diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000..a158744534 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +*.md linguist-detectable=true +*.md linguist-documentation=false diff --git a/.nojekyll b/.nojekyll new file mode 100644 index 0000000000..e69de29bb2 diff --git a/0000-template.md b/0000-template.md deleted file mode 100644 index fcc4a05252..0000000000 --- a/0000-template.md +++ /dev/null @@ -1,59 +0,0 @@ -# Feature name - -* Proposal: [SE-NNNN](NNNN-filename.md) -* Authors: [Author 1](https://github.com/swiftdev), [Author 2](https://github.com/swiftdev) -* Review Manager: TBD -* Status: **Awaiting review** - -*During the review process, add the following fields as needed:* - -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/), [Additional Commentary](https://lists.swift.org/pipermail/swift-evolution/) -* Bugs: [SR-NNNN](https://bugs.swift.org/browse/SR-NNNN), [SR-MMMM](https://bugs.swift.org/browse/SR-MMMM) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md) -* Previous Proposal: [SE-XXXX](XXXX-filename.md) - -## Introduction - -A short description of what the feature is. Try to keep it to a -single-paragraph "elevator pitch" so the reader understands what -problem this proposal is addressing. - -Swift-evolution thread: [Discussion thread topic for that proposal](https://lists.swift.org/pipermail/swift-evolution/) - -## Motivation - -Describe the problems that this proposal seeks to address. If the -problem is that some common pattern is currently hard to express, show -how one can currently get a similar effect and describe its -drawbacks. If it's completely new functionality that cannot be -emulated, motivate why this new functionality would help Swift -developers create better Swift code. - -## Proposed solution - -Describe your solution to the problem. Provide examples and describe -how they work. Show how your solution is better than current -workarounds: is it cleaner, safer, or more efficient? - -## Detailed design - -Describe the design of the solution in detail. If it involves new -syntax in the language, show the additions and changes to the Swift -grammar. If it's a new API, show the full API and its documentation -comments detailing what it does. The detail in this section should be -sufficient for someone who is *not* one of the authors to be able to -reasonably implement the feature. - -## Impact on existing code - -Describe the impact that this change will have on existing code. Will some -Swift applications stop compiling due to this change? Will applications still -compile but produce different behavior than they used to? Is it -possible to migrate existing Swift code to use a new feature or API -automatically? - -## Alternatives considered - -Describe alternative approaches to addressing the same problem, and -why you chose this approach instead. - diff --git a/404.html b/404.html new file mode 100644 index 0000000000..13f44ec1f4 --- /dev/null +++ b/404.html @@ -0,0 +1,9 @@ + + + + +Swift Evolution diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000..dce8b59b25 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,18 @@ +# Welcome to the Swift community! + +Contributions to Swift are welcomed and encouraged! Please see the [Contributing to Swift guide](https://www.swift.org/contributing/) and check out the [structure of the community](https://www.swift.org/community/#community-structure). + +To be a truly great community, Swift needs to welcome developers from all walks of life, with different backgrounds, and with a wide range of experience. A diverse and friendly community will have more great ideas, more unique perspectives, and produce more great code. We will work diligently to make the Swift community welcoming to everyone. + +To give clarity of what is expected of our members, Swift has adopted the code of conduct defined by the Contributor Covenant. This document is used across many open source communities, and we think it articulates our values well. For more, see the [Code of Conduct](https://www.swift.org/code-of-conduct/). + +## Contributing to Swift Evolution + +This repository is not your standard codebase. It houses Swift evolution proposals and related process documents, mostly composed of markdown and text files. Pull requests that make minor editorial and administrative changes are always welcome, including fixing typos and grammar mistakes, repairing links, and maintaining document and repository metadata. Other pull requests must follow the [Swift evolution process](process.md): + +- New proposals and substantive changes to existing proposals should be [pitched on the evolution forums](https://forums.swift.org/c/evolution/pitches/5) before a PR is opened here. +- Substantive changes to existing proposals require the permission of the proposal authors. +- Substantive changes to existing proposals require the approval of the appropriate evolution workgroup if the proposal is in an Active Review, Accepted, or Rejected state. +- New vision documents and substantive changes to existing vision documents require the approval of the appropriate evolution workgroup. +- Substantive changes to the evolution process require the approval of the Core Team. +- Conversations about the substance of a proposal should be held in an appropriate forums thread rather than in PR comments. This centralizes the discussion and allows more of the community to participate. diff --git a/README.md b/README.md index eef9df1b3b..47a172b81a 100644 --- a/README.md +++ b/README.md @@ -1,113 +1,33 @@ -# Swift Programming Language Evolution - -**Before you initiate a pull request**, please read the process document. Ideas should be thoroughly discussed on the [swift-evolution mailing list](https://swift.org/community/#swift-evolution) first. - -This repository tracks the ongoing evolution of Swift. It contains: - -* Goals for upcoming Swift releases (this document). -* The [Swift evolution review status][proposal-status] tracking proposals to change Swift. -* The [Swift evolution process](process.md) that governs the evolution of Swift. -* [Commonly Rejected Changes](commonly_proposed.md), proposals which have been denied in the past. - -This document describes goals for the Swift language on a per-release -basis, usually listing minor releases adding to the currently shipping -version and one major release out. Each release will have many -smaller features or changes independent of these larger goals, and not -all goals are reached for each release. - -Goals for past versions are included at the bottom of the document for -historical purposes, but are not necessarily indicative of the -features shipped. The release notes for each shipped version are the -definitive list of notable changes in each release. - -## Development major version: Swift 3.0 - -Expected release date: Late 2016 - -The primary goal of this release is to solidify and mature the Swift language and -development experience. While source breaking changes to the language have been -the norm for Swift 1 through 3, we would like the Swift 3.x (and Swift 4+) -languages to be as *source compatible* with Swift 3.0 as reasonably possible. -However, this will still be best-effort: if there is a really good reason to -make a breaking change beyond Swift 3, we will consider it and find the least -invasive way to roll out that change (e.g. by having a long deprecation cycle). - -To achieve this end, Swift 3 focuses on getting the basics right for the -long term: - -* **API design guidelines**: The way in which Swift is used in popular - libraries has almost as much of an effect on the character of Swift - code as the Swift language itself. The [API naming and design - guidelines](https://swift.org/documentation/api-design-guidelines/) are a - carefully crafted set of guidelines for building great Swift APIs. - -* **Automatic application of naming guidelines to imported Objective-C APIs**: - When importing Objective-C APIs, the Swift 3 compiler - [automatically maps](proposals/0005-objective-c-name-translation.md) methods - into the new Swift 3 naming guidelines, and provides a number of Objective-C - features to control and adapt this importing. - -* **Adoption of naming guidelines in key APIs**: The Swift Standard Library has - been significantly overhauled to embrace these guidelines, and key libraries - like [Foundation](proposals/0069-swift-mutability-for-foundation.md) and - [libdispatch](proposals/0088-libdispatch-for-swift3.md) have seen major - updates, which provide the consistent development experience we seek. - -* **Swiftification of imported Objective-C APIs**: Beyond the naming guidelines, - Swift 3 provides an improved experience for working with Objective-C APIs. - This includes importing - [Objective-C generic classes](proposals/0057-importing-objc-generics.md), - providing the ability to [import C APIs](proposals/0044-import-as-member.md) - into an "Object Oriented" style, much nicer - [imported string enums](proposals/0033-import-objc-constants.md), safer - syntax to work with [selectors](proposals/0022-objc-selectors.md) and - [keypaths](proposals/0062-objc-keypaths.md), etc. - -* **Focus and refine the language**: Since Swift 3 is the last release to make - major source breaking changes, it is also the right release to reevaluate the - syntax and semantics of the core language. This means that some obscure or - problematic features will be removed, we focus on improving consistency of - syntax in many small ways (e.g. by - [revising handling of parameter labels](proposals/0046-first-label.md), and - focus on forward looking improvements to the type system. This serves the - overall goal of making Swift a simpler, more predictable, and more consistent - language over the long term. - -* **Improvements to tooling quality**: The overall quality of the compiler is - really important to us: it directly affects the joy of developing in Swift. - Swift 3 focuses on fixing bugs in the compiler and IDE features, improving the - speed of compile times and incremental builds, improving the performance of - the generated code, improving the precision of error and warning messages, etc. - -One of the reasons that stability is important is that **portability** to non-Apple -systems is also a strong goal of Swift 3. This release enables -broad scale adoption across multiple platforms, including significant -functionality in the [Swift core libraries](https://swift.org/core-libraries/) -(Foundation, libdispatch, XCTest, etc). A useful Linux/x86 port is -already available (enabling many interesting server-side scenarios), and work is -underway across the community to bring Swift to FreeBSD, Raspberry Pi, Android, -Windows, and others. While we don't know which platforms will reach a useful -state by the launch of Swift 3, significant effort continues to go into making -the compiler and runtime as portable as practically possible. - -Finally, Swift 3 also includes a mix of relatively small but important additions -to the language and standard library that make solving common problems easier and -make everything feel nicer. A detailed list of accepted proposals is included -on the [proposal status page][proposal-status]. - - -## Swift 2.2 - Released on March 21, 2016 - -[This release](https://swift.org/blog/swift-2-2-released/) focused on fixing -bugs, improving quality-of-implementation (QoI) -with better warnings and diagnostics, improving compile times, and improving -performance. It put some finishing touches on features introduced in Swift 2.0, -and included some small additive features that don't break Swift code or -fundamentally change the way Swift is used. As a step toward Swift 3, it -introduced warnings about upcoming source-incompatible changes in Swift 3 -so that users can begin migrating their code sooner. - -Aside from warnings, a major goal of this release was to be as source compatible -as practical with Swift 2.0. - -[proposal-status]: https://apple.github.io/swift-evolution/ +# Swift Evolution + + + +This repository tracks the ongoing evolution of the Swift programming language, standard library, and package manager. + +## Goals and Release Notes + +* [Swift Language focus areas heading into 2025](https://forums.swift.org/t/swift-language-focus-areas-heading-into-2025/76611) +* [CHANGELOG](https://github.com/apple/swift/blob/main/CHANGELOG.md) + +| Version | Announced | Released | +| :-------- | :----------------------------------------------------------------------- | :----------------------------------------------------------- | +| Swift 6.2 | [2025-03-08](https://forums.swift.org/t/swift-6-2-release-process/78371) | [2025-09-15](https://www.swift.org/blog/swift-6.2-released/) | +| Swift 6.1 | [2024-10-17](https://forums.swift.org/t/swift-6-1-release-process/75442) | [2025-03-31](https://www.swift.org/blog/swift-6.1-released/) | +| Swift 6.0 | [2024-02-22](https://forums.swift.org/t/swift-6-0-release-process/70220) | [2024-09-17](https://www.swift.org/blog/announcing-swift-6/) | +| Swift 5.10 | [2023-08-23](https://forums.swift.org/t/swift-5-10-release-process/66911) | [2024-03-05](https://www.swift.org/blog/swift-5.10-released/) | +| Swift 5.9 | [2023-03-06](https://forums.swift.org/t/swift-5-9-release-process/63557) | [2023-09-18](https://www.swift.org/blog/swift-5.9-released/) | +| Swift 5.8 | [2022-11-19](https://forums.swift.org/t/swift-5-8-release-process/61540) | [2023-03-30](https://www.swift.org/blog/swift-5.8-released/) | +| Swift 5.7 | [2022-03-29](https://forums.swift.org/t/swift-5-7-release-process/56316) | [2022-09-12](https://www.swift.org/blog/swift-5.7-released/) | +| Swift 5.6 | [2021-11-10](https://forums.swift.org/t/swift-5-6-release-process/53412) | [2022-03-14](https://www.swift.org/blog/swift-5.6-released/) | +| Swift 5.5 | [2021-03-12](https://forums.swift.org/t/swift-5-5-release-process/45644) | [2021-09-20](https://www.swift.org/blog/swift-5.5-released/) | +| Swift 5.4 | [2020-11-11](https://forums.swift.org/t/swift-5-4-release-process/41936) | [2021-04-26](https://www.swift.org/blog/swift-5.4-released/) | +| Swift 5.3 | [2020-03-25](https://www.swift.org/blog/5.3-release-process/) | [2020-09-16](https://www.swift.org/blog/swift-5.3-released/) | +| Swift 5.2 | [2019-09-24](https://www.swift.org/blog/5.2-release-process/) | [2020-03-24](https://www.swift.org/blog/swift-5.2-released/) | +| Swift 5.1 | [2019-02-18](https://www.swift.org/blog/5.1-release-process/) | [2019-09-20](https://www.swift.org/blog/swift-5.1-released/) | +| Swift 5.0 | [2018-09-25](https://www.swift.org/blog/5.0-release-process/) | [2019-03-25](https://www.swift.org/blog/swift-5-released/) | +| Swift 4.2 | [2018-02-28](https://www.swift.org/blog/4.2-release-process/) | [2018-09-17](https://www.swift.org/blog/swift-4.2-released/) | +| Swift 4.1 | [2017-10-17](https://www.swift.org/blog/swift-4.1-release-process/) | [2018-03-29](https://www.swift.org/blog/swift-4.1-released/) | +| Swift 4.0 | [2017-02-16](https://www.swift.org/blog/swift-4.0-release-process/) | [2017-09-19](https://www.swift.org/blog/swift-4.0-released/) | +| Swift 3.1 | [2016-12-09](https://www.swift.org/blog/swift-3.1-release-process/) | [2017-03-27](https://www.swift.org/blog/swift-3.1-released/) | +| Swift 3.0 | [2016-05-06](https://www.swift.org/blog/swift-3.0-release-process/) | [2016-09-13](https://www.swift.org/blog/swift-3.0-released/) | +| Swift 2.2 | [2016-01-05](https://www.swift.org/blog/swift-2.2-release-process/) | [2016-03-21](https://www.swift.org/blog/swift-2.2-released/) | diff --git a/commonly_proposed.md b/commonly_proposed.md index 5f92c308fc..d2d2bdc8ca 100644 --- a/commonly_proposed.md +++ b/commonly_proposed.md @@ -1,49 +1,37 @@ # Commonly Rejected Changes - + This is a list of changes to the Swift language that are frequently proposed but that are unlikely to be accepted. If you're interested in pursuing something in this space, please familiarize yourself with the discussions that we have already had. In order to bring one of these topics up, you'll be expected to add new information to the discussion, not just to say, "I really want this" or "this exists in some other language and I liked it there". -Additionally, proposals for out-of-scope changes will not be scheduled for review. The [readme file](README.md) identifies a list of priorities for the next major release of Swift, and the [proposal review status page](https://apple.github.io/swift-evolution/) includes a list of changes that have been deferred for future discussion because they were deemed to be out of scope at the time of review (in addition to a list of changes proposed and rejected after a formal review). +Additionally, proposals for out-of-scope changes will not be scheduled for review. The [readme file](README.md) identifies a list of priorities for the next major release of Swift, and the [dashboard](https://www.swift.org/swift-evolution/) includes a list of changes that have been rejected after a formal review. -Several of the discussions below refer to "C family" languages. This is intended to mean the extended family of languages that resemble C at a syntactic level. This includes languages like C++, C#, Objective-C, Java, and Javascript. +Several of the discussions below refer to "C family" languages. This is intended to mean the extended family of languages that resemble C at a syntactic level, such as C++, C#, Objective-C, Java, and JavaScript. Swift embraces its C heritage. Where it deviates from other languages in the family, it does so because the feature was thought actively harmful (such as the pre/post-increment `++`) or to reduce needless clutter (such as `;` or parentheses in `if` statements). ## Basic Syntax and Operators - * [Replace `{}` brace syntax with Python-style indentation](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003656.html): Surely a polarizing issue, but Swift will not change to use indentation for scoping instead of curly braces. - - * [Remove `;` semicolons](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002421.html): Semicolons within a line are an intentional expressivity feature. Semicolons at the end of the line should be handled by a linter, not by the compiler. + * [Replace `{}` brace syntax with Python-style indentation](https://forums.swift.org/t/brace-syntax/678/3): Surely a polarizing issue, but Swift will not change to use indentation for scoping instead of curly braces. - * [Replace logical operators (`&&`, `||`, etc.) with words like "and" and "or"](https://lists.swift.org/pipermail/swift-evolution/2015-December/000032.html), [allowing non-punctuation as operators](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005669.html) and infix functions: The operator and identifier grammars are intentionally partitioned in Swift, which is a key part of how user-defined overloaded operators are supported. Requiring the compiler to see the "operator" declaration to know how to parse a file would break the ability to be able to parse a Swift file without parsing all of its imports. This has a major negative effect on tooling support. + * [Remove `;` semicolons](https://forums.swift.org/t/proposal-to-remove-semicolons/523/3): Semicolons within a line are an intentional expressivity feature. Semicolons at the end of the line should be handled by a linter, not by the compiler. - * [Replace `?:` ternary operator](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002609.html): Definitely magical, but it serves a very important use-case for terse selection of different values. Proposals for alternatives have been intensely discussed, but none have been "better enough" for it to make sense to diverge from the precedent established by the C family of languages. + * [Replace logical operators (`&&`, `||`, `!`, etc.) with words like "and", "or", "not"](https://forums.swift.org/t/change-the-name-of-the-boolean-operators/30/2), and [allow non-punctuation operators](https://forums.swift.org/t/allowing-characters-for-use-as-custom-operators/952) and infix functions: The operator and identifier grammars are intentionally partitioned in Swift, which is a key part of how user-defined overloaded operators are supported. Requiring the compiler to see the "operator" declaration to know how to parse a file would break the ability to be able to parse a Swift file without parsing all of its imports. This has a major negative effect on tooling support. While not needing infix support, `not` would need operator or keyword status to omit the parentheses as `!` can, and `not somePredicate()` visually binds too loosely compared to `!somePredicate()`. -## Strings, Characters, and Collection Types + * [Replace `?:` ternary operator](https://forums.swift.org/t/ternary-operator-suggestion/49/148): Definitely magical, but it serves a very important use-case for terse selection of different values. Proposals for alternatives have been intensely discussed, but none have been "better enough" for it to make sense to diverge from the precedent established by the C family of languages. - * [Single-quotes `''` for character literals](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/003977.html): Swift takes the approach of highly valuing Unicode. However, there are multiple concepts of a character that could make sense in Unicode, and none is so much more commonly used than the others that it makes sense to privilege them. We'd rather save single quoted literals for a greater purpose (e.g. non-escaped string literals). +## Strings, Characters, and Collection Types - * Make `Array` subscript access return `T?` or `T!` instead of `T`: The current array behavior is [intentional](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002446.html), as it accurately reflects the fact that out-of-bounds array access is a logic error. Changing the current behavior would slow `Array` accesses to an unacceptable degree. This topic has come up [multiple](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002425.html) times before but is very unlikely to be accepted. + * Change `Array` subscript access to return `T?` or `T!` instead of `T`: The current array behavior is [intentional](https://forums.swift.org/t/proposal-add-safe-subquence-access-via-subscript-for-colloctiontype/516/7), as it accurately reflects the fact that out-of-bounds array access is a logic error. Changing the current behavior would slow `Array` accesses to an unacceptable degree. Changing the unlabeled array subscript to return an optional has come up [multiple](https://forums.swift.org/t/proposal-add-safe-subquence-access-via-subscript-for-colloctiontype/516/5) times before and is very unlikely to be accepted. ## Control Flow, Closures, Optional Binding, and Error Handling - * [`if/else` and `switch` as expressions](https://lists.swift.org/pipermail/swift-evolution/2015-December/000393.html): These are conceptually interesting things to support, but many of the problems solved by making these into expressions are already solved in Swift in other ways. Making them expressions introduces significant tradeoffs, and on balance, we haven't found a design that is clearly better than what we have so far. - - * [Replace `continue` keyword with synonyms from other scripting languages (e.g. next, skip, advance, etc)](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/004407.html): Swift is designed to feel like a member of the C family of languages. Switching keywords away from C precedent without strong motivation is a non-goal. - - * [Remove support for `default:` in `switch` and just use `case _:`](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001422.html): `default` is widely used, `case _` is too magical, and `default` is widely precedented in many C family languages. - - * [Rename `guard` to `unless`](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005534.html): It is a common request that `guard` be renamed `unless`. People requesting this change argue that `guard` is simply a logically inverted `if` statement, and therefore `unless` is a more obvious keyword. However, such requests stem from a fundamental misunderstanding of the functionality provided by `guard`. Unlike `if`, `guard` *enforces* that the code within its curly braces provides an early exit from the codepath. In other words, a `guard` block **must** `return`, `throw`, `break`, `continue` or call a function that does not return, such as `fatalError()`. This differs from `if` quite significantly, and therefore the parallels assumed between `guard` and `if` are not valid. - - * [Change closure literal syntax](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002583.html): Closure syntax in Swift has been carefully debated internally, and aspects of the design have strong motivations. It is unlikely that we'll find something better, and any proposals to change it should have a very detailed understanding of the Swift grammar. - - * [Use pattern-matching in `if let` instead of optional-unwrapping](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160201/008979.html): We actually tried this and got a lot of negative feedback, for several reasons: (1) Most developers don't think about things in "pattern matching" terms, they think about "destructuring". (2) The vastly most common use case for `if let` is actually for optional matching, and this change made the common case more awkward. (3) This change increases the learning curve of Swift, changing pattern matching from being a concept that can be learned late to something that must be confronted early. (4) The current design of `if case` unifies "pattern matching" around the `case` keyword. (5) If a developer unfamiliar with `if case` runs into one in some code, they can successfully search for it in a search engine or Stack Overflow. - - * [Syntactic sugar for `if let` self-assignment](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160829/026796.html): An alternative syntax (such as `if let foo? { ... }` or `if let foo=? { ... }`) to serve as a shorthand for `if let foo = foo { ... }` is often proposed and rejected because it is favoring terseness over clarity by introducing new magic syntactic sugar. - - * [Replace the `do`/`try`/`repeat` keywords with C++-style syntax](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151228/004630.html): Swift's error handling approach is carefully designed to make it obvious to maintainers of code when a call can "throw" an error. It is intentionally designed to be syntactically similar in some ways, but different in other key ways, to exception handling in other languages. Its design is a careful balance that favors maintainers of code that uses errors, to make sure someone reading the code understands what can throw. Before proposing a change to this system, please read the [Error Handling Rationale and Proposal](https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst) in full to understand why the current design is the way it is, and be ready to explain why your changes would be worth unbalancing this design. + * [Replace `continue` keyword with synonyms from other scripting languages (e.g. next, skip, advance, etc)](https://forums.swift.org/t/replace-continue-keyword/764/2): Swift is designed to feel like a member of the C family of languages. Switching keywords away from C precedent without strong motivation is a non-goal. + * [Remove support for `default:` in `switch` and just use `case _:`](https://forums.swift.org/t/remove-default-case-in-switch-case/360/4): `default` is widely used, `case _` is too magical, and `default` is widely precedented in many C family languages. + * [Rename `guard` to `unless`](https://forums.swift.org/t/rename-guard-to-unless/934/7): It is a common request that `guard` be renamed `unless`. People requesting this change argue that `guard` is simply a logically inverted `if` statement, and therefore `unless` is a more obvious keyword. However, such requests stem from a fundamental misunderstanding of the functionality provided by `guard`. Unlike `if`, `guard` *enforces* that the code within its curly braces provides an early exit from the codepath. In other words, a `guard` block **must** `return`, `throw`, `break`, `continue` or call a function that does not return, such as `fatalError()`. This differs from `if` quite significantly, and therefore the parallels assumed between `guard` and `if` are not valid. + * [Infer `return` for omitted `guard` body](https://forums.swift.org/t/inferred-return-for-guard-statement/12099/11): It has been proposed many times to allow omission of the `guard` body for the sake of brevity. However, a core principle of Swift is to make control flow explicit and visible. For example, the `try` keyword exists solely to indicate to the human reader where thrown errors can happen. Implicit returns would violate this principle, favoring terseness over clarity in a way that isn't typical of Swift. Furthermore, there are many ways of exiting the scope other than `return` (loops may want `break` or `continue`), and not every function has an obvious default value to return. + * [Change closure literal syntax](https://forums.swift.org/t/streamlining-closures/487/3): Closure syntax in Swift has been carefully debated internally, and aspects of the design have strong motivations. It is unlikely that we'll find something better, and any proposals to change it should have a very detailed understanding of the Swift grammar. + * [Use pattern-matching in `if let` instead of optional-unwrapping](https://forums.swift.org/t/obsoleting-if-let/1301/4): We actually tried this and got a lot of negative feedback, for several reasons: (1) Most developers don't think about things in "pattern matching" terms, they think about "destructuring". (2) The vastly most common use case for `if let` is actually for optional matching, and this change made the common case more awkward. (3) This change increases the learning curve of Swift, changing pattern matching from being a concept that can be learned late to something that must be confronted early. (4) The current design of `if case` unifies "pattern matching" around the `case` keyword. (5) If a developer unfamiliar with `if case` runs into one in some code, they can successfully search for it in a search engine or Stack Overflow. + * [Remove or deprecate the force-unwrap operator `!`](https://forums.swift.org/t/moving-toward-deprecating-force-unwrap-from-swift/43455/82): Force-unwrap and force-try are legitimately useful parts of the language, and not just for source stability reasons. Therefore, proposals to deprecate or remove the force-unwrap operator (or `try!`), even in a mode enabled via compiler flag, will not be considered by the core team. Whether the Swift compiler should gain a more general "linting" capability to guide coding style remains a possible topic of discussion. + * [Replace the `do`/`try`/`repeat` keywords with C++-style syntax](https://forums.swift.org/t/use-standard-syntax-instead-of-do-and-repeat/791/2): Swift's error handling approach is carefully designed to make it obvious to maintainers of code when a call can "throw" an error. It is intentionally designed to be syntactically similar in some ways, but different in other key ways, to exception handling in other languages. Its design is a careful balance that favors maintainers of code that uses errors, to make sure someone reading the code understands what can throw. Before proposing a change to this system, please read the [Error Handling Rationale and Proposal](https://github.com/swiftlang/swift/blob/main/docs/ErrorHandlingRationale.md) in full to understand why the current design is the way it is, and be ready to explain why your changes would be worth unbalancing this design. ## Miscellaneous - * [Use garbage collection (GC) instead of automatic reference counting (ARC)](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160208/009403.html): Mark-and-sweep garbage collection is a well-known technique used in many popular and widely used languages (e.g., Java and JavaScript) and it has the advantage of automatically collecting reference cycles that ARC requires the programmer to reason about. That said, garbage collection has a [large number of disadvantages](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160208/009422.html) and using it would prevent Swift from successfully targeting a number of systems programming domains. For example, real-time systems (video or audio processing), deeply embedded controllers, and most kernels find GC to be generally unsuitable. Further, GC is only efficient when given 3–4× more memory to work with than the process is using at any time, and this tradeoff is not acceptable for Swift. - - * [Disjunctions (logical ORs) in type constraints](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000182.html): These include anonymous union-like types (e.g. `(Int | String)` for a type that can be inhabited by either an integer or a string). "[This type of constraint is] something that the type system cannot and should not support." - - * [Rewrite the Swift compiler in Swift](https://github.com/apple/swift/blob/2c7b0b22831159396fe0e98e5944e64a483c356e/www/FAQ.rst): This would be a lot of fun someday, but (unless you include rewriting all of LLVM) requires the ability to import C++ APIs into Swift. Additionally, there are lots of higher priority ways to make Swift better. + * [Use garbage collection (GC) instead of automatic reference counting (ARC)](https://forums.swift.org/t/what-about-garbage-collection/1360): Mark-and-sweep garbage collection is a well-known technique used in many popular and widely used languages (e.g., Java and JavaScript) and it has the advantage of automatically collecting reference cycles that ARC requires the programmer to reason about. That said, garbage collection has a [large number of disadvantages](https://forums.swift.org/t/what-about-garbage-collection/1360/6) and using it would prevent Swift from successfully targeting a number of systems programming domains. For example, real-time systems (video or audio processing), deeply embedded controllers, and most kernels find GC to be generally unsuitable. Further, GC is only efficient when given 3–4× more memory to work with than the process is using at any time, and this tradeoff is not acceptable for Swift. + * [Disjunctions (logical ORs) in type constraints](https://forums.swift.org/t/returned-for-revision-se-0095-replace-protocol-p1-p2-syntax-with-any-p1-p2/2855): These include anonymous union-like types (e.g. `(Int | String)` for a type that can be inhabited by either an integer or a string). "[This type of constraint is] something that the type system cannot and should not support." diff --git a/index.xml b/index.xml deleted file mode 100644 index f87caf7330..0000000000 --- a/index.xml +++ /dev/null @@ -1,151 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/index.xslt b/index.xslt deleted file mode 100644 index 3c1a7900cf..0000000000 --- a/index.xslt +++ /dev/null @@ -1,261 +0,0 @@ - - - - - - - - - - - - Swift-Evolution Proposal Status - - - -

Swift Programming Language Evolution: Proposal Status

- -

The Swift evolution process describes the process by which Swift evolves. This page tracks the currently active proposals in that process.

- - - Active reviews - - - - - Upcoming reviews - - - - - Proposals awaiting scheduling - - - - - Accepted (awaiting implementation) - This is the list of proposals which have been accepted for inclusion into Swift, but they are not implemented yet, and may not have anyone signed up to implement them. If they are not implemented in time for Swift 3, they will roll into a subsequent release. - - - - - Implemented (Swift 3.1) - - - - - Implemented (Swift 3) - - - - - Implemented (Swift 2.2) - - - - - Deferred for future discussion - - - - - Returned for revision - - - - - Rejected - - - - - Withdrawn - - - - -
- - - - - - -
-

-

- - -

(none)

-
- - - - - -
-
-
-
-
- - - - - SE- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
diff --git a/process.md b/process.md index 9c1b8fac72..3408e2d567 100644 --- a/process.md +++ b/process.md @@ -1,10 +1,28 @@ # Swift Evolution Process -Swift is a powerful and intuitive programming language that is designed to make writing and maintaining correct programs easier. Swift is growing and evolving, guided by a community-driven process referred to as the Swift evolution process. This document outlines the Swift evolution process and how a feature grows from a rough idea into something that can improve the Swift development experience for millions of programmers. +Swift is a powerful and intuitive programming language that is designed to make writing and maintaining correct programs easier. Swift is growing and evolving, guided by a community-driven process referred to as the Swift evolution process, maintained by the [Language Steering Group][language-steering-group]. This document outlines the Swift evolution process and how a feature grows from a rough idea into something that can improve the Swift development experience for millions of programmers. ## Scope -The Swift evolution process covers all changes to the Swift language and the public interface of the Swift standard library, including new language features and APIs (no matter how small), changes to existing language features or APIs, removal of existing features, and so on. Smaller changes, such as bug fixes, optimizations, or diagnostic improvements can be contributed via the normal contribution process; see [Contributing to Swift](https://swift.org/community/#contributing). +The Swift evolution process covers all design changes, no matter how small, to the Swift language, its standard library, and the core tools necessary to build Swift programs. This includes additions, removals, and changes to: + +- the features of the Swift language, +- the public interface of the Swift standard library, +- the configuration of the Swift compiler, +- the core tools of the Swift package ecosystem, including the configuration of + the [Swift package manager](https://www.swift.org/package-manager/) and the + design of its manifest files, and +- the public interfaces of the following libraries: + - [Swift Testing](https://github.com/swiftlang/swift-testing) + - [XCTest](https://github.com/swiftlang/swift-corelibs-xctest) + +The design of other tools, such as IDEs, debuggers, and documentation generators, is not covered by the evolution process. The Core Team may create workgroups to guide and make recommendations about the development of these tools, but the output of those workgroups is not reviewed. + +The evolution process does not cover experimental features, which can be added, changed, or removed at any time. Implementors should take steps to prevent the accidental use of experimental features, such as by enabling them only under explicitly experimental options. Features should not be allowed to remain perpetually experimental; a feature with no clear path for development into an official feature should be removed. + +Changes such as bug fixes, optimizations, or diagnostic improvements can be contributed via the normal contribution process; see [Contributing to Swift](https://www.swift.org/contributing/). Some bug fixes are effectively substantial changes to the design, even if they're just making the implementation match the official documentation; whether such a change requires evolution review is up to the appropriate evolution workgroup. + +Which parts of the Swift project are covered by the evolution process is ultimately up to the judgment of the Core Team. ## Goals @@ -15,17 +33,39 @@ The Swift evolution process aims to leverage the collective ideas, insights, and There is a natural tension between these two goals. Open evolution processes are, by nature, chaotic. Yet, maintaining a coherent vision for something as complicated as a programming language requires some level of coordination. The Swift evolution process aims to strike a balance that best serves the Swift community as a whole. +## Community structure + +The [Core Team](https://www.swift.org/community/#core-team) is responsible for the strategic direction of Swift. The Core Team creates workgroups focused on specific parts of the project. When the Core Team gives a workgroup authority over part of the evolution of the project, that workgroup is called an evolution workgroup. Evolution workgroups manage the evolution process for proposals under their authority, working together with other workgroups as needed. + +Currently, there are three evolution workgroups: + +* The [Language Steering Group][language-steering-group] has authority over the evolution of the Swift language, its standard library, and any language configuration features of the Swift package manager. +* The [Platform Steering Group][platform-steering-group] has authority over the evolution of all other features of the Swift package manager and its manifest files. +* The [Testing Workgroup][testing-workgroup] has authority over the evolution of + the Swift Testing and Corelibs XCTest projects. + +The Core Team manages (or delegates) the evolution process for proposals outside these areas. The Core Team also retains the power to override the evolution decisions of workgroups when necessary. + +## Proposals, roadmaps, and visions + +There are three kinds of documents commonly used in the evolution process. + +* An evolution *proposal* describes a specific proposed change in detail. All evolution changes are advanced as proposals which will be discussed in the community and given a formal open review. + +* An evolution *roadmap* describes a concrete plan for how a complex change will be broken into separate proposals that can be individually pitched and reviewed. Considering large changes in small pieces allows the community to provide more focused feedback about each part of the change. A roadmap makes this organization easier for community members to understand. + + Roadmaps are planning documents that do not need to be reviewed. + +* An evolution *vision* describes a high-level design for a broad topic (for example, string processing or concurrency). A vision creates a baseline of understanding in the community for future conversations on that topic, setting goals and laying out a possible program of work. + + Visions must be approved by the appropriate evolution workgroup. This approval is an endorsement of the vision's basic ideas, but not of any of its concrete proposals, which must still be separately developed and reviewed. + ## Participation Everyone is welcome to propose, discuss, and review ideas to improve -the Swift language and standard library on the [swift-evolution -mailing list][swift-evolution-mailing-list]. Before posting a review, -please see the section "What goes into a review?" below. - -The Swift [core team](https://swift.org/community/#core-team) is -responsible for the strategic direction of Swift. Core team members -initiate, participate in, and manage the public review of proposals -and have the authority to accept or reject changes to Swift. +the Swift language and standard library in the +[Evolution section of the Swift forums](https://forums.swift.org/c/evolution). +Before posting a review, please see the section "What goes into a review?" below. ## What goes into a review? @@ -44,105 +84,196 @@ Please state explicitly whether you believe that the proposal should be accepted ## How to propose a change -* **Check prior proposals**: many ideas come up frequently, and may either be in active discussion on the mailing list, or may have been discussed already and have joined the [Commonly Rejected Proposals](commonly_proposed.md) list. Please check the mailing list archives and this list for context before proposing something new. +1. **Check prior proposals** + + Many ideas come up frequently, and may either be in active discussion on the forums, or may have been discussed already and have joined the [Commonly Rejected Proposals](commonly_proposed.md) list. Please [search the forums](https://forums.swift.org/search) for context before proposing something new. + +1. **Consider the goals of the upcoming Swift release** + + Each major Swift release is focused on a [specific set of goals](README.md) + described early in the release cycle. When proposing a change to + Swift, please consider how your proposal fits in with the larger goals + of the upcoming Swift release. Proposals that are clearly out of scope + for the upcoming Swift release will not be brought up for review. If you can't resist discussing a proposal that you know is out of scope, please include the tag `[Out of scope]` in the subject. + +1. **Socialize the idea** + + Propose a rough sketch of the idea in the ["pitches" section of the Swift forums](https://forums.swift.org/c/evolution/pitches), the problems it solves, what the solution looks like, etc., to gauge interest from the community. + +1. **Develop the proposal and implementation** + + 1. Expand the rough sketch into a formal proposal using the + [relevant proposal template](#proposal-templates). + 1. In the [swift-evolution repository][swift-evolution-repo], open a + [draft pull request][draft-pr] that adds your proposal to the appropriate + [proposal directory](#proposal-locations). + 1. Announce the pull request on the forums and edit the root post to link out to the pull request. + 1. Refine the formal proposal in the open as you receive further feedback on the forums or the pull request. + A ripe proposal is expected to address commentary from present and past + discussions of the idea. + + Meanwhile, start working on an implementation. + Prototyping an implementation and its uses *alongside* the formal proposal + is important because it helps to determine an adequate scope, ensure + technical feasibility, and validate that the proposal lives up to + its motivation. + + A pull request with a working implementation is *required* for the + proposal to be accepted for review. + Proposals that can ship as part of the [Standard Library Preview package][preview-package] + should be paired with a pull request against the [swift-evolution-staging repository][swift-evolution-staging]. + All other proposals should be paired with an implementation pull request + against the [main Swift repository](https://github.com/swiftlang/swift). + + The preview package can accept new types, new protocols, and extensions to + existing types and protocols that can be implemented without access to + standard library internals or other non-public features. + For more information about the kinds of changes that can be implemented in + the preview package, see [SE-0264](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0264-stdlib-preview-package.md). + +1. **Request a review** + + Once you have a working implementation and believe the proposal is sufficiently detailed and clear, mark the draft pull request in the [swift-evolution repository][swift-evolution-repo] as ready for review to indicate to the appropriate evolution workgroup that you would like the proposal to be reviewed. + +> [!IMPORTANT] +> In general, and especially [during the review period](#review-process), be responsive to questions and feedback about the proposal. + +[draft-pr]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#draft-pull-requests + +### Proposal templates + +When writing a formal proposal document, start with the most relevant template +given the primary subject of the proposal: + +- Use the [Swift template][swift-template] for proposals concerning the Swift + language, compiler, or standard library. +- Use the [Swift Package Manager template][swiftpm-template] for proposals + related to SwiftPM, including the design of its package manifest files and + command-line tools. +- Use the [Swift Testing template][swift-testing-template] for proposals focused + on Swift Testing features or public interfaces. + +### Proposal locations -* **Socialize the idea**: propose a rough sketch of the idea on the [swift-evolution mailing list][swift-evolution-mailing-list], the problems it solves, what the solution looks like, etc., to gauge interest from the community. -* **Develop the proposal**: expand the rough sketch into a complete proposal, using the [proposal template](0000-template.md), and continue to refine the proposal on the evolution mailing list. Prototyping an implementation and its uses along with the proposal is encouraged, because it helps ensure both technical feasibility of the proposal as well as validating that the proposal solves the problems it is meant to solve. -* **Request a review**: initiate a pull request to the [swift-evolution repository][swift-evolution-repo] to indicate to the core team that you would like the proposal to be reviewed. When the proposal is sufficiently detailed and clear, and addresses feedback from earlier discussions of the idea, the pull request will be accepted. The proposal will be assigned a proposal number as well as a core team member to manage the review. -* **Address feedback**: in general, and especially [during the review period][proposal-status], be responsive to questions and feedback about the proposal. +When opening a pull request to add a new proposal to the +[swift-evolution repository][swift-evolution-repo], place proposals which use +the [Swift][swift-template] or [Swift Package Manager][swiftpm-template] +templates in the top-level [proposals](/proposals) directory. Place proposals +which use the newer [Swift Testing template][swift-testing-template] in the +[proposals/testing](/proposals/testing) subdirectory. ## Review process The review process for a particular proposal begins when a member of -the core team accepts a pull request of a new or updated proposal into -the [swift-evolution repository][swift-evolution-repo]. That core team +the appropriate evolution workgroup accepts a pull request of a new or updated proposal into +the [swift-evolution repository][swift-evolution-repo]. That member becomes the *review manager* for the proposal. The proposal -is assigned a proposal number (if it is a new proposal), then enters -the review queue. +is assigned a proposal number (if it is a new proposal), and then enters +the review queue. If your proposal's accompanying implementation takes the form of a package, the review manager will merge your pull request into a new branch in the [swift-evolution-staging repository][swift-evolution-staging]. The review manager will work with the proposal authors to schedule the review. Reviews usually last a single week, but can run longer for particularly large or complex proposals. When the scheduled review period arrives, the review manager will post -the proposal to the [swift-evolution mailing -list][swift-evolution-mailing-list] with the subject "[Review]" -followed by the proposal title and update the list of active +the proposal to the ["Proposal reviews" section of the Swift forums][proposal-reviews] +with the proposal title and update the list of active reviews. To avoid delays, it is important that the proposal authors be available to answer questions, address feedback, and clarify their intent during the review period. -After the review has completed, the core team will make a decision on +After the review has completed, the managing evolution workgroup will make a decision on the proposal. The review manager is responsible for determining -consensus among the core team members, then reporting their decision -to the proposal authors and mailing list. The review manager will +consensus among the workgroup members, then reporting their decision +to the proposal authors and forums. The review manager will update the proposal's state in the [swift-evolution repository][swift-evolution-repo] to reflect that decision. ## Proposal states + +```mermaid +flowchart LR + %% + + %% Nodes: + 1{{"Awaiting
review"}} + 2{{"Scheduled
for review"}} + 3{"Active
review"} + 4["Returned
for revision"] + 5(["Withdrawn"]) + 6(["Rejected"]) + 7_8["Accepted
(with revisions)"] + 9[["Previewing"]] + 10(["Implemented"]) + + %% Links: + 1 ==> 3 ==> 7_8 ==> 10 + 1 -.-> 2 -.-> 3 -.-> 4 -.-> 5 & 1 + 3 -.-> 6 + 7_8 -.-> 9 -.-> 10 +``` + A given proposal can be in one of several states: -* **Awaiting review**: The proposal is awaiting review. Once known, - the dates for the actual review will be placed in the proposal - document and updated in the [list of proposals](index.xml). When the - review period begins, the review manager will update the state to - *Active review*. -* **Scheduled for review (MONTH DAY...MONTH DAY)**: The public review of the proposal - on the [swift-evolution mailing list][swift-evolution-mailing-list] +* **Awaiting review**: The proposal is awaiting review. Once known, the dates + for the actual review will be placed in the proposal document. When the review + period begins, the review manager will update the state to *Active review*. +* **Scheduled for review (...)**: The public review of the proposal + in the [Swift forums][proposal-reviews] has been scheduled for the specified date range. -* **Active review (MONTH DAY...MONTH DAY)**: The proposal is undergoing public review - on the [swift-evolution mailing list][swift-evolution-mailing-list]. +* **Active review (...)**: The proposal is undergoing public review + in the [Swift forums][proposal-reviews]. The review will continue through the specified date range. * **Returned for revision**: The proposal has been returned from review for additional revision to the current draft. * **Withdrawn**: The proposal has been withdrawn by the original submitter. -* **Deferred**: Consideration of the proposal has been deferred - because it does not meet the [goals of the upcoming major Swift - release](README.md). Deferred proposals will be reconsidered when - scoping the next major Swift release. +* **Rejected**: The proposal has been considered and rejected. * **Accepted**: The proposal has been accepted and is either awaiting implementation or is actively being implemented. * **Accepted with revisions**: The proposal has been accepted, contingent upon the inclusion of one or more revisions. -* **Rejected**: The proposal has been considered and rejected. -* **Implemented (Swift VERSION)**: The proposal has been implemented. - Append the version number in parentheses—for example: Implemented (Swift 2.2). +* **Previewing**: The proposal has been accepted and is available for preview + in the [Standard Library Preview package][preview-package]. +* **Implemented (Swift Next)**: + The proposal has been implemented (for the specified version of Swift). If the proposal's implementation spans multiple version numbers, write the version number for which the implementation will be complete. -[swift-evolution-repo]: https://github.com/apple/swift-evolution "Swift evolution repository" -[swift-evolution-mailing-list]: https://swift.org/community/#swift-evolution "Swift evolution mailing list" -[proposal-status]: https://apple.github.io/swift-evolution/ +[swift-evolution-repo]: https://github.com/swiftlang/swift-evolution "Swift evolution repository" +[swift-evolution-staging]: https://github.com/swiftlang/swift-evolution-staging "Swift evolution staging repository" +[proposal-reviews]: https://forums.swift.org/c/evolution/proposal-reviews "'Proposal reviews' subcategory of the Swift forums" +[status-page]: https://www.swift.org/swift-evolution +[preview-package]: https://github.com/apple/swift-standard-library-preview/ +[language-steering-group]: https://www.swift.org/language-steering-group +[platform-steering-group]: https://www.swift.org/platform-steering-group +[testing-workgroup]: https://www.swift.org/testing-workgroup "Testing Workgroup page on Swift.org" +[swift-template]: proposal-templates/0000-swift-template.md "Swift proposal template" +[swiftpm-template]: proposal-templates/0000-swiftpm-template.md "Swift Package Manager proposal template" +[swift-testing-template]: proposal-templates/0000-swift-testing-template.md "Swift Testing proposal template" ## Review announcement -When a proposal enters review, an email using the following template will be -sent to the swift-evolution-announce and swift-evolution mailing lists: +When a proposal enters review, a new topic will be posted to the +["Proposal Reviews" subcategory of the Swift forums][proposal-reviews] using the +relevant announcement template below: --- +
+Swift language, compiler, and standard library + Hello Swift community, The review of "\<\>" begins now and runs through \<\>. The proposal is available here: -> +> https://linkToProposal -Reviews are an important part of the Swift evolution process. All reviews -should be sent to the swift-evolution mailing list at +Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager. When emailing the review manager directly, please keep the proposal link at the top of the message. -> +##### Trying it out -or, if you would like to keep your feedback private, directly to the -review manager. When replying, please try to keep the proposal link at -the top of the message: - -> Proposal link: ->> http://linkToProposal - -> Reply text - ->> Other replies +If you'd like to try this proposal out, you can [download a toolchain supporting it here](). You will need to add `-enable-experimental-feature FLAGNAME` to your build flags. \<\\> ##### What goes into a review? @@ -162,7 +293,7 @@ answer in your review: More information about the Swift evolution process is available at -> +> Thank you, @@ -170,4 +301,72 @@ Thank you, Review Manager ---- +
+ +
+Swift Testing public interfaces and features + +Hello Swift community, + +The review of "\<\>" begins now and runs through \<\>. +The proposal is available here: + +> https://linkToProposal + +Reviews are an important part of the Swift evolution process. All review +feedback should be either on this forum thread or, if you would like to keep +your feedback private, directly to the review manager. When emailing the review +manager directly, please keep the proposal link at the top of the message. + +##### Trying it out + +To try this feature out, add a dependency to the `main` branch of +`swift-testing` to your package: + +```swift +dependencies: [ + ... + .package(url: "https://github.com/swiftlang/swift-testing.git", branch: "main"), +] +``` + +Then, add a target dependency to your test target: + +```swift +.testTarget( + ... + dependencies: [ + ... + .product(name: "Testing", package: "swift-testing"), + ] +``` + +Finally, import Swift Testing using `@_spi(Experimental) import Testing`. + +##### What goes into a review? + +The goal of the review process is to improve the proposal under review through +constructive criticism and, eventually, determine the direction of Swift. When +writing your review, here are some questions you might want to answer in your +review: + +* What is your evaluation of the proposal? +* Is the problem being addressed significant enough to warrant a change to Swift + Testing? +* Does this proposal fit well with the feel and direction of Swift Testing? +* If you have used other languages or libraries with a similar feature, how do + you feel that this proposal compares to those? +* How much effort did you put into your review? A glance, a quick reading, or an + in-depth study? + +More information about the Swift evolution process is available at + +> https://github.com/swiftlang/swift-evolution/blob/main/process.md + +Thank you, + +-\<\> + +Review Manager + +
diff --git a/proposal-templates/0000-swift-template.md b/proposal-templates/0000-swift-template.md new file mode 100644 index 0000000000..c73d88ff67 --- /dev/null +++ b/proposal-templates/0000-swift-template.md @@ -0,0 +1,293 @@ +# Feature name + +* Proposal: [SE-NNNN](NNNN-filename.md) +* Authors: [Author 1](https://github.com/swiftdev), [Author 2](https://github.com/swiftdev) +* Review Manager: TBD +* Status: **Awaiting implementation** or **Awaiting review** +* Vision: *if applicable* [Vision Name](https://github.com/swiftlang/swift-evolution/visions/NNNNN.md) +* Roadmap: *if applicable* [Roadmap Name](https://forums.swift.org/...) +* Bug: *if applicable* [swiftlang/swift#NNNNN](https://github.com/swiftlang/swift/issues/NNNNN) +* Implementation: [swiftlang/swift#NNNNN](https://github.com/swiftlang/swift/pull/NNNNN) or [swiftlang/swift-evolution-staging#NNNNN](https://github.com/swiftlang/swift-evolution-staging/pull/NNNNN) +* Upcoming Feature Flag: *if applicable* `MyFeatureName` +* Previous Proposal: *if applicable* [SE-XXXX](XXXX-filename.md) +* Previous Revision: *if applicable* [1](https://github.com/swiftlang/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md) +* Review: ([pitch](https://forums.swift.org/...)) + +When filling out this template, you should delete or replace all of +the text except for the section headers and the header fields above. +For example, you should delete everything from this paragraph down to +the Introduction section below. + +As a proposal author, you should fill out all of the header fields +except `Review Manager`. The review manager will set that field and +change several others as part of initiating the review. Delete any +header fields marked *if applicable* that are not applicable to your +proposal. + +When sharing a link to the proposal while it is still a PR, be sure +to share a live link to the proposal, not an exact commit, so that +readers will always see the latest version when you make changes. +On GitHub, you can find this link by browsing the PR branch: from the +PR page, click the "username wants to merge ... from username:my-branch-name" +link and find the proposal file in that branch. + +`Status` should reflect the current implementation status while the +proposal is still a PR. The proposal cannot be reviewed until an +implementation is available, but early readers should see the correct +status. + +`Vision` should link to the [vision document](https://forums.swift.org/t/the-role-of-vision-documents-in-swift-evolution/62101) +for this proposal, if it is part of a vision. Most proposals are not +part of a vision. If a vision has been written but not yet accepted, +link to the discussion thread for the vision. + +`Roadmap` should link to the discussion thread for the roadmap for +this proposal, if applicable. When a complex feature is broken down +into several closely-related proposals to make evolution review easier +and more focused, it's helpful to make a forum post explaining what's +going on and detailing how the proposals are expected to be submitted +to review. That post is called a "roadmap". Most proposals don't need +roadmaps, but if this proposal was part of one, this field should link +to it. + +`Bug` should be used when this proposal is fixing a bug with significant +discussion in the bug report. It is not necessary to link bugs that do +not contain significant discussion or that merely duplicate discussion +linked somewhere else. Do not link bugs from private bug trackers. + +`Implementation` should link to the PR(s) implementing the feature. +If the proposal has not been implemented yet, or if it simply codifies +existing behavior, just say that. If the implementation has already +been committed to the main branch (as an experimental feature), say +that and specify the experimental feature flag. If the implementation +is spread across multiple PRs, just link to the most important ones. + +`Upcoming Feature Flag` should be the feature name used to identify this +feature under [SE-0362](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md#proposals-define-their-own-feature-identifier). +Not all proposals need an upcoming feature flag. You should think about +whether one would be useful for your proposal as part of filling this +field out. + +`Previous Proposal` should be used when there is a specific line of +succession between this proposal and another proposal. For example, +this proposal might have been removed from a previous proposal so +that it can be reviewed separately, or this proposal might supersede +a previous proposal in some way that was felt to exceed the scope of +a "revision". Include text briefly explaining the relationship, +such as "Supersedes SE-1234" or "Extracted from SE-01234". If possible, +link to a post explaining the relationship, such as a review decision +that asked for part of the proposal to be split off. Otherwise, you +can just link to the previous proposal. + +`Previous Revision` should be added after a major substantive revision +of a proposal that has undergone review. It links to the previously +reviewed revision. It is not necessary to add or update this field +after minor editorial changes. + +`Review` is a history of all discussion threads about this proposal, +in chronological order. Use these standardized link names: `pitch` +`review` `revision` `acceptance` `rejection`. If there are multiple +such threads, spell the ordinal out: `first pitch` `second review` etc. + +## Introduction + +A short description of what the feature is. Try to keep it to a +single-paragraph "elevator pitch" so the reader understands what +problem this proposal is addressing. + +## Motivation + +Describe the problems that this proposal seeks to address. If the +problem is that some common pattern is currently hard to express, show +how one can currently get a similar effect and describe its +drawbacks. If it's completely new functionality that cannot be +emulated, motivate why this new functionality would help Swift +developers create better Swift code. + +## Proposed solution + +Describe your solution to the problem. Provide examples and describe +how they work. Show how your solution is better than current +workarounds: is it cleaner, safer, or more efficient? + +This section doesn't have to be comprehensive. Focus on the most +important parts of the proposal and make arguments about why the +proposal is better than the status quo. + +## Detailed design + +Describe the design of the solution in detail. If it involves new +syntax in the language, show the additions and changes to the Swift +grammar. If it's a new API, show the full API and its documentation +comments detailing what it does. The detail in this section should be +sufficient for someone who is *not* one of the authors to be able to +reasonably implement the feature. + +## Source compatibility + +Describe the impact of this proposal on source compatibility. As a +general rule, all else being equal, Swift code that worked in previous +releases of the tools should work in new releases. That means both that +it should continue to build and that it should continue to behave +dynamically the same as it did before. Changes that cannot satisfy +this must be opt-in, generally by requiring a new language mode. + +This is not an absolute guarantee, and the Language Workgroup will +consider intentional compatibility breaks if their negative impact +can be shown to be small and the current behavior is causing +substantial problems in practice. + +For proposals that affect parsing, consider whether existing valid +code might parse differently under the proposal. Does the proposal +reserve new keywords that can no longer be used as identifiers? + +For proposals that affect type checking, consider whether existing valid +code might type-check differently under the proposal. Does it add new +conversions that might make more overload candidates viable? Does it +change how names are looked up in existing code? Does it make +type-checking more expensive in ways that might run into implementation +limits more often? + +For proposals that affect the standard library, consider the impact on +existing clients. If clients provide a similar API, will type-checking +find the right one? If the feature overloads an existing API, is it +problematic that existing users of that API might start resolving to +the new API? + +## ABI compatibility + +Describe the impact on ABI compatibility. As a general rule, the ABI +of existing code must not change between tools releases or language +modes. This rule does not apply as often as source compatibility, but +it is much stricter, and the Language Workgroup generally cannot allow +exceptions. + +The ABI encompasses all aspects of how code is generated for the +language, how that code interacts with other code that has been +compiled separately, and how that code interacts with the Swift +runtime library. Most ABI changes center around interactions with +specific declarations. Proposals that do not affect how code is +generated to interact with an external declaration usually do not +have ABI impact. + +For proposals that affect general code generation rules, consider +the impact on code that's already been compiled. Does the proposal +affect declarations that haven't explicitly adopted it, and if so, +does it change ABI details such as symbol names or conventions +around their use? Will existing code change its dynamic behavior +when running against a new version of the language runtime or +standard library? Conversely, will code compiled in the new way +continue to run on old versions of the language runtime or standard +library? + +For proposals that affect the standard library, consider the impact +on any existing declarations. As above, does the proposal change symbol +names, conventions, or dynamic behavior? Will newly-compiled code work +on old library versions, and will new library versions work with +previously-compiled code? + +This section will often end up very short. A proposal that just +adds a new standard library feature, for example, will usually +say either "This proposal is purely an extension of the ABI of the +standard library and does not change any existing features" or +"This proposal is purely an extension of the standard library which +can be implemented without any ABI support" (whichever applies). +Nonetheless, it is important to demonstrate that you've considered +the ABI implications. + +If the design of the feature was significantly constrained by +the need to maintain ABI compatibility, this section is a reasonable +place to discuss that. + +## Implications on adoption + +The compatibility sections above are focused on the direct impact +of the proposal on existing code. In this section, describe issues +that intentional adopters of the proposal should be aware of. + +For proposals that add features to the language or standard library, +consider whether the features require ABI support. Will adopters need +a new version of the library or language runtime? Be conservative: if +you're hoping to support back-deployment, but you can't guarantee it +at the time of review, just say that the feature requires a new +version. + +Consider also the impact on library adopters of those features. Can +adopting this feature in a library break source or ABI compatibility +for users of the library? If a library adopts the feature, can it +be *un*-adopted later without breaking source or ABI compatibility? +Will package authors be able to selectively adopt this feature depending +on the tools version available, or will it require bumping the minimum +tools version required by the package? + +If there are no concerns to raise in this section, leave it in with +text like "This feature can be freely adopted and un-adopted in source +code with no deployment constraints and without affecting source or ABI +compatibility." + +## Future directions + +Describe any interesting proposals that could build on this proposal +in the future. This is especially important when these future +directions inform the design of the proposal, for example by making +sure an attribute encodes enough information to be used for other +purposes. + +The rest of the proposal should generally not talk about future +directions except by referring to this section. It is important +not to confuse reviewers about what is covered by this specific +proposal. If there's a larger vision that needs to be explained +in order to understand this proposal, consider starting a discussion +thread on the forums to capture your broader thoughts. + +Avoid making affirmative statements in this section, such as "we +will" or even "we should". Describe the proposals neutrally as +possibilities to be considered in the future. + +Consider whether any of these future directions should really just +be part of the current proposal. It's important to make focused, +self-contained proposals that can be incrementally implemented and +reviewed, but it's also good when proposals feel "complete" rather +than leaving significant gaps in their design. For example, when +[SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md) +introduced the `@inlinable` attribute, it also included the +`@usableFromInline` attribute so that declarations used in inlinable +functions didn't have to be `public`. This was a relatively small +addition to the proposal which avoided creating a serious usability +problem for many adopters of `@inlinable`. + +## Alternatives considered + +Describe alternative approaches to addressing the same problem. +This is an important part of most proposal documents. Reviewers +are often familiar with other approaches prior to review and may +have reasons to prefer them. This section is your first opportunity +to try to convince them that your approach is the right one, and +even if you don't fully succeed, you can help set the terms of the +conversation and make the review a much more productive exchange +of ideas. + +You should be fair about other proposals, but you do not have to +be neutral; after all, you are specifically proposing something +else. Describe any advantages these alternatives might have, but +also be sure to explain the disadvantages that led you to prefer +the approach in this proposal. + +You should update this section during the pitch phase to discuss +any particularly interesting alternatives raised by the community. +You do not need to list every idea raised during the pitch, just +the ones you think raise points that are worth discussing. Of course, +if you decide the alternative is more compelling than what's in +the current proposal, you should change the main proposal; be sure +to then discuss your previous proposal in this section and explain +why the new idea is better. + +## Acknowledgments + +If significant changes or improvements suggested by members of the +community were incorporated into the proposal as it developed, take a +moment here to thank them for their contributions. Swift evolution is a +collaborative process, and everyone's input should receive recognition! + +Generally, you should not acknowledge anyone who is listed as a +co-author or as the review manager. diff --git a/proposal-templates/0000-swift-testing-template.md b/proposal-templates/0000-swift-testing-template.md new file mode 100644 index 0000000000..642ac7bd53 --- /dev/null +++ b/proposal-templates/0000-swift-testing-template.md @@ -0,0 +1,188 @@ +# Swift Testing Feature name + +* Proposal: [ST-NNNN](NNNN-filename.md) +* Authors: [Author 1](https://github.com/author1), [Author 2](https://github.com/author2) +* Review Manager: TBD +* Status: **Awaiting implementation** or **Awaiting review** +* Bug: _if applicable_ [swiftlang/swift-testing#NNNNN](https://github.com/swiftlang/swift-testing/issues/NNNNN) +* Implementation: [swiftlang/swift-testing#NNNNN](https://github.com/swiftlang/swift-testing/pull/NNNNN) +* Previous Proposal: _if applicable_ [ST-XXXX](XXXX-filename.md) +* Previous Revision: _if applicable_ [1](https://github.com/swiftlang/swift-evolution/blob/...commit-ID.../proposals/testing/NNNN-filename.md) +* Review: ([pitch](https://forums.swift.org/...)) + +When filling out this template, you should delete or replace all of the text +except for the section headers and the header fields above. For example, you +should delete everything from this paragraph down to the Introduction section +below. + +As a proposal author, you should fill out all of the header fields except +`Review Manager`. The review manager will set that field and change several +others as part of initiating the review. Delete any header fields marked _if +applicable_ that are not applicable to your proposal. + +When sharing a link to the proposal while it is still a PR, be sure to share a +live link to the proposal, not an exact commit, so that readers will always see +the latest version when you make changes. On GitHub, you can find this link by +browsing the PR branch: from the PR page, click the "username wants to merge ... +from username:my-branch-name" link and find the proposal file in that branch. + +`Status` should reflect the current implementation status while the proposal is +still a PR. The proposal cannot be reviewed until an implementation is available, +but early readers should see the correct status. + +`Bug` should be used when this proposal is fixing a bug with significant +discussion in the bug report. It is not necessary to link bugs that do not +contain significant discussion or that merely duplicate discussion linked +somewhere else. Do not link bugs from private bug trackers. + +`Implementation` should link to the PR(s) implementing the feature. If the +proposal has not been implemented yet, or if it simply codifies existing +behavior, just say that. If the implementation has already been committed to the +main branch (as an experimental feature or SPI), mention that. If the +implementation is spread across multiple PRs, just link to the most important +ones. + +`Previous Proposal` should be used when there is a specific line of succession +between this proposal and another proposal. For example, this proposal might +have been removed from a previous proposal so that it can be reviewed separately, +or this proposal might supersede a previous proposal in some way that was felt +to exceed the scope of a "revision". Include text briefly explaining the +relationship, such as "Supersedes ST-1234" or "Extracted from ST-01234". If +possible, link to a post explaining the relationship, such as a review decision +that asked for part of the proposal to be split off. Otherwise, you can just +link to the previous proposal. + +`Previous Revision` should be added after a major substantive revision of a +proposal that has undergone review. It links to the previously reviewed revision. +It is not necessary to add or update this field after minor editorial changes. + +`Review` is a history of all discussion threads about this proposal, in +chronological order. Use these standardized link names: `pitch` `review` +`revision` `acceptance` `rejection`. If there are multiple such threads, spell +the ordinal out: `first pitch` `second review` etc. + +## Introduction + +A short description of what the feature is. Try to keep it to a single-paragraph +"elevator pitch" so the reader understands what problem this proposal is +addressing. + +## Motivation + +Describe the problems that this proposal seeks to address. If the problem is +that some common pattern is currently hard to express, show how one can +currently get a similar effect and describe its drawbacks. If it's completely +new functionality that cannot be emulated, motivate why this new functionality +would help Swift developers test their code more effectively. + +## Proposed solution + +Describe your solution to the problem. Provide examples and describe how they +work. Show how your solution is better than current workarounds: is it cleaner, +safer, or more efficient? + +This section doesn't have to be comprehensive. Focus on the most important parts +of the proposal and make arguments about why the proposal is better than the +status quo. + +## Detailed design + +Describe the design of the solution in detail. If it includes new API, show the +full API and its documentation comments detailing what it does. If it involves +new macro logic, describe the behavior changes and include a succinct example of +the additions or modifications to the macro expansion code. The detail in this +section should be sufficient for someone who is *not* one of the authors to be +able to reasonably implement the feature. + +## Source compatibility + +Describe the impact of this proposal on source compatibility. As a general rule, +all else being equal, test code that worked in previous releases of the testing +library should work in new releases. That means both that it should continue to +build and that it should continue to behave dynamically the same as it did +before. + +This is not an absolute guarantee, and the testing library administrators will +consider intentional compatibility breaks if their negative impact can be shown +to be small and the current behavior is causing substantial problems in practice. + +For proposals that affect testing library API, consider the impact on existing +clients. If clients provide a similar API, will type-checking find the right one? +If the feature overloads an existing API, is it problematic that existing users +of that API might start resolving to the new API? + +## Integration with supporting tools + +In this section, describe how this proposal affects tools which integrate with +the testing library. Some features depend on supporting tools gaining awareness +of the new feature for users to realize new benefits. Other features do not +strictly require integration but bring improvement opportunities which are worth +considering. Use this section to discuss any impact on tools. + +This section does need not to include details of how this proposal may be +integrated with _specific_ tools, but it should consider the general ways that +tools might support this feature and note any accompanying SPI intended for +tools which are included in the implementation. Note that tools may evolve +independently and have differing release schedules than the testing library, so +special care should be taken to ensure compatibility across versions according +to the needs of each tool. + +## Future directions + +Describe any interesting proposals that could build on this proposal in the +future. This is especially important when these future directions inform the +design of the proposal, for example by making sure an interface meant for tools +integration can be extended to include additional information. + +The rest of the proposal should generally not talk about future directions +except by referring to this section. It is important not to confuse reviewers +about what is covered by this specific proposal. If there's a larger vision that +needs to be explained in order to understand this proposal, consider starting a +discussion thread on the forums to capture your broader thoughts. + +Avoid making affirmative statements in this section, such as "we will" or even +"we should". Describe the proposals neutrally as possibilities to be considered +in the future. + +Consider whether any of these future directions should really just be part of +the current proposal. It's important to make focused, self-contained proposals +that can be incrementally implemented and reviewed, but it's also good when +proposals feel "complete" rather than leaving significant gaps in their design. +An an example from the Swift project, when +[SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md) +introduced the `@inlinable` attribute, it also included the `@usableFromInline` +attribute so that declarations used in inlinable functions didn't have to be +`public`. This was a relatively small addition to the proposal which avoided +creating a serious usability problem for many adopters of `@inlinable`. + +## Alternatives considered + +Describe alternative approaches to addressing the same problem. This is an +important part of most proposal documents. Reviewers are often familiar with +other approaches prior to review and may have reasons to prefer them. This +section is your first opportunity to try to convince them that your approach is +the right one, and even if you don't fully succeed, you can help set the terms +of the conversation and make the review a much more productive exchange of ideas. + +You should be fair about other proposals, but you do not have to be neutral; +after all, you are specifically proposing something else. Describe any +advantages these alternatives might have, but also be sure to explain the +disadvantages that led you to prefer the approach in this proposal. + +You should update this section during the pitch phase to discuss any +particularly interesting alternatives raised by the community. You do not need +to list every idea raised during the pitch, just the ones you think raise points +that are worth discussing. Of course, if you decide the alternative is more +compelling than what's in the current proposal, you should change the main +proposal; be sure to then discuss your previous proposal in this section and +explain why the new idea is better. + +## Acknowledgments + +If significant changes or improvements suggested by members of the community +were incorporated into the proposal as it developed, take a moment here to thank +them for their contributions. This is a collaborative process, and everyone's +input should receive recognition! + +Generally, you should not acknowledge anyone who is listed as a co-author or as +the review manager. diff --git a/proposal-templates/0000-swiftpm-template.md b/proposal-templates/0000-swiftpm-template.md new file mode 100644 index 0000000000..bd65132b9e --- /dev/null +++ b/proposal-templates/0000-swiftpm-template.md @@ -0,0 +1,62 @@ +# Package Manager Feature name + +* Proposal: [SE-NNNN](NNNN-filename.md) +* Authors: [Author 1](https://github.com/swiftdev), [Author 2](https://github.com/swiftdev) +* Review Manager: TBD +* Status: **Awaiting implementation** + +*During the review process, add the following fields as needed:* + +* Implementation: [swiftlang/swift-package-manager#NNNNN](https://github.com/swiftlang/swift-package-manager/pull/NNNNN) +* Decision Notes: [Rationale](https://forums.swift.org/), [Additional Commentary](https://forums.swift.org/) +* Bugs: [SR-NNNN](https://bugs.swift.org/browse/SR-NNNN), [SR-MMMM](https://bugs.swift.org/browse/SR-MMMM) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md) +* Previous Proposal: [SE-XXXX](XXXX-filename.md) + +## Introduction + +A short description of what the feature is. Try to keep it to a single-paragraph +"elevator pitch" so the reader understands what problem this proposal is +addressing. + +Swift-evolution thread: [Discussion thread topic for that +proposal](https://forums.swift.org/) + +## Motivation + +Describe the problems that this proposal seeks to address. If the problem is +that some functionality is currently hard to use, show how it is currently used +and describe its drawbacks. If it's completely new functionality that cannot be +emulated, motivate why this new functionality would help Swift developers create +better Swift packages. + +## Proposed solution + +Describe your solution to the problem. Provide examples and describe how they +work. Show how your solution is better than current workarounds: is it cleaner, +easier, or more efficient? + +## Detailed design + +Describe the design of the solution in detail. If it involves adding or +modifying functionality in the package manager, explain how the package manager +behaves in different scenarios and with existing features. If it's a new API in +the `Package.swift` manifest, show the full API and its documentation comments +detailing what it does. The detail in this section should be sufficient for +someone who is *not* one of the author of the proposal to be able to reasonably +implement the feature. + +## Security + +Does this change have any impact on security, safety, or privacy? + +## Impact on existing packages + +Explain if and how this proposal will affect the behavior of existing packages. +If there will be impact, is it possible to gate the changes on the tools version +of the package manifest? + +## Alternatives considered + +Describe alternative approaches to addressing the same problem, and +why you chose this approach instead. diff --git a/proposals/0001-keywords-as-argument-labels.md b/proposals/0001-keywords-as-argument-labels.md index db324b7f0b..cd0c7279af 100644 --- a/proposals/0001-keywords-as-argument-labels.md +++ b/proposals/0001-keywords-as-argument-labels.md @@ -68,7 +68,7 @@ does not change the behavior of any well-formed code. ## Alternatives considered -The primarily alternative here is to do nothing: Swift APIs will +The primary alternative here is to do nothing: Swift APIs will continue to avoid keywords for argument labels, even when they are the most natural word for the label, and imported APIs will either continue to use backticks or will need to be renamed. This alternative diff --git a/proposals/0002-remove-currying.md b/proposals/0002-remove-currying.md index cb7fc5411d..9761465a5c 100644 --- a/proposals/0002-remove-currying.md +++ b/proposals/0002-remove-currying.md @@ -2,8 +2,8 @@ * Proposal: [SE-0002](0002-remove-currying.md) * Author: [Joe Groff](https://github.com/jckarter) -* Status: **Implemented (Swift 3)** -* Commit: [apple/swift@983a674](https://github.com/apple/swift/commit/983a674e0ca35a85532d70a3eb61e71a6d024108) +* Status: **Implemented (Swift 3.0)** +* Implementation: [apple/swift@983a674](https://github.com/apple/swift/commit/983a674e0ca35a85532d70a3eb61e71a6d024108) ## Introduction diff --git a/proposals/0003-remove-var-parameters.md b/proposals/0003-remove-var-parameters.md index 42bd4d44cc..a78dc80b0c 100644 --- a/proposals/0003-remove-var-parameters.md +++ b/proposals/0003-remove-var-parameters.md @@ -1,11 +1,11 @@ # Removing `var` from Function Parameters * Proposal: [SE-0003](0003-remove-var-parameters.md) -* Author: [David Farler](https://github.com/bitjammer) +* Author: [Ashley Garland](https://github.com/bitjammer) * Review Manager: [Joe Pamer](https://github.com/jopamer) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160125/008145.html) -* Commit: [apple/swift@8a5ed40](https://github.com/apple/swift/commit/8a5ed405bf1f92ec3fc97fa46e52528d2e8d67d9) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0003-removing-var-from-function-parameters-and-pattern-matching/1230) +* Implementation: [apple/swift@8a5ed40](https://github.com/apple/swift/commit/8a5ed405bf1f92ec3fc97fa46e52528d2e8d67d9) ## Note @@ -97,16 +97,16 @@ this language change. This proposal originally included removal of `var` bindings for all refutable patterns as well as function parameters. -[Original SE-0003 Proposal](https://github.com/apple/swift-evolution/blob/8cd734260bc60d6d49dbfb48de5632e63bf200cc/proposals/0003-remove-var-parameters-patterns.md) +[Original SE-0003 Proposal](https://github.com/swiftlang/swift-evolution/blob/8cd734260bc60d6d49dbfb48de5632e63bf200cc/proposals/0003-remove-var-parameters-patterns.md) Removal of `var` from refutable patterns was reconsidered due to the burden it placed on valid mutation patterns already in use in Swift 2 code. You can view the discussion on the swift-evolution mailing list here: -[Initial Discussion of Reconsideration](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007326.html) +[Initial Discussion of Reconsideration](https://forums.swift.org/t/reconsidering-se-0003-removing-var-from-function-parameters-and-pattern-matching/1159) The rationale for a final conclusion was also sent to the swift-evolution list, which you can view here: -[Note on Revision of the Proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160125/008145.html) +[Note on Revision of the Proposal](https://forums.swift.org/t/se-0003-removing-var-from-function-parameters-and-pattern-matching/1230) diff --git a/proposals/0004-remove-pre-post-inc-decrement.md b/proposals/0004-remove-pre-post-inc-decrement.md index 5e7d57640b..c33507a664 100644 --- a/proposals/0004-remove-pre-post-inc-decrement.md +++ b/proposals/0004-remove-pre-post-inc-decrement.md @@ -2,8 +2,8 @@ * Proposal: [SE-0004](0004-remove-pre-post-inc-decrement.md) * Author: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Commit: [apple/swift@8e12008](https://github.com/apple/swift/commit/8e12008d2b34a605f8766310f53d5668f3d50955) +* Status: **Implemented (Swift 3.0)** +* Implementation: [apple/swift@8e12008](https://github.com/apple/swift/commit/8e12008d2b34a605f8766310f53d5668f3d50955) ## Introduction diff --git a/proposals/0005-objective-c-name-translation.md b/proposals/0005-objective-c-name-translation.md index da47e03465..9a494db4c3 100644 --- a/proposals/0005-objective-c-name-translation.md +++ b/proposals/0005-objective-c-name-translation.md @@ -3,8 +3,8 @@ * Proposal: [SE-0005](0005-objective-c-name-translation.md) * Authors: [Doug Gregor](https://github.com/DougGregor), [Dave Abrahams](https://github.com/dabrahams) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160229/011770.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modification-se-0005-better-translation-of-objective-c-apis-into-swift/1668) ## Reviewer notes @@ -12,11 +12,11 @@ This review is part of a group of three related reviews, running concurrently: * [SE-0023 API Design Guidelines](0023-api-guidelines.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007353.html)) + ([Review](https://forums.swift.org/t/review-se-0023-api-design-guidelines/1162)) * [SE-0006 Apply API Guidelines to the Standard Library](0006-apply-api-guidelines-to-the-standard-library.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007354.html)) + ([Review](https://forums.swift.org/t/review-se-0006-apply-api-guidelines-to-the-standard-library/1163)) * [SE-0005 Better Translation of Objective-C APIs Into Swift](0005-objective-c-name-translation.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007355.html)) + ([Review](https://forums.swift.org/t/review-se-0005-better-translation-of-objective-c-apis-into-swift/1164)) These reviews are running concurrently because they interact strongly (e.g., an API change in the standard library will correspond to a @@ -48,15 +48,15 @@ APIs that feel right in Objective-C can feel wordy when used in Swift. For example: ```swift - let content = listItemView.text.stringByTrimmingCharactersInSet( - NSCharacterSet.whitespaceAndNewlineCharacterSet()) +let content = listItemView.text.stringByTrimmingCharactersInSet( + NSCharacterSet.whitespaceAndNewlineCharacterSet()) ``` The APIs used here follow the Objective-C guidelines. A more "Swifty" version of the same code might instead look like this: ```swift - let content = listItemView.text.trimming(.whitespaceAndNewlines) +let content = listItemView.text.trimming(.whitespaceAndNewlines) ``` The latter example more closely adheres to the [Swift API Design @@ -140,43 +140,43 @@ To get a sense of what these transformations do, consider a portion of the imported `UIBezierPath` API in Swift 2: ```swift - class UIBezierPath : NSObject, NSCopying, NSCoding { - convenience init(ovalInRect: CGRect) - func moveToPoint(_: CGPoint) - func addLineToPoint(_: CGPoint) - func addCurveToPoint(_: CGPoint, controlPoint1: CGPoint, controlPoint2: CGPoint) - func addQuadCurveToPoint(_: CGPoint, controlPoint: CGPoint) - func appendPath(_: UIBezierPath) - func bezierPathByReversingPath() -> UIBezierPath - func applyTransform(_: CGAffineTransform) - var empty: Bool { get } - func containsPoint(_: CGPoint) -> Bool - func fillWithBlendMode(_: CGBlendMode, alpha: CGFloat) - func strokeWithBlendMode(_: CGBlendMode, alpha: CGFloat) - func copyWithZone(_: NSZone) -> AnyObject - func encodeWithCoder(_: NSCoder) - } +class UIBezierPath : NSObject, NSCopying, NSCoding { + convenience init(ovalInRect: CGRect) + func moveToPoint(_: CGPoint) + func addLineToPoint(_: CGPoint) + func addCurveToPoint(_: CGPoint, controlPoint1: CGPoint, controlPoint2: CGPoint) + func addQuadCurveToPoint(_: CGPoint, controlPoint: CGPoint) + func appendPath(_: UIBezierPath) + func bezierPathByReversingPath() -> UIBezierPath + func applyTransform(_: CGAffineTransform) + var empty: Bool { get } + func containsPoint(_: CGPoint) -> Bool + func fillWithBlendMode(_: CGBlendMode, alpha: CGFloat) + func strokeWithBlendMode(_: CGBlendMode, alpha: CGFloat) + func copyWithZone(_: NSZone) -> AnyObject + func encodeWithCoder(_: NSCoder) +} ``` And the same API imported under our current, experimental implementation of this proposal: ```swift - class UIBezierPath : NSObject, NSCopying, NSCoding { - convenience init(ovalIn rect: CGRect) - func move(to point: CGPoint) - func addLine(to point: CGPoint) - func addCurve(to endPoint: CGPoint, controlPoint1 controlPoint1: CGPoint, controlPoint2 controlPoint2: CGPoint) - func addQuadCurve(to endPoint: CGPoint, controlPoint controlPoint: CGPoint) - func append(_ bezierPath: UIBezierPath) - func reversing() -> UIBezierPath - func apply(_ transform: CGAffineTransform) - var isEmpty: Bool { get } - func contains(_ point: CGPoint) -> Bool - func fill(_ blendMode: CGBlendMode, alpha alpha: CGFloat) - func stroke(_ blendMode: CGBlendMode, alpha alpha: CGFloat) - func copy(with zone: NSZone = nil) -> AnyObject - func encode(with aCoder: NSCoder) - } +class UIBezierPath : NSObject, NSCopying, NSCoding { + convenience init(ovalIn rect: CGRect) + func move(to point: CGPoint) + func addLine(to point: CGPoint) + func addCurve(to endPoint: CGPoint, controlPoint1 controlPoint1: CGPoint, controlPoint2 controlPoint2: CGPoint) + func addQuadCurve(to endPoint: CGPoint, controlPoint controlPoint: CGPoint) + func append(_ bezierPath: UIBezierPath) + func reversing() -> UIBezierPath + func apply(_ transform: CGAffineTransform) + var isEmpty: Bool { get } + func contains(_ point: CGPoint) -> Bool + func fill(_ blendMode: CGBlendMode, alpha alpha: CGFloat) + func stroke(_ blendMode: CGBlendMode, alpha alpha: CGFloat) + func copy(with zone: NSZone = nil) -> AnyObject + func encode(with aCoder: NSCoder) +} ``` In the latter case, a number of words that restated type information @@ -279,14 +279,14 @@ A couple of basic rules govern all matches: acronyms or prefixes:
-func documentForURL(_: NSURL) -> NSDocument?
-
+ func documentForURL(_: NSURL) -> NSDocument? + - while preventing partial-word mismatches: + while preventing partial-word mismatches: -
-   var thumbnailPreview : UIView  // not matched
-   
+
+  var thumbnailPreview : UIView  // not matched
+  
* **Matched text extends to the end of the type name**. Because we accept a match for *any suffix* of the type name, this code: @@ -313,14 +313,14 @@ Matches are a sequence of one or more of the following: matches `String` in `NSString`:
-func appendString(_: NSString)
-
+ func appendString(_: NSString) + * `Index` in the selector piece matches `Int` in the type name:
-func characterAtIndex(_: Int) -> unichar
-
+ func characterAtIndex(_: Int) -> unichar + * **Collection matches** @@ -328,33 +328,33 @@ func characterAtIndex(_: Int) -> unichar the type name:
-func removeObjectsAtIndexes(_: NSIndexSet)
-
+ func removeObjectsAtIndexes(_: NSIndexSet) + * A plural noun in the selector piece matches a collection type name if the noun's singular form matches the name of the collection's element type: -
-func arrangeObjects(_: [AnyObject]) -> [AnyObject]
-
+
+    func arrangeObjects(_: [AnyObject]) -> [AnyObject]
+    
* **Special suffix matches** * The empty string in the selector piece matches `Type` or `_t` in the type name:
-func writableTypesForSaveOperation(_: NSSaveOperationType) -> [String]
-func objectForKey(_: KeyType) -> AnyObject
-func startWithQueue(_: dispatch_queue_t, completionHandler: MKMapSnapshotCompletionhandler)
-
+ func writableTypesForSaveOperation(_: NSSaveOperationType) -> [String] + func objectForKey(_: KeyType) -> AnyObject + func startWithQueue(_: dispatch_queue_t, completionHandler: MKMapSnapshotCompletionhandler) + * The empty string in the selector piece matches *one or more digits followed by "D"* in the type name:
-func pointForCoordinate(_: CLLocationCoordinate2D) -> NSPoint
-
+ func pointForCoordinate(_: CLLocationCoordinate2D) -> NSPoint + In the examples above, the italic text is effectively skipped, so the bold part of the selector piece can be matched and pruned. @@ -378,27 +378,27 @@ skipped. user to write backticks. For example,
-extension NSParagraphStyle {
-  class func defaultParagraphStyle() -> NSParagraphStyle
-}
-let defaultStyle = NSParagraphStyle.defaultParagraphStyle()  // OK
-
+ extension NSParagraphStyle { +   class func defaultParagraphStyle() -> NSParagraphStyle + } + let defaultStyle = NSParagraphStyle.defaultParagraphStyle() // OK + would become:
-extension NSParagraphStyle {
-  class func `default`() -> NSParagraphStyle
-}
-let defaultStyle = NSParagraphStyle.`default`()              // Awkward
-
+ extension NSParagraphStyle { +   class func `default`() -> NSParagraphStyle + } + let defaultStyle = NSParagraphStyle.`default`() // Awkward + By contrast, later selector pieces become argument labels, which are allowed to match Swift keywords without requiring backticks:
-receiver.handle(someMessage, for: somebody)  // OK
-
+ receiver.handle(someMessage, for: somebody) // OK + * **Never transform a name into "get", "set", "with", "for", or "using"**, just to avoid creating absurdly vacuous names. @@ -413,19 +413,19 @@ receiver.handle(someMessage, for: somebody) // OK the parameter that follows. For example,
-func setTextColor(_: UIColor)
-...
-button.setTextColor(.red())  // clear
-
+ func setTextColor(_: UIColor) + ... + button.setTextColor(.red()) // clear + If we were to drop `Color`, leaving just `Text`, call sites would become confusing:
-func setText(_: UIColor)
-...
-button.setText(.red())      // appears to be setting the text!
-
+ func setText(_: UIColor) + ... + button.setText(.red()) // appears to be setting the text! + Note: We don't maintain a list of nouns, but if we did, this rule could be more simply phrased as "don't prune a suffix @@ -438,9 +438,9 @@ button.setText(.red()) // appears to be setting the text! of the class.
-var gestureRecognizers: [UIGestureRecognizer]
-func addGestureRecognizer(_: UIGestureRecognizer)
-
+ var gestureRecognizers: [UIGestureRecognizer] + func addGestureRecognizer(_: UIGestureRecognizer) + If we were to drop `GestureRecognizer`, leaving just `add`, we end up with a method that conceptually modifies the @@ -448,9 +448,9 @@ func addGestureRecognizer(_: UIGestureRecognizer) do so:
-var gestureRecognizers: [UIGestureRecognizer]
-func add(_: UIGestureRecognizer) // should indicate that we're adding to the property
-
+ var gestureRecognizers: [UIGestureRecognizer] + func add(_: UIGestureRecognizer) // should indicate that we're adding to the property + #### Pruning Steps @@ -471,20 +471,20 @@ shown: receiver. For example:
-extension NSColor {
-  func colorWithAlphaComponent(_: CGFloat) -> NSColor
-}
-let translucentForeground = foregroundColor.colorWithAlphaComponent(0.5)
-
+ extension NSColor { +   func colorWithAlphaComponent(_: CGFloat) -> NSColor + } + let translucentForeground = foregroundColor.colorWithAlphaComponent(0.5) + becomes:
-extension NSColor {
-  func withAlphaComponent(_: CGFloat) -> NSColor
-}
-let translucentForeground = foregroundColor.withAlphaComponent(0.5)
-
+ extension NSColor { +   func withAlphaComponent(_: CGFloat) -> NSColor + } + let translucentForeground = foregroundColor.withAlphaComponent(0.5) + 2. **Prune an additional hanging "By"**. Specifically, if @@ -497,20 +497,20 @@ let translucentForeground = foregroundColor.withAlphaComponent(0.5 b.frobnicating(c)`. For example:
-extension NSString {
-  func stringByApplyingTransform(_: NSString, reverse: Bool) -> NSString?
-}
-let sanitizedInput = rawInput.stringByApplyingTransform(NSStringTransformToXMLHex, reverse: false)
-
+ extension NSString { +   func stringByApplyingTransform(_: NSString, reverse: Bool) -> NSString? + } + let sanitizedInput = rawInput.stringByApplyingTransform(NSStringTransformToXMLHex, reverse: false) + becomes:
-extension NSString {
-  func applyingTransform(_: NSString, reverse: Bool) -> NString?
-}
-let sanitizedInput = rawInput.applyingTransform(NSStringTransformToXMLHex, reverse: false)
-
+ extension NSString { +   func applyingTransform(_: NSString, reverse: Bool) -> NString? + } + let sanitizedInput = rawInput.applyingTransform(NSStringTransformToXMLHex, reverse: false) + 3. **Prune a match for any type name in the signature from the tail of the preceding selector piece**. Specifically, @@ -524,54 +524,54 @@ let sanitizedInput = rawInput.applyingTransform(NSStringTransformToXMLHex For example,
-extension NSDocumentController {
-  func documentForURL(_ url: NSURL) -> NSDocument? // parameter introducer
-}
-extension NSManagedObjectContext {
-  var parentContext: NSManagedObjectContext?       // property
-}
-extension UIColor {
-  class func darkGrayColor() -> UIColor            // zero-argument method
-}
-...
-myDocument = self.documentForURL(locationOfFile)
-if self.managedObjectContext.parentContext != changedContext { return }
-foregroundColor = .darkGrayColor()
-
+ extension NSDocumentController { +   func documentForURL(_ url: NSURL) -> NSDocument? // parameter introducer + } + extension NSManagedObjectContext { +   var parentContext: NSManagedObjectContext? // property + } + extension UIColor { +   class func darkGrayColor() -> UIColor // zero-argument method + } + ... + myDocument = self.documentForURL(locationOfFile) + if self.managedObjectContext.parentContext != changedContext { return } + foregroundColor = .darkGrayColor() + becomes:
-extension NSDocumentController {
-  func documentFor(_ url: NSURL) -> NSDocument?
-}
-extension NSManagedObjectContext {
-  var parent : NSManagedObjectContext?
-}
-extension UIColor {
-  class func darkGray() -> UIColor
-}
-...
-myDocument = self.documentFor(locationOfFile)
-if self.managedObjectContext.parent != changedContext { return }
-foregroundColor = .darkGray()
-
+ extension NSDocumentController { +   func documentFor(_ url: NSURL) -> NSDocument? + } + extension NSManagedObjectContext { +   var parent : NSManagedObjectContext? + } + extension UIColor { +   class func darkGray() -> UIColor + } + ... + myDocument = self.documentFor(locationOfFile) + if self.managedObjectContext.parent != changedContext { return } + foregroundColor = .darkGray() + 3. **Prune a match for the enclosing type from the base name of a method so long as the match starts after a verb**. For example,
-extension UIViewController {
-  func dismissViewControllerAnimated(flag: Bool, completion: (() -> Void)? = nil)
-}
-
+ extension UIViewController { +   func dismissViewControllerAnimated(flag: Bool, completion: (() -> Void)? = nil) + } + becomes:
-extension UIViewController {
-  func dismissAnimated(flag: Bool, completion: (() -> Void)? = nil)
-}
-
+ extension UIViewController { +   func dismissAnimated(flag: Bool, completion: (() -> Void)? = nil) + } + ##### Why Does Order Matter? @@ -581,10 +581,12 @@ prevent both the head and tail from being pruned, prioritizing head-pruning steps can keep method families together. For example, in NSFontDescriptor: - func fontDescriptorWithSymbolicTraits(_: NSFontSymbolicTraits) -> NSFontDescriptor - func fontDescriptorWithSize(_: CGFloat) -> UIFontDescriptor - func fontDescriptorWithMatrix(_: CGAffineTransform) -> UIFontDescriptor - ... +```swift +func fontDescriptorWithSymbolicTraits(_: NSFontSymbolicTraits) -> NSFontDescriptor +func fontDescriptorWithSize(_: CGFloat) -> UIFontDescriptor +func fontDescriptorWithMatrix(_: CGAffineTransform) -> UIFontDescriptor +... +``` becomes: @@ -633,9 +635,11 @@ UIView.animateWithDuration( to become: - rootViewController.present(alert, animated: true) - UIView.animateWithDuration( - 0.2, delay: 0.0, animations: { self.logo.alpha = 0.0 }) { _ in self.logo.hidden = true } +```swift +rootViewController.present(alert, animated: true) +UIView.animateWithDuration( + 0.2, delay: 0.0, animations: { self.logo.alpha = 0.0 }) { _ in self.logo.hidden = true } +``` #### Add First Argument Labels @@ -684,8 +688,10 @@ array.enumerateObjects() { // OK **For Boolean properties, use the name of the getter as the property name in Swift*. For example: - @interface NSBezierPath : NSObject - @property (readonly,getter=isEmpty) BOOL empty; +```swift +@interface NSBezierPath : NSObject +@property (readonly,getter=isEmpty) BOOL empty; +``` will become @@ -712,13 +718,13 @@ as adopting `Comparable`. A survey of Foundation classes reveals not just NSDate but a few other classes that would be affected by this change. -
+```swift
 func compare(other: NSDate) -> NSComparisonResult
 func compare(decimalNumber: NSNumber) -> NSComparisonResult
 func compare(otherObject: NSIndexPath) -> NSComparisonResult
 func compare(string: String) -> NSComparisonResult
 func compare(otherNumber: NSNumber) -> NSComparisonResult
-
+``` ## Impact on existing code diff --git a/proposals/0006-apply-api-guidelines-to-the-standard-library.md b/proposals/0006-apply-api-guidelines-to-the-standard-library.md index d36883ac4e..76349a2287 100644 --- a/proposals/0006-apply-api-guidelines-to-the-standard-library.md +++ b/proposals/0006-apply-api-guidelines-to-the-standard-library.md @@ -3,8 +3,8 @@ * Proposal: [SE-0006](0006-apply-api-guidelines-to-the-standard-library.md) * Authors: [Dave Abrahams](https://github.com/dabrahams), [Dmitri Gribenko](https://github.com/gribozavr), [Maxim Moiseev](https://github.com/moiseev) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000054.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0006-apply-api-guidelines-to-the-standard-library/1667) ## Reviewer notes @@ -12,11 +12,11 @@ This review is part of a group of three related reviews, running concurrently: * [SE-0023 API Design Guidelines](0023-api-guidelines.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007353.html)) + ([Review](https://forums.swift.org/t/review-se-0023-api-design-guidelines/1162)) * [SE-0006 Apply API Guidelines to the Standard Library](0006-apply-api-guidelines-to-the-standard-library.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007354.html)) + ([Review](https://forums.swift.org/t/review-se-0006-apply-api-guidelines-to-the-standard-library/1163)) * [SE-0005 Better Translation of Objective-C APIs Into Swift](0005-objective-c-name-translation.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007355.html)) + ([Review](https://forums.swift.org/t/review-se-0005-better-translation-of-objective-c-apis-into-swift/1164)) These reviews are running concurrently because they interact strongly (e.g., an API change in the standard library will correspond to a @@ -1144,7 +1144,7 @@ public struct OpaquePointer : ... { - allowedCharacters: NSCharacterSet - ) -> String? + public func addingPercentEncoding( -+ withAllowedCharaters allowedCharacters: NSCharacterSet ++ withAllowedCharacters allowedCharacters: NSCharacterSet + ) -> String? - public func stringByAddingPercentEscapesUsingEncoding( diff --git a/proposals/0007-remove-c-style-for-loops.md b/proposals/0007-remove-c-style-for-loops.md index def47e117c..1bef3ab55a 100644 --- a/proposals/0007-remove-c-style-for-loops.md +++ b/proposals/0007-remove-c-style-for-loops.md @@ -3,8 +3,8 @@ * Proposal: [SE-0007](0007-remove-c-style-for-loops.md) * Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2015-December/000001.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0007-remove-c-style-for-loops-with-conditions-and-incrementers/512) * Bugs: [SR-226](https://bugs.swift.org/browse/SR-226), [SR-227](https://bugs.swift.org/browse/SR-227) ## Introduction @@ -19,7 +19,7 @@ language. The value of this construct is limited and I believe its removal should be seriously considered. -This proposal was discussed on the Swift Evolution list in the [C-style For Loops](https://lists.swift.org/pipermail/swift-evolution/2015-December/000053.html) thread and reviewed in the [\[Review\] Remove C-style for-loops with conditions and incrementers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/000913.html) thread. +This proposal was discussed on the Swift Evolution list in the [C-style For Loops](https://forums.swift.org/t/c-style-for-loops/31) thread and reviewed in the [\[Review\] Remove C-style for-loops with conditions and incrementers](https://forums.swift.org/t/review-remove-c-style-for-loops-with-conditions-and-incrementers/255) thread. ## Advantages of For Loops @@ -101,6 +101,6 @@ for(var i=0 ; i < array.count ;i++){ * "For what it's worth we don't have a single C style for loop in the Lyft codebase." -- Keith Smiley, keithbsmiley@gmail.com * "Just checked; ditto Khan Academy." -- Andy Matsuchak, andy@andymatuschak.org * "We’ve developed a number of Swift apps for various clients over the past year and have not needed C style for loops either." -- Eric Chamberlain, eric.chamberlain@arctouch.com -* "Every time I've tried to use a C-style for loop, I've ended up switching to a while loop because my iteration variable ended up having the wrong type (e.g. having an optional type when the value must be non-optional for the body to execute). The Postmates codebase contains no instances of C-style for loops in Swift." -- Kevin Ballard, kevin@sb.org +* "Every time I've tried to use a C-style for loop, I've ended up switching to a while loop because my iteration variable ended up having the wrong type (e.g. having an optional type when the value must be non-optional for the body to execute). The Postmates codebase contains no instances of C-style for loops in Swift." -- Lily Ballard, lily@sb.org * "I found a couple of cases of them in my codebase, but they were trivially transformed into “proper” Swift-style for loops that look better anyway. If it were a vote, I’d vote for eliminating C-style." -- Sean Heber, sean@fifthace.com diff --git a/proposals/0008-lazy-flatmap-for-optionals.md b/proposals/0008-lazy-flatmap-for-optionals.md index 8ec4f70956..2abda3ceee 100644 --- a/proposals/0008-lazy-flatmap-for-optionals.md +++ b/proposals/0008-lazy-flatmap-for-optionals.md @@ -3,8 +3,8 @@ * Proposal: [SE-0008](0008-lazy-flatmap-for-optionals.md) * Author: [Oisin Kidney](https://github.com/oisdk) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/004418.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0008-add-a-lazy-flatmap-for-sequences-of-optionals/748) * Bug: [SR-361](https://bugs.swift.org/browse/SR-361) ## Introduction ## @@ -39,7 +39,7 @@ However, there is only a lazy implementation for the first version: // [1, 2, 3, 4, 5] ``` -Swift Evolution Discussions: [Lazy flatMap for Optionals](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151130/000534.html), [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002592.html) +Swift Evolution Discussions: [Lazy flatMap for Optionals](https://forums.swift.org/t/lazy-flatmap-for-optionals/127/3), [Review](https://forums.swift.org/t/review-add-a-lazy-flatmap-for-sequences-of-optionals/548) ## Motivation ## diff --git a/proposals/0009-require-self-for-accessing-instance-members.md b/proposals/0009-require-self-for-accessing-instance-members.md index 737a62e7ff..626f78665e 100644 --- a/proposals/0009-require-self-for-accessing-instance-members.md +++ b/proposals/0009-require-self-for-accessing-instance-members.md @@ -4,13 +4,13 @@ * Author: [David Hart](https://github.com/hartbit) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005478.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0009-require-self-for-accessing-instance-members/930) ## Introduction The current version of Swift (2.1) requires using `self` when accessing instance members in closures. The proposal suggests extending this to all member accesses (as is intrinsically the case in Objective-C). It has the benefit of documenting instance properties vs local variables and instance functions vs local functions or closures. -[Swift Evolution Discussion Thread](https://lists.swift.org/pipermail/swift-evolution/2015-December/000209.html) +[Swift Evolution Discussion Thread](https://forums.swift.org/t/proposal-re-instate-mandatory-self-for-accessing-instance-properties-and-functions/125) ## Motivation @@ -21,7 +21,7 @@ This proposal makes it obvious which are instance properties vs local variables, * Less confusing from a learning point of view. * Lets the compiler warn users (and avoids bugs) where the authors mean to use a local variable but instead are unknowingly using an instance property (and the other way round). -One example of a bug avoidable by the proposal ([provided by Rudolf Adamkovic](https://lists.swift.org/pipermail/swift-evolution/2015-December/000243.html)): +One example of a bug avoidable by the proposal ([provided by Rudolf Adamkovic](https://forums.swift.org/t/proposal-re-instate-mandatory-self-for-accessing-instance-properties-and-functions/125/4)): ```swift class MyViewController : UIViewController { diff --git a/proposals/0010-add-staticstring-unicodescalarview.md b/proposals/0010-add-staticstring-unicodescalarview.md index dc4d4c1a5e..ac3a792444 100644 --- a/proposals/0010-add-staticstring-unicodescalarview.md +++ b/proposals/0010-add-staticstring-unicodescalarview.md @@ -1,17 +1,17 @@ # Add StaticString.UnicodeScalarView * Proposal: [SE-0010](0010-add-staticstring-unicodescalarview.md) -* Author: [Kevin Ballard](https://github.com/kballard) +* Author: [Lily Ballard](https://github.com/lilyball) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-February/000045.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0010-add-staticstring-unicodescalarview/1530) ## Introduction There is no way to create a substring of a `StaticString` that is still typed as `StaticString`. There should be. -[Swift Evolution Discussion Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151130/000535.html), [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005609.html) +[Swift Evolution Discussion Thread](https://forums.swift.org/t/proposal-add-staticstring-unicodescalarview/199), [Review](https://forums.swift.org/t/review-se-0010-add-staticstring-unicodescalarview/942) ## Motivation diff --git a/proposals/0011-replace-typealias-associated.md b/proposals/0011-replace-typealias-associated.md index dc17e45ba9..89d7a3d25e 100644 --- a/proposals/0011-replace-typealias-associated.md +++ b/proposals/0011-replace-typealias-associated.md @@ -4,7 +4,7 @@ * Author: [Loïc Lecrenier](https://github.com/loiclec) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Implemented (Swift 2.2)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-January/000014.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0011-replace-typealias-keyword-with-associatedtype-for-associated-type-declarations/990) * Bug: [SR-511](https://bugs.swift.org/browse/SR-511) @@ -21,7 +21,7 @@ confusion surrounding the use of associated types. The proposed new keyword is `associatedtype`. -[Review Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151228/005123.html) +[Review Thread](https://forums.swift.org/t/review-replace-typealias-keyword-with-associatedtype-for-associated-type-declarations/880) ## Motivation @@ -105,5 +105,5 @@ could be easily automated without any risk of breaking existing code. ## Mailing List -- [Original](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151130/000470.html) -- [Alternative Keywords](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003551.html) +- [Original](https://forums.swift.org/t/introduce-associated-type-keyword/201) +- [Alternative Keywords](https://forums.swift.org/t/se-0011-re-considering-the-replacement-keyword-for-typealias/669) diff --git a/proposals/0012-add-noescape-to-public-library-api.md b/proposals/0012-add-noescape-to-public-library-api.md index 9678927780..9e9c0d95e6 100644 --- a/proposals/0012-add-noescape-to-public-library-api.md +++ b/proposals/0012-add-noescape-to-public-library-api.md @@ -4,7 +4,7 @@ * Author: [Jacob Bandes-Storch](https://github.com/jtbandes) * Review Manager: [Philippe Hausler](https://github.com/phausler) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160530/019902.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0097-normalizing-naming-for-negative-attributes/2854/9) ##### Revision history @@ -18,7 +18,7 @@ * We propose exposing this attribute in CF and Foundation as `CF_NOESCAPE` and `NS_NOESCAPE` * We also propose applying this declaration to a number of closure-taking APIs in CF and Foundation -[Swift Evolution Discussion Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160201/009270.html) +[Swift Evolution Discussion Thread: \[Pitch\] make @noescape the default](https://forums.swift.org/t/pitch-make-noescape-the-default/664) ## Introduction diff --git a/proposals/0013-remove-partial-application-super.md b/proposals/0013-remove-partial-application-super.md index 17a9ee43af..40cfaa459c 100644 --- a/proposals/0013-remove-partial-application-super.md +++ b/proposals/0013-remove-partial-application-super.md @@ -1,10 +1,10 @@ # Remove Partial Application of Non-Final Super Methods (Swift 2.2) * Proposal: [SE-0013](0013-remove-partial-application-super.md) -* Author: [David Farler](https://github.com/bitjammer) +* Author: [Ashley Garland](https://github.com/bitjammer) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007316.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0013-remove-partial-application-of-non-final-super-methods/1157) ## Introduction @@ -24,7 +24,7 @@ those mechanisms, I propose that we disallow partial application of non-final methods through `super`, except where the `self` parameter is implicitly captured. -[Swift Evolution Discussion Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/000947.html) +[Swift Evolution Discussion Thread](https://forums.swift.org/t/swift-2-2-removing-partial-application-of-super-method-calls/264) ## Motivation diff --git a/proposals/0014-constrained-AnySequence.md b/proposals/0014-constrained-AnySequence.md index e699b371a7..48ddfbc8ed 100644 --- a/proposals/0014-constrained-AnySequence.md +++ b/proposals/0014-constrained-AnySequence.md @@ -4,7 +4,7 @@ * Author: [Max Moiseev](https://github.com/moiseev) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Implemented (Swift 2.2)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-January/000008.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0014-constraining-anysequence-init/924) * Bug: [SR-474](https://bugs.swift.org/browse/SR-474) @@ -13,7 +13,7 @@ In order to allow `AnySequence` delegate calls to the underlying sequence, its initializer should have extra constraints. -[Swift Evolution Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/000910.html) +[Swift Evolution Discussion](https://forums.swift.org/t/restricting-anysequence-init/254) ## Motivation diff --git a/proposals/0015-tuple-comparison-operators.md b/proposals/0015-tuple-comparison-operators.md index 61a7363469..9722ed2b76 100644 --- a/proposals/0015-tuple-comparison-operators.md +++ b/proposals/0015-tuple-comparison-operators.md @@ -1,17 +1,17 @@ # Tuple comparison operators * Proposal: [SE-0015](0015-tuple-comparison-operators.md) -* Author: [Kevin Ballard](https://github.com/kballard) +* Author: [Lily Ballard](https://github.com/lilyball) * Review Manager: [Dave Abrahams](https://github.com/dabrahams) * Status: **Implemented (Swift 2.2)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/004423.html) -* Pull Request: [apple/swift#408](https://github.com/apple/swift/pull/408) +* Decision Notes: [Rationale](https://forums.swift.org/t/review-add-a-lazy-flatmap-for-sequences-of-optionals/695/4) +* Implementation: [apple/swift#408](https://github.com/apple/swift/pull/408) ## Introduction Implement comparison operators on tuples up to some arity. -[Swift Evolution Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/000892.html), [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/003907.html) +[Swift Evolution Discussion](https://forums.swift.org/t/proposal-implement-and-for-tuples-where-possible-up-to-some-high-arity/251), [Review](https://forums.swift.org/t/review-add-a-lazy-flatmap-for-sequences-of-optionals/695) Note: The review was initially started on the wrong thread with the wrong title and subsequently corrected. diff --git a/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md b/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md index 468ae1c6aa..ed282b7e13 100644 --- a/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md +++ b/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md @@ -3,10 +3,10 @@ * Proposal: [SE-0016](0016-initializers-for-converting-unsafe-pointers-to-ints.md) * Author: [Michael Buckley](https://github.com/MichaelBuckley) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000083.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0016-adding-initializers-to-int-and-uint-to-convert-from-unsafepointer-and-unsafemutablepointer/2005) * Bug: [SR-1115](https://bugs.swift.org/browse/SR-1115) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/ae2d7c24fff7cbdff754d9a4339e4fb02df5c690/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/ae2d7c24fff7cbdff754d9a4339e4fb02df5c690/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md) ## Introduction @@ -16,7 +16,7 @@ allow users to call C functions with `intptr_t` and `uintptr_t` parameters, and allow users to perform more advanced pointer arithmetic than is allowed by `UnsafePointer`s. -[Swift Evolution Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001213.html), [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013119.html) +[Swift Evolution Discussion](https://forums.swift.org/t/proposal-add-initializers-for-converting-unsafepointers-to-int-and-unit/331), [Review](https://forums.swift.org/t/review-se-0016-adding-initializers-to-int-and-uint-to-convert-from-unsafepointer-and-unsafemutablepointer/1899) ## Motivation diff --git a/proposals/0017-convert-unmanaged-to-use-unsafepointer.md b/proposals/0017-convert-unmanaged-to-use-unsafepointer.md index 9bcade90e3..cc2bb68860 100644 --- a/proposals/0017-convert-unmanaged-to-use-unsafepointer.md +++ b/proposals/0017-convert-unmanaged-to-use-unsafepointer.md @@ -3,15 +3,15 @@ * Proposal: [SE-0017](0017-convert-unmanaged-to-use-unsafepointer.md) * Author: [Jacob Bandes-Storch](https://github.com/jtbandes) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000133.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0017-change-unmanaged-to-use-unsafepointer/2461) * Bug: [SR-1485](https://bugs.swift.org/browse/SR-1485) ## Introduction The standard library [`Unmanaged` struct](https://github.com/apple/swift/blob/master/stdlib/public/core/Unmanaged.swift) provides a type-safe object wrapper that does not participate in ARC; it allows the user to make manual retain/release calls. -[Swift Evolution Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001046.html), [Proposed Rewrite Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003243.html), [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/016034.html) +[Swift Evolution Discussion](https://forums.swift.org/t/unmanaged-and-copaquepointer-vs-unsafe-mutable-pointer/295), [Proposed Rewrite Discussion](https://forums.swift.org/t/rfc-proposed-rewrite-of-unmanaged-t/612), [Review](https://forums.swift.org/t/review-se-0017-change-unmanaged-to-use-unsafepointer/2380) ## Motivation @@ -83,5 +83,5 @@ Code previously calling `Unmanaged` API with `COpaquePointer` will need to chang ## Alternatives considered -- Make no change. However, it has been [said on swift-evolution](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001096.html) that `COpaquePointer` is vestigial, and better bridging of C APIs is desired, so we do want to move in this direction. +- Make no change. However, it has been [said on swift-evolution](https://forums.swift.org/t/unmanaged-and-copaquepointer-vs-unsafe-mutable-pointer/295/3) that `COpaquePointer` is vestigial, and better bridging of C APIs is desired, so we do want to move in this direction. diff --git a/proposals/0018-flexible-memberwise-initialization.md b/proposals/0018-flexible-memberwise-initialization.md index 032f70a9ff..2bddbf171c 100644 --- a/proposals/0018-flexible-memberwise-initialization.md +++ b/proposals/0018-flexible-memberwise-initialization.md @@ -3,15 +3,13 @@ * Proposal: [SE-0018](0018-flexible-memberwise-initialization.md) * Author: [Matthew Johnson](https://github.com/anandabits) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006469.html) +* Status: **Returned for revision** +* Review: ([pitch](https://forums.swift.org/t/proposal-draft-flexible-memberwise-initialization/698)) ([review](https://forums.swift.org/t/review-se-0018-flexible-memberwise-initialization/939)) ([deferral](https://forums.swift.org/t/review-se-0018-flexible-memberwise-initialization/939/22)) ([return for revision](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction The Swift compiler is currently able to generate a memberwise initializer for use in some circumstances, however there are currently many limitations to this. This proposal builds on the idea of a compiler generated memberwise initializer, making the capability available to any initializer that opts in. -Swift-evolution thread: [Proposal Draft: flexible memberwise initialization](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/003902.html) - ## Motivation When designing initializers for a type we are currently faced with the unfortunate fact that the more flexibility we wish to offer users the more boilerplate we are required to write and maintain. We usually end up with more boilerplate and less flexibility than desired. There have been various strategies employed to mitigate this problem, including: @@ -22,9 +20,9 @@ When designing initializers for a type we are currently faced with the unfortuna Underlying this problem is the fact that initialization scales with M x N complexity (M members, N initializers). We need as much help from the compiler as we can get! -Flexible and concise initialization for both type authors and consumers will encourages using immutability where possible and removes the need for boilerplate from the concerns one must consider when designing the intializers for a type. +Flexible and concise initialization for both type authors and consumers will encourages using immutability where possible and removes the need for boilerplate from the concerns one must consider when designing the initializers for a type. -Quoting [Chris Lattner](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151130/000518.html): +Quoting [Chris Lattner](https://forums.swift.org/t/proposal-helpers-for-initializing-properties-of-same-name-as-parameters/129/8): The default memberwise initializer behavior of Swift has at least these deficiencies (IMO): 1) Defining a custom init in a struct disables the memberwise initializer, and there is no easy way to get it back. @@ -33,23 +31,23 @@ Quoting [Chris Lattner](https://lists.swift.org/pipermail/swift-evolution/Week-o 4) var properties with default initializers should have their parameter to the synthesized initializer defaulted. 5) lazy properties with memberwise initializers have problems (the memberwise init eagerly touches it). -Add to the list “all or nothing”. The compiler generates the entire initializer and does not help to eliminate boilerplate for any other initializers where it may be desirable to use memberwise intialization for a subset of members and initialize others manually. +Add to the list “all or nothing”. The compiler generates the entire initializer and does not help to eliminate boilerplate for any other initializers where it may be desirable to use memberwise initialization for a subset of members and initialize others manually. -It is common to have a type with a number of public members that are intended to be configured by clients, but also with some private state comprising implementation details of the type. This is especially prevalent in UI code which may expose many properties for configuring visual appearance, etc. Flexibile memberwise initialization can provide great benefit in these use cases, but it immediately becomes useless if it is "all or nothing". +It is common to have a type with a number of public members that are intended to be configured by clients, but also with some private state comprising implementation details of the type. This is especially prevalent in UI code which may expose many properties for configuring visual appearance, etc. Flexible memberwise initialization can provide great benefit in these use cases, but it immediately becomes useless if it is "all or nothing". -We need a flexible solution that can synthesize memberwise initialization for some members while allowing the type auther full control over initialization of implementation details. +We need a flexible solution that can synthesize memberwise initialization for some members while allowing the type author full control over initialization of implementation details. ## Proposed solution I propose adding a `memberwise` declaration modifier for initializers which allows them to *opt-in* to synthesis of memberwise initialization. -This proposal adopts a model for property eligibility where stored properties automatically recieve memberwise initialization parameters unless they are deemed ineligible for one of several reasons. An *opt-in* model using a `memberwise` declaration modifier allowing properties to *opt-in* to memberwise initialization synthesis is also possible. +This proposal adopts a model for property eligibility where stored properties automatically receive memberwise initialization parameters unless they are deemed ineligible for one of several reasons. An *opt-in* model using a `memberwise` declaration modifier allowing properties to *opt-in* to memberwise initialization synthesis is also possible. The two approaches are not mutually exclusive: it is possible to use the *automatic* model when no properties have the `memberwise` declaration modifier and the *opt-in* model when one or more properties do have the `memberwise` declaration modifier. A future enhancement to this proposal may introduce the *opt-in* model, allowing programmers to choose which model is preferred for a specific type they are authoring. The *automatic* model of the current proposal determines the set of properties that receive memberwise initialization parameters by considering *only* the initializer declaration and the declarations for all properties that are *at least* as visible as the initializer (including any behaviors attached to the properties). The rules are as follows: -1. The access level of the property is *at least* as visible as the memberwise initializer. The visiblity of the **setter** is used for `var` properties. +1. The access level of the property is *at least* as visible as the memberwise initializer. The visibility of the **setter** is used for `var` properties. 2. They do not have a behavior which prohibits memberwise initialization (e.g. the 'lazy' behavior). 3. If the property is a `let` property it *may not* have an initial value. @@ -194,7 +192,7 @@ Throughout this design the term **memberwise initialization parameter** is used 1. Determine the set of properties eligible for memberwise initialization synthesis. Properties are eligible for memberwise initialization synthesis if: - 1. The access level of the property is *at least* as visible as the memberwise initializer. The visiblity of the **setter** is used for `var` properties. + 1. The access level of the property is *at least* as visible as the memberwise initializer. The visibility of the **setter** is used for `var` properties. 2. They do not have a behavior which prohibits memberwise initialization. 3. If the property is a `let` property it *may not* have an initial value. @@ -218,7 +216,7 @@ This proposal will also support generating an *implicit* memberwise initializer 2. The type is: 1. a struct 2. a root class - 3. a class whose superclass has a designated intializer requiring no arguments + 3. a class whose superclass has a designated initializer requiring no arguments The implicitly generated memberwise initializer will have the highest access level possible while still allowing all stored properties to be eligible for memberwise parameter synthesis, but will have at most `internal` visibility. Currently this means its visibility will be `internal` when all stored properties of the type have setters with *at least* `internal` visibility, and `private` otherwise (when one or more stored properties are `private` or `private(set)`). @@ -230,7 +228,7 @@ The changes described in this proposal are *almost* entirely additive. The only 1. If the implicitly synthesized memberwise initializer was only used *within* the same source file no change is necessary. An implicit `private` memberwise initializer will still be synthesized by the compiler. 2. A mechanical migration could generate the explicit code necessary to declare the previously implicit initializer. This would be an `internal` memberwise initializer with *explicit* parameters used to manually initialize the stored properties with `private` setters. -3. If the "Access control for init" enhancement were accepted the `private` members could have their access control modified to `private internal(init)` which would allow the implict memberwise intializer to continue to have `internal` visibility as all stored properties would be eligible for parameter synthesis by an `internal` memberwise initializer. +3. If the "Access control for init" enhancement were accepted the `private` members could have their access control modified to `private internal(init)` which would allow the implicit memberwise initializer to continue to have `internal` visibility as all stored properties would be eligible for parameter synthesis by an `internal` memberwise initializer. The only other impact on existing code is that memberwise parameters corresponding to `var` properties with initial values will now have default values. This will be a change in the behavior of the implicit memberwise initializer but will not break any code. The change will simply allow new code to use that initializer without providing an argument for such parameters. @@ -295,7 +293,7 @@ The rules of the current proposal are designed to synthesize memberwise paramete Introducing a `memberwise` declaration modifier for properties would allow programmers to specify exactly which properties should participate in memberwise initialization synthesis. It allows full control and has the clarity afforded by being explicit. -Specifc use cases this feature would support include allowing `private` properties to receive synthesized memberwise parameters in a `public` initializer, or allow `public` properties to be ommitted from parameter synthesis. +Specific use cases this feature would support include allowing `private` properties to receive synthesized memberwise parameters in a `public` initializer, or allow `public` properties to be omitted from parameter synthesis. An example of this @@ -325,7 +323,7 @@ struct S { ### Access control for init -In some cases it may be desirable to be able to specify distinct access control for memberwise initialization when using the *automatic* model, for example if that model *almost* has the desired behavior, but the initialization visibiltiy of one property must be adjusted to produce the necessary result. +In some cases it may be desirable to be able to specify distinct access control for memberwise initialization when using the *automatic* model, for example if that model *almost* has the desired behavior, but the initialization visibility of one property must be adjusted to produce the necessary result. The syntax used would be identical to that used for specifying distinct access control for a setter. This feature would likely have its greatest utility in allowing more-private members to participate in more-public memberwise initializers. It may also be used to inhibit memberwise initialization for some members, although that use would usually be discouraged if the `@nomemberwise` proposal were also accepted. @@ -350,7 +348,7 @@ struct S { If this enhancement were submitted the first property eligibility rule would be updates as follows: -1. Their **init** access level is *at least* as visible as the memberwise initializer. If the property does not have an **init** acccess level, the access level of its **setter** must be *at least* as visible as the memberwise initializer. +1. Their **init** access level is *at least* as visible as the memberwise initializer. If the property does not have an **init** access level, the access level of its **setter** must be *at least* as visible as the memberwise initializer. ### @nomemberwise @@ -402,7 +400,7 @@ struct S { init(s: String) { /* synthesized */ self.s = s - // body of the user's intializer remains + // body of the user's initializer remains i = 42 } } @@ -410,7 +408,7 @@ struct S { ### Memberwise initializer chaining / parameter forwarding -Ideally it would be possible to define convenience and delegating initializers without requiring them to manually declare parameters and pass arguments to the designated initializer for memberwise intialized properties. It would also be ideal if designated initializers also did not have to the same for memberwise intialization parmaeters of super. +Ideally it would be possible to define convenience and delegating initializers without requiring them to manually declare parameters and pass arguments to the designated initializer for memberwise initialized properties. It would also be ideal if designated initializers also did not have to the same for memberwise initialization parameters of super. A general solution for parameter forwarding would solve this problem. A future parameter forwarding proposal to support this use case and others is likely to be pursued. @@ -436,8 +434,8 @@ Obviously supporting memberwise initialization with Cocoa classes would require This is a reasonable option and and I expect a healthy debate about which default is better. The decision to adopt the *automatic* model by default was made for several reasons: -1. The memberwise initializer for structs does not currently require an annotation for properties to opt-in. Requiring an annotation for a mechanism designed to supercede that mechanism may be viewed as boilerplate. -2. Stored properties with public visibility are often intialized directly with a value provided by the caller. +1. The memberwise initializer for structs does not currently require an annotation for properties to opt-in. Requiring an annotation for a mechanism designed to supersede that mechanism may be viewed as boilerplate. +2. Stored properties with public visibility are often initialized directly with a value provided by the caller. 3. Stored properties with **less visibility** than a memberwise initializer are not eligible for memberwise initialization. No annotation is required to indicate that and it is usually not desired. 4. The *automatic* model cannot exist unless it is the default. The *opt-in* model can exist alongside the *automatic* model and itself be opted-into simply by specifying the `memberwise` declaration modifier on one or more properties. @@ -462,9 +460,9 @@ Reasons to limit memberwise parameter synthesis to members which are *at least* 3. It is likely the more common desire of the author of an initializer. If the caller can’t see a member it probably doesn’t make sense to allow them to initialize it. 4. If we expose more private-members by default then memberwise initialization is useless under the current proposal in many cases. There would be no way to prevent synthesis of parameters for more-private members. We have to choose between allowing callers to initialize our internal state or forgoing the benefit of memberwise initialization. 5. If a proposal for `@nomemberwise` is put forward and adopted that would allow us to prevent synthesis of parameters for members as desired. Unfortunately `@nomemberwise` would need to be used much more heavily than it otherwise would (i.e. to prevent synthesis of memberwise parameters for more-private members). It would be better if `@nomemberwise` was not necessary most of the time. -6. If callers must be able to provide memberwise arguments for more-private members directly it is still possible to allow that while taking advantage of memberwise initialization for same-or-less-private members. You just need to declare a `memberwise init` with explicitly declared parameters for the more-private members and initialize them manually in the body. If the "Access control for init" enhancement is accepted another option would be upgrading the visibility of `init` for the more-private member while retaining its access level for the getter and setter. Requiring the programmer to explicitly expose a more-private member either via `init` access control or by writing code that it directly is a arguably a very good thing. +6. If callers must be able to provide memberwise arguments for more-private members directly it is still possible to allow that while taking advantage of memberwise initialization for same-or-less-private members. You just need to declare a `memberwise init` with explicitly declared parameters for the more-private members and initialize them manually in the body. If the "Access control for init" enhancement is accepted another option would be upgrading the visibility of `init` for the more-private member while retaining its access level for the getter and setter. Requiring the programmer to explicitly expose a more-private member either via `init` access control or by writing code that it directly is arguably a very good thing. -Reasons we might want to allow memberwise parameter synthesis for members with lower visiblity than the initializer: +Reasons we might want to allow memberwise parameter synthesis for members with lower visibility than the initializer: 1. Not doing so puts tension between access control for stored properties and memberwise inits. You have to choose between narrower access control or getting the benefit of a memberwise init. Another way to say it: this design means that narrow access control leads to boilerplate. @@ -472,7 +470,7 @@ NOTE: The tension mentioned here is lessened by #6 above: memberwise initializat ### Require initializers to explicitly specify memberwise initialization parameters -The thread "[helpers for initializing properties of the same name as parameters](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151130/000428.html)" discussed an idea for synthesizing property initialization in the body of the initializer while requiring the parameters to be declard explicitly. +The thread "[helpers for initializing properties of the same name as parameters](https://forums.swift.org/t/proposal-helpers-for-initializing-properties-of-same-name-as-parameters/129/3)" discussed an idea for synthesizing property initialization in the body of the initializer while requiring the parameters to be declared explicitly. ```swift struct Foo { @@ -496,7 +494,7 @@ Under the current proposal full control is still available. It requires initial I believe the `memberwise` declaration modifier on the initializer and the placeholder in the parameter list make it clear that the compiler will synthesize additional parameters. Furthermore, IDEs and generated documentation will contain the full, synthesized signature of the initializer. -Finally, this idea is not mutually exclusive with the current proposal. It could even work in the declaration of a memberwise initializer, so long the corresponding property was made ineligible for memberwise intialization synthesis. +Finally, this idea is not mutually exclusive with the current proposal. It could even work in the declaration of a memberwise initializer, so long the corresponding property was made ineligible for memberwise initialization synthesis. ### Adopt "type parameter list" syntax like Kotlin and Scala @@ -542,7 +540,7 @@ Responses to these points follow: 1. If the expansion of this syntax does not supply initial values to the synthesized properties and only uses the default value for parameters of the synthesized initializer this is true. The downside of doing this is that `var` properties no longer have an initial value which may be desirable if you write additional initializers for the type. I believe we should continue the discussion about default values for `let` properties. Ideally we can find an acceptable solution that will work with the current proposal, as well as any additional syntactic sugar we add in the future. -2. I don't believe allowing parameter labels for memberwise initialization parameters is a good idea. Callers are directly initializing a property and are best served by a label that matches the name of the property. If you really need to provide a different name you can still do so by writing your initializer manually. With future enhancements to the current proposal you may be able to use memberwise intialization for properties that do not require a custom label while manually initialzing properties that do need one. +2. I don't believe allowing parameter labels for memberwise initialization parameters is a good idea. Callers are directly initializing a property and are best served by a label that matches the name of the property. If you really need to provide a different name you can still do so by writing your initializer manually. With future enhancements to the current proposal you may be able to use memberwise initialization for properties that do not require a custom label while manually initializing properties that do need one. 3. The Scala / Kotlin syntax is indeed more concise in some cases, but not in all cases. Under this proposal the example given above is actually more concise than it is with that syntax: ```swift diff --git a/proposals/0019-package-manager-testing.md b/proposals/0019-package-manager-testing.md index 98fd318b53..252c9bdaa0 100644 --- a/proposals/0019-package-manager-testing.md +++ b/proposals/0019-package-manager-testing.md @@ -3,8 +3,8 @@ * Proposal: [SE-0019](0019-package-manager-testing.md) * Authors: [Max Howell](https://github.com/mxcl), [Daniel Dunbar](https://github.com/ddunbar), [Mattt Thompson](https://github.com/mattt) * Review Manager: [Rick Ballard](https://github.com/rballard) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007278.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0019-swift-testing-package-manager/1155) * Bug: [SR-592](https://bugs.swift.org/browse/SR-592) ## Introduction @@ -13,12 +13,12 @@ Testing is an essential part of modern software development. Tight integration of testing into the Swift Package Manager will help ensure a stable and reliable packaging ecosystem. -[SE Review Link](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005397.html), [Second Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006758.html) +[SE Review Link](https://forums.swift.org/t/review-se-0019-swift-testing-package-manager/909), [Second Review](https://forums.swift.org/t/review-se-0019-swift-testing-package-manager/1084) ## Proposed Solution We propose to extend our conventional package directory layout -to accomodate test modules. +to accommodate test modules. Any subdirectory of the package root directory named "Tests" or any subdirectory of an existing module directory named "Tests" will comprise a test module. diff --git a/proposals/0020-if-swift-version.md b/proposals/0020-if-swift-version.md index 2ad7621e01..a59e6ad4dc 100644 --- a/proposals/0020-if-swift-version.md +++ b/proposals/0020-if-swift-version.md @@ -1,10 +1,10 @@ # Swift Language Version Build Configuration * Proposal: [SE-0020](0020-if-swift-version.md) -* Author: [David Farler](https://github.com/bitjammer) +* Author: [Ashley Garland](https://github.com/bitjammer) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Implemented (Swift 2.2)** -* Commit: [apple/swift@c32fb8e](https://github.com/apple/swift/commit/c32fb8e7b9a67907e8b6580a46717c6a345ec7c6) +* Implementation: [apple/swift@c32fb8e](https://github.com/apple/swift/commit/c32fb8e7b9a67907e8b6580a46717c6a345ec7c6) ## Introduction @@ -12,8 +12,8 @@ This proposal aims to add a new build configuration option to Swift 2.2: `#if swift`. Swift-evolution threads: -- [Swift 2.2: #if swift language version](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003385.html) -- [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006398.html) +- [Swift 2.2: #if swift language version](https://forums.swift.org/t/proposal-swift-2-2-if-swift-language-version/640) +- [Review](https://forums.swift.org/t/review-se-0020-swift-language-version-build-configuration/1034) ## Motivation diff --git a/proposals/0021-generalized-naming.md b/proposals/0021-generalized-naming.md index fd42e004f4..20256b9ba5 100644 --- a/proposals/0021-generalized-naming.md +++ b/proposals/0021-generalized-naming.md @@ -4,8 +4,8 @@ * Author: [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Joe Groff](https://github.com/jckarter) * Status: **Implemented (Swift 2.2)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-January/000021.html) -* Commit: [apple/swift@ecfde0e](https://github.com/apple/swift/commit/ecfde0e71c61184989fde0f93f8d6b7f5375b99a) +* Decision Notes: [Rationale](https://forums.swift.org/t/review-naming-functions-with-argument-labels/1046/11) +* Implementation: [apple/swift@ecfde0e](https://github.com/apple/swift/commit/ecfde0e71c61184989fde0f93f8d6b7f5375b99a) ## Introduction @@ -13,10 +13,10 @@ Swift includes support for first-class functions, such that any function (or method) can be placed into a value of function type. However, when specifying the name of a function, one can only provide the base name, (e.g., `insertSubview`) without the argument labels. For overloaded functions, this means that one must disambiguate based on type information, which is awkward and verbose. This proposal allows one to provide argument labels when referencing a function, eliminating the need to provide type context in most cases. -Swift-evolution thread: The first draft of this proposal was discussed [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/004555.html). It included support for naming getters/setters (separately brought up by Michael Henson -[here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/002168.html), +Swift-evolution thread: The first draft of this proposal was discussed [here](https://forums.swift.org/t/proposal-draft-generalized-naming-for-any-function/787). It included support for naming getters/setters (separately brought up by Michael Henson +[here](https://forums.swift.org/t/proposal-expose-getter-setters-in-the-same-way-as-regular-methods/501), continued -[here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002203.html)). Joe Groff [convinced](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/004579.html) me that lenses are a better approach for working with getters/setters, so I've dropped them from this version of the proposal. +[here](https://forums.swift.org/t/proposal-expose-getter-setters-in-the-same-way-as-regular-methods/501/5)). Joe Groff [convinced](https://forums.swift.org/t/proposal-draft-generalized-naming-for-any-function/787/4) me that lenses are a better approach for working with getters/setters, so I've dropped them from this version of the proposal. ## Motivation @@ -137,12 +137,12 @@ code. ## Alternatives considered * Joe Groff - [notes](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003008.html) + [notes](https://forums.swift.org/t/proposal-expose-getter-setters-in-the-same-way-as-regular-methods/501/3) that *lenses* are a better solution than manually retrieving getter/setter functions when the intent is to actually operate on the properties. -* Bartlomiej Cichosz [suggests](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151228/004739.html) a general partial application syntax using `_` as a placeholder, e.g., +* Bartlomiej Cichosz [suggests](https://forums.swift.org/t/proposal-draft-generalized-naming-for-any-function/787/6) a general partial application syntax using `_` as a placeholder, e.g., ```swift aGameView.insertSubview(_, aboveSubview: playingSurfaceView) diff --git a/proposals/0022-objc-selectors.md b/proposals/0022-objc-selectors.md index e0569414ff..05f5d73410 100644 --- a/proposals/0022-objc-selectors.md +++ b/proposals/0022-objc-selectors.md @@ -4,8 +4,8 @@ * Author: [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Joe Groff](https://github.com/jckarter) * Status: **Implemented (Swift 2.2)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160125/007797.html) -* Pull Request: [apple/swift#1170](https://github.com/apple/swift/pull/1170) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0022-referencing-the-objective-c-selector-of-a-method/1194) +* Implementation: [apple/swift#1170](https://github.com/apple/swift/pull/1170) ## Introduction @@ -15,7 +15,7 @@ In Swift 2, Objective-C selectors are written as string literals with `Selector` initialization syntax that refers to a specific method via its Swift name. -Swift-evolution thread: [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006282.html), [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/006913.html), [Amendments after acceptance](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160523/018698.html) +Swift-evolution thread: [here](https://forums.swift.org/t/proposal-draft-referencing-the-objective-c-selector-of-a-method/1022), [Review](https://forums.swift.org/t/review-se-0022-referencing-the-objective-c-selector-of-a-method/1118), [Amendments after acceptance](https://forums.swift.org/t/amendment-se-0022-referencing-the-objective-c-selector-of-a-method/2737) ## Motivation @@ -53,7 +53,7 @@ having to do the naming translation manually and get static checking that the method exists and is exposed to Objective-C. This proposal composes with the [Naming Functions with Argument Labels -proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006262.html), which lets us name methods along with their argument labels, e.g.: +proposal](https://forums.swift.org/t/proposal-draft-2-naming-functions-with-argument-labels/1018), which lets us name methods along with their argument labels, e.g.: let sel = #selector(UIView.insertSubview(_:atIndex:)) // produces the Selector "insertSubview:atIndex:" @@ -111,8 +111,8 @@ from a string. ## Alternatives considered The primary alternative is [type-safe -selectors](https://lists.swift.org/pipermail/swift-evolution/2015-December/000233.html), -which would introduce a new "selector" calling convetion to capture +selectors](https://forums.swift.org/t/type-safe-selectors/108), +which would introduce a new "selector" calling convention to capture the type of an `@objc` method, including its selector. One major benefit of type-safe selectors is that they can carry type information, improving type safety. From that discussion, referencing diff --git a/proposals/0023-api-guidelines.md b/proposals/0023-api-guidelines.md index 7fe81a55c9..e8e5c3fe02 100644 --- a/proposals/0023-api-guidelines.md +++ b/proposals/0023-api-guidelines.md @@ -1,10 +1,10 @@ # API Design Guidelines * Proposal: [SE-0023](0023-api-guidelines.md) -* Authors: [Dave Abrahams](https://github.com/dabrahams), [Doug Gregor](https://github.com/DougGregor), [Dmitri Gribenko](https://github.com/gribozavr), [Ted Kremenek](https://github.com/tkremenek), [Chris Lattner](http://github.com/lattner), Alex Migicovsky, [Max Moiseev](https://github.com/moiseev), Ali Ozer, [Tony Parker](https://github.com/parkera) +* Authors: [Dave Abrahams](https://github.com/dabrahams), [Doug Gregor](https://github.com/DougGregor), [Dmitri Gribenko](https://github.com/gribozavr), [Ted Kremenek](https://github.com/tkremenek), [Chris Lattner](https://github.com/lattner), Alex Migicovsky, [Max Moiseev](https://github.com/moiseev), Ali Ozer, [Tony Parker](https://github.com/parkera) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000053.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0023-api-design-guidelines/1666) ## Reviewer notes @@ -12,11 +12,11 @@ This review is part of a group of three related reviews, running concurrently: * [SE-0023 API Design Guidelines](0023-api-guidelines.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007353.html)) + ([Review](https://forums.swift.org/t/review-se-0023-api-design-guidelines/1162)) * [SE-0006 Apply API Guidelines to the Standard Library](0006-apply-api-guidelines-to-the-standard-library.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007354.html)) + ([Review](https://forums.swift.org/t/review-se-0006-apply-api-guidelines-to-the-standard-library/1163)) * [SE-0005 Better Translation of Objective-C APIs Into Swift](0005-objective-c-name-translation.md) - ([Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007355.html)) + ([Review](https://forums.swift.org/t/review-se-0005-better-translation-of-objective-c-apis-into-swift/1164)) These reviews are running concurrently because they interact strongly (e.g., an API change in the standard library will correspond to a diff --git a/proposals/0024-optional-value-setter.md b/proposals/0024-optional-value-setter.md index 5a3cd05f1e..98f45801cc 100644 --- a/proposals/0024-optional-value-setter.md +++ b/proposals/0024-optional-value-setter.md @@ -4,18 +4,18 @@ * Author: [James Campbell](https://github.com/jcampbell05) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-February/000043.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0024-optional-value-setter/1528) ## Introduction Introduce a new operator an "Optional Value Setter". If the optional is set via this operator then the new value is only set if there isn't an already existing value. -Swift-evolution thread: [link to the discussion thread for that proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002624.html) +Swift-evolution thread: [link to the discussion thread for that proposal](https://forums.swift.org/t/optional-setting/553) ## Motivation -In certain cases the `??` operation doesn't help with lengthly variable names, i.e., `really.long.lvalue[expression] = really.long.lvalue[expression] ?? ""`. In addtition to this other languages such as Ruby contain a pipe operator `really.long.lvalue[expression] ||= ""` which works the same way and which is very popular. This lowers the barrier of entry for programmers from that language. +In certain cases the `??` operation doesn't help with lengthy variable names, i.e., `really.long.lvalue[expression] = really.long.lvalue[expression] ?? ""`. In addtition to this other languages such as Ruby contain a pipe operator `really.long.lvalue[expression] ||= ""` which works the same way and which is very popular. This lowers the barrier of entry for programmers from that language. In the interest in conciseness and clarity I feel this would be a great addition to swift and would bring the length of that previous statement from diff --git a/proposals/0025-scoped-access-level.md b/proposals/0025-scoped-access-level.md index 5aa6de905b..470666dd8b 100644 --- a/proposals/0025-scoped-access-level.md +++ b/proposals/0025-scoped-access-level.md @@ -2,29 +2,29 @@ * Proposal: [SE-0025](0025-scoped-access-level.md) * Author: Ilya Belenkiy -* Status: **Implemented (Swift 3)** -* Review Manager: [Doug Gregor](http://github.com/DougGregor) -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160404/014116.html) +* Status: **Implemented (Swift 3.0)** +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0025-scoped-access-level-next-steps/1797/131) * Bug: [SR-1275](https://bugs.swift.org/browse/SR-1275) -* Previous revision: [1](https://github.com/apple/swift-evolution/blob/e4328889a9643100177aef19f6f428855c5d0cf2/proposals/0025-scoped-access-level.md) +* Previous revision: [1](https://github.com/swiftlang/swift-evolution/blob/e4328889a9643100177aef19f6f428855c5d0cf2/proposals/0025-scoped-access-level.md) ## Introduction -Scoped access level allows to hide implementation details of a class or a class extension at the class/extension level, instead of a file. It is a concise expression of the intent that a particular part of a class or extension definition is there only to implement a public API for other classes or extensions, and must not be used directly anywhere outside of the scope of the class or the extension. +Scoped access level allows hiding implementation details of a class or a class extension at the class/extension level, instead of a file. It is a concise expression of the intent that a particular part of a class or extension definition is there only to implement a public API for other classes or extensions and must not be used directly anywhere outside of the scope of the class or the extension. -[Swift Evolution Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011162.html), [Next Steps Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160314/012604.html) +[Swift Evolution Discussion](https://forums.swift.org/t/review-se-0025-scoped-access-level/1588), [Next Steps Discussion](https://forums.swift.org/t/se-0025-scoped-access-level-next-steps/1797) ## Motivation -Currently, the only reliable way to hide implementations details of a class is to put the code in a separate file and mark it as private. This is not ideal for the following reasons: +Currently, the only reliable way to hide implementation details of a class is to put the code in a separate file and mark it as private. This is not ideal for the following reasons: - It is not clear whether the implementation details are meant to be completely hidden or can be shared with some related code without the danger of misusing the APIs marked as private. If a file already has multiple classes, it is not clear if a particular API is meant to be hidden completely or can be shared with the other classes. -- It forces a one class per file structure, which is very limiting. Putting related APIs and/or related implementations in the same file helps ensure consistency and reduces the time to find a particular API or implementation. This does not mean that the classes in the same file need to share otherwise hidden APIs, but there is no way to express it with the current access levels. +- It forces a one class per file structure, which is very limiting. Putting related APIs and/or related implementations in the same file helps ensure consistency and reduces the time to find a particular API or implementation. This does not mean that the classes in the same file need to share otherwise hidden APIs, but there is no way to express such sharability with the current access levels. -Another, less reliable, way is to prefix APIs that are meant to be hidden with a `_` or do something similar. That works, but it’s not enforced by the compiler, and those APIs show up in tools like code completion, so the programmer has to filter out the noise - although these tools could quite easily support hiding methods with the `_` prefix standard. Also, there is a greater danger of using private APIs if they do something similar to public APIs but are somehow more optimized (because they make additional assumptions about the internal state). +Another, less reliable, way is to prefix APIs that are meant to be hidden with a `_` or do something similar. That works, but it’s not enforced by the compiler, and those APIs show up in tools like code completion, so the programmer has to filter out the noise — although these tools could quite easily support hiding methods with the `_` prefix standard. Also, there is a greater danger of using private APIs if they do something similar to public APIs but are somehow more optimized (because they make additional assumptions about the internal state). -The existing solutions are in some ways similar to those for untyped collections. It is usually possible to give collection a name that would imply the type of elements it holds (similar to using _ to indicate private), but it is not the same as specifying it explicitly. Just as with generics, the intent not to share the implementation details with any other class is much clearer with support from the language as opposed to where the code is in the project. Also, with untyped collections, it is possible to add an element of a different type (deliberately or not). Generics make it impossible, and it’s enforced by the compiler. Similarly, a dedicated access level modifier could enforce hiding implementation details at the compiler level and make it impossible to accidentally misuse or (deliberately use) implementation details in a context that the class meant not to share. +The existing solutions are in some ways similar to those for untyped collections. It is usually possible to give a collection a name that would imply the type of elements it holds (similar to using _ to indicate private), but it is not the same as specifying it explicitly. Just as with generics, the intent not to share the implementation details with any other class is much clearer with support from the language as opposed to relying on where the code is in the project. Also, with untyped collections, it is possible to add an element of a different type (deliberately or not). Generics make that impossible, and it’s enforced by the compiler. Similarly, a dedicated access level modifier could enforce hiding implementation details at the compiler level and make it impossible to accidentally misuse or (deliberately use) implementation details in a context that the class meant not to share. ## Proposed solution @@ -72,7 +72,7 @@ extension A { ### Complications with private types -When a type is defined with the `private` access modifier, things become a little more complicated. Of course the type itself is only visible within the lexical scope it is defined in, but what about members of the type? +When a type is defined with the `private` access modifier, things become a little more complicated. Of course the type itself is visible only within the lexical scope it is defined in, but what about members of the type? ```swift class Outer { @@ -89,15 +89,15 @@ class Outer { } ``` -If the members of a private type are themselves considered `private`, it is very clear that they cannot be used outside of the type itself. However, it is also not currently permitted for a member to have an access level greater than its enclosing type. This produces a conundrum: the type can be referenced within its enclosing lexical scope, but none of its members can. +If the members of a private type are themselves considered `private`, it is very clear that they cannot be used outside of the type itself. However, it is also not currently permitted for a member to have an access level greater than its enclosing type. This restriction produces a conundrum: the type can be referenced within its enclosing lexical scope, but none of its members can. Ignoring formal concerns, the most likely expected behavior is that members not explicitly marked `private` are permitted to be accessed within the enclosing scope of the private type. To achieve this goal, we relax a few of the existing rules: - The default level of access control anywhere is `internal`. -- The compiler should not warn when a broader level of access control is used within a type with more restrictive access, such as `internal` within a `private` type. This allows the owner of the type to design the access they would use were they to make the type more widely accessible. (The members still cannot be accessed outside the enclosing lexical scope because the type itself is still restricted, i.e. outside code will never encounter a value of that type.) +- The compiler should not warn when a broader level of access control is used within a type with more restrictive access, such as `internal` within a `private` type. This allows the designer of the type to select the access they would use were they to make the type more widely accessible. (The members still cannot be accessed outside the enclosing lexical scope because the type itself is still restricted, i.e. outside code will never encounter a value of that type.) -- A member may not have a type that references any declarations that aren't accessible wherever the member is accessible. (This replaces an existing rule that states that the type of a declaration may not reference any declarations that have broader access.) This permits the following code: +- The type of a member can reference only declarations that are accessible wherever the member is accessible. (This relaxes an existing rule that states that the type of a declaration may not reference any declarations that have broader access.) The change permits the following code: ```swift struct Outer { @@ -121,33 +121,33 @@ Ignoring formal concerns, the most likely expected behavior is that members not - A member that satisfies a protocol requirement may never be `private`. Similarly, a `required` initializer may never be `private`. -- Extensions with explicit access modifiers continue to override the default `internal` access by specifying a default *scope.* Therefore, within an extension marked `private`, the default access level is `fileprivate` (since extensions are always declared at file scope). This matches the behavior of types declared `private` at file scope. +- As before, an extension with an explicit access modifier overrides the default `internal` access by specifying a default *scope*. Therefore, within an extension marked `private`, the default access level is `fileprivate` (since extensions are always declared at file scope). This matches the behavior of types declared `private` at file scope. - The explicit access modifier on an extension also continues to set the maximum allowed access within that extension. The compiler will continue to warn on overly broad access within an extension with an explicit access modifier. +- As before, an explicit access modifier on an extension sets the maximum allowed access within that extension, and the compiler will warn on overly broad access within an extension that has an explicit access modifier. ## Impact on existing code -The existing code will need to rename `private` to `fileprivate` to achieve the same semantics. In many cases the new meaning of `private` is likely to compile as well and the code will then run exactly as before. +Existing code will need to rename `private` to `fileprivate` to achieve the same semantics, although in many cases the new meaning of `private` is likely to still compile and to run exactly as before. ## Alternatives considered -1. Do nothing and use `_` and `/` or split the code into more files and use the private modifier. The proposed solution makes the intent much clearer, it would be enforced by the compiler, and the language does not dictate how the code must be organized. +1. Do nothing and use `_` and `/` or split the code into more files and use the `private` modifier. The proposed solution makes the intent much clearer and enforced by the compiler, and the language does not dictate how the code must be organized. 2. Introduce a scoped namespace that would make it possible to hide APIs in part of the file. This introduces an extra level of grouping and nesting and forces APIs to be grouped by access level instead of a logical way that may make more sense. -3. Introduce a different access modifier and keep the current names unchanged. The proposal followed this approach to be completely compatible with the existing code, but the core team decided that it was better to use `private` for this modifier because it’s much closer to what the term means in other languages. +3. Introduce a different access modifier and keep the current names unchanged. The proposal originally followed this approach to be completely compatible with the existing code, but the core team decided that it was better to use `private` for this modifier because it’s much closer to what the term means in other languages. ### Alternatives considered for "the private type issue" 1. Use `fileprivate` rather than `internal` as the default access level within `private` and `fileprivate` types. This is a more narrow change from the original model, but didn't have any benefits once we determined that the warning for unnecessarily broad access wasn't useful. -2. Introduce a new "parent" access level that declares an entity to be accessible within the *parent* lexical scope, rather than the immediately enclosing scope. This seems effective for `private` but overly specific within types with any broader access, and not worth the added complexity. We would also have to determine its name within the language, or decide that this level of access could not be spelled explicitly and was only available as the default access within private types. +2. Introduce a new "parent" access level that declares an entity to be accessible within the *parent* lexical scope, rather than the immediately enclosing scope. This idea seems effective for `private` but is overly specific within types with any broader access and not worth the added complexity. We would also have to determine its name within the language, or decide that this level of access could not be spelled explicitly and was available only as the default access within private types. -3. Introduce a new "default" access level that names the default access within a scope. Within a `private` type, this would have the "parent" semantics from (2); elsewhere it would follow the rules laid down in previous versions of Swift. This likewise added complexity to the model for only a small gain in expressivity, and we would likewise have to determine a name for it within the language. +3. Introduce a new "default" access level that names the default access within a scope. Within a `private` type, this would have the "parent" semantics from (2); elsewhere it would follow the rules laid down in previous versions of Swift. This idea likewise added complexity to the model for only a small gain in expressivity, and we would likewise have to determine a name for it within the language. ## Changes from revision 1 -- The proposal was amended post-acceptance by [Robert Widmann][] and [Jordan Rose][] to account for "[the private type issue](#complications-with-private-types)". Only this section was added; there were no semantic changes to the rest of the proposal. This amendment requires a small amount of work to implement compared to the [alternatives considered](#alternatives-considered-for-the-private-type-issue), and was determined by the Core Team to be a small enough set of changes in the spirit of the original proposal that a full review was not necessary. +- The proposal was amended post-acceptance by [Robert Widmann][] and [Jordan Rose][] to account for "[the private type issue](#complications-with-private-types)". Only that section was added; there were no semantic changes to the rest of the proposal. This amendment requires a small amount of work to implement compared to the [alternatives considered](#alternatives-considered-for-the-private-type-issue), and was determined by the Core Team to be a small enough set of changes in the spirit of the original proposal that a full review was not necessary. [Robert Widmann]: https://github.com/CodaFi [Jordan Rose]: https://github.com/jrose-apple diff --git a/proposals/0026-abstract-classes-and-methods.md b/proposals/0026-abstract-classes-and-methods.md old mode 100755 new mode 100644 index 5b663a6a74..24b89a5eb9 --- a/proposals/0026-abstract-classes-and-methods.md +++ b/proposals/0026-abstract-classes-and-methods.md @@ -3,8 +3,8 @@ * Proposal: [SE-0026](0026-abstract-classes-and-methods.md) * Author: David Scrève * Review Manager: [Joe Groff](https://github.com/jckarter/) -* Status: **Deferred** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000056.html) +* Status: **Rejected** +* Review: ([pitch](https://forums.swift.org/t/proposal-draff-abstract-classes-and-methods/965)) ([review](https://forums.swift.org/t/review-se-0026-abstract-classes-and-methods/1580)) ([deferral](https://forums.swift.org/t/deferred-se-0026-abstract-classes-and-methods/1705)) ([rejection](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction @@ -14,11 +14,9 @@ they cannot have attributes as classes have. A partial class combines the behavior of a class with the requirement of implementing methods in inherited class like protocols. -[Swift-Evolution Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005728.html) - ## Motivation -like pure virtual methods in C++ and abtract classes in Java and C#, frameworks development +like pure virtual methods in C++ and abstract classes in Java and C#, frameworks development sometimes required abstract classes facility. An abstract class is like a regular class, but some methods/properties are not implemented and must be implemented in one of inherited classes. @@ -38,7 +36,7 @@ class RESTClient { var timeout = 3000 var url : String { - assert(false,"Must be overriden") + assert(false,"Must be overridden") return "" } @@ -99,7 +97,7 @@ class MyRestServiceClient : RESTClient { ``` ## Detailed design -An abstract class cannot be instanciated. +An abstract class cannot be instantiated. Abstract method/property cannot have implementation. @@ -133,8 +131,8 @@ stabilizing in Swift 3.0. ## Alternatives considered As first reading, it seems that protocols and protocol extensions might fit the need. It -actually does not because abstract classes can have attributs and properties that -protocols does not support. +actually does not because abstract classes can have attributes and properties that +protocols do not support. An alternative solution would be to add attributes to protocols and protocol extensions, but this might break compatibility with Objective-C runtime. diff --git a/proposals/0027-string-from-code-units.md b/proposals/0027-string-from-code-units.md index 115db88c4c..ddae996dba 100644 --- a/proposals/0027-string-from-code-units.md +++ b/proposals/0027-string-from-code-units.md @@ -4,13 +4,13 @@ * Author: [Zachary Waldowski](https://github.com/zwaldowski) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-February/000044.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0027-expose-code-unit-initializers-on-string/1529) ## Introduction Going back and forth from Strings to their byte representations is an important part of solving many problems, including object serialization, binary and text file formats, wire/network interfaces, and cryptography. Swift has such utilities, but currently only exposed through `String.Type.fromCString(_:)` and `String.Type.fromCStringRepairingIllFormedUTF8(_:)`. -See swift-evolution [thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005951.html) and [draft proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006295.html). +See swift-evolution [thread](https://forums.swift.org/t/faster-lower-level-external-string-initialization/974) and [draft proposal](https://forums.swift.org/t/faster-lower-level-external-string-initialization/974/4). ## Motivation diff --git a/proposals/0028-modernizing-debug-identifiers.md b/proposals/0028-modernizing-debug-identifiers.md index e21d5f88b7..9e51fb46f9 100644 --- a/proposals/0028-modernizing-debug-identifiers.md +++ b/proposals/0028-modernizing-debug-identifiers.md @@ -1,17 +1,17 @@ # Modernizing Swift's Debugging Identifiers * Proposal: [SE-0028](0028-modernizing-debug-identifiers.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Implemented (Swift 2.2)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-February/000030.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0028-modernizing-swifts-debugging-identifiers-line-etc/1303) * Bug: [SR-669](https://bugs.swift.org/browse/SR-669) ## Introduction This proposal aims to eliminate Swift's use of "[screaming snake case](https://en.wikipedia.org/wiki/Snake_case)" like `__FILE__` and `__FUNCTION__` and replacing identifier instances with common [octothorpe-prefixed](https://en.wiktionary.org/wiki/octothorpe) lowercase `#identifier` representations. -*The Swift-Evolution discussion of this topic took place in the "[Review] SE-0022: Referencing the Objective-C selector of a method" thread and then in its own "[\[Proposal\] Eliminating Swift's Screaming Snake Case Identifiers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007347.html)" thread* +*The Swift-Evolution discussion of this topic took place in the "[Review] SE-0022: Referencing the Objective-C selector of a method" thread and then in its own "[\[Proposal\] Eliminating Swift's Screaming Snake Case Identifiers](https://forums.swift.org/t/proposal-eliminating-swifts-screaming-snake-case-identifiers/1165)" thread* ## Motivation diff --git a/proposals/0029-remove-implicit-tuple-splat.md b/proposals/0029-remove-implicit-tuple-splat.md index bd0c5fcc0f..e16653aef4 100644 --- a/proposals/0029-remove-implicit-tuple-splat.md +++ b/proposals/0029-remove-implicit-tuple-splat.md @@ -1,11 +1,11 @@ # Remove implicit tuple splat behavior from function applications * Proposal: [SE-0029](0029-remove-implicit-tuple-splat.md) -* Author: [Chris Lattner](http://github.com/lattner) -* Review Manager: [Joe Groff](http://github.com/jckarter) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-February/000033.html) -* Commit: [apple/swift@8e12008](https://github.com/apple/swift/commit/8e12008d2b34a605f8766310f53d5668f3d50955) +* Author: [Chris Lattner](https://github.com/lattner) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0029-remove-implicit-tuple-splat-behavior-from-function-applications/1380) +* Implementation: [apple/swift@8e12008](https://github.com/apple/swift/commit/8e12008d2b34a605f8766310f53d5668f3d50955) ## Introduction @@ -30,7 +30,7 @@ foo(x) This proposal recommends removing the later form, which I affectionately refer to as the "tuple splat" form. This feature is purely a sugar feature, it does not provide any expressive ability beyond passing the parameters manually. -Swift-evolution thread: [Proposal: Remove implicit tuple splat behavior from function applications](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160125/007856.html) +Swift-evolution thread: [Proposal: Remove implicit tuple splat behavior from function applications](https://forums.swift.org/t/proposal-remove-implicit-tuple-splat-behavior-from-function-applications/1201) ## Motivation diff --git a/proposals/0030-property-behavior-decls.md b/proposals/0030-property-behavior-decls.md index 781cd2f8d8..2a48404d78 100644 --- a/proposals/0030-property-behavior-decls.md +++ b/proposals/0030-property-behavior-decls.md @@ -3,8 +3,9 @@ * Proposal: [SE-0030](0030-property-behavior-decls.md) * Author: [Joe Groff](https://github.com/jckarter) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Deferred** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-February/000047.html) +* Status: **Withdrawn** +* Superseded by: [SE-0258](0258-property-wrappers.md) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0030-property-behaviors/1546) ## Introduction @@ -13,8 +14,8 @@ Rather than hardcode a fixed set of patterns into the compiler, we should provide a general "property behavior" mechanism to allow these patterns to be defined as libraries. -[Swift Evolution Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/003148.html)
-[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160208/009603.html) +[Swift Evolution Discussion](https://forums.swift.org/t/proposal-property-behaviors/594)
+[Review](https://forums.swift.org/t/review-se-0030-property-behaviors/1385) ## Motivation diff --git a/proposals/0031-adjusting-inout-declarations.md b/proposals/0031-adjusting-inout-declarations.md index 8aa74dbda5..a49eadb4eb 100644 --- a/proposals/0031-adjusting-inout-declarations.md +++ b/proposals/0031-adjusting-inout-declarations.md @@ -1,19 +1,20 @@ # Adjusting `inout` Declarations for Type Decoration * Proposal: [SE-0031](0031-adjusting-inout-declarations.md) -* Authors: [Joe Groff](https://github.com/jckarter), [Erica Sadun](http://github.com/erica) +* Authors: [Joe Groff](https://github.com/jckarter), [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010571.html) -* Pull Request: [apple/swift#1333](https://github.com/apple/swift/pull/1333) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0031-adjusting-inout-declarations-for-type-decoration/1478) +* Implementation: [apple/swift#1333](https://github.com/apple/swift/pull/1333) ## Introduction The `inout` keyword indicates copy-in/copy-out argument behavior. In its current implementation the keyword prepends argument names. We propose to move the `inout` keyword to the right side of the colon to decorate the type instead of the parameter label. -*The initial Swift-Evolution discussion of this topic took place in the "[Replace 'inout' with &](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005511.html)" thread.* +*The initial Swift-Evolution discussion of this topic took place in the "[Replace 'inout' with &](https://forums.swift.org/t/pitch-replace-inout-with/652/29)" thread.* + +[Thread to Proposal](https://forums.swift.org/t/proposal-adjusting-inout-declarations-for-type-decoration/1239), [Review](https://forums.swift.org/t/review-se-0031-adjusting-inout-declarations-for-type-decoration/1399) -[Thread to Proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160125/008264.html), [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160208/009793.html) ## Motivation In Swift 2, the `inout` parameter lives on the label side rather than the type side of the colon diff --git a/proposals/0032-sequencetype-find.md b/proposals/0032-sequencetype-find.md index eadeee7f55..7dd6b27397 100644 --- a/proposals/0032-sequencetype-find.md +++ b/proposals/0032-sequencetype-find.md @@ -1,12 +1,12 @@ # Add `first(where:)` method to `Sequence` * Proposal: [SE-0032](0032-sequencetype-find.md) -* Author: [Kevin Ballard](https://github.com/kballard) +* Author: [Lily Ballard](https://github.com/lilyball) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000134.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0032-add-find-method-to-sequence/2462) * Bug: [SR-1519](https://bugs.swift.org/browse/SR-1519) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0032-sequencetype-find.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0032-sequencetype-find.md) ## Introduction @@ -15,9 +15,9 @@ found element. Discussion on swift-evolution started with a proposal with title **Add find method to SequenceType** -Swift-evolution thread: [Proposal: Add function SequenceType.find()](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151228/004814.html) +Swift-evolution thread: [Proposal: Add function SequenceType.find()](https://forums.swift.org/t/proposal-add-function-sequencetype-find/825) -[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/016035.html) +[Review](https://forums.swift.org/t/review-se-0032-add-find-method-to-sequencetype/2381) ## Motivation diff --git a/proposals/0033-import-objc-constants.md b/proposals/0033-import-objc-constants.md index 71e4bf0ce4..06e12fcb71 100644 --- a/proposals/0033-import-objc-constants.md +++ b/proposals/0033-import-objc-constants.md @@ -3,16 +3,16 @@ * Proposal: [SE-0033](0033-import-objc-constants.md) * Author: [Jeff Kelley](https://github.com/SlaunchaMan) * Review Manager: [John McCall](https://github.com/rjmccall) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000097.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0033-import-objective-c-constants-as-swift-types/1706) ## Introduction Given a list of constants in an Objective-C file, add an attribute that will enable Swift to import them as either an Enum or a Struct, using `RawRepresentable` to convert to their original type. This way, instead of passing strings around for APIs, we can use more type-safe objects and take advantage of Swift’s code completion, as well as making our Swift (and Objective-C!) code more readable and more approachable to beginners. -Swift-evolution thread: [Original E-Mail](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006893.html), [Replies](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/006904.html) +[Swift-evolution thread](https://forums.swift.org/t/pitch-import-objective-c-constants-as-enums/1114) -[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010625.html) +[Review](https://forums.swift.org/t/review-import-objective-c-constants-as-swift-types/1486) ## Motivation @@ -49,7 +49,7 @@ Sometimes, as with `NSError` domains, it’s OK to use your own values when ther ## Proposed solution -As [suggested by Doug Gregor on swift-evolution](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/006904.html), combining a `typedef` with a new attribute could clean this up while still keeping everything as the original type internally. Here are examples using both structs and enums: +As [suggested by Doug Gregor on swift-evolution](https://forums.swift.org/t/pitch-import-objective-c-constants-as-enums/1114/2), combining a `typedef` with a new attribute could clean this up while still keeping everything as the original type internally. Here are examples using both structs and enums: ### Enums diff --git a/proposals/0034-disambiguating-line.md b/proposals/0034-disambiguating-line.md index cab8a53630..61b51ffe47 100644 --- a/proposals/0034-disambiguating-line.md +++ b/proposals/0034-disambiguating-line.md @@ -1,10 +1,10 @@ # Disambiguating Line Control Statements from Debugging Identifiers * Proposal: [SE-0034](0034-disambiguating-line.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011337.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0034-disambiguating-line-control-statements-from-debugging-identifiers/1614) * Bug: [SR-840](https://bugs.swift.org/browse/SR-840) @@ -13,11 +13,11 @@ In being accepted, Swift Evolution SE-0028 (0028-modernizing-debug-identifiers.md) overloads the use of `#line` to mean both an identifier that maps to a calling site's line number within a file and acts as part of a line control statement. This proposal nominates `#setline` to replace `#line` for file and line syntactic source control. -The discussion took place on-line in the [*\[Discussion\]: Renaming #line, the line control statement*](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160208/009390.html) thread. +The discussion took place on-line in the [*\[Discussion\]: Renaming #line, the line control statement*](https://forums.swift.org/t/discussion-renaming-line-the-line-control-statement/1307/24) thread. -[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010563.html) +[Review](https://forums.swift.org/t/review-se-0034-disambiguating-line-control-statements-from-debugging-identifiers/1477) -[Revision](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012432.html) +[Revision](https://forums.swift.org/t/accepted-se-0034-disambiguating-line-control-statements-from-debugging-identifiers/1614/3) ## Motivation @@ -45,7 +45,7 @@ file-name → static-string-literal­ ## Alternatives considered -A more flexible grammar was suggested, however, as Kevin Ballard pointed out: +A more flexible grammar was suggested, however, as Lily Ballard pointed out: > This feature isn't something end users are going to use. And it's not something that will ever reasonably apply to anything except `#file` and `#line`. This feature is only ever intended to be used by tools that auto-generate source files. The most important concerns here really should just be that whatever we use is trivial to generate correctly by even the simplest of tools and is readable. And since this won't ever apply to anything beyond `#file` and `#line`, there's no need to try to generalize this feature at all. diff --git a/proposals/0035-limit-inout-capture.md b/proposals/0035-limit-inout-capture.md index fb0c91a950..4a44fd09ff 100644 --- a/proposals/0035-limit-inout-capture.md +++ b/proposals/0035-limit-inout-capture.md @@ -3,8 +3,8 @@ * Proposal: [SE-0035](0035-limit-inout-capture.md) * Author: [Joe Groff](https://github.com/jckarter) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-February/000046.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0035-limiting-inout-capture-to-noescape-contexts/1544) * Bug: [SR-807](https://bugs.swift.org/browse/SR-807) @@ -13,9 +13,9 @@ Swift's behavior when closures capture `inout` parameters and escape their enclosing context is a common source of confusion. We should disallow implicit capture of `inout` parameters except in `@noescape` closures. -Swift-evolution thread: [only allow capture of inout parameters in @noescape closures](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160125/008074.html) +Swift-evolution thread: [only allow capture of inout parameters in @noescape closures](https://forums.swift.org/t/pitch-only-allow-capture-of-inout-parameters-in-noescape-closures/1223) -[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010465.html) +[Review](https://forums.swift.org/t/review-se-0035-limiting-inout-capture-to-noescape-contexts/1461) ## Motivation @@ -63,7 +63,7 @@ one in a long line of complaints on the topic. ## Proposed solution -I propose we make it so that implicitly capturing an `inout` parameter into a escapable +I propose we make it so that implicitly capturing an `inout` parameter into an escapable closure is an error. We added the explicit `@noescape` annotation in Swift 1.2, and have since adopted it throughout the standard library where appropriate, so the compromise has outlived its usefulness and become a source of confusion. diff --git a/proposals/0036-enum-dot.md b/proposals/0036-enum-dot.md index 9fc36c59d4..c9cd52fbcd 100644 --- a/proposals/0036-enum-dot.md +++ b/proposals/0036-enum-dot.md @@ -1,10 +1,10 @@ # Requiring Leading Dot Prefixes for Enum Instance Member Implementations * Proposal: [SE-0036](0036-enum-dot.md) -* Authors: [Erica Sadun](http://github.com/erica), [Chris Lattner](https://github.com/lattner) +* Authors: [Erica Sadun](https://github.com/erica), [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000100.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0036-requiring-leading-dot-prefixes-for-enum-instance-member-implementations/2196) * Bug: [SR-1236](https://bugs.swift.org/browse/SR-1236) @@ -17,10 +17,10 @@ This makes little sense. In no other case can an instance implementation directl This proposal introduces a rule that requires leading dots or fully qualified references (EnumType.caseMember) to provide a more consistent developer experience to clearly disambiguate static cases from instance members. -*Discussion took place on the Swift Evolution mailing list in the [\[Discussion\] Enum Leading Dot Prefixes](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160208/009861.html) thread. This proposal uses lowerCamelCase enumeration cases in compliance with +*Discussion took place on the Swift Evolution mailing list in the [\[Discussion\] Enum Leading Dot Prefixes](https://forums.swift.org/t/discussion-enum-leading-dot-prefixes/1404) thread. This proposal uses lowerCamelCase enumeration cases in compliance with current [API Guideline Working Group guidance](https://swift.org/documentation/api-design-guidelines/).* -[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/013956.html) +[Review](https://forums.swift.org/t/review-se-0036-requiring-leading-dot-prefixes-for-enum-instance-member-implementations/2020) ## Motivation diff --git a/proposals/0037-clarify-comments-and-operators.md b/proposals/0037-clarify-comments-and-operators.md index 36ce6bc37b..9866d04c29 100644 --- a/proposals/0037-clarify-comments-and-operators.md +++ b/proposals/0037-clarify-comments-and-operators.md @@ -2,9 +2,9 @@ * Proposal: [SE-0037](0037-clarify-comments-and-operators.md) * Author: [Jesse Rusak](https://github.com/jder) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000066.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0037-clarify-interaction-between-comments-operators/1833) * Bug: [SR-960](https://bugs.swift.org/browse/SR-960) @@ -17,11 +17,11 @@ whether they are to the left or right of an operator, and the contents of the comment itself. This proposal suggests a uniform set of rules for how these cases should be parsed. -Swift-evolution thread: [started here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/006030.html) +Swift-evolution thread: [started here](https://forums.swift.org/t/draft-clarify-interaction-between-comments-operators/984) A draft implementation is [available here](https://github.com/apple/swift/compare/master...jder:comment-operator-fixes). -[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012398.html) +[Review](https://forums.swift.org/t/review-se-0037-clarify-interaction-between-comments-operators/1762) ## Motivation diff --git a/proposals/0038-swiftpm-c-language-targets.md b/proposals/0038-swiftpm-c-language-targets.md index 53d377148a..d86482524e 100644 --- a/proposals/0038-swiftpm-c-language-targets.md +++ b/proposals/0038-swiftpm-c-language-targets.md @@ -3,8 +3,8 @@ * Proposal: [SE-0038](0038-swiftpm-c-language-targets.md) * Author: [Daniel Dunbar](https://github.com/ddunbar) * Review Manager: [Rick Ballard](https://github.com/rballard) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011097.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0038-package-manager-c-language-target-support/1569) * Bug: [SR-821](https://bugs.swift.org/browse/SR-821) @@ -16,7 +16,7 @@ languages). This proposal is limited in scope to only supporting targets consisting entirely of C languages; there is no provision for supporting targets which include both C and Swift sources. -[Swift Evolution Review Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010470.html) +[Swift Evolution Review Thread](https://forums.swift.org/t/review-se-0038-package-manager-c-language-target-support/1462) ## Motivation @@ -111,7 +111,7 @@ addressed with this proposal: 2. No provision is made in this proposal for controlling compiler arguments. We will support the existing debug and release configurations using a fixed set - of compiler flags. We expect future proposals to accomodate the need to + of compiler flags. We expect future proposals to accommodate the need to modify those flags. 3. We intend for the feature to be built in such a way as to support any diff --git a/proposals/0039-playgroundliterals.md b/proposals/0039-playgroundliterals.md index 611eadbf2f..c03981cf64 100644 --- a/proposals/0039-playgroundliterals.md +++ b/proposals/0039-playgroundliterals.md @@ -1,10 +1,10 @@ # Modernizing Playground Literals * Proposal: [SE-0039](0039-playgroundliterals.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000060.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0039-modernizing-playground-literals/1746) * Bug: [SR-917](https://bugs.swift.org/browse/SR-917) ## Introduction @@ -15,9 +15,9 @@ These literals are built using a simple square bracket syntax that, in the curre conflicts with collection literals. This proposal redesigns playground literals to follow the precedent of #available and #selector. -*Discussion took place on the Swift Evolution mailing list in the [\[Discussion\] Modernizing Playground Literals](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010301.html) thread. Thanks to [Chris Lattner](https://github.com/lattner) for suggesting this enhancement.* +*Discussion took place on the Swift Evolution mailing list in the [\[Discussion\] Modernizing Playground Literals](https://forums.swift.org/t/discussion-modernizing-playground-literals/1443) thread. Thanks to [Chris Lattner](https://github.com/lattner) for suggesting this enhancement.* -[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012025.html) +[Review](https://forums.swift.org/t/review-se-0039-modernizing-playground-literals/1707) ## Motivation diff --git a/proposals/0040-attributecolons.md b/proposals/0040-attributecolons.md index 010637bf06..17c49bbd26 100644 --- a/proposals/0040-attributecolons.md +++ b/proposals/0040-attributecolons.md @@ -1,11 +1,11 @@ # Replacing Equal Signs with Colons For Attribute Arguments * Proposal: [SE-0040](0040-attributecolons.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012100.html) -* Pull Request: [apple/swift#1537](https://github.com/apple/swift/pull/1537) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0040-replacing-equal-signs-with-colons-for-attribute-arguments/1719) +* Implementation: [apple/swift#1537](https://github.com/apple/swift/pull/1537) ## Introduction @@ -13,7 +13,7 @@ Attribute arguments are unlike other Swift language arguments. At the call site, to distinguish argument names from passed values. This proposal brings attributes into compliance with Swift standard practices by replacing the use of "=" with ":" in this one-off case. -*Discussion took place on the Swift Evolution mailing list in the [\[Discussion\] Replacing Equal Signs with Colons For Attribute Arguments](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010448.html) thread. Thanks to [Doug Gregor](https://github.com/DougGregor) for suggesting this enhancement.* +*Discussion took place on the Swift Evolution mailing list in the [\[Discussion\] Replacing Equal Signs with Colons For Attribute Arguments](https://forums.swift.org/t/discussion-replacing-equal-signs-with-colons-for-attribute-arguments/1459) thread. Thanks to [Doug Gregor](https://github.com/DougGregor) for suggesting this enhancement.* ## Motivation diff --git a/proposals/0041-conversion-protocol-conventions.md b/proposals/0041-conversion-protocol-conventions.md index f41ea52888..9b61c3cc3b 100644 --- a/proposals/0041-conversion-protocol-conventions.md +++ b/proposals/0041-conversion-protocol-conventions.md @@ -1,16 +1,16 @@ # Updating Protocol Naming Conventions for Conversions * Proposal: [SE-0041](0041-conversion-protocol-conventions.md) -* Authors: [Matthew Johnson](https://github.com/anandabits), [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Authors: [Matthew Johnson](https://github.com/anandabits), [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000160.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0041-updating-protocol-naming-conventions-for-conversions/2684) ## Introduction We propose to expand and improve the naming conventions established by the API Guidelines and the standard library with regard to conversion related protocols. We believe common protocol naming patterns should be clear, consistent, and meaningful. The Swift standard library includes slightly north of eighty protocols. Of these, about 15% concern themselves with type initialization and conversion. This proposal assigns specific conventional suffixes to these tasks. We present this proposal to improve overall language coherence. -*The Swift-evolution thread about this topic can be found here: [Proposal: conversion protocol naming conventions](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/002052.html)* +*The Swift-evolution thread about this topic can be found here: [Proposal: conversion protocol naming conventions](https://forums.swift.org/t/proposal-conversion-protocol-naming-conventions/478)* ## Motivation @@ -151,7 +151,7 @@ particularly similar to one another. **Our Response** -The semantics cover "converting to a type", "converting from a type", and "converting to and from a type". We have examples from our own code and from third party code on github that suggest conversion tasks are common enough that standardizing API naming conventions will be valuable. +The semantics cover "converting to a type", "converting from a type", and "converting to and from a type". We have examples from our own code and from third party code on GitHub that suggest conversion tasks are common enough that standardizing API naming conventions will be valuable. ## Updated Approach diff --git a/proposals/0042-flatten-method-types.md b/proposals/0042-flatten-method-types.md index 3fb1350444..3cd6411664 100644 --- a/proposals/0042-flatten-method-types.md +++ b/proposals/0042-flatten-method-types.md @@ -3,8 +3,9 @@ * Proposal: [SE-0042](0042-flatten-method-types.md) * Author: [Joe Groff](https://github.com/jckarter) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Accepted** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013251.html) +* Status: **Rejected** +* Decision Notes: [Original acceptance](https://forums.swift.org/t/accepted-se-0042-flattening-the-function-type-of-unapplied-method-references/1926). The proposal was not implemented in time for Swift 3, and is now source-breaking. + * Bug: [SR-1051](https://bugs.swift.org/browse/SR-1051) ## Introduction @@ -35,7 +36,7 @@ let f = Type.instanceMethod // f: (Type, y: Int) -> Int f(Type(x: 1), y: 2) // ==> 3 ``` -Swift-evolution thread: [Flattening the function type of unapplied instance methods](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/010843.html) +Swift-evolution thread: [Flattening the function type of unapplied instance methods](https://forums.swift.org/t/pitch-flattening-the-function-type-of-unapplied-instance-methods/1523) ## Motivation diff --git a/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md b/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md index e829ceb59b..abc9390c74 100644 --- a/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md +++ b/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md @@ -3,9 +3,9 @@ * Proposal: [SE-0043](0043-declare-variables-in-case-labels-with-multiple-patterns.md) * Author: [Andrew Bennett](https://github.com/therealbnut) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013250.html) -* Pull Request: [apple/swift#1383](https://github.com/apple/swift/pull/1383) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0043-declare-variables-in-case-labels-with-multiple-patterns/1925) +* Implementation: [apple/swift#1383](https://github.com/apple/swift/pull/1383) ## Introduction @@ -33,7 +33,7 @@ The error is: This proposal aims to remove this error when each pattern declares the same variables with the same types. -Swift-evolution thread: [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007431.html) +Swift-evolution thread: [here](https://forums.swift.org/t/draft-proposal-declare-variables-in-case-labels-with-multiple-patterns/1173) ## Motivation diff --git a/proposals/0044-import-as-member.md b/proposals/0044-import-as-member.md index 0131f05dc1..a978919016 100644 --- a/proposals/0044-import-as-member.md +++ b/proposals/0044-import-as-member.md @@ -2,9 +2,9 @@ * Proposal: [SE-0044](0044-import-as-member.md) * Author: [Michael Ilseman](https://github.com/milseman) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013265.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0044-import-as-member/1929) * Bug: [SR-1053](https://bugs.swift.org/browse/SR-1053) ## Introduction @@ -16,8 +16,9 @@ authors to specify the capability of importing functions and variables as members on imported Swift types. It also seeks to provide an automatic inference option for APIs that follow a consistent, disciplined naming convention. -[Swift-evolution thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160229/011617.html)
-[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160314/012695.html) +[Swift-evolution thread](https://forums.swift.org/t/proposal-draft-import-as-member/1650)
+[Review](https://forums.swift.org/t/review-se-0044-import-as-member/1805) + ## Motivation C APIs and frameworks currently import into Swift as global functions and global diff --git a/proposals/0045-scan-takewhile-dropwhile.md b/proposals/0045-scan-takewhile-dropwhile.md index b88f37e2cb..c4f3a751c0 100644 --- a/proposals/0045-scan-takewhile-dropwhile.md +++ b/proposals/0045-scan-takewhile-dropwhile.md @@ -1,12 +1,12 @@ # Add prefix(while:) and drop(while:) to the stdlib * Proposal: [SE-0045](0045-scan-takewhile-dropwhile.md) -* Author: [Kevin Ballard](https://github.com/kballard) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Author: [Lily Ballard](https://github.com/lilyball) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Implemented (Swift 3.1)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000136.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0045-add-scan-prefix-while-drop-while-and-unfold-to-the-stdlib/2466) * Bug: [SR-1516](https://bugs.swift.org/browse/SR-1516) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/b39d653f7e3d5e982b562664343f26c826652291/proposals/0045-scan-takewhile-dropwhile.md), [2](https://github.com/apple/swift-evolution/blob/baec22a8a5ddaa0407086380da32b5cad2144800/proposals/0045-scan-takewhile-dropwhile.md), [3](https://github.com/apple/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0045-scan-takewhile-dropwhile.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/b39d653f7e3d5e982b562664343f26c826652291/proposals/0045-scan-takewhile-dropwhile.md), [2](https://github.com/swiftlang/swift-evolution/blob/baec22a8a5ddaa0407086380da32b5cad2144800/proposals/0045-scan-takewhile-dropwhile.md), [3](https://github.com/swiftlang/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0045-scan-takewhile-dropwhile.md) ## Introduction @@ -15,9 +15,9 @@ overrides as appropriate on `Collection`, `LazySequenceProtocol`, and `LazyCollectionProtocol`. Swift-evolution thread: -[Proposal: Add scan, takeWhile, dropWhile, and iterate to the stdlib](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160229/011923.html) +[Proposal: Add scan, takeWhile, dropWhile, and iterate to the stdlib](https://forums.swift.org/t/proposal-add-scan-takewhile-dropwhile-and-iterate-to-the-stdlib/806) -[Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/016036.html) +[Review](https://forums.swift.org/t/review-se-0045-add-scan-prefix-while-drop-while-and-iterate-to-the-stdlib/2382) ## Motivation @@ -34,12 +34,10 @@ protocol Sequence { // ... /// Returns a subsequence by skipping elements while `predicate` returns /// `true` and returning the remainder. - @warn_unused_result - func drop(@noescape while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> Self.SubSequence + func drop(while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> Self.SubSequence /// Returns a subsequence containing the initial elements until `predicate` /// returns `false` and skipping the remainder. - @warn_unused_result - func prefix(@noescape while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> Self.SubSequence + func prefix(while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> Self.SubSequence } ``` @@ -63,10 +61,8 @@ extension Sequence where SubSequence.Iterator.Element == Iterator.Element, SubSequence.SubSequence == SubSequence { - @warn_unused_result - public func drop(@noescape while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> AnySequence - @warn_unused_result - public func prefix(@noescape while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> AnySequence + public func drop(while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> AnySequence + public func prefix(while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> AnySequence } ``` @@ -79,8 +75,8 @@ Provide default implementations on `Collection` as well: ```swift extension Collection { - func drop(@noescape while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> Self.SubSequence - func prefix(@noescape while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> Self.SubSequence + func drop(while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> Self.SubSequence + func prefix(while predicate: (Self.Iterator.Element) throws -> Bool) rethrows -> Self.SubSequence } ``` @@ -154,4 +150,4 @@ and `unfold(_:applying:)` (see [revision 3][rev-3]). This proposal was partially accepted, with `scan(_:combine:)` rejected on grounds of low utility and `unfold(_:applying:)` rejected on grounds of poor naming (see [rationale][]). -[rationale]: +[rationale]: https://forums.swift.org/t/accepted-with-modifications-se-0045-add-scan-prefix-while-drop-while-and-unfold-to-the-stdlib/2466 diff --git a/proposals/0046-first-label.md b/proposals/0046-first-label.md index a3d9f9f221..6b401da8aa 100644 --- a/proposals/0046-first-label.md +++ b/proposals/0046-first-label.md @@ -1,10 +1,10 @@ # Establish consistent label behavior across all parameters including first labels * Proposal: [SE-0046](0046-first-label.md) -* Authors: [Jake Carter](https://github.com/JakeCarter), [Erica Sadun](http://github.com/erica) +* Authors: [Jake Carter](https://github.com/JakeCarter), [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000067.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0046-establish-consistent-label-behavior-across-all-parameters-including-first-labels/1834) * Bug: [SR-961](https://bugs.swift.org/browse/SR-961) ## Introduction @@ -17,7 +17,7 @@ declaration throughout the Swift programming language and bring method and function declarations in-sync with initializers, which already use this standard. -*Discussion took place on the Swift Evolution mailing list in the [Make the first parameter in a function declaration follow the same rules as the others](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012209.html) thread.* +*Discussion took place on the Swift Evolution mailing list in the [Make the first parameter in a function declaration follow the same rules as the others](https://forums.swift.org/t/pitch-make-the-first-parameter-in-a-function-declaration-follow-the-same-rules-as-the-others/1734) thread.* ## Motivation In the current state of the art, Swift 2 methods and functions combine local and external names to diff --git a/proposals/0047-nonvoid-warn.md b/proposals/0047-nonvoid-warn.md index 404b38d0d2..ab940ba456 100644 --- a/proposals/0047-nonvoid-warn.md +++ b/proposals/0047-nonvoid-warn.md @@ -1,10 +1,10 @@ # Defaulting non-Void functions so they warn on unused results * Proposal: [SE-0047](0047-nonvoid-warn.md) -* Authors: [Erica Sadun](http://github.com/erica), [Adrian Kashivskyy](https://github.com/akashivskyy) +* Authors: [Erica Sadun](https://github.com/erica), [Adrian Kashivskyy](https://github.com/akashivskyy) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000075.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0047-defaulting-non-void-functions-so-they-warn-on-unused-results/1927) * Bug: [SR-1052](https://bugs.swift.org/browse/SR-1052) @@ -20,7 +20,7 @@ public func sort() -> [Self.Generator.Element] This proposal flips this default behavior. Unused results are more likely to indicate programmer error than confusion between mutating and non-mutating function pairs. This proposal makes "warn on unused result" the *default* behavior for Swift methods and functions. Developers must override this default to enable the compiler to ignore unconsumed values. -This proposal was discussed on-list in a variety of threads, most recently [Make non-void functions @warn_unused_result by default](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/010926.html). +This proposal was discussed on-list in a variety of threads, most recently [Make non-void functions @warn_unused_result by default](https://forums.swift.org/t/make-non-void-functions-warn-unused-result-by-default/1553). #### Acceptance Notes @@ -145,5 +145,5 @@ Upon acceptance, this proposal removes two of the last remaining instances of sn ## Acknowledgements -Changing the behavior of non-void functions to use default warnings for unused results was initially introduced by Adrian Kashivskyy. Additional thanks go out to Chris Lattner, Gwendal Roué, Dmitri Gribenko, Jeff Kelley, David Owens, Jed Lewison, Stephen Cellis, Ankit Aggarwal, Paul Ossenbruggen,Brent Royal-Gordon, Tino Heth, Haravikk, Félix Cloutier,Yuta Koshizawa, +Changing the behavior of non-void functions to use default warnings for unused results was initially introduced by Adrian Kashivskyy. Additional thanks go out to Chris Lattner, Gwendal Roué, Dmitri Gribenko, Jeff Kelley, David Owens, Jed Lewison, Stephen Cellis, Ankit Aggarwal, Paul Ossenbruggen, Becca Royal-Gordon, Tino Heth, Haravikk, Félix Cloutier,Yuta Koshizawa, for their feedback on this topic. diff --git a/proposals/0048-generic-typealias.md b/proposals/0048-generic-typealias.md index 774dc58c22..4f6efecb5a 100644 --- a/proposals/0048-generic-typealias.md +++ b/proposals/0048-generic-typealias.md @@ -3,15 +3,15 @@ * Proposal: [SE-0048](0048-generic-typealias.md) * Author: [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000098.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0048-generic-type-aliases/2192) ## Introduction This proposal aims to add generic typealiases to Swift. -Swift-evolution thread: [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012289.html) +Swift-evolution thread: [here](https://forums.swift.org/t/proposal-generic-type-aliases/1740) ## Motivation @@ -51,7 +51,7 @@ typealias DictionaryOfStrings = Dictionary // error: type 'T' does not conform to protocol 'Hashable' ``` -However, because this proposal is targetted at supporting aliases, it does not +However, because this proposal is targeted at supporting aliases, it does not allow *additional* constraints to be added to type parameters. For example, you can't write: diff --git a/proposals/0049-noescape-autoclosure-type-attrs.md b/proposals/0049-noescape-autoclosure-type-attrs.md index 020f44b3b1..a6572f2c82 100644 --- a/proposals/0049-noescape-autoclosure-type-attrs.md +++ b/proposals/0049-noescape-autoclosure-type-attrs.md @@ -3,8 +3,8 @@ * Proposal: [SE-0049](0049-noescape-autoclosure-type-attrs.md) * Author: [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000099.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0049-move-noescape-and-autoclosure-to-be-type-attributes/2194) * Bug: [SR-1235](https://bugs.swift.org/browse/SR-1235) ## Introduction @@ -15,7 +15,7 @@ attributes. This improves consistency and reduces redundancy within the language, e.g. aligning with [SE-0031](0031-adjusting-inout-declarations.md), which moved `inout`, making declaration and type syntax more consistent. -Swift-evolution thread: [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012292.html) +Swift-evolution thread: [here](https://forums.swift.org/t/proposal-move-noescape-and-autoclosure-to-type-attributes/1741) ## Motivation diff --git a/proposals/0050-floating-point-stride.md b/proposals/0050-floating-point-stride.md index 5aa7f67c32..63031b0f67 100644 --- a/proposals/0050-floating-point-stride.md +++ b/proposals/0050-floating-point-stride.md @@ -1,14 +1,14 @@ # Decoupling Floating Point Strides from Generic Implementations * Proposal: [SE-0050](0050-floating-point-stride.md) -* Authors: [Erica Sadun](http://github.com/erica), [Xiaodi Wu](http://github.com/xwu) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Authors: [Erica Sadun](https://github.com/erica), [Xiaodi Wu](https://github.com/xwu) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Withdrawn** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000178.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/returned-for-revision-se-0050-decoupling-floating-point-strides-from-generic-implementations/2823) Swift strides create progressions along "notionally continuous one-dimensional values" using a series of offset values. This proposal supplements Swift's generic stride implementation with separate algorithms for floating point strides that avoid error accumulation. -This proposal was discussed on-list in the ["\[Discussion\] stride behavior and a little bit of a call-back to digital numbers"](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011194.html) thread. +This proposal was discussed on-list in the ["\[Discussion\] stride behavior and a little bit of a call-back to digital numbers"](https://forums.swift.org/t/discussion-stride-behavior-and-a-little-bit-of-a-call-back-to-digital-numbers/1597) thread. ## Motivation diff --git a/proposals/0051-stride-semantics.md b/proposals/0051-stride-semantics.md index 917f5f582e..aabe1b2cad 100644 --- a/proposals/0051-stride-semantics.md +++ b/proposals/0051-stride-semantics.md @@ -1,13 +1,13 @@ # Conventionalizing `stride` semantics * Proposal: [SE-0051](0051-stride-semantics.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: N/A * Status: **Withdrawn** Swift offers two stride functions, `stride(to:, by:)` and `stride(through:, by:)`. This proposal introduces a third style and renames the existing `to` and `through` styles. -This proposal was discussed on-list in the ["\[Discussion\] stride behavior and a little bit of a call-back to digital numbers"](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011194.html) thread. +This proposal was discussed on-list in the ["\[Discussion\] stride behavior and a little bit of a call-back to digital numbers"](https://forums.swift.org/t/discussion-stride-behavior-and-a-little-bit-of-a-call-back-to-digital-numbers/1597) thread. ## Motivation diff --git a/proposals/0052-iterator-post-nil-guarantee.md b/proposals/0052-iterator-post-nil-guarantee.md index 1be29dce17..7b9c287fe7 100644 --- a/proposals/0052-iterator-post-nil-guarantee.md +++ b/proposals/0052-iterator-post-nil-guarantee.md @@ -3,9 +3,9 @@ * Proposal: [SE-0052](0052-iterator-post-nil-guarantee.md) * Author: [Patrick Pijnappel](https://github.com/PatrickPijnappel) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000135.html) -* Pull Request: [apple/swift#1702](https://github.com/apple/swift/pull/1702) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0052-change-iteratortype-post-nil-guarantee/2463) +* Implementation: [apple/swift#1702](https://github.com/apple/swift/pull/1702) ## Introduction @@ -19,7 +19,7 @@ are likely unaware of the precondition, expecting all iterators to return code will usually run fine, until someone does in fact pass in an iterator not repeating `nil` (it's a silent corner case). -Swift-evolution thread: [\[Proposal\] Change guarantee for GeneratorType.next() to always return nil past end](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160229/011699.html) +Swift-evolution thread: [\[Proposal\] Change guarantee for GeneratorType.next() to always return nil past end](https://forums.swift.org/t/proposal-change-guarantee-for-generatortype-next-to-always-return-nil-past-end/1673) ## Motivation diff --git a/proposals/0053-remove-let-from-function-parameters.md b/proposals/0053-remove-let-from-function-parameters.md index 6f7820c6c7..c3b22b4907 100644 --- a/proposals/0053-remove-let-from-function-parameters.md +++ b/proposals/0053-remove-let-from-function-parameters.md @@ -3,9 +3,9 @@ * Proposal: [SE-0053](0053-remove-let-from-function-parameters.md) * Author: [Nicholas Maccharoli](https://github.com/nirma) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000082.html) -* Pull Request: [apple/swift#1812](https://github.com/apple/swift/pull/1812) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0053-remove-explicit-use-of-let-from-function-parameters/1966) +* Implementation: [apple/swift#1812](https://github.com/apple/swift/pull/1812) ## Introduction @@ -15,7 +15,7 @@ Not allowing function parameters to be explicitly declared as `let` would permit Furthermore proposal [SE-0003​: "Removing `var` from Function Parameters"](0003-remove-var-parameters.md) removes `var` from function parameters removing any possible ambiguity as to whether a function parameter is immutable or not. -Swift-evolution thread: [Removing explicit use of `let` from Function Parameters](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160314/012851.html) +Swift-evolution thread: [Removing explicit use of `let` from Function Parameters](https://forums.swift.org/t/removing-explicit-use-of-let-from-function-parameters/1856) ## Motivation Now that proposal [SE-0003​: "Removing `var` from Function Parameters"](0003-remove-var-parameters.md) has been accepted, it would make sense that the syntax for function parameters being explicitly declared as `let` would be removed as well. diff --git a/proposals/0054-abolish-iuo.md b/proposals/0054-abolish-iuo.md index 05d87f0173..f8f12c9fe2 100644 --- a/proposals/0054-abolish-iuo.md +++ b/proposals/0054-abolish-iuo.md @@ -1,11 +1,11 @@ # Abolish `ImplicitlyUnwrappedOptional` type * Proposal: [SE-0054](0054-abolish-iuo.md) -* Author: [Chris Willmore](http://github.com/cwillmor) +* Author: [Chris Willmore](https://github.com/cwillmor) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000084.html) -* Pull Request: [apple/swift#2322](https://github.com/apple/swift/pull/2322) +* Status: **Implemented (Swift 4.2)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-pending-implementation-se-0054-abolish-implicitlyunwrappedoptional-type/2009) +* Implementation: [apple/swift#14299](https://github.com/apple/swift/pull/14299) ## Introduction @@ -15,7 +15,7 @@ Appending `!` to the type of a Swift declaration will give it optional type and annotate the declaration with an attribute stating that it may be implicitly unwrapped when used. -Swift-evolution thread: ["Abolish IUO Type"](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160314/012752.html) +Swift-evolution thread: ["Abolish IUO Type"](https://forums.swift.org/t/draft-abolish-iuo-type/1831) ## Motivation diff --git a/proposals/0055-optional-unsafe-pointers.md b/proposals/0055-optional-unsafe-pointers.md index 2d823d5c2d..6c69e20408 100644 --- a/proposals/0055-optional-unsafe-pointers.md +++ b/proposals/0055-optional-unsafe-pointers.md @@ -3,9 +3,9 @@ * Proposal: [SE-0055](0055-optional-unsafe-pointers.md) * Author: [Jordan Rose](https://github.com/jrose-apple) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-March/000086.html) -* Pull Request: [apple/swift#1878](https://github.com/apple/swift/pull/1878) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0055-make-unsafe-pointer-nullability-explicit-using-optional/2012) +* Implementation: [apple/swift#1878](https://github.com/apple/swift/pull/1878) ## Introduction @@ -21,7 +21,7 @@ We already have a way to describe this: Optionals. This proposal makes information about pointer nullability available in header files for imported C and Objective-C APIs. -swift-evolution thread: +swift-evolution thread: ## Motivation diff --git a/proposals/0056-trailing-closures-in-guard.md b/proposals/0056-trailing-closures-in-guard.md index 2e9c0cfcc3..ee77c952fe 100644 --- a/proposals/0056-trailing-closures-in-guard.md +++ b/proposals/0056-trailing-closures-in-guard.md @@ -4,11 +4,11 @@ * Author: [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000108.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0056-allow-trailing-closures-in-guard-conditions/2279) ## Introduction and Motivation -Swift-evolution thread: ["Allow trailing closures in 'guard' conditions"](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013141.html) +Swift-evolution thread: ["Allow trailing closures in 'guard' conditions"](https://forums.swift.org/t/proposal-allow-trailing-closures-in-guard-conditions/1909) The three statements `if`, `while`, and `guard` form a family that all take a rich form of conditions that can include one or more boolean conditions, @@ -58,7 +58,7 @@ guard let object = someSequence.findElement { $0.passesTest() } else { ## Detailed design The compiler change to do this is extremely straight-forward, the patch is -[available here](https://lists.swift.org/pipermail/swift-evolution/attachments/20160322/50c40166/attachment.obj). +[available here](https://forums.swift.org/uploads/short-url/pjcocZclSJ5owrodpA9tk4F1bmO.enc). ## Impact on existing code @@ -90,7 +90,7 @@ There are four primary alternatives: in the Swift 2 timeframe. I feel that it has stood the test of time well since then. - * *Change the syntax of `if` and `while`*: Brent Royal-Gordon points out that + * *Change the syntax of `if` and `while`*: Becca Royal-Gordon points out that we could change `if` and `while` to use a keyword after their condition as well, e.g.: diff --git a/proposals/0057-importing-objc-generics.md b/proposals/0057-importing-objc-generics.md index 4d816aa8b3..7faf19fa4b 100644 --- a/proposals/0057-importing-objc-generics.md +++ b/proposals/0057-importing-objc-generics.md @@ -3,8 +3,10 @@ * Proposal: [SE-0057](0057-importing-objc-generics.md) * Author: [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000097.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0057-importing-objective-c-lightweight-generics/2185) +* Previous Revision: [Originally Accepted Proposal](https://github.com/swiftlang/swift-evolution/blob/3abbed3edd12dd21061181993df7952665d660dd/proposals/0057-importing-objc-generics.md) + ## Introduction @@ -20,7 +22,7 @@ collections (`NSArray`, `NSDictionary`, `NSSet`) don't benefit in Swift. This proposal introduces a way to import the type parameters of Objective-C classes into Swift. -Swift-evolution thread: [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006790.html) +Swift-evolution thread: [here](https://forums.swift.org/t/proposal-draft-importing-objective-c-lightweight-generics/991) ## Motivation @@ -51,13 +53,15 @@ requirements on the generic type parameters in Swift: The following Objective-C code: - @interface MySet> : NSObject - -(MySet *)unionWithSet:(MySet *)otherSet; - @end +``` +@interface MySet> : NSObject +-(MySet *)unionWithSet:(MySet *)otherSet; +@end - @interface MySomething : NSObject - - (MySet *)valueSet; - @end +@interface MySomething : NSObject +- (MySet *)valueSet; +@end +``` will be imported as: @@ -72,6 +76,7 @@ class MySomething : NSObject { ``` ### Importing unspecialized types + When importing an unspecialized Objective-C type into Swift, we will substitute the bounds for the type arguments. For example: @@ -131,143 +136,11 @@ share a metaclass). This leads to several restrictions: } ``` -### Opting in to type argument discovery - -Some Objective-C parameterized classes do carry information about -their type arguments. When this is the case, it is possible to lift -some of the restrictions described in the above section. There are two -distinct cases: - -* *Abstract parameterized classes whose concrete subclasses are not - parameterized*: - [`NSLayoutAnchor`](https://developer.apple.com/library/mac/documentation/AppKit/Reference/NSLayoutAnchor_ClassReference/index.html#//apple_ref/occ/cl/NSLayoutAnchor) - is one such example: it is parameterized on the anchor type, but - there is a fixed set of such anchor types that are represented by - subclasses: `NSLayoutXAxisAnchor` subclasses - `NSLayoutAnchor`, `NSLayoutDimension` - subclasses `NSLayoutAnchor`, etc. Therefore, - the type arguments can be recovered by looking at the actual - metaclass. - -* *Parameterized classes that store their type arguments in - instances*: - [`GKComponentSystem`](https://developer.apple.com/library/ios/documentation/GameplayKit/Reference/GKComponentSystem_Class/) - is one such example: it is parameterized on the component type it - stores, but it's initializer (`-initWithComponentClass:`) requires - one to pass the component type's metaclass. Therefore, every - *instance* of `GKComponentSystem` knows its type arguments. - -A parameterized Objective-C class can opt in to providing information -about its type argument by implementing a method -`classForGenericArgumentAtIndex:` either as a class method (for the first case -described above) or as an instance method (for the second case -described above). The method returns the metaclass for the type -argument at the given, zero-based index. - -For example, `NSLayoutAnchor` would provide a class method -`classForGenericArgumentAtIndex:` that must be implemented by each of its -subclasses: - - @interface NSLayoutAnchor (SwiftSupport) - /// Note: must be implemented by each subclass - +(nonnull Class)classForGenericArgumentAtIndex:(NSUInteger)index; - @end - - @implementation NSLayoutAnchor - +(nonnull Class)classForGenericArgumentAtIndex:(NSUInteger)index { - NSAssert(false, @"subclass must override +classForGenericArgumentAtIndex:"); - } - @end - - @implementation NSLayoutXAxisAnchor (SwiftSupport) - +(nonnull Class)classForGenericArgumentAtIndex:(NSUInteger)index { - return [NSLayoutXAxisAnchor class]; - } - @end - - @implementation NSLayoutYAxisAnchor (SwiftSupport) - +(nonnull Class)classForGenericArgumentAtIndex:(NSUInteger)index { - return [NSLayoutYAxisAnchor class]; - } - @end - - @implementation NSLayoutDimension (SwiftSupport) - +(nonnull Class)classForGenericArgumentAtIndex:(NSUInteger)index { - return [NSLayoutDimension class]; - } - @end - -On the other hand, `GKComponentSystem` would implement an instance -method `classForGenericArgumentAtIndex:`: - - @interface GKComponentSystem (SwiftSupport) - - (nonnull Class)classForGenericArgumentAtIndex:(NSUInteger)index; - @end - - @implementation GKComponentSystem (SwiftSupport) - - (nonnull Class)classForGenericArgumentAtIndex:(NSUInteger)index { - return self.componentClass; - } - @end - -Note that many parameterized Objective-C classes cannot provide either -of these methods, because they don't carry enough information in their -instances. For example, an `NSMutableArray` has no record of what the -element type of the array is intended to be. - -However, when a parameterized class does provide this information, we -can lift some of the restrictions from the previous section: - -* If the parameterized class provides an instance method - `classForGenericArgumentAtIndex:`, the extension can use the type arguments - in its instance methods, including accessors for instance properties - and subscripts. For example: - - ```swift - extension GKComponentSystem { - var reversedComponents: [ComponentType] { - return components.reversed() - } - - static func notifyComponents(components: [ComponentType]) { - // error: cannot use `ComponentType` in a static method - } - } - ``` - -* If the parametized class provides a class method - `classForGenericArgumentAtIndex:`, the extension can use type arguments - anywhere. - - ```swift - extension NSLayoutAnchor { - func doSomething(x: AnchorType) { ... } - class func doSomethingClassy(x: AnchorType) { ... } - } - ``` - ### Subclassing parameterized Objective-C classes from Swift When subclassing a parameterized Objective-C class from Swift, the -Swift compiler will define `+classForGenericArgumentAtIndex:` and -`-classForGenericArgumentAtIndex:`. The Swift compiler has the -complete type metadata required, because it is stored in the (Swift) -type metadata, so these definitions will be correct. For example: - -```swift -class Employee : NSObject { ... } - -class EmployeeArray : NSMutableArray { - // +[EmployeeArray classForGenericArgumentAtIndex:] always returns - // ObjC type metadata for Employee -} - -class MyMutableArray : NSMutableArray { - // +[MyMutableArray classForGenericArgumentAtIndex:] returns the - // ObjC type metadata for T, extracted from the Swift metatype for - // `self`. -} -``` +Swift compiler has the complete type metadata required, because it is +stored in the (Swift) type metadata. ## Impact on existing code @@ -314,3 +187,19 @@ conceptual burden as well as a high implementation cost. The proposed solution implies less implementation cost and puts the limitations on what one can express when working with parameterized Objective-C classes without fundamentally changing the Swift model. + +## Revision history + +The [originally accepted proposal](https://github.com/swiftlang/swift-evolution/blob/3abbed3edd12dd21061181993df7952665d660dd/proposals/0057-importing-objc-generics.md) +included a mechanism by which Objective-C generic classes could implement +an informal protocol to provide reified generic arguments to Swift clients: + +> A parameterized Objective-C class can opt in to providing information +> about its type argument by implementing a method +> `classForGenericArgumentAtIndex:` either as a class method (for the first case +> described above) or as an instance method (for the second case +> described above). The method returns the metaclass for the type +> argument at the given, zero-based index. + +As of Swift 5, this feature has not been implemented, so it has been withdrawn +from the proposal. diff --git a/proposals/0058-objectivecbridgeable.md b/proposals/0058-objectivecbridgeable.md index 92307fe350..478a23adf4 100644 --- a/proposals/0058-objectivecbridgeable.md +++ b/proposals/0058-objectivecbridgeable.md @@ -3,21 +3,18 @@ * Proposal: [SE-0058](0058-objectivecbridgeable.md) * Authors: [Russ Bishop](https://github.com/russbishop), [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Joe Groff](https://github.com/jckarter) -* Status: **Deferred** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000095.html) +* Status: **Rejected** +* Review: ([pitch](https://forums.swift.org/t/idea-objectivecbridgeable/1559)) ([review](https://forums.swift.org/t/review-se-0058-allow-swift-types-to-provide-custom-objective-c-representations/2054)) ([deferral](https://forums.swift.org/t/deferred-se-0058-allow-swift-types-to-provide-custom-objective-c-representations/2167)) ([rejection](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction Provide an `ObjectiveCBridgeable` protocol that allows a Swift type to control how it is represented in Objective-C by converting into and back from an entirely separate `@objc` type. This frees library authors to create truly native Swift APIs while still supporting Objective-C. -Swift-evolution thread: [\[Idea\] ObjectiveCBridgeable](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011032.html) - - ## Motivation There is currently no good way to define a Swift-y API that makes use of generics, enums with associated values, structs, protocols with associated types, and other Swift features while still exposing that API to Objective-C. -This is especially prevelant in a mixed codebase. Often an API must be dumbed-down or Swift features eschewed because rewriting the entire codebase is impractical and Objective-C code must be able to call the new Swift code. This results in a situation where new code or refactored code adopts an Objective-C compatible API which is compromised, less type safe, and isn't as nice to work with as a truly native Swift API. +This is especially prevelent in a mixed codebase. Often an API must be dumbed-down or Swift features eschewed because rewriting the entire codebase is impractical and Objective-C code must be able to call the new Swift code. This results in a situation where new code or refactored code adopts an Objective-C compatible API which is compromised, less type safe, and isn't as nice to work with as a truly native Swift API. The cascading effect is even worse because when the last vestiges of Objective-C have been swept away, you're left with a mountain of Swift code that essentially looks like a direct port of Objective-C code and doesn't take advantage of any of Swift's modern features. @@ -80,7 +77,7 @@ public protocol ObjectiveCBridgeable { /// Objective-C thunk or when calling Objective-C code. /// /// - note: This initializer should eagerly perform the - /// conversion without defering any work for later, + /// conversion without deferring any work for later, /// returning `nil` if the conversion fails. init?(bridgedFromObjectiveC: ObjectiveCType) @@ -147,7 +144,7 @@ The compiler generates automatic thunks only when there is no ambiguity, while e 3. Bridged collection types will still observe the protocol conformance if cast to a Swift type (eg: `NSArray as? [Int]` will call the `ObjectiveCBridgeable` implementation on `Array`, which itself will call the implementation on `Int` for the elements) 2. A Swift type may bridge to an Objective-C base class then provide different subclass instances at runtime, but no other Swift type may bridge to that base class or any of its subclasses. 1. The compiler should emit a diagnostic when it detects two Swift types attempting to bridge to the same `ObjectiveCType`. -3. An exception to these rules exists for trivially convertable built-in types like `NSInteger` <--> `Int` when specified outside of a bridged collection type. In those cases the compiler will continue the existing behavior, bypassing the `ObjectiveCBridgeable` protocol. The effect is that types like `Int` will not bridge to `NSNumber` unless contained inside a collection type (see `BuiltInBridgeable below`). +3. An exception to these rules exists for trivially convertible built-in types like `NSInteger` <--> `Int` when specified outside of a bridged collection type. In those cases the compiler will continue the existing behavior, bypassing the `ObjectiveCBridgeable` protocol. The effect is that types like `Int` will not bridge to `NSNumber` unless contained inside a collection type (see `BuiltInBridgeable below`). ### Resiliance @@ -230,7 +227,7 @@ However there doesn't appear to be a convincing case to support such complex beh ### BuiltInBridgeable -On the mailing list the idea of a protocol to supercede `ObjectiveCBridgeable` for built-in types like `Int` was brought up but not considered essential for this proposal. (The protocol would be decorative only, not having any functions or properties). +On the mailing list the idea of a protocol to supersede `ObjectiveCBridgeable` for built-in types like `Int` was brought up but not considered essential for this proposal. (The protocol would be decorative only, not having any functions or properties). These types are special because they bridge differently inside collections vs outside. The compiler already has *magic* knowledge of these types and I don't anticipate the list of types will ever get any longer. The only benefit of a `BuiltInBridgeable` protocol would be to explicitly declare which types have this "magic". diff --git a/proposals/0059-updated-set-apis.md b/proposals/0059-updated-set-apis.md index a19948e8ab..1d69edb0c4 100644 --- a/proposals/0059-updated-set-apis.md +++ b/proposals/0059-updated-set-apis.md @@ -3,8 +3,8 @@ * Proposal: [SE-0059](0059-updated-set-apis.md) * Author: [Dave Abrahams](https://github.com/dabrahams) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000105.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0059-update-api-naming-guidelines-and-rewrite-set-apis-accordingly/2251) ## Introduction diff --git a/proposals/0060-defaulted-parameter-order.md b/proposals/0060-defaulted-parameter-order.md index b142369f9c..f6cd8d01f8 100644 --- a/proposals/0060-defaulted-parameter-order.md +++ b/proposals/0060-defaulted-parameter-order.md @@ -2,9 +2,9 @@ * Proposal: [SE-0060](0060-defaulted-parameter-order.md) * Author: [Joe Groff](https://github.com/jckarter) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000146.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0060-enforcing-order-of-defaulted-parameters/2573) * Bug: [SR-1489](https://bugs.swift.org/browse/SR-1489) ## Introduction @@ -14,7 +14,7 @@ method names with significant, order-sensitive argument labels, but an exception is made for parameters with default arguments. We should remove this exception. -Swift-evolution thread: [Enforce argument order for defaulted parameters](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/013789.html) +Swift-evolution thread: [Enforce argument order for defaulted parameters](https://forums.swift.org/t/pitch-enforce-argument-order-for-defaulted-parameters/1989) ## Motivation @@ -72,7 +72,7 @@ the arguments. This should be easy to automate. ## Alternatives considered -[Matthew Johnson](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/013802.html) +[Matthew Johnson](https://forums.swift.org/t/pitch-enforce-argument-order-for-defaulted-parameters/1989/7) raises an interesting point in favor of our current behavior. For memberwise initializers, it makes sense to allow reordering, because declared member order is not usually significant otherwise: @@ -81,7 +81,7 @@ is not usually significant otherwise: > In fact, I have found myself wishing non-defaulted memberwise initializer parameters were re-orderable at times, especially when using the implicit memberwise initializer for a struct. Source order for property declarations does not always match what makes the most sense at the initialization site (something that was pointed out numerous times during the review of my memberwise init proposal). -[Erica Sadun](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/013791.html) +[Erica Sadun](https://forums.swift.org/t/pitch-enforce-argument-order-for-defaulted-parameters/1989/3) notes that defaulted arguments are useful to simulate an ad-hoc sum type parameter: diff --git a/proposals/0061-autoreleasepool-signature.md b/proposals/0061-autoreleasepool-signature.md index c170fb686b..e9dc5a961c 100644 --- a/proposals/0061-autoreleasepool-signature.md +++ b/proposals/0061-autoreleasepool-signature.md @@ -2,9 +2,9 @@ * Proposal: [SE-0061](0061-autoreleasepool-signature.md) * Author: [Timothy J. Wood](https://github.com/tjw) -* Review Manager: [Dave Abrahams](http://github.com/dabrahams) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000123.html) +* Review Manager: [Dave Abrahams](https://github.com/dabrahams) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0061-add-generic-result-and-error-handling-to-autoreleasepool/2425) * Bugs: [SR-842](https://bugs.swift.org/browse/SR-842), [SR-1394](https://bugs.swift.org/browse/SR-1394) ## Introduction @@ -14,10 +14,10 @@ support a return value or error handling, making it difficult and error-prone to pass results or errors from the body to the calling context. Swift-evolution thread: A first call for discussion was -[made here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160314/013054.html). +[made here](https://forums.swift.org/t/update-the-signature-of-objectivec-autoreleasepool-sr-842/1886). Dmitri Gribenko pointed out that adding a generic return type would be useful -(first in my premature pull request) and then also [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013059.html). -Jordan Rose [pointed out](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013077.html) +(first in my premature pull request) and then also [here](https://forums.swift.org/t/update-the-signature-of-objectivec-autoreleasepool-sr-842/1886/2). +Jordan Rose [pointed out](https://forums.swift.org/t/update-the-signature-of-objectivec-autoreleasepool-sr-842/1886/3) that care was needed to avoid inferring an incorrect return type for the body block, but after testing we confirmed that this is handled correctly by the compiler. @@ -101,13 +101,13 @@ suggested adding `throws`, but Dmitri Gribenko pointed out that adding a generic return type would be better. Further discussion raised the question of [whether `autoreleasepool` should -behave like a statement](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/013697.html) +behave like a statement](https://forums.swift.org/t/update-the-signature-of-objectivec-autoreleasepool-sr-842/1886/13) in the future, or whether it should behave like an expression by returning the result of the passed in body, with some weighing in on either side. -Chris Lattner drew an [analogy to `forEach`](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/013697.html) +Chris Lattner drew an [analogy to `forEach`](https://forums.swift.org/t/update-the-signature-of-objectivec-autoreleasepool-sr-842/1886/13) and pointed out that `@autoreleasepool` *is* a statement in Objective-C, while Jordan Rose found this case [more like `withCString`, or -`withUnsafeMutablePointer`](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/013698.html), +`withUnsafeMutablePointer`](https://forums.swift.org/t/update-the-signature-of-objectivec-autoreleasepool-sr-842/1886/14), where having them return a value yields nice simplifications and avoids optional `var`s. diff --git a/proposals/0062-objc-keypaths.md b/proposals/0062-objc-keypaths.md index fcab029edd..54fa946647 100644 --- a/proposals/0062-objc-keypaths.md +++ b/proposals/0062-objc-keypaths.md @@ -3,15 +3,15 @@ * Proposal: [SE-0062](0062-objc-keypaths.md) * Author: [David Hart](https://github.com/hartbit) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000101.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0062-referencing-objective-c-key-paths/2198) * Bug: [SR-1237](https://bugs.swift.org/browse/SR-1237) ## Introduction In Objective-C and Swift, key-paths used by KVC and KVO are represented as string literals (e.g., `"friend.address.streetName"`). This proposal seeks to improve the safety and resilience to modification of code using key-paths by introducing a compiler-checked expression. -[SE Draft](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160229/011845.html), [Review thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160404/014435.html) +[SE Draft](https://forums.swift.org/t/draft-obejctive-c-keypaths/1688), [Review thread](https://forums.swift.org/t/review-se-0062-referencing-objective-c-key-paths/2086) ## Motivation diff --git a/proposals/0063-swiftpm-system-module-search-paths.md b/proposals/0063-swiftpm-system-module-search-paths.md index 7fe3a6f670..cb11562abf 100644 --- a/proposals/0063-swiftpm-system-module-search-paths.md +++ b/proposals/0063-swiftpm-system-module-search-paths.md @@ -3,9 +3,9 @@ * Proposal: [SE-0063](0063-swiftpm-system-module-search-paths.md) * Author: [Max Howell](https://github.com/mxcl) * Review Manager: [Anders Bertelrud](https://github.com/abertelrud) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000103.html) -* Pull Request: [apple/swift-package-manager#257](https://github.com/apple/swift-package-manager/pull/257) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0063-swiftpm-system-module-search-paths/2218) +* Implementation: [apple/swift-package-manager#257](https://github.com/apple/swift-package-manager/pull/257) ## Introduction @@ -17,7 +17,7 @@ The current system for using these module-map files with SwiftPM works, but with a number of caveats that must be addressed. -[swift-evolution thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013201.html) +[swift-evolution thread](https://forums.swift.org/t/draft-proposal-swiftpm-system-module-search-paths/1914) ## Terminology @@ -133,10 +133,10 @@ parameters, they can be added on a per enum basis. #### Install-names are not standard -`apt` is used across multiple distirbutions and the install-names for +`apt` is used across multiple distributions and the install-names for tools vary. Even for the same distribution install-names may vary across releases (eg. from Ubuntu 15.04 to Ubuntu 15.10) or even on -ocassion at finer granularity. +occasion at finer granularity. We will not add explicit handling for this, but one can imagine the enums for different system packagers could be supplemented in a backwards diff --git a/proposals/0064-property-selectors.md b/proposals/0064-property-selectors.md index cc606ce558..e3e7b61e35 100644 --- a/proposals/0064-property-selectors.md +++ b/proposals/0064-property-selectors.md @@ -3,16 +3,16 @@ * Proposal: [SE-0064](0064-property-selectors.md) * Author: [David Hart](https://github.com/hartbit) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000102.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0064-referencing-the-objective-c-selector-of-property-getters-and-setters/2199) * Bug: [SR-1239](https://bugs.swift.org/browse/SR-1239) ## Introduction Proposal [SE-0022](0022-objc-selectors.md) was accepted and implemented to provide a `#selector` expression to reference Objective-C method selectors. Unfortunately, it does not allow referencing the getter and setter methods of properties. This proposal seeks to provide a design to reference those methods for the Swift 3.0 timeframe. -* [Original swift-evolution thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010791.html) -* [Follow-up swift-evolution thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/010960.html) +* [Original swift-evolution thread](https://forums.swift.org/t/argument-of-selector-cannot-refer-to-a-property/1517) +* [Follow-up swift-evolution thread](https://forums.swift.org/t/discussion-referencing-the-objective-c-selector-of-property-getters-and-setters/1556) ## Motivation @@ -53,5 +53,5 @@ The introduction of the new `#selector` overrides has no impact on existing code ## Alternatives considered -A long term alternive could arrise from the design of lenses in Swift. But as this is purely hypothetical and out of scope for Swift 3, this proposal fixes the need for referencing property selectors in a type-safe way straight-away. +A long term alternative could arise from the design of lenses in Swift. But as this is purely hypothetical and out of scope for Swift 3, this proposal fixes the need for referencing property selectors in a type-safe way straight-away. diff --git a/proposals/0065-collections-move-indices.md b/proposals/0065-collections-move-indices.md index 3ba442df0f..2353847ca5 100644 --- a/proposals/0065-collections-move-indices.md +++ b/proposals/0065-collections-move-indices.md @@ -3,10 +3,10 @@ * Proposal: [SE-0065](0065-collections-move-indices.md) * Authors: [Dmitri Gribenko](https://github.com/gribozavr), [Dave Abrahams](https://github.com/dabrahams), [Maxim Moiseev](https://github.com/moiseev) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-April/000115.html), [Swift-evolution thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160229/011552.html) -* Pull Request: [apple/swift#2108](https://github.com/apple/swift/pull/2108) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/21fac2e8034e79e4f44c1c8799808fc8cba83395/proposals/0065-collections-move-indices.md), [2](https://github.com/apple/swift-evolution/blob/1a821cf7ccbdf1d7566e9ce2e991bdd835ba3b7d/proposals/0065-collections-move-indices.md), [3](https://github.com/apple/swift-evolution/blob/d44c3e7c189ba39ddf8a914ae8b78b71f88fdcdf/proposals/0065-collections-move-indices.md), [4](https://github.com/apple/swift-evolution/blob/57639040dc08d2f0b16d9bda527db069589b58d1/proposals/0065-collections-move-indices.md) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0065-a-new-model-for-collections/2371), [Swift-evolution thread](https://forums.swift.org/t/rfc-new-collections-model-collections-advance-indices/1643) +* Implementation: [apple/swift#2108](https://github.com/apple/swift/pull/2108) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/21fac2e8034e79e4f44c1c8799808fc8cba83395/proposals/0065-collections-move-indices.md), [2](https://github.com/swiftlang/swift-evolution/blob/1a821cf7ccbdf1d7566e9ce2e991bdd835ba3b7d/proposals/0065-collections-move-indices.md), [3](https://github.com/swiftlang/swift-evolution/blob/d44c3e7c189ba39ddf8a914ae8b78b71f88fdcdf/proposals/0065-collections-move-indices.md), [4](https://github.com/swiftlang/swift-evolution/blob/57639040dc08d2f0b16d9bda527db069589b58d1/proposals/0065-collections-move-indices.md) ## Summary diff --git a/proposals/0066-standardize-function-type-syntax.md b/proposals/0066-standardize-function-type-syntax.md index 950867a313..e5cd1c72a7 100644 --- a/proposals/0066-standardize-function-type-syntax.md +++ b/proposals/0066-standardize-function-type-syntax.md @@ -2,10 +2,10 @@ * Proposal: [SE-0066](0066-standardize-function-type-syntax.md) * Author: [Chris Lattner](https://github.com/lattner) -* Review Manager: [Doug Gregor](https://github/com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000138.html) -* Commit: [apple/swift@3d2b5bc](https://github.com/apple/swift/commit/3d2b5bcc5350e1dea2ed8a0a95cd12ff5c760f24) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0066-standardize-function-type-argument-syntax-to-require-parentheses/2488) +* Implementation: [apple/swift@3d2b5bc](https://github.com/apple/swift/commit/3d2b5bcc5350e1dea2ed8a0a95cd12ff5c760f24) ## Introduction @@ -37,7 +37,7 @@ function types themselves, and offers no additional expressive capability (this is just syntactic sugar). This proposal suggests that we simply eliminate the special case and require parentheses on all argument lists for function types. -Swift-evolution thread: [\[pitch\] Eliminate the "T1 -> T2" syntax, require "(T1) -> T2"](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160411/014986.html) +Swift-evolution thread: [\[pitch\] Eliminate the "T1 -> T2" syntax, require "(T1) -> T2"](https://forums.swift.org/t/pitch-eliminate-the-t1-t2-syntax-require-t1-t2/2211) ## Motivation diff --git a/proposals/0067-floating-point-protocols.md b/proposals/0067-floating-point-protocols.md index 70bbc952d4..b93f354014 100644 --- a/proposals/0067-floating-point-protocols.md +++ b/proposals/0067-floating-point-protocols.md @@ -3,10 +3,10 @@ * Proposal: [SE-0067](0067-floating-point-protocols.md) * Author: [Stephen Canon](https://github.com/stephentyrone) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000121.html) -* Pull Request: [apple/swift#2453](https://github.com/apple/swift/pull/2453) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/fb1368a6a5474f57aa8f1846b5355d18753098f3/proposals/0067-floating-point-protocols.md) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0067-enhanced-floating-point-protocols/2420) +* Implementation: [apple/swift#2453](https://github.com/apple/swift/pull/2453) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/fb1368a6a5474f57aa8f1846b5355d18753098f3/proposals/0067-floating-point-protocols.md) ## Introduction @@ -19,15 +19,15 @@ for generic programming with the most commonly used types. Revision 2 reintroduces some of the details of the `Arithmetic` protocol from earlier drafts of this proposal, but as methods in the `FloatingPoint` API, with the goal of reducing the number of overloads for basic operations. This -change was requested by some members of the core langauge team. +change was requested by some members of the core language team. Revision 2 also incorporates a number of suggestions from the review list and corrects some typos; thanks especially to Xiaodi Wu for thoughtful feedback. Consult the changelog at the end of this document for full details. -* [Proposal draft](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160411/014969.html) -* [Review #1](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160418/015300.html) -* [Review #2](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/015733.html) +* [Proposal draft](https://forums.swift.org/t/proposal-draft-enhanced-floating-point-protocols/2201) +* [Review #1](https://forums.swift.org/t/review-se-0067-enhanced-floating-point-protocols/2264) +* [Review #2](https://forums.swift.org/t/review-2-se-0067-enhanced-floating-point-protocols/2339) ## Motivation @@ -188,7 +188,7 @@ public protocol FloatingPoint: Comparable, IntegerLiteralConvertible, SignedNumb /// This quantity, or a related quantity is sometimes called "epsilon" or /// "machine epsilon". We avoid that name because it has different meanings /// in different languages, which can lead to confusion, and because it - /// suggests that it is an good tolerance to use for comparisons, + /// suggests that it is a good tolerance to use for comparisons, /// which is almost never is. /// /// (See https://en.wikipedia.org/wiki/Machine_epsilon for more detail) @@ -449,7 +449,7 @@ public protocol FloatingPoint: Comparable, IntegerLiteralConvertible, SignedNumb @warn_unused_result func isLessThanOrEqualTo(_ other: Self) -> Bool - /// True if and only if `self` preceeds `other` in the IEEE 754 total order + /// True if and only if `self` precedes `other` in the IEEE 754 total order /// relation. /// /// This relation is a refinement of `<=` that provides a total order on all @@ -762,7 +762,7 @@ and `isNaN`. ## Changes from the draft proposal 1. Removed the `Arithmetic` protocol; it may be circulated again in the future -as an independent proposal, or as part of an new model for integers. +as an independent proposal, or as part of a new model for integers. 2. Removed the `add[ing]`, `subtract[ing]`, etc methods, which were hooks for `Arithmetic`. This proposal now includes only the existing operator forms. diff --git a/proposals/0068-universal-self.md b/proposals/0068-universal-self.md index ca35591a25..f129386def 100644 --- a/proposals/0068-universal-self.md +++ b/proposals/0068-universal-self.md @@ -1,12 +1,13 @@ # Expanding Swift `Self` to class members and value types * Proposal: [SE-0068](0068-universal-self.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Accepted with revisions** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/015977.html) +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 5.1)** +* Implementation: [apple/swift#22863](https://github.com/apple/swift/pull/22863) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modification-se-0068-expanding-swift-self-to-class-members-and-value-types/2373) * Bug: [SR-1340](https://bugs.swift.org/browse/SR-1340) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/bcd77b028cb2fc9f07472532b120e927c7e48b34/proposals/0068-universal-self.md), [2](https://github.com/apple/swift-evolution/blob/13d9771e86c5639b8320f05e5daa31a62bac0f07/proposals/0068-universal-self.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/bcd77b028cb2fc9f07472532b120e927c7e48b34/proposals/0068-universal-self.md), [2](https://github.com/swiftlang/swift-evolution/blob/13d9771e86c5639b8320f05e5daa31a62bac0f07/proposals/0068-universal-self.md) ## Introduction @@ -15,7 +16,7 @@ by renaming `dynamicType` to `Self`. This establishes a universal and consistent way to refer to the dynamic type of the current receiver. -*This proposal was discussed on the Swift Evolution list in the [\[Pitch\] Adding a Self type name shortcut for static member access](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160404/014132.html) thread.* +*This proposal was discussed on the Swift Evolution list in the [\[Pitch\] Adding a Self type name shortcut for static member access](https://forums.swift.org/t/pitch-adding-a-self-type-name-shortcut-for-static-member-access/2056) thread.* ## Motivation @@ -24,7 +25,7 @@ a static member or passing types for unsafe bitcasts, among other uses. You can either specify a type by its full name or use `self.dynamicType` to access an instance's dynamic runtime type as a value. -``` +```swift struct MyStruct { static func staticMethod() { ... } func instanceMethod() { @@ -54,13 +55,13 @@ You will continue to specify full type names for any other use. Joe Groff writes Not at this time -## Acknowlegements +## Acknowledgements -Thanks Sean Heber, Kevin Ballard, Joe Groff, Timothy Wood, Brent Royal-Gordon, Andrey Tarantsov, Austin Zheng +Thanks Sean Heber, Lily Ballard, Joe Groff, Timothy Wood, Becca Royal-Gordon, Andrey Tarantsov, Austin Zheng ## Rationale -On [April 27, 2016](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/015977.html), the core team decided to **accept a subset of** this proposal. +On [April 27, 2016](https://forums.swift.org/t/accepted-with-modification-se-0068-expanding-swift-self-to-class-members-and-value-types/2373), the core team decided to **accept a subset of** this proposal. > This proposal had light discussion in the community review process, but the core team heavily debated it. It includes two pieces: > diff --git a/proposals/0069-swift-mutability-for-foundation.md b/proposals/0069-swift-mutability-for-foundation.md index c5f0b80c35..e16169cf97 100644 --- a/proposals/0069-swift-mutability-for-foundation.md +++ b/proposals/0069-swift-mutability-for-foundation.md @@ -3,8 +3,8 @@ * Proposal: [SE-0069](0069-swift-mutability-for-foundation.md) * Author: [Tony Parker](https://github.com/parkera) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000132.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0069-mutability-and-foundation-value-types/2460) ## Introduction @@ -14,7 +14,7 @@ One of the core principles of Swift is "mutability when you need it." This is es * [Building Better Apps with Value Types in Swift - WWDC 2015 (Doug Gregor)](https://developer.apple.com/videos/play/wwdc2015/414/) * [Swift Programming Language - Classes and Structures](https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/ClassesAndStructures.html#//apple_ref/doc/uid/TP40014097-CH13-ID82) -[Swift Evolution Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160418/015503.html), [Swift Evolution Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/015682.html) +[Swift Evolution Discussion](https://forums.swift.org/t/mutability-for-foundation-types-in-swift/2300), [Swift Evolution Review](https://forums.swift.org/t/review-se-0069-mutability-and-foundation-value-types/2332) This concept is so important that it is literally the second thing taught in _The Swift Programming Language_, right after `print("Hello, world!")`: @@ -22,7 +22,7 @@ This concept is so important that it is literally the second thing taught in _Th > > Use `let` to make a constant and `var` to make a variable. The value of a constant doesn’t need to be known at compile time, but you must assign it a value exactly once. > -> __Excerpt From: Apple Inc. “[The Swift Programming Language (Swift 2.1).](https://itun.es/us/jEUH0.l__)__” +> __Excerpt From: Apple Inc. “[The Swift Programming Language (Swift 3.0.1).](https://itun.es/us/jEUH0.l)__” When certain Foundation types are imported into Swift, they do not fully take advantage of the features that Swift has to offer developers for controlling mutability of their objects. diff --git a/proposals/0070-optional-requirements.md b/proposals/0070-optional-requirements.md index 64255d64ad..0fa44d5c14 100644 --- a/proposals/0070-optional-requirements.md +++ b/proposals/0070-optional-requirements.md @@ -2,9 +2,9 @@ * Proposal: [SE-0070](0070-optional-requirements.md) * Author: [Doug Gregor](https://github.com/DougGregor) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000124.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0070-make-optional-requirements-objective-c-only/2426) * Bug: [SR-1395](https://bugs.swift.org/browse/SR-1395) ## Introduction @@ -21,11 +21,11 @@ feature. Swift-evolution threads: -* [Is there an underlying reason why optional protocol requirements need @objc?](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160229/011854.html) -* [\[Proposal\] Make optional protocol methods first class citizens](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/013770.html) -* [\[Idea\] How to eliminate 'optional' protocol requirements](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160404/014471.html) -* [\[Proposal draft\] Make Optional Requirements Objective-C-only](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160418/015552.html) -* [\[Review\] SE-0070: Make Optional Requirements Objective-C only](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/015681.html) +* [Is there an underlying reason why optional protocol requirements need @objc?](https://forums.swift.org/t/is-there-an-underlying-reason-why-optional-protocol-requirements-need-objc/1681) +* [\[Proposal\] Make optional protocol methods first class citizens](https://forums.swift.org/t/proposal-make-optional-protocol-methods-first-class-citizens/1990) +* [\[Idea\] How to eliminate 'optional' protocol requirements](https://forums.swift.org/t/idea-how-to-eliminate-optional-protocol-requirements/2095) +* [\[Proposal draft\] Make Optional Requirements Objective-C-only](https://forums.swift.org/t/proposal-draft-make-optional-requirements-objective-c-only/2310) +* [\[Review\] SE-0070: Make Optional Requirements Objective-C only](https://forums.swift.org/t/review-se-0070-make-optional-requirements-objective-c-only/2343) ## Motivation @@ -169,7 +169,7 @@ example, Objective-C protocols could be annotated with attributes that say what the default implementation for each optional requirement is (to be used only in Swift), but such a massive auditing effort is impractical. There is a related notion of [caller-site default -implementations](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160404/014471.html) +implementations](https://forums.swift.org/t/idea-how-to-eliminate-optional-protocol-requirements/2095) that was not well-received due to its complexity. Initially, this proposal introduce a new keyword diff --git a/proposals/0071-member-keywords.md b/proposals/0071-member-keywords.md index 44103a4d9c..3ac330d7b0 100644 --- a/proposals/0071-member-keywords.md +++ b/proposals/0071-member-keywords.md @@ -3,8 +3,8 @@ * Proposal: [SE-0071](0071-member-keywords.md) * Author: [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000122.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0071-allow-most-keywords-in-member-references/2421) ## Introduction @@ -21,8 +21,8 @@ after a ".", similarly to how [SE-0001](0001-keywords-as-argument-labels.md) allows keywords are argument labels. -* [\[Idea\] Allowing most keywords after "."](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/011169.html) -* [\[Review\] SE-0071: Allow (most) keywords in member references](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/015760.html) +* [\[Idea\] Allowing most keywords after "."](https://forums.swift.org/t/idea-allowing-most-keywords-after/1592) +* [\[Review\] SE-0071: Allow (most) keywords in member references](https://forums.swift.org/t/review-se-0071-allow-most-keywords-in-member-references/2346) ## Motivation diff --git a/proposals/0072-eliminate-implicit-bridging-conversions.md b/proposals/0072-eliminate-implicit-bridging-conversions.md index b1d9006042..47910a312a 100644 --- a/proposals/0072-eliminate-implicit-bridging-conversions.md +++ b/proposals/0072-eliminate-implicit-bridging-conversions.md @@ -3,17 +3,15 @@ * Proposal: [SE-0072](0072-eliminate-implicit-bridging-conversions.md) * Author: [Joe Pamer](https://github.com/jopamer) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000137.html) -* Pull Requests: [apple/swift#2419](https://github.com/apple/swift/pull/2419), - [apple/swift#2440](https://github.com/apple/swift/pull/2440), - [apple/swift#2441](https://github.com/apple/swift/pull/2441) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0072-fully-eliminate-implicit-bridging-conversions-from-swift/2487) +* Implementation: [apple/swift#2419](https://github.com/apple/swift/pull/2419) ## Introduction In Swift 1.2, we attempted to remove all implicit bridging conversions from the language. Unfortunately, problems with how the v1.2 compiler imported various un-annotated Objective-C APIs caused us to scale back on our ambitions. In the interest of further simplifying our type system and our user model, we would like to complete this work and fully remove implicit bridging conversions from the language in Swift 3. -This was discussed in [this swift-evolution thread.](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160418/015238.html) +This was discussed in [this swift-evolution thread.](https://forums.swift.org/t/pitch-fully-eliminate-implicit-bridging-conversions-in-swift-3/2256) ## Motivation Prior to Swift 1.2, conversions between bridged Swift value types and their associated Objective-C types could be implicitly inferred in both directions. For example, you could pass an NSString object to a function expecting a String value, and vice versa. diff --git a/proposals/0073-noescape-once.md b/proposals/0073-noescape-once.md index 385b6810ec..793f313095 100644 --- a/proposals/0073-noescape-once.md +++ b/proposals/0073-noescape-once.md @@ -2,9 +2,9 @@ * Proposal: [SE-0073](0073-noescape-once.md) * Authors: [Félix Cloutier](https://github.com/zneak), [Gwendal Roué](https://github.com/groue) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000147.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0073-marking-closures-as-executing-exactly-once/2575) ## Introduction @@ -14,7 +14,7 @@ escape, and that it is run exactly once on any code path returning from the function. For clients, it allows the compiler to relax initialization requirements and close the gap between closure and "inline code" a little bit. -Swift-evolution thread: [Guaranteed closure execution](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160125/008167.html) +Swift-evolution thread: [Guaranteed closure execution](https://forums.swift.org/t/guaranteed-closure-execution/1233) ## Motivation @@ -128,7 +128,7 @@ expected. ## Not requiring exactly one execution Assuming that the main goal of this proposal is to relax initialization -requirements, a unique invocation of the closure is not stricly required. +requirements, a unique invocation of the closure is not strictly required. However the requirement of unique invocation makes the proposal simpler to understand. @@ -183,7 +183,7 @@ future proposals. ## Rationale -On [May 11, 2016](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000147.html), the core team decided to **Reject** this proposal for Swift 3. +On [May 11, 2016](https://forums.swift.org/t/rejected-se-0073-marking-closures-as-executing-exactly-once/2575), the core team decided to **Reject** this proposal for Swift 3. The feedback on the proposal was generally positive both from the community and core team. That said, it is being rejected for Swift 3 two reasons: diff --git a/proposals/0074-binary-search.md b/proposals/0074-binary-search.md index da14970b48..356615ad03 100644 --- a/proposals/0074-binary-search.md +++ b/proposals/0074-binary-search.md @@ -2,16 +2,16 @@ * Proposal: [SE-0074](0074-binary-search.md) * Authors: [Lorenzo Racca](https://github.com/lorenzoracca), [Jeff Hajewski](https://github.com/j-haj), [Nate Cook](https://github.com/natecook1000) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000148.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0074-implementation-of-binary-search-functions/2576) ## Introduction Swift does not offer any way to efficiently search sorted collections. This proposal seeks to add a few different functions that implement the binary search algorithm. -- Swift-evolution thread: [\[Proposal\] Add Binary Search functions to SequenceType](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160314/012680.html) +- Swift-evolution thread: [\[Proposal\] Add Binary Search functions to SequenceType](https://forums.swift.org/t/proposal-add-binary-search-functions-to-sequencetype/1827) - JIRA: [Swift/SR-368](https://bugs.swift.org/browse/SR-368) ## Motivation @@ -193,7 +193,7 @@ extension Collection { ## Example usage -As an example of how the `partitionedIndex(of:)` method enables heterogenous binary search, this `SortedDictionary` type uses an array of `(Word, Definition)` tuples as its storage, sorted by `Word`. +As an example of how the `partitionedIndex(of:)` method enables heterogeneous binary search, this `SortedDictionary` type uses an array of `(Word, Definition)` tuples as its storage, sorted by `Word`. Better explained examples can be found in the Swift playground available [here to download](https://github.com/lorenzoracca/Swift-binary-search/blob/binarysearch/Binary%20Search%20Proposal.playground.zip). @@ -280,4 +280,4 @@ The authors considered a few alternatives to the current proposal: ## Rationale -On [May 11, 2016](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000148.html), the core team decided to **Reject** this proposal. The feedback on the proposal was generally positive about the concept of adding binary search functionality, but negative about the proposal as written, with feedback that it was adding too much complexity to the API. +On [May 11, 2016](https://forums.swift.org/t/rejected-se-0074-implementation-of-binary-search-functions/2576), the core team decided to **Reject** this proposal. The feedback on the proposal was generally positive about the concept of adding binary search functionality, but negative about the proposal as written, with feedback that it was adding too much complexity to the API. diff --git a/proposals/0075-import-test.md b/proposals/0075-import-test.md index b61015c984..7844243832 100644 --- a/proposals/0075-import-test.md +++ b/proposals/0075-import-test.md @@ -1,23 +1,23 @@ # Adding a Build Configuration Import Test * Proposal: [SE-0075](0075-import-test.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Accepted** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000159.html) +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 4.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0075-adding-a-build-configuration-import-test/2683) * Bug: [SR-1560](https://bugs.swift.org/browse/SR-1560) ## Introduction Expanding the build configuration suite to test for the ability to import certain -modules was [first introduced](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010693.html) -on the Swift-Evolution list by Kevin Ballard. Although his initial idea (checking for Darwin +modules was [first introduced](https://forums.swift.org/t/idea-support-if-os-darwin-as-shorthand-for-os-ios-os-osx-os-watchos-os-tvos/1493) +on the Swift-Evolution list by Lily Ballard. Although her initial idea (checking for Darwin to differentiate Apple targets from non-Apple targets) proved problematic, developers warmly greeted the notion of an import-based configuration test. Dmitri Gribenko wrote, "There's a direction that we want to move to a unified name for the libc module for all platform, so 'can import Darwin' might not be a viable long-term strategy." Testing for imports offers advantages that stand apart from this one use-case: to test for API availability before use. -[Swift Evolution Review Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160509/017044.html) +[Swift Evolution Review Thread](https://forums.swift.org/t/review-se-0075-adding-a-build-configuration-import-test/2542) ## Motivation @@ -25,20 +25,20 @@ Swift's existing set of build configurations specify platform differences, not m ```swift #if canImport(UIKit) - // UIKit-based code - #elseif canImport(Cocoa) - // OSX code - #elseif - // Workaround/text, whatever + // UIKit-based code +#elseif canImport(Cocoa) + // OSX code +#elseif + // Workaround/text, whatever #endif ``` -Guarding code with operating system tests can be less future-proofed than testing for module support. Excluding OS X to use UIColor creates code that might eventually find its way to a Linux plaform. Targeting Apple platforms by inverting a test for Linux essentially broke after the introduction of `Windows` and `FreeBSD` build configurations: +Guarding code with operating system tests can be less future-proofed than testing for module support. Excluding OS X to use UIColor creates code that might eventually find its way to a Linux platform. Targeting Apple platforms by inverting a test for Linux essentially broke after the introduction of `Windows` and `FreeBSD` build configurations: ```swift // Exclusive os tests are brittle #if !os(Linux) - // Matches OSX, iOS, watchOS, tvOS, Windows, FreeBSD + // Matches OSX, iOS, watchOS, tvOS, Windows, FreeBSD #endif ``` @@ -73,7 +73,7 @@ frameworks at runtime is to do it via Obj-C. Some sort of check like the ones yo #if canImport(module) // provide solution with module APIs - #else +#else // provide alternative solution that does not depend on that module #endif ``` diff --git a/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md b/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md index 25f5c8722a..a84194e2f9 100644 --- a/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md +++ b/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md @@ -2,16 +2,16 @@ * Proposal: [SE-0076](0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md) * Author: [Janosch Hildebrand](https://github.com/Jnosh) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000149.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0076-add-overrides-taking-an-unsafepointer-source-to-non-destructive-copying-methods-on-unsafemutablepointer/2577) * Bug: [SR-1490](https://bugs.swift.org/browse/SR-1490) ## Introduction `UnsafeMutablePointer` includes several methods to non-destructively copy elements from memory pointed to by another `UnsafeMutablePointer` instance. I propose adding overloads of these methods to `UnsafeMutablePointer` that allow an `UnsafePointer` source. -Swift-evolution thread: [\[Pitch\] Add overrides with UnsafePointer sources to non-destructive copying methods on UnsafeMutablePointer](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160201/008827.html) +Swift-evolution thread: [\[Pitch\] Add overrides with UnsafePointer sources to non-destructive copying methods on UnsafeMutablePointer](https://forums.swift.org/t/pitch-add-overrides-with-unsafepointer-sources-to-non-destructive-copying-methods-on-unsafemutablepointer/1294) ## Motivation diff --git a/proposals/0077-operator-precedence.md b/proposals/0077-operator-precedence.md index 3ffdc4f420..aaecb84955 100644 --- a/proposals/0077-operator-precedence.md +++ b/proposals/0077-operator-precedence.md @@ -2,14 +2,14 @@ * Proposal: [SE-0077](0077-operator-precedence.md) * Author: [Anton Zhilin](https://github.com/Anton3) -* Review Manager: [Joe Groff](http://github.com/jckarter) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160704/023745.html) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0077-v2-improved-operator-declarations/3321) **Revision history** -- **[v1](https://github.com/apple/swift-evolution/blob/40c2acad241106e1cfe697d0f75e1855dc9e96d5/proposals/0077-operator-precedence.md)** Initial version -- **[v2](https://github.com/apple/swift-evolution/blob/1f3ae8bfecb2ba70d30767607f0bd3279feeec90/proposals/0077-operator-precedence.md)** After the first review +- **[v1](https://github.com/swiftlang/swift-evolution/blob/40c2acad241106e1cfe697d0f75e1855dc9e96d5/proposals/0077-operator-precedence.md)** Initial version +- **[v2](https://github.com/swiftlang/swift-evolution/blob/1f3ae8bfecb2ba70d30767607f0bd3279feeec90/proposals/0077-operator-precedence.md)** After the first review - **v3** After the second review ## Introduction @@ -28,7 +28,7 @@ precedencegroup ComparisonPrecedence { infix operator <> : ComparisonPrecedence ``` -[Swift-evolution discussion thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160328/014062.html) +[Swift-evolution discussion thread](https://forums.swift.org/t/proposal-custom-operators/2046) ## Motivation @@ -421,7 +421,7 @@ precedence Multiplicative > Additive precedence Exponentiative > Multiplicative ``` -Appearence of precedence group name in any of these "declarations" would mean declaration of the precedence group. +Appearance of precedence group name in any of these "declarations" would mean declaration of the precedence group. Precedence relationship declaration would only allow `>` sign for consistency. Limitations on connecting unrelated imported groups could still hold. @@ -432,7 +432,7 @@ It would make each operator define precedence relationships. The graph of relationships would be considerably larger and less understandable in this case. -Precedence groups concept would still be present, but it would make one operator in each group "priveleged": +Precedence groups concept would still be present, but it would make one operator in each group "privileged": ```swift precedence - = + diff --git a/proposals/0078-rotate-algorithm.md b/proposals/0078-rotate-algorithm.md index bc400be476..23a82ae469 100644 --- a/proposals/0078-rotate-algorithm.md +++ b/proposals/0078-rotate-algorithm.md @@ -2,16 +2,16 @@ * Proposal: [SE-0078](0078-rotate-algorithm.md) * Authors: [Nate Cook](https://github.com/natecook1000), [Sergey Bolshedvorsky](https://github.com/bolshedvorsky) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000165.html) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/f5936651da1a08e2335a4991831db61da29aba15/proposals/0078-rotate-algorithm.md), [2](https://github.com/apple/swift-evolution/blob/8d45024ed7baacce94e22080d74f136bebc5c075/proposals/0078-rotate-algorithm.md) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Returned for revision** +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/f5936651da1a08e2335a4991831db61da29aba15/proposals/0078-rotate-algorithm.md), [2](https://github.com/swiftlang/swift-evolution/blob/8d45024ed7baacce94e22080d74f136bebc5c075/proposals/0078-rotate-algorithm.md) +* Review: ([pitch](https://forums.swift.org/t/proposal-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/491)) ([review](https://forums.swift.org/t/review-se-0078-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/2440)) ([return for revision](https://forums.swift.org/t/review-se-0078-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/2440/3)) ([immediate deferral](https://forums.swift.org/t/deferred-se-0078-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/2744)) ([return for revision #2](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction This proposal is to add rotation and in-place reversing methods to Swift's standard library collections. -[Swift-evolution thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002213.html), [Proposal review feedback](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160502/016642.html) +[Swift-evolution thread](https://forums.swift.org/t/proposal-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/491), [Proposal review feedback](https://forums.swift.org/t/review-se-0078-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/2440/3) ## Motivation diff --git a/proposals/0079-upgrade-self-from-weak-to-strong.md b/proposals/0079-upgrade-self-from-weak-to-strong.md index bcaf20abc8..f5fb30488a 100644 --- a/proposals/0079-upgrade-self-from-weak-to-strong.md +++ b/proposals/0079-upgrade-self-from-weak-to-strong.md @@ -3,7 +3,8 @@ * Proposal: [SE-0079](0079-upgrade-self-from-weak-to-strong.md) * Author: [Evan Maloney](https://github.com/emaloney) * Review Manager: TBD -* Status: **Deferred** +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift#15306](https://github.com/apple/swift/pull/15306) ## Introduction @@ -138,7 +139,7 @@ guard let `self` = self else { } ``` -Apple’s Chris Lattner has stated that “[this is a compiler bug](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160118/007425.html)”. +Apple’s Chris Lattner has stated that “[this is a compiler bug](https://forums.swift.org/t/allowing-guard-let-self-self-else-for-weakly-captured-self-in-a-closure/931/12)”. Therefore, we should not rely on this “feature” to work in the future, because the bug will (presumably) be fixed eventually. @@ -150,12 +151,12 @@ Although the alternate proposal received a favorable response from the Swift Evo ## Citations -Variations on this proposal were discussed earlier in the following [swift-evolution](https://lists.swift.org/mailman/listinfo/swift-evolution) threads: +Variations on this proposal were discussed earlier in the following [swift-evolution](https://forums.swift.org/c/evolution/18) threads: -- [Wanted: syntactic sugar for [weak self] callbacks](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160201/008713.html) -- [Allowing `guard let self = self else { … }` for weakly captured self in a closure.](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160201/009023.html) -- [[Draft Proposal] A simplified notation for avoiding the weak/strong dance with closure capture lists](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160201/009241.html) -- [[Proposal Update 1] A simplified notation for avoiding the weak/strong dance with closure capture lists](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160208/009972.html) -- [[Proposal] Allow upgrading weak self to strong self by assignment](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010691.html) -- [[Proposal] Allow using optional binding to upgrade self from a weak to strong reference](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160215/010759.html) -- [[Last Call] Allow using optional binding to upgrade self from a weak to strong reference](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160222/010904.html) +- [Wanted: syntactic sugar for [weak self] callbacks](https://forums.swift.org/t/wanted-syntactic-sugar-for-weak-self-callbacks/1274) +- [Allowing `guard let self = self else { … }` for weakly captured self in a closure.](https://forums.swift.org/t/allowing-guard-let-self-self-else-for-weakly-captured-self-in-a-closure/931) +- [[Draft Proposal] A simplified notation for avoiding the weak/strong dance with closure capture lists](https://forums.swift.org/t/draft-proposal-a-simplified-notation-for-avoiding-the-weak-strong-dance-with-closure-capture-lists/1332) +- [[Proposal Update 1] A simplified notation for avoiding the weak/strong dance with closure capture lists](https://forums.swift.org/t/proposal-update-1-a-simplified-notation-for-avoiding-the-weak-strong-dance-with-closure-capture-lists/1415) +- [[Proposal] Allow upgrading weak self to strong self by assignment](https://forums.swift.org/t/proposal-allow-upgrading-weak-self-to-strong-self-by-assignment/1496) +- [[Proposal] Allow using optional binding to upgrade self from a weak to strong reference](https://forums.swift.org/t/proposal-allow-using-optional-binding-to-upgrade-self-from-a-weak-to-strong-reference/1509) +- [[Last Call] Allow using optional binding to upgrade self from a weak to strong reference](https://forums.swift.org/t/last-call-allow-using-optional-binding-to-upgrade-self-from-a-weak-to-strong-reference/1542) diff --git a/proposals/0080-failable-numeric-initializers.md b/proposals/0080-failable-numeric-initializers.md index 880c1676bd..f03aeec441 100644 --- a/proposals/0080-failable-numeric-initializers.md +++ b/proposals/0080-failable-numeric-initializers.md @@ -2,9 +2,9 @@ * Proposal: [SE-0080](0080-failable-numeric-initializers.md) * Author: [Matthew Johnson](https://github.com/anandabits) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Accepted with revisions** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000150.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0080-failable-numeric-conversion-initializers/2578) * Bug: [SR-1491](https://bugs.swift.org/browse/SR-1491) ## Introduction @@ -13,7 +13,7 @@ Swift numeric types all currently have a family of conversion initializers. In This proposal is to add a new family of conversion initializers to all numeric types that either complete successfully without loss of information or return nil. -Swift-evolution thread: [Proposal: failable numeric conversion initializers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151130/000623.html) +Swift-evolution thread: [Proposal: failable numeric conversion initializers](https://forums.swift.org/t/proposal-failable-numeric-conversion-initializers/216) ## Motivation diff --git a/proposals/0081-move-where-expression.md b/proposals/0081-move-where-expression.md index 91facde806..700590d4cb 100644 --- a/proposals/0081-move-where-expression.md +++ b/proposals/0081-move-where-expression.md @@ -2,16 +2,16 @@ * Proposal: [SE-0081](0081-move-where-expression.md) * Authors: [David Hart](https://github.com/hartbit), [Robert Widmann](https://github.com/CodaFi), [Pyry Jahkola](https://github.com/pyrtsa) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000161.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0081-move-where-clause-to-end-of-declaration/2685) * Bug: [SR-1561](https://bugs.swift.org/browse/SR-1561) ## Introduction This proposal suggests moving the `where` clause to the end of the declaration syntax, but before the body, for readability reasons. It has been discussed at length on the following swift-evolution thread: -[\[Pitch\] Moving where Clauses Out Of Parameter Lists](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160404/014309.html) +[\[Pitch\] Moving where Clauses Out Of Parameter Lists](https://forums.swift.org/t/pitch-moving-where-clauses-out-of-parameter-lists/2076) ## Motivation diff --git a/proposals/0082-swiftpm-package-edit.md b/proposals/0082-swiftpm-package-edit.md index ee6ac06e6d..5e3e24bfed 100644 --- a/proposals/0082-swiftpm-package-edit.md +++ b/proposals/0082-swiftpm-package-edit.md @@ -3,8 +3,8 @@ * Proposal: [SE-0082](0082-swiftpm-package-edit.md) * Author: [Daniel Dunbar](https://github.com/ddunbar) * Review Manager: [Anders Bertelrud](https://github.com/abertelrud) -* Status: **Accepted** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160509/017038.html) +* Status: **Implemented (Swift 3.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0082-package-manager-editable-packages/2540) ## Introduction @@ -15,9 +15,9 @@ those sources, and add a new feature for allowing iterative development. These features are tightly interrelated, which is why they are combined into one proposal. -[Proposal Announcement](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/015686.html) +[Proposal Announcement](https://forums.swift.org/t/rfc-swift-package-manager-editable-packages-proposal/2333) -[Review announcement](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160502/016502.html) +[Review announcement](https://forums.swift.org/t/review-se-0082-package-manager-editable-packages/2450) ## Motivation @@ -70,7 +70,7 @@ intended user action with the current contents of the tree. Our proposed solution is as follows: -1. Move the default location for checked depencency sources to be "hidden" (an +1. Move the default location for checked dependency sources to be "hidden" (an implementation detail). The package manager build system will by default try to ensure that any normal build always runs against the exact sources specified by the tag which was selected by dependency resolution. @@ -81,19 +81,19 @@ Our proposed solution is as follows: If a such an editable package is present in `Packages`, then `swift build` will always use the exact sources in this directory to build, regardless of - it's state, git repository status, tags, or the tag desired by dependency + its state, git repository status, tags, or the tag desired by dependency resolution. In other words, this will "just build" against the sources that are present. When an editable package is present, it will be used to satisfy all instances - of that Package in the depencency graph. It should be possible to edit all, + of that Package in the dependency graph. It should be possible to edit all, some, or none of the packages in a dependency graph, without restriction. This solution is intended to directly address the desired behaviors of the package manager: * By hiding the sources by default, we minimize the distractions in the common - case where a user is programming against a known, well-establised, library + case where a user is programming against a known, well-established, library they do not need to modify. * By adding a new, explicit workflow for switching to an "editable" package, we diff --git a/proposals/0083-remove-bridging-from-dynamic-casts.md b/proposals/0083-remove-bridging-from-dynamic-casts.md index b91191058d..eb64f9e2b1 100644 --- a/proposals/0083-remove-bridging-from-dynamic-casts.md +++ b/proposals/0083-remove-bridging-from-dynamic-casts.md @@ -2,9 +2,9 @@ * Proposal: [SE-0083](0083-remove-bridging-from-dynamic-casts.md) * Author: [Joe Groff](https://github.com/jckarter) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000173.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Rejected** +* Review: ([pitch](https://forums.swift.org/t/pitch-reducing-the-bridging-magic-in-dynamic-casts/2398)) ([review](https://forums.swift.org/t/review-se-0083-remove-bridging-conversion-behavior-from-dynamic-casts/2544)) ([deferral](https://forums.swift.org/t/deferred-to-later-in-swift-3-se-0083-remove-bridging-conversion-behavior-from-dynamic-casts/2780)) ([rejection](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction @@ -16,8 +16,6 @@ easier to understand. To replace this functionality, initializers should be added to bridged types, providing an interface for these conversions that's more consistent with the conventions of the standard library. -Swift-evolution thread: [Reducing the bridging magic in dynamic casts](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/016171.html) - ## Motivation When we introduced Swift, we wanted to provide value types for common diff --git a/proposals/0084-trailing-commas.md b/proposals/0084-trailing-commas.md index 3cae236cd2..44b0d5d01b 100644 --- a/proposals/0084-trailing-commas.md +++ b/proposals/0084-trailing-commas.md @@ -1,16 +1,16 @@ # Allow trailing commas in parameter lists and tuples * Proposal: [SE-0084](0084-trailing-commas.md) -* Authors: [Grant Paul](https://github.com/grp), [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Authors: [Grant Paul](https://github.com/grp), [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000171.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0084-allow-trailing-commas-in-parameter-lists-and-tuples/2777) ## Introduction Swift permits trailing commas after the last element in array or dictionary literal. This proposal extends that to parameters and tuples. -Original swift-evolution discussion: [Allow trailing commas in argument lists](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012112.html) +Original swift-evolution discussion: [Allow trailing commas in argument lists](https://forums.swift.org/t/draft-allow-trailing-commas-in-argument-lists/1729) ## Motivation @@ -79,13 +79,13 @@ Allowing cut and paste or commenting of entire parameter lines means simple chan > "Having used, more or less continuously for my 20 years as a professional programmer, both a language that allows trailing commas and one that does not, I come down pretty strongly on the side of allowing trailing commas (for all the reasons already stated in this thread). If it means requiring a newline after the last comma to make some people feel better about it, so be it." - John Siracusa -> "I was skeptical of this until a week or two ago, when I had some code where I ended up commenting out certain parameters. Removing the now-trailing commas was an inconvenience. So, +1 from me." - Brent Royal-Gordon +> "I was skeptical of this until a week or two ago, when I had some code where I ended up commenting out certain parameters. Removing the now-trailing commas was an inconvenience. So, +1 from me." - Becca Royal-Gordon > "We should be consistent in either accepting or rejecting trailing commas everywhere we have comma-delimited syntax. I'm in favor of accepting it, since it's popular in languages where it's supported to enable a minimal-diff style, so that changes to code don't impact neighboring lines for purely syntactic reasons. > -> If you add an argument to a function, without trailing comma support, a comma has to be added to dirty the previous line In response to observations that tuples and function arguments are somehow different from collection literals because they generally have fixed arity, I'll note that we have an very prominent variadic function in the standard library, "print", and that adding or removing values to a "print" is a very common and natural thing to do +> If you add an argument to a function, without trailing comma support, a comma has to be added to dirty the previous line In response to observations that tuples and function arguments are somehow different from collection literals because they generally have fixed arity, I'll note that we have a very prominent variadic function in the standard library, "print", and that adding or removing values to a "print" is a very common and natural thing to do > -> We've generally shied away from legislating style; see our rationale behind not requiring `self.` ([example](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/005478.html)) In languages where trailing commas are pervasively allowed, such as Perl, Python, Ruby, and modern Javascript, I haven't seen any indication that this is a major problem. Less blood has definitely been shed over it than over bracing style and other style wars." - Joe Groff +> We've generally shied away from legislating style; see our rationale behind not requiring `self.` ([example](https://forums.swift.org/t/rejected-se-0009-require-self-for-accessing-instance-members/930)) In languages where trailing commas are pervasively allowed, such as Perl, Python, Ruby, and modern Javascript, I haven't seen any indication that this is a major problem. Less blood has definitely been shed over it than over bracing style and other style wars." - Joe Groff ## Impact on Existing Code @@ -95,4 +95,4 @@ The acceptance of SE-0084 will not affect existing code. * Chris Lattner: A narrower way to solve the same problem would be to allow a comma before the `)`, but *only* when there is a newline between them. -* Vlad S suggests introducing "newlines as separators for any comma-separated list, not limited by funcs/typles but also array/dicts/generic type list etc." +* Vlad S suggests introducing "newlines as separators for any comma-separated list, not limited by funcs/tuples but also array/dicts/generic type list, etc." diff --git a/proposals/0085-package-manager-command-name.md b/proposals/0085-package-manager-command-name.md index 50f564df41..44c299dbe5 100644 --- a/proposals/0085-package-manager-command-name.md +++ b/proposals/0085-package-manager-command-name.md @@ -1,11 +1,11 @@ # Package Manager Command Names * Proposal: [SE-0085](0085-package-manager-command-name.md) -* Authors: [Rick Ballard](https://github.com/rballard), [Daniel Dunbar](http://github.com/ddunbar) -* Review Manager: [Daniel Dunbar](http://github.com/ddunbar) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/017728.html) -* Pull Request: [apple/swift-package-manager#364](https://github.com/apple/swift-package-manager/pull/364) +* Authors: [Rick Ballard](https://github.com/rballard), [Daniel Dunbar](https://github.com/ddunbar) +* Review Manager: [Daniel Dunbar](https://github.com/ddunbar) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/review-se-0085-package-manager-command-names/2530/6) +* Implementation: [apple/swift-package-manager#364](https://github.com/apple/swift-package-manager/pull/364) ## Note @@ -20,9 +20,7 @@ and `swift test`, we will introduce a new `swift package` command with multiple subcommands. `swift build` and `swift test` will remain as top-level commands due to their frequency of use. -[Swift Build Review Thread](https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160509/000438.html) - -[Swift Evolution Review Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160509/016931.html) +[Swift Evolution Review Thread](https://forums.swift.org/t/review-se-0085-package-manager-command-names/2530) ## Motivation @@ -37,7 +35,7 @@ these are not really flags modifying a build, and should be full commands in the The intent of this proposal is to establish a forward-looking syntax for supporting the full range of future package manager functionality in a clean, expressive, and -clear manner, without using command-line flags (which should be modifiers on a commmand) +clear manner, without using command-line flags (which should be modifiers on a command) to express commands. ## Proposed solution @@ -85,7 +83,7 @@ the new `swift package` subcommands are added, as aliases to those subcommands, for compatibility. They will be removed before Swift 3 is released. We acknowledge the possible need for a shorter version of the `swift package` -command, and believe we can revisit this to add an shorter alias for this in the +command, and believe we can revisit this to add a shorter alias for this in the future if necessary. See the alternatives section below. ## Impact on existing packages diff --git a/proposals/0086-drop-foundation-ns.md b/proposals/0086-drop-foundation-ns.md index daed527919..1fb18ea38a 100644 --- a/proposals/0086-drop-foundation-ns.md +++ b/proposals/0086-drop-foundation-ns.md @@ -3,8 +3,8 @@ * Proposal: [SE-0086](0086-drop-foundation-ns.md) * Authors: [Tony Parker](https://github.com/parkera), [Philippe Hausler](https://github.com/phausler) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000229.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0086-drop-ns-prefix-in-swift-foundation/3382) ##### Related radars or Swift bugs @@ -19,9 +19,9 @@ As part of _Swift 3 API Naming_ and the introduction of _Swift Core Libraries_, we are dropping the `NS` prefix from key Foundation types in Swift. -[Swift Evolution Discussion Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160502/016723.html) +[Swift Evolution Discussion Thread](https://forums.swift.org/t/dropping-ns-prefix-in-foundation/2494) -[Review Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160509/016934.html) +[Review Thread](https://forums.swift.org/t/review-se-0086-drop-ns-prefix-in-swift-foundation/2531) ## Motivation diff --git a/proposals/0087-lazy-attribute.md b/proposals/0087-lazy-attribute.md index 239342453f..161687367e 100644 --- a/proposals/0087-lazy-attribute.md +++ b/proposals/0087-lazy-attribute.md @@ -2,9 +2,9 @@ * Proposal: [SE-0087](0087-lazy-attribute.md) * Author: [Anton3](https://github.com/Anton3) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000172.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0087-rename-lazy-to-lazy/2778) ## Introduction @@ -16,7 +16,7 @@ struct ResourceManager { } ``` -Swift-evolution thread: [link to the discussion thread for that proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160502/016325.html) +Swift-evolution thread: [link to the discussion thread for that proposal](https://forums.swift.org/t/idea-make-lazy-an-attribute/2424) ## Motivation diff --git a/proposals/0088-libdispatch-for-swift3.md b/proposals/0088-libdispatch-for-swift3.md index 14da842372..19bfd9d3bc 100644 --- a/proposals/0088-libdispatch-for-swift3.md +++ b/proposals/0088-libdispatch-for-swift3.md @@ -2,10 +2,10 @@ * Proposal: [SE-0088](0088-libdispatch-for-swift3.md) * Author: [Matt Wright](https://github.com/mwwa) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000163.html) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/ef372026d5f7e46848eb2a64f292328028b667b9/proposals/0088-libdispatch-for-swift3.md) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0088-modernize-libdispatch-for-swift-3-naming-conventions/2697) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/ef372026d5f7e46848eb2a64f292328028b667b9/proposals/0088-libdispatch-for-swift3.md) ## Introduction @@ -13,7 +13,7 @@ The existing libdispatch module imports the C API almost verbatim. To move towar This discussion focuses on the transformation of the existing libdispatch API. -[Review thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160509/017170.html) +[Review thread](https://forums.swift.org/t/review-se-0088-modernize-libdispatch-for-swift-3-naming-conventions/2552) ## Motivation @@ -144,7 +144,7 @@ queue.setSpecific(key: akey, value: 42) #### Work Items -The existing ```dispatch_block_*``` API group exposes functionality that produces ```dispatch_block_t``` blocks that are wrapped with additional metadata. That behaviour in C has multiple cases where this API group can be accidentally misused because the C types are ambiguously overloaded. This proposal will introduce a new explict class to cover this functionality, ```DispatchWorkItem``` that provides more explicit, safer typing. +The existing ```dispatch_block_*``` API group exposes functionality that produces ```dispatch_block_t``` blocks that are wrapped with additional metadata. That behaviour in C has multiple cases where this API group can be accidentally misused because the C types are ambiguously overloaded. This proposal will introduce a new explicit class to cover this functionality, ```DispatchWorkItem``` that provides more explicit, safer typing. ```swift class DispatchWorkItem { @@ -259,7 +259,7 @@ struct DispatchData : RandomAccessCollection, _ObjectiveCBridgeable { } ``` -This proposal will introduce new accessor methods to access the bytes in a Data object. Along with becoming iteratable, several methods will be introduced that replace the ```dispatch_data_create_map``` approach used in C: +This proposal will introduce new accessor methods to access the bytes in a Data object. Along with becoming iterable, several methods will be introduced that replace the ```dispatch_data_create_map``` approach used in C: ```swift struct DispatchData : RandomAccessCollection, _ObjectiveCBridgeable { diff --git a/proposals/0089-rename-string-reflection-init.md b/proposals/0089-rename-string-reflection-init.md index d6c7657700..e6808a3683 100644 --- a/proposals/0089-rename-string-reflection-init.md +++ b/proposals/0089-rename-string-reflection-init.md @@ -1,18 +1,18 @@ # Renaming `String.init(_: T)` * Proposal: [SE-0089](0089-rename-string-reflection-init.md) -* Authors: [Austin Zheng](https://github.com/austinzheng), [Brent Royal-Gordon](https://github.com/brentdax) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000190.html) +* Authors: [Austin Zheng](https://github.com/austinzheng), [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0089-renaming-string-init-t-t/3097) * Bug: [SR-1881](https://bugs.swift.org/browse/SR-1881) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/40aecf3647c19ae37730e39aa9e54b67fcc2be86/proposals/0089-rename-string-reflection-init.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/40aecf3647c19ae37730e39aa9e54b67fcc2be86/proposals/0089-rename-string-reflection-init.md) ## Introduction Swift's `String` type ships with a large number of initializers that take one unlabeled argument. One of these initializers, defined as `init(_: T)`, is used to create a string containing the textual representation of an object. It is very easy to write code which accidentally invokes this initializer, when one of the other synonymous initializers was desired. Such code will compile without warnings and can be very difficult to detect. -Discussion threads: [pre-proposal part 1](https://lists.swift.org/pipermail/swift-users/Week-of-Mon-20160502/001846.html), [pre-proposal part 2](https://lists.swift.org/pipermail/swift-users/Week-of-Mon-20160509/001867.html), [review thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/017881.html), [post-review thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160523/019018.html) +Discussion threads: [pre-proposal](https://forums.swift.org/t/string-initializers-and-developer-ergonomics/2507), [review thread](https://forums.swift.org/t/review-se-0089-renaming-string-init-t-t/2663), [post-review thread](https://forums.swift.org/t/returned-for-revision-se-0089-renaming-string-init-t-t/2782) ## Motivation diff --git a/proposals/0090-remove-dot-self.md b/proposals/0090-remove-dot-self.md index 22dd2436e6..c736c5069c 100644 --- a/proposals/0090-remove-dot-self.md +++ b/proposals/0090-remove-dot-self.md @@ -2,10 +2,9 @@ * Proposal: [SE-0090](0090-remove-dot-self.md) * Authors: [Joe Groff](https://github.com/jckarter), [Tanner Nelson](https://github.com/tannernelson) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000174.html) -* Revision: 2 +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Returned for revision** +* Review: ([pitch](https://forums.swift.org/t/making-self-after-type-optional/1737)) ([review](https://forums.swift.org/t/review-se-0090-remove-self-and-freely-allow-type-references-in-expressions/2664)) ([deferral](https://forums.swift.org/t/deferred-se-0090-remove-self-and-freely-allow-type-references-in-expressions/2781)) ([return for revision](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction @@ -15,8 +14,6 @@ for `T`, one must refer to the special member `T.self`. I propose allowing type references to appear freely in expressions and removing the `.self` member from the language. -Swift-evolution thread: [Making `.self` After `Type` Optional](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160307/012239.html) - ## Motivation The constructor-or-member restriction on type references exists to provide diff --git a/proposals/0091-improving-operators-in-protocols.md b/proposals/0091-improving-operators-in-protocols.md index 06dfd87b56..940e84b6c2 100644 --- a/proposals/0091-improving-operators-in-protocols.md +++ b/proposals/0091-improving-operators-in-protocols.md @@ -2,11 +2,11 @@ * Proposal: [SE-0091](0091-improving-operators-in-protocols.md) * Authors: [Tony Allevato](https://github.com/allevato), [Doug Gregor](https://github.com/DougGregor) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000232.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0091-improving-operator-requirements-in-protocols/3390) * Bug: [SR-2073](https://bugs.swift.org/browse/SR-2073) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/eaab20ed34df1dc8ba8aa07e49abc8c5fa216f3e/proposals/0091-improving-operators-in-protocols.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/eaab20ed34df1dc8ba8aa07e49abc8c5fa216f3e/proposals/0091-improving-operators-in-protocols.md) ## Introduction @@ -20,7 +20,7 @@ the conforming type implements it) and that Swift use universal lookup for operators that finds candidates both at the global scope and within types. Swift-evolution thread: -[Discussion about operators and protocols in the context of `FloatingPoint`](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/015807.html) +[Discussion about operators and protocols in the context of `FloatingPoint`](https://forums.swift.org/t/review-se-0067-enhanced-floating-point-protocols/2264/31) ## Motivation diff --git a/proposals/0092-typealiases-in-protocols.md b/proposals/0092-typealiases-in-protocols.md index b54fcb1eac..d6d3912fc7 100644 --- a/proposals/0092-typealiases-in-protocols.md +++ b/proposals/0092-typealiases-in-protocols.md @@ -2,9 +2,9 @@ * Proposal: [SE-0092](0092-typealiases-in-protocols.md) * Authors: [David Hart](https://github.com/hartbit), [Doug Gregor](https://github.com/DougGregor) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decison Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/017742.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0092-typealiases-in-protocols-and-protocol-extensions/2639) * Bug: [SR-1539](https://bugs.swift.org/browse/SR-1539) ## Introduction diff --git a/proposals/0093-slice-base.md b/proposals/0093-slice-base.md index 8dd553ffbd..361c42c159 100644 --- a/proposals/0093-slice-base.md +++ b/proposals/0093-slice-base.md @@ -3,9 +3,9 @@ * Proposal: [SE-0093](0093-slice-base.md) * Author: [Max Moiseev](https://github.com/moiseev) * Review Manager: [Dave Abrahams](https://github.com/dabrahams) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160523/019109.html) -* Pull Request: [apple/swift#2929](https://github.com/apple/swift/pull/2929) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/review-se-0093-adding-a-public-base-property-to-slices/2695/4) +* Implementation: [apple/swift#2929](https://github.com/apple/swift/pull/2929) ## Introduction diff --git a/proposals/0094-sequence-function.md b/proposals/0094-sequence-function.md index dd1d6a6a30..f96272e1da 100644 --- a/proposals/0094-sequence-function.md +++ b/proposals/0094-sequence-function.md @@ -1,12 +1,12 @@ # Add sequence(first:next:) and sequence(state:next:) to the stdlib * Proposal: [SE-0094](0094-sequence-function.md) -* Authors: [Kevin Ballard](https://github.com/kballard), [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000170.html) +* Authors: [Lily Ballard](https://github.com/lilyball), [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0094-add-sequence-initial-next-and-sequence-state-next-to-the-stdlib/2775) * Bug: [SR-1622](https://bugs.swift.org/browse/SR-1622) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/7d220a152a681e28761493c7d9781dd867a04cf7/proposals/0094-sequence-function.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7d220a152a681e28761493c7d9781dd867a04cf7/proposals/0094-sequence-function.md) * Previous Proposal: [SE-0045](0045-scan-takewhile-dropwhile.md) ## Introduction @@ -17,14 +17,13 @@ applications of a closure to an initial value or a mutable state. Swift-evolution thread: -[Discussion thread topic for SE-0045](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160229/011923.html) +[Discussion thread topic for SE-0045](https://forums.swift.org/t/proposal-add-scan-takewhile-dropwhile-and-iterate-to-the-stdlib/806/6) - -[Initial Discussion](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006193.html) +[Initial Discussion](https://forums.swift.org/t/proposal-add-scan-takewhile-dropwhile-and-iterate-to-the-stdlib/806/15) ## Motivation -[SE-0045](0045-scan-takewhile-dropwhile.md), originally proposed `iterate(_:apply:)` (see [SE-0045r1](https://github.com/apple/swift-evolution/blob/dd0a39dd051b11e4460accad5af0e74223533e95/proposals/0045-scan-takewhile-dropwhile.md)), a method that +[SE-0045](0045-scan-takewhile-dropwhile.md), originally proposed `iterate(_:apply:)` (see [SE-0045r1](https://github.com/swiftlang/swift-evolution/blob/dd0a39dd051b11e4460accad5af0e74223533e95/proposals/0045-scan-takewhile-dropwhile.md)), a method that was subsequently changed to `unfold(_:applying:)`. The proposal was accepted with modifications. The core team rejected `unfold` based on its naming. As its core utility remains unquestionably high, this proposal re-introduces the method with better, more Swift-appropriate naming. @@ -51,8 +50,8 @@ See also: * [SE-0007 Remove C-style For Loops](0007-remove-c-style-for-loops.md), * [SE-0045](0045-scan-takewhile-dropwhile.md), -* [SE-0045r1](https://github.com/apple/swift-evolution/blob/b39d653f7e3d5e982b562664343f26c826652291/proposals/0045-scan-takewhile-dropwhile.md), -* [SE-0045r3](https://github.com/apple/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0045-scan-takewhile-dropwhile.md) +* [SE-0045r1](https://github.com/swiftlang/swift-evolution/blob/b39d653f7e3d5e982b562664343f26c826652291/proposals/0045-scan-takewhile-dropwhile.md), +* [SE-0045r3](https://github.com/swiftlang/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0045-scan-takewhile-dropwhile.md) ## Detailed design diff --git a/proposals/0095-any-as-existential.md b/proposals/0095-any-as-existential.md index f5436ae113..d8379f45f6 100644 --- a/proposals/0095-any-as-existential.md +++ b/proposals/0095-any-as-existential.md @@ -2,20 +2,20 @@ * Proposal: [SE-0095](0095-any-as-existential.md) * Authors: [Adrian Zubarev](https://github.com/DevAndArtist), [Austin Zheng](https://github.com/austinzheng) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000198.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0095-replace-protocol-p1-p2-syntax-with-p1-p2-syntax/3198) * Bug: [SR-1938](https://bugs.swift.org/browse/SR-1938) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/a4356fee94c06181715fad83aa61e923eb73f8ec/proposals/0095-any-as-existential.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/a4356fee94c06181715fad83aa61e923eb73f8ec/proposals/0095-any-as-existential.md) ## Introduction The current `protocol<>` construct, which defines an existential type consisting of zero or more protocols, should be replaced by an infix `&` type operator joining bare protocol type names. Discussion threads: -[pre-proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/018109.html), -[review thread 1](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021713.html), -[return for revision thread](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000182.html) +[pre-proposal](https://forums.swift.org/t/pitch-rename-protocol-to-any/2687), +[review thread 1](https://forums.swift.org/t/review-se-0095-replace-protocol-p1-p2-syntax-with-any-p1-p2/3081), +[return for revision thread](https://forums.swift.org/t/returned-for-revision-se-0095-replace-protocol-p1-p2-syntax-with-any-p1-p2/2855) ## Motivation @@ -72,5 +72,5 @@ The original proposal suggested replacing `protocol<>` with either `Any<>` or `a ## Acknowledgements -[Matthew Johnson](https://github.com/anandabits) and [Brent Royal-Gordon](https://github.com/brentdax) provided valuable input which helped shape the first version of this proposal. +[Matthew Johnson](https://github.com/anandabits) and [Becca Royal-Gordon](https://github.com/beccadax) provided valuable input which helped shape the first version of this proposal. diff --git a/proposals/0096-dynamictype.md b/proposals/0096-dynamictype.md index 4cfa5647da..bc7a45ba8d 100644 --- a/proposals/0096-dynamictype.md +++ b/proposals/0096-dynamictype.md @@ -2,9 +2,9 @@ * Proposal: [SE-0096](0096-dynamictype.md) * Author: [Erica Sadun](https://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000180.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0098-converting-dynamictype-from-a-property-to-an-operator/2853) * Bug: [SR-2218](https://bugs.swift.org/browse/SR-2218) ## Introduction @@ -12,7 +12,7 @@ This proposal establishes `dynamicType` as a named operator rather than a property. Swift-evolution thread: -[RFC: didset and willset](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/017959.html) +[RFC: didset and willset](https://forums.swift.org/t/rfc-didset-and-willset/2669) ## Motivation @@ -38,7 +38,7 @@ At this time, this operation cannot be written as a stdlib feature and it will b ## Impact on Existing Code -Adopting this proposal will break code and require migration support. The postfix property syntax must change to a operator call. +Adopting this proposal will break code and require migration support. The postfix property syntax must change to an operator call. ## Alternatives Considered diff --git a/proposals/0097-negative-attributes.md b/proposals/0097-negative-attributes.md index e6ba2931c1..6a902d535a 100644 --- a/proposals/0097-negative-attributes.md +++ b/proposals/0097-negative-attributes.md @@ -2,9 +2,9 @@ * Proposal: [SE-0097](0097-negative-attributes.md) * Author: [Erica Sadun](https://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000181.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0097-normalizing-naming-for-negative-attributes/2854) ## Introduction @@ -13,7 +13,7 @@ that replaces property names starting with `no` with adjectives starting with `non`. Swift-evolution thread: -[RFC: didset and willset](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/017959.html) +[RFC: didset and willset](https://forums.swift.org/t/rfc-didset-and-willset/2669) ## Motivation diff --git a/proposals/0098-didset-capitalization.md b/proposals/0098-didset-capitalization.md index 035f7d93bc..52e9279924 100644 --- a/proposals/0098-didset-capitalization.md +++ b/proposals/0098-didset-capitalization.md @@ -2,16 +2,16 @@ * Proposal: [SE-0098](0098-didset-capitalization.md) * Author: [Erica Sadun](https://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000179.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0098-lowercase-didset-and-willset-for-more-consistent-keyword-casing/2852) ## Introduction This proposal adopts consistent conjoined keyword lowercasing. Swift-evolution thread: -[RFC: didset and willset](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/017959.html) +[RFC: didset and willset](https://forums.swift.org/t/rfc-didset-and-willset/2669) ## Motivation diff --git a/proposals/0099-conditionclauses.md b/proposals/0099-conditionclauses.md index 8ad5140caf..95ca694357 100644 --- a/proposals/0099-conditionclauses.md +++ b/proposals/0099-conditionclauses.md @@ -3,16 +3,16 @@ * Proposal: [SE-0099](0099-conditionclauses.md) * Authors: [Erica Sadun](https://github.com/erica), [Chris Lattner](https://github.com/lattner) * Review Manager: [Joe Groff](https://github.com/jckarter) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](#rationale) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/83053c5f5395987caf2ecb3830a5cd8dc6213237/proposals/0099-conditionclauses.md) +* Status: **Implemented (Swift 3.0)** +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/83053c5f5395987caf2ecb3830a5cd8dc6213237/proposals/0099-conditionclauses.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-making-where-and-interchangeable-in-guard-conditions/2702)), ([review](https://forums.swift.org/t/review-se-0099-restructuring-condition-clauses/2808)), ([acceptance](https://forums.swift.org/t/accepted-with-revision-se-0099-restructuring-condition-clauses/2921)) ## Introduction -Swift condition clauses appear in `guard`, `if`, and `while` statements. This proposal re-architects the condition grammar to enable an arbitrary mix of Boolean expressions, `let` conditions (which test and unwrap optionals), general `case` clauses for arbitrary pattern matching, and availability tests. It removes `where` clauses from optional binding conditions and case conditions, and eliminates gramatical ambiguity by using commas for separation between clauses instead of using them both to separate clauses and terms within each clause. These modifications streamline Swift's syntax and alleviate the situation where many Swift developers don't know they can use arbitrary Boolean conditions after a value binding. +Swift condition clauses appear in `guard`, `if`, and `while` statements. This proposal re-architects the condition grammar to enable an arbitrary mix of Boolean expressions, `let` conditions (which test and unwrap optionals), general `case` clauses for arbitrary pattern matching, and availability tests. It removes `where` clauses from optional binding conditions and case conditions, and eliminates grammatical ambiguity by using commas for separation between clauses instead of using them both to separate clauses and terms within each clause. These modifications streamline Swift's syntax and alleviate the situation where many Swift developers don't know they can use arbitrary Boolean conditions after a value binding. Swift-evolution thread: -[\[Pitch\] making where and , interchangeable in guard conditions](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/018352.html) +[\[Pitch\] making where and , interchangeable in guard conditions](https://forums.swift.org/t/pitch-making-where-and-interchangeable-in-guard-conditions/2702) ## Motivation diff --git a/proposals/0100-add-sequence-based-init-and-merge-to-dictionary.md b/proposals/0100-add-sequence-based-init-and-merge-to-dictionary.md index c75df238b8..441c7ca533 100644 --- a/proposals/0100-add-sequence-based-init-and-merge-to-dictionary.md +++ b/proposals/0100-add-sequence-based-init-and-merge-to-dictionary.md @@ -3,13 +3,13 @@ * Proposal: [SE-0100](0100-add-sequence-based-init-and-merge-to-dictionary.md) * Author: [Nate Cook](https://github.com/natecook1000) * Review Manager: TBD -* Status: **Deferred** +* Status: **Withdrawn** ## Introduction The `Dictionary` type should allow initialization from a sequence of `(Key, Value)` tuples and offer methods that merge a sequence of `(Key, Value)` tuples into a new or existing dictionary, using a closure to combine values for duplicate keys. -Swift-evolution thread: [First message of thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160104/006124.html), [Initial proposal draft](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160111/006665.html) +Swift-evolution thread: [First message of thread](https://forums.swift.org/t/map-like-operation-that-returns-a-dictionary/999), [Initial proposal draft](https://forums.swift.org/t/map-like-operation-that-returns-a-dictionary/999/18) ## Motivation diff --git a/proposals/0101-standardizing-sizeof-naming.md b/proposals/0101-standardizing-sizeof-naming.md index 71284326ff..ec656c098b 100644 --- a/proposals/0101-standardizing-sizeof-naming.md +++ b/proposals/0101-standardizing-sizeof-naming.md @@ -1,10 +1,10 @@ # Reconfiguring `sizeof` and related functions into a unified `MemoryLayout` struct * Proposal: [SE-0101](0101-standardizing-sizeof-naming.md) -* Authors: [Erica Sadun](http://github.com/erica), [Dave Abrahams](https://github.com/dabrahams) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000244.html) +* Authors: [Erica Sadun](https://github.com/erica), [Dave Abrahams](https://github.com/dabrahams) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0101-reconfiguring-sizeof-and-related-functions-into-a-unified-memorylayout-struct/3477) ## Introduction @@ -12,14 +12,14 @@ This proposal addresses `sizeof`, `sizeofValue`, `strideof`, `strideofValue`, `a Review 1: -* [Swift Evolution Review Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021527.html) -* [Original Proposal](https://github.com/apple/swift-evolution/blob/26e1e5b546b13fb66ee8877ad7018a7856e467ca/proposals/0101-standardizing-sizeof-naming.md) +* [Swift Evolution Review Thread](https://forums.swift.org/t/review-se-0101-rename-sizeof-and-related-functions-to-comply-with-api-guidelines/3060) +* [Original Proposal](https://github.com/swiftlang/swift-evolution/blob/26e1e5b546b13fb66ee8877ad7018a7856e467ca/proposals/0101-standardizing-sizeof-naming.md) Prior Discussions: -* Swift Evolution Pitch: [\[Pitch\] Renaming sizeof, sizeofValue, strideof, strideofValue](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160530/019884.html) -* [Earlier Discussions](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160425/016042.html) -* [SE-0101 Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021527.html) +* Swift Evolution Pitch: [\[Pitch\] Renaming sizeof, sizeofValue, strideof, strideofValue](https://forums.swift.org/t/pitch-renaming-sizeof-sizeofvalue-strideof-strideofvalue/2857) +* [Earlier Discussions](https://forums.swift.org/t/trial-balloon-conforming-sizeof-sizeofvalue-etc-to-naming-guidelines/2388) +* [SE-0101 Review](https://forums.swift.org/t/review-se-0101-rename-sizeof-and-related-functions-to-comply-with-api-guidelines/3060) ## Motivation @@ -146,7 +146,7 @@ According to Joe Groff, concerns about existential values (it's illegal to ask f This proposal uses `` / `T.Type` to reflect Swift's current implementation. -**Note:** There is a [known bug](https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160530/002150.html) (cite D. Gregor) that does not enforce `.self` when used with `sizeof`, allowing `sizeof(UInt)`. This call should be `sizeof(UInt.self)`. This proposal is written as if the bug were resolved without relying on adoption of [SE-0090](0090-remove-dot-self.md). +**Note:** There is a [known bug](https://forums.swift.org/t/delaying-the-enforcement-of-self-out-of-swift-3/2862) (cite D. Gregor) that does not enforce `.self` when used with `sizeof`, allowing `sizeof(UInt)`. This call should be `sizeof(UInt.self)`. This proposal is written as if the bug were resolved without relying on adoption of [SE-0090](0090-remove-dot-self.md). ## Impact on Existing Code diff --git a/proposals/0102-noreturn-bottom-type.md b/proposals/0102-noreturn-bottom-type.md index 42c1b3ff51..4a91344483 100644 --- a/proposals/0102-noreturn-bottom-type.md +++ b/proposals/0102-noreturn-bottom-type.md @@ -2,9 +2,9 @@ * Proposal: [SE-0102](0102-noreturn-bottom-type.md) * Author: [Joe Groff](https://github.com/jckarter) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000205.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0102-remove-noreturn-attribute-and-introduce-an-empty-never-type/3213) * Bug: [SR-1953](https://bugs.swift.org/browse/SR-1953) ## Introduction @@ -15,9 +15,9 @@ uninhabited type. Swift-evolution threads: -- [SE-0097: Normalizing naming for "negative" attributes](https://lists.swift.org/pipermail/swift-evolution-announce/2016-May/000167.html) +- [SE-0097: Normalizing naming for "negative" attributes](https://forums.swift.org/t/review-se-0097-normalizing-naming-for-negative-attributes/2746) was the review discussion from which this proposal arose. -- [Change @noreturn to unconstructible return type](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160530/020140.html) +- [Change @noreturn to unconstructible return type](https://forums.swift.org/t/draft-change-noreturn-to-unconstructible-return-type/2888) ## Motivation @@ -66,7 +66,7 @@ The `@noreturn` attribute is removed from the language. Where `@noreturn` is currently used to exempt nonterminating code paths from control flow requirements such as exiting a `guard...else` clause or `return`-ing from a non-`Void` function, that exemption is -transfered to expressions of *uninhabited type*. +transferred to expressions of *uninhabited type*. ## Detailed design diff --git a/proposals/0103-make-noescape-default.md b/proposals/0103-make-noescape-default.md index a96a75ce6a..1948030cd1 100644 --- a/proposals/0103-make-noescape-default.md +++ b/proposals/0103-make-noescape-default.md @@ -2,11 +2,11 @@ * Proposal: [SE-0103](0103-make-noescape-default.md) * Author: [Trent Nadeau](https://github.com/tanadeau) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000204.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0103-make-non-escaping-closures-the-default/3212) * Bug: [SR-1952](https://bugs.swift.org/browse/SR-1952) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/833afd64b5d24a777fe2c42800d4b4dcd52bb487/proposals/0103-make-noescape-default.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/833afd64b5d24a777fe2c42800d4b4dcd52bb487/proposals/0103-make-noescape-default.md) ## Introduction @@ -16,11 +16,11 @@ This proposal switches the default to be non-escaping and requires an `@escaping Swift-evolution threads: -* [Make non-escaping closures the default](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160530/020181.html) +* [Make non-escaping closures the default](https://forums.swift.org/t/proposal-make-non-escaping-closures-the-default/2889) ## Motivation -Per Chris Lattner [on swift-evolution](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160530/019880.html): +Per Chris Lattner [on swift-evolution](https://forums.swift.org/t/rejected-se-0097-normalizing-naming-for-negative-attributes/2854/2): > To provide some more details, this approach has the following advantages: > diff --git a/proposals/0104-improved-integers.md b/proposals/0104-improved-integers.md index 297559abd7..5dbf33b262 100644 --- a/proposals/0104-improved-integers.md +++ b/proposals/0104-improved-integers.md @@ -1,31 +1,45 @@ # Protocol-oriented integers * Proposal: [SE-0104](0104-improved-integers.md) -* Authors: [Dave Abrahams](https://github.com/dabrahams), [Dmitri Gribenko](https://github.com/gribozavr), [Maxim Moiseev](https://github.com/moiseev) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Accepted** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000206.html) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/0440700fc555a6c72abb4af807c8b79fb1bec592/proposals/0104-improved-integers.md) +* Authors: [Dave Abrahams](https://github.com/dabrahams), [Maxim Moiseev](https://github.com/moiseev) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 4.0)** +* Bug: [SR-3196](https://bugs.swift.org/browse/SR-3196) +* Previous Revisions: + [1](https://github.com/swiftlang/swift-evolution/blob/0440700fc555a6c72abb4af807c8b79fb1bec592/proposals/0104-improved-integers.md), + [2](https://github.com/swiftlang/swift-evolution/blob/957ab545e05adb94507792e7871b38e34b56a0a5/proposals/0104-improved-integers.md), + [3](https://github.com/swiftlang/swift-evolution/blob/80f57a6b7645126fe0220dcb91c19565e447d5d8/proposals/0104-improved-integers.md) +* Discussion on swift-evolution: [here](https://forums.swift.org/t/protocol-oriented-integers-take-2/4884). +* Decision notes: [Rationale](https://forums.swift.org/t/accepted-se-0104-protocol-oriented-integers/5346) + ## Introduction This proposal cleans up Swifts integer APIs and makes them more useful for generic programming. +The language has evolved in ways that affect integers APIs since the time the +original proposal was approved for Swift 3. We also attempted to implement +the proposed model in the standard library and found that some essential APIs +were missing, whereas others could be safely removed. + +Major changes to the APIs introduced by this revision are listed in a +[dedicated section](#whats-new-in-this-revision). + ## Motivation Swift's integer protocols don't currently provide a suitable basis for generic programming. See [this blog post](http://blog.krzyzanowskim.com/2015/03/01/swift_madness_of_generic_integer/) for an example of an attempt to implement a generic algorithm over integers. -The way the `Arithmetic` protocol is defined, it does not generalize to floating -point numbers and also slows down compilation by requiring every concrete -type to provide an implementation of arithmetic operators as free functions, -thus polluting the overload set. +The way the `IntegerArithmetic` protocol is defined, it does not generalize to +floating point numbers and also slows down compilation by requiring every +concrete type to provide an implementation of arithmetic operators, thus +polluting the overload set. Converting from one integer type to another is performed using the concept of the 'maximum width integer' (see `MaxInt`), which is an artificial limitation. -The very existence of `MaxInt` makes it unclear what to do, should someone +The very existence of `MaxInt` makes it unclear what to do should someone implement `Int256`, for example. Another annoying problem is the inability to use integers of different types in @@ -45,9 +59,8 @@ Currently, bit-shifting a negative number of (or too many) bits causes a trap on some platforms, which makes low-level bit manipulations needlessly dangerous and unpredictable. -Finally, the current design predates many of the improvements that came in -Swift 2, and hasn't been revised since then. - +Finally, the current design predates many of the improvements that came since +Swift 1, and hasn't been revised since then. ## Proposed solution @@ -55,29 +68,29 @@ We propose a new model that does not have above mentioned problems and is more easily extensible. ~~~~ - +--------------+ +-------------+ - +------>+ Arithmetic | | Comparable | - | | (+,-,*,/) | | (==,<,>,...)| - | +-------------++ +---+---------+ + +-------------+ +-------------+ + +------>+ Numeric | | Comparable | + | | (+,-,*) | | (==,<,>,...)| + | +------------++ +---+---------+ | ^ ^ +-------+------------+ | | -| SignedArithmetic | +-+-------+------+ -| (unary -) | | BinaryInteger | -+------+-------------+ | (words,%,...) | - ^ ++---+-----+-----+ - | +-----------^ ^ ^-----------+ - | | | | -+------+---------++ +---------+-------------+ ++------------------+ -| SignedInteger | | FixedWidthInteger | | UnsignedInteger | -| | | (bitwise,overflow,...)| | | -+---------------+-+ +-+-----------------+---+ ++------------------+ - ^ ^ ^ ^ - | | | | - | | | | - ++--------+-+ +-+-------+-+ - |Int family |-+ |UInt family|-+ - +-----------+ | +-----------+ | - +-----------+ +-----------+ +| SignedNumeric | +-+-------+-----------+ +| (unary -) | | BinaryInteger | ++------+-------------+ |(words,%,bitwise,...)| + ^ ++---+-----+----------+ + | +-----------^ ^ ^---------------+ + | | | | ++------+---------++ +---------+---------------+ +--+----------------+ +| SignedInteger | | FixedWidthInteger | | UnsignedInteger | +| | |(endianness,overflow,...)| | | ++---------------+-+ +-+--------------------+--+ +-+-----------------+ + ^ ^ ^ ^ + | | | | + | | | | + ++--------+-+ +-+-------+-+ + |Int family |-+ |UInt family|-+ + +-----------+ | +-----------+ | + +-----------+ +-----------+ ~~~~ @@ -86,41 +99,37 @@ There are several benefits provided by this model over the old one: - It allows mixing integer types in generic functions. The possibility to initialize instances of any concrete integer type with -values of any other concrete integer type enables writing functions that -operate on more than one type conforming to `BinaryInteger`, such as heterogeneous -comparisons or bit shifts, described later. + values of any other concrete integer type enables writing functions that + operate on more than one type conforming to `BinaryInteger`, such as + heterogeneous comparisons or bit shifts, described later. - It removes the overload resolution overhead. - Arithmetic and bitwise operations can now be defined as free functions -delegating work to concrete types. This approach significantly reduces the -number of overloads for those operations, which used to be defined for every -single concrete integer type. + Arithmetic and bitwise operations can now be defined as generic operators on + protocols. This approach significantly reduces the number of overloads for + those operations, which used to be defined for every single concrete integer + type. - It enables protocol sharing between integer and floating point types. - Note the exclusion of the `%` operation from `Arithmetic`. Its behavior for -floating point numbers is sufficiently different from the one for integers that -using it in generic context would lead to confusion. The `FloatingPoint` protocol -introduced by -[SE-0067](https://github.com/apple/swift-evolution/blob/0440700fc555a6c72abb4af807c8b79fb1bec592/proposals/0104-improved-integers.md) -should now refine `SignedArithmetic`. + Note the exclusion of the `%` operation from `Numeric`. Its behavior for + floating point numbers is sufficiently different from the one for integers + that using it in generic context would lead to confusion. The `FloatingPoint` + protocol introduced by [SE-0067](0067-floating-point-protocols.md) should now + refine `SignedNumeric`. - It makes future extensions possible. - The proposed model eliminates the 'largest integer type' concept previously used -to interoperate between integer types (see `toIntMax` in the current model) and -instead provides access to machine words. It also introduces the -`doubleWidthMultiply`, `doubleWidthDivide`, and `quotientAndRemainder` methods. -Together these changes can be used to provide an efficient implementation of -bignums that would be hard to achieve otherwise. - -The prototype implementation, available -[here](https://github.com/apple/swift/blob/master/test/Prototypes/Integers.swift.gyb) -contains a `DoubleWidth` generic type that uses two values of any -`FixedInteger` type to represent a value of twice the width, demonstrating the -suitability of the proposed model for generic programming. + The proposed model eliminates the 'largest integer type' concept previously + used to interoperate between integer types (see `toIntMax` in the current + model) and instead provides access to machine words. It also introduces the + `multipliedFullWidth(by:)`, `dividingFullWidth(_:)`, and + `quotientAndRemainder` methods. Together these changes can be used to provide + an efficient implementation of bignums that would be hard to achieve + otherwise. +The implementation of proposed model in the standard library is available +[in the new-integer-protocols branch][impl]. ### A note on bit shifts @@ -134,28 +143,90 @@ as shown in the examples below: - `x << -2` is equivalent to `x >> 2` -- `(1 as UInt8) >> 42)` will evaluate to `0` +- `(1 as UInt8) >> 42` will evaluate to `0` -- `(-128 as Int8) >> 42)` will evaluate to `0xff` or `-1` +- `(-128 as Int8) >> 42` will evaluate to `0xff` or `-1` In most scenarios, the right hand operand is a literal constant, and branches for handling under- and over-shift cases can be optimized away. For other cases, this proposal provides *masking shifts*, implemented by `&>>` and `&<<`. A masking shift logically preprocesses the right hand operand by masking its -bits to produce a value in the range `0...(x-1)` where `x` is the number of -bits in the left hand operand. On most architectures this masking is already -performed by the CPU's shift instructions and has no cost. Both kinds of shift +bits to produce a value in the range `0...(x-1)` where `x` is the number of bits +in the left hand operand. On most architectures this masking is already +performed by the CPU's shift instructions and has no cost. Both kinds of shift avoid undefined behavior and produce uniform semantics across architectures. ## Detailed design +### What's new in this revision + +* Shift operators were reorganized + ([pull request](https://github.com/apple/swift/pull/11044)) + + - Masking shift operators were moved from `BinaryInteger` to + `FixedWidthInteger`. + - Non-masking shift operators were moved from `FixedWidthInteger` to + `BinaryInteger`. + + **Rationale:** attempts to implement masking shifts for + arbitrary-precision integers have failed because + the + [semantics aren't clear](https://gist.github.com/xwu/d68baefaae9e9291d2e65bd12ad51be2#semantics-of-masking-shifts-for-arbitrary-precision-integers). + Attempts to clarify the definition of the semantics of masking shift + for `BinaryInteger` have failed, indicating that the operation + doesn't actually make sense outside of `FixedWidthInteger`. + +* `ArithmeticOverflow` was removed in favor of using a simple `Bool`. + + **Rationale:** the enum + [proves to have poor ergonomics](https://gist.github.com/xwu/d68baefaae9e9291d2e65bd12ad51be2#arithmeticoverflow). + It only appears as part of a result tuple, where a label already + helps counteract the readability deficit usually caused by + returning an un-labeled `Bool`. + +* `BinaryInteger`'s initializers from floating point values were + changed from: + + ```swift + init?(exactly source: T) + init(_ source: T) + ``` + + to: + + ```swift + init?(exactly source: T) + init(_ source: T) + ``` + + **Rationale:** Attempts to implement these initializers for + arbitrary models of `FloatingPoint` have failed + (see + [here](https://github.com/apple/swift/blob/826f8daf4a25657965f65cbb7343e751c76fe2e1/stdlib/public/core/DoubleWidth.swift.gyb#L100-L106) and + [here](https://github.com/apple/swift/blob/826f8daf4a25657965f65cbb7343e751c76fe2e1/test/Prototypes/BigInt.swift#L145-L151)) + whereas they are + clearly + [implementable](https://github.com/apple/swift/blob/826f8daf4a25657965f65cbb7343e751c76fe2e1/stdlib/public/core/DoubleWidth.swift.gyb#L108-L159) for + models of `BinaryFloatingPoint`, suggesting that the operation + doesn't actually make sense outside of `BinaryFloatingPoint`. + +* `BinaryInteger`'s `init(extendingOrTruncating:)` was renamed to + `init(truncatingIfNeeded:)` + + **Rationale:** `extendingOrTruncating` emphasizes a part of the + semantics (“extending”) that is lossless and thus doesn't warrant + the implied warning of an argument label. The two use cases for + this initializer are intentional truncation and optimizing out range + checks that are known by the programmer to be un-needed. Both these + cases are better served by `truncatingIfNeeded`. + ### Protocols -#### `Arithmetic` +#### `Numeric` -The `Arithmetic` protocol declares methods backing binary arithmetic -operators—such as `+`, `-` and `*`—and their mutating counterparts. +The `Numeric` protocol declares binary arithmetic operators – such as `+`, +`-`, and `*` — and their mutating counterparts. It provides a suitable basis for arithmetic on scalars such as integers and floating point numbers. @@ -165,7 +236,7 @@ only the mutating ones are required, as default implementations of the non-mutating ones are provided by a protocol extension. The `Magnitude` associated type is able to hold the absolute value of any -possible value of `Self`. Concrete types do not have to provide a typealias for +possible value of `Self`. Concrete types do not have to provide a type alias for it, as it can be inferred from the `magnitude` property. This property can be useful in operations that are simpler to implement in terms of unsigned values, for example, printing a value of an integer, which is just printing a @@ -177,426 +248,801 @@ as its argument. ```Swift -public protocol Arithmetic : Equatable, IntegerLiteralConvertible { - /// Initializes to the value of `source` if it is representable exactly, - /// returns `nil` otherwise. +public protocol Numeric : Equatable, ExpressibleByIntegerLiteral { + /// Creates a new instance from the given integer, if it can be represented + /// exactly. + /// + /// If the value passed as `source` is not representable exactly, the result + /// is `nil`. In the following example, the constant `x` is successfully + /// created from a value of `100`, while the attempt to initialize the + /// constant `y` from `1_000` fails because the `Int8` type can represent + /// `127` at maximum: + /// + /// let x = Int8(exactly: 100) + /// // x == Optional(100) + /// let y = Int8(exactly: 1_000) + /// // y == nil + /// + /// - Parameter source: A BinaryInteger value to convert to an integer. init?(exactly source: T) - associatedtype Magnitude : Arithmetic + /// A type that can represent the absolute value of any possible value of the + /// conforming type. + associatedtype Magnitude : Numeric, Comparable + + /// The magnitude of this value. + /// + /// For any numeric value `x`, `x.magnitude` is the absolute value of `x`. + /// You can use the `magnitude` property in operations that are simpler to + /// implement in terms of unsigned values, such as printing the value of an + /// integer, which is just printing a '-' character in front of an absolute + /// value. + /// + /// let x = -200 + /// // x.magnitude == 200 + /// + /// The global `abs(_:)` function provides more familiar syntax when you need + /// to find an absolute value. In addition, because `abs(_:)` always returns + /// a value of the same type, even in a generic context, using the function + /// instead of the `magnitude` property is encouraged. + /// + /// - SeeAlso: `abs(_:)` var magnitude: Magnitude { get } - func adding(_ other: Self) -> Self - func subtracting(_ other: Self) -> Self - func multiplied(by other: Self) -> Self - func divided(by other: Self) -> Self + /// Returns the sum of the two given values. + /// + /// The sum of `lhs` and `rhs` must be representable in the same type. In the + /// following example, the result of `100 + 200` is greater than the maximum + /// representable `Int8` value: + /// + /// let x: Int8 = 10 + 21 + /// // x == 31 + /// let y: Int8 = 100 + 121 + /// // Overflow error + static func +(_ lhs: Self, _ rhs: Self) -> Self + + /// Adds the given value to this value in place. + /// + /// For example: + /// + /// var x = 15 + /// x += 7 + /// // x == 22 + static func +=(_ lhs: inout Self, rhs: Self) + + /// Returns the difference of the two given values. + /// + /// The difference of `lhs` and `rhs` must be representable in the same type. + /// In the following example, the result of `10 - 21` is less than zero, the + /// minimum representable `UInt` value: + /// + /// let x: UInt = 21 - 10 + /// // x == 11 + /// let y: UInt = 10 - 21 + /// // Overflow error + static func -(_ lhs: Self, _ rhs: Self) -> Self + + /// Subtracts the given value from this value in place. + /// + /// For example: + /// + /// var x = 15 + /// x -= 7 + /// // x == 8 + static func -=(_ lhs: inout Self, rhs: Self) + + /// Returns the product of the two given values. + /// + /// The product of `lhs` and `rhs` must be representable in the same type. In + /// the following example, the result of `10 * 50` is greater than the + /// maximum representable `Int8` value. + /// + /// let x: Int8 = 10 * 5 + /// // x == 50 + /// let y: Int8 = 10 * 50 + /// // Overflow error + static func *(_ lhs: Self, _ rhs: Self) -> Self - mutating func add(_ other: Self) - mutating func subtract(_ other: Self) - mutating func multiply(by other: Self) - mutating func divide(by other: Self) + /// Multiples this value by the given value in place. + /// + /// For example: + /// + /// var x = 15 + /// x *= 7 + /// // x == 105 + static func *=(_ lhs: inout Self, rhs: Self) } -extension Arithmetic { - public init() { self = 0 } +extension Numeric { + public static prefix func + (x: Self) -> Self { + return x + } } ``` -#### `SignedArithmetic` +#### `SignedNumeric` -The `SignedArithmetic` protocol is for numbers that can be negated. +The `SignedNumeric` protocol is for numbers that can be negated. ```Swift -public protocol SignedArithmetic : Arithmetic { - func negated() -> Self +public protocol SignedNumeric : Numeric { + /// Returns the additive inverse of this value. + /// + /// let x = 21 + /// let y = -x + /// // y == -21 + /// + /// - Returns: The additive inverse of this value. + /// + /// - SeeAlso: `negate()` + static prefix func - (_ operand: Self) -> Self + + /// Replaces this value with its additive inverse. + /// + /// The following example uses the `negate()` method to negate the value of + /// an integer `x`: + /// + /// var x = 21 + /// x.negate() + /// // x == -21 + /// + /// - SeeAlso: The unary minus operator (`-`). mutating func negate() } -extension SignedArithmetic { - public func negated() -> Self { - return Self() - self +extension SignedNumeric { + public static prefix func - (_ operand: Self) -> Self { + var result = operand + result.negate() + return result } + public mutating func negate() { - self = negated() + self = 0 - self } } ``` #### `BinaryInteger` -The `BinaryInteger` protocol is the basis for all the integer types provided by the -standard library. - -The `isEqual(to:)` and `isLess(than:)` methods provide implementations for -`Equatable` and `Comparable` protocol conformances. Similar to how arithmetic -operations are dispatched in `Arithmetic`, `==` and `<` operators for -homogeneous comparisons are implemented as generic free functions invoking the -`isEqual(to:)` and `isLess(than:)` protocol methods respectively. +The `BinaryInteger` protocol is the basis for all the integer types provided by +the standard library. -This protocol adds 4 new initializers. One of them allows to create integers -from floating point numbers, if the value is representable exactly, others -support construction from instances of any type conforming to `BinaryInteger`, using -different strategies: +This protocol adds a few new initializers. Two of them allow to create integers +from floating point numbers, others support construction from instances of any +type conforming to `BinaryInteger`, using different strategies: - - Initialze `Self` with the value, provided that the value is representable. + - Initialize `Self` with the value, provided that the value is representable. The precondition should be satisfied by the caller. - Extend or truncate the value to fit into `Self` - Clamp the value to the representable range of `Self` +`BinaryInteger` also declares bitwise and shift operators. + ```Swift -public protocol BinaryInteger: - Comparable, Arithmetic, - IntegerLiteralConvertible, CustomStringConvertible { +public protocol BinaryInteger : + Comparable, Hashable, Numeric, CustomStringConvertible, Strideable { - static var isSigned: Bool { get } + associatedtype Words : Collection // where Iterator.Element == UInt - func isEqual(to other: Self) -> Bool - func isLess(than other: Self) -> Bool + /// A Boolean value indicating whether this type is a signed integer type. + /// + /// *Signed* integer types can represent both positive and negative values. + /// *Unsigned* integer types can represent only nonnegative values. + static var isSigned: Bool { get } - /// Creates an instance of `Self` that has the exact value of `source`, - /// returns `nil` otherwise. - init?(exactly source: T) + /// Creates an integer from the given floating-point value, if it can be + /// represented exactly. + /// + /// If the value passed as `source` is not representable exactly, the result + /// is `nil`. In the following example, the constant `x` is successfully + /// created from a value of `21.0`, while the attempt to initialize the + /// constant `y` from `21.5` fails: + /// + /// let x = Int(exactly: 21.0) + /// // x == Optional(21) + /// let y = Int(exactly: 21.5) + /// // y == nil + /// + /// - Parameter source: A floating-point value to convert to an integer. + init?(exactly source: T) - /// Truncates the `source` to the closest representable value of `Self`. - init(_ source: T) + /// Creates an integer from the given floating-point value, truncating any + /// fractional part. + /// + /// Truncating the fractional part of `source` is equivalent to rounding + /// toward zero. + /// + /// let x = Int(21.5) + /// // x == 21 + /// let y = Int(-21.5) + /// // y == -21 + /// + /// If `source` is outside the bounds of this type after truncation, a + /// runtime error may occur. + /// + /// let z = UInt(-21.5) + /// // Error: ...the result would be less than UInt.min + /// + /// - Parameter source: A floating-point value to convert to an integer. + /// `source` must be representable in this type after truncation. + init(_ source: T) - /// Creates an instance of `Self` from `source` if it is representable. + /// Creates an new instance from the given integer. + /// + /// If the value passed as `source` is not representable in this type, a + /// runtime error may occur. + /// + /// let x = -500 as Int + /// let y = Int32(x) + /// // y == -500 + /// + /// // -500 is not representable as a 'UInt32' instance + /// let z = UInt32(x) + /// // Error /// - /// - Precondition: the value of `source` is representable in `Self`. + /// - Parameter source: An integer to convert. `source` must be representable + /// in this type. init(_ source: T) - /// Creates in instance of `Self` from `source` by sign-extending it - /// indefinitely and then truncating to fit `Self`. - init(extendingOrTruncating source: T) + /// Creates a new instance from the bit pattern of the given instance by + /// sign-extending or truncating to fit this type. + /// + /// When the bit width of `T` (the type of `source`) is equal to or greater + /// than this type's bit width, the result is the truncated + /// least-significant bits of `source`. For example, when converting a + /// 16-bit value to an 8-bit type, only the lower 8 bits of `source` are + /// used. + /// + /// let p: Int16 = -500 + /// // 'p' has a binary representation of 11111110_00001100 + /// let q = Int8(truncatingIfNeeded: p) + /// // q == 12 + /// // 'q' has a binary representation of 00001100 + /// + /// When the bit width of `T` is less than this type's bit width, the result + /// is *sign-extended* to fill the remaining bits. That is, if `source` is + /// negative, the result is padded with ones; otherwise, the result is + /// padded with zeros. + /// + /// let u: Int8 = 21 + /// // 'u' has a binary representation of 00010101 + /// let v = Int16(truncatingIfNeeded: u) + /// // v == 21 + /// // 'v' has a binary representation of 00000000_00010101 + /// + /// let w: Int8 = -21 + /// // 'w' has a binary representation of 11101011 + /// let x = Int16(truncatingIfNeeded: w) + /// // x == -21 + /// // 'x' has a binary representation of 11111111_11101011 + /// let y = UInt16(truncatingIfNeeded: w) + /// // y == 65515 + /// // 'y' has a binary representation of 11111111_11101011 + /// + /// - Parameter source: An integer to convert to this type. + init(truncatingIfNeeded source: T) - /// Creates in instance of `Self` containing the closest representable - /// value of `source`. + /// Creates a new instance with the representable value that's closest to the + /// given integer. + /// + /// If the value passed as `source` is greater than the maximum representable + /// value in this type, the result is the type's `max` value. If `source` is + /// less than the smallest representable value in this type, the result is + /// the type's `min` value. + /// + /// In this example, `x` is initialized as an `Int8` instance by clamping + /// `500` to the range `-128...127`, and `y` is initialized as a `UInt` + /// instance by clamping `-500` to the range `0...UInt.max`. + /// + /// let x = Int8(clamping: 500) + /// // x == 127 + /// // x == Int8.max + /// + /// let y = UInt(clamping: -500) + /// // y == 0 + /// + /// - Parameter source: An integer to convert to this type. init(clamping source: T) - /// Returns the n-th word, counting from the least significant to most - /// significant, of the underlying representation of `self`. - /// Should return `0` for positive numbers and `~0` for negative ones if `n` - /// is greater than the number of words in current representation of `self`. - func word(at n: Int) -> UInt + /// The collection of words in two's complement representation of the value, + /// from the least significant to most. + var words: Words { get } - /// The number of bits in current representation of `self` - /// Will be constant for fixed-width integer types. + /// The number of bits in the current binary representation of this value. + /// + /// This property is a constant for instances of fixed-width integer + /// types. var bitWidth : Int { get } - /// The number of bits required to represent the value of `self` in a signed - /// type using two's complement representation. The minimum value for this - /// property should naturally be 1. - var minimumSignedRepresentationBitWidth: Int { get } + /// The number of trailing zeros in this value's binary representation. + /// + /// For example, in a fixed-width integer type with a `bitWidth` value of 8, + /// the number -8 has three trailing zeros. + /// + /// let x = Int8(bitPattern: 0b1111_1000) + /// // x == -8 + /// // x.trailingZeroBits == 3 + var trailingZeroBits: Int { get } - /// Returns the remainder of division of `self` by `other`. - func remainder(dividingBy other: Self) -> Self - /// Replaces `self` with the remainder of division of `self` by `other`. - mutating func formRemainder(dividingBy other: Self) + /// Returns the quotient of dividing the first value by the second. + /// + /// For integer types, any remainder of the division is discarded. + /// + /// let x = 21 / 5 + /// // x == 4 + static func /(_ lhs: Self, _ rhs: Self) -> Self - /// Returns a pair of values, containing the quotient and the remainder of - /// division of `self` by `other`. + /// Divides this value by the given value in place. + /// + /// For example: + /// + /// var x = 15 + /// x /= 7 + /// // x == 2 + static func /=(_ lhs: inout Self, rhs: Self) + + /// Returns the remainder of dividing the first value by the second. + /// + /// The result has the same sign as `lhs` and is less than `rhs.magnitude`. + /// + /// let x = 22 % 5 + /// // x == 2 + /// let y = 22 % -5 + /// // y == 2 + /// let z = -22 % -5 + /// // z == -2 + /// + /// - Parameters: + /// - lhs: The value to divide. + /// - rhs: The value to divide `lhs` by. `rhs` must not be zero. + static func %(_ lhs: Self, _ rhs: Self) -> Self + + /// Replaces this value with the remainder of itself divided by the given + /// value. For example: + /// + /// var x = 15 + /// x %= 7 + /// // x == 1 + /// + /// - Parameter rhs: The value to divide this value by. `rhs` must not be + /// zero. + /// + /// - SeeAlso: `remainder(dividingBy:)` + static func %=(_ lhs: inout Self, _ rhs: Self) + + /// Returns the inverse of the bits set in the argument. + /// + /// The bitwise NOT operator (`~`) is a prefix operator that returns a value + /// in which all the bits of its argument are flipped: Bits that are `1` in + /// the argument are `0` in the result, and bits that are `0` in the argument + /// are `1` in the result. This is equivalent to the inverse of a set. For + /// example: + /// + /// let x: UInt8 = 5 // 0b00000101 + /// let notX = ~x // 0b11111010 + /// + /// Performing a bitwise NOT operation on 0 returns a value with every bit + /// set to `1`. + /// + /// let allOnes = ~UInt8.min // 0b11111111 + /// + /// - Complexity: O(1). + static prefix func ~ (_ x: Self) -> Self + + /// Returns the result of performing a bitwise AND operation on this value + /// and the given value. + /// + /// A bitwise AND operation results in a value that has each bit set to `1` + /// where *both* of its arguments have that bit set to `1`. For example: + /// + /// let x: UInt8 = 5 // 0b00000101 + /// let y: UInt8 = 14 // 0b00001110 + /// let z = x & y // 0b00000100 + static func &(_ lhs: Self, _ rhs: Self) -> Self + static func &=(_ lhs: inout Self, _ rhs: Self) + + /// Returns the result of performing a bitwise OR operation on this value and + /// the given value. + /// + /// A bitwise OR operation results in a value that has each bit set to `1` + /// where *one or both* of its arguments have that bit set to `1`. For + /// example: + /// + /// let x: UInt8 = 5 // 0b00000101 + /// let y: UInt8 = 14 // 0b00001110 + /// let z = x | y // 0b00001111 + static func |(_ lhs: Self, _ rhs: Self) -> Self + static func |=(_ lhs: inout Self, _ rhs: Self) + + /// Returns the result of performing a bitwise XOR operation on this value + /// and the given value. + /// + /// A bitwise XOR operation, also known as an exclusive OR operation, results + /// in a value that has each bit set to `1` where *one or the other but not + /// both* of its arguments had that bit set to `1`. For example: + /// + /// let x: UInt8 = 5 // 0b00000101 + /// let y: UInt8 = 14 // 0b00001110 + /// let z = x ^ y // 0b00001011 + static func ^(_ lhs: Self, _ rhs: Self) -> Self + static func ^=(_ lhs: inout Self, _ rhs: Self) + + /// Returns the result of shifting this value's binary representation the + /// specified number of digits to the right. + static func >>(_ lhs: Self, _ rhs: RHS) -> Self + + /// Stores the result of shifting a value's binary representation the + /// specified number of digits to the right in the left-hand-side variable. + static func >>=(_ lhs: inout Self, _ rhs: RHS) + + /// Returns the result of shifting a value's binary representation the + /// specified number of digits to the left. + static func << (_ lhs: Self, _ rhs: RHS) -> Self + + /// Stores the result of shifting a value's binary representation the + /// specified number of digits to the left in the left-hand-side variable + static func <<= (_ lhs: inout Self, _ rhs: RHS) + + /// Returns the quotient and remainder of this value divided by the given + /// value. + /// + /// Use this method to calculate the quotient and remainder of a division at + /// the same time. + /// + /// let x = 1_000_000 + /// let (q, r) = x.quotientAndRemainder(dividingBy: 933) + /// // q == 1071 + /// // r == 757 /// - /// The default implementation simply invokes `divided(by:)` and - /// `remainder(dividingBy:)`, which in case of built-in types will be fused - /// into a single instruction by the compiler. + /// - Parameter rhs: The value to divide this value by. + /// - Returns: A tuple containing the quotient and remainder of this value + /// divided by `rhs`. + func quotientAndRemainder(dividingBy rhs: Self) + -> (quotient: Self, remainder: Self) + + /// Returns `-1` if this value is negative and `1` if it's positive; + /// otherwise, `0`. /// - /// Conforming types can override the default behavior in order to - /// provide a more efficient implementation. - func quotientAndRemainder(dividingBy other: Self) -> (Self, Self) + /// - Returns: The sign of this number, expressed as an integer of the same + /// type. + func signum() -> Self +} + +extension BinaryInteger { + init() { self = 0 } } ``` #### `FixedWidthInteger` -The `FixedWidthInteger` protocol adds binary bitwise operations and bit shifts -to the `BinaryInteger` protocol. +The `FixedWidthInteger` protocol adds the notion of endianness as well as static +properties for type bounds and bit width. -The `WithOverflow` family of methods is used in default implementations of -mutating arithmetic methods (see the `Arithmetic` protocol). Having these +The `ReportingOverflow` family of methods is used in default implementations of +mutating arithmetic methods (see the `Numeric` protocol). Having these methods allows the library to provide both bounds-checked and masking implementations of arithmetic operations, without duplicating code. -Bitwise binary and shift operators are implemented the same way as arithmetic -operations: a free function dispatches a call to a corresponding protocol -method. - -The `doubleWidthMultiply` and `doubleWidthDivide` methods are necessary building -blocks to implement support for integer types of a greater width such as -arbitrary-precision integers. +The `multipliedFullWidth(by:)` and `dividingFullWidth(_:)` methods are +necessary building blocks to implement support for integer types of a greater +width such as arbitrary-precision integers. ```Swift + public protocol FixedWidthInteger : BinaryInteger { - /// Returns the bit width of the underlying binary - /// representation of values of `self`. + /// The number of bits used for the underlying binary representation of + /// values of this type. + /// + /// An unsigned, fixed-width integer type can represent values from 0 through + /// `(2 ** bitWidth) - 1`, where `**` is exponentiation. A signed, + /// fixed-width integer type can represent values from + /// `-(2 ** bitWidth - 1)` through `(2 ** bitWidth - 1) - 1`. For example, + /// the `Int8` type has a `bitWidth` value of 8 and can store any integer in + /// the range `-128...127`. static var bitWidth : Int { get } - /// Returns the maximum value representable by `Self`. + /// The maximum representable integer in this type. + /// + /// For unsigned integer types, this value is `(2 ** bitWidth) - 1`, where + /// `**` is exponentiation. For signed integer types, this value is + /// `(2 ** bitWidth - 1) - 1`. static var max: Self { get } - /// Returns the minimum value representable by 'Self'. - static var min: Self { get } - - /// Adds `other` to `self` returning a pair containing the partial result - /// of addition and an overflow flag. - func addingWithOverflow( - other: Self - ) -> (partialValue: Self, overflow: ArithmeticOverflow) - - /// Subtracts `other` from `self` returning a pair containing the partial - /// result of subtraction and an overflow flag. - func subtractingWithOverflow( - other: Self - ) -> (partialValue: Self, overflow: ArithmeticOverflow) - - /// Multiplies `self` by `other` returning a pair containing the partial - /// result of multiplication and an overflow flag. - func multipliedWithOverflow( - by other: Self - ) -> (partialValue: Self, overflow: ArithmeticOverflow) - - /// Divides `self` by `other` returning a pair containing the partial - /// result of division and an overflow flag. - func dividedWithOverflow( - by other: Self - ) -> (partialValue: Self, overflow: ArithmeticOverflow) - - /// Returns the partial result of getting a remainder of division of `self` - /// by `other`, and an overflow flag. - func remainderWithOverflow( - dividingBy other: Self - ) -> (partialValue: Self, overflow: ArithmeticOverflow) - - /// Returns the result of the 'bitwise and' operation, applied - /// to `self` and `other`. - func bitwiseAnd(other: Self) -> Self - - /// Returns the result of the 'bitwise or' operation, applied - /// to `self` and `other`. - func bitwiseOr(other: Self) -> Self - - /// Returns the result of the 'bitwise exclusive or' operation, applied - /// to `self` and `other`. - func bitwiseXor(other: Self) -> Self - - /// Returns the result of shifting the binary representation - /// of `self` by `other` binary digits to the right. - func maskingShiftRight(other: Self) -> Self - - /// Returns the result of shifting the binary representation - /// of `self` by `other` binary digits to the left. - func maskingShiftLeft(other: Self) -> Self - - /// Returns a pair containing the `high` and `low` parts of the result - /// of `lhs` multiplied by `rhs`. - static func doubleWidthMultiply(_ lhs: Self, _ rhs: Self) - -> (high: Self, low: Magnitude) - - /// Returns a pair containing a quotient and a remainder of `lhs` divided by - /// `rhs`, where `lhs` is itself a pair of `high` and `low` words of a double - /// width number. - static func doubleWidthDivide( - _ lhs: (high: Self, low: Magnitude), _ rhs: Self - ) -> (quotient: Self, remainder: Self) - - - /// Returns a number of set (i.e. equal to 1) bits in the representation of - /// `self`. - var popcount: Int { get } - - /// Returns the number of leading zeros in the representation of `self`. - var leadingZeros: Int { get } -} -``` -#### Auxiliary protocols + /// The minimum representable value. + /// + /// For unsigned integer types, this value is always `0`. For signed integer + /// types, this value is `-(2 ** bitWidth - 1)`, where `**` is + /// exponentiation. + static var min: Self { get } -```Swift -public protocol UnsignedInteger : BinaryInteger { - associatedtype Magnitude : BinaryInteger -} -public protocol SignedInteger : BinaryInteger, SignedArithmetic { - associatedtype Magnitude : BinaryInteger -} -``` + /// Returns the sum of this value and the given value along with a flag + /// indicating whether overflow occurred in the operation. + /// + /// - Parameter other: The value to add to this value. + /// - Returns: A tuple containing the result of the addition along with a + /// flag indicating whether overflow occurred. If the `overflow` component + /// is `.none`, the `partialValue` component contains the entire sum. If + /// the `overflow` component is `.overflow`, an overflow occurred and the + /// `partialValue` component contains the truncated sum of this value and + /// `other`. + /// + /// - SeeAlso: `+` + func addingReportingOverflow(_ other: Self) + -> (partialValue: Self, overflow: Bool) + /// Returns the difference of this value and the given value along with a + /// flag indicating whether overflow occurred in the operation. + /// + /// - Parameter other: The value to subtract from this value. + /// - Returns: A tuple containing the result of the subtraction along with a + /// flag indicating whether overflow occurred. If the `overflow` component + /// is `.none`, the `partialValue` component contains the entire + /// difference. If the `overflow` component is `.overflow`, an overflow + /// occurred and the `partialValue` component contains the truncated + /// result of `other` subtracted from this value. + /// + /// - SeeAlso: `-` + func subtractingReportingOverflow(_ other: Self) + -> (partialValue: Self, overflow: Bool) -### Operators + /// Returns the product of this value and the given value along with a flag + /// indicating whether overflow occurred in the operation. + /// + /// - Parameter other: The value to multiply by this value. + /// - Returns: A tuple containing the result of the multiplication along with + /// a flag indicating whether overflow occurred. If the `overflow` + /// component is `.none`, the `partialValue` component contains the entire + /// product. If the `overflow` component is `.overflow`, an overflow + /// occurred and the `partialValue` component contains the truncated + /// product of this value and `other`. + /// + /// - SeeAlso: `*`, `multipliedFullWidth(by:)` + func multipliedReportingOverflow(by other: Self) + -> (partialValue: Self, overflow: Bool) -#### Arithmetic + /// Returns the quotient of dividing this value by the given value along with + /// a flag indicating whether overflow occurred in the operation. + /// + /// For a value `x`, if zero is passed as `other`, the result is + /// `(x, .overflow)`. + /// + /// - Parameter other: The value to divide this value by. + /// - Returns: A tuple containing the result of the division along with a + /// flag indicating whether overflow occurred. If the `overflow` component + /// is `.none`, the `partialValue` component contains the entire quotient. + /// If the `overflow` component is `.overflow`, an overflow occurred and + /// the `partialValue` component contains the truncated quotient. + /// + /// - SeeAlso: `/`, `dividingFullWidth(_:)` + func dividedReportingOverflow(by other: Self) + -> (partialValue: Self, overflow: Bool) -```Swift -public func + (lhs: T, rhs: T) -> T -public func += (lhs: inout T, rhs: T) -public func - (lhs: T, rhs: T) -> T -public func -= (lhs: inout T, rhs: T) -public func * (lhs: T, rhs: T) -> T -public func *= (lhs: inout T, rhs: T) -public func / (lhs: T, rhs: T) -> T -public func /= (lhs: inout T, rhs: T) -public func % (lhs: T, rhs: T) -> T -public func %= (lhs: inout T, rhs: T) -``` + /// Returns a double-width value containing the high and low parts of the + /// result of multiplying this value by an argument. + /// + /// Use this method to calculate the full result of a product that would + /// otherwise overflow. Unlike traditional truncating multiplication, the + /// `multipliedFullWidth(by:)` method returns both the high and low + /// parts of the product of `self` and `other`. The following example uses + /// this method to multiply two `UInt8` values that normally overflow when + /// multiplied: + /// + /// let x: UInt8 = 100 + /// let y: UInt8 = 20 + /// let result = x.multipliedFullWidth(by: y) + /// // result.high == 0b00000111 + /// // result.low == 0b11010000 + /// + /// The product of `x` and `y` is 2000, which is too large to represent in a + /// `UInt8` instance. The `high` and `low` components of the `result` + /// represent 2000 when concatenated to form a double-width integer; that + /// is, using `result.high` as the high byte and `result.low` as the low byte + /// of a `UInt16` instance. + /// + /// let z = UInt16(result.high) << 8 | UInt16(result.low) + /// // z == 2000 + /// + /// - Parameters: + /// - other: A value to multiplied `self` by. + /// - Returns: A tuple containing the high and low parts of the result of + /// multiplying `self` and `other`. + /// + /// - SeeAlso: `multipliedReportingOverflow(by:)` + func multipliedFullWidth(by other: Self) -> DoubleWidth -##### Implementation example + /// Returns a tuple containing the quotient and remainder of dividing the + /// first argument by this value. + /// + /// The resulting quotient must be representable within the bounds of the + /// type. If the quotient of dividing `lhs` by `self` is too large to + /// represent in the type, a runtime error may occur. + /// + /// - Parameters: + /// - lhs: A value containing the high and low parts of a double-width + /// integer. The `high` component of the tuple carries the sign, if the + /// type is signed. + /// - Returns: A tuple containing the quotient and remainder of `lhs` divided + /// by `self`. + func dividingFullWidth(_ lhs: DoubleWidth) + -> (quotient: Self, remainder: Self) + + /// The number of bits equal to 1 in this value's binary representation. + /// + /// For example, in a fixed-width integer type with a `bitWidth` value of 8, + /// the number 31 has five bits equal to 1. + /// + /// let x: Int8 = 0b0001_1111 + /// // x == 31 + /// // x.populationCount == 5 + var populationCount: Int { get } -_Only homogeneous arithmetic operations are supported._ + /// The number of leading zeros in this value's binary representation. + /// + /// For example, in a fixed-width integer type with a `bitWidth` value of 8, + /// the number 31 has three leading zeros. + /// + /// let x: Int8 = 0b0001_1111 + /// // x == 31 + /// // x.leadingZeroBits == 3 + /// - SeeAlso: `BinaryInteger.trailingZeroBits` + var leadingZeroBits: Int { get } -```Swift -public func + (lhs: T, rhs: T) -> T { - return lhs.adding(rhs) -} + /// Creates an integer from its big-endian representation, changing the + /// byte order if necessary. + init(bigEndian value: Self) -extension Arithmetic { - public func adding(_ rhs: Self) -> Self { - var lhs = self - lhs.add(rhs) - return lhs - } -} + /// Creates an integer from its little-endian representation, changing the + /// byte order if necessary. + init(littleEndian value: Self) -extension FixedWidthInteger { - public mutating func add(_ rhs: Self) { - let (result, overflow) = self.addingWithOverflow(rhs) - self = result - } -} + /// The big-endian representation of this integer. + /// + /// If necessary, the byte order of this value is reversed from the typical + /// byte order of this integer type. On a big-endian platform, for any + /// integer `x`, `x == x.bigEndian`. + /// + /// - SeeAlso: `littleEndian` + var bigEndian: Self { get } -public struct Int8 { - public func addingWithOverflow(_ rhs: DoubleWidth) - -> (partialValue: DoubleWidth, overflow: ArithmeticOverflow) { - // efficient implementation - } + /// The little-endian representation of this integer. + /// + /// If necessary, the byte order of this value is reversed from the typical + /// byte order of this integer type. On a little-endian platform, for any + /// integer `x`, `x == x.littleEndian`. + /// + /// - SeeAlso: `bigEndian` + var littleEndian: Self { get } + + /// A representation of this integer with the byte order swapped. + var byteSwapped: Self { get } + + + /// Returns the result of shifting a value's binary representation the + /// specified number of digits to the right, masking the shift amount to the + /// type's bit width. + static func &>>(_ lhs: Self, _ rhs: Self) -> Self + + /// Calculates the result of shifting a value's binary representation the + /// specified number of digits to the right, masking the shift amount to the + /// type's bit width, and stores the result in the left-hand-side variable. + static func &>>=(_ lhs: inout Self, _ rhs: Self) + + /// Returns the result of shifting a value's binary representation the + /// specified number of digits to the left, masking the shift amount to the + /// type's bit width. + static func &<<(_ lhs: Self, _ rhs: Self) -> Self + + /// Returns the result of shifting a value's binary representation the + /// specified number of digits to the left, masking the shift amount to the + /// type's bit width, and stores the result in the left-hand-side variable. + static func &<<=(_ lhs: inout Self, _ rhs: Self) } ``` - -#### Masking arithmetic +#### Auxiliary protocols ```Swift -public func &* (lhs: T, rhs: T) -> T -public func &- (lhs: T, rhs: T) -> T -public func &+ (lhs: T, rhs: T) -> T +public protocol UnsignedInteger : BinaryInteger { + associatedtype Magnitude : BinaryInteger +} +public protocol SignedInteger : BinaryInteger, SignedNumeric { + associatedtype Magnitude : BinaryInteger +} ``` -##### Implementation +### DoubleWidth\ -These operators call `WithOverflow` family of methods from `FixedWidthInteger` -and simply return the `partialValue` part, ignoring the possible overflow. +The `DoubleWidth` type allows to create wider fixed-width integer types from +the ones available in the standard library. -```Swift -public func &+ (lhs: T, rhs: T) -> T { - return lhs.addingWithOverflow(rhs).partialValue -} +Standard library currently provides fixed-width integer types of up to 64 bits. +A value of `DoubleWidth` will double the range of the underlying type and +implement all the `FixedWidthInteger` requirements. _Please note_ though that +the implementation will not necessarily be the most efficient one, so it would +not be a good idea to use `DoubleWidth` instead of a built-in `Int64`. -public struct Int8 { - public func addingWithOverflow(_ other: DoubleWidth) - -> (partialValue: DoubleWidth, overflow: ArithmeticOverflow) { - // efficient implementation - } +```swift +public enum DoubleWidth { + case .parts(high: T, low: T.Magnitude) + + public var high: T { get } + public var low: T.Magnitude { get } } ``` +Representing it as an `enum` instead of a simple struct allows to use it both +as a single value, as well as in destructuring matches. -#### Homogeneous comparison +```swift +let high = doubleWidthValue.high +let low = doubleWidthValue.low -```Swift -public func == (lhs:T, rhs: T) -> Bool -public func != (lhs:T, rhs: T) -> Bool -public func < (lhs: T, rhs: T) -> Bool -public func > (lhs: T, rhs: T) -> Bool -public func >= (lhs: T, rhs: T) -> Bool -public func <= (lhs: T, rhs: T) -> Bool +// or + +case let (high, low) = doubleWidthValue ``` -The implementation is similar to the homogeneous arithmetic operators above. +### Extra operators -#### Heterogeneous comparison +In addition to the operators described in the [protocols section](#protocols), +we also provide a few extensions: + +#### Non-mutating homogeneous shifts ```Swift -public func == (lhs:T, rhs: U) -> Bool -public func != (lhs:T, rhs: U) -> Bool -public func < (lhs: T, rhs: U) -> Bool -public func > (lhs: T, rhs: U) -> Bool -public func >= (lhs: T, rhs: U) -> Bool -public func <= (lhs: T, rhs: U) -> Bool +extension FixedWidthInteger { + public static func &>> (lhs: Self, rhs: Self) -> Self + public static func &<< (lhs: Self, rhs: Self) -> Self ``` -##### Implementation example +#### Heterogeneous shifts ```Swift -public func == (lhs:T, rhs: U) -> Bool { - return (lhs > 0) == (rhs > 0) - && T(extendingOrTruncating: rhs) == lhs - && U(extendingOrTruncating: lhs) == rhs +extension BinaryInteger { + // 'Smart' shifts + static func >> (lhs: Self, rhs: Other) -> Self + static func >>= (lhs: inout Self, rhs: Other) + static func << (lhs: Self, rhs: Other) -> Self + static func <<= (lhs: inout Self, rhs: Other) } - extension FixedWidthInteger { - public init(extendingOrTruncating source: T) { - // converting `source` into the value of `Self` - } + public static func &>> (lhs: Self, rhs: Other) -> Self + public static func &>>= (lhs: inout Self, rhs: Other) + public static func &<< (lhs: Self, rhs: Other) -> Self + public static func &<<= (lhs: inout Self, rhs: Other) } ``` - -#### Shifts +#### Heterogeneous equality and comparison ```Swift -public func << (lhs: T, rhs: U) -> T -public func << (lhs: T, rhs: Word) -> T -public func <<= (lhs: inout T, rhs: U) -public func <<= (lhs: inout T, rhs: T) - -public func >> (lhs: T, rhs: U) -> T -public func >> (lhs: T, rhs: Word) -> T -public func >>= (lhs: inout T, rhs: T) -public func >>= (lhs: inout T, rhs: U) - -public func &<< (lhs: T, rhs: U) -> T -public func &<< (lhs: T, rhs: T) -> T -public func &<<= (lhs: inout T, rhs: U) -public func &<<= (lhs: inout T, rhs: T) - -public func &>> (lhs: T, rhs: U) -> T -public func &>> (lhs: T, rhs: T) -> T -public func &>>= (lhs: inout T, rhs: U) -public func &>>= (lhs: inout T, rhs: T) +extension BinaryInteger { + // Equality + static func == (lhs: Self, rhs: Other) -> Bool + static func != (lhs: Self, rhs: Other) -> Bool + + // Comparison + static func < (lhs: Self, rhs: Other) -> Bool + static func <= (lhs: Self, rhs: Other) -> Bool + static func > (lhs: Self, rhs: Other) -> Bool + static func >= (lhs: Self, rhs: Other) -> Bool +} ``` -##### Notes on the implementation of mixed-type shifts - -The implementation is similar to the heterogeneous comparison. The only -difference is that because shifting left truncates the high bits of fixed-width -integers, it is hard to define what a left shift would mean to an -arbitrary-precision integer. Therefore we only allow shifts where the left -operand conforms to the `FixedWidthInteger` protocol. The right operand, -however, can be an arbitrary `BinaryInteger`. - -#### Bitwise operations +#### Masking arithmetic ```Swift -public func | (lhs: T, rhs: T) -> T -public func |= (lhs: inout T, rhs: T) -public func & (lhs: T, rhs: T) -> T -public func &= (lhs: inout T, rhs: T) -public func ^ (lhs: T, rhs: T) -> T -public func ^= (lhs: inout T, rhs: T) +public func &* (lhs: T, rhs: T) -> T +public func &- (lhs: T, rhs: T) -> T +public func &+ (lhs: T, rhs: T) -> T ``` -## Impact on existing code - -The new model is designed to be a drop-in replacement for the current one. One -feature that has been deliberately removed is the concept of `the widest -integer type`, which will require a straightforward code migration. - -Existing code that does not implement its own integer types (or rely on the -existing protocol hierarchy in any other way) should not be affected. It may -be slightly wordier than necessary due to all the type conversions that are -no longer required, but will continue to work. - - ## Non-goals This proposal: @@ -608,5 +1054,29 @@ This proposal: - *DOES NOT* include the implementation of a `BigInt` type, but allows it to be implemented in the future. -- *DOES NOT* propose including a `DoubleWidth` integer type in the standard - library, but provides a proof-of-concept implementation. + +## Source compatibility + +The proposed change is designed to be as non-breaking as possible, and it has +been proven that it does not break code on concrete integer types. However, +there are still a few API breaking changes in the realm of generic code: + +* Integer protocols in Swift up to and including version 3 were not particularly +useful for generic programming, but were rather a means of sharing +implementation between conforming types. Therefore we believe the amount of code +that relied on these protocols is relatively small. The breakage can be further +reduced by introducing proper aliases for the removed protocols with deprecation +warnings. + +* Deprecation of the `BitwiseOperations` protocol. We find it hard to imagine a +type that conforms to this protocol, but is *not* a binary integer type. + +* Addition of 'smart' shifts will change the behavior of existing code. It will +still compile, but will be potentially less performant due to extra logic +involved. In a case, where this becomes a problem, newly introduced masking +shift operators can be used instead. Unfortunately, performance characteristics +of the code cannot be statically checked, and thus migration cannot be provided. + + +[se91]: 0091-improving-operators-in-protocols.md +[impl]: https://github.com/apple/swift/tree/new-integer-protocols diff --git a/proposals/0105-remove-where-from-forin-loops.md b/proposals/0105-remove-where-from-forin-loops.md index acfd954df4..3df0c2d63e 100644 --- a/proposals/0105-remove-where-from-forin-loops.md +++ b/proposals/0105-remove-where-from-forin-loops.md @@ -1,16 +1,16 @@ # Removing Where Clauses from For-In Loops * Proposal: [SE-0105](0105-remove-where-from-forin-loops.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000199.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0105-removing-where-clauses-from-for-in-loops/3205) ## Introduction This proposal removes `where` clauses from `for-in` loops, where they are better expressed (and read) as guard conditions. -Swift Evolution Discussion: [\[Pitch\] Retiring `where` from for-in loops](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160606/020566.html) +Swift Evolution Discussion: [\[Pitch\] Retiring `where` from for-in loops](https://forums.swift.org/t/pitch-retiring-where-from-for-in-loops/2926) ## Motivation @@ -26,7 +26,7 @@ public/core/Algorithm.swift: for value in rest where value < minValue { public/core/Algorithm.swift: for value in rest where value >= maxValue { ``` -I pulled down a random sample of popular Swift repositories from github and found one use of `for-in-where` among my sample vs over 650 `for-in` uses. +I pulled down a random sample of popular Swift repositories from GitHub and found one use of `for-in-where` among my sample vs over 650 `for-in` uses. ``` Carthage/Source/CarthageKit/Algorithms.swift: for (node, var incomingEdges) in workingGraph where incomingEdges.contains(lastSource) { @@ -153,4 +153,4 @@ Code must be refactored to move the where clause into `guard` (or, for less styl ## Acknowledgements -Big thanks to Joe Groff, Brent Royal-Gordon, Xiaodi Wu +Big thanks to Joe Groff, Becca Royal-Gordon, Xiaodi Wu diff --git a/proposals/0106-rename-osx-to-macos.md b/proposals/0106-rename-osx-to-macos.md index 38f4231117..e5f4714024 100644 --- a/proposals/0106-rename-osx-to-macos.md +++ b/proposals/0106-rename-osx-to-macos.md @@ -1,10 +1,10 @@ # Add a `macOS` Alias for the `OSX` Platform Configuration Test * Proposal: [SE-0106](0106-rename-osx-to-macos.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decison Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-June/000193.html) +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0106-add-a-macos-alias-for-the-osx-platform-configuration-test/3176) * Bugs: [SR-1823](https://bugs.swift.org/browse/SR-1823), [SR-1887](https://bugs.swift.org/browse/SR-1887) @@ -14,7 +14,7 @@ Starting in Sierra, Apple's Mac-based OS (OS X) will be renamed "macOS". All use This proposal adds the `#if os(macOS)` platform configuration test to alias the current `#if os(OSX)` -Swift Evolution Discussion: [\[DRAFT\] Aliasing the OS X Platform Configuration Test](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160613/021239.html) +Swift Evolution Discussion: [\[DRAFT\] Aliasing the OS X Platform Configuration Test](https://forums.swift.org/t/draft-aliasing-the-os-x-platform-configuration-test/2999) ## Motivation @@ -73,7 +73,7 @@ This proposal is purely additive. It will not affect existing code other than ad Instead of retaining and aliasing `os(OSX)`, it can be fully replaced by `os(macOS)`. This mirrors the situation with the phoneOS to iOS rename and would require a migration assistant to fixit old-style use. -Charlie Monroe points out: "Since Swift 3.0 is a code-breaking change my guess is that there is no burden if the Xcode migration assistent automatically changes all `#if os(OSX)` to `#if os(macOS)`, thus deprecating the term OSX, not burdening the developer at all. If iOS was renamed to phoneOS and kept versioning, you'd still expect `#if os(iOS)` to be matched when targetting phoneOS and vice-versa." +Charlie Monroe points out: "Since Swift 3.0 is a code-breaking change my guess is that there is no burden if the Xcode migration assistant automatically changes all `#if os(OSX)` to `#if os(macOS)`, thus deprecating the term OSX, not burdening the developer at all. If iOS was renamed to phoneOS and kept versioning, you'd still expect `#if os(iOS)` to be matched when targeting phoneOS and vice-versa." ## Unaddressed Issues diff --git a/proposals/0107-unsaferawpointer.md b/proposals/0107-unsaferawpointer.md index 0f44e6bebd..6acefab32f 100644 --- a/proposals/0107-unsaferawpointer.md +++ b/proposals/0107-unsaferawpointer.md @@ -2,11 +2,17 @@ * Proposal: [SE-0107](0107-unsaferawpointer.md) * Author: [Andrew Trick](https://github.com/atrick) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000231.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0107-unsaferawpointer-api/3389) -For quick reference, jump to: +For detailed instructions on how to migrate your code to this new +Swift 3 API refer to the +[UnsafeRawPointer Migration Guide](https://swift.org/migration-guide-swift3/se-0107-migrate.html). See +also: See `bindMemory(to:capacity:)`, `assumingMemoryBound(to:)`, and +`withMemoryRebound(to:capacity:)`. + +For quick reference on the full API, jump to: - [Full UnsafeRawPointer API](#full-unsaferawpointer-api) Contents: @@ -89,14 +95,11 @@ This proposal aims to achieve several goals in one coherent design: Swift-evolution threads: -- [\[RFC\] UnsafeBytePointer API for In-Memory Layout](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160509/thread.html#16909) +- [\[RFC\] UnsafeBytePointer API for In-Memory Layout](https://forums.swift.org/t/rfc-unsafebytepointer-api-for-in-memory-layout/2526) -- [\[RFC\] UnsafeBytePointer API for In-Memory Layout (Round 2)](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160516/thread.html#18156) +- [\[RFC\] UnsafeBytePointer API for In-Memory Layout (Round 2)](https://forums.swift.org/t/rfc-unsafebytepointer-api-for-in-memory-layout/2526/7) -- [RFC] UnsafeRawPointer API (Round 3) - - [Week #1](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/thread.html#22005) - - [Week #2](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/thread.html#22230) - - [Week #3](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160704/thread.html#23384) +- [\[RFC\] UnsafeRawPointer API (Round 3)](https://forums.swift.org/t/draft-unsaferawpointer-api/3123) [1]:https://github.com/atrick/swift/blob/type-safe-mem-docs/docs/TypeSafeMemory.rst @@ -157,7 +160,7 @@ value argument could result in miscompilation if the inferred type ever deviates from the user's original expectations. The type parameter also importantly conveys that the raw memory becomes accessible via a pointer to that type at the point of the call. The -type should be explicitly spelled at this point because accesing the +type should be explicitly spelled at this point because accessing the memory via a typed pointer of an unrelated type could also result in miscompilation. @@ -248,7 +251,7 @@ UnsafeMutableRawPointer { } ``` -The `load` and `storeBytes` operations are assymetric. `load` reads raw +The `load` and `storeBytes` operations are asymmetric. `load` reads raw bytes but properly constructs a new value of type `T` with its own lifetime. Any copied references will be retained. In contrast, `storeBytes` only operates on a value's raw bytes, writing them into @@ -732,7 +735,7 @@ func stringFromBytes(size: Int, value: UInt8) { bytes.initialize(to: value, count: size) bytes[size] = 0 - // Unsafe pointer conversion is requred to invoke readCString. + // Unsafe pointer conversion is required to invoke readCString. // If readCString is inlineable and compiled with strict aliasing, // then it could read uninitialized memory. readCStr(UnsafePointer(bytes)) @@ -1765,7 +1768,7 @@ to build the standard library with the changes: - The type system handles implicit conversions to UnsafeRawPointer. - `UnsafeRawPointer` replaces both `UnsafePointer` and - `UnsafeMutablePointer` (Recent feedback suggestes that + `UnsafeMutablePointer` (Recent feedback suggests that `UnsafeMutablePointer` should also be introduced). - The standard library was relying on inferred `UnsafePointer` diff --git a/proposals/0108-remove-assoctype-inference.md b/proposals/0108-remove-assoctype-inference.md index 5e2ed6e3a3..b1373d8a75 100644 --- a/proposals/0108-remove-assoctype-inference.md +++ b/proposals/0108-remove-assoctype-inference.md @@ -2,9 +2,9 @@ * Proposal: [SE-0108](0108-remove-assoctype-inference.md) * Authors: [Douglas Gregor](https://github.com/DougGregor), Austin Zheng -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000214.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0108-remove-associated-type-inference/3304) ## Introduction @@ -28,7 +28,7 @@ In this example, the typechecker deduces that `StringBag.Element` is `String` th In order to simplify the compiler and typechecker, we propose to **remove associated type witness inference**. -swift-evolution thread: [pre-proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/022138.html) +swift-evolution thread: [pre-proposal](https://forums.swift.org/t/pitch-remove-type-inference-for-associated-types/3135) ## Motivation @@ -38,7 +38,7 @@ According to *[Completing Generics](https://github.com/apple/swift/blob/master/d The main advantage of removing associated type witness inference is that it decreases the complexity of the type checker. Doing so removes the only aspect of Swift that depends upon global type inference. Simplifying the type checker makes it easier to improve the performance and correctness of the type checker code. Given that both are widely acknowledged issues with current versions of Swift, any opportunity for improvement should be carefully considered. -As Douglas Gregor (original author of the relevant type inference code) [puts it](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/022483.html): +As Douglas Gregor (original author of the relevant type inference code) [puts it](https://forums.swift.org/t/pitch-remove-type-inference-for-associated-types/3135/23): > Because this is the only place we do global type inference, it’s put tremendous pressure on the type checker that caused a huge number of bugs, crashes, and outright incomprehensible behavior. [...] [The re-implementation is] still not global *enough* to actually be predictable, and the legacy of this mis-feature manifests in a number of weird ways (e.g., typealiases in protocol extensions cannot be used to satisfy associated type requirements, weird rules for when a defaulted associated type gets used). @@ -147,7 +147,7 @@ Currently, `C.A` for the previous example would be inferred to be `String`, and If associated type inference were to be removed, `C.A` would be bound as `Int` (since there would be no explicit `typealias` declaration overriding the default type value), and the `doSomething()` implementation returning `Int` would be considered to fulfill the protocol requirement. Thus, the semantics of the code listing above would change even though the source itself remained unchanged. -To some extent, this is an issue inherent to any design which makes no distinctions at the site of implementation between members intended to satisfy protocol requirements and members that are explicitly not intended to satisfy protocol requirements. Rather than adding keywords to create this distinction, Douglas Gregor has [proposed and implemented type checker heuristics](https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20151228/000643.html) that will generate warnings when a programmer implements a member that "looks like" it should fulfill a protocol requirement but does not actually do so. This is one possible mitigation strategy that should be revisited as a way to decrease the possible impact of removing associated type witness inference from the compiler. +To some extent, this is an issue inherent to any design which makes no distinctions at the site of implementation between members intended to satisfy protocol requirements and members that are explicitly not intended to satisfy protocol requirements. Rather than adding keywords to create this distinction, Douglas Gregor has [proposed and implemented type checker heuristics](https://forums.swift.org/t/warning-when-overriding-an-extension-method-thats-not-in-the-protocol/861/2) that will generate warnings when a programmer implements a member that "looks like" it should fulfill a protocol requirement but does not actually do so. This is one possible mitigation strategy that should be revisited as a way to decrease the possible impact of removing associated type witness inference from the compiler. ## Impact on existing code @@ -163,9 +163,9 @@ The current behavior is kept. Swift will continue to allow associated types to b There are some advantages to this approach. Brevity is slightly improved. A type's associated types don't "stand out" in the type declaration, being unobtrusively and implicitly defined through the implementation of protocol requirements. -As well, Dave Abrahams expresses a [potential issue](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/022316.html): +As well, Dave Abrahams expresses a [potential issue](https://forums.swift.org/t/pitch-remove-type-inference-for-associated-types/3135/17): -> Finally, I am very concerned that there are protocols such as `Collection`, with many inferrable associated types, and that conforming to these protocols could become *much* uglier. +> Finally, I am very concerned that there are protocols such as `Collection`, with many inferable associated types, and that conforming to these protocols could become *much* uglier. As with many proposals, there is a tradeoff between the status quo and the proposed behavior. As *Completing Generics* puts it, @@ -173,4 +173,4 @@ As with many proposals, there is a tradeoff between the status quo and the propo ### Require explicit declaration using `associatedtype` -An [earlier draft of this proposal](https://github.com/apple/swift-evolution/blob/18a1781d930034583ffc0325a180099f15fbb834/proposals/XXXX-remove-assoctype-inference.md) detailed a design in which types would explicitly bind their associated types using an `associatedtype` declaration. It is presented as an alternative for consideration. +An [earlier draft of this proposal](https://github.com/swiftlang/swift-evolution/blob/18a1781d930034583ffc0325a180099f15fbb834/proposals/XXXX-remove-assoctype-inference.md) detailed a design in which types would explicitly bind their associated types using an `associatedtype` declaration. It is presented as an alternative for consideration. diff --git a/proposals/0109-remove-boolean.md b/proposals/0109-remove-boolean.md old mode 100755 new mode 100644 index 88d2cd8ad8..89a62aabd3 --- a/proposals/0109-remove-boolean.md +++ b/proposals/0109-remove-boolean.md @@ -2,11 +2,11 @@ * Proposal: [SE-0109](0109-remove-boolean.md) * Authors: [Anton Zhilin](https://github.com/Anton3), [Chris Lattner](https://github.com/lattner) -* Review Manager: [Doug Gregor](http://github.com/DougGregor) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160711/024270.html) -* Commits: [apple/swift@76cf339](https://github.com/apple/swift/commit/76cf339694a41293dbbec9672b6df87a864087f2), - [apple/swift@af30ae3](https://github.com/apple/swift/commit/af30ae32226813ec14c2bef80cb090d3e6c586fb) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0109-remove-the-boolean-protoco/3380) +* Implementation: [apple/swift@76cf339](https://github.com/apple/swift/commit/76cf339694a41293dbbec9672b6df87a864087f2), + [apple/swift@af30ae3](https://github.com/apple/swift/commit/af30ae32226813ec14c2bef80cb090d3e6c586fb) ## Introduction @@ -29,7 +29,7 @@ consistently in APIs that take Boolean parameters: almost everything takes `Bool` concretely. This means that its supposed abstraction isn't useful. The only significant users are the unary `!`, and binary `&&` and `||` operators. -[Discussion thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021983.html) +[Discussion thread](https://forums.swift.org/t/proposal-remove-boolean/3124) ## Proposal diff --git a/proposals/0110-distingish-single-tuple-arg.md b/proposals/0110-distingish-single-tuple-arg.md deleted file mode 100644 index ee0388889e..0000000000 --- a/proposals/0110-distingish-single-tuple-arg.md +++ /dev/null @@ -1,56 +0,0 @@ -# Distinguish between single-tuple and multiple-argument function types - -* Proposal: [SE-0110](0110-distingish-single-tuple-arg.md) -* Authors: Vladimir S., Austin Zheng -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Accepted** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000215.html) -* Bug: [SR-2008](https://bugs.swift.org/browse/SR-2008) - -## Introduction - -Swift's type system should properly distinguish between functions that take one tuple argument, and functions that take multiple arguments. - -Discussion: [pre-proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021793.html) - -## Motivation - -Right now, the following is possible: - -```swift -let fn1 : (Int, Int) -> Void = { x in - // The type of x is the tuple (Int, Int). - // ... -} - -let fn2 : (Int, Int) -> Void = { x, y in - // The type of x is Int, the type of y is Int. - // ... -} -``` - -A variable of function type where there exist _n_ parameters (where _n_ > 1) can be assigned a value (whether it be a named function, a closure literal, or other acceptable value) which either takes in _n_ parameters, or one tuple containing _n_ elements. This seems to be an artifact of the tuple splat behavior removed in [SE-0029](0029-remove-implicit-tuple-splat.md). - -The current behavior violates the principle of least surprise and weakens type safety, and should be changed. - -## Proposed solution - -We propose that this behavior should be fixed in the following ways: - -* A function type declared with _n_ parameters (_n_ > 1) can only be satisfied by a function value which takes in _n_ parameters. In the above example, only the `fn2` expression would be considered valid. - -* To declare a function type with one tuple parameter containing _n_ elements (where _n_ > 1), the function type's argument list must be enclosed by double parentheses: - - ```swift - let a : ((Int, Int, Int)) -> Int = { x in return x.0 + x.1 + x.2 } - ``` - - We understand that this may be a departure from the current convention that a set of parentheses enclosing a single object are considered semantically meaningless, but it is the most natural way to differentiate between the two situations described above and would be a clearly-delineated one-time-only exception. - -## Impact on existing code - -Minor changes to user code may be required if this proposal is accepted. - -## Alternatives considered - -Don't make this change. diff --git a/proposals/0110-distinguish-single-tuple-arg.md b/proposals/0110-distinguish-single-tuple-arg.md new file mode 100644 index 0000000000..be56d47d1c --- /dev/null +++ b/proposals/0110-distinguish-single-tuple-arg.md @@ -0,0 +1,70 @@ +# Distinguish between single-tuple and multiple-argument function types + +* Proposal: [SE-0110](0110-distinguish-single-tuple-arg.md) +* Authors: Vladimir S., [Austin Zheng](https://github.com/austinzheng) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0110-distinguish-between-single-tuple-and-multiple-argument-function-types/3305), [Additional Commentary](https://forums.swift.org/t/core-team-addressing-the-se-0110-usability-regression-in-swift-4/6147) +* Bug: [SR-2008](https://bugs.swift.org/browse/SR-2008) +* Previous Revision: [Originally Accepted Proposal](https://github.com/swiftlang/swift-evolution/blob/9e44932452e1daead98f2bc2e58711eb489e9751/proposals/0110-distingish-single-tuple-arg.md) + +## Introduction + +Swift's type system should properly distinguish between functions that take one tuple argument, and functions that take multiple arguments. + +Discussion: [pre-proposal](https://forums.swift.org/t/partial-list-of-open-swift-3-design-topics/3094) + +## Motivation + +Right now, the following is possible: + +```swift +let fn1 : (Int, Int) -> Void = { x in + // The type of x is the tuple (Int, Int). + // ... +} + +let fn2 : (Int, Int) -> Void = { x, y in + // The type of x is Int, the type of y is Int. + // ... +} +``` + +A variable of function type where there exist _n_ parameters (where _n_ > 1) can be assigned a value (whether it be a named function, a closure literal, or other acceptable value) which either takes in _n_ parameters, or one tuple containing _n_ elements. This seems to be an artifact of the tuple splat behavior removed in [SE-0029](0029-remove-implicit-tuple-splat.md). + +The current behavior violates the principle of least surprise and weakens type safety, and should be changed. + +## Proposed solution + +We propose that this behavior should be fixed in the following ways: + +* A function type declared with _n_ parameters (_n_ > 1) can only be satisfied by a function value which takes in _n_ parameters. In the above example, only the `fn2` expression would be considered valid. + +* To declare a function type with one tuple parameter containing _n_ elements (where _n_ > 1), the function type's argument list must be enclosed by double parentheses: + + ```swift + let a : ((Int, Int, Int)) -> Int = { x in return x.0 + x.1 + x.2 } + ``` + + We understand that this may be a departure from the current convention that a set of parentheses enclosing a single object are considered semantically meaningless, but it is the most natural way to differentiate between the two situations described above and would be a clearly-delineated one-time-only exception. + +Existing Swift code widely takes advantage of the ability to pass a multi-parameter closure or function value to a higher-order function that operates on tuples, particularly with collection operations: + +``` +zip([1, 2, 3], [3, 2, 1]).filter(<) // => [(1, 3)] +zip([1, 2, 3], [3, 2, 1]).map(+) // => [4, 4, 4] +``` + +Without the implicit conversion, this requires invasive changes to explicitly destructure the tuple argument. In order to gain most of the type system benefits of distinguishing single-tuple-argument functions from multiple-argument functions, while maintaining the fluidity of functional code like the above, arguments of type `(T, U, ...) -> V` in call expressions are allowed to be converted to parameters of the corresponding single-tuple parameter type `((T, U, ...)) -> V`, so the two examples above will continue to be accepted. + +## Impact on existing code + +Minor changes to user code may be required if this proposal is accepted. + +## Alternatives considered + +Don't make this change. + +## Revision history + +The [original proposal as reviewed](https://github.com/swiftlang/swift-evolution/blob/9e44932452e1daead98f2bc2e58711eb489e9751/proposals/0110-distingish-single-tuple-arg.md) did not include the special-case conversion from `(T, U, ...) -> V` to `((T, U, ...)) -> V` for function arguments. In response to community feedback, [this conversion was added](https://forums.swift.org/t/core-team-addressing-the-se-0110-usability-regression-in-swift-4/6147) as part of the Core Team's acceptance of the proposal. diff --git a/proposals/0111-remove-arg-label-type-significance.md b/proposals/0111-remove-arg-label-type-significance.md index aa82fcce62..0c6db853bd 100644 --- a/proposals/0111-remove-arg-label-type-significance.md +++ b/proposals/0111-remove-arg-label-type-significance.md @@ -2,16 +2,16 @@ * Proposal: [SE-0111](0111-remove-arg-label-type-significance.md) * Author: Austin Zheng -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000216.html), [Additional Commentary](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000233.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0111-remove-type-system-significance-of-function-argument-labels/3306), [Additional Commentary](https://forums.swift.org/t/update-commentary-se-0111-remove-type-system-significance-of-function-argument-labels/3391) * Bug: [SR-2009](https://bugs.swift.org/browse/SR-2009) ## Introduction Swift's type system should not allow function argument labels to be expressed as part of a function type. -Discussion: [pre-proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021793.html) +Discussion: [pre-proposal](https://forums.swift.org/t/partial-list-of-open-swift-3-design-topics/3094) ## Motivation diff --git a/proposals/0112-nserror-bridging.md b/proposals/0112-nserror-bridging.md index e68ea6716a..da41dc387f 100644 --- a/proposals/0112-nserror-bridging.md +++ b/proposals/0112-nserror-bridging.md @@ -2,9 +2,9 @@ * Proposal: [SE-0112](0112-nserror-bridging.md) * Authors: [Doug Gregor](https://github.com/DougGregor), [Charles Srstka](https://github.com/CharlesJS) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000222.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0112-improved-nserror-bridging/3362) ## Introduction @@ -38,9 +38,9 @@ proposal attempts to bridge those gaps. Swift-evolution thread: [Charles Srstka's pitch for Consistent bridging for NSErrors at the language -boundary](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160502/016618.html), +boundary](https://forums.swift.org/t/pitch-consistent-bridging-for-nserrors-at-the-language-boundary/2482), which discussed Charles' [original -proposal](https://github.com/apple/swift-evolution/pull/331) that +proposal](https://github.com/swiftlang/swift-evolution/pull/331) that addressed these issues by providing ``NSError`` to ``ErrorProtocol`` bridging and exposing the domain, code, and user-info dictionary for all errors. This proposal expands upon that work, but without directly diff --git a/proposals/0113-rounding-functions-on-floatingpoint.md b/proposals/0113-rounding-functions-on-floatingpoint.md index a99999732d..d3d78f5036 100644 --- a/proposals/0113-rounding-functions-on-floatingpoint.md +++ b/proposals/0113-rounding-functions-on-floatingpoint.md @@ -2,9 +2,9 @@ * Proposal: [SE-0113](0113-rounding-functions-on-floatingpoint.md) * Author: [Karl Wagner](https://github.com/karwa) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000217.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0113-add-integral-rounding-functions-to-floatingpoint/3308) * Bug: [SR-2010](https://bugs.swift.org/browse/SR-2010) ## Introduction, Motivation @@ -14,7 +14,7 @@ The standard library lacks equivalents to the `floor()` and `ceil()` functions f In general, rounding of floating-point numbers for predictable conversion in to integers is something we should provide natively. Swift-evolution initial discussion thread: [\[Proposal\] Add floor() and ceiling() functions to FloatingPoint -](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/022146.html) +](https://forums.swift.org/t/proposal-add-floor-and-ceiling-functions-to-floatingpoint/3139) ## Proposed Solution diff --git a/proposals/0114-buffer-naming.md b/proposals/0114-buffer-naming.md index 9bc4db0560..1be984c287 100644 --- a/proposals/0114-buffer-naming.md +++ b/proposals/0114-buffer-naming.md @@ -1,11 +1,11 @@ # Updating Buffer "Value" Names to "Header" Names * Proposal: [SE-0114](0114-buffer-naming.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000221.html) -* Pull Request: [apple/swift#3374](https://github.com/apple/swift/pull/3374) +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0114-updating-buffer-value-names-to-header-names/3359) +* Implementation: [apple/swift#3374](https://github.com/apple/swift/pull/3374) ## Introduction @@ -13,7 +13,7 @@ This proposal updates parameters and generic type parameters from `value` names All user-facing Swift APIs must go through Swift Evolution. While this is a trivial API change with an existing implementation, this formal proposal provides a paper trail as is normal and usual for this process. -[Swift Evolution Thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/022551.html) +[Swift Evolution Thread](https://forums.swift.org/t/request-for-quickie-proposal-and-review/3175) [Patch](https://github.com/apple/swift/commit/eb7311de065df7ea332cdde8782cb44f9f4a5121) diff --git a/proposals/0115-literal-syntax-protocols.md b/proposals/0115-literal-syntax-protocols.md index c222bb8132..c5353edbdf 100644 --- a/proposals/0115-literal-syntax-protocols.md +++ b/proposals/0115-literal-syntax-protocols.md @@ -2,18 +2,18 @@ * Proposal: [SE-0115](0115-literal-syntax-protocols.md) * Author: [Matthew Johnson](https://github.com/anandabits) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000220.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0115-rename-literal-syntax-protocols/3358) * Bug: [SR-2054](https://bugs.swift.org/browse/SR-2054) ## Introduction This proposal renames the `*LiteralConvertible` protocols to `ExpressibleBy*Literal`. -Swift-evolution thread: [Literal Syntax Protocols](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021865.html) +Swift-evolution thread: [Literal Syntax Protocols](https://forums.swift.org/t/proposal-draft-literal-syntax-protocols/3109) -An earlier thread that resulted in this proposal: [Revisiting SE-0041 Names](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021714.html) +An earlier thread that resulted in this proposal: [Revisiting SE-0041 Names](https://forums.swift.org/t/revisiting-se-0041-names/3084) ## Motivation @@ -21,15 +21,15 @@ The standard library currently has protocols that use the term `Convertible` in Further, the standard library team has observed: - The "literal" protocols are not about conversion, they are about adopting - a certain syntax provided by the language. "Convertible" in the name is - a red herring: a type can't be convertible from an integer literal because - there is no "IntegerLiteral" entity in the type system. - The literal *becomes* typed as the corresponding literal type - (e.g., Int or String), and as far as the user at the call site is concerned, - there is no visible conversion (even if one is happening behind the scenes). +> The "literal" protocols are not about conversion, they are about adopting +> a certain syntax provided by the language. "Convertible" in the name is +> a red herring: a type can't be convertible from an integer literal because +> there is no "IntegerLiteral" entity in the type system. +> The literal *becomes* typed as the corresponding literal type +> (e.g., Int or String), and as far as the user at the call site is concerned, +> there is no visible conversion (even if one is happening behind the scenes). -[An earlier proposal](0041-conversion-protocol-conventions.md) was intended to address the first problem by introducing strong naming conventions for three kinds of conversion protocols (*from*, *to*, and *bidirectional*). The review highlighted the difficulity in establishing conventions that everyone is happy with. This proposal takes a different approach to solving the problem that originally inspired that proposal while also solving the awkwardness of the current names described by the standard library team. +[An earlier proposal](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0041-conversion-protocol-conventions.md) was intended to address the first problem by introducing strong naming conventions for three kinds of conversion protocols (*from*, *to*, and *bidirectional*). The review highlighted the difficulty in establishing conventions that everyone is happy with. This proposal takes a different approach to solving the problem that originally inspired that proposal while also solving the awkwardness of the current names described by the standard library team. ## Proposed solution @@ -79,30 +79,30 @@ All code that references any of the `*LiteralConvertible` protocols will need to Discussion of the pros and cons of the proposed and alternative naming schemes is encouraged. The core team should feel free to choose names they deem best suited for Swift after community discussion and review if they decide to accept this proposal. -The discussion thread for this proposal includes abundant bike shedding on the names. This section includes selected examples to highlight different directions that have been discussed. Reviewers are encouraged to read the discussion thread if they wish to see all of the alternatives. The thread includes abundant discusison of the pros and cons of many naming ideas. +The discussion thread for this proposal includes abundant bike shedding on the names. This section includes selected examples to highlight different directions that have been discussed. Reviewers are encouraged to read the discussion thread if they wish to see all of the alternatives. The thread includes abundant discussion of the pros and cons of many naming ideas. Some of the names that have been suggested have been inaccurate due to a misunderstanding of what the protocols do. Dave Abrahams explained during the discussion: - No, it's exactly the opposite, as I keep saying. Conformance to this - protocol does *not* mean you can initialize the type with a literal. - Proof: - -```swift - func f() -> T { - return T(integerLiteral: 43) // Error - return T(43) // Also an Error - } - -// It means an instance of the type can be *written* as a literal: - - func f() -> T { - return 43 // OK - } -``` - - Everybody's confused about the meaning of the protocol, and doesn't like - the proposed names because they imply exactly the actual meaning of the - protocol, which they misunderstand. +> No, it's exactly the opposite, as I keep saying. Conformance to this +> protocol does *not* mean you can initialize the type with a literal. +> Proof: +> +> ```swift +> func f() -> T { +> return T(integerLiteral: 43) // Error +> return T(43) // Also an Error +> } +> +> // It means an instance of the type can be *written* as a literal: +> +> func f() -> T { +> return 43 // OK +> } +>``` +> +> Everybody's confused about the meaning of the protocol, and doesn't like +> the proposed names because they imply exactly the actual meaning of the +> protocol, which they misunderstand. ### Previous Version @@ -121,35 +121,35 @@ Several commenters suggested that this naming scheme is confusing at the site of Nate Cook provided the best explanation of the potential confusion: - Primarily, the new names read like we're saying that a conforming type is a - literal, compounding a common existing confusion between literals and types - that can be initialized with a literal. Swift's type inference can sometimes - make it seem like dark magic is afoot with literal conversions—for example, - you need to understand an awful lot about the standard library to figure out - why line 1 works here but not line 2: - -Note: The comment above is still valid if it is corrected to say "types that can have instances *written as* a literal" rather than "types that can be *initialized with* a literal". - -```swift -var x = [1, 2, 3, 4, 5] -let y = [10, 20] - -x[1..<2] = [10, 20] // 1 -x[1..<2] = y // 2 -``` - - These new names are a (small) step in the wrong direction. While it's true - that the type system doesn't have an IntegerLiteral type, the language does - have integer literals. If someone reads: - -```swift -extension MyInt : Syntax.IntegerLiteral { ... } -``` - - the implication is that MyInt is an integer literal, and therefore instances - of MyInt should be usable wherever an integer literal is usable. - The existing "Convertible" wording may be a red herring, but it at least - suggests that there's a difference between a literal and a concrete type. +> Primarily, the new names read like we're saying that a conforming type is a +> literal, compounding a common existing confusion between literals and types +> that can be initialized with a literal. Swift's type inference can sometimes +> make it seem like dark magic is afoot with literal conversions—for example, +> you need to understand an awful lot about the standard library to figure out +> why line 1 works here but not line 2: + +>```swift +>var x = [1, 2, 3, 4, 5] +>let y = [10, 20] +> +>x[1..<2] = [10, 20] // 1 +>x[1..<2] = y // 2 +>``` + +(Note: The comment above is still valid if it is corrected to say "types that can have instances *written as* a literal" rather than "types that can be *initialized with* a literal".) + +> These new names are a (small) step in the wrong direction. While it's true +> that the type system doesn't have an IntegerLiteral type, the language does +> have integer literals. If someone reads: +> +>```swift +>extension MyInt : Syntax.IntegerLiteral { ... } +>``` +> +> the implication is that MyInt is an integer literal, and therefore instances +> of MyInt should be usable wherever an integer literal is usable. +> The existing "Convertible" wording may be a red herring, but it at least +> suggests that there's a difference between a literal and a concrete type. ### Namespace names diff --git a/proposals/0116-id-as-any.md b/proposals/0116-id-as-any.md index b23f75dada..f1cb4d528e 100644 --- a/proposals/0116-id-as-any.md +++ b/proposals/0116-id-as-any.md @@ -2,17 +2,17 @@ * Proposal: [SE-0116](0116-id-as-any.md) * Author: [Joe Groff](https://github.com/jckarter) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000243.html) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/b9a0ab5f7db4d3806c7941a07acedc5f0fe36e55/proposals/0116-id-as-any.md) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0116-import-objective-c-id-as-swift-any-type/3476) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/b9a0ab5f7db4d3806c7941a07acedc5f0fe36e55/proposals/0116-id-as-any.md) ## Introduction Objective-C interfaces that use `id` and untyped collections should be imported into Swift as taking the `Any` type instead of `AnyObject`. -Swift-evolution thread: [Importing Objective-C `id` as Swift `Any`](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/023203.html) +Swift-evolution thread: [Importing Objective-C `id` as Swift `Any`](https://forums.swift.org/t/pitch-importing-objective-c-id-as-swift-any/3236) ## Motivation @@ -172,7 +172,7 @@ that is itself `Hashable`, for use as the upper-bound type of heterogeneous `Dictionary`s and `Set`s. The user model for this type would ideally align with our long-term goal of supporting `Hashable` existentials directly, so the type deserves some short-term compiler support to help us get there. -This type deserves its own proposal and design discussion, so thi +This type deserves its own proposal and design discussion. ## Future Directions diff --git a/proposals/0117-non-public-subclassable-by-default.md b/proposals/0117-non-public-subclassable-by-default.md index 97be8b0ec1..50e2d07743 100644 --- a/proposals/0117-non-public-subclassable-by-default.md +++ b/proposals/0117-non-public-subclassable-by-default.md @@ -2,11 +2,11 @@ * Proposal: [SE-0117](0117-non-public-subclassable-by-default.md) * Authors: [Javier Soto](https://github.com/JaviSoto), [John McCall](https://github.com/rjmccall) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000268.html) -* Pull Request: [apple/swift#3882](https://github.com/apple/swift/pull/3882) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/367086f18a5deaf8f9dfbe3f5a4846ef19addf38/proposals/0117-non-public-subclassable-by-default.md), [2](https://github.com/apple/swift-evolution/blob/2989538daa1640cfa6a56f80b5c7599967af0905/proposals/0117-non-public-subclassable-by-default.md), [3](https://github.com/apple/swift-evolution/blob/15c18d24adb7e701ae831b643e0803f1b6e601d9/proposals/0117-non-public-subclassable-by-default.md) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0117-allow-distinguishing-between-public-access-and-public-overridability/3578) +* Implementation: [apple/swift#3882](https://github.com/apple/swift/pull/3882) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/367086f18a5deaf8f9dfbe3f5a4846ef19addf38/proposals/0117-non-public-subclassable-by-default.md), [2](https://github.com/swiftlang/swift-evolution/blob/2989538daa1640cfa6a56f80b5c7599967af0905/proposals/0117-non-public-subclassable-by-default.md), [3](https://github.com/swiftlang/swift-evolution/blob/15c18d24adb7e701ae831b643e0803f1b6e601d9/proposals/0117-non-public-subclassable-by-default.md) ## Introduction @@ -31,7 +31,7 @@ that is `open`, nor does it dissuade one from using `open` in their APIs. In fact, with this proposal, `open` APIs are syntactically lighter-weight than `public` ones. -Swift-evolution thread: https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/022354.html +Swift-evolution thread: ## Motivation @@ -172,7 +172,7 @@ current module and that superclass's access level is not `open`. An `open` class may not also be declared `final`. The superclass of an `open` class must be `open`. This is consistent -with the existing access rule for superclasses. It may be desireable +with the existing access rule for superclasses. It may be desirable to lift this restriction in a future proposal. ### `open` class members @@ -352,7 +352,7 @@ That is, a `public` class could be used as a compositional superclass, useful for adding new storage to an existing identity but not for messing with its invariants. This would prevent the creation of sealed hierarchies and is inconsistent with the general principle -that restrictons on future evolution should be opt-in. Authors would +that restrictions on future evolution should be opt-in. Authors would have no ability to reserve the right to decide later whether to allow subclasses; declaring something `final` is irrevocable. This could be added in a future extension, but it is not the right rule @@ -396,7 +396,7 @@ We may want to reconsider the need for `final` in the light of this change. ## Impact on existing code This would be a backwards-breaking change for all classes and methods that are -public and non-final, which code outside of their module has overriden. +public and non-final, which code outside of their module has overridden. Those classes/methods would fail to compile. Their superclass would need to be changed to `open`. diff --git a/proposals/0118-closure-parameter-names-and-labels.md b/proposals/0118-closure-parameter-names-and-labels.md index 39fff01a3f..2f89190564 100644 --- a/proposals/0118-closure-parameter-names-and-labels.md +++ b/proposals/0118-closure-parameter-names-and-labels.md @@ -2,14 +2,14 @@ * Proposal: [SE-0118](0118-closure-parameter-names-and-labels.md) * Authors: [Dave Abrahams](https://github.com/dabrahams), [Dmitri Gribenko](https://github.com/gribozavr), [Maxim Moiseev](https://github.com/moiseev) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000230.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0118-closure-parameter-names-and-labels/3387) * Bug: [SR-2072](https://bugs.swift.org/browse/SR-2072) ## Revision History -- [v1](https://github.com/apple/swift-evolution/blob/ae4a55ab217cc9755004cbf2b29db24e28645d15/proposals/0118-closure-parameter-names-and-labels.md) (as proposed) +- [v1](https://github.com/swiftlang/swift-evolution/blob/ae4a55ab217cc9755004cbf2b29db24e28645d15/proposals/0118-closure-parameter-names-and-labels.md) (as proposed) - v2: fixed spelling of identifiers containing `Utf8` to read `UTF8` per convention. ## Introduction @@ -18,10 +18,10 @@ We propose a revision to the names and argument labels of closure parameters in standard library APIs. Swift-evolution thread: -[Take 2: Stdlib closure argument labels and parameter names](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/022612.html) +[Take 2: Stdlib closure argument labels and parameter names](https://forums.swift.org/t/take-2-stdlib-closure-argument-labels-and-parameter-names/3180) Discussion of earlier revision of the proposal: -[Stdlib closure argument labels and parameter names](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021470.html) +[Stdlib closure argument labels and parameter names](https://forums.swift.org/t/stdlib-closure-argument-labels-and-parameter-names/3046) ## Motivation diff --git a/proposals/0119-extensions-access-modifiers.md b/proposals/0119-extensions-access-modifiers.md index 40a8e311c0..9112147c9a 100644 --- a/proposals/0119-extensions-access-modifiers.md +++ b/proposals/0119-extensions-access-modifiers.md @@ -2,21 +2,21 @@ * Proposal: [SE-0119](0119-extensions-access-modifiers.md) * Author: [Adrian Zubarev](https://github.com/DevAndArtist) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000250.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0119-remove-access-modifiers-from-extensions/3493) ## Introduction -

One great goal for Swift 3 is to sort out any source breaking language changes. This proposal aims to fix access modifier inconsistency on extensions compared to other scope declarations types.

+One great goal for Swift 3 is to sort out any source breaking language changes. This proposal aims to fix access modifier inconsistency on extensions compared to other scope declarations types. -Swift-evolution thread: [\[Proposal\] Revising access modifiers on extensions](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/022144.html) +Swift-evolution thread: [\[Proposal\] Revising access modifiers on extensions](https://forums.swift.org/t/proposal-revising-access-modifiers-on-extensions/3138) ## Motivation -

The access control of classes, enums and structs in Swift is very easy to learn and memorize. It also disallows to suppress the access modifier of implemented conformance members to lower access modifier if the host type has an access modifier of higher or equal level.

+The access control of classes, enums and structs in Swift is very easy to learn and memorize. It also disallows to suppress the access modifier of implemented conformance members to lower access modifier if the host type has an access modifier of higher or equal level. -
`public` > `internal` > `fileprivate` >= `private`
+`public` > `internal` > `fileprivate` >= `private` ```swift public class A { @@ -56,7 +56,7 @@ This simple access control model also allows us to nest types inside each other *Extensions* however behave differently when it comes to their access control: -* The *access modifier* of an *extension* sets the default modifier of its members which do not have their own localy defined modifier. +* The *access modifier* of an *extension* sets the default modifier of its members which do not have their own locally defined modifier. ```swift public struct D {} @@ -102,7 +102,7 @@ public protocol SomeProtocol {} public extension A : SomeProtocol {} ``` -*Extensions* are also used for *protocol default implementations* in respect to the mentioned rules. That means that if someone would want to provide a public default implementation for a specific protocol there are three different ways to achive this goal: +*Extensions* are also used for *protocol default implementations* in respect to the mentioned rules. That means that if someone would want to provide a public default implementation for a specific protocol there are three different ways to achieve this goal: ```swift public protocol G { @@ -150,7 +150,7 @@ I propose to revise the access control on extensions by removing access modifier > That way, access for members follows the same defaults as in the original type. > -> [Jordan Rose](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/022341.html) +> [Jordan Rose](https://forums.swift.org/t/proposal-revising-access-modifiers-on-extensions/3138/15) * It would be possible to conform types to a protocol using an *extension* which has an explicit *access modifier*. The *access modifier* respects the modifier of the extended type and the protocol to which it should be conformed. @@ -234,7 +234,7 @@ I propose to revise the access control on extensions by removing access modifier fileprivate group { - // Every group memebr is `fileprivate` + // Every group member is `fileprivate` func member1() {} func member2() {} func member3() {} @@ -243,7 +243,7 @@ I propose to revise the access control on extensions by removing access modifier } } ``` - Such a mechanism could also be used outside extensions! This idea has its own discussion [thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/022644.html). + Such a mechanism could also be used outside extensions! This idea has its own discussion [thread](https://forums.swift.org/t/post-swift-3-proposal-introducing-group-mechanism/3196). ## Proposed solution @@ -286,7 +286,7 @@ Iff the *access-level-modifier* is not present, the access modifier on extension ```diff - extension SomeType : SomeProtocol { + public extension SomeType : SomeProtocol { - public func someMemeber() + public func someMember() } ``` diff --git a/proposals/0120-revise-partition-method.md b/proposals/0120-revise-partition-method.md index 223d4ecb92..a222aaaa45 100644 --- a/proposals/0120-revise-partition-method.md +++ b/proposals/0120-revise-partition-method.md @@ -2,17 +2,17 @@ * Proposal: [SE-0120](0120-revise-partition-method.md) * Authors: [Lorenzo Racca](https://github.com/lorenzoracca), [Jeff Hajewski](https://github.com/j-haj), [Nate Cook](https://github.com/natecook1000) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000242.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0120-revise-partition-method-signature/3475) * Bug: [SR-1965](https://bugs.swift.org/browse/SR-1965) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/1dcfd35856a6f9c86af2cf7c94a9ab76411739e3/proposals/0120-revise-partition-method.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/1dcfd35856a6f9c86af2cf7c94a9ab76411739e3/proposals/0120-revise-partition-method.md) ## Introduction This proposal revises the signature for the collection partition algorithm. Partitioning is a foundational API for sorting and for searching through sorted collections. -- Swift-evolution thread: [Feedback from standard library team](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160502/016729.html) +- Swift-evolution thread: [Feedback from standard library team](https://forums.swift.org/t/review-se-0074-implementation-of-binary-search-functions/2438/5) - Swift Bug: [SR-1965](https://bugs.swift.org/browse/SR-1965) ## Motivation diff --git a/proposals/0121-remove-optional-comparison-operators.md b/proposals/0121-remove-optional-comparison-operators.md index ed5ec55e53..45073740df 100644 --- a/proposals/0121-remove-optional-comparison-operators.md +++ b/proposals/0121-remove-optional-comparison-operators.md @@ -2,14 +2,14 @@ * Proposal: [SE-0121](0121-remove-optional-comparison-operators.md) * Author: [Jacob Bandes-Storch](https://github.com/jtbandes) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000245.html) -* Pull Request: [apple/swift#3637](https://github.com/apple/swift/pull/3637) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0121-remove-optional-comparison-operators/3478) +* Implementation: [apple/swift#3637](https://github.com/apple/swift/pull/3637) ## Introduction -Swift's [`Comparable` protocol](https://developer.apple.com/reference/swift/comparable) requires 4 operators, [`<`, `<=`, `>`, and `>=`](https://github.com/apple/swift/blob/master/stdlib/public/core/Policy.swift#L729-L763), beyond the requirements of Equatable. +Swift's [`Comparable` protocol](https://developer.apple.com/reference/swift/comparable) requires 4 operators, [`<`, `<=`, `>`, and `>=`](https://github.com/apple/swift/blob/5868f9c597088793f7131d4655dd0f702a04dea3/stdlib/public/core/Policy.swift#L729-L763), beyond the requirements of Equatable. The standard library [additionally defines](https://github.com/apple/swift/blob/2a545eaa1bfd7d058ef491135cca270bc8e4be5f/stdlib/public/core/Optional.swift#L383-L419) the following 4 variants, which accept operands of Optional type, with the semantics that `.none < .some(_)`: @@ -23,9 +23,9 @@ public func >= (lhs: T?, rhs: T?) -> Bool This proposal removes the above 4 functions. swift-evolution discussion threads: -- [Optional comparison operators](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160711/024121.html) -- [Possible bug with arithmetic optional comparison ?](https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20160523/002095.html) -- [? suffix for <, >, <=, >= comparisons with optionals to prevent subtle bugs](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001264.html) +- [Optional comparison operators](https://forums.swift.org/t/optional-comparison-operators/3320) +- [Possible bug with arithmetic optional comparison ?](https://forums.swift.org/t/possible-bug-with-arithmetic-optional-comparison/2749) +- [? suffix for <, >, <=, >= comparisons with optionals to prevent subtle bugs](https://forums.swift.org/t/suffix-for-comparisons-with-optionals-to-prevent-subtle-bugs/350) ## Motivation @@ -47,7 +47,7 @@ a < b // b is coerced from "Int" to "Int?" to match the parameter type. [SE-0123](0123-disallow-value-to-optional-coercion-in-operator-arguments.md) seeks to remove this coercion (for arguments to operators) for a variety of reasons. -If the coercion is not removed (if no change is made), the results of comparisons with Optional values are sometimes **surprising**, making it easy to write bugs. In a thread from December 2015, [Al Skipp offers](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001267.html) the following example: +If the coercion is not removed (if no change is made), the results of comparisons with Optional values are sometimes **surprising**, making it easy to write bugs. In a thread from December 2015, [Al Skipp offers](https://forums.swift.org/t/suffix-for-comparisons-with-optionals-to-prevent-subtle-bugs/350/3) the following example: ```swift struct Pet { diff --git a/proposals/0122-use-colons-for-subscript-type-declarations.md b/proposals/0122-use-colons-for-subscript-type-declarations.md index 378c8ce7ea..cfabbd5dbd 100644 --- a/proposals/0122-use-colons-for-subscript-type-declarations.md +++ b/proposals/0122-use-colons-for-subscript-type-declarations.md @@ -2,9 +2,9 @@ * Proposal: [SE-0122](0122-use-colons-for-subscript-type-declarations.md) * Author: [James Froggatt](https://github.com/MutatingFunk) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decison Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000258.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0122-use-colons-for-subscript-declarations/3545) ## Introduction @@ -19,7 +19,7 @@ subscript(externalName internalName: ParamType) -> ElementType { The initial keyword `subscript` is followed by a parameter list, followed by an arrow to the accessed type. This proposal is to replace the arrow with a colon, to match accessor declarations elsewhere in the language. -Swift-evolution thread: [Discussion thread topic for that proposal](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160704/023883.html) +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/change-subscripts-to-use-colons/3346) ## Motivation diff --git a/proposals/0123-disallow-value-to-optional-coercion-in-operator-arguments.md b/proposals/0123-disallow-value-to-optional-coercion-in-operator-arguments.md index 0d27ca6a9e..29ecfb3225 100644 --- a/proposals/0123-disallow-value-to-optional-coercion-in-operator-arguments.md +++ b/proposals/0123-disallow-value-to-optional-coercion-in-operator-arguments.md @@ -2,9 +2,9 @@ * Proposal: [SE-0123](0123-disallow-value-to-optional-coercion-in-operator-arguments.md) * Authors: [Mark Lacey](https://github.com/rudkx), [Doug Gregor](https://github.com/DougGregor), [Jacob Bandes-Storch](https://github.com/jtbandes) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000246.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0123-disallow-coercion-to-optionals-in-operator-arguments/3479) ## Introduction @@ -46,7 +46,7 @@ This coercion happens for normal function calls, the assignment statement, and for operators defined with optional parameter types, e.g. the comparison operators and the nil-coalescing operator (`??`). -Swift-evolution thread: [Optional comparison operators](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160711/024121.html) +Swift-evolution thread: [Optional comparison operators](https://forums.swift.org/t/optional-comparison-operators/3320) ## Proposal diff --git a/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md b/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md index 56b80bca4c..892a806a96 100644 --- a/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md +++ b/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md @@ -2,9 +2,9 @@ * Proposal: [SE-0124](0124-bitpattern-label-for-int-initializer-objectidentfier.md) * Author: [Arnold Schwaighofer](https://github.com/aschwaighofer) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000241.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0124-int-init-objectidentifier-and-uint-init-objectidentifier-should-have-a-bitpattern-label/3474) * Bug: [SR-2064](https://bugs.swift.org/browse/SR-2064) ## Introduction @@ -19,7 +19,7 @@ as a bit pattern. } ``` -- Swift-evolution thread: [Pitch](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160711/024323.html) +- Swift-evolution thread: [Pitch](https://forums.swift.org/t/pitch-int-init-objectidentifier-and-uint-init-objectidentifier-should-have-a-bitpattern-label/3384) - Swift Bug: [SR-2064](https://bugs.swift.org/browse/SR-2064) - Branch with change to stdlib: [int_init_objectidentifier_label] (https://github.com/aschwaighofer/swift/tree/int_init_objectidentifier_label) diff --git a/proposals/0125-remove-nonobjectivecbase.md b/proposals/0125-remove-nonobjectivecbase.md index 2a58572709..d9b887fed1 100644 --- a/proposals/0125-remove-nonobjectivecbase.md +++ b/proposals/0125-remove-nonobjectivecbase.md @@ -2,9 +2,9 @@ * Proposal: [SE-0125](0125-remove-nonobjectivecbase.md) * Author: [Arnold Schwaighofer](https://github.com/aschwaighofer) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000261.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0125-remove-nonobjectivecbase-and-isuniquelyreferenced/3548) * Bug: [SR-1962](http://bugs.swift.org/browse/SR-1962) ## Introduction @@ -15,14 +15,14 @@ Remove `NonObjectiveCBase` and `isUniquelyReferencedNonObjC(_ object: T)`. This replacement is as performant as the call to `isUniquelyReferenced` in cases where the compiler has static knowledge that the type of `object` is a native -Swift class and dyamically has the same semantics for native swift classes. +Swift class and dynamically has the same semantics for native swift classes. This change will remove surface API. Rename `isUniquelyReferencedNonObjC` to `isKnownUniquelyReferenced` and no longer promise to return false for `@objc` class instances. Cleanup the `ManagedBufferPointer` API by renaming `holdsUniqueReference` to `isUniqueReference` and removing `holdsUniqueOrPinnedReference`. -- Swift-evolution thread: [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160718/024806.html) +- Swift-evolution thread: [Review](https://forums.swift.org/t/review-se-0125-remove-nonobjectivecbase-and-isuniquelyreferenced/3462) - Branch with change to stdlib: [remove_nonobjectivecbase_2] (https://github.com/aschwaighofer/swift/commits/remove_nonobjectivecbase_2) @@ -91,7 +91,7 @@ Rename `ManagedBufferPointer.holdsUniqueReference` to `ManagedBufferPointer.isUniqueReference` to avoid confusion. Remove `ManagedBufferPointer.holdsUniqueOrPinnedReference` because there is no -public pinning API so having this public API is not neccessary. +public pinning API so having this public API is not necessary. ## Detailed design diff --git a/proposals/0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md b/proposals/0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md index a32284d1fd..6901aa0b6e 100644 --- a/proposals/0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md +++ b/proposals/0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md @@ -2,9 +2,9 @@ * Proposal: [SE-0126](0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md) * Authors: [Adrian Zubarev](https://github.com/DevAndArtist), [Anton Zhilin](https://github.com/Anton3) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Withdrawn** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000251.html) +* Decision Notes: [Rationale](https://forums.swift.org/t/withdrawn-for-revision-se-0126-refactor-metatypes-repurpose-t-self-and-mirror/3499) ## Introduction @@ -12,9 +12,9 @@ This proposal wants to revise metatypes `T.Type`, repurpose *public* `T.self` no Swift-evolution threads: -* [\[Proposal\] Refactor Metatypes, repurpose T[dot]self and Mirror](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160718/024772.html) -* [\[Discussion\] Seal `T.Type` into `Type`](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160704/023818.html) -* [\[Discussion\] Can we make `.Type` Hashable?](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160627/023067.html) +* [\[Proposal\] Refactor Metatypes, repurpose T[dot]self and Mirror](https://forums.swift.org/t/proposal-refactor-metatypes-repurpose-t-dot-self-and-mirror/3460) +* [\[Discussion\] Seal `T.Type` into `Type`](https://forums.swift.org/t/discussion-seal-t-type-into-type-t/3340) +* [\[Discussion\] Can we make `.Type` Hashable?](https://forums.swift.org/t/discussion-can-we-make-type-hashable/3232) GitHub Gist thread: diff --git a/proposals/0127-cleaning-up-stdlib-ptr-buffer.md b/proposals/0127-cleaning-up-stdlib-ptr-buffer.md index de5932c659..f00bb6631a 100644 --- a/proposals/0127-cleaning-up-stdlib-ptr-buffer.md +++ b/proposals/0127-cleaning-up-stdlib-ptr-buffer.md @@ -2,9 +2,9 @@ * Proposal: [SE-0127](0127-cleaning-up-stdlib-ptr-buffer.md) * Author: [Charlie Monroe](https://github.com/charlieMonroe) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000262.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0127-cleaning-up-stdlib-pointer-and-buffer-routines/3549) * Bugs: [SR-1937](https://bugs.swift.org/browse/SR-1937), [SR-1955](https://bugs.swift.org/browse/SR-1955), [SR-1957](https://bugs.swift.org/browse/SR-1957) @@ -15,7 +15,7 @@ This proposal deals with three routines and one class related to pointers and bu The goal of this proposal is to update the API to match new API guidelines and remove redundant identifiers. -Swift-evolution thread: [Cleaning up stdlib Pointer and Buffer Routines](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160704/023518.html) +Swift-evolution thread: [Cleaning up stdlib Pointer and Buffer Routines](https://forums.swift.org/t/discussion-cleaning-up-stdlib-pointer-and-buffer-routines-open-issues-affecting-standard-library-api-stability/3295) ## Motivation @@ -61,13 +61,11 @@ withUnsafePointer(to: &x) { (ptrX) -> Void in --- -`unsafeAddressOf` is removed, in favor of adding a `unsafeAddress` field on `ObjectIdentifier`. -`ObjectIdentifier` already contains a raw pointer in the internal `_value` field and -can be initialized with `AnyObject` just like the argument of `unsafeAddressOf`. +`unsafeAddressOf` is removed. Suggested replacement is to use `Unmanged.passUnretained`. ``` let obj = NSObject() -let ptr = ObjectIdentifier(obj).unsafeAddress // instead of unsafeAddress(of: obj) +let ptr = Unmanaged.passUnretained(obj) // instead of unsafeAddress(of: obj) ``` --- @@ -82,7 +80,7 @@ will be moved onto `ManagedBuffer` instead. and the multi-pointer versions will need to be removed by the user and nested calls to single-pointer variants need to be used instead. -Use of `unsafeAddressOf(x)` will need to be changed to `ObjectIdentifier(x).unsafeAddress` +Use of `unsafeAddressOf(x)` will need to be changed to `Unmanaged.passUnretained(x)` instead. Since `ManagedProtoBuffer` doesn't have any accessible initializers, it can only be @@ -101,9 +99,4 @@ them as immutable (`let`). Discussion on the mailing list brought up two suggest - The second suggestion was to introduce two variants of `withUnsafePointer` - one that maintains current behavior and one that that doesn't require `inout` argument. This has been viewed on as an additive change not in scope for Swift 3. -- Remove `unsafeAddressOf` and use `Unmanaged.takeUnretainedValue(_:)` instead. This, -however, requires the caller to deal with retain logic for something as simple as -getting an object address. -- Alternative names for the `unsafeAddress` property on `ObjectIdentifier` - `value`, -`pointerValue`, `pointer`. - Instead of removing `ManagedProtoBuffer`, rename it to `ManagedBufferBase`. diff --git a/proposals/0128-unicodescalar-failable-initializer.md b/proposals/0128-unicodescalar-failable-initializer.md index 63e83f92c2..29e75877b8 100644 --- a/proposals/0128-unicodescalar-failable-initializer.md +++ b/proposals/0128-unicodescalar-failable-initializer.md @@ -2,10 +2,10 @@ * Proposal: [SE-0128](0128-unicodescalar-failable-initializer.md) * Author: [Xin Tong](https://github.com/trentxintong) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000259.html) -* Pull Request: [apple/swift#3662](https://github.com/apple/swift/pull/3662) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0128-change-failable-unicodescalar-initializers-to-failable/3546) +* Implementation: [apple/swift#3662](https://github.com/apple/swift/pull/3662) ## Introduction diff --git a/proposals/0129-package-manager-test-naming-conventions.md b/proposals/0129-package-manager-test-naming-conventions.md index a82e746afa..f3e218b70c 100644 --- a/proposals/0129-package-manager-test-naming-conventions.md +++ b/proposals/0129-package-manager-test-naming-conventions.md @@ -3,8 +3,8 @@ * Proposal: [SE-0129](0129-package-manager-test-naming-conventions.md) * Author: [Anders Bertelrud](https://github.com/abertelrud) * Review Manager: [Daniel Dunbar](https://github.com/ddunbar) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-build-dev/Week-of-Mon-20160725/000572.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0129-package-manager-test-naming-conventions/3574) ## Introduction diff --git a/proposals/0130-string-initializers-cleanup.md b/proposals/0130-string-initializers-cleanup.md index b2b3c18878..05a3e7f821 100644 --- a/proposals/0130-string-initializers-cleanup.md +++ b/proposals/0130-string-initializers-cleanup.md @@ -2,9 +2,9 @@ * Proposal: [SE-0130](0130-string-initializers-cleanup.md) * Author: Roman Levenstein -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000260.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0130-replace-repeating-character-and-unicodescalar-forms-of-string-init/3547) ## Introduction diff --git a/proposals/0131-anyhashable.md b/proposals/0131-anyhashable.md index 13555ffc69..aa7547d367 100644 --- a/proposals/0131-anyhashable.md +++ b/proposals/0131-anyhashable.md @@ -2,9 +2,9 @@ * Proposal: [SE-0131](0131-anyhashable.md) * Author: [Dmitri Gribenko](https://github.com/gribozavr) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000263.html) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0131-add-anyhashable-to-the-standard-library/3553) ## Introduction @@ -19,7 +19,7 @@ hashable values. From SE-0116: > hashable type that is itself `Hashable`, for use as the upper-bound > type of heterogeneous `Dictionary`s and `Set`s. -Swift-evolution thread: [Add AnyHashable to the standard library](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160718/025264.html). +Swift-evolution thread: [Add AnyHashable to the standard library](https://forums.swift.org/t/add-anyhashable-to-the-standard-library/3517). ## Motivation diff --git a/proposals/0132-sequence-end-ops.md b/proposals/0132-sequence-end-ops.md index 30b13f9d87..a9136f73e5 100644 --- a/proposals/0132-sequence-end-ops.md +++ b/proposals/0132-sequence-end-ops.md @@ -1,10 +1,10 @@ # Rationalizing Sequence end-operation names * Proposal: [SE-0132](0132-sequence-end-ops.md) -* Authors: [Brent Royal-Gordon](https://github.com/brentdax), [Dave Abrahams](https://github.com/dabrahams) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000267.html) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax), [Dave Abrahams](https://github.com/dabrahams) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Rejected** +* Decision Notes: [Rationale](https://forums.swift.org/t/deferred-se-0132-rationalizing-sequence-end-operation-names/3577) ## Introduction @@ -14,7 +14,7 @@ inconsistent naming which can make it difficult to find inverses or remember what the standard library offers. We propose that we standardize these names so they follow consistent, predictable patterns. -Swift-evolution thread: [[Draft] Rationalizing Sequence end-operation names](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160620/021872.html) +Swift-evolution thread: [[Draft] Rationalizing Sequence end-operation names](https://forums.swift.org/t/draft-rationalizing-sequence-end-operation-names/3103) ### Scope @@ -152,7 +152,7 @@ the direction, but do not indicate the direction in their names: Adding a direction to these APIs would make their behavior clearer and permit us to offer opposite-end equivalents in the future. (Unmerged -[swift-evolution pull request 329](https://github.com/apple/swift-evolution/pull/329) +[swift-evolution pull request 329](https://github.com/swiftlang/swift-evolution/pull/329) would add `lastIndex` methods.) ### Operations taking an index are really slicing diff --git a/proposals/0133-rename-flatten-to-joined.md b/proposals/0133-rename-flatten-to-joined.md index 583ef23d94..4408b1a775 100644 --- a/proposals/0133-rename-flatten-to-joined.md +++ b/proposals/0133-rename-flatten-to-joined.md @@ -2,12 +2,12 @@ * Proposal: [SE-0133](0133-rename-flatten-to-joined.md) * Author: [Jacob Bandes-Storch](https://github.com/jtbandes) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000265.html) -* Pull Requests: [apple/swift#3809](https://github.com/apple/swift/pull/3809), - [apple/swift#3838](https://github.com/apple/swift/pull/3838), - [apple/swift#3839](https://github.com/apple/swift/pull/3839) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0133-rename-flatten-to-joined/3575) +* Implementation: [apple/swift#3809](https://github.com/apple/swift/pull/3809), + [apple/swift#3838](https://github.com/apple/swift/pull/3838), + [apple/swift#3839](https://github.com/apple/swift/pull/3839) ## Introduction @@ -31,8 +31,8 @@ https://github.com/apple/swift/blob/c6e828f761fc30f7ce444431de7da52814f96595/std https://github.com/apple/swift/blob/f72a82327b172e1a2979e46cb7a579e3cc2f3bd6/stdlib/public/core/Flatten.swift.gyb Swift-evolution threads: -- [[Pitch] Unify joined(separator:) and flatten()](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160718/025136.html) -- [[Pitch] Rename flatten() to joined() and give joined() for string sequences the empty string as the default parameter](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160718/025234.html) +- [[Pitch] Unify joined(separator:) and flatten()](https://forums.swift.org/t/pitch-unify-joined-separator-and-flatten/3505) +- [[Pitch] Rename flatten() to joined() and give joined() for string sequences the empty string as the default parameter](https://forums.swift.org/t/pitch-rename-flatten-to-joined-and-give-joined-for-string-sequences-the-empty-string-as-the-default-parameter/3522) ## Motivation diff --git a/proposals/0134-rename-string-properties.md b/proposals/0134-rename-string-properties.md index 2e12c1545a..6d9abf3cd9 100644 --- a/proposals/0134-rename-string-properties.md +++ b/proposals/0134-rename-string-properties.md @@ -2,17 +2,17 @@ * Proposal: [SE-0134](0134-rename-string-properties.md) * Authors: [Xiaodi Wu](https://github.com/xwu), [Erica Sadun](https://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution-announce/2016-July/000266.html) -* Pull Request: [apple/swift#3816](https://github.com/apple/swift/pull/3816) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/aea8b836d21051076663c5692ec1d09bb3222527/proposals/0134-rename-string-properties.md) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0134-rename-two-utf8-related-properties-on-string/3576) +* Implementation: [apple/swift#3816](https://github.com/apple/swift/pull/3816) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/aea8b836d21051076663c5692ec1d09bb3222527/proposals/0134-rename-string-properties.md) ## Introduction This proposal removes `nulTerminatedUTF8` and renames `nulTerminatedUTF8CString` to enhance clarity and reduce mismatch between user expectations and the Swift programming language. -Swift-evolution thread: [Discussion thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160718/025378.html) +Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/draft-fix-a-typo-in-two-string-methods/3524) ## Motivation diff --git a/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md b/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md index fd5f610baf..ef8c7fbad4 100644 --- a/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md +++ b/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md @@ -3,8 +3,8 @@ * Proposal: [SE-0135](0135-package-manager-support-for-differentiating-packages-by-swift-version.md) * Author: [Anders Bertelrud](https://github.com/abertelrud) * Review Manager: [Daniel Dunbar](https://github.com/ddunbar) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160801/025955.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0135-package-manager-support-for-differentiating-packages-by-swift-version/3687) ## Introduction diff --git a/proposals/0136-memory-layout-of-values.md b/proposals/0136-memory-layout-of-values.md index 864fd48dc8..9705baaa85 100644 --- a/proposals/0136-memory-layout-of-values.md +++ b/proposals/0136-memory-layout-of-values.md @@ -3,15 +3,15 @@ * Proposal: [SE-0136](0136-memory-layout-of-values.md) * Author: [Xiaodi Wu](https://github.com/xwu) * Review Manager: [Dave Abrahams](https://github.com/dabrahams) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160808/026164.html) -* Pull Request: [apple/swift#4041](https://github.com/apple/swift/pull/4041) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0136-memory-layout-of-values/3760) +* Implementation: [apple/swift#4041](https://github.com/apple/swift/pull/4041) ## Introduction This proposal is to introduce, as a bugfix, a replacement for `sizeofValue(_:)` and related functions. -Swift-evolution thread: [MemoryLayout for a value](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160801/025890.html) +Swift-evolution thread: [MemoryLayout for a value](https://forums.swift.org/t/memorylayout-for-a-value/3671) ## Motivation diff --git a/proposals/0137-avoiding-lock-in.md b/proposals/0137-avoiding-lock-in.md index 7cc0ec72a0..43136b648a 100644 --- a/proposals/0137-avoiding-lock-in.md +++ b/proposals/0137-avoiding-lock-in.md @@ -3,15 +3,15 @@ * Proposal: [SE-0137](0137-avoiding-lock-in.md) * Authors: [Dave Abrahams](https://github.com/dabrahams), [Dmitri Gribenko](https://github.com/gribozavr) * Review Manager: [John McCall](https://github.com/rjmccall) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160815/026300.html) +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0137-avoiding-lock-in-to-legacy-protocol-designs/3781) ## Introduction We propose to deprecate or move protocols that shouldn't be a part of the standard library's public API going forward. -Swift-evolution threads: [Late Pitch](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160808/026071.html), [Review](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160808/026103.html) +Swift-evolution threads: [Late Pitch](https://forums.swift.org/t/late-pitch-deprecations-moves-and-renames/3723), [Review](https://forums.swift.org/t/review-se-0137-avoiding-lock-in-to-legacy-protocol-designs/3739) ## Motivation diff --git a/proposals/0138-unsaferawbufferpointer.md b/proposals/0138-unsaferawbufferpointer.md index 4abcc1a4b1..b425d8356a 100644 --- a/proposals/0138-unsaferawbufferpointer.md +++ b/proposals/0138-unsaferawbufferpointer.md @@ -2,8 +2,9 @@ * Proposal: [SE-0138](0138-unsaferawbufferpointer.md) * Author: [Andrew Trick](https://github.com/atrick) -* Status: **Active Review (August 31...September 14)** * Review manager: [Dave Abrahams](https://github.com/dabrahams) +* Status: **Implemented (Swift 3.0.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0138-unsaferawbufferpointer/4063) Contents: - [Introduction](#introduction) @@ -31,7 +32,7 @@ for binding memory to a type for subsequent normal typed access. However, migration is not always straightforward because SE-0107 provided only minimal support for raw pointers. Extending raw pointer support to the `UnsafeBufferPointer` type will fill in this -funcionality gap. This is especially important for code that currently +functionality gap. This is especially important for code that currently views "raw" bytes of memory as `UnsafeBufferPointer`. Converting between `UInt8` and the client's element type at every API transition is difficult to do @@ -41,12 +42,8 @@ changing the type the represents a view into raw bytes to [UnsafeRawPointer Migration Guide](https://swift.org/migration-guide/se-0107-migrate.html). Swift-evolution threads: -- [Week #1](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160808/thread.html#26173) -- [Week #2](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160815/thread.html#26254) -- [Week #3](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160822/thread.html#26553) -- [Week #4 (1)](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160829/thread.html#26812) -- [Week #4 (2)](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160829/thread.html#26844) -- [Week #5](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160905/thread.html#26947) +- [Pitch](https://forums.swift.org/t/late-pitch-unsafebytes-proposal/3762) +- [Review thread 1](https://forums.swift.org/t/se-0138-unsafebytes/3926), [Review thread 2](https://forums.swift.org/t/review-se-0138-unsafebytes/3917) ## Motivation @@ -72,7 +69,7 @@ is natural for the same type that encapsulates a raw pointer and length to also allow clients to view that memory as raw bytes without the need to explicitly bind the memory type each time memory is accessed. This would also improve performance in some cases that I've -encoutered by avoiding array copies. Let's call this new type +encountered by avoiding array copies. Let's call this new type `Unsafe[Mutable]RawBufferPointer`. Any array could be viewed as `UnsafeRawBufferPointer`, and that raw @@ -153,6 +150,57 @@ Add an `Array.withUnsafe[Mutable]Bytes(_)` method that passes an Add a `withUnsafeMutableBytes(of:_)` function that passes an `UnsafeRawBufferPointer` view of a value of type `T` to the closure body. +## Amendment to normalize the slice type + +The original version of this proposal defined +`Unsafe[Mutable]BufferPointer` to be its own `SubSequence` type. This +is the `Collection`'s slice type returned by a range subscript +getter. Using a single type for buffers and buffer slices is very +convenient for working with flat memory regions, but it is +inconsistent with Swift's `Collection` semantics. The problem is that +it transparently rebases the slice's `Int`-type indices to zero. This +has the potential to break generic algorithms, which expect to use the +same indices across both the original `Collection` and its slices. + +The amended version of this proposal changes `SubSequence` to `[Mutable]RandomAccessSlice` + +`rebasing` initializers have been added to allow explicit conversion +from a slice to a zero-based `Unsafe[Mutable]RawBufferPointer`. + +This results in the following behavioral changes: + +Passing a region within buffer to another function that takes a buffer +can no longer be done via subscript: + +Incorrect: `takesRawBuffer(buffer[i.. /// An iterator for the bytes referenced by `${Self}`. public struct Iterator : IteratorProtocol, Sequence { @@ -865,6 +914,44 @@ public struct Unsafe${Mutable}RawBufferPointer } % end # !mutable +% if not mutable: + /// Creates a raw buffer over the same memory as the given raw buffer slice. + /// + /// The new raw buffer will represent the same region of memory as the slice, + /// but it's indices will be rebased to zero. Given: + /// + /// let slice = buffer[n..) { + self.init(start: slice.base.baseAddress! + slice.startIndex, + count: slice.count) + } +% end # !mutable + + /// Creates a raw buffer over the same memory as the given raw buffer slice. + /// + /// The new raw buffer will represent the same region of memory as the slice, + /// but it's indices will be rebased to zero. Given: + /// + /// let slice = buffer[n.. + ) { + self.init(start: slice.base.baseAddress! + slice.startIndex, + count: slice.count) + } + /// Always zero, which is the index of the first byte in a /// non-empty buffer. public var startIndex: Int { @@ -904,16 +991,14 @@ public struct Unsafe${Mutable}RawBufferPointer /// Accesses the bytes in the memory region within `bounds` as a `UInt8` /// values. - public subscript(bounds: Range) -> Unsafe${Mutable}RawBufferPointer { + public subscript(bounds: Range) -> ${Mutable}RandomAccessSlice { get { _debugPrecondition(bounds.lowerBound >= startIndex) _debugPrecondition(bounds.upperBound <= endIndex) - return Unsafe${Mutable}RawBufferPointer( - start: baseAddress.map { $0 + bounds.lowerBound }, - count: bounds.count) + return ${Mutable}RandomAccessSlice(base: self, bounds: bounds) } % if mutable: - set { + nonmutating set { _debugPrecondition(bounds.lowerBound >= startIndex) _debugPrecondition(bounds.upperBound <= endIndex) _debugPrecondition(bounds.count == newValue.count) @@ -963,12 +1048,12 @@ extension Unsafe${Mutable}RawBufferPointer : CustomDebugStringConvertible { % if mutable: public func withUnsafeMutableBytes( of arg: inout T, - _ body: (inout UnsafeMutableRawBufferPointer) throws -> Result + _ body: (UnsafeMutableRawBufferPointer) throws -> Result ) rethrows -> Result { return try withUnsafeMutablePointer(to: &arg) { - var bytes = UnsafeMutableRawBufferPointer(start: $0, count: MemoryLayout.size) - return try body(&bytes) + return try body(UnsafeMutableRawBufferPointer( + start: $0, count: MemoryLayout.size)) } } % else: @@ -1012,11 +1097,10 @@ extension ${Self} { /// /// - SeeAlso: `withUnsafeBytes`, `UnsafeRawBufferPointer` public mutating func withUnsafeMutableBytes( - _ body: (inout UnsafeMutableRawBufferPointer) throws -> R + _ body: (UnsafeMutableRawBufferPointer) throws -> R ) rethrows -> R { return try self.withUnsafeMutableBufferPointer { - var bytes = UnsafeMutableRawBufferPointer($0) - return try body(&bytes) + return try body(UnsafeMutableRawBufferPointer($0)) } } @@ -1042,7 +1126,7 @@ extension ${Self} { /// - Returns: The return value of the `body` closure parameter, if any. /// /// - SeeAlso: `withUnsafeBytes`, `UnsafeRawBufferPointer` - public mutating func withUnsafeBytes( + public func withUnsafeBytes( _ body: (UnsafeRawBufferPointer) throws -> R ) rethrows -> R { return try self.withUnsafeBufferPointer { @@ -1051,6 +1135,50 @@ extension ${Self} { } } %end + +% for mutable in (True, False): +% Mutable = 'Mutable' if mutable else '' + +extension Unsafe${Mutable}BufferPointer { + +% if not Mutable: + /// Creates a buffer over the same memory as the given buffer slice. + /// + /// The new buffer will represent the same region of memory as the slice, + /// but it's indices will be rebased to zero. Given: + /// + /// let slice = buffer[n..>) { + self.init(start: slice.base.baseAddress! + slice.startIndex, + count: slice.count) + } +% end # !mutable + + /// Creates a buffer over the same memory as the given buffer slice. + /// + /// The new buffer will represent the same region of memory as the slice, + /// but it's indices will be rebased to zero. Given: + /// + /// let slice = buffer[n..> + ) { + self.init(start: slice.base.baseAddress! + slice.startIndex, + count: slice.count) + } +} +% end ``` ## Implementation status diff --git a/proposals/0139-bridge-nsnumber-and-nsvalue.md b/proposals/0139-bridge-nsnumber-and-nsvalue.md index 6c680a1514..c6d3a65d1b 100644 --- a/proposals/0139-bridge-nsnumber-and-nsvalue.md +++ b/proposals/0139-bridge-nsnumber-and-nsvalue.md @@ -3,8 +3,8 @@ * Proposal: [SE-0139](0139-bridge-nsnumber-and-nsvalue.md) * Author: [Joe Groff](https://github.com/jckarter) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Accepted** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160912/027060.html) +* Status: **Implemented (Swift 3.0.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0139-bridge-numeric-types-to-nsnumber-and-cocoa-structs-to-nsvalue/4004) ## Introduction @@ -13,7 +13,7 @@ into Objective-C object contexts. We should extend this bridging behavior to all Swift numeric types. We should also bridge common Cocoa structs such as `NSRange` by boxing them into `NSValue` objects. -Swift-evolution thread: [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160822/026560.html) +Swift-evolution thread: [here](https://forums.swift.org/t/proposal-draft-bridge-numeric-types-to-nsnumber-and-cocoa-structs-to-nsvalue/3841) ## Motivation @@ -29,12 +29,12 @@ as `NSNumber`, other-sized numeric types fall back to opaque boxing: let i = 17 let plist = ["seventeen": i] // OK -try! NSJSONSerialization.data(withJSONObject: plist) +try! JSONSerialization.data(withJSONObject: plist) let j: UInt8 = 38 let brokenPlist = ["thirty-eight": j] // Will throw because `j` didn't bridge to a JSON type -try! NSJSONSerialization.data(withJSONObject: brokenPlist) +try! JSONSerialization.data(withJSONObject: brokenPlist) ``` We had shied away from enabling this bridging for all numeric types in diff --git a/proposals/0140-bridge-optional-to-nsnull.md b/proposals/0140-bridge-optional-to-nsnull.md index 7fa37ec43a..e5eca4d8a5 100644 --- a/proposals/0140-bridge-optional-to-nsnull.md +++ b/proposals/0140-bridge-optional-to-nsnull.md @@ -3,8 +3,8 @@ * Proposal: [SE-0140](0140-bridge-optional-to-nsnull.md) * Author: [Joe Groff](https://github.com/jckarter) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Accepted** -* Decision Notes: [Rationale](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160912/027062.html) +* Status: **Implemented (Swift 3.0.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0140-bridge-optional-as-its-payload-or-nsnull/4005) ## Introduction @@ -42,7 +42,7 @@ ObjCClass().imported(s1) ObjCClass().imported(s2) ``` -Swift-evolution thread: [here](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160822/026561.html) +Swift-evolution thread: [here](https://forums.swift.org/t/proposal-draft-bridge-optional-as-its-payload-or-nsnull/3842) ## Motivation @@ -195,9 +195,11 @@ error, so should fail early at runtime: This point of view is understandable, but is inconsistent with how Swift itself dynamically treats Optionals inside Anys: +```swift let a: Int? = 3 let b = a as Any let c = a as! Int // Casts '3' out of the Optional as a non-optional Int +``` And while it's true that Cocoa uses `NSNull` sparingly, it *is* the standard sentinel used in the few places where a null-like object is expected, such as diff --git a/proposals/0141-available-by-swift-version.md b/proposals/0141-available-by-swift-version.md new file mode 100644 index 0000000000..b3bc4fda3e --- /dev/null +++ b/proposals/0141-available-by-swift-version.md @@ -0,0 +1,115 @@ +# Availability by Swift version + +* Proposal: [SE-0141](0141-available-by-swift-version.md) +* Author: [Graydon Hoare](https://github.com/graydon) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 3.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0141-availability-by-swift-version/4190) +* Bug: [SR-2709](https://bugs.swift.org/browse/SR-2709) + +## Introduction + +Swift's existing `@available(...)` attribute indicates the lifecycle of a +given declaration, either unconditionally or relative to a particular +platform or OS version range. + +It does not currently support indicating declaration lifecycle relative to +Swift language versions. This proposal seeks to extend it to do so. + +Swift-evolution threads: +[Draft](https://forums.swift.org/t/draft-availability-by-swift-version/4083), +[Review](https://forums.swift.org/t/review-se-0141-availability-by-swift-version/4096) + +## Motivation + +As the Swift language progresses from one version to the next, some +declarations will be added, renamed, deprecated or removed from the +standard library. Existing code written for earlier versions of Swift will +be supported through a `-swift-version N` command-line flag, that runs the +compiler in a backward-compatibility mode for the specified "effective" +language version. + +When running in a backward-compatibility mode, the set of available +standard library declarations should change to match expectations of older +code. Currently the only mechanism for testing a language version is the +compiler-control statement `#if swift(>= N)` which is a static construct: +it can be used to compile-out a declaration from the standard library, but +evolving the standard library through this mechanism would necessitate +compiling the standard library once for each supported older language +version. + +It would be preferable to compile the standard library _once_ for all +supported language versions, but make declarations _conditionally +available_ depending on the effective language version of a _user_ of the +library. The existing `@available(...)` attribute is similar to this +use-case, and this proposal seeks to extend the attribute to support it. + +## Proposed solution + +The `@available(...)` attribute will be extended to support specifying +`swift` version numbers, in addition to its existing platform versions. + +As an example, an API that is removed in Swift 3.1 will be written +as: + +~~~~ +@available(swift, obsoleted: 3.1) +class Foo { + //... +} +~~~~ + +When compiling _user code_ in `-swift-version 3.0` mode, this declaration +would be available, but not when compiling in subsequent versions. + +## Detailed design + +The token `swift` will be added to the set of valid initial arguments +to the `@available(...)` attribute. It will be treated similarly, +but slightly differently, than the existing platform arguments. In +particular: + + - As with platform-based availability judgments, a declaration's + `swift` version availability will default to available-everywhere + if unspecified. + + - A declaration's `swift` version availability will be considered + in logical conjunction with its platform-based availability. + That is, a given declaration will be available if and only + if it is _both_ available to the current effective `swift` version + _and_ available to the current deployment-target platform. + + - Similar to the abbreviated form of platform availability, an + abbreviated form `@available(swift N)` will be permitted as a synonym + for `@available(swift, introduced: N)`. However, adding `swift` to + a platform availability abbreviation list will not be allowed. That is, + writing the following examples is not permitted: + + - `@available(swift 3, *)` + - `@available(swift 3, iOS 10, *)` + + This restriction is due to the fact that platform-availability lists + are interpreted disjunctively (as a logical-_OR_ of their arguments), + and adding a conjunct (logical-_AND_) to such a list would make + the abbreviation potentially ambiguous to readers. + +## Impact on existing code + +Existing code does not use this form of attribute, so will not be +affected at declaration-site. + +As declarations are annotated as unavailable or obsoleted via +this attribute, some user code may stop working, but the same risk exists +(with a worse user experience) in today's language any time declarations +are removed or conditionally-compiled out. The purpose of this proposal +is to provide a better user experience around such changes, and facilitate +backward-compatibility modes. + +## Alternatives considered + +The main alternative is compiling libraries separately for each language +version and using `#if swift(>=N)` to conditionally include varying APIs. +For a library used locally within a single project, recompiling for a +specific language version may be appropriate, but for shipping the standard +library it is more economical to compile once with all declarations, and +select a subset based on language version. diff --git a/proposals/0142-associated-types-constraints.md b/proposals/0142-associated-types-constraints.md new file mode 100644 index 0000000000..2c561ef782 --- /dev/null +++ b/proposals/0142-associated-types-constraints.md @@ -0,0 +1,106 @@ +# Permit where clauses to constrain associated types + +* Proposal: [SE-0142](0142-associated-types-constraints.md) +* Authors: [David Hart](https://github.com/hartbit), [Jacob Bandes-Storch](https://github.com/jtbandes), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0142-permit-where-clauses-to-constrain-associated-types/4191) +* Bugs: [SR-4506](https://bugs.swift.org/browse/SR-4506) + +## Introduction + +This proposal seeks to introduce a `where` clause to associated type +declarations and improvements to protocol constraints to bring associated types +the same expressive power as generic type parameters. + +This proposal was discussed twice on the Swift Evolution list in the following +threads: + +* [\[Completing Generics\] Arbitrary requirements in protocols](https://forums.swift.org/t/completing-generics-arbitrary-requirements-in-protocols/2135) +* [\[Proposal\] More Powerful Constraints for Associated Types](https://forums.swift.org/t/proposal-more-powerful-constraints-for-associated-types/2328) + +## Motivation + +Currently, associated type declarations can only express simple inheritance +constraints and not the more sophisticated constraints available to generic +types with the `where` clause. Some designs, including many in the Standard +Library, require more powerful constraints for associated types to be truly +elegant. For example, the `SequenceType` protocol could be declared as follows +if the current proposal was accepted: + +```swift +protocol Sequence { + associatedtype Iterator : IteratorProtocol + associatedtype SubSequence : Sequence where SubSequence.Iterator.Element == Iterator.Element + ... +} +``` + +## Detailed Design + +First of all, this proposal modifies the grammar for a protocol's associated types +to the following: + +*protocol-associated-type-declaration* → + *attributesopt* + *access-level-modifieropt* + **associatedtype** + *typealias-name* + ­*type-inheritance-clause­opt* + *typealias-assignment­opt* + *requirement-clauseopt* + +The new requirement-clause is then used by the compiler to validate the +associated types of conforming types. + +Secondly, the proposal also allows protocols to use the associated types of +their conforming protocols in their declaration `where` clause as below: + +```swift +protocol IntSequence : Sequence where Iterator.Element == Int { + ... +} +``` + +Name lookup semantics in the protocol declaration `where` clause only looks at +associated types in the parent protocols. For example, the following code would +cause an error: + +```swift +protocol SomeSequence : Sequence where Counter : SomeProtocol { // error: Use of undefined associated type 'Counter' + associatedtype Counter +} +``` + +But instead should be written on the associated type itself: + +```swift +protocol IntSequence : Sequence { + associatedtype Counter : SomeProtocol +} +``` + +## Effect on ABI Stability + +As mentioned previously, there are a number of places in the standard library where this feature would be adopted (such as the `SubSequence.Iterator.Element == Iterator.Element` example), each of which will change the mangling of any generic function/type that makes use of them. + +## Alternatives + +Douglas Gregor argues that the proposed syntax is redundant when adding new +constraints to an associated type declared in a parent protocol and proposes +another syntax: + +```swift +protocol Collection : Sequence { + where SubSequence : Collection +} +``` + +But as Douglas notes himself, that syntax is ambiguous since we adopted the +generic `where` clause at the end of declarations of the following proposal: +[SE-0081: Move where clause to end of declaration](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0081-move-where-expression.md). For those reasons, it might be wiser not to introduce the shorthand syntax. + +## Acknowledgements + +Thanks to Dave Abrahams and Douglas Gregor for taking the time to help me +through this proposal. diff --git a/proposals/0143-conditional-conformances.md b/proposals/0143-conditional-conformances.md new file mode 100644 index 0000000000..f6cfaf5a66 --- /dev/null +++ b/proposals/0143-conditional-conformances.md @@ -0,0 +1,615 @@ +# Conditional conformances + +* Proposal: [SE-0143](0143-conditional-conformances.md) +* Author: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 4.2)** +* Decision Notes: [Review extended](https://forums.swift.org/t/review-se-0143-conditional-conformances/4130/10), [Rationale](https://forums.swift.org/t/accepted-se-0143-conditional-conformances/4537) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/91725ee83fa34c81942a634dcdfa9d2441fbd853/proposals/0143-conditional-conformances.md) + +## Introduction + +Conditional conformances express the notion that a generic type will +conform to a particular protocol only when its type arguments meet +certain requirements. For example, the `Array` collection can +implement the `Equatable` protocol only when its elements are +themselves `Equatable`, which can be expressed via the following +conditional conformance on `Equatable`: + +```swift +extension Array: Equatable where Element: Equatable { + static func ==(lhs: Array, rhs: Array) -> Bool { ... } +} +``` + +This feature is part of the [generics +manifesto](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#conditional-conformances-) +because it's something that fits naturally into the generics model and +is expected to have a high impact on the Swift standard library. + +Swift-evolution thread: [here](https://forums.swift.org/t/proposal-draft-conditional-conformances/4110) + +## Motivation + +Conditional conformances address a hole in the composability of the +generics system. Continuing the `Array` example from above, it's +always been the case that one could use the `==` operator on two +arrays of `Equatable` type, e.g., `[Int] == [Int]` would +succeed. However, it doesn't compose: arrays of arrays of `Equatable` +types cannot be compared (e.g., `[[Int]] == [[Int]]` will fail to +compile) because, even though there is an `==` for arrays of +`Equatable` type, the arrays themselves are never `Equatable`. + +Conditional conformances are particularly powerful when building +generic adapter types, which are intended to reflect the capabilities +of their type arguments. For example, consider the "lazy" +functionality of the Swift standard library's collections: using the +`lazy` member of a sequence produces a lazy adapter that conforms to +the `Sequence` protocol, while using the `lazy` member of a collection +produces a lazy adapter that conforms to the `Collection` +protocol. In Swift 3, the only way to model this is with different +types. For example, the Swift standard library has four similar +generic types to handle a lazy collection: `LazySequence`, +`LazyCollection`, `LazyBidirectionalCollection`, and +`LazyRandomAccessCollection`. The Swift standard library uses +overloading of the `lazy` property to decide among these: + +```swift +extension Sequence { + var lazy: LazySequence { ... } +} + +extension Collection { + var lazy: LazyCollection { ... } +} + +extension BidirectionalCollection { + var lazy: LazyBidirectionalCollection { ... } +} + +extension RandomAccessCollection { + var lazy: LazyRandomAccessCollection { ... } +} +``` + +This approach causes an enormous amount of repetition, and doesn't +scale well because each more-capable type has to re-implement (or +somehow forward the implementation of) all of the APIs of the +less-capable versions. With conditional conformances, one can provide +a single generic wrapper type whose basic requirements meet the lowest +common denominator (e.g., `Sequence`), but which scale their +capabilities with their type argument (e.g., the `LazySequence` +conforms to `Collection` when the type argument does, and so on). + +## Proposed solution +In a nutshell, the proposed solution is to allow a constrained +extension of a `struct`, `enum`, or `class` (but [not a protocol](#alternatives-considered)) to declare protocol +conformances. No additional syntax is necessary for this change, +because it already exists in the grammar; rather, this proposal +removes the limitation that results in the following error: + +``` +t.swift:1:1: error: extension of type 'Array' with constraints cannot have an inheritance clause +extension Array: Equatable where Element: Equatable { } +^ ~~~~~~~~~ +``` + +Conditional conformances can only be used when the additional +requirements of the constrained extension are satisfied. For example, +given the aforementioned `Array` conformance to `Equatable`: + +```swift +func f(_: T) { ... } + +struct NotEquatable { } + +func test(a1: [Int], a2: [NotEquatable]) { + f(a1) // okay: [Int] conforms to Equatable because Int conforms to Equatable + f(a2) // error: [NotEquatable] does not conform to Equatable because NotEquatable has no conformance to Equatable +} +``` + +Conditional conformances also have a run-time aspect, because a +dynamic check for a protocol conformance might rely on the evaluation +of the extra requirements needed to successfully use a conditional +conformance. For example: + +```swift +protocol P { + func doSomething() +} + +struct S: P { + func doSomething() { print("S") } +} + +// Array conforms to P if it's element type conforms to P +extension Array: P where Element: P { + func doSomething() { + for value in self { + value.doSomething() + } + } +} + +// Dynamically query and use conformance to P. +func doSomethingIfP(_ value: Any) { + if let p = value as? P { + p.doSomething() + } else { + print("Not a P") + } +} + +doSomethingIfP([S(), S(), S()]) // prints "S" three times +doSomethingIfP([1, 2, 3]) // prints "Not a P" +``` + +The `if-let` in `doSomethingIfP(_:)` dynamically queries whether the type stored in `value` conforms to the protocol `P`. In the case of an `Array`, that conformance is conditional, which requires another dynamic lookup to determine whether the element type conforms to `P`: in the first call to `doSomethingIfP(_:)`, the lookup finds the conformance of `S` to `P`. In the second case, there is no conformance of `Int` to `P`, so the conditional conformance cannot be used. The desire for this dynamic behavior motivates some of the design decisions in this proposal. + +## Detailed design +Most of the semantics of conditional conformances are +obvious. However, there are a number of issues (mostly involving +multiple conformances) that require more in-depth design. + +### Multiple conformances + +Swift already bans programs that attempt to make the same type conform +to the same protocol twice, e.g.: + +```swift +protocol P { } + +struct X : P { } +extension X : P { } // error: X already stated conformance to P +``` + +This existing ban on multiple conformances is extended to conditional +conformances, including attempts to conform to the same protocol in +two different ways. For example: + +```swift +struct SomeWrapper { + let wrapped: Wrapped +} + +protocol HasIdentity { + static func ===(lhs: Self, rhs: Self) -> Bool +} + +extension SomeWrapper: Equatable where Wrapped: Equatable { + static func ==(lhs: SomeWrapper, rhs: SomeWrapper) -> Bool { + return lhs.wrapped == rhs.wrapped + } +} + +// error: SomeWrapper already stated conformance to Equatable +extension SomeWrapper: Equatable where Wrapped: HasIdentity { + static func ==(lhs: SomeWrapper, rhs: SomeWrapper) -> Bool { + return lhs.wrapped === rhs.wrapped + } +} +``` + +Furthermore, for consistency, the ban extends even to multiple +conformances that are "clearly" disjoint, e.g., + +```swift +extension SomeWrapper: Equatable where Wrapped == Int { + static func ==(lhs: SomeWrapper, rhs: SomeWrapper) -> Bool { + return lhs.wrapped == rhs.wrapped + } +} + +// error: SomeWrapper already stated conformance to Equatable +extension SomeWrapper: Equatable where Wrapped == String { + static func ==(lhs: SomeWrapper, rhs: SomeWrapper) -> Bool { + return lhs.wrapped == rhs.wrapped + } +} +``` + +The section [overlapping +conformances](#overlapping-conformances) describes some of the +complexities introduced by multiple conformances, to justify their +exclusion from this proposal. A follow-on proposal could introduce +support for multiple conformances, but should likely also cover related +features such as [private +conformances](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#private-conformances) +that are orthogonal to conditional conformances. + + +### Implied conditional conformances + +Stating a non-conditional conformance to a protocol implicitly states +conformances to any of the protocols that the protocol inherits: one +can declare conformance to the `Collection` protocol, and it implies +conformance to `Sequence` as well. However, with conditional +conformances, the constraints for the conformance to the inherited +protocol may not be clear, and even when there is a clear choice, it +will often be incorrect, so the conformance to the inherited protocol +will need to be stated explicitly. For example, for the first case: + +```swift +protocol P { } +protocol Q : P { } +protocol R : P { } + +struct X { } + +extension X: Q where T: Q { } +extension X: R where T: R { } + +// error: X does not conform to protocol P; add +// +// extension X: P where <#constraints#> { ... } +// +// to state conformance to P. +``` + +Note that both of the constrained extensions could imply the +conformance to `P`. However, because the two extensions have disjoint +sets of constraints (one requires `T: Q`, the other `T: R`), it +becomes unclear which constraints should apply to the conformance to +`P`: picking one set of constraints (e.g. `T: Q`, from the conformance +of `X` to `Q`) makes the inherited conformance unusable for `X` +instances where `T: R`, which would break type safety (because we +could have `X` instances that conform to `R` but not `P`!). Moreover, +the previously-discussed ban on multiple conformances prohibits +introducing two different conformances of `X` to `P` (one where `T: Q` +and one where `T: R`). Therefore, the program above is ill-formed, and +the correct fix is for the user to introduce an explicit conformance +of `X` to `P` with the appropriate set of constraints, e.g.: + +```swift +extension X: P where T: P { } +``` + +For the second problem mentioned above, when there is an obvious set +of requirements to use in an implied conformance, it is likely to be +wrong, because of how often conditional conformances are used for +wrapper types. For instance: + +```swift +protocol R: P { } +protocol S: R { } + +struct Y { } + +extension Y: R where T: R { } +extension Y: S where T: S { } +``` + +The conformances of `Y: R` and `Y: S` both imply the conformance +`Y: P`, however the constraints `T: R` are less specialized (more +general) than the constraints `T: S`, because every `S` is also an +`R`. Therefore, it could be that `Y` will conform to `P` when `T: R`, e.g.: + +```swift +/// compiler produces the following implied inherited conformance: +extension Y: P where T: R { } +``` + +However, it is likely that the best conformance is actually the more +relaxed (that is, applicable for more choices of `T`): + +``` swift +extension Y: P where T: P { } +``` + +This is the case for almost all wrappers for the +`Sequence`/`Collection`/`BidirectionalCollection`/... hierarchy (for +instance, as discussed below, `Slice : BidirectionalCollection where +Base : BidirectionalCollection` and similarly for +`RandomAccessCollection`), and for most types conforming to several of +`Equatable`, `Comparable` and `Hashable`. + +Implicitly constructing these conformances could be okay if it were +possible to relax the overly-strong requirements when they're noticed +in future. However, it can be backwards incompatible, and so not doing +it implicitly is defaulting to the safer option. The backwards +incompatibility comes from how requirements are inferred in function +signatures: given `struct Z {}`, Swift notices that a +declaration `func foo(x: Z>)` requires that `Y : P`, since +it is used in `Z`, and thus, if the implicit inherited conformance +above existed, `A: R`. This conformance is part of the function's +signature (and mangling!) and is available to be used inside `foo`: +that function can use requirements from `R` on values of type `A`. If +the library declaring `Y` was to change to the declaration of the +conformance `Y: P`, the inferred requirement becomes `A: P`, which +changes the `foo`'s mangled name, and what can be done with values of +type `A`. This breaks both API and ABI compatibility. + +(Note: the inference above is driven by having a unique conformance, +and thus `Y: P` if *and only if* `A: P`. If overlapping conformances +were allowed, this inference would not be possible. A possible +alternative that's more directly future-proof with overlapping +conformances would be to disable this sort of inference from +conditional conformances, and instead require the user to write `func +foo`. This could also allow the conformances to be implied, +since it would no longer be such a backwards-compatibility problem.) + +On the other hand, not allowing implicit inherited conformances means +that one cannot insert a superprotocol to an existing protocol: for +instance, if the second example started as `protocol R { }` and was +changed to `protocol R: P { }`. However, we believe this is already +incompatible, for unrelated reasons. + +Finally, it is a small change to get implicit behaviour explicitly, by +adding the conformance declaration to the extension that would be +implying the conformance. For instance, if it is correct for the +second example to have `T: R` as the requirement on `Y: P`, the `Y: R` +extension only needs to be changed to include `, P`: + +``` swift +extension Y: R, P where T: R { } +``` + +This is something compilers can, and should, suggest as a fixit. + +## Standard library adoption + +Adopt conditional conformances to make various standard library types +that already have a suitable `==` conform to `Equatable`. Specifically: + +```swift +extension Optional: Equatable where Wrapped: Equatable { /*== already exists */ } +extension Array: Equatable where Element: Equatable { /*== already exists */ } +extension ArraySlice: Equatable where Element: Equatable { /*== already exists */ } +extension ContiguousArray: Equatable where Element: Equatable { /*== already exists */ } +extension Dictionary: Equatable where Value: Equatable { /*== already exists */ } +``` + +In addition, implement conditional conformances to `Hashable` for the +types above, as well as `Range` and `ClosedRange`: + +```swift +extension Optional: Hashable where Wrapped: Hashable { /*...*/ } +extension Array: Hashable where Element: Hashable { /*...*/ } +extension ArraySlice: Hashable where Element: Hashable { /*...*/ } +extension ContiguousArray: Hashable where Element: Hashable { /*...*/ } +extension Dictionary: Hashable where Value: Hashable { /*...*/ } +extension Range: Hashable where Bound: Hashable { /*...*/ } +extension ClosedRange: Hashable where Bound: Hashable { /*...*/ } +``` + +While the standard library did not previously provide existing +implementations of `hashValue` for these types, conditional `Hashable` +conformance is a natural expectation for them. + +Note that `Set` is already (unconditionally) `Equatable` and `Hashable`. + +In addition, it is intended that the standard library adopt conditional conformance +to collapse a number of "variants" of base types where other generic parameters +enable conformance to further protocols. + +For example, there is a type: + +```swift +ReversedCollection: BidirectionalCollection +``` + +that provides a low-cost lazy reversal of any bidirecitonal collection. +There is a variation on that type, + +```swift +ReversedRandomAccessCollection: RandomAccessCollection +``` + + that additionally conforms to `RandomAccessCollection` when its base does. +Users create these types via the `reversed()` extension method on +`BidirectionalCollection` and `RandomAccessCollection` respectively. + +With conditional conformance, the `ReversedRandomAccessCollection` variant can +be replaced with a conditional extension: + +```swift +extension ReversedCollection: RandomAccessCollection where Base: RandomAccessCollection { } + +@available(*, deprecated, renamed: "ReversedCollection") +public typealias ReversedRandomAccessCollection = ReversedCollection +``` + +Similar techniques can be used for variants of `Slice`, `LazySequence`, +`DefaultIndices`, `Range` and others. These refactorings are considered an +implementation detail of the existing functionality standard library and should +be applied across the board where applicable. + +## Source compatibility + +From the language perspective, conditional conformances are purely additive. They introduce no new syntax, but instead provide semantics for existing syntax---an extension that both declares a protocol conformance and has a `where` clause---whose use currently results in a type checker failure. That said, this is a feature that is expected to be widely adopted within the Swift standard library, which may indirectly affect source compatibility. + +## Effect on ABI Stability + +As noted above, there are a number of places where the standard library is expected to adopt this feature, which fall into two classes: + +1. Improve composability: the example in the [introduction](#introduction) made `Array` conform to `Equatable` when its element type does; there are many places in the Swift standard library that could benefit from this form of conditional conformance, particularly so that collections and other types that contain values (e.g., `Optional`) can compose better with generic algorithms. Most of these changes won't be ABI- or source-breaking, because they're additive. +2. Eliminating repetition: the `lazy` wrappers described in the [motivation](#motivation) section could be collapsed into a single wrapper with several conditional conformances. A similar refactoring could also be applied to the range abstractions and slice types in the standard library, making the library itself simpler and smaller. All of these changes are potentially source-breaking and ABI-breaking, because they would remove types that could be used in Swift 3 code. However, there are mitigations: generic typealiases could provide source compatibility to Swift 3 clients, and the ABI-breaking aspect is only relevant if conditional conformances and the standard library changes they imply aren't part of Swift 4. + +Aside from the standard library, conditional conformances have an impact on the Swift runtime, which will require specific support to handle dynamic casting. If that runtime support is not available once ABI stability has been declared, then introducing conditional conformances in a later language version either means the feature cannot be deployed backward or that it would provide only more limited, static behavior when used on older runtimes. Hence, there is significant motivation for doing this feature as part of Swift 4. Even if we waited to introduce conditional conformances, we would want to include a hook in the runtime to allow them to be implemented later, to avoid future backward-compatibility issues. + +## Effect on Resilience + +One of the primary goals of Swift 4 is resilience, which allows libraries to evolve without breaking binary compatibility with the applications that use them. While the specific details of the impact of conditional conformances on resilience will be captured in a more-complete proposal on resilience, possible rules are summarized here: + +* A conditional conformance cannot be removed in the new version of a library, because existing clients might depend on it. +* A conditional conformance can be added in a new version of a library, roughly following the rules described in the [library evolution document](https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst#new-conformances). The conformance itself will need to be annotated with the version in which it was introduced. +* A conditional conformance can be *generalized* in a new version of the library, i.e., it can be effectively replaced by a (possibly conditional) conformance in a new version of the library that is less specialized than the conditional conformance in the older version of the library. For example. + + ```swift + public struct X { } + + // Conformance in version 1.0 + public extension X: Sequence where T: Collection { ... } + + // Can be replaced by this less-specialized conformance in version 1.1 + public extension X: Sequence where T: Sequence { ... } + ``` + + Such conformances would likely need some kind of annotation. + +## Alternatives considered + +### Overlapping conformances + +As noted in the section on [multiple +conformances](#multiple-conformances), Swift already bans programs +that attempt to make the same type conform to the same protocol +twice. This proposal extends the ban to cases where the conformances +are conditional. Reconsider the example from that section: + +```swift +struct SomeWrapper { + let wrapped: Wrapped +} + +protocol HasIdentity { + static func ===(lhs: Self, rhs: Self) -> Bool +} + +extension SomeWrapper: Equatable where Wrapped: Equatable { + static func ==(lhs: SomeWrapper, rhs: SomeWrapper) -> Bool { + return lhs.wrapped == rhs.wrapped + } +} + +extension SomeWrapper: Equatable where Wrapped: HasIdentity { + static func ==(lhs: SomeWrapper, rhs: SomeWrapper) -> Bool { + return lhs.wrapped === rhs.wrapped + } +} +``` + +Note that, for an arbitrary type `T`, there are four potential answers to +the question of whether `SomeWrapper` conforms to `Equatable`: + +1. No, it does not conform because `T` is neither `Equatable` nor +`HasIdentity`. +2. Yes, it conforms via the first extension of `SomeWrapper` because +`T` conforms to `Equatable`. +3. Yes, it conforms via the second extension of `SomeWrapper` because +`T` conforms to `HasIdentity`. +4. Ambiguity, because `T` conforms to both `Equatable` and +`HasIdentity`. + +It is due to the possibility of #4 occurring that we refer to the two conditional conformances in the example as *overlapping*. There are designs that would allow one to address the ambiguity, for example, by writing a third conditional conformance that addresses #4: + +```swift +// Possible tie-breaker conformance +extension SomeWrapper: Equatable where Wrapped: Equatable & HasIdentity, { + static func ==(lhs: SomeWrapper, rhs: SomeWrapper) -> Bool { + return lhs.wrapped == rhs.wrapped + } +} +``` + +The design is consistent, because this third conditional conformance is more *specialized* than either of the first two conditional conformances, meaning that its requirements are a strict superset of the requirements of those two conditional conformances. However, there are a few downsides to such a system: + +1. To address all possible ambiguities, one has to write a conditional conformance for every plausible combination of overlapping requirements. To *statically* resolve all ambiguities, one must also cover nonsensical combinations where the two requirements are mutually exclusive (or invent a way to state mutual-exclusivity). +2. It is no longer possible to uniquely say what is required to make a generic type conform to a protocol, because there might be several unrelated possibilities. This makes reasoning about the whole system more complex, because it admits divergent interfaces for the same generic type based on their type arguments. At its extreme, this invites the kind of cleverness we've seen in the C++ community with template metaprogramming, which is something Swift has sought to avoid. +3. All of the disambiguation machinery required at compile time (e.g., to determine whether one conditional conformance is more specialized than another to order them) also needs to implements in the run-time, as part of the dynamic casting machinery. One must also address the possibility of ambiguities occurring at run-time. This is both a sharp increase in the complexity of the system and a potential run-time performance hazard. + +For these reasons, this proposal *bans overlapping conformances* entirely. While the resulting system is less flexible than one that allowed overlapping conformances, the gain in simplicity in this potentially-confusing area is well worth the cost. + +There are several potential solutions to the problem of overlapping conformances (e.g., admitting some form of overlapping conformances that can be resolved at runtime or introducing the notion of conformances that cannot be queried a runtime), but the feature is large enough to warrant a separate proposal that explores the solutions in greater depth. + +### Extending protocols to conform to protocols +The most common request related to conditional conformances is to allow a (constrained) protocol extension to declare conformance to a protocol. For example: + +```swift +extension Collection: Equatable where Iterator.Element: Equatable { + static func ==(lhs: Self, rhs: Self) -> Bool { + // ... + } +} +``` + +This protocol extension would make any `Collection` of `Equatable` elements `Equatable`, which is a powerful feature that could be put to good use. Introducing conditional conformances for protocol extensions would exacerbate the problem of overlapping conformances, because it would be unreasonable to say that the existence of the above protocol extension means that no type that conforms to `Collection` could declare its own conformance to `Equatable`, conditional or otherwise. + +### Overloading across constrained extensions + +Conditional conformances may exacerbate existing problems with +overloading behaving differently with concrete types vs. in a generic +context. For example, consider: + +```swift +protocol P { + func f() +} + +protocol Q: P { } +protocol R: Q { } + +struct X1 { } + +extension X1: Q where T: Q { + func f() { + // #1: basic implementation of 'f()' + } +} + +extension X1: R where T: R { + func f() { + // #2: superfast implementation of f() using some knowledge of 'R' + } +} + +// note: compiler implicitly creates conformance `X1: P` equivalent to +// extension X1: P where T: Q { } + +struct X2: R { + func f() { } +} + +(X1() as P).f() // calls #1, which was used to satisfy the requirement for 'f' +X1().f() // calls #2, which is preferred by overload resolution +``` + +When satisfying a protocol requirement, Swift chooses the most +specific member that can be used *given the constraints of the +conformance*. In this case, the conformance of `X1` to `P` has the +constraints `T: Q`, so the only `f()` that can be used under those +constraints is the `f()` from the first extension. The `f()` in the +second extension won't necessarily always be available, because `T` +may not conform to `R`. Hence, the call that treats an `X1` as a +`P` gets the first implementation of `X1.f()`. When using the concrete +type `X1`, where `X2` conforms to `R`, both `X1.f()` +implementations are visible... and the second is more specialized. + +This is not a new problem to Swift. We can write a similar example +using a constrained extension and non-conditional conformances: + +```swift +protocol P { + func f() +} + +protocol Q: P { } + +struct X3 { } + +extension X3: Q { + func f() { + // #1: basic implementation of 'f()' + } +} + +extension X3 where T: R { + func f() { + // #2: superfast implementation of f() using some knowledge of 'R' + } +} + +// note: compiler implicitly creates conformance `X3: P` equivalent to +// extension X3: P { } + +struct X2: R { + func f() { } +} + +(X3() as P).f() // calls #1, which was used to satisfy the requirement for 'f' +X3().f() // calls #2, which is preferred by overload resolution +``` + +That said, the introduction of conditional conformances might increase +the likelihood of these problems surprising developers. diff --git a/proposals/0144-allow-single-dollar-sign-as-valid-identifier.md b/proposals/0144-allow-single-dollar-sign-as-valid-identifier.md new file mode 100644 index 0000000000..3b5d3a9ea1 --- /dev/null +++ b/proposals/0144-allow-single-dollar-sign-as-valid-identifier.md @@ -0,0 +1,69 @@ +# Allow Single Dollar Sign as a Valid Identifier + +* Proposal: [SE-0144](0144-allow-single-dollar-sign-as-valid-identifier.md) +* Author: [Ankur Patel](https://github.com/ankurp) +* Review manager: [Chris Lattner](https://github.com/lattner) +* Status: **Rejected** +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0144-allow-single-dollar-sign-as-a-valid-identifier/4340) + +## Introduction + +The mainline Swift compiler emits an error message when the `$` character +(U+0024) is used as an identifier by itself, which is a source breaking +change from Swift 3.0. For example: + +```swift +let $ = 10 +// OR +let $ : (Int) -> (Int) = { $0 * $0 } +// OR +class $ {} +``` + +This proposal suggests reverting this change, enabling the use of `$` as a +valid identifier in future versions of Swift (>= 3.1). + +## Motivation + +Some projects depend on the +[Dollar library](https://github.com/ankurp/Dollar), which uses `$` +as a namespace. +The core team has decided to remove it as a valid character by merging this +[Pull Request](https://github.com/apple/swift/pull/3901) + +The reason behind the removal of `$` character as a valid identifier is: + +1. it was never intended to be an identifier, and was never documented (e.g. by [TSPL](https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/LexicalStructure.html#//apple_ref/swift/grammar/identifier)). +2. the $ sigil is more operator/punctuation-like than identifier-like. +3. the $ namespace is already used for anonymous closure arguments (when + followed by a number) and by debugger/REPL temporaries (when followed by + a letter). Misuses of these cases were already properly rejected by the + compiler, it is just the bare `$` identifier that was accepted: + +```swift +ERROR at line 1, col 5: expected numeric value following '$' +var $a = 5 +``` + + +## Proposed solution + +Allow `$` character (U+0024) to be used as a valid identifier without use of +any tick marks `` `$` ``. + +## Impact on existing code + +If this proposal is accepted, it will preserve the Swift 3.0 syntax which allows +`$` to be used as a valid identifier by itself, so there will be no impact on +existing code (the TSPL will be updated to reflect this expansion of the grammar +though). + +If this proposal is rejected, then the `$` identifier will be rejected in the +Swift 4 compiler with a migration hint that changes uses to `` `$` ``. Users +of the [Dollar](https://github.com/ankurp/Dollar) library will be affected. + +## Alternatives considered + +The primarily alternative here is to allow for the breaking change and use +`` `$` `` as the identifier in the [Dollar](https://github.com/ankurp/Dollar) +library, or for Dollar to adapt to another namespace. diff --git a/proposals/0145-package-manager-version-pinning.md b/proposals/0145-package-manager-version-pinning.md new file mode 100644 index 0000000000..6900359fa2 --- /dev/null +++ b/proposals/0145-package-manager-version-pinning.md @@ -0,0 +1,380 @@ +# Package Manager Version Pinning + +* Proposal: [SE-0145](0145-package-manager-version-pinning.md) +* Author: [Daniel Dunbar](https://github.com/ddunbar), [Ankit Aggarwal](https://github.com/aciidb0mb3r), [Graydon Hoare](https://github.com/graydon) +* Review Manager: [Anders Bertelrud](https://github.com/abertelrud) +* Status: **Implemented (Swift 3.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/swift-evolution-accepted-se-0145-package-manager-version-pinning-revised/4653) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/91725ee83fa34c81942a634dcdfa9d2441fbd853/proposals/0145-package-manager-version-pinning.md) +* Previous Discussion: [Email Thread](https://forums.swift.org/t/review-se-0145-package-manager-version-pinning/4405/15) + +## Introduction + +This is a proposal for adding package manager features to "pin" or "lock" package dependencies to particular versions. + +## Motivation + +As used in this proposal, version pinning refers to the practice of controlling +*exactly* which specific version of a dependency is selected by the dependency +resolution algorithm, *independent from* the semantic versioning +specification. Thus, it is a way of instructing the package manager to select a +particular version from among all of the versions of a package which could be +chosen while honoring the dependency constraints. + +### Terminology + +*We have chosen to use "pinning" to refer to this feature, over "lockfiles", since +the term "lock" is already overloaded between POSIX file locks and locks in +concurrent programming.* + +### Use Cases + +Our proposal is designed to satisfy several different use cases for such a behavior: + +1. Standardizing team workflows + + When collaborating on a package, it can be valuable for team members (and + continuous integration) to all know they are using the same exact version of + dependencies, to avoid "works for me" situations. + + This can be particularly important for certain kinds of open source projects + which are actively being cloned by new users, and which want to have some + measure of control around exactly which available version of a dependency is + selected. + +2. Difficult to test packages or dependencies + + Complex packages which have dependencies which may be hard to test, or hard + to analyze when they break, may choose to maintain careful control over what + versions of their upstream dependencies they recommend -- even if + conceptually they regularly update those recommendations following the true + semantic version specification of the dependency. + +3. Dependency locking w.r.t. deployment + + When stabilizing a release for deployment, or building a version of a package + for deployment, it is important to be able to lock down the exact versions of + dependencies in use, so that the resulting product can be exactly recreated + later if necessary. + +### Current Behavior + +The package manager *NEVER* updates a locally cloned package from its current +version without explicit user action (`swift package update`). We anticipate +encouraging users to update to newer versions of packages when viable, but this +has not yet been proposed or implemented. + +Whenever a package is operated on locally in a way that requires its +dependencies be present (typically a `swift build`, but it could also be `swift +package fetch` or any of several other commands), the package manager will fetch +a complete set of dependencies. However, when it does so, it attempts to get +versions of the missing dependencies compatible with the existing dependencies. + +From a certain perspective, the package manager today is acting as if the local +clones were "pinned", however, there has heretofore been no way to share that +pinning information with other team members. This proposal is aimed at +addressing that. + +## Proposed solution + +We will introduce support for an **optional** new file `Package.pins` adjacent +to the `Package.swift` manifest, called the "pins file". We will also introduce +a number of new commands (see below) for maintaining the pins file. + +This file will record the active version pin information for the package, +including data such as the package identifier, the pinned version, and explicit +information on the pinned version (e.g., the commit hash/SHA for the resolved +tag). + +The exact file format is unspecified/implementation defined, however, in +practice it will be a JSON data file. + +This file *may* be checked into SCM by the user, so that its effects apply to +all users of the package. However, it may also be maintained only locally (e.g., +placed in the `.gitignore` file). We intend to leave it to package authors to +decide which use case is best for their project. We will **recommend** that it +not be checked in by library authors, at least for released versions, since pins +are not inherited and thus this information may be confusing. We may codify this +recommendation into a warning in a future package manager workflow which +provided assistance in publishing package versions. + +In the presence of a top-level `Package.pins` file, the package manager will +respect the pinned dependencies recorded in the file whenever it needs to do +dependency resolution (e.g., on the initial checkout or when updating). + +In the absence of a top-level `Package.pins` file, the package manager will +operate based purely on the requirements specified in the package manifest, but +will then automatically record the choices it makes into a `Package.pins` file +as part of the "automatic pinning" feature. The goal of this behavior is to +encourage reproducible behavior among package authors who share the pin file +(typically by checking it in). + +We will also provide an explicit mechanism by which package authors can opt out +of the automatic pinning default for their package. + +The pins file will *not* override manifest specified version requirements and it +will be a warning if there is a conflict between the pins and the manifest +specification. + +The pins file will also not influence dependency resolution for dependent packages; +for example if application A depends on library B which in turn depends on library C, +then package resolution for application A will use the manifest of library B to learn +of the dependency on library C, but ignore any `Package.pins` file belonging to +library B when deciding which version of library C to use. + +## Detailed Design + +1. We will add a new command `pin` to `swift package` tool with following semantics: + + ``` + $ swift package pin ( [--all] | [] [] ) [--message ] + ``` + + The `package-name` refers to the name of the package as specified in its manifest. + + This command pins one or all dependencies. The command which pins a single version can optionally take a specific version to pin to, if unspecified (or with `--all`) the behavior is to pin to the current package version in use. Examples: + * `$ swift package pin --all` - pins all the dependencies. + * `$ swift package pin Foo` - pins `Foo` at current resolved version. + * `$ swift package pin Foo 1.2.3` - pins `Foo` at 1.2.3. The specified version should be valid and resolvable. + + The `--message` option is an optional argument to document the reason for pinning a dependency. This could be helpful for user to later remember why a dependency was pinned. Example: + + `$ swift package pin Foo --message "The patch updates for Foo are really unstable and need screening."` + + NOTE: When we refer to dependencies in the context of pinning, we are + referring to *all* dependencies of a package, i.e. the transitive closure of + its immediate dependencies specified in the package manifest. One of the + important ways in which pinning is useful is because it allows specifying a + behavior for the closure of the dependencies outside of them being named in + the manifest. + +2. We will add two additional commands to `pin` as part of the automatic pinning + workflow (see below): + + ``` + $ swift package pin ( [--enable-autopin] | [--disable-autopin] ) + ``` + + These will enable or disable automatic pinning for the package (this state is + recorded in the `Package.pins` file). + + These commands are verbose, but the expectation is that they are very + infrequently run, just to establish the desired behavior for a particular + project, and then the pin file (containing this state) is checked in to + source control. + +3. We will add a new command `unpin`: + + ``` + $ swift package unpin ( [--all] | [] ) + ``` + This is the counterpart to the pin command, and unpins one or all packages. + + It is an error to attempt to `unpin` when automatic pinning is enabled. + +4. We will fetch and resolve the dependencies when running the pin commands, in case we don't have the complete dependency graph yet. + +5. We will extend the workflow for update to honor version pinning, that is, it will only update packages which are unpinned, and it will only update to versions which can satisfy the existing pins. The update command will, however, also take an optional argument `--repin`: + + ``` + $ swift package update [--repin] + ``` + + * The update command will warn if there are no unpinned packages which can be updated. + + * Otherwise, the behavior is to update all unpinned packages to the latest possible versions which can be resolved while respecting the existing pins. + + * The `[--repin]` argument can be used to lift the version pinning restrictions. In this case, the behavior is that all packages are updated, and packages which were previously pinned are then repinned to the latest resolved versions. + + When automatic pinning is enabled, `package update` would by default have absolutely no effect without `--repin`. Thus, we will make `package update` act as if `--repin` was specified whenever automatic pinning is enabled. This is a special case, but we believe it is most likely to match what the user expects, and avoids have a command syntax which has no useful behavior in the automatic pinning mode. + +6. The update and checkout will both emit logs, notifying the user that pinning is in effect. + +7. The `swift package show-dependencies` subcommand will be updated to indicate if a dependency is pinned. + + +### Automatic Pinning + +The package manager will have automatic pinning enabled by default (this is +equivalent to `swift package pin --enable-autopin`), although package project +owners can choose to disable this if they wish to have more fine grained control +over their pinning behavior. + +When automatic pinning is enabled, the package manager will automatic record all +package dependencies in the `Package.pins` file. If package authors do not check +this file into their source control, the behavior will typically be no different +than the existing package manager behavior (one exception is the `package +update` behavior described above). + +If a package author does check the file into source control, the effect will be +that anyone developing directly on this package will end up sharing the same +dependency versions (and modifications will be committed as part of the SCM +history). + +The automatic pinning behavior is an extension of the behaviors above, and works +as follows: + + * When enabled, the package manager will write all dependency versions into the + pin file after any operation which changes the set of active working + dependencies (for example, if a new dependency is added). + + * A package author can still change the individual pinned versions using the + `package pin` commands, these will simply update the pinned state. + + * Some commands do not make sense when automatic pinning is enabled; for + example, it is not possible to `unpin` and attempts to do so will produce an + error. + +Since package pin information is **not** inherited across dependencies, our +recommendation is that packages which are primarily intended to be consumed by +other developers either *disable* automatic pinning or put the `Package.pins` +file into `.gitignore`, so that users are not confused why they get different +versions of dependencies that are those being used by the library authors while +they develop. + + +## Future Directions + +We have intentionally kept the pin file format an implementation detail in order +to allow for future expansion. For example, we would like to consider embedding +additional information on a known tag (like its SHA, when using Git) in the pins +file as a security feature, to prevent man-in-the-middle attacks on parts of the +package graph. + +## Impact on existing code + +There will be change in the behaviors of `swift build` and `swift package update` in presence of the pins file, as noted in the proposal, however the existing package will continue to build without any modifications. + +## Alternative considered + +### Minimal pin feature set + +A prior version of this proposal did not pin by default. Since this proposal +includes this behavior, we could in theory eliminate the fine grained pinning +feature set we expose, like `package pin ` and `package unpin`. + +However, we believe it is important for package authors to retain a large amount +of control over how their package is developed, and we wish the community to +aspire to following semantic versioning strictly. For that reason, we wanted to +support mechanisms so that package authors wishing to follow this model could +still pin individual dependencies. + + +### Pin by default + +_This discussion is historical, from a prior version of a proposal which did not +include the automatic pinning behavior; which we altered the proposal for. We +have left it in the proposal for historical context._ + +Much discussion has revolved around a single policy-default question: whether +SwiftPM should generate a pins file as a matter of course any time it +builds. This is how some other package managers work, and it is viewed as a +conservative stance with respect to making repeatable builds more likely between +developers. Developers will see the pins file and will be likely to check it in +to their SCM system as a matter of convention. As a side effect, other +developers cloning and trying out the package will then end up using the same +dependencies the developer last published. + +While pinning does reduce the risk of packages failing to build, this practice +discourages the community from relying on semver compatibility to completely +specify what packages are compatible. That in turn makes it more likely for +packages to fail to correctly follow the semver specification when publishing +versions. Unfortunately, when packages don't correctly follow semver then it +requires downstream clients to overspecify their dependency constraints since +they cannot rely on the package manager automatically picking the appropriate +version. + +Overconstraint is much more of a risk in Swift than in other languages +using this style of package management. Specifically: Swift does not support +linking multiple versions of a dependency into the same artifact at the same +time. Therefore the risk of producing a "dependency hell" situation, in which +two packages individually build but _cannot be combined_ due to over-constrained +transitive dependencies, is significantly higher than in other languages. +Changing the compiler support in this area is not something which is currently +planned as a feature, so our expectation is that we will have this limitation +for a significant time. + +For example, if package `Foo` depends on library `LibX` version 1.2, +and package `Bar` depends on `LibX` 1.3, and these are _specific_ +version constraints that do not allow version-range variation, +then SwiftPM will _not_ allow building a product that depends on +both `Foo` and `Bar`: their requirements for `LibX` are incompatible. +Where other package managers will simultaneously link two versions +of `LibX` -- and hope that the differing simultaneous uses of +`LibX` do not cause other compile-time or run-time errors -- +SwiftPM will simply fail to resolve the dependencies. + +We therefore wish to encourage library authors to keep their +packages building and testing with as recent and as wide a range +of versions of their dependencies as possible, and guard more +vigorously than other systems against accidental overconstraint. +One way to encourage this behavior is to avoid emitting pins files +by default. + +We also believe that if packages default to exposing their pin files as part of +their public package, there is a substantial risk that when developers encounter +build failures they will default to copying parts of those pinned versions into +their manifest, rather than working to resolve the semver specification issues +in the dependencies. If this behavior becomes common place, it may even become +standard practice to do this proactively, simply to avoid the potential of +breakage. + +This practice is likely because it resolves the immediate issue (a build +failure) without need for external involvement, but if it becomes widespread +then it has a side-effect of causing significant overconstraint of packages +(since a published package may end up specifying only a single version it is +compatible with). + +Finally, we are also compelled by several pragmatic implications of an approach +which optimizes for reliance on the semver specifications: + +1. We do not yet have a robust dependency resolution algorithm we can rely + on. The complexity of the algorithm is in some ways relative to the degree of + conflicts we expect to be present in the package graph (for example, this may + mean we need to investigate significantly more work in optimizing its + performance, or in managing its diagnostics). + +2. The Swift package manager and its ecosystem is evolving quickly, and we + expect it will continue to do so for some time. As a consequence, we + anticipate that packages will frequently be updated simply to take advantage + of new features. Optimizing for an ecosystem where everyone can reliably live + on the latest semver-compatible release of a package should help make that a + smoother process. + +If, in practice, the resulting ecosystem either contains too many packages that +fail to build, or if a majority of users emit pins files manually regardless of +default, this policy choice can be revisited. + +We considered approaches to "pin by default" that used separate mechanisms when +publishing a package to help address the potential for overconstraint, but were +unable to find a solution we felt was workable. + + +### Naming Choice + +This feature is called "locking" and the files are "lockfiles" in many other +package managers, and there has been considerable discussion around whether the +Swift package manager should follow that precedent. + +In Swift, we have tried to choose the "right" answer for names in order to make +the resulting language consistent and beautiful. + +We have found significant consensus that without considering the prededent, the +"lock" terminology is conceptually the *wrong* word for the operation being +performed here. We view pinning as a workflow-focused feature, versus the +specification in the manifest (which is the "requirement"). The meaning of pin +connotes this transient relationship between the pin action and the underlying +dependency. + +In contrast, not only does lock have the wrong connotation, but it also is a +heavily overloaded word which can lead to confusion. For example, if the package +manager used POSIX file locking to prevent concurrent manipulation of packages +(a feature we intend to implement), and we also referred to the pinning files as +"lock files", then any diagnostics using the term "lock file" would be confusing +to a newcomer to the ecosystem familiar with the pinning mechanism but +unfamiliar with the concept of POSIX file locking. + +We believe that there are many more potential future users of the Swift package +manager than there are current users familiar with the lock, and chose the "pin" +terminology to reflect what we thought was ultimately the best word for the +operation, in order to contribute to the best long term experience. diff --git a/proposals/0146-package-manager-product-definitions.md b/proposals/0146-package-manager-product-definitions.md new file mode 100644 index 0000000000..ce87eeb8fe --- /dev/null +++ b/proposals/0146-package-manager-product-definitions.md @@ -0,0 +1,260 @@ +# Package Manager Product Definitions + +* Proposal: [SE-0146](0146-package-manager-product-definitions.md) +* Author: [Anders Bertelrud](https://github.com/abertelrud) +* Review manager: Daniel Dunbar +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/review-se-0146-package-manager-product-definitions/4540/2) +* Bug: [SR-3606](https://bugs.swift.org/browse/SR-3606) + +## Introduction + +This proposal introduces the concept of *products* to the Swift Package Manager, and proposes enhancements to the `Package.swift` syntax to let packages define products that can be referenced by other packages. + +## Motivation + +Currently, the Swift Package Manager has only a limited notion of the products that are built for a package. It does have a set of rules by which it infers implicit products based on the contents of the targets in the package, and it also has a small amount of undocumented, unsupported package manifest syntax for explicitly declaring products, but it provides no supported way for package authors to declare what their packages produce. + +Also, while the Swift Package Manager currently supports dependencies between packages, it has no support for declaring a dependency on anything more fine-grained than a package. + +Such fine-grained dependencies are often desired, and this desire has in the past been expressed as a request for the ability to declare a dependency on an arbitrary target in a package. That, in turn, leads to requests to provide access control for targets, since a package author may want control over which targets can be independently accessed from outside the package. + +Even if visibility control for targets were to be provided (indeed, an early draft of the package manifest syntax had the notion of "publishing" a target to external clients), there would still be no way for the package author to declare anything about the kind of product that should be built. + +One consequence of the lack of ability to define dependencies at subpackage granularity is that package authors have to break up packages into several smaller packages in order to achieve the layering they want. Such package decomposition may be appropriate in some cases, but should be done based on what makes sense from a conceptual standpoint and not because the Swift Package Manager doesn't allow the author to express their intent. + +For example, consider the package for a component that has both a library API and command line tools. Such a package is also likely to be partitioned into a set of core libraries on which both the public library and the command line tools depend, but which should remain a private implementation detail as far as clients are concerned. + +Such a package would currently need to be split up into three separate packages in order to provide the appropriate dependency granularity: one for the public library, another for the command line tools, and a third, private package to provide the shared implementation used by the other two packages. In the case of a single conceptual component that should have a single version number, this fracturing into multiple packages is directly contrary to the developer's preferred manner of packaging. + +What is needed is a way to allow package authors to define conceptually distinct products of a package, and to allow client packages to declare dependencies on individual products in a package. + +Furthermore, explicit product definitions would allow the package author to control the types of artifacts produced from the targets in the package. This would include such things as whether a library is built as a static archive or a dynamic library. We expect that additional product types will be added over time to let package authors build more kinds of artifacts. + +## Proposed solution + +We will introduce a documented and supported concept of a package product, along with package manifest improvements to let package authors define products and to let package clients define dependencies on such products. In defining a product, a package author will be able specify the type of product as well as its characteristics. + +### Product definitions + +A package will be able to define an arbitrary number of products that are visible to all direct clients of the package. A product definition consists of a product type, a name (which must be unique among the products in the package), and the root targets that comprise the implementation of the product. There may also be additional properties depending on the type of product (for example, a library may be static or dynamic). + +Any target may be included in multiple products, though not all kinds of targets are usable in every kind of product; for example, a test target is not able to be included in a library product. + +The products represent the publicly vended package outputs on which client packages can depend. Other artifacts might also be created as part of building the package, but what the products specifically define are the conceptual "outputs" of the package, i.e. those that make sense to think of as something a client package can depend on and use. Examples of artifacts that are not necessarily products include built unit tests and helper tools that are used only during the build of the package. + +An example of a package that defines two library products and one executable product: + +```swift +let package = Package( + name: "MyServer", + targets: [ + Target(name: "Utils"), + Target(name: "HTTP", dependencies: ["Utils"]), + Target(name: "ClientAPI", dependencies: ["HTTP", "Utils"]), + Target(name: "ServerAPI", dependencies: ["HTTP"]), + Target(name: "ServerDaemon", dependencies: ["ServerAPI"]), + ], + products: [ + .Library(name: "ClientLib", type: .static, targets: ["ClientAPI"]), + .Library(name: "ServerLib", type: .dynamic, targets: ["ServerAPI"]), + .Executable(name: "myserver", targets: ["ServerDaemon"]), + ] +) +``` + +The initial types of products that can be defined are executables and libraries. Libraries can be declared as either static or dynamic, but when possible, the specific type of library should be left unspecified, letting the build system chooses a suitable default for the platform. + +Note that tests are not considered to be products, and do not need to be explicitly defined. + +A product definition lists the root targets to include in the product; for product types that vend interfaces (e.g. libraries), the root targets are those whose modules will be available to clients. Any dependencies of those targets will also be included in the product, but won't be made visible to clients. The Swift compiler does not currently provide this granularity of module visibility control, but the set of root targets still constitutes a declaration of intent that can be used by IDEs and other tools. We also hope that the compiler will one day support this level of visibility control. See [SR-3205](https://bugs.swift.org/browse/SR-3205) for more details. + +For example, in the package definition shown above, the library product `ClientLib` would only vend the interface of the `ClientAPI` module to clients. Since `ClientAPI` depends on `HTTP` and `Utilities`, those two targets would also be compiled and linked into `ClientLib`, but their interfaces should not be visible to clients of `ClientLib`. + +### Implicit products + +SwiftPM 3 applies a set of rules to infer products based on the targets in the package. For backward compatibility, SwiftPM 4 will apply the same rules to packages that use the SwiftPM 3 PackageDescription API. The `package describe` command will show implied product definitions. + +When switching to the SwiftPM 4 PackageDescription API, the package author takes over responsibility for defining the products. There will be tool support (probably in the form of a fix-it on a "package defines no products" warning) to make it easy for the author to add such definitions. Also, the `package init` command will be extended to automatically add the appropriate product definitions to the manifest when it creates the package. + +There was significant discussion about whether the implicit product rules should continue to be supported alongside the explicit product declarations. The tradeoffs are described in the "Alternatives considered" section. + +### Product dependencies + +A target will be able to declare its use of the products that are defined in any of the external package dependencies. This is in addition to the existing ability to declare dependencies on other targets in the same package. + +To support this, the `dependencies` parameter of the `Target()` initializer will be extended to also allow product references. + +To see how this works, remember that each string listed for the `dependencies` parameter is just shorthand for `.target(name: "...")`, i.e. a dependency on a target within the same package. + +For example, the target definition: + +```swift +Target(name: "Foo", dependencies: ["Bar", "Baz"]) +``` + +is shorthand for: + +```swift +Target(name: "Foo", dependencies: [.target(name: "Bar"), .target(name: "Baz")]) +``` + +This will be extended to support product dependencies in addition to target dependencies: + +```swift +let package = Package( + name: "MyClientLib", + dependencies: [ + .package(url: "https://github.com/Alamofire/Alamofire", majorVersion: 3), + ], + targets: [ + Target(name: "MyUtils"), + Target(name: "MyClientLib", dependencies: [ + .target(name: "MyUtils"), + .product(name: "Alamofire", package: "Alamofire") + ]) + ] +) +``` + +The package name is the canonical name of the package, as defined in the manifest of the package that defines the product. The product name is the name specified in the product definition of that same manifest. + +The package name is optional, since the product name is almost always unambiguous (and is frequently the same as the package name). The package name must be specified if there is more than one product with the same name in the package graph (this does not currently work from a technical perspective, since Swift module names must currently be unique within the package graph). + +In order to continue supporting the convenience of being able to use plain strings as shorthand, and in light of the fact that most of the time the names of packages and products are unique enough to avoid confusion, we will extend the short-hand notation so that a string can refer to either a target or a product. + +The Package Manager will first try to resolve the name to a target in the same package; if there isn't one, it will instead to resolve it to a product in one of the packages specified in the `dependencies` parameter of the `Package()` initializer. + +For both the shorthand form and the complete form of product references, the only products that will be visible to the package are those in packages that are declared as direct dependencies -- the products of indirect dependencies are not visible to the package. + +## Detailed design + +### Product definitions + +We will add a `products` parameter to the `Package()` initializer: + +```swift +Package( + name: String, + pkgConfig: String? = nil, + providers: [SystemPackageProvider]? = nil, + targets: [Target] = [], + dependencies: [Package.Dependency] = [], + products: [Product] = [], + exclude: [String] = [] +) +``` + +The definition of the `Product` type will be: + +```swift +public enum Product { + public class Executable { + public init(name: String, targets: [String]) + } + + public class Library { + public enum LibType { + case .static + case .dynamic + } + public init(name: String, type: LibType? = nil, targets: [String]) + } +} +``` + +The namespacing allows the names of the product types to be kept short. + +As with targets, there is no semantic significance to the order of the products in the array. + +### Implicit products + +If the `products` array is omitted, and if the package uses version 3 of the PackageDescription API, the Swift Package Manager will infer a set of implicit products based on the targets in the package. + +The rules for implicit products are the same as in SwiftPM 3: + +1. An executable product is implicitly defined for any target that produces an executable module (as defined here: https://github.com/apple/swift-package-manager/blob/master/Documentation/Reference.md). + +2. If there are any library targets, a single library product is implicitly defined for all the library targets. The name of this product is based on the name of the package. + +### Product dependencies + +We will add a new enum case to the `TargetDependency` enum to represent dependencies on products, and another enum case to represent an unbound by-name dependency. The string-literal conversion will create a by-name dependency instead of a target dependency, and there will be logic to bind the by-name dependencies to either target or product dependencies once the package graph has been resolved: + +```swift +public final class Target { + + /// Represents a target's dependency on another entity. + public enum TargetDependency { + /// A dependency on a target in the same project. + case Target(name: String) + + /// A dependency on a product from a package dependency. The package name match the name of one of the packages named in a `.package()` directive. + case Product(name: String, package: String?) + + /// A by-name dependency that resolves to either a target or a product, as above, after the package graph has been loaded. + case ByName(name: String) + } + + /// The name of the target. + public let name: String + + /// Dependencies on other entities inside or outside the package. + public var dependencies: [TargetDependency] + + /// Construct a target. + public init(name: String, dependencies: [TargetDependency] = []) +} +``` + +For compatibility reasons, packages using the Swift 3 version of the PackageDescription API will have implicit dependencies on the directly specified packages. Packages that have adopted the Swift 4 version need to declare their product dependencies explicitly. + +## Impact on existing code + +There will be no impact on existing packages that follow the documented Swift Package Manager 3.0 format of the package manifest. Until the package is upgraded to the 4.0 format, the Swift Package Manager will continue to infer products based the existing rules. + +## Alternatives considered + +Many alternatives were considered during the development of this proposal, and in many cases the choice was between two distinct approaches, each with clear advantages and disadvantages. This section summarizes the major alternate approaches. + +### Not adding product definitions + +Instead of product definitions, fine-grained dependencies could be introduced by allowing targets to be marked as public or private to the package. + +_Advantage_ + +It would avoid the need to introduce new concepts. + +_Disadvantage_ + +It would not provide any way for a package author to control the type and characteristics of the various artifacts. Relying on the implied products that result from the inference rules is not likely to be scalable in the long term as we introduce new kinds of product types. + +### Inferring implicit products + +An obvious alternative to the proposal would be to keep the inference rules even in v4 of the PackageDescription API. + +_Advantage_ + +It can be a lot more convenient to not have to declare products, and to be able to add new products just by adding or renaming files and directories. This is particularly true for small, simple packages. + +_Disadvantages_ + +The very fact that it is so easy to change the set of products without modifying the package manifest can lead to unexpected behavior due to seemingly unrelated changes (e.g. creating a file called `main.swift` in a module changes that module from a library to an executable). + +Also, as packages become more complex and new conceptual parts on which clients can depend are introduced, the interaction of implicit rules with the explicit product definitions can become very complicated. + +We plan to provide some of the convenience through tooling. For example, an IDE (or `swift package` itself on the command line) can offer to add product definitions when it notices certain types of changes to the structure of the package. We believe that having defined product types lets the tools present a lot better diagnostics and other forms of help to users, since it provides a clear statement of intent on the part of the package author. + +### Distinguishing between target dependencies and product dependencies + +The proposal extends the `Target()` initializer's `dependencies` parameter to allow products as well as targets; another approach would have been to add a new parameter, e.g. `externalDependencies` or `productDependencies`. + +_Advantage_ + +Targets and products are different types of entities, and it is possible for a target and a product to have the same name. Having the `dependencies` parameter as a heterogeneous list can lead to ambiguity. + +_Disadvantages_ + +Conceptually the *dependency* itself is the same concept, even if type of entity being depended on is technically different. In both cases (target and product), it is up to the build system to determine the exact type of artifacts that should be produced. In most cases, there is no actual semantic ambiguity, since the name of a target, product, and package are often the same uniquely identifiable "brand" name of the component. + +Also, separating out each type of dependency into individual homogeneous lists doesn't scale. If a third type of dependency needs to be introduced, a third parameter would also need to be introduced. Keeping the list heterogeneous avoids this. diff --git a/proposals/0147-move-unsafe-initialize-from.md b/proposals/0147-move-unsafe-initialize-from.md new file mode 100644 index 0000000000..7964d5d408 --- /dev/null +++ b/proposals/0147-move-unsafe-initialize-from.md @@ -0,0 +1,165 @@ +# Move UnsafeMutablePointer.initialize(from:) to UnsafeMutableBufferPointer + +* Proposal: [SE-0147](0147-move-unsafe-initialize-from.md) +* Author: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 3.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0147-move-unsafemutablepointer-initialize-from-to-unsafemutablebufferpointer/4823) +* Implementation: [apple/swift#6601](https://github.com/apple/swift/pull/6601) + +## Introduction + +The version of `UnsafeMutablePointer.initialize(from:)` that takes a `Collection` should be deprecated in favor of a new method on `UnsafeMutableBufferPointer` that takes a `Sequence`, with a goal of improving memory safety and enabling faster initialization of memory from sequences. Similarly, `UnsafeMutableRawPointer.initializeMemory(as:from:)` should be deprecated in favor of a new `UnsafeMutableRawBufferPointer.initialize(as:from:)`. + +## Motivation + +`UnsafeMutablePointer.initialize(from:)` underpins implementations of collections, such as `Array`, which are backed by a buffer of contiguous memory. When operations like `Array.append` are implemented, they first ensure their backing store can accommodate the number of elements in a source collection, then pass that collection into the `initialize` method of their backing store. + +Unfortunately there is a major flaw in this design: a collection's `count` might not accurately reflect the number of elements returned by its iterator. For example, some collections can be misused to return different results on each pass. Or a collection could just be implemented incorrectly. + +If the collection's `count` ends up being lower than the actual number of elements yielded by its iterator, the caller may not allocate enough memory for them. Since `UnsafeMutablePointer.initialize(from:)` does not receive a limiting capacity, this method would then scribble past the end of the buffer, resulting in undefined behavior. + +Normally when using `Unsafe...` constructs in Swift the burden of ensuring safety falls on the caller. When using this method with something known to have correct behavior, like an `Array`, you can do that. But when used in a generic context like `Array.append(contentsOf:)`, where the caller of `initialize` does not know exactly what kind of collection they are passing in, it is impossible to use this method safely. You can see the impact of this by running the following code. which exhibits memory-unsafe behavior despite only using “safe” constructs from the standard library, something that shouldn’t be possible: + +```swift +var i = 0 +let c = repeatElement(42, count: 10_000).lazy.filter { _ in + // capture i and use it to exhibit inconsistent + // behavior across iteration of c + i += 1; return i > 10_000 +} +var a: [Int] = [] +// a will allocate insufficient memory before +// calling self._buffer.initialize(from: c) +a.append(contentsOf: c) // memory access violation +``` + +While a collection returning an inconsistent count is a programming error (in this case, use of the lazy filter in combination with an logically impure function, breaking value semantics), and it would be reasonable for the standard library to trap under these circumstances, undefined behavior like this is not OK. + +In addition, the requirement to pre-allocate enough memory to accommodate `from.count` elements rules out using this method to initialize memory from a sequence, since sequences don't have a `count` property (they have an `underestimatedCount` but this isn't enough since underestimated counts are exactly the problem described above). The proposed solution would allow for this, enabling some internal performance optimizations for generic code. + +## Proposed solution + +The existing `initialize` method should be altered to receive a capacity, to avoid running beyond what the caller has allocated. Since `UnsafeMutableBufferPointer` already exists to encapsulate "pointer plus a count", the method should be moved to that type and the old method deprecated. + +This new method should take a `Sequence` as its `from` argument, and handle possible element overflow, returning an `Iterator` of any elements not written due to a lack of space. It should also return an index into the buffer to indicate where the elements were written up to in cases of underflow. + +Once this has been done, the version of `Array.append(contentsOf:)` that takes a collection can be eliminated, since the performance benefits of the collection version could be incorporated into the implementation of the one that takes a sequence. + +The intention of this change is to add memory safety, not to allow the flexibility of collections giving inconsistent counts. Therefore the precondition should remain that the caller should ensure enough memory is allocated to accommodate `source.underestedCount` elements. The only difference is if they don’t, the behaviour should be well-defined (ideally by trapping, if this can be done efficiently). + +Therefore: + +- Under-allocating the destination buffer relative to `underestimatedCount` may trap at runtime. _May_ rather than _will_ because this is an `O(n)` operation on some collections, so may only be enforced in debug builds. +- Over-allocating the destination buffer relative to `underestimatedCount` is valid and simply results in sequence underflow with potentially uninitialized buffer memory (a likely case with arrays that reserve more than they need). +- The source sequence's actual count may exceed both `underestimatedCount` and the destination buffer size, resulting in sequence overflow. This is also valid and handled by returning an iterator to the uncopied elements as an overflow sequence. + +A matching change should also be made to `UnsafeRawBufferPointer.initializeMemory(from:)`. The one difference is that for convenience this should return an `UnsafeMutableBufferPointer` of the (typed) initialized elements instead of an index into the raw buffer. + +## Detailed design + +The following API changes would be made: + +```swift +extension UnsafeMutablePointer { + @available(*, deprecated, message: "it will be removed in Swift 4.0. Please use 'UnsafeMutableBufferPointer.initialize(from:)' instead") + public func initialize(from source: C) + where C.Iterator.Element == Pointee +} + +extension UnsafeMutableBufferPointer { + /// Initializes memory in the buffer with the elements of `source`. + /// Returns an iterator to any elements of `source` that didn't fit in the + /// buffer, and an index to the point in the buffer one past the last element + /// written (so `startIndex` if no elements written, `endIndex` if the buffer + /// was completely filled). + /// + /// - Precondition: The memory in `self` is uninitialized. The buffer must contain + /// sufficient uninitialized memory to accommodate `source.underestimatedCount`. + /// + /// - Postcondition: The returned iterator + /// - Postcondition: The `Pointee`s at `self[startIndex..( + from source: S + ) -> (unwritten: S.Iterator, initializedUpTo: Index) + where S.Iterator.Element == Iterator.Element +} + +extension UnsafeMutableRawPointer { + @available(*, deprecated, message: "it will be removed in Swift 4.0. Please use 'UnsafeMutableRawBufferPointer.initialize(from:)' instead") + @discardableResult + public func initializeMemory( + as: C.Iterator.Element.Type, from source: C + ) -> UnsafeMutablePointer +} + +extension UnsafeMutableRawBufferPointer { + /// Initializes memory in the buffer with the elements of + /// `source` and binds the initialized memory to type `T`. + /// + /// Returns an iterator to any elements of `source` that didn't fit in the + /// buffer, and an index into the buffer one past the last byte written. + /// + /// - Precondition: The memory in `self` is uninitialized or initialized to a + /// trivial type. + /// + /// - Precondition: The buffer must contain sufficient memory to + /// accommodate at least `source.underestimatedCount` elements. + /// + /// - Postcondition: The memory at `self[startIndex...stride] is bound to type `S.Iterator`. + /// + /// - Postcondition: The memory at `self[startIndex...stride] are initialized.. + @discardableResult + public func initializeMemory( + as: S.Iterator.Element.Type, from source: S + ) -> (unwritten: S.Iterator, initialized: UnsafeMutableBufferPointer) +} + +``` + +The `+=` operators and `append(contentsOf newElements: C)` methods on `Array`, `ArraySlice` and `ContiguousArray` will be removed as no-longer needed, since the implementation that takes a sequence can be made to be as efficient. The `+=` can be replaced by a generic one that calls `RangeReplaceableCollection.append(contenstsOf:)`: + +(note, because it forwards on to a protocol requirement, it itself does not need to be a static operator protocol requirement) + +```swift +/// Appends the elements of a sequence to a range-replaceable collection. +/// +/// Use this operator to append the elements of a sequence to the end of +/// range-replaceable collection with same `Element` type. This example +/// appends the elements of a `Range` instance to an array of +/// integers. +/// +/// var numbers = [1, 2, 3, 4, 5] +/// numbers += 10...15 +/// print(numbers) +/// // Prints "[1, 2, 3, 4, 5, 10, 11, 12, 13, 14, 15]" +/// +/// - Parameters: +/// - lhs: The array to append to. +/// - rhs: A collection or finite sequence. +/// +/// - Complexity: O(*n*), where *n* is the length of the resulting array. +public func += < + R : RangeReplaceableCollection, S : Sequence +>(lhs: inout R, rhs: S) + where R.Iterator.Element == S.Iterator.Element +``` + +## Source compatibility + +The addition of the new method does not affect source compatibility. The deprecation of the old method does, but since this is a fundamentally unsound operation that cannot be fixed except via a source-breaking change, it should be aggressively deprecated and then removed. + +The knock-on ability to remove the version of `Array.append(contentsOf:)` that takes a collection does not affect source compatibility since the version for sequences will be called for collections instead. + +## Effect on ABI stability + +This change must be made prior to declaring ABI stability, since it is currently called from the `Array.append` method, which is inlineable. + +## Alternatives considered + +Overflow (elements remain on the returned iterator) and underflow (`initializedUpTo != endIndex`) are almost but not quite mutually exclusive – if the buffer is exactly used, the caller must call `.next()` to check for any unwritten elements, which means the returned value must be declared `var`, and the check can't be chained. This is a little ugly, but is the unavoidable consequence of how iterators work: since iterating is consuming, the `initialize` method cannot easily test for this and indicate it back to the caller in some other way (such as returning an optional iterator). + diff --git a/proposals/0148-generic-subscripts.md b/proposals/0148-generic-subscripts.md new file mode 100644 index 0000000000..1eb2e0dae8 --- /dev/null +++ b/proposals/0148-generic-subscripts.md @@ -0,0 +1,79 @@ +# Generic Subscripts + +* Proposal: [SE-0148](0148-generic-subscripts.md) +* Author: [Chris Eidhof](https://github.com/chriseidhof) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0148-generic-subscripts/5017) +* Bug: [SR-115](https://bugs.swift.org/browse/SR-115) + +## Introduction + +Make it possible to have generic subscripts. Example: + +```swift +extension Collection { + subscript(indices: Indices) -> [Iterator.Element] where Indices.Iterator.Element == Index { + // ... + } +} +``` + +Or a generic return type: + +```swift +extension JSON { + subscript(key: String) -> T? { + // ... + } +} +``` + +Swift-evolution thread: [Generic Subscripts](https://forums.swift.org/t/generic-subscripts/4858). + +## Motivation + +Currently, subscripts can't be generic. This is limiting in a number of ways: + +- Some subscripts are very specific and could be made more generic. +- Some generic methods would feel more natural as a subscript, but currently can't be. This also makes it impossible to use them as lvalues. + +This feature is also mentioned in the generics manifesto under [generic subscripts](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#generic-subscripts). The [Rationalizing Sequence end-operation names](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0132-sequence-end-ops.md) proposal could greatly benefit from this, as well as the ideas in the [String Manifesto](https://github.com/apple/swift/blob/master/docs/StringManifesto.md). + +## Proposed solution + +Add generics to subscripts. There are two pieces to this: where to add the generic parameter list, and where to add the `where`-clause. The most straightforward way would be to use the same syntax as methods: + +```swift +extension Dictionary { + subscript(indices: Indices) -> [Iterator.Element] where Indices.Iterator.Element == Index { + // ... + } +} +``` + +*Update Jan 20*: during the review it came up that while we're at it, we should add default arguments to subscripts. For example, the following (contrived) example: + +```swift +subscript(index: A? = nil) -> Element { + // ... +} +``` + +Adding default arguments would unify the compiler's handling of subscripts and functions. + +## Source compatibility + +This is a purely additive change. We don't propose changing the Standard Library to use this new feature, that should be part of a separate proposal. (Likewise, we should consider making subscripts `throws` in a [separate proposal](https://github.com/beccadax/swift-evolution/blob/throwing-properties/proposals/0000-throwing-properties.md)). + +## Effect on ABI stability + +It won’t change the ABI of existing subscript calls. + +## Effect on API resilience + +It won’t change the ABI of existing subscript calls, but if the standard library introduces new generic subscripts that replace older non-generic subscripts, it will impact ABI. + +## Alternatives considered + +None. diff --git a/proposals/0149-package-manager-top-of-tree.md b/proposals/0149-package-manager-top-of-tree.md new file mode 100644 index 0000000000..8b30f65cde --- /dev/null +++ b/proposals/0149-package-manager-top-of-tree.md @@ -0,0 +1,42 @@ +# Package Manager Support for Top of Tree development + +* Proposal: [SE-0149](0149-package-manager-top-of-tree.md) +* Author: [Boris Bügling](https://github.com/neonichu) +* Review Manager: [Daniel Dunbar](https://github.com/ddunbar) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0149-package-manager-support-for-top-of-tree-development/5072) +* Bug: [SR-3709](https://bugs.swift.org/browse/SR-3709) + +## Introduction + +This proposal adds enhancements to `swift package edit` to support development of packages without strict versioning ("top of tree" development). + +## Motivation + +The package manager currently supports package dependencies which are strictly versioned according to semantic versioning. This works well for users of packages and it is possible to edit a package in place using `swift package edit` already, but we want to allow developers to manually check out repositories on their machines as overrides. This is useful when developing multiple packages in tandem or when working on packages alongside an application. + +When a developer owns multiple packages that depend on each other, it can be necessary to work on a feature across more than one of them at the same time, without having to tag versions in between. The repositories for each package would usually already be checked out and managed manually by the developer, allowing them to switch branches or perform other SCM operations at will. + +A similar situation will arise when working on a feature that requires code changes to both an application and a dependent package. Developers want to iterate on the package by directly in the context of the application without having to release spurious versions of the package. Allowing developers to provide their own checkouts to the package manager as overrides will make this workflow much easier than it currently is. + +## Proposed solution + +As a solution to this problem, we propose to extend `swift package edit` to take an optional path argument to an existing checkout so that users can manually manage source control operations. + +## Detailed Design + +We will extend the `edit` subcommand with a new optional argument `--path`: + +```bash +$ swift package edit bar --path ../bar +``` + +This allows users to manage their own checkout of the `bar` repository and will make the package manager use that instead of checking out a tagged version as it normally would. Concretely, this will make `./Packages/bar` a symbolic link to the given path in the local filesystem, store this mapping inside the workspace and will ensure that the package manager will no longer be responsible for managing checkouts for the `bar` package, instead the user is responsible for managing the source control operations on their own. This is consistent with the current behavior of `swift edit`. Using `swift package unedit` will also work unchanged, but the checkout itself will not be deleted, only the symlink. If there is no existing checkout at the given filesystem location, the package manager will do an initial clone on the user's behalf. + +## Impact on existing code + +There will no impact on existing code. + +## Alternative considered + +We could have used the symlink in the `Packages` directory as primary data, but decided against it in order to be able to provide better diagnostics and distinguish use of `swift package edit` from the user manually creating symbolic links. This makes the symlink optional, but we decided to still create it in order to keep the structure of the `Packages` directory consistent independently of the use of this feature. diff --git a/proposals/0150-package-manager-branch-support.md b/proposals/0150-package-manager-branch-support.md new file mode 100644 index 0000000000..6fa969c889 --- /dev/null +++ b/proposals/0150-package-manager-branch-support.md @@ -0,0 +1,88 @@ +# Package Manager Support for branches + +* Proposal: [SE-0150](0150-package-manager-branch-support.md) +* Author: [Boris Bügling](https://github.com/neonichu) +* Review Manager: [Daniel Dunbar](https://github.com/ddunbar) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0150-package-manager-support-for-branches/5074) +* Bug: [SR-666](https://bugs.swift.org/browse/SR-666) + +## Introduction + +This proposal adds enhancements to the package manifest to support development of packages without strict versioning. This is one of two features, along with "Package Manager Support for Top of Tree development", being proposed to enable use of SwiftPM to develop on "top of tree" of related packages. + +## Motivation + +The package manager currently supports packages dependencies which are strictly versioned according to semantic versioning. This is how a package's dependencies should be specified when that package is released, but this requirement hinders some development workflows: + +- bootstrapping a new package which does not yet have a version at all +- developing related packages in tandem in between releases, when one package may depend on the latest revision of another, which has not yet been tagged for release + +## Proposed solution + +As a solution to this problem, we propose to extend the package manifest to allow specifying a branch or revision instead of a version to support revlocked packages and initial bootstrapping. In addition, we will also allow specifying a branch or revision as an option to the `pin` subcommand. + +## Detailed Design + +### Specifying branches or revisions in the manifest + +We will introduce a second initializer for `.Package` which takes a branch instead of a version range: + +```swift +import PackageDescription + +let package = Package( + name: "foo", + dependencies: [ + .Package(url: "http://url/to/bar", branch: "development"), + ] +) +``` + +In addition, there is also the option to use a concrete revision instead: + +```swift +import PackageDescription + +let package = Package( + name: "foo", + dependencies: [ + .Package(url: "http://url/to/bar", revision: "0123456789012345678901234567890123456789"), + ] +) +``` + +Note that the revision parameter is a string, but it will still be sanity checked by the package manager. It will only accept the full 40 character commit hash here for Git and not a commit-ish or tree-ish. + +Whenever dependencies are checked out or updated, if a dependency on a package specifies a branch instead of a version, the latest commit on that branch will be checked out for that package. If a dependency on a package specifies a branch instead of a version range, it will override any versioned dependencies present in the current package graph that other packages might specify. + +For example, consider this graph with the packages A, B, C and D: + +A -> (B:master, C:master) + B -> D:branch1 + C -> D:branch2 + +The package manager will emit an error in this case, because there are dependencies on package D for both `branch1` and `branch2`. + +While this feature is useful during development, a package's dependencies should be updated to point at versions instead of branches before that package is tagged for release. This is because a released package should provide a stable specification of its dependencies, and not break when a branch changes over time. To enforce this, it is an error if a package referenced by a version-based dependency specifies a branch in any of its dependencies. + +Running `swift package update` will update packages referencing branches to their latest remote state. Running `swift package pin` will store the commit hash for the currently checked out revision in the pins file, as well as the branch name, so that other users of the package will receive the exact same revision if pinning is enabled. If a revision was specified, users will always receive that specific revision and `swift package update` becomes a no op. + +### Pinning to a branch or revision + +In addition to specifying a branch or revision in the manifest, we will also allow specifying it when pinning: + +```bash +$ swift pin --branch +$ swift pin --revision +``` + +This is meant to be used for situations where users want to temporarily change the source of a package, but it is just an alternative way to get the same semantics and error handling described in the previous section. + +## Impact on existing code + +There will no impact on existing code. + +## Alternative considered + +We decided to make using a version-based package dependency with unversioned dependencies an error, because a released package should provide a stable specification of its dependencies. A dependency on a branch could break at any time when the branch is being changed over time. diff --git a/proposals/0151-package-manager-swift-language-compatibility-version.md b/proposals/0151-package-manager-swift-language-compatibility-version.md new file mode 100644 index 0000000000..e9421cf04b --- /dev/null +++ b/proposals/0151-package-manager-swift-language-compatibility-version.md @@ -0,0 +1,145 @@ +# Package Manager Swift Language Compatibility Version + +* Proposal: [SE-0151](0151-package-manager-swift-language-compatibility-version.md) +* Authors: [Daniel Dunbar](https://github.com/ddunbar), [Rick Ballard](https://github.com/rballard) +* Review Manager: [Anders Bertelrud](https://github.com/abertelrud) +* Status: **Implemented (Swift 3.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0151-package-manager-swift-language-compatibility-version/5183) +* Bug: [SR-3964](https://bugs.swift.org/browse/SR-3964) + +## Introduction + +This proposal adds support for the Swift compiler's new "language compatibility +version" feature to the package manager. + +## Motivation + +The Swift compiler now supports a "language compatibility version" flag which +specifies the Swift major language version that the compiler should try to +accept. We need support for an additional package manager manifest feature in +order for this feature to be used by Swift packages. + +## Proposed solution + +We will add support to the package manifest declaration to specify a set of +supported Swift language versions: + +```swift +let package = Package( + name: "HTTP", + ... + swiftLanguageVersions: [3, 4]) +``` + +When building a package, we will always select the Swift language version that +is most close to (but not exceeding) the major version of the Swift compiler in +use. + +If a package does not support any version compatible with the current compiler, +we will report an error. + +If a package does not specify any Swift language versions, the +language version to be used will match the major version of the the +package's Swift tools version (as discussed in a separate evolution proposal). A +Swift tools version with a major version of '3' will imply a default Swift +language version of '3', and a Swift tools version with a major version +of '4' will imply a default Swift language version of '4'. + +## Detailed design + +We are operating under the assumption that for the immediate future, the Swift +language compatibility version accepted by the compiler will remain an integer +major version. + +With this change, the complete package initializer will be: + +```swift + public init( + name: String, + pkgConfig: String? = nil, + providers: [SystemPackageProvider]? = nil, + targets: [Target] = [], + products: [Product] = [], + dependencies: [Dependency] = [], + swiftLanguageVersions: [Int]? = nil, + exclude: [String] = [] +``` + +where absence of the optional language version list indicates the default +behavior should be used for this package. + +### Example Behaviors + +Here are concrete examples of how the package manager will compile code, +depending on its language version declarations: + +* Version 3 Packager Manager & Swift 3 (only) Language Package + + The package manager will compile the code with `-swift-version 3`. + +* Version 3 Packager Manager & Swift 4 (only) Language Package + + The package manager will report an error, since the package supports no language + version compatible with the tools. + +* Version 3 Packager Manager & Swift [3, 4] Language Package + + The package manager will compile the code with `-swift-version 3`, matching the + major version of the tools in use. + +* Version 4 Packager Manager & Swift 3 (only) Language Package + + The package manager will compile the code with `-swift-version 3`. + +* Version 4 Packager Manager & Swift 4 (only) Language Package + + The package manager will compile the code with `-swift-version 4`. + +* Version 4 Packager Manager & Swift [3, 4] Language Package + + The package manager will compile the code with `-swift-version 4`. + + Clients wishing to validate actual Swift 3 compatibility are expected to do so + by using an actual Swift 3 implementation to build, since the Swift compiler + does not commit to maintaining pedantic Swift 3 compatibility (that is, it is + designed to *accept* any valid Swift 3 code in the `-swift-version 3` + compatibility mode, but not necessarily to *reject* any code which the Swift 3 + compiler would not have accepted). + +## Impact on existing code + +Since this is a new API, all packages will use the default behavior once +implemented. Because the default Swift tools version of existing packages +is "3.0.0" (pending approval of the Swift tools version proposal), the Swift +4 package manager will build such packages in Swift 3 mode. When packages +wish to migrate to the Swift 4 language, they can either update their +Swift tools version or specify Swift 4 as their language version. + +New packages created with `swift package init` by the Swift 4 tools will +build with the Swift 4 language by default, due to the Swift tools version +that `swift package init` chooses. + +This is a new manifest API, so packages which adopt this API will no longer be +buildable with package manager versions which do not recognize that +API. We expect to add support for this API to Swift 3.1, so it will be possible +to create packages which support the Swift 4 and 3 languages and the Swift +4 and 3.1 tools. + +## Alternatives considered + +We could have made the Swift language version default to the version of the +Swift tools in use if not specified. However, tying this to the Swift tools +version instead allows existing Swift 3 language packages to build with the +Swift 4 tools without changes, as they won't need to explicitly specify a Swift +language version in order to continue to build with the Swift 3 language. + +We chose not to support any command line features to modify the selected version +(e.g., to force a Swift 4 compiler to use Swift 3 mode where acceptable) in +order to keep this proposal simple. We will consider these in the future if they +prove necessary. + +We considered supporting a way to set the Swift language version on +a per-module basis, instead of needing to set it for the entire package. +We think this would be best done using build settings, which the package +manager does not yet support. We are pending per-module support for this +feature until build settings are supported. diff --git a/proposals/0152-package-manager-tools-version.md b/proposals/0152-package-manager-tools-version.md new file mode 100644 index 0000000000..92394f4156 --- /dev/null +++ b/proposals/0152-package-manager-tools-version.md @@ -0,0 +1,438 @@ +# Package Manager Tools Version + +* Proposal: [SE-0152](0152-package-manager-tools-version.md) +* Author: [Rick Ballard](https://github.com/rballard) +* Review Manager: [Anders Bertelrud](https://github.com/abertelrud) +* Status: **Implemented (Swift 3.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0152-package-manager-tools-version/5184) +* Bug: [SR-3965](https://bugs.swift.org/browse/SR-3965) + +## Introduction + +This proposal introduces a "Swift tools version" which is declared for each Swift package. +The tools version declares the minimum version of the Swift tools required to +use the package, determines what version of the PackageDescription API should +be used in the Package.swift manifest, and determines which Swift language +compatibility version should be used to parse the Package.swift manifest. + +This feature shall be added to Swift 3.1, to allow packages to manage the transition +from Swift 3 to Swift 4 compatibility. + +## Motivation + +This proposal addresses three problems with one mechanism. + +First, when a package adopts new features of Swift or the Swift Package Manager, +it may no longer be able to be compiled by an older version of the Swift tools, +and older tools may not even be able to interpret its Package.swift manifest. +Without a mechanism to handle this, a package author who tags a version +of their package which uses new features may break the builds of all +clients of that package who are using older tools unless that package adjusts its +major semantic version, which would unnecessarily stop clients who are using the +latest tools from getting that version. This is especially problematic during development of +a new version of the Swift tools, when new features have been added which are not yet +supported by the current release of the Swift tools. + +Second, one specific planned change for the Swift Package Manager is a revision +of the Package.swift PackageDescription API for Swift 4, to make it conform to +Swift conventions and to correct historical mistakes. In order to support backwards +compatibility, the old version of the PackageDescription API must remain available. +We need some way to determine which version of the PackageDescription API a package +wishes to use. + +Finally, as the Package.swift manifest is itself written in Swift, some +mechanism is needed to control which Swift language compatibility version should +be used when interpreting the manifest. This cannot be determined by a property +on the Package object in the manifest itself, as we must know what compatibility +version to interpret the manifest with before we have access to data specified +by Swift code in the manifest. + +## Proposed solution + +Each package will specify a Swift version (the "Swift tools version") which is the minimum version +of the Swift tools currently needed to build that package. This minimum version +will be specified in a file in the package, so it is managed in source +control just like any other package data and may differ for different +tagged versions of the package. + +When adopting new or revised PackageDescription API, or when making changes to a Package's source +code which require a new version of Swift, users will be expected to update +the package's Swift tools version to specify that it requires that version +of the Swift tools. + +The Swift Package Manager will continue to include the Swift 3 version of the +PackageDescription module for backwards compatibility, in addition to including +the new version of the PackageDescription module, as designed in a forthcoming +evolution proposal. The Swift tools version shall determine which version of the +PackageDescription module will be used when interpreting the Package.swift +manifest. + +The Swift Tools Version will also determine which Swift language compatibility +version should be used when interpreting the Package.swift manifest. +And it will determine the default Swift language compatibility version used to compile +the package's sources if not otherwise specified. + +## Detailed design + +When resolving package dependencies, if the version of a dependency that would normally +be chosen specifies a Swift tools version which is greater than the version in use, that +version of the dependency will be considered ineligible and dependency +resolution will continue with evaluating the next-best version. If +no version of a dependency (which otherwise meets the version requirements +from the package dependency graph) supports the version of the Swift tools in use, a +dependency resolution error will result. + +New PackageDescription API will be added as needed for upcoming features. When +new API is used in a Package.swift manifest, it will cause the package manager +to validate that the Swift tools version of that package specifies a version of +the tools which understands that API, or to emit an error with instructions to +update the Swift tools version if not. Note that if a [version-specific manifest](https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#version-specific-manifest-selection) +is present for the older tools version, that tools version will validate as +allowable even when newer features are adopted in the main Package.swift +manifest. + +The Swift tools version will determine the default Swift language compatibility version +used to compile the package's Swift sources if unspecified, but the Swift language +compatibility version for the package's sources is otherwise decoupled from the +Swift tools version. A separate Swift evolution proposal will describe how +to specify a Swift language compatibility version for package sources. + +The Swift Package Manager may consider the Swift tools version of a package +for other compatibility-related purposes as well. For example, if a +bugfix in a new version of the Swift Package Manager might break older +packages, the fixed behavior might only be applied to packages which have +adopted the newer Swift tools version. + +A new `swift package tools-version` command will be added to manage the +Swift tools version. This command will behave as follows: + +* `swift package tools-version` will report the Swift tools version of the package. + +* `swift package tools-version --set ` will set the Swift tools version to `value`. +It will also print an informational message advising the user of any changes that this +tools version change will necessitate, depending on what the prior and new tools versions +were. For example, changing the tools version might require converting the Package.swift manifest to +a different Swift language version, and to a different version of the PackageDescription API. + +* `swift package tools-version --set-current` will set the Swift tools version to the version +of the tools currently in use. + +If a package does not specify a Swift tools version, the Swift tools version is +"3.0.0". It is expected that in the future all Swift packages will specify a +Swift tools version. `swift package init` will set the Swift tools version of a +package it creates to the version of the tools in use. + +### How the Swift tools version is specified + +The Swift tools version will be specified by a special comment in the first line +of the Package.swift manifest. This is similar to how a DTD is defined for XML +documents. To specify a tools version, a Package.swift file must begin with the +string `// swift-tools-version:`, followed by a version number specifier. + +Though the Swift tools version refers to a Swift marketing version number and is +not a proper semantic version, the version number specifier shall follow the +syntax defined by [semantic versioning 2.0.0](http://semver.org/spec/v2.0.0.html), +with an amendment that the patch version component shall be optional and shall +be considered to be `0` if not specified. As we expect that patch versions will +not affect tools compatibility, the package manager will automatically elide the +patch version component when appropriate, including when setting a version using +the `swift package tools-version --set-current` command, to avoid unnecessarily +restricting package compatibility to specific patch versions. The semver syntax +allows for an optional pre-release version component or build version component; +those components will not be used by the package manager currently, but may be +used in a future release to provide finer-grained compatibility controls during +the development of a new version of Swift. + +After the version number specifier, an optional `;` character may be present; +it, and anything else after it until the end of the first line, will be ignored by +this version of the package manager, but is reserved for the use of future +versions of the package manager. + +The package manager will attempt to detect approximate misspellings of the Swift +tools version comment. As such, it is an error if the first line of the file +begins with `//`, contains the string `swift-tools-version` (with any +capitalization), but is not otherwise a valid tools version comment. Any other first +line of the file will not be considered to be a Swift tools version comment, in +which case the Swift tools version will be considered to be `3.0.0`. + +## Examples + +* The author of a package created with Swift 3 wishes to adopt the new Swift 4 +product definitions API in their manifest. Using a Swift 4 toolchain, the author +first runs `swift package tools-version --update` to make their package require +the Swift 4.0 tools. They then make any changes to their Package.swift manifest +needed to make it compatible with the Swift 4 language version and the revised +Swift 4 Package Manager PackageDescription API. Since their package sources are +still written with Swift 3, they should specify the Swift 3 language +compatibility version in their manifest, if they didn't already, so that it +doesn't start defaulting to building their sources as Swift 4 code. The author +is now free to adopt new PackageDescription API in their Package.swift manifest. +They are not required to update the language version of their package sources at +the same time. + +* A package author wishes to support both the Swift 3 and Swift 4 tools, while +conditionally adopting Swift 4 language features. The author specifies both +Swift language compatibility versions for their package sources (using a +mechanism discussed in a separate evolution proposal). Because their package +needs to support Swift 3 tools, the package's Swift tools version must be set +to `3.1`. Their Package.swift manifest must continue to be compatible with the +Swift 3 language, and must continue to use the Swift 3.1 version of the +PackageDescription API. + +* The author of a package created with the Swift 3 tools wishes to convert the +package's sources to the Swift 4 language version. They specify Swift 4 as their +package's language compatibility version (using a mechanism discussed in a +separate evolution proposal). When they try to build their package, the package +manager emits an error informing them that they must update their Swift tools +version to 4.0 or later, because the Swift 4 tools are required to build a +package when it no longer supports the Swift 3 language version. The author runs +`swift package tools-version --update` to make their package require the Swift +4.0 tools. They then make any changes to their Package.swift manifest required +to make it compatible with the Swift 4 language version and the revised Swift 4 +Package Manager PackageDescription API. + +## Impact on existing code + +There is no impact on existing packages. Since existing packages do not specify +a Swift tools version, they will default to building their sources +with the Swift 3 language compatibility mode, and will be able to be built +by both Swift 3 and Swift 4 tools. + +Use of the package manager's [version-specific tag selection](https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#version-specific-tag-selection) +mechanism will no longer be necessary in many situations. Previously, authors needed to employ +that mechanism in order to tag the last version of a package compatible with an old +version of the Swift tools before adopting new Swift features, to avoid break clients +of the package still using the old version of the Swift tools. Now, when adopting new Swift features, a +package author merely needs to set their Swift tools version to that new version, and +dependency resolution performed by an older version of the Swift tools (starting with Swift 3.1) will consider +those new package versions ineligible. + +The existing version-specific tag selection mechanism may still be useful for +authors who wish to publish new parallel versions of their package for multiple +versions of the Swift tools. + +Use of the package manager's [version-specific manifest selection](https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#version-specific-manifest-selection) +mechanism may still be useful for authors who wish to conditionally adopt new features of the Swift tools in +their Package.swift manifest without needing to update their Swift tools version to +exclude older versions of Swift. + +Packages which have used conditional compilation blocks in the Package.swift +manifest to adopt new PackageDescription features while remaining compatible +with older versions of the Swift tools will no longer be able to do so for future versions +of Swift, and must instead use version-specific manifest selection. This is +because when the newer tools interpret the Package.swift manifest, those tools will see +that new PackageDescription APIs are in use, will not detect the alternate code behind +the conditional compilation blocks, and will thus emit an error requiring the +user to update the Swift tools version to a version which supports those new +APIs. + +The following table shows an example of which Swift language version will be +used to interpret the Package.swift manifest, and to interpret the package's +sources, based on the Swift tools in use and the parameters specified by the +package. + +| Swift Tools | Swift Tools Version | [Swift Language Compatibility Version](http://link/to/proposal) | Language Version Used | +|:---:|:---:|:---:| --- | +| 3.1 | Not Present | Not Present | Manifest: 3 Sources: 3 | +| 3.1 | 3.1 | Not Present | Manifest: 3 Sources: 3 | +| 3.1 | 3.1 | 3 | Manifest: 3 Sources: 3 | +| 3.1 | 3.1 | 3, 4 | Manifest: 3 Sources: 3 | +| Any | 3.1 | 4 | Error | +| 3.1 | 4.0 | Any | Error | +| 4.0 | Not Present | Not Present | Manifest: 3 Sources: 3 | +| 4.0 | 3.1 | Not Present | Manifest: 3 Sources: 3 | +| 4.0 | 3.1 | 3 | Manifest: 3 Sources: 3 | +| 4.0 | 3.1 | 3, 4 | Manifest: 3 Sources: 4 | +| 4.0 | 4.0 | Not Present | Manifest: 4 Sources: 4 | +| 4.0 | 4.0 | 3 | Manifest: 4 Sources: 3 | +| 4.0 | 4.0 | 3, 4 | Manifest: 4 Sources: 4 | +| 4.0 | 4.0 | 4 | Manifest: 4 Sources: 4 | + +## Alternatives considered + +We considered a number of alternative approaches that might avoid the need for +adding this new Swift tools version; however, we think that this proposal +is compelling compared to the alternatives considered. + +### Don't change the PackageDescription manifest API + +If we chose not to change the PackageDescription API, we would not need a way +to determine which version of the module to use when interpreting a manifest. +However, we think that it is important for this API to be made compliant with +the Swift language conventions, and to review the API with our community. +It would be best to do this now, while the Swift package ecosystem is relatively +young; in the future, when the ecosystem is more mature, it will be more +painful to make significant changes to this API. + +Not changing this API would still leave the problem of figuring out which +Swift language compatibility version to interpret the manifest in. It's possible +that Package.swift manifests won't be significantly affected by Swift +language changes in Swift 4, and could mostly work in either language compatibility +mode without changes. However, we don't know whether that will be the case, +and it would be a significant risk to assume that it will be. + +Finally, we will need to add new API to the PackageDescription module to +support new features, and without a Swift tools version, adoption of new features +would break existing clients of a package that aren't using the latest tools. + +### Rely on conditional compilation blocks + +We could choose to ask package authors to use Swift conditional compilation +blocks to make their manifests compatible with both Swift 3 and Swift 4. +Unfortunately, this might be a lot of work for package authors, and result in a +hard-to-read manifests, if the PackageDescription API changes or Swift 4 +language changes are significant. + +Another major downside of this approach is that until package authors do the +work of adding conditional compilation blocks, their packages would fail to +build with the Swift 4 tools. In order to build with the Swift 4 tools, you'd +both need to update your own packages with conditional compilation, and you'd +need to wait for any packages you depend upon to do the same. This could be a +major obstacle to adopting the Swift 4 tools. + +Finally, we are not convinced that all authors would bother to add conditional +compilation blocks to preserve Swift 3 compatibility when they update their +packages for the Swift 4 tools. Any packages which were updated but not given +conditional compilation blocks would now break the builds of any clients still +using the Swift 3 tools. + +### Rely on semantic versioning + +We could expect that package authors bump their packages' major semantic version +when updating those packages for the Swift 4 tools, thereby preventing clients +who were still using the Swift 3 tools from automatically getting the updated +version of their dependency and failing to build. There are several problems with +this approach. + +First, this does nothing to allow packages to be used with new Swift tools without +needing to be updated for those tools. We don't want package authors to need +to immediately adopt the Swift 4 language compatibility version and PackageDescription +API before they can build their package with the new tools. Using a Swift +tools version allows us to support multiple versions of the PackageDescription +API and the Swift language, so existing packages will continue to work +with newer tools automatically. + +Second, this forces clients of a package to explicitly opt-in to updated +versions of their dependencies, even if there was otherwise no API change. +The Swift tools version mechanism that we have proposed allows packages +to automatically get updated versions of their dependencies when using Swift +tools that are new enough to be able to build them, which is preferable. + +Finally, we are not confident that all package authors would reliably update +their semantic version when updating their package for newer tools. If they failed +to do so, clients still using the older Swift tools would fail to build. + +### Relying on the package manager's existing versioning mechanisms + +In Swift 3, the package manager introduced two versioning mechanisms: +[version-specific tag selection](https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#version-specific-tag-selection), +and [version-specific manifest selection](https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#version-specific-manifest-selection). +These mechanisms can be used to publish updated versions of a package without +breaking the builds of clients who are still using older Swift tools. We think +that these mechanisms are still useful and can be used in concert with the +Swift tools version, as described in the "Impact on existing code" section. +However, they are insufficient to completely solve the versioning problem. + +These mechanisms allow a package to be updated for new Swift tools without +breaking clients who are still using older Swift tools, but they do not allow +a package that has not been updated to be built with new Swift tools. Again, +we don't want package authors to need to immediately adopt the Swift 4 language +compatibility version and PackageDescription API before they can build their +package with the new tools. + +These mechanisms are also opt-in and may not be known to all package authors. +If a package author fails to explicitly adopt these mechanisms when updating +a package, they will break the builds of clients that are still using older +Swift tools. In contrast, the Swift tools version mechanism that we have proposed works +by default without requiring package authors to know about extra opt-in +mechanisms. + +### Automatically re-interpret Package.swift manifest in different modes + +We considered having the package manager automatically try to reinterpret +a Package.swift manifest in different modes until it finds a mode that +can successfully interpret it, so that we wouldn't need an explicit specifier +of which Swift language compatibility version or PackageDescription module version +the Package.swift manifest is using. + +We saw three major problems with this. First, this would make it very difficult +to provide high quality diagnostics when a manifest has an error or warning. +If the manifest cannot be interpreted cleanly in any of the supported modes, +we'd have no way to know which mode it should have been interpreted in -- +or whether the required mode is even known to the version of the Swift +tools in use. That means that the errors we provide might be incorrect +with respect to the actual version of the Swift language or the PackageDescription +module that the manifest targets. + +Second, any subtle incompatibilities introduced by a difference in Swift language +compatibility versions could cause the manifest to interpret without errors +in the wrong mode, but result in unexpected behavior. + +Finally, this could cause performance problems for the package manager. Because +the package manager needs to interpret Package.swift manifests from potentially +many packages during dependency resolution, forcing it to interpret +each manifest multiple times could add an undesirable delay to the +`swift build` or `swift package update` commands. + +### Provide finer-grained versioning controls + +In this proposal we've provided a single versioning mechanism which controls multiple +things: the Swift language compatibility version used to parse the manifest, +the version of the PackageDescription module to use, and the minimum version +of the tools which will consider a package version eligible during dependency +resolution. We could instead provide separate controls for each of these +things. However, we feel that doing so would add unnecessary complexity to +the Swift package manager. We do not see a compelling use-case for needing +to control these different features independently, so we are consolidating +them into one version mechanism as a valuable simplification. + +### Rename the PackageDescription module + +We considered giving the new version of the PackageDescription module +a different name, and having users change the import statement in their +Package.swift when they want to adopt the revised PackageDescription API. +This would mean that the package manager would not automatically switch +which version of the PackageDescription module to use based on the Swift +tools version. However, this did not seem like a better experience for our +users. It would also allow users to import both modules, which we do not +want to support. And it would allow users to continue using the old +PackageDescription API in a manifest that is otherwise updated for +Swift 4, which we also do not want to support. + +### Store the Swift tools version in a separate file + +Instead of storing the Swift tools version in a comment at the top of the +Package.swift manifest, we considered storing it in a separate file. +Possibilities considered were to either store it as JSON in a +`.swift-tools-version` file, or to store it in a `.swift-version` file in the +format already established by the [swiftenv](https://github.com/kylef/swiftenv/) +tool. + +Reasons we prefer to store this in the Package.swift manifest include: + +* Keeping the Swift tools version in the same file as the rest of the manifest +data eliminates the possibility of forgetting to include both the Package.swift +manifest and a file specifying the Swift tools version in every commit which +should affect both. Users may also find it more convenient to only need to +commit a single file when making manifest changes. + +* Users may like being able to see the Swift tools version when reading a +Package.swift manifest, instead of needing to look in a separate file. + +* Supporting the `.swift-version` standard would require supporting toolchain +names as a stand-in for a Swift tools version, which complicates this mechanism +significantly. + +### Specify the Swift tools version with a more "Swifty" syntax + +Several alternative suggestions were given for how to spell the comment that +specifies the Swift tools version. We could make it look like a line of Swift +code (`let toolsVersion = ToolsVersion._3_1`), a compiler directive +(`#swift-pm(tools-version: 3.1)`), or use camel case in the comment (`// +swiftToolsVersion: 3.1`). We rejected the former two suggestions because this +version is not actually going to be interpreted by the Swift compiler, so it's +misleading to make it appears as if it is a valid line of Swift code. Embedding +metadata in a leading comment is a strategy with a clear precedent in XML & +SGML. For the latter suggestion, we are preferring kebab-case +(`swift-tools-version`) because it is distinct from normal Swift naming, and +more clearly stands out as its own special (tiny) language, which it is. diff --git a/proposals/0153-compensate-for-the-inconsistency-of-nscopyings-behaviour.md b/proposals/0153-compensate-for-the-inconsistency-of-nscopyings-behaviour.md new file mode 100644 index 0000000000..f5005565bd --- /dev/null +++ b/proposals/0153-compensate-for-the-inconsistency-of-nscopyings-behaviour.md @@ -0,0 +1,219 @@ +# Compensate for the inconsistency of `@NSCopying`'s behaviour + +* Proposal: [SE-0153](0153-compensate-for-the-inconsistency-of-nscopyings-behaviour.md) +* Author: [Torin Kwok](https://github.com/TorinKwok) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Rejected** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0153-compensate-for-the-inconsistency-of-nscopying-s-behaviour/5341), [Additional Commentary](https://forums.swift.org/t/addressing-unimplemented-evolution-proposals/40322) +* Bug: [SR-4538](https://bugs.swift.org/browse/SR-4538) + +**Note**: This proposal was initially accepted without an implementation on 2017-03-01, and has not been implemented since. The Core Team has decided to retroactively reject the proposal. + +## Introduction + +First of all, in Swift, the Objective-C `copy` property attribute translates to `@NSCopying`. + +Like Objective-C, in Swift, avoiding accessing ivar via setter methods in initializer is considered as the best practice. Unlike Objective-C, which gives developers the freedom to decide on whether assign a value to a property by invoking setter or by accessing ivar directly, accessing a property in Swift from within an initializer always does direct access to the storage rather than going through the setter, even if using `dot` syntax. + +However, as a side-effect, `@NSCopying` attribute does not work as consistently as we usually expected in Swift initializers after developers declared a property as `@NSCopying`. + +This proposal is intent on proposing several solutions to this inconsistency. + +## Swift-evolution thread + +[@NSCopying currently does not affect initializers](https://forums.swift.org/t/nscopying-currently-does-not-affect-initializers/5019) + +## Motivation + +Here's an example of the inconsistency mentioned above: + +```swift +class Person: NSObject, NSCopying { + + var firstName: String + var lastName: String + var job: String? + + init( firstName: String, lastName: String, job: String? = nil ) { + self.firstName = firstName + self.lastName = lastName + self.job = job + + super.init() + } + + /// Conformance to protocol + func copy( with zone: NSZone? = nil ) -> Any { + let theCopy = Person.init( firstName: firstName, lastName: lastName ) + theCopy.job = job + + return theCopy + } + + /// For convenience of debugging + override var description: String { + return "\(firstName) \(lastName)" + ( job != nil ? ", \(job!)" : "" ) + } + + } +``` + +`Person` class has promised that it conforms to `` protocol. + +``` swift +let johnAppleseed = Person( firstName: "John", lastName: "Appleseed", job: "CEO" ) +var refJohnAppleseed = johnAppleseed // assigning without copying semantic + +refJohnAppleseed.job = "Engineer" + +// `cloneJohnAppleseed` and `johnAppleseed` have the identical `job` ... + +print( refJohnAppleseed ) // Prints "John Appleseed, Engineer" +print( johnAppleseed ) // Prints "John Appleseed, Engineer" too + +// ... and the assertion **would not** fail: +assert( refJohnAppleseed === johnAppleseed ) + +// Assigning a copy of johnAppleseed to clonedJohnAppleseed, +// which was returned by `copy( zone: ) -> Any` +var clonedJohnAppleseed = johnAppleseed/* refJohnAppleseed is also okay */.copy() as! Person + +clonedJohnAppleseed.job = "Designer" +print( clonedJohnAppleseed ) // Prints "John Appleseed, Designer" +print( johnAppleseed ) // Prints "John Appleseed, Engineer" +// Alright as what you see, setting the job of `clonedJohnAppleseed` doesn't affect the +// job stored in `johnAppleseed`. +``` + +Up to now, everything seems to run right. However, problems will soon emerge once we begin introducing a new class consuming instances of `Person` class: + +``` swift +class Department: NSObject { + + // Here, we're expecting that `self.employee` would automatically + // store the deeply-copied instance of `Person` class + @NSCopying var employee: Person + + init( employee candidate: Person ) { + + // CAUTION! That's the key point: + // `self.employee` has been marked with `@NSCopying` attribute + // but what would take place here is only the shallow-copying. + // + // In the other words, `self.employee` will share identical underlying + // object with `candidate`. + self.employee = candidate + super.init() + + // Assertion will definitely fail since Swift do not actually + // copy the value assigned to this property even though + // `self.employee` has been marked as `@NSCoyping`: + + /* assert( self.employee !== employee ) */ + } + + override var description: String { + return "A Department: [ ( \(employee) ) ]" + } + + } +``` + +`Department`'s designated initializer receives an external instance of `Person` and expects to assign its deeply-copied value to `self.employee` property. + +``` swift +let isaacNewton = Person( firstName: "Isaac", lastName: "Newton", job: "Mathematician" ) +let lab = Department.init( employee: isaacNewton ) + +isaacNewton.job = "Astronomer" + +print( isaacNewton ) // Prints "Isaac Newton, Astronomer" + +print( lab.employee ) // Prints "Isaac Newton, Astronomer" +// Expected output printed here is "Isaac Newton, Mathematician" instead +``` + +Setting the job of `isaacNewton` affects the job stored in `lab.employee`. That's an unexpected behavior as we have declared `employee` property as `@NSCopying`. Obviously, `@NSCopying` semantic became effectless implicitly in the initializer of `Department` class. + +For the moment, if we indeed require copy we have to invoke `copy()` method explicitly on instances that want to be copied to make sure that classes' properties are able to store deeply-copied results during the initialization: + +``` swift +init( employee candidate: Person ) { + // ... + self.employee = candidate.copy() as! Person + // ... + } +``` + +The reason why it is considered *inconsistency* is that `@NSCopying` contract will be well respected within the rest of class definition: + +``` swift +lab.employee = isaacNewton +isaacNewton.job = "Physicist" + +print( isaacNewton ) // Prints "Isaac Newton, Physicist" +print( lab.employee ) // Prints "Isaac Newton, Astronomer" +``` + +It is undeniably reasonable to enforce programmers to access instance variables directly from initializer methods because of the potential troubles made by setter methods' additional side-effects when the initialization is not complete yet. However, I believe we at least should be warned by the Swift compiler when we assigned an instance of `NSCopying` conforming class to a class's property declared as `@NSCopying` during the initialization. + +In Objective-C, developers can make a decision on this process explicitly by writing done either: + +```objc +- ( instancetype )initWithName: ( NSString* )name { + // ... + self->_name = [ name copy ]; + // ... + } +``` + +or: + +```objc +- ( instancetype )initWithName: ( NSString* )name { + // ... + self.name = name; /* self.name has been qualified with @property ( copy ) */ + // ... + } +``` + +Speaking of Swift, however, there is no stuff like `->` operator to access ivar directly. As a result, with property marked with `@NSCopying` attribute, developers who are new to this language, especially those who have had experience of writing Objective-C, are likely to automatically suppose it acts normally when they're writing down code like `self.employee = candidate` in initializer. That's bug-prone. + +## Proposed solution + +Do the compiler magic to call `copy( with: )` in the initializer so that `@NSCopying` attribute no longer subjects to the fact that setter methods would not be invoked in initializers. **Copying should always take place after a property has been declared as `@NSCopying`**. It seems like the most direct way to maintain the `@NSCopying` contract without changing the underlying direct-storage model. + +## Source compatibility + +Projects written with prior versions of Swift that have not yet adopted this proposal may fail to be built due to the compile-time error. But overall, it will be easy to be resolved. IDEs' Fix-it and auto migrator tools will deal with all works painlessly. + +## Effect on ABI stability + +The proposal doesn't change the ABI of existing language features. + +## Alternatives Considered + +### Compile-time checking + +Instead of introducing the copy within the initializer, have the compiler emit a **compile-time error or warning** if developers are performing an assignment operation from within an initializer between a property declared as `@NSCopying` and an instance of a `` protocol conforming class. Also, speaking of GUI integrated development environments such as Xcode, leaving this kind of error or warning **FIXABLE** would be needed in order to make them can be quickly fixed by both IDEs and migrator tools through simply appending `.copy() as! AutoInferredClassType`. + +With the adjustment mentioned above, following code fragment, for instance, will no longer be successfully compiled: + +``` swift +... +class Person: NSObject, NSCopying { /* ... */ } +... +@NSCopying var employee: Person +... +init( employee candidate: Person ) { + // ... + self.employee = candidate + // ... + } +``` + +GUI IDE will be expected to leave developers a fixable error or warning, and thus if we hit the either red or yelloe point in Xcode, or something similar to those in other IDEs, they will automatically append the lacked statement: + +> self.employee = candidate***.copy() as! Person*** + +Inferring `AutoInferredClassType` from context should be the responsibility of compiler. diff --git a/proposals/0154-dictionary-key-and-value-collections.md b/proposals/0154-dictionary-key-and-value-collections.md new file mode 100644 index 0000000000..f5ad8d6d71 --- /dev/null +++ b/proposals/0154-dictionary-key-and-value-collections.md @@ -0,0 +1,177 @@ +# Provide Custom Collections for Dictionary Keys and Values + +* Proposal: [SE-0154](0154-dictionary-key-and-value-collections.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0154-provide-custom-collections-for-dictionary-keys-and-values/5322) + +## Introduction + +This proposal addresses significant unexpected performance gaps when using dictionaries. It introduces type-specific collections for a `Dictionary` instance's `keys` and `values` properties. + +New collection types provide efficient key lookup and mutable access to dictionary values, allowing in-place updates and copy-on-write optimization of stored values. The addition of these new types impacts the standard library ABI, since we won't be able to use types aliases from the existing types for `keys` and `values`. + +Swift-evolution thread: [[Proposal Draft] Provide Custom Collections for Dictionary Keys and Values](https://forums.swift.org/t/proposal-draft-provide-custom-collections-for-dictionary-keys-and-values/4244) + + +## Motivation + +This proposal address two problems: + +* While a dictionary's `keys` collection is fine for iteration, its implementation is inefficient when looking up a specific key, because `LazyMapCollection` doesn't know how to forward lookups to the underlying dictionary storage. +* Dictionaries do not offer value-mutating APIs. The mutating key-based subscript wraps values in an `Optional`. This prevents types with copy-on-write optimizations from recognizing they are singly referenced. + +This proposal uses the following `[String: [Int]]` dictionary to demonstrate these problems: + +```swift +var dict = ["one": [1], "two": [2, 2], "three": [3, 3, 3]] +``` + +### Inefficient `dict.keys` Search + +Swift coders normally test key membership using `nil` checks or underscored optional bindings: + +```swift +if dict["one"] != nil { + // ... +} +if let _ = dict["one"] { + // ... +} +``` + +These approaches provide the expected performance of a dictionary lookup but they read neither well nor "Swifty". Checking the `keys` view reads much better but introduces a serious performance penalty: this approach requires a linear search through a dictionary's keys to find a match. + +```swift +if dict.keys.contains("one") { + // ... +} +``` + +A similar dynamic plays out when comparing `dict.index(forKey:)` and `dict.keys.index(of:)`. + +### Inefficient Value Mutation + +Dictionary values can be modified through the keyed subscript by direct reassignment or by using optional chaining. Both of these statements append `1` to the array stored by the key `"one"`: + +```swift +// Direct re-assignment +dict["one"] = (dict["one"] ?? []) + [1] + +// Optional chaining +dict["one"]?.append(1) +``` + +Both approaches present problems. The first is complex and hard to read. The second ignores the case where `"one"` is not a key in the dictionary, and is therefore less useful even if more streamlined. Furthermore, neither approach allows the array to grow in place—they introduce an unnecessary copy of the array's contents even though `dict` is the sole holder of its storage. + +Adding mutation to a dictionary's index-based subscripting isn't possible. Changing a key stored at a particular index would almost certainly modify its hash value, rendering the index incorrect. This violates the requirements of the `MutableCollection` protocol. + +## Proposed Solution + +This proposal adds custom collections for the `keys` and `values` dictionary properties. This follows the example set by `String`, which presents multiple views of its contents. A new `Keys` collection introduces efficient key lookup, while a new `Values` collection provides a mutable collection interface to dictionary values. Both new collections are nested in the `Dictionary` type. + +These changes make the simple approach for testing whether a dictionary contains a key an efficient one: + +```swift +// Fast, not slow +if dict.keys.contains("one") { + // ... +} +``` + +As a mutable collection, `values` enables modification without copies or clumsy code: + +```swift +if let i = dict.index(forKey: "one") { + dict.values[i].append(1) // no copy here +} else { + dict["one"] = [1] +} +``` + +Both the `keys` and `values` collections share the same index type as `Dictionary`. This allows the above sample to be rewritten as: + +```swift +// Using `dict.keys.index(of:)` +if let i = dict.keys.index(of: "one") { + dict.values[i].append(1) +} else { + dict["one"] = [1] +} +``` + +## Detailed design + +* The standard library introduces two new collection types: `Dictionary.Keys` and `Dictionary.Values`. +* A `Dictionary`'s `keys` and `values` properties change from `LazyMapCollection` to these new types. +* The new collection types are not directly constructable. They are presented only as views into a dictionary. + +```swift +struct Dictionary: ... { + /// A collection view of a dictionary's keys. + struct Keys: Collection { + subscript(i: Index) -> Key { get } + // Other `Collection` requirements + } + + /// A mutable collection view of a dictionary's values. + struct Values: MutableCollection { + subscript(i: Index) -> Value { get set } + // Other `Collection` requirements + } + + var keys: Keys { get } + var values: Values { get set } + // Remaining Dictionary declarations +} +``` + +## Impact on existing code + +The performance improvements of using the new `Dictionary.Keys` type and the mutability of the `Dictionary.Values` collection are both additive in nature. + +Most uses of these properties are transitory in nature. Adopting this proposal should not produce a major impact on existing code. The only impact on existing code exists where a program explicitly specifies the type of a dictionary's `keys` or `values` property. In those cases, the fix would be to change the specified type. + + +## Alternatives considered + +1. Add additional compiler features that manage mutation through existing key-based subscripting without the copy-on-write problems of the current implementation. This could potentially be handled by upcoming changes to copy-on-write semantics and/or inout access. + +2. Provide new APIs for updating dictionary values with a default value, eliminating the double-lookup for a missing key. The approach outlined in this proposal provides a way to remove one kind of double-lookup (mutating a value that exists) but doesn't eliminate all of them (in particular, checking for the existence of a key before adding). + + These could be written in a variety of ways: + + ```swift + // Using a 'SearchState' type to remember key position + dict.entries["one"] + .withDefault([]) + .append(1) + + // Using a two-argument subscript + dict["one", withDefault: []].append(1) + + // Using a closure with an inout argument + dict.withValue(forKey: "one") { (v: inout Value?) in + if v != nil { + v!.append(1) + } else { + v = [1] + } + } + ``` + +3. Restructure `Dictionary`'s collection interface such that the `Element` type of a dictionary is its `Value` type instead of a `(Key, Value)` tuple. That would allow the `Dictionary` type itself to be a mutable collection with an `entries` or `keysAndValues` view similar to the current collection interface. This interface might look a bit like this: + + ```swift + let valuesOnly = Array(dict) + // [[2, 2], [1], [3, 3, 3]] + let keysAndValues = Array(dict.entries) + // [("two", [2, 2]), ("one", [1]), ("three", [3, 3, 3])] + + let foo = dict["one"] + // Optional([1]) + + let i = dict.keys.index(of: "one")! + dict[i].append(1) + ``` diff --git a/proposals/0155-normalize-enum-case-representation.md b/proposals/0155-normalize-enum-case-representation.md new file mode 100644 index 0000000000..8e0474d166 --- /dev/null +++ b/proposals/0155-normalize-enum-case-representation.md @@ -0,0 +1,273 @@ +# Normalize Enum Case Representation + +* Proposal: [SE-0155][] +* Authors: [Daniel Duan][], [Joe Groff][] +* Review Manager: [John McCall][] +* Status: **Implemented (Swift 3.0)** +* Decision Notes: [Rationale][] +* Previous Revision: [1][Revision 1], [Originally Accepted Proposal][], [Expired Proposal][] +* Bugs: [SR-4691](https://bugs.swift.org/browse/SR-4691), [SR-12206](https://bugs.swift.org/browse/SR-12206), [SR-12229](https://bugs.swift.org/browse/SR-12229) + +## Introduction + +In Swift 3, associated values of an enum case are represented by a tuple. This +implementation causes inconsistencies in case declaration, construction and +pattern matching in several places. + +Enums, therefore, can be made more "regular" when we replace tuple as the +representation of associated case values. This proposal aims to define the +effect of doing so on various parts of the language. + +Swift-evolution thread: [Normalize Enum Case Representation (rev. 2)][] + +## Motivation + +When user declares a case for an enum, a function which constructs the +corresponding case value is declared. We'll refer to such functions as _case +constructors_ in this proposal. + +```swift +enum Expr { + // this case declares the case constructor `Expr.elet(_:_:)` + indirect case elet(locals: [(String, Expr)], body: Expr) +} + +// f's signature is f(_: _), type is ([(String, Expr)], Expr) -> Expr +let f = Expr.elet + +// `f` is just a function +f([], someExpr) // construct a `Expr.elet` +``` + +There are many surprising aspects of enum constructors, however: + +1. After [SE-0111][], Swift function's fully qualified name consists of its base + name and all of its argument labels. User can use the full name of the + function at use site. In the example above, `locals` and `body` are currently + not part of the case constructors name, therefore the expected syntax is + invalid. + + ```swift + func f(x: Int, y: Int) {} + f(x: y:)(0, 0) // Okay, this is equivalent to f(x: 0, y: 0) + Expr.elet(locals: body:)([], someExpr) // this doesn't work in Swift 3 + ``` +2. Case constructors cannot include a default value for each parameter. This + is yet another feature available to functions. + +As previous mentioned, these are symptoms of associated values being a tuple +instead of having its own distinct semantics. This problem manifests more in +Swift 3's pattern matching: + +1. A pattern with a single value would match and result in a tuple: + + ```swift + // this works for reasons most user probably don't expect! + if case .elet(let wat) = anExpr { + eval(wat.body) + } + ``` + +2. Labels in patterns are not enforced: + + ```swift + // note: there's no label in the first sub-pattern + if case .elet(let p, let body: q) = anExpr { + // code + } + ``` + +These extra rules makes pattern matching difficult to teach and to expand to +other types. + +## Proposed Solution + +We'll add first class syntax (which largely resemble the syntax in Swift 3) for +declaring associated values with labels. Tuple will no longer be used to +represent the aggregate of associated values for an enum case. This means +pattern matching for enum cases needs its own syntax as well (as opposed to +piggybacking on tuple patterns, which remains in the language for tuples.). + +## Detailed Design + +### Compound Names For Enum Constructors + +Associated values' labels should be part of the enum case's constructor name. +When constructing an enum value with the case name, label names must either be +supplied in the argument list it self, or as part of the full name. + +```swift +Expr.elet(locals: [], body: anExpr) // Okay, the Swift 3 way. +Expr.elet(locals: body:)([], anExpr) // Okay, equivalent to the previous line. +Expr.elet(locals: body:)(locals: 0, body: 0) // This would be an error, however. +``` + +Note that since the labels aren't part of a tuple, they no longer participate in +type checking, behaving consistently with functions. + +```swift +let f = Expr.elet // f has type ([(String, Expr)], Expr) -> Expr +f([], anExpr) // Okay! +f(locals: [], body: anExpr) // Won't compile. +``` + +Enum cases should have distinct *full* names. Therefore, shared base name will +be allowed: + +```swift +enum SyntaxTree { + case type(variables: [TypeVariable]) + case type(instantiated: [Type]) +} +``` + +Using only the base name in pattern matching for the previous example would be +ambiguous and result in an compile error. In this case, the full name must be +supplied to disambiguate. + +```swift +case .type // error: ambiguous +case .type(variables: let variables) // Okay +``` + +### Default Parameter Values For Enum Constructors + +From a user's point view, declaring an enum case should remain the same as Swift +3 except now it's possible to add `= expression` after the type of an +associated value to convey a default value for that field. + +```swift +enum Animation { + case fadeIn(duration: TimeInterval = 0.3) // Okay! +} +let anim = Animation.fadeIn() // Great! +``` + +Updated syntax: + +```ebnf +union-style-enum-case = enum-case-name [enum-case-associated-value-clause]; +enum-case-associated-value-clause = "(" ")" + | "(" enum-case-associated-value-list ")"; +enum-case-associated-value-list = enum-associated-value-element + | enum-associated-value-element "," + enum-case-associated-value-list; +enum-case-associated-value-element = element-name type-annotation + [enum-case-element-default-value-clause] + | type + [enum-case-element-default-value-clause]; +element-name = identifier; +enum-case-element-default-value-clause = "=" expression; +``` + +### Alternative Payload-less Case Declaration + +In Swift 3, the following syntax is valid: + +```swift +enum Tree { + case leaf() // the type of this constructor is confusing! +} +``` + +`Tree.leaf` has a very unexpected type to most Swift users: `(()) -> Tree` + +We propose this syntax become illegal. User must explicitly declare +associated value of type `Void` if needed: + +```swift +enum Tree { + case leaf(Void) +} +``` + +## Source compatibility + +Despite a few additions, case declaration remain mostly source-compatible with +Swift 3, with the exception of the change detailed in "Alternative Payload-less +Case Declaration". + +Syntax for case constructor at use site remain source-compatible. + +## Effect on ABI stability and resilience + +After this proposal, enum cases may have compound names. This means the standard +library will expose different symbols for enum constructors. The name mangling +rules should also change accordingly. + +## Alternative Considered + +Between case declaration and pattern matching, there exist many reasonable +combinations of improvement. On one hand, we can optimize for consistency, +simplicity and teachability by bringing in as much similarity between enum and +other part of the language as possible. Many decisions in the first revision +were made in favor if doing so. Through the feedbacks from swift-evolution, we +found that some of the changes impedes the ergonomics of these features too much +. In this section, we describe some of the alternatives that were raised and +rejected in hope to strike a balance between the two end of the goals. + +We discussed allowing user to declare a *parameter name* ("internal names") +for each associated value. Such names may be used in various rules in pattern +matching. Some feedback suggested they maybe used as property names when we +make enum case subtypes of the enum and resembles a struct. This feature is not +included in this proposal because parameter names are not very useful *today*. +Using them in patterns actually don't improve consistency as users don't use +them outside normal function definitions at all. If enum case gains a function +body in a future proposal, it'd be better to define the semantics of parameter +names then, as opposed to locking it down now. + +To maintain ergonomics/source compatibility, we could allow user to choose +arbitrary bindings for each associated value. The problem is it makes the +pattern deviate a lot from declaration and makes it hard for beginners to +understand. This also decrease readability for seasoned users. + +Along the same line, a pattern that gets dropped is binding all associated +values as a labeled tuple, which tuple pattern allowed in Swift 3. As T.J. +Usiyan [pointed out][TJs comment], implementation of the equality protocol would +be simplified due to tuple's conformance to `Equatable`. This feature may still +be introduced with alternative syntax (perhaps related to splats) later without +source-breakage. And the need to implement `Equatable` may also disappear with +auto-deriving for `Equatable` conformance. + +## Revision History + +The [first revision of this proposal][Revision 1] mandated that the labeled form of +sub-pattern (`case .elet(locals: let x, body: let y)`) be the only acceptable +pattern. Turns out the community considers this to be too verbose in some cases. + +A drafted version of this proposal considered allowing "overloaded" declaration +of enum cases (same full-name, but with associated values with different types). +We ultimately decided that this feature is out of the scope of this proposal. + +The [second revision of this proposal][Originally Accepted Proposal] was accepted with revisions. As originally written, the proposal required that pattern matching against an enum match either by the name of the bound variables, or by explicit labels on the parts of the associated values: + +``` +enum Foo { + case foo(bar: Int) +} + +func switchFoo(x: Foo) { + switch x { + case .foo(let bar): // ok + case .foo(bar: let bar): // ok + case .foo(bar: let bas): // ok + case .foo(let bas): // not ok + } +} +``` + +However, it was decided in review that this was still too restrictive and +source-breaking, and so the core team [accepted the proposal][Rationale] with the modification that pattern matches only had to match the case declaration in arity, and case labels could be either provided or elided in their entirety, unless there was an ambiguity. Even then, as of Swift 5.2, this part of the proposal has not been implemented, and it would be a source breaking change to do so. Therefore, the "Pattern Consistency" section of the original proposal has been removed, and replaced with a ["Disambiguating pattern matches" section](https://github.com/swiftlang/swift-evolution/blob/aecced4919ab297f343dafd7235d392d8b859839/proposals/0155-normalize-enum-case-representation.md), which provided a minimal disambiguation rule for pattern matching cases that share a +base name. This new design still had not been implemented at the time the [core team adopted a new expiration policy for unimplemented proposals](https://forums.swift.org/t/addressing-unimplemented-evolution-proposals/40322), so it has expired. + +[SE-0155]: 0155-normalize-enum-case-representation.md +[SE-0111]: 0111-remove-arg-label-type-significance.md +[Daniel Duan]: https://github.com/dduan +[Joe Groff]: https://github.com/jckarter +[John McCall]: https://github.com/rjmccall +[TJs comment]: https://forums.swift.org/t/draft-compound-names-for-enum-cases/4933/33 +[Revision 1]: https://github.com/swiftlang/swift-evolution/blob/43ca098355762014f53e1b54e02d2f6a01253385/proposals/0155-normalize-enum-case-representation.md +[Normalize Enum Case Representation (rev. 2)]: https://forums.swift.org/t/normalize-enum-case-representation-rev-2/5395 +[Originally Accepted Proposal]: https://github.com/swiftlang/swift-evolution/blob/4cbb1f1fa836496d4bfba95c4b78a9754690956d/proposals/0155-normalize-enum-case-representation.md +[Expired Proposal]: https://github.com/swiftlang/swift-evolution/blob/aecced4919ab297f343dafd7235d392d8b859839/proposals/0155-normalize-enum-case-representation.md +[Rationale]: https://forums.swift.org/t/accepted-se-0155-normalize-enum-case-representation/5732 diff --git a/proposals/0156-subclass-existentials.md b/proposals/0156-subclass-existentials.md new file mode 100644 index 0000000000..a3107aef79 --- /dev/null +++ b/proposals/0156-subclass-existentials.md @@ -0,0 +1,217 @@ +# Class and Subtype existentials + +* Proposal: [SE-0156](0156-subclass-existentials.md) +* Authors: [David Hart](https://github.com/hartbit), [Austin Zheng](https://github.com/austinzheng) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0156-class-and-subtype-existentials/5477) +* Bug: [SR-4296](https://bugs.swift.org/browse/SR-4296) + +## Introduction + +This proposal brings more expressive power to the type system by allowing Swift to represent existentials of classes and subtypes which conform to protocols. + +[Mailing list discussion](https://forums.swift.org/t/subclass-existentials/5024) + +## Motivation + +Currently, the only existentials which can be represented in Swift are conformances to a set of protocols, using the `&` protocol composition syntax: + +```swift +Protocol1 & Protocol2 +``` + +On the other hand, Objective-C is capable of expressing existentials of classes and subclasses conforming to protocols with the following syntax: + +```objc +id +Base* +``` + +We propose to provide similar expressive power to Swift, which will also improve the bridging of those types from Objective-C. + +## Proposed solution + +The proposal keeps the existing `&` syntax but allows one of the elements to be either `AnyObject` or of class type. The equivalent to the above Objective-C types would look like this: + +```swift +AnyObject & Protocol1 & Protocol2 +Base & Protocol +``` + +As in Objective-C, the first line is an existential of classes which conform to `Protocol1` and `Protocol2`, and the second line is an existential of subtypes of `Base` which conform to `Protocol`. + +Here are the new proposed rules for what is valid in a existential conjunction syntax: + +### 1. An element in the protocol composition syntax can be the `AnyObject` keyword to enforce a class constraint: + +```swift +protocol P {} +struct S : P {} +class C : P {} +class D { } +let t: AnyObject & P = S() // Compiler error: S is not of class type +let u: AnyObject & P = C() // Compiles successfully +let v: P & AnyObject = C() // Compiles successfully +let w: P & AnyObject = D() // Compiler error: class D does not conform to protocol P +``` + +### 2. An element in the protocol composition syntax can be a class type to enforce the existential to be a subtype of the class: + +```swift +protocol P {} +struct S {} +class C {} +class D : P {} +class E : C, P {} +let u: S & P // Compiler error: S is not of class type +let v: C & P = D() // Compiler error: D is not a subtype of C +let w: C & P = E() // Compiles successfully +``` + +### 3. If a protocol composition contains both a class type and `AnyObject`, the class type supersedes the `AnyObject` constraint: + +```swift +protocol P {} +class C {} +class D : C, P { } +let u: AnyObject & C & P = D() // Okay: D is a subclass of C and conforms to P +let v: C & P = u // Okay: C & P is equivalent to AnyObject & C & P +let w: AnyObject & C & P = v // Okay: AnyObject & C & P is equivalent to C & P +``` + +### 4. If a protocol composition contains two class types, either the class types must be the same or one must be a subclass of the other. In the latter case, the subclass type supersedes the superclass type: + +```swift +protocol P {} +class C {} +class D : C { } +class E : C { } +class F : D, P { } +let t: C & D & P = F() // Okay: F is a subclass of D and conforms to P +let u: D & P = t // Okay: D & P is equivalent to C & D & P +let v: C & D & P = u // Okay: C & D & P is equivalent to D & P +let w: D & E & P // Compiler error: D is not a subclass of E or vice-versa +``` + +### 5. When a protocol composition type contains one or more typealiases, the validity of the type is determined by expanding the typealiases into their component protocols, class types, and `AnyObject` constraints, then following the rules described above: + +```swift +class C {} +class D : C {} +class E {} +protocol P1 {} +protocol P2 {} +typealias TA1 = AnyObject & P1 +typealias TA2 = AnyObject & P2 +typealias TA3 = C & P2 +typealias TA4 = D & P2 +typealias TA5 = E & P2 + +typealias TA5 = TA1 & TA2 +// Expansion: typealias TA5 = AnyObject & P1 & AnyObject & P2 +// Normalization: typealias TA5 = AnyObject & P1 & P2 +// TA5 is valid + +typealias TA6 = TA1 & TA3 +// Expansion: typealias TA6 = AnyObject & P1 & C & P2 +// Normalization (AnyObject < C): typealias TA6 = C & P1 & P2 +// TA6 is valid + +typealias TA7 = TA3 & TA4 +// Expansion: typealias TA7 = C & P2 & D & P2 +// Normalization (C < D): typealias TA7 = D & P2 +// TA7 is valid + +typealias TA8 = TA4 & TA5 +// Expansion: typealias TA8 = D & P2 & E & P2 +// Normalization: typealias TA8 = D & E & P2 +// TA8 is invalid because the D and E constraints are incompatible +``` + +## `class` and `AnyObject` + +This proposal merges the concepts of `class` and `AnyObject`, which now have the same meaning: they represent an existential for classes. To get rid of the duplication, we suggest only keeping `AnyObject` around. To reduce source-breakage to a minimum, `class` could be redefined as `typealias class = AnyObject` and give a deprecation warning on `class` for the first version of Swift this proposal is implemented in. Later, `class` could be removed in a subsequent version of Swift. + +## Inheritance clauses and `typealias` + +To improve readability and reduce confusion, a class conforming to a typealias which contains a class type constraint does not implicitly inherit the class type: inheritance should stay explicit. Here are a few examples to remind what the current rules are and to make the previous sentence clearer: + +The proposal does not change the rule which forbids using the protocol composition syntax in the inheritance clause: + +```swift +protocol P1 {} +protocol P2 {} +class C {} + +class D : P1 & P2 {} // Compiler error +class E : C & P1 {} // Compiler error +``` + +Class `D` in the previous example does not inherit a base class so it can be expressed using the inheritance/conformance syntax or through a typealias: + +```swift +class D : P1, P2 {} // Valid +typealias P12 = P1 & P2 +class D : P12 {} // Valid +``` + +Class `E` above inherits a base class. The inheritance must be explicitly declared in the inheritance clause and can't be implicitly derived from a typealias: + +```swift +class E : C, P1 {} // Valid +typealias CP1 = C & P1 +class E : CP1 {} // Compiler error: class 'E' does not inherit from class 'C' +class E : C, CP1 {} // Valid: the inheritance is explicitly declared +``` + +## Source compatibility + +This change will not break Swift 3 compatibility mode because Objective-C types will continue to be imported as before. But in Swift 4 mode, all types bridged from Objective-C which use the equivalent Objective-C existential syntax could break code which does not meet the new protocol requirements. For example, the following Objective-C code: + +```objc +@interface MyViewController +- (void)setup:(nonnull UIViewController*)tableViewController; +@end +``` + +is imported into Swift-3 mode as: + +```swift +class MyViewController { + func setup(tableViewController: UIViewController) {} +} +``` + +which allows calling the function with an invalid parameter: + +```swift +let myViewController = MyViewController() +myViewController.setup(UIViewController()) +``` + +The previous code continues to compile but still crashes if the Objective-C code calls a method of `UITableViewDataSource` or `UITableViewDelegate`. But if this proposal is accepted and implemented as-is, the Objective-C code will be imported in Swift 4 mode as: + +```swift +class MyViewController { + func setup(tableViewController: UIViewController & UITableViewDataSource & UITableViewDelegate) {} +} +``` + +That would then cause the Swift code run in version 4 mode to fail to compile with an error which states that `UIViewController` does not conform to the `UITableViewDataSource` and `UITableViewDelegate` protocols. + +## Alternatives considered + +An alternative solution to the `class`/`AnyObject` duplication was to keep both, redefine `AnyObject` as `typealias AnyObject = class` and favor the latter when used as a type name. + +The [reviewed version of the +proposal](https://github.com/swiftlang/swift-evolution/blob/78da25ec4acdc49ad9b68fb58300e49c33bc6355/proposals/0156-subclass-existentials.md) +included rules that required the class type (or `AnyObject`) to be +first within the protocol composition, e.g., `AnyObject & Protocol1` +was well-formed but `Protocol1 & AnyObject` would produce a compiler +error. When accepting this proposal, the core team removed these +rules; see the decision notes at the top for more information. + +## Acknowledgements + +Thanks to [Austin Zheng](https://github.com/austinzheng) and [Matthew Johnson](https://github.com/anandabits) who brought a lot of attention to existentials in this mailing-list and from whom most of the ideas in the proposal come from. diff --git a/proposals/0157-recursive-protocol-constraints.md b/proposals/0157-recursive-protocol-constraints.md new file mode 100644 index 0000000000..81c1328494 --- /dev/null +++ b/proposals/0157-recursive-protocol-constraints.md @@ -0,0 +1,195 @@ +# Support recursive constraints on associated types + +* Proposal: [SE-0157](0157-recursive-protocol-constraints.md) +* Authors: [Douglas Gregor](https://github.com/DougGregor), [Erica Sadun](https://github.com/erica), [Austin Zheng](https://github.com/austinzheng) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 4.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0157-support-recursive-constraints-on-associated-types/5494) +* Bug: [SR-1445](https://bugs.swift.org/browse/SR-1445) + +## Introduction + +This proposal lifts restrictions on associated types in protocols. Their constraints will be allowed to reference any +protocol, including protocols that depend on the enclosing one (recursive constraints). + +Further reading: [swift-evolution thread](https://forums.swift.org/t/pitch-plea-recursive-protocol-constraints/4507), _[Completing Generics](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#recursive-protocol-constraints-)_ + +## Motivation + +Swift supports defining _associated types_ on protocols using the `associatedtype` keyword. + +```swift +protocol Sequence { + associatedtype Subsequence +} +``` + +Swift also supports defining _constraints_ on those associated types, for example: + +```swift +protocol Foo { + // For all types X conforming to Foo, X.SomeType must conform to Bar + associatedtype SomeType: Bar +} +``` + +However, Swift does not currently support defining _constraints on an associated type that recursively reference the +enclosing protocol_. It would make sense for `SubSequence` to be constrained to be a `Sequence`, as all subsequences +are themselves sequences: + +```swift +// Will not currently compile +protocol Sequence { + associatedtype SubSequence: Sequence + where Iterator.Element == SubSequence.Iterator.Element, SubSequence.SubSequence == SubSequence + + // Returns a subsequence containing all but the first 'n' items + // in the original sequence. + func dropFirst(_ n: Int) -> Self.SubSequence + // ... +} +``` + +However, Swift currently doesn't support expressing this constraint at the point where `SubSequence` is declared. +Instead, we must specify it in documentation and/or at each site of use. This results in more verbose code and obscures +intent: + +```swift +protocol Sequence { + // SubSequences themselves must be Sequences. + // The element type of the subsequence must be identical to the element type of the sequence. + // The subsequence's subsequence type must be itself. + associatedtype SubSequence + + func dropFirst(_ n: Int) -> Self.SubSequence + // ... +} + +struct SequenceOfInts : Sequence { + // This concrete implementation of `Sequence` happens to work correctly. + // Implicitly: + // The subsequence conforms to Sequence. + // The subsequence's element type is the same as the parent sequence's element type. + // The subsequence's subsequence type is the same as itself. + func dropFirst(_ n: Int) -> SimpleSubSequence { + // ... + } +} + +struct SimpleSubSequence : Sequence { + typealias SubSequence = SimpleSubSequence + typealias Iterator.Element = Element + // ... +} +``` + +## Proposed solution + +The first part of the solution we propose is to lift this restriction. From the perspective of the end user, this is a +relatively simple change. It is only a new feature in the sense that certain associated type definitions which were +previously disallowed will now be accepted by the compiler. + +Implementation details regarding the compiler changes necessary to implement the first part of the solution can be +found in [this document](https://gist.github.com/DougGregor/e7c4e7bb4465d6f5fa2b59be72dbdba6). + +The second part of the solution involves updating the standard library to take advantage of the removal of this +restriction. Such changes are made with [SE-0142](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md) +in mind, and incorporate both recursive constraints and `where` clauses. The changes necessary for this are described +in the _Detailed Design_ section below. + +This second change will affect the sort of user code which is accepted by the compiler. User code which uses the +affected protocols and types will require fewer generic parameter constraints to be considered valid. Conversely, +user code which (incorrectly) uses the private protocols removed by this proposal, or which uses the affected public +protocols in an incorrect manner, might cease to be accepted by the compiler after this change is implemented. + +## Detailed design + +The following standard library protocols and types will change in order to support recursive protocol constraints. + +Note that since the specific collection types conform to `Collection`, and `Collection` refines `Sequence`, not all the +constraints need to be defined on every collection-related associated type. + +Default values of all changed associated types remain the same, unless explicitly noted otherwise. + +All "Change associated type" entries reflect the complete, final state of the associated type definition, including +removal of underscored protocols and addition of any new constraints. + +### `Arithmetic` + +* Change associated type: `associatedtype Magnitude : Arithmetic` + +### `BidirectionalCollection` + +* Remove conformance to `_BidirectionalIndexable` +* Change associated type: `associatedtype SubSequence : BidirectionalCollection` +* Change associated type: `associatedtype Indices : BidirectionalCollection` + +### `Collection` + +* Remove conformance to `_Indexable` +* Change associated type: `associatedtype SubSequence : Collection where SubSequence.Index == Index` +* Change associated type: `associatedtype Indices : Collection where Indices.Iterator.Element == Index, Indices.Index == Index` + +### `Default*Indices` (all variants) + +* Declarations changed to `public struct Default*Indices : *Collection` + +### `IndexingIterator` + +* Declaration changed to `public struct IndexingIterator : IteratorProtocol, Sequence` + +### `LazyFilter*Collection` (all variants) + +* Add default associated type conformance: `typealias SubSequence = ${Self}` + +### `LazyMap*Collection` (all variants) + +* Add default associated type conformance: `typealias SubSequence = ${Self}` + +### `MutableCollection` + +* Change associated type: `associatedtype SubSequence : MutableCollection` + +### `RandomAccessCollection` + +* Change associated type: `associatedtype SubSequence : RandomAccessCollection` +* Change associated type: `associatedtype Indices : RandomAccessCollection` + +### `RangeReplaceableCollection` + +* Change associated type: `associatedtype SubSequence : RangeReplaceableCollection` + +### `Sequence` + +* Change associated type: `associatedtype SubSequence : Sequence where Iterator.Element == SubSequence.Iterator.Element, SubSequence.SubSequence == SubSequence` + +### `*Slice` (all variants) + +* Add default associated type conformance: `typealias Indices = Base.Indices` + +## Source compatibility + +From a source compatibility perspective, this is a purely additive change if the user's code is correctly written. It +is possible that users may have written code which defines semantically incorrect associated types, which the compiler +now rejects because of the additional constraints. We do not consider this scenario "source-breaking". + +An example of code that currently compiles but is semantically invalid is an implementation of a range-replacable +collection's subsequence that isn't itself range-replaceable. This is a constraint that cannot be enforced by the compiler +without this change. For some time, the `Data` type in Foundation violated this constraint; user-written code that is +similarly problematic will cease to compile using a Swift toolchain that includes these standard library and compiler +changes. + +## Impact on ABI stability + +Since this proposal involves modifying the standard library, it changes the ABI. In particular, ABI changes enabled by +this proposal are critical to getting the standard library to a state where it more closely resembles the design +envisioned by its engineers. + +## Impact on API resilience + +This feature cannot be removed without breaking API compatibility, but since it forms a necessary step in crystallizing +the standard library for future releases, it is very unlikely that it will be removed after being accepted. + +## Alternatives considered + +n/a diff --git a/proposals/0158-package-manager-manifest-api-redesign.md b/proposals/0158-package-manager-manifest-api-redesign.md new file mode 100644 index 0000000000..adbe933201 --- /dev/null +++ b/proposals/0158-package-manager-manifest-api-redesign.md @@ -0,0 +1,696 @@ +# Package Manager Manifest API Redesign + +* Proposal: [SE-0158](0158-package-manager-manifest-api-redesign.md) +* Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Rick Ballard](https://github.com/rballard) +* Status: **Implemented (Swift 4.0)** +* Bug: [SR-3949](https://bugs.swift.org/browse/SR-3949) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0158-package-manager-manifest-api-redesign/5468) + +## Introduction + +This is a proposal for redesigning the `Package.swift` manifest APIs provided +by Swift Package Manager. +This proposal only redesigns the existing public APIs and does not add any +new functionality; any API to be added for new functionality will happen in +separate proposals. + +## Motivation + +The `Package.swift` manifest APIs were designed prior to the [API Design +Guidelines](https://swift.org/documentation/api-design-guidelines/), and their +design was not reviewed by the evolution process. Additionally, there are +several small areas which can be cleaned up to make the overall API more +"Swifty". + +We would like to redesign these APIs as necessary to provide clean, +conventions-compliant APIs that we can rely on in the future. Because we +anticipate that the user community for the Swift Package Manager will grow +considerably in Swift 4, we would like to make these changes now, before +more packages are created using the old API. + +## Proposed solution + +Note: Access modifier is omitted from the diffs and examples for brevity. The +access modifier is `public` for all APIs unless specified. + +* Remove `successor()` and `predecessor()` from `Version`. + + These methods neither have well defined semantics nor are used a lot + (internally or publicly). For e.g., the current implementation of + `successor()` always just increases the patch version. + + +
+ View diff +

+ + ```diff + struct Version { + - func successor() -> Version + + - func predecessor() -> Version + } + ``` +

+ +* Convert `Version`'s `buildMetadataIdentifier` property to an array. + + According to SemVer 2.0, build metadata is a series of dot separated + identifiers. Currently this is represented as an optional string property + in the `Version` struct. We propose to change this property to an array + (similar to `prereleaseIdentifiers` property). To maintain backwards + compatiblility in PackageDescription 3 API, we will keep the optional + string as a computed property based on the new array property. We will also + keep the version initializer that takes the `buildMetadataIdentifier` string. + +* Make all properties of `Package` and `Target` mutable. + + Currently, `Package` has three immutable and four mutable properties, and + `Target` has one immutable and one mutable property. We propose to make all + properties mutable to allow complex customization on the package object + after initial declaration. + +
+ View diff and example +

+ + Diff: + ```diff + final class Target { + - let name: String + + var name: String + } + + final class Package { + - let name: String + + var name: String + + - let pkgConfig: String? + + var pkgConfig: String? + + - let providers: [SystemPackageProvider]? + + var providers: [SystemPackageProvider]? + } + ``` + + Example: + ```swift + let package = Package( + name: "FooPackage", + targets: [ + Target(name: "Foo", dependencies: ["Bar"]), + ] + ) + + #if os(Linux) + package.targets[0].dependencies = ["BarLinux"] + #endif + ``` +

+ +* Change `Target.Dependency` enum cases to lowerCamelCase. + + According to API design guidelines, everything other than types should be in lowerCamelCase. + +
+ View diff and example +

+ + Diff: + ```diff + enum Dependency { + - case Target(name: String) + + case target(name: String) + + - case Product(name: String, package: String?) + + case product(name: String, package: String?) + + - case ByName(name: String) + + case byName(name: String) + } + ``` + + Example: + ```diff + let package = Package( + name: "FooPackage", + targets: [ + Target( + name: "Foo", + dependencies: [ + - .Target(name: "Bar"), + + .target(name: "Bar"), + + - .Product(name: "SwiftyJSON", package: "SwiftyJSON"), + + .product(name: "SwiftyJSON", package: "SwiftyJSON"), + ] + ), + ] + ) + ``` +

+ +* Add default parameter to the enum case `Target.Dependency.product`. + + The associated value `package` in the (enum) case `product`, is an optional + `String`. It should have the default value `nil` so clients don't need to + write it if they prefer using explicit enum cases but don't want to specify + the package name i.e. it should be possible to write `.product(name: + "Foo")` instead of `.product(name: "Foo", package: nil)`. + + If + [SE-0155](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0155-normalize-enum-case-representation.md) + is accepted, we can directly add a default value. Otherwise, we will use a + static factory method to provide default value for `package`. + +* Rename all enums cases to have a suffix `Item` and favor static methods. + + Since static methods are more extensible than enum cases right now, we + should discourage use of direct use of enum initializers and provide a + static method for each case. It is not possible to have overloads on enum + cases, so as a convention we propose to rename all enum cases to have a + suffix "Item" for future extensibility. + +* Change `SystemPackageProvider` enum cases to lowerCamelCase and their payloads to array. + + According to API design guidelines, everything other than types should be + in lowerCamelCase. + + This enum allows SwiftPM System Packages to emit hints in case of build + failures due to absence of a system package. Currently, only one system + package per system packager can be specified. We propose to allow + specifying multiple system packages by changing the payload to be an array. + +
+ View diff and example +

+ + Diff: + ```diff + enum SystemPackageProvider { + - case Brew(String) + + case brew([String]) + + - case Apt(String) + + case apt([String]) + } + ``` + + Example: + + ```diff + let package = Package( + name: "Copenssl", + pkgConfig: "openssl", + providers: [ + - .Brew("openssl"), + + .brew(["openssl"]), + + - .Apt("openssl-dev"), + + .apt(["openssl", "libssl-dev"]), + ] + ) + ``` +

+ + +* Remove implicit target dependency rule for test targets. + + There is an implicit test target dependency rule: a test target "FooTests" + implicitly depends on a target "Foo", if "Foo" exists and "FooTests" doesn't + explicitly declare any dependency. We propose to remove this rule because: + + 1. It is a non obvious "magic" rule that has to be learned. + 2. It is not possible for "FooTests" to remove dependency on "Foo" while + having no other (target) dependency. + 3. It makes real dependencies less discoverable. + 4. It may cause issues when we get support for mechanically editing target + dependencies. + +* Use factory methods for creating objects. + + We propose to always use factory methods to create objects except for the + main `Package` object. This gives a lot of flexibility and extensibility to + the APIs because Swift's type system can infer the top level type in a + context and allow using the shorthand dot syntax. + + Concretely, we will make these changes: + + * Add a factory method `target` to `Target` class and change the current + initializer to private. + +
+ View example and diff +

+ + Example: + + ```diff + let package = Package( + name: "Foo", + target: [ + - Target(name: "Foo", dependencies: ["Utility"]), + + .target(name: "Foo", dependencies: ["Utility"]), + ] + ) + ``` +

+ + * Introduce a `Product` class with two subclasses: `Executable` and + `Library`. These subclasses will be nested inside `Product` class + instead of being a top level declaration in the module. Nesting will give + us a namespace for products and it is easy to find all the supported + products when the product types grows to a large number. We will add two + factory methods to `Product` class: `library` and `executable` to create + respective products. + + ```swift + /// Represents a product. + class Product { + + /// The name of the product. + let name: String + + private init(name: String) { + self.name = name + } + + /// Represents an executable product. + final class Executable: Product { + + /// The names of the targets in this product. + let targets: [String] + + private init(name: String, targets: [String]) + } + + /// Represents a library product. + final class Library: Product { + /// The type of library product. + enum LibraryType: String { + case `static` + case `dynamic` + } + + /// The names of the targets in this product. + let targets: [String] + + /// The type of the library. + /// + /// If the type is unspecified, package manager will automatically choose a type. + let type: LibraryType? + + private init(name: String, type: LibraryType? = nil, targets: [String]) + } + + /// Create a library product. + static func library(name: String, type: LibraryType? = nil, targets: [String]) -> Library + + /// Create an executable product. + static func executable(name: String, targets: [String]) -> Library + } + ``` + +
+ View example +

+ + Example: + + ```swift + let package = Package( + name: "Foo", + target: [ + .target(name: "Foo", dependencies: ["Utility"]), + .target(name: "tool", dependencies: ["Foo"]), + ], + products: [ + .executable(name: "tool", targets: ["tool"]), + .library(name: "Foo", targets: ["Foo"]), + .library(name: "FooDy", type: .dynamic, targets: ["Foo"]), + ] + ) + ``` +

+ + +* Special syntax for version initializers. + + A simplified summary of what is commonly supported in other package managers: + + | Package Manager | x-ranges | tilde (`~` or `~>`) | caret (`^`) | + |-----------------|---------------|-------------------------|---------------| + | npm | Supported | Allows patch-level changes if a minor version is specified on the comparator. Allows minor-level changes if not. | patch and minor updates | + | Cargo | Supported | Same as above | Same as above | + | CocoaPods | Not supported | Same as above | Not supported | + | Carthage | Not supported | patch and minor updates | Not supported | + + Some general notes: + + Every package manager we looked at supports the tilde `~` operator in some form, and it's generally + recommended as "the right thing", because package maintainers often fail to increment their major package + version when they should, incrementing their minor version instead. See e.g. how Google created a + [6-minute instructional video](https://www.youtube.com/watch?v=x4ARXyovvPc) about this operator for CocoaPods. + This is a form of version overconstraint; your package should be compatible with everything with the same + major version, but people don't trust that enough to rely on it. But version overconstraint is harmful, + because it leads to "dependency hell" (unresolvable dependencies due to conflicting requirements for a package + in the dependency graph). + + We'd like to encourage a better standard of behavior in the Swift Package Manager. In the future, we'd like + to add tooling to let the package manager automatically help you use Semantic Versioning correctly, + so that your clients can trust your major version. If we can get package maintainers to use SemVer correctly, + through automatic enforcement in the future or community norms for now, then caret `^` becomes + the best operator to use most of the time. That is, you should be able to specify a minimum version, + and you should be willing to let your package use anything after that up to the next major version. This + means you'll get safe updates automatically, and you'll avoid overconstraining and introducing dependency hell. + + Caret `^` and tilde `~` syntax is somewhat standard, but is syntactically non-obvious; we'd prefer a syntax + that doesn't require reading a manual for novices to understand, even if that means we break with the + syntactic convention established by the other package managers which support caret `^` and tilde `~`. + We'd like to make it possible to follow the tilde `~` use case (with different syntax), but caret `^` + should be the most convenient, to encourage its use. + + What we propose: + + * We will introduce a factory method which takes a lower bound version and + forms a range that goes upto the next major version (i.e. caret). + + ```swift + // 1.0.0 ..< 2.0.0 + .package(url: "/SwiftyJSON", from: "1.0.0"), + + // 1.2.0 ..< 2.0.0 + .package(url: "/SwiftyJSON", from: "1.2.0"), + + // 1.5.8 ..< 2.0.0 + .package(url: "/SwiftyJSON", from: "1.5.8"), + ``` + + * We will introduce a factory method which takes `Requirement`, to + conveniently specify common ranges. + + `Requirement` is an enum defined as follows: + + ```swift + enum Requirement { + /// The requirement is specified by an exact version. + case exact(Version) + + /// The requirement is specified by a version range. + case range(Range) + + /// The requirement is specified by a source control revision. + case revision(String) + + /// The requirement is specified by a source control branch. + case branch(String) + + /// Creates a specified for a range starting at the given lower bound + /// and going upto next major version. + static func upToNextMajor(from version: Version) -> Requirement + + /// Creates a specified for a range starting at the given lower bound + /// and going upto next minor version. + static func upToNextMinor(from version: Version) -> Requirement + } + ``` + + Examples: + + ```swift + // 1.5.8 ..< 2.0.0 + .package(url: "/SwiftyJSON", .upToNextMajor(from: "1.5.8")), + + // 1.5.8 ..< 1.6.0 + .package(url: "/SwiftyJSON", .upToNextMinor(from: "1.5.8")), + + // 1.5.8 + .package(url: "/SwiftyJSON", .exact("1.5.8")), + ``` + + * This will also give us ability to add more complex features in future: + + Examples: + > Note that we're not actually proposing these as part of this proposal. + + ```swift + .package(url: "/SwiftyJSON", .upToNextMajor(from: "1.5.8").excluding("1.6.4")), + + .package(url: "/SwiftyJSON", .exact("1.5.8", "1.6.3")), + ``` + + * We will introduce a factory method which takes `Range`, to specify + arbitrary open range. + + ```swift + // Constraint to an arbitrary open range. + .package(url: "/SwiftyJSON", "1.2.3"..<"1.2.6"), + ``` + + * We will introduce a factory method which takes `ClosedRange`, to specify + arbitrary closed range. + + ```swift + // Constraint to an arbitrary closed range. + .package(url: "/SwiftyJSON", "1.2.3"..."1.2.8"), + ``` + * As a slight modification to the + [branch proposal](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0150-package-manager-branch-support.md), + we will add cases for specifying a branch or revision, rather than + adding factory methods for them: + + ```swift + .package(url: "/SwiftyJSON", .branch("develop")), + .package(url: "/SwiftyJSON", .revision("e74b07278b926c9ec6f9643455ea00d1ce04a021")) + ``` + + * We will remove all of the current factory methods: + + ```swift + // Constraint to a major version. + .Package(url: "/SwiftyJSON", majorVersion: 1), + + // Constraint to a major and minor version. + .Package(url: "/SwiftyJSON", majorVersion: 1, minor: 2), + + // Constraint to an exact version. + .Package(url: "/SwiftyJSON", "1.2.3"), + + // Constraint to an arbitrary range. + .Package(url: "/SwiftyJSON", versions: "1.2.3"..<"1.2.6"), + + // Constraint to an arbitrary closed range. + .Package(url: "/SwiftyJSON", versions: "1.2.3"..."1.2.8"), + ``` + +* Adjust order of parameters on `Package` class: + + We propose to reorder the parameters of `Package` class to: `name`, + `pkgConfig`, `products`, `dependencies`, `targets`, `swiftLanguageVersions`. + + The rationale behind this reorder is that the most interesting parts of a + package are its product and dependencies, so they should be at the top. + Targets are usually important during development of the package. Placing + them at the end keeps it easier for the developer to jump to end of the + file to access them. Note that the `swiftLanguageVersions` property will likely + be removed once we support Build Settings, but that will be discussed in a separate proposal. + + +
+ View example +

+ + Example: + + ```swift + let package = Package( + name: "Paper", + products: [ + .executable(name: "tool", targets: ["tool"]), + .library(name: "Paper", type: .static, targets: ["Paper"]), + .library(name: "PaperDy", type: .dynamic, targets: ["Paper"]), + ], + dependencies: [ + .package(url: "http://github.com/SwiftyJSON/SwiftyJSON", from: "1.2.3"), + .package(url: "../CHTTPParser", .upToNextMinor(from: "2.2.0")), + .package(url: "http://some/other/lib", .exact("1.2.3")), + ] + targets: [ + .target( + name: "tool", + dependencies: [ + "Paper", + "SwiftyJSON" + ]), + .target( + name: "Paper", + dependencies: [ + "Basic", + .target(name: "Utility"), + .product(name: "CHTTPParser"), + ]) + ] + ) + ``` +

+ +* Eliminate exclude in future (via custom layouts feature). + + We expect to remove the `exclude` property after we get support for custom + layouts. The exact details will be in the proposal of that feature. + +## Example manifests + +* A regular manifest. + +```swift +let package = Package( + name: "Paper", + products: [ + .executable(name: "tool", targets: ["tool"]), + .library(name: "Paper", targets: ["Paper"]), + .library(name: "PaperStatic", type: .static, targets: ["Paper"]), + .library(name: "PaperDynamic", type: .dynamic, targets: ["Paper"]), + ], + dependencies: [ + .package(url: "http://github.com/SwiftyJSON/SwiftyJSON", from: "1.2.3"), + .package(url: "../CHTTPParser", .upToNextMinor(from: "2.2.0")), + .package(url: "http://some/other/lib", .exact("1.2.3")), + ] + targets: [ + .target( + name: "tool", + dependencies: [ + "Paper", + "SwiftyJSON" + ]), + .target( + name: "Paper", + dependencies: [ + "Basic", + .target(name: "Utility"), + .product(name: "CHTTPParser"), + ]) + ] +) +``` + +* A system package manifest. + +```swift +let package = Package( + name: "Copenssl", + pkgConfig: "openssl", + providers: [ + .brew(["openssl"]), + .apt(["openssl", "libssl-dev"]), + ] +) +``` + +## Impact on existing code + +The above changes will be implemented only in the new Package Description v4 +library. The v4 runtime library will release with Swift 4 and packages will be +able to opt-in into it as described by +[SE-0152](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md). + +There will be no automatic migration feature for updating the manifests from v3 +to v4. To indicate the replacements of old APIs, we will annotate them using +the `@unavailable` attribute where possible. Unfortunately, this will not cover +all the changes for e.g. rename of the target dependency enum cases. + +All new packages created with `swift package init` command in Swift 4 tools +will by default to use the v4 manifest. It will be possible to switch to v3 +manifest version by changing the tools version using `swift package +tools-version --set 3.1`. However, the manifest will needed to be adjusted to +use the older APIs manually. + +Unless declared in the manifest, existing packages automatically default +to the Swift 3 minimum tools version; since the Swift 4 tools will also include +the v3 manifest API, they will build as expected. + +A package which needs to support both Swift 3 and Swift 4 tools will need to +stay on the v3 manifest API and support the Swift 3 language version for its +sources, using the API described in the proposal +[SE-0151](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0151-package-manager-swift-language-compatibility-version.md). + +An existing package which wants to use the new v4 manifest APIs will need to bump its +minimum tools version to 4.0 or later using the command `$ swift package tools-version +--set-current`, and then modify the manifest file with the changes described in +this proposal. + +## Alternatives considered + +* Add variadic overloads. + + Adding variadic overload allows omitting parenthesis which leads to less + cognitive load on eyes, especially when there is only one value which needs + to be specified. For e.g.: + + Target(name: "Foo", dependencies: "Bar") + + might looked better than: + + Target(name: "Foo", dependencies: ["Bar"]) + + However, plurals words like `dependencies` and `targets` imply a collection + which implies brackets. It also makes the grammar wrong. Therefore, we + reject this option. + +* Version exclusion. + + It is not uncommon to have a specific package version break something, and + it is undesirable to "fix" this by adjusting the range to exclude it + because this overly constrains the graph and can prevent picking up the + version with the fix. + + This is desirable but it should be proposed separately. + +* Inline package declaration. + + We should probably support declaring a package dependency anywhere we + support spelling a package name. It is very common to only have one target + require a dependency, and annoying to have to specify the name twice. + + This is desirable but it should be proposed separately. + +* Introduce an "identity rule" to determine if an API should use an initializer + or a factory method: + + Under this rule, an entity having an identity, will use a type initializer + and everything else will use factory methods. `Package`, `Target` and + `Product` are identities. However, a product referenced in a target + dependency is not an identity. + + We rejected this because it may become a source of confusion for users. + Another downside is that the product initializers will have to used with + the dot notation (e.g.: `.Executable(name: "tool", targets: ["tool"])`) + which is a little awkward because we expect factory methods and enum cases + to use the dot syntax. This can be solved by moving these products outside + of `Product` class but we think having a namespace for product provides a + lot of value. + +* Upgrade `SystemPackageProvider` enum to a struct. + + We thought about upgrading `SystemPackageProvider` to a struct when we had + the "identity" rule but since we're dropping that, there is no need for + this change. + + ```swift + public struct SystemPackageProvider { + enum PackageManager { + case apt + case brew + } + + /// The system package manager. + let packageManager: PackageManager + + /// The array of system packages. + let packages: [String] + + init(_ packageManager: PackageManager, packages: [String]) + } + ``` diff --git a/proposals/0159-fix-private-access-levels.md b/proposals/0159-fix-private-access-levels.md new file mode 100644 index 0000000000..45660527a8 --- /dev/null +++ b/proposals/0159-fix-private-access-levels.md @@ -0,0 +1,51 @@ +# Fix Private Access Levels + +* Proposal: [SE-0159](0159-fix-private-access-levels.md) +* Author: [David Hart](https://github.com/hartbit) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Rejected** +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0159-fix-private-access-levels/5576) + +## Introduction + +This proposal presents the problems that came with the the access level modifications in [SE-0025](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0025-scoped-access-level.md) and proposes reverting to Swift 2 behaviour. + +## Motivation + +Since the release of Swift 3, the access level change of SE-0025 was met with dissatisfaction by a substantial proportion of the general Swift community. Those changes can be viewed as *actively harmful*, the new requirement for syntax/API changes. + +The `private` keyword is a "soft default" access modifier for restricting access within a file. Scoped access is not a good behavior for a "soft default" because it is extremely common to use several extensions within a file. A "soft default" (and therefore `private`) should work well with this idiom. It is fair to say that changing the behavior of `private` such that it does not work well with extensions meets the criteria of actively harmful in the sense that it subtly encourages overuse of scoped access control and discourages the more reasonable default by giving it the awkward name `fileprivate`. + +Compared to a file-based access level, the scoped-based access level adds meaningful information by hiding implementation details which do not concern other types or extensions in the same file. But is that distinction between `private` and `fileprivate` actively used by the larger community of Swift developers? And if it were used pervasively, would it be worth the cognitive load and complexity of keeping two very similar access levels in the language? This proposal argues that answer to both questions is no and therefore wish to simplify Swift's access control story by removing scoped access and leaving more design breathing space for future discussions around submodules. + +## Detailed design + +The `private` keyword should be reverted back to its Swift 2 file-based meaning and the `fileprivate` keyword should be deprecated. + +## Source compatibility + +In Swift 3 compatibility mode, the compiler will continue to treat `private` and `fileprivate` as was previously the case. + +In Swift 4 mode, the compiler will deprecate the `fileprivate` keyword and revert the semantics of the `private` access level to be file based. The migrator will rename all uses of `fileprivate` to `private`. + +Cases where a type had `private` declarations with the same signature in different scopes will produce a compiler error. For example, the following piece of code compiles in Swift 3 compatibility mode but generates a `Invalid redeclaration of 'bar()'` error in Swift 4 mode. + +```swift +struct Foo { + private func bar() {} +} + +extension Foo { + private func bar() {} +} +``` + +## Alternatives Considered + +1. Deprecate `fileprivate` and modify the semantics of `private` to include same-type extension scopes in the same file. +2. Deprecate `fileprivate` and modify the semantics of `private` to include same-type extension scopes in the same module. +3. Revert `private` to be file-based and introduce the scope-based access level under a new name. + +## Thanks + +I'd like to extend my thanks to Xiaodi Wu and Matthew Johnson for their respective contributions. diff --git a/proposals/0160-objc-inference.md b/proposals/0160-objc-inference.md new file mode 100644 index 0000000000..8c595e5318 --- /dev/null +++ b/proposals/0160-objc-inference.md @@ -0,0 +1,611 @@ +# Limiting `@objc` inference + +* Proposal: [SE-0160](0160-objc-inference.md) +* Author: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0160-limiting-objc-inference/5621) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/0389b1f49fc55b1a898701c549ce89738307b9fc/proposals/0160-objc-inference.md) +* Implementation: [apple/swift#8379](https://github.com/apple/swift/pull/8379) +* Bug: [SR-4481](https://bugs.swift.org/browse/SR-4481) + +## Introduction + +One can explicitly write `@objc` on any Swift declaration that can be +expressed in Objective-C. As a convenience, Swift also *infers* +`@objc` in a number of places to improve interoperability with +Objective-C and eliminate boilerplate. This proposal scales back the +inference of `@objc` to only those cases where the declaration *must* +be available to Objective-C to maintain semantic coherence of the model, +e.g., when overriding an `@objc` method or implementing a requirement +of an `@objc` protocol. Other cases currently supported (e.g., a +method declared in a subclass of `NSObject`) would no longer infer +`@objc`, but one could continue to write it explicitly to produce +Objective-C entry points. + +Swift-evolution thread: [here](https://forums.swift.org/t/pitch-align-objc-inference-with-the-semantic-model/2563) and [here](https://forums.swift.org/t/proposal-draft-limiting-objc-inference/4812) + +## Motivation + +There are several observations motivating this proposal. The first is +that Swift's rules for inference of `@objc` are fairly baroque, and it +is often unclear to users when `@objc` will be inferred. This proposal +seeks to make the inference rules more straightforward. The second +observation is that it is fairly easy to write Swift classes that +inadvertently cause Objective-C selector collisions due to +overloading, e.g., + +```swift +class MyNumber : NSObject { + init(_ int: Int) { } + init(_ double: Double) { } // error: initializer 'init' with Objective-C selector 'init:' + // conflicts with previous declaration with the same Objective-C selector +} +``` + +The example above also illustrates the third observation, which is +that code following the [Swift API Design +Guidelines](https://swift.org/documentation/api-design-guidelines/) +will use Swift names that often translate into very poor Objective-C +names that violate the [Objective-C Coding Guidelines for +Cocoa](https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/CodingGuidelines/CodingGuidelines.html). Specifically, +the Objective-C selectors for the initializers above should include a noun +describing the first argument, e.g., `initWithInteger:` and +`initWithDouble:`, which requires explicit `@objc` annotations anyway: + +```swift +class MyNumber : NSObject { + @objc(initWithInteger:) init(_ int: Int) { } + @objc(initWithDouble:) init(_ double: Double) { } +} +``` + +The final observation is that there is a cost for each Objective-C +entry point, because the Swift compiler must create a "thunk" method +that maps from the Objective-C calling convention to the Swift calling +convention and is recorded within Objective-C metadata. This increases +the size of the binary (preliminary tests on some Cocoa[Touch] apps +found that 6-8% of binary size was in these thunks alone, some of +which are undoubtedly unused), and can have some impact on load time +(the dynamic linker has to sort through the Objective-C metadata for +these thunks). + +## Proposed solution + +The proposed solution is to limit the inference of `@objc` to only +those places where it is required for semantic consistency of the +programming model. Then, add some class-level and extension-level +annotations to reduce boilerplate for cases where one wants to +enable/disable `@objc` inference more widely. + +### Constructs that (still) infer `@objc` + +Specifically, `@objc` will continue to be inferred +for a declaration when: + +* The declaration is an override of an `@objc` declaration, e.g., + + ```swift + class Super { + @objc func foo() { } + } + + class Sub : Super { + /* inferred @objc */ + override func foo() { } + } + ``` + + This inference is required so that Objective-C callers to the method + `Super.foo()` will appropriately invoke the overriding method + `Sub.foo()`. + +* The declaration satisfies a requirement of an `@objc` protocol, + e.g., + + ```swift + @objc protocol MyDelegate { + func bar() + } + + class MyClass : MyDelegate { + /* inferred @objc */ + func bar() { } + } + ``` + + This inference is required because anyone calling + `MyDelegate.bar()`, whether from Objective-C or Swift, will do so + via an Objective-C message send, so conforming to the protocol + requires an Objective-C entry point. + +* The declaration has the `@IBAction` or `@IBOutlet` attribute. This + inference is required because the interaction with Interface Builder + occurs entirely through the Objective-C runtime, and therefore + depends on the existence of an Objective-C entrypoint. + +* The declaration has the `@NSManaged` attribute. This inference is + required because the interaction with CoreData occurs entirely + through the Objective-C runtime, and therefore depends on the + existence of an Objective-C entrypoint. + +The list above describes cases where Swift 3 already performs +inference of `@objc` and will continue to do so if this proposal is +accepted. + +### Additional constructs that will infer `@objc` + +These are new cases that *should* infer `@objc`, but currently don't +in Swift. `@objc` should be inferred when: + +* The declaration has the `@GKInspectable` attribute. This inference + is required because the interaction with GameplayKit occurs entirely + through the Objective-C runtime. + +* The declaration has the `@IBInspectable` attribute. This inference + is required because the interaction with Interface Builder occurs entirely + through the Objective-C runtime. + +### `dynamic` no longer infers `@objc` + +A declaration that is `dynamic` will no longer infer `@objc`. For example: + +```swift +class MyClass { + dynamic func foo() { } // error: 'dynamic' method must be '@objc' + @objc dynamic func bar() { } // okay +} +``` + +This change is intended to separate current implementation +limitations from future language evolution: the current +implementation supports `dynamic` by always using the Objective-C +message send mechanism, allowing replacement of `dynamic` +implementations via the Objective-C runtime (e.g., `class_addMethod` +and `class_replaceMethod`). In the future, it is plausible that the +Swift language and runtime will evolve to support `dynamic` without +relying on the Objective-C runtime, and it's important that we leave +the door open for that language evolution. + +This change therefore does two things. First, it makes it clear that +the `dynamic` behavior is tied to the Objective-C runtime. Second, +it means that well-formed Swift 4 code will continue to work in the +same way should Swift gain the ability to provide `dynamic` without +relying on Objective-C: at that point, the method `foo()` above will +become well-formed, and the method `bar()` will continue to work as +it does today through the Objective-C runtime. Indeed, this change +is the right way forward even if Swift never supports `dynamic` in +its own runtime, following the precedent of +[SE-0070](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0070-optional-requirements.md), +which required the Objective-C-only protocol feature "optional +requirements" to be explicitly marked with `@objc`. + +### `NSObject`-derived classes no longer infer `@objc` + +A declaration within an `NSObject`-derived class will no longer infer +`@objc`. For example: + +```swift +class MyClass : NSObject { + func foo() { } // not exposed to Objective-C in Swift 4 +} +``` + +This is the only major change of this proposal, because it means +that a large number of methods that Swift 3 would have exposed to +Objective-C (and would, therefore, be callable from Objective-C code +in a mixed project) will no longer be exposed. On the other hand, +this is the most unpredictable part of the Swift 3 model, because +such methods infer `@objc` only when the method can be expressed in +Objective-C. For example: + +```swift +extension MyClass { + func bar(param: ObjCClass) { } // exposed to Objective-C in Swift 3; not exposed by this proposal + func baz(param: SwiftStruct) { } // not exposed to Objective-C +} +``` + +With this proposal, neither method specifies `@objc` nor is either +required by the semantic model to expose an Objective-C entrypoint, +so they don't infer `@objc`: there is no need to reason about the +type of the parameter's suitability in Objective-C. + +### Re-enabling `@objc` inference within a class hierarchy + +Some libraries and systems still depend greatly on the Objective-C +runtime's introspection facilities. For example, XCTest uses +Objective-C runtime metadata to find the test cases in `XCTestCase` +subclasses. To support such systems, introduce a new attribute for +classes in Swift, spelled `@objcMembers`, that re-enables `@objc` +inference for the class, its extensions, its subclasses, and (by +extension) all of their extensions. For example: + +```swift +@objcMembers +class MyClass : NSObject { + func foo() { } // implicitly @objc + + func bar() -> (Int, Int) // not @objc, because tuple returns + // aren't representable in Objective-C +} + +extension MyClass { + func baz() { } // implicitly @objc +} + +class MySubClass : MyClass { + func wibble() { } // implicitly @objc +} + +extension MySubClass { + func wobble() { } // implicitly @objc +} +``` + +This will be paired with an Objective-C attribute, spelled +`swift_objc_members`, that allows imported Objective-C classes to be +imported as `@objcMembers`: + +```objective-c +__attribute__((swift_objc_members)) +@interface XCTestCase : XCTest +/* ... */ +@end +``` + +will be imported into Swift as: + +```swift +@objcMembers +class XCTestCase : XCTest { /* ... */ } +``` + +### Enabling/disabling `@objc` inference within an extension + +There might be certain regions of code for which all of (or none of) +the entry points should be exposed to Objective-C. Allow either +`@objc` or `@nonobjc` to be specified on an `extension`. The `@objc` +or `@nonobjc` will apply to any member of that extension that does not +have its own `@objc` or `@nonobjc` annotation. For example: + +```swift +class SwiftClass { } + +@objc extension SwiftClass { + func foo() { } // implicitly @objc + func bar() -> (Int, Int) // error: tuple type (Int, Int) not + // expressible in @objc. add @nonobjc or move this method to fix the issue +} + +@objcMembers +class MyClass : NSObject { + func wibble() { } // implicitly @objc +} + +@nonobjc extension MyClass { + func wobble() { } // not @objc, despite @objcMembers +} +``` + +Note that `@objc` on an extension provides less-surprising behavior +than the implicit `@objc` inference of Swift 3, because it indicates +the intent to expose *everything* in that extension to Objective-C. If +some member within that extension cannot be exposed to Objective-C, +such as `SwiftClass.bar()`, the compiler will produce an error. + +## Side benefit: more reasonable expectations for `@objc` protocol extensions + +Users are often surprised to realize that extensions of `@objc` +protocols do not, in fact, produce Objective-C entrypoints: + +```swift +@objc protocol P { } + +extension P { + func bar() { } +} + +class C : NSObject, P { } + +let c = C() +print(c.responds(to: Selector("bar"))) // prints "false" +``` + +The expectation that `P.bar()` has an Objective-C entry point is set +by the fact that `NSObject`-derived Swift classes do implicitly create +Objective-C entry points for declarations within class extensions when +possible, but Swift does not (and, practically speaking, cannot) do +the same for protocol extensions. + +A previous mini-proposal [discussed +here](https://forums.swift.org/t/mini-proposal-require-nonobjc-on-members-of-objc-protocol-extensions/905) +suggested requiring `@nonobjc` for members of `@objc` protocol +extensions. However, limiting inference of `@objc` eliminates the +expectation itself, addressing the problem from a different angle. + +## Source compatibility + +The two changes that remove inference of `@objc` are both +source-breaking in different ways. The `dynamic` change mostly +straightforward: + +* In Swift 4 mode, introduce an error when a `dynamic` declaration + does not explicitly state `@objc` (or infer it based on one of the + `@objc` inference rules that still applies in Swift 4), with a + Fix-It to add the `@objc`. + +* In Swift 3 compatibility mode, continue to infer `@objc` for + `dynamic` methods. However, introduce a warning that such code will + be ill-formed in Swift 4, along with a Fix-It to add the + `@objc`. + +* A Swift 3-to-4 migrator could employ the same logic as Swift 3 + compatibility mode to update `dynamic` declarations appropriately. + +The elimination of inference of `@objc` for declarations in `NSObject` +subclasses is more complicated. Considering again the three cases: + +* In Swift 4 mode, do not infer `@objc` for such declarations. + Source-breaking changes that will be introduced include: + + * If `#selector` or `#keyPath` refers to one such declaration, an + error will be produced on previously-valid code that the + declaration is not `@objc`. In most cases, a Fix-It will suggest + the addition of `@objc`. + + * If a message is sent to one of these declarations via + `AnyObject`, the compiler may produce an error (if no `@objc` + entity by that name exists anywhere) or a failure might occur at + runtime (if another, unrelated `@objc` entity exists with that + same name). For example: + + ```swift + class MyClass : NSObject { + func foo() { } + func bar() { } + } + + class UnrelatedClass : NSObject { + @objc func bar() { } + } + + func test(object: AnyObject) { + object.foo?() // Swift 3: can call method MyClass.foo() + // Swift 4: compiler error, no @objc method "foo()" + object.bar?() // Swift 3: can call MyClass.bar() or UnrelatedClass.bar() + // Swift 4: can only call UnrelatedClass.bar() + } + ``` + + * If one of these declarations is written in a class extension and + is overridden, the override will produce an error in Swift 4 + because Swift's class model does not support overriding + declarations introduced in class extensions. For example: + + ```swift + class MySuperclass : NSObject { } + + extension MySuperclass { + func extMethod() { } // implicitly @objc in Swift 3, not in Swift 4 + } + + class MySubclass : MySuperclass { + override func extMethod() { } // Swift 3: okay + // Swift 4: error "declarations in extensions cannot override yet" + } + ``` + + * Objective-C code in mixed-source projects won't be able to call + these declarations. Most problems caused by this will result in + warnings or errors from the Objective-C compiler (due to + unrecognized selectors); some may only be detected at runtime, + similarly to the `AnyObject` case described above. + + * Other tools and frameworks that rely on the presence of + Objective-C entrypoints (e.g., via strings) but do not make use + of Swift's facilities for referring to them will fail. This case + is particularly hard to diagnose well, and failures of this sort + are likely to cause runtime failures (e.g., unrecoignized + selectors) that only the developer can diagnose and correct. + +* In Swift 3 compatibility mode, continue to infer `@objc` for these + declarations. We can warn about uses of the `@objc` entrypoints in + cases where the `@objc` is inferred in Swift 3 but will not be in + Swift 4. + +* A Swift 3-to-4 migrator is the hardest part of the story. The + migrator should have a switch: a "conservative" option and a + "minimal" option. + + * The "conservative" option (which is the best default) simply adds + explicit `@objc` annotations to every entity that was implicitly + `@objc` in Swift 3 but would not implicitly be `@objc` in Swift + 4. Migrated projects won't get the benefits of the more-limited + `@objc` inference, but they will work out-of-the-box. + + * The "minimal" option attempts to only add `@objc` in places where + it is needed to maintain the semantics of the program. It would be + driven by the diagnostics mentioned above (for `#selector`, + `#keyPath`, `AnyObject` messaging, and overrides), but some manual + intervention will be involved to catch the runtime cases. More + discussion of the migration workflow follows. + + +## "Minimal" migration workflow + +To migrate a Swift 3 project to Swift 4 without introducing spurious +Objective-C entry points, we can apply the following workflow: + +1. In Swift 4 mode, address all of the warnings about uses of +declarations for which `@objc` was inferred based on the deprecated +rule. +2. Set the environment variable `SWIFT_DEBUG_IMPLICIT_OBJC_ENTRYPOINT` +to a value between 1 and 3 (see below) and test the application. Clean +up any "deprecated `@objc` entrypoint` warnings. +3. Migrate to Swift 4 with "minimal" migration, which at this point +will only add `@objc` to explicitly `dynamic` declarations. + +The following subsections describe this migration in more detail. + +### Step 1: Address compiler warnings + +The compiler can warn about most instances of the source-breaking +changes outlined above. Here is an example that demonstrates the +warnings in Swift code, all of which are generated by the Swift +compiler: + +```swift +class MyClass : NSObject { + func foo() { } + + var property: NSObject? = nil + + func baz() { } +} + +extension MyClass { + func bar() { } +} + +class MySubClass : MyClass { + override func foo() { } // okay + + override func bar() { } // warning: override of instance method + // 'bar()' from extension of 'MyClass' depends on deprecated inference + // of '@objc' +} + +func test(object: AnyObject, mine: MyClass) { + _ = #selector(MyClass.foo) // warning: argument of `#selector` + // refers to instance method `foo()` in `MyClass` that uses deprecated + // `@objc` inference + + _ = #keyPath(MyClass.property) // warning: argument of '#keyPath' + // refers to property 'property' in 'MyClass' that uses deprecated + // `@objc` inference + + _ = object.baz?() // warning: reference to instance + // method 'baz()' of 'MyClass' that uses deprecated `@objc` inference +} +``` + +For mixed-source projects, the Swift compiler will annotate the +generated Swift header with "deprecation" attributes, so that any +references to those declarations *from Objective-C code* will also +produce warnings. For example: + +```objective-c +#import "MyApp-Swift.h" + +void test(MyClass *mine) { + [mine foo]; // warning: -[MyApp.MyClass foo] uses deprecated + // '@objc' inference; add '@objc' to provide an Objective-C entrypoint +} +``` + +### Step 2: Address (opt-in) runtime warnings + +Swift 3 compatibility mode augments each of the Objective-C +entrypoints introduced based on the deprecated `@objc` inference rules +with a call to a new runtime function +`swift_objc_swift3ImplicitObjCEntrypoint`. This entry point can be +used in two ways to find cases where an Objective-C entry point that +will be eliminated by the migration to Swift 4: + +* In a debugger, one can set a breakpoint on +`swift_objc_swift3ImplicitObjCEntrypoint` to catch specific cases +where the Objective-C entry point is getting called. + +* One can set the environment variable +`SWIFT_DEBUG_IMPLICIT_OBJC_ENTRYPOINT` to one of three different +values to cause the Swift runtime to log uses of these Objective-C +entry points: + 1. Log calls to these entry points with a message such as: + ```***Swift runtime: entrypoint -[MyApp.MyClass foo] generated by implicit @objc inference is deprecated and will be removed in Swift 4``` + 2. Log (as in #1) and emit a backtrace showing how that Objective-C + entry point was invoked. + 3. Log with a backtrace (as in #2), then crash. This last stage is + useful for automated testing leading up to the migration to Swift 4. + +Testing with logging enabled should uncover uses of the Objective-C +entry points that use the deprecated rules. As explicit `@objc` is +added to each case, the runtime warnings will go away. + +### Step 3: Migrate to Swift 4 + +At this point, one can migrate to Swift 4. Building in Swift 4 will +remove the Objective-C entry points for any remaining case where +`@objc` was inferred based on the deprecated rules. + +## Effect on ABI stability + +This proposal has no effect on the Swift ABI, because it only concerns the Objective-C entry points for Swift entities, which have always been governed by the already-set-in-stone Objective-C ABI. Whether a particular Swift entity is `@objc` or not does not affect its Swift ABI. + +## Effect on API resilience + +The [library evolution document](https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst) notes that adding or removing `@objc` is not a resilient API change. Therefore, changing the inference behavior of `@objc` doesn't really have an impact on API resilience beyond the normal concerns about errors of omission: prior to this proposal, forgetting to add `@nonobjc` meant that an API might be stuck vending an Objective-C entry point it didn't want to expose; with this proposal, forgetting to add `@objc` means that an API might fail to be usable from Objective-C. The latter problem, at least, can be addressed by exposing an additional entrypoint. Moreover, adding an Objective-C entrypoint is "less" ABI-breaking that removing an Objective-C entrypoint, because the former is only breaking for `open` or `dynamic` members. + +## Alternatives considered + +Aside from the obvious alternative of "do nothing", there are ways to +address some of the problems called out in the +[Motivation](#motivation) section without eliminating inference in the +cases we're talking about, or to soften the requirements on some +constructs. + +### Mangling Objective-C selectors +Some of the problems with Objective-C selector collisions could be +addressed by using "mangled" selector names for Swift-defined +declarations. For example, given: + +```swift +class MyClass : NSObject { + func print(_ value: Int) { } +} +``` + +Instead of choosing the Objective-C selector "print:" by default, +which is likely to conflict, we could use a mangled selector name like +`__MyModule__MyClass__print__Int:` that is unlikely to conflict with +anything else in the program. However, this change would also be +source-breaking for the same reasons that restricting `@objc` +inference is: dynamic behavior that constructs Objective-C selectors +or tools outside of Swift that expect certain selectors will break at +run-time. + +### Completely eliminating `@objc` inference + +Another alternative to this proposal is to go further and completely +eliminate `@objc` inference. This would simplify the programming model +further---it's exposed to Objective-C only if it's marked +`@objc`---but at the cost of significantly more boilerplate for +applications that use Objective-C frameworks. For example: + +```swift +class Sub : Super { + @objc override func foo() { } // @objc is now required +} + +class MyClass : MyDelegate { + @objc func bar() { } // @objc is now required +} +``` + +I believe that this proposal strikes the right balance already, where +`@objc` is inferred when it's needed to maintain the semantic model, +and can be explicitly added to document those places where the user is +intentionally exposing an Objective-C entrypoint for some +reason. Thus, explicitly writing `@objc` indicates intent without +creating boilerplate. + +# Acknowledgments + +Thanks to Brian King for noting the inference of `dynamic` and its +relationship to this proposal. + +# Revision history + +[Version 1](https://github.com/swiftlang/swift-evolution/blob/0389b1f49fc55b1a898701c549ce89738307b9fc/proposals/0160-objc-inference.md) +of this proposal did not include the use of `@objcMembers` on classes +or the use of `@objc`/`@nonobjc` on extensions to mass-annotate. diff --git a/proposals/0161-key-paths.md b/proposals/0161-key-paths.md new file mode 100644 index 0000000000..b945854b41 --- /dev/null +++ b/proposals/0161-key-paths.md @@ -0,0 +1,201 @@ +# Smart KeyPaths: Better Key-Value Coding for Swift + +* Proposal: [SE-0161](0161-key-paths.md) +* Authors: [David Smith](https://github.com/Catfish-Man), [Michael LeHew](https://github.com/mlehew), [Joe Groff](https://github.com/jckarter) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0161-smart-keypaths-better-key-value-coding-for-swift/5690) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/55e61f459632eca2face40e571a517919f846cfb/proposals/0161-key-paths.md) + +## Introduction +We propose a family of concrete _Key Path_ types that represent uninvoked references to properties that can be composed to form paths through many values and directly get/set their underlying values. + +## Motivation +#### We Can Do Better than String +On Darwin platforms Swift's existing `#keyPath()` syntax provides a convenient way to safely *refer* to properties. Unfortunately, once validated, the expression becomes a `String` which has a number of important limitations: + +* Loss of type information (requiring awkward `Any` APIs) +* Unnecessarily slow to parse +* Only applicable to `NSObjects` +* Limited to Darwin platforms + +#### Use/Mention Distinctions +While methods can be referred to without invoking them (`let x = foo.bar` instead of `let x = foo.bar()`), this is not currently possible for properties and subscripts. + +Making indirect references to a properties' concrete types also lets us expose metadata about the property, and in the future additional behaviors. + +#### More Expressive KeyPaths +We would also like to support being able to use _Key Paths_ to access into collections and other subscriptable types, which is not currently possible. + +## Proposed solution +We propose introducing a new expression similar to function type references (e.g. `Type.method`), but for properties and subscripts. To avoid ambiguities with type properties, we propose we escape such expressions using `\` to indicate that you are talking about the property, not its invocation. A key path expression takes the general form `\.`, where `` is a type name, and `` is a chain of one or more property, subscript, or optional chaining/forcing operators. If the type name can be inferred from context, then it can be elided, leaving `\.`. + +These property reference expressions produce `KeyPath` objects, rather than `Strings`. `KeyPaths` are a family of generic classes _(structs and protocols here would be ideal, but requires generalized existentials)_ which encapsulate a property reference or chain of property references, including the type, mutability, property name(s), and ability to set/get values. + +Here's a sample of it in use: + +```swift +class Person { + var name: String + var friends: [Person] = [] + var bestFriend: Person? = nil + init(name: String) { + self.name = name + } +} + +var han = Person(name: "Han Solo") +var luke = Person(name: "Luke Skywalker") +luke.friends.append(han) + +// create a key path and use it +let firstFriendsNameKeyPath = \Person.friends[0].name +let firstFriend = luke[keyPath: firstFriendsNameKeyPath] // "Han Solo" + +// or equivalently, with type inferred from context +luke[keyPath: \.friends[0].name] // "Han Solo" +// The path must always begin with a dot, even if it starts with a +// subscript component +luke.friends[keyPath: \.[0].name] // "Han Solo" +luke.friends[keyPath: \[Person].[0].name] // "Han Solo" + +// rename Luke's first friend +luke[keyPath: firstFriendsNameKeyPath] = "A Disreputable Smuggler" + +// optional properties work too +let bestFriendsNameKeyPath = \Person.bestFriend?.name +let bestFriendsName = luke[keyPath: bestFriendsNameKeyPath] // nil, if he is the last Jedi +``` + +## Detailed design +### Core KeyPath Types +`KeyPaths` are a hierarchy of progressively more specific classes, based on whether we have prior knowledge of the path through the object graph we wish to traverse. + +##### Unknown Path / Unknown Root Type +`AnyKeyPath` is fully type-erased, referring to 'any route' through an object/value graph for 'any root'. Because of type-erasure many operations can fail at runtime and are thus optional. + +```swift +class AnyKeyPath: CustomDebugStringConvertible, Hashable { + // MARK - Composition + // Returns nil if path.rootType != self.valueType + func appending(path: AnyKeyPath) -> AnyKeyPath? + + // MARK - Runtime Information + class var rootType: Any.Type + class var valueType: Any.Type + + static func == (lhs: AnyKeyPath, rhs: AnyKeyPath) -> Bool + var hashValue: Int +} +``` +##### Unknown Path / Known Root Type +If we know a little more type information (what kind of thing the key path is relative to), then we can use `PartialKeyPath`, which refers to an 'any route' from a known root: + +```swift +class PartialKeyPath: AnyKeyPath { + // MARK - Composition + // Returns nil if Value != self.valueType + func appending(path: AnyKeyPath) -> PartialKeyPath? + func appending(path: KeyPath) -> KeyPath? + func appending(path: ReferenceKeyPath) -> ReferenceKeyPath? +} +``` + +##### Known Path / Known Root Type +When we know both what the path is relative to and what it refers to, we can use `KeyPath`. Thanks to the knowledge of the `Root` and `Value` types, all of the failable operations lose their `Optional`. + +```swift +public class KeyPath: PartialKeyPath { + // MARK - Composition + func appending(path: KeyPath) -> KeyPath + func appending(path: WritableKeyPath) -> Self + func appending(path: ReferenceWritableKeyPath) -> ReferenceWritableKeyPath +} +``` + +##### Value/Reference Mutation Semantics Mutation +Finally, we have a pair of subclasses encapsulating value/reference mutation semantics. These have to be distinct because mutating a copy of a value is not very useful, so we need to mutate an inout value. + +```swift +class WritableKeyPath: KeyPath { + // MARK - Composition + func appending(path: WritableKeyPath) -> WritableKeyPath +} + +class ReferenceWritableKeyPath: WritableKeyPath { + override func appending(path: WritableKeyPath) -> ReferenceWritableKeyPath +} +``` + +### Access and Mutation Through KeyPaths +To get or set values for a given root and key path we effectively add the following subscripts to all Swift types. + +```swift +extension Any { + subscript(keyPath path: AnyKeyPath) -> Any? { get } + subscript(keyPath path: PartialKeyPath) -> Any { get } + subscript(keyPath path: KeyPath) -> Value { get } + subscript(keyPath path: WritableKeyPath) -> Value { set, get } +} +``` + +This allows for code like + +```swift +let someKeyPath = ... +person[keyPath: someKeyPath] +``` + +which is both appealingly readable, and doesn't require read-modify-write copies (subscripts access `self` inout). Conflicts with existing subscripts are avoided by using a named parameter and generics to only accept key paths with a `Root` of the type in question. + +### Referencing Key Paths +Forming a `KeyPath` utilizes a new escape sigil `\`. We feel this best serves our needs of disambiguating from existing `#keyPath` expressions (which will continue to produce `Strings`) and existing type properties. + +Optionals are handled via optional-chaining. Multiply dotted expressions are allowed as well, and work just as if they were composed via the `appending` methods on `KeyPath`. + +Forming a key path through subscripts (e.g. Array / Dictionary) will have the limitation that the parameter's type(s) must be `Hashable`. Should the archival and serialization proposal be accepted, we would also like to include `Codable` with an eye towards being able to make key paths `Codable` themselves in the future. + +### Performance +The performance of interacting with a property/subscript via `KeyPaths` should be close to the cost of calling the property directly. + +## Source compatibility +This change is additive and there should no affect on existing source. + +## Effect on ABI stability +This feature adds the following requirements to ABI stability: + +- mechanism to access key paths of public properties + +We think a protocol-based design would be preferable once the language has sufficient support for generalized existentials to make that ergonomic. By keeping the class hierarchy closed and the concrete implementations private to the implementation it should be tractable to provide compatibility with an open protocol-based design in the future. + +## Effect on API resilience +This should not significantly impact API resilience, as it merely provides a new mechanism for operating on existing APIs. + +## Alternatives considered + +#### More Features +Various drafts of this proposal have included additional features (decomposable key paths, prefix comparisons, support for custom `KeyPath` subclasses, creating a `KeyPath` from a `String` at runtime, `KeyPaths` conforming to `Codable`, bound key paths as a concrete type, etc.). We anticipate approaching these enhancements additively once the core `KeyPath` functionality is in place. + +#### Spelling +We also explored many different spellings, each with different strengths. We have chosen the current syntax for the clarity and discoverability it provides in practice. + +| Case | `#keyPath` | Function Type Reference | Escape | +| --- | --- | --- | --- | +| Fully qualified | `#keyPath(Person, .friends[0].name)` | `Person.friends[0].name` | `\Person.friends[0].name` | +| Type Inferred | `#keyPath(.friends[0].name)` |`Person.friends[0].name` | `\.friends[0].name` | + +While the crispness of the function-type-reference is appealing, it becomes ambiguous when working with type properties. The escape-sigil variant avoids this, and remains quite readable. + +#### Why `\`? +During review many different sigils were considered: + +**No Sigil**: This matches function type references, but suffers from ambiguity with wanting to actually call a type property. Having to type `let foo: KeyPath` while consistent with function type references, really is not that great (even for function type references). + +**Back Tick**: Borrowing from lisp, back-tick was what we used in initial discussions of this proposal (it was easy to write on a white-board), but it was not chosen because it is hard to type in markdown, and comes dangerously close to conflicting with other parser intrinsics. + +**Pound**: We considered `#` as well, and while it is appealing, we'd like to save it for the future. `#` also has a slightly more computational connotation in Swift so far. For instance, `#keyPath` 'identifies if its valid and returns a String', `#available` does the necessary computation to verify availability and yields a boolean. + +**Back Slash**: Where `#` is computational, `\` in Swift has more of a 'behave differently for a moment' connotation, and that seems to fit exactly what we want when forming a key path. + +#### Function Type References +We think the disambiguating benefits of the escape-sigil would greatly benefit function type references, but such considerations are outside the scope of this proposal. diff --git a/proposals/0162-package-manager-custom-target-layouts.md b/proposals/0162-package-manager-custom-target-layouts.md new file mode 100644 index 0000000000..78d428e350 --- /dev/null +++ b/proposals/0162-package-manager-custom-target-layouts.md @@ -0,0 +1,262 @@ +# Package Manager Custom Target Layouts + +* Proposal: [SE-0162](0162-package-manager-custom-target-layouts.md) +* Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Rick Ballard](https://github.com/rballard) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0162-package-manager-custom-target-layouts/5647) +* Bug: [SR-29](https://bugs.swift.org/browse/SR-29) + +## Introduction + +This proposal enhances the `Package.swift` manifest APIs to support custom +target layouts, and removes a convention which allowed omission of targets from +the manifest. + +## Motivation + +The Package Manager uses a convention system to infer targets structure from +disk layout. This works well for most packages, which can easily adopt the +conventions, and frees users from needing to update their `Package.swift` file +every time they add or remove sources. Adopting the conventions is more +difficult for some packages, however – especially existing C libraries or large +projects, which would be difficult to reorganize. We intend to give users a way +to make such projects into packages without needing to conform to our +conventions. + +The current convention rules make it very convenient to add new targets and +source files by inferring them automatically from disk, but they also can be +confusing, overly-implicit, and difficult to debug; for example, if the user +does not follow the conventions correctly which determine their targets, they +may wind up with targets they don't expect, or not having targets they did +expect, and either way their clients can't easily see which targets are available +by looking at the `Package.swift` manifest. We want to retain convenience where it +really matters, such as easy addition of new source files, but require explicit +declarations where being explicit adds significant value. We also want to make +sure that the implicit conventions we keep are straightforward and easy to +remember. + +## Proposed solution + +* We propose to stop inferring targets from disk. They must be explicitly declared + in the manifest file. The inference was not very useful, as targets eventually + need to be declared in order to use common features such as product and target + dependencies, or build settings (which are planned for Swift 4). Explicit + target declarations make a package easier to understand by clients, and allow us + to provide good diagnostics when the layout on disk does not match the + declarations. + +* We propose to remove the requirement that name of a test target must have + suffix "Tests". Instead, test targets will be explicitly declared as such + in the manifest file. + +* We propose a list of pre-defined search paths for declared targets. + + When a target does not declare an explicit path, these directories will be used + to search for the target. The name of the directory must match the name of + the target. The search will be done in order and will be case-sensitive. + + Regular targets: package root, Sources, Source, src, srcs. + Test targets: Tests, package root, Sources, Source, src, srcs. + + It is an error if a target is found in more than one of these paths. In + such cases, the path should be explicitly declared using the path property + proposed below. + +* We propose to add a factory method `testTarget` to the `Target` class, to define + test targets. + + ```swift + .testTarget(name: "FooTests", dependencies: ["Foo"]) + ``` + +* We propose to add three properties to the `Target` class: `path`, `sources` and + `exclude`. + + * `path`: This property defines the path to the top-level directory containing the + target's sources, relative to the package root. It is not legal for this path + to escape the package root, i.e., values like "../Foo", "/Foo" are invalid. The + default value of this property will be `nil`, which means the target will be + searched for in the pre-defined paths. The empty string ("") or dot (".") implies + that the target's sources are directly inside the package root. + + * `sources`: This property defines the source files to be included in the + target. The default value of this property will be nil, which means all + valid source files found in the target's path will be included. This can + contain directories and individual source files. Directories will be + searched recursively for valid source files. Paths specified are relative + to the target path. + + Each source file will be represented by String type. In future, we will + consider upgrading this to its own type to allow per-file build settings. + The new type would conform to `CustomStringConvertible`, so existing + declarations would continue to work (except where the strings were + constructed programmatically). + + * `exclude`: This property can be used to exclude certain files and + directories from being picked up as sources. Exclude paths are relative + to the target path. This property has more precedence than `sources` + property. + + _Note: We plan to support globbing in future, but to keep this proposal short + we are not proposing it right now._ + +* It is an error if the paths of two targets overlap (unless resolved with `exclude`). + + ```swift + // This is an error: + .target(name: "Bar", path: "Sources/Bar"), + .testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"), + + // This works: + .target(name: "Bar", path: "Sources/Bar", exclude: ["Tests"]), + .testTarget(name: "BarTests", dependencies: ["Bar"], path: "Sources/Bar/Tests"), + ``` + +* For C family library targets, we propose to add a `publicHeadersPath` + property. + + This property defines the path to the directory containing public headers of + a C target. This path is relative to the target path and default value of + this property is `include`. This mechanism should be further improved + in the future, but there are several behaviors, such as modulemap generation, + which currently depend of having only one public headers directory. We will address + those issues separately in a future proposal. + + _All existing rules related to custom and automatic modulemap remain intact._ + +* Remove exclude from `Package` class. + + This property is no longer required because of the above proposed + per-target exclude property. + +* The templates provided by the `swift package init` subcommand will be updated + according to the above rules, so that users do not need to manually + add their first target to the manifest. + +## Examples: + +* Dummy manifest containing all Swift code. + +```swift +let package = Package( + name: "SwiftyJSON", + targets: [ + .target( + name: "Utility", + path: "Sources/BasicCode" + ), + + .target( + name: "SwiftyJSON", + dependencies: ["Utility"], + path: "SJ", + sources: ["SwiftyJSON.swift"] + ), + + .testTarget( + name: "AllTests", + dependencies: ["Utility", "SwiftyJSON"], + path: "Tests", + exclude: ["Fixtures"] + ), + ] +) +``` + +* LibYAML + +```swift +let packages = Package( + name: "LibYAML", + targets: [ + .target( + name: "libyaml", + sources: ["src"] + ) + ] +) +``` + +* Node.js http-parser + +```swift +let packages = Package( + name: "http-parser", + targets: [ + .target( + name: "http-parser", + publicHeaders: ".", + sources: ["http_parser.c"] + ) + ] +) +``` + +* swift-build-tool + +```swift +let packages = Package( + name: "llbuild", + targets: [ + .target( + name: "swift-build-tool", + path: ".", + sources: [ + "lib/Basic", + "lib/llvm/Support", + "lib/Core", + "lib/BuildSystem", + "products/swift-build-tool/swift-build-tool.cpp", + ] + ) + ] +) +``` + +## Impact on existing code + +These enhancements will be added to the version 4 manifest API, which will +release with Swift 4. There will be no impact on packages using the version 3 +manifest API. When packages update their minimum tools version to 4.0, they +will need to update the manifest according to the changes in this proposal. + +There are two flat layouts supported in Swift 3: + +1. Source files directly in the package root. +2. Source files directly inside a `Sources/` directory. + +If packages want to continue using either of these flat layouts, they will need +to explicitly set a target path to the flat directory; otherwise, a directory +named after the target is expected. For example, if a package `Foo` has +following layout: + +``` +Package.swift +Sources/main.swift +Sources/foo.swift +``` + +The updated manifest will look like this: + +```swift +// swift-tools-version:4.0 +import PackageDescription + +let package = Package( + name: "Foo", + targets: [ + .target(name: "Foo", path: "Sources"), + ] +) +``` + +## Alternatives considered + +We considered making a more minimal change which disabled the flat layouts +by default, and provided a top-level property to allow opting back in to them. +This would allow us to discourage these layouts – which we would like +to do before the package ecosystem grows – without needing to add a fully +customizable API. However, we think the fuller API we've proposed here is fairly +straightforward and provides the ability to make a number of existing projects +into packages, so we think this is worth doing at this time. diff --git a/proposals/0163-string-revision-1.md b/proposals/0163-string-revision-1.md new file mode 100644 index 0000000000..f584cc67f7 --- /dev/null +++ b/proposals/0163-string-revision-1.md @@ -0,0 +1,453 @@ +# String Revision: Collection Conformance, C Interop, Transcoding + +* Proposal: [SE-0163](0163-string-revision-1.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Dave Abrahams](https://github.com/dabrahams/) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 4.0)** +* Revision: 2 +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7513547ddac66b06770a1fd620aad915d75987ff/proposals/0163-string-revision-1.md) +* Decision Notes: [Rationale #1](https://forums.swift.org/t/accepted-se-0163-string-revision-collection-conformance-c-interop-transcoding/5716/2), [Rationale #2](https://forums.swift.org/t/accepted-se-0163-string-revision-collection-conformance-c-interop-transcoding/5952) + +## Introduction + +This proposal is to implement a subset of the changes from the [Swift 4 +String +Manifesto](https://github.com/apple/swift/blob/master/docs/StringManifesto.md). + +Specifically: + + * Make `String` conform to `BidirectionalCollection` + * Make `String` conform to `RangeReplaceableCollection` + * Create a `Substring` type for `String.SubSequence` + * Create a `StringProtocol` protocol to allow for generic operations over both types. + * Consolidate on a concise set of C interop methods. + * Revise the transcoding infrastructure. + * Sink Unicode-specific functionality into a `Unicode` namespace. + +Other existing aspects of `String` remain unchanged for the purposes of this +proposal. + +## Motivation + +This proposal follows up on a number of recommendations found in the manifesto: + +`Collection` conformance was dropped from `String` in Swift 2. After +reevaluation, the feeling is that the minor discrepancies with +required `RangeReplaceableCollection` semantics (the fact that some +characters may merge when Strings are concatenated) are outweighed by +the significant benefits of restoring these conformances. For more +detail on the reasoning, +see +[here](https://github.com/apple/swift/blob/master/docs/StringManifesto.md#string-should-be-a-collection-of-characters-again) + +While it is not a collection, the Swift 3 string does have slicing operations. +`String` is currently serving as its own subsequence, allowing substrings +to share storage with their "owner". This can lead to memory leaks when small substrings of larger +strings are stored long-term (see [here](https://github.com/apple/swift/blob/master/docs/StringManifesto.md#substrings) +for more detail on this problem). Introducing a separate type of `Substring` to +serve as `String.Subsequence` is recommended to resolve this issue, in a similar +fashion to `ArraySlice`. + +As noted in the manifesto, support for interoperation with nul-terminated C +strings in Swift 3 is scattered and incoherent, with 6 ways to transform a C +string into a `String` and four ways to do the inverse. These APIs should be +replaced with a simpler set of methods on `String`. + +## Proposed solution + +A new type, `Substring`, will be introduced. Similar to `ArraySlice` it will +be documented as only for short- to medium-term storage: + +> **Important** +> +> Long-term storage of `Substring` instances is discouraged. A substring holds a +> reference to the entire storage of a larger string, not just to the portion it +> presents, even after the original string’s lifetime ends. Long-term storage of +> a substring may therefore prolong the lifetime of elements that are no longer +> otherwise accessible, which can appear to be memory leakage. + +Aside from minor differences, such as having a `SubSequence` of `Self` +and a larger size to describe the range of the subsequence, +`Substring` will be near-identical from a user perspective. + +In order to be able to write extensions across both `String` and +`Substring`, a new `StringProtocol` protocol to which the two types +will conform will be introduced. For the purposes of this proposal, +`StringProtocol` will be defined as a protocol to be used whenever you +would previously extend `String`. It should be possible to substitute +`extension StringProtocol { ... }` in Swift 4 wherever +`extension String { ... }` was written in Swift 3, with one exception: any +passing of `self` into an API that takes a concrete `String` will need to be +rewritten as `String(self)`. If `Self` is a `String` then this should +effectively optimize to a no-op, whereas if `Self` is a `Substring` then this +will force a copy, helping to avoid the "memory leak" problems described above. + +The exact nature of the protocol – such as which methods should be +protocol requirements vs which can be implemented as protocol +extensions, are considered implementation details and so not covered +in this proposal. + +`StringProtocol` will conform to `BidirectionalCollection`. +`RangeReplaceableCollection` conformance will be added directly onto +the `String` and `Substring` types, as it is possible future +`StringProtocol`-conforming types might not be range-replaceable +(e.g. an immutable type that wraps a `const char *`). + +The C string interop methods will be updated to a variant of those +described +[here](https://github.com/apple/swift/blob/master/docs/StringManifesto.md#c-string-interop): +two `withCString` operations and two `init(cString:)` constructors, +one each for UTF8 and for arbitrary encodings. The primary change is +to remove "non-repairing" variants of construction from nul-terminated +C strings. In both of the construction APIs, any invalid encoding +sequence detected will have its longest valid prefix replaced by +`U+FFFD`, the Unicode replacement character, per the Unicode +specification. This covers the common case. The replacement can be +done physically in the underlying storage and the validity of the +result can be recorded in the String's encoding such that future +accesses need not be slowed down by possible error repair +separately. Construction that is aborted when encoding errors are +detected can be accomplished using APIs on the encoding. + +Additionally, an `init` that takes a collection of code units and an encoding +will allow for construction of a `String` from arbitrary collections – for example, +an `UnsafeBufferPointer` containing a non-nul-terminated C string. + +The current transcoding support will be updated to improve usability and +performance. The primary changes will be: + + - to allow transcoding directly from one encoding to another without having + to triangulate through an intermediate scalar value + - to add the ability to transcode an input collection in reverse, allowing the + different views on `String` to be made bi-directional + - to ensure that the APIs can be + used to create performant bidirectional decoded and transcoded + views of underlying code units. + - to replace the `UnicodeCodec` with a stateless `Unicode.Encoding` + protocol having associated `ForwardParser` and `ReverseParser` + types for decoding. + +The standard library currently lacks a `Latin1` codec, so a +`enum Latin1: Unicode.Encoding` type will be added. + +## Detailed design + +### The `Unicode` Namespace + +A `Unicode` “namespace” will be added for components related to +low-level Unicode operations such as transcoding and grapheme +breaking. Absent more direct language support, `Unicode` will, for the +time being, be implemented as a caseless `enum`. [The caseless `enum` +technique is precedented by `CommandLine`, which vends the equivalent +of `argc` and `argv` for command-line applications.] + +```swift +enum Unicode { + enum ASCII : Unicode.Encoding { ... } + enum UTF8 : Unicode.Encoding { ... } + enum UTF16 : Unicode.Encoding { ... } + enum UTF32 : Unicode.Encoding { ... } + ... + enum ParseResult { ... } + struct Scalar { ... } +} +``` + +The names `UTF8`, `UTF16`, `UTF32`, and `Scalar` correspond +to entities that exist in Swift 3. For backward compatibility they will +be exposed to Swift 3 programs with their legacy spellings: + +```swift +@available(swift, obsoleted: 4.0, renamed: "Unicode.UTF8") +public typealias UTF8 = Unicode.UTF8 +@available(swift, obsoleted: 4.0, renamed: "Unicode.UTF16") +public typealias UTF16 = Unicode.UTF16 +@available(swift, obsoleted: 4.0, renamed: "Unicode.UTF32") +public typealias UTF32 = Unicode.UTF32 +@available(swift, obsoleted: 4.0, renamed: "Unicode.Scalar") +public typealias UnicodeScalar = Unicode.Scalar +``` + +Unicode-specific protocols will be presented as members of this +namespace. Pending the addition of more direct language support, +typealiases will be used to bring them in from underscored names in +the `Swift` namespace. The intention is that diagnostics and +documentation will display the nested, non-underscored names. + +```swift +protocol _UnicodeEncoding { ... } +protocol _UnicodeParser { ... } +extension Unicode { + typealias Encoding = _UnicodeEncoding + typealias Parser = _UnicodeParser +} +``` + +`UnicodeCodec` will be updated to refine `Unicode.Encoding`, and +deprecated for Swift 4. Existing models of `UnicodeCodec` such as +`UTF8` will inherit `Unicode.Encoding` conformance for Swift 3. + +As noted [below](#higher-level-unicode-processing) we anticipate +adding many more Unicode-specific components to the `Unicode` +namespace in the near future. + +### `String`, `Substring`, and `StringProtocol` + +The following additions will be made to the standard library: + +```swift +protocol StringProtocol : BidirectionalCollection { + // Implementation detail as described above +} + +extension String : StringProtocol, RangeReplaceableCollection { + typealias SubSequence = Substring + subscript(bounds: Range) -> Substring { + ... + } +} + +struct Substring : StringProtocol, RangeReplaceableCollection { + typealias SubSequence = Substring + // near-identical API surface area to String +} +``` + +The slicing operations on `String` will be amended to return +`Substring`: + +```swift +struct String { + subscript(bounds: Range) -> Substring { ... } +} +``` + +Note that properties or methods that due to their nature create new +`String` storage (such as `lowercased()`) will _not_ change. + +C string interopability will be consolidated on the following methods: + +```swift +extension String { + /// Constructs a `String` having the same contents as `codeUnits`. + /// + /// - Parameter codeUnits: a collection of code units in + /// the given `encoding`. + /// - Parameter encoding: describes the encoding in which the code units + /// should be interpreted. + init( + decoding codeUnits: C, as encoding: Encoding.Type + ) + where C.Iterator.Element == Encoding.CodeUnit + + /// Constructs a `String` having the same contents as `nulTerminatedUTF8`. + /// + /// - Parameter nulTerminatedUTF8: a sequence of contiguous UTF-8 encoded + /// bytes ending just before the first zero byte (NUL character). + init(cString nulTerminatedUTF8: UnsafePointer) + + /// Constructs a `String` having the same contents as `nulTerminatedCodeUnits`. + /// + /// - Parameter nulTerminatedCodeUnits: a sequence of contiguous code units in + /// the given `encoding`, ending just before the first zero code unit. + /// - Parameter encoding: describes the encoding in which the code units + /// should be interpreted. + init( + decodingCString nulTerminatedCodeUnits: UnsafePointer, + as: Encoding.Type) + + /// Invokes the given closure on the contents of the string, represented as a + /// pointer to a null-terminated sequence of UTF-8 code units. + func withCString( + _ body: (UnsafePointer) throws -> Result) rethrows -> Result + + /// Invokes the given closure on the contents of the string, represented as a + /// pointer to a null-terminated sequence of code units in the given encoding. + func withCString( + encodedAs: Encoding.Type, + _ body: (UnsafePointer) throws -> Result + ) rethrows -> Result +} +``` + +Additionally, the current ability to pass a Swift `String` directly +into methods that take a C string (`UnsafePointer`) will remain +as-is. + +### Low-level Unicode Processing + +A new protocol, `Unicode.Encoding`, will be added to replace the +current `UnicodeCodec` protocol. + +```swift +extension Unicode { typealias Encoding = _UnicodeEncoding } + +public protocol _UnicodeEncoding { + /// The basic unit of encoding + associatedtype CodeUnit : UnsignedInteger, FixedWidthInteger + + /// A valid scalar value as represented in this encoding + associatedtype EncodedScalar : BidirectionalCollection + where EncodedScalar.Iterator.Element == CodeUnit + + /// A unicode scalar value to be used when repairing + /// encoding/decoding errors, as represented in this encoding. + /// + /// If the Unicode replacement character U+FFFD is representable in this + /// encoding, `encodedReplacementCharacter` encodes that scalar value. + static var encodedReplacementCharacter : EncodedScalar { get } + + /// Converts from encoded to encoding-independent representation + static func decode(_ content: EncodedScalar) -> Unicode.Scalar + + /// Converts from encoding-independent to encoded representation, returning + /// `nil` if the scalar can't be represented in this encoding. + static func encode(_ content: Unicode.Scalar) -> EncodedScalar? + + /// Converts a scalar from another encoding's representation, returning + /// `nil` if the scalar can't be represented in this encoding. + /// + /// A default implementation of this method will be provided + /// automatically for any conforming type that does not implement one. + static func transcode( + _ content: FromEncoding.EncodedScalar, from _: FromEncoding.Type + ) -> EncodedScalar? + + /// A type that can be used to parse `CodeUnits` into + /// `EncodedScalar`s. + associatedtype ForwardParser : Unicode.Parser + where ForwardParser.Encoding == Self + + /// A type that can be used to parse a reversed sequence of + /// `CodeUnits` into `EncodedScalar`s. + associatedtype ReverseParser : Unicode.Parser + where ReverseParser.Encoding == Self +} +``` + +Parsing `CodeUnits` into `EncodedScalar`s, in either direction, is +done with models of `Unicode.Parser`: + +```swift +extension Unicode { typealias Parser = _UnicodeParser } + +/// Types that separate streams of code units into encoded Unicode +/// scalar values. +public protocol _UnicodeParser { + /// The encoding with which this parser is associated + associatedtype Encoding : Unicode.Encoding + + /// Constructs an instance that can be used to begin parsing `CodeUnit`s at + /// any Unicode scalar boundary. + init() + + /// Parses a single Unicode scalar value from `input`. + mutating func parseScalar( + from input: inout I + ) -> Unicode.ParseResult + where I.Element == Encoding.CodeUnit +} + +extension Unicode { + /// The result of attempting to parse a `T` from some input. + public enum ParseResult { + /// A `T` was parsed successfully + case valid(T) + + /// The input was entirely consumed. + case emptyInput + + /// An encoding error was detected. + /// + /// `length` is the number of underlying code units consumed by this + /// error (when decoding, the length of the longest prefix that + /// could be recognized of a valid encoding sequence). + case error(length: Int) + } +} +``` + +### Higher-Level Unicode Processing + +The Unicode processing APIs proposed here are intentionally extremely +low-level. We have proven that they are sufficient to implement +higher-level constructs, but those designs are still baking and not +yet ready for review. We expect to propose generic `Iterator`, +`Sequence`, and `Collection` views that expose transcoded or segmented +views of arbitrary underlying storage, as separate components in the +`Unicode` namespace. + +## Source compatibility + +Adding collection conformance to `String` should not materially impact source +stability as it is purely additive: Swift 3's `String` interface currently +fulfills all of the requirements for a bidirectional range replaceable +collection. + +Altering `String`'s slicing operations to return a different type is source +breaking. The following mitigating steps are proposed: + + - Add a deprecated subscript operator that will run in Swift 3 compatibility + mode and which will return a `String` not a `Substring`. + + - Add deprecated versions of all current slicing methods to similarly return a + `String`. + +i.e.: + +```swift +extension String { + @available(swift, obsoleted: 4) + subscript(bounds: Range) -> String { + return String(characters[bounds]) + } + + @available(swift, obsoleted: 4) + subscript(bounds: ClosedRange) -> String { + return String(characters[bounds]) + } +} +``` + +In a review of 77 popular Swift projects found on GitHub, these changes +resolved any build issues in the 12 projects that assumed an explicit `String` +type returned from slicing operations. + +Due to the change in internal implementation, this means that these operations +will be _O(n)_ rather than _O(1)_. This is not expected to be a major concern, +based on experiences from a similar change made to Java, but projects will be +able to work around performance issues without upgrading to Swift 4 by +explicitly typing slices as `Substring`, which will call the Swift 4 variant, +and which will be available but not invoked by default in Swift 3 mode. + +The C string interoperability methods outside the ones described in the +detailed design will remain in Swift 3 mode, be deprecated in Swift 4 mode, and +be removed in a subsequent release. `UnicodeCodec` will be similarly deprecated. + +## Effect on ABI stability + +As a fundamental currency type for Swift, it is essential that the +`String` type (and its associated subsequence) is in a good long-term +state before being locked down when Swift declares ABI stability. +Shrinking the size of `String` to be 64 bits is an important part of +the story. As full ABI stablity is not planned for Swift 4, it is +currently unclear when the transition to a 64-bit memory layout will +occur. + +## Effect on API resilience + +Decisions about the API resilience of the `String` type are still to be +determined, but are not adversely affected by this proposal. + +## Alternatives considered + +For a more in-depth discussion of some of the trade-offs in string design, see +the manifesto and associated [evolution thread](https://forums.swift.org/t/strings-in-swift-4/4939). + +This proposal does not yet introduce an implicit conversion from `Substring` to +`String`. The decision on whether to add this will be deferred pending feedback +on the initial implementation. The intention is to make a preview toolchain +available for feedback, including on whether this implicit conversion is +necessary, prior to the release of Swift 4. + + diff --git a/proposals/0164-remove-final-support-in-protocol-extensions.md b/proposals/0164-remove-final-support-in-protocol-extensions.md new file mode 100644 index 0000000000..5e807e4bd3 --- /dev/null +++ b/proposals/0164-remove-final-support-in-protocol-extensions.md @@ -0,0 +1,53 @@ +# Remove final support in protocol extensions + +* Proposal: [SE-0164](0164-remove-final-support-in-protocol-extensions.md) +* Author: [Brian King](https://github.com/KingOfBrian) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0164-remove-final-support-in-protocol-extensions/5687) +* Bug: [SR-1762](https://bugs.swift.org/browse/SR-1762) + +## Introduction +This proposal disallows the `final` keyword when declaring functions in protocol +extensions. + +*Discussion took place on the Swift Evolution mailing list in the [Remove support for final in protocol extensions](https://forums.swift.org/t/draft-remove-support-for-final-in-protocol-extensions/5390) thread.* + +## Motivation + +In the current version of Swift, the `final` keyword does not modify +dispatch behavior in any way, and it does not generate an error message. +This keyword has no use in Swift's current protocol model. Functions in +protocol extensions cannot be overridden and will always use direct dispatch. + +Jordan Rose described the history behind this behavior: +``` +We originally required `final` to signify that there was no +dynamic dispatch going on. Once we started allowing protocol extension +methods to fulfill requirements, it became more confusing than useful. +``` + +## Detailed design + +If adopted, the compiler will flag the use of the `final` keyword on functions +declared within a protocol extension, and emit an error or warning. This +behavior is consistent with `final` use in structures and enumerations. + +## Source compatibility + +This change will impact source compatibility. Existing use of `final` in +protocol extensions will raise a compilation error. The compiler will address +this by source migration and fixits. When running in Swift 3 mode, a warning +will be generated instead of an error. + +## Effect on ABI stability + +This proposal does not affect ABI stability. + +## Effect on API resilience + +This proposal does not affect API resilience. + +## Alternatives considered + +There are no alternatives considered at this time. diff --git a/proposals/0165-dict.md b/proposals/0165-dict.md new file mode 100644 index 0000000000..73897eccda --- /dev/null +++ b/proposals/0165-dict.md @@ -0,0 +1,516 @@ +# Dictionary & Set Enhancements + +* Proposal: [SE-0165](0165-dict.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale][rationale] + +## Introduction + +This proposal comprises a variety of commonly (and less commonly) suggested improvements to the standard library's `Dictionary` type, from merging initializers to dictionary-specific `filter` and `mapValues` methods. The proposed additions to `Dictionary`, and the corresponding changes to `Set`, are detailed in the sections below. + +- Suggested Improvements: + - [Merging initializers and methods](#1-merging-initializers-and-methods) + - [Key-based subscript with default value](#2-key-based-subscript-with-default-value) + - [Dictionary-specific map and filter](#3-dictionary-specific-map-and-filter) + - [Visible `Dictionary` capacity](#4-visible-dictionary-capacity) + - [Grouping sequence elements](#5-grouping-sequence-elements) + - [Relevant changes to `Set`](#6-apply-relevant-changes-to-set) +- [Detailed Design](#detailed-design) + - [Sandbox with Prototype][sandbox] +- [Source Compatibility](#source-compatibility) + +--- + +## 1. Merging initializers and methods + +The `Dictionary` type should allow initialization from a sequence of `(Key, Value)` tuples and offer methods that merge a sequence of `(Key, Value)` tuples into a new or existing dictionary, using a closure to combine values for duplicate keys. + +- [First message of discussion thread](https://forums.swift.org/t/map-like-operation-that-returns-a-dictionary/999) +- [Initial proposal draft](https://forums.swift.org/t/map-like-operation-that-returns-a-dictionary/999/18) +- [Prior standalone proposal (SE-100)](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0100-add-sequence-based-init-and-merge-to-dictionary.md) + +`Array` and `Set` both have initializers that create a new instance from a sequence of elements. The `Array` initializer is useful for converting other sequences and collections to the "standard" collection type, while the `Set` initializer is essential for recovering set operations after performing any functional operations on a set. For example, filtering a set produces a collection without any set operations available. + +```swift +let numberSet = Set(1 ... 100) +let fivesOnly = numberSet.lazy.filter { $0 % 5 == 0 } +``` + +`fivesOnly` is a `LazyFilterCollection>` instead of a `Set`—sending that back through the `Set` sequence initializer restores the expected methods. + +```swift +let fivesOnlySet = Set(numberSet.lazy.filter { $0 % 5 == 0 }) +fivesOnlySet.isSubsetOf(numberSet) // true +``` + +`Dictionary`, on the other hand, has no such initializer, so a similar operation leaves no room to recover dictionary functionality without building a mutable `Dictionary` via iteration or functional methods. These techniques also don't support type inference from the source sequence, increasing verbosity. + +```swift +let numberDictionary = ["one": 1, "two": 2, "three": 3, "four": 4] +let evenOnly = numberDictionary.lazy.filter { (_, value) in + value % 2 == 0 +} + +var viaIteration: [String: Int] = [:] +for (key, value) in evenOnly { + viaIteration[key] = value +} + +let viaReduce: [String: Int] = evenOnly.reduce([:]) { (cumulative, kv) in + var dict = cumulative + dict[kv.key] = kv.value + return dict +} +``` + +Beyond initialization, `Array` and `Set` both also provide a method to add a new block of elements to an existing collection. `Array` provides this via `append(contentsOf:)` for the common appending case or `replaceSubrange(_:with:)` for general inserting or replacing, while the unordered `Set` type lets you pass any sequence to `unionInPlace(_:)` to add elements to an existing set. + +Once again, `Dictionary` has no corresponding API -- looping and adding elements one at a time as shown above is the only way to merge new elements into an existing dictionary. + + +### Proposed solution + +This proposal puts forward two new ways to convert `(Key, Value)` sequences to dictionary form: an initializer and a set of merging APIs that handle input data with duplicate keys. + +#### Sequence-based initializer + +The proposed solution would add a new initializer to `Dictionary` that accepts any sequence of `(Key, Value)` tuple pairs. + +```swift +init( + uniqueKeysWithValues: S) +``` + +With the proposed initializer, creating a `Dictionary` instance from a sequence of key/value pairs is as easy as creating an `Array` or `Set`: + +```swift +let viaProposed = Dictionary(uniqueKeysWithValues: evenOnly) +``` + +Like `Array.init(_:)` and `Set.init(_:)`, this is a full-width initializer. To ensure this, the initializer has a precondition that each key in the supplied sequence is unique, and traps whenever that condition isn't met. When duplicate keys are possible in the input, the merging initializer described in the next section must be used instead. + +The new initializer allows for some convenient uses that aren't currently possible. + +- Initializing from a `DictionaryLiteral` (the type, not an actual literal) + + ```swift + let literal: DictionaryLiteral = ["a": 1, "b": 2, "c": 3, "d": 4] + let dictFromDL = Dictionary(uniqueKeysWithValues: literal) + ``` + +- Converting an array to an indexed dictionary (popular on the thread) + + ```swift + let names = ["Cagney", "Lacey", "Bensen"] + let dict = Dictionary(uniqueKeysWithValues: names.enumerated().map { (i, val) in (i + 1, val) }) + // [2: "Lacey", 3: "Bensen", 1: "Cagney"] + ``` + +- Initializing from a pair of zipped sequences (examples abound) + + ```swift + let letters = "abcdef".characters.lazy.map(String.init) + let dictFromZip = Dictionary(uniqueKeysWithValues: zip(letters, 1...10)) + // ["b": 2, "e": 5, "a": 1, "f": 6, "d": 4, "c": 3] + ``` + + > This particular use is currently blocked by [SR-922](https://bugs.swift.org/browse/SR-922). As a workaround, add `.map {(key: $0, value: $1)}`. + +#### Merging initializer and methods + +Creating a `Dictionary` from a dictional literal checks the keys for uniqueness, trapping on a duplicate. The sequence-based initializer shown above has the same requirement. + +```swift +let duplicates: DictionaryLiteral = ["a": 1, "b": 2, "a": 3, "b": 4] +let letterDict = Dictionary(uniqueKeysWithValues: duplicates) +// error: Duplicate key found: "a" +``` + +Because some use cases can be forgiving of duplicate keys, this proposal includes a second new initializer. This initializer allows the caller to supply, along with the sequence, a combining closure that's called with the old and new values for any duplicate keys. + +```swift +init( + _ keysAndValues: S, + uniquingKeysWith combine: (Value, Value) throws -> Value +) rethrows where S.Iterator.Element == (key: Key, value: Value) +``` + +This example shows how one could keep the first value of all those supplied for a duplicate key. + +```swift +let letterDict2 = Dictionary(duplicates, uniquingKeysWith: { (first, _) in first }) +// ["b": 2, "a": 1] +``` + +Or the largest value for any duplicate keys. + +```swift +let letterDict3 = Dictionary(duplicates, uniquingKeysWith: max) +// ["b": 4, "a": 3] +``` + +At other times the merging initializer could be used to combine values for duplicate keys. Donnacha Oisín Kidney wrote a neat `frequencies()` method for sequences as an example of such a use in the thread. + +```swift +extension Sequence where Iterator.Element: Hashable { + func frequencies() -> [Iterator.Element: Int] { + return Dictionary(self.lazy.map { v in (v, 1) }, uniquingKeysWith: +) + } +} +[1, 2, 2, 3, 1, 2, 4, 5, 3, 2, 3, 1].frequencies() +// [2: 4, 4: 1, 5: 1, 3: 3, 1: 3] +``` + +This proposal also includes new mutating and non-mutating methods for `Dictionary` that merge the contents of a sequence of `(Key, Value)` tuples into an existing dictionary, `merge(_:uniquingKeysWith:)` and `merging(_:uniquingKeysWith:)`. + +```swift +mutating func merge( + _ other: S, + uniquingKeysWith combine: (Value, Value) throws -> Value +) rethrows where S.Iterator.Element == (key: Key, value: Value) + +func merging( + _ other: S, + uniquingKeysWith combine: (Value, Value) throws -> Value +) rethrows -> [Key: Value] where S.Iterator.Element == (key: Key, value: Value) +``` + +As above, there are a wide variety of uses for the merge. + +```swift +// Adding default values +let defaults: [String: Bool] = ["foo": false, "bar": false, "baz": false] +var options: [String: Bool] = ["foo": true, "bar": false] +options.merge(defaults) { (old, _) in old } +// options is now ["foo": true, "bar": false, "baz": false] + +// Summing counts repeatedly +var bugCounts: [String: Int] = ["bees": 9, "ants": 112, ...] +while bugCountingSource.hasMoreData() { + bugCounts.merge(bugCountingSource.countMoreBugs(), uniquingKeysWith: +) +} +``` + +--- + +## 2. Key-based subscript with default value + +Another common challenge with dictionaries is iteratively making changes to key/value pairs that may or may not already be present. For example, to iteratively add count the frequencies of letters in a string, one might write something like the following: + +```swift +let source = "how now brown cow" +var frequencies: [Character: Int] = [:] +for c in source.characters { + if frequencies[c] == nil { + frequencies[c] = 1 + } else { + frequencies[c]! += 1 + } +} +``` + +Testing for `nil` and assigning through the force unwrapping operator are awkward at best for such a common operation. Furthermore, the `Optional` return type of the current keyed subscript complicates efficiencies that could be gained for this type of `modify` action under a future ownership model. + +### Proposed solution + +A keyed subscript with a default value neatly simplifies this usage. Instead of checking for `nil`, one can pass the default value along with the key as a `default` subscript parameter. + +```swift +let source = "how now brown cow" +var frequencies: [Character: Int] = [:] +for c in source.characters { + frequencies[c, default: 0] += 1 +} +``` + +The return type of this subscript is a non-optional `Value`. Note that accessing the subscript as a getter does not store the default value in the dictionary—the following two lines are equivalent: + +```swift +let x = frequencies["a", default: 0] +let y = frequencies["a"] ?? 0 +``` + +--- + +## 3. Dictionary-specific map and filter + +The standard `map` and `filter` methods, while always useful and beloved, aren't ideal when applied to dictionaries. In both cases, the desired end-result is frequently another dictionary instead of an array of key-value pairs—even with the sequence-based initializer proposed above this is an inefficient way of doing things. + +Additionally, the standard `map` method doesn't gracefully support passing a function when transforming only the *values* of a dictionary. The transform function must accept and return key/value pairs, necessitating a custom closure in nearly every case. + +Assuming the addition of a sequence-based initializer, the current `filter` and `map` look like the following: + +```swift +let numbers = ["one": 1, "two": 2, "three": 3, "four": 4] +let evens = Dictionary(numbers.lazy.filter { $0.value % 2 == 0 })! +// ["four": 4, "two": 2] +let strings = Dictionary(numbers.lazy.map { (k, v) in (k, String(v)) })! +// ["three": "3", "four": "4", "one": "1", "two": "2"] +``` + +### Proposed solution + +This proposal adds two new methods for `Dictionary`: + +1. A `mapValues` method that keeps a dictionary's keys intact while transforming the values. Mapping a dictionary's key/value pairs can't always produce a new dictionary, due to the possibility of key collisions, but mapping only the values can produce a new dictionary with the same underlying layout as the original. + + ```swift + let strings = numbers.mapValues(String.init) + // ["three": "3", "four": "4", "one": "1", "two": "2"] + ``` + +2. A `Dictionary`-returning `filter` method. While transforming the keys and values of a dictionary can result in key collisions, filtering the elements of a dictionary can at worst replicate the entire dictionary. + + ```swift + let evens = numbers.filter { $0.value % 2 == 0 } + // ["four": 4, "two": 2] + ``` + +Both of these can be made significantly more efficient than their `Sequence`-sourced counterparts. For example, the `mapValues` method can simply copy the portion of the storage that holds the keys to the new dictionary before transforming the values. + +--- + +## 4. Visible dictionary capacity + +As you add elements to a dictionary, it automatically grows its backing storage as necessary. This reallocation is a significant operation—unlike arrays, where the existing elements can be copied to a new block of storage en masse, every key/value pair must be moved over individually, recalculating the hash value for the key to find its position in the larger backing buffer. + +While dictionaries uses an exponential growth strategy to make this as efficient as possible, beyond the `init(minimumCapacity:)` initializer they do not expose a way to reserve a specific capacity. In addition, adding a key/value pair to a dictionary is guaranteed not to invalidate existing indices as long as the capacity doesn't change, yet we don't provide any way of seeing a dictionary's current or post-addition capacity. + +### Proposed solution + +`Dictionary` should add a `capacity` property and a `reserveCapacity(_:)` method, like those used in range-replaceable collections. The `capacity` of a dictionary is the number of elements it can hold without reallocating a larger backing storage, while calling `reserveCapacity(_:)` allocates a large enough buffer to hold the requested number of elements without reallocating. + +```swift +var numbers = ["one": 1, "two": 2, "three": 3, "four": 4] +numbers.capacity // 6 +numbers.reserveCapacity(20) +numbers.capacity // 24 +``` + +Because hashed collections use extra storage capacity to reduce the likelihood and cost of collisions, the value of the `capacity` property won't be equal to the actual size of the backing storage. Likewise, the capacity after calling `reserveCapacity(_:)` will be at least as large as the argument, but usually larger. (In its current implementation, `Dictionary` always has a power of 2-sized backing storage.) + +--- + +## 5. Grouping sequence elements + +As a final `Dictionary`-related issue, grouping elements in a sequence by some computed key is a commonly requested addition that we can add as part of this omnibus proposal. Call a new `Dictionary(grouping:by:)` initializer with a closure that converts each value in a sequence to a hashable type `T`; the result is a dictionary with keys of type `T` and values of type `[Iterator.Element]`. + +```swift +let names = ["Patti", "Aretha", "Anita", "Gladys"] + +// By first letter +Dictionary(grouping: names, by: { $0.characters.first! }) +// ["P": ["Patti"], "G": ["Gladys"], "A": ["Aretha", "Anita"]] + +// By name length +Dictionary(grouping: names) { $0.utf16.count } +// [5: ["Patti", "Anita"], 6: ["Aretha", "Gladys"]] +``` + +--- + +## 6. Apply relevant changes to `Set` + +As the `Set` and `Dictionary` types are similar enough to share large chunks of their implementations, it makes sense to look at which of these `Dictionary` enhancements can also be applied to `Set`. Of the symbols proposed above, the following additions would also be useful and appropriate for the `Set` type: + +- Add a `Set`-specific `filter` method that returns a new set—this would function essentially as a predicate-based `intersection` method. +- Add a `capacity` property and `reserveCapacity()` method, for the reasons listed above. + + + +## Detailed design + +With the exception of the proposed capacity property and method, the proposed additions to `Dictionary`, `Set`, and `Sequence` are available in [this Swift Sandbox][sandbox]. Note that this prototype is not a proposed implementation; rather a way to try out the behavior of the proposed changes. + +Collected in one place, these are the new APIs for `Dictionary`, `Set`, and `Sequence`: + +```swift +struct Dictionary { + typealias Element = (key: Key, value: Value) + + // existing declarations + + /// Creates a new dictionary using the key/value pairs in the given sequence. + /// If the given sequence has any duplicate keys, the result is `nil`. + init(uniqueKeysWithValues: S) where S.Iterator.Element == Element + + /// Creates a new dictionary using the key/value pairs in the given sequence, + /// using a combining closure to determine the value for any duplicate keys. + init( + _ keysAndValues: S, + uniquingKeysWith combine: (Value, Value) throws -> Value + ) rethrows where S.Iterator.Element == Element + + /// Creates a new dictionary where the keys are the groupings returned by + /// the given closure and the values are arrays of the elements that + /// returned each specific key. + init( + grouping values: S, + by keyForValue: (S.Iterator.Element) throws -> Key + ) rethrows where Value == [S.Iterator.Element] + + /// Merges the key/value pairs in the given sequence into the dictionary, + /// using a combining closure to determine the value for any duplicate keys. + mutating func merge( + _ other: S, + uniquingKeysWith combine: (Value, Value) throws -> Value + ) rethrows where S.Iterator.Element == Element + + /// Returns a new dictionary created by merging the key/value pairs in the + /// given sequence into the dictionary, using a combining closure to determine + /// the value for any duplicate keys. + func merging( + _ other: S, + uniquingKeysWith combine: (Value, Value) throws -> Value + ) rethrows -> [Key: Value] where S.Iterator.Element == Element + + /// Accesses the element with the given key, or the specified default value, + /// if the dictionary doesn't contain the given key. + subscript(key: Key, default defaultValue: Value) -> Value { get set } + + /// Returns a new dictionary containing the key/value pairs that satisfy + /// the given predicate. + func filter(_ isIncluded: (Key, Value) throws -> Bool) rethrows -> [Key: Value] + + /// Returns a new dictionary containing the existing keys and the results of + /// mapping the given closure over the dictionary's values. + func mapValues(_ transform: (Value) throws -> T) rethrows -> [Key: T] + + /// The number of key/value pairs that can be stored by the dictionary without + /// reallocating storage. + var capacity: Int { get } + + /// Ensures that the dictionary has enough storage for `capacity` key/value + /// pairs. + var reserveCapacity(_ capacity: Int) +} + +struct Set { + // existing declarations + + /// Returns a new set containing the elements that satisfy the given predicate. + func filter(_ isIncluded: (Element) throws -> Bool) rethrows -> Set + + /// The number of elements that can be stored by the set without + /// reallocating storage. + var capacity: Int { get } + + /// Ensures that the set has enough storage for `capacity` elements. + var reserveCapacity(_ capacity: Int) +} +``` + +## Alternatives Considered + +- An earlier version of this proposal declared the first merging initializer as failable, returning `nil` when a sequence with duplicate keys was passed. That initializer is really only appropriate when working with known unique keys, however, as there isn't a feasible recovery path when there can be duplicate keys. That leaves two possibilities: + 1. The source input can *never* have duplicate keys, and the programmer has to write `!` or unwrap in some other way, or + 2. The source input *can* have duplicate keys, in which case the programmer should be using `init(merging:mergingValues:)` instead. + +- An earlier version of this proposal included the addition of two new top-level functions to the standard library: `first(_:_:)` and `last(_:_:)`, which return their first and last arguments, respectively. These new functions would be passed instead of a custom closure to a merging method or initializer: + + ```swift + let firstWins = Dictionary(merging: duplicates, mergingValues: first) + // ["b": 2, "a": 1] + + let lastWins = Dictionary(merging: duplicates, mergingValues: last) + // ["b": 4, "a": 3] + ``` + + As an alternative to the `first(_:_:)` and `last(_:_:)` functions, at the cost of three additional overloads for the merging initializer and methods, we could allow users to specify the `useFirst` and `useLast` cases of a `MergeCollisionStrategy` enumeration. + + ```swift + extension Dictionary { + /// The strategy to use when merging a sequence of key-value pairs into a dictionary. + enum MergeCollisionStrategy { + /// If there is more than one instance of a key in the sequence to merge, use + /// only the first value for the dictionary. + case useFirst + /// If there is more than one instance of a key in the sequence to merge, use + /// only the last value for the dictionary. + case useLast + } + + init( + merging keysAndValues: S, + mergingValues strategy: MergeCollisionStrategy + ) + + // other merging overloads + } + ``` + + In use, this overload would look similar to the functional version, but may aid in discoverability: + + ```swift + let firstWins = Dictionary(merging: duplicates, mergingValues: .useFirst) + // ["b": 2, "a": 1] + + let lastWins = Dictionary(merging: duplicates, mergingValues: .useLast) + // ["b": 4, "a": 3] + ``` + +- An earlier version of this proposal included a change to both collections' `remove(at:)` method, such that it would return both the removed element and the next valid index in the iteration sequence. This change lets a programmer write code that would remove elements of a dictionary or a set in place, which can't currently be done with as much efficiency due to index invalidation: + + ```swift + var i = dict.startIndex + while i != dict.endIndex { + if shouldRemove(dict[i]) { + (_, i) = dict.remove(at: i) + } else { + dict.formIndex(after: &i) + } + } + ``` + + This change to `remove(at:)` has been deferred to a later proposal that can better deal with the impact on existing code. + +- The reviewed version of this proposal had different spelling of the merging initializers and methods: + + ```swift + init( + _ keysAndValues: S) + + init( + merging keysAndValues: S, + mergingValues combine: (Value, Value) throws -> Value + ) rethrows where S.Iterator.Element == (key: Key, value: Value) + + mutating func merge( + _ other: S, + mergingValues combine: (Value, Value) throws -> Value + ) rethrows where S.Iterator.Element == (key: Key, value: Value) + + func merged( + with other: S, + mergingValues combine: (Value, Value) throws -> Value + ) rethrows -> [Key: Value] where S.Iterator.Element == (key: Key, value: Value) + ``` + + In addition, the `Dictionary(grouping:by:)` initializer was proposed as a `grouped(by:)` method on the `Sequence` protocol. The [rationale][] for accepting the proposal explains the reasons for these changes. + +## Source compatibility + +All the proposed additions are purely additive and should impose only a minimal source compatibility burden. The addition of same-type returning `filter(_:)` methods to `Dictionary` and `Set` could cause problems if code is relying on the previous behavior of returning an array. For example, the following code would break after this change: + +```swift +let digits = Set(0..<10) +let evens = digits.filter { $0 % 2 == 0 } +functionThatTakesAnArray(evens) +``` + +To mitigate the impact of this on code compiled in Swift 3 mode, `Dictionary` and `Set` will have an additional overload of the `Array`-returning `filter(_:)` that is marked as obsoleted in Swift 4. This will allow Swift 3 code to continue to compile and work as expected: + +```swift +extension Set { + @available(swift, obsoleted: 4) + func filter(_ isIncluded: (Element) throws -> Bool) rethrows -> [Element] + + @available(swift, introduced: 4) + func filter(_ isIncluded: (Element) throws -> Bool) rethrows -> Set +} +``` + +[sandbox]: http://swift.sandbox.bluemix.net/#/repl/58f8e3aee785f75678f428a8 +[rationale]: https://forums.swift.org/t/accepted-se-165-dictionary-set-enhancements/5727 diff --git a/proposals/0166-swift-archival-serialization.md b/proposals/0166-swift-archival-serialization.md new file mode 100644 index 0000000000..53a9a69229 --- /dev/null +++ b/proposals/0166-swift-archival-serialization.md @@ -0,0 +1,2205 @@ +# Swift Archival & Serialization + +* Proposal: [SE-0166](0166-swift-archival-serialization.md) +* Authors: [Itai Ferber](https://github.com/itaiferber), [Michael LeHew](https://github.com/mlehew), [Tony Parker](https://github.com/parkera) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0166-swift-archival-serialization/5780) +* Implementation: [apple/swift#9004](https://github.com/apple/swift/pull/9004) + +## Introduction + +Foundation's current archival and serialization APIs (`NSCoding`, `NSJSONSerialization`, `NSPropertyListSerialization`, etc.), while fitting for the dynamism of Objective-C, do not always map optimally into Swift. This document lays out the design of an updated API that improves the developer experience of performing archival and serialization in Swift. + +Specifically: + +* It aims to provide a solution for the archival of Swift `struct` and `enum` types +* It aims to provide a more type-safe solution for serializing to external formats, such as JSON and plist + +## Motivation + +The primary motivation for this proposal is the inclusion of native Swift `enum` and `struct` types in archival and serialization. Currently, developers targeting Swift cannot participate in `NSCoding` without being willing to abandon `enum` and `struct` types — `NSCoding` is an `@objc` protocol, conformance to which excludes non-`class` types. This can be limiting in Swift because small `enums` and `structs` can be an idiomatic approach to model representation; developers who wish to perform archival have to either forgo the Swift niceties that constructs like `enums` provide, or provide an additional compatibility layer between their "real" types and their archivable types. + +Secondarily, we would like to refine Foundation's existing serialization APIs (`NSJSONSerialization` and `NSPropertyListSerialization`) to better match Swift's strong type safety. From experience, we find that the conversion from the unstructured, untyped data of these formats into strongly-typed data structures is a good fit for archival mechanisms, rather than taking the less safe approach that 3rd-party JSON conversion approaches have taken (described further in an appendix below). + +We would like to offer a solution to these problems without sacrificing ease of use or type safety. + +## Agenda + +This proposal is the first stage of three that introduce different facets of a whole Swift archival and serialization API: + +1. This proposal describes the basis for this API, focusing on the protocols that users adopt and interface with +2. The next stage will propose specific API for new encoders +3. The final stage will discuss how this new API will interop with `NSCoding` as it is today + +[SE-0167](0167-swift-encoders.md) provides stages 2 and 3. + +## Proposed solution + +We will be introducing the `Encodable` and `Decodable` protocols, adoption of which will allow end user types to participate in encoding and decoding: + +```swift +// Codable implies Encodable and Decodable +// If all properties are Codable, protocol implementation is automatically generated by the compiler: +public struct Location : Codable { + public let latitude: Double + public let longitude: Double +} + +public enum Animal : Int, Codable { + case chicken = 1 + case dog + case turkey + case cow +} + +public struct Farm : Codable { + public let name: String + public let location: Location + public let animals: [Animal] +} +``` + +With developer participation, we will offer encoders and decoders (described in [SE-0167](0167-swift-encoders.md), not here) that take advantage of this conformance to offer type-safe serialization of user models: + +```swift +let farm = Farm(name: "Old MacDonald's Farm", + location: Location(latitude: 51.621648, longitude: 0.269273), + animals: [.chicken, .dog, .cow, .turkey, .dog, .chicken, .cow, .turkey, .dog]) +let payload: Data = try JSONEncoder().encode(farm) + +do { + let farm = try JSONDecoder().decode(Farm.self, from: payload) + + // Extracted as user types: + let coordinates = "\(farm.location.latitude, farm.location.longitude)" +} catch { + // Encountered error during deserialization +} +``` + +This gives developers access to their data in a type-safe manner and a recognizable interface. + +## Detailed design + +We will be introducing the following new types to the Swift standard library: + + * `protocol Encodable` & `protocol Decodable`: Adopted by types to opt into archival. Implementation can be synthesized by the compiler in cases where all properties are also `Encodable` or `Decodable` + * `protocol CodingKey`: Adopted by types used as keys for keyed containers, replacing `String` keys with semantic types. Implementation can be synthesized by the compiler in most cases + * `protocol Encoder`: Adopted by types which can take `Encodable` values and encode them into a native format + * `protocol KeyedEncodingContainerProtocol`: Adopted by types which provide a concrete way to store encoded values by `CodingKey`. Types adopting `Encoder` should provide types conforming to `KeyedEncodingContainerProtocol` to vend + * `struct KeyedEncodingContainer`: A concrete type-erased box for exposing `KeyedEncodingContainerProtocol` types; this is a type consumers of the API interact with directly + * `protocol UnkeyedEncodingContainer`: Adopted by types which provide a concrete way to stored encoded values with no keys. Types adopting `Encoder` should provide types conforming to `UnkeyedEncodingContainer` to vend + * `protocol SingleValueEncodingContainer`: Adopted by types which provide a concrete way to store a single encoded value. Types adopting `Encoder` should provide types conforming to `SingleValueEncodingContainer` to vend + * `protocol Decoder`: Adopted by types which can take payloads in a native format and decode `Decodable` values out of them + * `protocol KeyedDecodingContainerProtocol`: Adopted by types which provide a concrete way to retrieve encoded values from storage by `CodingKey`. Types adopting `Decoder` should provide types conforming to `KeyedDecodingContainerProtocol` to vend + * `struct KeyedDecodingContainer`: A concrete type-erased box for exposing `KeyedDecodingContainerProtocol` types; this is a type consumers of the API interact with directly + * `protocol UnkeyedDecodingContainer`: Adopted by types which provide a concrete way to retrieve encoded values from storage with no keys. Types adopting `Decoder` should provide types conforming to `UnkeyedDecodingContainer` to vend + * `protocol SingleValueDecodingContainer`: Adopted by types which provide a concrete way to retrieve a single encoded value from storage. Types adopting `Decoder` should provide types conforming to `SingleValueDecodingContainer` to vend + * `struct CodingUserInfoKey`: A `String RawRepresentable struct` for representing keys to use in `Encoders`' and `Decoders`' `userInfo` dictionaries + +To support user types, we expose the `Encodable` and `Decodable` protocols: + +```swift +/// Conformance to `Encodable` indicates that a type can encode itself to an external representation. +public protocol Encodable { + /// Encodes `self` into the given encoder. + /// + /// If `self` fails to encode anything, `encoder` will encode an empty keyed container in its place. + /// + /// - parameter encoder: The encoder to write data to. + /// - throws: An error if any values are invalid for `encoder`'s format. + func encode(to encoder: Encoder) throws +} + +/// Conformance to `Decodable` indicates that a type can decode itself from an external representation. +public protocol Decodable { + /// Initializes `self` by decoding from `decoder`. + /// + /// - parameter decoder: The decoder to read data from. + /// - throws: An error if reading from the decoder fails, or if read data is corrupted or otherwise invalid. + init(from decoder: Decoder) throws +} + +/// Conformance to `Codable` indicates that a type can convert itself into and out of an external representation. +public typealias Codable = Encodable & Decodable +``` + +By adopting these protocols, user types opt in to this system. + +Structured types (i.e. types which encode as a collection of properties) encode and decode their properties in a keyed manner. Keys are semantic `String`-convertible `enums` which map properties to encoded names. Keys must conform to the `CodingKey` protocol: + +```swift +/// Conformance to `CodingKey` indicates that a type can be used as a key for encoding and decoding. +public protocol CodingKey { + /// The string to use in a named collection (e.g. a string-keyed dictionary). + var stringValue: String { get } + + /// Initializes `self` from a string. + /// + /// - parameter stringValue: The string value of the desired key. + /// - returns: An instance of `Self` from the given string, or `nil` if the given string does not correspond to any instance of `Self`. + init?(stringValue: String) + + /// The int to use in an indexed collection (e.g. an int-keyed dictionary). + var intValue: Int? { get } + + /// Initializes `self` from an integer. + /// + /// - parameter intValue: The integer value of the desired key. + /// - returns: An instance of `Self` from the given integer, or `nil` if the given integer does not correspond to any instance of `Self`. + init?(intValue: Int) +} +``` + +For performance, where relevant, keys may be `Int`-convertible, and `Encoders` may choose to make use of `Ints` over `Strings` as appropriate. Framework types should provide keys which have both for flexibility and performance across different types of `Encoders`. + +By default, `CodingKey` conformance can be derived for `enums` which have no raw type and no associated values, or `String` or `Int` backing: + +```swift +enum Keys1 : CodingKey { + case a // (stringValue: "a", intValue: nil) + case b // (stringValue: "b", intValue: nil) + + // The compiler automatically generates the following: + var stringValue: String { + switch self { + case .a: return "a" + case .b: return "b" + } + } + + init?(stringValue: String) { + switch stringValue { + case "a": self = .a + case "b": self = .b + default: return nil + } + } + + var intValue: Int? { + return nil + } + + init?(intValue: Int) { + return nil + } +} + +enum Keys2 : String, CodingKey { + case c = "foo" // (stringValue: "foo", intValue: nil) + case d // (stringValue: "d", intValue: nil) + + // stringValue, init?(stringValue:), intValue, and init?(intValue:) are generated by the compiler as well +} + +enum Keys3 : Int, CodingKey { + case e = 4 // (stringValue: "e", intValue: 4) + case f // (stringValue: "f", intValue: 5) + case g = 9 // (stringValue: "g", intValue: 9) + + // stringValue, init?(stringValue:), intValue, and init?(intValue:) are generated by the compiler as well +} +``` + +Coding keys which are not `enum`s, have associated values, or have other raw representations must implement these methods manually. + +In addition to automatic `CodingKey` requirement synthesis for `enums`, `Encodable` & `Decodable` requirements can be automatically synthesized for certain types as well: + +1. Types conforming to `Encodable` whose properties are all `Encodable` get an automatically generated `String`-backed `CodingKey` `enum` mapping properties to case names. Similarly for `Decodable` types whose properties are all `Decodable` +2. Types falling into (1) — and types which manually provide a `CodingKey` `enum` (named `CodingKeys`, directly, or via a `typealias`) whose cases map 1-to-1 to `Encodable`/`Decodable` properties by name — get automatic synthesis of `init(from:)` and `encode(to:)` as appropriate, using those properties and keys +3. Types which fall into neither (1) nor (2) will have to provide a custom key type if needed and provide their own `init(from:)` and `encode(to:)`, as appropriate + +This synthesis can always be overridden by a manual implementation of any protocol requirements. Many types will either allow for automatic synthesis of all of codability (1), or provide a custom key subset and take advantage of automatic method synthesis (2). + +### Encoding and Decoding + +Types which are `Encodable` encode their data into a container provided by their `Encoder`: + +```swift +/// An `Encoder` is a type which can encode values into a native format for external representation. +public protocol Encoder { + /// Returns an encoding container appropriate for holding multiple values keyed by the given key type. + /// + /// - parameter type: The key type to use for the container. + /// - returns: A new keyed encoding container. + /// - precondition: May not be called after a prior `self.unkeyedContainer()` call. + /// - precondition: May not be called after a value has been encoded through a previous `self.singleValueContainer()` call. + func container(keyedBy type: Key.Type) -> KeyedEncodingContainer + + /// Returns an encoding container appropriate for holding multiple unkeyed values. + /// + /// - returns: A new empty unkeyed container. + /// - precondition: May not be called after a prior `self.container(keyedBy:)` call. + /// - precondition: May not be called after a value has been encoded through a previous `self.singleValueContainer()` call. + func unkeyedContainer() -> UnkeyedEncodingContainer + + /// Returns an encoding container appropriate for holding a single primitive value. + /// + /// - returns: A new empty single value container. + /// - precondition: May not be called after a prior `self.container(keyedBy:)` call. + /// - precondition: May not be called after a prior `self.unkeyedContainer()` call. + /// - precondition: May not be called after a value has been encoded through a previous `self.singleValueContainer()` call. + func singleValueContainer() -> SingleValueEncodingContainer + + /// The path of coding keys taken to get to this point in encoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } +} + +// Continuing examples from before; below is automatically generated by the compiler if no customization is needed. +public struct Location : Codable { + private enum CodingKeys : CodingKey { + case latitude + case longitude + } + + public func encode(to encoder: Encoder) throws { + // Generic keyed encoder gives type-safe key access: cannot encode with keys of the wrong type. + var container = encoder.container(keyedBy: CodingKeys.self) + + // The encoder is generic on the key -- free key autocompletion here. + try container.encode(latitude, forKey: .latitude) + try container.encode(longitude, forKey: .longitude) + } +} + +public struct Farm : Codable { + private enum CodingKeys : CodingKey { + case name + case location + case animals + } + + public func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(name, forKey: .name) + try container.encode(location, forKey: .location) + try container.encode(animals, forKey: .animals) + } +} +``` + +Similarly, `Decodable` types initialize from data read from their `Decoder`'s container: + +```swift +/// A `Decoder` is a type which can decode values from a native format into in-memory representations. +public protocol Decoder { + /// Returns the data stored in `self` as represented in a container keyed by the given key type. + /// + /// - parameter type: The key type to use for the container. + /// - returns: A keyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a keyed container. + func container(keyedBy type: Key.Type) throws -> KeyedDecodingContainer + + /// Returns the data stored in `self` as represented in a container appropriate for holding values with no keys. + /// + /// - returns: An unkeyed container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not an unkeyed container. + func unkeyedContainer() throws -> UnkeyedDecodingContainer + + /// Returns the data stored in `self` as represented in a container appropriate for holding a single primitive value. + /// + /// - returns: A single value container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a single value container. + func singleValueContainer() throws -> SingleValueDecodingContainer + + /// The path of coding keys taken to get to this point in decoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } +} + +// Continuing examples from before; below is automatically generated by the compiler if no customization is needed. +public struct Location : Codable { + public init(from decoder: Decoder) throws { + var container = try decoder.container(keyedBy: CodingKeys.self) + latitude = try container.decode(Double.self, forKey: .latitude) + longitude = try container.decode(Double.self, forKey: .longitude) + } +} + +public struct Farm : Codable { + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + name = try container.decode(String.self, forKey: .name) + location = try container.decode(Location.self, forKey: .location) + animals = try container.decode([Animal].self, forKey: .animals) + } +} +``` + +### Keyed Containers + +Keyed containers are the primary interface that most `Codable` types interact with for encoding and decoding. Through these, `Codable` types have strongly-keyed access to encoded data by using keys that are semantically correct for the operations they want to express. + +Since semantically incompatible keys will rarely (if ever) share the same key type, it is impossible to mix up key types within the same container (as is possible with `String` keys), and since the type is known statically, keys get autocompletion by the compiler. + +```swift +/// Conformance to `KeyedEncodingContainerProtocol` indicates that a type provides a view into an `Encoder`'s storage and is used to hold the encoded properties of an `Encodable` type in a keyed manner. +/// +/// Encoders should provide types conforming to `KeyedEncodingContainerProtocol` for their format. +public protocol KeyedEncodingContainerProtocol { + associatedtype Key : CodingKey + + /// Encodes the given value for the given key. + /// + /// - parameter value: The value to encode. + /// - parameter key: The key to associate the value with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: T?, forKey key: Key) throws + + /// Encodes the given value for the given key. + /// + /// - parameter value: The value to encode. + /// - parameter key: The key to associate the value with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: Bool?, forKey key: Key) throws + mutating func encode(_ value: Int?, forKey key: Key) throws + mutating func encode(_ value: Int8?, forKey key: Key) throws + mutating func encode(_ value: Int16?, forKey key: Key) throws + mutating func encode(_ value: Int32?, forKey key: Key) throws + mutating func encode(_ value: Int64?, forKey key: Key) throws + mutating func encode(_ value: UInt?, forKey key: Key) throws + mutating func encode(_ value: UInt8?, forKey key: Key) throws + mutating func encode(_ value: UInt16?, forKey key: Key) throws + mutating func encode(_ value: UInt32?, forKey key: Key) throws + mutating func encode(_ value: UInt64?, forKey key: Key) throws + mutating func encode(_ value: Float?, forKey key: Key) throws + mutating func encode(_ value: Double?, forKey key: Key) throws + mutating func encode(_ value: String?, forKey key: Key) throws + + /// Encodes the given object weakly for the given key. + /// + /// For `Encoder`s that implement this functionality, this will only encode the given object and associate it with the given key if it is encoded unconditionally elsewhere in the payload (either previously or in the future). + /// + /// For formats which don't support this feature, the default implementation encodes the given object unconditionally. + /// + /// - parameter object: The object to encode. + /// - parameter key: The key to associate the object with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encodeWeak(_ object: T?, forKey key: Key) throws + + /// The path of coding keys taken to get to this point in encoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } +} + +/// `KeyedEncodingContainer` is a type-erased box for `KeyedEncodingContainerProtocol` types, similar to `AnyCollection` and `AnyHashable`. This is the type which consumers of the API interact with directly. +public struct KeyedEncodingContainer : KeyedEncodingContainerProtocol { + associatedtype Key = K + + /// Initializes `self` with the given container. + /// + /// - parameter container: The container to hold. + init(_ container: Container) where Container.Key == Key + + // + methods from KeyedEncodingContainerProtocol +} + +/// Conformance to `KeyedDecodingContainerProtocol` indicates that a type provides a view into a `Decoder`'s storage and is used to hold the encoded properties of a `Decodable` type in a keyed manner. +/// +/// Decoders should provide types conforming to `KeyedDecodingContainerProtocol` for their format. +public protocol KeyedDecodingContainerProtocol { + associatedtype Key : CodingKey + + /// All the keys the `Decoder` has for this container. + /// + /// Different keyed containers from the same `Decoder` may return different keys here; it is possible to encode with multiple key types which are not convertible to one another. This should report all keys present which are convertible to the requested type. + var allKeys: [Key] { get } + + /// Returns whether the `Decoder` contains a value associated with the given key. + /// + /// The value associated with the given key may be a null value as appropriate for the data format. + /// + /// - parameter key: The key to search for. + /// - returns: Whether the `Decoder` has an entry for the given key. + func contains(_ key: Key) -> Bool + + /// Decodes a value of the given type for the given key. + /// + /// A default implementation is given for these types which calls into the `decodeIfPresent` implementations below. + /// + /// - parameter type: The type of value to decode. + /// - parameter key: The key that the decoded value is associated with. + /// - returns: A value of the requested type, if present for the given key and convertible to the requested type. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the given key or if the value is null. + func decode(_ type: Bool.Type, forKey key: Key) throws -> Bool + func decode(_ type: Int.Type, forKey key: Key) throws -> Int + func decode(_ type: Int8.Type, forKey key: Key) throws -> Int8 + func decode(_ type: Int16.Type, forKey key: Key) throws -> Int16 + func decode(_ type: Int32.Type, forKey key: Key) throws -> Int32 + func decode(_ type: Int64.Type, forKey key: Key) throws -> Int64 + func decode(_ type: UInt.Type, forKey key: Key) throws -> UInt + func decode(_ type: UInt8.Type, forKey key: Key) throws -> UInt8 + func decode(_ type: UInt16.Type, forKey key: Key) throws -> UInt16 + func decode(_ type: UInt32.Type, forKey key: Key) throws -> UInt32 + func decode(_ type: UInt64.Type, forKey key: Key) throws -> UInt64 + func decode(_ type: Float.Type, forKey key: Key) throws -> Float + func decode(_ type: Double.Type, forKey key: Key) throws -> Double + func decode(_ type: String.Type, forKey key: Key) throws -> String + func decode(_ type: T.Type, forKey key: Key) throws -> T + + /// Decodes a value of the given type for the given key, if present. + /// + /// This method returns `nil` if the container does not have a value associated with `key`, or if the value is null. The difference between these states can be distinguished with a `contains(_:)` call. + /// + /// - parameter type: The type of value to decode. + /// - parameter key: The key that the decoded value is associated with. + /// - returns: A decoded value of the requested type, or `nil` if the `Decoder` does not have an entry associated with the given key, or if the value is a null value. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + func decodeIfPresent(_ type: Bool.Type, forKey key: Key) throws -> Bool? + func decodeIfPresent(_ type: Int.Type, forKey key: Key) throws -> Int? + func decodeIfPresent(_ type: Int8.Type, forKey key: Key) throws -> Int8? + func decodeIfPresent(_ type: Int16.Type, forKey key: Key) throws -> Int16? + func decodeIfPresent(_ type: Int32.Type, forKey key: Key) throws -> Int32? + func decodeIfPresent(_ type: Int64.Type, forKey key: Key) throws -> Int64? + func decodeIfPresent(_ type: UInt.Type, forKey key: Key) throws -> UInt? + func decodeIfPresent(_ type: UInt8.Type, forKey key: Key) throws -> UInt8? + func decodeIfPresent(_ type: UInt16.Type, forKey key: Key) throws -> UInt16? + func decodeIfPresent(_ type: UInt32.Type, forKey key: Key) throws -> UInt32? + func decodeIfPresent(_ type: UInt64.Type, forKey key: Key) throws -> UInt64? + func decodeIfPresent(_ type: Float.Type, forKey key: Key) throws -> Float? + func decodeIfPresent(_ type: Double.Type, forKey key: Key) throws -> Double? + func decodeIfPresent(_ type: String.Type, forKey key: Key) throws -> String? + func decodeIfPresent(_ type: T.Type, forKey key: Key) throws -> T? + + /// The path of coding keys taken to get to this point in decoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } +} + +/// `KeyedDecodingContainer` is a type-erased box for `KeyedDecodingContainerProtocol` types, similar to `AnyCollection` and `AnyHashable`. This is the type which consumers of the API interact with directly. +public struct KeyedDecodingContainer : KeyedDecodingContainerProtocol { + associatedtype Key = K + + /// Initializes `self` with the given container. + /// + /// - parameter container: The container to hold. + init(_ container: Container) where Container.Key == Key + + // + methods from KeyedDecodingContainerProtocol +} +``` + +These `encode(_:forKey:)` and `decode(_:forKey:)` overloads give strong, static type guarantees about what is encodable (preventing accidental attempts to encode an invalid type), and provide a list of primitive types which are common to all encoders and decoders that users can rely on. + +When the conditional conformance feature lands in Swift, the ability to express that "a collection of things which are `Codable` is `Codable`" will allow collections (`Array`, `Dictionary`, etc.) to be extended and fall into these overloads as well. + +### Unkeyed Containers + +For some types, when the source and destination of a payload can be guaranteed to agree on the payload layout and format (e.g. in cross-process communication, where both sides agree on the payload format), it may be appropriate to eschew the encoding of keys and encode sequentially, without keys. In this case, a type may choose to make use of an unkeyed container for its properties: + +```swift +/// Conformance to `UnkeyedEncodingContainer` indicates that a type provides a view into an `Encoder`'s storage and is used to hold the encoded properties of an `Encodable` type sequentially, without keys. +/// +/// Encoders should provide types conforming to `UnkeyedEncodingContainer` for their format. +public protocol UnkeyedEncodingContainer { + /// Encodes the given value. + /// + /// - parameter value: The value to encode. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: T?) throws + + /// Encodes the given value. + /// + /// - parameter value: The value to encode. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: Bool?) throws + mutating func encode(_ value: Int?) throws + mutating func encode(_ value: Int8?) throws + mutating func encode(_ value: Int16?) throws + mutating func encode(_ value: Int32?) throws + mutating func encode(_ value: Int64?) throws + mutating func encode(_ value: UInt?) throws + mutating func encode(_ value: UInt8?) throws + mutating func encode(_ value: UInt16?) throws + mutating func encode(_ value: UInt32?) throws + mutating func encode(_ value: UInt64?) throws + mutating func encode(_ value: Float?) throws + mutating func encode(_ value: Double?) throws + mutating func encode(_ value: String?) throws + + /// Encodes the given object weakly. + /// + /// For `Encoder`s that implement this functionality, this will only encode the given object if it is encoded unconditionally elsewhere in the payload (either previously or in the future). + /// + /// For formats which don't support this feature, the default implementation encodes the given object unconditionally. + /// + /// - parameter object: The object to encode. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encodeWeak(_ object: T?) throws + + /// Encodes the elements of the given sequence. + /// + /// A default implementation of these is given in an extension. + /// + /// - parameter sequence: The sequences whose contents to encode. + /// - throws: An error if any of the contained values throws an error. + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Bool + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Int + // ... + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element : Encodable + + /// The path of coding keys taken to get to this point in encoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } +} + +/// Conformance to `UnkeyedDecodingContainer` indicates that a type provides a view into a `Decoder`'s storage and is used to hold the encoded properties of a `Decodable` type sequentially, without keys. +/// +/// Decoders should provide types conforming to `UnkeyedDecodingContainer` for their format. +public protocol UnkeyedDecodingContainer { + /// Returns the number of elements (if known) contained within this container. + var count: Int? { get } + + /// Returns whether there are no more elements left to be decoded in the container. + var isAtEnd: Bool { get } + + /// Decodes a value of the given type. + /// + /// A default implementation is given for these types which calls into the `decodeIfPresent` implementations below. + /// + /// - parameter type: The type of value to decode. + /// - returns: A value of the requested type, if present for the given key and convertible to the requested type. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + /// - throws: `CocoaError.coderValueNotFound` if the encountered encoded value is null, or of there are no more values to decode. + mutating func decode(_ type: Bool.Type) throws -> Bool + mutating func decode(_ type: Int.Type) throws -> Int + mutating func decode(_ type: Int8.Type) throws -> Int8 + mutating func decode(_ type: Int16.Type) throws -> Int16 + mutating func decode(_ type: Int32.Type) throws -> Int32 + mutating func decode(_ type: Int64.Type) throws -> Int64 + mutating func decode(_ type: UInt.Type) throws -> UInt + mutating func decode(_ type: UInt8.Type) throws -> UInt8 + mutating func decode(_ type: UInt16.Type) throws -> UInt16 + mutating func decode(_ type: UInt32.Type) throws -> UInt32 + mutating func decode(_ type: UInt64.Type) throws -> UInt64 + mutating func decode(_ type: Float.Type) throws -> Float + mutating func decode(_ type: Double.Type) throws -> Double + mutating func decode(_ type: String.Type) throws -> String + mutating func decode(_ type: T.Type) throws -> T + + /// Decodes a value of the given type, if present. + /// + /// This method returns `nil` if the container has no elements left to decode, or if the value is null. The difference between these states can be distinguished by checking `isAtEnd`. + /// + /// - parameter type: The type of value to decode. + /// - returns: A decoded value of the requested type, or `nil` if the value is a null value, or if there are no more elements to decode. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + mutating func decodeIfPresent(_ type: Bool.Type) throws -> Bool? + mutating func decodeIfPresent(_ type: Int.Type) throws -> Int? + mutating func decodeIfPresent(_ type: Int8.Type) throws -> Int8? + mutating func decodeIfPresent(_ type: Int16.Type) throws -> Int16? + mutating func decodeIfPresent(_ type: Int32.Type) throws -> Int32? + mutating func decodeIfPresent(_ type: Int64.Type) throws -> Int64? + mutating func decodeIfPresent(_ type: UInt.Type) throws -> UInt? + mutating func decodeIfPresent(_ type: UInt8.Type) throws -> UInt8? + mutating func decodeIfPresent(_ type: UInt16.Type) throws -> UInt16? + mutating func decodeIfPresent(_ type: UInt32.Type) throws -> UInt32? + mutating func decodeIfPresent(_ type: UInt64.Type) throws -> UInt64? + mutating func decodeIfPresent(_ type: Float.Type) throws -> Float? + mutating func decodeIfPresent(_ type: Double.Type) throws -> Double? + mutating func decodeIfPresent(_ type: String.Type) throws -> String? + mutating func decodeIfPresent(_ type: T.Type) throws -> T? + + /// The path of coding keys taken to get to this point in decoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } +} +``` + +Unkeyed encoding is fragile and generally not appropriate for archival without specific format guarantees, so keyed encoding remains the recommended approach (and is why `CodingKey` `enums` are synthesized by default unless otherwise declined). + +### Single Value Containers + +For other types, an array or dictionary container may not even make sense (e.g. values which are `RawRepresentable` as a single primitive value). Those types may encode and decode directly as a single value, instead of requesting an outer container: + +```swift +/// A `SingleValueEncodingContainer` is a container which can support the storage and direct encoding of a single non-keyed value. +public protocol SingleValueEncodingContainer { + /// Encodes a single value of the given type. + /// + /// - parameter value: The value to encode. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + /// - precondition: May not be called after a previous `self.encode(_:)` call. + mutating func encode(_ value: Bool) throws + mutating func encode(_ value: Int) throws + mutating func encode(_ value: Int8) throws + mutating func encode(_ value: Int16) throws + mutating func encode(_ value: Int32) throws + mutating func encode(_ value: Int64) throws + mutating func encode(_ value: UInt) throws + mutating func encode(_ value: UInt8) throws + mutating func encode(_ value: UInt16) throws + mutating func encode(_ value: UInt32) throws + mutating func encode(_ value: UInt64) throws + mutating func encode(_ value: Float) throws + mutating func encode(_ value: Double) throws + mutating func encode(_ value: String) throws +} + +/// A `SingleValueDecodingContainer` is a container which can support the storage and direct decoding of a single non-keyed value. +public protocol SingleValueDecodingContainer { + /// Decodes a single value of the given type. + /// + /// - parameter type: The type to decode as. + /// - returns: A value of the requested type. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value cannot be converted to the requested type. + func decode(_ type: Bool.Type) throws -> Bool + func decode(_ type: Int.Type) throws -> Int + func decode(_ type: Int8.Type) throws -> Int8 + func decode(_ type: Int16.Type) throws -> Int16 + func decode(_ type: Int32.Type) throws -> Int32 + func decode(_ type: Int64.Type) throws -> Int64 + func decode(_ type: UInt.Type) throws -> UInt + func decode(_ type: UInt8.Type) throws -> UInt8 + func decode(_ type: UInt16.Type) throws -> UInt16 + func decode(_ type: UInt32.Type) throws -> UInt32 + func decode(_ type: UInt64.Type) throws -> UInt64 + func decode(_ type: Float.Type) throws -> Float + func decode(_ type: Double.Type) throws -> Double + func decode(_ type: String.Type) throws -> String +} + +// Continuing example from before; below is automatically generated by the compiler if no customization is needed. +public enum Animal : Int, Codable { + public func encode(to encoder: Encoder) throws { + // Encode as a single value; no keys. + try encoder.singleValueContainer().encode(self.rawValue) + } + + public init(from decoder: Decoder) throws { + // Decodes as a single value; no keys. + let intValue = try decoder.singleValueContainer().decode(Int.self) + if let value = Self(rawValue: intValue) { + self = value + } else { + throw CocoaError.error(.coderReadCorrupt) + } + } +} +``` + +In the example given above, since `Animal` uses a single value container, `[.chicken, .dog, .cow, .turkey, .dog, .chicken, .cow, .turkey, .dog]` would encode directly as `[1, 2, 4, 3, 2, 1, 4, 3, 2]`. + +### Nesting + +In practice, some types may also need to control how data is nested within their container, or potentially nest other containers within their container. Keyed containers allow this by returning nested containers of differing types: + +```swift +// Continuing from before +public protocol KeyedEncodingContainerProtocol { + /// Stores a keyed encoding container for the given key and returns it. + /// + /// - parameter keyType: The key type to use for the container. + /// - parameter key: The key to encode the container for. + /// - returns: A new keyed encoding container. + mutating func nestedContainer(keyedBy keyType: NestedKey.Type, forKey key: Key) -> KeyedEncodingContainer + + /// Stores an unkeyed encoding container for the given key and returns it. + /// + /// - parameter key: The key to encode the container for. + /// - returns: A new unkeyed encoding container. + mutating func nestedUnkeyedContainer(forKey key: Key) -> UnkeyedEncodingContainer +} + +public protocol KeyedDecodingContainerProtocol { + /// Returns the data stored for the given key as represented in a container keyed by the given key type. + /// + /// - parameter type: The key type to use for the container. + /// - parameter key: The key that the nested container is associated with. + /// - returns: A keyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a keyed container. + func nestedContainer(keyedBy type: NestedKey.Type, forKey key: Key) throws -> KeyedDecodingContainer + + /// Returns the data stored for the given key as represented in an unkeyed container. + /// + /// - parameter key: The key that the nested container is associated with. + /// - returns: An unkeyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not an unkeyed container. + func nestedUnkeyedContainer(forKey key: Key) throws -> UnkeyedDecodingContainer +} +``` + +This can be common when coding against specific external data representations: + +```swift +// User type for interfacing with a specific JSON API. JSON API expects encoding as {"id": ..., "properties": {"name": ..., "timestamp": ...}}. Swift type differs from encoded type, and encoding needs to match a spec: +struct Record : Codable { + // We care only about these values from the JSON payload + let id: Int + let name: String + let timestamp: Double + + // ... + + private enum Keys : CodingKey { + case id + case properties + } + + private enum PropertiesKeys : CodingKey { + case name + case timestamp + } + + public func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: Keys.self, type: .dictionary) + try container.encode(id, forKey: .id) + + // Set a dictionary for the "properties" key + let nested = container.nestedContainer(keyedBy: PropertiesKeys.self, forKey: .properties) + try nested.encode(name, forKey: .name) + try nested.encode(timestamp, forKey: .timestamp) + } + + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: Keys.self) + id = try container.decode(Int.self, forKey: .id) + + let nested = try container.nestedContainer(keyedBy: PropertiesKeys.self, forKey: .properties) + name = try nested.decode(String.self, forKey: .name) + timestamp = try nested.decode(Double.self, forKey: .timestamp) + } +} +``` + +Unkeyed containers allow for the same types of nesting: + +```swift +// Continuing from before +public protocol UnkeyedEncodingContainer { + /// Encodes a nested container keyed by the given type and returns it. + /// + /// - parameter keyType: The key type to use for the container. + /// - returns: A new keyed encoding container. + mutating func nestedContainer(keyedBy keyType: NestedKey.Type) -> KeyedEncodingContainer + + /// Encodes an unkeyed encoding container and returns it. + /// + /// - returns: A new unkeyed encoding container. + mutating func nestedUnkeyedContainer() -> UnkeyedEncodingContainer +} + +public protocol UnkeyedDecodingContainer { + /// Decodes a nested container keyed by the given type. + /// + /// - parameter type: The key type to use for the container. + /// - returns: A keyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a keyed container. + mutating func nestedContainer(keyedBy type: NestedKey.Type) throws -> KeyedDecodingContainer + + /// Decodes an unkeyed nested container. + /// + /// - returns: An unkeyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not an unkeyed container. + mutating func nestedUnkeyedContainer() throws -> UnkeyedDecodingContainer +} +``` + +### Dynamic Context-Based Behavior + +In some cases, types may need context in order to decide on their external representation. Some types may choose a different representation based on the encoding format that they are being read from or written to, and others based on other runtime contextual information. To facilitate this, `Encoders` and `Decoders` expose user-supplied context for consumption: + +```swift +/// Represents a user-defined key for providing context for encoding and decoding. +public struct CodingUserInfoKey : RawRepresentable, Hashable { + typealias RawValue = String + let rawValue: String + init?(rawValue: String) +} + +// Continuing from before: +public protocol Encoder { + /// Any contextual information set by the user for encoding. + var userInfo: [CodingUserInfoKey : Any] { get } +} + +public protocol Decoder { + /// Any contextual information set by the user for decoding. + var userInfo: [CodingUserInfoKey : Any] { get } +} +``` + +Consuming types may then support setting contextual information to inform their encoding and decoding: + +```swift +public struct Person : Encodable { + public static let codingUserInfoKey = CodingUserInfoKey("com.foocorp.person.codingUserInfoKey") + + public struct UserInfo { + let shouldEncodePrivateFields: Bool + // ... + } + + func encode(to encoder: Encoder) throws { + if let context = encoder.userInfo[Person.codingUserInfoKey] as? Person.UserInfo { + if context.shouldEncodePrivateFields { + // Do something special. + } + } + + // Fall back to default. + } +} + +let encoder = ... +encoder.userInfo[Person.codingUserInfoKey] = Person.UserInfo(...) +let data = try encoder.encode(person) +``` + +`Encoders` and `Decoders` may choose to expose contextual information about their configuration as part of the context as well if necessary. + +### Inheritance + +Inheritance in this system is supported much like it is with `NSCoding` — on encoding, objects which inherit from a type that is `Encodable` encode `super` using their encoder, and pass a decoder to `super.init(from:)` on decode if they inherit from a type that is `Decodable`. With the existing `NSCoding` API, this is most often done like so, by convention: + +```objc +- (void)encodeWithCoder:(NSCoder *)encoder { + [super encodeWithCoder:encoder]; + // ... encode properties +} + +- (instancetype)initWithCoder:(NSCoder *)decoder { + if ((self = [super initWithCoder:decoder])) { + // ... decode properties + } + + return self; +} +``` + +In practice, this approach means that the properties of `self` and the properties of `super` get encoded into the same container: if `self` encodes values for keys `"a"`, `"b"`, and `"c"`, and `super` encodes `"d"`, `"e"`, and `"f"`, the resulting object is encoded as `{"a": ..., "b": ..., "c": ..., "d": ..., "e": ..., "f": ...}`. This approach has two drawbacks: + +1. Things which `self` encodes may overwrite `super`'s (or vice versa, depending on when `-[super encodeWithCoder:]` is called +2. `self` and `super` may not encode into different container types (e.g. `self` in a sequential fashion, and `super` in a keyed fashion) + +The second point is not an issue for `NSKeyedArchiver`, since all values encode with keys (sequentially coded elements get autogenerated keys). This proposed API, however, allows for `self` and `super` to explicitly request conflicting containers (`.array` and `.dictionary`, which may not be mixed, depending on the data format). + +To remedy both of these points, we adopt a new convention for inheritance-based coding — encoding `super` as a sub-object of `self`: + +```swift +public class MyCodable : SomethingCodable { + public func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + // ... encode some properties + + // superEncoder() gives `super` a nested container to encode into (for + // a predefined key). + try super.encode(to: container.superEncoder()) + } + + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + // ... decode some properties + + // Allow `super` to decode from the nested container. + try super.init(from: container.superDecoder()) + } +} +``` + +If a shared container is desired, it is still possible to call `super.encode(to: encoder)` and `super.init(from: decoder)`, but we recommend the safer containerized option. + +`superEncoder()` and `superDecoder()` are provided on containers to provide handles to nested containers for `super` to use. + +```swift +// Continuing from before +public protocol KeyedEncodingContainerProtocol { + /// Stores a new nested container for the default `super` key and returns a new `Encoder` instance for encoding `super` into that container. + /// + /// Equivalent to calling `superEncoder(forKey:)` with `Key(stringValue: "super", intValue: 0)`. + /// + /// - returns: A new `Encoder` to pass to `super.encode(to:)`. + mutating func superEncoder() -> Encoder + + /// Stores a new nested container for the given key and returns a new `Encoder` instance for encoding `super` into that container. + /// + /// - parameter key: The key to encode `super` for. + /// - returns: A new `Encoder` to pass to `super.encode(to:)`. + mutating func superEncoder(forKey key: Key) -> Encoder +} + +public protocol KeyedDecodingContainerProtocol { + /// Returns a `Decoder` instance for decoding `super` from the container associated with the default `super` key. + /// + /// Equivalent to calling `superDecoder(forKey:)` with `Key(stringValue: "super", intValue: 0)`. + /// + /// - returns: A new `Decoder` to pass to `super.init(from:)`. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the default `super` key, or if the stored value is null. + func superDecoder() throws -> Decoder + + /// Returns a `Decoder` instance for decoding `super` from the container associated with the given key. + /// + /// - parameter key: The key to decode `super` for. + /// - returns: A new `Decoder` to pass to `super.init(from:)`. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the given key, or if the stored value is null. + func superDecoder(forKey key: Key) throws -> Decoder +} + +public protocol UnkeyedEncodingContainer { + /// Encodes a nested container and returns an `Encoder` instance for encoding `super` into that container. + /// + /// - returns: A new `Encoder` to pass to `super.encode(to:)`. + mutating func superEncoder() -> Encoder +} + +public protocol UnkeyedDecodingContainer { + /// Decodes a nested container and returns a `Decoder` instance for decoding `super` from that container. + /// + /// - returns: A new `Decoder` to pass to `super.init(from:)`. + /// - throws: `CocoaError.coderValueNotFound` if the encountered encoded value is null, or of there are no more values to decode. + mutating func superDecoder() throws -> Decoder +} +``` + +### Primitive `Codable` Conformance + +The encoding container types offer overloads for working with and processing the API's primitive types (`String`, `Int`, `Double`, etc.). However, for ease of implementation (both in this API and others), it can be helpful for these types to conform to `Codable` themselves. Thus, along with these overloads, we will offer `Codable` conformance on these types: + +```swift +extension Bool : Codable { + public init(from decoder: Decoder) throws { + self = try decoder.singleValueContainer().decode(Bool.self) + } + + public func encode(to encoder: Encoder) throws { + try encoder.singleValueContainer().encode( self) + } +} + +// Repeat for others... +``` + +This conformance allows one to write functions which accept `Codable` types without needing specific overloads for the fifteen primitive types as well. + +Since Swift's function overload rules prefer more specific functions over generic functions, the specific overloads are chosen where possible (e.g. `encode("Hello, world!", forKey: .greeting)` will choose `encode(_: String, forKey: Key)` over `encode(_: T, forKey: Key)`). + +#### Additional Extensions + +Along with the primitive `Codable` conformance above, extensions on `Codable` `RawRepresentable` types whose `RawValue` is a primitive types will provide default implementations for encoding and decoding: + +```swift +public extension RawRepresentable where RawValue == Bool, Self : Codable { + public init(from decoder: Decoder) throws { + let decoded = try decoder.singleValueContainer().decode(RawValue.self) + guard let value = Self(rawValue: decoded) else { + throw CocoaError.error(.coderReadCorrupt) + } + + self = value + } + + public func encode(to encoder: Encoder) throws { + try encoder.singleValueContainer().encode(self.rawValue) + } +} + +// Repeat for others... +``` + +This allows for trivial `Codable` conformance of `enum` types (and manual `RawRepresentable` implementations) with primitive backing. + +## Source compatibility + +This proposal is additive — existing code will not have to change due to this API addition. This implementation can be made available in both Swift 4 and the Swift 3 compatibility mode. + +## Effect on ABI stability + +The addition of this API will not be an ABI-breaking change. However, this will add limitations for changes in future versions of Swift, as parts of the API will have to remain unchanged between versions of Swift (barring some additions, discussed below). + +## Effect on API resilience + +Much like new API added to the standard library, once added, many changes to this API will be ABI- and source-breaking changes. In particular, changes which change the types or names of methods or arguments, add required methods on protocols or classes, or remove supplied default implementations will break client behavior. + +The following types may not have methods added to them without providing default implementations: + +* `Encodable` +* `Decodable` +* `CodingKey` +* `Encoder` +* `KeyedEncodingContainerProtocol` + * `KeyedEncodingContainer` +* `UnkeyedEncodingContainer` +* `SingleValueEncodingContainer` +* `Decoder` +* `KeyedDecodingContainerProtocol` + * `KeyedDecodingContainer` +* `UnkeyedEncodingContainer` +* `SingleValueDecodingContainer` + +Various extensions to Swift primitive types (`Bool`, `Int`, `Double`, etc.) and to `RawRepresentable` types (`where RawValue == Bool`, `== Int`, `== Double`, etc.) may also not be removed. + +In general, changes to the proposed types will be restricted as described in the [library evolution document](https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst) in the Swift repository. + +## Alternatives considered + +The following are a few of the more notable approaches considered for the problem: + +1. Leverage the existing `NSCoding` implementation by adding support for `struct` and `enum` types, either through `NSCoding` itself, or through a similar protocol. + + * Although technically feasible, this can feel like a "missed opportunity" for offering something better tuned for Swift. This approach would also not offer any additional integration with `JSONSerialization` and `PropertyListSerialization`, unless JSON and plist archivers were added to offer support. + +2. The following type-erased, declarative approach: + + ```swift + // Similar hack to AnyHashable; these wrap values which have not yet been + // encoded, or not yet decoded. + struct AnyEncodable { ... } + struct AnyDecodable { ... } + + protocol CodingPrimitive {} + protocol PrimitiveCodable { /* same as above */ } + + protocol OrderedCodable { + init(from: [AnyDecodable]) throws + var encoded: [AnyEncodable] { get } + } + + protocol KeyedCodable { + init(from: [String: AnyDecodable]) throws + var encoded: [String: AnyEncodable] { get } + } + + // Same as above + protocol OrderedEncoder { ... } + protocol OrderedDecoder { ... } + protocol KeyedEncoder { ... } + protocol KeyedDecoder { ... } + + // Sample: + struct Location: OrderedCodable { + let latitude: Double + let longitude: Double + + init(from array: [AnyDecodable]) throws { + guard array.count == 2 else { /* throw */ } + + // These `.as()` calls perform the actual decoding, and fail by + // throwing an error. + let latitude = try array[0].as(Double.self) + let longitude = try array[1].as(Double.self) + try self.init(latitutde: latitude, longitude: longitude) + } + + var encoded: [AnyEncodable] { + // With compiler support, AnyEncodable() can be automatic. + return [AnyEncodable(latitude), AnyEncodable(longitude)] + } + } + + struct Farm: KeyedCodable { + let name: String + let location: Location + let animals: [Animal] + + init(from dictionary: [String: AnyDecodable]) throws { + guard let name = dictionary["name"], + let location = dictionary["location"], + let animals = dictionary["animals"] else { /* throw */ } + + self.name = try name.as(String.self) + self.location = try location.as(Location.self) + self.animals = try animals.asArrayOf(Animal.self) + } + + var encoded: [String: AnyEncodable] { + // Similarly, AnyEncodable() should go away. + return ["name": AnyEncodable(name), + "location": AnyEncodable(location), + "animals": AnyEncodable(animals)] + } + } + ``` + + Although the more declarative nature of this approach can be appealing, this suffers from the same problem that `JSONSerialization` currently does: as-casting. Getting an intermediate type-erased value requires casting to get a "real" value out. Doing this with an `as?`-cast requires compiler support to interject code to decode values of a given type out of their type-erased containers (similar to what happens today with `AnyHashable`). If the user requests a value of a different type than what is stored, however, the `as?`-cast will fail by returning `nil` — there is no meaningful way to report the failure. Getting the code to `throw` in cases like this requires methods on `AnyDecodable` (as shown above), but these can be confusing (when should you use `.as()` and when should you use `as?`?). + + Modifications can be made to improve this: + + ```swift + protocol OrderedCodable { + // AnyDecodable can wrap anything, including [AnyDecodable]; unwrapping + // these can be tedious, so we want to give default implementations + // that do this. + // Default implementations for these are given in terms of the + // initializer below. + init?(from: AnyDecodable?) throws + init(from: AnyDecodable) throws + + init(from: [AnyDecodable]) throws + var encoded: [AnyEncodable] { get } + } + + protocol KeyedCodable { + // AnyDecodable can wrap anything, including [String: AnyDecodable]; + // unwrapping these can be tedious, so we want to give default + // implementations that do this. + // Default implementations for these are given in terms of the + // initializer below. + init?(from: AnyDecodable?) throws + init(from: AnyDecodable) throws + + init(from: [String: AnyDecodable]) throws + var encoded: [String: AnyEncodable] { get } + } + + // Sample: + struct Location: OrderedCodable { + // ... + init(from array: [AnyDecodable]) throws { + guard array.count == 2 else { /* throw */ } + let latitude = try Double(from: array[0]) + let longitude = try Double(from: array[1]) + try self.init(latitude: latitude, longitude: longitude) + } + // ... + } + + struct Farm: KeyedCodable { + // ... + init(from dictionary: [String: AnyDecodable]) throws { + guard let name = try String(from: dictionary["name"]), + let Location = try Location(from: dictionary["location"]) + let animals = try [Animal](from: dictionary["animals"]) else { + /* throw */ + } + + self.name = name + self.location = location + self.animals = animals + } + // ... + } + ``` + + By providing the new initializer methods, we can perform type casting via initialization, rather than by explicit casts. This pushes the `.as()` calls into the Swift primitives (`CodingPrimitive`s, `Array`, `Dictionary`), hiding them from end users. However, this has a different problem, namely that by offering the same type-erased initializers, `OrderedCodable` and `KeyedCodable` now conflict, and it is impossible to conform to both. + + The declarative benefits here are not enough to outweigh the fact that this does not effectively remove the need to `as?`-cast. + +3. The following approach, which relies on compiler code generation: + + ```swift + protocol Codable { + /// `EncodedType` is an intermediate representation of `Self` -- it has + /// the properties from `Self` that need to be archived and unarchived + /// (and performs that archival work), but represents at type that is + /// not yet domain-validated like `self` is. + associatedtype EncodedType: CodingRepresentation + init(from encoded: EncodedType) + var encoded: EncodedType { get } + } + + protocol CodingPrimitive {} + protocol CodingRepresentation {} + protocol PrimitiveCodingRepresentation: CodingRepresentation { + /* Similar to PrimitiveCodable above */ + } + + protocol OrderedCodingRepresentation: CodingRepresentation { + /* Similar to OrderedCodable above */ + } + + protocol KeyedCodingRepresentation: CodingRepresentation { + /* Similar to KeyedCodable above */ + } + + // Sample: + struct Location : Codable { + let latitude: Double + let longitude: Double + + // --------------------------------------------------------------------- + // Ideally, the following could be generated by the compiler (in simple + // cases; developers can choose to implement subsets of the following + // code based on where they might need to perform customizations. + init(from encoded: Encoded) throws { + latitude = encoded.latitude + longitude = encoded.longitude + } + + var encoded: Encoded { + return Encoded(self) + } + + // Keyed coding is the default generated by the compiler; consumers who + // want OrderedCodingRepresentation need to provide their own encoded + // type. + struct Encoded: OrderedCodingRepresentation { + let latitude: String + let longitude: String + + init(_ location: Location) { + latitude = location.latitude + longitude = location.longitude + } + + init(from: KeyedDecoder) throws { ... } + func encode(to: KeyedEncoder) { ... } + } + // --------------------------------------------------------------------- + } + ``` + + This approach separates encoding and decoding into constituent steps: + + 1. Converting `self` into a representation fit for encoding (`EncodedType`, particularly if `EncodedType` has different properties from `Self`) + 2. Converting that representation into data (`encode(into:)`) + 3. Converting arbitrary bytes into validated types (`EncodedType.init(from:)`) + 4. Converting validated data and types into a domain-validated value (`Self.init(from:)`). + + These steps can be generated by the compiler in simple cases, with gradations up to the developer providing implementations for all of these. With this approach, it would be possible to: + + 1. Have a type where all code generation is left to the compiler + 2. Have a type where `EncodedType` is autogenerated, but the user implements `init(from:)` (allowing for custom domain validation on decode) or `var encoded`, or both + 3. Have a type where the user supplies `EncodedType`, `Self.init(from:)`, and `var encoded`, but the compiler generates `EncodedType.init(from:)` and `EncodedType.encode(into:)`. This allows the user to control what properties `EncodedType` has (or control its conformance to one of the `CodingRepresentation` types) without having to perform the actual `encode` and `decode` calls + 4. Have a type where the user supplies everything, giving them full control of encoding and decoding (for implementing archive versioning and other needs) + + While cases 1 and 2 save on boilerplate, types which need to be customized have significantly more boilerplate to write by hand. + +4. The following approach, which delineates between keyed encoding (with `String` keys) and ordered encoding (this is the approach proposed in v1 and v2 of this proposal): + + ```swift + protocol PrimitiveCodable { + associatedtype Atom: CodingAtom + var atomValue: Atom { get } + init(atomValue value: Atom) + } + + protocol OrderedCodable { + init(from decoder: OrderedDecoder) throws + func encode(into encoder: OrderedEncoder) + } + + protocol KeyedCodable { + init(from decoder: KeyedDecoder) throws + func encode(into encoder: KeyedEncoder) + } + + protocol OrderedEncoder { + func encode(_ value: Value?) where Value: CodingAtom + func encode(_ value: Value?) where Value: PrimitiveCodable + func encode(_ value: Value?) where Value: OrderedCodable + func encode(_ value: Value?) where Value: KeyedCodable + func encode(_ value: Value?) where Value: OrderedCodable & KeyedCodable + } + + protocol OrderedDecoder { + var count: Int { get } + func decode(_ type: Value.Type) throws -> Value? where Value: CodingAtom + func decode(_ type: Value.Type) throws -> Value? where Value: PrimitiveCodable + func decode(_ type: Value.Type) throws -> Value? where Value: OrderedCodable + func decode(_ type: Value.Type) throws -> Value? where Value: KeyedCodable + func decode(_ type: Value.Type) throws -> Value? where Value: OrderedCodable & KeyedCodable + } + + protocol KeyedEncoder { + func encode(_ value: Value?, forKey key: String) where Value: CodingPrimitive + func encode(_ value: Value?, forKey key: String) where Value: PrimitiveCodable + func encode(_ value: Value?, forKey key: String) where Value: OrderedCodable + func encode(_ value: Value?, forKey key: String) where Value: KeyedCodable + func encode(_ value: Value?, forKey key: String) where Value: OrderedCodable & KeyedCodable + } + + protocol KeyedDecoder { + var allKeys: [String] { get } + func hasValue(forKey key: String) -> Bool + func decode(_ type: Value.Type, forKey key: String) throws -> Value? where Value: CodingPrimitive + func decode(_ type: Value.Type, forKey key: String) throws -> Value? where Value: PrimitiveCodable + func decode(_ type: Value.Type, forKey key: String) throws -> Value? where Value: OrderedCodable + func decode(_ type: Value.Type, forKey key: String) throws -> Value? where Value: KeyedCodable + func decode(_ type: Value.Type, forKey key: String) throws -> Value? where Value: OrderedCodable & KeyedCodable + } + ``` + + Although this semantically separates between different types of encoding, the multiple protocols can be confusing, and it is not immediately apparent which to adopt and use. This also specifically calls out a difference between string-keyed and non-keyed coding, which is unnecessary. + +5. A closure-based version of the current approach which scopes keyed encoders/decoders to call sites via closures: + + ```swift + protocol Encoder { + func encode(as value: Bool) throws + // ... + + func with(keys type: Key.Type, _ block: (KeyedEncoder) throws -> Void) rethrows + // ... + } + + internal struct Record : Codable { + let id: Int + let name: String + let timestamp: Double + + // ... + + public func encode(into encoder: Encoder) throws { + try encoder.with(keys: Keys.self) { keyedEncode in + try keyedEncode.encode(id, forKey: .id) + + try keyedEncode.encode(.dictionary, forKey: .properties, keys: PropertiesKeys.self) { properties in + try properties.encode(name, forKey: .name) + try properties.encode(timestamp, forKey: .timestamp) + } + } + } + } + ``` + + However, this cannot currently be applied to decoding: + + ```swift + public init(from decoder: Decoder) throws { + // This closure implicitly references self. Since Swift has no + // guarantees that this closure will get called exactly once, self must + // be fully initialized before this call. + // + // This would require all instance variables to be vars with default + // values. + try decoder.with(keys: Keys.self) { keyedDecoder in + id = try keyedDecoder.decode(Int.self, forKey: .id) + // ... + } + } + ``` + + Although it is not currently possible to initialize `self` within a closure in Swift, this may be added in the future as annotations make these guarantees possible. + +6. A previous approach similar to the current approach with single value encode calls available directly on `Encoder`, and a `KeyedEncoder` type instead of `KeyedEncodingContainer`: + + ```swift + public protocol Encoder { + func keyed(by: Key.Type) throws -> KeyedEncoder + + func encode(as: Bool) throws + func encode(as: Int) throws + func encode(as: Int8) throws + func encode(as: Int16) throws + func encode(as: Int32) throws + // ... + } + + public class KeyedEncoder { + // Identical to KeyedEncodingContainer + } + ``` + +# Appendix + +## `JSONSerialization` Friction and Third-Party Solutions (Motivation) + +The following example usage of `JSONSerialization` is taken from the README of [SwiftyJSON](https://github.com/swiftyjson/swiftyjson), a third-party library that many developers use to interface with JSON models: + +```swift +if let statusesArray = try? JSONSerialization.jsonObject(with: data, options: .allowFragments) as? [[String: Any]], + let user = statusesArray[0]["user"] as? [String: Any], + let username = user["name"] as? String { + // Finally we got the username +} +``` + +SwiftyJSON attempts to elide the verbosity of casting by offering the following solution instead: + +```swift +let json = JSON(data: dataFromNetworking) +if let userName = json[0]["user"]["name"].string { + // Now you got your value +} +``` + +This friction is not necessarily a design flaw in the API, simply a truth of interfacing between JavaScript and JSON's generally untyped, unstructured contents, and Swift's strict typing. Some libraries, like SwiftyJSON, do this at the cost of type safety; others, like ObjectMapper and Argo below, maintain type safety by offering archival functionality for JSON types: + +```swift +// Taken from https://github.com/Hearst-DD/ObjectMapper +class User: Mappable { + var username: String? + var age: Int? + var weight: Double! + var array: [AnyObject]? + var dictionary: [String : AnyObject] = [:] + var bestFriend: User? // Nested User object + var friends: [User]? // Array of Users + var birthday: NSDate? + + required init?(map: Map) { + + } + + // Mappable + func mapping(map: Map) { + username <- map["username"] + age <- map["age"] + weight <- map["weight"] + array <- map["arr"] + dictionary <- map["dict"] + bestFriend <- map["best_friend"] + friends <- map["friends"] + birthday <- (map["birthday"], DateTransform()) + } +} + +struct Temperature: Mappable { + var celsius: Double? + var fahrenheit: Double? + + init?(map: Map) { + + } + + mutating func mapping(map: Map) { + celsius <- map["celsius"] + fahrenheit <- map["fahrenheit"] + } +} +``` + +or the more functional + +```swift +// Taken from https://github.com/thoughtbot/Argo +struct User { + let id: Int + let name: String + let email: String? + let role: Role + let companyName: String + let friends: [User] +} + +extension User: Decodable { + static func decode(j: JSON) -> Decoded { + return curry(User.init) + <^> j <| "id" + <*> j <| "name" + <*> j <|? "email" // Use ? for parsing optional values + <*> j <| "role" // Custom types that also conform to Decodable just work + <*> j <| ["company", "name"] // Parse nested objects + <*> j <|| "friends" // parse arrays of objects + } +} + +// Wherever you receive JSON data: +let json: Any? = try? NSJSONSerialization.JSONObjectWithData(data, options: []) +if let j: Any = json { + let user: User? = decode(j) +} +``` + +There are tradeoffs made here as well. ObjectMapper requires that all of your properties be optional, while Argo relies on a vast collection of custom operators and custom curried initializer functions to do its work. (While not shown in the snippet above, `User.init` code in reality is effectively implemented as `User.init(id)(name)(email)(role)(companyName)(friends)`.) + +We would like to provide a solution that skirts neither type safety, nor ease-of-use and -implementation. + +## Unabridged API + +```swift +/// Conformance to `Encodable` indicates that a type can encode itself to an external representation. +public protocol Encodable { + /// Encodes `self` into the given encoder. + /// + /// If `self` fails to encode anything, `encoder` will encode an empty keyed container in its place. + /// + /// - parameter encoder: The encoder to write data to. + /// - throws: An error if any values are invalid for `encoder`'s format. + func encode(to encoder: Encoder) throws +} + +/// Conformance to `Decodable` indicates that a type can decode itself from an external representation. +public protocol Decodable { + /// Initializes `self` by decoding from `decoder`. + /// + /// - parameter decoder: The decoder to read data from. + /// - throws: An error if reading from the decoder fails, or if read data is corrupted or otherwise invalid. + init(from decoder: Decoder) throws +} + +/// Conformance to `Codable` indicates that a type can convert itself into and out of an external representation. +public typealias Codable = Encodable & Decodable + +/// Conformance to `CodingKey` indicates that a type can be used as a key for encoding and decoding. +public protocol CodingKey { + /// The string to use in a named collection (e.g. a string-keyed dictionary). + var stringValue: String { get } + + /// Initializes `self` from a string. + /// + /// - parameter stringValue: The string value of the desired key. + /// - returns: An instance of `Self` from the given string, or `nil` if the given string does not correspond to any instance of `Self`. + init?(stringValue: String) + + /// The int to use in an indexed collection (e.g. an int-keyed dictionary). + var intValue: Int? { get } + + /// Initializes `self` from an integer. + /// + /// - parameter intValue: The integer value of the desired key. + /// - returns: An instance of `Self` from the given integer, or `nil` if the given integer does not correspond to any instance of `Self`. + init?(intValue: Int) +} + +/// An `Encoder` is a type which can encode values into a native format for external representation. +public protocol Encoder { + /// The path of coding keys taken to get to this point in encoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } + + /// Any contextual information set by the user for encoding. + var userInfo: [CodingUserInfoKey : Any] { get } + + /// Returns an encoding container appropriate for holding multiple values keyed by the given key type. + /// + /// - parameter type: The key type to use for the container. + /// - returns: A new keyed encoding container. + /// - precondition: May not be called after a prior `self.unkeyedContainer()` call. + /// - precondition: May not be called after a value has been encoded through a previous `self.singleValueContainer()` call. + func container(keyedBy type: Key.Type) -> KeyedEncodingContainer + + /// Returns an encoding container appropriate for holding multiple unkeyed values. + /// + /// - returns: A new empty unkeyed container. + /// - precondition: May not be called after a prior `self.container(keyedBy:)` call. + /// - precondition: May not be called after a value has been encoded through a previous `self.singleValueContainer()` call. + func unkeyedContainer() -> UnkeyedEncodingContainer + + /// Returns an encoding container appropriate for holding a single primitive value. + /// + /// - returns: A new empty single value container. + /// - precondition: May not be called after a prior `self.container(keyedBy:)` call. + /// - precondition: May not be called after a prior `self.unkeyedContainer()` call. + /// - precondition: May not be called after a value has been encoded through a previous `self.singleValueContainer()` call. + func singleValueContainer() -> SingleValueEncodingContainer +} + +/// A `Decoder` is a type which can decode values from a native format into in-memory representations. +public protocol Decoder { + /// The path of coding keys taken to get to this point in decoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } + + /// Any contextual information set by the user for decoding. + var userInfo: [CodingUserInfoKey : Any] { get } + + /// Returns the data stored in `self` as represented in a container keyed by the given key type. + /// + /// - parameter type: The key type to use for the container. + /// - returns: A keyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a keyed container. + func container(keyedBy type: Key.Type) throws -> KeyedDecodingContainer + + /// Returns the data stored in `self` as represented in a container appropriate for holding values with no keys. + /// + /// - returns: An unkeyed container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not an unkeyed container. + func unkeyedContainer() throws -> UnkeyedDecodingContainer + + /// Returns the data stored in `self` as represented in a container appropriate for holding a single primitive value. + /// + /// - returns: A single value container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a single value container. + func singleValueContainer() throws -> SingleValueDecodingContainer +} + +/// Conformance to `KeyedEncodingContainerProtocol` indicates that a type provides a view into an `Encoder`'s storage and is used to hold the encoded properties of an `Encodable` type in a keyed manner. +/// +/// Encoders should provide types conforming to `KeyedEncodingContainerProtocol` for their format. +public protocol KeyedEncodingContainerProtocol { + associatedtype Key : CodingKey + + /// The path of coding keys taken to get to this point in encoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } + + /// Encodes the given value for the given key. + /// + /// - parameter value: The value to encode. + /// - parameter key: The key to associate the value with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: T?, forKey key: Key) throws + + /// Encodes the given value for the given key. + /// + /// - parameter value: The value to encode. + /// - parameter key: The key to associate the value with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: Bool?, forKey key: Key) throws + mutating func encode(_ value: Int?, forKey key: Key) throws + mutating func encode(_ value: Int8?, forKey key: Key) throws + mutating func encode(_ value: Int16?, forKey key: Key) throws + mutating func encode(_ value: Int32?, forKey key: Key) throws + mutating func encode(_ value: Int64?, forKey key: Key) throws + mutating func encode(_ value: UInt?, forKey key: Key) throws + mutating func encode(_ value: UInt8?, forKey key: Key) throws + mutating func encode(_ value: UInt16?, forKey key: Key) throws + mutating func encode(_ value: UInt32?, forKey key: Key) throws + mutating func encode(_ value: UInt64?, forKey key: Key) throws + mutating func encode(_ value: Float?, forKey key: Key) throws + mutating func encode(_ value: Double?, forKey key: Key) throws + mutating func encode(_ value: String?, forKey key: Key) throws + + /// Encodes the given object weakly for the given key. + /// + /// For `Encoder`s that implement this functionality, this will only encode the given object and associate it with the given key if it is encoded unconditionally elsewhere in the payload (either previously or in the future). + /// + /// For formats which don't support this feature, the default implementation encodes the given object unconditionally. + /// + /// - parameter object: The object to encode. + /// - parameter key: The key to associate the object with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encodeWeak(_ object: T?, forKey key: Key) throws + + /// Stores a keyed encoding container for the given key and returns it. + /// + /// - parameter keyType: The key type to use for the container. + /// - parameter key: The key to encode the container for. + /// - returns: A new keyed encoding container. + mutating func nestedContainer(keyedBy keyType: NestedKey.Type, forKey key: Key) -> KeyedEncodingContainer + + /// Stores an unkeyed encoding container for the given key and returns it. + /// + /// - parameter key: The key to encode the container for. + /// - returns: A new unkeyed encoding container. + mutating func nestedUnkeyedContainer(forKey key: Key) -> UnkeyedEncodingContainer + + /// Stores a new nested container for the default `super` key and returns a new `Encoder` instance for encoding `super` into that container. + /// + /// Equivalent to calling `superEncoder(forKey:)` with `Key(stringValue: "super", intValue: 0)`. + /// + /// - returns: A new `Encoder` to pass to `super.encode(to:)`. + mutating func superEncoder() -> Encoder + + /// Stores a new nested container for the given key and returns a new `Encoder` instance for encoding `super` into that container. + /// + /// - parameter key: The key to encode `super` for. + /// - returns: A new `Encoder` to pass to `super.encode(to:)`. + mutating func superEncoder(forKey key: Key) -> Encoder +} + +/// `KeyedEncodingContainer` is a type-erased box for `KeyedEncodingContainerProtocol` types, similar to `AnyCollection` and `AnyHashable`. This is the type which consumers of the API interact with directly. +public struct KeyedEncodingContainer : KeyedEncodingContainerProtocol { + associatedtype Key = K + + /// Initializes `self` with the given container. + /// + /// - parameter container: The container to hold. + init(_ container: Container) where Container.Key == Key + + /// The path of coding keys taken to get to this point in encoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } + + /// Encodes the given value for the given key. + /// + /// - parameter value: The value to encode. + /// - parameter key: The key to associate the value with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: T?, forKey key: Key) throws + + /// Encodes the given value for the given key. + /// + /// - parameter value: The value to encode. + /// - parameter key: The key to associate the value with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: Bool?, forKey key: Key) throws + mutating func encode(_ value: Int?, forKey key: Key) throws + mutating func encode(_ value: Int8?, forKey key: Key) throws + mutating func encode(_ value: Int16?, forKey key: Key) throws + mutating func encode(_ value: Int32?, forKey key: Key) throws + mutating func encode(_ value: Int64?, forKey key: Key) throws + mutating func encode(_ value: UInt?, forKey key: Key) throws + mutating func encode(_ value: UInt8?, forKey key: Key) throws + mutating func encode(_ value: UInt16?, forKey key: Key) throws + mutating func encode(_ value: UInt32?, forKey key: Key) throws + mutating func encode(_ value: UInt64?, forKey key: Key) throws + mutating func encode(_ value: Float?, forKey key: Key) throws + mutating func encode(_ value: Double?, forKey key: Key) throws + mutating func encode(_ value: String?, forKey key: Key) throws + + /// Encodes the given object weakly for the given key. + /// + /// For `Encoder`s that implement this functionality, this will only encode the given object and associate it with the given key if it is encoded unconditionally elsewhere in the payload (either previously or in the future). + /// + /// For formats which don't support this feature, the default implementation encodes the given object unconditionally. + /// + /// - parameter object: The object to encode. + /// - parameter key: The key to associate the object with. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encodeWeak(_ object: T?, forKey key: Key) throws + + /// Stores a keyed encoding container for the given key and returns it. + /// + /// - parameter keyType: The key type to use for the container. + /// - parameter key: The key to encode the container for. + /// - returns: A new keyed encoding container. + mutating func nestedContainer(keyedBy keyType: NestedKey.Type, forKey key: Key) -> KeyedEncodingContainer + + /// Stores an unkeyed encoding container for the given key and returns it. + /// + /// - parameter key: The key to encode the container for. + /// - returns: A new unkeyed encoding container. + mutating func nestedUnkeyedContainer(forKey key: Key) -> UnkeyedEncodingContainer + + /// Stores a new nested container for the default `super` key and returns a new `Encoder` instance for encoding `super` into that container. + /// + /// Equivalent to calling `superEncoder(forKey:)` with `Key(stringValue: "super", intValue: 0)`. + /// + /// - returns: A new `Encoder` to pass to `super.encode(to:)`. + mutating func superEncoder() -> Encoder + + /// Stores a new nested container for the given key and returns a new `Encoder` instance for encoding `super` into that container. + /// + /// - parameter key: The key to encode `super` for. + /// - returns: A new `Encoder` to pass to `super.encode(to:)`. + mutating func superEncoder(forKey key: Key) -> Encoder +} + +/// Conformance to `KeyedDecodingContainerProtocol` indicates that a type provides a view into a `Decoder`'s storage and is used to hold the encoded properties of a `Decodable` type in a keyed manner. +/// +/// Decoders should provide types conforming to `KeyedDecodingContainerProtocol` for their format. +public protocol KeyedDecodingContainerProtocol { + associatedtype Key : CodingKey + + /// The path of coding keys taken to get to this point in decoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } + + /// All the keys the `Decoder` has for this container. + /// + /// Different keyed containers from the same `Decoder` may return different keys here; it is possible to encode with multiple key types which are not convertible to one another. This should report all keys present which are convertible to the requested type. + var allKeys: [Key] { get } + + /// Returns whether the `Decoder` contains a value associated with the given key. + /// + /// The value associated with the given key may be a null value as appropriate for the data format. + /// + /// - parameter key: The key to search for. + /// - returns: Whether the `Decoder` has an entry for the given key. + func contains(_ key: Key) -> Bool + + /// Decodes a value of the given type for the given key. + /// + /// A default implementation is given for these types which calls into the `decodeIfPresent` implementations below. + /// + /// - parameter type: The type of value to decode. + /// - parameter key: The key that the decoded value is associated with. + /// - returns: A value of the requested type, if present for the given key and convertible to the requested type. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the given key or if the value is null. + func decode(_ type: Bool.Type, forKey key: Key) throws -> Bool + func decode(_ type: Int.Type, forKey key: Key) throws -> Int + func decode(_ type: Int8.Type, forKey key: Key) throws -> Int8 + func decode(_ type: Int16.Type, forKey key: Key) throws -> Int16 + func decode(_ type: Int32.Type, forKey key: Key) throws -> Int32 + func decode(_ type: Int64.Type, forKey key: Key) throws -> Int64 + func decode(_ type: UInt.Type, forKey key: Key) throws -> UInt + func decode(_ type: UInt8.Type, forKey key: Key) throws -> UInt8 + func decode(_ type: UInt16.Type, forKey key: Key) throws -> UInt16 + func decode(_ type: UInt32.Type, forKey key: Key) throws -> UInt32 + func decode(_ type: UInt64.Type, forKey key: Key) throws -> UInt64 + func decode(_ type: Float.Type, forKey key: Key) throws -> Float + func decode(_ type: Double.Type, forKey key: Key) throws -> Double + func decode(_ type: String.Type, forKey key: Key) throws -> String + func decode(_ type: T.Type, forKey key: Key) throws -> T + + /// Decodes a value of the given type for the given key, if present. + /// + /// This method returns `nil` if the container does not have a value associated with `key`, or if the value is null. The difference between these states can be distinguished with a `contains(_:)` call. + /// + /// - parameter type: The type of value to decode. + /// - parameter key: The key that the decoded value is associated with. + /// - returns: A decoded value of the requested type, or `nil` if the `Decoder` does not have an entry associated with the given key, or if the value is a null value. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + func decodeIfPresent(_ type: Bool.Type, forKey key: Key) throws -> Bool? + func decodeIfPresent(_ type: Int.Type, forKey key: Key) throws -> Int? + func decodeIfPresent(_ type: Int8.Type, forKey key: Key) throws -> Int8? + func decodeIfPresent(_ type: Int16.Type, forKey key: Key) throws -> Int16? + func decodeIfPresent(_ type: Int32.Type, forKey key: Key) throws -> Int32? + func decodeIfPresent(_ type: Int64.Type, forKey key: Key) throws -> Int64? + func decodeIfPresent(_ type: UInt.Type, forKey key: Key) throws -> UInt? + func decodeIfPresent(_ type: UInt8.Type, forKey key: Key) throws -> UInt8? + func decodeIfPresent(_ type: UInt16.Type, forKey key: Key) throws -> UInt16? + func decodeIfPresent(_ type: UInt32.Type, forKey key: Key) throws -> UInt32? + func decodeIfPresent(_ type: UInt64.Type, forKey key: Key) throws -> UInt64? + func decodeIfPresent(_ type: Float.Type, forKey key: Key) throws -> Float? + func decodeIfPresent(_ type: Double.Type, forKey key: Key) throws -> Double? + func decodeIfPresent(_ type: String.Type, forKey key: Key) throws -> String? + func decodeIfPresent(_ type: T.Type, forKey key: Key) throws -> T? + + /// Returns the data stored for the given key as represented in a container keyed by the given key type. + /// + /// - parameter type: The key type to use for the container. + /// - parameter key: The key that the nested container is associated with. + /// - returns: A keyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a keyed container. + func nestedContainer(keyedBy type: NestedKey.Type, forKey key: Key) throws -> KeyedDecodingContainer + + /// Returns the data stored for the given key as represented in an unkeyed container. + /// + /// - parameter key: The key that the nested container is associated with. + /// - returns: An unkeyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not an unkeyed container. + func nestedUnkeyedContainer(forKey key: Key) throws -> UnkeyedDecodingContainer + + /// Returns a `Decoder` instance for decoding `super` from the container associated with the default `super` key. + /// + /// Equivalent to calling `superDecoder(forKey:)` with `Key(stringValue: "super", intValue: 0)`. + /// + /// - returns: A new `Decoder` to pass to `super.init(from:)`. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the default `super` key, or if the stored value is null. + func superDecoder() throws -> Decoder + + /// Returns a `Decoder` instance for decoding `super` from the container associated with the given key. + /// + /// - parameter key: The key to decode `super` for. + /// - returns: A new `Decoder` to pass to `super.init(from:)`. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the given key, or if the stored value is null. + func superDecoder(forKey key: Key) throws -> Decoder +} + +/// `KeyedDecodingContainer` is a type-erased box for `KeyedDecodingContainerProtocol` types, similar to `AnyCollection` and `AnyHashable`. This is the type which consumers of the API interact with directly. +public struct KeyedDecodingContainer : KeyedDecodingContainerProtocol { + associatedtype Key = K + + /// Initializes `self` with the given container. + /// + /// - parameter container: The container to hold. + init(_ container: Container) where Container.Key == Key + + /// The path of coding keys taken to get to this point in decoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } + + /// All the keys the `Decoder` has for this container. + /// + /// Different keyed containers from the same `Decoder` may return different keys here; it is possible to encode with multiple key types which are not convertible to one another. This should report all keys present which are convertible to the requested type. + var allKeys: [Key] { get } + + /// Returns whether the `Decoder` contains a value associated with the given key. + /// + /// The value associated with the given key may be a null value as appropriate for the data format. + /// + /// - parameter key: The key to search for. + /// - returns: Whether the `Decoder` has an entry for the given key. + func contains(_ key: Key) -> Bool + + /// Decodes a value of the given type for the given key. + /// + /// A default implementation is given for these types which calls into the `decodeIfPresent` implementations below. + /// + /// - parameter type: The type of value to decode. + /// - parameter key: The key that the decoded value is associated with. + /// - returns: A value of the requested type, if present for the given key and convertible to the requested type. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the given key or if the value is null. + func decode(_ type: Bool.Type, forKey key: Key) throws -> Bool + func decode(_ type: Int.Type, forKey key: Key) throws -> Int + func decode(_ type: Int8.Type, forKey key: Key) throws -> Int8 + func decode(_ type: Int16.Type, forKey key: Key) throws -> Int16 + func decode(_ type: Int32.Type, forKey key: Key) throws -> Int32 + func decode(_ type: Int64.Type, forKey key: Key) throws -> Int64 + func decode(_ type: UInt.Type, forKey key: Key) throws -> UInt + func decode(_ type: UInt8.Type, forKey key: Key) throws -> UInt8 + func decode(_ type: UInt16.Type, forKey key: Key) throws -> UInt16 + func decode(_ type: UInt32.Type, forKey key: Key) throws -> UInt32 + func decode(_ type: UInt64.Type, forKey key: Key) throws -> UInt64 + func decode(_ type: Float.Type, forKey key: Key) throws -> Float + func decode(_ type: Double.Type, forKey key: Key) throws -> Double + func decode(_ type: String.Type, forKey key: Key) throws -> String + func decode(_ type: T.Type, forKey key: Key) throws -> T + + /// Decodes a value of the given type for the given key, if present. + /// + /// This method returns `nil` if the container does not have a value associated with `key`, or if the value is null. The difference between these states can be distinguished with a `contains(_:)` call. + /// + /// - parameter type: The type of value to decode. + /// - parameter key: The key that the decoded value is associated with. + /// - returns: A decoded value of the requested type, or `nil` if the `Decoder` does not have an entry associated with the given key, or if the value is a null value. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + func decodeIfPresent(_ type: Bool.Type, forKey key: Key) throws -> Bool? + func decodeIfPresent(_ type: Int.Type, forKey key: Key) throws -> Int? + func decodeIfPresent(_ type: Int8.Type, forKey key: Key) throws -> Int8? + func decodeIfPresent(_ type: Int16.Type, forKey key: Key) throws -> Int16? + func decodeIfPresent(_ type: Int32.Type, forKey key: Key) throws -> Int32? + func decodeIfPresent(_ type: Int64.Type, forKey key: Key) throws -> Int64? + func decodeIfPresent(_ type: UInt.Type, forKey key: Key) throws -> UInt? + func decodeIfPresent(_ type: UInt8.Type, forKey key: Key) throws -> UInt8? + func decodeIfPresent(_ type: UInt16.Type, forKey key: Key) throws -> UInt16? + func decodeIfPresent(_ type: UInt32.Type, forKey key: Key) throws -> UInt32? + func decodeIfPresent(_ type: UInt64.Type, forKey key: Key) throws -> UInt64? + func decodeIfPresent(_ type: Float.Type, forKey key: Key) throws -> Float? + func decodeIfPresent(_ type: Double.Type, forKey key: Key) throws -> Double? + func decodeIfPresent(_ type: String.Type, forKey key: Key) throws -> String? + func decodeIfPresent(_ type: T.Type, forKey key: Key) throws -> T? + + /// Returns the data stored for the given key as represented in a container keyed by the given key type. + /// + /// - parameter type: The key type to use for the container. + /// - parameter key: The key that the nested container is associated with. + /// - returns: A keyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a keyed container. + func nestedContainer(keyedBy type: NestedKey.Type, forKey key: Key) throws -> KeyedDecodingContainer + + /// Returns the data stored for the given key as represented in an unkeyed container. + /// + /// - parameter key: The key that the nested container is associated with. + /// - returns: An unkeyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not an unkeyed container. + func nestedUnkeyedContainer(forKey key: Key) throws -> UnkeyedDecodingContainer + + /// Returns a `Decoder` instance for decoding `super` from the container associated with the default `super` key. + /// + /// Equivalent to calling `superDecoder(forKey:)` with `Key(stringValue: "super", intValue: 0)`. + /// + /// - returns: A new `Decoder` to pass to `super.init(from:)`. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the default `super` key, or if the stored value is null. + func superDecoder() throws -> Decoder + + /// Returns a `Decoder` instance for decoding `super` from the container associated with the given key. + /// + /// - parameter key: The key to decode `super` for. + /// - returns: A new `Decoder` to pass to `super.init(from:)`. + /// - throws: `CocoaError.coderValueNotFound` if `self` does not have an entry for the given key, or if the stored value is null. + func superDecoder(forKey key: Key) throws -> Decoder +} + +/// Conformance to `UnkeyedEncodingContainer` indicates that a type provides a view into an `Encoder`'s storage and is used to hold the encoded properties of an `Encodable` type sequentially, without keys. +/// +/// Encoders should provide types conforming to `UnkeyedEncodingContainer` for their format. +public protocol UnkeyedEncodingContainer { + /// The path of coding keys taken to get to this point in encoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } + + /// Encodes the given value. + /// + /// - parameter value: The value to encode. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: T?) throws + + /// Encodes the given value. + /// + /// - parameter value: The value to encode. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encode(_ value: Bool?) throws + mutating func encode(_ value: Int?) throws + mutating func encode(_ value: Int8?) throws + mutating func encode(_ value: Int16?) throws + mutating func encode(_ value: Int32?) throws + mutating func encode(_ value: Int64?) throws + mutating func encode(_ value: UInt?) throws + mutating func encode(_ value: UInt8?) throws + mutating func encode(_ value: UInt16?) throws + mutating func encode(_ value: UInt32?) throws + mutating func encode(_ value: UInt64?) throws + mutating func encode(_ value: Float?) throws + mutating func encode(_ value: Double?) throws + mutating func encode(_ value: String?) throws + + /// Encodes the given object weakly. + /// + /// For `Encoder`s that implement this functionality, this will only encode the given object if it is encoded unconditionally elsewhere in the payload (either previously or in the future). + /// + /// For formats which don't support this feature, the default implementation encodes the given object unconditionally. + /// + /// - parameter object: The object to encode. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + mutating func encodeWeak(_ object: T?) throws + + /// Encodes the elements of the given sequence. + /// + /// A default implementation of these is given in an extension. + /// + /// - parameter sequence: The sequences whose contents to encode. + /// - throws: An error if any of the contained values throws an error. + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Bool + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Int + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Int8 + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Int16 + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Int32 + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Int64 + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == UInt + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == UInt8 + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == UInt16 + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == UInt32 + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == UInt64 + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Float + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == Double + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element == String + mutating func encode(contentsOf sequence: Sequence) throws where Sequence.Iterator.Element : Encodable + + /// Encodes a nested container keyed by the given type and returns it. + /// + /// - parameter keyType: The key type to use for the container. + /// - returns: A new keyed encoding container. + mutating func nestedContainer(keyedBy keyType: NestedKey.Type) -> KeyedEncodingContainer + + /// Encodes an unkeyed encoding container and returns it. + /// + /// - returns: A new unkeyed encoding container. + mutating func nestedUnkeyedContainer() -> UnkeyedEncodingContainer + + /// Encodes a nested container and returns an `Encoder` instance for encoding `super` into that container. + /// + /// - returns: A new `Encoder` to pass to `super.encode(to:)`. + mutating func superEncoder() -> Encoder +} + +/// Conformance to `UnkeyedDecodingContainer` indicates that a type provides a view into a `Decoder`'s storage and is used to hold the encoded properties of a `Decodable` type sequentially, without keys. +/// +/// Decoders should provide types conforming to `UnkeyedDecodingContainer` for their format. +public protocol UnkeyedDecodingContainer { + /// The path of coding keys taken to get to this point in decoding. + /// A `nil` value indicates an unkeyed container. + var codingPath: [CodingKey?] { get } + + /// Returns the number of elements (if known) contained within this container. + var count: Int? { get } + + /// Returns whether there are no more elements left to be decoded in the container. + var isAtEnd: Bool { get } + + /// Decodes a value of the given type. + /// + /// A default implementation is given for these types which calls into the `decodeIfPresent` implementations below. + /// + /// - parameter type: The type of value to decode. + /// - returns: A value of the requested type, if present for the given key and convertible to the requested type. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + /// - throws: `CocoaError.coderValueNotFound` if the encountered encoded value is null, or of there are no more values to decode. + mutating func decode(_ type: Bool.Type) throws -> Bool + mutating func decode(_ type: Int.Type) throws -> Int + mutating func decode(_ type: Int8.Type) throws -> Int8 + mutating func decode(_ type: Int16.Type) throws -> Int16 + mutating func decode(_ type: Int32.Type) throws -> Int32 + mutating func decode(_ type: Int64.Type) throws -> Int64 + mutating func decode(_ type: UInt.Type) throws -> UInt + mutating func decode(_ type: UInt8.Type) throws -> UInt8 + mutating func decode(_ type: UInt16.Type) throws -> UInt16 + mutating func decode(_ type: UInt32.Type) throws -> UInt32 + mutating func decode(_ type: UInt64.Type) throws -> UInt64 + mutating func decode(_ type: Float.Type) throws -> Float + mutating func decode(_ type: Double.Type) throws -> Double + mutating func decode(_ type: String.Type) throws -> String + mutating func decode(_ type: T.Type) throws -> T + + /// Decodes a value of the given type, if present. + /// + /// This method returns `nil` if the container has no elements left to decode, or if the value is null. The difference between these states can be distinguished by checking `isAtEnd`. + /// + /// - parameter type: The type of value to decode. + /// - returns: A decoded value of the requested type, or `nil` if the value is a null value, or if there are no more elements to decode. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value is not convertible to the requested type. + mutating func decodeIfPresent(_ type: Bool.Type) throws -> Bool? + mutating func decodeIfPresent(_ type: Int.Type) throws -> Int? + mutating func decodeIfPresent(_ type: Int8.Type) throws -> Int8? + mutating func decodeIfPresent(_ type: Int16.Type) throws -> Int16? + mutating func decodeIfPresent(_ type: Int32.Type) throws -> Int32? + mutating func decodeIfPresent(_ type: Int64.Type) throws -> Int64? + mutating func decodeIfPresent(_ type: UInt.Type) throws -> UInt? + mutating func decodeIfPresent(_ type: UInt8.Type) throws -> UInt8? + mutating func decodeIfPresent(_ type: UInt16.Type) throws -> UInt16? + mutating func decodeIfPresent(_ type: UInt32.Type) throws -> UInt32? + mutating func decodeIfPresent(_ type: UInt64.Type) throws -> UInt64? + mutating func decodeIfPresent(_ type: Float.Type) throws -> Float? + mutating func decodeIfPresent(_ type: Double.Type) throws -> Double? + mutating func decodeIfPresent(_ type: String.Type) throws -> String? + mutating func decodeIfPresent(_ type: T.Type) throws -> T? + + /// Decodes a nested container keyed by the given type. + /// + /// - parameter type: The key type to use for the container. + /// - returns: A keyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not a keyed container. + mutating func nestedContainer(keyedBy type: NestedKey.Type) throws -> KeyedDecodingContainer + + /// Decodes an unkeyed nested container. + /// + /// - returns: An unkeyed decoding container view into `self`. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered stored value is not an unkeyed container. + mutating func nestedUnkeyedContainer() throws -> UnkeyedDecodingContainer + + /// Decodes a nested container and returns a `Decoder` instance for decoding `super` from that container. + /// + /// - returns: A new `Decoder` to pass to `super.init(from:)`. + /// - throws: `CocoaError.coderValueNotFound` if the encountered encoded value is null, or of there are no more values to decode. + mutating func superDecoder() throws -> Decoder +} + +/// A `SingleValueEncodingContainer` is a container which can support the storage and direct encoding of a single non-keyed value. +public protocol SingleValueEncodingContainer { + /// Encodes a single value of the given type. + /// + /// - parameter value: The value to encode. + /// - throws: `CocoaError.coderInvalidValue` if the given value is invalid in the current context for this format. + /// - precondition: May not be called after a previous `self.encode(_:)` call. + mutating func encode(_ value: Bool) throws + mutating func encode(_ value: Int) throws + mutating func encode(_ value: Int8) throws + mutating func encode(_ value: Int16) throws + mutating func encode(_ value: Int32) throws + mutating func encode(_ value: Int64) throws + mutating func encode(_ value: UInt) throws + mutating func encode(_ value: UInt8) throws + mutating func encode(_ value: UInt16) throws + mutating func encode(_ value: UInt32) throws + mutating func encode(_ value: UInt64) throws + mutating func encode(_ value: Float) throws + mutating func encode(_ value: Double) throws + mutating func encode(_ value: String) throws +} + +/// A `SingleValueDecodingContainer` is a container which can support the storage and direct decoding of a single non-keyed value. +public protocol SingleValueDecodingContainer { + /// Decodes a single value of the given type. + /// + /// - parameter type: The type to decode as. + /// - returns: A value of the requested type. + /// - throws: `CocoaError.coderTypeMismatch` if the encountered encoded value cannot be converted to the requested type. + func decode(_ type: Bool.Type) throws -> Bool + func decode(_ type: Int.Type) throws -> Int + func decode(_ type: Int8.Type) throws -> Int8 + func decode(_ type: Int16.Type) throws -> Int16 + func decode(_ type: Int32.Type) throws -> Int32 + func decode(_ type: Int64.Type) throws -> Int64 + func decode(_ type: UInt.Type) throws -> UInt + func decode(_ type: UInt8.Type) throws -> UInt8 + func decode(_ type: UInt16.Type) throws -> UInt16 + func decode(_ type: UInt32.Type) throws -> UInt32 + func decode(_ type: UInt64.Type) throws -> UInt64 + func decode(_ type: Float.Type) throws -> Float + func decode(_ type: Double.Type) throws -> Double + func decode(_ type: String.Type) throws -> String +} + +/// Represents a user-defined key for providing context for encoding and decoding. +public struct CodingUserInfoKey : RawRepresentable, Hashable { + typealias RawValue = String + let rawValue: String + init?(rawValue: String) + init(_ value: String) +} + +// Repeat for all primitive types... +extension Bool : Codable { + public init(from decoder: Decoder) throws { + self = try decoder.singleValueContainer().decode(Bool.self) + } + + public func encode(to encoder: Encoder) throws { + try encoder.singleValueContainer().encode( self) + } +} + +// Repeat for all primitive types... +public extension RawRepresentable where RawValue == Bool, Self : Codable { + public init(from decoder: Decoder) throws { + let decoded = try decoder.singleValueContainer().decode(RawValue.self) + guard let value = Self(rawValue: decoded) else { + throw CocoaError.error(.coderReadCorrupt) + } + + self = value + } + + public func encode(to encoder: Encoder) throws { + try encoder.singleValueContainer().encode(self.rawValue) + } +} +``` diff --git a/proposals/0167-swift-encoders.md b/proposals/0167-swift-encoders.md new file mode 100644 index 0000000000..223aa7f390 --- /dev/null +++ b/proposals/0167-swift-encoders.md @@ -0,0 +1,408 @@ +# Swift Encoders + +* Proposal: [SE-0167](0167-swift-encoders.md) +* Authors: [Itai Ferber](https://github.com/itaiferber), [Michael LeHew](https://github.com/mlehew), [Tony Parker](https://github.com/parkera) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0167-swift-encoders/5781) +* Implementation: [apple/swift#9005](https://github.com/apple/swift/pull/9005) + +## Introduction + +As part of the proposal for a Swift archival and serialization API ([SE-0166](0166-swift-archival-serialization.md)), we are also proposing new API for specific new encoders and decoders, as well as introducing support for new `Codable` types in `NSKeyedArchiver` and `NSKeyedUnarchiver`. + +This proposal composes the latter two stages laid out in [SE-0166](0166-swift-archival-serialization.md). + +## Motivation + +With the base API discussed in [SE-0166](0166-swift-archival-serialization.md), we want to provide new encoders for consumers of this API, as well as provide a consistent story for bridging this new API with our existing `NSCoding` implementations. We would like to offer a base level of support that users can depend on, and set a pattern that third parties can follow in implementing and extending their own encoders. + +## Proposed solution + +We will: + +1. Add two new encoders and decoders to support encoding Swift value trees in JSON and property list formats +2. Add support for passing `Codable` Swift values to `NSKeyedArchiver` and `NSKeyedUnarchiver`, and add `Codable` conformance to our Swift value types + +## Detailed design + +### New Encoders and Decoders + +#### JSON + +One of the key motivations for the introduction of this API was to allow safer interaction between Swift values and their JSON representations. For values which are `Codable`, users can encode to and decode from JSON with `JSONEncoder` and `JSONDecoder`: + +```swift +open class JSONEncoder { + // MARK: Top-Level Encoding + + /// Encodes the given top-level value and returns its JSON representation. + /// + /// - parameter value: The value to encode. + /// - returns: A new `Data` value containing the encoded JSON data. + /// - throws: `CocoaError.coderInvalidValue` if a non-conforming floating-point value is encountered during archiving, and the encoding strategy is `.throw`. + /// - throws: An error if any value throws an error during encoding. + open func encode(_ value: T) throws -> Data + + // MARK: Customization + + /// The formatting of the output JSON data. + public enum OutputFormatting { + /// Produce JSON compacted by removing whitespace. This is the default formatting. + case compact + + /// Produce human-readable JSON with indented output. + case prettyPrinted + } + + /// The strategy to use for encoding `Date` values. + public enum DateEncodingStrategy { + /// Defer to `Date` for choosing an encoding. This is the default strategy. + case deferredToDate + + /// Encode the `Date` as a UNIX timestamp (as a JSON number). + case secondsSince1970 + + /// Encode the `Date` as UNIX millisecond timestamp (as a JSON number). + case millisecondsSince1970 + + /// Encode the `Date` as an ISO-8601-formatted string (in RFC 3339 format). + @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *) + case iso8601 + + /// Encode the `Date` as a string formatted by the given formatter. + case formatted(DateFormatter) + + /// Encode the `Date` as a custom value encoded by the given closure. + /// + /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place. + case custom((_ value: Date, _ encoder: Encoder) throws -> Void) + } + + /// The strategy to use for encoding `Data` values. + public enum DataEncodingStrategy { + /// Encoded the `Data` as a Base64-encoded string. This is the default strategy. + case base64 + + /// Encode the `Data` as a custom value encoded by the given closure. + /// + /// If the closure fails to encode a value into the given encoder, the encoder will encode an empty `.default` container in its place. + case custom((_ value: Data, _ encoder: Encoder) throws -> Void) + } + + /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN). + public enum NonConformingFloatEncodingStrategy { + /// Throw upon encountering non-conforming values. This is the default strategy. + case `throw` + + /// Encode the values using the given representation strings. + case convertToString(positiveInfinity: String, negativeInfinity: String, nan: String) + } + + /// The output format to produce. Defaults to `.compact`. + open var outputFormatting: OutputFormatting + + /// The strategy to use in encoding dates. Defaults to `.deferredToDate`. + open var dateEncodingStrategy: DateEncodingStrategy + + /// The strategy to use in encoding binary data. Defaults to `.base64`. + open var dataEncodingStrategy: DataEncodingStrategy + + /// The strategy to use in encoding non-conforming numbers. Defaults to `.throw`. + open var nonConformingFloatEncodingStrategy: NonConformingFloatEncodingStrategy + + /// Contextual information to expose during encoding. + open var userInfo: [CodingUserInfoKey : Any] +} + +open class JSONDecoder { + // MARK: Top-Level Decoding + + /// Decodes a top-level value of the given type from the given JSON representation. + /// + /// - parameter type: The type of the value to decode. + /// - parameter data: The data to decode from. + /// - returns: A value of the requested type. + /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not valid JSON. + /// - throws: An error if any value throws an error during decoding. + open func decode(_ type: T.Type, from data: Data) throws -> Value + + // MARK: Customization + + /// The strategy to use for decoding `Date` values. + public enum DateDecodingStrategy { + /// Defer to `Date` for decoding. This is the default strategy. + case deferredToDate + + /// Decode the `Date` as a UNIX timestamp from a JSON number. + case secondsSince1970 + + /// Decode the `Date` as UNIX millisecond timestamp from a JSON number. + case millisecondsSince1970 + + /// Decode the `Date` as an ISO-8601-formatted string (in RFC 3339 format). + @available(OSX 10.12, iOS 10.0, watchOS 3.0, tvOS 10.0, *) + case iso8601 + + /// Decode the `Date` as a string parsed by the given formatter. + case formatted(DateFormatter) + + /// Decode the `Date` as a custom value decoded by the given closure. + case custom((_ decoder: Decoder) throws -> Date) + } + + /// The strategy to use for decoding `Data` values. + public enum DataDecodingStrategy { + /// Decode the `Data` from a Base64-encoded string. This is the default strategy. + case base64 + + /// Decode the `Data` as a custom value decoded by the given closure. + case custom((_ decoder: Decoder) throws -> Data) + } + + /// The strategy to use for non-JSON-conforming floating-point values (IEEE 754 infinity and NaN). + public enum NonConformingFloatDecodingStrategy { + /// Throw upon encountering non-conforming values. This is the default strategy. + case `throw` + + /// Decode the values from the given representation strings. + case convertFromString(positiveInfinity: String, negativeInfinity: String, nan: String) + } + + /// The strategy to use in decoding dates. Defaults to `.deferredToDate`. + open var dateDecodingStrategy: DateDecodingStrategy + + /// The strategy to use in decoding binary data. Defaults to `.base64`. + open var dataDecodingStrategy: DataDecodingStrategy + + /// The strategy to use in decoding non-conforming numbers. Defaults to `.throw`. + open var nonConformingFloatDecodingStrategy: NonConformingFloatDecodingStrategy + + /// Contextual information to expose during decoding. + open var userInfo: [CodingUserInfoKey : Any] +} +``` + +Usage: + +```swift +var encoder = JSONEncoder() +encoder.dateEncodingStrategy = .iso8601 +encoder.dataEncodingStrategy = .custom(myBase85Encoder) + +// Since JSON does not natively allow for infinite or NaN values, we can customize strategies for encoding these non-conforming values. +encoder.nonConformingFloatEncodingStrategy = .convertToString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN") + +// MyValue conforms to Codable +let topLevel = MyValue(...) + +let payload: Data +do { + payload = try encoder.encode(topLevel) +} catch { + // Some value threw while encoding. +} + +// ... + +var decoder = JSONDecoder() +decoder.dateDecodingStrategy = .iso8601 +decoder.dataDecodingStrategy = .custom(myBase85Decoder) + +// Look for and match these values when decoding `Double`s or `Float`s. +decoder.nonConformingFloatDecodingStrategy = .convertFromString(positiveInfinity: "INF", negativeInfinity: "-INF", nan: "NaN") + +let topLevel: MyValue +do { + topLevel = try decoder.decode(MyValue.self, from: payload) +} catch { + // Data was corrupted, or some value threw while decoding. +} +``` + +It should be noted here that `JSONEncoder` and `JSONDecoder` do not themselves conform to `Encoder` and `Decoder`; instead, they contain private nested types which do conform to `Encoder` and `Decoder`, which are passed to values' `encode(to:)` and `init(from:)`. This is because `JSONEncoder` and `JSONDecoder` must present a different top-level API than they would at intermediate levels. + +#### Property List + +We also intend to support the property list format, with `PropertyListEncoder` and `PropertyListDecoder`: + +```swift +open class PropertyListEncoder { + // MARK: Top-Level Encoding + + /// Encodes the given top-level value and returns its property list representation. + /// + /// - parameter value: The value to encode. + /// - returns: A new `Data` value containing the encoded property list data. + /// - throws: An error if any value throws an error during encoding. + open func encode(_ value: T) throws -> Data + + // MARK: Customization + + /// The output format to write the property list data in. Defaults to `.binary`. + open var outputFormat: PropertyListSerialization.PropertyListFormat + + /// Contextual information to expose during encoding. + open var userInfo: [CodingUserInfoKey : Any] +} + +open class PropertyListDecoder { + // MARK: Top-Level Decoding + + /// Decodes a top-level value of the given type from the given property list representation. + /// + /// - parameter type: The type of the value to decode. + /// - parameter data: The data to decode from. + /// - returns: A value of the requested type. + /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list. + /// - throws: An error if any value throws an error during decoding. + open func decode(_ type: T.Type, from data: Data) throws -> Value + + /// Decodes a top-level value of the given type from the given property list representation. + /// + /// - parameter type: The type of the value to decode. + /// - parameter data: The data to decode from. + /// - parameter format: The parsed property list format. + /// - returns: A value of the requested type along with the detected format of the property list. + /// - throws: `CocoaError.coderReadCorrupt` if values requested from the payload are corrupted, or if the given data is not a valid property list. + /// - throws: An error if any value throws an error during decoding. + open func decode(_ type: Value.Type, from data: Data, format: inout PropertyListSerialization.PropertyListFormat) throws -> Value + + // MARK: Customization + + /// Contextual information to expose during decoding. + open var userInfo: [CodingUserInfoKey : Any] +} +``` + +Usage: + +```swift +let encoder = PropertyListEncoder() +let topLevel = MyValue(...) +let payload: Data +do { + payload = try encoder.encode(topLevel) +} catch { + // Some value threw while encoding. +} + +// ... + +let decoder = PropertyListDecoder() +let topLevel: MyValue +do { + topLevel = try decoder.decode(MyValue.self, from: payload) +} catch { + // Data was corrupted, or some value threw while decoding. +} +``` + +Like with JSON, `PropertyListEncoder` and `PropertyListDecoder` also provide private nested types which conform to `Encoder` and `Decoder` for performing the archival. + +### Foundation-Provided Errors + +Along with providing the above encoders and decoders, we would like to promote the use of a common set of error codes and messages across all new encoders and decoders. A common vocabulary of expected errors allows end-users to write code agnostic about the specific encoder/decoder implementation they are working with, whether first-party or third-party: + +```swift +extension CocoaError.Code { + /// Thrown when a value incompatible with the output format is encoded. + public static var coderInvalidValue: CocoaError.Code + + /// Thrown when a value of a given type is requested but the encountered value is of an incompatible type. + public static var coderTypeMismatch: CocoaError.Code + + /// Thrown when read data is corrupted or otherwise invalid for the format. This value already exists today. + public static var coderReadCorrupt: CocoaError.Code + + /// Thrown when a requested key or value is unexpectedly null or missing. This value already exists today. + public static var coderValueNotFound: CocoaError.Code +} + +// These reexpose the values above. +extension CocoaError { + public static var coderInvalidValue: CocoaError.Code + + public static var coderTypeMismatch: CocoaError.Code +} +``` + +The localized description strings associated with the two new error codes are: + +* `.coderInvalidValue`: "The data is not valid for encoding in this format." +* `.coderTypeMismatch`: "The data couldn't be read because it isn't in the correct format." (Precedent from `NSCoderReadCorruptError`.) + +All of these errors will include the coding key path that led to the failure in the error's `userInfo` dictionary under `NSCodingPathErrorKey`, along with a non-localized, developer-facing failure reason under `NSDebugDescriptionErrorKey`. + +### `NSKeyedArchiver` & `NSKeyedUnarchiver` Changes + +Although our primary objectives for this new API revolve around Swift, we would like to make it easy for current consumers to make the transition to `Codable` where appropriate. As part of this, we would like to bridge compatibility between new `Codable` types (or newly-`Codable`-adopting types) and existing `NSCoding` types. + +To do this, we want to introduce changes to `NSKeyedArchiver` and `NSKeyedUnarchiver` in Swift that allow archival of `Codable` types intermixed with `NSCoding` types: + +```swift +// These are provided in the Swift overlay, and included in swift-corelibs-foundation. +extension NSKeyedArchiver { + public func encodeCodable(_ codable: Encodable?, forKey key: String) { ... } +} + +extension NSKeyedUnarchiver { + public func decodeCodable(_ type: T.Type, forKey key: String) -> T? { ... } +} +``` + +> NOTE: Since these changes are being made in extensions in the Swift overlay, it is not yet possible for these methods to be overridden. These can therefore not be added to `NSCoder`, since `NSKeyedArchiver` and `NSKeyedUnarchiver` would not be able to provide concrete implementations. In order to call these methods, it is necessary to downcast from an `NSCoder` to `NSKeyedArchiver`/`NSKeyedUnarchiver` directly. Since subclasses of `NSKeyedArchiver` and `NSKeyedUnarchiver` in Swift will inherit these implementations without being able to override them (which is wrong), we will `NSRequiresConcreteImplementation()` dynamically in subclasses. + +The addition of these methods allows the introduction of `Codable` types into existing `NSCoding` structures, allowing for a transition to `Codable` types where appropriate. + +#### Refining `encode(_:forKey:)` + +Along with these extensions, we would like to refine the import of `-[NSCoder encodeObject:forKey:]`, which is currently imported into Swift as `encode(_: Any?, forKey: String)`. This method currently accepts Objective-C and Swift objects conforming to `NSCoding` (non-conforming objects produce a runtime error), as well as bridgeable Swift types (`Data`, `String`, `Array`, etc.); we would like to extend it to support new Swift `Codable` types, which would otherwise produce a runtime error upon call. + +`-[NSCoder encodeObject:forKey:]` will be given a new Swift name of `encodeObject(_:forKey:)`, and we will provide a replacement `encode(_: Any?, forKey: String)` in the overlay which will funnel out to either `encodeCodable(_:forKey:)` or `encodeObject(_:forKey:)` as appropriate. This should maintain source compatibility for end users already calling `encode(_:forKey:)`, as well as behavior compatibility for subclassers of `NSCoder` and `NSKeyedArchiver` who may be providing their own `encode(_:forKey:)`. + +#### Semantics of `Codable` Types in Archives + +There are a few things to note about including `Codable` values in `NSKeyedArchiver` archives: + +* Bridgeable Foundation types will always bridge before encoding. This is to facilitate writing Foundation types in a compatible format from both Objective-C and Swift + * On decode, these types will decode either as their Objective-C or Swift version, depending on user need (`decodeObject(forKey:)` will decode as an Objective-C object; `decodeCodable(_:forKey:)` as a Swift value) +* User types, which are not bridgeable, do not write out a `$class` and can only be decoded in Swift. In the future, we may add API to allow Swift types to provide an Objective-C class to decode as, effectively allowing for user bridging across archival + +##### Foundation Types Adopting `Codable` + +The following Foundation Swift types will be adopting `Codable`, and will encode as their bridged types when encoded through `NSKeyedArchiver`, as mentioned above: + +* `AffineTransform` +* `Calendar` +* `CharacterSet` +* `Date` +* `DateComponents` +* `DateInterval` +* `Decimal` +* `IndexPath` +* `IndexSet` +* `Locale` +* `Measurement` +* `Notification` +* `PersonNameComponents` +* `TimeZone` +* `URL` +* `URLComponents` +* `URLRequest` +* `UUID` + +Along with these, the `Array`, `Dictionary`, and `Set` types will gain `Codable` conformance (as part of the Conditional Conformance feature), and encode through `NSKeyedArchiver` as `NSArray`, `NSDictionary`, and `NSSet` respectively. + +## Source compatibility + +The majority of this proposal is additive. The changes to `NSKeyedArchiver` are intended to be non-source-breaking changes, and non-behavior-breaking changes for subclasses in Objective-C and Swift. + +## Effect on ABI stability + +The addition of this API will not be an ABI-breaking change. However, this will add limitations for changes in future versions of Swift, as parts of the API will have to remain unchanged between versions of Swift (barring some additions, discussed below). + +## Effect on API resilience + +Much like new API added to the standard library, once added, some changes to this API will be ABI- and source-breaking changes. Changes to the new encoder and decoder classes provided above will be restricted as described in the [library evolution document](https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst) in the Swift repository; in particular, the removal of methods or nested types or changes to argument types will break client behavior. Additionally, additions to provided options `enum`s will be a source-breaking change for users performing an exhaustive switch over their cases; removal of cases will be ABI-breaking. + diff --git a/proposals/0168-multi-line-string-literals.md b/proposals/0168-multi-line-string-literals.md new file mode 100644 index 0000000000..bf8baa4a2a --- /dev/null +++ b/proposals/0168-multi-line-string-literals.md @@ -0,0 +1,125 @@ +# Multi-Line String Literals + +* Proposal: [SE-0168](0168-multi-line-string-literals.md) +* Authors: [John Holdsworth](https://github.com/johnno1962), [Becca Royal-Gordon](https://github.com/beccadax), [Tyler Cloutier](https://github.com/TheArtOfEngineering) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 4.0)** +* Implementation: [apple/swift#8813](https://github.com/apple/swift/pull/8813) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0168-multi-line-string-literals/5715) +* Bugs: [SR-170](https://bugs.swift.org/browse/SR-170), [SR-4701](https://bugs.swift.org/browse/SR-4701), [SR-4708](https://bugs.swift.org/browse/SR-4708), [SR-4874](https://bugs.swift.org/browse/SR-4874) + +## Introduction + +This proposal introduces multi-line string literals to Swift source code. +This has been discussed a few times on swift-evolution most recently +putting forward a number of different syntaxes that could achieve this goal +each of which has their own use case and constituency for discussion. + +[Swift-evolution thread](https://forums.swift.org/t/multi-line-string-literals/409) + +## Motivation + +Multi-line String literals are a common programming-language feature that is, to date, still missing in +Swift. Whether generating XML/JSON messages or building usage messages in Swift scripts, providing string +literals that extend over multiple lines offers a simple way to represent text without having to manually +break lines using string concatenation. Concatenation is ungainly and may result in slow compilation. + +## Proposed solution + +After consideration this proposal puts forward a single simple syntax for inclusion: `"""long strings"""`. +This has the advantage that it is well supported by the syntax highlighters on GitHub and existing editors +and is a relatively minor change to the Swift Lexer. Interpolation would work as before. + +### Long strings + +Long strings are strings delimited by `"""triple quotes"""` that can contain newlines and individual `"` +characters without the need to escape them. + + assert( xml == """ + + + + \(author) + XML Developer's Guide + Computer + 44.95 + 2000-10-01 + An in-depth look at creating applications with XML. + + + """ ) + +To allow free formatting of the literal an indentation stripping operation is applied whereby +any whitespace characters in front of the closing delimiter are removed from each of the lines +in the literal. As part of this process any initial linefeed is also removed. This allows the +developer to paste literal content directly into the string without modification. Some concern +has been expressed about could introduce confusion if the prefixing indentation of each line does +not contain the same whitespace characters, though this can be checked for by a compiler warning. + +## Detailed design + +These changes are envisaged to be mostly confined to the Swift tokeniser: lib/Parse/Lexer.cpp. +The new string literals would be presented to the grammar as simply being string literals. +This has been explored in a PR for a [prototype toolchain](https://github.com/apple/swift/pull/2275) +and seems to be a robust approach. Other details are explored in the prototype such as +escaping the newline in literals resulting in it not being included in the final literal. + +## Impact on existing code + +As this proposal is additive it does not affect existing code. + +## Alternatives considered + +Two other alternative syntaxes were discussed in the swift-evolution thread. +It became apparent that each syntax had its own, at times, non-overlapping +constituency of supporters. + +### Continuation quotes + +The basic idea of continuation quotes is straightforward. If a quoted string literal is not closed by " +before the end of the line and the first non-whitespace character on the next line is " it is taken to +be a continuation of the previous literal. + + let xml = " + " + " + " \(author) + " XML Developer's Guide + " Computer + " 44.95 + " 2000-10-01 + " An in-depth look at creating applications with XML. + " + " + "" + +The advantage of this format is it gives precise control of exactly what is included in the literal. It also +allows code to be formatted in an aesthetically pleasing manner. Its main disadvantage is that some external +editors will not be familiar with the format and will be unable to correctly syntax highlight literals. + +### Heredoc + +Taking a precedent from other languages, a syntax such as the following could be used to introduce +literals into a codebase. + + assert( xml == <<"XML" ) + + + + \(author) + XML Developer's Guide + Computer + 44.95 + 2000-10-01 + An in-depth look at creating applications with XML. + + + XML + +The same indentation stripping rules would be applied as for long strings. This syntax has the +advantage of being able to paste content in directly and the additional advantage that the +literal is separated from the line of code using it, increasing clarity. Its main disadvantage +is a more practical one: it is a more major departure for the compiler in that tokens +in the AST are no longer in source file order. Testing has, however, shown the toolchain +to be surprisingly robust in dealing with this change once a few assertions were removed. + diff --git a/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md b/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md new file mode 100644 index 0000000000..043f58d5b6 --- /dev/null +++ b/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md @@ -0,0 +1,161 @@ +# Improve Interaction Between `private` Declarations and Extensions + +* Proposal: [SE-0169](0169-improve-interaction-between-private-declarations-and-extensions.md) +* Authors: [David Hart](https://github.com/hartbit), [Chris Lattner](https://github.com/lattner) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0169-improve-interaction-between-private-declarations-and-extensions/5692) +* Previous Revision: [1][Revision 1] +* Bug: [SR-4616](https://bugs.swift.org/browse/SR-4616) + +## Introduction + +In Swift 3, a declaration marked `private` may be accessed by anything nested in the scope of the private declaration. For example, a private property or method defined on a struct may be accessed by other methods defined within that struct. + +This model was introduced by [SE-0025](0025-scoped-access-level.md) and with nearly a year of experience using this model, it has worked well in almost all cases. The primary case it falls down is when the implementation of a type is split into a base definition and a set of extensions. Because of the SE-0025 model, extensions to the type are not allowed to access private members defined on that type. + +This proposal recommends extending `private` access control so that members defined in an extension of a type have the same access as members defined on the type itself, so long as the type and extension are in the same source file. We expect this to dramatically reduce the number of uses of `fileprivate` in practice. + +## Motivation + +[SE-0025](0025-scoped-access-level.md) defined the `private` access control level to be used for scoped access, and introduced `fileprivate` for the case when a declaration needs to be visible across declarations, but not only within a file. The goal of the proposal was for `fileprivate` to be used rarely. However, that goal of the proposal has not been realized: Swift encourages developers to use extensions as a logical grouping mechanism and requires them for conditional conformances. Because of this, the SE-0025 design makes `fileprivate` more necessary than expected, and reduces the appeal of using extensions. + +The prevalence of `fileprivate` in practice has caused mixed reactions from Swift developers, culminating in proposal [SE-0159](0159-fix-private-access-levels.md) which suggested reverting the access control model to Swift 2’s design. That proposal was rejected for two reasons: scoped access is something that many developers use and value, and because it was seen as too large of a change given Swift 4’s source compatibility goals. + +In contrast to SE-0159, this proposal is an extremely narrow change (which is almost completely additive) to the SE-0025 model, which embraces the extension-oriented design of Swift. The authors believe that this change will not preclude introduction of submodules in the future. + + +## Detailed Design + +For purposes of access control, extensions to any given type `T` within a file are +considered to be a single access control scope, and if `T` is defined within the +file, the extensions use the same access control scope as `T`. + +This has two ramifications: + +* Declarations in one of these extensions get access to the `private` members of the type. + +* If the declaration in the extension itself is defined as `private`, then they are accessible to declarations in the type, and other extensions of that type (in the same file). + +Here is a simple code example that demonstrates this: + +```swift +struct S { + private var p: Int + + func f() { + use(g()) // ok, g() is accessible within S + } +} + +extension S { + private func g() { + use(p) // ok, g() has access to p, since it is in an extension on S. + } +} + +extension S { + func h() { + use(g()) // Ok, h() has access to g() since it defined in the access control scope for S. + } +} +``` + +Please note: + +* This visibility does **not** extend to subclasses of a class in the same file, it only affects extensions of the type itself. + +* Constrained extensions are extensions, so this visibility **does** extend to them as well. For example, the body of `extension Optional where Wrapped == String { }` would have access to `private` members of Optional, assuming the extension is defined in the same file as `Optional`. + +* This proposal does change the behavior of extensions that are not in the same file as the type - those extensions are merged together into a single access control scope: + +```swift +// FileA.swift +struct A { + private var aMember : Int +} + +// FileB.swift +extension A { + private func foo() { + bar() // ok, foo() does have access to bar() + } +} + +extension A { + private func bar() { + aMember = 42 // not ok, private members may not be accessed outside their file. + } +} +``` + +* This proposal does not change access control behavior for types nested within each other. As in Swift 3, inner types have access to the private members of outer types, but outer types cannot refer to private members of inner types. For example: + +```swift +struct Outer { + private var outerValue = 42 + + struct Inner { + private var innerValue = 57 + + func innerTest(_ o: Outer) { + print(o.outerValue) // still ok. + } + } + + func test(_ i: Inner) { + print(i.innerValue) // still an error + } +} +``` + +## Source Compatibility + +In Swift 3 compatibility mode, the compiler will continue to treat `private` as before. In Swift 4 mode, the compiler will modify the semantics of `private` to follow the rules of this proposal. No migration will be necessary as this proposal merely broadens the visibility of `private`. + +Cases where a type had `private` declarations with the same signature in the same type/extension but in different scopes will produce a compiler error in Swift 4. For example, the following piece of code compiles in Swift 3 compatibility mode but generates a `Invalid redeclaration of 'bar()'` error in Swift 4 mode: + +```swift +struct Foo { + private func bar() {} +} + +extension Foo { + private func bar() {} +} +``` + +## Alternatives Considered + +Access control has been hotly debated on the swift-evolution mailing list, both in the Swift 3 cycle (leading to [SE-0025](0025-scoped-access-level.md) and most recently in Swift 4 which led to [SE-0159](0159-fix-private-access-levels.md). There have been too many proposals to summarize here, including the introduction of a `scoped` keyword. + +The core team has made it clear that most of those proposals are not in scope for discussion in Swift 4 (or any later release), given the significant impact on source compatibility. This is the primary reason for this narrow scope proposal. + +A future direction that may be interesting to consider and debate (as a separate proposal, potentially after Swift 4) is whether extensions within the same file as a type should be treated as parts of the extended type **in general**. This would allow idioms like this, for example: + + +```swift +struct Foo { + var x: Int +} +// ... +extension Foo { + var y: Int +} +``` + +However, this is specifically **not** part of this proposal at this time. It is merely a possible avenue to consider in the future. + + +Another alternative considered is to allow `private` members of a type to be +accessible to extensions outside of the current source file: either within +the current module, or anywhere in the program. This is rejected because it +violates an important principle of our access control system: that `private` is +narrower than `fileprivate`. Allowing `private` to be narrower in some ways, +but broader in other ways (allow access across files) would lead to a more +confusing and fractured model. + + +[Revision 1]: https://github.com/swiftlang/swift-evolution/blob/e0e04f785dbf5bff138b75e9c47bf94e7db28447/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md + + diff --git a/proposals/0170-nsnumber_bridge.md b/proposals/0170-nsnumber_bridge.md new file mode 100644 index 0000000000..3dff41e9b7 --- /dev/null +++ b/proposals/0170-nsnumber_bridge.md @@ -0,0 +1,310 @@ +# NSNumber bridging and Numeric types + +* Proposal: [SE-0170](0170-nsnumber_bridge.md) +* Author: [Philippe Hausler](https://github.com/phausler) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0170-nsnumber-bridging-and-numeric-types/5801) + +##### Revision history + +* **v1** Initial version + +## Introduction + +`NSNumber` has been a strange duck in the Swift world especially when it has come to bridging and interacting with other protocols. An attempt was made to make a type preserving `NSNumber` subclass; however that defeated numerous optimizations in Foundation and also caused some rather unfortunate disparity between where and how the NSNumbers were created. + +## Motivation + +Swift should have a consistent experience across all the SDK. No matter if you are using Objective-C in your app/framework in addition to Swift or not the behavior should be easily understood and consistent. Furthermore performance optimizations like tagged pointers or other fast-path accessors in Objective-C or CoreFoundation based code should just work no matter the context in which a `NSNumber` is created. + +There are some really spooky behaviors as of current; here are a few extreme examples - + +#### Example 1 +```swift +let n = NSNumber(value: Int64.max) +if n is Int16 { + // this is true as of the current build of Swift! +} +``` + +#### Example 2 +```swift +let n = NSNumber(value: Int64.max) +if n is Int16 { + // this is true as of the current build of Swift! +} else if n is Int64 { + // Ideally we should get to here +} +``` + +#### Example 3 +```swift +let n = NSNumber(value: Int64(42)) +if let value = n as? UInt8 { + // This makes sense if NSNumber is viewed as an abstract numeric box +} +``` + +But there are some subtle failures that occur as well - + +#### Example 4 +```swift +// Serialization behind the scenes causes numeric failures if we were 100% strict +open class MyDocument : NSDocument { + var myStateValue: Int16 = 0 + + public static func restoreWindow(withIdentifier identifier: String, state: NSCoder, completionHandler: @escaping (NSWindow?, Error?) -> Swift.Void) { + myStateValue = state.decodeObject(forKey: "myStateValue") as! Int16 // this is expected to pass since we encoded a Int16 value and we expect at least to get an Int16 out. + } + + open func encodeRestorableState(with coder: NSCoder) { + coder.encodeObject(myStateValue, forKey: "myStateValue") + } +} +``` + +#### Example 5 +```swift +// lets say you have a remote server you have no control over except getting JSON requests back +// the format specifies that the response will have a timestamp in a value that is the integral number of milliseconds since 1970 +// e.g. { "timestamp" : 1487980252519 } is valid JSON, but a recent server upgrade changes it to be "timestamp" : 1487980252519.0 } which is also valid JSON + +// initially code could be written validly as + +let payload = JSONSerialization.jsonObject(with: response, options: []) as? [String : Int] + +// but a remote server change could break things if we were strict and requires this code change + +let payload = JSONSerialization.jsonObject(with: response, options: []) as? [String : Double] + +// But the responses are still integral! + +``` + +#### Example 6 +```swift +let itDoesntDoWhatYouThinkItDoes = Int8(NSNumber(value: Int64.max)) // counterintuitively (as legacy baggage of C) this is Int8(-1) +``` + +All of these examples boil down to the simple question: what does `as?` mean for NSNumber. It is worth noting that `is` should follow the same logic as `as?`. + +## Proposed solution + +`as?` for NSNumber should mean "Can I safely express the value stored in this opaque box called a NSNumber as the value I want?". + +#### Solution Example 1 +```swift +// The trivial case of "what you put in the box is what you get out" +let n = NSNumber(value: UInt32(543)) +let v = n as? UInt32 +// v is UInt32(543) +``` + +#### Solution Example 2 +```swift +// The trivial case of the other way around from Solution Example 1 +let v: UInt32 = 543 +let n = v as NSNumber +// n == NSNumber(value: 543) +``` + +#### Solution Example 3 +```swift +// Safe "casting" +let n = NSNumber(value: UInt32(543)) +let v = n as? Int16 +// v is Int16(543) +``` + +In this case `n` is storing a value of 543 obtained from a `UInt32`. Asking the question "Can I safely express the value stored in this opaque box called a `NSNumber` as the value I want?"; we have 543, can it be expressed safely as a `Int16`? Absolutely! + +#### Solution Example 4 +```swift +// Failures when casting that lose the value stored +let n = NSNumber(value: UInt32(543)) +let v = n as? Int8 +// v is nil +``` + +In this case `n` is storing a value of 543 obtained from a `UInt32` but when asked can it be safely represented as the requested type `Int8` the answer is; of course not. + +#### Solution Example 5 +```swift +let v = Int8(NSNumber(value: Int64.max)) +// v is nil because Int64.max cannot be safely expressed as Int8 +``` + + +## Detailed design + +The following methods will be changed in swift 4 because the behavior is truncating, not an assertion of exactly: + +```swift +extension Int8 { + init(_ number: NSNumber) +} + +extension UInt8 { + init(_ number: NSNumber) +} + +extension Int16 { + init(_ number: NSNumber) +} + +extension UInt16 { + init(_ number: NSNumber) +} + +extension Int32 { + init(_ number: NSNumber) +} + +extension UInt32 { + init(_ number: NSNumber) +} + +extension Int64 { + init(_ number: NSNumber) +} + +extension UInt64 { + init(_ number: NSNumber) +} + +extension Float { + init(_ number: NSNumber) +} + +extension Double { + init(_ number: NSNumber) +} + +extension CGFloat { + init(_ number: NSNumber) +} + +extension Bool { + init(_ number: NSNumber) +} + +extension Int { + init(_ number: NSNumber) +} + +extension UInt { + init(_ number: NSNumber) +} +``` + +To the optional variants as such: + +```swift +extension Int8 { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension UInt8 { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension Int16 { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension UInt16 { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension Int32 { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension UInt32 { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension Int64 { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension UInt64 { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension Float { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension Double { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension CGFloat { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension Bool { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension Int { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +extension UInt { + init?(exactly number: NSNumber) + init(truncating number: NSNumber) +} + +``` + +The behavior of exactly will be an exact bitwise representation for stored integer types such that any initialization must be able to express the initialized type with the stored representation in the NSNumber or it will fail and return nil. The truncating versions on the other hand will be similar to fetching the value via that type accessor. e.g. `Int8(truncating: myNSNumber)` will be tantamount to calling `int8Value `on said number + +## Impact on existing code + +The one big repercussion is that the bridging methods cannot be versioned for swift 3 and swift 4 availability so we must pick one implementation or another for the bridging. Swift 4 modules or frameworks must bridge in the same manner as Swift 3 since the implementation is the behavioral difference, not the calling module. + +There are a number of places in frameworks where NSNumbers are used to provide results. For example CoreData queries can potentially yield NSNumbers, Coding and Archival use NSNumbers for storage under the hood, Serialization uses NSNumbers for numeric representations (there are many more that apply these are just a few that can easily lead to corruption or malformed data when users accidentally truncate via the current bridging behavior). + +Overall I am of the belief that any case that this would break existing code it was potentially incorrect behavior on the part of the developer using the bridge of NSNumber and if it is not the facilities of NSNumber are still present (you can still call `NSNumber(value: Int64.max).int8Value` to get `-1` if that is what you really mean. That way is unambiguous and distinctly clear for maintainable code and reduces the overall magic/spooky behavior. + +## Source compatibility + +This does change the result of cast via `as?` so it does not change source compatibility but it does change runtime compatibility. In the end, round tripped NSNumbers into collections that bridge across are relatively rare and it is much more common to get NSNumbers out of APIs. The original `as?` cast required developers to be aware of potential failure by syntax, the change here is a safer way of expressing values for all numbers in one uniform methodology that is both more performant as well as more correct in more common cases. In short the runtime effect is a step in the right direction. + +## Effect on ABI stability + +This change should have no direct impact upon ABI. + +## Effect on API resilience + +The initializers that had no decoration are marked as deprecated so the remaining portion is considered additive only. + +## Considerations for Objective-C platforms + +This brings in line the concepts of the existing Objective-C APIs to the intent that was originally used for NSNumber and the usage. Bridging NSNumbers (for platforms that support it and cases that are supported) should always respect the concept of tagged pointers. In just property list cases alone it is approximately a 2% performance loss to avoid the tagged pointer cases; just by using swift there should be little to no performance penalty for sending these types either direction on the bridge in comparison to the equivalent Objective-C implementations. + +## Considerations for Linux platforms + +We do not have bridging on Linux so the `as?` cast is less important; but if it were to have bridging this would be the desired functionality. + +## Alternatives considered + +We have explored making NSNumbers created in Swift to be strongly type preserving. This unfortunately results in a severe inconsistency between APIs implemented in Swift and those implemented in Objective-C. Instead of having a disparate behavior depending on the spooky action determined in an opaque framework it is better to have a simple story no matter what language the NSNumber was made in and no matter what facility it was created with. To do so we have only one real way of representing numeric types; meet halfway in the middle between erasing all type information and requiring a pedantic matching of type information and always allow a safe expression of the numeric value. + +Using initializers for `Integer` or `FloatingPoint` protocol adopters on NSNumber was considered but was determined out of scope for this change and may be considered separately from the behavior of NSNumber bridging. diff --git a/proposals/0171-reduce-with-inout.md b/proposals/0171-reduce-with-inout.md new file mode 100644 index 0000000000..d1ab3d448c --- /dev/null +++ b/proposals/0171-reduce-with-inout.md @@ -0,0 +1,97 @@ +# Reduce with `inout` + +* Proposal: [SE-0171](0171-reduce-with-inout.md) +* Author: [Chris Eidhof](https://github.com/chriseidhof) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0171-reduce-with-inout/5769) + +## Introduction + +A new variant of `reduce` should be added to the standard library. Instead of taking a `combine` function that is of type `(A, Iterator.Element) -> A`, the full type and implementation of the added `reduce` will be: + +```swift +extension Sequence { + func reduce
(into initial: A, _ combine: (inout A, Iterator.Element) -> ()) -> A { + var result = initial + for element in self { + combine(&result, element) + } + return result + } +} +``` + +Swift-evolution thread: [Reduce with inout](https://forums.swift.org/t/reduce-with-inout/4897) + +## Motivation + +The current version of `reduce` needs to make copies of the result type for each element in the sequence. The proposed version can eliminate the need to make copies (when the `inout` is optimized away). + +## Proposed solution + +The first benefit of the proposed solution is efficiency. The new version of `reduce` can be used to write efficient implementations of methods that work on `Sequence`. For example, consider an implementation of `uniq`, which filters adjacent equal entries: + +```swift +extension Sequence where Iterator.Element: Equatable { + func uniq() -> [Iterator.Element] { + return reduce(into: []) { (result: inout [Iterator.Element], element) in + if result.last != element { + result.append(element) + } + } + } +} +``` + +In the code above, the optimizer will usually optimize the `inout` array into an `UnsafeMutablePointer`, or when inlined, into a simple mutable variable. With that optimization, the complexity of the algorithm is `O(n)`. The same algorithm, but implemented using the existing variant of reduce, will be `O(n²)`, because instead of `append`, it copies the existing array in the expression `result + [element]`: + +```swift +extension Sequence where Iterator.Element: Equatable { + func uniq() -> [Iterator.Element] { + return reduce([]) { (result: [Iterator.Element], element) in + guard result.last != element else { return result } + return result + [element] + } + } +} +``` + +The second benefit is that the new version of `reduce` is more natural when dealing with `mutating` methods. For example, consider a function that computes frequencies in a `Sequence`: + +```swift +extension Sequence where Iterator.Element: Hashable { + func frequencies() -> [Iterator.Element: Int] { + return reduce(into: [:]) { (result: inout [Iterator.Element:Int], element) in + if let value = result[element] { + result[element] = value + 1 + } else { + result[element] = 1 + } + } + } +} +``` + +Without the `inout` parameter, we'd first have to make a `var` copy of the existing result, and have to remember to return that copy instead of the `result`. (The method above is probably clearer when written with a `for`-loop, but that's not the point). + + +## Source compatibility + +This is purely additive, we don't propose removing the existing `reduce`. Additionally, because the first argument will have a label `into`, it doesn't add any extra burden to the type checker. + +## Effect on ABI stability + +N/A + +## Effect on API resilience + +N/A + +## Alternatives considered + +We considered removing the existing `reduce`, but the problem with that is two-fold. First, removing it breaks existing code. Second, it's useful for algorithms that don't use mutating methods within `combine`. We considered overloading `reduce`, but that would stress the type checker too much. + +There has been a really active discussion about the naming of the first parameter. Naming it `mutating:` could deceive people into thinking that the value would get mutated in place. Naming it `mutatingCopyOf:` is also tricky: even though a copy of the struct gets mutated, copying is always implicit when using structs, and it wouldn't copy an instance of a class. `into:` seems the best name so far. + +Under active discussion: the naming of this method. See the [swift-evolution thread](https://forums.swift.org/t/reduce-with-inout/4897). diff --git a/proposals/0172-one-sided-ranges.md b/proposals/0172-one-sided-ranges.md new file mode 100644 index 0000000000..0864e86a10 --- /dev/null +++ b/proposals/0172-one-sided-ranges.md @@ -0,0 +1,235 @@ +# One-sided Ranges + +* Proposal: [SE-0172](0172-one-sided-ranges.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Dave Abrahams](https://github.com/dabrahams), [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0172-one-sided-ranges/5768) + +## Introduction + +This proposal introduces the concept of a "one-sided" range, created via +prefix/postfix versions of the existing range operators. + +It also introduces a new protocol, `RangeExpression`, to simplify the creation +of methods that take different kinds of ranges. + +## Motivation + +It is common, given an index into a collection, to want a slice up to or from +that index versus the start/end. + +For example (assuming `String` is once more a `Collection`): + +```swift +let s = "Hello, World!" +let i = s.index(of: ",")! +let greeting = s[s.startIndex..` suitable for slicing `collection`. + /// The return value is *not* guaranteed to be inside + /// its bounds. Callers should apply the same preconditions + /// to the return value as they would to a range provided + /// directly by the user. + func relative(to collection: C) -> Range where C.Index == Bound + + func contains(_ element: Bound) -> Bool +} + +extension RangeExpression { + public static func ~= (pattern: Self, value: Bound) -> Bool +} + +prefix operator ..< +public struct PartialRangeUpTo: RangeExpression { + public init(_ upperBound: T) { self.upperBound = upperBound } + public let upperBound: T +} +extension Comparable { + public static prefix func ..<(x: Self) -> PartialRangeUpTo +} + +prefix operator ... +public struct PartialRangeThrough: RangeExpression { + public init(_ upperBound: T) + public let upperBound: T +} +extension Comparable { + public static prefix func ...(x: Self) -> PartialRangeThrough +} + +postfix operator ... +public struct PartialRangeFrom: RangeExpression { + public init(_ lowerBound: T) + public let lowerBound: T +} +extension Comparable { + public static postfix func ...(x: Self) -> PartialRangeFrom +} + +// The below relies on Conditional Conformance. Pending that feature, +// this may require an additional CountablePartialRangeFrom type temporarily. +extension PartialRangeFrom: Sequence + where Index: _Strideable, Index.Stride : SignedInteger + + +extension Collection { + public subscript(r: R) -> SubSequence + where R.Bound == Index { get } +} +extension MutableCollection { + public subscript(r: R) -> SubSequence + where R.Bound == Index { get set } +} + +extension RangeReplaceableCollection { + public mutating func replaceSubrange( + _ subrange: ${Range}, with newElements: C + ) where C.Iterator.Element == Iterator.Element, R.Bound == Index + + public mutating func removeSubrange( + _ subrange: ${Range} + ) where R.Bound == Index +} +``` + +Additionally, these new ranges will implement appropriate protocols such as +`CustomStringConvertible`. + +It is important to note that these new methods and range types are _extensions +only_. They are not protocol requirements, as they should not need to be +customized for specific collections. They exist only as shorthand to expand +out to the full slicing operation. + +Where `PartialRangeFrom` is a `Sequence`, it is left up to the type of `Index` +to control the behavior when the type is incremented past its bounds. In the +case of an `Int`, the iterator will trap when iterating past `Int.max`. Other +types, such as a `BigInt` that could be incremented indefinitely, would behave +differently. + +The `prefix` and `suffix` methods that take an index _are_ currently protocol +requirements, but should not be. This proposal will fix that as a side-effect. + +## Source compatibility + +The new operators/types are purely additive so have no source compatibility +consequences. Replacing the overloads taking concrete ranges other than `Range` +with a single generic version is source compatible. `prefix` and `suffix` will +be deprecated in Swift 4 and later removed. + +## Effect on ABI stability + +The `prefix`/`suffix` methods being deprecated should be eliminated before +declaring ABI stability. + +## Effect on API resilience + +The new operators/types are purely additive so have no resilience +consequences. + +## Alternatives considered + +`i...` is favored over `i..<` because the latter is ugly. We have to pick one, +two would be redundant and likely to cause confusion over which is the "right" one. +Either would be reasonable on pedantic correctness grounds – `(i as Int)...` +includes `Int.max` consistent with `...`, whereas `a[i...]` is interpreted as +`a[i.. "Omit all labels when arguments can’t be usefully distinguished” + +and: + +> "When the first argument forms part of a prepositional phrase, give it an argument label...An exception arises when the first two arguments represent parts of a single abstraction….In such cases, begin the argument label after the preposition, to keep the abstraction clear." + + diff --git a/proposals/0174-filter-range-replaceable.md b/proposals/0174-filter-range-replaceable.md new file mode 100644 index 0000000000..ead1768f74 --- /dev/null +++ b/proposals/0174-filter-range-replaceable.md @@ -0,0 +1,110 @@ +# Change `RangeReplaceableCollection.filter` to return `Self` + +* Proposal: [SE-0174](0174-filter-range-replaceable.md) +* Author: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.2)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0174-change-filter-to-return-an-associated-type/5866) +* Bug: [SR-3444](https://bugs.swift.org/browse/SR-3444) + +## Introduction + +This proposal implements the `filter` operation on `RangeReplaceableCollection` +to return the same type as the filtered collection. + +## Motivation + +The recently accepted +[SE-165](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0165-dict.md) +introduced a version of `filter` on `Dictionary` that returned a +`Dictionary`. This had both performance and usability benefits: in most cases, +a `Dictionary` is what the user wanted from the filter, and creating one +directly from the filter operation is much more efficient than first creating +an array then creating a `Dictionary` from it. + +However, this does result in some inconsistencies. Users may be surprised that +this one specific collection returns `Self`, while other collections that would +benefit from the same change still return `[Element]`. And some collections, +such as `String`, might also benefit from usability and performance win similar +to `Dictionary`. Additionally, these wins will be lost in generic code – if you +pass a `Dictionary` into an algorithm that takes a `Sequence`, then when you +filter it, you will still get an `Array`. + +## Proposed solution + +An implementation of `filter` on `RangeReplaceableCollection` will be provided, +using `init()` and `append(_:)`, so all range-replaceable collections will +have a `filter` method returning of `Self`. Per [SE-163](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0163-string-revision-1.md), +this will include `String`. + +Note, many sequences (for example, strides or ranges), cannot represent a +filtered `self` as `Self` and will continue to return an array. If this is a +performance problem, `lazy` remains a good solution. + +## Detailed design + +Add a default implementation of `filter` to `RangeReplaceableCollection` +returning `Self`: + +```swift +extension RangeReplaceableCollection { + func filter(_ isIncluded: (Iterator.Element) throws -> Bool) rethrows -> Self { + var result = Self() + for element in self { + if try isIncluded(element) { + result.append(element) + } + } + return result + } +} +``` + +Specific concrete collections may choose to implement a faster version, but +this is an implementation detail. + +## Source compatibility + +This change is subtly source breaking. In most cases users will not notice. +They may be be relying on an array being returned (albeit often in order to +then transform it back into the original type), but this version will still +be available (via the extension on `Sequence`) and will be called if forced +through type context. The only code that will break is if this operation spans +multiple lines: + +```swift +// filtered used to be [Character], now String +let filtered = "abcd".filter { $0 == "a" } +useArray(filtered) // won't compile +``` + +Because of this, the new implementation of `RangeReplaceableCollection.filter` +will only be available in Swift 4. + +## Effect on ABI stability + +This change will affect the ABI of `RangeReplaceableCollection` and needs to be made before +declaring ABI stability. + +## Effect on API resilience + +N/A + +## Alternatives considered + +Status-quo. There are benefits to the consistency of always returning `[Element]`. +The version on `Sequence` can be reached via type context (`"abc".filter(predicate) as [Element]`). + +It could be worthwhile to make a similar change to `map`, but this is beyond +the capabilities of the current generics system because `map` does not preserve +the element type (more specifically, you cannot express a type that is `Self` +except with a different `Element` in order to provide the +implementation on `RangeReplaceableCollection`). + +## History + +This proposal originally included a new associated type `Filtered` on `Sequence`. However, this +was unimplementable due to requiring a recursive type constraint (`Filtered: Sequence`). While +these were supported in later Swift versions, the additional associated type was not implemented +and that portion of the proposal has [expired](https://forums.swift.org/t/addressing-unimplemented-evolution-proposals/). + diff --git a/proposals/0175-package-manager-revised-dependency-resolution.md b/proposals/0175-package-manager-revised-dependency-resolution.md new file mode 100644 index 0000000000..9dbc7e116f --- /dev/null +++ b/proposals/0175-package-manager-revised-dependency-resolution.md @@ -0,0 +1,94 @@ +# Package Manager Revised Dependency Resolution + +* Proposal: [SE-0175](0175-package-manager-revised-dependency-resolution.md) +* Author: [Rick Ballard](https://github.com/rballard) +* Review Manager: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0175-package-manager-revised-dependency-resolution/5896) + +## Introduction +This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (`swift package pin` & `swift package unpin`), replaces the `swift package fetch` command with a new `swift package resolve` command with improved behavior, and replaces the optional `Package.pins` file with a `Package.resolved` file which is always created during dependency resolution. + +## Motivation +When [SE-0145 Package Manager Version Pinning](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md) was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like `swift package update`, which has a `--repin` flag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing. + +In the existing design, when autopinning is on (which is true by default) the `swift package pin` command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the `Package.swift` manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning. + +The problems here stem from trying to use one mechanism (pinning) to solve two different use cases: wanting to record and share resolved dependency versions, vs wanting to keep a badly-behaved package at a specific version. We think the package manager could be simplified by splitting these two use cases out into different mechanisms ("resolved versions" vs "pinning"), instead of using an "autopinning" option which makes these two features mutually-exclusive and confusing. + +Additionally, some dependency resolution behaviors were not well-specified and do not behave well. The package manager is lax about detecting changes to the versions specified in the `Package.swift` manifest or `Package.pins` pinfile, and fails to automatically update packages when needed, or to issue errors if the version requirements are unsatisfiable, until the user explicitly runs `swift package update`, or until a new user without an existing checkout attempts to build. We'd like to clarify and revise the rules around when and how the package manager performs dependency resolution. + +## Proposed solution +The pinning feature will be removed. This removes the `swift package pin` and `swift package unpin` commands, the `--repin` flag to `swift package update`, and use of the `Package.pins` file. + +In a future version of the package manager we may re-introduce pinning. If we do, pins will only be recorded in the `Package.pins` file when explicitly set with `swift package pin`, and any pinned dependencies will _not_ be updated by the `swift package update` command; instead, they would need to be unpinned to be updated. This would be a purely additive feature which packages could use in addition to the resolved versions feature when desired. + +A new "resolved versions" feature will be added, which behaves very similarly to how pinning previously behaved when autopinning was on. The version of every resolved dependency will be recorded in a `Package.resolved` file in the top-level package, and when this file is present in the top-level package it will be used when performing dependency resolution, rather than the package manager finding the latest eligible version of each package. `swift package update` will update all dependencies to the latest eligible versions and update the `Package.resolved` file accordingly. + +Resolved versions will always be recorded by the package manager. Some users may chose to add the `Package.resolved` file to their package's `.gitignore` file. When this file is checked in, it allows a team to coordinate on what versions of the dependencies they should use. If this file is gitignored, each user will separately choose when to get new versions based on when they run the `swift package update` command, and new users will start with the latest eligible version of each dependency. Either way, for a package which is a dependency of other packages (e.g. a library package), that package's `Package.resolved` file will not have any effect on its client packages. + +The existing `swift package fetch` command will be deprecated, removed from the help message, and removed completely in a future release of the Package Manager. In its place, a new `swift package resolve` command will be added. The behavior of `resolve` will be to resolve dependencies, taking into account the current version restrictions in the `Package.swift` manifest and `Package.resolved` resolved versions file, and issuing an error if the graph cannot be resolved. For packages which have previously resolved versions recorded in the `Package.resolved` file, the `resolve` command will resolve to those versions as long as they are still eligible. If the resolved versions file changes (e.g. because a teammate pushed a new version of the file) the next `resolve` command will update packages to match that file. After a successful `resolve` command, the checked out versions of all dependencies and the versions recorded in the resolved versions file will match. In most cases the `resolve` command will perform no changes unless the `Package.swift` manifest or `Package.resolved` file have changed. + +The following commands will implicitly invoke the `swift package resolve` functionality before running, and will cancel with an error if dependencies cannot be resolved: + +* `swift build` +* `swift test` +* `swift package generate-xcodeproj` + +The `swift package show-dependencies` command will also implicitly invoke `swift package resolve`, but it will show whatever information about the dependency graph is available even if the resolve fails. + +The `swift package edit` command will implicitly invoke `swift package resolve`, but if the resolve fails yet did identify and fetch a package with the package name the command supplied, the command will allow that package to be edited anyway. This is useful if you wish to use the `edit` command to edit version requirements and fix an unresolvable dependency graph. `swift package unedit` will unedit the package and _then_ perform a `resolve`. + +## Detailed design +The `resolve` command is allowed to automatically add new dependencies to the resolved versions file, and to remove dependencies which are no longer in the dependency graph. It can also automatically update the recorded versions of any package whose previously-resolved version is no longer allowed by the version requirements from the `Package.swift` manifests. When changed version requirements force a dependency to be automatically re-resolved, the latest eligible version will be chosen; any other dependencies affected by that change will prefer to remain at their previously-resolved versions as long as those versions are eligible, and will otherwise update likewise. + +The `Package.resolved` resolved versions file will record the git revision used for each resolved dependency in addition to its version. In future versions of the package manager we may use this information to detect when a previously-resolved version of a package resolves to a new revision, and warn the user if this happens. + +The `swift package resolve` command will not actually perform a `git fetch` on any dependencies unless it needs to in order to correctly resolve dependencies. As such, if all dependencies are already resolved correctly and allowed by the version constraints in the `Package.swift` manifest and `Package.resolved` resolved versions file, the `resolve` command will not need to do anything (e.g. a normal `swift build` won't hit the network or make unnecessary changes during its implicit `resolve`). + +If a dependency is in edit mode, it is allowed to have a different version checked out than that recorded in the resolved versions file. The version recorded for an edited package will not change automatically. If a `swift package update` operation is performed while any packages are in edit mode, the versions of those edited packages will be removed from the resolved versions file, so that when those packages leave edit mode the next resolution will record a new version for them. Any packages in the dependency tree underneath an edited package will also have their resolved version removed by `swift package update`, as otherwise the resolved versions file might record versions that wouldn't have been chosen without whatever edited package modifications have been made. + +## Alternatives considered + +We considered repurposing the existing `fetch` command for this new behavior, instead of renaming the command to `resolve`. However, the name `fetch` is defined by `git` to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name `fetch`. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word `fetch` in flag names that control that behavior. + +We considered continuing to write out the `Package.pins` file for packages whose [Swift tools version](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md) was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools. + +We considered keeping the `pin` and `unpin` commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the `Package.swift` manifest). + +We considered using an `install` verb instead of `resolve`, as many other package managers use `install` for a very similar purpose. However, almost all of those package managers are for non-compiled languages, where downloading the source to a dependency is functionally equivalent to "installing" it as a product ready for use. In contrast, Swift is a compiled language, and our dependencies must be built (e.g. into libraries) before they can be installed. As such, `install` would be a misnomer for this workflow. In the future we may wish to add an `install` verb which actually does install built products, similar to `make install`. + +### Why we didn't use "Package.lock" + +We considered using the `.lock` file extension for the new resolved versions file, to be consistent with many other package managers. We expect that the decision not to use this extension will be controversial, as following established precedent is valuable. However, we think that a "lockfile" is a very poor name for this concept, and that using that name would cause confusion when we re-introduce pins. Specifically: + +- Calling this a "lock" implies a stronger lockdown of dependencies than is supported by the actual behavior. As a simple `update` command will reset the locks, and a change to the specified versions in `Package.swift` will override them, they're not really "locked" at all. This is misleading. +- When we re-introduce pinning, it would be very confusing to have both "locks" and "pins". Having "resolved versions" and "pins" is not so confusing. +- The term "lock" is already overloaded between POSIX file locks and locks in concurrent programming. + + +For comparison, here is a list of other package managers which implement similar behavior and their name for this file: + +| Package Manager | Language | Resolved versions file name | +| --- | --- | --- | +| Yarn | JS | yarn.lock | +| Composer | PHP | composer.lock | +| Cargo | Rust | Cargo.lock | +| Bundler | Ruby | Gemfile.lock | +| CocoaPods | ObjC/Swift | Podfile.lock | +| Glide | Go | glide.lock | +| Pub | Dart | pubspec.lock | +| Mix | Elixir | mix.lock | +| rebar3 | Erlang | rebar.lock | +| Carton | Perl | carton.lock | +| Carthage | ObjC/Swift | Cartfile.resolved | +| Pip | Python | requirements.txt | +| NPM | JS | npm-shrinkwrap.json | +| Meteor | JS | versions | + +Some arguments for using ".lock" instead of ".resolved" are: + +- Users of other package managers will already be familiar with the terminology and behavior. +- For packages which support multiple package managers, it will be possible to put "\*.lock" into the gitignore file instead of needing a separate entry for "\*.resolved". + +However, we do not feel that these arguments outweigh the problems with the term "lock". If providing feedback asking that we reconsider this decision, please be clear about why the above decision is incorrect, with new information not already considered. diff --git a/proposals/0176-enforce-exclusive-access-to-memory.md b/proposals/0176-enforce-exclusive-access-to-memory.md new file mode 100644 index 0000000000..f11512ff96 --- /dev/null +++ b/proposals/0176-enforce-exclusive-access-to-memory.md @@ -0,0 +1,748 @@ +# Enforce Exclusive Access to Memory + +* Proposal: [SE-0176](0176-enforce-exclusive-access-to-memory.md) +* Author: [John McCall](https://github.com/rjmccall) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 4.0)** +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7e6816c22a29b0ba9bdf63ff92b380f9e963860a/proposals/0176-enforce-exclusive-access-to-memory.md) +* Previous Discussion: [Email Thread](https://forums.swift.org/t/review-se-0176-enforce-exclusive-access-to-memory/5836) + +## Introduction + +In Swift 3, it is possible to modify a variable while it's being used or +modified by another part of the program. This can lead to unexpected and +confusing results. It also forces a great deal of conservatism onto the +implementation of the compiler and the standard libraries, which must +generally ensure the basic soundness of the program (no crashes or +undefined behavior) even in unusual circumstances. + +We propose that Swift should instead enforce a general rule that potential +modifications of variables must be exclusive with any other access to that +variable. + +This proposal is a core part of the Ownership feature, which was described +in the [ownership manifesto](https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md). +That document presents the high-level objectives of Ownership, develops +a more rigorous theory of memory access in Swift, and applies it in detail +to a variety of different language features. In that document, the rule +we're proposing here is called the Law of Exclusivity. We will not be +going into that level of detail in this proposal. Instead, we will +lay out the basic rule, how it will be enforced, and the implications for +programming in Swift. It should be possible to understand this proposal +without actually having read the ownership manifesto at all. That said, +if you are interested in the technical details, that document is probably +the right place to turn. + +## Motivation + +### Instantaneous and non-instantaneous accesses + +On a basic level, Swift is an imperative language which allows programmers +to directly access mutable memory. + +Many of the language features that access memory, like simply loading +from or assigning to a variable, are "instantaneous". This means that, +from the perspective of the current thread, the operation completes +without any other code being able to interfere. For example, when you +assign to a stored property, the current value is just replaced with +the new value. Because arbitrary other code can't run during an +instantaneous access, it's never possible for two instantaneous accesses +to overlap each other (without introducing concurrency, which we'll +talk about later). That makes them very easy to reason about. + +However, not all accesses are instantaneous. For example, when you +call a ``mutating`` method on a stored property, it's really one long +access to the property: ``self`` just becomes another way of referring +to the property's storage. This access isn't instantaneous because +all of the code in the method executes during it, so if that code +manages to access the same property again, the accesses will overlap. +There are several language features like this already in Swift, and +Ownership will add a few more. + +### Examples of problems due to overlap + +Here's an example: + +```swift +// These are simple global variables. +var global: Int = 0 +var total: Int = 0 + +extension Int { + // Mutating methods access the variable they were called on + // for the duration of the method. + mutating func increaseByGlobal() { + // Any accesses they do will overlap the access to that variable. + + total += self // Might access 'total' through both 'total' and 'self' + self += global // Might access 'global' through both 'global' and 'self' + } +} +``` + +If ``self`` is ``total`` or ``global``, the low-level semantics of this +method don't change, but the programmer's high-level understanding +of it almost certainly does. A line that superficially seems to not +change 'global' might suddenly start doubling it! And the data dependencies +between the two lines instantly go from simple to very complex. That's very +important information for someone maintaining this method, who might be +tempted to re-arrange the code in ways that seem equivalent. That kind of +maintenance can get very frustrating because of overlap like this. + +The same considerations apply to the language implementation. +The possibility of overlap means the language has to make +pessimistic assumptions about the loads and stores in this method. +For example, the following code avoids a seemingly-redundant load, +but it's not actually equivalent because of overlap: + +```swift + let value = self + total += value + self = value + global +``` + +Because these variables just have type ``Int``, the cost of this pessimism +is only an extra load. If the types were more complex, like ``String``, +it might mean doing extra copies of the ``String`` value, which would +translate to extra retains and releases of the string's buffer; in a more +complex example, that could even lead to the underlying data being copied +unnecessarily. + +In the above examples, we've made the potentially-overlapping accesses +obvious, but they don't have to be. For example, here is another method +that takes a closure as an argument: + +```swift +extension Array { + mutating func modifyElements(_ closure: (inout Element) -> ()) { + var i = startIndex + while i != endIndex { + closure(&self[i]) + i = index(after: i) + } + } +} +``` + +This method's implementation seems straightforwardly correct, but +unfortunately it doesn't account for overlap. Absolutely nothing +prevents the closure from modifying ``self`` during the iteration, +which means that ``i`` can suddenly become an invalid index, which +could lead to all sorts of unwanted behavior. Even if this never +happen in reality, the fact that it's *possible* means that the +implementation is blocked from pursuing all sorts of important +optimizations. + +For example, the compiler has an optimization that "hoists" the +uniqueness check on a copy-on-write collection from the inside of +a loop (where it's run on each iteration) to the outside (so that +it's only checked once, before the loop begins). But that optimization +can't be applied in this example because the closure might change or +copy ``self``. The only realistic way to tell the compiler that +that can't happen is to enforce exclusivity on ``self``. + +The same considerations that apply to ``self`` in a ``mutating`` +method also apply to ``inout`` parameters. For example: + +```swift +open class Person { + open var title: String +} + +func collectTitles(people: [Person], into set: inout Set) { + for person in people { + set.insert(person.title) + } +} +``` + +This function mutates a set of strings, but it also repeatedly +calls a class method. The compiler cannot know how this method +is implemented, because it is ``open`` and therefore overridable +from an arbitrary module. Therefore, because of overlap, the +compiler must pessimistically assume that each of these method +calls might somehow find a way to modify the original variable +that ``set`` was bound to. (And if the method did manage to do +so, the resulting strange behavior would probably be seen as a bug +by the caller of ``collectTitles``.) + +### Eliminating non-instantaneous accesses? + +If non-instantaneous accesses create all of these problems with +overlapping accesses, should we just eliminate non-instantaneous +accesses completely? Well, no, and there's two big reasons why not. +In order to make something like a ``mutating`` method not access +the original storage of ``self`` for the duration of the method, +we would need to make it access a temporary copy instead, which +we would assign back to the storage after the method is complete. +That is, suppose we had the following Swift code: + +```swift +var numbers = [Int]() +numbers.appendABunchOfStuff() +``` + +Currently, behind the scenes, this is implemented somewhat like +the following C code: + +```c +struct Array numbers = _Array_init(); +_Array_appendABunchOfStuff(&numbers); +``` + +You can see clearly how ``_Array_appendABunchOfStuff`` will be working +directly with the storage of ``numbers``, creating the abstract +possibility of overlapping accesses to that variable. To prevent +this in general, we would need to pass a temporary copy instead: + +```c +struct Array numbers = _Array_init(); +struct Array temp = _Array_copy(numbers); +_Array_appendABunchOfStuff(&temp); +_Array_assign(&numbers, temp); +``` + +Like we said, there's two big problems with this. + +The first problem is that it's awful for performance. Even for a +normal type, doing extra copies is wasteful, but doing it with ``Array`` +is even worse because it's a copy-on-write type. The extra copy here +means that there will be multiple references to the buffer, which +means that ``_Array_appendABunchOfStuff`` will be forced to copy the +buffer instead of modifying it in place. Removing these kinds of +copies, and making it easier to reason about when they happen, is a +large part of the goal of the Ownership feature. + +The second problem is that it doesn't even eliminate the potential +confusion. Suppose that ``_Array_appendABunchOfStuff`` somehow +reads or writes to ``numbers`` (perhaps because ``numbers`` is +captured in a closure, or it's actually a global variable or a class +property or something else that can be potentially accessed from +anywhere). Because the method is now modifying the copy in ``temp``, +any reads it makes from ``numbers`` won't see any of the changes +it's made to ``temp``, and any changes it makes to ``numbers`` will +be silently lost when it returns and the caller unconditionally +overwrites ``numbers`` with ``temp``. + +### Consequences of non-instantaneous accesses + +So we have to accept that accesses can be non-instantaneous. That +means programmers can write code that would naturally cause overlapping +accesses to the same variable. We currently allow this to happen +and make a best effort to live with the consequences. The costs, +in general, are a lot of complexity and lost performance. + +For example, the ``Array`` type has an optimization in its ``subscript`` +operator which allows callers to directly access the storage of array +elements. This is a very important optimization which, among other +things, allows arrays to efficiently hold values of copy-on-write types. +However, because the caller can execute arbitrary code while they're +working with the array element storage, and that code might do something +like assign a new value to the original array variable and therefore drop +the last reference to the array buffer, this optimization has to create a +new strong reference to the buffer until the caller is done with the element, +which itself causes a whole raft of complexity. + +Similarly, when the compiler is optimizing a ``mutating`` method, it has +to assume that an arbitrary call might completely rewrite ``self``. +This makes it very difficult to perform any meaningful optimization +at all, especially in generic code. It also means that the compiler +must generally emit a large number of conservative copies just in case +things are modified in unexpected ways. + +Furthermore, the possibility of overlapping accesses has a continued +impact on language evolution. Many of the features laid out in the +Ownership manifesto rely on static guarantees that Swift simply cannot +make without stronger rules about when a variable can be modified. + +Therefore we think it best to simply disallow overlapping accesses +as best as we can. + +## Proposed solution + +We should add a rule to Swift that two accesses to the same variable +are not allowed to overlap unless both accesses are reads. By +"variable", we mean any kind of mutable memory: global variables, +local variables, class and struct properties, and so on. + +This rule should be enforced as strongly as possible, depending on +what sort of variable it is: + +* Local variables, inout parameters, and struct properties can + generally enforce the rule statically. The compiler can analyze + all the accesses to the variable and emit an error if it sees + any conflicts. + +* Class properties and global variables will have to enforce the + rule dynamically. The runtime can keep track of what accesses + are underway and report any conflicts. Local variables will + sometimes have to use dynamic enforcement when they are + captured in closures. + +* Unsafe pointers will not use any active enforcement; it is the + programmer's responsibility to follow the rule. + +* No enforcement is required for immutable memory, like a ``let`` + binding or property, because all accesses must be reads. + +Examples: + +```swift +var x = 0, y = 0 + +// NOT A CONFLICT. These two accesses to 'x' are both reads. +// Each completes instantaneously, so the accesses do not overlap and +// therefore do not conflict. Even if they were not instantaneous, they +// are both reads and therefore do no conflict. +let z = x + x + +// NOT A CONFLICT. The right-hand side of the assignment is a read of +// 'x' which completes instantaneously. The assignment is a write to 'x' +// which completes instantaneously. The accesses do not overlap and +// therefore do not conflict. +x = x + +// NOT A CONFLICT. The right-hand side is a read of 'x' which completes +// instantaneously. Calling the operator involves passing 'x' as an inout +// argument; this is a write access for the duration of the call, but it does +// not begin until immediately before the call, after the right-hand side is +// fully evaluated. Therefore the accesses do not overlap and do not conflict. +x += x + +// CONFLICT. Passing 'x' as an inout argument is a write access for the +// duration of the call. Passing the same variable twice means performing +// two overlapping write accesses to that variable, which therefore conflict. +swap(&x, &x) + +extension Int { + mutating func assignResultOf(_ function: () -> Int) { + self = function() + } +} + +// CONFLICT. Calling a mutating method on a value type is a write access +// that lasts for the duration of the method. The read of 'x' in the closure +// is evaluated while the method is executing, which means it overlaps +// the method's formal access to 'x'. Therefore these accesses conflict. +x.assignResultOf { x + 1 } +``` + +## Detailed design + +### Concurrency + +Swift has always considered read/write and write/write races on the same +variable to be undefined behavior. It is the programmer's responsibility +to avoid such races in their code by using appropriate thread-safe +programming techniques. + +We do not propose changing that. Dynamic enforcement is not required to +detect concurrent conflicting accesses, and we propose that by default +it should not make any effort to do so. This should allow the dynamic +bookkeeping to avoid synchronizing between threads; for example, it +can track accesses in a thread-local data structure instead of a global +one protected by locks. Our hope is that this will make dynamic +access-tracking cheap enough to enable by default in all programs. + +The implementation should still be *permitted* to detect concurrent +conflicting accesses, of course. Some programmers may wish to use an +opt-in thread-safe enforcement mechanism instead, at least in some +build configurations. + +Any future concurrency design in Swift will have the elimination +of such races as a primary goal. To the extent that it succeeds, +it will also define away any specific problems for exclusivity. + +### Value types + +Calling a method on a value type is an access to the entire value: +a write if it's a ``mutating`` method, a read otherwise. This is +because we have to assume that a method might read or write an +arbitrary part of the value. Trying to formalize rules like +"this method only uses these properties" would massively complicate +the language. + +For similar reasons, using a computed property or subscript on a +value type generally has to be treated as an access to the entire +value. Whether the access is a read or write depends on how the +property/subscript is used and whether either the getter or the setter +is ``mutating``. + +Accesses to different stored properties of a ``struct`` or different +elements of a tuple are allowed to overlap. However, note that +modifying part of a value type still requires exclusive access to +the entire value, and that acquiring that access might itself prevent +overlapping accesses. For example: + +```swift +struct Pair { + var x: Int + var y: Int +} + +class Paired { + var pair = Pair(x: 0, y: 0) +} + +let object = Paired() +swap(&object.pair.x, &object.pair.y) +``` + +Here, initiating the write-access to ``object.pair`` for the first +argument will prevent the write-access to ``object.pair`` for the +second argument from succeeding because of the dynamic enforcement +used for the property. Attempting to make dynamic enforcement +aware of the fact that these accesses are modifying different +sub-components of the property would be prohibitive, both in terms +of the additional performance cost and in terms of the complexity +of the implementation. + +However, this limitation can be worked around by binding +``object.pair`` to an ``inout`` parameter: + +```swift +func modifying(_ value: inout T, _ function: (inout T) -> ()) { + function(&value) +} + +modifying(&object.pair) { pair in swap(&pair.x, &pair.y) } +``` + +This works because now there is only a single access to +``object.pair`` and because, once the the ``inout`` parameter is +bound to that storage, accesses to the parameter within the +function can use purely static enforcement. + +We expect that workarounds like this will only rarely be required. + +Note that two different properties can only be assumed to not +conflict when they are both known to be stored. This means that, +for example, it will not be allowed to have overlapping accesses +to different properties of a resilient value type. This is not +expected to be a significant problem for programmers. + +### Arrays + +Collections do not receive any special treatment in this proposal. +For example, ``Array``'s indexed subscript is an ordinary computed +subscript on a value type. Accordingly, mutating an element of an +array will require exclusive access to the entire array, and +therefore will disallow any other simultaneous accesses to the +array, even to different elements. For example: + +```swift +var array = [[1,2], [3,4,5]] + +// NOT A CONFLICT. These accesses to the elements of 'array' each +// complete instantaneously and do not overlap each other. Even if they +// did overlap for some reason, they are both reads and therefore +// do not conflict. +print(array[0] + array[1]) + +// NOT A CONFLICT. The access done to read 'array[1]' completes +// before the modifying access to 'array[0]' begins. Therefore, these +// accesses do not conflict. +array[0] += array[1] + +// CONFLICT. Passing 'array[i]' as an inout argument performs a +// write access to it, and therefore to 'array', for the duration of +// the call. This call makes two such accesses to the same array variable, +// which therefore conflict. +swap(&array[0], &array[1]) + +// CONFLICT. Calling a non-mutating method on 'array[0]' performs a +// read access to it, and thus to 'array', for the duration of the method. +// Calling a mutating method on 'array[1]' performs a write access to it, +// and thus to 'array', for the duration of the method. These accesses +// therefore conflict. +array[0].forEach { array[1].append($0) } +``` + +It's always been somewhat fraught to do simultaneous accesses to +an array because of copy-on-write. The fact that you should not +create an array and then fork off a bunch of threads that assign +into different elements concurrently has been independently +rediscovered by a number of different programmers. (Under this +proposal, we will still not be reliably detecting this problem +by default, because it is a race condition; see the section on +concurrency.) The main new limitation here is that some idioms +which did work on a single thread are going to be forbidden. +This may just be a cost of progress, but there are things we +can do to mitigate the problem. + +In the long term, the API of ``Array`` and other collections +should be extended to ensure that there are good ways of achieving +the tasks that exclusivity enforcement has made difficult. +It will take experience living with exclusivity in order to +understand the problems and propose the right API additions. +In the short term, these problems can be worked around with +``withUnsafeMutableBufferPointer``. + +We do know that swapping two array elements will be problematic, +and accordingly we are (separately proposing)[https://github.com/swiftlang/swift-evolution/blob/master/proposals/0173-swap-indices.md] to add a +``swapAt`` method to ``MutableCollection`` that takes two indices +rather than two ``inout`` arguments. The Swift 3 compatibility +mode should recognize the swap-of-elements pattern and automatically +translate it to use ``swapAt``, and the 3-to-4 migrator should +perform this rewrite automatically. + +### Class properties + +Unlike value types, calling a method on a class doesn't formally access +the entire class instance. In fact, we never try to enforce exclusivity +of access on the whole object at all; we only enforce it for individual +stored properties. Among other things, this means that an access to a +class property never conflicts with an access to a different property. + +There are two major reasons for this difference between value +and reference types. + +The first reason is that it's important to allow overlapping method +calls to a single class instance. It's quite common for an object +to have methods called on it concurrently from different threads. +These methods may access different properties, or they may synchronize +their accesses to the same properties using locks, dispatch queues, +or some other thread-safe technique. Regardless, it's a widespread +pattern. + +The second reason is that there's no benefit to trying to enforce +exclusivity of access to the entire class instance. For a value +type to be mutated, it has to be held in a variable, and it's +often possible to reason quite strongly about how that variable +is used. That means that the exclusivity rule that we're proposing +here allows us to make some very strong guarantees for value types, +generally making them an even tighter, lower-cost abstraction. +In contrast, it's inherent to the nature of reference types that +references can be copied pretty arbitrarily throughout a program. +The assumptions we want to make about value types depend on having +unique access to the variable holding the value; there's no way +to make a similar assumption about reference types without knowing +that we have a unique reference to the object, which would +radically change the programming model of classes and make them +unacceptable for the concurrent patterns described above. + +### Disabling dynamic enforcement. + +We could add an attribute which allows dynamic enforcement to +be downgraded to an unsafe-pointer-style undefined-behavior rule +on a variable-by-variable basis. This would allow programmers to +opt out of the expense of dynamic enforcement when it is known +to be unnecessary (e.g. because exclusivity is checked at some +higher level) or when the performance burden is simply too great. + +There is some concern that adding this attribute might lead to +over-use and that we should only support it if we are certain +that the overheads cannot be reduced in some better way. + +Since the rule still applies, and it's merely no longer being checked, +it makes sense to borrow the "checked" and "unchecked" terminology +from the optimizer settings. + +```swift +class TreeNode { + @exclusivity(unchecked) var left: TreeNode? + @exclusivity(unchecked) var right: TreeNode? +} +``` + +### Closures + +A closure (including both local function declarations and closure +expressions, whether explicit or autoclosure) is either "escaping" or +"non-escaping". Currently, a closure is considered non-escaping +only if it is: + +- a closure expression which is immediately called, + +- a closure expression which is passed as a non-escaping function + argument, or + +- a local function which captures something that is not allowed + to escape, like an ``inout`` parameter. + +It is likely that this definition will be broadened over time. + +A variable is said to be escaping if it is captured in an escaping +closure; otherwise, it is non-escaping. + +Escaping variables generally require dynamic enforcement instead of +static enforcement. This is because Swift cannot reason about when +an escaping closure will be called and thus when the variable will +be accessed. There are some circumstances where static enforcement +may still be allowed, for example when Swift can reason about how +the variable will be used after it is escaped, but this is only +possible as a best-effort improvement for special cases, not as a +general rule. + +In contrast, non-escaping variables can always use static enforcement. +(In order to achieve this, we must impose a new restriction on +recursive uses of non-escaping closures; see below.) This guarantee +aligns a number of related semantic and performance goals. For +example, a non-escaping variable does not need to be allocated +on the heap; by also promising to only use static enforcement for +the variable, we are essentially able to guarantee that the variable +will have C-like performance, which can be important for some kinds +of program. This guarantee also ensures that only static enforcement +is needed for ``inout`` parameters, which cannot be captured in +escaping closures; this substantially simplifies the implementation +model for capturing ``inout`` parameters. + +### Diagnosing dynamic enforcement violations statically + +In general, Swift is permitted to upgrade dynamic enforcement to +static enforcement when it can prove that two accesses either +always or never conflict. This is analogous to Swift's rules +about integer overflow. + +For example, if Swift can prove that two accesses to a global +variable will always conflict, then it can report that error +statically, even though global variables use dynamic enforcement: + +```swift +var global: Int +swap(&global, &global) // Two overlapping modifications to 'global' +``` + +Swift is not required to prove that both accesses will actually +be executed dynamically in order to report a violation statically. +It is sufficient to prove that one of the accesses cannot ever +be executed without causing a conflict. For example, in the +following example, Swift does not need to prove that ``mutate`` +actually calls its argument function: + +```swift +// The inout access lasts for the duration of the call. +global.mutate { return global + 1 } +``` + +When a closure is passed as a non-escaping function argument +or captured in a closure that is passed as a non-escaping function +argument, Swift may assume that any accesses made by the closure +will be executed during the call, potentially conflicting with +accesses that overlap the call. + +### Restrictions on recursive uses of non-escaping closures + +In order to achieve the goal of guaranteeing the use of static +enforcement for variables that are captured only by non-escaping +closures, we do need to impose an additional restriction on +the use of such closures. This rule is as follows: + +> A non-escaping closure ``A`` may not be recursively invoked +> during the execution of a non-escaping closure ``B`` which +> captures the same local variable or ``inout`` parameter unless: +> +> - ``A`` is defined within ``B`` or +> +> - ``A`` is a local function declaration which is referenced +> directly by ``B``. + +For clarity, we will call this rule the Non-Escaping Recursion +Restriction, or NRR. The NRR is sufficient to prove that +non-escaping variables captured by ``B`` will not be interfered +with unless ``B`` delegates to something which is locally known by +``B`` to have access to those variables. This, together with the +fact that the uses of ``B`` itself can be statically analyzed by +its defining function, is sufficient to allow static enforcement +for the non-escaping variables it captures. (It also enables some +powerful analyses of captured variables within non-escaping +closures; we do not need to get into that here.) + +Because of the tight restrictions on how non-escaping closures +can be used in Swift today, it's already quite difficult to +violate the NRR. The following user-level restrictions are +sufficient to ensure that the NRR is obeyed: + +- A function may not call a non-escaping function parameter + passing a non-escaping function parameter as an argument. + + For the purposes of this rule, a closure which captures + a non-escaping function parameter is treated the same as + the parameter. + + We will call this rule the Non-Escaping Parameter Call + Restriction, or NPCR. + +- Programmers using ``withoutActuallyEscaping`` should take + care not to allow the result to be recursively invoked. + +The NPCR is a conservative over-approximation: that is, there +is code which does not violate the NRR which will be considered +ill-formed under the NPCR. This is unfortunate but inevitable. + +Here is an example of the sort of code that will be disallowed +under the NPCR: + +```swift +func recurse(fn: (() -> ()) -> ()) { + // Invoke the closure, passing a closure which, if invoked, + // will invoke the closure again. + fn { fn { } } +} + +func apply(argProvider: () -> T, fn: (() -> T) -> T) { + // Pass the first argument function to the second. + fn(argProvider) +} +``` + +Note that it's quite easy to come up with ways to use these +functions that wouldn't violate the NRR. For example, if +either argument to ``apply`` is not a closure, the call +cannot possibly violate the NRR. Nonetheless, we feel that +the NPCR is a reasonable restriction: + +- Functions like ``recurse`` that apply a function to itself are + pretty much just of theoretical interest. Recursion is an + important programming tool, but nobody writes it like this + because it's just unnecessarily more difficult to reason about. + +- Functions like ``apply`` that take two closures are not uncommon, + but they're likely to either invoke the closures sequentially, + which would not violate the NPCR, or else be some sort of + higher-order combinator, which would require the closures to be + ``@escaping`` and thus also not violate the NPCR. + +Note that passing two non-escaping functions as arguments to the +same call does not violate the NPCR. This is because the NPCR +will be enforced, recursively, in the callee. (Imported C +functions which take non-escaping block parameters can, of +course, easily violate the NPCR. They can also easily allow +the block to escape. We do not believe there are any existing +functions or methods on our target platforms that directly +violate the NPCR.) + +In general, programmers who find the NPCR an unnecessarily +overbearing restriction can simply declare their function parameter +to be ``@escaping`` or, if they are certain that their code will +not violate the NRR, use ``withoutActuallyEscaping`` to disable +the NPCR check. + +## Source compatibility + +In order to gain the performance and language-design benefits of +exclusivity, we will have to enforce it in all language modes. +Therefore, exclusivity will eventually demand a source break. + +We can mitigate some of the impact of this break by implicitly migrating +code matching certain patterns to use different patterns that are known +to satisfy the exclusivity rule. For example, it would be straightforward +to automatically translate calls like ``swap(&array[i], &array[j])`` to +``array.swapAt(i, j)``. Whether this makes sense for any particular +migration remains to be seen; for example, ``swap`` does not appear to be +used very often in practice outside of specific collection algorithms. + +Overall, we do not expect that a significant amount of code will violate +exclusivity. This has been borne out so far by our testing. Often the +examples that do violate exclusivity can easily be rewritten to avoid +conflicts. In some of these cases, it may make sense to do the rewrite +automatically to avoid source-compatibility problems. + +## Effect on ABI stability and resilience + +In order to gain the performance and language-design benefits of +exclusivity, we must be able to assume that it is followed faithfully +in various places throughout the ABI. Therefore, exclusivity must be +enforced before we commit to a stable ABI, or else we'll be stuck with +the current conservatism around ``inout`` and ``mutating`` methods +forever. diff --git a/proposals/0177-add-clamped-to-method.md b/proposals/0177-add-clamped-to-method.md new file mode 100644 index 0000000000..37d3dfafc3 --- /dev/null +++ b/proposals/0177-add-clamped-to-method.md @@ -0,0 +1,102 @@ +# Add clamp(to:) to the stdlib + +* Proposal: [SE-0177](0177-add-clamped-to-method.md) +* Author: [Nicholas Maccharoli](https://github.com/Nirma) +* Review Manager: TBD +* Status: **Returned for revision** + +## Introduction + +This proposal aims to add functionality to the standard library for clamping a value to a provided `Range`. +The proposed function would allow the user to specify a range to clamp a value to where if the value fell within the range, the value would be returned as is, if the value being clamped exceeded the upper or lower bound then the upper or lower bound would be returned respectively. + +Swift-evolution thread: [Add a `clamp` function to Algorithm.swift](https://forums.swift.org/t/add-a-clamp-function-to-algorithm-swift/5405) + +## Motivation + +There have been quite a few times in my professional and personal programming life where I reached for a function to limit a value to a given range and was disappointed that it was not part of the standard library. + +There already exists an extension to `CountableRange` in the standard library implementing `clamped(to:)` that will limit the calling range to that of the provided range, so having the same functionality but just for types that conform to the `Comparable` protocol would be conceptually consistent. + +Having functionality like `clamped(to:)` added to `Comparable` as a protocol extension would benefit users of the Swift language who wish +to guarantee that a value is kept within bounds, perhaps one example of this coming in handy would be to limit the result of some calculation between two acceptable numerical limits, say the bounds of a coordinate system. + +## Proposed solution + +The proposed solution is to add a `clamped(to:)` function to the Swift Standard Library as an extension to `Comparable` and to `Strideable`. +The function would return a value within the bounds of the provided range, if the value `clamped(to:)` is being called on falls within the provided range then the original value would be returned. +If the value was less or greater than the bounds of the provided range then the respective lower or upper bound of the range would be returned. + +Clamping on an empty range simply returns the value clamped to the `lowerBound` / `upperBound` of the `Range` no different from clamping on a non-empty range. + +Given a `clamped(to:)` function existed it could be called in the following way, yielding the results in the adjacent comments: + +```swift +// Closed range variant + +100.clamped(to: 0...50) // 50 +100.clamped(to: 200...300) // 200 +100.clamped(to: 0...150) // 100 + +// Half-Open range variant + +100.clamped(to: 0..<50) // 49 +100.clamped(to: 200..<300) // 200 +100.clamped(to: 0..<150) // 100 +100.clamped(to: 42..<42) // 42 +``` + +## Detailed design + +The implementation of `clamped(to:)` that is being proposed is composed of two protocol extensions; one protocol extension on `Comparable` and another on `Strideable`. + +The implementation for `clamped(to:)` as an extension to `Comparable` accepting a range of type `ClosedRange` would look like the following: + +```swift +extension Comparable { + func clamped(to range: ClosedRange) -> Self { + if self > range.upperBound { + return range.upperBound + } else if self < range.lowerBound { + return range.lowerBound + } else { + return self + } + } +} +``` + +The implementation of `clamped(to:)` as an extension on `Strideable` would be confined to cases where the stride is of type `Integer`. +The implementation would be as follows: + +```swift +extension Strideable where Stride: Integer { + func clamped(to range: Range) -> Self { + let clampRange: ClosedRange + + if range.lowerBound == range.upperBound { + clampRange = range.lowerBound...range.upperBound + } else { + clampRange = range.lowerBound...(range.upperBound - 1) + } + + return clamped(to: clampRange) + } +} +``` + +## Source compatibility + +This feature is purely additive; it has no effect on source compatibility. + +## Effect on ABI stability + +This feature is purely additive; it has no effect on ABI stability. + +## Effect on API resilience + +The proposed function would become part of the API but purely additive. + +## Alternatives considered + +Aside from doing nothing, no other alternatives were considered. diff --git a/proposals/0178-character-unicode-view.md b/proposals/0178-character-unicode-view.md new file mode 100644 index 0000000000..64264a6e97 --- /dev/null +++ b/proposals/0178-character-unicode-view.md @@ -0,0 +1,93 @@ +# Add `unicodeScalars` property to `Character` + +* Proposal: [SE-0178](0178-character-unicode-view.md) +* Author: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0178-add-unicodescalars-property-to-character/5941) +* Implementation: [apple/swift#9675](https://github.com/apple/swift/pull/9675) + +## Introduction + +This proposal adds a `unicodeScalars` view to `Character`, similar to that on `String`. + +## Motivation + +The `Character` element type of `String` is currently a black box that provides +little functionality besides comparison, literal construction, and to be used +as an argument to `String.init`. + +Many operations on `String` could be neatly/readably implemented as operations +on each character in the string, if `Character` exposed its scalars more +directly. Many useful things can be determined by examining the scalars in a +grapheme (for example is this an ASCII character?). + +For example, today you can write this: + +```swift +let s = "one two three" +s.split(separator: " ") +``` + +But you cannot write this: + +```swift +let ws = CharacterSet.whitespacesAndNewlines +s.split { $0.unicodeScalars.contains(where: ws.contains) } +``` + +## Proposed solution + +Add a `unicodeScalars` property to `Character`, presenting a lazy view of the +scalars in the character, along similar lines to the one on `String`. + +Unlike the view on `String`, this will _not_ be a mutable view – it will be +read-only. The preferred method for creating and manipulating non-literal +`Character` values will be through `String`. While there may be some good +use cases to manipulating a `Character` directly, these are outweighed by the +complexity of ensuring the invariant that it contain exactly one grapheme. + +## Detailed design + +Add the following nested type to `Character`: + +```swift +extension Character { + public struct UnicodeScalarView : BidirectionalCollection { + public struct Index + public var startIndex: Index { get } + public var endIndex: Index { get } + public func index(after i: Index) -> Index + public func index(before i: Index) -> Index + public subscript(i: Index) -> UnicodeScalar + } + public var unicodeScalars: UnicodeScalarView { get } +} +``` + +Additionally, this type will conform to appropriate convenience protocols such +as `CustomStringConvertible`. + +All initializers will be declared internal, as unlike the `String` equivalent, +this type will only ever be vended by `Character`. + +## Source compatibility + +Purely additive, so no impact. + +## Effect on ABI stability + +Purely additive, so no impact. + +## Effect on API resilience + +Purely additive, so no impact. + +## Alternatives considered + +Adding other views, such as `utf8` or `utf16`, was considered but not deemed useful +enough compared to using these operations on `String` instead. + +In the future, this feature could be used to implement convenience methods such as +`isASCII` on `Character`. This could be done additively, given this building block, +and is outside the scope of this initial proposal. diff --git a/proposals/0179-swift-run-command.md b/proposals/0179-swift-run-command.md new file mode 100644 index 0000000000..f43b30b473 --- /dev/null +++ b/proposals/0179-swift-run-command.md @@ -0,0 +1,65 @@ +# Swift `run` Command + +* Proposal: [SE-0179](0179-swift-run-command.md) +* Author: [David Hart](https://github.com/hartbit/) +* Review Manager: [Daniel Dunbar](https://github.com/ddunbar) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0179-swift-run-command/6031) +* Implementation: [apple/swift-package-manager#1187](https://github.com/apple/swift-package-manager/pull/1187) + +## Introduction + +The proposal introduces a new `swift run` command to build and run an executable defined in the current package. + +## Motivation + +It is common to want to build and run an executable during development. For now, one must first build it and then execute it from the build folder: + +```bash +$ swift build +$ .build/debug/myexecutable +``` + +In Swift 4, the Swift Package Manager will build to a different path, containing a platform sub-folder (`.build/macosx-x86_64/debug` for mac and `.build/linux-x86_64/debug` for linux), making it more cumbersome to run the executable from the command line. + +To improve the development workflow, the proposal suggests introducing a new first-level `swift run` command that will build if necessary and then run an executable defined in the `Package.swift` manifest, reducing the above steps to just one. + +## Proposed solution + +The swift `run` command would be defined as: + +```bash +$ swift run --help +OVERVIEW: Build and run executable + +USAGE: swift run [options] [executable [arguments]] + +OPTIONS: + --build-path Specify build/cache directory [default: ./.build] + --chdir, -C Change working directory before any other operation + --color Specify color mode (auto|always|never) [default: auto] + --configuration, -c Build with configuration (debug|release) [default: debug] + --enable-prefetching Enable prefetching in resolver + --skip-build Skip building the executable product + --verbose, -v Increase verbosity of informational output + -Xcc Pass flag through to all C compiler invocations + -Xlinker Pass flag through to all linker invocations + -Xswiftc Pass flag through to all Swift compiler invocations + --help Display available options +``` + +If needed, the command will build the product before running it. As a result, it can be passed any options `swift build` accepts. As for `swift test`, it also accepts an extra `--skip-build` option to skip the build phase. + +After the options, the command optionally takes the name of an executable product defined in the `Package.swift` manifest and introduced in [SE-0146](0146-package-manager-product-definitions.md). If called without an executable and the manifest defines one and only one executable product, it will default to running that one. In any other case, the command fails. + +If the executable is explicitly defined, all remaining arguments are passed as-is to the executable. + +```bash +$ swift run # .build/debug/exe +$ swift run exe # .build/debug/exe +$ swift run exe arg1 arg2 # .build/debug/exe arg1 arg2 +``` + +## Alternatives considered + +One alternative to the Swift 4 change of build folder would be for the Swift Package Manager to create and update a symlink at `.build/debug` and `.build/release` that point to the latest build folder for that configuration. Although that should probably be done to retain backward-compatibility with tools that depended on the build location, it does not completely invalidate the usefulness of the `run` command. diff --git a/proposals/0180-string-index-overhaul.md b/proposals/0180-string-index-overhaul.md new file mode 100644 index 0000000000..abe8de6074 --- /dev/null +++ b/proposals/0180-string-index-overhaul.md @@ -0,0 +1,389 @@ +# String Index Overhaul + +* Proposal: [SE-0180](0180-string-index-overhaul.md) +* Author: [Dave Abrahams](https://github.com/dabrahams) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0180-string-index-overhaul/6286) +* Implementation: [apple/swift#9806](https://github.com/apple/swift/pull/9806) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/72b8d90becd60b7cc7695607ae908ef251f1e966/proposals/0180-string-index-overhaul.md) + +## Introduction + +Today `String` shares an `Index` type with its `CharacterView` but not +with its `UTF8View`, `UTF16View`, or `UnicodeScalarView`. This +proposal redefines `String.UTF8View.Index`, `String.UTF16View.Index`, +and `String.CharacterView.Index` as typealiases for `String.Index`, +and exposes a public `encodedOffset` property and initializer that can +be used to serialize and deserialize positions in a `String` or +`Substring`. + +Swift-evolution thread: [Pitch: String Index Overhaul](https://forums.swift.org/t/pitch-string-index-overhaul/6017) + +## Motivation + +The different index types are supported by a set of `Index` +initializers, which are failable whenever the source index might not +correspond to a position in the target view: + +```swift +if let j = String.UnicodeScalarView.Index( + someUTF16Position, within: s.unicodeScalars) { + ... +} +``` + +The current API is as follows: + +```swift +public extension String.Index { + init?(_: String.UnicodeScalarIndex, within: String) + init?(_: String.UTF16Index, within: String) + init?(_: String.UTF8Index, within: String) +} + +public extension String.UTF16View.Index { + init?(_: String.UTF8Index, within: String.UTF16View) + init(_: String.UnicodeScalarIndex, within: String.UTF16View) + init(_: String.Index, within: String.UTF16View) +} + +public extension String.UTF8View.Index { + init?(_: String.UTF16Index, within: String.UTF8View) + init(_: String.UnicodeScalarIndex, within: String.UTF8View) + init(_: String.Index, within: String.UTF8View) +} + +public extension String.UnicodeScalarView.Index { + init?(_: String.UTF16Index, within: String.UnicodeScalarView) + init?(_: String.UTF8Index, within: String.UnicodeScalarView) + init(_: String.Index, within: String.UnicodeScalarView) +} +``` + +These initializers are supplemented by a corresponding set of +convenience conversion methods: + +```swift +if let j = someUTF16Position.samePosition(in: s.unicodeScalars) { + ... +} +``` + +with the following API: + +```swift +public extension String.Index { + func samePosition(in: String.UTF8View) -> String.UTF8View.Index + func samePosition(in: String.UTF16View) -> String.UTF16View.Index + func samePosition( + in: String.UnicodeScalarView) -> String.UnicodeScalarView.Index +} + +public extension String.UTF16View.Index { + func samePosition(in: String) -> String.Index? + func samePosition(in: String.UTF8View) -> String.UTF8View.Index? + func samePosition( + in: String.UnicodeScalarView) -> String.UnicodeScalarView.Index? +} + +public extension String.UTF8View.Index { + func samePosition(in: String) -> String.Index? + func samePosition(in: String.UTF16View) -> String.UTF16View.Index? + func samePosition( + in: String.UnicodeScalarView) -> String.UnicodeScalarView.Index? +} + +public extension String.UnicodeScalarView.Index { + func samePosition(in: String) -> String.Index? + func samePosition(in: String.UTF8View) -> String.UTF8View.Index + func samePosition(in: String.UTF16View) -> String.UTF16View.Index +} +``` + +The result is a great deal of API surface area for apparently little +gain in ordinary code, that normally only interchanges indices among +views when the positions match up exactly (i.e. when the conversion is +going to succeed). Also, the resulting code is needlessly awkward. + +Finally, the opacity of these index types makes it difficult to record +`String` or `Substring` positions in files or other archival forms, +and reconstruct the original positions with respect to a deserialized +`String` or `Substring`. + +## Proposed solution + +All `String` views will use a single index type (`String.Index`), so +that positions can be interchanged without awkward explicit +conversions: + +```swift +let html: String = "See swift.org" + +// Search the UTF16, instead of characters, for performance reasons: +let open = "<".utf16.first!, close = ">".utf16.first! +let tagStart = html.utf16.index(of: open) +let tagEnd = html.utf16[tagStart...].index(of: close) + +// Slice the String with the UTF-16 indices to retrieve the tag. +let tag = html[tagStart...tagEnd] +``` + +A property and an initializer will be added to `String.Index`, exposing +the offset of the index in code units (currently only UTF-16) from the +beginning of the string: + +```swift +let n: Int = html.endIndex.encodedOffset +let end = String.Index(encodedOffset: n) +assert(end == String.endIndex) +``` + +# Comparison and Subscript Semantics + +When two indices being compared correspond to positions that are valid +in any single `String` view, comparison semantics are already fully +specified by the `Collection` requirements. The other cases occur +when indices fall between Unicode scalar boundaries in views having +distinct encodings. For example, the string `"\u{1f773}"` (“🝳”) is +encoded as `0xD83D, 0xDF73` in UTF-16 and `0xF0, 0x9F, 0x9D, 0xB3` in +UTF-8, and there is no obvious way to compare the second positions in +each of those sequences. The proposed rule is that such indices are +compared by comparing their `encodedOffset`s. Such index values are +not totally ordered but do satisfy strict weak ordering requirements, +which is sufficient for algorithms such as `sort` to exhibit sensible +behavior. We might consider loosening the specified requirements on +these algorithms and on `Comparable` to support strict weak ordering, +but for now we can treat such index pairs as being formally outside +the domain of comparison, like any other indices from completely +distinct collections. + +With respect to subscripts, an index that does not fall on an exact +boundary in a given `String` or `Substring` view will be treated as +falling at its `encodedOffset` in the underlying code units, with the +actual contents of the result being an emergent property of applying +the usual Unicode rules for decoding those code units. For example, +when slicing a string with an index `i` that falls between two +`Character` boundaries, `i.encodedOffset` is treated as a position in +the string's underlying code units, and the `Character`s of the result +are determined by performing standard Unicode grapheme breaking on the +resulting sequence of code units. + +```swift +let s = "e\u{301}galite\u{301}" // "égalité" +let i = Array(s.unicodeScalars.indices) +print(s[i[1]...]) // "◌́galité" +print(s[.. String.Index? + func samePosition(in: String.UTF8View) -> String.Index? + func samePosition(in: String.UTF16View) -> String.Index? + func samePosition(in: String.UnicodeScalarView) -> String.Index? +} +``` + +## Source compatibility + +Because of the collapse of index +types, [existing non-failable APIs](#motivation) become failable. To +avoid breaking Swift 3 code, the following overloads of existing +functions are added, allowing the resulting optional indices to be +used where previously non-optional indices were used. These overloads +were driven by making the new APIs work with existing code, including +the Swift source compatibility test suite, and should be viewed as +migration aids only, rather than additions to the Swift 3 API. + +```swift +extension Optional where Wrapped == String.Index { + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional indices") + public static func ..<( + lhs: String.Index?, rhs: String.Index? + ) -> Range { + return lhs! ..< rhs! + } + + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional indices") + public static func ...( + lhs: String.Index?, rhs: String.Index? + ) -> ClosedRange { + return lhs! ... rhs! + } +} + +// backward compatibility for index interchange. +extension String.UTF16View { + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public func index(after i: Index?) -> Index { + return index(after: i) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public func index( + _ i: Index?, offsetBy n: IndexDistance) -> Index { + return index(i!, offsetBy: n) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional indices") + public func distance(from i: Index?, to j: Index?) -> IndexDistance { + return distance(from: i!, to: j!) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public subscript(i: Index?) -> Unicode.UTF16.CodeUnit { + return self[i!] + } +} + +extension String.UTF8View { + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public func index(after i: Index?) -> Index { + return index(after: i!) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public func index(_ i: Index?, offsetBy n: IndexDistance) -> Index { + return index(i!, offsetBy: n) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional indices") + public func distance( + from i: Index?, to j: Index?) -> IndexDistance { + return distance(from: i!, to: j!) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public subscript(i: Index?) -> Unicode.UTF8.CodeUnit { + return self[i!] + } +} + +// backward compatibility for index interchange. +extension String.UnicodeScalarView { + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public func index(after i: Index?) -> Index { + return index(after: i) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public func index(_ i: Index?, offsetBy n: IndexDistance) -> Index { + return index(i!, offsetBy: n) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional indices") + public func distance(from i: Index?, to j: Index?) -> IndexDistance { + return distance(from: i!, to: j!) + } + @available( + swift, deprecated: 3.2, obsoleted: 4.0, + message: "Any String view index conversion can fail in Swift 4; please unwrap the optional index") + public subscript(i: Index?) -> Unicode.Scalar { + return self[i!] + } +} +``` + +- **Q**: Will existing correct Swift 3 applications stop compiling due + to this change? + + **A**: it is possible but unlikely. The existing index conversion + APIs are relatively rarely used, and the overloads listed above + handle the common cases in Swift 3 compatibility mode. + +- **Q**: Will applications still compile but produce + different behavior than they used to? + + **A**: No. + +- **Q**: Is it possible to automatically migrate from the old syntax + to the new syntax? + + **A**: Yes, although usages of these APIs may be rare enough that it + isn't worth the trouble. + +- **Q**: Can Swift applications be written in a common subset that works + both with Swift 3 and Swift 4 to aid in migration? + + **A**: Yes, the Swift 4 APIs will all be available in Swift 3 mode. + +## Effect on ABI stability + +This proposal changes the ABI of the standard library. + +## Effect on API resilience + +This proposal makes no changes to the resilience of any APIs. + +## Alternatives considered + +The only alternative considered was no action. diff --git a/proposals/0181-package-manager-cpp-language-version.md b/proposals/0181-package-manager-cpp-language-version.md new file mode 100644 index 0000000000..5fd52c8f02 --- /dev/null +++ b/proposals/0181-package-manager-cpp-language-version.md @@ -0,0 +1,84 @@ +# Package Manager C/C++ Language Standard Support + +* Proposal: [SE-0181](0181-package-manager-cpp-language-version.md) +* Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Daniel Dunbar](https://github.com/ddunbar) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0181-package-manager-c-c-language-standard-support/6353) +* Implementation: [apple/swift-package-manager#1264](https://github.com/apple/swift-package-manager/pull/1264) + +## Introduction + +This proposal adds support for declaring the language standard for C and C++ +targets in a SwiftPM package. + +## Motivation + +The C++ language standard is one of the most important build setting needed to +compile C++ targets. We want to add some mechanism to declare it until we get +the complete build settings feature, which is deferred from the Swift 4 release. + +## Proposed solution + +We will add support to the package manifest declaration to specify the C and C++ +language standards: + +```swift +let package = Package( + name: "CHTTP", + ... + cLanguageStandard: .c89, + cxxLanguageStandard: .cxx11 +) +``` + +These settings will apply to all the C and C++ targets in the package. The +default value of these properties will be `nil`, i.e., a language standard flag +will not be passed when invoking the C/C++ compiler. + +_Once we get the build settings feature, we will deprecate these properties._ + +## Detailed design + +The C/C++ language standard will be defined by the enums below and +updated as per the Clang compiler [repository](https://github.com/llvm-mirror/clang/blob/master/include/clang/Frontend/LangStandards.def). + +```swift +public enum CLanguageStandard { + case c89 + case c90 + case iso9899_1990 + case iso9899_199409 + case gnu89 + case gnu90 + case c99 + case iso9899_1999 + case gnu99 + case c11 + case iso9899_2011 + case gnu11 +} + +public enum CXXLanguageStandard { + case cxx98 + case cxx03 + case gnucxx98 + case gnucxx03 + case cxx11 + case gnucxx11 + case cxx14 + case gnucxx14 + case cxx1z + case gnucxx1z +} +``` +## Impact on existing code + +There will be no impact on existing packages because this is a new API and the +default behaviour remains unchanged. + +## Alternatives considered + +We considered adding this property at target level but we think that will +pollute the target namespace. Moreover, this is a temporary measure until we get +the build settings feature. diff --git a/proposals/0182-newline-escape-in-strings.md b/proposals/0182-newline-escape-in-strings.md new file mode 100644 index 0000000000..6f3cfb9fc9 --- /dev/null +++ b/proposals/0182-newline-escape-in-strings.md @@ -0,0 +1,106 @@ +# String Newline Escaping + +* Proposal: [SE-0182](0182-newline-escape-in-strings.md) +* Authors: [John Holdsworth](https://github.com/johnno1962), [David Hart](https://github.com/hartbit), [Adrian Zubarev](https://github.com/DevAndArtist) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 4.0)** +* Implementation: [apple/swift#11080](https://github.com/apple/swift/pull/11080) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0182-string-newline-escaping/6355) +* Previous Proposal: [SE-0168](0168-multi-line-string-literals.md) + +## Introduction + +This proposal is a refinement of [SE-0168](0168-multi-line-string-literals.md) which introduces the ability to escape newlines in single and multi-line strings to improve readability and maintenance of source material containing excessively long lines. + +Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/accepted-se-0168-multi-line-string-literals/5715) + +## Motivation + +Escaping newlines in multi-line strings was removed from the [SE-0168](0168-multi-line-string-literals.md) proposal by the Core Team with the following rational: + +> Discussion on the list raised the idea of allowing a line to end with \ to "escape" the newline and elide it from the value of the literal; the core team had concerns about only allowing that inside multi-line literals and felt that that could also be considered later as an additive feature. + +Adding them to multi-line strings would have introduced an inconsistency with respect to conventional string literals. This proposal conforms both multi-line and conventional string construction to allow newline escaping, enabling developers to split text over multiple source lines without introducing new line breaks. This approach enhances source legibility. For example: + +```swift +// Excessively long line that requires scrolling to read +let text = """ + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. + """ + +// Shorter lines that are easier to read, but represent the same long line +let text = """ + Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod \ + tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, \ + quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. + """ +``` + +Incorporating a string continuation character is well founded, used in other development languages, and carries little risk of confusing naive users. + +## Detailed design + +This proposal introduces `\` as a line continuation character which escapes newlines matching the following regular-expression: `/\\[ \t]*\n/`. In other terms, line continuation requires a `\` character, followed by zero or more horizontal whitespace characters, followed by a newline character. All those characters are omitted from the resulting string. + +As a consequence, these rules follow: + +* All whitespace characters between `\` and the newline are disregarded. +* All whitespace characters before `\` are included in the string as is. +* An escape character at the end of the last line of a literal is an error, as no newlines follow. + +For example: + +```swift +let str1 = """↵ + line one \↵ + line two \ ↵ + line three↵ + """↵ + +let str2 = "line one \↵ +line two \ ↵ +line three"↵ + +assert(str1 == "line one line two line three") +assert(str2 == "line one line two line three") +assert(str1 == str2) +``` + +This does not affect the indentation removal feature of multiline strings and does not suggest that indentation removal should be added to conventional strings but it does give them consistent treatment. + +## Source compatibility + +This proposal does not affect existing source, because it is purely additive - enabling syntax that is not currently allowed in Swift. + +## Effect on ABI stability + +This proposal does not affect ABI stability. + +## Effect on API resilience + +This proposal does not affect ABI resilience. + +## Alternatives considered + +It has been heavily debated between the authors of the proposals whether newline escaping should be supported in single-line strings. One argument against it is that the lack of indentation stripping in single-line strings forces strings to include no indentation, hindering the readability of code by visually breaking scopes when returning the column 1: + +```swift +class Messager { + let defaultMessage = "This is a long string that will wrap over multiple \ +lines. Because we don't strip indentation like with multi-line strings, the \ +author has no choice but to remove all indentation." + + func send(message: String?) { + precondition(message == nil || !message!.isEmpty, "You can't send an \ +empty message, it has no meaning.") + print(message ?? defaultMessage) + } +} +``` + +Another argument against it is that lines that are sufficiently long and/or complex could use +multi-line string literals with newline escaping, so there is no need to include them in single +quoted strings. + +A counter-argument is that it is important to keep single and triple quoted strings consistent +with each other. diff --git a/proposals/0183-substring-affordances.md b/proposals/0183-substring-affordances.md new file mode 100644 index 0000000000..eb83064787 --- /dev/null +++ b/proposals/0183-substring-affordances.md @@ -0,0 +1,84 @@ +# Substring performance affordances + +* Proposal: [SE-0183](0183-substring-affordances.md) +* Author: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0183-substring-performance-affordances/6393) +* Bug: [SR-4933](https://bugs.swift.org/browse/SR-4933) + +## Introduction + +This proposal modifies a small number of methods in the standard library that +are commonly used with the `Substring` type: + + - Modify the `init` on floating point and integer types, to construct them + from `StringProtocol` rather than `String`. +- Change `join` to be an extension `where Element: StringProtocol` +- Have `Substring.filter` to return a `String` + +## Motivation + +Swift 4 introduced `Substring` as the slice type for `String`. Previously, +`String` had been its own slice type, but this leads to issues where string +buffers can be unexpectedly retained. This approach was adopted instead of the +alternative of having the slicing operation make a copy. A copying slicing +operation would have negative performance consequences, and would also conflict +with the requirement that `Collection` be sliceable in constant time. In cases +where an API requires a `String`, the user must construct a new `String` from a +`Substring`. This can be thought of as a "deferral" of the copy that was +avoided at the time of the slice. + +There are a few places in the standard library where it is notably inefficient +to force a copy of a substring in order to use it with a string: joining +substrings, and converting substrings to integers. In particular, these +operations are likely to be used inside a loop over a number of substrings +extracted from a string – for example, splitting a string into substrings, +then rejoining them. + +Additionally, per SE-163, operations on `Substring` that produce a fresh string +(such as `.uppercase`) should return a `String`. This changes +`Substring.filter` to do so. + +## Proposed solution + +Add the following to the standard library: + +```swift +extension FixedWidthInteger { + public init?(_ text: S, radix: Int = 10) +} + +extension Float/Double/Float80 { + public init?(_ text: S, radix: Int = 10) +} + +extension Sequence where Element: StringProtocol { + public func joined(separator: String = "") -> String +} + +extension Substring { + public func filter(_ isIncluded: (Element) throws -> Bool) rethrows -> String +} +``` + +These additions are deliberately narrow in scope. They are _not_ intended to +solve a general problem of being able to interchange substrings for strings (or +more generally slices for collections) generically in different APIs. See the +alternatives considered section for more on this. + +## Source compatibility + +No impact, these are generalizing an existing API to a protocol (in case of numeric conversion/joining) or modify a type newly introduced in Swift 4 (in +case of `filter`). + +## Effect on ABI stability + +The switch from concrete to generic types needs to be made before ABI stability. + +## Alternatives considered + +The goal of this proposal is to generalize existing methods that are specific +to string processing. Further affordances, such as implicit or explicit +conversions of `String` to `Substring`, might solve this problem more generally +but are considered out of scope for this proposal. diff --git a/proposals/0184-unsafe-pointers-add-missing.md b/proposals/0184-unsafe-pointers-add-missing.md new file mode 100644 index 0000000000..a8cff99c6b --- /dev/null +++ b/proposals/0184-unsafe-pointers-add-missing.md @@ -0,0 +1,589 @@ +# Unsafe[Mutable][Raw][Buffer]Pointer: add missing methods, adjust existing labels for clarity, and remove deallocation size + +* Proposal: [SE-0184](0184-unsafe-pointers-add-missing.md) +* Author: [Diana Ma (“Taylor Swift”)](https://github.com/tayloraswift) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.1)** +* Implementation: [apple/swift#12200](https://github.com/apple/swift/pull/12200) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revisions-se-0184-unsafe-mutable-raw-buffer-pointer-add-missing-methods-adjust-existing-labels-for-clarity-and-remove-deallocation-size/6772) + + +## Introduction + +*This document is a spin-off from a much larger [original proposal](https://github.com/tayloraswift/swift-evolution/blob/e888af466c9993de977f6999a131eadd33291b06/proposals/0184-unsafe-pointers-add-missing.md), which covers only those aspects of SE-0184 which do not deal with partial buffer memory state. Designing the partial buffer memory state API clearly requires more work, and has been left out of the scope of this document.* + +Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. + +Swift-evolution threads: [Pitch: Improved Swift pointers](https://forums.swift.org/t/pitch-improved-swift-pointers/6318), [Pitch: More Improved Swift pointers](https://forums.swift.org/t/pitch-improved-swift-pointers/6318) + +Implementation branch: [`tayloraswift:se-0184a`](https://github.com/tayloraswift/swift/tree/se-0184a) + +## Background + +There are four binary memorystate operations: *initialization*, *move-initialization*, *assignment*, and *move-assignment*, and two unary memorystate operations: *deinitialization* and *type rebinding*. The binary operations can be grouped according to how they affect the source buffer and the destination buffer. **Copy** operations only read from the source buffer, leaving it unchanged. **Move** operations deinitialize the source memory, decrementing the reference count by 1 if the memory type is not a trivial type. **Retaining** operations initialize the destination memory, incrementing the reference count by 1 if applicable. **Releasing** operations deinitialize the destination memory before reinitializing it with the new values, resulting in a net change in the reference count of 0, if applicable. + +| | Copy (+0) | Move (−1) | +| -------------: |----------: | ---------: | +| **Retaining (+1)** | initialize | move-initialize | +| **Releasing (+0)** | assign | move-assign | + +Raw pointers also have a unique operation, *bytewise-copying*, which we will lump together with the memorystate functions, but does not actually change a pointer’s memory state. + +Most of these operations become more relevant in the discussion of partial buffer memory state, which is not in the scope of this document. This document only proposes changes related to memory allocation, type-rebinding, and two special *unary* forms of initialization and assignment which initialize memory to a fixed, repeating value. + +## Motivation + +Right now, `UnsafeMutableBufferPointer` is kind of a black box when it comes to producing and modifying instances of it. Much of the API present on `UnsafeMutablePointer` is absent on its buffer variant. To create, bind, allocate, initialize, and deallocate them, you have to extract `baseAddress`es and `count`s. This is unfortunate because `UnsafeMutableBufferPointer` provides a handy container for tracking the size of a memory buffer, but to actually make use of this information, the buffer pointer must be disassembled. In practice, this means the use of memory buffers requires frequent (and annoying) conversion back and forth between buffer pointers and base address–count pairs. For example, buffer allocation requires the creation of a temporary `UnsafeMutablePointer` instance. This means that the following “idiom” is very common in Swift code: + +```swift +let buffer = UnsafeMutableBufferPointer(start: UnsafeMutablePointer.allocate(capacity: byteCount), count: byteCount) +``` + +Aside from being extremely long and unwieldy, and requiring the creation of a temporary, `byteCount` must appear twice. + +You can’t even cast buffer pointer types to their mutable or immutable forms without creating a temporary. + +```swift + +var mutableBuffer = UnsafeMutableBufferPointer(start: UnsafeMutablePointer(mutating: immutableBuffer.baseAddress!), count: immutableBuffer.count) +``` + +Currently, memory is deallocated by an instance method on `UnsafeMutablePointer`, `deallocate(count:)`. Like much of the Swift pointer API, performing this operation on a buffer pointer requires extracting `baseAddress!` and `count`. It is very common for the allocation code above to be immediately followed by: + +```swift +defer +{ + buffer.baseAddress?.deallocate(capacity: buffer.count) +} +``` + +The `?` is sometimes exchanged with an `!` depending on the personality of the author, as normally, neither operator is meaningful here — the `baseAddress` is never `nil` if the buffer pointer was created around an instance of `UnsafeMutablePointer`. + +This method is extremely problematic because nearly all users, on first seeing the signature of `deallocate(capacity:)`, will naturally conclude from the `capacity` label that `deallocate(capacity:)` is equivalent to some kind of `realloc()` that can only shrink the buffer. However this is not the actual behavior — `deallocate(capacity:)` actually *ignores* the `capacity` argument and just calls `free()` on `self`. The current API is not only awkward and suboptimal, it is *misleading*. You can write perfectly legal Swift code that shouldn’t segfault, but still can, for example + +```swift +var ptr = UnsafeMutablePointer.allocate(capacity: 1000000) +ptr.initialize(to: 13, count: 1000000) +ptr.deallocate(capacity: 500000) // deallocate the second half of the memory block +ptr[0] // segmentation fault +``` + +where the first 500000 addresses should still be valid if the [documentation](https://developer.apple.com/documentation/swift/unsafemutablepointer/2295090-deallocate) is to be read literally. + +Users who are *aware* of this behavior may also choose to disregard the `capacity` argument and write things like this: + +```swift +defer +{ + buffer.baseAddress?.deallocate(capacity: 42) +} +``` + +which is functionally equivalent. However this will lead to disastrous source breakage if the implementation of `deallocate(capacity:)` is ever “corrected”. Since the API would not change, such code would still compile, but suddenly start failing at runtime. Thus, the current API, combined with incorrect documentation, is serving as a vector for introducing memory bugs into Swift code. + +Finally, some of the naming choices in the current API deserve a second look. While the original API intended to introduce a naming convention where `bytes` refers to uninitialized memory, `capacity` to uninitialized elements, and `count` to initialized elements, the actual usage of the three words does not always agree. In `copyBytes(from:count:)`, `count` refers to the number of *bytes*, which may or may not be initialized. Similarly, the `UnsafeMutableRawBufferPointer` `allocate(count:)` type method includes a `count` argument which actually refers to uninitialized bytes. The argument label `to:` is also excessively overloaded; sometimes it refers to a type `T.Type`, and sometimes it refers to a repeated value parameter. This becomes problematic when both parameters appear in the same method, as in `initializeMemory(as:at:count:to)`. + +## Proposed solution + +The ultimate goal of the API redesign is to bring all of the functionality in `UnsafeMutablePointer` and `UnsafeMutableRawPointer` to their buffer types, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer`. Operations which are covered by this proposal are in **bold**. + +The full toolbox of methods that we could possibly support includes: + + - **allocation** + - **deallocation** + + - initialization + - move-initialization + + - assignment + - move-assignment + + - deinitialization + + - **type rebinding** + - bytewise copying + +Because copy operations (initialization and assignment) don’t mutate the source argument, they can also come in a form which takes a repeated-value source instead of a buffer source. + + - **initialization (repeated-value)** + - **assignment (repeated-value)** + +`UnsafeMutablePointer` and `UnsafeMutableRawPointer` already contain repeated-value methods for initialization in the form of `initialize(to:count:)` and `initializeMemory(as:at:count:to:)`. This proposal will add the assignment analogues. For reasons explained later, the argument label for the repeated-value parameter will be referred to as `repeating:`, not `to:`. + +### `UnsafePointer` + +``` +func deallocate() +func withMemoryRebound(to:capacity:_:) -> Result +``` + +`UnsafePointer` does not get an allocator static method, since you almost always want a mutable pointer to newly allocated memory. Its type rebinding method is also written as a decorator, taking a trailing closure, for memory safety. + +Most immutable pointer types currently do not have a deallocation method. This proposal adds them, fixing [SR-3309](https://bugs.swift.org/browse/SR-3309). Note, immutable raw buffer pointers already support this API. + +### `UnsafeMutablePointer` + +``` +static +func allocate(capacity:) -> UnsafeMutablePointer +func deallocate() + +func initialize(repeating:count:) +func initialize(to:) + +func assign(repeating:count:) + +func withMemoryRebound(to:capacity:_:) -> Result +``` + +Like `UnsafePointer`, `UnsafeMutablePointer`’s type rebinding method is written as a decorator. + +Previously, the single-element repeated-initialization case was supported by a default argument of `1` on `initialize(repeating:count:)`’s `count:` parameter, but it was decided this was too confusing in terms of API readability. For example, calls to `initialize(repeating:count:)` and its corresponding method on `UnsafeMutableBufferPointer` were prone to look the same. + +```swift +plainPointer.initialize(repeating: pointee) +bufferPointer.initialize(repeating: repeatedValue) +``` + +Increasing API surface by adding this method is justified by the large number of calls to `initialize(to:count:)` in the standard library (and likely other code) which rely on the default argument of `1`. We do *not* need to add a corresponding `assignPointee(to:)` method since this can be done with the assignment operator. + +```swift +ptr.pointee = newValue +``` + +### `UnsafeRawPointer` + +``` +func deallocate() + +func bindMemory(to:capacity:) -> UnsafePointer +``` + +### `UnsafeMutableRawPointer` + +``` +static +func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer +func deallocate() + +func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer + +func bindMemory(to:capacity:) -> UnsafeMutablePointer +``` + +Currently, `UnsafeMutableRawPointer`’s methods take an `at:` offset argument that is interpreted in strides. This argument is not currently in use in the entire Swift standard library, and we believe that it is not useful in practice. + +Unlike `UnsafeMutablePointer`, we do not add a single-instance initialize method to `UnsafeMutableRawPointer`, as such a method would probably not be useful. However, we still remove the default argument of `1` from the `count:` argument in `initializeMemory(as:repeating:count:)` to prevent confusion with calls to its buffer variant. + +### `UnsafeBufferPointer` + +``` +func deallocate() +func withMemoryRebound(to:_:) -> Result +``` + +The buffer type rebind method dynamically computes the new count by performing multiplication and integer division, since the target type may have a different stride than the original type. This is in line with existing precedent in the generic buffer method `initializeMemory(as:from:)` on `UnsafeMutableRawBufferPointer`. + +> Note: **calling `deallocate()` on a buffer pointer is only defined behavior if the buffer pointer references a complete heap memory block**. This operation may become supported in a wider variety of cases in the future if Swift gets a more sophisticated heap allocation backend. + +### `UnsafeMutableBufferPointer` + +``` +static +func allocate(capacity:) -> UnsafeMutableBufferPointer +func deallocate() + +func initialize(repeating:) + +func assign(repeating:) + +func withMemoryRebound(to:_:) -> Result +``` + +The buffer type rebind method works the same way as in `UnsafeBufferPointer`. (Type rebinding never cares about mutability.) + +### `UnsafeRawBufferPointer` + +``` +func deallocate() + +func bindMemory(to:) -> UnsafeBufferPointer +``` + +### `UnsafeMutableRawBufferPointer` + +``` +static +func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer +func deallocate() + +func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer + +func bindMemory(to:) -> UnsafeMutableBufferPointer +``` + +> note: `initializeMemory(as:repeating:)` performs integer division on `self.count` (just like `bindMemory(to:)`) + +> note: the return value of `initializeMemory(as:repeating:)` should be marked as `@discardableResult`. + +We also make several miscellaneous changes to the API in order to tidy things up. + +- **rename `copyBytes(from:count:)` and `copyBytes(from:)` to `copyMemory(from:byteCount:)` and `copyMemory(from:)`** + +This brings the method names in line with the rest of the raw pointer API. + +- **add an `init(mutating:)` initializer to `UnsafeMutableBufferPointer`** + +This makes it much easier to make a mutable copy of an immutable buffer pointer. Such an initializer already exists on `UnsafeMutableRawBufferPointer`, so adding one to `UnsafeMutableBufferPointer` is also necessary for consistency. The reverse initializer, from `UnsafeMutableBufferPointer` to `UnsafeBufferPointer` should also be added for completeness. + +- **deprecate the sized deallocation API** + +Removing `capacity` from `deallocate(capacity:)` will end the confusion over what `deallocate()` does, making it obvious that `deallocate()` will free the *entire* memory block at `self`, just as if `free()` were called on it. + +The old `deallocate(capacity:)` method should be marked as `deprecated` and eventually removed since it currently encourages dangerously incorrect code. This avoids misleading future users, encourages current users to address this potentially catastrophic memory bug, and leaves the possibility open for us to add a `deallocate(capacity:)` method in the future, or perhaps even a `reallocate(toCapacity:)` method. + +Along similar lines, the `bytes` and `alignedTo` parameters should be removed from the `deallocate(bytes:alignedTo:)` method on `UnsafeMutableRawPointer` and `UnsafeRawPointer`. + +As discussed earlier, an unsized `deallocate()` method should be added to all pointer types, even immutable ones, as Swift’s memory model does not require memory to be mutable for deallocation. + +> note: the deallocation size parameters were originally included in early versions of Swift in order to support a more sophisticated hypothetical heap allocator backend that we wanted to have in the future. (Swift currently calls `malloc(_:)` and `free()`.) While such a backend would theoretically run more efficiently than the C backend, overengineering Swift to support it in the future has proven to be a detriment to users right now. By removing the size parameters now, we make it easier and safer to reintroduce such an API in the future without inadvertently causing silent source breakage. + +> note: changes to deallocation methods are not listed in the type-by-type overview below. All items in the following list are either non-source breaking, or trivially automigratable. + +### `UnsafePointer` + +#### Existing methods + +``` +func withMemoryRebound(to:capacity:_:) -> Result +``` + +#### New methods + +```diff ++++ func deallocate() +``` + +### `UnsafeMutablePointer` + +#### Existing methods + +``` +static +func allocate(capacity:) -> UnsafeMutablePointer + +func withMemoryRebound(to:capacity:_:) -> Result +``` + +#### Renamed methods + +```diff +--- func initialize(to:count:) ++++ func initialize(repeating:count:) +``` + +#### New methods + +```diff ++++ func deallocate() + ++++ func initialize(to:) ++++ func assign(repeating:count:) +``` + +### `UnsafeRawPointer` + +#### Existing methods + +``` +func bindMemory(to:capacity:) -> UnsafePointer +``` + +#### New methods + +```diff ++++ func deallocate() +``` + +### `UnsafeMutableRawPointer` + +#### Existing methods + +``` +func bindMemory(to:capacity:) -> UnsafeMutablePointer +``` + +#### New methods + +```diff ++++ func deallocate() +``` + +#### Renamed methods and dropped arguments + +```diff +--- static +--- func allocate(bytes:alignedTo:) -> UnsafeMutableRawPointer + ++++ static ++++ func allocate(byteCount:alignment:) -> UnsafeMutableRawPointer + +--- func initializeMemory(as:at:count:to:) -> UnsafeMutablePointer ++++ func initializeMemory(as:repeating:count:) -> UnsafeMutablePointer + +--- func copyBytes(from:count:) ++++ func copyMemory(from:byteCount:) +``` + +### `UnsafeBufferPointer` + +#### New methods + +```diff ++++ func deallocate() + ++++ withMemoryRebound(to:_:) -> Result +``` + +### `UnsafeMutableBufferPointer` + +#### New methods + +```diff ++++ static ++++ func allocate(capacity:) -> UnsafeMutableBufferPointer ++++ func deallocate() + ++++ func initialize(repeating:) ++++ func assign(repeating:) + ++++ func withMemoryRebound(to:_:) -> Result +``` + +### `UnsafeRawBufferPointer` + +#### Existing methods + +``` +deallocate() +``` + +#### New methods + +```diff ++++ func bindMemory(to:) -> UnsafeBufferPointer +``` + +### `UnsafeMutableRawBufferPointer` + +#### Existing methods + +``` +deallocate() +``` + +#### Renamed methods and new/renamed arguments + +```diff +--- static +--- func allocate(count:) -> UnsafeMutableRawBufferPointer ++++ static ++++ func allocate(byteCount:alignment:) -> UnsafeMutableRawBufferPointer + +--- func copyBytes(from:) ++++ func copyMemory(from:) +``` + +#### New methods + +```diff ++++ func initializeMemory(as:repeating:) -> UnsafeMutableBufferPointer + ++++ func bindMemory(to:) -> UnsafeMutableBufferPointer +``` + +## What this proposal does not do + +- **attempt to fully partial initialization** + +This proposal does not attempt to fill in most of the memory state APIs for buffer pointers, as doing so necessitates designing a partial initialization system, as well as a possible buffer slice rework. + +- **address problems relating to the generic `Sequence` buffer API** + +Buffer pointers are currently missing generic `assign(from:S)` and `initializeMemory(as:S.Element.Type, from:S)` methods. The existing protocol oriented API also lacks polish and is inconvenient to use. (For example, it returns tuples.) This is an issue that can be tackled separately from the lower-level buffer-pointer-to-buffer-pointer API. + +## Detailed design + +```diff +struct UnsafePointer +{ ++++ func deallocate() + + func withMemoryRebound(to:T.Type, capacity:Int, _ body:(UnsafePointer) -> Result) + -> Result +} + +struct UnsafeMutablePointer +{ + static func allocate(capacity:Int) -> UnsafeMutablePointer + +--- func deallocate(capacity:Int) ++++ func deallocate() + +--- func initialize(to:Pointee, count:Int = 1) ++++ func initialize(repeating:Pointee, count:Int) ++++ func initialize(to:Pointee) + func initialize(from:UnsafePointer, count:Int) + func moveInitialize(from:UnsafeMutablePointer, count:Int) + ++++ func assign(repeating:Pointee, count:Int) + func assign(from:UnsafePointer, count:Int) + func moveAssign(from:UnsafeMutablePointer, count:Int) + + func deinitialize(count:Int) + + func withMemoryRebound(to:T.Type, capacity:Int, _ body:(UnsafeMutablePointer) -> Result) + -> Result +} + +struct UnsafeRawPointer +{ +--- func deallocate(bytes:Int, alignedTo:Int) ++++ func deallocate() + + func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer +} + +struct UnsafeMutableRawPointer +{ +--- static +--- func allocate(bytes:Int, alignedTo:Int) -> UnsafeMutableRawPointer ++++ static ++++ func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawPointer +--- func deallocate(bytes:Int, alignedTo:Int) ++++ func deallocate() + +--- func initializeMemory(as:T.Type, at:Int = 0, count:Int = 1, to:T) -> UnsafeMutablePointer ++++ func initializeMemory(as:T.Type, repeating:T, count:Int) -> UnsafeMutablePointer + + func initializeMemory(as:T.Type, from:UnsafePointer, count:Int) -> UnsafeMutablePointer + func moveInitializeMemory(as:T.Type, from:UnsafeMutablePointer, count:Int) + -> UnsafeMutablePointer + + func bindMemory(to:T.Type, count:Int) -> UnsafeMutablePointer + +--- func copyBytes(from:UnsafeRawPointer, count:Int) ++++ func copyMemory(from:UnsafeRawPointer, byteCount:Int) +} + +struct UnsafeBufferPointer +{ ++++ init(_:UnsafeMutableBufferPointer) + ++++ func deallocate() + ++++ func withMemoryRebound ++++ (to:T.Type, _ body:(UnsafeBufferPointer) -> Result) +} + +struct UnsafeMutableBufferPointer +{ ++++ init(mutating:UnsafeBufferPointer) + ++++ static ++++ func allocate(capacity:Int) -> UnsafeMutableBufferPointer + ++++ func initialize(repeating:Element) ++++ func assign(repeating:Element) + ++++ func withMemoryRebound ++++ (to:T.Type, _ body:(UnsafeMutableBufferPointer) -> Result) +} + +struct UnsafeRawBufferPointer +{ + func deallocate() + ++++ func bindMemory(to:T.Type) -> UnsafeBufferPointer +} + +struct UnsafeMutableRawBufferPointer +{ +--- static func allocate(count:Int) -> UnsafeMutableRawBufferPointer ++++ static func allocate(byteCount:Int, alignment:Int) -> UnsafeMutableRawBufferPointer + func deallocate() + ++++ func initializeMemory(as:T.Type, repeating:T) -> UnsafeMutableBufferPointer + ++++ func bindMemory(to:T.Type) -> UnsafeMutableBufferPointer + +--- func copyBytes(from:UnsafeRawBufferPointer) ++++ func copyMemory(from:UnsafeRawBufferPointer) +} +``` + +## Source compatibility + +Everything is additive except the following. Can we deprecate all of +the original functions in Swift 5, then drop those from the binary +later in Swift 6? + +- **add `deallocate()` to all pointer types, replacing any existing deallocation methods** + +The migrator needs to drop the existing `capacity` and `alignedTo` arguments. + +- **in `UnsafeMutableRawPointer.allocate(count:alignedTo:)` rename `count` to `byteCount` and `alignedTo` to `alignment`** + +- **in `UnsafeMutableRawBufferPointer.allocate(count:)` rename `count` to `byteCount` and add an `alignment` parameter** + +This change is source breaking but can be trivially automigrated. The +`alignment:` parameter can be filled in with `MemoryLayout.stride`. + +- **fix the arguments to `initialize(repeating:Pointee, count:Int)`** + +Note: initialize(to:Pointee) is backward compatible whenever the +caller relied on a default `count = 1`. + +An annotation could otherwise rename `to` to `repeating`, but we don't +want that to interfere with the default count case, so this might need to be a migrator rule. + +- **fix the ordering of the arguments in `initializeMemory(as:at:count:to:)`, rename `to:` to `repeating:`, and remove the `at:` argument** + +This change is source breaking but can be trivially automigrated. The +`to` argument changes position and is relabeled as `repeating`. + +The migrator could be taught to convert the `at:` argument into +pointer arithmetic on `self`. However, we found no code on Github that +uses the `at:` argument, so it is low priority. + +- **rename `copyBytes(from:count:)` to `copyMemory(from:byteCount:)`** + +This change is source breaking but can be trivially automigrated. + +## Effect on ABI stability + +Removing sized deallocators changes the existing ABI, as will renaming some of the methods and their argument labels. + +## Effect on API resilience + +Some proposed changes in this proposal change the public API. + +Removing sized deallocators right now will break ABI, but offers increased ABI and API stability in the future as reallocator methods can be added in the future without having to rename `deallocate(capacity:)` which currently occupies a “reallocator” name, but has “`free()`” behavior. + +## Alternatives considered + +- **keeping sized deallocators and fixing the stdlib implementation instead** + +Instead of dropping the `capacity` parameter from `deallocate(capacity:)`, we could fix the underlying implementation so that the function actually deallocates `capacity`’s worth of memory. However this would be catastrophically, and silently, source-breaking as existing code would continue compiling, but suddenly start leaking or segfaulting at runtime. `deallocate(capacity:)` can always be added back at a later date without breaking ABI or API, once users have been forced to address this potential bug. + +- **adding an initializer `UnsafeMutableBufferPointer.init(allocatingCount:)` instead of a type method to `UnsafeMutableBufferPointer`** + +The allocator could be expressed as an initializer instead of a type method. However since allocation involves acquisition of an external resource, perhaps it is better to keep with the existing convention that allocation is performed differently than regular buffer pointer construction. + +- **using the argument label `value:` instead of `repeating:` in methods such as `initialize(repeating:count:)` (originally `initialize(to:count:)`)** + +The label `value:` or `toValue:` doesn’t fully capture the repeating nature of the argument, and is inconsistent with `Array.init(repeating:count:)`. While `value:` sounds less strange when `count == 1`, on consistency and technical correctness, `repeating:` is the better term. Furthermore, `value` is a common variable name, meaning that function calls with `value:` as the label would be prone to looking like this: + +```swift +ptr.initialize(value: value) +``` diff --git a/proposals/0185-synthesize-equatable-hashable.md b/proposals/0185-synthesize-equatable-hashable.md new file mode 100644 index 0000000000..40a20d97d8 --- /dev/null +++ b/proposals/0185-synthesize-equatable-hashable.md @@ -0,0 +1,342 @@ +# Synthesizing `Equatable` and `Hashable` conformance + +* Proposal: [SE-0185](0185-synthesize-equatable-hashable.md) +* Author: [Tony Allevato](https://github.com/allevato) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 4.1)** +* Implementation: [apple/swift#9619](https://github.com/apple/swift/pull/9619) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0185-synthesizing-equatable-and-hashable-conformance/6493) + +## Introduction + +Developers have to write large amounts of boilerplate code to support +equatability and hashability of complex types. This proposal offers a way for +the compiler to automatically synthesize conformance to `Equatable` and +`Hashable` to reduce this boilerplate, in a subset of scenarios where generating +the correct implementation is known to be possible. + +Swift-evolution thread: [Universal Equatability, Hashability, and Comparability +](https://forums.swift.org/t/universal-equatability-hashability-and-comparability/1718) + +## Motivation + +Building robust types in Swift can involve writing significant boilerplate +code to support hashability and equatability. By eliminating the complexity for +the users, we make `Equatable`/`Hashable` types much more appealing to users and +allow them to use their own types in contexts that require equatability and +hashability with no added effort on their part (beyond declaring the +conformance). + +Equality is pervasive across many types, and for each one users must implement +the `==` operator such that it performs a fairly rote memberwise equality test. +As an example, an equality test for a basic `struct` is fairly uninteresting: + +```swift +struct Person: Equatable { + static func == (lhs: Person, rhs: Person) -> Bool { + return lhs.firstName == rhs.firstName && + lhs.lastName == rhs.lastName && + lhs.birthDate == rhs.birthDate && + ... + } +} +``` + +What's worse is that this operator must be updated if any properties are added, +removed, or changed, and since it must be manually written, it's possible to get +it wrong, either by omission or typographical error. + +Likewise, hashability is necessary when one wishes to store a type in a +`Set` or use one as a multi-valued `Dictionary` key. Writing high-quality, +well-distributed hash functions is not trivial so developers may not put a great +deal of thought into them—especially as the number of properties +increases—not realizing that their performance could potentially suffer +as a result. And as with equality, writing it manually means there is the +potential for it to not only be inefficient, but incorrect as well. + +In particular, the code that must be written to implement equality for +`enum`s is quite verbose: + +```swift +enum Token: Equatable { + case string(String) + case number(Int) + case lparen + case rparen + + static func == (lhs: Token, rhs: Token) -> Bool { + switch (lhs, rhs) { + case (.string(let lhsString), .string(let rhsString)): + return lhsString == rhsString + case (.number(let lhsNumber), .number(let rhsNumber)): + return lhsNumber == rhsNumber + case (.lparen, .lparen), (.rparen, .rparen): + return true + default: + return false + } + } +} +``` + +Crafting a high-quality hash function for this `enum` would be similarly +inconvenient to write. + +Swift already derives `Equatable` and `Hashable` conformance for a small subset +of `enum`s: those for which the cases have no associated values (which includes +enums with raw types). Two instances of such an `enum` are equal if they are the +same case, and an instance's hash value is its ordinal: + +```swift +enum Foo { + case zero, one, two +} + +let x = (Foo.one == Foo.two) // evaluates to false +let y = Foo.one.hashValue // evaluates to 1 +``` + +Likewise, conformance to `RawRepresentable` is automatically derived for `enum`s +with a raw type, and the recently approved `Encodable`/`Decodable` protocols +also support synthesis of their operations when possible. Since there is +precedent for synthesized conformances in Swift, we propose extending it to +these fundamental protocols. + +## Proposed solution + +In general, we propose that a type synthesize conformance to +`Equatable`/`Hashable` if all of its members are `Equatable`/`Hashable`. We +describe the specific conditions under which these conformances are synthesized +below, followed by the details of how the conformance requirements are +implemented. + +### Requesting synthesis is opt-in + +Users must _opt-in_ to automatic synthesis by declaring their type as +`Equatable` or `Hashable` without implementing any of their +requirements. This conformance must be part of the original type +declaration or in an extension in the same file (to ensure that +`private` and `fileprivate` members can be accessed from the +extension). + +Any type that declares such conformance and satisfies the conditions below +will cause the compiler to synthesize an implementation of `==`/`hashValue` +for that type. + +Making the synthesis opt-in—as opposed to automatic derivation without +an explicit declaration—provides a number of benefits: + +* The syntax for opting in is natural; there is no clear analogue in Swift + today for having a type opt out of a feature. + +* It requires users to make a conscious decision about the public API surfaced + by their types. Types cannot accidentally "fall into" conformances that the + user does not wish them to; a type that does not initially support `Equatable` + can be made to at a later date, but the reverse is a breaking change. + +* The conformances supported by a type can be clearly seen by examining + its source code; nothing is hidden from the user. + +* We reduce the work done by the compiler and the amount of code generated + by not synthesizing conformances that are not desired and not used. + +* As will be discussed later, explicit conformance significantly simplifies + the implementation for recursive types. + +There is one exception to this rule: the current behavior will be preserved that +`enum` types with cases that have no associated values (including those with raw +values) conform to `Equatable`/`Hashable` _without_ the user explicitly +declaring those conformances. While this does add some inconsistency to `enum`s +under this proposal, changing this existing behavior would be source-breaking. +The question of whether such `enum`s should be required to opt-in as well can +be revisited at a later date if so desired. + +Synthesis is supported in same-file extensions to ensure that generic +types can synthesize a conditional conformance, since the properties +may only satisfy the requirements for synthesis (see below) with extra +bounds: + +``` swift +struct Bad: Equatable { // synthesis not possible, T is not Equatable + var x: T +} + +struct Good { + var x: T +} +extension Good: Equatable where T: Equatable {} // synthesis works, T is Equatable +``` + +### Overriding synthesized conformances + +Any user-provided implementations of `==` or `hashValue` will override the +default implementations that would be provided by the compiler. + +### Conditions where synthesis is allowed + +For brevity, let `P` represent either the protocol `Equatable` or `Hashable` in +the descriptions below. + +#### Synthesized requirements for `enum`s + +For an `enum`, synthesis of `P`'s requirements is based on the conformances of +its cases' associated values. Computed properties are not considered. + +The following rules determine whether `P`'s requirements can be synthesized for +an `enum`: + +* The compiler does **not** synthesize `P`'s requirements for an `enum` with no + cases because it is not possible to create instances of such types. + +* The compiler synthesizes `P`'s requirements for an `enum` with one or more + cases if and only if all of the associated values of all of its cases conform + to `P`. + +#### Synthesized requirements for `struct`s + +For a `struct`, synthesis of `P`'s requirements is based on the conformances of +**only** its stored instance properties. Neither static properties nor computed +instance properties (those with custom getters) are considered. + +The following rules determine whether `P`'s requirements can be synthesized for +a `struct`: + +* The compiler trivially synthesizes `P`'s requirements for a `struct` with *no* + stored properties. (All instances of a `struct` with no stored properties can + be considered equal and hash to the same value if the user opts in to this.) + +* The compiler synthesizes `P`'s requirements for a `struct` with one or more + stored properties if and only if all of the types of all of its stored + properties conform to `P`. + +### Considerations for recursive types + +By making the synthesized conformances opt-in, recursive types have their +requirements fall into place with no extra effort. In any cycle belonging to a +recursive type, every type in that cycle must declare its conformance +explicitly. If a type does so but cannot have its conformance synthesized +because it does not satisfy the conditions above, then it is simply an error for +_that_ type and not something that must be detected earlier by the compiler in +order to reason about _all_ the other types involved in the cycle. (On the other +hand, if conformance were implicit, the compiler would have to fully traverse +the entire cycle to determine eligibility, which would make implementation much +more complex). + +### Implementation details + +An `enum T: Equatable` that satisfies the conditions above will receive a +synthesized implementation of `static func == (lhs: T, rhs: T) -> Bool` that +returns `true` if and only if `lhs` and `rhs` are the same case and have +payloads that are memberwise-equal. + +An `enum T: Hashable` that satisfies the conditions above will receive a +synthesized implementation of `var hashValue: Int { get }` that uses an +unspecified hash function to compute the hash value by incorporating +the case's ordinal (i.e., definition order) followed by the hash values of its +associated values as its terms, also in definition order. + +A `struct T: Equatable` that satisfies the conditions above will receive a +synthesized implementation of `static func == (lhs: T, rhs: T) -> Bool` that +returns `true` if and only if `lhs.x == rhs.x` for all stored properties `x` in +`T`. If the `struct` has no stored properties, this operator simply returns +`true`. + +A `struct T: Hashable` that satisfies the conditions above will receive a +synthesized implementation of `var hashValue: Int { get }` that uses an +unspecified hash function to compute the hash value by incorporating +the hash values of the fields as its terms, in definition order. If the `struct` +has no stored properties, this property evaluates to a fixed value not specified +here. + + The choice of hash function is left as an implementation detail, +not a fixed part of the design; as such, users should not depend on specific +characteristics of its behavior. The most likely implementation would call the +standard library's `_mixInt` function on each member's hash value and then +combine them with exclusive-or (`^`), which mirrors the way `Collection` types +are hashed today. + +## Source compatibility + +By making the conformance opt-in, this is a purely additive change that does +not affect existing code. We also avoid source-breaking changes by not changing +the behavior for `enum`s with no associated values, which will continue to +implicitly conform to `Equatable` and `Hashable` even without explicitly +declaring the conformance. + +## Effect on ABI stability + +This feature is purely additive and does not change ABI. + +## Effect on API resilience + +N/A. + +## Alternatives considered + +In order to realistically scope this proposal, we considered but ultimately +deferred the following items, some of which could be proposed additively in the +future. + +### Synthesis for `class` types and tuples + +We do not synthesize conformances for `class` types. The conditions above become +more complicated in inheritance hierarchies, and equality requires that +`static func ==` be implemented in terms of an overridable instance method for +it to be dispatched dynamically. Even for `final` classes, the conditions are +not as clear-cut as they are for value types because we have to take superclass +behavior into consideration. Finally, since objects have reference identity, +memberwise equality may not necessarily imply that two instances are equal. + +We do not synthesize conformances for tuples at this time. While this would +nicely round out the capabilities of value types, allow the standard library to +remove the hand-crafted implementations of `==` for up-to-arity-6 tuples, and +allow those types to be used in generic contexts where `Equatable` conformance +is required, adding conformances to non-nominal types would require additional +work. + +### Omitting fields from synthesized conformances + +Some commenters have expressed a desire to tag certain properties of a `struct` +from being included in automatically generated equality tests or hash value +computations. This could be valuable, for example, if a property is merely used +as an internal cache and does not actually contribute to the "value" of the +instance. Under the rules above, if this cached value was equatable, a user +would have to override `==` and `hashValue` and provide their own +implementations to ignore it. + +Such a feature, which could be implemented with an attribute such as +`@transient`, would likely also play a role in other protocols like +`Encodable`/`Decodable`. This could be done as a purely additive change on top +of this proposal, so we propose not doing this at this time. + +### Implicit derivation + +An earlier draft of this proposal made derived conformances implicit (without +declaring `Equatable`/`Hashable` explicitly). This has been changed +because—in addition to the reasons mentioned earlier in the +proposal—`Encodable`/`Decodable` provide a precedent for having the +conformance be explicit. More importantly, however, determining derivability for +recursive types is _significantly more difficult_ if conformance is implicit, +because it requires examining the entire dependency graph for a particular type +and to properly handle cycles in order to decide if the conditions are +satisfied. + +### Support for `Comparable` + +The original discussion thread also included `Comparable` as a candidate for +automatic generation. Unlike equatability and hashability, however, +comparability requires an ordering among the members being compared. +Automatically using the definition order here might be too surprising for users, +but worse, it also means that reordering properties in the source code changes +the code's behavior at runtime. (This is true for hashability as well if a +multiplicative hash function is used, but hash values are not intended to be +persistent and reordering the terms does not produce a significant _behavioral_ +change.) + +## Acknowledgments + +Thanks to Joe Groff for spinning off the original discussion thread, Jose Cheyo +Jimenez for providing great real-world examples of boilerplate needed to support +equatability for some value types, Mark Sands for necromancing the +swift-evolution thread that convinced me to write this up, and everyone on +swift-evolution since then for giving me feedback on earlier drafts. diff --git a/proposals/0186-remove-ownership-keyword-support-in-protocols.md b/proposals/0186-remove-ownership-keyword-support-in-protocols.md new file mode 100644 index 0000000000..7ec3fa0194 --- /dev/null +++ b/proposals/0186-remove-ownership-keyword-support-in-protocols.md @@ -0,0 +1,64 @@ +# Remove ownership keyword support in protocols + +* Proposal: [SE-0186](0186-remove-ownership-keyword-support-in-protocols.md) +* Author: [Greg Spiers](https://github.com/gspiers) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 4.1)** +* Implementation: [apple/swift#11744](https://github.com/apple/swift/pull/11744) +* [Review thread on swift-evolution](https://forums.swift.org/t/se-0186-remove-ownership-keyword-support-in-protocols/6678) +* Bug: [SR-479](https://bugs.swift.org/browse/SR-479) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0186-remove-ownership-keyword-support-in-protocols/6736) + +## Introduction + +This proposal removes support for the keywords `weak` and `unowned` for property declarations in a protocol. + +Swift-evolution thread: [Ownership on protocol property requirements](https://forums.swift.org/t/ownership-on-protocol-property-requirements/5872) thread. + +## Motivation + +Currently it's possible to use the weak/unowned keywords for a property requirement in a protocol. This can lead to confusion as specifying one of these keywords does not enforce or raise any warnings in the adopting type of that protocol: + +```swift + +class A {} + +protocol P { + weak var weakVar: A? { get set } +} + +class B: P { + var weakVar: A? // Not declared weak, no compiler warning/error +} + +``` + +This can lead to unexpected and surprising behaviour from the point of view of users. The keywords themselves are currently meaningless inside of a protocol but look like they would have an effect when the protocol is adopted. + +This change is consistent with removing keywords that do not have any meaning like `final` in protocol extensions: [SE-0164](0164-remove-final-support-in-protocol-extensions.md). + +## Proposed solution + +Although the case could be made that the keywords should have meaning in a protocol, as they are currently implemented today they don't have an effect. This proposal aims to cleanup the misleading syntax and isn't meant to remove functionality only correct to existing behaviour. + +This proposal suggests removing support for `weak` and `unowned` in a protocol. + +## Detailed design + +In existing Swift modes, 3 and 4, the compiler will warn about the use of `weak` and `unowned` in a protocol and suggest a fix to remove the keywords. In Swift 5 mode the compiler will error and offer a fixit to remove the keywords. + +## Source compatibility + +This is a source breaking change but one that would only correct code that already has broken assumptions. For existing Swift modes, 3 and 4, the compiler will raise a compilation warning instead of an error. + +## Effect on ABI stability + +This proposal does not affect ABI stability. + +## Effect on API resilience + +This proposal does not affect API resilience. + +## Alternatives considered + +There is an argument in making `weak` and `unowned` have meaning in a protocol but this does open up other questions and is probably better as a topic of a separate discussion/proposal. As this would be additive it can be addressed at a later point when we have a clearer understanding. diff --git a/proposals/0187-introduce-filtermap.md b/proposals/0187-introduce-filtermap.md new file mode 100644 index 0000000000..a03909edc1 --- /dev/null +++ b/proposals/0187-introduce-filtermap.md @@ -0,0 +1,131 @@ +# Introduce Sequence.compactMap(_:) + +* Proposal: [SE-0187](0187-introduce-filtermap.md) +* Author: [Max Moiseev](https://github.com/moiseev) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 4.1)** +* Implementation: [apple/swift#12819](https://github.com/apple/swift/pull/12819) +* Decision Notes: + [Review #0](https://forums.swift.org/t/draft-introduce-sequence-filteredmap/6872), + [Review #1](https://forums.swift.org/t/review-se-0187-introduce-sequence-filtermap/6977), + [Review #2](https://forums.swift.org/t/accepted-and-focused-re-review-se-0187-introduce-sequence-filtermap/7076), + [Rationale](https://forums.swift.org/t/accepted-with-revisions-se-0187-introduce-sequence-filtermap/7290) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/2d24b0ce9f138858b8341467170d6d8ba973827f/proposals/0187-introduce-filtermap.md) + +## Introduction + +We propose to deprecate the controversial version of a `Sequence.flatMap` method +and provide the same functionality under a different, and potentially more +descriptive, name. + +## Motivation + +The Swift standard library currently defines 3 distinct overloads for `flatMap`: + +~~~swift +Sequence.flatMap(_: (Element) -> S) -> [S.Element] + where S : Sequence +Optional.flatMap(_: (Wrapped) -> U?) -> U? +Sequence.flatMap(_: (Element) -> U?) -> [U] +~~~ + +The last one, despite being useful in certain situations, can be (and often is) +misused. Consider the following snippet: + +~~~swift +struct Person { + var age: Int + var name: String +} + +func getAges(people: [Person]) -> [Int] { + return people.flatMap { $0.age } +} +~~~ + +What happens inside `getAges` is: thanks to the implicit promotion to +`Optional`, the result of the closure gets wrapped into a `.some`, then +immediately unwrapped by the implementation of `flatMap`, and appended to the +result array. All this unnecessary wrapping and unwrapping can be easily avoided +by just using `map` instead. + +~~~swift +func getAges(people: [Person]) -> [Int] { + return people.map { $0.age } +} +~~~ + +It gets even worse when we consider future code modifications, like the one +where Swift 4 introduced a `String` conformance to the `Collection` protocol. +The following code used to compile (due to the `flatMap` overload in question). + +~~~swift +func getNames(people: [Person]) -> [String] { + return people.flatMap { $0.name } +} +~~~ + +But it no longer does, because now there is a better overload that does not +involve implicit promotion. In this particular case, the compiler error would be +obvious, as it would point at the same line where `flatMap` is used. Imagine +however if it was just a `let names = people.flatMap { $0.name }` statement, and +the `names` variable were used elsewhere. The compiler error would be +misleading. + +## Proposed solution + +We propose to deprecate the controversial overload of `flatMap` and re-introduce +the same functionality under a new name. The name being `compactMap(_:)` as we +believe it best describes the intent of this function. + +For reference, here are the alternative names from other languages: +- Haskell, Idris + `
mapMaybe :: (a -> Maybe b) -> [a] -> [b]` +- Ocaml (Core and Batteries)
 + `filter_map : 'a t -> f:('a -> 'b option) -> 'b t` +- F#
 + `List.choose : ('T -> 'U option) -> 'T list -> 'U list` +- Rust
 + `fn filter_map(self, f: F) -> FilterMap
 where F: FnMut(Self::Item) -> Option` +- Scala + `
def collect[B](pf: PartialFunction[A, B]): List[B]` + +Filtering `nil` elements from the `Sequence` is very common, therefore we also +propose adding a `Sequence.compact()` function. This function should only be +available for sequences of optional elements, which is not expressible in +current Swift syntax. Until we have the missing features, using +`xs.compactMap { $0 }` is an option. + +## Source compatibility + +Since the old function will still be available (although deprecated) all +the existing code will compile, producing a deprecation warning and a fix-it. + +## Effect on ABI stability + +This is an additive API change, and does not affect ABI stability. + +## Effect on API resilience + +Ideally, the deprecated `flatMap` overload would not exist at the time when ABI +stability is declared, but in the worst case, it will be available in a +deprecated form from a library post-ABI stability. + +## Alternatives considered + +It was attempted in the past to warn about this kind of misuse and do the right +thing instead by means of a deprecated overload with a non-optional-returning +closure. The attempt failed due to another implicit promotion (this time to +`Any`). + +The following alternative names for this function were considered: +- `mapNonNil(_:)
` + Does not communicate what happens to nil’s +- `mapSome(_:)
` + Reads more like «map some elements of the sequence, but not the others» + rather than «process only the ones that produce an Optional.some» +- `filterMap(_:)` + Considered confusing, due to similarity with `filter`, but without any control + over what gets filtered out. Besides, even though it can be implemented as a + series of calls to `filter` and `map`, the order of these calls is different + from what the `filterMap` name suggests. diff --git a/proposals/0188-stdlib-index-types-hashable.md b/proposals/0188-stdlib-index-types-hashable.md new file mode 100644 index 0000000000..a1e2c5a83c --- /dev/null +++ b/proposals/0188-stdlib-index-types-hashable.md @@ -0,0 +1,68 @@ +# Make Standard Library Index Types Hashable + +* Proposal: [SE-0188](0188-stdlib-index-types-hashable.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 4.1)** +* Implementation: [apple/swift#12777](https://github.com/apple/swift/pull/12777) + +## Introduction + +Key-path expressions can now include subscripts to reference individual positions in collections and other subscriptable types, but only when the subscript parameters are `Hashable`. To provide maximum utility, the standard library index types should all have `Hashable` conformance added. + +Swift-evolution "thread:" [[draft] Make Standard Library Index Types Hashable](https://forums.swift.org/t/draft-make-standard-library-index-types-hashable/6946) + +## Motivation + +You can only use subscripts in key-path expressions when the subscript parameter type is `Hashable`. This means that you can use a subscript as part of a key-path expression with an array, which uses `Int` as its index type, but not with a string, which uses a custom index type. + +```swift +let numbers = [10, 20, 30, 40, 50] +let firstValue = \[Int].[0] +print(numbers[keyPath: firstValue]) // 10 + +let string = "Helloooo!" +let firstChar = \String.[string.startIndex] +// error: subscript index of type 'String.Index' in a key path must be Hashable +``` + +## Proposed solution + +This proposal would add `Hashable` conformance to all the index types in the standard library. With that done, `[Int]`, `String`, and all other standard library collections would have the same behavior when using subscripts in key paths. + +## Detailed design + +For index types that wrap an internal offset or other value, adding `Hashable` conformance will be simple. For index types that wrap another index type, such as `ReversedIndex`, `Hashable` conformance must wait until the implementation of conditional conformance is complete. + +This is the breakdown of the standard library's index types: + +#### Simple Index Types + +- `Int` (already `Hashable`) +- `Dictionary.Index` +- `Set.Index` +- `String.Index` + +#### Wrapping Index Types + +- `ClosedRangeIndex` +- `FlattenCollectionIndex` +- `LazyDropWhileIndex` +- `LazyFilterIndex` +- `LazyPrefixWhileIndex` +- `ReversedIndex` + +`AnyIndex`, which type erases any index type at run-time, would not be hashable since it might wrap a non-hashable type. + +## Source compatibility + +This is an additive change in the behavior of standard library index types, so it should pose no source compatibility burden. Specifically, this proposal does *not* change the requirements for an index type in the `Collection` protocol, so collections and custom index types that have been written in prior versions of Swift will be unaffected. + +## Effect on ABI stability & API resilience + +Beyond an additional conformance for the types mentioned above, this proposal has no effect on ABI stability or API resilience. + +## Alternatives considered + +None. + diff --git a/proposals/0189-restrict-cross-module-struct-initializers.md b/proposals/0189-restrict-cross-module-struct-initializers.md new file mode 100644 index 0000000000..a587ec18c4 --- /dev/null +++ b/proposals/0189-restrict-cross-module-struct-initializers.md @@ -0,0 +1,133 @@ +# Restrict Cross-module Struct Initializers + +* Proposal: [SE-0189](0189-restrict-cross-module-struct-initializers.md) +* Author: [Jordan Rose](https://github.com/jrose-apple) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 4.1)** +* Implementation: [apple/swift#12834](https://github.com/apple/swift/pull/12834) +* Pre-review discussion: [Restrict Cross-module Struct Initializers](https://forums.swift.org/t/pitch-restrict-cross-module-struct-initializers/6780) +* [Swift Evolution Review Thread](https://forums.swift.org/t/review-se-0189-restrict-cross-module-struct-initializers/7066) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0189-restrict-cross-module-struct-initializers/7164) + +## Introduction + +Adding a property to a public struct in Swift ought to not be a source-breaking change. However, a client in another target can currently extend a struct with a new initializer that directly initializes the struct's fields. This proposal forbids that, requiring any cross-target initializers to use `self.init(…)` or assign to `self` instead. This matches an existing restriction for classes, where cross-module initializers must be convenience initializers. + + +## Motivation + +Swift structs are designed to be flexible, allowing library authors to change their implementation between releases. This goes all the way to changing the set of stored properties that make up the struct. Since initializers have to initialize every stored property, they have two options: + +- Assign each property before returning or using `self`. +- Assign all properties at once by using `self.init(…)` or `self = …`. + +The former requires knowing every stored property in the struct. If all of those properties happen to be public, however, a client in another target can implement their own initializer, and suddenly adding a new stored property (public or not) becomes a source-breaking change. + +Additionally, initializers are often used with `let` properties to enforce a struct's invariants. Consider this (contrived) example: + +```swift +public struct BalancedPair { + public let positive: Int + public let negative: Int + public init(absoluteValue: Int) { + assert(absoluteValue >= 0) + self.positive = absoluteValue + self.negative = -absoluteValue + } +} +``` + +At this point a user of BalancedPair ought to be able to assume that `positive` and `negative` always hold opposite values. However, an unsuspecting (or malicious) client could add their own initializer that breaks this invariant: + +```swift +import ContrivedExampleKit +extension BalancedPair { + init(positiveOnly value: Int) { + self.positive = value + self.negative = 0 + } +} +``` + +Anything that prevents the library author from enforcing the invariants of their type is a danger and contrary to the spirit of Swift. + + +## Proposed solution + +If an initializer is declared in a different module from a struct, it must use `self.init(…)` or `self = …` before returning or accessing `self`. Failure to do so will produce a warning in Swift 4 and an error in Swift 5. + +The recommendation for library authors who wish to continue allowing this is to explicitly declare a public memberwise initializer for clients in other modules to use. + + +### C structs + +C structs are not exempt from this rule, but all C structs are imported with a memberwise initializer anyway. This *still* does not guarantee source compatibility because C code owners occasionally decide to split up or rename members of existing structs, but this proposal does not make that situation worse. Most C structs also have a no-argument initializer that fills the struct with zeros unless one of the members is marked `_Nonnull`. + + +## Source compatibility + +This makes existing code invalid in Swift 5, which is a source compatibility break. + +This makes adding a stored property to a struct a source-compatible change (except for Swift 4 clients that choose to ignore the warning). + + +## Effect on ABI stability + +This is required for structs to avoid exposing the layout of their properties in a library's binary interface. + + +## Effect on Library Evolution + +It is now a binary-compatible change to add a public or non-public stored property to a struct. + +It is still not a binary-compatible change to remove a public stored property from a struct. + + +## Alternatives considered + +### Do nothing + +We've survived so far, so we can live without this for libraries that don't have binary compatibility concerns, but not being able to enforce invariants is still a motivating reason to do this proposal. + + +### Distinguish between "structs with a fixed set of stored properties" and "structs that may get new stored properties later" + +This actually *is* a distinction we want to make for code in frameworks with binary compatibility constraints, where the ability to add new members to a struct forces client code to use extra indirection. (We've been spelling this `@_fixed_layout`, though that's just a placeholder.) However, enforcing invariants may still be relevant for such a "fixed-layout" struct, and a library author can get nearly the same effect simply by defining a public memberwise initializer, something that's common to do anyway. (If performance is a concern, the initializer can also be marked inlinable.) We don't think there should ever be a reason to annotate a struct as "fixed-layout" in a source package, and we wouldn't want this to become one. + + +### Allow stored-property-wise initialization just for C structs + +C structs are similar to the "fixed-layout" structs described above in that their layout is known at compile time, and since that's just a property of C there's no annotation cost. However, allowing this would create an unnecessary distinction between C structs and Swift structs. + +Additionally, there have been requests in the past for a C-side annotation to restrict access to the implicit no-argument and memberwise initializers provided by the Swift compiler. This has been motivated by C structs that do effectively have invariants; just as C++ allows a library author to restrict how a struct may be initialized, so could Swift. This is just a possible future change (and probably unlikely to happen in Swift 5), but it works better with this proposal than without it. + + +### Add an exception for unit tests + +An earlier version of the proposal included an exception for structs in modules imported as `@testable`, allowing unit tests to bypass the restriction that required calling an existing initializer. However, this can already be accomplished by providing an initializer marked `internal` in the original library. + +```swift +public struct ExportConfiguration { + public let speed: Int + public let signature: String + public init(from fileURL: URL) {…} + internal init(manualSpeed: Int, signature: String) {…} +} +``` + +```swift +import XCTest +@testable import MyApp + +class ExportTests: XCTestCase { + func testSimple() { + // Still avoids having to load from a file. + let config = ExportConfiguration(manualSpeed: 5, signature: "abc") + let op = ExportOperation(config) + } +} +``` + +The downside is that the initializer is available to the rest of the module, which probably is not supposed to call it. + +Allowing per-stored-property initializers for `@testable` imports is an additive feature; if it turns out to be a common pain point, we can add it in a later proposal. Leaving it out means `@testable` remains primarily about access control. diff --git a/proposals/0190-target-environment-platform-condition.md b/proposals/0190-target-environment-platform-condition.md new file mode 100644 index 0000000000..8de170fe27 --- /dev/null +++ b/proposals/0190-target-environment-platform-condition.md @@ -0,0 +1,143 @@ +# Target environment platform condition + +* Proposal: [SE-0190](0190-target-environment-platform-condition.md) +* Authors: [Erica Sadun](https://github.com/erica), [Graydon Hoare](https://github.com/graydon) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 4.1)** +* [Swift Evolution Review Thread](https://forums.swift.org/t/review-se-0190-target-environment-platform-condition/7082) +* Implementation: [apple/swift#12964](https://github.com/apple/swift/pull/12964) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0190-target-environment-platform-condition/7138) + +## Introduction + +This proposal introduces a platform condition to differentiate device and simulator builds. +This condition subsumes a common pattern of conditional compilation for Metal, Keychain, and +AVFoundation Camera code. + +Swift-evolution threads: + +* [Expanding Build Configuration Tests for Simulator and Device targets](https://forums.swift.org/t/draft-expanding-build-configuration-tests-for-simulator-and-device-targets/1793) +* [Target environment platform condition](https://forums.swift.org/t/draft-target-environment-platform-condition/6884) + +## Motivation + +A common developer requirement is to conditionally compile code based on whether +the current compilation target is a simulator or a real device. The current technique +for accomplishing this involves testing for particular combinations of presumed mismatch +between architecture and operating system. This is fragile and non-obvious, and +requires reasoning about complex nested conditions that obscure the user's purpose. + +For example, code often looks like this: + + +```swift +// Test for a simulator destination +#if (arch(i386) || arch(x86_64)) && (!os(macOS)) + print("Simulator") +#else + print("Device") +#endif + +// More restrictive test for iOS simulator +// Adjust the os test for watchOS, tvOS +#if (arch(i386) || arch(x86_64)) && os(iOS) + // iOS simulator code +#endif +``` + +## Proposed Solution + +This proposal adds a new platform condition `targetEnvironment` with a single valid +argument: `simulator`. + +In other words, the proposal is to enable conditional compilation of the form +`#if targetEnvironment(simulator)`. + +## Detailed Design + +When the compiler is targeting simulator environments, the `targetEnvironment(simulator)` +condition will evaluate to `true`. Otherwise it will evaluate as `false`. + +In the future, other target environments may be indicated using different arguments to +the `targetEnvironment` condition. It is a general extension point for disambiguating +otherwise-similar target environments. + +The name of the condition is motivated by the fact that an unambiguous indication of +target environment can be made using the 4th (seldom used, but valid) _environment_ field of +the _target triple_ provided to the compiler. + +In other words, if the compiler's target triple is specified with an _environment_ +field such as `arm64-apple-tvos-simulator`, the `targetEnvironment(simulator)` condition +will be set. + +As a transitionary measure: until users have migrated to consistent use of _target triples_ +with an explicit `simulator` value in the _environment_ field, the Swift compiler +will infer it from the remaining components of the target triple, without requiring the +user to approximate the condition through combinations of `os` and `arch` platform +conditions. + +In other words, while a given _target triple_ may be missing the _environment_ field, +the `targetEnvironment(simulator)` condition may still be `true`, if it is inferred +that the current target triple denotes a simulator environment. + +## Source compatibility + +This is an additive proposal, existing code will continue to work. + +A warning and fixit may be provided for migrating recognizable cases in existing code, +but this will necessarily be best-effort, as existing conditions may be arbitrarily +complex. + +## Effect on ABI stability + +None + +## Effect on API resilience + +None + +## Current Art + +Swift currently supports the following platform conditions: + +* The `os()` function that tests for `macOS, iOS, watchOS, tvOS, Linux, Windows, FreeBSD, + Android, PS4, Cygwin and Haiku` +* The `arch()` function that tests for `x86_64, arm, arm64, i386, powerpc64, powerpc64le and s390x` +* The `swift()` function that tests for specific Swift language releases, e.g. `swift(>=2.2)` + +## Comparison with other languages + +1. [Rust's conditional compilation system](https://doc.rust-lang.org/reference/attributes.html#conditional-compilation) + includes the `target_env` configuration option, which similarly presents the _environment_ + field of the target triple. +2. In Clang, several _environment_-based preprocessor symbols can be used to achieve similar + effects (`__CYGWIN__`, `__ANDROID__`, etc.) though the mapping is quite ad-hoc and the + 4th field of the _target triple_ is + [officially documented](https://clang.llvm.org/docs/CrossCompilation.html#target-triple) + as representing the target _ABI_. In + [the implementation](https://github.com/apple/swift-llvm/blob/352a3d745c4ed4d24c1e5a86ec0e1b2af2f0dfa4/include/llvm/ADT/Triple.h#L219-L245), + however, the 4th field is treated as _environment_ (subsuming _ABI_) and a 5th field + for _object format_ is supported. +3. Clang also supports various flags such as `-mtvos-simulator-version-min` which define a + simulator-specific preprocessor symbol `__APPLE_EMBEDDED_SIMULATOR__`. + + +## Alternatives Considered + +Some possible alternatives were considered: + + 1. As in the first round of this proposal, `target(simulator)`. This has the advantage + of brevity, but the disadvantage of using a relatively overloaded term, and contradicts + the existing design of using a separate condition per-component of the target triple + (`os()` and `arch()`). + 2. A similarly brief `environment(simulator)` condition, which has the disadvantage that + users may mistake it for a means of accessing environment variables of the compiler + process. + 3. An additional state for the `os` or `arch` conditions, such as `os(simulator)`. + This would complicate both the definition and implementation of platform conditions, + while blurring the notion of an operating system. + 4. Avoidance of the target triple altogether, and use of a dedicated `simulator()` + platform condition. This is the simplest option, but is less-similar to existing + conditions and may introduce more meaningless combinations of flags as the set of + target environments grows (rather than mutually exclusive arguments to + `targetEnvironment`). diff --git a/proposals/0191-eliminate-indexdistance.md b/proposals/0191-eliminate-indexdistance.md new file mode 100644 index 0000000000..73a23ed9d8 --- /dev/null +++ b/proposals/0191-eliminate-indexdistance.md @@ -0,0 +1,87 @@ +# Eliminate `IndexDistance` from `Collection` + +* Proposal: [SE-0191](0191-eliminate-indexdistance.md) +* Author: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.1)** +* Implementation: [apple/swift#12641](https://github.com/apple/swift/pull/12641) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0191-eliminate-indexdistance-from-collection/7191) + +## Introduction + +Eliminate the associated type `IndexDistance` from `Collection`, and modify all uses to the concrete type `Int` instead. + +## Motivation + +`Collection` allows for the distance between two indices to be any `SignedInteger` type via the `IndexDistance` associated type. While in practice the distance between indices is almost always +an `Int`, generic algorithms on `Collection` need to either constrain `IndexDistance == Int` or write their algorithm to be generic over any `SignedInteger`. + +Swift 4.0 introduced the ability to constrain associated types with `where` clauses +([SE-142](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md)) and will soon allow protocol constraints +to be recursive ([SE-157](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0157-recursive-protocol-constraints.md)). With these features, +writing generic algorithms against `Collection` is finally a realistic tool for intermediate Swift programmers. You no longer need to know to +constrain `SubSequence.Element == Element` or `SubSequence: Collection`, missing constraints that previously led to inexplicable error messages. + +At this point, the presence of `IndexDistance` is of of the biggest hurdles that new users trying to write generic algorithms face. If you want to +write code that will compile against any distance type, you need to constantly juggle with explicit type annotations (i.e. you need to write `let i: +IndexDistance = 0` instead of just `let i = 0`), and perform `numericCast` to convert from one distance type to another. + +But these `numericCasts` are hard to use correctly. Given two collections with different index distances, it's very hard to reason about whether your +`numericCast` is casting from the smaller to larger type correctly. This turns any problem of writing a generic collection algorithm into both a collection _and_ +numeric problem. And chances are you are going to need to interoperate with a method that takes or provides a concrete `Int` anyway (like `Array.reserveCapacity` inside +`Collection.map`). Much of the generic code in the standard library would trap if ever presented with a collection with a distance greater than `Int.max`. +Additionally, this generalization makes specialization less likely and increases compile-time work. + +For these reasons, it's common to see algorithms constrained to `IndexDistance == Int`. In fact, the inconvenience of having to deal with generic index +distances probably encourages more algorithms to be constrained to `Index == Int`, such as [this +code](https://github.com/airspeedswift/swift-package-manager/blob/472c647dcad3adf4344a06ef7ba91d2d4abddc94/Sources/Basic/OutputByteStream.swift#L119) in +the Swift Package Manager. Converting this function to work with any index type would be straightforward. Converting it to work with any index distance +as well would be much trickier. + +The general advice from [The Swift Programming +Language](https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/TheBasics.html#//apple_ref/doc/uid/TP40014097-CH5-ID309) when writing Swift code is to encourage users to stick to using `Int` unless they have a special reason not to: + +> Unless you need to work with a specific size of integer, always use `Int` for integer values in your code. [...] `Int` is preferred, even when the values to be stored are known to be nonnegative. A consistent use of Int for integer values aids +code interoperability, avoids the need to convert between different number types, and matches integer type inference[.] + +There are two main use cases for keeping `IndexDistance` as an associated type rather than concretizing it to be `Int`: tiny collections that might +benefit from tiny distances, and huge collections that need to address greater than `Int.max` elements. For example, it may seem wasteful to force a +type that presents the bits in a `UInt` as a collection to need to use a whole `Int` for its distance type. Or you may want to create a gigantic +collection, such as one backed by a memory mapped file, with a size great than `Int.max`. The most likely scenario for this is on 32-bit processors where a collection would be constrained to 2 billion elements. + +These use cases are very niche, and do not seem to justify the considerable impedance to generic programming that `IndexDistance` causes. Therefore, +this proposal recommends removing the associated type and replacing all references to it with `Int`. + +## Proposed solution + +Scrap the `IndexDistance` associated type. Switch all references to it in the standard library to the concrete `Int` type: + +```swift +protocol Collection { + var count: Int { get } + func index(_ i: Index, offsetBy n: Int) -> Index + func index(_ i: Index, offsetBy n: Int, limitedBy limit: Index) -> Index? + func distance(from start: Index, to end: Index) -> Int +} +// and in numerous extensions in the standard library +``` + +The one instance where a concrete type uses an `IndexDistance` other than `Int` in the standard library is `AnyCollection`, which uses `Int64`. This would be changed to `Int`. + +## Source compatibility + +This can be split into 2 parts: + +Algorithms that currently constrain `IndexDistance` to `Int` in their `where` clause, and algorithms that use `IndexDistance` within the body of a +method, can be catered for by a deprecated typealias for `IndexDistance` inside an extension on `Collection`. This is the common case. + +Collections that truly take advantage of the ability to define non-`Int` distances would be source-broken, with no practical way of making this +compatible in a 4.0 mode. It's worth noting that there are no such types in the Swift source compatibility suite. + +## Effect on ABI stability + +This removes an associated type and changes function signatures, so must be done before declaring ABI stability + +## Alternatives considered + +None other than status quo. diff --git a/proposals/0192-non-exhaustive-enums.md b/proposals/0192-non-exhaustive-enums.md new file mode 100644 index 0000000000..534cef4c91 --- /dev/null +++ b/proposals/0192-non-exhaustive-enums.md @@ -0,0 +1,653 @@ +# Handling Future Enum Cases + +* Proposal: [SE-0192](0192-non-exhaustive-enums.md) +* Author: [Jordan Rose](https://github.com/jrose-apple) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#14945](https://github.com/apple/swift/pull/14945) +* Previous revision: [1](https://github.com/swiftlang/swift-evolution/blob/a773d07ff4beab8b7855adf0ac56d1e13bb7b44c/proposals/0192-non-exhaustive-enums.md), [2 (informal)](https://github.com/jrose-swiftlang/swift-evolution/blob/57dfa2408fe210ed1d5a1251f331045b988ee2f0/proposals/0192-non-exhaustive-enums.md), [3](https://github.com/swiftlang/swift-evolution/blob/af284b519443d3d985f77cc366005ea908e2af59/proposals/0192-non-exhaustive-enums.md) +* Pre-review discussion: [Enums and Source Compatibility](https://forums.swift.org/t/enums-and-source-compatibility/6460), with additional [orphaned thread](https://forums.swift.org/t/enums-and-source-compatibility/6651) +* Review discussion: [Review author summarizes some feedback from review discussion and proposes alternatives](https://forums.swift.org/t/se-0192-non-exhaustive-enums/7291/26), [full discussion thread](https://forums.swift.org/t/se-0192-non-exhaustive-enums/7291/337), plus [Handling unknown cases in enums](https://forums.swift.org/t/handling-unknown-cases-in-enums-re-se-0192/7388/) +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0192-non-exhaustive-enums-review-2/11043/62) + +## Introduction + +Currently, adding a new case to an enum is a source-breaking change, something that's at odds with Apple's established process for evolving APIs. This proposal aims to distinguish between enums that are _frozen_ (meaning they will never get any new cases) and those that are _non-frozen,_ and to ensure that clients handle any future cases when dealing with the latter. + +A key note: in this version of the proposal, *nothing changes for user-defined Swift enums.* This only affects C enums and enums in the standard library and overlays today. (This refers to libraries that Apple could hypothetically ship with its OSs, as it does with Foundation.framework and the Objective-C runtime.) The features described here may be used by third-party libraries in the future. + + +### Post-acceptance revision + +- Since the proposal was accepted months after it was written, the rollout plan turned out to be a little too aggressive. Therefore, in Swift 5 the diagnostic for omitting `@unknown default:` or `@unknown case _:` will only be a warning, and in Swift 4 mode there will be no diagnostic at all. (The previous version of the proposal used an error and a warning, respectively.) Developers are still free to use `@unknown` in Swift 4 mode, in which case the compiler will still produce a warning if all known cases are not handled. + + +### Revision at acceptance + +- The "new case" `unknown:` was changed to the `unknown` attribute, which can only be applied to `default:` and `case _:`. + + +### Differences from the first revision + +- [This now only affects C enums and enums defined in the standard library and overlays](https://forums.swift.org/t/se-0192-non-exhaustive-enums/7291/337) +- The `unknown` case has been added, to preserve exhaustivity checking +- The term used to describe enums that will not change is now "frozen" rather than "exhaustive" +- The proposal now describes what will happen if you "break the contract" in a new library version +- Much more discussion of future directions and alternatives considered + +Thanks to everyone who offered feedback! + + +## Motivation + +It's well-established that many enums need to grow new cases in new versions of a library. For example, in last year's release of iOS 10, Foundation's [DateComponentsFormatter.UnitsStyle][] gained a `brief` case and UIKit's [UIKeyboardType][] gained an `asciiCapableNumberPad` case. Large error enums also often grow new cases to go with new operations supported by the library. This all implies that library authors *must* have a way to add new cases to enums without breaking binary compatibility. + +At the same time, we really like that you can exhaustively switch over enums. This feature helps prevent bugs and makes it possible to enforce [definitive initialization][DI] without having `default` cases in every `switch`. So we don't want to get rid of enums where every case is known, either. This calls for a distinction between enums where every case can be known statically and enums that might grow new cases in the future. + +To see how this distinction will play out in practice, I investigated the public headers of Foundation in the macOS SDK. Out of all 60 or so `NS_ENUM`s in Foundation, only 6 of them are clearly intended to be switched exhaustively: + +- [ComparisonResult](https://developer.apple.com/documentation/foundation/comparisonresult) +- [NSKeyValueChange](https://developer.apple.com/documentation/foundation/nskeyvaluechange) / [NSKeyValueSetMutationKind](https://developer.apple.com/documentation/foundation/nskeyvaluesetmutationkind) +- [NSRectEdge](https://developer.apple.com/documentation/foundation/nsrectedge) +- [FileManager.URLRelationship](https://developer.apple.com/documentation/foundation/filemanager.urlrelationship) +- *maybe* [Decimal.CalculationError](https://developer.apple.com/documentation/foundation/nsdecimalnumber.calculationerror) + +...with a handful more that could go either way, such as [Stream.Status](https://developer.apple.com/documentation/foundation/stream.status). This demonstrates that there is a clear default for public enums, at least in Objective-C. + + [DateComponentsFormatter.UnitsStyle]: https://developer.apple.com/documentation/foundation/datecomponentsformatter.unitsstyle + [UIKeyboardType]: https://developer.apple.com/documentation/uikit/uikeyboardtype + [DI]: https://developer.apple.com/swift/blog/?id=28 + + +## Proposed solution + +In Swift 4.2, enums imported from C and enums defined in the standard library and overlays are either *frozen* or *non-frozen.* (Grammatical note: they are not "unfrozen" because that implies that they were frozen at one point.) + +When a client tries to switch over a non-frozen enum, they should include a "catch-all" case of some kind (`default`, `case _`, etc). In Swift 5 mode, omitting this case will result in a warning. + +All enums written in Swift outside of the standard library and overlays will implicitly be considered frozen in Swift 4.2. Enums imported from C will be non-frozen by default, with a new C-side annotation to treat them as frozen. + + +## Detailed design + +When switching over a non-frozen enum, the switch statement that matches against it must include a catch-all case (usually `default` or an "ignore" `_` pattern). + +```swift +switch excuse { +case .eatenByPet: + // … +case .thoughtItWasDueNextWeek: + // … +} +``` + +Failure to do so will produce a warning in Swift 5. A program will trap at run time if an unknown enum case is actually encountered. + +All other uses of enums (`if case`, creation, accessing members, etc) do not change. Only the exhaustiveness checking of switches is affected by the frozen/non-frozen distinction. Non-exhaustive switches over frozen enums (and boolean values) will continue to be invalid in all language modes. + +Here's a more complicated example: + +```swift +switch (excuse, notifiedTeacherBeforeDeadline) { +case (.eatenByPet, true): + // … +case (.thoughtItWasDueNextWeek, true): + // … +case (_, false): + // … +} +``` + +This switch handles all *known* patterns, but still doesn't account for the possibility of a new enum case when the second tuple element is `true`. This should result in a warning in Swift 5, like the first example. + + +### `@unknown` + +The downside of using a `default` case is that the compiler can no longer alert a developer that a particular enum has elements that aren't explicitly handled in the `switch`. To remedy this, switch cases will gain a new attribute, `@unknown`. + +```swift +switch excuse { +case .eatenByPet: + // … +case .thoughtItWasDueNextWeek: + // … +@unknown default: + // … +} +``` + +Like the regular `default`, `@unknown default` matches any value; it is a "catch-all" case. However, the compiler will produce a *warning* if all known elements of the enum have not already been matched. This is a warning rather than an error so that adding new elements to the enum remains a source-compatible change. (This is also why `@unknown default` matches any value rather than just those not seen at compile-time.) + +`@unknown` may only be applied to `default` or a case consisting of the single pattern `_`. Even in the latter case, `@unknown` must be used with the last case in a `switch`. This restriction is discussed further in the "`unknown` patterns" section under "Future directions". + +The compiler will warn if all enums in the pattern being matched by `@unknown` are explicitly annotated as frozen, or if there are no enums in the pattern at all. This is a warning rather than an error so that annotating an enum as frozen remains a source-compatible change. If the pattern contains any enums that are implicitly frozen (i.e. because it is a user-defined Swift enum), `@unknown` is permitted, in order to make it easier to adapt to newly-added cases. + +`@unknown` has a downside that it is not testable, since there is no way to create an enum value that does not match any known cases, and there wouldn't be a safe way to use it if there was one. However, combining `@unknown` with other cases using `fallthrough` can get the effect of following another case's behavior while still getting compiler warnings for new cases. + +```swift +switch excuse { +case .eatenByPet: + showCutePicturesOfPet() + +case .thoughtItWasDueNextWeek: + fallthrough +@unknown default: + askForDueDateExtension() +} +``` + + +### C enums + +Enums imported from C are tricky, because it's difficult to tell whether they're part of the current project or not. An `NS_ENUM` in Apple's SDK should probably be treated as non-frozen, but one in your own framework might be frozen. Even there, though, it's possible that there's a "private case" defined in a .m file: + +```objc +// MyAppPaperSupport.h +typedef NS_ENUM(NSInteger, PaperSize) { + PaperSizeUSLetter = 0, + PaperSizeA4 = 1, + PaperSizePhoto4x6 = 2 +}; +``` +```objc +// MyAppPaperSupport.m +static const PaperSize PaperSizeStickyNote = 255; +``` + +(While this pattern may be unfamiliar, it is used in Apple's SDKs, though not often.) + +Therefore, enums imported from C will be treated conservatively: an otherwise-unannotated `NS_ENUM` will be imported as non-frozen and treated as such in all contexts. The newly-added C attribute `enum_extensibility` can be used to override this behavior: + +```objc +typedef NS_ENUM(NSInteger, GregorianMonth) { + GregorianMonthJanuary = 1, + GregorianMonthFebruary, + GregorianMonthMarch, + GregorianMonthApril, + GregorianMonthMay, + GregorianMonthJune, + GregorianMonthJuly, + GregorianMonthAugust, + GregorianMonthSeptember, + GregorianMonthOctober, + GregorianMonthNovember, + GregorianMonthDecember, +} __attribute__((enum_extensibility(closed))); +``` + +Apple doesn't speak about future plans for its SDKs, so having an alternate form of `NS_ENUM` that includes this attribute is out of scope for this proposal. + +Apart from the effect on switches, a frozen C enum's `init(rawValue:)` will also enforce that the case is one of those known at compile time. Imported non-frozen enums will continue to perform no checking on the raw value. + +> This section only applies to enums that Swift considers "true enums", rather than option sets or funny integer values. In the past, the only way to get this behavior was to use the `NS_ENUM` or `CF_ENUM` macros, but the presence of `enum_extensibility(closed)` *or* `enum_extensibility(open)` will instruct Swift to treat the enum as a "true enum". Similarly, the newly-added `flag_enum` C attribute can be used to signify an option set like `NS_OPTIONS`. + + +### Effect on the standard library and overlays + +The majority of enums defined in the standard library do not need the flexibility afforded by being non-frozen, and so will be marked as frozen. This includes the following enums: + +- ❄️ ClosedRange.Index +- ❄️ FloatingPointSign +- ❄️ FloatingPointClassification +- ❄️ Never +- ❄️ Optional +- ❄️ UnicodeDecodingResult +- ❄️ Unicode.ParseResult + +The following public enums in the standard library will *not* be marked as frozen: + +- DecodingError +- EncodingError +- FloatingPointRoundingRule +- Mirror.AncestorRepresentation +- Mirror.DisplayStyle +- PlaygroundQuickLook (deprecated anyway) + +And while the overlays are not strictly part of the Swift Open Source project (since they are owned by framework teams at Apple), the tentative plan would be to mark these two enums as frozen: + +- ❄️ ARCamera.TrackingState (a tri-state of "on", "off", and "limited(Reason)") +- ❄️ DispatchTimeoutResult ("success" and "timed out") + +And the other public enums in the overlays would be non-frozen: + +- ARCamera.TrackingState.Reason +- Calendar.Component +- Calendar.Identifier +- Calendar.MatchingPolicy +- Calendar.RepeatedTimePolicy +- Calendar.SearchDirection +- CGPathFillRule +- Data.Deallocator +- DispatchData.Deallocator +- DispatchIO.StreamType +- DispatchPredicate +- DispatchQoS.QoSClass +- DispatchQueue.AutoreleaseFrequency +- DispatchQueue.GlobalQueuePriority (deprecated anyway) +- DispatchTimeInterval +- JSONDecoder.DataDecodingStrategy +- JSONDecoder.DateDecodingStrategy +- JSONDecoder.KeyDecodingStrategy +- JSONDecoder.NonConformingFloatDecodingStrategy +- JSONEncoder.DataEncodingStrategy +- JSONEncoder.DateEncodingStrategy +- JSONEncoder.KeyEncodingStrategy +- JSONEncoder.NonConformingFloatEncodingStrategy +- MachErrorCode +- POSIXErrorCode + + +## Comparison with other languages + +"Enums", "unions", "variant types", "sum types", or "algebraic data types" are present in a number of other modern languages, most of which don't seem to treat this as an important problem. + + +### Languages without non-frozen enums + +**Haskell** and **OCaml** make heavy use of enums ("algebraic data types", or just "types") without any feature like this; adding a new "case" is always a source-breaking change. (Neither of these languages seems to care much about binary compatibility.) This is definitely a sign that you can have a successful language without a form of non-frozen enum other than "protocols". **Kotlin** also falls in this bucket, although it uses enums ("enum classes") less frequently. + +The **C#** docs have a nice section on [how the language isn't very helpful][c-sharp] for distinguishing frozen and non-frozen enums. **Objective-C**, of course, is in the same bucket, though Apple could start doing things with the `enum_extensibility` Clang attribute that was recently added. + + [c-sharp]: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/enum#robust-programming + + +### Languages with alternate designs + +**F#** enums ("unions") [either expose all of their "cases" or none of them][f-sharp]. The Swift equivalent of this would be not allowing you to switch on such an enum at all, as if it were a struct with private fields. + +Enums in **D** are like enums in C, but D distinguishes `switch` from `final switch`, and only the latter is exhaustive. That is, it's a client-side decision at the use site, rather than a decision by the definer of the enum. + +**Scala** has enums, but the pattern most people seem to use is "sealed traits", which in Swift terms would be "protocols where all conforming types are known, usually singletons". A non-frozen enum would then just be a normal protocol. Some downsides of applying this to Swift are discussed below under "Use protocols instead". + + [f-sharp]: https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/signatures + + +### Languages with designs similar to this proposal + +**Rust** has an [accepted proposal][rust] to add non-frozen enums that looks a lot like this one, but where "frozen" is still the default to not break existing Rust programs. (There are some interesting differences that come up in Rust but not Swift; in particular they need a notion of non-frozen structs because their structs can be decomposed in pattern-matching as well.) + + [rust]: https://github.com/rust-lang/rfcs/blob/master/text/2008-non-exhaustive.md + + + +## Source compatibility + +It is now a source-compatible change to add a case to a non-frozen enum (whether imported from C or defined in the standard library). + +It is not a source-compatible change to add a case to a frozen enum. + +It is still not a source-compatible change to remove a case from a public enum (frozen or non-frozen). + +It is a source-compatible change to change a non-frozen enum into a frozen enum, but not vice versa. + + +### Breaking the contract + +If a library author adds a case to a frozen enum, any existing switches will likely not handle this new case. The compiler will produce an error for any such switch (i.e. those without a `default` case or `_` pattern to match the enum value), noting that the case is unhandled; this is the same error that is produced for a non-exhaustive switch in Swift 4. + +If a library author changes an enum previously marked frozen to make it non-frozen, the compiler will produce a warning for any switch that does not have a catch-all case. + + +## Effect on ABI stability + +The layout of a non-frozen Swift enum must not be exposed to clients, since the library may choose to add a new case that does not fit in that layout in its next release. This results in extra indirection when that enum appears in public API. The layout of a frozen enum will continue to be made available to clients for optimization purposes. + +This change does not affect the layout of `@objc` enums, whether imported from C or defined in Swift. (Note that the representation of a non-`@objc` enum's case may differ from its raw value; this improves the efficiency of `switch` statements when all cases are known at compile time.) + + +## Effect on Library Evolution + +It is now a binary-compatible change to add a case to a non-frozen enum. + +It is still not a binary-compatible change to remove a case from a public enum (frozen or non-frozen). + +It is not a binary-compatible change to add `@objc` to an enum, nor to remove it. + +Taking an existing non-frozen enum and making it frozen is something we'd like to support without breaking binary compatibility, but there is no design for that yet. The reverse will not be allowed. + + +### Breaking the contract + +Because the compiler uses the set of cases in a frozen enum to determine its in-memory representation and calling convention, adding a new case or marking such an enum as non-frozen will result in "undefined behavior" from any client apps that have not been recompiled. This means a loss of memory-safety and type-safety on par with a misuse of "unsafe" types, which would most likely lead to crashes but could lead to code unexpectedly being executed or skipped. In short, things would be very bad. + +Some ideas for how to prevent library authors from breaking the rules accidentally are discussed in "Compatibility checking" under "Future directions". + +As a special case, switching over an unexpected value in an `@objc` enum (whether imported or defined in Swift) will always result in a trap rather than "undefined behavior", even if the enum is frozen. + + +## Future directions + +### Non-frozen Swift enums outside the standard library + +Earlier versions of this proposal included syntax that allowed *all* public Swift enums to have a frozen/non-frozen distinction, rather than just those in the standard library and overlays. This is still something we want to support, but the core team has made it clear that such a distinction is only worth it for libraries that have binary compatibility concerns (such as those installed into a standard location and used by multiple clients), at least without a more developed notion of versioning and version-locking. Exactly what it means to be a "library with binary compatibility concerns" is a large topic that deserves its own proposal. + + +### `unknown` patterns + +As described, `@unknown` cases can only be used to match the entire switched value; it does not work when trying to match a tuple element, or another enum's associated type. In theory, we could make a new *pattern* kind that allows matching unknown cases anywhere within a larger pattern: + +```swift +switch (excuse, notifiedTeacherBeforeDeadline) { +case (.eatenByPet, true): + // … +case (.thoughtItWasDueNextWeek, true): + // … +case (#unknown, true): + // … +case (_, false): + // … +} +``` + +(The `#unknown` spelling is chosen by analogy with `#selector` to not conflict with existing syntax; it is not intended to be a final proposal.) + +However, this produces potentially surprising results when followed by a case that could also match a particular input. Because `@unknown` is only supported on catch-all cases, the input `(.thoughtItWasDueNextWeek, true)` would result in case 2 being chosen rather than case 3. + +```swift +switch (excuse, notifiedTeacherBeforeDeadline) { +case (.eatenByPet, true): // 1 + // … +case (#unknown, true): // 2 + // … +case (.thoughtItWasDueNextWeek, _): // 3 + // … +case (_, false): // 4 + // … +} +``` + +The compiler would warn about this, at least, since there is a known value that can reach the `unknown` pattern. + +`@unknown` must appear only on the last case in a switch to avoid this issue. However, it's not possible to enforce the same thing for arbitrary patterns because there may be multiple enums in the pattern whose unknown cases need to be treated differently. + +A key point of this discussion is that as proposed `@unknown` merely produces a *warning* when the compiler can see that some enum cases are unhandled, rather than an error. If the compiler produced an error instead, it would make more sense to use a pattern-like syntax for `unknown` (see the naming discussions under "Alternatives considered"). However, if the compiler produced an error, then adding a new case would not be a source-compatible change. + +For these reasons, generalized `unknown` patterns are not being included in this proposal. + + +### Using `@unknown` with other catch-all cases + +At the moment, `@unknown` is only supported on cases that are written as `default:` or as `case _:`. However, there are other ways to form catch-all cases, such as `case let value:`, or `case (_, let b):` for a tuple input. Supporting `@unknown` with these cases was considered outside the scope of this proposal, which had already gone on for quite a while, but there are no known technical issues with lifting this restriction. + + +### Non-public cases + +The work required for non-frozen enums also allows for the existence of non-public cases in a public enum. This already shows up in practice in Apple's SDKs, as described briefly in the section on "C enums" above. Like "enum inheritance", this kind of behavior can mostly be emulated by using a second enum inside the library, but that's not sufficient if the non-public values need to be vended opaquely to clients. + +Were such a proposal to be written, I advise that a frozen enum not be permitted to have non-public cases. An enum in a user-defined library would then be implicitly considered frozen if and only if it had no non-public cases. + + +### Compatibility checking + +Of course, the compiler can't stop a library author from adding a new case to a frozen enum, even though that will break source and binary compatibility. We already have two ideas on how we could catch mistakes of this nature: + +- A checker that can compare APIs across library versions, using swiftmodule files or similar. + +- Encoding the layout of a type in a symbol name. Clients could link against this symbol so that they'd fail to launch if it changes, but even without that an automated system could check the list of exported symbols to make sure nothing was removed. + +Frozen enums remain useful even without any automated checking, and such checking should account for more than just enums, so it's not being included in this proposal. + + +### Efficient representation of enums with raw types + +For enums with raw types, a 32-bit integer can be used as the representation rather than a fully opaque value, on the grounds that 4 billion is a reasonable upper limit for the number of distinct cases in an enum without payloads. However, this would make it an ABI-breaking change to add or remove a raw type from an enum, and would make the following definitions not equivalent: + +```swift +/* non-frozen */ public enum HTTPMethod: String { + case get = "GET" + case put = "PUT" + case post = "POST" + case delete = "DELETE" +} +``` + +```swift +/* non-frozen */ public enum HTTPMethod: RawRepresentable { + case get + case put + case post + case delete + + public init?(rawValue: String) { + switch rawValue { + case "GET": return .get + case "PUT": return .put + case "POST": return .post + case "DELETE": return .delete + default: return nil + } + } + + public var rawValue: String { + switch self { + case .get: return "GET" + case .put: return "PUT" + case .post: return "POST" + case .delete: return "DELETE" + } + } +} +``` + +As such, this representation change is out of scope for this proposal. + + +## Alternatives considered + +### Terminology and syntax + +#### Terminology: "closed" and "open" + +The original description of the problem used "closed" and "open" to describe frozen and non-frozen enums, respectively. However, this conflicts with the use of `open` in classes and their members. In this usage, `open` is clearly a greater level of access than `public`, in that clients of an `open` class can do everything they can with a `public` class and more; it is source-compatible to turn a `public` class into an `open` one. For enums, however, it is frozen enums that are "greater": you can do everything you can with a non-frozen enum and more, and it would be source-compatible for a standard library contributor to turn a non-frozen enum into a frozen one (at the cost of a warning). + + +#### Terminology: other options + +Several more options were suggested during initial discussions: + +- complete / incomplete +- covered +- exhaustive / non-exhaustive +- non-extensible +- final / non-final +- finite / non-finite (not "infinite") +- fixed +- locked +- sealed / non-sealed +- total / partial + +I didn't have a strong preference for any particular choice as long as it *isn't* "closed" / "open", for the reasons described above. In the first revision of this proposal I picked "exhaustive" because it matches the name proposed [in Rust][rust]. (Unfortunately, Clang's `enum_extensibility` attribute, recently added by us at Apple, uses `open` and `closed`.) + +Note that "nonextensible" does have one problem: Apple already uses [`NS_TYPED_EXTENSIBLE_ENUM `][NS_TYPED_EXTENSIBLE_ENUM] to refer to enum-like sets of constants (usually strings) that *clients* can add "cases" to. That's not the same meaning as the exhaustiveness discussed in this proposal. + +During the first review for this proposal, Becca Royal-Gordon suggested "frozen", which was met with general approval or at least no major objections. + + [NS_TYPED_EXTENSIBLE_ENUM]: https://developer.apple.com/library/content/documentation/Swift/Conceptual/BuildingCocoaApps/InteractingWithCAPIs.html#//apple_ref/doc/uid/TP40014216-CH8-ID206 + + +#### `unknown` naming + +The first version of this proposal did not include `unknown`, but did discuss it as a "considered alternative" under the name `future`. Previous discussions have also used `unexpected` or `undeclared` to describe this feature as well. + +It was pointed out that neither `future` nor `unexpected` really described the feature being provided. `unknown` does not just handle cases added in the future; it also handles private cases and invalid values for C enums. Nor are such cases entirely unexpected, since the compiler is telling the developer to expect them. `undeclared` has fewer issues, but certainly private cases can be declared *somewhere;* the declarations just aren't visible. + +The "intermediate" revision of this proposal where `unknown` was first added used the spelling `unknown case`, but restricted the new case to only match values that *were* enums rather than values *containing* enums. When that restriction was loosened, the reading of `unknown case` as "(enum) cases that I don't know about" no longer made as much sense. + +During discussion, the name `unknown default` (or `@unknown default`) was suggested as an alternative to `unknown case`, since the semantics behave very much like `default`. However, it isn't the "default" that's "unknown". Other proposed spellings included `default unknown` (a simple attempt to avoid reading "unknown" as an adjective modifying "default") and `default(unknown)` (by analogy with `private(set)`). Nevertheless, this attribute syntax won out in the end by not tying it to `default`; the alternate spelling `@unknown case _` is also accepted. + +Moving away from "unknown", `@unused default` was also suggested, but the case is *not* unused. A more accurate `@runtimeReachableOnly` (or even `@runtimeOnly`) was proposed instead, but that's starting to get overly verbose for something that will appear reasonably often. + +For standalone names, `fallback` was also suggested, but semantically that seems a little too close to "default", and in terms of actual keywords it was pointed out that this was very similar to `fallthrough` despite having no relation. `invisible` was suggested as well (though in the context of patterns rather than cases), but that doesn't exactly apply to future cases. + +To summarize, the following spellings were considered for `unknown`: + +- `future:` +- `unexpected:` +- `undeclared:` +- `unknown case:` +- `unknown default:` +- `@unknown default:` +- `@unused default:` +- `@runtimeReachableOnly default:` +- `default unknown:` +- `default(unknown):` +- `fallback:` +- `invisible:` + +For the review of the proposal, I picked `unknown:` as the best option, admittedly as much for *not* having *unwanted* connotations as for having *good* connotations. The core team ultimately went with `@unknown default:` / `@unknown case _:` instead. + +A bigger change would be to make a custom *pattern* instead of a custom *case,* even if it were subject to the same restrictions in implementation (see "`unknown` patterns" above). This usually meant using a symbol of some kind to distinguish the "unknown" from a normal label or pattern, leading to `case #unknown` or similar. This makes the new feature less special, since it's "just another pattern". However, it would be surprising to have such a pattern but keep the restrictions described in this proposal; thus, it would only make sense to do this if we were going to implement fully general pattern-matching for this feature. See "`unknown` patterns" above for more discussion. + +Finally, there was the option to put an annotation on a `switch` instead of customizing the catch-all case, e.g. `@warnUnknownCases switch x {`. This is implementable but feels easier for a developer to forget to write, and the compiler can only help if the developer actually *has* implemented all of the current cases alongside their `default` case. + + +### `switch!` + +`switch!` was an alternative to `@unknown` that would not support any action other than trapping when the enum is not one of the known cases. This avoids some of the problems with `@unknown` (such as making it much less important to test), but isn't exactly in the spirit of non-frozen enums, where you *know* there will be more cases in the future. + +The following two examples would be equivalent (except perhaps in the form of the diagnostic produced). + +```swift +switch! excuse { +case .eatenByPet: + // … +case .thoughtItWasDueNextWeek: + // … +} +``` + +```swift +switch excuse { +case .eatenByPet: + // … +case .thoughtItWasDueNextWeek: + // … +unknown: + fatalError("unknown case in switch: \(excuse)") +} +``` + + +### Testing invalid cases + +Another issue with non-frozen enums is that clients cannot properly test what happens when a new case is introduced, almost by definition. Becca Royal-Gordon came up with the idea to have a new type annotation that would allow the creation of an invalid enum value. Since this is only something to use for testing, the initial version of the idea used `@testable` as the spelling for the annotation. The tests could then use a special expression, `#invalid`, to pass this invalid value to a function with a `@testable` enum parameter. + +However, this would only work in cases where the action to be taken does not actually depend on the enum value. If it needs to be passed to the original library that owns the enum, or used with an API that is not does not have this annotation, the code still cannot be tested properly. + +```swift +override func process(_ transaction: @testable Transaction) { + switch transaction { + case .deposit(let amount): + // … + case .withdrawal(let amount): + // … + default: + super.process(transaction) // hmm… + } +} +``` + +This is an additive feature, so we can come back and consider it in more detail even if we leave it out of the language for now. Meanwhile, the effect can be imitated using an Optional or ImplicitlyUnwrappedOptional parameter. + + +### Allow enums defined in source packages to be considered non-frozen + +The first version of this proposal applied the frozen/non-frozen distinction to all public enums, even those in user-defined libraries. The motivation for this was to allow package authors to add cases to their enums without it being a source-breaking change, meaning it can be done in a minor version release of a library (i.e. one intended to be backwards-compatible). Like deprecations, this can produce new warnings, but not new errors, and it should not (if done carefully) break existing code. + +The core team decided that this feature was not worth the disruption and long-term inconvenience it would cause for users who did not care about this capability. + + +### Leave out `@unknown` + +The [initial version][] of this proposal did not include `@unknown`, and required people to use a normal `default` to handle cases added in the future instead. However, many people were unhappy with the loss of exhaustivity checking for `switch` statements, both for enums in libraries distributed as source and enums imported from Apple's SDKs. While this is an additive feature that does not affect ABI, it seems to be one that the community considers a necessary part of a language model that provides non-frozen enums. + + [initial version]: https://github.com/swiftlang/swift-evolution/blob/a773d07ff4beab8b7855adf0ac56d1e13bb7b44c/proposals/0192-non-exhaustive-enums.md + + +### Mixing `@unknown` with other catch-all cases + +The proposal as written forbids having two catch-all cases in the same `switch` where only one is marked `@unknown`. Most people would expect this to have the following behavior: + +```swift +switch excuse { +case .eatenByPet: + // Specific known case +@unknown case _: + // Any cases not recognized by the compiler +case _: + // Any other cases the compiler *does* know about, + // such as .thoughtItWasDueNextWeek +} +``` + +However, I can't think of an actual use case for this; it's not clear what action one would take in the `@unknown` case that they wouldn't take in the later default case. Furthermore, this becomes a situation where the same code behaves differently before and after recompilation: + +1. A new case is added to the HomeworkExcuse enum, say, `droppedInMud`. +2. When using the new version of the library with an existing built client app, the `droppedInMud` case will end up in the `@unknown` part of the `switch`. +3. When the client app *is* recompiled, the `droppedInMud` case will end up in the `case _` case. The compiler will not (and cannot) provide any indication that the behavior has changed. + +Without a resolution to these concerns, this feature does not seem worth including in the proposal. It's also additive and has no ABI impact, so if we do find use cases for it in the future we can always add it then. + + +### Introduce a new declaration kind instead + +There have been a few suggestions to distinguish `enum` from some other kind of declaration that acts similarly but allows adding cases: + +```swift +choices HomeworkExcuse { + case eatenByPet + case thoughtItWasDueNextWeek +} +``` + +My biggest concern with this is that if we ever *do* expand this beyond the standard library and overlays, it increases the possibility of a library author accidentally publishing a (frozen) `enum` when they meant to publish a (non-frozen) `choices`. As described above, the opposite mistake is one that can be corrected without breaking source compatibility, but this one cannot. + +A smaller concern is that both `enum` and `choices` would behave the same when they *aren't* `public`. + +Stepping back, increasing the surface area of the language in this way does not seem desirable. Exhaustive switching has been a key part of how Swift enums work, but it is not their only feature. Given how people already struggle with the decision of "struct vs. class" when defining a new type, introducing another pair of "similar but different" declaration kinds would have to come with strong benefits. + +My conclusion is that it is better to think of frozen and non-frozen enums as two variants of the same declaration kind, rather than as two different declaration kinds. + + +### Use protocols instead + +Everything you can do with non-frozen enums, you can do with protocols as well, except for: + +- exhaustivity checking with `@unknown` +- forbidding others from adding their own "cases" + +```swift +protocol HomeworkExcuse {} +struct EatenByPet: HomeworkExcuse {} +struct ThoughtItWasDueNextWeek: HomeworkExcuse {} + +switch excuse { +case is EatenByPet: + // … +case is ThoughtItWasDueNextWeek: + // … +default: + // … +} +``` + +(Associated values are a little harder to get out of the cases, but let's assume we could come up for syntax as well.) + +This is a valid model; it's close to what Scala does (as mentioned above), and is independently useful in Swift. However, using this as the only way to get non-frozen enum semantics would lead to a world where `enum` is dangerous for library authors, because `public enum` is now a promise that no new cases will be added. Nothing else in Swift works that way. More practically, getting around this restriction would mean rewriting existing code to use the more verbose syntax of separate types conforming to a common protocol. + +(If Swift were younger, perhaps we would consider using protocols for *all* non-imported enums, not just non-frozen ones. But at this point that would be *way* too big a change to the language.) + + +### Import non-frozen C enums as RawRepresentable structs + +The Swift compiler already makes a distinction between plain C enums, enums marked with the `flag_enum` Clang attribute (`NS_OPTIONS`), and enums marked with the `enum_extensibility` Clang attribute (`NS_ENUM`). The first two categories were deemed to not be sufficiently similar to Swift enums and are imported instead as structs. Given that we're most immediately concerned about *C* enums growing new cases (specifically, those in Apple's existing Objective-C SDKs), we could sidestep the problem by importing *all* C enums as structs except for those marked `enum_extensibility(closed)`. However, this doesn't solve the problem for future Swift libraries, while still requiring changes to existing `switch` statements across many many projects, and it doesn't support the exhaustivity checking provided by `unknown`. Furthermore, it would probably be harder to implement high-quality migration support from Swift 4 to Swift 5, since the structs-formerly-enums will look like any other structs imported from C. + + +### Get Apple to stop adding new cases to C enums + +This isn't going to happen, but I thought I'd mention it since it was brought up during discussion. While some may consider this a distasteful use of the C language, it's an established pattern for Apple frameworks and is not going to change. + + +### "Can there be a kind of open enum where you can add new cases in extensions?" + +There is no push to allow adding new cases to an enum from *outside* a library. This use case (no pun intended) is more appropriate for a RawRepresentable struct, where the library defines some initial values as static properties. (You can already switch over struct values in Swift as long as they are Equatable.) diff --git a/proposals/0193-cross-module-inlining-and-specialization.md b/proposals/0193-cross-module-inlining-and-specialization.md new file mode 100644 index 0000000000..bbdcc7305a --- /dev/null +++ b/proposals/0193-cross-module-inlining-and-specialization.md @@ -0,0 +1,186 @@ +# Cross-module inlining and specialization + +* Proposal: [SE-0193](0193-cross-module-inlining-and-specialization.md) +* Author: [Slava Pestov](https://github.com/slavapestov) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 4.2)** +* Evolution review thread: [https://forums.swift.org/t/se-0193-cross-module-inlining-and-specialization/7310](https://forums.swift.org/t/se-0193-cross-module-inlining-and-specialization/7310) +* Implementation: [apple/swift#15787](https://github.com/apple/swift/pull/15787) + +## Introduction + +We propose introducing a pair of new attributes, `@inlinable` and `@usableFromInline`. The `@inlinable` attribute exports the body of a function as part of a module's interface, making it available to the optimizer when referenced from other modules. The `@usableFromInline` attribute marks an internal declaration as being part of the binary interface of a module, allowing it to be used from `@inlinable` code without exposing it as part of the module's source interface. + +## Motivation + +One of the top priorities of the Swift 5 release is a design and implementation of _the Swift ABI_. This effort consists of three major tasks: + +* Finalizing the low-level function calling convention, layout of data types, and various runtime data structures. The goal here is to maintain compatibility across compiler versions, ensuring that we can continue to make improvements to the Swift compiler without breaking binaries built with an older version of the compiler. + +* Implementing support for _library evolution_, or the ability to make certain source-compatible changes, without breaking binary compatibility. Examples of source-compatible changes we are considering include adding new stored properties to structs and classes, removing private stored properties from structs and classes, adding new public methods to a class, or adding new protocol requirements that have a default implementation. The goal here is to maintain compatibility across framework versions, ensuring that framework authors can evolve their API without breaking binaries built against an older version of the framework. For more information about the resilience model, see the +[library evolution +document](https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst) +in the Swift repository. + +* Stabilizing the API of the standard library. The goal here is to ensure that the standard library can be deployed separately from client binaries and frameworks, without forcing recompilation of existing code. + +All existing language features of Swift were designed with these goals in mind. In particular, the implementation of generic types and functions relies on runtime reified types to allow separate compilation and type checking of generic code. + +Within the scope of a single module, the Swift compiler performs very aggressive optimization, including full and partial specialization of generic functions, inlining, and various forms of interprocedural analysis. + +On the other hand, across module boundaries, runtime generics introduce unavoidable overhead, as reified type metadata must be passed between functions, and various indirect access patterns must be used to manipulate values of generic type. We believe that for most applications, this overhead is negligible compared to the actual work performed by the code itself. + +However, for some advanced use cases, and in particular for the standard library, the overhead of runtime generics can dominate any useful work performed by the library. Examples include the various algorithms defined in protocol extensions of `Sequence` and `Collection`, for instance the `map` method of the `Sequence` protocol. Here the algorithm is very simple and spends most of its time manipulating generic values and calling to a user-supplied closure; specialization and inlining can completely eliminate the algorithm of the higher-order function call and generate equivalent code to a hand-written loop manipulating concrete types. + +The library author can annotate such published APIs with the `@inlinable` attribute. This will make their bodies available to the optimizer when building client code in other modules that call those APIs. The optimizer may or may not make use of the function body; it might be inlined, specialized, or ignored, in which case the compiler will continue to reference the public entry point in the framework. If the framework were to change the definition of such a function, only binaries built against the newer version of library might continue using the old, inlined definition, they may use the new definition, or even a mix depending if certain call sites inlined the function or not. + +## Proposed solution + +The `@inlinable` attribute causes the body of a function to be emitted as part of the module interface. For example, a framework can define a rather impractical implementation of an algorithm which returns `true` if all elements of a sequence are equal or if the sequence is empty, and `false` otherwise: + +```swift +@inlinable public func allEqual(_ seq: T) -> Bool + where T : Sequence, T.Element : Equatable { + var iter = seq.makeIterator() + guard let first = iter.next() else { return true } + + func rec(_ iter: inout T.Iterator) -> Bool { + guard let next = iter.next() else { return true } + return next == first && rec(&iter) + } + + return rec(&iter) +} +``` + +A client binary built against this framework can call `allEqual()` and enjoy a possible performance improvement when built with optimizations enabled, due to the elimination of abstraction overhead. + +On the other hand, once the framework author comes to their senses and implements an iterative solution to replace the recursive algorithm defined above, the client binary might not be able to make use of the more efficient implementation until recompiled. + +## Detailed design + +### The `@inlinable` attribute + +The `@inlinable` attribute can be applied to the following kinds of declarations: + +* Functions and methods +* Subscripts +* Computed properties +* Initializers +* Deinitializers + +The attribute can only be applied to declarations with `public` or `internal` visibility. + +The attribute cannot be applied to local declarations, that is, declarations nested inside functions or statements. However, local functions and closure expressions defined inside public `@inlinable` functions are always implicitly `@inlinable`. + +When applied to a subscript or computed property, the attribute applies to both the getter and setter. + +Note that only delegating initializers (those that assign to `self` or call another initializer via `self.init`) can be inlinable. Root initializers which initialize the stored properties of a struct or class directly cannot be inlinable. For motivation, see [SE-0189 Restrict Cross-module Struct Initializers](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0189-restrict-cross-module-struct-initializers.md). + +### Inlinable contexts + +The body of an inlinable declaration is an example of an _inlinable context_. The compiler enforces certain restrictions within inlinable contexts: + +* **Inlinable declarations cannot define local types.** This is because all types have a unique identity in the Swift runtime, visible to the language in the form of the `==` operator on metatype values. It is not clear what it would mean if two different libraries inline the same local type from a third library, with all three libraries linked together into the same binary. This becomes even worse if two _different_ versions of the same inlinable function appear inside the same binary. + +* **Inlinable declarations can only reference ABI-public declarations.** This is because they can be emitted into the client binary, and are therefore limited to referencing symbols that the client binary can reference. + +**Note:** Future evolution proposals may add new kinds of inlinable contexts. + +### The `@usableFromInline` attribute + +This attribute allows us to introduce a notion of an _ABI-public_ declaration. A declaration is _ABI-public_ if both of the following conditions hold: + +- The declaration is a top-level declaration, or it is nested inside an ABI-public type. +- The declaration is `public`, or is `internal` and annotated with either the `@usableFromInline` attribute or `@inlinable` attribute. + +In the following example, the method `C.f` is ABI-public: + +```swift +public class C { + public func f() {} +} +``` + +Two more examples of ABI-public declarations are the methods `C.D.f` and `C.D.g` below: + +```swift +public class C { + @usableFromInline internal class D { + @usableFromInline internal func f() {} + + @inlinable internal func g() {} + } +} +``` + +In the following, the method `C.f` is **not** ABI-public, because it is nested inside a type that is not `@usableFromInline` or `public`: + +```swift +internal class C { + public func f() {} +} +``` + +The `@usableFromInline` attribute can be applied to all declarations which support access control modifiers. This includes almost all kinds of declarations, except for the following, which always have the same effective visibility as their containing declaration: + +* Protocol requirements +* Enum cases +* Class destructors + +When applied to a subscript or computed property, the attribute applies to both the getter and setter, if present. + +The `@usableFromInline` attribute can only be applied to `internal` declarations. It does not make sense on `public` declarations, which are already ABI-public. It also cannot be applied to `private` and `fileprivate` declarations. and not `private`, `fileprivate` or `public` declarations. The `@usableFromInline` attribute does not affect source-level visibility of a declaration; it only results in the entry point being exported at the ABI level, allowing it to be referenced from `@inlinable` functions. + +**Note:** On an internal declaration, `@inlinable` implies `@usableFromInline`. The compiler will emit a warning if a declaration has both attributes. + +### Future directions + +We would also like to add the ability to specify versioning information. This capability is not part of this proposal, but will be explored in the future, possibly using syntax like `@inlinable(2.0)` or `@available(inlinable, 2.0)`. + +This is needed when a function introduced in the original release of a framework becomes inlinable in a later release of the framework. The function body might use ABI-public functions that are only part of the later release, and therefore the function is only available for inlining if the client is deploying against the newer version of the framework. + +This versioning capability will also be required for non-exhaustive enums and fixed-contents structs, since enums can become exhaustive, and structs can become fixed-contents, after the fact, and the compiler can only make use of this information of deploying against a sufficiently-recent version of the framework. + +## Source compatibility + +The introduction of the `@inlinable` and `@usableFromInline` attributes is an additive change to the language and has no impact on source compatibility. + +## Effect on ABI stability + +The following changes are ABI compatible: + +- Adding `@inlinable` to a public or internal declaration +- Removing `@inlinable` from a public declaration +- Replacing `@inlinable` with `@usableFromInline` on an internal declaration +- Adding `@usableFromInline` to an existing declaration + +## Effect on API resilience + +Any changes to the body of an `@inlinable` declaration should be considered very carefully. As a general guideline, we feel that `@inlinable` makes the most sense with "obviously correct" algorithms which manipulate other data types abstractly through protocols, so that any future changes to an `@inlinable` declaration are optimizations that do not change observed behavior. + +An `@inlinable` function implementation must be prepared to interact with multiple versions of the same function linked into a single binary. For example, if a hashing function is `@inlinable`, the hash algorithm must not be changed to avoid introducing inconsistency. + +## Comparison with other languages + +The closest language feature to the `@inlinable` attribute is found in C and C++. In C and C++, the concept of a header file is similar to Swift's binary `swiftmodule` files, except they are written by hand and not generated by the compiler. Swift's `public` declarations are roughly analogous to declarations whose prototypes appear in a header file. + +Header files mostly contain declarations without bodies, but can also declare `inline` functions with bodies. Such functions are not part of the binary interface of the library, and are instead emitted into client code when referenced. As with `@inlinable` declarations, `inline` functions can only reference other "public" declarations, that is, those that are defined in other header files. Note that while `static inline` is the most commonly-used incarnation of this feature in C, our proposed `@inlinable` attribute is most similar to `extern inline`, were that easier to use. + +The closest analogue in C to `@usableFromInline` is a non-`static` function that is not declared in a framework's header file. External clients cannot see it directly, but they can call it if they provide a local `extern` declaration. + +## Alternatives considered + +One possible alternative would be to add a new compiler mode where _all_ declarations become implicitly `@inlinable`, and _all_ private and internal declarations become `@usableFromInline`. + +However, such a compilation mode would not solve the problem of delivering a stable ABI and standard library which can be deployed separately from user code. We _don't want_ all declaration bodies in the standard library to be available to the optimizer when building user code. + +While such a feature might be useful for users who build private frameworks that are always shipped together their application without resilience concerns, we do not feel it aligns with our goals for ABI stability, and at best it should be a separate discussion. + +For similar reasons, we do not feel that an "opt-out" attribute that can be applied to declarations to mark them non-inlinable makes sense. + +We have also considered generalizing `@inlinable` to allow it to be applied to entire blocks of declarations, for example at the level of an extension. As we gain more experience with using this attribute in the standard library we might decide this would be a useful addition, but we feel that for now, it makes sense to focus on the case of a single inlinable declaration instead. Any future generalizations can be introduced as additive language features. + +We originally used the spelling `@inlineable` for the attribute. However, we settled on `@inlinable` for consistency with the `Decodable` and `Encodable` protocols, which are named as they are and not `Decodeable` and `Encodeable`. + +Finally, we have considered some alternate spellings for this attribute. The name `@inlinable` is somewhat of a misnomer, because nothing about it actually forces the compiler to inline the declaration; it might simply generate a concrete specialization of it, or look at the body as part of an interprocedural analysis, or completely ignore the body. However, nothing seemed to read as well as `@inlinable`. diff --git a/proposals/0194-derived-collection-of-enum-cases.md b/proposals/0194-derived-collection-of-enum-cases.md new file mode 100644 index 0000000000..ee60a8d2fb --- /dev/null +++ b/proposals/0194-derived-collection-of-enum-cases.md @@ -0,0 +1,281 @@ +# Derived Collection of Enum Cases + +* Proposal: [SE-0194](0194-derived-collection-of-enum-cases.md) +* Authors: [Jacob Bandes-Storch](https://github.com/jtbandes), [Becca Royal-Gordon](https://github.com/beccadax), [Robert Widmann](https://github.com/CodaFi) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Review Thread: [SE-0194 review][se8] +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift#13655](https://github.com/apple/swift/pull/13655) +* Bugs: [SR-7151](https://bugs.swift.org/browse/SR-7151), [SR-7152](https://bugs.swift.org/browse/SR-7152) + +## Introduction + +> *It is a truth universally acknowledged, that a programmer in possession of an `enum` with many cases, must eventually be in want of dynamic enumeration over them.* + +[Enumeration types][enums] without associated values (henceforth referred to as "*simple enums*") have a finite, fixed number of cases, yet working with them programmatically is challenging. It would be natural to enumerate all the cases, count them, determine the highest `rawValue`, or produce a Collection of them. However, despite the fact that both the Swift compiler and the Swift runtime are aware of this information, there is no safe and sanctioned way for users to retrieve it. Users must resort to various [workarounds](#workarounds) in order to iterate over all cases of a simple enum. + +This topic was brought up [three][se1] [different][se2] [times][se3] in just the first two months of swift-evolution's existence. It was the [very first language feature request][SR-30] on the Swift bug tracker. It's a [frequent][so1] [question][so2] on Stack Overflow (between them, these two questions have over 400 upvotes and 60 answers). It's a [popular][nateblog] [topic][ericablog] on blogs. It is one of just eight [examples][sourcery] shipped with Sourcery. + +We propose the introduction of a protocol, `CaseIterable`, to indicate that a type has a finite, enumerable set of values. Moreover, we propose an opt-in derived implementation of `CaseIterable` for the common case of a simple enum. + +### Prior discussion on Swift-Evolution +- [List of all Enum values (for simple enums)][se1] (December 8, 2015) +- [Proposal: Enum 'count' functionality][se2] (December 21, 2015) +- [Draft Proposal: count property for enum types][se3] (January 17, 2016) +- † [Pre-proposal: CaseEnumerable protocol (derived collection of enum cases)][se4] (January 17, 2016) +- † [ValueEnumerable protocol with derived implementation for enums][se5] (April 15, 2016) + +…more than a year passes… + +- [[Pitch] Synthesized static enum property to iterate over cases][se6] (September 8, 2017) +- † [Re-pitch: Deriving collections of enum cases][se7] (November 6, 2017) +- **Official Review**: [SE-0194: Derived Collection of Enum Cases][se8] (January 6, 2018) + +_**†** = a precursor to this proposal_ + +[se1]: https://forums.swift.org/t/list-of-all-enum-values-for-simple-enums/337 +[se2]: https://forums.swift.org/t/proposal-enum-count-functionality/711 +[se3]: https://forums.swift.org/t/draft-proposal-count-property-for-enum-types/1103 +[se4]: https://forums.swift.org/t/pre-proposal-caseenumerable-protocol-derived-collection-of-enum-cases/1109 +[se5]: https://forums.swift.org/t/valueenumerable-protocol-with-derived-implementation-for-enums/2226 +[se6]: https://forums.swift.org/t/pitch-synthesized-static-enum-property-to-iterate-over-cases/6623 +[se7]: https://forums.swift.org/t/re-pitch-deriving-collections-of-enum-cases/6956 +[se8]: https://forums.swift.org/t/review-se-0194-derived-collection-of-enum-cases/7377 + +[so1]: http://stackoverflow.com/questions/24007461/how-to-enumerate-an-enum-with-string-type +[so2]: http://stackoverflow.com/questions/27094878/how-do-i-get-the-count-of-a-swift-enum + +[enums]: https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/Enumerations.html +[SR-30]: https://bugs.swift.org/browse/SR-30 +[nateblog]: http://natecook.com/blog/2014/10/loopy-random-enum-ideas/ +[ericablog]: http://ericasadun.com/2015/07/12/swift-enumerations-or-how-to-annoy-tom/ +[sourcery]: https://cdn.rawgit.com/krzysztofzablocki/Sourcery/master/docs/enum-cases.html + + +## Motivation + +### Use cases + +Examples online typically pose the question "How do I get all the enum cases?", or even "How do I get the count of enum cases?", without fully explaining what the code will do with that information. To guide our design, we focus on two categories of use cases: + +1. The code must greedily iterate over all possible cases, carrying out some action for each case. For example, imagine enumerating all combinations of suit and rank to build a deck of playing cards: + + ```swift + let deck = Suit.magicListOfAllCases.flatMap { suit in + (1...13).map { rank in + PlayingCard(suit: suit, rank: rank) + } + } + ``` + +2. The code must access information about all possible cases on demand. For example, imagine displaying all the cases through a lazy rendering mechanism like `UITableViewDataSource`: + + ```swift + class SuitTableViewDataSource: NSObject, UITableViewDataSource { + func tableView(_: UITableView, numberOfRowsInSection section: Int) -> Int { + return Suit.magicListOfAllCases.count + } + + func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell { + let cell = tableView.dequeueReusableCell(withIdentifier: "Cell", for: indexPath) + let suit = Suit.magicListOfAllCases[indexPath.row] + cell.titleView!.text = suit.localizedName + return cell + } + } + ``` + +To limit our scope, we are primarily interested in *simple* enums—those without any associated values—although we would also like to allow more complicated enums and structs to manually participate in this mechanism. + +The second use case suggests that access by contiguous, zero-based integer index is important for at least some uses. At minimum, it should be easy to construct an `Array` from the list of cases, but ideally the list could be used like an array directly, at least for simple enums. + + +### Workarounds + +The most basic approach to producing a collection of all cases is by manual construction: + +```swift +enum Attribute { + case date, name, author +} +protocol Entity { + func value(for attribute: Attribute) -> Value +} +// Cases must be listed explicitly: +[Attribute.date, .name, .author].map{ entity.value(for: $0) }.joined(separator: "\n") +``` + +For `RawRepresentable` enums, users have often relied on iterating over the known (or assumed) allowable raw values: + +*Excerpt from Nate Cook's post, [Loopy, Random Ideas for Extending "enum"][nateblog] (October 2014):* + +```swift +enum Reindeer: Int { + case Dasher, Dancer, Prancer, Vixen, Comet, Cupid, Donner, Blitzen, Rudolph +} +extension Reindeer { + static var allCases: [Reindeer] { + var cur = 0 + return Array( + GeneratorOf { + return Reindeer(rawValue: cur++) + } + ) + } + static var caseCount: Int { + var max: Int = 0 + while let _ = self(rawValue: ++max) {} + return max + } + static func randomCase() -> Reindeer { + // everybody do the Int/UInt32 shuffle! + let randomValue = Int(arc4random_uniform(UInt32(caseCount))) + return self(rawValue: randomValue)! + } +} +``` + +Or creating the enums by `unsafeBitCast` from their hashValue, which is assumed to expose their memory representation: + +*Excerpt from Erica Sadun's post, [Swift: Enumerations or how to annoy Tom][ericablog], with full implementation in [this gist](https://gist.github.com/erica/dd1f6616b4124c588cf7) (July 12, 2015):* + +```swift +static func fromHash(hashValue index: Int) -> Self { + let member = unsafeBitCast(UInt8(index), Self.self) + return member +} + +public init?(hashValue hash: Int) { + if hash >= Self.countMembers() {return nil} + self = Self.fromHash(hashValue: hash) +} +``` + +Or using a `switch` statement, making it a compilation error to forget to add a case, as with Dave Sweeris's [Enum Enhancer](https://github.com/TheOtherDave/EnumEnhancer) (which includes some extra functionality to avoid the boilerplate required for `.cases` and `.labels`): + +```swift +enum RawValueEnum : Int, EnumeratesCasesAndLabels { + static let enhancer:EnumEnhancer = EnhancedGenerator { + // `$0` is a RawValueEnum? + switch $0 { + case .none: $0 = .zero + case .some(let theCase): + switch theCase { + case .zero: $0 = .one + case .one: $0 = nil + } + } + } + case zero = 0 + case one = 1 +} +``` + +There are many problems with these existing techniques: + +- They are ad-hoc and can't benefit every enum type without duplicated code. +- They are not standardized across codebases, nor provided automatically by libraries such as Foundation and {App,UI}Kit. +- They are dangerous at worst, bug-prone in most cases (such as when enum cases are added, but the user forgets to update a hard-coded static collection), and awkward at best. + +### Resilience implications + +This last point is especially important as we begin to concern ourselves with library resilience. Future versions of Swift should allow library authors to add and deprecate public enum cases without breaking binary compatibility. But if clients are manually constructing arrays of "all cases", those arrays will not correspond to the version of the library they are running against. + +At the same time, the principle that libraries ought to control the promises they make should apply to any case-listing feature. Participation in a "list all cases" mechanism should be optional and opt-in. + +### Precedent in other languages + +- Rust does not seem to have a solution for this problem. + +- C#'s Enum has several [methods](https://msdn.microsoft.com/en-us/library/system.enum_methods.aspx) available for reflection, including `GetValues()` and `GetNames()`. + +- Java [implicitly declares](http://docs.oracle.com/javase/specs/jls/se8/html/jls-8.html#jls-8.9.3) a static `values()` function, returning an array of enum values, and `valueOf(String name)` which takes a String and returns the enum value with the corresponding name (or throws an exception). More examples [here](http://docs.oracle.com/javase/specs/jls/se8/html/jls-8.html#jls-8.9.3). + +- The Template Haskell extension to Haskell provides a function `reify` which extracts [info about types](http://hackage.haskell.org/package/template-haskell-2.10.0.0/docs/Language-Haskell-TH-Syntax.html#t:Info), including their constructors. + +## Proposed solution + +We propose introducing a `CaseIterable` protocol to the Swift Standard Library. The compiler will derive an implementation automatically for simple enums when the conformance is specified. + +```swift +enum Ma: CaseIterable { case 马, 吗, 妈, 码, 骂, 麻, 🐎, 🐴 } + +Ma.allCases // returns some Collection whose Iterator.Element is Ma +Ma.allCases.count // returns 8 +Array(Ma.allCases) // returns [Ma.马, .吗, .妈, .码, .骂, .麻, .🐎, .🐴] +``` + +## Detailed design + +- The `CaseIterable` protocol will have the following declaration: + + ```swift + public protocol CaseIterable { + associatedtype AllCases: Collection where AllCases.Element == Self + static var allCases: AllCases { get } + } + ``` + +- The compiler will synthesize an implementation of CaseIterable for an enum type if and only if: + + - the enum contains only cases without associated values; + - the enum declaration has an explicit `CaseIterable` conformance (and does not fulfill the protocol's requirements). + +- Enums imported from C/Obj-C headers will **not** participate in the derived CaseIterable conformance. + +- Cases marked `unavailable` will **not** be included in `allCases`. + +- The implementation will **not** be synthesized if the conformance is on an `extension` — it must be on the original `enum` declaration. + + +## Source compatibility + +This proposal only adds functionality, so existing code will not be affected. (The identifier `CaseIterable` doesn't make very many appearances in Google and GitHub searches.) + + +## Effect on ABI stability + +The proposed implementation adds a derived conformance that makes use of no special ABI or runtime features. + +## Effect on API resilience + +User programs will come to rely on the `CaseIterable` protocol and its `allCases` and `AllCases` requirements. Due to the use of an associated type for the property's type, the derived implementation is free to change the concrete Collection it returns without breaking the API. + + +## Alternatives considered + +The functionality could also be provided entirely through the `Mirror`/reflection APIs. This would likely result in much more obscure and confusing usage patterns. + +#### Provide a default collection + +Declaring this in the Standard Library reduces the amount of compiler magic required +to implement the protocol. However, it also exposes a public unsafe entrypoint to the +reflection API that we consider unacceptable. + +```swift +extension CaseIterable where AllCases == DefaultCaseCollection { + public static var allCases: DefaultCaseCollection { + return DefaultCaseCollection(unsafeForEnum: Self.self) + } +} + +public struct DefaultCaseCollection: RandomAccessCollection { + public var startIndex: Int { return 0 } + public let endIndex: Int + + public init(unsafeForEnum _: Enum.Type) { + endIndex = _countCaseValues(Enum.self) + } + + public subscript(i: Int) -> Enum { + precondition(indices.contains(i), "Case index out of range") + return Builtin.reinterpretCast(i) as Enum + } +} +``` + +### [Metatype conformance to Collection](https://forums.swift.org/t/re-pitch-deriving-collections-of-enum-cases/6956/11) + +A *type* inherently represents a set of its possible values, and as such, `for val in MyEnum.self { }` could be a natural way to express iteration over all cases of a type. Specifically, if the metatype `MyEnum.Type` could conform to `Collection` or `Sequence` (with `Element == MyEnum`), this would allow `for`-loop enumeration, `Array(MyEnum.self)`, and other use cases. In a generic context, the constraint `T: CaseIterable` could be expressed instead as `T.Type: Collection`. + +Absent the ability for a metatype to actually conform to a protocol, the compiler could be taught to treat the special case of enum types *as if* they conformed to Collection, enabling this syntax before metatype conformance became a fully functional feature. diff --git a/proposals/0195-dynamic-member-lookup.md b/proposals/0195-dynamic-member-lookup.md new file mode 100644 index 0000000000..3989a15b50 --- /dev/null +++ b/proposals/0195-dynamic-member-lookup.md @@ -0,0 +1,656 @@ +# Introduce User-defined "Dynamic Member Lookup" Types + +* Proposal: [SE-0195](0195-dynamic-member-lookup.md) +* Author: [Chris Lattner](https://github.com/lattner) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Implementation: [apple/swift#14546](https://github.com/apple/swift/pull/14546) +* [Previous Revision #1](https://github.com/swiftlang/swift-evolution/commit/59c7455170c231f3df9ab0ba923262e126afaa06#diff-b3460d13f154c3d6b1d8396e4159a1d2) +* Status: **Implemented (Swift 4.2)** +* Decision Notes: [Review extended](https://forums.swift.org/t/se-0195-introduce-user-defined-dynamic-member-lookup-types/8658/126), [Rationale](https://forums.swift.org/t/se-0195-introduce-user-defined-dynamic-member-lookup-types/8658/160) + + +## Introduction + +This proposal introduces a new `@dynamicMemberLookup` attribute. Types that use it provide +"dot" syntax for arbitrary names which are resolved +at runtime - in a **completely type safe** way. This provides syntactic sugar that allows the +user to write: + +```swift + a = someValue.someMember + someValue.someMember = a + mutateParameter(&someValue.someMember) +```` + +and have it be interpreted by the compiler as: + +```swift + a = someValue[dynamicMember: "someMember"] + someValue[dynamicMember: "someMember"] = a + mutateParameter(&someValue[dynamicMember: "someMember"]) +``` + +This allows the static type of `someValue` to decide how to implement these dynamic member +references. + +Many other languages have analogous features e.g., the [dynamic](https://docs.microsoft.com/en-us/dotnet/framework/reflection-and-codedom/dynamic-language-runtime-overview) feature in C#, the [Dynamic trait in Scala](https://blog.scalac.io/2015/05/21/dynamic-member-lookup-in-scala.html), the composition of Objective-C's +[explicit properties](https://developer.apple.com/library/content/documentation/General/Conceptual/DevPedia-CocoaCore/DeclaredProperty.html) and underlying [messaging infrastructure](https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtHowMessagingWorks.html). This sort +of functionality is great for implementing dynamic language interoperability, dynamic +[proxy APIs](https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtForwarding.html), and other APIs (e.g. for JSON processing). + +The driving motivation for this feature is to improve interoperability with inherently dynamic +languages like Python, Javascript, Ruby and others. That said, this feature is designed such +that it can be applied to other inherently dynamic domains in a modular way. **NOTE**: if +you are generally supportive of interoperability with dynamic languages but are +concerned about the potential for abuse of this feature, please see the [Reducing Potential +Abuse](#reducing-potential-abuse) section in the Alternatives Considered section and voice +your support for one of those directions that limit the feature. + +## Motivation and Context + +Swift is well known for being exceptional at interworking with existing C and Objective-C +APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby +is quite lacking. + +C and Objective-C are integrated into Swift by expending a heroic amount of effort into +integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by +providing a large number of attributes and customization points for changing the behavior +of this integration when writing an Objective-C header. The end result of this massive +investment of effort is that Swift tries to provide an (arguably) *better* experience when +programming against these legacy APIs than Objective-C itself does. + +When considering the space of dynamic languages, four things are clear: 1) there are several +different languages of interest, and they each have significant communities in different areas: +for example, Python is big in data science and machine learning, Ruby is popular for building +server side apps, a few people apparently use Javascript, and even Perl is still widely used. +2) These languages have decades of library building behind them, sometimes with [significant +communities](https://pandas.pydata.org) and 3) there are one or two orders of magnitude +more users of these libraries than there are people currently using Swift. 4) APIs written in +these languages will never feel "Swifty", both because of serious differences between the +type systems of Swift and these languages, and because of runtime issues like the Python +GIL. + +In the extensive discussion and development of this feature (including hundreds of emails to +swift-evolution) we considered many different implementation approaches. These include +things like Objective-C style bridging support, F# type providers, generated wrappers, foreign +class support, and others described in the [Alternative Python Interoperability +Approaches](#alternative-python-interoperability-approaches) section. + +The conclusion of these many discussions was that it is better to embrace the fact +that these languages are inherently dynamic and meet them where they are: F# type providers +and generated wrappers require this proposal to work in the first place (because, for example, +Javascript does not have classes and Python doesn't have stored property declarations) and +providing a good code completion experience for dynamic languages requires incorporation +of flow-sensitive analysis into SourceKit (something that is [fully compatible](#future-directions-python-code-completion) +with this proposal). + +Given that Swift already has an intentionally incredibly syntax-extensible design, we only need +two minor enhancements to the language to support these dynamic languages in an +ergonomic way: this proposal (which introduces the `@dynamicMemberLookup` attribute) and a +related [`@dynamicCallable`](0216-dynamic-callable.md) proposal. + +To show the impact of these proposals, consider this Python code: + +```Python +class Dog: + def __init__(self, name): + self.name = name + self.tricks = [] # creates a new empty list for each dog + def add_trick(self, trick): + self.tricks.append(trick) + return self +``` + +we would like to be able to use this from Swift like this (the comments show the +corresponding syntax you would use in Python): + +```swift + // import DogModule + // import DogModule.Dog as Dog // an alternate + let Dog = Python.import("DogModule.Dog") + + // dog = Dog("Brianna") + let dog = Dog("Brianna") + + // dog.add_trick("Roll over") + dog.add_trick("Roll over") + + // cuteDog = Dog("Kaylee").add_trick("snore") + let cuteDog = Dog("Kaylee").add_trick("snore") +``` + +Of course, this would also apply to standard Python APIs as well. Here is an example +working with the Python `pickle` API and the builtin Python function `open`: + +```swift + // import pickle + let pickle = Python.import("pickle") + + // file = open(filename) + let file = Python.open(filename) + + // blob = file.read() + let blob = file.read() + + // result = pickle.loads(blob) + let result = pickle.loads(blob) +``` + +This can all be expressed today as library functionality written in Swift, but without this +proposal, the code required is unnecessarily verbose and gross. Without it (but *with* the +related [`@dynamicCallable`](0216-dynamic-callable.md) proposal) +the code would have explicit member lookups all over the place: + +```swift + // import pickle + let pickle = Python.get(member: "import")("pickle") + + // file = open(filename) + let file = Python.get(member: "open")(filename) + + // blob = file.read() + let blob = file.get(member: "read")() + + // result = pickle.loads(blob) + let result = pickle.get(member: "loads")(blob) + + // dog2 = Dog("Kaylee").add_trick("snore") + let dog2 = Dog("Kaylee").get(member: "add_trick")("snore") +``` + +If you'd like to explore what Python interoperability looks like with plain Swift 4 (i.e. without +either of these proposals) then check out the [Xcode 9 playground demonstrating Python +interoperability](https://forums.swift.org/t/python-interop-with-swift-4/7109/9) +that has been periodically posted to swift-evolution over the last few months. + +While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in +important new domains. In addition to dynamic language interoperability, this sort of +functionality is useful for other APIs, e.g. when working with dynamically typed unstructured +data like JSON, which could provide an API like `jsonValue?.jsonField1?.jsonField2` +where each field is dynamically looked up. An example of this is shown below in the +"Example Usage" section. + +## Proposed solution + +We propose introducing a new attribute to the Swift language, `@dynamicMemberLookup`. + +Types with this attribute on their primary type declaration have the behavior that member +lookup - accessing `someval.member` will always succeed. Failures to find normally +declared members of `member` will be turned into subscript references using the +`someval[dynamicMember: member]` member. It is an error to put the +`@dynamicMemberLookup` attribute on a type but not have this subscript declared. + +This attribute extends the language such that member lookup syntax (`x.y`) - when it +otherwise fails (because there is no member `y` defined on the type of `x`) and when applied +to a value with the `@dynamicMemberLookup` attribute - is accepted and +transformed into a subscript on `x`. The produced value is a mutable +L-value if the type implements a mutable +subscript, or immutable otherwise. This allows the type to perform arbitrary runtime +processing to calculate the value to return. The dynamically computed property can be used +the same way as an explicitly declared computed property, including being passed `inout` if +mutable. + +The attribute is intentionally designed to be flexible: the implementation can take the member +name through any `ExpressibleByStringLiteral` type, including `StaticString` and of +course `String`. The result type may also be any type the implementation desires, including +an `Optional`, `ImplicitlyUnwrappedOptional` or some other type, which allows the +implementation to reflect dynamic failures in a way the user can be expected to process (e.g., +see the JSON example below). + +## Example Usage + +While there are many potential uses of this sort of API, one motivating example comes from a +[prototype Python interoperability layer](https://forums.swift.org/t/python-interop-with-swift-4/7109/9). +There are many ways to implement this, and the details are not +particularly important, but it is perhaps useful to know that this is directly useful to address +the motivation section described above. Given a currency type of `PyVal`, an implementation +may look like: + +```swift +@dynamicMemberLookup +struct PyVal { + ... + subscript(dynamicMember member: String) -> PyVal { + get { + let result = PyObject_GetAttrString(borrowedPyObject, member)! + return PyVal(owned: result) + } + set { + PyObject_SetAttrString(borrowedPyObject, member, + newValue.borrowedPyObject) + } + } +} +``` + +Another example use are JSON libraries which represent JSON blobs as a Swift enum, e.g.: + +```swift +enum JSON { + case IntValue(Int) + case StringValue(String) + case ArrayValue(Array) + case DictionaryValue(Dictionary) +} +``` + +Today, it is not unusual for them to implement members like this to allow drilling down into +the JSON value: + +```swift +extension JSON { + var stringValue : String? { + if case .StringValue(let str) = self { + return str + } + return nil + } + subscript(index: Int) -> JSON? { + if case .ArrayValue(let arr) = self { + return index < arr.count ? arr[index] : nil + } + return nil + } + subscript(key: String) -> JSON? { + if case .DictionaryValue(let dict) = self { + return dict[key] + } + return nil + } +} +``` + +This allows someone to drill into a JSON value with code like: +`json[0]?["name"]?["first"]?.stringValue`. On the other hand, if we add the +`@dynamicMemberLookup` attribute and an implementation like this: + +```swift +@dynamicMemberLookup +enum JSON { + ... + subscript(dynamicMember member: String) -> JSON? { + if case .DictionaryValue(let dict) = self { + return dict[member] + } + return nil + } +} +``` + +Now clients are able to write more natural code like: +`json[0]?.name?.first?.stringValue` which is close to the expressivity of Javascript... +while being fully type safe! + +It is important to note that this proposal does *not* include introducing or changing an existing +JSON type to use this mechanic, we only point out that this proposal allows subsequent work +to turn this on if sagacious API authors approve of it. + + +### Future Directions: Python Code Completion + +In the extensive discussion of alternative implementation approaches, one concern raised +was whether or not we could ever get a good code completion experience with this design. +It would be nice to get at least something along the lines of what Swift provides for +`AnyObject` lookup, where you can get a "big list" and filter down quickly as you type. + +After extensive discussion at the Core Team, we concluded that the *best* way to get a good +code completion for Python APIs in Swift (if and when that becomes a priority) is to build such +functionality into SourceKit, and model it directly after the way that existing Python IDEs +provide their code completion. + +The observation is that a state of the art Python code completion experience requires +incorporating simple control flow analysis and unsound heuristics into the the model in order +to take local hints into account, pre-filtering the lists. These heuristics would be inappropriate +to include in the static type system of the Swift language (in any form), and thus we believe it +is better to build this as special support in SourceKit (e.g. special casing code completion on +`Python.PyVal` (or whatever the currency type ends up being called) to incorporate these +techniques. + +In any case, while we would like to see such future developments, but they are beyond the +scope of this proposal, and are a tooling discussion for "swift-dev", not a language evolution +discussion. + + +## Source compatibility + +This is a strictly additive proposal with no source breaking changes. + +## Effect on ABI stability + +This is a strictly additive proposal with no ABI breaking changes. + +## Effect on API resilience + +Types with this attribute will always succeed at member lookup (`x.foo` will +always be accepted by the compiler): members that are explicitly declared in the type or in +a visible extension will be found and referenced, and anything else will be handled by the +dynamic lookup feature. + +That behavior could lead to a surprising behavior change if the API evolves over time: adding +a new statically declared member to the type or an extension will cause clients to resolve that +name to the static declaration instead of being dynamically dispatched. This is inherent to +this sort of feature, and means it should not be used on types that have a large amount of +API, API that is likely to change over time, or API with names that are likely to conflict. + +## Alternatives considered + +A few alternatives were considered: + +### Spelling: Attribute vs Declaration Modifier + +This proposal argues for spelling this as a `@dynamicMemberLookup` attribute that is +applied to a type, but there are many other ways this can be spelled. There could be other +]attribute names that are worth considering (suggestions welcome!), and it is also possible to +spell this as a declaration modifier like `dynamicMemberLookup`. + +### Make this a marker protocol + +We started with the approach of making this be a protocol that types conform to to get this +behavior. It turns out that this behavior is very non-protocol like: it is not useful to define +generic algorithms over, and existential values are only useful if they define a specific +subscript that implements the requirements implicit in this attribute. + +For these and other reasons, defining this as a protocol doesn't really fit into the design of +Swift. + +### Model this with methods, instead of a labeled subscript + +It may be surprising to some that this functionality is modeled as a subscript instead of a +get/set method pair. This is intentional though: subscripts are the way that Swift supports +parameterized l-values like we're are trying to expose here. Exposing this as two methods +doesn't fit into the language as cleanly, and would make the compiler implementation more +invasive. It is better to use the existing model for l-values directly. + +### Reducing Potential Abuse + +In the discussion cycle, there was significant concern about abuse of this feature, particularly +when this was spelled as a protocol that would allow someone to retroactively conform a type +to the `DynamicMemberLookupProtocol` protocol. For this and other reasons, this proposal +has been revised to be an attribute that can be applied to a primary type declaration, not to +be a protocol. + +On the other hand, the potential for abuse has never been a strong guiding principle for Swift +features (and many existing features could theoretically be abused (including operator +overloading, emoji identifiers, +`AnyObject` lookup, `ImplicitlyUnwrappedOptional`, and many more). If we look to +other language communities, we find that Objective-C has many inherently dynamic features. +Furthermore, directly +analogous "dynamic" features were [added to C#](https://docs.microsoft.com/en-us/dotnet/framework/reflection-and-codedom/dynamic-language-runtime-overview) and Scala late in +their evolution cycle, and no one has produced evidence that they led to abuse. + +Finally, despite **extensive discussion** on the mailing list and lots of concern +about how this feature could be abused, no one has been able to produce (even one) +non-malicious example where someone would use this feature inappropriately and lead to +harm for users (and of course, if you're consuming an API produced by a malicious entity, you +are already doomed. `:-)`). + +Fortunately (if and only if a compelling example of harm were demonstrated) there are +different ways to assuage concerns of "abuse" of this feature, for example we could have +the compiler specifically bless individual well-known types, e.g. `Python.PyVal` (or +whatever it is eventually named) by baking in awareness of these types into the compiler. +Such an approach would require a Swift evolution proposal to add a new type that +conforms to this. + +Despite such possibilities, this doesn't seem like a likely future direction. + +### Increasing Visibility of Dynamic Member Lookups + +People have suggested that we add some explicit syntax to accesses to make the dynamic +lookup visible, e.g.: `foo.^bar` or `foo->bar` or some other punctuation character we +haven't already used. In my opinion, this is the wrong thing to do for several reasons: + +1) Swift's type system already includes features (optionals, IUOs, runtime failure) for handling + failability. Keeping that orthogonal to this proposal is good because that allows API + authors to make the right decision for expressing the needs of their use-case. +2) Swift already has a dynamic member lookup feature, "`AnyObject` dispatch" which does + not use additional punctuation, so this would break precedent. The syntax and behavior + of AnyObject dispatch was carefully considered in a situation that was directly analogous + to this - Swift 1 days - where nullability audited headers were rare. +3) Adding punctuation to the lookup itself reduces the likelihood that an API author would + make the lookups return strong optionals, because of increased syntactic noise. +4) The point of this proposal is to make use of dynamic language APIs more elegant than + what is already possible: making use of them ugly (this punctuation character would be + pervasive through use of the APIs and just add visual noise, not clarity) undermines the + entire purpose of this proposal. +5) There are already other features (including operator overloading, subscripts, forthcoming + `@dynamicCallable`, etc) that are just as dynamic as property lookup when implemented + on a type like `PyVal`. Requiring additional syntax for "`a.b`" but not "`a + b`" (which can + be just as dynamic) would be inconsistent. +6) Language syntax is not the only way to handle this. IDEs like Xcode could color code + dynamic member lookups differently, making their semantics visible without adversely + affecting the code that is written. It is true that not all developers use Xcode, but since + many other major pieces of Swift assume a rich editor experience, it is consistent for this + one to expect it too. + +It probably helps to consider an example. Assume we used the `^` sigil to represent a dynamic +operation (member lookup, call, dynamic operator, etc). This would give us syntax like +`foo.^bar` for member lookup, `baz^(42)` for calls. The API author isn't forced to pick an +operator that follows this scheme, but has the option to do so. Such a design would +change reasonable code like this: + +```swift + let np = Python.import("numpy") + let x = np.array([6, 7, 8]) + let y = np.arange(24).reshape(2, 3, 4) + + let a = np.ones(3, dtype: np.int32) + let b = np.linspace(0, pi, 3) + let c = a+b + let d = np.exp(c) + print(d) +``` + +into: + +```swift + let np = Python.import("numpy") + let b = np^.array^([6, 7, 8]) + let y = np^.arange^(24)^.reshape^(2, 3, 4) + + let a = np^.ones^(3, dtype: np^.int32) + let b = np^.linspace^(0, pi, 3) + let c = a+^b + let d = np^.exp^(c) +``` + +This does not improve clarity of code, it merely serves to obfuscate logic. It is immediately +apparent from the APIs being used, the API style, and the static types (in Xcode or through +static declarations) that this is all Python stuff. When you start mixing in use of native Swift +types like dictionaries (something we want to encourage because they are typed!) you end up +with an inconsistent mishmash where people would just try adding syntax or applying fixits +continuously until the code builds. + +The example gets ever worse if the implementation chooses to return a strong optional value, +because you'd end up with something like this (just showing a couple lines): + +```swift + let y = np^.arange?^(24)?^.reshape?^(2, 3, 4)! + let a = np^.ones?^(3, dtype: np^.int32!)! +``` + +This is so bad that no one would actually do this. Making the Python operators return +optionals would be even worse, since binary operators break optional chaining. + +## Alternative Python Interoperability Approaches + +In addition to the alternatives above (which provide different approaches to refine a +proposal along the lines of this one), there have also been extensive discussion of different +approaches to the problem of dynamic language interoperability on the whole. Before +we explore those alternatives, we need to understand the common sources of concern. + +The biggest objection to this proposal is that it is "completely dynamic", and some +people have claimed that Swift intentionally pushes people towards static types. Others +claim that dynamic features like this are unsafe (what if a member doesn't exist?), and claim +that Swift does not provide unsafe features. + +While the aims are largely correct for pure Swift code, the only examples of language +interoperability we have so far is with C and Objective-C, and Swift's interop with those is +indeed already memory unsafe, unsound, and far more invasive than what we propose. +Observe: + +1) Swift does have completely dynamic features, even ones that can be used unsafely or + unwisely. There was a recent talk at Swift Summit 2017 that explored some of these. +2) Bridging to dynamic languages inherently requires "fully dynamic" facilities, because the + imported language has fully dynamic capabilities, and programmers use them. This is + pervasive in Python code, and is also reasonable common in Objective-C "the `id` type". +3) The Objective-C interoperability approach to handling the "inherently dynamic" parts of + Objective-C is a feature called "AnyObject dispatch". If you aren't familiar, it returns + members lookup as `ImplicitlyUnwrappedOptional` types (aka, `T!` types), which are + extremely dangerous to work with - though not "unsafe" in the Swift sense. +4) Beyond the problems with IUOs, Anyobject lookup is *also* completely non-type safe when + the lookup is successful: it is entirely possible to access a property or method as "String" + even if it were declared as an integer in Objective-C: the bridging logic doesn't even return + such a lookup as `nil` in general. +5) The implementation of "AnyObject lookup" is incredibly invasive across the compiler and + has been the source of a large number of bugs. +6) Beyond AnyObject lookup, the Clang importer itself is ~25 thousand lines of code, and + while it is the source of great power, it is also a continuous source of bugs and problems. + Beyond the code in the Clang importer library itself, it also has tendrils in almost + every part of the compiler and runtime. + +in contrast, the combination of the `@dynamicMemberLookup` and `@dynamicCallable` +proposals are both minimally invasive in the compiler, and fully type safe. Implementations of +these proposals may choose to provide one of three styles of fully type safe implementation: + +1) Return optional types, forcing clients to deal with dynamic failures. We show an example + of this in the JSON example above. +2) Return IUO types, providing the ability for clients to deal with dynamic failures, but allow + them to ignore them otherwise. This approach is similar to AnyObject dispatch, but is + more type safe. +3) Return non-optional types, and enforce type safety at runtime with traps. This is the + approach followed in the Python interop example above, but that bridge is under + development and may switch to another approach if it provides a better experience. The + actual prototype already has a more holistic approach for handling failability that isn't + describe here. + +The style of design depends on the exact bridging problem being solved, e.g. the customs of +the language being interoperated with. Also, yes, it is possible to use these APIs to provide +an unsafe API, but that is true of literally every feature in Swift - Swift provides support +for unsafe features as part of its goals to be pragmatic. + +With this as background, let's explore the proposed alternatives approaches to dynamic +language interoperability: + +### Direct language support for Python (and all the other languages) + +We considered implementing something analogous to the Clang importer for Python, which +would add a first class Python specific type(s) to Swift language and/or standard library. We +rejected this option because: + +1) Such integration would **require that we do something like this proposal anyway**, + because Python (like Objective-C) is a fundamentally dynamic language, and we need a + way to express that fundamental dynamism: either this proposal or something like + "`AnyObject` for Python". +2) Python is actually far less "inherently typed" than Objective-C is, because everything + is typed with the equivalent of `id`, whereas the Objective-C community has used concrete + types for many things (and with the introduction of Objective-C Generics and other + features, this has become even more common). +3) it would be *significantly* more invasive in the compiler than this proposal. While it is + unlikely to be as large a scope of impact as the Clang importer in terms of bulk of code, it + would require just as many tendrils spread throughout the compiler. +4) Taking this approach would set the precedent for all other dynamic languages to get first + class language support baked into the Swift compiler, leading to an even greater + complexity spiral down the road. +5) The proposed "first class support" doesn't substantially improve the experience of working + with Python over this proposal, so it is all pain and no gain. + +Several people have suggested that a "Clang-importer" style Python interoperability approach +could use the "Type Hints" introduced in +[PEP 484](https://www.python.org/dev/peps/pep-0484/), which would invalidate that last +point above. This approach to progressive typing for Python was prototyped in +[mypy](http://mypy-lang.org/) and first [shipped in Python +3.5](https://docs.python.org/3/library/typing.html) in September 2015. + +Unfortunately, it isn't reasonable to expect Type Hints to significantly improve the experience +working with Python in Swift for several reasons, including: + +1) PEP 484 states: "It should also be emphasized that **Python will remain a dynamically + typed language, and the authors have no desire to ever make type hints mandatory, + even by convention.**" (the emphasis is by the authors of PEP 484). This means we + need this proposal or something like AnyObject lookup ... forever. +2) These annotations are only currently supported on a [provisional + basis](https://docs.python.org/3/glossary.html#term-provisional-api), which means that + they are "deliberately excluded from .. backwards compatibility guarantees... up to and + including removal of the interface". Because they are subject to change and potentially + removal, they haven't gotten wide adoption. +3) Unlike Objective-C type annotations, these type hints are inherently unsound by design. + Further, Python APIs often creep to accepting many more types than a Swift programmer + would expect, which means that a type annotation is often "so broad as to be unhelpful" or + "too narrow to be correct". +4) These annotations are not dynamically enforced, and they are specifically designed to + "influence" static analysis tools, which means that they are also frequently wrong. In the + context of a narrowly used static analysis tool, a wrong type annotation is merely annoying. + In the context of bridging into Swift, being incorrect is a real problem. +5) The fact that they have only been available in Python **3** (which has a smaller community + than Python 2) and have only been shipping for 2 years, means that they haven't been used + by many people, and which contributes to their low adoption. +6) It is a goal to work with other dynamic language communities beyond Python, and not + all have progressive typing facilities like this one. +7) Type annotations help the most in situations when you are "mixing and matching" Swift + code with an existing body of some other code that you are able to modify. While it is + possible that some people will want to mix and match Swift and Python, by far the most + common reason for wanting to interoperate with a dynamic language is to leverage the + existing APIs that the community provides in a black box manner. Being black box means + that you want to reuse the code, but you don't want to touch and own it yourself. +8) Finally, the idea of Swift providing a "better Python than Python itself does" + under-appreciates the effort that the Python community has spent trying to achieve the + same goals. Its community includes a lot of people in it that understand the + benefits of static and progressive typing (e.g. the [mypy](http://mypy-lang.org/) community) + and they have spent a lot of time on this problem. Despite their efforts, the ideas have low + adoption: Swift bridging doesn't change the fundamental reasons for this. + +Finally, it is important to realize that Swift and Clang have a +special relationship because they were designed by many of the same people - so their +implementations are similar in many respects. Further, the design of both Clang and the +Objective-C language shifted in major ways to support interoperability with Swift - including +the introduction of ARC, Modules, Objective-C generics, pervasive lazy decl resolution, and +many more minor features). + +It is extremely unlikely that another established language/compiler community would accept +the scope of changes necessary to provide great importer support for them, and it is also +extremely unlikely that one would just magically work out of the box for what we need it to +do. That said, our goals aren't to provide a better Python than Python, only to embrace +Python (and other dynamic languages) for what they are, without polluting the Swift compiler +and runtime with a ton of complexity that we'll have to carry around forever. + + +### Introduce F# style "Type Providers" into Swift + +[Type providers](https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/type-providers/) are +a cool feature of the F# language. They are an expansive (and quite +complex) metaprogramming system which permits a "metaprogrammed library" to synthesize +types and other language features into the program based on statically knowable type +databases. This leads to significantly improved type safety in the case where schemas for +dynamic APIs are available (e.g. a JSON schema) but which are not apparent in the source +code. + +While type providers are extremely interesting and might be considered for inclusion into a +future macro system in Swift, it is important to understand that they just provide a way for +library developers to extend the compiler along the lines of the Clang importer in Swift. As +such, they aren't actually helpful for this proposal for all of the reasons described +in the section above, but the most important point is that: + +Type providers can only "provide" a type that is expressible in the language, and dynamic + languages do have completely dynamic features, so **something that provides the +semantics of `DynamicMemberLookup` will be required** with that approach anyway. We'd +have to take this proposal (or something substantially similar) before type providers would be +useful for Python in the first place. + +Because of this, we believe that type providers and other optional typing facilities are +a potentially interesting follow-on to this proposal which should be measured according to +their own merits. The author of this proposal is also personally skeptical that type providers +could ever be done in a way that is acceptable within the goals of Swift (e.g. to +someday have fast compile times). + +### Introduce a language independent "foreign class" feature to Swift + +One suggestion was to introduce a [general "foreign class" feature to Swift](https://forums.swift.org/t/proposal-introduce-user-defined-dynamic-member-lookup-types/7129/90). The core team met to discuss this and concluded that it was the wrong direction to go. Among opinions held by core team members, several believed that forcing other languages models into the Swift model would violate their fundamental principles (e.g. Go and Javascript don't *have* classes), some felt it would be too invasive into the compiler, and others believed that such an approach ends up requiring a `DynamicMemberLookup` related feature anyway - because +e.g. Python doesn't require property declarations. + +### Use "automatically generated wrappers" to interface with Python + +This approach has [numerous problems](https://forums.swift.org/t/dynamicmemberlookup-proposal-status-update/7358). Beyond that, wrappers also fundamentally require that we take (something like) this proposal +to work in the first place. The primary issue is that Python (and other dynamic languages) +require no property declarations. If you have no property declaration, there is nothing for the +wrapper generator to [w]rap or generate. + diff --git a/proposals/0196-diagnostic-directives.md b/proposals/0196-diagnostic-directives.md new file mode 100644 index 0000000000..2b69a98991 --- /dev/null +++ b/proposals/0196-diagnostic-directives.md @@ -0,0 +1,170 @@ +# Compiler Diagnostic Directives + +* Proposal: [SE-0196](0196-diagnostic-directives.md) +* Author: [Harlan Haskins](https://github.com/harlanhaskins) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Implementation: [apple/swift#14048](https://github.com/apple/swift/pull/14048) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/ab0c22a2340be9bfcb82e6f237752b4d959a93b7/proposals/0196-diagnostic-directives.md) +* Status: **Implemented (Swift 4.2)** + +## Introduction + +This proposal introduces `#warning` and `#error` directives that will cause +the Swift compiler to emit a custom warning or an error during compilation. + +## Motivation + +During the development lifecycle, it's common to leave bits of code unfinished +while focusing on other parts. Frequently, developers leave `TODO` or `FIXME` +comments around unfinished or suboptimal portions of their code to signal +to themselves and their teammates that the code needs work. Many editors pick +up on these comments and present them in code navigators, but these don't +materialize in command-line builds or continuous integration. Allowing #warning +gives a predictable, universal way to ensure a message will be displayed during +compilation. + +Additionally, it's possible for two build configurations to be mutually +exclusive, and there isn't a canonical way to ensure those configurations +don't happen. `#error` solves this problem: + +```swift +#if MYLIB_VERSION < 3 && os(macOS) +#error("MyLib versions < 3 are not supported on macOS") +#endif +``` + +`#warning` can be used for code templates where the user is meant to fill in +the values of global constants or implement missing routines: + +```swift +enum APICredentials { + #warning("fill in your API key below") + static let key = "" + #warning("fill in your API secret below") + static let secret = "" +} +``` + +## Proposed solution + +Add `#warning` and `#error` as compiler directives that emit an appropriate +diagnostic with the contents, pointing to the start of the message. + +```swift +func configPath() -> String { + #warning("this should be made more safe") + return Bundle.main().path(forResource: "Config", ofType: "plist")! +} +``` + +## Detailed design + +This will add four new productions to the Swift grammar: + +``` +compiler-control-statement → warning-directive +compiler-control-statement → error-directive +warning-directive → #warning '(' static-string-literal ')' +error-directive → #error '(' static-string-literal ')' +``` + +Upon parsing a `#error` directive, the Swift compiler will emit the provided +string literal as an error, pointing to the beginning of the +string, and ignore the directive. + +Upon parsing a `#warning` directive, the Swift compiler will emit the provided +string literal as a warning, pointing to the beginning of the +string, and ignore the directive. + +If a `#warning` or `#error` exists inside a branch of a `#if` directive that is +not taken, then no diagnostic is emitted. + +```swift +#if false +#warning("this will not trigger a warning") +#error("this will not trigger an error") +#endif + +#if true +#warning("this will trigger a warning") +#error("this will trigger an error") +#endif +``` + +## Impact on existing code + +This change is purely additive; no migration will be required. + +## Alternatives considered + +- We could do some kind of comment-parsing based approach to surface + `TODO`s and `FIXME`s, but `#warning` serves as a general-purpose facility + for reporting at compile time. It is also likely that there are `TODO` and + `FIXME` comments that are not urgent, but that should be revisited. + +- [Alexander Momchilov](https://forums.swift.org/t/pitch-warning/2819/41) brought + up the idea of using `TODO` and `warning` as functions in the standard + library with special compiler magic that will warn on their uses. + + ```swift + func TODO(_ message: StaticString? = nil) { + if let s = message { print("TODO: \(s)") } + } + + @discardableResult + func TODO(_ message: StaticString? = nil, _ temporaryValue: T) -> T { + if let s = message { print("TODO: \(s)") } + return temporaryValue + } + ``` + While these could be useful, I think `#warning` and `#error` have uses beyond + just marking unfinished code that would be unwieldy or impossible with + just an expression-oriented approach. + +- [Erik Little](https://forums.swift.org/t/pitch-warning/2819/42?) refined that + to instead use special directives `#warning` and `#error` in expression + position, like: + + ```swift + let somethingSuspect = #warning("This is really the wrong function to call, but I'm being lazy", suspectFunction()) + ``` + However, I think there's not much of a benefit to this syntax vs. just + adding a `#warning` above the line: + ```swift + #warning "This is really the wrong function to call, but I'm being lazy" + let somethingSuspect = suspectFunction() + ``` + +- [A few](https://forums.swift.org/t/pitch-warning/2819/39) + [people](https://forums.swift.org/t/pitch-warning/2819/37) have requested + `#message` or `#info`, as an analogue for Clang's `#pragma message`. This may + be something we want, but I didn't include it in this proposal because as of + this writing, Clang treats `#pragma message` as a warning and flags it + as `-W#pragma-message`. + +## Future directions + +Both `#message` and an expression-based `#warning` are additive with respect +to this proposal, and both could be addressed in future proposals. + +------------------------------------------------------------------------------- + +# Rationale + +On February 1, 2018 the Core Team decided to **accept** this proposal with +slight revision over the [original proposal](https://github.com/swiftlang/swift-evolution/blob/ab0c22a2340be9bfcb82e6f237752b4d959a93b7/proposals/0196-diagnostic-directives.md). + +The only revision over the original proposal is to change the syntax to use +`#warning()` instead of `#warning `. This fits well with +most of Swift's existing compiler directives, and was strongly supported in +the [review discussion](https://forums.swift.org/t/se-0196-compiler-diagnostic-directives/8734). + +The review discussion also covered a variety of possible extensions or +variants to this proposal, including support for using `#warning` [as an +expression](https://forums.swift.org/t/se-0196-compiler-diagnostic-directives/8734/21) +instead of a line directive and [support for runtime issues](https://forums.swift.org/t/se-0196-compiler-diagnostic-directives/8734/6). +The Core Team decided that while these directions are interesting and worth +exploring, they are complementary to the core functionality serviced by this +proposal. Further, keeping `#warning` as a line directive allows it to be +used in a wide variety of contexts, and serves a different need than using it +as a placeholder expression. diff --git a/proposals/0197-remove-where.md b/proposals/0197-remove-where.md new file mode 100644 index 0000000000..514aabc2cf --- /dev/null +++ b/proposals/0197-remove-where.md @@ -0,0 +1,140 @@ +# Adding in-place `removeAll(where:)` to the Standard Library + +* Proposal: [SE-0197](0197-remove-where.md) +* Author: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift#11576](https://github.com/apple/swift/pull/11576) +* Review: [Thread](https://forums.swift.org/t/se-0197-add-in-place-remove-where/8872) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/feec7890d6c193e9260ac9905456f25ef5656acd/proposals/0197-remove-where.md) + +## Introduction + +It is common to want to remove all occurrences of a certain element from a +collection. This proposal is to add a `removeAll` algorithm to the +standard library, which will remove all entries in a collection in-place +matching a given predicate. + +## Motivation + +Removing all elements matching some criteria is a very common operation. +However, it can be tricky to implement correctly and +efficiently. + +The easiest way to achieve this effect in Swift 3 is to use `filter` and assign +back, negating the thing you want to remove (because `filter` takes a closure +of items to "keep"): + +```swift +var nums = [1,2,3,4,5] +// remove odd elements +nums = nums.filter { !isOdd($0) } +``` + +In addition to readability concerns, this has two performance problems: fresh +memory allocation, and a copy of all the elements in full even if none need to +be removed. + +The alternative is to open-code a `for` loop. The simplest performant solution +is the "shuffle-down" approach. While not especially complex, it is certainly +non-trivial: + +```swift +if var i = nums.index(where: isOdd) { + var j = i + 1 + while j != nums.endIndex { + let e = nums[j] + if !isOdd(nums[j]) { + nums[i] = nums[j] + i += 1 + } + j += 1 + } + nums.removeSubrange(i...) +} +``` + +Possibilities for logic and performance errors abound. There are probably some +in the above code. + +Additionally, this approach does not work for range-replaceable collections +that are _not_ mutable i.e. collections that can replace subranges, but can't +guarantee replacing a single element in constant time. `String` is the most +important example of this, because its elements (graphemes) are variable width. + +## Proposed solution + +Add the following method to `RangeReplaceableCollection`: + +```swift +nums.removeAll(where: isOdd) +``` + +The default implementation will use the protocol's `init()` and `append(_:)` +operations to implement a copy-based version. Collections which also conform to +`MutableCollection` will get the more efficient "shuffle-down" implementation, +but still require `RangeReplaceableCollection` as well because of the need to +trim at the end. Other types may choose + +Collections which are range replaceable but _not_ mutable (like `String`) will +be able to implement their own version which makes use of their internal +layout. Collections like `Array` may also implement more efficient versions +using memory copying operations. + +Since `Dictionary` and `Set` would benefit from this functionality as well, but +are not range-replaceable, they should be given concrete implementations for +consistency. + +## Detailed design + +Add the following to `RangeReplaceableCollection`: + +```swift +protocol RangeReplaceableCollection { + /// Removes every element satisfying the given predicate from the collection. + mutating func removeAll(where: (Iterator.Element) throws -> Bool) rethrows +} + +extension RangeReplaceableCollection { + mutating func removeAll(where: (Iterator.Element) throws -> Bool) rethrows { + // default implementation similar to self = self.filter + } +} +``` + +Other protocols or types may also have custom implementations for a faster +equivalent. For example `RangeReplaceableCollection where Self: +MutableCollection` can provide a more efficient non-allocating default +implementation. `String` is also likely to benefit from a custom implementation. + +## Source compatibility + +This change is purely additive so has no source compatibility consequences. + +## Effect on ABI stability + +This change is purely additive so has no ABI stability consequences. + +## Effect on API resilience + +This change is purely additive so has no API resilience consequences. + +## Alternatives considered + +`removeAll(where:)` takes a closure with `true` for elements to remove. +`filter` takes a closure with elements to keep. In both cases, `true` is the +"active" case, so likely to be what the user wants without having to apply a +negation. The naming of `filter` is unfortunately ambiguous as to whether it's +a removing or keeping operation, but re-considering that is outside the scope +of this proposal. + +Several collection methods in the standard library (such as `index(where:)`) +have an equivalent for collections of `Equatable` elements. A similar addition +could be made that removes every element equal to a given value. This could +easily be done as a further additive proposal later. + +The initial proposal of this feature named it `remove(where:)`. During review, +it was agreed that this was unnecessarily ambiguous about whether all the +matching elements should be removed or just the first, and so the method was +renamed to `removeAll(where:)`. + diff --git a/proposals/0198-playground-quicklook-api-revamp.md b/proposals/0198-playground-quicklook-api-revamp.md new file mode 100644 index 0000000000..7b6608d542 --- /dev/null +++ b/proposals/0198-playground-quicklook-api-revamp.md @@ -0,0 +1,426 @@ +# Playground QuickLook API Revamp # + +* Proposal: [SE-0198](0198-playground-quicklook-api-revamp.md) +* Author: [Connor Wakamo](https://github.com/cwakamo) +* Implementation: Swift 4.1 deprecation ([apple/swift#13911](https://github.com/apple/swift/pull/13911)], introduction of new protocol ([apple/swift-xcode-playground-support#21](https://github.com/apple/swift-xcode-playground-support/pull/21)), Swift 5 removal + shim library ([apple/swift#14252](https://github.com/apple/swift/pull/14252), [apple/swift-corelibs-foundation#1415](https://github.com/apple/swift-corelibs-foundation/pull/1415), [apple/swift-xcode-playground-support#20](https://github.com/apple/swift-xcode-playground-support/pull/20)) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift/) +* Review thread: [SE-0198 review](https://forums.swift.org/t/se-0198-playground-quicklook-api-revamp/9448/16) +* Status: **Implemented (Swift 4.1)** + +## Introduction ## + +The standard library currently includes API which allows a type to customize its +description in Xcode playgrounds and Swift Playgrounds. This API takes the +form of the `PlaygroundQuickLook` enum which enumerates types which are +supported for quick looks, and the `CustomPlaygroundQuickLookable` protocol +which allows a type to return a custom `PlaygroundQuickLook` value for an +instance. + +This is brittle, and to avoid dependency inversions, many of the cases are typed +as taking `Any` instead of a more appropriate type. This proposal suggests that +we deprecate `PlaygroundQuickLook` and `CustomPlaygroundQuickLookable` in Swift +4.1 so they can be removed entirely in Swift 5, preventing them from being +included in the standard library's stable ABI. To maintain compatibility with +older playgrounds, the deprecated symbols will be present in a temporary +compatibility shim library which will be automatically imported in playground +contexts. (This will represent an intentional source break for projects, +packages, and other non-playground Swift code which use `PlaygroundQuickLook` or +`CustomPlaygroundQuickLookable` when they switch to the Swift 5.0 compiler, even +in the compatibility modes.) + +Since it is still useful to allow types to provide alternate descriptions for +playgrounds, we propose to add a new protocol to the PlaygroundSupport framework +which allows types to do just that. (PlaygroundSupport is a framework delivered +by the [swift-xcode-playground-support project](https://github.com/apple/swift-xcode-playground-support) +which provides API specific to working in the playgrounds environment). The new +`CustomPlaygroundDisplayConvertible` protocol would allow instances to return an +alternate object or value (as an `Any`) which would serve as their +description. The PlaygroundLogger framework, also part of +swift-xcode-playground-support, will be updated to understand this protocol. + +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/proposal-revamp-the-playground-quicklook-apis/7392) + +## Motivation ## + +The `PlaygroundQuickLook` enum which currently exists in the standard library is +substandard: + +```swift +public enum PlaygroundQuickLook { + case text(String) + case int(Int64) + case uInt(UInt64) + case float(Float32) + case double(Float64) + case image(Any) + case sound(Any) + case color(Any) + case bezierPath(Any) + case attributedString(Any) + case rectangle(Float64, Float64, Float64, Float64) + case point(Float64, Float64) + case size(Float64, Float64) + case bool(Bool) + case range(Int64, Int64) + case view(Any) + case sprite(Any) + case url(String) + case _raw([UInt8], String) +} +``` + +The names of these enum cases do not necessarily match current Swift naming +conventions (e.g. `uInt`), and many cases are typed as `Any` to avoid dependency +inversions between the standard library and higher-level frameworks like +Foundation and AppKit or UIKit. It also contains cases which the +PlaygroundLogger framework does not understand (e.g. `sound`), and this listing +of cases introduces revlock between PlaygroundLogger and the standard library +that makes it challenging to introduce support for new types of quick looks. + +Values of this enum are provided to the PlaygroundLogger framework by types via +conformances to the `CustomPlaygroundQuickLookable` protocol: + +```swift +public protocol CustomPlaygroundQuickLookable { + var customPlaygroundQuickLook: PlaygroundQuickLook { get } +} +``` + +This protocol itself is not problematic, but if `PlaygroundQuickLook` is being +removed, then it needs to be removed as well. Additionally, there is a companion +underscored protocol which should be removed as well: + +```swift +public protocol _DefaultCustomPlaygroundQuickLookable { + var _defaultCustomPlaygroundQuickLook: PlaygroundQuickLook { get } +} +``` + +## Proposed solution ## + +To solve this issue, we propose the following changes: + + - Introduce a new `CustomPlaygroundDisplayConvertible` protocol in + PlaygroundSupport in Swift 4.1 to allow types to provide an alternate + description for playground logging + - Deprecate `PlaygroundQuickLook` and `CustomPlaygroundQuickLookable` in Swift + 4.1, suggesting users use `CustomPlaygroundDisplayConvertible` instead + - Remove `PlaygroundQuickLook` and `CustomPlaygroundQuickLookable` from the + standard library in Swift 5.0 + - Provide an automatically-imported shim library for the playgrounds context + to provide the deprecated instances of `PlaygroundQuickLook` and + `CustomPlaygroundQuickLookable` for pre-Swift 5 playgrounds + +## Detailed design ## + +To provide a more flexible API, we propose deprecating and ultimately removing +the `PlaygroundQuickLook` enum and `CustomPlaygroundQuickLookable` protocol in +favor of a simpler design. Instead, we propose introducing a protocol which just +provides the ability to return an `Any` that serves as a stand-in for the +instance being logged: + +```swift +/// A type that supplies a custom description for playground logging. +/// +/// All types have a default description for playgrounds. This protocol +/// allows types to provide custom descriptions which are then logged in +/// place of the original instance. +/// +/// Playground logging can generate, at a minimum, a structured description +/// of any type. Playground logging is also capable of generating a richer, +/// more specialized description of core types -- for instance, the contents +/// of a `String` are logged, as are the components of an `NSColor` or +/// `UIColor`. +/// +/// The current playground logging implementation logs specialized +/// descriptions of at least the following types: +/// +/// - `String` and `NSString` +/// - `Int` and `UInt` (including the sized variants) +/// - `Float` and `Double` +/// - `Bool` +/// - `Date` and `NSDate` +/// - `NSAttributedString` +/// - `NSNumber` +/// - `NSRange` +/// - `URL` and `NSURL` +/// - `CGPoint`, `CGSize`, and `CGRect` +/// - `NSColor`, `UIColor`, `CGColor`, and `CIColor` +/// - `NSImage`, `UIImage`, `CGImage`, and `CIImage` +/// - `NSBezierPath` and `UIBezierPath` +/// - `NSView` and `UIView` +/// +/// Playground logging may also be able to support specialized descriptions +/// of other types. +/// +/// Implementors of `CustomPlaygroundDisplayConvertible` may return a value of +/// one of the above types to also receive a specialized log description. +/// Implementors may also return any other type, and playground logging will +/// generated structured logging for the returned value. +public protocol CustomPlaygroundDisplayConvertible { + /// Returns the custom playground description for this instance. + /// + /// If this type has value semantics, the instance returned should be + /// unaffected by subsequent mutations if possible. + var playgroundDescription: Any { get } +} +``` + +Additionally, instead of placing this protocol in the standard library, we +propose placing this protocol in the PlaygroundSupport framework, as it is only +of interest in the playgrounds environment. Should demand warrant it, a future +proposal could suggest lowering this protocol into the standard library. + +If this proposal is accepted, then code like the following: + +```swift +extension MyStruct: CustomPlaygroundQuickLookable { + var customPlaygroundQuickLook: PlaygroundQuickLook { + return .text("A description of this MyStruct instance") + } +} +``` + +would be replaced with something like the following: + +```swift +extension MyStruct: CustomPlaygroundDisplayConvertible { + var playgroundDescription: Any { + return "A description of this MyStruct instance" + } +} +``` + +This proposal also allows types which wish to be represented structurally +(like an array or dictionary) to return a type which is logged structurally +instead of requiring an implementation of the `CustomReflectable` protocol: + +```swift +extension MyStruct: CustomPlaygroundDisplayConvertible { + var playgroundDescription: Any { + return [1, 2, 3] + } +} +``` + +This is an enhancement over the existing `CustomPlaygroundQuickLookable` +protocol, which only supported returning opaque, quick lookable values for +playground logging. + +Implementations of `CustomPlaygroundDisplayConvertible` may potentially chain +from one to another. For instance, with: + +```swift +extension MyStruct: CustomPlaygroundDisplayConvertible { + var playgroundDescription: Any { + return "MyStruct description for playgrounds" + } +} + +extension MyOtherStruct: CustomPlaygroundDisplayConvertible { + var playgroundDescription: Any { + return MyStruct() + } +} +``` + +Playground logging for `MyOtherStruct` would generate the string "MyStruct +description for playgrounds" rather than the structural view of `MyStruct`. It +is legal, however, for playground logging implementations to cap chaining to a +reasonable limit to guard against infinite recursion. + +## Source compatibility ## + +This proposal is explicitly suggesting that we make a source-breaking change in +Swift 5 to remove `PlaygroundQuickLook`, `CustomPlaygroundQuickLookable`, and +`_DefaultCustomPlaygroundQuickLookable`. Looking at a GitHub search, there are +fewer than 900 references to `CustomPlaygroundQuickLookable` in Swift source +code; from a cursory glance, many of these are duplicates, from forks of the +Swift repo itself (i.e. the definition of `CustomPlaygroundQuickLookable` in +the standard library), or are clearly implemented using pre-Swift 3 names of the +enum cases in `PlaygroundQuickLook`. (As a point of comparison, there are over +185,000 references to `CustomStringConvertible` in Swift code on GitHub, and +over 145,000 references to `CustomDebugStringConvertible`, so +`CustomPlaygroundQuickLookable` is clearly used many orders of magnitude less +than those protocols.) Furthermore, it does not appear that any projects +currently in the source compatibility suite use these types. + +However, to mitigate the impact of this change, we propose to provide a limited +source compatibility shim for the playgrounds context. This will be delivered as +part of the swift-xcode-playground-support project as a library containing the +deprecated `PlaygroundQuickLook` and `CustomPlaygroundQuickLookable` protocols. +This library would be imported automatically in playgrounds. This source +compatibility shim would not be available outside of playgrounds, so any +projects, packages, or other Swift code would be intentionally broken by this +change when upgrading to the Swift 5.0 compiler, even when compiling in a +compatibility mode. + +Due to the limited usage of these protocols, and the potential challenge in +migration, this proposal does not include any proposed migrator changes to +support the replacement of `CustomPlaygroundQuickLookable` with +`CustomPlaygroundDisplayConvertible`. Instead, we intend for Swift 4.1 to be a +deprecation period for these APIs, allowing any code bases which implement +`CustomPlaygroundQuickLookable` to manually switch to the new protocol. While +this migration may not be trivial programmatically, it should -- in most cases -- +be fairly trivial for someone to hand-migrate to +`CustomPlaygroundDisplayConvertible`. During the deprecation period, the +PlaygroundLogger framework will continue to honor implementations of +`CustomPlaygroundQuickLookable`, though it will prefer implementations of +`CustomPlaygroundDisplayConvertible` if both are present on a given type. + +## Effect on ABI stability ## + +This proposal affects ABI stability as it removes an enum and a pair of +protocols from the standard library. Since this proposal proposes adding +`CustomPlaygroundDisplayConvertible` to PlaygroundSupport instead of the +standard library, there is no impact of ABI stability from the new protocol, as +PlaygroundSupport does not need to maintain a stable ABI, as its clients -- +playgrounds -- are always recompiled from source. + +Since playgrounds are always compiled from source, the temporary shim library +does not represent a new ABI guarantee, and it may be removed if the compiler +drops support for the Swift 3 and 4 compatibility modes in a future Swift +release. + +Removing `PlaygroundQuickLook` from the standard library also potentially allows +us to remove a handful of runtime entry points which were included to support +the `PlaygroundQuickLook(reflecting:)` API. + +## Effect on API resilience ## + +This proposal does not impact API resilience. + +## Alternatives considered ## + +### Do nothing ### + +One valid alternative to this proposal is to do nothing: we could continue to +live with the existing enum and protocol. As noted above, these are fairly poor, +and do not serve the needs of playgrounds particularly well. Since this is our +last chance to remove them prior to ABI stability, we believe that doing nothing +is not an acceptable alternative. + +### Provide type-specific protocols ### + +Another alternative we considered was to provide type-specific protocols for +providing playground descriptions. We would introduce new protocols like +`CustomNSColorConvertible`, `CustomNSAttributedStringConvertible`, etc. which +would allow types to provide descriptions as each of the opaquely-loggable +types supported by PlaygroundLogger. + +This alternative was rejected as it would balloon the API surface for +playgrounds, and it also would not provide a good way to select a preferred +description. (That is, what would PlaygroundLogger select as the +description of an instance if it implemented both `CustomNSColorConvertible` +*and* `CustomNSAttributedStringConvertible`?) + +### Implement `CustomPlaygroundDisplayConvertible` in the standard library ### + +As an alternative to implementing `CustomPlaygroundDisplayConvertible` in +PlaygroundSupport, we could implement it in the standard library. This would +make it available in all contexts (i.e. in projects and packages, not just in +playgrounds), but this protocol is not particularly useful outside of the +playground context, so this proposal elects not to place +`CustomPlaygroundDisplayConvertible` in the standard library. + +Additionally, it should be a source-compatible change to move this protocol to +the standard library in a future Swift version should that be desirable. Since +playgrounds are always compiled from source, the fact that this would be an ABI +change for PlaygroundSupport does not matter, and a compatibility typealias +could be provided in PlaygroundSupport to maintain compatibility with code which +explicitly qualified the name of the `CustomPlaygroundDisplayConvertible` +protocol. + +### Have `CustomPlaygroundDisplayConvertible` return something other than `Any` ### + +One minor alternative considered was to have +`CustomPlaygroundDisplayConvertible` return a value with a more specific type +than `Any`. For example: + +```swift +protocol CustomPlaygroundDisplayConvertible { + var playgroundDescription: CustomPlaygroundDisplayConvertible { get } +} +``` + +or: + +```swift +protocol PlaygroundDescription {} + +protocol CustomPlaygroundDisplayConvertible { + var playgroundDescription: PlaygroundDescription { get } +} +``` + +In both cases, core types which the playground logger supports would conform to +the appropriate protocol such that they could be returned from implementations +of `playgroundDescription`. + +The benefit to this approach is that it is more self-documenting than the +approach proposed in this document, as a user can look up all of the types which +conform to a particular protocol to know what the playground logger understands. +However, this approach has a number of pitfalls, largely because it's +intentional that the proposal uses `Any` instead of a more-constrained protocol. +It should be possible to return anything as the stand-in for an instance, +including values without opaque playground quick look views, so that it's easier +to construct an alternate structured view of a type (without having to override +the more complex `CustomReflectable` protocol). Furthermore, by making the API +in the library use a general type like `Any`, this proposal prevents revlock +from occurring between IDEs and the libraries, as the IDE's playground logger +can implement support for opaque logging of new types without requiring library +changes. (And IDEs can opt to support a subset of types if they prefer, whereas +if the libraries promised support an IDE would effectively be compelled to +provide it.) + +### Have `CustomPlaygroundDisplayConvertible` return an `Any?` instead of an `Any` ### + +One alternative considered was to have `CustomPlaygroundDisplayConvertible` +return an `Any?` instead of an `Any`. This would permit individual instances to +opt-out of a custom playground description by returning nil instead of a +concrete value or object. + +Although that capability is no longer present, in most cases implementors of +`CustomPlaygroundDisplayConvertible` may return a custom description which +closely mirrors their default description. One big exception to this are classes +which are considered core types, such as `NSView` and `UIView`, as one level of +subclass may wish to customize its description while deeper level may wish to +use the default description (which is currently a rendered image of the view). +This proposal does not permit that; the second-level subclass must return a +custom description one way or another, and due to the chaining nature of +`CustomPlaygroundDisplayConvertible` implementations, it cannot return `self` +and have that reliably indicate to the playground logger implementation that +that means "don't use a custom description". + +This issue seems to be limited enough that it should not tarnish the API design +as a whole. Returning `Any` and not `Any?` is easier to understand, so this +proposal opts to do that. Should this be a larger issue than anticipated, a +future proposal could introduce a struct like `DefaultPlaygroundDescription` +which the playground logger would understand to mean "don't check for a +`CustomPlaygroundDisplayConvertible` conformance on the wrapped value". + +### Alternate Names for `CustomPlaygroundDisplayConvertible` ### + +Finally, as this introduces a new protocol, there are other possible names: + +- `CustomPlaygroundRepresentable` +- `CustomPlaygroundConvertible` +- `CustomPlaygroundPreviewConvertible` +- `CustomPlaygroundQuickLookConvertible` +- `CustomPlaygroundValuePresentationConvertible` +- `CustomPlaygroundPresentationConvertible` + +`CustomPlaygroundRepresentable` was rejected as it does not match the naming +convention established by +`CustomStringConvertible`/`CustomDebugStringConvertible`. +`CustomPlaygroundConvertible` was rejected as not being specific enough -- types +conforming to this protocol are not themselves convertible to playgrounds, but +are instead custom convertible for playground display. +`CustomPlaygroundPreviewConvertible` is very similar to +`CustomPlaygroundDisplayConvertible`, but implies more about the presentation +than is appropriate as a playground environment is free to display it any way it +wants, not just as a "preview". `CustomPlaygroundQuickLookConvertible` was +rejected as it potentially invokes the to-be-removed `PlaygroundQuickLook` enum. +`CustomPlaygroundValuePresentationConvertible` and +`CustomPlaygroundPresentationConvertible` were rejected as too long of names for +the protocol. diff --git a/proposals/0199-bool-toggle.md b/proposals/0199-bool-toggle.md new file mode 100644 index 0000000000..b9a4c22384 --- /dev/null +++ b/proposals/0199-bool-toggle.md @@ -0,0 +1,80 @@ +# Adding `toggle` to `Bool` + +* Proposal: [SE-0199](0199-bool-toggle.md) +* Author: [Chris Eidhof](https://github.com/chriseidhof) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift/) +* Status: **Implemented (Swift 4.2)** +* Decision notes: [Rationale](https://forums.swift.org/t/accepted-se-199-add-toggle-to-bool/10681) +* Implementation: [apple/swift#14586](https://github.com/apple/swift/pull/14586) +* Review thread: [Swift evolution forum](https://forums.swift.org/t/se-0199-adding-toggle-method-to-bool/) + + +## Introduction + +I propose adding a `mutating func toggle` to `Bool`. It toggles the `Bool`. + +- Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/adding-toggle-to-bool/7414) +- Swift forums thread: [pitch: adding toggle to Bool](https://forums.swift.org/t/pitch-adding-toggle-to-bool/7414) + +## Motivation + +For `Bool` variables, it is common to want to toggle the state of the variable. In larger (nested) structs, the duplication involved can become especially annoying: + +```swift +myVar.prop1.prop2.enabled = !myVar.prop1.prop2.enabled +``` + +It's also easy to make a mistake in the code above if there are multiple `Bool` vars. + +## Proposed solution + +Add a method `toggle` on `Bool`: + +```swift +extension Bool { + /// Equivalent to `someBool = !someBool` + /// + /// Useful when operating on long chains: + /// +  /// myVar.prop1.prop2.enabled.toggle() +  mutating func toggle() { + self = !self + } +} +``` + +This allows us to write the example above without duplication: + +```swift +myVar.prop1.prop2.enabled.toggle() +``` + +`!` and `toggle()` mirror the API design for `-` and `negate()`. (Thanks to Xiaodi Wu for pointing this out). + +## Detailed design + +N/A + +## Source compatibility + +This is strictly additive. + +## Effect on ABI stability + +N/A + +## Effect on API resilience + +N/A + +## Alternatives considered + +Other names could be: + +- `invert` +- `negate` +- `flip` + +From the brief discussion on SE, it seems like `toggle` is the clear winner. + +Some people also suggested adding a non-mutating variant (in other words, a method with the same semantics as the prefix `!` operator), but that's out of scope for this proposal, and in line with commonly rejected proposals. diff --git a/proposals/0200-raw-string-escaping.md b/proposals/0200-raw-string-escaping.md new file mode 100644 index 0000000000..b08508ca0c --- /dev/null +++ b/proposals/0200-raw-string-escaping.md @@ -0,0 +1,483 @@ +# Enhancing String Literals Delimiters to Support Raw Text + +* Proposal: [SE-0200](0200-raw-string-escaping.md) +* Authors: [John Holdsworth](https://github.com/johnno1962), [Becca Royal-Gordon](https://github.com/beccadax), [Erica Sadun](https://github.com/erica) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/102b2f2770f0dab29f254a254063847388647a4a/proposals/0200-raw-string-escaping.md) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#17668](https://github.com/apple/swift/pull/17668) +* Bugs: [SR-6362](https://bugs.swift.org/browse/SR-6362) +* Review: [Discussion thread](https://forums.swift.org/t/se-0200-enhancing-string-literals-delimiters-to-support-raw-text/15420), [Announcement thread](https://forums.swift.org/t/accepted-se-0200-enhancing-string-literals-delimiters-to-support-raw-text/15822/2) + +## Introduction + +Like many computer languages, Swift uses an escape character (`\`) to create a special interpretation of subsequent characters within a string literal. Escape character sequences represent a set of predefined, non-printing characters as well as string delimiters (the double quote), the escape character (the backslash itself), and (uniquely in Swift) to allow in-string expression interpolation. + +Escape characters provide useful and necessary capabilities but strings containing many escape sequences are difficult to read. Other languages have solved this problem by providing an alternate "raw" string literal syntax which does not process escape sequences. As the name suggests, raw string literals allow you to use "raw" text, incorporating backslashes and double quotes without escaping. + +We propose to alter Swift's string literal design to do the same, using a new design which we believe fits Swift's simple and clean syntax. This design supports both single-line and multi-line string literals, and can contain any content whatsoever. + +This proposal has been extensively revised based on the Core Team feedback for [SE-0200](https://forums.swift.org/t/returned-for-revision-se-0200-raw-mode-string-literals/11630). It was discussed on the [Swift online forums](https://forums.swift.org/t/pure-bikeshedding-raw-strings-why-yes-again/13866). + +### Discussion + +Raw strings and their design have been discussed in the following Evolution forum threads: + +* [\[Pitch\] Raw mode string literals](https://forums.swift.org/t/pitch-raw-mode-string-literals/7120/5) +* [SE-0200: "Raw" mode string literals (Review thread)](https://forums.swift.org/t/se-0200-raw-mode-string-literals/11048) +* [\[Returned for revision\] SE-0200: "Raw" mode string literals +](https://forums.swift.org/t/returned-for-revision-se-0200-raw-mode-string-literals/11630) +* [\[Pitch v2\]: Raw strings and SE-0200](https://forums.swift.org/t/pitch-v2-raw-strings-and-se-0200/11660) +* [Pure Bikeshedding: Raw Strings (why yes, again!)](https://forums.swift.org/t/pure-bikeshedding-raw-strings-why-yes-again/13866) + +## Background + +Modern programming languages use two approaches to represent string literals. + +* A **conventional string literal** is exactly what you use in Swift today. It allows you to use escape sequences like `\\` and `\"` and `\u{n}` to express backslashes, quotes, and unicode scalars, among other special character sequences. +* A **raw string literal** ignores escape sequences. It allows you to paste raw code. In a raw string literal the sequence `\\\n` represents three backslashes followed by the letter "n", not a backslash followed by a line feed. + +This proposal uses the following terms. + +* **String literals** represent a sequence of characters in source. +* **String delimiters** establish the boundaries at the start and end of a character sequence. Swift's string delimiter is `"`, the double quote (U+0022). +* **Escape characters** create a special interpretation of one or more subsequent characters within a string literal. Swift's escape character is `\`, the backslash (U+005C). +* **Escape character sequences** (shortened to _escape sequence_) represent special characters. In the current version of Swift, the backslash escape character tells the compiler that a sequence should combine to produce one of these special characters. + +## Motivation + +Raw strings support non-trivial content which belongs directly in source code -- not in an external file -- but cannot be satisfactorily maintained or read in escaped form. + +Hand-escaped strings require time and effort to transform source material to an escaped form. It is difficult to validate the process to ensure the escaped form properly represents the original text. This task is also hard to automate as it may not pick up intended nuances, such as recognizing embedded dialog quotes. + +Escaping actively interferes with inspection. Developers should be able to inspect and modify raw strings in-place without removing that text from source code. This is especially important when working with precise content such as code sources and regular expressions. + +Backslash escapes are common in other languages and formats, from JSON to LaTeX to Javascript to regular expressions. Embedding these in a string literal currently requires doubling-up escapes, or even quadrupling if the source is pre-escaped. Pre-escaped source should be maintained exactly as presented so it can be used, for example, when contacting web-based services. + +Importantly, raw strings are transportable. They allow developers to cut and paste content both from and to the literal string. This allows testing, reconfiguration, and adaption of raw content without the hurdles escaping and unescaping that limit development. + +In short, a good raw string feature should let users embed any valid Unicode text snippet in a Swift string literal merely by surrounding it with appropriate delimiters, without altering the content itself. + +### Examples + +Raw string literals may include characters normally used for escaping (such as the backslash `\` character) and characters normally requiring escaping (such as a double quote `"`). For example, consider the following multiline string. It represents code to be output at some point in the program execution: + +``` +let separators = """ + public static var newlineSeparators: Set = [ + // [Zl]: 'Separator, Line' + "\u{2028}", // LINE SEPARATOR + + // [Zp]: 'Separator, Paragraph' + "\u{2029}", // PARAGRAPH SEPARATOR + ] + """ +``` + +Unescaped backslash literals cause the unicode escape sequences to be evaluated and replaced in-string. This produces the following result: + +``` +public static var newlineSeparators: Set = [ + // [Zl]: 'Separator, Line' + " +", // LINE SEPARATOR + + // [Zp]: 'Separator, Paragraph' + " +", // PARAGRAPH SEPARATOR +] +``` + +To preserve the intended text, each backslash must be escaped, for example `\\u{2029}`. This is a relatively minor edit but if the code is being copied in and out of the source to permit testing and modification, then each hand-escaped cycle introduces the potential for error. + +Single-line string literals may similarly be peppered with backslashes to preserve their original intent, as in the following examples. + +``` +// Quoted Text +let quote = "Alice: "How long is forever?" White Rabbit: "Sometimes, just one second."" +let quote = "Alice: \"How long is forever?\" White Rabbit: \"Sometimes, just one second.\"" + +// and + +// Regular Expression +let ucCaseCheck = "enum\s+.+\{.*case\s+[:upper:]" +let ucCaseCheck = "enum\\s+.+\\{.*case\\s+[:upper:]" +``` + +Escaping hinders readability and interferes with inspection, especially in the latter example, where the content contains secondary escape sequences. Using a raw form ensures the expression can be read and updated as needed in the form that will be passed by the literal string. + +### Candidates + +A good candidate for using raw strings is non-trivial and is burdened by escaping because it: + +* **Is obscured by escaping.** Escaping actively harms code review and validation. +* **Is already escaped.** Escaped material should not be pre-interpreted by the compiler. +* **Requires easy transport between source and code in both directions**, whether for testing or just updating source. + +The following example is a poor case for using a raw string: + +``` +let path = "C:\\AUTOEXEC.BAT" +``` + +The example is trivial and the escaping is not burdensome. It's unlikely that the string contents will require any further modification or reuse in a raw form. + +### Utility + +Raw strings are most valuable for the following scenarios. + +**Metaprogramming**: Use cases include code-producing-code. This incorporates utility programming and building test cases. Apps may generate color scheme type extensions (in Swift, ObjC, for SpriteKit/SceneKit, literals, etc) or date formatters, perform language-specific escaping, create markup, and more. + +Escaping complicates copying and pasting from working code into your source and back. When you're talking about code, and using code, having that code be formatted as an easily updated raw string is especially valuable. + +Examples of popular apps that perform these tasks include Kite Compositor and PaintCode. Any utility app that outputs code would benefit in some form. + +**Regular expressions**: While we have bigger plans for regular expressions in the future, we think they will be a primary use case for raw strings in the short term, and will continue to have an important place in regex usage in the long term. + +Even if we introduce native regular expressions in a future version of Swift, users will still sometimes have to write regular expressions intended for use in other systems. For instance, if you need to send a regex to a server, or embed it in Javascript, or put it in a SQL query, or construct an `NSRegularExpression` and pass it to an existing API which uses that type, you'll still express that regular expression as a string literal, not a native regex. And when you do, raw strings will make that much easier. + +A raw string feature would thus help with all regular expressions now and some regular expressions in the future. And if the native regular expression feature involves some form of quoting and escaping, it can follow the by-then-established precedent of this proposal to support "raw regexes". + +**Pedagogy**: Not all Swift learning takes place in the playground and not all code described in Swift source files use the Swift programming language. + +Code snippets extend beyond playground-only solutions for many applications. Students may be presented with source code, which may be explained in-context within an application or used to populate text edit areas as a starting point for learning. + +Removing escaped snippets to external files makes code review harder. Escaping (or re-escaping) code is a tedious process, which is hard to inspect and validate. + +**Data Formats and Domain Specific Languages**: It's useful to incorporate short sections of unescaped or pre-escaped JSON and XML. It may be impractical to use external files and databases for each inclusion. Doing so impacts inspection, maintenance, and updating. + +**Windows paths**: Windows uses backslashes to delineate descent through a directory tree: e.g., `C:\Windows\All Users\Application Data`. The more complex the path, the more intrusive the escapes. + +## Initial Proposal + +"Raw-mode" strings were first discussed during the [SE-0168 Multi-Line String literals](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0168-multi-line-string-literals.md) review and postponed for later consideration. This proposal focuses on raw strings to allow the entry of single and multi-line string literals. + +The first iteration of [SE-0200](https://github.com/swiftlang/swift-evolution/blob/102b2f2770f0dab29f254a254063847388647a4a/proposals/0200-raw-string-escaping.md) proposed adopting Python's model, using `r"...raw string..."`. The proposal was returned for revision with the [following feedback](https://forums.swift.org/t/returned-for-revision-se-0200-raw-mode-string-literals/11630): + +> During the review discussion, a few issues surfaced with the proposal, including: +> +> The proposed r"..." syntax didn’t fit well with the rest of the language. The most-often-discussed replacement was #raw("..."), but the Core Team felt more discussion (as a pitch) is necessary. +> +> The proposal itself leans heavily on regular expressions as a use case for raw string literals. Several reviewers remarked that the motivation wasn’t strong enough to justify the introduction of new syntax in the language, so a revised proposal will need additional motivating examples in other domains. + +To move forward, the new raw string design must provide a suitable Swift-appropriate syntax that works within the language's culture and conventions. + +## Prior Art + +The following links explore the existing art in other languages. We were inspired by the [Rust raw string RFC discussion](https://github.com/rust-lang/rust/issues/9411) when researching these features. + +| Syntax | Language(s) | Possible in Swift? | Swifty? | +| ----- | --------- | ------------- | -------- | +| `'Hello, world!'` | Bourne shell, Perl, PHP, Ruby, Windows PowerShell | Yes | Yes if Rust-style multiplicity allows incorporating `'` into raw strings. May be too narrow a use-case to burn `'`. | +| `q(Hello, world!)` | [Perl](https://en.wikipedia.org/wiki/String_literal) (alternate) | Maybe (depends on delimiter) | No | +| `%q(Hello, world!)` | Ruby (alternate) | No (`%` is a valid prefix operator) | No | +| `@"Hello, world!"` | [C#](https://msdn.microsoft.com/en-us/library/69ze775t.aspx), F# | Yes (but would be awful for Obj-C switchers) | No | +| `R"(Hello, world!)"` | [C++11](https://en.cppreference.com/w/cpp/language/string_literal) | Yes | No | +| `r"Hello, world!"` | [D](https://tour.dlang.org/tour/en/basics/alias-strings), [Python](http://wiki.c2.com/?RawStrings) | Yes | No | +| `r#"Hello, world!"#` | [Rust](https://doc.rust-lang.org/reference/tokens.html#raw-string-literals) | Yes | Would need to drop the opening `r` and maybe change the delimiter from `#`. | +| `"""hello \' world"""` and `raw"Hello, world!"` | [Scala](https://www.scala-lang.org/files/archive/spec/2.13/13-syntax-summary.html) | No/Yes | No | +| ``` `Hello, world!` ``` | [D](https://tour.dlang.org/tour/en/basics/alias-strings), [Go](https://golang.org/ref/spec), \`...\` | No (conflicts with escaped identifiers) | No, needs Rust multiplicity | +| ``` ``...`` ``` | [Java](http://openjdk.java.net/jeps/326), any number of \` | No (conflicts with escaped identifiers) | Yes | + + +## Design + +We determined that Rust's approach to raw string literals is the best starting point, offering the greatest flexibility in the smallest syntactic footprint. + +In Rust, raw string literals are written as `r"..."`. To embed double-quotes in a Rust raw string literal, you add one or more pound signs before the opening quote, and put a matching number of pound signs after the closing quote: `r#"..."..."#`, `r##"...#"..."##`, etc. Rust developers assured us that even one pound sign was unusual and more than one almost never needed but it's nice to have the flexibility in the rare cases where you need it. + +### Swiftifying Rust's Design + +Rust's design distinguishes between conventional and raw string literals. It also includes an asymmetric `r` off its leading edge. We found these distinctions unnecessary and the `r` aesthetically displeasing. Instead, our design powers up a conventional Swift `String` literal and in doing so, allows you to access features normally associated with raw literals. + +In this design, there is no separate "raw" syntax; rather, there is a small extension of the conventional string literal syntax. A conventional string literal is either: + +* a sequence of characters surrounded by double quotation marks ("), or +* a string that spans several lines surrounded by three double quotation marks. + +These are examples of conventional Swift string literals: + +``` +"This is a single line Swift string literal" + +""" + This is a multi line + Swift string literal + """ +``` + +In this form, the revised string design acts exactly like any other string. You use escape character sequences including string interpolation exactly as you would today. A backslash escape character tells the compiler that a sequence should be interpolated, interpreted as an escaped character, or represent a unicode scalar. + +Swift's escape sequences include: + +* The special characters `\0` (null character), `\\` (backslash), `\t` (horizontal tab), `\n` (line feed), `\r` (carriage return), `\"` (double quotation mark) and `\'` (single quotation mark) +* Arbitrary Unicode scalars, written as `\u{n}`, where *n* is a 1–8 digit hexadecimal number with a value equal to a valid Unicode code point +* Interpolated expressions, introduced by `\(` and terminated by `)` + +### Expanding Delimiters + +Our design adds customizable string delimiters. You may pad a string literal with one or more `#` (pound, Number Sign, U+0023) characters: + +``` +"This is a Swift string literal" + +#"This is also a Swift string literal"# + +####"So is this"#### +``` + +The number of pound signs at the start of the string (in these examples, zero, one, and four) must match the number of pound signs at the end of the string. `"This"`, `#"This"#`, and `##"This"##` represent identical string values: + +``` +static-string-literal -> " quoted-text " | + """ multiline-quoted-text """ | + # static-string-literal # +``` + +Any instance of the delimiter which is not followed by the appropriate number of pound signs is treated as literal string contents, rather than as the end of the string literal. That is, the leading pound signs *change the string's delimiter* from `"` to `"#` (or `"##`', etc.). A plain `"` without pound signs after it is just a double-quote character inside the string. + +``` +#"She said, "This is dialog!""# +// Equivalent to "She said, \"This is dialog!\"" +``` + +If you do add a backslash, it is interpreted as an extra character. This string literal includes the backslash and both double quote marks inside the string delimiters (`#"` and `"#`): + +``` +#"A \"quote"."# +``` + +When you need to include `#"` (pound-quote) or `"#` (quote-pound) in your character sequence, adjust the number of delimiting pound signs. This need should be rare. + +### Customized Escape Delimiters + +This design uses an *escape delimiter*, that is a sequence of one or more characters to indicate the beginning of an escape character sequence, rather than a single escape character. Like Swift today, the escape delimiter begins with a backslash (Reverse Solidus, U+005C), but it is now followed by zero or more pound signs +(Number Sign, U+0023). An escape delimiter in a string literal must match the number of pound signs used to delimit either end of the string. + +Here is the degenerate case. It is a normal string with no pound signs. The escape delimiter therefore needs no pound signs and a single backslash is sufficient to establish the escape character sequence: + +``` +"This string has an \(interpolated) item" +``` + +Strings using custom boundary delimiters mirror their pound sign(s) after the leading backslash, as in these examples which produce identical results to the preceding string literal: + +``` +#"This string has an \#(interpolated) item"# + +####"This string has an \####(interpolated) item"#### +``` + +The escape delimiter customization matches the string. Any backslash that is not followed by the correct number of pound signs is treated as raw text. It is not an escape: + +``` +#"This is not \(interpolated)"# +``` + +| String Start Delimiter | Escape Delimiter | String End Delimiter | +| ---------------------- | ---------------- | -------------------- | +| `"` | `\` | `"` | +| `#"` | `\#` | `"#` | +| `##"` | `\##` | `"##` | +| `######"` | `\######` | `"######` | + +Inside the string, any backslash that is followed by too few pound signs (like `\#` in a `##""##` string) is not an escape delimiter. It is just that exact string. Any backslash followed by too many pound signs (like `\##` in a `#""#` string) creates an invalid escape sequence because it is an escape delimiter followed by one or more pound signs. + +This escaping rule supports several important features: it provides for raw string support, *and* string interpolation. We feel this is a huge win, especially for code generation applications. We believe this conceptual leap of elegance simplifies all our previous design workarounds and collapses them into one general solution. + +This design retains Rust-inspired custom delimiters, offers all the features of "raw" strings, introduces raw string interpolation, and does this _all_ without adding a new special-purpose string type to Swift. + +Yes, this approach requires work: + +* You must use pound signs for any raw string. +* You must use a more cumbersome interpolation sequence for raw strings than conventional strings. + +Hopefully the tradeoffs are worth it in terms of added expressibility and the resulting design is sufficiently elegant to pass muster. + +### Updated Escape Character Sequences Reference + +Swift string literals may include the following special character sequences. Swift's escape delimiter begins with a backslash (Reverse Solidus, U+005C), and is followed by zero or more pound signs (Number Sign, U+0023). An escape delimiter in a string literal must match the number of pound signs used to delimit either end of the string. + +| Sequence | Escape Characters | Result | +|----------|------------|--------| +| Escape Delimiter + `0` | Digit Zero (U+0030)| Boundary Neutral / Null character U+0000 | +| Escape Delimiter + `\` | Reverse Solidus (U+005C) | Punctuation / Backslash U+005C | +| Escape Delimiter + `t` | Latin Small Letter T (U+0074) | Character Tabulation / Horizontal Tab U+0009 | +| Escape Delimiter + `n` | Latin Small Letter N (U+006E) | Paragraph Separator / Line Feed U+000A | +| Escape Delimiter + `r` | Latin Small Letter R (U+0072) | Paragraph Separator / Carriage Return U+000D | +| Escape Delimiter + `"` | Quotation Mark (U+0022) | Punctuation / Quotation Mark U+0022 | +| Escape Delimiter + `'` | Apostrophe (U+0027) | Punctuation / Apostrophe (Single Quote) U+0027 | +| Escape Delimiter + `u{n}` | Latin Small Letter U (U+0055), Left Curly Bracket (U+007B), Right Curly Bracket (U+007D) | An arbitrary Unicode scalar, where *n* is a 1–8 digit hexadecimal number with a value equal to a valid Unicode code point | +| Escape Delimiter + `(...)` | Left Parenthesis (U+0028), Right Parenthesis (U+0029) | An interpolated expression | + + +## Examples + +Consider the text `\#1`: + +``` +// What you type today +"\\#1" // escape delimiter + backslash + # + 1 + +// What you type using the new system +"\\#1" // the same + +// What you type with a single pound string delimiter +#"\#\#1"# // escape delimiter + backslash + # + 1 + +// Or you can adjust the string delimiter for a raw string: +##"\#1"## // backslash + # + 1, no escape sequence +``` + +Adjusting string delimiters allows you to eliminate escape sequences to present text as intended for use: + +``` +#"c:\windows\system32"# // vs. "c:\\windows\\system32" +#"\d{3) \d{3} \d{4}"# // vs "\\d{3) \\d{3} \\d{4}" + +#"a string with "double quotes" in it"# + +##"a string that needs "# in it"## + +#""" + a string with + """ + in it + """# +``` + +The following example terminates with backslash-r-backslash-n, not a carriage return and line feed: + +``` +#"a raw string containing \r\n"# +// vs "a raw string containing \\r\\n" +``` + +The same behavior is extended to multi-line strings: + +``` +#""" + a raw string containing \r\n + """# +``` + +New line escaping works as per [SE-182](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0182-newline-escape-in-strings.md): + +``` +#""" + this backslash and newline will be present in the string \ + this newline will take it's escaped value in this case \# + and the line continues here with only a space joining. + """# +``` + +Custom-delimited strings allow you to incorporate already-escaped text. For example, you can paste static data without having to worry about re-escaping a JSON message + +``` +#""" + [ + { + "id": "12345", + "title": "A title that \"contains\" \\\"" + } + ] + """# +``` + +Without custom delimiters, Swift would silently unescape this content, yielding an invalid JSON message. Even if you did remember to escape, this process would be error-prone and difficult to maintain. + +However, if you wanted to interpolate the value of the "id" field, you could +still do that without having to double-escape the other backslashes: + +``` +#""" + [ + { + "id": "\#(idNumber)", + "title": "A title that \"contains\" \\\"" + } + ] + """# +``` + +It is anticipated raw strings will also work in attributes: +``` +@available(swift, deprecated: 4.2, message: + #"Note: "\r\n" (CR-LF) is normalized to "\n" (LF)"#) +``` + +## Errors + +The compiler errors when an escape delimiter is followed by an unrecognized value to complete an escape sequence. For example, using one-`#`-delimited strings: + +``` +#"printf("%s\n", value_string)"# // no error, no escape sequence +#"printf("%s\#n", value_string)"# // newline escape sequence +#"printf("%s\#x\#n", value_string)"# // error: escape delimiter + x. +``` + +The last example can introduce a fixit by adding another `#` to either side of the string so `\#` is no longer the escape delimiter. However, this eliminates the subsequent line feed (a valid escape sequence) that follows unless that, too, is appropriately updated. + +There are also wrong ways to add interpolated text. These examples are both errors. The escape delimiter in each case (respectively `\` and `\#`) is followed by `#`, forming an invalid escape sequence: + +``` +"This is not \#(correct)" + +#"This is not \##(correct)"# + +``` + +An escape with too many pounds should be an error with a special message and fix-it. The fixit should suggest that `escape-delimiter + #` instances remove the extra pound or add further `#`-signs to each end of the string. + +## Discoverability and Recognition + +There are two questions of developer approach: discoverability ("how do I use raws string in Swift") and recognition ("Why do some strings in Swift start with `#`?" and "Why are there `#` signs after backslashes?"). When presented to developers unfamiliar with the new string syntax, we feel that it isn't overly burdensome to search the web for: + +* "What do #/pound/number/etc signs mean in Swift strings?" +* "How do I use raw strings in Swift?" +* "How do I add quote marks to strings without escaping in Swift?" +* "How do I interpolate in raw Swift strings?" + +## Implementation + +Changes are largely confined to the file lib/Parse/Lexer.cpp. They involve a slight modification to the main lexer loop Lexer::lexImpl() to detect strings that have a custom delimiter/are surrounded by 1 or more `#` characters. When the start of a custom-delimited string is detected Lexer::lexStringLiteral() is called with the delimiter’s length. Targeted changes to Lexer::lexCharacter() and Lexer::getEncodedStringSegment() bypass processing of the escape character `\` if the delimiter length is not equal to the number of `#` characters after the `\`, the degenerate case being delimiter length of 0 which is normal string processing. + +A new field `StringDelimiterLength` in Token.h carries the string escaping mode from the parsing to code generation phases of compilation. + +## Source compatibility + +This is a purely additive change. The syntax proposed is not currently valid Swift. + +## Effect on ABI stability + +None. + +## Effect on API resilience + +None. + +## Alternatives considered + +We evaluated many, *many* designs from other languages and worked through a long thread full of bikeshedding. We will list the most notable rejected designs here, but these are just the tip of the iceberg. + +### Excluding single quotes and backticks + +Single quotes are a common syntax for raw strings in other languages. However, they're also commonly used for character literals (i.e. integer literals containing the value of a Unicode scalar) in other languages. If we use single quotes for raw strings, we cannot use them for character literals or any other future proposal. We see no need to burn single quotes on this feature. + +Similarly, while backticks preserve the meaning of "code voice" and "literal", as you are used to in Markdown, they would conflict with escaped identifiers. + +### Using "raw" and "rawString" + +The original design `r"..."` was rejected in part for not being Swifty, that is, not taking on the look and feel and characteristics of existing parts of the language. Similar approaches like `raw"..."` and `#raw"..."` carry the same issues. Leading text is distracting and competes for attention with the content of the string that follows. And function-like constructs like `#raw("...")` harmonize better with the language syntax, but are even worse for readability. + +### Using user-specified delimiter characters + +We felt user-specified delimiters overly complicated the design space, were harder to discover and use, and were generally un-Swifty. The pound sign is rarely used and a minor burden on the syntax. + +``` +@rawString(delimiter: @) @"Hello"@ // no +``` + +We considered a Perl/Ruby-like approach to arbitrary delimiters, but quickly rejected it. The arbitrary delimiter rules in these languages are complex and have many corner cases; we don't need or want that complexity. + +We also rejected a standalone raw string attribute for being wordy and heavy, especially for short literals. diff --git a/proposals/0201-package-manager-local-dependencies.md b/proposals/0201-package-manager-local-dependencies.md new file mode 100644 index 0000000000..de681db9e5 --- /dev/null +++ b/proposals/0201-package-manager-local-dependencies.md @@ -0,0 +1,74 @@ +# Package Manager Local Dependencies + +* Proposal: [SE-0201](0201-package-manager-local-dependencies.md) +* Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Boris Bügling](https://github.com/neonichu) +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift-package-manager#1583](https://github.com/apple/swift-package-manager/pull/1583) +* Bug: [SR-7433](https://bugs.swift.org/browse/SR-7433) + +## Introduction + +This proposal adds a new API in `PackageDescription` to support declaring +dependency on a package using its path on disk instead of the git URL. + +## Motivation + +There are two primary motivations for this proposal: + +1. Reduce friction in bringing up a set of packages. + + Currently, there is a lot of friction in setting up a new set of interconnected +packages. The packages must be in a git repository and the package author needs +to run the `swift package edit` command for each dependency to develop them in tandem. + +2. Some package authors prefer to keep their packages in a single repository. + This could be because of several reasons, for e.g.: + + * The packages are in very early design phase and are not ready to be published + in their own repository yet. + * The author has a set of loosely related packages which they may or may not + want to publish as a separate repository in future. + +## Proposed solution + +We propose to add the following API in the `PackageDescription` module: + +```swift +extension Package.Dependency { + public static func package(path: String) -> Package.Dependency +} +``` + +* This API will only be available if the tools version of the manifest is + greater than or equal to the Swift version this proposal is implemented in. + +* The value of `path` must be a valid absolute or relative path string. + +* The package at `path` must be a valid Swift package. + +* The local package will be used as-is and the package manager will not try to + perform any git operation on the package. + +* The local package must be declared in the root package or in other local + dependencies. This means, it is not possible to depend on a regular versioned + dependency that declares a local package dependency. + +* A local package dependency will override any regular dependency in the package + graph that has the same package name. + +* A local dependency will not be recorded in the `Package.resolved` file and + if it overrides a dependency in the package graph, any existing entry will be + removed. This is similar to the edit mode behaviour. + +* The package manager will not allow using the edit feature on local dependencies. + +## Impact on existing packages + +None. + +## Alternatives considered + +We considered piggybacking this on the multi-package repository feature, which +would also allow authors to publish subpackages from a repository. However, we +think local dependencies is a much simpler feature that stands on its own. diff --git a/proposals/0202-random-unification.md b/proposals/0202-random-unification.md new file mode 100644 index 0000000000..48eeb31a90 --- /dev/null +++ b/proposals/0202-random-unification.md @@ -0,0 +1,387 @@ +# Random Unification + +* Proposal: [SE-0202](0202-random-unification.md) +* Author: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Ben Cohen](https://github.com/AirspeedSwift/) +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift#12772](https://github.com/apple/swift/pull/12772) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-020-random-unification/12040) + +## Introduction + +This proposal's main focus is to create a unified random API, and a secure random API for all platforms. + +*This idea has been floating around swift-evolution for a while now, but this is the thread that started this proposal: https://forums.swift.org/t/proposal-random-unification/6626* + +## Motivation + +The current random functionality that Swift provides is through imported C APIs. This of course is dependent on what system is running the code. This means that a different random implementation is provided on a per system basis. Swift relies on Darwin to provide developers the `arc4random(3)` approach. While on Linux, many developers tend to use `random(3)`, from the POSIX standard, or `rand(3)`, from the C standard library, for a quick and dirty solution for Linux systems. A very common implementation that developers tend to use for quick random functionality is here: + +```swift +// This is confusing because some may look at this and think Foundation provides these functions, +// when in reality Foundation imports Darwin and Glibc from which these are defined and implemented. +// You get the same behavior when you import UIKit, or AppKit. +import Foundation + +// We can't forget to seed the Linux implementation. +// This is unsecure as time is a very predictable seed. +// This raises more questions to whether or not Foundation, UIKit, or AppKit provided these functions. +#if os(Linux) +srandom(UInt32(time(nil))) +#endif + +// We name this randomNumber because random() interferes with Glibc's random()'s namespace +func randomNumber() -> Int { +#if !os(Linux) + return Int(arc4random()) // This is inefficient as it doesn't utilize all of Int's range +#else + return random() // or Int(rand()) +#endif +} + +// Many tend to opt in for ranges here, but for this example I opt for a start and end argument +func random(from start: Int, to end: Int) -> Int { +#if !os(Linux) + var random = Int(arc4random_uniform(UInt32(end - start))) +#else + // Not recommended as it introduces modulo bias + var random = random() % (end - start) +#endif + + random += start + + return random +} + +// Alternatively, an easier solution would be: +/* + func random(from start: Int, to end: Int) -> Int { + var random = randomNumber() % (end - start) + + random += start + + return random + } +*/ +// However this approach introduces modulo bias for all systems rather than just Linux. +``` + +While although this does work, it just provides a quick workaround to a much larger outstanding problem. Below is a list outlining the problems the example contains, and the problems the example introduces. + +1. In order to define these workarounds, developers must utilize a few platform checks. Developers should not be forced to define platform checks for such trivial requests like this one. + +2. Unexperienced developers who rely on workarounds like these, may be pushing unsecure code that is used by tens of hundreds or thousands of users. Starting with Darwin's `arc4random(3)`, pre macOS 10.12 (Sierra) and iOS 10, the implementation of `arc4random(3)` utilized the RC4 algorithm. This algorithm is now considered non-cryptographically secure due to RC4 weakness. Post macOS 10.12 (Sierra) and iOS 10, "...it was replaced with the NIST-approved AES cipher", as stated from the man pages in terminal (`man arc4random`). Moving on to Linux we see that using `random()` or `rand()` to generate numbers make it completely predictable as these weren't designed to be at a crypto level. + +3. In the example, it uses modulo to generate the number within the upper bound. Many developers may not realize it, but for a quick workaround, it introduces modulo bias in which modulo does not correctly distribute the probability in dividing the upper bound equally within the range. + +4. Highly inefficient as creating a new `Int` from a `UInt32` doesn't utilize the full extent of `Int`'s range. + +5. Could confuse some users as Foundation, AppKit, or UIKit don't provide these random functionalities. + +Considering all of this, I believe it is very important that Swift provides developers a simple, easy to use, and powerful random API. Although this is out of focus for Swift 5, this proposal solves a huge pain point for many Swift developers. + +## Proposed Solution + +### Random Number Generator + +To kick this off, the standard library will provide a default RNG. Each platform vendor will have the freedom to decide the specific implementation for this RNG for their platform. The standard library should document what specific RNG implementation is used on a specific platform. The aspiration is that this RNG should be cryptographically secure, provide reasonable performance, and should be thread safe. If a vendor is unable to provide these goals, they should document it clearly. It is also worth mentioning, that if an RNG on a platform has the possibility of failing, then it must fail when it is unable to complete its operation. An example of this is reading from `/dev/urandom`. If an error is to occur during reading, then it should produce a fatal error and abort the application. Reasons why I went with this approach in Alternatives Considered at the bottom of this proposal. + +### Random API + +For the core API, introduce a new protocol named `RandomNumberGenerator`. This type is used to define RNGs that can be used within the stdlib. Developers can conform to this type and use their own custom RNG throughout their whole application. + +Then for the stdlib's default RNG implementation, introduce a new struct named `SystemRandomNumberGenerator`. + +Next, we will make extension methods for `FixedWidthInteger`, `BinaryFloatingPoint` and `Bool`. For numeric types, this allows developers to select a value within a range and swap out the RNG used to select a value within the range. + +`FixedWidthInteger` example: +```swift +// Utilizes the standard library's default random +// Alias to: +// var rng = SystemRandomNumberGenerator() +// Int.random(in: 0 ..< 10, using: &rng) +let randomIntFrom0To10 = Int.random(in: 0 ..< 10) +let randomUIntFrom10Through100 = UInt.random(in: 10 ... 100, using: &myCustomRandomNumberGenerator) + +// The following are examples on how to get full width integers + +let randomInt = Int.random(in: .min ... .max) +let randomUInt = UInt.random(in: .min ... .max, using: &myCustomRandomNumberGenerator) +``` + +`BinaryFloatingPoint` example: +```swift +// Utilizes the standard library's default random +// Alias to: +// var rng = SystemRandomNumberGenerator() +// Float.random(in: 0 ..< 1, using: &rng) +let randomFloat = Float.random(in: 0 ..< 1) +let randomDouble = Double.random(in: 0 ... .pi, using: &myCustomRandomNumberGenerator) +``` + +`Bool` example: +```swift +// Utilizes the standard library's default random +// Alias to: +// var rng = SystemRandomNumberGenerator() +// Bool.random(using: &rng) +let randomBool1 = Bool.random() +let randomBool2 = Bool.random(using: &myCustomRandomNumberGenerator) +``` + +### Collection Additions + +#### Random Element + +For `Collection` we add an extension method for collections to get a random element. + +`Collection` example: +```swift +let greetings = ["hey", "hi", "hello", "hola"] + +// Utilizes the standard library's default random +// Alias to: +// var rng = SystemRandomNumberGenerator() +// greetings.randomElement(using: &rng)! +print(greetings.randomElement()!) // This returns an Optional +print(greetings.randomElement(using: &myCustomRandomNumberGenerator)!) // This returns an Optional +``` + +Note that some types make it easy to form collections with more elements than +can be represented as an `Int`, such as the range `Int.min...Int.max`, and +`randomElement` will likely trap on such collections. However, such ranges +are likely to trap when used with almost any collection API, and the +`random(in:)` method on `FixedWidthInteger` can be used for this purpose +instead. + +#### Shuffle API + +As a result of adding the random API, it only makes sense to utilize that power to fuel the shuffle methods. We extend `MutableCollection` to add a method to shuffle the collection itself, and extend `Sequence` to add a method to return a shuffled version of itself in a new array. Example: + +```swift +var greetings = ["hey", "hi", "hello", "hola"] + +// Utilizes the standard library's default random +// Alias to: +// var rng = SystemRandomNumberGenerator() +// greetings.shuffle(using: &rng) +greetings.shuffle() +print(greetings) // A possible output could be ["hola", "hello", "hey", "hi"] + +let numbers = 0 ..< 5 +print(numbers.shuffled(using: &myCustomRandomNumberGenerator)) // A possible output could be [1, 3, 0, 4, 2] +``` + +## Detailed Design + +The actual implementation can be found here: [apple/swift#12772](https://github.com/apple/swift/pull/12772) + +```swift +public protocol RandomNumberGenerator { + // This determines the functionality for producing a random number. + // Required to implement by all RNGs. + mutating func next() -> UInt64 +} + +// These sets of functions are not required and are provided by the stdlib by default +extension RandomNumberGenerator { + // This function provides generators a way of generating other unsigned integer types + public mutating func next() -> T + + // This function provides generators a mechanism for uniformly generating a number from 0 to upperBound + // Developers can extend this function themselves and create different behaviors for different distributions + public mutating func next(upperBound: T) -> T +} + +// The stdlib RNG. +public struct SystemRandomNumberGenerator : RandomNumberGenerator { + + public init() {} + + // Conformance for `RandomNumberGenerator`, calls one of the crypto functions. + public mutating func next() -> UInt64 + + // We override the protocol defined one to prevent unnecessary work in generating an + // unsigned integer that isn't a UInt64 + public mutating func next() -> T +} + +extension Collection { + // Returns a random element from the collection + // Can return nil if isEmpty is true + public func randomElement( + using generator: inout T + ) -> Element? + + /// Uses the standard library's default RNG + public func randomElement() -> Element? { + var g = SystemRandomNumberGenerator() + return randomElement(using: &g) + } +} + +// Enables developers to use things like Int.random(in: 5 ..< 12) which does not use modulo bias. +// It is worth noting that any empty range entered here will abort the program. +// We do this to preserve a general use case design that the core team expressed. +// For those that are that unsure whether or not their range is empty or not, +// they can if/guard check whether or not the range is empty beforehand, then +// use these functions. +extension FixedWidthInteger { + + public static func random( + in range: Range, + using generator: inout T + ) -> Self + + /// Uses the standard library's default RNG + public static func random(in range: Range) -> Self { + var g = SystemRandomNumberGenerator() + return Self.random(in: range, using: &g) + } + + public static func random( + in range: ClosedRange, + using generator: inout T + ) -> Self + + /// Uses the standard library's default RNG + public static func random(in range: ClosedRange) -> Self { + var g = SystemRandomNumberGenerator() + return Self.random(in: range, using: &g) + } +} + +// Enables developers to use things like Double.random(in: 5 ..< 12) which does not use modulo bias. +// It is worth noting that any empty range entered here will abort the program. +// We do this to preserve a general use case design that the core team expressed. +// For those that are that unsure whether or not their range is empty or not, +// they can simply if/guard check the bounds to make sure they can correctly form +// ranges which a random number can be formed from. +extension BinaryFloatingPoint where Self.RawSignificand : FixedWidthInteger { + + public static func random( + in range: Range, + using generator: inout T + ) -> Self + + /// Uses the standard library's default RNG + public static func random(in range: Range) -> Self { + var g = SystemRandomNumberGenerator() + return Self.random(in: range, using: &g) + } + + public static func random( + in range: ClosedRange, + using generator: inout T + ) -> Self + + /// Uses the standard library's default RNG + public static func random(in range: ClosedRange) -> Self { + var g = SystemRandomNumberGenerator() + return Self.random(in: range, using: &g) + } +} + +// We add this as a convenience to something like: +// Int.random(in: 0 ... 1) == 1 +// To the unexperienced developer they might have to look at this a few times to +// understand what is going on. This extension methods helps bring clarity to +// operations like these. +extension Bool { + public static func random( + using generator: inout T + ) -> Bool + + /// Uses the standard library's default RNG + public static func random() -> Bool { + var g = SystemRandomNumberGenerator() + return Bool.random(using: &g) + } +} + +// Shuffle API + +// The shuffle API will utilize the Fisher Yates algorithm + +extension Sequence { + public func shuffled( + using generator: inout T + ) -> [Element] + + /// Uses the standard library's default RNG + public func shuffled() -> [Element] { + var g = SystemRandomNumberGenerator() + return shuffled(using: &g) + } +} + +extension MutableCollection { + public mutating func shuffle( + using generator: inout T + ) + + /// Uses the standard library's default RNG + public mutating func shuffle() { + var g = SystemRandomNumberGenerator() + shuffle(using: &g) + } +} +``` + +## Source compatibility + +This change is purely additive, thus source compatibility is not affected. + +## Effect on ABI stability + +This change is purely additive, thus ABI stability is not affected. + +## Effect on API resilience + +This change is purely additive, thus API resilience is not affected. + +## Alternatives considered + +There were very many alternatives to be considered in this proposal. + +### Why would the program abort if it failed to generate a random number? + +I spent a lot of time deciding what to do if it failed. Ultimately it came down to the fact that many RNGs for platforms will almost never fail. In the cases where this can fail is where an RNG like `/dev/urandom` doesn't exist, or there were too many file descriptors open at once. In the case where `/dev/urandom` doesn't exist, either the kernel is too old to generate that file by itself on a fresh install, or a privileged user deleted it. Both of which are way out of scope for Swift in my opinion. In the case where there are too many file descriptors, with modern technology this should almost never happen. If the process has opened too many descriptors then it should be up to the developer to optimize opening and closing descriptors. + +In a world where this did return an error to Swift, it would require types like `Int` to return an optional on its static function. This would defeat the purpose of those static functions as it creates a double spelling for the same operation. + +```swift +let randomDice = Int.random(in: 1 ... 6)! +``` + +"I just want a random dice roll, what is this ! the compiler is telling me to add?" + +This syntax wouldn't make sense for a custom RNG that deterministically generates numbers with no fail. This also goes against the "general use" design that the core team and much of the community expressed. + +Looking at Rust, we can observe that they also abort when an unexpected error occurs with any of the forms of randomness. [source](https://doc.rust-lang.org/rand/src/rand/os.rs.html) + +It would be silly to account for these edge cases that would only happen to those who need to update their os, optimize their file descriptors, or deleted their `/dev/urandom`. Accounting for these edge cases for specific platforms sacrifices the clean API for everyone else. + +### Shouldn't this fallback on something more secure at times of low entropy? + +Thomas Hühn explains it very well [here](https://www.2uo.de/myths-about-urandom/). There is also a deeper discussion [here talking about python's implementation](https://www.python.org/dev/peps/pep-0524). Both articles discuss that even though `/dev/urandom` may not have enough entropy at a fresh install, "It doesn't matter. The underlying cryptographic building blocks are designed such that an attacker cannot predict the outcome." Using `getrandom(2)` on linux systems where the kernel version is >= 3.17, will block if it decides that the entropy pool is too small. In python's implementation, they fallback to reading `/dev/urandom` if `getrandom(2)` decides there is not enough entropy. + +### Why not make the default RNG non-secure? + +Swift is a safe language which means that it shouldn't be encouraging non-experienced developers to be pushing unsecure code. Making the default secure removes this issue and gives developers a feeling of comfort knowing their code is ready to go out of the box. + +### Rename `RandomNumberGenerator` + +It has been discussed to give this a name such as `RNG`. I went with `RandomNumberGenerator` because it is clear, whereas `RNG` has a level of obscurity to those who don't know the acronym. + +### Add static `.random()` to numerical types + +There were a bit of discussion for and against this. Initially I was on board with adding this as it provided a rather simple approach to getting full width integers/floating points within the range of `[0, 1)`. However, discussion came up that `Int.random()` encourages the misuse of modulo to get a number within x and y. This is bad because of modulo bias as I discussed in Motivation. For unexperienced developers, they might not know what values are going to be produced from those functions. It would require documentation reading to understand the range at which those functions pick from. They would also have inconsistent ranges that they choose from (`[.min, .max]` for `FixedWidthInteger`s and `[0, 1)` for `BinaryFloatingPoint`s). I thought about this for a very long time and I came to the conclusion that I would rather have a few extra characters than to have the potential for abuse with modulo bias and confusion around the ranges that these functions use. + +### Choose `range.random()` over static `.random(in:)` + +This was a very heavily discussed topic that we can't skip over. + +I think we came into agreement that `range.randomElement()` should be possible, however the discussion was around whether or not this is the primary spelling for getting random numbers. Having a range as the primary spelling makes it fairly simple to get a random number from. Ranges are also very desirable because it doesn't encourage modulo bias. Also, since we decided pretty early on that we're going to trap if for whatever reason we can't get a random number, this gave `range.randomElement()` the excuse to return a non optional. + +On the other end of the spectrum, we came into early agreement that `Collection.randomElement()` needs to return an optional in the case of an empty collection. If ranges were the primary spelling, then we would need to create exceptions for them to return non optionals. This would satisfy the general use design, but as we agreed that `.randomElement()` behaves more like `.first`, `.last`, `.min()`, and `.max()`. Because of this, `.randomElement()` has to return an optional to keep the consistent semantics. This justifies the static functions on the numeric types as the primary spelling as they can be the ones to return non optionals. These static functions are also the spelling for how developers think about going about random numbers. "Ok, I need a random integer from x and y." This helps give these functions the upper hand in terms of discoverability. diff --git a/proposals/0203-rename-sequence-elements-equal.md b/proposals/0203-rename-sequence-elements-equal.md new file mode 100644 index 0000000000..4d5391bee1 --- /dev/null +++ b/proposals/0203-rename-sequence-elements-equal.md @@ -0,0 +1,139 @@ +# Rename Sequence.elementsEqual + +* Proposal: [SE-0203](0203-rename-sequence-elements-equal.md) +* Author: [Xiaodi Wu](https://github.com/xwu) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Rejected** +* Implementation: [apple/swift#12884](https://github.com/apple/swift/pull/12884) +* Bugs: [SR-6102](https://bugs.swift.org/browse/SR-6102) +* Review thread: [Swift evolution forum](https://forums.swift.org/t/se-0203-rename-sequence-elementsequal/11482) + +## Introduction + +The behavior of `Sequence.elementsEqual` is confusing to users given its name. Having surveyed alternative solutions to this problem, it is proposed that the method be renamed to `Sequence.elementsEqualInIterationOrder`. + +Swift-evolution thread: [Rename Sequence.elementsEqual](https://forums.swift.org/t/draft-rename-sequence-elementsequal/6821) + +## Motivation + +[As Ole Begemann describes](https://twitter.com/olebegemann/status/916291785185529857), use of `Sequence.elementsEqual(_:)` can lead to surprising results for two instances of `Set`: + +```swift +var set1: Set = Set(1...5) +var set2: Set = Set((1...5).reversed()) + +set1 == set2 // true +set1.elementsEqual(set2) // false +``` + +In almost all circumstances where a set is compared to another set or a dictionary is compared to another dictionary, users should use `==`, which is order-insensitive, instead of `elementsEqual(_:)`, which is order-sensitive. + +[As Michael Ilseman explains](https://forums.swift.org/t/draft-rename-sequence-elementsequal/6821/152): + +> We have two forms of equality we're talking about: equality of Sequence and equality of the elements of Sequences in their respective ordering. `==` covers the former, and I'll use the existing (harmful) name of `elementsEqual` for the latter. +> +> `==` conveys substitutability of the two Sequences. This does not necessarily entail anything about their elements, how those elements are ordered, etc., it just means two Sequences are substitutable. `elementsEqual` means that the two Sequences produce substitutable elements. These are different concepts and both are independently useful. +> +> Cases: +> +> 1. Two Sequences are substitutable and produce substitutable elements when iterated. `==` and `elementsEqual` both return true. +> +> Example: Two arrays with the same elements in the same order. +> +> 2. Two Sequences are substitutable, but do not produce substitutable elements when iterated. `==` returns true, while `elementsEqual` returns false. +> +> Example: Two Sets that contain the same elements but in a different order. +> +> Contrived Example: Two Lorem Ipsum generators are the same generator (referentially equal, substitutable for the purposes of my library), but they sample the user’s current battery level (global state) each time they produce text to decide how fancy to make the faux Latin. They’re substitutable, but don’t generate the same sequence. +> +> 3. Two Sequences are not substitutable, but produce substitutable elements when iterated. `==` returns false, while `elementsEqual` returns true. +> +> Example: Consider two sequences that have differing identity. `==` operates on an identity level, `elementsEqual` operates at an element level. +> +> Contrived Example: InfiniteMonkeys and Shakespeare both produce the same sonnet, but they’re not substitutable for my library’s purposes. +> +> 4. Two Sequences are not substitutable and don’t produce substitutable elements when iterated. `==` and `elementsEqual` both return false. +> +> Example: `[1,2,3]` compared to `[4,5,6]` +> +> It is true that situations #2 and #3 are a little harder to grok, but they are what illustrate the subtle difference at hand. I think situation #2 is the most confusing, and has been the primary focus of this thread as Set exists and exhibits it. + +## Proposed solution + +The method `elementsEqual(_:)` is listed as an ["order-dependent operation"](https://developer.apple.com/documentation/swift/set/order_dependent_operations_on_set) in Apple documentation. However, its name does not suggest that it performs an order-sensitive comparison. (Other "order-dependent operations" incorporate words that clearly suggest order dependence in the name, such as "first," "last," "prefix," "suffix," and so on.) + +> These "order-dependent operations" are available for use because `Set` and `Dictionary` conform to `Sequence`. Major changes to the protocol hierarchy for Swift standard library collection types are out of the scope of this proposal, if not out of scope for Swift 5 entirely. + +The proposed solution is the result of an iterative process of reasoning, presented here: + +The first and most obvious solution is to remove the `elementsEqual(_:)` method altogether in favor of `==`. This prevents its misuse. However, because `elementsEqual(_:)` is a generic method on `Sequence`, we can use it to compare an instance of `UnsafeBufferPointer` to an instance of `[Int]`. This is a potentially useful and non-redundant feature which would be eliminated if the method is removed altogether. + +[A second solution](https://github.com/apple/swift/pull/12318) is to create overloads that forbid the use of `elementsEqual(_:)` method specifically in non-generic code that uses sets or dictionaries. This would certainly prevent misuse in non-generic code. However, it would also forbid legitimate mixed-type comparisons in non-generic code, and it would not prevent misuse in generic code. This solution also creates a difference in the behavior of generic and non-generic code that calls the same method, which is confusing, without solving the problem completely. + +A third solution is proposed here. It is predicated on the following observation: + +*Another method similar to `elementsEqual(_:)` exists on `Sequence` named `lexicographicallyPrecedes(_:)`. Like `prefix(_:)` and others, it is an order-dependent operation not completely suitable for an unordered collection. However, like `prefix(_:)` and unlike `elementsEqual(_:)`, this fact is called out in the name of the method. Unsurprisingly, like `prefix(_:)` and unlike `elementsEqual(_:)`, there is no evidence that `lexicographicallyPrecedes(_:)` has been a pitfall for users.* + +This observation suggests that a major reason for confusion over `elementsEqual(_:)` stems from its name. So, __it is proposed that `elementsEqual(_:)` should be renamed__. + +Initially, the suggested name for this method was **`lexicographicallyMatches`**, to parallel `lexicographicallyPrecedes`. This was opposed on two grounds: + +* The term __matches__ suggests a connection to pattern matching which does not exist. +* The term __lexicographically__ is unfamiliar to users, is inaccurate in the absence of a total ordering (i.e., where `Sequence.Element` does not conform to `Comparable`), and could erroneously suggest that the receiver and argument could themselves be re-ordered for comparison. + +Alternative suggestions used terms such as __pairwise__, __iterative__, __ordered__, or __sequential__. A revised name that aims for call-site clarity while incorporating these suggestions would be **`equalsInIterationOrder`**. + +However, the name should reflect the distinction, explained above, between equality of sequences and equality of their elements. It remains important for the renamed method to clarify that it is performing a comparison operation based on `Sequence.Element.==` and not `Sequence.==`. Therefore, incorporating this insight, the proposed name for the method is **`elementsEqualInIterationOrder`**. + +## Detailed design + +```swift +extension Sequence where Element : Equatable { + @available(swift, deprecated: 5, renamed: "elementsEqualInIterationOrder(_:)") + public func elementsEqual( + _ other: Other + ) -> Bool where Other.Element == Element { + return elementsEqualInIterationOrder(other) + } + + public func elementsEqualInIterationOrder( + _ other: Other + ) -> Bool where Other.Element == Element { + // The body of this method is unchanged. + var iter1 = self.makeIterator() + var iter2 = other.makeIterator() + while true { + switch (iter1.next(), iter2.next()) { + case let (e1?, e2?): + if e1 != e2 { return false } + case (_?, nil), (nil, _?): + return false + case (nil, nil): + return true + } + } + } +} +``` + +A parallel change will be made with respect to `elementsEqual(_:by:)`; that is, it will be renamed to `elementsEqualInIterationOrder(_:by:)`. + +## Source compatibility + +Existing code that uses `elementsEqual` will gain a deprecation warning. + +## Effect on ABI stability + +None. + +## Effect on API resilience + +This proposal adds new methods to the public API of `Sequence` and conforming types. + +## Alternatives considered + +It is to be noted that `lexicographicallyPrecedes(_:by:)` and `elementsEqual(_:by:)` are very closely related methods. However, they cannot be unified because they handle sequences of dissimilar count differently. + +Along those same lines, the manner in which `lexicographicallyPrecedes(_:)` and `lexicographicallyPrecedes(_:by:)` compare sequences of dissimilar count is described by the term __lexicographical__ ("a" < "aa" < "aaa" < "ab"), in contradistinction to __length-lexicographical__ (__shortlex__) ordering ("a" < "aa" < "ab" < "aaa") or __Kleene–Brouwer__ ordering ("aaa" < "aa" < "ab" < "a"). Though strictly redundant, the name could be modestly clarified by expanding it to `lexicographicallyPrecedesInIterationOrder`; however, the name cannot be shortened to `precedesInIterationOrder` without losing information as to the comparison performed. In the absence of evidence that users are using this method incorrectly, the additional consistency gained by adding "in iteration order" to the name is outweighed by the source-breaking nature of the change and the unwieldy end result. + +An entirely different solution to the motivating problem is to have sets internally reorder themselves prior to any order-dependent operations so that iteration order is guaranteed to be the same for sets which compare equal. As a result, `==` and `elementsEqual` would become equivalent when comparing two sets. Few (if any) other order-dependent operations would benefit from such a change, however, as the result of `first` remains arbitrary with or without it. There could potentially be a performance cost imposed on all users of sets, and the resulting benefit would be two functionally equivalent methods, which does not make possible any additional uses for the type. diff --git a/proposals/0204-add-last-methods.md b/proposals/0204-add-last-methods.md new file mode 100644 index 0000000000..8726f9cadb --- /dev/null +++ b/proposals/0204-add-last-methods.md @@ -0,0 +1,130 @@ +# Add `last(where:)` and `lastIndex(where:)` Methods + +* Proposal: [SE-0204](0204-add-last-methods.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 4.2)** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0204-add-last-where-and-lastindex-where-methods-rename-index-of-and-index-where-methods/11486/21) +* Implementation: [apple/swift#13337](https://github.com/apple/swift/pull/13337) +* Bug: [SR-1504](https://bugs.swift.org/browse/SR-1504) + +## Introduction + +The standard library should include methods for finding the last element in a sequence, and the index of the last element in a collection, that match a given predicate. + +* Swift-evolution thread: [\[swift-evolution\] (Draft) Add last(where:) and lastIndex(where:) methods](https://forums.swift.org/t/draft-add-last-where-and-lastindex-where-methods/7131) + +## Motivation + +The standard library currently has methods that perform a linear search to find an element or the index of an element that matches a predicate: + +```swift +let a = [20, 30, 10, 40, 20, 30, 10, 40, 20] +a.first(where: { $0 > 25 }) // 30 +a.index(where: { $0 > 25 }) // 1 +a.index(of: 10) // 2 +``` + +Unfortunately, there are no such methods that search from the end. Finding the last of a particular kind of element has multiple applications, particularly with text, such as wrapping a long string into lines of a maximum length or trimming whitespace from the beginning and end of a string. + +You can work around this limitation by using the methods above on a reversed view of a collection, but the resulting code is frankly appalling. For example, to find the corresponding last index to `a.index(where: { $0 > 25 })`, something like this unholy incantation is required: + +```swift +(a.reversed().index(where: { $0 > 25 })?.base).map({ a.index(before: $0) }) +``` + +## Proposed solution + +The `Sequence` protocol should add a `last(where:)` method, and the `Collection` protocol should add `lastIndex(where:)` and `lastIndex(of:)` methods. These new methods create symmetry with the existing forward-searching APIs that are already part of `Sequence` and `Collection`. + +These additions remove the need for searching in a reversed collection and allow code like this: + +```swift +a.last(where: { $0 > 25 }) // 40 +a.lastIndex(where: { $0 > 25 }) // 7 +a.lastIndex(of: 10) // 6 +``` + +Much better! + +## Detailed design + +`last(where:)` and `lastIndex(where:)` will be added to the standard library as `Sequence` and `Collection` protocol requirements, respectively. These methods will have default implementations in their respective protocols and in `BidirectionalCollection`, which can provide a more efficient implementation. `lastIndex(of:)` will be provided in `Collection` and `BidirectionalCollection` extensions constrained to `Equatable` elements. + +In addition, the `index(of:)` and `index(where:)` methods will be renamed to `firstIndex(of:)` and `firstIndex(where:)`, respectively. This change is beneficial for two reasons. First, it groups the `first...` and `last...` methods into two symmetric analogous groups, and second, it leaves the `index...` methods as solely responsible for index manipulation. + +The new APIs are shown here: + +```swift +protocol Sequence { + // Existing declarations... + + /// Returns the last element of the collection that satisfies the given + /// predicate, or `nil` if no element does. The sequence must be finite. + func last(where predicate: (Element) throws -> Bool) + rethrows -> Element? +} + +protocol Collection { + // Existing declarations... + + // Renamed from index(where:) + func firstIndex(where predicate: (Element) throws -> Bool) + rethrows -> Index? + + /// Returns the index of the last element of the collection that satisfies + /// the given predicate, or `nil` if no element does. + func lastIndex(where predicate: (Element) throws -> Bool) + rethrows -> Index? +} + +extension BidirectionalCollection { + func last(where predicate: (Element) throws -> Bool) + rethrows -> Element? { ... } + + func lastIndex(where predicate: (Element) throws -> Bool) + rethrows -> Index? { ... } +} + +extension Collection where Element: Equatable { + // Renamed from index(of:) + func firstIndex(of element: Element) -> Index? { ... } + + /// Returns the index of the last element equal to the given element, or + /// no matching element is found. + func lastIndex(of element: Element) -> Index? { ... } +} + +extension BidirectionalCollection where Element: Equatable { + func lastIndex(of element: Element) -> Index? { ... } +} +``` + +You can try out the usage (but not really the performance) of the `last...` methods by copying [this gist](https://gist.github.com/natecook1000/45c3df2aa5f84a834063329a478c7972) into a playground. + +## Source compatibility + +The addition of the `last(where:)`, `lastIndex(where:)`, and `lastIndex(of:)` methods is strictly additive and should have no impact on existing code. + +The name changes to `index(of:)` and `index(where:)` are easily migratable, and will be deprecated in Swift 4.2 before being removed in Swift 5. + +## Effect on ABI stability & API resilience + +This change does not affect ABI stability or API resilience beyond the addition of the new methods. + +## Alternatives considered + +- A previous proposal limited the new methods to the `BidirectionalCollection` protocol. This isn't a necessary limitation, as the standard library already has methods on sequences and forward collections with the same performance characteristics. + +- Another previous proposal included renaming `index(of:)` and `index(where:)` to `firstIndex(of:)` and `firstIndex(where:)`, respectively. A proposal after *that one* removed that source-breaking change. *This* version of the proposal adds the source-breaking change back in again. + +- An [alternative approach](https://github.com/swiftlang/swift-evolution/pull/773#issuecomment-351148673) is to add a defaulted `options` parameter to the existing searching methods, like so: + + ```swift + let a = [20, 30, 10, 40, 20, 30, 10, 40, 20] + a.index(of: 10) // 2 + a.index(of: 10, options: .backwards) // 6 + ``` + + This alternative approach matches the convention used in Foundation, such as `NSString.range(of:options:)` and `NSArray.indexOfObject(options:passingTest:)`, but in the opinion of this author, separate functions better suit the structure of the Swift standard library. + diff --git a/proposals/0205-withUnsafePointer-for-lets.md b/proposals/0205-withUnsafePointer-for-lets.md new file mode 100644 index 0000000000..f85cd48bcd --- /dev/null +++ b/proposals/0205-withUnsafePointer-for-lets.md @@ -0,0 +1,88 @@ +# `withUnsafePointer(to:_:)` and `withUnsafeBytes(of:_:)` for immutable values + +* Proposal: [SE-0205](0205-withUnsafePointer-for-lets.md) +* Authors: [Joe Groff](https://github.com/jckarter) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift#15608](https://github.com/apple/swift/pull/15608) + +## Introduction + +We propose to extend the toplevel `withUnsafePointer(to:_:)` and +`withUnsafeBytes(of:_:)` functions to work with read-only values. + +Swift-evolution thread: [`withUnsafePointer` and `withUnsafeBytes` for immutable arguments](https://forums.swift.org/t/withunsafepointer-and-withunsafebytes-for-immutable-arguments/11493/5) + +## Motivation + +`withUnsafePointer` and `withUnsafeBytes` provide temporary scoped access to +the in-memory representation of variables and properties via pointers. They +currently only accept `inout` arguments, which makes working with the +in-memory representation of immutable values more awkward and inefficient +than it ought to be, requiring a copy: + +``` +let x = 1 + 2 +var x2 = x +withUnsafeBytes(of: &x2) { ... } +``` + +Even for `inout` arguments, the semantics of `withUnsafeBytes` and +`withUnsafePointer` are extremely narrow: the pointer cannot be used after +the closure returns, cannot be mutably aliased, and is not guaranteed to +have a stable identity across `withUnsafe*` applications to the same +property. There's therefore no fundamental reason these operations couldn't +also be offered on arbitrary read-only variables or temporary values, since +memory does not need to be persisted for longer than a call. It is a +frequent request to extend these APIs to work with read-only values. + +## Proposed solution + +We add overloads of `withUnsafeBytes` and `withUnsafePointer` that accept +their `to`/`of` argument by value (by shared borrow, when we get moveonly +types). + +## Detailed design + +The following top-level functions are added to the standard library: + +``` +public func withUnsafePointer( + to value: /*borrowed*/ T, + _ body: (UnsafePointer) throws -> Result +) rethrows -> Result + +public func withUnsafeBytes( + of value: /*borrowed*/ T, + _ body: (UnsafeRawBufferPointer) throws -> Result +) rethrows -> Result +``` + +Like their `inout`-taking siblings, each method produces a pointer to +the in-memory representation of its `value` argument and invokes the `body` +closure with the pointer. The pointer is only valid for the duration of `body`. +It is undefined behavior to write through the pointer or access it after +`body` returns. It is unspecified whether taking a pointer to the same +variable or property produces a consistent pointer value across different +`withUnsafePointer`/`Bytes` calls. + +We do not propose removing the `inout`-taking forms of `withUnsafePointer` +and `withUnsafeBytes`, both for source compatibility, and because in Swift +today, using `inout` is still the only reliably way to assert exclusive +access to storage and suppress implicit copying. Optimized Swift code +ought to in principle avoid copying when `withUnsafePointer(to: x)` is passed +a stored `let` property `x` that is known to be immutable, and with moveonly +types it would be possible to statically guarantee this, but in Swift today +this cannot be guaranteed. + +## Source compatibility + +Existing code can continue to use the `inout`-taking forms without modification. + +## Effect on ABI stability/API resilience + +This is a purely additive change to the standard library. In anticipation +of a future version of Swift adding moveonly types, we should ensure that +the implementations of `withUnsafePointer` and `withUnsafeBytes` takes their +arguments +0 so that they will be ABI-compatible with shared borrows of +moveonly types in the future. diff --git a/proposals/0206-hashable-enhancements.md b/proposals/0206-hashable-enhancements.md new file mode 100644 index 0000000000..289cb6d985 --- /dev/null +++ b/proposals/0206-hashable-enhancements.md @@ -0,0 +1,921 @@ +# Hashable Enhancements + +* Proposal: [SE-0206](0206-hashable-enhancements.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey), [Vincent Esche](https://github.com/regexident) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 4.2)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0206-hashable-enhancements/11675/115) +* Implementation:
+ [apple/swift#14913](https://github.com/apple/swift/pull/14913) (standard library, underscored),
+ [apple/swift#16009](https://github.com/apple/swift/pull/16009) (`Hasher` interface),
+ [apple/swift#16073](https://github.com/apple/swift/pull/16073) (automatic synthesis, de-underscoring)
+* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/f5a020ec79cdb64fc8700af91b1a1ece2d2fb141/proposals/0206-hashable-enhancements.md) + + + +## Table of contents + +* [Introduction](#intro) +* [Motivation](#why) + - [The status quo](#status-quo) + - [Universal hash functions](#universal-hashing) +* [Proposed solution](#proposed-solution) + - [The `Hasher` struct](#hasher) + - [The `hash(into:)` requirement](#hash-into) +* [Detailed design](#detailed-design) + - [`Hasher`](#hasher-details) + - [`Hashable`](#hashable-details) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#abi) +* [Effect on API resilience](#resilience) +* [Alternatives considered](#alternatives) + - [Leaving `Hashable` as is](#leave-hashable-alone) + - [Defining a new protocol](#new-protocol) + - [Making `hash(into:)` generic over a `Hasher` protocol](#generic-hasher) + - [Change `hash(into:)` to take a closure instead of a new type](#closure-hasher) + +## Introduction + +This proposal introduces a new `Hasher` type representing the standard +library's universal hash function, and it extends the `Hashable` +protocol with a new `hash(into:)` requirement that expresses hashing +in terms of `Hasher`. This new requirement is intended to replace the +old `hashValue` property, which is deprecated. + +Switching to `hash(into:)` moves the choice of a hash function out of +`Hashable` implementations, and into the standard library. This makes +it considerably easier to manually conform types to `Hashable` -- the +task of writing a custom implementation reduces to identifying the +parts of a type that should contribute to the hash value. + +Standardizing on a single, high-quality hash function greatly improves +the reliability of `Set` and `Dictionary`. The hash function can be +specially selected and tuned to work well with the hash tables used by +these collections, preventing hash collision patterns that would break +the expected performance of common operations. + +`Hasher` is a resilient struct, enabling future versions of the +standard library to improve the hash function without the need to +change (or even recompile) existing code that implements `Hashable`. +Not baking any particular hash function into `Hashable` types is +especially important in case a weakness is found in the current +algorithm. + +Swift-evolution thread: [Combining Hashes](https://forums.swift.org/t/combining-hashes/9082) + +[SE-0185]: 0185-synthesize-equatable-hashable.md +[SE-0143]: 0143-conditional-conformances.md + +## Motivation + +The Swift Standard Library includes two general-purpose hashing +collections, `Set` and `Dictionary`. These collections are built +around hash tables, whose performance is critically dependent on the +expected distribution of the elements stored in them, along with the +quality of the hash function that is used to derive bucket indices for +individual elements. + +With a good hash function, simple lookups, insertions and removals +take constant time on average. However, when the hash function isn't +carefully chosen to suit the data, the expected time of such +operations can become proportional to the number of elements stored in +the table. If the table is large enough, such a regression can easily +lead to unacceptable performance. When they're overwhelmed with hash +collisions, applications and network services may stop processing new +events for prolonged periods of time; this can easily be enough to +make the app unresponsive or to bring down the service. + +### The status quo + +Since Swift version 1.0, `Hashable` has had a single requirement on +top of `Equatable`: the `hashValue` property. `hashValue` looks +deceptively simple, but implementing it is unreasonably hard: Not only +do we need to decide which components of our type should be involved +in hashing, but we also have to come up with a way to somehow distill +these components down into a single integer value. The API is +essentially asking us to implement a new hash function, from scratch, +every single time we conform to `Hashable`. + +Given adequate documentation, it is reasonable to expect that an +experienced programmer implementing a custom type would be able to +identify what parts need to be hashed. On the other hand, implementing +a good hash function requires careful consideration and specialist +knowledge. It is unreasonable to expect Swift programmers to invest +time and effort to get this right for every `Hashable` type out +there. + +For example, consider the code below, extracted directly from the +documentation of Swift 4.1's `Hashable`. Is this a good implementation +of `hashValue`? + +```swift +struct GridPoint { + var x: Int + var y: Int +} + +extension GridPoint: Hashable { + var hashValue: Int { + return x.hashValue ^ y.hashValue &* 16777619 + } + + static func == (lhs: GridPoint, rhs: GridPoint) -> Bool { + return lhs.x == rhs.x && lhs.y == rhs.y + } +} +``` + +The answer is that it depends; while the hash values it produces are +perfectly fine if `x` and `y` are expected to be small integers coming +from trusted sources, this hash function does have some undesirable +properties: + +1. The clever bit manipulations make it hard to understand what the + code does, or how it works. For example, can you tell what makes + 16777619 a better choice for the multiplier than, say, 16777618? + What are the precedence rules for `^` and `&*` again? What's with + the ampersand, anyway? + + We just wanted to use `GridPoint` values as keys in a + `Dictionary`, but first, we need to spend a couple of hours + learning about bitwise operations, integer overflows and the + exciting properties of coprime numbers. + + (For what it's worth, the magic constant used in the example above + is the same as the one used for the 32-bit version of the [FNV-1a] + hashing algorithm, which uses a similar (if a little more + complicated) method to distill arbitrary byte sequences down into + a single integer.) + +2. It is trivially easy to construct an arbitrarily large set of + `GridPoint` values that aren't equal, but have the same hash + value. If the values come from an untrusted source, they may + sometimes be deliberately chosen to induce collisions. + +3. The hash function doesn't do a particularly great job at mixing up + the input data; the hash values it produces tend to form long + chains of sequential integer clusters. While these aren't as bad + as hash collisions, some hash table operations can slow down + drastically when such clusters are present. (In Swift 4.1, `Set` + and `Dictionary` use open addressing with linear probing, and they + have to do some clever postprocessing of hash values to get rid of + such patterns.) + +It seems desirable for the standard library to provide better guidance +for people implementing `hashValue`. + +### Universal hash functions + +With [SE-0185], Swift 4.1 introduced compiler support for automatic +synthesis of `Hashable` conformance for certain types. For example, +the `GridPoint` struct above can be made to conform to `Hashable` +without explicitly defining `hashValue` (or `==`): + +```swift +struct GridPoint: Hashable { + var x: Int + var y: Int + + // hashValue and == are automatically synthesized by the compiler +} +``` + +[SE-0185] did not specify a hash function to be used for such +conformances, leaving it as an implementation detail of the compiler +and the standard library. Doing this well requires the use of a hash +function that works equally well on any number of components, +regardless of their expected distributions. + +[SE-0185]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0185-synthesize-equatable-hashable.md + +Luckily, this problem has occurred in other contexts before, and there +is an extensive list of hash functions that have been designed for +exactly such cases: [Foller-Noll-Vo][FNV-1a], [MurmurHash], +[CityHash], [SipHash], and [HighwayHash] are just a small selection of +these. The last two algorithms include some light cryptographic +elements so that they provide a level of protection against deliberate +hash collision attacks. This makes them a better choice for +general-purpose hashed collections like `Set` and `Dictionary`. + +[FNV-1a]: http://www.isthe.com/chongo/tech/comp/fnv/index.html +[MurmurHash]: https://github.com/aappleby/smhasher +[CityHash]: https://github.com/google/cityhash +[SipHash]: https://131002.net/siphash/ +[HighwayHash]: https://github.com/google/highwayhash + +Since [SE-0185] required the standard library to implement a +high-quality universal hash function, it seems like a good idea to +expose it as public API, so that manual `Hashable` implementations can +take advantage of it, too. + +Universal hash functions work by maintaining some internal state -- +this can be as simple as a single 32/64-bit integer value (for +e.g. [FNV-1a]), but it is usually much wider than that. For example, +[SipHash] maintains a state of 256 bits, while [HighwayHash] uses 1024 +bits. + +## Proposed solution + +We solve `Hashable`'s implementation problems in two parts. First, we +make the standard library's hash function public. Second, we replace +`hashValue` with a requirement that is designed specifically to +eliminate the guesswork from manual `Hashable` implementations. + +### The `Hasher` struct + +We propose to expose the standard library's standard hash function as +a new, public struct type, called `Hasher`. This new struct captures +the state of the hash function, and provides the following operations: + +1. An initializer to create an empty state. To make hash values less + predictable, the standard hash function uses a per-execution random + seed, so that generated hash values will be different in each + execution of a Swift program. + + ```swift + public struct Hasher { + public init() // Uses a per-execution random seed value + } + ``` + +2. Operations to feed new bytes to the hash function, mixing them + into its state: + + ```swift + extension Hasher { + public mutating func combine(bytes buffer: UnsafeRawBufferPointer) + public mutating func combine(_ value: H) + } + ``` + + `combine(bytes:)` is the most general operation, suitable for use + when the bytes to be hashed are available as a single contiguous + region in memory. + + `combine(_:)` is a convenience operation that can take any + `Hashable` value; we expect this will be more frequently + useful. (We'll see how this is implemented in the next section.) + +3. An operation to finalize the state, extracting the hash value from it. + + ```swift + extension Hasher { + public __consuming func finalize() -> Int + } + ``` + + Finalizing the hasher state consumes it; it is illegal to call + `combine` or `finalize` on a hasher you don't own, or on one that's + already been finalized. + +For example, here is how one may use `Hasher` as a standalone type: + +```swift +var hasher = Hasher() // Initialize state, usually by random seeding +hasher.combine(23) // Mix in several integer's worth of bytes +hasher.combine(42) +print(hasher.finalize()) // Finalize the state and return the hash +``` + +Within the same execution of a Swift program, `Hasher`s are guaranteed +to return the same hash value in `finalize()`, as long as they are fed +the exact same sequence of bytes. (Note that the order of `combine` +operations matters; swapping the two integers above will produce a +completely different hash.) + +However, `Hasher` may generate entirely different hash values in other +executions, *even if it is fed the exact same byte sequence*. This +randomization is a critical feature, as it makes it much harder for +potential attackers to predict hash values. `Hashable` has always been +documented to explicitly allow such nondeterminism: + +> - Important: Hash values are not guaranteed to be equal across +> different executions of your program. Do not save hash values to +> use in a future execution. +> +> -- `Hashable` documentation + +(Random seeding can be disabled by setting a special environment +variable; see [Effect on ABI stability](#abi) for details.) + +The choice of which hash function `Hasher` implements is an +implementation detail of the standard library, and may change in any +new release. This includes the size and internal layout of `Hasher` +itself. (The current implementation uses SipHash-1-3 with 320 bits of +state.) + +### The `hash(into:)` requirement + +Introducing `Hasher` is a big improvement, but it's only half of the +story: `Hashable` itself needs to be updated to make better use of it. + +We propose to change the `Hashable` protocol by adding a new +`hash(into:)` requirement: + +```swift +public protocol Hashable: Equatable { + var hashValue: Int { get } + func hash(into hasher: inout Hasher) +} +``` + +At the same time, we deprecate custom implementations of the +`hashValue` property. (Please see the section on [Source +compatibility](#source-compatibility) on how we'll do this without +breaking code written for previous versions of Swift.) In some future +language version, we intend to convert `hashValue` to an extension +method. + +To make it easier to express `hash(into:)` in terms of `Hashable` +components, `Hasher` provides a variant of `combine` that simply calls +`hash(into:)` on the supplied value: + +```swift +extension Hasher { + @inlinable + public mutating func combine(_ value: H) { + value.hash(into: &self) + } +} +``` + +This is purely for convenience; `hasher.combine(foo)` is slightly +easier to type than `foo.hash(into: &hasher)`. + + +At first glance, it may not be obvious why we need to replace +`hashValue`. After all, `Hasher` can be used to take the guesswork out +of its implementation: + +```swift +extension GridPoint: Hashable { + var hashValue: Int { + var hasher = Hasher() + hasher.combine(x) + hasher.combine(y) + return hasher.finalize() + } +} +``` + +What's wrong with this? What makes `hash(into:)` so much better that's +worth the cost of a change to a basic protocol? + +* **Better Discoverability** -- With `hashValue`, you need to know + about `Hasher` to make use of it: the API does not direct you to do + the right thing. Worse, you have to do extra busywork by manually + initializing and finalizing a `Hasher` instance directly in your + `hashValue` implementation. Compare the code above to the + `hash(into:)` implementation below: + + ```swift + extension GridPoint: Hashable { + func hash(into hasher: inout Hasher) { + hasher.combine(x) + hasher.combine(y) + } + } + ``` + + This is nice and easy, with minimal boilerplate. `Hasher` is part of + the function signature; people who need to implement `hash(into:)` + are naturally guided to learn about it. + +* **Guaranteed Dispersion Quality** -- Keeping the existing + `hashValue` interface would mean that there was no way for `Set` and + `Dictionary` to guarantee that `Hashable` types produce hash values + with good enough dispersion. Therefore, these collections would need + to keep postprocessing hash values. We'd like to eliminate + postprocessing overhead for types that upgrade to `Hasher`. + +* **Hasher Customizability** -- `hash(into:)` moves the initialization + of `Hasher` out of `Hashable` types, and into hashing + collections. This allows us to customize `Hasher` to the needs of + each hashing data structure. For example, the stdlib could start + using a different seed value for every new `Set` and `Dictionary` + instance; this somewhat improves reliability by making hash values + even less predictable, but (probably more importantly), it + drastically improves the performance of some relatively common + operations involving [copying data between `Set`/`Dictionary` + instances][quadratic-copy]. + +* **Better Performance** -- Similarly, `hash(into:)` moves the + finalization step out of `Hashable`. Finalization is a relatively + expensive operation; for example, in SipHash-1-3, it costs three + times as much as a single 64-bit `combine`. Repeating it for every + single component of a composite type would make hashing unreasonably + slow. + + For example, consider the `GridRectangle` type below: + ```swift + struct GridRectangle { + let topLeft: GridPoint + let bottomRight: GridPoint + } + ``` + + With `hashValue`, its `Hashable` implementation would look like this: + ```swift + extension GridRectangle: Hashable { + var hashValue: Int { // SLOW, DO NOT USE + var hasher = Hasher() + hasher.combine(bits: topLeft.hashValue) + hasher.combine(bits: bottomRight.hashValue) + return hasher.finalize() + } + } + ``` + + Both of the `hashValue` invocations above create and finalize + separate hashers. Assuming finalization takes three times as much + time as a single combine call (and generously assuming that initialization + is free) this takes 15 combines' worth of time: + + ``` + 1 hasher.combine(topLeft.hashValue) + 1 hasher.combine(topLeft.x) (in topLeft.hashValue) + 1 hasher.combine(topLeft.y) + 3 hasher.finalize() + 1 hasher.combine(bottomRight.hashValue) + 1 hasher.combine(bottomRight.x) (in bottomRight.hashValue) + 1 hasher.combine(bottomRight.y) + 3 hasher.finalize() + 3 hasher.finalize() + --- + 15 + ``` + + Switching to `hash(into:)` gets us the following code: + + ```swift + extension GridRegion: Hashable { + func hash(into hasher: inout Hasher) { + hasher.combine(topLeft) + hasher.combine(bottomRight) + } + } + ``` + + This reduces the cost of hashing to just four combines and a single + finalization, which takes less than half the time of our original + approach: + + ``` + 1 hasher.combine(topLeft.x) (in topLeft.hash(into:)) + 1 hasher.combine(topLeft.y) + 1 hasher.combine(bottomRight.x) (in bottomRight.hash(into:)) + 1 hasher.combine(bottomRight.y) + 3 hasher.finalize() (outside of GridRectangle.hash(into:)) + --- + 7 + ``` + +Switching to `hash(into:)` gets us more robust hash values faster, and +with cleaner, simpler code. + + +[quadratic-copy]: https://bugs.swift.org/browse/SR-3268 + + +## Detailed design + +### `Hasher` + +Add the following type to the standard library: + +```swift +/// Represents the universal hash function used by `Set` and `Dictionary`. +/// +/// The hash function is a mapping from a 128-bit seed value and an +/// arbitrary sequence of bytes to an integer hash value. The seed value +/// is specified during `Hasher` initialization, while the byte sequence +/// is fed to the hasher using a series of calls to mutating `combine` +/// methods. When all bytes have been fed to the hasher, the hash value +/// can be retrieved by calling `finalize()`: +/// +/// var hasher = Hasher() +/// hasher.combine(23) +/// hasher.combine("Hello") +/// let hashValue = hasher.finalize() +/// +/// The underlying hash algorithm is designed to exhibit avalanche +/// effects: slight changes to the seed or the input byte sequence +/// will produce drastic changes in the generated hash value. +/// +/// - Note: `Hasher` is usually randomly seeded, which means it will return +/// different values on every new execution of your program. The hash +/// algorithm implemented by `Hasher` may itself change between +/// any two versions of the standard library. Do not save or otherwise +/// reuse hash values across executions of your program. +public struct Hasher { + /// Initialize a new hasher using a per-execution seed value. + /// The seed is set during process startup, usually from a high-quality + /// random source. + public init() + + /// Feed `value` to this hasher, mixing its essential parts into + /// the hasher state. + @inlinable + public mutating func combine(_ value: H) { + value.hash(into: &self) + } + + /// Feed the raw bytes in `buffer` into this hasher, mixing its bits into + /// the hasher state. + public mutating func combine(bytes buffer: UnsafeRawBufferPointer) + + /// Finalize the hasher state and return the hash value. + /// + /// Finalizing consumes the hasher: it is illegal to finalize a hasher you + /// don't own, or to perform operations on a finalized hasher. (These may + /// become compile-time errors in the future.) + public __consuming func finalize() -> Int +} +``` + +## `Hashable` + +Change the `Hashable` protocol as follows. + +```swift +/// A type that can be hashed into a `Hasher`. +/// +/// You can use any type that conforms to the `Hashable` protocol in a set or as +/// a dictionary key. Many types in the standard library conform to `Hashable`: +/// Strings, integers, floating-point and Boolean values, and even sets are +/// hashable by default. Some other types, such as optionals, arrays and ranges +/// automatically become hashable when their type arguments implement the same. +/// +/// Your own custom types can be hashable as well. When you define an +/// enumeration without associated values, it gains `Hashable` conformance +/// automatically, and you can add `Hashable` conformance to your other custom +/// types by implementing the `hash(into:)` function. For structs whose stored +/// properties are all `Hashable`, and for enum types that have all-`Hashable` +/// associated values, the compiler is able to provide an implementation of +/// `hash(into:)` automatically. +/// +/// Hashing a value means feeding its essential components into a hash function, +/// represented by the `Hasher` type. Essential components are those that +/// contribute to the type's implementation of `Equatable`. Two instances that +/// are equal must feed the same values to `Hasher` in `hash(into:)`, in the +/// same order. +/// +/// Conforming to the Hashable Protocol +/// =================================== +/// +/// To use your own custom type in a set or as the key type of a dictionary, +/// add `Hashable` conformance to your type. The `Hashable` protocol inherits +/// from the `Equatable` protocol, so you must also satisfy that protocol's +/// requirements. +/// +/// A custom type's `Hashable` and `Equatable` requirements are automatically +/// synthesized by the compiler when you declare `Hashable` conformance in the +/// type's original declaration and your type meets these criteria: +/// +/// - For a `struct`, all its stored properties must conform to `Hashable`. +/// - For an `enum`, all its associated values must conform to `Hashable`. (An +/// `enum` without associated values has `Hashable` conformance even without +/// the declaration.) +/// +/// To customize your type's `Hashable` conformance, to adopt `Hashable` in a +/// type that doesn't meet the criteria listed above, or to extend an existing +/// type to conform to `Hashable`, implement the `hash(into:)` function in your +/// custom type. To ensure that your type meets the semantic requirements of the +/// `Hashable` and `Equatable` protocols, it's a good idea to also customize +/// your type's `Equatable` conformance to match the `hash(into:)` definition. +/// +/// As an example, consider a `GridPoint` type that describes a location in a +/// grid of buttons. Here's the initial declaration of the `GridPoint` type: +/// +/// /// A point in an x-y coordinate system. +/// struct GridPoint { +/// var x: Int +/// var y: Int +/// } +/// +/// You'd like to create a set of the grid points where a user has already +/// tapped. Because the `GridPoint` type is not hashable yet, it can't be used +/// as the `Element` type for a set. To add `Hashable` conformance, provide an +/// `==` operator function and a `hash(into:)` function. +/// +/// extension GridPoint: Hashable { +/// func hash(into hasher: inout Hasher) { +/// hasher.combine(x) +/// hasher.combine(y) +/// } +/// +/// static func == (lhs: GridPoint, rhs: GridPoint) -> Bool { +/// return lhs.x == rhs.x && lhs.y == rhs.y +/// } +/// } +/// +/// The `hash(into:)` property in this example feeds the properties `x` and `y` +/// to the supplied hasher; these are the same properties compared by the +/// implementation of the `==` operator function. +/// +/// Now that `GridPoint` conforms to the `Hashable` protocol, you can create a +/// set of previously tapped grid points. +/// +/// var tappedPoints: Set = [GridPoint(x: 2, y: 3), GridPoint(x: 4, y: 1)] +/// let nextTap = GridPoint(x: 0, y: 1) +/// if tappedPoints.contains(nextTap) { +/// print("Already tapped at (\(nextTap.x), \(nextTap.y)).") +/// } else { +/// tappedPoints.insert(nextTap) +/// print("New tap detected at (\(nextTap.x), \(nextTap.y)).") +/// } +/// // Prints "New tap detected at (0, 1).") +public protocol Hashable: Equatable { + /// The hash value. + /// + /// Hash values are not guaranteed to be equal across different executions of + /// your program. Do not save hash values to use during a future execution. + var hashValue: Int { get } + + /// Hash the essential components of this value into the hash function + /// represented by `hasher`, by feeding them into it using its `combine` + /// methods. + /// + /// Essential components are precisely those that are compared in the type's + /// implementation of `Equatable`. + func hash(into hasher: inout Hasher) +} +``` + +## Source compatibility + +The introduction of the new `Hasher` type is a purely additive change. +However, adding the `hash(into:)` requirement is potentially source +breaking. To ensure we keep compatibility with code written for +previous versions of Swift, while also allowing new code to only +implement `hash(into:)`, we extend [SE-0185]'s automatic `Hashable` +synthesis to automatically derive either of these requirements when +the other has been manually implemented. + +Code written for Swift 4.1 or earlier will continue to compile (in the +corresponding language mode) after this proposal is implemented. The +compiler will synthesize the missing `hash(into:)` requirement +automatically: + +``` +struct GridPoint41: Hashable { // Written for Swift 4.1 or below + let x: Int + let y: Int + + var hashValue: Int { + return x.hashValue ^ y.hashValue &* 16777619 + } + + // Supplied automatically by the compiler: + func hash(into hasher: inout Hasher) { + hasher.combine(self.hashValue) + } +} +``` + +The compiler will emit a deprecation warning when it needs to do this +in 4.2 mode. (However, note that code that merely uses `hashValue` +will continue to compile without warnings.) + +Code written for Swift 4.2 or later should conform to `Hashable` by +implementing `hash(into:)`. The compiler will then complete the +conformance with a suitable `hashValue` definition: + +``` +struct GridPoint42: Hashable { // Written for Swift 4.2 + let x: Int + let y: Int + + func hash(into hasher: inout Hasher) { + hasher.combine(x) + hasher.combine(y) + } + + // Supplied automatically by the compiler: + var hashValue: Int { + var hasher = Hasher() + self.hash(into: &hasher) + return hasher.finalize() + } +} +``` + +When upgrading to Swift 4.2, `Hashable` types written for earlier +versions of Swift should be migrated to implement `hash(into:)` +instead of `hashValue`. + +For types that satisfy [SE-0185]'s requirements for `Hashable` +synthesis, this can be as easy as removing the explicit `hashValue` +implementation. Note that Swift 4.2 implements conditional `Hashable` +conformances for many standard library types that didn't have it +before; this enables automatic `Hashable` synthesis for types that use +these as components. + +For types that still need to manually implement `Hashable`, the +migrator can be updated to help with this process. For example, the +`GridPoint41.hashValue` implementation above can be mechanically +rewritten as follows: + +``` +struct GridPoint41: Hashable { // Migrated from Swift 4.1 + let x: Int + let y: Int + + // Migrated from hashValue: + func hash(into hasher: inout Hasher) { + hash.combine(x.hashValue ^ y.hashValue &* 16777619) + } +} +``` + +This will not provide the same hash quality as combining both members +one by one, but it may be useful as a starting point. + + +## Effect on ABI stability + +`Hasher` and `hash(into:)` are additive changes that extend the ABI of +the standard library. `Hasher` is a fully resilient struct, with +opaque size/layout and mostly opaque members. (The only exception is +the generic function `combine(_:)`, which is provided as a syntactic +convenience.) + +While this proposal deprecates the `hashValue` requirement, it doesn't +remove it. Types implementing `Hashable` will continue to provide an +implementation for it, although the implementation may be provided +automatically by compiler synthesis. + +To implement nondeterminism, `Hasher` uses an internal seed value +initialized by the runtime during process startup. The seed is usually +produced by a random number generator, but this may be disabled by +defining the `SWIFT_DETERMINISTIC_HASHING` environment variable with a +value of `1` prior to starting a Swift process. + +## Effect on API resilience + +Replacing `hashValue` with `hash(into:)` moves the responsibility of +choosing a suitable hash function out of `Hashable` implementations +and into the standard library, behind a resiliency boundary. + +`Hasher` is explicitly designed so that future versions of the +standard library will be able to replace the hash function. +`Hashable` implementations compiled for previous versions will +automatically pick up the improved algorithm when linked with the new +release. This includes changing the size or internal layout of the +`Hasher` state itself. + +(We foresee several reasons why we may want to replace the hash +function. For example, we may need to do so if a weakness is +discovered in the current function, to restore the reliability of +`Set` and `Dictionary`. We may also want to tweak the hash function to +adapt it to the special requirements of certain environments (such as +network services), or to generally improve hashing performance.) + +## Alternatives considered + +### Leaving `Hashable` as is + +One option that we considered is to expose `Hasher`, but to leave the +`Hashable` protocol as is. Individual `Hashable` types would be able +to choose whether or not to use it or to roll their own hash +functions. + +We felt this was an unsatisfying approach; the rationale behind this +is explained in the section on [The `hash(into:)` requirement](#hash-into). + +### Defining a new protocol + +There have been several attempts to fix `Hashable` by creating a new +protocol to replace it. For example, there's a prototype +implementation of a [`NewHashable` protocol][h1] in the Swift test +suite. The authors of this proposal have done their share of this, +too: Karoy has previously published an open-source [hashing +package providing an opt-in replacement for `Hashable`][h2], while +Vincent wrote [a detailed pitch for adding a `HashVisitable` protocol +to the standard library][h3a] -- these efforts were direct precursors +to this proposal. + +In these approaches, the new protocol could either be a refinement of +`Hashable`, or it could be unrelated to it. Here is what a refinement +would look like: + +[h1]: https://github.com/apple/swift/blob/swift-4.1-branch/validation-test/stdlib/HashingPrototype.swift +[h2]: https://github.com/attaswift/SipHash +[h3a]: https://blog.definiteloops.com/ha-r-sh-visitors-8c0c3686a46f +[h3b]: https://gist.github.com/regexident/1b8e84974da2243e5199e760508d2d25 + +```swift +protocol Hashable: Equatable { + var hashValue: Int { get } +} + +protocol Hashable2: Hashable { + func hash(into hasher: inout Hasher) +} + +extension Hashable2 { + var hashValue: Int { + var hasher = Hasher() + hash(into: &hasher) + return hasher.finalize() + } +} +``` + +While this is a great approach for external hashing packages, we +believe it to be unsuitable for the standard library. Adding a new +protocol would add a significant new source of user confusion about +hashing, and it would needlessly expand the standard library's API +surface area. + +The new protocol would need to have a new name, but `Hashable` already +has the perfect name for a protocol representing hashable things -- so +we'd need to choose an imperfect name for the "better" protocol. + +While deprecating a protocol requirement is a significant change, we +believe it to be less harmful overall than leaving `Hashable` +unchanged, or trying to have two parallel protocols for the same thing. + +Additionally, adding a second protocol would lead to complications +with `Hashable` synthesis. It's also unclear how `Set` and `Dictionary` +would be able to consistently use `Hasher` for their primary hashing +API. (These problems are not unsolvable, but they may involve adding +special one-off compiler support for the new protocol. For example, we +may want to automatically derive `Hashable2` conformance for all types +that implement `Hashable`.) + +### Making `hash(into:)` generic over a `Hasher` protocol + +It would be nice to allow Swift programmers to define their own hash +functions, and to plug them into any `Hashable` type: + +```swift +protocol Hasher { + func combine(bytes: UnsafeRawBufferPointer) +} +protocol Hashable { + func hash(into hasher: inout H) +} +``` + +However, we believe this would add a degree of generality whose costs +are unjustifiable relative to their potential gain. We expect the +ability to create custom hashers would rarely be exercised. For +example, we do not foresee a need for adding support for custom +hashers in `Set` and `Dictionary`. On the other hand, there are +distinct advantages to standardizing on a single, high-quality hash +function: + +* Adding a generic type parameter to `hash(into:)` would complicate + the `Hashable` API. +* By supporting only a single `Hasher`, we can concentrate our efforts + on making sure it runs fast. For example, we know that the + standard hasher's opaque mutating functions won't ever perform any + retain/release operations, or otherwise mutate any of the + reference types we may encounter during hashing; describing this + fact to the compiler enables optimizations that would not + otherwise be possible. +* Generics in Swift aren't zero-cost abstractions. We may be tempted + to think that we could gain some performance by plugging in a less + sophisticated hash function. This is not necessarily the case -- + support for custom hashers comes with significant overhead that + can easily overshadow the (slight, if any) algorithmic + disadvantage of the standard `Hasher`. + +Note that the proposed non-generic `Hasher` still has full support for +Bloom filters and other data structures that require multiple hash +functions. (To select a different hash function, we just need to +supply a new seed value.) + +### Change `hash(into:)` to take a closure instead of a new type + +A variant of the previous idea is to represent the hasher by a simple +closure taking an integer argument: + +```swift +protocol Hashable { + func hash(into hasher: (Int) -> Void) +} + +extension GridPoint: Hashable { + func hash(into hasher: (Int) -> Void) { + hasher(x) + hasher(y) + } +} +``` + +While this is an attractively minimal API, it has problems with +granularity -- it doesn't allow adding anything less than an +`Int`'s worth of bits to the hash state. + +Additionally, like generics, the performance of such a closure-based +interface would compare unfavorably to `Hasher`, since the compiler +wouldn't be able to guarantee anything about the potential +side-effects of the closure. diff --git a/proposals/0207-containsOnly.md b/proposals/0207-containsOnly.md new file mode 100644 index 0000000000..c82a4d6fa2 --- /dev/null +++ b/proposals/0207-containsOnly.md @@ -0,0 +1,78 @@ +# Add an `allSatisfy` algorithm to `Sequence` + +* Proposal: [SE-0207](0207-containsOnly.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Dave Abrahams](https://github.com/dabrahams) +* Implementation: [apple/swift#15120](https://github.com/apple/swift/pull/15120) +* Status: **Implemented (Swift 4.2)** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0207-add-a-containsonly-algorithm-to-sequence/11686/102) + +## Introduction + +It is common to want to confirm that every element of a sequence equals a +value, or matches certain criteria. Many implementations of this can be found +in use on GitHub. This proposal adds such a method to `Sequence`. + +## Motivation + +You can achieve this in Swift 3 with `contains` by negating both the criteria +and the result: + +```swift +// every element is 9 +!nums.contains { $0 != 9 } +// every element is odd +!nums.contains { !isOdd($0) } +``` + +but these are a readability nightmare. Additionally, developers may not make +the leap to realize `contains` can be used this way, so may hand-roll their own +`for` loop, which could be buggy, or compose other inefficient alternatives: + +```swift +// misses opportunity to bail early +nums.reduce(true) { $0.0 && $0.1 == 9 } +// the most straw-man travesty I could think of... +Set(nums).count == 1 && Set(nums).first == 9 +``` + +## Proposed solution + +Introduce an algorithm on `Sequence` which tests every element and returns +`true` if they all match a given predicate: + +```swift +nums.allSatisfy(isOdd) +``` + +on the basis that it aids readability and avoids performance pitfalls from the composed alternatives. + +## Detailed design + +Add the following extensions to `Sequence`: + +```swift +extension Sequence { + /// Returns a Boolean value indicating whether every element of the sequence + /// satisfies the given predicate. + func allSatisfy(_ predicate: (Element) throws -> Bool) rethrows -> Bool +} +``` + +## Source compatibility + +This change is purely additive so has no source compatibility consequences. + +## Effect on ABI stability + +This change is purely additive so has no ABI stability consequences. + +## Effect on API resilience + +This change is purely additive so has no API resilience consequences. + +## Alternatives considered + +Not adding it, since it can be trivially (if confusingly) composed. + +Much name bikeshedding has ensued. Names considered included `containsOnly` and `all`. `all` has strong precedent in other languages, but was considered unclear at the call site (adding an argument label does not help here given trailing closures omit them). Naming it `all` suggests a renaming of `contains` to `any` would be appropriate – but this is already a fairly heavily-used term elsewhere in Swift, and is less explicit. `containsOnly` is more explicit, and echoes the existing `contains`, but is too easily misread at the use-site as “contains one instance equal to,” especially when considering empty collections. `contains(only:)` was discounted due to trailing closures dropping the argument label, rendering it indistinguishable from `contains(where:)`. diff --git a/proposals/0208-package-manager-system-library-targets.md b/proposals/0208-package-manager-system-library-targets.md new file mode 100644 index 0000000000..7c7d322625 --- /dev/null +++ b/proposals/0208-package-manager-system-library-targets.md @@ -0,0 +1,133 @@ +# Package Manager System Library Targets + +* Proposal: [SE-0208](0208-package-manager-system-library-targets.md) +* Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r), [Daniel Dunbar](https://github.com/ddunbar) +* Review Manager: [Boris Bügling](https://github.com/neonichu) +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift-package-manager#1586](https://github.com/apple/swift-package-manager/pull/1586) +* Bug: [SR-7434](https://bugs.swift.org/browse/SR-7434) + +## Introduction + +This proposal introduces a new type of target "system library target", which +moves the current system-module packages feature from package to target level. + +## Motivation + +The package manager currently supports "system-module packages" which are +intended to adapt a system installed dependency to work with the package +manager. However, this feature is only supported on *package* declarations, +which mean libraries that need it often need to create a separate repository +containing the system package and refer to it as a dependency. + +Our original motivation in forcing system packages to be declared as standalone +packages was to encourage the ecosystem to standardize on them, their names, +their repository locations, and their owners. In practice, this effort did not +work out and it only made the package manager harder to use. + +## Terminology + +**Swift N**: The tools version this proposal is implemented in. + +## Proposed solution + +We propose adding a new "system library target", which would supply the same +metadata needed to adapt system libraries to work with the package manager, but +as a target. This would allow packages to embed these targets with the libraries +that need them. + +We propose to deprecate the legacy system-module package declaration for +packages that are only compatible with Swift N or later. + +## Detailed design + +We will add a new factory method for system library target: + +```swift +extension Target { + public static func systemLibrary( + name: String, + path: String? = nil, + pkgConfig: String? = nil, + providers: [SystemPackageProvider]? = nil + ) -> Target +} +``` + +* This target factory function will only be available if the tools version of + the manifest is greater than or equal to Swift N. + +* During dependency resolution, the package manager will emit a deprecation + warning if there is a legacy system-module package in the package graph and + the tools version of the root package is greater than or equal to Swift N. + +* Currently, the package manage implicitly assumes a dependency on system-module + packages when they are included as a dependency. In the new model, for + consistency with the existing target/product model, clients of a system + package must also specify explicit dependencies from the targets which use the + system library. + +* System library targets _may_ be exported from a package as products. To do so, + they *must* be exported via a library product with exactly one member target + i.e. the system library target. + +## Examples + +For example, an existing package which defines only a system library adaptor +would be described: + +```swift +// swift-tools-version:N +import PackageDescription + +let package = Package( + name: "CZLib", + products: [ + .library(name: "CZLib", targets: ["CZLib"]), + ], + targets: [ + .systemLibrary( + name: "CZLib", + pkgConfig: "zlib", + providers: [ + .brew(["zlib"]), + .apt(["zlib"]), + ] + ) + ] +) +``` + +A similar package which exported a Swift interface for zlib might look like the +following example, which is not expressible today without using a separate +repository. + +```swift +// swift-tools-version:N +import PackageDescription + +let package = Package( + name: "ZLib", + products: [ + .library(name: "ZLib", targets: ["ZLib"]), + ], + targets: [ + .target( + name: "ZLib", + dependencies: ["CZLib"]), + .systemLibrary( + name: "CZLib") + ] +) +``` + +In this case, the system library is not an exported product, and would not be +available to other packages. + +## Impact on existing packages + +None. + +## Alternatives considered + +None. diff --git a/proposals/0209-package-manager-swift-lang-version-update.md b/proposals/0209-package-manager-swift-lang-version-update.md new file mode 100644 index 0000000000..78b2825f32 --- /dev/null +++ b/proposals/0209-package-manager-swift-lang-version-update.md @@ -0,0 +1,166 @@ +# Package Manager Swift Language Version API Update + +* Proposal: [SE-0209](0209-package-manager-swift-lang-version-update.md) +* Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Boris Bügling](https://github.com/neonichu) +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift-package-manager#1563](https://github.com/apple/swift-package-manager/pull/1563) +* Bug: [SR-7464](https://bugs.swift.org/browse/SR-7464) + +## Introduction + +This proposal changes the current `Package.swift` manifest API for declaring for +Swift language versions from freeform Integer array to a new `SwiftVersion` enum +array. + +## Motivation + +The Swift compiler now allows `4.2` as an accepted value of Swift version flag +(`-swift-version`). The `swiftLanguageVersions` API in `Package.swift` currently +accepts an integer array and we need to update this API in order for packages +to declare this language version if required. + +## Proposed solution + +We propose to change the type of `swiftLanguageVersions` property from `[Int]` +to `[SwiftVersion]` in the manifest API used for Swift tools version 4.2. The +`SwiftVersion` will be an enum that contains all known Swift language version +values and will provide an option to declare custom version values: + +```swift +/// Represents the version of the Swift language that should be used for +/// compiling Swift sources in the package. +public enum SwiftVersion { + case v3 + case v4 + case v4_2 + + /// User-defined value of Swift version. + /// + /// The value is passed as-is to Swift compiler's `-swift-version` flag. + case version(String) +} +``` + +The existing package manifests that use `swiftLanguageVersions` will need to +migrate to the new enum when their tools version is updated to 4.2. + +## Detailed design + +The custom version string will be passed as-is to the value of `-swift-version` +flag. The custom version string allows a package to support and make use of new +language versions which are not known to the manifest API of the current tools +version. This is important for packages which want to add support for a newer +Swift language version but also want to retain compatibility with an older +language and tools version, where the new language version isn't known in the +manifest API. + +The package manager will use standard version numbering rules to determine +precedence of language versions. For e.g. 5 > 4.2.1 > 4.2 > 4. + +We will ship a new `PackageDescription` runtime in the Swift 4.2 toolchain. This +runtime will be selected if tools version of a package greater than or equal to +version 4.2. + +When building a package, we will always select the Swift language version that +is most close to (but not exceeding) a valid language version of the Swift +compiler in use. + +If a package does not specify a Swift language version, the tools version of the +manifest will be used to derive the value. + +The `swift package init` command in Swift 4.2 will create packages with +`// swift-tools-version:4.2`. This is orthogonal to this proposal but is +probably worth mentioning. + +### Examples + +Here are some concrete examples: + +* Example 1: + +```swift +// swift-tools-version:4.0 + +import PackageDescription + +let package = Package( + name: "HTTPClient", + ... + swiftLanguageVersions: [4] +) +``` + +The sources will be compiled with `-swift-version 4`. + +* Example 2: + +```swift +// swift-tools-version:4.2 + +import PackageDescription + +let package = Package( + name: "HTTPClient", + ... + swiftLanguageVersions: [.v4, .v4_2] +) +``` + +The sources will be compiled with `-swift-version 4.2`. + +* Example 3: + +```swift +// swift-tools-version:4.0 + +import PackageDescription + +let package = Package( + name: "HTTPClient", + ... + swiftLanguageVersions: [.v4_2, .version("5")] +) +``` + +The package manager will emit an error because this is not possible in +PackageDescription 4 runtime. + +* Example 4: + +```swift +// swift-tools-version:4.2 + +import PackageDescription + +let package = Package( + name: "HTTPClient", + ... + swiftLanguageVersions: [.v4, .version("5")] +) +``` + +The sources will be compiled with `-swift-version 4`. + +## Impact on existing code + +Existing packages will not be impacted but they will not be able to use Swift +language version 4.2 unless they update their manifest to tools version 4.2. + +Once they do update their tools version to 4.2, they will need to migrate to the +new enum if they use `swiftLanguageVersions` property. + +## Alternatives considered + +We considered adding a new property to existing `PackageDescription` runtime so +we don't need to ship additional runtime. However that approach is not very +scalable as it requires us to come up with another name for the APIs that are +breaking. + +We considered making `SwiftVersion` a struct which conforms to +`ExpressibleByIntegerLiteral` and `ExpressibleByStringLiteral`. However, we +think the enum approach is better for several reasons: + +- It is consistent with the C/C++ language version settings. +- The package authors can easily tell which versions are available. +- The package authors don't need to concern themselves with how to spell a valid version. diff --git a/proposals/0210-key-path-offset.md b/proposals/0210-key-path-offset.md new file mode 100644 index 0000000000..1d7f606b67 --- /dev/null +++ b/proposals/0210-key-path-offset.md @@ -0,0 +1,173 @@ +# Add an `offset(of:)` method to `MemoryLayout` + +* Proposal: [SE-0210](0210-key-path-offset.md) +* Authors: [Joe Groff](https://github.com/jckarter) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 4.2)** +* Implementation: [apple/swift#15519](https://github.com/apple/swift/pull/15519) + +## Introduction + +This proposal introduces the ability for Swift code to query the in-memory +layout of stored properties in aggregates using key paths. Like the +`offsetof` macro in C, `MemoryLayout.offset(of:)` returns the distance in +bytes between a pointer to a value and a pointer to one of its fields. + +Swift-evolution thread: [Pitch: “offsetof”-like functionality for stored property key paths](https://forums.swift.org/t/pitch-offsetof-like-functionality-for-stored-property-key-paths/11309/13) + +## Motivation + +Many graphics and math libraries accept input data in arbitrary input formats, +which the user has to describe to the API when setting up their input buffers. +For example, OpenGL lets you describe the layout of vertex buffers using +series of calls to the `glVertexAttribPointer` API. In C, you can use the +standard `offsetof` macro to get the offset of fields within a struct, allowing +you to use the compiler's knowledge of a type's layout to fill out these +function calls: + +```c +// Layout of one of our vertex entries +struct MyVertex { + float position[4]; + float normal[4]; + uint16_t texcoord[2]; +}; + +enum MyVertexAttribute { Position, Normal, TexCoord }; + +glVertexAttribPointer(Position, 4, GL_FLOAT, GL_FALSE, + sizeof(MyVertex), (void*)offsetof(MyVertex, position)); +glVertexAttribPointer(Normal, 4, GL_FLOAT, GL_FALSE, + sizeof(MyVertex), (void*)offsetof(MyVertex, normal)); +glVertexAttribPointer(TexCoord, 2, GL_UNSIGNED_BYTE, GL_TRUE, + sizeof(MyVertex), (void*)offsetof(MyVertex, texcoord)); +``` + +There's currently no equivalent to `offsetof` in Swift, so users of these kinds +of APIs must either write those parts of their code in C or else do Swift +memory layout in their heads, which is error-prone if they ever change their +data layout or the Swift compiler implementation changes its layout algorithm +(which it reserves the right to do). + +## Proposed solution + +Key paths now provide a natural way to refer to fields in Swift. We can add +an API to the `MemoryLayout` type to ask for the offset of the field +represented by a key path. + +## Detailed design + +A new API is added to `MemoryLayout`: + +```swift +extension MemoryLayout { + public static func offset(of key: PartialKeyPath) -> Int? +} +``` + +If the given `key` refers to inline storage within the +in-memory representation of `T`, and the storage is directly +addressable (meaning that accessing it does not need to trigger any +`didSet` or `willSet` accessors, perform any representation changes +such as bridging or closure reabstraction, or mask the value out of +overlapping storage as for packed bitfields), then the return value +is a distance in bytes that can be added to a pointer of type `T` to +get a pointer to the storage accessed by `key`. In other words, if the return +value is non-nil, then these formulations are equivalent: + +```swift +var root: T, value: U +var key: WritableKeyPath +// Mutation through the key path... +root[keyPath: \.key] = value +// ...is exactly equivalent to mutation through the offset pointer... +withUnsafePointer(to: &root) { + (UnsafeMutableRawPointer($0) + MemoryLayout.offset(of: \.key)) + // ...which can be assumed to be bound to the target type + .assumingMemoryBound(to: U.self).pointee = value +} +``` + +One possible set of answers for a Swift struct might look like this: + +```swift +struct Point { + var x, y: Double +} + +struct Size { + var w, h: Double + + var area: Double { return w*h } +} + +struct Rect { + var origin: Point + var size: Size +} + +MemoryLayout.offset(of: \.origin.x) // => 0 +MemoryLayout.offset(of: \.origin.y) // => 8 +MemoryLayout.offset(of: \.size.w) // => 16 +MemoryLayout.offset(of: \.size.h) // => 24 +MemoryLayout.offset(of: \.size.area) // => nil +``` + +In Swift today, only key paths that refer to +struct fields would support taking their offset, though if support for tuple +elements in key paths were added in the future, tuple elements could +as well. Class properties are always stored out-of-line, and require runtime +exclusivity checking to access, so their offsets would not be available by this +mechanism. + +## Source compatibility + +This is an additive change to the API of `MemoryLayout`. + +## Effect on ABI stability + +`KeyPath` objects already encode the offset information for stored properties +necessary to implement this, so this has no additional demands from the ABI. + +## Effect on API resilience + +Clients of an API could potentially use this functionality to dynamically +observe whether a public property is implemented as a stored property from +outside of the module. If a client assumes that a property will always be +stored by force-unwrapping the optional result of `offset(of:)`, that could +lead to compatibility problems if the library author changes the property to +computed in a future library version. Client code using offsets should be +careful not to rely on the stored-ness of properties in types they don't +control. + +## Alternatives considered + +Instead of a new static method on `MemoryLayout`, this functionality could also +be expressed as an `offset` property on `KeyPath`. All of the information +necessary to answer the offset question is in the `KeyPath` value itself. +Nonetheless, `MemoryLayout` seems like the natural place to put this API. + +A related API that might be useful to build on top of this functionality would +be to add methods to `UnsafePointer` and `UnsafeMutablePointer` for projecting +a pointer to a field from a pointer to a base value, for example: + +```swift +extension UnsafePointer { + subscript(field: KeyPath) -> UnsafePointer { + return (UnsafeRawPointer(self) + MemoryLayout.offset(of: field)) + .assumingMemoryBound(to: Field.self) + } +} + +extension UnsafeMutablePointer { + subscript(field: KeyPath) -> UnsafePointer { + return (UnsafeRawPointer(self) + MemoryLayout.offset(of: field)) + .assumingMemoryBound(to: Field.self) + } + + subscript(field: WritableKeyPath) -> UnsafeMutablePointer { + return (UnsafeMutableRawPointer(self) + MemoryLayout.offset(of: field)) + .assumingMemoryBound(to: Field.self) + } +} +``` diff --git a/proposals/0211-unicode-scalar-properties.md b/proposals/0211-unicode-scalar-properties.md new file mode 100644 index 0000000000..0da039fdf3 --- /dev/null +++ b/proposals/0211-unicode-scalar-properties.md @@ -0,0 +1,410 @@ +# Add Unicode Properties to `Unicode.Scalar` + +* Proposal: [SE-0211](0211-unicode-scalar-properties.md) +* Author: [Tony Allevato](https://github.com/allevato) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#15593](https://github.com/apple/swift/pull/15593) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0211-add-unicode-properties-to-unicode-scalar/13857), [Update](https://forums.swift.org/t/update-se-0211-add-unicode-properties-to-unicode-scalar/59727) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/9b1c670206052f5c94bcb20df1c30c27a06e9755/proposals/0211-unicode-scalar-properties.md) + +## Introduction + +We propose adding a number of properties to the `Unicode.Scalar` type to support +both common and advanced text processing use cases, filling in a number of gaps +in Swift's text support compared to other programming languages. + +Swift-evolution thread: [Adding Unicode properties to UnicodeScalar/Character](https://forums.swift.org/t/adding-unicode-properties-to-unicodescalar-character/9310) + +## Motivation + +The Swift `String` type, and related types like `Character` and +`Unicode.Scalar`, provide very rich support for Unicode-correct operations. +String comparisons are normalized, grapheme cluster boundaries are automatically +detected, and string contents can be easily accessed in terms of grapheme +clusters, code points, and UTF-8 and -16 code units. + +However, when drilling down to lower levels, like individual code points (i.e., +`Unicode.Scalar` elements), the current APIs are missing a number of fundamental +features available in other programming languages. `Unicode.Scalar` lacks the +ability to ask whether a scalar is upper/lowercase or what its upper/lowercase +mapping is, if it is a whitespace character, and so forth. + +Without pulling in third-party code, users can currently import the +`Darwin/Glibc` module and access C functions like `isspace`, but these only work +with ASCII characters. + +### Issues Linking with ICU + +The Swift standard library uses the system's ICU libraries to implement its +Unicode support. A third-party developer may expect that they could also link +their application directly to the system ICU to access the functionality that +they need, but this proves problematic on both Apple and Linux platforms. + +On Apple operating systems, `libicucore.dylib` is built with function renaming +disabled (function names lack the `_NN` version number suffix). This makes it +fairly straightforward to import the C APIs and call them from Swift without +worrying about which version the operating system is using. + +Unfortunately, `libicucore.dylib` is considered to be private API for +submissions to the App Store, so applications doing this will be rejected. +Instead, users must built their own copy of ICU from source and link that into +their applications. This is significant overhead. + +On Linux, system ICU libraries are built with function renaming enabled (the +default), so function names have the `_NN` version number suffix. Function +renaming makes it more difficult to use these APIs from Swift; even though the C +header files contain `#define`s that map function names like `u_foo_59` to +`u_foo`, these `#define`s are not imported into Swift—only the suffixed function +names are available. This means that Swift bindings would be fixed to a specific +version of the library without some other intermediary layer. Again, this is +significant overhead. + +Therefore, this proposal not only fills in important gaps in the standard +library's capabilities, but removes a significant pain point for users who may +try to access that functionality through other means. + +## Proposed solution + +We propose adding a nested struct, `Unicode.Scalar.Properties`, which will +encapsulate many of the properties that the Unicode specification defines on +scalars. Supporting types, such as enums representing the values of certain +properties, will also be added to the `Unicode` enum "namespace." + +### Scope of This Proposal + +This proposal is restricted, by design, to add functionality to `Unicode.Scalar` +only. While we believe that some of the properties described here (and others) +would be valuable on `Character` as well, we have intentionally saved those for +a future proposal in order to keep this one small and focused. Such a future +proposal would likely depend on the design and implementation herein. + +## Detailed design + +The code snippets below reflect an elided sketch of the proposed public API +only. Full details can be found in the implementation pull request. + +In general, the names of the properties inside the `Properties` struct are +derived directly from the names of the properties as they are defined in the +[Unicode Standard](http://unicode.org/reports/tr44/#Property_Index). + +```swift +extension Unicode.Scalar { + + // NOT @_fixed_layout + public struct Properties { + // Remaining API is defined in the subsections below. + } + + /// The value that encapsulates the properties exposed. + public var properties: Properties { get } +} +``` + +### Boolean Properties + +Each of the Boolean properties in the first block below would be implemented by +calling ICU's `u_hasBinaryProperty` function. The official Unicode name of each +property is indicated by the comment next to the computed properties below. + +We propose supporting all of the Boolean properties that are currently available +using ICU's `u_hasBinaryProperty` that correspond to properties in the Unicode +Standard (but _not_ ICU-specific properties), with the following exceptions: + +* `Grapheme_Link` is omitted because it is deprecated and equivalent to + canonical combining class 9. +* `Hyphen` is omitted because is deprecated in favor of the `Line_Break` + property. + +```swift +extension Unicode.Scalar.Properties { + + public var isAlphabetic: Bool { get } // Alphabetic + public var isASCIIHexDigit: Bool { get } // ASCII_Hex_Digit + public var isBidiControl: Bool { get } // Bidi_Control + public var isBidiMirrored: Bool { get } // Bidi_Mirrored + public var isDash: Bool { get } // Dash + public var isDefaultIgnorableCodePoint: Bool { get } // Default_Ignorable_Code_Point + public var isDeprecated: Bool { get } // Deprecated + public var isDiacritic: Bool { get } // Diacritic + public var isExtender: Bool { get } // Extender + public var isFullCompositionExclusion: Bool { get } // Full_Composition_Exclusion + public var isGraphemeBase: Bool { get } // Grapheme_Base + public var isGraphemeExtend: Bool { get } // Grapheme_Extend + public var isHexDigit: Bool { get } // Hex_Digit + public var isIDContinue: Bool { get } // ID_Continue + public var isIDStart: Bool { get } // ID_Start + public var isIdeographic: Bool { get } // Ideographic + public var isIDSBinaryOperator: Bool { get } // IDS_Binary_Operator + public var isIDSTrinaryOperator: Bool { get } // IDS_Trinary_Operator + public var isJoinControl: Bool { get } // Join_Control + public var isLogicalOrderException: Bool { get } // Logical_Order_Exception + public var isLowercase: Bool { get } // Lowercase + public var isMath: Bool { get } // Math + public var isNoncharacterCodePoint: Bool { get } // Noncharacter_Code_Point + public var isQuotationMark: Bool { get } // Quotation_Mark + public var isRadical: Bool { get } // Radical + public var isSoftDotted: Bool { get } // Soft_Dotted + public var isTerminalPunctuation: Bool { get } // Terminal_Punctuation + public var isUnifiedIdeograph: Bool { get } // Unified_Ideograph + public var isUppercase: Bool { get } // Uppercase + public var isWhitespace: Bool { get } // Whitespace + public var isXIDContinue: Bool { get } // XID_Continue + public var isXIDStart: Bool { get } // XID_Start + public var isCaseSensitive: Bool { get } // Case_Sensitive + public var isSentenceTerminal: Bool { get } // Sentence_Terminal (S_Term) + public var isVariationSelector: Bool { get } // Variation_Selector + public var isNFDInert: Bool { get } // NFD_Inert + public var isNFKDInert: Bool { get } // NFKD_Inert + public var isNFCInert: Bool { get } // NFC_Inert + public var isNFKCInert: Bool { get } // NFKC_Inert + public var isSegmentStarter: Bool { get } // Segment_Starter + public var isPatternSyntax: Bool { get } // Pattern_Syntax + public var isPatternWhitespace: Bool { get } // Pattern_White_Space + public var isCased: Bool { get } // Cased + public var isCaseIgnorable: Bool { get } // Case_Ignorable + public var changesWhenLowercased: Bool { get } // Changes_When_Lowercased + public var changesWhenUppercased: Bool { get } // Changes_When_Uppercased + public var changesWhenTitlecased: Bool { get } // Changes_When_Titlecased + public var changesWhenCaseFolded: Bool { get } // Changes_When_Casefolded + public var changesWhenCaseMapped: Bool { get } // Changes_When_Casemapped + public var changesWhenNFKCCaseFolded: Bool { get } // Changes_When_NFKC_Casefolded + public var isEmoji: Bool { get } // Emoji + public var isEmojiPresentation: Bool { get } // Emoji_Presentation + public var isEmojiModifier: Bool { get } // Emoji_Modifier + public var isEmojiModifierBase: Bool { get } // Emoji_Modifier_Base +} +``` + +### Case Mappings + +The properties below provide full case mappings for scalars. Since a handful of +mappings result in multiple scalars (e.g., "ß" uppercases to "SS"), these +properties are `String`-valued, not `Unicode.Scalar`. + +These properties are also common enough that they could be reasonably hoisted +out of `Unicode.Scalar.Properties` and made into instance properties directly on +`Unicode.Scalar`. + +```swift +extension Unicode.Scalar.Properties { + + public var lowercaseMapping: String { get } // u_strToLower + public var titlecaseMapping: String { get } // u_strToTitle + public var uppercaseMapping: String { get } // u_strToUpper +} +``` + +### Identification and Classification + +We add the following properties for the purposes of identifying and categorizing +scalars. + +The `Canonical_Combining_Class` property is somewhat unique in that the Unicode +standard provides names for some, but not all, of the 255 valid values. We +choose to represent this in Swift as a `RawRepresentable` `struct` that wraps +the raw integer value, while still being `Comparable` for the purposes of +implementing manual decomposition logic if necessary, and still providing the +named values through `static let` constants. + +```swift +extension Unicode.Scalar.Properties { + + /// Corresponds to the `Age` Unicode property, when a code point was first + /// defined. + public var age: Unicode.Version? { get } + + /// Corresponds to the `Name` Unicode property. + public var name: String? { get } + + /// Corresponds to the `Name_Alias` Unicode property. + public var nameAlias: String? { get } + + /// Corresponds to the `General_Category` Unicode property. + public var generalCategory: Unicode.GeneralCategory { get } + + /// Corresponds to the `Canonical_Combining_Class` Unicode property. + public var canonicalCombiningClass: Unicode.CanonicalCombiningClass { get } +} + +extension Unicode { + + /// Represents the version of Unicode in which a scalar was introduced. + public typealias Version = (major: Int, minor: Int) + + /// General categories returned by + /// `Unicode.Scalar.Properties.generalCategory`. Listed along with their + /// two-letter code. + public enum GeneralCategory { + case uppercaseLetter // Lu + case lowercaseLetter // Ll + case titlecaseLetter // Lt + case modifierLetter // Lm + case otherLetter // Lo + + case nonspacingMark // Mn + case spacingMark // Mc + case enclosingMark // Me + + case decimalNumber // Nd + case letterlikeNumber // Nl + case otherNumber // No + + case connectorPunctuation //Pc + case dashPunctuation // Pd + case openPunctuation // Ps + case closePunctuation // Pe + case initialPunctuation // Pi + case finalPunctuation // Pf + case otherPunctuation // Po + + case mathSymbol // Sm + case currencySymbol // Sc + case modifierSymbol // Sk + case otherSymbol // So + + case spaceSeparator // Zs + case lineSeparator // Zl + case paragraphSeparator // Zp + + case control // Cc + case format // Cf + case surrogate // Cs + case privateUse // Co + case unassigned // Cn + } + + public struct CanonicalCombiningClass: + Comparable, Hashable, RawRepresentable + { + public static let notReordered = CanonicalCombiningClass(rawValue: 0) + public static let overlay = CanonicalCombiningClass(rawValue: 1) + public static let nukta = CanonicalCombiningClass(rawValue: 7) + public static let kanaVoicing = CanonicalCombiningClass(rawValue: 8) + public static let virama = CanonicalCombiningClass(rawValue: 9) + public static let attachedBelowLeft = CanonicalCombiningClass(rawValue: 200) + public static let attachedBelow = CanonicalCombiningClass(rawValue: 202) + public static let attachedAbove = CanonicalCombiningClass(rawValue: 214) + public static let attachedAboveRight = CanonicalCombiningClass(rawValue: 216) + public static let belowLeft = CanonicalCombiningClass(rawValue: 218) + public static let below = CanonicalCombiningClass(rawValue: 220) + public static let belowRight = CanonicalCombiningClass(rawValue: 222) + public static let left = CanonicalCombiningClass(rawValue: 224) + public static let right = CanonicalCombiningClass(rawValue: 226) + public static let aboveLeft = CanonicalCombiningClass(rawValue: 228) + public static let above = CanonicalCombiningClass(rawValue: 230) + public static let aboveRight = CanonicalCombiningClass(rawValue: 232) + public static let doubleBelow = CanonicalCombiningClass(rawValue: 233) + public static let doubleAbove = CanonicalCombiningClass(rawValue: 234) + public static let iotaSubscript = CanonicalCombiningClass(rawValue: 240) + + public let rawValue: UInt8 + + public init(rawValue: UInt8) + } +} +``` + +### Numerics + +Many Unicode scalars have associated numeric values. These are not only the +common digits zero through nine, but also vulgar fractions and various other +linguistic characters and ideographs that have an innate numeric value. These +properties are exposed below. They can be useful for determining whether +segments of text contain numbers or non-numeric data, and can also help in the +design of algorithms to determine the values of such numbers. + +```swift +extension Unicode.Scalar.Properties { + + /// Corresponds to the `Numeric_Type` Unicode property. + public var numericType: Unicode.NumericType? + + /// Corresponds to the `Numeric_Value` Unicode property. + public var numericValue: Double? +} + +extension Unicode { + + public enum NumericType { + case decimal + case digit + case numeric + } +} +``` + +## Source compatibility + +These changes are strictly additive. This proposal does not affect source +compatibility. + +## Effect on ABI stability + +These changes are strictly additive. This proposal does not affect the ABI of +existing language features. + +## Effect on API resilience + +The `Unicode.Scalar.Properties` struct is currently defined as a resilient +(non-`@_fixed_layout`) struct whose only stored property in the initial +implementation is the integer value of the scalar whose properties are being +retrieved. All other properties are computed properties, and new properties can +be added without breaking the ABI. + +In the future, we may choose to cache certain properties if data show that they +are more frequently accessed than others and that there would be a meaningful +performance improvement by computing them early. It is too soon to make such a +determination now, however. + +## Alternatives considered + +### API Designs + +We considered other representations for the Boolean properties of a scalar: + +* A `BooleanProperty` enum with a case for each property, and a + `Unicode.Scalar.hasProperty` method used to query it. This is very close to + the underlying ICU C APIs and does not bloat the `Unicode.Scalar` API, but + makes the kinds of queries users would commonly make less discoverable. +* A `Unicode.Scalar.properties` property whose type conforms to `OptionSet`. + This would allow us to use the underlying property enum constants as the + bit-shifts for the option set values, but there are already 64 Boolean + properties defined by ICU. Since the underlying integral type is part of the + public API/ABI of the option set, we would not be able to change it in the + future without breaking compatibility. +* A `Unicode.Scalar.properties` property whose type is a `Set`, + but we would not be able to form this collection without querying all Boolean + properties upon any access (the `OptionSet` solution above suffers the same + problem). This would be needlessly inefficient in almost all usage. + +We feel that by putting the properties into a separate +`Unicode.Scalar.Properties` struct, the large number of advanced properties does +not contribute to bloat of the main `Unicode.Scalar` API, and allows us to +cleanly represent not only Boolean properties but other types of properties with +ease. + +### Naming + +The names of the Boolean properties are all of the form `is`, with the exception of a small number of properties whose names already +start with indicative verb forms and read as assertions (e.g., +`changesWhenUppercased`). This leads to some technical and/or awkward property +names, like `isXIDContinue`. + +We considered modifying these names to make them read more naturally like other +Swift APIs; for example, `extendsPrecedingScalar` instead of `isExtender`. +However, since these properties are intended for advanced users who are likely +already somewhat familiar with the Unicode Standard and its definitions, we +decided to keep the names directly derived from the Standard, which makes them +more discoverable to the intended audience. + +### `isDefined` Property + +The original version of this proposal also included a Boolean `isDefined` +property that was equivalent to the `u_isdefined` function from ICU, which +would evaluate to true for exactly the code points that are assigned (those +with general category other than "Cn"). In Swift, however, this would be +redundant and was thus dropped from the final implementation; its value would +differ from `Unicode.Scalar.Properties.generalCategory != .unassigned` only +for surrogate code points, which cannot be created as `Unicode.Scalar` values. diff --git a/proposals/0212-compiler-version-directive.md b/proposals/0212-compiler-version-directive.md new file mode 100644 index 0000000000..a05a4e06a0 --- /dev/null +++ b/proposals/0212-compiler-version-directive.md @@ -0,0 +1,108 @@ +# Compiler Version Directive + +* Proposal: [SE-0212](0212-compiler-version-directive.md) +* Author: [David Hart](https://github.com/hartbit) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Implementation: [apple/swift#15977](https://github.com/apple/swift/pull/15977) +* Status: **Implemented (Swift 4.2)** + +## Introduction + +This proposal introduces a `compiler` directive that is syntactically equivalent to the `#if swift` version check but checks against the version of the compiler, regardless of which compatibility mode it's currently running in. + +## Motivation + +The `#if swift` check allows conditionally compiling code depending on the version of the language. Prior to Swift 4, the version of the compiler and the language were one and the same. But since Swift 4, the compiler can run in a compatibility mode for previous Swift versions, introducing a new version dimension. To support code across multiple compiler versions and compatibility modes, extra language versions are regularly introduced to represent old language versions running under compatibility mode. + +For example, the release of Swift 4 introduced a Swift 3.2 language version representing the Swift 4 compiler in version 3 compatibility mode. Here is the current language matrix, as well as guesses as to what those versions will be for Swift 5.0 and 5.1. + +| Swift | --swift-version 3 | --swift-version 4 | --swift-version 4.2 | --swift-version 5 | +|:----- |:----------------- |:----------------- |:------------------- |:----------------- | +| 3.0 | N/A | N/A | N/A | N/A | +| 3.1 | N/A | N/A | N/A | N/A | +| 4.0 | 3.2 | 4.0 | N/A | N/A | +| 4.1 | 3.3 | 4.1 | N/A | N/A | +| 4.2 | 3.4 | 4.1.50 | 4.2 | N/A | +| 5.0 | 3.5 | 4.1.51 | 4.3 | 5.0 | +| 5.1 | 3.6 | 4.1.52 | 4.4 | 5.1 | + +This solution is problematic for several reasons: + +* It creates a quadratic growth in the number of Swift versions for each new compatibility version. +* Conditionally compiling for a version of the compiler, regardless of the compatibility mode, is difficult and error prone: + +```swift +#if swift(>=4.1) || (swift(>=3.3) && !swift(>=4.0)) +// Code targeting the Swift 4.1 compiler and above. +#endif + +#if swift(>=4.1.50) || (swift(>=3.4) && !swift(>=4.0)) +// Code targeting the Swift 4.2 compiler and above. +#endif + +#if swift(>=5.0) || (swift(>=4.1.50) && !swift(>=4.2)) || (swift(>=3.5) && !swift(>=4.0)) +// Code targeting the Swift 5.0 compiler and above. +#endif +``` + +## Proposed solution + +This proposal suggests: + +* introducing a new `compiler` directive that is syntactically equivalent to the `swift` directive but checks against the version of the compiler, +* stop bumping old Swift versions when new versions are introduced. + +This will simplify future Swift versions by stopping the artificial growth of old language versions: + +| Invocation | Compiler Version | Language Version | +|:------------------------- |:---------------- |:---------------- | +| 3.0 | N/A | 3.0 | +| 3.1 | N/A | 3.1 | +| 4.0 | N/A | 4.0 | +| 4.0 (--swift-version 3) | N/A | 3.2 | +| 4.1 | N/A | 4.1 | +| 4.1 (--swift-version 3) | N/A | 3.3 | +| 4.2 | 4.2 | 4.2 | +| 4.2 (--swift-version 3) | 4.2 | 3.3 | +| 4.2 (--swift-version 4) | 4.2 | 4.1 | +| 5.0 | 5.0 | 5.0 | +| 5.0 (--swift-version 3) | 5.0 | 3.3 | +| 5.0 (--swift-version 4) | 5.0 | 4.1 | +| 5.0 (--swift-version 4.2) | 5.0 | 4.2 | +| 5.1 | 5.1 | 5.1 | +| 5.1 (--swift-version 3) | 5.1 | 3.3 | +| 5.1 (--swift-version 4) | 5.1 | 4.1 | +| 5.1 (--swift-version 4.2) | 5.1 | 4.2 | + +This change is possible because it retains the ability to conditionally compile code targeting a compiler in compatibility mode: + +```swift +#if swift(>=4.1) && compiler(>=5.0) +// Code targeting the Swift 5.0 compiler and above in --swift-version 4 mode and above. +#endif +``` + +It will also greatly simplify conditional compilation based on compiler version alone: + +```swift +#if swift(>=4.1) || (swift(>=3.3) && !swift(>=4.0)) +// Code targeting the Swift 4.1 compiler and above. +// This can't change because it needs to continue working with older compilers. +#endif + +#if compiler(>=4.2) +// Code targeting the Swift 4.2 compiler and above. +#endif + +#if compiler(>=5.0) +// Code targeting the Swift 5.0 compiler and above. +#endif +``` + +## Impact on existing code + +This is a purely additive change and will have no impact on existing code. + +## Alternatives considered + +No other alternative naming was considered for this new directive. diff --git a/proposals/0213-literal-init-via-coercion.md b/proposals/0213-literal-init-via-coercion.md new file mode 100644 index 0000000000..9fe4ecbd98 --- /dev/null +++ b/proposals/0213-literal-init-via-coercion.md @@ -0,0 +1,95 @@ +# Literal initialization via coercion + +* Proposal: [SE-0213](0213-literal-init-via-coercion.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#17860](https://github.com/apple/swift/pull/17860) + +## Introduction + +`T(literal)` should construct T using the appropriate literal protocol if possible. + +Swift-evolution thread: [Literal initialization via coercion](https://forums.swift.org/t/literal-initialization-via-coercion/11251) + +## Motivation + +Currently types conforming to literal protocols are type-checked using regular +initializer rules, which means that for expressions like `UInt32(42)` the +type-checker is going to look up a set of available initializer choices and +attempt them one-by-one trying to deduce the best solution. + +This is not always a desired behavior when it comes to numeric and +other literals, because it means that argument is going to be type-checked +separately (most likely to some default literal type like `Int`) and passed +to an initializer call. At the same time coercion behavior would treat +the expression above as `42 as UInt32` where `42` is ascribed to be `UInt32` +and constructed without an intermediate type. + +## Proposed solution + +The proposed change makes all initializer expressions involving literal types +behave like coercion of literal to specified type if such type conforms to the +expected literal protocol. As a result expressions like `UInt64(0xffff_ffff_ffff_ffff)`, +which result in compile-time overflow under current rules, become valid. It +also simplifies type-checker logic and leads to speed-up in some complex expressions. + +This change also makes some of the errors which currently only happen at runtime +become compile-time instead e.g. `Character("ab")`. + +## Detailed design + +Based on the [previous discussion on this topic](https://forums.swift.org/t/proposal-t-literal-should-construct-t-using-the-appropriate-literal-protocol-if-possible/2861) here is the formal typing rule: + +``` +Given a function call expression of the form `A(B)` (that is, an expr-call with a single, +unlabelled argument) where B is a literal expression, if `A` has type `T.Type` +for some type `T` and there is a declared conformance of `T` to an appropriate literal protocol +for `B`, then `A` is directly constructed using `init` witness to literal protocol +(as if the expression were written "B as A"). +``` + +This behavior could be avoided by spelling initializer call verbosely e.g. `UInt32.init(42)`. + +Implementation is going to transform `CallExpr` with `TypeExpr` as a applicator into +implicit `CoerceExpr` if the aforementioned typing rule holds before forming constraint system. + +## Source compatibility + +This is a source breaking change because it’s possible to declare a conformance to +a literal protocol and also have a failable initializer with the same parameter type: + +```swift +struct Q: ExpressibleByStringLiteral { + typealias StringLiteralType = String + + var question: String + + init?(_ possibleQuestion: StringLiteralType) { + return nil + } + + init(stringLiteral str: StringLiteralType) { + self.question = str + } +} + +_ = Q("ultimate question") // 'nil' +_ = "ultimate question" as Q // Q(question: 'ultimate question') +``` + +Although such situations are possible, we consider them to be quite rare +in practice. FWIW, none were found in the compatibility test suite. + + +## Effect on ABI stability + +Does not affect ABI stability + +## Effect on API resilience + +Does not affect API resilience + +## Alternatives considered + +Not to make this change. diff --git a/proposals/0214-DictionaryLiteral.md b/proposals/0214-DictionaryLiteral.md new file mode 100644 index 0000000000..c5d06b0f9c --- /dev/null +++ b/proposals/0214-DictionaryLiteral.md @@ -0,0 +1,102 @@ +# Renaming the `DictionaryLiteral` type to `KeyValuePairs` + +* Proposal: [SE-0214](0214-DictionaryLiteral.md) +* Authors: [Erica Sadun](https://github.com/erica), [Chéyo Jiménez](https://github.com/masters3d) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#16577](https://github.com/apple/swift/pull/16577) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0214-renaming-the-dictionaryliteral-type-to-keyvaluepairs/13661) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/12315c44dd6b36fec924f4f6c30f48d8784ae4cc/proposals/0214-DictionaryLiteral.md) + +## Introduction + +This proposal renames the confusing and misnamed [`DictionaryLiteral`](https://github.com/apple/swift/blob/c25188bafd1c775d4ceecc4a795f614f00451bf9/stdlib/public/core/Mirror.swift#L646) type to `KeyValuePairs`. This type is neither a dictionary nor a literal. It is a list of key-value pairs. + +There is no strong motivation to deprecate. The type does not produce active harm. Instead, it adds measurable (if small) utility and will be part of the ABI. A sensible renaming mitigates the most problematic issue with the type. + +*This proposal was discussed in the Swift Forums on the [100% bikeshed topic: DictionaryLiteral](https://forums.swift.org/t/100-bikeshed-topic-dictionaryliteral/7385) thread.* + +## Motivation + +This proposal renames the standard library's `DictionaryLiteral` before the Swift Language declares ABI stability. The type is confusingly misnamed. A `DictionaryLiteral` is neither a dictionary nor a literal. + +* It offers no key-based value lookup. +* It does not represent a fixed value in source code. + +It seems reasonable to give the type to a better name that fits its role and purpose. + +#### Current Use: + +`DictionaryLiteral` establishes the `Mirror` API's children: + +``` +public init( + _ subject: Subject, + children: DictionaryLiteral, + displayStyle: Mirror.DisplayStyle? = default, + ancestorRepresentation: Mirror.AncestorRepresentation = default +) +``` + +* This implementation depends on `DictionaryLiteral`'s continued existence. +* [The `@dynamicCallable` proposal](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0216-dynamic-callable.md) will provide another use case for this type. + +Even when narrowly used, a type's reach is no longer a sufficient reason to deprecate it or remove it from the language. Absent *active harm*, source stability takes precedence. In this case, the `DictionaryLiteral` type causes no measurable harm beyond API sprawl and the issues with its name. The latter is easily fixed. + +#### Current Limits: + +The type's doc comments note an inefficient lookup implementation. This issue can be resolved in future Swift releases if needed: + +``` +/// Some operations that are efficient on a dictionary are slower when using +/// `DictionaryLiteral`. In particular, to find the value matching a key, you +/// must search through every element of the collection. The call to +/// `index(where:)` in the following example must traverse the whole +/// collection to find the element that matches the predicate +``` + +#### Utility: + +The type's support of duplicate keys could become be a feature for scanning key-value pairs: + +``` +/// You initialize a `DictionaryLiteral` instance using a Swift dictionary +/// literal. Besides maintaining the order of the original dictionary literal, +/// `DictionaryLiteral` also allows duplicates keys. +``` + +This key-value pair processing might support custom initializers. It allows duplicate keys and preserves declaration order, which are both reasonably useful features. + +## Detailed Design + +`DictionaryLiteral` is renamed to `KeyValuePairs`. A typealias preserves the old name for compatibility but can be deprecated as of Swift 5.0. + +This name was extensively bikeshedded on the [Swift Forum thread](https://forums.swift.org/t/100-bikeshed-topic-dictionaryliteral/7385) before proposal. The runner up name was `KeyValueArray`. + +## Source compatibility + +The old name can live on indefinitely via a typealias (which has no ABI consequences, so could be retired at a later date once everyone has had plenty of time to address the deprecation warnings). + +Removing it as not carrying its weight (and instead using `[(Key,Value)]`, which is basically what it’s a wrapper for) is probably off the table for reasons of source stability. + +## Effect on ABI stability + +Should be decided before ABI Stability is declared. + +## Effect on API resilience + +None. + +## Alternatives and Future Directions + +* This proposal does not change syntax. It processes an ordered immutable list of pairs declared using `[:]` syntax. This syntax offers better visual aesthetics than `[(,)]`. + +* Swift cannot yet replace `DictionaryLiteral` with conditional conformance using `Array: ExpressibleByDictionaryLiteral where Element = (Key,Value)` because [parameterized extensions](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#parameterized-extensions) are not yet available. Further, creating an array from a dictionary literal may be unnecessarily confusing. + +* This proposal does not deprecate and remove the type, as `Mirror` relies on its existence. This proposal does not recommend the removal of the `Mirror` type as third party custom debugging features rely on this feature. + +* A forum discussion considered a one-time split of the standard library, creating a "SwiftDeprecated" module that could eventually be phased out. That idea lies outside the scope of this proposal and involves a tradeoff between sunsetting APIs in the future for a slight reduction in size of today's standard library. Most applications will not use these APIs, whether such an approach is taken or not. + +## Acknowledgments + +Thanks, Ben Cohen, for pointing out this problem and starting the forum thread to arrive at a better name. Thanks Chris Lattner and Ted Kremenek for design direction. diff --git a/proposals/0215-conform-never-to-hashable-and-equatable.md b/proposals/0215-conform-never-to-hashable-and-equatable.md new file mode 100644 index 0000000000..43828f15ba --- /dev/null +++ b/proposals/0215-conform-never-to-hashable-and-equatable.md @@ -0,0 +1,87 @@ +# Conform `Never` to `Equatable` and `Hashable` + +* Proposal: [SE-0215](0215-conform-never-to-hashable-and-equatable.md) +* Author: [Matt Diephouse](https://github.com/mdiep) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0215-conform-never-to-equatable-and-hashable/13586/45) +* Implementation: [apple/swift#16857](https://github.com/apple/swift/pull/16857) + +## Introduction +Extend `Never` so it conforms to `Equatable` and `Hashable`. + +Swift-evolution thread: [Conform Never to Equatable and Hashable](https://forums.swift.org/t/conform-never-to-equatable-and-hashable/12934) + +## Motivation +`Never` is very useful for representing impossible code. Most people are familiar with it as the return type of functions like `fatalError`, but `Never` is also useful when working with generic classes. + +For example, a `Result` type might use `Never` for its `Value` to represent something that _always_ errors or use `Never` for its `Error` to represent something that _never_ errors. + +Conditional conformances to `Equatable` and `Hashable` are also very useful when working with `enum`s so you can test easily or work with collections. + +But those don’t play well together. Without conformance to `Equatable` and `Hashable`, `Never` disqualifies your generic type from being `Equatable` and `Hashable`. + +## Proposed solution +The standard library should add `Equatable` and `Hashable` implementations for `Never`: + +```swift +extension Never: Equatable { + public static func == (lhs: Never, rhs: Never) -> Bool { + switch (lhs, rhs) { + } + } +} + +extension Never: Hashable { + public func hash(into hasher: inout Hasher) { + } +} +``` + +## Detailed design +The question that most often comes up is how `Never` should implement `Equatable`. How do you compare to `Never` values? + +But there are no `Never` values; it’s an uninhabitable type. Thankfully Swift makes this easy. By switching over the left- and right-hand sides, Swift correctly notices that there are no missing `case`s. Since there are no missing `case`s and every `case` returns a `Bool`, the function compiles. + +The new `Hashable` design makes its implementation even easier: the function does nothing. + +## Source compatibility +Existing applications may have their own versions of these conformances. In this case, Swift will give a redundant conformance error. + +## Effect on ABI stability +None. + +## Effect on API resilience +None. + +## Alternatives considered +### Make `Never` conform to _all_ protocols +As a bottom type, `Never` could conceivably conform to every protocol automatically. This would have some advantages and might be ideal, but would require a lot more work to determine the design and implement the behavior. + +### Don’t include this functionality in the standard library +This creates a significant headache—particularly for library authors. Since redundant conformance would be an error, the community would need to settle on a de facto library to add this conformance. + +### Require generic types to add conditional conformances with `Never` +An example `Result` type could manually add `Equatable` and `Hashable` implementations for `Never`s: + +```swift +extension Result: Equatable where Value == Never, Error: Equatable { + … +} + +extension Result: Equatable where Value: Hashable, Error == Never { + … +} + +extension Result: Equatable where Value == Never, Error == Never { + … +} +``` + +Adding so many extra conditional conformances is an unreasonable amount of work. + +### Amendment from Core Team + +As part of the [review decision](https://forums.swift.org/t/se-0215-conform-never-to-equatable-and-hashable/13586/45) from the Core Team +when accepting this proposal, in addition to `Equatable` and `Hashable` conformances being added to `Never` this proposal +now also includes adding conformances to the `Comparable` and `Error` protocols as well. diff --git a/proposals/0216-dynamic-callable.md b/proposals/0216-dynamic-callable.md new file mode 100644 index 0000000000..a407d18d97 --- /dev/null +++ b/proposals/0216-dynamic-callable.md @@ -0,0 +1,477 @@ +# Introduce user-defined dynamically "callable" types + +* Proposal: [SE-0216](0216-dynamic-callable.md) +* Authors: [Chris Lattner](https://github.com/lattner), [Dan Zheng](https://github.com/dan-zheng) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Implementation: [apple/swift#20305](https://github.com/apple/swift/pull/20305) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-216-user-defined-dynamically-callable-types/14110) +* Status: **Implemented (Swift 5.0)** + +## Introduction + +This proposal is a follow-up to [SE-0195 - Introduce User-defined "Dynamic Member +Lookup" Types](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0195-dynamic-member-lookup.md), +which shipped in Swift 4.2. It introduces a new `@dynamicCallable` attribute, which marks +a type as being "callable" with normal syntax. It is simple syntactic sugar +which allows the user to write: + +```swift +a = someValue(keyword1: 42, "foo", keyword2: 19) +```` + +and have it be rewritten by the compiler as: + +```swift +a = someValue.dynamicallyCall(withKeywordArguments: [ + "keyword1": 42, "": "foo", "keyword2": 19 +]) +``` + +Many other languages have analogous features (e.g. Python "callables", C++ `operator()`, and +[functors in many other languages](https://en.wikipedia.org/wiki/Function_object)), but the +primary motivation of this proposal is to allow elegant and natural interoperation with +dynamic languages in Swift. + +Swift-evolution threads: + - [Pitch: Introduce user-defined dynamically "callable" + types](https://forums.swift.org/t/pitch-introduce-user-defined-dynamically-callable-types/7038). + - [Pitch #2: Introduce user-defined dynamically “callable” + types](https://forums.swift.org/t/pitch-2-introduce-user-defined-dynamically-callable-types/7112). + - Current pitch thread: [Pitch #3: Introduce user-defined dynamically “callable” + types](https://forums.swift.org/t/pitch-3-introduce-user-defined-dynamically-callable-types/12232) + +## Motivation and context + +Swift is exceptional at interworking with existing C and Objective-C APIs and +we would like to extend this interoperability to dynamic languages like Python, +JavaScript, Perl, and Ruby. We explored this overall goal in a long design +process wherein the Swift evolution community evaluated multiple different +implementation approaches. The conclusion was that the best approach was to put +most of the complexity into dynamic language specific bindings written as +pure-Swift libraries, but add small hooks in Swift to allow these bindings to +provide a natural experience to their clients. +[SE-0195](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0195-dynamic-member-lookup.md) +was the first step in this process, which introduced a binding to naturally +express member lookup rules in dynamic languages. + +What does interoperability with Python mean? Let's explain this by looking at +an example. Here's some simple Python code: + +```Python +class Dog: + def __init__(self, name): + self.name = name + self.tricks = [] # creates a new empty list for each `Dog` + + def add_trick(self, trick): + self.tricks.append(trick) +``` + +With the [SE-0195 `@dynamicMemberLookup` feature](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0195-dynamic-member-lookup.md) +introduced in Swift 4.2, it is possible to implement a [Python interoperability +layer](https://github.com/apple/swift/tree/tensorflow/stdlib/public/Python) +written in Swift. It interoperates with the Python runtime, and project all +Python values into a single `PythonObject` type. It allows us to call into the +`Dog` class like this: + +```swift +// import DogModule.Dog as Dog +let Dog = Python.import.call(with: "DogModule.Dog") + +// dog = Dog("Brianna") +let dog = Dog.call(with: "Brianna") + +// dog.add_trick("Roll over") +dog.add_trick.call(with: "Roll over") + +// dog2 = Dog("Kaylee").add_trick("snore") +let dog2 = Dog.call(with: "Kaylee").add_trick.call(with: "snore") +``` + +This also works with arbitrary other APIs as well. Here is an example working +with the Python `pickle` API and the builtin Python function `open`. Note that +we choose to put builtin Python functions like `import` and `open` into a +`Python` namespace to avoid polluting the global namespace, but other designs +are possible: + +```swift +// import pickle +let pickle = Python.import.call(with: "pickle") + +// file = open(filename) +let file = Python.open.call(with: filename) + +// blob = file.read() +let blob = file.read.call() + +// result = pickle.loads(blob) +let result = pickle.loads.call(with: blob) +``` + +This capability works well, but the syntactic burden of having to use +`foo.call(with: bar, baz)` instead of `foo(bar, baz)` is significant. Beyond +the syntactic weight, it directly harms code clarity by making code hard to +read and understand, cutting against a core value of Swift. + +The proposed `@dynamicCallable` attribute directly solves this problem. +With it, these examples become more natural and clear, effectively matching the +original Python code in expressiveness: + +```swift +// import DogModule.Dog as Dog +let Dog = Python.import("DogModule.Dog") + +// dog = Dog("Brianna") +let dog = Dog("Brianna") + +// dog.add_trick("Roll over") +dog.add_trick("Roll over") + +// dog2 = Dog("Kaylee").add_trick("snore") +let dog2 = Dog("Kaylee").add_trick("snore") +``` + +Python builtins: + +```swift +// import pickle +let pickle = Python.import("pickle") + +// file = open(filename) +let file = Python.open(filename) + +// blob = file.read() +let blob = file.read() + +// result = pickle.loads(blob) +let result = pickle.loads(blob) +``` + +This proposal merely introduces a syntactic sugar - it does not add any new +semantic model to Swift. We believe that interoperability with scripting +languages is an important and rising need in the Swift community, particularly +as Swift makes inroads into the server development and machine learning +communities. This feature is also precedented in other languages (e.g. Scala's +[`Dynamic`](https://www.scala-lang.org/api/current/scala/Dynamic.html) trait), and +can be used for other purposes besides language interoperability (e.g. +implementing dynamic proxy objects). + +## Proposed solution + +We propose introducing a new `@dynamicCallable` attribute to the Swift language +which may be applied to structs, classes, enums, and protocols. This follows +the precedent of +[SE-0195](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0195-dynamic-member-lookup.md). + +Before this proposal, values of these types are not valid in a call expression: +the only existing callable values in Swift are those with function types +(functions, methods, closures, etc) and metatypes (which are initializer +expressions like `String(42)`). Thus, it is always an error to "call" an +instance of a nominal type (like a struct, for instance). + +With this proposal, types with the `@dynamicCallable` attribute on their +primary type declaration become "callable". They are required to implement at +least one of the following two methods for handling the call behavior: + +```swift +func dynamicallyCall(withArguments: <#Arguments#>) -> <#R1#> +// `<#Arguments#>` can be any type that conforms to `ExpressibleByArrayLiteral`. +// `<#Arguments#>.ArrayLiteralElement` and the result type `<#R1#>` can be arbitrary. + +func dynamicallyCall(withKeywordArguments: <#KeywordArguments#>) -> <#R2#> +// `<#KeywordArguments#>` can be any type that conforms to `ExpressibleByDictionaryLiteral`. +// `<#KeywordArguments#>.Key` must be a type that conforms to `ExpressibleByStringLiteral`. +// `<#KeywordArguments#>.Value` and the result type `<#R2#>` can be arbitrary. + +// Note: in these type signatures, bracketed types like <#Arguments#> and <#KeywordArguments#> +// are not actual types, but rather any actual type that meets the specified conditions. +``` + +As stated above, `<#Arguments#>` and `<#KeywordArguments#>` can be any types +that conform to the +[`ExpressibleByArrayLiteral`](https://developer.apple.com/documentation/swift/expressiblebyarrayliteral) +and +[`ExpressibleByDictionaryLiteral`](https://developer.apple.com/documentation/swift/expressiblebydictionaryliteral) +protocols, respectively. The latter is inclusive of +[`KeyValuePairs`](https://developer.apple.com/documentation/swift/keyvaluepairs), +which supports duplicate keys, unlike [`Dictionary`](https://developer.apple.com/documentation/swift/dictionary). +Thus, using `KeyValuePairs` is recommended to support duplicate keywords and +positional arguments (because positional arguments are desugared as keyword +arguments with the empty string `""` as the key). + +If a type implements the `withKeywordArguments:` method, it may be dynamically +called with both positional and keyword arguments: positional arguments have +the empty string `""` as the key. If a type only implements the +`withArguments:` method but is called with keyword arguments, a compile-time +error is emitted. + +Since dynamic calls are syntactic sugar for direct calls to `dynamicallyCall` +methods, additional behavior of the `dynamicallyCall` methods is directly +forwarded. For example, if a `dynamicallyCall` method is marked with `throws` +or `@discardableResult`, then the corresponding sugared dynamic call will +forward that behavior. + +### Ambiguity resolution: most specific match + +Since there are two `@dynamicCallable` methods, there may be multiple ways to +handle some dynamic calls. What happens if a type specifies both the +`withArguments:` and `withKeywordArguments:` methods? + +We propose that the type checker resolve this ambiguity towards the tightest +match based on syntactic form of the expression. The exact rules are: + +- If a `@dynamicCallable` type implements the `withArguments:` method and it is + called with no keyword arguments, use the `withArguments:` method. +- In all other cases, attempt to use the `withKeywordArguments:` method. + - This includes the case where a `@dynamicCallable` type implements the + `withKeywordArguments:` method and it is called with at least one keyword + argument. + - This also includes the case where a `@dynamicCallable` type implements only + the `withKeywordArguments:` method (not the `withArguments:` method) and + it is called with no keyword arguments. + - If `@dynamicCallable` type does not implement the `withKeywordArguments:` + method but the call site has keyword arguments, an error is emitted. + +Here are some toy illustrative examples: + +```swift +@dynamicCallable +struct Callable { + func dynamicallyCall(withArguments args: [Int]) -> Int { return args.count } +} +let c1 = Callable() +c1() // desugars to `c1.dynamicallyCall(withArguments: [])` +c1(1, 2) // desugars to `c1.dynamicallyCall(withArguments: [1, 2])` +c1(a: 1, 2) // error: `Callable` does not define the 'withKeywordArguments:' method + +@dynamicCallable +struct KeywordCallable { + func dynamicallyCall(withKeywordArguments args: KeyValuePairs) -> Int { + return args.count + } +} +let c2 = KeywordCallable() +c2() // desugars to `c2.dynamicallyCall(withKeywordArguments: [:])` +c2(1, 2) // desugars to `c2.dynamicallyCall(withKeywordArguments: ["": 1, "": 2])` +c2(a: 1, 2) // desugars to `c2.dynamicallyCall(withKeywordArguments: ["a": 1, "": 2])` + +@dynamicCallable +struct BothCallable { + func dynamicallyCall(withArguments args: [Int]) -> Int { return args.count } + func dynamicallyCall(withKeywordArguments args: KeyValuePairs) -> Int { + return args.count + } +} +let c3 = BothCallable() +c3() // desugars to `c3.dynamicallyCall(withArguments: [])` +c3(1, 2) // desugars to `c3.dynamicallyCall(withArguments: [1, 2])` +c3(a: 1, 2) // desugars to `c3.dynamicallyCall(withKeywordArguments: ["a": 1, "": 2])` +``` + +This ambiguity resolution rule works out naturally given the behavior of the +Swift type checker, because it only resolves call expressions when the type +of the base expression is known. At that point, it knows whether the base is a +function type, metatype, or a valid `@dynamicCallable` type, and it knows the +syntactic form of the call. + +This proposal does not require massive or invasive changes to the constraint +solver. Please look at the implementation for more details. + +## Example usage + +Here, we sketch some example bindings to show how this could be used in +practice. Note that there are lots of design decisions that are orthogonal to +this proposal (e.g. how to handle exceptions) that we aren't going into here. +This is just to show how this feature provides an underlying facility that +language bindings authors can use to achieve their desired result. These +examples also show `@dynamicMemberLookup` to illustrate how they work together, +but elides other implementation details. + +JavaScript supports callable objects but does not have keyword arguments. + +Here is a sample JavaScript binding: + +```swift +@dynamicCallable @dynamicMemberLookup +struct JSValue { + // JavaScript doesn't have keyword arguments. + @discardableResult + func dynamicallyCall(withArguments: [JSValue]) -> JSValue { ... } + + // This is a `@dynamicMemberLookup` requirement. + subscript(dynamicMember member: JSValue) -> JSValue {...} + + // ... other stuff ... +} +``` + +On the other hand, a common JavaScript pattern is to take a dictionary of +values as a stand-in for argument labels (called like +`example({first: 1, second: 2, third: 3})` in JavaScript). A JavaScript bridge +in Swift could choose to implement keyword argument support to allow this to be +called as `example(first: 1, second: 2, third: 3)` from Swift code (kudos to +Ben Rimmington for [this +observation](https://forums.swift.org/t/pitch-3-introduce-user-defined-dynamically-callable-types/12232/45)). + +Python does support keyword arguments. While a Python binding could implement +only the `withKeywordArguments:` method, it is be better to implement both the +non-keyword and keyword forms to make the non-keyword case slightly more +efficient (avoid allocating temporary storage) and to make direct calls with +positional arguments nicer (`x.dynamicallyCall(withArguments: 1, 2)` instead of +`x.dynamicallyCall(withKeywordArguments: ["": 1, "": 2])`). + +Here is a sample Python binding: + +```swift +@dynamicCallable @dynamicMemberLookup +struct PythonObject { + // Python supports arbitrary mixes of keyword arguments and non-keyword + // arguments. + @discardableResult + func dynamicallyCall( + withKeywordArguments: KeyValuePairs + ) -> PythonObject { ... } + + // An implementation of a Python binding could choose to implement this + // method as well, avoiding allocation of a temporary array. + @discardableResult + func dynamicallyCall(withArguments: [PythonObject]) -> PythonObject { ... } + + // This is a `@dynamicMemberLookup` requirement. + subscript(dynamicMember member: String) -> PythonObject {...} + + // ... other stuff ... +} +``` + +## Limitations + +Following the precedent of SE-0195, this attribute must be placed on the +primary definition of a type, not on an extension. + +This proposal does not introduce the ability to provide dynamically callable +`static`/`class` members. We don't believe this is important given the goal of +supporting dynamic languages like Python, but it could be explored if a use +case is discovered in the future. Such future work should keep in mind that +call syntax on metatypes is already meaningful, and that ambiguity would have +to be resolved somehow (e.g. through the most specific rule). + +This proposal supports direct calls of values and methods, but subsets out +support for currying methods in Smalltalk family languages. This is just an +implementation limitation given the current state of currying in the Swift +compiler. Support can be added in the future if there is a specific need. + +## Source compatibility + +This is a strictly additive proposal with no source breaking changes. + +## Effect on ABI stability + +This is a strictly additive proposal with no ABI breaking changes. + +## Effect on API resilience + +This has no impact on API resilience which is not already captured by other +language features. + +## Future directions + +### Dynamic member calling (for Smalltalk family languages) + +In addition to supporting languages like Python and JavaScript, we would also +like to grow to support Smalltalk derived languages like Ruby and Squeak. These +languages resolve *method* calls using both the base name as well as the +keyword arguments at the same time. For example, consider this Ruby code: + +```Ruby +time = Time.zone.parse(user_time) +``` + +The `Time.zone` reference is a member lookup, but `zone.parse(user_time)` is a +method call, and needs to be handled differently than a lookup of `zone.parse` +followed by a direct function call. + +This can be handled by adding a new `@dynamicMemberCallable` attribute, which +acts similarly to `@dynamicCallable` but enables dynamic member calls (instead +of dynamic calls of `self`). + +`@dynamicMemberCallable` would have the following requirements: + +```swift +func dynamicallyCallMethod(named: S1, withArguments: [T5]) -> T6 +func dynamicallyCallMethod(named: S2, withKeywordArguments: [S3 : T7]) -> T8 +``` + +Here is a sample Ruby binding: + +```swift +@dynamicMemberCallable @dynamicMemberLookup +struct RubyObject { + @discardableResult + func dynamicallyCallMethod( + named: String, withKeywordArguments: KeyValuePairs + ) -> RubyObject { ... } + + // This is a `@dynamicMemberLookup` requirement. + subscript(dynamicMember member: String) -> RubyObject {...} + + // ... other stuff ... +} +``` + +### General callable behavior + +This proposal is mainly directed at dynamic language interoperability. For this +use case, it makes sense for the `dynamicallyCall` method to take a +variable-sized list of arguments where each argument has the same type. +However, it may be useful to support general callable behavior (akin to +`operator()` in C++) where the desugared "callable" method can have a fixed +number of arguments and arguments of different types. + +For example, consider something like: + +```swift +struct BinaryFunction { + func call(_ argument1: T1, _ argument1: T2) -> U { ... } +} +``` + +It is not unreasonable to look ahead to a day where sugaring such things is +supported, particularly when/if Swift gets [variadic +generics](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#variadic-generics). +This could allow typesafe n-ary smart function pointer types. + +We feel that the approach outlined in this proposal supports this direction. +When/if a motivating use case for general callable behavior comes up, we can +simply add a new form to represent it and enhance the type checker to prefer +that during ambiguity resolution. If this is a likely direction, then it may be +better to name the attribute `@callable` instead of `@dynamicCallable` in +anticipation of that future growth. + +We believe that general callable behavior and `@dynamicCallable` are orthogonal +features and should be evaluated separately. + +## Alternatives considered + +Many alternatives were considered and discussed. Most of them are captured in +the ["Alternatives Considered" section of +SE-0195](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0195-dynamic-member-lookup.md#alternatives-considered). + +Here are a few points raised in the discussion: + +- It was suggested that we use subscripts to represent the call + implementations instead of a function call, aligning with + `@dynamicMemberLookup`. We think that functions are a better fit here: the + reason `@dynamicMemberLookup` uses subscripts is to allow the members to be + l-values, but call results are not l-values. + +- It was requested that we design and implement the 'static callable' version + of this proposal in conjunction with the dynamic version proposed here. In + the author's opinion, it is important to consider static callable support as + a likely future direction to make sure that the two features sit well next + to each other and have a consistent design (something we believe this + proposal has done) but it doesn't make sense to join the two proposals. So + far, there have been no strong motivating use case presented for the static + callable version, and Swift lacks certain generics features (e.g. variadics) + that would be necessary to make static callables general. We feel that static + callable should stand alone on its own merits. diff --git a/proposals/0217-bangbang.md b/proposals/0217-bangbang.md new file mode 100644 index 0000000000..b689e549ee --- /dev/null +++ b/proposals/0217-bangbang.md @@ -0,0 +1,643 @@ +# Introducing the `!!` "Unwrap or Die" operator to the Swift Standard Library + +* Proposal: [SE-0217](0217-bangbang.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Dave DeLong](https://github.com/davedelong), [Paul Cantrell](https://github.com/pcantrell), [Erica Sadun](https://github.com/erica), and several other folk +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Rejected** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0217-the-unwrap-or-die-operator/14107/222) + +## Introduction + +This proposal introduces an annotating forced-unwrapping operator to the Swift standard library. It augments the `?`, `??`, and `!` family, adding `!!`. This "unwrap or die" operator provides code-sourced rationales for failed unwraps, supporting self-documentation and safer development. The `!!` operator is commonly implemented in the wider Swift Community and should be considered for official adoption. + +The new operator benefits both experienced and new Swift users. It takes this form: + +```let value = wrappedValue !! <# "Explanation why lhs cannot be nil." #>``` + +It provides a sugared and succinct equivalent of the following `guard`, unwrapping its left hand value and branching to a fatal error on nil values: + +``` +guard let value = wrappedValue else { + fatalError(<# "Explanation why lhs cannot be nil." #>) +} +``` + +This approach replaces commented versions that do not emit explanations when an unwrap fails: + +``` +let value = wrappedValue! // Explanation why lhs cannot be nil +``` + +Adopting this operator: + +* Encourages a thoughtful approach to unwrapping, +* Promotes reasons for trapping during unwraps, which are visible at run time, and +* Provides a succinct and easily-taught form for new Swift learners. + + +*This proposal was first discussed on the Swift Evolution list in the +[\[Pitch\] Introducing the "Unwrap or Die" operator to the standard library](https://forums.swift.org/t/pitch-introducing-the-unwrap-or-die-operator-to-the-standard-library/6207) thread. It has been further discussed in the Swift Forums on the [Resolved: Insert "!" is a bad fixit"](https://forums.swift.org/t/resolved-insert-is-a-bad-fixit/10764) thread.* + +## Motivation + +"Unwrap or Die" has been widely adopted in the Swift community. This approach provides a best practices approach that establishes informative run-time diagnostics with compiler-checked rationales (the rhs is mandatory). Relying solely on comments only conveys in the source code itself the reason why the developer thinks the value cannot be nil. Requiring a string on the rhs of `!!` provides useful information all the way from source to the console should the underlying guarantee fail at runtime. This produces "better" failures with explicit explanations. If you’re going to write a comment, why not make that comment useful for debugging at the same time? + +### The Naive User / Fixit Problem + +Swift's [`Insert "!"` fixit](https://i.imgur.com/I05TkbJ.jpg) is a dangerous enticement to new Swift language users. Force-unwrapping should be used sparingly and thoughtfully. Beginners have a strong tendency to throw code at the compiler until it runs. Consider the following line of code and its fixit. Mashing the "Fix" button inserts an unwrap after the `url` instance. + +![](https://i.imgur.com/I05TkbJ.jpg) + +```swift +let resourceData = try String(contentsOf: url, encoding: .utf8) + +// Error: Value of optional type 'URL?' not unwrapped; did you mean to use '!' or '?'? +// Fix: Insert '!' +``` + +Experienced developers easily distinguish whether an optional value reflects an overlooked error condition, in which case they rearchitect to introduce a better pattern, or if the value in question is guaranteed to never contain nil and can safely support an unwrap. + +Inexperienced developers, unless they’re moving from a language with similar constructs, usually will not make such distinctions. They’re focused on getting past a compilation barrier, without realizing the hazard of nil values at the point of use. + +Quincey Morris writes with respect to the `url` example: + +> The problem with the fixit (for inexperienced Swift programmers) is that this is almost certainly not the point at which things went wrong. Almost always (for the inexperienced) the problem is that url is optional by accident, and _the correct fix is to add a ! to the RHS of the assignment from which url’s type was inferred — or, at least, to handle the optional there)_. + +The "fixit" solution adds an unwrap at the point of use: + +```swift +let resourceData = try String(contentsOf: url!, encoding: .utf8) +``` + +A better solution tests and unwraps _at the point of declaration_ using `if-let` or `guard-let`: + +```swift +let destination = "http://swift.org" +guard let url = URL(string: destination) else { + fatalError("Invalid URL string: \(destination)") +} + +let destination = "☹️" +guard let url = URL(string: destination) else { + fatalError("Invalid URL string: \(destination)") +} +``` + +Swift's "Add !" is a syntactic fix, not a semantic one. Swift cannot holistically evaluate user code intent. It cannot recommend or fixit a `guard let` or `if let` alternative at a disjoint location that leads to the point of error. + +### Moral Hazards + +Fixits should not be used as bandaids to make things compile. Swift's "Add !" fixit is a modest courtesy for experienced users and a moral hazard for the inexperienced. Stack Overflow and the developer forums are littered with questions regarding ["unexpectedly found nil" errors at runtime](https://duckduckgo.com/?q=unexpectedly+found+nil&bext=msl&atb=v99-7_g&ia=qa): + +> If you go through the Swift tab on Stack Overflow and take a drink every time you see a blatantly inappropriate use of ! in a questioner’s code due to them just mashing the fix-it, you will soon be dead of alcohol poisoning. +> +> -- _Charles Srstka_ + +Introducing `!!` allows Swift to offer a better, more educational fixit that guides new users to better solutions. If the expression cannot guarantee that the lhs is non-nil, coders are better served using `if let`/`guard let` constructions. Replacing the `!` fixit with `!! <# "Explanation why the left hand value cannot be nil." #>` offers a user-supporting hint that counters "Value of optional type not unwrapped" confusion. + +### Runtime Diagnostics + +Forced unwraps are an essential part of the Swift programming language. In many cases, a `!` forced unwrap is the correct way to unwrap an optional value. This proposal does not challenge this. The new operator promotes safety, maintainability, and readability as an alternative, not evolutionary, use. `!!`'s documentary "guided landing" explains why unwrapping is guaranteed to be non-nil. + +It takes its cue from existing Swift constructs (like `precondition` and `assert`) that incorporate meaningful output strings when the app traps: + +```swift +assert(!array.isEmpty, "Array guaranteed to be non-empty because...") +let lastItem = array.last! +``` + +Guard statements, assertions, preconditions, and fatal errors allow developers to backtrack and correct assumptions that underlie their design. When an application traps from annotated assertions and preconditions, debug console output explains *why* straight away. You don't have to hunt down the code, read the line that failed, then establish the context around the line to understand the reason. Embedded rationales are even more important when you didn’t write this code yourself. + +Incorporating messages explains the use of forced unwraps in code. This provides better self documentation of facts that are known to be true. These messages support better code reading, maintenance, and future modifications. + +When an optional can a priori be *guaranteed to be non-nil*, using guards, assertions, preconditions, etc, are relatively heavy-handed approaches to document these established truths. When you already know that an array is not-empty, you should be able to specify this in a single line using a simple operator: + +```swift +// Existing force-unwrap +let lastItem = array.last! // Array guaranteed to be non-empty because... + +// Proposed unwrap operator with fallback explanation +let lastItem = array.last !! "Array guaranteed to be non-empty because..." +``` + +Consider the following scenario of a custom view controller subclass that only accepts children of a certain kind. It is known a priori that the cast will succeed: + +```swift +let existing = childViewControllers as? Array + !! "TableViewController must only have TableRowViewControllers as children" +``` + +This pattern extends to any type-erased or superclass where a cast will always be valid. + +## Examples of Real-World Use + +Here are a variety of examples that demonstrate the `!!` operator in real-world use: + +```swift +// In a right-click gesture recognizer action handler +let event = NSApp.currentEvent + !! "Trying to get current event for right click, but there's no event" + +// In a custom view controller subclass that only +// accepts children of a certain kind: +let existing = childViewControllers as? Array + !! "TableViewController must only have TableRowViewControllers as children" + +// Providing a value based on an initializer that returns an optional: +lazy var sectionURL: URL = { + return URL(string: "myapp://section/\(identifier)") + !! "can't create URL for section \(identifier)" +}() + +// Retrieving an image from an embedded framework: +private static let addImage: NSImage = { + let bundle = Bundle(for: FlagViewController.self) + let image = bundle.image(forResource: "add") !! "Missing 'add' image" + image.isTemplate = true + return image +}() + +// Asserting consistency of an internal model: +let flag = command.flag(with: flagID) !! "Unable to retrieve non-custom flag for id \(flagID.string)" + +// drawRect: +override draw(_ rect: CGRect) { + let context = UIGraphicsGetCurrentContext() !! "`drawRect` context guarantee was breeched" +} +``` + +The `!!` operator generally falls in two use scenarios: + +1. **Asserting System Framework Correctness**: The `NSApp.currentEvent` property returns an `Optional` as there’s not always a current event going on. It is always safe to assert an actual event in the action handler of a right-click gesture recognizer. If this ever fails, `!!` provides an immediately and clear description of where the system framework has not worked according to expectations. + +2. **Asserting Application Logic Correctness**: The `!!` operator ensures that outlets are properly hooked up and that the internal data model is in a consistent state. The related error messages explicitly mention specific outlet and data details. + +These areas identify when resources haven't been added to the right target, when a URL has been mis-entered, or when a model update has not propagated completely to its supporting use. Incorporating a diagnostic message, provides immediate feedback as to why the code is failing and where. + +### The Black Swan Deployment + +In one [real-world case](http://ericasadun.com/2017/01/23/safe-programming-optionals-and-hackintoshes/), a developer's deployed code crashed when querying Apple's [smart battery interface](https://developer.apple.com/library/content/documentation/DeviceDrivers/Conceptual/IOKitFundamentals/PowerMgmt/PowerMgmt.html) on a Hackintosh. Since the laptop in question wasn’t an actual Apple platform, it used a simulated AppleSmartBatteryManager interface. In this case, the simulated manager didn’t publish the full suite of values normally guaranteed by the manager’s API. The developer’s API-driven contract assumptions meant that forced unwraps broke his app: + +> Since IOKit just gives you back dictionaries, a missing key, is well… not there, and nil. you know how well Swift likes nils… + +Applications normally can’t plan for, anticipate, or provide workarounds for code running on unofficial platforms. There are too many unforeseen factors that cannot be incorporated into realistic code that ships. Adopting a universal "unwrap or die" style with explanations enables you to "guide the landing" on these unforeseen ["Black Swan"](https://en.wikipedia.org/wiki/Black_swan_theory) failures: + +``` +guard let value = dict[guaranteedKey] + else { + fatalError("Functionality compromised when unwrapping " + + "Apple Smart Battery Dictionary value.") + return +} + +// or more succinctly + +let value = dict[guaranteedKey] !! "Functionality compromised when unwrapping Apple Smart Battery Dictionary value." +``` + +The `!!` operator reduces the overhead involved in debugging unexpected Black Swan deployments. This practice adds robustness and assumes that in reality bad execution can happen for the oddest of reasons. Providing diagnostic information even when your assumptions are "guaranteed" to be correct is a always positive coding style. + +### Real World Deployment + +The `!!` operator is in reasonably wide use in the Swift community, including in the personal toolboxes of this proposal's authors. For example, Constantino Tsarouhas explains how `!!` has affected his development, enhancing code reading and introducing mindful unwrapping practices: + +> I’ve been happily using !! in my codebases for over a year. It significantly improved understanding what my code does, even after several months of not reading it. I almost never use ! now. The extra keystrokes also act as a deterrent to use this escape hatch too often. + +## On Forced Unwraps + +This proposal _does not_ eliminate or prejudge the `!`operator. Using `!!` should be a positive house standards choice, especially when the use of explanatory text becomes cumbersome. The following example using an unwrapping operator: + +```swift +// Constructed date values will always produce valid results +return Date.sharedCalendar.date(byAdding: rhs, to: lhs)! +``` + +is far simpler than an equivalent version using `guard`: + +```swift +guard let date = Date.sharedCalendar.date(byAdding: lhs, to: rhs) else { + // This should never happen + fatalError("Constructed date values will always produce valid results") +} +return date +``` + +and slightly more succinct than: + +```swift +return Date.sharedCalendar.date(byAdding: lhs, to: rhs) + !! "Constructed date values will always produce valid results" +``` + +The major difference is that the first example lacks the runtime diagnostic output of the latter. This last example accrues all the benefits of `!!`. Those benefits are ultimately the choice of the adopter. + +An often-touted misconception exists that force unwraps are, in and of themselves, *bad*: that they were only created to accommodate legacy apps, and that you should never use force-unwrapping in your code. This isn’t true. + +There are many good reasons to use force unwraps, though if you're often reaching for it, it's a bad sign. Force-unwrapping can be a better choice than throwing in meaningless default values with nil-coalescing or applying optional chaining when the presence of nil would indicate a serious failure. + +Introducing the `!!` operator endorses and encourages the use of "mindful force-unwrapping". It incorporates the reason *why* the forced unwrap should be safe (for example, *why* the array can’t be empty at this point, not just that it is unexpectedly empty). If you’re already going to write a comment, why not make that comment useful for debugging at the same time? + +Using `!!` provides syntactic sugar for the following common unwrap pattern: + +```swift +guard let y = x + else { fatalError("reason") } + +// becomes + +let y = x !! "reason" + +// and avoids + +let y = x! // reason +``` + +Although comments document in-code reasoning, these explanations are not emitted when the application traps on the forced unwrap: + +> As the screener of a non-zero number of radars resulting from unwrapped nils, I would certainly appreciate more use of `guard let x = x else { fatalError("explanation") }` and hope that `!!` would encourage it. +> -- Ben Cohen + +Sometimes it’s not necessary to explain your use of a forced unwrap. In those cases the normal `!` operator will remain, even after the introduction of `!!`. You can continue using `!`, as before, just as you can leave off the string from a precondition. + +Similarly, updating Swift's recommended fixit from `Insert !` to `Insert <# "Explanation why lhs cannot be nil." #>` does not prevent the use of `!` for the experienced user and requires no more keystrokes or actions on the part of the experienced user. + +## On Adding a New Operator to Swift + +Although burning a new operator is a serious choice, `!!` is a good candidate for adoption: + +* It matches and parallels the existing `??` operator. +* It fosters better understanding of optionals and the legitimate use of force-unwrapping in a way that encourages safe coding and good documentation, both in source and at run-time. +* `!!` sends the right semantic message. It communicates that "unwrap or die" is an unsafe operation and that failures should be both extraordinary and explained. + +The new operator is consciously based on `!`, the *unsafe* forced unwrap operator, and not on `??`, the *safe* fallback nil-coalescing operator. Its symbology therefore follows `!` and not `?`. + +## Detailed Design + +```swift +infix operator !!: NilCoalescingPrecedence + +extension Optional { + /// Performs an unsafe forced-unwrap operation, returning + /// the wrapped value of an `Optional` instance or + /// executing `fatalError` using the message on the rhs + /// of the operator. + /// + /// The result of a successful operation will be the same type + /// as the wrapped value of the left-hand side argument. + /// + /// The `optional` lhs is checked first, and a `fatalError` + /// is called only if the left hand side is nil. + /// + /// Use this operator when the lhs is guaranteed to be non-nil, + /// for example: + /// + /// ``` + /// // This URL is well formed + /// let normalURL = URL(string: "http://swift.org") + /// !! "URL is not well formed" + /// + /// // An emoji character cannot be used to construct a URL + /// let badURL = URL(string: "😱") + /// !! "URL is not well formed" + /// ``` + /// + /// - Parameters: + /// - optional: An optional value. + /// - message: A message to emit via `fatalError` upon + /// failing to unwrap the optional. + public static func !!( + optional: Optional, + errorMessage: @autoclosure () -> String + ) -> Wrapped { + guard let wrapped = optional else { fatalError(errorMessage()) } + return wrapped + } +} +``` + +### Updating Fixits + +A well-designed error message explains why an error occurred and offer user-supporting directions for remedying the situation. The current error/fixit works like this: + +```C++ +ERROR(missing_unwrap_optional,none, + "value of optional type %0 not unwrapped; did you mean to use '!' " + "or '?'?", + (Type)) + +diag.fixItInsertAfter(affected->getEndLoc(), "!"); +``` + +Adopting `!!` allows Swift to refresh the error message in question and provide multiple fixits: + +> Found unexpected value of optional type %0; did you mean to unwrap this +> optional or provide a default fallback value? Use `if-let` and +> `if-guard` statements to unwrap optionals before use in statements +> and expressions. +> +> * Insert '!!' to unwrap values that cannot be nil: `!! <# "statement why lhs cannot be nil." #>` +> * Insert '??' to provide a default fallback value: `?? <# default value #>` + +### Optimized Builds and Runtime Errors + +With one notable exception, the `!!` operator should follow the same semantics as `Optional.unsafelyUnwrapped`, which establishes a precedent for this approach: + +> "The unsafelyUnwrapped property provides the same value as the forced unwrap operator (postfix !). However, in optimized builds (-O), no check is performed to ensure that the current instance actually has a value. Accessing this property in the case of a nil value is a serious programming error and could lead to undefined behavior or a runtime error." + +By following `Optional.unsafelyUnwrapped`, this approach is consistent with Swift's [error handling system](https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst#logic-failures): + +> "Logic failures are intended to be handled by fixing the code. It means checks of logic failures can be removed if the code is tested enough. Actually checks of logic failures for various operations, `!`, `array[i]`, `&+` and so on, are designed and implemented to be removed when we use `-Ounchecked`. It is useful for heavy computation like image processing and machine learning in which overhead of those checks is not permissible." + +Like `assert`, `unsafelyUnwrapped` does not perform a check in optimized builds. The forced unwrap `!` operator does as does `precondition`. The new "unwrap or die" operator, `!!`, should behave like `precondition` and not `assert` to preserve trapping information in optimized builds. + +Unfortunately, there is no direct way at this time to emit the `#file` name and `#line` number with the above code. We hope the dev team can somehow work around this limitation to produce that information at the `!!` site. The `!!`-alternative design that uses a `() -> Never` closure in the "Alternative Designs" section provides that information today. + +## Future Directions + +#### Calling Context + +Right now, `fatalError` reports the line and file of the `!!` operator implementation rather than the code where the operator is used. At some point Swift may allow operator implementations with more than two parameters. At such time, the `!!` operator should incorporate the source line and file of the forced unwrap call: + +```swift +public static func !!(optional: Optional, errorMessage: @autoclosure () -> String, file: StaticString = #file, line: UInt = #line) -> Wrapped +``` + +This could be a minimal modification to lib/AST/Decl.cpp:4919, to the `FuncDecl::isBinaryOperator()` implementation, enabling `size() > 2` if `get(2+)->isDefaultArgument()`: + +```c++ + bool FuncDecl::isBinaryOperator() const { + if (!isOperator()) + return false; + + auto *params = getParameterList(getDeclContext()->isTypeContext()); + return params->size() == 2 && + !params->get(0)->isVariadic() && + !params->get(1)->isVariadic(); +} +``` + +Having a line and file reference for the associated failure point would be a major advantage in adopting this proposal, even if some under-the-covers "Swift Magic™" must be applied to the `!!` implementation. + +#### `Never` as a Bottom Type + +If `Never` ever becomes a true bottom type as in [SE-0102](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0102-noreturn-bottom-type.md), Swift will be able to use `fatalError()` on the right hand side of nil-coalescing. + +> If [`Never` as a bottom type] were supported by the compiler, it would enable some potentially useful things, for instance using a nonreturning function directly as a parameter to a higher-order function that expects a result...or allowing a subclass to override a method and covariantly return `Never`. +> +> -- [SE-0102](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0102-noreturn-bottom-type.md#never-as-a-universal-bottom-subtype) + +```swift +// Legal if a (desirable) `Never` bottom type is adopted +let x = y ?? fatalError("reason") +``` + +This proposal supports using a `String` (or more properly a string autoclosure) on the rhs of a `!!` operator in preference to a `Never` bottom type or a `() -> Never` closure with `??` for the reasons that are enumerated here: + +- A string provides the cleanest user experience, and allows the greatest degree of in-place self-documentation. + +- A string respects DRY, and avoids using *both* the operator and the call to `fatalError` or `preconditionFailure` to signal an unsafe condition: +`let last = array.last !! "Array guaranteed non-empty because..." // readable` +versus: +`let last = array.last !! fatalError("Array guaranteed non-empty because...") // redundant` + +- A string allows the operator *itself* to unsafely fail, just as the unary version of `!` does now. It does this with additional feedback to the developer during testing, code reading, and code maintenance. The string provides a self-auditing in-line annotation of the reason why the forced unwrap has been well considered, using a language construct to support this. + +- A string disallows a potentially unsafe `Never` call that does not reflect a serious programming error, for example: +`let last = array.last !! f() + // where func f() -> Never { while true {} }` + +- Using `foo ?? Never` requires a significant foundational understanding of the language, which includes a lot of heavy lifting to understand how and why it works. Using `!!` is a simpler approach that is more easily taught to new developers: "This variation of forced unwrap emits this message if the lhs is nil." Although a `Never` closure solution can be cobbled together in today's Swift, `!!` operator solution can be as well. Neither one requires a fundamental change to the language. + +Pushing forward on this proposal does not in any way reflect on adopting the still-desirable `Never` bottom type. + +## Alternatives Considered + +We present two alternative designs that cover a similar space and an extension of `!!` that adds throwing and `Never` closures. Error-throwing operaters was previously discussed in an early proposal by Pyry Jahkola and Erica Sadun: [Introducing an error-throwing nil-coalescing operator](https://gist.github.com/erica/5a26d523f3d6ffb74e34d179740596f7). + +Reasons why these alternatives may not be ideal are covered in the future directions section, such as why extending the `??` operator to fail or throw creates a foundational re-architecting of its semantics. + +### Introducing `fail(_:)` + +Fail allows you to supply a string or error on the right hand side of the coalescing operator. It infers the `Wrapped` type from context: + +``` +// This URL is well formed +let normalURL = URL(string: "http://swift.org") ?? fail("URL is not well formed") + +// This URL is not well formed and should raise a fatal error +let emojiURL = URL(string: "😱") ?? fail("URL is not well formed") + + // A custom error +let illformedURL = ErrorMessage("URL is not well formed") + +do { + This URL is fine + let normalURL = try URL(string: "http:swift.org") ?? fail(illformedURL) + + This URL is not well formed and will throw + let emojiURL = try URL(string: "😱") ?? fail(illformedURL) +} catch { + print(error) +} +``` + +Here is the basic implementation. It avoids using `Wrapped` as its generic variable as `fail` can be used to replace any contextual type. + +``` +// Inspired by Maz Jaleel, https://github.com/Mazyod + +/// Unconditionally prints a given message and stops execution, +/// allowing placement at any point where a contextual type +/// is required. +/// +/// - Parameters: +/// - message: The string to print. The default is an empty string. +/// - file: The file name to print with `message`. The default is the file +/// where `fail(_:file:line:)` is called. +/// - line: The line number to print along with `message`. The default is the +/// line number where `fail(_:file:line:)` is called. +public func fail( + _ message: String = "", + file: StaticString = #file, + line: UInt = #line +) -> T { + fatalError(message, file: file, line: line) +} + +/// Unconditionally throw an error, allowing placement at +/// any point where a contextual type is required. +/// +/// - Parameters: +/// - error: The error to throw. +public func fail(_ error: E) throws -> T { + throw error +} +``` + +### Extending `??` for throwing and `Never` + +A second alternate design overloads `??`, as was discussed several years ago in [this preliminary proposal](https://gist.github.com/erica/5a26d523f3d6ffb74e34d179740596f7), adopting Dave DeLong's suggestion of overriding `??`: + +``` +// where illFormedURL is an error +let normalURL = try URL(string: "http://swift.org") ?? illFormedURL +let badURL = try URL(string: "😱") ?? illFormedURL + +// Using a `Never` call +let normalURL = URL(string: "http://swift.org") ?? fatalError("URL is not well formed") +let badURL = URL(string: "😱") ?? fatalError("URL is not well formed") +``` + +Here is the design: + +``` +extension Optional { + /// Performs an unwrapping operation, returning the wrapped value of an + /// `Optional` instance or throwing an error corresponding to + /// the rhs of the operation. For example: + /// + /// ``` + /// let normalURL = try URL(string: "http://swift.org") ?? illFormedURL + /// let badURL = try URL(string: "😱") ?? illFormedURL + /// ``` + /// - Parameters: + /// - optionalValue: An optional value. + /// - error: an error to throw should the optional be nil. + public static func ?? ( + optionalValue: Wrapped?, + error: E + ) throws -> Wrapped { + guard let value = optionalValue else { throw error } + return value + } + + /// Performs an unwrapping operation, returning the wrapped value of an + /// `Optional` instance or executing a closure that is guaranteed + /// to never return, such as fatalError(). Use this operator + /// when the rhs should never execute, for example: + /// + /// ``` + /// let normalURL = URL(string: "http://swift.org") + /// ?? fatalError("URL is not well formed") + /// let badURL = URL(string: "😱") ?? fatalError("URL is not well formed") + /// ``` + /// + /// - Parameters: + /// - optionalValue: An optional value. + /// - never: a `Never` closure to execute should the optional be nil. + public static func ?? ( + optionalValue: Wrapped?, + never: @autoclosure () -> Never + ) -> Wrapped { + guard let value = optionalValue else { never() } + return value + } +} +``` + +## Extending `!!` beyond strings + +Both throwing and `Never` behaviors can be added to `!!`. These offer the same community sourced features used in the nil-coalescing design. The `Never` variation provides line and file information at the point of use. + +``` +extension Optional { + /// Performs an unsafe forced-unwrap operation, returning + /// the wrapped value of an `Optional` instance or + /// throwing an error corresponding to the rhs of the operator. + /// + /// The result of a successful operation will be the same type + /// as the wrapped value of the left-hand side argument. + /// + /// The `optional` lhs is checked first, and an error is thrown + /// only if the left hand side is nil. + /// + /// Use this operator when the lhs is guaranteed to be non-nil, + /// for example: + /// + /// ``` + /// // `illFormedURL` is an `Error`-conforming instance. + /// + /// // This URL is well formed + /// let normalURL = try URL(string: "http://swift.org") !! illFormedURL + /// + /// // An emoji character cannot be used to construct a URL + /// let badURL = try URL(string: "😱") !! illFormedURL + /// ``` + /// + /// - Parameters: + /// - optional: An optional value. + /// - error: an error to throw should the optional be nil. + public static func !!( + optional: Optional, + error: E + ) throws -> Wrapped { + guard let wrapped = optional else { throw error } + return value + } + + /// Performs an unsafe forced-unwrap operation, returning + /// the wrapped value of an `Optional` instance or + /// executing a closure that is guaranteed to never return, + /// such as fatalError(). + /// + /// The result of a successful operation will be the same type + /// as the wrapped value of the left-hand side argument. + /// + /// The `optional` lhs is checked first, and an error is thrown + /// only if the left hand side is nil. + /// + /// Use this operator when the rhs should never execute, + /// for example: + /// + /// ``` + /// // This URL is well formed + /// let normalURL = URL(string: "http://swift.org") + /// !! fatalError("URL is not well formed") + /// + /// // An emoji character cannot be used to construct a URL + /// let badURL = URL(string: "😱") + /// !! fatalError("URL is not well formed") + /// ``` + /// + /// - Parameters: + /// - optional: An optional value. + /// - error: an error to throw should the optional be nil. + public static func !!( + optional: Optional, + never: @autoclosure () -> Never + ) throws -> Wrapped { + guard let wrapped = optionalValue else { never() } + return wrapped +} +``` + +## Additional Operators Considered + +The throwing variation of `!!` could be named `?!`, taking an `Error` on the rhs. This extension introduces a non-fatal operator that creates variation of `try` specifically for optionals. + +Adding `?!` produces the following operator family: + +| Operator | Use | +| :------- | :----------------- | +| `!` | Force unwrap | +| `?` | Optional chaining | +| `!!` | Unwrap or die | +| `??` | Nil coalescing | +| `?!` | Unwrap or throw | +| `try` | Throw on error | +| `try!` | Die on error | +| `try?` | Nil on error | +| `as!` | Die on failed cast | +| `as?` | Nil on failed cast | + +## Source compatibility + +This proposal is strictly additive. + +## Effect on ABI stability + +This proposal does not affect ABI stability. + +## Effect on API resilience + +This proposal does not affect ABI resilience. diff --git a/proposals/0218-introduce-compact-map-values.md b/proposals/0218-introduce-compact-map-values.md new file mode 100644 index 0000000000..66c3e2e8ce --- /dev/null +++ b/proposals/0218-introduce-compact-map-values.md @@ -0,0 +1,96 @@ +# Introduce `compactMapValues` to Dictionary + +* Proposal: [SE-0218](0218-introduce-compact-map-values.md) +* Author: [Daiki Matsudate](https://github.com/d-date) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#15017](https://github.com/apple/swift/pull/15017) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0218-introduce-compactmapvalues-to-dictionary/14448) + +## Introduction + +This proposal adds a combined filter/map operation to `Dictionary`, as a companion to the `mapValues` and filter methods introduced by [SE-0165](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0165-dict.md). The new compactMapValues operation corresponds to compactMap on Sequence. + +- Swift forums pitch: [Add compactMapValues to Dictionary](https://forums.swift.org/t/add-compactmapvalues-to-dictionary/8741) + +## Motivation + +Swift 4 introduced two new `Dictionary` operations: the new method `mapValues` and a new version of `filter`. They correspond to the `Sequence` methods `map` and `filter`, respectively, but they operate on `Dictionary` values and return dictionaries rather than arrays. + +However, [SE-0165](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0165-dict.md) left a gap in the API: it did not introduce a `Dictionary`-specific version of `compactMap`. We sometimes need to transform and filter values of a `Dictionary` at the same time, and `Dictionary` does not currently provide an operation that directly supports this. + +For example, consider the task of filtering out `nil` values from a `Dictionary` of optionals: + +```swift +let d: [String: String?] = ["a": "1", "b": nil, "c": "3"] +let r1 = d.filter { $0.value != nil }.mapValues { $0! } +let r2 = d.reduce(into: [String: String]()) { (result, item) in result[item.key] = item.value } +// r1 == r2 == ["a": "1", "c": "3"] +``` + +Or try running a failable conversion on dictionary values: + +```swift +let d: [String: String] = ["a": "1", "b": "2", "c": "three"] +let r1 = d.mapValues(Int.init).filter { $0.value != nil }.mapValues { $0! } +let r2 = d.reduce(into: [String: Int]()) { (result, item) in result[item.key] = Int(item.value) } +// r == ["a": 1, "b": 2] +``` + +While `mapValues` and `filter` can be combined to solve this tasks, the solution needs multiple passes on the input dictionary, which is not particularly efficient. `reduce(into:)` provides a more efficient solution, but it is rather tricky to get right, and it obscures the intended meaning of the code with implementation details. + +It seems worth adding an extra extension method to `Dictionary` for this operation; its obvious name is `compactMapValues(_:)`, combining the precedents set by `compactMap` and `mapValues`. + +```swift +let r3 = d.compactMapValues(Int.init) +``` + +## Proposed solution + +Add the following to `Dictionary`: + +```swift +let d: [String: String?] = ["a": "1", "b": nil, "c": "3"] +let r4 = d.compactMapValues({$0}) +// r4 == ["a": "1", "c": "3"] +``` + +Or, + +```swift +let d: [String: String] = ["a": "1", "b": "2", "c": "three"] +let r5 = d.compactMapValues(Int.init) +// r5 == ["a": 1, "b": 2] +``` + +## Detailed design + +Add the following to `Dictionary`: + +```swift +extension Dictionary { + public func compactMapValues(_ transform: (Value) throws -> T?) rethrows -> [Key: T] { + return try self.reduce(into: [Key: T](), { (result, x) in + if let value = try transform(x.value) { + result[x.key] = value + } + }) + } +} +``` + +## Source compatibility + +This change is purely additive so has no source compatibility consequences. + +## Effect on ABI stability + +This change is purely additive so has no ABI stability consequences. + +## Effect on API resilience + +This change is purely additive so has no API resilience consequences. + +## Alternatives considered + +We can simply omit this method from the standard library -- however, we already have `mapValues` and `filter`, and it seems reasonable to fill the API hole left between them with a standard extension. diff --git a/proposals/0219-package-manager-dependency-mirroring.md b/proposals/0219-package-manager-dependency-mirroring.md new file mode 100644 index 0000000000..ecf960870f --- /dev/null +++ b/proposals/0219-package-manager-dependency-mirroring.md @@ -0,0 +1,91 @@ +# Package Manager Dependency Mirroring + +* Proposal: [SE-0219](0219-package-manager-dependency-mirroring.md) +* Authors: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Boris Bügling](https://github.com/neonichu) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift-package-manager#1776](https://github.com/apple/swift-package-manager/pull/1776) +* Bug: [SR-8328](https://bugs.swift.org/browse/SR-8328) + +## Introduction + +A dependency mirror refers to an alternate source location which exactly replicates the contents of the original source. This is a proposal for adding support for dependency mirroring in SwiftPM. + +## Motivation + +Dependency mirroring is useful for several reasons: + +- **Availability**: Mirrors can ensure that a dependency can be always fetched, in case the original source is unavailable or even deleted. +- **Cache**: Access to the original source location could be slow or forbidden in the current environment. +- **Validation**: Mirrors can help with screening the upstream updates before making them available internally within a company. + +## Proposed solution + +We propose to introduce a "package configuration" file to store per-dependency mirroring information that SwiftPM can use as additional input. + +We propose to allow registering a mirror using the following command: + +```sh +$ swift package config set-mirror \ + --package-url \ + --mirror-url + +# Example: + +$ swift package config set-mirror \ + --package-url https://github.com/Core/libCore.git \ + --mirror-url https://mygithub.com/myOrg/libCore.git +``` + +A dependency's mirror URL will be used instead of its original URL to perform all relevant git operations, such as fetching and updating the dependency. It will be possible to mirror both direct and transitive dependencies of a package. + +## Detailed design + +### Package Configuration File + +The package configuration file will be expected at this location: + + /.swiftpm/config + +Similar to the `Package.resolved` file, the configuration file of a dependency will not affect a top-level package. + +This file will be managed through SwiftPM commands and users are not expected to edit it by hand. The format of this file is an implementation detail but it will be JSON in practice. + +The configuration file can be expanded to add other information if it makes sense to add it there. Other tools and IDEs written on top of SwiftPM, can also use the `.swiftpm` directory to store their auxiliary files. + +### Dependency Mirroring + +In addition to the `set-mirror` command described above, SwiftPM will provide a command to unset mirror URLs: + +```sh +$ swift package config unset-mirror \ + (--mirror-url | --package-url | --all) + +# Examples: + +$ swift package config unset-mirror --package-url https://github.com/Core/libCore.git +$ swift package config unset-mirror --mirror-url https://mygithub.com/myOrg/libCore.git +$ swift package config unset-mirror --all +``` + +A dependency can have only one mirror URL at a time; `set-mirror` command will replace any previous mirror URL for that dependency. + +SwiftPM will allow overriding the path of the configuration file using the environment variable `SWIFTPM_MIRROR_CONFIG`. This allows using mirrors on arbitrary packages that don't have a config file or require different configurations in different environments. Note that the file at this variable will override only the mirror configuration, if in future we have other configuration stored in the configuration file. + +The `Package.resolved` file will contain the mirror URLs that were used during dependency resolution. + +## Security + +There is no security impact since mirrors only work for the top-level package, and dependencies can't add mirrors on downstream packages. There is a potential privacy concern in case someone accidentally commits their private mirror configuration file in a public package. + +## Impact on existing packages + +This is an additive feature and doesn't impact existing packages. + +## Alternatives considered + +We considered using a dedicated file for storing mirror information. However, there is no good reason to have a new file specifically for mirrors. A generic file gives us flexibility if we discover the need to store more configuration. + +We considered adding a global configuration file for storing the mirror information. A global file could be convenient for some users, but it can also cause "gotcha" moments if the file is used when it shouldn't be used and vice versa. Users who understand this risk can export the `SWIFTPM_MIRROR_CONFIG` in their shell to achieve a similar effect. + +We considered storing the mirroring information in the `Package.swift` manifest file but that doesn't fit well with several of the use-cases of mirrors. The manifest file is also fundamentally different from the configuration file. The manifest file defines how a package is configured and built, whereas the mirror configuration provides overrides for fetching the package dependencies. For e.g., different users would want to use different mirror files depending on their environment, or a user may want to use a mirror on a package they don't have permission to edit. diff --git a/proposals/0220-count-where.md b/proposals/0220-count-where.md new file mode 100644 index 0000000000..a723213269 --- /dev/null +++ b/proposals/0220-count-where.md @@ -0,0 +1,70 @@ +# `count(where:)` + +* Proposal: [SE-0220](0220-count-where.md) +* Author: [Soroush Khanlou](https://github.com/khanlou) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.0)** +* Implementation: [apple/swift#72705](https://github.com/apple/swift/pull/72705) +* Review: ([first pitch](https://forums.swift.org/t/count-where-on-sequence/11186)) ([first review](https://forums.swift.org/t/se-0220-count-where/15048)) ([first acceptance](https://forums.swift.org/t/accepted-se-0220-count-where/15280)) ([second pitch](https://forums.swift.org/t/pitch-restore-count-where-from-se-0220/65859)) ([second review](https://forums.swift.org/t/refresh-review-se-0220-count-where/66235)) ([second acceptance](https://forums.swift.org/t/accepted-again-se-0220-count-where/66659)) + +## Introduction + +While Swift's `Sequence` models brings a lot of niceties that we didn't have access to in Objective-C, like `map` and `filter`, there are other useful operations on sequences that the standard library doesn't support yet. One current missing operation is `count(where:)`, which counts the number of elements in a `Sequence` that pass some test. + +## Motivation + +Counting the number of objects that pass a test has a wide range of uses in many domains. However, Swift currently doesn't give its users a simple way to perform this operation. While the behavior can currently be approximated with a `filter` and a `count`, this approach creates an intermediate array which it immediately discards. This is a bit wasteful. + + [1, 2, 3, -1, -2].filter({ $0 > 0 }).count // => 3 + +To correctly avoid a potentially expensive intermediate array, you can use the Swift's `lazy` subsystem: + + [1, 2, 3, -1, -2].lazy.filter({ $0 > 0 }).count // => 3 + +However, using `lazy` comes with the downside of being forced to use an `@escaping` block. Lastly, you could rely on an eminently unreadable `reduce`: + + [1, 2, 3, -1, -2].reduce(0) { $1 > 0 ? $0 + 1 : $0 } + +These three solutions lie on a spectrum between "easy to write, but include performance traps" to "performant, but require Swift arcana to write". + +## Proposed solution + +The proposed solution would avoid a performance trap and provide a simple interface for users to both read and write. Autocomplete should present it to them handily as well. + + [1, 2, 3, -1, -2].count(where: { $0 > 0 }) // => 3 + +I use it as an extension in my code regularly, and I think it'd make a nice addition to the standard library. + +## Detailed design + +A reference implementation for the function is included here: + + extension Sequence { + func count(where predicate: (Element) throws -> Bool) rethrows -> Int { + var count = 0 + for element in self { + if try predicate(element) { + count += 1 + } + } + return count + } + } + +The recommended implementation can be found [in a pull request to `apple/swift`](https://github.com/apple/swift/pull/16099). + +## Source compatibility + +This change is additive only. + +## Effect on ABI stability + +This change is additive only. + +## Effect on API resilience + +This change is additive only. + +## Alternatives considered + +One alternative worth discussing is the addition of `count(of:)`, which can be implemented on sequences where `Element: Equatable`. This function returns the count of all objects that are equal to the parameter. I'm open to amending this proposal to include this function, but in practice I've never used or needed this function, so I've omitted it here. diff --git a/proposals/0221-character-properties.md b/proposals/0221-character-properties.md new file mode 100644 index 0000000000..d47a95842c --- /dev/null +++ b/proposals/0221-character-properties.md @@ -0,0 +1,295 @@ +# Character Properties + +* Proposal: [SE-0221](0221-character-properties.md) +* Authors: [Michael Ilseman](https://github.com/milseman), [Tony Allevato](https://github.com/allevato) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#20520](https://github.com/apple/swift/pull/20520) +* Review: [Discussion thread](https://forums.swift.org/t/se-0221-character-properties/14686), [Announcement thread](https://forums.swift.org/t/accepted-with-modification-se-0221-character-properties/14944/2) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/fdb725c240033c5273860b0a66d2189d62a97608/proposals/0221-character-properties.md) + +## Introduction + +@allevato (a co-author here) proposed [Add Unicode Properties to Unicode.Scalar](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0211-unicode-scalar-properties.md), which exposes Unicode properties from the [Unicode Character Database](http://unicode.org/reports/tr44/). These are Unicode expert/enthusiast oriented properties that give a finer granularity of control and answer highly-technical and specific Unicody enquiries. + +However, they are not ergonomic and Swift makes no attempt to clarify their interpretation or usage: meaning and proper interpretation is directly tied to the Unicode Standard and the version of Unicode available at run time. There’s some low-hanging ergo-fruit ripe for picking by exposing properties directly on `Character`. + +Pitch thread: [Character and String properties](https://forums.swift.org/t/pitch-character-and-string-properties/11620) + +## Motivation + +`String` is a collection whose element is `Character`, which represents an [extended grapheme cluster](https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) (commonly just called “grapheme”). This makes `Character` one of the first types encountered both by newcomers to Swift as well as by experienced Swift developers playing around in new domains (e.g. scripting). Yet `Character` exposes little functionality other than the ability to order it with respect to other characters, and a way to access the raw [Unicode scalar values](https://unicode.org/glossary/#unicode_scalar_value) that comprise it. + +This proposal adds several queries to increase the usefulness of `Character` and approachability of programming in Swift. It tries to walk the fuzzy line between what Swift can give reasonably good answers to, and what would require the user to adopt more elaborate linguistic analysis frameworks or techniques. + +## Proposed Solution +(Note that Unicode does not define properties on graphemes in general. Swift is defining its own semantics in terms of Unicode semantics derived from scalar properties, semantics on strings, or both) + +### Character Properties + +```swift +extension Character { + /// Whether this Character is ASCII. + @inlinable + public var isASCII: Bool { ... } + + /// Returns the ASCII encoding value of this Character, if ASCII. + /// + /// Note: "\r\n" (CR-LF) is normalized to "\n" (LF), which will return 0x0A + @inlinable + public var asciiValue: UInt8? { ... } + + /// Whether this Character represents whitespace, including newlines. + /// + /// Examples: + /// * "\t" (U+0009 CHARACTER TABULATION) + /// * " " (U+0020 SPACE) + /// * U+2029 PARAGRAPH SEPARATOR + /// * U+3000 IDEOGRAPHIC SPACE + /// + public var isWhitespace: Bool { ... } + + /// Whether this Character represents a newline. + /// + /// Examples: + /// * "\n" (U+000A): LINE FEED (LF) + /// * "\r" (U+000D): CARRIAGE RETURN (CR) + /// * "\r\n" (U+000A U+000D): CR-LF + /// * U+0085: NEXT LINE (NEL) + /// * U+2028: LINE SEPARATOR + /// * U+2029: PARAGRAPH SEPARATOR + /// + public var isNewline: Bool { ... } + + /// Whether this Character represents a number. + /// + /// Examples: + /// * "7" (U+0037 DIGIT SEVEN) + /// * "⅚" (U+215A VULGAR FRACTION FIVE SIXTHS) + /// * "㊈" (U+3288 CIRCLED IDEOGRAPH NINE) + /// * "𝟠" (U+1D7E0 MATHEMATICAL DOUBLE-STRUCK DIGIT EIGHT) + /// * "๒" (U+0E52 THAI DIGIT TWO) + /// + public var isNumber: Bool { ... } + + /// Whether this Character represents a whole number. See + /// `Character.wholeNumberValue` + @inlinable + public var isWholeNumber: Bool { ... } + + /// If this Character is a whole number, return the value it represents, else + /// nil. + /// + /// Examples: + /// * "1" (U+0031 DIGIT ONE) => 1 + /// * "५" (U+096B DEVANAGARI DIGIT FIVE) => 5 + /// * "๙" (U+0E59 THAI DIGIT NINE) => 9 + /// * "万" (U+4E07 CJK UNIFIED IDEOGRAPH-4E07) => 10_000 + /// + public var wholeNumberValue: Int? { ... } + + /// Whether this Character represents a hexadecimal digit. + /// + /// Hexadecimal digits include 0-9, Latin letters a-f and A-F, and their + /// fullwidth compatibility forms. To get their value, see + /// `Character.hexadecimalDigitValue` + @inlinable + public var isHexadecimalDigit: Bool { ... } + + /// If this Character is a hexadecimal digit, returns the value it represents, + /// else nil. + /// + /// Hexadecimal digits include 0-9, Latin letters a-f and A-F, and their + /// fullwidth compatibility forms. + public var hexadecimalDigitValue: Int? { ... } + + /// Whether this Character is a letter. + /// + /// Examples: + /// * "A" (U+0041 LATIN CAPITAL LETTER A) + /// * "é" (U+0065 LATIN SMALL LETTER E, U+0301 COMBINING ACUTE ACCENT) + /// * "ϴ" (U+03F4 GREEK CAPITAL THETA SYMBOL) + /// * "ڈ" (U+0688 ARABIC LETTER DDAL) + /// * "日" (U+65E5 CJK UNIFIED IDEOGRAPH-65E5) + /// * "ᚨ" (U+16A8 RUNIC LETTER ANSUZ A) + /// + public var isLetter: Bool { ... } + + /// Perform case conversion to uppercase + /// + /// Examples: + /// * "é" (U+0065 LATIN SMALL LETTER E, U+0301 COMBINING ACUTE ACCENT) + /// => "É" (U+0045 LATIN CAPITAL LETTER E, U+0301 COMBINING ACUTE ACCENT) + /// * "и" (U+0438 CYRILLIC SMALL LETTER I) + /// => "И" (U+0418 CYRILLIC CAPITAL LETTER I) + /// * "π" (U+03C0 GREEK SMALL LETTER PI) + /// => "Π" (U+03A0 GREEK CAPITAL LETTER PI) + /// * "ß" (U+00DF LATIN SMALL LETTER SHARP S) + /// => "SS" (U+0053 LATIN CAPITAL LETTER S, U+0053 LATIN CAPITAL LETTER S) + /// + /// Note: Returns a String as case conversion can result in multiple + /// Characters. + public func uppercased() -> String { ... } + + /// Perform case conversion to lowercase + /// + /// Examples: + /// * "É" (U+0045 LATIN CAPITAL LETTER E, U+0301 COMBINING ACUTE ACCENT) + /// => "é" (U+0065 LATIN SMALL LETTER E, U+0301 COMBINING ACUTE ACCENT) + /// * "И" (U+0418 CYRILLIC CAPITAL LETTER I) + /// => "и" (U+0438 CYRILLIC SMALL LETTER I) + /// * "Π" (U+03A0 GREEK CAPITAL LETTER PI) + /// => "π" (U+03C0 GREEK SMALL LETTER PI) + /// + /// Note: Returns a String as case conversion can result in multiple + /// Characters. + public func lowercased() -> String { ... } + + /// Whether this Character is considered uppercase. + /// + /// Uppercase Characters vary under case-conversion to lowercase, but not when + /// converted to uppercase. + /// + /// Examples: + /// * "É" (U+0045 LATIN CAPITAL LETTER E, U+0301 COMBINING ACUTE ACCENT) + /// * "И" (U+0418 CYRILLIC CAPITAL LETTER I) + /// * "Π" (U+03A0 GREEK CAPITAL LETTER PI) + /// + @inlinable + public var isUppercase: Bool { ... } + + /// Whether this Character is considered lowercase. + /// + /// Lowercase Characters vary under case-conversion to uppercase, but not when + /// converted to lowercase. + /// + /// Examples: + /// * "é" (U+0065 LATIN SMALL LETTER E, U+0301 COMBINING ACUTE ACCENT) + /// * "и" (U+0438 CYRILLIC SMALL LETTER I) + /// * "π" (U+03C0 GREEK SMALL LETTER PI) + /// + @inlinable + public var isLowercase: Bool { ... } + + /// Whether this Character changes under any form of case conversion. + @inlinable + public var isCased: Bool { ... } + + /// Whether this Character represents a symbol + /// + /// Examples: + /// * "®" (U+00AE REGISTERED SIGN) + /// * "⌹" (U+2339 APL FUNCTIONAL SYMBOL QUAD DIVIDE) + /// * "⡆" (U+2846 BRAILLE PATTERN DOTS-237) + /// + public var isSymbol: Bool { ... } + + /// Whether this Character represents a symbol used in mathematical formulas + /// + /// Examples: + /// * "+" (U+002B PLUS SIGN) + /// * "∫" (U+222B INTEGRAL) + /// * "ϰ" (U+03F0 GREEK KAPPA SYMBOL) + /// + /// Note: This is not a strict subset of isSymbol. This includes characters + /// used both as letters and commonly in mathematical formulas. For example, + /// "ϰ" (U+03F0 GREEK KAPPA SYMBOL) is considered a both mathematical symbol + /// and a letter. + /// + public var isMathSymbol: Bool { ... } + + /// Whether this Character represents a currency symbol + /// + /// Examples: + /// * "$" (U+0024 DOLLAR SIGN) + /// * "¥" (U+00A5 YEN SIGN) + /// * "€" (U+20AC EURO SIGN) + public var isCurrencySymbol: Bool { ... } + + /// Whether this Character represents punctuation + /// + /// Examples: + /// * "!" (U+0021 EXCLAMATION MARK) + // * "؟" (U+061F ARABIC QUESTION MARK) + /// * "…" (U+2026 HORIZONTAL ELLIPSIS) + /// * "—" (U+2014 EM DASH) + /// * "“" (U+201C LEFT DOUBLE QUOTATION MARK) + /// + public var isPunctuation: Bool { ... } +} +``` + +## Detailed Semantics and Rationale + +Some fuzziness is inherent in modeling human writing systems and the rules of grapheme breaking allow for semantically meaningless, yet technically valid, graphemes. In light of all this, we make a best effort and try to discover some principle to follow. Principles are useful for evaluating tradeoffs, but are not hard rules that always lead to a single clear answer. + +The closest applicable principle might be something similar to W3C’s [Principle of Tolerance](https://www.w3.org/DesignIssues/Principles.html), paraphrased as “Be liberal in what you accept, conservative in what you produce”. Character properties can be roughly grouped into those that “produce” specific values or behaviors, and those that “accept” graphemes under a fuzzy classification. + +### Restrictive Properties + +Properties that provide a clear interpretation or which the stdlib produces a specific value for should be restrictive. One example is `wholeNumberValue`. `wholeNumberValue` *produces* an `Int` from a `Character`, which means it needs to be *restrictive*, permitting only the graphemes with unambiguous whole number values. It only returns a value for single-scalar graphemes whose sole scalar has an integral numeric value. Thus, `wholeNumberValue` returns nil for “7̅” (7 followed by U+0305 COMBINING OVERLINE) as there is no clear interpretation of the value. Any attempt to produce a specific integer from “7̅” would be suspect. + +Restrictive properties typically accept/reject based on an analysis of the entire grapheme. + +* Values: `isASCII` / `asciiValue`, `isWholeNumber` / `wholeNumberValue`, `isHexDigit` / `hexDigitValue` +* Casing: `isUppercase` / `uppercased()`, `isLowercase` / `lowercased()`, `isCased` + +### Permissive Properties + +Where there is no clear interpretation or specific value to produce, we try to be as permissive as reasonable. For example, `isLetter` just queries the first scalar to see if it is “letter-like”, and thus handles unforeseeable combinations of a base letter-like scalar with subsequent combining, modifying, or extending scalars. `isLetter` merely answers a general (fuzzy) question, but doesn’t prescribe further interpretation. + +Permissive APIs should in general be non-inlinable and their documentation may be less precise regarding details and corner cases. This allows for a greater degree of library evolution. Permissive properties typically accept/reject based on an analysis of part of the grapheme. + +* Fuzzy queries: `isNumber`, `isLetter`, `isSymbol` / `isMathSymbol` / `isCurrencySymbol`, `isPunctuation` + +#### Newlines and Whitespace + +Newlines encompass more than hard line-breaks in traditional written language; they are common terminators for programmer strings. Whether a `Character` such as `"\n\u{301}"` (a newline with a combining accent over it) is a newline is debatable. Either interpretation can lead to inconsistencies. If true, then a program might skip the first scalar in a new entry (whatever such a combining scalar at the start could mean). If false, then a `String` with newline terminators inside of it would return false for `myStr.contains { $0.isNewline }`, which is counter-intuitive. The same is true of whitespace. + +We recommend that the precise semantics of `isWhitespace` and `isNewline` be unspecified regarding graphemes consisting of leading whitespace/newlines followed by combining scalars. + +## Source Compatibility +The properties on `Character` are strictly additive. + +## Effect on ABI Stability +These changes are ABI-additive: they introduce new ABI surface area to keep stable. + +The proposed solution includes recommended `@inlinable` annotations on properties which derive their value from other properties (thus benefitting from optimizations), or which are well-defined and stable under future Unicode versions (e.g. ASCII-related properties). + +## Additions and Alternatives Considered + +### Titlecase + +Titlecase can be useful for some legacy scalars (ligatures) as well as for Strings when combined with word-breaking logic. However, it seems pretty obscure to surface on Character directly. + +### String.Lines, String.Words + +These have been deferred from this pitch to keep focus and await a more generalized lazy split collection. + +### Rename Permissive `isFoo` to `hasFoo` + +This was mentioned above in discussion of `isNewline` semantics and could also apply to `isWhitespace`. However, it would be awkward for `isNumber` or `isLetter`. What the behavior should be for exotic whitespace and newlines is heavily debatable. We’re sticking to `isNewline/isWhitespace` for now, but are open to argument. + +### Design as `Character.has(OptionSet<…>, exclusively: …)` + +There could be something valuable to glean from this, but we reject this approach as somewhat un-Swifty with a poor discovery experience, especially for new or casual users. It does, however, make the semantic distinctions above very explicit at the call site. + +### Add Failable FixedWidthInteger/FloatingPoint Initializers Taking Character + +In addition to (or perhaps instead of) properties like `wholeNumberValue`, add `Character`-based `FixedWidthInteger.init?(_:Character)`. Similarly `FloatingPoint.init?(_:Character)` which includes vulgar fractions and the like (if single-scalar, perhaps). However, these do not have direct counterparts in this pitch as named, at least without an explicit argument label clarifying their semantics. + +We could consider adding something like `FixedWidthInteger.init?(hexDigit: Character)` and `FixedWithInteger.init?(wholeNumber: Character)` which correspond to `hexDigitValue` and `wholeNumberValue`, and similarly a counterpart for `String`. But, we don’t feel this carries its weight as surfaced directly at the top level of e.g. `Int`. We prefer to keep this avenue open for future directions involving more general number parsing and grapheme evaluation logic. + +### Drop `isASCII/HexDigit/WholeNumber`: Check for `nil` Instead + +This alternative is to drop `isASCII`, `isHexDigit`, and `isWholeNumber` and instead use `if let` or compare explicitly to `nil`. + +We decided to provide these convenience properties both for discoverability as well as use in more complex expressions: `c.isHexDigit && c.isLetter`, `c.isASCII && c.isWhitespace`, etc. We don’t think they add significant weight or undue API surface area. + +### Add `numericValue: Double?` in addition to, or instead of, `wholeNumberValue: Int?`. Alternatively, add a `rationalValue: (numerator: Int, denominator: Int)?`. + +Unicode defines numeric values for whole numbers, hex digits, and rational numbers (vulgar fractions). As an implementation artifact (ICU only vends a double), Unicode.Scalar.Properties’s `numericValue` is a double rather than an enum of a rational or whole number. We could follow suit and add such a value to `Character`, restricted to single-scalar graphemes. We could also remove `wholeNumberValue`, letting users test if the double is integral. Alternatively or additionally, we could provide a `rationalValue` capable of handling non-whole-numbers. + +As far as adding a `rationalValue` is concerned, we do not feel that support for vulgar fractions and other obscure Unicode scalars (e.g. baseball score-keeping) warrants an addition to `Character` directly. `wholeNumberValue` producing an Int is a more fluid solution than a Double which happens to be integral, so we’re hesitant to replace `wholeNumberValue` entirely with a `numericValue`. Since `numericValue` would only add utility for these obscure characters, we’re not sure if it’s worth adding. + +Suggestions for alternative names for `wholeNumberValue`would be appreciated. diff --git a/proposals/0222-lazy-compactmap-sequence.md b/proposals/0222-lazy-compactmap-sequence.md new file mode 100644 index 0000000000..c5cbcdb402 --- /dev/null +++ b/proposals/0222-lazy-compactmap-sequence.md @@ -0,0 +1,187 @@ +# Lazy CompactMap Sequence + +* Proposal: [SE-0222](0222-lazy-compactmap-sequence.md) +* Authors: [TellowKrinkle](https://github.com/TellowKrinkle), [Johannes Weiß](https://github.com/weissi) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Rejected** +* Implementation: [apple/swift#14841](https://github.com/apple/swift/pull/14841) +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0222-lazy-compactmap-sequence/14850/16) + +## Introduction + +Chaining multiple `.map()`s and `.filter()`s on a lazy collection leads +to suboptimal codegen, as well as large, painful type names. +To improve this, we propose adding a `LazyCompactMap{Sequence, Collection}` +type along with some overloads on the other lazy collection types' `.map(_:)` +and `.filter(_:)` functions which return this type to get better codegen +and shorter type names. + +Swift-evolution thread: [Discussion thread topic for the proposal](https://forums.swift.org/t/introduce-lazy-version-of-compactmap/9835/1) + +## Motivation + +The current lazy system is very good for easily defining transforms on collections, but certain constructs can lead to less-than-optimal codegen. + +For example, the code collection.map(map1).filter(filter1).map(map2).filter(filter2) will lead to code like this in the formIndex(after:) method: + +```swift +do { + do { + collection.formIndex(after: &i) + } while !filter1(map1(collection[i])) +} while !filter2(map2(map1(collection[i]))) +``` +while it could be represented with this more efficient single loop: +```swift +do { + collection.formIndex(after: &i) +} while !filter1(map1(collection[i])) && !filter2(map2(map1(collection[i]))) +``` +Currently, you can get a single loop by instead using this compactMap: +```swift +collection.compactMap { + let a = map1($0) + guard filter1(a) else { return nil } + let b = map2(a) + guard filter2(b) else { return nil } + return b +} +``` +but this removes the nice composability of the chained map/filter combination. + +The standard library recently got an override on LazyMapCollection and LazyFilterCollection +which combines multiple filters and maps in a row, however it does not work with alternating maps and filters. + +## Proposed solution + +Define a `LazyCompactMapCollection` collection (and sequence) which represents a compactMap. +Then, add overrides on `LazyMapCollection.filter`, `LazyFilterCollection.map`, +`Lazy*Collection.compactMap`, and `LazyCompactMapCollection.{filter, map}` +to return a `LazyCompactMapCollection` that combines all the maps and filters. + +As an added bonus, you’ll never see a giant chain of +`LazyMapCollection, ...>` again + +## Detailed design + +A new `LazyCompactMapCollection` and equivalent Sequence should be defined like so: +```swift +public struct LazyCompactMapCollection { + internal var _base: Base + internal let _transform: (Base.Element) -> Element? + + internal init(_base: Base, transform: @escaping (Base.Element) -> Element?) { + self._base = _base + self._transform = transform + } +} +``` +with a very similar set of overrides to the current `LazyFilterCollection` + +Then, the following extensions should be added (with equivalent ones for Lazy Sequences): +```swift +extension LazyMapCollection { + public func compactMap(_ transform: @escaping (Element) -> U?) -> LazyCompactMapCollection { + let mytransform = self._transform + return LazyCompactMapCollection( + _base: self._base, + transform: { transform(mytransform($0)) } + ) + } + + public func filter(_ isIncluded: @escaping (Element) -> Bool) -> LazyCompactMapCollection { + let mytransform = self._transform + return LazyCompactMapCollection( + _base: self._base, + transform: { + let transformed = mytransform($0) + return isIncluded(transformed) ? transformed : nil + } + ) + } +} + +extension LazyFilterCollection { + public func compactMap(_ transform: @escaping (Base.Element) -> U?) -> LazyCompactMapCollection { + let mypredicate = self._predicate + return LazyCompactMapCollection( + _base: self._base, + transform: { mypredicate($0) ? transform($0) : nil } + ) + } + + public func map(_ transform: @escaping (Base.Element) -> U) -> LazyCompactMapCollection { + let mypredicate = self._predicate + return LazyCompactMapCollection( + _base: self._base, + transform: { mypredicate($0) ? transform($0) : nil } + ) + } +} + +extension LazyCompactMapCollection { + public func compactMap(_ transform: @escaping (Element) -> U?) -> LazyCompactMapCollection { + let mytransform = self._transform + return LazyCompactMapCollection( + _base: self._base, + transform: { + guard let halfTransformed = mytransform($0) else { return nil } + return transform(halfTransformed) + } + ) + } + + public func map(_ transform: @escaping (Element) -> U) -> LazyCompactMapCollection { + let mytransform = self._transform + return LazyCompactMapCollection( + _base: self._base, + transform: { + guard let halfTransformed = mytransform($0) else { return nil } + return transform(halfTransformed) + } + ) + } + + public func filter(_ isIncluded: @escaping (Element) -> Bool) -> LazyCompactMapCollection { + let mytransform = self._transform + return LazyCompactMapCollection( + _base: self._base, + transform: { + guard let halfTransformed = mytransform($0), isIncluded(halfTransformed) else { return nil } + return halfTransformed + } + ) + } +} +``` + +## Source compatibility + +In Swift 5, while most code will work with the new extensions, code that relies on +the return type of `LazyCollection.compactMap(_:)` will break. + +In addition, code like following code will break: +```swift +let array = [0, 1, 22] +let tmp = array.lazy.map(String.init).filter { $0.count == 1 } +let filtered: LazyFilterCollection> = tmp +``` + +However, this type of code is probably rare and similar code will already +be broken by the previously mentioned change that coalesces +`.filter(_:).filter(_:)` and `.map(_:).map(_:)` + +## Effect on ABI stability + +N/A + +## Effect on API resilience + +N/A + +## Alternatives considered + +The main alternative would be to not do this at all. This alternative +isn't great, as it can be many times slower when the map/filter functions +do little work, as shown by [this test](https://gist.github.com/tellowkrinkle/818c8d9ce467f272c889bdd503784d63) + diff --git a/proposals/0223-array-uninitialized-initializer.md b/proposals/0223-array-uninitialized-initializer.md new file mode 100644 index 0000000000..cd78c92bc4 --- /dev/null +++ b/proposals/0223-array-uninitialized-initializer.md @@ -0,0 +1,350 @@ +# Accessing an Array's Uninitialized Buffer + +* Proposal: [SE-0223](0223-array-uninitialized-initializer.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Withdrawn** +* Next Proposal: [SE-0245](0245-array-uninitialized-initializer.md) +* Decision Notes: [Returned for revision](https://forums.swift.org/t/se-0223-accessing-an-arrays-uninitialized-buffer/15194/41) +* Implementation: [apple/swift#17389](https://github.com/apple/swift/pull/17389) +* Bug: [SR-3087](https://bugs.swift.org/browse/SR-3087) + +## Introduction + +This proposal suggests a new initializer and method for `Array` and `ContiguousArray` +that provide access to an array's uninitialized storage buffer. + +Swift-evolution thread: [https://forums.swift.org/t/array-initializer-with-access-to-uninitialized-buffer/13689](https://forums.swift.org/t/array-initializer-with-access-to-uninitialized-buffer/13689) + +## Motivation + +Some collection operations require working on a fixed-size buffer of uninitialized memory. +For example, one O(*n*) algorithm for performing a stable partition of an array is as follows: + +1. Create a new array the same size as the original array. +2. Iterate over the original array, + copying matching elements to the beginning of the new array + and non-matching elements to the end. +3. When finished iterating, reverse the slice of non-matching elements. + +Unfortunately, the standard library provides no way to create an array +of a particular size without allocating every element, +or to copy elements to the end of an array's buffer +without initializing every preceding element. +Even if we avoid initialization by manually allocating the memory using an `UnsafeMutableBufferPointer`, +there's no way to convert that buffer into an array without copying the contents. +There simply isn't a way to implement this particular algorithm with maximum efficiency in Swift. + +We also see this limitation when working with C APIs +that fill a buffer with an unknown number of elements and return the count. +The workarounds are the same as above: +either initialize an array before passing it +or copy the elements from an unsafe mutable buffer into an array after calling. + +## Proposed solution + +Adding a new `Array` initializer +that lets a program work with an uninitialized buffer, +and a method for accessing an existing array's buffer +of both initialized and uninitialized memory, +would fill in this missing functionality. + +The new initializer takes a closure that operates on an `UnsafeMutableBufferPointer` +and an `inout` count of initialized elements. +This closure has access to the uninitialized contents +of the newly created array's storage, +and must set the initialized count of the array before exiting. + +```swift +var myArray = Array(unsafeUninitializedCapacity: 10) { buffer, initializedCount in + for x in 1..<5 { + buffer[x] = x + } + buffer[0] = 10 + initializedCount = 5 +} +// myArray == [10, 1, 2, 3, 4] +``` + +With this new initializer, it's possible to implement the stable partition +as an extension to the `Collection` protocol, without any unnecessary copies: + +```swift +func stablyPartitioned(by belongsInFirstPartition: (Element) throws -> Bool) rethrows -> [Element] { + return try Array(unsafeUninitializedCapacity: count) { + buffer, initializedCount in + var low = buffer.baseAddress! + var high = low + buffer.count + do { + for element in self { + if try belongsInFirstPartition(element) { + low.initialize(to: element) + low += 1 + } else { + high -= 1 + high.initialize(to: element) + } + } + + let highIndex = high - buffer.baseAddress! + buffer[highIndex...].reverse() + initializedCount = buffer.count + } catch { + let lowCount = low - buffer.baseAddress! + let highCount = (buffer.baseAddress! + buffer.count) - high + buffer.baseAddress!.deinitialize(count: lowCount) + high.deinitialize(count: highCount) + throw error + } + } +} +``` + +## Detailed design + +The new initializer and method are added to both `Array` and `ContiguousArray`. + +```swift +/// Creates an array with the specified capacity, then calls the given closure +/// with a buffer covering the array's uninitialized memory. +/// +/// The closure must set its second parameter to a number `c`, the number +/// of elements that are initialized. The memory in the range `buffer[0.., + _ initializedCount: inout Int + ) throws -> Void +) rethrows + +/// Calls the given closure with a buffer of the array's mutable contiguous +/// storage, reserving the specified capacity if necessary. +/// +/// The closure must set its second parameter to a number `c`, the number +/// of elements that are initialized. The memory in the range `buffer[0..( + capacity: Int, + _ body: ( + _ buffer: inout UnsafeMutableBufferPointer, + _ initializedCount: inout Int + ) throws -> Result +) rethrows -> Result +``` + +### Specifying a capacity + +Both the initializer and the mutating method take +the specific capacity that a user wants to work with as a parameter. +In each case, the buffer passed to the closure has a count +that is exactly the same as the specified capacity, +even if the ultimate capacity of the new or existing array is larger. +This helps avoid bugs where a user assumes that the capacity they observe +before calling the mutating method would match the size of the buffer. + +The method requires that the capacity specified be at least the current `count` of the array +to prevent nonsensical operations, +like reducing the size of the array from the middle. +That is, this will result in a runtime error: + +```swift +var a = Array(1...10) +a.withUnsafeMutableBufferPointerToStorage(capacity: 5) { ... } +``` + +### Guarantees after throwing + +If the closure parameter to either the initializer +or the mutating method throws, +the `initializedCount` value at the time an error is thrown is assumed to be correct. +This means that a user who needs to throw from inside the closure has one of two options. +Before throwing, they must: + +1. deinitialize any newly initialized instances or re-initialize any deinitialized instances, or +2. update `initializedCount` to the new count. + +In either case, +the postconditions that `buffer[0.., + _ initializedCount: inout Int + ) throws -> Void + ) rethrows { + self = [] + try self.withUnsafeMutableBufferPointerToStorage(capacity: unsafeUninitializedCapacity, initializer) + } + + public mutating func withUnsafeMutableBufferPointerToStorage( + capacity: Int, + _ body: ( + _ buffer: inout UnsafeMutableBufferPointer, + _ initializedCount: inout Int + ) throws -> Result + ) rethrows -> Result { + var buffer = UnsafeMutableBufferPointer.allocate(capacity: capacity) + buffer.initialize(from: self) + var initializedCount = self.count + defer { + buffer.baseAddress?.deinitialize(count: initializedCount) + buffer.deallocate() + } + + let result = try body(&buffer, &initializedCount) + self = Array(buffer[..=4.2) +// This will only be executed if the Swift version is less than 4.2. +#endif + +#if !compiler(>=4.2) +// This will only be executed if the Swift compiler version is less than 4.2. +#endif +``` + +With the introduction of support for the "<" unary operator, the +refactored code would be more clear and readable: + +```swift +#if swift(<4.2) +// This will only be executed if the Swift version is less than 4.2. +#endif + +#if compiler(<4.2) +// This will only be executed if the Swift compiler version is less than 4.2. +#endif +``` + +In the former snippet, the `!` can be easily missed in a code +review. The latter snippet reads more like plain English. + +Support for other operators like "<=" and ">" is not desired, as they +make a statement about future releases and they don't account for +patch releases. That means that client code will need to be updated if +a patch release didn't fix a particular issue with the compiler, for +example. + +## Proposed solution + +The solution is small change in the parser so that the operator "<" is +supported for both the `#if swift` and `#if compiler` conditions. Diagnostic +messages about invalid unary operators must be updated as well. + +## Detailed design + +The place in the parser where `#if swift(...)` is parsed is +`ParseIfConfig.cpp`. There are two classes that will require +modification: `ValidateIfConfigCondition`, to take into account the +"<" operator, and `EvaluateIfConfigCondition`, to actually evaluate +the new operator semantics. A new '<' operator for `Version` will also +need to be implemented. + +The diagnostic message when the operator is not valid also needs to +change. I propose changing it from + +``` +unexpected platform condition argument: expected a unary comparison, such as '>=2.2' +``` + +to + +``` +unexpected platform condition argument: expected a unary comparison '>=' or '<'; for example, '>=2.2' or '<2.2' +``` + +## Source compatibility + +This has no effect in source compatibility. + +## Effect on ABI stability + +This has no effect in ABI stability. + +## Effect on API resilience + +This has no effect in API resilience. diff --git a/proposals/0225-binaryinteger-iseven-isodd-ismultiple.md b/proposals/0225-binaryinteger-iseven-isodd-ismultiple.md new file mode 100644 index 0000000000..cdcb90b7e1 --- /dev/null +++ b/proposals/0225-binaryinteger-iseven-isodd-ismultiple.md @@ -0,0 +1,187 @@ +# Adding `isMultiple` to `BinaryInteger` + +* Proposal: [SE-0225](0225-binaryinteger-iseven-isodd-ismultiple.md) +* Authors: [Robert MacEachern](https://github.com/robmaceachern), [Micah Hansonbrook](https://github.com/SiliconUnicorn) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.0)** (with modifications, see Implementation Notes) +* Implementation: [apple/swift#18689](https://github.com/apple/swift/pull/18689) +* Review: [Discussion thread](https://forums.swift.org/t/se-0225-adding-iseven-isodd-ismultiple-to-binaryinteger/15382), [Announcement thread](https://forums.swift.org/t/accepted-with-modifications-se-0225-adding-iseven-isodd-ismultiple-to-binaryinteger/15689) + +Note: the title of this proposal has been modified to reflect what was accepted. The original title was "Adding `isEven`, `isOdd`, and `isMultiple` to `BinaryInteger`". + +## Introduction + +This proposal adds `var isEven: Bool`, `var isOdd: Bool`, and `func isMultiple(of other: Self) -> Bool` to the `BinaryInteger` protocol. `isEven` and `isOdd` are convenience properties for querying the [parity](https://en.wikipedia.org/wiki/Parity_(mathematics)) of the integer and `isMultiple` is a more general function to determine whether an integer is a multiple of another integer. + +Swift-evolution thread: [Even and Odd Integers](https://forums.swift.org/t/even-and-odd-integers/11774) + +## Motivation + +It is sometimes necessary to know whether or not an integer is a multiple of another. The most common case is testing if a value is a multiple of 2 (even and oddness). + +**Commonality:** Testing if a value is a multiple of another shows up in a surprising number of contexts including UI code, algorithm implementations (often in the form of assertions), tests, benchmarks, documentation and tutorial/educational code. + +Currently, the most common way to test if a value is a multiple is by using the remainder operator (`%`) checking for a remainder of zero: `12 % 2 == 0 // returns true. 12 is a multiple of 2`. Similarly, testing that a value is _not_ a multiple of another is done by checking for a remainder other than zero: `13 % 2 != 0 // returns true. 13 is not a multiple of 2`. + +Alternatively, it is also possible to use the bitwise AND operator (`&`) to check the even/oddness of a value: `12 & 1 == 0 // returns true`. + +Some examples of testing multiples in code (see more in appendix): + +```swift +// UITableView alternating row colour +cell.contentView.backgroundColor = indexPath.row % 2 == 0 ? .gray : .white + +// Codable.swift.gyb in apple/swift +guard count % 2 == 0 else { throw DecodingError.dataCorrupted(...) } + +// Bool.swift in apple/swift +public static func random(using generator: inout T) -> Bool { + return (generator.next() >> 17) & 1 == 0 +} + +// KeyPath.swift in apple/swift +_sanityCheck(bytes > 0 && bytes % 4 == 0, "capacity must be multiple of 4 bytes") + +// ReversedCollection Index.base documentation https://developer.apple.com/documentation/swift/reversedcollection/index/2965437-base +guard let i = reversedNumbers.firstIndex(where: { $0 % 2 == 0 }) +``` + +Determining whether a value is even or odd is a common question across programming languages, at least based on these Stack Overflow questions: +[c - How do I check if an integer is even or odd?](https://stackoverflow.com/questions/160930/how-do-i-check-if-an-integer-is-even-or-odd) 300,000+ views +[java - Check whether number is even or odd](https://stackoverflow.com/questions/7342237/check-whether-number-is-even-or-odd) 350,000+ views +[Check if a number is odd or even in python](https://stackoverflow.com/questions/21837208/check-if-a-number-is-odd-or-even-in-python) 140,000+ views + +Convenience properties or functions equivalent to `isEven` and `isOdd` are available in the standard libraries of many other programming languages, including: [Ruby](https://ruby-doc.org/core-2.2.2/Integer.html#method-i-odd-3F), [Haskell](http://hackage.haskell.org/package/base-4.11.1.0/docs/Prelude.html#v:even), [Clojure](https://clojuredocs.org/clojure.core/odd_q), and according to [RosettaCode](https://www.rosettacode.org/wiki/Even_or_odd): Julia, Racket, Scheme, Smalltalk, Common Lisp. + +**Readability:** This proposal significantly improves readability, as expressions read like straightforward English sentences. There is no need to mentally parse and understand non-obvious operator precedence rules (`%` has higher precedence than `==`). + +The `isEven` and `isOdd` properties are also fewer characters wide than the remainder approach (maximum 7 characters for `.isEven` vs 9 for ` % 2 == 0`) which saves horizontal space while being clearer in intent. + +```swift +// UITableView alternating row colour +cell.contentView.backgroundColor = indexPath.row.isEven ? .gray : .white + +// Codable.swift.gyb in apple/swift +guard count.isEven else { throw DecodingError.dataCorrupted(...) } + +// Bool.swift in apple/swift +public static func random(using generator: inout T) -> Bool { + return (generator.next() >> 17).isEven +} + +// KeyPath.swift in apple/swift +_sanityCheck(bytes > 0 && bytes.isMultiple(of: 4), "capacity must be multiple of 4 bytes") +``` + +**Discoverability:** IDEs will be able to suggest `isEven`, `isOdd`, and `isMultiple` as part of autocomplete on integer types which will aid discoverability. It will also be familiar to users coming from languages that also support functionality similar to `isEven` and `isOdd`. + +**Trivially composable:** It would be relatively easy to reproduce the proposed functionality in user code but there would be benefits to having a standard implementation. It may not be obvious to some users exactly which protocol these properties belong on (`Int`?, `SignedInteger`?, `FixedWidthInteger`?, `BinaryInteger`?). This inconsistency can be seen in a [popular Swift utility library](https://github.com/SwifterSwift/SwifterSwift/blob/9eb6259faf6689a161825cc91cccec0c82edea8d/Sources/Extensions/SwiftStdlib/SignedIntegerExtensions.swift#L28) which defines `isEven` and `isOdd` on `SignedInteger` which results in the properties being inaccessible for unsigned integers. + +Testing the parity of integers is also relatively common in sample code and educational usage. In this context, it’s usually not appropriate for an author to introduce this functionality (unless they are teaching extensions!) in order to avoid distracting from the main task at hand (e.g. filter, map, etc). It may also be the same situation for authoring test code: it'd be used if it existed but it's not worth the overhead of defining it manually. + +This functionality will also eliminate the need to use the remainder operator or bitwise AND when querying the divisibility of an integer. + +**Correctness:** It isn't [uncommon](https://github.com/apple/swift/blob/4a43ee83e701145d69141adca311497c082b7170/stdlib/public/core/RangeReplaceableCollection.swift#L1090) to see tests for oddness written as `value % 2 == 1` in Swift, but this is incorrect for negative odd values. The semantics of the `%` operator vary between programming languages, such as Ruby and Python, which can be surprising. + +``` +// Swift: +7 % 2 == 1 // true +-7 % 2 == 1 // false. -7 % 2 evaluates to -1 + +// Ruby and Python +7 % 2 == 1 // true +-7 % 2 == 1 // true +``` + +The `%` operator will also trap when the righthand side is zero. The proposed solution does not. + +There is also a minor correctness risk in misinterpreting something like `value % 2 == 0`, particularly when used in a more complex statement, when compared to `value.isEven`, e.g. `bytes > 0 && bytes % 4 == 0`. + +**Performance:** It's _possible_ that `isMultiple` could be implemented in a more performant way than `% divisor == 0` for more complex types, such as a BigInteger/BigNum type. + +The addition of `isEven` and `isOdd` likely won’t have a major positive impact on performance but it should not introduce any additional overhead thanks to `@_transparent`. + +## Proposed solution + +Add two computed properties, `isEven` and `isOdd`, and a function `isMultiple` to the `BinaryInteger` protocol. + +```swift +// Integers.swift.gyb +// On protocol BinaryInteger + + @_transparent + public var isEven: Bool { return _lowWord % 2 == 0 } + + @_transparent + public var isOdd: Bool { return !isEven } + + func isMultiple(of other: Self) -> Bool +``` + +## Detailed design + +N/A + +## Source compatibility + +This is strictly additive. + +## Effect on ABI stability + +N/A + +## Effect on API resilience + +N/A + +## Alternatives considered + +### `isDivisible` instead of `isMultiple` + +The original discussions during the pitch phase where related to an `isDivisible(by:)` alternative to `isMultiple`. [Issues](https://forums.swift.org/t/even-and-odd-integers/11774/83) related to divisibility and division by zero were discussed and `isMultiple` was proposed as a solution that 1) avoids trapping on zero, and 2) avoids confusion where a value that is _divisible_ by zero would not be _dividable_ in Swift. e.g. + +``` +let y = 0 +if 10.isDivisible(by: y) { + let val = 10 / y // traps +} +``` + +### Only `isEven/isOdd` or only `isMultiple`. + +During the pitch phase there were discussions about including only one of `isEven/isOdd` or `isMultiple` in the proposal. + +On the one hand there were concerns that `isEven/isOdd` would not provide enough utility to justify inclusion into the standard library and that `isMultiple` was preferable as it was more general. `isEven/isOdd` are also trivial inverses of each other which Swift, as a rule, doesn't include in the standard library. + +On the other hand there was some unscientific analysis that indicated that even/oddness accounted for 60-80% of the operations in which the result of the remainder operator was compared against zero. This lent some support to including `isEven/isOdd` over `isMultiple`. There is also more precedence in other languages for including `isEven/isOdd` over `isMultiple`. + +The authors decided that both were worthy of including in the proposal. Odd and even numbers have had special [shorthand labels](http://mathforum.org/library/drmath/view/65413.html) for thousands of years and are used frequently enough to justify the small additional weight `isEven/isOdd` would add to the standard library. `isMultiple` will greatly improve clarity and readability for arbitrary divisibility checks and also avoid potentially surprising `%` operator semantics with negative values. + +## Implementation Notes + +Only `isMultiple(of:)` was approved during review, so the final implementation does not include `isEven` or `isOdd`. Two default implementations are provided in the standard library; one on `BinaryInteger` and one on `FixedWidthInteger & SignedInteger`. For concrete signed and unsigned fixed-size integers, like the standard library types, these two implementations should be nearly optimal. + +For some user-defined types, especially bignum types, you may want to implement your own conformance for this function. Specifically, if your type does not have bounded min and max values, you should be able to do the divisibility check directly on the values rather than on the magnitudes, which may be more efficient. + +## Appendix + +### Other example uses in code (and beyond) + +* `% 2 == 0` appears 63 times in the [Apple/Swift](https://github.com/apple/swift) repository. +* [Initializing a cryptographic cipher](https://github.com/krzyzanowskim/CryptoSwift/blob/31efdd85ceb4190ee358a0516c6e82d8fd7b9377/Sources/CryptoSwift/Rabbit.swift#L89) +* Colouring a checker/chess board and [determining piece colour](https://github.com/nvzqz/Sage/blob/dec5edd97ba45d46bf94bc2e264d3ce2be6404ad/Sources/Piece.swift#L276) +* Multiple occurrences in cpp Boost library. [Example: handleResizingVertex45](https://www.boost.org/doc/libs/1_62_0/boost/polygon/polygon_45_set_data.hpp) +* Alternating bar colours in a chart +* [VoronoiFilter implementation](https://github.com/BradLarson/GPUImage/blob/167b0389bc6e9dc4bb0121550f91d8d5d6412c53/framework/Source/GPUImageJFAVoronoiFilter.m#L428) and [PoissonBlendFilter](https://github.com/BradLarson/GPUImage/blob/167b0389bc6e9dc4bb0121550f91d8d5d6412c53/framework/Source/GPUImagePoissonBlendFilter.m#L142) +* [Image blurring library](https://github.com/nicklockwood/FXBlurView/blob/9530adfc62fa682d0d8b3f612d3bb3af7a60ab7e/FXBlurView/FXBlurView.m#L58) +* UPC check digit calculation [(spec)](http://www.gs1.org/how-calculate-check-digit-manually) (values in odd digit indices are multiplied by 3) +* [Precondition check in sqlite function](https://github.com/sqlcipher/sqlcipher/blob/c6f709fca81c910ba133aaf6330c28e01ccfe5f8/src/crypto_impl.c#L1296) +* [Alternating UITableView cell background style](https://github.com/alloy/HockeySDK-CocoaPods/blob/978f3f072d206cfa35f4789d3a5b3abb31b9df11/Pods/HockeySDK/Classes/BITFeedbackListViewController.m#L502). NSTableView [has built in support](https://developer.apple.com/documentation/appkit/nstableview/1533967-usesalternatingrowbackgroundcolo?language=objc) for this. +* [Barcode reading](https://github.com/TheLevelUp/ZXingObjC/blob/d952cc02beb948ab49832661528c5e3e4953885e/ZXingObjC/oned/rss/expanded/ZXRSSExpandedReader.m#L449) +* [CSS row and column styling](https://www.w3.org/Style/Examples/007/evenodd.en.html) with `nth-child(even)` and `nth-child(odd)` (h/t @beccadax) +* Various test code + +Some _really_ real-world examples: +* [Printing even or odd pages only](https://i.stack.imgur.com/TE4Xn.png) +* New [Hearthstone even/odd cards](https://blizzardwatch.com/2018/03/16/hearthstones-new-even-odd-cards-going-shake-meta/) +* A variety of [even-odd rationing](https://en.m.wikipedia.org/wiki/Odd–even_rationing) rules: [parking rules](https://www.icgov.org/city-government/departments-and-divisions/transportation-services/parking/oddeven-parking), [water usage restrictions](https://vancouver.ca/home-property-development/understanding-watering-restrictions.aspx), [driving restrictions](https://auto.ndtv.com/news/odd-even-in-delhi-5-things-you-need-to-know-1773720) diff --git a/proposals/0226-package-manager-target-based-dep-resolution.md b/proposals/0226-package-manager-target-based-dep-resolution.md new file mode 100644 index 0000000000..503b1a71cd --- /dev/null +++ b/proposals/0226-package-manager-target-based-dep-resolution.md @@ -0,0 +1,260 @@ +# Package Manager Target Based Dependency Resolution + +* Proposal: [SE-0226](0226-package-manager-target-based-dep-resolution.md) +* Authors: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Boris Bügling](https://github.com/neonichu) +* Status: **Partially implemented (Swift 5.2):** Implemented the manifest API to disregard targets not concerned by any dependency products, which avoids building dependency test targets. +* Bug: [SR-8658](https://bugs.swift.org/browse/SR-8658) +* Previous Revision: [1](hhttps://github.com/swiftlang/swift-evolution/blob/e833c4d00bef253452b7d546e1303565ba584b58/proposals/0226-package-manager-target-based-dep-resolution.md) +* Review: [Review](https://forums.swift.org/t/se-0226-package-manager-target-based-dependency-resolution/), [Acceptance](https://forums.swift.org/t/accepted-se-0226-package-manager-target-based-dependency-resolution/15616), [Amendment](https://forums.swift.org/t/amendment-se-0226-package-manager-target-based-dependency-resolution/), [Amendment Acceptance](https://forums.swift.org/t/accepted-se-0226-amendment-package-manager-target-based-dependency-resolution/) + +## Introduction + +This is a proposal for enhancing the package resolution process to resolve +the minimal set of dependencies that are used in a package graph. + +## Motivation + +The current package resolution process resolves all declared dependencies in +the package graph. Some of the declared dependencies may not be required by the +products that are being used in the package graph. For e.g., a package may be +using some additional dependencies for its test targets. The packages that +depend on this package doesn't need to resolve such additional dependencies. These +dependencies increase the overall constraint in the dependency resolution process +that can otherwise be avoided. It can cause more cases of dependency hell if two +packages want to use incompatible versions of a dependency that they only use +for their unexported products. Cloning unnecessary dependencies also impacts the +performance of the resolution process. + +Another example of packages requiring additional dependencies is for sample code +targets. A library package may want to create an executable target which +demonstrates some functionality of the library. This executable may require +other dependencies such as a command-line argument parser. + +## Proposed solution + +We propose to enhance the dependency resolution process to resolve only the +dependencies that are actually being used in the package graph. The resolution +process can examine the target dependencies to figure out which package +dependencies require resolution. Since we will only resolve what is required, it +may reduce the odds of dependency hell situations. + +To achieve this, the package manager needs to associate the product dependencies +with the packages which provide those products without cloning them. We propose +to make the `package` parameter in the product dependency declaration +non-optional. This is necessary because if the package is not specified, +the package manager is forced to resolve all package dependencies just to figure +out which products are vended by which packages. + +```swift +extension Target.Dependency { + static func product(name: String, package: String) -> Target.Dependency +} +``` + +e.g. + +``` +.target( + name: "foo", + dependencies: [.product(name: "bar", package: "bar-package")] +), +``` + +SwiftPM will retain support for the simplified `byName` declaration for products +that are named after the package. +This provides a shorthand for the common case of small packages that vend just +one product, and the product and package names are aligned. + +```swift +extension Target.Dependency { + static func byName(_ name: String) -> Target.Dependency +} +``` + +e.g. + +``` +.target( + name: "foo", + dependencies: ["bar"] +), +``` + +To correlate between the package referenced in the target and the declared dependency, +SwiftPM will need to compute an identity for the declared dependencies without +cloning them. + +When this feature was first implemented in SwiftPM version 5.2, the package identity needed +to be specified using the `name` attribute on the package dependency declaration. +Such `name` attribute had to also align to the name declared in the dependency's manifest, +which led to adoption friction given that the manifest name is rarely known by +the users of the dependency. + +**5.2 version** + +```swift +extension Package.Dependency { + static func package( + name: String? = nil, // Proposed + url: String, + from version: Version + ) -> Package.Dependency + + static func package( + name: String? = nil, // Proposed + url: String, + _ requirement: Package.Dependency.Requirement + ) -> Package.Dependency + + static func package( + name: String? = nil, // Proposed + url: String, + _ range: Range + ) -> Package.Dependency + + static func package( + name: String? = nil, // Proposed + url: String, + _ range: ClosedRange + ) -> Package.Dependency +} +``` + +Starting with the SwiftPM version following 5.4 (exact number TBD), SwiftPM will actively discourage the use of the +`name` attribute on the package dependency declaration (will emit warning when used with tools-version >= TBD) +and instead will compute an identity for the declared dependency by using the last path +component of the dependency URL (or path in the case of local dependencies) in the dependencies section. +The dependency URL used for this purpose is as-typed (no percent encoding or other transformation), and regardless of any configured mirrors. +With this change, the name specified in the dependency manifest will have no bearing +over target based dependencies (other than for backwards compatibility). + +Note: [SE-0292] (when accepted and implemented) will further refine how package identities are computed. + + [SE-0292]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0292-package-registry-service.md + +**TBD version** + +```swift +extension Package.Dependency { + static func package( + name: String? = nil, // Usage will emit warning and eventually be deprecated + url: String, + from version: Version + ) -> Package.Dependency + + static func package( + name: String? = nil, // Usage will emit warning and eventually be deprecated + url: String, + _ requirement: Package.Dependency.Requirement + ) -> Package.Dependency + + static func package( + name: String? = nil, // Usage will emit warning and eventually be deprecated + url: String, + _ range: Range + ) -> Package.Dependency + + static func package( + name: String? = nil, // Usage will emit warning and eventually be deprecated + url: String, + _ range: ClosedRange + ) -> Package.Dependency +} +``` + +## Detailed design + +The resolution process will start by examining the products used in the target +dependencies and figure out the package dependencies that vend these products. +For each dependency, the resolver will only clone what is necessary to build +the products that are used in the dependees. + +The products declared in the target dependencies will need to provide their +package identity unless the package and product have the same name. SwiftPM will +diagnose the invalid product declarations and emit an error. + +As an example, consider the following package manifests: + +```swift +// IRC package +let package = Package( + name: "irc", + products: [ + .library(name: "irc", targets: ["irc"]), + .executable(name: "irc-sample", targets: ["irc-sample"]), + ], + dependencies: [ + .package(url: "https://github.com/apple/swift-nio.git", from: "1.0.0"), + + .package(url: "https://github.com/swift/ArgParse.git", from: "1.0.0"), + .package(url: "https://github.com/swift/TestUtilities.git", from: "1.0.0"), + ], + targets: [ + .target( + name: "irc", + dependencies: [.product(name: "NIO", package: "swift-nio"),] + ), + + .target( + name: "irc-sample", + dependencies: ["irc", "ArgParse"] + ), + + .testTarget( + name: "ircTests", + dependencies: [ + "irc", + .product(name: "Nimble", package: "TestUtilities"), + ] + ) + ] +) + +// IRC Client package +let package = Package( + name: "irc-client", + products: [ + .executable(name: "irc-client", targets: ["irc-client"]), + ], + dependencies: [ + .package(url: "https://github.com/swift/irc.git", from: "1.0.0"), + ], + targets: [ + .target(name: "irc-client", dependencies: ["irc"]), + ] +) +``` + +When the package "irc-client" is resolved, the package manager will only create +checkouts for the packages "irc" and "swift-nio" as "ArgParse" is used by +"irc-sample" but that product is not used in the "irc-client" package and +"Nimble" is used by the test target of the "irc" package. + +## Impact on existing packages + +There will be no impact on the existing packages. All changes, both behavioral +and API, will be guarded against the tools version this proposal is implemented +in. It is possible to form a package graph with mix-and-match of packages +with different tools versions. Packages with the older tools version will +resolve all package dependencies, while packages using the newer tools version +will only resolve the package dependencies that are required to build the used +products. + +As described in the proposal, the package manager will stop resolving the unused +dependencies. There will be no `Package.resolved` entries and checkouts for such +dependencies. + +Declaring target dependency on a product from an already resolved dependency +could potentially trigger the dependency resolution process, which in turn could +lead to cloning more repositories or even dependency hell. Note that such +dependency hell situations will always happen in the current implementation. + +## Alternatives considered + +We considered introducing a way to mark package dependencies as "development" or +"test-only". Adding these types of dependencies would have introduced new API +and new concepts, increasing package manifest complexity. It could also require +complicated rules, or new workflow options, dictating when these dependencies +would be resolved. Instead, we rejected adding new APIs as the above proposal +can handle these cases without any new API, and in a more intuitive manner. diff --git a/proposals/0227-identity-keypath.md b/proposals/0227-identity-keypath.md new file mode 100644 index 0000000000..1151ae35ab --- /dev/null +++ b/proposals/0227-identity-keypath.md @@ -0,0 +1,122 @@ +# Identity key path + +* Proposal: [SE-0227](0227-identity-keypath.md) +* Author: [Joe Groff](https://github.com/jckarter) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#18804](https://github.com/apple/swift/pull/18804), [apple/swift#19382](https://github.com/apple/swift/pull/19382) +* Review: [Discussion thread](https://forums.swift.org/t/se-0227-identity-key-path/15830), [Announcement thread](https://forums.swift.org/t/accepted-se-0227-identity-key-path/16278) + +## Introduction + +Add the ability to reference the identity key path, which refers to the entire +input value it is applied to. + +Swift-evolution thread: [Some small keypath extensions: identity and tuple components](https://forums.swift.org/t/some-small-keypath-extensions-identity-and-tuple-components/13729) + +## Motivation + +Key paths provide a means to refer to part of a value or a path through an +object graph independent of any specific instance. In most places where this +is useful, it is also useful to be able to refer to the entire value. +For instance, one could have a coordinator object that owns a state value and +notifies observers of changes to the state by allowing modification through +key paths: + +```swift +class ValueController { + private var state: T + + private var observers: [(T) -> ()] + + subscript(key: WritableKeyPath) { + get { return state[keyPath: key] } + set { + state[keyPath: key] = newValue + for observer in observers { + observer(state) + } + } + } +} +``` + +With such an interface, it'd be useful to be able to update the entire state +object at once. + +## Proposed solution + +We add a way to refer to the **identity key path**, which refers to the entire +input value a key path applies to. + +## Detailed design + +Every value in Swift has a special pseudo-property `.self`, which refers to +the entire value: + +```swift +var x = 1 +x.self = 2 +print(x.self) // prints 2 +``` + +By analogy, we could spell the identity key path `\.self`, since it notionally +refers to this `self` member: + +```swift +let id = \Int.self + +x[keyPath: id] = 3 +print(x[keyPath: id]) // prints 3 + +struct Employee { + var name: String + var position: String +} + +func updateValue(of vc: ValueController) { + vc[\.self] = Employee(name: "Cassius Green", position: "Power Caller") +} +``` + +The identity key path is a `WritableKeyPath`, since it can be used to +mutate a mutable value, but cannot mutate immutable references. It also +makes sense to give the identity key path special behavior with other +key path APIs: + +- Appending an identity key path produces a key path equal to the other + operand: + + ```swift + kp.appending(path: \.self) // == kp + (\.self).appending(path: kp) // == kp + ``` + +- Asking for the `offset(of:)` the identity key path produces `0`, since + reading and writing a `T` at offset zero from an `Unsafe(Mutable)Pointer` + is of course equivalent to reading the entire value: + + ``` + MemoryLayout.offset(of: \.self) // == 0 + ``` + +Also, for compatibility with Cocoa KVC, the identity key path maps to the +`@"self"` KVC key path. + +## Source compatibility + +This is an additive feature. + +## Effect on ABI stability + +The Swift standard library required some modifications to correctly handle +identity key paths. + +## Alternatives considered + +The biggest design question here is the syntax. Some other alternatives +include: + +- The special syntax `\.`, a key path with "no components". +- A static property on `KeyPath` and/or `WritableKeyPath`. + diff --git a/proposals/0228-fix-expressiblebystringinterpolation.md b/proposals/0228-fix-expressiblebystringinterpolation.md new file mode 100644 index 0000000000..701036698b --- /dev/null +++ b/proposals/0228-fix-expressiblebystringinterpolation.md @@ -0,0 +1,517 @@ +# Fix `ExpressibleByStringInterpolation` + +* Proposal: [SE-0228](0228-fix-expressiblebystringinterpolation.md) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax), [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.0)** +* Review: [Discussion thread](https://forums.swift.org/t/se-0228-fix-expressible-by-string-interpolation/16031), [Announcement thread](https://forums.swift.org/t/accepted-se-0228-fix-expressible-by-string-interpolation/16548) +* Implementation: [apple/swift#20214](https://github.com/apple/swift/pull/20214) + +## Introduction + +String interpolation is a simple and powerful feature for expressing complex, runtime-created strings, but the current version of the `ExpressibleByStringInterpolation` protocol has been deprecated since Swift 3. We propose a new design that improves its performance, clarity, and efficiency. + +Swift-evolution thread: [\[Draft\] Fix ExpressibleByStringInterpolation](https://forums.swift.org/t/draft-fix-expressiblebystringinterpolation/5403), [String interpolation revamp](https://forums.swift.org/t/string-interpolation-revamp/9302), [String interpolation revamp: design decisions](https://forums.swift.org/t/string-interpolation-revamp-design-decisions/12624) + +## Motivation + +### Background + +An interpolated string literal contains one or more embedded expressions, delimited by `\(` and `)`. At runtime, these expressions are evaluated and concatenated with the string literal to produce a value. They are typically more readable than code that switches between string literals, concatenation operators, and arbitrary expressions. + +Like most literal features in Swift, interpolated string literals are implemented with a protocol, `ExpressibleByStringInterpolation`. However, this protocol has been known to have issues since Swift 3, so it is currently deprecated. + +### Desired use cases + +We see three general classes of types that might want to conform to `ExpressibleByStringInterpolation`: + +1. **Simple textual data**: Types that represent simple, unconstrained text, like `Swift.String` itself. String types from foreign languages (like `JavaScriptCore.JSValue`) and alternative representations of strings (like a hypothetical `ASCIIString` type) might also want to participate in string interpolation. + +2. **Structured textual data**: Types that represent text but have some additional semantics. For example, a `Foundation.AttributedString` type might allow you to interpolate dictionaries to set or clear attributes. [This gist’s `LocalizableString`][loc] type creates a format string, which can be looked up in a `Foundation.Bundle`’s localization tables. + +3. **Machine-readable code fragments**: Types that represent data in a format that will be understood by a machine, like [`SQLKit.SQLStatement`][sql] or [this blog post’s `SanitizedHTML`][html]. These types often require data included in them to be escaped or passed out-of-band; a good `ExpressibleByStringInterpolation` design might allow this to be done automatically without the programmer having to do anything explicit. They may only support specific types, or may want to escape by default but also have a way to insert unescaped data. + +The current design handles simple textual data, but struggles to support structured textual data and machine-readable code fragments. + + [sql]: https://github.com/beccadax/SQLKit/blob/master/Sources/SQLKit/SQLStatement.swift + [html]: https://oleb.net/blog/2017/01/fun-with-string-interpolation/ + [loc]: https://gist.github.com/beccadax/79fa038c0af0cafb52dd + +### Current design + +The compiler parses a string literal into a series of *segments*, each of which is either a *literal segment* containing characters and escapes, or an *interpolated segment* containing an expression to be interpolated. If there is more than one segment, it wraps each segment in a call to `init(stringInterpolationSegment:)`, then wraps all of the segments together in a call to `init(stringInterpolation:)`: + +```swift +// Semantic expression for: "hello \(name)!" +String(stringInterpolation: + String(stringInterpolationSegment: "hello "), + String(stringInterpolationSegment: name), + String(stringInterpolationSegment: "!")) +``` + +The type checker considers all overloads of `init(stringInterpolationSegment:)`, not just the one that implements the protocol requirement. `Swift.String` uses this to add fast paths for types conforming to `CustomStringConvertible` and `TextOutputStreamable`. + +### Issues with the current design + +The current design is inefficient and inflexible for conformers. It does not permit special handling such as formatting or interpolation options. + +#### Inefficient: Naïve memory management + +Each `init(stringInterpolationSegment:)` call creates a temporary instance of `Self`; these instance are then concatenated together. Depending on the conformer or the segment size, this may trigger a heap allocation and ARC overhead for every single interpolated segment. + +Furthermore, while the compiler knows the sizes and numbers of literal and interpolated segments, this is not communicated to the conformer. + +If size information were available to the conformer, they could estimate the final size of the value and preallocate capacity. If segments were not converted to `Self` before concatenation, their data could be directly written to this preallocated capacity without using temporary instances. + +#### Inflexible: No extra parameters, unconstrained segments, lost segment semantics + +The current approach does not permit conformers to specify additional parameters or options to govern the evaluation of an interpolated expression. Many conformers may want to provide alternative interpolation behaviors, such as disabling escaping in `SanitizedHTML`. Others may want to accept options, like controlling the format string used in a `LocalizableString`. `String` itself would like to support a format argument eventually. + +`init(stringInterpolationSegment:)` takes an unconstrained generic value, so its parameter can be of any type. However, some conformers may want to limit the types that can be interpolated. For example, `SQLKit.SQLStatement` can only bind certain types, like integers and strings, to a SQL statement’s parameters. + +This unconstrained generic parameter causes a second problem: when a literal is passed to `init(stringInterpolationSegment:)`, it defaults to forming a `String`, the default literal type. This deviates from the standard library’s common practice of allowing the conformer to supply a literal type for use. + +Finally, the conformer cannot easily determine whether an incoming segment was from a literal or an expression without resorting to hacks baking in compiler-internal details. + +
Compiler-internal details + +##### Baking in assumptions + +An `init(stringInterpolationSegment:)` implementation cannot determine whether its parameter is a literal segment or an interpolated segment. However, the `init(stringInterpolation:)` call can exploit a compiler quirk to do so: the parser always generates a literal segment first, and always alternates between literal and interpolated segments (generating empty literal segments if necessary), so the position of a segment can tell you whether it is literal or interpolated. + +Needless to say, this is the sort of obscure implementation detail we don’t want users to depend upon. And preserving enough data for `init(stringInterpolation:)` to treat a segment as either type often requires conformers to add extra properties or otherwise alter the type's design purely to support string interpolation. + +##### Type-checker hacks + +If semantic analysis simply generated the semantic expression and then type-checked it normally, many string interpolations would be too complex to type-check. Instead, it type-checks each segment separately, then creates the `init(stringInterpolationSegment:)` call for the segment and type-checks just the one call to resolve its overload. + +String interpolation is the only remaining client of this type-checker entry point; we want to get rid of it. + +
+ +### Potential uses + +An improved string interpolation design could open many doors for future functionality in the standard library, in framework overlays, and in user code. To illustrate, here are some things we could use it for in code shipped with the Swift compiler. We're not proposing any of this, and any future proposal might look different—we're just demonstrating what's possible. + +#### Constructing formatted strings + +There are a number of approaches we could take to formatting values interpolated into strings. Here are a few examples with numbers: + +```swift +// Use printf-style format strings: +"The price is $\(cost, format: "%.2f")" + +// Use UTS #35 number formats: +"The price is \(cost, format: "¤###,##0.00")" + +// Use Foundation.NumberFormatter, or a new type-safe native formatter: +"The price is \(cost, format: moneyFormatter)" + +// Mimic String.init(_:radix:uppercase:) +"The checksum is 0x\(checksum, radix: 16)" +``` + +You could imagine analogous formatting tools for other types, like `Data`, `Date`, or even just `String` itself. + +#### Logging + +Some logging facilities restrict the kinds of data that can be logged or require extra metadata on certain values; a more powerful interpolation feature could support that: + +```swift +log("Processing \(public: tagName) tag containing \(private: contents)") +``` + +#### Constructing attributed strings + +`NSAttributedString` or a value-type wrapper around it could allow users to interpolate dictionaries of attributes to enable and disable them: + +```swift +"\([.link: supportURL])Click here\([.link: nil]) to visit our support site" +``` + +#### Localization + +A `LocalizableString` type could be expressed by a string literal, which would be used to generate a format string key and a list of arguments; converting a `LocalizableString` to an ordinary `String` would look up the key in a `Bundle`'s localization table, then format the value with the arguments. + +```swift +// Builds a LocalizableString(key: "The document “%@” could not be saved.", arguments: [name]) +let message: LocalizableString = "The document “\(name)” could not be saved." +alert.messageText = String(localized: message) +``` + +## Proposed solution + +We propose completely reworking the currently-deprecated `ExpressibleByStringInterpolation` as follows (doc comments omitted for brevity): + +```swift +public protocol ExpressibleByStringInterpolation + : ExpressibleByStringLiteral { + + associatedtype StringInterpolation : StringInterpolationProtocol + = String.StringInterpolation + where StringInterpolation.StringLiteralType == StringLiteralType + + init(stringInterpolation: StringInterpolation) +} + +public protocol StringInterpolationProtocol { + associatedtype StringLiteralType : _ExpressibleByBuiltinStringLiteral + + init(literalCapacity: Int, interpolationCount: Int) + + mutating func appendLiteral(_ literal: StringLiteralType) + + // Informal requirement: mutating func appendInterpolation(...) +} +``` + +An interpolated string will be converted into code that: + +1. Initializes an instance of an associated `StringInterpolation` type, passing the total literal segment size and interpolation count as parameters. + +2. Calls its `appendLiteral(_:)` method to append literal values, and `appendInterpolation` to append its interpolated values, one at a time. Interpolations are treated as call parentheses—that is, `\(x, with: y)` becomes a call to `appendInterpolation(x, with: y)`. + +3. Passes the instance to `init(stringInterpolation:)` to produce a final value. + +Below is code roughly similar to what the compiler would generate: + +```swift +// Semantic expression for: "hello \(name)!" +String(stringInterpolation: { + var temp = String.StringInterpolation(literalCapacity: 7, interpolationCount: 1) + temp.appendLiteral("hello ") + temp.appendInterpolation(name) + temp.appendLiteral("!") + return temp +}()) +``` + +[We have written a few examples of conforming types.][examples] + + [examples]: https://gist.github.com/beccadax/0b46ce25b7da1049e61b4669352094b6 + +## Detailed design + +This design has been implemented in [apple/swift#18590](https://github.com/apple/swift/pull/18590). + +### The `StringInterpolation` type + +The associated `StringInterpolation` type is a sort of buffer or scratchpad where the value of an interpolated string literal is accumulated. By having it be an associated type, rather than `Self` as it currently is, we realize a few benefits: + +1. A new type can serve as a namespace for the various `appendLiteral` and `appendInterpolation` methods. This allows conformers to add new interpolation methods without them showing up in code completion, documentation, etc. + +2. A separate type can store extra temporary state involved in the formation of the result. For example, `Foundation.AttributedString` might need to track the current attributes in a property; a type backed by a parsed data structure, like a `LambdaCalculusExp` or `Regexp` type, could store an unparsed string or parser state. When a type does *not* need any extra state, the associated type does not add any overhead. + +3. Several different types can share an implementation. For instance, `String` and `Substring` both use a common `StringInterpolationProtocol`-conforming type. + +The standard library will provide a `DefaultStringInterpolation` type; `StringProtocol`, and therefore `String` and `Substring`, will use this type for their interpolation. (`Substring` did not previously permit interpolation.) + +The standard library will also provide two sets of default implementations: + +* For types using `DefaultStringInterpolation`, it will provide a default `init(stringInterpolation:)` that extracts the value after interpolation and forwards it to `init(stringLiteral:)`. Thus, types that currently conform to `ExpressibleByStringLiteral` and use `String` as their literal type can add simple interpolation support by merely changing their conformance to `ExpressibleByStringInterpolation`. + +* For other types, it will provide a default `init(stringLiteral:)` that constructs a `Self.StringInterpolation` instance, calls its `appendLiteral(_:)` method, and forwards it to `init(stringInterpolation:)`. (An unavailable or deprecated `init(stringLiteral:)` will ensure that this is never used with the `init(stringInterpolation:)` provided for `DefaultStringInterpolation`-using types, which would cause infinite recursion.) + +### The `appendInterpolation` method(s) + +`StringInterpolation` types must conform to a `StringInterpolationProtocol`, which requires the `init(literalCapacity:interpolationCount:)` and `appendLiteral(_:)` methods. + +Non-literal segments are restricted at compile time to the overloads of `appendInterpolation` supplied by the conformer. This allows conforming types to restrict the values that can be interpolated into them by implementing only methods that accept the types they want to support. `appendInterpolation` can be overloaded to support several unrelated types. + +`appendInterpolation` methods can specify any parameter signature they wish. An `appendInterpolation` method can accept multiple parameters (with or without default values), can require a label on any parameter (including the first one), and can have variadic parameters. `appendInterpolation` methods can also throw; if one does, the string literal must be covered by a `try`, `try?`, or `try!` keyword. Future work includes enhancing String to accept formatting control. + +While this part of the design gives us great flexibility, it does introduce an implicit relationship between the compiler and ad-hoc methods declared by the conformer. It also restricts what values can be interpolated in a context generic over `StringInterpolationProtocol`, though further constraints can lift this restriction. + +Even though there is no formal requirement listed in the protocol, we have modified the compiler to emit an error if a `StringInterpolationProtocol`-conforming type does not have at least one overload of `appendInterpolation` that is as public as the type, does not return a value (or returns a discardable value), and is not static. + +### Interpolation parsing changes + +Interpolations will be parsed as argument lists; labels and multiple parameters will be permitted, but trailing closures will not. + +This change is slightly source-breaking: a 4.2 interpolation like `\(x, y)`, which tries to interpolate a tuple, would need to be written `\((x, y))`. While we could address un-labeled tuples with n-arity overloads of `appendInterpolation`, labeled tuples would still break. We emulate the current behavior in Swift 4.2 mode, and we can easily correct it during migration to Swift 5. + +### Ancillary changes + +We will add `ExpressibleByStringInterpolation` conformance to `StringProtocol`, and thus to `Susbtring`, allowing interpolations in string literals used to create `Substring`s. + +We will add `TextOutputStreamable` conformances to `Float`, `Double`, and `Float80`, along with an underscored, defaulted method for writing raw ASCII buffers to `TextOutputStream`s. These changes together reduce a regression in `Float` interpolation benchmarks and completely reverse regressions in `Double` and `Float80` interpolation benchmarks. + +### Implementation details + +
The `DefaultStringInterpolation` type + +The standard library uses `make()` to extract the final value; `CustomStringConvertible` is provided as a public equivalent for types that want to use `DefaultStringInterpolation` but do some processing in their `init(stringInterpolation:)` implementation. + +```swift +/// Represents a string literal with interpolations while it is being built up. +/// +/// Do not create an instance of this type directly. It is used by the compiler +/// when you create a string using string interpolation. Instead, use string +/// interpolation to create a new string by including values, literals, +/// variables, or expressions enclosed in parentheses, prefixed by a +/// backslash (`\(`...`)`). +/// +/// let price = 2 +/// let number = 3 +/// let message = "If one cookie costs \(price) dollars, " + +/// "\(number) cookies cost \(price * number) dollars." +/// print(message) +/// // Prints "If one cookie costs 2 dollars, 3 cookies cost 6 dollars." +/// +/// When implementing an `ExpressibleByStringInterpolation` conformance, +/// set the `StringInterpolation` associated type to `DefaultStringInterpolation` +/// to get the same interpolation behavior as Swift's built-in `String` type and +/// construct a `String` with the results. If you don't want the default behavior +/// or don't want to construct a `String`, use a custom type conforming to +/// `StringInterpolationProtocol` instead. +/// +/// Extending default string interpolation behavior +/// =============================================== +/// +/// Code outside the standard library can extend string interpolation on +/// `String` and many other common types by extending +/// `DefaultStringInterpolation` and adding an `appendInterpolation(...)` +/// method. For example: +/// +/// extension DefaultStringInterpolation { +/// fileprivate mutating func appendInterpolation( +/// escaped value: String, asASCII forceASCII: Bool = false) { +/// for char in value.unicodeScalars { +/// appendInterpolation(char.escaped(asASCII: forceASCII)) +/// } +/// } +/// } +/// +/// print("Escaped string: \(escaped: string)") +/// +/// See `StringInterpolationProtocol` for details on `appendInterpolation` +/// methods. +/// +/// `DefaultStringInterpolation` extensions should add only `mutating` members +/// and should not copy `self` or capture it in an escaping closure. +@_fixed_layout +public struct DefaultStringInterpolation: StringInterpolationProtocol { + /// The string contents accumulated by this instance. + @usableFromInline + internal var _storage: String = "" + + /// Creates a string interpolation with storage pre-sized for a literal + /// with the indicated attributes. + /// + /// Do not call this initializer directly. It is used by the compiler when + /// interpreting string interpolations. + @inlinable + public init(literalCapacity: Int, interpolationCount: Int) { + let capacityPerInterpolation = 2 + let initialCapacity = literalCapacity + interpolationCount * capacityPerInterpolation + _storage.reserveCapacity(initialCapacity) + } + + /// Appends a literal segment of a string interpolation. + /// + /// Do not call this method directly. It is used by the compiler when + /// interpreting string interpolations. + @inlinable + public mutating func appendLiteral(_ literal: String) { + _storage += literal + } + + /// Interpolates the given value's textual representation into the + /// string literal being created. + /// + /// Do not call this method directly. It is used by the compiler when + /// interpreting string interpolations. Instead, use string + /// interpolation to create a new string by including values, literals, + /// variables, or expressions enclosed in parentheses, prefixed by a + /// backslash (`\(`...`)`). + /// + /// let price = 2 + /// let number = 3 + /// let message = "If one cookie costs \(price) dollars, " + + /// "\(number) cookies cost \(price * number) dollars." + /// print(message) + /// // Prints "If one cookie costs 2 dollars, 3 cookies cost 6 dollars." + @inlinable + public mutating func appendInterpolation(_ value: T) { + value.write(to: &_storage) + } + + /// Interpolates the given value's textual representation into the + /// string literal being created. + /// + /// Do not call this method directly. It is used by the compiler when + /// interpreting string interpolations. Instead, use string + /// interpolation to create a new string by including values, literals, + /// variables, or expressions enclosed in parentheses, prefixed by a + /// backslash (`\(`...`)`). + /// + /// let price = 2 + /// let number = 3 + /// let message = "If one cookie costs \(price) dollars, " + + /// "\(number) cookies cost \(price * number) dollars." + /// print(message) + /// // Prints "If one cookie costs 2 dollars, 3 cookies cost 6 dollars." + @inlinable + public mutating func appendInterpolation(_ value: T) { + value.write(to: &_storage) + } + + /// Interpolates the given value's textual representation into the + /// string literal being created. + /// + /// Do not call this method directly. It is used by the compiler when + /// interpreting string interpolations. Instead, use string + /// interpolation to create a new string by including values, literals, + /// variables, or expressions enclosed in parentheses, prefixed by a + /// backslash (`\(`...`)`). + /// + /// let price = 2 + /// let number = 3 + /// let message = "If one cookie costs \(price) dollars, " + + /// "\(number) cookies cost \(price * number) dollars." + /// print(message) + /// // Prints "If one cookie costs 2 dollars, 3 cookies cost 6 dollars." + @inlinable + public mutating func appendInterpolation(_ value: T) { + _storage += value.description + } + + /// Interpolates the given value's textual representation into the + /// string literal being created. + /// + /// Do not call this method directly. It is used by the compiler when + /// interpreting string interpolations. Instead, use string + /// interpolation to create a new string by including values, literals, + /// variables, or expressions enclosed in parentheses, prefixed by a + /// backslash (`\(`...`)`). + /// + /// let price = 2 + /// let number = 3 + /// let message = "If one cookie costs \(price) dollars, " + + /// "\(number) cookies cost \(price * number) dollars." + /// print(message) + /// // Prints "If one cookie costs 2 dollars, 3 cookies cost 6 dollars." + @inlinable + public mutating func appendInterpolation(_ value: T) { + _print_unlocked(value, &_storage) + } + + /// Creates a string from this instance, consuming the instance in the process. + @inlinable + internal __consuming func make() -> String { + return _storage + } +} + +extension DefaultStringInterpolation: CustomStringConvertible { + @inlinable + public var description: String { + return _storage + } +} +``` + +
+ +
Generating the append calls + +This design puts every `appendLiteral(_:)` and `appendInterpolation` call in its own statement, so there’s no need for special type checker treatment. Each interpolation will naturally be type-checked separately, and the overloads of `appendInterpolation` will be resolved at the same time as the value being interpolated. This helps us with ongoing refactoring of the type checker. + +Due to issues with capturing of partially initialized variables, we do not enclose these statements in a closure. Instead, we use a new kind of AST node. + +
+ +
Performance + +While some string interpolation benchmarks show regressions of 20–30%, most show improvements, sometimes dramatic ones. + +| Benchmark | -O speed improvement | -Osize speed improvement | +| -------------------------------------------- | -------------------- | ------------------------ | +| `StringInterpolationManySmallSegments` | 2.15x | 1.80x | +| `StringInterpolationSmall` | 2.01x | 2.03x | +| `ArrayAppendStrings` | 1.16x | 1.14x | +| `FloatingPointPrinting_Double_interpolated` | 1.15x | 1.16x | +| `FloatingPointPrinting_Float80_interpolated` | 1.09x | 1.08x | +| `StringInterpolation` | 0.82x | 0.79x | +| `FloatingPointPrinting_Float_interpolated` | 0.82x | 0.73x | + +The `StringInterpolation` benchmark's regression is caused by the specific sizes of literal and interpolated segment sizes; in the new design, these happen to cause the benchmark to grow its buffer an extra time. We don't think it's representative of the design's performance. + +Initially, all three `FloatingPointPrinting__interpolated` tests regressed with the new design. We conformed these types to `TextOutputStreamable` and added a private ASCII-only fast path in `TextOutputStream`; this increased the performance of `Double` and `Float80` to be small improvements, but did little to help `Float`. + +Benchmark code size slightly improved on average: + +| Benchmark file | -O size improvement | -Osize size improvement | +| ------------------------------ | ------------------- | ----------------------- | +| StringInterpolation.o | 1.18x | 1.16x | +| FloatingPointPrinting.o | 1.12x | 1.11x | +| All files with notable changes | 1.02x | 1.02x | + +So did Swift library code size: + +| Library | Size improvement | +| ------------------------------------ | ---------------- | +| libswiftSwiftPrivateLibcExtras.dylib | 1.20x | +| libswiftFoundation.dylib | 1.15x | +| libswiftXCTest.dylib | 1.10x | +| libswiftStdlibUnittest.dylib | 1.06x | +| libswiftCore.dylib. | 1.04x | +| libswiftNetwork.dylib | 1.02x | +| libswiftSwiftOnoneSupport.dylib | 1.02x | +| libswiftsimd.dylib | 1.01x | +| libswiftMetal.dylib | 0.90x | +| libswiftSwiftReflectionTest.dylib | 0.92x | + +We believe the current results already look pretty good, and further performance tuning is possible in the future. Other types can likely improve interpolation performance using `TextOutputStreamable`. Overall, this design has nowhere to go but up. + +The default `init(stringLiteral:)` (which is only used for types implementing fully custom string interpolation) is currently about 0.5x the speed of a manually-implemented `init(stringLiteral:)`, but prototyping indicates that inlining certain fast paths from `String.reserveCapacity(_:)` and `String.append(_:)` can reduce that penalty to 0.93x, and we may be able to squeeze out gains beyond that. Even if we cannot close this gap completely, performance-sensitive types can always implement `init(stringLiteral:)` manually. + +
+ +## Source compatibility + +Since `ExpressibleByStringInterpolation` has been deprecated since Swift 3, we need not maintain source compatibility with existing conformances, nor do we propose preserving existing conformances to `ExpressibleByStringInterpolation` even in Swift 4 mode. + +We do not propose preserving existing `init(stringInterpolation:)` or `init(stringInterpolationSegment:)` initializers, since they have always been documented as calls that should not be used directly. However, the source compatibility suite contains code that accidentally uses `init(stringInterpolationSegment:)` by writing `String.init` in a context expecting a `CustomStringConvertible` or `TextOutputStreamable` type. We have devised a set of overloads to `init(describing:)` that will match these accidental, implicit uses of `init(stringInterpolationSegment:)` without preserving explicit uses of `init(stringInterpolationSegment:)`. + +We propose a set of `String.StringInterpolation.appendInterpolation` overloads that exactly match the current `init(stringInterpolationSegment:)` overloads, so “normal” interpolations will work exactly as before. + +“Strange” interpolations like `\(x, y)` or `\(foo: x)`, which are currently accepted by the Swift compiler will be errors in Swift 5 mode. In Swift 4.2 mode, we will preserve the existing behavior with a warning; this means that Swift 4.2 code will only be able to use `appendInterpolation` overloads with a single unlabeled parameter, unless all other parameters have default values. Migration involves inserting an extra pair of parens or removing an argument label to preserve behavior. + +## Effect on ABI stability + +`ExpressibleByStringInterpolation` will need to be ABI-stable starting in Swift 5; we should adopt this proposal or some alternative and un-deprecate `ExpressibleByStringInterpolation` before that. + +## Effect on API resilience + +This API is pretty foundational and it would be difficult to change compatibly in the future. + +## Alternatives considered + +### Variadic-based designs + +We considered several designs that, like the current design, passed segments to a variadic parameter. For example, we could wrap literal segments in `init(stringLiteral:)` instead of `init(stringInterpolationSegment:)` and otherwise keep the existing design: + +```swift +String(stringInterpolation: + String(stringLiteral: "hello "), + String(stringInterpolationSegment: name), + String(stringLiteral: "!")) +``` + +Or we could use an enum to differentiate literal segments from interpolated ones: + +```swift +String(stringInterpolation: + .literal("hello "), + .interpolation(String.StringInterpolationType(name)), + .literal("!")) +``` + +However, this requires that conformers expose a homogeneous return value, which has expressibility and/or efficiency drawbacks. The proposed approach, which is statement based, keeps this as a detail internal to the conformer. + +### Have a formal `appendInterpolation(_:)` requirement + +We considered having a formal `appendInterpolation(_:)` requirement with an unconstrained generic parameter to mimic current behavior. We could even have a default implementation that vends strings and still honors overloading. + +However, we would have to give up on conformers being able to restrict the types or interpolation segment forms permitted. diff --git a/proposals/0229-simd.md b/proposals/0229-simd.md new file mode 100644 index 0000000000..9be0cc1afe --- /dev/null +++ b/proposals/0229-simd.md @@ -0,0 +1,646 @@ +# SIMD Vectors + +* Proposal: [SE-0229](0229-simd.md) +* Author: [Stephen Canon](https://github.com/stephentyrone) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#20344](https://github.com/apple/swift/pull/20344) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0229-simd/18066) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/7e4f09fbe11e412326fe19aa5a6121efecc15b3e/proposals/0229-simd.md), + [2](https://github.com/swiftlang/swift-evolution/blob/fb85e6772d181800231e71438c7240507d2f04d9/proposals/0229-simd.md) + +## Introduction + +This proposal would expose a common subset of operations on the SIMD types supported +by most processors in the standard library. It is based on Apple's module, +which is used throughout Apple's platforms as the common currency type for fixed-size +vectors and matrices. It is not a complete re-implementation; rather it provides the low-level +support needed to import any such library, and tries to make a number of things much nicer +in Swift than they are in C or C++. + +Preliminary Swift-evolution [discussion](https://forums.swift.org/t/simd-vector-types/16254/). + +## Motivation + +### Task 1: SIMD programming +Essentially every modern CPU has support for [SIMD](https://en.wikipedia.org/wiki/SIMD) +("Single Instruction, Multiple Data") instructions in hardware. Without getting into a long +discussion of the architectural details, effective use of these instructions allows 2-10x better +performance than is otherwise possible for a large class of data-parallel problems, without +incurring the synchronization and data-movement hassles of working with the GPU. + +Historically, there have been a number of obstacles to taking advantage of this hardware. +Four programming models have been commonly used, all of which the author has considerable +experience with: +- Assembly: this has the advantage that you get exactly the code you want. It has numerous +disadvantages--another language to learn, requiring either macro soup or separate +implementations for every target (in effectively a different language for each), there are few +good learning resources, and you have to slog through all the tedious things that the compiler +normally does for you, like allocating registers and getting the calling conventions right (or +wrong, in subtle ways that bite your users many years later). +- Intrinsics: The model historically pushed by hardware vendors. Each architecture has its own +set of C "intrinsic" types like `__m128` (x86) or `int8x8_t` (ARM). These are superficially nicer +than assembly, but in practice incur nearly all of the downsides, plus a few additional ones. +The biggest problem is that these types are often bolted awkwardly onto the language, and +so are incompatible with fundamental language or runtime assumptions about size or +alignment. Innumerable bugs have been created by people attempting to use vector intrinsic +types together with C++ containers, for example. These types move your implementation into +a portable language (C or C++), and then immediately remove that portability by being tied to +a specific architecture. +- "Vector class" libraries: Agner's is the most well-known. What these generally bring to the +table is support for familiar operators rather than arcane intrinsics (e.g. `a + b` instead of +`_mm_addps(a, b)`). Most are narrowly focused on a single architecture (almost always x86, +so they still don't provide real portability). Apple's is similar to these, but with +full support for every architecture that Apple uses, which in practice makes code written +against it fairly portable. +- Autovectorization: works well for simple problems, but no one has yet demonstrated a +satisfactory solution for more general tasks. One of several problems is that the underlying +machine model assumed by vector code is fundamentally distinct from the underlying +machine model that most scalar code is written against, forcing an autovectorizing compiler +to map between the two. Historically this was largely done via an ad-hoc set of +transformations, which never really worked all that well. Newer approaches show some +promise, but explicit manual vectorization is still frequently needed. + +A major goal of these new data types in Swift is to provide a *better* API for vector +programming. We want to capture the cross-platform niceties of the module, +but also add some new features that were difficult or impossible to do in C; things like enabling +generic programming, but also things as simple as a native way to express vector permutations +and unaligned loads, or even just conversions between different vector types with the same +number of elements (this requires a function call rather than a cast with most vector libraries). + +Looking ahead, I expect that we will use these primitives to expose things like iterating over +vectors extracted from a collection of scalars, to make explicitly vectorized code much more +approachable. + +### Task 2: geometry primitives +There is a large class of computational tasks (graphics and animation, image processing, +AR, VR, computer vision) that want to have 2, 3, and 4 dimensional vector and matrix types. +For these applications, these types are just as fundamental as `Int` and `Array` are for +"normal" programming--they are the foundation upon which everything else is constructed. + +These tasks require both elementwise operations, as well as some operations on types as +abstract vectors--things like the dot and cross products, vector length, and orientation tests. + +### Task 3: GPU data structures +Closely related to part 2, short vectors are also essential data types for representing GPU data +structures. It's frequently necessary to handle these on the CPU side do do pre/post +processing, or data marshalling, and exposing these types in Swift enables that task. + +### Putting it together +Superficially, the only thing that these tasks have in common is that the fundamental types +are "homogeneous aggregates", and that they want to benefit from the SIMD hardware that is +available. Two or even three sets of types may seem more appropriate. However, our +experience with clients of is that all of our clients use a diverse subset of +operations from the module, and that it's difficult to draw clear boundaries of what belongs +where. While it may be reasonable to refine the underlying *protocols*, the types should +probably remain unified. + +Looking ahead, we would like to enable more sophisticated storage layouts and transforms +to and from them, such as SoA - AoS conversion (a fancy way of saying "matrix transpose" or +"interleave / deinterleave"; these are all the same thing). Once you start thinking about such +transforms, the distinction between these types really goes out the window, because you want +to map between things like 16 vectors of 3 floats and 3 vectors of 16 floats. + +## Proposed solution + +This proposal adds types like `SIMD2`, `SIMD16` etc. to the language, +with operations whose semantics map to SIMD types available on most architectures. The +full set of types are `SIMD2`, `SIMD3`, `SIMD4`, `SIMD8`, +`SIMD16`, `SIMD32`, and `SIMD64`, where `T` is any type that conforms +to the `SIMDScalar` protocol (discussed in detail below). All of the standard library +integer and floating-point types (except for `Float80`) conform to `SIMDScalar`. + +## Detailed design + +To support SIMD vectors, a type conforms to the `SIMDScalar` protocol: + +```swift +public protocol SIMDScalar { + associatedtype SIMDMaskScalar : SIMDScalar & FixedWidthInteger & SignedInteger + associatedtype SIMD2Storage : SIMDStorage where SIMD2Storage.Scalar == Self + associatedtype SIMD4Storage : SIMDStorage where SIMD4Storage.Scalar == Self + associatedtype SIMD8Storage : SIMDStorage where SIMD8Storage.Scalar == Self + associatedtype SIMD16Storage : SIMDStorage where SIMD16Storage.Scalar == Self + associatedtype SIMD32Storage : SIMDStorage where SIMD32Storage.Scalar == Self + associatedtype SIMD64Storage : SIMDStorage where SIMD64Storage.Scalar == Self +} +``` +Let's leave aside `SIMDMaskScalar` for now, and consider the other six `associatedtype` requirements. +They all have the same form; a type that provides storage for a vector of the designated size +with the correct scalar type. + +The `SIMDStorage` protocol is extremely simple: +```swift +public protocol SIMDStorage { + /// The type of scalars in the vector space. + associatedtype Scalar : Hashable + + /// The number of elements in the vector. + var scalarCount: Int { get } + + /// A vector with zero or the default value in all lanes. + init() + + /// Element access to the vector. + subscript(index: Int) -> Scalar { get set } +} +``` +This is the bare minimum functionality upon which everything else is built; the expectation is +that types conforming to this protocol will wrap LLVM `Builtin` vector types, but this is not +necessary. Users can conform types to this protocol with any backing storage that can provide +the necessary operations, which allows them to benefit from the semantics (and many +operations will be autovectorized by the compiler in most cases, even without explicit +vector types as storage). + +The `SIMD` protocol refines `SIMDStorage`, adding some additional +requirements: +```swift +public protocol SIMD : SIMDStorage, Hashable, CustomStringConvertible { + associatedtype MaskStorage : SIMD + where MaskStorage.Scalar : FixedWidthInteger & SignedInteger +} +``` +Let's discuss `Mask`s. `SIMD`s are `Equatable`, so they have the +`==` and `!=` operators, but they also provide the "pointwise comparison" `.==` and `.!=` +operators, which compare the lanes of two vectors, and produce a `Mask`, which is a vector +of boolean values. Each lane of the mask is either `true` or `false`, depending on the result +of comparing the values in the corresponding lanes. An example: +```swift +(swift) let x = SIMD4(1,2,3,4) +// x : SIMD4 = SIMD4(1, 2, 3, 4) +(swift) let y = SIMD4(3,2,1,0) +// y : SIMD4 = SIMD4(3, 2, 1, 0) +(swift) x .== y +// r0 : SIMDMask> = SIMDMask>(false, true, false, false) +``` +here, the second lane is `true`, because `2 == 2`, while all other lanes are false because +the elements of `x` and `y` in those lanes are not equal. + +Because of some technical details of how these comparisons happen in SIMD hardware on +CPUs, it's often advantageous to store these vectors-of-bools as vectors-of-ints whose width +matches that of the vectors being compared. Because of this, there isn't a single +`SIMD4` type, but rather a whole family of `SIMDMask>` types. The +`MaskStorage` associatedtype on `SIMDScalar` allows us to get to the desired +mask type from `SIMD4`. + +The concrete `SIMD2`, `SIMD3`, etc types conform to `SIMD`; and almost all +of their operations are provided by extensions on the protocol: +```swift +public extension SIMD { + /// The valid indices for subscripting the vector. + var indices: Range { get } + + /// A vector with value in all lanes. + init(repeating value: Scalar) + + /// Conformance to Equatable + static func ==(lhs: Self, rhs: Self) -> Bool + + /// Conformance to Hashable + func hash(into hasher: inout Hasher) + + /// Conformance to CustomStringConvertible + var description: String + + /// Pointwise equality and inequality + static func .==(lhs: Self, rhs: Self) -> SIMDMask + static func .!=(lhs: Self, rhs: Self) -> SIMDMask + static func .==(lhs: Scalar, rhs: Self) -> SIMDMask + static func .!=(lhs: Scalar, rhs: Self) -> SIMDMask + static func .==(lhs: Self, rhs: Scalar) -> SIMDMask + static func .!=(lhs: Self, rhs: Scalar) -> SIMDMask + + /// Replaces elements of this vector with `other` in the lanes where + /// `mask` is `true`. + mutating func replace(with other: Self, where mask: SIMDMask) + mutating func replace(with other: Scalar, where mask: SIMDMask) + func replacing(with other: Self, where mask: SIMDMask) -> Self + func replacing(with other: Scalar, where mask: SIMDMask) -> Self + + /// Initialize from array literal + /// + /// Precondition: the array must have exactly scalarCount elements. + init(arrayLiteral elements: Scalar...) + + /// Initialize from sequence + /// + /// Precondition: the sequence must have exactly scalarCount elements. + init(_ elements: S) where S.Element == Scalar +} + +public extension SIMD where Scalar : Comparable { + /// Pointwise ordered comparisons + static func .<(lhs: Self, rhs: Self) -> SIMDMask + static func .<=(lhs: Self, rhs: Self) -> SIMDMask + static func .>=(lhs: Self, rhs: Self) -> SIMDMask + static func .>(lhs: Self, rhs: Self) -> SIMDMask + + static func .<(lhs: Scalar, rhs: Self) -> SIMDMask + static func .<=(lhs: Scalar, rhs: Self) -> SIMDMask + static func .>=(lhs: Scalar, rhs: Self) -> SIMDMask + static func .>(lhs: Scalar, rhs: Self) -> SIMDMask + + static func .<(lhs: Self, rhs: Scalar) -> SIMDMask + static func .<=(lhs: Self, rhs: Scalar) -> SIMDMask + static func .>=(lhs: Self, rhs: Scalar) -> SIMDMask + static func .>(lhs: Self, rhs: Scalar) -> SIMDMask +} + +public extension SIMDMask { + /// Boolean operations on Masks; these use the .-prefixed form because + /// they do not short-circuit like `&&` and `||`, but the normal + /// boolean operators have the wrong precedence for the typical use + /// of these operators. + static prefix func .!(rhs: SIMDMask) -> SIMDMask + + static func .&(lhs: SIMDMask, rhs: SIMDMask) -> SIMDMask + static func .^(lhs: SIMDMask, rhs: SIMDMask) -> SIMDMask + static func .|(lhs: SIMDMask, rhs: SIMDMask) -> SIMDMask + + static func .&(lhs: Bool, rhs: SIMDMask) -> SIMDMask + static func .^(lhs: Bool, rhs: SIMDMask) -> SIMDMask + static func .|(lhs: Bool, rhs: SIMDMask) -> SIMDMask + + static func .&(lhs: SIMDMask, rhs: Bool) -> SIMDMask + static func .^(lhs: SIMDMask, rhs: Bool) -> SIMDMask + static func .|(lhs: SIMDMask, rhs: Bool) -> SIMDMask + + static func .&=(lhs: inout SIMDMask, rhs: SIMDMask) + static func .^=(lhs: inout SIMDMask, rhs: SIMDMask) + static func .|=(lhs: inout SIMDMask, rhs: SIMDMask) + + static func .&=(lhs: inout SIMDMask, rhs: Bool) + static func .^=(lhs: inout SIMDMask, rhs: Bool) + static func .|=(lhs: inout SIMDMask, rhs: Bool) + + static func random(using generator: inout T) -> SIMDMask + static func random() -> SIMDMask +} + +public extension SIMD where Scalar : FixedWidthInteger { + + static var zero: Self + var leadingZeroBitCount: Self + var trailingZeroBitCount: Self + var nonzeroBitCount: Self + + static prefix func ~(rhs: Self) -> Self + static func &(lhs: Self, rhs: Self) -> Self + static func ^(lhs: Self, rhs: Self) -> Self + static func |(lhs: Self, rhs: Self) -> Self + static func &<<(lhs: Self, rhs: Self) -> Self + static func &>>(lhs: Self, rhs: Self) -> Self + static func &+(lhs: Self, rhs: Self) -> Self + static func &-(lhs: Self, rhs: Self) -> Self + static func &*(lhs: Self, rhs: Self) -> Self + static func /(lhs: Self, rhs: Self) -> Self + static func %(lhs: Self, rhs: Self) -> Self + + static func &(lhs: Scalar, rhs: Self) -> Self + static func ^(lhs: Scalar, rhs: Self) -> Self + static func |(lhs: Scalar, rhs: Self) -> Self + static func &<<(lhs: Scalar, rhs: Self) -> Self + static func &>>(lhs: Scalar, rhs: Self) -> Self + static func &+(lhs: Scalar, rhs: Self) -> Self + static func &-(lhs: Scalar, rhs: Self) -> Self + static func &*(lhs: Scalar, rhs: Self) -> Self + static func /(lhs: Scalar, rhs: Self) -> Self + static func %(lhs: Scalar, rhs: Self) -> Self + + static func &(lhs: Self, rhs: Scalar) -> Self + static func ^(lhs: Self, rhs: Scalar) -> Self + static func |(lhs: Self, rhs: Scalar) -> Self + static func &<<(lhs: Self, rhs: Scalar) -> Self + static func &>>(lhs: Self, rhs: Scalar) -> Self + static func &+(lhs: Self, rhs: Scalar) -> Self + static func &-(lhs: Self, rhs: Scalar) -> Self + static func &*(lhs: Self, rhs: Scalar) -> Self + static func /(lhs: Self, rhs: Scalar) -> Self + static func %(lhs: Self, rhs: Scalar) -> Self + + static func &=(lhs: inout Self, rhs: Self) + static func ^=(lhs: inout Self, rhs: Self) + static func |=(lhs: inout Self, rhs: Self) + static func &<<=(lhs: inout Self, rhs: Self) + static func &>>=(lhs: inout Self, rhs: Self) + static func &+=(lhs: inout Self, rhs: Self) + static func &-=(lhs: inout Self, rhs: Self) + static func &*=(lhs: inout Self, rhs: Self) + static func /=(lhs: inout Self, rhs: Self) + static func %=(lhs: inout Self, rhs: Self) + + static func &=(lhs: inout Self, rhs: Scalar) + static func ^=(lhs: inout Self, rhs: Scalar) + static func |=(lhs: inout Self, rhs: Scalar) + static func &<<=(lhs: inout Self, rhs: Scalar) + static func &>>=(lhs: inout Self, rhs: Scalar) + static func &+=(lhs: inout Self, rhs: Scalar) + static func &-=(lhs: inout Self, rhs: Scalar) + static func &*=(lhs: inout Self, rhs: Scalar) + static func /=(lhs: inout Self, rhs: Scalar) + static func %=(lhs: inout Self, rhs: Scalar) + + static func random( + in range: Range, + using generator: inout T + ) -> Self + + static func random(in range: Range) -> Self + + static func random( + in range: ClosedRange, + using generator: inout T + ) -> Self + + static func random(in range: ClosedRange) -> Self +} + +public extension SIMD where Scalar : FloatingPoint { + + static var zero: Self + + static func +(lhs: Self, rhs: Self) -> Self + static func -(lhs: Self, rhs: Self) -> Self + static func *(lhs: Self, rhs: Self) -> Self + static func /(lhs: Self, rhs: Self) -> Self + + static func +(lhs: Scalar, rhs: Self) -> Self + static func -(lhs: Scalar, rhs: Self) -> Self + static func *(lhs: Scalar, rhs: Self) -> Self + static func /(lhs: Scalar, rhs: Self) -> Self + + static func +(lhs: Self, rhs: Scalar) -> Self + static func -(lhs: Self, rhs: Scalar) -> Self + static func *(lhs: Self, rhs: Scalar) -> Self + static func /(lhs: Self, rhs: Scalar) -> Self + + static func +=(lhs: inout Self, rhs: Self) + static func -=(lhs: inout Self, rhs: Self) + static func *=(lhs: inout Self, rhs: Self) + static func /=(lhs: inout Self, rhs: Self) + + static func +=(lhs: inout Self, rhs: Scalar) + static func -=(lhs: inout Self, rhs: Scalar) + static func *=(lhs: inout Self, rhs: Scalar) + static func /=(lhs: inout Self, rhs: Scalar) + + func addingProduct(_ lhs: Self, _ rhs: Self) -> Self + func addingProduct(_ lhs: Scalar, _ rhs: Self) -> Self + func addingProduct(_ lhs: Self, _ rhs: Scalar) -> Self + mutating func addProduct(_ lhs: Self, _ rhs: Self) + mutating func addProduct(_ lhs: Scalar, _ rhs: Self) + mutating func addProduct(_ lhs: Self, _ rhs: Scalar) + + func squareRoot( ) -> Self + mutating func formSquareRoot( ) + + func rounded(_ rule: FloatingPointRoundingRule) -> Self + mutating func round(_ rule: FloatingPointRoundingRule) +} + +public extension SIMD where Scalar : BinaryFloatingPoint, + Scalar.RawSignificand : FixedWidthInteger { + + static func random( + in range: Range, + using generator: inout T + ) -> Self + + static func random(in range: Range) -> Self + + static func random( + in range: ClosedRange, + using generator: inout T + ) -> Self + + static func random(in range: ClosedRange) -> Self +} +``` +As you can see, this is a fairly huge API. + +These operations are mostly implemented in terms of each other; the base operations are +implemented as for-loops over the elements of the vector. Most of these are already auto- +vectorized by the compiler when optimization is enabled, but this will not be the full long- +term solution. Instead, they will be annotated with `@_semantics` to allow them to be +lowered to special SIL nodes and from there to the corresponding LLVM IR vector nodes +so that we can guarantee vector codegen always occurs. This optimization work does not +effect the *semantics* of the operations at the Swift language level, so it will happen over the +next few months as bug fixes. + +After all that, the actual machinery of the concrete types is pretty boring: +```swift +public struct SIMD4 : SIMD + where Scalar : SIMDScalar { + + public var _storage: Scalar.SIMD4Storage + + public var scalarCount: Int { return 4 } + + public init() { _storage = Scalar.SIMD4Storage() } + + public subscript(index: Int) -> Scalar { + get { + _precondition(indices.contains(index)) + return _storage[index] + } + set { + _precondition(indices.contains(index)) + _storage[index] = newValue + } + } + + public init(_ v0: Scalar, _ v1: Scalar, _ v2: Scalar, _ v3: Scalar) { + self.init() + self[0] = v0 + self[1] = v1 + self[2] = v2 + self[3] = v3 + } + + public init(x: Scalar, y: Scalar, z: Scalar, w: Scalar) { + self.init(x, y, z, w) + } + + public var x: Scalar { + get { return self[0]} + set { self[0] = newValue } + } + + public var y: Scalar { + get { return self[1]} + set { self[1] = newValue } + } + + public var z: Scalar { + get { return self[2]} + set { self[2] = newValue } + } + + public var w: Scalar { + get { return self[3]} + set { self[3] = newValue } + } + + public init(lowHalf: SIMD2, highHalf: SIMD2) { + self.init() + self.lowHalf = lowHalf + self.highHalf = highHalf + } + + public var lowHalf: SIMD2 { + get { + var result = SIMD2() + for i in result.indices { result[i] = self[i] } + return result + } + set { + for i in newValue.indices { self[i] = newValue[i] } + } + } + + public var highHalf: SIMD2 { + get { + var result = SIMD2() + for i in result.indices { result[i] = self[2+i] } + return result + } + set { + for i in newValue.indices { self[2+i] = newValue[i] } + } + } + + public var evenHalf: SIMD2 { + get { + var result = SIMD2() + for i in result.indices { result[i] = self[2*i] } + return result + } + set { + for i in newValue.indices { self[2*i] = newValue[i] } + } + } + + public var oddHalf: SIMD2 { + get { + var result = SIMD2() + for i in result.indices { result[i] = self[2*i+1] } + return result + } + set { + for i in newValue.indices { self[2*i+1] = newValue[i] } + } + } +} +``` +Beyond the new Swift API for vector types, there's an accompanying change to the clang +importer that will allow C functions and structures using vector types to be imported. When +we find a "clang extended vector type" or "extvector" whose underlying scalar type is +imported as a `SIMDScalar` type in the standard library and whose element count is +2, 3, 4, 8, 16, 32, or 64, we'll import it as the corresponding standard library vector type. + +## Source compatibility + +No source compatibility changes. + +## Effect on ABI stability + +No effects on ABI stability at the language level. There are some minor changes around +how the types are imported on Apple platforms, but these will have no effect +on source compatibility; they only may tweak some low-level ABI details. + +## Effect on API resilience + +Because these are entirely new types, they come with a large set of API that will become part +of the standard library interface. New API can be added in the future, including to protocols, +because it is generally possible to provide good default implementations for simd operations. + +## Alternatives considered + +The main alternative is "don't do anything, let people write structs, and trust the autovectorizer." +This might even mostly work, but even if it worked flawlessly (which it doesn't today), you +would be left with the problem that these types are common, and everyone would be using +their own set of structs, with slightly incompatible size and alignment or operations provided. +Even when the layout matches up, you would still need to provide conversion shims between +each and every library you work with. There is a lot of merit in providing a single ground-truth +for low-level vectors in the stdlib, over which libraries and programs can build additional +operations. + +### Notes from pre-review discussion: + +1. I have added a static `.zero` property to integer and floating point vectors based on email +conversation with Rick Roe. This is a departure from scalar numeric types, but he convinced +me that these are good to have as an explicit alternative to `T()`. + +2. As mentioned above, `.`-prefixes have been added to some elementwise operations. I see +pretty good arguments both in favor of and against this change. I think that it largely comes +down to a matter of taste. There are two reasonable alternative positions here. (a) don't add the +prefixes; this makes type checking a little more complex. (b) add `.`-prefixes to *all* lanewise +operations; this adds a lot of new operators and a fair bit of noise, but there's reasonable +precedent for it as well (julialang). + +3. Richard Wei has argued strongly against the spelling of `replacing(with: where:)`, +objecting to both the lack of an explicit object of the verb `replacing` and to the +use of `where:` as a label for an argument that is not a predicate function. I am not ignoring +his concerns, but from a pragmatic point of view, I believe that the fact that this operates on +elements is implicit, and `where:` simply reads more clearly (to me and the other people that +I surveyed) than any other option we could come up with. + +## Changelog: + +1. The first version of this proposal used the `<`, `<=`, `>=`, and `>` operators. These have been +replaced with the `.`-prefixed operators. + +2. The first version of this proposal used `.*` and `.&*` for elementwise multiplication; these +have been replaced with the "normal" arithmetic operations, matching `+`, `-`, `/`, and `%`. + +3. We have adopted `.|`, `.&`, etc on the principle that `.`-prefixed operators are "pointwise". + +4. An earlier version of this proposal used names of the form `Float.Vector4` instead of +`Vector4`. The core team feels that the latter name is a better direction to take. + +5. We have removed several operations whose naming was not totally satisfactory. We +expect to add those operations and more in future additive proposals; only the types are +necessary for ABI stability. + +6. Updates based on discussion on Swift-Evolution and with core team: I have eliminated +`_`-prefixed protocol requirements, to make it somewhat more obvious how user types +can be made to conform. This required some renaming to make the function of these +associatedtypes more clear. I have also systematically removed `Vector` from the naming +scheme, in order to remove confusion with C++-style vectors or "mathematical" vector +structures that might be added at some future point. The resulting names are generally +shorter, which is an nice benefit, and they now have a uniform `SIMD` prefix. + + Protocols renamed: + + | Old name | New name | + | --- | --- | + | `SIMDVector` | `SIMD` | + | `SIMDVectorizable` | `SIMDScalar` | + | `SIMDVectorStorage` | `SIMDStorage` | + | `SIMDMaskVector` | N/A* | + + Associated types renamed: + + | Old name | New name | + | --- | --- | + | `Element` | `Scalar` | + | `_MaskElement` | `SIMDMaskScalar` | + | `_Vector${n}` | `SIMD${n}Storage` | + | `Mask` | N/A* | + | N/A* | `MaskStorage` | + + Concrete types renamed: + + | Old name | New name | + | --- | --- | + | `Vector2` | `SIMD2` | + | `Vector3` | `SIMD3` | + | `Vector4` | `SIMD4` | + | `Vector8` | `SIMD8` | + | `Vector16` | `SIMD16` | + | `Vector32` | `SIMD32` | + | `Vector64` | `SIMD64` | + | N/A* | `SIMDMask` | + + (*) The way in which masks are implemented has also been simplified somewhat, which + allowed me to eliminate the `SIMDMaskVector` protocol entirely. diff --git a/proposals/0230-flatten-optional-try.md b/proposals/0230-flatten-optional-try.md new file mode 100644 index 0000000000..62d9faef58 --- /dev/null +++ b/proposals/0230-flatten-optional-try.md @@ -0,0 +1,341 @@ +# Flatten nested optionals resulting from 'try?' + +* Proposal: [SE-0230](0230-flatten-optional-try.md) +* Author: [BJ Homer](https://github.com/bjhomer) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#16942](https://github.com/apple/swift/pull/16942) +* Review: ([forum thread](https://forums.swift.org/t/se-0230-flatten-nested-optionals-resulting-from-try/16570)) ([acceptance](https://forums.swift.org/t/accepted-se-230-flatten-nested-optionals-resulting-from-try/17376)) + +## Introduction + +Swift's `try?` statement currently makes it easy to introduce a +nested optional. Nested optionals are difficult for users to reason +about, and Swift tries to avoid producing them in other common cases. + +This document proposes giving `try?` the same optional-flattening +behavior found in other common Swift features, to avoid the common +occurrence of a nested optional. + +Swift-evolution thread: [Make try? + optional chain flattening work together](https://forums.swift.org/t/make-try-optional-chain-flattening-work-together/7415) + +## Motivation + +It's currently quite easy to end up with a nested `Optional` type when +using `try?`. Although it is valid to construct a nested optional, it +is usually not what the developer intended. + +Swift has various mechanisms to avoid accidentally creating nested optionals. For example: + +```swift +// Note how 'as?' produces the same type regardless of whether the value +// being cast is optional or not. +let x = nonOptionalValue() as? MyType // x is of type 'MyType?' +let y = optionalValue() as? MyType // y is of type 'MyType?' + +// Note how optional chaining produces the same type whether or not the +// call produces an optional value. +let a = optThing?.pizza() // a is of type 'Pizza?' +let b = optThing?.optionalPizza() // b is of type 'Pizza?' +``` + +However, `try?` behaves differently: + +```swift +let q = try? harbor.boat() // q is of type 'Boat?' +let r = try? harbor.optionalBoat() // r is of type 'Boat??' +``` + +The above examples are contrived, but it's actually quite common to end +up with a nested optional in production code. For example: + +```swift +// The result of 'foo?.makeBar()' is 'Bar?' because of the optional +// chaining on 'foo'. The 'try?' adds an additional layer of +// optionality. So the type of 'x' is 'Bar??' +let x = try? foo?.makeBar() + +// JSONSerialization.jsonObject(with:) returns an 'Any'. We use 'as?' to +// verify that the result is of the expected type, but the result is that 'dict' +// is now of type '[String: Any]??' because 'try?' added an additional layer. +let dict = try? JSONSerialization.jsonObject(with: data) as? [String: Any] +``` + +Although it is fairly easy to produce a nested optional using `try?`, a survey of +existing code suggests that it is almost never the desired outcome. Code that uses +`try?` with nested optionals is almost always accompanied by one of the following patterns: + +```swift +// Pattern 1: Double if-let or guard-let +if let optionalX = try? self.optionalThing(), + let x = optionalX { + // Use 'x' here +} + +// Pattern 2: Introducing parentheses to let 'as?' flatten for us +if let x = (try? somethingAsAny()) as? JournalEntry { + // use 'x' here +} + +// Pattern 3: Pattern matching +if case let x?? = try? optionalThing() { + // use 'x' here +} +``` + +The need for these workarounds makes the language more difficult to +learn and use, and they don't really give us any benefit in return. + +Code using `try?` generally does not care to distinguish between the error case +and the nil-result case, which is why all these patterns focus on extracting the +value and ignore the error. If the developer does care to specifically detect errors, +they should probably be using `do/try/catch` instead. + +## Proposed solution + +In Swift 5, `try? someExpr()` will mirror the behavior of `foo?.someExpr()`: + +- If `someExpr()` produces a non-optional value, it will be wrapped in an Optional. +- If `someExpr()` produces an `Optional`, then no additional optional-ness is added. + +This results in the following changes to the type of a `try?` expression: + +```swift +// Swift 4: 'Int??' +// Swift 5: 'Int?' +let result = try? database?.countOfRows(matching: predicate) + + +// Swift 4: 'String??' +// Swift 5: 'String?' +let myString = try? String(data: someData, encoding: .utf8) + +// Swift 4: '[String: Any]??' +// Swift 5: '[String: Any]?' +let dict = try? JSONSerialization.jsonObject(with: data) as? [String: Any] +``` + +There are no changes to the overall type when the sub-expression produces a non-optional. + +```swift +// Swift 4: 'String?' +// Swift 5: 'String?' +let fileContents = try? String(contentsOf: someURL) +``` + +If the sub-expression already produces a nested optional, the result is equally +nested: + +```swift +func doubleOptionalInt() throws -> Int?? { + return 3 +} + +// Swift 4: 'Int???' +// Swift 5: 'Int??' +let x = try? doubleOptionalInt() +``` + +> ### A side note about `try?` and `as?` +> +> Although `as?` often has the effect of flattening Optionals (as shown in +> the example in the Motivation section) it does not exhibit exactly +> the same behavior as proposed here for `try?`. Because `as?` takes +> an explicit type, it can actually flatten multiple levels of nested +> Optionals. `foo as? T` will always produce an `Optional`, regardless +> of how many optionals were on `foo`. This can potentially _add or subtract_ +> levels of optionals, depending on the type specified. (It can also cast between +> subtypes and supertypes, which is unrelated to the behavior under consideration.) +> +> In practice, the most common use of `as?` with nested optionals is to reduce from +> `T??` to `T?`, which makes it superficially similar to the use optional-chaining +> use case and the proposed behavior of `try?`. But `as?` is a more powerful +> and versatile construct than what is proposed for `try?` here. + + +## Detailed design + +The type of a `try?` expression in Swift 4 is defined as `Optional`, +where `T` is the type of the expression following the `try?` keyword. + +In Swift 5, the type of a `try?` expression will be some type `U`, where +`U` is an `Optional<_>` type and where the sub-expression type `T` is +coercible to `U`. The type constraint system automatically chooses the +smallest level of optional-nesting needed to satisfy this constraint, +which results in the behavior described in this proposal. + +#### Generics + +Some questions have been raised regarding the interoperability with +generic code, as in the following example: + +```swift +func test(fn: () throws -> T) -> T? { + + // Will this line change behavior if T is an Optional? + if let result = try? fn() { + print("We got a result!") + return result + } + else { + print("There was an error") + return nil + } +} + +// T is inferred as 'Int' here +let value = test({ return 15 }) + +// T is inferred as 'Int?' here +let value2 = test({ return 15 as Int? }) +``` + +The answer is that it does not matter if `T` is optional at runtime. +At compile time, `result` has a clearly-defined type: `T`. This is +true in both Swift 4 and Swift 5 modes; because `T` is not known to be +an `Optional` type at compile-time, a single layer of `Optional` is added +via the `try?` expression and then unwrapped via `if let`. + +Generic code that uses `try?` can continue to use it as always without +concern for whether the generic type might be optional at runtime. No +behavior is changed in this case. + + +## Source compatibility + +This is a source-breaking change for `try?` expressions that operate on an +`Optional` sub-expression _if they do not explicitly flatten the optional +themselves_. It appears that those cases are rare, though; see the analysis +below for details. We can provide backward-compatible behavior when the compiler +is running in Swift 4 mode, and a migration should be possible for most common +cases. + +The bar for including source-breaking changes in Swift 5 is high, but I believe +it passes the bar. These are the criteria listed in the swift-evolution README: + +### 1. The current syntax/API must be shown to actively cause problems for users. + +Nested optionals are a complex concept. They have value in the language, but their +use should be intentional. Currently, it's far too easy for beginners to create a +nested optional without understanding why, due to the current interaction of +`try?` and optional chaining or `as?` casting. + +Code that uses `try?` to actually detect errors is _more_ difficult to understand +than just using `try`. Compare: + +```swift +// Using 'try?' +if let result = try? foo?.bar() { + // Do something with 'result', which may be nil + // even though we are in an 'if let'. +} +else { + // There was an error, but we don't know what it is +} +``` + +```swift +// Using 'try/catch' +do { + let result = try foo?.bar() + // Do something with 'result', which may be nil due to optional chaining +} +catch { + // Handle the error +} +``` + +The variant using `try?` is significantly less clear (what is the type of +`result`?), and has no obvious advantages. Using `try?` _is_ better if you +don't care about the _else_ clause and only want to handle a value if one +exists, but that use case is better served by the proposed change. + +The current syntax is also harmful because of its interaction with `as?` casting, +as seen here: + +```swift +if let x = try? foo() as? String { + // We specifically called out `String` as the desired type here, + // so it unexpected that `x` is of type `Optional` +} +``` + +### 2. The new syntax/API must be clearly better and must not conflict with existing Swift syntax. + +The proposed change resolves all of the above problems, which is better. +This change also clarifies the role of `try?` as a means for accessing +values when possible, rather than as an alternative error-handling +mechanism to `try/catch`. + +### 3. There must be a reasonably automated migration path for existing code. + +As shown in the analysis below, most source code will require no migration; +developers who have encountered code that produces nested Optionals are likely +already using patterns that are source-compatible with this change. This proposal +simply provides a way to simplify that code. + +Automated migration is implemented for the double `if/guard let` and +`case let value??:` patterns mentioned above. + +## Swift Source Compatibility Suite analysis + +The Swift Source Compatibility Suite suggests that this is unlikely to +be a breaking change for most users. I manually inspected the use +cases of `try?` in the compatibility suite. Here are the results: + +* There are **613** total instances of `try?` in the compatibility suite. The vast majority of those appear to use non-optional sub-expressions, and would be unaffected by this proposal. + +* There are **4** instances of `try? ... as?`. All four of them wrap the `try?` in parentheses to get the flattening behavior of `as?`, and are source-compatible with this change. They all look something like this: + + ```swift + (try? JSONSerialization.jsonObject(with: $0)) as? NSDictionary + ``` + +* There are **12** cases of `try? foo?.bar()` across 3 projects. +**10** of those assign it to `_ = try? foo?.bar()` , so the resulting type does not matter. +**2** of those cases have a `Void` sub-expression type, and do not assign the result to any variable. + +* There are **6** instances of `try? somethingReturningOptional()` . They all flatten it manually using `flatMap { $0 }`, and are thus source-compatible with this change, as the return type of that expression is the same under either behavior. + + ```swift + (try? get(key)).flatMap { $0 } + ``` + +* As far as I can tell, there are **zero** cases in the entire suite where a double-optional is actually used to distinguish between the error case and the nil-as-a-value case. + +* As far as I can tell, there are **zero** cases of source incompatibility found in the compatibility suite. + + +## Effect on ABI stability + +No impact on ABI. + +## Effect on API resilience + +A `try?` expression is never exposed at function boundaries, so API +resilience should be unaffected. + +## Alternatives considered + +### Alter the binding precedence of try? + +For expressions like `let x = try? getObject() as? Foo`, the nested optional +can be eliminated by parsing the expression as `(try? getObject) as? Foo`. +Adding explicit parentheses like this is already a common workaround for +the double-optional problem. + +However, this change would not resolve cases of `try?` with optional chaining +(e.g. `try? foo?.bar?.baz()`), nor cases where an optional result is returned +directly from a function (e.g. `try? cachedValue(for: key)`). + +Altering the binding precedence of `try?` would also be *far* more +source-breaking than this proposal. + +### Do nothing + +It is possible to write correct code under the current model. We are not +proposing to eliminate nested Optionals from the language entirely, so +we could just expect users to figure them out. + +This is a workable solution, but it is odd to have such a complex structure +produced as part of a syntactic sugar designed to simplify a common case. diff --git a/proposals/0231-optional-iteration.md b/proposals/0231-optional-iteration.md new file mode 100644 index 0000000000..ec8dd5f227 --- /dev/null +++ b/proposals/0231-optional-iteration.md @@ -0,0 +1,141 @@ +# Optional Iteration + +* Proposal: [SE-0231](0231-optional-iteration.md) +* Author: [Anthony Latsis](https://github.com/AnthonyLatsis) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Rejected** +* Implementation: [apple/swift#19207](https://github.com/apple/swift/pull/19207) +* Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0231-optional-iteration/17805) + +## Introduction + +Optionals are a key feature of Swift and a powerful tool that seamlessly interacts with code. In particular, they serve a great means in expressing "act accordingly if there's a value, skip otherwise". Some vivid examples of such behavior are optional chaining, optional invocation `foo?()`, `if let`, [optional patterns](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_optional-pattern), optional assignments and `guard let`. This proposal considers further supporting this convenience in `for-in` loops. + +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/another-try-at-allowing-optional-iteration/14376?u=anthonylatsis) + +## Motivation + +Most Swift statements provide convenience patterns and handling for optionals. We have optional binding patterns for `while`, `if` and `guard`. Consider `switch`, that can be used directly on an optional to match against the unwrapped value. Nevertheless, it is important to keep in mind that exhaustiveness still applies, that is, the `nil` case must also be handled either explicitly or via the `default` clause: + +```swift +let str: Int? = nil + +switch str { +case 0: print() +case 1: print() +default: print() +} +``` + +Optional patterns bring a succinct way of handling the `.some` case when optional binding is unavailable: + +```swift +for case let unwrapped? in sequence { ... } +``` + +Optional assignment lets you skip an assignment if the lvalue is nil, sparing the need to write an entire `if-else` or deal with access exclusivity when using the ternary operator. A very useful albeit sparsely documented feature. + +```swift +var ages = ["Amy" : 30, "Graham" : 5] +ages["Anthony"]? = 21 +ages["Graham"]? = 6 +print(ages) // ["Amy" : 30, "Graham" : 6] +``` + +Loops are a common statement in almost every codebase. Similarly, a possibility to optionally iterate over a sequence (iterate if there is a value, otherwise skip) when the `nil` case is of no interest is self-explanatory. While usage of optional sequences is often treated as misconception, there are several common ways one could end up with an optional sequence through Standard Library APIs and language constructs themselves. Amongst the most prevalent are optional chaining and dictionary getters. An indentation-sensitive area of which optional arrays are an integral part is decoding and deserialization, i.e parsing a JSON response. +Swift currently doesn't offer a mechanism for expressing optional iteration directly: optional sequences are illegal as a `for-in` loop argument. For a safe option, developers often resort to optional binding, which requires additional nesting: + +```swift +if let sequence = optionalSequence { + for element in sequence { ... } +} +``` +There are several workarounds to avoid that extra level of indentation, none of which can be called a general solution: +* `guard` is a pretty straight-forward option for a simple scenario, but `guard` doesn't fall through – if handling the nil case is unnecessary and there follows flow-sensitive logic that is resistant to `nil` or doesn't depend on that whatsoever, rearranging the flow with `guard` is likely to become a counterproductive experiment that affects readability while still keeping the indentation. +* Coalescing `??` with an empty literal is only valid with types that conform to a corresponding `ExpressibleByLiteral` protocol. Just in the Standard Library, there is a considerable amount of sequence types that cannot be expressed literally. Most of them are frequently used indirectly: + * [Type-erasing](https://developer.apple.com/documentation/swift/anysequence#see-also) and [lazy](https://developer.apple.com/documentation/swift/lazysequence#see-also) wrappers + * [Zipped](https://developer.apple.com/documentation/swift/zip2sequence) and [enumerated](https://developer.apple.com/documentation/swift/enumeratedsequence) sequences + * `String` views ([`String.UTF8View`](https://developer.apple.com/documentation/swift/string/utf8view), [`String.UTF16View`](https://developer.apple.com/documentation/swift/string/utf16view), [`String.UnicodeScalarView`](https://developer.apple.com/documentation/swift/string/unicodescalarview)) + * [Default indices](https://developer.apple.com/documentation/swift/defaultindices) + * [Reversed](https://developer.apple.com/documentation/swift/reversedcollection) and [repeated](https://developer.apple.com/documentation/swift/repeated) collections + * Strides ([`StrideTo`](https://developer.apple.com/documentation/swift/strideto), [`StrideThrough`](https://developer.apple.com/documentation/swift/stridethrough)) + * [Flattened](https://developer.apple.com/documentation/swift/flattensequence), [joined](https://developer.apple.com/documentation/swift/joinedsequence) and [unfolding](https://developer.apple.com/documentation/swift/unfoldsequence) sequences. + + An empty instance is not guaranteed to exist for an arbitrary sequence regardless of whether it can be expressed literally. This helps to see another flaw in the `?? #placeholder#` fix-it from an engineer's perspective. There are potentially untraceable cases when the fix-it is wrong. Furthermore, literals are unavailable in generic contexts that aren't additionally constrained to an `ExpressibleBy*Literal` protocol. + +* Reaching for `sequence?.forEach` is not an alternative if you are using control transfer statements, such as `continue` and `break`. The differences are clearly listed in the [documentation](https://developer.apple.com/documentation/swift/sequence/3018367-foreach): + + > Using the forEach method is distinct from a for-in loop in two important ways: + > + > 1. You cannot use a `break` or `continue` statement to exit the current call of the body closure or skip subsequent calls. + > + > 2. Using the `return` statement in the body closure will exit only from the current call to body, not from any outer + > scope, and won’t skip subsequent calls. + +## Proposed solution + +This proposal introduces optional iteration (`for?`) and hence the possibility to use optional sequences as the corresponding attribute in `for-in` loops. + +``` swift +let array: [Int]? = nil + +for? element in array { ... } +// Equivalent to +if let unwrappedArray = array { + for element in unwrappedArray { ... } +} +``` + +The `?` notation here is a semantic emphasis rather than a functional unit: there is no `for!`. Syntactically marking an optional iteration is redundant, however, in contrast to `switch`, nil values are *skipped silently*. Swift strives to follow a style where silent handling of `nil` is acknowledged via the `?` sigil, distinctly reflected in optional chaining syntax. This decision was primarily based on inconsistency and potential confusion that an otherwise left without syntactic changes `for-in` loop could potentially lead to ("clarity over brevity"). + +``` swift +for element in optionalArray { ... } // Silently handling optionals implicitly is a style that Swift prefers to eschew. +``` + +From the author's point of view, the solution's most significant advantage is generality and hence scalability. `for?` is independent of the nature and form of the sequence argument and freely composes with any possible expression, be it a cast, `try`, or a mere optional chain. Albeit being an unprecedented optional handling case among statements on the grounds of the need to always omit the identifier to which the unwrapped value is bound, the community points out inconsistency in relation to other statements. + +## Detailed design + +An optional `for-in` loop over a nil sequence does nothing. To be precise, it trips over nil when `sequence?.makeIterator()` is invoked and continues execution. Otherwise, it iterates normally. One can roughly imagine an optional `for-in` loop as `sequence?.forEach` with all the pattern-matching features and benefits of a `for-in` statement. + +The `?` notation in `for?` is required when the passed sequence is optional and disallowed otherwise. +```swift +let array: [Int] = [1, 2, 3] +let optArray: [Int]? = nil + +for element in optArray { // The usual 'must be force-unwrapped' error, but with the preferred fixit to use 'for?' +... +} + +for? element in array { // error: optional for-in loop must not be used on a non-optional sequence of type '[Int]' +... +} +``` + +## Source compatibility + +This feature is purely additive. + +## Effect on ABI stability + +None + +## Alternatives considered + +### Imitating optional chaining + +A syntactically less disruptive approach, the idea of which is denoting an optional iteration by selectively following the sequence expression with `?`: + +```swift +let array: [Int]? = [1, 2, 3] +for element in sequence? { ... } +``` +A terminating `?` sigil here can be thought of as bringing the `for` loop into the optional chain with the sequence and mirrors the force-unwrapping case (`sequence!`). The technique implies that a degenerate optional chain (`sequence?`) should end with the `?` sigil, but expressions that already acknowledge optionality, for instance `sequence?.reversed()`, `data as? [T]`, `try? sequenceReturningMethod()`, may be left as-is. It is unclear how to address expressions that don't acknowledge optionality but carry their own syntax without resorting to parenthesizing. One example of such an expression would be `try methodReturningOptionalSequence()`. + +#### Nested optionals + +As a mechanism that inherently runs only on *non-optional* sequences, `for-in` asks for optional flattening. The position inclines for expressions that acknowledge optionality to keep their optional flattening behavior, while enabling optional flattening on degenerate optional chains, so that types such as `[T]???...` can be iterated without *additional* syntactic load. + +### Purely implicit + +The option of leaving out any syntactic changes was also discussed and met concern from the community. The drawback is briefly explained in the [Proposed solution](#proposed-solution) section. diff --git a/proposals/0232-remove-customization-points.md b/proposals/0232-remove-customization-points.md new file mode 100644 index 0000000000..a9fc872825 --- /dev/null +++ b/proposals/0232-remove-customization-points.md @@ -0,0 +1,124 @@ +# Remove Some Customization Points from the Standard Library's `Collection` Hierarchy + +* Proposal: [SE-0232](0232-remove-customization-points.md) +* Author: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#19995](https://github.com/apple/swift/pull/19995) +* Review: [Discussion thread](https://forums.swift.org/t/se-0232-remove-some-customization-points-from-the-standard-librarys-collection-hierarchy/17265), [Announcement thread](https://forums.swift.org/t/accepted-se-0232-remove-some-customization-points-from-the-standard-librarys-collection-hierarchy/17560) + +## Introduction + +This proposal removes four customization points from protocols in the +standard library: + +- `map`, `filter`, and `forEach` from `Sequence` +- `first`, `prefix(upTo:)`, `prefix(through:)`, and `suffix(from:)` from `Collection` +- `last` on `BidirectionalCollection` + +The default implementations of these symbols will remain, so sequences and +collections will continue to have the same operations available that they +do today. + +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/pitch-remove-some-customization-points-from-the-std-lib-collection-protocols/16911/) + +## Motivation + +Customization points are methods that are declared on the protocol, as well as +given default implementations in an extension. This way, when a type provides +its own non-default implementation, this will be dispatched to in a generic +context (e.g. when another method defined in an extension on the protocol calls +the customized method). Without a customization point, the default +implementation is called in the generic context. + +This serves broadly two purposes: + +1. Allowing for differences in behavior. For example, an "add element" method on + a set type might exclude duplicates while on a bag it might allow them. + +2. Allowing for more efficient implementations. For example, `count` on + forward-only collections takes O(n) (because without random access, the + implementation needs to iterate the collection to count it). But some + collection types might know their `count` even if they aren't random access. + +Once ABI stability has been declared for a framework, customization points can +never be removed, though they can be added. + +Customization points aren't free – they add a small cost at both compile time +and run time. So they should only be added if there is a realistic possibility +that either of the two reasons above apply. + +In the case of the customization points in this proposal, reason 1 does not +apply. In fact it could be considered a serious bug if any type implemented +these features with anything other than the default observable behavior. + +It is also hard to find a good use case for reason 2 – whereas slight slowdowns +and code size bloat from the presence of the customization points have been observed. +In some cases (for example `suffix(from:)`), the implementation is so simple that +there is no reasonable alternative implementation. + +While it is possible that a resilient type's `forEach` implementation might be able +to eke out a small performance benefit (for example, to avoid the reference count +bump of putting `self` into an iterator), it is generally harmful to encourage this +kind of "maybe forEach could be faster" micro-optimization. For example, see +[here](https://github.com/apple/swift/pull/17387), where error control flow was +used in order to break out of the `forEach` early, causing unpleasant +interference for debugging workflows that detected when errors were thrown. + +### Future move-only type considerations + +In the case of `first` and `last` there is an additional consideration: in the +future, collections of move-only types (including `Array`) will not be able +to reasonably fulfil these requirements. + +A collection that contains move-only types will only allow elements to be +either removed and returned (e.g. with `popLast()`), or borrowed (e.g. via +`subscript`). + +Returning an optional to represent the first element fits into neither of these +buckets. You cannot write a generic implementation of `first` that fetches the +first move-only element of a collection using a subscript, moves it into an +optional, and then returns that optional. + +This means `first` and `last` need to be removed as requirements on the +collection protocols in order to make it possible for collections of move only +types to conform to them. + +They would remain on `Collection` via extensions. When move-only types are +introduced, those extensions will be constrained to the collection element +being copyable. + +Once the final functionality for move-only types is designed, it may be that +language features will be added that allow for borrowing into an optional, +allowing even collections of move-only types to implement a `first` property. +But it's better to err on the side of caution for now and remove them from +the protocol. + +## Proposed solution + +Remove these customization points from the `Collection` protocols. The +default implementations will remain. + +## Source compatibility + +These are customization points with an existing default implementation, so +there is no effect on source stability. + +It is theoretically possible that removing these customization points could +result in a behavior change on types that rely on the dynamic dispatch to add +additional logic. However, this would be an extremely dubious practice e.g. +`MyCollection.first` should really never do anything more than return the first +element. + +## Effect on ABI stability + +Removing customization points is not an ABI-stable operation. The driver for +this proposal is to do this before declaring ABI stability. + +## Effect on API resilience + +None + +## Alternatives considered + +Leave them in. Live with the slight code/performance impact in the case of `map` and `forEach`, and work around the issue when designing move-only types. diff --git a/proposals/0233-additive-arithmetic-protocol.md b/proposals/0233-additive-arithmetic-protocol.md new file mode 100644 index 0000000000..dca603f18c --- /dev/null +++ b/proposals/0233-additive-arithmetic-protocol.md @@ -0,0 +1,210 @@ +# Make `Numeric` Refine a new `AdditiveArithmetic` Protocol + +* Proposal: [SE-0233](0233-additive-arithmetic-protocol.md) +* Author: [Richard Wei](https://github.com/rxwei) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#20422](https://github.com/apple/swift/pull/20422) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0233-make-numeric-refine-a-new-additivearithmetic-protocol/17751) + +## Introduction + +This proposal introduces a weakening of the existing `Numeric` protocol named `AdditiveArithmetic` , which defines additive arithmetic operators and a zero, making conforming types roughly correspond to the mathematic notion of an [additive group](https://en.wikipedia.org/wiki/Additive_group). This makes it possible for vector types to share additive arithmetic operators with scalar types, which enables generic algorithms over `AdditiveArithmetic` to apply to both scalars and vectors. + +Discussion thread: [Should Numeric not refine ExpressibleByIntegerLiteral](https://forums.swift.org/t/should-numeric-not-refine-expressiblebyintegerliteral/15106) + +## Motivation + +The `Numeric` protocol today refines `ExpressibleByIntegerLiteral` and defines all arithmetic operators. The design makes it easy for scalar types to adopt arithmetic operators, but makes it hard for vector types to adopt arithmetic operators by conforming to this protocol. + +What's wrong with `Numeric`? Assuming that we need to conform to `Numeric` to get basic arithmetic operators and generic algorithms, we have three problems. + +### 1. Vectors conforming to `Numeric` would be mathematically incorrect. + +`Numeric` roughly corresponds to a [ring](https://en.wikipedia.org/wiki/Ring_(mathematics)). Vector spaces are not rings. Multiplication is not defined between vectors. Requirements `*` and `*=` below would make vector types inconsistent with the mathematical definition. + +```swift +static func * (lhs: Self, rhs: Self) -> Self +static func *= (lhs: inout Self, rhs: Self) +``` + +### 2. Literal conversion is undefined for dynamically shaped vectors. + +Vectors can be dynamically shaped, in which case the the shape needs to be provided when we initialize a vector from a scalar. Dynamically shaped vector types often have an initializer `init(repeating:shape:)`. + +Conforming to `Numeric` requires a conformance to `ExpressibleByIntegerLiteral`, which requires `init(integerLiteral:)`. However, the conversion from a scalar to a dynamically shaped vector is not defined when there is no given shape. + +```swift +struct Vector: Numeric { + // Okay! + init(repeating: Scalar, shape: [Int]) { ... } + + // What's the shape? + init(integerLiteral: Int) +} +``` + +### 3. Common operator overloading causes type checking ambiguity. + +Vector types mathematically represent [vector spaces](https://en.wikipedia.org/wiki/Vector_space). Vectors by definition do not have multiplication between each other, but they come with scalar multiplication. + +```swift +static func * (lhs: Vector, rhs: Scalar) -> Vector { ... } +``` + +By established convention in numerical computing communities such as machine learning, many libraries define a multiplication operator `*` between vectors as element-wise multiplication. Given that scalar multiplication has to exist by definition, element-wise multiplication and scalar multiplication would overload the `*` operator. + +```swift +static func * (lhs: Vector, rhs: Vector) -> Vector { ... } +static func * (lhs: Vector, rhs: Scalar) -> Vector { ... } +``` + +This compiles, but does not work in practice. The following trivial use case would fail to compile, because literal `1` can be implicitly converted to both a `Scalar` and a `Vector` , and `*` is overloaded for both `Vector` and `Scalar` . + +```swift +let x = Vector(...) +x * 1 // Ambiguous! Can be both `x * Vector(integerLiteral: 1)` and `x * (1 as Int)`. +``` + +## Proposed solution + +We keep `Numeric` 's behavior and requirements intact, and introduce a new protocol that +- does not require `ExpressibleByIntegerLiteral` conformance, and +- shares common properties and operators between vectors and scalars. + +To achieve these, we can try to find a mathematical concept that is close enough to makes practical sense without depending on unnecessary algebraic abstractions. This concept is additive group, containing a zero and all additive operators that are defined on `Numeric` today. `Numeric` will refine this new protocol, and vector types/protocols will conform to/refine the new protocol as well. + +## Detailed design + +We define a new protocol called `AdditiveArithmetic` . This protocol requires all additive arithmetic operators that today's `Numeric` requires, and a zero. Zero is a fundamental property of an additive group. + +```swift +public protocol AdditiveArithmetic: Equatable { + /// A zero value. + static var zero: Self { get } + + /// Adds two values and produces their sum. + /// + /// The addition operator (`+`) calculates the sum of its two arguments. For + /// example: + /// + /// 1 + 2 // 3 + /// -10 + 15 // 5 + /// -15 + -5 // -20 + /// 21.5 + 3.25 // 24.75 + /// + /// You cannot use `+` with arguments of different types. To add values of + /// different types, convert one of the values to the other value's type. + /// + /// let x: Int8 = 21 + /// let y: Int = 1000000 + /// Int(x) + y // 1000021 + /// + /// - Parameters: + /// - lhs: The first value to add. + /// - rhs: The second value to add. + static func + (lhs: Self, rhs: Self) -> Self + + /// Adds two values and stores the result in the left-hand-side variable. + /// + /// - Parameters: + /// - lhs: The first value to add. + /// - rhs: The second value to add. + static func += (lhs: inout Self, rhs: Self) -> Self + + /// Subtracts one value from another and produces their difference. + /// + /// The subtraction operator (`-`) calculates the difference of its two + /// arguments. For example: + /// + /// 8 - 3 // 5 + /// -10 - 5 // -15 + /// 100 - -5 // 105 + /// 10.5 - 100.0 // -89.5 + /// + /// You cannot use `-` with arguments of different types. To subtract values + /// of different types, convert one of the values to the other value's type. + /// + /// let x: UInt8 = 21 + /// let y: UInt = 1000000 + /// y - UInt(x) // 999979 + /// + /// - Parameters: + /// - lhs: A numeric value. + /// - rhs: The value to subtract from `lhs`. + static func - (lhs: Self, rhs: Self) -> Self + + /// Subtracts the second value from the first and stores the difference in the + /// left-hand-side variable. + /// + /// - Parameters: + /// - lhs: A numeric value. + /// - rhs: The value to subtract from `lhs`. + static func -= (lhs: inout Self, rhs: Self) -> Self +} +``` + +Remove arithmetic operator requirements from `Numeric` , and make `Numeric` refine `AdditiveArithmetic` . + +```swift +public protocol Numeric: AdditiveArithmetic, ExpressibleByIntegerLiteral { + associatedtype Magnitude: Comparable, Numeric + init?(exactly source: T) where T : BinaryInteger + var magnitude: Self.Magnitude { get } + static func * (lhs: Self, rhs: Self) -> Self + static func *= (lhs: inout Self, rhs: Self) -> Self +} +``` + +To make sure today's `Numeric` -conforming types do not have to define a `zero` , we provide an extension to `AdditiveArithmetic` constrained on `Self: ExpressibleByIntegerLiteral` . + +```swift +extension AdditiveArithmetic where Self: ExpressibleByIntegerLiteral { + public static var zero: Self { + return 0 + } +} +``` + +In the existing standard library, prefix `+` is provided by an extension to +`Numeric`. Since additive arithmetics are now defined on `AdditiveArithmetic`, +we change this extension to apply to `AdditiveArithmetic`. + +```swift +extension AdditiveArithmetic { + /// Returns the given number unchanged. + /// + /// You can use the unary plus operator (`+`) to provide symmetry in your + /// code for positive numbers when also using the unary minus operator. + /// + /// let x = -21 + /// let y = +21 + /// // x == -21 + /// // y == 21 + /// + /// - Returns: The given argument without any changes. + public static prefix func + (x: Self) -> Self { + return x + } +} +``` + +## Source compatibility + +The proposed change is fully source-compatible. + +## Effect on ABI stability + +The proposed change will affect the existing ABI of the standard library, because it changes the protocol hierarchy and protocol requirements. As such, this protocol must be considered before the Swift 5 branching date. + +## Effect on API resilience + +The proposed change will affect the existing ABI, and there is no way to make it not affect the ABI because it changes the protocol hierarchy and protocol requirements. + +## Alternatives considered + +1. Make `Numeric` no longer refine `ExpressibleByIntegerLiteral` and not introduce any new protocol. This can solve the type checking ambiguity problem in vector protocols, but will break existing code: Functions generic over `Numeric` may use integer literals for initialization. Plus, Steve Canon also pointed out that it is not mathematically accurate -- there's a canonical homomorphism from the integers to every ring with unity. Moreover, it makes sense for vector types to conform to `Numeric` to get arithmetic operators, but it is uncommon to make vectors, esp. fixed-rank vectors, be expressible by integer literal. + +2. On top of `AdditiveArithmetic`, add a `MultiplicativeArithmetic` protocol that refines `AdditiveArithmetic`, and make `Numeric` refine `MultiplicativeArithmetic`. This would be a natural extension to `AdditiveArithmetic`, but the practical benefit of this is unclear. + +3. Instead of a `zero` static computed property requirement, an `init()` could be used instead, and this would align well with Swift's preference for initializers. However, this would force conforming types to have an `init()`, which in some cases could be confusing or misleading. For example, it would be unclear whether `Matrix()` is creating a zero matrix or an identity matrix. Spelling it as `zero` eliminates that ambiguity. diff --git a/proposals/0234-remove-sequence-subsequence.md b/proposals/0234-remove-sequence-subsequence.md new file mode 100644 index 0000000000..6cf850fccb --- /dev/null +++ b/proposals/0234-remove-sequence-subsequence.md @@ -0,0 +1,325 @@ +# Remove `Sequence.SubSequence` + +* Proposal: [SE-0234](0234-remove-sequence-subsequence.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#20221](https://github.com/apple/swift/pull/20221) +* Review: ([review thread](https://forums.swift.org/t/se-0234-removing-sequence-subsequence/17750)) ([acceptance](https://forums.swift.org/t/accepted-se-0234-remove-sequence-subsequence/18002)) + +## Introduction + +This proposal recommends eliminating the associated type from `Sequence`, +moving it up to start at `Collection`. Current customization points on +`Sequence` returning a `SubSequence` will be amended to be extensions returning +concrete types. + +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/rationalizing-sequence-subsequence/17586) + +## Motivation + +### Current usage + +Today, `Sequence` declares several methods that return a `SubSequence`: + +```swift +func dropFirst(_:) -> SubSequence +func dropLast(_:) -> SubSequence +func drop(while:) -> SubSequence +func prefix(_:) -> SubSequence +func prefix(while:) -> SubSequence +func suffix(_:) -> SubSequence +func split(separator:) -> [SubSequence] +``` + +You don't have to implement them to implement a `Sequence`. They all have default implementations for the default type for `SubSequence`. + +But if you think about _how_ you'd implement them generically on a single-pass sequence, you'll quickly realize there is a problem. They all call for completely different return types in their implementation. For example, the ideal way to implement `dropFirst` would be to return a wrapper type that first drops n elements, then starts returning values. `prefix` would ideally be implemented the other way around: return elements until you've returned n, then stop. `suffix` and `dropLast` need to consume the entire sequence to get to the end, buffering as they go, then return what they buffered. `drop(while:)` needs to eagerly drop non-matching elements as soon as it's called (because the closure is not `@escaping`), which means it needs to buffer one element in case that's the first one it needs to return later. `prefix(while:)` also needs to eagerly search and buffer _everything_ it reads. + +But the protocol requires all these methods return the same type – `SubSequence`. They can't return specific types for their specific needs. In theory, that could be resolved by having them all return `[Element]`, but that would be wasteful of memory in cases like `prefix(_:)`. + +The way the std lib works around this is to make the default `SubSequence` an `AnySequence`. So internally, there is a `DropFirstSequence` type, that is created when you call `dropFirst` on a `Sequence`. But it is type-erased to be the same type as returned by `prefix`, `suffix` etc., which also return their own custom types, but type erased. + +Unfortunately this has two major consequences: +- performance is bad: type-erased wrappers are an optimization barrier; and +- it blocks conditional conformance going from `Sequence` to `Collection`. + +Additionally, it makes implementing your own custom `SubSequence` that _isn't_ `AnySequence` extremely hard, because you need to then implement your own version of all these methods, even `split`. So in practice, this is never done. + +### Type erasure performance + +There is a prototype in [this PR](https://github.com/apple/swift/pull/20175) that replaces the customization points on `Sequence` with regular extensions that each return a specific type. + +To see the performance problem, here is how the [benchmarks](https://github.com/apple/swift/pull/20175#issuecomment-434661846) improve if you return a non-type-erased type instead: + +#### Performance: -O + +TEST | OLD | NEW | DELTA | RATIO +--- | --- | --- | --- | --- +**Improvement** | | | | +DropWhileSequence | 2214 | 29 | -98.7% | **76.34x** +PrefixSequenceLazy | 2265 | 52 | -97.7% | **43.56x** +PrefixSequence | 2213 | 52 | -97.7% | **42.56x** +DropFirstSequenceLazy | 2310 | 59 | -97.4% | **39.15x** +DropFirstSequence | 2240 | 59 | -97.4% | **37.97x** + +#### Performance: -Osize + +TEST | OLD | NEW | DELTA | RATIO +--- | --- | --- | --- | --- +**Improvement** | | | | +DropWhileAnySeqCRangeIter | 17631 | 163 | -99.1% | **108.16x** +DropFirstAnySeqCRangeIterLazy | 21259 | 213 | -99.0% | **99.81x** +PrefixAnySeqCRangeIterLazy | 16679 | 176 | -98.9% | **94.77x** +PrefixAnySeqCntRangeLazy | 15810 | 168 | -98.9% | **94.11x** +DropFirstAnySeqCntRangeLazy | 15717 | 213 | -98.6% | **73.79x** +DropWhileSequence | 2582 | 35 | -98.6% | **73.77x** +DropFirstSequenceLazy | 2671 | 58 | -97.8% | **46.05x** +DropFirstSequence | 2649 | 58 | -97.8% | **45.67x** +PrefixSequence | 2705 | 70 | -97.4% | **38.64x** +PrefixSequenceLazy | 2670 | 70 | -97.4% | **38.14x** + +These performance improvements are all down to how well the optimizer can eliminate the wrapper abstractions when there isn’t the barrier of type erasure in the way. In -Onone builds, you don’t see any speedup. + +### How does it block conditional conformance? + +The problem with `SubSequence` really became clear when conditional conformance +was implemented. With conditional conformance, it becomes really important that +an associated type be able to grow and take on capabilities that line up with +the capabilities you are adding with each new conformance. + +For example, the `Slice` type that is the default `SubSequence` for +`Collection` grows in capabilities as it’s base grows. So for example, if the +`Base` is a `RandomAccessCollection`, then so can the `Slice` be. This then +works nicely when you add new conformances to a `Collection` that _uses_ +`Slice` as it’s `SubSequence` type. For more detail on this, watch Doug’s +explanation in our WWDC [Swift +Generics](https://developer.apple.com/videos/play/wwdc2018/406/?time=1614) talk +(starts at about minute 26). + +But the default type for `Sequence.SubSequence` is `AnySequence`, which is a +conformance dead end. You cannot add additional capabilities to `AnySequence` +because there is nothing to drive them: the type erases all evidence of it’s +wrapped type – that’s it’s point. + +This in turn forces two implementations of types that would ideally have a +single unified implementation. For example, suppose you wanted to write +something similar to `EnumeratedSequence` from the standard library, but have +it be a `Collection` as well when it could support it. + +First you start with the basic type (note, all this code takes shortcuts for +brevity): + +```swift +struct Enumerated { + let _base: Base +} +extension Sequence { + func enumerated() -> Enumerated { + return Enumerated(_base: self) + } +} +``` + +And add `Sequence` conformance: + +```swift +extension Enumerated: Sequence { + typealias Element = (Int,Base.Element) + struct Iterator: IteratorProtocol { + var _count: Int + let _base: Base.Iterator + mutating func next() -> Element? { + defer { _count += 1 } + return _base.next().map { (_count,$0) } + } + } + func makeIterator() -> Enumerated.Iterator { + return Iterator(_count: 0, _base: _base.makeIterator()) + } +} +``` + +Then, you’d want to add `Collection` conformance when the underlying base is +also a collection. Something like this: + +```swift +extension Enumerated: Collection where Base: Collection { + struct Index: Comparable { + let _count: Int, _base: Base.Index + static func < (lhs: Index, rhs: Index) -> Bool { + return lhs._base < rhs._base + } + } + var startIndex: Index { return Index(_count: 0, _base: _base.startIndex) } + var endIndex: Index { return Index(_count: Int.max, _base: _base.endIndex) } + subscript(i: Index) -> Element { + return (i._count, _base[i._base]) + } +} +``` + +You’d then follow through with conformance to `RandomAccessCollection` too when +the base was. You can see this pattern used throughout the standard library. + +But this code won’t compile. The reason is that `Collection` requires that +`SubSequence` also be a collection (and `BidirectionalCollection` requires it +be bi-directional, and so on). This all works perfectly for `Slice`, the +default value for `SubSequence` from `Collection` down, which progressively +acquires these capabilities and grows along with the protocols it supports. But +`AnySequence` can’t, as described above. It blocks all further capabilities. + +Because of this, if you want to support collection-like behavior, you’re back +to the bad old days before conditional conformance. You have to declare two +separate types: `EnumeratedSequence` and `EnumeratedCollection`, and define the +`enumerated` function twice. This is bad for code size, and also leaks into +user code, where these two different types appear. + +### Why is `Sequence` like this? + +The reason why `Sequence` declares this associated type and then forces all +these requirements to return it is to benefit from a specific use case: writing +a generic algorithm on `Sequence`, and then passing a `Collection` to it. +Because these are customization points, when `Collection` is able to provide a +better implementation, that generic algorithm can benefit from it. + +For example, suppose you pass an `Array`, which provides random-access, into an +algorithm that then calls `suffix`. Instead of needing to buffer all the +elements, requiring linear time _and_ memory allocation, it can just return a +slice in constant time and no allocation. + +You can see this in the regressions from the same PR: + +#### Performance: -O + +TEST | OLD | NEW | DELTA | RATIO +--- | --- | --- | --- | --- +**Regression** | | | | +DropLastAnySeqCntRangeLazy | 9 | 20366 | +226163.8% | **0.00x** +SuffixAnySeqCntRangeLazy | 14 | 20699 | +147739.4% | **0.00x** +DropLastAnySeqCntRange | 9 | 524 | +5721.6% | **0.02x** +SuffixAnySeqCntRange | 14 | 760 | +5328.2% | **0.02x** + +#### Performance: -Osize + +TEST | OLD | NEW | DELTA | RATIO +--- | --- | --- | --- | --- +**Regression** | | | | +DropLastAnySeqCRangeIterLazy | 3684 | 20142 | +446.7% | **0.18x** +SuffixAnySeqCRangeIterLazy | 3973 | 20223 | +409.0% | **0.20x** +SuffixAnySeqCntRangeLazy | 5225 | 20235 | +287.3% | **0.26x** +DropLastAnySeqCntRangeLazy | 5256 | 20113 | +282.7% | **0.26x** +DropFirstAnySeqCntRange | 15730 | 20645 | +31.2% | **0.76x** + +What is happening in these is a random-access collection (a `CountableRange`) +is being put inside the type-erasing `AnySequence`, then `suffix` or `dropLast` +is being called on it. The type-erased wrapper is then forwarding on the call +to the wrapped collection. Fetching the suffix of a countable range is +incredibly fast (it basically does nothing, ranges are just two numbers so it’s +just adjusting the lower bound upwards), whereas after removing the +customization points, the `suffix` function that’s called is the one for a +`Sequence`, which needs to iterate the range and buffer the elements. + +This is a nice performance tweak, but it doesn’t justify the significant +downsides listed above. It is essentially improving generic performance at the +expense of concrete performance. Normally, these kind of generic improvements +come at just a tiny cost (e.g. slight increase in compile time or binary size) +rather than a significant runtime performance penalty. + +## Proposed solution + +Remove the `SubSequence` associated type from `Sequence`. It should first +appear from `Collection` onwards. + +Remove the methods on `Sequence` that return `SubSequence` from the protocol. +They should remain as extensions only. Each one should return a specific type +best suited to the task: + +```swift +extension Sequence { + public func dropFirst(_ k: Int = 1) -> DropFirstSequence + public func dropLast(_ k: Int = 1) -> [Element] + public func suffix(_ maxLength: Int) -> [Element] + public func prefix(_ maxLength: Int) -> PrefixSequence + public func drop(while predicate: (Element) throws -> Bool) rethrows -> DropWhileSequence + public func prefix(while predicate: (Element) throws -> Bool) rethrows -> [Element] + public func split( + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true, + whereSeparator isSeparator: (Element) throws -> Bool + ) rethrows -> [ArraySlice] +} +``` + +`DropFirstSequence`, `DropWhileSequence` and `PrefixSequence` already exist in +the standard library in underscored form. + +This will also have the useful side-effect that these methods can also be +removed as customization points from `Collection` as well, similar to removing +`prefix(upTo:)` in +[SE-0232](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0232-remove-customization-points.md), +because there’s no longer any reasonable customization to be done on a per-collection basis. + +Doing this will have considerable benefits to code size as well. For example, +the CoreAudio overlay that declares a handful of collections reduces in size by +25% after applying these changes. + +Once done, this change will allow the following simplifying changes and +enhancements to standard library types: + +- `LazySequenceProtocol` and `LazyCollectionProtocol` can be collapsed into a + single protocol. + +- The following types can be collapsed, dropping the collection variant (with a + typealias provided for source compatibility): + + - `FlattenSequence` and `-Collection` + - `LazySequence` and `-Collection` + - `LazyMapSequence` and `-Collection` + - `LazyFilterSequence` and `-Collection` + - `LazyDropWhileSequence` and `-Collection` + - `LazyPrefixWhileSequence` and `-Collection` + +- The following types can be extended to support `Collection`: + + - `EnumeratedSequence` + - `JoinedSequence` + - `Zip2Sequence` + +`SubSequence` will continue to be an associated type on `Collection`, and the +equivalent take/drop methods (and split) will continue to return it. Once +the methods are removed from the `Sequence` protocol, they will also no longer +need to be customizable at the `Collection` level so can be extensions only. + +## Source compatibility + +This is a source-breaking change, in that any code that relies on +`Sequence.SubSequence` will no longer work. For example: + +```swift +extension Sequence { + func dropTwo() -> SubSequence { + return dropFirst(2) + } +} +``` + +There are no examples of this kind of code in the compatibility suite. +Unfortunately there is no way to remove an associated type in a way that only +affects a specific language version, so this would not be something you could +handle as part of upgrading to 5.0. + +Additionally, any sequences that define a custom `SubSequence` of their own +will no longer work. This is really a non-problem, because doing so is almost +impossible (it means you even have to implement your own `split`). + +## Effect on ABI stability + +This is an ABI-breaking change, so must happen before 5.0 is released. + +## Alternatives considered + +Other than not doing it, a change that split `Sequence.SubSequence` into two +(say `Prefix` and `Suffix`) was considered. This implementation added +significant complexity to the standard library, impacting compile time and code +size, without being a significant enough improvement over the current situation. diff --git a/proposals/0235-add-result.md b/proposals/0235-add-result.md new file mode 100644 index 0000000000..f4b979f6f7 --- /dev/null +++ b/proposals/0235-add-result.md @@ -0,0 +1,365 @@ +# Add Result to the Standard Library + +* Proposal: [SE-0235](0235-add-result.md) +* Author: [Jon Shier](https://github.com/jshier) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#21073](https://github.com/apple/swift/pull/21073), + [apple/swift#21225](https://github.com/apple/swift/pull/21225), + [apple/swift#21378](https://github.com/apple/swift/pull/21378) +* Review: ([initial review](https://forums.swift.org/t/se-0235-add-result-to-the-standard-library/17752)) ([second review](https://forums.swift.org/t/revised-se-0235-add-result-to-the-standard-library/18371)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0235-add-result-to-the-standard-library/18603)) + +## Introduction + +Swift's current error-handling, using `throws`, `try`, and `catch`, offers automatic and synchronous handling of errors through explicit syntax and runtime behavior. However, it lacks the flexibility needed to cover all error propagation and handling in the language. `Result` is a type commonly used for manual propagation and handling of errors in other languages and within the Swift community. Therefore this proposal seeks to add such a type to the Swift standard library. + +## Motivation + +Swift's [Error Handling Rationale and Proposal](https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst) document lays out the reasoning behind and high level details of the current Swift error handling story. Types conforming to `Error` can be propagated and handled using the `do` `try` `catch` `throw` syntax. The rationale document refers to this model as a typed and automatically propagating error system. However, this system has several drawbacks, some of which are mentioned in the rationale document. Namely, it cannot compose with asynchronous work, more complex error handling, or with failure values which don't conform to `Error`. The added flexibility of typed, marked, but manually propagating error type can address these shortcomings. Namely `Result`. + +## Proposed solution + +```swift +public enum Result { + case success(Success), failure(Failure) +} +``` + +`Result` is a pragmatic compromise between competing error handling concerns both present and future. + +The `Failure` type captured in the `.failure` case is constrained to `Error` to simplify and underline `Result`'s intended use for manually propagating the result of a failable computation. + +### Usage + +#### Asynchronous APIs + +Most commonly, and seen in abundance when using Apple or Foundation APIs, `Result` can serve to unify the awkwardly disparate parameters seen in asynchronous completion handlers. For instance, `URLSession`'s completion handlers take three optional parameters: + +```swift +func dataTask(with url: URL, completionHandler: @escaping (Data?, URLResponse?, Error?) -> Void) -> URLSessionDataTask +``` + +This can make it quite difficult to elegantly consume the results of these APIs: + +```swift +URLSession.shared.dataTask(with: url) { (data, response, error) in + guard error == nil else { return self.handleError(error!) } + + guard let data = data, let response = response else { return // Impossible? } + + handleResponse(response, data: data) +} +``` + +While this code is only a few lines long, it exposes Swift's complete lack of automatic error handling for asynchronous APIs. Not only was the `error` forcibly unwrapped (or perhaps handled using a slightly less elegant `if` statement), but a possibly impossible scenario was created. What happens if `response` or `data` are `nil`? Is it even possible? It shouldn't be, but Swift currently lacks the ability to express this impossibility. Using `Result` for the same scenario allows for much more elegant code: + +```swift +URLSession.shared.dataTask(with: url) { (result: Result<(response: URLResponse, data: Data), Error>) in // Type added for illustration purposes. + switch result { + case let .success(success): + handleResponse(success.response, data: success.data) + case let .error(error): + handleError(error) + } +} +``` + +This API expresses exactly the intended result (either an error or data and response, never all or none) and allows them to be handled much more clearly. + +#### More General Usage + +More generally, there are several scenarios in which `Result` can make error handling and the surrounding APIs more elegant. + +#### Delayed Handling + +There may be times where a developer wants to immediately execute a `throw`ing function but otherwise delay handling the error until a later time. Currently, if they wish to preserve the error, they must break the value and error apart, as is typically seen in completion handlers (see above). This is made even more obnoxious if the developer needs to store the results of more than one function. + +```swift +// Properties +var configurationString: String? +var configurationReadError: Error? + +do { + string = try String(contentsOfFile: configuration) +} catch { + readError = error +} + +// Sometime later... + +func doSomethingWithConfiguration() { + guard let configurationString = configurationString else { handle(configurationError!) } + + ... +} + +``` + +This can be made somewhat cleaner by using a `(string: String?, error: Error?)` to store the result, but it would still have the same usage issues as in the asynchronous case. Using a `Result` is the appropriate answer here, especially with the convenience API for creating a result from a `throw`ing function. + +```swift +let configuration = Result { try String(contentsOfFile: configuration) } + +// Sometime later... + +func doSomethingWithConfiguration() { + switch configuration { + ... + } +} + +``` + +#### Separating Errors + +It's occasionally useful to be able to run `throw`able functions in such way as to allow the developer to disambiguate between the sources of the errors, especially if the errors don't contain the information necessary to do so, or the developer doesn't want to implement such a check. For instance, if we needed to disambiguate between the errors possible when reading files: + +```swift +do { + handleOne(try String(contentsOfFile: oneFile)) +} catch { + handleOneError(error) +} + +do { + handleTwo(try String(contentsOfFile: twoFile)) +} catch { + handleTwoError(error) +} + +do { + handleThree(try String(contentsOfFile: threeFile)) +} catch { + handleThreeError(error) +} +``` +This case can be expressed much more clearly using `Result`: + +```swift +let one = Result { try String(contentsOfFile: oneFile) } +let two = Result { try String(contentsOfFile: twoFile) } +let three = Result { try String(contentsOfFile: threeFile) } + +handleOne(one) +handleTwo(two) +handleThree(three) +``` +Additional convenience API on `Result` could make many of these cases even more elegant. + +## Detailed design + +As implemented in the PR (annotations pending): + +```swift +/// A value that represents either a success or a failure, including an +/// associated value in each case. +@_frozen +public enum Result { + /// A success, storing a `Success` value. + case success(Success) + + /// A failure, storing a `Failure` value. + case failure(Failure) + + /// Returns a new result, mapping any success value using the given + /// transformation. + /// + /// Use this method when you need to transform the value of a `Result` + /// instance when it represents a success. The following example transforms + /// the integer success value of a result into a string: + /// + /// func getNextInteger() -> Result { /* ... */ } + /// + /// let integerResult = getNextInteger() + /// // integerResult == .success(5) + /// let stringResult = integerResult.map({ String($0) }) + /// // stringResult == .success("5") + /// + /// - Parameter transform: A closure that takes the success value of this + /// instance. + /// - Returns: A `Result` instance with the result of evaluating `transform` + /// as the new success value if this instance represents a success. + public func map( + _ transform: (Success) -> NewSuccess + ) -> Result { } + + /// Returns a new result, mapping any failure value using the given + /// transformation. + /// + /// Use this method when you need to transform the value of a `Result` + /// instance when it represents a failure. The following example transforms + /// the error value of a result by wrapping it in a custom `Error` type: + /// + /// struct DatedError: Error { + /// var error: Error + /// var date: Date + /// + /// init(_ error: Error) { + /// self.error = error + /// self.date = Date() + /// } + /// } + /// + /// let result: Result = // ... + /// // result == .failure() + /// let resultWithDatedError = result.mapError({ e in DatedError(e) }) + /// // result == .failure(DatedError(error: , date: )) + /// + /// - Parameter transform: A closure that takes the failure value of the + /// instance. + /// - Returns: A `Result` instance with the result of evaluating `transform` + /// as the new failure value if this instance represents a failure. + public func mapError( + _ transform: (Failure) -> NewFailure + ) -> Result { } + + /// Returns a new result, mapping any success value using the given + /// transformation and unwrapping the produced result. + /// + /// - Parameter transform: A closure that takes the success value of the + /// instance. + /// - Returns: A `Result` instance with the result of evaluating `transform` + /// as the new failure value if this instance represents a failure. + public func flatMap( + _ transform: (Success) -> Result + ) -> Result { } + + /// Returns a new result, mapping any failure value using the given + /// transformation and unwrapping the produced result. + /// + /// - Parameter transform: A closure that takes the failure value of the + /// instance. + /// - Returns: A `Result` instance, either from the closure or the previous + /// `.success`. + public func flatMapError( + _ transform: (Failure) -> Result + ) -> Result { } + + /// Returns the success value as a throwing expression. + /// + /// Use this method to retrieve the value of this result if it represents a + /// success, or to catch the value if it represents a failure. + /// + /// let integerResult: Result = .success(5) + /// do { + /// let value = try integerResult.get() + /// print("The value is \(value).") + /// } catch error { + /// print("Error retrieving the value: \(error)") + /// } + /// // Prints "The value is 5." + /// + /// - Returns: The success value, if the instance represents a success. + /// - Throws: The failure value, if the instance represents a failure. + public func get() throws -> Success { } +} + +extension Result where Failure == Swift.Error { + /// Creates a new result by evaluating a throwing closure, capturing the + /// returned value as a success, or any thrown error as a failure. + /// + /// - Parameter body: A throwing closure to evaluate. + @_transparent + public init(catching body: () throws -> Success) { } +} + +extension Result: Equatable where Success: Equatable, Failure: Equatable { } + +extension Result: Hashable where Success: Hashable, Failure: Hashable { } +``` + +## Adding `Swift.Error` self-conformance + +As part of the preparatory work for this proposal, self-conformance was added for `Error` (and only `Error`). This is also generally useful for working with errors in a generic context. + +This self-conformance does not extend to protocol compositions including the `Error` protocol, only the exact type `Error`. It will be possible to add such compositions in the future, but that is out of scope for Swift 5. + +## Other Languages +Many other languages have a `Result` type or equivalent: + +- Kotlin: [`Result`](https://github.com/Kotlin/KEEP/blob/master/proposals/stdlib/result.md) +- Scala: [`Try[T]`](https://www.scala-lang.org/api/current/scala/util/Try.html) +- Rust: [`Result`](https://doc.rust-lang.org/std/result/) +- Haskell: [`Exceptional e t`](https://wiki.haskell.org/Exception) +- C++ (proposed): [`expected`](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4015.pdf) + +## Source compatibility + +This is an additive change which could conflict with existing `Result` types, but work done to improve type name shadowing has made it a non-issue. + +## Effect on ABI stability + +This proposal adds a type to the standard library and so will affect the ABI once added. + +## Effect on API resilience + +Addition of `Result` should be future proof against additional needs surrounding error handling. + +## Alternatives considered + +### Alternative Spellings of `Result` +A few alternate spellings were proposed: + +A previously revised version of this proposal proposed: +```swift +enum Result { + case value(Value) + case error(Error) +} +``` + +However, community opposition to this spelling resulted in the current, final spelling. + +Additionally, a hybrid spelling was proposed: + +```swift +enum Result { + case success(Value) + case failure(Error) +} +``` + +This creates an unfortunate asymmetry between the names of the payload types and the names of the cases. This is additional complexity that has to be remembered. + +Finally, spellings using `Wrapped` as the success type name were proposed: + +```swift +enum Result { + case value(Wrapped) + case error(Failure) +} +``` +```swift +enum Result { + case some(Wrapped) + case error(Failure) +} +``` + +The use of `Wrapped` in these spellings emphasizes more of a similarity to `Optional` than seems appropriate. + +### Alternatives to `Result` + +- `Result`: A `Result` without a generic error type fits well with the current error design in Swift. However, it prevents the future addition of typed error handling (typed `throws`). + +- `Either`: Rather than adopting `Result` directly, basing it on an `Either` type has been considered. However, it's felt that a `Result` type is a more generally useful case of `Either`, and `Either` gives users little in the way of actual API. Also, the two types can exist peacefully alongside each other with little conflict. Additionally, given the relatively unpopularity of the `Either` type in the community (59 apps use the [Either](https://github.com/runkmc/either) CocoaPod) seems to indicate that lacking this type isn't affecting Swift users that much. + +### Constraint on the `Error` type + +A previous version of this proposal did not constrain the `Failure` type to conform to `Error`. This was largely because adding such a requirement would block `Error` from being used as the `Error` type, since `Error` does not conform to itself. Since we've now figured out how how to make that conformance work, this constraint is unblocked. + +Constraining the error type to conform to `Error` is a very low burden, and it has several benefits: + +- It simplifies interoperation with error handling by making such operations unconditionally available. + +- It encourages better practices for error values, such as using meaningful wrapper types for errors instead of raw `Int`s and `String`s. + +- It immediately catches the simple transposition error of writing `Result`. Programmers coming from functional languages that use `Either` as a `Result` type are especially prone to this mistake: such languages often write the error type as the first argument due to the useful monadic properties of `Either E`. + +### Operations + +A previous version of this proposal included operations for optionally projecting out the `value` and `error` cases. These operations are useful, but they should be added uniformly for all `enum`s, and `Result` should not commit to providing hard-coded versions that may interfere with future language evolution. In the meantime, it is easy for programmers to add these operations with extensions in their own code. + +A previous version of this proposal included a `fold` operation. This operation is essentially an expression-based `switch`, and like the optional case projections, it would be better to provide a general language solution for it than to add a more limited design that covers only a single type. + +A previous version of this proposal did not label the closure parameter for the catching initializer. Single-argument unlabeled initializers are conventionally used for conversions, which this is not; usually the closure will be written explicitly, but in case it isn't, a parameter label is appropriate. diff --git a/proposals/0236-package-manager-platform-deployment-settings.md b/proposals/0236-package-manager-platform-deployment-settings.md new file mode 100644 index 0000000000..afde4c7e14 --- /dev/null +++ b/proposals/0236-package-manager-platform-deployment-settings.md @@ -0,0 +1,121 @@ +# Package Manager Platform Deployment Settings + +* Proposal: [SE-0236](0236-package-manager-platform-deployment-settings.md) +* Authors: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Boris Bügling](https://github.com/neonichu) +* Status: **Implemented (Swift 5.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0236-package-manager-platform-deployment-settings/18420) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/aebb22c0f3e139fd921d14f79c3945af99d0342d/proposals/0236-package-manager-platform-deployment-settings.md) + +## Introduction + +This is a proposal for adding support for specifying a per-platform minimum required deployment target in the `Package.swift` manifest file. + +## Motivation + +Packages should be able to declare the minimum required platform deployment target version. SwiftPM currently uses a hardcoded value for the macOS deployment target. This creates friction for packages which want to use APIs that were introduced after the hardcoded deployment target version. + +There are two ways to work around this limitation: 1) using availability checks, or 2) passing the deployment target on the command line while building the package. However, these workarounds are not ideal and the package manager should really provide a proper API for setting the required deployment target version. + +## Proposed solution + +We propose to add the following API to `PackageDescription`: + +```swift +/// Represents a supported platform. +struct SupportedPlatform { + static func macOS(_ version: MacOSVersion) -> SupportedPlatform + static func macOS(_ versionString: String) -> SupportedPlatform + + static func tvOS(_ version: TVOSVersion) -> SupportedPlatform + static func tvOS(_ versionString: String) -> SupportedPlatform + + static func iOS(_ version: IOSVersion) -> SupportedPlatform + static func iOS(_ versionString: String) -> SupportedPlatform + + static func watchOS(_ version: WatchOSVersion) -> SupportedPlatform + static func watchOS(_ versionString: String) -> SupportedPlatform +} + +/// List of known versions. +extension SupportedPlatform.MacOSVersion { + static let v10_10: MacOSVersion + static let v10_11: MacOSVersion + static let v10_12: MacOSVersion + ... +} + +final class Package { + + init( + name: String, + platforms: [SupportedPlatform]? = nil, + ... + ) +} + +// Example usage: + +let package = Package( + name: "NIO", + platforms: [ + .macOS(.v10_13), .iOS(.v12), + ], + products: [ + .library(name: "NIO", targets: ["NIO"]), + ], + targets: [ + .target(name: "NIO"), + ] +) +``` + +A package will be assumed to support all platforms using a predefined minimum deployment version. This predefined deployment version will be the oldest deployment target version supported by the installed SDK for a given platform. One exception to this rule is macOS, for which the minimum deployment target version will start from 10.10. Packages can choose to declare the minimum deployment target version for some platform by using the above APIs. For example: + +* This declaration means that the package should use 10.13 on macOS, 12.0 on iOS and the default deployment target when compiled on other platforms: + +```swift + ... + platforms: [ + .macOS(.v10_13), .iOS(.v12), + ], + ... +``` + +## Detailed design + +Changes in deployment target versions should be considered as a major breaking change for the purposes of semantic versioning since the dependees of a library package can break when a library makes such a change. + +SwiftPM will emit an error if a dependency is not compatible with the top-level package's deployment version, i.e., the deployment target of dependencies must be lower than or equal to top-level package's deployment target version for a particular platform. + +Each package will be compiled with the deployment target specified by it. In theory, SwiftPM can use the top-level package's deployment version to compile the entire package graph since the deployment target versions of dependencies are guaranteed to be compatible. This might even produce more efficient compilation output but it also means that the users might start seeing a lot of warnings due to use of a higher version. + +Each platform API has a string overload that can be used to construct versions that are not already provided by PackageDescription APIs. This could be because the new version was recently released or isn't appropriate to be included in the APIs (for e.g. dot versions). The version format for each platform will be documented in the API. Invalid values will be diagnosed and presented as manifest parsing errors. + +SwiftPM will emit appropriate errors when an invalid value is provided for supported platforms. For e.g., an empty array, multiple declarations for the same platform, invalid version specification. + +The generated Xcode project command will set the deployment target version for each platform. + +### Future directions + +The Swift compiler supports several platforms like macOS, iOS, Linux, Windows, Android. However, the runtime availability checks currently only work for Apple platforms. It is expected that support for more platforms in the availability APIs will be added gradually as the community and support for Swift compiler grows. We think that the package manager can use a similar direction for declaring the supported platforms. We can start by allowing packages to declare support for Apple platforms with version specifications as proposed by the above APIs. Depending on the need in the community, these APIs can be evolved over time by adding support for declaring the minimum deployment version for other platforms that Swift supports. For e.g., the API can be enhanced to declare the minimum required version for Windows and the API level for Android. + +This proposal doesn't handle these problems: + +1. **Restricting supported platforms**: Some packages are only meant to work on certain platforms. This proposal doesn't provide ability to restrict the list of supported platforms and only allows customizing the deployment target versions. Restricting supported platforms is orthogonal to customizing deployment target and will be explored separately. + +2. **Platform-specific targets or products**: Consider that a package supports multiple platforms but has a product that should be only built for Linux. There is no way to express this intent and the build system may end up trying to build such targets on all platforms. A simple workaround is to use `#if` to conditionalize the source code of the target. Another use case comes up when you want to keep deployment target of a certain target lower than other targets in a package. A workaround is factoring out the target into its own package. A proper solution to this problem will be explored in a separate proposal, which should provide target-level settings. + +3. **Platform-specific package dependencies**: Similar to the above issue, a package may want to use a certain dependency only when building for a specific platform. It is currently not possible to declare such a dependency without using `#if os` checks in the manifest file, which doesn't interact nicely with other features like `Package.resolved` and the cross-compilation support. This can be solved by providing APIs to declare platform-specific package dependencies in the manifest file. This will also be explored in a separate proposal. + +## Impact on existing packages + +Existing packages will not be impacted as the behavior described in this proposal is compatible with SwiftPM's current behavior, i.e., a package is assumed to support all platforms and SwiftPM picks a default deployment target version for the macOS platform. + +The new APIs will be guarded against the tools version this proposal is implemented in. Packages that want to use this feature will need to update their tools version. + +## Alternatives considered + +We considered making the supported platform field mandatory. We think that it would provide little value in practice and cause more friction instead. One advantage would be that SwiftPM could use that information to produce error messages when a package tries to use a dependency which is not tested on some platform. We think that is not really a big enough issue. Since Swift is cross-platform, Swift packages should generally work on all platforms that Swift supports. However, there are platform-specific packages that are only meant for certain platforms and the proposed API does provide an option to declare that intent. + +We considered taking the deployment target version into the dependency resolution process to automatically find a compatible version with the top-level package. This too doesn't provide enough value in practice. Library packages generally tend to stick with a deployment target to avoid breaking their dependees. When they do bump the version, it is generally a well-thought decision and may require a semver upgrade anyway due to updates in the API. diff --git a/proposals/0237-contiguous-collection.md b/proposals/0237-contiguous-collection.md new file mode 100644 index 0000000000..4bd00aa477 --- /dev/null +++ b/proposals/0237-contiguous-collection.md @@ -0,0 +1,136 @@ +# Introduce `withContiguous{Mutable}StorageIfAvailable` methods + +* Proposal: [SE-0237](0237-contiguous-collection.md) +* Author: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#21138](https://github.com/apple/swift/pull/21138) +* Decision notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0237-introduce-with-contiguous-mutable-storage-if-available-methods/18713) +* Prior revisions: [Version 1](https://github.com/swiftlang/swift-evolution/commit/be787dee0732895d35e0aba8f2f69d1f310b4e99) + +## Introduction + +This proposal introduces two new methods, on `Sequence` and +`MutableCollection`. These methods will allow generic code to make use of the +`withUnsafe{Mutable}BufferPointer` idiom, as well as provide fast paths in the +standard library for adopting types. + +Swift-evolution thread: [Contiguous Collection Protocols](https://forums.swift.org/t/contiguous-collection-protocols/17875) + +## Motivation + +Almost every feature of `Array` is made available via one of the protocols +in the standard library, and so most code written against `Array` can be +rewritten generically as an extension of one or more protocols. + +The exceptions to this are the operations `withUnsafeBufferPointer` and +`withUnsafeMutableBufferPointer`, which are only available on the concrete +types. Given the usefulness of these methods, they should also be made +available generically. + +In addition, it is common to be able to provide an optimized fast path +for many operations that are generic over `Sequence` or `MutableCollection`, +when an unsafe buffer is available. For example, initializing a `Data` from +a `[UInt8]` should be a memcpy when calling the generic initializer +from a `Sequence where Element == UInt8`. `sort` currently is implemented +as a merge sort, and the movements to/from its auxiliary storage can be +done using memory moves for non-trivial types when the sorted collection +is contiguously stored. + +## Proposed solution + +Introduce two new methods, providing access to the with-unsafe +capabilities of `Array` & co when operating generically +on protocols: + +```swift +protocol Sequence { + /// Call `body(p)`, where `p` is a pointer to the collection's + /// contiguous storage. If no such storage exists, it is + /// first created. If the collection does not support an internal + /// representation in a form of contiguous storage, `body` is not + /// called and `nil` is returned. + /// + /// A `Collection` that provides its own implementation of this method + /// must also guarantee that an equivalent buffer of its `SubSequence` + /// can be generated by advancing the pointer by the distance to the + /// slice's `startIndex`. + func withContiguousStorageIfAvailable( + _ body: (UnsafeBufferPointer) throws -> R + ) rethrows -> R? +} + +protocol MutableCollection { + /// Call `body(p)`, where `p` is a pointer to the collection's + /// mutable contiguous storage. If no such storage exists, it is + /// first created. If the collection does not support an internal + /// representation in a form of mutable contiguous storage, `body` is not + /// called and `nil` is returned. + /// + /// A `Collection` that provides its own implementation of this method + /// must also guarantee that an equivalent buffer of its `SubSequence` + /// can be generated by advancing the pointer by the distance to the + /// slice's `startIndex`. + public mutating func withContiguousMutableStorageIfAvailable( + _ body: (inout UnsafeMutableBufferPointer) throws -> R + ) rethrows -> R? +} +``` + +Existing types (such as `Array` or `ArraySlice`) that currently support +`withUnsafe{MutableBuffer}Pointer` will forward on to this method. The default +implementations will return `nil`. Additionally, `Unsafe{Mutable}BufferPointer` +will respond using`self`, and `Slice` will provide an implementation that +forwards on to its `Base`. + +A customization point already exists with an underscore in the standard library +for the mutable version (it just returns `nil` by default), and should be +exposed to general users. In addition, there exist underscored +customization points that could be replaced by the immutable variant. + +There are no guarantees made by the mutable version about the state left behind +if the closure throws during mutation. The updates made may or may not be +reflected in the collection (which might have given a direct pointer to its +internal storage, or could have handed out a temporary buffer that it then does +not write back after the error is thrown). The closure should always perform +any cleanup it thinks is necessary itself. + +It should be documented that successive calls to +`withUnsafe{Mutable}BufferPointer` are not guaranteed to give you the same +pointer with each call. For example, a packed small string implementation may +be giving you a pointer to that string temporarily expanded to a buffer. + +Use of this entry point can provide significant speedups in some +algorithms, e.g. our current +[`sort`](https://github.com/apple/swift/blob/6662ccc16dba27418eefd3cb7856bddda5a33386/stdlib/public/core/Sort.swift#L249) +which needs to move elements of a collection back and forth between +some storage. + +## Source compatibility + +These are additive changes and do not affect source compatibility. + +## Effect on ABI stability + +These are additive changes and so can be done without affecting ABI stability. +However, some existing underscored entry points could be altered as a result +if done before ABI stability is declared. + +## Alternatives considered + +This proposal originally introduced two new protocols: `ContiguouslyStored` +and `MutableContiguouslyStored`. The introduction of customization points +lower in the stack mean that these protocols are not necessary. It may be +that valid use cases for asserting contiguity at compile time exist, in +which case these protocols could be re-proposed. + +Some collections are not fully contiguous, but instead consist of multiple +contiguous regions (for example, a ring buffer is one or two separate +contiguous regions). Protocols or methods that handle this situation are +left to subsequent proposals. + +The `inout` argument to the closure in the mutating variant is debatable. It +does imply the user can change the buffer to a totally different one. +Nonetheless, this is better handled in documentation, since the improved +ergonomics of the `inout` version are considerable. It would also be a +source-breaking change to alter `Array`'s implementation at this point. diff --git a/proposals/0238-package-manager-build-settings.md b/proposals/0238-package-manager-build-settings.md new file mode 100644 index 0000000000..721434b15e --- /dev/null +++ b/proposals/0238-package-manager-build-settings.md @@ -0,0 +1,172 @@ +# Package Manager Target Specific Build Settings + +* Proposal: [SE-0238](0238-package-manager-build-settings.md) +* Decision Notes: [Draft Thread](https://forums.swift.org/t/draft-proposal-target-specific-build-settings/18031) +* Authors: [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Boris Bügling](https://github.com/neonichu) +* Status: **Implemented (Swift 5.0)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0238-package-manager-target-specific-build-settings/18590) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/1a2801a3dc912b093f2cda13eafd54f0d98b3c8e/proposals/0238-package-manager-build-settings.md) + +## Introduction + +This is a proposal for adding support for declaring some commonly used target-specific build settings in the `Package.swift` manifest file. As the name suggests, target-specific build settings are only applied to a particular target. SwiftPM also aims to support cross-target build settings that go across the target boundary and impart certain settings on a target's dependees, but this proposal is only concerned with the former type of build settings and the latter will be explored with a future proposal. + +## Motivation + +SwiftPM currently has little facility for customizing how the build tools (compilers, linker, etc.) are invoked during a build. This causes a lot of friction for package authors who want to do some basic customizations in order to build their targets. They often have to resort to awkward workarounds like creating custom modulemaps for linking system libraries, symlinking private headers inside the include directory, changing the include statements, and so on. + +We think most of these workarounds can be removed by providing support for some common build settings at the target level. This proposal will also set the stage for a richer build settings API in the future that has support for various conditional expressions, deployment options, inheritance of build settings, etc. + +## Proposed solution + +We propose to add four new arguments to the target factory method: `cSettings`, `cxxSettings`, `swiftSettings` and `linkerSettings`. The build settings specified in these arguments will be used to compile a particular target and the settings will not affect any other target in the package or the package graph. The API will also allow conditionalization using a `.when` modifier on two parameters: platforms and build configuration. + +We propose to add the following build settings in this proposal: + +*Note: `` represents the concrete type of a certain setting. Possible types are `CSetting`, `CXXSetting`, `SwiftSetting` or `LinkerSetting`. Each build setting in the upcoming section contains the method signature that will be available in their corresponding .* + +### Header search path (C/CXX) + +```swift +static func headerSearchPath(_ path: String, _ condition: BuildSettingCondition? = nil) -> +``` + +Many C-family projects are structured in a way that requires adding header search paths to different directories of the project. Currently, SwiftPM only adds a search path to the `include` directory which makes it difficult for many C projects to add support for building with SwiftPM. This specified path should be relative to the target and should not escape the package boundary. Absolute paths are disallowed. + +*Note: It is not recommended to use this setting for adding search paths of public headers as the target-specific settings are not imparted onto other targets.* + +### Define (C/CXX) + +```swift +static func define(_ name: String, to value: String? = nil, _ condition: BuildSettingCondition? = nil) -> +``` + +This setting will add the `-D=` flag during a target's compilation. This is useful for projects that want to specify a compile-time condition. + +*Note: It is not recommended to use this setting for public headers as the target-specific settings are not imparted.* + +### Define (Swift) + +```swift +static func define(_ name: String, _ condition: BuildSettingCondition? = nil) -> SwiftSetting +``` + +This setting enables the specified compilation condition for Swift targets. Unlike C/CXX's define, it doesn't have an associated value. + +### Link library (Linker) + +```swift +static func linkedLibrary(_ libraryName: String, _ condition: BuildSettingCondition? = nil) -> +``` + +This is useful for packages that want to link against a library present in the system. The current approach requires them to create a module map using system library targets or a fake C target in order to achieve this effect. There is also no provision for conditionalization based on the platform in the existing approach, which is valuable when writing cross-platform packages. + +### Link framework (Linker) + +```swift +static func linkedFramework(_ frameworkName: String, _ condition: BuildSettingCondition? = nil) -> +``` + +Frameworks are autolinked for Swift and C/ObjC targets so most packages shouldn't require this build setting. However, packages that contain C++ files can't autolink the frameworks. Since frameworks are widely used on Apple platforms, it is recommended to use this setting with a platform conditional. + +### Unsafe flags (All) + +```swift +static func unsafeFlags(_ flags: [String], _ condition: BuildSettingCondition? = nil) -> +``` + +This is an escape hatch that will allow targets to pass arbitrary command-line flags to the corresponding build tool. The "unsafe" here implies that SwiftPM can't safely determine if the build flags will have any negative side-effect to the build since certain flags can change the behavior of how a build is performed. It is similar to how the `-Xcc`, `-Xswiftc`, `-Xlinker` option work in the command-line SwiftPM tools. + +The primary purpose of this escape hatch is to enable experimentation and exploration for packages that currently use a makefile or script to pass the `-X*` flags. Products that contain a target which uses an unsafe flag will be ineligible to act as a dependency for other packages. + +We have several such conditions (use of local dependencies, branch-based dependencies etc) that makes a package (individual products in this case) ineligible for acting as a dependency. This feature would be one more in that category. SwiftPM could provide a "pre-publish" command to detect and report such cases. RFC: https://forums.swift.org/t/rfc-swift-package-publish-precheck/15398 + +### Conditionalization + +`static func when(platforms: [Platform]? = nil, configuration: BuildConfiguration? = nil) -> BuildSettingCondition` + +By default, build settings will be applicable for all platforms and build configurations. The `.when` modifier can be used to conditionalize a build setting. SwiftPM will diagnose invalid usage of `.when` and emit a manifest parsing error. For e.g., it is invalid to specify a when condition with both parameter as `nil`. + +### Example + +Here is an example usage of the proposed APIs: + +```swift +... +.target( + name: "MyTool", + dependencies: ["Yams"], + cSettings: [ + .define("BAR"), + .headerSearchPath("path/relative/to/my/target"), + + .define("DISABLE_SOMETHING", .when(platforms: [.iOS], configuration: .release)), + .define("ENABLE_SOMETHING", .when(configuration: .release)), + + // Unsafe flags will be rejected by SwiftPM when a product containing this + // target is used as a dependency. + .unsafeFlags(["-B=imma/haxx0r"]), + ], + swiftSettings: [ + .define("API_VERSION_5"), + ], + linkerSettings: [ + .linkLibrary("z"), + .linkFramework("CoreData"), + + .linkLibrary("openssl", .when(platforms: [.linux])), + .linkFramework("CoreData", .when(platforms: [.macOS], configuration: .debug)), + + // Unsafe flags will be rejected by SwiftPM when a product containing this + // target is used as a dependency. + .unsafeFlags(["-L/path/to/my/library", "-use-ld=gold"], .when(platforms: [.linux])), + ] +), +... +``` + +## Detailed design + +#### Use of a declarative model + +Using a declarative model for build settings (and, in general, all PackageDescription APIs) allows SwiftPM to understand the complete package manifest, including the conditionals that may currently evaluate to false. This information can be used to build some advanced features like mechanically editing the manifest file and it also allows possibility for a “migrator” feature for upgrading the APIs as they evolve. + +It is important to consider the impact of each build setting that is allowed to be used in a package. Certain build flags can be unsafe when configured without a more expressive build settings model, which can lead to non-hermetic builds. They can also cause bad interaction with the compilation process as certain flags can have a large impact on how the build is performed (for e.g. Swift compiler's `-wmo`). Some flags can even be exploited to link pre-compiled binaries without being officially supported by the package manager, which can be a huge security issue. Other flags (like `-B`) can be used to change the directory where the tools are looked up by the compiler. In the future, we can enhance the build system to perform builds in a highly sandboxed environment and potentially loosen the restrictions from unsafe flags as such vulnerabilities will no longer be possible. The package author will immediately run into build errors in such a sandbox. + +#### Sharing build settings between tools + +If a Swift target specifies both Swift and C settings, the flags produced by C settings will be added to Swift compiler by prefixing each flag with `-Xcc`. Similarly, if a C-family target specifies both C and CXX settings, the flags produced by C settings will be added to C++ compiler by prefixing the flags with `-Xcc`. This behavior is similar to what the command-line SwiftPM does for the `-X*` overrides. This strategy doesn't allow passing C flags that should be only passed to the C++ compiler but that is a very rare case. + +## Future direction + +One of the major goal of this proposal is to introduce the infrastructure for build settings with some frequently used ones. There are many other build settings that can be safely added to the proposed API. Such additional build settings can be explored in a separate proposal. + +In the long term, SwiftPM aims to have a more complex build settings model with a rich API that allows expressing conditionalization on the various parameter, macro expansion, etc. We believe that the experience we gain with the proposed API will help us in fleshing out the future build settings API. + +## Impact on existing packages + +There is no impact on existing packages as this is an additive feature. Packages that want to use the new build settings APIs will need to upgrade their manifest's tools version to the version this proposal implemented in. + +## Alternatives considered + +We considered making the API to be just an array of `String` that can take arbitrary build flags. However, that requires SwiftPM to implement parsing logic in order to determine if the flags are in the whitelist or not. The compiler flags are very difficult to parse and there are several variations accepted by the compiler for different flags. The other option was standardizing on the syntax of each whitelisted flag but that would require package authors to lookup SwiftPM's documentation to figure out which variation is accepted. + +We considered another spelling for the proposal API but rejected it because we expect that most packages that need to add a build settings will require only one of the four type. It seems unnecessary to have package authors do nesting in order to add a single flag. + +```swift +... +.target( + name: "foo", + dependencies: ["Yams"], + settings: [ + .swift([ + .define("BAR"), + ]), + .linker([ + .linkLibrary("z"), + ]), + ] +), +... +``` diff --git a/proposals/0239-codable-range.md b/proposals/0239-codable-range.md new file mode 100644 index 0000000000..2acdb63d64 --- /dev/null +++ b/proposals/0239-codable-range.md @@ -0,0 +1,111 @@ +# Add Codable conformance to Range types + +* Proposal: [SE-0239](0239-codable-range.md) +* Authors: [Dale Buckley](https://github.com/dlbuckley), [Ben Cohen](https://github.com/airspeedswift), [Maxim Moiseev](https://github.com/moiseev) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Implementation: [apple/swift#19532](https://github.com/apple/swift/pull/19532), + [apple/swift#21857](https://github.com/apple/swift/pull/21857) +* Status: **Implemented (Swift 5.0)** +* Review: [Discussion thread](https://forums.swift.org/t/se-0239-add-codable-conformance-to-range-types/18794/51) + + +## Introduction + +[SE-0167](0167-swift-encoders.md) introduced `Codable` conformance for some types in the standard +library, but not the `Range` family of types. This proposal adds that +conformance. + +Swift-evolution thread: [Range conform to Codable](https://forums.swift.org/t/range-conform-to-codable/15552) + +## Motivation + +`Range` is a very useful type to have conform to `Codable`. A good usage example is a range being sent to/from a client/server to convey a range of time using `Date`, or a safe operating temperature range using `Measurement`. + +## Proposed solution + +The following Standard Library range types will gain `Codable` conformance +when their `Bound` is also `Codable`: + + * `Range` + * `ClosedRange` + * `PartialRangeFrom` + * `PartialRangeThrough` + * `PartialRangeUpTo` + +These types will use an unkeyed container of their lower/upper bound (in sorted order, in cases of `Range` and `ClosedRange`). + +In addition, `ContiguousArray` is also missing a conformance to `Codable`, which will be added. + +## Effect on ABI stability, resilience, and source stability + +This is a purely additive change, and so has no impact. + +## Alternatives considered + +One area of concern mentioned during the discussion is that of +potential data corruption. Consider this: if an application implements its own +`Codable` conformance for `Range` or `ClosedRange` (which is, by the way, not +recommended, as both the protocol and the type are defined in the standard +library), there might be data permanently stored somewhere (in the database, in +a JSON or PLIST file, etc.) which was serialized using that conformance. Unless +the serialization format is exactly the same as the one used in the standard +library, that data will be considered invalid. + +When this proposal is implemented, any `Codable` conformance to the `Range` type +outside standard library will result in a compiler error as duplicate +conformance. At which point, we suggest the following course of action to avoid +data loss. + +1. Define your own type wrapping `Range`. + + ```swift + public struct MyRangeWrapper + where Bound: Comparable { + public var range: Range + } + ``` + +2. Port `Codable` conformance from `Range` to this new type. + + ```swift + extension MyRangeWrapper { + enum CodingKeys: CodingKey { + case lowerBound + case upperBound + } + } + + extension MyRangeWrapper: Decodable + where Bound: Decodable { + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + let lowerBound = try container.decode(Bound.self, forKey: .lowerBound) + let upperBound = try container.decode(Bound.self, forKey: .upperBound) + self.range = lowerBound ..< upperBound + } + } + + ``` + + Note that unless you wish to keep using this serialization format in the + future, and are OK with using what's provided by the standard library, only + the `Decodable` conformance is needed for data migration purposes. + +3. Support both old and new formats when deserializing your data that contains + ranges. + + ```swift + extension JSONDecoder { + func decodeRange( + _ type: Range.Type, from data: Data + ) throws -> Range + where Bound: Decodable { + do { + return try self.decode(Range.self, from: data) + } + catch DecodingError.typeMismatch(_, _) { + return try self.decode(MyRangeWrapper.self, from: data).range + } + } + } + ``` diff --git a/proposals/0240-ordered-collection-diffing.md b/proposals/0240-ordered-collection-diffing.md new file mode 100644 index 0000000000..b75c0a8448 --- /dev/null +++ b/proposals/0240-ordered-collection-diffing.md @@ -0,0 +1,307 @@ +# Ordered Collection Diffing + +* Proposal: [SE-0240](0240-ordered-collection-diffing.md) +* Authors: [Scott Perry](https://github.com/numist), [Kyle Macomber](https://github.com/kylemacomber) +* Review Manager: [Doug Gregor](https://github.com/DougGregor), [Ben Cohen](https://github.com/AirspeedSwift) +* Status: **Implemented (Swift 5.1)** +* Amendment status: **Implemented (Swift 5.1)** +* Implementation: [apple/swift#21845](https://github.com/apple/swift/pull/21845) +* Decision notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0240-ordered-collection-diffing/20008) + +## Introduction + +This proposal describes additions to the standard library that provide an interchange format for diffs as well as diffing/patching functionality for appropriate collection types. + +## Motivation + +Representing, manufacturing, and applying transactions between states today requires writing a lot of error-prone code. This proposal is inspired by the convenience of the `diffutils` suite when interacting with text files, and the reluctance to solve similar problems in code by linking `libgit2`. + +Many state management patterns would benefit from improvements in this area, including undo/redo stacks, generational stores, and syncing differential content to/from a service. + +## Proposed solution + +A new type representing the difference between collections is introduced along with methods that support its production and application. + +Using this API, a line-by-line three-way merge can be performed in a few lines of code: + +``` swift +// Split the contents of the sources into lines +let baseLines = base.components(separatedBy: "\n") +let theirLines = theirs.components(separatedBy: "\n") +let myLines = mine.components(separatedBy: "\n") + +// Create a difference from base to theirs +let diff = theirLines.difference(from:baseLines) + +// Apply it to mine, if possible +guard let patchedLines = myLines.applying(diff) else { + print("Merge conflict applying patch, manual merge required") + return +} + +// Reassemble the result +let patched = patchedLines.joined(separator: "\n") +print(patched) +``` + +## Detailed design + +### Producing diffs + +Most diffing algorithms have collection access patterns that are not appropriate for basic collections, so difference production is dependant on conformance to `BidirectionalCollection`: + +``` swift +@available(swift, introduced: 5.1) +extension BidirectionalCollection { + /// Returns the difference needed to produce the receiver's state from the + /// parameter's state, using the provided closure to establish equivalence + /// between elements. + /// + /// This function does not infer moves. + /// + /// - Parameters: + /// - other: The base state. + /// - areEquivalent: A closure that returns whether the two + /// parameters are equivalent. + /// + /// - Returns: The difference needed to produce the receiver's state from + /// the parameter's state. + /// + /// - Complexity: For pathological inputs, worst case performance is + /// O(`self.count` * `other.count`). Faster execution can be expected + /// when the collections share many common elements. + public func difference( + from other: C, by areEquivalent: (Element, C.Element) -> Bool + ) -> CollectionDifference + where C : BidirectionalCollection, C.Element == Self.Element +} + +extension BidirectionalCollection where Element: Equatable { + /// Returns the difference needed to produce the receiver's state from the + /// parameter's state, using equality to establish equivalence between + /// elements. + /// + /// This function does not infer element moves, but they can be computed + /// using `CollectionDifference.inferringMoves()` if desired. + /// + /// - Parameters: + /// - other: The base state. + /// + /// - Returns: The difference needed to produce the receiver's state from + /// the parameter's state. + /// + /// - Complexity: For pathological inputs, worst case performance is + /// O(`self.count` * `other.count`). Faster execution can be expected + /// when the collections share many common elements, or if `Element` + /// also conforms to `Hashable`. + public func difference(from other: C) -> CollectionDifference + where C: BidirectionalCollection, C.Element == Self.Element +``` + +The `difference(from:)` method produces an instance of a difference type, defined as: + +``` swift +/// A type that represents the difference between two collection states. +@available(swift, introduced: 5.1) +public struct CollectionDifference { + /// A type that represents a single change to a collection. + /// + /// The `offset` of each `insert` refers to the offset of its `element` in + /// the final state after the difference is fully applied. The `offset` of + /// each `remove` refers to the offset of its `element` in the original + /// state. Non-`nil` values of `associatedWith` refer to the offset of the + /// complementary change. + public enum Change { + case insert(offset: Int, element: ChangeElement, associatedWith: Int?) + case remove(offset: Int, element: ChangeElement, associatedWith: Int?) + } + + /// Creates an instance from a collection of changes. + /// + /// For clients interested in the difference between two collections, see + /// `BidirectionalCollection.difference(from:)`. + /// + /// To guarantee that instances are unambiguous and safe for compatible base + /// states, this initializer will fail unless its parameter meets to the + /// following requirements: + /// + /// 1) All insertion offsets are unique + /// 2) All removal offsets are unique + /// 3) All offset associations between insertions and removals are symmetric + /// + /// - Parameter changes: A collection of changes that represent a transition + /// between two states. + /// + /// - Complexity: O(*n* * log(*n*)), where *n* is the length of the + /// parameter. + public init?(_ c: C) where C.Element == Change + + /// The `.insert` changes contained by this difference, from lowest offset to highest + public var insertions: [Change] { get } + + /// The `.remove` changes contained by this difference, from lowest offset to highest + public var removals: [Change] { get } + + /// Produces a difference that is the functional inverse of `self` + public func inverse() -> CollectionDifference +} + +/// A CollectionDifference is itself a Collection. +/// +/// The enumeration order of `Change` elements is: +/// +/// 1. `.remove`s, from highest `offset` to lowest +/// 2. `.insert`s, from lowest `offset` to highest +/// +/// This guarantees that applicators on compatible base states are safe when +/// written in the form: +/// +/// ``` +/// for c in diff { +/// switch c { +/// case .remove(offset: let o, element: _, associatedWith: _): +/// arr.remove(at: o) +/// case .insert(offset: let o, element: let e, associatedWith: _): +/// arr.insert(e, at: o) +/// } +/// } +/// ``` +extension CollectionDifference : Collection { + public typealias Element = CollectionDifference.Change + public struct Index: Comparable, Hashable {} +} + +extension CollectionDifference.Change: Equatable where ChangeElement: Equatable {} +extension CollectionDifference: Equatable where ChangeElement: Equatable {} + +extension CollectionDifference.Change: Hashable where ChangeElement: Hashable {} +extension CollectionDifference: Hashable where ChangeElement: Hashable { + /// Infers which `ChangeElement`s have been both inserted and removed only + /// once and returns a new difference with those associations. + /// + /// - Returns: an instance with all possible moves inferred. + /// + /// - Complexity: O(*n*) where *n* is `self.count` + public func inferringMoves() -> CollectionDifference +} + +extension CollectionDifference: Codable where ChangeElement: Codable {} +``` + +A `Change` is a single mutating operation, a `CollectionDifference` is a plurality of such operations that represents a complete transition between two states. Given the interdependence of the changes, `CollectionDifference` has no mutating members, but it does allow index- and `Slice`-based access to its changes via `Collection` conformance as well as a validating initializer taking a `Collection`. + +Fundamentally, there are only two operations that mutate collections, `insert(_:at:)` and `remove(_:at:)`, but there are benefits from being able to represent other operations such as moves and replacements, especially for UIs that may want to animate a move differently from an `insert`/`remove` pair. These operations are represented using `associatedWith:`. When non-`nil`, they refer to the offset of the counterpart as described in the headerdoc. + +### Application of instances of `CollectionDifference` + +``` swift +extension RangeReplaceableCollection { + /// Applies a difference to a collection. + /// + /// - Parameter difference: The difference to be applied. + /// + /// - Returns: An instance representing the state of the receiver with the + /// difference applied, or `nil` if the difference is incompatible with + /// the receiver's state. + /// + /// - Complexity: O(*n* + *c*), where *n* is `self.count` and *c* is the + /// number of changes contained by the parameter. + @available(swift, introduced: 5.1) + public func applying(_ difference: CollectionDifference) -> Self? +} +``` + +Applying a diff to an incompatible base state is the only way application can fail. `applying(_:)` expresses this by returning nil. + +## Source compatibility + +This proposal is additive and the names of the types it proposes are not likely to already be in wide use, so it does not represent a significant risk to source compatibility. + +## Effect on ABI stability + +This proposal does not affect ABI stability. + +## Effect on API resilience + +This feature is additive and symbols marked with `@available(swift, introduced: 5.1)` as appropriate. + +## Alternatives considered + +The following is an incomplete list based on common feedback received during the process of developing this API: + +### Communicating changes via a series of callbacks + +Breaking up a transaction into a sequence of imperative events is not very Swifty, and the pattern has proven to be fertile ground for defects. + +### More cases in `CollectionDifference.Change` + +While other cases such as `.move` are tempting, the proliferation of code in switch statements is unwanted overhead for clients that don't care about the "how" of a state transition so much as the "what". + +The use of associated offsets allows for more information to be encoded into the diff without making it more difficult to use. You've already seen how associated offsets can be used to illustrate moves (as produced by `inferringMoves()`): + +``` swift +CollectionDifference([ + .remove(offset:0, element: "value", associatedWith: 4), + .insert(offset:4, element: "value", associatedWith: 0) +]) +``` + +But they can also be used to illustrate replacement when the offsets refer to the same position (and the element is different): + +``` swift +CollectionDifference([ + .remove(offset:0, element: "oldvalue", associatedWith: 0), + .insert(offset:0, element: "newvalue", associatedWith: 0) +]) +``` + +Differing offsets and elements can be combined when a value is both moved and replaced (or changed): + +``` swift +CollectionDifference([ + .remove(offset:4, element: "oldvalue", associatedWith: 0), + .insert(offset:0, element: "newvalue", associatedWith: 4) +]) +``` + +Neither of these two latter forms can be inferred from a diff by inferringMoves(), but they can be legally expressed by any API that vends a difference. + +### `applying(_:) throws -> Self` instead of `applying(_:) -> Self?` + +Applying a diff can only fail when the base state is incompatible. As such, the additional granularity provided by an error type does not add any value. + +### Use `Index` instead of offset in `Change` + +Because indexes cannot be navigated in the absence of the collection instance that generated them, a diff based on indexes instead of offsets would be much more limited in usefulness as a boundary type. If indexes are required, they can be rehydrated from the offsets in the presence of the collection(s) to which they belong. + +### `Change` generic on both `BaseElement` and `OtherElement` instead of just `Element` + +Application of differences would only be possible when both `Element` types were equal, and there would be additional cognitive overhead with comparators with the type `(Element, Other.Element) -> Bool`. + +Since the comparator forces both types to be effectively isomorphic, a diff generic over only one type can satisfy the need by mapping one (or both) collections to force their `Element` types to match. + +### `difference(from:using:)` with an enum parameter for choosing the diff algorithm instead of `difference(from:)` + +This is an attractive API concept, but it can be very cumbersome to extend. This is especially the case for types like `OrderedSet` that—through member uniqueness and fast membership testing—have the capability to support very fast diff algorithms that aren't appropriate for other types. + +### `CollectionDifference` or just `Difference` instead of `CollectionDifference` + +The name `CollectionDifference` gives us the opportunity to build a family of related types in the future, as the difference type in this proposal is (intentionally) unsuitable for representing differences between keyed collections (which don't shift their elements' keys on insertion/removal) or structural differences between treelike collections (which are multidimensional). + +## Intentional omissions: + +### Further adoption + +This API allows for more interesting functionality that is not included in this proposal. + +### `mutating apply(_:)` + +There is no mutating applicator because there is no algorithmic advantage to in-place application. + +### `mutating inferringMoves()` + +While there may be savings to be had from in-place move inferencing; we're holding this function for a future proposal. + +### Formalizing the concept of an ordered collection + +This problem warrants a proposal of its own. diff --git a/proposals/0241-string-index-explicit-encoding-offset.md b/proposals/0241-string-index-explicit-encoding-offset.md new file mode 100644 index 0000000000..bb0a847080 --- /dev/null +++ b/proposals/0241-string-index-explicit-encoding-offset.md @@ -0,0 +1,228 @@ +# Deprecate String Index Encoded Offsets + +* Proposal: [SE-0241](0241-string-index-explicit-encoding-offset.md) +* Author: [Michael Ilseman](https://github.com/milseman) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.0)** +* Implementation: [apple/swift#22108](https://github.com/apple/swift/pull/22108) +* Review: ([review](https://forums.swift.org/t/se-0241-explicit-encoded-offsets-for-string-indices/19929)) ([acceptance](https://forums.swift.org/t/accepted-se-0241-explicit-encoded-offsets-for-string-indices/20540)) + +## Introduction + +[SE-0180][] introduced a computed variable and initializer surrounding the concept of an `encodedOffset` for serialization purposes. Unfortunately, that approach is flawed for its intended purpose and is commonly misused in ways that Swift 5 is [more likely to expose](https://bugs.swift.org/browse/SR-9749). It is too late in the Swift 5.0 release to solve all existing problems, so we propose deprecating `encodedOffset` and introducing a targeted, semantics-preserving alternative. + +## Motivation + +String abstracts away details about the underlying encoding used in its storage. String.Index is opaque and represents a position within a String or Substring. This can make serializing a string alongside its indices difficult, and for that reason [SE-0180][] added a computed variable and initializer `encodedOffset` in Swift 4.0. + +String was always meant to be capable of handling multiple backing encodings for its contents, and this is realized in Swift 5. [String now uses UTF-8](https://forums.swift.org/t/string-s-abi-and-utf-8/17676) for its preferred “fast” native encoding, but has a resilient fallback for strings of different encodings. Currently, we only use this fall-back for lazily-bridged Cocoa strings, which are commonly encoded as UTF-16, though it can be extended in the future thanks to resilience. + +Unfortunately, [SE-0180][]’s approach of a single notion of `encodedOffset` is flawed. A string can be serialized with a choice of encodings, and the offset is therefore encoding-dependent and requires access to the contents of the string to calculate. A comment in [SE-0180][]’s example source mentioned that `encodedOffset` assumes UTF-16, which happened to be the only encoding used internally by String at the time (for offset purposes). + +Furthermore, the majority of uses of `encodedOffset` in the wild are not following [SE-0180][]’s intended purpose and are sensitive to encoding changes. `encodedOffset` is frequently misused under the assumption that all Characters are comprised of a single code unit, which is error-prone and Swift 5 might surface the underlying bugs in more situations. It is also sometimes used for mapping Cocoa string indices, which happens to work in Swift 4 but might not in Swift 5, and Foundation already provides better alternatives. + + + +## Proposed solution + +We propose a targeted semantics-preserving off-ramp for uses of `encodedOffset`. Because Swift 5 may introduce a semantic difference in behavior, it is important to rush this fix into the 5.0 release so that developers can preserve existing semantics. Potential solutions to other problems, including the original intended purpose of `encodedOffset`, are highlighted in “Alternatives Considered”. + + +## Detailed design + +```swift +extension String.Index { + /// The UTF-16 code unit offset corresponding to this Index + public func utf16Offset(in s: S) -> Int { + return s.utf16.distance(from: s.utf16.startIndex, to: self) + } + /// Creates a new index at the specified UTF-16 code unit offset + /// + /// - Parameter offset: An offset in UTF-16 code units. + public init(utf16Offset offset: Int, in s: S) { + let (start, end) = (s.utf16.startIndex, s.utf16.endIndex) + guard offset >= 0, + let idx = s.utf16.index(start, offsetBy: offset, limitedBy: end) + else { + self = end.nextEncoded // internal method returning endIndex+1 + return + } + self = idx + } +} + +``` + +We try to match the original semantics as close as we reasonably can. If the user supplies an out-of-bounds offset to the initializer, we will form an invalid index. If the out-of-bounds offset is exactly equivalent to the count, the returned index will compare equal with `endIndex`, otherwise it will compare greater than `endIndex`. + + +## Source Compatibility + +This deprecates existing API and provides a semantics-preserving alternative. Deprecation preserves source compatibility and strongly hints towards correct usage. But, other changes in Swift 5 introduce potential semantic drift in old code. + +## Effect of ABI stability + +This change is ABI-additive, but necessary due to other ABI changes in Swift 5. + +## Effect on API resilience + +Added APIs are all resilient and can be replaced with more efficient implementations that preserve correctness as String evolves. + +## Alternatives Considered + +### Do Nothing + +If `encodedOffset` was only used for serialization, *and* such serialization/deserialization would record and preserve the original encoding, *and* we amend [SE-0180][]’s comment to avoid nailing it down to any given encoding, no change would be necessary. Unfortunately, there is no way to query or preserve internal encoding and there is considerable use and misuse in the wild, as mentioned in the “Uses in the Wild” disclosure section. + +### Fix all the Bugs + +This proposal originally introduced a set of API attempting to solve 3 problems: + +1. [SE-0180][]’s `encodedOffset`, meant for serialization purposes, needs to be parameterized over the encoding in which the string will be serialized in +2. Existing uses of `encodedOffset` need a semantics-preserving off-ramp for Swift 5, which is expressed in terms of UTF-16 offsets +3. Existing misuses of `encodedOffset`, which assume all characters are a single UTF-16 code unit, need a semantics-fixing alternative + + +
Details: String’s views and encodings + +String has 3 views which correspond to the most popular Unicode encodings: UTF-8, UTF-16, and UTF-32 (via the Unicode scalar values). String’s default view is of Characters. + +```swift +let myString = "abc\r\nいろは" +Array(myString.utf8) // UTF-8 encoded +Array(myString.utf16) // UTF-16 encoded +Array(myString.unicodeScalars.lazy.map { $0.value }) // UTF-32 encoded +Array(myString); Array(myString.indices) // Not an encoding, but provides offset-based access to `Characters` +``` +
+ +#### Uses in the Wild +
+ +GitHub code search yields [nearly 1500 uses](https://github.com/search?l=Swift&q=encodedOffset&type=Code) , and nearly-none of them are for [SE-0180][]’s intended purpose. Below I present the 3 most common uses. + +```swift +// Common code for these examples +let myString: String = ... +let start: String.Index = ... +let end: String.Index = ... +let utf16OffsetRange: Range = ... +let nsRange: NSRange = ... +``` + + +#### Offset-based `Character` indexing + +The most common misuse of `encodedOffset` assumes that all Characters in a String are comprised of a single code unit. This is wrong and a source of surprising bugs, even for exclusively ASCII content: `"\r\n".count == 1`. + +```swift +let (i, j): (Int, Int) = ... // Something computed in terms of myString.count + +// Problematic code +myString[String.Index(encodedOffset: i]..` and `NSRange`. Foundation already provides convenient initializers for this purpose already, and using them is the preferred approach: + +```swift +// Problematic code +let myNSRange = NSRange(location: start.encodedOffset, length: end.encodedOffset - start.encodedOffset) +let myStrRange = String.Index(encodedOffset: nsRange.lowerBound).. + +#### Potential Solution + +
Original Proposed Solution + +Here is a (slightly revised) version of the original proposal: + +```swift + /// The UTF-16 code unit offset corresponding to this Index + public func offset(in utf16: S.UTF16View) -> Int { ... } + + /// The UTF-8 code unit offset corresponding to this Index + public func offset(in utf8: S.UTF8View) -> Int { ... } + + /// The Unicode scalar offset corresponding to this Index + public func offset(in scalars: S.UnicodeScalarView) -> Int { ... } + + /// The Character offset corresponding to this Index + public func offset(in str: S) -> Int { ... } + + /// Creates a new index at the specified UTF-16 code unit offset + /// + /// - Parameter offset: An offset in UTF-16 code units. + public init(offset: Int, in utf16: S.UTF16View) { ... } + + /// Creates a new index at the specified UTF-8 code unit offset + /// + /// - Parameter offset: An offset in UTF-8 code units. + public init(offset: Int, in utf8: S.UTF8View) { ... } + + /// Creates a new index at the specified Unicode scalar offset + /// + /// - Parameter offset: An offset in terms of Unicode.Scalars + public init(offset: Int, in scalars: S.UnicodeScalarView) { ... } + + /// Creates a new index at the specified Character offset + /// + /// - Parameter offset: An offset in terms of Characters + public init(offset: Int, in str: S) { ... } +} +``` + + +This gives developers: + +1. The ability to choose a specific encoding for serialization, the original intended purpose. +2. The ability to fix any code that assumed fixed-encoding-width Characters by choosing the most-natural variant that just takes a String. +3. The ability to migrate their uses for Cocoa index mapping by choosing UTF-16. + +However, it’s not clear this is the best approach for Swift and more design work is needed: + +* Overloading only on view type makes it easy to accidentally omit a view and end up with character offsets. E.g. `String.Index(offset: myUTF16Offset, in: myUTF16String)` instead of `String.Index(offset: myUTF16Offset, in: myUTF16String.utf16)`. +* Producing new indices is usually done by the collection itself rather than parameterizing an index initializer. This should be handled with something more ergonomic such as offset-based indexing in a future release. +* In real code in the wild, almost all created indices are immediately used to subscript the string or one of its views. This should be handled with something more ergonomic such as [offset-based subscripting](https://forums.swift.org/t/shorthand-for-offsetting-startindex-and-endindex/9397) in a future release. + +
+ +#### Conclusion + +It is too late in the Swift 5.0 release to design and add all of these API. Instead, we’re proposing an urgent, targeted fix for the second problem. + +[SE-0180]: diff --git a/proposals/0242-default-values-memberwise.md b/proposals/0242-default-values-memberwise.md new file mode 100644 index 0000000000..4b6522f683 --- /dev/null +++ b/proposals/0242-default-values-memberwise.md @@ -0,0 +1,132 @@ +# Synthesize default values for the memberwise initializer + +* Proposal: [SE-0242](0242-default-values-memberwise.md) +* Author: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.1)** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0242-synthesize-default-values-for-the-memberwise-initializer/20618/98) +* Implementation: [apple/swift#19743](https://github.com/apple/swift/pull/19743) + +## Introduction + +This proposal aims to solve a simple outstanding problem with the way the Swift compiler currently synthesizes the memberwise initializer for structures by synthesizing default values for properties with default initializers. + +*This is mentioned in the "State of the Memberwise Initializer" forum post: [here](https://forums.swift.org/t/state-of-the-memberwise-initializer/17168)* + +## Motivation + +Currently the Swift compiler is able to synthesize a fairly basic memberwise initializer for structures. + +```swift +struct Dog { + var age: Int + var name: String +} +``` + +The compiler is able to synthesize a memberwise iniailizer for this structure which simply looks like this: + +```swift +init(age: Int, name: String) +``` + +But, lets say we want all dogs to have a default value of `0` for the age: + +```swift +struct Dog { + var age: Int = 0 + var name: String +} +``` + +A user might naively try using this default value when constructing their `Dog` instance: + +```swift +// I just want to set the name of Dog, sparky is a newborn +let sparky = Dog(name: "Sparky") +``` + +To their surprise, they can't. `missing argument for parameter 'age' in call`. Using the compiler synthesized memberwise initializer has turned to become a nuisance rather than a nice removal of boilerplate. In many cases the user may optionally just define their own initializer with a default value for the age parameter. + +```swift +struct Dog { + var age: Int = 0 + var name: String + + // This is defined because the Swift compiler can't generate default values for properties with an initial value + init(age: Int = 0, name: String) { + self.age = age + self.name = name + } +} +``` + +## Proposed solution + +I propose simply doing the obvious and synthesizing default values for properties with default initializers in the memberwise initializer. Simple code like the following will simply work: + +```swift +struct Dog { + var age: Int = 0 + var name: String + + // The generated memberwise init: + init(age: Int = 0, name: String) +} + +// This now works +let sparky = Dog(name: "Sparky") // Dog(age: 0, name: "Sparky") +``` + +The following example displays the memberwise initializer being produced by the compiler with a combination of variables with default values. + +```swift +struct Alphabet { + var a: Int = 97 + let b: String + var c: String = "c" + let d: Bool = true + var e: Double = Double.random(in: 0 ... .pi) + + // The generated memberwise init: + init( + a: Int = 97, + b: String, + c: String = "c", + e: Double = Double.random(in: 0 ... .pi) + ) +} +``` + +Notice the `d` variable does not get an entry in the memberwise initializer because it is a constant whose value is already assigned. This behavior already exists with the current initializer. + +In the case where multiple variables are being initialized together, we cannot generate a default value for them in the memberwise initializer. For example: + +```swift +struct Person { + var (firstName, lastName) = ("First", "Last") + + // The generated memberwise init: + init(firstName: String, lastName: String) +} +``` + +## Detailed design + +This change does not alter the requirements needed to synthesize the memberwise initializer, but rather if we can synthesize the memberwise initializer, also synthesize default values for properties with default initializers. Note that we can only synthesize values for *variables* that have declared default initializers and not *constants*. + +## Source compatibility + +This is a purely additive feature, thus source compatibility is not affected. + +## Effect on ABI stability + +This feature does not alter ABI, thus ABI stability is not affected. + +## Effect on API resilience + +As the memberwise initializer is only synthesized as an internal initializer, this feature does not affect API resilience. + +## Alternatives considered + +We could simply not do this and save this proposal for a solution much larger in regards to fixing more problems the memberwise initializer has. The downside is that we put off obvious changes like this for much longer because of wanting to solve a bigger problem. I agree we should solve the bigger problems, but by solving problems like this it aids in the solution of the larger problem. diff --git a/proposals/0243-codepoint-and-character-literals.md b/proposals/0243-codepoint-and-character-literals.md new file mode 100644 index 0000000000..7df6c0ee75 --- /dev/null +++ b/proposals/0243-codepoint-and-character-literals.md @@ -0,0 +1,226 @@ +# Integer-convertible character literals + +* Proposal: [SE-0243](0243-codepoint-and-character-literals.md) +* Authors: [Diana Ma (“Taylor Swift”)](https://github.com/tayloraswift), [Chris Lattner](https://github.com/lattner), [John Holdsworth](https://github.com/johnno1962) +* Review manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Rejected** ([Rationale](https://forums.swift.org/t/se-0243-codepoint-and-character-literals/21188/341)) +* Implementation: [apple/swift#21873](https://github.com/apple/swift/pull/21873) +* Threads: [1](https://forums.swift.org/t/prepitch-character-integer-literals/10442) + +## Introduction + +Swift’s `String` type is designed for Unicode correctness and abstracts away the underlying binary representation of the string to model it as a `Collection` of grapheme clusters. This is an appropriate string model for human-readable text, as to a human reader, the atomic unit of a string is (usually) the extended grapheme cluster. When treated this way, many logical string operations “just work” the way users expect. + +However, it is also common in programming to need to express values which are intrinsically numeric, but have textual meaning, when taken as an ASCII value. We propose adding a new literal syntax takes single-quotes (`'`), and is transparently convertible to Swift’s integer types. This syntax, but not the behavior, will extend to all “single element” text literals, up to and including `Character`, and will become the preferred literal syntax these types. + +## Motivation + +A pain point of using characters in Swift is they lack a first-class literal syntax. Users have to manually coerce string literals to a `Character` or `Unicode.Scalar` type using `as Character` or `as Unicode.Scalar`, respectively. Having the collection share the same syntax as its element also harms code clarity and makes it difficult to tell if a double-quoted literal is being used as a string or character in some cases. + +Additional challenges arise when using ASCII scalars in Swift. Swift currently provides no static mechanism to assert that a unicode scalar literal is restricted to the ASCII range, and lacks a readable literal syntax for such values as well. In C, `'a'` is a `uint8_t` literal, equivalent to `97`. Swift has no such equivalent, requiring awkward spellings like `UInt8(ascii: "a")`, or spelling out the values in hex or decimal directly. This harms readability of code, and makes bytestring processing in Swift painful. + +```c +static char const hexcodes[16] = { + '0', '1', '2', '3', '4' ,'5', '6', '7', '8', '9', + 'a', 'b', 'c', 'd', 'e', 'f' +}; +``` + +```swift +let hexcodes = [ + UInt8(ascii: "0"), UInt8(ascii: "1"), UInt8(ascii: "2"), UInt8(ascii: "3"), + UInt8(ascii: "4"), UInt8(ascii: "5"), UInt8(ascii: "6"), UInt8(ascii: "7"), + UInt8(ascii: "8"), UInt8(ascii: "9"), UInt8(ascii: "a"), UInt8(ascii: "b"), + UInt8(ascii: "c"), UInt8(ascii: "d"), UInt8(ascii: "e"), UInt8(ascii: "f") +] +``` + +Higher-level constructs can regain some readability, + +```swift +let hexcodes = [ + "0", "1", "2", "3", + "4", "5", "6", "7", + "8", "9", "a", "b", + "c", "d", "e", "f" +].map{ UInt8(ascii: $0) } +``` + +but may not be familiar to all users, and can come at a runtime cost. + +In addition, the `init(ascii:)` initializer only exists on `UInt8`. If you're working with other types like `Int8` (common when dealing with C APIs that take `char`), it is much more awkward. Consider scanning through a `char*` buffer as an `UnsafeBufferPointer`: + +```swift +for scalar in int8buffer { + switch scalar { + case Int8(UInt8(ascii: "a")) ... Int8(UInt8(ascii: "f")): + // lowercase hex letter + case Int8(UInt8(ascii: "A")) ... Int8(UInt8(ascii: "F")): + // uppercase hex letter + case Int8(UInt8(ascii: "0")) ... Int8(UInt8(ascii: "9")): + // hex digit + default: + // something else + } +} +``` + +Transforming `Unicode.Scalar` literals also sacrifices compile-time guarantees. The statement `let char: UInt8 = 1989` is a compile time error, whereas `let char: UInt8 = .init(ascii: "߅")` is a run time error. + +ASCII scalars are inherently textual, so it should be possible to express them with a textual literal directly. Just as applying the `String` APIs runs counter to Swift’s stated design goals of safety and efficiency, requiring users to express basic data values in such a verbose way runs counter to our design goal of [expressiveness](https://swift.org/about/#swiftorg-and-open-source). + +Integer character literals would provide benefits to `String` users. One of the [future directions](https://gist.github.com/milseman/bb39ef7f170641ae52c13600a512782f#unmanaged-strings) for `String` is to provide performance-sensitive or low-level users with direct access to code units. Having numeric character literals for use with this API is highly motivating. Furthermore, improving Swift’s bytestring ergonomics is an [important part](https://forums.swift.org/t/prepitch-character-integer-literals/10442/140?u=taylorswift) of our long term goal of expanding into embedded platforms. + +## Proposed solution + +The most straightforward solution is to conform Swift’s integer types to `ExpressibleByUnicodeScalarLiteral`. Due to ABI constraints, it is not currently possible to add this conformance, so we will add the conformance *implementations* to the standard library, and allow users to “enable” to the feature by declaring this conformance in user code, for example: + +```swift +extension Int8: ExpressibleByUnicodeScalarLiteral { } +``` +Once the Swift ABI supports retroactive conformances, this conformance can be declared in the standard library, making it available by default. + +These integer conversions will only be valid for the ASCII range `U+0 ..< U+128`; unicode scalar literals outside of that range will be invalid and will generate compile-time errors similar to the way we currently diagnose overflowing integer literals. This is a conservative approach, as allowing transparent unicode conversion to integer types carries encoding pitfalls users may not anticipate or easily understand. + +Because it is currently possible to call literal initializers at run-time, a run-time precondition failure will occur if a non-ASCII value is passed to the integer initializer. (We expect the compiler to elide this precondition check for “normal” invocations.) + +```swift +let u: Unicode.Scalar = '\u{FF}' +let i1: Int = '\u{FF}' // compile-time error +let i2: Int = .init(unicodeScalarLiteral: u) // run-time error +``` + +| `ExpressibleBy`… | `UnicodeScalarLiteral` | `ExtendedGraphemeClusterLiteral` | `StringLiteral` | +| --- | --- | --- | --- | +| `UInt8:`, … , `Int:` | yes* (initially opt-in) | no | no | +| `Unicode.Scalar:` | yes | no | no | +| `Character:` | yes (inherited) | yes | no | +| `String:` | yes | yes | yes | +| `StaticString:` | yes | yes | yes | + +> Cells marked with an asterisk `*` indicate behavior that is different from the current language behavior. + +The ASCII range restriction will only apply to single-quote literals coerced to a `Unicode.Scalar` and (either statically or dynamically) converted to an integer type. Any valid `Unicode.Scalar` can be written as a single-quoted unicode scalar literal, and any valid `Character` can be written as a single-quoted character literal. + +| | `'a'` | `'é'` | `'β'` | `'𓀎'` | `'👩‍✈️'` | `"ab"` | +| --- | --- | --- | --- | --- | --- | --- | +| `:String` | `"a"` | `"é"` | `"β"` | `"𓀎"` | `"👩‍✈️"` | "ab" +| `:Character` | `'a'` | `'é'` | `'β'` | `'𓀎'` | `'👩‍✈️'` +| `:Unicode.Scalar` | U+0061 | U+00E9 | U+03B2 | U+1300E +| `:UInt32` | 97 | +| `:UInt16` | 97 | +| `:UInt8` | 97 | +| `:Int8` | 97 | + +With these changes, the hex code example can be written much more naturally: + +```swift +let hexcodes: [UInt8] = [ + '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', + 'a', 'b', 'c', 'd', 'e', 'f' +] + +for scalar in int8buffer { + switch scalar { + case 'a' ... 'f': + // lowercase hex letter + case 'A' ... 'F': + // uppercase hex letter + case '0' ... '9': + // hex digit + default: + // something else + } +} +``` + +### Choice of single quotes + +We propose to adopt the `'x'` syntax for all textual literal types up to and including `ExtendedGraphemeClusterLiteral`, but not including `StringLiteral`. These literals will be used to express integer types, `Character`, `Unicode.Scalar`, and types like `UTF16.CodeUnit` in the standard library. + +The default inferred literal type for `let x = 'a'` will be `Character`, following the principle of least surprise. This also allows for a natural user-side syntax for differentiating methods overloaded on both `Character` and `String`. + +Use of single quotes for character/scalar literals is precedented in other languages, including C, Objective-C, C++, Java, and Rust, although different languages have slightly differing ideas about what a “character” is. We choose to use the single quote syntax specifically because it reinforces the notion that strings and character values are different: the former is a sequence, the later is an element. Character types also don't support string literal interpolation, which is another reason to move away from double quotes. + +### Single quotes in Swift, a historical perspective + +In Swift 1.0, single quotes were reserved for some yet-to-be determined syntactical purpose. Since then, pretty much all of the things that might have used single quotes have already found homes in other parts of the Swift syntactical space: + +- syntax for [multi-line string literals](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0168-multi-line-string-literals.md) uses triple quotes (`"""`) + +- string interpolation syntax uses standard double quote syntax. + +- raw-mode string literals settled into the `#""#` syntax. + +- In current discussions around [regex literals](https://forums.swift.org/t/string-update/7398/6), most people seem to prefer slashes (`/`). + +Given that, and the desire for lightweight syntax for single character syntax, and the precedent in other languages for characters, it is natural to use single quotes for this purpose. + +### Existing double quote initializers for characters + +We propose deprecating the double quote literal form for `Character` and `Unicode.Scalar` types and slowly migrating them out of Swift. + +```swift +let c2 = 'f' // preferred +let c1: Character = "f" // deprecated +``` + +## Detailed Design + +The only standard library changes will be to extend `FixedWidthInteger` to have the `init(unicodeScalarLiteral:)` initializer required by `ExpressibleByUnicodeScalarLiteral`. Static ASCII range checking will be done in the type checker, dynamic ASCII range checking will be done by a runtime precondition. + +```swift +extension FixedWidthInteger { + init(unicodeScalarLiteral: Unicode.Scalar) +} +``` + +The default inferred type for all single-quoted literals will be `Character`. This addresses a language pain point where declaring a `Character` requires type context. + +## Source compatibility + +This proposal could be done in a way that is strictly additive, but we feel it is best to deprecate the existing double quote initializers for characters, and the `UInt8.init(ascii:)` initializer. + +Here is a specific sketch of a deprecation policy: + + * Continue accepting these in Swift 5 mode with no change. + + * Introduce the new syntax support into Swift 5.1. + + * Swift 5.1 mode would start producing deprecation warnings (with a fixit to change double quotes to single quotes.) + + * The Swift 5 to 5.1 migrator would change the syntax (by virtue of applying the deprecation fixits.) + + * Swift 6 would not accept the old syntax. + +During the transition period, `"a"` will remain a valid unicode scalar literal, but attempting to initialize integer types with double-quoted ASCII literals will produce an error. + +``` +let ascii:Int8 = "a" // error +``` + +However, as this will only be possible in new code, and will produce a deprecation warning from the outset, this should not be a problem. + +## Effect on ABI stability + +All changes except deprecating the `UInt8.init(ascii:)` initializer are either additive, or limited to the type checker, parser, or lexer. + +Removing `UInt8.init(ascii:)` would break ABI, but this is not necessary to implement the proposal, it’s merely housekeeping. + +## Effect on API resilience + +None. + +## Alternatives considered + +### Integer initializers + +Some have proposed extending the `UInt8(ascii:)` initializer to other integer types (`Int8`, `UInt16`, … , `Int`). However, this forgoes compile-time validity checking, and entails a substantial increase in API surface area for questionable gain. + +### Lifting the ASCII range restriction + +Some have proposed allowing any unicode scalar literal whose codepoint index does not overflow the target integer type to be convertible to that integer type. Consensus was that this is an easy source of unicode encoding bugs, and provides little utility to the user. If people change their minds in the future, this restriction can always be lifted in a source and ABI compatible way. + +### Single-quoted ASCII strings + +Some have proposed allowing integer *array* types to be expressible by *multi-character* ASCII strings such as `'abcd'`. We consider this to be out of scope of this proposal, as well as unsupported by precedent in C and related languages. diff --git a/proposals/0244-opaque-result-types.md b/proposals/0244-opaque-result-types.md new file mode 100644 index 0000000000..a6404963e1 --- /dev/null +++ b/proposals/0244-opaque-result-types.md @@ -0,0 +1,644 @@ +# Opaque Result Types + +* Proposal: [SE-0244](0244-opaque-result-types.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Joe Groff](https://github.com/jckarter) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.1)** +* Implementation: [apple/swift#22072](https://github.com/apple/swift/pull/22072) +* Toolchain: https://github.com/apple/swift/pull/22072#issuecomment-483495849 +* Previous revisions: ([1](https://github.com/swiftlang/swift-evolution/commit/e60bac23bf0d6f345ddb48fbf64ea8324fce79a9)) +* Previous review threads: https://forums.swift.org/t/se-0244-opaque-result-types/21252 +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0244-opaque-result-types-reopened/22942/57) + +## Introduction + +This proposal is the first part of a group of changes we're considering in a [design document for improving the UI of the generics model](https://forums.swift.org/t/improving-the-ui-of-generics/22814). We'll try to make this proposal stand alone to describe opaque return types, their design, and motivation, but we also recommend reading the design document for more in-depth exploration of the relationships among other features we're considering. We'll link to relevant parts of that document throughout this proposal. + +This specific proposal addresses the problem of [type-level abstraction for returns](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--missing-type-level-abstraction). Many libraries consist of composable generic components. For example, a graphics library might provide primitive types for basic shapes: + +```swift +protocol Shape { + func draw(to: Surface) + + func collides(with: Other) -> Bool +} + +struct Rectangle: Shape { /* ... */ } +struct Circle: Shape { /* ... */ } +``` + +along with composable transformations to combine and modify primitive shapes into more complex ones: + +```swift +struct Union: Shape { + var a: A, b: B + // ... +} + +struct Intersect: Shape { + var a: A, b: B + // ... +} + +struct Transformed: Shape { + var shape: S + var transform: Matrix3x3 + // ... +} +``` + +One could compose these transformations by using the existential type `Shape` instead of generic arguments, but doing so would imply more dynamism and runtime overhead than may be desired. If we directly compose the generic containers, maintaining the concrete types, then generic specialization can more readily optimize the composed operations together, and the type system can also be used. A game or graphics app may want to define objects in terms of their shapes: + +```swift +protocol GameObject { + // The shape of the object + associatedtype Shape: Shapes.Shape + + var shape: Shape { get } +} +``` + +However, users of the `GameObject` protocol would now be burdened with writing out long, explicit types for their shapes: + +```swift +struct EightPointedStar: GameObject { + var shape: Union> { + return Union(Rectangle(), Transformed(Rectangle(), by: .fortyFiveDegrees) + } +} +``` + +This is unsightly because it's verbose, but it's also not very helpful for someone reading this declaration. The exact return type doesn't really matter, only the fact that it conforms to `Shape`. Spelling out the return type also effectively reveals most of the implementation of `shape`, making the declaration brittle; clients of `EightPointedStar` could end up relying on its exact return type, making it harder if the author of `EightPointedStar` wants to change how they implement its shape, such as if a future version of the library provides a generic `NPointedStar` primitive. Right now, if you want to abstract the return type of a declaration from its signature, existentials or manual type erasure are your only options, and these [come with tradeoffs](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--limits-of-existentials) that are not always acceptable. + +## Proposed solution + +Instead of declaring the specific return type of `EightPointedStar.shape`'s current implementation, +all we really want to say is that it returns something that conforms to `Shape`. We propose +the syntax `some Protocol`: + +```swift +struct EightPointedStar: GameObject { + var shape: some Shape { + return Union(Rectangle(), Transformed(Rectangle(), by: .fortyFiveDegrees) + } +} +``` + +to declare that an `EightPointedStar` has some `Shape` without having to specify exactly what shape that is. The underlying concrete type is hidden, and can even change from one version of the library to the next without breaking those clients, because the underlying type identity is never exposed to clients. Unlike an existential, though, clients still have access to the type identity. This allows the library to provide a potentially-more-efficient design that leverages Swift's type system, without expanding the surface area of the library or making implementors of the library's protocols rely on exposing verbose implementation types. + +An opaque type behaves like a ["reverse generic"](https://forums.swift.org/t/reverse-generics-and-opaque-result-types/21608). In a traditional generic function, the caller decides what types get bound to the callee's generic arguments: + +```swift +func generic() -> T { ... } + +let x: Rectangle = generic() // T == Rectangle, chosen by caller +let x: Circle = generic() // T == Circle, chosen by caller +``` + +An opaque return type can be thought of as putting the generic signature "to the right" of the function arrow; instead of being a type chosen by the caller that the callee sees as abstracted, the return type is chosen by the callee, and comes back to the caller as abstracted: + +```swift +// Strawman syntax +func reverseGeneric() -> T { return Rectangle(...) } + +let x = reverseGeneric() // abstracted type chosen by reverseGeneric's implementation +``` + +Reverse generics are a great mental model for understanding opaque return types, but the notation +is admittedly awkward. We expect the common use case for this feature to be a single return value behind a set of protocol conformances, so we're proposing to start with the more concise `some Shape` syntax: + +```swift +// Proposed syntax +func reverseGeneric() -> some Shape { return Rectangle(...) } + +let x = reverseGeneric() // abstracted type chosen by reverseGeneric's implementation +``` + +Following the `some` keyword is a set of constraints on the implicit generic type variable: a class, protocol, `Any`, `AnyObject`, or some composition thereof (joined with `&`). +This `some Protocol` sugar [can be generalized to generic arguments and structural positions in return types](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--directly-expressing-constraints) in the future, and we could also eventually support a fully general generic signature for opaque returns. To enable incremental progress on the implementation, we propose starting by only supporting the `some` syntax in return position. + +### Type identity + +Generics give us an idea of what to expect from opaque return types. The return values from different calls to the same function have the same return type, like two variables of the same generic argument type: + +```swift +func foo(x: T, y: T) -> some Equatable { + let condition = x == y // OK to use ==, x and y are the same generic type T + return condition ? 1738 : 679 +} + +let x = foo("apples", "bananas") +let y = foo("apples", "some fruit nobody's ever heard of") + +print(x == y) // also OK to use ==, x and y are the same opaque return type +``` + +If the opaque type exposes associated types, those associated types' identities are also maintained. This allows the full API of protocols like the `Collection` family to be used: + +```swift +func makeMeACollection(with: T) -> some RangeReplaceableCollection & MutableCollection { ... } + +var c = makeMeACollection(with: 17) +c.append(c.first!) // ok: it's a RangeReplaceableCollection +c[c.startIndex] = c.first! // ok: it's a MutableCollection +print(c.reversed()) // ok: all Collection/Sequence operations are available + +func foo(_ : C) { } +foo(c) // ok: C inferred to opaque result type of makeMeACollection +``` + +Moreover, opaque result types preserve their identity when composed into other types, such as when forming a collection of the results: + +```swift +var cc = [c] +cc.append(c) // ok: cc's Element == the result type of makeMeACollection +var c2 = makeMeACollection(with: 38) +cc.append(c2) // ok: Element == the result type of makeMeACollection +``` + +The opaque return type can however depend on the generic arguments going into the function when +it's called, so the return types of the same function invoked with different generic arguments are +different: + +```swift +var d = makeMeACollection(with: "seventeen") +c = d // error: types of makeMeACollection and makeMeACollection are different +``` + +Like a generic argument, the static type system does not consider the opaque type to be statically equivalent to the type it happens to be bound to: + +```swift +func foo() -> some BinaryInteger { return 219 } +var x = foo() +let i = 912 +x = i // error: Int is not known to be the same as the return type as foo() +``` + +However, one can inspect an opaque type's underlying type at runtime using dynamic casting: + +```swift +if let x = foo() as? Int { + print("It's an Int, \(x)\n") +} else { + print("Guessed wrong") +} +``` + +In other words, like generic arguments, opaque result types are only opaque to the static type system. They don't have an independent existence at runtime. + +### Implementing a function returning an opaque type + +The implementation of a function returning an opaque type must return a value of the same concrete type `T` from each `return` statement, and `T` must meet all of the constraints stated on the opaque type. For example: + +```swift +protocol P { } +extension Int : P { } +extension String : P { } + +func f1() -> some P { + return "opaque" +} + +func f2(i: Int) -> some P { // ok: both returns produce Int + if i > 10 { return i } + return 0 +} + +func f2(flip: Bool) -> some P { + if flip { return 17 } + return "a string" // error: different return types Int and String +} + +func f3() -> some P { + return 3.1419 // error: Double does not conform to P +} + +func f4() -> some P { + let p: P = "hello" + return p // error: protocol type P does not conform to P +} + +func f5() -> some P { + return f1() // ok: f1() returns an opaque type that conforms to P +} + +protocol Initializable { init() } + +func f6(_: T.Type) -> some P { + return T() // ok: T will always be a concrete type conforming to P +} +``` + +These rules guarantee that there is a single concrete type produced by any call to the function. The concrete type can depend on the generic type arguments (as in the `f6()` example), but must be consistent across all `return` statements. + +Note that recursive calls *are* allowed, and are known to produce a value of the same concrete type, but the concrete type itself is not known: + +```swift +func f7(_ i: Int) -> some P { + if i == 0 { + return f7(1) // ok: returning our own opaque result type + } else if i < 0 { + let result: Int = f7(-i) // error: opaque result type of f7() is not convertible to Int + return result + } else { + return 0 // ok: grounds the recursion with a concrete type + } +} +``` + +Of course, there must be at least one `return` statement that provides a concrete type. + +Note that a function cannot call itself recursively in a way that forms a type parameterized on the function's own opaque result type, since this would mean that the opaque type is infinitely recursive: + +```swift +struct Wrapper: P { var value: T } + +func f8(_ i: Int) -> some P { + // invalid; this binds the opaque result type to Wrapper, + // which is Wrapper>, which is + // Wrapper>>... + return Wrapper(f8(i + 1)) +} +``` + +A function with an opaque result type is also required to have a `return` statement even if it does not terminate: + +```swift +func f9() -> some P { + fatalError("not implemented") + + // error: no return statement to get opaque type +} +``` + +This requirement is necessary because, even though `f9`'s return value cannot be reached, the return type of `f9` can still be propagated by local type inference or generic instantiation in ways that don't require evaluating `f9`, so a type for `f9` must be available: + +```swift +let delayedF9 = { f9() } // closure has type () -> return type of f9 +``` + +We can't necessarily default the underlying type to `Never`, since `Never` may not +conform to the constraints of the opaque type. If `Never` does conform, and it +is desired as the underlying return type, that can be written explicitly as a +`return` statement: + +```swift +extension Never: P {} +func f9b() -> some P { + return fatalError("not implemented") // OK, explicitly binds return type to Never +} +``` + +This restriction on non-terminating functions could be something we lift in the future, by synthesizing bottom-type conformances to the protocols required by the opaque type. We leave that as a future extension. + +### Properties and subscripts + +Opaque result types can also be used with properties and subscripts: + +```swift +struct GameObject { + var shape: some Shape { /* ... */ } +} +``` + +For computed properties, the concrete type is determined by the `return` statements in the getter. Opaque result types can also be used in stored properties that have an initializer, in which case the concrete type is the type of the initializer: + +```swift +let strings: some Collection = ["hello", "world"] +``` + +Properties and subscripts of opaque result type can be mutable. For example: + +```swift +// Module A +public protocol P { + mutating func flip() +} + +private struct Witness: P { + mutating func flip() { /* ... */ } +} + +public var someP: some P = Witness() + +// Module B +import A +someP.flip() // ok: flip is a mutating function called on a variable +``` + +With a subscript or a computed property, the type of the value provided to the setter (e.g., `newValue`) is determined by the `return` statements in the getter, so the type is consistent and known only to the implementation of the property or subscript. For example: + +```swift +protocol P { } +private struct Impl: P { } + +public struct Vendor { + private var storage: [Impl] = [/* ... */] + + public var count: Int { + return storage.count + } + + public subscript(index: Int) -> some P { + get { + return storage[index] + } + set (newValue) { + storage[index] = newValue + } + } +} + +var vendor = Vendor() +vendor[0] = vendor[2] // ok: can move elements around +``` + +### Associated type inference + +While one can use type inference to declare variables of the opaque result type of a function, there is no direct way to name the opaque result type: + +```swift +func f1() -> some P { /* ... */ } + +let vf1 = f1() // type of vf1 is the opaque result type of f1() +``` + +However, type inference can deduce an opaque result type as the associated type of the protocol: + +```swift +protocol GameObject { + associatedtype ObjectShape: Shape + + var shape: ObjectShape +} + +struct Player: GameObject { + var shape: some Shape { /* ... */ } + + // infers typealias Shape = opaque result type of Player.shape +} + +let pos: Player.ObjectShape // ok: names the opaque result type of S.someValue() +pos = Player().shape // ok: returns the same opaque result type +``` + +Note that having a name for the opaque result type still doesn't give information about the underlying concrete type. For example, the only way to create an instance of the type `S.SomeType` is by calling `S.someValue()`. + +Associated type inference can only infer an opaque result type for a non-generic requirement, because the opaque type is parameterized by the function's own generic arguments. For instance, in: + +```swift +protocol P { + associatedtype A: P + func foo(x: T) -> A +} + +struct Foo: P { + func foo(x: T) -> some P { + return x + } +} +``` + +there is no single underlying type to infer `A` to, because the return type +of `foo` is allowed to change with `T`. + +## Detailed design + +### Grammar of opaque result types + +The grammatical production for opaque result types is straightforward: + +``` +type ::= opaque-type + +opaque-type ::= 'some' type +``` + +The `type` following the `'some'` keyword is semantically restricted to be a class or existential type, meaning it must consist only of `Any`, `AnyObject`, protocols, or base classes, possibly composed using `&`. This type is used to describe the constraints on the implicit "reverse generic" parameter. + +### Restrictions on opaque result types + +Opaque result types can only be used as the result type of a function, the type of a variable, or the result type of a subscript. The opaque type must be the entire return type of the function, For example, one cannot return an optional opaque result type: + +```swift +func f(flip: Bool) -> (some P)? { // error: `some P` is not the entire return type + // ... +} +``` + +This restriction could be lifted in the future. + +More fundamentally, opaque result types cannot be used in the requirements of a protocol: + +```swift +protocol Q { + func f() -> some P // error: cannot use opaque result type within a protocol +} +``` + +Associated types provide a better way to model the same problem, and the requirements can then be satisfied by a function that produces an opaque result type. (There might be an interesting shorthand feature here, where using `some` in a protocol requirement implicitly introduces an associated type, but we leave that for future language design to explore.) + +Similarly to the restriction on protocols, opaque result types cannot be used for a non-`final` declaration within a class: + +```swift +class C { + func f() -> some P { /* ... */ } // error: cannot use opaque result type with a non-final method + final func g() -> some P { /* ... */ } // ok +} +``` + +This restriction could conceivably be lifted in the future, but it would mean that override implementations would be constrained to returning the same type as their super implementation, meaning they must call `super.method()` to produce a valid return value. + +### Uniqueness of opaque result types + +Opaque result types are uniqued based on the function/property/subscript and any generic type arguments. For example: + +```swift +func makeOpaque(_: T.Type) -> some Any { /* ... */ } +var x = makeOpaque(Int.self) +x = makeOpaque(Double.self) // error: "opaque" type from makeOpaque is distinct from makeOpaque +``` + +This includes any generic type arguments from outer contexts, e.g., + +```swift +extension Array where Element: Comparable { + func opaqueSorted() -> some Sequence { /* ... */ } +} + +var x = [1, 2, 3]. opaqueSorted() +x = ["a", "b", "c"].opaqueSorted() // error: opaque result types for [Int].opaqueSorted() and [String].opaqueSorted() differ +``` + +### Implementation strategy + +From an implementation standpoint, a client of a function with an opaque result type needs to treat values of that result type like any other resilient value type: its size, alignment, layout, and operations are unknown. + +However, when the body of the function is known to the client (e.g., due to inlining or because the client is in the same compilation unit as the function), the compiler's optimizer will have access to the specific concrete type, eliminating the indirection cost of the opaque result type. + +## Source compatibility + +Opaque result types are purely additive. They can be used as a tool to improve long-term source (and binary) stability, by not exposing the details of a result type to clients. + +If opaque result types are retroactively adopted in a library, it would initially break source compatibility (e.g., if types like `EnumeratedSequence`, `FlattenSequence`, and `JoinedSequence` were removed from the public API) but could provide longer-term benefits for both source and ABI stability because fewer details would be exposed to clients. There are some mitigations for source compatibility, e.g., a longer deprecation cycle for the types or overloading the old signature (that returns the named types) with the new signature (that returns an opaque result type). + +## Effect on ABI stability + +Opaque result types are an ABI-additive feature, so do not in and of themselves impact existing ABI. However, additional runtime support is however needed to support instantiating opaque result types across ABI boundaries, meaning that *a Swift 5.1 runtime will be required to deploy code that uses opaque types in public API*. Also, changing an existing API to make use of opaque result types instead of returning concrete types would be an ABI-breaking change, so one of the source compatibility mitigations mentioned above would also need to be deployed to maintain ABI compatibility with existing binary clients. + +## Effect on API resilience + +Opaque result types are part of the result type of a function/type of a variable/element type of a subscript. The requirements that describe the opaque result type cannot change without breaking the API/ABI. However, the underlying concrete type *can* change from one version to the next without breaking ABI, because that type is not known to clients of the API. + +One notable exception to the above rule is `@inlinable`: an `@inlinable` declaration with an opaque result type requires that the underlying concrete type be `public` or `@usableFromInline`. Moreover, the underlying concrete type *cannot be changed* without breaking backward compatibility, because it's identity has been exposed by inlining the body of the function. That makes opaque result types somewhat less compelling for the `compactMap` example presented in the introduction, because one cannot have `compactMap` be marked `@inlinable` with an opaque result type, and then later change the underlying concrete type to something more efficient. + +We could allow an API originally specified using an opaque result type to later evolve to specify the specific result type. The result type itself would have to become visible to clients, and this might affect source compatibility, but (mangled name aside) such a change would be resilient. + +## Rust's `impl Trait` + +The proposed Swift feature is largely inspired by Rust's `impl Trait` language feature, described by [Rust RFC 1522](https://github.com/rust-lang/rfcs/blob/master/text/1522-conservative-impl-trait.md) and extended by [Rust RFC 1951](https://github.com/rust-lang/rfcs/blob/master/text/1951-expand-impl-trait.md). There are only a small number of differences between this feature as expressed in Swift vs. Rust's `impl Trait` as described in RFC 1522: + +* Swift's need for a stable ABI necessitates translation of opaque result types as resilient types, which is unnecessary in Rust's model, where the concrete type can always be used for code generation. +* Swift's opaque result types are fully opaque, because Swift doesn't have pass-through protocols like Rust's `Send` trait, which simplifies the type checking problem slightly. +* Due to associated type inference, Swift already has a way to "name" an opaque result type in some cases. + +## Alternatives considered + +### Return type inference + +Part of the motivation of this feature is to avoid having to spell out elaborate return types. This proposal achieves that for types that can be abstracted behind protocols, but in doing so introduces complexity in the form of a new kind of "reverse generic" type. Meanwhile, there are kinds of verbose return types that can't be effectively hidden behind protocol interfaces, like deeply nested collections: + +```swift +func jsonBlob() -> [String: [String: [[String: Any]]]] { ... } +``` + +We could theoretically address the verbosity problem in its full generality and without introducing new type system features by allowing return types to be inferred, like C++14's or D's `auto` return types: + +```swift +func jsonBlob() -> auto { ... } +``` + +Although this would superficially address the problem of verbose return types, that isn't really the primary problem this proposal is trying to solve, which is to allow for *more precise description of interfaces*. The case of a verbose composed generic adapter type is fundamentally different from a deeply nested collection; in the former case, the concrete type is not only verbose, but it's largely irrelevant, because clients should only care about the common protocols the type conforms to. For a nested collection, the verbose type *is* the interface: the full type is necessary to fully describe how someone interacts with that collection. + +Return type inference also has several undesirable traits as a language feature: + +- It violates separate compilation of function bodies, since the body of the function must be + type-checked to infer the return type. Swift already has type inference for + stored property declarations, and we consider this a mistake, since it has + been an ongoing source of implementation complexity and performance problems + due to the need to type-check the property initializer across files. This is + not a problem for opaque return types, because callers only interface with the + declaration through the opaque type's constraints. Code can be compiled against a + function with an opaque return type without having to know what the + underlying type is; it is "just an optimization" to specialize away the + opaque type when the underlying type is known. +- Similarly, inferred return types wouldn't provide any semantic abstraction once the type is inferred. Code that calls the function would still see the full concrete type, allowing clients to rely on unintentional details of the concrete type, and the implementation would be bound by ABI and source compatibility constraints if it needed to change the return type. Module interfaces and documentation would also still expose the full return type, meaning they don't get the benefit of the shorter notation. + +We see opaque return types as not only sugar for syntactically heavy return types, but also a tool for writing clearer, more resilient APIs. Return type inference would achieve the former but not the latter, while also introducing another compile-time performance footgun into the language. + +### Syntax for opaque return types + +This proposal suggests the word `some` to introduce opaque return types, since it has the right connotation by analogy to `Any` which is used to describe dynamically type-erased containers (and has been proposed to be a general way of referring to existential types)--a function that has an opaque return type returns *some* specific type +that conforms to the given constraints, whereas an existential can contain *any* type dynamically at any point in time. The spelling would also work well if [generalized to implicit generic arguments](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--directly-expressing-constraints) in the future. There are nonetheless reasonable objections to this keyword: + +- `some` is already used in the language as one of the case names for `Optional`; it is rare to need to spell `Optional.some` or `.some` explicitly, but it could nonetheless introduce confusion. +- In spoken language, `some type` and `sum type` sound the same. +- `swift some` could be a difficult term to search for. (However, on Google it currently gives [reasonable results](https://www.google.com/search?client=safari&rls=en&q=swift+some&ie=UTF-8&oe=UTF-8) about Optional in Swift.) + +Another obvious candidate is `opaque`, following the title of this very proposal. The word "opaque" is itself an overloaded term with existing meaning in many domains, which is unfortunate: + +```swift +protocol Shape {} + +func translucentRectangle() -> opaque Shape { /* ... */ } +``` + +The term "opaque" is also fairly heavily overloaded in the Swift implementation (albeit not so much the source-level programming model). It may be that there is a better term, such as "abstract return types", to refer to this feature in its entirety. `opaque` is also not as good a fit for generalization to implicit generic arguments. + +In swift-evolution discussion, several other names came up, including: + +``` +func translucentRectangle() -> unspecified Shape { ... } + +func translucentRectangle() -> anyOld Shape { ... } + +func translucentRectangle() -> hazyButSpecific Shape { ... } + +func translucentRectangle() -> someSpecific Shape { ... } + +func translucentRectangle() -> someConcrete Shape { ... } + +func translucentRectangle() -> nonspecific Shape { ... } + +func translucentRectangle() -> unclear Shape { ... } + +func translucentRectangle() -> arbitrary Shape { ... } + +func translucentRectangle() -> someThing Shape { ... } + +func translucentRectangle() -> anonymized Shape { ... } + +func translucentRectangle() -> nameless Shape { ... } +``` + +In our opinion, most of these are longer and not much clearer, and many wouldn't work well as generalized sugar for arguments and returns. + +### Opaque type aliases + +As proposed, opaque result types are tied to a specific declaration. They offer no way to state that two related APIs with opaque result types produce the *same* underlying concrete type. The idea of "reverse generics" could however be decoupled from function declarations, if you could write an "opaque typealias" that describes the abstracted interface to a type, while giving it a name, then you could express that relationship across declarations. For example: + +```swift +public typealias LazyCompactMapCollection + -> C where C.Element == ElementOfResult + = LazyMapSequence>, ElementOfResult> +``` + +In this strawman syntax, the "reverse generic" signature following the `->` is how clients see `LazyCompactMapCollection`. The underlying concrete type, spelled after the `=`, is visible only to the implementation (in some way that would have to be designed). With this feature, multiple APIs could be described as returning a `LazyCompactMapCollection`: + +```swift +extension LazyMapCollection { + public func compactMap(_ transform: @escaping (Element) -> U?) -> LazyCompactMapCollection { + // ... + } + + public func filter(_ isIncluded: @escaping (Element) -> Bool) -> LazyCompactMapCollection { + // ... + } +} +``` + +From the client perspective, both APIs would return the same type, but the specific underlying type would not be known. + +```swift +var compactMapOp = values.lazy.map(f).compactMap(g) +if Bool.random() { + compactMapOp = values.lazy.map(f).filter(h) // ok: both APIs have the same type +} +``` + +This would be a great feature to explore as a future direction, since it has some important benefits relative to opaque result types. However, we don't think it's the best place to start with this feature. Compare a declaration using opaque typealiases like: + +```swift +func foo() -> ReturnTypeOfFoo { return 1 } + +opaque typealias ReturnTypeOfFoo -> P = Int +``` + +to one using opaque return types: + +```swift +func foo() -> some P { return 1 } +``` + +The one using opaque typealiases requires an intermediate name, which one must read and follow to its definition to understand the interface of `foo`. The definition of `ReturnTypeOfFoo` also needs to spell out the underlying concrete return type of `foo`, and the two declarations are tightly coupled; a change to `foo` will likely require a lockstep change to `ReturnTypeOfFoo`. We expect that, in the common use case for this feature, the types being abstracted are going to be tied to specific declarations, and there wouldn't be any better name to really give than "return type of (decl)," so making opaque type aliases the only way of expressing return type abstraction would introduce a lot of obscuring boilerplate. + +## Future Directions + +As noted in the introduction, this proposal is the first part of a group of changes we're considering in a [design document for improving the UI of the generics model](https://forums.swift.org/t/improving-the-ui-of-generics/22814). That design document lays out a number of related directions we can go based on the foundation established by this proposal, including: + +- allowing [fully generalized reverse generics](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--reverse-generics) +- [generalizing the `some` syntax](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--directly-expressing-constraints) as shorthand for generic arguments, and allowing structural use in generic returns +- [more compact constraint syntax](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--directly-expressing-constraints) that can also work with generalized existentials +- introducing [`any` as a dual to `some`](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--clarifying-existentials) for explicitly spelling existential types + +We recommend reading that document for a more in-depth exploration of these related ideas. diff --git a/proposals/0245-array-uninitialized-initializer.md b/proposals/0245-array-uninitialized-initializer.md new file mode 100644 index 0000000000..28c017a2b9 --- /dev/null +++ b/proposals/0245-array-uninitialized-initializer.md @@ -0,0 +1,291 @@ +# Add an Array Initializer with Access to Uninitialized Storage + +* Proposal: [SE-0245](0245-array-uninitialized-initializer.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.1)** +* Previous Proposal: [SE-0223](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0223-array-uninitialized-initializer.md) +* Implementation: [apple/swift#23134](https://github.com/apple/swift/pull/23134) +* Bug: [SR-3087](https://bugs.swift.org/browse/SR-3087) + +## Introduction + +This proposal suggests a new initializer for `Array` and `ContiguousArray` +that provides access to an array's uninitialized storage buffer. + +Swift-evolution thread: [https://forums.swift.org/t/array-initializer-with-access-to-uninitialized-buffer/13689](https://forums.swift.org/t/array-initializer-with-access-to-uninitialized-buffer/13689) + +## Motivation + +Some collection operations require working on a fixed-size buffer of uninitialized memory. +For example, one O(*n*) algorithm for performing a stable partition of an array is as follows: + +1. Create a new array the same size as the original array. +2. Iterate over the original array, + copying matching elements to the beginning of the new array + and non-matching elements to the end. +3. When finished iterating, reverse the slice of non-matching elements. + +Unfortunately, the standard library provides no way to create an array of a +particular size without initializing every element. Even if we +avoid initialization by manually allocating the memory using an +`UnsafeMutableBufferPointer`, there's no way to convert that buffer into an +array without copying the contents. There simply isn't a way to implement this +particular algorithm with maximum efficiency in Swift. + +We also see this limitation when working with C APIs that fill a buffer with an +unknown number of elements and return the count. The workarounds are the same +as above: either initialize an array before passing it or copy the elements +from an unsafe mutable buffer into an array after calling. + +## Proposed solution + +Add a new `Array` initializer that lets a program work with an uninitialized +buffer. + +The new initializer takes a closure that operates on an +`UnsafeMutableBufferPointer` and an `inout` count of initialized elements. This +closure has access to the uninitialized contents of the newly created array's +storage, and must set the initialized count of the array before exiting. + +```swift +var myArray = Array(unsafeUninitializedCapacity: 10) { buffer, initializedCount in + for x in 1..<5 { + buffer[x] = x + } + buffer[0] = 10 + initializedCount = 5 +} +// myArray == [10, 1, 2, 3, 4] +``` + +With this new initializer, it's possible to implement the stable partition +as an extension to the `Collection` protocol, without any unnecessary copies: + +```swift +func stablyPartitioned(by belongsInFirstPartition: (Element) throws -> Bool) rethrows -> [Element] { + return try Array(unsafeUninitializedCapacity: count) { + buffer, initializedCount in + var low = buffer.baseAddress! + var high = low + buffer.count + do { + for element in self { + if try belongsInFirstPartition(element) { + low.initialize(to: element) + low += 1 + } else { + high -= 1 + high.initialize(to: element) + } + } + + let highIndex = high - buffer.baseAddress! + buffer[highIndex...].reverse() + initializedCount = buffer.count + } catch { + let lowCount = low - buffer.baseAddress! + let highCount = (buffer.baseAddress! + buffer.count) - high + buffer.baseAddress!.deinitialize(count: lowCount) + high.deinitialize(count: highCount) + throw error + } + } +} +``` + +This also facilitates efficient interfacing with C functions. For example, +suppose you wanted to wrap the function `vDSP_vsadd` in a Swift function that +returns the result as an array. This function requires you give it an unsafe +buffer into which it writes results. This is easy to do with an array, but you +would have to initialize the array with zeroes first. With a function like +`vDSP_vsadd`, this unnecessary zeroing out would eat into the slight speed edge +that the function gives you, defeating the point. This can be neatly solved +by using the proposed initializer: + +```swift +extension Array where Element == Float { + func dspAdd(scalar: Float) -> [Float] { + let n = self.count + return self.withUnsafeBufferPointer { buf in + var scalar = scalar + return Array(unsafeUninitializedCapacity: n) { rbuf, count in + vDSP_vsadd(buf.baseAddress!, 1, &scalar, rbuf.baseAddress!, 1, UInt(n)) + count = n + } + } + } +} +``` + +## Detailed design + +The new initializer is added to both `Array` and `ContiguousArray`. + +```swift +/// Creates an array with the specified capacity, then calls the given closure +/// with a buffer covering the array's uninitialized memory. +/// +/// The closure must set its second parameter to a number `c`, the number +/// of elements that are initialized. The memory in the range `buffer[0.., + _ initializedCount: inout Int + ) throws -> Void +) rethrows +``` + +### Specifying a capacity + +The initializer takes the specific capacity that a user wants to work with as a +parameter. The buffer passed to the closure has a count that is exactly the +same as the specified capacity, even if the ultimate capacity of the new array is larger. + +### Guarantees after throwing + +If the closure parameter to the initializer throws, the `initializedCount` +value at the time an error is thrown is assumed to be correct. This means that +a user who needs to throw from inside the closure has one of two options. +Before throwing, they must: + +1. deinitialize any newly initialized instances, or +2. update `initializedCount` to the correct count. + +In either case, the postconditions that `buffer[0.., + _ initializedCount: inout Int + ) throws -> Void + ) rethrows { + var buffer = UnsafeMutableBufferPointer + .allocate(capacity: unsafeUninitializedCapacity) + defer { buffer.deallocate() } + + var count = 0 + do { + try initializer(&buffer, &count) + } catch { + buffer.baseAddress!.deinitialize(count: count) + throw error + } + self = Array(buffer[0..(_ x: T) -> T where T: FloatingPoint { + return 1/(1 + exp(-x)) +} +``` +This doesn't work, because `exp` is not available on the `FloatingPoint` protocol. +Currently, you might work around this limitation by doing something like: +```swift +func sigmoid(_ x: T) -> T where T: FloatingPoint { + return 1/(1 + T(exp(-Double(x)))) +} +``` +but that's messy, inefficient if T is less precise than Double, and inaccurate if +T is more precise than Double. We can and should do better in Swift. + +With the changes in this proposal, the full implementation would become: +```swift +import Math + +func sigmoid(_ x: T) -> T where T: Real { + return 1/(1 + exp(-x)) +} +``` + +## Proposed solution + +There are four pieces of this proposal: first, we introduce the protocol +`ElementaryFunctions`: +```swift +public protocol ElementaryFunctions { + + /// The cosine of `x`. + static func cos(_ x: Self) -> Self + + /// The sine of `x`. + static func sin(_ x: Self) -> Self + + /// The tangent of `x`. + static func tan(_ x: Self) -> Self + + ... +} +``` +Conformance to this protocol means that the elementary functions are available as +static functions: +```swift +(swift) Float.exp(1) +// r0 : Float = 2.7182817 +``` +(For the full set of functions provided, see Detailed Design below). All of the standard +library `FloatingPoint` types conform to `ElementaryFunctions`; a future `Complex` +type would also conform. `SIMD` types do not conform themselves, but the operations +are defined on them when their scalar type conforms to the protocol. + +The second piece of the proposal is the protocol `Real`: +```swift +public protocol Real: FloatingPoint, ElementaryFunctions { + + /// `atan(y/x)` with quadrant fixup. + /// + /// There is an infinite family of angles whose tangent is `y/x`. + /// `atan2` selects the representative that is the angle between + /// the vector `(x, y)` and the real axis in the range [-π, π]. + static func atan2(y: Self, x: Self) -> Self + + /// The error function evaluated at `x`. + static func erf(_ x: Self) -> Self + + /// The complimentary error function evaluated at `x`. + static func erfc(_ x: Self) -> Self + + /// sqrt(x*x + y*y) computed without undue overflow or underflow. + /// + /// Returns a numerically precise result even if one or both of x*x or + /// y*y overflow or underflow. + static func hypot(_ x: Self, _ y: Self) -> Self + + ... +} +``` +This protocol does not add much API surface, but it is what most users will +write generic code against. The benefit of this protocol is that it allows +us to avoid multiple constraints for most simple generic functions, and to +adopt a clearer naming scheme for the protocol that most users see, while +also giving `ElementaryFunctions` a more precise name, suitable for +sophisticated uses. + +The third piece of the proposal is the introduction of generic free function +implementations of math operations: + +```swift +(swift) exp(1.0) +// r0 : Float = 2.7182817 +``` + +Finally, we will update the platform imports to obsolete existing functions +covered by the new free functions, and also remove the +imports of the suffixed functions (which were actually never +intended to be available in Swift). Updates will only be necessary with functions like `atan2(y: x:)` +where we are adding argument labels or `logGamma( )` where we have new +function names. In these cases we will deprecate the old functions instead +of obsoleting them to allow users time to migrate. + +## Detailed design + +The full API provided by `ElementaryFunctions` is as follows: +```swift +/// The square root of `x`. +/// +/// For real types, if `x` is negative the result is `.nan`. For complex +/// types there is a branch cut on the negative real axis. +static func sqrt(_ x: Self) -> Self + +/// The cosine of `x`, interpreted as an angle in radians. +static func cos(_ x: Self) -> Self + +/// The sine of `x`, interpreted as an angle in radians. +static func sin(_ x: Self) -> Self + +/// The tangent of `x`, interpreted as an angle in radians. +static func tan(_ x: Self) -> Self + +/// The inverse cosine of `x` in radians. +static func acos(_ x: Self) -> Self + +/// The inverse sine of `x` in radians. +static func asin(_ x: Self) -> Self + +/// The inverse tangent of `x` in radians. +static func atan(_ x: Self) -> Self + +/// The hyperbolic cosine of `x`. +static func cosh(_ x: Self) -> Self + +/// The hyperbolic sine of `x`. +static func sinh(_ x: Self) -> Self + +/// The hyperbolic tangent of `x`. +static func tanh(_ x: Self) -> Self + +/// The inverse hyperbolic cosine of `x`. +static func acosh(_ x: Self) -> Self + +/// The inverse hyperbolic sine of `x`. +static func asinh(_ x: Self) -> Self + +/// The inverse hyperbolic tangent of `x`. +static func atanh(_ x: Self) -> Self + +/// The exponential function applied to `x`, or `e**x`. +static func exp(_ x: Self) -> Self + +/// Two raised to to power `x`. +static func exp2(_ x: Self) -> Self + +/// Ten raised to to power `x`. +static func exp10(_ x: Self) -> Self + +/// `exp(x) - 1` evaluated so as to preserve accuracy close to zero. +static func expm1(_ x: Self) -> Self + +/// The natural logarithm of `x`. +static func log(_ x: Self) -> Self + +/// The base-two logarithm of `x`. +static func log2(_ x: Self) -> Self + +/// The base-ten logarithm of `x`. +static func log10(_ x: Self) -> Self + +/// `log(1 + x)` evaluated so as to preserve accuracy close to zero. +static func log1p(_ x: Self) -> Self + +/// `x**y` interpreted as `exp(y * log(x))` +/// +/// For real types, if `x` is negative the result is NaN, even if `y` has +/// an integral value. For complex types, there is a branch cut on the +/// negative real axis. +static func pow(_ x: Self, _ y: Self) -> Self + +/// `x` raised to the `n`th power. +/// +/// The product of `n` copies of `x`. +static func pow(_ x: Self, _ n: Int) -> Self + +/// The `n`th root of `x`. +/// +/// For real types, if `x` is negative and `n` is even, the result is NaN. +/// For complex types, there is a branch cut along the negative real axis. +static func root(_ x: Self, _ n: Int) -> Self +``` +`Real` builds on this set by adding the following additional operations that are either +difficult to implement for complex types or only make sense for real types: +```swift +/// `atan(y/x)` with quadrant fixup. +/// +/// There is an infinite family of angles whose tangent is `y/x`. `atan2` +/// selects the representative that is the angle between the vector `(x, y)` +/// and the real axis in the range [-π, π]. +static func atan2(y: Self, x: Self) -> Self + +/// The error function evaluated at `x`. +static func erf(_ x: Self) -> Self + +/// The complimentary error function evaluated at `x`. +static func erfc(_ x: Self) -> Self + +/// sqrt(x*x + y*y) computed without undue overflow or underflow. +/// +/// Returns a numerically precise result even if one or both of x*x or +/// y*y overflow or underflow. +static func hypot(_ x: Self, _ y: Self) -> Self + +/// The gamma function evaluated at `x`. +/// +/// For integral `x`, `gamma(x)` is `(x-1)` factorial. +static func gamma(_ x: Self) -> Self + +/// `log(gamma(x))` computed without undue overflow. +/// +/// `log(abs(gamma(x)))` is returned. To recover the sign of `gamma(x)`, +/// use `signGamma(x)`. +static func logGamma(_ x: Self) -> Self + +/// The sign of `gamma(x)`. +/// +/// This function is typically used in conjunction with `logGamma(x)`, which +/// computes `log(abs(gamma(x)))`, to recover the sign information that is +/// lost to the absolute value. +/// +/// `gamma(x)` has a simple pole at each non-positive integer and an +/// essential singularity at infinity; we arbitrarily choose to return +/// `.plus` for the sign in those cases. For all other values, `signGamma(x)` +/// is `.plus` if `x >= 0` or `trunc(x)` is odd, and `.minus` otherwise. +static func signGamma(_ x: Self) -> FloatingPointSign +``` +These functions directly follow the math library names used in most other +languages, as there is not a good reason to break with existing precedent. +The changes worth noting are as follows: +- `exp10` does not exist in most C math libraries. It is a generally useful +function, corresponding to `log10`. We'll fall back on implementing it as +`pow(10, x)` on platforms that don't have it in the system library. +- There are *two* functions named `pow` with different signatures. One +implements the IEEE 754 `powr` function (nan if `x` is negative), the other +restricts the exponent to have type `Int`, and implements the IEEE 754 `pown` +function. +- The function `root` does not exist in most math libraries; it computes the +nth root of x. For now this is implemented in terms of `pow`, but we may +adopt other implementations for better speed or accuracy in the future. +- Argument labels have been added to `atan2(y:x:)`. This is the only math.h +function whose argument order regularly causes bugs, so it would be good +to clarify here. +- `logGamma` is introduced instead of the existing `lgamma`, and returns a +single value instead of a tuple. The sign is still available via a new `signGamma` +function, but requires a separate function call. The +motivation for this approach is two-fold: first, the more common use case is +to want only the first value, so returning a tuple creates noise: + + let (result, _) = lgamma(x) + + Second, there's an outstanding bug that results from the C interfaces being +re-exported in Swift where `lgamma` is ambiguous; it can be either the +platform shim returning `(T, Int)`, or the C library function returning +`Double`; we want to deprecate the first and make the second unavailable. +Simultaneously introducing yet another function with the same name would +create a bit of a mess. + +### Future expansion +The following functions recommended by IEEE 754 are not provided at this point +(because implementations are not widely available), but are planned for future expansion, +possibly with implementation directly in Swift: `cospi`, `sinpi`, `tanpi`, `acospi`, `asinpi`, +`atanpi`, `exp2m1`, `exp10m1`, `log2p1`, `log10p1`, `compound` (these are the names +used by IEEE 754; Swift can use different names if we like). + +### Functions not defined on ElementaryFunctions +The following functions are exported by , but will not be defined on ElementaryFunctions: +`frexp`, `ilogb`, `ldexp`, `logb`, `modf`, `scalbn`, `scalbln`, `fabs`, `ceil`, +`floor`, `nearbyint`, `rint`, `lrint`, `llrint`, `round`, `lround`, `llround`, `trunc`, `fmod`, +`remainder`, `remquo`, `copysign`, `nan`, `nextafter`, `nexttoward`, `fdim`, `fmin`, `fmax`, +`fma`. + +Most of these are not defined on ElementaryFunctions because they are inherently bound to the +semantics of `FloatingPoint` or `BinaryFloatingPoint`, and so cannot be defined for +types such as Complex or Decimal. Equivalents to many of them are already defined on +`[Binary]FloatingPoint` anyway--in those cases free functions are defined as generic over `FloatingPoint` or `BinaryFloatingPoint`: +```swift +@available(swift, introduced: 5.1) +@_alwaysEmitIntoClient +public func ceil(_ x: T) -> T where T: FloatingPoint { + return x.rounded(.up) +} + +@available(swift, introduced: 5.1) +@_alwaysEmitIntoClient +public func floor(_ x: T) -> T where T: FloatingPoint { + return x.rounded(.down) +} + +@available(swift, introduced: 5.1) +@_alwaysEmitIntoClient +public func round(_ x: T) -> T where T: FloatingPoint { + return x.rounded() +} + +@available(swift, introduced: 5.1) +@_alwaysEmitIntoClient +public func trunc(_ x: T) -> T where T: FloatingPoint { + return x.rounded(.towardZero) +} + +@available(swift, introduced: 5.1) +@_alwaysEmitIntoClient +public func fma(_ x: T, _ y: T, _ z: T) -> T where T: FloatingPoint { + return z.addingProduct(x, y) +} + +@available(swift, introduced: 5.1) +@_alwaysEmitIntoClient +public func remainder(_ x: T, _ y: T) -> T where T: FloatingPoint { + return x.remainder(dividingBy: y) +} + +@available(swift, introduced: 5.1) +@_alwaysEmitIntoClient +public func fmod(_ x: T, _ y: T) -> T where T: FloatingPoint { + return x.truncatingRemainder(dividingBy: y) +} +``` +These definitions replace the definitions existing in the platform module. + +A few of the other functions (`nearbyint`, `rint`) are fundamentally tied to the +C language notion of dynamic floating-point rounding-modes, which is not modeled +by Swift (and which we do not have plans to support--even if Swift adds rounding-mode +control, we should avoid the C fenv model). These are deprecated. + +The remainder will not be moved into the Math module at this point, as they can +be written more naturally in terms of the `FloatingPoint` API. We intend to +deprecate them. + +## Source compatibility +This is an additive change, but it entails some changes for platform modules; the existing +platform implementations provided by the Darwin or GLibc should be deprecated and +made to redirect people to the new operations. + +## Effect on ABI stability +For the standard library, this is an additive change. We'll need to continue to support the +old platform hooks to provide binary stability, but will mark them deprecated or obsoleted. + +## Effect on API resilience +This is an additive change. + +## Alternatives considered +1. The name `ElementaryFunctions` is a marked improvement on the earlier `Mathable`, +but is still imperfect. As discussed above, the introduction of the `Real` protocol mostly +renders this issue moot; most code will be constrained to that instead. + +2. The names of these functions are strongly conserved across languages, but they are +not universal; we *could* consider more verbose names inline with Swift usual patterns. +`sine`, `cosine`, `inverseTangent`, etc. This holds some appeal especially for the more +niche functions (like `expm1`), but the weight of common practice is pretty strong here; +almost all languages use essentially the same names for these operations. Another +alternative would be to break these up into `TrigonometricFunctions`, +`HyperbolicFunctions`, `ExponentialFunctions`, etc, but I don't think that actually +buys us very much. + +3. We may also want to add `log(_ base: T, _ x: T) -> T` at some future point as a +supplement to the existing `log`, `log2`, and `log10` functions. Python and Julia both +provide a similar interface. Doing this correctly requires a building block that the C math +library doesn't provide (an extra-precise `log` or `log2` that returns a head-tail +representation of the result); without this building block rounding errors creep in even for +exact cases: + + >>> from math import log + >>> log(3**20, 3) + 19.999999999999996 + + Julia includes a warning about this in their documentation that basically says "Use log2 +or log10 instead if base is 2 or 10". We could take that approach, but base 2 and 10 +cover 99% of uses, so I would rather wait to provide this function until we have time to +do it correctly. + +4. We could spell `log` (the natural logarithm) `ln`. This would resolve some ambiguity for +users with a background in certain fields, at the cost of diverging from the vast majority +of programming languages. Rust and Kotlin do spell it this way, so we wouldn't be completely +alone. It would also avoid using a function name that potentially conflicts (visually or +syntactically) with an obvious name for logging facilities. However, depending on font, +`ln` is easily confused with `in`, and it breaks the similarity with the other `log` functions. +As an assistance, we will add `ln` in the `Math` module but mark it unavailable, referring +users to `log`. + +5. We could put the free functions into a separate module instead of the standard library. +Having them in a separate module helps avoid adding stuff to the global namespace +unless you're actually using it, which is generally nice, and the precedent from other +languages is pretty strong here: `#include `, `import math`, etc. However, +the general utility of having these functions available in the global namespace is felt +to outweigh this. + +6. We could define an operator like `^` or `**` for one or both definitions of `pow`. I +have opted to keep new operators out of this proposal, in the interests of focusing on +the functions and their implementation hooks. I would consider such an operator to be an +additive change to be considered in a separate proposal. + +7. Add the constants `pi` and `e` to `T.Math`. There's a bit of a question about how to +handle these with hypothetical arbitrary-precision types, but that's not a great reason +to keep them out for the concrete types we already have. Plus we already have `pi` on +FloatingPoint, so someone implementing such a type already needs to make a decision about +how to handle it. There's a second question of how to handle these with `Complex` or +`SIMD` types; one solution would be to only define them for types conforming to `Real`. + +## Changes from previous revisions +1. A number of functions (`atan2`, `erf`, `erfc`, `gamma`, `logGamma`) have been moved +from `ElementaryFunctions` onto `Real`. The rationale for this is threefold: `atan2` never +made much sense for non-real arguments. Implementations of `erf`, `erfc`, `gamma` and +`logGamma` are not available on all platforms. Finally, those four functions are not actually +"elementary", so the naming is more accurate following this change. + +2. `hypot` has been added to `Real`. We would like to have a more general solution for +efficiently rescaling computations in the future, but `hypot` is a tool that people can use +today. + +3. I have dropped the `.Math` pseudo-namespace associatedtype from the protocols. +In earlier versions of the proposal, the static functions were spelled `Float.Math.sin(x)`. +This was motivated by a desire to avoid "magic" underscore-prefixed implementation +trampolines, while still grouping these functions together under a single `.Math` in +autocompletion lists. Whatever stylistic benefits this might have were judged to be not +worth the extra layer of machinery that would be fixed into the ABI even if we get a "real" +namespace mechanism at some future point. These functions now appear simply as +`Float.sin(x)`. diff --git a/proposals/0247-contiguous-strings.md b/proposals/0247-contiguous-strings.md new file mode 100644 index 0000000000..69527d9163 --- /dev/null +++ b/proposals/0247-contiguous-strings.md @@ -0,0 +1,228 @@ +# Contiguous Strings + +* Proposal: [SE-0247](0247-contiguous-strings.md) +* Author: [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.1)** +* Implementation: [apple/swift#23051](https://github.com/apple/swift/pull/23051) +* Bugs: [SR-6475](https://bugs.swift.org/browse/SR-6475) + +## Introduction + +One of the most common API requests from performance-minded users of string is a way to get direct access to the raw underlying code units. Now that [Swift 5 uses UTF-8](https://forums.swift.org/t/string-s-abi-and-utf-8/17676) for its preferred encoding, we can provide this. + +“Contiguous strings” are strings that are capable of providing a pointer and length to [validly encoded](https://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences) UTF-8 contents in constant time. Contiguous strings include: + +* All native Swift string forms +* Lazily bridged Cocoa strings that provide a pointer to contiguous ASCII +* Any “shared string” concept we may add in the future + +Noncontiguous strings include: + +* Lazily-bridged Cocoa strings that don’t provide a pointer to contiguous ASCII (even if they do have contiguous UTF-16 code units) +* Any “foreign string” concept we may add in the future + +Contiguous strings are heavily optimized and more efficient to process than noncontiguous strings. However, noncontiguous string contents don’t have to be copied when imported into Swift, which is profitable for strings which may never be read from, such as imported NSString dictionary keys. + +Swift-evolution thread: [Pitch: Contiguous Strings](https://forums.swift.org/t/pitch-contiguous-strings/21206) + +## Motivation + +In Swift 5.0, `String.UTF8View` supports `withContiguousStorageIfAvailable()`, which succeeds on contiguous strings but returns `nil` on noncontiguous strings. While it’s nice to run a fast algorithm when you can, users are left in the dark if this doesn’t succeed. Even if it’s known to probably succeed, guarding against `nil` and having a valid fall-back path is not ergonomic. + +## Proposed solution + +We propose adding to String and Substring: + +* A way to query if a string is contiguous +* A way to force a string to be contiguous +* A way to run a closure over the raw UTF-8 contents of a string + +(For rationale on why StringProtocol was excluded, see “What about StringProtocol?” in Alternatives Considered) + +## Detailed design + +```swift +extension String { + /// Returns whether this string is capable of providing access to + /// validly-encoded UTF-8 contents in contiguous memory in O(1) time. + /// + /// Contiguous strings always operate in O(1) time for withUTF8 and always + /// give a result for String.UTF8View.withContiguousStorageIfAvailable. + /// Contiguous strings also benefit from fast-paths and better optimizations. + /// + public var isContiguousUTF8: Bool { get } + + /// If this string is not contiguous, make it so. If this mutates the string, + /// it will invalidate any pre-existing indices. + /// + /// Complexity: O(n) if non-contiguous, O(1) if already contiguous + /// + public mutating func makeContiguousUTF8() + + /// Runs `body` over the content of this string in contiguous memory. If this + /// string is not contiguous, this will first make it contiguous, which will + /// also speed up subsequent access. If this mutates the string, + /// it will invalidate any pre-existing indices. + /// + /// Note that it is unsafe to escape the pointer provided to `body`. For + /// example, strings of up to 15 UTF-8 code units in length may be represented + /// in a small-string representation, and thus will be spilled into + /// temporary stack space which is invalid after `withUTF8` finishes + /// execution. + /// + /// Complexity: O(n) if non-contiguous, O(1) if already contiguous + /// + public mutating func withUTF8( + _ body: (UnsafeBufferPointer) throws -> R + ) rethrows -> R +} + +// Contiguous UTF-8 strings +extension Substring { + /// Returns whether this string is capable of providing access to + /// validly-encoded UTF-8 contents in contiguous memory in O(1) time. + /// + /// Contiguous strings always operate in O(1) time for withUTF8 and always + /// give a result for String.UTF8View.withContiguousStorageIfAvailable. + /// Contiguous strings also benefit from fast-paths and better optimizations. + /// + public var isContiguousUTF8: Bool { get } + + /// If this string is not contiguous, make it so. If this mutates the + /// substring, it will invalidate any pre-existing indices. + /// + /// Complexity: O(n) if non-contiguous, O(1) if already contiguous + /// + public mutating func makeContiguousUTF8() + + /// Runs `body` over the content of this substring in contiguous memory. If + /// this substring is not contiguous, this will first make it contiguous, + /// which will also speed up subsequent access. If this mutates the substring, + /// it will invalidate any pre-existing indices. + /// + /// Note that it is unsafe to escape the pointer provided to `body`. For + /// example, strings of up to 15 UTF-8 code units in length may be represented + /// in a small-string representation, and thus will be spilled into + /// temporary stack space which is invalid after `withUTF8` finishes + /// execution. + /// + /// Complexity: O(n) if non-contiguous, O(1) if already contiguous + /// + public mutating func withUTF8( + _ body: (UnsafeBufferPointer) throws -> R + ) rethrows -> R +} +``` + +(For rationale as to why `withUTF8` is marked `mutating`, see “Wait, why is `withUTF8` marked as `mutating`?” in Alternatives Considered.) + +## Source compatibility + +All changes are additive. + + +## Effect on ABI stability + +All changes are additive. ABI-relevant attributes are provided in “Detailed design”. + + +## Effect on API resilience + +The APIs for String and Substring have their implementations exposed, as they are expressed in terms of assumptions and mechanisms already (non-resiliently) present in Swift 5.0. + + +## Alternatives considered + +### Wait, why is `withUTF8` marked as `mutating`? + +If the string is noncontiguous, something has to happen to get contiguous UTF-8 contents in memory. We have 3 basic options here: + +##### 1. Copy into a new heap allocation, run the closure, then throw it away + +This approach takes inspiration from Array’s `withUnsafeBufferPointer`, which runs directly on its contents if it can, otherwise it makes a temporary copy to run over. However, unlike Array, noncontiguous strings are much more common and there are no type-level guarantees of contiguity. For example, Array with a value-type `Element` is always contiguous, so the vast majority of performance sensitive operations over arrays (e.g. `Array`) can never trigger this allocation. + +Because String does not have Array’s guarantee, this approach would harm the ability to reason about the performance characteristics of code. This approach would also keep the string on the slow-path for access after the method is called. If the contents are worth reading, they’re worth ensuring contiguity. + +##### 2. Trap if noncontiguous + +This would be the ideal API for platforms without Cocoa interoperability (unless/until we introduce noncontiguous foreign strings, at least), as it will always succeed. However, this would be a terrible source of crashes in libraries and applications that forget to exhaustively test their code with noncontiguous strings. This would hit cross-platform packages especially hard. + +Even for libraries confined to Cocoa-interoperable platforms, whether an imported NSString is contiguous or not could change version-to-version of the OS, producing difficult to reason about bugs. + +##### 3. Fine, make it `mutating` + +The proposed compromise makes `withUTF8` mutating, forcing the contents to be bridged if noncontiguous. This has the downside of forcing such strings to be declared `var` rather than `let`, but mitigates the downsides of the other approaches. If the contents are worth reading, they’re worth ensuring contiguity. + +##### Or, introduce a separate type, `ContiguousUTF8String`, instead + +Introducing a new type would introduce an API schism and further complicate the +nature of StringProtocol. We don't consider the downsides to be worth the upside +of dropping `mutable`. + +### What about StringProtocol? + +Adding these methods to StringProtocol would allow code to be generic over String and Substring (and in the somewhat unlikely event we add new conformers, those as well). This can be done at any time, but there are some issues with doing this right now. + +When it comes to adding these methods on StringProtocol, we have two options: + +##### 1. Add it as a protocol extension + +This approach would add the functions in an extension, ideally resilient and versioned. Unfortunately, `makeContiguousUTF8()` wouldn’t be implementable, as we cannot reassign self to a concrete type, so we’d have to add some additional requirements to StringProtocol surrounding forming a new contiguous string of the same concrete type. + +##### 2. Add it as a requirement with a default implementation + +This approach would add it as a customization hook that’s dynamically dispatched to the concrete type’s real implementations. The default implementation would be resilient and versioned and trap if called; any new conformers to StringProtocol would need to be versioned and accommodated here. + +Adding new versioned-and-defaulted requirements to a protocol can be done at any point while preserving ABI stability. For now, we’re not sure it’s worth the extra witness table entries at this point. This also hits some of the pain points of option 1: any conformers to StringProtocol must satisfy `makeContiguousUTF8`’s post-condition of ensuring contiguity without changing concrete type. + +This can be added in the future. + +### Put this on the UTF8View + +Contiguous UTF-8 is baked into String's ABI as the fastest representation; this +is about directly exposing that fact to performance-minded users. Other views +will not have this, so we believe it makes sense to put directly on String. +UTF-8 is special for performance and contiguity concerns. + +### The Most Viable Alternative + +Here's an alternative formulation that avoids `mutating` and +index-invalidation (by using strategy \#1 above). + +```swift +extension [Sub]String { + /// + public var isContiguousUTF8: Bool { get } + + /// Produce a contiguous UTF-8 string with the same contents as `self`. + /// Returns `self` is already contiguous, otherwise returns a new copy. + /// + /// Complexity: O(1) if `self` is contiguous UTF-8, otherwise O(n) + /// + public var contiguousUTF8: [Sub]String { get } + + /// Runs `body` over the content of this string in contiguous memory. If this + /// string is not contiguous UTF-8, this will first make a contiguous copy and + /// run over that. + /// + /// Note that it is unsafe to escape the pointer provided to `body`, even if + /// `self` is contiguous UTF-8. For example, strings of up to 15 UTF-8 code + /// units in length may be represented in a small-string representation, and + /// thus will be spilled into temporary stack space which is invalid after + /// `withUTF8` returns. + /// + /// Complexity: O(1) if `self` is contiguous UTF-8, otherwise O(n) + /// + public func withUTF8( + _ body: (UnsafeBufferPointer) throws -> R + ) rethrows -> R +} +``` + +This formulation would be preferable for situations where strings are nearly- +always contiguous UTF-8. Also, the computed property is less invasive than a +mutating method as it can be used on the right-hand-side of a let binding. + +Choosing between this and the version proposed depends on how users expect to +use it. We're eagerly awaiting feedback during the review. diff --git a/proposals/0248-string-gaps-missing-apis.md b/proposals/0248-string-gaps-missing-apis.md new file mode 100644 index 0000000000..75b250aa83 --- /dev/null +++ b/proposals/0248-string-gaps-missing-apis.md @@ -0,0 +1,237 @@ +# String Gaps and Missing APIs + +* Proposal: [SE-0248](0248-string-gaps-missing-apis.md) +* Author: [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.1)** +* Implementation: [apple/swift#22869](https://github.com/apple/swift/pull/22869) +* Bugs: [SR-9955](https://bugs.swift.org/browse/SR-9955) + +## Introduction + +String and related types are missing trivial and obvious functionality, much of which currently exists internally but has not been made API. We propose adding 9 new methods/properties and 3 new code unit views. + +Swift-evolution thread: [Pitch: String Gaps and Missing APIs](https://forums.swift.org/t/pitch-string-gaps-and-missing-apis/20984) + +## Motivation + +These missing APIs address [commonly encountered](https://forums.swift.org/t/efficiently-retrieving-utf8-from-a-character-in-a-string/19916) gaps and [missing functionality](https://bugs.swift.org/browse/SR-9955) for users of String and its various types, often leading developers to [reinvent](https://github.com/apple/swift-nio-http2/blob/master/Sources/NIOHPACK/HPACKHeader.swift#L412) the same trivial definitions. + +## Proposed solution + +We propose: + +* 6 simple APIs on Unicode’s various encodings +* 2 generic initializers for string indices and ranges of indices +* `Substring.base`, equivalent to `Slice.base` +* Make `Character.UTF8View` and `Character.UTF16View` public +* Add `Unicode.Scalar.UTF8View` + +## Detailed design + +### 1. Unicode obvious/trivial additions + +This functionality existed internally as helpers and is generally useful (even if they’re simple) for anyone working with Unicode. + +```swift + +extension Unicode.ASCII { + /// Returns whether the given code unit represents an ASCII scalar + public static func isASCII(_ x: CodeUnit) -> Bool +} + +extension Unicode.UTF8 { + /// Returns the number of code units required to encode the given Unicode + /// scalar. + /// + /// Because a Unicode scalar value can require up to 21 bits to store its + /// value, some Unicode scalars are represented in UTF-8 by a sequence of up + /// to 4 code units. The first code unit is designated a *lead* byte and the + /// rest are *continuation* bytes. + /// + /// let anA: Unicode.Scalar = "A" + /// print(anA.value) + /// // Prints "65" + /// print(UTF8.width(anA)) + /// // Prints "1" + /// + /// let anApple: Unicode.Scalar = "🍎" + /// print(anApple.value) + /// // Prints "127822" + /// print(UTF16.width(anApple)) + /// // Prints "4" + /// + /// - Parameter x: A Unicode scalar value. + /// - Returns: The width of `x` when encoded in UTF-8, from `1` to `4`. + public static func width(_ x: Unicode.Scalar) -> Int + + /// Returns whether the given code unit represents an ASCII scalar + public static func isASCII(_ x: CodeUnit) -> Bool +} + +extension Unicode.UTF16 { + /// Returns a Boolean value indicating whether the specified code unit is a + /// high or low surrogate code unit. + public static func isSurrogate(_ x: CodeUnit) -> Bool + + /// Returns whether the given code unit represents an ASCII scalar + public static func isASCII(_ x: CodeUnit) -> Bool +} + +extension Unicode.UTF32 { + /// Returns whether the given code unit represents an ASCII scalar + public static func isASCII(_ x: CodeUnit) -> Bool +} + +``` + +### 2. Generic initializers for String.Index and Range + +Concrete versions of this exist parameterized over String, but versions generic over StringProtocol are missing. + +```swift +extension String.Index { + /// Creates an index in the given string that corresponds exactly to the + /// specified position. + /// + /// If the index passed as `sourcePosition` represents the start of an + /// extended grapheme cluster---the element type of a string---then the + /// initializer succeeds. + /// + /// The following example converts the position of the Unicode scalar `"e"` + /// into its corresponding position in the string. The character at that + /// position is the composed `"é"` character. + /// + /// let cafe = "Cafe\u{0301}" + /// print(cafe) + /// // Prints "Café" + /// + /// let scalarsIndex = cafe.unicodeScalars.firstIndex(of: "e")! + /// let stringIndex = String.Index(scalarsIndex, within: cafe)! + /// + /// print(cafe[...stringIndex]) + /// // Prints "Café" + /// + /// If the index passed as `sourcePosition` doesn't have an exact + /// corresponding position in `target`, the result of the initializer is + /// `nil`. For example, an attempt to convert the position of the combining + /// acute accent (`"\u{0301}"`) fails. Combining Unicode scalars do not have + /// their own position in a string. + /// + /// let nextScalarsIndex = cafe.unicodeScalars.index(after: scalarsIndex) + /// let nextStringIndex = String.Index(nextScalarsIndex, within: cafe) + /// + /// print(nextStringIndex) + /// // Prints "nil" + /// + /// - Parameters: + /// - sourcePosition: A position in a view of the `target` parameter. + /// `sourcePosition` must be a valid index of at least one of the views + /// of `target`. + /// - target: The string referenced by the resulting index. + public init?( + _ sourcePosition: String.Index, within target: S + ) +} + +extension Range where Bound == String.Index { + public init?(_ range: NSRange, in string: __shared S) +} +``` + +### 3. Substring provides access to its base + +Slice, the default SubSequence type, provides `base` for accessing the original Collection. Substring, String’s SubSequence, should as well. + +```swift +extension Substring { + /// Returns the underlying string from which this Substring was derived. + public var base: String { get } +} + +``` + +### 4. Add in missing views on Character + +Character’s UTF8View and UTF16View has existed internally, but we should make it public. + +```swift + +extension Character { + /// A view of a character's contents as a collection of UTF-8 code units. See + /// String.UTF8View for more information + public typealias UTF8View = String.UTF8View + + /// A UTF-8 encoding of `self`. + public var utf8: UTF8View { get } + + /// A view of a character's contents as a collection of UTF-16 code units. See + /// String.UTF16View for more information + public typealias UTF16View = String.UTF16View + + /// A UTF-16 encoding of `self`. + public var utf16: UTF16View { get } +} +``` + + +### 5. Add in a RandomAccessCollection UTF8View on Unicode.Scalar + +Unicode.Scalar has a UTF16View with is a RandomAccessCollection, but not a UTF8View. + +```swift +extension Unicode.Scalar { + public struct UTF8View { + internal init(value: Unicode.Scalar) + internal var value: Unicode.Scalar + } + + public var utf8: UTF8View { get } +} + +extension Unicode.Scalar.UTF8View : RandomAccessCollection { + public typealias Indices = Range + + /// The position of the first code unit. + public var startIndex: Int { get } + + /// The "past the end" position---that is, the position one + /// greater than the last valid subscript argument. + /// + /// If the collection is empty, `endIndex` is equal to `startIndex`. + public var endIndex: Int { get } + + /// Accesses the code unit at the specified position. + /// + /// - Parameter position: The position of the element to access. `position` + /// must be a valid index of the collection that is not equal to the + /// `endIndex` property. + public subscript(position: Int) -> UTF8.CodeUnit +} +``` + +## Source compatibility + +All changes are additive. + +## Effect on ABI stability + +All changes are additive. + +## Effect on API resilience + +* Unicode encoding additions and `Substring.base` are trivial and can never change in definition, so their implementations are exposed. +* `String.Index` initializers are resilient and versioned. +* Character’s views already exist as inlinable in 5.0, we just replace `internal` with `public` +* Unicode.Scalar.UTF8View's implementation is fully exposed (for performance), but is versioned + +## Alternatives considered + +### Do Nothing + +Various flavors of “do nothing” include stating a given API is not useful or waiting for a rethink of some core concept. Each of these API gaps frequently come up on the forums, bug reports, or seeing developer usage in the wild. Rethinks are unlikely to happen anytime soon. We believe these gaps should be closed immediately. + +### Do More + +This proposal is meant to round out holes and provide some simple additions, keeping the scope narrow for Swift 5.1. We could certainly do more in all of these areas, but that would require a more design iteration and could be dependent on other missing functionality. + diff --git a/proposals/0249-key-path-literal-function-expressions.md b/proposals/0249-key-path-literal-function-expressions.md new file mode 100644 index 0000000000..2986b4b772 --- /dev/null +++ b/proposals/0249-key-path-literal-function-expressions.md @@ -0,0 +1,178 @@ +# Key Path Expressions as Functions + +* Proposal: [SE-0249](0249-key-path-literal-function-expressions.md) +* Authors: [Stephen Celis](https://github.com/stephencelis), [Greg Titus](https://github.com/gregomni) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.2)** +* Implementation: [apple/swift#26054](https://github.com/apple/swift/pull/26054) + +## Introduction + +This proposal introduces the ability to use the key path expression `\Root.value` wherever functions of `(Root) -> Value` are allowed. + +Swift-evolution thread: [Key Path Expressions as Functions](https://forums.swift.org/t/key-path-expressions-as-functions/19587) + +Previous discussions: + +- [Allow key path literal syntax in expressions expecting function type](https://forums.swift.org/t/allow-key-path-literal-syntax-in-expressions-expecting-function-type/16453) +- [Key path getter promotion](https://forums.swift.org/t/key-path-getter-promotion/11185) +- [[Pitch] KeyPath based map, flatMap, filter](https://forums.swift.org/t/pitch-keypath-based-map-flatmap-filter/6266) + +## Motivation + +One-off closures that traverse from a root type to a value are common in Swift. Consider the following `User` struct: + +```swift +struct User { + let email: String + let isAdmin: Bool +} +``` + +Applying `map` allows the following code to gather an array of emails from a source user array: + +```swift +users.map { $0.email } +``` + +Similarly, `filter` can collect an array of admins: + +```swift +users.filter { $0.isAdmin } +``` + +These ad hoc closures are short and sweet but Swift already has a shorter and sweeter syntax that can describe this: key paths. The Swift forum has [previously proposed](https://forums.swift.org/t/pitch-support-for-map-and-flatmap-with-smart-key-paths/6073) adding `map`, `flatMap`, and `compactMap` overloads that accept key paths as input. Popular libraries [define overloads](https://github.com/ReactiveCocoa/ReactiveSwift/search?utf8=✓&q=KeyPath&type=) of their own. Adding an overload per function, though, is a losing battle. + +## Proposed solution + +Swift should allow `\Root.value` key path expressions wherever it allows `(Root) -> Value` functions: + +```swift +users.map(\.email) + +users.filter(\.isAdmin) +``` + +## Detailed design + +As implemented in [apple/swift#19448](https://github.com/apple/swift/pull/19448), occurrences of `\Root.value` are implicitly converted to key path applications of `{ $0[keyPath: \Root.value] }` wherever `(Root) -> Value` functions are expected. For example: + +``` swift +users.map(\.email) +``` + +Is equivalent to: + +``` swift +users.map { $0[keyPath: \User.email] } +``` + +The implementation is limited to key path literal expressions (for now), which means the following is not allowed: + +``` swift +let kp = \User.email // KeyPath +users.map(kp) +``` + +> 🛑 Cannot convert value of type 'WritableKeyPath' to expected argument type '(Person) throws -> String' + +But the following is: + +``` swift +let f1: (User) -> String = \User.email +users.map(f1) + +let f2: (User) -> String = \.email +users.map(f2) + +let f3 = \User.email as (User) -> String +users.map(f3) + +let f4 = \.email as (User) -> String +users.map(f4) +``` + +Any key path expression can be used where a function of the same shape is expected. A few more examples include: + +``` swift +// Multi-segment key paths +users.map(\.email.count) + +// `self` key paths +[1, nil, 3, nil, 5].compactMap(\.self) +``` + +### Precise semantics + +*(Note: Added after acceptance to clarify the proposed behavior.)* + +When inferring the type of a key path literal expression like `\Root.value`, the type checker will prefer `KeyPath` or one of its subtypes, but will also allow `(Root) -> Value`. If it chooses `(Root) -> Value`, the compiler will generate a closure with semantics equivalent to capturing the key path and applying it to the `Root` argument. For example: + +```swift +// You write this: +let f: (User) -> String = \User.email + +// The compiler generates something like this: +let f: (User) -> String = { kp in { root in root[keyPath: kp] } }(\User.email) +``` + +The compiler may generate any code that has the same semantics as this example; it might not even use a key path at all. + +Any side effects of the key path expression are evaluated when the closure is formed, not when it is called. In particular, if the key path contains subscripts, their arguments are evaluated once, when the closure is formed: + +```swift +var nextIndex = 0 +func makeIndex() -> Int { + defer { nextIndex += 1 } + return nextIndex +} + +let getFirst: ([Int]) -> Int = \Array.[makeIndex()] // Calls makeIndex(), gets 0, forms \Array.[0] +let getSecond: ([Int]) -> Int = \Array.[makeIndex()] // Calls makeIndex(), gets 1, forms \Array.[1] + +assert(getFirst([1, 2, 3]) == 1) // No matter how many times +assert(getFirst([1, 2, 3]) == 1) // you call getFirst(), +assert(getFirst([1, 2, 3]) == 1) // it always returns root[0]. + +assert(getSecond([1, 2, 3]) == 2) // No matter how many times +assert(getSecond([1, 2, 3]) == 2) // you call getSecond(), +assert(getSecond([1, 2, 3]) == 2) // it always returns root[1]. +``` + +## Effect on source compatibility, ABI stability, and API resilience + +This is a purely additive change and has no impact. + +## Future direction + +### `@callable` + +It was suggested in [the proposal thread](https://forums.swift.org/t/key-path-expressions-as-functions/19587/4) that a future direction in Swift would be to introduce a `@callable` mechanism or `Callable` protocol as a static equivalent of `@dynamicCallable`. Functions could be treated as the existential of types that are `@callable`, and `KeyPath` could be `@callable` to adopt the same functionality as this proposal. Such a change would be backwards-compatible with this proposal and does not need to block its implementation. + +### `ExpressibleByKeyPathLiteral` protocol + +It was also suggested [in the implementation's discussion](https://github.com/apple/swift/pull/19448) that it might be appropriate to define an `ExpressibleByKeyPathLiteral` protocol, though discussion in [the proposal thread](https://forums.swift.org/t/key-path-expressions-as-functions/19587/14) questioned the limited utility of such a protocol. + +## Alternatives considered + +### `^` prefix operator + +The `^` prefix operator offers a common third party solution for many users: + +```Swift +prefix operator ^ + +prefix func ^ (keyPath: KeyPath) -> (Root) -> Value { + return { root in root[keyPath: keyPath] } +} + +users.map(^\.email) + +users.filter(^\.isAdmin) +``` + +Although handy, it is less readable and less convenient than using key path syntax alone. + +### Accept `KeyPath` instead of literal expressions + +There has been some concern expressed that accepting the literal syntax but not key paths may be confusing, though this behavior is in line with how other literals work, and the most general use case will be with literals, not key paths that are passed around. Accepting key paths directly would also be more limiting and prevent exploring the [future directions](#future-direction) of `Callable` or `ExpressibleByKeyPathLiteral` protocols. diff --git a/proposals/0250-swift-style-guide-and-formatter.md b/proposals/0250-swift-style-guide-and-formatter.md new file mode 100644 index 0000000000..d73191a9fb --- /dev/null +++ b/proposals/0250-swift-style-guide-and-formatter.md @@ -0,0 +1,211 @@ +# Swift Code Style Guidelines and Formatter + +* Proposal: [SE-0250](0250-swift-style-guide-and-formatter.md) +* Authors: [Tony Allevato](https://github.com/allevato), [Dave Abrahams](https://github.com/dabrahams) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Returned for revision** +* Discussion thread: [An Official Style Guide and Formatter for Swift](https://forums.swift.org/t/pitch-an-official-style-guide-and-formatter-for-swift/21025) + +## Introduction + +We propose that the Swift project adopt a set of code style +guidelines and provide a formatting tool that lets users easily +diagnose and update their code according to those guidelines. +These guidelines would _not_ be mandatory for all projects, but +encouraged for Swift code to follow for general consistency. + +## Motivation + +At the time of this writing, there is no single style agreed on +by developers using Swift. Indeed, even Apple's own Swift +projects on GitHub—such as the standard library, Foundation, +Swift NIO, and so forth—have adopted their own varying styles. +Furthermore, in many cases the code in those projects—despite +the owners' best efforts—is not always completely consistent in +terms of style. + +In the absence of strict or recommended language-specific +guidelines, many organizations adopt company-wide or project-wide +style guides, which other developers may, and do, choose to +adopt. But creating code style guidelines that are maintained by +the language owners and community, along with tooling that allows +users to easily adopt those guidelines, provides a number of +additional benefits: + +1. The guidelines serve as a clear and easily referenceable + source of language best practices and patterns, rather than + developers trying to glean these by reading existing code. +1. Developers can move from one codebase to another without + incurring the mental load of learning and conforming to a + different style or being required to reconfigure their + development environment. +1. Developers spend less time worrying about how to format their + code and more on the program's logic. +1. Likewise, code reviewers spend less time commenting on code + formatting issues and more on its logic and design. + +The first two points in particular align well with the Swift +team's goal of making the language easy to learn. They also +remove learning barriers for developers who want to contribute +to a new open-source project, or to the language itself. + +## Proposed solution + +This proposal consists of two parts, discussed below: + +1. We propose that Swift adopt a set of code style guidelines for + the Swift language. +2. We propose formally adopting a formatting tool into the Swift + project that will allow users to update their code in + accordance with those guidelines. + +### Style Guide + +This meta-proposal does not attempt to define any specific style +guidelines. Its purpose is to answer the following existential +question: + +> Should the Swift language adopt a set of code style guidelines +> and a formatting tool? + +If the answer to this is "yes", then **subsequent proposals** +will be pitched to discuss and ratify those guidelines. In order +to keep those discussions scoped and focused, we plan to present +proposed guidelines to the community in multiple phases with each +centered around a particular theme, such as (but not necessarily +limited to) **typographical concerns,** **code consistency,** +and **best practices.** + +The proposal authors wish to emphasize that **we are not +proposing that users be required or forced** to use a particular +set of style conventions. The Swift compiler will **not** be +changed in any way that would prevent otherwise syntactically +valid code from compiling. Users who wish to reject the style +guidelines and adopt a different style for their own projects are +free to do so without the tooling pushing back on that decision. + +### Formatting Tool + +If the proposal is accepted, the Swift project will adopt an +official code formatting tool. The adoption of such a tool into +the Swift project will not preclude other similar tools being +written, but the expectation is that this tool will be officially +maintained as part of the Swift project and will (once the +details are decided) format users' code in accordance with the +accepted code style guidelings. + +The proposal authors ([among others](#acknowledgments)) have +collaborated on the `swift-format` tool currently hosted at +https://github.com/google/swift/tree/format and propose its +adoption into the Swift project. We propose this specific tool +because it satisfies all of the following goals: + +* It is syntax-oriented, which provides high reliability and + performance (especially once it adopts recently developed + in-process parsing APIs) as compared to SourceKit-based + solutions. +* It uses SwiftSyntax to process code—the Swift project's + preferred method of developing such tools—rather than a + distinct parsing implementation that must separately track + language evolution. +* It comes with a continuing support commitment from active + maintainers. + +The tool will be used as part of evaluating options for the +proposed code style guidelines, as part of a follow-up proposal on +the details of the guidelines themselves. + +### Configurability of Formatting + +`swift-format` will allow configuration of some practical +formatting decisions like indentation size, line length, and +respecting existing newlines. In mixed-language projects, some +tools in a developer's workflow may not easily support +configuring these on a per-language basis. + +We are also willing to consider additional degrees of +configurability. A tool that is not configurable only works for +users who are completely satisfied with the defaults. A tool that +is configurable is still usable by anyone who wants to leave it +configured to the default settings, but can also be tailored to +the unique needs of individual code bases. Even if style +guidelines ratified later encourage a particular default +configuration, users with different needs should still be able to +reap benefits from using the tool. + +As with the style guidelines above, the adopted formatting tool +will not be forced upon a developer's workflow by any part of the +Swift toolchain. Users who wish not to use it will have the +option to simply not run it on their code. + +## Alternatives considered + +We could not propose any particular style guidelines and leave it +to individual developers and teams to create their own (if they +so desired). That does not address the points listed in +[Motivation](#motivation) above. + +We could propose style guidelines but no official formatting +tool. However, we feel that a tool that works out-of-the-box +without any other installation requirements or mandatory +configuration is a major benefit to users. The existence of such +a tool does not diminish the value of other tools that aim to +enforce good coding patterns, be they the same, complementary, or +outright different patterns than those proposed in future Swift +coding style guidelines. + +We could make style guidelines mandatory, or at least enforced in +a very opinionated manner by the formatter (similar to Go). We +have chosen not to do so given that Swift is a well-established +language. Users who are happy with the default guidelines can +simply use them as-is, developers who have different preferences +are not unnecessarily constrained. + +Some Swift users have suggested that instead of proposing any +style guidelines, tooling should be able to transform code +to the developer's personal style upon checkout and then back to +some canonical style upon check-in, allowing individual +developers to code in whatever style they wished. While such +ideas are intriguing, we find them to be more of an academic +curiosity than a practical solution: + +* Varying personal styles would hinder team communication. Team + members should be able to discuss code on a whiteboard without + it looking foreign to other people in the room, and to make + API and language design decisions based on a clear idea of how + the code will look. +* This approach assumes that all tools on all platforms used in + the developer's workflow support this approach. The development + experience would suffer if the code does not use the same + format locally as it does on their code review system, if + remote builds reported errors at different line numbers because + they used a checked-in snapshot with a different style, or if + symbolicated crash logs contain line numbers that must be + matched to one specific "rendering" of the project's source + code long after the fact. +* If the source of truth of the source code is saved in some + canonical format and transformed when checked in/out, then + there must still be some decision about what that canonical + style _is._ + +Indeed, nothing in this proposal would _prevent_ a developer from +using a workflow like the one described above, if they wished to +implement it. + +## Acknowledgments + +We gratefully acknowledge the following contributors, without +whom this work would not have been possible: + +* the other contributors to `swift-format`: Austin Belknap + ([@dabelknap](https://github.com/dabelknap)), + Harlan Haskins ([@harlanhaskins](https://github.com/harlanhaskins)), + Alexander Lash ([@abl](https://github.com/abl)), + Lauren White ([@LaurenWhite](https://github.com/LaurenWhite)), + and Andrés Tamez Hernandez ([@atamez31](https://github.com/atamez31)), +* Argyrios Kyrtzidis ([@akyrtzi](https://github.com/akyrtzi)) for + his insight and help on using SwiftSyntax, +* and Kyle Macomber ([@kylemacomber](https://github.com/kylemacomber)), + who advocated for using results of existing research for + `swift-format`'s implementation and found the Oppen paper, + instead of inventing solutions from whole cloth. diff --git a/proposals/0251-simd-additions.md b/proposals/0251-simd-additions.md new file mode 100644 index 0000000000..9b5e896eff --- /dev/null +++ b/proposals/0251-simd-additions.md @@ -0,0 +1,361 @@ +# SIMD additions + +* Proposal: [SE-0251](0251-simd-additions.md) +* Author: [Stephen Canon](https://github.com/stephentyrone) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented with Modifications (Swift 5.1)** +* Implementation: [apple/swift#23421](https://github.com/apple/swift/pull/23421) and [apple/swift#24136](https://github.com/apple/swift/pull/24136) +* Review: ([review](https://forums.swift.org/t/se-0251-simd-additions/21957)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0251-simd-additions/22801)) + + + +## Introduction + +Early adopters of SIMD types and protocols have encountered a few missing things as +they've started to write more code that uses them. In addition, there are some +features we punted out of the original review because we were up against a hard time +deadline to which we would like to give further consideration. + +This is a bit of a grab bag of SIMD features, so I'm deviating from the usual proposal +structure. Each new addition has its own motivation, proposed solution, and alternatives +considered section. + +## Table of Contents +1. [Static `scalarCount`](#scalarCount) +2. [Extending Vectors](#extending) +3. [Swizzling](#swizzling) +4. [Reductions](#reduction) +5. [Lanewise `min`, `max`, and `clamp`](#minMaxClamp) +6. [`.one`](#one) +7. [`any` and `all`](#anyAndAll) + + + +## Static scalarCount +### Motivation +In functions that construct new SIMD vectors, especially initializers, one frequently wants +to perform some validation involving `scalarCount` *before* doing the work to create the +vector. Currently, `scalarCount` is defined as an instance property (following the pattern +of `count` on collection). + +However, all SIMD vectors of a given type have the same `scalarCount`, so semantically +it makes sense to have available as a static property as well. There's precedent for having +this duplication in the `.bitWidth` property on fixed-width integers. + +### Detailed Design +```swift +extension SIMDStorage { + /// The number of scalars, or elements, in a vector of this type. + public static var scalarCount: Int { + return Self().scalarCount + } +} +``` +The property is defined as an extension on SIMDStorage because it makes semantic +sense there (`SIMD` refines `SIMDStorage`). It is defined in terms of the existing member +property (instead of the apparently more-logical vise-versa) because that way it +automatically works for all existing SIMD types with no changes. In practice this +introduces no extra overhead at runtime. + +### Alternatives Considered +Not doing anything. Users can always fall back on the weird-but-effective +`Self().scalarCount`. + + + +## Extending vectors +### Motivation +When working with homogeneous coordinates in graphics, the last component frequently +needs to be treated separately--this means that you frequently want to extract the first +(n-1) components, do arithmetic on them and the final component separately, and then +re-assemble them. At API boundaries, you frequently take in (n-1) component vectors, +immediately extend them to perform math, and then return out only the first (n-1) +components. + +### Detailed design +In order to support extending vectors from (n-1) to n components, add the following two +initializers: +```swift +extension SIMD3 { + public init(_ xy: SIMD2, _ z: Scalar) { + self.init(xy.x, xy.y, z) + } +} + +extension SIMD4 { + public init(_ xyz: SIMD3, _ w: Scalar) { + self.init(xyz.x, xyz.y, xyz.z, w) + } +} +``` + +### Alternatives Considered +These could alternatively be spelled like `xy.appending(z)`; there are two reasons that +I'm avoiding that: +- I would expect `appending( )` to return the same type; but the result type is different. +- I would expect `appending( )` to be available on all SIMD types, but it breaks down +beyond `SIMD4`, because there is no `SIMD5` in the standard library. + +We could have also used an explicit parameter label. +```swift +let v = SIMD3(...) +let x = SIMD4(v, 1) // proposed above +let y = SIMD4(v, appending: 1) // with parameter label +``` +My feeling is that the behavior is clear without the label, but it's very reasonable to argue +for an explicit label instead. + + + +## Swizzling +### Motivation +In C-family languages, clang defines "vector swizzles" (aka permutes, aka shuffles, ... ) +that let you select and re-arrange elements from a vector: +```c +#import +simd_float4 x = { 1, 2, 3, 4}; +x.zyx; // (simd_float3){3, 2, 1}; +``` +This comes from an identical feature in graphics shader-languages, where it is very +heavily used. + +### Detailed design +For Swift, we want to restrict the feature somewhat, but also make it more powerful. +In shader languages and clang extensions, you can even use swizzled vectors as *lvalues*, +so long as the same element does not appear twice. I proposed to define general +permutes as get-only subscripts. By restricting them from appearing as setters, we +gain the flexibility to not require they be compile-time constants: +```swift +extension SIMD { + /// Extracts the scalars at specified indices to form a SIMD2. + /// + /// The elements of the index vector are wrapped modulo the count of elements + /// in this vector. Because of this, the index is always in-range and no trap + /// can occur. + public subscript(index: SIMD2) -> SIMD2 + where Index: FixedWidthInteger { + var result = SIMD2() + for i in result.indices { + result[i] = self[Int(index[i]) % scalarCount] + } + return result + } +} + +let v = SIMD4(1,2,3,4) +let xyz = SIMD3(2,1,0) +let w = v[xyz] // SIMD3(3,2,1) +``` +### Alternatives Considered +1. We might want an explicit label on this subscript, but as with the extending inits, I +believe that its use is generally clear enough in context. + +2. The main question is "what should the behavior for out-of-range indices be?" The +definition I have chosen here is simple to explain and maps efficiently to the hardware, +but there are at least two other good options: it could be a precondition failure, or it +could fill the vector with zero in lanes that have out of range indices. The first option +(trapping) is undesirable because it's less efficient with dynamic indices. The second +would be slightly more efficient on some architectures, but is also significantly more +magic. I believe that the proposed alternative has the best balance of explainable +behavior and efficiency. + + + +## Reductions (or "Horizontal Operations") +### Motivation +Generally in SIMD programming you try to avoid horizontal operations as much as +possible, but frequently you need to do a few of them at the end of a chain of +computations. For example, if you're summing an array, you would sum into a bank +of vector accumulators first, then sum those down to a single vector. Now you need +to get from that vector to a scalar by summing the elements. This is where reductions +enter. + +`sum` is also a basic building block for things like the dot product (and hence matrix +multiplication), so it's very valuable to have an efficient implementation provided by the +standard library. Similarly you want to have `min` and `max` to handle things like rescaling +for computational geometry. + +### Detailed design +```swift +extension SIMD where Scalar: Comparable { + /// The least element in the vector. + public func min() -> Scalar + + /// The greatest element in the vector. + public func max() -> Scalar +} + +extension SIMD where Scalar: FixedWidthInteger { + /// Returns the sum of the scalars in the vector, computed with + /// wrapping addition. + /// + /// Equivalent to indices.reduce(into: 0) { $0 &+= self[$1] }. + public func wrappedSum() -> Scalar +} + +extension SIMD where Scalar: FloatingPoint { + /// Returns the sum of the scalars in the vector. + public func sum() -> Scalar +} +``` + +### Alternatives Considered +We could call the integer operation `sum` as well, but it seems better to reserve that name +for the trapping operation in case we ever want to add it (just like we use `&+` for integer +addition on vectors, even though there is no `+`). We may want to define a floating-point +sum with relaxed semantics for accumulation ordering at some point in the future (I plan +to define `sum` as the binary tree sum here--that's the best tradeoff between reproducibility +and performance). + +I dropped `indexOfMinValue` and `indexOfMaxValue` from this proposal for two reasons: +- there's some disagreement about whether or not they're important enough to include +- it's not clear what we should name them; If they're sufficiently important, we probably +want to have them on Collection some day, too, so the bar for the naming pattern that we +establish is somewhat higher. + + + +## `any` and `all` +### Motivation +`any` and `all` are special reductions that operate on boolean vectors (`SIMDMask`). They +return `true` if and only if *any* (or *all*) lanes of the boolean vector are `true`. These are +used to do things like branch around edge-case fixup: +```swift +if any(x .< 0) { // handle negative x } +``` + +### Detailed design +`any` and `all` are free functions: +```swift +public func any(_ mask: SIMDMask) -> Bool { + return mask._storage.min() < 0 +} + +public func all(_ mask: SIMDMask) -> Bool { + return mask._storage.max() < 0 +} +``` + +### Alternatives Considered +*Why* are `any` and `all` free functions while `max` and `min` and `sum` are member +properties? Because of readability in their typical use sites. `min`, `max`, and `sum` are +frequently applied to a named value: +```swift +let accumulator = /* do some work */ +return accumulator.sum +``` +`any` and `all` are most often used with nameless comparison results: +```swift +if any(x .< minValue .| x .> maxValue) { + // handle special cases +} +``` +To my mind, this would read significantly less clearly as +```swift +if (x .< minValue .| x .> maxValue).any` { +``` +or +```swift +if (x .< minValue .| x .> maxValue).anyIsTrue` { +``` +because there's no "noun" that the property applies to. There was a proposal in the fall +to make them static functions on `Bool` so that one could write +```swift +if .any(x .< minValue) { +} +``` +but I'm not convinced that's actually better than a free function. + + + +## `min`, `max`, and `clamp` +### Motivation +We have lanewise arithmetic on SIMD types, but we don't have lanewise `min` and `max`. +We're also missing `clamp` to restrict values to a specified range. + +### Detailed design +```swift +extension SIMD where Scalar: Comparable { + /// Replace any values less than lowerBound with lowerBound, and any + /// values greater than upperBound with upperBound. + /// + /// For floating-point vectors, `.nan` is replaced with `lowerBound`. + public mutating func clamp(lowerBound: Self, upperBound: Self) { + self = self.clamped(lowerBound: lowerBound, upperBound: upperBound) + } + + /// The vector formed by replacing any values less than lowerBound + /// with lowerBound, and any values greater than upperBound with + /// upperBound. + /// + /// For floating-point vectors, `.nan` is replaced with `lowerBound`. + public func clamped(lowerBound: Self, upperBound: Self) -> Self { + return Self.min(upperBound, Self.max(lowerBound, self)) + } +} + +/// The lanewise minimum of two vectors. +/// +/// Each element of the result is the minimum of the corresponding +/// elements of the inputs. +public func min(_ lhs: V, _ rhs: V) -> V where V: SIMD, V.Scalar: Comparable + +/// The lanewise maximum of two vectors. +/// +/// Each element of the result is the maximum of the corresponding +/// elements of the inputs. +public func max(_ lhs: V, _ rhs: V) -> V where V: SIMD, V.Scalar: Comparable +``` + +### Alternatives Considered +These could be spelled out `lanewiseMaximum` or similar, to clarify that they operate +lanewise (Chris suggested this in the pitch thread), but we don't spell out `+` as +"lanewise-plus", so it seems weird to do it here. The default assumption is that SIMD +operations are lanewise. + + + +## `.one` +### Motivation +SIMD types cannot be `ExpressibleByIntegerLiteral` (it results in type system +ambiguity for common expressions). We already have `.zero`, so adding `.one` makes +sense as a convenience. + +### Detailed design +```swift +extension SIMD where Scalar: ExpressibleByIntegerLiteral { + public static var one: Self { + return Self(repeating: 1) + } +} +``` + +### Alternatives Considered +- Do nothing. We don't *need* this, but it has turned out to be a useful convenience. +- Why stop at `one`? Why not `two`? Because that way lies madness. + +## Source compatibility + +These are all purely additive changes with no effect on source stability. + +## Effect on ABI stability + +These are all purely additive changes with no effect on source stability. + +## Effect on API resilience + +These are all purely additive changes with no effect on source stability. + +## Alternatives Considered + +The pitch for this proposal included some operations for loading and storing from a +collection. As Jordan pointed out in the pitch thread, we already have an init from +Sequence, which together with slices makes the load mostly irrelevant. The store +operation did not have satisfactory naming, and I would like to come up with a better +pattern for these that handles iterating over a sequence of SIMD vectors loaded from +a collection of scalars and storing them out as a single pattern, rather than building +it up one piece at a time. + +## Implementation Notes + +Due to a desire to avoid collision between the `min(u, v)` (pointwise minimum on SIMD vectors) and `min(u, v)` (minimum defined on `Comparable`, if a user adds a retroactive conformance), the core team decided to rename the SIMD operations to `pointwiseMin(u, v)` and `pointwiseMax(u, v)`. diff --git a/proposals/0252-keypath-dynamic-member-lookup.md b/proposals/0252-keypath-dynamic-member-lookup.md new file mode 100644 index 0000000000..b0ef216db5 --- /dev/null +++ b/proposals/0252-keypath-dynamic-member-lookup.md @@ -0,0 +1,125 @@ +# Key Path Member Lookup + +* Proposal: [SE-0252](0252-keypath-dynamic-member-lookup.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.1)** +* Implementation: [PR #23436](https://github.com/apple/swift/pull/23436) + +## Introduction + +This proposal attempts to enable stronger-typed version of the dynamic member lookup by extending functionality of an existing `@dynamicMemberLookup` attribute with key path based variants. + +[Swift Evolution Pitch](https://forums.swift.org/t/pitch-key-path-member-lookup/21579) + +## Motivation + +Dynamic member lookup allows a type to opt in to extending member lookup ("dot" syntax) for arbitrary member names, turning them into a string that can then be resolved at runtime. Dynamic member lookup allows interoperability with dynamic languages where the members of a particular instance can only be determined at runtime... but no earlier. Dynamic member lookups, therefore, tend to work with type-erased wrappers around foreign language objects (e.g., `PyVal` for an arbitrary Python object), which don't provide much static type information. + +On the other hand, key paths provide a dynamic representation of a property that can be used to read or write the referenced property. Key paths maintain static type information about the type of the property being accessed, making them a good candidate for abstractly describing a reference to data that is modeled via Swift types. However, key paths can be cumbersome to create and apply. Consider a type `Lens` that abstractly refers to some value of type `T`, through which one can read (and possibly write) the value of that `T`: + +```swift +struct Lens { + let getter: () -> T + let setter: (T) -> Void + + var value: T { + get { + return getter() + } + nonmutating set { + setter(newValue) + } + } +} +``` + +Given some `Lens`, we would like to produce a new `Lens` referring to a property of the value produced by the lens. Key paths allow us to write such a projection function directly: + +```swift +extension Lens { + func project(_ keyPath: WritableKeyPath) -> Lens { + return Lens( + getter: { self.value[keyPath: keyPath] }, + setter: { self.value[keyPath: keyPath] = $0 }) + } +} +``` + +As an example, consider a `Lens`: + +```swift +struct Point { + var x, y: Double +} + +struct Rectangle { + var topLeft, bottomRight: Point +} + +func projections(lens: Lens) { + let topLeft = lens.project(\.topLeft) // inferred type is Lens + let top = lens.project(\.topLeft.y) // inferred type is Lens +} +``` + +Forming the projection is a bit unwieldy: it's a call to `project` in which we need to use `\.` to then describe the key path. Why not support the most direct syntax to form a lens referring to some part of the stored value, e.g., `lens.topLeft` or `lens.topLeft.y`, respectively? + +## Proposed solution + +Augment existing `@dynamicMemberLookup` attribute to support key path based dynamic member lookup by rewriting "dot" and "subscript" syntax into a call to a special subscript whose argument is a key path describing the member. Here, we reimplement `Lens` in terms of new `@dynamicMemberLookup` capabilities: + + +```swift +@dynamicMemberLookup +struct Lens { + let getter: () -> T + let setter: (T) -> Void + + var value: T { + get { + return getter() + } + nonmutating set { + setter(newValue) + } + } + + subscript(dynamicMember keyPath: WritableKeyPath) -> Lens { + return Lens( + getter: { self.value[keyPath: keyPath] }, + setter: { self.value[keyPath: keyPath] = $0 }) + } +} +``` + +Given a `Lens` named `lens`, the expression `lens.topLeft` will be evaluated as `lens[dynamicMember: \.topLeft]`, allowing normal member accesses on a `Lens` to produce a new `Lens`. + +The formation of the key path follows a "single step" approach where each key path component is split into a separate `[dynamicMember: KeyPath]` invocation. For example, the expression `lens.topLeft.y` will be evaluated as `lens[dynamicMember: \.topLeft][dynamicMember: \.y]`, producing a `Lens`. + +## Detailed design + +Proposed solution builds on existing functionality of the `@dynamicMemberLookup` attribute. It adopts restrictions associated with existing string-based design as well as a couple of new ones: + +* Key path member lookup only applies when the `@dynamicMemberLookup` type does not contain a member with the given name. This privileges the members of the `@dynamicMemberLookup` type (e.g., `Lens`), hiding those of whatever type is that the root of the keypath (e.g., `Rectangle`). +* `@dynamicMemberLookup` can only be written directly on the definition of a type, not an an extension of that type. +* A `@dynamicMemberLookup` type must define a subscript with a single, non-variadic parameter whose argument label is `dynamicMember` and that accepts one of the key path types (e.g., `KeyPath`, `WritableKeyPath`). +* In case both string-based and keypath-based overloads match, keypath takes priority as one carrying more typing information. + +## Source compatibility + +This is an additive proposal, which makes ill-formed syntax well-formed but otherwise does not affect existing code. First, only types that opt in to `@dynamicMemberLookup` will be affected. Second, even for types that adopt `@dynamicMemberLookup`, the change is source-compatible because the transformation to use `subscript(dynamicMember:)` is only applied when there is no member of the given name. + +## Effect on ABI stability + +This feature is implementable entirely in the type checker, as (effectively) a syntactic transformation on member access expressions. It, therefore, has no impact on the ABI. + +## Effect on API resilience + +Adding `@dynamicMemberLookup` is a resilient change to a type, as is the addition of the subscript. + +## Alternatives considered + +The main alternative would be to not do this at all. + +Another alternative would be to use a different attribute to separate this feature from `@dynamicMemberLookup`, e.g. `@keyPathMemberLookup` since string based design doesn't, at the moment, provide any static checking for member access. We recognize this as a valid concern, but at the same time consider both to be fundamentally the same feature with different amount of static checking. Using the same attribute allows us to adhere to "fewer conceptular features" concept, as well as, enables powerful combinations where string based dynamic lookup could be used as a fallback when key path dynamic lookup fails. diff --git a/proposals/0253-callable.md b/proposals/0253-callable.md new file mode 100644 index 0000000000..035161bc7b --- /dev/null +++ b/proposals/0253-callable.md @@ -0,0 +1,664 @@ +# Callable values of user-defined nominal types + +* Proposal: [SE-0253](0253-callable.md) +* Authors: [Richard Wei](https://github.com/rxwei), [Dan Zheng](https://github.com/dan-zheng) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 5.2)** +* Implementation: [apple/swift#24299](https://github.com/apple/swift/pull/24299) +* Previous Revisions: + [[1]](https://github.com/swiftlang/swift-evolution/blob/36ea8be09508db9380949954d0c7a101fdb15226/proposals/0253-callable.md), + [[2]](https://github.com/swiftlang/swift-evolution/blob/e1dd65469e3525e5e230b877e1539bff1e1cc5e3/proposals/0253-callable.md) +* Decision Notes: + ([rationale](https://forums.swift.org/t/accepted-with-modification-se-0253-callable-values-of-user-defined-nominal-types/24605)), + ([final revision note](https://forums.swift.org/t/accepted-with-modification-se-0253-callable-values-of-user-defined-nominal-types/24605/166)) + +## Introduction + +This proposal introduces "statically" +[callable](https://en.wikipedia.org/wiki/Callable_object) values to Swift. +Callable values are values that define function-like behavior and can be called +using function call syntax. In contrast to dynamically callable values +introduced in +[SE-0216](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0216-dynamic-callable.md), +this feature supports statically declared arities, argument labels, and +parameter types, and is not constrained to primary type declarations. + +In a nutshell, values that have a method whose base name is `callAsFunction` +(referred to as a "`callAsFunction` method" for the rest of this proposal) can +be called like a function. The function call syntax forwards arguments to the +corresponding `callAsFunction` method. + +```swift +struct Adder { + var base: Int + func callAsFunction(_ x: Int) -> Int { + return base + x + } +} + +let add3 = Adder(base: 3) +add3(10) // => 13 +``` + +## Motivation + +Currently, in Swift, only a few kinds of values are syntactically callable: +- Values with function types. +- Type names (e.g. `T` can be called like `T(...)`, which is desugared to `T.init(...)`). +- Values with a `@dynamicCallable` type. + +However, call-syntax can also be useful for other values, primarily those that behave like functions. This includes: +- Values that represent functions: mathematical functions, function expressions, etc. +- Values that have one main use and want to provide a simple call-syntax interface: neural network layers, parsers, efficient bound closures, etc. + +Here are some concrete sources of motivation. + +### Values representing functions + +Values of some nominal types exactly represent functions: in the mathematical sense (a mapping from inputs to outputs), or in the context of programming languages. + +Here are some examples: + +```swift +/// Represents a polynomial function, e.g. `2 + 3x + 4x²`. +struct Polynomial { + /// Represents the coefficients of the polynomial, starting from power zero. + let coefficients: [Float] +} +``` + +Since these types represent functions, naturally they can be applied to inputs. However, currently in Swift, the "function application" functionality must be defined as a method. + +```swift +extension Polynomial { + func evaluated(at input: Float) -> Float { + var result: Float = 0 + for (i, c) in coefficients.enumerated() { + result += c * pow(input, Float(i)) + } + return result + } +} + +let f = Polynomial(coefficients: [2, 3, 4]) +print(f.evaluated(at: 2)) // => 24 +``` + +The mathematical notation for function application is simply `output = f(input)`. Using subscript methods achieve a similar application syntax `f[x]`, but subscripts and square brackets typically connote "indexing into a collection", which is not the behavior here. + +```swift +extension Polynomial { + subscript(input: Float) -> Float { + ... + } +} +let f = Polynomial(coefficients: [2, 3, 4]) +// Subscript syntax, may be confusing. +print(f[2]) // => 24 +``` + +The proposed feature enables the same call syntax as the mathematical notation: +```swift +extension Polynomial { + func callAsFunction(_ input: Float) -> Float { + ... + } +} +let f = Polynomial(coefficients: [2, 3, 4]) +// Call syntax. +print(f(2)) // => 24 +``` + +#### Bound closures + +Variable-capturing closures can be modeled explicitly as structs that store the bound variables. This representation is more performant and avoids the type-erasure of closure contexts. + +```swift +// Represents a nullary function capturing a value of type `T`. +struct BoundClosure { + var function: (T) -> Void + var value: T + + func callAsFunction() { return function(value) } +} + +let x = "Hello world!" +let closure = BoundClosure(function: { print($0) }, value: x) +closure() // prints "Hello world!" +``` + +A call syntax sugar would enable `BoundClosure` instances to be applied like normal functions. + +### Nominal types with one primary method + +Some nominal types have a "primary method" that performs their main use. For example: + +* Calculators *calculate*: `calculator.calculating(query)`. +* Parsers *parse*: `parser.parsing(text)`. +* Neural network layers *apply to inputs*: `layer.applied(to: input)`. +* Types representing functions *apply to arguments*: `function.applied(to: arguments)`. + +Types that have a primary method usually call that method frequently. Thus, it may be desirable to sugar applications of the main method with call syntax to reduce noise. + +Let's explore neural network layers and string parsers in detail. + +#### Neural network layers + +[Machine learning](https://en.wikipedia.org/wiki/Machine_learning) models often represent a function that contains an internal state called "trainable parameters", and the function takes an input and predicts the output. In code, models are often represented as a data structure that stores trainable parameters, and a method that defines the transformation from an input to an output in terms of these trained parameters. Here's an example: + +```swift +struct Perceptron { + var weight: Vector + var bias: Float + + func applied(to input: Vector) -> Float { + return weight • input + bias + } +} +``` + +Stored properties `weight` and `bias` are considered as trainable parameters, and are used to define the transformation from model inputs to model outputs. Models can be [trained ](https://developers.google.com/machine-learning/glossary/#training), during which parameters like `weight` are updated, thus changing the behavior of `applied(to:)`. When a model is used, the call site looks just like a function call. + +```swift +let model: Perceptron = ... +let ŷ = model.applied(to: x) +``` + +Many deep learning models are composed of layers, or layers of layers. In the definition of those models, repeated calls to `applied(to:)` significantly complicate the look of the program and reduce the clarity of the resulting code. + +```swift +struct Model { + var conv = Conv2D(filterShape: (5, 5, 3, 6)) + var maxPool = MaxPool2D(poolSize: (2, 2), strides: (2, 2)) + var flatten = Flatten() + var dense = Dense(inputSize: 36 * 6, outputSize: 10) + + func applied(to input: Tensor) -> Tensor { + return dense.applied(to: flatten.applied(to: maxPool.applied(to: conv.applied(to: input)))) + } +} +``` + +These repeated calls to `applied(to:)` harm clarity and makes code less readable. If `model` could be called like a function, which it mathematically represents, the definition of `Model` becomes much shorter and more concise. The proposed feature [promotes clear usage by omitting needless words](https://swift.org/documentation/api-design-guidelines/#promote-clear-usage). + +```swift +struct Model { + var conv = Conv2D(filterShape: (5, 5, 3, 6)) + var maxPool = MaxPool2D(poolSize: (2, 2), strides: (2, 2)) + var flatten = Flatten() + var dense = Dense(inputSize: 36 * 6, outputSize: 10) + + func callAsFunction(_ input: Tensor) -> Tensor { + // Call syntax. + return dense(flatten(maxPool(conv(input)))) + } +} + +let model: Model = ... +let ŷ = model(x) +``` + +There are more ways to further simplify model definitions, but making models callable like functions is a good first step. + +#### Domain specific languages + +DSL constructs like string parsers represent functions from inputs to outputs. Parser combinators are often implemented as higher-order functions operating on parser values, which are themselves data structures—some implementations store closures, while some other [efficient implementations ](https://infoscience.epfl.ch/record/203076/files/p637-jonnalagedda.pdf) store an expression tree. They all have an "apply"-like method that performs an application of the parser (i.e. parsing). + +```swift +struct Parser { + // Stored state... + + func applied(to input: String) throws -> Output { + // Using the stored state... + } + + func many() -> Parser<[Output]> { ... } + func many(separatedBy separator: Parser) -> Parser<[Output]> { ... } +} +``` + +When using a parser, one would need to explicitly call `applied(to:)`, but that is a bit cumbersome. Since parsers are like functions, it would be cleaner if parsers themself were callable. + +```swift +func callAsFunction(_ input: String) throws -> Output { + // Using the stored state... +} +``` + +```swift +let sexpParser: Parser = ... +// Call syntax. +let sexp = sexpParser("(+ 1 2)") +``` + +### A static counterpart to `@dynamicCallable` + +[SE-0216](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0216-dynamic-callable.md) introduced user-defined dynamically callable values. In its [alternatives considered](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0216-dynamic-callable.md#alternatives-considered) section, it was requested that we design and implement the "static callable" version of this proposal in conjunction with the dynamic version proposed. See its [pitch thread](https://forums.swift.org/t/pitch-3-introduce-user-defined-dynamically-callable-types/12232) for discussions about "static callables". + +### Prior art + +Many languages offer the call syntax sugar: + +* Python: [ `object.__call__(self[, args...])` ](https://docs.python.org/3/reference/datamodel.html#emulating-callable-objects) +* C++: [ `operator()` (function call operator)](https://en.cppreference.com/w/cpp/language/operators) +* Scala: [ `def apply(...)` (apply methods) ](https://twitter.github.io/scala_school/basics2.html#apply) + +### Unifying compound types and nominal types + +A long term goal with the type system is to unify compound types (e.g. function types and tuple types) and nominal types, to allow compound types to conform to protocols and have members. When function types can have members, it will be most natural for them to have a `call` method, which can help unify the compiler's type checking rules for call expressions. + +## Proposed design + +We propose to introduce a syntactic sugar for values that have an instance +method whose base name is `call` (a `call` method). + +```swift +struct Adder { + var base: Int + func callAsFunction(_ x: Int) -> Int { + return base + x + } +} +``` + +Values that have a `callAsFunction` method can be called like a function, forwarding +arguments to the `callAsFunction` method. + +```swift +let add3 = Adder(base: 3) +add3(10) // => 13 +``` + +Note: There are many alternative syntaxes for marking "call-syntax delegate +methods". These are listed and explored in the ["Alternatives +considered"](#alternative-ways-to-declare-call-syntax-delegate-methods) section. + +## Detailed design + +### `callAsFunction` methods + +Instance methods whose base name is `callAsFunction` will be recognized as an +implementation that makes a value of the enclosing type "callable" like a +function. + +When type-checking a call expression, the type checker will try to resolve the +callee. Currently, the callee can be a value with a function type, a type name, +or a value of a `@dynamicCallable` type. This proposal adds a fourth kind of a +callee: a value with a matching `callAsFunction` method. + +```swift +struct Adder { + var base: Int + + func callAsFunction(_ x: Int) -> Int { + return base + x + } + + func callAsFunction(_ x: Float) -> Float { + return Float(base) + x + } + + func callAsFunction(_ x: T, bang: Bool) throws -> T where T: BinaryInteger { + if bang { + return T(Int(exactly: x)! + base) + } else { + return T(Int(truncatingIfNeeded: x) + base) + } + } +} +``` + +```swift +let add1 = Adder(base: 1) +add1(2) // => 3 +try add1(4, bang: true) // => 5 +``` + +When type-checking fails, error messages look like those for function calls. +When there is ambiguity, the compiler will show relevant `callAsFunction` method +candidates. + +```swift +add1("foo") +// error: cannot invoke 'add1' with an argument list of type '(String)' +// note: overloads for functions named 'callAsFunction' exist with these partially matching parameter lists: (Float), (Int) +add1(1, 2, 3) +// error: cannot invoke 'add1' with an argument list of type '(Int, Int, Int)' +``` + +### Direct reference to `callAsFunction` + +Since a `callAsFunction` method is a normal method, one can refer to a +`callAsFunction` method using its declaration name and get a closure where +`self` is captured. This is exactly how method references work today. + +```swift +let add1 = Adder(base: 1) +let f1: (Int) -> Int = add1.callAsFunction +let f2: (Float) -> Float = add1.callAsFunction(_:) +let f3: (Int, Bool) throws -> Int = add1.callAsFunction(_:bang:) +``` + +### When the type is also `@dynamicCallable` + +A type can both have `callAsFunction` methods and be declared with +`@dynamicCallable`. When type-checking a call expression, the type checker will +first try to resolve the call to a function or initializer call, then a +`callAsFunction` method call, and finally a dynamic call. + +### Implementation + +The implementation is very simple and non-invasive: [less than 200 lines of +code](https://github.com/apple/swift/pull/24299) in the type checker that +performs lookup and expression rewrite. + +```swift +let add1 = Adder(base: 1) +add1(0) // Rewritten to `add1.callAsFunction(0)` after type checking. +``` + +## Source compatibility + +This is a strictly additive proposal with no source-breaking changes. + +## Effect on ABI stability + +This is a strictly additive proposal with no ABI-breaking changes. + +## Effect on API resilience + +This has no impact on API resilience which is not already captured by other language features. + +## Future directions + +### Implicit conversions to function + +A value cannot be implicitly converted to a function when the destination +function type matches the type of the `callAsFunction` method. Since +`callAsFunction` methods are normal methods, you can [refer to them +directly](#direct-reference-to-call) via `.callAsFunction` and get a function. + +Implicit conversions impact the entire type system and require runtime support +to work with dynamic casts; thus, further exploration is necessary for a formal +proposal. This base proposal is self-contained; incremental proposals involving +conversion can come later. + +```swift +let h: (Int) -> Int = add1 +``` + +A less controversial future direction is to support explicit conversion via `as`: + +```swift +let h = add1 as (Int) -> Int +``` + +### Function type as a constraint + +On the [pitch +thread](https://forums.swift.org/t/pitch-introduce-static-callables/21732/2), +[Joe Groff](https://github.com/jckarter) brought up the possibility of allowing +function types to be used as conformance constraints. Performance-minded +programmers can define custom closure types where the closure context is not +fully type-erased. + +```swift +struct BoundClosure ()>: () -> () { + var function: F + var value: T + + func callAsFunction() { return function(value) } +} + +let f = BoundClosure({ print($0) }, x) // instantiates BoundClosure<(underlying type of closure), Int> +f() // invokes call on BoundClosure +``` + +In this design, the function type constraint behaves like a protocol that +requires a `callAsFunction` method whose parameter types are the same as the +function type's parameter types. + +## Alternatives considered + +### Alternative names for call-syntax delegate methods + +In addition to the word `call` in `callAsFunction`, there are other words that +can be used to denote the function call syntax. The most common ones are `apply` +and `invoke` as they are used to declare call-syntax delegate methods in other +programming languages. + +Both `apply` and `invoke` are good one-syllable English words that are +technically correct, but we feel there are two concerns with these names: + +* They are officially completely new terminology to Swift. In [the _Functions_ +chapter of _The Swift Programming Language_ +book](https://docs.swift.org/swift-book/LanguageGuide/Functions.html), there is +no mention of "apply" or "invoke" anywhere. Function calls are officially +called "function calls". + +* They do not work very well with Swift's API naming conventions. According to + [Swift API Design Guidelines - Strive for Fluent + Usage](https://swift.org/documentation/api-design-guidelines/#strive-for-fluent-usage), + functions should be named according to their side-effects. + + > Those with side-effects should read as imperative verb phrases, e.g., + > `print(x)`, `x.sort()`, `x.append(y)`. + + Both `apply` and `invoke` are clearly imperative verbs. If call-syntax + delegate methods must be named `apply` or `invoke`, their declarations and + direct references will almost certainly read like a mutating function while + they may not be. + + In contrast, `call` is both a noun and a verb. It is perfectly + suited for describing the precise functionality while not having a strong + implication about the function's side-effects. + + **call** + - _v._ Cause (a subroutine) to be executed. + - _n._ A command to execute a subroutine. + +After the second round of proposal review, the core team accepted the proposal +while revising the function base name to `callFunction`. The revision invoked a +significant amount of bikeshedding on [the +thread](https://forums.swift.org/t/accepted-with-modification-se-0253-callable-values-of-user-defined-nominal-types/24605). +After considering the feedback, the core team +[decided](https://forums.swift.org/t/accepted-with-modification-se-0253-callable-values-of-user-defined-nominal-types/24605/166) +to further revise the proposal to name the call-syntax delegate method +`callAsFunction`. + +### Alternative ways to declare call-syntax delegate methods + +#### Create a new declaration kind like `subscript` and `init` + +Declarations that are associated with special invocation syntax often have their +own declaration kind. For example, subscripts are implemented with a `subscript` +declaration, and initialization calls are implemented with an `init` +declaration. Since the function call syntax is first-class, one direction is to +make the declaration be as first-class as possible. + +```swift +struct Adder { + var base: Int + call(_ x: Int) -> Int { + return base + x + } +} +``` + +This alternative is in fact what's proposed in [the first revision of this +proposal](https://github.com/swiftlang/swift-evolution/blob/36ea8be09508db9380949954d0c7a101fdb15226/proposals/0253-callable.md), +which got [returned for +revision](https://forums.swift.org/t/returned-for-revision-se-0253-static-callables/23290). + +#### Use unnamed `func` declarations to mark call-syntax delegate methods + +```swift +struct Adder { + var base: Int + // Option: `func` with literally no name. + func(_ x: Int) -> Int { ... } + + // Option: `func` with an underscore at the base name position. + func _(_ x: Int) -> Int + + // Option: `func` with a `self` keyword at the base name position. + func self(_ x: Int) -> Int + + // Option: `call` method modifier on unnamed `func` declarations. + // Makes unnamed `func` less weird and clearly states "call". + call func(_ x: Int) -> Int { ... } +} +``` + +This approach represents call-syntax delegate methods as unnamed `func` +declarations instead of `func callAsFunction`. + +One option is to use `func(...)` without an identifier name. Since the word +"call" does not appear, it is less clear that this denotes a call-syntax +delegate method. Additionally, it's not clear how direct references would work: +the proposed design of referencing `callAsFunction` methods via `foo.call` is +clear and consistent with the behavior of `init` declarations. + +To make unnamed `func(...)` less weird, one option is to add a `call` +declaration modifier: `call func(...)`. The word `call` appears in both this +option and the proposed design, clearly conveying "call-syntax delegate method". +However, declaration modifiers are currently also treated as keywords, so with +both approaches, parser changes to ensure source compatibility are necessary. +`call func(...)` requires additional parser changes to allow `func` to sometimes +not be followed by a name. The authors lean towards `callAsFunction` methods for +simplicity and uniformity. + +#### Use an attribute to mark call-syntax delegate methods + +```swift +struct Adder { + var base: Int + @callDelegate + func addingWithBase(_ x: Int) -> Int { + return base + x + } +} +``` + +This approach achieves a similar effect as `callAsFunction` methods, except that +it allows call-syntax delegate methods to have a custom name and be directly +referenced by that name. This is useful for types that want to make use of the +call syntax sugar, but for which the name "call" does not accurately describe +the callable functionality. + +However, there are two concerns. + +* First, we feel that using a `@callableMethod` method attribute is more noisy, + as many callable values do not need a special name for its call-syntax + delegate methods. + +* Second, custom names often involve argument labels that form a phrase with the + base name in order to be idiomatic. The grammaticality will be lost in the + call syntax when the base name disappears. + + ```swift + struct Layer { + ... + @callDelegate + func applied(to x: Int) -> Int { ... } + } + + let layer: Layer = ... + layer.applied(to: x) // Grammatical. + layer(to: x) // Broken. + ``` + +In contrast, standardizing on a specific name defines these problems away and +makes this feature easier to use. + +For reference: Other languages with callable functionality typically require +call-syntax delegate methods to have a particular name (e.g. `def __call__` in +Python, `def apply` in Scala). + +#### Use a type attribute to mark types with call-syntax delegate methods + +```swift +@staticCallable // alternative name `@callable`; similar to `@dynamicCallable` +struct Adder { + var base: Int + // Informal rule: all methods with a particular name (e.g. `func callAsFunction`) are deemed call-syntax delegate methods. + // + // `StringInterpolationProtocol` has a similar informal requirement for + // `func appendInterpolation` methods. + // https://github.com/swiftlang/swift-evolution/blob/master/proposals/0228-fix-expressiblebystringinterpolation.md#proposed-solution + func callAsFunction(_ x: Int) -> Int { + return base + x + } +} +``` + +We feel this approach is not ideal because a marker type attribute is not +particularly meaningful. The call-syntax delegate methods of a type are what +make values of that type callable - a type attribute means nothing by itself. +There's also an unfortunate edge case that must be explicitly handled: if a +`@staticCallable` type defines no call-syntax delegate methods, an error must be +produced. + +After the first round of review, the core team also did not think a type-level +attribute is necessary. + +> After discussion, the core team doesn't think that a type level attribute is +> necessary, and there is no reason to limit this to primal type declarations - +> it is fine to add callable members (or overloads) in extensions, just as you +> can add subscripts to a type in extensions today. + +#### Use a `Callable` protocol to represent callable types + +```swift +// Compiler-known `Callable` marker protocol. +struct Adder: Callable { + var base: Int + // Informal rule: all methods with a particular name (e.g. `func callAsFunction`) are deemed call-syntax delegate methods. + func callAsFunction(_ x: Int) -> Int { + return base + x + } +} +``` + +We feel this approach is not ideal for the same reasons as the marker type attribute. A marker protocol by itself is not meaningful and the name for call-syntax delegate methods is informal. Additionally, protocols should represent particular semantics, but call-*syntax* behavior has no inherent semantics. + +### Also allow `static`/`class` `callAsFunction` methods + +Static `callAsFunction` methods could in theory look like initializers at the +call site. + +```swift +extension Adder { + static func callAsFunction(base: Int) -> Int { + ... + } + static func callAsFunction(_ x: Int) -> Int { + ... + } +} +Adder(base: 3) // error: ambiguous static member; do you mean `init(base:)` or `call(base:)`? +Adder(3) // okay, returns an `Int`, but it looks really like an initializer that returns an `Adder`. +``` + +This is an interesting direction, but parentheses followed by a type identifier +often connote initialization and it is not source-compatible. We believe this +would make call sites look very confusing. + +### Unify callable functionality with `@dynamicCallable` + +Both `@dynamicCallable` and the proposed `callAsFunction` methods involve +syntactic sugar related to function applications. However, the rules of the +sugar are different, making unification difficult. In particular, +`@dynamicCallable` provides a special sugar for argument labels that is crucial +for usability. + +```swift +// Let `PythonObject` be a `@dynamicMemberLookup` type with callable functionality. +let np: PythonObject = ... +// `PythonObject` with `@dynamicCallable. +np.random.randint(-10, 10, dtype: np.float) +// `PythonObject` with `callAsFunction` methods. The empty strings are killer. +np.random.randint(["": -10, "": 10, "dtype": np.float]) +``` + +[static-and-class-subscripts]: https://forums.swift.org/t/pitch-static-and-class-subscripts/21850 diff --git a/proposals/0254-static-subscripts.md b/proposals/0254-static-subscripts.md new file mode 100644 index 0000000000..fdc1935b81 --- /dev/null +++ b/proposals/0254-static-subscripts.md @@ -0,0 +1,133 @@ +# Static and class subscripts + +* Proposal: [SE-0254](0254-static-subscripts.md) +* Author: [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.1)** +* Implementation: [apple/swift#23358](https://github.com/apple/swift/pull/23358) +* Review: ([review](https://forums.swift.org/t/se-0254-static-and-class-subscripts/22537), [acceptance](https://forums.swift.org/t/accepted-se-0254-static-and-class-subscripts/22941)) + +## Introduction + +We propose allowing `static subscript` and, in classes, `class subscript` declarations. These could be used through either `TypeName[index]` or `TypeName.self[index]` and would have all of the capabilities you would expect of a subscript. We also propose extending dynamic member lookup to static properties by using static subscripts. + +Swift-evolution thread: [Static Subscript](https://forums.swift.org/t/static-subscript/1229) (2016), [Pitch: Static and class subscripts](https://forums.swift.org/t/pitch-static-and-class-subscripts/21850) + +## Motivation + +Subscripts have a unique and useful combination of features. Like functions, they can take arguments to change their behavior and generic parameters to support many types; like properties, they are permitted as lvalues so their results can be set, modified, and passed as inout. This is a powerful feature set, which is why they are used for features like key paths and `@dynamicMemberLookup`. + +Unfortunately, unlike functions and properties, Swift only supports subscripts on regular types, not metatypes. This not only makes the language inconsistent, it also prevents us from supporting important language features on metatypes. + +>
+> (Wait, what the heck is a "metatype"?) +> +> A type like `Int` has many instances, like `0` and `-42`. But Swift also creates a special instance representing the `Int` type itself, as opposed to any specific `Int` belonging to that type. This special instance can be directly accessed by writing `Int.self`; it is also returned by `type(of:)` and used in various other places. In fact, static members of `Int` are instance members of `Int.self`, so you use it any time you call one of those. +> +> Since `Int.self` is an instance, it must have a type, but the type of `Int.self` is not `Int`; after all, `Int.self` cannot do the things an `Int` can do, like arithmetic and comparison. Instead, `Int.self` is an instance of the type `Int.Type`. Because `Int.Type` is the type of a type, it is called a "metatype". +>
+ +And occasionally a subscript on a type is truly the best way to represent an operation. For instance, suppose you're offering access to the process's environment variables. Since the environment is global and environment variables can be both retrieved and set, a static subscript would be an excellent representation of them. Without them, users must either introduce a singleton instance or [use static properties or subscripts to expose the same operations with less fidelity](https://github.com/apple/swift-package-manager/blob/master/Sources/Basic/ProcessEnv.swift#L15). + +Swift originally omitted static subscripts for a good reason: They conflicted with an early sugar syntax for arrays, `Element[]`. But we have long since changed that syntax to `[Element]` and we aren't going back. There is no longer a technical obstacle to supporting them, and there never was a philosophical one. The only obstacle to this feature is inertia. + +It's time we gave it a little push. + +## Proposed solution + +In any place where it was previously legal to declare a `subscript`, it will now be legal to declare a `static subscript` as well. In classes it will also be legal to declare a `class subscript`. + +```swift +public enum Environment { + public static subscript(_ name: String) -> String? { + get { + return getenv(name).map(String.init(cString:)) + } + set { + guard let newValue = newValue else { + unsetenv(name) + return + } + setenv(name, newValue, 1) + } + } +} +``` + +The static and class subscripts on a type `T` can be used on any expression of type `T.Type`, including `T.self[...]` and plain `T[...]`. + +```swift +Environment["PATH"] += ":/some/path" +``` + +A static subscript with the parameter label `dynamicMember` can also be used to look up static properties on types marked with `@dynamicMemberLookup`. + +```swift +@dynamicMemberLookup +public enum Environment { + public static subscript(_ name: String) -> String? { + // as before + } + + public static subscript(dynamicMember name: String) -> String? { + get { return self[name] } + set { self.name = newValue } + } +} + +Environment.PATH += ":/some/path" +``` + +We do not currently propose to add support for metatype key paths, but this proposal is a necessary prerequisite for any future work on them. + +One concern brought up during the pitch phase was discoverability. We think that code completion changes will help with this, but those are outside the scope of an Evolution proposal. + +## Detailed design + +### Static subscripts + +Static and class subscripts can be declared everywhere static and class computed properties can be, with analogous language rules. In particular, static and class subscript accessors are implicitly `nonmutating` and cannot be made `mutating`, just like static and class computed property accessors. + +If a static or class subscript is declared on a type `T`, it can be applied to any value of type `T`, including `T.self`, `T`, and variables or other expressions evaluating to a value of type `T.Type`. + +Objective-C class methods with the same selectors as instance subscript methods (like `+objectAtIndexedSubscript:`) will not be imported to Swift as class subscripts; Objective-C technically allows them but doesn't make them usable in practice, so this is no worse than the native experience. Likewise, it will be an error to mark a static or class subscript with `@objc`. + +### Dynamic member lookup + +`@dynamicMemberLookup` can be applied to any type with an appropriate `subscript(dynamicMember:)` or `static subscript(dynamicMember:)` (or `class subscript(dynamicMember:)`, of course). If `subscript(dynamicMember:)` is present, it will be used to find instance members; if `static subscript(dynamicMember:)` is present, it will be used to find static members. A type can provide both. + +## Source compatibility + +This proposal is purely additive; it does not change any previously existing behavior. All syntax it will add support for was previously illegal. + +## ABI compatibility and backwards deployment + +Static subscripts are an additive change to the ABI. They do not require any runtime support; the Swift 5.0 runtime should even demangle their names correctly. Dynamic member lookup is implemented in the type checker and has no backwards deployment concerns. + +## Effect on API resilience + +The rules for the resilience of static and class subscripts will be the same as the rules of their instance subscript equivalents. Dynamic member lookup does not impact resilience. + +## Alternatives considered + +### Leave our options open + +The main alternative is to defer this feature again, leaving this syntax unused and potentially available for some other purpose. + +The most compelling suggestion we've seen is using `Element[n]` as type sugar for fixed-size arrays, but we don't think we would want to do that anyway. If fixed-size arrays need a sugar at all, we would want one that looked like an extension of the existing `Array` sugar, like `[Element * n]`. We can't really think of any other possibilities, so we feel confident that we won't want the syntax back in a future version of Swift. + +## Future directions + +### Metatype key paths + +Swift does not currently allow you to form keypaths to or through static properties. This was no loss before static subscripts, since you wouldn't have been able to apply them to a metatype anyway. But now that we have static subscripts, metatype keypaths could be supported. + +Metatype key paths were left out of this proposal because they are more complex than dynamic member lookup: + +1. Making them backwards deploy requires certain compromises, such as always using opaque accessors. Are these worth the cost? + +2. If we allow users to form key paths to properties in resilient libraries built before static key paths are supported, their `Equatable` and `Hashable` conformances may return incorrect results. Should we accept that as a minor issue, or set a minimum deployment target? + +3. Metatypes have kinds of members not seen in instances; should we allow you to form key paths to them? (Forming a key path to a no-argument case may be particularly useful.) + +These issues both require more implementation effort and deserve more design attention than metatype key paths could get as part of this proposal, so it makes sense to defer them. Nevertheless, this proposal is an important step in the right direction. diff --git a/proposals/0255-omit-return.md b/proposals/0255-omit-return.md new file mode 100644 index 0000000000..1392f5f411 --- /dev/null +++ b/proposals/0255-omit-return.md @@ -0,0 +1,362 @@ +# Implicit returns from single-expression functions + +* Proposal: [SE-0255](0255-omit-return.md) +* Author: [Nate Chandler](https://github.com/nate-chandler) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.1)** +* Implementation: [apple/swift#23251](https://github.com/apple/swift/pull/23251) +* Previous Revision: [1](https://github.com/DevAndArtist/swift-evolution/blob/single_expression_optional_return/proposals/nnnn-single-expression-optional-return.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-implicit-returns-from-single-expression-functions/21898)), ([review](https://forums.swift.org/t/se-0255-implicit-returns-from-single-expression-functions/22544)), ([acceptance](https://forums.swift.org/t/accepted-se-0255-implicit-returns-from-single-expression-functions/23581/1)) + +## Introduction + +Swift provides a pleasant shorthand for short closures: if a closure contains just a single expression, that expression is implicitly returned--the `return` keyword can be omitted. We should provide this shorthand for functions as well. + +Swift-evolution thread: [Pitch: Implicit Returns from Single-Expression Functions](https://forums.swift.org/t/pitch-implicit-returns-from-single-expression-functions/21898) + +## Motivation + +Consider the following implementation of the popular `Collection.sum()` extension: + +```swift +extension Sequence where Element == Int { + + func sum() -> Element { + return reduce(0, +) + } + +} +``` + +The implementation is extremely brief, weighing in at 19 characters. Of those 19, however, 7 are consumed by the `return` keyword. + +## Proposed solution + +Here's how that same property would look if the single-expression in the body is implicitly returned: + +```swift +func sum() -> Element { + reduce(0, +) +} +``` + +The expression which is being returned is front and center, not tucked away behind the `return` keyword. + +For readers previously exposed to single-expression closure syntax, this will feel completely familiar. + +Even to readers without such exposure, though, the meaning will be clear: When reading the implementation--after the `var` keyword and name--you first encounter the return type and then the single expression within the body. Since you've just read the return type, can see that there's only one expression in the body, and are told by the compiler that the code is legal, you are forced to conclude that the expression must indeed be returned. + +In fact, exposure to functions--whose types are always stated--without an explicit `return` keyword will help prepare new Swift users to understand code like + +```swift +let names = persons.map { $0.name } +``` + +Their prior exposure to functions which implicitly return the single expression in their bodies will lead them to conclude that the closure being passed to map is returning the expression `$0.name` and that the return type of the closure is `String` (the type of `name`, here). + +## Detailed design + +Interpret the bodies of function-like entities which consist of a single expression as returning that expression--unless the entity's return type is `Void` or the single expression's type is uninhabited. + +#### Function-like entities + +The following are the function-like entities eligible to implicitly return the single expression in their bodies: + +- Functions. For example: +```swift +func add(lhs: Int, rhs: Int) -> Int { lhs + rhs } +``` + +- Property accessors. + +With an implicit getter: + +```swift +var location: Location { .init(latitude: lat, longitude: long) } +``` + +With an explicit getter and setter: + +```swift +var location: Location { + get { + .init(latitude: lat, longitude: long) + } + set { + self.lat = newValue.latitude + self.long = newValue.longitude + } +} +``` + +Since only the `get` accessor may return a value, implicit returns from single-expression accessors will only affect them. + +- Subscript accessors. + +With an implicit getter: + +```swift +struct Echo { + subscript(_ value: T) -> T { value } +} +``` + +With an explicit getter and setter: + +```swift +struct GuaranteedDictionary { + var storage: [Key: Value] + var fallback: Value + subscript(key: Key) -> Value { + get { + storage[key] ?? fallback + } + set { + storage[key] = newValue + } + } +} +``` + +As with property accessors, since only the `get` accessor may return a value, implicit returns only affect them. + +- Initializers. + +```swift +class Derived: Base { + required init?() { nil } +} +``` + +The only legal return from an initializer is `nil`, and that only in the context of a failable initializer. As a result, that is the only place where an implicit return from an initializer can occur. + +#### Exceptions + +When a function-like entity's body consists of a single expression, there are two cases where no implicit return will be inserted: + +- `Void` return. In the following code + +```swift +func foo() { + logAndReturn("foo was called") +} + +@discardableResult +func logAndReturn(_ string: String) -> String { ... } +``` + +adding an implicit return to `foo` would result in a type error, namely, `unexpected non-void return in a void function`. It is reasonable to be able to call a function (here, `logAndReturn`) which returns a value as the only operation performed by another function (here `foo`) which does not return a value. Moreover, `foo` as written is legal code, so we want to avoid treating this as a type error since doing so would result in source breakage. + +- Uninhabited expressions. In the following code + +```swift +func vendAnInteger() -> Int { + fatalError() +} +``` + +adding an implicit return would result in the analogous type error (`cannot convert return expression of type 'Never' to return type 'Int'`). Functions which return values but whose implementations consist solely of a single call to a `Never` returning function are an established practice in Swift--they allow users to put off defining their functions until they are ready to (or forever). With implicit returns, this function's implementation will have the same meaning as it has today: The code will compile. No implicit return will be inserted. And at runtime the call to `fatalError()` will never return. Source compatibility will be preserved. + +There is one exception, as described in the section below: + +## Source compatibility + +For the most part, the change is additive, making legal code that is currently illegal. It does, however, break source compatibility in one case. + +In current Swift, when the following code is compiled + +``` +func bad(value: Int = 0) -> Int { return 0 } +func bad() -> Never { return fatalError() } + +func callBad() -> Int { bad() } +``` + +the call to `bad()` in `callBad()` resolves to the second overload of that name (whose signature is `() -> Never`). With implicit return, the call will instead resolve to the first overload. + +The large downside of breaking source-compatibility is mitigated by the fact that overload sets of which only one member returns `Never` are very rare: Extensive source compatibility tests have been run against this change without issue. + +## Effect on ABI stability + +None. Implementation is only in the parser and type checker. + +## Effect on API resilience + +None. + +## Alternatives considered + +- **Maintain source compatibility.** + +Maintaining source compatibility entails teaching the overload resolution system a special case for single-expression functions. It is possible to do this but would require complicating the type checker. Far worse it would complicate the language model: + +If source compatibility were maintained, the following two functions + +``` +func callBad_1() -> Int { bad() } + + +func callBad_2() -> Int { return bad() } +``` + +would have different behaviors: `callBad_1` would trap and `callBad_2` would return an `Int`. + +In a Swift with implicit return for single-expression functions, the mental model for users should be that a `return` can be omitted in certain cases and that it doesn't matter whether one is included or not. Preserving source-compatibility in this case would break that mental model. + +  + +- **Permit implicit return for a subset of declarations.** + +This document proposes allowing `return` to be omitted from the following declarations: +- functions +- properties +- subscripts +- initializers + +An alternative would be to allow that omission in only a subset of these. + +Concretely, several reasons were given for allowing it in only get-only computed properties: + +(1) Unlike functions, get-only properties already have one shorthand, the omission of `get`. By analogy to the situation with closures, that indicates that they are eligible for the further shorthand of omitting `return`. + +Response: This argument applies equally to subscripts which support the same shorthand as properties. If the reason to permit the `return` to be omitted from properties is that `get` can already be omitted, then that reason leads also to permitting `return` to be omitted from get-only subscripts. + +The differences between get-only subscripts and functions are already few and may be getting fewer ( https://forums.swift.org/t/pitch-static-and-class-subscripts/21850 , https://forums.swift.org/t/draft-throwing-properties-and-subscripts/1786 ). It would amount to a language inconsistency to allow get-only subscripts but not functions to omit `return`. + +(2) Unlike functions, get-only properties always have a return type. + +Response: In standard usage, it is much more common to encounter functions which return `Void` than properties. However, while that usage is far more common, the following is still part of the language: + +```swift +var doWork: Void { + work() +} +``` + +  + +- **Making uninhabited types be bottom types.** + +As currently implemented, an implicit conversion from an uninhabited type to any arbitrary type is permitted only if the uninhabited type is the type of the expression in a single-argument function and the arbitrary type is the result type of that function. If every uninhabited type were a subtype of every type, this implicit conversion could be applied across the board without special casing for the single-argument return scenario. + +While such a feature can be implemented (see the [uninhabited-upcast](https://github.com/nate-chandler/swift/tree/nate/uninhabited-upcast) branch), it doesn't maintain source compatibility or otherwise relate to this feature except in terms of the compiler's implementation. + +  + +- **Use braceless syntax for single-expression functions.** + +Some other languages such as Scala and Kotlin allow single-expression functions to be declared without braces. In Kotlin, this looks like + +```kotlin +fun squareOf(x: Int): Int = x * x +``` + +and in Scala, it looks almost identical (the only difference being the use of `def` instead of `fun`). + +```scala +def squareOf(x: Int): Int = x * x +``` + +Those languages' syntax suggests a similar approach be taken in Swift: + +```swift +func square(of x: Int) -> Int = x * x +``` + +For functions, this might be fine. For Swift to be self-consistent, a somewhat similar would be needed for properties and subscripts. + +```swift +var value: Int { + get = _storedValue + set { _storedValue = newValue } +} +``` + +Unfortunately, this begins moving into ambiguous territory: + +```swift +var get: () +var value: Void { + get = () +} +``` + +In this example, it's unclear whether the braces of `value` either (1) enclose an explicit getter for `value` whose implementation is a single-expression function returning `()` or alternatively (2) enclose the body of an implicit getter whose implementation sets the `get` property to `()`. + +  + +- **Allow implicit return of the last expression even from bodies which consist of more than a single expression.** + +Rust, for example, permits this. Given functions `foo`, `bar`, and `baz`, all which return integers, the following is a legal function in Rust: + +```rust +fn call() -> i64 { + foo(); + bar(); + baz() +} +``` + +While this could be permitted in Swift, doing so would lead to asymmetry in code resulting from the fact that Swift is not expression-oriented as Rust is. Consider a function with some basic branching: + +```swift +func evenOrOdd(_ int: Int) -> EvenOrOdd { + if int % 2 == 0 { + return .even + } + .odd +} +``` + +Here `.even` is returned for even `Int`s and `.odd` for odd. Notice that only one of the two returns from the function uses the return keyword! The same unpleasant function could be written in Rust: + +```rust +fn even_or_odd(i: i64) -> EvenOrOdd { + if i % 2 == 0 { + return EvenOrOdd::Even + } + EvenOrOdd::Odd +} +``` + +In Rust, though, the asymmetry could be resolved by implicitly returning the entire `if` expression: + +```rust +fn even_or_odd(i: i64) -> EvenOrOdd { + if i % 2 == 0 { + EvenOrOdd::Even + } else { + EvenOrOdd::Odd + } +} +``` + +That option is not open to us in Swift because conditionals are statements, not expressions in Swift. Changing Swift into an expression-oriented language would be a radical transformation to the language and is beyond the scope of this change. + +  + +- **Allow the return type to be omitted from the function declarations.** + +Scala, for example, permits this. In the following code + +```scala +def squareOf(x: Int) = x * x +``` + +the compiler infers that the type of `squareOf` is `(Int) -> Int`. + +Haskell takes this further, permitting functions to be written without either explicit inputs or outputs: + +```haskell +{-# LANGUAGE PartialTypeSignatures #-} +fac :: _ +fac 0 = 1 +fac n = n * fac (n - 1) +``` + +While these features are arguably nice, they greatly increase the complexity of type inference, and are out of scope for this change. + +## Acknowledgments + +A special thanks to Adrian Zubarev for his prior exploration of the design space culminating in his [proposal](https://github.com/DevAndArtist/swift-evolution/blob/single_expression_optional_return/proposals/nnnn-single-expression-optional-return.md). + diff --git a/proposals/0256-contiguous-collection.md b/proposals/0256-contiguous-collection.md new file mode 100644 index 0000000000..5c60a8517b --- /dev/null +++ b/proposals/0256-contiguous-collection.md @@ -0,0 +1,224 @@ +# Introduce `{Mutable}ContiguousCollection` protocol + +* Proposal: [SE-0256](0256-contiguous-collection.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Rejected** +* Previous Proposal: [SE-0237](0237-contiguous-collection.md) +* Implementation: [apple/swift#23616](https://github.com/apple/swift/pull/23616) +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0256-introduce-mutable-contiguouscollection-protocol/22569/8) + +## Introduction + +This proposal introduces two new protocols: `ContiguousCollection`, which +refines `Collection`, and `MutableContiguousCollection`, which refines +`MutableCollection`. Both provide guaranteed access to an underlying +unsafe buffer. + +## Motivation + +[SE-0237](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0237-contiguous-collection.md) +introduced two new methods, `withContiguous{Mutable}StorageIfAvailable`, to +allow generic algorithms to test for and use an underlying unsafe buffer when +it is available. This has significant performance benefit for certain +algorithms, such as `sort`, which can achieve big speedups when they +have access to the unsafe buffer, but can still operate without that fast path +if needed. + +There is another class of operation that _only_ wants to be available when +there is a fast path. A good example would be a Swift-friendly wrapper for +the `vDSP` suite of algorithms. + +For example, you might want to write a convenient wrapper for `vDSP_vadd`. + +```swift +// note this is **not** a proposal about vDSP wrappers, this is just a +// simplified example :) +func dspAdd( + _ a: A, _ b: B, _ result: inout [Float] +) where A.Element == Float, B.Element == Float { + let n = a.count + // try accessing contiguous underlying buffers: + let wasContiguous: ()?? = + a.withContiguousStorageIfAvailable { abuf in + b.withContiguousStorageIfAvailable { bbuf in + vDSP_vadd(abuf.baseAddress!, 1, bbuf.baseAddress!, 1, &result, 1, UInt(n)) + } + } + // if they weren't contiguous, create two arrays try again + if wasContiguous == nil || wasContiguous! == nil { + dspAdd(Array(a), Array(b), &result) + } +} +``` + +This follows a similar pattern to `sort`: provide a fast path when available, +but fall back to a slower path when it isn't. + +But in the case of functions like `vDSP_vsaddi` this is very much the wrong +thing to do. These functions often operate on a thin (but very material) +performance edge over their open-coded equivalent, and allocating and +initializing two arrays purely to be able to call it would probably vastly +outweigh the speed benefits gained by using the function instead of a regular +loop. This encourages misuse by the caller, who might not realize they are +getting worse performance than if they reorganized their code. + +Trapping on non-contiguous inputs would flag the problem more clearly to the +user, who could realize immediately on first use, that types like `Range` +should not be used with this API. But ideally we would use Swift's type system +to enforce this instead, guiding the user to a better solution. + +## Proposed solution + +Introduce two new protocols which guarantee access to a contiguous underlying +buffer. + +```swift +/// A collection that supports access to its underlying contiguous storage. +public protocol ContiguousCollection: Collection +where SubSequence: ContiguousCollection { + /// Calls a closure with a pointer to the array's contiguous storage. + func withUnsafeBufferPointer( + _ body: (UnsafeBufferPointer) throws -> R + ) rethrows -> R +} + +/// A collection that supports mutable access to its underlying contiguous +/// storage. +public protocol MutableContiguousCollection: ContiguousCollection, MutableCollection +where SubSequence: MutableContiguousCollection { + /// Calls the given closure with a pointer to the array's mutable contiguous + /// storage. + mutating func withUnsafeMutableBufferPointer( + _ body: (inout UnsafeMutableBufferPointer) throws -> R + ) rethrows -> R +} +``` + +Conformances will be added for the following types: + +- `Array`, `ArraySlice` and `ContiguousArray` will conform to `MutableContiguousCollection` +- `UnsafeBufferPointer` will conform to `ContiguousCollection` +- `UnsafeMutableBufferPointer` will conform to `MutableContiguousCollection` +- `Slice` will conditionally conform: + - to `ContiguousCollection where Base: ContiguousCollection` + - to `MutableContiguousCollection where Base: MutableContiguousCollection` + +Conforming to to `ContiguousCollection` should also provide types with a default +implementation of `Collection.withContiguousStorageIfAvailable`, via an extension +that calls `withUnsafeBufferPointer`. Same for `MutableContiguousCollection` and +`Collection.withMutableContiguousStorageIfAvailable`. + +## Detailed design + +The introduction of these protocols allows an appropriate constraint that would +prevent a user passing a `Range` or `Repeating` collection into our `dspAdd` +function. It also allows an easy path to a generic result buffer instead of a +concrete array; this is important as often these functions are used in a tiled +mode where you would want to repeatedly pass in an array slice. As a nice +side-benefit, it also cleans up the function implementation: + +```swift +func dspAdd( + _ a: A, _ b: B, _ result: inout R +) where A.Element == Float, B.Element == Float, R.Element == Float { + let n = a.count + a.withUnsafeBufferPointer { abuf in + b.withUnsafeBufferPointer { bbuf in + result.withUnsafeMutableBufferPointer { rbuf in + vDSP_vadd(abuf.baseAddress!, 1, bbuf.baseAddress!, 1, rbuf.baseAddress!, 1, UInt(n)) + } + } + } +} +``` + +## Source compatibility + +These are additive changes and do not affect source compatibility. + +## Effect on ABI stability + +These are additive changes of new protocols and so can be introduced in an +ABI-stable way. On platforms that have declared ABI stability, they will need +to have availability annotations. + +## Effect on API resilience + +N/A + +## Alternatives considered + +This is a re-pitch of these protocols. They originally appeared in SE-0237, but +were deferred pending further use cases. This proposal is motivated by such +cases. + +As mentioned in the motivation, there are some alternatives available in the +absence of this protocol: + +- Libraries can add their own version of the protocol, and retroactively + conform standard library types to it. This is a workable solution, but has some downsides: + + - Non-standard types will not conform. Callers will need to conform these types to the + library's protocol themselves manually. A standard library protocol is more likely to + be adopted by other types. + + - If this pattern proves common, it will lead to multiple libraries declaring + the same protocol. + +- Libraries can use the `IfAvailable` variant, and document that using types + without contiguous storage is inefficient. This leaves enforcing the correct + usage to the user. This is not always possible in a generic context, where + the calling function does not know exactly what concrete type they are using. + +- Libraries can use the `IfAvailable` variant, and trap when not available. + This could alert callers to inefficient usage on first use. For example, if + you ever pass in a `Range`, your call will always fail, detectable by any + testing. Some types, however, may respond to the `IfAvailable` call in some + cases but not others. For example, a ring buffer might often not have wrapped + around, so can provide a single contiguous buffer sometimes, but not always. + Trapping then would lead to unpredictable crashes. + +### `Array` and lazy bridging + +The conformance of `Array` to these protocols presents a particular concern. + +`Array` at its core is a contiguously stored block of memory, and so naturally +lends itself to conformance to `ContiguousCollection`. However, this has one +specific carved-out exception on Darwin platforms for the purposes of +Objective-C interop. When an `NSArray` of classes is bridged from Objective-C +code, it remains as an `NSArray`, and the `Array` forwards element accesses to +that bridged `NSArray`. + +This is very different from `NSArray` itself, which abstracts away the storage +to a much greater degree, giving it flexibility to present an "array-like" +interface to multiple different backings. + +Here's a run-down of when Array will be contiguously stored: + +- Arrays created in Swift will **always** be contiguously stored; + +- Arrays of structs and enums will **always** be contiguously stored; + +- Arrays on platforms without an Objective-C runtime (i.e. non-Darwin + platforms) are **always** contiguously stored; + +- The only time an array **won't** be contiguously stored is if it is of + classes and has been bridged from an `NSArray`. Even then, in many cases, the + NSArray will be contiguously stored and could present a pointer at no or + amortized cost. + +These caveats should be documented clearly on both the protocol and on `Array`. +Note that in use cases such as the `vDSP` family of functions, this is not a +concern as the element types involved are structs. The documented caveat +approach has precedent in several places already in Swift: the conformance of +floating-point types to `Comparable`, the complexity of operations like `first` +and `last` on some lazy random-access types, and of the existing implementation +of `withUnsafeBufferPointer` on `Array` itself. + +Note that these caveats only apply to the un-mutable variant. The +first thing an array does when you call a mutating method is ensure that it's +uniquely referenced and contiguous, so even lazily bridged arrays will become +contiguous at that point. This copying occurs naturally in other cases, such +as multiply-referenced CoW buffers. + diff --git a/proposals/0257-elide-comma.md b/proposals/0257-elide-comma.md new file mode 100644 index 0000000000..9f08aa63f1 --- /dev/null +++ b/proposals/0257-elide-comma.md @@ -0,0 +1,667 @@ +# Eliding commas from multiline expression lists + +* Proposal: [SE-0257](0257-elide-comma.md) +* Author: [Nate Chandler](https://github.com/nate-chandler), [Matthew Johnson](https://github.com/anandabits) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Returned for revision** +* Implementation: [apple/swift#21876](https://github.com/apple/swift/pull/22714) +* Review: ([previous pitch 1](https://forums.swift.org/t/se-0084-spinoff-newlines-as-item-separators/2659)) ([previous pitch 2](https://forums.swift.org/t/trailing-commas-in-all-expression-lists/19527)) ([pitch](https://forums.swift.org/t/pitch-eliding-commas-from-multiline-expression-lists/22558)) ([review](https://forums.swift.org/t/se-0257-eliding-commas-from-multiline-expression-lists/22889)) ([returned for revision](https://forums.swift.org/t/se-0257-eliding-commas-from-multiline-expression-lists/22889/191)) + +# Introduction + +Swift requires a semicolon "`;`" to separate statements unless those statements are separated by newlines, in which case the semicolon can be elided. Currently, Swift requires a comma "`,`" to separate expressions even when those statements are separated by newlines. We should ease this restriction, allowing the comma between two expressions to be elided when they are separated by a newline. + +* Implementation: [apple/swift#21876](https://github.com/apple/swift/pull/22714) +* Previous Pitch: [SE-0084 spinoff: Newlines as item separators](https://forums.swift.org/t/se-0084-spinoff-newlines-as-item-separators/2659) + +## Reducing visual clutter + +When writing a list of expressions, you insert commas to tell the compiler where one expression ends and the next begins. When you add a newline between the expressions, though, you provide that same information by way of the newline characters. When newlines are present, commas provide clarity neither to human readers nor to the compiler. + +In these cases, at best, commas can be overlooked by human readers. At worst, they cause visual clutter and obscure the meaning you are trying to communicate in our code. Consider the following sample, taken from [Alamofire](https://github.com/Alamofire/Alamofire/blob/4df5912df4ebb138454aeba78f1470acc52dd4ae/Source/Request.swift#L649-L655): + +```swift +let protectionSpace = URLProtectionSpace( + host: host, + port: url.port ?? 0, + protocol: url.scheme, + realm: host, + authenticationMethod: NSURLAuthenticationMethodHTTPBasic +) +``` + +The commas here are not communicating anything. The writer has to put them in, the compiler has to observe that they're there and move along, and the reader has to filter them out. They're noise for all parties involved. Compare that to the following: + +```swift +let protectionSpace = URLProtectionSpace( + host: host + port: url.port ?? 0 + protocol: url.scheme + realm: host + authenticationMethod: NSURLAuthenticationMethodHTTPBasic +) +``` + +The difference is small, but significant: +1. **There are no spurious characters between the parentheses**. You are presented with a list of the names of the ingredients being used to construct the `URLProtectionSpace` on the left hand side of the colons and on the right hand side you see the values which are serving as those ingredients. +2. **The lines are symmetric**. the last line, lacking a comma, looks no different from the others, because they all lack a comma. +3. **Each line stands on its own**. Because they appear in single line argument lists, a comma at the end of a line has the effect of drawing your eye down to the next line. Without the commas, you have a moment to breathe at the end of the line, maybe to glance back at the argument label before moving on to the next line. + +Let's take a look at a couple more examples. + +To begin with, let's look at constructing another [type](https://github.com/apple/swift/blob/master/benchmark/utils/DriverUtils.swift#L59-L98) which takes many arguments. Here it is with commas: + +```swift +let config = TestConfig( + delim: "abc", + sampleTime: 42.0, + numIters: nil, + numSamples: nil, + quantile: nil, + delta: true, + verbose: false, + logMemory: true, + afterRunSleep: nil +) +``` + +and without: + +```swift +let config = TestConfig( + delim: "abc" + sampleTime: 42.0 + numIters: nil + numSamples: nil + quantile: nil + delta: true + verbose: false + logMemory: true + afterRunSleep: nil +) +``` + +Once again, the result is cleaner. All the characters that you see are relevant and meaningful. Each line you see is like the others. You're not drawn from one line to the next by the comma, you're free to scan through the items at your leisure. + +These same improvements are visible in expression lists besides the arguments to an initializer. Consider the following [function calls](https://github.com/apple/swift/blob/master/benchmark/single-source/StringComparison.swift), first with commas: + +```swift +StringTests.test("AssociatedTypes-UTF8View") { + typealias View = String.UTF8View + expectCollectionAssociatedTypes( + collectionType: View.self, + iteratorType: View.Iterator.self, + subSequenceType: Substring.UTF8View.self, + indexType: View.Index.self, + indicesType: DefaultIndices.self) +} + +StringTests.test("AssociatedTypes-UTF16View") { + typealias View = String.UTF16View + expectCollectionAssociatedTypes( + collectionType: View.self, + iteratorType: View.Iterator.self, + subSequenceType: Substring.UTF16View.self, + indexType: View.Index.self, + indicesType: View.Indices.self) +} +``` + +and then without: + + +```swift +StringTests.test("AssociatedTypes-UTF8View") { + typealias View = String.UTF8View + expectCollectionAssociatedTypes( + collectionType: View.self + iteratorType: View.Iterator.self + subSequenceType: Substring.UTF8View.self + indexType: View.Index.self + indicesType: DefaultIndices.self) +} + +StringTests.test("AssociatedTypes-UTF16View") { + typealias View = String.UTF16View + expectCollectionAssociatedTypes( + collectionType: View.self + iteratorType: View.Iterator.self + subSequenceType: Substring.UTF16View.self + indexType: View.Index.self + indicesType: View.Indices.self) +} +``` + +The difference is subtle but striking. + +## Making domain-specific languages first-class + +One case where the problem of visual clutter is especially pronounced is in domain-specific languages embedded (EDSLs) within Swift. Particularly when those EDSLs are "declarative". + +Consider an EDSL for specifying a table in a database: + +```swift +let table = Table( + name: "Employees", + columns: + guid ("record_id", isPrimaryKey: true, nullable: false), + guid ("manager_id", isPrimaryKey: false, nullable: true), + string ("name", length: 1024, nullable: false), + int64 ("employee_id", nullable: false), + date ("start_date", nullable: false) +) +``` + +Beyond merely defining a table, the intent is to do so in a way that feels natural in this context, to define a table "in its own terms". The majority of defining the table is taken up providing a list of columns, declaring them type first, in C style. + +The corresponding declaration in native Swift would look something like this: + +```swift +let recordID = SQLGUIDColumn(isPrimaryKey: true, nullable: false); +let managerID = SQLGUIDColumn(isPrimaryKey: true, nullable: true); +let name = SQLStringColumn(length: 1024, nullable: false); +let employeeID = SQLInt64Column(nullable: false); +let startDate = SQLDateColumn(nullable: false); +``` + +Note in particular the use of semicolons at the end. While the semicolons are not a huge problem, they are unpleasant. The main reason that they are unpleasant is that they are superfluous. We are merely going through some ceremony that is of service to nobody: not the writer, not the reader, and not the compiler. For that reason, semicolons are not required to end statements that are terminated by line endings in Swift. Instead, Swift allows you to write + +```swift +let recordID = SQLGUIDColumn(isPrimaryKey: true, nullable: false) +let managerID = SQLGUIDColumn(isPrimaryKey: true, nullable: true) +let name = SQLStringColumn(length: 1024, nullable: false) +let employeeID = SQLInt64Column(nullable: false) +let startDate = SQLDateColumn(nullable: false) +``` + +The situation with definition of the table in the EDSL is the same. The commas are providing information to nothing and nobody. They are ceremony for its own sake. Moreover, they are a constant visual reminder that you are looking at a list of arguments being passed to a function rather than a list of declarations in the EDSL. + +Just as Swift allows semicolons to be omitted, it should also allow the commas to be omitted, permitting + +```swift +let table = Table( + name: "Employees" + columns: + guid ("record_id", isPrimaryKey: true, nullable: false) + guid ("manager_id", isPrimaryKey: false, nullable: true) + string ("name", length: 1024, nullable: false) + int64 ("employee_id", nullable: false) + date ("start_date", nullable: false) +) +``` + +With the commas present, the reader is saddled with a constant reminder that she is "just" reading a list of arguments, not language uses that stand on their own. Once they are removed, the author can express her intent directly, that the table is defined by a series of declarations, *not* by passing a series of arguments to a function. That fact, while still visible, is reduced from its current overbearing status to an implementation detail of the EDSL. + +Without the commas, each column can be viewed as intended: *as a **declaration** in the EDSL*. + +By allowing statement separators to be omitted when a newline is present while requiring expression separators Swift exhibits a subtle bias, making imperative code look clean where declarative code must be cluttered with punctuation. By providing the same affordance for declarative style, this bias towards imperative style is lifted, allowing declarative EDSLs to feel just as natural as imperative Swift code. + +This example EDSL is not taken from a real library. There are, despite the current limitation, many shipping Swift libraries that make extensive use of EDSLs. Let's take a look at the EDSLs from those in several different domains to see how they would be improved by comma elision: + +### Networking EDSL case study: HTTP Requests + +HTTP requests are pervasive in modern applications. To avoid dealing directly with networking-layer API, higher-level abstractions are often used to specify requests. Here's an example taken from some real-world code: + +```swift +Request( + method: .get, + host: .someHost, + path: "/path/to/model", + query: [ + "page": 10, + "order": "ascending", + ] +) +``` + +A request is specified by a series of declarations. The request will use the GET verb. It will be made against `someHost` at "/path/to/model". The content should be ordered ascending and there should be ten entities per page. + +The commas here add no value. At the least, they are line noise. At worst, they obscure the fact that there is a series of declarations, forcing the reader back into viewing the each line as *just* an argument to an initializer. + +Here's the same code without commas: + +```swift +Request( + method: .get + host: .someHost + path: "/path/to/model" + query: [ + "page": 10 + "order": "ascending" + ] +) +``` + +As you read the code, first you see that a Request is being initialized. In that context, you see a number of declarations about the request, that it uses the GET verb and so on. The elision of commas allows you to focus on what's important, the declarations that make up the definition of the `Request`. You are not constantly reminded that each line is an argument passed to an initializer but are instead free to think of each as a declaration in the language in which HTTP requests are specified. + +### App Routing EDSL case study: Jason Prasad's Routing library + +The [routing](https://github.com/jjgp/Routing) library provides a convenient API for specifying which screen of an app should be displayed when the app is launched via a URL. Here's an example usage taken from the project's README: + +```swift +router.map("routingexample://present/login", + source: .storyboard(storyboard: "Main", identifier: "LoginViewController", bundle: nil), + style: .inNavigationController(.present(animated: true)), + setup: presentationSetup) +``` + +This code specifies that when the app is launched with the `present/login` path, the `LoginViewController` from the `Main` storyboard will be presented in a navigation controller. + +Here's how that code looks without commas and with newlines added instead: + +```swift +router.map( + "routingexample://present/login" + source: .storyboard(name: "Main" + identifier: "LoginViewController" + bundle: nil) + style: .inNavigationController(.present(animated: true)) + setup: presentationSetup +) +``` + +Without the commas, and with newlines added to the storyboard example, the code looks much cleaner. Moreover, while in Swift an instance of the `ControllerSource` enum is being instantiated via the `.storyboard` implicit member access, in this code snippet, you are free to ignore those mechanics and focus instead on the declaration of a location from which to obtain a view controller in a storyboard named "Main" under the identifier "LoginViewController". + +## Improving the editing experience + +Another, more minor point is that commas in these positions cause problems when editing code. In Swift today, you can easily add or remove any item--even the last--from a collection literal by commenting it in or out: + +```swift +let colors = [ + "red", + "green", + "blue", +// "cerulean" +] +``` + +Unfortunately that convenience is not available fully in the other expression lists. For example, in a multiline function call, it is a breeze to comment out any argument + +```swift +print( + "red", +// "green", // ok + "blue", + "cerulean" +) +``` + +*except* the last; commenting it out raises an error: + + +```swift +print( + "red", + "green", + "blue", // error: unexpected ',' separator +// "cerulean" +) +``` + +The reason for these inconsistent behaviors is that trailing commas are permitted in collection literals but not in any other expression list. + +One solution would be to allow trailing commas in all expression lists. That change, however, only addresses part of the problem. The visual noise that the commas cause not only remains but is magnified: to get this convenience, we would be incentivized to write our code with trailing commas in all multiline expression lists. + +Instead, we should allow commas to be elided from multiline expression lists entirely. Without commas, the original function call would instead look like + +```swift +print( + "red" + "green" + "blue" + "cerulean" +) +``` + +with its arguments untarnished by commas. We would be free to comment out any line of it, including the last + +```swift +print( + "red" + "green" + "blue" +// "cerulean" +) +``` + +because what remains is again a multiline expression list with commas elided. + +## Proposed solution + +Rather than allowing comma elision in just some expression lists in an ad hoc fashion, this document proposes allowing commas to be elided uniformly in all multiline expression lists. + +### When will you still use commas? + +When parsing an expression, the compiler keeps going until it can't any longer, following the [maximal munch](https://en.wikipedia.org/wiki/Maximal_munch) principle. Sometimes, though, you want one expression to end and the next to begin before the parser would otherwise stop. In those situations, you will use a comma to communicate that intent. + +There are two scenarios where you will use commas to clarify that one expression is ending and the next is beginning: + +#### Implicit members + +When parsing a multiline expression list featuring a member expression which appears after a newline + +```swift +foo( + bar + .baz +) +``` + +the member expression will be interpreted as modifying the expression that preceded it + +```swift +foo(bar.baz) +``` + +rather than as a separate expression + +```swift +foo(bar, .baz) +``` + +If you actually want `.baz` to be as an implicit member, an expression in its own right, you will add a comma: + +```swift +foo( + bar, + .baz +) +``` + +##### Diagnostics + +When this situation comes up, the issue can be ameliorated via a good diagnostic experience. If you attempt to compile the original function call + +```swift +foo( + bar + .baz +) +``` + +in a context where `baz` is not an instance member of `bar`'s type but is a static member of the type of the second parameter accepted by `foo` + +```swift +enum E { + case baz +} + +func foo(_ bar: Bar, _ e: E) {...} +``` + +the error can be diagnosed with a fixit to insert a comma at the end of the line before `.baz`, right after `bar`, leaving you with the code you intended: + +```swift +foo( + bar, + .baz +) +``` + +If you attempt to compile the original function call + +```swift +foo( + bar + .baz +) +``` + +in a context where both `foo(bar.baz)` and `foo(bar, .baz)` are legal, a warning can be emitted with several fixits to clarify the intent by doing one of the following: + +1. inserting a comma + +```swift +foo( + bar, + .baz +) +``` + +2. eliminating the newline + +```swift +foo( + bar.baz +) +``` + +3. indenting the second expression + +```swift +foo( + bar + .baz +) +``` + +as is suggested today for an expression on the line after the `return` keyword at its same indentation level. + +#### Closures + +In a similar vein, when parsing a multiline expression list featuring a closure which appears after a newline + +```swift +foo( + bar + { print("baz") } +) +``` + +the closure will be interpreted as a trailing closure passed as an argument to the expression that preceded it. + +```swift +foo(bar { print("baz") }) +``` + +rather than as a separate expression + +```swift +foo(bar, { print("baz") } +``` + +If you actually want the closure to stand on its own, to be its own expression, you will add a comma to separate it from the preceding expression. + +```swift +foo( + bar, + { print("baz") } +) +``` + +##### Diagnostics + +As with implicit members, we will be able to provide a good diagnostic experience here including both + +1. an error with a fixit to insert a comma before the closure which is being parsed as a trailing closure in the case where using the closure as a trailing closure doesn't typecheck +2. a warning with multiple fixits to insert a comma or change whitespace in the case where the closure could be used both as a trailing closure and as a top level expression in the list + +#### Alternatives to adding commas + +These situations may sound familiar--they are exactly the same situations where we need to use semicolons to separate items in statement lists, even in the presence of newlines. In practice, you will need to use commas more often than semicolons because it is more often for these expressions to appear in expression lists than in statement lists. + +That said, you will need to use them less often than it might at first seem. + +Consider closures: Trailing closure syntax means that most of the time, closures appear after the end of the expression list. Typically, the above example would actually be written + +```swift +foo( + bar +) +{ + print("baz") +} +``` + +What about implicit members? Consider a function call like this: + +```swift +buyTreat( + .sweet + .orange +) +``` + +This would be parsed as `.sweet.orange`, which may not be what you want. Even to a human, reader, though, it's not clear what is meant. To make code obvious to readers, you often use argument labels (`flavor: .orange`) to provide a hint to readers of what the implicit member may be a member of: + +```swift +buyTreat( + .sweet + flavor: .orange +) +``` + +If you would prefer to leave out an argument label, you could also provide the type `Flavor.orange` in order to provide a reader with that context: + +```swift +buyTreat( + .sweet + Flavor.orange +) +``` + +If you don't want to use either of those approaches, only then will you end the prior expression with a comma. + +Without this change, you are forced to use commas everywhere. In multiline expression lists, they are reduced to line noise and meaninglessness. A comma is a character to be ignored. With this change, if you omit commas whenever possible, when you write a comma, you will mean something: "without this, the previous expression would keep going; I want it to end here." + +## Detailed design + +Swift will allow the comma separating two expressions in an expression list to be elided provided that there is a newline separating the expressions. + +The grammatical productions from The Swift Programming Language will be modified as follows: + +
+expression-list -> expression | expression , expression-list | expression \n expression-list 
+function-call-argument-list -> function-call-argument | function-call-argument , function-call-argument-list | function-call-argument \n function-call-argument-list
+tuple-element-list -> tuple-element | tuple-element , tuple-element-list | tuple-element \n tuple-element-list
+
+ +With these few changes to the grammatical productions, comma elision will be accepted in the following positions: + +- array literals +```swift +[ + "red" + "green" +] +``` +- dictionary literals +```swift +[ + "red" : 4 + "green" : 8 +] +``` +- free function calls +```swift +print( + "red" + "green" +) +``` +- method calls +```swift +foo.print( + "red" + "green" +) +``` +- initializer calls +```swift +let instance = Widget( + "red" + "green" +) +``` +- subscript reads +```swift +foo[ + "red" + "green" +] +``` +- subscript writes +```swift +foo[ + "red" + "green" +] = "yellow" +``` +- super method calls +```swift +super.print( + "red" + "green" +) +``` +- super initializer calls +```swift +super.init( + "red" + "green" +) +``` +- super subscript reads +```swift +super[ + "red" + "green" +] +``` +- super subscript writes +```swift +super[ + "red" + "green" +] = "yellow" +``` +- enum instantiations +```swift +let e = E.foo( + "red" + "green" +) +``` +- tuple instantiations +```swift +let t = ( + "red" + "green" +) +``` +- key-path subscripts +```swift +let path = \Gadget[ + 0 + 1 +] +``` + +## Source compatibility + +This is not a source-breaking change. Extensive compatibility tests have been run against the change. + +This document does *not* propose removing commas from the language. All code that is legal today will continue to be legal. This document proposes easing a restriction, making more code legal. + +@blangmuir looked into SourceKit's under the change and determined everything just works without any other changes. Autocomplete continues to function as before. + +Because statement separator (i.e. semicolon) elision has been in the language for so long, all the engineering problems for expression separator (i.e. comma) elision have already been solved. + +## Effect on ABI stability + +N/A + +## Effect on API resilience + +N/A + +## Alternatives considered + +- Allow trailing commas in expression lists. + +While trailing commas in expression lists would provide the same improvements in the editing experience that comma elision does, they do not bring the same readability improvements to the language as comma elision. + +- Base interpretation of arguments off of semantic information. + +The two cases where commas will still be written listed above may seem less than ideal. It is tempting to ask whether we could decide the number of expressions in the expression list based on the context in which it appears. Swift does not currently do this sort of interpretation based on semantic information and doing so massively complicates the language. + diff --git a/proposals/0258-property-wrappers.md b/proposals/0258-property-wrappers.md new file mode 100644 index 0000000000..68c8be7a77 --- /dev/null +++ b/proposals/0258-property-wrappers.md @@ -0,0 +1,1548 @@ +# Property Wrappers + +* Proposal: [SE-0258](0258-property-wrappers.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Joe Groff](https://github.com/jckarter) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.1)** +* Implementation: [Linux toolchain](https://ci.swift.org/job/swift-PR-toolchain-Linux/251//artifact/branch-master/swift-PR-25781-251-ubuntu16.04.tar.gz), [macOS toolchain](https://ci.swift.org/job/swift-PR-toolchain-osx/327//artifact/branch-master/swift-PR-25781-327-osx.tar.gz) +* Review: ([review #1](https://forums.swift.org/t/se-0258-property-delegates/23139)) ([revision announcement #1](https://forums.swift.org/t/returned-for-revision-se-0258-property-delegates/24080)) ([review #2](https://forums.swift.org/t/se-0258-property-wrappers-second-review/25843)) ([review #3](https://forums.swift.org/t/se-0258-property-wrappers-third-review/26399)) ([acceptance](https://forums.swift.org/t/accepted-with-modification-se-0258-property-wrappers/26828)) +* Previous versions: [Revision #3](https://github.com/swiftlang/swift-evolution/blob/e99ae69370f56ae84256b78902ab377cb8249cdd/proposals/0258-property-wrappers.md), [Revision #2](https://github.com/swiftlang/swift-evolution/blob/bb8709c2ddca25c21a3c1e0298ce9457911dbfba/proposals/0258-property-wrappers.md), [Revision #1](https://github.com/swiftlang/swift-evolution/commit/8c3499ec5bc22713b150e2234516af3cb8b16a0b) + +## Contents + ++ [Introduction](#introduction) ++ [Motivation](#motivation) ++ [Proposed solution](#proposed-solution) ++ [Examples](#examples) + - [Delayed Initialization](#delayed-initialization) + - [`NSCopying`](#nscopying) + - [Thread-specific storage](#thread-specific-storage) + - [User defaults](#user-defaults) + - [Copy-on-write](#copy-on-write) + - [`Ref` / `Box`](#ref--box) + - ["Clamping" a value within bounds](#clamping-a-value-within-bounds) + - [Property wrapper types in the wild](#property-wrapper-types-in-the-wild) ++ [Composition of property wrappers](#composition-of-property-wrappers) ++ [Detailed design](#detailed-design) + - [Property wrapper types](#property-wrapper-types) + - [Initialization of synthesized storage properties](#initialization-of-synthesized-storage-properties) + - [Type inference with property wrappers](#type-inference-with-property-wrappers) + - [Custom attributes](#custom-attributes) + - [Mutability of properties with wrappers](#mutability-of-properties-with-wrappers) + - [Out-of-line initialization of properties with wrappers](#out-of-line-initialization-of-properties-with-wrappers) + - [Memberwise initializers](#memberwise-initializers) + - [Codable, Hashable, and Equatable synthesis](#codable-hashable-and-equatable-synthesis) + - [$ identifiers](#-identifiers) + - [Projections](#projections) + - [Restrictions on the use of property wrappers](#restrictions-on-the-use-of-property-wrappers) ++ [Impact on existing code](#impact-on-existing-code) ++ [Backward compatibility](#backward-compatibility) ++ [Alternatives considered](#alternatives-considered) + - [Composition](#composition) + - [Composition via nested type lookup](#composition-via-nested-type-lookup) + - [Composition without nesting](#composition-without-nesting) + - [Using a formal protocol instead of `@propertyWrapper`](#using-a-formal-protocol-instead-of-propertywrapper) + - [Kotlin-like `by` syntax](#kotlin-like-by-syntax) + - [Alternative spellings for the `$` projection property](#alternative-spellings-for-the--projection-property) + - [The 2015-2016 property behaviors design](#the-2015-2016-property-behaviors-design) ++ [Future Directions](#future-directions) + - [Finer-grained access control](#finer-grained-access-control) + - [Referencing the enclosing 'self' in a wrapper type](#referencing-the-enclosing-self-in-a-wrapper-type) + - [Delegating to an existing property](#delegating-to-an-existing-property) ++ [Revisions](#revisions) + - [Changes from the accepted proposal](#changes-from-the-accepted-proposal) + - [Changes from the third reviewed version](#changes-from-the-third-reviewed-version) + - [Changes from the second reviewed version](#changes-from-the-second-reviewed-version) + - [Changes from the first reviewed version](#changes-from-the-first-reviewed-version) ++ [Acknowledgments](#acknowledgments) + +## Introduction + +There are property implementation patterns that come up repeatedly. +Rather than hardcode a fixed set of patterns into the compiler (as we have done for `lazy` and `@NSCopying`), +we should provide a general "property wrapper" mechanism to allow +these patterns to be defined as libraries. + +This is an alternative approach to some of the problems intended to be addressed by the [2015-2016 property behaviors proposal](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0030-property-behavior-decls.md). Some of the examples are the same, but this proposal takes a completely different approach designed to be simpler, easier to understand for users, and less invasive in the compiler implementation. There is a section that discusses the substantive differences from that design near the end of this proposal. + +[Pitch #1](https://forums.swift.org/t/pitch-property-delegates/21895)
+[Pitch #2](https://forums.swift.org/t/pitch-2-property-delegates-by-custom-attributes/22855)
+[Pitch #3](https://forums.swift.org/t/pitch-3-property-wrappers-formerly-known-as-property-delegates/24961)
+ +## Motivation + +We've tried to accommodate several important patterns for properties with +targeted language support, like `lazy` and `@NSCopying`, but this support has been narrow in scope and utility. For instance, Swift provides `lazy` properties as a primitive language feature, since lazy initialization is common and is often necessary to avoid having properties be exposed as `Optional`. Without this language support, it takes a lot of boilerplate to get the same effect: + +```swift +struct Foo { + // lazy var foo = 1738 + private var _foo: Int? + var foo: Int { + get { + if let value = _foo { return value } + let initialValue = 1738 + _foo = initialValue + return initialValue + } + set { + _foo = newValue + } + } +} +``` + +Building `lazy` into the language has several disadvantages. It makes the +language and compiler more complex and less orthogonal. It's also inflexible; +there are many variations on lazy initialization that make sense, but we +wouldn't want to hardcode language support for all of them. + +There are important property patterns outside of lazy initialization. It often +makes sense to have "delayed", once-assignable-then-immutable properties to +support multi-phase initialization: + +```swift +class Foo { + let immediatelyInitialized = "foo" + var _initializedLater: String? + + // We want initializedLater to present like a non-optional 'let' to user code; + // it can only be assigned once, and can't be accessed before being assigned. + var initializedLater: String { + get { return _initializedLater! } + set { + assert(_initializedLater == nil) + _initializedLater = newValue + } + } +} +``` + +Implicitly-unwrapped optionals allow this in a pinch, but give up a lot of +safety compared to a non-optional 'let'. Using IUO for multi-phase +initialization gives up both immutability and nil-safety. + +The attribute `@NSCopying` introduces a use of `NSCopying.copy()` to +create a copy on assignment. The implementation pattern may look familiar: + +```swift +class Foo { + // @NSCopying var text: NSAttributedString + var _text: NSAttributedString + var text: NSAttributedString { + get { return _text } + set { _text = newValue.copy() as! NSAttributedString } + } +} +``` + +## Proposed solution + +We propose the introduction of **property wrappers**, which allow a +property declaration to state which **wrapper** is used to implement +it. The wrapper is described via an attribute: + +```swift +@Lazy var foo = 1738 +``` + +This implements the property `foo` in a way described by the *property wrapper type* for `Lazy`: + +```swift +@propertyWrapper +enum Lazy { + case uninitialized(() -> Value) + case initialized(Value) + + init(wrappedValue: @autoclosure @escaping () -> Value) { + self = .uninitialized(wrappedValue) + } + + var wrappedValue: Value { + mutating get { + switch self { + case .uninitialized(let initializer): + let value = initializer() + self = .initialized(value) + return value + case .initialized(let value): + return value + } + } + set { + self = .initialized(newValue) + } + } +} +``` + +A property wrapper type provides the storage for a property that +uses it as a wrapper. The `wrappedValue` property of the wrapper type +provides the actual +implementation of the wrapper, while the (optional) +`init(wrappedValue:)` enables initialization of the storage from a +value of the property's type. The property declaration + +```swift +@Lazy var foo = 1738 +``` + +translates to: + +```swift +private var _foo: Lazy = Lazy(wrappedValue: 1738) +var foo: Int { + get { return _foo.wrappedValue } + set { _foo.wrappedValue = newValue } +} +``` + +The use of the prefix `_` for the synthesized storage property name is +deliberate: it provides a predictable name for the synthesized storage property that +fits established conventions for `private` stored properties. For example, +we could provide a `reset(_:)` operation on `Lazy` to set it back to a new +value: + +```swift +extension Lazy { + /// Reset the state back to "uninitialized" with a new, + /// possibly-different initial value to be computed on the next access. + mutating func reset(_ newValue: @autoclosure @escaping () -> Value) { + self = .uninitialized(newValue) + } +} + +_foo.reset(42) +``` + +The backing storage property can also be explicitly initialized. For example: + +```swift +extension Lazy { + init(body: @escaping () -> Value) { + self = .uninitialized(body) + } +} + +func createAString() -> String { ... } + +@Lazy var bar: String // not initialized yet +_bar = Lazy(body: createAString) +``` + +The property wrapper instance can be initialized directly by providing the initializer arguments in parentheses after the name. The above code can be written equivalently in a single declaration as: + +```swift +@Lazy(body: createAString) var bar: String +``` + +Property wrappers can be applied to properties at global, local, or type scope. Those properties can have observing accessors (`willSet`/`didSet`), but not explicitly-written getters or setters. + +The `Lazy` property wrapper has little or no interesting API outside of its initializers, so it is not important to export it to clients. However, property wrappers can also describe rich relationships that themselves have interesting API. For example, we might have a notion of a property wrapper that references a database field established by name (example inspired by [Tanner](https://forums.swift.org/t/se-0258-property-wrappers-second-review/25843/14)): + +```swift +@propertyWrapper +public struct Field { + public let name: String + private var record: DatabaseRecord? + private var cachedValue: Value? + + public init(name: String) { + self.name = name + } + + public func configure(record: DatabaseRecord) { + self.record = record + } + + public var wrappedValue: Value { + mutating get { + if cachedValue == nil { fetch() } + return cachedValue! + } + + set { + cachedValue = newValue + } + } + + public func flush() { + if let value = cachedValue { + record!.flush(fieldName: name, value) + } + } + + public mutating func fetch() { + cachedValue = record!.fetch(fieldName: name, type: Value.self) + } +} +``` + +We could define our model based on the `Field` property wrapper: + +```swift +public struct Person: DatabaseModel { + @Field(name: "first_name") public var firstName: String + @Field(name: "last_name") public var lastName: String + @Field(name: "date_of_birth") public var birthdate: Date +} +``` + +`Field` itself has API that is important to users of `Person`: it lets us flush existing values, fetch new values, and retrieve the name of the corresponding field in the database. However, the underscored variables for each of the properties of our model (`_firstName`, `_lastName`, and `_birthdate`) are `private`, so our clients cannot manipulate them directly. + +To vend API, the property wrapper type `Field` can provide a *projection* that allows us to manipulate the relationship of the field to the database. Projection properties are prefixed with a `$`, so the projection of the `firstName` property is called `$firstName` and is visible wherever `firstName` is visible. Property wrapper types opt into providing a projection by defining a `projectedValue` property: + +```swift +@propertyWrapper +public struct Field { + // ... API as before ... + + public var projectedValue: Self { + get { self } + set { self = newValue } + } +} +``` + +When `projectedValue` is present, the projection variable is created as a wrapper around `projectedValue`. For example, the following property: + +```swift +@Field(name: "first_name") public var firstName: String +``` + +expands to: + +```swift +private var _firstName: Field = Field(name: "first_name") + +public var firstName: String { + get { _firstName.wrappedValue } + set { _firstName.wrappedValue = newValue } +} + +public var $firstName: Field { + get { _firstName.projectedValue } + set { _firstName.projectedValue = newValue } +} +``` + +This allows clients to manipulate both the property and its projection, e.g., + +```swift +somePerson.firstName = "Taylor" +$somePerson.flush() +``` + +## Examples + +Before describing the detailed design, here are some more examples of +wrappers. + +### Delayed Initialization + +A property wrapper can model "delayed" initialization, where +the definite initialization (DI) rules for properties are enforced +dynamically rather than at compile time. This can avoid the need for +implicitly-unwrapped optionals in multi-phase initialization. We can +implement both a mutable variant, which allows for reassignment like a +`var`: + +```swift +@propertyWrapper +struct DelayedMutable { + private var _value: Value? = nil + + var wrappedValue: Value { + get { + guard let value = _value else { + fatalError("property accessed before being initialized") + } + return value + } + set { + _value = newValue + } + } + + /// "Reset" the wrapper so it can be initialized again. + mutating func reset() { + _value = nil + } +} +``` + +and an immutable variant, which only allows a single initialization like +a `let`: + +```swift +@propertyWrapper +struct DelayedImmutable { + private var _value: Value? = nil + + var wrappedValue: Value { + get { + guard let value = _value else { + fatalError("property accessed before being initialized") + } + return value + } + + // Perform an initialization, trapping if the + // value is already initialized. + set { + if _value != nil { + fatalError("property initialized twice") + } + _value = newValue + } + } +} +``` + +This enables multi-phase initialization, like this: + +```swift +class Foo { + @DelayedImmutable var x: Int + + init() { + // We don't know "x" yet, and we don't have to set it + } + + func initializeX(x: Int) { + self.x = x // Will crash if 'self.x' is already initialized + } + + func getX() -> Int { + return x // Will crash if 'self.x' wasn't initialized + } +} +``` + +### `NSCopying` + +Many Cocoa classes implement value-like objects that require explicit copying. +Swift currently provides an `@NSCopying` attribute for properties to give +them behavior like Objective-C's `@property(copy)`, invoking the `copy` method +on new objects when the property is set. We can turn this into a wrapper: + +```swift +@propertyWrapper +struct Copying { + private var _value: Value + + init(wrappedValue value: Value) { + // Copy the value on initialization. + self._value = value.copy() as! Value + } + + var wrappedValue: Value { + get { return _value } + set { + // Copy the value on reassignment. + _value = newValue.copy() as! Value + } + } +} +``` + +This implementation would address the problem detailed in +[SE-0153](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0153-compensate-for-the-inconsistency-of-nscopyings-behaviour.md). Leaving the `copy()` out of `init(wrappedValue:)` implements the pre-SE-0153 semantics. + +### Thread-specific storage + +Thread-specific storage (based on pthreads) can be implemented as a property wrapper, too (example courtesy of Daniel Delwood): + +```swift +@propertyWrapper +final class ThreadSpecific { + private var key = pthread_key_t() + private let initialValue: T + + init(key: pthread_key_t, wrappedValue: T) { + self.key = key + self.initialValue = wrappedValue + } + + init(wrappedValue: T) { + self.initialValue = wrappedValue + pthread_key_create(&key) { + // 'Any' erasure due to inability to capture 'self' or + $0.assumingMemoryBound(to: Any.self).deinitialize(count: 1) + $0.deallocate() + } + } + + deinit { + fatalError("\(ThreadSpecific.self).deinit is unsafe and would leak") + } + + private var box: UnsafeMutablePointer { + if let pointer = pthread_getspecific(key) { + return pointer.assumingMemoryBound(to: Any.self) + } else { + let pointer = UnsafeMutablePointer.allocate(capacity: 1) + pthread_setspecific(key, UnsafeRawPointer(pointer)) + pointer.initialize(to: initialValue as Any) + return pointer + } + } + + var wrappedValue: T { + get { return box.pointee as! T } + set (v) { + box.withMemoryRebound(to: T.self, capacity: 1) { $0.pointee = v } + } + } +} +``` + + +### User defaults + +Property wrappers can be used to provide typed properties for +string-keyed data, such as [user defaults](https://developer.apple.com/documentation/foundation/userdefaults) (example courtesy of Harlan Haskins), +encapsulating the mechanism for extracting that data in the wrapper type. +For example: + +```swift +@propertyWrapper +struct UserDefault { + let key: String + let defaultValue: T + + var wrappedValue: T { + get { + return UserDefaults.standard.object(forKey: key) as? T ?? defaultValue + } + set { + UserDefaults.standard.set(newValue, forKey: key) + } + } +} + +enum GlobalSettings { + @UserDefault(key: "FOO_FEATURE_ENABLED", defaultValue: false) + static var isFooFeatureEnabled: Bool + + @UserDefault(key: "BAR_FEATURE_ENABLED", defaultValue: false) + static var isBarFeatureEnabled: Bool +} +``` + +### Copy-on-write + +With some work, property wrappers can provide copy-on-write wrappers (original example courtesy of Becca Royal-Gordon): + +```swift +protocol Copyable: AnyObject { + func copy() -> Self +} + +@propertyWrapper +struct CopyOnWrite { + init(wrappedValue: Value) { + self.wrappedValue = wrappedValue + } + + private(set) var wrappedValue: Value + + var projectedValue: Value { + mutating get { + if !isKnownUniquelyReferenced(&wrappedValue) { + wrappedValue = wrappedValue.copy() + } + return wrappedValue + } + set { + wrappedValue = newValue + } + } +} +``` + +`projectedValue` provides projection for the synthesized storage property, allowing the copy-on-write wrapper to be used directly: + +```swift +@CopyOnWrite var storage: MyStorageBuffer + +// Non-modifying access: +let index = storage.index(of: …) + +// For modification, access $storage, which goes through `projectedValue`: +$storage.append(…) +``` + +### `Ref` / `Box` + +We can define a property wrapper type `Ref` that is an abstracted reference +to some value that can be get/set, which is effectively a programmatic computed property: + +```swift +@propertyWrapper +struct Ref { + let read: () -> Value + let write: (Value) -> Void + + var wrappedValue: Value { + get { return read() } + nonmutating set { write(newValue) } + } + + subscript(dynamicMember keyPath: WritableKeyPath) -> Ref { + return Ref( + read: { self.wrappedValue[keyPath: keyPath] }, + write: { self.wrappedValue[keyPath: keyPath] = $0 }) + } +} +``` + +The subscript is using [SE-0252 "Key Path Member Lookup"](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0252-keypath-dynamic-member-lookup.md) so that a `Ref` instance provides access to the properties of its value. Building on the example from SE-0252: + +```swift +@Ref(read: ..., write: ...) +var rect: Rectangle + +print(rect) // accesses the Rectangle +print(rect.topLeft) // accesses the topLeft component of the rectangle + +let rect2 = _rect // get the Ref +let topLeft2 = _rect.topLeft // get a Ref referring to the Rectangle's topLeft +``` + +The `Ref` type encapsulates read/write, and making it a property wrapper lets +us primarily see the underlying value. Often, one does not want to explicitly +write out the getters and setters, and it's fairly common to have a `Box` type that boxes up a value and can vend `Ref` instances referring into that box. We can do so with another property wrapper: + +```swift +@propertyWrapper +class Box { + var wrappedValue: Value + + init(wrappedValue: Value) { + self.wrappedValue = wrappedValue + } + + var projectedValue: Ref { + return Ref(read: { self.wrappedValue }, write: { self.wrappedValue = $0 }) + } +} +``` + +Now, we can define a new `Box` directly: + +```swift +@Box var rectangle: Rectangle = ... + +print(rectangle) // access the rectangle +print(rectangle.topLeft) // access the top left coordinate of the rectangle +let rect2 = $rectangle // through projectedValue, produces a Ref +let topLeft2 = $rectangle.topLeft // through projectedValue, produces a Ref +``` + +The use of `projectedValue` hides the box from the client (`_rectangle` remains private), providing direct access to the value in the box (the common case) as well as access to the box contents via `Ref` (referenced as `$rectangle`). + +### "Clamping" a value within bounds + +A property wrapper could limit the stored value to be within particular bounds. For example, the `Clamping` property wrapper provides min/max bounds within which values will be clamped: + +```swift +@propertyWrapper +struct Clamping { + var value: V + let min: V + let max: V + + init(wrappedValue: V, min: V, max: V) { + value = wrappedValue + self.min = min + self.max = max + assert(value >= min && value <= max) + } + + var wrappedValue: V { + get { return value } + set { + if newValue < min { + value = min + } else if newValue > max { + value = max + } else { + value = newValue + } + } + } +} +``` + +Most interesting in this example is how `@Clamping` properties can be +initialized given both an initial value and initializer arguments. In such cases, the `wrappedValue:` argument is placed first. For example, this means we can define a `Color` type that clamps all values in the range [0, 255]: + +```swift +struct Color { + @Clamping(min: 0, max: 255) var red: Int = 127 + @Clamping(min: 0, max: 255) var green: Int = 127 + @Clamping(min: 0, max: 255) var blue: Int = 127 + @Clamping(min: 0, max: 255) var alpha: Int = 255 +} +``` + +The synthesized memberwise initializer demonstrates how the initialization itself is formed: + +```swift +init(red: Int = 127, green: Int = 127, blue: Int = 127, alpha: Int = 255) { + _red = Clamping(wrappedValue: red, min: 0, max: 255) + _green = Clamping(wrappedValue: green, min: 0, max: 255) + _blue = Clamping(wrappedValue: blue, min: 0, max: 255) + _alpha = Clamping(wrappedValue: alpha, min: 0, max: 255) +} +``` + +(Example [courtesy of Avi](https://forums.swift.org/t/pitch-3-property-wrappers-formerly-known-as-property-delegates/24961/65)) + +### Property wrapper types in the wild + +There are a number of existing types that already provide the basic structure of a property wrapper type. One fun case is `Unsafe(Mutable)Pointer`, which we could augment to allow easy access to the pointed-to value: + +```swift +@propertyWrapper +struct UnsafeMutablePointer { + var pointee: Pointee { ... } + + var wrappedValue: Pointee { + get { return pointee } + set { pointee = newValue } + } +} +``` + +From a user perspective, this allows us to set up the unsafe mutable pointer's address once, then mostly refer to the pointed-to value: + +``` +@UnsafeMutablePointer(mutating: addressOfAnInt) +var someInt: Int + +someInt = 17 // equivalent to _someInt.pointee = 17 +print(someInt) + +_someInt.deallocate() +``` + +RxCocoa's [`BehaviorRelay`](https://github.com/ReactiveX/RxSwift/blob/master/RxCocoa/Traits/BehaviorRelay.swift) replays the most recent value provided to it for each of the subscribed observers. It is created with an initial value, has `wrappedValue` property to access the current value and a `projectedValue` to expose a projection providing API to `subscribe` a new observer: (Thanks to Adrian Zubarev for pointing this out) + +```swift +@BehaviorRelay +var myValue: Int = 17 + +let observer = $myValue.subscribe(...) // subscribe an observer +$myValue.accept(42) // set a new value via the synthesized storage property + +print(myValue) // print the most recent value +``` + +[Combine's `Published`](https://developer.apple.com/documentation/combine/published) property wrapper is similar in spirit, allowing clients to subscribe to `@Published` properties (via the `$` projection) to receive updates when the value changes. + +[SwiftUI](https://developer.apple.com/xcode/swiftui/) makes extensive use of +property wrappers to declare local state (`@State`) and express data dependencies on other state that can effect the UI (`@EnvironmentObject`, `@Environment`, `@ObjectBinding`). It makes extensive use of projections to the [`Binding`](https://developer.apple.com/documentation/swiftui/binding) property wrapper to allow controlled mutation of the state that affects UI. + +## Composition of property wrappers + +When multiple property wrappers are provided for a given property, +the wrappers are composed together to get both effects. For example, consider the composition of `DelayedMutable` and `Copying`: + +```swift +@DelayedMutable @Copying var path: UIBezierPath +``` + +Here, we have a property for which we can delay initialization until later. When we do set a value, it will be copied via `NSCopying`'s `copy` method. + +Composition is implemented by nesting later wrapper types inside earlier wrapper types, where the innermost nested type is the original property's type. For the example above, the backing storage will be of type `DelayedMutable>`, and the synthesized getter/setter for `path` will look through both levels of `.wrappedValue`: + +```swift +private var _path: DelayedMutable> = .init() +var path: UIBezierPath { + get { return _path.wrappedValue.wrappedValue } + set { _path.wrappedValue.wrappedValue = newValue } +} +``` + +Note that this design means that property wrapper composition is not commutative, because the order of the attributes affects how the nesting is performed: + +```swift +@DelayedMutable @Copying var path1: UIBezierPath // _path1 has type DelayedMutable> +@Copying @DelayedMutable var path2: UIBezierPath // error: _path2 has ill-formed type Copying> +``` + +In this case, the type checker prevents the second ordering, because `DelayedMutable` does not conform to the `NSCopying` protocol. This won't always be the case: some semantically-bad compositions won't necessarily be caught by the type system. Alternatives to this approach to composition are presented in "Alternatives considered." + +## Detailed design + +### Property wrapper types + +A *property wrapper type* is a type that can be used as a property +wrapper. There are two basic requirements for a property wrapper +type: + +1. The property wrapper type must be defined with the attribute +`@propertyWrapper`. The attribute indicates that the type is meant to +be used as a property wrapper type, and provides a point at which the +compiler can verify any other consistency rules. +2. The property wrapper type must have a property named `wrappedValue`, whose +access level is the same as that of the type itself. This is the +property used by the compiler to access the underlying value on the +wrapper instance. + +### Initialization of synthesized storage properties + +Introducing a property wrapper to a property makes that property +computed (with a getter/setter) and introduces a stored property whose +type is the wrapper type. That stored property can be initialized +in one of three ways: + +1. Via a value of the original property's type (e.g., `Int` in `@Lazy var + foo: Int`, using the property wrapper type's + `init(wrappedValue:)` initializer. That initializer must have a single + parameter of the same type as the `wrappedValue` property (or + be an `@autoclosure` thereof) and have the same access level as the + property wrapper type itself. When `init(wrappedValue:)` is present, + is always used for the initial value provided on the property + declaration. For example: + + ```swift + @Lazy var foo = 17 + + // ... implemented as + private var _foo: Lazy = Lazy(wrappedValue: 17) + var foo: Int { /* access via _foo.wrappedValue as described above */ } + ``` + + When there are multiple, composed property wrappers, all of them must provide an `init(wrappedValue:)`, and the resulting initialization will wrap each level of call: + + ```swift + @Lazy @Copying var path = UIBezierPath() + + // ... implemented as + private var _path: Lazy> = .init(wrappedValue: .init(wrappedValue: UIBezierPath())) + var path: UIBezierPath { /* access via _path.wrappedValue.wrappedValue as described above */ } + ``` + +2. Via a value of the property wrapper type, by placing the initializer + arguments after the property wrapper type: + + ```swift + var addressOfInt: UnsafePointer = ... + + @UnsafeMutablePointer(mutating: addressOfInt) + var someInt: Int + + // ... implemented as + private var _someInt: UnsafeMutablePointer = UnsafeMutablePointer(mutating: addressOfInt) + var someInt: Int { /* access via _someInt.wrappedValue */ } + ``` + + When there are multiple, composed property wrappers, only the first (outermost) wrapper may have initializer arguments. + +3. Implicitly, when no initializer is provided and the property wrapper type provides a no-parameter initializer (`init()`). In such cases, the wrapper type's `init()` will be invoked to initialize the stored property. + + ```swift + @DelayedMutable var x: Int + + // ... implemented as + private var _x: DelayedMutable = DelayedMutable() + var x: Int { /* access via _x.wrappedValue */ } + ``` + + When there are multiple, composed property wrappers, only the first (outermost) wrapper needs to have an `init()`. + +### Type inference with property wrappers + +If the first property wrapper type is generic, its generic arguments must either be given explicitly in the attribute or Swift must be able to deduce them from the variable declaration. That deduction proceeds as follows: + +* If the variable has an initial value expression `E`, then the first wrapper type is constrained to equal the type resulting from a call to `A(wrappedValue: E, argsA...)`, where `A` is the written type of the attribute and `argsA` are the arguments provided to that attribute. For example: + + ```swift + @Lazy var foo = 17 + // type inference as in... + private var _foo: Lazy = Lazy(wrappedValue: 17) + // infers the type of '_foo' to be 'Lazy' + ``` + + If there are multiple wrapper attributes, the argument to this call will instead be a nested call to `B(wrappedValue: E, argsB...)` for the written type of the next attribute, and so on recursively. For example: + + ```swift + @A @B(name: "Hello") var bar = 42 + // type inference as in ... + private var _bar = A(wrappedValue: B(wrappedValue: 42, name: "Hello")) + // infers the type of '_bar' to be 'A' + ``` + +* Otherwise, if the first wrapper attribute has direct initialization arguments `E...`, the outermost wrapper type is constrained to equal the type resulting from `A(E...)`, where `A` is the written type of the first attribute. Wrapper attributes after the first may not have direct initializers. For example: + + ```swift + @UnsafeMutablePointer(mutating: addressOfInt) + var someInt + // type inference as in... + private var _someInt: UnsafeMutablePointer = UnsafeMutablePointer.init(mutating: addressOfInt) + // infers the type of `_someInt` to be `UnsafeMutablePointer` + ``` + +* Otherwise, if there is no initialization, and the original property has a type annotation, the type of the `wrappedValue` property in the last wrapper type is constrained to equal the type annotation of the original property. For example: + + ```swift + @propertyWrapper + struct Function { + var wrappedValue: (T) -> U? { ... } + } + + @Function var f: (Int) -> Float? // infers T=Int, U=Float + ``` + +In any case, the first wrapper type is constrained to be a specialization of the first attribute's written type. Furthermore, for any secondary wrapper attributes, the type of the wrappedValue property of the previous wrapper type is constrained to be a specialization of the attribute's written type. Finally, if a type annotation is given, the type of the wrappedValue property of the last wrapper type is constrained to equal the type annotation. If these rules fail to deduce all the type arguments for the first wrapper type, or if they are inconsistent with each other, the variable is ill-formed. For example: + +```swift +@Lazy var foo: Int // okay +@Lazy var bar: Double // error: Lazy.wrappedValue is of type Int, not Double +``` + +The deduction can also provide a type for the original property (if a type annotation was omitted) or deduce generic arguments that have been omitted from the type annotation. For example: + +```swift +@propertyWrapper +struct StringDictionary { + var wrappedValue: [String: String] +} + +@StringDictionary var d1. // infers Dictionary +@StringDictionary var d2: Dictionary // infers +``` + +### Custom attributes + +Property wrappers are a form of custom attribute, where the attribute syntax +is used to refer to entities declared in Swift. Grammatically, the use of property wrappers is described as follows: + +``` +attribute ::= '@' type-identifier expr-paren? +``` + +The *type-identifier* must refer to a property wrapper type, which can include generic arguments. Note that this allows for qualification of the attribute names, e.g., + +```swift +@Swift.Lazy var foo = 1742 +``` + +The *expr-paren*, if present, provides the initialization arguments for the wrapper instance. + +This formulation of custom attributes fits in with a [larger proposal for custom attributes](https://forums.swift.org/t/pitch-introduce-custom-attributes/21335/47), which uses the same custom attribute syntax as the above but allows for other ways in which one can define a type to be used as an attribute. In this scheme, `@propertyWrapper` is just one kind of custom attribute: there will be other kinds of custom attributes that are available only at compile time (e.g., for tools) or runtime (via some reflection capability). + +### Mutability of properties with wrappers + +Generally, a property that has a property wrapper will have both a getter and a setter. However, the setter may be missing if the `wrappedValue` property of the property wrapper type lacks a setter, or its setter is inaccessible. + +The synthesized getter will be `mutating` if the property wrapper type's `wrappedValue` property is `mutating` and the property is part of a `struct`. Similarly, the synthesized setter will be `nonmutating` if either the property wrapper type's `wrappedValue` property has a `nonmutating` setter or the property wrapper type is a `class`. For example: + +```swift +@propertyWrapper +struct MutatingGetterWrapper { + var wrappedValue: Value { + mutating get { ... } + set { ... } + } +} + +@propertyWrapper +struct NonmutatingSetterWrapper { + var wrappedValue: Value { + get { ... } + nonmutating set { ... } + } +} + +@propertyWrapper +class ReferenceWrapper { + var wrappedValue: Value +} + +struct Usewrappers { + // x's getter is mutating + // x's setter is mutating + @MutatingGetterWrapper var x: Int + + // y's getter is nonmutating + // y's setter is nonmutating + @NonmutatingSetterWrapper var y: Int + + // z's getter is nonmutating + // z's setter is nonmutating + @ReferenceWrapper var z: Int +} +``` + +### Out-of-line initialization of properties with wrappers + +A property that has a wrapper can be initialized after it is defined, +either via the property itself (if the wrapper type has an +`init(wrappedValue:)`) or via the synthesized storage property. For +example: + + +```swift +@Lazy var x: Int +// ... +x = 17 // okay, treated as _x = .init(wrappedValue: 17) +``` + +The synthesized storage property can also be initialized directly, +e.g., + +```swift +@UnsafeMutable var y: Int +// ... +_y = UnsafeMutable(pointer: addressOfInt) // okay +``` + +Note that the rules of [definite +initialization](https://developer.apple.com/swift/blog/?id=28) (DI) +apply to properties that have wrappers. Let's expand the example of +`x` above to include a re-assignment and use `var`: + +```swift +@Lazy var x2: Int +// ... +x2 = 17 // okay, treated as _x2 = .init(wrappedValue: 17) +// ... +x2 = 42 // okay, treated as x2 = 42 (calls the Lazy.wrappedValue setter) +``` + +### Memberwise initializers + +Structs implicitly declare memberwise initializers based on the stored +properties of the struct. With a property that has a wrapper, the +property is technically computed because it's the synthesized property +(of the wrapper's type) that is stored. Instance properties that have a +property wrapper will have a corresponding parameter in the memberwise +initializer, whose type will either be the original property type or +the wrapper type, depending on the wrapper type and the initial value +(if provided). Specifically, the memberwise initializer parameter for +an instance property with a property wrapper will have the original +property type if either of the following is true: + +* The corresponding property has an initial value specified with the +`=` syntax, e.g., `@Lazy var i = 17`, or +- The corresponding property has no initial value, but the property +wrapper type has an `init(wrappedValue:)`. + +Otherwise, the memberwise initializer parameter will have the same +type as the wrapper. For example: + +```swift +struct Foo { + @UserDefault(key: "FOO_FEATURE_ENABLED", defaultValue: false) + var x: Bool + @Lazy var y: Int = 17 + @Lazy(closure: { getBool() }) var z: Bool + @CopyOnWrite var w: Image + + // implicit memberwise initializer: + init(x: UserDefault = UserDefault(key: "FOO_FEATURE_ENABLED", defaultValue: false), + y: Int = 17, + z: Lazy = Lazy(closure: { getBool() }), + w: Image) { + self._x = x + self._y = Lazy(wrappedValue: y) + self._z = z + self._w = CopyOnWrite(wrappedValue: w) + } +} +``` + +### Codable, Hashable, and Equatable synthesis + +Synthesis for `Encodable`, `Decodable`, `Hashable`, and `Equatable` +use the backing storage property. This allows property wrapper types to determine their own serialization and equality behavior. For `Encodable` and `Decodable`, the name used for keyed archiving is that of the original property declaration (without the `_`). + +### $ identifiers + +Currently, identifiers starting with a `$` are not permitted in Swift programs. Today, such identifiers are only used in LLDB, where they can be used to name persistent values within a debugging session. + +This proposal loosens these rules slightly: the Swift compiler will introduce identifiers that start with `$` (for the projection property), and Swift code can reference those properties. However, Swift code cannot declare any new entities with an identifier that begins with `$`. For example: + +```swift +@CopyOnWrite var x = UIBezierPath() +print($x) // okay to refer to compiler-defined $x +let $y = UIBezierPath() // error: cannot declare entity with $-prefixed name '$y' +``` + +### Projections + +A property wrapper type can choose to provide a projection property (e.g., `$foo`) to expose more API for each wrapped property by defining a `projectedValue` property. +As with the `wrappedValue` property and `init(wrappedValue:)`, the `projectedValue` property must have the +same access level as its property wrapper type. For example: + +```swift +class StorageManager { + func allocate(_: T.Type) -> UnsafeMutablePointer { ... } +} + +@propertyWrapper +struct LongTermStorage { + let pointer: UnsafeMutablePointer + + init(manager: StorageManager, initialValue: Value) { + pointer = manager.allocate(Value.self) + pointer.initialize(to: initialValue) + } + + var wrappedValue: Value { + get { return pointer.pointee } + set { pointer.pointee = newValue } + } + + var projectedValue: UnsafeMutablePointer { + return pointer + } +} +``` + +When we use the `LongTermStorage` wrapper, it handles the coordination with the `StorageManager` and provides either direct access or an `UnsafeMutablePointer` with which to manipulate the value: + +```swift +let manager = StorageManager(...) + +@LongTermStorage(manager: manager, initialValue: "Hello") +var someValue: String + +print(someValue) // prints "Hello" +someValue = "World" // update the value in storage to "World" + +// $someValue accesses the projectedValue property of the wrapper instance, which +// is an UnsafeMutablePointer +let world = $someValue.move() // take value directly from the storage +$someValue.initialize(to: "New value") +``` + +The projection property has the same access level as the original property: +```swift +@LongTermStorage(manager: manager, initialValue: "Hello") +public var someValue: String +``` + +is translated into: + +```swift +private var _someValue: LongTermStorage = LongTermStorage(manager: manager, initialValue: "Hello") + +public var $someValue: UnsafeMutablePointer { + get { return _someValue.projectedValue } +} + +public var someValue: String { + get { return _someValue.wrappedValue } + set { _someValue.wrappedValue = newValue } +} +``` + +Note that, in this example, `$someValue` is not writable, because `projectedValue` is a get-only property. + +When multiple property wrappers are applied to a given property, only the outermost property wrapper's `projectedValue` will be considered. + +### Restrictions on the use of property wrappers + +There are a number of restrictions on the use of property wrappers when defining a property: + +* A property with a wrapper may not be declared inside a protocol. +* An instance property with a wrapper may not be declared inside an extension. +* An instance property may not be declared in an `enum`. +* A property with a wrapper that is declared within a class cannot override another property. +* A property with a wrapper cannot be `lazy`, `@NSCopying`, `@NSManaged`, `weak`, or `unowned`. +* A property with a wrapper must be the only property declared within its enclosing declaration (e.g., `@Lazy var (x, y) = /* ... */` is ill-formed). +* A property with a wrapper shall not define a getter or setter. +* The `wrappedValue` property and (if present) `init(wrappedValue:)` of a property wrapper type shall have the same access as the property wrapper type. +* The `projectedValue` property, if present, shall have the same access as the property wrapper type. +* The `init()` initializer, if present, shall have the same access as the property wrapper type. + +## Impact on existing code + +By itself, this is an additive feature that doesn't impact existing +code. However, with some of the property wrappers suggested, it can +potentially obsolete existing, hardcoded language +features. `@NSCopying` could be completely replaced by a `Copying` +property wrapper type introduced in the `Foundation` module. `lazy` +cannot be completely replaced because it's initial value can refer to +the `self` of the enclosing type; see 'deferred evaluation of +initialization expressions_. However, it may still make sense to +introduce a `Lazy` property wrapper type to cover many of the common +use cases, leaving the more-magical `lazy` as a backward-compatibility +feature. + +## Backward compatibility + +The property wrappers language feature as proposed has no impact on the ABI or runtime. Binaries that use property wrappers can be backward-deployed to the Swift 5.0 runtime. + +## Alternatives considered + +### Composition + +Composition was left out of the [first revision](https://github.com/swiftlang/swift-evolution/commit/8c3499ec5bc22713b150e2234516af3cb8b16a0b) of this proposal, because one can manually compose property wrapper types. For example, the composition `@A @B` could be implemented as an `AB` wrapper: + +```swift +@propertyWrapper +struct AB { + private var storage: A> + + var wrappedValue: Value { + get { storage.wrappedValue.wrappedValue } + set { storage.wrappedValue.wrappedValue = newValue } + } +} +``` + +The main benefit of this approach is its predictability: the author of `AB` decides how to best achieve the composition of `A` and `B`, names it appropriately, and provides the right API and documentation of its semantics. On the other hand, having to manually write out each of the compositions is a lot of boilerplate, particularly for a feature whose main selling point is the elimination of boilerplate. It is also unfortunate to have to invent names for each composition---when I try to compose `A` and `B` via `@A @B`, how do I know to go look for the manually-composed property wrapper type `AB`? Or maybe that should be `BA`? + +### Composition via nested type lookup +One proposed approach to composition addresses only the last issue above directly, treating the attribute-composition syntax `@A @B` as a lookup of the nested type `B` inside `A` to find the wrapper type: + +```swift +@propertyWrapper +struct A { + var wrappedValue: Value { ... } +} + +extension A { + typealias B = AB +} +``` + +This allows the natural composition syntax `@A @B` to work, redirecting to manually-written property wrappers that implement the proper semantics and API. Additionally, this scheme allows one to control which compositions are valid: if there is no nested type `B` in `A`, the composition is invalid. If both `A.B` and `B.A` exist, we have a choice: either enforce commutative semantics as part of the language (`B.A` and `A.B` must refer to the same type or the composition `@A @B` is ill-formed), or allow them to differ (effectively matching the semantics of this proposal). + +This approach addresses the syntax for composition while maintaining control over the precise semantics of composition via manually-written wrapper types. However, it does not address the boilerplate problem. + +### Composition without nesting + +There has been a desire to effect composition of property wrappers without having to wrap one property wrapper type in the other. For example, to have `@A @B` apply the policies of both `A` and `B` without producing a nested type like `A>`. This would potentially make composition more commutative, at least from the type system perspective. However, this approach does not fit with the "wrapper" approach taken by property wrappers. In a declaration + +```swift +@A @B var x: Int +``` + +the `Int` value is conceptually wrapped by a property wrapper type, and the property wrapper type's `wrappedValue` property guards access to that (conceptual) `Int` value. That `Int` value cannot be wrapped both by instances of both `A` and `B` without either duplicating data (both `A` and `B` have a copy of the `Int`) or nesting one of the wrappers inside the other. With the copying approach, one must maintain consistency between the copies (which is particularly hard when value types are involved) and there will still be non-commutative compositions. Nesting fits better with the "wrapper" model of property wrappers. + +### Using a formal protocol instead of `@propertyWrapper` + +Instead of a new attribute, we could introduce a `PropertyWrapper` +protocol to describe the semantic constraints on property wrapper +types. It might look like this: + +```swift +protocol PropertyWrapper { + associatedtype Value + var wrappedValue: Value { get } +} +``` + +There are a few issues here. First, a single protocol +`PropertyWrapper` cannot handle all of the variants of `wrappedValue` that +are implied by the section on mutability of properties with wrappers, +because we'd need to cope with `mutating get` as well as `set` and +`nonmutating set`. Moreover, protocols don't support optional +requirements, like `init(wrappedValue:)` (which also has two +forms: one accepting a `Value` and one accepting an `@autoclosure () +-> Value`) and `init()`. To cover all of these cases, we would need +several related-but-subtly-different protocols. + +The second issue is that, even if there were a single `PropertyWrapper` +protocol, we don't know of any useful generic algorithms or data +structures that seem to be implemented in terms of only +`PropertyWrapper`. + + +### Kotlin-like `by` syntax + +A previous iteration of this proposal (and its [implementation](https://github.com/apple/swift/pull/23440)) used `by` syntax similar to that of [Kotlin's delegated +properties](https://kotlinlang.org/docs/reference/delegated-properties.html), where the `by` followed the variable declaration. For example: + +```swift +var foo by Lazy = 1738 + +static var isFooFeatureEnabled: Bool by UserDefault(key: "FOO_FEATURE_ENABLED", defaultValue: false) +``` + +There are some small advantages to this syntax over the attribute formulation: + +* For cases like `UserDefault` where the wrapper instance is initialized directly, the initialization happens after the original variable declaration, which reads better because the variable type and name come first, and how it's implemented come later. (Counter point: Swift developers are already accustomed to reading past long attributes, which are typically placed on the previous line) +* The `by wrapperType` formulation leaves syntactic space for add-on features like specifying the access level of the wrapper instance (`by private wrapperType`) or delegating to an existing property (`by someInstanceProperty`). + +The main problem with `by` is its novelty: there isn't anything else in Swift quite like the `by` keyword above, and it is unlikely that the syntax would be re-used for any other feature. As a keyword, `by` is quite meaningless, and brainstorming during the [initial pitch](https://forums.swift.org/t/pitch-property-delegates/21895) didn't find any clearly good names for this functionality. + +### Alternative spellings for the `$` projection property + +The prefix `$` spelling for the projection property has been the source of +much debate. A number of alternatives have been proposed, including longer `#`-based spellings (e.g., `#storage(of: foo)`) and postfix `$` (e.g., `foo$`). The postfix `$` had the most discussion, based on the idea that it opens up more extension points in the future (e.g., `foo$storage` could refer to the backing storage, `foo$databaseHandle` could refer to a specific "database handle" projection for certain property wrappers, etc.). However, doing so introduces yet another new namespace of names to the language ("things that follow `$`) and isn't motivated by enough strong use cases. + +### The 2015-2016 property behaviors design + +Property wrappers address a similar set of use cases to *property behaviors*, which were [proposed and +reviewed](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0030-property-behavior-decls.md) +in late 2015/early 2016. The design did not converge, and the proposal +was deferred. This proposal picks up the thread, using much of the +same motivation and some design ideas, but attempting to simplify the +feature and narrow the feature set. Some substantive differences from +the prior proposal are: + +* Behaviors were introduced into a property with the `[behavior]` + syntax, rather than the attribute syntax described here. See the + property behaviors proposal for more information. +* Wrappers are always expressed by a (generic) type. Property behaviors + had a new kind of declaration (introduced by the + `behavior` keyword). Having a new kind of declaration allowed for + the introduction of specialized syntax, but it also greatly + increased the surface area (and implementation cost) of the + proposal. Using a generic type makes property wrappers more of a + syntactic-sugar feature that is easier to implement and explain. +* Wrappers cannot declare new kinds of accessors (e.g., the + `didChange` example from the property behaviors proposal). +* Wrappers used for properties declared within a type cannot refer to + the `self` of their enclosing type. This eliminates some use cases + (e.g., implementing a `Synchronized` property wrapper type that + uses a lock defined on the enclosing type), but simplifies the + design. +* Wrappers can be initialized out-of-line, and one + can use the `$`-prefixed name to refer to the storage property. + These were future directions in the property behaviors proposal. + + +## Future Directions + +### Finer-grained access control + +By default, the synthesized storage property will have `private` access, and the projection property (when available) will have the same access as the original wrapped property. However, there are various circumstances where it would be beneficial to expose the synthesized storage property. This could be performed "per-property", e.g., by introducing a syntax akin to `private(set)`: + +```swift +// both foo and _foo are publicly visible, $foo remains private +@SomeWrapper +public public(storage) private(projection) var foo: Int = 1738 +``` + +One could also consider having the property wrapper types themselves declare that the synthesized storage properties for properties using those wrappers should have the same access as the original property. For example: + +```swift +@propertyWrapper(storageIsAccessible: true) +struct SomeWrapper { + var wrappedValue: T { ... } +} + +// both bar and _bar are publicly visible +@SomeWrapper +public var bar: Int = 1738 +``` + +The two features could also be combined, allowing property wrapper types to provide the default behavior and the `access-level(...)` syntax to change the default. The current proposal's rules are meant to provide the right defaults while allowing for a separate exploration into expanding the visibility of the synthesized properties. + +### Referencing the enclosing 'self' in a wrapper type + +Manually-written getters and setters for properties declared in a type often refer to the `self` of their enclosing type. For example, this can be used to notify clients of a change to a property's value: + +```swift +public class MyClass: Superclass { + private var backingMyVar: Int + public var myVar: Int { + get { return backingMyVar } + set { + if newValue != backingMyVar { + self.broadcastValueWillChange(newValue: newValue) + } + backingMyVar = newValue + } + } +} +``` + +This "broadcast a notification that the value has changed" implementation cannot be cleanly factored into a property wrapper type, because it needs access to both the underlying storage value (here, `backingMyVar`) and the `self` of the enclosing type. We could require a separate call to register the `self` instance with the wrapper type, e.g., + +```swift +protocol Observed { + func broadcastValueWillChange(newValue: T) +} + +@propertyWrapper +public struct Observable { + public var stored: Value + var observed: Observed? + + public init(wrappedValue: Value) { + self.stored = wrappedValue + } + + public func register(_ observed: Observed) { + self.observed = observed + } + + public var wrappedValue: Value { + get { return stored } + set { + if newValue != stored { + observed?.broadcastValueWillChange(newValue: newValue) + } + stored = newValue + } + } +} +``` + +However, this means that one would have to manually call `register(_:)` in the initializer for `MyClass`: + +```swift +public class MyClass: Superclass { + @Observable public var myVar: Int = 17 + + init() { + // self._myVar gets initialized with Observable(wrappedValue: 17) here + super.init() + self._myVar.register(self) // register as an Observable + } +} +``` + +This isn't as automatic as we would like, and it requires us to have a separate reference to the `self` that is stored within `Observable`. Moreover, it is hiding a semantic problem: the observer code that runs in the `broadcastValueWillChange(newValue:)` must not access the synthesized storage property in any way (e.g., to read the old value through `myVal` or subscribe/unsubscribe an observer via `_myVal`), because doing so will trigger a [memory exclusivity](https://swift.org/blog/swift-5-exclusivity/) violation (because we are calling `broadcastValueWillChange(newValue:)` from within the a setter for the same synthesized storage property). + +To address these issues, we could extend the ad hoc protocol used to access the storage property of a `@propertyWrapper` type a bit further. Instead of a `wrappedValue` property, a property wrapper type could provide a static `subscript(instanceSelf:wrapped:storage:)`that receives `self` as a parameter, along with key paths referencing the original wrapped property and the backing storage property. For example: + + +```swift +@propertyWrapper +public struct Observable { + private var stored: Value + + public init(wrappedValue: Value) { + self.stored = wrappedValue + } + + public static subscript( + instanceSelf observed: OuterSelf, + wrapped wrappedKeyPath: ReferenceWritableKeyPath, + storage storageKeyPath: ReferenceWritableKeyPath + ) -> Value { + get { + observed[keyPath: storageKeyPath].stored + } + set { + let oldValue = observed[keyPath: storageKeyPath].stored + if newValue != oldValue { + observed.broadcastValueWillChange(newValue: newValue) + } + observed[keyPath: storageKeyPath].stored = newValue + } + } +} +``` + +The (generic) subscript gets access to the enclosing `self` type via its subscript parameter, eliminating the need for the separate `register(_:)` step and the (type-erased) storage of the outer `self`. The desugaring within `MyClass` would be as follows: + +```swift +public class MyClass: Superclass { + @Observable public var myVar: Int = 17 + + // desugars to... + private var _myVar: Observable = Observable(wrappedValue: 17) + public var myVar: Int { + get { Observable[instanceSelf: self, wrapped: \MyClass.myVar, storage: \MyClass._myVar] } + set { Observable[instanceSelf: self, wrapped: \MyClass.myVar, storage: \MyClass._myVar] = newValue } + } +} +``` + +The design uses a `static` subscript and provides key paths to both the original property declaration (`wrapped`) and the synthesized storage property (`storage`). A call to the static subscript's getter or setter does not itself constitute an access to the synthesized storage property, allowing us to address the memory exclusivity violation from the early implementation. The subscript's implementation is given the means to access the synthesized storage property (via the enclosing `self` instance and `storage` key path). In our `Observable` property wrapper, the static subscript setter performs two distinct accesses to the synthesized storage property via `observed[keyPath: storageKeyPath]`: + +1. The read of the old value +2. A write of the new value + +In between these operations is the broadcast operation to any observers. Those observers are permitted to read the old value, unsubscribe themselves from observation, etc., because at the time of the `broadcastValueWillChange(newValue:)` call there is no existing access to the synthesized storage property. + +There is a secondary benefit to providing the key paths, because it allows the property wrapper type to reason about its different instances based on the identity of the `wrapped` key path. + +This extension is backward-compatible with the rest of the proposal. Property wrapper types could opt in to this behavior by providing a `static subscript(instanceSelf:wrapped:storage:)`, which would be used in cases where the property wrapper is being applied to an instance property of a class. If such a property wrapper type is applied to a property that is not an instance property of a class, or for any property wrapper types that don't have such a static subscript, the existing `wrappedValue` could be used. One could even allow `wrappedValue` to be specified to be unavailable within property wrapper types that have the static subscript, ensuring that such property wrapper types could only be applied to instance properties of a class: + +```swift +@availability(*, unavailable) +var wrappedValue: Value { + get { fatalError("only works on instance properties of classes") } + set { fatalError("only works on instance properties of classes") } +} +``` + +The same model could be extended to static properties of types (passing the metatype instance for the enclosing `self`) as well as global and local properties (no enclosing `self`), although we would also need to extend key path support to static, global, and local properties to do so. + +### Delegating to an existing property + +When specifying a wrapper for a property, the synthesized storage property is implicitly created. However, it is possible that there already exists a property that can provide the storage. One could provide a form of property delegation that creates the getter/setter to forward to an existing property, e.g.: + +```swift +lazy var fooBacking: SomeWrapper +@wrapper(to: fooBacking) var foo: Int +``` + +One could express this either by naming the property directly (as above) or, for an even more general solution, by providing a keypath such as `\.someProperty.someOtherProperty`. + +## Revisions + +### Changes from the accepted proposal + +This proposal originally presented an example of implementing atomic operations using a property wrapper interface. This example was misleading because it would require additional compiler and library features to work correctly. + +Programmers looking for atomic operations can use the [Swift Atomics](https://github.com/apple/swift-atomics) package. + +For those who have already attempted to implement something similar, here is the original example, and why it is incorrect: + +```swift +@propertyWrapper +class Atomic { + private var _value: Value + + init(wrappedValue: Value) { + self._value = wrappedValue + } + + var wrappedValue: Value { + get { return load() } + set { store(newValue: newValue) } + } + + func load(order: MemoryOrder = .relaxed) { ... } + func store(newValue: Value, order: MemoryOrder = .relaxed) { ... } + func increment() { ... } + func decrement() { ... } +} +``` + +As written, this property wrapper does not access its wrapped value atomically. `wrappedValue.getter` reads the entire `_value` property nonatomically, *before* calling the atomic `load` operation on the copied value. Similarly, `wrappedValue.setter` writes the entire `_value` property nonatomically *after* calling the atomic `store` operation. So, in fact, there is no atomic access to the shared class property, `_value`. + +Even if the getter and setter could be made atomic, useful atomic operations, like increment, cannot be built from atomic load and store primitives. The property wrapper in fact encourages race conditions by allowing mutating methods, such as `atomicInt += 1`, to be directly invoked on the nonatomic copy of the wrapped value. + +### Changes from the third reviewed version + +* `init(initialValue:)` has been renamed to `init(wrappedValue:)` to match the name of the property. + +### Changes from the second reviewed version + +* The synthesized storage property is always named with a leading `_` and is always `private`. +* The `wrapperValue` property has been renamed to `projectedValue` to make it sufficiently different from `wrappedValue`. This also gives us the "projection" terminology to talk about the `$` property. +* The projected property (e.g., `$foo`) always has the same access as the original wrapped property, rather than being artificially limited to `internal`. This reflects the idea that, for property wrapper types that have a projection, the projection is equal in importance to the wrapped value. + +### Changes from the first reviewed version + +* The name of the feature has been changed from "property delegates" to "property wrappers" to better communicate how they work and avoid the existing uses of the term "delegate" in the Apple developer community +* When a property wrapper type has a no-parameter `init()`, properties that use that wrapper type will be implicitly initialized via `init()`. +* Support for property wrapper composition has been added, using a "nesting" model. +* A property with a wrapper can no longer have an explicit `get` or `set` declared, to match with the behavior of existing, similar features (`lazy`, `@NSCopying`). +* A property with a wrapper does not need to be `final`. +* Reduced the visibility of the synthesized storage property to `private`. +* When a wrapper type provides `wrapperValue`, the (computed) `$` variable is `internal` (at most) and the backing storage variable gets the prefix `$$` (and remains private). +* Removed the restriction banning property wrappers from having names that match the regular expression `_*[a-z].*`. +* `Codable`, `Hashable`, and `Equatable` synthesis are now based on the backing storage properties, which is a simpler model that gives more control to the authors of property wrapper types. +* Improved type inference for property wrapper types and clarified that the type of the `wrappedValue` property is used as part of this inference. See the "Type inference" section. +* Renamed the `value` property to `wrappedValue` to avoid conflicts. +* Initial values and explicitly-specified initializer arguments can both be used together; see the `@Clamping` example. + +## Acknowledgments + +This proposal was greatly improved throughout its [first pitch](https://forums.swift.org/t/pitch-property-delegates/21895) by many people. Harlan Haskins, Becca Royal-Gordon, Adrian Zubarev, Jordan Rose and others provided great examples of uses of property wrappers (several of which are in this proposal). Adrian Zubarev and Kenny Leung helped push on some of the core assumptions and restrictions of the original proposal, helping to make it more general. Vini Vendramini and David Hart helped tie this proposal together with custom attributes, which drastically reduced the syntactic surface area of this proposal. diff --git a/proposals/0259-approximately-equal.md b/proposals/0259-approximately-equal.md new file mode 100644 index 0000000000..c8a1249e4f --- /dev/null +++ b/proposals/0259-approximately-equal.md @@ -0,0 +1,266 @@ +# Approximate Equality for Floating Point + +* Proposal: [SE-0259](0259-approximately-equal.md) +* Author: [Stephen Canon](https://github.com/stephentyrone) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Returned for revision** +* Implementation: [apple/swift#23839](https://github.com/apple/swift/pull/23839) + +## Introduction + +The internet is full advice about what not to do when comparing floating-point values: + +- "Never compare floats for equality." +- "Always use an epsilon." +- "Floating-point values are always inexact." + +Much of this advice is false, and most of the rest is technically correct but misleading. +Almost none of it provides specific and correct recommendations for what you *should* +do if you need to compare floating-point numbers. + +There is no uniformly correct notion of "approximate equality", and there is no uniformly +correct tolerance that can be applied without careful analysis, but we can define +approximate equality functions that are better than what most people will come up with +without assistance from the standard library. + +Pitch thread: [Approximate equality for floating-point](https://forums.swift.org/t/approximate-equality-for-floating-point/22420) + +## Motivation + +Almost all floating-point computations incur rounding error, and many additionally need +to account for model errors (numerical solutions for physical problems are often best +analyzed as solving a *related* equation exactly, rather than approximately solving the +original equation, for example). Because of this, it is frequently desirable to consider two +results to be equivalent if they are equal within an approximate tolerance. + +However, people tend to blindly apply this notion without careful analysis, and without +much understanding of the mechanics of floating-point. This leads to a lot of C-family +code that looks like: +```C +if (fabs(x - y) < DBL_EPSILON) { + // equal enough! +} +``` +This is almost surely incorrect, however. One of the key features of floating-point arithmetic +is *scale invariance*, which means that comparing with an *absolute* tolerance will +essentially always be incorrect. Further, in the most likely scaling (x and y are both of +roughly unit scale), the tolerance `DBL_EPSILON` is *far* too small for any nontrivial +computation. + +Even though these are relatively simple things to account for, people get them wrong more +often than they get them right, so it's a great opportunity for the standard library to help +improve software quality with a small well-considered API. + +## Proposed solution +```Swift +if x.isAlmostEqual(to: y) { + // equal enough! +} +``` +This predicate is reflexive (except for NaN, like all floating-point comparisons) and +symmetric (many implementations of approximately equality you'll find on the internet +are not, which sets people up for hard to find bugs or broken tests). It gracefully handles +subnormal numbers in the only sensible fashion possible, and uses a tolerance that is +acceptable for most computations. For cases that want to manually specify another +tolerance, we have: +```Swift +if x.isAlmostEqual(to: y, tolerance: 0.001) { + // equal enough! +} +``` +Both of the aforementioned functions use a *relative* tolerance. There is a single value +for which that's unsuitable--comparison with zero (because no number will ever +compare equal to zero with a sensible relative tolerance applied). To address this, we +provide one additional function: +```Swift +if x.isAlmostZero( ) { + // zero enough! +} +``` +## Detailed design +```swift +extension FloatingPoint { + /// Test approximate equality with relative tolerance. + /// + /// Do not use this function to check if a number is approximately + /// zero; no reasoned relative tolerance can do what you want for + /// that case. Use `isAlmostZero` instead for that case. + /// + /// The relation defined by this predicate is symmetric and reflexive + /// (except for NaN), but *is not* transitive. Because of this, it is + /// often unsuitable for use for key comparisons, but it can be used + /// successfully in many other contexts. + /// + /// The internet is full advice about what not to do when comparing + /// floating-point values: + /// + /// - "Never compare floats for equality." + /// - "Always use an epsilon." + /// - "Floating-point values are always inexact." + /// + /// Much of this advice is false, and most of the rest is technically + /// correct but misleading. Almost none of it provides specific and + /// correct recommendations for what you *should* do if you need to + /// compare floating-point numbers. + /// + /// There is no uniformly correct notion of "approximate equality", and + /// there is no uniformly correct tolerance that can be applied without + /// careful analysis. This function considers two values to be almost + /// equal if the relative difference between them is smaller than the + /// specified `tolerance`. + /// + /// The default value of `tolerance` is `sqrt(.ulpOfOne)`; this value + /// comes from the common numerical analysis wisdom that if you don't + /// know anything about a computation, you should assume that roughly + /// half the bits may have been lost to rounding. This is generally a + /// pretty safe choice of tolerance--if two values that agree to half + /// their bits but are not meaningfully almost equal, the computation + /// is likely ill-conditioned and should be reformulated. + /// + /// For more complete guidance on an appropriate choice of tolerance, + /// consult with a friendly numerical analyst. + /// + /// - Parameters: + /// - other: the value to compare with `self` + /// - tolerance: the relative tolerance to use for the comparison. + /// Should be in the range (.ulpOfOne, 1). + /// + /// - Returns: `true` if `self` is almost equal to `other`; otherwise + /// `false`. + @inlinable + public func isAlmostEqual( + to other: Self, + tolerance: Self = Self.ulpOfOne.squareRoot() + ) -> Bool { + // tolerances outside of [.ulpOfOne,1) yield well-defined but useless results, + // so this is enforced by an assert rather than a precondition. + assert(tolerance >= .ulpOfOne && tolerance < 1, "tolerance should be in [.ulpOfOne, 1).") + // The simple computation below does not necessarily give sensible + // results if one of self or other is infinite; we need to rescale + // the computation in that case. + guard self.isFinite && other.isFinite else { + return rescaledAlmostEqual(to: other, tolerance: tolerance) + } + // This should eventually be rewritten to use a scaling facility to be + // defined on FloatingPoint suitable for hypot and scaled sums, but the + // following is good enough to be useful for now. + let scale = max(abs(self), abs(other), .leastNormalMagnitude) + return abs(self - other) < scale*tolerance + } + + /// Test if this value is nearly zero with a specified `absoluteTolerance`. + /// + /// This test uses an *absolute*, rather than *relative*, tolerance, + /// because no number should be equal to zero when a relative tolerance + /// is used. + /// + /// Some very rough guidelines for selecting a non-default tolerance for + /// your computation can be provided: + /// + /// - If this value is the result of floating-point additions or + /// subtractions, use a tolerance of `.ulpOfOne * n * scale`, where + /// `n` is the number of terms that were summed and `scale` is the + /// magnitude of the largest term in the sum. + /// + /// - If this value is the result of floating-point multiplications, + /// consider each term of the product: what is the smallest value that + /// should be meaningfully distinguished from zero? Multiply those terms + /// together to get a tolerance. + /// + /// - More generally, use half of the smallest value that should be + /// meaningfully distinct from zero for the purposes of your computation. + /// + /// For more complete guidance on an appropriate choice of tolerance, + /// consult with a friendly numerical analyst. + /// + /// - Parameter absoluteTolerance: values with magnitude smaller than + /// this value will be considered to be zero. Must be greater than + /// zero. + /// + /// - Returns: `true` if `abs(self)` is less than `absoluteTolerance`. + /// `false` otherwise. + @inlinable + public func isAlmostZero( + absoluteTolerance tolerance: Self = Self.ulpOfOne.squareRoot() + ) -> Bool { + assert(tolerance > 0) + return abs(self) < tolerance + } + + /// Rescales self and other to give meaningful results when one of them + /// is infinite. We also handle NaN here so that the fast path doesn't + /// need to worry about it. + @usableFromInline + internal func rescaledAlmostEqual(to other: Self, tolerance: Self) -> Bool { + // NaN is considered to be not approximately equal to anything, not even + // itself. + if self.isNaN || other.isNaN { return false } + if self.isInfinite { + if other.isInfinite { return self == other } + // Self is infinite and other is finite. Replace self with the binade + // of the greatestFiniteMagnitude, and reduce the exponent of other by + // one to compensate. + let scaledSelf = Self(sign: self.sign, + exponent: Self.greatestFiniteMagnitude.exponent, + significand: 1) + let scaledOther = Self(sign: .plus, + exponent: -1, + significand: other) + // Now both values are finite, so re-run the naive comparison. + return scaledSelf.isAlmostEqual(to: scaledOther, tolerance: tolerance) + } + // If self is finite and other is infinite, flip order and use scaling + // defined above, since this relation is symmetric. + return other.rescaledAlmostEqual(to: self, tolerance: tolerance) + } +} +``` + +## Source compatibility + +This is a purely additive change. These operations may collide with existing extensions +that users have defined with the same names, but the user-defined definitions will win, so +there is no ambiguity introduced, and it will not change the behavior of existing programs. +Those programs can opt-in to the stdlib notion of approximate equality by removing their +existing definitions. + +## Effect on ABI stability + +Additive change only. + +## Effect on API resilience + +This introduces two leaf functions that will become part of the public API for the +FloatingPoint protocol, and one `@usableFromInline` helper function. We may make small +changes to these implementations over time for the purposes of performance optimization, +but the APIs are expected to be stable. + +## Alternatives considered + +**Why not change the behavior of `==`?** +Approximate equality is useful to have, but is a terrible default because it isn't transitive, +and does not imply substitutability under any useful model of floating-point values. This +breaks all sorts of implicit invariants that programs depend on. + +**Why not introduce a fancy new operator like `a ≅ b` or `a == b ± t`?** +There's a bunch of clever things that one might do in this direction, but it's not at all +obvious which (if any) of them is actually a good idea. Defining good semantics for +approximate equality via a method in the standard library makes it easy for people +to experiment with clever operators if they want to, and provides very useful functionality +now. + +**Why is a relative tolerance used?** +Because--as a library--we do not know the scaling of user's inputs a priori, a relative +tolerance is the only sensible option. Any absolute tolerance will be wrong more often +than it is right. + +**Why do we need a special comparison for zero?** +Because zero is the one and only value with which a relative comparison tolerance is +essentially never useful. For well-scaled values, `x.isAlmostEqual(to: 0, tolerance: t)` +will be false for any valid `t`. + +**Why not a single function using a hybrid relative/absolute tolerance?** +Because--as a library--we do not know the scaling of user's inputs a priori, this will be +wrong more often than right. The scale invariance (up to underflow) of the main +`isAlmostEqual` function is a highly-desirable property, because it mirrors the way +the rest of the floating-point number system works. diff --git a/proposals/0260-library-evolution.md b/proposals/0260-library-evolution.md new file mode 100644 index 0000000000..e14420908c --- /dev/null +++ b/proposals/0260-library-evolution.md @@ -0,0 +1,361 @@ +# Library Evolution for Stable ABIs + +* Proposal: [SE-0260](0260-library-evolution.md) +* Authors: [Jordan Rose](https://github.com/jrose-apple), [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.1)** +* Implementation: Implemented in Swift 5 for standard library use. PR with renamed attributes [here](https://github.com/apple/swift/pull/24185). +* Pre-review discussion: [Forum thread](https://forums.swift.org/t/pitch-library-evolution-for-stable-abis/) +* Review: ([review](https://forums.swift.org/t/se-0260-library-evolution-for-stable-abis/24260)) ([acceptance](https://forums.swift.org/t/accepted-se-0260-library-evolution-for-stable-abis/24845)) + +## Introduction + +One of Swift's goals is to be a good language for libraries with binary compatibility concerns, such as those shipped as part of Apple's OSs. This includes giving library authors the flexibility to add to their public interface, and to change implementation details, without breaking binary compatibility. At the same time, it's important that library authors be able to opt out of this flexibility in favor of performance. + +This proposal introduces: + +- a "library evolution" build mode for libraries that are declaring ABI stability, which preserves the ability to make certain changes to types without breaking the library's ABI; and +- an attribute for such libraries to opt out of this flexibility on a per-type basis, allowing certain compile-time optimizations. + +The mechanisms for this are already in place, and were used to stabilize the ABI of the standard library. This proposal makes them features for use by any 3rd-party library that wishes to declare itself ABI stable. + +**This build mode will have no impact on libraries built and distributed with an app**. Such libraries will receive the same optimization that they have in previous versions of Swift. + +_Note: this proposal will use the word "field" to mean "stored instance property"._ + +## Motivation + +As of Swift 5, libraries are able to declare a stable ABI, allowing a library binary to be replaced with a newer version without requiring client programs to be recompiled. + +What constitutes the ABI of a library differs from language to language. In C and C++, the public ABI for a library includes information that ideally would be kept purely as an implementation detail. For example, the _size_ of a struct is fixed as part of the ABI, and is known to the library user at compile time. This prevents adding new fields to that type in later releases once the ABI is declared stable. If direct access to fields is allowed, the _layout_ of the struct can also become part of the ABI, so fields cannot then be reordered. + +This often leads to manual workarounds. A common technique is to have the struct hold only a pointer to an "impl" type, which holds the actual stored properties. All access to these properties is made via function calls, which can be updated to handle changes to the layout of the "impl" type. This has some obvious downsides: + +- it introduces indirection and function call overhead when accessing fields; +- it means the fields of the struct must be stored on the heap even when they could otherwise be stored on the stack, with that heap allocation needing to be managed via reference counting or some other mechanism; +- it means that the fields of the struct data are not stored in contiguous memory when placed in an array; and +- the implementation of all this must be provided by the library author. + +A similar challenge occurs with Swift enums. As discussed in [SE-0192][], introducing new cases to an enum can break source compatibility. There are also ABI consequences to this: adding new enum cases sometimes means increasing the storage needed for the enum, which can affect the size and layout of an enum in the same way adding fields to a struct can. + +The goal of this proposal is to reduce the burden on library developers by building into the compiler an automatic mechanism to reserve the flexibility to alter the internal representation of structs and enums, without manual workarounds. This mechanism can be implemented in such a way that optimizations such as stack allocation or contiguous inline storage of structs can still happen, while leaving the size of the type to be determined at runtime. + + [SE-0192]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0192-non-exhaustive-enums.md + +## Proposed solution + +The compiler will gain a new mode, called "library evolution mode". This mode will be off by default. A library compiled with this mode enabled is said to be _ABI-stable_. + +When a library is compiled with library evolution mode enabled, it is not an ABI-breaking change to modify the fields in a struct (to add, remove, or reorder them), and to add new enum cases (including with associated values). This implies that clients must manipulate fields and enum cases indirectly, via non-inlinable function calls. Information such as the size of the type, and the layout of its fields, becomes something that can only be determined at runtime. Types that reserve this flexibility are referred to as "resilient types". + +A new `@frozen` attribute will be introduced to allow library authors to opt out of this flexibility on a per-type basis. This attribute promises that stored instance properties within the struct will not be *added,* *removed,* or *reordered*, and that an enum will never *add,* *remove,* or *reorder* its cases (note removing, and sometimes reordering, cases can already be source breaking, not just ABI breaking). The compiler will use this for optimization purposes when compiling clients of the type. The precise set of allowed changes is defined below. + +This attribute has does not affect a compiled binary when library evolution mode is off. In that case, the compiler acts as if *every* struct or enum in the library being compiled is "frozen". However, restrictions such as requiring stored properties on frozen structs to be ABI-public will still be enforced to avoid confusion when moving between modes. + +Binaries compiled without this mode should not be declared as ABI-stable. While they should be stable even without library evolution mode turned on, doing this will not be a supported configuration. + +## Detailed design + +### Library evolution mode + +A new command-line argument, `-enable-library-evolution`, will enable this new mode. + +Turning on library evolution mode will have the following effects on source code: + +- The default behavior for enums will change to be non-frozen. This is the only change visible to _users_ of the library, who will need to use the `@unknown default:` technique described in SE-0192. +- To ensure that all fields are initialized, inlinable `init`s of the type must delegate to a non-inlinable `init`. + +### `@frozen` on `struct` types + +When a library author is certain that there will never be a need to add fields to a struct in future, they may mark that type as `@frozen`. + +This will allow the compiler to optimize away at compile time some calls that would otherwise need to be made at runtime (for example, it might access fields directly without the indirection). + +When compiling with binary stability mode on, a struct can be marked `@frozen` as long as it meets all of the following conditions: + +- The struct is *ABI-public* (see [SE-0193][]), i.e. `public` or marked `@usableFromInline`. +- Every class, enum, struct, protocol, or typealias mentioned in the types of the struct's fields is ABI-public. +- No fields have observing accessors (`willSet` or `didSet`). +- If a field has an initial value, the expression computing the initial value does not reference any types or functions that are not ABI-public. + +```swift +@frozen +public struct Point { + public var x: Double + public var y: Double + + public init(_ x: Double, _ y: Double) { + self.x = x + self.y = y + } +} +``` + +This affects what changes to the struct's fields affect the ABI of the containing library: + +| Change | Normal struct | `@frozen` struct +|---|:---:|:---: +| Adding fields | Allowed | **Affects ABI** +| Reordering fields | Allowed | **Affects ABI** +| Removing ABI-public fields | Affects ABI | Affects ABI +| Removing non-ABI-public fields | Allowed | **Affects ABI** +| Changing the type of an ABI-public field | Affects ABI | Affects ABI +| Changing the type of a non-ABI-public field | Allowed | **Affects ABI** +| Changing a stored instance property to computed | Allowed | **Affects ABI** +| Changing a computed instance property to stored | Allowed | **Affects ABI** +| Changing the access of a non-ABI-public field | Allowed | Allowed +| Marking an `internal` field as `@usableFromInline` | Allowed | Allowed +| Changing an `internal` ABI-public field to be `public` | Allowed | Allowed + +> Note: This proposal is implemented already and in use by the standard library, albeit under different names. The command-line flag is `-enable-library-evolution`; the attribute is `@_fixed_layout` for structs, and `@_frozen` for enums. + + [SE-0193]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0193-cross-module-inlining-and-specialization.md + +#### Guarantees + +Marking a struct `@frozen` only guarantees that its stored instance properties won't change. This allows the compiler to perform certain optimizations, like ignoring properties that are never accessed, or eliminating redundant loads from the same instance property. However, it does not provide a handful of other guarantees that a C struct might: + +- **It is not guaranteed to be "trivial"** ([in the C++ sense][trivial]). A frozen struct containing a class reference or closure still requires reference-counting when copied and when it goes out of scope. + +- **It does not necessarily have a known size or alignment.** A generic frozen struct's layout might depend on the generic argument provided at run time. + +- **Even concrete instantiations may not have a known size or alignment.** A frozen struct with a field that's a *non-*frozen, has a size that may not be known until run time. + +- **It is not guaranteed to use the same layout as a C struct with a similar "shape".** If such a struct is necessary, it should be defined in a C header and imported into Swift. + +- **The fields are not guaranteed to be laid out in declaration order.** The compiler may choose to reorder fields, for example to minimize padding while satisfying alignment requirements. + +That said, the compiler is allowed to use its knowledge of the struct's contents and layout to derive any of these properties. For instance, the compiler can statically prove that copying `Point` is "trivial", because each of its members has a statically-known type that is "trivial". However, depending on this at the language level is not supported, with two exceptions: + +1. The run-time memory layout of a struct with a single field is always identical to the layout of the instance property on its own, whether the struct is declared `@frozen` or not. This has been true since Swift 1. (This does not extend to the calling convention, however. If the struct is not frozen, it will be passed indirectly even if its single field is frozen and thus _can_ be passed directly) + +2. The representation of `nil` for any type that is "nullable" in C / Objective-C is the same as the representation of `nil` or `NULL` in those languages. This includes class references, class-bound protocol references, class type references, unsafe-pointers, `@convention(c)` functions, `@convention(block)` functions, OpaquePointer, Selector, and NSZone. This has been true since Swift 3 ([SE-0055][]). + +This proposal does not change either of these guarantees. + + [trivial]: https://docs.microsoft.com/en-us/cpp/cpp/trivial-standard-layout-and-pod-types + [SE-0055]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0055-optional-unsafe-pointers.md + +### `@frozen` on `enum` types + +Marking an enum as `@frozen` will similarly allow the compiler to optimize away runtime calls. + +In addition, marking an enum as frozen restores the ability of a library user to exhaustively switch over that enum without an `@unknown default:`, because it guarantees no further cases will be added. + +Once frozen, all changes made to an enum's cases affect its ABI. + +## Naming + +`@frozen` was used here, as originally used in [SE-0192][], for both fixed-layout structs and enums. + +- While they share a fixing of their layout going forward, declaring an enum as frozen has additional meaning to _users_ of the library, whereas a frozen struct is an implementation detail of the library. + +- It would be reasonable to use `@frozen` to describe an enum without associated values that promises not to *add* any cases with associated values. This would allow for similar optimization as `@frozen` on a struct, while still not declaring that the enum is frozen. + +- Since this feature is only relevant for libraries with binary compatibility concerns, it may be more useful to tie its syntax to `@available` from the start, as with [SE-0193][]. + +But that said, `@frozen` still has the right *connotations* to describe a type whose stored instance properties or cases can no longer be changed. So the final candidates for names are: + +- `@frozen` for both, to match the term used in [SE-0192][] +- `@frozen` for enums but `@fixedContents`, or something else, specific to structs +- `@available(*, frozen)`, to leave space for e.g. "frozen as of dishwasherOS 5" + +Other names are discussed in "Alternatives considered" + +## Comparison with other languages + +### C + +C structs have a very simple layout algorithm, which guarantees that fields are laid out in source order with adjustments to satisfy alignment restrictions. C has no formal notion of non-public fields; if used in an ABI-stable interface, the total size and alignment of a struct and its fields must not change. On the other hand, because the layout algorithm is so well-known and all fields are "trivial", C struct authors can often get away with adding one or more "reserved" fields to the end of their struct, and then renaming and possibly splitting those fields later to get the effect of adding new fields. A fixed-contents struct in Swift can get the same effect with a private field that backs public computed properties. + +When a C library author wants to reserve the right to *arbitrarily* change the layout of a struct, they will usually not provide a definition for the struct at all. Instead, they will forward-declare the type and vend pointers to instances of the struct, usually allocated on the heap. Clients have to use "getter" and "setter" functions to access the fields of the struct. Non-fixed-contents Swift structs work in much the same way, except that the compiler can be a little more efficient about it. + +Finally, C has an interesting notion of a *partially* fixed-contents struct. If a developer knows that two C structs start with the same fields and have the same alignment, they can reinterpret a pointer to one struct as a pointer to the other, as long as they only access those shared fields. This can be used to get fast access to some fields while going through indirect "getter" functions for others. Swift does not have any equivalent feature at this time. + + +### C++ + +C++ shares many of the same features as C, except that C++ does not guarantee that all fields are laid out in order unless they all have the same access control. C++ access control does not otherwise affect the ABI of a type, at least not with regards to layout; if a private field is removed from a struct (or class), the layout changes. + +C++ structs may also contain non-trivial fields. Rather than client code being able to copy them element-by-element, C++ generates a *copy constructor* that must be called whenever a struct value is copied (including when it is passed to a function). Copy constructors may also be written explicitly, in which case they may run arbitrary code; this affects what optimizations a C++ compiler can do. However, if the compiler can *see* that the copy constructor has not been customized, it may be able to optimize copies similarly to Swift. + +(This is overly simplified, but sufficient for this comparison.) + + +### Rust + +Rust likewise has a custom layout algorithm for the fields of its structs. While Rust has not put too much effort into having a stable ABI (other than interoperation with C), their current design behaves a lot like C's in practice: layout is known at compile-time, and changing a struct's fields will change the ABI. + +(there is an interesting post a couple of years ago about [optimizing the layout of Rust structs, tuples, and enums][rust].) + +Like C++ (and Swift), the fields of a Rust struct may be non-trivial to copy; unlike C++ (or Swift), Rust simply does not provide an implicit copy operation if that is the case. In fact, in order to preserve source compatibility, the author of the struct must *promise* that the type will *always* be trivial to copy by implementing a particular trait (protocol), if they want to allow clients to copy the struct through simple assignment. + +Rust also allows explicitly opting into C's layout algorithm. We could add that to Swift too if we wanted; for now we continue encouraging people to define such structs in C instead. + +[rust]: http://camlorn.net/posts/April%202017/rust-struct-field-reordering.html + + +### VM/JIT/Interpreted languages + +Languages where part of the compilation happens at run time are largely immune to these issues. In theory, the layout of a "struct" can be changed at any time, and any client code will automatically be updated to deal with that, because the runtime system will know either to handle the struct indirectly, or to emit fresh JITed code that can dynamically access the struct directly. + +### Objective-C + +Objective-C uses C structs for its equivalent of fixed-contents structs, but *non-*fixed-contents structs are instead represented using classes, often immutable classes. (An immutable class sidesteps any questions about value semantics as long as you don't test its identity, e.g. with `===`.) The presence of these classes in Apple's frameworks demonstrates the need for non-fixed-contents structs in Swift. + +Old versions of Objective-C (as in, older than iOS) had the same restrictions for the instance variables of a class as C does for the fields of a struct: any change would result in a change in layout, which would break any code that accessed instance variables directly. This was largely acceptable for classes that didn't need to be subclassed, but those that did had to keep their set of instance variables fixed. (This led to the same sort of "reserved" tricks used by C structs.) This restriction was lifted at the cost of some first-use set-up time with Apple's "non-fragile Objective-C runtime", introduced with iOS and with the 64-bit version of Mac OS X; it is now also supported in the open-source GNUstep runtime. + + +### Other languages + +Many other languages allow defining structs, but most of them either don't bother to define a stable ABI or don't allow modifying the fields of a struct once it becomes part of an ABI. Haskell and Ocaml fall into this bucket. + + +## Source compatibility + +This change does not impact source compatibility. It has always been a source-compatible change to modify a struct's non-public stored instance properties, and to add a new public stored instance property, as long as the struct's existing API does not change (including any initializers). + + +## Effect on ABI stability + +Currently, the layout of a public struct is known at compile time in both the defining library and in its clients. For a library concerned about binary compatibility, the layout of a non-fixed-contents struct must not be exposed to clients, since the library may choose to add new stored instance properties that do not fit in that layout in its next release, or even remove existing properties as long as they are not public. + +These considerations should not affect libraries shipped with their clients, including SwiftPM packages. These libraries should always have library evolution mode turned off, indicating that the compiler is free to optimize based on the layout of a type (because the library won't change). + + +## Effect on Library Evolution + +Both structs and enums can gain new protocol conformances or methods, even when `@frozen`. Binary compatibility only affects additions of fields or cases. + +The set of binary-compatible changes to a struct's stored instance properties is described above. There are no binary-compatible changes to an enum's cases. + +Taking an existing struct or enum and marking it `@frozen` is something we'd like to support without breaking binary compatibility, but there is no design for that yet. + +Removing `@frozen` from a type is not allowed; this would break any existing clients that rely on the existing layout. + +### Breaking the contract + +Because the compiler uses the set of fields in a fixed-contents struct to determine its in-memory representation and calling convention, adding a new field or removing `@frozen` from a type in a library will result in "undefined behavior" from any client apps that have not been recompiled. This means a loss of memory-safety and type-safety on par with a misuse of "unsafe" types, which would most likely lead to crashes but could lead to code unexpectedly being executed or skipped. In short, things would be very bad. + +Some ideas for how to prevent library authors from breaking the rules accidentally are discussed in "Compatibility checking" under "Future directions". + + +## Future directions + +### Compatibility checking + +Of course, the compiler can't stop a library author from modifying the fields of a fixed-contents struct, even though that will break binary compatibility. We already have two ideas on how we could catch mistakes of this nature: + +- A checker that can compare APIs across library versions, using swiftmodule files or similar. + +- Encoding the layout of a type in a symbol name. Clients could link against this symbol so that they'd fail to launch if it changes, but even without that an automated system could check the list of exported symbols to make sure nothing was removed. + + +### Allowing observing accessors + +The limitation on observing accessors for stored instance properties is an artificial one; clients could certainly access the field directly for "read" accesses but go through a setter method for "write" accesses. However, the expected use for `@frozen` is for structs that need to be as efficient as C structs, and that includes being able to do direct stores into public instance properties. The effect of observing accessors can be emulated in a fixed-contents struct with a private stored property and a public computed property. + +If we wanted to add this later, the way we would probably implement it is by saying that accessors for stored instance properties of `@frozen` structs must be marked `@inlinable`, with all of the implications that has for being able to make changes in the future. + + +## Alternatives considered + +### Annotation syntax + +#### Naming + +`@frozen` comes from [SE-0192][], where it was originally used as a term for for enums that will not gain new cases. + +Other than `@frozen`, most names have revolved around the notion of "fixed" somehow: `@fixedContents`, `@fixedLayout`, or even just `@fixed`. We ultimately chose `@frozen` as there is a strong similarity between this feature for both enums and structs. + +`final` was suggested at one point, but the meaning is a little far from that of `final` on classes. + +#### Modifier or attribute? + +This proposal suggests a new *attribute*, `@frozen`; it could also be a modifier `frozen`, implemented as a context-sensitive keyword. Because this annotation only affects the type's *implementation* rather than anything about its use, an attribute seemed more appropriate. + +### Command-line flag naming + +As mentioned above, the current spelling of the flag`-enable-library-evolution`. + +The term "resilience" has been an umbrella name for the Swift project's efforts to support evolving an API without breaking binary compatibility, so for some time the flag was flag `-enable-resilience`. But it isn't well-known outside of the Swift world. It's better to have this flag be more self-explanatory. + +Alternate bikeshed colors: + +- `-enable-abi-stability` +- `-enable-stable-abi` +- just `-stable-abi` + +In practice, the precise name of this flag won't be very important, because (1) the number of people outside of Apple making libraries with binary stability concerns won't be very high, and (2) most people will be building those libraries through Xcode or SwiftPM, which will have its own name for this mode. But naming the *feature* is still important. + +### "No reordering" vs. "No renaming" + +As written, this proposal disallows *reordering* field of a fixed-contents struct. This matches the restrictions of C structs, but is different from everything else in Swift, which freely allows reordering. Why? + +Let's step back and remember our goal: a stable, unchanging ABI for a fixed-contents struct. That means that whatever changes are made to the struct, the in-memory representation and the rest of the ABI must be consistent across library versions. We can think of this as a function that takes fields and produces a layout: + +```swift +struct Field { + var name: String + var type: ValueType +} +typealias Offset = Int +func layoutForFixedContentsStruct(fields: [Field]) -> [Offset: Field] +``` + +To make things simple, let's assume all the fields in the struct have the same type, like this PolarPoint: + +```swift +@frozen public struct PolarPoint { + // Yes, these have different access levels. + // You probably wouldn't do that in real life, + // but it's needed for this contrived example. + public var radius: Double + private var angle: Double + + public init(x: Double, y: Double) { + self.radius = sqrt(x * x, y * y) + self.angle = atan2(y, x) + } + + public var x: Double { + return radius * cos(angle) + } + public var y: Double { + return radius * sin(angle) + } +} +``` + +We have two choices for how to lay out the PolarPoint struct: + +- `[0: radius, 8: angle]` +- `[0: angle, 8: radius]` + +*It's important that everyone using the struct agrees on which layout we're using.* Otherwise, an angle could get misinterpreted as a radius when someone outside the module tries to access the `radius` property. + +So how do we decide? We have three choices: + +1. Sort the fields by declaration order. +2. Sort the fields by name. +3. Do something clever with access levels. + +Which translates to three possible restrictions: + +1. Fields of a fixed-contents struct may not be reordered, even non-public ones. +2. Fields of a fixed-contents struct may not be renamed, even non-public ones. +3. Copying the struct requires calling a function because we can't see the private fields. + +(3) is a non-starter, since it defeats the performance motivation for `@frozen`. That leaves (1) and (2), and we decided it would be *really weird* if renaming the private property `angle` to `theta` changed the layout of the struct. So the restriction on reordering fields was considered the least bad solution. It helps that it matches the behavior of C, Rust, and other languages. + + +### Fixed-contents by default + +No major consideration was given towards making structs fixed-contents by default, with an annotation to make them "changeable". While this is the effective behavior of other languages, the library design philosophy in Swift is that changes that don't break source compatibility shouldn't break binary compatibility whenever possible. Furthermore, while this may be the behavior of other languages, it's also been a longstanding *complaint* in other languages like C. We want to make structs in ABI-stable libraries as easy to evolve over time as classes are in Objective-C, while still providing an opt-out for structs that have high performance requirements. + + +### Binary stability mode by default + +Rather than introduce a new compiler flag, Swift could just treat *all* library code as if it defined ABI-stable libraries. However, this could result in a noticeable performance hit when it isn't really warranted; if a library does *not* have binary-compatibility concerns (say, if it is distributed with the app that uses it), there is no benefit to handling structs and enums indirectly. In practice, it's likely we'd start seeing people put `@frozen` in their source packages. Keeping this as a flag means that the notion of a "frozen" type can be something that most users and even most library authors don't have to think about. + +(Note that we'd love to have this be true for inlinable functions too, but that's a lot trickier on the implementation side, which is why `@inlinable` was considered worth adding in [SE-0193][] despite the additional complexity.) diff --git a/proposals/0261-identifiable.md b/proposals/0261-identifiable.md new file mode 100644 index 0000000000..c19eda84de --- /dev/null +++ b/proposals/0261-identifiable.md @@ -0,0 +1,264 @@ +# Identifiable Protocol + +* Proposal: [SE-0261](0261-identifiable.md) +* Authors: [Matthew Johnson](https://github.com/anandabits), [Kyle Macomber](https://github.com/kylemacomber) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.1)** +* Implementation: [apple/swift#26022](https://github.com/apple/swift/pull/26022) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0261-identifiable-protocol/27358) + +## Introduction + +This proposal introduces an `Identifiable` protocol, a general concept that is broadly useful— +for diff algorithms, user interface libraries, and other generic code—to +correlate snapshots of the state of an entity in order to identify changes. It +is a fundamental notion that deserves representation in the standard library. + +Swift-evolution thread: [Move SwiftUI's `Identifiable` and related types into +the standard library](https://forums.swift.org/t/move-swiftuis-identifiable-protocol-and-related-types-into-the-standard-library/25713) + +## Motivation + +There are many use cases for identifying distinct values as belonging to a +single logical entity. Consider a `Contact` record: + +```swift +struct Contact { + var id: Int + var name: String +} + +let john = Contact(id: 1000, name: "John Appleseed") +var johnny = john +johnny.name = "Johnny Appleseed" +``` + +Snapshots of a `Contact`, like `john` and `johnny`, refer to the same logical +person, even though that person may change their name over time and at any +moment, may share any number of other details with distinct persons. Being able +to determine that two such snapshots belong to the same logical entity is a +broadly useful capability. + +Representing such identity as simply the `ObjectIdentifier` of a class instance +(or using `===` directly) sometimes works, but there are cases, such as when the +instances are persistent or distributed across processes, where it simply +doesn't, and even when it does work, allocating class instances to represent +identity of value types is needlessly costly. + +### Diffing + +User interfaces often involve collections of elements, each of which represents +an entity. Consider a list of favorite contacts: + +```swift +struct FavoriteContactList: View { + var favorites: [Contact] + + var body: some View { + List(favorites) { contact in + FavoriteCell(contact) + } + } +} +``` + +In order to provide a high quality user experience when updating such a user +interface with new content it is necessary to distinguish between the identity +of the represented entity and the representation of the state of the entity that +is presented to the user. Content in an interface representing an entity whose +state has changed but identity has not should be updated in place (rather than +resorting to removing the old content and inserting the new content). + +A user interface component is capable of making such a distinction if its +represented entities are `Identifiable`: + +```swift +struct List { + init( + _ data: Data, + @ViewBuilder rowContent: @escaping (Data.Element) -> RowContent + ) where Data.Element: Identifiable +} +``` + +`Identifiable` supports diff algorithms that are able to report entity insertions, +moves and removals. These algorithms are also able to detect _changes_ to the +state of an entity that is represented in both collections. This can include +changes to the state of an entity that _also_ moves in the collection. + +While diffs are often applied to the user interface layer of a program the diff +algorithm does not necessarily need to run in the user interface layer. It can +be desirable to compute a diff in the model layer. For example, the model layer +updates may be processed in the background and the diff can be computed before +moving back to the main thread to apply the changes to the UI. There may also +be more than one simultaneous presentation of the same data in the UI, in which +case computing the diff in the UI layer is redundant. + +Model layer code that performs these computations often has no dependencies +outside the standard library itself. It is unlikely to accept a dependency on +a UI framework that defines its own `Identifiable` protocol. If `Identifiable` +doesn't move to the standard library Swift programmers +will need to continue using their own variation of this protocol and will need +to ensure it is able co-exist with other similar definitions found in other frameworks higher +up the dependency stack. Unfortunately none these variations +are likely to be compatible with one another. + +## Proposed solution + +The proposed solution is to define a new `Identifiable` protocol: + +```swift +/// A class of types whose instances hold the value of an entity with stable identity. +protocol Identifiable { + + /// A type representing the stable identity of the entity associated with `self`. + associatedtype ID: Hashable + + /// The stable identity of the entity associated with `self`. + var id: ID { get } +} +``` + +This protocol will be used by diff algorithms, user interface libraries and other +generic code to correlate snapshots of the state of an entity in order to identify +changes to that state from one snapshot to another. + +An example conformance follows: + +```swift +struct Contact: Identifiable { + var id: Int + var name: String +} +``` + +There are a variety of considerations (value or reference semantics, persisted, +distributed, performance, convenience, etc.) to weigh when choosing the +appropriate representation of identity for an entity. `ID` is an associatedtype +because no single concrete type of identifier is appropriate in all cases. + +`id` was chosen as the name of the requirement over the unabbreviated form +because it is a [frequently used](https://www.swiftbysundell.com/posts/type-safe-identifiers-in-swift) +term of art that will allow easy conformance. + +## Detailed design + +### Object identifiability + +In order to make it as convenient as possible to conform to `Identifiable`, a +default `id` is provided for all class instances: + +```swift +extension Identifiable where Self: AnyObject { + var id: ObjectIdentifier { + return ObjectIdentifier(self) + } +} +``` + +Then, a class whose instances are identified by their object identities need not +explicitly provide an `id`: + +```swift +final class Contact: Identifiable { + var name: String + + init(name: String) { + self.name = name + } +} +``` + +Note, a class may provide a custom implementation of `id`: + +```swift +final class Contact: Identifiable { + let id: Int + let name: String + + init(id: Int, name: String) { + self.id = id + self.name = name + } +} +``` + +## Source compatibility + +This is a purely additive change. + +## Effect on ABI stability + +This is a purely additive change. + +## Effect on API resilience + +This has no impact on API resilience which is not already captured by other +language features. + +## Alternatives considered + +### Per-use identification + +Instead of constraining a collection's elements to an `Identifiable` protocol, +generic code could take an additional parameter that projects the identity of an +entity from its representation: + +```swift +struct FavoriteContactList: View { + var favorites: [Contact] + + var body: some View { + List(favorites, id: \.id) { contact in + FavoriteCell(contact) + } + } +} + +struct List { + public init( + _ data: Data, + id: KeyPath, + @ViewBuilder rowContent: @escaping (Data.Element) -> RowContent + ) +} +``` + +This is undesirable because a type generally has a single, canonical identity, +but this approach unnecessarily re-defines an entity's identity at every use +site, which is error-prone. + +Furthermore, this isn't a practical alternative because there is evidence that +if Swift doesn't define an `Identifiable` concept, libraries will opt to define +their own rather than take an identifier at the use-site. + +### Concrete conformances + +The purpose of `Identifiable` is to distinguish the identity of an entity from +the state of an entity. Concrete types like `UUID`, `Int`, and `String` are +commonly _used as identifiers_, however they do not _have an identifier_, so +they should not conform to `Identifiable`. + +## Future directions + +### Collection diffing + +Today there is a collection diffing convenience for `Equatable` elements: + +```swift +extension BidirectionalCollection where Element: Equatable { + func difference( + from other: C + ) -> CollectionDifference where C.Element == Self.Element +} +``` + +It may be desirable to add a similar convenience for `Identifiable` elements +(and prefer use of `Identifiable` to `Equatable` when a type conforms to both). +This is omitted from the immediate proposal in order to keep it focused. + +### Conditional conformances + +It may be desirable to provide the conditional conformance +`Optional: Identifiable where Wrapped: Identifiable`. This is omitted from the +immediate proposal in order to keep it focused. diff --git a/proposals/0262-demangle.md b/proposals/0262-demangle.md new file mode 100644 index 0000000000..447a02c7f9 --- /dev/null +++ b/proposals/0262-demangle.md @@ -0,0 +1,146 @@ +# Demangle Function + +* Proposal: [SE-0262](0262-demangle.md) +* Author: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Returned for revision** +* Implementation: [apple/swift#25314](https://github.com/apple/swift/pull/25314) +* Decision Notes: [Returned for revision](https://forums.swift.org/t/returned-for-revision-se-0262-demangle-function/28186) + +## Introduction + +Introduce a new standard library function, `demangle`, that takes a mangled Swift symbol, like `$sSS7cStringSSSPys4Int8VG_tcfC`, and output the human readable Swift symbol, like `Swift.String.init(cString: Swift.UnsafePointer) -> Swift.String`. + +Swift-evolution thread: [Demangle Function](https://forums.swift.org/t/demangle-function/25416) + +## Motivation + +Currently in Swift, if a user is given an unreadable mangled symbol, they're most likely to use the `swift-demangle` tool to get the demangled version. However, this is a little awkward when you want to demangle a symbol in-process in Swift. One could create a new `Process` from Foundation and set it up to launch a new process within the process to use `swift-demangle`, but the standard library can do better and easier. + +## Proposed solution + +The standard library will add the following 3 new functions. + +```swift +// Given a mangled Swift symbol, return the demangled symbol. +public func demangle(_ input: String) -> String? + +// Given a mangled Swift symbol in a buffer and a preallocated buffer, +// write the demangled symbol into the buffer. +public func demangle( + _ mangledNameBuffer: UnsafeBufferPointer, + into buffer: UnsafeMutableBufferPointer +) -> DemangleResult + +// Given a mangled Swift symbol and a preallocated buffer, +// write the demangle symbol into the buffer. +public func demangle( + _ input: String, + into buffer: UnsafeMutableBufferPointer +) -> DemangleResult +``` + +as well as the following enum to indicate success or the different forms of failure: + +```swift +public enum DemangleResult: Equatable { + // The demangle was successful + case success + + // The result was truncated. Payload contains the number of bytes + // required for the complete demangle. + case truncated(Int) + + // The given Swift symbol was invalid. + case invalidSymbol +} +``` + +Examples: + +```swift +print(demangle("$s8Demangle3FooV")!) // Demangle.Foo + +// Demangle.Foo is 13 characters + 1 null terminator +let buffer = UnsafeMutableBufferPointer.allocate(capacity: 14) +defer { buffer.deallocate() } + +let result = demangle("$s8Demangle3BarV", into: buffer) + +guard result == .success else { + // Handle failure here + switch result { + case let .truncated(required): + print("We need \(required - buffer.count) more bytes!") + case .invalidSymbol: + print("I was given a faulty symbol?!") + default: + break + } + + return +} + +print(String(cString: buffer.baseAddress!)) // Demangle.Foo +``` + +## Detailed design + +If one were to pass a string that wasn't a valid Swift mangled symbol, like `abc123`, then the `(String) -> String?` would simply return nil to indicate failure. With the `(String, into: UnsafeMutableBufferPointer) -> DemangleResult` version and the buffer input version, we wouldn't write the passed string into the buffer if it were invalid. + +This proposal includes a trivial `(String) -> String?` version of the function, as well as a version that takes a buffer. In addition to the invalid input error case, the buffer variants can also fail due to truncation. This occurs when the output buffer doesn't have enough allocated space for the entire demangled result. In this case, we return `.truncated(Int)` where the payload is equal to the total number of bytes required for the entire demangled result. We're still able to demangle a truncated version of the symbol into the buffer, but not the whole symbol if the buffer is smaller than needed. E.g. + +```swift +// Swift.Int requires 10 bytes = 9 characters + 1 null terminator +// Give this 9 to exercise truncation +let buffer = UnsafeMutableBufferPointer.allocate(capacity: 9) +defer { buffer.deallocate() } + +if case let .truncated(required) = demangle("$sSi", into: buffer) { + print(required) // 10 (this is the amount needed for the full Swift.Int) + let difference = required - buffer.count + print(difference) // 1 (we only need 1 more byte in addition to the 9 we already allocated) +} + +print(String(cString: buffer.baseAddress!)) // Swift.In (notice the missing T) +``` + +This implementation relies on the Swift runtime function `swift_demangle` which accepts symbols that start with `_T`, `_T0`, `$S`, and `$s`. + +## Source compatibility + +These are completely new standard library functions, thus source compatibility is unaffected. + +## Effect on ABI stability + +These are completely new standard library functions, thus ABI compatibility is unaffected. + +## Effect on API resilience + +These are completely new standard library functions, thus API resilience is unaffected. + +## Alternatives considered + +We could choose to only provide one of the proposed functions, but each of these brings unique purposes. The trivial take a string and return a string version is a very simplistic version in cases where maybe you're not worried about allocating new memory, and the buffer variants where you don't want to alloc new memory and want to pass in some memory you've already allocated. + +## Future Directions + +The `swift_demangle` runtime function has an extra `flags` parameter, but currently it is not being used for anything. In the future if that function ever supports any flags, it would make sense to introduce new overloads or something similar to expose those flags to the standard library as well. E.g. + +```swift +public func demangle(_ input: String, flags: DemangleFlags) -> String? + +public func demangle( + _ mangledNameBuffer: UnsafeBufferPointer, + into buffer: UnsafeMutableBufferPointer, + flags: DemangleFlags +) -> DemangleResult + +public func demangle( + _ input: String, + into buffer: UnsafeMutableBufferPointer, + flags: DemangleFlags +) -> DemangleResult +``` + +where `DemangleFlags` could be an enum, `OptionSet`, `[DemangleFlag]`, etc. diff --git a/proposals/0263-string-uninitialized-initializer.md b/proposals/0263-string-uninitialized-initializer.md new file mode 100644 index 0000000000..826f27abe9 --- /dev/null +++ b/proposals/0263-string-uninitialized-initializer.md @@ -0,0 +1,139 @@ +# Add a String Initializer with Access to Uninitialized Storage + +* Proposal: [SE-0263](0263-string-uninitialized-initializer.md) +* Author: [David Smith](https://github.com/Catfish-Man) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift#26007](https://github.com/apple/swift/pull/26007), + [apple/swift#30106](https://github.com/apple/swift/pull/30106) +* Bug: [SR-10288](https://bugs.swift.org/browse/SR-10288) + +## Introduction + +This proposal suggests a new initializer for `String` that provides access to a String's uninitialized storage buffer. + +## Motivation + +`String` today is well-suited to interoperability with raw memory buffers when a contiguous buffer is already available, such as when dealing with `malloc`ed C strings. However, there are quite a few situations where no such buffer is available, requiring a temporary one to be allocated and copied into. One example is bridging `NSString` to `String`, which currently uses standard library internals to get good performance when using `CFStringGetBytes`. Another, also from the standard library, is `Int` and `Float`, which currently create temporary stack buffers and do extra copying. We expect libraries like SwiftNIO will also find this useful for dealing with streaming data. + +## Proposed solution + +Add a new `String` initializer that lets a program work with an uninitialized +buffer. + +The new initializer takes a closure that operates on an +`UnsafeMutableBufferPointer` referencing the uninitialized contents of the newly created +String's storage, and returns a count of initialized elements or 0. + +```swift +let myCocoaString = NSString("The quick brown fox jumps over the lazy dog") as CFString +var myString = String(unsafeUninitializedCapacity: CFStringGetMaximumSizeForEncoding(myCocoaString, …)) { buffer in + var initializedCount = 0 + CFStringGetBytes( + myCocoaString, + buffer, + …, + &initializedCount + ) + return initializedCount +} +// myString == "The quick brown fox jumps over the lazy dog" +``` + +Without this initializer we would have had to heap allocate an `UnsafeMutableBufferPointer`, copy the `NSString` contents into it, and then copy the buffer again as we initialized the `String`. + +## Detailed design + +```swift + /// Creates a new String with the specified capacity in UTF-8 code units then + /// calls the given closure with a buffer covering the String's uninitialized + /// memory. + /// + /// The closure should return the number of initialized code units, + /// or 0 if it couldn't initialize the buffer (for example if the + /// requested capacity was too small). + /// + /// This method replaces ill-formed UTF-8 sequences with the Unicode + /// replacement character (`"\u{FFFD}"`); This may require resizing + /// the buffer beyond its original capacity. + /// + /// The following examples use this initializer with the contents of two + /// different `UInt8` arrays---the first with well-formed UTF-8 code unit + /// sequences and the second with an ill-formed sequence at the end. + /// + /// let validUTF8: [UInt8] = [67, 97, 102, -61, -87, 0] + /// let s = String(unsafeUninitializedCapacity: validUTF8.count, + /// initializingUTF8With: { ptr in + /// ptr.initializeFrom(validUTF8) + /// return validUTF8.count + /// }) + /// // Prints "Café" + /// + /// let invalidUTF8: [UInt8] = [67, 97, 102, -61, 0] + /// let s = String(unsafeUninitializedCapacity: invalidUTF8.count, + /// initializingUTF8With: { ptr in + /// ptr.initializeFrom(invalidUTF8) + /// return invalidUTF8.count + /// }) + /// // Prints "Caf�" + /// + /// let s = String(unsafeUninitializedCapacity: invalidUTF8.count, + /// initializingUTF8With: { ptr in + /// ptr.initializeFrom(invalidUTF8) + /// return 0 + /// }) + /// // Prints "" + /// + /// - Parameters: + /// - capacity: The number of UTF-8 code units worth of memory to allocate + /// for the String. + /// - initializer: A closure that initializes elements and sets the count of + /// the new String + /// - Parameters: + /// - buffer: A buffer covering uninitialized memory with room for the + /// specified number of UTF-8 code units. + public init( + unsafeUninitializedCapacity capacity: Int, + initializingUTF8With initializer: ( + _ buffer: UnsafeMutableBufferPointer, + ) throws -> Int + ) rethrows +``` + +### Specifying a capacity + +The initializer takes the specific capacity that a user wants to work with as a +parameter. The buffer passed to the closure has a count that is at least the +same as the specified capacity, even if the ultimate size of the new `String` is larger. + +### Guarantees after throwing + +Because `UTF8.CodeUnit` is a trivial type, there are no special considerations about the state of the buffer when an error is thrown, unlike `Array`. + +## Source compatibility + +This is an additive change to the standard library, +so there is no effect on source compatibility. + +## Effect on ABI stability + +The new initializer will be part of the ABI, and may result in calls to a new @usableFromInline symbol being inlined into client code. Use of the new initializer is gated by @availability though, so there's no back-deployment concern. + +## Effect on API resilience + +The additional APIs will be a permanent part of the standard library, +and will need to remain public API. + +## Alternatives considered + +### Taking an inout count in the initializer rather than returning the new count + +Consistency with `Array` (which has to use the inout count for correctness) has some appeal here, but ultimately we decided that the value of consistency lies in allowing skill transfer and in repeated use, and these highly specialized initializers don't really allow for that. + +### Returning a `Bool` to indicate success from the closure + +Requiring people to either `throw` or check in the caller for an empty `String` return if the initializing closure fails is slightly awkward, but making the initializer failable is at least as awkward, and would be inconsistent with `Array`. + +### Validating UTF-8 instead of repairing invalid UTF-8 + +Matching the behavior of most other `String` initializers here also makes it more ergonomic to use, since it can be non-failable this way. diff --git a/proposals/0264-stdlib-preview-package.md b/proposals/0264-stdlib-preview-package.md new file mode 100644 index 0000000000..bb37b2d3d6 --- /dev/null +++ b/proposals/0264-stdlib-preview-package.md @@ -0,0 +1,176 @@ +# Standard Library Preview Package + +* Proposal: [SE-0264](0264-stdlib-preview-package.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Max Moiseev](https://github.com/moiseev), [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0264-review-2-standard-library-preview-package/31288/16) +* Implementation: + 1. [swiftlang/swift-evolution#1089](https://github.com/swiftlang/swift-evolution/pull/1089) + 2. [swiftlang/swift-evolution#1090](https://github.com/swiftlang/swift-evolution/pull/1090) + 3. [swiftlang/swift-evolution#1091](https://github.com/swiftlang/swift-evolution/pull/1091) +* Pitch Discussion: [Pitch: Standard Library Preview Package](https://forums.swift.org/t/pitch-standard-library-preview-package/27202) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/1547e503376bca2c64c57c96b1f87d5e01a094c3/proposals/0264-stdlib-preview-package.md) +* Previous Review: [SE-0264 — Standard Library Preview Package](https://forums.swift.org/t/se-0264-standard-library-preview-package/29068) +* Previous Decision: [Returned for Revision](https://forums.swift.org/t/returned-for-revision-se-0264-standard-library-preview-package/29865) + +## Introduction + +We propose changing the Swift Evolution process to publish accepted proposals as individual SwiftPM packages, as well as a `SwiftPreview` package that bundles these proposal packages together. This group of packages will form the initial landing spot for certain additions to the Swift standard library. + +Adding these packages serves the goal of allowing for rapid adoption of new standard library features, enabling sooner real-world feedback, and allowing for an initial period of time where that feedback can lead to source- and ABI-breaking changes if needed. + +As a secondary benefit, it will reduce technical challenges for new community members implementing new features in the standard library. + +In the first iteration, this package will take the following: + +- free functions and methods, subscripts, and computed properties via extensions, that do not require access to the internal implementation of standard library types and that could reasonably be emitted into the client +- new types (for example, a sorted dictionary, or useful property wrapper implementations) +- new protocols, with conformance of existing library types to those protocols as appropriate + +These packages will not include features that need to be matched with language changes, nor functions that cannot practically be implemented without access to existing type internals or other non-public features such as LLVM builtins. + +For the purposes of this document, we will refer to the individual packages and the `SwiftPreview` package as "the preview packages" and the standard library that ships with the toolchain as "the library". + +## Motivation + +### Facilitating Rapid Adoption and Feedback + +It is common for a feature proposed and accepted through Swift Evolution and added to the standard library to wait for months before it gets any real life usage. Take [SE-0199](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0199-bool-toggle.md) (the introduction of `toggle`) for example. It was [accepted](https://forums.swift.org/t/accepted-se-199-add-toggle-to-bool/10681) almost 6 months before Swift 4.2 was released. + +Even though all additions go through a thorough review, there is no substitute for feedback from real-world usage in production. Sometimes we discover that the API is not quite as good as it might have been. + +While the toolchains downloadable from swift.org are useful for experimentation, they cannot be used to ship applications, so are rarely used for much more than "kicking the tires" on a feature. The beta period for Xcode provides slightly more real usage, but is still relatively short, with feedback often coming too late to be applied, given that any changes would require a further review on Swift evolution as well as time to integrate into the converging release. And the nature of standard library additions are such that you do not always have an immediate need early in the beta to try out a feature such as partial sort or a min heap. + +Once a feature ships as part of a Swift release, any future changes resulting from feedback from real usage must clear the very high bar of source and ABI stability. Even if the change is merited and a source break is justified, the absolute need for ABI stability can rule out certain changes entirely on technical grounds. Furthermore, for performance reasons, standard library types that rely on specialization need to expose some of their internal implementation as part of their ABI, closing off future optimizations or performance fixes that are only discovered through subsequent usage and feedback. + +### Technical Challenges for Contributors + +The requirements for contributing to the standard library can be prohibitive. Not everybody has the time and resources to build a whole stack including LLVM, Clang, and the Swift compiler itself just to change a part of the standard library. Additionally, Xcode and XCTest cannot easily be used to maintain and test standard library code. + +Integrating changes into the standard library requires knowledge of non-public features and idioms relating to ABI-stability. In particular, implementing a performant, specializable, ABI-stable collection type while preserving partial future flexibility is _significantly_ harder than writing one that is only source-stable. Harder still is knowing what internal implementation details can be changed after such a partially-transparent type has been published as ABI. + +It would be of great benefit to the community to allow proposals and contributions to the standard library without these requirements, leaving integration of the final ABI-stable version to a later date and possibly other contributors with more experience maintaining ABI-stable code. + +## Proposed solution + +We propose introducing individual SwiftPM packages for each approved proposal that meets the criteria above, as well as a `SwiftPreview` package that imports and re-exports each of the individual proposal packages. These packages will be standalone projects hosted on GitHub under the Apple organization, and will be part of the overall Swift project. + +Whenever possible, proposals for additions to the standard library will land first as a package before migrating to the standard library. A PR against the `SwiftPreview` package will be sufficient to fulfill the implementation requirement for an evolution proposal. Proposals that are accepted will be published as packages immediately, and then be integrated into the standard library. For more detail on this process, see _Evolution_ below. + +All additions to the standard library will **continue to use the Swift Evolution process**, no matter whether they first land as a package or not. All proposed additions to the standard library should be made with the understanding that they will migrate to the ABI-stable standard library module that ships as part of the Swift toolchain. + +If usage after acceptance of a proposal reveals unanticipated problems, a follow-up proposal or amendment will be able to make source-breaking changes. Unlike the very high bar for source-breaking changes to the standard library, and the absolute rules around ABI stability, the bar for changes to API that haven't shipped as part of a Swift release will be that of any other proposal of a new API. + +Starting life as a package will not be mandatory. Proposed additions can instead go straight into the standard library, if they do not meet the criteria for suitable package additions. See the detailed design section for an expanded discussion of these criteria. + +Since the packages will not be ABI-stable, they will not ship as a binary on any platform, or be a dependency of any ABI-stable package. This allows for changes to the internal implementation of any type, and the change/removal of any function, as part of implementation changes, follow-on proposal amendments, or subsequent proposals. + +## Detailed design + +The following additive evolution proposals could be made using the preview package process: + +- New algorithms implemented as extensions on library protocols +- Utility types (e.g. wrapper types returned from functions like the `Lazy` types underlying `.lazy`) +- New protocols, including conformances by standard library types +- New collection types +- Property wrappers, such as a late-initialized wrapper + +The following are examples of changes that would _not_ be published in the preview packages: + +- Types introduced as part of and inseparable from language features (like `Optional` or `Error`) +- Implementations that rely on builtins for a reasonable implementation (like atomics or SIMD types) +- Implementations that require access to other types internals to be performant +- Changes that can't be done via a package, like adding customization points to protocols + +Some of these cases will require a judgement call. For example, making an extension method a customization point may bring a minor performance optimization — and so not prevent initial deployment in the package — or it may be a major part of the implementation, making the difference between an `O(1)` and `O(n)` implementation. Whether a proposal will be published as a package should be part of the evolution pitch and review discussions. + +### Evolution + +The introduction of the preview packages does not change much in the process of Swift Evolution. Changes to the standard library API surface should go through a well-developed pitch - discussion - implementation - proposal - decision life-cycle. + +Proposal authors will provide the required implementation that accompanies their proposal by opening a pull request against a new `swift-evolution-staging` repository. This implementation PR will be merged into its own branch before the start of a review to facilitate experimentation during the review process. Upon completion of the review, the proposed package will be moved into its own repository, or removed from the `swift-evolution-staging` repository if the proposal was rejected. + +The main difference from the existing process is that the final result will become available for general use immediately in the preview packages. As users have real-world experience with the accepted functionality, any proposed amendments to the proposal must go through the same evolution process as an original proposal. + +To provide time for feedback from real-world use of new additions, the Swift Evolution process should favor landing new features in a window immediately after branching for a major release. This doesn’t mean that important and valuable proposals can’t be added at different times, but they’ll be subject to increased scrutiny. + +No proposal should be accepted into the preview packages on a provisional basis: Reviewers should assume that every proposal will be migrated as-is unless feedback reveals that it was a clear misadventure. The review manager for any revising proposals will be responsible for ensuring that the review discussion focuses on feedback from real-world use, and not relitigation of previously settled decisions made during the original review. + +### Migration + +An important aspect of the design is the experience of users of the preview packages when features that have been available from the packages become available in the standard library in a new Swift or platform release. This is handled differently for functions and types. + +#### Types + +The preview packages will have different module names (e.g. `SwiftPreview`, `SE250_LeftPad`) than that of the standard library (`Swift`). As such, every type it re-exports will be distinct from the type once it migrates into the standard library, and can co-exist with it. Because the preview packages are "above" the `Swift` module in the stack, users of the packages will get the package version of the type by default in source files that have imported a preview package. They will be able to specify the library version instead by prepending `Swift.` to the type name. This has the benefit that addition of the type into the library is source-compatible with code already using the package version. + +It does have a downside for code size. Package users do not benefit from the code-size wins of not including the new type in their app and instead using a version in the OS (though, given that most types in the standard library are generic and need to be specialized, this is not as big a problem as it might be). It would also mean package users miss out on optimizations that may be possible with the library implementation. Once inside the library, fast paths could benefit from internals of other types. For example, a ring buffer might be implemented in terms of the same storage as an `Array`, and conversion from one type to another could just be a question of copying a reference to the other's storage. + +In these cases where the standard library's version of the type would be better, the user can easily switch to it by adding a `typealias TheType = Swift.TheType` to their code, or by prefixing the module on individual declarations if needed. + +#### Functions + +Unlike types, methods cannot be disambiguated. That is, you cannot write something like `myCollection.SwiftPreview.partialSort`. Swift does not have this feature yet (and while desirable, it should not be a dependency of this proposal). So there is no way similar to the typealias approach above to prefer the standard library's implementation of a protocol extension over the packages. + +Since Swift 5.1, the standard library has had an internal feature that allows it to force-emit the body of a function into the client code, as a way of back-deploying critical bug fixes. This `@_alwaysEmitIntoClient` attribute can be used from within the standard library to deploy functions only to prior platforms, at the cost of binary size of the client (when they use them). Again, this cost is mitigated in the case of protocol extensions by the fact that the specialized implementation is already inlined. + +This allows use of `#if compiler` directives to simultaneously obsolete implementations in the package for the latest version of Swift when introducing new functions into the standard library. As long as the new library definitions are marked as `@_alwaysEmitIntoClient` the source compatibility of this should not be noticeable. + +#### Type/Function Combinations + +A common pattern used in the standard library is to return a type from a function, with the type driving much of the functionality initiated by the function call. For example, `myArray.lazy.map` returns a `LazySequence` which maps elements on the fly using the supplied closure. + +Since types cannot be emitted into client code, these combination function/type features will follow the pattern for types. This will mean that the package version will be used by default, and type context will be needed to force a call to the standard library if desired. + +#### Retirement from the `SwiftPreview` Package + +In order to keep the size of the `SwiftPreview` package manageable, re-exported proposal packages will be removed from the bundling package in a version update **1 year** after migration to the standard library. This is a necessary balance between maintainability and convenience. Since the proposal packages will still be available, it should be possible for users of the `SwiftPreview` package to import an individual package if they need to keep it while also wanting to upgrade to the latest version of the `SwiftPreview` package _without_ upgrading to the latest compiler. + +### Testing + +The standard library currently uses a combination of `StdlibUnittest` tests and raw `lit` tests. This works well, but is unfamiliar to most developers. + +The preferred style of test for Swift packages is `XCTest`, and the preview packages will adopt this approach. This has the benefit of familiarity, and integrates well with Xcode. The work to convert tests from this format to lit can be done at the point of migration into the library. + +Certain features of `StdlibUnittest` – in particular crashLater and `StdlibCollectionUnittest`'s `checkCollection`, as well as the helper types like the minimal collections – should be ported over to `XCTest` if possible. This would make for an excellent starter bug, or could be done as part of introducing the first full-featured collection to the package that had a significant need for this kind of testing, but are not a requirement for accepting this proposal. + +### GYB + +There will be no use of [gyb](https://github.com/apple/swift/blob/master/utils/gyb.py) in the preview packages. The need for gyb has been gradually reducing over time as language features like conditional conformance and improved optimization eliminated the need for it. Gyb is a powerful and useful tool, but it can cause code that is difficult to read and write, and does not interact well with Xcode, so runs counter one of the goals of this package: to simplify contribution. + +### Versioning + +Individual proposal packages will use semantic versioning, with source-breaking changes only allowed at major version revisions. Once accepted via the Swift Evolution process, each proposal package will be published at version `1.0.0`. + +The fundamental goals of the bundling `SwiftPreview` package, on the other hand, are somewhat at odds with the usual goals of semantic versioning. The `SwiftPreview` package will always re-export the latest versions of the most recent proposals, so source-breaking changes, including retirements after promoting into the standard library, or source-breaking changes resulting from evolution proposals prior to promotion, will be common. Unlike many packages, including the individual proposal packages, the `SwiftPreview` package will not converge over time, which normally reduces the frequency of a package's version bumps. + +As such, the `SwiftPreview` package will remain at major version `0` **permanently**. Minor version bumps will be able to include source-breaking changes. Patch updates should not break source and will just be used for bug fixes. + +The use of a `0` major version at all times will act as a signifier that adopters need to be comfortable with the requirement to continuously update to the latest version of the package. It should not be taken as an indication that the package shouldn't be used for real-world code: the code should be considered production-worthy. But it is a preview, and not source stable. Where practical, deprecated shims could be used to preserve source compatibility between versions, but this may not always be practical. + +The decision when to tag minor versions, largely whenever proposal packages are added or removed, will be made by the release manager for the standard library as chosen by the Swift project lead. + +### Source compatibility + +None on Swift itself. The intention of the preview packages is to facilitate long-term source stability by detecting issues with APIs prior to their integration into the library. + +## Effect on ABI stability + +None. The preview packages will not be declared ABI stable. ABI stability will only happen when proposals are migrated to the standard library. This means that the preview packages may not be used within binary frameworks. + +Because we do not have cross-module optimization, package implementations should make use of `@inlinable` annotations. However, these annotations should be in the context of source stability only and should be fully reevaluated by the library integrator when stabilizing the ABI. + +## Alternatives considered + +### Additional Review Before Library Integration + +A previous version of this proposal prescribed a window of time before promoting accepted features into the standard library, to provide additional bake time and an additional review point for the feature. However, this approach delays some important feedback that can only be gained from integration into the standard library, where overload resolution and performance can have different outcomes than the same APIs shipped in a package. + +### No Semantic Versioning + +An alternative to keeping the semver major version at `0` is to not version the package at all. With this approach, the package could only be included by packages by specifying a branch-based dependency. However, this would be too restricting at this time, as a branch-based dependency can only be imported by other branch-based dependencies. This would effectively limit use of the `SwiftPreview` package to top-level applications only. + +### A Monolithic `SwiftPreview` Package + +A previous version of this proposal described a single preview package that included all of the approved proposal implementations. A monolithic package like that one poses a challenge to adopters attempting to reason about the size and scope of their dependencies. This is particularly an issue for packages like `SwiftPreview`, which is expected to break source compatibility as proposals are added and removed. Providing individual, versioned packages for each approved proposal, in addition to the umbrella `SwiftPreview` package, provides more control to adopters of preview functionality. diff --git a/proposals/0265-offset-indexing-and-slicing.md b/proposals/0265-offset-indexing-and-slicing.md new file mode 100644 index 0000000000..fcb605aee2 --- /dev/null +++ b/proposals/0265-offset-indexing-and-slicing.md @@ -0,0 +1,832 @@ +# Offset-Based Access to Indices, Elements, and Slices + +* Proposal: [SE-0265](0265-offset-indexing-and-slicing.md) +* Author: [Michael Ilseman](https://github.com/milseman) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Returned for revision** +* Implementation: [apple/swift#24296](https://github.com/apple/swift/pull/24296) +* Review: ([first review](https://forums.swift.org/t/se-0265-offset-based-access-to-indices-elements-and-slices/29596)) ([revision announcement](https://forums.swift.org/t/returned-for-revision-se-0265-offset-based-access-to-indices-elements-and-slices/30503)) + +## Introduction + +This proposal introduces `OffsetBound`, which can represent a position in a collection specified as an offset from either the beginning or end of the collection (i.e. the collection’s “bounds”). Corresponding APIs provide a more convenient abstraction over indices. The goal is to alleviate an expressivity gap in collection APIs by providing easy and safe means to access elements, indices, and slices from such offsets. + +If you would like to try it out, you can just copy-paste from [this gist](https://gist.github.com/milseman/1461e4f3e195974a5d1ad76cefdd6961), which includes the functionality as well as test cases and examples. This work is the culmination of prior discussion from an [earlier thread](https://forums.swift.org/t/pitch-offsetting-indices-and-relative-ranges/23837), the [thread before that](https://forums.swift.org/t/call-for-users-and-authors-offset-indexing-pitch/21444), and [@Letan](https://forums.swift.org/u/letan) ’s [original thread](https://forums.swift.org/t/shorthand-for-offsetting-startindex-and-endindex/9397). The latest pitch thread can be found [here](https://forums.swift.org/t/offset-indexing-and-slicing/28333). + + +## Motivation + +### Easily and Safely Getting Indices, Elements, and Slices from Offsets + +`Collection`’s current index manipulation methods are meant to represent the lowest level programming interface, and as such they impose requirements on their use which are important for performance. Violations of these requirements are treated as irrecoverable logic errors, trapping whenever they lead to a potential memory safety issue. But, `Collection` lacks higher-level APIs that allow the programmer to treat such violations as recoverable domain errors. This proposal addresses the gap. + +Extracting elements and slices from offsets into a collection is an onerous task, requiring manually advancing indices. This proposal offers an ergonomic approach to offsetting from the start or end of collections, including their slices. + +This commonly comes up with casual `String` usage, and aligns with the [String Essentials](https://forums.swift.org/t/string-essentials/21909) effort. + +For a simple example taken from [Advent of Code 2018 Day 7](https://adventofcode.com/2018/day/7) , we want a function taking a string where each line is of the form `Step C must be finished before step A can begin.` , and returns an array representing the requirement `(finish: "C", before: "A")`. + +```swift +func parseRequirements(_ s: String) -> [(finish: Character, before: Character)] { + s.split(separator: "\n").map { line in + let finishIdx = line.index(line.startIndex, offsetBy: 5) // 5 after first + let beforeIdx = line.index(line.endIndex, offsetBy: -12) // 11 before last + return (line[finishIdx], line[beforeIdx]) + } +} +``` + +Advancing indices by hand through `line.index(line.startIndex, offsetBy: 5)` is fairly obnoxious and distracts from the intent of the code. + +Alternatively, we could take a detour through forming `SubSequence`s: + +```swift +func parseRequirements(_ s: String) -> [(finish: Character, before: Character)] { + s.split(separator: "\n").map { line in + (line.dropFirst(5).first!, line.dropLast(11).last!) + } +} +``` + +This results in less boilerplate code, but the detour through slicing APIs increases the cognitive load. Anyone reading the code has to jump through mental hoops, and the code author has more details to reason through (when we first wrote this example, we had an off-by-one error). + +Instead, this proposal provides a way to directly extract elements from known offsets. + + +### Common Bugs and Assumptions in Int-Indexed Collections + +When a collection’s index type happens to be `Int`, it’s a common mistake to assume that such indices start from zero. For an example from [this forum post](https://forums.swift.org/t/subscripting-a-range-seems-unintuitive/23278), an `Int` range’s indices are the very elements themselves; they don’t start at zero. + +```swift +print((3..<10)[5...]) // 5..<10 +``` + +Slices share the same indices with the base collection. Assuming indices start from zero can be especially pernicious in generic code and when working with self-sliced types such as `Data`. + +```swift +func fifth(_ c: C) -> C.Element? where C.Index == Int { + return c.count >= 5 ?? c[4] : nil +} + +let array = [1,2,3,4,5,6,7,8,9] +print(fifth(array)!) // 5 +print(fifth(array[2...])!) // still `5`, but `7` would be the real fifth item + +func fifth(_ data: Data) -> UInt8? { + return data.count >= 5 ?? data[4] : nil +} + +var data = Data([1, 2, 3, 4, 5, 6, 7, 8, 9]) +print(fifth(data)!) // 5 + +data = data.dropFirst(2) +print(fifth(data)!) // still `5`, but `7` is the real fifth item +``` + +Common advice when working with `Data` is to index by adding to the start index, as in `data[data.startIndex + 4]`. However, even this approach is not valid in a generic context (even for random access collections). Fetching an index and then performing integer arithmetic is different than advancing a position: + +```swift +struct EveryOther: RandomAccessCollection { + internal var storage: C + + var startIndex: C.Index { storage.startIndex } + var endIndex: C.Index { + if storage.count % 2 == 0 { return storage.endIndex } + return storage.index(before: storage.endIndex) + } + + subscript(position: C.Index) -> C.Element { storage[position] } + + func index(before i: C.Index) -> C.Index { storage.index(i, offsetBy: -2) } + func index(after i: C.Index) -> C.Index { storage.index(i, offsetBy: 2) } + // ... and override `distance`, `index(_:offsetBy:)` for performance ... +} + +let everyOther = EveryOther(storage: [1,2,3,4,5,6,7,8]) +print(everyOther.contains(2)) // false +print(everyOther.contains(3)) // true + +let startIdx = everyOther.startIndex +print(everyOther[startIdx + 1]) // 2, but everyOther doesn't even contain 2! +print(everyOther[everyOther.index(after: startIdx)]) // 3 +``` + +This proposal provides a way to have offset-based element access for such collections, with similar expressivity but with explicit bounds for clarity. + + +## Proposed solution + +We propose convenient subscripts for slicing, single-element retrieval, and fetching an index from an offset: + +```swift +let str = "abcdefghijklmnopqrstuvwxyz" +print(str[.first + 3 ..< .first + 6]) // "def" +print(str[.first + 3 ..< .last - 2]) // "defghijklmnopqrstuvw" +print(str[.first + 3 ..< .last - 22]) // "", +print(str[.last]) // Optional("z") +print(str[.last - 1]) // Optional("y") +print(str[.first + 26]) // nil +print(str[(.last - 3)...]) // "wxyz" + +print(str.index(at: .last - 1)) // Optional(... index of "y") +print(str.index(at: .last - 25)) // Optional(... index of "a") +print(str.index(at: .last - 26)) // nil +``` + +The `parseRequirements` example from above can be written as: + +```swift +func parseRequirements(_ s: String) -> [(finish: Character, before: Character)] { + s.split(separator: "\n").map { line in + (line[.first + 5]!, line[.last - 11]!) + } +} +``` + +These APIs are available on all Collections, allowing a more general solution. The Advent of Code exercise only requires the extraction and comparison of the elements at the corresponding positions, so we can generalize to: + +```swift +func parseRequirements( + _ c: C, lineSeparator: C.Element +) -> [(finish: C.Element, before: C.Element)] where C.Element: Comparable { + c.split(separator: lineSeparator).map { line in + (line[.first + 5]!, line[.last - 11]!) + } +} +``` + +Here, the `line[.last - 11]` will run in constant-time if `c` conforms to `BidirectionalCollection`, or in linear-time if it does not (since we have to count from the front). This algorithmic guarantee is added as part of this proposal, without which this generalization cannot be done. + +These also address the expressivity issues and assumptions with Int-indexed Collections above: + +```swift +print((3..<10)[(.first + 5)...]) // 8..<10 + +func fifth(_ c: C) -> C.Element? { return c[.first + 4] } + +let array = [1,2,3,4,5,6,7,8,9] +print(fifth(array)!) // 5 +print(fifth(array[2...])!) // 7 + +var data = Data([1, 2, 3, 4, 5, 6, 7, 8, 9]) +print(fifth(data)!) // 5 +data = data.dropFirst(2) +print(fifth(data)!) // 7 + +let everyOther = EveryOther(storage: [1,2,3,4,5,6,7,8]) +print(everyOther[.first + 1]!) // 3 +``` + + + +## Detailed design + +This proposal adds an `OffsetBound` struct, representing a position at an offset from the start or end of a collection. + +```swift +/// A position in a collection specified as an offset from either the first +/// or last element of the collection (i.e. the collection's bounds). +/// +/// You can use an `OffsetBound` to access an index or element of a collection +/// as well as extract a slice. For example: +/// +/// let str = "abcdefghijklmnopqrstuvwxyz" +/// print(str[.last]) // Optional("z") +/// print(str[.last - 2]) // Optional("x") +/// print(str[.first + 26]) // nil +/// print(str[.first + 3 ..< .first + 6]) // "def" +/// print(str[.first + 3 ..< .last - 2]) // "defghijklmnopqrstuvw" +/// +/// `OffsetBound`s also provide a convenient way of working with slice types +/// over collections whose index type is `Int`. Slice types share indices with +/// their base collection, so `0` doesn't always mean the first element. For +/// example: +/// +/// let array = [1,2,3,4,5,6] +/// print(array[2...][3) // 4 +/// print(array[2...][.first + 3]!) // 6 +/// +public struct OffsetBound { + /* internally stores an enum, not ABI/API */ + + /// The position of the first element of a nonempty collection, corresponding + /// to `startIndex`. + public static var first: OffsetBound + + /// The position of the last element of a nonempty collection, corresponding + /// to `index(before: endIndex)`. + public static var last: OffsetBound + + /// Returns a bound that offsets the given bound by the specified distance. + /// + /// For example: + /// + /// .first + 2 // The position of the 3rd element + /// .last + 1 // One past the last element, corresponding to `endIndex` + /// + public static func +(_ lhs: OffsetBound, _ rhs: Int) -> OffsetBound + + /// Returns a bound that offsets the given bound by the specified distance + /// backwards. + /// + /// For example: + /// + /// .last - 2 // Two positions before the last element's position + /// + public static func -(_ lhs: OffsetBound, _ rhs: Int) -> OffsetBound +} +``` + + +`OffsetBound` is `Comparable`, and as such can be used as a bound type for `RangeExpression`s. + +```swift +extension OffsetBound: Comparable { + /// Compare the positions represented by two `OffsetBound`s. + /// + /// Offsets relative to `.first` are always less than those relative to + /// `.last`, as there are arbitrarily many offsets between the two + /// extremities. Offsets from the same bound are ordered by their + /// corresponding positions. For example: + /// + /// .first + n < .last - m // true for all values of n and m + /// .first + n < .first + m // equivalent to n < m + /// .last - n < .last - m // equivalent to n > m + /// + public static func < (_ lhs: OffsetBound, _ rhs: OffsetBound) -> Bool + + /// Compare two `OffsetBound`s to see if they represent equivalent positions. + /// + /// This is only true if both offset the same bound by the same amount. For + /// example: + /// + /// .first + n == .last - m // false for all values of n and m + /// .first + n == .first + m // equivalent to n == m + /// + public static func == (_ lhs: OffsetBound, _ rhs: OffsetBound) -> Bool +} +``` + + +`Collection` gets an API to retrieve an index from an `OffsetBound` if it exists, a subscript to retrieve an element from an `OffsetBound` if it exists, and a slicing subscript to extract a range. + +```swift +extension Collection { + /// Returns the corresponding index for the provided offset, if it exists, + /// else returns nil. + /// + /// - Complexity: + /// - O(1) if the collection conforms to `RandomAccessCollection`. + /// - O(*k*) where *k* is equal to the offset if the collection conforms to + /// `BidirectionalCollection`. + /// - O(*k*) if `position` is `.first + n` for any n, or `.last + 1`. + /// - Otherwise, O(*n*) where *n* is the length of the collection. + public func index(at position: OffsetBound) -> Index? + + /// Returns the corresponding element for the provided offset, if it exists, + /// else returns nil. + /// + /// Example: + /// + /// let abcs = "abcdefg" + /// print(abcs[.last]) // Optional("g") + /// print(abcs[.last - 2]) // Optional("e") + /// print(abcs[.first + 8]) // nil + /// + /// - Complexity: + /// - O(1) if the collection conforms to `RandomAccessCollection`. + /// - O(*k*) where *k* is equal to the offset if the collection conforms to + /// `BidirectionalCollection`. + /// - O(*k*) if `position` is `.first + n` for any n, or `.last + 1`. + /// - Otherwise, O(*n*) where *n* is the length of the collection. + public subscript(position: OffsetBound) -> Element? + + /// Returns the contiguous subrange of elements corresponding to the provided + /// offsets. + /// + /// Example: + /// + /// let abcs = "abcdefg" + /// print(abcs[.first + 1 ..< .first + 6]) // "bcdef" + /// print(abcs[.first + 1 ..< .last - 1]) // "bcde" + /// + /// - Complexity: + /// - O(1) if the collection conforms to `RandomAccessCollection`. + /// - O(*k*) where *k* is equal to the larger offset if the collection + /// conforms to `BidirectionalCollection`. + /// - O(*k*) if the offsets are `.first + n` for any n or `.last + 1`. + /// - Otherwise, O(*n*) where *n* is the length of the collection. + public subscript( + range: ORE + ) -> SubSequence where ORE.Bound == OffsetBound +} +``` + + +RangeReplaceableCollection gets corresponding APIs in terms of OffsetBound, as well as subscript setters. + +```swift +extension RangeReplaceableCollection { + /// Replaces the specified subrange of elements with the given collection. + /// + /// This method has the effect of removing the specified range of elements + /// from the collection and inserting the new elements at the same location. + /// The number of new elements need not match the number of elements being + /// removed. + /// + /// In this example, two characters in the middle of a string are + /// replaced by the three elements of a `Repeated` instance. + /// + /// var animals = "🐕🐈🐱🐩" + /// let dogFaces = repeatElement("🐶" as Character, count: 3) + /// animals.replaceSubrange(.first + 1 ... .last - 1, with: dogFaces) + /// print(animals) + /// // Prints "🐕🐶🐶🐶🐩" + /// + /// If you pass a zero-length range as the `subrange` parameter, this method + /// inserts the elements of `newElements` at `subrange.startIndex`. Calling + /// the `insert(contentsOf:at:)` method instead is preferred. + /// + /// Likewise, if you pass a zero-length collection as the `newElements` + /// parameter, this method removes the elements in the given subrange + /// without replacement. Calling the `removeSubrange(_:)` method instead is + /// preferred. + /// + /// Calling this method may invalidate any existing indices for use with this + /// collection. + /// + /// - Parameters: + /// - subrange: The subrange of the collection to replace, specified as + /// offsets from the collection's bounds. + /// - newElements: The new elements to add to the collection. + /// + /// - Complexity: O(*n* + *m*), where *n* is length of this collection and + /// *m* is the length of `newElements`. If the call to this method simply + /// appends the contents of `newElements` to the collection, the complexity + /// is O(*m*). + public mutating func replaceSubrange( + _ subrange: R, with newElements: __owned C + ) where C.Element == Element, R.Bound == OffsetBound + + /// Inserts a new element into the collection at the specified position. + /// + /// The new element is inserted before the element currently at the specified + /// offset. If you pass `.last + 1` as the `position` parameter, corresponding + /// to the collection's `endIndex`, the new element is appended to the + /// collection. + /// + /// var numbers = "12345" + /// numbers.insert("Ⅸ", at: .first + 1) + /// numbers.insert("𐄕", at: .last + 1) + /// + /// print(numbers) + /// // Prints "1Ⅸ2345𐄕" + /// + /// Calling this method may invalidate any existing indices for use with this + /// collection. + /// + /// - Parameter newElement: The new element to insert into the collection. + /// - Parameter `position`: The position at which to insert the new element, + /// specified as offsets from the collection's bounds + /// + /// - Complexity: O(*n*), where *n* is the length of the collection. If + /// `position == .last + 1`, this method is equivalent to `append(_:)`. + public mutating func insert( + _ newElement: __owned Element, at position: OffsetBound + ) + + /// Inserts the elements of a sequence into the collection at the specified + /// position. + /// + /// The new elements are inserted before the element currently at the + /// specified offset. If you pass `.last + 1` as the `position` parameter, + /// corresponding to the collection's `endIndex`, the new elements are + /// appended to the collection. + /// + /// Here's an example of inserting vulgar fractions in a string of numbers. + /// + /// var numbers = "12345" + /// numbers.insert(contentsOf: "↉⅖⅑", at: .first + 2) + /// print(numbers) + /// // Prints "12↉⅖⅑345" + /// + /// Calling this method may invalidate any existing indices for use with this + /// collection. + /// + /// - Parameter newElements: The new elements to insert into the collection. + /// - Parameter `position`: The position at which to insert the new elements, + /// specified as offsets from the collection's bounds + /// + /// - Complexity: O(*n* + *m*), where *n* is length of this collection and + /// *m* is the length of `newElements`. If `position == .last + 1`, this + /// method is equivalent to `append(contentsOf:)`. + public mutating func insert( + contentsOf newElements: __owned S, at position: OffsetBound + ) where S.Element == Element + + /// Removes and returns the element at the specified position, if it exists, + /// else returns nil. + /// + /// All the elements following the specified position are moved to close the + /// gap. + /// + /// Example: + /// var measurements = [1.2, 1.5, 2.9, 1.2, 1.6] + /// let removed = measurements.remove(at: .last - 2) + /// print(measurements) + /// // Prints "[1.2, 1.5, 1.2, 1.6]" + /// print(measurements.remove(at: .first + 4)) + /// // Prints nil + /// + /// Calling this method may invalidate any existing indices for use with this + /// collection. + /// + /// - Parameter position: The position of the element to remove, specified as + /// an offset from the collection's bounds. + /// - Returns: The removed element if it exists, else nil + /// + /// - Complexity: O(*n*), where *n* is the length of the collection. + public mutating func remove(at position: OffsetBound) -> Element? + + /// Removes the elements in the specified subrange from the collection. + /// + /// All the elements following the specified position are moved to close the + /// gap. This example removes two elements from the middle of a string of + /// rulers. + /// + /// var rulers = "📏🤴👑📐" + /// rulers.removeSubrange(.first + 1 ... .last - 1) + /// print(rulers) + /// // Prints "📏📐" + /// + /// Calling this method may invalidate any existing indices for use with this + /// collection. + /// + /// - Parameter range: The range of the collection to be removed, specified + /// as offsets from the collection's bounds. + /// + /// - Complexity: O(*n*), where *n* is the length of the collection. + public mutating func removeSubrange( + _ range: R + ) where R.Bound == OffsetBound + + /// Accesses the element corresponding to the provided offset. If no element + /// exists, `nil` is returned from the getter. Similarly, setting an element + /// to `nil` will remove the element at that offset. + /// + /// Example: + /// + /// let abcs = "abcdefg" + /// print(abcs[.last]) // Optional("g") + /// print(abcs[.last - 2]) // Optional("e") + /// print(abcs[.first + 8]) // nil + /// abcs[.first + 2] = "©" + /// print(abcs) // "ab©defg" + /// abcs[.last - 1] = nil + /// print(abcs) // "ab©deg" + /// + /// - Complexity (get): + /// - O(1) if the collection conforms to `RandomAccessCollection`. + /// - O(*k*) where *k* is equal to the offset if the collection conforms to + /// `BidirectionalCollection`. + /// - O(*k*) if `position` is `.first + n` for any n, or `.last + 1`. + /// - Otherwise, O(*n*) where *n* is the length of the collection. + /// + /// - Complexity (set): + /// - O(*n*) where *n* is the length of the collection. + public subscript(position: OffsetBound) -> Element? { get set } + + /// Accesses the contiguous subrange of elements corresponding to the provided + /// offsets. + /// + /// Example: + /// + /// var abcs = "abcdefg" + /// print(abcs[.first + 1 ..< .first + 6]) // "bcdef" + /// print(abcs[.first + 1 ..< .last - 1]) // "bcde" + /// abcs[.first ... .first + 3] = "🔡" + /// print(abcs) // "🔡efg" + /// + /// - Complexity (get): + /// - O(1) if the collection conforms to `RandomAccessCollection`. + /// - O(*k*) where *k* is equal to the larger offset if the collection + /// conforms to `BidirectionalCollection`. + /// - O(*k*) if the offsets are `.first + n` for any n or `.last + 1`. + /// - Otherwise, O(*n*) where *n* is the length of the collection. + /// + /// - Complexity (set): + /// - O(*n*) where *n* is the length of the collection. + public subscript( + range: ORE + ) -> SubSequence where ORE.Bound == OffsetBound { get set } +``` + + +This proposal adds a new “internal” (i.e. underscored) customization hook to `Collection` to apply a reverse offset to a given index. Unlike `index(_:offsetBy:limitedBy:)` which will trap if the collection is not bidirectional, this will instead advance `startIndex`. + +```swift +public protocol Collection: Sequence { + /// Returns an index `distance` positions prior to `i` if it exists. + /// + /// Other methods such as `index(_:offetBy:)` must not be passed a negative + /// offset if the collection is bidirectional. This method will perform a + /// negative offset even if the collection is not bidirectional, by using a + /// less efficient means. `BidirectionalCollection` customizes this with a + /// more efficient implementation. + /// + /// - Parameters + /// - i: a valid index of the collection. + /// - distance: The distance to offset `i` backwards. `distance` must be + /// positive or zero. + /// - Returns: The index `distance` positions prior to `i` if in bounds, else + /// `nil`. + /// + /// - Complexity: + /// - O(1) if the collection conforms to `RandomAccessCollection`. + /// - O(*k*), where *k* is equal to `distance` if the collection conforms + /// to `BidirectionalCollection`. + /// - Otherwise, O(*n*), where *n* is the length of the collection. + func _reverseOffsetIndex(_ i: Index, by distance: Int) -> Index? +} + +extension Collection { + // Scans from the start + public func _reverseOffsetIndex(_ i: Index, by distance: Int) -> Index? +} +extension BidirectionalCollection { + // Reverse offsets + public func _reverseOffsetIndex(_ i: Index, by distance: Int) -> Index? +} +``` + + +Change `Collection.suffix()` and `Collection.dropLast()` to use this hook, which will improve algorithmic complexity in generic code when the collection happens to be bidirectional. + +```diff +extension Collection { + /// Returns a subsequence containing all but the specified number of final + /// elements. + /// + /// If the number of elements to drop exceeds the number of elements in the + /// collection, the result is an empty subsequence. + /// + /// let numbers = [1, 2, 3, 4, 5] + /// print(numbers.dropLast(2)) + /// // Prints "[1, 2, 3]" + /// print(numbers.dropLast(10)) + /// // Prints "[]" + /// + /// - Parameter k: The number of elements to drop off the end of the + /// collection. `k` must be greater than or equal to zero. + /// - Returns: A subsequence that leaves off the specified number of elements + /// at the end. + /// +- /// - Complexity: O(1) if the collection conforms to +- /// `RandomAccessCollection`; otherwise, O(*n*), where *n* is the length of +- /// the collection. ++ /// - Complexity: ++ /// - O(1) if the collection conforms to `RandomAccessCollection`. ++ /// - O(*k*), where *k* is equal to `distance` if the collection conforms ++ /// to `BidirectionalCollection`. ++ /// - Otherwise, O(*n*), where *n* is the length of the collection. + @inlinable + public __consuming func dropLast(_ k: Int = 1) -> SubSequence + + /// Returns a subsequence, up to the given maximum length, containing the + /// final elements of the collection. + /// + /// If the maximum length exceeds the number of elements in the collection, + /// the result contains all the elements in the collection. + /// + /// let numbers = [1, 2, 3, 4, 5] + /// print(numbers.suffix(2)) + /// // Prints "[4, 5]" + /// print(numbers.suffix(10)) + /// // Prints "[1, 2, 3, 4, 5]" + /// + /// - Parameter maxLength: The maximum number of elements to return. The + /// value of `maxLength` must be greater than or equal to zero. + /// - Returns: A subsequence terminating at the end of the collection with at + /// most `maxLength` elements. + /// +- /// - Complexity: O(1) if the collection conforms to +- /// `RandomAccessCollection`; otherwise, O(*n*), where *n* is the length of +- /// the collection. ++ /// - Complexity: ++ /// - O(1) if the collection conforms to `RandomAccessCollection`. ++ /// - O(*k*), where *k* is equal to `maxLength` if the collection conforms ++ /// to `BidirectionalCollection`. ++ /// - Otherwise, O(*n*), where *n* is the length of the collection. + @inlinable + public __consuming func suffix(_ maxLength: Int) -> SubSequence +} +``` + +Finally, `BidirectionalCollection`’s overloads are obsoleted in new versions as they are fully redundant with `Collection`’s. + +## Source compatibility + +This change preserves source compatibility. + +## Effect on ABI stability + +This does not change any existing ABI. + +## Effect on API resilience + +All additions are versioned. The `_reverseOffsetIndex` customization hook is `@inlinable`, while all other added ABI are fully resilient. + +## Alternatives considered + +### Add a `.start` and `.end` + +The previous version of this pitch also had a `.start` and `.end`, where `.start == .first && .end == .last + 1`. Having `.end` made it easier to refer to an open upper bound corresponding to the collection’s `endIndex`, which mostly manifests in documentation. However, in actual usage, `.first` and `.last` are almost always clearer. + +We chose to eschew `.start` and `.end` members, simplifying the programming model and further distinguishing `OffsetBound` as an element-position abstraction more akin to working with `Collection.first/last` than `Collection.startIndex/endIndex`. + + +### Don’t add the customization hook + +An alternative to the customization hook is adding overloads to both `Collection` and `BidirectionalCollection` for every API. This would result in slower code in a generic context over `Collection` even if that collection happened to also conform to `BidirectionalCollection`, as this is statically dispatched. Since this is a frequent enough access pattern in the standard library, we feel a customization hook to share and accelerate the functionality is warranted. + +This proposal adapts Collection’s `suffix` and `dropLast` to use the new hook, improving their performance as well. + + +### Offset Arbitrary Indices + +More general than offsetting from the beginning or end of a Collection is offsetting from a given index. Relative indices, that is indices with offsets applied to them, could be expressed in a `RelativeBound` struct, where `Bound` is the index type of the collection it will be applied to (phantom-typed for pure-offset forms). The prior pitch proposed this feature alongside the `++`/`--` operators, but doing this presents problems in `RangeExpression` conformance as well as type-checker issues with generic operator overloading. + +These issues can be worked around (as shown below), but each workaround comes with its own drawbacks. All in all, offsetting from an arbitrary index isn’t worth the tradeoffs and can be mimicked with slicing (albeit with more code). + +#### The Trouble with Generic Operator Overloading + +Overloading an operator for a type with a generic parameter complicates type checking for that operator. The type checker has to open a type variable and associated constraints for that parameter, preventing the constraint system from being split. This increases the complexity of type checking *all* expressions involving that operator, not just those using the overload. + +This increased complexity may be tolerable for a brand new operator, such as the previously pitched `++` and `--`, but it is a downside of overloading an existing-but-narrowly-extended operator such as `..<` and `...`. It is a total non-starter for operators such as `+` and `-`, which already have complex resolution spaces. + +#### The Trouble with RangeExpression + +`RangeExpression` requires the ability to answer whether a range contains a bound, where `Bound` is constrained to be `Comparable`. Pure-offset ranges can answer this similarly to other partial ranges by putting the partial space between `.first` and `.last`. + +Unlike pure-offset ranges, containment cannot be consistently answered for a range of relative indices without access to the original collection. Similarly, `RelativeBound` cannot be comparable. + +A workaround could be to introduce a new protocol `IndexRangeExpression` without these requirements and add new overloads for it. Some day in the future when the compiler supports it, `RangeExpression` can conform to it and the existing `RangeExpression` overloads would be deprecated. + +This would also require a new `RelativeRange` type and new generic overloads for range operators `..<` and `...` producing it. This is a hefty amount of additional machinery and would complicate type checking of all ranges, as explained next. + + +### Other Syntax + +#### `idx++2` or `idx-->2` + +The original pitch introduced `++` and `--` whose left-hand side could be omitted. Alternatives included symmetric `-->` and `<--`. This syntax was used for alternatives that offset existing indices, omitting a side for an implied start or end, and thus did not introduce a generic overload of `+`. However, in addition to the issues mentioned above in “Offset Arbitrary Indices”, these operators were met with considerable resistance: `++/--` carries C-baggage for many developers, and `++/--` and `-->/<--` are both new glyphs that don’t carry their weight when limited to the start or end of a collection. + +#### Use an `offset:` label and literal convention + +An alternative syntax (prototyped in [this gist](https://gist.github.com/milseman/7f7cf3b764618ead6011700fdce2ad83)) is to have `OffsetBound` be `ExpressibleByIntegerLiteral` with the convention that a negative literal produces an offset from the end. E.g.: + +```swift +// Proposed +"abc"[.first + 1 ..< .last] // "b" + +// Offset label +"abc"[offset: 1..<(-1)] // "b" +``` + +Negative values meaning from-the-end is only a convention on top of literals, i.e. this would *not* provide wrap-around semantics for a value that happens to be negative. + +In the end, we believe that the proposed syntax is more readable. There is less cognitive load in reading `.last` than mentally mapping the literal convention of `-1`, and that literal convention would be inconsistent with run-time values. + + +#### No Syntax + +Alternative approaches include avoiding any syntax such as `+` or the use of the range operators `..<` and `...` by providing labeled subscript overloads for all combinations. + +```swift +// Proposed +collection[.first + 5 ..< .last - 1] + +// No ranges +collection[fromStart: 5, upToEndOffsetBy: -2] +``` + +We feel range-less variants are not in the direction and spirit of Swift. Swift uses range syntax for subscripts rather than multiple parameter labels: + +```swift +// Existing Swift style +collection[lowerIdx ..< upperIdx] // Up-to +collection[lowerIdx ... upperIdx] // Up-through + +// Not Swift style +collection[from: lowerIdx, upTo: upperIdx] +collection[from: lowerIdx, upThrough: upperIdx] +``` + + + +### Don’t Make it Easy + +One objection to this approach is that it makes it less obnoxious to write poorly performing formulations of simple iteration patterns. Advancing the start index in every iteration of a loop can be a quadratic formulation of an otherwise linear algorithm. Currently, writing the quadratic formulation requires significantly more complex and unwieldy code compared to efficient approaches. The fear is that ergonomic improvements to legitimate use cases in this pitch would also improve the ergonomics of inefficient code. + +```swift +// Linear, for when you want `element` +for element in collection { ... } + +// Linear, for when you want `idx` and `element` +for idx in collection.indices { + let element = collection[idx] + ... +} + +// Linear time and linear space, for when you want `element`, `i`, and random-access to `indices`. +let indices = Array(collection.indices) +for i in 0.. Element` must be passed a valid index less than `endIndex`. + +A violation of the above requirements leading to a potential memory safety issue causes a trap. I.e. such violations are [irrecoverable logic errors](https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst#logic-failures) and the process is safely taken down. + +Additionally, such low-level index-manipulation-heavy code is most often written within a context where indices are known to be valid, e.g. because the indices were vended by the collection itself. Subscript taking an index is similarly non-optional, as an index is an assumed-valid “key” to a specific element. Swift, being a memory-safe language, will still bounds-check the access, but this is not surfaced to the programmer’s code and bounds checks can be eliminated by the optimizer if safe to do so. Again, invalid indices for these operations represent irrecoverable logic errors, so a trap is issued. + +#### Optional Returning APIs + +In contrast, higher-level operations are often written in a context where the existence of a desired element is not known. Whether there is or is not such an element represents cases that should be handled explicitly by code, most often through the use of an optional result. That is, non-existence is not a logic error but a simple domain error, for which Optional is [the best tool for the job](https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst#simple-domain-errors). + +For example, Dictionary has the regular `subscript(_:Index) -> Element` trapping subscript for indices derived from the dictionary itself, but additionally provides a `subscript(_:Key) -> Value?`, which returns `nil` if the key is not present in the dictionary. A missing key is not necessarily a logic error, just another case to handle in code. Similarly, `first`, `last`, `.randomElement()`, `min()`, etc., all return optionals, where emptiness is not necessarily a logic error, but a case to be handled in code. + +Optional handling is a strength of Swift and known-valid access is an acceptable use of `!`, which has the effect of converting a simple domain error into an irrecoverable logic error. + +Single element offset-based subscripting follows this pattern by returning `nil` if the offset is not valid for the collection. While this does introduce optional-handling code at the use site, handling such access would be necessary anyways outside of a context where the offset is known to be less than the count. Otherwise, in these contexts, the programmer would have to remember to guard a trapping subscript’s use site by an explicit check against `count`. Offset ranges are clamped, resulting in empty slice return values, just like other partial ranges. + +This means that `collection.last` matches `collection.index(at: .last)` and `collection[.last]` in optionality, and the latter two can be offset. + +#### Example of Known-Valid Offset Context + +For an example of a known-valid-offset context highlighting the differences between the lowest level and higher level APIs, consider the below binary-search algorithm on Collection. This formulation is constant in space and linear in indexing complexity if the collection is not random-access (and logarithmic if it is). Fetching `idx` in the loop body could be written using either `index(_:offsetBy:)`, `index(atOffset:)`, or `index(_:offsetBy:limitedBy:)`, all of which have the same complexity. However, `index(_:offsetBy)` is the lowest-level API available and assumes the given offset is valid (a trap will be issued only for those misuses that lead to a memory safety violation). That is, it treats an invalid offset as a logic error and doesn’t surface this distinction in code, unlike the higher level `index(atOffset:)` and `index(_:offsetBy:limitedBy)` APIs which return optionals. This allows its implementation to skip branches checking the past-the-end condition on every loop iteration, and it allows the caller to skip a branch checking if there was a result. It is unreasonable (at least in general) to expect an optimizer to perform this transformation automatically based on scalar evolution of `count` and an understanding of the implementation of the customizable indexing APIs (which may be dynamically dispatched). + +```swift +extension Collection { + func binarySearch(for element: Element) -> Index? { + assert(self.elementsEqual(self.sorted()), "only valid if sorted") + + var slice = self[...] + var count = self.count // O(n) if non-RAC + while count > 1 { + defer { assert(slice.count == count) } + + let middle = count / 2 + + // Either of the below formulations sum to a total of O(n) index + // advancement operations across all loop iterations if non-RAC. + let idx = slice.index(slice.startIndex, offsetBy: middle) + // let idx = slice.index(at: .first + middle)! + // let idx = slice.index( + // slice.startIndex, offsetBy: middle, limitedBy: slice.endIndex)! + + let candidate = self[idx] + if candidate == element { return idx } + if candidate < element { + slice = slice[idx...] + count = count &- middle // because division truncates + } else { + slice = slice[.. Bool { + return lhs.rawValue < rhs.rawValue + } +} +``` + +* Manually implementing the `<` operator with a private `minimum(_:_:)` helper function. This is the “proper” implementation, but is fairly verbose and error-prone to write, and does not scale well with more enumeration cases. + +```swift +enum Brightness: Comparable { + case low + case medium + case high + + private static func minimum(_ lhs: Self, _ rhs: Self) -> Self { + switch (lhs, rhs) { + case (.low, _), (_, .low ): + return .low + case (.medium, _), (_, .medium): + return .medium + case (.high, _), (_, .high ): + return .high + } + } + + static func < (lhs: Self, rhs: Self) -> Bool { + return (lhs != rhs) && (lhs == Self.minimum(lhs, rhs)) + } +} +``` + +As the second workaround is non-obvious to many, users also often attempt to implement “private” integer values for enumeration cases by manually numbering them. Needless to say, this approach scales very poorly, and incurs a high code maintenance cost as simple tasks like adding a new enumeration case require manually re-numbering all the other cases. Workarounds for the workaround, such as numbering by tens (to “make room” for future cases) or using `Double` as the key type (to allow [numbering “on halves”](https://youtu.be/KWcxgrg4eQI?t=113)) reflect poorly on the language. + +```swift +enum Membership: Comparable { + case premium + case preferred + case general + + private var comparisonValue: Int { + switch self { + case .premium: + return 0 + case .preferred: + return 1 + case .general: + return 2 + } + } + + static func < (lhs: Self, rhs: Self) -> Bool { + return lhs.comparisonValue < rhs.comparisonValue + } +} +``` + +## Proposed solution + +Enumeration types which opt-in to a synthesized `Comparable` conformance would compare according to case declaration order, with later cases comparing greater than earlier cases. Only `enum` types with no associated values and `enum` types with only `Comparable` associated values would be eligible for synthesized conformances. The latter kind of `enum`s will compare by case declaration order first, and then lexicographically by payload values. No `enum` types with raw values would qualify. + +While basing behavior off of declaration order is unusual for Swift, as we generally hew to the “all fields are reorderable by the compiler” principle, it is not a foreign concept to `enums`. For example, reordering cases in a numeric-backed raw `enum` already changes its runtime behavior, since the case declaration order is taken to be meaningful in that context. I also believe that `enum` cases and `struct`/`class` fields are sufficiently distinct concepts that making enumeration case order meaningful would not make the language incoherent. + +Later cases will compare greater than earlier cases, as Swift generally views sort orders to be “ascending” by default. It also harmonizes with the traditional C/C++ paradigm where a sequence of enumeration cases is merely a sequence of incremented integer values. + +## Detailed design + +Synthesized `Comparable` conformances for eligible types will work exactly the same as synthesized `Equatable`, `Hashable`, and `Codable` conformances today. A conformance will not be synthesized if a type is ineligible (has raw values or non recursively-conforming associated values) or already provides an explicit `<` implementation. + +```swift +enum Membership: Comparable { + case premium(Int) + case preferred + case general +} + +([.preferred, .premium(1), .general, .premium(0)] as [Membership]).sorted() +// [Membership.premium(0), Membership.premium(1), Membership.preferred, Membership.general] +``` + +## Source compatibility + +This feature is strictly additive. + +## ABI compatibility + +This feature does not affect the ABI. + +## API stability + +This feature does not affect the standard library. + +## Alternatives considered + +* Basing comparison order off of raw values or `RawRepresentable`. This alternative is inapplicable, as enumerations with “raw” representations don’t always have an obvious sort order anyway. Raw `String` backings are also commonly (ab)used for debugging and logging purposes making them a poor source of intent for a comparison-order definition. + +```swift +enum Month: String, Comparable { + case january + case february + case march + case april + ... +} +// do we compare alphabetically or by declaration order? +``` diff --git a/proposals/0267-where-on-contextually-generic.md b/proposals/0267-where-on-contextually-generic.md new file mode 100644 index 0000000000..812fb39da5 --- /dev/null +++ b/proposals/0267-where-on-contextually-generic.md @@ -0,0 +1,99 @@ +# `where` clauses on contextually generic declarations + +* Proposal: [SE-0267](0267-where-on-contextually-generic.md) +* Author: [Anthony Latsis](https://github.com/AnthonyLatsis) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift#23489](https://github.com/apple/swift/pull/23489) +* Decision Notes: [Review](https://forums.swift.org/t/se-0267-where-clauses-on-contextually-generic-declarations/30051/49), [Announcement](https://forums.swift.org/t/accepted-se-0267-where-clauses-on-contextually-generic-declarations/30474) + +## Introduction + +This proposal aims to lift the restriction on attaching `where` clauses to member declarations that can reference only outer generic parameters. Simply put, this means the `'where' clause cannot be attached` error will be relaxed for most declarations nested inside generic contexts: + +```swift +struct Box { + func boxes() -> [Box] where Wrapped: Sequence { ... } +} + +``` + +> The current proposal only covers member declarations that already support a generic parameter list, that is, properties and unsupported constraints on protocol requirements are out of scope. +> To illustrate, the following remains invalid: +> ```swift +> protocol P { +> // error: Instance method requirement 'foo(arg:)' cannot add constraint 'Self: Equatable' on 'Self' +> func foo() where Self: Equatable +> } +> +> class C { +> // error: type 'Self' in conformance requirement does not refer to a generic parameter or associated type +> func foo() where Self: Equatable +> } +> ``` +> Whereas placing constraints on an extension member rather than the extension itself becomes possible: +> ```swift +> extension P { +> func bar() where Self: Equatable { ... } +> } +> ``` + +[Swift-evolution thread](https://forums.swift.org/t/where-clauses-on-contextually-generic-declaractions/22449) + +## Motivation + +Today, a contextual where clause on a member declaration can only be expressed indirectly by placing the member inside a dedicated constrained extension. Unless constraints are identical, every such member requires a separate extension. +This lack of flexibility is a potential obstacle to grouping semantically related APIs, stacking up constraints, and the legibility of heavily generic interfaces. + +It is reasonable to expect a `where` clause to "work" anywhere a constraint is meaningful, which is to say both of these structuring styles should be available to the user: + +```swift +// 'Foo' can be any kind of nominal type declaration. +// For a protocol, 'T' would be Self or an associatedtype. +struct Foo + +extension Foo where T: Sequence, T.Element: Equatable { + func slowFoo() { ... } +} +extension Foo where T: Sequence, T.Element: Hashable { + func optimizedFoo() { ... } +} +extension Foo where T: Sequence, T.Element == Character { + func specialCaseFoo() { ... } +} + +extension Foo where T: Sequence, T.Element: Equatable { + func slowFoo() { ... } + + func optimizedFoo() where T.Element: Hashable { ... } + + func specialCaseFoo() where T.Element == Character { ... } +} +``` +A step towards generalizing `where` clause use is an obvious and farsighted improvement to the generics +system with numerous future applications, including generic properties, [opaque types](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0244-opaque-result-types.md), [generalized +existentials](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#generalized-existentials) and constrained protocol requirements. + +## Detailed design +### Overrides + +Because contextual `where` clauses are effectively visibility constraints, overrides adopting this feature must be at least as visible as the overridden method: + +```swift +class Base { + func foo() where T == Int { ... } +} + +class Derived: Base { + // OK, the substitutions for are a superset of those for + override func foo() where T: Equatable { ... } +} +``` + +## Source compatibility and ABI stability + +This is an additive change with no impact on the ABI and existing code. + +## Effect on API resilience + +For public declarations in resilient libraries such as the Standard Library, moving a constraint from an extension to a member and vice versa will break the ABI due to subtle mangling differences as of the current implementation. diff --git a/proposals/0268-didset-semantics.md b/proposals/0268-didset-semantics.md new file mode 100644 index 0000000000..782c01c1d9 --- /dev/null +++ b/proposals/0268-didset-semantics.md @@ -0,0 +1,192 @@ +# Refine `didSet` Semantics + +* Proposal: [SE-0268](0268-didset-semantics.md) +* Author: [Suyash Srijan](https://github.com/theblixguy) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift#26632](https://github.com/apple/swift/pull/26632) +* Bug: [SR-5982](https://bugs.swift.org/browse/SR-5982) + +## Introduction + +Introduce two changes to `didSet` semantics - + +1. If a `didSet` observer does not reference the `oldValue` in its body, then the call to fetch the `oldValue` will be skipped. We refer to this as a "simple" didSet. +2. If we have a "simple" `didSet` and no `willSet`, then we could allow modifications to happen in-place. + +Swift-evolution thread: [didSet Semantics](https://forums.swift.org/t/pitch-didset-semantics/27858) + +## Motivation + +Currently, Swift always calls the property's getter to get the `oldValue` if we have a `didSet` observer, even if the observer does not refer to the `oldValue` in its body. For example: + +```swift +class Foo { + var bar: Int { + didSet { print("didSet called") } + } + + init(bar: Int) { self.bar = bar } +} + +let foo = Foo(bar: 0) +// This calls the getter on 'bar' to get +// the 'oldValue', even though we never +// refer to the oldValue inside bar's 'didSet' +foo.bar = 1 +``` + +This might look harmless, but it is doing redundant work (by allocating storage and loading a value which isn't used). It could also be expensive if the getter performs some non-trivial task and/or returns a large value. + +For example: + +```swift +struct Container { + var items: [Int] = .init(repeating: 1, count: 100) { + didSet { + // Do some stuff, but don't access oldValue + } + } + + mutating func update() { + for index in 0.. { + var wrappedValue: Value { + get { + guard let value = value else { + preconditionFailure("Property \(String(describing: self)) has not been set yet") + } + return value + } + + set { + guard value == nil else { + preconditionFailure("Property \(String(describing: self)) has already been set") + } + value = newValue + } + } + + var value: Value? +} + +class Foo { + @Delayed var bar: Int { + didSet { print("didSet called") } + } +} + +let foo = Foo() +foo.bar = 1 +``` + +However, this code will currently crash when we set `bar`'s value to be `1`. This is because Swift will fetch the `oldValue`, which is `nil` initially and thus will trigger the precondition in the getter. + +## Proposed Solution + +The property's getter is no longer called if we do not refer to the `oldValue` inside the body of the `didSet`. + +```swift +class Foo { + var bar = 0 { + didSet { print("didSet called") } + } + + var baz = 0 { + didSet { print(oldValue) } + } +} + +let foo = Foo() +// This will not call the getter to fetch the oldValue +foo.bar = 1 +// This will call the getter to fetch the oldValue +foo.baz = 2 +``` + +This applies to a `didSet` on an overridden property as well - the call to the superclass getter will be skipped if the `oldValue` is not referenced in the body of the overridden property's `didSet`. + +This also resolves some pending bugs such as [SR-11297](https://bugs.swift.org/browse/SR-11297) and [SR-11280](https://bugs.swift.org/browse/SR-11280). + +As a bonus, if the property has a "simple" `didSet` and no `willSet`, then we could allow for modifications to happen in-place. For example: + +```swift +// This is how we currently synthesize the _modify coroutine +_modify { + var newValue = underlyingStorage + yield &newValue + // Call the setter, which then calls + // willSet (if present) and didSet + observedStorage = newValue +} + +// This is how we're going to synthesize it instead +_modify { + // Since we don't have a willSet and + // we have a "simple" didSet, we can + // yield the storage directly and + // call didSet + yield &underlyingStorage + didSet() +} +``` + +This will provide a nice performance boost in some cases (for example, in the earlier array copying example). + +## Source compatibility + +This does not break source compatibility, _unless_ someone is explicitly relying on the current buggy behavior (i.e. the property's getter being called even if the `oldValue` isn't referenced). However, I think the possibility of that is very small. + +It would still be possible to preserve the old behavior by either: + +1. Explicitly providing the `oldValue` argument to `didSet`: +```swift +didSet(oldValue) { + // The getter is called to fetch + // the oldValue, even if it's not + // used in this body. +} +``` +2. Forcing the getter to be called by simply ignoring its value in the body of the `didSet`: +```swift +didSet { + // Calls the getter, but the value + // is ignored. + _ = oldValue +} +``` + +## Effect on ABI stability + +This does not affect the ABI as observers are not a part of it. + +## Effect on API resilience + +This does not affect API resilience - library authors can freely switch between a `didSet` which does not refer to the `oldValue` in its body and one which does and freely add or remove `didSet` from the property. + +## Alternatives considered + +- Explicitly require an `oldValue` parameter to use it, such as `didSet(oldValue) { ... }`, otherwise it is an error to use `oldValue` in the `didSet` body. This will be a big source breaking change. It will also cause a regression in usability and create an inconsistency with other accessors, such as `willSet` or `set`, which can be declared with or without an explicit parameter. The source compatibility problem can be mitigated by deprecating the use of implicit `oldValue` and then making it an error in the next language version, however the usability regression would remain. +- Introduce a new `didSet()` syntax that will suppress the read of the `oldValue` (and it will be an error to use `oldValue` in the `didSet` body). This will prevent any breakage since it's an additive change, but will reduce the positive performance gain (of not calling the getter when `oldValue` is not used) to zero unless people opt-in to the new syntax. Similar to the previous solution, it will create an inconsistency in the language, since it will be the only accessor that can be declared with an empty parameter list and will become yet another thing to explain to a newcomer. +- Leave the existing behavior as is. + +## Future Directions + +We can apply the same treatment to `willSet` i.e. not pass the `newValue` if it does not refer to it in its body, although it wouldn't provide any real benefit as not passing `newValue` to `willSet` does not avoid anything, where as not passing `oldValue` to `didSet` avoids loading it. + +We can also deprecate the implicit `oldValue` and request users to explicitly provide `oldValue` in parenthesis (`didSet(oldValue) { ... }`) if they want to use it in the body of the observer. This will make the new behavior more obvious and self-documenting. diff --git a/proposals/0269-implicit-self-explicit-capture.md b/proposals/0269-implicit-self-explicit-capture.md new file mode 100644 index 0000000000..272a4386ab --- /dev/null +++ b/proposals/0269-implicit-self-explicit-capture.md @@ -0,0 +1,195 @@ +# Increase availability of implicit `self` in `@escaping` closures when reference cycles are unlikely to occur + +* Proposal: [SE-0269](0269-implicit-self-explicit-capture.md) +* Author: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift#23934](https://github.com/apple/swift/pull/23934) +* Bug: [SR-10218](https://bugs.swift.org/browse/SR-10218) +* Forum threads: [Discussion](https://forums.swift.org/t/review-capture-semantics-of-self/22017), [Pitch](https://forums.swift.org/t/allow-implicit-self-in-escaping-closures-when-self-is-explicitly-captured/22590), [Review thread](https://forums.swift.org/t/se-0269-increase-availability-of-implicit-self-in-escaping-closures-when-reference-cycles-are-unlikely-to-occur/30376) + +## Introduction + +Modify the rule that all uses of `self` in escaping closures must be explicit by allowing for implicit uses of `self` in situations where the user has already made their intent explicit, or where strong reference cycles are otherwise unlikely to occur. There are two situations covered by this proposal. The first is when the user has explicitly captured `self` in the closure's capture list, so that the following would compile without error: + +```swift +class Test { + var x = 0 + func execute(_ work: @escaping () -> Void) { + work() + } + func method() { + execute { [self] in + x += 1 + } + } +} +``` + +Secondly, this proposal would make implicit `self` available in escaping closures when `self` is a value type, so that the following would become valid: + +```swift +struct Test { + var x = 0 + func execute(_ work: @escaping () -> Void) { + work() + } + func method() { + execute { + print(x) + } + } +} +``` + +## Motivation + +In order to prevent users from inadvertently creating retain cycles, the Swift compiler today requires all uses of `self` in escaping closures to be explicit. Attempting to reference a member `x` of `self` without the `self` keyword gives the error: + +``` +error: reference to property 'x' in closure requires explicit 'self.' to make capture semantics explicit +``` + +In codebases that choose to omit `self` where possible, this can result in a lot of unwanted noise, if many properties of `self` are used in a row: + +```swift +execute { + let foo = self.doFirstThing() + performWork(with: self.bar) + self.doSecondThing(with: foo) + self.cleanup() +} +``` + +It also results in a lot of unnecessary repetition. The motivation for requiring explicit usage of `self` is to force the user to make the intent to capture `self` explicit, but that goal is accomplished after the first explicit usage of `self` and is not furthered by any of the subsequent usages. + +In codebases that make heavy use of asynchronous code, such as clients of [PromiseKit](https://github.com/mxcl/PromiseKit), or even Apple's new [Combine](https://developer.apple.com/documentation/combine) and [SwiftUI](https://developer.apple.com/documentation/swiftui) libraries, maintainers must either choose to adopt style guides which require the use of explicit `self` in all cases, or else resign to the reality that explicit usage of `self` will appear inconsistently throughout the code. Given that Apple's preferred/recommended style is to omit `self` where possible (as evidenced by examples throughout [The Swift Programming Language](https://docs.swift.org/swift-book/) and the [SwiftUI Tutorials](https://developer.apple.com/tutorials/swiftui)), having any asynchronous code littered with `self.` is a suboptimal state of affairs. + +The error is also overly conservative—it will prevent the usage of implicit `self` even when it is very unlikely that the capture of `self` will somehow cause a reference cycle. With function builders in SwiftUI, the automatic resolution to the "make capture semantics explicit" error (and indeed, the fix-it that is currently supplied to the programmer) is just to slap a `self.` on wherever the compiler tells you to. While this is likely fine when `self` is a value type, building this habit could cause users to ignore the error and apply the fix-it in cases where it is not in fact appropriate. + +There are also cases that are just as (if not more) likely to cause reference cycles that the tool currently misses. Once such instance is discussed in **Future Directions** below. These false negatives are not addressed in this proposal, but improving the precision of this diagnostic will make the tool more powerful and less likely to be dismissed if (or when) new diagnostic cases are introduced. + +## Proposed solution + +First, allow the use of implicit `self` when it appears in the closure's capture list. The above code could then be written as: + +```swift +execute { [self] in + let foo = doFirstThing() + performWork(with: bar) + doSecondThing(with: foo) + cleanup() +} +``` + +This change still forces code which captures `self` to be explicit about its intentions, but reduces the cost of that explicitness to a single declaration. With this change explicit capture of `self` would be one of *two* ways to get rid of the error, with the current method of adding `self.` to each property/method access (without adding `self` to the capture list) remaining as a viable option. + +The compiler would also offer an additional fix-it when implicit `self` is used: + +```swift +execute { // <- Fix-it: capture 'self' explicitly to enable implicit 'self' in this closure. Fix-it: insert '[self] in' + let foo = doFirstThing() + performWork(with: bar) + doSecondThing(with: foo) + cleanup() +} +``` + +Second, if `self` is a value type, we will not require any explicit usage of `self` (at the call/use site or in the capture list), so that if `self` were a `struct` or `enum` then the above could be written as simply: + +```swift +execute { + let foo = doFirstThing() + performWork(with: bar) + doSecondThing(with: foo) + cleanup() +} +``` + +## Detailed design + +Whenever `self` is declared explicitly in an escaping closure's capture list, or its type is a value type, any code inside that closure can use names which resolve to members of the enclosing type, without specifying `self.` explicitly. In nested closures, the *innermost* escaping closure must capture `self`, so the following code would be invalid: + +```swift +execute { [self] in + execute { // Fix-it: capture 'self' explicitly to enable implicit 'self' in this closure. + x += 1 // Error: reference to property 'x' in closure requires explicit use of 'self' to make capture semantics explicit. Fix-it: reference 'self.' explicitly. + } +} +``` + +This new behavior will also be available for closures with `unowned(safe)` and `unowned(unsafe)` captures of `self` since the programmer has clearly declared their intent to capture `self` (as for strong captures). For `weak` captures of `self`, it's not immediately clear what bare reference to an instance property even *means* when the local `self` is bound weakly—should it be treated as though the programmer had written `self?.`? Should there be a special syntax available in this context, such as `?.x`? Should we offer a diagnostic offering to insert `self?.`? Or should we instead suggest that the programmer re-bind `self` strongly via a `guard` statement? There are enough open questions here that have not been sufficiently discussed, so this proposal leaves the handling of `weak self` captures as a future direction worthy of further consideration. + +The existing errors and fix-its have their language updated accordingly to indicate that there are now multiple ways to resolve the error. In addition to the changes noted above, we will also have: + +``` +Error: call to method in closure requires explicit use of 'self' to make capture semantics explicit. +``` + +The new fix-it for explicitly capturing `self` will take slightly different forms depending on whether there is a capture list already present ("insert '`self, `'"), whether there are explicit parameters ("insert '`[self]`'"), and whether the user has already captured a variable with the name `self` (in which case the fix-it would not be offered). Since empty capture lists are allowed (`{ [] in ... }`), the fix-it will cover this case too. + +This new rule would only apply when `self` is captured directly, and with the name `self`. This includes captures of the form `[self = self]` but would still not permit implicit `self` if the capture were `[y = self]`. In the unusual event that the user has captured another variable with the name `self` (e.g. `[self = "hello"]`), we will offer a note that this does not enable use of implicit `self` (in addition to the existing error attached to the attempted use of implicit `self`): + +```swift +Note: variable other than 'self' captured here under the name 'self' does not enable implicit 'self'. +``` + +If the user has a capture of `weak self` already, we offer a special diagnostic note indicating that `weak` captures of `self` do not enable implicit `self`: + +```swift +Note: weak capture of 'self' here does not enable implicit 'self'. +``` + +If either of the two above notes is present, we will not offer the usual fix-its for resolving this error, since the code inserted would be erroneous. + +## Source compatibility + +This proposal makes previously illegal syntax legal, and has no effect on source compatibility. + +## Effect on ABI stability + +This is an entirely frontend change and has no effect on ABI stability. + +## Effect on API resilience + +This is not an API-level change, and has no effect on API resilience. + +## Future Directions + +### Bound method references + +While this proposal opens up implicit `self` in situations where we can be reasonably sure that we will not cause a reference cycle, there are other cases where implicit `self` is currently allowed that we may want to disallow in the future. One of these is allowing bound method references to be passed into escaping contexts without making it clear that such a reference captures `self`. For example: + +```swift +class Test { + var x = 0 + func execute(_ work: @escaping () -> Void) { + work() + } + func method() { + execute(inc) // self is captured, but no error! + execute { inc() } // error + } + func inc() { + x += 1 + } +} +``` + +This would help reduce the false negatives of the "make capture semantics explicit" error, making it more useful and hopefully catching reference cycles that the programmer was previously unaware of. + +### Weak captures of `self` + +It might be desirable to add improved diagnostic information (or similarly do away with the error) in the case where the programmer has explicitly captured `self`, but only as a weak reference. For the reasons noted in this proposal, this was not pursued, but answering the lingering questions and issuing a follow-up proposal would fit nicely with this direction. + +## Alternatives considered + +### Always require `self` in the capture list + +The rule requiring the use of explicit `self` is helpful when the code base does not already require `self` to be used on all instance accesses. However, many code bases have linters or style guides that require `self` to be used explicitly always, making the capture semantics opaque. Always requiring `self` to be captured in the capture list explicitly would ensure that there are no `self` captures that the programmer is unaware of, even if they naturally use `self` for instance accesses. This would be a more drastic, source breaking change (and is not ruled out by adopting this change), so it was not seriously pursued as part of this proposal. + +### Eliminate the former fix-it + +A less extreme solution to the problem described above is to simply stop offering the current fix-it that suggests adding the explicit `self.` at the point of reference in favor of only recommending the explicit capture list fix-it, when possible. + + diff --git a/proposals/0270-rangeset-and-collection-operations.md b/proposals/0270-rangeset-and-collection-operations.md new file mode 100644 index 0000000000..f9c3e224f7 --- /dev/null +++ b/proposals/0270-rangeset-and-collection-operations.md @@ -0,0 +1,454 @@ +# Add Collection Operations on Noncontiguous Elements + +* Proposal: [SE-0270](0270-rangeset-and-collection-operations.md) +* Authors: [Nate Cook](https://github.com/natecook1000), [Jeremy Schonfeld](https://github.com/jmschonfeld) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Implementation: [apple/swift#69766](https://github.com/apple/swift/pull/69766) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/9b5957c00e7483ab8664afe921f989ed1394a666/proposals/0270-rangeset-and-collection-operations.md), [2](https://github.com/swiftlang/swift-evolution/blob/b17d85fcaf38598fd2ea19641d0e9c26c96747ec/proposals/0270-rangeset-and-collection-operations.md), [3](https://github.com/swiftlang/swift-evolution/blob/54d85f65fefce924eb4d5bf10dd633e81f063d11/proposals/0270-rangeset-and-collection-operations.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-add-rangeset-and-related-collection-operations/29961)) ([first review](https://forums.swift.org/t/se-0270-add-collection-operations-on-noncontiguous-elements/30691)) ([first revision](https://forums.swift.org/t/returned-for-revision-se-0270-add-collection-operations-on-noncontiguous-elements/31484)) ([second review](https://forums.swift.org/t/se-0270-review-2-add-collection-operations-on-noncontiguous-elements/31653)) ([second revision](https://forums.swift.org/t/revised-se-0270-add-collection-operations-on-noncontiguous-elements/32840)) ([third review](https://forums.swift.org/t/se-0270-review-3-add-collection-operations-on-noncontiguous-elements/32839)) ([acceptance into preview package](https://forums.swift.org/t/accepted-se-0270-add-collection-operations-on-noncontiguous-elements/33270)) ([fourth pitch](https://forums.swift.org/t/pitch-revision-4-add-collection-operations-on-noncontiguous-elements/68345)) ([fourth review](https://forums.swift.org/t/se-0270-fourth-review-add-collection-operations-on-noncontiguous-elements/68855)) ([acceptance](https://forums.swift.org/t/accepted-se-0270-add-collection-operations-on-noncontiguous-elements/69080)) + +## Introduction + +We can use a `Range` to refer to a group of consecutive positions in a collection, but the standard library doesn't currently provide a way to refer to discontiguous positions in an arbitrary collection. I propose the addition of a `RangeSet` type that can represent any number of positions, along with collection algorithms that operate on those positions. + +## Motivation + +There are varied uses for tracking multiple elements in a collection, such as maintaining the selection in a list of items, or refining a filter or search result set after getting more input from a user. + +The Foundation data type most suited for this purpose, `IndexSet`, uses integers only, which limits its usefulness to arrays and other random-access collection types. The standard library is missing a collection that can efficiently store ranges of indices, and is missing the operations that you might want to perform with such a collection. These operations themselves can be challenging to implement correctly, and have performance traps as well — see WWDC 2018's [Embracing Algorithms](https://developer.apple.com/videos/wwdc/2018/?id=223) talk for a demonstration. + +## Proposed solution + +This proposal adds a `RangeSet` type for representing multiple, noncontiguous ranges, as well as a variety of collection operations for creating and working with range sets. + +```swift +var numbers = Array(1...15) + +// Find the indices of all the even numbers +let indicesOfEvens = numbers.indices(where: { $0.isMultiple(of: 2) }) + +// Perform an operation with just the even numbers +let sumOfEvens = numbers[indicesOfEvens].reduce(0, +) +// sumOfEvens == 56 + +// You can gather the even numbers at the beginning +let rangeOfEvens = numbers.moveSubranges(indicesOfEvens, to: numbers.startIndex) +// numbers == [2, 4, 6, 8, 10, 12, 14, 1, 3, 5, 7, 9, 11, 13, 15] +// numbers[rangeOfEvens] == [2, 4, 6, 8, 10, 12, 14] +``` + + +## Detailed design + +`RangeSet` is generic over any `Comparable` type, and supports fast containment checks for ranges and individual values, as well as adding and removing ranges of that type. + +```swift +/// A set of values of any comparable value, represented by ranges. +public struct RangeSet: Equatable, CustomStringConvertible { + /// Creates an empty range set. + public init() {} + + /// Creates a range set containing the values in the given range. + public init(_ range: Range) + + /// Creates a range set containing the values in the given ranges. + public init(_ ranges: S) where S.Element == Range + + /// A Boolean value indicating whether the range set is empty. + public var isEmpty: Bool { get } + + /// Returns a Boolean value indicating whether the given value is + /// contained by the range set. + /// + /// - Complexity: O(log *n*), where *n* is the number of ranges in the + /// range set. + public func contains(_ value: Bound) -> Bool + + /// Adds the values represented by the given range to the range set. + /// + /// If `range` overlaps or adjoins any existing ranges in the set, the + /// ranges are merged together. Empty ranges are ignored. + public mutating func insert(contentsOf range: Range) + + /// Removes the given range of values from the range set. + /// + /// The values represented by `range` are removed from this set. This may + /// result in one or more ranges being truncated or removed, depending on + /// the overlap between `range` and the set's existing ranges. + public mutating func remove(contentsOf range: Range) +} + +extension RangeSet: Sendable where Bound: Sendable {} +extension RangeSet: Hashable where Bound: Hashable {} +``` + +#### Conveniences for working with collection indices + +Although a range set can represent a set of values of any `Comparable` type, the primary intended use case is to maintain a set of indices into a collection. To streamline this workflow, `RangeSet` includes an additional initializer and methods for inserting and removing individual indices. + +```swift +extension RangeSet { + /// Creates a new range set containing ranges that contain only the + /// specified indices in the given collection. + /// + /// - Parameters: + /// - index: The index to include in the range set. `index` must be a + /// valid index of `collection` that isn't the collection's `endIndex`. + /// - collection: The collection that contains `index`. + public init(_ indices: S, within collection: C) + where S: Sequence, C: Collection, S.Element == C.Index, C.Index == Bound + + /// Inserts a range that contains only the specified index into the range + /// set. + /// + /// - Parameters: + /// - index: The index to insert into the range set. `index` must be a + /// valid index of `collection` that isn't the collection's `endIndex`. + /// - collection: The collection that contains `index`. + /// + /// - Returns: `true` if the range set was modified, or `false` if + /// the given `index` was already in the range set. + /// + /// - Complexity: O(*n*), where *n* is the number of ranges in the range + /// set. + @discardableResult + public mutating func insert(_ index: Bound, within collection: C) -> Bool + where C: Collection, C.Index == Bound + + /// Removes the range that contains only the specified index from the range + /// set. + /// + /// - Parameters: + /// - index: The index to remove from the range set. `index` must be a + /// valid index of `collection` that isn't the collection's `endIndex`. + /// - collection: The collection that contains `index`. + /// + /// - Complexity: O(*n*), where *n* is the number of ranges in the range + /// set. + public mutating func remove(_ index: Bound, within collection: C) + where C: Collection, C.Index == Bound +} +``` + + +#### Accessing underlying ranges + +`RangeSet` provides access to its ranges as a random-access collection via the `ranges` property. +You can access the individual indices represented by the range set by using it as a subscript parameter to a collection's `indices` property. + +```swift +extension RangeSet { + public struct Ranges: RandomAccessCollection, Equatable, CustomStringConvertible { + public var startIndex: Int { get } + public var endIndex: Int { get } + public subscript(i: Int) -> Range + } + + /// A collection of the ranges that make up the range set. + public var ranges: Ranges { get } +} + +extension RangeSet.Ranges: Sendable where Bound: Sendable {} +extension RangeSet.Ranges: Hashable where Bound: Hashable {} +``` + +The ranges that are exposed through this collection are always in ascending order, are never empty, and never overlap or adjoin. For this reason, inserting an empty range or a range that is subsumed by the ranges already in the set has no effect on the range set. Inserting a range that adjoins an existing range simply extends that range. + +```swift +var set = RangeSet([0..<5, 10..<15]) +set.insert(contentsOf: 7..<7) +set.insert(contentsOf: 11.<14) +// Array(set.ranges) == [0..<5, 10..<15] + +set.insert(contentsOf: 5..<7) +// Array(set.ranges) == [0..<7, 10..<15] + +set.insert(contentsOf: 7..<10) +// Array(set.ranges) == [0..<15] +``` + +#### `SetAlgebra`-like methods + +`RangeSet` implements a subset of the `SetAlgebra` protocol, +for working with more than one `RangeSet`. + +```swift +extension RangeSet { + public func union(_ other: RangeSet) -> RangeSet + public mutating func formUnion(_ other: RangeSet) + + public func intersection(_ other: RangeSet) -> RangeSet + public mutating func formIntersection(_ other: RangeSet) + + public func symmetricDifference(_ other: RangeSet) -> RangeSet + public mutating func formSymmetricDifference(_ other: RangeSet) + + public func subtracting(_ other: RangeSet) -> RangeSet + public mutating func subtract(_ other: RangeSet) + + public func isSubset(of other: RangeSet) -> Bool + public func isSuperset(of other: RangeSet) -> Bool + public func isStrictSubset(of other: RangeSet) -> Bool + public func isStrictSuperset(of other: RangeSet) -> Bool + public func isDisjoint(with other: RangeSet) -> Bool +} +``` + +### New `Collection` APIs + +#### Finding multiple elements + +Akin to the `firstIndex(...)` and `lastIndex(...)` methods, this proposal introduces `indices(where:)` and `indices(of:)` methods that return a range set with the indices of all matching elements in a collection. + +```swift +extension Collection { + /// Returns the indices of all the elements that match the given predicate. + /// + /// For example, you can use this method to find all the places that a + /// vowel occurs in a string. + /// + /// let str = "Fresh cheese in a breeze" + /// let vowels: Set = ["a", "e", "i", "o", "u"] + /// let allTheVowels = str.indices(where: { vowels.contains($0) }) + /// // str[allTheVowels].count == 9 + /// + /// - Complexity: O(*n*), where *n* is the length of the collection. + public func indices(where predicate: (Element) throws -> Bool) rethrows + -> RangeSet +} + +extension Collection where Element: Equatable { + /// Returns the indices of all the elements that are equal to the given + /// element. + /// + /// For example, you can use this method to find all the places that a + /// particular letter occurs in a string. + /// + /// let str = "Fresh cheese in a breeze" + /// let allTheEs = str.indices(of: "e") + /// // str[allTheEs].count == 7 + /// + /// - Complexity: O(*n*), where *n* is the length of the collection. + public func indices(of element: Element) -> RangeSet +} +``` + +#### Accessing elements via `RangeSet` + +When you have a `RangeSet` describing a group of indices for a collection, you can access those elements via subscript. This subscript returns a new `DiscontiguousSlice` type, which couples the collection and range set to provide access. + +```swift +extension Collection { + /// Accesses a view of this collection with the elements at the given + /// indices. + /// + /// - Complexity: O(1) + public subscript(subranges: RangeSet) -> DiscontiguousSlice { get } +} + +extension MutableCollection { + /// Accesses a mutable view of this collection with the elements at the + /// given indices. + /// + /// - Complexity: O(1) to access the elements, O(*m*) to mutate the + /// elements at the positions in `subranges`, where *m* is the number of + /// elements indicated by `subranges`. + public subscript(subranges: RangeSet) -> DiscontiguousSlice { get set } +} + +/// A collection wrapper that provides access to the elements of a collection, +/// indexed by a set of indices. +public struct DiscontiguousSlice: Collection, CustomStringConvertible { + /// The collection that the indexed collection wraps. + public var base: Base { get set } + + /// The set of index ranges that are available through this indexing + /// collection. + public var subranges: RangeSet { get set } + + public typealias SubSequence = Self + + /// A position in an `DiscontiguousSlice`. + public struct Index: Comparable { + public let base: Base.Index + } + + public var startIndex: Index { get } + public var endIndex: Index { set } + public subscript(i: Index) -> Base.Element { get } + public subscript(bounds: Range) -> Self { get } +} + +extension DiscontiguousSlice: BidirectionalCollection where Base: BidirectionalCollection {} +extension DiscontiguousSlice: Sendable where Base: Sendable, Base.Index: Sendable {} +extension DiscontiguousSlice: Equatable where Base.Element: Equatable {} +extension DiscontiguousSlice: Hashable where Base.Element: Hashable {} + +extension DiscontiguousSlice.Index: Sendable where Base.Index: Sendable {} +extension DiscontiguousSlice.Index: Hashable where Base.Index: Hashable {} +``` + +#### Moving elements + +Within a mutable collection, you can move the elements represented by a range set to be in a contiguous range before the element at a specific index, while otherwise preserving element order. When moving elements, other elements slide over to fill gaps left by the elements that move. For that reason, this method returns the new range of the elements that were previously located at the indices within the provided `RangeSet`. + +```swift +extension MutableCollection { + /// Collects the elements at the given indices just before the element at + /// the specified index. + /// + /// This example finds all the uppercase letters in the array and gathers + /// them between `"i"` and `"j"`. + /// + /// var letters = Array("ABCdeFGhijkLMNOp") + /// let uppercaseRanges = letters.indices(where: { $0.isUppercase }) + /// let rangeOfUppercase = letters.moveSubranges(uppercaseRanges, to: 10) + /// // String(letters) == "dehiABCFGLMNOjkp" + /// // rangeOfUppercase == 4..<13 + /// + /// - Parameters: + /// - subranges: The indices of the elements to move. + /// - insertionPoint: The index to use as the destination of the elements. + /// - Returns: The new bounds of the moved elements that were previously located at + /// the indices provided by the RangeSet. + /// + /// - Complexity: O(*n* log *n*) where *n* is the length of the collection. + @discardableResult + public mutating func moveSubranges( + _ subranges: RangeSet, to insertionPoint: Index + ) -> Range +} +``` + +#### Removing elements + +Within a range-replaceable collection, you can remove the elements represented by a range set. `removeSubranges(_:)` is a new `RangeReplaceableCollection` requirement with a default implementation, along with an overload for collections that also conform to `MutableCollection`. + +```swift +extension RangeReplaceableCollection { + /// Removes the elements at the given indices. + /// + /// For example, this code sample finds the indices of all the vowel + /// characters in the string, and then removes those characters. + /// + /// var str = "The rain in Spain stays mainly in the plain." + /// let vowels: Set = ["a", "e", "i", "o", "u"] + /// let vowelIndices = str.indices(where: { vowels.contains($0) }) + /// + /// str.removeSubranges(vowelIndices) + /// // str == "Th rn n Spn stys mnly n th pln." + /// + /// - Parameter subranges: A range set representing the elements to remove. + /// + /// - Complexity: O(*n*), where *n* is the length of the collection. + public mutating func removeSubranges(_ subranges: RangeSet) +} + +extension Collection { + /// Returns a collection of the elements in this collection that are not + /// represented by the given range set. + /// + /// For example, this code sample finds the indices of all the vowel + /// characters in the string, and then retrieves a collection that omits + /// those characters. + /// + /// let str = "The rain in Spain stays mainly in the plain." + /// let vowels: Set = ["a", "e", "i", "o", "u"] + /// let vowelIndices = str.indices(where: { vowels.contains($0) }) + /// + /// let disemvoweled = str.removingSubranges(vowelIndices) + /// print(String(disemvoweled)) + /// // Prints "Th rn n Spn stys mnly n th pln." + /// + /// - Parameter subranges: A range set representing the elements to remove. + /// - Returns: A collection of the elements that are not in `indices`. + public func removingSubranges(_ subranges: RangeSet) -> DiscontiguousSlice +} +``` + +## Other considerations + +### Source compatibility + +These additions preserve source compatibility. + +### Effect on ABI stability + +This proposal only makes additive changes to the existing ABI. + +### Effect on API resilience + +All the proposed additions are versioned. + +## Alternatives considered + +### `SetAlgebra` Conformance + +An earlier version of this proposal included `SetAlgebra` conformance when the `Bound` type was `Stridable` with an integer `Stride`. The idea behind this constraint was that with an integer-strideable bound, `RangeSet` could translate an individual value into a `Range` which contained just that value. This translation enabled the implementation of `SetAlgebra` methods like insertion and removal of individual elements. + +However, when working with collection indices, there is no guarantee that the correct stride distance is the same for all integer-stridable types. For example, when working with a collection `C` that uses even integers as its indices, removing a single integer from a `RangeSet` could leave an odd number as the start of one of the constitutive ranges: + +```swift +let numbers = (0..<20).lazy.filter { $0.isMultiple(of: 2) } +var set = RangeSet(0..<10) +set.remove(4) +// set.ranges == [0..<4, 5..<10] +``` + +Since `5` is not a valid index of `numbers`, it's an error to use it when subscripting the collection. + +One way of avoiding this issue would be to change the internal representation of `RangeSet` to store an enum of single values, ranges, and fully open ranges: + +```swift +enum _RangeSetValue { + case single(Bound) + case halfOpen(Range) + case fullyOpen(lower: Bound, upper: Bound) +} +``` + +This implementation would allow correct `SetAlgebra` conformance, but lose the ability to always have a maximally efficient representation of ranges and a single canonical empty state. Through additions and removals, you could end up with a series of a fully-open ranges, none of which actually contain any values in the `Bound` type. + +```swift +var set: RangeSet = [1..<4] +set.remove(2) +set.remove(3) +set.remove(4) +// set.ranges == [.fullyOpen(1, 2), .fullyOpen(2, 3), .fullyOpen(3, 4)] +``` + +### `Elements` view + +In an earlier version of the proposal, `RangeSet` included a collection view of the indices, conditionally available when the `Bound` type was strideable with an integer stride. This collection view was removed for the same reasons as covered in the section above. Users can access these indices by using a `RangeSet` as a subscript for the source collection's `indices` property. + + +### More helpers for working with individual `Collection` indices + +In an earlier version of this proposal, `RangeSet` included several methods that accepted range expressions as a parameter, along with the matching collection, so that the `RangeSet` could convert the range expression into a concrete range. These methods have been removed; instead, convert range expressions to concrete ranges before using `RangeSet` methods. + +There has been some concern that any APIs that take both an index and a collection represent a violation of concerns. However, these kinds of methods are already well-represented in the standard library (such as `RangeExpression.relative(to:)`), and are necessary for building readable interfaces that work with the design of Swift's collections and indices. + + +### A predicate-based `gather` + +This proposal originally included a predicate-based `gather` method (at a time when `moveSubranges(_:to:)` was named `gather(in:at:)`). This predicate-based method has been removed to allow design work to continue on the larger issue of predicate-based mutating collection methods. + +In particular, the issue with this method stems from the fact that a collection user may sometimes want to use a predicate that operates on collection elements (e.g. to check for even elements), and may sometimes want to use a predicate that operates on collection indices (e.g. to test for indices that are part of a known group). For non-mutating methods, this poses no problem, as one can call the predicate-based method on the collection's `indices` property instead. However, with mutating methods, this kind of access poses issues with copy-on-write and exclusivity. + +One potential solution is to offer an `(Index, Element) -> Bool` predicate for mutating methods instead of the currently standard `(Element) -> Bool` predicate. This kind of change should be considered for existing mutating methods, like `removeAll(where:)` and `partition(by:)`, as well as any future additions. + +### `DiscontiguousSlice` conformance to `MutableCollection` + +Previously, this proposal included a `MutableCollection` conformance for `DiscontiguousSlice` where the `Base` collection conformed to `MutableCollection`. However, after further consideration, this conformance was removed. The semantics of how this conformance should behave could be potentially unexpected and in addition to being slightly misleading the conformance would likely be hard to use in the first place. For these reasons, the conformance was removed. + +### Other bikeshedding + +The review garnered several alternative names for the `RangeSet` type. Some were too tied to the index use case (such as `IndexRangeSet`, `DiscontiguousIndices`, `SomeIndices`), while others didn't represent enough of an obvious improvement to supplant the proposed name (such as `SparseRange`, `DiscontiguousRange`, or `RangeBasedSet`). + +There were also a suggestion that the methods for inserting and removing ranges of values should be aligned with the `SetAlgebra` methods `formUnion` and `subtract`. Instead, to keep the `RangeSet` API aligned with user expectations, these operations will keep the names `insert(contentsOf:)` and `remove(contentsOf:)`. + +As a result of other feedback, some of the collection operations have been renamed. Instead of `removeAll(in:)` to match `removeAll(where:)`, removing the elements represented by a `RangeSet` is now `removeSubranges(_:)`, as a partner to `removeSubrange(_:)`. Similarly, the `gather(in:at:)` method has been renamed to `moveSubranges(_:to:)`. These names do better at continuing the naming scheme set up by `RangeReplaceableCollection`. + +Additionally, we have also chosen the names `indices(of:)` and `indices(where:)` as opposed to the previously suggested names of `ranges(of:)` and `ranges(where:)`. Since this proposal's last revision, `Collection` now has a `ranges(of:)` API that accepts a `RegexComponent` and returns a `[Range]`, so to avoid ambiguity we use the term "indices" here to provide a `RangeSet` value. The compiler does not find these functions ambiguous with the existing `indices` property on `Collection`. + diff --git a/proposals/0271-package-manager-resources.md b/proposals/0271-package-manager-resources.md new file mode 100644 index 0000000000..f83bbad6ef --- /dev/null +++ b/proposals/0271-package-manager-resources.md @@ -0,0 +1,240 @@ +# Package Manager Resources + +* Proposal: [SE-0271](0271-package-manager-resources.md) +* Authors: [Anders Bertelrud](https://github.com/abertelrud), [Ankit Aggarwal](https://github.com/aciidb0mb3r) +* Review Manager: [Boris Buegling](https://github.com/neonichu) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift-package-manager#2381](https://github.com/apple/swift-package-manager/pull/2381), + [apple/swift-package-manager#2510](https://github.com/apple/swift-package-manager/pull/2510), + [apple/swift-package-manager#2520](https://github.com/apple/swift-package-manager/pull/2520), + [apple/swift-package-manager#2607](https://github.com/apple/swift-package-manager/pull/2607) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0271-package-manager-resources/31021) + +## Introduction + +Packages should be able to contain images, data files, and other resources needed at runtime. This proposal describes SwiftPM support for specifying such package resources, and introduces a consistent way of accessing them from the source code in the package. + +## Motivation + +Packages consist primarily of source code intended to be compiled and linked into client executables. Sometimes, however, the code needs additional resources that are expected to be available at runtime. Such resources could include images, sounds, user interface specifications, OpenGL or Metal shaders, and other typical runtime resources. During the build, some package resources might be copied verbatim into the product, while others might need to be processed in some way. + +Resources aren't always intended for use by clients of the package; one use of resources might include test fixtures that are only needed by unit tests. Such resources would not be incorporated into clients of the package along with the library code, but would only be used while running the package's tests. + +One of the fundamental principles behind SwiftPM is that packages should be as platform-independent and client-agnostic as possible: in particular, packages should make as few assumptions as possible about the details of how they will be incorporated into a particular client on a particular platform. + +For example, a package product might in one case be built as a dynamic library or a framework that is embedded into an application bundle, and might in another case be statically linked into the client executable. These differences might be due to requirements of the platform for which the package is being built, or might be a result of deployment packaging choices made by the client. + +Packages should therefore be able to specify resources in a platform-independent way, and SwiftPM should provide a consistent way to access those resources without requiring the source code to make assumptions about exactly where the resources will be at runtime. For example, the source code cannot assume that the resources will be in the same bundle as the compiled code (if bundles even exist as separate entities on a given platform). + +Build systems (such as those in SwiftPM, Xcode, etc) are then free to build resources for runtime use in any way they choose, as long as the API for accessing those resources at runtime continues to work as described in this document. + +## Goals + +The most important goals of this design include: + +* Making it easy to add resource files in a package. + +* Avoiding unintentionally copying files not intended to be resources (e.g. design documents) into the product. + +* Supporting platform-specific resource types for packages written using specific APIs (e.g. Storyboards, XIBs, and Metal shaders on Apple platforms). + +* Supporting localization. + +* Letting package authors put resources where they like in their source directories (to allow functional organization of large code bases). + +## Proposed Solution + +We propose the following to support resources in Swift packages: + +- Scope resources to targets i.e. resource files will be part of a target just like source files. + +- Extend SwiftPM's file detection capabilities and apply a "rule" to every file in the target. SwiftPM will emit an error for any file in a target for which it is not able to automatically determine the rule. + +- Add two new rules "copy" and "process" to existing list of rules. At this time, there will be no additional built-in file type that use these two new rules. + +- Vend an API in libSwiftPM to allow its clients to register additional file types supported by those clients. SwiftPM will automatically detect the matching files in a target and report them in the package model data structure. Similar to source files, package authors will not need to explicitly declare these files in the package manifest. This ability is useful for clients like Xcode to support platform-specific file types (such as `.metal`) without having to bake in a hardcoded list in SwiftPM’s codebase. + +- Add a new `resources` parameter in `target` and `testTarget` APIs to allow declaring resource files explicitly. + +- Create a bundle for each module with resources and generate a `Resources` struct for each module it compiles for accessing the bundle. + +## Detailed Design + +### Declaring Resources + +The `target` and `testTarget` function in the PackageDescription API will be extended to have an optional `resources` parameter: + +```swift +public static func target( + name: String, + dependencies: [Target.Dependency] = [], + path: String? = nil, + exclude: [String] = [], + sources: [String]? = nil, + resources: [Resource]? = nil, // <=== NEW + publicHeadersPath: String? = nil, + cSettings: [CSetting]? = nil, + cxxSettings: [CXXSetting]? = nil, + swiftSettings: [SwiftSetting]? = nil, + linkerSettings: [LinkerSetting]? = nil +) -> Target +``` + +Where the `Resource` type is defined as: + +```swift +/// Represents an individual resource file. +public struct Resource { + /// Apply the platform-specific rule to the given path. + /// + /// Matching paths will be processed according to the platform for which this + /// target is being built. For example, image files might get optimized when + /// building for platforms that support such optimizations. + /// + /// By default, a file will be copied if there is no specialized processing + /// for its file type. + /// + /// If path is a directory, the rule is applied recursively to each file in the + /// directory. + public static func process(_ path: String) -> Resource + + /// Apply the copy rule to the given path. + /// + /// Matching paths will be copied as-is and will be at the top-level + /// in the bundle. The structure is retained for if path is a directory. + public static func copy(_ path: String) -> Resource +} +``` + +The exact API in `libSwiftPM` for registering additional resource file types will be an implementation detail since it currently does not have a stable API. However, there will be *some* API that provides this functionality. Such file types will be automatically picked up by SwiftPM and will not require explicit declaration using the `resources` parameter. SwiftPM contributors will iterate on this API with existing and future clients. For example, if Xcode registers `.metal` as a file type and there is a metal file in a target, the package will not need to declare that file in the resources list. However, the package can override the default rule for this file by adding it to the exclude list or by specifying the "copy" rule for this file. + +Packages will also be able to explicitly declare the resource files using the two new APIs in `PackageDescription`. These APIs have some interesting behavior when the given path is a directory: + +- The `.copy` API allows copying directories as-is which is useful if a package author wants to retain the directory structure. + +- The `.process` API allows applying the rule recursively to files inside a directory. This is useful for packages that have all of its resources contained in directories for organization. + +Each file in a target will be required to have a rule. SwiftPM will emit an error if it is not able to automatically determine the rule for a file. For example, if there is a `README.md` in the target, the package will need to make an explicit decision about this file by either adding to the exclude list or in the resources list. + +### Resources Bundle + +For each target that defines at least one resource file, SwiftPM will produce a Foundation-style bundle whose name and identifier are derived from an implementation-defined unique combination of the package name and the target name (the package source code should not concern itself with the exact name). On Linux, each such resource bundle will be located next to the built executable that is the ultimate client of the target containing the resources. On macOS and related platforms, each such bundle is located at a place of the build system's choosing (typically nested inside the ultimate client's main bundle). + +Note that the construction of a bundle to hold the resources does not necessarily mean that the code is in the same bundle, i.e. it is not necessarily a framework. Whether or not the code is colocated with the resources is an implementation detail. The key part of this proposal is that the package's code not make any assumptions that limits the build system's choices in where to put the resources, as long as it can make them accessible to the package code at runtime. + +### Runtime Access to Resources Bundle + +SwiftPM will generate an internal static extension on `Bundle` for each module it compiles: + +```swift +extension Bundle { + /// The bundle associated with the current Swift module. + static let module: Bundle = { ... }() +} +``` + +Because this is an internal static property, it would be visible only to code within the same module, and the implementations for each module would not interfere with each other. The implementation generated by SwiftPM would use information about the layout of the built product to instantiate and cache the bundle that contains the resources. + +The first access to the `module` property would cause the bundle to be instantiated. Modules without any resources would not have resource bundles, and for such modules, no declaration would be created. + +Some examples: + +```swift +// Get path to DefaultSettings.plist file. +let path = Bundle.module.path(forResource: "DefaultSettings", ofType: "plist") + +// Load an image that can be in an asset archive in a bundle. +let image = UIImage(named: "MyIcon", in: Bundle.module, compatibleWith: UITraitCollection(userInterfaceStyle: .dark)) + +// Find a vertex function in a compiled Metal shader library. +let shader = try mtlDevice.makeDefaultLibrary(bundle: Bundle.module).makeFunction(name: "vertexShader") + +// Load a texture. +let texture = MTKTextureLoader(device: mtlDevice).newTexture(name: "Grass", scaleFactor: 1.0, bundle: Bundle.module, options: options) +``` + +Because the name of the accessor would always be `module`, code could be moved between modules (as long as the resources were moved as well) without requiring any source code changes. + +This would be an improvement over the status quo in Cocoa code, which involves either `Bundle.init(for:)` or `Bundle.init(identifier:)`. The former is problematic because it assumes that the resources will necessarily be in the same bundle as the code (which won't be true for statically linked code), and the latter is problematic because it hardcodes the identifier name (and also because it doesn't automatically cause the bundle to be loaded if it hasn't been loaded already). + +This approach also allows creating codeless resource bundles. Any package target that really wants to just vend the bundle of resources could implement a single property to publicly expose the bundle to clients. It seems reasonable that the package authors have to explicitly vend them. + +For Objective-C, the build system will add a preprocessor define called `SWIFTPM_MODULE_BUNDLE` which can be used to access the bundle from any `.m` file. + +### Localization Support + +Localization support will build on top of the resources feature described in this proposal. We think resources and localization support are separable and it would be better to discuss localization in its own proposal. + +## Rationale + +This section describes the rationale for some of the design decisions taken in this proposal. + +### Scoping of Resources to targets + +Resources are most commonly referenced by the code that needs them, so it seems natural to consider source files and resource files as conceptually being a part of the same target. Any client that depends on the target's code also needs the associated resources to be available at runtime (though it usually doesn't access them directly, but instead through the code that makes use of them). + +Furthermore, the processing of some types of resources might result in the generation of more source code, which needs to be compiled into the same module as the code that needs the resource. For this reason, too, it is natural to consider the resources (and any generated code) as conceptually being a part of the same target as the code that uses it. + +Scoping resources by target also aligns naturally with how targets are currently built in other development environments that support resources: in Xcode, for example, each target is built into a CFBundle (usually a framework or an application), and any resources associated with the target are copied into that bundle. Since a bundle provides a namespace for the resources in it, scoping resources by target is natural. + +If resources were not scoped by target, there would need to be some way to associate the resources with the source code that needs them. This could take the form of defining a separate resource target, and making the code target depend on that resource target. A library that needs just one or two resources would then require the use of two separate targets (one for the code and another for the resources). This seems needlessly complicated, and is therefore not the approach used by this proposal. + +### Rules for determining resource files + +SwiftPM uses file system conventions for determining the set of source files that belongs to each target in a package: specifically, a target's source files are those that are located underneath the designated "target directory" for the target. By default this is a directory that has the same name as the target and is located in "Sources" (for a regular target) or "Tests" (for a test target), but this location can be customized in the package manifest. + +Given that resources are conceptually associated with targets (as discussed in the previous section), then it seems logical for those resources to also reside inside the directories of the targets to which they belong. + +The question then becomes how to determine which of the target files to treat as resources and which to treat as source files. + +A flexible approach would be to treat resource files the same as source files, using rules to determine how to process each file inside the target directory. Just as for source files, each resource file's role would be determined by its file type (as indicated by its filename suffix). + +For example, SwiftPM already recognizes files with suffixes such as `.swift`, `.c`, `.s`, etc as source files, and has built-in rules to determine which build tool to invoke for each known type of source file. + +This could be viewed as simply broadening the notion of what constitutes a "source file" beyond just source code to be compiled and linked into the executable part of the product; the executable code is, after all, just one aspect of the built artifact. In this broadened view, Storyboards, XIBs, Metal shaders, and Xcode Asset Catalogs are all just compiled to produce other kinds of files that then get incorporated into the built product. + +Processing resource files the same way as source files is especially natural for file types that don't fit neatly into a source file / resource file dichotomy, such as Metal shaders and CoreData models. Metal shaders, for example, are compiled as source code, and linked together to produce binary resource files that are loaded at runtime. CoreData models are also compiled, and may generate source code as part of that compilation. + +Specialized file types such as Storyboards, XIBs, CoreData models, or Metal shaders doesn't introduce much risk of mischaracterizing the intent of having the file in the target; these are specialized file types with unambiguous filename suffixes and a clear purpose. + +But multi-purpose file types such as `.md`, `.png`, `.jpg`, `.txt`, or `.pdf` are more problematic: should a Markdown file or a PDF found in a source directory be treated as a resource to be copied into the built product, or is it just a document describing internal implementation details? Similarly, is any given PNG file an artwork resource, or does it perhaps belong to a Markdown file containing internal design notes? And files of an unknown type altogether might or might not be resources, without any clear way for SwiftPM to tell. + +On several occasions, developers have accidentally ended up shipping internal design documents or other files because they were automatically included as resources in their apps based on their types. + +This design aims to reduce that risk as much as possible by limiting the additional built-in rules to only those file types where the intent is clear (Storyboards, XIBs, Metal shaders, CoreData models, Xcode Asset Catalogs, etc). Resource files of more ambiguous types have to be explicitly designated as resources. + +## Impact on existing packages + +The new APIs and behavior described in this proposal will be guarded against the tools version this proposal is implemented in. Packages that want to use this feature will need to update their tools version. + +Once a package updates its tools version, it might need to make some changes to make explicit decisions about any files that were previously being ignored implicitly. + +## Alternatives considered + +### Requiring all resources to be in a special directory + +While an approach involving a magical directory (such as one in a particular location and with a particular name) might be simpler, it would cause problems with adding package support for the large number of CocoaPods and other projects that don't have all of their resources confined to a particular directory. + +One possible approach would be consider resource files as being completely disjoint from source files, requiring all of a target's resources to be located in a separate "Resources" directory inside the target directory. + +While this might be appropriate for some source package layouts, many existing source hierarchies are not structured this way. For example, many CocoaPods have resource files interspersed with source files; this is also common in Xcode projects. + +This practice is especially common for types of resource files that are closely associated with source files; for example, Storyboards and XIB files are commonly located together with the source code that loads those Storyboards and XIB files, grouped by functional component. + +A variation of this would be to have a new top-level "Resources" directory next to "Sources" and "Targets", and to rely on naming conventions to associate a target's resources with a target's sources, but that moves the resources even further from the existing source hierarchy for the target. + +## Future Directions + +### Type-safe access to individual resource files + +Historically, macOS and iOS resources have been accessed primarily by name, using untyped Bundle APIs such as `path(forResource:ofType:)` that assume that each resource is stored as a separate file. Missing resources or mismatched types (e.g. trying to load a font as an image) have historically resulted in runtime errors rather than being detected at build time. + +In the long run, we would like to do better. Because SwiftPM knows the names and types of resource files, it should be able provide type-safe access to those resources by, for example, generating the right declarations that the package's authored code could reference. Missing or mismatched resources would produce build-time errors, leading to early detection of problems. + +This would also serve to separate the referencing of a resource from the details about how that resource is stored in the built artifact; instead of getting the path to an image, for example, and then loading that image using a separate API (which assumes that the image is stored in a separate file on disk), the image accessor could do whatever is needed to load the image based on the platform and build-time processing that was done, so that the package code doesn't have to be aware of such details. + +In the short term, however, we want to keep things simple and allow existing code to work without major modifications. Therefore, the short-term approach suggested here is to stay with Bundle APIs for the actual resource access, and to provide a very simple way for code in a package to access the bundle associated with that code. A future proposal could introduce typed resource references as additional functionality. + +### Filename patterns + +As a separate enhancement, we would like to consider enhancing sources, exclude and resources array to accept a glob pattern using `fnmatch()` semantics. The semantics involving the presence or absence of a path separator are useful in that they provide great expressibility in a manner that is intuitive for anyone familiar with shell name-vs-path lookup semantics etc. diff --git a/proposals/0272-swiftpm-binary-dependencies.md b/proposals/0272-swiftpm-binary-dependencies.md new file mode 100644 index 0000000000..993b70a5e6 --- /dev/null +++ b/proposals/0272-swiftpm-binary-dependencies.md @@ -0,0 +1,473 @@ +# Package Manager Binary Dependencies + +* Proposal: [SE-0272](0272-swiftpm-binary-dependencies.md) +* Authors: [Braden Scothern](https://github.com/bscothern), [Daniel Dunbar](https://github.com/ddunbar), [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Boris Bügling](https://github.com/neonichu) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift-package-manager#2509](https://github.com/apple/swift-package-manager/pull/2509), + [apple/swift-package-manager#2511](https://github.com/apple/swift-package-manager/pull/2511), + [apple/swift-package-manager#2514](https://github.com/apple/swift-package-manager/pull/2514), + [apple/swift-package-manager#2588](https://github.com/apple/swift-package-manager/pull/2588) +* Decision Notes: [First Review](https://forums.swift.org/t/returned-for-revision-se-0272-package-manager-binary-dependencies/30994), [Second Review](https://forums.swift.org/t/accepted-with-modifications-se-0272-package-manager-binary-dependencies/31926) + +## Contents + ++ [Introduction](#introduction) ++ [Motivation](#motivation) ++ [Proposed solution](#proposed-solution) ++ [Detailed design](#detailed-design) ++ [New `PackageDescription` API](#new-packagedescription-api) ++ [New `Package.resolved` Behavior](#new-packageresolved-behavior) ++ [Binary Target Artifact Format](#binary-target-artifact-format) ++ [Security](#security) ++ [Impact on existing packages](#impact-on-existing-packages) ++ [Future directions](#future-directions) ++ [Alternatives considered](#alternatives-considered) + +## Introduction + +SwiftPM currently supports source-only packages for several languages, and with +a very proscriptive build model which considerably limits exactly how the +compilation of the source can be performed. While this makes packages consistent +and to some extent "simple", it limits their use in several important cases: +* Software vendors who wish to provide easy integration with the package + manager, but do not deliver source code, cannot integrate. +* Existing code bases which would like to integrate "simply" with SwiftPM, but + require more complicated build processes, have no recourse. + +For example, consider these use cases: + + * Someone wants to create a Swift package for + generating [LLVM](https://llvm.org) code. However, LLVM's build process is + far more complex than can be currently fit into SwiftPM's build model. This + makes building an *easy to use* package difficult. + * A third-party wants to provide a Swift SDK for easily integrating their + service with server-side Swift applications. The SDK itself relies on + substantial amounts of internal infrastructure the company does not want to + make available as open source. + * A large company has an internal team which wants to deliver a Swift package + for use in their iOS applications, but for for business reasons cannot publish + the source code. + +This proposal defines a new SwiftPM feature to allow SwiftPM to accept some +forms of "binary packages". This proposal is intentionally written to +address the above use cases *explicitly*, it **does not** define a general +purpose "binary artifact" mechanism intended to address other use cases (such as +accelerating build performance). The motivations for this are discussed in more +detail below. + +Swift-evolution thread: [\[PITCH\] Support for binary dependencies](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620) + +## Motivation + +SwiftPM has a large appeal to certain developer communities, like the iOS +ecosystem, where it is currently very common to rely on closed source +dependencies such as Firebase, GoogleAnalytics, Adjust and many more. Existing +package managers like Cocoapods support these use cases. By adding such support +to SwiftPM, we will unblock substantially more adoption of SwiftPM within those +communities. + +Prior to Swift 5.1, the Swift compiler itself did not expose all of the features +(like ABI compatibility) required to build a workable solution. Now that those +features are present, it makes sense to re-evaluate the role of binary packages. + +The goal of this proposal is to make *consumption* of binary packages as +described above *easy*, *intuitive*, *safe*, and *consistent*. This proposal +**does not** attempt to provide any affordances for the creation of the binary +package itself. The overall intent of this proposal is to allow consumption of +binary packages *where necessary*, but not to encourage their use or facilitate a +transition from the existing source-based ecosystem to a binary one. + +This proposal is also focused at packages which come exclusively in binary form, +it explicitly **does not** introduce a mechanism which allows a package to be +present in either source or binary form. See alternatives considered for more +information on this choice. + +## Proposed solution + +To enable binary dependencies we have to make changes in the `Package.swift` manifest file. First, we propose to add a new target type which describes a binary target. Such a target needs to declare where to retrieve the artifact from and the checksum of the expected artifact. An example of such a package can be seen below: + +```swift +let package = Package( + name: "SomePackage", + platforms: [ + .macOS(.v10_10), .iOS(.v8), .tvOS(.v9), .watchOS(.v2), + ], + products: [ + .library(name: "SomePackage", targets: ["SomePackageLib"]) + ], + targets: [ + .binaryTarget( + name: "SomePackageLib", + url: "https://github.com/some/package/releases/download/1.0.0/SomePackage-1.0.0.zip", + checksum: "839F9F30DC13C30795666DD8F6FB77DD0E097B83D06954073E34FE5154481F7A" + ), + .binaryTarget( + name: "SomeLibOnDisk", + path: "artifacts/SomeLibOnDisk.zip" + ) + ] +) +``` + +Packages are allowed to contain a mix of binary and source targets. This is +useful when, for example, providing a pre-built or closed source C library +alongside an open source set of Swift bindings for the library. + +The use case will be limited to Apple platforms in the beginning. In the future, we can add support for other platforms. A potential approach is outlined in the future directions section. + +## Detailed design + +The design consists of the following key points: +* New `PackageDescription` API for defining a binary target. +* New requirements for the `Package.resolved` file when using binary packages. +* A new command to compute a checksum for a file. +* A new mechanism for downloading binary target artifacts. +* Support for artifact mirroring. + +Terminology: + +* Technically, a *target* is binary or not. However, we anticipate that often a + single package will consist of either exclusively source or binary targets. We + will use the term *binary package* to refer to any package which contains at + least one binary product. Similarly, a *binary product* is one which contains + at least one binary target. + +Our design attempts to optimize for the following goals: + +* Ease of use for clients +* Ease of implementation in existing SwiftPM +* Ease of maintenance in the face of an evolving SwiftPM +* Understandable composition with current and upcoming SwiftPM features +* Support existing well-known occurrences of binary artifacts in the existing + (often iOS focused) target developer market. + +while keeping the following as non-goals: + +* Ease of production of binary packages +* Simplicity of binary artifact distribution mechanism +* Widespread use of binary packages + +## New `PackageDescription` API + +### BinaryTarget +Since a binary target is different compared to a source only target, we propose to introduce two new static method on `Target` to declare a binary target. We propose to support local and remote artifacts from the beginning. In the alternatives considered section is a larger collection of potential artifact stores. However we opted to simplify the initial implementation by just supporting a url and a path based definition. Later, we can implement different types of providers with different authentication methods. + +```swift +extension Target { + /// Declare a binary target with the given url. + public static func binaryTarget( + name: String, + url: String, + checksum: String + ) -> Target + + /// Declare a binary target with the given path on disk. + public static func binaryTarget( + name: String, + path: String + ) -> Target +} +``` + +## Checksum computation +We propose to add a new command to SwiftPM `swift package compute-checksum ` which is going to be used to compute the checksum of individual files. This implementation can then evolve in the future and is tied to the tools version of the package to avoid breaking compatibility with older tools. + +## New `Package.resolved` Behavior + +For binary targets we will validate the commit hashes from the resolved file for any dependencies from now on to ensure the checksums of binaries cannot be changed for a specific version. This lets us check for errors during resolution where a package's version did not change but the checksum did. In this case we will throw an error alerting the user about this. + +### Resolution + +Package resolution and dependency expression will not be impacted by this change (except where explicitly noted). + +#### Exported product with binary dependency that specifies a type +SwiftPM will emit an error during resolution when a product that directly exports a binary dependency declares a type, e.g.: `.product(name: "MyBinaryLib", type: .static, targets: ["MyBinaryLib"])`. + +#### Resolution on non-Apple platforms +When resolving a package that contains a binary dependency on non-Apple platforms, SwiftPM will throw an error and explicitly state that this dependency is not valid for the current platform. During the review it was brought up that we could ignore these dependencies but that would make the behavior of SwiftPM very unexpected. In the future, when properly supporting other platforms this can be solved easily with a proper condition mechanism. + +## Binary Target Artifact Format + +SwiftPM currently supports multiple platforms; however, this proposal only adds support for binary targets on Apple platforms. The reason for this is that Apple platforms provide ABI guarantees and an already existing format we can leverage to simplify the initial implementation. For Apple platforms we propose to use the `XCFramework` format for artifacts. This format already supports dynamic and static linking. Furthermore, it can contain products for every individual Apple platform at once. + +SwiftPM expects url-based artifacts to be packaged inside a `.zip` file where the artifact is lying at the root of the archive. Furthermore, the artifact needs to have the same module name as the target name provided inside the manifest file. + +For path-based artifact SwiftPM supports artifacts as a `.zip` and as a raw `XCFramework`. + +During resolution SwiftPM won't do any verification of the format of the artifact. This is up to the vendor to provide correct and valid artifact. In the future, this can be extended and further validation, such as checking that the module name matches, can be implemented. + +## Security + +When adding new external dependencies, it is always important to consider the security implication that it will bring with it. Comparing the trust level of a source-based to a binary-based dependency the first thought is that the trust level of the source-based dependency is higher since on can inspect its source code. However, there is no difference between a binary and source dependency since source-based dependencies can have security issues as well. One should have better reasons to trust a dependency than source being inspectable. + +There is still a significant difference between having a dependency with zero vs. any binary dependency. For example, the portability of a library with binary dependencies is far worse than the one with only source-based dependencies. + +However, there are still some security related aspects when it comes to binary artifacts that we should mitigate. For example, when declaring a `binaryTarget` the hash of the artifact is required similar to Homebrew. By doing this an attacker needs to compromise both the server which provides the artifact as well as the git repository which provides the package manifest. A secondary reason is that the server providing the binary might be out of the package author's control and this way we can ensure that the expected binary is used. + +Lastly, the hash of the binary is stored in the package resolved to avoid that the vendor changes the artifact behind a version without anyone noticing. + +## Mirroring support +Binary artifacts can also be mirrored. We propose to deprecate the existing `--package-url` option and to replace it with a `--original-url` option which will work for both package URLs as well as artifact URLs: + +``` +$ swift package config set-mirror \ + --original-url \ + --mirror-url + +# Example: + +$ swift package config set-mirror \ + --original-url https://github.com/Core/core/releases/download/1.0.0/core.zip \ + --mirror-url https://mygithub.com/myOrg/core/releases/download/1.0.0/core.zip +``` + +Additionally, we propose to add a command to unset a mirror URL for an artifact: + +``` +$ swift package config unset-mirror \ + --original-url https://github.com/Core/core/releases/download/1.0.0/core.zip +``` + +The other unset command options `--mirror-url` and `--all` will be working the same for artifacts as they do for packages. + +## Impact on existing packages + +No current package should be affected by this change since this is only an additive change in enabling SwiftPM to use binary dependencies. + +## Future directions + +### Support for non-Apple platforms +Non-Apple platforms provide non-trivial challenges since they are not always giving guarantees of the ABI of the platform. Additionally, further conditions such as the corelibs-foundation ABI or if the hardware supports floating points need to be taken into consideration when declaring a package for non-Apple platforms. Various other communities tried to solve this, e.g. Python's [manylinux](https://www.python.org/dev/peps/pep-0600/). + +In the future, we could add an `Artifact` struct and `ArtifactCondition`s to SwiftPM which provides the possibility to declare under which conditions a certain artifact can be used. Below is a potential `Artifact` and `ArtifactCondition` struct which does **not** include a complete set of conditions that need to be taken into consideration. + +```swift +public struct Artifact { + public enum Source { + case url(String, checksum: String) + case path + } + + public let source: Source +} + +public struct ArtifactCondition: Encodable { + public struct LLVMTriplet: Encodable { + // Should be only the subset that Swift supports + enum ArchType: String, Encodable { + case arm5 + case arm7 + case x86 + case x86_64 + // And the rest + } + + // Should be only the subset that Swift supports + enum Vendor: String, Encodable { + case apple + case ibm + case bgp + case suse + // And the rest + } + + // Should be only the subset that Swift supports + enum OSType: String, Encodable { + case linux + case openBSD + case win32 + case darwin + case iOS + case macOSX + // And the rest + } + + let archType: ArchType + let vendor: Vendor + let osType: OSType + // Do we need the LLVM environment here? + public init(archType: ArchType, vendor: Vendor, osType: OSType) { + self.archType = archType + self.vendor = vendor + self.osType = osType + } + } + + private let llvmTriplets: [LLVMTriplet] + + private init(llvmTriplets [LLVMTriplet]) { + self.llvmTriplets = llvmTriplets + } + + /// Create an artifact condition. + /// + /// - Parameters: + /// - llvmTriplets: The llvm triplets for which this condition will be applied. + public static func when( + llvmTriplets: [LLVMTriplet] + ) -> ArtifactCondition { + return ArtifactCondition(llvmTriplets: llvmTriplets) + } +} +``` + +## Alternatives considered + +### General Approach + +There are three popular use cases for binary packages (terminology courtesy +of +[Tommaso Piazza](https://forums.swift.org/t/spm-support-for-binaries-distribution/25549/32)). They +are all related, but for the purposes of this proposal we will distinguish them: + +1. "Vendored binaries" (no source available, or cannot be built from source) +2. "Artifact cache" (pre-built version of packages which are available in source form) +3. "Published & tagged binaries" (the package manager heavily depends on + published and tagged binary artifacts) + +In the first case, binary packages are used because there is no other viable +alternative. In the second case, binary artifacts are used to either accelerate +development (by eliminating existing build or analysis steps), or to simplify +cognitive load (e.g. by removing uninteresting sources from display in an IDE +with package integration). In the third case, the very mechanism the package +manager uses to resolve dependencies is deeply integrated with the publishing of +a binary artifact. While the third approach is popular in certain ecosystems and +package managers like Maven, we consider it out of scope given SwiftPM's current +decentralized architecture, and we will ignore it for the remained of this +proposal. + +The proposal explicit sets out to solve the first use case; a natural question +is should the second use case be supported by the same feature. In this +proposal, we chose not to go that route, for the following reasons: + +* When used as a build or space optimization, artifact caching is a general + purpose strategy which can be applied to *any* package. SwiftPM was explicitly + designed in order to allow the eventual implementation of performant, + scalable, and even distributed caches for package artifacts. Artifact caching + is something we would like to "just work" in order to give the best possible + user experience. + + In particular, when artifact is employed "manually" to achieve the above + goals, it often introduces certain amounts of ambiguity or risk. From the + user's perspective, when the source of a package is available, then one would + typically like to think of the artifact cache as a perfect reproduction of + "what would have been built, if I built it myself". However, leveraging a + binary package like mechanism instead of explicit tool support for this often + means: + + * There is almost no enforcement that the consumed binary artifact matches the + source. The above presumption of equivalence makes such artifact caches a + ripe opportunity for embedding malware into an ecosystem. + + * The consumer does not always have control over the artifact production. This + interacts adversely with potential future SwiftPM features which would allow + the build of a package to be more dependent on its consumer (e.g. allowing + compile-time configuration "knobs & switches"). + + * The artifact cache "optimization" may not apply to all packages, or may + require substantial manual effort to maintain. + +* When used as a workflow improvement (e.g. to reduce the scope of searches), + our position is that the user would ultimately have a better user experience + by explicitly enumerating and designing features (either in SwiftPM, or in + related tools) to address these use cases. When analyzed, it may become clear + that there is more nuance to the solution than an artifact caching scheme + could reasonably support. + +* The choice to support both source and binary packages in the same mechanism + imposes certain requirements on the design which make it more complex than the + existing proposal. In particular, it means that the metadata about how the + source and artifacts are mapped must be kept somewhere adjacent to but + distinct from the package description (since a source package needs to define + its source layout). However, such a mechanism must also be defined in a way + that works when no source layout is present to support binary only packages. + + Finally, since it would be a feature with user-authored metadata, such a + mechanism would need to be updated when any other SwiftPM enhancement + introduces or changes the nature of the source layout specification. + +Taken together, the above points led us to focus on a proposal focused at +"vendored binaries", while our hope is that artifact caching eventually becomes +a built-in and automatic feature of the package manager which applies to all +packages. + +### Binary Signatures + +We considered adding signature checks during the checkout of binary dependencies but when these have transitive dependencies it gets complicated expressing that in the `Package.swift`. + +``` + let package = Package( + name: "Paper", + products: [...], + dependencies: [ + .package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .any), + .package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .gpg("XXX")"), + ], + targets: [...] + ) +``` + +### Binary target vs. binary product +During the discussion, it was brought up whether a binary dependency should be declared as a target or a product. Below is a list of what we took into consideration when deciding between target and product. In the end, a target seems like a better choice for a binary dependency. + +- Targets allow configuring linker/compiler flags; this ability might be necessary for static libraries +- There are already `systemLibraryTargets` which are essentially dependencies on pre-existing binaries on the host system +- Targets represent a single module, the same is true for one XCFramework +- Currently there is no way to depend on products, so mixed binary and source packages might be harder if we go with a binary product approach +- Analogy with what currently is being produced by source packages (dylibs => product) + +### .o file format +During the discussion of the proposal, the idea of using `.o` files was brought up. This would follow what SwiftPM creates for source-based dependencies right now; therefore, making the integration potentially easier. The main benefit of using `.o` files would be that the product linking the binary artifact can decide whether to link it dynamically or statically. However, further discussion needs to happen here how this would work. For now XCFrameworks are the initial format. XCFrameworks will allow current framework authors to package their existing XCFrameworks and distribute them via SwiftPM. + +### Avoiding duplicate symbols with static libraries/frameworks +When multiple products depend on a static library or framework it will result in duplicated symbols. SwiftPM could be smart enough to figure this out from the manifest and provide an error. This is a potential future improvement, but would require SwiftPM to know the linkage type of the artifact. + +### Opt-in to allow binaries +In the beginning, we considered making binary support an opt-in feature so that only when somebody explicitly allows it then SwiftPM tried to use them. However, after discussion in the Swift forum we came to the conclusion that the trust one has to extend to a dependency is no different between a source-based and binary-based one; therefore, we removed the opt-in behavior but added an opt-out behavior. + +Using an opt-in mechanism for binary dependencies would also mean that any package that adds a binary dependencies would need to do a major version bump, because it will require any client to change something in their manifests. + +### Whitelist for allowed URLs for binary dependencies +During the discussion of this proposal another solution to the `allowsBinary` flag was brought up. That is to create a whitelist for URLs that are allowed origins for binary artifacts. This way one can still control from where binary dependencies come but it doesn't require to allow them for a complete dependency tree; therefore, giving more fine-grained control. However, we propose an opt-out mechanism instead. + +### Opt-out configuration in separate file +During the discussion of this proposal it was decided that an opt-out mechanism was good to give package users and vendors an escape hatch. However, it was discussed whether this configuration should live inside the manifest or a separate configuration file. In this proposal, we opted to keep the configuration inside the manifest file. + +### Opt-out in package manifest +In the first round, we proposed to add a configuration flag in the manifest to opt-out of binary dependencies; however, during the review it became apparent that this flag doesn't provide as much value and can make some dependencies actually more restricted when they add this flag. Therefor, we opted to not include such a configuration flag and let workflow tooling provide this functionality if needed. + +```swift +public final class Package { + ... + /// This disallows any binary dependency or any transitive binary dependency. + public var disallowsBinaryDependencies: Bool + ... +} +``` + +```swift +let package = Package( + name: "SomeOtherPackage", + disallowsBinaryDependencies: true, + products: [ + .library(name: "SomeOtherPackage", targets: ["SomeOtherPackageLib"]) + ], + targets: [ + .target(name: "SomeOtherPackageLib") + ] +) +``` + +### Support for various artifact stores +Initially, we considered the various artifact stores on the market and how we can integrate with them. We decided to support a URL based artifact definition for the first implementation since the various providers require each their own method of authentication. However, we wanted to keep the possibility for future additions of providers open; therefore, we made the source of an artifact an enum which can be extended. + +Possible artifact stores we considered: +- Github releases +- Github packages +- Gitlab +- Bitbucket +- Artifactory, Nexus etc. + +### Conditional Linkage +During the discussion of this proposal it was brought up to support conditional linkage of binary targets. This is in itself a very useful feature; however, it applies to binary and source based targets. In the end, conditional linkage is an orthogonal feature which can be pitched separately. diff --git a/proposals/0273-swiftpm-conditional-target-dependencies.md b/proposals/0273-swiftpm-conditional-target-dependencies.md new file mode 100644 index 0000000000..720d6bc9f6 --- /dev/null +++ b/proposals/0273-swiftpm-conditional-target-dependencies.md @@ -0,0 +1,113 @@ +# Package Manager Conditional Target Dependencies + +* Proposal: [SE-0273](0273-swiftpm-conditional-target-dependencies.md) +* Authors: [David Hart](https://github.com/hartbit) +* Review Manager: [Boris Buegling](https://github.com/neonichu) +* Status: **Partially implemented (Swift 5.3 supports platform conditionals, but not configuration conditionals)** +* Implementation: [apple/swift-package-manager#2428](https://github.com/apple/swift-package-manager/pull/2428), + [apple/swift-package-manager#2598](https://github.com/apple/swift-package-manager/pull/2598) + +## Introduction + +This proposal introduces the ability for Swift package authors to conditionalize target dependencies on platform and configuration with a similar syntax to the one introduced in [SE-0238](0238-package-manager-build-settings.md) for build settings. This gives developers more flexibility to describe complex target dependencies to support multiple platforms or different configuration environments. + +## Motivation + +This proposal resolves two use cases that the current version of the Package Manager doesn't support very well. In the first scenario, packages that span multiple platforms may need to depend on different libraries depending on the platform, as can be the case for low-level, platform-specific code. In a second scenario, packages may want to link against libraries only in certain configurations, for example when importing debug libraries, which do not make sense to build and link in release builds, or when importing instrumentation logic, which only make sense in release builds when the developer can not benefit from debugging. + +This proposal attempts to bring solutions to those use cases by allowing package authors to define under what build environments dependencies need to be built and linked against targets. + +## Proposed solution + +To allow package authors to append conditions to target dependencies, we introduce new APIs to the `Package.swift` manifest library. The `.target`, `.product`, and `.byName` will be optionally configurable with a `condition` argument of the same format as for build settings, to specify the platforms and configuration under which that dependency should be enabled. For example: + +```swift +// swift-tools-version:5.3 + +import PackageDescription + +let package = Package( + name: "BestPackage", + dependencies: [ + .package(url: "https://github.com/pureswift/bluetooth", .branch("master")), + .package(url: "https://github.com/pureswift/bluetoothlinux", .branch("master")), + ], + targets: [ + .target( + name: "BestExecutable", + dependencies: [ + .product(name: "Bluetooth", condition: .when(platforms: [.macOS])), + .product(name: "BluetoothLinux", condition: .when(platforms: [.linux])), + .target(name: "DebugHelpers", condition: .when(configuration: .debug)), + ] + ), + .target(name: "DebugHelpers") + ] +) +``` + +It is important to note that this proposal has no effect on dependency resolution, but only affects which targets are built and linked against each other during compilation. In the previous example, both the `Bluetooth` and `BluetoothLinux` packages will be resolved regardless of the platform, but when building, the Package Manager will avoid building the disabled dependency and will link the correct library depending on the platform. It will also build and link against the `DebugHelpers` target, but only in debug builds. + +## Detailed design + +### New `PackageDescription` API + +All the cases of the `Target.Dependency` enum will gain a new optional `TargetDependencyCondition` argument, a type with the same API as `BuildSettingCondition`. Creating a new type allows the APIs to evolve independently, if ever they have to, to support different condition types. The current static factory functions for initializing those enums will be obsoleted in the version of the tools this proposal will appear in, and new functions will take their place, introducing a new optional argument: + +```swift +extension Target.Dependency { + /// Creates a dependency on a target in the same package. + /// + /// - Parameters: + /// - name: The name of the target. + /// - condition: The condition under which the dependency is exercised. + @available(_PackageDescription, introduced: 5.3) + public static func target( + name: String, + condition: TargetDependencyCondition? = nil + ) -> Target.Dependency { + // ... + } + + /// Creates a dependency on a product from a package dependency. + /// + /// - Parameters: + /// - name: The name of the product. + /// - package: The name of the package. + /// - condition: The condition under which the dependency is exercised. + @available(_PackageDescription, introduced: 5.3) + public static func product( + name: String, + package: String? = nil, + condition: TargetDependencyCondition? = nil + ) -> Target.Dependency { + // ... + } + + /// Creates a by-name dependency that resolves to either a target or a product but + /// after the package graph has been loaded. + /// + /// - Parameters: + /// - name: The name of the dependency, either a target or a product. + /// - condition: The condition under which the dependency is exercised. + @available(_PackageDescription, introduced: 5.3) + public static func byName( + name: String, + condition: TargetDependencyCondition? = nil + ) -> Target.Dependency { + // ... + } +} +``` + +## Security + +This proposal has no impact on security, safety, or privacy. + +## Impact on existing packages + +Current packages will not be impacted by this change as all `PackageDescription` changes will be gated by a new tools version. As always, the Package Manager will support package hierarchies with heterogeneous tools versions, so authors will be able to adopt those new APIs with minimal impact to end-users. + +## Alternatives considered + +No alternatives were considered for now. diff --git a/proposals/0274-magic-file.md b/proposals/0274-magic-file.md new file mode 100644 index 0000000000..68a7016951 --- /dev/null +++ b/proposals/0274-magic-file.md @@ -0,0 +1,233 @@ +# Concise magic file names + +* Proposal: [SE-0274](0274-magic-file.md) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax), [Dave DeLong](https://github.com/davedelong) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift/) +* Status: **Implemented (Swift 5.8)** +* Upcoming Feature Flag: `ConciseMagicFile` +* Decision Notes: [Review #1](https://forums.swift.org/t/se-0274-concise-magic-file-names/32373/50), [Review #2](https://forums.swift.org/t/re-review-se-0274-concise-magic-file-names/33171/11), [Additional Commentary](https://forums.swift.org/t/revisiting-the-source-compatibility-impact-of-se-0274-concise-magic-file-names/37720) +* Next Proposal: [SE-0285](0285-ease-pound-file-transition.md) + +## Introduction + +Today, `#file` evaluates to a string literal containing the full path to the current source file. We propose to instead have it evaluate to a human-readable string containing the filename and module name, while preserving the existing behavior in a new `#filePath` expression. + +Swift-evolution thread: [Concise Magic File Names](https://forums.swift.org/t/concise-magic-file-names/31297), [We need `#fileName`](https://forums.swift.org/t/we-need-filename/19781) + +## Changes since the first review + +* We now specify that the `#file` string will have the format `/` (with a second form for future expansion) and discuss how to parse it. The previous revision left this format as a compiler implementation detail that tools should always treat as opaque. + +* We now discuss the behavior of `#sourceLocation`'s `file` parameter and provide for a warning when it creates conflicts. The previous revision did not discuss these topics. + +* We now mention the need for tooling to map `#file` strings back to paths. + +* We now provide for a warning when a wrapper around a `#filePath`-defaulting function passes it `#file` instead, or vice versa. The previous revision did not discuss this. + +* We have added several suggestions from the first review to the "alternatives considered" section to explain why we aren't proposing them. + +## Motivation + +In Swift today, the magic identifier `#file` evaluates to a string literal containing the full path (that is, the path passed to `swiftc`) to the current file. It's a nice way to trace the location of logic occurring in a Swift process, but its use of a full path has a lot of drawbacks: + +* It can inadvertently reveal private or sensitive information. The full path to a source file may contain a developer's username, hints about the configuration of a build farm, proprietary versions or identifiers, or the Sailor Scout you named an external disk after. Developers probably don't know that this information is embedded in their binaries and may not want it to be there. And most uses of `#file` are in default arguments, which makes this information capture invisible at the use site. The information leaks here are quite serious; if other languages hadn't already normalized this, I doubt we would think that `#file` was an acceptable design. + +* It bloats binaries produced by the Swift compiler. In testing with the Swift benchmark suite, a shorter `#file` string reduced code size by up to 5%. The large code also impacts runtime performance—in the same tests, a couple dozen benchmarks ran noticeably faster, with several taking 22% less time. While we didn't benchmark app launch times, it's likely that they are also adversely affected by lengthy `#file` strings. + +* It introduces artificial differences between binaries built on different machines. For instance, the same code built in two different environments might produce different binaries with different hashes. This makes life difficult for anyone trying to cache build artifacts, find the differences between two binaries, or otherwise hoping to benefit from reproducible builds. + +Meanwhile, the benefits you might expect from using a full path aren't really there: + +* The paths are not portable between machines, so you can't use it to automatically match an error to a line of source code unless your machine has the same directory layout as the machine that built the binary. + +* The paths are not guaranteed to be absolute—XCBuild and SwiftPM happen to use absolute paths, but other build systems like Bazel don't—so you can't be sure that you can resolve paths to a file on disk. + +* The paths are not even guaranteed to be unique within a process because the same file could be included in several different modules, or different projects could be located at the same paths on different machines or at different times. + +For `#file`'s most important use case, it seems like the current string computed by `#file` is suboptimal and a better default would improve Swift in several ways. + +## Proposed solution + +We propose changing the string that `#file` evaluates to—instead of evaluating to the full path, it will now have the format `/`. For those applications which still need a full path, we will provide a new magic identifier, `#filePath`. Both of these features will otherwise behave the same as the old `#file`, including capturing the call site location when used in default arguments. The standard library's assertion and error functions will continue to use `#file`. + +With this proposal, a file at `/Users/becca/Desktop/0274-magic-file.swift` in a module named `MagicFile` with this content: + +```swift +print(#file) +print(#filePath) +fatalError("Something bad happened!") +``` + +Would produce this output: + +```text +MagicFile/0274-magic-file.swift +/Users/becca/Desktop/0274-magic-file.swift +Fatal error: Something bad happened!: file MagicFile/0274-magic-file.swift, line 3 +``` + +The Swift compiler does not currently allow two files with the same name (regardless of path) to be compiled into the same module.[1] This means that `/` does a better job of uniquely identifying a file than the full path without taking up nearly as much space. It is also easy for a human to read and map back to a source file; tools can automatically map these strings back to paths if they know which files are in each module, which compiler tooling could help them to determine. + +> [1] This limitation ensures that identically-named `private` and `fileprivate` declarations in different files will have unique mangled names. A future version of the Swift compiler could lift this rule. + +## Detailed design + +### Specification of the `#file` string format + +The formal grammar of the string generated by `#file` is: + +```text +file-string → module-name "/" file-name +file-string → module-name "/" disambiguator "/" file-name // Reserved for future use + +module-name → [same as Swift identifier] +disambiguator → [all characters except U+0000 NULL] +file-name → [all characters except U+002F SOLIDUS and U+0000 NULL] +``` + +`"/"` means `"/"` in this grammar, even on systems with a different path separator. + +Note that *disambiguator* may contain `"/"` characters. To parse *file-string* correctly, developers should split the string on `"/"` characters and use the first element for the module name, or the last element for the file name, without assuming that there are exactly two elements. Alternatively, they may use the index of the first `"/"` as the end of the module name and the index after the last `"/"` as the start of the file name. + +### Computation of *file-string* components + +The components of *file-string* are currently computed from the module name and `#filePath` at the source location. Specifically: + +* *module-name* is the name of the module being compiled. + +* *file-name* is the substring of the `#filePath` string falling after the last path separator. "Path separator" here is either `"/"` or the host's path separator character (e.g. `"\"` on Windows). + +* *disambiguator* is currently always omitted. + +Future compilers may also use the other potential `#filePath` strings in the module as an input to this function and use them to compute a *disambiguator* field to distinguish between `#filePath`s that would otherwise produce the same *file-string*. We do not currently specify how they will do this; we simply constrain future compilers to fit this information into the *disambiguator* field. + +### Interaction with `#sourceLocation` + +The `file` parameter of a `#sourceLocation` directive will specify the content of `#filePath`. Since *file-string* is calculated from the `#filePath`, this means that the last component of the `file` parameter will be used as the *file-name*. We do not provide a way to change the *module-name* or to directly specify the *file-string*. + +In current compilers, a `#sourceLocation` directive may produce `#file` strings which collide with the `#file` strings produced by other `#sourceLocation`-introduced paths or physical files. The compiler will warn about this. Future compilers may use the *disambiguator* field to differentiate them instead. + +### Tooling + +SourceKit will provide facilities for mapping `#file` strings back to their matching `#filePath`s. + +### Default argument mismatch diagnostics + +To help users who are wrapping a `#file` or `#filePath`-defaulting function avoid using the wrong default argument in their wrappers, the compiler will emit a warning when a parameter which captures one of the magic identifiers is passed to a default argument which captures a different one. This warning will also apply to existing magic identifiers, like `#file` vs. `#function` or `#line` vs. `#column`. + +We do not specify the exact diagnostics in this proposal, but as an illustration, the mismatch in this code: + +```swift +func fn1(file: String = #filePath) { ... } +func fn2(file: String = #file) { + fn1(file: file) +} +``` + +Might be diagnosed like: + +``` +sample.swift:3: warning: parameter 'file' with default argument '#file' passed to parameter 'file', whose default argument is '#filePath' + fn1(file: file) + ^~~~ +sample.swift:2: note: did you mean for parameter 'file' to default to '#filePath'? +func fn2(file: String = #file) { + ^~~~~ + #filePath +sample.swift:3: note: add parentheses to silence this warning + fn1(file: file) + ^ ^ + ( ) +``` + +### Implementation status + +A prototype of this feature is already in master; it can be enabled by passing `-Xfrontend -enable-experimental-concise-pound-file` to the Swift compiler. This prototype does not include some of the revisions for the second review: + +* The specified `#file` string format (the prototype uses `file-name (module-name)` instead). +* The warning for colliding `#sourceLocation`s. +* Proof-of-concept for tooling support, in the form of a table printed into comments in `-emit-sil` output. + +These are implemented in the unmerged [apple/swift#29412](https://github.com/apple/swift/pull/29412). + +## Source compatibility + +All existing source code will continue to compile with this change, and `#file`'s documentation never specified precisely what its contents were; in one of the pitch threads, [Ben Cohen](https://forums.swift.org/t/concise-magic-file-names/31297/19) said that this is sufficient to satisfy Swift's source compatibility requirements. However, the proposal *will* cause behavior to change in existing code, and in some cases it will change in ways that cause existing code to behave incorrectly when run. Code that is adversely affected by this change can access the previous behavior by using `#filePath` instead of `#file`. + +The change to the behavior of `#file` was deferred to the next major language version (Swift 6). However, it can be enabled with the [upcoming feature flag](0362-piecemeal-future-features.md) `ConciseMagicFile`. + +## Effect on ABI stability + +None. `#file` is a compile-time feature; existing binaries will continue to work as they did before. + +## Effect on API resilience + +As with any addition to Swift's syntax, older compilers won't be able to use module interface files that have `#filePath` in default arguments or inlinable code. Otherwise, none. + +## Alternatives considered + +### Deprecate `#file` and introduce two new syntaxes + +Rather than changing the meaning of `#file`, we could keep its existing behavior, deprecate it, and provide two alternatives: + +* `#filePath` would continue to use the full path. +* `#fileName` would use the new concise string suggested by this proposal. + +This is a more conservative approach that would avoid breaking any existing uses. We choose not to propose it for three reasons: + +1. The name `#fileName` is misleading because it sounds like the string only contains the file name, but it also contains the module name. `#file` is more vague, so we're more comfortable saying that it's "a string that identifies the file". + +2. This alternative will force developers to update every use of `#file` to one or the other option. We feel this is burdensome and unnecessary given how much more frequently the `#fileName` behavior would be appropriate. + +3. This alternative gives users no guidance on which feature developers ought to use. We feel that giving `#file` a shorter name gives them a soft push towards using it when they can, while resorting to `#filePath` only when necessary. + +4. Since all uses of `#file`—not just ones that require a full path—would change, all uses of `#file` in module interfaces—not just ones that require a full path—would become stumbling blocks for backwards compatibility. This includes uses in the swiftinterface files for the standard library and XCTest. + +However, if the core team feels that changing `#file`'s behavior will cause unacceptable behavior changes, this ready-made alternative would accomplish most of the goals of this proposal. + +### Support more than two `#file` variants + +We considered introducing additional `#file`-like features to generate other strings, selecting between them either with a compiler flag or with different magic identifiers. The full set of behaviors we considered included: + +1. Path as written in the compiler invocation +2. Guaranteed-absolute path +3. Path relative to the Xcode `SOURCE_DIR` value, or some equivalent +4. Last component of the path (file name only) +5. File name plus module name +6. Empty string (sensible as a compiler flag) + +We ultimately decided that supporting only 1 (as `#filePath`) and 5 (as `#file`) would adequately cover the use cases for `#file`. Five different syntaxes would devote a lot of language surface area to a small niche, and controlling the behavior with a compiler flag would create six language dialects that might break some code. Some of these behaviors would also require introducing new concepts into the compiler or would cause trouble for distributed build systems. + +### Make `#filePath` always absolute + +While we're looking at this area of the language, we could change `#filePath` to always generate an absolute path. This would make `#filePath` more stable and useful, but it would cause problems for distributed builds unless it respected `-debug-prefix-map` or something similar. It would also mean that there'd be no simple way to get the *exact* same behavior as Swift previously provided, which would make it more difficult to adapt code to this change. + +Ultimately, we think changing to an absolute path is severable from this proposal and that, if we want to do this, we should consider it separately. + +### Provide separate `#moduleName` and `#fileName` magic identifiers + +Rather than combining the module and file names into a single string, we could provide separate ways to retrieve each of them. There are several reasons we chose not to do this: + +* `#fileName` is so ambiguous between modules that good uses for it by itself are few and far between. You really need either the module name or the path to tell which file by that name you're talking about. The full path can be so verbose that it's better to truncate to the filename and deal with the ambiguity, but module names are short enough that module name + filename doesn't really have this problem. + +* Existing clients, like `fatalError(_:file:line:)`, have only one parameter for filename information. Adding a second would break ABI, and there would not be a way to concatenate the caller's `#moduleName` and `#fileName` into a single string in a default argument. (Magic identifiers only give you the caller's location if they are the only thing in the default argument; something like `file: String = "\(#moduleName)/\(#fileName)"` would capture the callee's location instead.) + +* `#file` provides a standard format for this information so that different tools are all on the same page. + +While we don't see many practical uses for `#fileName`, we *do* think there are reasonable uses for `#moduleName`. However, this information can now be easily parsed out of the `#file` string. If we want to also provide a separate `#moduleName`, we can consider that in a separate proposal. + +### Other alternatives + +We considered leaving the *file-string* unspecified rather than specifying it with an unused *disambiguator* field for future expansion. Feedback from the first review convinced us that clients will want to interpret this string, which requires a specified format. + +We considered renaming the `file` parameter of `#sourceLocation` to `filePath`. This would make its meaning clearer, but it seems like unnecessary churn, and `#sourceLocation` is begging for a redesign to address its other shortcomings anyway. + +We considered adding a new magic identifier like `#context` which represents several pieces of contextual information simultaneously. This would not give us most of the privacy and code size improvements we seek because, when passed across optimization barriers like module boundaries, we would have to conservatively generate all of the information the callee might want to retrieve from `#context`. + +We considered making `#file`'s behavior change conditional on enabling a new language version mode. While we're not opposed to that in principle, we don't think the breakage from this change will be severe enough to justify delaying this proposal's implementation until the next release with a language version mode. + +We considered introducing a new alternative to `#file` (e.g. `#fileName`) while preserving the existing meaning of `#file`. However, a great deal of code already uses `#file` and would in practice probably never be converted to `#fileName`. Most of this code would benefit from the new behavior, so we think it would be better to automatically adopt it. (Note that clang supports a `__FILE_NAME__` alternative, but most code still uses `__FILE__` anyway.) + +We considered switching between the old and new `#file` behavior with a compiler flag. However, this creates a language dialect, and compiler flags are not a natural interface for users. + +Finally, we could change the behavior of `#file` without offering an escape hatch. However, we think that the existing behavior is useful in rare circumstances and should not be totally removed. diff --git a/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md b/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md new file mode 100644 index 0000000000..c2423c71e4 --- /dev/null +++ b/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md @@ -0,0 +1,135 @@ +# Allow more characters (like whitespaces and punctuations) for escaped identifiers + +* Proposal: [SE-0275](0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md) +* Author: [Alfredo Delli Bovi](https://github.com/adellibovi) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Rejected** +* Implementation: [apple/swift#28966](https://github.com/apple/swift/pull/28966) +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers/32538/46) + +## Introduction +Swift has a beautiful concise yet expressive syntax. +As part of that, escaped identifiers are adopted to allow usage of reserved keywords. +This proposal wants to extend the character allowance for escaped identifiers with more Unicode scalars, like whitespace and punctuation. + +## Motivation + +Naming could be hard and having descriptive methods, like in tests, may result in declarations that are hard to read because of its lack of whitespace and punctuations or other symbols. Enabling natural language would improve readability. + +```swift +func `test validation should succeed when input is less than ten`() // currently not possible +// vs +func testValidationShouldSucceedWhenInputIsLessThanTen() // camelCase +func test_Validation_Should_Succeed_When_Input_Is_Less_Than_Ten() // camel_Case_Mixed_Snake_Case +func test_validationShouldSucceed_whenInputIs_lessThanTen() //camelCase_Mixed_SnakeCase_Grouped +``` + +Maintainers of different projects under the [Swift Source Compatibility](https://swift.org/source-compatibility/#current-list-of-projects) uses, instead of Swift's method declaration, testing frameworks, like [Quick](https://github.com/Quick/Quick), because (among other reasons) how they can elegantly express tests descriptions. + +Other modern languages like [F#](https://fsharp.org) and [Kotlin](https://kotlinlang.org) saw the value in supporting natural language for escaped identifiers. Today, naming methods with spaces and punctuation are, for those languages, a standard for tests, widely adopted and supported by different test runners and reporting tools. + +Another limitation of identifiers is that they can include digits (0-9), but they can not start with one. +There are certain scenarios where it would be beneficial to be able to start with a digit. +```swift +enum Version { + case `1` // currently not possible + case `1.2` // currently not possible +} +enum HTTPStatus { + case `300` // currently not possible +} +``` +Code generators are also affected, as they are required to mangle invalid code points in identifiers in order to produce correct Swift code. +As an example, asset names to typed values tools, like [R.swift](https://github.com/mac-cain13/R.swift), can not express identifiers like `10_circle` (and others from Apple's SF Symbols), without losing a 1 to 1 map to their original asset names. Having to prefix (i.e.: `_10_circle`), replace (i.e.: `ten_circle`) or strip (i.e.: `_circle`) will affect discoverability of those. + +Non-English idioms, like French, heavily rely on apostrophes or hyphens and trying to replace those code points in an identifier will likely result in a less readable version. + +## Proposed solution +This proposal wants to extend the current grammar for every escaped identifier (properties, methods, types, etc...) by allowing every Unicode scalar except every type of line terminators and back-ticks. + +A declaration to an escaped identifier will follow the existing back-ticked syntax. +```swift +func `test validation should succeed when input is less than ten`() +var `some var` = 0 +``` + +As per referencing. +```swift +`test validation should succeed when input is less than ten`() +foo.`property with space` +``` + +It is important to clarify how escaped identifiers will behave. +In general, an escaped identifier will respect any meaning the non-escaping version may have, if rappresentable. +This means that any current semantic restriction will be respected: +* Dollar identifiers are compiler-reserved names and defining ``` `$identifierNames` ``` will produce an error, as it is already happening. +* Escaped identifiers that can be expressed, within their context, without back-ticks as operators, will be considered operators, therefore they will respect operators semantics. +```swift +static func `+`(lhs: Int, rhs: Int) -> Int // Is an operator +func `test +`() // Is not an operator but a valid method +``` + +The proposal, by allowing a larger set of characters will remove other limitations as, for example, referencing to operators. +```swift +let add = Int.`+` // currently not possible +``` + +### Grammar +This proposal wants to replace the following grammar: +``` +identifier → ` identifier-head identifier-characters opt ` +``` +with: +``` +identifier → ` escaped-identifier ` +escaped-identifier -> Any Unicode scalar value except U+000A (line feed), U+000B (vertical tab), U+000C (form feed), U+000D (carriage return), U+0085 (next line), U+2028 (line separator), U+2029 (paragraph separator) or U+0060 (back-tick) +``` + +### Objective-C Interoperability +Objective-C declarations do not support every type of Unicode scalar value. +If willing to expose an escaped identifier that includes a not supported Objective-C character, we can sanitize it using the existing `@objc` annotation like the following: +```swift +@objc(sanitizedName) +``` + +## Source compatibility +This feature is strictly additive. + +## Effect on ABI stability +This feature does not affect the ABI. + +## Effect on API resilience +This feature does not affect the API. + +## Alternatives considered +It was considered to extend the grammars for methods declaration only, this was later discarded because we want to keep usage consistency and it would be hard to explain why an escaped identifier may support a certain set of characters in a context and a different one in another context. + +It was suggested, as an alternative, for the testing method names use case to add a method attribute: +``` +@test("test validation should succeed when input is less than ten") +func testValidationShouldSucceedWhenInputIsLessThanTen() {} +``` +It was not considered a valid option for few reasons: +* it introduces information redundancy +* it is not applicable for the rest of the issues mentioned above +* adding a new attribute would likely to introduce more complexity to the compiler and to the test runner + +Swift currently treats Unicode values individually, this means that a character that can have different representations (like a-grave: U+00E0 'à' or U+0061 U+0300 'a' + combining grave accent) will be treated as different identifiers. Swift also supports unprintable characters like zero-width character (U+200B) that allows identifiers that look the same be treated as different. +```swift +let ​ = 3 // Currently valid, zero-width character identifier +let space​Here = 3 // Currently valid, with zero-width character between `space` and `Here` +let spaceHere = 3 // Currently valid, does not conflict from the above because represented differently, no zero width character +let à = 3 // U+00E0 // Currently valid +let à = 3 // U+0061 U+0300 // Currently valid, does not conflict from the above because represented differently +``` +While this issue can be related to escaped identifiers too, we believe it should be addressed separately as it is an existing issue that is affecting non-escaping identifiers and other grammars tokens. + +## Future direction +It may be possible, if relevant, to support new lines or back-ticks using a similar raw string literals approach. +```swift +func #`this has a ` back-tick`#() +func ###`this has a +new line`###() +``` + +It was considered that the proposal, by including code points like `<`, `>`, `.` may confuse a possible future runtime API for type retrieval by a string (i.e.: `typeByName("Foo.Bar")`). In that hypothetical scenario using back-ticks as part of string could be sufficient in order to resolve ambiguity. diff --git a/proposals/0276-multi-pattern-catch-clauses.md b/proposals/0276-multi-pattern-catch-clauses.md new file mode 100644 index 0000000000..48bb923696 --- /dev/null +++ b/proposals/0276-multi-pattern-catch-clauses.md @@ -0,0 +1,126 @@ +# Multi-Pattern Catch Clauses + +* Proposal: [SE-0276](0276-multi-pattern-catch-clauses.md) +* Author: [Owen Voorhees](https://github.com/owenv) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift#27776](https://github.com/apple/swift/pull/27776) +* Decision Notes: [Review](https://forums.swift.org/t/accepted-se-0276-multi-pattern-catch-clauses/33220) + +## Introduction + +Currently, each catch clause in a do-catch statement may only contain a single pattern and where clause. This is inconsistent with the behavior of cases in switch statements, which provide similar functionality. It also makes some error handling patterns awkward to express. This proposal extends the grammar of catch clauses to support a comma-separated list of patterns (with optional where clauses), resolving this inconsistency. + +Swift-evolution thread: [Thread](https://forums.swift.org/t/multi-pattern-and-conditionally-compiled-catch-clauses/30246) + +## Motivation + +Currently, Swift only allows up to one pattern and where clause for each catch clause in a do-catch statement, so the following code snippet is not allowed: + +```swift +do { + try performTask() +} catch TaskError.someRecoverableError { // OK + recover() +} catch TaskError.someFailure(let msg), + TaskError.anotherFailure(let msg) { // Not currently valid + showMessage(msg) +} +``` + +Because the above snippet is not valid today, developers frequently end up duplicating code between catch clauses, or writing something like the following instead: + +```swift +do { + try performTask() +} catch let error as TaskError { + switch error { + case TaskError.someRecoverableError: + recover() + case TaskError.someFailure(let msg), + TaskError.anotherFailure(let msg): + showMessage(msg) + } +} +``` + +Nesting the switch inside of the catch clause is awkward and defeats the purpose of supporting pattern matching in catch clauses. Splitting the code up into multiple catch clauses requires duplicating the body, which is also undesirable. Supporting a multi-pattern catch clause would allow for code which is both clearer and more concise. + +## Proposed solution + +Catch clauses should allow the user to specify a comma-separated list of patterns. If an error thrown at runtime in a do block matches any of the patterns in a corresponding catch clause, that catch clause's body should be executed. Similar to switch cases, a user should be able to bind a variable in all patterns of a catch clause and then use it in the body. + +With this change, the code snippet from the motivation section is now valid: + +```swift +do { + try performTask() +} catch TaskError.someRecoverableError { // OK + recover() +} catch TaskError.someFailure(let msg), + TaskError.anotherFailure(let msg) { // Also Allowed + showMessage(msg) +} +``` + +Now, if `performTask` throws either `TaskError.someFailure("message")` or `TaskError.anotherFailure("message")`, the body of the second catch clause will be executed and `showMessage` will be called. + +## Detailed design + +### Grammar Changes + +The revised catch clause grammar is as follows: + +``` +catch-clauses -> catch-clause catch-clauses? + +catch-clause -> 'catch' catch-item-list? code-block + +catch-item-list -> catch-item | + catch-item ',' catch-item-list + +catch-item -> pattern where-clause? | + where-clause +``` + +Note: Expressions with trailing closures are not allowed in any of a catch clause's items to avoid parsing ambiguity. This differs from the behavior of switch cases. + +### Semantics + +If a catch clause has multiple patterns, then its body will be executed if a thrown error matches any one of those patterns, and has not already matched a pattern from a preceding catch clause. Similar to switch cases, catch clauses with multiple patterns may still contain value bindings. However, those bindings must have the same name and type in each pattern. + +## Source compatibility + +This proposal maintains source compatibility. It will only result in code compiling which was considered invalid by older compiler versions. + +## Effect on ABI stability + +This feature has no ABI impact. + +## Effect on API resilience + +This proposal does not introduce any new features which could become part of a public API. + +## Alternatives considered + +### Do nothing + +This is a relatively minor addition to the language, and arguably would see rather limited usage. However, in the cases where it's needed, it helps the user avoid hard-to-maintain error handling code which either duplicates functionality or nests control flow statements in a confusing manner. This proposal also simplifies Swift's pattern matching model by unifying some of the semantics of switch and do-catch statements. + +## Future Directions + +There are a number of possible future directions which could increase the expressiveness of catch clauses. + +### Implicit `$error` or `error` binding for all catch clauses + +Currently, only catch clauses without a pattern have an implicit `error: Error` binding. However, there are some cases where it would be useful to have this binding in all catch clauses to make, for example, re-throwing errors easier. However, using `error` as the identifier for this binding would be a medium-to-large source-breaking change. Instead, we could continue the trend of compiler defined identifiers and use `$error`. `error` in empty catch clauses could then be deprecated and eventually removed in future language versions, a smaller source break. + +This change was not included in this proposal because it is source-breaking and orthogonal. If there is interest in this feature, we should probably consider it as an independent improvement which deserves its own proposal. + +### `fallthrough` support in catch clauses + +Allowing `fallthrough` statements in catch clauses would further unify the semantics of switch cases and catches. However, it is currently undesirable for a number of reasons. First, it would be a source-breaking change for existing do-catch statements which trigger `fallthrough`s in an enclosing switch. Also, there is no historical precedent from other languages for supporting `fallthrough`s in catch statements, unlike switches. Finally, there have not yet been any identified examples of code which would benefit from this functionality. + +### Conditional compilation blocks around catch clauses + +An earlier draft of this proposal also added support for wrapping catch clauses in conditional compilation blocks, similar to the existing support for switch cases. This was removed in favor of keeping this proposal focused on a single topic, and leaving room for a more comprehensive redesign of conditional compilation in the future, as described in [https://forums.swift.org/t/allow-conditional-inclusion-of-elements-in-array-dictionary-literals/16171](https://forums.swift.org/t/allow-conditional-inclusion-of-elements-in-array-dictionary-literals/16171/29). diff --git a/proposals/0277-float16.md b/proposals/0277-float16.md new file mode 100644 index 0000000000..83e40bf265 --- /dev/null +++ b/proposals/0277-float16.md @@ -0,0 +1,126 @@ +# Float16 + +* Proposal: [SE-0277](0277-float16.md) +* Author: [Stephen Canon](https://github.com/stephentyrone) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift#30130](https://github.com/apple/swift/pull/30130) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0277-float16/34121) + +## Introduction + +Introduce the `Float16` type conforming to the `BinaryFloatingPoint` and `SIMDScalar` +protocols, binding the IEEE 754 *binary16* format (aka *float16*, *half-precision*, or *half*), +and bridged by the compiler to the C `_Float16` type. + +* Old pitch thread: [Add `Float16`](https://forums.swift.org/t/add-float16/19370). +* New pitch thread: [Add Float16](https://forums.swift.org/t/add-float16/33019) + +## Motivation + +The last decade has seen a dramatic increase in the use of floating-point types smaller +than (32-bit) `Float`. The most widely implemented is `Float16`, which is used +extensively on mobile GPUs for computation, as a pixel format for HDR images, and as +a compressed format for weights in ML applications. + +Introducing the type to Swift is especially important for interoperability with shader-language +programs; users frequently need to set up data structures on the CPU to +pass to their GPU programs. Without the type available in Swift, they are forced to use +unsafe mechanisms to create these structures. + +In addition, C APIs that use these types simply cannot be imported, making them +unavailable in Swift. + +## Proposed solution + +Add `Float16` to the standard library. + +## Detailed design + +There is shockingly little to say here. We will add: +``` +@frozen +struct Float16: BinaryFloatingPoint, SIMDScalar, CustomStringConvertible { } +``` +The entire API falls out from that, with no additional surface outside that provided by those +protocols. `Float16` will provide exactly the operations that `Float` and `Double` and `Float80` +do for their conformance to these protocols. + +We also need to ensure that the parameter passing conventions followed by the compiler +for `Float16` are what we want; these values should be passed and returned in the +floating-point registers, and vectors should be passed and returned in SIMD registers. + +On platforms that do not have native arithmetic support, we will convert `Float16` to +`Float` and use the hardware support for `Float` to perform the operation. This is +correctly-rounded for every operation except fused multiply-add. A software sequence +will be used to emulate fused multiply-add in these cases (the easiest option is to convert +to `Double`, but other options may be more efficient on some architectures, especially +for vectors). + +## Source compatibility + +N/A + +## Effect on ABI stability + +There is no change to existing ABI. We will be introducing a new type, which will have +appropriate availability annotations. + +## Effect on API resilience + +The `Float16` type would become part of the public API. It will be `@frozen`, so no +further changes will be possible, but its API and layout are entirely constrained by +IEEE 754 and conformance to `BinaryFloatingPoint`, so there are no alternatives +possible anyway. + +## Alternatives considered + +Q: Why isn't it called `Half`? + +A: `FloatN` is the more consistent pattern. Swift already has `Float32`, +`Float64` and `Float80` (with `Float` and `Double` as alternative spellings of `Float32` +and `Float64`, respectively). At some future point we will add `Float128`. Plus, the C +language type that this will bridge to is named `_Float16`. + +`Half` is not completely outrageous as an alias, but we shouldn't add aliases unless +there's a really compelling reason. + +During the pitch phase, feedback was fairly overwhelmingly in favor of `Float16`, though +there are a few people who would like to have both names available. Unless significantly +more people come forward, however, we should make the "opinionated" choice to have a single +name for the type. An alias could always be added with a subsequent minor proposal if +necessary. + +Q: What about ARM's ["alternative half precision"](https://en.wikipedia.org/wiki/Half-precision_floating-point_format#ARM_alternative_half-precision)? +What about [bfloat16](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format)? + +A: Alternative half-precision is no longer supported; ARMv8.2-FP16 only uses the IEEE 754 +encoding. Bfloat is something that we should *also* support eventually, but it's a separate +proposal. and we should wait a little while before doing it. Conformance to IEEE 754 fully +defines the semantics of `Float16` (and several hardware vendors, including Apple have +been shipping processors that implement those semantics for a few years). By contrast, +there are a few proposals for hardware that implements bfloat16; ARM and Intel designed +different multiply-add units for it, and haven't shipped yet (and haven't defined any +operations *other* than a non-homogeneous multiply-add). Other companies have HW in +use, but haven't (publicly) formally specified their arithmetic. It's a moving target, and it +would be a mistake to attempt to specify language bindings today. + +Q: Do we need conformance to `BinaryFloatingPoint`? What if we made it a storage-only format? + +A: We could make it a type that can only be used to convert to/from `Float` and `Double`, +forcing all arithmetic to be performed in another format. However, this would mean that +it would be much harder, in some cases, to get the same numerical result on the CPU and +GPU (when GPU computation is performed in half-precision). It's a very nice convenience +to be able to do a computation in the REPL and see what a GPU is going to do. + +Q: Why not put it in Swift Numerics? + +A: The biggest reason to add the type is for interoperability with C-family and GPU programs +that want to use their analogous types. In order to maximize the support for that interoperability, +and to get the calling conventions that we want to have in the long-run, it makes more sense to put +this type in the standard library. + +Q: What about math library support? + +A: If this proposal is approved, I will add conformance to `Real` in Swift Numerics, providing +the math library functions (by using the corresponding `Float` implementations initially). diff --git a/proposals/0278-package-manager-localized-resources.md b/proposals/0278-package-manager-localized-resources.md new file mode 100644 index 0000000000..5f253cdbf3 --- /dev/null +++ b/proposals/0278-package-manager-localized-resources.md @@ -0,0 +1,316 @@ +# Package Manager Localized Resources + +* Proposal: [SE-0278](0278-package-manager-localized-resources.md) +* Author: [David Hart](https://github.com/hartbit) +* Review Manager: [Boris Buegling](https://github.com/neonichu) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift-package-manager#2535](https://github.com/apple/swift-package-manager/pull/2535), + [apple/swift-package-manager#2606](https://github.com/apple/swift-package-manager/pull/2606) + +## Introduction + +This proposal builds on top of the [Package Manager Resources](0271-package-manager-resources.md) proposal to allow defining localized versions of resources in the SwiftPM manifest and have them automatically accessible at runtime using the same APIs. + +## Motivation + +The recently accepted [Package Manager Resources](0271-package-manager-resources.md) proposal allows SwiftPM users to define resources (images, data file, etc...) in their manifests and have them packaged inside a bundle to be accessible at runtime using the Foundation `Bundle` APIs. Bundles support storing different versions of resources for different localizations and can retrieve the version which makes most sense depending on the runtime environment, but SwiftPM currently offers no way to define those localized variants. + +While it is technically possible to benefit from localization today by setting up a resource directory structure that the `Bundle` API expects and specifying it with a `.copy` rule in the manifest (to have SwiftPM retain the structure), this comes at a cost: it bypasses any platform-custom processing that comes with `.process`, and doesn't allow SwiftPM to provide diagnostics when localized resources are mis-configured. + +Without a way to define localized resources, package authors are missing out on powerful Foundation APIs to have their applications, libraries and tools adapt to different regions and languages. + +## Goals + +The goals of this proposal builds on those of the [Package Manager Resources](0271-package-manager-resources.md) proposal: + +* Making it easy to add localized variants of resources with minimal change to the manifest. + +* Avoiding unintentionally copying files not intended to be localized variants into the product. + +* Supporting platform-specific localized resource types for packages written using specific APIs (e.g. Storyboards, XIBs, strings, and stringsdict files on Apple platforms). + +## Proposed Solution + +The proposed solution for supporting localized resources in Swift packages is to: + +* Add a new optional `defaultLocalization` parameter to the `Package` initializer to define the default localization for the resource bundle. The default localization will be used as a fallback when no other localization for a resource fits the runtime environment. SwiftPM will require that parameter be set if the package contains localized resources. + +* Require localized resources to be placed in directories named after the [IETF Language Tag](https://en.wikipedia.org/wiki/IETF_language_tag) they represent followed by an `.lproj` suffix, or in a special `Base.lproj` directory to open up future support for [Base Internationalization](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPInternational/InternationalizingYourUserInterface/InternationalizingYourUserInterface.html#//apple_ref/doc/uid/10000171i-CH3-SW2) on Apple platforms. While Foundation supports several localization directories which are not valid IETF Language Tags, like `English` or `en_US`, it is recommended to use `en-US` style tags with a two-letter ISO 639-1 or three-letter ISO 639-2 language code, followed by optional region and/or dialect codes separated by a hyphen (see the [CFBundleDevelopmentRegion documentation](https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundledevelopmentregion#)). + +* Add an optional `localization` parameter to the `Resource.process` factory function to allow declaring files outside of `.lproj` directories as localized for the default or base localization. + +* Have SwiftPM diagnose incoherent resource configurations. For example, if a resource has both an un-localized and a localized variant, the localized variant can never be selected by `Foundation` (see the documentation on [The Bundle Search Pattern](https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFBundles/AccessingaBundlesContents/AccessingaBundlesContents.html#//apple_ref/doc/uid/10000123i-CH104-SW7)). + +* Have SwiftPM copy the localized resource to the resource bundle in the right locations for the `Foundation` APIs to find and use them, and generate a `Info.plist` for the resources bundle containing the `CFBundleDevelopmentRegion` key set to the `defaultLocalization`. + +## Detailed Design + +### Declaring Localized Resources + +The `Package` initializer in the `PackageDescription` API gains a new optional `defaultLocalization` parameter with type `LocalizationTag` and a default value of `nil`: + +```swift +public init( + name: String, + defaultLocalization: LocalizationTag = nil, // New defaultLocalization parameter. + pkgConfig: String? = nil, + providers: [SystemPackageProvider]? = nil, + products: [Product] = [], + dependencies: [Dependency] = [], + targets: [Target] = [], + swiftLanguageVersions: [Int]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil +) +``` +`LocalizationTag` is a wrapper around a [IETF Language Tag](https://en.wikipedia.org/wiki/IETF_language_tag), with a `String` initializer and conforming to `Hashable`, `RawRepresentable`, `CustomStringConvertible` and `ExpressibleByStringLiteral`. While a `String` would suffice for now, the type allows for future expansion. + +```swift +/// A wrapper around a [IETF Language Tag](https://en.wikipedia.org/wiki/IETF_language_tag). +public struct LocalizationTag: Hashable { + + /// A IETF language tag. + public let tag: String + + /// Creates a `LocalizationTag` from its IETF string representation. + public init(_ tag: String) { + self.tag = tag + } +} + +extension LocalizationTag: RawRepresentable, ExpressibleByStringLiteral, CustomStringConvertible { + // Implementation. +} +``` + +To allow marking files outside of `.lproj` directories as localized, the `Resource.process` factory function gets a new optional `localization` parameter typed as an optional `LocalizationType`, an enum with two cases: `.default` for declaring a default localized variant, and `.base` for declaring a base-localized resource: + +```swift +public struct Resource { + public enum LocalizationType { + case `default` + case base + } + + public static func process(_ path: String, localization: LocalizationType? = nil) -> Resource +} +``` + +### Localized Resource Discovery + +SwiftPM will only detect localized resources if they are defined with the `.process` rule. When scanning for files with that rule, SwiftPM will tag files inside directories with an `.lproj` suffix as localized variants of a resource. The name of the directory before the `.lproj` suffix identifies which localization they correspond to. For example, an `en.lproj` directory contains resources localized to English, while a `fr-CH.lproj` directory contains resources localized to French for Swiss speakers. + +Files in those special directories represent localized variants of a "virtual" resource with the same name in the parent directory, and the manifest must use that path to reference them. For example, the localized variants in `Resources/en.lproj/Icon.png` and `Resources/fr.lproj/Icon.png` are english and french variants of the same "virtual" resource with the `Resources/Icon.png` path, and a reference to it in the manifest would look like: + +```swift +let package = Package( + name: "BestPackage", + defaultLocalization: "en", + targets: [ + .target(name: "BestTarget", resources: [ + .process("Resources/Icon.png"), + ]) + ] +) +``` + +To support SwiftPM clients for Apple platform-specific resources, SwiftPM will also recognize resources located in `Base.lproj` directories as resources using [Base Internationalization](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPInternational/InternationalizingYourUserInterface/InternationalizingYourUserInterface.html#//apple_ref/doc/uid/10000171i-CH3-SW2) and treat them as any other localized variants. + +In addition to localized resources detected by scanning `.lproj` directories, SwiftPM will also take into account processed resources declared with a `localization` parameter in the manifest. This allows package authors to mark files outside of `.lproj` directories as localized, for example to keep localized and un-localized resources together. Separate post-processing done outside of SwiftPM can provide additional localizations in this case. + +### Validating Localized Resources + +SwiftPM can help package authors by diagnosing mis-configurations of localized resources and other inconsistencies that may otherwise only show up at runtime. To illustrate the diagnostics described below, we define a `Package.swift` manifest with a default localization of `"en"`, and two resource paths with the `.process` rule an one with the `.copy` rule: + +```swift +let package = Package( + name: "BestPackage", + defaultLocalization: "en", + targets: [ + .target(name: "BestTarget", resources: [ + .process("Resources/Processed"), + .copy("Resources/Copied"), + ]) + ] +``` + +#### Sub-directory in Localization Directory + +To avoid overly-complex and ambiguous resource directory structures, SwiftPM with emit an error when a localization directory in a `.process` resource path contains a sub-directory. For example, the following directory structure: + +``` +BestPackage +|-- Sources +| `-- BestTarget +| |-- Resources +| | |-- Processed +| | | `-- en.lproj +| | | `-- directory +| | | `-- file.txt +| | `-- Copied +| | `-- en.lproj +| | `-- directory +| | `-- file.txt +| `-- main.swift +`-- Package.swift +``` + +will emit the following diagnostic: + +``` +error: localization directory `Resources/Processed/en.lproj` in target `BestTarget` contains sub-directories, which is forbidden +``` + +#### Missing Default Localized Variant + +When a localized resource is missing a variant for the default localization, `Foundation` may not be able to find the resource depending on the run environment. SwiftPM will emit a warning to warn against it. For example, the following directory structure: + +``` +BestPackage +|-- Sources +| `-- BestTarget +| |-- Resources +| | |-- Processed +| | | `-- fr.lproj +| | | `-- Image.png +| | `-- Copied +| | `-- fr.lproj +| | `-- Image.png +| `-- main.swift +`-- Package.swift +``` + +will emit the following diagnostic: + + +``` +warning: resource 'Image.png' in target 'BestTarget' is missing a localization for the default localization 'en'; the default localization is used as a fallback when no other localization matches +``` + +#### Un-localized and Localized Variants + +When there exists both an un-localized and localized variant of the same resource, SwiftPM will emit a warning to let users know that the localized variants will never be chosen at runtime, due to the search pattern of `Foundation` APIs (see the documentation on [The Bundle Search Pattern](https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFBundles/AccessingaBundlesContents/AccessingaBundlesContents.html#//apple_ref/doc/uid/10000123i-CH104-SW7)). For example, the following directory structure: + +``` +BestPackage +|-- Sources +| `-- BestTarget +| |-- Resources +| | |-- Processed +| | | |-- en.lproj +| | | | `-- Image.png +| | | `-- Image.png +| | `-- Copied +| | |-- en.lproj +| | | `-- Image.png +| | `-- Image.png +| `-- main.swift +`-- Package.swift +``` + +will emit the following diagnostic: + +``` +warning: resource 'Image.png' in target 'BestTarget' has both localized and un-localized variants; the localized variants will never be chosen +``` + +### Missing Default Localization + +The `defaultLocalization` property is optional and has a default value of `nil`, but its required to provide a valid `LocalizationTag` in the presence of localized resources. SwiftPM with emit an error if that is not the case. For example, the following directory structure: + +``` +BestPackage +|-- Sources +| `-- BestTarget +| |-- Resources +| | `-- en.lproj +| | `-- Localizable.strings +| `-- main.swift +`-- Package.swift +``` + +with the following manifest: + +```swift +let package = Package( + name: "BestPackage", + targets: [ + .target(name: "BestTarget", resources: [ + .process("Resources"), + ]) + ] +``` + +will emit the following diagnostic: + +``` +error: missing manifest property 'defaultLocalization'; it is required in the presence of localized resources +``` + +#### Explicit Localization Resource in Localization Directory + +Explicit resource localization declarations exist to avoid placing resources in localization directories. To avoid any ambiguity, SwiftPM will emit an error when a resource with an explicit localization declaration is inside a localization directory. + +``` +BestPackage +|-- Sources +| `-- BestTarget +| |-- Resources +| | `-- en.lproj +| | `-- Storyboard.storyboard +| `-- main.swift +`-- Package.swift +``` + +with the following manifest: + +```swift +let package = Package( + name: "BestPackage", + defaultLocalization: "en", + targets: [ + .target(name: "BestTarget", resources: [ + .process("Resources", localization: .base), + ]) + ] +``` + +will emit the following diagnostic: + +``` +error: resource 'Storyboard.storyboard' in target 'BestTarget' is in a localization directory and has an explicit localization declaration; choose one or the other to avoid any ambiguity +``` + +### Resource Bundle Generation + +SwiftPM will copy localized resources into the correct locations of the resources bundle for them to be picked up by Foundation. It will also generate a `Info.plist` for that bundle with the `CFBundleDevelopmentRegion` value declared in the manifest: + +```xml + + + + + CFBundleDevelopmentRegion + fr-CH + + +``` + +### Runtime Access + +The Foundation APIs already used to load resources will automatically pick up the correct localization: + +```swift +// Get path to a file, which can be localized. +let path = Bundle.module.path(forResource: "TOC", ofType: "md") + +// Load an image from the bundle, which can be localized. +let image = UIImage(named: "Sign", in: .module, with: nil) +``` + +And other APIs will now work as expected on all platforms Foundation is supported on: + +```swift +// Get localization out of strings files. +var localizedGreeting = NSLocalizedString("greeting", bundle: .module) +``` diff --git a/proposals/0279-multiple-trailing-closures.md b/proposals/0279-multiple-trailing-closures.md new file mode 100644 index 0000000000..0f1b03e4f5 --- /dev/null +++ b/proposals/0279-multiple-trailing-closures.md @@ -0,0 +1,385 @@ +# Multiple Trailing Closures + +* Proposal: [SE-0279](0279-multiple-trailing-closures.md) +* Authors: [Kyle Macomber](https://github.com/kylemacomber), [Pavel Yaskevich](https://github.com/xedin), [Doug Gregor](https://github.com/douggregor), [John McCall](https://github.com/rjmccall) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.3)** +* Previous Revisions: [1st](https://github.com/swiftlang/swift-evolution/blob/d923209a05c3c38c8b735510cf1525d27ed4bd14/proposals/0279-multiple-trailing-closures.md) +* Reviews: [1st](https://forums.swift.org/t/se-0279-multiple-trailing-closures/34255), + [2nd](https://forums.swift.org/t/se-0279-multiple-trailing-closures-amended/35435), + [3rd](https://forums.swift.org/t/accepted-se-0279-multiple-trailing-closures/36141) +* Implementation: [apple/swift#31052](https://github.com/apple/swift/pull/31052) + +## Motivation + +Since its inception, Swift has supported *trailing closure* syntax: a bit of syntactic sugar that lets you "pop" the final argument to a function out of the parentheses when it's a closure. + +This example uses [`UIView.animate(withDuration:animations:)`](https://developer.apple.com/documentation/uikit/uiview/1622418-animate) to fade out a view: + +```swift +// Without trailing closure: +UIView.animate(withDuration: 0.3, animations: { + self.view.alpha = 0 +}) +// With trailing closure: +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} +``` + +Trailing closure syntax has proven to be very popular, and it's not hard to guess why. Especially when an API is crafted with trailing closure syntax in mind, the call site is *easier to read*: it is more concise and less nested, without loss of clarity. + +However, the restriction of trailing closure syntax to *only the final closure* has limited its applicability. This limitation was noticed [very early on in Swift's lifetime](https://www.natashatherobot.com/swift-trailing-closure-syntax/) as "the" problem with trailing closure syntax. + +Consider using [`UIView.animate(withDuration:animations:completion:)`](https://developer.apple.com/documentation/uikit/uiview/1622515-animate) to remove the view once it has finished fading out: + +```swift +// Without trailing closure: +UIView.animate(withDuration: 0.3, animations: { + self.view.alpha = 0 +}, completion: { _ in + self.view.removeFromSuperview() +}) +// With trailing closure +UIView.animate(withDuration: 0.3, animations: { + self.view.alpha = 0 +}) { _ in + self.view.removeFromSuperview() +} +``` + +In this case, the trailing closure syntax is *harder to read*: the role of the trailing closure is unclear, the first closure remains nested, and something about the asymmetry is unsettling. + +Concerns about call-site confusion have led Swift style guides to include rules that prohibit the use of trailing closure syntax when a function call has multiple closure arguments ([1](https://google.github.io/swift/#trailing-closures), [2](https://rules.sonarsource.com/swift/RSPEC-2958)). + +As a result, if we ever need to append an additional closure argument to a function, many of us find ourselves having to rejigger our code more than may seem necessary: + +```swift +// Single closure argument -> trailing closure +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} +// Multiple closure arguments -> no trailing closure +UIView.animate(withDuration: 0.3, animations: { + self.view.alpha = 0 +}, completion: { _ in + self.view.removeFromSuperview() +}) +``` + +## Proposed Solution + +This proposal extends trailing closure syntax to allow additional *labeled* closures to follow the initial unlabeled closure: + +```swift +// Single trailing closure argument +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} +// Multiple trailing closure arguments +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} completion: { _ in + self.view.removeFromSuperview() +} +``` + +This extends the concision and denesting of trailing closure syntax to function calls with multiple closures arguments. And there's no rejiggering required to append an additional trailing closure argument! +Informally, the new syntax rules are: + +* The first trailing closure drops its argument label (like today). +* Subsequent trailing closures require argument labels. + +These rules seem to work well in practice, because functions with multiple closure arguments tend to have one that is more primary. Often any additional closure arguments are optional (via default parameter values or overloading), in order to provide progressive disclosure: + + +1. We've already seen UIKit's [`UIView.animate(withDuration:animations:)`](https://developer.apple.com/documentation/uikit/uiview/1622418-animate) and [`UIView.animate(withDuration:animations:completion:)`](https://developer.apple.com/documentation/uikit/uiview/1622515-animate) functions. + +2. Consider Combine's [sink](https://developer.apple.com/documentation/combine/publisher/3343978-sink) operator, which today contorts itself to the existing trailing closure rules: + + ```swift + ipAddressPublisher + .sink { identity in + self.hostnames.insert(identity.hostname!) + } + + ipAddressPublisher + .sink(receiveCompletion: { completion in + // handle error + }) { identity in + self.hostnames.insert(identity.hostname!) + } + ``` + + ... but could be re-worked in light of multiple trailing closure syntax: + + ```swift + ipAddressPublisher + .sink { identity in + self.hostnames.insert(identity.hostname!) + } + + ipAddressPublisher + .sink { identity in + self.hostnames.insert(identity.hostname!) + } receiveCompletion: { completion in + // handle error + } + ``` + +3. Consider SwiftUI's [Section](https://developer.apple.com/documentation/swiftui/section) view, which today avoids using `@ViewBuilder` closures for its optional header and footer: + + ```swift + Section { + // content + } + Section(header: ...) { + // content + } + Section(footer: ...) { + // content + } + Section( + header: ..., + footer: ... + ) { + // content + } + ``` + + ... but could be re-worked in light of multiple trailing closure syntax: + + ```swift + Section { + // content + } + Section { + // content + } header: { + ... + } + Section { + // content + } footer: { + ... + } + Section { + // content + } header: { + ... + } footer: { + ... + } + ``` + +When using multiple trailing closure syntax, these APIs are all [clear at the point of use](https://swift.org/documentation/api-design-guidelines/#fundamentals), without the need to label the first trailing closure. + +If labelling the first trailing closure were allowed, users would have to evaluate whether to include the label on a case by case basis, which would inevitably lead to linter and style guide rules to prohibit it. So, in conjunction with the new syntax rules, we propose an amendment to the [API Design Guidelines](https://swift.org/documentation/api-design-guidelines): + +> Name functions assuming that the argument label of the first trailing closure will be dropped. Include meaningful argument labels for all subsequent trailing closures. + +## Detailed Design + +The grammar is modified as follows to accommodate labeled trailing closures following an unlabeled trailing closure: + +``` +expr-trailing-closure: + expr-postfix(trailing-closure) trailing-closures + +trailing-closures: + expr-closure + trailing-closures (identifier|keyword|'_') ':' expr-closure +``` + +This introduces a zero-lookahead ambiguity between the start of a labeled trailing closure and the start of either a new expression, a labeled statement, or a `default:` label followed by `'{'`. The first two ambiguities can be resolved by looking forward at most two tokens, because `(identifier|keyword|'_') ':'` can never start an expression, only a labeled statement, and the statement in a labeled statement can never start with `'{'` while `expr-closure` must start with it. The ambiguity with `default:` can be resolved by not allowing the unescaped `default` keyword as a label in this syntax; it can still be used if necessary by escaping it (i.e. `` `default`: ``). Source compatibility requires the existing use of `default:` to be preferred, and it's better to do this uniformly (even if the syntax does not appear in a `switch`) in order to discourage the use of `default` as a trailing-closure label in APIs, rather than leaving a trap in the language if an API like that is used within a `switch`. + +The labeled trailing closures are associated with the base expression in the same way as the unlabeled trailing closure is today: + +* if the base expression is a call, they are added as extra arguments to that call; +* if the base expression is a subscript, they are added as extra index arguments to that subscript, +* otherwise, an implicit call of the base expression is created using only the trailing closures as arguments. + +The existing trailing-closure feature requires special treatment of the trailing-closure argument by the type checker because of the special power of label omission: the trailing closure can be passed to a parameter that would ordinarily require an argument label. Currently, the special treatment is specific to the final argument. Because this proposal still has an unlabeled trailing-closure argument, we have to generalize that treatment to allow label omission at an intermediate argument. + +Note that labeled trailing closures are required to match labels with a parameter. A labeled trailing closure can use the special label `_` to indicate that it matches an unlabeled parameter, but this *only* matches an unlabeled parameter; it does not have the special label-omission powers of the initial unlabeled trailing closure. For example: + +```swift +func pointFromClosures( + x: () -> Int, + _ y: () -> Int +) -> (Int, Int) { + (x(), y()) +} +pointFromClosures { 10 } _: { 20 } // Ok + +func performAsync( + action: @escaping () -> Void, + completionOnMainThread: @escaping () -> Void +) { + ... +} +performAsync { + // some action +} _: { // Not okay: must use completionOnMainThread: + window.exit() +} +``` + +In the current representation, the AST maintains a flag indicating whether the last argument was a trailing closure. This is no longer sufficient, and instead the AST must maintain the exact position of the first trailing closure in the argument list. + +The current type-checking rule for trailing closures performs a limited backwards scan through the parameters looking for a parameter that is compatible with a trailing closure. The proposed type-checking rule builds on this while seeking to degenerate to the old behavior when there are no labeled trailing closures. This is done by performing a backwards scan through the parameters to bind all the labeled trailing closures to parameters using label-matching, then doing the current limited scan for the unlabeled trailing closure starting from the last labeled parameter that was matched. + +For example, given this function: + +```swift +func when( + _ condition: @autoclosure () -> Bool, + then: () -> T, + `else`: () -> T +) -> T { + condition() ? then() : `else`() +} +``` + +The following call using the new syntax: + +```swift +when(2 < 3) { + print("then") +} else: { + print("else") +} +``` + +is equivalent to: + +```swift +when(2 < 3, then: { print("then") }, else: { print("else") }) +``` + +It's important to note that the handling of default arguments in relation to trailing closures is maintained as-is. For example: + +```swift +func foo(a: () -> Int = { 42 }, b: Any? = nil) {} + +foo { + 42 +} +``` + +Although trailing closure matches parameter for `a:` by type, existing trailing closure behavior would match trailing closure argument to parameter labeled as `b:`, which means that previous call to foo is equivalent to: + +```swift +foo(b: { 42 }) +``` + +Now let's add one more parameter to foo to see how this applies to the new multiple trailing closures syntax: + +```swift +func foo(a: () -> Int = { 42 }, b: Any? = nil, c: () -> Void) {} + +foo { + 42 +} c: { + ... +} +``` + +Since the new type-checking rule dictates a backwards scan starting for the last (labeled) trailing closure before attempting to match an unlabeled argument, this call is equivalent to a following "old" syntax: + +```swift +foo(b: { 42 }, c: { ... }) +``` + +This shows that unlabeled trailing closure matching behaves exactly the same way in both scenarios. + +There are reasonable arguments against the backwards-scan design for type-checking trailing closures. Perhaps the strongest argument is that this interaction with default arguments is unintuitive and limiting. For example, it is natural to want to take a primary, required closure, followed by some optional closures: + +```swift +func resolve( + id: UUID, + action: (Object) -> Void, + completion: (() -> Void)? = nil, + onError: ((Error) -> Void)? = nil +) { + ... +} +``` + +Under the proposed type-checking rule, code like the following will not type-check as expected: + +```swift +resolve(id: paulID) { paul in + // do something with object +} onError: { error in + // handle error +} +``` + +It is tempting to try to take advantage of the introduction of this new syntax to use a better type-checking rule that would handle this correctly. This would help in some cases. However, unfortunately, when the programmer omits both of the optional closures, they’re no longer using this new syntax, because they’ve dropped back to the existing trailing-closure case: + +```swift +resolve(id: paulID) { paul in + // do something with object +} +``` + +The behavior of this call can’t be changed without potentially breaking source compatibility. That might be worthwhile to do in order to enable these sorts of APIs and get more consistent type-checking behavior for trailing closures; however, it will need its own proposal, and it will only be feasible under a new source-compatibility mode. We recommend considering this for Swift 6. In the meantime, library designers will have to use overloading to get this effect instead of default arguments. + +## Alternatives Considered + +### Trailing Block of Closures + +Multiple trailing closures could alternatively be specified within a trailing block, with each trailing closure indicated by an argument label: + +```swift +UIView.animate(withDuration: 0.3) { + animations: { + self.view.alpha = 0 + } + completion: { _ in + self.view.removeFromSuperview() + } +} +``` + +While this syntax is clear at the point of use, pleasant to read, and provides contextual cues by separating the trailing closures from the rest of the arguments, it risks evolving into an alternative calling syntax. The proposed syntax is more concise and less nested, without loss of clarity: + +```swift +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} completion: { _ in + self.view.removeFromSuperview() +} +``` + +### Optionally Labeled First Trailing Closure + +The proposed syntax could be extended to allow users to optionally label the first trailing closure: + +```swift +ipAddressPublisher + .sink receiveCompletion: { completion in + // handle error + } +``` + +This would allow the user to disambiguate when the backwards-scan would have otherwise resolved differently, in this case for the declaration: + +```swift +public func sink( + receiveCompletion: ((Subscribers.Completion) -> Void)? = nil, + receiveValue: ((Output) -> Void)? = nil +) -> Subscribers.Sink +``` + +However, it shouldn’t be used to disambiguate for fellow humans. Recall: API authors should be naming functions assuming that the argument label of the first trailing closure will be dropped. Swift users aren’t used to seeing function names and argument labels juxtaposed without parenthesis. Many find this spelling unsettling. + +Improving the type-checking rule, as described in Detailed Design, is a more promising avenue for addressing this use case. In the meantime, users that find themselves in this situation can use the existing syntax: + +```swift +ipAddressPublisher + .sink(receiveCompletion: { completion in + // handle error + }) +``` diff --git a/proposals/0280-enum-cases-as-protocol-witnesses.md b/proposals/0280-enum-cases-as-protocol-witnesses.md new file mode 100644 index 0000000000..b837d99485 --- /dev/null +++ b/proposals/0280-enum-cases-as-protocol-witnesses.md @@ -0,0 +1,189 @@ +# Enum cases as protocol witnesses + +* Proposal: [SE-0280](0280-enum-cases-as-protocol-witnesses.md) +* Author: [Suyash Srijan](https://github.com/theblixguy) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift#28916](https://github.com/apple/swift/pull/28916) +* Bug: [SR-3170](https://bugs.swift.org/browse/SR-3170) +* Toolchain: [macOS](https://ci.swift.org/job/swift-PR-toolchain-osx/477/artifact/branch-master/swift-PR-28916-477-osx.tar.gz) & [Linux](https://ci.swift.org/job/swift-PR-toolchain-Linux/346/artifact/branch-master/swift-PR-28916-346-ubuntu16.04.tar.gz) +* Review: ([1](https://forums.swift.org/t/se-0280-enum-cases-as-protocol-witnesses/34257)) ([Acceptance](https://forums.swift.org/t/acceepted-se-0280-enum-cases-as-protocol-witnesses/34850)) + +## Introduction + +The aim of this proposal is to lift an existing restriction, which is that enum cases cannot participate in protocol witness matching. + +Swift-evolution thread: [Enum cases as protocol witnesses](https://forums.swift.org/t/enum-cases-as-protocol-witnesses/32753) + +## Motivation + +Currently, Swift has a very restrictive protocol witness matching model where a protocol witness has to match _exactly_ with the requirement, with some exceptions (see [Protocol Witness Matching Manifesto](https://forums.swift.org/t/protocol-witness-matching-mini-manifesto/32752)). + +For example, if one writes a protocol with static requirements: + +```swift +protocol DecodingError { + static var fileCorrupted: Self { get } + static func keyNotFound(_ key: String) -> Self +} +``` + +and attempts to conform an enum to it, then writing a case with the same name (and arguments) is not considered a match: + +```swift +enum JSONDecodingError: DecodingError { + case fileCorrupted // error, because it is not a match + case keyNotFound(_ key: String) // error, because it is not a match +} +``` + +This is quite surprising, because even though cases are not _written_ as a `static var` or `static func`, they do _behave_ like one both syntactically and semantically throughout the language. For example: + +```swift +enum Foo { + case bar(_ value: Int) + case baz +} + +let f = Foo.bar // `f` is a function of type (Int) -> Foo +let bar = f(2) // Returns Foo +let baz = Foo.baz // Returns Foo +``` + +is the same as: + +```swift +struct Foo { + static func bar(_ value: Int) -> Self { ... } + static var baz: Self { ... } +} + +let f = Foo.bar // `f` is a function of type (Int) -> Foo +let bar = f(2) // Returns Foo +let baz = Foo.baz // Returns Foo +``` + +Such "spelling" exceptions exist when matching other kinds of requirements as well, for example: + +```swift +protocol Foo { + var somePropertyA: Self { get } +} + +struct ImplementsFoo: Foo { + // This can be a 'let' because even though the + // keywords don't match, and a variable and a + // constant are two different things, the + // *semantics* of 'var ... { get }' and 'let' + // do match. + let somePropertyA: Self + // and you can write it as a 'var' if you want + // and still keep the semantics the same. + var somePropertyA: Self +} +``` + +Now, because enum cases are not considered as a "witness" for static protocol requirements, one has to provide a manual implementation instead: + +```swift +enum JSONDecodingError: DecodingError { + case _fileCorrupted + case _keyNotFound(_ key: String) + static var fileCorrupted: Self { return ._fileCorrupted } + static func keyNotFound(_ key: String) -> Self { + return ._keyNotFound(key) + } +} +``` + +This leads to some rather unfortunate consequences: + +1. The ability to write a case with the same name as the requirement is lost. Now, you can rename the case to something different, but it might not always be ideal, especially because naming things right is a really hard problem. In most cases, you expect the case to be named the same as the requirement. +2. The namespace of the enum is now polluted with both cases and requirements (for example, in the snippet above we have `_fileCorrupted` and `fileCorrupted`), which can be confusing during code completion. +3. There's extra code that now has to be maintained and which arguably should not exist in the first place. + +In almost every corner of the language, enum cases and static properties/functions are indistinguishable from each other, *except* when it comes to matching protocol requirements, which is very inconsistent, so it is not unreasonable to think of a enum case without associated values as a `static`, get-only property that returns `Self` or an enum case with associated values as a `static` function (with arguments) that returns `Self`. + +Lifting this restriction can also lead to other improvements, for example, one can conform `DispatchTimeInterval` directly to Combine's `SchedulerTypeIntervalConvertible` instead of having to go through a much more complicated type like `DispatchQueue.SchedulerTimeType.Stride`: + +```swift +extension DispatchTimeInterval: SchedulerTimeIntervalConvertible { + public static func seconds(_ s: Double) -> Self { + return DispatchTimeInterval.seconds(Int((s * 1000000000.0).rounded())) + } + // Remaining requirements already satisfied by cases +} +``` + +## Proposed Solution + +The current restriction is lifted and the compiler allows a static protocol requirement to be witnessed by an enum case, under the following rules: + +1. A static, get-only protocol requirement having an enum type or `Self` type can be witnessed by an enum case with no associated values. +2. A static function requirement with arguments and returning an enum type or `Self` type can be witnessed by an enum case with associated values having the same argument list as the function's. + +This means the example from the motivation section will successfully compile: + +```swift +enum JSONDecodingError: DecodingError { + case fileCorrupted // okay + case keyNotFound(_ key: String) // okay +} +``` + +This also means the mental model of an enum case will now be _more_ consistent with static properties/methods and an inconsistency in the language will be removed. + +You will still be able to implement the requirement manually if you want and code that currently compiles today (with the manual implementation) will continue to compile. However, you will now have the option to let the case satisfy the requirement directly. + +Here are a few more examples that demonstrate how cases will be matched with the requirements: + +```swift +protocol Foo { + static var zero: FooEnum { get } + static var one: Self { get } + static func two(arg: Int) -> FooEnum + static func three(_ arg: Int) -> Self + static func four(_ arg: String) -> Self + static var five: Self { get } + static func six(_: Int) -> Self + static func seven(_ arg: Int) -> Self + static func eight() -> Self +} + +enum FooEnum: Foo { + case zero // okay + case one // okay + case two(arg: Int) // okay + case three(_ arg: Int) // okay + case four(arg: String) // not a match + case five(arg: Int) // not a match + case six(Int) // okay + case seven(Int) // okay + case eight // not a match +} +``` + +The last one is intentional - there is no way to declare a `case eight()` today (and even when you could in the past, it actually had a different type). In this case, the requirement `static func eight()` can in fact be better expressed as a `static var eight`. In the future, this limitation may be lifted when other kinds of witness matching is considered. + +## Source compatibility + +This does not break source compatibility since it's a strictly additive change and allows code that previously did not compile to now compile and run successfully. + +## Effect on ABI stability + +This does not affect the ABI and does not require new runtime support. + +## Effect on API resilience + +Switching between enum cases and static properties/methods is not a resilient change due to differences in ABI and mangling. Doing so will break binary compatibility, or source compatibility if clients are pattern matching on the cases for example. + + +## Alternatives considered + +- Allow protocol requirements to be declared as `case` instead of `static var` or `static func` - the obvious downside of doing this would be that only enums would be able to adopt such a protocol, which would be unreasonably restrictive because other types like classes and structs having satisfying witnesses would no longer be able to adopt such a protocol. +- Only allow enum cases without associated values to participate out of the box. Ones with associated values will be disallowed unless explicitly marked with a specific annotation to allow them to be used as "factories". It seems unnecessarily restrictive to impose another syntactic barrier, one which does not exist in other types. The semantics of a protocol requirement is up to the author to document and for clients to read and verify before implementing, so adding another annotation does not provide any language improvements. +- Leave the existing behaviour as-is. + +## Future directions + +We can allow for more kinds of witness matching, as described in the [Protocol Witness Matching Manifesto](https://forums.swift.org/t/protocol-witness-matching-mini-manifesto/32752), such as subtyping and default arguments. diff --git a/proposals/0281-main-attribute.md b/proposals/0281-main-attribute.md new file mode 100644 index 0000000000..204ff7e2ba --- /dev/null +++ b/proposals/0281-main-attribute.md @@ -0,0 +1,208 @@ +# `@main`: Type-Based Program Entry Points + +* Proposal: [SE-0281](0281-main-attribute.md) +* Authors: [Nate Cook](https://github.com/natecook1000), [Nate Chandler](https://github.com/nate-chandler), [Matt Ricketson](https://github.com/ricketson) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.3)** +* Implementation: [apple/swift#30693](https://github.com/apple/swift/pull/30693) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0281-main-type-based-program-entry-points/35400) + + +## Introduction + +A Swift language feature for designating a type as the entry point for beginning program execution. Instead of writing top-level code, users can use the `@main` attribute on a single type. Libraries and frameworks can then provide custom entry-point behavior through protocols or class inheritance. + +Related forum threads: + +- [Initial pitch](https://forums.swift.org/t/main-type-based-program-execution/34624) +- [Control of Swift Entry Point](https://forums.swift.org/t/control-of-swift-entry-point/25069) + +## Motivation + +Swift programs start execution at the beginning of a file. This works great for procedural code, and allows simple Swift programs to be as short as a single line, with no special syntax required. + +```swift +// A self-contained Swift program: +print("Hello, world!") +``` + +However, not all kinds of programs naturally fit into this model. For example, a user-facing app launches and runs until it is quit. User interface frameworks like UIKit and AppKit take care of the complexities of launching an application, providing high-level API hooks for defining app behavior. A developer using these frameworks typically does not care about nor interact with the literal starting point of their app’s execution. + +In order to resolve these two models, apps need a small amount of “boot-loading” code to kick off the framework’s preferred execution entry point. Ever since its initial release, Swift has provided the domain-specific attributes `@UIApplicationMain` and `@NSApplicationMain` to smooth over this startup process for developers of UIKit and AppKit applications. + +Instead of these hard-coded, framework-specific attributes, it would be ideal for Swift to offer a more general purpose and lightweight mechanism for delegating a program’s entry point to a designated type. A type-based approach to program execution fits within Swift's general pattern of solving problems through the type system and allows frameworks to use standard language features to provide clean, simple entry point APIs. + +## Proposed solution + +The Swift compiler will recognize a type annotated with the `@main` attribute as providing the entry point for a program. Types marked with `@main` have a single implicit requirement: declaring a static `main()` method. This `main()` method will typically be provided by libraries or frameworks, so that the author of a Swift program will only need to use the `@main` attribute to indicate the correct starting point. + +When the program starts, the static `main()` method is called on the type marked with `@main`. For example, this code: + +```swift +// In a framework: +public protocol ApplicationRoot { + // ... +} +extension ApplicationRoot { + public static func main() { + // ... + } +} + +// In MyProgram.swift: +@main +struct MyProgram: ApplicationRoot { + // ... +} +``` + +is equivalent to this code: + +```swift +// In a framework: +public protocol ApplicationRoot { + // ... +} +extension ApplicationRoot { + public static func main() { + // ... + } +} + +// In MyProgram.swift: +struct MyProgram: ApplicationRoot { + // ... +} + +// In 'main.swift': +MyProgram.main() +``` + +Since `main()` is a regular static method, it can be supplied by a protocol as an extension method or by a base class. This allows frameworks to easily define custom entry point behavior without additional language features. + +For example, the `ArgumentParser` library offers a `ParsableCommand` protocol that provides a default implementation for `main():` + +```swift +// In ArgumentParser: +public protocol ParsableCommand { + // Other requirements +} + +extension ParsableCommand { + static func main() { + // Parses the command-line arguments and then + // creates and runs the selected command. + } +} +``` + +The `@main` attribute would allow clients to focus on just the requirements of their command-line tool, rather than how to launch its execution: + +```swift +@main +struct Math: ParsableCommand { + @Argument(help: "A group of integers to operate on.") + var values: [Int] + + func run() throws { + let result = values.reduce(0, +) + print(result) + } +} +``` + +Likewise, UIKit and AppKit could add the static `main()` method to the `UIApplicationDelegate` and `NSApplicationDelegate` protocols, allowing authors to use the single `@main` attribute no matter the user interface framework, and allowing for the deprecation of the special-purpose attributes. (Note: These changes are not a part of this proposal.) + +## Detailed design + +The compiler will ensure that the author of a program only specifies one entry point: either a single, non-generic type designated with the `@main` attribute *or* a single `main.swift` file. The type designated as the entry point with `@main` can be defined in any of a module's source files. `@UIApplicationMain` and `@NSApplicationMain` will be counted the same way as the `@main` attribute when guaranteeing the uniqueness of a program's entry point. + +A `main.swift` file is always considered to be an entry point, even if it has no top-level code. Because of this, placing the `@main`-designated type in a `main.swift` file is an error. + +`@main` can be applied to either a type declaration or to an extension of an existing type. The `@main`-designated type can be declared in the application target or in an imported module. `@main` can be applied to the base type of a class hierarchy, but is not inherited — only the specific annotated type is treated as the entry point. + +The rules for satisfying the `main()` requirement are the same as for satisfying a hypothetical protocol with a single requirement: + +```swift +protocol ProvidesMain { + static func main() throws +} +``` + +The `main()` method can be provided by the type itself, inherited from a superclass, or declared in an extension to a protocol the type conforms to. The `main()` method can either be declared as `throws` or not; errors thrown from a `main()` method will have the same behavior as errors thrown from top-level code. + +## Other considerations + +### Source compatibility + +This is a purely additive change and has no source compatibility impacts. + +### Effect on ABI stability and API resilience + +The new attribute is only applicable to application code, so there is no effect on ABI stability or API resilience. + +### Effect on SwiftPM packages + +The `@main` attribute will currently not be usable by Swift packages, since SwiftPM recognizes executable targets by looking for a `main.swift` file. + +## Alternatives + +### Use a special protocol instead of an attribute + +The standard library includes several protocols that the compiler imbues with special functionality, such as expressing instances of a type using literals and enabling `for`-`in` iteration for sequences. It would similarly be possible to define a `protocol Main` instead of creating a `@main` attribute, with the same requirements and special treatment. However, such a protocol wouldn’t enable any useful generalizations or algorithms, and the uniqueness requirement is totally non-standard for protocol conformance. These factors, plus the precedent of `@UIApplicationMain` and `@NSApplicationMain`, make the attribute a more appropriate way to designate a type as an entry point. + +### Allow `@main` to be applied to protocols + +One or more protocols could be attributed with `@main` to make any type that conforms an automatic entry point, with the compiler ensuring that only one such type exists in an application. As noted above, however, this uniqueness requirement is non-standard for protocols. In addition, this would make the entry point less explicit from the perspective of the program's author and maintainers, since the entry-point conforming type would look the same as any other. Likewise, this would prevent using _manual_ execution if a programmer still wanted to have custom logic in a `main.swift` file. + +### Use a `@propertyWrapper`-style type instead of an attribute + +Instead of a dedicated `@main` attribute, the compiler could let libraries declare a type that could act as an attribute used to denote a program's entry point. This approach is largely isomorphic to the proposed `@main` attribute, but loses the consistency of having a single way to spell the entry point, no matter which library you're using. + +### Use an instance instead of static method + +Instead of requiring a static `main()` method, the compiler could instead require `main()` as an instance method and a default initializer. This, however, would increase the implicit requirements of the `@main` attribute and split the entry point into two separate calls. + +In addition, a default initializer may not make sense for every type. For example, a web framework could offer a `main()` method that loads a configuration file and instantiates the type designated with `@main` using data from that configuration. + +### Use a different name for `main()` + +Some types may already define a static `main()` method with a different purpose than being a program’s entry point. A different, more specific name could avoid some of these potential collisions. + +However, adding `@main` isn’t source- or ABI-breaking for those types, as authors would already need to update their code with the new attribute and recompile to see any changed behavior. In addition, the `main` name matches the existing behavior of `main.swift`, as well as the name of the entry point in several other languages—C, Java, Rust, Kotlin, etc. all use functions or static methods named `main` as entry points. + +### Use `(Int, [String]) -> Int` or another signature for `main()` + +C programs define a function with the signature `int main(int argc, char *argv[])` as their entry point, with access to any arguments and returning a code indicating the status of the program. Swift programs have access to arguments through the `CommandLine` type, and can use `exit` to provide a status code, so the more complicated signature isn't strictly necessary. + +To eliminate any overhead in accessing arguments via `CommandLine` and to provide a way to handle platform-specific entry points, a future proposal could expand the ways that types can satisfy the `@main` requirement. For example, a type could supply either `main()` or `main(Int, [String]) -> Int`. + +Some platforms, such as Windows, base an executable's launch behavior on the specific entry point that the executable provides. A future direction could be to allow `@main` designated types to supply other specific entry points, such as `wWindowsMain(Int, UnsafeMutablePointer>) -> Int`, and to allow additional arguments to be given with the `@main` attribute: + +```swift +// In a framework: +extension ApplicationRoot { + static func main(console: Bool = true, ...) { ... } +} + +@main(console: false) +struct MyApp: Application { + // ... +} +``` + +Alternatively, a future proposal could add an attribute that would let a library designate a different symbol name for the entry point: + +```swift +// In a framework: +extension ApplicationRoot { + @entryPoint(symbolName: "wWinMain", convention: stdcall) + static func main(_ hInstance: HINSTANCE, _ hPrevInstance: HINSTANCE, lpCmdLine: LPWSTR, _ nCmdShow: Int32) { ... } +} +``` + +### Return `Never` instead of `Void` + +A previous design of this feature required the static `main()` method to return `Never` instead of `Void`, since that more precisely matches the semantics of how the method is used when invoked by the compiler. That said, because you can’t provide any top-level code when specifying the `@main` attribute, the `Void`-returning version of the method effectively acts like a `() -> Never` function. + +In addition, returning `Void` allows for more flexibility when the `main()` method is called manually. For example, an author might want to leave out the `@main` attribute and use top-level code to perform configuration, diagnostics, or other behavior before or after calling the static `main()` method. diff --git a/proposals/0282-atomics.md b/proposals/0282-atomics.md new file mode 100644 index 0000000000..21419fbb3c --- /dev/null +++ b/proposals/0282-atomics.md @@ -0,0 +1,314 @@ +# Clarify the Swift memory consistency model ⚛︎ + +* Proposal: [SE-0282](0282-atomics.md) +* Author: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Bug: [SR-9144](https://bugs.swift.org/browse/SR-9144) +* Implementation: Proof of concept [swift-atomics package][package] +* Previous Revision: [v1][SE-0282v1] ([Returned for revision](https://forums.swift.org/t/se-0282-low-level-atomic-operations/35382/69)) +* Status: **Implemented (Swift 5.3)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0282-interoperability-with-the-c-atomic-operations-library/38050) + +[SE-0282v1]: https://github.com/swiftlang/swift-evolution/blob/3a358a07e878a58bec256639d2beb48461fc3177/proposals/0282-atomics.md +[package]: https://github.com/apple/swift-atomics + +## Introduction + +This proposal adopts a C/C++-style weak concurrency memory model in Swift, describing how Swift code interoperates with concurrency primitives imported from C. + +This enables intrepid library authors to start building concurrency constructs in (mostly) pure Swift. + +Original swift-evolution thread: [Low-Level Atomic Operations](https://forums.swift.org/t/low-level-atomic-operations/34683) + +## Revision History + +- 2020-04-13: Initial proposal version. +- 2020-06-05: First revision. + - Removed all new APIs; the proposal is now focused solely on C interoperability. + +## Table of Contents + + * [Motivation](#motivation) + * [Proposed Solution](#proposed-solution) + * [Amendment to The Law of Exclusivity](#amendment-to-the-law-of-exclusivity) + * [Considerations for Library Authors](#considerations-for-library-authors) + * [Interaction with Non\-Instantaneous Accesses](#interaction-with-non-instantaneous-accesses) + * [Interaction with Implicit Pointer Conversions](#interaction-with-implicit-pointer-conversions) + * [Source Compatibility](#source-compatibility) + * [Effect on ABI Stability](#effect-on-abi-stability) + * [Effect on API Resilience](#effect-on-api-resilience) + * [Alternatives Considered](#alternatives-considered) + * [References](#references) + +## Motivation + +In Swift today, application developers use dispatch queues and Foundation's NSLocking protocol to synchronize access to mutable state across concurrent threads of execution. + +However, for Swift to be successful as a systems programming language, it needs to also be possible to implement such synchronization constructs (and many more!) directly within Swift. To allow this, we need to start describing a concurrency memory model. + +Given how deeply Swift interoperates with C, it seems reasonable to assume that Swift's concurrency memory model is compatible with the one described in the C standard. In fact, given that a large amount of existing Swift code deeply depends on concurrency constructs imported from C (most prominently the Dispatch library, but also POSIX Threads, `stdatomic.h` and others), fully embracing interoperability is very likely to be the only practical choice. Therefore, this proposal does exactly that -- it describes how C's atomic operations and memory orderings interact with Swift code. + +Having a reasonably well-defined meaning for the low-level atomic constructs defined for the C (and C++) memory model is crucial for people who wish to implement synchronization constructs or concurrent data structures directly in Swift. (Note that this is a hazardous area that is full of pitfalls. We expect that the higher-level synchronization tools that can be built on top of these atomic primitives will provide a nicer abstraction layer.) + +Note that while this proposal doesn't include a high-level concurrency design for Swift, it also doesn't preclude the eventual addition of one. Indeed, we expect that embracing a compatible concurrency memory model will serve as an important step towards language-level concurrency, by making it easier for motivated people to explore the design space on a library level. + +## Proposed Solution + +We propose to adopt a C/C++-style concurrency memory model for Swift code: + +* Concurrent write/write or read/write access to the same location in memory generally remains undefined/illegal behavior, unless all such access is done through a special set of primitive *atomic operations*. + +* The same atomic operations can also apply *memory ordering* constraints that establish strict before/after relationships for accesses across multiple threads of execution. Such constraints can also be established by explicit *memory fences* that aren't tied to a particular atomic operation. + +This document does not define a formal concurrency memory model in Swift, although we believe the methodology and tooling introduced for the C and C++ memory model and other languages could be adapted to work for Swift, too [[C18], [C++17], [Boehm 2008], [Batty 2011], [Nienhuis 2016], [Mattarei 2018]]. This proposal also doesn't come with any native concurrency primitives; it merely describes how C's preexisting constructs (`atomic_load_explicit`, `atomic_thread_fence`, etc.) can be used to synchronize Swift code. + +When applied carefully, atomic operations and memory ordering constraints can be used to implement higher-level synchronization algorithms that guarantee well-defined behavior for arbitrary variable accesses across multiple threads, by strictly confining their effects into some sequential timeline. + +For now, we will be heavily relying on the Law of Exclusivity as defined in [[SE-0176]] and the [[Ownership Manifesto]], and we'll explain to what extent C's memory orderings apply to Swift's variable accesses. The intention is that Swift's memory model will be fully interoperable with its C/C++ counterparts. + +This proposal does not specify whether/how dependency chains arising from the C/C++ `memory_order_consume` memory ordering work in Swift. The consume ordering as specified in the C/C++ standards is not implemented in any C/C++ compiler, and we join the current version of the C++ standard in encouraging Swift programmers not to use it. We expect to tackle the problem of efficient traversal of concurrent data structures in future proposals. Meanwhile, Swift programmers can start building useful concurrency constructs using relaxed, acquire/release, and sequentially consistent memory orderings imported from C. + + +### Amendment to The Law of Exclusivity + +While the declarations in C's `stdatomic.h` header don't directly import into Swift, it is still possible to access these constructs from Swift code by [wrapping them into plain structs and functions][package] that can be imported. This way, `_Atomic` values can end up being stored within a Swift variable. + +When Swift code is able to acquire a stable pointer to the storage location of such a variable (by e.g. manually allocating it), it ought to be possible to pass this pointer to C's atomic functions to perform atomic operations on its value. Because C's atomic operations (`atomic_load`, `atomic_store`, `atomic_compare_exchange`, etc.) are inherently safe to execute concurrently, we must make sure that the Law of Exclusivity won't disallow them. + +While [[SE-0176]] didn't introduce any active enforcement of the Law of Exclusivity for unsafe pointers, it still defined overlapping read/write access to their pointee as an exclusivity violation. + +To resolve this problem, we propose to introduce the concept of *atomic access*, and to amend the Law of Exclusivity as follows: + +> Two accesses to the same variable aren't allowed to overlap unless both accesses are reads **or both accesses are atomic**. + +We define *atomic access* as a call to one of the following functions in the C atomic operation library: + +```text + atomic_flag_test_and_set atomic_flag_test_and_set_explicit + atomic_flag_clear atomic_flag_clear_explicit + atomic_store atomic_store_explicit + atomic_load atomic_load_explicit + atomic_exchange atomic_exchange_explicit + atomic_compare_exchange_strong atomic_compare_exchange_strong_explicit + atomic_compare_exchange_weak atomic_compare_exchange_weak_explicit + atomic_fetch_add atomic_fetch_add_explicit + atomic_fetch_sub atomic_fetch_sub_explicit + atomic_fetch_or atomic_fetch_or_explicit + atomic_fetch_xor atomic_fetch_xor_explicit + atomic_fetch_and atomic_fetch_and_explicit +``` + +We consider two of these operations to *access the same variable* if they operate on the same memory location. (Future proposals may introduce additional ways to perform atomic access, including native support for atomic operations in the Swift Standard Library.) + +We view the amendment above as merely formalizing pre-existing practice, rather than introducing any actual new constraint. + +> **Note:** As such, this proposal does not need to come with an associated implementation -- there is no need to change how the Swift compiler implements the Swift memory model. For example, there is no need to relax any existing compile-time or runtime checks for exclusivity violations, because unsafe pointer operations aren't currently covered by such checks. Similarly, the existing llvm-based Thread Sanitizer tool [[Tsan1], [TSan2]] already assumes a C-compatible memory model when it is run on Swift code. + +Like C, we leave mixed atomic/non-atomic access to the same memory location as undefined behavior, even if these mixed accesses are guaranteed to never overlap. (This restriction does not apply to accesses during storage initialization and deinitialization; those are always nonatomic.) + +## Considerations for Library Authors + +While this proposal enables the use of C's atomics operations in Swift code, we don't generally recommend calling C atomics API directly. Rather, we suggest wrapping the low-level atomic invocations in more appropriate Swift abstractions. As an example of how this can be done, we've made available a [proof of concept package][package] implementing the APIs originally included in the first version of this proposal. + +In this section we highlight some preexisting aspects of Swift's memory model that need to be taken into account when designing or using a C-based atomics library. + +This section doesn't propose any changes to the language or the Standard Library. + +### Interaction with Non-Instantaneous Accesses + +As described in [[SE-0176]], Swift allows accesses that are non-instantaneous. For example, calling a `mutating` method on a variable counts as a single write access that is active for the entire duration of the method call: + +```swift +var integers: [Int] = ... +... +integers.sort() // A single, long write access +``` + +The Law of Exclusivity disallows overlapping read/write and write/write accesses to the same variable, so while one thread is performing `sort()`, no other thread is allowed to access `integers` at all. Note that this is independent of `sort()`'s implementation; it is merely a consequence of the fact that it is declared `mutating`. + +> **Note:** One reason for this is that the compiler may decide to implement the mutating call by first copying the current value of `integers` into a temporary variable, running `sort` on that, and then copying the resulting value back to `integers`. If `integers` had a computed getter and setter, this is in fact the only reasonable way to implement the mutating call. If overlapping access wasn't disallowed, such implicit copying would lead to race conditions even if the `mutating` method did not actually mutate any data at all. + +While C's atomic memory orderings do apply to Swift's variable accesses, and we can use them to reliably synchronize Swift code, they can only apply to accesses whose duration doesn't overlap with the atomic operations themselves. They inherently cannot synchronize variable accesses that are still in progress while the atomic operation is being executed. Code that relies on memory orderings must be carefully written to take this into account. + +For example, it isn't possible to implement any "thread-safe" `mutating` methods, no matter how much synchronization we add to their implementation. The following attempt to implement an "atomic" increment operation on `Int` is inherently doomed to failure: + +```swift +import Dispatch +import Foundation + +let _mutex = NSLock() + +extension Int { + mutating func atomicIncrement() { // BROKEN, DO NOT USE + _mutex.lock() + self += 1 + _mutex.unlock() + } +} + +var i: Int +... +i = 0 +DispatchQueue.concurrentPerform(iterations: 10) { _ in + for _ in 0 ..< 1_000_000 { + i.atomicIncrement() // Exclusivity violation + } +} +print(i) +``` + +Even though `NSLock` does guarantee that the `self += 1` line is always serialized, any concurrent `atomicIncrement` invocations still count as an exclusivity violation, because the write access to `i` starts when the function call begins, before the call to `_mutex.lock()`. Therefore, the code above has undefined behavior, despite all the locking. (For example, it may print any value between one and ten million, or it may trap in a runtime exclusivity check, or indeed it may do something else.) + +The same argument also applies to property and subscript setters (unless they are declared `nonmutating`), and to `inout` arguments of any function call. + +Methods of types with reference semantics (such as classes) can modify their instance variables without declaring themselves `mutating`, so they aren't constrained by this limitation. (Of course, the implementation of the method must still guarantee that the Law is upheld for any stored properties they themselves access -- but synchronization tools such as locks do work in this context.) + +### Interaction with Implicit Pointer Conversions + + +To simplify interoperability with functions imported from C, Swift provides several forms of implicit conversions from Swift values to unsafe pointers. This often requires the use of Swift's special `&` syntax for passing inout values. At first glance, this use of the ampersand resembles C's address-of operator, and it seems to work in a similar way: + +```swift +func test(_ address: UnsafePointer) + +var value = 42 + +// Implicit conversion from `inout Int` to `UnsafePointer` +test(&value) +``` + +However, despite the superficial similarity, the `&` here isn't an address-of operator at all. Swift variables do not necessarily have a stable location in memory, and even in case they happen to get assigned one, there is generally no reliable way to retrieve the address of their storage. (The obvious exceptions are dynamic variables that we explicitly allocate ourselves.) What the `&`-to-pointer conversion actually does here is equivalent to a call to `withUnsafePointer(to:)`: + +```swift +withUnsafePointer(to: &value) { pointer in + test(pointer) +} +``` + +This counts as a write access to the original value, and (unlike with C) the generated pointer may address a temporary copy of the value -- so it is only considered valid for the duration of the closure call, and the addressed memory location may change every time the code is executed. Because of these two reasons, inout-to-pointer conversions must not be employed to pass "the address" of an atomic value to an atomic operation. + +For example, consider the following constructs, imported from C wrappers of `_Atomic intptr_t`, `atomic_load` and `atomic_fetch_add`: + +```swift +struct AtomicIntStorage { ... } +func atomicLoadInt(_ address: UnsafePointer) -> Int +func atomicFetchAddInt( + _ address: UnsafeMutablePointer, + _ delta: Int +) -> Int +``` + +It is tempting to call these by simply passing an inout reference to a Swift variable of type `AtomicIntStorage`: + +```swift +// BROKEN, DO NOT USE +var counter = AtomicIntStorage() // zero init +DispatchQueue.concurrentPerform(iterations: 10) { _ in + for _ in 0 ..< 1_000_000 { + atomicFetchAddInt(&counter, 1) // Exclusivity violation + } +} +print(atomicLoadInt(&counter) // ??? +``` + +Unfortunately, this code has undefined behavior. `&counter` counts as a write access to `counter`, and as we explained in the previous section, this leads to a clear exclusivity violation. Additionally, `&counter` may result in a different pointer in each thread of execution (or, perhaps even each iteration of the loop), which would defeat atomicity. + +Given that the concurrency in this example is neatly isolated to a single section of code, we could wrap it in a `withUnsafeMutablePointer(to:)` invocation that generates a single (but still temporary) pointer. This resolves the problem: + +```swift +var counter = AtomicIntStorage() // zero init +withUnsafeMutablePointer(to: counter) { pointer in + DispatchQueue.concurrentPerform(iterations: 10) { _ in + for _ in 0 ..< 1_000_000 { + atomicFetchAddInt(pointer, 1) // OK + } + } + print(atomicLoadInt(pointer) // 10_000_000 +} +``` + +However, it isn't always possible to do this. In cases where thread lifetime cannot be restricted to a single code block, the best way to produce a pointer that is suitable for atomic operations is either by manually allocating a dynamic variable, or by using `ManagedBuffer` APIs to retrieve stable pointers to inline storage inside a class instance. + + +## Source Compatibility + +This proposal requires no changes to Swift's implementation; as such, it has no source compatibility impact. + +## Effect on ABI Stability + +None. + +## Effect on API Resilience + +None. + +## Alternatives Considered + +A previous version of this proposal included a large set of APIs implementing a native Swift atomics facility. We expect a revised version of these APIs will return in a followup proposal later (following further work on Swift's [Ownership Manifesto]); however, for now, we expect to develop them [as a standalone package][package], implemented around the operations provided by the C standard library. This C-based reimplementation of the module exports the same public interface and it has the same performance characteristics as the originally proposed native implementation. + +## References + +[Ownership Manifesto]: https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md +**\[Ownership Manifesto]** John McCall. "Ownership Manifesto." *Swift compiler documentation*, May 2, 2017. https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md + +[SE-0176]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0176-enforce-exclusive-access-to-memory.md +**\[SE-0176]** John McCall. "Enforce Exclusive Access to Memory. *Swift Evolution Proposal,* SE-0176, May 2, 2017. https://github.com/swiftlang/swift-evolution/blob/master/proposals/0176-enforce-exclusive-access-to-memory.md + +[Generics Manifesto]: https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md +**\[Generics Manifesto]** Douglas Gregor. "Generics Manifesto." *Swift compiler documentation*, 2016. https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md + +[C++17]: https://isocpp.org/std/the-standard +**\[C++17]** ISO/IEC. *ISO International Standard ISO/IEC 14882:2017(E) – Programming Language C++.* 2017. +https://isocpp.org/std/the-standard + +[C18]: https://www.iso.org/standard/74528.html +**\[C18]** *ISO International Standard ISO/IEC 9899:2018 - Information Technology -- Programming Languages -- C.*. 2018. +https://www.iso.org/standard/74528.html + +**\[Williams 2019]** Anthony Williams. *C++ Concurrency in Action.* 2nd ed., Manning, 2019. + +**\[Nagarajan 2020]** Vijay Nagarajan, Daniel J. Sorin, Mark D. Hill, David A. Wood. *A Primer on Memory Consistency and Cache Coherence.* 2nd ed., Morgan & Claypool, February 2020. https://doi.org/10.2200/S00962ED2V01Y201910CAC049 + +**\[Herlihy 2012]** Maurice Herlihy, Nir Shavit. *The Art of Multiprocessor Programming.* Revised 1st ed., Morgan Kauffmann, May 2012. + +[Boehm 2008]: https://doi.org/10.1145/1375581.1375591 +**\[Boehm 2008]** Hans-J. Boehm, Sarita V. Adve. "Foundations of the C++ Concurrency Memory Model." In *PLDI '08: Proc. of the 29th ACM SIGPLAN Conf. on Programming Language Design and Implementation*, pages 68–78, June 2008. + https://doi.org/10.1145/1375581.1375591 + +[Batty 2011]: https://doi.org/10.1145/1925844.1926394 +**\[Batty 2011]** Mark Batty, Scott Owens, Susmit Sarkar, Peter Sewell, Tjark Weber. "Mathematizing C++ Concurrency." In *ACM SIGPlan Not.,* volume 46, issue 1, pages 55–66, January 2011. https://doi.org/10.1145/1925844.1926394 + +[Boehm 2012]: https://doi.org/10.1145/2247684.2247688 +**\[Boehm 2012]** Hans-J. Boehm. "Can Seqlocks Get Along With Programming Language Memory Models?" In *MSPC '12: Proc. of the 2012 ACM SIGPLAN Workshop on Memory Systems Performance and Correctness*, pages 12–20, June 2012. https://doi.org/10.1145/2247684.2247688 + +[Nienhuis 2016]: https://doi.org/10.1145/2983990.2983997 +**\[Nienhuis 2016]** Kyndylan Nienhuis, Kayvan Memarian, Peter Sewell. "An Operational Semantics for C/C++11 Concurrency." In *OOPSLA 2016: Proc. of the 2016 ACM SIGPLAN Conf. on Object Oriented Programming, Systems, Languages, and Applications,* pages 111–128, October 2016. https://doi.org/10.1145/2983990.2983997 + +[Mattarei 2018]: https://doi.org/10.1007/978-3-319-89963-3_4 +**\[Mattarei 2018]** Christian Mattarei, Clark Barrett, Shu-yu Guo, Bradley Nelson, Ben Smith. "EMME: a formal tool for ECMAScript Memory Model Evaluation." In *TACAS 2018: Lecture Notes in Computer Science*, vol 10806, pages 55–71, Springer, 2018. https://doi.org/10.1007/978-3-319-89963-3_4 + +[N2153]: http://wg21.link/N2153 +**\[N2153]** Raúl Silvera, Michael Wong, Paul McKenney, Bob Blainey. *A simple and efficient memory model for weakly-ordered architectures.* WG21/N2153, January 12, 2007. http://wg21.link/N2153 + +[N4455]: http://wg21.link/N4455 +**\[N4455]** JF Bastien *No Sane Compiler Would Optimize Atomics.* WG21/N4455, April 10, 2015. http://wg21.link/N4455 + +[P0124]: http://wg21.link/P0124 +**\[P0124]** Paul E. McKenney, Ulrich Weigand, Andrea Parri, Boqun Feng. *Linux-Kernel Memory Model.* WG21/P0124r6. September 27, 2018. http://wg21.link/P0124 + +[TSan1]: https://developer.apple.com/documentation/code_diagnostics/thread_sanitizer +**\[TSan1]** *Thread Sanitizer -- Audit threading issues in your code.* Apple Developer Documentation. Retrieved March 2020. https://developer.apple.com/documentation/code_diagnostics/thread_sanitizer + +[TSan2]: https://clang.llvm.org/docs/ThreadSanitizer.html +**\[TSan2]** *ThreadSanitizer*. Clang 11 documentation. Retrieved March 2020. https://clang.llvm.org/docs/ThreadSanitizer.html + + +⚛︎︎ + + + + + + + + diff --git a/proposals/0283-tuples-are-equatable-comparable-hashable.md b/proposals/0283-tuples-are-equatable-comparable-hashable.md new file mode 100644 index 0000000000..963cf56eaf --- /dev/null +++ b/proposals/0283-tuples-are-equatable-comparable-hashable.md @@ -0,0 +1,156 @@ +# Tuples Conform to `Equatable`, `Comparable`, and `Hashable` + +* Proposal: [SE-0283](0283-tuples-are-equatable-comparable-hashable.md) +* Author: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Accepted (2020-05-19)** +* Implementation: [apple/swift#28833](https://github.com/apple/swift/pull/28833) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0283-tuples-conform-to-equatable-comparable-and-hashable/36658), [Additional Commentary](https://forums.swift.org/t/implementation-issues-with-se-0283-tuples-are-ehc/46946) + +## Introduction + +Introduce `Equatable`, `Comparable`, and `Hashable` conformance for all tuples whose elements are themselves `Equatable`, `Comparable`, and `Hashable`. + +Swift-evolution thread: [Tuples Conform to Equatable, Comparable, and Hashable](https://forums.swift.org/t/tuples-conform-to-equatable-comparable-and-hashable/34156) + +## Motivation + +Tuples in Swift currently lack the ability to conform to protocols. This has led many users to stop using tuples altogether in favor of structures that can conform to protocols. The shift — from tuples to structures — has made tuples almost feel like a second-class type in the language, because of them not being able to do simple operations that should *just* work. + +Consider the following snippet of code that naively tries to use tuples for simple operations, but instead is faced with ugly errors. + +```swift +let points = [(x: 128, y: 316), (x: 0, y: 0), (x: 100, y: 42)] +let origin = (x: 0, y: 0) + +// error: type '(x: Int, y: Int)' cannot conform to 'Equatable'; +// only struct/enum/class types can conform to protocols +if points.contains(origin) { + // do some serious calculations here +} + +// error: type '(x: Int, y: Int)' cannot conform to 'Comparable'; +// only struct/enum/class types can conform to protocols +let sortedPoints = points.sorted() + +// error: type '(x: Int, y: Int)' cannot conform to 'Hashable'; +// only struct/enum/class types can conform to protocols +let uniquePoints = Set(points) +``` + +This also creates friction when one needs to conditionally conform to a type, or if a type is just trying to get free conformance synthesis for protocols like `Equatable` or `Hashable`. + +```swift +struct Restaurant { + let name: String + let location: (latitude: Int, longitude: Int) +} + +// error: type 'Restaurant' does not conform to protocol 'Equatable' +extension Restaurant: Equatable {} + +// error: type 'Restaurant' does not conform to protocol 'Hashable' +extension Restaurant: Hashable {} +``` + +These are simple and innocent examples of trying to use tuples in one's code, but currently the language lacks the means to get these examples working and prevents the user from writing this code. + +After all the errors, one decides to give in and create a structure to mimic the tuple layout. From a code size perspective, creating structures to mimic each unique tuple need adds a somewhat significant amount of size to one's binary. + +## Proposed solution + +Introduce `Equatable`, `Comparable`, and `Hashable` conformance for all tuples whose elements themselves conform to said protocols. While this isn't a general-purpose "conform any tuple to any protocol" proposal, `Equatable`, `Comparable`, and `Hashable` are crucial protocols to conform to, because it allows for all of the snippets above in Motivation to compile and run as expected, along with many other standard library operations to work nicely with tuples. + +### Equatable + +The rule is simple: if all of the tuple elements are themselves `Equatable` then the overall tuple itself conforms to `Equatable`. + +```swift +// Ok, Int is Equatable thus the tuples are Equatable +(1, 2, 3) == (1, 2, 3) // true + +struct EmptyStruct {} + +// error: type '(EmptyStruct, Int, Int)' does not conform to protocol 'Equatable' +// note: value of type 'EmptyStruct' does not conform to protocol 'Equatable', +// preventing conformance of '(EmptyStruct, Int, Int)' to 'Equatable' +(EmptyStruct(), 1, 2) == (EmptyStruct(), 1, 2) +``` + +It's also important to note that this conformance does not take into account the tuple labels in consideration for equality. If both tuples have the same element types, then they can be compared for equality. This mimics the current behavior of the operations introduced in [SE-0015](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0015-tuple-comparison-operators.md). + +```swift +// We don't take into account the labels for equality. +(x: 0, y: 0) == (0, 0) // true +``` + +### Comparable + +Comparable conformance for tuples works just like `Equatable`, if all the elements themselves are `Comparable`, then the tuple itself is `Comparable`. Comparing a tuple to a tuple works elementwise: + +> Look at the first element, if they are equal move to the second element. +Repeat until we find elements that are not equal and compare them. + +If all of the elements are equal, we cannot compare them, thus the result is `false`. Of course if we're using `<=` or `>=` and the tuples are exactly the same then the output would be `true`. + +```swift +let origin = (x: 0, y: 0) +let randomPoint = (x: Int.random(in: 1 ... 10), y: Int.random(in: 1 ... 10)) + +// In this case, the first element of origin is 0 and the first element +// of randomPoint is somewhere between 1 and 10, so they are not equal. +// origin's element is less than randomPoint's, thus true. +print(origin < randomPoint) // true +``` + +Just like in `Equatable`, the comparison operations do not take tuple labels into consideration when determining comparability. This mimics the current behavior of the operations introduced in [SE-0015](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0015-tuple-comparison-operators.md). + +```swift +// We don't take into account the labels for comparison. +(x: 0, y: 0) < (1, 0) // true +``` + +### Hashable + +The same rule applies to `Hashable` as it does for `Comparable` and `Equatable`, if all the elements are `Hashable` then the tuple itself is `Hashable`. When hashing a value of a tuple, all of the elements are combined into the hasher to produce the tuple's hash value. Now that tuples are `Hashable`, one can make a set of tuples or create dictionaries with tuple keys: + +```swift +let points = [(x: 0, y: 0), (x: 1, y: 2), (x: 0, y: 0)] +let uniquePoints = Set(points) + +// Create a grid system to hold game entities. +var grid = [(x: Int, y: Int): Entity]() + +for point in uniquePoints { + grid[point]?.move(up: 10) +} +``` + +Once again, `Hashable` doesn't take tuple element labels into consideration when evaluating the hash value of a tuple. Because of this, one is able to index into a set or dictionary with an unlabeled tuple and retrieve elements whose keys were labeled tuples: + +```swift +// We don't take into account the labels for hash value. +(x: 0, y: 0).hashValue == (0, 0).hashValue // true + +grid[(x: 100, y: 200)] = Entity(name: "Pam") + +print(grid[(100, 200)]) // Entity(name: "Pam") +``` + +## Source compatibility + +These are completely new conformances to tuples, thus source compatibility is unaffected as they were previously not able to conform to protocols. + +## Effect on ABI stability + +The conformances to `Equatable`, `Comparable`, and `Hashable` are all additive to the ABI. While at the time of writing this, there is no way to spell a new conformance to an existing type. However, these conformances are being implemented within the runtime which allows us to backward deploy these conformance to Swift 5.0, 5.1, and 5.2 clients. Because these are special conformances being added before other language features allows us to create real conformances, there is a level of runtime support needed to enable these conformances to work properly. Going forward this means we'll need to keep the entry points needed for these to work even after tuples are able to properly conform to protocols. + +## Alternatives considered + +Besides not doing this entirely, the only alternative here is whether or not we should hold off on this before we get proper protocol conformances for tuples which allow them to conform to any protocol. Doing this now requires a lot of builtin machinery in the compiler which some may refer to as technical debt. While I agree with this statement, I don't believe we should be holding off on features like this that many are naturally reaching for until bigger and more complex proposals that allow this feature to natively exist in Swift. I also believe it is none of the user's concern for what technical debt is added to the compiler that allows them to write the Swift code that they feel comfortable writing. In any case, the technical debt to be had here should only be the changes to the runtime (or at least the symbols needed) which allow this feature to work. + +## Future Directions + +With this change, other conformances such as `Codable` might make sense for tuples as well. It also makes sense to implement other conformances for other structural types in the language such as metatypes being `Hashable`, existentials being `Equatable` and `Hashable`, etc. + +In the future when we have proper tuple extensions along with variadic generics and such, implementing these conformances for tuples will be trivial and I imagine the standard library will come with these conformances for tuples. When that happens all future usage of those conformances will use the standard library's implementation, but older clients that have been compiled with this implementation will continue using it as normal. diff --git a/proposals/0284-multiple-variadic-parameters.md b/proposals/0284-multiple-variadic-parameters.md new file mode 100644 index 0000000000..3ec42bab99 --- /dev/null +++ b/proposals/0284-multiple-variadic-parameters.md @@ -0,0 +1,164 @@ +# Allow Multiple Variadic Parameters in Functions, Subscripts, and Initializers + +* Proposal: [SE-0284](0284-multiple-variadic-parameters.md) +* Author: [Owen Voorhees](https://github.com/owenv) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.4)** +* Implementation: [apple/swift#29735](https://github.com/apple/swift/pull/29735) + +## Introduction + +Currently, variadic parameters in Swift are subject to two main restrictions: + +- Only one variadic parameter is allowed per parameter list +- If present, the parameter which follows a variadic parameter must be labeled + +This proposal seeks to remove the first restriction while leaving the second in place, allowing a function, subscript, or initializer to have multiple variadic parameters so long as every parameter which follows a variadic one has a label. + +Swift-evolution thread: [Lifting the 1 variadic param per function restriction](https://forums.swift.org/t/lifting-the-1-variadic-param-per-function-restriction/33787?u=owenv) + +## Motivation + +Variadic parameters allow programmers to write clear, succinct APIs which operate on a variable, but compile-time fixed number of inputs. One prominent example is the standard library's `print` function. However, restricting each function to a single variadic parameter can sometimes be limiting. For example, consider the following example from the `swift-driver` project: + +```swift +func assertArgs( + _ args: String..., + parseTo driverKind: DriverKind, + leaving remainingArgs: ArraySlice, + file: StaticString = #file, line: UInt = #line + ) throws { /* Implementation Omitted */ } + +try assertArgs("swift", "-foo", "-bar", parseTo: .interactive, leaving: ["-foo", "-bar"]) +``` + +Currently, the `leaving:` parameter cannot be variadic because of the preceding unnamed variadic parameter. This results in an odd inconsistency, where the first list of arguments does not require brackets, but the second does. By allowing multiple variadic parameters, it could be rewritten like so: + +```swift +func assertArgs( + _ args: String..., + parseTo driverKind: DriverKind, + leaving remainingArgs: String..., + file: StaticString = #file, line: UInt = #line + ) throws { /* Implementation Omitted */ } + +try assertArgs("swift", "-foo", "-bar", parseTo: .interactive, leaving: "-foo", "-bar") +``` + +This results in a cleaner, more consistent interface. + +Multiple variadic parameters can also be used to streamline lightweight DSL-like functions. For example, one could write a simple autolayout wrapper like the following: + +```swift +extension UIView { + func addSubviews(_ views: UIView..., constraints: NSLayoutConstraint...) { + views.forEach { + addSubview($0) + $0.translatesAutoresizingMaskIntoConstraints = false + } + constraints.forEach { $0.isActive = true } + } +} + +myView.addSubviews(v1, v2, constraints: v1.widthAnchor.constraint(equalTo: v2.widthAnchor), + v1.heightAnchor.constraint(equalToConstant: 40), + /* More Constraints... */) +``` + +## Proposed solution + +Lift the arbitrary restriction on variadic parameter count and allow a function/subscript/initializer to have any number of them. Leave in place the restriction which requires any parameter following a variadic one to have a label. + +## Detailed design + +A variadic parameter can already appear anywhere in a parameter list, so the behavior of multiple variadic parameters in functions and initializers is fully specified by the existing language rules. + +```swift +// Note the label on the second parameter is required because it follows a variadic parameter. +func twoVarargs(_ a: Int..., b: Int...) { } +twoVarargs(1, 2, 3, b: 4, 5, 6) + +// Variadic parameters can be omitted because they default to []. +twoVarargs(1, 2, 3) +twoVarargs(b: 4, 5, 6) +twoVarargs() + +// The third parameter does not require a label because the second isn't variadic. +func splitVarargs(a: Int..., b: Int, _ c: Int...) { } +splitVarargs(a: 1, 2, 3, b: 4, 5, 6, 7) +// a is [1, 2, 3], b is 4, c is [5, 6, 7]. +splitVarargs(b: 4) +// a is [], b is 4, c is []. + +// Note the third parameter doesn't need a label even though the second has a default expression. This +// is consistent with the current behavior, which allows a variadic parameter followed by a labeled, +// defaulted parameter, followed by an unlabeled required parameter. +func varargsSplitByDefaultedParam(_ a: Int..., b: Int = 42, _ c: Int...) { } +varargsSplitByDefaultedParam(1, 2, 3, b: 4, 5, 6, 7) +// a is [1, 2, 3], b is 4, c is [5, 6, 7]. +varargsSplitByDefaultedParam(b: 4, 5, 6, 7) +// a is [], b is 4, c is [5, 6, 7]. +varargsSplitByDefaultedParam(1, 2, 3) +// a is [1, 2, 3], b is 42, c is []. +// Note: it is impossible to call varargsSplitByDefaultedParam providing a value for the third parameter +// without also providing a value for the second. +``` + +This proposal also allows subscripts to have more than one variadic parameter. Like in functions and initializers, a subscript parameter which follows a variadic parameter must have an external label. However, the syntax differs slightly because of the existing labeling rules for subscript parameters: + +```swift +struct HasSubscript { + // Not allowed because the second parameter does not have an external label. + subscript(a: Int..., b: Int...) -> [Int] { a + b } + + // Allowed + subscript(a: Int..., b b: Int...) -> [Int] { a + b } +} +``` + +Note that due to a long-standing bug, the following subscript declarations are accepted by the current compiler: + +```swift +struct HasBadSubscripts { + // Shouldn't be allowed because the second parameter follows a variadic one and has no + // label. Is accepted by the current compiler but can't be called. + subscript(a: Int..., b: String) -> Int { 0 } + + // Shouldn't be allowed because the second parameter follows a variadic one and has no + // label. Is accepted by the current compiler and can be called, but the second + // parameter cannot be manually specified. + subscript(a: Int..., b: String = "hello, world!") -> Bool { false } +} +``` + +This proposal makes both declarations a compile time error. This is a source compatibility break, but a very small one which only affects declarations with no practical use. This bug also affects closure parameter lists: + +```swift +// Currently allowed, but impossible to call. +let closure = {(a: Int..., b: Int) in} +``` + +Under this proposal, the above code also becomes a compile-time error. Note that because closures do not allow external parameter labels, they cannot support multiple variadic parameters. + +## Source compatibility + +As noted above, this proposal is source-breaking for any program which has a subscript declaration or closure having an unlabeled parameter following a variadic parameter. With the exception of very specific subscript declarations making use of default parameters, this only affects parameter lists which are syntactically impossible to fulfill. As a result, the break should have no impact on the vast majority of existing codebases. It does not cause any failures in the source compatibility suite. + +If this source-breaking change is considered unacceptable, there are two alternatives. One would be to change the error to a warning when applied to subscripts and closures. The other would be to preserve the buggy behavior and emit no diagnostics. In both cases, multiple variadic parameters would continue to be supported by subscripts, but users would retain the ability to write parameter lists which can't be fulfilled in some contexts. + +## Effect on ABI stability + +This proposal does not require any changes to the ABI. The current ABI representation of variadic parameters already supports more than one per function/subscript/initializer. + +## Effect on API resilience + +An ABI-public function may not add, remove, or reorder parameters, whether or not they have default arguments or are variadic. This rule is unchanged and applies to all variadic parameters. + +## Alternatives considered + +Two alternative labeling rules were considered. + +1. If a parameter list has more than one variadic parameter, every variadic parameter must have a label. +2. If a parameter list has more than one variadic parameter, every variadic parameter except for the first must have a label. + +Both alternatives are more restrictive in terms of the declarations they allow. This increases complexity and makes the parameter labeling rules harder to reason about. However, they might make it more difficult to write confusing APIs which mix variadic, defaulted, and required parameters. Overall, it seems better to trust programmers with greater flexibility, while also minimizing the number of rules they need to learn. diff --git a/proposals/0285-ease-pound-file-transition.md b/proposals/0285-ease-pound-file-transition.md new file mode 100644 index 0000000000..fd73519258 --- /dev/null +++ b/proposals/0285-ease-pound-file-transition.md @@ -0,0 +1,175 @@ +# Ease the transition to concise magic file strings + +* Proposal: [SE-0285](0285-ease-pound-file-transition.md) +* Author: [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Implementation: [apple/swift#32700](https://github.com/apple/swift/pull/32700) +* Status: **Implemented (Swift 5.3)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0285-ease-the-transition-to-concise-magic-file-strings/38516) + +## Introduction + +In [SE-0274][], the core team accepted a proposal to change the behavior of `#file`. This proposal modifies that plan to transition into new behavior more gradually, treating it as a source break requiring a new language version mode to fully adopt. + +Swift-evolution thread: [Revisiting the source compatibility impact of SE-0274: Concise magic file names](https://forums.swift.org/t/revisiting-the-source-compatibility-impact-of-se-0274-concise-magic-file-names/37720/) + + [SE-0274]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0274-magic-file.md + +## Motivation + +SE-0274 made the following changes: + +1. Introduced a new `#filePath` magic identifier which, like `#file` in prior versions of Swift, evaluates to the source file path passed when the compiler was invoked. + +2. Modified the `#file` magic identifier to instead evaluate to a string of the form `ModuleName/FileName.swift`. (See SE-0274 for discussions of the motivations for this.) + +3. Added a warning when, essentially, a wrapper around a function with a `#filePath` default argument inadvertently captured `#file` instead, or vice versa. + +After acceptance, changes 1 and 3 were enabled in the release/5.3 branch. Change 2 was gated by a staging flag and was expected to be enabled in the release after 5.3. + +On paper, this seemed like a workable plan, giving developers the entire Swift 5.3 cycle to update code like this: + +```swift +/// Prints the path to the file it was called from. +func printSourcePath(file: String = #file) { print(file) } +``` + +To this when necessary: + +```swift +/// Prints the path to the file it was called from. +func printSourcePath(file: String = #filePath) { print(file) } +``` + +However, many source-distributed libraries like SwiftNIO need more than one version of backwards compatibility in use sites which will need to transition to `#filePath`. Their authors had believed they could achieve this using `#if compiler(>=5.3)` directives, but when they began to implement these workarounds, they found they were stymied by language limitations. + +For instance, they believed they would be able to use a helper function to abstract the choice of `#file` or `#filePath`, not realizing that the magic call-site behavior of `#file` and `#filePath` in default arguments isn't transitive: + +```swift +/// Prints the path to the file it was called from. +/// - Warning: Actually prints the path to the file `printSourcePath(file:)` was +/// defined in instead! +func printSourcePath(file: String = filePathOrFile()) { print(file) } + +#if compiler(>=5.3) + func filePathOrFile(_ value: String = #filePath) -> String { value } +#else + func filePathOrFile(_ value: String = #file) -> String { value } +#endif +``` + +They also believed they could use `#if` in default arguments, not realizing that `#if` directives can only be wrapped around whole statements or declarations: + +```swift +/// Prints the path to the file it was called from. +/// - Warning: Actually fails to compile with a syntax error! +func printSourcePath(file: String = #if compiler(>=5.3) + #filePath + #else + #file + #endif) { print(file) } +``` + +That left them with with only one reasonable option—conditionally-selected wrapper functions—which they understandably considered too burdensome: + +```swift +fileprivate func _printSourcePathImpl(file: String) { print(file) } + +#if compiler(>=5.3) +/// Prints the path to the file it was called from. +func printSourcePath(file: String = #filePath) { _printSourcePathImpl(file: file) } +#else +/// Prints the path to the file it was called from. +func printSourcePath(file: String = #file) { _printSourcePathImpl(file: file) } +#endif +``` + +While we still want to reach the same endpoint—most code using a syntax with concise file strings while full paths are still available to the use sites that need them—it has become clear that this transition needs to be more gradual, with `#file` continuing to have its current meaning in Swift 5 projects indefinitely. + +## Proposed solution + +We propose to modify all three aspects of SE-0274's changes: + +1. In addition to `#filePath`, we will introduce a new `#fileID` magic identifier, which generates the new concise file string in all language modes. + +2. `#file` will continue to produce the same string as `#filePath` in the Swift 5 and earlier language modes. When code is compiled using future language modes (e.g. a hypothetical "Swift 6 mode"[1]), `#file` will generate the same string as `#fileID`, and `#fileID` will be deprecated. + +3. In Swift 5 mode, for the purposes of the "wrong magic identifier literal" warning, `#file` will be treated as compatible with *both* `#filePath` and `#fileID`. In future language modes, it will be treated as compatible with only `#fileID`. + +
+ +[1] + +> "Swift 6" is purely illustrative. There is no guarantee that Swift 6 will be a source-breaking version, that there won't be a source-breaking version before Swift 6, or indeed that there will ever be a Swift 6 at all. Offer void where prohibited. + +
+ +## Detailed design + +### The `#fileID` magic identifier literal + +In addition to the existing `#file` and the new `#filePath`, Swift will support `#fileID`. This new magic identifier will generate the same module-and-filename string SE-0274 specified for `#file`. It will do so immediately, without any way to reverse its behavior. + +Standard library assertion functions like `assert`, `precondition`, and `fatalError` will switch from `#file` to `#fileID`. When a filename is included in a compiler-generated trap, such as a force unwrap, it will also emit a literal equivalent to using `#fileID`. + +`#fileID` is intended to allow Swift 5.3 code to adopt the new, more compact literals before the behavior of `#file` changes. In language version modes where `#file` produces the same string as `#fileID`, `#fileID` will be deprecated. + +### Transitioning the `#file` magic identifier + +The `#file` identifier will continue to generate the same string as `#filePath` in Swift 4, 4.2, and 5 modes. In any future language version modes, it will instead generate the `#fileID` string. This means that the change in `#file`'s behavior—and therefore the requirement to change some uses of `#file` to `#filePath`—will be delayed until Swift next makes source-breaking changes, and even then, the current behavior will be preserved in Swift 5 mode. + +If a client compiled in Swift 5 mode or earlier uses a library compiled with a later language mode, the library's `#file` default arguments will be treated as though they were `#fileID`; if a client compiled with a later language mode uses a library compiled in Swift 5 mode or earlier, the library's `#file` default arguments will be treated as though they were `#filePath`. + +### Magic identifier default argument mismatch warnings + +Swift will not warn when: + +* A parameter with a default argument of `#file` is passed to one with a default argument of `#fileID`. +* A parameter with a default argument of `#fileID` is passed to one with a default argument of `#file`. + +Additionally, Swift 5 mode and earlier will not warn when: + +* A parameter with a default argument of `#file` is passed to one with a default argument of `#filePath`. +* A parameter with a default argument of `#filePath` is passed to one with a default argument of `#file`. + +In Swift 5 mode, substituting `#file` for `#fileID` *will* actually result in different behavior, so ideally we would warn about this. However, doing that would cause new warnings in existing functions which wrap functions that adopt `#fileID`. In practice, most `#fileID` adopters will work fine when they're passed `#file`—they'll just generate unnecessarily large file strings. + +### Swift API Design Guidelines amendment + +This guideline will be added to [the "Parameters" section][api-params] of the Swift API Design Guidelines: + +> * **If your API will run in production, prefer `#fileID`** over alternatives. +> `#fileID` saves space and protects developers’ privacy. Use `#filePath` in +> APIs that are never run by end users (such as test helpers and scripts) if +> the full path will simplify development workflows or be used for file I/O. +> Use `#file` to preserve source compatibility with Swift 5.2 or earlier. + + [api-params]: https://swift.org/documentation/api-design-guidelines/#parameter-names + +### Schedule + +Ideally, these changes will be included in Swift 5.3, with the "future language mode" parts tied to a frontend flag for testing, rather than a language version mode. + +## Source compatibility + +Significantly improved compared to SE-0274. Code that uses `#filePath` or `#fileID` will not be compilable in Swift 5.2 and earlier, but such code can continue to use `#file` until a future language version mode permits breaking changes. + +## Effect on ABI stability + +See SE-0274. + +## Effect on API resilience + +See SE-0274. + +## Alternatives considered + +The SE-0274 design was considered and accepted, but did not hold up to real-world use. + +In future language version modes, we could deprecate `#file` instead of `#fileID`, creating a situation where users must choose whether to update their uses of `#file` to either `#fileID` or `#filePath`. This would prompt future users to make a choice of behavior, but we don't want to force those users to make an explicit choice; we continue to believe that the concise file string (i.e. `#fileID` string) is the right choice for most uses. + +Similarly, we could simply introduce `#fileID` and change the standard library to use it, without adding `#filePath` or changing `#file`'s behavior. But since we think the `#fileID` string is the right default, this would leave us with a design that encouraged the wrong defaults. + +We could remove the `#fileID` syntax from this proposal, probably retaining it in underscored form as an implementation detail used to transition the standard library assertions to the new strings in Swift 5 mode. This would give us a cleaner "Swift 6 mode" without a deprecated `#fileID` hanging around, but we don't want to keep potential adopters outside the standard library from gaining the advantages of the new strings if they want them. + +There are many colors other than `#fileID` that we could use to paint this bikeshed; we like this spelling because it's shorter than `#filePath` and suitably vague about the exact format of the string it produces. diff --git a/proposals/0286-forward-scan-trailing-closures.md b/proposals/0286-forward-scan-trailing-closures.md new file mode 100644 index 0000000000..972038c5a6 --- /dev/null +++ b/proposals/0286-forward-scan-trailing-closures.md @@ -0,0 +1,343 @@ +# Forward-scan matching for trailing closures + +* Proposal: [SE-0286](0286-forward-scan-trailing-closures.md) +* Author: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.3)** +* Upcoming Feature Flag: `ForwardTrailingClosures` (implemented in Swift 5.8) +* Implementation: [apple/swift#33092](https://github.com/apple/swift/pull/33092) +* Toolchains: [Linux](https://ci.swift.org/job/swift-PR-toolchain-Linux/404//artifact/branch-master/swift-PR-33092-404-ubuntu16.04.tar.gz), [macOS](https://ci.swift.org/job/swift-PR-toolchain-osx/579//artifact/branch-master/swift-PR-33092-579-osx.tar.gz) +* Discussion: ([Pitch #1](https://forums.swift.org/t/pitch-1-forward-scan-matching-for-trailing-closures-source-breaking/38162)), ([Pitch #2](https://forums.swift.org/t/pitch-2-forward-scan-matching-for-trailing-closures/38491)) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/07bcb908125e1795a08d47391b5d866eb782639e/proposals/0286-forward-scan-trailing-closures.md) +* Review: ([Review](https://forums.swift.org/t/se-0286-forward-scan-for-trailing-closures/38529)), ([Acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0286-forward-scan-for-trailing-closures/38836)) + +## Introduction + +[SE-0279 "Multiple Trailing Closures"](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0279-multiple-trailing-closures.md) threaded the needle between getting the syntax we wanted for multiple trailing closures without breaking source compatibility. One aspect of that compromise was to extend (rather than replace) the existing rule for matching a trailing closure to a parameter by scanning *backward* from the end of the parameter list. + +However, the backward-scan matching rule makes it hard to write good API that uses trailing closures, especially multiple trailing closures. This proposal replaces the backward scan with a forward scan wherever possible, which is simpler, more in line with normal argument matching in a call, and works better for APIs that support trailing closures (whether single or multiple) and default arguments. This change introduces a *minor source break* for code involving multiple, defaulted closure parameters, but that source break is staged over multiple Swift versions. + +## Motivation + +Several folks noted the downsides of the "backward" matching rule. The rule itself is described in the [detailed design](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0279-multiple-trailing-closures.md#detailed-design) section of SE-0279 (search for "backward"). To understand the problem with the backward rule, let's try to declare the UIView [`animate(withDuration:animations:completion:)`](https://developer.apple.com/documentation/uikit/uiview/1622515-animate) method in the obvious way to make use of SE-0279: + +```swift +class func animate( + withDuration duration: TimeInterval, + animations: @escaping () -> Void, + completion: ((Bool) -> Void)? = nil +) +``` + +SE-0279 matches the named trailing closure arguments backward, matching the last (labeled) trailing closure argument from the back of the parameter list, then proceeding to move to earlier trailing closures and function parameters. Consider the following example (straight from SE-0279): + +```swift +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} completion: { _ in + self.view.removeFromSuperview() +} +``` + +The `completion:` trailing closure matches the last parameter, and the unnamed trailing closure matches `animations:`. The backward rule worked fine here. + +However, things fall apart when a single (therefore unnamed) trailing closure is provided to this API: + +```swift +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} +``` + +Now, the backward rule matches the unnamed trailing closure to `completion:`. The compiler produces an error: + +``` +error: missing argument for parameter 'animations' in call + animate(withDuration: 0.3) { +``` + +Note that the "real" UIView API actually has two different methods---`animate(withDuration:animations:completion:)` and `animate(withDuration:animations:)`---where the latter looks like this: + +```swift +class func animate( + withDuration duration: TimeInterval, + animations: @escaping () -> Void +) +``` + +This second overload only has a closure argument, so the backward-matching rule handles the single-trailing-closure case. These overloads exist because these UIView APIs were imported from Objective-C, which does not have default arguments. A new Swift API would not be written this way---except that SE-0279 forces it due to the backward-matching rule. + +## Proposed solution + +The idea of the forward-scan matching rule is to match trailing closure arguments to parameters in the same forward, left-to-right manner that other arguments are matched to parameters. The unlabeled trailing closure will be matched to the next parameter that is either unlabeled or has a declared type that structurally resembles a function type (defined below). For the example above, this means the following: + +```swift +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} +// equivalent to +UIView.animate(withDuration: 0.3, animations: { + self.view.alpha = 0 +}) +``` + +and + +```swift +UIView.animate(withDuration: 0.3) { + self.view.alpha = 0 +} completion: { _ in + self.view.removeFromSuperview() +} +// equivalent to +UIView.animate(withDuration: 0.3, animations: { + self.view.alpha = 0 +}, completion: { _ in + self.view.removeFromSuperview() +}) +``` + +Note that the unlabeled trailing closure matches `animations:` in both cases; specifying additional trailing closures fills out later parameters but cannot shift the unlabeled trailing closure to an earlier parameter. + +Note that you can still have the unlabeled trailing closure match a later parameter, by specifying earlier ones: + +```swift +UIView.animate(withDuration: 0.3, animations: self.doAnimation) { _ in + self.view.removeFromSuperview() +} +// equivalent to +UIView.animate(withDuration: 0.3, animations: self.doAnimation, completion: { _ in + self.view.removeFromSuperview() +}) +``` + +This is both a consequence of forward matching and also a necessity for source compatibility. + +### Structural resemblance to a function type + +When a function parameter does not require an argument (e.g., because it is variadic or has a default argument), the call site can skip mentioning that parameter entirely, and the default will be used instead (e.g., an empty variadic argument or the specified default argument). The matching of arguments to parameters tends to rely on argument labels to determine when a particular parameter has been skipped. For example: + +```swift +func nameMatchingExample(x: Int = 1, y: Int = 2, z: Int = 3) { } + +nameMatchingExample(x: 5) // equivalent to nameMatchingExample(x: 5, y: 2, z: 3) +nameMatchingExample(y: 4) // equivalent to nameMatchingExample(x: 1, y: 4, z: 3) +nameMatchingExample(x: -1, z: -3) // equivalent to nameMatchingExample(x: -1, y: 2, z: -3) +``` + +The unlabeled trailing closure ignores the (otherwise required) argument label, which would prevent the use of argument labels for deciding which parameter should be matched with the unlabeled trailing closure. Let's bring that back to the UIView example by adding a default argument to `withDuration:` + +```swift +class func animate( + withDuration duration: TimeInterval = 1.0, + animations: @escaping () -> Void, + completion: ((Bool) -> Void)? = nil +) +``` + +Consider a call: + +```swift +UIView.animate { + self.view.alpha = 0 +} +``` + +The first parameter is `withDuration`, but there is no argument in parentheses. Unlabeled trailing closures ignore the parameter name, so without some additional rule, the unlabeled trailing closure would try to match `withDuration:` and this call would be ill-formed. + +The forward-scan matching rule skips over any parameters that do not "structurally resemble" a function type. A parameter structurally resembles a function type if both of the following are true: + +* The parameter is not `inout` +* The adjusted type of the parameter (defined below) is a function type + +The adjusted type of the parameter is the parameter's type as declared in the function, looking through any type aliases whenever they appear, and performing three adjustments: + +* If the parameter is an `@autoclosure` , use the result type of the parameter's declared (function) type, before performing the second adjustment. +* If the parameter is variadic, looking at the element type of the (implied) array type. +* Remove all outer "optional" types. + +Following this rule, the `withDuration` parameter (a `TimeInterval`) does not resemble a function type. However, `@escaping () -> Void` does, so the unlabeled trailing closure matches `animations`. `@autoclosure () -> ((Int) -> Int)` and `((Int) -> Int)?` would also resemble a function type. + +### Mitigating the source compatibility impact (all language versions) + +The forward-scanning rule, as described above, is source-breaking. A run over Swift's [source compatibility suite](https://swift.org/source-compatibility/) with this change enabled in all language modes turned up source compatibility breaks in three projects. The first problem occurs with a SwiftUI API [`View.sheet(isPresented:onDismiss:content:)`](https://developer.apple.com/documentation/swiftui/view/sheet(ispresented:ondismiss:content:)): + +```swift +func sheet( + isPresented: Binding, + onDismiss: (() -> Void)? = nil, + content: @escaping () -> Content +) -> some View +``` + +Note that `onDismiss` and `content` both structurally resemble a function type. This API fits well with the backward-matching rule, because the unlabeled trailing closure in the following example is always ascribed to `content:`. The `onDismiss:` argument gets the default argument `nil`: + +```swift +sheet(isPresented: $isPresented) { Text("Hello") } +``` + +With the forward-scanning rule, the unlabeled trailing closure matches the `onDismiss:` parameter, and there is no suitable argument for `content:`. Therefore, the well-formed code above would be rejected by the rule as proposed above. + +However, it is clear from the function signature that (1) `onDismiss:` could have used the default argument, and (2) `content:` therefore won't have an argument if it is not paired with the unlabeled trailing closure. We can turn this into an heuristic to accept more existing code, reducing the source breaking impact of the proposal. Specifically, if + +* the parameter that would match the unlabeled trailing closure +argument does not require an argument (because it is variadic or has a default argument), and +* there are parameters *after* that parameter that require an argument, up until the first parameter whose label matches that of the *next* trailing closure (if any) + +then do not match the unlabeled trailing closure to that parameter. Instead, skip it and examine the next parameter to see if that should be matched against the unlabeled trailing closure. For the `View.sheet(isPresented:onDismiss:content:)` API, this means that `onDismiss`, which has a default argument, will be skipped in the forward match so that the unlabeled trailing closure will match `content:`, allowing this code to continue to compile correctly. + +This heuristic is remarkably effective: in addition to fixing 2 of the 3 failures from the Swift source compatibility suite (the remaining failure will be discussed below), it resolved most of the failures we observed in a separate (larger) testsuite comprising a couple of million lines of Swift. + +One practical effect of this heuristic is that it makes the forward scan as proposed here produce the same results as the existing backward scan in many, many more cases. + +### Mitigating the source compatibility impact (Swift < 6) + +Even with the heuristic, the forward-scan matching rule will still fail to compile some existing code, and can change the meaning of some code, when there are multiple, defaulted parameters of closure type. As an example, the remaining source compatibility failure in the Swift source compatibility suite, a project called [ModelAssistant](https://github.com/ssamadgh/ModelAssistant), is due to [this API](https://github.com/ssamadgh/ModelAssistant/blob/c96335280a3aba5f8e14955ecaf38dc25a0872b6/Source/Libraries/AOperation/Observers/BlockObserver.swift#L22-L26): + +```swift +init( + startHandler: ((AOperation) -> Void)? = nil, + produceHandler: ((AOperation, Foundation.Operation) -> Void)? = nil, + finishHandler: ((AOperation, [NSError]) -> Void)? = nil +) { + self.startHandler = startHandler + self.produceHandler = produceHandler + self.finishHandler = finishHandler +} +``` + +Note that this API takes three closure parameters. The (existing) backward scan will match `finishHandler:`, while the forward scan will match `startHandler:`. The heuristic described in the previous section does not apply, because all of the closure parameters have default arguments. Existing code that uses trailing closures with this API will break. + +Note that this API interacts poorly with SE-0279 multiple trailing closures, because the unlabeled trailing closure "moves" backwards as additional trailing closures are provided at the call site: + +```swift +// SE-0279 backward scan behavior +BlockObserver { (operation, errors) in + print("finishHandler!") +} + +// label finishHandler, unlabeled moves "back" to produceHandler + +BlockObserver { (aOperation, foundationOperation) in + print("produceHandler!") +} finishHandler: { (operation, errors) in + print("finishHandler!") +} + +// label produceHandler, unlabeled moves "back" to startHandler +BlockObserver { aOperation in + print("startHandler!") +} produceHandler: { (aOperation, foundationOperation) in + print("produceHandler!") +} finishHandler: { (operation, errors) in + print("finishHandler!") +} +``` + +The forward scan provides a consistent unlabeled trailing closure anchor, and later (labeled) trailing closures can be tacked on: + +```swift +// Proposed forward scan +BlockObserver { aOperation in + print("startHandler!") { +} + +// add another +BlockObserver { aOperation in + print("startHandler!") +} produceHandler: { (aOperation, foundationOperation) in + print("produceHandler!") +} + +// specify everything +BlockObserver { aOperation in + print("startHandler!") +} produceHandler: { (aOperation, foundationOperation) in + print("produceHandler!") +} finishHandler: { (operation, errors) in + print("finishHandler!") +} + +// skip the middle one! +BlockObserver { aOperation in + print("startHandler!") +} finishHandler: { (operation, errors) in + print("finishHandler!") +} +``` + +The forward-scan matching rule provides more predictable results, making it easier to understand how to use this API properly. However, maintaining backward compatibility requires that the backward scan be considered in places where it differs from the forward scan. + +To address this remaining source compatibility problem, Swift minor versions (prior to Swift 6) shall implement an additional rule for calls that involve a single (unlabeled) trailing closure. If the forward and backward-scan rules produce *different* assignments of arguments to parameters, then the Swift compiler will attempt both: if only one succeeds, use it. If both succeed, prefer the backward-scanning rule (for source compatibility reasons) and produce a warning about the use of the backward scan. For example: + +```swift +BlockObserver { (operation, errors) in + print("finishHandler!") +} +``` + +Here, the forward scan fails to type-check, because the closure accepts two parameters whereas `startHandler` accepts a single parameter. Therefore, the backward scan is selected, maintaining source compatibility, and produces a warning with a Fix-It to make the trailing closure a regular argument: + +``` +warning: backward matching of the unlabeled trailing closure is deprecated; label the argument with 'finishHandler' to suppress this warning +BlockObserver { (operation, errors) in + ^ + (finishHandler: +``` + +If there truly is an ambiguity, where both the forward scan and backward scan type-check but would do so differently, we prefer the backward scan to maintain source compatibility: + +```swift +func trailingClosureBothDirections( + f: (Int, Int) -> Int = { $0 + $1 }, g: (Int, Int) -> Int = { $0 - $1 } +) { } +trailingClosureBothDirections { $0 * $1 } +``` + +Here, the forward scan would bind the trailing closure to `f:` (for Swift 6 and newer) while the backward scan would bind the trailing closure to `g:` (for Swift < 6). The same warning will apply when the backward scan result is chosen, with a Fix-It to rewrite the code to: + +```swift +trailingClosureBothDirections(g: { $0 * $1 }) +``` + +This suppresses the warning and eliminates the ambiguity, so the code behaves the same across all overload sets. + +The Swift 6 and newer behavior can be enabled in existing language modes with the [upcoming feature flag](0362-piecemeal-future-features.md) `ForwardTrailingClosures`. + +### Workaround via overload sets + +APIs like the above that depend on the backward scan can be reworked to provide the same client API. The basic technique involves removing the default arguments, then adding additional overloads to create the same effect. For example, drop the default argument of `finishHandler` so that the heuristic will kick in to fix calls with a single unlabeled trailing closure: + +```swift +init( + startHandler: ((AOperation) -> Void)? = nil, + produceHandler: ((AOperation, Foundation.Operation) -> Void)? = nil, + finishHandler: ((AOperation, [NSError]) -> Void)? +) { + self.startHandler = startHandler + self.produceHandler = produceHandler + self.finishHandler = finishHandler +} +``` + +One can then add overloads to handle other cases, e.g., the zero-argument case: + +```swift +init() { + self.init(startHandler: nil, produceHandler: nil, finishHandler: nil) +} +``` + +## Future directions + +The proposal specifies that the "backward" scan be removed in Swift 6, which introduces a small source break that is staged in over time. However, the heuristic (that skips matching the unnamed trailing closure argument to a parameter that doesn't require an argument when the unnamed trailing closure is needed to match a later parameter) is retained. However, some future language version (Swift 6 or even later) might accept more source breakage by removing this heuristic---leaving only the forward scan in place---and find a better way to express APIs such as `View.sheet(isPresented:onDismiss:content:)` in the language. Possibilities include (but are not limited to): + +* A parameter attribute `@noTrailingClosure` that prevents the use of trailing closure syntax for a given parameter entirely. +* Eliminating the allowance for matching the first (unlabeled) trailing closure to a parameter that has an argument label, so normal argument matching rules would apply. +* Allowing an argument label on the first trailing closure to let the caller select which parameter to match explicitly. + +This proposal leaves open all of these possibilities, and makes their changes less drastic because each of the ideas involves moving to a forward-scan matching rule. As such, this proposal makes SE-0279's multiple trailing closures immediately useful with minimal (or no) source breakage, paving the way for more significant changes if we so choose in the future. + +## Revision history + +* **Version 2**: Improved source compatibility by performing both the forward and backward scans in Swift < 6 mode ([originally suggested](https://forums.swift.org/t/se-0286-forward-scan-for-trailing-closures/38529/9) by Pavel Yaskevich) and adopting the [specific proposal](https://forums.swift.org/t/se-0286-forward-scan-for-trailing-closures/38529/30) from Xiaodi Wu to prefer the backward scan result in Swift < 6 when the two scans differ. diff --git a/proposals/0287-implicit-member-chains.md b/proposals/0287-implicit-member-chains.md new file mode 100644 index 0000000000..27fd9da22c --- /dev/null +++ b/proposals/0287-implicit-member-chains.md @@ -0,0 +1,252 @@ +# Extend implicit member syntax to cover chains of member references + +* Proposal: [SE-0287](0287-implicit-member-chains.md) +* Author: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.4)** +* Implementation: [apple/swift#31679](https://github.com/apple/swift/pull/31679) +* Review: [Review](https://forums.swift.org/t/se-0287-extend-implicit-member-syntax-to-cover-chains-of-member-references/39398), [Acceptance](https://forums.swift.org/t/accepted-se-0287-extend-implicit-member-syntax-to-cover-chains-of-member-references/39714) + +## Introduction + +When the type of an expression is implied by the context, Swift allows developers to use what is formally referred to as an "implicit member expression," sometimes referred to as "leading dot syntax": + +```swift +class C { + static let zero = C(0) + var x: Int + + init(_ x: Int) { + self.x = x + } +} + +func f(_ c: C) { + print(c.x) +} + +f(.zero) // prints '0' +``` + +This allows for the omission of repetitive type information in contexts where the type information is already obvious to the reader: +```swift +view.backgroundColor = .systemBackground +``` + +This proposal suggests the expansion of implicit member syntax to more complex expressions than just a single static member or function. Specifically, implicit member syntax would be allowed to cover chains of member references. + +## Motivation + +Today, attempting to use implicit member syntax with a chain of member references fails: + +```swift +extension C { + var incremented: C { + return C(self.x + 1) + } +} + +f(.zero.incremented) // Error: Type of expression is ambiguous without more context +``` + +This error breaks the mental model that many users likely have for implicit member syntax, which boils down to a simple lexical omission of the type name in contexts where the type is clear. I.e., users expect that writing: + +```swift +let one: C = .zero.incremented +``` + +is just the same as writing + +```swift +let one = C.zero.incremented +``` + +This issue arises in practice with any type that offers "modifier" methods that vend updated instances according to some rule. For example, `UIColor` offers the `withAlphaComponent(_:)` modifier for constructing new colors, which cannot be used with implicit member syntax: + +```swift +let milky: UIColor = .white.withAlphaComponent(0.5) // error +``` + +## Proposed solution + +Improve implicit member expression syntax to handle multiple chained member accesses. The base type of the implicit member expression would be constrained to match the contextual/resultant type of the whole chain. Under this proposal, all of the following would successfully typecheck: + +```swift +let milky: UIColor = .white.withAlphaComponent(0.5) +let milky2: UIColor = .init(named: "white")!.withAlphaComponent(0.5) +let milkyChance: UIColor? = .init(named: "white")?.withAlphaComponent(0.5) + +struct Foo { + static var foo = Foo() + + var anotherFoo: Foo { Foo() } + func getFoo() -> Foo { Foo() } + var optionalFoo: Foo? { Foo() } + var fooFunc: () -> Foo { { Foo() } } + var optionalFooFunc: () -> Foo? { { Foo() } } + var fooFuncOptional: (() -> Foo)? { { Foo() } } + subscript() -> Foo { Foo() } +} + +let _: Foo = .foo.anotherFoo +let _: Foo = .foo.anotherFoo.anotherFoo.anotherFoo.anotherFoo +let _: Foo = .foo.getFoo() +let _: Foo = .foo.optionalFoo!.getFoo() +let _: Foo = .foo.fooFunc() +let _: Foo = .foo.optionalFooFunc()! +let _: Foo = .foo.fooFuncOptional!() +let _: Foo = .foo.optionalFoo! +let _: Foo = .foo[] +let _: Foo = .foo.anotherFoo[] +let _: Foo = .foo.fooFuncOptional!()[] + +struct Bar { + var anotherFoo = Foo() +} + +extension Foo { + static var bar = Bar() + var anotherBar: Bar { Bar() } +} + +let _: Foo = .bar.anotherFoo +let _: Foo = .foo.anotherBar.anotherFoo +``` + +## Detailed design + +This proposal would provide the model mentioned earlier for implicit member expressions: anywhere that a contextual type `T` can be inferred, writing + +```swift +.member1.member2.(...).memberN +``` + +Will behave as if the user had written: + +```swift +T.member1.member2.(...).memberN +``` + +Note: if `T` is an optional type `R?` for some type `R`, we maintain the existing rule that lookup of `member1` in an implicit base will proceed in both `R?` and `R` (which allows the `milkyChance` example above to compile). + +Members of this "implicit member chain" can be any of the following: +- Property references +- Method calls +- Forced unwrapping expressions +- Optional-chaining question marks +- Subscripts + +When any of the above is encountered by the type checker, it will determine two things: + +1. Whether this expression sits at the tail of the chain. +2. Whether the base of the chain is an implicit member expression. + +If those two conditions are met, then a constraint is introduced requiring the result of the whole chain to equal the type of the base of the implicit member expression. + +Members of the chain are allowed to participate in generic parameter inference as well. Thus, the following code is valid: + +```swift +struct Foo { + static var foo: Foo { Foo() } + var anotherFoo: Foo { Foo() } + func getAnotherFoo() -> Foo { + Foo() + } +} + +extension Foo where T == Int { + static var fooInt: Foo { Foo() } + var anotherFooInt: Foo { Foo() } + var anotherFooIntString: Foo { Foo() } + func getAnotherFooInt() -> Foo { + Foo() + } +} + +extension Foo where T == String { + var anotherFooStringInt: Foo { Foo() } +} + +func implicit(_ arg: Foo) {} + +// T inferred as Foo in all of the following +implicit(.fooInt) +implicit(.foo.anotherFooInt) +implicit(.foo.anotherFooInt.anotherFoo) +implicit(.foo.anotherFoo.anotherFooInt) +implicit(.foo.getAnotherFooInt()) +implicit(.foo.anotherFoo.getAnotherFooInt()) +implicit(.foo.getAnotherFoo().anotherFooInt) +implicit(.foo.getAnotherFooInt()) +implicit(.foo.getAnotherFoo().getAnotherFooInt()) +// Members types along the chain can have different generic arguments +implicit(.foo.anotherFooIntString.anotherFooStringInt) +``` + +If `T` is the contextually inferred type but `memberN` has non-convertible type `R` , a diagnostic of the form: + +```swift +Error: Cannot convert value of type 'R' to expected type 'T' +``` + +will be produced. The exact form of the diagnostic will depend on how `T` was contextually inferred (e.g. as an argument, as an explicit type annotation, etc.). + +## Source compatibility + +This is a purely additive change and does not have any effect on source compatibility. + +## Effect on ABI stability + +This change is frontend only and would not impact ABI. + +## Effect on API resilience + +This is not an API-level change and would not impact resilience. + +## Alternatives considered + +### Require homogeneously-typed chains + +While overall discussion around this feature was very positive, one point of minor disagreement was whether chains should be required to have the same type along the length of the chain. Such a rule would prohibit constructs like this: + +```swift +struct S { + static var foo = T() +} + +struct T { + var bar = S() +} + +let _: S = .foo.bar // error! +``` + +Proponents of this rule argued that the most common use case for these member chains (the aforementioned "modifier" or "builder" methods) doesn't require heterogeneously-typed chains, and that supporting them would introduce a cognitive load for readers of code that relies on heterogeneously-typed chains. + +A rule of this form was explored during implementation, but was abandoned for several reasons. One was simply that the implementation complexity would have been greatly increased in order to properly support the additional constraints along the chain while still offering helpful diagnostics. Another is that such a rule is far less flexible in situations that seem like they should work even with homogeneously-typed chains. For instance, allowing heterogeneously-typed chains easily enables the following syntax to compile: + +```swift +struct HasClosure { + static var factoryOpt: ((Int) -> HasClosure)? = { _ in .init() } +} + +var _: HasClosure = .factoryOpt!(4) +``` + +Trying to support this construction with "homogeneously-typed chains" rule in place would require significantly more interaction between the different segments of the chain in order to decide whether certain constructions should be allowed. + +Lastly, the author makes the subjective determination that such a rule would be at odds with the expectations of most users when using implicit member chains. Visually, implicit member chains appear very similar to keypath expressions, and indeed support all the same elements as keypath expressions (additionally supporting method/function calls). There is no such restriction that keypath expressions refer to the same type along their length (even when the keypath base type is omitted), so users may find it surprising that the compiler does not accept identical syntax for direct property accesses: + +```swift +struct S { + static var foo = T() + var foo: T { T() } +} + +struct T { + var bar = S() +} + +let _: KeyPath = \.foo.bar +let _: S = .foo.bar // error? +``` diff --git a/proposals/0288-binaryinteger-ispower.md b/proposals/0288-binaryinteger-ispower.md new file mode 100644 index 0000000000..b48fde326d --- /dev/null +++ b/proposals/0288-binaryinteger-ispower.md @@ -0,0 +1,274 @@ +# Adding `isPower(of:)` to `BinaryInteger` + +* Proposal: [SE-0288](0288-binaryinteger-ispower.md) +* Author: [Ding Ye](https://github.com/dingobye) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Previewing** +* Implementation: [apple/swift#24766](https://github.com/apple/swift/pull/24766) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0288-adding-ispower-of-to-binaryinteger/40325) + +## Introduction + +Checking some mathematical properties of integers (e.g. parity, divisibility, etc.) is widely used in scientific and engineering applications. Swift brings a lot of convenience when performing such checks, thanks to the relevant methods (e.g. `isMultiple(of:)`) provided by the standard library. However there are still some other cases not yet supported. One of those useful checks that are currently missing is to tell if an integer is power of another, of which the implementation is non-trivial. Apart from inconvenience, user-implemented code can bring inefficiency, poor readability, and even incorrectness. To address this problem, this proposal would like to add a public API `isPower(of:)`, as an extension method, to the `BinaryInteger` protocol. + +Swift-evolution thread: [Pitch](https://forums.swift.org/t/adding-ispowerof2-to-binaryinteger/24087) + + +## Motivation + +Checking whether an integer is power of a given base is a typical integral query in a wide range of applications, and is especially common when the base value is two. A question about [How to check if a number is a power of 2](https://stackoverflow.com/questions/600293/how-to-check-if-a-number-is-a-power-of-2) on Stack Overflow grows for a decade with ~200K views. Since there are public demands for such functionality, it has been or will be officially supported by the libraries of many popular languages, such as [C++20](https://en.cppreference.com/w/cpp/numeric/ispow2), [D Programming Language](https://dlang.org/library/std/math/is_power_of2.html), and [.NET Framework](https://docs.microsoft.com/en-us/dotnet/api/system.numerics.biginteger.ispoweroftwo?view=netframework-4.8). + +Swift, as a general-purpose programming language, is also experiencing such demands covering a variety of domains, including numerical and mathematical libraries (e.g. [#1](https://github.com/jsbean/ArithmeticTools/blob/cb6dae327baf53cdf614d26e630833efa00eda3f/ArithmeticTools/IntegerExtensions.swift#L48), [#2](https://github.com/Orbifold/XAct/blob/9acad78e5571aa93fb52d88f539459effab1d5f7/XAct/Numbers.swift#L49), [#3](https://github.com/donald-pinckney/SwiftNum/blob/b92e3b964268ebf62d99f488fcdf438574974f0d/Sources/SignalProcessing/IntExtensions.swift#L12), [#4](https://github.com/dn-m/Math/blob/d1284e043377c0b924cba2ffa2ab0b9aa9dd246f/Sources/Math/IntegerExtensions.swift#L48)), programming language and compiler components (e.g. [#1](https://github.com/kai-language/kai/blob/41268660a01e0d6d1f0ac8de743d91700707135e/Sources/Core/Helpers/Helpers.swift#L205), [#2](https://github.com/llvm-swift/LLVMSwift/blob/162f1632e017349b17146e33c5905f88148e55f1/Sources/LLVM/Units.swift#L125)), image/video manipulation (e.g. [#1](https://github.com/schulz0r/CoreImage-HDR/blob/c7f5264929338beebfcfbf2e420594aa2952ef6c/CoreImage-HDR/Convenience/extensions.swift#L20), [#2](https://github.com/OpsLabJPL/MarsImagesIOS/blob/f2109f38b31bf1ad2e7b5aae6916da07d2d7d08e/MarsImagesIOS/Math.swift#L17), [#3](https://github.com/chingf/CarmenaCamera/blob/bfa928ca1595770c3b99bb8aa95dc5340a0f3284/VideoCapture/Common.swift#L22), [#4](https://github.com/dehancer/IMProcessing/blob/7a7d48edb7ceeb2635219c8139aa6fb8dbf1525d/IMProcessing/Classes/Common/IMPNumericTypes.swift#L132)), and other applications and utilities such as [geography kit](https://github.com/tokyovigilante/CesiumKit/blob/7983bd742a85982d9c3303cdacc039dcb44c8a42/CesiumKit/Core/Math.swift#L597), [quantum circuit simulator](https://github.com/indisoluble/SwiftQuantumComputing/blob/008e82e0f38792372df1a428884cccb74c2732b3/SwiftQuantumComputing/Extension/Int%2BIsPowerOfTwo.swift#L24), [tournament](https://github.com/eaheckert/Tournament/blob/c09c6b3634da9b2666b8b1f8990ff62bdc4fd625/Tournament/Tournament/ViewControllers/TCreateTournamentVC.swift#L215), [blockchain](https://github.com/yeeth/BeaconChain.swift/blob/954bcb6e47b51f90eff16818719320a228afe891/Sources/BeaconChain/Extensions/Int.swift#L5), [3D-engine](https://github.com/xuzhao-nick/3DEngine/blob/c3aab94f2bce5e29f7988b0d7c1e075d74076ad7/3DEngine/3DEngine/MoreMath.swift#L50), etc. + +Apart from the *is-power-of-two* usage, queries on *non-2* bases may also be practical, though they are much less common. In signal processing, for example, the efficient radix-4 algorithms can be applied when the FFT size is a power of 4. + +As a result, it would be beneficial if we could have such an API supported in the standard library. To be more specific, it is an extension method in the form of `isPower(of:)` to the `BinaryInteger` protocol, checking whether the `BinaryInteger` *self* is a power of the base specified by the parameter. Let us discuss the impacts in the following aspects: + +### Readability + +A classic approach to check if an integer `n` is power of two is to test whether the condition `n > 0 && n & (n - 1) == 0` holds. Although such code is efficient, its underlying functionality is not intuitive to many people. As a result, programmers would have to put additional information somewhere (e.g. in the comments nearby) to make it clear. + +Below is an example when making an assertion that `bucketCount` is power of two. The classic approach is applied, followed by an additional error message for necessary clarification purpose. However, if we had the proposed API available, the code would become more fluent and concise. +```swift +// Example (1) - apple/swift/stdlib/public/core/HashTable.swift +internal struct _HashTable { + internal init(words: UnsafeMutablePointer, bucketCount: Int) { + _internalInvariant(bucketCount > 0 && bucketCount & (bucketCount - 1) == 0, + "bucketCount must be a power of two") + +// _internalInvariant(bucketCount.isPower(of: 2)) --- proposed solution + + ... + } +} +``` + +### Efficiency + +The user-implemented code may be less performant (e.g. [#1](https://github.com/OpsLabJPL/MarsImagesIOS/blob/f2109f38b31bf1ad2e7b5aae6916da07d2d7d08e/MarsImagesIOS/Math.swift#L17), [#2](https://github.com/eaheckert/Tournament/blob/c09c6b3634da9b2666b8b1f8990ff62bdc4fd625/Tournament/Tournament/ViewControllers/TCreateTournamentVC.swift#L215)), since some developers are not aware of the classic approach as described above. + +The example below shows a controversial approach, which employs the `llvm.ctpop` intrinsic to count the number of 1 bits. It can be expensive when the hardware does not have relevant `popcount` instruction support. +```swift +// Example (2) - apple/swift/stdlib/public/core/Integers.swift +extension BinaryInteger { + internal func _description(radix: Int, uppercase: Bool) -> String { + // Bit shifting can be faster than division when `radix` is a power of two + let isRadixPowerOfTwo = radix.nonzeroBitCount == 1 + ... + } +} +``` + +### Abstraction + +Some developers, especially those unfamiliar to the integer type hierarchy, may have their own implementation targeting inappropriate types. + +The following example presents very similar implementation individually for `UInt` and `Int`. Such code duplication could be avoided if it targeted `BinaryInteger`. +```swift +// Example (3) - apple/swift/stdlib/public/core/Misc.swift +func _isPowerOf2(_ x: UInt) -> Bool { + // implementation +} + +func _isPowerOf2(_ x: Int) -> Bool { + // implementation very similar to above +} +``` + +### Discoverability + +IDEs can aid discoverability by suggesting `isPower(of:)` as part of autocompletion on integer types. Notably, it works as a companion to other existing integral query APIs (e.g. `isMultiple(of:)`), and they can help improve the discoverability of each other. + + +## Proposed solution + +Our solution is to introduce a public API `isPower(of:)`, as an extension method, to the `BinaryInteger` protocol. It provides a standard implementation, which can be adopted by any type that conforms to this protocol. With regard to semantics, it returns `true` iff `self` is a power of the input `base`. To be more specific, it holds when (1) `self` is one (i.e., any base to the zero power), or (2) `self` equals the product of one or more `base`s. Note that this API sits at the `BinaryInteger` protocol level, and it is expected to properly handle negative integers when the type `Self` is signed. + +```swift +// In the standard library +extension BinaryInteger { + @inlinable public func isPower(of base: Self) -> Bool { + // implementation described in Detailed Design section + } +} + +// In user code +let x: Int = Int.random(in: 0000..<0288) +1.isPower(of: x) // 'true' since x^0 == 1 +1000.isPower(of: 10) // 'true' since 10^3 == 1000 +(-1).isPower(of: 1) // 'false' +(-32).isPower(of: -2) // 'true' since (-2)^5 == -32 +``` + + +## Detailed design + +A reference implementation can be found in [pull request #24766](https://github.com/apple/swift/pull/24766). + +### Overall Design + +To make the API efficient for most use cases, the implementation is based on a fast-/slow-path pattern. We presents a generic implementation, which is suitable for all inputs, as the slow path; and meanwhile provide some particularly optimized implementation for frequently used inputs (e.g. 2) individually as the fast paths. The high-level solution is illustrated below. + +```swift +extension BinaryInteger { + @inlinable public func isPower(of base: Self) -> Bool { + // Fast path when base is one of the common cases. + if base == common_base_A { return self._isPowerOfCommonBaseA } + if base == common_base_B { return self._isPowerOfCommonBaseB } + if base == common_base_C { return self._isPowerOfCommonBaseC } + ... + // Slow path for other bases. + return self._slowIsPower(of: base) + } + + @inlinable internal var _isPowerOfCommonBaseA: Bool { /* optimized implementation */ } + @inlinable internal var _isPowerOfCommonBaseB: Bool { /* optimized implementation */ } + @inlinable internal var _isPowerOfCommonBaseC: Bool { /* optimized implementation */ } + ... + @usableFromInline internal func _slowIsPower(of base: Self) -> Bool { /* generic implementation */ } +} +``` +Calling the public API `isPower(of: commonBaseK)` is expected to be as performant as directly calling the optimized version `_isPowerOfCommonBaseK`, if argument `commonBaseK` is a constant and the type `Self` is obvious enough (e.g. built-in integers) to the compiler to apply **constant-folding** to the `base == commonBaseK` expression, followed by a **simplify-cfg** transformation. + +### Fast path when base is two + +As for this fast path, it is **not** recommended to directly apply the classic approach to any type that conforms to `BinaryInteger` like this: +```swift +extension BinaryInteger { + @inlinable internal var _isPowerOfTwo: Bool { + return self > 0 && self & (self - 1) == 0 + } +} +``` +Because when `Self` is some complicated type, the arithmetic, bitwise and comparison operations can be expensive and thus lead to poor performance. In this case, the `BinaryInteger.words`-based implementation below is preferred, where word is supported by the hardware and the operations are expected to be efficient. +```swift +extension BinaryInteger { + @inlinable internal var _isPowerOfTwo: Bool { + let words = self.words + guard !words.isEmpty else { return false } + + // If the value is represented in a single word, perform the classic check. + if words.count == 1 { + return self > 0 && self & (self - 1) == 0 + } + + // Return false if it is negative. Here we only need to check the most + // significant word (i.e. the last element of `words`). + if Self.isSigned && Int(bitPattern: words.last!) < 0 { + return false + } + + // Check if there is exactly one non-zero word and it is a power of two. + var found = false + for word in words { + if word != 0 { + if found || word & (word - 1) != 0 { return false } + found = true + } + } + return found + } +} +``` + +### Fast path when base itself is a power of two + +As long as we have `_isPowerOfTwo` available, it becomes easy to implement such fast path when the base itself is power of two. It takes advantages of the existing APIs `isMultiple(of:)` and `trailingZeroBitCount`. The code can be written as below. + +```swift +extension BinaryInteger { + @inlinable internal func _isPowerOf(powerOfTwo base: Self) -> Bool { + _precondition(base._isPowerOfTwo) + guard self._isPowerOfTwo else { return false } + return self.trailingZeroBitCount.isMultiple(of: base.trailingZeroBitCount) + } +} +``` + +### Fast path when base is ten + +Unfortunately, there isn't any known super efficient algorithm to test if an integer is power of ten. Some optimizations are discussed in the forum to help boost the performance to some extent. Since the optimizations are conceptually non-trivial, they are not described here. Please refer to the [pitch thread](https://forums.swift.org/t/adding-ispowerof2-to-binaryinteger/24087/38) for details. + + +### Slow path + +The slow path is generic for any input base, and the algorithm is standard. It handles some corner cases when `base.magnitude <= 1` in the first place; and then it repeatedly multiplies `base` to see if it can be equal to `self` at some point. Attentions are required to avoid overflow of the multiplications. + +```swift +extension BinaryInteger { + @usableFromInline internal func _slowIsPower(of base: Self) -> Bool { + // If self is 1 (i.e. any base to the zero power), return true. + if self == 1 { return true } + + // Here if base is 0, 1 or -1, return true iff self equals base. + if base.magnitude <= 1 { return self == base } + + // At this point, we have base.magnitude >= 2. Repeatedly perform + // multiplication by a factor of base, and check if it can equal self. + let (bound, remainder) = self.quotientAndRemainder(dividingBy: base) + guard remainder == 0 else { return false } + var x: Self = 1 + while x.magnitude < bound.magnitude { x *= base } + return x == bound + } +} +``` + +## Source compatibility + +The proposed solution is an additive change. + +## Effect on ABI stability + +The proposed solution is an additive change. + + +## Effect on API resilience + +The proposed solution is an additive change. + + +## Alternatives considered + +### `isPower(of:)` as a protocol requirement + +Making the public API as a protocol requirement instead of an extension methods reserves future flexibility. It can give users a chance to provide their own optimization on some custom types where the default implementation is inefficient. However, it would increase the interface complexity of the heavily-used `BinaryInteger` protocol, so it may not be worthy. + +### `isPowerOfTwo` instead of `isPower(of:)` + +In fact, [the pitch](https://forums.swift.org/t/adding-ispowerof2-to-binaryinteger/24087) originally intended to add an API to check if an integer is power of two only. However, the more generic form `isPower(of:)` is favored by the community. This case is similar to [SE-0225](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0225-binaryinteger-iseven-isodd-ismultiple.md), which initially proposed `isEven` and `isOdd` as well, but ended up with `isMultiple(of:)` accepted only. + +### Choices for fast paths + +In the Detailed Design section, there are three specific cases of fast paths presented. Of course, we can have other choices. The fast path for base two is essential, since it is the most common case in real world applications and should always be kept as a fast path. We may remove the fast path for base ten if it is considered introducing too much complexity to the standard library. + +### Alternative optimizations for `_isPowerOfTwo` + +Apart from the classic approach, there are other optimization schemes to check if an integer is power of two. Two candidates are `ctpop`-based and `cttz`-based approaches, whose implementation is shown below. + +```swift +extension FixedWidthInteger { + internal var _isPowerOfTwo_ctpop: Bool { + return self > 0 && self.nonzeroBitCount == 1 + } +} + +extension BinaryInteger { + internal var _isPowerOfTwo_cttz: Bool { + return (self > 0) && (self == (1 as Self) << self.trailingZeroBitCount) + } +} +``` +As per [the experimental results](https://github.com/apple/swift/pull/24766#issuecomment-492237146), they are overall less performant than the proposed solution on the types and platforms tested. In addition, the `ctpop`-based approach narrows down the scope from `BinaryInteger` to `FixedWidthInteger`. + +## Acknowledgments + +This proposal has been greatly improved by the community. Below are some cases of significant help. + +- Steve Canon followed the pitch all the way through and provided a lot of valuable comments to keep the work on the right track. +- Jens Persson showed some inspiration in [an earlier thread](https://forums.swift.org/t/even-and-odd-integers/11774/117), and discussed some use cases as well as its extendability to floating point types. +- Nevin Brackett-Rozinsky gave many details to optimize the implementation, and discovered [SR-10657](https://bugs.swift.org/browse/SR-10657) during the discussion. +- Michel Fortin provided an efficient solution to the fast path for base 10 (i.e. checking if an integer is power of ten), together with thorough explanation. +- Jordan Rose gave prompt and continued comments on the PR, and advised the API should better be an extension method rather than a protocol requirement. +- Erik Strottmann suggested a more appropriate naming `_isPowerOfTwo` instead of `_isPowerOf2`. +- Antoine Coeur had valuable discussions. diff --git a/proposals/0289-result-builders.md b/proposals/0289-result-builders.md new file mode 100644 index 0000000000..2e6a4fdef6 --- /dev/null +++ b/proposals/0289-result-builders.md @@ -0,0 +1,1270 @@ +# Result builders + +* Proposal: [SE-0289](0289-result-builders.md) +* Authors: [John McCall](https://github.com/rjmccall), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.4)** +* Previous Revisions: [1st](https://github.com/swiftlang/swift-evolution/blob/51c99447562e749b23f82184c99c0ddfb07a71df/proposals/0289-function-builders.md) + +Table of Contents +================= + * [Result builders](#result-builders) + * [Changes from the first revision](#changes-from-the-first-revision) + * [Introduction](#introduction) + * [Motivation](#motivation) + * [Detailed design](#detailed-design) + * [Result builder types](#result-builder-types) + * [Result builder attributes](#result-builder-attributes) + * [Result-building methods](#result-building-methods) + * [The result builder transform](#the-result-builder-transform) + * [Statement blocks](#statement-blocks) + * [Declaration statements](#declaration-statements) + * [Expression statements](#expression-statements) + * [Assignments](#assignments) + * [Selection statements](#selection-statements) + * [Imperative control-flow statements](#imperative-control-flow-statements) + * [Exception-handling statements](#exception-handling-statements) + * [do statements](#do-statements) + * [for..in loops](#forin-loops) + * [Compiler Diagnostic Directives](#compiler-diagnostic-directives) + * [Availability](#availability) + * [Example](#example) + * [Type inference](#type-inference) + * [Result builder bodies](#result-builder-bodies) + * [Inferring result builders from protocol requirements](#inferring-result-builders-from-protocol-requirements) + * [Implicit memberwise initializer](#implicit-memberwise-initializer) + * [Source compatibility](#source-compatibility) + * [Effect on ABI stability and API resilience](#effect-on-abi-stability-and-api-resilience) + * [Future Directions](#future-directions) + * ["Simple" result builder protocol](#simple-result-builder-protocol) + * [Stateful result builders](#stateful-result-builders) + * [Transforming declarations](#transforming-declarations) + * [Virtualized Abstract Syntax Trees (ASTs)](#virtualized-abstract-syntax-trees-asts) + * [Alternatives considered](#alternatives-considered) + * [Additional control-flow statements](#additional-control-flow-statements) + * [Builder-scoped name lookup](#builder-scoped-name-lookup) + * [Dropping Void/Never values](#dropping-voidnever-values) + +## Changes from the first revision + +* The feature is now called *result builders* (rather than "function builders"). James Dempsey provided some [exploration and rationale](https://forums.swift.org/t/se-0289-function-builders/39889/75) for naming that led to this choice. +* Although not part of the proposal itself, the [implementation quality has been improved](https://github.com/apple/swift/pull/33972) to help guide users in writing their result builders, with code completions and Fix-Its to help define the builder methods. +* Added a section on [dropping `Void`/`Never` values](#dropping-voidnever-values) to the list of alternatives considered. +* Clarified the role of each of the [result-building methods](#result-building-methods), and provided declarations for each that are easier to understand and copy/paste. + +## Introduction + +This proposal describes *result builders*, a new feature which allows certain functions (specially-annotated, often via context) to implicitly build up a result value from a sequence of components. + +The basic idea is that the results of the function's statements are collected using a builder type, like so: + +```swift +// Original source code: +@TupleBuilder +func build() -> (Int, Int, Int) { + 1 + 2 + 3 +} + +// This code is interpreted exactly as if it were this code: +func build() -> (Int, Int, Int) { + let _a = TupleBuilder.buildExpression(1) + let _b = TupleBuilder.buildExpression(2) + let _c = TupleBuilder.buildExpression(3) + return TupleBuilder.buildBlock(_a, _b, _c) +} +``` + +In this example, all the statements are expressions and so produce a single value apiece. Other statements, like `let`, `if`, and `while`, are variously either handled differently or prohibited; see the proposal details below. + +In effect, this proposal allows the creation of a new class of embedded domain-specific languages in Swift by applying *builder transformations* to the statements of a function. The power of these builder transformations is intentionally limited so that the result preserves the dynamic semantics of the original code: the original statements of the function are still executed as normal, it's just that values which would be ignored under normal semantics are in fact collected into the result. The use of an *ad hoc* protocol for the builder transformation leaves room for a wide variety of future extension, whether to support new kinds of statements or to customize the details of the transformation. A similar builder pattern was used successfully for string interpolation in [SE-0228](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0228-fix-expressiblebystringinterpolation.md). + +Result builders have been a "hidden" feature since Swift 5.1, under the name "function builder", and the implementation (and its capabilities) have evolved since then. They are used most famously by [SwiftUI](https://developer.apple.com/xcode/swiftui/) to declaratively describe user interfaces, but others have also experimented with [building Swift syntax trees](https://swiftpack.co/package/akkyie/SyntaxBuilder), [testing](https://www.dotconferences.com/2020/02/kaya-thomas-swift-techniques-for-testing), +[a Shortcuts DSL](https://github.com/a2/swift-shortcuts), [a CSS DSL](https://github.com/carson-katri/swift-css/blob/master/Sources/CSS/CSSBuilder.swift), and [an alternative SwiftPM manifest format](https://forums.swift.org/t/declarative-package-description-for-swiftpm-using-function-builders/28699). There's a GitHub repository dedicated to [awesome function builders](https://github.com/carson-katri/awesome-function-builders) with more applications. + +## Motivation + +It's always been a core goal of Swift to allow the creation of great libraries. A lot of what makes a library great is its interface, and Swift is designed with rich affordances for building expressive, type-safe interfaces for libraries. In some cases, a library's interface is distinct enough and rich enough to form its own miniature language within Swift. We refer to this as a *Domain Specific Language* (DSL), because it lets you better describe solutions within a particular problem domain. + +Result builders target a specific kind of interface that involves the declaration of list and tree structures, which are useful in many problem domains, including generating structured data (e.g. XML or JSON), UI view hierarchies (notably including Apple's SwiftUI framework, mentioned above), and similar use cases. In this proposal, we will be primarily working with code which generates an HTML DOM hierarchy, somewhat like a web templating language except in code; credit goes to Harlan Haskins for this example. + +Suppose that you have a program, part of which generates HTML. You could, of course, directly generate a `String`, but that's error-prone (you have to make sure you're handling escapes and closing tags correctly throughout your code) and would make it hard to structurally process the HTML before sending it out. An alternative approach is to generate a DOM-like representation of the HTML and then render it to a `String` as a separate pass: + + +```swift +protocol HTML { + func renderAsHTML(into stream: HTMLOutputStream) +} + +extension String: HTML { + func renderAsHTML(into stream: HTMLOutputStream) { + stream.writeEscaped(self) + } +} + +struct HTMLNode: HTML { + var tag: String + var attributes: [String: String] = [:] + var children: [HTML] = [] + + func renderAsHTML(into stream: HTMLOutputStream) { + stream.write("<") + stream.write(tag) + for (k, v) in attributes.sort { $0.0 < $1.0 } { + stream.write(" ") + stream.write(k) + stream.write("=") + stream.writeDoubleQuoted(v) + } + if children.isEmpty { + stream.write("/>") + } else { + stream.write(">") + for child in children { + child.renderAsHTML(into: stream) + } + stream.write("") + } + } +} +``` + +To make it easier to build these HTML hierarchies, we can define a bunch of convenient node-builder functions corresponding to common nodes: + +```swift +func body(_ children: [HTML]) -> HTMLNode { ... } +func division(_ children: [HTML]) -> HTMLNode { ... } +func paragraph(_ children: [HTML]) -> HTMLNode { ... } +func header1(_ text: String) -> HTMLNode { ... } +``` + +Unfortunately, even with these helper functions, it's still pretty awkward to actually produce a complex hierarchy because of all the lists of children: + +```swift +return body([ + division([ + header1("Chapter 1. Loomings."), + paragraph(["Call me Ishmael. Some years ago"]), + paragraph(["There is now your insular city"]) + ]), + division([ + header1("Chapter 2. The Carpet-Bag."), + paragraph(["I stuffed a shirt or two"]) + ]) +]) +``` + +The first problem is that there's a lot of punctuation here: commas, parentheses, and brackets. This is a pretty superficial problem, and it's probably not a showstopper by itself, but it is something that it'd be nice to avoid, because it does distract a little from the content. + +The second problem is that, because we're using array literals for the children, the type-checker is going to require the elements to have a homogeneous type. That's fine for our HTML example, but it's limiting in general, because some trees are more generic and would benefit from propagating the types of the children into the type of the node. For example, SwiftUI uses this for various optimizations within the view hierarchy. + +The biggest problem, though, is that it's awkward to change the structure of this hierarchy. That's fine if our hierarchy is just the static contents of *Moby Dick*, but in reality, we're probably generating HTML that should vary significantly based on dynamic information. For example, if we wanted to allow chapter titles to be turned off in the output, we'd have to restructure that part of the code like this: + +```swift +division((useChapterTitles ? [header1("Chapter 1. Loomings.")] : []) + + [paragraph(["Call me Ishmael. Some years ago"]), + paragraph(["There is now your insular city"])]) +``` + +It's also harder to use good coding practices within this hierarchy. For example, suppose there's a common string we want to use many times, and so we want to create a variable for it: + +```swift +let chapter = spellOutChapter ? "Chapter " : "" + ... +header1(chapter + "1. Loomings.") + ... +header1(chapter + "2. The Carpet-Bag.") +``` + +Most programmers would advise declaring this variable in as narrow a scope as possible, and as close as possible to where it's going to be used. But because the entire hierarchy is an expression, and there's no easy way to declare variables within expressions, every variable like this has to be declared above the entire hierarchy. (Now, it's true that there's a trick for declaring locals within expressions: you can start a closure, which gives you a local scope that you can use to declare whatever you want, and then immediately call it. But this is awkward in its own way and significantly adds to the punctuation problem.) + +Some of these problems would be solved, at least in part, if the hierarchy was built up by separate statements: + +```swift +let chapter = spellOutChapter ? "Chapter " : "" + +let d1header = useChapterTitles ? [header1(chapter + "1. Loomings.")] : [] +let d1p1 = paragraph(["Call me Ishmael. Some years ago"]) +let d1p2 = paragraph(["There is now your insular city"]) +let d1 = division(d1header + [d1p1, d1p2]) + +let d2header = useChapterTitles ? [header1(chapter + "2. The Carpet-Bag.")] : [] +let d2p1 = paragraph(["I stuffed a shirt or two"]) +let d2 = division(d2header + [d2p1]) + +return body([d1, d2]) +``` + +But in most ways, this is substantially worse. There's quite a lot of extra code that's made it much more difficult to track what's really going on. That's especially true with all the explicit data flow, where it can be tough to piece together what nodes are children of others; moreover, that code is as tedious to write as it is to read, making it very prone to copy-paste bugs. Furthermore, the basic structure of the hierarchy used to be clear from the code, and that's been completely lost: all of the nicely-nested calls to node builders have been flattened into one sequence. Also, while optional children are a common problem for this hierarchy, the actual code to handle them has to be repeated over and over again, leading to boilerplate and bugs. Overall, this is not a good approach at all. + +What we really want is a compromise between these two approaches: + +* We want the programming flexibility that comes from building things up with ordinary blocks of code: the ability to have local declarations and explicit control flow. + +* We want the explicit recursive structure and implicit data flow that comes from building things up with expression operands. + +This suggests a straightforward resolution: allow certain blocks of code to have implicit data flow out of the block and into the context which entered the block. The goal here is to allow this node hierarchy to be declared something like this: + +```swift +return body { + let chapter = spellOutChapter ? "Chapter " : "" + division { + if useChapterTitles { + header1(chapter + "1. Loomings.") + } + paragraph { + "Call me Ishmael. Some years ago" + } + paragraph { + "There is now your insular city" + } + } + division { + if useChapterTitles { + header1(chapter + "2. The Carpet-Bag.") + } + paragraph { + "I stuffed a shirt or two" + } + } +} +``` + +The above has to be embedded into the ordinary language somehow, which means that at least the outermost layer must obey something like ordinary language rules. Under ordinary language rules, this is a function call to `body` passing a trailing closure. It makes sense, then, that what we're doing is taking the body of the anonymous function and applying some sort of transformation to it. This raises a series of separate questions: + +1. What is it about this source code that triggers the transformation? We have chosen not to require an explicit annotation on every closure that needs transformation; see Alternatives Considered for a discussion. So somehow this must get picked up from the fact that we're passing the closure to `body`. + +2. Given that the transformation is going to collect a bunch of information, how does that information get back to `body`? Since the transformation is working on a function body, this one's straightforward: the collected information will just be the return value of the function. There's no requirement to support this transformation simultaneously with ordinary `return`. + +3. Given that the transformation has collected a sequence of partial results, how do they get combined to produce a return value? We could simply always produce a tuple, but that isn't necessarily what the caller actually wants. In particular, it may be useful to allow the DSL to do different things for different partial results. The logical answer is that there should be a function somehow provided by the DSL to combine the partial results, and this function might need to be generic or overloaded. + +4. Given that the transformation might collect a partial result within optionally-executed code (e.g. in an `if` statement), what should be passed to the combining function? The transformation can be taught to produce an optional value of the partial result type, but the DSL needs to be able to distinguish between a partial result that is optionally produced and a partial result that coincidentally happens to be of optional type. The logical answer is that there should be a function provided by the DSL to translate optional partial results into "ordinary" partial results that can be collected normally. + +These last two points (and some other considerations) strongly suggest that the DSL should be identified with a type that can provide an arbitrary namespace of functions that can be used as *ad hoc* customization points for the transformation. + +## Detailed design + +[SE-0258](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0258-property-wrappers.md) introduced the concept of custom attributes to Swift, an approach we build on here. + +### Result builder types + +A *result builder type* is a type that can be used as a result builder, which is to say, as an embedded DSL for collecting partial results from the expression-statements of a function and combining them into a return value. + +A result builder type must satisfy two basic requirements: + +* It must be annotated with the `@resultBuilder` attribute, which indicates that it is intended to be used as a result builder type and allows it to be used as a custom attribute. + +* It must supply at least one static `buildBlock` result-building method. + +In addition, it may supply a sufficient set of other result-building methods to translate the kinds of functions which the DSL desires to translate. + +### Result builder attributes + +A result builder type can be used as an attribute in two different syntactic positions. The first position is on a `func`, `var`, or `subscript` declaration. For the `var` and `subscript`, the declaration must define a getter, and the attribute is treated as if it were an attribute on that getter. A result builder attribute used in this way causes the result builder transform to be applied to the body of the function; it is not considered part of the interface of the function and does not affect its ABI. + +A result builder type can also be used as an attribute on a parameter of function type, including on parameters of protocol requirements. A result builder attribute used in this way causes the result builder transform to be applied to the body of any explicit closures that are passed as the corresponding argument, unless the closure contains a `return` statement. This is considered part of the interface of the function and can affect source compatibility, although it does not affect its ABI. + +### Result-building methods + +To be useful as a result builder, the result builder type must provide a sufficient subset of the result-building methods. The protocol between the compiler's generated code and the result builder type is intended to be *ad hoc* and arbitrarily extensible in the future. + +Result-building methods are `static` methods that can be called on the result builder type. Calls to result-building methods are type-checked as if a programmer had written `BuilderType.()`, where the arguments (including labels) are as described below; therefore, all the ordinary overload resolution rules apply. However, in some cases the result builder transform changes its behavior based on whether a result builder type declares a certain method at all; it is important to note that this is a weaker check, and it may be satisfied by a method that cannot in fact ever be successfully called on the given builder type. + +This is a quick reference for the result-building methods currently proposed. The typing here is subtle, as it often is in macro-like features. In the following descriptions, `Expression` stands for any type that is acceptable for an expression-statement to have (that is, a raw partial result), `Component` stands for any type that is acceptable for a partial or combined result to have, and `FinalResult` stands for any type that is acceptable to be ultimately returned by the transformed function. + +* `buildBlock(_ components: Component...) -> Component` is used to build combined results for statement blocks. It is required to be a static method in every result builder. + +* `buildOptional(_ component: Component?) -> Component` is used to handle a partial result that may or may not be available in a given execution. When a result builder provides `buildOptional(_:)`, the transformed function can include `if` statements without an `else`. + +* `buildEither(first: Component) -> Component` and `buildEither(second: Component) -> Component` are used to build partial results when a selection statement produces a different result from different paths. When a result builder provides these methods, the transformed function can include `if` statements with an `else` statement as well as `switch` statements. + +* `buildArray(_ components: [Component]) -> Component` is used to build a partial result given the partial results collected from all of the iterations of a loop. When a result builder provides `buildArray(_:)`, the transformed function can include `for..in` statements. + +* `buildExpression(_ expression: Expression) -> Component` is used to lift the results of expression-statements into the `Component` internal currency type. It is optional, but when provided it allows a result builder to distinguish `Expression` types from `Component` types or to provide contextual type information for statement-expressions. + +* `buildFinalResult(_ component: Component) -> FinalResult` is used to finalize the result produced by the outermost `buildBlock` call for top-level function bodies. It is optional, and allows a result builder to distinguish `Component` types from `FinalResult` types, e.g. if it wants builders to internally traffic in some type that it doesn't really want to expose to clients. + +* `buildLimitedAvailability(_ component: Component) -> Component` is used to transform the partial result produced by `buildBlock` in a limited-availability context (such as `if #available`) into one suitable for any context. It is optional, and is only needed by result builders that might carry type information from inside an `if #available` outside it. + +The set of requirements can be summarized by the following example result builder: + +```swift +@resultBuilder +struct ExampleResultBuilder { + /// The type of individual statement expressions in the transformed function, + /// which defaults to Component if buildExpression() is not provided. + typealias Expression = ... + + /// The type of a partial result, which will be carried through all of the + /// build methods. + typealias Component = ... + + /// The type of the final returned result, which defaults to Component if + /// buildFinalResult() is not provided. + typealias FinalResult = ... + + /// Required by every result builder to build combined results from + /// statement blocks. + static func buildBlock(_ components: Component...) -> Component { ... } + + /// If declared, provides contextual type information for statement + /// expressions to translate them into partial results. + static func buildExpression(_ expression: Expression) -> Component { ... } + + /// Enables support for `if` statements that do not have an `else`. + static func buildOptional(_ component: Component?) -> Component { ... } + + /// With buildEither(second:), enables support for 'if-else' and 'switch' + /// statements by folding conditional results into a single result. + static func buildEither(first component: Component) -> Component { ... } + + /// With buildEither(first:), enables support for 'if-else' and 'switch' + /// statements by folding conditional results into a single result. + static func buildEither(second component: Component) -> Component { ... } + + /// Enables support for 'for..in' loops by combining the + /// results of all iterations into a single result. + static func buildArray(_ components: [Component]) -> Component { ... } + + /// If declared, this will be called on the partial result of an 'if + /// #available' block to allow the result builder to erase type + /// information. + static func buildLimitedAvailability(_ component: Component) -> Component { ... } + + /// If declared, this will be called on the partial result from the outermost + /// block statement to produce the final returned result. + static func buildFinalResult(_ component: Component) -> FinalResult { ... } +} +``` + +### The result builder transform + +The result builder transform is a recursive transformation operating on statement blocks and the individual statements within them. + +#### Statement blocks + +Within a statement block, the individual statements are separately transformed into sequences of statements which are then concatenated. Each such sequence may optionally produce a single *partial result*, which is an expression (typically a reference to a unique local variable defined as part of the transformation) which can be used later in the block. + +After the transformation has been applied to all the statements, a call to `buildBlock` is generated to form a *combined result*, with all the partial results as unlabelled arguments. + +If the statement block is the top-level body of the function being transformed, the final statement in the transformed block is a `return` with the combined result expression as its operand, wrapped in `buildFinalResult` if it was provided by the builder. Otherwise, the combined result is propagated outwards, typically by assigning it (possibly after a transformation) to a local variable in the enclosing block; the exact rule is specified in the section for the enclosing statement below. + +#### Declaration statements + +Local declarations are left alone by the transformation. This allows developers to factor out subexpressions freely to clarify their code, without affecting the result builder transformation. + +#### Expression statements + +An expression statement which does not perform an assignment is transformed as follows: + +* If the result builder type declares the `buildExpression` result-building method, the transformation calls it, passing the expression-statement as a single unlabeled argument. This call expression is hereafter used as the expression statement. This call is type-checked together with the statement-expression and may influence its type. + +* The statement-expression is used to initialize a unique variable of the statement-expression's result type, as if by `let v = `. This variable is treated as a partial result of the containing block. References to this variable are type-checked independently from it so that they do not affect the type of the expression. + +The ability to control how expressions are embedded into partial results is an important advantage for certain kinds of DSL, including our HTML example. In the original HTML example, we have an `HTML` protocol with a very small (and essentially fixed) number of implementing types. This would probably be more natural to represent in Swift as an `enum` rather than a `protocol`, but that would prevent a `String` from being written directly wherever an `HTML` was required, which would make complex `HTML` literals even more onerous to write in our original, pre-DSL solution: + +```swift +return body([ + division([ + header1("Chapter 1. Loomings."), + paragraph([.text("Call me Ishmael. Some years ago")]), + paragraph([.text("There is now your insular city")]) + ]), + division([ + header1("Chapter 2. The Carpet-Bag."), + paragraph([.text("I stuffed a shirt or two")]) + ]) +]) +``` + +By using a DSL with `buildExpression`, however, we can use an `enum` for its natural representational, pattern-matching, and other advantages, then just add overloads to `buildExpression` to make it easier to build common cases within the DSL: + +```swift +static func buildExpression(_ text: String) -> [HTML] { + return [.text(text)] +} + +static func buildExpression(_ node: HTMLNode) -> [HTML] { + return [.node(node)] +} + +static func buildExpression(_ value: HTML) -> [HTML] { + return [value] +} +``` + +#### Assignments + +An expression statement which performs an assignment is treated in the same manner as all other expression statements, although it will always return `()`. A result builder could choose to handle `()`-returning expression statements specially by overloading `buildExpression`, e.g., + +```swift +static func buildExpression(_: ()) -> Component { ... } +``` + +#### Selection statements + +`if`/`else` and `switch` statements produce values conditionally depending on their cases. There are two basic transformation patterns which can be used, depending on the selection statement itself; we'll show examples of each, then explain the details of the transformation. + +Consider a simple "if" statement without an "else" block: + +```swift +if i == 0 { + "0" +} +``` + +The first transformation pattern for selection statements turns the case into its own optional partial result in the enclosing block. This is a simple transformation handling code that is optionally executed: + +```swift +var vCase0: String? +if i == 0 { + var thenVar = "0" + var thenBlock = BuilderType.buildBlock(thenVar) + vCase0 = BuilderType.buildOptional(.some(thenBlock)) +} +``` + +If `if` statement doesn't have a corresponding `else` block, like in our example, the result builder transform is going create one implicitly and inject a call to `buildOptional(.none)` as follows: + +```swift +var vCase0: String? +if i == 0 { + var thenVar = "0" + var thenBlock = BuilderType.buildBlock(thenVar) + vCase0 = BuilderType.buildOptional(.some(thenBlock)) +} else { + vCase0 = BuilderType.buildOptional(.none) +} +``` + +The second transformation pattern produces a balanced binary tree of injections into a single partial result in the enclosing block. It supports `if`-`else` and `switch`. Consider the following code: + +```swift +if i == 0 { + "0" +} else if i == 1 { + "1" +} else { + generateFibTree(i) +} +``` + +Under this pattern, the example code becomes something like the following: + +```swift +let vMerged: PartialResult +if i == 0 { + var firstVar = "0" + var firstBlock = BuilderType.buildBlock(firstVar) + vMerged = BuilderType.buildEither(first: firstBlock) +} else if i == 1 { + var secondVar = "1" + var secondBlock = BuilderType.buildBlock(secondVar) + vMerged = BuilderType.buildEither(second: + BuilderType.buildEither(first: secondBlock)) +} else { + var elseVar = generateFibTree(i) + var elseBlock = BuilderType.buildBlock(elseVar) + vMerged = BuilderType.buildEither(second: + BuilderType.buildEither(second: elseBlock)) +} +``` + +The detailed transformation of selection statements proceeds as follows. The child blocks of the statement are first analyzed to determine the number *N* of cases that can produce results and whether there are any cases that don't. The implementation is permitted to analyze multiple nesting levels of statements at once; e.g. if a `case` consists solely of an `if` chain, the cases of the `if` can be treated recursively as cases of the `switch` at the implementation's discretion. A missing `else` is a separate case for the purposes of this analysis, and will be handled by `buildOptional(_:)`. + +If *N* = 0, the statement is ignored by the transformation. Otherwise, an injection strategy is chosen: + +* If the result builder type declares the `buildEither(first:)` and `buildEither(second:)` result-building methods, a full binary tree with *N* leaves (the *injection tree*) is chosen, and each result-producing case is uniquely assigned a leaf in it; these decisions are implementation-defined. A unique variable `vMerged` of fresh type is declared before the statement. + +* Otherwise, unique variables `vCase` are declared before the statement for each result-producing case. + +The transformation then proceeds as follows: + +* In each result-producing case, the transformation is applied recursively. As the final statement in the case, the combined result is injected and assigned outwards: + + * If the statement is not using an injection tree, the combined result is wrapped in `Optional.Some` and assigned to the appropriate `vCase`. + + * Otherwise, the path from the root of the injection tree to the appropriate leaf is considered. An expression is formed by the following rules and then assigned to `vMerged`: + + * For an empty path, the original combined result from the case. + + * For a left branch through the tree, a call to the result-building method `buildEither(first:)` with the argument being the injection expression for the remainder of the path. + + * For a right branch through the tree, the same but with `buildEither(second:)`. + + * Finally, if there are any non-result-producing cases, the expression is wrapped in `Optional.some`. + + For example, if the path to the case's leaf is `left`, `left`, `right`, and there are non-result-producing cases, and the original combined result is `E`, then the injection expression assigned to `vMerged` is + + ```swift + Optional.some( + BuilderType.buildEither(first: + BuilderType.buildEither(first: + BuilderType.buildEither(second: E)))) + ``` + + Note that all of the assignments to `vMerged` will be type-checked together, which should allow any free generic arguments in the result types of the injections to be unified. + +* After the statement, if there is an `if` that does not have a corresponding `else`, a new unique variable `v2` is initialized by calling the result-building method `buildOptional(_:)` with `v` as the argument, and `v2` is then a partial result of the surrounding block. Otherwise, there is a unique variable `vMerged`, and `vMerged` is a partial result of the surrounding block. + +#### Imperative control-flow statements + +`return` statements are ill-formed when they appear within a transformed function. However, note that the transformation is suppressed in closures that contain a `return` statement, so this rule is only applicable in `func`s and getters that explicitly provide the attribute. + +`break` and `continue` statements are ill-formed when they appear within a transformed function. These statements may be supported in some situations in the future, for example by treating all potentially-skipped partial results as optional. + +`guard` is provisionally ill-formed when it appears within a transformed function. Situations in which this statement may appear may be supported in the future, such as when the statement does not produce a partial result. + +#### Exception-handling statements + +`throw` statements are left alone by the transformation. + +`defer` statements are ill-formed when encountered in transformed functions. + +`do` statements with `catch` blocks are ill-formed when encountered in transformed functions. + +#### `do` statements + +`do` statements without `catch` blocks are effectively wrappers around a block statement, and are transformed accordingly: + +* A unique variable `v` is declared immediately prior to the `do`. + +* The transformation is applied recursively to the child statement block. + +* The combined result is assigned to `v` as the final statement in the child block, and `v` becomes a partial result of the containing block. + +#### `for`..`in` loops + +`for`...`in` statements execute each of the iterations of the loop, collecting the partial results from all iterations into an array. That array is then passed into `buildArray`. Specifically: + +* A unique variable `v` is declared immediately prior to the `for`. +* A unique variable `vArray` is declared immediately prior to the `for`, is given `Array` type (with as-yet-undetermined element type), and is initialized to `[]`. +* The transformation is applied recursively to the body of the `for`..`in` loop, except that the partial result produced by the body is appended to the array via a call to `vArray.append`. +* The result of calling `buildArray(vArray)` is assigned to `v`, and `v` becomes a partial result of the containing block. + +If no `buildArray` is provided, `for`..`in` loops are not supported in the body. + +### Compiler Diagnostic Directives + +`#warning` and `#error` have no run-time impact and are left unchanged by the result builder transformation. + +### Availability + +Statements that introduce limited availability contexts, such as `if #available(...)`, allow the use of newer APIs while still making the code backward-deployable to older versions of the libraries. A result builder that carries complete type information (such as SwiftUI's [`ViewBuilder`](https://developer.apple.com/documentation/swiftui/viewbuilder)) may need to "erase" type information from a limited availability context using `buildLimitedAvailability`. Here is a SwiftUI example borrowed from [Paul Hudson](https://www.hackingwithswift.com/quick-start/swiftui/how-to-lazy-load-views-using-lazyvstack-and-lazyhstack): + +```swift +@available(macOS 10.15, iOS 13.0, *) +struct ContentView: View { + var body: some View { + ScrollView { + if #available(macOS 11.0, iOS 14.0, *) { + LazyVStack { + ForEach(1...1000, id: \.self) { value in + Text("Row \(value)") + } + } + } else { + VStack { + ForEach(1...1000, id: \.self) { value in + Text("Row \(value)") + } + } + } + } + } +} +``` + +`LazyVStack` was introduced in macOS 11/iOS 14.0, but this view is also available on macOS 10.15/iOS 13.0, so it uses `if #available`. SwiftUI carries complete type information throughout the view builder closure, including conditional branches: + +```swift +static func buildEither(first: TrueContent) -> _ConditionalContent +``` + +This means that the type of the `ScrollView` will refer to `LazyVStack`, even on macOS 10.15/iOS 13.0, which results in a compilation error. `buildLimitedAvailability` provides a way for the result builder to "erase" type information it would normally keep, specifically in these situations: + +```swift +static func buildLimitedAvailability(_ content: Content) -> AnyView { .init(content) } +``` + +Consider a cut-down example focusing on the `if #available`: + +```swift +if #available(macOS 11.0, iOS 14.0, *) { + LazyVStack { } +} else { + VStack { } +} +``` + +This will be transformed as: + +```swift +let vMerged: *inferred type* +if #available(macOS 11.0, iOS 14.0, *) { + let v0 = LazyVStack { } + let v1 = ViewBuilder.buildBlock(v0) + let v2 = ViewBuilder.buildLimitedAvailability(v1) + vMerged = ViewBuilder.buildEither(first: v2) +} else { + let v3 = VStack { } + let v4 = ViewBuilder.buildBlock(v3) + vMerged = ViewBuilder.buildEither(second: v4) +} +``` + +### **Example** + +Let's return to our earlier example and work out how to define a result-builder DSL for it. First, we need to define a basic result builder type: + +```swift +@resultBuilder +struct HTMLBuilder { + // We'll use these typealiases to make the lifting rules clearer in this example. + // Result builders don't really require these to be specific types that can + // be named this way! For example, Expression could be "either a String or an + // HTMLNode", and we could just overload buildExpression to accept either. + // Similarly, Component could be "any Collection of HTML", and we'd just have + // to make buildBlock et al generic functions to support that. But we'll keep + // it simple so that we can write these typealiases. + + // Expression-statements in the DSL should always produce an HTML value. + typealias Expression = HTML + + // "Internally" to the DSL, we'll just build up flattened arrays of HTML + // values, immediately flattening any optionality or nested array structure. + typealias Component = [HTML] + + // Given an expression result, "lift" it into a Component. + // + // If Component were "any Collection of HTML", we could have this return + // CollectionOfOne to avoid an array allocation. + static func buildExpression(_ expression: Expression) -> Component { + return [expression] + } + + // Build a combined result from a list of partial results by concatenating. + // + // If Component were "any Collection of HTML", we could avoid some unnecessary + // reallocation work here by just calling joined(). + static func buildBlock(_ children: Component...) -> Component { + return children.flatMap { $0 } + } + + // We can provide this overload as a micro-optimization for the common case + // where there's only one partial result in a block. This shows the flexibility + // of using an ad-hoc builder pattern. + static func buildBlock(_ component: Component) -> Component { + return component + } + + // Handle optionality by turning nil into the empty list. + static func buildOptional(_ children: Component?) -> Component { + return children ?? [] + } + + // Handle optionally-executed blocks. + static func buildEither(first child: Component) -> Component { + return child + } + + // Handle optionally-executed blocks. + static func buildEither(second child: Component) -> Component { + return child + } +} +``` + +Next, we need to adjust our convenience functions to use it: + +```swift +func body(@HTMLBuilder makeChildren: () -> [HTML]) -> HTMLNode { + return HTMLNode(tag: "body", attributes: [:], children: makeChildren()) +} +func division(@HTMLBuilder makeChildren: () -> [HTML]) -> HTMLNode { ... } +func paragraph(@HTMLBuilder makeChildren: () -> [HTML]) -> HTMLNode { ... } +``` + +Now we can go back to the example code and see how the transformation acts on a small part of it: + +```swift +division { + if useChapterTitles { + header1(chapter + "1. Loomings.") + } + paragraph { + "Call me Ishmael. Some years ago" + } + paragraph { + "There is now your insular city" + } +} +``` + +The transformation proceeds one-by-one through the top-level statements of the closure body passed to `division`. + +For the `if` statement, we see that there are two cases: the “then” case and the implicit “else” case. The first produces a result (because it has a non-assignment expression-statement), the second does not. We apply the recursive transformation to the “then” case: + +```swift + if useChapterTitles { + let v0: [HTML] = HTMLBuilder.buildExpression(header1(chapter + "1. Loomings.")) + // combined result is HTMLBuilder.buildBlock(v0) + } +``` + +We're not using an injection tree because this is a single `if` without an `else`: + +```swift + var v0_opt: [HTML]? + if useChapterTitles { + let v0: [HTML] = HTMLBuilder.buildExpression(header1(chapter + "1. Loomings.")) + v0_opt = v0 + } + let v0_result = HTMLBuilder.buildOptional(v0_opt) + // partial result is v0_result +``` + +The two calls to `paragraph` happen to involve arguments which are also transformed blocks; we'll leave those alone for now, but +suffice it to say that they'll also get transformed in time when the type-checker gets around to checking those calls. These are just non-assignment expression-statements, so we just lift them into the `Component` type: + +```swift +division { + var v0_opt: [HTML]? + if useChapterTitles { + let v0: [HTML] = HTMLBuilder.buildExpression(header1(chapter + "1. Loomings.")) + v0_opt = v0 + } + let v0_result = HTMLBuilder.buildOptional(v0_opt) + + let v1 = HTMLBuilder.buildExpression(paragraph { + "Call me Ishmael. Some years ago" + }) + + let v2 = HTMLBuilder.buildExpression(paragraph { + "There is now your insular city" + }) + + // partial results are v0_result, v1, v2 +} +``` + +Finally, we finish this block with a call to `buildBlock`. This is the top-level body of the function, but the result builder type doesn't declare `buildFinalResult`, that result is returned directly: + +```swift +division { + var v0_opt: [HTML]? + if useChapterTitles { + let v0: [HTML] = HTMLBuilder.buildExpression(header1(chapter + "1. Loomings.")) + v0_opt = v0 + } + let v0_result = HTMLBuilder.buildOptional(v0_opt) + + let v1 = HTMLBuilder.buildExpression(paragraph { + "Call me Ishmael. Some years ago" + }) + + let v2 = HTMLBuilder.buildExpression(paragraph { + "There is now your insular city" + }) + + return HTMLBuilder.buildBlock(v0_result, v1, v2) +} +``` + +This closure has now been completely transformed (except for the nested closures passed to `paragraph`). + +## Type inference + +### Result builder bodies +Type inference in result builder bodies follows from the syntactic effects of the result builder transformation. +For example, when applying the result builder to the following closure: + +```swift +{ + 42 + 3.14159 +} +``` + +the result builder transformation produces + +```swift +let v1 = 42 +let v2 = 3.14159 +return Builder.buildBlock(v1, v2) +``` + +The types for `v1` and `v2` are determined independently by the normal type inference rules to `Int` and `Double`, +respectively, then `buildBlock` can operate on both types to produce the final result of the closure. +However, the type of `buildBlock` cannot have any effect on how the types of `v1` and `v2` are +computed. For example, if the builder contained a `buildBlock` like the following: + +```swift +func buildBlock(_ a: T, _ b: T) -> T { ... } +``` + +Then the call to `buildBlock(v1, v2)` will fail because `Int` and `Double` have different types, even though the integer literal `42` could have been treated as a `Double` if type inference were permitted to propagate information "backward" to affect `v1`. + +Note that the first implementation of result builders in Swift 5.1 used a different syntactic transform that *did* allow such backward propagation, e.g., + +```swift +return Builder.buildBlock(42, 3.14159) // not proposed; example only +``` + +in which case the `42` would be treated as a `Double`. There are several reasons why allowing such "backward" propagation of type information is undesirable for result builders: +* The type inference model would be different from normal closures or function bodies, which is a divergence that makes the mental model more complicated +* Type checker performance with moderate-to-large result builder bodies was unacceptable, because backward propagation introduced exponential behavior. The implementation of [one-way constraints](https://github.com/apple/swift/pull/26661) for result builders (which introduced the current behavior) resolved most reported "expression too complex to be solved in a reasonable time" issues with SwiftUI code. + +### Inferring result builders from protocol requirements + +Most result builder transformations are applied implicitly, without the client of the API writing the name of the result builder. For example, given the following API: + +```swift +func paragraph(@HTMLBuilder makeChildren: () -> [HTML]) -> HTMLNode { ... } +``` + +The result builder `HTMLBuilder` is applied at each call site, implicitly, when the closure argument is matched to the parameter that has a result builder attribute: + +```swift +paragraph { + "Call me Ishmael. Some years ago" +} +``` + +Most function declarations are standalone, so only the explicit result builder annotation can enable the transformation. However, result builder DSLs like SwiftUI tend to have a central protocol to which many different types conform. A typical SwiftUI view might look something like this: + +```swift +struct ContentView: View { + @ViewBuilder var body: some View { + Image(named: "swift") + Text("Hello, Swift!") + } +} +``` + +Nearly every `body` for a SwiftUI view can use `@ViewBuilder`, because `body` defines a `View`, and those are best built with a `ViewBuilder`. To eliminate the boilerplate from writing `@ViewBuilder` on each, one can annotate `body` with `@ViewBuilder` in the `View` protocol itself: + +```swift +protocol View { + associatedtype Body: View + @ViewBuilder var body: Body { get } +} +``` + +When a `View`-conforming type defines its `body`, the `@ViewBuilder` attribute is inferred from the protocol requirement it satisfies, implicitly applying the result builder transform. This inference occurs unless: +* The function or property already has a result builder attribute explicitly written on it, or +* The body of the function or property getter contains an explicit `return` statement. + +### Implicit memberwise initializer + +Result builders are designed with composition in mind, and it is common to have a number of small structures that use result builders to describe their child content. For example, a custom VStack in SwiftUI might look like this: + +```swift +struct CustomVStack: View { + let content: () -> Content + + var body: some View { + VStack { + // custom stuff here + content() + } + } +} +``` + +However, this custom VStack doesn't work with result builder syntax without writing an initializer to introduce the `@ViewBuilder` attribute: + +```swift +init(@ViewBuilder content: @escaping () -> Content) { + self.content = content +} +``` + +A result builder attribute can be placed on a stored property. This introduces the result builder attribute on the corresponding parameter of the implicit memberwise initializer. In other words, changing the `CustomVStack` definition to the following: + +```swift +struct CustomVStack: View { + @ViewBuilder let content: () -> Content + + var body: some View { + VStack { + // custom stuff here + content() + } + } +} +``` + +implicitly produces the memberwise initializer shown above. + +A result builder attribute can also be placed on a stored property whose type does not [structurally resemble function type](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0286-forward-scan-trailing-closures.md#structural-resemblance-to-a-function-type). In this case, the implicit memberwise initializer will have a corresponding function parameter that is a result-builder-attributed closure returning the property's type, and the body of the initializer will call the function. For example, given: + +```swift +struct CustomHStack: View { + @ViewBuilder let content: Content + + var body: some View { + HStack { + // custom stuff here + content + } + } +} +``` + +the implicit memberwise initializer would have the following definition: + +```swift +init(@ViewBuilder content: () -> Content) { + self.content = content() +} +``` + +This idea was reported as [SR-13188](https://bugs.swift.org/browse/SR-13188), and the examples are pulled from there. + +## Source compatibility + +Result builders are an additive feature which should not affect existing source code. + +Because some decisions with result builders are implementation-defined, e.g. the structure of the injection tree for `switch` statements, it is possible that certain DSLs will observe differences in behavior across future compiler versions if they are written to observe such differences. + +## Effect on ABI stability and API resilience + +Result builders are based on compile-time code generation and do not require support from the language runtime or standard library. + +Because result builders are essentially a kind of macro system, where the details of expansion are basically an aspect of the current implementation rather than necessarily a stable interface, library authors are encouraged to make as much as possible inlinable and, if possible, non-ABI. + +## Future Directions + +There are a number of future directions that could be layered on top of this proposal without compromising its basic design. Several of them are covered here. + +### "Simple" result builder protocol + +On the Swift forums, @anreitersimon [demonstrated](https://forums.swift.org/t/pitch-2-function-builders/39410/6) the ability to use protocols to make it easier to define new result builders that support all of the syntax, and for which all expressions have the same type. Their example (slightly tuned) follows. The basic idea is to form a tree describing the result: + +```swift +enum Either { + case first(T) + case second(U) +} + +indirect enum ResultBuilderTerm { + case expression(Expression) + case block([ResultBuilderTerm]) + case either(Either) + case optional(ResultBuilderTerm?) +} +``` + +and then define a `ResultBuilder` protocol with only a single requirement, `buildFinalResult`, to take all of the values and form the final result: + +```swift +protocol ResultBuilder { + associatedtype Expression + typealias Component = ResultBuilderTerm + associatedtype FinalResult + + static func buildFinalResult(_ component: Component) -> FinalResult +} +``` + +All of the other `build` methods---to enable `if`, `switch`, and so on---are implemented in an extension of `ResultBuilder`: + +```swift +extension ResultBuilder { + static func buildExpression(_ expression: Expression) -> Component { .expression(expression) } + static func buildBlock(_ components: Component...) -> Component { .block(components) } + static func buildOptional(_ component: Component?) -> Component { .optional(component) } + static func buildArray(_ components: [Component]) -> Component { .block(components) } + static func buildLimitedAvailability(_ component: Component) -> Component { component } +} +``` + +It then becomes possible to define new version builders with only a small amount of code, e.g., here is a builder that flattens the result term into an array: + +```swift +@resultBuilder +enum ArrayBuilder: ResultBuilder { + typealias Expression = E + typealias FinalResult = [E] + + static func buildFinalResult(_ component: Component) -> FinalResult { + switch component { + case .expression(let e): return [e] + case .block(let children): return children.flatMap(buildFinalResult) + case .either(.first(let child)): return buildFinalResult(child) + case .either(.second(let child)): return buildFinalResult(child) + case .optional(let child?): return buildFinalResult(child) + case .optional(nil): return [] + } + } +} +``` + +With some more experience, a facility like this could become part of the standard library. + +### Stateful result builders +All of the `build*` methods for a result builder type are defined to be `static`, and no instances of the result builder type are created by the result builder transform. However, the transform could create an instance of the result builder type at the beginning of the transformed function, then call `build*` instance methods on it. For example, this code adapted from [Constantino Tsarouhas](https://forums.swift.org/t/pitch-2-function-builders/39410/67)): + +```swift +struct Heading : Element { + init(@ElementBuilder(containerStyle: .inline) content: () -> Content) { … } +} + +Heading { + Text("Loomings") + Text("Call me Ishmael. Some years ago") +} +``` + +would be translated as: + +```swift +Heading { + var builder = ElementBuilder(containerStyle: .inline) + let v0 = builder.buildExpression(Text("Loomings")) + let v1 = builder.buildExpression(Text("Call me Ishmael. Some years ago")) + let v2 = builder.buildBlock(v0, v1) + return builder.buildFinalResult(v2) +} +``` + +The introduction of stateful result builders would be a pure extension. However, the same effect can be achieved by the "simple result builder protocol" described above, so it is not clear that this facility belongs in the language. + +### Transforming declarations + +Result builders leave all declaration statements unmodified. However, some DSLs might want to incorporate declarations, notifying the builder of such declarations so it can incorporate them. Here's an abstracted example based on one [provided by Konrad Malawski](https://forums.swift.org/t/function-builders-and-including-let-declarations-in-built-result/37184): + +```swift +Definition { + Thing { + // "import" the Def people, in order to be able to use it in this Thing + // there may be many Defs, but this Thing only uses people + let people = Defs.people // <1> + let other = Def.otherThings // <2> + + show { + people.name // <1> + people.surname // <1> + other.information // <2> + } + filter { + equals(people.age, 42) + } + } +} +``` + +The builder could be informed of each `let` declaration, allowing it to produce a partial result, e.g., the innermost `Thing` could be transformed to: + +```swift +let people = Defs.people +let v0 = ThingBuilder.buildDeclaration(people) +let other = Def.otherThings +let v1 = ThingBuilder.buildDeclaration(other) + +let v2 = +let v3 = +let v4 = ThingBuilder.buildBlock(v0, v1, v2, v3) +``` + +Such DSLs would not change the way declarations are type checked, but would have the option to produce partial results for them, which could further eliminate boilerplate. On the other hand, without this feature one can freely use `let` to pull out subexpressions and partial computations without changing the code. For example, today one expects to be able to refactor + +```swift +a.doSomething().doSomethingElse() +``` + +into + +```swift +let b = a.doSomething() +b.doSomethingElse() +``` + +without changing semantics. That would no longer be the case for result builders that opt in to this feature by providing `buildDeclaration`. + +### Virtualized Abstract Syntax Trees (ASTs) + +The result builder model executes the transformed function directly, collecting the partial results that get passed into the builder. Some DSLs might prefer to "virtualize" the structure of the transformed function, such that the DSL can determine how the evaluation happens. A simple form of this can be achieved by using escaping autoclosures in `buildExpression:`: + +```swift +typealias DelayedValue = () -> Any + +static func buildExpression(_ value: @autoclosure @escaping () -> Any) -> DelayedValue { + return value +} +``` + +Here, the partial results provided to other build methods (e.g., `buildBlock`) will be functions that produce the value. The DSL is free to call those functions whenever it wants to produce the values. + +However, virtualizing any kind of control flow would require a new kind of build method that describes more of the structure of the AST. For example, consider a `for..in` loop: + +```swift +for person in employees { + "Hello, \(person.preferredName)" +} +``` + +Result builders currently will execute all iterations of the loop. Virtualizing the execution means passing along the means to execute the loop to the result builder, e.g., via a `buildVirtualFor` operation: + +```swift +Builder.buildVirtualFor(employees, { person in + let v0 = Builder.buildExpression("Hello, \(person.preferredName)") + return Builder.buildBlock(v0) +} +``` + +The builder's `buildVirtualFor` would have a signature such as: + +```swift +static func buildVirtualFor(_ sequence: S, @escaping (S.Element) -> T) -> ForEach { ... } +``` + +Such a facility could be used to map the language's `for..in` syntax to a lazily-evaluated construct like SwiftUI's [`ForEach`](https://developer.apple.com/documentation/swiftui/foreach). Similar builder methods would need to be developed for each supported syntax, e.g., `if` statements where the condition and then/else blocks are provided via closures. + +Virtualized ASTs would be a powerful extension to result builders. However, they will require an additional set of builder methods that match more closely with the syntax of the function being transformed. + +## Alternatives considered + +### Additional control-flow statements +The set of statements that are permitted within a transformed function are intentionally limited to those that are "strictly structural", and could reasonably be thought of as being part of a single, functional expression, aggregating values but without complicated control flow. However, one could go beyond this model to accept additional statements in a transformed function: + +* Local control flow statements that aren't “strictly structural”, like `break`, `continue`, and `do/catch`, could be handled by treating subsequent partial results as optional, as if they appeared within an `if`. +* Iteration statements other than `for`..`in` (i.e., `while` and `repeat`..`while`) could be supported via `buildArray`. + +Support for additional control-flow statements would weaken the declarative feel of result builders, and makes the "tree" structure of the DSL harder to reason about. + +It has been suggested that there could be two "forms" of result builders, one that matches the design in this proposal and a second, "simpler" one that handles the full breadth of the statement grammar (including all loops, `break`, `continue`, and so on) but sees only the partial results (e.g., via `buildExpression`) and not the structure (`buildBlock`, `buildEither(first:)`, etc. would not get called). The "simple result builder protocol" described above illustrates how one can get the second part of this--defining a simple result builder that receives all of the values without the structure--by building on top of this proposal. However, we should not have two forms of result builders in the language itself, with different capabilities, because it leads to confusion. If result builders gain support for additional control-flow statements (as a general feature), that should be reflected in the "simple result builder protocol" to extend the feature set for result builders that don't want the structure. + +### Builder-scoped name lookup +It is common for DSLs to want to introduce shorthands which might be unreasonable to introduce into the global scope. For example, `p` might be a reasonable name in the context of our `HTMLBuilder` DSL (rather than `paragraph`), but actually introducing a global function named `p` just for DSL use is quite unfortunate. Contextual lookups like `.p` will generally not work at the top level in DSLs because they will be interpreted as continuations of the previous statement. One could imagine having some way for the DSL to affect lexical lookup within transformed functions so that, e.g., within the transformed function one could use short names like `p`, `div`, and `h1`: + +```swift +return body { + let chapter = spellOutChapter ? "Chapter " : "" + div { + if useChapterTitles { + h1(chapter + "1. Loomings.") + } + p { + "Call me Ishmael. Some years ago" + } + p { + "There is now your insular city" + } + } +} +``` + +which are defined in the result builder type itself, e.g., + +```swift +extension HTMLBuilder { + static func body(@HTMLBuilder _ children: () -> [HTML]) -> HTMLNode { ... } + static func div(@HTMLBuilder _ children: () -> [HTML]) -> HTMLNode { ... } + static func p(@HTMLBuilder _ children: () -> [HTML]) -> HTMLNode { ... } + static func h1(_ text: String) -> HTMLNode { ... } +} +``` + +Name lookup that doesn't find the names `p`, `div`, or `h1` within the closure would look into the result builder being used to transform the closure before continuing lexical name lookup. + +Note that one can simulate this kind of effect by following the common pattern set out by SwiftUI, where the transformed function is usually within a type that conforms to some common protocol. For example, if we were to always say that HTML documents are types that conform to an `HTMLDocument` protocol, like this: + +```swift +protocol HTMLDocument { + @HTMLBuilder var body: HTML { get } +} + +struct MobyDick: HTMLDocument { + var body: HTML { + let chapter = spellOutChapter ? "Chapter " : "" + div { + if useChapterTitles { + h1(chapter + "1. Loomings.") + } + p { + "Call me Ishmael. Some years ago" + } + p { + "There is now your insular city" + } + } + } +} +``` + +Here, one can put the shorthand names (or, indeed, everything defined for the DSL) into an extension of the protocol: + +```swift +extension HTMLDocument { + static func body(@HTMLBuilder _ children: () -> [HTML]) -> HTMLNode { ... } + static func div(@HTMLBuilder _ children: () -> [HTML]) -> HTMLNode { ... } + static func p(@HTMLBuilder _ children: () -> [HTML]) -> HTMLNode { ... } + static func h1(_ text: String) -> HTMLNode { ... } +} +``` + +### Dropping Void/Never values + +During the first review, it was [suggested](https://forums.swift.org/t/se-0289-function-builders/39889/33) that a `Void` or `Never`-returning `buildExpression` method should cause the corresponding value to be dropped from the result builder itself. For example, the following result builder collects string values and puts them into a comma-separated string: + +```swift +@resultBuilder +struct StringConcatenator { + static func buildExpression(_ string: String) -> String { string } + + static func buildBlock(_ strings: String...) -> String { + strings.joined(separator: ", ") + } +} +``` + +However, it will reject any statements that don't produce a `String`. What if the builder wanted to allow other statements, but without collecting the values? One could re-work the builder like this: + +```swift +@resultBuilder +struct StringConcatenator { + static func buildExpression(_ string: String) -> String? { string } + static func buildExpression(_: T) -> String? { nil } + + static func buildBlock(_ strings: String?...) -> String { + strings.compactMap({$0}).joined(separator: ", ") + } +} +``` + +However, it's a non-obvious change to the result builder. The suggestion is to allow the original builder to be extended with a `Void`-producing `buildExpression`, e.g., + +```swift +static func buildExpression(_: T) { } +``` + +and have any `Void`-producing partial results be dropped. For example, applying +the result builder to this closure: + +```swift +func applyStringConcatenator(@StringConcatenator _: () -> String) { ... } + +applyStringConcatenator { + "hello" + 3.14159 + "world" +} +``` + +would produce "hello, world". While this is perhaps easier for the the author of the result builder, it makes the translation of the closure much less predictable. The basic translation of that closure would be to: + +```swift +applyStringConcatenator { + let a = StringConcatenator.buildExpression("hello") + let b = StringConcatenator.buildExpression(3.14159) + let c = StringConcatenator.buildExpression("world") + return StringConcatenator.buildBlock(a, b, c) +} +``` + +With this proposal, the formation of the `buildBlock` call would depend on the type inference for the partial result variables `a`, `b`, and `c`. In other words, because `b` will be inferred to type `Void`, the actual `buildBlock` call would end up being `StringConcatenator.buildBlock(a, c)`. This complication to the mental model outweighs the benefits to authors of result builders, because this feature isn't adding any expressive power--it's a shortcut to make it more convenient to address this case. diff --git a/proposals/0290-negative-availability.md b/proposals/0290-negative-availability.md new file mode 100644 index 0000000000..2d43fdcce4 --- /dev/null +++ b/proposals/0290-negative-availability.md @@ -0,0 +1,253 @@ +# Unavailability Condition + +* Proposal: [SE-0290](0290-negative-availability.md) +* Author: [Bruno Rocha](https://github.com/rockbruno) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Implementation: [apple/swift#33932](https://github.com/apple/swift/pull/33932) +* Status: **Implemented (Swift 5.6)** +* Previous revision: [1](https://github.com/swiftlang/swift-evolution/blob/066545c1cc9ff2b87ce233e0f8936f8d53724bdb/proposals/0290-negative-availability.md) +* Decision Notes: [Review #1](https://forums.swift.org/t/se-0290-unavailability-condition/41873/34), [Review #2](https://forums.swift.org/t/se-290-second-review-unavailability-condition/43544/59) + +## Introduction + +Swift historically supported the `#available` condition to check if a specific symbol **is** available for usage, but not the opposite. In this proposal, we'll present cases where checking for the **unavailability** of something is necessary, the ugly workaround needed to achieve it today and how a new `#unavailable` condition can fix it. + +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/support-negative-availability-literals/39946) + +## Motivation + +Checking whether a specific platform/symbol is **not** available is necessary when the changes made to an API are so extreme that you cannot represent it in a single if/else statement. The most common example is how building an iOS app's main `UIWindow` changed with the introduction of `SceneDelegates`. While a basic iOS 12 `AppDelegate` app sets up its window when the app finishes launching, apps that support `SceneDelegates` should instead do it *later* in the app's lifecycle -- more specifically, when the AppDelegate connects the app's main `UIScene`. Since this happens outside the usual `didFinishLaunching` flow, this extreme difference in behavior cannot be conveyed by a simple if/else availability check. Instead, the `AppDelegate` will require a *negative* platform check that makes it sure it only sets up the window if the user is **not** running iOS 13: + +```swift +// if NOT in iOS 13, load the window. +// Post iOS 13 the window is loaded later in the lifecycle, in the SceneDelegate. +if #available(iOS 13, *) { + +} else { + loadMainWindow() +} +``` + +### Readability + +As you might notice, the current way to achieve negative availability checks is by working around the current `#available` keyword. Because the availability condition is not parsed as an expression, it cannot be negated with regular boolean operations (`!`/`== false`). The way instead is to make use of the `else` clause, but as unavailability checks are not interested at all in the positive portion of the check, doing so will leave behind an empty `if` branch. For context, this problem does not exist in Objective-C as `if (@available(iOS 13.0, *) == NO)` is a valid expression. + +With the exception of this very specific case, an empty statement is a sure sign that there's something wrong in the developer's code, and it is likely that every unavailability check like this had to include a comment to indicate that it was done on purpose due to the compiler's limitations. In some cases, it might even be necessary to add an exclusion rule to the project's linters as most would assume that this is a mistake and incorrectly suggest that it can be fixed by negating the statement. + +This workaround has a clear negative impact on the readability of the unavailability check, as no one would expect an empty `if` statement to not be a coding mistake. Most developers will attempt to hide the problem by putting the positive portion in a single line: + +```swift +// if NOT in iOS 13, load the window. +// Post iOS 13 the window is loaded later in the lifecycle, in the SceneDelegate. +if #available(iOS 13, *) {} else { + loadMainWindow() +} +``` + +Unfortunately, this makes it easy to mistake it for a regular availability check. If the comment isn't clear that the statement is wrong on purpose, problems involving this check could easily land unnoticed in code-reviews. A less noisy way would be to put it under a `guard`: + +```swift +guard #available(iOS 13, *) else { + loadMainWindow() + return +} +// no-op +``` + +However, this goes against the code style recommendations involving the usage of `guard`. The guarded part should be the happy path, which is not the case when doing unavailability checks. As shown, it's currently impossible to write an unavailability check that properly fits a developer's engineering expectations and Swift's general style guide. + +Currently, any iOS application that supports UIScenes will face this issue and have to write this workaround. To describe it in a generic way, this issue will be encountered when dealing with any API addition or change that requires more than one availability condition to be implemented. + +### Usage of deprecated APIs + +A negative availability condition might also be necessary in cases where an API is marked as deprecated (and documented as non-functional) in newer OS versions. + +An example is the deprecation of the [`isAdvertisingTrackingEnabled`](https://developer.apple.com/documentation/adsupport/asidentifiermanager/1614148-isadvertisingtrackingenabled) property. Apps supporting iOS 14 must now use the new App Tracking Transparency framework for user tracking purposes, which involves displaying a permission alert for the user that explains why they are going to be tracked. A developer might determine that this large change in functionality might warrant a complete refactor of this feature, or simply conclude that the negative UX of displaying a new alert is not worth it and that they should remove this feature entirely. In any case however, the property still works when used with older iOS versions: + +```swift +// If NOT on iOS 14, track this action. +// Post iOS 14, we must ask for permission to do this. +// The UX impact is not worth it. Let's not do this at all. +if #available(iOS 14.0) { + +} else { + oldIos13TrackingLogic(isEnabled: ASIdentifierManager.shared().isAdvertisingTrackingEnabled) +} +``` + +In this specific case, a company that wants to adopt the new privacy practices will require unavailability checks to prevent breaking old versions of the app. In general, this will be the case when dealing with any API that is now deprecated. + +### Code Structure + +Besides cases where having an unavailability check is mandatory, supporting them would give developers more options when structuring their code in cases where they are not mandatory. By not being forced to consider the availability of something as the happy path, developers would have more choices when considering how to architect and abstract certain pieces of code. + +## Proposed solution + +Given that Objective-C is capable of negating availability checks, we believe that this not being supported in Swift was simply an oversight. We would like to propose this feature back to Swift in the shape of a new `#unavailable` condition: + +```swift +if #unavailable(iOS 13, *) { + loadMainWindow() +} +``` + +As dictated by the name, `#unavailable` is the reverse version of `#available`. Having a proper unavailability check will eliminate the need to use the current workaround and makes it clear to the reader that the statement is checking for the *lack* of a specific version, eliminating the need to provide a comment explaining what that piece of code is trying to achieve. + +Like with `#available`, `#unavailable` has the capacity to increase the symbol availability of a scope. As opposed to `#available`, the availability is increased at the `else` clause. + +```swift +if #unavailable(iOS 13, *) { + // Symbol Availability: Default (deployment target) +} else { + // Symbol Availability: iOS 13 +} +``` + +## Detailed design + +As the compiler is already able to calculate the symbol availability for both the positive and negative flows of the check, implementing `#unavailable` is simply a matter of introducing a new keyword that reverses them. This allows `#unavailable` to be internally implemented as a simple boolean that flips `#available's` functionality. + +Implementation: [apple/swift#33932](https://github.com/apple/swift/pull/33932) + +### Preventing impossible conditions + +The ability to use several availability checks in a single statement allows positive and negative availability checks to be mixed. This will lead to an ambiguous symbol availability: + +```swift +// User running something between iOS 9 and 12 +if #available(iOS 9.0, *), #unavailable(iOS 13.0, *) { + // Symbol Availability: iOS 9.0 +} else { + // Symbol Availability: ??? +} +``` + +The availability of the else block cannot be determined because it depends on which of the two conditions is false. To prevent this from happening, the compiler will emit a diagnostic whenever this happens. + +```swift +if #available(iOS 9.0, *), #unavailable(iOS 13.0, *) +// error: #available and #unavailable cannot be in the same statement +``` + +Technically we could support this by *not* improving the symbol availability of a scope if it's ambiguous, but as there are currently no legitimate cases where one would have to mix availability with unavailability, the author believes the work necessary to support this and its edge cases is not worth it at the time this proposal was written. However, you can still use them as separate statements. + +```swift +if #available(iOS 9.0, *) { + // Symbol Availability: iOS 9.0 + if #unavailable(iOS 13.0, *) { + // Symbol Availability: iOS 9.0 + } else { + // Symbol Availability: iOS 13.0 + } +} else { + // Symbol Availability: Default (deployment target) +} +``` + +As they are separate statements, there's no ambiguity. + +### Multiple Elses + +In the case of multiple else flows, the compiler will increase the symbol availability in **all** of them. + +```swift +if #unavailable(iOS 9.0, *) { + // Symbol Availability: Default (deployment target) +} else if a == b { + // Symbol Availability: iOS 9.0 +} else if b == c { + // Symbol Availability: iOS 9.0 +} else { + // Symbol Availability: iOS 9.0 +} +``` + +### Semantics of `*` for unavailability + +Availability statements are composed by a list of platform specs. The purpose of the list is to answer what's the platform version requirement necessary for the statement to return true in the platform where the code is being compiled. + +For this to work properly, the spec list must always contain an entry that matches the platform that's being compiled. An entry can be added to the list by either writing an explicit platform requirement (like `iOS 12`) or by using the generic platform wildcard `*`. Essentially, the wildcard signals that the expression being written is unrelated to the current platform, meaning that no version specification is needed. In practice, "no version specification needed" is done by treating the wildcard as the minimum deployment target of the platform, which essentially will cause the statement to never be false. + +```swift +// Example: Code is compiling in iOS + +error: condition required for target platform 'iOS' +if #available(macOS 11) { + ^ + +/// Examples of possible solutions: +if #available(macOS 11, *) +if #available(macOS 11, iOS 12) +if #available(iOS 12) +// In practice, the last two will not compile due to an additional requirement that is mentioned below. +``` + +Even if you have no plans to use your code in a different platform, your statement must still define the wildcard as a way to define what should happen in all unspecified current and potential new future platforms. The compiler uses the wildcard to ease porting to new platforms -- because the platform being compiled must always have an entry in the spec list, the wildcard allows these platforms to compile your code without requiring a modification to every availability guard in the program. Additionally, because new platforms typically branch from existing platforms, the wildcard also makes sure these ported checks will always return `true` by default when checking for availability. + +It's important to note that availability spec lists **are not boolean expressions**. For example, it's not possible to add multiple versions of a platform to the statement: + +```swift +if #available(iOS 12, *) +if #available(iOS 12, iOS 13, *) // Error: Version for 'iOS' already specified +``` + +Additionally, the specification of different platforms have no effect on the final result -- it depends *only* on the (unique) spec that matches the current platform being compiled. A check like `#available(iOS 12, watchOS 4, *)` compiled in iOS doesn't mean "return true if (iOS 12 **||** watchOS 4 **||** the current platform's minimum deployment target) is available", it simply means "return true if iOS 12 is available". The specs that refer to different platforms are ignored. + +Finally, the wildcard represents *only* the platforms that were not explicitly specified in the spec list. When `#available(iOS 13, *)` is compiled for iOS, the wildcard will be ignored in favor of the explicitly defined iOS 13 spec. As mentioned before, a platform can only be mentioned once. + +For unavailability, the semantics mentioned above means that `#unavailable(*)` and `#unavailable(notWhatImCompiling X, *)` should do the opposite of `#available` and return `false`. Since the minimum deployment target will always be present, the statement can never be true. This behavior is exactly how the current workaround works, and it also matches how the theoretical `!#available(*)` would behave. + +```swift +if #unavailable(*) { + // Will never be executed +} else { + // ... +} +``` + +As an interesting side effect, this means that having multiple unavailability checks in the same statement (`#unavailable(iOS 13, *), #unavailable(watchOS 3, *)` as opposed to `#unavailable(iOS 13, watchOS 3, *)`) would cause the statement to always be false if they are triggering the wildcard (in this case, because they cover different platforms). + +In these cases, since wildcard checks are eventually optimized to boolean literals, the compiler will already emit a warning indicating that the code will never be executed. Still, we could provide a more descriptive diagnostic that suggests using a single check that considers all platforms. + +```swift +if #unavailable(iOS 13, *), #unavailable(watchOS 3, *) { + // ... + // Warning: code will never be executed + // Warning: unavailability checks in this statement are canceling each other, use a single check that treats all platforms + // fix-it: #unavailable(iOS 13, watchOS 3, *) +} +``` + +### Result builders + +As `#unavailable` behaves exactly like `#available`, [`ViewBuilder`](https://developer.apple.com/documentation/swiftui/viewbuilder) does not need to be modified to support it. Using `#unavailable` on a builder will simply instead trigger [`buildLimitedAvailability(_:)`](0289-result-builders.md#availability) in the `else` block. + +## Source compatibility and ABI + +This change is purely additive. + +## Alternatives considered + +### `!#available(...)` and `#available(...) == false` + +While allowing the original condition to be reversed seems to be the obvious choice, supporting it in practice would require hardcoding all of this behavior as `#available` cannot be used as an expression. The author would rather not add tech debt to the compiler. + +Refactoring `#available` to be usable as an expression would likely require refactoring the entire symbol availability system and has an extensive amount of implications and edge cases. The work to support it would be considerably beyond what is proposed here. + +Supporting it by hardcoding this behavior is possible though, and could be implemented if the core team is willing and has a plan to eliminate the resulting tech debt in the future. + +On the other hand, given that it's fair to consider that this is a developer's first guess when attempting to do unavailability, the compiler will provide fix-its for each of these spellings. + +### `#unavailable(iOS 12)`, `#unavailable(iOS 12 && *)`, `#available(iOS < 12, *)` and other alternatives that involve reworking spec lists + +One point of discussion was the importance of the wildcard in the case of unavailability. Because the wildcard achieves nothing in terms of functionality, we considered alternatives that involved omitting or removing it completely. However, the wildcard is still important from a platform migration point of view, because although we don't need to force the guarded branch to be executed like in `#available`, the presence of the wildcard still play its intended role of allowing code involving unavailability statements to be ported to different platforms without requiring every single statement to be modified. + +Additionally, we had lengthy discussions about the *readability* of unavailability statements. We've noticed that even though availability in Swift has never been a boolean expression, it was clear that pretty much every developer's first instinct is to assume that `(iOS 12, *)` is equivalent to `iOS 12 || *`. The main point of discussion then was that the average developer might not understand why a call like `#unavailable(iOS 12, *)` will return `false` in non-iOS platforms, because they will assume the list means `iOS 12 || *` (`true`), while in reality (and as described in the `Semantics` section) the list means just `*` (`false`). During the pitch we tried to come up with alternatives that could eliminate this, and although some of them *did* succeed in doing that, they were doing so at the cost of making `#unavailable` "misleading", just like in the case of `!#available`. We ultimately decided that these improvements would be better suited for a *separate* proposal that focused on improving spec lists in general, which will be mentioned again at the end of this section. + +In general, there was much worry that this confusion could cause developers to misuse `#unavailable` and introduce bugs in their applications. We can prove that this feeling cannot happen in practice by how `#unavailable` doesn't introduce any new behavior -- it's nothing more than a reversed `#available` with a reversed literal name, which is semantically no different than the current workaround of using the `else` branch. Any confusing `#unavailable` scenario can also be conveyed as a confusing `#available` scenario by simply swapping the branches. + +Additionally, we were unable to locate concrete examples where this confusion could *actually* cause the feature to be misused. This is because if someone *does* misunderstand the branches, the project will simply fail to compile as there are symbols being used outside of an availability range. This was the case even when we tried to make the statements as vague as possible in an attempt to introduce a bug on purpose, and should hopefully make it clear for a confused developer that their code is simply upside-down. + +Although it's possible for developers to feel confused by the syntax, this is something that already exists with `#available` and cannot result in the feature being misused. As `#unavailable` does not introduces any new functionality that could change this, the author personally believes that this issue could be considered harmless and orthogonal to this proposal. + +However, we *do* have an unanimous agreement that spec lists can be confusing and that a new proposal should be created that re-evaluates how they are defined in code. Some members have also shared their belief that this re-evaluation should also happen *before* this proposal is introduced, which the author also agrees if there is a strong argument that `#unavailable` as is can be harmful for Swift. diff --git a/proposals/0291-package-collections.md b/proposals/0291-package-collections.md new file mode 100644 index 0000000000..19635a6616 --- /dev/null +++ b/proposals/0291-package-collections.md @@ -0,0 +1,342 @@ +# Package Collections + +* Proposal: [SE-0291](0291-package-collections.md) +* Authors: [Boris Bügling](https://github.com/neonichu), [Yim Lee](https://github.com/yim-lee), [Tom Doron](https://github.com/tomerd) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.5)** +* Implementation: [apple/swift-package-manager#3030](https://github.com/apple/swift-package-manager/pull/3030) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7c45e22557a0ef726dea9787f0fae9dac3ed7856/proposals/0291-package-collections.md), [2](https://github.com/swiftlang/swift-evolution/blob/3e56b936a2398b7bd57c09dc39a845336d2543fe/proposals/0291-package-collections.md) +* Review: [Review](https://forums.swift.org/t/se-0291-package-collections), [Review 2](https://forums.swift.org/t/se-0291-2nd-review-package-collections), [Acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0291-package-collections), [Amendment](https://forums.swift.org/t/amendment-se-0291-package-collection-signing/), [Amendment Acceptance](https://forums.swift.org/t/accepted-se-0291-amendment-package-collection-signing/45126) + +## Introduction + +This is a proposal for adding support for **Package Collections** to SwiftPM. A package collection is a curated list of packages and associated metadata which makes it easier to discover an existing package for a particular use case. SwiftPM will allow users to subscribe to these collections, search them via the `swift package-collection` command-line interface, and will make their contents accessible to any clients of libSwiftPM. This proposal is focused on the shape of the command-line interface and the format of configuration data related to package collections. + +We believe there are three different components in the space of package discovery with different purposes: + +**Package Registry** is focused on hosting and serving package sources as an alternative to fetching them directly from git. The goal is to provide better immutability, durability and potentially improve performance and security. This initiative is in-progress and governed by [a separate proposal](https://github.com/swiftlang/swift-evolution/pull/1179). + +**Package Index** is focused for providing a search index for packages. The goal is to improve discoverability of packages that may be hosted anywhere, and provide a rich set of metadata that helps making informed decisions when choosing dependencies. The Index indexes the package core metadata available in `Package.swift` as well as additional metadata from additional and external sources. An example of a package index is https://swiftpackageindex.com. + +**Package Collections** which are the subject of this proposal are closer to the Package Index than to the Package Registry. Collections are also designed to make discovery easier, but focused on simpler curation lists that can be easily shared rather than on larger scale indexing and ranking system that requires infrastructure. This design is the first step in teaching SwiftPM about discovery of packages and future work to support Package Indexes can build on this initial design. + + +## Motivation + +Currently, it can be difficult to discover packages that fit particular use cases. There is also no standardized way of accessing metadata about a package which is not part of the package manifest itself. We envision educators and community influencers publishing package collections to go along with course materials or blog posts, removing the friction of using packages for the first time and the cognitive overload of deciding which packages are useful for a particular task. We also envision enterprises using collections to narrow the decision space for their internal engineering teams, focusing them on a trusted set of vetted packages. + +Exposing the data of package collections via libSwiftPM and the `swift package-collection` command-line interface will also allow other tools to leverage this information and provide a richer experience for package discovery that is configured by the user in one central place. + + +## Proposed solution + +We propose to introduce a new concept called **Package Collections** to the Swift package ecosystem. Collections are authored as static JSON documents and contain a list of packages and additional metadata per package. They are published to a web server or CDN-like infrastructure making them accessible to users. SwiftPM will gain new command-line interface for adding and removing collections and will index them in the background, allowing users to more easily discover and integrate packages that are included in the collections. + +For example, a course instructor knows they intend to teach with a set of several packages for their class. They can construct a collection JSON file, representing those packages. Then, they can post that JSON file to a GitHub repo or a website, giving the URL to that JSON file to all their students. Students use SwiftPM to add the instructor's collection to their SwiftPM configuration, and any packages the instructor puts into that collection can be easily used by the students. + + +## Detailed design + +We propose to add a new sets of commands under a `swift package-collection` command-line interface that support the following workflows: + +1. Managing collections +2. Querying metadata for individual packages +3. Searching for packages and modules across collections + +We also propose adding a new per-user SwiftPM configuration file which will initially store the list of collections a user has configured, but can later be used for other per-user configuration for SwiftPM. + +### Example + +A course instructor shares a collection with packages needed for some assignments. The participants can add it to their set of collections: + +``` +$ swift package-collection add https://www.example.com/packages.json +Added "Packages for course XYZ" to your package collections. +``` + +This will add the given collection to the user's set of collections for querying metadata and search. + +One of the assignments requires parsing a YAML file and instead of searching the web, participants can search the curated collection for packages that could help with their task: + +``` +$ swift package-collection search --keywords yaml +https://github.com/jpsim/yams: A sweet and swifty YAML parser built on LibYAML. +... +``` + +This will perform a string-based search across various metadata fields of all packages, such as the description and name. Results will contain URL and description (if any) of each match. + +Once a suitable package has been identified, there will also be the ability to query for more metadata, such as available versions, which will be required to actually depend on the package in practice. + +``` +$ swift package-collection describe https://github.com/jpsim/yams +Description: A sweet and swifty YAML parser built on LibYAML. +Available Versions: 4.0.0, 3.0.0, ... +Watchers: 14 +Readme: https://github.com/jpsim/Yams/blob/master/README.md +Authors: @norio-nomura, @jpsim +-------------------------------------------------------------- +Latest Version: 4.0.0 +Package Name: Yams +Modules: Yams, CYaml +Supported Platforms: iOS, macOS, Linux, tvOS, watchOS +Supported Swift Versions: 5.3, 5.2, 5.1, 5.0 +License: MIT +CVEs: ... +``` + +This will list the basic metadata for the given package, as well as more detailed metadata for the latest version of the package. Available metadata for a package can incorporate data from the collection itself, as well as data discovered by SwiftPM. For example, by querying the package's repository or gathering data from the source code hosting platform being used by the package. + + + +### Manage Package Collections + +#### List + +The `list` command lists all collections that are configured by the user. The result can optionally be returned as JSON for integration into other tools. + +``` +$ swift package-collection list [--json] +My organisation's packages - https://example.com/packages.json +... +``` + + +#### Manual refresh + +The `refresh` command refreshes any cached data manually. SwiftPM will also automatically refresh data under various conditions, but some queries such as search will rely on locally cached data. + +``` +$ swift package-collection refresh +Refreshed 23 configured package collections. +``` + + +#### Add + +The `add` command adds a collection by URL, with an optional order hint, to the user's list of configured collections. The order hint will influence ranking in search results and can also potentially be used by clients of SwiftPM to order results in a UI, for example. + + +``` +$ swift package-collection add https://www.example.com/packages.json [--order N] +Added "My organisation's packages" to your package collections. +``` + + +#### Remove + +The `remove` command removes a collection by URL from the user's list of configured collections. + +``` +$ swift package-collection remove https://www.example.com/packages.json +Removed "My organisation's packages" from your package collections. +``` + + +#### Metadata and packages of a single collection + +The `describe` command shows the metadata and included packages for a single collection. This can be used for both collections that have been previously added to the list of the user's configured collections, as well as to preview any other collections. + +``` +$ swift package-collection describe https://www.example.com/packages.json +Name: My organisation's packages +Source: https://www.example.com/packages.json +Description: ... +Keywords: best, packages +Created At: 2020-05-30 12:33 +Packages: + https://github.com/jpsim/yams + ... +``` + + + +### Get metadata for a single package + +Note: Collections will be limited in the number of major and minor versions they store per package. For each major/minor combination that is being stored, only data for the latest patch version will be present. + +#### Metadata for the package itself + +The `describe` command shows the metadata from the package itself. The result can optionally be returned as JSON for integration into other tools. + +``` +$ swift package-collection describe [--json] https://github.com/jpsim/yams +Description: A sweet and swifty YAML parser built on LibYAML. +Available Versions: 4.0.0, 3.0.0, ... +Watchers: 14 +Readme: https://github.com/jpsim/Yams/blob/master/README.md +Authors: @norio-nomura, @jpsim +-------------------------------------------------------------- +Latest Version: 4.0.0 +Package Name: Yams +Modules: Yams, CYaml +Supported Platforms: iOS, macOS, Linux, tvOS, watchOS +Supported Swift Versions: 5.3, 5.2, 5.1, 5.0 +License: MIT +CVEs: ... +``` + + +#### Metadata for a package version + +When passing an additional `--version` parameter, the `describe` command shows the metadata for a single package version. The result can optionally be returned as JSON for integration into other tools. + +``` +$ swift package-collection describe [--json] --version 4.0.0 https://github.com/jpsim/yams +Package Name: Yams +Version: 4.0.0 +Modules: Yams, CYaml +Supported Platforms: iOS, macOS, Linux, tvOS, watchOS +Supported Swift Versions: 5.3, 5.2, 5.1, 5.0 +License: MIT +CVEs: ... +``` + +Note: since the `describe` action is shared between showing metadata for both whole collections as well as individual packages, it will first check if the given URL matches a known package and otherwise will treat the argument as a collection URL. If the `--version` parameter is passed, the collection fallback will not be done since the user already explicitly requested information about a package. + + + +### Search + +#### String-based search + +The search command does a string-based search when using the `--keyword` option and returns the list of packages that match the query. The result can optionally be returned as JSON for integration into other tools. + +``` +$ swift package-collection search [--json] --keywords yaml +https://github.com/jpsim/yams: A sweet and swifty YAML parser built on LibYAML. +... +``` + + +#### Module-based search + +The search command does a search for a specific module name when using the `--module` option. The result can optionally be returned as JSON for integration into other tools. Lists the newest version the matching module can be found in. This will display more metadata per package than the string-based search as we expect just one or very few results for packages with a particular module name. + +``` +$ swift package-collection search [--json] --module yams +Package Name: Yams +Latest Version: 4.0.0 +Description: A sweet and swifty YAML parser built on LibYAML. +-------------------------------------------------------------- +... +``` + + + +### Configuration file + +The global configuration file will be expected at this location: + + +``` +~/.swiftpm/config +``` + + +This file will be stored in a `.swiftpm` directory in the user's home directory (or its equivalent on the specific platform SwiftPM is running on). + +This file will be managed through SwiftPM commands and users are not expected to edit it by hand. The format of this file is an implementation detail but it will be human readable format, likely JSON in practice. + +There could be a supplemental file providing key-value pairs whose keys can be referenced by the main configuration file. This can be used as an override mechanism that allows sharing the main configuration file between different users and machines by keeping user-specific configuration information out of the main configuration file. The use of this additional file will be optional and it will be managed by the user. The syntax of the format will be based on git's configuration files, described [here](https://git-scm.com/docs/git-config#_syntax), but it will not support all of its semantics. + + +### Data format + +Package collections must adhere to a specific JSON format for SwiftPM to be able to consume them. The current proposed JSON format can be found [here](https://github.com/apple/swift-package-collection-generator/blob/main/PackageCollectionFormats/v1.md). It is not part of this proposal because it is not considered stable API. Over time as the data format matures, we will consider making it stable API in a separate proposal. + +Since the data format is unstable, users should avoid generating package collections on their own. This proposal includes providing the necessary [tooling](https://github.com/apple/swift-package-collection-generator) for package collection generations. + +### Package collection signing + +Package collections can be signed to establish authenticity and protect their integrity. Doing this is optional and users will not be blocked from adding unsigned package collections. + +There will be [tooling](https://github.com/apple/swift-package-collection-generator) to help publishers sign their package collections. To generate a signature one must provide: +- The package collection file to be signed +- A code signing certificate +- The certificate's private key +- The certificate's chain in its entirety + +The signature will include the certificate's public key and chain so that they can be used for verification later. + +A signed package collection will have an extra `signature` object: + +``` +{ + // Package collection JSON + ..., + "signature": { + ... + } +} +``` + +#### Requirements on signing certificate + +The following conditions are checked and enforced during signature generation and verification: +- The timestamp at which signing/verification is done must fall within the signing certificate's validity period. +- The certificate's "Extended Key Usage" extension must include "Code Signing". +- The certificate must use either 256-bit EC (recommended) or 2048-bit RSA key. +- The certificate must not be revoked. The certificate authority must support OCSP, which means the certificate must have the "Certificate Authority Information Access" extension that includes OCSP as a method, specifying the responder's URL. +- The certificate chain is valid and root certificate must be trusted. + +##### Trusted root certificates + +On Apple platforms, all root certificates that come preinstalled with the OS are automatically trusted. Users may specify additional certificates to trust by placing them in the `~/.swiftpm/config/trust-root-certs` directory. + +On non-Apple platforms, there are no trusted root certificates by default. Only those found in `~/.swiftpm/config/trust-root-certs` are trusted. + +#### Add a signed package collection + +When adding a signed package collection, SwiftPM will check that: +- The file content (excluding `signature`) is what was used to generate the signature. In other words, this checks to see if the collection has been altered since it was signed. +- The signing certificate meets all of the [requirements](#requirements-on-signing-certificate). + +SwiftPM will not import a collection if any of these checks fails. + +User may opt to skip the signature check on a collection by passing the `--skip-signature-check` flag during `add`: + +```bash +$ swift package-collection add https://www.example.com/packages.json --skip-signature-check +``` + +Since there are no trusted root certificates by default on non-Apple platforms, the signature check will always fail. SwiftPM will detect this and instruct user to either set up the `~/.swiftpm/config/trust-root-certs` directory or use `--skip-signature-check`. + +#### Add an unsigned package collection + +When adding an unsigned package collection, user must confirm their trust by passing the `--trust-unsigned` flag: + +```bash +$ swift package-collection add https://www.example.com/unsigned-packages.json --trust-unsigned +``` + +The `--skip-signature-check` flag has no effects on unsigned collections. + +#### Security risks + +Signed package collections as currently designed are susceptible to the following attack vectors: + +- **Signature stripping**: This involves attackers removing signature from a signed collection, causing it to be downloaded as an unsigned collection and bypassing signature check. In this case, publishers should make it known that the collection is signed, and users should abort the `add` operation when the "unsigned" warning shows up on a supposedly signed collection. +- **Signature replacement**: Attackers may modify a collection then re-sign it using a different certificate, and SwiftPM will accept it as long as the signature is valid. + +To defend against these attacks, SwiftPM will offer a way for collection publishers to: +1. Require signature check on their collections - this defends against "signature stripping". +2. Restrict what certificate can be used for signing - this defends against "signature replacement". + +The process will involve submitting a pull request to modify SwiftPM certificate pinning configuration. + +## Future direction + +This proposal shows an initial set of metadata that package collections will offer, but the exact metadata provided can be evolved over time as needed. The global configuration files introduced here can be used for future features which require storing per-user configuration data. + +This design is the first step in teaching SwiftPM about sets of curated packages, with the goal of allowing the community to build trusted working-sets of packages. Future work to support more dynamic server-based indexes can build on this initial design. + + +## Impact on existing packages + +There is no impact on existing packages as this is a discovery feature that's being added to SwiftPM's command-line interface. + + +## Alternatives considered + +The initial pitch considered adding the new CLI functionality under the existing `swift package` command, but that caused deeper nesting of commands and also did not fit with the existing functionality under this command. diff --git a/proposals/0292-package-registry-service.md b/proposals/0292-package-registry-service.md new file mode 100644 index 0000000000..a981a2b4cf --- /dev/null +++ b/proposals/0292-package-registry-service.md @@ -0,0 +1,1267 @@ +# Package Registry Service + +* Proposal: [SE-0292](0292-package-registry-service.md) +* Authors: [Bryan Clark](https://github.com/clarkbw), + [Whitney Imura](https://github.com/whitneyimura), + [Mattt Zmuda](https://github.com/mattt) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift-package-manager#3023](https://github.com/apple/swift-package-manager/pull/3023) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0292-package-registry-service/49849) +* Review: + [1](https://forums.swift.org/t/se-0292-package-registry-service/) + [2](https://forums.swift.org/t/se-0292-2nd-review-package-registry-service/) + [3](https://forums.swift.org/t/se-0292-3rd-review-package-registry-service/) + [Amendment](https://forums.swift.org/t/amendment-se-0292-package-registry-service/) +* Previous Revision: + [1](https://github.com/swiftlang/swift-evolution/blob/b48527526b5748a60b0b23846d5880e9cc2c4711/proposals/0292-package-registry-service.md) + [2](https://github.com/swiftlang/swift-evolution/blob/53bd6d3813c40ebd07701727c8cfb6fedd751e2a/proposals/0292-package-registry-service.md) + [3](https://github.com/swiftlang/swift-evolution/blob/971d1f43bce718a45227432782a312cc5de99870/proposals/0292-package-registry-service.md) + +## Introduction + +Swift Package Manager downloads dependencies using Git. +Our proposal defines a standard web service interface +that it can also use to download dependencies from a package registry. + +Swift-evolution thread: +[Swift Package Registry Service](https://forums.swift.org/t/swift-package-registry-service/37219) + +## Motivation + +A package dependency is currently specified by a URL to its source repository. +When Swift Package Manager builds a project for the first time, +it clones the Git repository for each dependency +and attempts to resolve the version requirements from the available tags. + +Although Git is a capable version-control system, +it's not well-suited to this kind of workflow for the following reasons: + +* **Reproducibility**: + A version tag in the Git repository for a dependency + can be reassigned to another commit at any time. + This can cause a project to produce different build results + depending on when it was built. +* **Availability**: + The Git repository for a dependency can be moved or deleted, + which can cause subsequent builds to fail. +* **Efficiency**: + Cloning the Git repository for a dependency + downloads all versions of a package when only one is used at a time. +* **Speed**: + Cloning a Git repository for a dependency can be slow + if it has a large history. + Also, cloning a Git repository is expensive for both the server and client, + and may be significantly slower than downloading the same content + using HTTP through a [content delivery network (CDN)][CDN]. + +Many language ecosystems have a *package registry*, including +[RubyGems] for Ruby, +[PyPI] for Python, +[npm] for JavaScript, and +[crates.io] for Rust. +In fact, +many Swift developers build apps today using +[CocoaPods] and its index of libraries. + +A package registry for Swift Package Manager +could offer faster and more reliable dependency resolution +than downloading dependencies using Git. +It could also support other useful functionality, +including package search, security audits, and local offline caches. + +## Proposed solution + +This proposal defines a standard interface for package registry services +and describes how Swift Package Manager integrates with them +to download dependencies. + +A user may [configure](#registry-configuration-subcommands) +a package registry for their project +by specifying a URL to a [conforming web service](#package-registry-service-1). +When a registry is configured, +Swift Package Manager resolves external dependencies +in the project's package manifest (`Package.swift`) file +that are [declared](#new-packagedescription-api) +with a [scoped package identifier](#package-identity) in the form +`scope.package-name`. +These package identifiers resolve potential +[name collisions](#package-name-collision-resolution) +across build targets. + +For each external dependency declared in the package manifest, +Swift Package Manager first sends a +`GET` request to `/{scope}/{name}` +to fetch a list of available releases +from the configured registry. +If a release is found that satisfies the declared version requirement +(for example, `.upToNextMinor(from: "1.1.0")`), +Swift Package Manager sends a +`GET` request to `/{scope}/{name}/{version}/Package.swift` +to fetch the manifest for that release. +This process continues with the package manifests of each dependency, +each of their respective dependencies, +and so on. +Once the dependency graph is [resolved](#dependency-graph-resolution), +Swift Package Manager downloads the +[source archive](#archive-source-subcommand) for each dependency +by sending a `GET` request to `/{scope}/{name}/{version}.zip`. + +## Detailed design + +### Package registry service + +A package registry service implements the following REST API endpoints +for listing releases for a package, +fetching information about a release, +and downloading the source archive for a release: + +| Method | Path | Description | +| ------ | --------------------------------------------------------- | ----------------------------------------------- | +| `GET` | `/{scope}/{name}` | List package releases | +| `GET` | `/{scope}/{name}/{version}` | Fetch metadata for a package release | +| `GET` | `/{scope}/{name}/{version}/Package.swift{?swift-version}` | Fetch manifest for a package release | +| `GET` | `/{scope}/{name}/{version}.zip` | Download source archive for a package release | +| `GET` | `/identifiers{?url}` | Lookup package identifiers registered for a URL | + +A formal specification for the package registry interface is provided +[alongside this proposal](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md). +In addition, +an OpenAPI (v3) document +and a reference implementation written in Swift +are provided for the convenience of developers interested +in building their own package registry. + +### Changes to Swift Package Manager + +#### Package identity + +Currently, the identity of a package is computed from +the last path component of its effective URL +(which can be changed with dependency mirroring). +However, this approach can lead to a conflation of +distinct packages with similar names +and the duplication of the same package under different names. + +We propose using a scoped identifier +in the form `scope.package-name` +to identify package dependencies. + +A *scope* provides a namespace for related packages within a package registry. +A package scope consists of +alphanumeric characters and hyphens. +Hyphens may not occur at the beginning or end, +nor consecutively within a scope. +The maximum length of a package scope is 39 characters. +A valid package scope matches the following regular expression pattern: + +```regexp +\A[a-zA-Z\d](?:[a-zA-Z\d]|-(?=[a-zA-Z\d])){0,38}\z +``` + +A package's *name* uniquely identifies a package in a scope. +A package name consists of alphanumeric characters, underscores, and hyphens. +Hyphens and underscores may not occur at the beginning or end, +nor consecutively within a name. +The maximum length of a package name is 100 characters. +A valid package scope matches the following regular expression pattern: + +```regexp +\A[a-zA-Z0-9](?:[a-zA-Z0-9]|[-_](?=[a-zA-Z0-9])){0,99}\z +``` + +Package scopes and names are compared using locale-independent case folding. + +#### New `PackageDescription` API + +The `Package.Dependency` type adds the following static method: + +```swift +extension Package.Dependency { + /// Adds a dependency on a package with the specified identifier + /// that uses the provided version requirement. + public static func package( + id: String, + _ requirement: Package.Dependency.VersionBasedRequirement + ) -> Package.Dependency +} +``` + +These methods may be called in the `dependencies` field of a package manifest +to declare one or more dependencies by their respective package identifier. + +```swift +dependencies: [ + .package(id: "mona.LinkedList", .upToNextMinor(from: "1.1.0")), + .package(id: "mona.RegEx", .exact("2.0.0")) +] +``` + +A package dependency declared with an identifier using this method +may only specify a version-based requirement. +`Package.Dependency.VersionBasedRequirement` is a new type +that provides the same interface as `Package.Dependency.Requirement` +for version-based requirements, +but excluding branch-based and commit-based requirements. + +#### Package name collision resolution + +Consider a dependency graph that includes both +a package declared with the identifier `mona.LinkedList` and +an equivalent package declared with the URL `https://github.com/mona/LinkedList`. + +When Swift Package Manager fetches a list of releases for the identified package +(`GET /mona/LinkedList`), +the response includes a `Link` header field +with URLs to that project's source repository +that are known to the registry. + +```http +Link: ; rel="canonical", + ; rel="alternate" +``` + +Swift Package Manager uses this information +to reconcile the URL-based dependency declaration with +the package identifier `mona.LinkedList`. +Link relation URLs may also be normalized to mitigate insignificant variations. +For example, +a package with an ["scp-style" URL][scp-url] like +`git@github.com:mona/LinkedList.git` +is determined to be equivalent to a URL with an explicit scheme like +`ssh:///git@github.com/mona/LinkedList`. +Swift Package Manager may additionally consult the registry +to associate a URL-based package declaration with a package identifier +by sending a `GET /identifiers{?url}` request with that package's URL. + +A package identifier serves as the package name +in target-based dependency declarations — +that is, the `package` parameter in `.product(name:package)` method calls. + +```diff + targets: [ + .target(name: "MyLibrary", + dependencies: [ + .product(name: "LinkedList", +- package: "LinkedList") ++ package: "mona.LinkedList") + ] + ] +``` + +Any path-based dependency declaration +or URL-based declaration without an associated package identifier +will continue to synthesize its identity from +the last path component of its location. + +#### Dependency graph resolution + +In its `PackageGraph` module, Swift Package Manager defines +the `PackageContainer` protocol as the top-level unit of package resolution. +Conforming types are responsible for +determining the available tags for a package +and its contents at a particular revision. +A `PackageContainerProvider` protocol adds a level of indirection +for resolving package containers. + +There are currently two concrete implementations of `PackageContainer`: +`LocalPackageContainer` and `RepositoryPackageContainer`. +This proposal adds a new `RegistryPackageContainer` type +that adopts `PackageContainer` +and performs equivalent operations with HTTP requests to a registry service. +These client-server interactions are facilitated by +a new `RegistryManager` type. +When requesting resources from a registry, +Swift Package Manager will employ techniques like +exponential backoff, circuit breakers, and client-side validation +to safeguard against adverse network conditions and malicious server responses. + +The following table lists the +tasks performed by Swift Package Manager during dependency resolution +alongside the Git operations used +and their corresponding package registry API calls. + +| Task | Git operation | Registry request | +| ------------------------------------- | --------------------------- | --------------------------------------------- | +| Fetch the contents of a package | `git clone && git checkout` | `GET /{scope}/{name}/{version}.zip` | +| List the available tags for a package | `git tag` | `GET /{scope}/{name}` | +| Fetch a package manifest | `git clone` | `GET /{scope}/{name}/{version}/Package.swift` | + +Package registries support +[version-specific _manifest_ selection][version-specific-manifest-selection] +by providing a list of versioned manifest files for a package +(for example, `Package@swift-5.3.swift`) +in its response to `GET /{scope}/{name}/{version}/Package.swift`. +However, package registries don't support +[version-specific _tag_ selection][version-specific-tag-selection]. + +### Changes to `Package.resolved` + +Swift package registry releases are archived as Zip files. + +When an external package dependency is downloaded through a registry, +Swift Package Manager compares the integrity checksum provided by the server +against any existing checksum for that release in the `Package.resolved` file +as well as the integrity checksum reported by the `compute-checksum` subcommand: + +```console +$ swift package compute-checksum LinkedList-1.2.0.zip +1feec3d8d144814e99e694cd1d785928878d8d6892c4e59d12569e179252c535 +``` + +If no prior checksum exists, +it's saved to `Package.resolved`. + +```json +{ + "object": { + "pins": [ + { + "package": "mona.LinkedList", + "state": { + "checksum": "ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d", + "version": "1.2.0" + } + } + ] + }, + "version": 1 +} +``` + +Suppose the checksum reported by the server +is different from the existing checksum +(or the checksum of the downloaded artifact is different from either of them). +In that case, +a package's contents may have changed at some point. +Swift Package Manager will refuse to download dependencies +if there's a mismatch in integrity checksums. + +```console +$ swift build +error: checksum of downloaded source archive of dependency 'mona.LinkedList' (c2b934fe66e55747d912f1cfd03150883c4f037370c40ca2ad4203805db79457) does not match checksum specified by the manifest (ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d) +``` + +Once the correct checksum is determined, +the user can update `Package.resolved` with the correct value +and try again. + +### Archive-source subcommand + +An anecdotal look at other package managers suggests that +a checksum mismatch is more likely to be a +disagreement in how to create the archive and/or calculate the checksum +than, say, a forged or corrupted package. + +This proposal adds a new `swift package archive-source` subcommand +to provide a standard way to create source archives for package releases. + +```manpage +SYNOPSIS + swift package archive-source [--output=] + +OPTIONS + -o , --output= + Write the archive to . + If unspecified, the package is written to `\(PackageName).zip`. +``` + +Run the `swift package archive-source` subcommand +in the root directory of a package +to generate a source archive for the current working tree. +For example: + +```console +$ tree -a -L 1 +LinkedList +├── .git +├── Package.swift +├── README.md +├── Sources +└── Tests + +$ head -n 5 Package.swift +// swift-tools-version:5.3 +import PackageDescription + +let package = Package( +name: "LinkedList", + +$ swift package archive-source +Created LinkedList.zip +``` + +By default, +generated archive's filename is +the name of the package with a `.zip` extension +(for example, "LinkedList.zip"). +You can override this behavior with the `--output` option: + +```console +$ git checkout 1.2.0 +$ swift package archive-source --output="LinkedList-1.2.0.zip" +# Created LinkedList-1.2.0.zip +``` + +The `archive-source` subcommand has the equivalent behavior of +[`git-archive(1)`] using the `zip` format at its default compression level, +with entries prefixed by the basename of the generated archive's filename. +Therefore, the following command produces +equivalent output to the previous example: + +```console +$ git archive --format zip \ + --prefix LinkedList-1.2.0 + --output LinkedList-1.2.0.zip \ + 1.2.0 +``` + +If desired, this behavior could be changed in future tool versions. + +> **Note**: +> `git-archive` ignores files with the `export-ignore` Git attribute. +> By default, this ignores hidden files and directories, +> including`.git` and `.build`. + +### Registry configuration subcommands + +This proposal adds a new `swift package-registry` subcommand +for managing the registry used for all packages +and/or packages in a particular scope. + +Custom registries can serve a variety of purposes: + +- **Private dependencies**: + Users may configure a custom registry for a particular scope + to incorporate private packages with those fetched from a public registry. +- **Geographic colocation**: + Developers working under adverse networking conditions can + host a mirror of official package sources on a nearby network. +- **Policy enforcement**: + A corporate network can enforce quality or licensing standards, + so that only approved packages are available through a custom registry. +- **Auditing**: + A custom registry may analyze or meter access to packages + for the purposes of ranking popularity or charging licensing fees. + +#### Setting a custom registry + +```manpage +SYNOPSIS + swift package-registry set [options] +OPTIONS: + --global Apply settings to all projects for this user + --scope Associate the registry with a given scope + --login Specify a user name for the remote machine + --password Supply a password for the remote machine +``` + +Running the `package-registry set` subcommand +in the root directory of a package +creates or updates the `.swiftpm/configuration/registries.json` file +with a new top-level `registries` key +that's associated with an object containing the specified registry URLs. +The default, unscoped registry is associated with the key `[default]`. +Any scoped registries are keyed by their case-folded name. + +For example, +a build server that doesn't allow external network connections +may configure a registry URL to resolve dependencies +using an internal registry service. + +```console +$ swift package-registry set https://internal.example.com/ +$ cat .swiftpm/configuration/registries.json +``` + +```json +{ + "registries": { + "[default]": { + "url": "https://internal.example.com" + } + }, + "version": 1 +} + +``` + +If no registry is configured, +Swift Package Manager commands like +`swift package resolve` and `swift package update` +fail with an error. + +```console +$ swift package resolve +error: cannot resolve dependency 'mona.LinkedList' without a configured registry +``` + +#### Associating a registry with a scope + +The user can associate a package scope with a custom registry +by passing the `--scope` option. + +For example, +a user might resolve all packages with the package scope `example` +(such as `example.PriorityQueue`) +to a private registry. + +```console +$ swift package-registry set https://internal.example.com/ --scope example +$ cat .swiftpm/configuration/registries.json +``` + +```json +{ + "registries": { + "example": { + "url": "https://internal.example.com" + } + }, + "version": 1 +} + +``` + +When a custom registry is associated with a package scope, +package dependencies with that scope are resolved through the provided URL. +A custom registry may be associated with one or more scopes, +but a scope may be associated with only a single registry at a time. +Scoped custom registries override any unscoped custom registry. + +#### Unsetting a custom registry + +This proposal also adds a new `swift package-registry unset` subcommand +to complement the `package-registry set` subcommand. + +```manpage +SYNOPSIS + swift package-registry unset [options] +OPTIONS: + --global Apply settings to all projects for this user + --scope Removes the registry's association to a given scope +``` + +Running the `package-registry unset` subcommand +in the root directory of a package +updates the `.swiftpm/configuration/registries.json` file +to remove the `default` entry in the top-level `registries` key, if present. +If a `--scope` option is passed, +only the entry for the specified scope is removed, if present. + +#### Global registry configuration + +The user can pass the `--global` option to the `set` or `unset` subcommands +to update the user-level configuration file located at +`~/.swiftpm/configuration/registries.json`. + +Any default or scoped registries configured locally in a project directory +override any values configured globally for the user. +For example, +consider the following global and local registry configuration files: + +```jsonc +// Global configuration (~/.swiftpm/configuration/registries.json) +{ + "registries": { + "[default]": { + "url": "https://global.example.com" + }, + "foo": { + "url": "https://global.example.com" + }, + }, + "version": 1 +} + +// Local configuration (.swiftpm/configuration/registries.json) +{ + "registries": { + "foo": { + "url": "https://local.example.com" + } + }, + "version": 1 +} + +``` + +Running the `swift package resolve` command with these configuration files +resolves packages with the `foo` scope +using the registry located at "https://local.example.com", +and all other packages +using the registry located at "https://global.example.com". + +In summary, +the behavior of `swift package resolve` and related commands +depends on the following factors, +in descending order of precedence: + +* The package manifest in the current directory (`./Package.swift`) +* Any existing lock file (`./Package.resolved`) +* Any local configuration (`./.swiftpm/configuration/registries.json`) +* Any global configuration file (`~/.swiftpm/configuration/registries.json`) + +#### Specifying credentials for a custom registry + +Some servers may require a username and password. +The user can provide credentials when setting a custom registry +by passing the `--login` and `--password` options. + +When credentials are provided, +the corresponding object in the `registries.json` file +includes a `login` key with the passed value. +If the project's `.netrc` file has an existing entry +for a given machine and login, +it's updated with the new password; +otherwise, a new entry is added. +If no `.netrc` file exists, +a new one is created and populated with the new entry. + +```console +$ swift package-registry set https://internal.example.com/ \ + --login jappleseed --password alpine + +$ cat .netrc +machine internal.example.com +login jappleseed +password alpine + +$ cat .swiftpm/configuration/registries.json + +{ + "registries": { + "[default]": { + "url": "https://internal.example.com" + "login": "jappleseed" + } + }, + "version": 1 +} +``` + +If the user passes the `--login` and `--password` options +to the `set` subcommand along with the `--global` option, +the user-level `.netrc` file is updated instead. +When Swift Package Manager connects to a custom registry, +it first consults the project's `.netrc` file, if one exists. +If no entry is found for the custom registry, +Swift Package Manager then consults the user-level `.netrc` file, if one exists. + +If the provided credentials are missing or invalid, +Swift Package Manager commands like +`swift package resolve` and `swift package update` +fail with an error. + +### Changes to config subcommand + +#### Set-mirror option for package identifiers + +A user can currently specify an alternate location for a package +by setting a [dependency mirror][SE-0219] for that package's URL. + +```console +$ swift package config set-mirror \ + --original-url https:///github.com/mona/linkedlist \ + --mirror-url https:///github.com/octocorp/swiftlinkedlist +``` + +This proposal updates the `swift package config set-mirror` subcommand +to accept a `--package-identifier` option in place of an `--original-url`. +Running this subcommand with a `--package-identifier` option +creates or updates the `.swiftpm/configuration/mirrors.json` file, +modifying the array associated with the top-level `object` key +to add a new entry or update an existing entry +for the specified package identifier, +that assigns its alternate location. + +```json +{ + "object": [ + { + "mirror": "https://github.com/OctoCorp/SwiftLinkedList.git", + "original": "mona.LinkedList" + } + ], + "version": 1 +} + +``` + +When a mirror URL is set for a package identifier, +Swift Package Manager resolves any dependencies with that identifier +through Git using the provided URL. + +## Security + +Adding external dependencies to a project +increases the attack surface area of your software. +However, much of the associated risk can be mitigated, +and a package registry can offer stronger guarantees for safety and security +compared to downloading dependencies using Git. + +Core security measures, +such as the use of HTTPS and integrity checksums, +are required by the registry service specification. +Additional decisions about security +are delegated to the registries themselves. +For example, +registries are encouraged to adopt a +scoped, revocable authorization framework like [OAuth 2.0][RFC 6749], +but this isn't a strict requirement. +Package maintainers and consumers should +consider a registry's security posture alongside its other features +when deciding where to host and fetch packages. + +Our proposal's package identity scheme is designed to prevent or mitigate +vulnerabilities common to packaging systems and networked applications: + +- Package scopes are restricted to a limited set of characters, + preventing [homograph attacks]. + For example, + "А" (U+0410 CYRILLIC CAPITAL LETTER A) is an invalid scope character + and cannot be confused for "A" (U+0041 LATIN CAPITAL LETTER A). +- Package scopes disallow leading, trailing, or consecutive hyphens (`-`), + and disallows underscores (`_`) entirely, + which mitigates look-alike package scopes + (for example, "llvm--swift" and "llvm_swift" are both invalid + and cannot be confused for "llvm-swift"). +- Package scopes disallow dots (`.`), + which prevents potential confusion with domain variants of scopes + (for example, "apple.com" is invalid + and cannot be confused for "apple"). +- Packages are registered within a scope, + which mitigates [typosquatting]. + Package registries may further restrict the assignment of new scopes + that are intentionally misleading + (for example, "G00gle", which looks like "Google"). +- Package names disallow punctuation and whitespace characters used in + [cross-site scripting][xss] and + [CRLF injection][http header injection] attacks. + +To better understand the security implications of this proposal — +and Swift dependency management more broadly — +we employ the + +[STRIDE] + mnemonic below: + +### Spoofing + +An attacker could interpose a proxy between the client and the package registry +to intercept credentials for that host +and use them to impersonate the user in subsequent requests. + +The impact of this attack is potentially high, +depending on the scope and level of privilege associated with these credentials. +However, the use of secure connections over HTTPS +goes a long way to mitigate the overall risk. + +Swift Package Manager could further mitigate this risk +by taking the following measures: + +* Enforcing HTTPS for all URLs +* Resolving URLs using DNS over HTTPS (DoH) +* Requiring URLs with Internationalized Domain Names (IDNs) + to be represented as Punycode + +### Tampering + +An attacker could interpose a proxy between the client and the package registry +to construct and send Zip files containing malicious code. + +Although the impact of such an attack is potentially high, +the risk is largely mitigated by the use of cryptographic checksums +to verify the integrity of downloaded source archives. + +```console +$ echo "$(swift package compute-checksum LinkedList-1.2.0.zip) *LinkedList-1.2.0.zip" | \ + shasum -a 256 -c - +LinkedList-1.2.0.zip: OK +``` + +Integrity checks alone can't guarantee +that a package isn't a forgery; +an attacker could compromise the website of the host +and provide a valid checksum for a malicious package. + +`Package.resolved` provides a [Trust on first use (TOFU)][TOFU] security model +that can offer strong guarantees about the integrity of dependencies over time. +A registry can further improve on this model by implementing a +[transparent log], +[checksum database], +or another comparable, tamper-proof system +for authenticating package contents. + +Distribution of packages through Zip files +introduces new potential attack vectors. +For example, +an attacker could maliciously tamper with a generated source archive +in an attempt to exploit +a known vulnerability like [Zip Slip], +or a common software weakness like susceptibility to a [Zip bomb]. +Swift Package Manager should take care to +identify and protect against these kinds of attacks +in its implementation of source archive decompression. + +### Repudiation + +A compromised host could serve a malicious package with a valid checksum +and be unable to deny its involvement in constructing the forgery. + +This threat is unique and specific to binary and source artifacts; +Git repositories can have their histories audited, +and individual commits may be cryptographically signed by authors. +Unless you can establish a direct connection between +an artifact and a commit in a source tree, +there's no way to determine the provenance of that artifact. + +Source archives generated by [`git-archive(1)`] +include the checksum of the `HEAD` commit as a comment. +If the history of a project is available +and the commit used to generate the source archive is signed with [GPG], +the cryptographic signature may be used to verify the authenticity. + +```console +$ git rev-parse HEAD +b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3 + +$ swift package archive-source -o LinkedList-1.2.0.zip +Generated LinkedList-1.2.0.zip + +$ zipnote LinkedList-1.2.0.zip | grep "@ (zip file comment below this line)" -A 1 | tail -n 1 +b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3 + +$ git verify-commit b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3 +gpg: Signature made Tue Dec 16 00:00:00 2020 PST +gpg: using RSA key BFAA7114B920808AA4365C203C5C1CF +gpg: Good signature from "Mona Lisa Octocat " [ultimate] +``` + +Otherwise, +a checksum database and the use of digital signatures +can both provide similar non-repudiation guarantees. + +### Information disclosure + +A user may inadvertently expose credentials +by checking in their project's configuration files. +An attacker could scrape public code repositories for configuration files +and attempt to reuse credentials to impersonate the user. + +The risk of leaking credentials can be mitigated by +storing them in a `.netrc` file located outside the project directory +(typically in the user's home directory). +However, +a user may run `swift package` subcommands with the `--netrc-file` option +to configure the location of their project's `.netrc` file. +To mitigate the risk of a user inadvertently +adding a local `.netrc` file to version control, +Swift Package Manager could add an entry to the `.gitignore` file template +for new projects created with `swift package init`. + +Code hosting providers can also help minimize this risk +by [detecting secrets][secret scanning] +that are committed to public repositories. + +Credentials may also be unintentionally disclosed +by Swift Package Manager or other tools in logging statements. +Care should be taken to redact usernames and passwords +when displaying feedback to the user. + +### Denial of service + +An attacker could scrape public code repositories +for `.swiftpm/configuration/registries.json` files +that declare one or more custom registries +and launch a denial-of-service attack +in an attempt to reduce the availability of those resources. + +```json +{ + "registries": { + "[default]": { + "url": "https://private.example.com" + } + }, + "version": 1 +} + +``` + +The likelihood of this attack is generally low +but could be used in a targeted way +against resources known to be important or expensive to distribute. + +This kind of attack can be mitigated on an individual basis +by adding `.swiftpm/configuration` to a project's `.gitignore` file. + +### Escalation of privilege + +Even authentic packages from trusted creators can contain malicious code. + +Code analysis tools can help to some degree, +as can system permissions and other OS-level security features. +However, developers are ultimately responsible for the code they ship to users. + +## Impact on existing packages + +Current packages won't be affected by this change, +as they'll continue to download dependencies directly through Git. + +## Alternatives considered + +### Use of alternative naming schemes + +Some package systems, +including [RubyGems], [PyPI], and [CocoaPods] +identify packages with bare names in a flat namespace +(for example, `rails`, `pandas`, or `Alamofire`). +Other systems, +including [Maven], +use [reverse domain name notation] to identify software components +(for example, `com.squareup.okhttp3`). + +We considered these and other schemes for identifying packages, +but they were rejected in favor of the scoped package identity +described in this proposal. + +### Use of `tar` or other archive formats + +Swift Package Manager currently uses Zip archives for binary dependencies, +which is reason enough to use it again here. + +Zip files are also a convenient format for package registries, +because they support the access of individual files within an archive. +This allows a registry to satisfy +the package manifest endpoint +(`GET /{scope}/{name}/{version}/Package.swift`) +without storing anything separately from the archive used for the +package archive endpoint +(`GET /{scope}/{name}/{version}.zip`). + +We briefly considered `tar` as an archive format +but concluded that its behavior of preserving symbolic links and executable bits +served no useful purpose in the context of package management, +and instead raised concerns about portability and security. + +### Inclusion of alternative source locations in package releases payload + +To maintain compatibility with existing, URL-based dependency declarations +Swift Package Manager needs to reconcile source locations +with their respective identifiers. +For example, +the declarations +`.package(url: "https://github.com/mona/LinkedList", .exact("1.1.0"))` and +`.package(id: "mona.LinkedList", .exact("1.1.0"))`, +must be deemed equivalent +to resolve a dependency graph that contains both of them. + +We considered including alternative source locations in the response body, +but rejected that in favor of using link relations. + +[Web linking][RFC 8288] provides a standard way to +describe the relationships between resources. +Standard `canonical` and `alternative` [IANA link relations] +convey precise semantics for +the relationship between a package and its source repositories +that are broadly useful beyond any individual client. + +### Addition of an `unarchive-source` subcommand + +This proposal adds an `archive-source` subcommand +as a standard way for developers and registries +to create source archives for packages. +Having a canonical tool for creating source archives +avoids any confusion when attempting to verify the integrity of +Zip files sent from a registry +with the source code for that package. + +We considered including a complementary `unarchive-source` subcommand +but ultimately decided against it, +the reason being that unarchiving a Zip archive +is unambiguous and well-supported on most platforms. + +### Use of digital signatures + +[SE-0272] includes a discussion about +the use of digital signatures for binary dependencies, +concluding that they were unsuitable +because of complexity around transitive dependencies. +However, it's unclear what specific objections were raised in this proposal. +We didn't see any inherent tension with the example provided, +and no further explanation was given. + +Without understanding the context of this decision, +we decided it was best to abide by their determination +and instead consider adding this functionality in a future proposal. +For the reasons outlined in the preceding Security section, +we believe that digital signatures may offer additional guarantees +of authenticity and non-repudiation beyond what's possible with checksums alone. + +## Future directions + +Defining a standard interface for package registries +lays the groundwork for several useful features. + +### Package publishing + +A package registry is responsible for determining +which package releases are made available to a consumer. +This proposal sets no policies for how +package releases are published to a registry. +Nor does it specify how package scopes are registered or verified. + +Many package managers — +including the ones mentioned above — +and artifact repository services, such as +[Docker Hub], +[JFrog Artifactory], +and [AWS CodeArtifact] +follow what we describe as a *"push"* model of publication: +When a package owner wants to releases a new version of their software, +they produce a build locally and push the resulting artifact to a server. +This model has the benefit of operational simplicity and flexibility. +For example, +maintainers have an opportunity to digitally sign artifacts +before uploading them to the server. + +Alternatively, +a system might incorporate build automation techniques like +continuous integration (CI) and continuous delivery (CD) +into what we describe as a *"pull"* model: +When a package owner wants to release a new version of their software, +their sole responsibility is to notify the package registry; +the server does all the work of downloading the source code +and packaging it up for distribution. +This model can provide strong guarantees about +reproducibility, quality assurance, and software traceability. + +We intend to work with industry stakeholders +to develop standards for publishing Swift packages +in an extension to the registry specification. + +### Package removal + +Removing a package from a registry +can break other packages that depend on it, +as demonstrated by the ["left-pad" incident][left-pad] in March 2016. +We believe package registries can and should +provide strong durability guarantees +to ensure the health of the ecosystem. + +At the same time, +there are valid reasons why a package release may be removed: + +* The package maintainer publishing a release by mistake +* A security researcher disclosing a vulnerability for a release +* The registry being compelled by law enforcement to remove a release + +It's unclear whether and to what extent package deletion policies +should be informed by the registry specification itself. +For now, +a registry is free to exercise its own discretion +about how to respond to out-of-band removal requests. + +We plan to consider these questions +as part of the future extension to the specification +described in the previous section. + +### Package dependency URL normalization + +As described in ["Package name collision resolution"](#package-name-collision-resolution) +Swift Package Manager cannot build a project +if two or more packages in the project +are located by URLs with the same (case-insensitive) last path component. +Swift Package Manager may improve support URL-based dependencies +by normalizing package URLs to mitigate insignificant variations. +For example, +a package with an ["scp-style" URL][scp-url] like +`git@github.com:mona/LinkedList.git` +may be determined to be equivalent to a package with an HTTPS scheme like +`https:///github.com/mona/LinkedList`. + +### Local offline cache + +Swift Package Manager could implement an [offline cache] +that would allow it to work without network access. +While this is technically possible today, +a package registry makes for a simpler and more secure implementation +than would otherwise be possible with Git repositories alone. + +### Binary framework distribution + +The registry specification could be amended to support the distribution of +[XCFramework] bundles or [artifact archives][SE-0305]. + +```http +GET /github.com/mona/LinkedList/1.1.1.xcframework HTTP/1.1 +Host: packages.github.com +Accept: application/vnd.swift.registry.v1+xcframework +``` + +Swift Package Manager could then use XCFramework archives as +[binary dependencies][SE-0272] +or as part of a future binary artifact distribution mechanism. + +```swift +let package = Package( + name: "SomePackage", + /* ... */ + targets: [ + .binaryTarget( + name: "LinkedList", + url: "https://packages.github.com/github.com/mona/LinkedList/1.1.1.xcframework", + checksum: "ed04a550c2c7537f2a02ab44dd329f9e74f9f4d3e773eb883132e0aa51438b37" + ), + ] +) +``` + +### Updates to package editor commands + +[Package editor commands][SE-0301] +could be extended to add dependencies using scoped identifiers +in addition to URLs. + +```console +$ swift package add-dependency mona.LinkedList +# Installed LinkedList 1.2.0 +``` + +```diff ++ .package(id: "mona.LinkedList", .exact("1.2.0")) +``` + +### Package manifest dependency migration + +Swift Package Manager could add tooling +to help package maintainers adopt registry-supported identifiers +in their projects. + +```console +$ swift package-registry migrate +``` + +```diff +- .package(url: "https://github.com/mona/LinkedList", .exact("1.2.0")) ++ .package(id: "mona.LinkedList", .exact("1.2.0")) +``` + +### Security audits + +The response for listing package releases could be updated to include +information about security advisories. + +```jsonc +{ + "releases": { /* ... */ }, + "advisories": [{ + "cve": "CVE-20XX-12345", + "cwe": "CWE-400", + "package": "mona.LinkedList", + "vulnerable_versions": "<=1.0.0", + "patched_versions": ">1.0.0", + "severity": "moderate", + "recommendation": "Update to version 1.0.1 or later.", + /* additional fields */ + }] +} +``` + +Swift Package Manager could communicate this information to users +when installing or updating dependencies +or as part of a new `swift package audit` subcommand. + +```console +$ swift package audit +┌───────────────┬────────────────────────────────────────────────┐ +│ High │ Regular Expression Denial of Service │ +├───────────────┼────────────────────────────────────────────────┤ +│ Package │ mona.RegEx │ +├───────────────┼────────────────────────────────────────────────┤ +│ Dependency of │ PatternMatcher │ +├───────────────┼────────────────────────────────────────────────┤ +│ Path │ SomePackage > PatternMatcher > RegEx │ +├───────────────┼────────────────────────────────────────────────┤ +│ More info │ https://example.com/advisories/526 │ +└───────────────┴────────────────────────────────────────────────┘ + +Found 3 vulnerabilities (1 low, 1 moderate, 1 high) in 8 scanned packages. + Run `swift package audit fix` to fix 3 of them. +``` + +### Package search + +The package registry API could be extended to add a search endpoint +to allow users to search for packages by name, keywords, or other criteria. +This endpoint could be used by clients like Swift Package Manager. + +```console +$ swift package search LinkedList +LinkedList (github.com/mona/LinkedList) - One thing links to another. + +$ swift package search --author "Mona Lisa Octocat" +LinkedList (github.com/mona/LinkedList) - One thing links to another. +RegEx (github.com/mona/RegEx) - Expressions on the reg. +``` + +[AWS CodeArtifact]: https://aws.amazon.com/codeartifact/ +[BCP 13]: https://tools.ietf.org/html/rfc6838 "Media Type Specifications and Registration Procedures" +[CDN]: https://en.wikipedia.org/wiki/Content_delivery_network "Content delivery network" +[checksum database]: https://sum.golang.org "Go Module Mirror, Index, and Checksum Database" +[CocoaPods]: https://cocoapods.org "A dependency manager for Swift and Objective-C Cocoa projects" +[crates.io]: https://crates.io "crates.io: The Rust community’s crate registry" +[Docker Hub]: https://hub.docker.com +[GPG]: https://gnupg.org +[homograph attacks]: https://en.wikipedia.org/wiki/IDN_homograph_attack +[http header injection]: https://en.wikipedia.org/wiki/HTTP_header_injection +[IANA link relations]: https://www.iana.org/assignments/link-relations/link-relations.xhtml "IANA Link Relation Types" +[ICANN]: https://www.icann.org +[JFrog Artifactory]: https://jfrog.com/artifactory/ +[JSON-LD]: https://w3c.github.io/json-ld-syntax/ "JSON-LD 1.1: A JSON-based Serialization for Linked Data" +[left-pad]: https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/ "How one programmer broke the internet by deleting a tiny piece of code" +[Maven]: https://maven.apache.org +[npm]: https://www.npmjs.com "The npm Registry" +[offline cache]: https://yarnpkg.com/features/offline-cache "Offline Cache | Yarn - Package Manager" +[PyPI]: https://pypi.org "PyPI: The Python Package Index" +[reverse domain name notation]: https://en.wikipedia.org/wiki/Reverse_domain_name_notation +[RFC 2119]: https://tools.ietf.org/html/rfc2119 "Key words for use in RFCs to Indicate Requirement Levels" +[RFC 3230]: https://tools.ietf.org/html/rfc5843 "Instance Digests in HTTP" +[RFC 3492]: https://tools.ietf.org/html/rfc3492 "Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)" +[RFC 3986]: https://tools.ietf.org/html/rfc3986 "Uniform Resource Identifier (URI): Generic Syntax" +[RFC 3987]: https://tools.ietf.org/html/rfc3987 "Internationalized Resource Identifiers (IRIs)" +[RFC 5234]: https://tools.ietf.org/html/rfc5234 "Augmented BNF for Syntax Specifications: ABNF" +[RFC 5843]: https://tools.ietf.org/html/rfc5843 "Additional Hash Algorithms for HTTP Instance Digests" +[RFC 6249]: https://tools.ietf.org/html/rfc6249 "Metalink/HTTP: Mirrors and Hashes" +[RFC 6570]: https://tools.ietf.org/html/rfc6570 "URI Template" +[RFC 6749]: https://tools.ietf.org/html/rfc6749 "The OAuth 2.0 Authorization Framework" +[RFC 7230]: https://tools.ietf.org/html/rfc7230 "Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing" +[RFC 7231]: https://tools.ietf.org/html/rfc7231 "Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content" +[RFC 7233]: https://tools.ietf.org/html/rfc7233 "Hypertext Transfer Protocol (HTTP/1.1): Range Requests" +[RFC 7234]: https://tools.ietf.org/html/rfc7234 "Hypertext Transfer Protocol (HTTP/1.1): Caching" +[RFC 7807]: https://tools.ietf.org/html/rfc7807 "Problem Details for HTTP APIs" +[RFC 8288]: https://tools.ietf.org/html/rfc8288 "Web Linking" +[RFC 8446]: https://tools.ietf.org/html/rfc8446 "The Transport Layer Security (TLS) Protocol Version 1.3" +[RubyGems]: https://rubygems.org "RubyGems: The Ruby community’s gem hosting service" +[Schema.org]: https://schema.org/ +[scp-url]: https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols#_the_ssh_protocol +[SE-0219]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0219-package-manager-dependency-mirroring.md "Package Manager Dependency Mirroring" +[SE-0272]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md "Package Manager Binary Dependencies" +[SE-0301]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0301-package-editing-commands.md "Package Editor Commands" +[SE-0305]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md "Package Manager Binary Target Improvements" +[secret scanning]: https://docs.github.com/en/github/administering-a-repository/about-secret-scanning +[SemVer]: https://semver.org/ "Semantic Versioning" +[SoftwareSourceCode]: https://schema.org/SoftwareSourceCode +[STRIDE]: https://en.wikipedia.org/wiki/STRIDE_(security) "STRIDE (security)" +[thundering herd effect]: https://en.wikipedia.org/wiki/Thundering_herd_problem "Thundering herd problem" +[TOFU]: https://en.wikipedia.org/wiki/Trust_on_first_use "Trust on First Use" +[transparent log]: https://research.swtch.com/tlog +[typosquatting]: https://en.wikipedia.org/wiki/Typosquatting +[UTI]: https://en.wikipedia.org/wiki/Uniform_Type_Identifier +[version-specific-manifest-selection]: https://github.com/apple/swift-package-manager/blob/main/Documentation/Usage.md#version-specific-manifest-selection "Swift Package Manager - Version-specific Manifest Selection" +[version-specific-tag-selection]: https://github.com/apple/swift-package-manager/blob/main/Documentation/Usage.md#version-specific-tag-selection "Swift Package Manager - Version-specific Tag Selection" +[XCFramework]: https://developer.apple.com/videos/play/wwdc2019/416/ "WWDC 2019 Session 416: Binary Frameworks in Swift" +[xss]: https://en.wikipedia.org/wiki/Cross-site_scripting +[Zip bomb]: https://en.wikipedia.org/wiki/Zip_bomb "Zip bomb" +[Zip Slip]: https://snyk.io/research/zip-slip-vulnerability "Zip Slip Vulnerability" diff --git a/proposals/0293-extend-property-wrappers-to-function-and-closure-parameters.md b/proposals/0293-extend-property-wrappers-to-function-and-closure-parameters.md new file mode 100644 index 0000000000..cb28e27d6c --- /dev/null +++ b/proposals/0293-extend-property-wrappers-to-function-and-closure-parameters.md @@ -0,0 +1,762 @@ +# Extend Property Wrappers to Function and Closure Parameters + +* Proposal: [SE-0293](0293-extend-property-wrappers-to-function-and-closure-parameters.md) +* Authors: [Holly Borla](https://github.com/hborla), [Filip Sakel](https://github.com/filip-sakel) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 5.5)** +* Implementation: [apple/swift#34272](https://github.com/apple/swift/pull/34272), [apple/swift#36344](https://github.com/apple/swift/pull/36344) +* Decision Notes: [Review #3](https://forums.swift.org/t/accepted-se-0293-extend-property-wrappers-to-function-and-closure-parameters/47030), [Review #2](https://forums.swift.org/t/returned-for-revision-2-se-0293-extend-property-wrappers-to-function-and-closure-parameters/44832), [Review #1](https://forums.swift.org/t/returned-for-revision-se-0293-extend-property-wrappers-to-function-and-closure-parameters/42953) +* Previous versions: [Revision #2](https://github.com/swiftlang/swift-evolution/blob/bdf12b26d15d63ab7e58dab635a55ffeca841389/proposals/0293-extend-property-wrappers-to-function-and-closure-parameters.md), [Revision #1](https://github.com/swiftlang/swift-evolution/blob/e5b2ce1fd6c1c2617a820a1e6f2b53a00e54fdce/proposals/0293-extend-property-wrappers-to-function-and-closure-parameters.md) + +## Contents + ++ [Introduction](#introduction) ++ [Motivation](#motivation) + - [Applying a common behavior via property wrapper](#applying-a-common-behavior-via-property-wrapper) + - [Arguments with auxiliary values via property wrapper projection](#arguments-with-auxiliary-values-via-property-wrapper-projection) ++ [Proposed solution](#proposed-solution) ++ [Detailed design](#detailed-design) + - [Passing a projected value argument](#passing-a-projected-value-argument) + - [Inference of API-level property wrappers](#inference-of-api-level-property-wrappers) + - [Implementation-detail property wrappers](#implementation-detail-property-wrappers) + - [Arguments in the property-wrapper attribute](#arguments-in-the-property-wrapper-attribute) + - [API-level property wrappers](#api-level-property-wrappers) + - [Function-body semantics](#function-body-semantics) + - [Call-site semantics](#call-site-semantics) + - [Unapplied function references](#unapplied-function-references) + - [Closures](#closures) + - [Overload resolution of backing property-wrapper initializer](#overload-resolution-of-backing-property-wrapper-initializer) + - [Restrictions on property-wrapper parameters](#restrictions-on-property-wrapper-parameters) ++ [Source compatibility](#source-compatibility) ++ [Effect on ABI stability](#effect-on-abi-stability) ++ [Effect on API resilience](#effect-on-api-resilience) ++ [Alternatives considered](#alternatives-considered) + - [Preserve property wrapper parameter attributes in the type system](#preserve-property-wrapper-parameter-attributes-in-the-type-system) + - [Only allow implementation-detail property wrappers on function parameters](#only-allow-implementation-detail-property-wrappers-on-function-parameters) + - [Pass a property-wrapper storage instance directly](#pass-a-property-wrapper-storage-instance-directly) ++ [Future directions](#future-directions) + - [The impact of formalizing separate property wrapper models](#the-impact-of-formalizing-separate-property-wrapper-models) + - [Explicit spelling for API-level property wrappers](#explicit-spelling-for-api-level-property-wrappers) + - [Generalized property-wrapper initialization from a projection](#generalized-property-wrapper-initialization-from-a-projection) + - [Static property-wrapper attribute arguments](#static-property-wrapper-attribute-arguments) + - [API property wrappers in protocol requirements](#api-property-wrappers-in-protocol-requirements) + - [Extend property wrappers to patterns](#extend-property-wrappers-to-patterns) + - [Support `inout` in wrapped function parameters](#support-inout-in-wrapped-function-parameters) ++ [Revisions](#revisions) + - [Changes from the second reviewed version](#changes-from-the-second-reviewed-version) + - [Changes from the first reviewed version](#changes-from-the-first-reviewed-version) ++ [Appendix](#appendix) + - [Mutability of composed `wrappedValue` accessors](#mutability-of-composed-wrappedValue-accessors) ++ [Acknowledgements](#acknowledgements) + + +## Introduction + +Property Wrappers were [introduced in Swift 5.1](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md), and have since become a popular mechanism for abstracting away common accessor patterns for properties. Currently, applying a property wrapper is solely permitted on local variables and type properties. However, with increasing adoption, demand for extending _where_ property wrappers can be applied has emerged. This proposal aims to extend property wrappers to function and closure parameters. + + +## Motivation + +Property wrappers have undoubtably been very successful. Applying a property wrapper to a property is enabled by an incredibly lightweight and expressive syntax. For instance, frameworks such as [SwiftUI](https://developer.apple.com/documentation/swiftui/) and [Combine](https://developer.apple.com/documentation/combine) introduce property wrappers such as [`State`](https://developer.apple.com/documentation/swiftui/state), [`Binding`](https://developer.apple.com/documentation/swiftui/binding) and [`Published`](https://developer.apple.com/documentation/combine/published) to expose elaborate behavior through a succinct interface, helping craft expressive yet simple APIs. However, property wrappers are only applicable to local variables and type properties, shattering the illusion that they helped realize in the first place when working with parameters. + +Property wrappers attached to parameters have a wide variety of use cases. We present a few examples here. + +### Applying a common behavior via property wrapper + +Property wrappers are often used as sugar for applying a common behavior to a value, such as asserting a precondition, transforming the value, or logging the value. Such behaviors are valuable to apply to function parameters. For example, using `Validation` from [`PropertyKit`](https://github.com/SvenTiigi/ValidatedPropertyKit), we can abstract various preconditions into a property wrapper: + +```swift +@propertyWrapper +struct Asserted { + init( + wrappedValue: Value, + validation: Validation, + ) { ... } + + var wrappedValue: Value { ... } +} +``` + +It would be useful to apply `@Asserted` to parameters to assert certain preconditions on argument values. For example, the following code asserts that the argument passed to the `quantity` parameter is greater than or equal to 1: + +```swift +func buy( + @Asserted(.greaterOrEqual(1)) quantity: Int, + of product: Product, +) { ... } +``` + +Similarly, one could write an `@Logged` property wrapper to be used as a light-weight debugging tool to see the arguments passed to a function each time that function is called: + +```swift +@propertyWrapper +struct Logged { + init(wrappedValue: Value) { + print(wrappedValue) + self.wrappedValue = wrappedValue + } + + var wrappedValue: Value { + didSet { + print(wrappedValue) + } + } +} + +// Every time `runAnimation` is called, the `duration` argument +// will be logged by the property wrapper. +func runAnimation(@Logged withDuration duration: Double) { ... } +``` + +### Arguments with auxiliary values via property wrapper projection + +Consider the following property wrapper, inspired by `@Traceable` from [David Piper's blog post](https://medium.com/better-programming/creating-a-history-with-property-wrappers-in-swift-5-1-4c0202060a7f), which tracks the history of a value: + +```swift +struct History { ... } + +@propertyWrapper +struct Traceable { + init(wrappedValue value: Value) { ... } + init(projectedValue: History) { ... } + + private var history: History + + var wrappedValue: Value { + get { + history.currentValue + } + set { + history.append(newValue) + } + } + + var projectedValue: History { + history + } +} +``` + +This property wrapper provides the history of the traced value via its projection, and it can be initialized with a value to be traced, or with an existing history of a traced value. Now consider the following model for a simple text editor that supports change tracking: + +```swift +struct TextEditor { + @Traceable var dataSource: String +} +``` + +Currently, property-wrapper attributes on struct properties interact with function parameters through the struct's synthesized member-wise initializer. Because the `@Traceable` property wrapper supports initialization from a wrapped value via `init(wrappedValue:)`, the member-wise initializer for `TextEditor` will take in a `String`. However, the programmer may want to initialize `TextEditor` with a string value that already has a history. Today, this behavior can be achieved with overloads, which can greatly impact compile-time performance and impose boilerplate on the programmer. Another approach is to expose the `Traceable` type through the `TextEditor` initializer, which is unfortunate since the backing storage is meant to be implementation detail. + +## Proposed solution + +We propose to allow application of property wrappers on function and closure parameters, allowing the call-site to pass a wrapped value, or a projected value if appropriate, which will be used to automatically initialize the backing property wrapper. Within the body of the function, the function author can use the property-wrapper syntax for accessing the backing wrapper and the projected value. + +It's clear from a survey of the use cases for property wrappers on parameters that there are two kinds of property wrappers. The first kind of property wrapper is an abstraction of a common behavior on a value, such as logging, transforming, or caching a value. For these property wrappers, you use the wrapped value generally the same way as you would if the value did not have the wrapper attached, and the wrapper itself is implementation detail. Callers that provide the value to initialize the property wrapper will always pass an instance of the wrapped-value type. + +The second kind of property wrapper attaches additional semantics to the value being wrapped that are fundamental to understanding how the wrapped value can be used. These wrappers tend to attach auxiliary API through the wrapper's `projectedValue`, and many of these wrappers cannot be initialized from an instance of the wrapped-value type. + +The natural model for these two kinds of wrappers is different when applied to parameters, because the second model must allow the caller to pass a different type of argument. We propose to formalize the difference between 1) API-level property wrappers that have an external effect on the function, and 2) implementation-detail property wrappers. The compiler will determine whether a property wrapper must have an external effect on the function by analyzing the property wrapper's initializers. + +## Detailed design + +### Passing a projected value argument + +Property-wrapper projections are designed to allow property wrappers to provide a representation of the storage type that can be used outside of the context that owns the property-wrapper storage. Typically, projections either expose the backing property wrapper directly, or provide an instance of a separate type that vends more restricted access to the functionality of the property wrapper. + +When a property-wrapper has a projection, it's often necessary to use the projection alongside the wrapped value. In such cases, the projection is equal in importance to the wrapped value in the API of the wrapped property, which is reflected in the access control of synthesized projection properties. With respect to function parameters, it's equally important to support passing a projection. + +Property wrappers can enable passing a projected-value argument to a property-wrapped parameter by declaring `var projectedValue`, and implementing an `init(projectedValue:)` that meets the following requirements: + +- The first parameter of this initializer must be labeled `projectedValue` and have the same type as the `var projectedValue` property. +- The initializer must have the same access level as the property-wrapper type. +- The initializer must not be failable. +- Any additional parameters to the initializer must have default arguments. + +This method of initialization is not mandatory for functions using supported wrapper types, and it can be disabled by providing arguments in the wrapper attribute, including empty attribute arguments: `func log(@Traceable() _ value: Value) { ... }`. + +### Inference of API-level property wrappers + +For a given property wrapper attached to a parameter, the compiler will infer whether that wrapper is part of the function signature based on whether the wrapper must have an external effect on the argument at the call-site. This proposal limits external argument effects to the case where the property wrapper allows the caller to pass an instance of the projected-value type, which means the property wrapper supports projected-value initialization via `init(projectedValue:)` and there are no arguments in the wrapper attribute. + +A property wrapper will only be inferred as API if `init(projectedValue:)` is declared directly in the nominal property wrapper type. This is to ensure that the same decision is always made regardless of which module the property wrapper is applied in. This is the same strategy that is used to determine whether a computed projection property with the `$` prefix should be synthesized when a property wrapper is applied, and whether a property wrapper supports initialization from a wrapped value. Once it is determined whether a property wrapper is API or implementation-detail, normal [overload resolution rules](#overload-resolution-of-backing-property-wrapper-initializer) will apply to the backing property wrapper initializer. + +### Implementation-detail property wrappers + +By default, property wrappers are implementation detail. Attaching an implementation-detail property wrapper attribute to a parameter will synthesize the following local variables in the function body: + +* A local `let`-constant representing the backing storage will be synthesized with the name of the parameter prefixed with an underscore. The backing storage is initialized by passing the parameter to `init(wrappedValue:)`. +* A local computed variable representing the `wrappedValue` of the innermost property wrapper will be synthesized with the same name as the original, unprefixed parameter name. If the innermost `wrappedValue` defines a setter, a setter will be synthesized for the local property if the [mutability of the composed setter](#mutability-of-composed-wrappedValue-accessors) is `nonmutating`. +* If the outermost property wrapper defines a `projectedValue` property with a `nonmutating` getter, a local computed variable representing the outermost `projectedValue` will be synthesized and named per the original parameter name prefixed with a dollar sign (`$`). If the outermost `projectedValue` defines a setter, a setter for the local computed variable will be synthesized if the `projectedValue` setter is `nonmutating`. + + +Consider the following code, which attaches the `@Logged` property wrapper to a parameter. + +```swift +func insert(@Logged text: String) { ... } +``` + +The above code is sugar for: + +```swift +func insert(text: String) { + let _text: Logged = Logged(wrappedValue: text) + + var text: String { _text.wrappedValue } +} +``` + +Note that the backing storage is a let-constant, and the local `text` property does not have a setter. + +> **Rationale**: The ability to mutate a wrapped parameter would likely confuse users into thinking that the mutations they make are observable by the caller; that's not the case. There was a similar feature in Swift which was removed in [SE-0003](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0003-remove-var-parameters.md). + +Implementation-detail property wrappers on parameters must support initialization from a wrapped value, and the parameter type must be equal to the wrapped value type of the wrapper. Because the backing storage is initialized locally, implementation-detail property wrappers have no external effect on the function. A function that uses implementation-detail property wrappers on parameters can fulfill protocol requirements that use the wrapped-value type: + +```swift +protocol P { + func requirement(value: Int) +} + +struct S: P { + func requirement(@Logged value: Int) { + ... + } +} +``` + +#### Arguments in the property-wrapper attribute + +Property-wrapper attributes with arguments applied to parameters are always implementation-detail property wrappers, even if the property wrapper supports initialization from a projected value. + +> **Rationale**: Arguments in the wrapper attribute only apply to `init(wrappedValue:)`. To ensure that these arguments never change, the property wrapper must always be initialized via `init(wrappedValue:)` and pass the additional attribute arguments. Because the caller can only pass a wrapped value, there is no reason for the property wrapper to affect the function externally. + +Because property wrappers with attribute arguments are always implementation-detail, the arguments will always be evaluated in the function body. + +### API-level property wrappers + +Property wrappers that declare an `init(projectedValue:)` initializer are inferred to be API-level wrappers. These wrappers become part of the function signature, and the property wrapper is initialized at the call-site of the function. + +#### Function-body semantics + +Attaching an API-level property wrapper to a parameter makes that parameter a computed variable local to the function body, and changes the parameter type to the backing wrapper type. The type of the parameter is only observable in compiled code; [unapplied references to functions with property-wrapped parameters](#unapplied-function-references) will not use the backing-wrapper type. + +The transformation of functions with a property-wrapped parameter will be performed as such: + +* The argument label will remain unchanged. +* The parameter name will be prefixed with an underscore. +* The type of the parameter will be the backing property-wrapper type. +* A local computed variable representing the `wrappedValue` of the innermost property wrapper will be synthesized with the same name as the original, unprefixed parameter name. If the innermost `wrappedValue` defines a setter, a setter will be synthesized for the local property if the [mutability of the composed setter](#mutability-of-composed-wrappedValue-accessors) is `nonmutating`. +* If the outermost property wrapper defines a `projectedValue` property with a `nonmutating` getter, a local computed variable representing the outermost `projectedValue` will be synthesized and named per the original parameter name prefixed with a dollar sign (`$`). If the outermost `projectedValue` defines a setter, a setter for the local computed variable will be synthesized if the `projectedValue` setter is `nonmutating`. + +Consider the following function which has a parameter with the `@Traceable` property wrapper attached: + +```swift +func copy(@Traceable text: String) { ... } +``` + +The compiler will synthesize computed `text` and `$text` variables in the body of `copy(text:)`: + +```swift +func copy(text _text: Traceable) { + var text: String { + get { _text.wrappedValue } + } + + var $text: History { + get { _text.projectedValue } + } + + ... +} +``` + +#### Call-site semantics + +When passing an argument to a parameter with an API-level property wrapper, the compiler will wrap the argument in a call to the appropriate initializer depending on the argument label. When using the original argument label (or no argument label), the compiler will wrap the argument in a call to `init(wrappedValue:)`. When using the argument label prefixed with `$` (or `$_` in the case of no argument label), the compiler will wrap the argument in a call to `init(projectedValue:)`. + +Consider the `@Traceable` property wrapper that implements both `init(wrappedValue:)` and `init(projectedValue:)`: + +```swift +struct History { ... } + +@propertyWrapper +struct Traceable { + init(wrappedValue value: Value) + init(projectedValue: History) + + var wrappedValue: Value + var projectedValue: History +} +``` + +A function with an `@Traceable` parameter can be called with either a wrapped value or a projected value: + +```swift +func log(@Traceable value: Value) { ... } + +let history: History = ... + +log(value: 10) +log($value: history) +``` + +The compiler will inject a call to the appropriate property-wrapper initializer into each call to `log` based on the argument label, so the above code is transformed to: + +```swift +log(value: Traceable(wrappedValue: 10)) +log(value: Traceable(projectedValue: history)) +``` + +Wrapped parameters with no argument label can still be passed a projection using the syntax `$_:`, as shown in the following example: + +```swift +func log(@Traceable _ value: Value) { ... } + +let history: History = ... + +log(10) +log(_: 10) +log($_: history) +``` + +For composed property wrappers, initialization of the backing wrapper via wrapped value will contain a call to `init(wrappedValue:)` for each property-wrapper attribute in the composition chain. However, initialization via projected value will only contain one call to `init(projectedValue:)` for the outermost wrapper attribute, because property wrapper projections are not composed. For example: + +```swift +func log(@Traceable @Traceable text: String) { ... } + +let history: History> = ... + +log(text: "Hello!") +log($text: history) +``` + +The above calls to `log` are transformed to: + +```swift +log(text: Traceable(wrappedValue: Traceable(wrappedValue: "Hello!")) +log(text: Traceable(projectedValue: history)) +``` + +This transformation at the call-site only applies when calling the function directly using the declaration name. + +#### Unapplied function references + +By default, unapplied references to functions that accept property-wrapped parameters use the wrapped-value type in the parameter list. + +Consider the `log` function from above, which uses the `@Traceable` property wrapper: + +```swift +func log(@Traceable value: Value) { ... } +``` + +The type of `log` is `(Value) -> Void`. These semantics can be observed when working with an unapplied reference to `log`: + +```swift +let logReference: (Int) -> Void = log +logReference(10) + +let labeledLogReference: (Int) -> Void = log(value:) +labeledLogReference(10) +``` + +The compiler will generate a thunk when referencing `log` to take in the wrapped-value type and initialize the backing property wrapper. Both references to `log` in the above example are transformed to: + +```swift +{ log(value: Traceable(wrappedValue: $0) } +``` + +The type of an unapplied function reference can be changed to instead take in the projected-value type using `$` in front of the argument label. Since `Traceable` implements `init(projectedValue:)`, the `log` function can be referenced in a way that takes in `History` by using `$` in front of `value`: + +```swift +let history: History = ... +let logReference: (History) -> Void = log($value:) +logReference(history) +``` + +If a wrapped parameter omits an argument label, the function can be referenced to take in the projected-value type using `$_`: + +```swift +func log(@Traceable _ value: Value) { ... } + +let history: History = ... +let logReference: (History) -> Void = log($_:) +logReference(history) +``` + +### Closures + +Property wrappers can be attached to closure parameter declarations in the closure expression. Property-wrapper attributes are not propagated through the type system, so a given closure can only be passed either a wrapped value or a projected value. Because of this, closures parameters do not distinguish between implementation-detail and API property wrappers; all property wrappers will be initialized from the appropriate argument in the order they appear in the parameter list before the closure body is executed. + +The `log` function from the previous section can be implemented as a closure that takes in the wrapped-value type: + +```swift +let log: (Int) -> Void = { (@Traceable value) in + ... +} +``` + +The closure can be written to instead take in the projected-value type by using the `$` prefix in the parameter name: + +```swift +let log: (History) -> Void = { (@Traceable $value) in + ... +} +``` + +For closures that take in a projected value, the property-wrapper attribute is not necessary if the backing property wrapper and the projected value have the same type, such as the [`@Binding`](https://developer.apple.com/documentation/swiftui/binding) property wrapper from SwiftUI. If `Binding` implemented `init(projectedValue:)`, it could be used as a property-wrapper attribute on closure parameters without explicitly writing the attribute: + +```swift +let useBinding: (Binding) -> Void = { $value in + ... +} +``` + +Since property-wrapper projections are not composed, `$` closure parameters can only have one property-wrapper attribute. + +### Overload resolution of backing property-wrapper initializer + +For both implementation-detail and API property wrappers, the type of the wrapped parameter (not the argument) is used for overload resolution of `init(wrappedValue:)` and `init(projectedValue:)`. For example: + +```swift +@propertyWrapper +struct Wrapper { + init(wrappedValue: Value) { ... } + + init(wrappedValue: Value) where Value: Collection { ... } +} + +func generic(@Wrapper value: T) { ... } +``` + +The above property wrapper defines overloads of `init(wrappedValue:)` with different generic constraints. When the property wrapper is applied to the function parameter `value` of generic parameter type `T`, overload resolution will choose which `init(wrappedValue:)` to call based on the constraints on `T`. `T` is unconstrained, so the unconstrained `init(wrappedValue:)` will always be called: + +```swift +// Both of the following calls use the unconstrained 'init(wrappedValue:)' +generic(value: 10) +generic(value: [1, 2, 3]) +``` + +The function `generic` could be overloaded where `T: Collection` to allow the constrained `init(wrappedValue:)` to be called: + +```swift +func generic(@Wrapper value: T) { ... } +func generic(@Wrapper value: T) { ... } + +generic(value: 10) // calls the unconstrained init(wrappedValue:) +generic(value: [1, 2, 3]) // calls init(wrappedValue:) where Value: Collection +``` + +### Restrictions on property-wrapper parameters + +Property wrappers attached to parameters must support either or both of `init(wrappedValue:)` and `init(projectedValue:)`. + +> **Rationale**: If a property wrapper does not support either of these initializers, the compiler does not know how to automatically initialize the property wrapper given an argument. + +The composed mutability of the innermost `wrappedValue` getter must be `nonmutating`. + +> **Rationale**: If the composed `wrappedValue` getter is `mutating`, then the local computed property for a property-wrapper parameter must mutate the backing wrapper, which is immutable. + +Property-wrapper parameters cannot have an `@autoclosure` type. + +> **Rationale**: A wrapped value cannot have an `@autoclosure` type. If `init(wrappedValue:)` needs to accept an `@autoclosure`, a warning will be emitted with a fix-it prompting the user to use a regular `@autoclosure` parameter and a local property wrapper instead. + +API property-wrapper parameters cannot also have an attached result builder attribute. + +> **Rationale**: Result-builder attributes can be applied to the parameters in `init(wrappedValue:)` and `init(projectedValue:)`. If there is a result builder attached to a property-wrapper parameter that already has a result builder in `init(wrappedValue:)`, it's unclear which result builder should be applied. + +Non-instance methods cannot use property wrappers that require the [enclosing `self` subscript](0258-property-wrappers.md#referencing-the-enclosing-self-in-a-wrapper-type). + +> **Rationale**: Non-instance methods do not have an enclosing `self` instance, which is required for the local computed property that represents `wrappedValue`. + +API property wrapper attributes can only be applied to parameters in overridden functions or protocol witnesses if the superclass function or protocol requirement, respectively, has the same property wrapper attributes. + +> **Rationale**: This restriction ensures that the call-site transformation is always the same for families of dynamically dispatched functions. + +API property wrappers must match the access level of the enclosing function. + +> **Rationale**: These property wrappers have an external effect on the argument at the call-site, so they must be accessible to all callers. + +## Source compatibility + +This is an additive change with no impact on source compatibility. + +## Effect on ABI stability + +This is an additive change with no impact on the existing ABI. + +## Effect on API resilience + +Implementation-detail property wrappers have no impact on API resilience. These property wrappers will not be preserved in the generated Swift interface for the module; they are entirely implementation details. + +API-level property wrappers applied to function parameters are part of the API and ABI of that function. A property wrapper applied to a function parameter changes the type of that parameter, which is reflected in the ABI; it also changes the way that function callers are compiled to pass an argument of that type. Thus, adding or removing a property wrapper on an ABI-public function parameter is not a resilient change. + +Property wrappers changing between implementation-detail and API-level is not a resilient change. Consider a property wrapper that is implementation-detail when applied to a parameter. Adding an `init(projectedValue:)` initializer to this property wrapper is a source-breaking change for clients that use this property wrapper in a function that is a protocol witness, and it is an ABI breaking change for any code that uses this property wrapper on a parameter. We expect this case to be very rare, and clients can work around the source and ABI break by either adding an argument to the wrapper attribute or using a local wrapped variable instead. + +## Alternatives considered + +### Preserve property-wrapper parameter attributes in the type system + +One approach to achieving the expected semantics for higher-order functions with property wrappers in the parameter list is to preserve property-wrapper attributes in parameter types. While this is feasible for plain property-wrapper attributes, it is not feasible in the case where the property-wrapper attribute has attribute arguments, because type equality cannot be dependent on expression equivalence. + +### Only allow implementation-detail property wrappers on function parameters + +Only allowing implementation-detail property wrappers on function parameters would eliminate the need for the API-level versus implementation-detail distinction for functions, because the property wrapper would never have an external effect on the argument. However, allowing property wrappers to have an external effect on the wrapped declaration is part of what makes the feature so powerful and applicable to a wide variety of use cases. One fairly common class of property wrappers are those which provide an abstracted reference to a value, such as the [`Ref` / `Box` example](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md#ref--box) from the SE-0258 proposal, and [`Binding`](https://developer.apple.com/documentation/swiftui/binding) from SwiftUI. It's common to pass these property wrappers around as projections, and there isn't currently a nice way to achieve the property wrapper sugar in a function body that uses such a property wrapper. The best way to achieve this currently is to use a local property wrapper and initialize the backing storage directly, e.g. + +```swift +func useReference(reference: Ref) { + @Ref var value: Int + _value = reference +} +``` + +Furthermore, property wrappers can already have an external effect on the wrapped declaration. For example, the compiler may change the type of the accessors of the wrapped declaration based on the mutability of the property wrapper's `wrappedValue` accessors, and the synthesized member-wise initializer of a type containing wrapped properties can change based on which initializers the property wrapper provides. Formalizing the distinction can only help the compiler provide the programmer with more tools to understand code that uses such property wrappers. + +### Pass a property-wrapper storage instance directly + +A previous revision of this proposal supported passing a property-wrapper storage instance to a function with a wrapped parameter directly because the function type was in terms of the property-wrapper type. A big point of criticism during the first review was that the backing storage type should be an artifact of the function implementation, and not exposed to function callers through the type system. + +Exposing the property-wrapper storage type through the type system has the following implications, summarized by [Frederick Kellison-Linn](https://forums.swift.org/u/jumhyn): + +* The addition/removal of a property-wrapper attribute on a function parameter is a source-breaking change for any code that references or curries the function. +* It prohibits the use of initializer arguments in the wrapper attribute. There's no point in declaring a wrapper as `@Asserted(.greaterOrEqual(1))` if any client can simply pass an `Asserted` instance with a completely different validation. +* It removes API control from both the property wrapper author and the author of the wrapped-argument function. + +Keeping the property-wrapper storage type private is consistent with how property wrappers work today. Unless a property wrapper projects its storage type via `projectedValue`, the storage type itself is meant to be a private implementation detail inaccessible to API clients. + +## Future directions + +### The impact of formalizing separate property wrapper models + +The design of this property wrapper extension includes a formalized distinction between property wrappers that are implementation detail and property wrappers that are API. These two kinds of wrappers will need to be modeled differently in certain places in the language. This section explores the impact that introducing two separate models for property wrappers will have on the language and the future design space for property wrappers. + +The property wrapper model inside the declaration context of the wrapped property will remain the same between these two kinds of property wrappers. Whether the property wrapper is API or implementation detail, the auxiliary declaration model is fundamental to programmers' understanding of how property wrappers work and how to use them, and this model should not be changed in any future enhancement to the property wrapper feature. Property wrappers are and will always be syntactic sugar for code that the programmer can write manually using exactly the strategy that the compiler uses — auxiliary variables and custom accessors on the wrapped property. Any enhancements to property wrappers that add capabilities to the auxiliary declarations, such as access to the enclosing `self` instance or delegating to an existing stored property, will not be impacted by the API versus implementation detail distinction. + +The distinction of API versus implementation detail _will_ have an impact outside of the enclosing context of the wrapped declaration. Conceptually, the API versus implementation-detail distinction should only impact the parts of the language where an abstraction that contains a property wrapper attribute is used. + +Across module boundaries, implementation-detail property wrappers become invisible, because these wrappers are purely a detail of how the module is implemented. Clients have no knowledge of these wrappers, so property wrapper attributes that appear in the module must be API property wrappers. + +The modeling difference between implementation-detail and API property wrappers is only observable when both are used within the same module, and the difference is mainly observable in the language restrictions on the use of API versus implementation-detail wrappers. These two models are designed such that nearly all observable semantics of property wrapper application do not differ based on where the wrapper is applied. The only observable semantic difference that the proposal authors can think of is evaluation order among property wrapper initialization and other arguments that are passed to the API, and the proposal authors believe it is extremely unlikely that this evaluation order will have any impact on the functionality of the code. For evaluation order to have a functional impact, both the property wrapper initializer _and_ another function argument would both need to call into a separate function that has some side effect. For example: + +```swift +func hasSideEffect() -> Int { + struct S { + static var state = 0 + } + + S.state += 1 + return S.state +} + +@propertyWrapper +struct Wrapper { + var wrappedValue: Int + + init(wrappedValue: Int) { + self.wrappedValue = wrappedValue + hasSideEffect() + } +} + +func demonstrateEvaluationOrder(@Wrapper arg1: Int, arg2: Int) { + print(arg1, arg2) +} + +demonstrateEvaluationOrder(arg1: 1, arg2: hasSideEffect()) +``` + +If the property wrapper initializer is evaluated in the caller, the output of this code is `2, 2`. If the property wrapper initializer is evaluated in the callee, the output of the code is `3, 1`. + +The proposal authors believe that these two kinds of property wrappers already exist, and formalizing the distinction is a first step in enhancing programmers' understanding of such a complex feature. Property wrappers are very flexible to cover a wide variety of use cases. Formalizing the two broad categories of use cases opens up many interesting possibilities for the language and compiler to enhance library documentation when API wrappers are used, provide better guidance to programmers, and even allow library authors to augment the guidance given to programmers on invalid code through, for example, library-defined diagnostic notes. + +### Explicit spelling for API-level property wrappers + +The scope of what is considered an API-level property wrapper is very limited in this proposal, and the external effect of an API-level property wrapper may be useful for wrappers that don't fit the current definition. The `@propertyWrapper` attribute could have an explicit `apiLevel` option that allows library authors to define whether the property wrapper has an external effect on the wrapped declaration: + +```swift +@propertyWrapper(apiLevel) +struct Asserted { + init( + wrappedValue: Value, + _ assertion: (Value) -> Bool, + file: StaticString = #file, + line: UInt = #line + ) { ... } +} +``` + +### Generalized property-wrapper initialization from a projection + +This proposal adds `init(projectedValue:)` as a new property-wrapper initialization mechanism for function parameters. This mechanism could also be used to support initialization from a projected value for properties and local variables via [definite initialization](https://en.wikipedia.org/wiki/Definite_assignment_analysis): + +```swift +struct TextEditor { + @Traceable var dataSource: String + + init(history: History) { + // treated as _dataSource = Traceable(projectedValue: history) + $dataSource = history + } +} +``` + +### Static property-wrapper attribute arguments + +This proposal does not allow API-level property wrappers to have arguments in the wrapper attribute to ensure that these arguments remain the same across the different initialization mechanisms. Instead of passing these arguments to the property-wrapper initializer, property wrappers could opt into storing these arguments in per-wrapped-declaration static storage that is shared across property-wrapper instances. Consider the following example, inspired by [ValidatedPropertyKit](https://github.com/SvenTiigi/ValidatedPropertyKit): + +```swift +@propertyWrapper(sharedInfo: Validation) +struct Asserted { + struct Validation { + private let predicate: (Value) -> Bool + + init(predicate: @escaping (Value) -> Bool) { + self.predicate = predicate + } + + init(_ validation: Validation) { + self.predicate = validation.predicate + } + + static func greaterOrEqual(_ value: Value) -> Self { + .init { $0 >= value } + } + } + + init(wrappedValue: Value) { ... } + + // This is the 'wrappedValue' + subscript(sharedInfo: Validation) -> Value { + get { ... } + set { ... } + } +} +``` + +When `Asserted` is applied as a property wrapper, the arguments to the wrapper attribute become arguments to the `Validation` initializer, which would have static storage that is shared across each instance of the `Asserted` property wrapper in the following struct: + +```swift +struct S { + @Asserted(.greaterOrEqual(1)) var value: Int = 10 +} + +// translated to --> + +struct S { + private static let _value$sharedInfo: Asserted.Validation + = .init(.greaterOrEqual(1)) + + private var _value: Asserted + = .init(wrappedValue: 10) + + var value: Int { + get { _value[sharedInfo: _value$sharedInfo] } + set { _value[sharedInfo: _value$sharedInfo] = newValue } + } +} +``` + +This static storage mechanism would eliminate a lot of unnecessary storage in property wrapper instances. It would also allow API property wrappers on parameters to have attribute arguments, because those arguments are guaranteed to never change regardless of how the property wrapper is initialized. + +### API property wrappers in protocol requirements + +Protocol requirements that include property wrappers was [pitched](https://forums.swift.org/t/property-wrapper-requirements-in-protocols/33953) a while ago, but there was a lot of disagreement about whether property wrappers are implementation detail or API. With this distinction formalized, we could allow only API-level property wrappers in protocol requirements. + +### Extend property wrappers to patterns + +Passing a property-wrapper storage instance directly to a property-wrapped closure parameter was supported in first revision. One suggestion from the core team was to imagine this functionality as an orthogonal feature to allow pattern matching to "unwrap" property wrappers. Though this proposal revised the design of closures to match the behavior of unapplied function references, extending property wrappers to all patterns is still a viable future direction. + +Enabling the application of property wrappers in value-binding patterns would facilitate using the intuitive property-wrapper syntax in more language constructs, as shown below: + +```swift +enum Review { + case revised(Traceable) + case original(String) +} + +switch Review(fromUser: "swiftUser5") { +case .revised(@Traceable let reviewText), + .original(let reviewText): + // do something with 'reviewText' +} +``` + +### Support `inout` in wrapped function parameters + +This proposal doesn't currently support marking property-wrapped function parameters `inout`. We deemed that this functionality would be better tackled by another proposal, due to its implementation complexity. Nonetheless, this would be useful for mutating a wrapped parameter with the changes written back to the argument that was passed. + +## Revisions + +### Changes from the second reviewed version + +* The distinction between API wrappers and implementation-detail wrappers is formalized, and determined by the compiler based on whether the property wrapper type allows the call-site to pass a different type of argument. +* Implementation-detail property wrappers on parameters use callee-side application of the property wrapper, and have no external effect on the function. +* API property wrappers on parameters use caller-side application of the property wrapper, and are part of the function signature. +* Overload resolution for property wrapper initializers will always be done at the property wrapper declaration. + +### Changes from the first reviewed version + +* Passing a projected value using the `$` calling syntax is supported via `init(projectedValue:)`. +* The type of the unapplied function reference uses the wrapped-value type by default. Referencing the function using the projected-value type is supported by writing `$` in front of the argument label, or by writing `$_` if there is no argument label. +* Closures with property-wrapper parameters have the same semantics as unapplied function references. +* Additional arguments in the wrapper attribute are supported, and these arguments have the same evaluation semantics as default function arguments. + +## Appendix + +#### Mutability of composed `wrappedValue` accessors + +The algorithm is computing the mutability of the synthesized accessors for a wrapped parameter (or property) with _N_ attached property wrapper attributes. Attribute 1 is the outermost attribute, and attribute _N_ is the innermost. The accessor mutability is the same as the mutability of the _N_ th .wrappedValue access, e.g. _param.wrappedValue1.wrappedValue2. [...] .wrappedValueN + +The mutability of the _N_ th access is defined as follows: + +* If _N = 1_, the mutability of the access is the same as the mutability of the wrappedValue accessor in the 1st property wrapper. +Otherwise: + * If the wrappedValue accessor in the _N_ th property wrapper is nonmutating, then the _N_ th access has the same mutability as the _N - 1_ th get access. + * If the wrappedValue accessor in the _N_ th property wrapper is mutating, then the _N_ th access is mutating if the _N - 1_ th get or set access is mutating. + +**Example**: Consider the following `Reference` property wrapper, which is composed with `Logged` and used on a function parameter: + +```swift +@propertyWrapper +struct Reference { + var wrappedValue: Value { + get { ... } + nonmutating set { ... } + } + var projectedValue: Reference { + self + } +} + +func useReference(@Reference @Logged reference: String) { + ... +} +``` + +In the above example, the function `useReference` is equivalent to: + +```swift +func useReference(reference _reference: Reference>) { + var reference: String { + get { + _reference.wrappedValue.wrappedValue + } + set { + _reference.wrappedValue.wrappedValue = newValue + } + } + + var $reference: Reference> { + get { + _reference.projectedValue + } + } + + ... +} +``` + +Since both the getter and setter of `Reference.wrappedValue` are `nonmutating`, a setter can be synthesized for `var reference`, even though `Logged.wrappedValue` has a `mutating` setter. `Reference` also defines a `projectedValue` property, so a local computed property called `$reference` is synthesized in the function body, but it does _not_ have a setter, because `Reference.projectedValue` only defines a getter. + +## Acknowledgements + +This proposal was greatly improved as a direct result of feedback from the community. [Doug Gregor](https://forums.swift.org/u/douglas_gregor) and [Dave Abrahams](https://forums.swift.org/u/dabrahams) surfaced more use cases for property-wrapper parameters. [Frederick Kellison-Linn](https://forums.swift.org/u/jumhyn) proposed the idea to change the behavior of unapplied function references based on argument labels, and provided [ample justification](#passing-a-property-wrapper-storage-instance-directly) for why the semantics in the first revision were unintuitive. [Lantua](https://forums.swift.org/u/lantua) pushed for the behavior of closures to be consistent with that of functions, and proposed the idea to use `$` on closure parameters in cases where the wrapper attribute is unnecessary. Finally, ideas from [Jens Jakob Jensen](https://forums.swift.org/u/jjj) and [John McCall](https://forums.swift.org/u/john_mccall) were combined to produce the 'inference of external property wrapper' design in its current form. + +Many others participated throughout the several pitches and reviews. This feature would not be where it is today without the thoughtful contributions from folks across our community. diff --git a/proposals/0294-package-executable-targets.md b/proposals/0294-package-executable-targets.md new file mode 100644 index 0000000000..31aa5593b7 --- /dev/null +++ b/proposals/0294-package-executable-targets.md @@ -0,0 +1,200 @@ +# Declaring executable targets in Package Manifests + +* Proposal: [SE-0294](0294-package-executable-targets.md) +* Authors: [Anders Bertelrud](https://github.com/abertelrud) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.4)** +* Implementation: [apple/swift-package-manager#3045](https://github.com/apple/swift-package-manager/pull/3045) +* Bugs: [SR-13924](https://bugs.swift.org/browse/SR-13924) +* Pitch: [Forum Discussion](https://forums.swift.org/t/pitch-ability-to-declare-executable-targets-in-swiftpm-manifests-to-support-main/) +* Review: [Forum Discussion](https://forums.swift.org/t/se-0294-declaring-executable-targets-in-package-manifests/) + +## Introduction + +This proposal lets Swift Package authors declare targets as executable in the +package manifest. This replaces the current approach of inferring executability +based on the presence of a source file with the base name `main` at the top +level of the target source directory. + +Letting package authors declare targets as executable allows the use of `@main` +in Swift package targets. It also allows for better diagnostics, since the +purpose of the target is unambiguous even if source files are moved or renamed. + +## Motivation + +The Swift Package Manager doesn’t currently provide a way for a package manifest +to declare that a target provides the main module for an executable. Instead, +SwiftPM infers this by looking for a compilable source file with a base name of +`main` at the top level of the target directory. + +It is important to know unambiguously whether or not a target is intended to be +executable, because it affects the flags that are passed to the compiler at +build time. It also affects the quality of diagnostics, such as detecting +product declarations that mistakenly include more or less than a single +executable target in an executable product. + +Relying on specially named source files also doesn’t work when using `@main` to +specify the entry point of an executable. In addition, there are ergonomic +problems with using specially named source files (e.g. +[SR-1379](https://bugs.swift.org/browse/SR-1379)) that would be addressed by +being able to explicitly declare a target as being executable in the manifest. + +## Proposed solution + +The most straightforward approach is to allow a target to be marked as +executable in the manifest. This could take the form of either a parameter to +the existing `target` type, or a new target type. + +There is already an established pattern of using the type itself to denote the +kind of target being declared (e.g. `testTarget` as a specialization of +`target`), so this proposal suggests adding a new `executableTarget` type for +this purpose. + +Using a separate target type in the manifest would also support any future +differences in parameters between an executable target and a library target. +It would also be easier to read in a package manifest that includes a long +list of target declarations. + +## Detailed design + +The `PackageDescription` API is updated to add an `executableTarget` function, +currently having the same parameters as the `target` function: + +```swift +/// Creates an executable target. +/// +/// An executable target can contain either Swift or C-family source files, but not both. It contains code that +/// is built as an executable module that can be used as the main target of an executable product. The target +/// is expected to either have a source file named `main.swift`, `main.m`, `main.c`, or `main.cpp`, or a source +/// file that contains the `@main` keyword. +/// +/// - Parameters: +/// - name: The name of the target. +/// - dependencies: The dependencies of the target. A dependency can be another target in the package or a product from a package dependency. +/// - path: The custom path for the target. By default, the Swift Package Manager requires a target's sources to reside at predefined search paths; +/// for example, `[PackageRoot]/Sources/[TargetName]`. +/// Don't escape the package root; for example, values like `../Foo` or `/Foo` are invalid. +/// - exclude: A list of paths to files or directories that the Swift Package Manager shouldn't consider to be source or resource files. +/// A path is relative to the target's directory. +/// This parameter has precedence over the `sources` parameter. +/// - sources: An explicit list of source files. If you provide a path to a directory, +/// the Swift Package Manager searches for valid source files recursively. +/// - resources: An explicit list of resources files. +/// - publicHeadersPath: The directory containing public headers of a C-family library target. +/// - cSettings: The C settings for this target. +/// - cxxSettings: The C++ settings for this target. +/// - swiftSettings: The Swift settings for this target. +/// - linkerSettings: The linker settings for this target. +@available(_PackageDescription, introduced: 999.0) +public static func executableTarget( + name: String, + dependencies: [Dependency] = [], + path: String? = nil, + exclude: [String] = [], + sources: [String]? = nil, + resources: [Resource]? = nil, + publicHeadersPath: String? = nil, + cSettings: [CSetting]? = nil, + cxxSettings: [CXXSetting]? = nil, + swiftSettings: [SwiftSetting]? = nil, + linkerSettings: [LinkerSetting]? = nil +) -> Target { + return Target( + name: name, + dependencies: dependencies, + path: path, + exclude: exclude, + sources: sources, + resources: resources, + publicHeadersPath: publicHeadersPath, + type: .executable, + cSettings: cSettings, + cxxSettings: cxxSettings, + swiftSettings: swiftSettings, + linkerSettings: linkerSettings + ) +} +``` + +A new `.executable` case is also added to the `TargetType` enum in the +`PackageDescription` API. + +These are the only API changes that are visible to package manifest authors. + +On the implementation side, this proposal also updates the logic in SwiftPM +so that: + +- if the package tools-version is less than 5.4, a target is considered to be + executable in exactly the same way as in versions of SwiftPM prior to 5.4 +- if the package tools-version is greater than or equal to 5.4, then: + - if the target is declared using `.executableTarget`, then it is considered + to be an executable target + - if the target is declared using `.target`, then source files are examined + using the same rules as prior to 5.4, and a warning suggesting that the + target be declared using `.executableTarget` is emitted if the target is + considered executable under those rules + +The effect of this logic is that, starting with SwiftPM tools-version 5.4, +declaring an executable target by using `.target` and having it inferred to be +an executable by the presence of a source file with a base name of `main` is +considered deprecated but still works. + +This approach eases the transition for any package that adopts tools-version +5.4, and provides better diagnostics to change the declarations of executable +targets to use `.executableTarget`. If technically feasible, the warning +should have a fix-it to change the associated `.target` declaration to +`.executableTarget`. + +A future tools-version will remove the warning and treat any target that is +declared using `.target` as a library target. + +SwiftPM already passes different flags when compiling executable targets and +library targets, and that remains unchanged. In particular, SwiftPM passes +`-parse-as-library` when compiling a non-executable target, and will continue +to do so with these changes. It will continue to not pass this flag when +compiling executable targets, so that the compiler will continue to interpret +`main.swift` as the main source file of an executable. + +## Impact on existing packages + +There is no impact on existing packages. Packages that specify a tools-version +of 5.4 or greater will get the new behavior. Any package graph can contain a +mixture of old and new packages, and each target's executability will be +determined according to the tools-version of the package that declares it. + +As with any Swift module, the Swift compiler will not let an executable have +multiple `@main` entry points, including one that is specified by having a +`main.swift` source file in the target. + +## Alternatives considered + +There are various other ways that could have been considered for designating +a target as executable: + +#### As a new parameter on `target` + +Target declaration functions already have many parameters, and this would +exacerbate that problem. It would also make it difficult to syntactically +distinguish between executable targets and library targets in the package +manifest. It would also make it difficult to add future parameters that +applied to only library targets or executable targets. + +#### Through some other designation outside the manifest + +Since the intended executability of a target is a fundamental statement of +purpose, it seems logical that this should be denoted in the package manifest. +The Swift package manifest provides all the information other than what can be +inferred from the file system. + +## Future directions + +It is somewhat redundant that a package having only a single executable target +that builds a single executable product will now have both a product and target +with the word "executable" in its type. This is mitigated by the existing Swift +Package Manager behavior of implicitly creating an executable product for any +executable target if there isn't already a product with the same name (this +behavior remains unchanged by this proposal). + +Since a future goal is to find a way to unify product and target declarations +in the package manifest, the redundant appearance of the word "executable" in +both product and target type names is expected to be only a short-term issue. diff --git a/proposals/0295-codable-synthesis-for-enums-with-associated-values.md b/proposals/0295-codable-synthesis-for-enums-with-associated-values.md new file mode 100644 index 0000000000..599e2db532 --- /dev/null +++ b/proposals/0295-codable-synthesis-for-enums-with-associated-values.md @@ -0,0 +1,412 @@ +# Codable synthesis for enums with associated values + +* Proposal: [SE-0295](0295-codable-synthesis-for-enums-with-associated-values.md) +* Authors: [Dario Rexin](https://github.com/drexin) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.5)** +* Implementation: [apple/swift#34855](https://github.com/apple/swift/pull/34855) +* Pitch: [Forum Discussion](https://forums.swift.org/t/codable-synthesis-for-enums-with-associated-values/41493) +* Previous Review: [Forum Discussion](https://forums.swift.org/t/se-0295-codable-synthesis-for-enums-with-associated-values/42408) +* Previous Review 2: [Forum Discussion](https://forums.swift.org/t/se-0295-codable-synthesis-for-enums-with-associated-values-second-review/44036) + +## Introduction + +Codable was introduced in [SE-0166](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0166-swift-archival-serialization.md) +with support for synthesizing `Encodable` and `Decodable` conformance for +`class` and `struct` types, that only contain values that also conform +to the respective protocols. + +This proposal will extend the support for auto-synthesis of these conformances +to enums with associated values. + +## Motivation + +Currently auto-synthesis only works for enums conforming to `RawRepresentable`. +There have been discussions about adding general support for enums in the past, +but the concrete structure of the encoded values was never agreed upon. +We believe that having a solution for this is an important quality of life +improvement. + +## Proposed solution + +There are two models of evolution, the one designated by the language and one that is useful in the regular use of enumerations. This proposal subsets the supported cases of automatic synthesis of Codable where the two models align. We believe this to be important to retain flexibility for the user to change the shape of the enumeration. + +### Structure of encoded enums + +The following enum with associated values + +```swift +enum Command: Codable { + case load(key: String) + case store(key: String, value: Int) +} +``` + +would be encoded to + +```json +{ + "load": { + "key": "MyKey" + } +} +``` + +and + +```json +{ + "store": { + "key": "MyKey", + "value": 42 + } +} +``` + +The top-level container contains a single key that matches the name of the enum case, +which points to another container that contains the values as they would be encoded +for structs and classes. + +Associated values can also be unlabeled, in which case an identifier will be generated in the form of `_$N`, where `$N` is the 0-based position of the parameter. Using generated identifiers allows more flexibility in evolution of models than using an `UnkeyedContainer` would. If a user defined parameter has an identifier that conflicts with a generated identifier, the compiler will produce a diagnostic message. + +```swift +enum Command: Codable { + case load(String) + case store(key: String, Int) +} +``` + +would encoded to + +```json +{ + "load": { + "_0": "MyKey" + } +} +``` + +and + +```json +{ + "store": { + "key": "MyKey", + "_1": 42 + } +} +``` + +An enum case without associated values would be encoded as an empty `KeyedContainer`, +i.e. + +```swift +enum Command: Codable { + case dumpToDisk +} +``` + +would encode to: + +```json +{ + "dumpToDisk": {} +} +``` + +This allows these cases to evolve in the same manner as cases with associated values, without breaking compatibility. + +### Synthesized code + +Given that enums are encoded into a nested structure, there are multiple `CodingKeys` declarations. One +that contains the keys for each of the enum cases, which as before is called `CodingKeys`, and one for each case that contain the keys for the +associated values, that are prefixed with the capilized case name, e.g. `LoadCodingKeys` for `case load`. + +```swift +enum Command: Codable { + case load(key: String) + case store(key: String, value: Int) +} +``` + +Would have the compiler generate the following `CodingKeys` declarations: + +```swift + +// contains keys for all cases of the enum +enum CodingKeys: CodingKey { + case load + case store +} + +// contains keys for all associated values of `case load` +enum LoadCodingKeys: CodingKey { + case key +} + +// contains keys for all associated values of `case store` +enum StoreCodingKeys: CodingKey { + case key + case value +} +``` + +The `encode(to:)` implementation would look as follows: + +```swift +public func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + switch self { + case let .load(key): + var nestedContainer = container.nestedContainer(keyedBy: LoadCodingKeys.self, forKey: .load) + try nestedContainer.encode(key, forKey: .key) + case let .store(key, value): + var nestedContainer = container.nestedContainer(keyedBy: StoreCodingKeys.self, forKey: .store) + try nestedContainer.encode(key, forKey: .key) + try nestedContainer.encode(value, forKey: .value) + } +} +``` + +and the `init(from:)` implementation would look as follows: + +```swift +public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + if container.allKeys.count != 1 { + let context = DecodingError.Context( + codingPath: container.codingPath, + debugDescription: "Invalid number of keys found, expected one.") + throw DecodingError.typeMismatch(Command.self, context) + } + + switch container.allKeys.first.unsafelyUnwrapped { + case .load: + let nestedContainer = try container.nestedContainer(keyedBy: LoadCodingKeys.self, forKey: .load) + self = .load( + key: try nestedContainer.decode(String.self, forKey: .key)) + case .store: + let nestedContainer = try container.nestedContainer(keyedBy: StoreCodingKeys.self, forKey: .store) + self = .store( + key: try nestedContainer.decode(String.self, forKey: .key), + value: try nestedContainer.decode(Int.self, forKey: .value)) + } +} +``` + +### User customization + +For the existing cases, users can customize which properties are included in the encoded representation +and map the property name to a custom name for the encoded representation by providing a custom `CodingKeys` +declaration, instead of having the compiler generate one. The same should apply to the enum case. + +Users can define custom `CodingKeys` declarations for all, or a subset +of the cases. If some of the cases in an enum should not be codable, +they can be excluded from the `CodingKeys` declaration. + +**Example** + +```swift +enum Command: Codable { + case load(key: String) + case store(key: String, value: Int) + case dumpToDisk + + enum CodingKeys: CodingKey { + case load + case store + // don't include `dumpToDisk` + } +} +``` + +The compiler will now only synthesize the code for the `load` and `store` +cases. An attempt to en- or decode a `dumpToDisk` value will cause an error +to be thrown. + +Customizing which values will be included follows the same rules as the +existing functionality. Values that are excluded must have a default value +defined, if a `Decodable` conformance should be synthesized. If only `Encodable` +is synthesized, this restriction does not apply. + +**Example** + +```swift +enum Command: Codable { + case load(key: String, someLocalInfo: Int) + + // invalid, because `someLocalInfo` has no default value + enum LoadCodingKeys: CodingKey { + case key + } +} +``` + +```swift +enum Command: Codable { + case load(key: String, someLocalInfo: Int = 0) + + // valid, because `someLocalInfo` has a default value + enum LoadCodingKeys: CodingKey { + case key + } +} +``` + +```swift +enum Command: Codable { + case load(key: String) + + // invalid, because `someUnknownKey` does not map to a parameter in `load` + enum LoadCodingKeys: CodingKey { + case key + case someUnknownKey + } +} +``` + +Keys can be mapped to other names by conforming to `RawRepresentable`: + +**Example** + +```swift +enum Command: Codable { + case load(key: String) + case store(key: String, Int) + + enum CodingKeys: String, CodingKey { + case load = "lade" + } + + enum LoadCodingKeys: String, CodingKey { + case key = "schluessel" + } +} +``` + +would encode to: + +```json +{ + "lade": { + "schluessel": "MyKey" + } +} +``` + +### Evolution and compatibility + +Enum cases can evolve in the same way as structs and classes. Adding new fields, or removing existing ones is compatible, as long as the values are optional and the identifiers for the other cases don't change. This is in opposition to the evolution model of the language, where adding or removing associated values is a source and binary breaking change. We believe that a lot of use cases benefit from the ability to evolve the model, where source and binary compatibility are not an issue, e.g. in applications, services, or for internal types. If binary compatibility is important, evolution can be supported by having a single struct or class with all the parameters as the associated value. + +### Unsupported cases + +This proposal specifically does not support auto-synthesis for enums with overloaded case identifiers. This decision has been made because there is no clear way to support the feature, while also allowing the model to evolve, without severe restrictions. The separate cases in an enum typically have different semantics associated with them, so it is crucial to be able to properly identify the different cases and reject unknown cases. + +#### Evolution with overloaded case identifiers + +In this proposal, we are using keys as descriminators. For overloaded case names that would not be sufficient to identify the different overloads. An alternative would be to use the full name, including labels, e.g. `"store(key:,value:)"` for `case store(key: String, value: Int)`. This leads to several problems. + +1. Not a valid enum case identifier, so no user customization possible +2. Not forward/backward compatible because it changes when parameters are added + +An alternative solution would be to match the keys against the parameter names when decoding an object. This approach also has issues. Overloads can share parameter names, so a message that contains additional keys, that the current code version does not know about, would cause ambiguity. + +**Example** + +```swift +enum Test: Codable { + case x(y: Int) +} +``` + +```json +{ + "x": { + "y": 42, + "z": "test" + } +} +``` + +This could either mean that a parameter has been added to the existing case in a new code version, which should be ok under backwards compatibility rules, but could also mean that an overload has been added that partially matches the original case. If it is a different case, we should reject the message, but there is no way for the old code to decide this. + +#### Ambiguities without evolution + +Even when ignoring evolution, ambiguities exist with overloaded case identifiers. If two overloads share the same parameter names and one of them has additional optional parameters, the encoder can decide to drop `nil` values. This would cause ambiguities when only the shared keys are present. + +**Example** + +```swift +enum Test: Codable { + case x(y: Int) + case x(y: Int, z: String?) +} +``` + +Both cases would match the following input: + +```json +{ + "x": { + "y": 42 + } +} +``` + +Another ambiguity can be created by using the same parameter names for two overloads, only in a different order. Some formats do not guarantee the ordering of the keys, so the following definition leads to ambiguity when decoding: + +```swift +enum Test: Codable { + case x(y: Int, z: String) + case x(z: String, y: Int) +} +``` + +Both cases would match the following input: + +```json +{ + "x" : { + "y": 42, + "z": "test" + } +} +``` + +## Source compatibility + +Existing source is not affected by these changes. + +## Effect on ABI stability + +None + +## Effect on API resilience + +None + +## Alternatives considered + +Previous discussions in the forums have been considered, specifically separating +the discriminator and value into individual key/value pairs discussed in [this forum thread](https://forums.swift.org/t/automatic-codable-conformance-for-enums-with-associated-values-that-themselves-conform-to-codable/11499). +While we do believe that there is value in doing this, we think that the default +behavior should more closely follow the structure of the types that are encoded. +A future proposal could add customization options to change the structure to meet +individual requirements. + +### Parameterless cases + +Alternative ways to encode these cases have been discussed and the following problems have been found: + +1. Encode as plain values, as we do today with `RawRepresentable` + +The problem with this is that `Decoder` does not have APIs to check which type of container is present and adding such APIs would be a breaking change. The existing APIs to read containers are `throws`, but it is not specified in which cases an error should occur. + +2. Encode as `nil` + +This representation is problematic, because `Encoder` implementations can decide to drop `nil` values, which would also mean losing the key and with it the ability to identify the case. + + +## Acknowledgements + +While iterating on this proposal, a lot of inspiration was drawn from the Rust library [serde](https://serde.rs/container-attrs.html). diff --git a/proposals/0296-async-await.md b/proposals/0296-async-await.md new file mode 100644 index 0000000000..6c32790ac6 --- /dev/null +++ b/proposals/0296-async-await.md @@ -0,0 +1,770 @@ +# Async/await + +* Proposal: [SE-0296](0296-async-await.md) +* Authors: [John McCall](https://github.com/rjmccall), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.5)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modification-se-0296-async-await/43318), [Amendment to allow overloading on `async`](https://forums.swift.org/t/accepted-amendment-to-se-0296-allow-overloads-that-differ-only-in-async/50117) + +## Table of Contents + + * [Async/await](#asyncawait) + * [Introduction](#introduction) + * [Motivation: Completion handlers are suboptimal](#motivation-completion-handlers-are-suboptimal) + * [Proposed solution: async/await](#proposed-solution-asyncawait) + * [Suspension points](#suspension-points) + * [Detailed design](#detailed-design) + * [Asynchronous functions](#asynchronous-functions) + * [Asynchronous function types](#asynchronous-function-types) + * [Await expressions](#await-expressions) + * [Closures](#closures) + * [Overloading and overload resolution](#overloading-and-overload-resolution) + * [Autoclosures](#autoclosures) + * [Protocol conformance](#protocol-conformance) + * [Source compatibility](#source-compatibility) + * [Effect on ABI stability](#effect-on-abi-stability) + * [Effect on API resilience](#effect-on-api-resilience) + * [Future Directions](#future-directions) + * [reasync](#reasync) + * [Alternatives Considered](#alternatives-considered) + * [Make await imply try](#make-await-imply-try) + * [Launching async tasks](#launching-async-tasks) + * [Await as syntactic sugar](#await-as-syntactic-sugar) + * [Revision history](#revision-history) + * [Related proposals](#related-proposals) + * [Acknowledgments](#acknowledgments) + +## Introduction + +Modern Swift development involves a lot of asynchronous (or "async") programming using closures and completion handlers, but these APIs are hard to use. This gets particularly problematic when many asynchronous operations are used, error handling is required, or control flow between asynchronous calls gets complicated. This proposal describes a language extension to make this a lot more natural and less error prone. + +This design introduces a [coroutine model](https://en.wikipedia.org/wiki/Coroutine) to Swift. Functions can opt into being `async`, allowing the programmer to compose complex logic involving asynchronous operations using the normal control-flow mechanisms. The compiler is responsible for translating an asynchronous function into an appropriate set of closures and state machines. + +This proposal defines the semantics of asynchronous functions. However, it does not provide concurrency: that is covered by a separate proposal to introduce structured concurrency, which associates asynchronous functions with concurrently-executing tasks and provides APIs for creating, querying, and cancelling tasks. + +Swift-evolution thread: [Pitch #1](https://forums.swift.org/t/concurrency-asynchronous-functions/41619), [Pitch #2](https://forums.swift.org/t/pitch-2-async-await/42420) + +## Motivation: Completion handlers are suboptimal + +Async programming with explicit callbacks (also called completion handlers) has many problems, which we’ll explore below. We propose to address these problems by introducing async functions into the language. Async functions allow asynchronous code to be written as straight-line code. They also allow the implementation to directly reason about the execution pattern of the code, allowing callbacks to run far more efficiently. + +#### Problem 1: Pyramid of doom + +A sequence of simple asynchronous operations often requires deeply-nested closures. Here is a made-up example showing this: + +```swift +func processImageData1(completionBlock: (_ result: Image) -> Void) { + loadWebResource("dataprofile.txt") { dataResource in + loadWebResource("imagedata.dat") { imageResource in + decodeImage(dataResource, imageResource) { imageTmp in + dewarpAndCleanupImage(imageTmp) { imageResult in + completionBlock(imageResult) + } + } + } + } +} + +processImageData1 { image in + display(image) +} +``` + +This "pyramid of doom" makes it difficult to read and keep track of where the code is running. In addition, having to use a stack of closures leads to many second order effects that we will discuss next. + +#### Problem 2: Error handling + +Callbacks make error handling difficult and very verbose. Swift 2 introduced an error handling model for synchronous code, but callback-based interfaces do not derive any benefit from it: + +```swift +// (2a) Using a `guard` statement for each callback: +func processImageData2a(completionBlock: (_ result: Image?, _ error: Error?) -> Void) { + loadWebResource("dataprofile.txt") { dataResource, error in + guard let dataResource = dataResource else { + completionBlock(nil, error) + return + } + loadWebResource("imagedata.dat") { imageResource, error in + guard let imageResource = imageResource else { + completionBlock(nil, error) + return + } + decodeImage(dataResource, imageResource) { imageTmp, error in + guard let imageTmp = imageTmp else { + completionBlock(nil, error) + return + } + dewarpAndCleanupImage(imageTmp) { imageResult, error in + guard let imageResult = imageResult else { + completionBlock(nil, error) + return + } + completionBlock(imageResult) + } + } + } + } +} + +processImageData2a { image, error in + guard let image = image else { + display("No image today", error) + return + } + display(image) +} +``` + +The addition of [`Result`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0235-add-result.md) to the standard library improved on error handling for Swift APIs. Asynchronous APIs were one of the [main motivators](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0235-add-result.md#asynchronous-apis) for `Result`: + +```swift +// (2b) Using a `do-catch` statement for each callback: +func processImageData2b(completionBlock: (Result) -> Void) { + loadWebResource("dataprofile.txt") { dataResourceResult in + do { + let dataResource = try dataResourceResult.get() + loadWebResource("imagedata.dat") { imageResourceResult in + do { + let imageResource = try imageResourceResult.get() + decodeImage(dataResource, imageResource) { imageTmpResult in + do { + let imageTmp = try imageTmpResult.get() + dewarpAndCleanupImage(imageTmp) { imageResult in + completionBlock(imageResult) + } + } catch { + completionBlock(.failure(error)) + } + } + } catch { + completionBlock(.failure(error)) + } + } + } catch { + completionBlock(.failure(error)) + } + } +} + +processImageData2b { result in + do { + let image = try result.get() + display(image) + } catch { + display("No image today", error) + } +} +``` + +```swift +// (2c) Using a `switch` statement for each callback: +func processImageData2c(completionBlock: (Result) -> Void) { + loadWebResource("dataprofile.txt") { dataResourceResult in + switch dataResourceResult { + case .success(let dataResource): + loadWebResource("imagedata.dat") { imageResourceResult in + switch imageResourceResult { + case .success(let imageResource): + decodeImage(dataResource, imageResource) { imageTmpResult in + switch imageTmpResult { + case .success(let imageTmp): + dewarpAndCleanupImage(imageTmp) { imageResult in + completionBlock(imageResult) + } + case .failure(let error): + completionBlock(.failure(error)) + } + } + case .failure(let error): + completionBlock(.failure(error)) + } + } + case .failure(let error): + completionBlock(.failure(error)) + } + } +} + +processImageData2c { result in + switch result { + case .success(let image): + display(image) + case .failure(let error): + display("No image today", error) + } +} +``` + +It's easier to handle errors when using `Result`, but the closure-nesting problem remains. + +#### Problem 3: Conditional execution is hard and error-prone + +Conditionally executing an asynchronous function is a huge pain. For example, suppose we need to "swizzle" an image after obtaining it. But, we sometimes have to make an asynchronous call to decode the image before we can swizzle. Perhaps the best approach to structuring this function is to write the swizzling code in a helper "continuation" closure that is conditionally captured in a completion handler, like this: + +```swift +func processImageData3(recipient: Person, completionBlock: (_ result: Image) -> Void) { + let swizzle: (_ contents: Image) -> Void = { + // ... continuation closure that calls completionBlock eventually + } + if recipient.hasProfilePicture { + swizzle(recipient.profilePicture) + } else { + decodeImage { image in + swizzle(image) + } + } +} +``` + +This pattern inverts the natural top-down organization of the function: the code that will execute in the second half of the function must appear *before* the part that executes in the first half. In addition to restructuring the entire function, we must now think carefully about captures in the continuation closure, because the closure is used in a completion handler. The problem worsens as the number of conditionally-executed async functions grows, yielding what is essentially an inverted "pyramid of doom." + +#### Problem 4: Many mistakes are easy to make + +It's quite easy to bail-out of the asynchronous operation early by simply returning without calling the correct completion-handler block. When forgotten, the issue is very hard to debug: + +```swift +func processImageData4a(completionBlock: (_ result: Image?, _ error: Error?) -> Void) { + loadWebResource("dataprofile.txt") { dataResource, error in + guard let dataResource = dataResource else { + return // <- forgot to call the block + } + loadWebResource("imagedata.dat") { imageResource, error in + guard let imageResource = imageResource else { + return // <- forgot to call the block + } + ... + } + } +} +``` + +When you do remember to call the block, you can still forget to return after that: + +```swift +func processImageData4b(recipient:Person, completionBlock: (_ result: Image?, _ error: Error?) -> Void) { + if recipient.hasProfilePicture { + if let image = recipient.profilePicture { + completionBlock(image) // <- forgot to return after calling the block + } + } + ... +} +``` + +Thankfully the `guard` syntax protects against forgetting to return to some degree, but it's not always relevant. + +#### Problem 5: Because completion handlers are awkward, too many APIs are defined synchronously + +This is hard to quantify, but the authors believe that the awkwardness of defining and using asynchronous APIs (using completion handlers) has led to many APIs being defined with apparently synchronous behavior, even when they can block. This can lead to problematic performance and responsiveness problems in UI applications, e.g. a spinning cursor. It can also lead to the definition of APIs that cannot be used when asynchrony is critical to achieve scale, e.g. on the server. + +## Proposed solution: async/await + +Asynchronous functions—often known as async/await—allow asynchronous code to be written as if it were straight-line, synchronous code. This immediately addresses many of the problems described above by allowing programmers to make full use of the same language constructs that are available to synchronous code. The use of async/await also naturally preserves the semantic structure of the code, providing information necessary for at least three cross-cutting improvements to the language: (1) better performance for asynchronous code; (2) better tooling to provide a more consistent experience while debugging, profiling, and exploring code; and (3) a foundation for future concurrency features like task priority and cancellation. The example from the prior section demonstrates how async/await drastically simplifies asynchronous code: + +```swift +func loadWebResource(_ path: String) async throws -> Resource +func decodeImage(_ r1: Resource, _ r2: Resource) async throws -> Image +func dewarpAndCleanupImage(_ i : Image) async throws -> Image + +func processImageData() async throws -> Image { + let dataResource = try await loadWebResource("dataprofile.txt") + let imageResource = try await loadWebResource("imagedata.dat") + let imageTmp = try await decodeImage(dataResource, imageResource) + let imageResult = try await dewarpAndCleanupImage(imageTmp) + return imageResult +} +``` + +Many descriptions of async/await discuss it through a common implementation mechanism: a compiler pass which divides a function into multiple components. This is important at a low level of abstraction in order to understand how the machine is operating, but at a high level we’d like to encourage you to ignore it. Instead, think of an asynchronous function as an ordinary function that has the special power to give up its thread. Asynchronous functions don’t typically use this power directly; instead, they make calls, and sometimes these calls will require them to give up their thread and wait for something to happen. When that thing is complete, the function will resume executing again. + +The analogy with synchronous functions is very strong. A synchronous function can make a call; when it does, the function immediately waits for the call to complete. Once the call completes, control returns to the function and picks up where it left off. The same thing is true with an asynchronous function: it can make calls as usual; when it does, it (normally) immediately waits for the call to complete. Once the call completes, control returns to the function and it picks up where it was. The only difference is that synchronous functions get to take full advantage of (part of) their thread and its stack, whereas *asynchronous functions are able to completely give up that stack and use their own, separate storage*. This additional power given to asynchronous functions has some implementation cost, but we can reduce that quite a bit by designing holistically around it. + +Because asynchronous functions must be able to abandon their thread, and synchronous functions don’t know how to abandon a thread, a synchronous function can’t ordinarily call an asynchronous function: the asynchronous function would only be able to give up the part of the thread it occupied, and if it tried, its synchronous caller would treat it like a return and try to pick up where it was, only without a return value. The only way to make this work in general would be to block the entire thread until the asynchronous function was resumed and completed, and that would completely defeat the purpose of asynchronous functions, as well as having nasty systemic effects. + +In contrast, an asynchronous function can call either synchronous or asynchronous functions. While it’s calling a synchronous function, of course, it can’t give up its thread. In fact, asynchronous functions never just spontaneously give up their thread; they only give up their thread when they reach what’s called a suspension point. A suspension point can occur directly within a function, or it can occur within another asynchronous function that the function calls, but in either case the function and all of its asynchronous callers simultaneously abandon the thread. (In practice, asynchronous functions are compiled to not depend on the thread during an asynchronous call, so that only the innermost function needs to do any extra work.) + +When control returns to an asynchronous function, it picks up exactly where it was. That doesn’t necessarily mean that it’ll be running on the exact same thread it was before, because the language doesn’t guarantee that after a suspension. In this design, threads are mostly an implementation mechanism, not a part of the intended interface to concurrency. However, many asynchronous functions are not just asynchronous: they’re also associated with specific actors (which are the subject of a separate proposal), and they’re always supposed to run as part of that actor. Swift does guarantee that such functions will in fact return to their actor to finish executing. Accordingly, libraries that use threads directly for state isolation—for example, by creating their own threads and scheduling tasks sequentially onto them—should generally model those threads as actors in Swift in order to allow these basic language guarantees to function properly. + +### Suspension points + +A suspension point is a point in the execution of an asynchronous function where it has to give up its thread. Suspension points are always associated with some deterministic, syntactically explicit event in the function; they’re never hidden or asynchronous from the function’s perspective. The primary form of suspension point is a call to an asynchronous function associated with a different execution context. + +It is important that suspension points are only associated with explicit operations. In fact, it’s so important that this proposal requires that calls that *might* suspend be enclosed in an `await` expression. These calls are referred to as *potential suspension points*, because it is not known statically whether they will actually suspend: that depends both on code not visible at the call site (e.g., the callee might depend on asynchronous I/O) as well as dynamic conditions (e.g., whether that asynchronous I/O will have to wait to complete). + +The requirement for `await` on potential suspension points follows Swift's precedent of requiring `try` expressions to cover calls to functions that can throw errors. Marking potential suspension points is particularly important because *suspensions interrupt atomicity*. For example, if an asynchronous function is running within a given context that is protected by a serial queue, reaching a suspension point means that other code can be interleaved on that same serial queue. A classic but somewhat hackneyed example where this atomicity matters is modeling a bank: if a deposit is credited to one account, but the operation suspends before processing a matched withdrawal, it creates a window where those funds can be double-spent. A more germane example for many Swift programmers is a UI thread: the suspension points are the points where the UI can be shown to the user, so programs that build part of their UI and then suspend risk presenting a flickering, partially-constructed UI. (Note that suspension points are also called out explicitly in code using explicit callbacks: the suspension happens between the point where the outer function returns and the callback starts running.) Requiring that all potential suspension points are marked allows programmers to safely assume that places without potential suspension points will behave atomically, as well as to more easily recognize problematic non-atomic patterns. + +Because potential suspension points can only appear at points explicitly marked within an asynchronous function, long computations can still block threads. This might happen when calling a synchronous function that just does a lot of work, or when encountering a particularly intense computational loop written directly in an asynchronous function. In either case, the thread cannot interleave code while these computations are running, which is usually the right choice for correctness, but can also become a scalability problem. Asynchronous programs that need to do intense computation should generally run it in a separate context. When that’s not feasible, there will be library facilities to artificially suspend and allow other operations to be interleaved. + +Asynchronous functions should avoid calling functions that can actually block the thread, especially if they can block it waiting for work that’s not guaranteed to be currently running. For example, acquiring a mutex can only block until some currently-running thread gives up the mutex; this is sometimes acceptable but must be used carefully to avoid introducing deadlocks or artificial scalability problems. In contrast, waiting on a condition variable can block until some arbitrary other work gets scheduled that signals the variable; this pattern goes strongly against recommendation. + +## Detailed design + +### Asynchronous functions + +Function types can be marked explicitly as `async`, indicating that the function is asynchronous: + +```swift +func collect(function: () async -> Int) { ... } +``` + +A function or initializer declaration can also be declared explicitly as `async`: + +```swift +class Teacher { + init(hiringFrom: College) async throws { + ... + } + + private func raiseHand() async -> Bool { + ... + } +} +``` + +> **Rationale**: The `async` follows the parameter list because it is part of the function's type as well as its declaration. This follows the precedent of `throws`. + +The type of a reference to a function or initializer declared `async` is an `async` function type. If the reference is a “curried” static reference to an instance method, it is the "inner" function type that is `async`, consistent with the usual rules for such references. + +Special functions like `deinit` and storage accessors (i.e., the getters and setters for properties and subscripts) cannot be `async`. + +> **Rationale**: Properties and subscripts that only have a getter could potentially be `async`. However, properties and subscripts that also have an `async` setter imply the ability to pass the reference as `inout` and drill down into the properties of that property itself, which depends on the setter effectively being an "instantaneous" (synchronous, non-throwing) operation. Prohibiting `async` properties is a simpler rule than only allowing get-only `async` properties and subscripts. + +If a function is both `async` and `throws`, then the `async` keyword must precede `throws` in the type declaration. This same rule applies if `async` and `rethrows`. + +> **Rationale** : This order restriction is arbitrary, but it's not harmful, and it eliminates the potential for stylistic debates. + +An `async` initializer of a class that has a superclass but lacks a call to a superclass initializer will get an implicit call to `super.init()` only if the superclass has a zero-argument, synchronous, designated initializer. + +> **Rationale**: If the superclass initializer is `async`, the call to the asynchronous initializer is a potential suspension point and therefore the call (and required `await`) must be visible in the source. + +### Asynchronous function types + +Asynchronous function types are distinct from their synchronous counterparts. However, there is an implicit conversion from a synchronous function type to its corresponding asynchronous function type. This is similar to the implicit conversion from a non-throwing function to its throwing counterpart, which can also compose with the asynchronous function conversion. For example: + +```swift +struct FunctionTypes { + var syncNonThrowing: () -> Void + var syncThrowing: () throws -> Void + var asyncNonThrowing: () async -> Void + var asyncThrowing: () async throws -> Void + + mutating func demonstrateConversions() { + // Okay to add 'async' and/or 'throws' + asyncNonThrowing = syncNonThrowing + asyncThrowing = syncThrowing + syncThrowing = syncNonThrowing + asyncThrowing = asyncNonThrowing + + // Error to remove 'async' or 'throws' + syncNonThrowing = asyncNonThrowing // error + syncThrowing = asyncThrowing // error + syncNonThrowing = syncThrowing // error + asyncNonThrowing = syncThrowing // error + } +} +``` + +### Await expressions + +A call to a value of `async` function type (including a direct call to an `async` function) introduces a potential suspension point. +Any potential suspension point must occur within an asynchronous context (e.g., an `async` function). Furthermore, it must occur within the operand of an `await` expression. + +Consider the following example: + +```swift +// func redirectURL(for url: URL) async -> URL { ... } +// func dataTask(with: URL) async throws -> (Data, URLResponse) { ... } + +let newURL = await server.redirectURL(for: url) +let (data, response) = try await session.dataTask(with: newURL) +``` + +In this code example, a task suspension may happen during the calls to `redirectURL(for:)` and `dataTask(with:)` because they are async functions. Thus, both call expressions must be contained within some `await` expression, because each call contains a potential suspension point. An `await` operand may contain more than one potential suspension point. For example, we can use one `await` to cover both potential suspension points from our example by rewriting it as: + +```swift +let (data, response) = try await session.dataTask(with: server.redirectURL(for: url)) +``` + +The `await` has no additional semantics; like `try`, it merely marks that an asynchronous call is being made. The type of the `await` expression is the type of its operand, and its result is the result of its operand. +An `await` operand may also have no potential suspension points, which will result in a warning from the Swift compiler, following the precedent of `try` expressions: + +```swift +let x = await synchronous() // warning: no calls to 'async' functions occur within 'await' expression +``` + +> **Rationale**: It is important that asynchronous calls are clearly identifiable within the function because they may introduce suspension points, which break the atomicity of the operation. The suspension points may be inherent to the call (because the asynchronous call must execute on a different executor) or simply be part of the implementation of the callee, but in either case it is semantically important and the programmer needs to acknowledge it. `await` expressions are also an indicator of asynchronous code, which interacts with inference in closures; see the section on [Closures](#closures) for more information. + +A potential suspension point must not occur within an autoclosure that is not of `async` function type. + +A potential suspension point must not occur within a `defer` block. + +If both `await` and a variant of `try` (including `try!` and `try?`) are applied to the same subexpression, `await` must follow the `try`/`try!`/`try?`: + +```swift +let (data, response) = await try session.dataTask(with: server.redirectURL(for: url)) // error: must be `try await` +let (data, response) = await (try session.dataTask(with: server.redirectURL(for: url))) // okay due to parentheses +``` + +> **Rationale**: this restriction is arbitrary, but follows the equally-arbitrary restriction on the ordering of `async throws` in preventing stylistic debates. + +### Closures + +A closure can have `async` function type. Such closures can be explicitly marked as `async` as follows: + +```swift +{ () async -> Int in + print("here") + return await getInt() +} +``` + +An anonymous closure is inferred to have `async` function type if it contains an `await` expression. + +```swift +let closure = { await getInt() } // implicitly async + +let closure2 = { () -> Int in // implicitly async + print("here") + return await getInt() +} +``` + +Note that inference of `async` on a closure does not propagate to its enclosing or nested functions or closures, because those contexts are separably asynchronous or synchronous. For example, only `closure6` is inferred to be `async` in this situation: + +```swift +// func getInt() async -> Int { ... } + +let closure5 = { () -> Int in // not 'async' + let closure6 = { () -> Int in // implicitly async + if randomBool() { + print("there") + return await getInt() + } else { + let closure7 = { () -> Int in 7 } // not 'async' + return 0 + } + } + + print("here") + return 5 +} +``` + +### Overloading and overload resolution + +Existing Swift APIs generally support asynchronous functions via a callback interface, e.g., + +```swift +func doSomething(completionHandler: ((String) -> Void)? = nil) { ... } +``` + +Many such APIs are likely to be updated by adding an `async` form: + +```swift +func doSomething() async -> String { ... } +``` + +These two functions have different names and signatures, even though they share the same base name. However, either of them can be called with no parameters (due to the defaulted completion handler), which would present a problem for existing code: + +```swift +doSomething() // problem: can call either, unmodified Swift rules prefer the `async` version +``` + +A similar problem exists for APIs that evolve into providing both a synchronous and an asynchronous version of the same function, with the same signature. Such pairs allow APIs to provide a new asynchronous function which better fits in the Swift asynchronous landscape, without breaking backward compatibility. New asynchronous functions can support, for example, cancellation (covered in the [Structured Concurrency](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md) proposal). + +```swift +// Existing synchronous API +func doSomethingElse() { ... } + +// New and enhanced asynchronous API +func doSomethingElse() async { ... } +``` + +In the first case, Swift's overloading rules prefer to call a function with fewer default arguments, so the addition of the `async` function would break existing code that called the original `doSomething(completionHandler:)` with no completion handler. This would get an error along the lines of: + +``` +error: `async` function cannot be called from non-asynchronous context +``` + +This presents problems for code evolution, because developers of existing asynchronous libraries would have to either have a hard compatibility break (e.g, to a new major version) or would need have different names for all of the new `async` versions. The latter would likely result in a scheme such as [C#'s pervasive `Async` suffix](https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/async/task-asynchronous-programming-model). + +The second case, where both functions have the same signature and only differ in `async`, is normally rejected by existing Swift's overloading rules. Those do not allow two functions to differ only in their *effects*, and one can not define two functions that only differ in `throws`, for example. + +``` +// error: redeclaration of function `doSomethingElse()`. +``` + +This also presents a problem for code evolution, because developers of existing libraries just could not preserve their existing synchronous APIs, and support new asynchronous features. + +Instead, we propose an overload-resolution rule to select the appropriate function based on the context of the call. Given a call, overload resolution prefers non-`async` functions within a synchronous context (because such contexts cannot contain a call to an `async` function). Furthermore, overload resolution prefers `async` functions within an asynchronous context (because such contexts should avoid stepping out of the asynchronous model into blocking APIs). When overload resolution selects an `async` function, that call is still subject to the rule that it must occur within an `await` expression. + +The overload-resolution rule depends on the synchronous or asynchronous context, in which the compiler selects one and only one overload. The selection of the async overload requires an `await` expression, as all introductions of a potential suspension point: + +```swift +func f() async { + // In an asynchronous context, the async overload is preferred: + await doSomething() + // Compiler error: Expression is 'async' but is not marked with 'await' + doSomething() +} +``` + +In non-`async` functions, and closures without any `await` expression, the compiler selects the non-`async` overload: + +```swift +func f() async { + let f2 = { + // In a synchronous context, the non-async overload is preferred: + doSomething() + } + f2() +} +``` + + +### Autoclosures + +A function may not take an autoclosure parameter of `async` function type unless the function itself is `async`. For example, the following declaration is ill-formed: + +```swift +// error: async autoclosure in a function that is not itself 'async' +func computeArgumentLater(_ fn: @escaping @autoclosure () async -> T) { } +``` + +This restriction exists for several reasons. Consider the following example: + + ```swift + // func getIntSlowly() async -> Int { ... } + + let closure = { + computeArgumentLater(await getIntSlowly()) + print("hello") + } + ``` + +At first glance, the `await` expression implies to the programmer that there is a potential suspension point *prior* to the call to `computeArgumentLater(_:)`, which is not actually the case: the potential suspension point is *within* the (auto)closure that is passed and used within the body of `computeArgumentLater(_:)`. This causes a few problems. First, the fact that `await` appears to be prior to the call means that `closure` would be inferred to have `async` function type, which is also incorrect: all of the code in `closure` is synchronous. Second, because an `await`'s operand only needs to contain a potential suspension point somewhere within it, an equivalent rewriting of the call should be: + +```swift +await computeArgumentLater(getIntSlowly()) +``` + +But, because the argument is an autoclosure, this rewriting is no longer semantics-preserving. Thus, the restriction on `async` autoclosure parameters avoids these problems by ensuring that `async` autoclosure parameters can only be used in asynchronous contexts. + +### Protocol conformance + +A protocol requirement can be declared as `async`. Such a requirement can be satisfied by an `async` or synchronous function. However, a synchronous function requirement cannot be satisfied by an `async` function. For example: + +```swift +protocol Asynchronous { + func f() async +} + +protocol Synchronous { + func g() +} + +struct S1: Asynchronous { + func f() async { } // okay, exactly matches +} + +struct S2: Asynchronous { + func f() { } // okay, synchronous function satisfying async requirement +} + +struct S3: Synchronous { + func g() { } // okay, exactly matches +} + +struct S4: Synchronous { + func g() async { } // error: cannot satisfy synchronous requirement with an async function +} +``` + +This behavior follows the subtyping/implicit conversion rule for asynchronous functions, as is precedented by the behavior of `throws`. + +## Source compatibility + +This proposal is generally additive: existing code does not use any of the new features (e.g., does not create `async` functions or closures) and will not be impacted. However, it introduces two new contextual keywords, `async` and `await`. + +The positions of the new uses of `async` within the grammar (function declarations and function types) allows us to treat `async` as a contextual keyword without breaking source compatibility. A user-defined `async` cannot occur in those grammatical positions in well-formed code. + +The `await` contextual keyword is more problematic, because it occurs within an expression. For example, one could define a function `await` in Swift today: + +```swift +func await(_ x: Int, _ y: Int) -> Int { x + y } + +let result = await(1, 2) +``` + +This is well-formed code today that is a call to the `await` function. With this proposal, this code becomes an `await` expression with the subexpression `(1, 2)`. This will manifest as a compile-time error for existing Swift programs, because `await` can only be used within an asynchronous context, and no existing Swift programs have such a context. Such functions do not appear to be common, so we believe this is an acceptable source break as part of the introduction of async/await. + +## Effect on ABI stability + +Asynchronous functions and function types are additive to the ABI, so there is no effect on ABI stability, because existing (synchronous) functions and function types are unchanged. + +## Effect on API resilience + +The ABI for an `async` function is completely different from the ABI for a synchronous function (e.g., they have incompatible calling conventions), so the addition or removal of `async` from a function or type is not a resilient change. + +## Future Directions + +### `reasync` + +Swift's `rethrows` is a mechanism for indicating that a particular function is throwing only when one of the arguments passed to it is a function that itself throws. For example, `Sequence.map` makes use of `rethrows` because the only way the operation can throw is if the transform itself throws: + +```swift +extension Sequence { + func map(transform: (Element) throws -> Transformed) rethrows -> [Transformed] { + var result = [Transformed]() + var iterator = self.makeIterator() + while let element = iterator.next() { + result.append(try transform(element)) // note: this is the only `try`! + } + return result + } +} +``` + +Here are uses of `map` in practice: + +```swift +_ = [1, 2, 3].map { String($0) } // okay: map does not throw because the closure does not throw +_ = try ["1", "2", "3"].map { (string: String) -> Int in + guard let result = Int(string) else { throw IntParseError(string) } + return result +} // okay: map can throw because the closure can throw +``` + +The same notion could be applied to `async` functions. For example, we could imagine making `map` asynchronous when its argument is asynchronous with `reasync`: + +```swift +extension Sequence { + func map(transform: (Element) async throws -> Transformed) reasync rethrows -> [Transformed] { + var result = [Transformed]() + var iterator = self.makeIterator() + while let element = iterator.next() { + result.append(try await transform(element)) // note: this is the only `try` and only `await`! + } + return result + } +} +``` + +*Conceptually*, this is fine: when provided with an `async` function, `map` will be treated as `async` (and you'll need to `await` the result), whereas providing it with a non-`async` function, `map` will be treated as synchronous (and won't require `await`). + +*In practice*, there are a few problems here: + +* This is probably not a very good implementation of an asynchronous `map` on a sequence. More likely, we would want a concurrent implementation that (say) processes up to number-of-cores elements concurrently. +* The ABI of throwing functions is intentionally designed to make it possible for a `rethrows` function to act as a non-throwing function, so a single ABI entry point suffices for both throwing and non-throwing calls. The same is not true of `async` functions, which have a radically different ABI that is necessarily less efficient than the ABI for synchronous functions. + +For something like `Sequence.map` that might become concurrent, `reasync` is likely the wrong tool: overloading for `async` closures to provide a separate (concurrent) implementation is likely the better answer. So, `reasync` is likely to be much less generally applicable than `rethrows`. + +There are undoubtedly some uses for `reasync`, such as the `??` operator for optionals, where the `async` implementation degrades nicely to a synchronous implementation: + +```swift +func ??( + _ optValue: T?, _ defaultValue: @autoclosure () async throws -> T +) reasync rethrows -> T { + if let value = optValue { + return value + } + + return try await defaultValue() +} +``` + +For such cases, the ABI concern described above can likely be addressed by emitting two entrypoints: one when the argument is `async` and one when it is not. However, the implementation is complex enough that the authors are not yet ready to commit to this design. + +## Alternatives Considered + +### Make `await` imply `try` + +Many asynchronous APIs involve file I/O, networking, or other failable operations, and therefore will be both `async` and `throws`. At the call site, this means `try await` will be repeated many times. To reduce the boilerplate, `await` could imply `try`, so the following two lines would be equivalent: + +```swift +let dataResource = await loadWebResource("dataprofile.txt") +let dataResource = try await loadWebResource("dataprofile.txt") +``` + +We chose not to make `await` imply `try` because they are expressing different kinds of concerns: `await` is about a potential suspension point, where other code might execute in between when you make the call and it when it returns, while `try` is about control flow out of the block. + +One other motivation that has come up for making `await` imply `try` is related to task cancellation. If task cancellation were modeled as a thrown error, and every potential suspension point implicitly checked whether the task was cancelled, then every potential suspension point could throw: in such cases `await` might as well imply `try` because every `await` can potentially exit with an error. +Task cancellation is covered in the [Structured Concurrency](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md) proposal, and does *not* model cancellation solely as a thrown error nor does it introduce implicit cancellation checks at each potential suspension point. + +### Launching async tasks + +Because only `async` code can call other `async` code, this proposal provides no way to initiate asynchronous code. This is intentional: all asynchronous code runs within the context of a "task", a notion which is defined in the [Structured Concurrency](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md) proposal. That proposal provides the ability to define asynchronous entry points to the program via `@main`, e.g., + +```swift +@main +struct MyProgram { + static func main() async { ... } +} +``` + +Additionally, top-level code is not considered an asynchronous context in this proposal, so the following program is ill-formed: + +```swift +func f() async -> String { "hello, asynchronously" } + +print(await f()) // error: cannot call asynchronous function in top-level code +``` + +This, too, will be addressed in a subsequent proposal that properly accounts for +top-level variables. + +None of the concerns for top-level code affect the fundamental mechanisms of async/await as defined in this proposal. + +### Await as syntactic sugar + +This proposal makes `async` functions a core part of the Swift type system, distinct from synchronous functions. An alternative design would leave the type system unchanged, and instead make `async` and `await` syntactic sugar over some `Future` type, e.g., + +```swift +async func processImageData() throws -> Future { + let dataResource = try loadWebResource("dataprofile.txt").await() + let imageResource = try loadWebResource("imagedata.dat").await() + let imageTmp = try decodeImage(dataResource, imageResource).await() + let imageResult = try dewarpAndCleanupImage(imageTmp).await() + return imageResult +} +``` + +This approach has a number of downsides vs. the proposed approach here: + +* There is no universal `Future` type on which to build it in the Swift ecosystem. If the Swift ecosystem had mostly settled on a single future type already (e.g., if there were already one in the standard library), a syntactic-sugar approach like the above would codify existing practice. Lacking such a type, one would have to try to abstract over all of the different kinds of future types with some kind of `Futurable` protocol. This may be possible for some set of future types, but would give up any guarantees about the behavior or performance of asynchronous code. +* It is inconsistent with the design of `throws`. The result type of asynchronous functions in this model is the future type (or "any `Futurable` type"), rather than the actual returned value. They must always be `await`'ed immediately (hence the postfix syntax) or you'll end up working with futures when you actually care about the result of the asynchronous operation. This becomes a programming-with-futures model rather than an asynchronous-programming model, when many other aspects of the `async` design intentionally push away from thinking about the futures. +* Taking `async` out of the type system would eliminate the ability to do overloading based on `async`. See the prior section on the reasons for overloading on `async`. +* Futures are relatively heavyweight types, and forming one for every async operation has nontrivial costs in both code size and performance. In contrast, deep integration with the type system allows `async` functions to be purpose-built and optimized for efficient suspension. All levels of the Swift compiler and runtime can optimize `async` functions in a manner that would not be possible with future-returning functions. + +## Revision history + +* Post-review changes: + * Replaced `await try` with `try await`. + * Added syntactic-sugar alternative design. + * Amended the proposal to allow [overloading on `async`](https://github.com/swiftlang/swift-evolution/pull/1392). +* Changes in the second pitch: + * One can no longer directly overload `async` and non-`async` functions. Overload resolution support remains, however, with additional justification. + * Added an implicit conversion from a synchronous function to an asynchronous function. + * Added `await try` ordering restriction to match the `async throws` restriction. + * Added support for `async` initializers. + * Added support for synchronous functions satisfying an `async` protocol requirement. + * Added discussion of `reasync`. + * Added justification for `await` not implying `try`. + * Added justification for `async` following the function parameter list. + +* Original pitch ([document](https://github.com/DougGregor/swift-evolution/blob/092c05eebb48f6c0603cd268b7eaf455865c64af/proposals/nnnn-async-await.md) and [forum thread](https://forums.swift.org/t/concurrency-asynchronous-functions/41619)). + +## Related proposals + +In addition to this proposal, there are a number of related proposals covering different aspects of the Swift Concurrency model: + +* [Concurrency Interoperability with Objective-C](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0297-concurrency-objc.md): Describes the interaction with Objective-C, especially the relationship between asynchronous Objective-C methods that accept completion handlers and `@objc async` Swift methods. +* [Structured Concurrency](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md): Describes the task structure used by asynchronous calls, the creation of both child tasks and detached tasks, cancellation, prioritization, and other task-management APIs. +* [Actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md): Describes the actor model, which provides state isolation for concurrent programs + +## Acknowledgments + +The desire for async/await in Swift has been around for a long time. This proposal draws some inspiration (and most of the Motivation section) from an earlier proposal written by +[Chris Lattner](https://github.com/lattner) and [Joe Groff](https://github.com/jckarter), available [here](https://gist.github.com/lattner/429b9070918248274f25b714dcfc7619). That proposal itself is derived from a proposal written by [Oleg Andreev](https://github.com/oleganza), available [here](https://gist.github.com/oleganza/7342ed829bddd86f740a). It has been significantly rewritten (again), and many details have changed, but the core ideas of asynchronous functions have remained the same. + +Efficient implementation is critical for the introduction of asynchronous functions, and Swift Concurrency as a whole. Nate Chandler, Erik Eckstein, Kavon Farvardin, Joe Groff, Chris Lattner, Slava Pestov, and Arnold Schwaighofer all made significant contributions to the implementation of this proposal. diff --git a/proposals/0297-concurrency-objc.md b/proposals/0297-concurrency-objc.md new file mode 100644 index 0000000000..a0caa5de14 --- /dev/null +++ b/proposals/0297-concurrency-objc.md @@ -0,0 +1,357 @@ +# Concurrency Interoperability with Objective-C + +* Proposal: [SE-0297](0297-concurrency-objc.md) +* Author: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 5.5)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0297-concurrency-interoperability-with-objective-c/43306) +* Implementation: Partially available in [recent `main` snapshots](https://swift.org/download/#snapshots) behind the flag `-Xfrontend -enable-experimental-concurrency` + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Proposed solution](#proposed-solution) +* [Detailed design](#detailed-design) + * [Asynchronous completion-handler methods](#asynchronous-completion-handler-methods) + * [Defining asynchronous @objc methods in Swift](#defining-asynchronous-objc-methods-in-swift) + * [Actor classes](#actor-classes) + * [Completion handlers must be called exactly once](#completion-handlers-must-be-called-exactly-once) + * [Additional Objective-C attributes](#additional-objective-c-attributes) +* [Source compatibility](#source-compatibility) +* [Revision history](#revision-history) +* [Future Directions](#future-directions) + * [NSProgress](#nsprogress) + +## Introduction + +Swift's concurrency feature involves asynchronous functions and actors. While Objective-C does not have corresponding language features, asynchronous APIs are common in Objective-C, expressed manually through the use of completion handlers. This proposal provides bridging between Swift's concurrency features (e.g., `async` functions) and the convention-based expression of asynchronous functions in Objective-C. It is intended to allow the wealth of existing asynchronous Objective-C APIs to be immediately usable with Swift's concurrency model. + +For example, consider the following Objective-C API in [CloudKit](https://developer.apple.com/documentation/cloudkit/ckcontainer/1640387-fetchshareparticipantwithuserrec): + +```objc +- (void)fetchShareParticipantWithUserRecordID:(CKRecordID *)userRecordID + completionHandler:(void (^)(CKShareParticipant * _Nullable, NSError * _Nullable))completionHandler; +``` + +This API is asynchronous. It delivers its result (or an error) via completion handler. The API directly translates into Swift: + +```swift +func fetchShareParticipant( + withUserRecordID userRecordID: CKRecord.ID, + completionHandler: @escaping (CKShare.Participant?, Error?) -> Void +) +``` + +Existing Swift code can call this API by passing a closure for the completion handler. This proposal provides an alternate Swift translation of the API into an `async` function, e.g., + +```swift +func fetchShareParticipant( + withUserRecordID userRecordID: CKRecord.ID +) async throws -> CKShare.Participant +``` + +Swift callers can invoke `fetchShareParticipant(withUserRecordID:)` within an `await` expression: + +```swift +guard let participant = try? await container.fetchShareParticipant(withUserRecordID: user) else { + return nil +} +``` + +Swift-evolution thread: [\[Concurrency\] Interoperability with Objective-C](https://forums.swift.org/t/concurrency-interoperability-with-objective-c/41616) + +## Motivation + +On Apple platforms, Swift's tight integration with Objective-C APIs is an important part of the developer experience. There are several core features: + +* Objective-C classes, protocols, and methods can be used directly from Swift. +* Swift classes can subclass Objective-C classes. +* Swift classes can declare conformance to Objective-C protocols. +* Swift classes, protocols, and methods can be made available to Objective-C via the `@objc` attribute. + +Asynchronous APIs abound in Objective-C code: the iOS 14.0 SDK includes nearly 1,000 methods that accept completion handlers. These include methods that one could call directly from Swift, methods that one would override in a Swift-defined subclass, and methods in protocols that one would conform to. Supporting these use cases in Swift's concurrency model greatly expands the reach of this new feature. + +## Proposed solution + +The proposed solution provides interoperability between Swift's concurrency constructs and Objective-C in various places. It has several inter-dependent pieces: + +* Translate Objective-C completion-handler methods into `async` methods in Swift. +* Allow `async` methods defined in Swift to be `@objc`, in which case they are exported as completion-handler methods. +* Provide Objective-C attributes to control over how completion-handler-based APIs are translated into `async` Swift functions. + +The detailed design section describes the specific rules and heuristics being applied. However, the best way to evaluate the overall effectiveness of the translation is to see its effect over a large number of Objective-C APIs. [This pull request](https://github.com/DougGregor/swift-concurrency-objc/pull/1) demonstrates the effect that this proposal has on the Swift translations of Objective-C APIs across the Apple iOS, macOS, tvOS, and watchOS SDKs. + +## Detailed design + +### Asynchronous completion-handler methods + +An Objective-C method is potentially an asynchronous completion-handler method if it meets the following requirements: + +* The method has a completion handler parameter, which is an Objective-C block that will receive the "result" of the asynchronous computation. It must meet the following additional constraints: + * It has a `void` result type. + * It is called exactly once along all execution paths through the implementation. + * If the method can deliver an error, one of the parameters of the block is of type `NSError *` that is not `_Nonnull`. A non-nil `NSError *` value typically indicates that an error occurred, although the C `swift_async` attribute can describe other conventions (discussed in the section on Objective-C attributes). +* The method itself has a `void` result type, because all results are delivered by the completion handler block. + +An Objective-C method that is potentially an asynchronous completion-handler method will be translated into an `async` method when it is either annotated explicitly with an appropriate `swift_async` attribute (described in the section on Objective-C attributes) or is implicitly inferred when the following heuristics successfully identify the completion handler parameter: + +* If the method has a single parameter, and the suffix of the first selector piece is one of the following phrases: + - `WithCompletion` + - `WithCompletionHandler` + - `WithCompletionBlock` + - `WithReplyTo` + - `WithReply` + the sole parameter is the completion handler parameter. The matching phrase will be removed from the base name of the function when it is imported. +* If the method has more than one parameter, the last parameter is the completion handler parameter if its selector piece or parameter name is `completion`, `withCompletion`, `completionHandler`, `withCompletionHandler`, `completionBlock`, `withCompletionBlock`, `replyTo`, `withReplyTo`, `reply`, or `replyTo`. +* If the method has more than one parameter, and the last parameter ends with one of the suffixes from the first bullet, the last parameter is the completion handler. The text preceding the suffix is appended to the base name of the function. + +When the completion handler parameter is inferred, the presence of an `NSError *` parameter that is not `_Nonnull` in the completion handler block type indicates that the translated method can deliver an error. + +The translation of an asynchronous Objective-C completion-handler method into an `async` Swift method follows the normal translation procedure, with the following alterations: + +* The completion handler parameter is removed from the parameter list of the translated Swift method. +* If the method can deliver an error, it is `throws` in addition to being `async`. +* The parameter types of the completion handler block type are translated into the result type of the `async` method, subject to the following additional rules: + * If the method can deliver an error, the `NSError *` parameter is ignored. + * If the method can deliver an error and a given parameter has the `_Nullable_result` nullability qualifier (see the section on Objective-C attributes below), it will be imported as optional. Otherwise, it will be imported as non-optional. + * If there are multiple parameter types, they will be combined into a tuple type. + +The following [PassKit API](https://developer.apple.com/documentation/passkit/pkpasslibrary/3543357-signdata?language=objc) demonstrates how the inference rule plays out: + +```objc +- (void)signData:(NSData *)signData +withSecureElementPass:(PKSecureElementPass *)secureElementPass + completion:(void (^)(NSData *signedData, NSData *signature, NSError *error))completion; +``` + +Today, this is translated into the following completion-handler function in Swift: + +```swift +@objc func sign(_ signData: Data, + using secureElementPass: PKSecureElementPass, + completion: @escaping (Data?, Data?, Error?) -> Void +) +``` + +This will be translated into the following `async` function: + +```swift +@objc func sign( + _ signData: Data, + using secureElementPass: PKSecureElementPass +) async throws -> (Data, Data) +``` + +When the compiler sees a call to such a method, it effectively uses `withUnsafeContinuation` to form a continuation for the rest of the function, then wraps the given continuation in a closure. For example: + +```swift +let (signedValue, signature) = try await passLibrary.sign(signData, using: pass) +``` + +becomes pseudo-code similar to + +```swift +try withUnsafeContinuation { continuation in + passLibrary.sign( + signData, using: pass, + completionHandler: { (signedValue, signature, error) in + if let error = error { + continuation.resume(throwing: error) + } else { + continuation.resume(returning: (signedValue!, signature!)) + } + } + ) +} +``` + +Additional rules are applied when translating an Objective-C method name into a Swift name of an `async` function: + +* If the base name of the method starts with `get`, the `get` is removed and the leading initialisms are lowercased. +* If the base name of the method ends with `Asynchronously`, that word is removed. + +If the completion-handler parameter of the Objective-C method is nullable and the translated `async` method returns non-`Void`, it will be marked with the `@discardableResult` attribute. For example: + +```objc +-(void)stopRecordingWithCompletionHandler:void(^ _Nullable)(RPPreviewViewController * _Nullable, NSError * _Nullable)handler; +``` + +will become: + +```swift +@discardableResult func stopRecording() async throws -> RPPreviewViewController +``` + +### Defining asynchronous `@objc` methods in Swift + +Many Swift entities can be exposed to Objective-C via the `@objc` attribute. With an `async` Swift method, the compiler will add an appropriate completion-handler parameter to the Objective-C method it creates, using what is effectively the inverse of the transformation described in the previous section, such that the Objective-C method produced is an asynchronous Objective-C completion-handler method. For example, a method such as: + +```swift +@objc func perform(operation: String) async -> Int { ... } +``` + +will translate into the following Objective-C method: + +```objc +- (void)performWithOperation:(NSString * _Nonnull)operation + completionHandler:(void (^ _Nullable)(NSInteger))completionHandler; +``` + +The Objective-C method implementation synthesized by the compiler will create a detached task that calls the `async` Swift method `perform(operation:)` with the given string, then (if the completion handler argument is not `nil`) forwards the result to the completion handler. + +For an `async throws` method, the completion handler is extended with an `NSError *` parameter to indicate the error, any non-nullable pointer type parameters are made `_Nullable`, and any nullable pointer type parameters are made `_Nullable_result`. For example, given: + +```swift +@objc func performDangerousTrick(operation: String) async throws -> String { ... } +``` + +the resulting Objective-C method will have the following signature: + +```objc +- (void)performDangerousTrickWithOperation:(NSString * _Nonnull)operation + completionHandler:(void (^ _Nullable)(NSString * _Nullable, NSError * _Nullable))completionHandler; +``` + +Again, the synthesized Objective-C method implementation will create a detached task that calls the `async throws` method `performDangerousTrick(operation:)`. If the method returns normally, the `String` result will be delivered to the completion handler in the first parameter and the second parameter (`NSError *`) will be passed `nil`. If the method throws, the first parameter will be passed `nil` (which is why it has been made `_Nullable` despite being non-optional in Swift) and the second parameter will receive the error. If there are non-pointer parameters, they will be passed zero-initialized memory in the non-error arguments to provide consistent behavior for callers. This can be demonstrated with Swift pseudo-code: + +```swift +// Synthesized by the compiler +@objc func performDangerousTrick( + operation: String, + completionHandler: ((String?, Error?) -> Void)? +) { + runDetached { + do { + let value = try await performDangerousTrick(operation: operation) + completionHandler?(value, nil) + } catch { + completionHandler?(nil, error) + } + } +} +``` + +### Actor classes + +Actor classes can be `@objc` and will be available in Objective-C as are other classes. Actor classes require that their superclass (if there is one) also be an actor class. However, this proposal loosens that requirement slightly to allow an actor class to have `NSObject` as its superclass. This is conceptually safe because `NSObject` has no state (and its layout is effectively fixed that way), and makes it possible both for actor classes to be `@objc` and also implies conformance to `NSObjectProtocol`, which is required when conforming to a number of Objective-C protocols and is otherwise unimplementable in Swift. + +A member of an actor class can only be `@objc` if it is either `async` or is outside of the actor's isolation domain. Synchronous code that is within the actor's isolation domain can only be invoked on `self` (in Swift). Objective-C does not have knowledge of actor isolation, so these members are not permitted to be exposed to Objective-C. For example: + +```swift +actor class MyActor { + @objc func synchronous() { } // error: part of actor's isolation domain + @objc func asynchronous() async { } // okay: asynchronous + @objc @actorIndependent func independent() { } // okay: actor-independent +} +``` + +### Completion handlers must be called exactly once + +A Swift `async` function will always suspend, return, or (if it throws) produce an error. For completion-handler APIs, it is important that the completion handler block be called exactly once on all paths, including when producing an error. Failure to do so will break the semantics of the caller, either by failing to continue or by executing the same code multiple times. While this is an existing problem, widespread use of `async` with incorrectly-implemented completion-handler APIs might exacerbate the issue. + +Fortunately, because the compiler itself is synthesizing the block that will be passed to completion-handler APIs, it can detect both problems by introducing an extra bit of state into the synthesized block to indicate that the block has been called. If the bit is already set when the block is called, then it has been called multiple times. If the bit is not set when the block is destroyed, it has not been called at all. While this does not fix the underlying problem, it can at least detect the issue consistently at run time. + +### Additional Objective-C attributes + +The transformation of Objective-C completion-handler-based APIs to async Swift APIs could benefit from the introduction of additional annotations (in the form of attributes) to guide the process. For example: + +* `_Nullable_result`. Like `_Nullable`, indicates that a pointer can be null (or `nil`). `_Nullable_result` differs from `_Nullable` only for parameters to completion handler blocks. When the completion handler block's parameters are translated into the result type of an `async` method, the corresponding result will be optional. +* `__attribute__((swift_async(...)))`. An attribute to control the translation of an asynchronous completion-handler method to an `async` function. It has several operations within the parentheses: + * `__attribute__((swift_async(none)))`. Disables the translation to `async`. + * `__attribute__((swift_async(not_swift_private, C)))`. Specifies that the method should be translated into an `async` method, using the parameter at index `C` as the completion handler parameter. The first (non-`self`) parameter has index 1. + * `__attribute__((swift_async(swift_private, C)))`. Specifies that the method should be translated into an `async` method that is "Swift private" (only for use when wrapping), using the parameter at index `C` as the completion handler parameter. The first (non-`self`) parameter has index 1. +* `__attribute__((swift_attr("swift attribute")))`. A general-purpose Objective-C attribute to allow one to provide Swift attributes directly. In the context of concurrency, this allows Objective-C APIs to be annotated with a global actor (e.g., `@MainActor`). +* `__attribute__((swift_async_name("method(param1:param2:)")))`. Specifies the Swift name that should be used for the `async` translation of the API. The name should not include an argument label for the completion handler parameter. +* `__attribute__((swift_async_error(...)))`. An attribute to control how passing an `NSError *` into the completion handle maps into the method being `async throws`. It has several possible parameters: + * `__attribute__((swift_async_error(none)))`: Do not import as `throws`. The `NSError *` parameter will be considered a normal parameter. + * `__attribute__((swift_async_error(zero_argument, N)))`: Import as `throws`. When the Nth parameter to the completion handler is passed the integral value zero (including `false`), the async method will throw the error. The Nth parameter is removed from the result type of the translated `async` method. The first parameter is numbered `1`. + * `__attribute__((swift_async_error(nonzero_argument, N)))`: Import as `throws`. When the Nth parameter to the completion handler is passed a non-zero integral value (including `true`), the async method will throw the error. The Nth parameter is removed from the result type of the translated `async` method. The first parameter is numbered `1`. + + +## Source compatibility + +Generally speaking, changes to the way in which Objective-C APIs are translated into Swift are source-breaking changes. To avoid breaking source compatibility, this proposal involves translating Objective-C asynchronous completion-handler methods as *both* their original completion-handler signatures and also with the new `async` signature. This allows existing Swift code bases to gradually adopt the `async` forms of API, rather than forcing (e.g.) an entire Swift module to adopt `async` all at once. + +Importing the same Objective-C API in two different ways causes some issues: + +* Overloading of synchronous and asynchronous APIs. Objective-C frameworks may have evolved to include both synchronous and asynchronous versions of the same API, e.g., + + ```objc + - (NSString *)lookupName; + - (void)lookupNameWithCompletionHandler:(void (^)(NSString *))completion; + ``` + which will be translated into three different Swift methods: + + ```swift + @objc func lookupName() -> String + @objc func lookupName(withCompletionHandler: @escaping (String) -> Void) + @objc func lookupName() async -> String + ``` + + The first and third signatures are identical except for being synchronous and asynchronous, respectively. The async/await design doesn't allow such overloading to be written in the same Swift module, but it can happen when translating Objective-C APIs or when importing methods from different Swift modules. The async/await design accounts for such overloading by favoring synchronous functions in synchronous contexts and asynchronous functions in asynchronous contexts. This overloading should avoid breaking source compatibility. + +* Another issue is when an asynchronous completion-handler method is part of an Objective-C protocol. For example, the [`NSURLSessionDataDelegate` protocol](https://developer.apple.com/documentation/foundation/nsurlsessiondatadelegate?language=objc) includes this protocol requirement: + + ```objc + @optional + - (void)URLSession:(NSURLSession *)session + dataTask:(NSURLSessionDataTask *)dataTask + didReceiveResponse:(NSURLResponse *)response + completionHandler:(void (^)(NSURLSessionResponseDisposition disposition))completionHandler; + ``` + + Existing Swift code might implement this requirement in a conforming type using its completion-handler signature + + ```swift + @objc + func urlSession( + _ session: URLSession, + dataTask: URLSessionDataTask, + didReceive response: URLResponse, + completionHandler: @escaping (URLSession.ResponseDisposition) -> Void + ) { ... } + ``` + + while Swift code designed to take advantage of the concurrency model would implement this requirement in a conforming type using its `async` signature + + ```swift + @objc + func urlSession( + _ session: URLSession, + dataTask: URLSessionDataTask, + didReceive response: URLResponse + ) async -> URLSession.ResponseDisposition { ... } + ``` + + Implementing both requirements would produce an error (due to two Swift methods having the same selector), but under the normal Swift rules implementing only one of the requirements will also produce an error (because the other requirement is unsatisfied). Swift’s checking of protocol conformances will be extended to handle the case where multiple (imported) requirements have the same Objective-C selector: in that case, only one of them will be required to be implemented. + +* Overriding methods that have been translated into both completion-handler and `async` versions have a similar problem to protocol requirements: a Swift subclass can either override the completion-handler version or the `async` version, but not both. Objective-C callers will always call to the subclass version of the method, but Swift callers to the "other" signature will not unless the subclass's method is marked with `@objc dynamic`. Swift can infer that the `async` overrides of such methods are `@objc dynamic` to avoid this problem (because such `async` methods are new code). However, inferring `@objc dynamic` on existing completion-handler overrides can change the behavior of programs and break subclasses of the subclasses, so at best the compiler can warn about this situation. + +## Revision history + +* Post-review: + * `await try` becomes `try await` based on result of SE-0296 review + * Added inference of `@discardableResult` for `async` methods translated from completion-handler methods with an optional completion handler. +* Changes in the second pitch: + * Removed mention of asynchronous handlers, which will be in a separate proposal. + * Introduced the `swift_async_error` Clang attribute to separate out "throwing" behavior from the `swift_async` attribute. + * Added support for "Swift private" to the `swift_async` attribute. + * Tuned the naming heuristics based on feedback to add (e.g) `reply`, `replyTo`, `completionBlock`, and variants. + * For the rare case where we match a parameter suffix, append the text prior to the suffix to the base name. + * Replaced the `-generateCGImagesAsynchronouslyForTimes:completionHandler:` example with one from PassKit. + * Added a "Future Directions" section about `NSProgress`. + +* Original pitch ([document](https://github.com/DougGregor/swift-evolution/blob/9b9bdfd16eb5ced390913ea170007a46eabb08eb/proposals/NNNN-concurrency-objc.md) and [forum thread](https://forums.swift.org/t/concurrency-interoperability-with-objective-c/41616)). + +## Future Directions + +### NSProgress + +Some Objective-C completion-handler methods return an [NSProgress](https://developer.apple.com/documentation/foundation/progress) to allow the caller to evaluate progress of the asynchronous operation. Such methods are *not* imported as `async` in this proposal, because the method does not return `void`. For example: + +```swift +- (NSProgress *)doSomethingThatTakesALongTimeWithCompletionHandler:(void (^)(MyResult * _Nullable, NSError * _Nullable))completionHandler; +``` + +To support such methods would require some kind of integration between `NSProgress` and Swift's tasks. For example, when calling such a method, the `NSProgress` returned from such a call to be recorded in the task (say, in some kind of task-local storage). The other direction, where a Swift-defined method overrides a method, would need to extract an `NSProgress` from the task to return. Such a design is out of scope for this proposal, but could be introduced at some later point. diff --git a/proposals/0298-asyncsequence.md b/proposals/0298-asyncsequence.md new file mode 100644 index 0000000000..d0899264df --- /dev/null +++ b/proposals/0298-asyncsequence.md @@ -0,0 +1,362 @@ +# Async/Await: Sequences + +* Proposal: [SE-0298](0298-asyncsequence.md) +* Authors: [Tony Parker](https://github.com/parkera), [Philippe Hausler](https://github.com/phausler) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.5)** +* Implementation: [apple/swift#35224](https://github.com/apple/swift/pull/35224) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modification-se-0298-async-await-sequences/44231) +* Revision: Based on [forum discussion](https://forums.swift.org/t/pitch-clarify-end-of-iteration-behavior-for-asyncsequence/45548) + +## Introduction + +Swift's [async/await](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md) feature provides an intuitive, built-in way to write and use functions that return a single value at some future point in time. We propose building on top of this feature to create an intuitive, built-in way to write and use functions that return many values over time. + +This proposal is composed of the following pieces: + +1. A standard library definition of a protocol that represents an asynchronous sequence of values +2. Compiler support to use `for...in` syntax on an asynchronous sequence of values +3. A standard library implementation of commonly needed functions that operate on an asynchronous sequence of values + +## Motivation + +We'd like iterating over asynchronous sequences of values to be as easy as iterating over synchronous sequences of values. An example use case is iterating over the lines in a file, like this: + +```swift +for try await line in myFile.lines() { + // Do something with each line +} +``` + +Using the `for...in` syntax that Swift developers are already familiar with will reduce the barrier to entry when working with asynchronous APIs. Consistency with other Swift types and concepts is therefore one of our most important goals. The requirement of using the `await` keyword in this loop will distinguish it from synchronous sequences. +### `for/in` Syntax + +To enable the use of `for in`, we must define the return type from `func lines()` to be something that the compiler understands can be iterated. Today, we have the `Sequence` protocol. Let's try to use it here: + +```swift +extension URL { + struct Lines: Sequence { /* ... */ } + func lines() async -> Lines +} +``` + +Unfortunately, what this function actually does is wait until *all* lines are available before returning. What we really wanted in this case was to await *each* line. While it is possible to imagine modifications to `lines` to behave differently (e.g., giving the result reference semantics), it would be better to define a new protocol to make this iteration behavior as simple as possible. + +```swift +extension URL { + struct Lines: AsyncSequence { /* ... */ } + func lines() async -> Lines +} +``` + +`AsyncSequence` allows for waiting on each element instead of the entire result by defining an asynchronous `next()` function on its associated iterator type. + +### Additional AsyncSequence functions + +Going one step further, let's imagine how it might look to use our new `lines` function in more places. Perhaps we want to process lines until we reach one that is greater than a certain length. + +```swift +let longLine: String? +do { + for try await line in myFile.lines() { + if line.count > 80 { + longLine = line + break + } + } +} catch { + longLine = nil // file didn't exist +} +``` + +Or, perhaps we actually do want to read all lines in the file before starting our processing: + +```swift +var allLines: [String] = [] +do { + for try await line in myFile.lines() { + allLines.append(line) + } +} catch { + allLines = [] +} +``` + +There's nothing wrong with the above code, and it must be possible for a developer to write it. However, it does seem like a lot of boilerplate for what might be a common operation. One way to solve this would be to add more functions to `URL`: + +```swift +extension URL { + struct Lines : AsyncSequence { } + + func lines() -> Lines + func firstLongLine() async throws -> String? + func collectLines() async throws -> [String] +} +``` + +It doesn't take much imagination to think of other places where we may want to do similar operations, though. Therefore, we believe the best place to put these functions is instead as an extension on `AsyncSequence` itself, specified generically -- just like `Sequence`. + +## Proposed solution + +The standard library will define the following protocols: + +```swift +public protocol AsyncSequence { + associatedtype AsyncIterator: AsyncIteratorProtocol where AsyncIterator.Element == Element + associatedtype Element + __consuming func makeAsyncIterator() -> AsyncIterator +} + +public protocol AsyncIteratorProtocol { + associatedtype Element + mutating func next() async throws -> Element? +} +``` + +The compiler will generate code to allow use of a `for in` loop on any type which conforms with `AsyncSequence`. The standard library will also extend the protocol to provide familiar generic algorithms. Here is an example which does not actually call an `async` function within its `next`, but shows the basic shape: + +```swift +struct Counter : AsyncSequence { + let howHigh: Int + + struct AsyncIterator : AsyncIteratorProtocol { + let howHigh: Int + var current = 1 + mutating func next() async -> Int? { + // We could use the `Task` API to check for cancellation here and return early. + guard current <= howHigh else { + return nil + } + + let result = current + current += 1 + return result + } + } + + func makeAsyncIterator() -> AsyncIterator { + return AsyncIterator(howHigh: howHigh) + } +} +``` + +At the call site, using `Counter` would look like this: + +```swift +for await i in Counter(howHigh: 3) { + print(i) +} + +/* +Prints the following, and finishes the loop: +1 +2 +3 +*/ + + +for await i in Counter(howHigh: 3) { + print(i) + if i == 2 { break } +} +/* +Prints the following: +1 +2 +*/ +``` + +## Detailed design + +Returning to our earlier example: + +```swift +for try await line in myFile.lines() { + // Do something with each line +} +``` + +The compiler will emit the equivalent of the following code: + +```swift +var it = myFile.lines().makeAsyncIterator() +while let line = try await it.next() { + // Do something with each line +} +``` + +All of the usual rules about error handling apply. For example, this iteration must be surrounded by `do/catch`, or be inside a `throws` function to handle the error. All of the usual rules about `await` also apply. For example, this iteration must be inside a context in which calling `await` is allowed like an `async` function. + +### Cancellation + +`AsyncIteratorProtocol` types should use the cancellation primitives provided by Swift's `Task` API, part of [structured concurrency](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md). As described there, the iterator can choose how it responds to cancellation. The most common behaviors will be either throwing `CancellationError` or returning `nil` from the iterator. + +If an `AsyncIteratorProtocol` type has cleanup to do upon cancellation, it can do it in two places: + +1. After checking for cancellation using the `Task` API. +2. In its `deinit` (if it is a class type). + +### Rethrows + +This proposal will take advantage of a separate proposal to add specialized `rethrows` conformance in a protocol, pitched [here](https://forums.swift.org/t/pitch-rethrowing-protocol-conformances/42373). With the changes proposed there for `rethrows`, it will not be required to use `try` when iterating an `AsyncSequence` which does not itself throw. + +The `await` is always required because the definition of the protocol is that it is always asynchronous. + +### End of Iteration + +After an `AsyncIteratorProtocol` types returns `nil` or throws an error from its `next()` method, all future calls to `next()` must return `nil`. This matches the behavior of `IteratorProtocol` types and is important, since calling an iterator's `next()` method is the only way to determine whether iteration has finished. + +## AsyncSequence Functions + +The existence of a standard `AsyncSequence` protocol allows us to write generic algorithms for any type that conforms to it. There are two categories of functions: those that return a single value (and are thus marked as `async`), and those that return a new `AsyncSequence` (and are not marked as `async` themselves). + +The functions that return a single value are especially interesting because they increase usability by changing a loop into a single `await` line. Functions in this category are `first`, `contains`, `min`, `max`, `reduce`, and more. Functions that return a new `AsyncSequence` include `filter`, `map`, and `compactMap`. + +### AsyncSequence to single value + +Algorithms that reduce a for loop into a single call can improve readability of code. They remove the boilerplate required to set up and iterate a loop. + +For example, here is the `contains` function: + +```swift +extension AsyncSequence where Element : Equatable { + public func contains(_ value: Element) async rethrows -> Bool +} +``` + +With this extension, our "first long line" example from earlier becomes simply: + +```swift +let first = try? await myFile.lines().first(where: { $0.count > 80 }) +``` + +Or, if the sequence should be processed asynchronously and used later: + +```swift +async let first = myFile.lines().first(where: { $0.count > 80 }) + +// later + +warnAboutLongLine(try? await first) +``` + +The following functions will be added to `AsyncSequence`: + +| Function | Note | +| - | - | +| `contains(_ value: Element) async rethrows -> Bool` | Requires `Equatable` element | +| `contains(where: (Element) async throws -> Bool) async rethrows -> Bool` | The `async` on the closure allows optional async behavior, but does not require it | +| `allSatisfy(_ predicate: (Element) async throws -> Bool) async rethrows -> Bool` | | +| `first(where: (Element) async throws -> Bool) async rethrows -> Element?` | | +| `min() async rethrows -> Element?` | Requires `Comparable` element | +| `min(by: (Element, Element) async throws -> Bool) async rethrows -> Element?` | | +| `max() async rethrows -> Element?` | Requires `Comparable` element | +| `max(by: (Element, Element) async throws -> Bool) async rethrows -> Element?` | | +| `reduce(_ initialResult: T, _ nextPartialResult: (T, Element) async throws -> T) async rethrows -> T` | | +| `reduce(into initialResult: T, _ updateAccumulatingResult: (inout T, Element) async throws -> ()) async rethrows -> T` | | + +### AsyncSequence to AsyncSequence + +These functions on `AsyncSequence` return a result which is itself an `AsyncSequence`. Due to the asynchronous nature of `AsyncSequence`, the behavior is similar in many ways to the existing `Lazy` types in the standard library. Calling these functions does not eagerly `await` the next value in the sequence, leaving it up to the caller to decide when to start that work by simply starting iteration when they are ready. + +As an example, let's look at `map`: + +```swift +extension AsyncSequence { + public func map( + _ transform: @escaping (Element) async throws -> Transformed + ) -> AsyncMapSequence +} + +public struct AsyncMapSequence: AsyncSequence { + public let upstream: Upstream + public let transform: (Upstream.Element) async throws -> Transformed + public struct Iterator : AsyncIterator { + public mutating func next() async rethrows -> Transformed? + } +} +``` + +For each of these functions, we first define a type which conforms with the `AsyncSequence` protocol. The name is modeled after existing standard library `Sequence` types like `LazyDropWhileCollection` and `LazyMapSequence`. Then, we add a function in an extension on `AsyncSequence` which creates the new type (using `self` as the `upstream`) and returns it. + +| Function | +| - | +| `map(_ transform: (Element) async throws -> T) -> AsyncMapSequence` | +| `compactMap(_ transform: (Element) async throws -> T?) -> AsyncCompactMapSequence` | +| `flatMap(_ transform: (Element) async throws -> SegmentOfResult) async rethrows -> AsyncFlatMapSequence` | +| `drop(while: (Element) async throws -> Bool) async rethrows -> AsyncDropWhileSequence` | +| `dropFirst(_ n: Int) async rethrows -> AsyncDropFirstSequence` | +| `prefix(while: (Element) async throws -> Bool) async rethrows -> AsyncPrefixWhileSequence` | +| `prefix(_ n: Int) async rethrows -> AsyncPrefixSequence` | +| `filter(_ predicate: (Element) async throws -> Bool) async rethrows -> AsyncFilterSequence` | + +## Future Proposals + +The following topics are things we consider important and worth discussion in future proposals: + +### Additional `AsyncSequence` functions + +We've aimed for parity with the most relevant `Sequence` functions. There may be others that are worth adding in a future proposal. + +API which uses a time argument must be coordinated with the discussion about `Executor` as part of the [structured concurrency proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md). + +We would like a `first` property, but properties cannot currently be `async` or `throws`. Discussions are ongoing about adding a capability to the language to allow effects on properties. If those features become part of Swift then we should add a `first` property to `AsyncSequence`. + +### AsyncSequence Builder + +In the standard library we have not only the `Sequence` and `Collection` protocols, but concrete types which adopt them (for example, `Array`). We will need a similar API for `AsyncSequence` that makes it easy to construct a concrete instance when needed, without declaring a new type and adding protocol conformance. + +## Source compatibility + +This new functionality will be source compatible with existing Swift. + +## Effect on ABI stability + +This change is additive to the ABI. + +## Effect on API resilience + +This change is additive to API. + +## Alternatives considered + +### Explicit Cancellation + +An earlier version of this proposal included an explicit `cancel` function. We removed it for the following reasons: + +1. Reducing the requirements of implementing `AsyncIteratorProtocol` makes it simpler to use and easier to understand. The rules about when `cancel` would be called, while straightforward, would nevertheless be one additional thing for Swift developers to learn. +2. The structured concurrency proposal already includes a definition of cancellation that works well for `AsyncSequence`. We should consider the overall behavior of cancellation for asynchronous code as one concept. + +### Asynchronous Cancellation + +If we used explicit cancellation, the `cancel()` function on the iterator could be marked as `async`. However, this means that the implicit cancellation done when leaving a `for/in` loop would require an implicit `await` -- something we think is probably too much to hide from the developer. Most cancellation behavior is going to be as simple as setting a flag to check later, so we leave it as a synchronous function and encourage adopters to make cancellation fast and non-blocking. + +### Opaque Types + +Each `AsyncSequence`-to-`AsyncSequence` algorithm will define its own concrete type. We could attempt to hide these details behind a general purpose type eraser. We believe leaving the types exposed gives us (and the compiler) more optimization opportunities. A great future enhancement would be for the language to support `some AsyncSequence where Element=...`-style syntax, allowing hiding of concrete `AsyncSequence` types at API boundaries. + +### Reusing Sequence + +If the language supported a `reasync` concept, then it seems plausible that the `AsyncSequence` and `Sequence` APIs could be merged. However, we believe it is still valuable to consider these as two different types. The added complexity of a time dimension in asynchronous code means that some functions need more configuration options or more complex implementations. Some algorithms that are useful on asynchronous sequences are not meaningful on synchronous ones. We prefer not to complicate the API surface of the synchronous collection types in these cases. + +### Naming + +The names of the concrete `AsyncSequence` types is designed to mirror existing standard library API like `LazyMapSequence`. Another option is to introduce a new pattern with an empty enum or other namespacing mechanism. + +We considered `AsyncGenerator` but would prefer to leave the `Generator` name for future language enhancements. `Stream` is a type in Foundation, so we did not reuse it here to avoid confusion. + +### `await in` + +We considered a shorter syntax of `await...in`. However, since the behavior here is fundamentally a loop, we feel it is important to use the existing `for` keyword as a strong signal of intent to readers of the code. Although there are a lot of keywords, each one has purpose and meaning to readers of the code. + +### Add APIs to iterator instead of sequence + +We discussed applying the fundamental API (`map`, `reduce`, etc.) to `AsyncIteratorProtocol` instead of `AsyncSequence`. There has been a long-standing (albeit deliberate) ambiguity in the `Sequence` API -- is it supposed to be single-pass or multi-pass? This new kind of iterator & sequence could provide an opportunity to define this more concretely. + +While it is tempting to use this new API to right past wrongs, we maintain that the high level goal of consistency with existing Swift concepts is more important. + +For example, `for...in` cannot be used on an `IteratorProtocol` -- only a `Sequence`. If we chose to make `AsyncIteratorProtocol` use `for...in` as described here, that leaves us with the choice of either introducing an inconsistency between `AsyncIteratorProtocol` and `IteratorProtocol` or giving up on the familiar `for...in` syntax. Even if we decided to add `for...in` to `IteratorProtocol`, it would still be inconsistent because we would be required to leave `for...in` syntax on the existing `Sequence`. + +Another point in favor of consistency is that implementing an `AsyncSequence` should feel familiar to anyone who knows how to implement a `Sequence`. + +We are hoping for widespread adoption of the protocol in API which would normally have instead used a `Notification`, informational delegate pattern, or multi-callback closure argument. In many of these cases we feel like the API should return the 'factory type' (an `AsyncSequence`) so that it can be iterated again. It will still be up to the caller to be aware of any underlying cost of performing that operation, as with iteration of any `Sequence` today. diff --git a/proposals/0299-extend-generic-static-member-lookup.md b/proposals/0299-extend-generic-static-member-lookup.md new file mode 100644 index 0000000000..89813ef0b8 --- /dev/null +++ b/proposals/0299-extend-generic-static-member-lookup.md @@ -0,0 +1,281 @@ +# Extending Static Member Lookup in Generic Contexts + +* Proposal: [SE-0299](0299-extend-generic-static-member-lookup.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin), [Sam Lazarus](https://github.com/sl), [Matt Ricketson](https://github.com/ricketson) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.5)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modification-se-0299-extending-static-member-lookup-in-generic-contexts/45238) +* Implementation: [apple/swift#34523](https://github.com/apple/swift/pull/34523) +* Decision Notes: [First return for revision](https://forums.swift.org/t/returned-for-revision-se-0299-extending-static-member-lookup-in-generic-contexts/44466), [First review thread](https://forums.swift.org/t/se-0299-extending-static-member-lookup-in-generic-contexts/43958) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/4dd3a9c85195185ab7ad99c468732c5b568d51ac/proposals/0299-extend-generic-static-member-lookup.md) + +## Introduction + +Using static member declarations to provide semantic names for commonly used values which can then be accessed via leading dot syntax is an important tool in API design, reducing type repetition and improving call-site legibility. Currently, when a parameter is generic, there is no effective way to take advantage of this syntax. This proposal aims to relax restrictions on accessing static members on protocols to afford the same call-site legibility to generic APIs. + +Swift-evolution thread: [Extending Static Member Lookup in Generic Contexts](https://forums.swift.org/t/proposal-static-member-lookup-on-protocol-metatypes/41946) + +## Motivation + +### Background + +Today, Swift supports static member lookup on concrete types. For example, SwiftUI extends types like `Font` and `Color` with pre-defined, commonly-used values as static properties: + +```swift +extension Font { + public static let headline: Font + public static let subheadline: Font + public static let body: Font + ... +} + +extension Color { + public static let red: Color + public static let green: Color + public static let blue: Color + ... +} +``` + +SwiftUI offers view modifiers that accept instances of `Font` and `Color`, including the static values offered above: + +```swift +VStack { + Text(item.title) + .font(Font.headline) + .foregroundColor(Color.primary) + Text(item.subtitle) + .font(Font.subheadline) + .foregroundColor(Color.secondary) +} +``` + +However, this example shows how “fully-qualified” accessors, include the `Font` and `Color` type names when accessing their static properties, are often redundant in context: we know from the `font()` and `foregroundColor()` modifier names that we’re expecting fonts and colors, respectively, so the type names just add unnecessary repetition. + +Fortunately, Swift’s static member lookup on concrete types is clever enough that it can infer the base type from context, allowing for the use of enum-like “leading dot syntax” (including a good autocomplete experience). This improves legibility, but without loss of clarity: + +```swift +VStack { + Text(item.title) + .font(.headline) + .foregroundColor(.primary) + Text(item.subtitle) + .font(.subheadline) + .foregroundColor(.secondary) +} +``` + +### The Problem + +**Swift static member lookup is not currently supported for members of protocols in generic functions, so there is no way to use leading dot syntax at a generic call site.** For example, SwiftUI defines a `toggleStyle` view modifier like so: + +```swift +extension View { + public func toggleStyle(_ style: S) -> some View +} +``` + +which accepts instances of the `ToggleStyle` protocol, e.g. + +```swift +public protocol ToggleStyle { + associatedtype Body: View + func makeBody(configuration: Configuration) -> Body +} + +public struct DefaultToggleStyle: ToggleStyle { ... } +public struct SwitchToggleStyle: ToggleStyle { ... } +public struct CheckboxToggleStyle: ToggleStyle { ... } +``` + +Today, SwiftUI apps must write the full name of the concrete conformers to `ToggleStyle` when using the `toggleStyle` modifier: + +```swift +Toggle("Wi-Fi", isOn: $isWiFiEnabled) + .toggleStyle(SwitchToggleStyle()) +``` + +However, this approach has a few downsides: + +* **Repetitive:** Only the “Switch” component of the style name is important, since we already know that the modifier expects a type of `ToggleStyle`. +* **Poor discoverability:** There is no autocomplete support to expose the available `ToggleStyle` types to choose from, so you have to know them in advance. + +These downsides are impossible to avoid for generic parameters like above, which discourages generalizing functions. API designers should not have to choose between good design and easy-to-read code. + +Instead, we could ideally support leading dot syntax for generic types with known protocol conformances, allowing syntax like this: + +```swift +Toggle("Wi-Fi", isOn: $isWiFiEnabled) + .toggleStyle(.switch) +``` + +### Existing Workarounds + +There are ways of achieving the desired syntax today without changing the language, however they are often too complex and too confusing for API clients. + +When SwiftUI was still in beta, it included one such workaround in the form of the `StaticMember` type: + +```swift +// Rejected SwiftUI APIs: + +public protocol ToggleStyle { + // ... + typealias Member = StaticMember +} + +extension View { + public func toggleStyle(_ style: S.Member) -> some View +} + +public struct StaticMember { + public var base: Base + public init(_ base: Base) +} + +extension StaticMember where Base: ToggleStyle { + public static var `default`: StaticMember { get } + public static var `switch`: StaticMember { get } + public static var checkbox: StaticMember { get } +} + +// Leading dot syntax (using rejected workaround): + +Toggle("Wi-Fi", isOn: $isWiFiEnabled) + .toggleStyle(.switch) +``` + +However, `StaticMember` *serves no purpose* outside of achieving a more ideal syntax elsewhere. Its inclusion is hard to comprehend for anyone looking at the public facing API, as the type itself is decoupled from its actual purpose. SwiftUI removed `StaticMember` before exiting beta for exactly that reason: developers were commonly confused by its existence, declaration complexity, and usage within the framework. + +In a [prior pitch](https://forums.swift.org/t/protocol-metatype-extensions-to-better-support-swiftui/25469), [Matthew Johnson](https://forums.swift.org/u/anandabits) rightly called out how framework-specific solutions like `StaticMember` are not ideal: this is a general-purpose problem, which demands a general-purpose solution, not framework-specific solutions like `StaticMember`. + +## Proposed solution + +We propose *partially* lifting the current limitation placed on referencing of static members from protocol metatypes in order to improve call site ergonomics of the language and make leading dot syntax behave consistently for all possible base types. + +More specifically, we propose allowing static members declared in extensions of protocols to be referenced by leading dot syntax if the declaring extension or member itself constrains `Self` to be a concrete type. + +The scope of this proposal is limited by design: partially lifting this restriction is an incremental step forward that doesn’t require making significant changes to the implementation of protocols, but also does not foreclose making further improvements in the future such as generally supporting protocol metatype extensions (more on this in *Alternatives Considered*, below). + +## Detailed design + +The type-checker is able to infer any protocol conformance requirements placed on a particular argument from the call site of a generic function. In our previous example, the `toggleStyle` function requires its argument conform to `ToggleStyle`. Based on that information, the type-checker should be able to resolve a base type for a leading dot syntax argument as a type which conforms to the `ToggleStyle` protocol. It can’t simply use the type `ToggleStyle` because only types conforming to a protocol can provide a witness method to reference. To discover such a type and produce a well-formed reference there are two options: + +* Do a global lookup for any type which conforms to the given protocol and use it as a base; +* Require that protocol extension declaring static member(s) or member itself (i.e. generic function/subscript) has 'Self' bound to a concrete type via a same-type generic requirement that would be used to provide a witness for the reference. + +The second option is a much better choice that avoids having to do a global lookup and conformance checking and is consistent with the semantics of leading dot syntax, namely, the requirement that result and base types of the chain have to be convertible. This leads to a new rule: if member either binds 'Self' directly (via same-type generic requirement), or is declared in a protocol extension that has `Self` bound to a concrete type, it should be possible to reference such a member on a protocol metatype, using leading dot syntax, by implicitly replacing the protocol with a conforming type referred by `Self`. + +This approach works well for references without an explicit base, let’s consider an example: + +```swift +// Existing SwiftUI APIs: + +public protocol ToggleStyle { ... } + +public struct DefaultToggleStyle: ToggleStyle { ... } +public struct SwitchToggleStyle: ToggleStyle { ... } +public struct CheckboxToggleStyle: ToggleStyle { ... } + +extension View { + public func toggleStyle(_ style: S) -> some View +} + +// Possible SwiftUI APIs: + +extension ToggleStyle where Self == DefaultToggleStyle { + public static var `default`: Self { .init() } +} + +extension ToggleStyle where Self == SwitchToggleStyle { + public static var `switch`: Self { .init() } +} + +extension ToggleStyle where Self == CheckboxToggleStyle { + public static var checkbox: Self { .init() } +} + +// Leading dot syntax (using proposed solution): + +Toggle("Wi-Fi", isOn: $isWiFiEnabled) + .toggleStyle(.switch) +``` + +In the case of `.toggleStyle(.switch)`, the reference to the member `.switch` is re-written to be `SwitchToggleStyle.switch` in the type-checked AST. + +Note that declaring members this way pollutes the namespace of each concrete type by creating members like `DefaultToggleStyle.default`, but we believe this is an acceptable trade-off to improve call site ergonomics. + +It's also possible to bind `Self` to a type with generic parameters: + +```swift +public struct CustomToggleStyle: ToggleStyle { + ... +} + +extension ToggleStyle { + public static func custom(_: T) -> Self where Self == CustomToggleStyle { + ... + } +} + +Toggle("Wi-Fi", isOn: $isWiFiEnabled) + .toggleStyle(.custom(42)) // base type is inferred to be `CustomToggleStyle` based on the argument type. +``` + +To make this work the type-checker would attempt to infer protocol conformance requirements from context, e.g. the call site of a generic function (in this case there is only one such requirement - the protocol `ToggleStyle`), and propagate them to the type variable representing the implicit base type of the chain. If there is no other contextual information available, e.g. the result type couldn’t be inferred to some concrete type, the type-checker would attempt to bind base to the type of the inferred protocol requirement. + +Member lookup filtering is adjusted to find static members declared in extension of a protocol metatype. Type-checker would then attempt to find innermost generic signature (either signature of context or itself, if it's some kind of a generic function) and make sure 'Self' parameter of a protocol is bound to a concrete type before accepting the member. When a reference to such a member is considered in expression context, type-checker would replace implicit base type with the concrete type referred by `Self` to form a valid reference to a static member. + +## Source compatibility + +This is a purely additive change and does not have any effect on source compatibility. + + + +## Effect on ABI stability + +This change is frontend only and would not impact ABI. + + + +## Effect on API resilience + +This is not an API-level change and would not impact resilience. + + +## Alternatives considered + +### Allow declaring static members directly on protocol metatypes + +There have been multiple discussions on this topic on the Swift forums. The most recent one being [a post from Matthew Johnson](https://forums.swift.org/t/protocol-metatype-extensions-to-better-support-swiftui/25469), which suggests adding a special syntax to the language to enable static member references on protocol metatypes. After investigating this direction we determined that supporting this would require significant changes to the implementation of protocols. + +Due to its narrow scope, the proposed design is simpler and does not require any syntax changes, while still satisfying all the intended use cases. We stress that this is an incremental improvement, which should not impede our ability to support protocol metatype extensions in the future. + +One concrete concern is whether the kind of static member lookup proposed here would be ambiguous with static member lookup on a hypothetical future protocol metatype property. We do not believe it would be, since lookup could be prioritized on the metatype over conforming types. Further, these kinds of namespace and lookup conflicts would likely need to be addressed in a future metatype extension proposal regardless of whether the lookup extension proposed here is accepted or not. + +### Allow leading dot syntax for any extensions on protocol metatypes where the return type can be used as the base type + +While technically feasible (as the compiler can use the concrete return type as the base type of the expression), this approach leads to the pollution of the protocols namespace. Consider the SwiftUI use case with this approach. The following use case would be valid as the types of the `default`, `switch` and `checkbox` static members all conform to `ToggleStyle`: + +```swift +extension ToggleStyle { + public static var `default`: DefaultToggleStyle { .init() } + public static var `switch`: SwitchToggleStyle { .init() } + public static var checkbox: CheckboxToggleStyle { .init() } +} +``` + +This unfortunately leads to all of the following being valid: + +```swift +DefaultToggleStyle.checkbox +SwitchToggleStyle.default +CheckboxToggleStyle.switch +// and so on and so forth +``` + +# Revision History + +The [initial revision of this proposal](https://github.com/swiftlang/swift-evolution/blob/4dd3a9c85195185ab7ad99c468732c5b568d51ac/proposals/0299-extend-generic-static-member-lookup.md) +allowed contextual member lookup to find protocol extension members without +the `Self` requirement, but as noted above, the Core Team rejected this ability +because of the potential for namespace pollution. diff --git a/proposals/0300-continuation.md b/proposals/0300-continuation.md new file mode 100644 index 0000000000..e6574abde4 --- /dev/null +++ b/proposals/0300-continuation.md @@ -0,0 +1,483 @@ +# Continuations for interfacing async tasks with synchronous code + +* Proposal: [SE-0300](0300-continuation.md) +* Authors: [John McCall](https://github.com/rjmccall), [Joe Groff](https://github.com/jckarter), [Doug Gregor](https://github.com/DougGregor), [Konrad Malawski](https://github.com/ktoso) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.5)** +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/5f79481244329ec2860951c0c49c101aef5069e7/proposals/0300-continuation.md), [2](https://github.com/swiftlang/swift-evolution/blob/61c788cdb9674c99fc8731b49056cebcb5497edd/proposals/0300-continuation.md) + +## Introduction + +Asynchronous Swift code needs to be able to work with existing synchronous +code that uses techniques such as completion callbacks and delegate methods to +respond to events. Asynchronous tasks can suspend themselves on +**continuations** which synchronous code can then capture and invoke to +resume the task in response to an event. + +Swift-evolution thread: + +- [Structured concurrency](https://forums.swift.org/t/concurrency-structured-concurrency/41622) +- [Continuations for interfacing async tasks with synchronous code](https://forums.swift.org/t/concurrency-continuations-for-interfacing-async-tasks-with-synchronous-code/43619) + +## Motivation + +Swift APIs often provide asynchronous code execution by way of a callback. This +may occur either because the code itself was written prior to the introduction +of async/await, or (more interestingly in the long term) because it ties in +with some other system that is primarily event-driven. In such cases, one may +want to provide an async interface to clients while using callbacks internally. +In these cases, the calling async task needs to be able to suspend itself, +while providing a mechanism for the event-driven synchronous system to resume +it in response to an event. + +## Proposed solution + +The library will provide APIs to get a **continuation** for the current +asynchronous task. Getting the task's continuation suspends the task, and +produces a value that synchronous code can then use a handle to resume the +task. Given a completion callback based API like: + +```swift +func beginOperation(completion: (OperationResult) -> Void) +``` + +we can turn it into an `async` interface by suspending the task and using its +continuation to resume it when the callback is invoked, turning the argument +passed into the callback into the normal return value of the async function: + +```swift +func operation() async -> OperationResult { + // Suspend the current task, and pass its continuation into a closure + // that executes immediately + return await withUnsafeContinuation { continuation in + // Invoke the synchronous callback-based API... + beginOperation(completion: { result in + // ...and resume the continuation when the callback is invoked + continuation.resume(returning: result) + }) + } +} +``` + +## Detailed design + +### Raw unsafe continuations + +The library provides two functions, `withUnsafeContinuation` and +`withUnsafeThrowingContinuation`, that allow one to call into a callback-based +API from inside async code. Each function takes an *operation* closure, +which is expected to call into the callback-based API. The closure +receives a continuation instance that must be resumed by the callback, +either to provide the result value or (in the throwing variant) the thrown +error that becomes the result of the `withUnsafeContinuation` call when the +async task resumes: + + +```swift +struct UnsafeContinuation { + func resume(returning: T) + func resume(throwing: E) + func resume(with result: Result) +} + +extension UnsafeContinuation where T == Void { + func resume() { resume(returning: ()) } +} + +extension UnsafeContinuation where E == Error { + // Allow covariant use of a `Result` with a stricter error type than + // the continuation: + func resume(with result: Result) +} + +func withUnsafeContinuation( + _ operation: (UnsafeContinuation) -> () +) async -> T + +func withUnsafeThrowingContinuation( + _ operation: (UnsafeContinuation) throws -> () +) async throws -> T +``` + +`withUnsafe*Continuation` will run its `operation` argument immediately in the +task's current context, passing in a *continuation* value that can be +used to resume the task. The `operation` function must arrange for the +continuation to be resumed at some point in the future; after the `operation` +function returns, the task is suspended. The task must then be brought out +of the suspended state by invoking one of the continuation's `resume` methods. +Note that `resume` immediately returns control to the caller after transitioning +the task out of its suspended state; the task itself does not actually resume +execution until its executor reschedules it. The argument to +`resume(returning:)` becomes the return value of `withUnsafe*Continuation` +when the task resumes execution. +`resume(throwing:)` can be used instead to make the task resume by propagating +the given error. As a convenience, given a `Result`, `resume(with:)` can be used +to resume the task by returning normally or raising an error according to the +state of the `Result`. If the `operation` raises an uncaught error before +returning, this behaves as if the operation had invoked `resume(throwing:)` with +the error. + +If the return type of `withUnsafe*Continuation` is `Void`, one must specify +a value of `()` when calling `resume(returning:)`. Doing so produces some +unsightly code, so `Unsafe*Continuation` has an extra member `resume()` +that makes the function call easier to read. + +After invoking `withUnsafeContinuation`, exactly one `resume` method must be +called *exactly-once* on every execution path through the program. +`Unsafe*Continuation` is an unsafe interface, so it is undefined behavior if +a `resume` method is invoked on the same continuation more than once. The +task remains in the suspended state until it is resumed; if the continuation +is discarded and never resumed, then the task will be left suspended until +the process ends, leaking any resources it holds. +Wrappers can provide checking for these misuses of continuations, and the +library will provide one such wrapper, discussed below. + +Using the `Unsafe*Continuation` API, one may for example wrap such +(purposefully convoluted for the sake of demonstrating the flexibility of +the continuation API) function: + +```swift +func buyVegetables( + shoppingList: [String], + // a) if all veggies were in store, this is invoked *exactly-once* + onGotAllVegetables: ([Vegetable]) -> (), + + // b) if not all veggies were in store, invoked one by one *one or more times* + onGotVegetable: (Vegetable) -> (), + // b) if at least one onGotVegetable was called *exactly-once* + // this is invoked once no more veggies will be emitted + onNoMoreVegetables: () -> (), + + // c) if no veggies _at all_ were available, this is invoked *exactly once* + onNoVegetablesInStore: (Error) -> () +) +// returns 1 or more vegetables or throws an error +func buyVegetables(shoppingList: [String]) async throws -> [Vegetable] { + try await withUnsafeThrowingContinuation { continuation in + var veggies: [Vegetable] = [] + + buyVegetables( + shoppingList: shoppingList, + onGotAllVegetables: { veggies in continuation.resume(returning: veggies) }, + onGotVegetable: { v in veggies.append(v) }, + onNoMoreVegetables: { continuation.resume(returning: veggies) }, + onNoVegetablesInStore: { error in continuation.resume(throwing: error) }, + ) + } +} + +let veggies = try await buyVegetables(shoppingList: ["onion", "bell pepper"]) +``` + +Thanks to weaving the right continuation resume calls into the complex +callbacks of the `buyVegetables` function, we were able to offer a much nicer +overload of this function, allowing async code to interact with this function in +a more natural straight-line way. + +### Checked continuations + +`Unsafe*Continuation` provides a lightweight mechanism for interfacing +sync and async code, but it is easy to misuse, and misuse can corrupt the +process state in dangerous ways. In order to provide additional safety and +guidance when developing interfaces between sync and async code, the +library will also provide a wrapper which checks for invalid use of the +continuation: + +```swift +struct CheckedContinuation { + func resume(returning: T) + func resume(throwing: E) + func resume(with result: Result) +} + +extension CheckedContinuation where T == Void { + func resume() +} + +extension CheckedContinuation where E == Error { + // Allow covariant use of a `Result` with a stricter error type than + // the continuation: + func resume(with result: Result) +} + +func withCheckedContinuation( + _ operation: (CheckedContinuation) -> () +) async -> T + +func withCheckedThrowingContinuation( + _ operation: (CheckedContinuation) throws -> () +) async throws -> T +``` + +The API is intentionally identical to the `Unsafe` variants, so that code +can switch easily between the checked and unchecked variants. For instance, +the `buyVegetables` example above can opt into checking merely by turning +its call of `withUnsafeThrowingContinuation` into one of `withCheckedThrowingContinuation`: + +```swift +// returns 1 or more vegetables or throws an error +func buyVegetables(shoppingList: [String]) async throws -> [Vegetable] { + try await withCheckedThrowingContinuation { continuation in + var veggies: [Vegetable] = [] + + buyVegetables( + shoppingList: shoppingList, + onGotAllVegetables: { veggies in continuation.resume(returning: veggies) }, + onGotVegetable: { v in veggies.append(v) }, + onNoMoreVegetables: { continuation.resume(returning: veggies) }, + onNoVegetablesInStore: { error in continuation.resume(throwing: error) }, + ) + } +} +``` + +Instead of leading to undefined behavior, `CheckedContinuation` will instead +trap if the program attempts to resume the continuation multiple times. +`CheckedContinuation` will also log a warning if the continuation +is discarded without ever resuming the task, which leaves the task stuck in its +suspended state, leaking any resources it holds. These checks happen regardless +of the optimization level of the program. + +## Additional examples + +Continuations can be used to interface with more complex event-driven +interfaces than callbacks as well. As long as the entirety of the process +follows the requirement that the continuation be resumed exactly once, there +are no other restrictions on where the continuation can be resumed. For +instance, an `Operation` implementation can trigger resumption of a +continuation when the operation completes: + +```swift +class MyOperation: Operation { + let continuation: UnsafeContinuation + var result: OperationResult? + + init(continuation: UnsafeContinuation) { + self.continuation = continuation + } + + /* rest of operation populates `result`... */ + + override func finish() { + continuation.resume(returning: result!) + } +} + +func doOperation() async -> OperationResult { + return await withUnsafeContinuation { continuation in + MyOperation(continuation: continuation).start() + } +} +``` + +Using APIs from the [structured concurrency proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md), +one can wrap up a `URLSession` in a task, allowing the task's cancellation +to control cancellation of the session, and using a continuation to respond +to data and error events fired by the network activity: + +```swift +func download(url: URL) async throws -> Data? { + var urlSessionTask: URLSessionTask? + + return try Task.withCancellationHandler { + urlSessionTask?.cancel() + } operation: { + let result: Data? = try await withUnsafeThrowingContinuation { continuation in + urlSessionTask = URLSession.shared.dataTask(with: url) { data, _, error in + if case (let cancelled as NSURLErrorCancelled)? = error { + continuation.resume(returning: nil) + } else if let error = error { + continuation.resume(throwing: error) + } else { + continuation.resume(returning: data) + } + } + urlSessionTask?.resume() + } + if let result = result { + return result + } else { + Task.cancel() + return nil + } + } +} +``` + +It is also possible for wrappers around callback based APIs to respect their parent/current tasks's cancellation, as follows: + +```swift +func fetch(items: Int) async throws -> [Items] { + let worker = ... + return try Task.withCancellationHandler( + handler: { worker?.cancel() } + ) { + return try await withUnsafeThrowingContinuation { c in + worker.work( + onNext: { value in c.resume(returning: value) }, + onCancelled: { value in c.resume(throwing: CancellationError()) }, + ) + } + } +} +``` + +If tasks were allowed to have instances, which is under discussion in the structured concurrency proposal, it would also be possible to obtain the task in which the fetch(items:) function was invoked and call isCanceled on it whenever the insides of the withUnsafeThrowingContinuation would deem it worthwhile to do so. + +## Alternatives considered + +### Name `CheckedContinuation` just `Continuation` + +We could position `CheckedContinuation` as the "default" API for doing +sync/async interfacing by leaving the `Checked` word out of the name. This +would certainly be in line with the general philosophy of Swift that safe +interfaces are preferred, and unsafe ones used selectively where performance +is an overriding concern. There are a couple of reasons to hesitate at doing +this here, though: + +- Although the consequences of misusing `CheckedContinuation` are not as + severe as `UnsafeContinuation`, it still only does a best effort at checking + for some common misuse patterns, and it does not render the consequences of + continuation misuse entirely moot: dropping a continuation without resuming + it will still leak the un-resumed task, and attempting to resume a + continuation multiple times will still cause the information passed through + the continuation to be lost. It is still a serious programming error if + a `with*Continuation` operation misuses the continuation; + `CheckedContinuation` only helps make the error more apparent. +- Naming a type `Continuation` now might take the "good" name away if, + after we have move-only types at some point in the future, we want to + introduce a continuation type that statically enforces the exactly-once + property. + +### Don't expose `UnsafeContinuation` + +One could similarly make an argument that `UnsafeContinuation` shouldn't be +exposed at all, since the `Checked` form can always be used instead. We think +that being able to avoid the cost of checking when interacting with +performance-sensitive APIs is valuable, once users have validated that their +interfaces to those APIs are correct. + +### Have `CheckedContinuation` trap on all misuses, or log all misuses + +`CheckedContinuation` is proposed to trap when the program attempts to +resume the same continuation twice, but only log a warning if a continuation +is abandoned without getting resumed. We think this is the right tradeoff +for these different situations for the following reasons: + +- With `UnsafeContinuation`, resuming multiple times corrupts the process and + leaves it in an undefined state. By trapping when the task is resumed + multiple times, `CheckedContinuation` turns undefined behavior into a well- + defined trap situation. This is analogous to other checked/unchecked + pairings in the standard library, such as `!` vs. `unsafelyUnwrapped` for + `Optional`. +- By contrast, failing to resume a continuation with `UnsafeContinuation` + does not corrupt the task, beyond leaking the suspended task's resources; + the rest of the program can continue executing normally. Furthermore, + the only way we can currently detect and report such a leak is by using + a class `deinit` in its implementation. The precise moment at which such + a deinit would execute is not entirely predictable because of refcounting + variability from ARC optimization. If `deinit` were made to trap, whether that + trap is executed and when could vary with optimization level, which we + don't think would lead to a good experience. + +### Expose more `Task` API on `*Continuation`, or allow a `Handle` to be recovered from a continuation + +The full `Task` and `Handle` API provides additional control over the task +state to holders of the handle, particularly the ability to query and set +cancellation state, as well as await the final result of the task, and one +might wonder why the `*Continuation` types do not also expose this functionality. +The role of a `Continuation` is very different from a `Handle`, in that a handle +represents and controls the entire lifetime of the task, whereas a continuation +only represents a *single suspension point* in the lifetime of the task. +Furthermore, the `*Continuation` API is primarily designed to allow for +interfacing with code outside of Swift's structured concurrency model, and +we believe that interactions between tasks are best handled inside that model +as much as possible. + +Note that `*Continuation` also does not strictly need direct support for any +task API on itself. If, for instance, someone wants a task to cancel itself +in response to a callback, they can achieve that by funneling a sentinel +through the continuation's resume type, such as an Optional's `nil`: + +```swift +let callbackResult: Result? = await withUnsafeContinuation { c in + someCallbackBasedAPI( + completion: { c.resume($0) }, + cancellation: { c.resume(nil) }) +} + +if let result = callbackResult { + process(result) +} else { + cancel() +} +``` + +### Provide API to resume the task immediately to avoid "queue-hopping" + +Some APIs, in addition to taking a completion handler or delegate, also allow +the client to control *where* that completion handler or delegate's methods are +invoked; for instance, some APIs on Apple platforms take an argument for the +dispatch queue the completion handler should be invoked by. In these cases, +it would be optimal if the original API could resume the task directly on the +dispatch queue (or whatever other scheduling mechanism, such as a thread or +run loop) that the task would normally be resumed on by its executor. To +enable this, we could provide a variant of `with*Continuation` that, in +addition to providing a continuation, also provides the dispatch queue that +the task expects to be resumed on. The `*Continuation` type in turn could +provide an `unsafeResumeImmediately` set of APIs, which would immediately +resume execution of the task on the current thread. This would enable something +like this: + +```swift +// Given an API that takes a queue and completion handler: +func doThingAsynchronously(queue: DispatchQueue, completion: (ResultType) -> Void) + +// We could wrap it in a Swift async function like: +func doThing() async -> ResultType { + await withUnsafeContinuationAndCurrentDispatchQueue { c, queue in + // Schedule to resume on the right queue, if we know it + doThingAsynchronously(queue: queue) { + c.unsafeResumeImmediately(returning: $0) + } + } +} +``` + +However, such an API would have to be used very carefully; the programmer +would have to be careful that `unsafeResumeImmediately` is in fact invoked +in the correct context, and that it is safe to take over control of the +current thread from the caller for a potentially unbounded amount of time. +If the task is resumed in the wrong context, it will break assumptions in the +written code as well as those made by the compiler and runtime, which will +lead to subtle bugs that would be difficult to diagnose. We can investigate +this as an addition to the core proposal, if "queue hopping" in continuation- +based adapters turns out to be a performance problem in practice. + +## Revision history + +Third revision: + +- Replaced separate `*Continuation` and `*ThrowingContinuation` types with a + single `Continuation` type parameterized on the error type. +- Added a convenience `resume()` equivalent to `resume(returning: ())` for + continuations with a `Void` return type. +- Changed `with*ThrowingContinuation` to take an `operation` block that may + throw, and to immediately resume the task throwing the error if an uncaught + error propagates from the operation. + +Second revision: + +- Clarified the execution behavior of `with*Continuation` and + `*Continuation.resume`, namely that `with*Continuation` immediately executes + its operation argument in the current context before suspending the task, + and that `resume` immediately returns to its caller after un-suspending the + task, leaving the task to be scheduled by its executor. +- Removed an unnecessary invariant on when `resume` must be invoked; it is valid + to invoke it exactly once at any point after the `with*Continuation` operation + has started executing; it does not need to run exactly when the operation + returns. +- Added "future direction" discussion of a potential more advanced API that + could allow continuations to directly resume their task when the correct + dispatch queue to do so is known. +- Added `resume()` on `Void`-returning `Continuation` types. diff --git a/proposals/0301-package-editing-commands.md b/proposals/0301-package-editing-commands.md new file mode 100644 index 0000000000..9778c01f1c --- /dev/null +++ b/proposals/0301-package-editing-commands.md @@ -0,0 +1,179 @@ +# Package Editor Commands + +* Proposal: [SE-0301](0301-package-editing-commands.md) +* Authors: [Owen Voorhees](https://github.com/owenv) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 6.0)** +* Implementation: [apple/swift-package-manager#3034](https://github.com/apple/swift-package-manager/pull/3034) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modification-se-0301-package-editor-commands/45069) + +## Introduction + +Because Swift package manifests are written in Swift using the PackageDescription API, it is difficult to automate common tasks like adding a new product, target, or dependency. This proposal introduces new `swift package` subcommands to perform some common editing tasks which can streamline users' workflows and enable new higher-level tools. + +Forums Discussion: https://forums.swift.org/t/pitch-package-editor-commands/42224/ + +## Motivation + +There are a number of reasons someone might want to make changes to a package using a CLI or library interface instead of editing a manifest by hand: + +- In some situations, it's less error-prone than editing the manifest manually. Package authors could provide a one line command in their README to easily integrate the latest version as a dependency. +- Because more of the process is automated, users would no longer need to remember details of the package layout convention, like which files and folders they need to create when adding a new library target. +- Using libSwiftPM, IDEs could offer to update the manifest automatically when the user tries to import a missing dependency or create a new target. +- Users could add packages from their package collections as dependencies by name instead of URL + +Additionally, many other package managers offer similar features: +- npm's [`npm install`](https://docs.npmjs.com/specifying-dependencies-and-devdependencies-in-a-package-json-file#adding-dependencies-to-a-packagejson-file-from-the-command-line) command for adding dependencies to its `package.json` +- Tools like [cargo-edit](https://github.com/killercup/cargo-edit) for editing the `cargo.toml` format used by Rust +- Elm's [`elm install`](https://elmprogramming.com/elm-install.html) command for adding dependencies to `elm.json` + +## Proposed solution + +This proposal introduces three new `swift package` subcommands: `add-product`, `add-target`, and `add-dependency`, which edit the manifest of the current package. Together, these encompass many of the most common editing operations performed by users when working on a package. + +## Detailed design + +### New Commands + +The following subcommands will be added to `swift package`: + +`swift package add-product [--type ] [--targets ]` + +- **name**: The name of the new product. +- **type**: _executable_, _library_, _static-library_, or _dynamic-library_. If unspecified, this will default to _library_. +- **targets**: A space separated list of target names to to add to the new product. +*** +`swift package add-target [--type ] [--no-test-target] [--dependencies ] [--url ] [--path ] [--checksum ]` +- **name**: The name of the new target. +- **type**: _library_, _executable_, _test_, or _binary_. If unspecified, this will default to _library_. Adding system library targets will not be supported by the initial version of the CLI due to their often complex configuration requirements. +- **--no-test-target**: By default, a test target is added for each library target unless this flag is present. +- **dependencies**: A space separated list of target dependency names. +- **url/path**: The URL for a remote binary target or path for a local one. +- **checksum**: The checksum for a remote binary target. + +In addition to editing the manifest, the add-target command will create the appropriate `Sources` or `Tests` subdirectories for new targets. +*** +`swift package add-dependency [--exact ] [--revision ] [--branch ] [--from ] [--up-to-next-minor-from ]` +- **dependency**: This may be the URL of a remote package, the path to a local package, or the name of a package in one of the user's package collections. + +The following options can be used to specify a package dependency requirement: +- **--exact **: Specifies a `.exact()` requirement in the manifest. +- **--revision **: Specifies a `.revision()` requirement in the manifest. +- **--branch **: Specifies a `.branch()` requirement in the manifest. +- **--up-to-next-minor-from **: Specifies a `.upToNextMinor()` requirement in the manifest. +- **--from **: Specifies a `.upToNextMajor()` requirement in the manifest when it appears alone. Optionally, **--to ** may be added to specify a custom range requirement, or **--through** may be added to specify a custom closed range requirement. + +If no requirement is specified, the command will default to a `.upToNextMajor` requirement on the latest version of the package. + +### Compatibility + +These new commands will be restricted to only operate on package manifests having a `swift-tools-version` of 5.2 or later. This decision was made to reduce the complexity of the feature, as there were a number of major changes to the `PackageDescription` module in Swift 5.2. It is expected that as the API continues to evolve in the future, support for editing manifests with older tools versions will be maintained whenever possible. + +### Editing Non-Declarative Manifests + +The subcommands described above support editing all fully declarative parts of a package manifest. For the purposes of this proposal, an entry in a manifest is considered fully declarative if it consists only of literals and calls to factory methods and initializers in the `PackageDescription` module. The vast majority of products, targets, and dependencies sections in existing package manifests are fully declarative. + +However, because manifests may contain arbitrary Swift code, not all of them meet this criteria. The proposed subcommands will be capable of making edits to many of these manifests, but they will do so on a best-effort basis. If the requested editing operation cannot be performed successfully, they will report an error and leave the manifest unchanged. + +### Example + +Given a simple initial package: +```swift +// swift-tools-version:5.3 +import PackageDescription + +// Description of my package +let package = Package( + name: "MyPackage", + targets: [ + .target( + name: "MyLibrary", // Utilities + dependencies: [] + ), + ] +) +``` + +The following commands can be used to add dependencies, targets, and products: +``` +swift package add-dependency https://github.com/apple/swift-argument-parser +swift package add-target MyLibraryTests --type test --dependencies MyLibrary +swift package add-target MyExecutable --type executable --dependencies MyLibrary ArgumentParser +swift package add-product MyLibrary --targets MyLibrary +``` +Resulting in this manifest: +```swift +// swift-tools-version:5.3 +import PackageDescription + +// Description of my package +let package = Package( + name: "MyPackage", + products: [ + .library( + name: "MyLibrary", + targets: [ + "MyLibrary", + ] + ), + ], + dependencies: [ + .package(name: "swift-argument-parser", url: "https://github.com/apple/swift-argument-parser", .upToNextMajor(from: "0.3.1")), + ], + targets: [ + .target( + name: "MyLibrary", // Utilities + dependencies: [] + ), + .testTarget( + name: "MyLibraryTests", + dependencies: [ + "MyLibrary", + ] + ), + .target( + name: "MyExecutable", + dependencies: [ + "MyLibrary", + .product(name: "ArgumentParser", package: "swift-argument-parser") + ] + ), + ] +) +``` + +Additionally, `Sources/MyExecutable/main.swift` and `Tests/MyLibraryTests/MyLibraryTests.swift` will be created for the new targets. + +## Security + +This proposal has minimal impact on the security of the package manager. Packages added using the `add-dependency` subcommand will be fetched and their manifests will be loaded, but this is no different than if the user manually edited the manifest to include them. + +## Impact on Existing Packages + +Because this proposal only includes new editing commands, it has no impact on package semantics. + +## Alternatives considered + +One alternative considered was not including this functionality at all. It's worth noting that maintaining package editor functionality over time and adapting it to changes in the `PackageDescription` API will require a nontrivial effort. However, the benefits of including the functionality are substantial enough that it seems like a worthwhile tradeoff. Beyond that, much of the required infrastructure can be reused to enable other new features like source locations in manifest diagnostics and commands like `swift package upgrade` discussed in the "Future Directions" section. + +Another suggested alternative was to provide an interface to run scripts which mutate the `Package` object after the manifest is run. A full discussion of such a feature is outside the scope of this proposal, but it would likely allow more flexible edits. However, this approach has several downsides. To persist edits after running such a script, the entire `Package.swift` would need to be rewritten, which would drop comments, the content of inactive conditional compilation blocks, and any other skipped branches in the manifest code. The proposed subcommands, which rely on a syntax-level transformation, are able to make more specific edits to the manifest and avoid these issues. + +--- + +During the review, an alternative spelling for the commands was proposed: +``` +swift package product add ... +swift package target add ... +swift package dependency add ... +``` +These spellings could scale better to introduce new 'verbs' in addition to `add` (for example, `rename`, `delete`, etc.), and there's some prior art (`git remote add`, for example). However, they'd also introduce a fourth level of subcommand nesting which could impact help output and ease-of-use, and might imply the existence of additional functionality which isn't likely to be added in the near future. Overall, both approaches to the subcommand spellings have legitimate tradeoffs, so it was decided to keep the original spellings. + +## Future Directions + +### Support for Deleting Products/Targets/Dependencies and Renaming Products/Targets + +This functionality was considered, but ultimately removed from the scope of the initial proposal. Most of these editing operations appear to be fairly uncommon, so it seems better to wait and see how the new commands are used in practice before rolling out more. + +### Add a `swift package upgrade` Command + +A hypothetical `swift package upgrade` command could automatically update the version specifiers of package dependencies to newer versions, similar to `npm upgrade`. This is another manifest editing operation very commonly provided by other package managers. diff --git a/proposals/0302-concurrent-value-and-concurrent-closures.md b/proposals/0302-concurrent-value-and-concurrent-closures.md new file mode 100644 index 0000000000..4012686d59 --- /dev/null +++ b/proposals/0302-concurrent-value-and-concurrent-closures.md @@ -0,0 +1,674 @@ +# `Sendable` and `@Sendable` closures + +* Proposal: [SE-0302](0302-concurrent-value-and-concurrent-closures.md) +* Authors: [Chris Lattner](https://github.com/lattner), [Doug Gregor](https://github.com/douggregor) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#35264](https://github.com/apple/swift/pull/35264) +* Major Contributors: Dave Abrahams, Paul Cantrell, Matthew Johnson, John McCall +* Review: ([first review](https://forums.swift.org/t/se-0302-Sendable-and-concurrent-closures/44919)) ([revision announcement](https://forums.swift.org/t/returned-for-revision-se-0302-concurrentvalue-and-concurrent-closures/45251)) ([second review](https://forums.swift.org/t/se-0302-second-review-sendable-and-sendable-closures/45253)) ([acceptance](https://forums.swift.org/t/accepted-se-0302-sendable-and-sendable-closures/45786)) + +## Contents + + * [Introduction](#introduction) + * [Motivation](#motivation) + * [💖 Swift Value Semantics](#-swift--value-semantics) + * [Value Semantic Composition](#value-semantic-composition) + * [Higher Order Functional Programming](#higher-order-functional-programming) + * [Immutable Classes](#immutable-classes) + * [Internally Synchronized Reference Types](#internally-synchronized-reference-types) + * [“Transferring” Objects Between Concurrency Domains](#transferring-objects-between-concurrency-domains) + * [Deep Copying Classes](#deep-copying-classes) + * [Motivation Conclusion](#motivation-conclusion) + * [Proposed Solution Detailed Design](#proposed-solution--detailed-design) + * [Marker Protocols](#marker-protocols) + * [Sendable Protocol](#sendable-protocol) + * [Tuple conformance to Sendable](#tuple-conformance-to-sendable) + * [Metatype conformance to Sendable](#metatype-conformance-to-sendable) + * [Sendable conformance checking for structs and enums](#sendable-conformance-checking-for-structs-and-enums) + * [Implicit struct/enum conformance to Sendable](#implicit-structenum-conformance-to-sendable) + * [Sendable conformance checking for classes](#sendable-conformance-checking-for-classes) + * [Actor types](#actor-types) + * [Key path literals](#key-path-literals) + * [New @Sendable attribute for functions](#new-sendable-attribute-for-functions) + * [Inference of @Sendable for Closure Expressions](#inference-of-sendable-for-closure-expressions) + * [Thrown errors](#thrown-errors) + * [Adoption of Sendable by Standard Library Types](#adoption-of-sendable-by-standard-library-types) + * [Support for Imported C / Objective-C APIs](#support-for-imported-c--objective-c-apis) + * [Future Work / Follow-on Projects](#future-work--follow-on-projects) + * [Adaptor Types for Legacy Codebases](#adaptor-types-for-legacy-codebases) + * [Objective-C Framework Support](#objective-c-framework-support) + * [Interaction of Actor self and @Sendable closures](#interaction-of-actor-self-and-sendable-closures) + * [Marker protocols as custom attributes](#marker-protocols-as-custom-attributes) + * [Source Compatibility](#source-compatibility) + * [Effect on API resilience](#effect-on-api-resilience) + * [Alternatives Considered](#alternatives-considered) + * [Exotic Type System Features](#exotic-type-system-features) + * [Support an explicit copy hook](#support-an-explicit-copy-hook) + * [Conclusion](#conclusion) + * [Revision history](#revision-history) + +## Introduction + +A key goal of the Swift Concurrency effort is to “provide a mechanism for isolating state in concurrent programs to eliminate data races.” Such a mechanism will be a major progression for widely used programming languages — most of them provide concurrent programming abstractions in a way that subjects programmers to a wide range of bugs, including race conditions, deadlocks and other problems. + +This proposal describes an approach to address one of the challenging problems in this space — how to type check value passing between structured concurrency constructs and actors messages. As such, this is a unifying theory that provides some of the underlying type system mechanics that make them both safe and work well together. + +This implementation approach involves a marker protocol named `Sendable`, as well as a `@Sendable` attribute that may be applied to functions. + +## Motivation + +Each actor instance and structured concurrency task in a program represents an “island of single threaded-ness”, which makes them a natural synchronization point that holds a bag of mutable state. These perform computation in parallel with other tasks, but we want the vast majority of code in such a system to be synchronization free — building on the logical independence of the actor, and using its mailbox as a synchronization point for its data. + +As such, a key question is: “when and how do we allow data to be transferred between concurrency domains?” Such transfers occur in arguments and results of actor method calls and tasks created by structured concurrency, for example. + +The Swift Concurrency features aspire to build a safe and powerful programming model. We want to achieve three things: + +1. We want Swift programmers to get a static compiler error when they try to pass across concurrency domains that could introduce unprotected shared mutable state. +2. We want advanced programmers to be able to implement libraries with sophisticated techniques (e.g. a concurrent hash table) that can be used in a safe way by others. +3. We need to embrace the existing world, which contains a lot of code that wasn’t designed with the Swift Concurrency model in mind. We need a smooth and incremental migration story. + +Before we jump into the proposed solution, let’s take a look at some common cases that we would like to be able to model along with the opportunities and challenges of each. This will help us reason about the design space we need to cover. + +### 💖 Swift + Value Semantics + +The first kind of type we need to support are simple values like integers. These can be trivially passed across concurrency domains because they do not contain pointers. + +Going beyond this, Swift has a strong emphasis on types with [value semantics](https://en.wikipedia.org/wiki/Value_semantics), which are safe to transfer across concurrent boundaries. Except for classes, Swift’s mechanisms for type composition provide value semantics when their elements do. This includes generic structs, as well as its core collections: for example, `Dictionary` can be directly shared across concurrency domains. Swift’s Copy on Write approach means that collections can be transferred without proactive data copying of their representations — an extremely powerful fact that I believe will make the Swift concurrency model more efficient than other systems in practice. + +However, everything isn’t simple here: the core collections can **not** be safely transferred across concurrency domains when they contain general class references, closures that capture mutable state, and other non-value types. We need a way to differentiate between the cases that are safe to transfer and those that are not. + +### Value Semantic Composition + +Structs, enums and tuples are the primary mode for composition of values in Swift. These are all safe to transfer across concurrency domains — so long as the data they contain is itself safe to transfer. + +### Higher Order Functional Programming + +It is common in Swift and other languages with functional programming roots to use [higher-order programming](https://en.wikipedia.org/wiki/Higher-order_function), where you pass functions to other functions. Functions in Swift are reference types, but many functions are perfectly safe to pass across concurrency domains — for example, those with an empty capture list. + +There are many useful reasons why you’d want to send bits of computation between concurrency domains in the form of a function — even trivial algorithms like `parallelMap` need this. This occurs at larger scale as well — for example, consider an actor example like this: + +```swift +actor MyContactList { + func filteredElements(_ fn: (ContactElement) -> Bool) async -> [ContactElement] { … } +} +``` + +Which could then be used like so: + +```swift +// Closures with no captures are ok! +list = await contactList.filteredElements { $0.firstName != "Max" } + +// Capturing a 'searchName' string by value is ok, because strings are +// ok to pass across concurrency domains. +list = await contactList.filteredElements { + [searchName] in $0.firstName == searchName +} +``` + +We feel that it is important to enable functions to be passed across concurrency domains, but we are also concerned that we should not allow capturing local state _by reference_ in these functions, and we should not allow capturing unsafe things by value. Both would introduce memory safety problems. + +### Immutable Classes + +One common and efficient design pattern in concurrent programming is to build immutable data structures — it is perfectly safe to transfer a reference to a class across concurrency domains if the state within it never mutates. This design pattern is extremely efficient (no synchronization beyond ARC is required), can be used to build [advanced data structures](https://en.wikipedia.org/wiki/Persistent_data_structure), and is widely explored by the pure-functional language community. + +### Internally Synchronized Reference Types + +A common design pattern in concurrent systems is for a class to provide a “thread-safe” API: they protect their state with explicit synchronization (mutexes, atomics, etc). Because the public API to the class is safe to use from multiple concurrency domains, the reference to the class can be directly transferred safely. + +References to actor instances themselves are an example of this: they are safe to pass between concurrency domains by passing a pointer, since the mutable state within an actor is implicitly protected by the actor mailbox. + +### “Transferring” Objects Between Concurrency Domains + +A fairly common pattern in concurrent systems is for one concurrency domain to build up a data structure containing unsynchronized mutable state, then “hand it off” to a different concurrency domain to use by transferring the raw pointer. This is correct without synchronization if (and only if) the sender stops using the data that it built up — the result is that only the sender or receiver dynamically accesses the mutable state at a time. + +There are both safe and unsafe ways to achieve this, e.g. see the discussion about “exotic” type systems in the [Alternatives Considered](#alternatives-considered) section at the end. + +### Deep Copying Classes + +One safe way to transfer reference types is to make a deep copy of the data structures, ensuring that the source and destination concurrency domains each have their own copy of mutable state. This can be expensive for large structures, but is/was commonly used in some Objective-C frameworks. General consensus is that this should be _explicit_, not something implicit in the definition of a type. + +### Motivation Conclusion + +This is just a sampling of patterns, but as we can see, there are a wide range of different concurrent design patterns in widespread use. The design center of Swift around value types and encouraging use of structs is a very powerful and useful starting point, but we need to be able to reason about the complex cases as well — both for communities that want to be able express high performance APIs for a given domain but also because we need to work with legacy code that won’t get rewritten overnight. + +As such, it is important to consider approaches that allow library authors to express the intent of their types, it is important for app programmers to be able to work with uncooperative libraries retroactively, and it is also important that we provide safety as well as unsafe escape hatches so we can all just “get stuff done” in the face of an imperfect world that is in a process of transition. + +Finally, our goal is for Swift (in general and in this specific case) to be a highly principled system that is sound and easy to use. In 20 years, many new libraries will be built for Swift and its ultimate concurrency model. These libraries will be built around value semantic types, but should also allow expert programmers to deploy state of the art techniques like lock-free algorithms, use immutable types, or whatever other design pattern makes sense for their domain. We want users of these APIs to not have to care how they are implemented internally. + +## Proposed Solution + Detailed Design + +The high level design of this proposal revolves around a `Sendable` marker protocol, adoption of `Sendable` by standard library types, and a new `@Sendable` attribute for functions. + +Beyond the basic proposal, in the future it could make sense to add a set of adapter types to handle legacy compatibility cases, and first class support for Objective-C frameworks. These are described in the following section. + +### Marker Protocols + +This proposal introduces the concept of a “marker” protocol, which indicates that the protocol has some semantic property but is entirely a compile-time notion that does not have any impact at runtime. Marker protocols have the following restrictions: + +* They cannot have requirements of any kind. +* They cannot inherit from non-marker protocols. +* A marker protocol cannot be named as the type in an `is` or `as?` check (e.g., `x as? Sendable` is an error). +* A marker protocol cannot be used in a generic constraint for a conditional protocol conformance to a non-marker protocol. + +We think this is a generally useful feature, but believe it should be a compiler-internal feature at this point. As such, we explain it and use this concept with the “`@_marker`” attribute syntax below. + +### `Sendable` Protocol + +The core of this proposal is a marker protocol defined in the Swift standard library , which has special conformance checking rules: + +```swift +@_marker +protocol Sendable {} +``` + +It is a good idea for types to conform to the `Sendable` protocol when they are designed so all of their public API is safe to use across concurrency domains. This is true for example, when there are no public mutators, if public mutators are implemented with COW, or if they are implemented with internal locking or some other mechanism. Types may of course have internal implementation details based on local mutation if they have locking or COW as part of their public API. + +The compiler rejects any attempts to pass data across concurrency domains, e.g. rejecting cases where the argument or result of an actor message send or structured concurrency call does not conform to the `Sendable` protocol: + +```swift +actor SomeActor { + // async functions are usable *within* the actor, so this + // is ok to declare. + func doThing(string: NSMutableString) async {...} +} + +// ... but they cannot be called by other code not protected +// by the actor's mailbox: +func f(a: SomeActor, myString: NSMutableString) async { + // error: 'NSMutableString' may not be passed across actors; + // it does not conform to 'Sendable' + await a.doThing(string: myString) +} +``` + +The `Sendable` protocol models types that are allowed to be safely passed across concurrency domains by copying the value. This includes value-semantic types, references to immutable reference types, internally synchronized reference types, `@Sendable` closures, and potentially other future type system extensions for unique ownership etc. + +Note that incorrect conformance to this protocol can introduce bugs in your program (just as an incorrect implementation of `Hashable` can break invariants), which is why the compiler checks conformance (see below). + +#### Tuple conformance to `Sendable` + +Swift has [hard coded conformances for tuples](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0283-tuples-are-equatable-comparable-hashable.md) to specific protocols, and this should be extended to `Sendable`, when the tuples elements all conform to `Sendable`. + +#### Metatype conformance to `Sendable` + +Metatypes (such as `Int.Type`, the type produced by the expression `Int.self`) always conform to `Sendable`, because they are immutable. + +#### `Sendable` conformance checking for structs and enums + +`Sendable` types are extremely common in Swift and aggregates of them are also safe to transfer across concurrency domains. As such, the Swift compiler allows direct conformance to `Sendable` for structs and enums that are compositions of other `Sendable` types: + +```swift +struct MyPerson : Sendable { var name: String, age: Int } +struct MyNSPerson { var name: NSMutableString, age: Int } + +actor SomeActor { + // Structs and tuples are ok to send and receive! + public func doThing(x: MyPerson, y: (Int, Float)) async {..} + + // error if called across actor boundaries: MyNSPerson doesn't conform to Sendable! + public func doThing(x: MyNSPerson) async {..} +} +``` + +While this is convenient, we would like to slightly increase friction of protocol adoption for cases that require more thought. As such, the compiler rejects conformance of structs and enums to the `Sendable` protocol when one of their members (or associated values) does not itself conform to `Sendable` (or is not known to conform to `Sendable` through a generic constraint): + +```swift +// error: MyNSPerson cannot conform to Sendable due to NSMutableString member. +// note: add '@unchecked' if you know what you're doing. +struct MyNSPerson : Sendable { + var name: NSMutableString + var age: Int +} + +// error: MyPair cannot conform to Sendable due to 'T' member which may not itself be a Sendable +// note: see below for use of conditional conformance to model this +struct MyPair : Sendable { + var a, b: T +} + +// use conditional conformance to model generic types +struct MyCorrectPair { + var a, b: T +} + +extension MyCorrectPair: Sendable where T: Sendable { } +``` + +As mentioned in the compiler diagnostic, any type can override this checking behavior by annotating the conformance to `Sendable` with `@unchecked`. This indicates that the type can safely be passed across concurrency domains, but requires the author of the type to ensure that this is safe. + +A `struct` or `enum` can only be made to conform to `Sendable` within the same source file in which the type was defined. This ensures that the stored properties in a struct and associated values in an enum are visible so that their types can be checked for `Sendable` conformance. For example: + +```swift +// MySneakyNSPerson.swift +struct MySneakyNSPerson { + private var name: NSMutableString + public var age: Int +} + +// in another source file or module... +// error: cannot declare conformance to Sendable outside of +// the source file defined MySneakyNSPerson +extension MySneakyNSPerson: Sendable { } +``` + +Without this restriction, another source file or module, which cannot see the private stored property name, would conclude that `MySneakyNSPerson` is properly a `Sendable`. One can declare conformance to `Sendable` as `@unchecked` to disable this check as well: + +```swift +// in another source file or module... +// okay: unchecked conformances in a different source file are permitted +extension MySneakyNSPerson: @unchecked Sendable { } +``` + +#### Implicit struct/enum conformance to `Sendable` + +Many structs and enums satisfy the requirements of `Sendable`, and having to explicitly write out "`: Sendable`" for every such type can feel like boilerplate. +For non-public structs and enums that are also not `@usableFromInline`, and for frozen public structs and enums, the `Sendable` conformance is implicitly provided when conformance checking (described in the previous section) succeeds: + +```swift +struct MyPerson2 { // Implicitly conforms to Sendable! + var name: String, age: Int +} + +class NotConcurrent { } // Does not conform to Sendable + +struct MyPerson3 { // Does not conform to Sendable because nc is of non-Sendable type + var nc: NotConcurrent +} +``` + +Public non-frozen structs and enums do not get an implicit conformance, because doing so would present a problem for API resilience: the implicit conformance to `Sendable` would become part of the contract with clients of the API, even if it was not intended to be. Moreover, this contract could easily be broken by extending the struct or enum with storage that does not conform to `Sendable`. + +> **Rationale**: Existing precedent from `Hashable`, `Equatable`, and `Codable` is to require explicit conformance, even when the details are synthesized. We break from that precedent for `Sendable` because (1) `Sendable` is likely to be even more common, (2) there is no impact on code size (or the binary at all) for `Sendable`, unlike with the other protocols, and (3) `Sendable` does not introduce any additional API beyond allowing the use of the type across concurrency domains. + +Note that implicit conformance to `Sendable` is only available for non-generic types and for generic types whose instance data is guaranteed to be of `Sendable` type. For example: + +```swift +struct X { // implicitly conforms to Sendable + var value: T +} + +struct Y { // does not implicitly conform to Sendable because T does not conform to Sendable + var value: T +} +``` + +Swift will not implicitly introduce a conditional conformance. It is possible that this could be introduced in a future proposal. + +#### `Sendable` conformance checking for classes + +Any class may be declared to conform to `Sendable` with an `@unchecked` conformance, allowing them to be passed between actors without semantic checks. This is appropriate for classes that use access control and internal synchronization to provide memory safety — these mechanisms cannot generally be checked by the compiler. + +In addition, a class may conform to `Sendable` and be checked for memory safety by the compiler in a specific limited case: when the class is a final class containing only immutable stored properties of types that conform to Sendable: + +```swift +final class MyClass : Sendable { + let state: String +} +``` + +Such classes may not inherit from classes other than NSObject (for Objective-C interoperability). `Sendable` classes have the same restriction as structs and enums that requires the `Sendable` conformance to occur in the same source file. + +This behavior makes it possible to safely create and pass around immutable bags of shared state between actors. There are several ways to generalize this in the future, but there are non-obvious cases to nail down. As such, this proposal intentionally keeps safety checking for classes limited to ensure we make progress on other aspects of the concurrency design. + +#### Actor types + +Actor types provide their own internal synchronization, so they implicitly conform to `Sendable`. The [actors proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md) provides more detail. + +#### Key path literals + +Key paths themselves conform to the `Sendable` protocol. However, to ensure that it is safe to share key paths, key path literals can only capture values of types that conform to the `Sendable` protocol. This affects uses of subscripts in key paths: + +```swift +class SomeClass: Hashable { + var value: Int +} + +class SomeContainer { + var dict: [SomeClass : String] +} + +let sc = SomeClass(...) + +// error: capture of 'sc' in key path requires 'SomeClass' to conform +// to 'Sendable' +let keyPath = \SomeContainer.dict[sc] +``` + +### New `@Sendable` attribute for functions + +While the `Sendable` protocol directly addresses value types and allows classes to opt-in to participation with the concurrency system, function types are also important reference types that cannot currently conform to protocols. Functions in Swift occur in several forms, including global func declarations, nested functions, accessors (getters, setters, subscripts, etc), and closures. It is useful and important to allow functions to be passed across concurrency domains where possible to allow higher order functional programming techniques in the Swift Concurrency model, for example to allow definition of `parallelMap` and other obvious concurrency constructs. + +We propose defining a new attribute on function types named `@Sendable`. A `@Sendable` function type is safe to transfer across concurrency domains (and thus, it implicitly conforms to the `Sendable` protocol). To ensure memory safety, the compiler checks several things about values (e.g. closures and functions) that have `@Sendable` function type: + +1. A function can be marked `@Sendable`. Any captures must also conform to `Sendable`. + +2. Closures that have `@Sendable` function type can only use by-value captures. Captures of immutable values introduced by `let` are implicitly by-value; any other capture must be specified via a capture list: + + ```swift + let prefix: String = ... + var suffix: String = ... + strings.parallelMap { [suffix] in prefix + $0 + suffix } + ``` + + The types of all captured values must conform to `Sendable`. + +3. Accessors are not currently allowed to participate with the `@Sendable` system as of this proposal. It would be straight-forward to allow getters to do so in a future proposal if there was demand for this. + +The `@Sendable` attribute to function types is orthogonal to the existing `@escaping` attribute, but it works the same way. `@Sendable` functions are always subtypes of non-`@Sendable` functions, and implicitly convert when needed. Similarly, closure expressions infer the `@Sendable` bit from context just like `@escaping` closures do. + +We can revisit the example from the motivation section — it may be declared like this: + +```swift +actor MyContactList { + func filteredElements(_ fn: @Sendable (ContactElement) -> Bool) async -> [ContactElement] { … } +} +``` + +Which could then be used like so: + +```swift +// Closures with no captures are ok! +list = await contactList.filteredElements { $0.firstName != "Max" } + +// Capturing a 'searchName' string is ok, because String conforms +// to Sendable. searchName is captured by value implicitly. +list = await contactList.filteredElements { $0.firstName == searchName } + +// @Sendable is part of the type, so passing a compatible +// function declaration works as well. +list = await contactList.filteredElements(dynamicPredicate) + +// Error: cannot capture NSMutableString in a @Sendable closure! +list = await contactList.filteredElements { + $0.firstName == nsMutableName +} + +// Error: someLocalInt cannot be captured by reference in a +// @Sendable closure! +var someLocalInt = 1 +list = await contactList.filteredElements { + someLocalInt += 1 + return $0.firstName == searchName +} +``` + +The combination of `@Sendable` closures and `Sendable` types allows type safe concurrency that is library extensible, while still being easy to use and understand. Both of these concepts are key foundations that actors and structured concurrency builds on top of. + +#### Inference of `@Sendable` for Closure Expressions + +The inference rule for `@Sendable` attribute for closure expressions is similar to closure `@escaping` inference. A closure expression is inferred to be `@Sendable` if either: + +* it is used in a context that expects a `@Sendable` function type (e.g. `parallelMap` or `Task.runDetached`) or +* `@Sendable` is in the closure's `in` specification. + +The difference from `@escaping` is that a context-less closure defaults to be non-`@Sendable`, but defaults to being `@escaping`: + +```swift +// defaults to @escaping but not @Sendable +let fn = { (x: Int, y: Int) -> Int in x+y } +``` + +Nested functions are also an important consideration, because they can also capture values just like a closure expression. The `@Sendable` attribute is used on nested function declarations to opt-into concurrency checking: + +```swift +func globalFunction(arr: [Int]) { + var state = 42 + + // Error, 'state' is captured immutably because closure is @Sendable. + arr.parallelForEach { state += $0 } + + // Ok, function captures 'state' by reference. + func mutateLocalState1(value: Int) { + state += value + } + + // Error: non-@Sendable function isn't convertible to @Sendable function type. + arr.parallelForEach(mutateLocalState1) + + @Sendable + func mutateLocalState2(value: Int) { + // Error: 'state' is captured as a let because of @Sendable + state += value + } + + // Ok, mutateLocalState2 is @Sendable. + arr.parallelForEach(mutateLocalState2) +} +``` + +This composes cleanly for both structured concurrency and actors. + +### Thrown errors + +A function or closure that `throws` can effectively return a value of any type that conforms to the `Error` protocol. If the function is called from a different concurrency domain, the thrown value can be passed across it. + +```swift +class MutableStorage { + var counter: Int +} +struct ProblematicError: Error { + var storage: MutableStorage +} + +actor MyActor { + var storage: MutableStorage + func doSomethingRisky() throws -> String { + throw ProblematicError(storage: storage) + } +} +``` + +A call to `myActor.doSomethingRisky()` from another concurrency domain would throw the problematic error, capturing part of the mutable state of `myActor`, then provide it to another concurrency domain, breaking actor isolation. Because there is no information in the signature of `doSomethingRisky()` about the types of errors thrown, and an error that propagates out from `doSomethingRisky()` could come from _any_ code that the function invokes, there is no place at which we could check that only `Sendable`-conforming errors are thrown. + +To close this safety hole, we alter the definition of the `Error` protocol to require that _all_ error types conform to `Sendable`: + +```swift +protocol Error: Sendable { … } +``` + +Now, the `ProblematicError` type will be rejected with an error because it conforms to `Sendable` but contains a stored property of non-`Sendable` type `MutableStorage`. + +Generally speaking, one cannot add a new inherited protocol to an existing protocol without breaking both source and binary compatibility. However, marker protocols have no impact on the ABI and no requirements, so binary compatibility is maintained. + +Source compatibility requires more care, however. `ProblematicError` is well-formed in today’s Swift, but will be rejected with the introduction of `Sendable`. To ease the transition, errors about types that get their `Sendable` conformances through `Error` will be downgraded to warnings in Swift < 6. + +### Adoption of `Sendable` by Standard Library Types + +It is important for standard library types to be passed across concurrency domains. The vast majority of standard library types provide value semantics, and therefore should conform to `Sendable`, e.g.: + +```swift +extension Int: Sendable {} +extension String: Sendable {} +``` + +Generic value-semantic types are safe to be passed across concurrency domains so long as any element types are safe to be passed across concurrency domains. This dependency can be modeled by conditional conformances: + +```swift +extension Optional: Sendable where Wrapped: Sendable {} +extension Array: Sendable where Element: Sendable {} +extension Dictionary: Sendable + where Key: Sendable, Value: Sendable {} +``` + +Except for the cases listed below, all struct, enum, and class types in the standard library conform to the `Sendable` protocol. Generic types conditionally conform to the `Sendable` protocol when all of their generic arguments conform to `Sendable`. The exceptions to these rules follow: + +* `ManagedBuffer`: this class is meant to provide mutable reference semantics for a buffer. It must not conform to `Sendable` (even unsafely). +* `Unsafe(Mutable)(Buffer)Pointer`: these generic types _unconditionally_ conform to the `Sendable` protocol. This means that an unsafe pointer to a non-concurrent value can potentially be used to share such values between concurrency domains. Unsafe pointer types provide fundamentally unsafe access to memory, and the programmer must be trusted to use them correctly; enforcing a strict safety rule for one narrow dimension of their otherwise completely unsafe use seems inconsistent with that design. +* Lazy algorithm adapter types: the types returned by lazy algorithms (e.g., as the result of `array.lazy.map` { … }) never conform to `Sendable`. Many of these algorithms (like the lazy `map`) take non-`@Sendable` closure values, and therefore cannot safely conform to `Sendable`. + +The standard library protocols `Error` and `CodingKey` inherit from the `Sendable` protocol: + +* `Error` inherits from `Sendable` to ensure that thrown errors can safely be passed across concurrency domains, as discussed in the previous section. +* `CodingKey` inherits from `Sendable` so that types like `EncodingError` and `DecodingError`, which store `CodingKey` instances, can correctly conform to `Sendable`. + +### Support for Imported C / Objective-C APIs + +Interoperability with C and Objective-C is an important part of Swift. C code will always be implicitly unsafe for concurrency, because Swift cannot enforce correct behavior of C APIs. However, we still define some basic interactions with the concurrency model by providing implicit `Sendable` conformances for many C types: + +* C enum types always conform to the `Sendable` protocol. +* C struct types conform to the `Sendable` protocol if all of their stored properties conform to `Sendable`. +* C function pointers conform to the `Sendable` protocol. This is safe because they cannot capture values. + +## Future Work / Follow-on Projects + +In addition to the base proposal, there are several follow-on things that could be explored as follow-on proposals. + +### Adaptor Types for Legacy Codebases + +**NOTE**: This section is NOT considered part of the proposal — it is included just to illustrate aspects of the design. + +The proposal above provides good support for composition and Swift types that are updated to support concurrency. Further, Swift’s support for retroactive conformance of protocols makes it possible for users to work with codebases that haven’t been updated yet. + +However, there is an additional important aspect of compatibility with existing frameworks that is important to confront: frameworks are sometimes designed around dense graphs of mutable objects with ad hoc structures. While it would be nice to “rewrite the world” eventually, practical Swift programmers will need support to “get things done” in the meantime. By analogy, when Swift first came out, most Objective-C frameworks were not audited for nullability. We introduced “`ImplicitlyUnwrappedOptional`” to handle the transition period, which gracefully faded from use over the years. + +To illustrate how we can do this with Swift concurrency, consider a pattern that is common in Objective-C frameworks: passing an object graph across threads by “transferring” the reference across threads — this is useful but not memory safe! Programmers will want to be able to express these things as part of their actor APIs within their apps. + +This can be achieved by the introduction of a generic helper struct: + +```swift +@propertyWrapper +struct UnsafeTransfer : @unchecked Sendable { + var wrappedValue: Wrapped + init(wrappedValue: Wrapped) { + self.wrappedValue = wrappedValue + } +} +``` + +For example, `NSMutableDictionary` isn’t safe to pass across concurrency domains, so it isn’t safe to conform to `Sendable`. The struct above allows you (as an app programmer) to write an actor API in your application like this: + +```swift +actor MyAppActor { + // The caller *promises* that it won't use the transferred object. + public func doStuff(dict: UnsafeTransfer) async +} +``` + +While this isn’t particularly pretty, it is effective at getting things done on the caller side when you need to work with unaudited and unsafe code. This can also be sugared into a parameter attribute using the recently proposed [extension to property wrappers for arguments](https://forums.swift.org/t/pitch-2-extend-property-wrappers-to-function-and-closure-parameters/40959), allowing a prettier declaration and caller-side syntax: + +```swift +actor MyAppActor { + // The caller *promises* that it won't use the transferred object. + public func doStuff(@UnsafeTransfer dict: NSMutableDictionary) async +} +``` + +### Objective-C Framework Support + +**NOTE**: This section is NOT considered part of the proposal — it is included just to illustrate aspects of the design. + +Objective-C has established patterns that would make sense to pull into this framework en-masse, e.g. the [`NSCopying` protocol](https://developer.apple.com/documentation/foundation/nscopying) is one important and widely adopted protocol that should be onboarded into this framework. + +General consensus is that it is important to make copies explicit in the model, so we can implement an `NSCopied` helper like so: + +```swift +@propertyWrapper +struct NSCopied: @unchecked Sendable { + let wrappedValue: Wrapped + + init(wrappedValue: Wrapped) { + self.wrappedValue = wrappedValue.copy() as! Wrapped + } +} +``` + +This would allow individual arguments and results of actor methods to opt-into a copy like this: + +```swift +actor MyAppActor { + // The string is implicitly copied each time you invoke this. + public func lookup(@NSCopied name: NSString) -> Int async +} +``` + +One random note: the Objective-C static type system is not very helpful to us with immutability here: statically typed `NSString`’s may actually be dynamically `NSMutableString`’s due to their subclass relationships. Because of this, it isn’t safe to assume that values of `NSString` type are dynamically immutable — they should be implemented to invoke the `copy()` method. + +### Interaction of Actor self and `@Sendable` closures + +Actors are a proposal that is conceptually layered on top of this one, but it is important to be aware of the actor design to make sure that this proposal addresses its needs. As described above, actor method sends across concurrency boundaries naturally require that arguments and results conform to `Sendable`, and thus implicitly require that closures passed across such boundaries are `@Sendable`. + +One additional detail that needs to be addressed is “when is something a cross actor call?”. For example, we would like these calls to be synchronous and not require an await: + +```swift +extension SomeActor { + public func oneSyncFunction(x: Int) {... } + public func otherSyncFunction() { + // No await needed: stays in concurrency domain of self actor. + self.oneSyncFunction(x: 42) + oneSyncFunction(x: 7) // Implicit self is fine. + } +} +``` + +However, we also need to consider the case when ‘self’ is captured into a closure within an actor method. For example: + +```swift +extension SomeActor { + public func thing(arr: [Int]) { + // This should obviously be allowed! + arr.forEach { self.oneSyncFunction(x: $0) } + + // Error: await required because it hops concurrency domains. + arr.parallelMap { self.oneSyncFunction(x: $0) } + + // Is this ok? + someHigherOrderFunction { + self.oneSyncFunction(x: 7) // ok or not? + } + } +} +``` + +We need the compiler to know whether there is a possible concurrency domain hop or not — if so, an await is required. Fortunately, this works out through straight-forward composition of the basic type system rules above: It is perfectly safe to use actor `self` in a non-`@Sendable` closure in an actor method, but using it in a `@Sendable` closure is treated as being from a different concurrency domain, and thus requires an `await`. + +### Marker protocols as custom attributes + +The marker protocol `Sendable` and the function attribute `@Sendable` are intentionally given the same name. There is a potential future direction here where `@Sendable` could move from a special attribute recognized by the compiler (as in this proposal), to having marker protocols like `Sendable` be custom attributes like [property wrappers](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md) and [result builders](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md). Such a change would have very little effect on existing code that uses `@Sendable` so long as users don't declare their own `Sendable` type that shadows the one from the standard library. However, it would make `@Sendable` less special and allow other marker protocols to be used similarly. + +## Source Compatibility + +This is almost completely source compatible with existing code bases. The introduction of the `Sendable` marker protocol and `@Sendable` functions are additive features that have no impact when not used and therefore do not affect existing code. + +There are a few new restrictions that could cause source breakage in exotic cases: + +* The change to keypath literals subscripts will break exotic keypaths that are indexed with non-standard types. +* `Error` and `CodingKey` inherit from `Sendable` and thus require that custom errors and keys conform to `Sendable`. + +Because of these changes, the new restrictions will only be enforced in Swift 6 mode, but will be warnings for Swift 5 and earlier. + +## Effect on API resilience + +This proposal has no effect on API resilience! + +## Alternatives Considered + +There are several alternatives that make sense to discuss w.r.t. this proposal. Here we capture some of the bigger ones. + +### Exotic Type System Features + +The [Swift Concurrency Roadmap](https://forums.swift.org/t/swift-concurrency-roadmap/41611) mentions that a future iteration of the feature set could introduce new type system features like “`mutableIfUnique`” classes, and it is easy to imagine that move semantics and unique ownership could get introduced into Swift someday. + +While it is difficult to understand the detailed interaction without knowing the full specification of future proposals, we believe that the checking machinery that enforces `Sendable` checking is simple and composable. It should work with any types that are safe to pass across concurrency boundaries. + +### Support an explicit copy hook + +The [first revision of this proposal](https://docs.google.com/document/d/1OMHZKWq2dego5mXQtWt1fm-yMca2qeOdCl8YlBG1uwg/edit#) allowed types to define custom behavior when they are sent across concurrency domains, through the implementation of an `unsafeSend` protocol requirement. This increased the complexity of the proposal, admitted undesired functionality (explicitly implemented copy behavior), made the recursive aggregate case more expensive, and would result in larger code size. + +## Conclusion + +This proposal defines a very simple approach for defining types that are safe to transfer across concurrency domains. It requires minimal compiler/language support that is consistent with existing Swift features, is extensible by users, works with legacy code bases, and provides a simple model that we can feel good about even 20 years from now. + +Because the feature is mostly a library feature that builds on existing language support, it is easy to define wrapper types that extend it for domain specific concerns (along the lines of the `NSCopied` example above), and retroactive conformance makes it easy for users to work with older libraries that haven’t been updated to know about the Swift Concurrency model yet. + +## Revision history + +* Changes from the second review: + * Renamed `@sendable` to `@Sendable`, per review feedback and Core Team decision. + * Add a future direction on marker protocols as custom attributes. + * Removed "Swift Concurrency 1.0" and "2.0" discussion in Alternatives Considered. +* Changes from the first review + * Renamed `ConcurrentValue` to `Sendable` and `@concurrent` to `@sendable`. + * Replaced `UnsafeConcurrentValue` with `@unchecked Sendable` conformances. + * Add implicit conformance to `Sendable` for non-public, non-frozen `struct` and `enum` types. diff --git a/proposals/0303-swiftpm-extensible-build-tools.md b/proposals/0303-swiftpm-extensible-build-tools.md new file mode 100644 index 0000000000..7761d1b5d5 --- /dev/null +++ b/proposals/0303-swiftpm-extensible-build-tools.md @@ -0,0 +1,1130 @@ +# Package Manager Extensible Build Tools + +* Proposal: [SE-0303](0303-swiftpm-extensible-build-tools.md) +* Authors: [Anders Bertelrud](https://github.com/abertelrud), [Konrad 'ktoso' Malawski](https://github.com/ktoso), [Tom Doron](https://github.com/tomerd) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (5.6)** +* Amendment status: **Accepted** +* Revision Pitch and Discussion: [Pitch: Amend SE-0303 Plugin API to Use `@main` for Plugin Entry Point](https://forums.swift.org/t/pitch-amend-se-0303-plugin-api-to-use-main-for-plugin-entry-point/51250) +* Original Pitch and Discussion: [Pitch: SwiftPM Extensible Build Tools](https://forums.swift.org/t/pitch-swiftpm-extensible-build-tools/44715) +* Original Reviews: + * [First review](https://forums.swift.org/t/se-0303-package-manager-extensible-build-tools/) + * [Second review](https://forums.swift.org/t/se-0303-2nd-review-package-manager-extensible-build-tools/) +* Amendment (Amend SE-0303 Plugin API to Use `@main` for Plugin Entry Point) + * [Pitch and Discussion](https://forums.swift.org/t/pitch-amend-se-0303-plugin-api-to-use-main-for-plugin-entry-point/51250) + * [Review](https://forums.swift.org/t/amendment-se-0303-package-manager-extensible-build-tools/) + * [Implementation](https://github.com/apple/swift-package-manager/pull/3712) +* Previous Revisions: + * [First revision](https://github.com/swiftlang/swift-evolution/blob/878e496eb799fa407ad704d89fb401952fe8fd02/proposals/0303-swiftpm-extensible-build-tools.md) + * [Second revision](https://github.com/swiftlang/swift-evolution/blob/38731efc140a53553aff923a6616a1dee28c973a/proposals/0303-swiftpm-extensible-build-tools.md) + * [Third revision](https://github.com/swiftlang/swift-evolution/blob/7c3de3eaed8e160feca1d39a35d2f8ba7b2add0d/proposals/0303-swiftpm-extensible-build-tools.md) + + +## Introduction + +This is a proposal for extensible build tools support in Swift Package Manager. The initial set of functionality is intentionally basic, and focuses on a general way of extending the build command graph through plugins. + +The approach is to: + +- provide a scalable way for packages to define plugins that can implement various capabilities +- define a narrowly scoped initial build tool capability that allows plugins to create new build commands + +The set of available plugin capabilities can then be extended in future SwiftPM versions. The goal of this proposal is to provide short-term support for common tasks such as source code generation, with a design that can scale to more complex tasks in the future. + +This proposal depends on improvements to the existing `binaryTarget` type in SwiftPM — those details are the subject of the separate proposal [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md). + +## Motivation + +SwiftPM doesn’t currently provide any means of performing custom actions during a build. This includes source generation as well as custom processing for special types of resources. + +This is quite restrictive, and affects even packages with relatively simple customization needs. Examples include invoking source generators such as [SwiftProtobuf](https://github.com/apple/swift-protobuf) or [SwiftGen](https://github.com/SwiftGen/SwiftGen), or running a custom command to modify a built artifact after it is produced. + +Providing even basic support for extensibility is expected to allow more codebases to be built using the Swift Package Manager, and to automate many steps that package authors currently have to do by hand (such as generate sources manually and commit them to their package repositories). + +## Proposed Solution + +This proposal introduces *package plugins* that implement various *capabilities*, and it defines a single initial *build tool* capability. + +Package plugins are Swift scripts that use API provided by a new `PackagePlugin` library to implement custom actions for SwiftPM to perform. Plugins are defined using package targets of a new type called `plugin`. + +Package plugins are in some ways similar to package manifests: both are Swift scripts that are evaluated in sandboxed environments and that use specialized APIs for a limited and specific purpose. In the case of a package manifest, that purpose is to define those characteristics of a package that cannot be inferred from the contents of the file system; in the case of a package plugin, the purpose is to extend SwiftPM with new functionality. + +Note that a plugin itself does *not* perform the actual work of a custom build tool or other action — that is done by a command invocation or other configured action that the plugin constructs and returns to SwiftPM. The plugin can be thought of as a procedural way of configuring commands, and it is typically quite small. + +Many different kinds of plugin capabilities are envisioned for future proposals. This proposal only defines an initial capability that allows plugins to create commands that should run at build time. + +Different kinds of plugins will be invoked in different ways, as appropriate for their capability. In the case of the build tool capability, plugins are applied on-demand by having targets opt into using those plugins. + +To support that, this proposal adds a new `plugins` parameter to the declarations of `target`, `executableTarget`, and `testTarget` types that allow these kinds of targets to use one or more build tool plugins. The `binaryTarget` and `systemLibrary` target types do not support plugins in this initial proposal, since they are pseudo-targets and are not actually built. + +A build tool plugin is invoked after package resolution and validation, and is given access to an input context that provides information about the target to which the plugin is applied. The plugin also has read-only access to the source directory of the target, and is also allowed to write to specially designated areas of the build output directory. + +This initial proposal does not directly provide a way for the client target to pass configuration parameters to a plugin. However, because plugins have read-only access to the package directory, they can read custom configuration files as needed. While this means that configuration of the plugin resides outside of the package manifest, it does allow each plugin to use a configuration format suitable for its own needs. This pattern is commonly used in practice already, in the form of JSON or YAML configuration files for source generators, etc. Future proposals are expected to let package plugins define options that can be controlled from the client package's manifest. + +A `plugin` target should declare dependencies on the targets that provide the executables needed during the build. The binary target type has been extended to let it vend pre-built executables for build tools that are not built using SwiftPM. This is the subject of the separate evolution proposal [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md). + +A plugin script itself cannot initially use custom libraries built by other SwiftPM targets. Initially, the only APIs available are `PackagePlugin` and the Swift standard libraries. This is somewhat limiting, but note that the plugin itself is only expected to contain a minimal amount of logic to construct a command that will then invoke a tool during the actual build. The tool that is invoked during the build — the one for which the plugin generates a command line — is either a regular `executableTarget` (which can depend on as many library targets as it wants to) or a `binaryTarget` that provides prebuilt binary artifacts. + +It is a future goal to allow package plugin scripts themselves to depend on custom libraries, but that will require larger changes to how SwiftPM — and any IDEs that use libSwiftPM — create their build plans and run their builds. In particular, it will require those build systems to be able to build any libraries that are needed by the plugin script before invoking it, prior to the actual build of the Swift package. SwiftPM's native build system does not currently support such tiered builds, and neither do the build systems of some of the IDEs that use libSwiftPM. + +In order to let other packages use a `plugin` target, it must made visible to other packages through a `plugin` product type (just as for other kinds of targets). If and when a future version of SwiftPM unifies the concepts of products and targets — which would be desirable for other reasons — then this distinction between plugin targets and plugin products will become unnecessary. + +A package plugin target can be used by other targets in the same package without declaring a corresponding product in the manifest. This can be useful for highly specialized build tool plugins that are defined and used in the same package. + +As with the `PackageDescription` API for package manifests, the `PackagePlugin` API availability annotations will be tied to the Swift Tools Version of the package that contains the `plugin` target. This will allow the API to evolve over time without affecting older plugins. + +## Detailed Design + +To allow plugins to be declared, the following API will be added to `PackageDescription`: + +```swift +extension Target { + /// Defines a new package plugin target with a given name, declaring it as + /// providing a capability of extending SwiftPM in a particular way. + /// + /// The capability determines the way in which the plugin extends SwiftPM, + /// which determines the context that is available to the plugin and the + /// kinds of commands it can create. The plugin capability also determines + /// how the plugin is activated. + /// + /// In the initial version of this proposal, only a single plugin capability + /// is defined: build tool. The intent is to define additional capabilities + /// in the future. + /// + /// Another possible capability that could be added in the future could be + /// a way to augment the testing support in SwiftPM. This could take the + /// form of allowing additional commands to run after the build and test + /// have completed, with a well-defined way to access build results and + /// test results. Another possible capability could be specific support + /// for code linters that could emit structured diagnostics with fix-its, + /// or for code formatters that can modify the source code as a separate + /// action outside the build. + /// + /// The package plugin itself is implemented using a Swift script that is + /// invoked for each target that uses it. The script is invoked after the + /// package graph has been resolved, but before the build system creates its + /// dependency graph. It is also invoked after changes to the target or the + /// build parameters. + /// + /// Note that the role of the package plugin is only to define the commands + /// that will run before the build and during the build. It does not itself + /// run those commands. The commands are defined in an IDE-neutral way, and + /// are run as appropriate by the build system that builds the package. The + /// plugin itself is only a procedural way of generating commands and their + /// input and output dependencies. + /// + /// The package plugin may specify the executable targets or binary targets + /// that provide the build tools that will be used by the generated commands + /// during the build. In the initial implementation, prebuild commands can + /// only depend on binary targets. Regular build commands can depend on exe- + /// cutables as well as binary targets. This is due to limitations in how + /// SwiftPM's build system constructs its build plan. It is a goal to remove + /// this restriction in a future release. + /// + /// The `path`, `exclude`, and `sources` parameters are the same as for any + /// other target, and allow flexibility in where the package author can put + /// the plugin scripts inside the package directory. The default subdirectory + /// for plugin targets is in a subdirectory called "Plugins", but this can + /// be customized using the `path` parameter. + public static func plugin( + name: String, + capability: PluginCapability, + dependencies: [Dependency] = [], + path: String? = nil, + exclude: [String] = [], + sources: [String]? = nil + ) -> Target +} + +extension Product { + /// Defines a product that vends a package plugin target for use by clients + /// of the package. It is not necessary to define a product for a plugin that + /// is only used within the same package as it is defined. All the targets + /// listed must be plugin targets in the same package as the product. They + /// will be applied to any client targets of the product in the same order + /// as they are listed. + public static func plugin( + name: String, + targets: [String] + ) -> Product +} + +final class PluginCapability { + /// Plugins that define a `buildTool` capability define commands to run at vari- + /// ous points during the build. + public static func buildTool( + /// Currently the plugin is invoked for every target that is specified as + /// using it. Future SwiftPM versions could refine this so that plugins + /// could, for example, provide input filename filters that further control + /// when they are invoked. + ) -> PluginCapability + + // The idea is to add additional capabilities in the future, each with its own + // semantics. A plugin can implement one or more of the capabilities, and it + // will be invoked within a context relevant for that capability. +} +``` + +To allow plugins to be applied to targets, the following API will be added to `PackageDescription`: + +```swift +extension Target { + .target( + . . . + plugins: [PluginUsage] = [] + ), + .executableTarget( + . . . + plugins: [PluginUsage] = [] + ), + .testTarget( + . . . + plugins: [PluginUsage] = [] + ) +} + +final class PluginUsage { + // Specifies the use of a package plugin with a given target or product name. + // In the case of a plugin target in the same package, no package parameter is + // provided; in the case of a plugin product in a different package, the name + // of the package that provides it needs to be specified. This is analogous to + // product dependencies and target dependencies. + public static func plugin( + _ name: String, + package: String? = nil + ) -> PluginUsage +} +``` + +The plugins that a target uses are applied to it in the order in which they are listed — this allows one plugin to act on output files produced by another plugin. + +#### Plugin API + +The API of the new `PackagePlugin` library lets the plugin construct build commands based on information about the target to which the plugin is applied. The context includes the target and module names, the set of source files in the target (including those generated by previously applied plugins), information about the target's dependency closure, and other inputs from the package. + +The initial proposed `PackagePlugin` API is the following: + +```swift +/// +/// PackagePlugin API +/// +/// Like package manifests, package plugins are Swift scripts that use API +/// from a special library provided by SwiftPM. In the case of plugins, this +/// library is `PackagePlugin`. Plugins run in a sandbox, and have read-only +/// access to the package directory. +/// +/// The input to a package plugin is provided by SwiftPM when it is invoked, +/// and can be accessed through the `context` parameter. The plugin defines +/// commands to run during the build by constructing and returning them from +/// the main plugin function. +/// +/// The `Plugin` protocol defines the functionality common to all types of +/// plugins, and there is a specific protocol for each of the defined plugin +/// capabilities (only the `BuildToolPlugin` is presently defined, but the +/// intent to is allow additional capabilities in the future). +/// +/// Each plugin defines a type that conforms to the protocol corresponding +/// to the capability it provides, and annotates that type with `@main`. +/// It then implements the corresponding methods, which will be called to +/// perform the functionality of the plugin. + +/// Defines functionality common to all plugins. +protocol Plugin { + /// Instantiates the plugin. This happens once per invocation of the + /// plugin; there is no facility for keeping in-memory state from one + /// invocation to the next. Most plugins do not need to implement the + /// initializer. + /// + /// If a future version of SwiftPM allows the usage of a plugin to + /// also provide configuration parameters for that plugin, then a new + /// initializer that accepts that configuration could be added here. + init() +} + +/// Defines functionality for all plugins having a `buildTool` capability. +protocol BuildToolPlugin: Plugin { + /// Invoked by SwiftPM to create build commands for a particular target. + /// The context parameter contains information about the package and its + /// dependencies, as well as other environmental inputs. + /// + /// This function should create and return build commands or prebuild + /// commands, configured based on the information in the context. + func createBuildCommands( + context: TargetBuildContext + ) async throws -> [Command] +} + +/// Provides information about the target being built, as well as contextual +/// information such as the paths of the directories to which commands should +/// be configured to write their outputs. This information should be used as +/// part of generating the commands to be run during the build. +protocol TargetBuildContext { + /// The name of the target being built, as specified in the manifest. + var targetName: String { get } + + /// The module name of the target. This is currently derived from the name, + /// but could be customizable in the package manifest in a future SwiftPM + /// version. + var moduleName: String { get } + + /// The path of the target source directory. + var targetDirectory: Path { get } + + /// That path of the package that contains the target. + var packageDirectory: Path { get } + + /// Information about the input files specified in the target being built, + /// including the sources, resources, and other files. This sequence also + /// includes any source files generated by other plugins that are listed + /// earlier than this plugin in the `plugins` parameter of the target + /// being built. + var inputFiles: FileList { get } + + /// Information about all targets in the dependency closure of the target + /// to which the plugin is being applied. This list is in topologically + /// sorted order, with immediate dependencies appearing earlier and more + /// distant dependencies later in the list. This is mainly intended for + /// generating lists of search path arguments, etc. + var dependencies: [DependencyTargetInfo] { get } + + /// Provides information about a target that appears in the dependency + /// closure of the target to which the plugin is being applied. + protocol DependencyTargetInfo { + + /// The name of the target. + var targetName: String { get } + + /// The module name of the target. This is currently derived from the + /// name, but could be customizable in the package manifest in a future + /// SwiftPM version. + var moduleName: String { get } + + /// Path of the target source directory. + var targetDirectory: Path { get } + + /// Path of the public headers directory, if any (Clang targets only). + var publicHeadersDirectory: Path? { get } + } + + /// The path of a writable directory into which the plugin or the build + /// commands it constructs can write anything it wants. This could include + /// any generated source files that should be processed further, and it + /// could include any caches used by the build tool or the plugin itself. + /// The plugin is in complete control of what is written under this di- + /// rectory, and the contents are preserved between builds. + /// + /// A plugin would usually create a separate subdirectory of this directory + /// for each command it creates, and the command would be configured to + /// write its outputs to that directory. The plugin may also create other + /// directories for cache files and other file system content that either + /// it or the command will need. + var pluginWorkDirectory: Path { get } + + /// The path of the directory into which built products associated with + /// the target are written. + var builtProductsDirectory: Path { get } + + /// Looks up and returns the path of a named command line executable tool. + /// The executable must be provided by an executable target or a binary + /// target on which the package plugin target depends. This function throws + /// an error if the tool cannot be found. The lookup is case sensitive. + func tool(named name: String) throws -> ToolInfo + + /// Information about a particular tool that is available to a plugin. + protocol ToolInfo { + /// Name of the tool, suitable for display purposes. + var name: String { get } + + /// Path of the built or provided tool in the file system. + var path: Path { get } + } +} + +/// A command to run during the build, including executable, command lines, +/// environment variables, initial working directory, etc. All paths should +/// be based on the ones passed to the plugin in the target build context. +enum Command { + + /// Creates a command to run during the build. The executable should be a + /// tool returned by `TargetBuildContext.tool(named:)`, and any paths in + /// the arguments list as well as in the input and output lists should be + /// based on the paths provided in the target build context structure. + /// + /// The build command will run whenever its outputs are missing or if its + /// inputs have changed since the command was last run. In other words, + /// it is incorporated into the build command graph. + /// + /// This is the preferred kind of command to create when the outputs that + /// will be generated are known ahead of time. + static func buildCommand( + /// An arbitrary string to show in build logs and other status areas. + displayName: String, + /// The executable to be invoked; should be a tool looked up using + /// `tool(named:)`, which may reference either a binary-provided tool + /// or a source-built tool. + executable: Path, + /// Arguments to be passed to the tool. Any paths should be based on + /// the paths provided in the target build context. + arguments: [String], + /// Optional initial working directory of the command. + workingDirectory: Path? = nil, + /// Any custom settings for the environment of the subprocess. + environment: [String: String] = [:], + /// Input files to the build command. Any changes to the input files + /// cause the command to be rerun. + inputFiles: [Path] = [], + /// Output files that should be processed further, according to the + /// rules defined by the build system. + outputFiles: [Path] = [] + ) -> Command + + /// Creates a command to run before the build. The executable should be a + /// tool returned by `TargetBuildContext.tool(named:)`, and any paths in + /// the arguments list and the output files directory should be based on + /// the paths provided in the target build context structure. + /// + /// The build command will run before the build starts, and is allowed to + /// create an arbitrary set of output files based on the contents of the + /// inputs. + /// + /// Because prebuild commands are run on every build, they are can have + /// significant performance impact and should only be used when there is + /// no way to know the names of the outputs before the command is run. + /// + /// The `outputFilesDirectory` parameter is the path of a directory into + /// which the command will write its output files. Any files that are in + /// that directory after the prebuild command finishes will be interpreted + /// according to same build rules as for sources. + static func prebuildCommand( + /// An arbitrary string to show in build logs and other status areas. + displayName: String, + /// The executable to be invoked; should be a tool looked up using + /// `tool(named:)`, which may reference either a binary-provided tool + /// or a source-built tool. + executable: Path, + /// Arguments to be passed to the tool. Any paths should be based on + /// the paths provided in the target build context. + arguments: [String], + /// Optional initial working directory of the command. + workingDirectory: Path? = nil, + /// Any custom settings for the environment of the subprocess. + environment: [String: String] = [:], + /// A directory into which the command can write output files that + /// should be processed further. + outputFilesDirectory: Path + ) -> Command +} + +/// Emits errors, warnings, and remarks to be shown as a result of running +/// the plugin. If any errors are emitted, the plugin is considered to have +/// have failed, which will be reported to users during the build. +struct Diagnostics { + /// Emits an error that is shown in SwiftPM or an IDE using it. If one + /// or more errors are emitted by a plugin, it is considered to have + /// failed to run. + static func error(_ message: String, file: Path? = #file, line: Int? = #line) + + /// Emits a warning that is shown in SwiftPM or an IDE using it. + static func warning(_ message: String, file: Path? = #file, line: Int? = #line) + + /// Emits a remark that may be shown in SwiftPM or an IDE using it. + static func remark(_ message: String, file: Path? = #file, line: Int? = #line) + + /// Emits a diagnostic with the specified severity and descriptive message. + static func emit(_ severity: Severity, _ message: String, file: Path? = #file, line: Int? = #line) + + /// The seriousness with which the diagnostic is treated. An error causes + /// SwiftPM to consider the plugin to have failed to run. + enum Severity { + case error, warning, remark + } +} + +/// Provides information about a list of files. The order is not defined +/// but is guaranteed to be stable. This allows the implementation to be +/// more efficient than a static file list. +protocol FileList: Sequence { + func makeIterator() -> FileListIterator +} +struct FileListIterator: IteratorProtocol { + mutating func next() -> FileInfo? +} + +/// Provides information about a single file in a FileList. +protocol FileInfo { + /// The absolute path in the local file system. + var path: Path { get } + + /// The role of the file in SwiftPM. + var type: FileType { get } +} + +/// Provides information about the type of a file. Any future cases will +/// use availability annotations to make sure existing plugins still work +/// until they increase their required tools version. +enum FileType { + /// A source file. + case source + /// A resource file (either processed or copied). + case resource + /// A file not covered by any other rule. + case unknown +} + +/// A simple representation of a path in the file system. This is aligned +/// with SwiftSystem.FilePath to minimize any changes if that is adopted +/// in the future. +protocol Path: ExpressibleByStringLiteral, CustomStringConvertible { + /// A string representation of the path. + public var string: String { get } + + /// The last path component (including any extension). + public var lastComponent: String { get } + /// The last path component (without any extension). + public var stem: String { get } + /// The filename extension, if any (without any leading dot). + public var `extension`: String? { get } + + /// The path except for the last path component. + public func removingLastComponent() -> Path + /// The result of appending one or more path components. + public func appending(_ other: [String]) -> Path + /// The result of appending one or more path components. + public func appending(_ other: String, ...) -> Path + + } +``` + +#### How SwiftPM Applies Plugins to Targets + +During package graph resolution, all package dependencies and binary artifacts are fetched as usual. As part of resolving the package graph, SwiftPM then checks the syntax of any plugin scripts and reports any errors as diagnostics. + +After package resolution but before building, SwiftPM applies build tool plugins to any target that has a `plugins` parameter in its declaration in the package manifest. + +An error is reported if any of the plugins specified by a target cannot be found in the dependency closure of the target, or if they don't support the `buildTool` capability. The plugins are applied to the target in the order in which they are listed in `plugins`. + +Applying a plugin to a target is done by executing the Swift script that implements it, passing inputs that describe the target to which the plugin is being applied, and interpreting its outputs as instructions for SwiftPM. The plugin is run — either by interpreting it or by compiling it to an executable and running it — in a sandbox that prevents network access and that restricts file system access. + +SwiftPM passes input to the plugin in serialized form, which is decoded and made available through the `context` parameter to the plugin function. + +Output from the plugin, in the form of created commands and diagnostics, is passed back to SwiftPM in a serialized form. + +Each build tool plugin is invoked once for each target to which it is applied, with that target’s context as its input. Plugins that are defined but never used are not invoked at all. The command definitions emitted by a package plugin are used to set up build commands to run before or during a build. + +If the plugin script throws an error or emits one or more diagnostics with an error severity, the invocation of the plugin is considered to have failed, and an error is reported to the user. Any console output emitted by the plugin script is shown as debug output. + +It is important to emphasize the distinction between a package plugin and the build commands it defines: + +- The *plugin* is invoked after the structure of the target that uses it is known, but before any build commands are run. It may also be invoked again if the target structure or other input conditions change. A build tool plugin is not invoked again when the *contents* of source files change. + +- The *commands* that are returned by a plugin are invoked either before or during the build, in accordance with their defined input and output dependencies. + +When SwiftPM applies a plugin to a particular target, it passes the path of a directory that is unique to that combination of target and plugin. The plugin and the commands it creates have write access to this work directory, and should use it for any intermediates or caches needed by the plugin or by the commands it creates. It should also write any derived sources inside this directory (or in subdirectories of it). + +The plugin should specify any generated source files that should be fed back into the build system by providing their paths when creating the command. Any other files written to the output directory are considered a private implementation detail of the plugin. The contents are preserved between builds but are removed along with any other intermediates when the build directory is removed. + +When SwiftPM invokes a plugin, it also passes information about any build tools on which the plugin has declared dependencies in the manifest. These build tools may be either executable targets in the package graph or may be binary targets containing prebuilt executables. + +The `binaryTarget` type in SwiftPM has been improved to allow binaries to contain command line tools and required support files (such as the system `.proto` files in the case of `protoc`, etc). Binary targets will need to support different executable binaries based on platform and architecture. This is the topic of the separate evolution proposal [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md), which extends binary targets to support executables in addition to XCFrameworks. + +#### Plugin Capabilities + +Although this proposal defines a general approach to plugins, it specifies only one initial capability: `buildTool`. This capability is used for plugins that generate various kinds of build commands to run during a build. + +Future proposals may extend the APIs to support new kinds of build commands for the `buildTool` capability, but the intent is also that entirely new kinds of capabilities can be defined in the future. New capabilities may focus on specific areas of functionality, such as source code formatters or linters, new types of unit tests, or actions that can be invoked on-demand to perform particular tasks. + +Significantly different types of plugin functionality are likely to use different capabilities. They would also be coupled with a new specialization of the `Plugin` protocol in the `PackagePlugin` API to define the methods that should be implemented for that capability. + +In the package manifest, the capability is expressed as a function invocation rather than an enum in order to make it easier to add arguments with default values in the future. + +#### Types of Build Commands + +The initial `PackagePlugin` API allows a plugin to define two different kinds of build commands: build commands and prebuild commands. They both involve invoking a build tool with a particular command line and environment, but the specific semantics of each make them suitable for different purposes. + +Whenever the command is used for generating other files and when the names of the output files can be determined before the command is run, a regular build command should be used. This lets the command be incorporated into the build system and the outputs be processed according to the build system's rules. + +Prebuild commands should be used when the tool being invoked can produce outputs whose names are determined by the _contents_ (not names) of the input files, or when there are other reasons why the names of the outputs cannot be known before actually running the command. + +##### Build Commands + +Commands of type `.buildCommand` that are returned by the plugin are incorporated into the build system's dependency graph, so that they run as needed during the build, based on their declared inputs and outputs. This requires that the paths of any outputs can be known before the command is run. This is usually done by forming the names of the outputs based on some combination of the output directory and the name of the input file. + +Examples of plugins that can use regular build commands include compiler-like translators such as Protobuf and other tools that take a fixed set of inputs and produce a fixed set of outputs. (note that one nuance with Protobuf in particular is that it is actually up to the source generator invoked by `protoc` to determine the output paths — however, the relevant source generators for Swift and C do produce output files with predictable names). + +Other examples include translators that "compile" data files in JSON or other editable formats to a suitable binary runtime representation. + +Regular build commands with defined outputs are preferable whenever possible, because such commands don't have to run unless their outputs are missing or their inputs have changed since the last time they ran. + +##### Prebuild Commands + +Commands of type `.prebuildCommand` run before the start of every build. When creating prebuild commands, the plugin needs to specify a directory into which the command will write its output files. This is how the prebuild command communicates its outputs to the build system. + +Before invoking a prebuild command, the build system will create the associated output directory if needed (but it will not remove any directory contents that already exist). After invoking the command, SwiftPM will use the contents of that directory as inputs to the construction of other build commands. The prebuild command should add or remove files so that the directory contents match the source files that should be processed by the build system. If the set of files in the directory has changed since the last time the prebuild command was run, the build system planning will be updated so that the changed file set is incorporated into the build. + +Every plugin invocation is passed the path of a directory in `TargetBuildContext.pluginWorkDirectory`. A plugin would usually create a separate subdirectory of this directory for each prebuild command it creates, and the command would be configured to write its output files into that directory. The plugin may also create other subdirectories for cache files and other file system content that either it or the command needs. If these additional files need to be available to the command, the plugin would construct the command line to include their paths, for the command to use. + +Examples of plugins that need to use prebuild commands include SwiftGen and other tools that need to see all the input files, and whose set of output files is determined by the *contents* (not just the paths) of the input files. Such a plugin usually generates just one command and configures it with an output directory into which all generated sources will be written. + +Because they run on every build, prebuild commands should use caching to do as little work as possible (ideally none) when there are no changes to their inputs. This should include preserving the timestamps of generated source files whose contents haven't changed, since most build systems (including SwiftPM's own build system) use timestamps as a shorthand for detecting whether files have changed. + +#### Using Command to Postprocess Built Artifacts + +This proposal focuses on tools that run during the build, as part of constructing artifacts. A future proposal could define new plugin capabilities for more general actions that should take place at the very end of a build, after tests have completed. + +Nonetheless it is sometimes useful to define commands that run at the end of the build in order to post-process the built artifacts. This may include modifying the contents of property lists or other data files in custom ways, or it might include modifying various properties of linked executables, etc. + +Regular build commands can be used for this purpose if they are configured with artifacts inside the built-products directory as their inputs. + +#### Plugin Errors + +Any errors emitted by the build command will be handled by the build system in the same way as for other build commands. SwiftPM will show the output in its console output, and IDEs that use libSwiftPM will show it in the same way as it does for the other build commands. + +Diagnostics from the plugin script itself can be reported using `Diagnostic()` APIs. Also, any error thrown from the main plugin function will be emitted as an error. In either case, if there are any errors, the plugin script will be considered to have failed, and SwiftPM and any IDEs that use libSwiftPM will emit them as it does with other configuration errors. In an IDE this would be similar to errors encountered during the rule matching of source file types, for example. + +The script can use `print()` statements to emit debug output, which will be shown in SwiftPM's console and as detailed output text in the build logs of IDEs that use libSwiftPM. + +## Example 1: SwiftGen + +This example is a package that uses SwiftGen to generate source code for accessing resources. The package plugin can be defined in the same package as the one that provides the source generation tool (SwiftGen, in this case), so that client packages access it just by adding a package dependency on the SwiftGen package. + +The `swiftgen` command may generate output files with any name, and they cannot be known without either running the command or separately parsing the configuration file. In this initial proposal for build plugins, this means that the SwiftGen plugin must construct a prebuild command in order for the source files it generates to be processed during the build. + +#### Client Package + +This is the structure of an example client package of SwiftGen: + +``` +MyPackage + ├ Package.swift + ├ swiftgen.yml + └ Sources + └ MyLibrary + ├ Assets.xcassets + └ SourceFile.swift +``` + + +SwiftGen supports using a config file named `swiftgen.yml` and this example implementation of the plugin assumes a convention that it is located in the package directory. A different plugin implementation might instead use a per-target convention, or might use a combination of the two. + +The package manifest has a dependency on the SwiftGen package, which vends a plugin product that the client package can use for any of its targets by adding a `plugins` parameter to the package manifest: + +```swift +// swift-tools-version: 999.0 +import PackageDescription + +let package = Package( + name: "MyPackage", + dependencies: [ + .package(url: "https://github.com/SwiftGen/SwiftGen", from: "6.4.0") + ] + targets: [ + .executable( + name: "MyLibrary", + plugins: [ + .plugin(name: "SwiftGenPlugin", package: "SwiftGen") + ] + ) + ] +) +``` + +By specifying that the `MyLibrary` target uses the plugin, `SwiftGenPlugin` will be invoked for the target. + +The order in which plugins are listed in `plugin` determines the order in which they will be applied compared with other plugins in the list that provide the same capability. This can be used to control which plugins will see the outputs of which other plugins. In this case, only one plugin is used. + +#### Plugin Package + +Using the facilities in this proposal, the SwiftGen package authors could implement a package plugin that creates a command to run `swiftgen` before the build. + +This is the hypothetical `SwiftGenPlugin` target referenced in the client package: + +``` +SwiftGen + ├ Package.swift + ├ Plugins + │ └ SwiftGenPlugin + │ └ plugin.swift + └ Sources + └ SwiftGen + └ ... +``` + +In this case, `plugin.swift` is the Swift script that implements the package plugin target. The plugin is treated as a Swift executable, and can consist of one or more Swift source files, once of which contains a type attributed with `@main`. + +The package manifest would have a `plugin` target in addition to the existing target that provides the `swiftgen` command line tool itself: + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "SwiftGen", + targets: [ + /// Package plugin that tells SwiftPM how to run `swiftgen` based on + /// the configuration file. Client targets use this plugin by listing + /// it in their `plugins` parameter. + .plugin( + name: "SwiftGenPlugin", + capability: .buildTool(), + dependencies: ["SwiftGen"] + ), + + /// Binary target that provides the built SwiftGen executables. + .binaryTarget( + name: "SwiftGen", + url: "https://url/to/the/built/swiftgen-executables.zip", + checksum: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + ), + ] +) +``` + +The package plugin script might look like this: + +```swift +import PackagePlugin + +@main struct SwiftGenPlugin: BuildToolPlugin { + /// This plugin's implementation returns a single `prebuild` command to run `swiftgen`. + func createBuildCommands(context: TargetBuildContext) throws -> [Command] { + // This example configures `swiftgen` to take inputs from a `swiftgen.yml` file + let swiftGenConfigFile = context.packageDirectory.appending("swiftgen.yml") + + // This example configures the command to write to a "GeneratedSources" directory. + let genSourcesDir = context.pluginWorkDirectory.appending("GeneratedSources") + + // Return a command to run `swiftgen` as a prebuild command. It will be run before + // every build and generates source files into an output directory provided by the + // build context. This example sets some environment variables that `swiftgen.yml` + // bases its output paths on. + return [.prebuildCommand( + displayName: "Running SwiftGen", + executable: try context.tool(named: "swiftgen").path, + arguments: [ + "config", "run", + "--config", "\(swiftGenConfigFile)" + ], + environment: [ + "PROJECT_DIR": "\(context.packageDirectory)", + "TARGET_NAME": "\(context.targetName)", + "DERIVED_SOURCES_DIR": "\(genSourcesDir)", + ], + outputFilesDirectory: genSourcesDir)] + } +} +``` + +An alternate use of `swiftgen` could instead invoke it once for each input file, passing it output files whose names are derived from the names of the input files. This might, however, make per-file configuration somewhat more difficult. + +There is a trade-off here between implementing a prebuild command or a regular build command. Future improvements to SwiftPM's build system — and to those of any IDEs that support Swift packages — could let it support commands whose outputs aren't known until it is run. That would allow the use of regular build commands to generate output files whose names aren't know until the command is known. + +Possibly, the `swiftgen` tool itself could also be modified to provide a simplified way to invoke it, to take advantage of SwiftPM's new ability to dynamically provide the names of the input files in the target. + +## Example 2: SwiftProtobuf + +This example is a package that uses SwiftProtobuf to generate source files from `.proto` files. In addition to the package plugin product, the package provides the runtime library that the generated Swift code uses. + +Since `protoc` isn’t built using SwiftPM, it also has a binary target with a reference to a `zip` archive containing the executable. + +#### Client Package + +This is the structure of an example package that uses Protobuf: + +``` +MyPackage + ├ Package.swift + └ Sources + └ MyExe + ├ messages.proto + └ main.swift +``` + +The `messages.proto` source file needs to be processed using the `protoc` compiler to generate Swift source files that are then compiled. In addition, the `protoc` compiler needs to be passed the path of a source generator plug-in built by the SwiftProtobuf package. + +The package manifest has a dependency on the `SwiftProtobuf` package, and references the hypothetical new `SwiftProtobuf` plugin defined in it: + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "MyPackage", + dependencies: [ + .package(url: "https://github.com/apple/swift-protobuf", from: "1.15.0") + ] + targets: [ + .executable( + name: "MyExe", + dependencies: [ + .product(name: "SwiftProtobufRuntimeLib", package: "swift-protobuf") + ], + plugins: [ + .plugin(name: "SwiftProtobuf", package: "swift-protobuf") + ] + ) + ] +) +``` + +As with the previous example, listing the plugin in the `plugins` parameter applies that plugin to the `MyExe` target. As with product dependencies, the package name needs to be provided when the plugin is defined in a different package. + +In this initial version of the proposal, the client target must also specify dependencies on any runtime libraries that will be needed, as this example shows with the hypothetical `SwiftProtobufRuntimeLib` library. A future improvement could extend the `PackagePlugin` API to let the plugin define additional dependencies that targets using the plugin would automatically get. + +This version of the initial proposal does not define a way to pass options to the plugin through the manifest. Because the plugin has read-only access to the package directory, it can define its own conventions for a configuration file in the package or target directory. A future improvement to the proposal should allow a way for the plugin to provide custom types that the client package manifest could use to specify options to the plugin. This is described in more detail under _Future Directions_, below. + +#### Plugin Package + +The structure of the hypothetical `SwiftProtobuf` target that provides the plugin is: + +``` +SwiftProtobuf + ├ Package.swift + ├ Plugins + │ └ SwiftProtobuf + │ └ plugin.swift + └ Sources + ├ SwiftProtobufRuntimeLib + │ └ ... + └ protoc-gen-swift + └ ... +``` + +The package manifest is: + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "SwiftProtobuf", + targets: [ + /// Package plugin that tells SwiftPM how to deal with `.proto` files. + .plugin( + name: "SwiftProtobuf", + capability: .buildTool(), + dependencies: ["protoc", "protoc-gen-swift"] + ), + + /// Binary target that provides the prebuilt `protoc` executable. + .binaryTarget( + name: "protoc", + url: "https://url/to/the/built/protoc-executables.zip", + checksum: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + ), + + /// Swift target that builds the plug-in executable that will be passed to + /// the `protoc` compiler. + .executableTarget( + name: "protoc-gen-swift" + ), + + /// Runtime library that clients will need to link against by specifying + /// it as a dependency. + .libraryTarget( + name: "SwiftProtobufRuntimeLib" + ), + ] +) +``` + +The `protoc-gen-swift` target is a regular executable target that implements a source generation plug-in to the Protobuf compiler. This is particular to the Protobuf compiler, which in addition to the `protoc` executable uses separate source code generator tools specific to the emitted languages. The implementation of the hypothetical `SwiftProtobuf` plugin generates the commands to invoke `protoc` passing it the `protoc-gen-swift` plug-in. + +The package plugin script might look like: + +```swift +import PackagePlugin +import Foundation + +@main struct MyPlugin: BuildToolPlugin { + /// This plugin's implementation returns multiple build commands, each of which + /// calls `protoc`. + func createBuildCommands(context: TargetBuildContext) throws -> [Command] { + // In this case we generate an invocation of `protoc` for each input file, + // passing it the path of the `protoc-gen-swift` generator tool. + let protocTool = try context.tool(named: "protoc") + let protocGenSwiftTool = try context.tool(named: "protoc-gen-swift") + + // Construct the search paths for the .proto files, which can include any + // of the targets in the dependency closure. Here we assume that the public + // ones are in a `protos` directory, but this can be made arbitrarily complex. + var protoSearchPaths = context.dependencies.map { target in + target.targetDirectory.appending("protos") + } + + // This example configures the commands to write to a "GeneratedSources" + // directory. + let genSourcesDir = context.pluginWorkDirectory.appending("GeneratedSources") + + // This example uses a different directory for other files generated by + // the plugin. + let otherFilesDir = context.pluginWorkDirectory.appending("OtherFiles") + + // Add the search path to the system proto files. This sample implementation + // assumes that they are located relative to the `protoc` compiler provided + // by the binary target, but real implementation could be more sophisticated. + protoSearchPaths.append(protocTool.path.removingLastComponent().appending("system-protos")) + + // Create a module mappings file. This is something that the Swift source + // generator `protoc` plug-in we are using requires. The details are not + // important for this proposal, except that it needs to be able to be con- + // structured from the information in the context given to the plugin, and + // to be written out to the intermediates directory. + let moduleMappingsFile = otherFilesDir.appending("module-mappings") + let outputString = ". . . module mappings file . . ." + let outputData = outputString.data(using: .utf8) + FileManager.default.createFile(atPath: moduleMappingsFile.string, contents: outputData) + + // Iterate over the .proto input files, creating a command for each. + let inputFiles = context.inputFiles.filter { $0.path.extension == "proto" } + return inputFiles.map { inputFile in + // The name of the output file is based on the name of the input file, + // in a way that's determined by the protoc source generator plug-in + // we're using. + let outputName = inputFile.path.stem + ".swift" + let outputPath = genSourcesDir.appending(outputName) + + // Specifying the input and output paths lets the build system know + // when to invoke the command. + let inputFiles = [inputFile.path] + let outputFiles = [outputPath] + + // Construct the command arguments. + var commandArgs = [ + "--plugin=protoc-gen-swift=\(protocGenSwiftTool.path)", + "--swift_out=\(genSourcesDir)", + "--swift_opt=ProtoPathModuleMappings=\(moduleMappingsFile)" + ] + commandArgs.append(contentsOf: protoSearchPaths.flatMap { ["-I", "\($0)"] }) + commandArgs.append("\(inputFile.path)") + + // Append a command containing the information we generated. + return .buildCommand( + displayName: "Generating \(outputName) from \(inputFile.path.stem)", + executable: protocTool.path, + arguments: commandArgs, + inputFiles: inputFiles, + outputFiles: outputFiles) + } + } +} +``` + +In this case, the script iterates over the input files that have a `.proto` suffix, creating a command to invoke the Protobuf compiler for each one. + +## Example 3: Source Analyzer + +A third important use case is source generators that analyze Swift files and generate some additional sources based on the Swift definitions found in them. Tool authors may use a package such as [Sourcery](https://github.com/krzysztofzablocki/Sourcery), or parse the sources using [SwiftSyntax](https://github.com/apple/swift-syntax) to generate some boilerplate code, in order to avoid having to maintain it manually. + +One could imagine a source generation tool called `GenSwifty` generating some additional “sugar” for existing definitions in a swift target. It would be configured like this by end users: + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "MyPackage", + dependencies: [ + .package(url: "https://github.com/example/gen-swifty", from: "1.0.0") + ] + targets: [ + .executable( + name: "MyExe", + plugins: [ + .plugin(name: "GenSwifty", package: "gen-swifty") + ] + ) + ] +) +``` + +The package manifest of the `gen-swifty` package would be as follows: + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "gen-swifty", + targets: [ + .plugin( + name: "GenSwifty", + capability: .buildTool(), + dependencies: ["GenSwiftyTool"] + ), + .executable( + name: "GenSwiftyTool", + ... // this is the target that builds the tool + // The tool can depend on swift-syntax or other tools + ), + ] +) +``` + +The implementation of the package plugin would be similar to the previous examples, but with a somewhat differently formed command line. + +## Example 4: Custom Source Generator + +This example uses a custom source generator implemented in the same package as the target that uses it. + +``` +MyPackage + ├ Package.swift + ├ Plugins + │ └ MySourceGenPlugin + │ └ plugin.swift + └ Sources + ├ MyExe + │ │ file.dat + │ └ main.swift + └ MySourceGenTool + └ main.swift +``` + +The manifest is: + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "MyPackage", + targets: [ + .executable( + name: "MyExe", + plugins: [ + .plugin("MySourceGenPlugin") + ] + ), + .plugin( + name: "MySourceGenPlugin", + capability: .buildTool(), + dependencies: ["MySourceGenTool"] + ), + .executableTarget( + name: "MySourceGenTool" + ), + ] +) +``` + +In this case, the `.plugin()` expression in the `plugin` parameter refers to a plugin target defined in the same package, so it does not need a `package` parameter and no product needs to be defined. + +The implementation of the package plugin script would be similar to the previous examples, but with a somewhat differently formed command line to transform the `.dat` file into something else. + +## Impact on Existing Packages + +The new API and functionality will only be available to packages that specify a tools version equal to or later than the SwiftPM version in which this functionality is implemented, so there will be no impact on existing packages. + +## Security + +Package plugins will be run in a sandbox that prevents network access and restricts writing to the file system to specific intermediate directories. This is the same sandbox that manifest parsing uses, except that it allows writing to a limited set of output directories. + +In addition, SwiftPM and IDEs that use libSwiftPM should run each command generated by an plugin in a sandbox. This sandbox should prevent all network access, and should only allow writing to the directories specified as outputs by the plugin. + +There is inherent risk in running build tools provided by other packages. This can possibly be mitigated by requiring the root package to list the approved package plugins, no matter where they are in the graph. This requires further consideration, and would be a subject of a future proposal. + +## Future Directions + +This proposal is intentionally fairly basic and leaves many improvements for the future. Among them are: + +- the ability for a build tool to have a separate dependency graph so that it can use different versions of the same packages as those used by the client of the build tool +- the ability for plugins to have access to the full build graph at a detailed level +- the ability for prebuild actions to depend on tools built by SwiftPM +- the ability for package plugin scripts to use libraries provided by other targets +- the ability for build commands to emit output files that feed back into the rule system to generate new work during the build (this requires support in SwiftPM's native build system as well as in the build systems of some IDEs that use libSwiftPM) +- the ability to provide per-target type-safe options to a use of an plugin (details below) +- the ability to define commands that run after building and testing is complete +- specific support for testing plugin scripts (they can currently be tested using test fixtures that exercise the plugins, but it would be useful to be able to invoke them directly, for example through a `swift package` command that invokes the plugin with specific inputs) + + +### Type-Safe Options + +We are aware that many plugins will want to allow specific per-target configuration which would be best done in the package manifest of the package that uses the plugin. + +We are purposefully leaving options out of this initial proposal, and plan to revisit and add these in a future proposal. + +In theory, options could just be done as a dictionary of string key-value pairs, like this: + +```swift +// NOT proposed +.plugin(name: "Foo", options: ["Visibility": "Public"]) // not type-safe! +``` + +However, we believe that this results in a suboptimal user experience. It is hard to know what the available keys are, and what values are accepted. Is only `"Public"` correct in this example, or would `"public"` work too? + +We would therefore prefer to exclude plugin options from this proposal and explore a type-safe approach to options in a future proposal. + +We would especially like to allow plugins to define some form of `struct MyOptions: PluginOptions` type, where `PluginOptions` is also `Codable`. SwiftPM could then take care of serializing these options and providing them to the plugin. + +This would allow something like this: + +```swift +.plugin(name: "Foo", options: FooOptions(visibility: .public)) // type-safe! +``` + +Such a design is difficult to implement well, because it would require the plugin to add a type that is accessible to the package package manifest. It would also mean that the package manifest couldn't be parsed at all until the plugin module that provides the types has been compiled. + +Designing such type safe options is out of scope for this initial proposal, as it involves many complexities with respect to how the types would be made available from the plugin definition to the package manifest that needs them. + +It is an area we are interested in exploring and improving in the near future, so rather than lock ourselves into supporting untyped dictionaries of strings, we suggest to introduce target specific, type safe plugin options in a future Swift evolution proposal. + +### Separate Dependency Graphs for Build Tools + +This proposal uses the existing facilities that SwiftPM provides for declaring dependencies on packages that provide plugins and tools. This keeps the scope of the proposal bounded enough to allow it to be implemented in the near term, but it does mean that it does not allow there to be any conflicts between the set of package dependencies that the build tool needs compared with those that the client package needs. + +For example, a situation in which a build tool needs [Yams](https://github.com/jpsim/Yams) v3.x while a package client needs v4.x would not be supported by this proposal, even though it would pose no real problem at runtime. It would be very desirable to support this, but that would require significant work on SwiftPM to allow it to keep multiple independent package graphs. It should be noted that if a package vends both a build tool and a runtime library that clients using the build tool must link against, then even this apparently simple case would get more complicated. In this case the runtime library would have to use Yams v4.x in order to be usable by the client package, even if the tool itself used Yams v3.x. + +It should also be noted that this is no different from a package that defines an executable and a library and would like to use a particular version of Yams without requiring the client of the library to use a compatible version. This is not supported in SwiftPM today either, and it is in fact another manifestation of the same problem. + +Solving this mixture of build-time and run-time package dependencies is possible but not without significant work in SwiftPM (and IDEs that use libSwiftPM). We think that this proposal could be extended to support declaring such dependencies in the future, and that even with this restriction, this proposal provides useful functionality for packages. + +### A More Sophisticated `Path` Type + +The current `Path` type provided in the `PackagePlugin` API is intentionally kept minimal, and should be sufficient for the needs of most plugins. A future version of this proposal would extend this type, possibly aligning it with `SwiftSystem`'s `FilePath` type. The initial API has purposefully been kept the same as `FilePath` to make such a transition easier. + +An alternate direction would be to replace it with a more domain-specific representation that could also keep track of the path root in relation to the directories that matter to packages and build systems, avoiding the need to form absolute paths. + +Since the API that is available to the plugin script is based on the tools version of the package that contains it, existing plugin scripts are expected to be able to run without modification even if the API does change. + +### Contextual Information About the Target Platform + +The current `TargetBuildContext` type in the `PackagePlugin` API provides minimal information about the structure and configuration of the target, but it does not in this initial proposal provide any information about the target platform for which the package is being built. + +This would be needed in order to implement more advanced tools such as code generators or linkers. + +### Specific Support for Code Linters and Formatters + +A future proposal should add specific support for code linters. In particular, there should be a way for build tools to convey fixits and other mechanical editing instructions to SwiftPM or to an IDE that uses libSwiftPM. + +One approach would be to use the existing Clang diagnostics file format, possibly together with a library making it easy to generate such files, and to extend the `PackagePlugin` API to allow the plugin to configure commands to emit this information in a way that SwiftPM and IDEs can apply it. Such a capability could also be useful for build tools such as source translators, if they want to be able to apply fixits to their input files. + +Code formatters (which typically modify the source code directly) should probably be supported using a new plugin capability that allows some specific action that a package author can take to run the formatter on their code, since it seems a bit subtle to allow source code to be modified as a side effect of a regular build action. + +### Postbuild Commands + +This proposal allows regular build commands to be configured to act on artifacts after they are built, but there is a greater need for a more general kind of postbuild command that can run after building and testing are complete for all the targets in the build graph. + +A future proposal should extend plugins to support commands that can be run after both building and testing is complete, possibly using a new plugin capability and additional `PackagePlugin` APIs. Such commands would be able to report on the results of the build and testing, to post notifications, etc. + +Many of the most useful cases need information about the results of the build and the testing. This includes detailed information about what was built, how it was built, and what the results were. This could include structured diagnostics, code coverage information, etc. + +A future proposal should define a structured format through which SwiftPM communicates this kind of information to postbuild commands, and should ideally provide a library with API that allows such a tool to load and query this structure. + +## Alternatives Considered + +A simpler approach would be to allow a package to contain shell scripts that are unconditionally invoked during the build. In order to support even moderately complex uses, however, there would still need to be some way for the script to get information about the target being built and its dependencies, and to know where to write output files. + +This information would need to be passed to the script through some means, such as command line flags or environment variables, or through file system conventions. This seems a more subtle and less clear approach than providing a Swift API to access this information. If shell scripts are needed for some use cases, it would be fairly straightforward for a package author to write a custom plugin to invoke that shell script and pass it inputs from the build context using either command line arguments or environment variables, as it sees fit. + +Even with an approach based on shell scripts, there would still need to be some way of defining the names of the output files ahead of time in order for SwiftPM to hook them into its build graph (needing to know the names of outputs ahead of time is a limitation of the native build system that SwiftPM currently uses, as well as of some of the IDEs that are based on libSwiftPM — these build systems currently have no way to apply build rules to the output of commands that are run in the same build). + +Defining these inputs and outputs using an API seems clearer than some approach based on naming conventions or requiring output files to be listed in the client package’s manifest. + +Another approach would be to delay providing extensible build tools until the build systems of SwiftPM and of IDEs using it can be reworked to support arbitrary discovery of new work. This would, however, delay these improvements for the various kinds of build tools that *can* take advantage of the support proposed here. + +We believe that the approach taken here — defining a common approach for specifying plugins that declare the capabilities they provide, and with a goal of defining more advanced capabilities over time — will provide some benefit now while still allowing more sophisticated behavior in the future. + +## References + +- https://github.com/aciidb0mb3r/swift-evolution/blob/extensible-tool/proposals/NNNN-package-manager-extensible-tools.md +- https://forums.swift.org/t/package-manager-extensible-build-tools/10900 diff --git a/proposals/0304-structured-concurrency.md b/proposals/0304-structured-concurrency.md new file mode 100644 index 0000000000..4051b87b1e --- /dev/null +++ b/proposals/0304-structured-concurrency.md @@ -0,0 +1,1611 @@ +# Structured concurrency + +* Proposal: [SE-0304](0304-structured-concurrency.md) +* Authors: [John McCall](https://github.com/rjmccall), [Joe Groff](https://github.com/jckarter), [Doug Gregor](https://github.com/DougGregor), [Konrad Malawski](https://github.com/ktoso) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.5)** [Acceptance post](https://forums.swift.org/t/accepted-with-modifications-se-0304-structured-concurrency/51850) +* Implementation: Available in [recent `main` snapshots](https://swift.org/download/#snapshots) behind the flag `-Xfrontend -enable-experimental-concurrency` +* Review: ([first pitch](https://forums.swift.org/t/concurrency-structured-concurrency/41622)) ([second pitch](https://forums.swift.org/t/pitch-2-structured-concurrency/43452)) ([third pitch](https://forums.swift.org/t/pitch-3-structured-concurrency/44496)) ([first review](https://forums.swift.org/t/se-0304-structured-concurrency/45314)) ([second review](https://forums.swift.org/t/se-0304-2nd-review-structured-concurrency/47217)) ([third review](https://forums.swift.org/t/se-0304-3rd-review-structured-concurrency/48847)) ([fourth review](https://forums.swift.org/t/se-0304-4th-review-structured-concurrency/50281)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0304-structured-concurrency/51850)) ([revision](https://forums.swift.org/t/accepted-with-modifications-se-0304-structured-concurrency/51850/9)) +* Previous revisions: ([first review](https://github.com/swiftlang/swift-evolution/blob/9b5e0cbd552b4c8b570aedcb94c0cb72b9f591b0/proposals/0304-structured-concurrency.md)) ([second review](https://github.com/swiftlang/swift-evolution/blob/3defec78bc3f7cbbc9a14f69511397d27899244d/proposals/0304-structured-concurrency.md)) ([third review](https://github.com/swiftlang/swift-evolution/blob/a0faeabd6718ba99865a940a085927a69494fe94/proposals/0304-structured-concurrency.md)) ([fourth review](https://github.com/swiftlang/swift-evolution/blob/2e9eda9f67ae7aabd89ea17baadd32affa416843/proposals/0304-structured-concurrency.md)) ([as accepted](https://github.com/swiftlang/swift-evolution/blob/c2f363236520589ab94ef91a1419e06f39346133/proposals/0304-structured-concurrency.md)) + +## Table of contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Structured concurrency](#structured-concurrency-1) + * [Tasks](#tasks) + * [Child tasks](#child-tasks) + * [Jobs](#jobs) + * [Executors](#executors) + * [Task priorities](#task-priorities) + * [Priority Escalation](#priority-escalation) +* [Proposed solution](#proposed-solution) + * [Task groups and child tasks](#task-groups-and-child-tasks) + * [Asynchronous programs](#asynchronous-programs) + * [Cancellation](#cancellation) + * [Unstructured tasks](#unstructured-tasks) + * [Context inheritance](#context-inheritance) + * [Detached tasks](#detached-tasks) +* [Detailed design](#detailed-design) + * [Task API](#task-api) + * [The Task type](#the-task-type) + * [UnsafeCurrentTask type](#unsafecurrenttask-type) + * [Task priorities](#task-priorities-1) + * [Unstructured tasks](#unstructured-tasks-1) + * [Priority propagation](#priority-propagation) + * [Actor context propagation](#actor-context-propagation) + * [Implicit "self"](#implicit-self) + * [Detached tasks](#detached-tasks-1) + * [Cancellation](#cancellation-1) + * [Cancellation handlers](#cancellation-handlers) + * [Voluntary Suspension](#voluntary-suspension) + * [Task Groups](#task-groups) + * [Creating TaskGroup child tasks](#creating-taskgroup-child-tasks) + * [Querying tasks in the group](#querying-tasks-in-the-group) + * [Task group cancellation](#task-group-cancellation) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) +* [Revision history](#revision-history) + * [Review changes](#review-changes) + * [Pitch changes](#pitch-changes) +* [Alternatives Considered](#alternatives-considered) + * [Prominent futures](#prominent-futures) +* [Future directions](#future-directions) + * [async let to create child tasks within a scope](#async-let-to-create-child-tasks-within-a-scope) + * [@Sendable closure checking for task groups](#sendable-closure-checking-for-task-groups) + * [Suspending await group.addTask](#suspending-await-groupaddtask) + +## Introduction + +[`async`/`await`](0296-async-await.md) is a language mechanism for writing natural, efficient asynchronous code. Asynchronous functions (introduced with `async`) can give up the thread on which they are executing at any given suspension point (marked with `await`), which is necessary for building highly-concurrent systems. + +However, the `async`/`await` proposal does not introduce concurrency *per se*: ignoring the suspension points within an asynchronous function, it will execute in essentially the same manner as a synchronous function. This proposal introduces support for [structured concurrency](https://en.wikipedia.org/wiki/Structured_concurrency) in Swift, enabling concurrent execution of asynchronous code with a model that is ergonomic, predictable, and admits efficient implementation. + +Swift-evolution threads: + +* [Pitch #1](https://forums.swift.org/t/concurrency-structured-concurrency/41622), +* [Pitch #2](https://forums.swift.org/t/pitch-2-structured-concurrency/43452), +* [Pitch #3](https://forums.swift.org/t/pitch-3-structured-concurrency/44496), +* [Review #1](https://forums.swift.org/t/se-0304-structured-concurrency/45314), +* [Review #2](https://forums.swift.org/t/se-0304-2nd-review-structured-concurrency/47217), +* [Review #3](https://forums.swift.org/t/se-0304-3rd-review-structured-concurrency/48847). + +## Motivation + +For a simple example, let's make dinner, asynchronously: + +```swift +func chopVegetables() async throws -> [Vegetable] { ... } +func marinateMeat() async -> Meat { ... } +func preheatOven(temperature: Double) async throws -> Oven { ... } + +// ... + +func makeDinner() async throws -> Meal { + let veggies = try await chopVegetables() + let meat = await marinateMeat() + let oven = try await preheatOven(temperature: 350) + + let dish = Dish(ingredients: [veggies, meat]) + return try await oven.cook(dish, duration: .hours(3)) +} +``` + +Each step in our dinner preparation is an asynchronous operation, so there are numerous suspension points. While waiting for the vegetables to be chopped, `makeDinner` won't block a thread: it will suspend until the vegetables are available, then resume. Presumably, many dinners could be in various stages of preparation, with most suspended until their current step is completed. + +However, even though our dinner preparation is asynchronous, it is still *sequential*. It waits until the vegetables have been chopped before starting to marinate the meat, then waits again until the meat is ready before preheating the oven. Our hungry patrons will be very hungry indeed by the time dinner is finally done. + +To make dinner preparation go faster, we need to perform some of these steps *concurrently*. To do so, we can break down our recipe into different tasks that can happen in parallel. The vegetables can be chopped at the same time that the meat is marinating and the oven is preheating. Sometimes there are dependencies between tasks: as soon as the vegetables and meat are ready, we can combine them in a dish, but we can't put that dish into the oven until the oven is hot. All of these tasks are part of the larger task of making dinner. When all of these tasks are complete, dinner is served. + +This proposal aims to provide the necessary tools to carve work up into smaller tasks that can run concurrently, to allow tasks to wait for each other to complete, and to effectively manage the overall progress of a task. + +## Structured concurrency + +Any concurrency system must offer certain basic tools. There must be some way to create a new thread that will run concurrently with existing threads. There must also be some way to make a thread wait until another thread signals it to continue. These are powerful tools, and you can write very sophisticated systems with them. But they're also very primitive tools: they make very few assumptions, but in return they give you very little support. + +Imagine there's a function which does a large amount of work on the CPU. We want to optimize it by splitting the work across two cores; so now the function creates a new thread, does half the work in each thread, and then has its original thread wait for the new thread to finish. (In a more modern system, the function might add a task to a global thread pool, but the basic concept is the same.) There is a relationship between the work done by these two threads, but the system doesn't know about it. That makes it much harder to solve systemic problems. + +For example, suppose a high-priority operation needs the function to hurry up and finish. The operation might know to escalate the priority of the first thread, but really it ought to escalate both. At best, it won't escalate the second thread until the first thread starts waiting for it. It's relatively easy to solve this problem narrowly, maybe by letting the function register a second thread that should be escalated. But it'll be an ad-hoc solution that might need to be repeated in every function that wants to use concurrency. + +Structured concurrency solves this by asking programmers to organize their use of concurrency into high-level tasks and their child component tasks. These tasks become the primary units of concurrency, rather than lower-level concepts like threads. Structuring concurrency this way allows information to naturally flow up and down the hierarchy of tasks which would otherwise require carefully-written support at every level of abstraction and on every thread transition. This in turn permits many high-level problems to be addressed with relative ease. + +For example: + +- It's common to want to limit the total time spent on a task. Some APIs support this by allowing a timeout to be passed in, but it takes a lot of work to propagate timeouts down correctly through every level of abstraction. This is especially true because end-programmers typically want to write timeouts as relative durations (e.g. 20ms), but the correctly-composing representation for libraries to pass around internally is an absolute deadline (e.g. now + 20ms). Under structured concurrency, a deadline can be installed on a task and naturally propagate through arbitrary levels of API, including to child tasks. + +- Similarly, it's common to want to be able to cancel an active task. Asynchronous interfaces that support this often do so by synchronously returning a token object that provides some sort of `cancel()` method. This significantly complicates the design of an API and so often isn't provided. Moreover, propagating tokens, or composing them to cancel all of the active work, can create significant engineering challenges for a program. Under structured concurrency, cancellation naturally propagates through APIs and down to child tasks, and APIs can install handlers to respond instantaneously to cancellation. + +- Graphical user interfaces often rely on task prioritization to ensure timely refreshes and responses to events. Under structured concurrency, child tasks naturally inherit the priority of their parent tasks. Furthermore, when higher-priority tasks wait for lower-priority tasks to complete, the lower-priority task and all of its child tasks can be escalated in priority, and this will reliably persist even if the task is briefly suspended. + +- Many systems want to maintain their own contextual information for an operation without having to pass it through every level of abstraction, such as a server that records information for the connection currently being serviced. Structured concurrency allows this to naturally propagate down through async operations as a sort of "task-local storage" which can be picked up by child tasks. + +- Systems that rely on queues are often susceptible to queue-flooding, where the queue accepts more work than it can actually handle. This is typically solved by introducing "back-pressure": a queue stops accepting new work, and the systems that are trying to enqueue work there respond by themselves stopping accepting new work. Actor systems often subvert this because it is difficult at the scheduler level to refuse to add work to an actor's queue, since doing so can permanently destabilize the system by leaking resources or otherwise preventing operations from completing. Structured concurrency offers a limited, cooperative solution by allowing systems to communicate up the task hierarchy that they are coming under distress, potentially allowing parent tasks to stop or slow the creation of presumably-similar new work. + +This proposal doesn't propose solutions for all of these, but early investigations show promise. + +### Tasks + +A task is the basic unit of concurrency in the system. Every asynchronous function is executing in a task. In other words, a _task_ is to _asynchronous functions_, what a _thread_ is to _synchronous functions_. That is: + +- All asynchronous functions run as part of some task. +- A task runs one function at a time; a single task has no concurrency. +- When a function makes an `async` call, the called function is still running as part of the same task (and the caller waits for it to return). +- Similarly, when a function returns from an `async` call, the caller resumes running on the same task. + +Synchronous functions do not necessarily run as part of a task. + +Swift assumes the existence of an underlying thread system. Tasks are scheduled by the system to run on these system threads. Tasks do not require special scheduling support from the underlying thread system, although a good scheduler could take advantage of some of the interesting properties of Swift's task scheduling. + +A task can be in one of three states: + +* A **suspended** task has more work to do but is not currently running. + - It may be **schedulable**, meaning that it’s ready to run and is just waiting for the system to instruct a thread to begin executing it, + - or it may be **waiting** on some external event before it can become schedulable. +* A **running** task is currently running on a thread. + - It will run until it either returns from its initial function (and becomes completed) or reaches a suspension point (and becomes suspended). At a suspension point, it may become immediately schedulable if, say, its execution just needs to change actors. +* A **completed** task has no more work to do and will never enter any other state. + - Code can wait for a task to become completed in various ways, most notably by `await`-ing on it. + +The way we talk about execution for tasks and asynchronous functions is more complicated than it is for synchronous functions. An asynchronous function is running as part of a task. If the task is running, it and its current function are also running on a thread. + +Note that, when an asynchronous function calls another asynchronous function, we say that the calling function is suspended, but that doesn’t mean the entire task is suspended. From the perspective of the function, it is suspended, waiting for the call to return. From the perspective of the task, it may have continued running in the callee, or it may have been suspended in order to, say, change to a different execution context. + +Tasks serve three high-level purposes: + +* They carry scheduling information, such as the task's priority. +* They serve as a handle through which the operation can be cancelled, queried, or manipulated. +* They can carry user-provided task-local data. + +At a lower level, the task allows the implementation to optimize the allocation of local memory, such as for asynchronous function contexts. It also allows dynamic tools, crash reporters, and debuggers to discover how a function is being used. + +### Child tasks + +An asynchronous function can create a child task. Child tasks inherit some of the structure of their parent task, including its priority, but can run concurrently with it. However, this concurrency is bounded: a function that creates a child task must wait for it to end before returning. This structure means that functions can locally reason about all the work currently being done for the current task, anticipate the effects of cancelling the current task, and so on. It also makes creating the child task substantially more efficient. + +Of course, a function’s task may itself be a child of another task, and its parent may have other children; a function cannot reason locally about these. But the features of this design that apply to an entire task tree, such as cancellation, only apply “downwards” and don’t automatically propagate upwards in the task hierarchy, and so the child tree still can be statically reasoned about. If child tasks did not have bounded duration and so could arbitrarily outlast their parents, the behavior of tasks under these features would not be easily comprehensible. + +In this proposal, the way to create child tasks is only within a `TaskGroup`, however there will be a follow-up proposal that enables creation of child tasks in any asynchronous context. + +### Jobs + +The execution of a task can be seen as a succession of periods where the task was running, each of which ends at a suspension point or — finally — at the completion of the task. These periods are called jobs. Jobs are the basic units of schedulable work in the system. They are also the primitive through which asynchronous functions interact with the underlying synchronous world. For the most part, programmers should not have to work directly with jobs unless they are implementing a custom executor. + +### Executors + +An executor is a service which accepts the submission of jobs and arranges for some thread to run them. The system assumes that executors are reliable and will never fail to run a job. + +An asynchronous function that is currently running always knows the executor that it's running on. This allows the function to avoid unnecessarily suspending when making a call to the same executor, and it allows the function to resume executing on the same executor it started on. + +An executor is called *exclusive* if the jobs submitted to it will never be run concurrently. (Specifically, the jobs must be totally ordered by the happens-before relationship: given any two jobs that were submitted and run, the end of one must happen-before the beginning of the other.) Executors are not required to run jobs in the order they were submitted; in fact, they should generally honor task priority over submission order. + +Swift provides a default executor implementation, but both [actors](0306-actors.md) and [global actors](0316-global-actors.md) (described in separate proposals) can suppress this and provide their own implementation. + +Generally end-users need not interact with executors directly, but rather use them implicitly by invoking functions which happen to use executors to perform the invoked asynchronous functions. + +### Task priorities + +A task is associated with a specific priority. + +Task priority may inform decisions an executor makes about how and when to schedule tasks submitted to it. An executor may utilize priority information to attempt to run higher priority tasks first, and then continuing to serve lower priority tasks. It may also use priority information to affect the platform thread priority. + +The exact semantics of how priority is treated are left up to each platform and specific executor implementation. + +Child tasks automatically inherit their parent task's priority. Detached tasks do not inherit priority (or any other information) because they semantically do not have a parent task. + +The priority of a task does not necessarily match the priority of its executor. For example, the UI thread on Apple platforms is a high-priority executor; any task submitted to it will be run with high priority for the duration of its time on the thread. This helps to ensure that the UI thread will be available to run higher-priority work if it is submitted later. This does not affect the formal priority of the task. + +### Priority Escalation + +In some situations the priority of a task must be escalated in order to avoid a priority inversion: + +- If a task is running on behalf of an actor, and a higher-priority task is enqueued on the actor, the task may temporarily run at the priority of the higher-priority task. This does not affect child tasks or the reported priority; it is a property of the thread running the task, not the task itself. + +- If a task is created with a task handle, and a higher-priority task waits for that task to complete, the priority of the task will be permanently increased to match the higher-priority task. This does affect child tasks and the reported task priority. + +## Proposed solution + +Our approach follows the principles of *structured concurrency* described above. All asynchronous functions run as part of an asynchronous task. Tasks can make child tasks that will perform work concurrently. This creates a hierarchy of tasks, and information can naturally flow up and down the hierarchy, making it convenient to manage the whole thing holistically. + +### Task groups and child tasks + +A *task group* defines a scope in which one can create new child tasks programmatically. As with all child tasks, the child tasks within the task group scope must complete when the scope exits, and will be implicitly cancelled first if the scope exits with a thrown error. + +To illustrate task groups, let's start by showing how we can introduce some +real concurrency to our `makeDinner` example: + +```swift +func makeDinner() async throws -> Meal { + // Prepare some variables to receive results from our concurrent child tasks + var veggies: [Vegetable]? + var meat: Meat? + var oven: Oven? + + enum CookingStep { + case veggies([Vegetable]) + case meat(Meat) + case oven(Oven) + } + + // Create a task group to scope the lifetime of our three child tasks + try await withThrowingTaskGroup(of: CookingStep.self) { group in + group.addTask { + try await .veggies(chopVegetables()) + } + group.addTask { + await .meat(marinateMeat()) + } + group.addTask { + try await .oven(preheatOven(temperature: 350)) + } + + for try await finishedStep in group { + switch finishedStep { + case .veggies(let v): veggies = v + case .meat(let m): meat = m + case .oven(let o): oven = o + } + } + } + + // If execution resumes normally after `withTaskGroup`, then we can assume + // that all child tasks added to the group completed successfully. That means + // we can confidently force-unwrap the variables containing the child task + // results here. + let dish = Dish(ingredients: [veggies!, meat!]) + return try await oven!.cook(dish, duration: .hours(3)) +} +``` + +Note that it would be illegal to say: + +```swift +var veggies: [Vegetable]? + +try await withThrowingTaskGroup(of: Void.self) { group in + group.addTask { + // error: mutation of captured var 'veggies' in concurrently-executing code + veggies = try await chopVegetables() + } +} +let dish = Dish(ingredients: [veggies!]) +``` + +This may be surprising, because the child tasks are guaranteed to have +completed in one way or another by the end of `withTaskGroup`, so it would +theoretically be safe for them to modify variables captured from their parent +context as long as sibling tasks or the parent task itself do not +simultaneously access those same variables until the task group completes. +However, Swift's `@Sendable` closure checking has to be conservative, unless +we give it special knowledge of task groups' semantics. We leave that to a +later proposal. + +The `withTaskGroup` API gives us access to a task group, and governs the +lifetime of the *child tasks* we subsequently add to the group using its +`addTask()` method. By the time `withTaskGroup` finishes executing, we know that all of +the subtasks have completed. A child task does not persist beyond the scope in +which it was created. By the time the scope exits, the child task must either +have completed, or it will be implicitly awaited. When the scope exits via a +thrown error, the child task will be implicitly cancelled before it is awaited. + +These properties allow us to nicely contain the effects of the concurrency we +introduce inside the task group: although `chopVegetables`, `marinateMeat`, +and `preheatOven` will run concurrently, and may make progress in any order, +we can be sure that they have all finished executing in one way or another +by the time `withTaskGroup` returns or throws an error. In either case, task groups +naturally propagate status from child tasks to the parent; in this example, +the `chopVegetables()` function might throw an error if, say, there is an +incident with the kitchen knife. That thrown error completes the child task for +chopping the vegetables. The error will then be propagated out of the +`makeDinner()` function, as expected. On exiting the body of the `makeDinner()` +function with this error, any child tasks that have not yet completed +(marinating the meat or preheating the oven, maybe both) will be automatically +cancelled. Structured concurrency means we don't have to manually propagate +errors and manage cancellation; if execution continues normally after a call +into `withTaskGroup`, we can assume that all of its child tasks completed +successfully. + +Let's stretch our example even further and focus in on our `chopVegetables()` operation, which produces an array of `Vegetable` values. With enough cooks, we could chop our vegetables even faster if we divided up the chopping for each kind of vegetable. Let's start with a sequential version of `chopVegetables()`: + +```swift +/// Sequentially chop the vegetables. +func chopVegetables() async throws -> [Vegetable] { + let rawVeggies: [Vegetable] = gatherRawVeggies() + var choppedVeggies: [Vegetable] = [] + for v in rawVeggies { + choppedVeggies.append(try await v.chopped()) + } + return choppedVeggies +} +``` + +Unlike the top-level `makeDinner` task, here we have a dynamic amount of +potential concurrency; depending on how many vegetables we can get from +`gatherRawVeggies`, each vegetable could in principle be chopped in parallel +with the rest. We also don't need to necessarily gather the chopped vegetables +in any specific order, and can collect the results as they become ready. + +To create a dynamic number of child tasks and gather their results, we still introduce a new task group via `withTaskGroup`, specifying a `ChildTaskResult.Type` +for the child tasks, and using the group's `next` method to collect those +results as they become ready: + +```swift +/// Concurrently chop the vegetables. +func chopVegetables() async throws -> [Vegetable] { + // Create a task group where each child task produces a Vegetable. + try await withThrowingTaskGroup(of: Vegetable.self) { group in + var rawVeggies: [Vegetable] = gatherRawVeggies() + var choppedVeggies: [Vegetable] = [] + + // Create a new child task for each vegetable that needs to be chopped. + for v in rawVeggies { + group.addTask { + try await v.chopped() + } + } + + // Wait for all of the chopping to complete, collecting the veggies into + // the result array in whatever order they're ready. + while let choppedVeggie = try await group.next() { + choppedVeggies.append(choppedVeggie) + } + + return choppedVeggies + } +} +``` + +As in the first example, if the closure passed to `withTaskGroup` exited without having completed all its child tasks, the task group will still wait until all child tasks have completed before returning. If the closure exits with a thrown error, the outstanding child tasks will first be cancelled before propagating +the error to the parent. + +By contrast with future-based task APIs, there is no way in which a reference to the child task can escape the scope in which the child task is created. This ensures that the structure of structured concurrency is maintained. It both makes it easier to reason about the concurrent tasks that are executing within a given scope, and also unlocks numerous optimization opportunities for the compiler and runtime. + +### Asynchronous programs + +A program can use `@main` with a `main()` function that is `async`: + +```swift +@main +struct Eat { + static func main() async throws { + let meal = try await makeDinner() + print(meal) + } +} +``` + +Semantically, Swift will create a new task that will execute `main()`. Once that task completes, the program terminates. + +Top-level code can also make use of asynchronous calls. For example: + + +```swift +// main.swift or a Swift script +let meal = try await makeDinner() +print(meal) +``` + +The model is the same as for `@main`: Swift creates a task to execute top-level code, and completion of that task terminates the program. + +### Cancellation + +A task can be cancelled asynchronously by any context that has a reference to a task or one of its parent tasks. Cancellation can be triggered explicitly by calling `cancel()` on the task handle. Cancellation can also trigger automatically, for example when a parent task throws an error out of a scope with unawaited child tasks. + +The effect of cancellation within the cancelled task is fully cooperative and synchronous. That is, cancellation has no effect at all unless something checks for cancellation. Conventionally, most functions that check for cancellation report it by throwing `CancellationError()`; accordingly, they must be throwing functions, and calls to them must be decorated with some form of `try`. As a result, cancellation introduces no additional control-flow paths within asynchronous functions; you can always look at a function and see the places where cancellation can occur. As with any other thrown error, `defer` blocks can be used to clean up effectively after cancellation. + +With that said, the general expectation is that asynchronous functions should attempt to respond to cancellation by promptly throwing or returning. In most functions, it should be sufficient to rely on lower-level functions that can wait for a long time (for example, I/O functions or `Task.value`) to check for cancellation and abort early. Functions which perform a large amount of synchronous computation may wish to periodically check for cancellation explicitly. + +Cancellation has two effects which trigger immediately with the cancellation: + +- A flag is set in the task which marks it as having been cancelled; once this flag is set, it is never cleared. Operations running synchronously as part of the task can check this flag and are conventionally expected to throw a `CancellationError`. + +- Any cancellation handlers which have been registered on the task are immediately run. This permits functions which need to respond immediately to do so. + +We can illustrate cancellation with a version of the `chopVegetables()` function we saw previously: + +```swift +func chopVegetables() async throws -> [Vegetable] { + return try await withThrowingTaskGroup(of: Vegetable.self) { group in + var veggies: [Vegetable] = [] + + group.addTask { + try await chop(Carrot()) // (1) throws UnfortunateAccidentWithKnifeError() + } + group.addTask { + try await chop(Onion()) // (2) + } + + for try await veggie in group { // (3) + veggies.append(veggie) + } + + return veggies + } +} +``` + +On line *(1)*, we start a new child task to chop a carrot. Suppose that this call to the `chop` function throws an error. Because this is asynchronous, that error is not immediately observed in `chopVegetables`, and we proceed to start a second child task to chop an onion *(2)*. On line *(3)*, we await the `next` completed task, which could be either of the child tasks we created, but for the sake of discussion we'll say happens to be the `chop(Carrot())` child task from *(1)*. This causes us to throw the error that was thrown from `chop`. Since we do not handle this error, we exit the scope without having yet awaited the onion-chopping task. This causes that task to be automatically cancelled. Because cancellation is cooperative, and because structured concurrency does not allow child tasks to outlast their parent context, control does not actually return until the onion-chopping task actually completes; any value it returns or throws will be discarded. + +As we mentioned before, the effect of cancellation on a task is synchronous and cooperative. Functions which do a lot of synchronous computation may wish to check explicitly for cancellation. They can do so by inspecting the task's cancelled status: + +```swift +func chop(_ vegetable: Vegetable) async throws -> Vegetable { + try Task.checkCancellation() // automatically throws `CancellationError` + // chop chop chop ... + // ... + + guard !Task.isCancelled else { + print("Cancelled mid-way through chopping of \(vegetable)!") + throw CancellationError() + } + // chop some more, chop chop chop ... +} +``` + +Note also that no information is passed to the task about why it was cancelled. A task may be cancelled for many reasons, and additional reasons may accrue after the initial cancellation (for example, if the task fails to immediately exit, it may pass a deadline). The goal of cancellation is to allow tasks to be cancelled in a lightweight way, not to be a secondary method of inter-task communication. + +### Unstructured tasks + +So far all types of tasks we discussed were child-tasks and respected the primary rule of structured concurrency: that a *child task* cannot live longer than the *parent task* (or scope) in which it was created. This is both true for task groups as well as [SE-0317 `async let`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0317-async-let.md). + +Sometimes however, these rigid rules end up being too restrictive. We might need to create new tasks whose lifetime is not bound to the creating task, for example in order to fire-and-forget some operation or to initiate asynchronous work from synchronous code. Unstructured tasks are not able to utilize some of the optimization techniques wrt. allocation and metadata propagation as child-tasks are, however they remain a very important building block especially for more free-form usages and integration with legacy APIs. + +All unstructured tasks are represented by a task handle, which can be used to retrieve the value (or thrown error) produced by the task, cancel the task, or perform queries of the task's status. A new task can be created with the `Task { ... }` initializer. For example: + +```swift +let dinnerHandle = Task { + try await makeDinner() +} +``` + +The initializer creates a new task that begins executing the provided closure. That new task is represented by the constructed task handle (in this case, `Task`) referencing the newly-launched task. Task handles can be used to await the result of the task, e.g., + +```swift +let dinner = try await dinnerHandle.value +``` + +Tasks run to completion even if there are no remaining uses of their task handle, so it is not necessary to retain the task handle or observe its value for the task to complete. However, the task handle can be used to explicitly cancel the operation, e.g., + +```swift +dinnerHandle.cancel() +``` + +#### Context inheritance + +Unstructured tasks created with the `Task` initializer inherit important metadata information from the context in which it is created, including priority, task-local values, and actor isolation. + +If called from the context of an existing task: + +- inherit the priority of the current task the _synchronous_ function is executing on +- inherit all task-local values by copying them to the new unstructured task +- if executed within the scope of a specific actor function: + - inherit the actor's execution context and run the task on its executor, rather than the global concurrent one, + - the closure passed to `Task {}` becomes actor-isolated to that actor, allowing access to the actor-isolated state, including mutable properties and non-sendable values. + +If called from a context that is _not_ running inside a task: + +- consult the runtime and infer the best possible priority to use (e.g. by asking for current thread priority), +- even though there is no `Task` to inherit task-local values from, check the fallback mechanism for any task-locals stored for the current synchronous context (this is discussed in depth in the [SE-0311](0311-task-locals.md) proposal) +- execute on the global concurrent executor and be non-isolated with regard to any actor. + +#### Detached tasks + +A *detached task* is an unstructured task that is independent of the context in which it is created, meaning that it does not inherit priority, task-local values, or the actor context. A new detached task can be created with the `Task.detached` function: + +```swift +let dinnerHandle = Task.detached { + try await makeDinner() +} +``` + +The `Task.detached` operation produces a new task instance (in this case, `Task`), in the same manner as the `Task` initializer. + +## Detailed design + +### Task API + +Much of the proposed implementation of structured concurrency is in the APIs for creating, querying, and managing tasks. + +#### The `Task` type + +The `Task` type describes a task and can be used to query or cancel that task. It's also used as a namespace for operations on the currently-executing task. + +```swift +struct Task: Equatable, Hashable, Sendable { ... } +``` + +An instance of the `Task` type can be used to retrieve the result (or thrown error) of executing the task. The operations are always `async`: + +```swift +extension Task { + /// Retrieve the result produced the task, if is the normal return value, or + /// throws the error that completed the task with a thrown error. + var value: Success { + get async throws + } + + /// Retrieve the result produced by the task as a \c Result instance. + var result: Result { get async } +} + +extension Task where Failure == Never { + /// Retrieve the result produced by a task that is known to never throw. + var value: Success { + get async + } +} +``` + +The `value` property is the primary consumer interface to a task instance: it returns the result produced by the task or (if the task exits via a thrown error) throws the error produced by the task. For example: + +```swift +func eat(mealHandle: Task) async throws { + let meal = try await mealHandle.value + meal.eat() // yum +} +``` + +Task instances also provide the ability to cancel a task programmatically: + +```swift +extension Task { + /// Cancel the task referenced by this handle. + func cancel() + + /// Determine whether the task was cancelled. + var isCancelled: Bool { get } +} +``` + +As noted elsewhere, cancellation is cooperative: the task will note that it has been cancelled and can choose to return earlier (either via a normal return or a thrown error, as appropriate). `isCancelled` can be used to determine whether a particular task was ever cancelled. + +#### `UnsafeCurrentTask` type + +The `UnsafeCurrentTask` offers non-static functions which may be used to interact with the running task itself. The unsafe task object must never be escaped or accessed from another task, and thus the API to obtain it takes the familiar scoped `with...` form of `withUnsafeCurrentTask`: + +```swift +func withUnsafeCurrentTask( + body: (UnsafeCurrentTask?) throws -> T +) rethrows -> T +``` + +The `withUnsafeCurrentTask` passes the current task into the operation or `nil` if the function is called from a context in which a Task is not available. In practice this means that nowhere in the call chain until this invocation, was any asynchronous function involved. If there is an asynchronous function in the call chain until the invocation of `unsafeCurrent`, that task will be returned. + +The `UnsafeCurrentTask` is purposefully named unsafe as it *may* expose APIs which can *only* be invoked safely from within task itself, and would exhibit undefined behavior if used from another task. It is therefore unsafe to store and "use later" an `UnsafeCurrentTask`. Examples of such unsafe API are interacting with task-local values on a task object, which must be equal to the "current" task to be performed safely. This is by design, and offers the runtime optimization opportunities for the normal, and safe, access patterns to task storage. + +Invoking some of its APIs from other tasks/threads will result in undefined behavior. + +Accessing this API performs a thread-local lookup of a specific thread-local variable that is maintained by the Swift Concurrency runtime. + +The `withUnsafeCurrentTask` function may be invoked from synchronous (as well as asynchronous) code: + +```swift +func synchronous() { + withUnsafeCurrentTask { maybeUnsafeCurrentTask in + if let unsafeCurrentTask = maybeUnsafeCurrentTask { + print("Seems I was invoked as part of a Task!") + } else { + print("Not part of a task.") + } + } +} + +func asynchronous() async { + // the following is safe, because withUnsafeCurrentTask is invoked from an 'async' function + withUnsafeCurrentTask { maybeUnsafeCurrentTask in + let task: UnsafeCurrentTask = maybeUnsafeCurrentTask! // always ok + } +} +``` + +The `withUnsafeCurrentTask` function returns an _optional_ `UnsafeCurrentTask`, this is because such synchronous function may be invoked from a task (i.e. from within asynchronous Swift code) or outside of it (e.g. some Task unaware API, like a raw pthread thread calling into Swift code). + +The `UnsafeCurrentTask` is also `Equatable` and `Hashable`, whose identity is based on the internal task object which is the same as the one used by `Task`. + +```swift +struct UnsafeCurrentTask: Equatable, Hashable { + public var isCancelled: Bool { get } + public var priority: TaskPriority { get } + public func cancel() +} +``` + +`UnsafeCurrentTask` has all the same query operations as `Task` (i.e. `isCancelled`, `priority`, ...) and the ability to cancel (`cancel()`), which are equally safe to invoke on the unsafe task as on a normal task, however it may define more APIs in the future that are more fragile and must only ever be invoked while executing on the same task (e.g. access to [Task-Local Values](0311-task-locals.md) which are defined in a separate proposal). + +#### Task priorities + +The priority of a task is used by the executor to help make scheduling decisions. + +The priorities are listed from highest (most important) to lowest (least important). + +In order to avoid forcing other platforms to use Darwin specific terminology priorities use generic terms such as "high" and "low". +However, the Darwin specific names exist as aliases and may be used interchangeably. + +```swift +/// Describes the priority of a task. +struct TaskPriority: Codable, Comparable, RawRepresentable, Sendable { + var rawValue: UInt8 { get set } + init(rawValue: UInt8) +} + +/// General, platform independent priority values. +/// +/// The priorities are ordered from highest to lowest as follows: +/// - `high` +/// - `medium` +/// - `low` +/// - `background` +extension TaskPriority { + static var high: TaskPriority { ... } + static var medium: TaskPriority { ... } + static var low: TaskPriority { ... } + static var background: TaskPriority { ... } +} + +/// Apple platform specific priority aliases. +/// +/// The priorities are ordered from highest to lowest as follows: +/// - `userInitiated` (alias for `high` priority) +/// - `utility` (alias for `low` priority) +/// - `background` +/// +/// The runtime reserves the right to use additional higher or lower priorities than those publicly listed here, +/// e.g. the main thread in an application might run at an user inaccessible `userInteractive` priority, however +/// any task created from it will automatically become `userInitiated`. +extension TaskPriority { + /// The task was initiated by the user and prevents the user from actively using + /// your app. + /// + /// Alias for `TaskPriority.high`. + static var userInitiated: TaskPriority { ... } + + /// Priority for a utility function that the user does not track actively. + /// + /// Alias for `TaskPriority.low` + static var utility: TaskPriority { ... } +} + +extension Task where Success == Never, Failure == Never { + /// Returns the `current` task's priority. + /// + /// When called from a context with no `Task` available, will return the best + /// approximation of the current thread's priority, e.g. userInitiated for + /// the "main thread" or default if no specific priority can be detected. + static var currentPriority: TaskPriority { ... } +} +``` + +The `priority` operation queries the priority of the task. + +The `currentPriority` operation queries the priority of the currently-executing task. Task priorities are set on task creation (e.g., `Task.detached` or `TaskGroup.addTask`) and can be escalated later, e.g., if a higher-priority task waits on the task handle of a lower-priority task. + +#### Unstructured tasks + +Unstructured tasks can be created using the `Task` initializer: + +```swift +extension Task where Failure == Never { + @discardableResult + init( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Success + ) +} + +extension Task where Failure == Error { + @discardableResult + init( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Success + ) +} +``` + +The initializers are marked with `@discardableResult` because the task itself will immediately execute the operation and run to completion when the handle is unused. This is a fairly common use case for fire-and-forget asynchronous operations. + +By default, the new task will be initially scheduled on the default global concurrent executor. Once custom executors are introduced in another proposal, these will be able to take an executor parameter to determine on which executor to schedule the new task instead. + +##### Priority propagation + +The `Task` initializer propagates priority from the point where it is called to the detached task that it creates: + +1. If the synchronous code is running on behalf of a task (i.e., `withUnsafeCurrentTask` provides a non-`nil` task), use the priority of that task; +2. If the synchronous code is running on behalf of the "UI" thread, use `.userInitiated`; otherwise +3. Query the system to determine the priority of the currently-executing thread and use that. + +The implementation will also propagate any other important OS-specific information from the synchronous code into the asynchronous task. + +##### Actor context propagation + +A closure passed to the `Task` initializer will implicitly inherit the actor execution context and isolation of the context in which the closure is formed. For example: + +```swift +func notOnActor(_: @Sendable () async -> Void) { } + +actor A { + func f() { + notOnActor { + await g() // must call g asynchronously, because it's a @Sendable closure + } + Task { + g() // okay to call g synchronously, even though it's @Sendable + } + } + + func g() { } +} +``` + +In a sense, the `Task` initializer counteracts the normal influence of `@Sendable` on a closure within an actor. Specifically, [SE-0306](0306-actors.md#closures) states that `@Sendable` closures are not actor-isolated: + +> Actors prevent this data race by specifying that a `@Sendable` closure is always non-isolated. + +Such semantics, where the closure is both `@Sendable` and actor-isolated, are only possible because the closure is also `async`. Effectively, when the closure is called, it will immediately "hop" over to the actor's context so that it runs within the actor. + +##### Implicit "self" + +Closures passed to the `Task` initializer are not required to explicitly acknowledge capture of `self` with `self.`. + +```swift +func acceptEscaping(_: @escaping () -> Void) { } + +class C { + var counter: Int = 0 + + func f() { + acceptEscaping { + counter = counter + 1 // error: must use "self." because the closure escapes + } + Task { + counter = counter + 1 // okay: implicit "self" is allowed here + } + } +} +``` + +The intent behind requiring `self.` when capturing `self` in an escaping closure is to warn the developer about potential reference cycles. The closure passed to `Task` is executed immediately, and the only reference to `self` is what occurs in the body. Therefore, the explicit `self.` isn't communicating useful information and should not be required. + +> **Note**: The same applies to the closure passed to `Task.detached` and `TaskGroup.addTask`. + +##### Detached tasks + +A new, detached task can be created with the `Task.detached` operation. The resulting task is represented by a `Task`. + +```swift +extension Task where Failure == Never { + /// Create a new, detached task that produces a value of type `Success`. + @discardableResult + static func detached( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Success + ) -> Task +} + +extension Task where Failure == Error { + /// Create a new, detached task that produces a value of type `Success` or throws an error. + @discardableResult + static func detached( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Success + ) -> Task +} +``` + +Detached tasks will typically be created using a trailing closure, e.g., + +```swift +let dinnerHandle: Task = Task.detached { + try await makeDinner() +} + +try await eat(mealHandle: dinnerHandle) +``` + +#### Cancellation + +The `isCancelled` property determines whether the given task has been cancelled: + +```swift +extension Task { + /// Returns `true` if the task is cancelled, and should stop executing. + var isCancelled: Bool { get } +} +``` + +It is possible to query for cancellation from within a synchronous task, e.g. while iterating over a loop and wanting to check if we should abort its execution by using the static `Task.isCancelled` property: + +```swift +extension Task where Success == Never, Failure == Never { + /// Returns `true` if the task is cancelled, and should stop executing. + /// + /// Always returns `false` when called from code not currently running inside of a `Task`. + static var isCancelled: Bool { get } +} +``` + +This works the same as its instance counterpart, except that if invoked from a context that has no `Task` available, e.g. if invoked from outside of Swift's concurrency model (e.g. directly from a pthread) a default value is returned. +The static `isCancelled` property is implemented as: + +```swift +extension Task where Success == Never, Failure == Never { + static var isCancelled: Bool { + withUnsafeCurrentTask { task in + task?.isCancelled ?? false + } + } +} +``` + +This static `isCancelled` property is always safe to invoke, i.e. it may be invoked from synchronous or asynchronous functions and will always return the expected result. Do note however that checking cancellation while concurrently setting cancellation may be slightly racy, i.e. if the `cancel` is performed from another thread, the `isCancelled` may not return `true`. + +For tasks that would prefer to immediately exit with a thrown error on cancellation, the task API provides a common error type, `CancellationError`, to communicate that the task was cancelled. The `Task.checkCancellation()` operation will throw `CancellationError` when the task has been cancelled, and is provided as a convenience. + +```swift +/// The default cancellation thrown when a task is cancelled. +/// +/// This error is also thrown automatically by `Task.checkCancellation()`, +/// if the current task has been cancelled. +struct CancellationError: Error { + // no extra information, cancellation is intended to be light-weight + init() {} +} + +extension Task where Success == Never, Failure == Never { + /// Returns `true` if the task is cancelled, and should stop executing. + /// + /// - SeeAlso: `checkCancellation()` + static func checkCancellation() throws +} +``` + +#### Cancellation handlers + +For tasks that want to react immediately to cancellation (rather than, say, waiting until a cancellation error propagates upward), one can install a cancellation handler: + +```swift +/// Execute an operation with cancellation handler which will immediately be +/// invoked if the current task is cancelled. +/// +/// This differs from the operation cooperatively checking for cancellation +/// and reacting to it in that the cancellation handler is _always_ and +/// _immediately_ invoked when the task is cancelled. For example, even if the +/// operation is running code which never checks for cancellation, a cancellation +/// handler still would run and give us a chance to run some cleanup code. +/// +/// Does not check for cancellation, and always executes the passed `operation`. +/// +/// This function returns instantly and will never suspend. +func withTaskCancellationHandler( + operation: () async throws -> T, + onCancel handler: @Sendable () -> Void +) async rethrows -> T +``` + +This function does not, by itself, create a new task, but rather executes the `operation` immediately, and once the `operation` returns the `withTaskCancellationHandler` returns as well (similarly with throwing behaviors). + +Note that the `handler` runs `@Sendable` with the rest of the task, because it +is executed immediately when the task is cancelled, which can happen at any +point. If the task has already been cancelled at the point `withTaskCancellationHandler` is called, the cancellation handler is invoked immediately, before the +`operation` block is executed. + +These properties place rather strict limitations on what a +cancellation handler closure can safely do, but the ability to be triggered at +any point makes cancellation handlers useful for managing the state of related +objects, in cases where either polling cancellation state from within the task +or else propagating it by throwing `CancellationError` is not possible. As one +example, cancellation handlers can be useful in conjunction with +[continuations](0300-continuation.md) to help thread cancellation through +non-`async` event-driven interfaces. For example, if one wanted to wrap up +Foundation's `URLSession` object in an async function interface, cancelling the +`URLSession` if the async task is itself cancelled, then it might look +something like this: + +```swift +func download(url: URL) async throws -> Data? { + var urlSessionTask: URLSessionTask? + + return try withTaskCancellationHandler { + return try await withUnsafeThrowingContinuation { continuation in + urlSessionTask = URLSession.shared.dataTask(with: url) { data, _, error in + if let error = error { + // Ideally translate NSURLErrorCancelled to CancellationError here + continuation.resume(throwing: error) + } else { + continuation.resume(returning: data) + } + } + urlSessionTask?.resume() + } + } onCancel: { + urlSessionTask?.cancel() // runs immediately when cancelled + } +} +``` + +#### Voluntary Suspension + +For long-running operations, say performing many computations in a tight loop +without natural suspend points, it might be beneficial to occasionally check in if the task should perhaps suspend and offer a chance for other tasks to proceed (e.g. if all are executing on a shared, limited-concurrency pool). For this use case, `Task` includes a `suspend()` operation, which is a way to explicitly suspend the current task and give other tasks a chance to run for a while. + +```swift +extension Task where Success == Never, Failure == Never { + static func suspend() async { ... } +} +``` + +We also offer an asynchronous sleep function, which accepts the number of nanoseconds to suspend for: + +```swift +extension Task where Success == Never, Failure == Never { + public static func sleep(nanoseconds duration: UInt64) async throws { ... } +} +``` + +The sleep function accepts a plain integer as nanoseconds to sleep for, which mirrors known top-level functions performing the same action in the synchronous world. It will throw `CancellationError` if the task is cancelled while it sleeps, without waiting for the full sleep duration. + +> The `sleep` function will gain nicer overloads once the standard library has time and deadline types, then the sleep will be able to be expressed as `await Task.sleep(until: deadline)` or `await Task.sleep(for: .seconds(1))` or similar. This proposal is not introducing those time types, so for now a bare bones sleep function is proposed. + +#### Task Groups + +Task groups are created using `withTaskGroup` in any asynchronous context, providing a scope in which new tasks can be created and executed concurrently. + +```swift +/// Starts a new task group which provides a scope in which a dynamic number of +/// tasks may be created. +/// +/// Tasks added to the group by `group.addTask()` will automatically be awaited on +/// when the scope exits. If the group exits by throwing, all added tasks will +/// be cancelled and their results discarded. +/// +/// ### Implicit awaiting +/// When the group returns it will implicitly await for all child tasks to +/// complete. The tasks are only cancelled if `cancelAll()` was invoked before +/// returning, the groups' task was cancelled, or the group body has thrown. +/// +/// When results of tasks added to the group need to be collected, one can +/// gather their results using the following pattern: +/// +/// while let result = await group.next() { +/// // some accumulation logic (e.g. sum += result) +/// } +/// +/// It is also possible to collect results from the group by using its +/// `AsyncSequence` conformance, which enables its use in an asynchronous for-loop, +/// like this: +/// +/// for await result in group { +/// // some accumulation logic (e.g. sum += result) +/// } +/// +/// ### Cancellation +/// If the task that the group is running in is cancelled, the group becomes +/// cancelled and all child tasks created in the group are cancelled as well. +/// +/// Since the `withTaskGroup` provided group is specifically non-throwing, +/// child tasks (or the group) cannot react to cancellation by throwing a +/// `CancellationError`, however they may interrupt their work and e.g. return +/// some best-effort approximation of their work. +/// +/// If throwing is a good option for the kinds of tasks created by the group, +/// consider using the `withThrowingTaskGroup` function instead. +/// +/// Postcondition: +/// Once `withTaskGroup` returns it is guaranteed that the `group` is *empty*. +/// +/// This is achieved in the following way: +/// - if the body returns normally: +/// - the group will await any not yet complete tasks, +/// - once the `withTaskGroup` returns the group is guaranteed to be empty. +func withTaskGroup( + of childTaskResult: ChildTaskResult.Type, + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout TaskGroup) async -> GroupResult +) async -> GroupResult { ... } + + +/// Starts a new throwing task group which provides a scope in which a dynamic +/// number of tasks may be created. +/// +/// Tasks added to the group by `group.addTask()` will automatically be awaited on +/// when the scope exits. If the group exits by throwing, all added tasks will +/// be cancelled and their results discarded. +/// +/// ### Implicit awaiting +/// When the group returns it will implicitly await for all created tasks to +/// complete. The tasks are only cancelled if `cancelAll()` was invoked before +/// returning, the groups' task was cancelled, or the group body has thrown. +/// +/// When results of tasks added to the group need to be collected, one can +/// gather their results using the following pattern: +/// +/// while let result = await try group.next() { +/// // some accumulation logic (e.g. sum += result) +/// } +/// +/// It is also possible to collect results from the group by using its +/// `AsyncSequence` conformance, which enables its use in an asynchronous for-loop, +/// like this: +/// +/// for try await result in group { +/// // some accumulation logic (e.g. sum += result) +/// } +/// +/// ### Thrown errors +/// When tasks are added to the group using the `group.addTask` function, they may +/// immediately begin executing. Even if their results are not collected explicitly +/// and such task throws, and was not yet cancelled, it may result in the `withTaskGroup` +/// throwing. +/// +/// ### Cancellation +/// If the task that the group is running in is cancelled, the group becomes +/// cancelled and all child tasks created in the group are cancelled as well. +/// +/// If an error is thrown out of the task group, all of its remaining tasks +/// will be cancelled and the `withTaskGroup` call will rethrow that error. +/// +/// Individual tasks throwing results in their corresponding `try group.next()` +/// call throwing, giving a chance to handle individual errors or letting the +/// error be rethrown by the group. +/// +/// Postcondition: +/// Once `withThrowingTaskGroup` returns it is guaranteed that the `group` is *empty*. +/// +/// This is achieved in the following way: +/// - if the body returns normally: +/// - the group will await any not yet complete tasks, +/// - once the `withTaskGroup` returns the group is guaranteed to be empty. +/// - if the body throws: +/// - all tasks remaining in the group will be automatically cancelled. +func withThrowingTaskGroup( + of childTaskResult: ChildTaskResult.Type, + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout ThrowingTaskGroup) async throws -> GroupResult +) async rethrows -> GroupResult { ... } + +/// A group of tasks, each of which produces a result of type `TaskResult`. +struct TaskGroup { + // No public initializers +} +``` + +`TaskGroup` has no public initializers; instead, an instance of `TaskGroup` is passed in to the `body` function of `withTaskGroup`. This instance should not be copied out of the `body` function, because doing so can break the child task structure. + +> **Note**: Swift does not currently have a way to ensure that the task group passed into the `body` function is not copied elsewhere, so we therefore rely on programmer discipline in a similar manner to, e.g., [`Array.withUnsafeBufferPointer`](https://developer.apple.com/documentation/swift/array/2994771-withunsafebufferpointer). However, in the case of task groups, we can at least provide a runtime assertion if one attempts to use the task group instance after its corresponding scope has ended. + +The result of `withTaskGroup` is the result produced by the `body` function. The `withThrowingTaskGroup` version of the function allows for the task group to throw, and if that happens all tasks it contained are implicitly cancelled (and awaited on) before rethrowing the error. + +> Note: Sadly it is not presently possible to implement this throwing/non-throwing functionality with a single function. The complex relationship of throwing `group.addTask` with a throwing `next` as well as corresponding throwing/non-throwing `AsyncSequence` conformances make it impossible to implement all in one function/type today. + +Note also that the `withThrowingTaskGroup` uses a `ThrowingTaskGroup`, however specifying the type of that error is not possible. This is because this Failure parameter on the `ThrowingTaskGroup` in only used as future-proof API in case Swift were to gain typed throwing at some point in time. This design makes no promises nor does it assume typed throws are actually going to happen though. + +A task group _guarantees_ that it will `await` all tasks that were added to it before it returns. + +This waiting can be performed either: +- by the code within the task group itself (e.g., using `next()` repeatedly until it returns `nil`, described below), or +- implicitly in the task group itself when returning from the `body`. + +By default, the task group will schedule child tasks added to the group on the default global concurrent executor. In the future it is likely that it will be possible to customize the executor tasks are started on with an optional executor parameter to `addTask`. + +##### Creating TaskGroup child tasks + +Within the `body` function, tasks may be added dynamically with the `addTask` operation. Each task produces a value of the same type (the `ChildTaskResult` generic parameter): + +```swift +extension TaskGroup { + /// Unconditionally create a child task in the group. + /// + /// The child task will be executing concurrently with the group, and its result + /// may be collected by calling `group.next()` or iterating over the group gathering + /// all submitted task results from the group. + mutating func addTask( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> ChildTaskResult + ) + + /// Attempts to create a child task in the group, unless the group is already cancelled. + /// + /// If the task that initiates the call to `addTaskUnlessCancelled` was already cancelled, + /// or if the group was explicitly cancelled by invoking `group.cancelAll()`, no child + /// task will be created. + /// + /// The child task will be executing concurrently with the group, and its result + /// may be collected by calling `group.next()` or iterating over the group gathering + /// all submitted task results from the group. + /// + /// Returns true if the task was created successfully, and false otherwise. + mutating func addTaskUnlessCancelled( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> ChildTaskResult + ) -> Bool +} + +extension ThrowingTaskGroup { + mutating func addTask( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> ChildTaskResult + ) + + mutating func addTaskUnlessCancelled( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> ChildTaskResult + ) -> Bool +} +``` + +`group.addTask` creates a child task in the task group to execute the given `operation` function concurrently. The task will be a child of the task that initially created the task group (via `withTaskGroup`), and will have the same priority as that task unless given a new priority with as an argument. Generally, it is recommended to not specify priority manually. + +The `addTask` operation always succeeds in adding a new child task to the group, even if the task running the group has been cancelled or the group was cancelled explicitly with `group.cancelAll`. In cases where the task group has already +been cancelled, the new child task will be created in the `cancelled` state. +To avoid this, the `addTaskUnlessCancelled` function checks if a group is cancelled before attempting to create the task, and returns a `Bool` that is true if +the task was successfully created. This allows for simple implementation of groups which should "keep creating tasks until cancelled". + +Cancelling a specific task group child task does _not_ cancel the entire group or any of its siblings. + +> Previously the `group.addTask` operation was designed to be a suspension point, which was intended to be a simple form of back-pressure where the group could decide to not allow more than N tasks to be running concurrently. This has not been fully designed nor implemented though, so currently has been moved to a future direction. + + +##### Querying tasks in the group + +The `next()` operation allows one to gather the results from the tasks that have been created in the group. It produces the result from one of the tasks in the group, whether it is the normal result or a thrown error. + +```swift +extension TaskGroup: AsyncSequence { + + /// Wait for the a child task that was added to the group to complete, + /// and return (or rethrow) the value it completed with. If no tasks are + /// pending in the task group this function returns `nil`, allowing the + /// following convenient expressions to be written for awaiting for one + /// or all tasks to complete: + /// + /// Await on a single completion: + /// + /// if let first = try await group.next() { + /// return first + /// } + /// + /// Wait and collect all group child task completions: + /// + /// while let first = try await group.next() { + /// collected += value + /// } + /// return collected + /// + /// Awaiting on an empty group results in the immediate return of a `nil` + /// value, without the group task having to suspend. + /// + /// It is also possible to use `for await` to collect results of a task groups: + /// + /// for await value in group { + /// collected += value + /// } + /// + /// ### Thread-safety + /// Please note that the `group` object MUST NOT escape into another task. + /// The `group.next()` MUST be awaited from the task that had originally + /// created the group. It is not allowed to escape the group reference. + /// + /// Note also that this is generally prevented by Swift's type-system, + /// as the `add` operation is `mutating`, and those may not be performed + /// from concurrent execution contexts, such as child tasks. + /// + /// ### Ordering + /// Order of values returned by next() is *completion order*, and not + /// submission order. I.e. if tasks are added to the group one after another: + /// + /// group.addTask { 1 } + /// group.addTask { 2 } + /// + /// print(await group.next()) + /// /// Prints "1" OR "2" + mutating func next() async -> ChildTaskResult? { ... } + + /// Wait for all of the child tasks to complete. + /// + /// This operation is the equivalent of + /// + /// for await _ in self { } + /// + mutating func waitForAll() async { ... } + + /// Query whether the group has any remaining tasks. + /// + /// Task groups are always empty upon entry to the `withTaskGroup` body, and + /// become empty again when `withTaskGroup` returns (either by awaiting on all + /// pending tasks or cancelling them). + /// + /// - Returns: `true` if the group has no pending tasks, `false` otherwise. + var isEmpty: Bool { ... } +} +``` + +```swift +extension ThrowingTaskGroup: AsyncSequence { + + /// Wait for the a child task that was added to the group to complete, + /// and return (or rethrow) the value it completed with. If no tasks are + /// pending in the task group this function returns `nil`, allowing the + /// following convenient expressions to be written for awaiting for one + /// or all tasks to complete: + /// + /// Await on a single completion: + /// + /// if let first = try await group.next() { + /// return first + /// } + /// + /// Wait and collect all group child task completions: + /// + /// while let first = try await group.next() { + /// collected += value + /// } + /// return collected + /// + /// Awaiting on an empty group results in the immediate return of a `nil` + /// value, without the group task having to suspend. + /// + /// It is also possible to use `for await` to collect results of a task groups: + /// + /// for await try value in group { + /// collected += value + /// } + /// + /// ### Thread-safety + /// Please note that the `group` object MUST NOT escape into another task. + /// The `group.next()` MUST be awaited from the task that had originally + /// created the group. It is not allowed to escape the group reference. + /// + /// Note also that this is generally prevented by Swift's type-system, + /// as the `add` operation is `mutating`, and those may not be performed + /// from concurrent execution contexts, such as child tasks. + /// + /// ### Ordering + /// Order of values returned by next() is *completion order*, and not + /// submission order. I.e. if tasks are added to the group one after another: + /// + /// group.addTask { 1 } + /// group.addTask { 2 } + /// + /// print(await group.next()) + /// /// Prints "1" OR "2" + /// + /// ### Errors + /// If an operation added to the group throws, that error will be rethrown + /// by the next() call corresponding to that operation's completion. + /// + /// It is possible to directly rethrow such error out of a `withTaskGroup` body + /// function's body, causing all remaining tasks to be implicitly cancelled. + mutating func next() async throws -> ChildTaskResult? { ... } + + /// Wait for a task to complete and return the result or thrown error packaged in + /// a `Result` instance. Returns `nil` only when there are no tasks left in the group. + mutating func nextResult() async -> Result? + + /// Wait for all of the child tasks to complete, or throws an error if any of the + /// child tasks throws. + /// + /// This operation is the equivalent of + /// + /// for try await _ in self { } + /// + mutating func waitForAll() async throws { ... } + + /// Query whether the task group has any remaining tasks. + var isEmpty: Bool { ... } +} +``` + +The `next()` operation may typically be used within a `while` loop to gather the results of all outstanding tasks in the group, e.g., + +```swift +while let result = await group.next() { + // some accumulation logic (e.g. sum += result) +} + +// OR + +while let result = try await group.next() { + // some accumulation logic (e.g. sum += result) +} +``` + +`TaskGroup` also conforms to the [`AsyncSequence` protocol](0298-asyncsequence.md), allowing the child tasks' results to be iterated in a `for await` loop: + +```swift +for await result in group { // non-throwing TaskGroup + // some accumulation logic (e.g. sum += result) +} + +// OR + +for try await result in group { // ThrowingTaskGroup + // some accumulation logic (e.g. sum += result) +} +``` + +With this pattern, if a single task throws an error, the error will be propagated out of the `body` function and the task group itself. + +To handle errors from individual tasks, one can use a do-catch block or the `nextResult()` method. For example, one might want to implement a function which starts `N` tasks, and reports back the first `m` successful results. This is simple to implement with a task group, by means of collecting results until the `results` array have accumulated `m` results, at which point we can cancel all remaining tasks and return from the group: + +```swift +func gather(first m: Int, of work: [Work]) async throws -> [WorkResult] { + assert(m <= work.count) + + return withTaskGroup(of: WorkResult.self) { group in + for w in work { + group.addTask { await w.doIt() } // create child tasks to perform the work + } + + var results: [WorkResult] = [] + while results.count <= m { + switch try await group.nextResult() { + case nil: return results + case .success(let r): results.append(r) + case .failure(let e): print("Ignore error: \(e)") + } + } + } +} +``` + +##### Task group cancellation + +There are several ways in which a task group can be cancelled. In all cases, all of the tasks in the group are cancelled, and any new tasks created in the group will start out cancelled. The three ways in which a task group can be cancelled are: + +1. When an error is thrown out of the `body` of `withTaskGroup`, +2. When the task in which the task group itself was created is cancelled, or +3. When the `cancelAll()` operation is invoked. + +A group's cancellation state can be queried by reading the `isCancelled` +property. + +```swift +extension TaskGroup { + /// Cancel all the remaining tasks in the group. + /// + /// A cancelled group will not will NOT accept new tasks being added into it. + /// + /// Any results, including errors thrown by tasks affected by this + /// cancellation, are silently discarded. + /// + /// This function may be called even from within child (or any other) tasks, + /// and will reliably cause the group to become cancelled. + /// + /// - SeeAlso: `Task.isCancelled` + /// - SeeAlso: `TaskGroup.isCancelled` + func cancelAll() { ... } + + /// Returns `true` if the group was cancelled, e.g. by `cancelAll`. + /// + /// If the task currently running this group was cancelled, the group will + /// also be implicitly cancelled, which will be reflected in the return + /// value of this function as well. + /// + /// - Returns: `true` if the group (or its parent task) was cancelled, + /// `false` otherwise. + var isCancelled: Bool { get } +} +``` + +For example: + +```swift +func chopVegetables() async throws -> [Vegetable] { + var veggies: [Vegetable] = [] + + try await withThrowingTaskGroup(of: Vegetable.self) { group in + print(group.isCancelled) // prints false + + group.addTask { + group.cancelAll() // Cancel all work in the group + throw UnfortunateAccidentWithKnifeError() + } + group.addTask { + return try await chop(Onion()) + } + + do { + while let veggie = try await group.next() { + veggies.append(veggie) + } + } catch { + print(group.isCancelled) // prints true now + let added = group.addTaskUnlessCancelled { + try await chop(SweetPotato()) + } + print(added) // prints false, no child was added to the cancelled group + } + } + + return veggies +} +``` + + +## Source compatibility + +This change is purely additive to the source language. + +## Effect on ABI stability + +This change is purely additive to the ABI. + +## Effect on API resilience + +All of the changes described in this document are additive to the language and are locally scoped, e.g., within function bodies. Therefore, there is no effect on API resilience. + +## Revision history + +### Review changes + +Changes after the fourth review: + +* added `UnsafeCurrentTask.cancel()` to cancel the task. + +Changes after the third review: + +- renamed `Task.sleep(_:)` to `Task.sleep(nanoseconds:)`. This makes it clear that the wait is in nanoseconds, and leaves API space open for a `sleep(_:)` based on a better duration type in the future. +- made `Task.sleep(nanoseconds:)` throwing; it will throw `CancellationError` if the sleeping task was cancelled. +- renamed `TaskGroup.async` and `TaskGroup.asyncUnlessCancelled` to `TaskGroup.addTask` and `TaskGroup.addTaskUnlessCancelled`. The fundamental behavior here is that we're adding a task to the group. `add` by itself does not suffice, because we aren't adding a value (accessible via `next()`), we are adding a task whose value will be accessible via `next()`. It also parallels the use of `Task { ... }` to create top-level tasks. +- renamed `TaskPriority.default` to `TaskPriority.medium`, because `nil` passed in to a `TaskPriority?` parameter is effectively the default for most APIs. +- added `TaskGroup.waitForAll` and `ThrowingTaskGroup.waitForAll`. +- renamed `Task.yield()` to `Task.suspend()`, which more accurately represents what this operation action does, and leaves the name "yield" for future work on generators. + +Changes after the second review: + +- remove `Priority.unspecified` and use `nil` as unspecified value. +- introduce platform independent priority names: `high`, `default`, `low`, `background`. The Apple platform specific names remain as aliases and can be used on apple platforms where they make sense. These names have a long history and were even originally used in dispatch itself. We discussed and confirmed with various teams inside Apple that those names work well for the future evolution of the platform. +- future-proof the `TaskPriority` type by changing it to a `RawRepresentable` `struct` with static computed properties. We do not immediately have any plans to introduce new priorities, but want to allow for such future extension if necessary. +- remove the ability to create new tasks at the `userInteractive` priority. This priority will be used only be the runtime itself, e.g. by the main thread and automatically inherited properly by any other tasks (and downgraded to `userInitiated`) +- `TaskGroup.spawn` and `TaskGroup.spawnUnlessCancelled` have been renamed to `TaskGroup.async` and `TaskGroup.asyncUnlessCancelled`. +- remove `Task.current` and the general ability to get hold of a child task instance. This change unlocks important optimizations in the compiler and runtime +- collapse `Task.Handle` into `Task`. This is the most-used type in the Task API and should have the shortest name. +- merge the `async { }` proposal ([pitched here](https://forums.swift.org/t/initiating-asynchronous-work-from-synchronous-code/47714)) into this proposal, such that we have always to create tasks in this proposal to review at-once, and make it the task instance initializer `Task { ... }` +- rename `detach` to `Task.detached` as it is similar, but less favored to use that API. +- re-order parameters of `withTaskCancellationHandler` from `handler, operation` to `(operation, onCancel handler)` which seems to be a more common pattern for such APIs where the "main closure" (the operation) comes first +- move `Task.Priority` out to `TaskPriority` and `Task.CancellationError` out to the top-level `CancellationError` +- replace `get()` with an async property `value`, and `getResult()` with an async property `result` + +Changes after first review: + +* `Task.current` now returns an optional `Task`: `var current: Task? { get }`, which depending on context it is called from might be `nil`. + * This API is not intended to be used "a lot", and if sure a task will be available one can always force unwrap it. + * Most usages of tasks are rather intended to go through the static functions/properties on Task which implicitly works on the current task. +* `Task.unsafeCurrent` becomes a top-level `withUnsafeCurrentTask { maybeUnsafeTask in }` + * This better explains the intended semantics of not escaping storing the unsafe task reference. +* Adopt `spawn...` terminology for "spawning tasks" + * `TaskGroup`'s `group.add` becomes `group.spawn` + * Creating a child task will eventually be `spawn ` +* Based on feedback, `runDetached` becomes `detach` because of how often it may be necessary to reach for. +* Moving away from using `Task` as namespace for everything + * rename `TaskGroup` to `TaskGroup`, and introduce `ThrowingTaskGroup` + * make `Task.unsafeCurrent` a free function`withUnsafeCurrentTask` +* Task group type parameter renames: `TaskGroup` becomes `ChildTaskResult` resulting in: `public func withTaskGroup(of childTaskResultType: ChildTaskResult.Type, returning returnType: GroupResult.Type = GroupResult.self, body: (inout TaskGroup) async throws -> GroupResult) async rethrows -> GroupResult` resulting in a more readable call site: `withTaskGroup(of: Int.self)` and optionally `withTaskGroup(of: Int.self, returning: Int.self)` +* For now remove `startingChildTasksOn` from `withTaskGroup` since this is only doable with Custom Executors which are pending review still. +* Move `Task.withCancellationHandler` to a top level function `withTaskCancellationHandler` which reads more logically, as it does not create a task by itself. +* Make `group.spawn` return `TaskGroup.Spawned` that serves both the purpose of knowing if the task was `spawned.successfully` and also obtaining the `Task.Handle` of a successfully spawned task. Thanks to Paulo Faria for reminding us to revisit this topic. +* The spawn parameter `overridingPriority` has been renamed to `priority` not to confuse existing users on Apple platforms where "override" has the specific meaning more similar to what we call "priority escalation". +* Task group `spawn` now always spawns a child task rather than only when the group is not cancelled. +* Task groups gain `spawnUnlessCancelled -> Bool` which explains the semantics intended by the previous spawn signature more clearly. The returned value is just a boolean signalling if the spawn was successfully or not. +* Some functions were accepting `Task.Priority?` which is unnecessary because we have `.unspecified`, so those functions now accept `Task.Priority` defaulting it to `.unspecified` + + +### Pitch changes + +* Changes in the third pitch: + * Factored `with*Continuation` into [its own proposal](https://github.com/swiftlang/swift-evolution/pull/1244). + * Factored `async let` into [its own proposal](https://github.com/DougGregor/swift-evolution/pull/50). + * `Task` becomes a `struct` with instance functions, introduction of `Task.current`, `Task.unsafeCurrent` and the `UnsafeCurrentTask` APIs + * `Task.Group` now conforms to [the `AsyncSequence` protocol](0298-asyncsequence.md). + * `runDetached` and `Task.Group.add` now accept [executor](https://github.com/swiftlang/swift-evolution/pull/1257) arguments to specify where the newly-spawned tasks are initially scheduled. +* Changes in the second pitch: + * Added a "desugaring" of `async let` to task groups and more motivation for the structured-concurrency parts of the design. + * Reflowed the entire proposal to focus on the general description of structured concurrency first, the programming model with syntax next, and then details of the language features and API design last. + * Reworked the presentation of the Task APIs with more rationale for the design. + * Added more discussion of why futures aren't more prominent. + * "Task nursery" has been replaced with "task group". + * Added support for asynchronous `@main` and top-level code. + * Specify that `try` is not required in the initializer of an `async let`, because the thrown error is only observable when reading from one of the variables. + * `withUnsafe(Throwing)Continuation` functions have been moved out of the `Task` type. + * Note that an `async let` variable can only be captured by a non-escaping closure. + * Removed the requirement that an `async let` variable be awaited on all paths. +* Original pitch [document](https://github.com/DougGregor/swift-evolution/blob/06fd6b3937f4cd2900bbaf7bb22889c46b5cb6c3/proposals/nnnn-structured-concurrency.md) + +## Alternatives Considered + +### Prominent futures + +The design of task groups intentionally avoids exposing any task handles (futures) for child tasks. This ensures that the structure of structured concurrency, where all child tasks complete before their parent task, is maintained. That helps various properties such as priorities, deadlines, and cancellation to propagate in a meaningful way down the task tree. + +However, an alternative design would bring futures to the forefront. One could introduce a new `runChild` operation that creates a new child task (of the current task), and then retrieve the result of that child task using the provided `Task`. To ensure that child tasks complete before the scope exits, we would require some kind of scoping mechanism that provides similar behavior to task groups. For example, the `makeDinner` example would be something like: + +```swift +func makeDinner() async throws -> Meal { + Task.withChildScope { scope in + let veggiesHandle = scope.runChild { try await chopVegetables() } + let meatHandle = scope.runChild { await marinateMeat() } + let ovenHandle = scope.runChild { await preheatOven(temperature: 350) } + + let dish = Dish(ingredients: await [try veggiesHandle.get(), meatHandle.get()]) + return try await ovenHandle.get().cook(dish, duration: .hours(3)) + } +} +``` + +The task handles produced by `runChild` should never escape the scope in which they are created, although there is no language mechanism to enforce this. Moreover, the difference between detached and child tasks becomes blurred: both return the same `Task` type, but some have extra restrictions while others don't. So while it is possible to maintain structured concurrency with a future-centric design, it requires more programmer discipline (even for otherwise simple tasks), and provides less structure for the Swift compiler, optimizer, and runtime to use to provide an efficient implementation of child tasks. + +## Future directions + +### `async let` to create child tasks within a scope + +Although our design de-emphasizes futures for structured tasks, for the reasons +delineated above, we acknowledge that it will be common to want to pass +heterogeneous values up from child tasks to their parent. This is possible +within the existing task group APIs, though not ideal, as illustrated by the +original `makeDinner` example. + +[SE-0317 `async let`](0317-async-let.md) provides syntactic sugar for the +creation of heterogenous child tasks, capturing their values in local +variables that use [effectful properties](0310-effectful-readonly-properties.md) +to wait for the result of the child task's completion. For example, this allows +the `makeDinner` example to be written as: + + +```swift +func makeDinner() async throws -> Meal { + async let veggies = chopVegetables() + async let meat = marinateMeat() + async let oven = preheatOven(temperature: 350) + + let dish = Dish(ingredients: await [try veggies, meat]) + return await oven.cook(dish, duration: .hours(3)) +} +``` + +This provides a lightweight syntax for a very common dataflow pattern +between child tasks and parents within a task group. + +### `@Sendable` closure checking for task groups + +In addition to `async let`, the scoped nature of task groups and child tasks would make it natural for child tasks to be able to do more ad-hoc mutation of captured state from their captured context. Because child tasks are guaranteed to +have completed by the time a `withTaskGroup` block finishes executing, it would theoretically be safe to allow them to mutate captured local variables, as long as every child task captures a disjoint set of variables, and the variables are not referenced in the enclosing context until the task group completes, as in: + +```swift +var numApplesProcessed = 0 +var numBananasProcessed = 0 +withTaskGroup { group in + // One child task handles apples: + group.addTask { + for apple in apples { + await processApple(apple) + numApplesProcessed += 1 + } + } + // And one child task handles bananas: + group.addTask { + for banana in bananas { + await processBanana(banana) + numBananasProcessed += 1 + } + } +} +print("\(numApplesProcessed + numBananasProcessed) fruits processed") +``` + +However, Swift's type checker does not have any special knowledge of `withTaskGroup`, and a conservative analysis of the `@Sendable` closures for each child +task has to assume that the closures could be executed at any time, and so apply +the usual rules banning capture of mutated variables. To allow for a more +natural coding style in these situations, it would be useful if the analysis +understood the special behavior of task groups and allowed for mutation in +captures when it's safe in cases like this. + +### Suspending `await group.addTask` + +Initially the `group.addTask` operation was designed with the idea of being an asynchronous function which might suspend if the group determined that it is "too full" and should apply this naive form of back-pressure to the task creating more tasks into the group. + +This was not implemented nor is it clear how efficient and meaningful this form of back-pressure really would be. A naive version of these semantics is possible to implement by balancing pending and completed task counts in the group by plain variables, so removing this implementation does not prevent developers from implementing such "width limited" operations per se. + +The way to back-pressure submissions should also be considered in terms of how it relates to async let and general task creation mechanisms, not only groups. We have not figured this out completely, and rather than introduce an not-implemented API which may or may not have the right shape, for now we decided to punt on this feature until we know precisely if and how to apply this style of back-pressure on creating tasks throughout the system. + +## Proposal history + +When this proposal was originally accepted, the description of `withThrowingTaskGroup` stated that any remaining tasks in the group will be cancelled if any task throws after the body function returns normally. This behavior was never implemented; throws from tasks are ignored in this state. By the time this discrepancy was detected, several years had passed, and the Language Steering Group decided that restoring the proposed behavior (if desired) would require a new proposal. This document has been revised to reflect the implemented behavior. diff --git a/proposals/0305-swiftpm-binary-target-improvements.md b/proposals/0305-swiftpm-binary-target-improvements.md new file mode 100644 index 0000000000..e525eeebb6 --- /dev/null +++ b/proposals/0305-swiftpm-binary-target-improvements.md @@ -0,0 +1,279 @@ +# Package Manager Binary Target Improvements + +* Proposal: [SE-0305](0305-swiftpm-binary-target-improvements.md) +* Authors: [Anders Bertelrud](https://github.com/abertelrud), [Tom Doron](https://github.com/tomerd) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.6)** +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0305-package-manager-binary-target-improvements/47742) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md) +* Forum Discussion: [SE-0305: Package Manager Binary Target Improvements](https://forums.swift.org/t/se-0305-package-manager-binary-target-improvements/45589) +* Review: [1](https://forums.swift.org/t/se-0305-package-manager-binary-target-improvements/) [2](https://forums.swift.org/t/se-0305-2nd-review-package-manager-binary-target-improvements/) +* Implementation: Available in [recent `main` snapshots](https://swift.org/download/#snapshots) + +## Introduction + +This proposal extends SwiftPM binary targets to also support other kinds of prebuilt artifacts, such as command line tools. It does not in and of itself add support for non-Darwin binary libraries, although the proposed improvements could be a step towards such support. + +## Motivation + +The Swift Package Manager’s [`binaryTarget` type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md) lets packages vend libraries that either cannot be built in Swift Package Manager for technical reasons, or for which the source code cannot be published for legal or other reasons. + +In the current version of SwiftPM, binary targets only support libraries in an Xcode-oriented format called *XCFramework*, and only for Apple platforms. + +As part of [SE-0303 SwiftPM Extensible Build Tools](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md), SwiftPM will need a way to allow packages to vend prebuilt binaries containing command line tools that can be invoked during the build. This is because: + +* many popular command line tools (such as `protoc`) do not build using SwiftPM, and +* tools that should run during “prebuild” (i.e. before the build starts) cannot themselves be built as part of the build + +There are many other reasons to extend binary targets in SwiftPM, such as to support binary libraries for non-Apple platforms. While this proposal is specifically focused on allowing the vending of command line tools and related kinds of artifacts, it tries to do so with an eye toward making binary targets more flexible and cross-platform in the future. + +## Proposed solution + +This proposal extends binary targets to allow a new “artifact bundle” format in addition to the XCFramework format that is supported in the current version of SwiftPM. This proposal does not change how `binaryTarget`s are declared in package manifests, nor how they are used for XCFrameworks. Instead, this proposal builds on the existing support for binary targets. + +In this proposal, *artifact bundles* are directory structures that can contain multiple *artifacts*, each having a unique identifier within the bundle. Each artifact consists of a set of variants that support various architectures and platforms. An artifact bundle also contains a manifest file that describes the artifacts and their variants. + +Prior to this proposal, a binary target could reference either: + +* a remote `.zip` file with an XCFramework directory at its top level, or +* less commonly, a plain XCFramework directory embedded inside the package directory + +This proposal extends each of these cases to alternatively support an artifact bundle in place of the XCFramework (a single binary target cannot contain both an XCFramework and an artifact bundle). + +In addition to allowing a URL reference to a `.zip` file containing a single artifact bundle, this proposal allows the remote URL to refer to an *artifact bundle index* file that in turn refers to multiple `.zip` files, each containing the artifact bundle for a variant or a set of variants of the same conceptual artifact. This optimization allows SwiftPM to download only the variants it needs. + +The artifact index file maps a variant selector to the `.zip` file that contains the appropriate variant of the artifact bundle. The *Detailed design* section below provides more information about the variant selector and how the appropriate variant is chosen. + +Once a `.zip` file has been selected using the index file, it is downloaded in the same way as in the case of a single `.zip` file (which is exactly the same as for all binary targets today). It is unarchived to a location in the local file system, and the path of the artifact bundle becomes available to the rest of SwiftPM. + +Regardless of whether the artifact bundle is local or remote, and whether or not it uses an index file, the appropriate variants of the artifacts declared in an artifact bundle will be made available to any package plugins that have a target dependency on the `binaryTarget`. This is done through API in the `PackagePlugin` library as described in [SE-0303](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md). + +## Detailed design + +This section describes *artifact bundles*, *artifact bundle manifests*, and *artifact bundle indices* in detail. + +### Artifact bundle + +The proposal defines the structure and semantics of an *artifact bundle* to be a directory that has the filename suffix `.artifactbundle` and which has the following content: + +* a set of one or more *artifacts*, each with an identifier string that is unique within the bundle +* within each artifact, a set of *variants*, each with an identifier string that is unique with the artifact +* a manifest file containing information about the individual artifacts and their variants + +Artifact bundles may appear as the referent of the `path` parameter of a `binaryTarget`, or as the sole top-level entity in a `.zip` file referenced by either a `url` parameter in the package manifest or through an artifact bundle index file (as described below). + +The structure of the artifact bundle is: + +``` +.artifactbundle + ├ info.json + ├ + │ ├ + │ │ ├ + │ │ └ + │ └ + │ ├ + │ └ + ├ + │ └ + │ ├ + │ └ + │ + ┆ └┄ +``` + +The manifest is always at the top level and is named `info.json`. Its contents are described in the next subsection. + +At the top level of the artifact bundle directory is a subdirectory for each artifact in the bundle. The names of the artifacts are arbitrary, but must be unique within the bundle. These are the names that are used in the plugin API when looking up a binary artifact. A plugin has access to the artifacts defined in the artifact bundles specified by all the binary targets on which it has declared a dependency. + +Within each artifact directory are variant directories. As with the names of artifacts, these names are arbitrary but must be unique within the artifact directory. + +All name lookup is case-sensitive. In order to be resilient on case-insensitive file systems, artifact bundles should avoid using pairs of names that differ only in case. + +Each artifact variant directory is the root of a file system hierarchy that can be made available to a plugin. The variant that is made available depends on the target triple of the host toolchain at the time of use. + +### Artifact bundle manifest + +The artifact bundle manifest is a JSON file named `info.json` at the top level of the artifact bundle. It has the following contents: + +```json +{ + "schemaVersion": "1.0", + "artifacts": { + "": { + "version": "", + "type": "executable", + "variants": [ + { + "path": "", + "supportedTriples": [ "", ... ] + }, + ... + ] + }, + ... + } +} +``` + +The top level of the artifact bundle manifest contains: + +* `schemaVersion` — in this proposal `1.0`; this allows changes to the format in the future +* `artifacts` — a mapping of artifact identifiers to artifact dictionaries + +Each artifact dictionary contains: + +* `version` — an arbitrary version number for informational purposes (available to plugins) +* `type` — in this proposal always `executable`; this allows further support for other types of artifacts in the future +* `variants` — an array of variant dictionaries + +Each variant dictionary contains: + +* `path` — the subpath of the command line executable (relative to the bundle) +* `supportedTriples` — array of target triples supported by this variant + +Note that although the `type` key is always `executable` in this proposal, this is expected to allow the support for artifact bundles to be extended in future proposals to support libraries, resource sets, and other types of binary artifacts. + +This proposal uses [target triples](https://clang.llvm.org/docs/CrossCompilation.html#target-triple) as the variant selectors. A single variant may support more than one target triple, as in the case of universal binaries. + +### How artifact bundles are processed + +As with binary targets today, after downloading the `.zip` file and validating its integrity using the checksum specified in the manifest, SwiftPM unarchives the contents of the `.zip` into an intermediate location in the local file system. If the archive does not contain a `.xcframework`, SwiftPM will look for a `.artifactbundle` directory instead. It is an error for both an `.artifactbundle` and a `.xcframework` to be present in the same `.zip`. + +If SwiftPM finds a `.artifactbundle` directory, it will try to load the `info.json` within it. If the `schemaVersion` is not recognized, then SwiftPM will emit an error and not process the artifact bundle further. Otherwise it will register the artifacts present in the artifact bundle, for later use by plugins. + +When a plugin asks for a tool with a particular identifier, SwiftPM will consider the artifact bundles specified by any binary targets on which the plugin depends. The artifacts defined in these bundles will be made available to the plugin and will be translated to the paths at which the executables have been unarchived. The plugin can access any support files provided with the executable in the same way. + +Any lookup of names within the artifact bundle is case sensitive. + +### Artifact bundle index + +To avoid downloading files that will not be needed, an artifact bundle can be split up into multiple bundles. Each of these bundles has the same format as any other artifact bundle, but contains only a subset of the variants. + +An example could include having one bundle for Apple platforms, another for Windows, and others for different Linux variants. + +An artifact bundle index is a JSON file with a `.artifactbundleindex` extension, and that has the following contents: + +```json +{ + "schemaVersion": "1.0", + "bundles": [ + { + "fileName": "", + "checksum": "", + "supportedTriples": [ "", ... ] + }, + ... + ] +} +``` + +The top level of the artifact bundle index contains: + +* `schemaVersion` — in this proposal `1.0`; this allows changes to the format in the future +* `bundles` — a list of `.zip` files that contain bundles containing subsets of variants of the same conceptual artifacts + +Each bundle dictionary contains: + +* `fileName` — the filename of the `.zip` archive containing the artifact bundle +* `checksum` — the checksum of the `.zip` archive, computed using `swift` `package` `checksum` +* `supportedTriples` — array of all the target triples supported by the variants in the artifact bundle + +The individual `.zip` files are expected to be located next to the `.artifactbundleindex` file, and thus only their filenames are listed in the index. + +The checksum for each `.zip` file in the index is computed in the same manner as for other binary `.zip` files, i.e. using `swift package compute-checksum`. The checksum in the binary target that references the `.artifactbundleindex` is the checksum of the `.artifactbundleindex` file itself. In this way, SwiftPM can validate the integrity of any of the `.zip` archives referenced by the index file. + +## Example + +Here is a hypothetical example of how the Protobuf compiler (`protoc`) could be vended as an artifact bundle: + +``` +protoc.artifactbundle +├── info.json +├── protoc-3.15.6-linux-gnu +│ ├── bin +│ │ └── protoc +│ └── include +│ └── etc.proto +├── protoc-3.15.6-macos +│ ├── bin +│ │ └── protoc +│ └── include +│ └── etc.proto +└── protoc-3.15.6-windows + ├── bin + │ └── protoc.exe + └── include + └── etc.proto +``` + +The contents of the `info.json` manifest would be: + +```json +{ + "schemaVersion": "1.0", + "artifacts": { + "protoc": { + "type": "executable", + "version": "3.15.6", + "variants": [ + { + "path": "protoc-3.15.6-linux-gnu/bin/protoc", + "supportedTriples": ["x86_64-unknown-linux-gnu"] + }, + { + "path": "protoc-3.15.6-macos/bin/protoc", + "supportedTriples": ["x86_64-apple-macosx", "arm64-apple-macosx"] + }, + { + "path": "protoc-3.15.6-windows/bin/protoc.exe", + "supportedTriples": ["x86_64-unknown-windows"] + }, + ] + } + } +} +``` + +In this hypothetical case, the `macos` variant supports both `x86_64` and `arm64`. + +## Security considerations + +The same checksum facility that binary targets already use will ensure that any downloaded `.zip` file will have the intended contents, exactly as for XCFrameworks. There is only a small conceptual difference between running a command during the build vs linking it into the built debug binary for a package using an XCFramework. Either way, the remote code will be run on the local machine, which has inherent security implications if the source is untrusted. + +## Impact on existing packages + +Artifact bundles would only be available for packages that specify the SwiftPM tools version in which this proposal is implemented. There will be no impact on existing packages. + +## Future directions + +### Binary compatibility for linux + +This initial proposal leaves unanswered some questions about binary compatibility, especially with regards to Linux. For Apple platforms, binary distribution is fairly straightforward, owing to of an ABI-compatible set of SDKs and a strict versioning scheme. + +For Linux, this is much more of a problem. Linux is not a single platform, and it is difficult to the variants in a way that will allow binary compatibility without having to provide an excessive number of specialized binaries. + +A future direction would be to adopt the concepts from [manylinux](https://www.python.org/dev/peps/pep-0513) and provide tooling to let package authors build Linux binaries in a way that makes them usable across many Linux installations. This would, among other things, require tools to statically link against any non-ABI-stable dependencies. + +### Support for executable scripts + +The proposed support for executables is mainly focused on compiled executables, where it makes sense to use a target triple to represent architectures and ABI requirements. For executables implemented as shell scripts (or other kinds of scripts), it might make sense to extend the notion of variant selectors to allow the cross-platform nature of scripts to be better represented. For example, it might be reasonable to allow a `*` for the architecture if there are no architecture contstraints. + +### Libraries on non-Darwin platforms + +Generalizing binary targets to support arbitrary artifacts moves SwiftPM closer to supporting binary libraries other than XCFrameworks. In order to make this usable, however, a future proposal would need to define exactly how to provide libraries on Windows and Linux, and this would encounter even more ABI compatibility issues than executables, since the workaround of statically linking troublesome dependencies would not be available to libraries that themselves need to be linked into the client. + +### Arbitrary binary artifacts + +This proposal focuses on executables, since that is the immediate need in order to support [SE-0303](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md). However, a future direction would be to allow distribution of libraries of resources such as 3D models, textures, fonts, or other large assets that a package may want to make available but not include in the package repository itself. The proposed `.artifactbundle` format is flexible enough to handle this, but there would need to be an API for plugins to access those artifacts, and possibly to vend them directly to client packages if no separate processing is necessary (for example in the form of resource bundles, which is a concept that SwiftPM already has). + +## Alternatives considered + +One alternative would be to not extend binary targets and to instead require any executables that are needed by plugins to be installed on the host before building the package. However, one of the goals of packages are to be self-describing, and for SwiftPM to be able to fetch any dependencies as needed. This should include binary executables. + +## References + +* https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md +* https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md +* https://www.python.org/dev/peps/pep-0513 + diff --git a/proposals/0306-actors.md b/proposals/0306-actors.md new file mode 100644 index 0000000000..4a1d6d0ee0 --- /dev/null +++ b/proposals/0306-actors.md @@ -0,0 +1,948 @@ +# Actors + +* Proposal: [SE-0306](0306-actors.md) +* Authors: [John McCall](https://github.com/rjmccall), [Doug Gregor](https://github.com/DougGregor), [Konrad Malawski](https://github.com/ktoso), [Chris Lattner](https://github.com/lattner) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.5)** +* Implementation: Partially available in [recent `main` snapshots](https://swift.org/download/#snapshots) behind the flag `-Xfrontend -enable-experimental-concurrency` +* Review: ([first review](https://forums.swift.org/t/se-0306-actors/45734)), ([second review](https://forums.swift.org/t/se-0306-second-review-actors/47291)), ([acceptance](https://forums.swift.org/t/accepted-with-modification-se-0306-actors/47662)) + +## Table of Contents + +* [Introduction](#introduction) +* [Proposed solution](#proposed-solution) + * [Actors](#actors-1) + * [Actor isolation](#actor-isolation) + * [Cross-actor references and Sendable types](#cross-actor-references-and-sendable-types) + * [Closures](#closures) + * [Actor reentrancy](#actor-reentrancy) + * ["Interleaving" execution with reentrant actors](#interleaving-execution-with-reentrant-actors) + * [Deadlocks with non-reentrant actors](#deadlocks-with-non-reentrant-actors) + * [Unnecessary blocking with non-reentrant actors](#unnecessary-blocking-with-non-reentrant-actors) + * [Existing practice](#existing-practice) + * [Reentrancy Summary](#reentrancy-summary) + * [Protocol conformances](#protocol-conformances) +* [Detailed design](#detailed-design) + * [Actors](#actors-2) + * [Actor isolation checking](#actor-isolation-checking) + * [References and actor isolation](#references-and-actor-isolation) + * [Protocol conformance](#protocol-conformance) + * [Partial applications](#partial-applications) + * [Key paths](#key-paths) + * [inout parameters](#inout-parameters) + * [Actor interoperability with Objective-C](#actor-interoperability-with-objective-c) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) +* [Future Directions](#future-directions) + * [Non-reentrancy](#non-reentrancy) + * [Task-chain reentrancy](#task-chain-reentrancy) +* [Alternatives considered](#alternatives-considered) + * [Actor inheritance](#actor-inheritance) + * [Cross-actor lets](#cross-actor-lets) +* [Revision history](#revision-history) + +## Introduction + +The Swift concurrency model intends to provide a safe programming model that statically detects [data races](https://en.wikipedia.org/wiki/Race_condition#Data_race) and other common concurrency bugs. The [Structured Concurrency][sc] proposal introduces a way to define concurrent tasks and provides data-race safety for functions and closures. This model is suitable for a number of common design patterns, including things like parallel maps and concurrent callback patterns, but is limited to working with state that is captured by closures. + +Swift includes classes, which provide a mechanism for declaring mutable state that is shared across the program. Classes, however, are notoriously difficult to correctly use within concurrent programs, requiring error-prone manual synchronization to avoid data races. We want to provide the ability to use shared mutable state while still providing static detection of data races and other common concurrency bugs. + +The [actor model](https://en.wikipedia.org/wiki/Actor_model) defines entities called *actors* that are perfect for this task. Actors allow you as a programmer to declare that a bag of state is held within a concurrency domain and then define multiple operations that act upon it. Each actor protects its own data through *data isolation*, ensuring that only a single thread will access that data at a given time, even when many clients are concurrently making requests of the actor. As part of the Swift Concurrency Model, actors provide the same race and memory safety properties as structured concurrency, but provide the familiar abstraction and reuse features that other explicitly declared types in Swift enjoy. + +Swift-evolution threads: + [Pitch #1](https://forums.swift.org/t/concurrency-actors-actor-isolation/41613), + [Pitch #2](https://forums.swift.org/t/pitch-2-actors/44094), + [Pitch #3](https://forums.swift.org/t/pitch-3-actors/44470), + [Pitch #4](https://forums.swift.org/t/pitch-4-actors/45215), + [Pitch #5](https://forums.swift.org/t/pitch-4-actors/45215/36), + [Pitch #6](https://forums.swift.org/t/pitch-6-actors/45519), + [Review #1](https://forums.swift.org/t/se-0306-actors/45734) + +## Proposed solution + +### Actors + +This proposal introduces *actors* into Swift. An actor is a reference type that protects access to its mutable state, and is introduced with the keyword `actor`: + +```swift +actor BankAccount { + let accountNumber: Int + var balance: Double + + init(accountNumber: Int, initialDeposit: Double) { + self.accountNumber = accountNumber + self.balance = initialDeposit + } +} +``` + +Like other Swift types, actors can have initializers, methods, properties, and subscripts. They can be extended and conform to protocols, be generic, and be used with generics. + +The primary difference is that actors protect their state from data races. This is enforced statically by the Swift compiler through a set of limitations on the way in which actors and their instance members can be used, collectively called *actor isolation*. + +### Actor isolation + +Actor isolation is how actors protect their mutable state. For actors, the primary mechanism for this protection is by only allowing their stored instance properties to be accessed directly on `self`. For example, here is a method that attempts to transfer money from one account to another: + +```swift +extension BankAccount { + enum BankError: Error { + case insufficientFunds + } + + func transfer(amount: Double, to other: BankAccount) throws { + if amount > balance { + throw BankError.insufficientFunds + } + + print("Transferring \(amount) from \(accountNumber) to \(other.accountNumber)") + + balance = balance - amount + other.balance = other.balance + amount // error: actor-isolated property 'balance' can only be referenced on 'self' + } +} +``` + +If `BankAccount` were a class, the `transfer(amount:to:)` method would be well-formed, but would be subject to data races in concurrent code without an external locking mechanism. + +With actors, the attempt to reference `other.balance` triggers a compiler error, because `balance` may only be referenced on `self`. The error message notes that `balance` is *actor-isolated*, meaning that it can only be accessed directly from within the specific actor it is tied to or "isolated by". In this case, it's the instance of `BankAccount` referenced by `self`. All declarations on an instance of an actor, including stored and computed instance properties (like `balance`), instance methods (like `transfer(amount:to:)`), and instance subscripts, are all actor-isolated by default. Actor-isolated declarations can freely refer to other actor-isolated declarations on the same actor instance (on `self`). Any declaration that is not actor-isolated is *non-isolated* and cannot synchronously access any actor-isolated declaration. + +A reference to an actor-isolated declaration from outside that actor is called a *cross-actor reference*. Such references are permissible in one of two ways. First, a cross-actor reference to immutable state is allowed from anywhere in the same module as the actor is defined because, once initialized, that state can never be modified (either from inside the actor or outside it), so there are no data races by definition. The reference to `other.accountNumber` is allowed based on this rule, because `accountNumber` is declared via a `let` and has value-semantic type `Int`. + +The second form of permissible cross-actor reference is one that is performed with an asynchronous function invocation. Such asynchronous function invocations are turned into "messages" requesting that the actor execute the corresponding task when it can safely do so. These messages are stored in the actor's "mailbox", and the caller initiating the asynchronous function invocation may be suspended until the actor is able to process the corresponding message in its mailbox. An actor processes the messages in its mailbox one-at-a-time, so that a given actor will never have two concurrently-executing tasks running actor-isolated code. This ensures that there are no data races on actor-isolated mutable state, because there is no concurrency in any code that can access actor-isolated state. For example, if we wanted to make a deposit to a given bank account `account`, we could make a call to a method `deposit(amount:)` on another actor, and that call would become a message placed in the actor's mailbox and the caller would suspend. When that actor processes messages, it will eventually process the message corresponding to the deposit, executing that call within the actor's isolation domain when no other code is executing in that actor's isolation domain. + +> **Implementation note**: At an implementation level, the messages are partial tasks (described by the [Structured Concurrency][sc] proposal) for the asynchronous call, and each actor instance contains its own serial executor (also in the [Structured Concurrency][sc] proposal). The default serial executor is responsible for running the partial tasks one-at-a-time. This is conceptually similar to a serial [`DispatchQueue`](https://developer.apple.com/documentation/dispatch/dispatchqueue), but with an important difference: tasks awaiting an actor are **not** guaranteed to be run in the same order they originally awaited that actor. Swift's runtime system aims to avoid priority inversions whenever possible, using techniques like priority escalation. Thus, the runtime system considers a task's priority when selecting the next task to run on the actor from its queue. This is in contrast with a serial DispatchQueue, which are strictly first-in-first-out. In addition, Swift's actor runtime uses a lighter-weight queue implementation than Dispatch to take full advantage of Swift's `async` functions. + +Compile-time actor-isolation checking determines which references to actor-isolated declarations are cross-actor references, and ensures that such references use one of the two permissible mechanisms described above. This ensures that code outside of the actor does not interfere with the actor's mutable state. + +Based on the above, we can implement a correct version of `transfer(amount:to:)` that is asynchronous: + +```swift +extension BankAccount { + func transfer(amount: Double, to other: BankAccount) async throws { + assert(amount > 0) + + if amount > balance { + throw BankError.insufficientFunds + } + + print("Transferring \(amount) from \(accountNumber) to \(other.accountNumber)") + + // Safe: this operation is the only one that has access to the actor's isolated + // state right now, and there have not been any suspension points between + // the place where we checked for sufficient funds and here. + balance = balance - amount + + // Safe: the deposit operation is placed in the `other` actor's mailbox; when + // that actor retrieves the operation from its mailbox to execute it, the + // other account's balance will get updated. + await other.deposit(amount: amount) + } +} +``` + +The `deposit(amount:)` operation needs involve the state of a different actor, so it must be invoked asynchronously. This method could itself be implemented as `async`: + +```swift +extension BankAccount { + func deposit(amount: Double) async { + assert(amount >= 0) + balance = balance + amount + } +} +``` + +However, this method doesn't really need to be `async`: it makes no asynchronous calls (note the lack of `await`). Therefore, it would be better defined as a synchronous function: + +```swift +extension BankAccount { + func deposit(amount: Double) { + assert(amount >= 0) + balance = balance + amount + } +} +``` + +Synchronous actor functions can be called synchronously on the actor's `self`, but cross-actor references to this method require an asynchronous call. The `transfer(amount:to:)` function calls it asynchronously (on `other`), while the following function `passGo` calls it synchronously (on the implicit `self`): + +```swift +extension BankAccount { + // Pass go and collect $200 + func passGo() { + self.deposit(amount: 200.0) // synchronous is okay because `self` is isolated + } +} +``` + +Cross-actor references to an actor property are permitted as an asynchronous call so long as they are read-only accesses: + +```swift +func checkBalance(account: BankAccount) async { + print(await account.balance) // okay + await account.balance = 1000.0 // error: cross-actor property mutations are not permitted +} +``` + +> **Rationale**: it is possible to support cross-actor property sets. However, cross-actor `inout` operations cannot be reasonably supported because there would be an implicit suspension point between the "get" and the "set" that could introduce what would effectively be race conditions. Moreover, setting properties asynchronously may make it easier to break invariants unintentionally if, e.g., two properties need to be updated at once to maintain an invariant. + +From outside a module, immutable `let`s must be referenced asynchronously from outside the actor. For example: + +```swift +// From another module +func printAccount(account: BankAccount) { + print("Account #\(await account.accountNumber)") +} +``` + +This preserves the ability for the module that defines `BankAccount` to evolve the `let` into a `var` without breaking clients, which is a property Swift has always maintained.: + +```swift +actor BankAccount { // version 2 + var accountNumber: Int + var balance: Double +} +``` + +Only code within the module will need to change to account for `accountNumber` becoming a `var`; existing clients will already use asynchronous access and be unaffected. + +### Cross-actor references and `Sendable` types + +[SE-0302][se302] introduces the `Sendable` protocol. Values of types that conform to the `Sendable` protocol are safe to share across concurrently-executing code. There are various kinds of types that work well this way: value-semantic types like `Int` and `String`, value-semantic collections of such types like `[String]` or `[Int: String]`, immutable classes, classes that perform their own synchronization internally (like a concurrent hash table), and so on. + +Actors protect their mutable state, so actor instances can be freely shared across concurrently-executing code, and the actor itself will internally maintain synchronization. Therefore, every actor type implicitly conforms to the `Sendable` protocol. + +All cross-actor references are, necessarily, working with values of types that are being shared across different concurrently-executed code. For example, let's say that our `BankAccount` includes a list of owners, where each owner is modeled by a `Person` class: + +```swift +class Person { + var name: String + let birthDate: Date +} + +actor BankAccount { + // ... + var owners: [Person] + + func primaryOwner() -> Person? { return owners.first } +} +``` + +The `primaryOwner` function can be called asynchronously from another actor, and then the `Person` instance can be modified from anywhere: + +```swift +if let primary = await account.primaryOwner() { + primary.name = "The Honorable " + primary.name // problem: concurrent mutation of actor-isolated state +} +``` + +Even non-mutating access is problematic, because the person's `name` could be modified from within the actor at the same time as the original call is trying to access it. To prevent this potential for concurrent mutation of actor-isolated state, all cross-actor references can only involve types that conform to `Sendable`. For a cross-actor asynchronous call, the argument and result types must conform to `Sendable`. For a cross-actor reference to an immutable property, the property type must conform to `Sendable`. By insisting that all cross-actor references only use `Sendable` types, we can ensure that no references to shared mutable state flow into or out of the actor's isolation domain. The compiler will produce a diagnostic for such issues. For example, the call to `account.primaryOwner()` about would produce an error like the following: + +``` +error: cannot call function returning non-Sendable type 'Person?' across actors +``` + +Note that the `primaryOwner()` function as defined above can still be used with actor-isolated code. For example, we can define a function to get the name of the primary owner, like this: + +```swift +extension BankAccount { + func primaryOwnerName() -> String? { + return primaryOwner()?.name + } +} +``` + +The `primaryOwnerName()` function is safe to asynchronously call across actors because `String` (and therefore `String?`) conforms to `Sendable`. + +### Closures + +The restrictions on cross-actor references only work so long as we can ensure that the code that might execute concurrently with actor-isolated code is considered to be non-isolated. For example, consider a function that schedules report generation at the end of the month: + +```swift +extension BankAccount { + func endOfMonth(month: Int, year: Int) { + // Schedule a task to prepare an end-of-month report. + Task.detached { + let transactions = await self.transactions(month: month, year: year) + let report = Report(accountNumber: self.accountNumber, transactions: transactions) + await report.email(to: self.accountOwnerEmailAddress) + } + } +} +``` + +A task created with `Task.detached` runs concurrently with all other code. If the closure passed to `Task.detached` were to be actor-isolated, we would introduce a data race on access to the mutable state on `BankAccount`. Actors prevent this data race by specifying that a `@Sendable` closure (described in [`Sendable` and `@Sendable` closures][se302], and used in the definition of `Task.detached` in the [Structured Concurrency][sc] proposal) is always non-isolated. Therefore, it is required to asynchronously access any actor-isolated declarations. + +A closure that is not `@Sendable` cannot escape the concurrency domain in which it was formed. Therefore, such a closure will be actor-isolated if it is formed within an actor-isolated context. This is useful, for example, when applying sequence algorithms like `forEach` where the provided closure will be called serially: + +```swift +extension BankAccount { + func close(distributingTo accounts: [BankAccount]) async { + let transferAmount = balance / accounts.count + + accounts.forEach { account in // okay, closure is actor-isolated to `self` + balance = balance - transferAmount + await account.deposit(amount: transferAmount) + } + + await thief.deposit(amount: balance) + } +} +``` + +A closure formed within an actor-isolated context is actor-isolated if it is non-`@Sendable`, and non-isolated if it is `@Sendable`. For the examples above: + +* The closure passed to `Task.detached` is non-isolated because that function requires a `@Sendable` function to be passed to it. +* The closure passed to `forEach` is actor-isolated to `self` because it takes a non-`@Sendable` function. + +### Actor reentrancy + +Actor-isolated functions are [reentrant](https://en.wikipedia.org/wiki/Reentrancy_(computing)). When an actor-isolated function suspends, reentrancy allows other work to execute on the actor before the original actor-isolated function resumes, which we refer to as *interleaving*. Reentrancy eliminates a source of deadlocks, where two actors depend on each other, can improve overall performance by not unnecessarily blocking work on actors, and offers opportunities for better scheduling of (e.g.) higher-priority tasks. However, it means that actor-isolated state can change across an `await` when an interleaved task mutates that state, meaning that developers must be sure not to break invariants across an await. In general, this is the [reason for requiring `await`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md#suspension-points) on asynchronous calls, because various state (e.g., global state) can change when a call suspends. + +This section explores the issue of reentrancy with examples that illustrate both the benefits and problems with both reentrant and non-reentrant actors, and settles on re-entrant actors. Alternatives Considered provides potential future directions to provide more control of re-entrancy, including [non-reentrant actors](#non-reentrancy) and [task-chain reentrancy](#task-chain-reentrancy). + +#### "Interleaving" execution with reentrant actors + +Reentrancy means that execution of asynchronous actor-isolated functions may "interleave" at suspension points, leading to increased complexity in programming with such actors, as every suspension point must be carefully inspected if the code *after* it depends on some invariants that could have changed before it suspended. + +Interleaving executions still respect the actor's "single-threaded illusion", i.e., no two functions will ever execute *concurrently* on any given actor. However they may *interleave* at suspension points. In broad terms this means that reentrant actors are *thread-safe* but are not automatically protecting from the "high level" kinds of races that may still occur, potentially invalidating invariants upon which an executing asynchronous function may be relying on. To further clarify the implications of this, let us consider the following actor, which thinks of an idea and then returns it, after telling its friend about it. + +```swift +actor DecisionMaker { + let friend: Friend + + // actor-isolated opinion + var opinion: Decision = .noIdea + + func thinkOfGoodIdea() async -> Decision { + opinion = .goodIdea // <1> + await friend.tell(opinion, heldBy: self) // <2> + return opinion // 🤨 // <3> + } + + func thinkOfBadIdea() async -> Decision { + opinion = .badIdea // <4> + await friend.tell(opinion, heldBy: self) // <5> + return opinion // 🤨 // <6> + } +} +``` + +In the example above the `DecisionMaker` can think of a good or bad idea, shares that opinion with a friend, and returns that opinion that it stored. Since the actor is reentrant this code is wrong and will return an arbitrary opinion if the actor begins to think of a few ideas at the same time. + +This is exemplified by the following piece of code, exercising the `decisionMaker` actor: + +```swift +let goodThink = Task.detached { await decisionMaker.thinkOfGoodIdea() } // runs async +let badThink = Task.detached { await decisionMaker.thinkOfBadIdea() } // runs async + +let shouldBeGood = await goodThink.get() +let shouldBeBad = await badThink.get() + +await shouldBeGood // could be .goodIdea or .badIdea ☠️ +await shouldBeBad +``` + +This snippet _may_ result (depending on timing of the resumptions) in the following execution: + +```swift +opinion = .goodIdea // <1> +// suspend: await friend.tell(...) // <2> +opinion = .badIdea // | <4> (!) +// suspend: await friend.tell(...) // | <5> +// resume: await friend.tell(...) // <2> +return opinion // <3> +// resume: await friend.tell(...) // <5> +return opinion // <6> +``` + +But it _may_ also result in the "naively expected" execution, i.e. without interleaving, meaning that the issue will only show up intermittently, like many race conditions in concurrent code. + +The potential for interleaved execution at suspension points is the primary reason for the requirement that every suspension point be [marked by `await`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md#suspension-points) in the source code, even though `await` itself has no semantic effect. It is an indicator that any shared state might change across the `await`, so one should avoid breaking invariants across an `await`, or otherwise depending on the state "before" to be identical to the state "after". + +Generally speaking, the easiest way to avoid breaking invariants across an `await` is to encapsulate state updates in synchronous actor functions. Effectively, synchronous code in an actor provides a [critical section](https://en.wikipedia.org/wiki/Critical_section), whereas an `await` interrupts a critical section. For our example above, we could effect this change by separating "opinion formation" from "telling a friend your opinion". Indeed, telling your friend your opinion might reasonably cause you to change your opinion! + +#### Deadlocks with non-reentrant actors + +The opposite of reentrant actor functions are "non-reentrant" functions and actors. This means that while an actor is processing an incoming actor function call (message), it will *not* process any other message from its mailbox until it has completed running this initial function. Essentially, the entire actor is blocked from executing until that task completes. + +If we take the example from the previous section and use a non-reentrant actor, it will execute correctly, because no work can be scheduled on the actor until `friend.tell` has completed: + +```swift +// assume non-reentrant +actor DecisionMaker { + let friend: DecisionMaker + var opinion: Decision = .noIdea + + func thinkOfGoodIdea() async -> Decision { + opinion = .goodIdea + await friend.tell(opinion, heldBy: self) + return opinion // ✅ always .goodIdea + } + + func thinkOfBadIdea() async -> Decision { + opinion = .badIdea + await friend.tell(opinion, heldBy: self) + return opinion // ✅ always .badIdea + } +} +``` + +However, non-entrancy can result in deadlock if a task involves calling back into the actor. For example, let's stretch this example further and have our friend try to convince us to change a bad idea: + +```swift +extension DecisionMaker { + func tell(_ opinion: Decision, heldBy friend: DecisionMaker) async { + if opinion == .badIdea { + await friend.convinceOtherwise(opinion) + } + } +} +``` + +With non-reentrant actors, `thinkOfGoodIdea()` will succeed under this implementation, because `tell` essentially does nothing. However, `thinkOfBadIdea()` will deadlock because the original decision maker (call it `A`) is locked when it calls `tell` on another decision maker (call it `B`). `B` then tries to convince `A` otherwise, but that call cannot execute because `A` is already locked. Hence, the actor itself deadlocks and cannot progress. + +> The term "deadlock" used in these discussions refer to actors asynchronously waiting on "each other," or on "future work of self". No thread blocking is necessary to manifest this issue. + +In theory, a fully non-reentrant model would also deadlock when calling asynchronous functions on `self`. However, since such calls are statically determinable to be on `self`, they would execute immediately and therefore not block. + +Deadlocks with non-reentrant actors could be detected with runtime tools that detect cyclic call graphs once they've occurred, much like tools exist to find reference cycles in data structures at runtime. However, such deadlocks cannot generally be identified statically (e.g., with the compiler or static analysis), because call graphs require whole-program knowledge and can change dynamically depending on the data provided to the program. + +Deadlocked actors would be sitting around as inactive zombies forever. Some runtimes solve deadlocks like this by making every single actor call have a timeout (such timeouts are already useful for distributed actor systems). This would mean that each `await` could potentially `throw`, and that either timeouts or deadlock detection would have to always be enabled. We feel this would be prohibitively expensive, because we envision actors being used in the vast majority of concurrent Swift applications. It would also muddy the waters with respect to cancellation, which is intentionally designed to be explicit and cooperative. Therefore, we feel that the approach of automatically cancelling on deadlocks does not fit well with the direction of Swift Concurrency. + +#### Unnecessary blocking with non-reentrant actors + +Consider an actor that handles the download of various images and maintains a cache of what it has downloaded to make subsequent accesses faster: + +```swift +// assume non-reentrant +actor ImageDownloader { + var cache: [URL: Image] = [:] + + func getImage(_ url: URL) async -> Image { + if let cachedImage = cache[url] { + return cachedImage + } + + let data = await download(url) + let image = await Image(decoding: data) + return cache[url, default: image] + } +} +``` + +This actor is functionally correct, whether it is re-entrant or not. However, if it is non-reentrant, it will completely serialize the download of images: once a single client asked for an image, all other clients are blocked from starting any requests--even ones that would hit the cache or which ask for images at different URLs---until that first client has had its image fully downloaded and decoded. + +With a reentrant actor, multiple clients can fetch images independently, so that (say) they can all be at different stages of downloading and decoding an image. The serialized execution of partial tasks on the actor ensures that the cache itself can never get corrupted. At worst, two clients might ask for the same image URL at the same time, in which there will be some redundant work. + +#### Existing practice + +There are a number of existing actor implementations that have considered the notion of reentrancy: + +* Erlang/Elixir ([gen_server](https://medium.com/@eduardbme/erlang-gen-server-never-call-your-public-interface-functions-internally-c17c8f28a1ee)) showcases a simple "loop/deadlock" scenario and how to detect and fix it, +* Akka ([Persistence persist/persistAsync](https://doc.akka.io/docs/akka/current/persistence.html#relaxed-local-consistency-requirements-and-high-throughput-use-cases) is effectively _non-reentrant behavior by default_, and specific APIs are designed to allow programmers to _opt into_ reentrant whenever it would be needed. In the linked documentation `persistAsync` is the re-entrant version of the API, and it is used _very rarely_ in practice. Akka persistence and this API has been used to implement bank transactions and process managers, by relying on the non-reentrancy of `persist()` as a killer feature, making implementations simple to understand and _safe_. Note that Akka is built on top of Scala, which does not provide `async`/`await`. This means that mailbox-processing methods are more synchronous in nature, and rather than block the actor while waiting for a response, they would handle the response as a separate message receipt. +* Orleans ([grains](https://dotnet.github.io/orleans/docs/grains/reentrancy.html)) are also non-reentrant by default, but offer extensive configuration around reentrancy. Grains and specific methods can be marked as being re-entrant, and there is even a dynamic mechanism by which one can implement a run-time predicate to determine whether an invocation can interleave. Orleans is perhaps closest to the Swift approach described here, because it is built on top of a language that provides `async`/`await` (C#). Note that Orleans *had* a feature called [call-chain reentrancy](https://dotnet.github.io/orleans/docs/grains/reentrancy.html#reentrancy-within-a-call-chain), which we feel is a promising potential direction: we cover it later in this proposal in our section on [task-chain reentrancy](#task-chain-reentrancy). + +#### Reentrancy Summary + +This proposal provides only reentrant actors. However, the [Future Directions](#future-directions) section describes potential future design directions that could add opt-in non-reentrancy. + +> **Rationale**: Reentrancy by default all but eliminates the potential for deadlocks. Moreover, it helps ensure that actors can make timely progress within a concurrent system, and that a particular actor does not end up unnecessarily blocked on a long-running asynchronous operation (say, downloading a file). The mechanisms for ensuring safe interleaving, such as using synchronous code when performing mutations and being careful not to break invariants across `await` calls, are already present in the proposal. + +### Protocol conformances + +All actor types implicitly conform to a new protocol, `Actor`: + +```swift +protocol Actor : AnyObject, Sendable { } +``` + +> **Note**: The definition of the `Actor` protocol is intentionally left blank. The [custom executors proposal][customexecs] will introduce requirements into the `Actor` protocol. These requirements will be implicitly synthesized by the implementation when not explicitly provided, but can be explicitly provided to allow actors to control their own serialized execution. + +The `Actor` protocol can be used to write generic operations that work across all actors, including extending all actor types with new operations. As with actor types, instance properties, functions, and subscripts defined on the `Actor` protocol (including extensions thereof) are actor-isolated to the `self` actor. For example, + +```swift +protocol DataProcessible: Actor { // only actor types can conform to this protocol + var data: Data { get } // actor-isolated to self +} + +extension DataProcessible { + func compressData() -> Data { // actor-isolated to self + // use data synchronously + } +} + +actor MyProcessor : DataProcessible { + var data: Data // okay, actor-isolated to self + + func doSomething() { + let newData = compressData() // okay, calling actor-isolated method on self + // use new data + } +} + +func doProcessing(processor: T) async { + await processor.compressData() // not actor-isolated, so we must interact asynchronously with the actor +} +``` + +No other kind of concrete type (class, enum, struct, etc.) can conform to the `Actor` protocol, because they cannot define actor-isolated operations. + +Actors can also conform to protocols with `async` requirements, because all clients will already have to interact with those requirements asynchronously, giving the actor the ability to protect its isolated state. For example: + +```swift +protocol Server { + func send(message: Message) async throws -> Message.Reply +} + +actor MyActor: Server { + func send(message: Message) async throws -> Message.Reply { // okay: this method is actor-isolated to 'self', satisfies asynchronous requirement + } +} +``` + +Actors cannot otherwise be made to conform to non-`Actor` protocols with synchronous requirements. However, there is a separate proposal on [controlling actor isolation][isolationcontrol] that allows such conformances when they can implemented in a manner that does not reference any mutable actor state. + +## Detailed design + +### Actors + +An actor type can be declared with the `actor` keyword: + +```swift +/// Declares a new type BankAccount +actor BankAccount { + // ... +} +``` + +Each instance of the actor represents a unique actor. The term "actor" can be used to refer to either an instance or the type; where necessary, one can refer to the "actor instance" or "actor type" to disambiguate. + +Actors are similar to other concrete nominal types in Swift (enums, structs, and classes). Actor types can have `static` and instance methods, properties, and subscripts. They have stored properties and initializers like structs and classes. They are reference types like classes, but do not support inheritance, and therefore do not have (or need) features such as `required` and `convenience` initializers, overriding, or `class` members, `open` and `final`. Where actor types differ in behavior from other types is primarily driven by the rules of actor isolation, described below. + +By default, the instance methods, properties, and subscripts of an actor have an isolated `self` parameter. This is true even for methods added retroactively on an actor via an extension, like any other Swift type. Static methods, properties, and subscripts do not have a `self` parameter that is an instance of the actor, so they are not actor-isolated. + +```swift +extension BankAccount { + func acceptTransfer(amount: Double) async { // actor-isolated + balance += amount + } +} +``` + +### Actor isolation checking + +Any given declaration in a program is either actor-isolated or is non-isolated. A function (including accessors) is actor-isolated if it is defined on an actor type (including protocols where `Self` conforms to `Actor`, and extensions thereof). A mutable instance property or instance subscript is actor-isolated if it is defined on an actor type. Declarations that are not actor-isolated are called non-isolated. + +The actor isolation rules are checked in a number of places, where two different declarations need to be compared to determine if their usage together maintains actor isolation. There are several such places: +* When the definition of one declaration (e.g., the body of a function) references another declaration, e.g., calling a function, accessing a property, or evaluating a subscript. +* When one declaration satisfies a protocol requirement. + +We'll describe each scenario in detail. + +#### References and actor isolation + +An actor-isolated non-`async` declaration can only be synchronously accessed from another declaration that is isolated to the same actor. For synchronous access to an actor-isolated function, the function must be called from another actor-isolated function. For synchronous access to an actor-isolated instance property or instance subscript, the instance itself must be actor-isolated. + +An actor-isolated declaration can be asynchronously accessed from any declaration, whether it is isolated to another actor or is non-isolated. Such accesses are asynchronous operations, and therefore must be annotated with `await`. Semantically, the program will switch to the actor to perform the synchronous operation, and then switch back to the caller's executor afterward. + +For example: + +```swift +actor MyActor { + let name: String + var counter: Int = 0 + func f() +} + +extension MyActor { + func g(other: MyActor) async { + print(name) // okay, name is non-isolated + print(other.name) // okay, name is non-isolated + print(counter) // okay, g() is isolated to MyActor + print(other.counter) // error: g() is isolated to "self", not "other" + f() // okay, g() is isolated to MyActor + await other.f() // okay, other is not isolated to "self" but asynchronous access is permitted + } +} +``` + +#### Protocol conformance + +When a given declaration (the "witness") satisfies a protocol requirement (the "requirement"), the protocol requirement can be satisfied by the witness if: + +* The requirement is `async`, or +* the requirement and witness are both actor-isolated. + +An actor can satisfy an asynchronous requirement because any uses of the requirement are asynchronous, and can therefore suspend until the actor is available to execute them. Note that an actor can satisfy an asynchronous requirement with a synchronous one, in which case the normal notion of asynchronously accessing a synchronous declaration on an actor applies. For example: + +```swift +protocol Server { + func send(message: Message) async throws -> Message.Reply +} + +actor MyServer : Server { + func send(message: Message) throws -> Message.Reply { ... } // okay, asynchronously accessed from clients of the protocol +} +``` + +### Partial applications + +Partial applications of isolated functions are only permitted when the expression is a direct argument whose corresponding parameter is non-Sendable. For example: + +```swift +func runLater(_ operation: @Sendable @escaping () -> T) -> T { ... } + +actor A { + func f(_: Int) -> Double { ... } + func g() -> Double { ... } + + func useAF(array: [Int]) { + array.map(self.f) // okay + Task.detached(operation: self.g) // error: self.g has non-sendable type () -> Double that cannot be converted to a @Sendable function type + runLater(self.g) // error: cannot convert value of non-sendable function type () -> Double to sendable function type + } +} +``` + +These restrictions follow from the actor isolation rules for the "desugaring" of partial applications to closures. The two erroneous cases above fall out from the fact that the closure would be non-isolated in a closure that performs the call, so access to the actor-isolated function `g` would have to be asynchronous. Here are the "desugared" forms of the partial applications: + + +```swift +extension A { + func useAFDesugared(a: A, array: [Int]) { + array.map { f($0) } ) // okay + Task.detached { g() } // error: self is non-isolated, so call to `g` cannot be synchronous + runLater { g() } // error: self is non-isolated, so the call to `g` cannot be synchronous + } +} +``` + +### Key paths + +A key path cannot involve a reference to an actor-isolated declaration: + +```swift +actor A { + var storage: Int +} + +let kp = \A.storage // error: key path would permit access to actor-isolated storage +``` + +> **Rationale**: Allowing the formation of a key path that references an actor-isolated property or subscript would permit accesses to the actor's protected state from outside of the actor isolation domain. As an alternative to this rule, we could remove the `Sendable` conformance from key paths, such that one could form key paths to actor-isolated state but they could not be shared. + +### inout parameters + +Actor-isolated stored properties can be passed into synchronous functions via `inout` parameters, but it is ill-formed to pass them to asynchronous functions via `inout` parameters. For example: + +```swift +func modifiesSynchronously(_: inout Double) { } +func modifiesAsynchronously(_: inout Double) async { } + +extension BankAccount { + func wildcardBalance() async { + modifiesSynchronously(&balance) // okay + await modifiesAsynchronously(&balance) // error: actor-isolated property 'balance' cannot be passed 'inout' to an asynchronous function + } +} + +class C { var state : Double } +struct Pair { var a, b : Double } +actor A { + let someC : C + var somePair : Pair + + func inoutModifications() async { + modifiesSynchronously(&someC.state) // okay + await modifiesAsynchronously(&someC.state) // not okay + modifiesSynchronously(&somePair.a) // okay + await modifiesAsynchronously(&somePair.a) // not okay + } +} +``` + +> **Rationale**: this restriction prevents exclusivity violations where the modification of the actor-isolated `balance` is initiated by passing it as `inout` to a call that is then suspended, and another task executed on the same actor then attempts to access `balance`. Such an access would then result in an exclusivity violation that will terminate the program. While the `inout` restriction is not required for memory safety (because errors will be detected at runtime), the default re-entrancy of actors makes it very easy to introduce non-deterministic exclusivity violations. Therefore, we introduce this restriction to eliminate that class of problems that where a race would trigger an exclusivity violation. + +### Actor interoperability with Objective-C + +An actor type can be declared `@objc`, which implicitly provides conformance to `NSObjectProtocol`: + +```swift +@objc actor MyActor { ... } +``` + +A member of an actor can only be `@objc` if it is either `async` or is not isolated to the actor. Synchronous code that is within the actor's isolation domain can only be invoked on `self` (in Swift). Objective-C does not have knowledge of actor isolation, so these members are not permitted to be exposed to Objective-C. For example: + +```swift +@objc actor MyActor { + @objc func synchronous() { } // error: part of actor's isolation domain + @objc func asynchronous() async { } // okay: asynchronous, exposed to Objective-C as a method that accepts a completion handler + @objc nonisolated func notIsolated() { } // okay: non-isolated +} +``` + +## Source compatibility + +This proposal is mostly additive, and should not break source compatibility. The `actor` contextual keyword to introduce actors is a parser change that does not break existing code, and the other changes are carefully staged so they do not change existing code. Only new code that introduces actors or actor-isolation attributes will be affected. + +## Effect on ABI stability + +This is purely additive to the ABI. Actor isolation itself is a static notion that is not part of the ABI. + +## Effect on API resilience + +Nearly all changes in actor isolation are breaking changes, because the actor isolation rules require consistency between a declaration and its users: + +* A class cannot be turned into an actor or vice versa. +* The actor isolation of a public declaration cannot be changed. + +## Future Directions + +### Non-reentrancy + +We could introduce a `@reentrant` attribute may be added to any actor-isolated function, actor, or extension of an actor to describe how it is reentrant. The attribute would have several forms: + +* `@reentrant`: Indicates that each potential suspension point within the function bodies covered by the attribute is reentrant. +* `@reentrant(never)`: Indicates that each potential suspension point within the function bodies covered by the attribute is non-reentrant. + +A non-reentrant potential suspension point prevents any other asynchronous call from executing on the actor until it has completed. Note that asynchronous calls to non-reentrant async functions directly on `self` are exempted from this check, so an actor can asynchronously call itself without producing a deadlock. + +> **Rationale**: Allowing direct calls on `self` eliminates an obvious set of deadlocks, and requires only the same static knowledge as actor-isolation checking for synchronous access to actor-isolated state. + +It is an error to have a `@reentrant` attribute on a non-isolated function, non-actor type, or extension of a non-actor type. Only one `@reentrant` attribute may occur on a given declaration. The reentrancy of an actor-isolated non-type declaration is determined by finding a suitable `@reentrant` attribute. The search is as follows: + +1. The declaration itself. +2. If the declaration is a non-type member of an extension, the extension. +3. If the declaration is a non-type member of a type (or extension thereof), the type definition. + +If there is no suitable `@reentrant` attribute, an actor-isolated declaration is reentrant. + +Here's an example illustrating how the `@reentrant` attribute can be applied at various points: + +```swift +actor Stage { + @reentrant(never) func f() async { ... } // not reentrant + func g() async { ... } // reentrant +} + +@reentrant(never) +extension Stage { + func h() async { ... } // not reentrant + @reentrant func i() async { ... } // reentrant + + actor InnerChild { // reentrant, not affected by enclosing extension + func j() async { ... } // reentrant + } + + nonisolated func k() async { .. } // okay, reentrancy is uninteresting + nonisolated @reentrant func l() async { .. } // error: @reentrant on non-actor-isolated +} + +@reentrant func m() async { ... } // error: @reentrant on non-actor-isolated +``` + +The attribute approach is not the only possible design here. At an implementation level, the actual blocking will be handled at each asynchronous call site. Instead of an attribute that affects potentially many asynchronous calls, we could introduce a different form of `await` that does the blocking, e.g., + +```swift +await(blocking) friend.tell(opinion, heldBy: self) +``` + +### Task-chain reentrancy + +The discussion of reentrant and non-reentrant actors treats reentrancy as a binary choice, where all forms of reentrancy are considered to be equally likely to introduce hard-to-reason-about data races. However, a frequent and usually quite understandable way of interacting between actors which are simply "conversations" between two or more actors in order fo fulfill some initial request. In synchronous code, it's common to have two or more different classes call back into each other with synchronous calls. For example, here is a silly implementation of `isEven` that uses mutual recursion between two classes: + +```swift +class OddOddySync { + let evan: EvenEvanSync! + + func isOdd(_ n: Int) -> Bool { + if n == 0 { return true } + return evan.isEven(n - 1) + } +} + +class EvenEvanSync { + let oddy: OddOddySync! + + func isEven(_ n: Int) -> Bool { + if n == 0 { return false } + return oddy.isOdd(n - 1) + } +} +``` + +This code is depending on the two methods of these classes to effectively be "reentrant" within the same call stack, because one will call into the other (and vice-versa) as part of the computation. Now, take this example and make it asynchronous using actors: + +```swift +@reentrant(never) +actor OddOddy { + let evan: EvenEvan! + + func isOdd(_ n: Int) async -> Bool { + if n == 0 { return true } + return await evan.isEven(n - 1) + } +} + +@reentrant(never) +actor EvenEvan { + let oddy: OddOddy! + + func isEven(_ n: Int) async -> Bool { + if n == 0 { return false } + return await oddy.isOdd(n - 1) + } +} +``` + +Under `@reentrant(never)`, this code will deadlock, because a call from `EvanEvan.isEven` to `OddOddy.isOdd` will then depend on another call to `EvanEvan.isEven`, which cannot proceed until the original call completes. One would need to make these methods to be reentrant to eliminate the deadlock. + +With Swift embracing [Structured Concurrency][sc] as a core building block of its concurrency story, we may be able to do better than outright banning reentrancy. In Swift, every asynchronous operation is part of a `Task` which encapsulates the general computation taking place, and every asynchronous operation spawned from such task becomes a child task of the current task. Therefore, it is possible to know whether a given asynchronous call is part of the same task hierarchy, which is the rough equivalent to being in the same call stack in synchronous code. + +We could introduce a new kind of reentrancy, *task-chain reentrancy*, which allows reentrant calls on behalf of the given task or any of its children. This resolves both the deadlock we encountered in the `convinceOtherwise` example from the section on [deadlocks](#deadlocks-with-non-reentrant-actors) as well as the mutually-recursive `isEven` example above, while still preventing reentrancy from unrelated tasks. This reentrancy therefore mimics synchronous code more closely, eliminating many deadlocks without allowing unrelated interleavings to break the high-level invariants of an actor. + +There are a few reasons why we are not currently comfortale including task-chain reentrancy in the proposal: +* The task-based reentrancy approach doesn't seem to have been tried at scale. Orleans documents support for [reentrancy in a call chain](https://dotnet.github.io/orleans/docs/grains/reentrancy.html#reentrancy-within-a-call-chain), but the implementation was fairly limited and it was eventually [removed](https://twitter.com/reubenbond/status/1349725703634251779). From the Orleans experience, it is hard to assess whether the problem is with the idea or the specific implementation. +* We do not yet know of an efficient implementation technique for this approach within the actor runtime. + +If we can address the above, task-chain reentrancy can be introduced into the actor model with another spelling of the reentrancy attribute such as `@reentrant(task)`, and may provide the best default. + +## Alternatives considered + +### Actor inheritance + +Earlier pitches and the first reviewed version of this proposal allowed actor inheritance. Actor inheritance followed the rules of class inheritance, albeit with specific additional rules required to maintain actor isolation: + +* An actor could not inherit from a class, and vice-versa. +* An overriding declaration must not be more isolated than the overridden declaration. + +Subsequent review discussion determined that the conceptual cost of actor inheritance outweighed its usefulness, so it has been removed from this proposal. The form that actor inheritance would take in the language is well-understand from prior iterations of this proposal and its implementation, so this feature could be re-introduced at a later time. + +### Cross-actor lets + +This proposal allows synchronous access to `let` properties on an actor instance from anywhere within the same module as the actor is defined: + +```swift +// in module BankActors +public actor BankAccount { + public let accountNumber: Int +} + +func print(account: BankAccount) { + print(account.accountNumber) // okay: synchronous access to an actor's let property +} +``` + +Outside of the module, access must be asynchronous: + +```swift +import BankActors + +func otherPrint(account: BankAccount) async { + print(account.accountNumber) // error: cannot synchronously access immutable 'let' outside the actor's module + print(await account.accountNumber) // okay to asynchronously access +} +``` + +The requirement for asynchronous access from outside of the module preserves a longstanding freedom for library implementors, which allows a public `let` to be refactored into a `var` without breaking any clients. It is consistent with Swift's policy of maximizing the freedom for library implementors to alter the implementation without breaking clients. Without requiring asynchronous access from other modules, the `otherPrint(account:)` function above were permitted to reference `accountNumber` synchronously. If the author of `BankActors` then changed the account number into a `var`, it would break existing client code: + +```swift +public actor BankAccount { + public var accountNumber: Int // version 2 makes this mutable, but would break clients if synchronous access to 'let's were allowed outside the module +} +``` + +There are a number of other language features that take this same approach of reducing boilerplate and simplifying the language within a module, then requiring the use of additional language features when an entity is used from outside the module. For example: + +* Access control defaults to `internal`, so you can use a declaration across your whole module but have to explicitly opt in to making it available outside your module (e.g., via `public`). In other words, you can ignore access control until the point where you need to make something `public` for use from another module. +* The implicit memberwise initializer of a struct is `internal`. You need to write a `public` initializer yourself to commit to allowing that struct to be initialized with exactly that set of parameters. +* Inheritance from a class is permitted by default when the superclass is in the same module. To inherit from a superclass defined in a different module, that superclass must be explicitly marked `open`. You can ignore `open` until you want to guarantee this ability to clients outside of the module. +* Overriding a declaration in a class is permitted by default when the overridden declaration is in the same module. To override from a declaration in a different module, that overrides declaration must be explicitly marked `open`. + +SE-0313 "[Improved control over actor isolation][isolationcontrol]" provides an explicit way to give clients the freedom to synchronously access immutable actor state via the `nonisolated` keyword, e.g., + +```swift +// in module BankActors +public actor BankAccount { + public nonisolated let accountNumber: Int // can be accessed synchronously from any module due to the explicit 'nonisolated' +} +``` + +The original accepted version of this proposal required *all* access to immutable actor storage be asynchronous, and left any synchronous access to explicit `nonisolated` annotations as spelled out in [SE-0313][isolationcontrol]. However, experience with that model showed that it had a number of problems that affected teachability of the model: + +* Developers were almost immediately confronted with the need to use `nonisolated` when writing actor code. This goes against the principle of [progressive disclosure](https://www.interaction-design.org/literature/topics/progressive-disclosure) that Swift tries to follow for advanced features. Aside from `nonisolated let`, uses of `nonisolated` are fairly rare. +* Immutable state is a key tool for writing safe concurrency code. A `let` of `Sendable` type is conceptually safe to reference from concurrency code, and works in other contexts (e.g., local variables). Making some immutable state concurrency-safe while other state is not complicates the story about data-race-safe concurrent programming. Here's an example of the existing restrictions around `@Sendable`, which were defined in [SE-0302][se302]: + + ```swift + func test() { + let total = 100 + var counter = 0 + + Task.detached { + print(total) // okay to reference immutable state + print(counter) // error, cannot reference a `var` from a @Sendable closure + } + + counter += 1 + } + ``` + +By allowing synchronous access to actor `let`s within a module, we provide a smoother learning curve for actor isolation and embrace (rather than subvert) the longstanding and pervasive idea that immutable data is safe for concurrency, while still addressing the concerns from the second review that unrestricted synchronous access to actor `let`s is implicitly committing a library author to never make that state mutable. It follows existing precedent in the Swift language of making in-module interactions simpler than interactions across modules. + +## Revision history + +* Changes in the post-review amendment to the proposal: + * Cross-after references to instance `let` properties from a different module must be asynchronous; within the same module they will be synchronous. +* Changes in the final accepted version of the proposal: + * Cross-actor references to instance `let` properties must be asynchronous. +* Changes in the second reviewed proposal: + * Escaping closures can now be actor-isolated; only `@Sendable` prevents isolation. + * Removed actor inheritance. It can be considered at some future point. + * Added "cross-actor lets" to Alternatives Considered. While there is no change to the proposed direction, the issue is explained here for further discussion. + * Replaced `detach` with `Task.detached` to match updates to the [Structured Concurrency proposal][sc]. +* Changes in the seventh pitch: + * Removed isolated parameters and `nonisolated` from this proposal. They'll come in a follow-up proposal on [controlling actor isolation][isolationcontrol]. +* Changes in the sixth pitch: + * Make the instance requirements of `Actor` protocols actor-isolated to `self`, and allow actor types to conform to such protocols using actor-isolated witnesses. + * Reflow the "Proposed Solution" section to get the bigger ideas out earlier. + * Remove `nonisolated(unsafe)`. +* Changes in the fifth pitch: + * Drop the prohibition on having multiple `isolated` parameters. We don't need to ban it. + * Add the `Actor` protocol back, as an empty protocol whose details will be filled in with a subsequent proposal for [custom executors][customexecs]. + * Replace `ConcurrentValue` with `Sendable` and `@concurrent` with `@Sendable` to track the evolution of [SE-0302][se302]. + * Clarify the presentation of actor isolation checking. + * Add more examples for non-isolated declarations. + * Added a section on isolated or "sync" actor types. +* Changes in the fourth pitch: + * Allow cross-actor references to actor properties, so long as they are reads (not writes or `inout` references) + * Added `isolated` parameters, to generalize the previously-special behavior of `self` in an actor and make the semantics of `nonisolated` more clear. + * Limit `nonisolated(unsafe)` to stored instance properties. The prior definition was far too broad. + * Clarify that `super` is isolated if `self` is. + * Prohibit references to actor-isolated declarations in key paths. + * Clarify the behavior of partial applications. + * Added a "future directions" section describing isolated protocol conformances. +* Changes in the third pitch: + * Narrow the proposal down to only support re-entrant actors. Capture several potential non-reentrant designs in the Alternatives Considered as possible future extensions. + * Replaced `@actorIndependent` attribute with a `nonisolated` modifier, which follows the approach of `nonmutating` and ties in better with the "actor isolation" terminology (thank you to Xiaodi Wu for the suggestion). + * Replaced "queue" terminology with the more traditional "mailbox" terminology, to try to help alleviate confusion with Dispatch queues. + * Introduced "cross-actor reference" terminology and the requirement that cross-actor references always traffic in `Sendable` types. + * Reference `@concurrent` function types from their separate proposal. + * Moved Objective-C interoperability into its own section. + * Clarify the "class-like" behaviors of actor types, such as satisfying an `AnyObject` conformance. +* Changes in the second pitch: + * Added a discussion of the tradeoffs with actor reentrancy, performance, and deadlocks, with various examples, and the addition of new attribute `@reentrant(never)` to disable reentrancy at the actor or function level. + * Removed global actors; they will be part of a separate document. + * Separated out the discussion of data races for reference types. + * Allow asynchronous calls to synchronous actor methods from outside the actor. + * Removed the `Actor` protocol; we'll tackle customizing actors and executors in a separate proposal. + * Clarify the role and behavior of actor-independence. + * Add a section to "Alternatives Considered" that discusses actor inheritance. + * Replace "actor class" with "actor". +* Original pitch [document](https://github.com/DougGregor/swift-evolution/blob/6fd3903ed348b44496b32a39b40f6b6a538c83ce/proposals/nnnn-actors.md) + + +[sc]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md +[se302]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md +[customexecs]: https://github.com/rjmccall/swift-evolution/blob/custom-executors/proposals/0000-custom-executors.md +[isolationcontrol]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0313-actor-isolation-control.md diff --git a/proposals/0307-allow-interchangeable-use-of-double-cgfloat-types.md b/proposals/0307-allow-interchangeable-use-of-double-cgfloat-types.md new file mode 100644 index 0000000000..1d58e9c5cb --- /dev/null +++ b/proposals/0307-allow-interchangeable-use-of-double-cgfloat-types.md @@ -0,0 +1,146 @@ +# Allow interchangeable use of `CGFloat` and `Double` types + +* Proposal: [SE-0307](0307-allow-interchangeable-use-of-double-cgfloat-types.md) +* Author: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.5)** +* Decision Notes: [Rationale](https://forums.swift.org/t/se-0307-allow-interchangeable-use-of-cgfloat-and-double-types/45756/55) +* Implementation: [apple/swift#34401](https://github.com/apple/swift/pull/34401) + +## Introduction + +I propose to extend the language and allow Double and CGFloat types to be used interchangeably by means of transparently converting one type into the other as a sort of retroactive typealias between these two types. This is a _narrowly_ defined implicit conversion intended to be part of the _existing family_ of implicit conversions (including NSType <=> CFType conversions) supported by Swift to strengthen Objective-C and Swift interoperability. The only difference between the proposed conversion and existing ones is related to the fact that interchangeability implies both narrowing conversion (`Double` -> `CGFloat`) and widening one (`CGFloat` -> `Double`) on 32-bit platforms. This proposal is not about generalizing support for implicit conversions to the language. + +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/pitch-allow-interchangeable-use-of-cgfloat-and-double-types/45324) + + +## Motivation + +When Swift was first released, the type of `CGFloat` presented a challenge. At the time, most iOS devices were still 32-bit. SDKs such as CoreGraphics provided APIs that took 32-bit floating point values on 32-bit platforms, and 64-bit values on 64-bit platforms. When these APIs were first introduced, 32-bit scalar arithmetic was faster on 32-bit platforms, but by the time of Swift’s release, this was no longer true: then, as today, 64-bit scalar arithmetic was just as fast as 32-bit even on 32-bit platforms (albeit with higher memory overhead). But the 32/64-bit split remained, mostly for source and ABI stability reasons. + +In (Objective-)C, this had little developer impact due to implicit conversions, but Swift did not have implicit conversions. A number of options to resolve this were considered: + +1. Keep `CGFloat` as a separate type, with explicit conversion from both `Float` and `Double` always needed on all platforms. +2. Do that, but introduce specific implicit conversions from `Double` to `CGFloat`. +3. Make `CGFloat` a typealias for either `Float` or `Double` based on the platform. +4. Consolidate on `Double` for all APIs using `CGFloat`, even on 32-bit platforms, with the Swift importer converting 32-bit `CGFloat` APIs to use `Double`, with the importer adding the conversion. + +One important goal was avoiding the need for users to build for both 32- and 64-bit platforms in order to know their code will work on both, so option 3 was ruled out. Option 4 was not chosen mainly because of concern over handling of pointers to `CGFloat` (including arrays). Option 2 was ruled out due to challenges in the type-checker. So option 1 was chosen. + +With several years’ hindsight and technical improvements, we can reevaluate these choices. 64-bit devices are now the norm, and even on 32-bit `Double` is now often the better choice for calculations. The concern around arrays or pointers to `CGFloat` turns out to be a minor concern as there are not many APIs that take them. And in the recent top-of-tree Swift compiler, the performance of the type-checker has significantly improved. + +As the world has moved on, the choice of creating a separate type has become a significant pain point for Swift users. As the language matured and new frameworks, such as SwiftUI, have been introduced, `CGFloat` has resulted in a significant impedance mismatch, without bringing any real benefit. Many newly introduced APIs have standardized on using `Double` in their arguments. Because of this discrepancy, it’s not always possible to choose a “correct” type while declaring variables, so projects end up with a mix-and-match of `Double` and `CGFloat` declarations, with constant type conversions (or refactoring of variable types) needed. This constant juggling of types is particularly frustrating given in practice the types are transparently identical to the compiler when building for 64-bit platforms. And when building for 32-bit platforms, the need to appease the type-checker with the API at hand is the overriding driver of conversions, rather than considerations of precision versus memory use that would in theory be the deciding factor. + +## Proposed Solution + +In order to address all of the aforementioned problems, I propose to extend the language and allow `Double` and `CGFloat` types to be used interchangeably by means of transparently converting one type into the other as a sort of retroactive `typealias` between these two types. This is option 2 in the list above. + +The compiler already implements similar conversions to improve interoperability ergonomics for types from CoreFoundation and Foundation, such as `CFString` <-> `NSString`. The difference between existing interchangeability conversions and the one proposed by this document lays in fact that new conversion implies both implicit widening (`CGFloat` to `Double`) and narrowing (`Double` to `CGFloat`) on 32-bit platforms, which results in loss of precision in the latter case. Precision loss could be deemed acceptable because users of the affected APIs incur it already by means of explicit conversions when passing `Double` values to APIs that expect `CGFloat` arguments (see `Detailed Design` section for more information regarding precision loss mitigation), and since `CGFloat` is a type that is defined by Apple frameworks, it makes this change specific to Apple platforms and a targeted conversion to address a particular ergonomics issue with `Double` and `CGFloat` types. + +Let’s consider an example where such a conversion might be useful in the real world: + +```swift +import UIKit + +struct Progress { + let startTime: Date + let duration: TimeInterval + + func drawPath(in rect: CGRect) -> UIBezierPath { + let elapsedTime = Date().timeIntervalSince(startTime) + let progress = min(elapsedTime / duration, 1.0) + + let path = CGMutablePath() + + let center = CGPoint(x: rect.midX, y: rect.midY) + path.move(to: center) + path.addLine(to: CGPoint(x: center.x, y: 0)) + + let adjustment = .pi / 2.0 + path.addRelativeArc(center: center, radius: rect.width / 2.0, + startAngle: CGFloat(0.0 - adjustment), + delta: CGFloat(2.0 * .pi * progress)) + path.closeSubpath() + + return UIBezierPath(cgPath: path) + } +} +``` + +Here, the `Progress` struct draws a progress circle given a start time and a duration. In Foundation, seconds are expressed using `[TimeInterval](https://developer.apple.com/documentation/foundation/timeinterval)`, which is a typealias for `Double`. However, the `[CGMutablePath](https://developer.apple.com/documentation/coregraphics/cgmutablepath)` APIs for drawing shapes require `CGFloat` arguments, forcing developers to explicitly convert between `Double` and `CGFloat` when working with these two frameworks together. Furthermore, because float literals default to `Double` in Swift, developers are forced to either explicitly annotate or convert simple constants when working with graphics APIs, such as `adjustment` in the above example. With an implicit conversion between `Double` and `CGFloat`, the call to `addRelativeArc` can be simplified to: + +```swift +path.addRelativeArc(center: center, radius: rect.width / 2.0, + startAngle: 0.0 - adjustment, + delta: 2.0 * .pi * progress) +``` + +## Detailed Design + +The type-checker will detect all of the suitable locations where `Double` is converted to `CGFloat` and vice versa and allow such conversion by inserting an implicit initializer call to the appropriate constructor - `(_: CGFloat) -> Double` or `(_: Double) -> CGFloat` depending on conversion direction. + +This new conversion has the following properties: + +* `Double` is always preferred over `CGFloat` where possible, in order to limit possibility of ambiguities, i.e. an overload that accepts a `Double` argument would be preferred over one that accepts a `CGFloat` if both require a conversion to type-check; +* `Double` <-> `CGFloat` conversion is introduced only if it has been determined by the type-checker that it would be impossible to type-check an expression without one; +* Any number of widening conversions (`CGFloat` -> `Double`) is preferred over a single narrowing one (`Double` -> `CGFloat`), and if narrowing is still contextually necessary, it would be attempted as late as possible to mitigate precision loss. + * Let’s consider following example: `let _: CGFloat = x / y` (where `x` is `CGFloat` and `y` is `Double`). Type-checked expression would be `let _: CGFloat = CGFloat.init(x / Double(y)` and not `let _: CGFloat = CGFloat(x) / y` to mitigate potential precision loss. +* Disallowed conversions: + * Arguments of explicit calls to the `CGFloat` initializer; + * Collections: arrays, sets, or dictionaries containing `CGFloat` or `Double` keys/values have to be explicitly converted by the user. Otherwise, implicit conversion could hide memory/cpu cost associated with per-element transformations; + * Explicit (conditional and checked) casts (`try`, `as` etc.) and runtime checks (`is`); + * Any position where such a conversion has already been introduced; to prevent converting back and forth between `Double` and `CGFloat` by means of other types. + + +Note that with the introduction of this new conversion, homogeneous overloads can be called with heterogeneous arguments, because it is possible to form a call that accepts both `Double` and `CGFloat` types or a combination thereof. This is especially common with operators. + +Let’s consider following example: + +```swift +func sum(_: Double, _: Double) -> Double { ... } +func sum(_: CGFloat, _: CGFloat) -> CGFloat { ... } + +var x: Double = 0 +var y: CGFloat = 42 + +_ = sum(x, y) +``` + +Although both overloads of `sum` are homogeneous, and under current rules the call is not going to be accepted, with introduction of `Double` <-> `CGFloat` conversion it’s possible to form two valid calls depending on what conversion direction is picked. Since it has been established that going `CGFloat` -> `Double` is always preferred, `sum(x, y)` is going to be type-checked as `(Double, Double) -> Double` (because there is no contextual type specified) and arguments are going to be `x` and `Double(y)`. + +The contextual type doesn’t affect the preferred conversion direction since the type-checker would always choose a solution with the fewest number of narrowing conversions possible, i.e.: + +```swift +let _: CGFloat = sum(x, y) +``` + +`sum(Double, Double) -> Double` is preferred in this case because it requires a single narrowing conversion at the last possible moment, so the type-checked expression would be `let _: CGFloat = CGFloat.init(sum(x, `Double(`y)))`. Such solution is numerically better than any other solution with fewer conversions where narrowing happens in the arguments i.e. `let _: CGFloat = sum(CGFloat(x), y)` because it incurs a more significant precision loss. + +## Source compatibility + +This is an additive change and does not have any material effect on source compatibility. This change has been tested on a very large body of code, and all of the expressions that previously type-checked with explicit conversions continued to do so. + +## Effect on ABI stability + +This change introduces new conversions at compile-time only, and so would not impact ABI. + +## Effect on API resilience + +This is not an API-level change and would not impact resilience. + +## Alternatives Considered + +Not to make this change and leave the painful ergonomics of the `CGFloat` type intact. + +Restrict implicit conversions to “API boundaries” only. Such a rule would either imply the same behavior as proposed if operators/functions accept `CGFloat` arguments or introduce arbitrary restrictions like allowing conversions only across module boundaries or resilient ABIs, which would lead to confusing and inconsistent behavior where some calls would fail to type-check without an explicit conversion for no apparent reason (e.g. after an application has been refactored and slit into multiple modules or vice versa). + +Another possible alternative would be to add new matching `Double` overloads to all APIs that currently accept `CGFloat` type. Such new APIs cannot be backward deployed unless they’re emitted directly into client binaries. In addition to bloating code size, this would also severely impact type checker performance much more than a targeted implicit conversion. + +A more general solution to type conversions may have provided a path to resolve this problem, but only partly. One could propose a language feature that permits implicit widening conversions – for example, from `Int8` to `Int`, or `Float` to `Double`, but not vice versa. Such a general language feature could allow user-defined (in this case CoreGraphics-defined) implicit conversion from, say, `Double` to `CGFloat` on 64-bit platforms, but probably not two-way conversion (`CGFloat` to `Double`) nor would it allow a narrowing conversion from `Double` to 32-bit `CGFloat`. This would result in users writing code for 64-bit platforms, but then finding their code does not compile for 32-bit platforms, and avoiding this remains a core goal for `CGFloat`. So instead, we propose a custom solution for `CGFloat`. Custom solutions like this are undesirable in general, but in this case the benefits to users and the ecosystem in general vastly weigh in its favor. That said, this general “implicit widening” has a lot of appeal, and the work done here with `CGFloat` does prove at least that these kind of implicit conversions can be handled by the Swift type-checker as it exists today. + +The choice of not converting arrays was deliberate, but does not match the current behavior with subtype conversions such as `[T] as [Any]`. Since graphical code involving buffers of `CGFloat` may be more performance, explicit conversion is preferred. This need for explicit conversion will likely lead the user to a more efficient solution of building an explicitly typed array instead in the first place, avoiding a linear conversion. The memory implications on 32-bit platforms are also likely to me most material when dealing with large buffers of values, again suggesting explicit conversion is preferable. There are relatively few APIs that traffic in buffers of `CGFloat` so this is not expected to be a concern. + + +## Acknowledgments + +[Holly Borla](https://forums.swift.org/u/hborla) and [Ben Cohen](https://forums.swift.org/u/Ben_Cohen) for all the help with writing and editing this proposal. Also [Xiaodi Wu](https://forums.swift.org/u/xwu) and other participants of the pitch discussion on Swift Forums. diff --git a/proposals/0308-postfix-if-config-expressions.md b/proposals/0308-postfix-if-config-expressions.md new file mode 100644 index 0000000000..c086807efb --- /dev/null +++ b/proposals/0308-postfix-if-config-expressions.md @@ -0,0 +1,310 @@ +# `#if` for postfix member expressions + +* Proposal: [SE-0308](0308-postfix-if-config-expressions.md) +* Author: [Rintaro Ishizaki](https://github.com/rintaro) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.5)** +* Implementation: [apple/swift#35097](https://github.com/apple/swift/pull/35097) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0308-postfix-if-config-expressions/47780) + +## Introduction + +Swift has conditional compilation block `#if ... #endif` which allows code to be conditionally compiled depending on the value of one or more compilation conditions. Currently, unlike `#if` in C family languages, the body of each clause must surround complete statements. However, in some cases, especially in result builder contexts, demand for applying `#if` to partial expressions has emerged. This proposal expands `#if ... #endif` to be able to surround postfix member expressions. + +## Motivation + +For example, when you have some SwiftUI code like this: + +```swift +VStack { + Text("something") +#if os(iOS) + .iOSSpecificModifier() +#endif + .commonModifier() +} +``` + +This doesn’t parse today, so you end up having to do something like: + +```swift +VStack { + let basicView = Text("something") +#if os(iOS) + basicView + .iOSSpecificModifier() + .commonModifier() +#else + basicView + .commonModifier() +#endif +} +``` + +which is ugly and has duplicated `.commonModifier()`. If you want to eliminate the duplication: + +```swift +VStack { + let basicView = Text("something") +#if os(iOS) + let tmpView = basicView.iOSSpecificModifier() +#else + let tmpView = basicView +#endif + tmpView.commonModifier() +} +``` + +...which is even uglier. + +## Proposed solution + +This proposal expands `#if` functionality to postfix member expressions. For example, in the following example: + +```swift +baseExpr +#if CONDITION + .someOptionalMember? + .someMethod() +#else + .otherMember +#endif +``` + +If `CONDITION` evaluates to `true`, the expression is parsed as + +```swift +baseExpr + .someOptionalMember? + .someMethod() +``` + +Otherwise, it’s parsed as + +``` +baseExpr + .otherMember +``` + +## Detailed design + +### Grammar changes + +This proposal adds `postfix-ifconfig-expression` to `postfix-expression`. `postfix-ifconfig-expression` +is a postfix-expression followed by a `#if ... #endif` clause. + +```diff ++ postfix-expression → postfix-ifconfig-expression ++ postfix-ifconfig-expression → postfix-expression conditional-compilation-block +``` + + `postfix-ifconfig-expression` is parsed only if the body of the `#if` clause starts with a period (`.`) followed by a identifier, a keyword or an integer-literal. For example: + +```swift +// OK +baseExpr +#if CONDITION_1 + .someMethod() +#else + .otherMethod() +#endif +``` + + But the following is not a `postfix-ifconfig-expression` because it does not start with `.`. In such cases, `#if ... #endif` is not considered a part of the expression, but is parsed as a normal compiler control statement. + +```swift +// ERROR +baseExpr // warning: expression of type 'BaseExpr' is unused. +#if CONDITION + { $0 + 1 } // error: closure expression is unused +#endif + +baseExpr // warning: expression of type 'BaseExpr' is unused. +#if CONDITION + + otherExpr // error: unary operator cannot be separated from its operand +#endif +``` + +Also, the body must not contain any other characters after the expression. + +```swift +// ERROR +baseExpr +#if CONDITION_1 + .someMethod() + +print("debug") // error: unexpected tokens in '#if' expression body +#endif +``` + +### Expression kind inside `#if`/`#elseif`/`#else` body + +There are several kinds of postfix expressions in Swift grammar. + +* initializer expression +* postfix self expression +* explicit member expression +* function call expression +* subscript expression +* forced value expression +* optional chaining expression +* postfix operator expression + +The body of a postfix `#if` expression must start with an explicit member expression, initializer expression, or postfix self expression (that is, the suffixes that begin with `.`). Once started this way, you can continue the expression with any other postfix expression suffixes. For example: + +```swift +// OK +baseExpr +#if CONDITION_1 + .someMember?.otherMethod()![idx]++ +#else + .otherMethod(arg) { + //... + } +#endif +``` + +However, you cannot continue the expression within the `#if` with non-postfix suffixes. For example, you cannot continue it with a binary operator, because a binary expression is not a postfix expression: + +```swift +// ERROR +baseExpr +#if CONDITION_1 + .someMethod() + 12 // error: unexpected tokens in '#if' expression body +#endif +``` + +Starting with other postfix expression suffixes besides those beginning with `.` is not allowed because this would be ambiguous with starting a new statement. These suffixes are generally required to start on the same line as the base expression. + +### `#elseif`/`#else` body + +While the body of the `#if` clause must begin with `.`, the body of any `#elseif` or `#else` clauses can be empty. + +```swift +// OK +baseExpr +#if CONDITION_1 + .someMethod() +#elseif CONDITION_2 + // OK. Do nothing. +#endif +``` + +If the clause is not empty, then it has the same requirements as the `#if` clause: it must begin with a postfix expression suffix starting with `.`, it may not continue into a non-postfix expression, and it must not contain an unrelated statement. + +```swift +// ERROR +baseExpr +#if CONDITION_1 + .someMethod() +#else +return 1 // error: unexpected tokens in '#if' expression body +#endif +``` + +### Consecutive postfix `#if` expressions + +`#if ... #endif` blocks for postfix expression can be followed by an additional postfix expression including another `#if ... #endif`: + +```swift +// OK +baseExpr +#if CONDITION_1 + .someMethod() +#endif +#if CONDITION_2 + .otherMethod() +#endif + .finalizeMethod() +``` + +### Nested `#if` blocks + +Nested `#if` blocks are supported as long as the first body starts with an explicit member-like expression. Each inner `#if` must follow the rule for `postfix-ifconfig-expression` too. + +```swift +// OK +baseExpr +#if CONDITION_1 + #if CONDITION_2 + .someMethod() + #endif + #if CONDITION_3 + .otherMethod() + #endif +#else + .someMethod() + #if CONDITION_4 + .otherMethod() + #endif +#endif +``` + +### Postfix `#if` expression inside another expression + +Postfix `#if` expressions can be nested inside another expression or statement. + +```swift +// OK +someFunc( + baseExpr + .someMethod() +#if CONDITION_1 + .otherMethod() +#endif +) +``` + +This is parsed as `someFunc(baseExpr.someMethod().otherMethod())` or `someFunc(baseExpr.someMethod())` depending on the condition. + +## Source compatibility + +This proposal does not have any source breaking changes. + +```swift +baseExpr +#if CONDITION_1 + .someMethod() +#endif +``` + +This is currently parsed as + +```swift +baseExpr +#if CONDITION_1.someMethod() +#endif +``` + +And it is error because `CONDITION_1.someMethod()` is not a valid compilation condition. This proposal changes the parser behavior so `.someMethod()` is *not* parsed as a part of the condition. As a bonus, this new behavior applies to non-postfix `#if` expressions too. Consequently, + +```swift +enum MyEnum { case foo, bar, baz } + +func test() -> MyEnum { +#if CONDITION_1 + .foo +#elseif CONDITION_2 + .bar +#else + .baz +#endif +} +``` + +Now becomes valid swift code. This change doesn’t break anything because explicit member expressions have always been invalid at the compilation condition position. + +## Effect on ABI stability + +This change is frontend-only and would not impact ABI. + +## Effect on API resilience + +This is not an API-level change and would not impact resilience. + +## Alternatives considered + +### Lexer based `#if` preprocessing + +Like C-family languages, we could pre-process conditional compilation directives purely in Lexer level as discussed in https://forums.swift.org/t/allow-conditional-inclusion-of-elements-in-array-dictionary-literals/16171/29. Although it is certainly a design we should explore some day, in this proposal, we would like to focus on expanding `#if` to postfix expressions. + diff --git a/proposals/0309-unlock-existential-types-for-all-protocols.md b/proposals/0309-unlock-existential-types-for-all-protocols.md new file mode 100644 index 0000000000..460d97f02d --- /dev/null +++ b/proposals/0309-unlock-existential-types-for-all-protocols.md @@ -0,0 +1,321 @@ +# Unlock existentials for all protocols + +* Proposal: [SE-0309](0309-unlock-existential-types-for-all-protocols.md) +* Authors: [Anthony Latsis](https://github.com/AnthonyLatsis), [Filip Sakel](https://github.com/filip-sakel), [Suyash Srijan](https://github.com/theblixguy) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#33767](https://github.com/apple/swift/pull/33767), [apple/swift#39492](https://github.com/apple/swift/pull/39492), [apple/swift#41198](https://github.com/apple/swift/pull/41198) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0309-unlock-existentials-for-all-protocols/47902), [Additional Commentary](https://forums.swift.org/t/se-0309-unlock-existential-types-for-all-protocols/47515/123) + +## Introduction + +Swift allows one to use a protocol as a type when its *requirements* meet a rather unintuitive list of criteria, among which is the absence of associated type requirements, and emits the following error otherwise: `Protocol can only be used as a generic constraint because it has 'Self' or associated type requirements`. Our objective is to *alleviate* this limitation so as to impact only the ability to access certain members (instead of preemptively sealing off the entire protocol interface), and adjust the specified criteria to further reduce the scope of the restriction. + +This proposal is a preamble to a series of changes aimed at generalizing value-level abstraction (existentials) and improving its interaction with type-level abstraction (generics). For an in-depth exploration of the relationships among different built-in abstractions models, we recommend reading the [design document for improving the UI of the generics model](https://forums.swift.org/t/improving-the-ui-of-generics/22814). + +Swift-Evolution Pitch Threads: [Thread #1](https://forums.swift.org/t/lifting-the-self-or-associated-type-constraint-on-existentials/18025), [Thread #2](https://forums.swift.org/t/unlock-existential-types-for-all-protocols/40665) + +## Motivation + +When a protocol is used as a type, that type is also known as an *existential type*. Unlike values of `some` types, which represent a value of some *specific* type that conforms to the given constraints, and cannot be reassigned to a value of a different conforming type, an existential value is akin to a box that can hold any value of any conforming type dynamically at any point in time. Existential types allow values of varying concrete types to be used interchangeably as values of the same existential type, abstracting the difference between the underlying conforming types at the *value level*. For convenience, we will be using the term "existential" to refer to such values and their protocol or protocol composition types throughout the proposal. We also wish to draw a distinction between an associated type *requirement* (the declaration), and an associated type (aka a dependent member type). For example, `Self.Element` and `Self.SubSequence.Element` are distinct associated types that point to the same associated type requirement. + +### Inconsistent Language Semantics + +The compiler permits the use of a protocol as a type *unless* +1) the protocol has an associated type requirement, or +2) the type of a method/property/subscript/initializer requirement contains a reference to `Self` in [non-covariant](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)) position: + +```swift +// 'Identifiable' has an associated type requirement. +public protocol Identifiable { + associatedtype ID: Hashable + + var id: ID { get } +} + +// 'Equatable' has a operator method requirement containing a `Self` reference in contravariant parameter position. +public protocol Equatable { + static func == (lhs: Self, rhs: Self) -> Bool +} +``` +The first condition is a relic of an incomplete implementation of protocol witness tables that didn't allow associated type metadata to be recovered dynamically. Despite having the same restrictive impact when violated, the second condition is actually meant to govern the type safety of individual member accesses. Consider the following protocol interface: + +```swift +protocol P { + func foo() -> Self + + func bar(_: Self) +} +``` + +Accessing a member on an existential value requires the ability to spell the type of that member outside its protocol context. Today, the one means to representing the dynamic `Self` type of an existential value is type erasure — the substitution of `Self` with a representable supertype, like `P`. On the other hand, type erasure is safe to perform only in [covariant](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)) position. For example, calling `foo` on a value of type `P` with its covariant `Self` result type-erased to `P` is safe, whereas allowing one to pass a type-erased value to `bar` would expose the opportunity to pass in an argument of non-matching type. + +In contrast to requirements, protocol extension members cannot afford to retroactively jeopardize the existential availability of the entire protocol. This is when the second condition shows its true colors, with the restriction forced to take on a more reasonable, on-demand manifestation in the form of a member access failure: + +```swift +protocol P {} +extension P { + func method(_: Self) {} +} + +func callMethod(p: P) { + p.method // error: member 'method' cannot be used on value of protocol type 'P'; use a generic constraint instead +} +``` + +As we hope it became clear, a requirement such as `func bar(_: Self)` or an associated type requirement alone cannot speak for the rest of the interface. Some protocols still have a useful subset of functionality that does not rely on `Self` and associated types whatsoever, or does so in a way that is compatible with existential values, like `func foo() -> Self`. In a refined protocol, some requirements may also happen to exclusively rely on an associated type with a fully concrete, known implementation, and become safe to invoke using a value of the refined protocol type. This brings us to a well-known implementation hole with counterintuitive behavior; in the snippet below, `Animal` is still assumed as having an associated type requirement despite the same-type constraint that effectively predefines a fully concrete implementation for it. + +```swift +protocol Animal: Identifiable where ID == String {} +extension Animal { + var name: String { id } +} +``` + +The current semantic inconsistency also discourages authors from refining their existing protocols with other, useful ones in fear of losing existential qualification. + +### Library Evolution + +Removing the type-level restriction would mean that adding defaulted requirements to a protocol is always both a binary- and source-compatible change, since it could no longer interfere with existing uses of the protocol. + +### Type-Erasing Wrappers + +Beyond making incremental progress toward the goal of [generalized existentials](https://github.com/apple/swift/blob/main/docs/GenericsManifesto.md#generalized-existentials), removing this restriction is a necessary — albeit not sufficient — condition for eliminating the need for manual type-erasing wrappers like [`AnySequence`](https://developer.apple.com/documentation/swift/anysequence). These containers are not always straightforward to implement, and can become a pain to maintain in resilient environments, since the wrapper must evolve in parallel to the protocol. In the meantime, wrapping the unconstrained existential type instead of resorting to `Any` or boxing the value in a subclass or closure will enable type-erasing containers to be written in a way that's easier for the compiler to optimize, and ABI-compatible with future generalized existentials. For requirements that cannot be accessed on the existential directly, it will be possible to forward the call through the convolution of writing protocol extension methods to open the value inside and have full access to the protocol interface inside the protocol extension: + +```swift +protocol Foo { + associatedtype Bar + + func foo(_: Bar) -> Bar +} + +private extension Foo { + // Forward to the foo method in an existential-accessible way, asserting that + // the '_Bar' generic argument matches the actual 'Bar' associated type of the + // dynamic value. + func _fooThunk<_Bar>(_ bar: _Bar) -> _Bar { + assert(_Bar.self == Bar.self) + let result = foo(unsafeBitCast(bar, to: Bar.self)) + return unsafeBitCast(result, to: _Bar.self) + } +} + +struct AnyFoo: Foo { + private var _value: Foo + + init(_ value: F) where F.Bar == Bar { + self._value = value + } + + func foo(_ bar: Bar) -> Bar { + return self._value._fooThunk(bar) + } +} +``` + +## Proposed Solution + +We suggest allowing any protocol to be used as a type and exercise the restriction on individual member accesses uniformly across extension members and requirements. Additionally, the adjusted access criteria for protocol members shall account for associated types with known implementations: + +```swift +protocol IntCollection: RangeReplaceableCollection where Self.Element == Int {} +extension Array : IntCollection where Element == Int {} + +var array: any IntCollection = [3, 1, 4, 1, 5] + +array.append(9) // OK, 'Self.Element' is known to be 'Int'. +``` + +Having lowered the limitation, the mere presence of an associated type requirement will no longer preclude member accesses, but references to `Self`-rooted associated types *will* for the same reasons some `Self` references do today. As alluded to back in [Inconsistent Language Semantics](#inconsistent-language-semantics), references to covariant `Self` are already getting automatically replaced with the base object type, permitting usage of `Self`-returning methods on existential values: + +```swift +protocol Copyable { + func copy() -> Self +} + +func test(_ c: Copyable) { + let x = c.copy() // OK, x is of type 'Copyable' +} +``` + +Because they tend to outnumber direct uses of `Self` in protocol contexts, and for the sake of consistency, we believe that extending covariant type erasure to associated types is a reasonable undertaking in light of the primary focus: + +```swift +func test(_ collection: RandomAccessCollection) { + // func dropLast(_ k: Int = 1) -> SubSequence + let x = collection.dropLast() // OK, x is of type 'RandomAccessCollection' +} +``` +___ + +This way, a protocol or protocol extension member (method/property/subscript/initializer) may be used on an existential value *unless*: +* The type of the invoked member (accessor — for storage declarations), as viewed in context of the *base type*, contains references to `Self` or `Self`-rooted associated types in [non-covariant](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)) position. + +> The following types will be considered covariant: +> * Function types in their result type. +> * Tuple types in either of their element types. +> * [Swift.Optional](https://developer.apple.com/documentation/swift/optional) in its `Wrapped` type. +> * [Swift.Array](https://developer.apple.com/documentation/swift/array) in its `Element` type. +> * [Swift.Dictionary](https://developer.apple.com/documentation/swift/dictionary) in its `Value` type. + +## Detailed Design + +Once more, we note that not all requirements will be accessible on existential values. For instance, the `==` operation still cannot be used on two values of the [`Equatable`](https://developer.apple.com/documentation/swift/equatable) type, because it cannot be proved that their dynamic types match without additional dynamic checks: + +```swift +let lhs: Equatable = "Paul" +let rhs: Equatable = "Alex" + +lhs == rhs ❌ + +if let ownerName = lhs as? String, let petName = rhs as? String { + print(ownerName == petName) ✅ // false +} +``` + +### Diagnostics + +Invoking an incompatible member on an existential value will trigger an error comprising a terse description of the issue and a suggestion to use the generic approach (if applicable) in order to gain full access to the protocol interface. For the common case when the existential base is a reference to a function or subscript parameter, the diagnostic will include a fix-it that turns it into a generic parameter (again, if applicable, since generic functions are not allowed in some local contexts). + +```swift +extension Sequence { + public func enumerated() -> EnumeratedSequence { + return EnumeratedSequence(_base: self) + } +} + +func printEnumerated(s: Sequence) { + // error: member 'enumerated' cannot be used on value of type protocol type 'Sequence' + // because it references 'Self' in invariant position; use a conformance constraint + // instead. [fix-it: printEnumerated(s: Sequence) -> printEnumerated(s: S)] + for (index, element) in s.enumerated() { + print("\(index) : \(element)") + } +} + +let collection: RangeReplaceableCollection = [1, 2, 3] +// error: member 'append' cannot be used on value of protocol type 'RangeReplaceableCollection' +// because it references associated type 'Element' in contravariant position; use a conformance +// constraint instead. +collection.append(4) +``` + +In an ideal world, one could imagine the compiler to accompany the error with a note pointing to the specific type reference that is preventing the member from being used. We are inclined toward leaving this out of scope for several reasons: +* Retrieval of the *relevant* source location information both logically and mechanically poses an earnest challenge with the current workings of various compiler components and the potential involvement of generic constraints. +* There is no certainty in whether a concept of these high-precision notes can outplay an educational note in the general case, or is worth indefinitely dragging out a resolution to this particular proposal. + +To showcase just one embodiment of the difficulties involved, consider this relatively simple code: +```swift +struct G {} + +protocol P { + associatedtype A + associatedtype B + + func method() -> B +} + +protocol Q: P where B == G
{} +``` +Notice how the associated type that would preclude a call to `method` on a value of type `Q` is actually `A`, not `B` as the result type may suggest, due to the same-type constraint on the protocol. + +#### Non-conformable Existentials + +A peculiar side effect of lowering the limitation is the expansion of the domain of existential types that cannot be conformed to. Some are such for fundamental reasons, and others could be made conformable with the adoption of appropriate features. One example of the latter is a composition between two unrelated protocols, each constraining the same associated type to different concrete types: +```swift +protocol P1 { + associatedtype A +} +protocol P2: P1 where A == Int {} + +protocol Q1 { + associatedtype A +} +protocol Q2: Q1 where A == Bool {} + +func foo(_: P2 & Q2) { + +} +``` +Any code relying on a non-conformable type is effectively dead and type-safe to keep around. Likewise, Swift may provide a way give formally distinct requirements, like `P1.A` and `Q1.A`, distinct implementations in the future. To spare us from having to deal with a source compatibility dilemma, we propose to at best warn about the ambiguities that arise in these types with messages we already use for generic parameters in similar circumstances. + +### Associated Types with Known Implementations + +By a known implementation, we mean that an associated type is bound to a concrete type under the generic signature of a given existential type. A known implementation has two embodiments: an explicit same-type constraint, i.e. `A == Int`, or an actual implementation for the associated type requirement, found via some other constraint, like a superclass requirement: + +```swift +class Class: P { + typealias A = Int +} + +protocol P { + associatedtype A +} +protocol Q: P { + func takesA(arg: A) +} + +func testComposition(arg: Q & Class) { + arg.takesA(arg: 0) // OK, 'A' is known to be 'Int'. +} +``` +A reference to a `Self`-rooted associated type with a known implementation will not prevent one from accessing a member. + +### Covariant Erasure for Associated Types + +When invoking a member, `Self`-rooted associated types that +* do **not** have a known implementation and +* appear in covariant position within the type of the member + +will be type-erased to their upper bounds as per the generic signature of the existential that is used to access the member. The upper bounds can be either a class, protocol, protocol composition, or `Any`, depending on the *presence* and *kind* of generic constraints on the associated type. As such, references to these associated types are also acceptable for accessing members on existential values. The essence of this behavior was presented a tad [earlier](#proposed-solution) alongside the proposed solution. + +## Source Compatibility & Effect on ABI Stability + +The proposed changes are ABI-additive and source-compatible. + +## Effect on API Resilience + +Adding defaulted requirements to a protocol will become an always-source-compatible change. + +## Alternatives Considered + +The concerns that were raised in regards to our stance can be attributed to one of the following observations: +* The fraction of available API is implicitly determined and requires careful inspection to be reasoned about. +* The current syntax for existentials is dangerously lightweight and strongly implies an API surface that may not match reality. + +Both statements are referring to *existing* problems — due to protocol extensions and the way we spell existential types — that have not been addressed and would become more widespread unless something else is done prior or in addition to implementing the proposed solution. + +### Let the User Decide on Availability + +Since the portion of available API is implicit, it is not apparent anywhere in code what API are being vended or claimed when using an existential, which could lead to developers proliferating the use of type-erasing wrapper dummies to avoid coding themselves into member unavailability time bombs (although, this technique would help only against unsolicited requirements, and not extension members). A solution suggests introducing +* a protocol-specific opt-in declaration modifier as a means of promising that the all *requirements* will always be available for use on the existential, and +* a second opt-in declaration modifier to explicitly mark protocol members that are intended as available on the existential. + +The first modifier is to statically prevent the addition of requirements that are incompatible with the existential, and the second is to forestall accidental unavailability and enhance discoverability. + +In our opinion, the pitfall of unexpected unavailability has to do mostly with inappropriate application of value-level abstraction, and is best addressed by reviewing the language guide and following the somewhat *established* roadmap for [generalized existentials](https://github.com/apple/swift/blob/main/docs/GenericsManifesto.md#generalized-existentials) (which includes syntax renovation and explicit opening of existential values), rather than taking a less principled detour. In swift-evolution discussion, the community pointed out several notable flaws: +* Using these modifiers feels like completely losing sight of generic programming, where no such usability limitations exist. +* The ability to access a member does not so much depend on its declared type as on the one of a multiple of existential types that is used to access it, and the invoked accessor (for storage declarations). +* This approach seems likely to lead to trade-offs between optimal design and compliance with the modifier. +* Being a source-compatible addition, modifiers can merely offer the *option* to be explicit. + +### Syntactic Matters First + +So far, existentials are the only built-in abstraction model (on par with generics and opaque types) that doesn't have its own idiosyncratic syntax; they are spelled as the bare protocol name or a composition thereof. Although the syntax strongly suggets that the protocol as a type and the protocol as a constraint are one thing, in practice, they serve different purposes, and this manifests most confusingly in the *"Protocol (the type) does not conform to Protocol (the constraint)"* paradox. This could be qualified as a missing feature in the language; the bottom line is that the syntax is tempting developers when it should be contributing to weighted decisions. Because using existential types is syntactically lightweight in comparison to using other abstractions, and similar to using a base class, the more possibilities they offer, the more *users are* vulnerable to unintended or inappropriate type erasure by following the path of initial least resistance. + +With regard to a source-compatible adoption of new adornments, we could +* force all newly supported existential types to use the new syntax, i.e. `Any

` or [`any P` as a dual to `some P`](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--clarifying-existentials), and +* allow the respective protocols of all newly supported existential types to be annotated with an attribute to opt into the old syntax (to allow for source-compatible addition of requirements). + +Though, if we *were* going to introduce a new syntax for existentials, we think it'd be much less confusing if we took the potentially source-breaking path and did so uniformly, deprecating the existing syntax after a late-enough language version, than to have yet another attribute and two syntaxes where one only works some of the time. We also believe that drawing a tangible line between protocols that "do" and "do not" have limited access to their API is ill-advised due to the relative nature of this phenomenon. + +## Future Directions + +* Simplify the implementation of Standard Library type-erasing wrappers, such as [`AnyHashable`](https://github.com/apple/swift/blob/main/stdlib/public/core/AnyHashable.swift) and [`AnyCollection`](https://github.com/apple/swift/blob/main/stdlib/public/core/ExistentialCollection.swift), using the practical advice from [earlier](#type-erasing-wrappers). +* Deemphasize existential types. + + It is often that people reach for existential types when they should be employing generic constraints — "should" not merely for performance reasons, but because they truly do not need or intend for any type erasure. Even though the compiler is sometimes able to back us up performance-wise by turning existential code into generic code (as in `func foo(s: Sequence)` vs `func foo(s: S)`), there is an important difference between the two abstractions. Existential types provide value-level abstraction, that is, they eliminate the type-level distinction between different values of the type, and cannot maintain type relationships between independent existential values. Under most circumstances, value-level abstraction only really makes sense in mutable state, in the elements of heterogeneous containers, or, unless our support of [`some` types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md) can turn the tide, in the storage of larger type-erasing constructs. A fitting starting point for giving value-level abstraction more careful consideration early on is the [Language Guide](https://docs.swift.org/swift-book/LanguageGuide/TheBasics.html). +* Make existential types "self-conforming" by automatically opening them when passed as generic arguments to functions. Generic instantiations could have them opened as opaque types. +* Add an `init(_ box: Hashable)` initializer to `AnyHashable` to alleviate confusion and aid in usability. This initializer would be treated as a workaround, and deprecated should automatic opening of existential types become available. +* Introduce [`any P` as a dual to `some P`](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--clarifying-existentials) for explicitly spelling existential types. +* Allow constraining existential types, i.e. `let collection: any Collection = [1, 2, 3]`. diff --git a/proposals/0310-effectful-readonly-properties.md b/proposals/0310-effectful-readonly-properties.md new file mode 100644 index 0000000000..af9dc2b7d8 --- /dev/null +++ b/proposals/0310-effectful-readonly-properties.md @@ -0,0 +1,402 @@ +# Effectful Read-only Properties + +* Proposal: [SE-0310](0310-effectful-readonly-properties.md) +* Author: [Kavon Farvardin](https://github.com/kavon) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.5)** +* Decision Notes: [Pitch](https://forums.swift.org/t/pitch-effectful-read-only-properties/44090), + [Acceptance](https://forums.swift.org/t/accepted-se-0310-effectful-read-only-properties/47739) +* Implementation: [apple/swift#36430](https://github.com/apple/swift/pull/36430), + [apple/swift#36670](https://github.com/apple/swift/pull/36670), + [apple/swift#37225](https://github.com/apple/swift/pull/37225) +* Available in [recent `main` snapshots](https://swift.org/download/#snapshots). + +## Introduction + +Nominal types such as classes, structs, and enums in Swift support [computed properties](https://docs.swift.org/swift-book/LanguageGuide/Properties.html) and [subscripts](https://docs.swift.org/swift-book/LanguageGuide/Subscripts.html), which are members of the type that invoke programmer-specified computations when getting or setting them. The recently accepted proposal [SE-0296](0296-async-await.md) introduced asynchronous functions via `async`, in conjunction with `await`, but did not specify that computed properties or subscripts can support effects like asynchrony. Furthermore, to take full advantage of `async` properties, the ability to specify that a property `throws` is also important. This document aims to partially fill in this gap by proposing a syntax and semantics for effectful read-only computed properties and subscripts. + +#### Terminology +A *read-only computed property* is a computed property that only defines a `get` accessor. Similarly, a *read-only subscript* is a subscript that only defines a `get` accessor. Throughout the remainder of this proposal, any unqualified mention of a "property" or "subscript" refers to a read-only version of that member. Furthermore, unless otherwise specified, the concepts of synchrony, asynchrony, and the definition of something being "async" or "sync" are as described in [SE-0296](0296-async-await.md). + +An *effect* is an observable behavior of a function. Swift's type system tracks a few kinds of effects: `throws` indicates that the function may return along an exceptional failure path with an `Error`, `rethrows` indicates that a throwing closure passed into the function may be invoked, and `async` indicates that the function may reach a suspension point. + +This proposal's examples use features from a number of other recent proposals, such as [structured concurrency](0304-structured-concurrency.md) and [actors](0306-actors.md). Overviews of those features are out of the scope of this proposal, but basic understanding of the importance of those features is required to fully grasp the motivation of this proposal. + +## Motivation + +An asynchronous function is designed for computations that may or always will suspend to perform a context switch before returning. Of primary concern in this proposal are scenarios where the use of Swift concurrency features are limited due to the lack of *effectful read-only computed properties and subscripts* (which will be referred to as simply "effectful properties" from now on), so we will consider those first. Then, we will consider programming patterns in existing Swift code where the availability of effectful properties would help simplify the code. + +#### Swift Concurrency + +An asynchronous call cannot appear within a synchronous context. This fundamental restriction means that computed properties and subscripts would be severely limited in their ability to use Swift's new concurrency features. The only concurrency capability available to them is creating detached tasks, but the completion of those tasks cannot be awaited in synchronous contexts in order to produce an answer: + +```swift +// ... +class Socket { + // ... + public var alive: Bool { + get { + let handle = detach { await self.checkSocketStatus() } + return await handle.get() + // ^~~~~ error: cannot 'await' in a sync context + } + } + + private func checkSocketStatus() async -> Bool { /* ... */ } +} +``` + +It would be better if the property could announce that it may require a suspension to retrieve an answer by allowing it to be marked as `async`. This way, `alive` could directly `await` the result of `checkSocketStatus`. + +As one might imagine, a type that would like to take advantage of actors to isolate concurrent access to resources, while exposing information about those resources through properties, is not possible because one must use `await` to interact with the actor from outside of its isolation context: + +```swift +struct Transaction { /* ... */ } +enum BankError: Error { /* ... */} + +actor AccountManager { + // NOTE: `getLastTransaction` is viewed as async + // when called from outside of the actor + func getLastTransaction() -> Transaction { /* ... */ } + func getTransactions(onDay: Date) async -> [Transaction] { /* ... */ } +} + +class BankAccount { + // ... + private let manager: AccountManager? + var lastTransaction: Transaction { + get { + guard let manager = manager else { + throw BankError.NoManager + // ^~~~~ error: cannot 'throw' in a non-throwing context + } + return await manager.getLastTransaction() + // ^~~~~ error: cannot 'await' in a sync context + } + } + + subscript(_ d: Date) -> [Transaction] { + return await manager?.getTransactions(onDay: d) ?? [] + // ^~~~~ error: cannot 'await' in a sync context + } +} +``` + + + +The use of `throw` in `lastTransaction` highlights a design pattern for properties and subscripts that is not available in Swift. Currently, `lastTransaction` would need to return values of type `Optional`, or some structurally similar `enum` or tuple, to account for the possibility of signaling failure. With the ability to `throw`, the property could describe what went wrong to its users, as opposed to simply returning `nil`, in a form compatible with the established error handling mechanisms in Swift. + +Furthermore, a computed property getter cannot accept any explicit arguments, such as a completion handler, because the syntax for accessing a property is fundamentally designed _not_ to accept such arguments. Such restrictions around input arguments are one of the key differences between computed properties and methods. But, with the advent of `async` functions, an explicit completion-handler argument is no longer required for the function to be asynchronous. Thus, having `async` computed properties does not go against the existing syntax for computed property accesses: it's mainly a distinction in the type system. + + +#### Existing Code + +According to the [API design guidelines](https://swift.org/documentation/api-design-guidelines/), computed properties that do not quickly return, which includes asynchronous operations, are not what programmers typically expect: + +> **Document the complexity of any computed property that is not O(1).** People often assume that property access involves no significant computation, because they have stored properties as a mental model. Be sure to alert them when that assumption may be violated. + +but, computed properties that may block or fail do appear in practice (see the motivation in [this pitch](https://github.com/beccadax/swift-evolution/blob/72c55f33b94749e22637bd8277661599e9cd8007/proposals/0000-throwing-properties.md)). + +As a real-world example of the need for effectful properties, [the SDK defines a protocol](https://developer.apple.com/documentation/avfoundation/avasynchronouskeyvalueloading) `AVAsynchronousKeyValueLoading`, which is solely dedicated to querying the status of a type's property, while offering an asynchronous mechanism to load the properties. The types that conform to this protocol include [AVAsset](https://developer.apple.com/documentation/avfoundation/avasset), which relies on this protocol because its read-only properties are blocking and failable. + +Let's distill the problem solved by `AVAsynchronousKeyValueLoading` into a simple example. In existing code, it is impossible for property `get` access to also accept a completion handler, i.e., a closure for the property to invoke with the result of the operation. Thus, existing code that wished to use computed properties in scenarios where the computation may be blocking must use various workarounds. One workaround is to define an additional asynchronous version of the property as a method that accepts a completion handler: + +```swift +class NetworkResource { + var isAvailable: Bool { + get { /* a possibly blocking operation */ } + } + func isAvailableAsync(completionHandler: ((Bool) -> Void)?) { + // method that returns without blocking. + // completionHandler is invoked once operation completes. + } +} +``` + +The problem with this code is that, even with a comment on `isAvailable` to document that a `get` on this property may block, the programmer may mistakenly use it instead of `isAvailableAsync` because it is easy to ignore a comment. But, if `isAvailable`'s `get` were marked with `async`, then the type system will force the programmer to use `await`, which tells the programmer that the property's access may suspend until the operation completes. Thus, this `async` effect specifier enhances the recommendation made in the [API design guidelines](https://swift.org/documentation/api-design-guidelines/) by leveraging the type checker to warn users that the property access may involve significant computation. + + +## Proposed solution + +For the problems detailed in the motivation section, the proposed solution is to allow `async`, `throws`, or both of these effect specifiers to be marked on a read-only computed property or subscript: + +```swift +// ... +class BankAccount { + // ... + var lastTransaction: Transaction { + get async throws { // <-- NEW: effects specifiers! + guard manager != nil else { + throw BankError.notInYourFavor + } + return await manager!.getLastTransaction() + } + } + + subscript(_ day: Date) -> [Transaction] { + get async { // <-- NEW: effects specifiers! + return await manager?.getTransactions(onDay: day) ?? [] + } + } +} +``` + +At corresponding access-sites of these properties, the expression will be treated as having the effects listed on the `get`-ter, requiring the usual `await` or `try` to surround it as-needed: + +```swift +extension BankAccount { + func meetsTransactionLimit(_ limit: Amount) async -> Bool { + return try! await self.lastTransaction.amount < limit + // ^~~~~~~~~~~~~~~~ + // this access is async & throws + } +} + + +func hadWithdrawlOn(_ day: Date, from acct: BankAccount) async -> Bool { + return await !acct[day].allSatisfy { $0.amount >= Amount.zero } + // ^~~~~~~~~ + // this access is async +} +``` + +Computed properties or subscripts only support effects specifiers if the only kind of accessor defined is a `get`. The main purpose of imposing this read-only restriction is to limit the scope of this proposal to a simple, useful, and easy-to-understand feature. Limiting effects specifiers to read-only properties and subscripts in this proposal does _not_ prevent future proposals from offering them for mutable members. For more discussion of why effectful setters are tricky, see the "Extensions considered" section of this proposal. + +## Detailed design + +This section takes a deep-dive into the changes made to Swift and its implementation as a result of this proposal. + +#### Syntax and Semantics + +Under the [grammar rules for declarations](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html), under "Type Variable Properties", the proposed modifications and additions are: + + +``` +getter-clause → attributes? mutation-modifier? "get" getter-effects? code-block +getter-effects → "throws" +getter-effects → "async" "throws"? +``` + +where `getter-effects` is a new production in the grammar. This production allows one of the three possible combinations of effects specifiers between `get` and `{`, while enforcing an order between `async` and `throws` that mirrors the existing one on functions. Additionally, one can declare (but not define) an effectful property (such as for a protocol) by adding the effect keywords following the `get`, as specified by this grammar: + +``` +getter-setter-keyword-block → "{" getter-keyword-clause setter-keyword-clause? "}" +getter-setter-keyword-block → "{" setter-keyword-clause getter-keyword-clause "}" +getter-keyword-clause → attributes? mutation-modifier? "get" getter-effects? +``` + +For example, one can write: + +```swift +protocol Account { + associatedtype Transaction + + var lastTransaction: Transaction { get async throws } + + subscript(_ day: Date) -> [Transaction] { get async } +} +``` + +to enforce that a type conforming to `Account` provides property and subscript witnesses that have the same or fewer effects than what is allowed by the protocol. + +The interpretation of an effectful property definition is straightforward: the `code-block` appearing in such a `get`-ter definition will be allowed to exhibit the effects specified, i.e., throwing and/or suspending such that `await` and `try` expressions are allowed in that `code-block`. Furthermore, expressions that evaluate to an access of the property or subscript will be treated as having the effects that are declared on that property. One can think of such expressions as a simple desugaring to a method call on the object. It is always possible to determine whether a property has such effects, because the declaration of the property is always known statically. Thus, it is a static error to omit the appropriate `await`, `try`, *etc*. + + +#### Protocol conformance + +In order for a type to conform to a protocol containing effectful properties, the type must contain a property (or subscript) that exhibits *the same or fewer effects than the protocol specifies* for that requirement. This rule mirrors how conformance checking happens for functions with effects: a witness can be missing an effect, but it cannot exhibit an effect that is not accounted for by the requirement. Here is a well-typed example without any superfluous `await`s or `try`s that follows this rule: + +```swift +protocol P { + var someProp: Int { get async throws } +} + +class NoEffects: P { var someProp: Int { get { 1 } } } + +class JustAsync: P { var someProp: Int { get async { 2 } } } + +struct JustThrows: P { var someProp: Int { get throws { 3 } } } + +struct Everything: P { var someProp: Int { get async throws { 4 } } } + +func exampleExpressions() async throws { + let _ = NoEffects().someProp + let _ = try! await (NoEffects() as P).someProp + + let _ = await JustAsync().someProp + let _ = try! await (JustAsync() as P).someProp + + let _ = try! JustThrows().someProp + let _ = try! await (JustThrows() as P).someProp + + let _ = try! await Everything().someProp + let _ = try! await (Everything() as P).someProp +} +``` + +Formally speaking, let us consider a getter `G` to have a set of effects `effects(G)` associated with it. This proposal adds one additional rule to conformance checking: if a getter definition `W` is said to satisfy the requirements of a protocol's getter declaration `R`, then `effects(W)` is a subset of `effects(R)`. + +#### Class Inheritance + +Effectful properties and subscripts can be inherited from a base class, and follow the usual visibility rules. The key difference is that, to override an inherited effectful property (or subscript) from the base class, *the subclass's property must have the same or fewer effects than the property being overridden*. This rule is a natural consequence of the subtyping relation for classes, where the base class must account for all of the effects that its subclasses may exhibit. In essence, this rule is the same as the one for protocol conformance. + +#### Objective-C bridging + +Some API designers may want to take advantage of Swift's effectful properties by having an Objective-C method imported as a property. Objective-C methods are normally imported as Swift methods, so their import as an effectful Swift property will be controlled through an opt-in annotation. This avoids any source compatibility issues for imported declarations. + +Due to the read-only restriction on Swift properties, and the fact that a large number of failable Objective-C methods are already imported as `throws` methods in Swift, support for Objective-C bridging in this proposal is scoped for the Swift concurrency features. Importing as an effectful subscript is not included in this proposal. Furthermore, exporting effectful properties to Objective-C as methods are left to future work. + +To import an Objective-C method as a Swift effectful property, the method must be compatible with the import rules for `async` Swift methods, as described by [SE-0297](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0297-concurrency-objc.md). An annotation changes this import behavior to produce an effectful Swift computed property, instead of an `async` Swift method. The original ObjC method is still imported as a normal Swift method, alongside the property. + +To summarize, an Objective-C method that meets the following requirements: + 1. The method takes exactly one argument, a completion handler, as recognized by + [SE-0297](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0297-concurrency-objc.md). + 2. The method returns `void`. + 3. The method is annotated with `__attribute__((swift_async_name("getter:myProp()")))`. Note the use of `getter:` to specify that it should be a property instead of a method. + +will be imported as an effectful read-only Swift property named `myProp`, instead of a Swift `async` (and possibly also `throws`) method. The following are Objective-C method examples from the SDK that have been annotated for import as an effectful Swift property: + +```objc +// from Safari Services +@interface SFSafariTab: NSObject +- (void)getPagesWithCompletionHandler:(void (^)(NSArray *pages))completionHandler +__attribute__((swift_async_name("getter:pages()"))); +// ... +@end + +// from Exposure Notification +@interface ENManager: NSObject +- (void)getUserTraveledWithCompletionHandler:(void (^)(BOOL traveled, NSError *error))completionHandler +__attribute__((swift_async_name("getter:userTraveled()")));; +// ... +@end +``` + +which would be imported into Swift as: + +```swift +class SFSafariTab: NSObject { + var pages: [SFSafariPage] { + get async { /* ... */ } + } + // ... +} + +class ENManager: NSObject { + var userTraveled: Bool { + get async throws { /* ... */ } + } +} +``` + +## Source compatibility + +The proposed syntactic changes are such that if they appeared in previous versions of the language, they would have been rejected as an error by the parser. + +## Effect on ABI stability + +This proposal is additive and limits its scope intentionally to avoid breaking ABI stability. + +## Effect on API resilience + +As an additive feature, this will not affect API resilience. But, existing APIs that adopt effectful read-only properties will break backwards compatibility, because users of the API will be required to wrap accesses of the property with `await` and/or `try`. + +## Extensions considered + +In this section, we will discuss extensions and additions to this proposal, and why they are not included in the proposed design above. + +#### Effectful settable properties + +Defining the interactions between async and/or throwing writable properties and features such as: + +1. `inout` +2. `_modify` +3. property observers, i.e., `didSet`, `willSet` +4. property wrappers +5. writable subscripts + +is a large project that requires a significant implementation effort. This proposal is primarily motivated by allowing the use of Swift concurrency features in computed properties and subscripts. The proposed design for effectful read-only properties is small and straightforward to implement, while still providing a notable benefit to real-world programs. + +#### Key Paths + +A [key-path expression](https://docs.swift.org/swift-book/ReferenceManual/Expressions.html) is syntactic sugar for instances of the `KeyPath` class and its type-erased siblings. The introduction of effectful properties would require changes to the synthesis of `subscript(keyPath:)` for each type. It is also likely to require restrictions on type-erasure for key-paths that can access effectful properties. + +For example, because we do not allow for function overloading based only on differences in effects, some sort of mechanism like `rethrows` and an equivalent version for `async` (such as a "reasync") would be required on `subscript(keyPath:)` as a starting-point. While a key-path literal can be [automatically treated as a function](0249-key-path-literal-function-expressions.md), a general `KeyPath` value is not a function, so it cannot carry effects in its type. This causes problems when trying to make, for example, a `rethrows` version of `subscript(keyPath:)` work. + +We could also introduce additional kinds of key-paths that have various capabilities, like the existing `WritableKeyPath` and `ReferenceWritableKeyPath`. Then, we could synthesize versions of `subscript` with the right effects specifiers on it, for example, `subscript(keyPath: T) throws`. This would require `KeyPath` kinds for all three new combinations of effects beyond "no effects". + +So, a non-trivial restructuring of the type system, or significant extensions to the `KeyPath` API, would be required to make key-paths work for effectful properties. Thus, for now, we will disallow accesses to effectful properties via key-paths. There already exist restrictions on key-paths to mutable properties based on the instance type (e.g., `WritableKeyPath`), so it would not be unusual to disallow key-paths to effectful properties. + +## Alternatives considered + +In this section, alternative designs for this proposal are discussed. + + + +#### Effects Specifiers Positions + +There are a number of places where the effects specifiers be placed: + +``` + var prop: Type { + get { } +} +``` + +Where `` refers to "position X" in the example. Consider each of these positions: + +* **Position A** is primarily used by access modifiers like `private(set)` or declaration modifiers like `override`. The more effect-like `mutating`/`nonmutating` is only allowed in Position C, which precedes the accessor declaration, just like a method within a struct. This position was not chosen because phrases like `override async throws var prop` or `async throws override var prop` do not read particularly well. +* **Position B** does not make much sense, because effects are only carried as part of a function type, not other types. So, it would be very confusing, leading people to think `Int async throws` is a type, when that is not. Introducing a new kind of punctuation here was ruled out because there are alternatives to this position. +* **Position C** is not bad; it's only occupied by `mutating`/`nonmutating`, but placing effects specifiers here is not consistent with the positioning for functions, which is *after* the subject. Since Position D is available, it makes more sense to use that instead of Position C. +* **Position D** is the one ultimately chosen for this proposal. It is an unused place in the grammar, places the effects on the accessor and not the variable or its type. Plus, it is consistent with where effects go on a function declaration, after the subject: `get throws` and `get async throws`, where get is the subject. Another benefit is that it is away from the variable, so it prevents confusion between the accessor's effects and the effects of a function being returned: + +```swift +var predicate: (Int) async throws -> Bool { + get throws { /* ... */ } +} +``` + +The access of `predicate` may throw, but if it doesn't, it results in a function that is async throws. + +There was also a desire to take advantage of the implicit-getter shorthand for the above: + +```swift +var predicate: (Int) async throws -> Bool { /* ... */ } +``` + +but there is no good place for effects specifiers here. Because this syntax is a short-hand / syntactic sugar, which necessarily has to trade some of its flexibility for conciseness. So, it was decided that it's OK to not allow effectful properties to be declared using this short-hand. The full syntax for computed properties explicitlys defines its accessors, and thus can declare effects on them. + +##### Subscripts +The major difference for subscripts is the method-like header syntax and support for the implicit-getter short-hand, which combined make it look like a method: + +``` +class C { + subscript(_ : InType) -> RetType { /* ... */ } +} +``` + +**Position E** in the above is a tempting place for effects specifiers for a subscript, but subscripts are not methods. They cannot be accessed a first-class function value with `c.subscript`, nor called with `c.subscript(0)`; they use an indexing syntax `c[0]`. Methods cannot be assigned to, but subscript index expressions can be. Thus, they are closer to properties that can accept an argument. + +Much like the short-hand for get-only properties, trying to find a position for effects specifiers on the short-hand form of get-only subscripts (whether its Position E or otherwise) will trap this feature in a corner if writable subscripts can support effects in the future. Why? Position E is a logically valid spot in the full-syntax *and* the short-hand syntax. Creating an inconsistency between the two would be bad. Then, using Position E + the full syntax creates an opportunity for confusion in situations like this: + +```swift +subscript(_ i : Int) throws -> Bool { + get async { } + set { } +} +``` + +Here, the only logical interpretation is that `set` is throws and `get` is async throws. The programmer needs to look in multiple places to add up the effects in their head when trying to determine what effects are allowed in an accessor. This may not seem so bad in this short example, but consider having to skip over a large `get` accessor definition to learn about *all* of the effects the set accessor is allowed to have for this subscript, when you do *not* need to do that for a computed property. + +So, Position D was chosen as the one true place where you can look to see whether there are effects for that type of accessor, both for subscripts and computed properties. + +#### Miscellany + +The `rethrows` specifier is excluded from this proposal because one cannot pass a closure (or any other explicit value) during a property `get` operation. + +The `async`/`await` feature is purpose-built for enabling asynchronous programming, so no consideration is given for alternative solutions that do not rely on that feature for asynchronous properties. The same reasoning applies to `throws`/`try`. + +## Acknowledgments + +Thanks to Doug Gregor and John McCall for their guidance while crafting this proposal. The feasibility and design choices for this proposal were influenced by [Becca Royal-Gordon's proposal for throwing property accessors](https://github.com/beccadax/swift-evolution/blob/72c55f33b94749e22637bd8277661599e9cd8007/proposals/0000-throwing-properties.md) and recent discussions with her. diff --git a/proposals/0311-task-locals.md b/proposals/0311-task-locals.md new file mode 100644 index 0000000000..d6f48593e5 --- /dev/null +++ b/proposals/0311-task-locals.md @@ -0,0 +1,1286 @@ +# Task Local Values + +* Proposal: [SE-0311](0311-task-locals.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.5)** +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/884df3ad6020f0724e06184534b21dd76bd6f4bf/proposals/0311-task-locals.md), [2](https://github.com/swiftlang/swift-evolution/blob/cd1aaef28802a26986094c1f851c261acc796cb6/proposals/0311-task-locals.md), [3](https://github.com/swiftlang/swift-evolution/blob/79b44f3cd15eefc675196136858a5f76a3e58656/proposals/0311-task-locals.md) +* Review: ([first review](https://forums.swift.org/t/se-0311-task-local-values/47478), of revision 1) ([second review](https://forums.swift.org/t/se-0311-2nd-review-task-local-values/47738), of revisions 2 and 3) ([third review](https://forums.swift.org/t/se-0311-3rd-review-task-local-values/49122), of revision 4) ([acceptance](https://forums.swift.org/t/accepted-se-0311-task-local-values/50120)) + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Proposed solution](#proposed-solution) + + [Task Local Values](#task-local-values-1) +* [Detailed design](#detailed-design) + + [Declaring task-local values](#declaring-task-local-values) + + [Binding task-local values](#binding-task-local-values) + - [Binding values for the duration of a child-task](#binding-values-for-the-duration-of-a-child-task) + - [Binding task-local values from synchronous functions](#binding-task-local-values-from-synchronous-functions) + - [Task-local value and tasks which outlive their scope](#task-local-value-and-tasks-which-outlive-their-scope) + + [Task-local value lifecycle](#task-local-value-lifecycle) + + [Reading task-local values](#reading-task-local-values) + - [Reading task-local values: implementation details](#reading-task-local-values-implementation-details) + - [Task-locals in contexts where no Task is available](#task-locals-in-contexts-where-no-task-is-available) + - [Child task and value lifetimes](#child-task-and-value-lifetimes) + * [Task-local value item allocations](#task-local-value-item-allocations) + + [Similarities and differences with SwiftUI's `Environment`](#similarities-and-differences-with-swiftuis-environment) +* [Prior Art](#prior-art) + + [Kotlin: CoroutineContext[T]](#kotlin-coroutinecontextt) + + [Java/Loom: Scope Variables](#javaloom-scope-variables) + + [Go: explicit context passing all the way](#go-explicit-context-passing-all-the-way) +* [Alternatives Considered](#alternatives-considered) + + [Surface API: Type-based key definitions](#surface-api-type-based-key-definitions) + + [Surface API: Key-less value definitions](#surface-api-key-less-value-definitions) +* [Rejected Alternatives](#rejected-alternatives) + + [Plain-old Thread-Local variables](#plain-old-thread-local-variables) + + [Dispatch Queue Specific Values](#dispatch-queue-specific-values) +* [Intended use-cases](#intended-use-cases) + + [Use case: Distributed Tracing & Contextual Logging](#use-case-distributed-tracing--contextual-logging) + - [Contextual Logging](#contextual-logging) + - [Function Tracing](#function-tracing) + - [Distributed Tracing](#distributed-tracing) + - [Future direction: Function wrapper interaction](#future-direction-function-wrapper-interaction) + + [Use case: Mocking internals (Swift System)](#use-case-mocking-internals-swift-system) + + [Use case: Progress Monitoring](#use-case-progress-monitoring) + + [Use case: Executor configuration](#use-case-executor-configuration) +* [Future Directions](#future-directions) + + [Additional configuration options for `@TaskLocal`](#additional-configuration-options-for-tasklocal) + + [Tracing annotations with Function Wrappers](#tracing-annotations-with-function-wrappers) + + [Language features to avoid nesting with `withValue`](#language-features-to-avoid-nesting-with-withvalue) + + [Specialized TaskLocal Value Inheritance Semantics](#specialized-tasklocal-value-inheritance-semantics) + - ["Never" task-local value inheritance](#never-task-local-value-inheritance) +* [Revision history](#revision-history) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) + + +## Introduction + +With Swift embracing asynchronous functions and actors, asynchronous code will be everywhere. + +Therefore, the need for debugging, tracing and otherwise instrumenting asynchronous code becomes even more necessary than before. At the same time, tools which instrumentation systems could have used before to carry information along requests — such as thread locals or queue-specific values — are no longer compatible with Swift's Task-focused take on concurrency. + +Previously, tool developers could have relied on thread-local or queue-specific values as containers to associate information with a task and carry it across suspension boundaries. However, these mechanisms do not compose well in general, have known "gotchas" (e.g., forgetting to carefully maintain and clear state from these containers after a task has completed to avoid leaking them), and do not compose at all with Swift's task-first approach to concurrency. Furthermore, those mechanisms do not feel "right" given Swift's focus on Structured Concurrency, because they are inherently unstructured. + +This proposal defines the semantics of _Task Local Values_. That is, values which are local to a `Task`. + +Task-local values set in a task _cannot_ out-live the task, solving many of the pain points relating to un-structured primitives such as thread-locals, as well as aligning this feature closely with Swift's take on Structured Concurrency. + +Swift-evolution threads: + +- [Review #1](https://forums.swift.org/t/se-0311-task-local-values/47478/11), +- [Pitch #1](https://forums.swift.org/t/pitch-task-local-values/42829/15). + +## Motivation + +Task Local Values are a significant improvement over thread-local storage in that it is a better fit for Swift's concurrency model because it takes advantage of its structured nature. In existing code, developers have used thread-local or queue-specific values to associate state with each thread/queue, however the exact semantics of those mechanisms made it difficult and error prone in reality. + +Specifically, previously developers could have used thread-local or queue-specific values to achieve some of these features, however the exact semantics of those made it difficult and error prone in reality. + +> The use of thread-locals in highly asynchronous libraries is generally frowned upon because it is so difficult to get right, and generally only adds to the confusion rather than helping one achieve transparent context propagation. +> +> This is why currently [Swift Distributed Tracing](https://github.com/apple/swift-distributed-tracing) had to revert to using explicit context passing, making asynchronous APIs even more verbose than they already are. + + +Finally, those mechanisms are outright incompatible with asynchronous code that hops between execution contexts, which also includes Swift's async/await execution semantics which guarantee specific _executors_ or _actors_ to be used for execution, however never guarantees any specific queue or thread use at all. + +> For discussion about alternative approaches to this problem please refer to [Prior Art](#prior-art) and [Alternatives Considered](#alternatives-considered). + +## Proposed solution + +### Task Local Values + +Tasks already require the capability to "carry" metadata with them, and that metadata used to implement both cancellation and priority propagation for a parent task and its child tasks. Specifically Task API's exhibiting similar behavior are: `Task.currentPriority`, and `Task.isCancelled`. Task-local values do not directly use the same storage mechanism, as cancellation and priority is somewhat optimized because _all_ tasks carry them, but the semantics are the same. + +We propose to expose the Task's internal ability to "carry metadata with it" via a Swift API, *aimed for library and instrumentation authors* such that they can participate in carrying additional information with Tasks, the same way as Tasks already do with priority and deadlines and other metadata. + +Task local values may be read from any function running within a task context. This includes *synchronous* functions which were called from an asynchronous function. + +The functionality is also available even if no Task is available in the call stack of a function at all. In such contexts, the task-local APIs will effectively work similar to thread-local storage meaning that they cannot automatically propagate to new (unstructured) threads (e.g. pthread) created from such context. They _will continue to work as expected_ with Task APIs nested inside such scopes however: for example, if `async{}` is used to create an asynchronous task from such synchronous function with no task available, it will inherit task-locals from the synchronous context. + +A task-local must be declared as a static stored property, and annotated using the `@TaskLocal` property wrapper. + +```swift +enum MyLibrary { + @TaskLocal + static var requestID: String? +} +``` + +> 💡 Note: Property wrappers are currently not allowed on global declarations. If this is changed, it should become possible to declare top-level task locals. + +Each task-local declaration represents its own, independent task-local storage. Reading from one declaration will never observe a value stored using a different declaration, even if the declarations look exactly the same. + +Because of those pitfalls with creating multiple instances of the same task local identifier, we propose to diagnose and fail at compile time if the `@TaskLocal` property wrapper is not defined on a static or global property. + +> In order to do so, we will extend the internal `public static subscript(_enclosingInstance object: T, ...)` subscript mechanism to require "no enclosing instance", which will cause the appropriate compile time error reporting to be triggered if such wrapper is used on a non-static or non-global property. + +The diagnosed error would look like this: + +```swift +enum MyLibrary { + @TaskLocal + var requestID: String? + // error: @TaskLocal declaration 'requestID' must be static. + // Task-local declarations must be static stored properties. +} +``` + +It is expected that task-local property declarations will often decide to use an optional type, and default it to `nil`. + +Some declarations however may have "good defaults", such as an empty container type, or some other representation of "not present". For example if `Task.Priority` were expressed using a task local, its `.unspecified` value would be a perfect default value to use in the task-local property declaration. We are certain similar cases exist, and thus want to allow users to retain full control over the type of the property, even if most often an optional is the right thing to use. + +Accessing the value of a task-local property is done by accessing the property wrapper annotated property. + + +```swift +func asyncPrintRequestID() async { + let id = MyLibrary.requestID + print(id ?? "no-request-id") +} + +func syncPrintRequestID() { // also works in synchronous functions + let id = MyLibrary.requestID + print(id ?? "no-request-id") +} +``` + +The task local value is accessible using the same API from async and non async functions, even though it relies on running inside a Task. The asynchronous function always performs a lookup inside the current task, since it is guaranteed to have a current task, while the synchronous function simply immediately returns the default value if it is _not_ called from within a Task context. + +> :warning: Task-local value lookups are more expensive than a direct static property lookup. They involve a thread-local access and scanning a stack of value bindings until a value is found, or the end of the stack is reached. As such, task-local values should be used with care, and e.g. hoisted out of for loops etc, so they are only looked up _once_ whenever possible. + +Binding values is the most crucial piece of this design, as it embraces the structured nature of Swift's concurrency. Unlike thread-local values, it is not possible to just "set" a task local value. This avoids the issue of values being set and forgotten about leading to leaks and hard to debug issues with unexpected values being read in other pieces of code which did not expect them. + +By using scopes and limiting a value's lifetime to the task's lifetime the implementation can use efficient task-local allocation techniques, thereby avoiding the system-wide allocator. Once the scope ends, the child task ends, and the associated task-local value is discarded. + +To bind a specific task-local declaration to a specific value, we can use the `withValue(_:operation:)` function which is declared on the property wrapper type. In order to access this function the `$` sign must be prefixed to the property name, to access the property wrapper's projected value rather than the wrapped value itself: + +```swift +await MyLibrary.$requestID.withValue("1234-5678") { + await asyncPrintRequestID() // prints: 1234-5678 + syncPrintRequestID() // prints: 1234-5678 +} + +await asyncPrintRequestID() // prints: no-request-id +syncPrintRequestID() // prints: no-request-id +``` + +The `withValue` operation is executed synchronously, and no additional tasks are created to execute them. + +It is also possible to bind the same key multiple times while executing in the same task. This can be thought of the most recent binding shadowing the previous one, like this: + +```swift +syncPrintRequestID() // prints: no-request-id + +await MyLibrary.$requestID.withValue("1111") { + syncPrintRequestID() // prints: 1111 + + await MyLibrary.$requestID.withValue("2222") { + syncPrintRequestID() // prints: 2222 + } + + syncPrintRequestID() // prints: 1111 +} + +syncPrintRequestID() // prints: no-request-id +``` + +A task local is readable by any function invoked from a context that has set the value, regardless of how nested it is. For example, it is possible for an asynchronous function to set the value, call through a few asynchronous functions, and finally one synchronous function. All the functions are able to read the bound value, like this: + +```swift +func outer() async -> String? { + await MyLibrary.$requestID.withValue("1234") { + MyLibrary.requestID // "1234" + return middle() // "1234" + } +} + +func middle() async -> String? { + MyLibrary.requestID // "1234" + return inner() // "1234" +} + + +func inner() -> String? { // synchronous function + return MyLibrary.requestID // "1234" +} +``` + +The same property holds for child tasks. For example, if we used a task group to create a child task, it would inherit and read the same value that was set in the outer scope by it's parent. Thanks to guarantees of structured concurrency and child tasks never out-living their parents this still is able to use the efficient storage allocation techniques, and does not need to employ any locking to implement the reads: + +```swift +await MyLibrary.$requestID.withValue("1234-5678") { + await withTaskGroup(of: String.self) { group in + group.addTask { // add child task running this closure + MyLibrary.requestID // returns "1234-5678", which was bound by the parent task + } + + return await group.next()! // returns "1234-5678" + } // returns "1234-5678" +} +``` + +The same operations also work and compose naturally with child tasks created by `async let` and any other future APIs that would allow creating child tasks. + +## Detailed design + +### Declaring task-local values + +Task-local values need to declare a _"key"_ which will be used to access them. This key is represented by the property wrapper instance that is created around the `@TaskLocal` annotated property. + +The `TaskLocal` property wrapper is used to declare task-local keys, based off a static property. + +The property wrapper is defined as: + +```swift +@propertyWrapper +public final class TaskLocal: Sendable, CustomStringConvertible { + let defaultValue: Value + + public init(wrappedValue defaultValue: Value) { + self.defaultValue = defaultValue + } + + @discardableResult + public func withValue(_ valueDuringOperation: Value, + operation: () async throws -> R, + file: String = #file, line: UInt = #line) async rethrows -> R { ... } + + public var wrappedValue: Value { + ... + } + + public var projectedValue: TaskLocal { + get { self } + + @available(*, unavailable, message: "use '$myTaskLocal.withValue(_:operation:)' instead") + set { + fatalError("Illegal attempt to set a \(Self.self) value, use `withValue(...) { ... }` instead.") + } + } + + + public var description: String { + "\(Self.self)(defaultValue: \(self.defaultValue))" + } + +} +``` + +Values stored in task-local storage must conform to the [`Sendable` marker protocol](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md), which ensures that such values are safe to be used from different tasks. Please refer to the `Sendable` proposal for more details on the guarantees and checks it introduces. + +The property wrapper itself must be a `class` because we use it's stable object identifier as *key* for the value lookups performed by the concurrency runtime. + +The implementation of task locals relies on the existence of `withUnsafeCurrentTask` from the [Structured Concurrency proposal](0304-structured-concurrency.md). This is how we are able to obtain a task reference, regardless if within or outside of an asynchronous context. + +### Binding task-local values + +Task locals cannot be "set" explicitly, rather, a scope must be formed within which the key is bound to a specific value. + +This addresses pain-points of task-local values predecessor: thread-locals, which are notoriously difficult to work with because, among other reasons, the hardships of maintaining the set/recover-previous value correctness of scoped executions. It also is cleanly inspired by structured concurrency concepts, which also operate in terms of such scopes (child tasks). + +> Please refer to [Rejected alternatives](#rejected-alternatives), for an in depth analysis of the shortcomings of thread-locals, and how task-local `withValue` scopes address them. + +Binding values is done by using the `$myTaskLocal.withValue(_:operation:)` function, which adds task-local values for the duration of the operation: + +```swift +@discardableResult +public func withValue( + _ valueDuringOperation: Value, + operation: () async throws -> R +) async rethrows -> R +``` + +A synchronous version of this function also exists, allowing users to spare the sometimes unnecessary `await` call, if all code called within the `operation` closure is synchronous as well: + +```swift +@discardableResult +public func withValue( + _ valueDuringOperation: Value, + operation: () throws -> R +) rethrows -> R +``` + +The synchronous version of this API can be called from synchronous functions, even if they are not running on behalf of a Task. The APIs will uphold their expected semantics. Details about how this is achieved will be explained in later sections. + +> In the future, if the `reasync` modifier is implemented and accepted, these two APIs could be combined into one. + +Task-local storage can only be modified by the "current" task itself, and it is not possible for a child task to mutate a parent's task-local values. + +#### Binding values for the duration of a child-task + +The scoped binding mechanism naturally composes with child tasks. + +Binding a task-local value for the entire execution of a child task is done by changing the following: + +```swift +async let dinner = cookDinner() +``` + +which–if we desugar the syntax a little bit to what is actually happening–is equivalent to the right hand side of the `async let` being a closure that will execute concurrently: + +```swift +async let dinner = { + cookDinner() +} +``` + +With that in mind, we only need to wrap the body of the right hand-side with the task-local binding to achieve the result of the value being bound for the entire duration of a specific child task. + +```swift +async let dinner = Lib.$wasabiPreference.withValue(.withWasabi) { + cookDinner() +} +``` + +This will set the wasabi preference task-local value _for the duration of that child task_ to `.withWasabi`. + +If we had two meals to prepare, we could either set the value for both of them, like this: + +```swift +await Lib.$wasabiPreference.withValue(.withWasabi) { + async let firstMeal = cookDinner() + async let secondMeal = cookDinner() + await firstMeal, secondMeal +} +``` + +And finally, if we wanted to set the `withWasabi` reference for most of the tasks in some scope, except one or two of them, we can compose the scopes to achieve this, as expected: + +```swift +await Lib.$wasabiPreference.withValue(.withWasabi) { + async let firstMeal = cookDinner() + async let secondMeal = cookDinner() + async let noWasabiMeal = Lib.$wasabiPreference.withValue(.withoutWasabi) { + cookDinner() + } + await firstMeal, secondMeal, noWasabiMeal +} +``` + +The example here is arguably a little silly, because we could just pass the wasabi preference to the functions directly in this case. But it serves well to illustrate the functioning of the scoping mechanisms. + +In practice, please be careful with the use of task-locals and don't use them in places where plain-old parameter passing would have done the job. Task-local values should be reserved to metadata that does not affect the logical outcome of function calls, but only affects side effects or other configuration parameters of functions. If unsure if a value should be passed directly or via a task-local, err on the side of passing it explicitly and keep in mind that task-locals are primarily designed for "context metadata" such as trace identifiers, authorization tokens etc. + +#### Binding task-local values from synchronous functions + +Reading and binding task-local values is also possible from synchronous functions. + +The same API is used to bind and read values from synchronous functions, however the closure passed to `withValue` when binding a key to a specific value cannot be asynchronous if called from a synchronous function itself (as usual with async functions). + +Sometimes, it may happen that the synchronous `withValue` function is called from a context that has no Task available to attach the task-local binding to. This should rarely be the case in typical Swift programs as all threads and calls should originate from _some_ initiating asynchronous function, however e.g. if the entry point is a call from a C-library or other library which manages it's own threads, a Task may not be available. The task-local values API _continues to work even in those (task-less) circumstances_, by simulating the task scope with the use of a special thread-local in which the task-local storage is written. + +This means that as long as the code remains synchronous, all the usual task-local operations will continue to work even if the functions are called from a task-less context. + +```swift +func synchronous() { // even if no Task is available to this function, the APIs continue to work as expected + printTaskLocal(TL.number) // 1111 + + TL.$number.withValue(2222) { // same as usual + printTaskLocal(TL.number) // 2222 + } + + printTaskLocal(TL.number) // 1111 +} +``` + +#### Task-local value and tasks which outlive their scope + +> Note: In the original pitch it was proposed to allow detached tasks to be forced into inheriting task-local values. We have since decided that detached tasks shall be _fully detached_, and a new API to introduce "continue work asynchronously, with carrying priority and task-local values" will be introduced shortly. +> +> This new core primitive has been implemented here: [Add "async" operation for continuing work asynchronously. #37007](https://github.com/apple/swift/pull/37007), and will be pitched to Swift Evolution shortly. This section expressess its semantics in terms of the new construct. + +Sometimes it may be necessary to "continue work asynchronously" _without waiting_ for the result of such operation. + +Today there exists the `detach` operation which steps out of the realm of Structured Concurrency entirely, and may out-live it's calling scope entirely. This is problematic for task-local values which are built and optimized entirely around the structured notion of child-tasks. Also, a detached task's purpose is to "start from a clean slate" (i.e. detach) from the context it was created from. In other words, detached tasks cannot and will not inherit task-local values (!), much in the same way as they would not inherit the execution context or priority of the calling context. + +To illustrate the interaction of detached tasks and task-locals, consider the following example: + +```swift +await Lib.$sugar.withValue(.noSugar) { + assert(Lib.sugar == .noSugar) + + detach { // completely detaches from enclosing context! + assert(Lib.sugar == .noPreference) // no preference was inherited; it's a detached task! + } + + assert(Lib.sugar == .noSugar) +} +``` + +As expected, because the *detached task* completely discards any contextual information from the creating task, no `.sugar` preferences were automatically carried through to it. This is similar to task priority, which also is never automatically inherited in detached tasks. + +If necessary, it is possible to make a detached task carry a specific priority, executor preference and even task-local value by handling the propagation manually: + +```swift +let sugarPreference = Lib.sugar // store the sugar preference in task-1 +detach(priority: Task.currentPriority) { // manually propagate priority + await Lib.$sugar.withValue(sugarPreference) { // restore the sugar preference in detached-task + assert(Lib.sugar == preference) + } +} +``` + +While this is quite labor intensive and boilerplate heavy, it is intentional that detached tasks never carry any of their legacy around with them. So if a detached task really has to carry some information, it should do so explicitly. + +At the same time, the new `async` (naming pending, perhaps `send` (?!)) operation _does_ inherit all of the following properties of the creating task: execution context, task priority, and task-local values. + +The async operation will be pitched independently, but for the sake of this proposal we only need to focus on the fact how it propagates task-local values. Consider the following snippet: + +```swift +// priority == .background +await Lib.$tea.withValue(.green) { + async { + await Task.sleep(10_000) + // assert(Task.currentPriority == .background) // inherited from creating task (!) + assert(Lib.tea == .green) // inherited from creating task + print("inside") + } +} + +print("outside") +``` + +Note that the `async` operation, similar to a `detach` operation, is allowed to out-live the creating task. I.e. the operation is __not__ a child-task, and as such the usual technique of task-locals to rely on the task tree for storage of the task-locals cannot be used here. + +The implementation ensures correctness of this by _copying_ all task-local value bindings over to the new async task at the point of creation (line 3 in the above example). This means that such operation is slightly heavier than creating a plain child-task, because not only does the task have to be likely heap allocated, it also needs to copy over all task-local bindings from the creating task. + +Please note that what is copied here are only the bindings, i.e. if a reference counted type was bound using `withValue` in the creating task, what is copied to the new task is a reference to the previous task, along with incrementing the reference count to it to keep the referenced object alive. + +--- + +One other situation where a task might out-live the `withValue` lexical-scope is a specific anti-pattern within task groups. This situation is reliabily detected at runtime and cause a crash when it is encountered, along with a detailed explanation of the issue. + +This one situation where a `withValue` scope is not enough to encapsulate the lifetime of a child-task is if the binding is performed _exactly_ around a TaskGroup's `group.addTask`, like this: + +```swift +withTaskGroup(of: String.self) { group in + + Trace.$name.withValue("some(func:)") { // RUNTIME CRASH! + // error: task-local value: detected illegal task-local value binding at Example.swift:68. + // <... more details ... > + + group.addTask { + Trace.name + } + } // end-withValue + + // the added child-task lives until it is pulled out of the group by next() here: + return group.next()! +} +``` + +This is an un-supported pattern because the purpose of `group.addTask` (and `group.addTaskUnlessCancelled`) is explicitly to add off a child-task and return immediately. While the _structure_ of these child-tasks is upheld by no child-task being allowed to escape the task group, the child-tasks do "escape" the scope of the `withValue` — which causes trouble for the internal workings of task locals, which are allocated using an efficient task-local allocation mechanism. + +At the same time, the just shown pattern can be seen as simply wrong usage of the API and programmer error, violating the structured nature of child-tasks. Instead, what the programmer should do in this case is either, set the value for the entire task group, such that all children inherit it: + +```swift +await Trace.$name.withValue("some(func:)") { // OK! + await withTaskGroup(...) { group in + group.addTask { ... } + } +} +``` + +or, set it _within_ the added child-task, as then the task-local allocation will take place inside the child-task, and the lifetime of the value will be correct again, i.e. bounded by the closure lifetime of the added child-task: + +```swift +await withTaskGroup(...) { group in + group.addTask { + await Trace.$name.withValue("some(func:)") { // OK! + ... + } + } +} +``` + +### Task-local value lifecycle + +Task-local values are retained until `withValue`'s `operation` scope exits. Effectively this means that the value is kept alive until all child tasks created in such scope exit as well. This is important because child tasks may be referring to this specific value in the parent task, so it cannot be released earlier. + +Both value and reference types are allowed to be stored in task-local storage, using their expected respective semantics: + +- values stored as task-locals are copied into the task's local storage, +- references stored as task-locals are retained and stored by reference in the task's local storage. + +Task local "item" storage allocations are performed using an efficient task local stack-discipline allocator, since it is known that those items can never out-live a task they are set on. This makes it slightly cheaper to allocate storage for values allocated this way than going through the global allocator, however task-local storage _should not_ be abused to avoid passing parameters explicitly, because it makes your code harder to reason about due to the "hidden argument" passing rather than plain old parameters in function calls. + +Task-local items which are copied to a different task, i.e. when `async{}` launches a new unstructured task, have independent lifecycles and attach to the newly spawned task. This means that, at the point of creating a new task with `async{}`, reference-counted types stored within task-local storage may be retained. + +### Reading task-local values + +Task-local variables are semantically _inherited_ the same way by _child tasks_ similar to some other properties of a task, such as `priority`. + +This implies that stored values may be accessed from different tasks executing concurrently. In order to guarantee safety, task-local values must conform to the `Sendable` protocol, introduced in [SE-0302](0302-concurrent-value-and-concurrent-closures.md). + +Accessing task-local values is synchronous and may be done from any context. If no task is available in the calling context, the default value for the task-local will be returned. The same default value is returned if the accessor is invoked from a context in which a task is present, however the task-local was never bound in this, or any of its parent tasks. + +The specific lookup mechanism used by this accessor will be explained in detail in the next sections. + +The example below explains the contextual awareness of task-local accessors when evaluated as a parameter for a synchronous function call (i.e. `print`): + +```swift +func simple() async { + print("number: \(Lib.number)") // number: 0 + await Lib.$number.withValue(42) { + print("number: \(Lib.number)") // number: 42 + } +} +``` + +The same would work if the second `print` would be multiple asynchronous function calls "deeper" from the `withValue` invocation. + +The same mechanism also works with tasks added in task groups or async let declarations, because those also construct child tasks, which then inherit the bound task-local values of the outer scope. + +```swift +await Lib.$number.withValue(42) { + + await withTaskGroup(of: Int.self) { group in + group.addTask { + Lib.number // task group child-task sees the "42" value + } + return group.next()! // 42 + } + +} +``` + +If a synchronous function is invoked from a context that was not running within a task, it will automatically return the `defaultValue` for given key — since there is no task available to read the value from. + +```swift +func simple() { + print("number: \(Lib.number)") +} +``` + +Usually it doesn’t matter if the function was invoked without first binding the task-local value, or if the execution context is outside the Task runtime, as we can simply return the default value. + +To check if the value was not bound albeit executing within a task, the following pattern can be used: + +````swift +withUnsafeCurrentTask { task in + guard task != nil else { + return "" + } + + return Library.example // e.g. "example" +} +```` + +#### Reading task-local values: implementation details + +There are two approaches possible to implement the necessary semantics. The naive approach being copying all task-local values to every created child task, which obviously creates a large overhead for "set once and then hundreds of tasks read the value" values. Because this is the usual access pattern for such values (e.g. request identifiers and similar), another approach is taken. + +Since the implementation effectively already is a linked list of tasks, where children are able to look up their parent task, we reuse this mechanism to avoid copying values into child tasks. Instead, the _read_ implementation first checks for presence of the key in the current task, if not present, it performs a lookup in its parent, and so on, until no parent is available at which point the default value for the task-local key is returned: + +``` +[detached] () + \ + |[child-task-1] (id:10) + | \ + | |[child-task-1-1] (id:20) + |[child-task-2] (name: "alice") +``` + +Looking up `name` from `child-task-2` will return "alice" immediately, while looking up the same `name` from `child-task-1-1` will have to 1) check in the child task itself, 2) check in `child-task-1`, and finally check in `detached`, all of which returning empty. Looking up `id` from `child-task-1-1` will also return immediately and return `20`, which is what we'd expect — it is the "more specific" value deeper in the call chain. + +We also notice that in many situations, the following chain will exist: + +``` +[detached] () + \ + [child-task-1] (requestID:10) + \ + |[child-task-2] () + \ + |[child-task-3] () + \ + |[child-task-4] () +``` + +Where many tasks can exist however they do not contribute any new task-local values to the chain. Thanks to task locals being immutable at task creation, we can guarantee that their known values never change, and thus we can optimize lookups from all tasks whose parent's do not contribute any additional task-local values. + +Specifically, at creation time of e.g. `child-task-3` we can notice that the parent (`child-task-2`) does not have any task-local values, and thus we can directly point at *its* parent instead: `child-task-1`, which indeed does contribute some values. More generally, the rule is expressed as pointing "up" to the first parent task that actually has any task-local values defined. Thanks to this, looking up `requestID` from `child-task-4` is only costing a single "hop" right into `child-task-1` which happens to define this key. If it didn't contain the key we were looking for, we would continue this search (including skipping empty tasks) until a detached task is reached. + +This approach is highly optimized for the kinds of use-cases such values are used for. Specifically, the following assumptions are made about the access patterns to such values: + +- **relatively few tasks read task-local values** + - there usually is one "root task" which has the task-local information set, and hundreds or thousands of small child tasks (throughout the lifetime of the "root") which may or may not read the value, + - _most_ child tasks do not read the task-local information; and even in tracing situations where potentially many tasks will read the value, this is only true in a fraction of the code's executions, + - **conclusion**: it is not worth aggressively copying the values into all child tasks; taking a small performance hit during lookups is acceptable. +- **there may be many tasks 'between' the task binding the values, and those reading them** + - quite often, values are set by a framework or runtime "once" before offering control flow to user code; usually none of the user-code adds any task-local values, but only uses the existing ones (e.g. in logging or tracing) + - **conclusion**: the "skip task-local 'empty' tasks" optimization is worth it, +- **tasks should never have to worry about "racing" access to task-local values** + - tasks must always be able to call `Lib.myValue` and get predictable values back; specifically, this means that a task _must not_ be able to mutate its task-local values — because child tasks run concurrently with it, this would mean that a child task invoking `Lib.myValue` twice, could get conflicting results, leading to a confusing programming model + - **conclusion**: task-local storage must be initialized at task creation time and cannot be mutated, values may only be "bound" by creating new scopes/tasks. + +> Note: This approach is similar to how Go's `Context` objects work -- they also cannot be mutated, but only `With(...)` copied, however the copies actually form a chain of contexts, all pointing to their parent context. In Swift, we simply reuse the Concurrency model's inherent `Task` abstraction to implement this pattern. + +#### Task-locals in contexts where no Task is available + +Task-locals are also able to function in contexts where no Task is available, they function just as a "dynamic scope" and simply utilize a single thread-local variable to store the task-local storage, rather than forming the chain of storages as is the case when tasks are available. + +The only context in which a Task is not available to the implementation are synchronous functions that were called from the outside of Swift Concurrency. These functions are very rare but can happen, e.g. when a callback is invoked by a C library on some thread that it managed itself. + +The following API continues to work as expected in those situations: + +```swift +func synchronous() { + withUnsafeCurrentTask { task in + assert(task == nil) // no task is available! + } + + Example.$local.withValue(13) { + other() + } +} + +func other() { + print(Example.local) // 13, works as expected +} +``` + +The only Swift Concurrency API that can be called from such synchronous functions and _will_ inherit task-local values is `async{}`, and it will copy any task-local values encountered as usual. This makes for a good interoperability story even with legacy libraries -- we never have to worry if we are on a thread owned by the Swift Concurrency runtime or not, and things continue to work as expected. + +#### Child task and value lifetimes + +It is also important to note that no additional synchronization is needed on the internal implementation of the task-local value stack / linked-list. This is because we strongly rely on guarantees of structured concurrency. Specifically, we exploit the guarantee that: + +> *By the time the scope exits, the child task must either have completed, or it will be implicitly awaited.* When the scope exits via a thrown error, the child task will be implicitly cancelled before it is awaited. + +Thanks to this guarantee child tasks may directly point at the head of the stack of their parent (or super-parent), and we need not implement any additional house-keeping for those references. We know that the parent task will always have values we pointed to from child tasks (defined within a `withValue` body) present, and the child tasks are guaranteed to complete before the `withValue` returns. We use this to automatically pop bound values from the value stack as we return from the `withValue` function, this is guaranteed to be safe, since by that time, all child tasks must have completed and no-one will refer to the task-local values at that point anymore. + +##### Task-local value item allocations + +It is worth calling out that the `withValue(_:) { ... }` API style enables crucial performance optimizations for internal storage of those tasks. + +Since the lifetime of values is bounded by the scope of a `withValue` function along with guarantees made by structured concurrency with reference to parent/child task lifetimes, we are able to use task-local allocation mechanisms, which avoid using the system allocator directly and can be vastly more efficient than global allocation (e.g. malloc). + +### Similarities and differences with SwiftUI's `Environment` + +Readers may be aware of SwiftUI's type [SwiftUI's Environment](https://developer.apple.com/documentation/swiftui/environment) which seemingly has a very similar purpose, however it is more focused on the view hierarchies, rather than "flow of a value _through_ asynchronous calls" which this API is focused on. + +One may think about the difference how these APIs differ in terms of where the "parent/child" relationship is represented. + +SwiftUI's environment considers relationships between views, while task-local values are about the relationship of asynchronous tasks. So while the general idea is similar, the actual semantics are quite different. It is best to visualize task-local values as "following" the execution path, regardless where (in which specific asynchronous function or actor) that execution takes place. + +Swift UI's `@Environment` can be used to define and store custom values, like so: + +```swift +struct Kitchen { + @Environment(\.oven) var oven: Oven +} +``` + +where keys are defined as: + +```swift +public protocol EnvironmentKey { + associatedtype Value + static var defaultValue: Self.Value { get } +} +``` + +and can be implemented as: + +```swift +struct OvenKey: EnvironmentKey { + static let defaultValue: Oven = DefaultOven() +} +extension EnvironmentValues { + var oven: Oven { + get { + return self[OvenKey.self] + } + set { + self[OvenKey.self] = newValue + } + } +} +``` + +Keeping the `OvenKey` `internal` or even `private` allows for fine grained control over who can set or read this value. + + +This API as well as the Swift Distributed Tracing `Baggage` type all adopt the same style and should be used in the same way to set custom keys. However it is NOT the primary purpose of task-local values to help create values — it is to use them _during_ execution of asynchronous functions. + +In other words: + +- **SwiftUI's `@Environment`** is useful for structurally configuring views etc. +- **Task-Local Values** are useful for _carrying_ metadata along through a series of asynchronous calls, where each call may want to access it, and the context is likely different for every single "incoming request" even while the structure of the system remains the same. + +## Prior Art + +### Kotlin: CoroutineContext[T] + +Kotlin offers an explicit API to interact with the coroutine "scope" and "context", these abstractions are very similar to Swift's `Task` abstraction. + +An explicit [`CoroutineContext`](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.coroutines/-coroutine-context/) API is offered to read the context from anywhere it can be accessed. It is semantically equivalent to `[CoroutineContext.Key<...>: CoroutineContext.Element]`, so again, very similar to what we discussed above. + +Usage typically is as follows: + +```kotlin +println("Running in ${coroutineContext[CoroutineName]}") +``` + +where `CoroutineName` is a `Key`, and when executed in a coroutine this yields the expected name. + +Setting a context again can only be done by nesting and scopes, as follows: + +```kotlin +suspend fun withContext( + context: CoroutineContext, + block: suspend CoroutineScope.() -> T +): T +``` + +used like this: + +```kotlin +withContext(Dispatchers.IO) { + // IO dispatcher context variable is in effect here +} +// IO dispatcher context variable is no longer set +``` + +which allows adding context variables while the `block` executes. + +See also [Structured concurrency, lifecycle and coroutine parent-child hierarchy](https://github.com/Kotlin/kotlinx.coroutines/blob/master/ui/coroutines-guide-ui.md#structured-concurrency-lifecycle-and-coroutine-parent-child-hierarchy). + +### Java/Loom: Scope Variables + +Java, with it's coroutine and green-thread based re-thinking of the JVM's execution model, is experimenting with introducing "[*Scope Variables*](https://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part2.html#scope-variables)" which address the same known pain-points of thread-local variables. + +Java's Loom-based concurrency does not expose coroutines or any new concepts into the language (nor does it have async/await or function coloring, because of the use of green threads). + +Snippet explaining their functioning: + +> ```java +> static final Scoped sv = Scoped.forType(String.class); +> +> void foo() { +> try (var __ = sv.bind("A")) { +> bar(); +> baz(); +> bar(); +> } +> } +> +> void bar() { +> System.out.println(sv.get()); +> } +> +> void baz() { +> try (var __ = sv.bind("B")) { +> bar(); +> } +> } +> ``` +> +> `baz` does not mutate `sv`’s binding but, rather introduces a new binding in a nested scope that shadows its enclosing binding. So foo will print: +> +> ``` +> A +> B +> A +> ``` + +This again is very similar to task-local variables, however it expresses it as an actual variable through the access must be performed. + +### Go: explicit context passing all the way + +Go's take on asynchronous context propagation takes the form of the [`Context`](https://golang.org/pkg/context/) type which is part of the standard library. The code of the library boils down to: + +```go +// A Context carries a deadline, cancelation signal, and request-scoped values +// across API boundaries. Its methods are safe for simultaneous use by multiple +// goroutines. +type Context interface { + // Done returns a channel that is closed when this Context is canceled + // or times out. + Done() <-chan struct{} + + // Err indicates why this context was canceled, after the Done channel + // is closed. + Err() error + + // Deadline returns the time when this Context will be canceled, if any. + Deadline() (deadline time.Time, ok bool) + + // Value returns the value associated with key or nil if none. + Value(key interface{}) interface{} +} + +var ( + background = new(emptyCtx) + todo = new(emptyCtx) +) +``` + +Go's `Context` is used for cancellation and deadline propagation as well as other values propagation, it is the _one_ bag for extra values that gets passed explicitly to all functions. + +Notice though that context variables are not typed, the Value returns an `interface{}` (which is like `Any` in Swift). Otherwise though, the general shape is very similar to what Swift is offering. + +The Go programming style is very strict about Context usage, meaning that _every, function that meaningfully can,_ **must** accept context parameter as its first parameter: + +```go +func DoSomething(ctx context.Context, arg Arg) error { + // ... use ctx ... +} +``` + +This results in the context being _everywhere_. Programmers learn to visually ignore the noise and live with it. + +Contexts are immutable, and modifying them is performed by making a new context: + +```go +func WithValue(parent Context, key interface{}, val interface{}) Context +``` + +The implementation is able to form a chain of contexts, such that each context points "back" to its parent forming a chain that is walked when we resolve a value by key. + +This blog post is fairly informative on how this is used in the real world: [ +Go Concurrency Patterns: Context](https://blog.golang.org/context). + +## Alternatives Considered + +### Surface API: Type-based key definitions + +The initially pitched approach to define task-local keys was impossible to get wrong thanks to the type always being unique. However declaring and using the keys was deemed too tiresome by the community during review, thus the proposal currently is pitching `@TaskLocal` property wrapper. + +The previous design required this boilerplate to declare a key: + +```swift +extension TaskLocalValues { + + public struct RequestIDKey: TaskLocalKey { + // alternatively, one may declare a nil default value: + // public static var defaultValue: String? { nil } + public static var defaultValue: String { "" } + + // additional options here, like e.g. + // static var inherit: TaskLocalValueInheritance = . never + } + + public var requestID: RequestIDKey { .init() } + +} +``` + +and usage would look like this: + +```swift +await Task.withLocal(\.requestID, boundTo: "abcd") { + _ = Task.local(\.requestID) // "abcd" +} +``` + +It was argued that the declaration is too boilerplate heavy and thus discarded and we moved towards the property wrapper based API. + +### Surface API: Key-less value definitions + +Stefano De Carolis proposed on the forums to simplify the definition sites to be: + +```swift +extension Task.Local { + var foo: String { "Swift" } +} +``` + +Our concerns about this shape of API are: + +- it prioritizes briefity and not clarity. It is not clear that the value returned by the computed property `foo` is the default value. And there isn't a good place to hint at this. In the `...Key` proposal we have plenty room to define a function `static var defaultValue` which developers need to implement, immediately explaining what this does. +- this shape of API means that we would need to actively invoke the key-path in order to obtain the value stored in it. We are concerned about the performance impact of having to invoke the key-path rather than invoke a static function on a key, however we would need to benchmark this to be sure about the performance impact. +- it makes it harder future extension, if we needed to allow special flags for some keys. Granted, we currently do not have an use-case for this, but with Key types it is trivial to add special "do not inherit" or "force a copy" or similar behaviors for specific keys. It is currently not planned to implement any such modifiers though. + +For completeness, the functions to read and bind values with this proposal would become: + +```swift +enum Task { + enum Local {} + + static func withLocal( + _ path: KeyPath, + boundTo value: Value, + body: @escaping () async -> R + ) async -> R { ... } + + static func local( + _ path: KeyPath + ) async -> Value { ... } +} +``` + +## Rejected Alternatives + +### Plain-old Thread-Local variables + +Thread-local storage _cannot_ work effectively with Swift's concurrency model. + +Swift's concurrency model deliberately abstains from using the thread terminology, because _no guarantees_ are made about specific threads where asynchronous functions are executed. Instead, guarantees are phrased in terms of Tasks and Executors (i.e. a thread pool, event loop or dispatch queue actually running the task). + +In other words: Thread locals cannot effectively work in Swift's concurrency model, because the model does not give _any_ guarantees about specific threads it will use for operations. + +We also specifically are addressing pain points of thread-locals with this proposal, as it is far too easy to make these mistakes with thread-local values: + +- it is hard to use thread locals in highly asynchronous code, e.g. relying on event loops or queue-hopping, because on every such queue hop the library or end-user must remember to copy and restore the values onto the thread which later-on is woken up to resume the work (i.e. in a callback), +- it is possible to "leak" values into thread locals, i.e. forgetting to clean up a thread-local value before returning it to a thread pool, may result in: + - a value never being released leading to memory leaks, + - or leading to new workloads accidentally picking up values previously set for other workloads; +- thread locals are not "inherited" so it is difficult to implement APIs which "carry all thread-local values to the underlying worker thread" or even jump to another worker thread. All involved libraries must be aware of the involved thread locals and copy them to the new thread — which is both inefficient, and error prone (easy to forget). + +None of those issues are possible with task-local values, because they are inherently scoped and cannot outlive the task with which they are associated. + +| **Issue** | **Thread-Local Variables** | **Task-Local Values** | +|---------------------|-----------------------------|-----------------------| +| "Leaking" values | Possible to forget to "unset" a value as a scope ends. | Impossible by construction; scopes are enforced on the API level, and are similar to scoping rules of async let and Task Groups. | +| Reasoning | Unstructured, no structural hints about when a variable is expected to be set, reset etc. | Simpler to reason about, follows the child-task semantics as embraced by Swift with `async let` and Task Groups. | +| Carrier type | Attached specific **threads**; difficult to work with in highly asynchronous APIs (such as async/await). | Attached to specific tasks, accessible through a task's child tasks as well, forming a hierarchy of values, which may be used to provide more specific values for children deeper in the hierarchy. +| Mutation | Thread locals may be mutated by *anyone*; A caller cannot assume that a function it called did not modify the thread-local that it has just set, and may need to be defensive about asserting this. | Task locals cannot modify their parent's values; They can only "modify" values by immutably adding new bindings in their own task; They cannot change values "in" their parent tasks. | + +### Dispatch Queue Specific Values + +Dispatch offers APIs that allow setting values that are _specific to a dispatch queue_: +- [`DispatchQueue.setSpecific(key:value:)`](https://developer.apple.com/documentation/dispatch/dispatchqueue/2883699-setspecific) +- [`DispatchQueue.getSpecific(key:)`](https://developer.apple.com/documentation/dispatch/dispatchqueue/1780751-getspecific). + +These APIs serve their purpose well, however they are incompatible with Swift Concurrency's task-focused model. Even if actors and asynchronous functions execute on dispatch queues, no capability to carry values over multiple queues is given, which is necessary to work well with Swift Concurrency, as execution may hop back and forth between queues. + +## Intended use-cases +It is important to keep in mind the intended use case of this API. Task-local values are not intended to replace passing parameters where doing so explicitly is the right tool for the job. Please note that task local storage is more expensive to access than parameters passed explicitly. They also are "invisible" in API, so take care to avoid accidentally building APIs which absolutely must have some task local value set when they are called as this is very surprising and hard to debug behavior. + +Only use task local storage for auxiliary _metadata_ or "_execution scoped configuration_", like mocking out some runtime bits for the duration of a _specific call_ but not globally, etc. + +### Use case: Distributed Tracing & Contextual Logging + +> This section refers to [Apple/Swift-Distributed-Tracing](https://github.com/apple/swift-distributed-tracing) + +Building complex server side systems is hard, especially as they are highly concurrent (serving many thousands of users concurrently) and distributed (spanning multiple services and nodes). Visibility into such system–i.e. if it is performing well, or lagging behind, dropping requests, or for some kinds of requests experiencing failures–is crucial for their successful deployment and operation. + +#### Contextual Logging + +Developers instrument their server side systems using logging, metrics and distributed tracing to gain some insight into how such systems are performing. Improving such observability of back-end systems is crucial to their success, yet also very tedious to manually propagate the context. + +Today developers must pass context explicitly, and with enough cooperation of libraries it is possible to make this process relatively less painful, however it adds a large amount of noise to the already noisy asynchronous functions: + +```swift +func chopVegetables(context: LoggingContext) async throws -> [Vegetable] { + context.log.info("\(#function)") +} +func marinateMeat(context: LoggingContext) async -> Meat { + context.log.info("\(#function)") +} +func preheatOven(temperature: Double, context: LoggingContext) async throws -> Oven { + context.log.info("\(#function)") +} + +// ... + +func makeDinner(context: LoggingContext) async throws -> Meal { + context.log.info("\(#function)") + + async let veggies = chopVegetables(context: context) + async let meat = marinateMeat(context: context) + async let oven = preheatOven(temperature: 350, context: context) + + let dish = Dish(ingredients: try await [veggies, meat]) + return try await oven.cook(dish, duration: .hours(3), context: context) +} +``` + +Thanks to the passed `LoggingContext` implementations may be invoked with a specific `"dinner-request-id"` and even if we are preparing multiple dinners in parallel, we know "which dinner" a specific operation belongs to: + +```swift +var context: LoggingContext = ... + +context.baggage.dinnerID = "1234" +async let first = makeDinner(context: context) + +context.baggage.dinnerID = "5678" +async let second = makeDinner(context: context) + +await first +await second +``` + +Resulting in logs like this: + +``` + dinner-id=1234 makeDinner + dinner-id=1234 chopVegetables + dinner-id=5678 makeDinner + dinner-id=5678 chopVegetables + dinner-id=1234 marinateMeat + dinner-id=1234 preheatOven + dinner-id=5678 marinateMeat + dinner-id=1234 cook + dinner-id=5678 preheatOven + dinner-id=5678 cook +``` + +Allowing developers to track down the request specific logs by filtering logs by the `dinner-id` that may be encountering some slowness, or other issues. + +#### Function Tracing + +We do not stop there however, by instrumenting all functions with swift-tracing, we can obtain a full trace of the execution (in production), and analyze it later on using tracing systems such as [zipkin](https://zipkin.io), [jaeger](https://www.jaegertracing.io/docs/1.20/#trace-detail-view), [Grafana](https://grafana.com/blog/2020/11/09/trace-discovery-in-grafana-tempo-using-prometheus-exemplars-loki-2.0-queries-and-more/), or others. + +By instrumenting all functions with tracing: + +```swift +// ... + +func chopVegetables(context: LoggingContext) async throws -> [Vegetable] { + let span = InstrumentationSystem.tracer.startSpan(#function, context: context) + defer { span.end() } + ... +} + +// ... +``` + +we are able to obtain visualizations of the asynchronous computation similar to the diagram shown below. The specific visualization depends on the tracing system used, but generally it yields such trace that can be inspected offline using server side trace visualization systems (such as Zipkin, Jaeger, Honeycomb, etc): + +``` +>-o-o-o----- makeDinner ----------------o---------------x [15s] + \-|-|- chopVegetables--------x | [2s] + | | \- chop -x | | [1s] + | | \--- chop -x | [1s] + \-|- marinateMeat -----------x | [3s] + \- preheatOven -----------------x | [10s] + \--cook---------x [5s] +``` +* diagram only for illustration purposes, generally this is displayed using fancy graphics and charts in tracing UIs. + +Such diagrams allow developers to naturally "spot" and profile the parallel execution of their code. Values declared as `async let` introduce concurrency to the program's execution, which can be visualized using such diagrams and also easily spot which operations dominate the execution time and need to be sped up or perhaps parallelized more. In the above example, we notice that since `preheatOven` takes 10 seconds in any case, even if we sped up `chopVegetables` we will _not_ have sped up the entire `makeDinner` task because it is dominated by the preheating. If we were able to optimize the `cook()` function though we could shave off 5 seconds off our dinner preparation! + +#### Distributed Tracing + +So far this is on-par with an always on "profiler" that is sampling a production service, however it does only sample a single node — all the code is on the same machine... + +The most exciting bit about distributed tracing is that the same trace graphs can automatically be produced even _across libraries_ and across nodes in a _distributed system_. Thanks to HTTP Clients, Servers and RPC systems being aware of the metadata carried by asynchronous tasks, we are able to carry tracing beyond single-nodes, and easily trace distributed systems. + +> For in depth details about this subject, please refer to [Swift Distribted Tracing](https://github.com/apple/swift-distributed-tracing). + +If, for whatever reason, we had to extract `chopVegetables()` into a _separate (web) service_, the exact same code can be written — and if the networking library used to make calls to this *"ChoppingService"* are made, the trace is automatically propagated to the remote node and the full trace now will include spans from multiple machines (!). To visualize this we can show this as: + +``` +>-o-o-o----- makeDinner ----------------o---------------x [15s] + | | | | | | +~~~~~~~~~ ChoppingService ~~|~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + \-|-|- chopVegetables-----x | [2s] \ + | | \- chop -x | | [1s] | Executed on different host (!) + | | \- chop --x [1s] / +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + \-|- marinateMeat -----------x | [3s] + \- preheatOven -----------------x | [10s] + \--cook---------x [5s] +``` + +Thanks to swift-tracing and automatic context propagation we can make tracing much simpler to adopt, and reap the benefits from it; Making complex server side systems (almost) as simple to debug as local single process concurrent applications. + + +#### Future direction: Function wrapper interaction + +> **CAVEAT**: This proposal does not imply/promise any future work on function wrappers, however **if** they were proposed and accepted at some point, this would be a natural use-case for them. + +If Swift were to get "function wrappers", tracing a set of asynchronous functions becomes trivial: + +```swift +@Traced func chopVegetables() async throws -> [Vegetable] { ... } +@Traced func marinateMeat() async -> Meat { ... } +@Traced func preheatOven(temperature: Double) async throws -> Oven { ... } + +// ... + +@Traced +func makeDinner() async throws -> Meal { + async let veggies = try await chopVegetables() + async let meat = await marinateMeat() + async let oven = try await preheatOven(temperature: 350) + + let dish = Dish(ingredients: await [veggies, meat]) + return try await oven.cook(dish, duration: .hours(3)) +} +``` + +Which would automatically *start* a tracing `Span` with an *operation name* "makePizza" and *end* it when the function returns, saving developers multiple layers of nesting and directly interacting with the Tracer API, and making Swift a truly great citizen among observability-first languages and best-in class for distributed systems programming in general. + +This way, eventually, we would have gained all benefits of (distributed) tracing and contextual logging, without any of the noise and complexity usually associated with it. + +### Use case: Mocking internals (Swift System) + +Some libraries may offer a special API that allows switching e.g. filesystem access to a "mock filesystem" if it is present in a task local value. + +This way developers could configure tasks used in their tests to bind a "mock filesystem" under a known to the underlying library task local value, and this way avoid writing/reading from a real filesystem in tests, achieving greater test isolation without having to pass a specific `Filesystem` instance through all API calls of the library. + +This pattern exists today in [Swift System](https://github.com/apple/swift-system) where the [withMockingEnabled](https://github.com/apple/swift-system/pull/8/files#diff-9e369bd109521aa185f8c63d962d415e58f03f6d8e80c3abd5e544511937452dR115-R128) function is used to set a thread local which changes how functions execute (and allows them to be traced). The mechanism used there, and in similar frameworks, _will not_ work in the future as Swift adopts `async` and `Swift System` itself would want to adopt async functions, since they are a prime candidate to suspend a task while a write is being handled asynchronously (e.g. if one were to implement APIs using `io_uring` or similar mechanisms). Task Local Values enable Swift System to keep it's mocking patterns working and efficient in the face of asynchronous functions. + +### Use case: Progress Monitoring + +In interactive applications asynchronous tasks frequently are linked with some progress indicator such that a user waiting for the task knows that it indeed is proceeding, and not just "stuck" on a never-ending "Loading..."-screen. + +Foundation offers the [Progress](https://developer.apple.com/documentation/foundation/progress) type which is used with UI frameworks, such as [SwiftUI](https://developer.apple.com/xcode/swiftui/), to easily report back progress of tasks back to users. Currently, `Progress` can be used by either passing it manually and explicitly, or accessing it through thread-local storage. + +`Progress` naturally has it's own child-progress semantics which exactly mirror how the compiler enforces child task relationships — child tasks contribute to the task's progress after all. Using task local values we could provide a nice API for progress monitoring that naturally works with tasks and child tasks, without causing noise in the APIs, and also avoiding the issues of thread-local style APIs which are notoriously difficult to use correctly. + +### Use case: Executor configuration + +A frequent requirement developers have voiced is to have some control and configurability over executor details on which tasks are launched. + +By using task locals we have a mechanism that flows naturally with the language, and due to inheritance of values also allows to automatically set up the preferred executor for tasks which do not have a preference. +For example, invoking such actor-independent functions `calcFoo` and `calcBar` could be scheduled on specific executors (or perhaps, allow configuring executor settings) by setting a task local value like this: + +```swift +// Just ideas, not actual API proposal (!) +async let foo = $myExecutor.withValue(someSpecificExecutor) { + calcFoo() +} +``` + +## Future Directions + +### Additional configuration options for `@TaskLocal` + +In our current work we discovered a number of special keys which we will be introducing in the future, e.g. to support operating system requirements for tracing calls, authentication or support for novel patterns such as a Swift Concurrency aware `Progress` type. + +Some of those keys will want to make different performance tradeoffs. For example, tracing IDs may want to require being propagated in an in-line storage and copied every time to a child task upon `spawn` rather than being lazily accessed on each read operation. Or certain keys may wish to propagate to child tasks only when called explicitly, so a "don't inherit" propagation policy could be used. + +These configuration options are able to be introduced in binary and source compatible ways to the property wrapper and backing storage. The storage requirements for those flags are minimal, and such flags will only ever be created once per specific task-local key. + +### Tracing annotations with Function Wrappers + +As discussed in the tracing use-case section, the ability to express `@Logged` or `@Traced` as annotations on existing functions to easily log and trace function invocations is definitely something various people have signalled a strong interest in. And this feature naturally enables the implementation of those features. + +Such annotations depend on the arrival of [Function Wrappers](https://forums.swift.org/t/prepitch-function-wrappers/33618) or a similar feature to them, which currently are not being actively worked on, however we definitely have in the back of our minds while designing this proposal. + +### Language features to avoid nesting with `withValue` + +It is necessary for task-local correctness to only bind values for a given scope. + +This scoping rule is enforced by the only API to bind a task-local value being the `withValue() { ... }` function. The function essentially does two operations: + +- pushes a new binding onto the task-local bindings stack in the current task +- (executes the user provided `body` closure) +- and pops the binding from the task-local bindings stack of the current task + +This structure allows us to use task-local allocation safely, and also ensure that no lingering values ever "leak" the scope where they were defined. + +It is, however, slightly cumbersome on a source level to have to indent code only in order to get this property when in reality most of the time a binding is going to be set for the entirety (or remaining part) of the current function. + +If Swift were to gain some "`using`" mechanism, that would encapsulate the pattern of doing one "start" operation now and an "end" operation at scope exit. That would be a very general feature with broad applicability. The same mechanism could be used by task-locals rather than resorting to nesting. + +```swift +scoped_overwrite requestID = id // <- intentionally terrible syntax +``` + +which by default would desugar as: + +```swift +let old = + = new +defer { = old } +``` + +but which property wrappers would have some ability to customize. Such feature may allow expressing the pattern necessary for task-local binding correctness without the cumbersome nesting. + +### Specialized TaskLocal Value Inheritance Semantics + +Some task local values may require specialized inheritance semantics. The default strategy simply means that child tasks "inherit" values from their parents. At runtime, this is not achieved by copying, but simply performing lookups through parent tasks as well, when a `TaskLocalInheritance.default` inherited key is being looked up. + +Some, specialized use-cases however can declare more specific inheritance semantics. It is *not* encouraged to use these specialized semantics nonchalantly, and their use should always be carefully considered and given much thought as they can lead to unexpected behaviors otherwise. + +A `TaskLocal` type may declare an inheritance semantics by defining the static `inherit` parameter when declaring the variable: `TaskLocal(inherit: .never)`. + +The semantics default to `.default`, which are what one would expect normally — that child tasks are able to lookup values defined in their parents, unless overridden in that specific child. We will discuss the exact semantics of lookups in depth in [Reading task-local values](#reading-task-local-values). + +In this proposal, we introduce two additional inheritance semantics: `.never` and `.alwaysBestEffort`: + +```swift +/// Allows configuring specialized inheritance strategies for task local values. +/// +/// By default, task local values are accessible by the current or any of its +/// child tasks (with this rule applying recursively). +/// +/// Some, rare yet important, use-cases may require specialized inheritance +/// strategies, and this property allows them to configure these for their keys. +public enum TaskLocalInheritance: UInt8, Equatable { + case `default` = 0 + case never = 1 + case alwaysBestEffort = 2 +} +``` + +Note that `TaskLocalInheritance` should remain extensible. + +Both these semantics are driven by specific use cases from the Swift ecosystem, highlighted during early design reviews of this proposal. First, the `.never` inheritance model allows for the design of a highly specialized task-aware `Progress` type, that is not part of this proposal. And second various Tracer implementations, including swift-distributed-tracing but also Instruments which will want to use special tracing metadata which should be carried "always" (at a best effort), even through detached tasks. + +#### "Never" task-local value inheritance + +The "never" inheritance semantics allow a task to set "truly local only to this specific task" values. I.e. if a parent task sets some value using an non-inherited key, it's children will not be able to read it. + +It is simplest to explain those semantics with an example, so let us do just that. First we define a key that uses the `.never` inheritance semantics. We could, for example, declare a `House?` task-local and make sure it will not be inherited by our children (child tasks): + +```swift +struct House { + + @TaskLocal(inherit: .never) + static var key: House? + +} +``` + +This way, only the current task which has bound this task local value to itself can access the house key. This key remains available throughout the entire `withValue`'s scope. However none of its child tasks, spawned either by async let, or task groups will inherit the `house`: + +```swift +House.$key.withValue(House(...)) { + async let child = assert(House.key == nil) // not available in child task +} +``` + +Addmitably, this is a fairly silly example, and in this small limited example it is trivial to replace this task local with a plain variable. Or rather, it *should* be replaced with a plain variable and not abuse task locals for this. + +We have specific designs in mind with regards to `Progress` monitoring types however which will greatly benefit from these semantics. The `Progress` API will be it's own swift evolution proposal however, so we do not dive much deeper into it's API design in this proposal. Please look forward to upcoming proposals with regards to monitoring + +## Revision history + +- v5: Allow usage even in contexts where no Task is available + - Fallback to additional thread-local storage if no Task is available to bind/get task local values from, + - remove API on UnsafeCurrentTask, its use-cases are now addressed by the core `withValue(_:)` API, +- v4.5: Drop the Access type and use the `projectedValue` to simplify read and declaration sites. +- v4: Changed surface API to be focused around `@TaskLocal` property wrapper-style key definitions. + - introduce API to bind task-local values in synchronous functions, through `UnsafeCurrentTask` + - allude to `async` (or `send`) as the way to carry task-local values rather than forcing them into a detached task + - explain an anti-pattern that will be detected and cause a crash if used around wrapping a `group.addTask` with a task local binding. Thank you to @Lantua over on the Swift Forums for noticing this specific issue. +- v3.2: Cleanups as the proposal used outdated wordings and references to proposals that since have either changed or been accepted already. + - No semantic changes in any of the mechanisms proposed. + - Change mentions of `ConcurrentValue` to `Sendable` as it was since revised and accepted. +- v3.1: Move specialized task semantics to future directions section + - Will adjust implementation to not offer the "do not inherit" mode when accepted +- v3: Prepare for review + - polish wording and API names of "task-local value inheritance" related functions and wording, + - discuss detached tasks and runDetached with inheritance, + - explain the use of task-local allocation as a core idea to those task local items. +- v2: Thanks to the introduction of `Task.unsafeCurrent` in Structured Concurrency, we're able to amend this proposal to: + - allow access to task-locals from *synchronous* functions, + - link to the [ConcurrentValue](https://forums.swift.org/t/pitch-3-concurrentvalue-and-concurrent-closures/43947) proposal and suggest it would be used to restrict what kinds of values may be stored inside task locals. + - introduce specialized limited storage for specialized trace keys, to be carried even through detached tasks. + - rewordings and clarifications. +- v1: Initial draft + +## Source compatibility + +This change is purely additive to the source language. + +## Effect on ABI stability + +This proposal is additive in nature. + +It adds one additional pointer for implementing the task local value stack in `AsyncTask`. + +## Effect on API resilience + +No impact. diff --git a/proposals/0312-indexed-and-enumerated-zip-collections.md b/proposals/0312-indexed-and-enumerated-zip-collections.md new file mode 100644 index 0000000000..0ead03b386 --- /dev/null +++ b/proposals/0312-indexed-and-enumerated-zip-collections.md @@ -0,0 +1,332 @@ +# Add `indexed()` and `Collection` conformances for `enumerated()` and `zip(_:_:)` + +* Proposal: [SE-0312](0312-indexed-and-enumerated-zip-collections.md) +* Author: [Tim Vermeulen](https://github.com/timvermeulen) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Returned for revision** +* Implementation: [apple/swift#36851](https://github.com/apple/swift/pull/36851) + +## Introduction +This proposal aims to fix the lack of `Collection` conformance of the sequences returned by `zip(_:_:)` and `enumerated()`, preventing them from being used in a context that requires a `Collection`. Also included is the addition of the `indexed()` method on `Collection` as a more ergonomic, efficient, and correct alternative to `c.enumerated()` and `zip(c.indices, c)`. + +Swift-evolution thread: [Pitch](https://forums.swift.org/t/pitch-add-indexed-and-collection-conformances-for-enumerated-and-zip/47288) + +## Motivation +Currently, the `Zip2Sequence` and `EnumeratedSequence` types conform to `Sequence`, but not to any of the collection protocols. Adding these conformances was impossible before [SE-0234 Remove `Sequence.SubSequence`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0234-remove-sequence-subsequence.md), and would have been an ABI breaking change before the language allowed `@available` annotations on protocol conformances ([PR](https://github.com/apple/swift/pull/34651)). Now we can add them! + +Conformance to the collection protocols can be beneficial in a variety of ways, for example: +* `(1000..<2000).enumerated().dropFirst(500)` becomes a constant time operation. +* `zip("abc", [1, 2, 3]).reversed()` will return a `ReversedCollection` rather than allocating a new array. +* SwiftUI’s `List` and `ForEach` views will be able to directly take an enumerated or zipped collection as their data. + +This proposal also includes the addition of the `indexed()` method (which can already be found in the [Swift Algorithms](https://github.com/apple/swift-algorithms) package) as an alternative for many use cases of `zip(_:_:)` and `enumerated()`. When the goal is to iterate over a collection’s elements and indices at the same time, `enumerated()` is often inadequate because it provides an offset, not a true index. For many collections this integer offset is different from the `Index` type, and in the case of `ArraySlice` in particular this offset is a common source of bugs when the slice’s `startIndex` isn’t `0`. `zip(c.indices, c)` solves these problems, but it is less ergonomic than `indexed()` and potentially less performant when traversing the indices of a collection is computationally expensive. + +## Detailed design +Conditionally conform `Zip2Sequence` to `Collection` and `BidirectionalCollection`. + +> **Note**: OS version 9999 is a placeholder and will be replaced with whatever actual OS versions this functionality will be introduced in. + +```swift +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension Zip2Sequence: Collection + where Sequence1: Collection, Sequence2: Collection +{ + // ... +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension Zip2Sequence: BidirectionalCollection + where Sequence1: BidirectionalCollection, Sequence2: BidirectionalCollection +{ + // ... +} +``` + +Add a `zip(_:_:)` overload that returns a random-access collection when given two random-access collections. + +```swift +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +public func zip( + _ base1: Base1, _ base2: Base2 +) -> Zip2RandomAccessCollection { + Zip2RandomAccessCollection(base1, base2) +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +public struct Zip2RandomAccessCollection + where Base1: RandomAccessCollection, Base2: RandomAccessCollection +{ + // ... +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension Zip2RandomAccessCollection: RandomAccessCollection { + // ... +} +``` + +Conditionally conform `EnumeratedSequence` to `Collection`, `BidirectionalCollection`, `RandomAccessCollection`, and `LazyCollectionProtocol`. + +```swift +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension EnumeratedSequence: Collection where Base: Collection { + // ... +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension EnumeratedSequence: BidirectionalCollection + where Base: BidirectionalCollection +{ + // ... +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension EnumeratedSequence: RandomAccessCollection + where Base: RandomAccessCollection {} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension EnumeratedSequence: LazySequenceProtocol + where Base: LazySequenceProtocol {} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension EnumeratedSequence: LazyCollectionProtocol + where Base: LazyCollectionProtocol {} +``` + +Add an `indexed()` method to `Collection` that returns a collection over (index, element) pairs of the original collection. + +```swift +extension Collection { + @available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) + public func indexed() -> IndexedCollection { + Indexed(_base: self) + } +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +public struct IndexedCollection { + // ... +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension IndexedCollection: Collection { + // ... +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension IndexedCollection: BidirectionalCollection where Base: BidirectionalCollection { + // ... +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension IndexedCollection: RandomAccessCollection where Base: RandomAccessCollection {} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension IndexedCollection: LazySequenceProtocol where Base: LazySequenceProtocol {} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension IndexedCollection: LazyCollectionProtocol where Base: LazyCollectionProtocol {} +``` + +## Source compatibility +Adding `LazySequenceProtocol` conformance for `EnumeratedSequence` is a breaking change for code that relies on the `enumerated()` method currently not propagating `LazySequenceProtocol` conformance in a lazy chain: + +```swift +extension Sequence { + func everyOther_v1() -> [Element] { + let x = self.lazy + .enumerated() + .filter { $0.offset.isMultiple(of: 2) } + .map(\.element) + + // error: Cannot convert return expression of type 'LazyMapSequence<...>' to return type '[Self.Element]' + return x + } + + func everyOther_v2() -> [Element] { + // will keep working, the eager overload of `map` is picked + return self.lazy + .enumerated() + .filter { $0.offset.isMultiple(of: 2) } + .map(\.element) + } +} +``` + +All protocol conformances of an existing type to an existing protocol are potentially source breaking because users could have added the exact same conformances themselves. However, given that `Zip2Sequence` and `EnumeratedSequence` do not expose their underlying sequences, there is no reasonable way anyone could have conformed either type to `Collection` themselves. The only sensible conformance that could conflict with one of the conformances added in this proposal is the conformance of `EnumeratedSequence` to `LazySequenceProtocol`. + +## Effect on ABI stability +This proposal does not affect ABI stability. + +## Alternatives considered +#### Don’t add `LazyCollectionProtocol` conformance for `EnumeratedSequence` for the sake of source compatibility. +We consider it a bug that `enumerated()` currently does not propagate laziness in a lazy chain. + +#### Keep `EnumeratedSequence` the way it is and add an `enumerated()` overload to `Collection` that returns a `Zip2Sequence, Self>`. +This is tempting because `enumerated()` is little more than `zip(0..., self)`, but this would cause an unacceptable amount of source breakage due to the lack of `offset` and `element` tuple labels that `EnumeratedSequence` provides. + +#### Add conditional conformance to `RandomAccessCollection` for `Zip2Sequence` rather than overloading `zip`. +It isn’t possible to conditionally conform `Zip2Sequence` to `RandomAccessCollection` in a way that has optimal performance in all cases. +Consider implementing `count`. Having it return `Swift.min(self._sequence1.count, self._sequence2.count)` works fine for random-access collections but is unexpectedly slow for collections that don’t support random-access: +```swift +let evenNumbers = (0 ..< 1_000_000).lazy.filter { $0.isMultiple(of: 2) } +let zipped = zip(evenNumbers, ["lorum", "ipsum", "dolor"]) +// This would traverse the entire `0 ..< 1_000_000` range, even though the +// zipped collection only has 3 elements! +_ = zipped.count +``` +But if `count` instead naively iterated over each pair of elements and counted them along the way, then this operation would always be O(n) and no longer meet the performance requirements of the `RandomAccessCollection` protocol. +The underlying issue is that the same implementation of `count` needs to work for random-access collections as well as non-random-access collections, meeting both of their individual performance needs. +The initial version of this proposal attempted to work around this problem by adding a `_hasFastCount` customisation point to the `Collection` protocol that can be checked at runtime inside the implementation of `count`: +```swift +protocol Collection: Sequence { + // ... + var _hasFastCount: Bool { get } +} +extension Collection { + var _hasFastCount: Bool { false } +} +extension RandomAccessCollection { + var _hasFastCount: Bool { true } +} +extension Zip2Sequence: Collection + where Sequence1.Collection, Sequence2.Collection +{ + // ... + var count: Int { + if self._sequence1._hasFastCount && self._sequence2._hasFastCount { + // It's fine to access each collection's `count` here. + return Swift.min(self._sequence1.count, self._sequence2.count) + } else { + // Use some other strategy that finds the number of pairs in O(n) + // without accessing the `count` property on the underlying collections. + // ... + } + } +} +``` +However, this didn't always work as intended. When a type conditionally conforms to `RandomAccessCollection`, accessing the value’s `_hasFastCount` property in a context where it is only statically known to be a `Collection` does not invoke the default implementation defined in the `RandomAccessCollection` extension: +```swift +// `ReversedCollection` conditionally conforms to `RandomAccessCollection` +// when the base collection does. +let reversedNumbers = (0 ..< 1_000_000).reversed() +let zipped = zip(reversedNumbers, ["lorum", "ipsum", "dolor"]) +// Accidentally an O(n) operation. +_ = zipped.count +``` +In this case, the `_hasFastCount` entry in the witness table of the `Collection` conformance of `reversedNumbers` would contain the default implementation defined in the extension on `Collection` (returning `false`) rather than the one on `RandomAccessCollection` (returning `true`), due to `ReversedCollection`’s conditional conformance to `RandomAccessCollection`. As a result, `self._sequence1._hasFastCount` inside `zipped.count` would evaluate to `false`, incorrectly triggering the code path meant for non-random-access collection. +A separate `Zip2RandomAccessCollection` type does not have this problem because the underlying collections are statically known to be random-access, and therefore `Swift.min(self._sequence1.count, self._sequence2.count)` suffices. + +#### Only conform `Zip2Sequence` and `EnumeratedSequence` to `BidirectionalCollection` when the base collections conform to `RandomAccessCollection` rather than `BidirectionalCollection`. +`EnumeratedSequence` is simpler, the trade-off will be presented in terms of that type, but all of the below applies to both types equally. + +Here’s what the `Collection` conformance could look like: + +```swift +extension EnumeratedSequence: Collection where Base: Collection { + struct Index { + let base: Base.Index + let offset: Int + } + var startIndex: Index { + Index(base: _base.startIndex, offset: 0) + } + var endIndex: Index { + Index(base: _base.endIndex, offset: 0) + } + func index(after index: Index) -> Index { + Index(base: _base.index(after: index.base), offset: index.offset + 1) + } + subscript(index: Index) -> (offset: Int, element: Base.Element) { + (index.offset, _base[index.base]) + } +} + +extension EnumeratedSequence.Index: Comparable { + static func == (lhs: Self, rhs: Self) -> Bool { + return lhs.base == rhs.base + } + static func < (lhs: Self, rhs: Self) -> Bool { + return lhs.base < rhs.base + } +} +``` + +Here’s what the `Bidirectional` conformance could look like. The question is: should `Base` be required to conform to `BidirectionalCollection` or `RandomAccessCollection`? + +```swift +extension EnumeratedSequence: BidirectionalCollection where Base: ??? { + func index(before index: Index) -> Index { + let currentOffset = index.base == _base.endIndex ? _base.count : index.offset + return Index(base: _base.index(before: index.base), offset: currentOffset - 1) + } +} +``` + +Notice that calling `index(before:)` with the end index requires computing the `count` of the base collection. This is an O(1) operation if the base collection is `RandomAccessCollection`, but O(n) if it's `BidirectionalCollection`. + +##### Option 1: `where Base: BidirectionalCollection` + +A direct consequence of `index(before:)` being O(n) when passed the end index is that some operations like `last` are also O(n): + +```swift +extension BidirectionalCollection { + var last: Element? { + isEmpty ? nil : self[index(before: endIndex)] + } +} + +// A bidirectional collection that is not random-access. +let evenNumbers = (0 ... 1_000_000).lazy.filter { $0.isMultiple(of: 2) } +let enumerated = evenNumbers.enumerated() + +// This is still O(1), ... +let endIndex = enumerated.endIndex + +// ...but this is O(n). +let lastElement = enumerated.last! +print(lastElement) // (offset: 500000, element: 1000000) +``` + +However, since this performance pitfall only applies to the end index, iterating over a reversed enumerated collection stays O(n): + +```swift +// A bidirectional collection that is not random-access. +let evenNumbers = (0 ... 1_000_000).lazy.filter { $0.isMultiple(of: 2) } + +// Reaching the last element is O(n), and reaching every other element is another combined O(n). +for (offset, element) in evenNumbers.enumerated().reversed() { + // ... +} +``` + +In other words, this could make some operations unexpectedly O(n), but it’s not likely to make operations unexpectedly O(n²). + +##### Option 2: `where Base: RandomAccessCollection` + +If `EnumeratedSequence`’s conditional conformance to `BidirectionalCollection` is restricted to when `Base: RandomAccessCollection`, then operations like `last` and `last(where:)` will only be available when they’re guaranteed to be O(1): + +```swift +// A bidirectional collection that is not random-access. +let str = "Hello" + +let lastElement = str.enumerated().last! // error: value of type 'EnumeratedSequence' has no member 'last' +``` + +That said, some algorithms that can benefit from bidirectionality such as `reversed()` and `suffix(_:)` are also available on regular collections, but with a less efficient implementation. That means that the code would still compile if the enumerated sequence is not bidirectional, it would just perform worse — the most general version of `reversed()` on `Sequence` allocates an array and adds every element to that array before reversing it: + +```swift +// A bidirectional collection that is not random-access. +let str = "Hello" + +// This no longer conforms to `BidirectionalCollection`. +let enumerated = str.enumerated() + +// As a result, this now returns a `[(offset: Int, element: Character)]` instead +// of a more efficient `ReversedCollection>`. +let reversedElements = enumerated.reversed() +``` + +The base collection needs to be traversed twice either way, but the defensive approach of giving the `BidirectionalCollection` conformance a stricter bound ultimately results in an extra allocation. + +Taking all of this into account, we've gone with option 1 for the sake of giving collections access to more algorithms and more efficient overloads of some algorithms. Conforming these collections to `BidirectionalCollection` when the base collection conforms to the same protocol is less surprising. We don’t think the possible performance pitfalls pose a large enough risk in practice to negate these benefits. diff --git a/proposals/0313-actor-isolation-control.md b/proposals/0313-actor-isolation-control.md new file mode 100644 index 0000000000..72f35a9d04 --- /dev/null +++ b/proposals/0313-actor-isolation-control.md @@ -0,0 +1,363 @@ +# Improved control over actor isolation + +* Proposal: [SE-0313](0313-actor-isolation-control.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Chris Lattner](https://github.com/lattner) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Implemented (Swift 5.5)** +* Previous revision: [1](https://github.com/swiftlang/swift-evolution/blob/ca2e3b43be77b7f20303e1c5cba98f22ebb0fcb0/proposals/0313-actor-isolation-control.md) +* Implementation: Partially available in [recent `main` snapshots](https://swift.org/download/#snapshots) behind the flag `-Xfrontend -enable-experimental-concurrency` + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Proposed design](#proposed-design) + * [Actor-isolated parameters](#actor-isolated-parameters) + * [Non-isolated declarations](#non-isolated-declarations) + * [Protocol conformances](#protocol-conformances) + * [Pre-async asynchronous protocols](#pre-async-asynchronous-protocols) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) +* [Future Directions](#future-directions) + * [Multiple isolated parameters](#multiple-isolated-parameters) + * [Isolated protocol conformances](#isolated-protocol-conformances) +* [Alternatives Considered](#alternatives-considered) + * [Isolated or sync actor types](#isolated-or-sync-actor-types) +* [Revision history](#revision-history) + +## Introduction + +The [Swift actors proposal][actors] introduces the notion of *actor-isolated* declarations, which are declarations that can safely access an actor's isolated state. In that proposal, all instance methods, instance properties, and instance subscripts on an actor type are actor-isolated, and they can synchronously use those declarations on `self`. This proposal generalizes the notion of actor isolation to allow better control, including the ability to have actor-isolated declarations that aren't part of an actor type (e.g., they can be non-member functions) and have non-isolated declarations that are instance members of an actor type (e.g., because they are based on immutable, non-isolated actor state). This allows better abstraction of the use of actors, additional actor operations that are otherwise not expressible safely in the system, and enables some conformances to existing, synchronous protocols. + +## Motivation + +The actors proposal uses a simple actor `BankAccount`, which has some immutable and some mutable state in it: + +```swift +actor BankAccount { + let accountNumber: Int + var balance: Double + + init(accountNumber: Int, initialDeposit: Double) { + self.accountNumber = accountNumber + self.balance = initialDeposit + } + + func deposit(amount: Double) { + assert(amount >= 0) + balance = balance + amount + } +} +``` + +There are a few seemingly obvious things that one cannot do with this actor: + +* We can't extract an operation like `deposit(amount:)` into a global function; it can only be written as a member of the actor. +* We can't write a computed property that provides a convenient display name for a bank account instance that's usable synchronously from outside the actor. +* We can't create a `Set` because there is no way to make `BankAccount` conform to the `Hashable` protocol. + +## Proposed design + +All of the limitations described above stem from the fact that instance methods (and properties, and subscripts) on an actor type are *always* actor-isolated; no other functions can be actor-isolated and there is no way to make an instance method (etc.) not be isolated. This proposal generalizes the notion of actor-isolated functions such that any function can choose to be actor-isolated by indicating which of its actor parameters is isolated, as well as making an instance declaration on an actor not be actor-isolated at all. + +### Actor-isolated parameters + +A function can become actor-isolated by indicating that one of its parameters is `isolated`. For example, the `deposit(amount:)` operation can now be expressed as a module-scope function as follows: + +```swift +func deposit(amount: Double, to account: isolated BankAccount) { + assert(amount >= 0) + account.balance = account.balance + amount +} +``` + +Because the `account` parameter is isolated, `deposit(amount:to:)` is actor-isolated (to its `account` parameter) and can access actor-isolated state directly on that parameter. The same actor-isolation rules apply: + +```swift +extension BankAccount { + func giveSomeGetSome(amount: Double, friend: BankAccount) async { + deposit(amount: amount, to: self) // okay to call synchronously, because self is isolated + await deposit(amount: amount, to: friend) // must call asynchronously, because friend is not isolated + } +} +``` + +This makes instance methods on actor types less special, because now they are expressible in terms of a general feature: they are methods for which the `self` parameter is `isolated`, which one can see when referencing the method's curried type: + +```swift +let fn = BankAccount.deposit(amount:) // type of fn is (isolated BankAccount) -> (Double) -> Void +``` + +A given function cannot have multiple `isolated` parameters: + +```swift +func f(a: isolated BankAccount, b: isolated BankAccount) { // error: multiple isolated parameters in function `f(a:b:)`. + // ... +} + +extension BankAccount { + func quickTransfer(amount: Double, to other: isolated BankAccount) { // error: multiple isolated parameters in function 'quickTransfer(amount:to:)' + // ... + } +} +``` + +### Non-isolated declarations + +Instance declarations on an actor type implicitly have an `isolated self`. However, one can disable this implicit behavior using the `nonisolated` keyword: + +```swift +actor BankAccount { + nonisolated let accountNumber: Int + var balance: Double + + // ... +} + +extension BankAccount { + // Produce an account number string with all but the last digits replaced with "X", which + // is safe to put on documents. + nonisolated func safeAccountNumberDisplayString() -> String { + let digits = String(accountNumber) // okay, because accountNumber is also nonisolated + return String(repeating: "X", count: digits.count - 4) + String(digits.suffix(4)) + } +} + +let fn2 = BankAccount.safeAccountNumberDisplayString // type of fn is (BankAccount) -> () -> String +``` + +Note that, because `self` is not actor-isolated, `safeAccountNumberDisplayString` can only refer to non-isolated data on the actor. An attempt to refer to any actor-isolated declaration will produce an error or require asynchronous access, as appropriate: + +```swift +extension BankAccount { + nonisolated func steal(amount: Double) { + balance -= amount // error: actor-isolated property 'balance' can not be referenced on non-isolated parameter 'self' + } +} +``` + +The types involved in a non-isolated declaration must all be `Sendable`, because a non-isolated declaration can be used from any actor or concurrently-executing code. For example, one could not return a non-`Sendable` class from a `nonisolated` function: + +```swift +class SomeClass { } // not Sendable + +extension BankAccount { + nonisolated func f() -> SomeClass? { nil } // error: `nonisolated` declaration returns non-Sendable type `SomeClass?` +} +``` + +### Protocol conformances + +The actors proposal describes the rule that an actor-isolated function cannot satisfy a protocol requirement that is neither actor-isolated nor asynchronous, because doing so would allow synchronous access to actor state. However, non-isolated functions don't have access to actor state, so they are free to satisfy synchronous protocol requirements of any kind. For example, we can make `BankAccount` conform to `Hashable` by basing the hashing on the account number: + +```swift +extension BankAccount: Hashable { + nonisolated func hash(into hasher: inout Hasher) { + hasher.combine(accountNumber) + } +} + +let fn = BankAccount.hash(into:) // type is (BankAccount) -> (inout Hasher) -> Void +``` + +Similarly, one can use a `nonisolated` computed property to conform to, e.g. `CustomStringConvertible`: + +```swift +extension BankAccount: CustomStringConvertible { + nonisolated var description: String { + "Bank account #\(safeAccountNumberDisplayString())" + } +} +``` + +### Pre-`async` asynchronous protocols + +Non-isolated declarations are particularly useful for adapting existing asynchronous protocols, expressed using completion handlers, to actors. For example, consider an existing simple "server" protocol that uses a completion handler: + +```swift +protocol OldServer { + func send( + message: Message, + completionHandler: (Result) -> Void + ) +} +``` + +Over time, this protocol should evolve to provide `async` requirements. However, one can make an actor type conform to this protocol using a non-isolated declaration that launches a detached task: + +```swift +actor MyActorServer { + func send(message: Message) async throws -> Message.Reply { ... } // this is the "real" asynchronous implementation we want +} + +extension MyActorServer : OldServer { + nonisolated func send( + message: Message, + completionHandler: (Result) -> Void + ) { + detach { + do { + let reply = try await send(message: message) + completionHandler(.success(reply)) + } catch { + completionHandler(.failure(error)) + } + } + } +} +``` + +This allows actors to more smoothly integrate into existing code bases, without having to first adopt `async` throughout. + +## Source compatibility + +This proposal is additive, extending the grammar in a space where new contextual keywords are commonly introduced (declaration modifiers), so it will not affect source compatibility. + +## Effect on ABI stability + +This is purely additive to the ABI. Function parameters can be marked `isolated`, which will be captured as part of the function type. However, this (like other modifiers on a function parameter) is an additive change that won't affect existing ABI. + +## Effect on API resilience + +Nearly all changes in actor isolation are breaking changes, because the actor isolation rules require consistency between a declaration. Therefore, a parameter cannot be changed between `isolated` and non-`isolated` (either directly, or indirectly via `nonisolated`) without breaking the API. + +## Future Directions + +### Multiple `isolated` parameters + +This proposal prohibits a function declaration that has more than one `isolated` parameter. We could lift this restriction in the future, to allow code such as: + +```swift +func f(a: isolated BankAccount, b: isolated BankAccount) { + // ... +} +``` + +However, there are very few ways to call such a function in [base actors proposal][actors], because one can only run on a single actor at a time. Therefore, the only way to safely call `f` is to pass the same actor twice: + +```swift +extension BankAccount { + func g() { + f(a: self, b: self) + } + + func h(other: BankAccount) async { + await f(a: self, b: other) // error: isolated parameters `a` and `b` passed values with potentially-different actors + } +} +``` + +There are unsafe mechanisms (e.g., unsafe casting of pointer types) that could be used to pass two different actors that are both isolated. The [custom executors proposal](https://forums.swift.org/t/support-custom-executors-in-swift-concurrency/44425) provides control over the concurrency domains in which actors execute, which could be used to dynamically ensure that two actors execute in the same concurrency domain. That proposal could be modified or extended to guarantee *statically* that some set of actors share a concurrency domain to make functions with more than one `isolated` parameter more useful in the future. + +### Isolated protocol conformances + +The conformance of an actor type to a protocol assumes that the client of the protocol is outside of the actor's isolation domain. Therefore, [protocol conformances](#protocol-conformance) require either the protocol to have `async` requirements or the actor to use non-isolated members to establish protocol conformance. The [Type System Considerations for Actor Protocol](https://forums.swift.org/t/exploration-type-system-considerations-for-actor-proposal/44540) pitch argues that actor types should be able to conform to protocols with the assumption that the conformance is only used within the actor's isolation context. That pitch provides the following example: + +```swift +public protocol DataProcessible { + var data: Data { get } +} +extension DataProcessible { + func compressData() -> Data { + use(data) + /// details omitted + } +} + +actor MyDataActor : DataProcessible { + // error: cannot fulfill sync requirement with isolated actor member. + var data: Data + + func doThing() { + // All sync, no problem! + let compressed = compressData() + } +} +``` + +That pitch suggests that the conformance of `MyDataActor : DataProcessible` be permitted, and introduces the notion of a `@sync` actor type to describe the actor when in its own isolation domain. Specifically, the type `@sync MyDataActor` conforms to `DataProcessible` but the type `@async MyDataActor` (which represents the actor outside of its isolation domain) does not. + +This proposal does not separate isolated from non-isolated actor types, and instead uses an `isolated` parameter to describe the actor that the code is executing on. The same notion can be extended to introduce isolated protocol conformances, which are conformances that can only be used with isolated values. For example, the conformance itself could have `isolated` applied to it to mark it as an isolated conformance: + +```swift +actor MyDataActor : isolated DataProcessible { + var data: Data // okay: satisfies "data" requirement + + func doThing() { + // okay, because self is isolated + let compressed = compressData() + } + + nonisolated failToDoTheThing() { + // error: isolated conformance MyDataActor : DataProcessible cannot be used when non-isolated + // value of type MyDataActor is passed to the generic function. + let compressed = compressData() + } +} +``` + +The use of isolated protocol conformances would require a number of other restrictions to ensure that the protocol conformance cannot be used on non-isolated instances of the actor. For example, this means that a non-isolated conformance can never be used along with `Sendable` on the same type, because that would permit a non-isolated instance of the actor to be passed outside of the actor's isolation domain along with a protocol conformance that assumes it is within the actor's isolation domain. + +## Alternatives Considered + +### Isolated or sync actor types + +The notion of "isolated" parameters grew out of a [proposal](https://forums.swift.org/t/exploration-type-system-considerations-for-actor-proposal/44540) that generalized the notion of actor isolation from something that only made sense on `self` to one that made sense for any parameter. That proposal modeled isolation directly in the type system by introducing a new kind of type: `@sync` actor types were used for values that have synchronous access to the actors they describe. Therefore, instead of saying that `self` is an `isolated` parameter of type `MyActor`, the proposal would say that `self` has the type `@sync MyActor`. The "isolated conformances" described in the future directions above are similar to (and directly influenced by) the notion that `@sync` actor types can conform to (synchronous) protocols as described in that proposal. + +At a high level, isolated parameters and isolated conformances are similar to parameters of `@sync` type and conformances of `@sync` types to protocols, and can address similar sets of use cases. This proposal chose to treat `isolated` as a parameter modifier rather than as a type because it provides a simpler, value-centric model that aligns more closely with the behavior of a similarly-constrained construct, `inout`. There are several inconsistencies to the `@sync` type approach that made it less desirable: + +* The type of an actor's `self` can change within nested contexts, such as closures, between `@sync` and non-`@sync`: + + ```swift + func f(_: T) { } + + actor MyActor { + func g() { + f(self) // T = @sync MyActor + + asyncDetached { + f(self) // T = MyActor + } + } + } + ``` + Generally speaking, a variable in Swift has the same type when it's captured in a nested context as it does in its enclosing context, which provides a level of predictability that would be lost with `@sync` types. In the example above, type inference for the call to `f` differs significantly whether you're in the closure or not. A [recent discussion on the forums](https://forums.swift.org/t/implicit-casts-for-verified-type-information/41035) about narrowing types showed resistance to the idea of changing the type of a variable in a nested context, even when doing so could eliminate additional boilerplate. + +* The design relies heavily on the implicit conversion from `@sync MyActor` to `MyActor`, e.g., + + ```swift + func acceptActor(_: MyActor) { } + func acceptSendable(_: T) { } + + extension MyActor { + func h() { + acceptActor(h) // okay, requires conversion of @sync MyActor to MyActor + acceptSendable(h) // okay, requires T=MyActor and conversion of @sync MyActor to MyActor + } + } + ``` + +* Conformance to `Sendable` doesn't follow the normal subtyping rules. Per the conversion above, a `@sync` actor type is a subtype of the corresponding (non-`@sync`) actor type. By definition, a subtype has all of the conformances of its supertype, and may of course add more capabilities. This is a general principle of type system design, and shows up in Swift in a number of places, e.g., with subclassing: + + ```swift + protocol P { } + + class C: P { } + class D: C { } + + func test(c: C, d: D) { + let _: P = c // okay, C conforms to P + let _: P = d // okay, D conforms to P because it is a subtype of C, which itself conforms to P + } + ``` + + However, `@sync` types don't behave this way with respect to `Sendable`. A non-`@sync` actor type conforms to `Sendable` (it's safe to share it across concurrency domains), but its corresponding `@sync` subtype does *not* conform to `Sendable`. This is why in the prior example's call to `acceptSendable`, the implicit conversion from `@sync MyActor` to `MyActor` is required. + +## Revision history + +* Changes in the accepted version of this proposal: + * Removed `isolated` captures. + * Prohibit multiple `isolated` parameters. + +[actors]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md diff --git a/proposals/0314-async-stream.md b/proposals/0314-async-stream.md new file mode 100644 index 0000000000..3abb7024f4 --- /dev/null +++ b/proposals/0314-async-stream.md @@ -0,0 +1,554 @@ +# `AsyncStream` and `AsyncThrowingStream` + +* Proposal: [SE-0314](0314-async-stream.md) +* Authors: [Philippe Hausler](https://github.com/phausler), [Tony Parker](https://github.com/parkera), [Ben D. Jones](https://github.com/bendjones), [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.5)** +* Review: ([first review](https://forums.swift.org/t/se-0314-asyncstream-and-asyncthrowingstream/48198), [revision announcement](https://forums.swift.org/t/returned-for-revision-se-0314-asyncstream-and-asyncthrowingstream/49718), [second review](https://forums.swift.org/t/se-0314-second-review-asyncstream-and-asyncthrowingstream/49803), [acceptance announcement](https://forums.swift.org/t/accepted-se-0314-asyncstream-and-asyncthrowingstream/50699)) +* Implementation: [apple/swift#36921](https://github.com/apple/swift/pull/36921) + +#### Change Log + +Changes for the second review: +* added `YieldResult` to express the action of yielding’s impact, either something is enqueued, dropped or the continuation is already terminated +* added `init(unfolding: @escaping () async -> Element?)` to offer an initializer for unfolding to handle back-pressure based APIs. +* made `AsyncThrowingStream` generic on Failure but the initializers only afford for creation `where Failure == Error` +* removed the example of `DispatchSource` signals since the other `DispatchSource` types might be actively harmful to use in *any* async context +* initialization now takes a buffering policy to both restrict the buffer size as well as configure how elements are dropped + +## Introduction + +The continuation types added in [SE-0300](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0300-continuation.md) act as adaptors for synchronous code that signals completion by calling a delegate method or callback function. For code that instead yields multiple values over time, this proposal adds new types to support implementing an `AsyncSequence` interface. + +Swift-evolution threads: + +- [[Pitch] AsyncStream and AsyncThrowingStream](https://forums.swift.org/t/pitch-asyncstream-and-asyncthrowingstream/47820) +- [[Concurrency] YieldingContinuation](https://forums.swift.org/t/concurrency-yieldingcontinuation/47126) + +## Motivation + +Swift’s new `async` / `await` features include support for adapting callback- or delegate-based APIs using `UnsafeContinuation` or `CheckedContinuation` . These single-use continuations work well for adapting APIs like the `getInt(completion:)` function defined here into asynchronous ones: + +```swift +func getInt(completion: @escaping (Int) -> Void) { + DispatchQueue(label: "myQueue").async { + sleep(1) + completion(42) + } +} +``` + +By calling one of the `with``*Continuation(_:)` functions, you can suspend the current task and resume it with a result or an error using the provided continuation. + +```swift +func getInt() async -> Int { + await withUnsafeContinuation { continuation in + getInt(completion: { result in + continuation.resume(returning: result) + } + } +} +``` + +This provides a great experience for APIs that asynchronously produce a single result, but some operations produce many values over time instead. Rather than being adapted to an `async` function, the appropriate solution for these operations is to create an `AsyncSequence` . + +Repeating asynchronous operations typically separate the consumption of values from their location of use, either within a callback function or a delegate method. Given an existing API that offers an interface as the following: + +```swift +class QuakeMonitor { + var quakeHandler: (Quake) -> Void + func startMonitoring() + func stopMonitoring() +} +``` + +The usage of this pattern would work similarly to this: + +```swift +let monitor = QuakeMonitor() +monitor.quakeHandler { quake in + // ... +} +monitor.startMonitoring() // start sending quakes to the handler +... +monitor.stopMonitoring() // cancel the quakes being sent to the handler +``` + +The same is true for delegates that are informative only, and need no feedback or exclusivity of execution to be valid. As one example, the AppKit [ `NSSpeechRecognizerDelegate` ](https://developer.apple.com/documentation/appkit/nsspeechrecognizerdelegate) is called whenever the system recognizes a spoken command: + +```swift +if let recognizer = NSSpeechRecognizer() { + class CommandDelegate: NSObject, NSSpeechRecognizerDelegate { + func speechRecognizer( + _ sender: NSSpeechRecognizer, + didRecognizeCommand command: String) + { + // do something on each recognized command + } + } + let delegate = CommandDelegate() + recognizer.delegate = delegate + ... +} +``` + +Both of these examples represent common design patterns in many applications, frameworks, and libraries. These delegate methods and callbacks are asynchronous in nature, but cannot be annotated as `async` functions because they don’t have a singular return value. While not all delegates or callbacks are suitable to be represented as asynchronous sequences, there are numerous enough cases that we would like to offer a safe and expressive way to represent producers that can yield multiple values or errors. + +## Proposed Solution + +In order to fill this gap, we propose adding two new types: `AsyncStream` and `AsyncThrowingStream` . These types fill a role similar to continuations, bridging the gap from non `async` / `await` based asynchronous behavior into the world of `async` / `await` . We anticipate that types that currently provide multiple-callback or delegate interfaces to asynchronous behavior can use these `AsyncStream` types to provide an interface for within `async` contexts. + +The two `AsyncStream` types each include a nested `Continuation` type; these outer and inner types represent the consuming and producing sides of operation, respectively. You send values, errors, and “finish” events via the `Continuation` , and clients consume those values and errors through the `AsyncStream` type’s `AsyncSequence` interface. + +### Creating a non-throwing `AsyncStream` + +When you create an `AsyncStream` instance, you specify the element type and pass a closure that operates on the series’s `Continuation` . You can yield values to this continuation type *multiple* times, and the series buffers any yielded elements until they are consumed via iteration. + +The `QuakeMonitor` above can be given an `AsyncStream` interface this way: + +```swift +extension QuakeMonitor { + static var quakes: AsyncStream { + AsyncStream { continuation in + let monitor = QuakeMonitor() + monitor.quakeHandler { quake in + continuation.yield(quake) + } + continuation.onTermination = { _ in + monitor.stopMonitoring() + } + monitor.startMonitoring() + } + } +} + +// elsewhere... + +for await quake in QuakeMonitor.quakes { + // ... +} +``` + +As each value is passed to the `QuakeMonitor` ’s event handler closure, the call to `continuation.yield(_:)` stores the value for access by a consumer of the sequence. With this implementation, quake events are buffered as they come in, and only consumed when an iterator requests a value. + +Alternatively if a source is just an async function (one that represents a backpressure) an AsyncStream can be constructed by unfolding a producing function and a cancellation handler. This affords the case in which that unfolded function can leave the specification adherence of AsyncSequence to AsyncStream. In short the `init(unfolding:onCancel:)` initializer handles the terminal cases as well as the cancellation. + +### Creating an `AsyncThrowingStream` + +Along with the potentially infinite sequence in the example above, `AsyncThrowingStream` can also adapt APIs like the slightly contrived one below. The `findVegetables` function uses callback closures that are called with each retrieved vegetable, as well as when the vegetables have all been returned or an error occurs. + +```swift +func buyVegetables( + shoppingList: [String], + + // a) invoked once for each vegetable in the shopping list + onGotVegetable: (Vegetable) -> Void, + + // b) invoked once all available veggies have been retrieved + onAllVegetablesFound: () -> Void, + + // c) invoked if a non-vegetable food item is encountered + // in the shopping list + onNonVegetable: (Error) -> Void +) + +// Returns a stream of veggies +func findVegetables(shoppingList: [String]) -> AsyncThrowingStream { + AsyncThrowingStream { continuation in + buyVegetables( + shoppingList: shoppingList, + onGotVegetable: { veggie in continuation.yield(veggie) }, + onAllVegetablesFound: { continuation.finish() }, + onNonVegetable: { error in continuation.finish(throwing: error) } + ) + } +} +``` + +Note that a call to the `finish()` method is required to end iteration for the consumer of an `AsyncStream` . Any buffered elements are provided to the sequence consumer before finishing with either a simple terminating `nil` or a thrown error. + +### Awaiting Values + +An `AsyncStream` provides an `AsyncSequence` interface to its values, so you can iterate over the elements in an `AsyncStream` by using `for` - `in` , or use any of the `AsyncSequence` methods added as part of [SE-0298](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0298-asyncsequence.md). + +```swift +for await notif in NotificationCenter + .notifications(for: ...) + .prefix(3) +{ + // update with notif +} +``` + +You may also create an iterator directly, and call its `next()` method for more control over iteration. Each call to `next()` , either through `for` - `in` iteration, an `AsyncSequence` method, or direct calls on an iterator, either immediately returns the earliest buffered element or `await` s the next element yielded to the stream’s continuation. + +As with any sequence, iterating over an `AsyncStream` multiple times, or creating multiple iterators and iterating over them separately, may produce an unexpected series of values. Neither `AsyncStream` nor its iterator are `@Sendable` types, and concurrent iteration is considered a programmer error. + +## Detailed design + +The full API of `AsyncStream` , `AsyncThrowingStream` , and their nested `Continuation` and `Iterator` types are as follows: + +```swift +/// An ordered, asynchronously generated sequence of elements. +/// +/// AsyncStream is an interface type to adapt from code producing values to an +/// asynchronous context iterating them. This is intended to allow +/// callback or delegation based APIs to participate with async/await. +/// +/// AsyncStream can be initialized with the option to buffer to a given limit. +/// The default value for this limit is Int.max. The buffering is only for +/// values that have yet to be consumed by iteration. Values can be yielded +/// to the continuation passed into the closure. That continuation +/// is Sendable, in that it is intended to be used from concurrent contexts +/// external to the iteration of the stream. +/// +/// A trivial use case producing values from a detached task would work as such: +/// +/// let digits = AsyncStream(Int.self) { continuation in +/// detach { +/// for digit in 0..<10 { +/// continuation.yield(digit) +/// } +/// continuation.finish() +/// } +/// } +/// +/// for await digit in digits { +/// print(digit) +/// } +/// +public struct AsyncStream { + public struct Continuation: Sendable { + /// Indication of the type of termination informed to + /// `onTermination`. + public enum Termination { + + /// The stream was finished via the `finish` method + case finished + + /// The stream was cancelled + case cancelled + } + + /// A result of yielding values. + public enum YieldResult { + + /// When a value is successfully enqueued, either buffered + /// or immediately consumed to resume a pending call to next + /// and a count of remaining slots available in the buffer at + /// the point in time of yielding. Note: transacting upon the + /// remaining count is only valid when then calls to yield are + /// mutually exclusive. + case enqueued(remaining: Int) + + /// Yielding resulted in not buffering an element because the + /// buffer was full. The element is the dropped value. + case dropped(Element) + + /// Indication that the continuation was yielded when the + /// stream was already in a terminal state: either by cancel or + /// by finishing. + case terminated + } + + /// A strategy that handles exhaustion of a buffer’s capacity. + public enum BufferingPolicy { + case unbounded + + /// When the buffer is full, discard the newly received element. + /// This enforces keeping the specified amount of oldest values. + case bufferingOldest(Int) + + /// When the buffer is full, discard the oldest element in the buffer. + /// This enforces keeping the specified amount of newest values. + case bufferingNewest(Int) + } + + /// Resume the task awaiting the next iteration point by having it return + /// normally from its suspension point. Buffer the value if nothing is awaiting + /// the iterator. + /// + /// - Parameter value: The value to yield from the continuation. + /// + /// This can be called more than once and returns to the caller immediately + /// without blocking for any awaiting consumption from the iteration. + /// + /// The `yield(_:)` function returns the state of any value yielded to the + /// continuation. This can be one of three states: `enqueued`, `dropped` or + /// `terminated`. Each of the states respectively represents if the value + /// was either buffered or resumed to active iteration, dropped because + /// the limit of the buffer was reached, or dropped because the AsyncStream + /// was at a terminal state either from being finished or cancelled. + @discardableResult + public func yield(_ value: Element) -> YieldResult + + /// Resume the task awaiting the next iteration point by having it return + /// nil. This signifies the end of the iteration. + /// + /// Calling this function more than once is idempotent. All values received + /// from the iterator after it is finished and after the buffer is exhausted + /// are nil. + public func finish() + + /// A callback to invoke when iteration of a AsyncStream is canceled. + /// + /// If an onTermination callback is set, when iteration of an AsyncStream is + /// canceled via task cancellation that callback is invoked. The callback + /// is disposed of after any terminal state is reached. + /// + /// Canceling an active iteration will first invoke the onCancel callback + /// and then resume yielding nil. This means that any cleanup state can be + /// emitted accordingly in the cancellation handler. + public var onTermination: (@Sendable (Termination) -> Void)? { get nonmutating set } + } + + /// Construct an AsyncStream buffering given an Element type. + /// + /// - Parameter elementType: The type the AsyncStream will produce. + /// - Parameter bufferingPolicy: The policy in which to buffer elements. + /// This controls the amount that can potentially be stored in the buffer + /// and the mechanism in which to drop values. + /// - Parameter build: The work associated with yielding values to the + /// AsyncStream. + /// + /// The maximum number of pending elements is enforced by dropping the oldest + /// value when a new value comes in. By default this limit is unlimited. + /// A value of 0 results in immediate dropping of values if there is no current + /// await on the iterator. + /// + /// The build closure passes in a Continuation which can be used in + /// concurrent contexts. It is thread safe to yield and finish. All calls are + /// to the continuation are serialized. However, calling yield from multiple + /// concurrent contexts could result in out of order delivery. + public init( + _ elementType: Element.Type = Element.self, + bufferingPolicy limit: Continuation.BufferingPolicy = .unbounded, + _ build: (Continuation) -> Void + ) { + + /// Construct an AsyncStream by unfolding the application of a function. + /// + /// - Parameter produce: The function to call when calculating the next value. + /// - Parameter onCancel: A closure to call when the AsyncStream is cancelled. + /// + /// Construction with this initializer handles the rules of AsyncSequence in + /// that after a nil is produced subsequent calls must produce nil. + public init( + unfolding produce: @escaping () async -> Element?, + onCancel: (@Sendable () -> Void)? = nil + ) +} + +extension AsyncStream: AsyncSequence { + /// The asynchronous iterator for iterating an AsyncStream. + /// + /// This type is not Sendable. It is not intended to be used + /// from multiple concurrent contexts. Any such case that next is invoked + /// concurrently and contends with another call to next is a programmer error + /// and will fatalError. + public struct Iterator: AsyncIteratorProtocol { + public mutating func next() async -> Element? + } + + /// Construct an iterator. + public func makeAsyncIterator() -> Iterator +} + +extension AsyncStream.Continuation { + /// Resume the task awaiting the next iteration point by having it return + /// normally from its suspension point or buffer the value if no awaiting + /// next iteration is active. + /// + /// - Parameter result: A result to yield from the continuation. + /// + /// This can be called more than once and returns to the caller immediately + /// without blocking for any awaiting consumption from the iteration. + @discardableResult + public func yield( + with result: Result + ) -> YieldResult + + /// Resume the task awaiting the next iteration point by having it return + /// normally from its suspension point or buffer the value if no awaiting + /// next iteration is active where the `Element` is `Void`. + /// + /// This can be called more than once and returns to the caller immediately + /// without blocking for any awaiting consumption from the iteration. + @discardableResult + public func yield() -> YieldResult where Element == Void +} + +public struct AsyncThrowingStream { + public struct Continuation: Sendable { + public enum Termination { + case finished(Failure?) + case cancelled + } + + public enum YieldResult { + case enqueued + case dropped + case terminated + } + + /// * See AsyncStream.Continuation.yield(_:) * + @discardableResult + public func yield(_ value: Element) -> YieldResult + + /// Resume the task awaiting the next iteration point with a terminal state. + /// If error is nil, this is a completion with out error. If error is not + /// nil, then the error is thrown. Both of these states indicate the + /// end of iteration. + /// + /// - Parameter error: The error to throw or nil to signify termination. + /// + /// Calling this function more than once is idempotent. All values received + /// from the iterator after it is finished and after the buffer is exhausted + /// are nil. + public func finish(throwing error: Failure? = nil) + + /// * See AsyncStream.Continuation.onTermination * + public var onTermination: (@Sendable (Termination) -> Void)? { get nonmutating set } + } + + /// * See AsyncStream.init * + public init( + _ elementType: Element.Type, + maxBufferedElements limit: Int = .max, + _ build: (Continuation) -> Void + ) where Failure == Error + + /// Construct an AsyncStream by unfolding the application of a function. + /// + /// - Parameter produce: The function to call when calculating the next value. + /// - Parameter onCancel: A closure to call when the AsyncStream is cancelled. + /// + /// Construction with this initializer handles the rules of AsyncSequence in + /// that after a nil is produced subsequent calls must produce nil. + public init( + unfolding produce: @escaping () async throws -> Element?, + onCancel: (@Sendable () -> Void)? = nil + ) +} + +extension AsyncThrowingStream: AsyncSequence { + public struct Iterator: AsyncIteratorProtocol { + public mutating func next() async throws -> Element? + } + + public func makeAsyncIterator() -> Iterator +} + +extension AsyncThrowingStream.Continuation { + /// Resume the task awaiting the next iteration point by having it return + /// normally from its suspension point or buffer the value if no awaiting + /// next iteration is active. + /// + /// - Parameter result: A result to yield from the continuation. + /// + /// This can be called more than once and returns to the caller immediately + /// without blocking for any awaiting consumption from the iteration. + @discardableResult + public func yield( + with result: Result + ) -> YieldResult + + /// * See AsyncStream.yield() * + @discardableResult + public func yield() -> YieldResult where Element == Void +} +``` + +### Yielding Values + +In some cases it is meaningful to manage the potential emissions in accordance with what the state of yielding values to the continuation might do. There are three potential states of yielding a value, it may be enqueued to the buffer (or even immediately resumed to an active iterator), dropped because the limit of the buffer has been reached, or dropped because the `AsyncStream` or `AsyncThrowingStream` have been terminated (either by finish or cancel). This is a meaningful return value in some cases but generally this is not always a meaningful/useful state. In the cases that it is meaningful; consumers may want to emit an error when the buffer has reached its limit. Or consumers may want to concatenate the value to the next emitted value. + +The yield function of the `Continuation` type for both `AsyncStream` and `AsyncThrowingStream` returns the state in which that yield transacted - it can return that the value was enqueued with a remaining count of slots available at the time of the yield for the backing buffer, or if the buffer is full it will return the element that was dropped, or if the stream was already terminal it returns an indication of that terminal state. + +### Buffering Values + +By default, every element yielded to an `AsyncStream` ’s continuation is buffered until consumed by iteration. This matches the expectation for most of the APIs we anticipate being adapted via `AsyncStream` — with a stream of notifications, database records, or other similar types, the caller needs to receive every single one. + +If the caller specifies a different value *n* for `maxBufferedElements` , then the most recent *n* elements are buffered until consumed by iteration. If the caller specifies `0` , the stream switches to a dropping behavior, dropping the value if nothing is `await` ing the iterator’s `next` . + +### Backpressure + +`AsyncStream` and `AsyncThrowingStream` both are types intended to interface systems in which back pressure is not present to the `AsyncSequence` interface which is a back pressure based system. `AsyncSequence` via it's iterator is back pressure based via the `next` method on the iterator. Each call to next is an asynchronous call that represents an applied demand of 1. This means that systems in which back pressure is not present; like callbacks that are called more than once, or some "informative" style delegates there must be some intermediary to offer behavior of either buffering, dropping or blocking. These three options are the only available mechanisms to stride that gap between the back pressure world and the non back pressure world. `AsyncStream` does not aim to resolve the blocking scenario (any callbacks that require that type of functionality are probably ill suited for async/await anyhow). The buffering and dropping scenarios can be represented with just buffering since the dropping scenario is just a buffer of 0 items. + +### Finishing the Stream + +Calling a continuation’s `finish()` method moves its stream into a “terminated” state. An implementor of an `AsyncThrowingStream` can optionally pass an error to be thrown by the iterator. After providing all buffered elements, the stream’s iterator will return `nil` or throw an error, as appropriate. + +The first call to `finish()` sets the terminating behavior of the stream (either returning `nil` or throwing); further calls to `finish()` or `yield(_:)` once a stream is in a terminated state have no effect. + +### Cancellation Handlers + +When defined, a continuation’s `onTermination` handler function is called when iteration ends, when the stream goes out of scope, or when the task containing the stream is canceled. You can safely use the `onTermination` handler to clean up resources that were opened or allocated at the start of the stream. + +This `onTermination` behavior is shown in the following example — once the task containing the stream is canceled, the `onTermination` handler for the stream’s continuation is called: + +```swift +let t = detach { + func make123Stream() -> AsyncStream { + AsyncStream { continuation in + continuation.onTermination = { termination in + switch termination { + case .finished: + print("Regular finish") + case .cancelled: + print("Cancellation") + } + } + detach { + for n in 1...3 { + continuation.yield(n) + sleep(2) + } + continuation.finish() + } + } + } + + for await n in make123Stream() { + print("for-in: \(n)") + } + print("After") +} +sleep(3) +t.cancel() + +// for-in: 1 +// for-in: 2 +// Cancellation +// After +``` + +### Convenience Methods + +As conveniences, both `AsyncStream` continuations include a `yield(with:)` method that takes a `Result` as a parameter and, when the stream’s `Element` type is `Void` , a `yield()` method that obviates the need for passing an empty tuple. + +## Alternatives considered + +### YieldingContinuation + +A `YieldingContinuation` type was pitched previously in [this forum thread](https://forums.swift.org/t/concurrency-yieldingcontinuation/47126), designed as a single type that could be yielded to and awaited on. `AsyncStream` adds automatic buffering and a simpler API to support most common use cases. + +### Users use `os_unfair_lock` / `unlock` or other locking primitives + +The buffering behavior in `AsyncStream` requires the use of locks in order to be thread safe. Without a native locking mechanism, Swift developers would be left to write their own custom (and potentially per-platform) locks. We think it is better for a standard library solution to be provided that removes this complexity and potential source of errors. + +## Source compatibility + +This proposal is purely additive and has no direct impact on existing source. + +## Effect on ABI stability + +This proposal is purely additive and has no direct impact on ABI stability. + +## Effect on API resilience + +The current implementation leaves room for future development of this type to offer different construction mechanisms and is encapsulated into it's own type hierarchy so it has no immediate impact upon API resilience and is future-proofed for future development. + +## Acknowledgments + +We would like to thank [Jon Shier](https://forums.swift.org/u/jon_shier) and [David Nadoba](https://forums.swift.org/u/dnadoba) for their helpful insight and feedback driving the discussion on YieldingContinuation to make us look further into making a more well rounded solution. Special thanks go to the creators of the [reactive-streams specification ](https://github.com/reactive-streams/reactive-streams-jvm/blob/v1.0.3/README.md#specification), without which numerous behavioral edge cases would not have been considered. diff --git a/proposals/0315-placeholder-types.md b/proposals/0315-placeholder-types.md new file mode 100644 index 0000000000..a52173e1f4 --- /dev/null +++ b/proposals/0315-placeholder-types.md @@ -0,0 +1,352 @@ +# Type placeholders (formerly, "Placeholder types") + +* Proposal: [SE-0315](0315-placeholder-types.md) +* Authors: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.6)** +* Implementation: [apple/swift#36740](https://github.com/apple/swift/pull/36740) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0315-placeholder-types/50671) + +Note: this feature was originally discussed and accepted under the "placeholder types" title. The official terminology for this feature is "type placeholders," so the terminology in this proposal has been updated accordingly. + +## Introduction + +When Swift's type inference is unable to work out the type of a particular expression, it requires the programmer to provide the necessary type context explicitly. However, all mechanisms for doing this require the user to write out the entire type signature, even if only one portion of that type is actually needed by the compiler. E.g., + +```swift +let losslessStringConverter = Double.init as (String) -> Double? + +losslessStringConverter("42") //-> 42.0 +losslessStringConverter("##") //-> nil +``` + +In the above example, we only really need to clarify the *argument* type—there's only one `Double.init` overload that accepts a `String`. This proposal allows the user to provide type hints which use *type placeholders* in such circumstances, so that the initialization of `stringTransform` could be written as: + +```swift +let losslessStringConverter = Double.init as (String) -> _? +``` + +Swift-evolution threads: [Partial type annotations ](https://forums.swift.org/t/partial-type-annotations/41239), [Placeholder types](https://forums.swift.org/t/placeholder-types/41329) + +## Motivation + +Swift's type inference system is quite powerful, but there are many situations where it is impossible (or simply infeasible) for the compiler to work out the type of an expression, or where the user needs to override the default types worked out by the compiler. Directly referencing the heavily-overloaded `Double.init` initializer, as seen above, is one such situation where the compiler does not have the necessary context to determine the type of the expression without additional context. + +Fortunately, Swift provides several ways for the user to provide type information explicitly. Common forms are: + +* Variable type annotations: + +```swift +let losslessStringConverter: (String) -> Double? = Double.init +``` + +* Type coercion via `as` (seen above): + +```swift +let losslessStringConverter = Double.init as (String) -> Double? +``` + +* Passing type parameters explicitly (e.g., `JSONDecoder` ): + +```swift +let dict = try JSONDecoder().decode([String: Int].self, from: data) +``` + +The downside of all of these approaches is that they require the user to write out the *entire* type, even when the compiler only needs guidance on some sub-component of that type. This can become particularly problematic in cases where a complex type that would normally be inferred has to be written out explicitly because some *unrelated* portion of the type signature is required. E.g., + +```swift +enum Either { + case left(Left) + case right(Right) + + init(left: Left) { self = .left(left) } + init(right: Right) { self = .right(right) } +} + +func makePublisher() -> Some>>>> { ... } +``` + +Attempting to initialize an `Either` from `makePublisher` isn't as easy as one might like: + +```swift +let publisherOrValue = Either(left: makePublisher()) // Error: generic parameter 'Right' could not be inferred +``` + +Instead, we have to write out the full generic type: + +```swift +let publisherOrValue = Either>>>>, Int>(left: makePublisher()) +``` + +The resulting expression is more difficult to write *and* read. If `Left` were the result of a long chain of Combine operators, the author may not even know the correct type to write and would have to glean it from several pages of documentation or compiler error messages. + +## Proposed solution + +Allow users to write types with designated *type placeholders* (spelled " `_` ") which indicate that the corresponding type should be filled in during type checking. For the above `publisherOrValue` example, this would look like: + +```swift +let publisherOrValue = Either<_, Int>(left: makePublisher()) +``` + +Because the generic argument to the `Left` parameter can be inferred from the return type of `makePublisher` , we do not need to write it out. Instead, during type checking, the compiler will see that the first generic argument to `Either` is a placeholder and leave it unresolved until other type information can be used to fill it in. + +## Detailed design + +### Grammar + +This proposal introduces the concept of a user-specified "type placeholder," which, in terms of the grammar, can be written anywhere a type can be written. In particular, the following productions will be introduced: + +```swift +type → placeholder-type +placeholder-type → '_' +``` + +Examples of types containing placeholders are: + +```swift +Array<_> // array with placeholder element type +[Int: _] // dictionary with placeholder value type +(_) -> Int // function type accepting a single type placeholder argument and returning 'Int' +(_, Double) // tuple type of placeholder and 'Double' +_? // optional wrapping a type placeholder +``` + +### Type inference + +When the type checker encounters a type containing a type placeholder, it will fill in all of the non-placeholder context exactly as before. Type placeholders will be treated as providing no context for that portion of the type, requiring the rest of the expression to be solvable given the partial context. Effectively, type placeholders act as user-specified anonymous type variables that the type checker will attempt to solve using other contextual information. + +Let's examine a concrete example: + +```swift +import Combine + +func makeValue() -> String { "" } +func makeValue() -> Int { 0 } + +let publisher = Just(makeValue()).setFailureType(to: Error.self).eraseToAnyPublisher() +``` + +As written, this code is invalid. The compiler complains about the "ambiguous use of `makeValue()` " because it is unable to determine which `makeValue` overload should be called. We could solve this by providing a full type annotation: + +```swift +let publisher: AnyPublisher = Just(makeValue()).setFailureType(to: Error.self).eraseToAnyPublisher() +``` + +Really, though, this is overkill. The generic argument to `AnyPublisher` 's `Failure` parameter is clearly `Error` , since the result of `setFailureType(to:)` has no ambiguity. Thus, we can substitute in a type placeholder for the `Failure` parameter, and still successfully typecheck this expression: + +```swift +let publisher: AnyPublisher = Just(makeValue()).setFailureType(to: Error.self).eraseToAnyPublisher() +``` + +Now, the type checker has all the information it needs to resolve the reference to `makeValue` : the ultimately resulting `AnyPublisher` must have `Output == Int` , so the result of `setFailureType(to:)` must have `Output == Int` , so the instance of `Just` must have `Output == Int` , so the argument to `Just.init` must have type `Int` , so `makeValue` must refer to the `Int` -returning overload! + +Note: it is not permitted to specify a type that is _just_ a placeholder—see the relevant subsection in **Future directions** for a discussion of the considerations. This means that, for example, the following would fail to compile: + +```swift +let percent: _ = 100.0 // error: placeholders are not allowed as top-level types +``` + +### Generic constraints + +In some cases, placeholders may be expected to conform to certain protocols. E.g., it is perfectly legal to write: + +```swift +let dict: [_: String] = [0: "zero", 1: "one", 2: "two"] +``` + +When examining the storage type for `dict` , the compiler will expect the key type placeholder to conform to `Hashable` . Conservatively, type placeholders are assumed to satisfy all necessary constraints, deferring the verification of these constraints until the checking of the initialization expression. + +### Generic parameter inference + +A limited version of this feature is already present in the language via generic parameter inference. When the generic arguments to a generic type can be inferred from context, you are permitted to omit them, like so: + +```swift +import Combine + +let publisher = Just(0) // Just is inferred! +``` + +With type placeholders, writing the bare name of a generic type (in most cases, see note below) becomes equivalent to writing the generic signature with type placeholders for the generic arguments. E.g., the initialization of `publisher` above is the same as: + +```swift +let publisher = Just<_>(0) +``` + +Note: there is an existing rule that *inside the body* of a generic type `S` , the bare name `S` is equivalent to `S` . This proposal does not augment this rule nor attempt to express this rule in terms of type placeholders. + +### Function signatures + +As is the case today, function signatures under this proposal are required to have their argument and return types fully specified. Generic parameters cannot be inferred and type placeholders are not permitted to appear within the signature, even if the type could ostensibly be inferred from e.g., a protocol requirement or default argument expression. + +Thus, it is an error under this proposal to write something like: + +```swift +func doSomething(_ count: _? = 0) { ... } +``` + +just as it would be an error to write: + +```swift +func doSomething(_ count: Optional = 0) { ... } +``` + +even though the type checker could infer the `Wrapped` type in an expression like: + +```swift +let count: _? = 0 +``` + +As a more comprehensive example, consider the following setup: + +```swift +struct Bar +where T: ExpressibleByIntegerLiteral, U: ExpressibleByIntegerLiteral { + var t: T + var u: U +} + +extension Bar { + func frobnicate() -> Bar { + return Bar(t: 42, u: 42) + } + func frobnicate2() -> Bar<_, _> { // error + return Bar(t: 42, u: 42) + } + func frobnicate3() -> Bar { + return Bar<_, _>(t: 42, u: 42) + } + func frobnicate4() -> Bar<_, _> { // error + return Bar<_, _>(t: 42, u: 42) + } + func frobnicate5() -> Bar<_, U> { // error + return Bar(t: 42, u: 42) + } + func frobnicate6() -> Bar { + return Bar<_, U>(t: 42, u: 42) + } + func frobnicate7() -> Bar<_, _> { // error + return Bar<_, U>(t: 42, u: 42) + } + func frobnicate8() -> Bar<_, U> { // error + return Bar<_, _>(t: 42, u: 42) + } +} +``` + +Under this proposal, only `frobnicate`, `frobnicate3` and `frobnicate6` would compile without error (`frobnicate`, of course, compiles without this proposal as well), since all others have placeholders appearing in at least one position in the function signature. + +### Dynamic casts + +In dynamic casts, unlike `as` coercions, there is no inherent relationship between the casted expression and the cast type. This is why we can write things like `0 as? String` or `[""] is Double` (albeit, with warnings that the casts will always fail). + +While this proposal does not *explicitly* disallow type placeholders in `is`, `as?`, and `as!` casts, it provides for no additional inference rules for matching the type of the casted expression to the cast type, meaning that in most cases type placeholders will fail to type check if used in these positions (e.g., `0 as? [_]`). + +This also applies to `is` and `as` patterns (e.g., `case let y as [_]`). + +## Source compatibility + +This is an additive change with no effect on source compatibility. Certain invalid code which previously produced errors like "'_' can only appear in a pattern or on the left side of an assignment" may now produce errors which complain about type placeholders. + +## Effect on ABI stability + +This feature does not have any effect on the ABI. + +## Effect on API resilience + +Type placeholders are not exposed as API. In a compiled interface, type placeholders (except for those within the bodies of `@inlinable` functions or default argument expressions) are replaced by whatever type the type checker fills in for the type placeholder. While the introduction or removal of a type placeholder *on its own* is not necessarily an API or ABI break, authors should be careful that the introduction/removal of the additional type context does not ultimately change the inferred type of the variable. + +## Alternatives considered + +### Alternative spellings + +Several potential spellings of the type placeholder were suggested, with most users preferring either " `_` " or " `?` ". The question mark version was rejected primarily for the fact that the existing usage of `?` in the type grammar for optionals would be confusing and or ambiguous if it were overloaded to also stand for a type placeholder. + +Some users also worried that the underscore spelling would preclude the same spelling from being used for automatically type-erased containers, e.g., + +```swift +var anyArray: Array<_> = [0] +anyArray = ["string"] +let stringArray = anyArray as? Array +``` + +This objection to the `_` is compelling, but it was noted during discussion that usage of an explicit existential marker keyword (a la `any Array<_>` ) could allow the usage of an underscore for both type placeholders and erased types. + +At the pitch phase, the author remains open to alternative spellings for this feature. In particular, the " `any Array<_>` " resolution does not address circumstances where an author may want to both erase some components of a type but allow inference to fill in others. + +## Future directions + +### Placeholders for generic bases and nested types + +In some examples, we're still technically providing more information than the compiler strictly needs to determine the type of an expression. E.g., in the example from the **Type inference** section, we could have conceivably written the type annotation as: + +```swift +let publisher: _ = Just(makeValue()).setFailureType(to: Error.self).eraseToAnyPublisher() +``` + +Since the type of the generic `AnyPublisher` base is fully determined from the result type of `eraseToAnyPublisher()` . + +Similarly, type placeholders could be used in type member positions to denote some type that is nested within another: + +```swift +struct S { + struct Inner {} + + func overloaded() -> Inner { } + func overloaded() -> Int { } +} + +func test(val: S) { + let result: S._ = val.overloaded() // Calls 'func overloaded() -> Inner' +} +``` + +The author is skeptical that either of these extensions of type placeholders ultimately results in clearer code, and so opts to defer consideration of such a feature until there is further discussion about potential uses/tradeoffs. + +### Attributed type placeholders + +Type placeholders could be used to apply an attribute when the rest of the type can be inferred from context, e.g.: + +```swift +let x: @convention(c) _ = { 0 } +``` + +Unfortunately, the current model for type attributes makes this somewhat problematic. Type attributes are closely tied to the syntactic form of type names, meaning that constructions like: + +```swift +typealias F = () -> Int +let x: @convention(c) F = { 0 } +``` + +are already illegal. + +Since there is more subtle design work to be done here, and because the use cases for this extension of type placeholders are comparatively narrow, the author opts to leave this as a future direction. + +### Top-level type placeholders + +An earlier draft of this proposal allowed for the use of placeholders as top-level types, so that one could write + +```swift +let x: _ = 0.0 // type of x is inferred as Double +``` + +Compared to other uses of this feature, top-level placeholders are clearly of more limited utility. In type annotations (as above), they merely serve as a slightly more explicit way to indicate "this type is inferred," and they are similarly unhelpful in `as` casts. There is *some* use for top-level placeholders in type expression position, particularly when passing a metatype value as a parameter. For instance, Combine's `setFailureType(to:)` operator could be used with a top-level placeholder to make conversions between failure types more lightweight when necessary: + +```swift +let p: AnyPublisher = Just().setFailureType(to: _.self).eraseToAnyPublisher() +``` + +However, as Xiaodi Wu points out, allowing placeholders in these positions would have the effect of permitting clients to leave out type names in circumstances where library authors intended the type information to be provided explicitly, such as when using `KeyedDecodingContainer.decode(_:forKey:)`. It is not obviously desirable for users to be able to write, e.g.: + +```swift +self.someProp = try container.decode(_.self, forKey: .someProp) +``` + +Due to the additional considerations here, the author has opted to leave top-level placeholders as a future direction, which could potentially be considered once there is more real-world usage of type placeholders that could inform the benefits and drawbacks. + +## Acknowledgments + +- Ben Rimmington and Xiaodi Wu suggested illustrative examples to help explain some more subtle aspects of the proposal. +- Xiaodi entertained extensive discussion about arcane syntactic forms that could be written using placeholders. +- Varun Gandhi and Rintaro Ishizaki helped come up with some edge cases to test and Varun provided valuable input regarding the scoping of this proposal. +- Holly Borla provided some well-timed encouragement and feedback to help push this proposal to completion. +- Pavel Yaskevich and Robert Widmann patiently reviewed the initial implementation. diff --git a/proposals/0316-global-actors.md b/proposals/0316-global-actors.md new file mode 100644 index 0000000000..79ba23067a --- /dev/null +++ b/proposals/0316-global-actors.md @@ -0,0 +1,549 @@ +# Global actors + +* Proposal: [SE-0316](0316-global-actors.md) +* Authors: [John McCall](https://github.com/rjmccall), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.5)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0316-global-actors/50116) + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Proposed solution](#proposed-solution) + * [Defining global actors](#defining-global-actors) + * [The main actor](#the-main-actor) + * [Using global actors on functions and data](#using-global-actors-on-functions-and-data) + * [Global actor function types](#global-actor-function-types) + * [Closures](#closures) + * [Global and static variables](#global-and-static-variables) + * [Using global actors on a type](#using-global-actors-on-a-type) + * [Global actor inference](#global-actor-inference) + * [Global actors and instance actors](#global-actors-and-instance-actors) +* [Detailed design](#detailed-design) + * [`GlobalActor` protocol](#globalactor-protocol) + * [Closure attributes](#closure-attributes) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) +* [Effect on runtime and standard library](#effect-on-runtime-and-standard-library) +* [Future directions](#future-directions) + * [Restricting global and static variables](#restricting-global-and-static-variables) + * [Global actor-constrained generic parameters](#global-actor-constrained-generic-parameters) +* [Alternatives considered](#alternatives-considered) + * [Singleton support](#singleton-support) + * [Propose only the main actor](#propose-only-the-main-actor) +* [Revision history](#revision-history) + +## Introduction + +[Actors](0306-actors.md) are a new kind of reference type that protect their instance data from concurrent access. Swift actors achieve this with *actor isolation*, which ensures (at compile time) that all accesses to that instance data go through a synchronization mechanism that serializes execution. + +This proposal introduces *global actors*, which extend the notion of actor isolation outside of a single actor type, so that global state (and the functions that access it) can benefit from actor isolation, even if the state and functions are scattered across many different types, functions and modules. Global actors make it possible to safely work with global variables in a concurrent program, as well as modeling other global program constraints such as code that must only execute on the "main thread" or "UI thread". + +Global actors also provide a means to eliminate data races on global and static variables, allowing access to such variables to be synchronized via a global actor. + +Swift-evolution threads: [Pitch #1](https://forums.swift.org/t/pitch-global-actors/45706), [Pitch #2](https://forums.swift.org/t/pitch-2-global-actors/48332) + +## Motivation + +Actors are fantastic for isolating instance data, providing a form of reference type that can be used in concurrent programs without introducing data races. However, when the data that needs to be isolated is scattered across a program, or is representing some bit of state that exists outside of the program, bringing all of that code and data into a single actor instance might be impractical (say, in a large program) or even impossible (when interacting with a system where those assumptions are pervasive). + +A primary motivator of global actors is to apply the actor model to the state and operations that can only be accessed by the *main thread*. In an application, the main thread is generally responsible for executing the primary event-handling loop that processes events from various sources and delivers them to application code. Graphical applications often deliver user-interaction events (a keyboard press, a touch interaction) on the main thread, and require that any stateful updates to the user interface occur there as well. Global actors provide the mechanism for describing the main thread in terms of actors, utilizing Swift's actor isolation model to aid in correct usage of the main thread. + +## Proposed solution + +A global actor is a globally-unique actor identified by a type. That type becomes a custom attribute (similar to [property wrapper types](0258-property-wrappers.md) or [result builder types](0289-result-builders.md)). Any declaration can state that it is actor-isolated to that particular global actor by naming the global actor type as an attribute, at which point all of the normal actor-isolation restrictions come into play: the declaration can only be synchronously accessed from another declaration on the same global actor, but can be asynchronously accessed from elsewhere. For example, this proposal introduces `MainActor` as a global actor describing the main thread. It can be used to require that certain functions only execute on the main thread: + +```swift +@MainActor var globalTextSize: Int + +@MainActor func increaseTextSize() { + globalTextSize += 2 // okay: +} + +func notOnTheMainActor() async { + globalTextSize = 12 // error: globalTextSize is isolated to MainActor + increaseTextSize() // error: increaseTextSize is isolated to MainActor, cannot call synchronously + await increaseTextSize() // okay: asynchronous call hops over to the main thread and executes there +} +``` + +### Defining global actors + +A global actor is a type that has the `@globalActor` attribute and contains a `static` property named `shared` that provides a shared instance of an actor. For example: + +```swift +@globalActor +public struct SomeGlobalActor { + public actor MyActor { } + + public static let shared = MyActor() +} +``` + +A global actor type can be a struct, enum, actor, or `final` class. It is essentially just a marker type that provides access to the actual shared actor instance via `shared`. The shared instance is a globally-unique actor instance that becomes synonymous with the global actor type, and will be used for synchronizing access to any code or data that is annotated with the global actor. + +Global actors implicitly conform to the `GlobalActor` protocol, which describes the `shared` requirement. The conformance of a `@globalActor` type to the `GlobalActor` protocol must occur in the same source file as the type definition, and the conformance itself cannot be conditional. + + +### The main actor + +The *main actor* is a global actor that describes the main thread: + +```swift +@globalActor +public actor MainActor { + public static let shared = MainActor(...) +} +``` + +> **Note**: integrating the main actor with the system's main thread requires support for [custom executors][customexecs], which is the subject of another proposal, as well as specific integration with the system's notion of the main thread. For systems that use the Apple's [Dispatch](https://developer.apple.com/documentation/DISPATCH) library as the underlying concurrency implementation, the main actor uses a custom executor that wraps the [main dispatch queue](https://developer.apple.com/documentation/dispatch/dispatchqueue/1781006-main). It also determines when code is dynamically executing on the main actor to avoid an extra "hop" when performing an asynchronous call to a `@MainActor` function. + +### Using global actors on functions and data + +As illustrated in our first example, both functions and data can be attributed with a global actor type to isolate them to that global actor. Note that global actors are not restricted to global functions or data as in the first example. One can mark members of types and protocols as belonging to a global actor as well. For example, in a view controller for a graphical UI, we would expect to receive notification of user interactions on the main thread, and must update the UI on the main thread. Therefore want both the methods called on notification and also the data they use to be on the main actor. Here's an small part of a view controller from some [AppKit sample code](https://developer.apple.com/documentation/appkit/cocoa_bindings/navigating_hierarchical_data_using_outline_and_split_views): + +```swift +class IconViewController: NSViewController { + @MainActor @objc private dynamic var icons: [[String: Any]] = [] + + @MainActor var url: URL? + + @MainActor private func updateIcons(_ iconArray: [[String: Any]]) { + icons = iconArray + + // Notify interested view controllers that the content has been obtained. + // ... + } +} +``` + +Note that the data in this view controller, as well as the method that performs the update of this data, is isolated to the `@MainActor`. That ensures that UI updates for this view controller only occur on the main thread, and any attempts to do otherwise will result in a compiler error. + +The sample code actually triggers an update when the `url` property is set. With global actors, that would look something like this: + +```swift +@MainActor var url: URL? { + didSet { + // Asynchronously perform an update + Task.detached { [url] in // not isolated to any actor + guard let url = url else { return } + let newIcons = self.gatherContents(url) + await self.updateIcons(newIcons) // 'await' required so we can hop over to the main actor + } + } +} +``` + +### Global actor function types + +A synchronous function type can be qualified to state that the function is only callable on a specific global actor: + +```swift +var callback: @MainActor (Int) -> Void +``` + +Such a function can only be synchronously called from code that is itself isolated to the same global actor. + +A reference to a function that is isolated to a global actor will have a function type with a global actor. The references themselves are not subject to actor-isolation checking, because the actor isolation is described by the resulting function type. For example: + +```swift +func functionsAsValues(controller: IconViewController) { + let fn = controller.updateIcons // okay, type is @MainActor ([[String: Any]]) -> Void + let fn2 = IconViewController.controller.updateIcons // okay, type is (IconViewController) -> (@MainActor ([[String: Any]]) -> Void) + fn([]) // error: cannot call main actor-isolated function synchronously from outside the actor +} +``` + +Values may be converted from a function type with no global actor qualifier to a function with a global actor qualifier. For example: + +```swift +func acceptInt(_: Int) { } // not on any actor + +callback = acceptInt // okay: conversion to @MainActor (Int) -> Void +``` + +The opposite conversion is not permitted for synchronous functions, because doing so would allow the function to be called without being on the global actor: + +```swift +let fn3: (Int) -> Void = callback // error: removed global actor `MainActor` from function type +``` + +However, it is permissible for the global actor qualifier to be removed when the result of the conversion is an `async` function. In this case, the `async` function will first "hop" to the global actor before executing its body: + +```swift +let callbackAsynchly: (Int) async -> Void = callback // okay: implicitly hops to main actor +``` + +This can be thought of as syntactic sugar for the following: + +```swift +let callbackAsynchly: (Int) async -> Void = { + await callback() // `await` is required because callback is `@MainActor` +} +``` + +A global actor qualifier on a function type is otherwise independent of `@Sendable`, `async`, `throws` and most other function type attributes and modifiers. The only exception is when the function itself is also isolated to an instance actor, which is discussed in the later section on [Global actors and instance actors](#global-actors-and-instance-actors). + +### Closures + +A closure can be explicitly specified to be isolated to a global actor by providing the attribute prior to the `in` in the closure specifier, e.g., + +```swift +callback = { @MainActor in + print($0) +} + +callback = { @MainActor (i) in + print(i) +} +``` + +When a global actor is applied to a closure, the type of the closure is qualified with that global actor. + +> **Note**: this can be used to replace the common pattern used with Apple's Dispatch library of executing main-thread code via `DispatchQueue.main.async { ... }`. One would instead write: +> ```swift +> Task.detached { @MainActor in +> // ... +> } +> ``` +> This formulation ensures that the closure body is executed on the main actor, and can synchronously use other `@MainActor`-annotated declarations. + +If a closure is used to directly initialize a parameter or other value of a global-actor-qualified function type, and the closure itself does not have a global actor explicitly specified on it, the closure will have that global actor inferred. For example: + +```swift +@MainActor var globalTextSize: Int + +var callback: @MainActor (Int) -> Void +callback = { // closure is inferred to be @MainActor due to the type of 'callback' + globalTextSize = $0 // okay: closure is on @MainActor +} +``` + +### Global and static variables + +Global and static variables can be annotated with a global actor. Such variables can only be accessed from the same global actor or asynchronously, e.g., + +```swift +@MainActor var globalCounter = 0 + +@MainActor func incrementGlobalCounter() { + globalCounter += 1 // okay, we are on the main actor +} + +func readCounter() async { + print(globalCounter) // error: cross-actor read requires 'await' + print(await globalCounter) // okay +} +``` + +As elsewhere, cross-actor references require the types involved to conform to `Sendable`. + +Global and static variables not annotated with a global actor can effectively be accessed from any concurrency context, and as such are prone to data races. Global actors provide one way to address such data races. The section on [future directions](#future-directions) considers whether to use global actors as a way to address data races for global and static variables comprehensively. + +### Using global actors on a type + +It is common for entire types (and even class hierarchies) to predominantly require execution on the main thread, and for asynchronous work to be a special case. In such cases, the type itself can be annotated with a global actor, and all of the methods, properties, and subscripts will implicitly be isolated to that global actor. Any members of the type that do not want to be part of the global actor can opt out, e.g., using the [`nonisolated` modifier][isolation]. For example: + +```swift +@MainActor +class IconViewController: NSViewController { + @objc private dynamic var icons: [[String: Any]] = [] // implicitly @MainActor + + var url: URL? // implicitly @MainActor + + private func updateIcons(_ iconArray: [[String: Any]]) { // implicitly @MainActor + icons = iconArray + + // Notify interested view controllers that the content has been obtained. + // ... + } + + nonisolated private func gatherContents(url: URL) -> [[String: Any]] { + // ... + } +} +``` + +A non-protocol type that is annotated with a global actor implicitly conforms to `Sendable`. Instances of such types are safe to share across concurrency domains because access to their +state is guarded by the global actor. + +A class can only be annotated with a global actor if it has no superclass, the superclass is annotated with the same global actor, or the superclass is `NSObject`. A subclass of a global-actor-annotated class must be isolated to the same global actor. + +### Global actor inference + +Declarations that are not explicitly annotated with either a global actor or `nonisolated` can infer global actor isolation from several different places: + +* Subclasses infer actor isolation from their superclass: + + ```swift + class RemoteIconViewController: IconViewController { // implicitly @MainActor + func connect() { ... } // implicitly @MainActor + } + ``` + +* An overriding declaration infers actor isolation from the declaration it overrides: + + ```swift + class A { + @MainActor func updateUI() { ... } + } + + class B: A { + override func updateUI() { ... } // implicitly @MainActor + } + ``` + +* A witness that is not inside an actor type infers actor isolation from a protocol requirement that is satisfies, so long as the protocol conformance is stated within the same type definition or extension as the witness: + + ```swift + protocol P { + @MainActor func f() + } + + struct X { } + + extension X: P { + func f() { } // implicitly @MainActor + } + + struct Y: P { } + + extension Y { + func f() { } // okay, not implicitly @MainActor because it's in a separate extension + // from the conformance to P + } + ``` + +* A non-actor type that conforms to a global-actor-qualified protocol within the same source file as its primary definition infers actor isolation from that protocol: + + ```swift + @MainActor protocol P { + func updateUI() { } // implicitly @MainActor + } + + class C: P { } // C is implicitly @MainActor + + // source file D.swift + class D { } + + // different source file D-extensions.swift + extension D: P { // D is not implicitly @MainActor + func updateUI() { } // okay, implicitly @MainActor + } + ``` + +* A struct or class containing a wrapped instance property with a global actor-qualified `wrappedValue` infers actor isolation from that property wrapper: + + ```swift + @propertyWrapper + struct UIUpdating { + @MainActor var wrappedValue: Wrapped + } + + struct CounterView { // infers @MainActor from use of @UIUpdating + @UIUpdating var intValue: Int = 0 + } + ``` + +### Global actors and instance actors + +A declaration cannot both be isolated to a global actor and isolated to an instance actor. If an instance declaration within an actor type is annotated with a global actor, it is isolated to the global actor but *not* its enclosing actor instance: + +```swift +actor Counter { + var value = 0 + + @MainActor func updateUI(view: CounterView) async { + view.intValue = value // error: `value` is actor-isolated to `Counter` but we are in a 'MainActor'-isolated context + view.intValue = await value // okay to asynchronously read the value + } +} +``` + +With the `isolated` parameters described in [SE-0313][isolation], no function type can contain both an `isolated` parameter and also a global actor qualifier: + +```swift +@MainActor func tooManyActors(counter: isolated Counter) { } // error: 'isolated' parameter on a global-actor-qualified function +``` + +## Detailed design + +Global actor attributes apply to declarations as follows: + +* A declaration cannot have multiple global actor attributes. The rules below say that, in some cases, a global actor attribute is propagated from one declaration to another. If the rules say that an attribute “propagates by default”, then no propagation is performed if the destination declaration has an explicit global actor attribute. If the rules say that attribute “propagates mandatorily”, then it is an error if the destination declaration has an explicit global actor attribute that does not identify the same actor. Regardless, it is an error if global actor attributes that do not identify the same actor are propagated to the same declaration. + +* A function declared with a global actor attribute becomes isolated to the given global actor. + +* A stored variable or constant declared with a global actor attribute becomes part of the isolated state of the given global actor. + +* The accessors of a variable or subscript declared with a global actor attribute become isolated to the given global actor. (This includes observing accessors on a stored variable.) + +* Local variables and constants cannot be marked with a global actor attribute. + +* A type declared with a global actor attribute propagates the attribute to all methods, properties, subscripts, and extensions of the type by default. + +* An extension declared with a global actor attribute propagates the attribute to all the members of the extension by default. + +* A protocol declared with a global actor attribute propagates the attribute to any type that conforms to it in the primary type definition by default. + +* A protocol requirement declared with a global actor attribute requires that a given witness must either have the same global actor attribute or be non-isolated. (This is the same rule observed by all witnesses for actor-isolated requirements). + +* A class declared with a global actor attribute propagates the attribute to its subclasses mandatorily. + +* An overridden declaration propagates its global actor attribute (if any) to its overrides mandatorily. Other forms of propagation do not apply to overrides. It is an error if a declaration with a global actor attribute overrides a declaration without an attribute. + +* An actor type cannot have a global actor attribute. Stored instance properties of actor types cannot have global actor attributes. Other members of an actor type can have global actor attributes; such members are isolated to the global actor, but not to the enclosing actor. (Per the proposal on [improved control over actor isolation][isolation], the `self` of such methods is not `isolated`). + +* A `deinit` cannot have a global actor attribute and is never a target for propagation. + +### `GlobalActor` protocol + +The `GlobalActor` protocol is defined as follows: + +```swift +/// A type that represents a globally-unique actor that can be used to isolate +/// various declarations anywhere in the program. +/// +/// A type that conforms to the `GlobalActor` protocol and is marked with the +/// the `@globalActor` attribute can be used as a custom attribute. Such types +/// are called global actor types, and can be applied to any declaration to +/// specify that such types are isolated to that global actor type. When using +/// such a declaration from another actor (or from nonisolated code), +/// synchronization is performed through the \c shared actor instance to ensure +/// mutually-exclusive access to the declaration. +public protocol GlobalActor { + /// The type of the shared actor instance that will be used to provide + /// mutually-exclusive access to declarations annotated with the given global + /// actor type. + associatedtype ActorType: Actor + + /// The shared actor instance that will be used to provide mutually-exclusive + /// access to declarations annotated with the given global actor type. + /// + /// The value of this property must always evaluate to the same actor + /// instance. + static var shared: ActorType { get } +} +``` + +### Closure attributes + +The global actor for a closure is one of a number of potentially-allowable attributes on a closure. The attributes precede the capture-list in the grammar: + +``` +closure-expression → { closure-signature opt statements opt } +closure-signature → attributes[opt] capture-list[opt] closure-parameter-clause async[opt] throws[opt] function-result[opt] in +closure-signature → attributes[opt] capture-list in +closure-signature → attributes in +``` + +## Source compatibility + +Global actors are an additive feature that have no impact on existing source code. + +## Effect on ABI stability + +A global actor annotation is part of the type of an entity, and is therefore part of its mangled name. Otherwise, a global actor has no effect on the ABI. + +## Effect on API resilience + +The `@globalActor` attribute can be added to a type without breaking API. + +A global actor attribute (such as `@MainActor`) can neither be added nor removed from an API; either will cause breaking changes for source code that uses the API. + +## Effect on runtime and standard library + +This proposal introduces a new kind of function type, a global-actor-qualified function type, which requires updates to the Swift runtime, metadata, name mangling scheme, and dynamic-casting machinery. For example, consider the following code: + +```swift +@MainActor func f(_ potentialCallback: Any) { + let Callback = @MainActor () -> Void + if let callback = potentialCallback as? Callback { + callback() + } +} +``` + +The dynamic cast to a global-actor-qualified function type requires changes to the Swift runtime to represent global-actor-qualified function types and model dynamic casts of unknown values to them. Similar changes for global-actor-qualified function types are required for name mangling, which also has runtime impact. + +## Future directions + +### Restricting global and static variables + +A global actor annotation on a global or static variable synchronizes all access to that variable through that global actor. We could require that *all* mutable global and static variables be annotated with a global actor, thereby eliminating those as a source of data races. Specifically, we can require that every global or static variable do one of the following: + +* Explicitly state that it is part of a global actor, or +* Be immutable (introduced via `let`), non-isolated, and of `Sendable` type. + +This allows global/static immutable constants to be used freely from any code, while any data that is mutable (or could become mutable in a future version of a library) must be protected by an actor. However, it comes with significant source breakage: every global variable that exists today would require annotation. Therefore, we aren't proposing to introduce this requirement, and instead leave the general data-race safety of global and static variables to a later proposal. + +### Global actor-constrained generic parameters + +A generic parameter that is constrained to `GlobalActor` could potentially be used as a global actor. For example: + +```swift +@T +class X { + func f() { ... } // constrained to the global actor T +} + +@MainActor func g(x: X, y: X) async { + x.f() // okay, on the main actor + await y.f() // okay, but requires asynchronous call because y.f() is on OtherGlobalActor +} +``` + +There are some complications here: without marking the generic parameter `T` with the `@globalActor` attribute, it wouldn't be clear what kind of custom attribute `T` is. Therefore, this might need to be expressed as, e.g., + +```swift +@T +class X<@globalActor T> { ... } +``` + +which would imply the requirement `T: GlobalActor`. However, doing this would require Swift to also support attributes on generic parameters, which currently don't exist. This is a promising direction for a follow-on proposal. + +## Alternatives considered + +### Singleton support + +Global actors are, effectively, baking a convention for singletons in the language. Singletons are occasionally used in Swift, and if they were to get special language syntax, global actors could be introduced with less boilerplate as "singleton actors", e.g., + +```swift +singleton actor MainActor { + // integration with system's main thread +} +``` + +This would eliminate the `@globalActor` attribute from the proposal, but would otherwise leave it unchanged. + +### Propose only the main actor + +The primary motivation for global actors is the main actor, and the semantics of this feature are tuned to the needs of main-thread execution. We know abstractly that there are other similar use cases, but it's possible that global actors aren't the right match for those use cases. Rather than provide a general feature for global actors now, we could narrow this proposal to `@MainActor` only, then provide global actors (or some other abstraction) at some later point to subsume `@MainActor` and other important use cases. + +## Revision history + +* Changes to the accepted version: + * Move global actor-constrained generic parameters to "future directions" + * Classes that are global actors must be `final`. +* Changes for the second review: + * Added the `GlobalActor` protocol, to which all global actors implicitly conform. + * Remove the requirement that all global and static variables be annotated with a global actor. + * Added a grammar for closure attributes. + * Clarified the interaction between the main actor and the main thread. Make the main actor a little less "special" in the initial presentation. +* Changes for the first review: + * Add inference of a global actor for a witness to a global-actor-qualified requirement. + * Extended inference of global actor-ness from protocols to conforming types to any extension within the same source file as the primary type definition. +* Changes in the second pitch: + * Clarify that the types of global-actor-qualified functions are global-actor-qualified. + * State that global-actor-qualified types are Sendable + * Expand on the implicit conversion rules for function types + * Require global and static variables to be immutable & non-isolated or global-actor-qualified. + * Describe the relationship between global actors and instance actors + * Update inference rules for global actors + + +[customexecs]: https://github.com/rjmccall/swift-evolution/blob/custom-executors/proposals/0000-custom-executors.md +[isolation]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0313-actor-isolation-control.md + diff --git a/proposals/0317-async-let.md b/proposals/0317-async-let.md new file mode 100644 index 0000000000..113e762a6a --- /dev/null +++ b/proposals/0317-async-let.md @@ -0,0 +1,878 @@ +# `async let` bindings + +* Proposal: [SE-0317](0317-async-let.md) +* Authors: [John McCall](https://github.com/rjmccall), [Joe Groff](https://github.com/jckarter), [Doug Gregor](https://github.com/DougGregor), [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.5)** + +## Introduction + +[Structured concurrency](0304-structured-concurrency.md) provides a paradigm for spawning concurrent *child tasks* in scoped *task groups*, establishing a well-defined hierarchy of tasks which allows for cancellation, error propagation, priority management, and other tricky details of concurrency management to be handled transparently. + +This proposal aims to make the common task of spawning child tasks to run asynchronously and pass their eventual +results up to their parent, using lightweight syntax similar to `let` bindings. + +Discussion threads: + +- Originally pitched as part of Structured Concurrency: + - [Pitch #1](https://forums.swift.org/t/concurrency-structured-concurrency/41622), + - [Pitch #2](https://forums.swift.org/t/pitch-2-structured-concurrency/43452). +- and later separated into its own proposal: + - [Pitch #3](https://forums.swift.org/t/pitch-3-async-let/48336). +- Separate discussion on [scoped suspension points](https://forums.swift.org/t/async-let-and-scoped-suspension-points/49846). + +[TOC] + + + +## Motivation + +In [SE-0304: Structured Concurrency](0304-structured-concurrency.md) we introduced the concept of tasks and task groups, which can be used to spawn multiple concurrently executing child-tasks and collect their results before exiting out of the task group. + +Task groups are a very powerful, yet low-level, building block useful for creating powerful parallel computing patterns, such as collecting the "first few" successful results, and other typical fan-out or scatter/gather patterns. They work best for spreading out computation of same-typed operations. For example, a parallelMap could be implemented in terms of a TaskGroup. In that sense, task groups are a low level implementation primitive, and not the end-user API that developers are expected to interact with a lot, rather, it is expected that more powerful primitives are built on top of task groups. + +Task Groups also automatically propagate task cancellation, priority, and task-local values through to child-tasks and offer a flexible API to collect results from those child-tasks _in completion order_, which is impossible to achieve otherwise using other structured concurrency APIs. They do all this while upholding the structured concurrency guarantees that a child-task may never "out-live" (i.e. keep running after the task group scope has exited) the parent task. + +While task groups are indeed very powerful, they are hard to use with *heterogeneous results* and step-by-step initialization patterns. + +The following example, an asynchronous `makeDinner` function, consists of both of those patterns. It consists of three tasks which can be performed in parallel, all yielding different result types. To proceed to the final step of the cooking process, all those results need to be obtained, and fed into the final `oven.cook(...)` function. In a way, this is the trickiest situation to implement well using task groups. Let us examine it more closely: + +```swift +func makeDinner() async -> Meal { + // Create a task group to scope the lifetime of our three child tasks + return try await withThrowingTaskGroup(of: CookingTask.self) { group in + // spawn three cooking tasks and execute them in parallel: + group.addTask { + CookingTask.veggies(try await chopVegetables()) + } + group.addTask { + CookingTask.meat(await marinateMeat()) + } + group.addTask { + CookingTask.oven(await preheatOven(temperature: 350)) + } + + // prepare variables to collect the results + var veggies: [Vegetable]? = nil + var meat: Meat? = nil + var oven: Oven? = nil + + // collect the results + for try await task in group { + switch task { + case .veggies(let v): + veggies = v + case .meat(let m): + meat = m + case .oven(let o): + oven = o + } + } + + // ensure every variable was initialized as expected + assert(veggies != nil) + assert(meat != nil) + assert(oven != nil) + + // prepare the ingredients + var ingredients: [Ingredient] = veggies! + ingredients.append(meat!) + + // and, finally, cook the meal, awaiting inside the group + let dish = Dish(ingredients: ingredients) + return try await oven!.cook(dish, duration: .hours(3)) + } +} +``` + +The `withThrowingTaskGroup` scope explicitly delineates any potential concurrency, because it guarantees that any child tasks spawned within it are awaited on as the group scope exits. Any results can be collected by iterating through the group. Errors and cancellation are handled automatically for us by the group. + +However, this example showcases the weaknesses of the TaskGroups very well: heterogeneous result processing and variable initialization become very boilerplate heavy. While there exist ideas to make this boilerplate go away in future releases, with smarter analysis and type checking, the fundamental issue remains. + +If we step back a little, we can notice that in the example each child task is really producing a *single value* and returning it back to the *parent task*, which then needs to assemble the pieces and proceed with calling some other function. We achieve this by preparing, and assigning into `Optional` variables dedicated for each of the spawned tasks. This is not ideal, since +although the code is correct as written, modifying this code to add a variable is not only boilerplate heavy, but also potentially quite error prone, leading to runtime crashes due to the force-unwraps which a well written Swift program usually would not have to resort to. + +This dataflow pattern from child tasks to parent is very common, and we want to make it as lightweight and safe as possible. + +## Proposed solution + +This proposal introduces a simple way to create child tasks and await their results: `async let` declarations. + +Using `async let`, our example looks like this: + +```swift +// given: +// func chopVegetables() async throws -> [Vegetables] +// func marinateMeat() async -> Meat +// func preheatOven(temperature: Int) async -> Oven + +func makeDinner() async throws -> Meal { + async let veggies = chopVegetables() + async let meat = marinateMeat() + async let oven = preheatOven(temperature: 350) + + let dish = Dish(ingredients: await [try veggies, meat]) + return try await oven.cook(dish, duration: .hours(3)) +} +``` + +`async let` is similar to a `let`, in that it defines a local constant that is initialized by the expression on the right-hand side of the `=`. However, it differs in that the initializer expression is evaluated in a separate, concurrently-executing child task. + +The child task begins running as soon as the `async let` is encountered. By default, child tasks use the global, width-limited, concurrent executor, in the same manner as task group child-tasks do. It is a future direction to allow customizing which executor these should be executing on. On normal completion, the child task will initialize the variables in the `async let`. + +The right-hand side of a `async let` expression can be thought of as an implicit `@Sendable closure`, similar to how the `Task.detached { ... }` API works, however the resulting task is a *child task* of the currently executing task. Because of this, and the need to suspend to await the results of such expression, `async let` declarations may only occur within an asynchronous context, i.e. an `async` function or closure. + +For single statement expressions in the `async let` initializer, the `await` and `try` keywords may be omitted. The effects they represent carry through to the introduced constant and will have to be used when waiting on the constant. In the example shown above, the veggies are declared as `async let veggies = chopVegetables()`, and even though `chopVegetables` is `async` and `throws`, the `await` and `try` keywords do not have to be used on that line of code. Once waiting on the value of that `async let` constant, the compiler will enforce that the expression where the `veggies` appear must be covered by both `await` and some form of `try`. + +Because the main body of the function executes concurrently with its child tasks, it is possible that the parent task (the body of `makeDinner` in this example) will reach the point where it needs the value of a `async let` (say,`veggies`) before that value has been produced. To account for that, reading a variable defined by a `async let` is treated as a potential suspension point, +and therefore must be marked with `await`. + +## Detailed design + +### Declaring `async let` constants + +`async let` declarations are similar to `let` declarations, however they can only appear in specific contexts. + +Because the asynchronous task must be able to be awaited on in the scope it is created in, it is only possible to declare `async let`s in contexts where it would also be legal to write an explicit `await`, i.e. asynchronous functions: + +```swift +func greet() async -> String { "hi" } + +func asynchronous() async { + async let hello = greet() + // ... + await hello +} +``` + +and inside asynchronous closures: + +```swift +func callMe(_ maybe: () async -> String) async -> String { + return await maybe() +} + +callMe { // async closure + async let hello = greet() + // ... + return await hello +} +``` + +It is not allowed to declare `async let` as top-level code, in synchronous functions or closures: + +```swift +async let top = ... // error: 'async let' in a function that does not support concurrency + +func sync() { // note: add 'async' to function 'sync()' to make it asynchronous + async let x = ... // error: 'async let' in a function that does not support concurrency +} + +func syncMe(later: () -> String) { ... } +syncMe { + async let x = ... // error: invalid conversion from 'async' function of type '() async -> String' to synchronous function type '() -> String' +} +``` + +A `async let` creates a child-task, which inherits its parent task's priority as well as task-local values. Semantically, this is equivalent to creating a one-off `TaskGroup` which spawns a single task and returns its result, however the implementation of `async let`s can make more assumptions and optimizations around the lifetime and usage of those values. + +The child-task created to initialize the `async let` by default runs on the global concurrent, width-limited, executor that comes with the Swift Concurrency runtime. + +> Customizing the execution context of async lets is a future direction we are likely to explore with the introduction of Custom Executors. + +The initializer of the `async let` can be thought of as a closure that runs the code contained within it in a separate task, very much like the explicit `group.addTask { }` API of task groups. + +Similarly to the `group.addTask()` function, the closure is `@Sendable` and `nonisolated`, meaning that it cannot access non-sendable state of the enclosing context. For example, it will result in a compile-time error, preventing a potential race condition, for a `async let` initializer to attempt mutating a closed-over variable: + +```swift +var localText: [String] = ... +async let w = localText.removeLast() // error: mutation of captured var 'localText' in concurrently-executing code +``` + +The async let initializer may refer to any sendable state, same as any non-isolated sendable closure. + +The initializer of a `async let` permits the omission of the `await` keyword if it is directly calling an asynchronous function, like this: + +```swift +func order() async -> Order { ... } + +async let o1 = await order() +// should be written instead as +async let o2 = order() +``` + +This is because by looking at the async let declaration, it is obvious that the right-hand side function will be used to initialize the left-hand side, by waiting on it. This is similar to single-expression `return` keyword omission, and also applies only to single expression initializers. + +It is illegal to declare an `async var`. This is due to the complex initialization that a `async let` represents, it does not make sense to allow further external modification of them. Doing so would tremendously complicate the understandability of such asynchronous code, and undermine potential optimizations by making it harder to make assumptions about the data-flow of the values. + +```swift +async var x = nope() // error: 'async' can only be used with 'let' declarations +``` + +Other than having to be awaited to access its value, a `async let` behaves just like a typical `let`, as such it is not possible to pass it `inout` to other functions - simply because it is a `let`, and those may not be passed as `inout`. + +#### Declaring `async let` with patterns + +It is possible to create a `async let` where the left-hand side is a pattern, e.g. a tuple, like this: + +```swift +func left() async -> String { "l" } +func right() async -> String { "r" } + +async let (l, r) = (left(), right()) + +await l // at this point `r` is also assumed awaited-on +``` + +To understand the execution semantics of the above snippet, we can remember the sugaring rule that the right-hand side of a `async let` effectively is just a concurrently executing asynchronous closure: + +```swift +async let (l, r) = { + return await (left(), right()) + // -> + // return (await left(), await right()) +} +``` + +meaning that the entire initializer of the `async let` is a single task, and if multiple asynchronous function calls are made inside it, they are performed one-by one. This is a specific application of the general rule of `async let` initializers being allowed to omit a single leading `await` keyword before their expressions. Because in this example, we invoke two asynchronous functions to form a tuple, the `await` can be moved outside the expression, and that await is what is omitted in the shorthand form of the `async let` that we've seen in the first snippet. + +This also means that as soon as we continue past the line of `await l`, it is known that the `r` value also has completed successfully (and will not need to emit an "implicit await" which we'll discuss in detail below). + +Another implication of these semantics is that if _any_ piece of the initializer throws, any await on such pattern declared `async let` shall be considered throwing, as they are initialized "together". To visualize this, let us consider the following: + +```swift +async let (yay, nay) = ("yay", throw Boom()) +try await yay // because the (yay, nay) initializer is throwing +``` + +Because we know that the right-hand side is simply a single closure, performing the entire initialization, we know that if any of the operations on the right-hand size is throwing, the entire initializer will be considered throwing. As such, awaiting even the `yay` here must be ready for that initializer to have thrown and therefore must include the `try` keyword in addition to `await`. + +### Awaiting `async let` values + +Since `async let`s introduce constants that will be "filled in later" by their right-hand-side concurrently-executing task, referring to them must be covered by an `await` keyword: + +```swift +async let name = getName() +async let surname = getSurname() +await name +await surname +``` + +It is also possible to simply cover the entire expression where a `async let` is used with just a single `await`, similar to how the same can be done with `try`: + +```swift +greet(await name, await surname) +await greet(name, surname) +// or even +await print("\(name) \(surname)") +``` + +If the initializer of the specific `async let` was throwing, then awaiting on the `async let` constant must be covered using a variant of the `try` keyword: + +```swift +async let ohNo = throwThings() +try await ohNo +try? await ohNo +try! await ohNo +``` + +Currently, it is required to cover every reference to a `async let` using the appropriate `try` and `await` keywords, like this: + +```swift +async let yes = "" +async let ohNo = throwThings() + +_ = await yes +_ = await yes +_ = try await ohNo +_ = try await ohNo +``` + +This is a simple rule and allows us to bring the feature forward already. It might be possible to employ control flow based analysis to enable "only the first reference to the specific `async let` on each control flow path has to be an `await`", as technically speaking, every following await will be a no-op and will not suspend as the value is already completed and the placeholder has been filled in. + +### Implicit `async let` awaiting + +A `async let` that was declared but never awaited on *explicitly*, as the scope in which it was declared exits, will be awaited on implicitly. These semantics are put in place to uphold the Structured Concurrency guarantees provided by `async let`. + +To showcase these semantics, let us have a look at this function which spawns two child tasks, `fast` and `slow`, but does not await on any of them: + +```swift +func go() async { + async let f = fast() // 300ms + async let s = slow() // 3seconds + return "nevermind..." + // implicitly: cancels f + // implicitly: cancels s + // implicitly: await f + // implicitly: await s +} +``` + +Assuming the execution times of `fast()` and `slow()` are as the comments next to them explain, the `go()` function will _always_ take at least 3 seconds to execute. Or to state the rule more generally, any structured invocation will take as much time to return as the longest of its child tasks takes to complete. + +As we return from the `go()` function without ever having awaited on the `f` or `s` values, both of them will be implicitly cancelled and awaited on before returning from the function `go()`. This is the very nature of structured concurrency, and avoiding this can _only_ be done by creating non-child tasks, e.g. by using `Task.detached` or other future APIs which would allow creation of non-child tasks. + +If we instead awaited on one of the values, e.g. the fast one (`f`), the emitted code would not need to implicitly cancel or await it, as this was already taken care of explicitly: + +```swift +func go2() async { + async let f = fast() + async let s = slow() + _ = await f + return "nevermind..." + // implicitly: cancels s + // implicitly: awaits s +} +``` + +The duration of the `go2()` call remains the same, it is always `time(go2) == max(time(f), time(s))`. + +Special attention needs to be given to the `async let _ = ...` form of declarations. This form is interesting because it creates a child-task from the right-hand-side initializer, however it actively chooses to ignore the result. Such a declaration (and the associated child-task) will run and be cancelled and awaited-on implicitly, as the scope it was declared in is about to exit — the same way as an unused `async let` declaration would be. + +### `async let` and closures + +Because `async let` tasks cannot out-live the scope in which they are defined, passing them to closures needs some further discussion for what is legal and not. + +It is legal to capture a `async let` in a non-escaping asynchronous closure, like this: + +```swift +func greet(_ f: () async -> String) async -> String { await f() } + +async let name = "Alice" +await greet { await name } +``` + +Notice how we are required to write the `await` inside the closure as well as in front of the `greet` function. This is on purpose as we do want to be explicit about the `await` inside the closure. + +The same applies to auto closures, in order to make it explicit that the `await` is happening _inside_ the closure rather than before it, it is required to await explicitly in parameter position where the auto closure is formed for the argument: + +```swift +func greet(_ f: @autoclosure () async -> String) async -> String { await f() } + +async let name = "Bob" +await greet(await name) // await on name is required, because autoclosure +``` + +It is *not* legal to escape a `async let` value to an escaping closure. This is because structures backing the async let implementation may be allocated on the stack rather than the heap. This makes them very efficient, and makes great use of the structured guarantees they have to adhere to. These optimizations, however, make it unsafe to pass them to any escaping contexts: + +```swift +func greet(_ f: @escaping () async -> String) async -> String { somewhere = f; somewhere() } + +async let name = "Bob" +await greet { await name } // error: cannot escape 'async let' value +``` + + + +### `async let` error propagation + +While it is legal to declare a `async let` and never explicitly `await` on it, it also implies that we do not particularly care about its result. + +This is the same as spawning a number of child-tasks in a task group and not collecting their results, like so: + +```swift +try await withThrowingTaskGroup(of: Int.self) { group in + group.addTask { throw Boom() } + + return 0 // we didn't care about the child-task at all(!) +} // returns 0 +``` + +The above TaskGroup example will ignore the `Boom` thrown by its child task. However, it _will_ await for the task (and any other tasks it has spawned) to run to completion before the `withThrowingTaskGroup` returns. If we wanted to surface all potential throws of tasks spawned in the group, we should have written: `for try await _ in group {}` which would have re-thrown the `Boom()`. + +The same concept carries over to `async let`, where the scope of the group is replaced by the syntactic scope in which the `async let` was declared. For example, the following snippet is semantically equivalent to the above TaskGroup one: + +```swift +// func boom() throws -> Int { throw Boom() } + +func work() async -> Int { + async let work: Int = boom() + // never await work... + return 0 + // implicitly: cancels work + // implicitly: awaits work, discards errors +} +``` + +This `work()` function will never throw, because we didn't await on the throwing `async let`. If we modified it to explicitly await on it, the compiler would force us to spell out not only the `await` but also the `try` keyword. The presence of the `try` keyword would then force us to annotate the `work()` function as `throws`, as expected from normal, non-asynchronous code in Swift: + +```swift +// func boom() throws -> Int { throw Boom() } + +func work() async throws -> Int { // throws is enforced due to 'try await' + async let work: Int = boom() + // ... + return try await work // 'try' is enforced since 'boom()' was throwing +} +``` + +Alternatively, we could have handled the error of `work` by wrapping it in a `do/catch`. + +### Cancellation and `async let` child tasks + +Cancellation propagates recursively through the task hierarchy from parent to child tasks. + +Because tasks spawned by `async let` are child tasks, they naturally participate in their parent's cancellation. + +Cancellation of the parent task means that the context in which the `async let` declarations exist is cancelled, and any tasks created by those declarations will be cancelled as well. Because cancellation in Swift is co-operative, it does not prevent the spawning of tasks, however tasks spawned from a cancelled context are *immediately* marked as cancelled. This exhibits the same semantics as `TaskGroup.addTask` which, when used from an already cancelled task, _will_ spawn more child-tasks, however they will be immediately created as cancelled tasks — which they can inspect by calling `Task.isCancelled`. + +We can observe this in the following example: + +```swift +let handle = Task.detached { + // don't write such spin loops in real code (!!!) + while !Task.isCancelled { + // keep spinning + await Task.sleep(...) + } + + assert(Task.isCancelled) // parent task is cancelled + async let childTaskCancelled = Task.isCancelled // child-task is spawned and is cancelled too + + assert(await childTaskCancelled) +} + +handle.cancel() +``` + +The example uses APIs defined in the Structured Concurrency proposal: `Task.detached` to obtain a handle for the detached task which we can cancel explicitly. This allows us to easily illustrate that a `async let` entered within a task that _already is cancelled_ still spawns the child task, yet the spawned task will be immediately cancelled - as witnessed by the `true` returned into the `childTaskCancelled` variable. + +This works well with the co-operative nature of task cancellation in Swift's concurrency story. Tasks which are able and willing to participate in cancellation handling, need to check for its status using `Task.isCancelled` or `try Task.checkCancellation()` where appropriate. + +### Analysis of limitations and benefits of `async let` + +#### Comparing with `TaskGroup` + +Semantically, one might think of a `async let` as sugar for manually using a task group, spawning a single task within it and collecting the result from `group.next()` wherever the `async let` declared value is `await`-ed on. As we saw in the [Motivation](#motivation) section of the proposal, such explicit usage of groups ends up very verbose and error prone in practice, thus the need for a "sugar" for the specific pattern. + +A `async let` declaration, in reality, is not just a plain sugar-syntax for task groups, and can make use of additional known-at-compile-time structure of the declared tasks. For example, it is possible to avoid heap allocations for small enough `async let` child tasks, avoid queues and other mechanisms which a task group must make use of to implement its "by completion order" yielding of values out of `next()`. + +This comes at a price though, async let declarations are less flexible than groups, and this is what we'll explore in this section. + +Specifically, `async let` declarations are not able to express dynamic numbers of tasks executing in parallel, like this group showcases: + +```swift +func toyParallelMap(_ items: [A], f: (A) async -> B) async -> [B] { + return await withTaskGroup(of: (Int, B).self) { group in + var bs = [B?](repeating: nil, count: items.count) + + // spawn off processing all `f` mapping functions in parallel + // in reality, one might want to limit the "width" of these + for i in items.indices { + group.addTask { (i, await f(items[i])) } + } + + // collect all results + for await (i, mapped) in group { + bs[i] = mapped + } + + return bs.map { $0! } + } +} +``` + +In the above `toyParallelMap` the number of child-tasks is _dynamic_ because it depends on the count of elements in the `items` array _at runtime_. Such patterns are not possible to express using `async let` because we'd have to know how many `async let` declarations to create *at compile time*. One might attempt to simulate these by: + +```swift +// very silly example to show limitations of `async let` when facing dynamic numbers of tasks +func toyParallelMapExactly2(_ items: [A], f: (A) async -> B) async -> [B] { + assert(items.count == 2) + async let f0 = f(items[0]) + async let f1 = f(items[1]) + + return await [f0, f1] +} +``` + +And while the second example reads very nicely, it cannot work in practice to implement such parallel map function, because the size of the input `items` is not known (and we'd have to implement `1...n` versions of such a function). + +Another API which is not implementable with `async let` and will require using a task group is anything that requires some notion of completion order. Because `async let` declarations must be awaited on it is not possible to express "whichever completes first", and a task group must be used to implement such API. + +For example, the `race(left:right:)` function shown below, runs two child tasks in parallel, and returns whichever completes first. Such API is not possible to implement using async let and must be implemented using a group: + +```swift +func race(left: () async -> Int, right: () async -> Int) async -> Int { + await withTaskGroup(of: Int.self) { group in + group.addTask { left() } + group.addTask { right() } + + let first = await group.next()! // !-safe, there is at-least one result to collect + group.cancelAll() // cancel the other task + return first + } +} +``` + +#### Comparing with Task, and (not proposed) futures + +It is worth comparing `async let` declarations with the one other API proposed so far that is able to start asynchronous tasks: `Task {}`, and `Task.detached {}`, proposed in [SE-0304: Structured Concurrency](0304-structured-concurrency.md). + +First off, `Task.detached` most of the time should not be used at all, because it does _not_ propagate task priority, task-local values or the execution context of the caller. Not only that but a detached task is inherently not _structured_ and thus may outlive its defining scope. + +This immediately shows how `async let` and the general concept of child-tasks are superior to detached tasks. They automatically propagate all necessary information about scheduling and metadata necessary for execution tracing. And they can be allocated more efficiently than detached tasks. + +So while in theory one can think of `async let` as introducing a (hidden) `Task` or future, which is created at the point of declaration of the `async let` and whose value is retrieved at the `await` in practice, this comparison fails to notice the primary strength of async lets: structured concurrency child-tasks. + +Child tasks in the proposed structured-concurrency model are (intentionally) more restricted than general-purpose futures. Unlike in a typical futures' implementation, a child task does not persist beyond the scope in which it was created. By the time the scope exits, the child task must either have completed, or it will be implicitly awaited. When the scope exits via a thrown error, the child task will be implicitly cancelled before it is awaited. These limitations intentionally preserve the same properties of structured concurrency that explicit task groups provide. + +It is also on purpose, and unlike Tasks and futures that it is not possible to pass a "still being computed" value to another function. With handles or futures one is quite used to "pass the handle" to another function like this: + +```swift +func take(h: Task) async -> String { + return await h.get() +} +``` + + +## Source compatibility + +This change is purely additive to the source language. + +## Effect on ABI stability + +This change is purely additive to the ABI. + +## Effect on API resilience + +All the changes described in this document are additive to the language and are locally scoped, e.g., within function bodies. Therefore, there is no effect on API resilience. + +## Future directions + +### Await in closure capture lists + +Because a `async let` cannot be closed over by an escaping closure, as it would unsafely extend its lifetime beyond the lifetime of the function in which it was declared, developers who need to wait for a value of an `async let` before passing it off to an escaping closure will have to write: + +```swift +func run() async { + async let alcatraz = "alcatraz" + // ... + escapeFrom { // : @escaping () async -> Void + alcatraz // error: cannot refer to 'async let' from @escaping closure + } + // implicitly: await alcatraz +} +``` + +The only legal way to achieve this in the present proposal is to introduce another value and store the awaited value in it: + +```swift +func run() async { + async let alcatraz = "alcatraz" + // ... + let awaitedAlcatraz = await alcatraz + escapeFrom { // : @escaping () async -> Void + awaitedAlcatraz // ok + } +} +``` + +This is correct, yet slightly annoying as we had to invent a new name for the awaited value. Instead, we could utilize capture lists enhanced with the ability to await on such value _at the creation point of the closure_: + +```swift +func escapeFrom(_ f: @escaping () -> ()) -> () {} + +func run() async { + async let alcatraz = "alcatraz" + // ... + escapeFrom { [await alcatraz] in // value awaited on at closure creation + alcatraz // ok + } +} +``` + +This snippet is semantically equivalent to the one before it, in that the `await alcatraz` happens before the `escapeFrom` function is able to run. + +While it is only a small syntactic improvement over the second snippet in this section, it is a welcome and consistent one with prior patterns in Swift, where it is possible to capture a `[weak variable]` in closures. + +The capture list is only necessary for `@escaping` closures, as non-escaping ones are guaranteed to not "out-live" the scope from which they are called, and thus cannot violate the structured concurrency guarantees an `async let` relies on. + +### Custom executors and `async let` + +It is reasonable to request that specific `async let` initializers run on specific executors. + +This is usually not necessary for actor based code, because actor invocations will implicitly "hop" to the right actor as it is called, like in the example below: + +```swift +actor Worker { func work() {} } +let worker: Worker = ... + +async let x = worker.work() // implicitly hops to the worker to perform the work +``` + +There are many reasons it may be beneficial to specify an executor in which child-tasks should run, and the list is by no means exhaustive, but to give an idea, specifying the executor of child-tasks may: + +- proactively fine-tune executors to completely avoid any thread and executor hopping in such tasks, +- execute child-tasks concurrently however _not_ in parallel with the creating task (e.g. make child tasks run on the same serial executor as the calling actor), +- if the child-task work is known to be heavy and blocking, it may be beneficial to delegate it to a specific "blocking executor" which would have a dedicated, small, number of threads on which it would execute the blocking work; thanks to such separation, the main global thread-pool would not be impacted by starvation issues which such blocking tasks would otherwise cause. +- various other examples where tight control over the execution context is required... + +We should be able to allow such configuration based on scope, like this: + +```swift +await withTask(executor: .globalConcurrentExecutor) { + async let x = ... + async let y = ... + // x and y execute in parallel; this is equal to the default semantics +} + +actor Worker { + func work(first: Work, second: Work) async { + await withTask(executor: self.serialExecutor) { + // using any serial executor, will cause the tasks to be completed one-by-one, + // concurrently, however without any real parallelism. + async let x = process(first) + async let y = process(second) + // x and y do NOT execute in parallel + } + } +} +``` + +The details of the API remain to be seen, but the general ability to specify an executor for child-tasks is useful and will be considered in the future. + +## Alternatives considered + +### Explicit futures + +As discussed in the [structured concurrency proposal](0304-structured-concurrency.md#prominent-futures), we choose not to expose futures or `Task`s for child tasks in task groups, because doing so can either undermine the hierarchy of tasks, by escaping from their parent task group and being awaited on indefinitely later, or would result in there being two kinds of future, one of which dynamically asserts that it's only used within the task's scope. `async let` allows for future-like data flow from child tasks to parent, without the need for general-purpose futures to be exposed. + +### "Don't spawn tasks when in cancelled parent" + +It would be very confusing to have `async let` tasks automatically "not run" if the parent task were cancelled. Such semantics are offered by task groups via the `group.asyncUnlessCancelled` API, however would be quite difficult to express using plain `let` declarations, as effectively all such declarations would have to become implicitly throwing, which would sacrifice their general usability. We are convinced that following through with the co-operative cancellation strategy works well for `async let` tasks, because it composes well with how all asynchronous functions should be handling cancellation to begin with: only when they want to, in appropriate places within their execution, and deciding by themselves if they prefer to throw a `Task.CancellationError` or rather return a partial result when cancellation occurs. + +### Requiring an `await`on any execution path that waits for an `async let` + +In initial versions of this proposal, we considered a rule to force an `async let` declaration to be awaited on each control-flow path that the execution of a function might take. This rule turned out to be too simplistic, because it isn't generally possible to annotate all of the control-flow edges that would result in waiting for a child task to complete. The most problematic case involves a control-flow edge due to a thrown exception, e.g., + +```swift +func runException() async { + do { + async let a = f() + try mayFail() // no way to "await a" only along the thrown-error edge; it is an implicit suspension point + ... await a ... + } catch { + ... + } +} +``` + +When `mayFail()` returns normally, we'll later `await a` so that `async let` will be associated with an explicit suspension point. However, when `mayFail()` throws an error, control flow jumps to the `catch` block and must wait for the child task that produces `a` to complete. This latter suspension point is implicit, and there is no direct way to make it explicit that doesn't also involve moving the definition of `a` outside of the `do...catch` block. + +There are other places where there are control-flow edges that will implicitly await the child tasks for `async let`s in scope, e.g., a function with an `async let` in a loop: + +```swift +func runLoop() async { + for e in list { + async let a = f(e) + guard else { + break // cancels and implicitly awaits the task that produces "a" + } + ... await a ... + } + foo() +} +``` + +The most promising approach to marking all `async let` suspension points explicitly involves marking the control-flow edges that can result in a potential suspension point with `await`. For the most recent example, this means using `await break`: + +```swift +func runLoop() async { + for e in list { + async let a = f(e) + guard else { + await break // awaits the child task that produces the value "a" + } + ... await a ... + } + foo() +} +``` + +One would similarly need an `await continue`. For the first example, this means marking the call to `mayFail()` with an `await`, because the potentially-throwing call creates a control-flow edge out of the scope: + +```swift +func runException() async { + do { + async let a = f() + try await mayFail() // awaits the child task that produces a; mayFail() itself may not even be "async" + ... await a ... + } catch { + ... + } +} +``` + +It is somewhat ambiguous what `try await` means in this case, because `mayFail()` may or may not be `async` at all. If it is, then `await` does double-duty covering both the potential suspension points for the call to `mayFail()` as well as the potential suspension point when waiting for the child task along the thrown-error control-flow-edge. + +Similarly, one would need `await throw` for cases where a directly-thrown expression would imply a suspension point to wait for an `async let` child task to complete: + +```swift +func runThrow() async { + do { + async let a = f() + if { + await throw SomeError() // awaits the child task that produces a + } + ... await a ... + } catch { + ... + } +} +``` + +However, not all control-flow edges involving implicit `async let` suspension points have a specific keyword to which we can attach `await`, because some come from fall-through to subsequent code. For such cases, one could have a standalone `await` statement marking that fall through: + +```swift +func runIfFallthrough() async { + if { + async let a = f() + ... code ... + // falling out of this block must await the child task that produces a, so require a freestanding "await" + await + } + ... more code ... +} +``` + +The same would be required in, e.g., the cases of a `switch` statement that introduce an `async let`: + +```swift +func runSwitchCase() async { + switch { + case .a: + async let a = f() + // falling out of this block must await the child task that produces a, so require a freestanding "await" + await + + default: + ... code ... + } + ... code ... +} +``` + +The above is a significant expansion of the grammar: introducing the `await` keyword in front of `break`, `continue`, `throw`, and `fallthrough`; requiring `await` on certain throwing expressions that don't otherwise involve `async` operations; and adding the freestanding `await` statement. It would also need to be coupled with rules that only require the new `await` when it is semantically meaningful. For example, the additional `await` shouldn't be required if all of the `async let` child tasks have already been explicitly awaited in some other manner, e.g., + +```swift +func runIfFallthroughOkay() async { + if { + async let a = f() + ... code ... + if { + ... await a ... + } else { + ... await a ... + } + // no need for "await" here because we've already waited for "a" along all paths + } + ... more code ... +} +``` + +Additionally, every `async` function is already called with an `await`, which covers any suspension points that occur when the function exits. Therefore, a control-flow edge that exits the function should not require any additional `await` for any `async let` child tasks that are awaited. For this reason, there is no `await return`. It also means that other control-flow edges that exit the function need not be annotated. For example: + +```swift +func runThrowsOkay() async { + async let a = f() + if { + throw SomeError() // no need for "await" because this edge exits the function + } + + // no need for "await" at the end because we are exiting the function +} +``` + +The rules above attempt to limit the places in which the new `await` syntaxes are required to only those where they are semantically meaningful, i.e., those places where the `async let` child tasks will not already have had their completion explicitly awaited. The rules are complicated enough that we would not expect programmers to be able to correctly write `await` in all of the places where it is required. Rather, the Swift compiler would need to provide error messages with Fix-Its to indicate the places where additional `await` annotations are required, and those `await`s will remain as an artifact for the reader. + +We feel that the complexity of the solution for marking all suspension points, which includes both the grammar expansion for marking control-flow edges and the flow-sensitive analysis to only require the additional `await` marking when necessary, exceeds the benefits of adding it. Instead, we feel that the presence of `async let` in a block with complicated control flow is sufficient to imply the presence of additional suspension points. + +### Property wrappers instead of `async let` + +The combination of [property wrappers](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md) and [effectful properties](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0310-effectful-readonly-properties.md) implies that one could approximate the behavior of `async let` with a property wrapper, e.g., + +```swift +@AsyncLet var veggies = try await chopVegetables() +``` + +One problem with this approach is that property wrappers cannot provide the semantics of structured concurrency. This becomes more apparent when trying to implement such a property wrapper: + +```swift +@propertyWrapper +class AsyncLet { + var task: Task + + init(wrappedValue fn: @Sendable @escaping @autoclosure () async throws -> Wrapped) { + self.task = Task.detached { // have to produce a detached task; cannot create a child task + try await fn() + } + } + + var wrappedValue: Wrapped { + get async throws { + try await task.value + } + } + + deinit { + // we can cancel the task... + task.cancel() + + // ... but we cannot wait for it to complete, because deinits cannot be async + } +} +``` + +A property-wrapper approach is forced to create unstructured concurrency to capture the task, which is then subject to escaping (e.g., the synthesized backing storage property `_veggies`). Once we have unstructured concurrency, there is no way to get the structure back: the deinitializer cannot wait on completion of the task, so the task would keep running after the `@AsyncLet` property has been destroyed. The lack of structure also affects the compiler's ability to reason about (and therefore optimize) the use of this feature: as a structured concurrency primitive, `async let` can be optimized by the compiler to (e.g.) share storage of its async stack frames with its parent async task, eliminating spurious allocations, and provide more optimal access patterns for the resulting value. To address the semantic and performance issues with using property wrappers, an `@AsyncLet` property wrapper would effectively be hard-coded syntax in the compiler that is property-wrapper-like, but not actually a property wrapper. + +One thing that is lost with the property-wrapper approach is that the definition of a property such as + +```swift +@AsyncLet var veggies = try await chopVegetables() +``` + +loses the `async` keyword. With `async let`, the names introduced are clearly `async` and therefore must be `await`'ed when they are used, as with other `async` entities in the language: + +```swift +async let veggies = chopVegetables() +... +await veggies +``` + +### Braces around the `async let` initializer + +The expression on the right-hand side of an `async let` declaration is executed in a separate child task that is running concurrently with the function that initiates the `async let`. It has been suggested that the task should be called out more explicitly by adding a separate set of braces around the expression, e.g., + +```swift +async let veggies = { try await chopVegetables() } +``` + +The problem with requiring braces is that it breaks the equivalence between the type of the entity being declared (`veggies` is of type `[Vegetable]`) and the value it is initialized with (which now appears to be `@Sendable () async throws -> [Vegetable]`). This equivalence holds throughout nearly all of the language; the only real exception is the `if let` syntax, which strips a level of optionality and is often considered a design mistake in Swift. For `async let`, requiring the braces would become particularly awkward if one were defining a value of closure type: + +```swift +async let closure = { { try await getClosure() } } +``` + +Requiring braces on the right-hand side of `async let` would be a departure from Swift's existing precedent with `let` declarations. In the cases where one is defining a syntactically larger child task, it is reasonable to create and immediately call a closure, which is common practice with `lazy` variables: + +```swift +async let image: Image = { + let data = try await download(url: url) + return try await Image(from: data) +}() +``` + +## Revision history + +After the first review: + +* Expanded the discussion of implicit suspension points in Alternatives Considered with a more comprehensive design sketch for making all suspension points explicit. +* Added discussion of the use of property wrappers instead of `async let` to Alternatives Considered. +* Added discussion about requiring braces around an `async let` initializer expression to Alternatives Considered. + +After initial pitch (as part of Structured Concurrency): + +- renamed back to `async let` to be consistent with updated naming in structured concurrency APIs, +- renamed `async let` to `spawn let` to be consistent with `spawn` usage in the rest of structured concurrency APIs, +- added details of cancellation handling +- added details of await handling diff --git a/proposals/0318-package-creation.md b/proposals/0318-package-creation.md new file mode 100644 index 0000000000..606cf9ad69 --- /dev/null +++ b/proposals/0318-package-creation.md @@ -0,0 +1,365 @@ +# Package Creation + +* Proposal: [SE-0318](0318-package-creation.md) +* Author: [Miguel Perez](https://github.com/miggs597) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Returned for revision** +* Implementation: [apple/swift-package-manager#3514](https://github.com/apple/swift-package-manager/pull/3514) +* Discussion: + [Pitch](https://forums.swift.org/t/pitch-new-command-for-package-creation-and-package-templates/47874/18), + [Review](https://forums.swift.org/t/se-0318-package-creation/) + +# Introduction + +In order to clearly separate the roles of transforming an existing directory of source files into a Swift package, from creating a new package from scratch we propose adding a new command `swift package create`. `swift package init` will continue to exist as is, but will be updated to focus on the former, while the new `swift package create` will focus on the latter. + + +# Motivation + +Currently `swift package init` handle two distinct use cases: + +1. Transforming an existing directory with sources into a Swift package +2. Creating a new package from scratch. + +This one-size-fits-all approach can be confusing and counter-productive, especially for users that are focused on the second use case. It assumes prior knowledge about the command behavior, and specifically about the need to create an empty directory upfront and naming the package after the directory name. + +We feel that separating the two concerns into separate commands, will allow SwiftPM to have better default behavior which is more aligned with the users expectations. + + +## Current Behavior + +### Creating new package + +To create a new package users perform the following steps: + +```console +$ mkdir MyLib +$ cd MyLib +$ swift package init +``` + +Resulting in the following structure: + +```console +. +├── .gitignore +├── Package.swift +├── README.md +├── Sources +│ └── MyLib +│ └── MyLib.swift +└── Tests + └── MyLibTests + └── MyLibTests.swift +``` + +Note that the user has to first make the `MyLib` directory and then navigate into it. + +By default, `swift package init` will use the directory name to define the package name, which can be changed by using the `--name` option. + +By default, `swift package init` will set the package type to a library , which can be changed by using the `—type` option. + + +### Transforming an existing directory of sources into a package + +To transform an existing directory of sources into a package using `swift package init` users perform the following steps: + +```console +$ cd MySources +$ swift package init +``` + +Resulting in the following structure: + +```console +. +├── .gitignore +├── Package.swift +├── README.md +├── Sources +│ └── MySources +│ └── MySources.swift +└── Tests + └── MySourcesTests + └── MySourcesTests.swift +``` + +In this case, SwiftPM will only “fill the gaps”, or in other words only add `Source`, `Tests`, and the other files if they are missing. + +By default, `swift package init` will use the directory name to define the package name, which can be changed by using the` —name` option. + +By default, `swift package init` will set the package type to a library , which can be changed by using the `—type` option. + + +## Problem Definition + +`swift package init` is a utility to get started quickly, and is especially important to new users. The current behavior as described above can often achieve the opposite given its ambiguity and reliance on prior knowledge. Specifically, the default behavior of `swift package init` is geared towards transforming existing source directory to packages, while most new users are interested in creating new programs from scratch so they can experiment with the language. + +A secondary issue is that `swift package init` uses a directory structure template which cannot be customized by the users. Given that SwiftPM is fairly flexible about the package’s directory structure, allowing users to define their own directory structure templates could be a good improvement for those that prefer a different default directory structure. + + +# Proposed Solution + +The identified problems could be solved by the introduction of a new command `swift package create`. This new command would live alongside of `swift package init.` + + `swift package create` would be used to create a new package from scratch. + + `swift package init` would be used to transform pre-existing source directory to a package. + +Both commands will gain the capability to use a templating system such that the directory structure used is customizable by the end user. + + +# Detailed Design + +## New command: `swift package create` + +Following, is the behavior of the new command: + +```console +$ swift package create MyApp +``` + +Will create a new package with the following directory structure: + +```console +. +├── Package.swift +├── Sources +│ └── MyApp +│ └── MyApp.swift +└── Tests + └── MyAppTests + └── MyAppTests.swift +``` + +Note that `swift package create` makes an executable package by default, which is important for new users trying to get their first Swift program up and running. Such users can immediately run the new package: + +```console +$ cd MyApp +$ swift run ## or omit cd, and use swift run --package-path MyApp +[3/3] Linking MyApp +Hello, world! +``` + + +### Customizing the package type + +The `--type` option is used to customize the type of package created. Available options include: `library`, `system-module`, or `executable`. For example + +``` +$ swift package create MyLib --type library +``` + +Will create a library package with the the following directory structure + +```console +. +├── Package.swift +├── Sources +│ └── MyLib +│ └── MyLib.swift +└── Tests + └── MyLibTests + └── MyLibTests.swift +``` + +Or, an example of creating a `system-module` package. + +```console +$ swift package create SysMod --type system-module +``` + +Will create a library package with the the following directory structure + +```console +. +├── Package.swift +└── module.modulemap +``` + + +## User defined templates + +By default, `swift package create` and `swift package init` uses the following directory structure: + +```console +. +├── Package.swift +├── Sources +│ └── +│ └── .swift +└── Tests + └──Tests + └── Tests.swift +``` + +To support use cases in which individuals or teams prefer a different directory structure that they can use consistently in their projects, the proposal introduces a new configuration option named “template”. + +Templates are defined by adding a directory to SwiftPM’s configuration directory, e.g. `~/.swiftpm/configuration/templates/new-package/` + +The template is a Swift package directory that SwiftPM copies and performs transformations on to create the new package. SwiftPM performs the following steps when creating a package from a template: + +1. Copy the template directory to the target location. +2. Substitute string placeholders with values that are derived from the new package request or context. +3. Strip git information from the template location. + + +For example, given a `test` template located in `~/.swiftpm/configuration/templates/new-package/test` with the directory structure: + +```console +. +├── .git +├── .gitignore +├── Package.swift +├── README.md +├── LICENSE.md +└── src + └── MyApp.swift +``` + +The following `Package.swift`: + +```swift +import PackageDescription + +let package = Package( + name: "___NAME___", + dependencies: [ + .package(url: "https://github.com/apple/swift-nio.git", from: "1.0.0"), + .package(url: "https://github.com/apple/swift-crypto.git", from: "1.0.0"), + ], + targets: [ + .executableTarget( + name: "___NAME_AS_C99___", + sources: "src", + dependencies: [ + .product(name: "NIO", package: "swift-nio"), + .product(name: "Crypto", package: "swift-crypto") + ] + ), + ] +) +``` + +And the following `README.md`: + +```markdown +### ___NAME___ + +This is the ___NAME___ package! +``` + +Running `swift package create HelloWorld --template test` + +Will result with the following directory structure: + +```console +. +├── Package.swift +├── .gitignore +├── README.md +├── LICENSE.md +└── src + └── MyApp.swift +``` + +The following `Package.swift`: + +```swift +import PackageDescription + +let package = Package( + name: "HelloWorld", + dependencies: [ + .package(url: "https://github.com/apple/swift-nio.git", from: "1.0.0"), + .package(url: "https://github.com/apple/swift-crypto.git", from: "1.0.0"), + ], + targets: [ + .executableTarget( + name: "HelloWorld", + sources: "src", + dependencies: [ + .product(name: "NIO", package: "swift-nio"), + .product(name: "Crypto", package: "swift-crypto") + ] + ), + ] +) +``` + +And the following `README.md`: + +```markdown +### HelloWorld + +This is the HelloWorld package! +``` + +When running `swift package init` with the `--name` option omitted, the name of the target directory will be used as the package name. + +### Substitutions + +While transforming the template directory into a package, SwiftPM performs string substitutions on all text files, using the following metadata fields: + +1. `___NAME___`: The name provided by the user using the `--name` flag +2. `___NAME_AS_C99___`: The name provided by the user using the `--name` flag, transformed to be C99 compliant + +Future iterations of this feature will include additional metadata fields that can be used in this context. + + +### Defining the default template + +To customize the default template (i.e. when `swift package create` or `swift package init` is invoked without the explicit `--template `argument), users may define a template named “default”, i.e. `~/.swiftpm/configuration/templates/new-package/default` + + +### Adding and updating templates + +Templates are designed to be shared as git repositories. The following commands will be added to SwiftPM to facilitate adding and updating templates: + +`swift package add-template [--name ]` + +Performs `git clone` of the provided URL into `~/.swiftpm/configuration/templates/new-package/,` making the template available to use immediately. The optional `--name` option can be used to set a different name from the one automatically given via the `git clone` operation. + +`swift package update-template ` + +Performs a `git update` on the template found at `~/.swiftpm/configuration/templates/new-package/`. + + +## Impact on SwiftPM + +When processing `swift package create` or `swift package init`, SwiftPM will do the following + +1. If a template is specified with `--template` option: try to load the template and use it as described above, exiting with an error if such template was not found or ran into parsing errors. +2. When no template is specified with `--template` option: + 1. Check if a default template is defined in `~/.swiftpm/configuration/templates/new-package/default.` If one is defined, use it as described in #1 above + 2. If no default template is defined, construct a default `PackageTemplate` based on the `--type` option when provided, or the default type when such is not. + + +## Changes to `swift package init` + +`swift package init` will be slightly updated to reflect it’s focus on transforming existing source directories to packages: + +1. `swift package init` will no longer add a `README.md`, and .`gitignore` files by default, reducing its impact on the existing sources directory. +2. When `swift package init` is used in an empty directory, it will create a new package as it does today but emit a diagnostics message encouraging the user to use `swift package create` in the future, to help transition to the more appropriate command. +3. `swift package init` will accept the new `--template` option and apply it as described above. + + +# Security + +No impact. + + +# Impact on existing packages + +No impact. + + +# Alternatives considered + +The main alternative is to modify the behavior of `swift package init` such that it better caters to the creation of new packages from scratch. The advantage of this alternative is that it maintains the API surface area. The disadvantages are that any changes to make it better for package creation are likely to make it confusing for transforming existing sources to package. More importantly, changes to the existing command may cause impact on users that have automation tied to the current behavior. + +For templates, the main alternative is to use a data file (e.g. JSON) that describes how the package should be constructed. This would hone in the implementation as it defines a finite set of capabilities driven by configuration. This was not selected in order to provide a better user experience, and greater flexibility with respect to including other files in a template. + +# Future Iterations + +In order to provide greater flexibility than what copying a Swift package directory can provide, a future version of SwiftPM could allow packages to be created in a procedural manner. SwiftPM could introduce new APIs that provide a toolbox of functionality for creating and configuration various aspects of packages, and could invoke Swift scripts that create new packages using those APIs. Such scripts could make decisions about what content to create based on input options or other external conditions. These APIs would also function when creating a Swift package from scratch, and or transforming existing sources into a Swift Package. diff --git a/proposals/0319-never-identifiable.md b/proposals/0319-never-identifiable.md new file mode 100644 index 0000000000..e298dcc55b --- /dev/null +++ b/proposals/0319-never-identifiable.md @@ -0,0 +1,68 @@ +# Conform Never to Identifiable + +* Proposal: [SE-0319](0319-never-identifiable.md) +* Author: [Kyle Macomber](https://github.com/kylemacomber) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.5)** +* Implementation: [apple/swift#38103](https://github.com/apple/swift/pull/38103) +* Review: [Forum discussion](https://forums.swift.org/t/se-0319-never-as-identifiable/50246) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0319-never-as-identifiable/50473) + +## Introduction + +This proposal conforms `Never` to `Identifiable` to make it usable as a "bottom type" for generic constraints that require `Identifiable`. + +## Motivation and Proposed Solution + +With the acceptance of [SE-0215](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0215-conform-never-to-hashable-and-equatable.md), `Never` was deemed as being a “blessed bottom type”, but that it wouldn’t implicitly conform to all protocols—instead explicit conformance would be added where valuable. + +The conformance of `Never` to `Equatable` and `Hashable` in SE-0215 was motivated by examples like using `Never` as a generic constraint in types like `Result` and in enumerations. These same use cases motivate the conformance of `Never` to `Identifiable`, which is pervasive in commonly used frameworks like SwiftUI. + +For example, the new `TableRowContent` protocol in SwiftUI follows a "recursive type pattern" and has the need for a primitive bottom type with an `Identifiable` associated type: + +```swift +extension Never: TableRowContent { + public typealias TableRowBody /* conforms to TableRowContent */ = Never + public typealias TableRowValue /* conforms to Identifiable */ = Never +} +``` + +## Detailed design + +```swift +@available(macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0, *) +extension Never: Identifiable { + public var id: Never { + switch self {} + } +} +``` + +## Source compatibility + +If another module has already conformed `Never` to `Identifiable`, the compiler will emit a warning: + +``` +MyFile.swift: warning: conformance of 'Never' to protocol 'Identifiable' was already stated in the type's module 'Swift' +extension Never: Identifiable { + ^ +MyFile.swift: note: property 'id' will not be used to satisfy the conformance to 'Identifiable' + var id: Never { + ^ +``` + +As the warning notes, the new conformance will be used to satisfy the protocol requirement. This difference shouldn't present an observable difference given that an instance of `Never` cannot be constructed. + +## Effect on ABI stability + +This change is additive. + +## Effect on API resilience + +As this change adds new ABI, it cannot be removed in the future without breaking the ABI. + +## Alternatives considered + +#### Add additional "missing" conformances to `Never` (e.g. `CaseIterable`) and other common types + +A more thorough audit of "missing" conformances is called for. With this proposal we chose the narrowest possible scope in order to prioritize the addition of important functionality in a timely manner. diff --git a/proposals/0320-codingkeyrepresentable.md b/proposals/0320-codingkeyrepresentable.md new file mode 100644 index 0000000000..cc8d6f1191 --- /dev/null +++ b/proposals/0320-codingkeyrepresentable.md @@ -0,0 +1,451 @@ +# Allow coding of non `String` / `Int` keyed `Dictionary` into a `KeyedContainer` + +* Proposal: [SE-0320](0320-codingkeyrepresentable.md) +* Author: [Morten Bek Ditlevsen](https://github.com/mortenbekditlevsen) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.6)** +* Implementation: [apple/swift#34458](https://github.com/apple/swift/pull/34458) +* Decision Notes: + [Review #1](https://forums.swift.org/t/se-0320-coding-of-non-string-int-keyed-dictionary-into-a-keyedcontainer/50903), + [Review #2](https://forums.swift.org/t/se-0320-2nd-review-coding-of-non-string-int-keyed-dictionary-into-a-keyedcontainer/51710), + [Rationale](https://forums.swift.org/t/accepted-se-0320-coding-of-non-string-int-keyed-dictionary-into-a-keyedcontainer/52057) + +## Introduction + +The current conformance of Swift's `Dictionary` to the `Codable` protocols has a somewhat-surprising limitation in that dictionaries whose key type is not `String` or `Int` (values directly representable in `CodingKey` types) encode not as `KeyedContainer`s but as `UnkeyedContainer`s. This behavior has caused much confusion for users and I would like to offer a way to improve the situation. + +Swift-evolution thread: [[Pitch] Allow coding of non-`String`/`Int` keyed `Dictionary` into a `KeyedContainer`](https://forums.swift.org/t/pitch-allow-coding-of-non-string-int-keyed-dictionary-into-a-keyedcontainer/44593) + +## Motivation + +The primary motivation for this pitch lays in the much-discussed confusion of this default behavior: + +* [Dictionarys encoding strategy](https://forums.swift.org/t/dictionarys-encoding-strategy/11973) +* [JSON Encoding / Decoding weird encoding of dictionary with enum values](https://forums.swift.org/t/json-encoding-decoding-weird-encoding-of-dictionary-with-enum-values/12995) +* [Bug or PEBKAC](https://forums.swift.org/t/bug-or-pebkac/33796) +* [Using RawRepresentable String and Int keys for Codable Dictionaries](https://forums.swift.org/t/using-rawrepresentable-string-and-int-keys-for-codable-dictionaries/26899) + +The common situations where people have found the behavior confusing include: + +* Using `enum`s as keys (especially when `RawRepresentable`, and backed by `String` or `Int` types) +* Using `String` wrappers (like the generic [Tagged](https://github.com/pointfreeco/swift-tagged) library or custom wrappers) as keys +* Using `Int8` or other `Int*` flavours as keys + +In the various discussions, there are clear and concise explanations for this behavior, but it is also mentioned that supporting encoding of `RawRepresentable` `String` and `Int` keys into keyed containers may indeed be considered to be a bug, and is an oversight in the implementation ([JSON Encoding / Decoding weird encoding of dictionary with enum values, reply by Itai Ferber](https://forums.swift.org/t/json-encoding-decoding-weird-encoding-of-dictionary-with-enum-values/12995/7)). + +There's a bug at [bugs.swift.org](http://bugs.swift.org) tracking the issue: [SR-7788](https://bugs.swift.org/browse/SR-7788) + +Unfortunately, it is too late to change the behavior now: + +1. It is a breaking change with respect to existing behavior, with backwards-compatibility ramifications (new code couldn't decode old data and vice versa), and +2. The behavior is tied to the Swift stdlib, so the behavior would differ between consumers of the code and what OS versions they are on + +Instead, I propose the addition of a new protocol to the standard library. Opting in to this protocol for the key type of a `Dictionary` will allow the `Dictionary` to encode/decode to/from a `KeyedContainer`. + +## Proposed Solution + +I propose adding a new protocol to the standard library: `CodingKeyRepresentable` + +Types conforming to `CodingKeyRepresentable` indicate that they can be represented by a `CodingKey` instance (which they can offer), allowing them to opt in to having dictionaries use their `CodingKey` representations in order to encode into `KeyedContainer`s. + +The opt-in can only happen for a version of Swift where the protocol is available, so the user will be in full control of the situation. For instance I am currently using my own workaround, but once I only support iOS versions running a specific future Swift version with this feature, I could skip my own workaround and rely on this behavior instead. + +I have a draft PR for the proposed solution: [#34458](https://github.com/apple/swift/pull/34458) + +## Examples + +```swift +// Same as stdlib's _DictionaryCodingKey +struct _AnyCodingKey: CodingKey { + let stringValue: String + let intValue: Int? + + init(stringValue: String) { + self.stringValue = stringValue + self.intValue = Int(stringValue) + } + + init(intValue: Int) { + self.stringValue = "\(intValue)" + self.intValue = intValue + } +} + +struct ID: Hashable, CodingKeyRepresentable { + static let knownID1 = ID(stringValue: "") + static let knownID2 = ID(stringValue: "") + + let stringValue: String + + var codingKey: CodingKey { + return _AnyCodingKey(stringValue: stringValue) + } + + init?(codingKey: T) { + stringValue = codingKey.stringValue + } + + init(stringValue: String) { + self.stringValue = stringValue + } +} + +let data: [ID: String] = [ + .knownID1: "...", + .knownID2: "...", +] + +let encoder = JSONEncoder() +try String(data: encoder.encode(data), encoding: .utf8) + +/* +{ + "": "...", + "": "...", +} +*/ +``` + +## Detailed Design + +### Adding `CodingKeyRepresentable` + +The proposed solution adds a new protocol, `CodingKeyRepresentable`: + +```swift +/// A type that can be converted to and from a `CodingKey` value. +/// +/// With a `CodingKeyRepresentable` type, you can switch back and forth between a +/// custom type and a `CodingKey` type without losing the value of +/// the original `CodingKeyRepresentable` type. +/// +/// Conforming a type to `CodingKeyRepresentable` lets you opt-in to encoding and +/// decoding `Dictionary` values keyed by the conforming type to and from a keyed +/// container - rather than an unkeyed container of alternating key-value pairs. +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +public protocol CodingKeyRepresentable { + var codingKey: CodingKey { get } + init?(codingKey: T) +} +``` + +### Handle `CodingKeyRepresentable` conforming types for `Dictionary` encoding + +In the conditional `Encodable` conformance on `Dictionary`, the following extra case can handle such conforming types: + +```swift + } else if #available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *), Key.self is CodingKeyRepresentable.Type { + // Since the keys are CodingKeyRepresentable, we can use the `codingKey` + // to create `_DictionaryCodingKey` instances. + var container = encoder.container(keyedBy: _DictionaryCodingKey.self) + for (key, value) in self { + let codingKey = (key as! CodingKeyRepresentable).codingKey + let dictionaryCodingKey = _DictionaryCodingKey(codingKey: codingKey) + try container.encode(value, forKey: dictionaryCodingKey) + } + } else { + // Keys are Encodable but not Strings or Ints, so we cannot arbitrarily + +``` + +### Handle `CodingKeyRepresentable` conforming types for `Dictionary` decoding + +In the conditional `Decodable` conformance on `Dictionary`, we can similarly handle conforming types: + +```swift + } else if #available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *), let codingKeyRepresentableType = Key.self as? CodingKeyRepresentable.Type { + // The keys are CodingKeyRepresentable, so we should be able to expect a keyed container. + let container = try decoder.container(keyedBy: _DictionaryCodingKey.self) + for dictionaryCodingKey in container.allKeys { + guard let key: Key = codingKeyRepresentableType.init( + codingKey: dictionaryCodingKey + ) as? Key else { + throw DecodingError.dataCorruptedError( + forKey: dictionaryCodingKey, + in: container, + debugDescription: "Could not convert key to type \(Key.self)" + ) + } + let value: Value = try container.decode( + Value.self, + forKey: dictionaryCodingKey + ) + self[key] = value + } + } else { + // We should have encoded as an array of alternating key-value pairs. +``` + +### Add `CodingKeyRepresentable` conformance to `String` and `Int` + +In order to allow the natural use of `String` and `Int` when `CodingKeyRepresentable` is used as a generic constraint, `Int` and `String` will be made to conform to `CodingKeyRepresentable`. + +```swift +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension Int: CodingKeyRepresentable { + public var codingKey: CodingKey { + _DictionaryCodingKey(intValue: self) + } + public init?(codingKey: T) { + if let intValue = codingKey.intValue { + self = intValue + } else { + return nil + } + } +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension String: CodingKeyRepresentable { + public var codingKey: CodingKey { + _DictionaryCodingKey(stringValue: self) + } + public init?(codingKey: T) { + self = codingKey.stringValue + } +} +``` + +### Provide a default implementation to `CodingKeyRepresentable` for `RawRepresentable` types where the raw value is `String` or `Int` + +In many use cases for this proposal, the types that are made to conform to `CodingKeyRepresentable` are already conforming to `RawRepresentable` (with `String` and `Int` raw values). In order to remove friction in these cases, `RawRepresentable` will have a default conformance to `CodingKeyRepresentable` when the raw value is `String` or `Int`: + +```swift +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension RawRepresentable where Self: CodingKeyRepresentable, RawValue == String { + public var codingKey: CodingKey { + _DictionaryCodingKey(stringValue: rawValue) + } + public init?(codingKey: T) { + self.init(rawValue: codingKey.stringValue) + } +} + +@available(macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, *) +extension RawRepresentable where Self: CodingKeyRepresentable, RawValue == Int { + public var codingKey: CodingKey { + _DictionaryCodingKey(intValue: rawValue) + } + public init?(codingKey: T) { + if let intValue = codingKey.intValue { + self.init(rawValue: intValue) + } else { + return nil + } + } +} +``` + +An example of the point of use for the default conformance. Assume that you have a type: `StringWrapper` that already conforms to `RawRepresentable` where `RawValue == String`: + +```swift +extension StringWrapper: CodingKeyRepresentable {} +``` +No boiler plate required. + +### Change internal type `_DictionaryCodingKey` to have non-failable initializers + +In the code above it may be noticed that the internal `_DictionaryCodingKey` type has been changed to have non-failable initializers: + +```swift +/// A wrapper for dictionary keys which are Strings or Ints. +internal struct _DictionaryCodingKey: CodingKey { + internal let stringValue: String + internal let intValue: Int? + + internal init(stringValue: String) { + self.stringValue = stringValue + self.intValue = Int(stringValue) + } + + internal init(intValue: Int) { + self.stringValue = "\(intValue)" + self.intValue = intValue + } + + fileprivate init(codingKey: CodingKey) { + self.stringValue = codingKey.stringValue + self.intValue = codingKey.intValue + } +} +``` + +This change is made to reflect the fact that initialization does in fact never fail, and it reduces the amount of unwrapping that would otherwise be needed elsewhere in the internal use of the type. + +## Impact on Existing Code + +No direct impact, since adoption of this protocol is additive. + +However, special care must be taken in *adopting* the protocol, since adoption on any type `T` which has previously been encoded as a dictionary key can introduce backwards incompatibility with archives. It is always safe to adopt `CodingKeyRepresentable` on new types, or types newly-conforming to `Codable`. + +## Other Considerations + +### Conforming stdlib types to `CodingKeyRepresentable` + +Along the above lines, we do not propose conforming any existing stdlib or Foundation type to `CodingKeyRepresentable` due to backwards-compatibility concerns. Should end-user code require this conversion on existing types, we recommend writing wrapper types which conform on those types' behalf (for example, a `MyUUIDWrapper` which contains a `UUID` and conforms to `CodingKeyRepresentable` to allow using `UUID`s as dictionary keys directly). + +### Adding an `AnyCodingKey` type to the standard library + +Since types that conform to `CodingKeyRepresentable` will need to supply a `CodingKey`, most likely generated dynamically from type contents, this may be a good time to introduce a general key type which can take on any `String` or `Int` value it is initialized from. + +`Dictionary` already uses exactly such a key type internally (`_DictionaryCodingKey`), as do `JSONEncoder` / `JSONDecoder` with `_JSONKey` (and `PropertyListEncoder` / `PropertyListDecoder` with `_PlistKey`), so generalization could be useful. The implementation of this type could match the implementation of `_AnyCodingKey` provided above. + +## Alternatives Considered + +### Why not just make the type conform to `CodingKey` directly? + +For two reasons: + +1. In the rare case in which a type already conforms to `CodingKey`, this runs the risk of behavior-breaking changes +2. `CodingKey` requires exposure of a `stringValue` and `intValue` property, which are only relevant when encoding and decoding; forcing types to expose these properties arbitrarily seems unreasonable + +### Why not refine `RawRepresentable`, or use a `RawRepresentable where RawValue == CodingKey` constraint? + +`RawRepresentable` conformance for types indicates a lossless conversion between the source type and its underlying `RawValue` type; this conversion is often the "canonical" conversion between a source type and its underlying representation, most commonly between `enum`s backed by raw values, and option sets similarly backed by raw values. + +In contrast, we expect conversion to and from `CodingKey` to be *incidental* , and representative only of the encoding and decoding process. We wouldn't suggest (or expect) a type's canonical underlying representation to be a `CodingKey`, which is what a `protocol CodingKeyRepresentable: RawRepresentable where RawValue == CodingKey` would require. Similarly, types which are already `RawRepresentable` with non- `CodingKey` raw values couldn't adopt conformance this way, and a big impetus for this feature is allowing `Int`- and `String`-backed `enum`s to participate as dictionary coding keys. + +### Why not use an Associated Type for `CodingKey` +It was suggested during the pitch phase to use an associated type for the `CodingKey` in the `CodingKeyRepresentable` protocol. + +The presented use case was perfectly valid - and demonstrated using the following example: + +```swift +enum MyKey: Int, CodingKey { + case a = 1 + case b = 3 + case c = 5 + + var intValue: Int? { rawValue } + + var stringValue: String { + switch self { + case .a: return "a" + case .b: return "b" + case .c: return "c" + } + } + + init?(intValue: Int) { self.init(rawValue: intValue) } + + init?(stringValue: String) { + guard let rawValue = RawValue(stringValue) else { return nil } + self.init(rawValue: rawValue) + } +} + +struct MyCustomType: CodingKeyRepresentable { + typealias CodingKey = MyKey + + var useB = false + + var codingKey: CodingKey { + useB ? .b : .a + } + + init?(codingKey: CodingKey) { + switch codingKey { + case .a: useB = false + case .b: useB = true + case .c: return nil // .c is unsupported + } + } +} +``` + +An analysis of this suggestion hints that the non-zero cost of doing type erasure for pulling out the key values at the consuming site might not carry it's weight ([https://forums.swift.org/t/pitch-allow-coding-of-non-string-int-keyed-dictionary-into-a-keyedcontainer/44593/9](https://forums.swift.org/t/pitch-allow-coding-of-non-string-int-keyed-dictionary-into-a-keyedcontainer/44593/9)): + +Because `associatedtype` s have non-zero cost on the consuming side (e.g. checking for `CodingKeyRepresentable` conformance, using the key type), I think that the associated type definition would need to carry its weight. Despite the name, I think that the key difference between `CodingKeyRepresentable` and `RawRepresentable` is that the *identity* of the `RawValue` type is crucial to `RawRepresentable` , but not so in the `CodingKeyRepresentable` case. + +On the *consuming* side of `CodingKeyRepresentable.codingKey` (e.g. in `Dictionary` ), I don't believe key type identity is necessarily useful enough: + +* The main use for the `.codingKey` value is immediate retrieval of the underlying `String` / `Int` values. `Dictionary` would either pull those values out for immediate use and throw away the original key +* In a non-generic context (or even one not predicated on `CodingKeyRepresentable` conformance), you can't meaningfully get at the key type. The type-erasure song and dance you have to do to get the key values won't be able to hand you a typed key (and the pain of doing that dance is that because it doesn't make sense to expose a public protocol for doing the erasure, every consumer that wants to do this needs to reinvent the wheel and add another protocol for doing it; we had to do it a few times for `Optional` s and it's a bit of a shame) +* Even if it were necessary to get a meaningful key type, the majority use-case for this feature, I believe, will be to provide dynamic-value keys for non-enumerable types (e.g. `struct` s like `UUID` [though yes, we can't make it conform]); for these types, you can't necessarily define a `CodingKey` s *`enum`* and instead, you'd likely want to use a more generic key type like `AnyCodingKey` (which by definition doesn't have identity) + +On the *producing* side (e.g. in `MyCustomType` ), I'm also not sure the utility is necessarily enough: in general, the majority of `CodingKeyRepresentable` types (I believe) will only really care about the `String` / `Int` values of the keys, since they will be initialized dynamically (again, I think of `UUID` initialization from a `CodingKey.stringValue` — you can do this from *any* `CodingKey` ). + +I believe that the constrained `MyKey` example above will be the minority use-case, but expressed without the `associatedtype` constraint too: + +```swift +enum MyKey: Int, CodingKey { + case a = 1, b = 3, c = 5 + + // There are several ways to express this, just an example: + init?(codingKey: CodingKey) { + if let key = codingKey.intValue.flatMap(Self.init(intValue:)) { + self = key + } else if let key = Self(stringValue: codingKey.stringValue) { + self = key + } else { + return nil + } + } +} + +struct MyCustomType: CodingKeyRepresentable { + var useB = false + + var codingKey: CodingKey { + useB ? MyKey.b : MyKey.a + } + + init?(codingKey: CodingKey) { + switch MyKey(codingKey: codingKey) { + case .a: useB = false + case .b: useB = true + default: return nil + } + } +} +``` + +I personally find this equally as expressive, and I think that not requiring the associated type gives more flexibility without a significant loss, especially with non- `enum` types in mind. + + +### Add workarounds to each `Encoder` / `Decoder` + +Following a suggestion from @itaiferber, I have previously tried to provide a solution to this issue — not in general, but instead solving it by providing a `DictionaryKeyEncodingStrategy` for `JSONEncoder` : [#26257](https://github.com/apple/swift/pull/26257) + +The idea there was to be able to express an opt-in to the new behavior directly in the `JSONEncoder` and `JSONDecoder` types by vending a new encoding/decoding 'strategy' configuration. I have since changed my personal opinion about this and I believe that the problem should not just be fixed for specific `Encoder` / `Decoder` pairs, but rather for all. + +The implementation of this was not very pretty, involving casts and iterations over the dictionaries to be encoded/decoded. + +### Await design of `newtype` + +I have heard mentions of a `newtype` design, that basically tries to solve the issue that the [Tagged](https://github.com/pointfreeco/swift-tagged) library solves: namely creating type safe wrappers around other primitive types. + +I am in no way an expert in this, and I don't know how this would be implemented, but *if* it were possible to tell that `SomeType` is a `newtype` of `String`, then this could be used to provide a new implementation in the `Dictionary` `Codable` conformance, and since this feature does not exist in older versions of Swift (providing that this is a feature that requires changes to the Swift run-time), then adding this to the `Dictionary` `Codable` conformance would not be behavior breaking. + +But those are an awful lot of ifs and buts, and it only solves one of the issues that people appear to run in to (the wrapping issue) — and not for instance `String` based enums or `Int8`-based keys. + +### Do nothing + +It is of course possible to handle this situation manually during encoding. + +A rather unintrusive way of handling the situation is by using a property wrapper as suggested here: [CodableKey](https://forums.swift.org/t/bug-or-pebkac/33796/12). + +This solution needs to be applied for each `Dictionary` and is a quite elegant workaround. But it is still a workaround for something that could be fixed in the stdlib. + +A few drawbacks to the property wrapper solution were given during the pitch phase: + +* Using `Int8` (or any other numeric stdlib type for that matter) as key requires it to conform to `CodingKey` . This conformance would have to come from the stdlib to prevent conformance collisions across e.g. Swift packages. And IMHO those types shouldn't provide a `CodingKey` conformance per se... +* It's not straightforward to simply encode/decode e.g. a `Dictionary` that is not a property of another `Codable` type (also mentioned in the example in the linked post). +* It's impossible to *add* a `Codable` conformance to an object that is already defined. So if I define a struct ( `MyType` ) having a `Dictionary` in one file, I can't simply put an `extension MyType: Codable { /* ... */ }` into another file. + +## Acknowledgements +Many thanks to [Itai Ferber](https://github.com/itaiferber) for providing input and feedback, for revising the pitch and for helping me shape the overall direction. + +Also many thanks to everyone providing feedback on the pitch and the first proposal review. + +## Revision history + +### Review changes + +Changes after the first review: + +* added conformance for `String` and `Int` to `CodingKeyRepresentable`. +* changed the initializer of `CodingKeyRepresentable` to be generic +* added default implementations for the conformance for `RawRepresentable` (with `String` and `Int` raw values). +* made the initializers of the internal `_DictionaryCodingKey` non-failable. + diff --git a/proposals/0321-package-registry-publish.md b/proposals/0321-package-registry-publish.md new file mode 100644 index 0000000000..098b217027 --- /dev/null +++ b/proposals/0321-package-registry-publish.md @@ -0,0 +1,325 @@ +# Package Registry Service - Publish Endpoint + +* Proposal: [SE-0321](0321-package-registry-publish.md) +* Authors: [Whitney Imura](https://github.com/whitneyimura), + [Mattt Zmuda](https://github.com/mattt) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Accepted (2021-09-01)** +* Implementation: [apple/swift-package-manager#3671](https://github.com/apple/swift-package-manager/pull/3671) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0321-package-registry-service-publish-endpoint/51660) + +## Introduction + +The [package registry service][SE-0292] defines endpoints for fetching packages. + +This proposal extends the existing [package registry specification][Registry.md] +with endpoints for publishing a package release. + +## Motivation + +A package registry is responsible for determining +which package releases are made available to a consumer. + +Currently, the availability of a package release +is determined by an out-of-band process. +For example, +a registry may consult an index of public Swift packages +and make releases available for each tag with a valid version number. + +Having a standard endpoint for publishing a new release to a package registry +would empower maintainers to distribute their software +and promote interoperability across service providers. + +## Proposed solution + +We propose to add the following endpoint to the registry specification: + +| Method | Path | Description | +| ------- | ----------------------------- | -------------------------| +| `PUT` | `/{scope}/{name}/{version}` | Create a package release | + +The goal of this proposal is to provide enough definition to ensure +a secure, robust mechanism for distributing software +while allowing individual registries enough flexibility in their +governance and operation. +For instance, +support for this endpoint would be optional, +so package registries may elect not to allow packages to be published. +And because there's an expectation of durability — +that is, package releases aren't removed after they're published — +registries make the ultimate determination of what is made available. + +## Detailed design + +This proposal amends the registry specification with a new, optional endpoint +that a registry may implement to support the publication of packages +through the web service interface. +To understand what the feature does and how it works, +consider the following use case: + +A maintainer of an open-source Swift package (`mona.LinkedList`) +creates a new release (version `1.1.1`), +and wants to submit it to a registry (`packages.example.com`) for distribution. + +First, they run the `swift package archive-source` subcommand +to generate a Zip file (`LinkedList-1.1.1.zip`) of their package. + +```console +$ swift package archive-source +``` + +Next, they upload their release to a package registry +by making the following request: + +```console +$ curl -X PUT --netrc \ + -H "Accept: application/vnd.swift.registry.v1+json" \ + -F source-archive="@LinkedList-1.1.1.zip" \ + "https://registry.example.com/mona/LinkedList?version=1.1.1" +``` + +The registry can respond to this request synchronously or asynchronously. +This allows the server an opportunity to perform any necessary +analysis and processing to ensure software quality and update its data stores. + +After receiving and processing this request, +the registry can make `mona.LinkedList` at version `1.1.1` available +by including it in the response to `GET /mona/LinkedList`. + +```console +$ curl -X GET -H "Accept: application/vnd.swift.registry.v1+json" \ + "https://registry.example.com/mona/LinkedList" \ + | jq ".[] | keys" +[ + "1.0.0", + "1.1.0", + "1.1.1", +] +``` + +The next time a developer with a package that depends on `mona.LinkedList` +resolves the dependencies of that package, +Swift Package Manager would see `1.1.1`, +and may attempt to update to this new version. +If the version is selected, +the client would download the source archive for this release +by sending the request `GET /mona/LinkedList/1.1.1.zip`. + +## Security + +Although this proposal has no direct impact on Swift Package Manager, +it's important to consider the security implications of +introducing a publishing endpoint to the Swift package ecosystem. +To do this, +we employ the + +[STRIDE] + mnemonic below: + +### Spoofing + +An attacker could attempt to impersonate a package maintainer +in order to publish a new release containing malicious code. + +Because the likelihood and potential impact of such an attack is high, +registry service providers should take all necessary precautions. +The registry specification recommends the use of multi-factor authentication +for all requests to publish a package release. + +Additional countermeasures like +rate-limiting suspicious requests and +analyzing uploaded source archives +can also help mitigate the risk of this kind of attack. + +An attacker could also attempt to trick users into downloading malicious code +by publishing a package with an identifier similar to a legitimate one +(for example, `4pple.swift-nio`, which looks like `apple/swift-nio`). +A registry can mitigate typosquatting attacks like this +by comparing the similarity of a new submission to existing package names +with a string metric like [Damerau–Levenshtein distance]. + +### Tampering + +An attacker could maliciously tamper with a generated source archive +in an attempt to exploit +a known vulnerability like [Zip Slip], +or a common software weakness like susceptibility to a [Zip bomb]. + +Registry services should take care to +identify and protect against these kinds of attacks +in its implementation of source archive decompression. + +To further improve the security of package submissions, +a registry could restrict publishing to trusted clients, +for which a chain of custody can be established. +(This is effectively the "pull" model described above). + +### Repudiation + +A dispute could arise between a package maintainer and a registry +about the content or existence of a package release. + +This proposal doesn't specifically provide a mechanism +for resolving such a dispute. +However, the design supports a variety of possible solutions. +For example, +a software bill of materials and the use of digital signatures +can both provide non-repudiation guarantees +about the provenance of package release artifacts. + +### Information disclosure + +A user could inadvertently expose credentials +when uploading a source archive for a package release. + +This threat isn't substantially different from that of +leaking credentials in source code with version management software, +so similar strategies can be employed here. +For example, +registry services can help minimize this risk +by rejecting any submissions that [contain sensitive information][secret scanning]. + +### Denial of service + +An attacker could upload large payloads +in an attempt to reduce the availability of a registry. + +This kind of attack is typical for any web service +with an endpoint for uploading resources. +A registry can mitigate this threat using defensive coding practices like +performing authentication checks before processing request bodies, +limiting the maximum allowed size of a message payload, and +routing requests through a reverse proxy or load balancer. + +### Escalation of privilege + +It's desirable for a registry to have information about +the content of a release submitted for publishing, +such as the package's supported platforms, products, and dependencies. +Swift package manifest files are executable code +and must be evaluated by the Swift toolchain to determine this information. +An attacker could construct a malicious `Package.swift` file +containing system calls in an attempt to perform remote code execution. + +Registry services should take care to evaluate package manifest files +in an unprivileged container to mitigate the risk of evaluating untrusted code. + +## Impact on existing packages + +This feature provides a mechanism for package maintainers and registries +to migrate existing packages from the current URL-based system +to the new registry scheme. + +The specific strategy for rolling out this functionality +is something to be determined by each registry operator +in advance of this feature. + +## Alternatives considered + +### Endpoint for scope registration + +This proposal sets no policies for how +package scopes are registered or verified. + +### Endpoint for publishing with "pull" model + +Many package managers and artifact repository services +follow what we describe as a *"push"* model of publication: +When a maintainer wants to releases a new version of their software, +they produce a build locally and push the resulting artifact to a server. + +For example, +a developer can distribute their Ruby library +by building a `.gem` archive and pushing it to a server like [RubyGems.org]. + +```console +$ gem build octokit.gemspec +$ gem push octokit-4.20.0.gem +``` + +This model has the benefit of operational simplicity and flexibility. +For example, +maintainers have an opportunity to digitally sign artifacts +before uploading them to the server. + +Alternatively, +a system might incorporate build automation techniques like +continuous integration (CI) and continuous delivery (CD) +into what we describe as a *"pull"* model: +When a maintainer wants to release a new version of their software, +their sole responsibility is to notify the registry; +the server does all the work of downloading the source code +and packaging it up for distribution. + +For example, +in addition to supporting the "push" model, +[Docker Hub] can [automatically build][autobuilds] images from source code +push the built image to a repository. + +This model can provide strong guarantees about +reproducibility, quality assurance, and software traceability. + +Initial drafts for this proposal +included separate endpoints for publishing with the "pull" and "push" models, +with a preference for the former and its stronger guarantees of traceability. +However, +we determined that while these models provide +a useful framework for understanding software distribution models, +they are both accommodated by a single endpoint; +a "pull" is equivalent a "push" +where the client and server are a single entity. + +## Future directions + +### Swift Package Manager subcommand for publishing + +Swift Package Manager could be updated to add +a new `swift package publish` subcommand, +that provides a more convenient interface for publishing packages to a registry. +For example, +it could automatically read the configuration in +`.swiftpm/config/registries.json` +to determine the correct registry endpoint, +or read the user's `.netrc` file to authenticate the request. + +The command could also subsume `swift package archive-source` +and perform additional tasks before uploading, +such as generating a software bill of materials +or signing the source archive. + +This feature wasn't included in the proposal +because it's unnecessary for the core publishing functionality. +We are also concerned that this command +could bloat to the command-line interface +and undermine the benefits of publishing within a CI/CD system. +However, +if the community finds this to be a useful feature, +we'd be happy to include it in an amendment to our proposal. + +### Mechanism for syndicating publishing activity + +A registry could syndicate new releases through +an [activity stream][activitystreams] or [RSS feed][rss]. +This functionality could be used as an information source by package indexes +or to provide federation across different registries. + +### Transparency logs + +Similar to a syndication feed, +each new package release could be added to an append-only log +like [Trillian] or [sigstore]. + +[activitystreams]: https://www.w3.org/TR/activitystreams-core/ "Activity Streams 2.0" +[autobuilds]: https://docs.docker.com/docker-hub/builds/ "Docker Hub: Set up Automated Builds" +[Damerau–Levenshtein distance]: https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance "Damerau–Levenshtein distance" +[Docker Hub]: https://hub.docker.com +[Registry.md]: https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md "Swift Package Registry Service Specification" +[rss]: https://validator.w3.org/feed/docs/rss2.html "RSS 2.0 specification" +[RubyGems.org]: https://rubygems.org/ +[SE-0292]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0292-package-registry-service.md "Package Registry Service" +[secret scanning]: https://docs.github.com/en/github/administering-a-repository/about-secret-scanning +[sigstore]: https://sigstore.dev/ "sigstore: A non-profit, public good software signing & transparency service" +[STRIDE]: https://en.wikipedia.org/wiki/STRIDE_(security) "STRIDE (security)" +[Trillian]: https://github.com/google/trillian "Trillian: A transparent, highly scalable and cryptographically verifiable data store." +[Zip bomb]: https://en.wikipedia.org/wiki/Zip_bomb "Zip bomb" +[Zip Slip]: https://snyk.io/research/zip-slip-vulnerability "Zip Slip Vulnerability" diff --git a/proposals/0322-temporary-buffers.md b/proposals/0322-temporary-buffers.md new file mode 100644 index 0000000000..05be569820 --- /dev/null +++ b/proposals/0322-temporary-buffers.md @@ -0,0 +1,320 @@ +# Temporary uninitialized buffers + +* Proposal: [SE-0322](0322-temporary-buffers.md) +* Author: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.6)** +* Implementation: [apple/swift#37666](https://github.com/apple/swift/pull/37666) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0322-temporary-uninitialized-buffers/52532) + +## Introduction + +This proposal introduces new Standard Library functions for manipulating temporary buffers that are preferentially allocated on the stack instead of the heap. + +Swift-evolution thread: [[Pitch] Temporary uninitialized buffers](https://forums.swift.org/t/pitch-temporary-uninitialized-buffers/48954) + +## Motivation + +Library-level code often needs to deal with C functions, and C functions have a wide variety of memory-management approaches. A common way to handle buffers of value types (i.e. structures) in C is to have a caller allocate a buffer of sufficient size and pass it to a callee to initialize; the caller is then responsible for deinitializing the buffer (if non-trivial) and then deallocating it. In C or Objective-C, it's easy enough to stack-allocate such a buffer, and the logic to switch to the heap for a larger allocation is pretty simple too. This sort of pattern is pervasive: + +```c +size_t tacoCount = ...; +taco_fillings_t fillings = ...; + +taco_t *tacos = NULL; +taco_t stackBuffer[SOME_LIMIT]; +if (tacoCount < SOME_LIMIT) { + tacos = stackBuffer; +} else { + tacos = calloc(tacoCount, sizeof(taco_t)); +} + +taco_init(tacos, tacoCount, &fillings); + +// do some work here +ssize_t tacosEatenCount = tacos_consume(tacos, tacoCount); +if (tacosEatenCount < 0) { + int errorCode = errno; + fprintf(stderr, "Error %i eating %zu tacos: %s", errorCode, tacoCount, strerror(errorCode)); + exit(EXIT_FAILURE); +} + +// Tear everything down. +taco_destroy(tacos, tacoCount); +if (buffer != stackBuffer) { + free(buffer); +} +``` + +In C++, we can make judicious use of `std::array` and `std::vector` to achieve the same purpose while generally preserving memory safety. + +But there's not really any way to express this sort of transient buffer usage in Swift. All values must be initialized in Swift before they can be used, but C typically claims responsibility for initializing the values passed to it, so a Swift caller ends up initializing the values _twice_. The caller can call `UnsafeMutableBufferPointer.allocate(capacity:)` to get uninitialized memory, of course. `allocate(capacity:)` places buffers on the heap by default, and the optimizer can only stack-promote them after performing escape analysis. + +Since Swift requires values be initialized before being used, and since escape analysis is [undecidable](https://duckduckgo.com/?q=escape+analysis+undecidable) for non-trivial programs, it's not possible to get efficient (i.e. stack-allocated and uninitialized) temporary storage in Swift. + +It is therefore quite hard for developers to provide Swift overlays for lower-level C libraries that use these sorts of memory-management techniques. This means that an idiomatic Swift program using a C library's Swift overlay will perform less optimally than the same program would if written in C. + +## Proposed solution + +I propose adding a new transparent function to the Swift Standard Library that would allocate a buffer of a specified type and capacity, provide that buffer to a caller-supplied closure, and then deallocate the buffer before returning. The buffer would be passed to said closure in an uninitialized state and treated as uninitialized on closure return—that is, the closure would be responsible for initializing and deinitializing the elements in the buffer. + +A typical use case might look like this: + +```swift +// Eat a temporary taco buffer. A buffet, if you will. +try taco_t.consume(count: tacoCount, filledWith: ...) + +// MARK: - Swift Overlay Implementation + +extension taco_t { + public static func consume(count: Int, filledWith fillings: taco_fillings_t) throws { + try withUnsafeTemporaryAllocation(of: taco_t.self, capacity: count) { buffer in + withUnsafePointer(to: fillings) { fillings in + taco_init(buffer.baseAddress!, buffer.count, fillings) + } + defer { + taco_destroy(buffer.baseAddress!, buffer.count) + } + + let eatenCount = tacos_consume(buffer.baseAddress!, buffer.count) + guard eatenCount >= 0 else { + let errorCode = POSIXErrorCode(rawValue: errno) ?? .ENOTHUNGRY + throw POSIXError(errorCode) + } + } + } +} +``` + +The proposed function allows developers to effectively assert to the compiler that the buffer pointer used in the closure cannot escape the closure's context (even if calls are made to non-transparent functions that might otherwise defeat escape analysis.) Because the compiler then "knows" that the pointer does not escape, it can optimize much more aggressively. + +A library developer going the extra mile would probably want to produce an interface that we would consider idiomatic in Swift. With the proposed function, the developer would be able to build such an interface without sacrificing C's performance. (The exact design of the hypothetical Swift "taco" interface is beyond the scope of this proposal.) + +## Detailed design + +A new free function would be introduced in the Standard Library: + +```swift +/// Provides scoped access to a buffer pointer to memory of the specified type +/// and with the specified capacity. +/// +/// - Parameters: +/// - type: The type of the buffer pointer being temporarily allocated. +/// - capacity: The capacity of the buffer pointer being temporarily +/// allocated. +/// - body: A closure to invoke and to which the allocated buffer pointer +/// should be passed. +/// +/// - Returns: Whatever is returned by `body`. +/// +/// - Throws: Whatever is thrown by `body`. +/// +/// This function is useful for cheaply allocating storage for a sequence of +/// values for a brief duration. Storage may be allocated on the heap or on the +/// stack, depending on the required size and alignment. +/// +/// When `body` is called, the contents of the buffer pointer passed to it are +/// in an unspecified, uninitialized state. `body` is responsible for +/// initializing the buffer pointer before it is used _and_ for deinitializing +/// it before returning, but deallocation is automatic. +/// +/// The implementation may allocate a larger buffer pointer than is strictly +/// necessary to contain `capacity` values of type `type`. The behavior of a +/// program that attempts to access any such additional storage is undefined. +/// +/// The buffer pointer passed to `body` (as well as any pointers to elements in +/// the buffer) must not escape—it will be deallocated when `body` returns and +/// cannot be used afterward. +public func withUnsafeTemporaryAllocation( + of type: T.Type, + capacity: Int, + _ body: (UnsafeMutableBufferPointer) throws -> R +) rethrows -> R +``` + +Two additional free functions, composed atop that one, would also be introduced for dealing with a raw buffer or a pointer to a single value: + +```swift +/// Provides scoped access to a raw buffer pointer with the specified byte count +/// and alignment. +/// +/// - Parameters: +/// - byteCount: The number of bytes to temporarily allocate. `byteCount` must +/// not be negative. +/// - alignment: The alignment of the new, temporary region of allocated +/// memory, in bytes. +/// - body: A closure to invoke and to which the allocated buffer pointer +/// should be passed. +/// +/// - Returns: Whatever is returned by `body`. +/// +/// - Throws: Whatever is thrown by `body`. +/// +/// This function is useful for cheaply allocating raw storage for a brief +/// duration. Storage may be allocated on the heap or on the stack, depending on +/// the required size and alignment. +/// +/// When `body` is called, the contents of the buffer pointer passed to it are +/// in an unspecified, uninitialized state. `body` is responsible for +/// initializing the buffer pointer before it is used _and_ for deinitializing +/// it before returning, but deallocation is automatic. +/// +/// The implementation may allocate a larger buffer pointer than is strictly +/// necessary to contain `byteCount` bytes. The behavior of a program that +/// attempts to access any such additional storage is undefined. +/// +/// The buffer pointer passed to `body` (as well as any pointers to elements in +/// the buffer) must not escape—it will be deallocated when `body` returns and +/// cannot be used afterward. +public func withUnsafeTemporaryAllocation( + byteCount: Int, + alignment: Int, + _ body: (UnsafeMutableRawBufferPointer) throws -> R +) rethrows -> R + +/// Provides scoped access to a pointer to memory of the specified type. +/// +/// - Parameters: +/// - type: The type of the pointer to allocate. +/// - body: A closure to invoke and to which the allocated pointer should be +/// passed. +/// +/// - Returns: Whatever is returned by `body`. +/// +/// - Throws: Whatever is thrown by `body`. +/// +/// This function is useful for cheaply allocating storage for a single value +/// for a brief duration. Storage may be allocated on the heap or on the stack, +/// depending on the required size and alignment. +/// +/// When `body` is called, the contents of the pointer passed to it are in an +/// unspecified, uninitialized state. `body` is responsible for initializing the +/// pointer before it is used _and_ for deinitializing it before returning, but +/// deallocation is automatic. +/// +/// The pointer passed to `body` must not escape—it will be deallocated when +/// `body` returns and cannot be used afterward. +public func withUnsafeTemporaryAllocation( + of type: T.Type, + _ body: (UnsafeMutablePointer) throws -> R +) rethrows -> R +``` + +Note the functions will be marked with attributes that ensure they are emitted into the calling frame. This is consistent with the annotations on most other pointer manipulation functions. + +### New builtins + +The proposed functions will need to invoke a new builtin function equivalent to C's `alloca()`, which I have named `Builtin.stackAlloc()`. A second builtin, `Builtin.stackDealloc()`, must then be invoked after the call to the body closure returns or throws in order to clean up the stack. + +If the alignment and size are known at compile-time, the compiler can convert a call to `stackAlloc()` into a single LLVM `alloca` instruction, which can then ultimately be lowered to a single CPU instruction adjusting the stack pointer. If either argument needs to be computed at runtime, a dynamic stack allocation can instead be emitted by the compiler. + +### Location of allocated buffers + +The proposed functions do _not_ guarantee that their buffers will be stack-allocated. This omission is intentional: guaranteed stack-allocation would make this feature equivalent to C99's variable-length arrays—a feature that is extremely easy to misuse and which is the cause of many [real-world security vulnerabilities](https://duckduckgo.com/?q=cve+variable-length+array). Instead, the proposed functions should stack-promote aggressively, but heap-allocate (just as `UnsafeMutableBufferPointer.allocate(capacity:)` does today) when passed overly large sizes. + +A common C approach is to say "anything larger than _n_ bytes uses `calloc()`", where _n_ is some moderately-sized constant. For allocations smaller than _n_, the compiler will simply allocate on the stack. This is as safe as the common Swift expression `let x = T()`, where `T` is a type whose size is smaller than _n_. + +### Runtime support for more complex heuristics + +For allocations **larger than** _n_, the compiler will emit a conditionalized call to a new Standard Library function `_swift_stdlib_isStackAllocationSafe()`. This function will be able to quickly determine if the desired allocation can fit in the available stack space. If it approves, the allocation will be performed on the stack. Otherwise, it will be performed on the heap. + +This function will be a no-op on platforms where the current stack cannot be queried (i.e. there is no platform API to do so) or cannot be queried efficiently (i.e. doing so is more expensive than allocating to the heap.) Because the Standard Library owns this function, enhancements can be made in future Swift revisions that improve the heuristic without requiring callers to recompile. + +The body of this function is an implementation detail of the Swift Standard Library and is **not** part of this proposal. A possible implementation might ask the operating system for details about the current thread's stack (e.g. by calling `GetCurrentThreadStackLimits()` on Windows or `pthread_getattr_np()` on Linux.) + +### Custom heuristics + +Several contributors to the pitch and review threads asked about providing their own heuristic for stack- vs. heap-allocation. It is unlikely that a caller will have sufficient additional information about the state of the program such that it can make better decisions about stack promotion than the compiler and/or Standard Library. An additional argument to the proposed functions that forces stack allocation is too visible a solution. Instead, callers that really do need larger stack allocations can pass `-stack-alloc-limit n` to the compiler frontend in order to specify a larger limit than the default for stack promotion. + +## Source compatibility + +This is new API, so there are no source compatibility considerations. + +## Effect on ABI stability + +This is new API, so there are no ABI stability considerations. The proposed functions should always be inlined into their calling frames, so they should be back-deployable to older Swift targets. + +## Effect on API resilience + +The addition of the proposed functions does not affect API resilience. If they were removed in a future release, it would be a source-breaking change but not an ABI-breaking change, because the proposed functions should always be inlined into their calling frames. + +## Alternatives considered + +In the pitch thread for this proposal, a number of alternatives were discussed: + +### Doing nothing + +* A number of developers both at Apple and in the broader Swift community have indicated that the performance costs cited above are measurably affecting the performance of their libraries and applications. +* The proposed functions would let developers build higher-order algorithms, structures, and interfaces that they cannot build properly today. + +### Naming the functions something different + +* Several commenters suggested making the proposed functions static members of `UnsafeMutableBufferPointer` (etc.) instead of free functions. The Standard Library has precedent for producing transient resources via free function, e.g. `withUnsafePointer(to:)` and `withUnsafeThrowingContinuation(_:)`. I am not immediately aware of counter-examples in the Standard Library. +* My initial proposal used longer names for each function, but review found they were too wordy. Several commenters proposed alternative names: `withEphemeral(...)`, `withUnsafeLocalStorage(...)`, and `withUnsafeUninitializedBuffer(...)` were all suggested, among others. + +### Exposing some subset of the three proposed functions + +* One commenter wanted to expose _only_ `withUnsafeTemporaryAllocation(byteCount:alignment:_:)` in order to add friction and reduce the risk that someone would adopt the function without understanding its behaviour. Since most adopters would immediately need to call `bindMemory(to:)` to get a typed buffer, my suspicion is that developers would quickly learn to do so anyway. +* Another commenter did not want to expose `withUnsafeTemporaryAllocation(of:_)` on the premise that it is trivial to get an `UnsafeMutablePointer` out of an `UnsafeMutableBufferPointer` with a `count` of `1`. It is indeed easy to do so, however the two types have different sets of member functions and I'm not sure that the added friction _improves_ adopting code. On the other hand, if anyone needs a _single_ stack-allocated value, they can use `Optional` today to get one. + +### Exposing this functionality as a type rather than as a scoped function + +* It is likely the capacity of such a type would need to be known at compile time. Swift already has a mechanism for declaring types with a fixed-size sequence of values: homogeneous tuples. In fact, Michael Gottesman has [a pitch](https://forums.swift.org/t/pitch-improved-compiler-support-for-large-homogenous-tuples/49023) open at the time of this writing to add syntactic sugar to make homogeneous tuples look more like C arrays. +* As a type, values thereof would need to be initialized before being used. They would impose the same initialization overhead we want to avoid. +* A type, even a value type, suffers from the same stack-promotion limitations as `UnsafeMutableBufferPointer` or `Array`, namely that the optimizer must act conservatively and may still need to heap-allocate. Value types also get copied around quite a bit (although the Swift compiler is quite good at copy elision.) +* One commenter suggested making this hypothetical type _only_ stack-allocatable, but no such type exists today in Swift. It would be completely new to both the Swift language and the Swift compiler. It would not generalize well, because (to my knowledge) there are no other use cases for stack-only types. + +### Telling adopters to use `ManagedBuffer` + +* `ManagedBuffer` has the same general drawbacks as any other type (see above.) +* `ManagedBuffer` is a reference type, not a value type, so the compiler _defaults_ to heap-allocating it. Stack promotion is possible but is not the common case. +* `ManagedBuffer` is not a great interface in and of itself. +* To me, `ManagedBuffer` says "I want to allocate a refcounted object with an arbitrarily long tail-allocated buffer" thus avoiding two heap allocations when one will do. I can then use that object as I would use any other object. This sort of use case doesn't really align with the use cases for the proposed functions. + +### Exposing an additional function to initialize a value by address without copying + +* One commenter suggested: + > A variation of the signature that gives you the initialized value you put in the memory as the return value, `makeValueAtUninitializedPointer(_: (UnsafeMutablePointer) -> Void) -> T`, which could be implemented with return value optimization to do the initialization in-place. + + The proposed functions can be used for this purpose: + + ```swift + let value = withUnsafeTemporaryAllocation(of: T.self) { ptr in + ... + return ptr.move() + } + ``` + + Subject to the optimizer eliminating the `move()`, which in the common case it should be able to do. + +### Eliminating "unsafe" interfaces from Swift entirely + +* Some commenters were concerned by the idea of adding more "unsafe" interfaces to the Standard Library. The proposed functions are "unsafe" by the usual Swift definition, but not moreso than existing functions such as `withUnsafePointer(to:_:)` or `Data.withUnsafeBytes(_:)`. +* As discussed previously, in order to provide a high-level "safe" interface, the language needs some amount of lower-level unsafety. `Data` cannot be implemented without `UnsafeRawPointer` (or equivalent,) nor `Array` without `UnsafeMutableBufferPointer`, nor `CheckedContinuation` without `UnsafeContinuation`. +* The need for unsafe interfaces is not limited to the Standard Library: developers working at every layer of the software stack can benefit from careful use of unsafe interfaces like the proposed functions. +* Creating a dialect of Swift that bans unsafe interfaces entirely is an interesting idea and is worth discussing in more detail, but it is beyond the scope of this proposal. + +### Using a different stack-promotion heuristic + +* The original proposal used only a trivial "allocation ≤ _n_" heuristic for stack allocations. A number of reviewers wanted stronger guarantees around stack promotion. By adding the Standard Library function `_swift_stdlib_isStackAllocationSafe()`, we give the language a hook it can use to implement more complex heuristics. +* Some reviewers wanted the proposed functions to yield an optional buffer and `nil` on stack allocation failure. The Swift standard library currently considers allocation to be a non-failable operation: if an allocation actually fails, the library terminates the program rather than yielding a `nil` pointer. Since the proposed functions are intended to be used in scenarios where callers are currently already using `UnsafeMutableBufferPointer.allocate(capacity:)`, a fallback path that uses that function does not represent a performance regression. +* Some reviewers wanted the proposed functions to _always_ stack-allocate and _never_ heap-allocate. Putting aside inputs that require heap allocation (e.g. `alignment` not known at compile-time, so the correct LLVM `alloca` instructions cannot be generated), such a function would be inherently _dangerous_ rather than simply unsafe. Consider this seemingly-harmless program: + + ```swift + print("How many tacos do you want?") + guard let n = readLine().flatMap(Int.init(_:)) else { + printUsageAndExit() + } + + withUnsafeTemporaryAllocation(of: taco_t, capacity: n) { buffer + taco_init(buffer.baseAddress!, buffer.count, fillings) + } + + ... + ``` + + If the user specifies a very large number of tacos (an understandable choice), this function will smash the stack. While this function is marked "unsafe", Swift still strives to be a safe language, and such a sharp edge on the proposed functions is unacceptable. + +## Acknowledgments + +Thank you to the Swift team for your help and patience as I learn how to write Swift proposals. And thank you to everyone who commented in the pitch thread—it was great to see your feedback and your ideas! diff --git a/proposals/0323-async-main-semantics.md b/proposals/0323-async-main-semantics.md new file mode 100644 index 0000000000..c3349c3424 --- /dev/null +++ b/proposals/0323-async-main-semantics.md @@ -0,0 +1,272 @@ +# Asynchronous Main Semantics + +* Proposal: [SE-0323](0323-async-main-semantics.md) +* Author: [Evan Wilde](https://github.com/etcwilde) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.5.2)** +* Implementation: [apple/swift#38604](https://github.com/apple/swift/pull/38604) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0323-asynchronous-main-semantics/52531) + +## Introduction + +Program setup generally occurs in the main function where developers expect to +perform operations before other parts of the program are run. +Objective-C, C++, and C have initializers that are run before the main +entrypoint runs and can interact with Swift's concurrency systems in ways that +are hard to reason about. +In the Swift concurrency model, the developer-written asynchronous main +function is wrapped in a task and enqueued on the main queue when the main +entrypoint is run. +If an initializer inserts a task on the main queue, that task may be executed +before the main function, so setup is performed after initializer tasks are run. + +Swift-evolution thread: [Pitch: Revisit the semantics of async main](https://forums.swift.org/t/pitch-revisit-the-semantics-of-async-main/51254) + +## Motivation + +Initializers in Objective-C, C++, and C can run code before the main entrypoint +while initializing global variables. If an initializer spawns a task on the main +queue, this initializer task will be enqueued before the task containing the +user-written asynchronous main function. This results in the initializer task +possibly being executed before the main function. +Comparatively, the synchronous main function is run immediately after the +initializers run, but before the tasks created by the initializers. + +Hand-waving around the Swift/C++ interoperability, the example below +demonstrates a C++ library that is incompatible with the current asynchronous +main function semantics because it expects that the `deviceHandle` member of the +`AudioManager` is initialized before the task is run. Instead, the program +asserts because the main function is executed after the task, so the +`deviceHandle` is not initialized by the time the task is run. + +```c++ +struct MyAudioManager { + int deviceHandle = 0; + + MyAudioManager() { + // 2. The constructor for the global variable inserts a task on the main + // queue. + dispatch_async(dispatch_get_main_queue(), ^{ + // 4. The deviceHandle variable is still 0 because the initialization + // hasn't run yet, so this assert fires + assert(deviceHandle != 0 && "Device handle not initialized!"); + }); + } +}; + +// 1. The global variable is dynamically initialized before the main entrypoint +MyAudioManager AudioManager; +``` + +```swift +@main struct Main { + // 3. main entrypoint implicitly wraps this function in a task and enqueues it + static func main() async { + // This line should be used to initialize the deviceHandle before the tasks + // are run, but it's enqueued after the crashing task, so we never get here. + AudioManager.deviceHandle = getAudioDevice(); + } +} +``` + +This behaviour is different from the behaviour of code before Swift concurrency. +Before Swift concurrency, the developer is able to run any setup code necessary +before explicitly starting a runloop to execute tasks that were enqueued on the +main queue. + +## Proposed Solution + +I propose the following changes: + - Run the main function up to the first suspension point synchronously. + - Make the main function implicitly `MainActor` protected. + +The asynchronous main function should run synchronously up to the first +suspension point to allow initialization of state that is required before the +tasks created by initializers are run. +At the suspension point, the current function suspends and other tasks on the +main queue are allowed to run. +This behaviour is consistent with the semantics of `await`, yielding for other +tasks to be executed. + +```swift +@main struct Main { + static func main() async { + // Executed synchronously before tasks created by the initializers run + AudioManager.device = getAudioDevice() + + // At this point, the continuation is enqueued on the main queue. + // Other code on the main queue can be run at this point. + await doSomethingCool() + } +} +``` + +The main entrypoint starts on the main thread. +In order to ensure that there are no suspension points related to thread +hopping, the main function will need to run on the MainActor. +This has the added benefit of making accesses to other MainActor operations +synchronous. +Since the main function must run on the main thread, it cannot be run on other +global actors, so we will need to ban that. + +```swift +@MainActor +var variable : Int = 32 + +@main struct Main { + static func main() async { + // not a suspension point because main is implicitly on the MainActor + print(variable) + } +} +``` + +## Detailed Design + +Asynchronous functions are broken into continuation functions at each suspension +point. +There is an entry function and separate continuation functions for each +suspension. +The example below is a high-level analog of how the asynchronous main function +is broken: + +```swift +@main struct Main { + static func main() async { + print("Hello1") + await foo() + await bar() + } +} +``` + +The asynchronous main function above is broken into three synchronous +continuation functions. +`_main1` is the entrypoint to the main function, while `_main2` is enqueued by +`_main1`, and `_main3` is enqueued by `_main2`. + +```swift +@main struct Main { + static func _main3() { + bar() + } + static func _main2() { + foo() + enqueue(_main3) + } + static func _main1() { + print("Hello1") + enqueue(_main2) + } +} +``` + +The snippet below describes how the main entrypoint starts the program, by +enqueuing the first continuation, `_main1`, before starting a runloop to run +the tasks enqueued on the main queue. + +```swift +// The main entrypoint to the program with old async main semantics +func @main(_ argc: Int32, _ argv: UnsafeMutablePointer>>) { + enqueue(_main1) + drainQueues() +} +``` + +Instead of enqueuing the first continuation, we can execute it directly and let +it enqueue the next continuation. + +```swift +// The main entrypoint to the program with the new async main semantics +func @main(_ argc: Int32, _ argv: UnsafeMutablePointer>>) { + _main1() + drainQueues() +} +``` + +## Source Compatibility + +There are no changes to the source representation of the asynchronous main +function. It will still be written with the same syntax as what is proposed in +[Structured Concurrency](0304-structured-concurrency.md). + +Enforcing that the main function be run on the MainActor will result in new +error messages on code that previously compiled when the main function was +annotated with a non-MainActor global actor. Additionally, there will be new +warning messages emitted when accessing variables or calling functions protected +by the MainActor due to the unnecessary `await` keywords. + +There shouldn't be any change at call-sites, where folks are calling the main +function from another function. The main function is asynchronous, so an await +will already be required. The change will be that this suspension now may +involve a hop to the main actor. + +## Effect on ABI Stability + +These changes can be implemented entirely in the compiler, so we will not need +to change the runtime. I can't think of anywhere else where there may be issues +with ABI and the main function. + +## Effect on API Resilience + +This shouldn't affect the API resilience. + +## Alternatives Considered + +### Separate Synchronous Setup Function + +```swift +@main struct Main { + // Effectively like the synchronous main, run by the main entrypoint of the + // program. + static func setup() { + } + + // Behaves the same way as it does currently + static func main() async { + } +} +``` + +We could allow programmers to implement a secondary `setup` function that is run +after the initializers, but before the concurrency systems are running, allowing +programmers to setup any necessary global state. + +This makes design makes it very clear where setup is to be done and disallows +any implicit asynchronous behaviour from creeping in. A benefit of this is that +you can't accidentally insert an `await` between lines that are initializing +state. + +I don't see anything technically wrong with this approach, but I think that the +model described in the proposal is more consistent with how synchronous code is +written as well as being more aesthetically pleasing. + +### Global Runloop + +Python 3.4 introduced an `asyncio` concurrency library which was driven with an +event loop object. One would need two main functions, one synchronous, and the +other asynchronous. In the synchronous function, you would initialize any +necessary state, grab the event loop with the `asyncio.get_event_loop()` +function, and tell it to run the asynchronous main function. + +Python has since migrated to `asycio.run()` to reduce the boilerplate of +grabbing the event loop and ensuring that it gets closed appropriately, but the +issue of using multiple main function still exists. + +In order to implement this design, we need to provide an analog to the event +loop type, providing a function to run asynchronous code inside of. The problem +with providing this type is that it is available from everywhere, not just the +main function, which would enable programmers to call asynchronous code from a +synchronous function, the model for which hasn't been designed +yet. + +Additionally, this design results in the programmer writing two main functions, +an asynchronous main function to perform asynchronous work and setup work, and +another function that gets the event loop and executes the asynchronous main +function. We can do this work implicitly to reduce the amount of boilerplate +code that a developer needs to write. + +## Acknowledgments + + - Thanks Doug for helping with this proposal and suggesting that we extend the + main function to be MainActor instead of just running on the main thread. diff --git a/proposals/0324-c-lang-pointer-arg-conversion.md b/proposals/0324-c-lang-pointer-arg-conversion.md new file mode 100644 index 0000000000..1c1834e9f8 --- /dev/null +++ b/proposals/0324-c-lang-pointer-arg-conversion.md @@ -0,0 +1,173 @@ +# Relax diagnostics for pointer arguments to C functions + +* Proposal: [SE-0324](0324-c-lang-pointer-arg-conversion.md) +* Authors: [Andrew Trick](https://github.com/atrick), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.6)** +* Implementation: [apple/swift#37956](https://github.com/apple/swift/pull/37956) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0324-relax-diagnostics-for-pointer-arguments-to-c-functions/52599) +* Bugs: [SR-10246](https://bugs.swift.org/browse/SR-10246) + +## Introduction + +C has special rules for pointer aliasing, for example allowing `char *` to alias other pointer types, and allowing pointers to signed and unsigned types to alias. The usability of some C APIs relies on the ability to easily cast pointers within the boundaries of those rules. Swift generally disallows typed pointer conversion. See [SE-0107 UnsafeRawPointer API](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0107-unsaferawpointer.md). Teaching the Swift compiler to allow pointer conversion within the rules of C when invoking functions imported from C headers will dramatically improve interoperability with no negative impact on type safety. + +Swift-evolution thread: [Pitch: Implicit Pointer Conversion for C Interoperability](https://forums.swift.org/t/pitch-implicit-pointer-conversion-for-c-interoperability/51129) + +## Motivation + +Swift exposes untyped, contiguous byte sequences using `UnsafeRawPointer`. This completely bypasses thorny strict aliasing rules when encoding and decoding byte streams. However, Swift programmers often need to call into low-level C functions to help implement the encoding. Those C functions commonly expect a `char *` pointer rather than a `void *` pointer to the contiguous bytes. Swift does not allow raw pointers to be passed as typed pointers because it can easily introduce undefined behavior. + +Calling a C function from Swift that takes a byte sequence as a typed pointer currently requires confusing, ugly, and likely incorrect workarounds. Swift programmers typically reach for `UnsafeRawPointer`'s "memory binding" APIs. Either `bindMemory(to:capacity:)` or `assumingMemoryBound(to:)`. We regularly see reports from programmers who were blocked while attempting a seemingly trivial task and needed to reach out to Swift experts to understand how to call a simple C API. + +Memory binding APIs were never intended for regular Swift programming. Any use of them outside of low-level libraries is a usability bug. Furthermore, observing how the memory binding APIs are commonly used to workaround compiler errors reveals that they are often used incorrectly. And sometimes there is no correct alternative short of copying memory. Swift's model for typed memory was designed to be completely verifiable with a runtime sanitizer. When such a sanitizer is deployed, many of these workarounds will again raise an issue. + +Consider using Foundation's `OutputStream.write` API. The programmer's initial attempt will look like this: + + func write(messageData: Data, output: OutputStream) -> Int { + return messageData.withUnsafeBytes { rawBuffer in + guard let rawPointer = rawBuffer.baseAddress else { return 0 } + return output.write(rawPointer, maxLength: rawBuffer.count) + } + } + +The compiler issues an unhelpful error: + + error: cannot convert value of type 'UnsafeRawPointer' to expected argument type 'UnsafePointer' + +There's no way to make the diagnostic helpful because there's no way to make this conversion generally safe. A determined programmer will eventually figure out how to defeat the compiler's type check by arbitrarily picking either `bindMemory` or `assumingMemoryRebound`, both of which require global understanding of how `messageData`'s memory is used to be correct. Now the code may look like this, or worse: + + func write(messageData: Data, output: OutputStream) -> Int { + return messageData.withUnsafeBytes { rawBuffer in + guard let rawPointer = rawBuffer.baseAddress else { return 0 } + let bufferPointer = rawPointer.assumingMemoryBound(to: UInt8.self) + return output.write(bufferPointer, maxLength: rawBuffer.count) + } + } + +This problem crops up regularly in compression and cryptographic APIs. You can see a couple examples from CommonCrypto in the forums: [CryptoKit: SHA256 much much slower than CryptoSwift](https://forums.swift.org/t/cryptokit-sha256-much-much-slower-than-cryptoswift/27983/12), and [withUnsafeBytes Data API confusion](https://forums.swift.org/t/withunsafebytes-data-api-confusion/22142/10) + +As a generalization of this problem, consider a toy example: + +*encrypt.h* + + #include + + struct DigestWrapper { + unsigned char digest[20]; + }; + + int computeDigest(unsigned char *output, + const unsigned char *input, + size_t length); + +It should be possible to call `computeDigest` from Swift as follows: + +*encrypt.swift* + + func makeDigest(data: Data, wrapper: inout DigestWrapper) -> Int32 { + data.withUnsafeBytes { inBytes in + withUnsafeMutableBytes(of: &wrapper.digest) { outBytes in + computeDigest(outBytes.baseAddress, inBytes.baseAddress, + inBytes.count) + } + } + } + +Without implicit conversion we need to write something like this instead: + + func makeDigest(data: Data, wrapper: inout DigestWrapper) -> Int32 { + data.withUnsafeBytes { inBytes in + withUnsafeMutableBytes(of: &wrapper.digest) { outBytes in + let inPointer = + inBytes.baseAddress?.assumingMemoryBound(to: UInt8.self) + let outPointer = + outBytes.baseAddress?.assumingMemoryBound(to: UInt8.self) + return computeDigest(outPointer, inPointer, inBytes.count) + } + } + } + +In some cases, a typed Swift pointer, rather than a raw pointer must be converted to `char *`. It is always safe to construct a raw pointer from a typed pointer, so the same implicit conversion to C arguments should work for both `UnsafePointer` and `UnsafeRawPointer`. A common use case involves a sequence of characters stored a buffer of any type other than CChar that needs to be passed to a C helper that takes `char *`. The character data may reside in an imported tuple (of any element type) or in a Swift array of UInt8 serving as a byte buffer. + +The implicit conversion issue isn't limited to `char *`. It also comes up when APIs expect signed/unsigned pointer conversion. This has been a problem in practice for Swift programmers calling the mach kernel's task_info API. Wherever the compiler C language's special aliasing rules apply, they should all apply consistently. + +The problematic cases that are documented in bug reports and forum posts are just a very small sampling of the issues that we've been made aware of both from direct communication with programmers and by searching Swift code bases for suspicious uses of "bind memory" APIs. + +## Proposed solution + +For imported C functions, allow implicit pointer conversion between pointer types that are allowed to alias according to C language rules: + +1. A raw or typed unsafe pointer, `Unsafe[Mutable]RawPointer` or + `Unsafe[Mutable]Pointer`, will be convertible to a typed + pointer, `Unsafe[Mutable]Pointer`, whenever `T2` is + `[U]Int8`. This allows conversion to any pointer type declared in C + as `[signed|unsigned] char *`. + +2. A typed unsafe pointer, `Unsafe[Mutable]Pointer`, will be + convertible to `Unsafe[Mutable]Pointer` whenever `T1` and `T2` + are integers that differ only in their signedness. + +The conversions automatically apply to any function imported by the compiler frontend that handles the C family of languages. As a consequence, a Swift programmer's initial attempt to call a C, Objective-C, or C++ function will just work in most cases. See the above Motivation section for examples. + +This solution does not affect type safety because the C compiler must already assume pointers of either type may alias. + +Note that implicit conversion to a `const` pointer type was implemented when unsafe pointers were introduced. The new conversions extend the existing design. In fact, this extension was anticipated when raw pointers were introduced, but the implementation was deferred until developers had experience using raw pointers. + +This solution does not cover C APIs that take function pointers. However, that case is much less common. For function pointer based APIs, its more appropriate to provide a Swift shim around the C API to encapsulate both the workaround for converting the pointer type and the function pointer handling in general. + +## Detailed design + +Implementation of this feature is based on the constraint restriction mechanism also used for other implicit conversions such as pointer/optional conversions. It introduces a new `PointerToCPointer` restriction kind which is only applied in argument positions when call is referencing an C/ObjC imported declaration and argument is either `Unsafe[Mutable]RawPointer` or `Unsafe[Mutable]Pointer` and parameter is a pointer type or an optional (however deep) type wrapping a pointer. + +To support new conversion in interaction with optional types e.g. `UnsafeRawPointer` -> `UnsafePointer?` new restriction won't be recorded until there are other restrictions left to try (e.g. value-to-optional or optional-to-optional conversions), doing so makes sure that optional promotion or unwrap happens before new implicit conversion is considered. + +Note that only conversions between typed signed and unsigned integral +pointers are commutative, conversions from raw pointers are more +restrictive: + +|Actual Swift Argument|Parameter Imported from C|Is Commutative| +---|---|--- +|`UnsafeRawPointer`|`UnsafePointer<[U]Int8>`|No| +|`UnsafeMutableRawPointer`|`Unsafe[Mutable]Pointer<[U]Int8>`|No| +|`UnsafePointer`|`UnsafePointer<[U]Int8>`|No| +|`UnsafeMutablePointer`|`Unsafe[Mutable]Pointer<[U]Int8>`|No| +|`UnsafePointer`|`UnsafePointer`|Yes| +|`UnsafePointer`|`UnsafePointer`|Yes| +|`UnsafePointer`|`UnsafePointer`|Yes| +|`UnsafePointer`|`UnsafePointer`|Yes| +|`UnsafeMutablePointer`|`Unsafe[Mutable]Pointer`|Yes| +|`UnsafeMutablePointer`|`Unsafe[Mutable]Pointer`|Yes| +|`UnsafeMutablePointer`|`Unsafe[Mutable]Pointer`|Yes| +|`UnsafeMutablePointer`|`Unsafe[Mutable]Pointer`|Yes| + + +## Source compatibility + +No effect. + +In general, adding implicit conversions is not source compatible. But this proposal only adds implicit conversions for function argument types that would already cause an override conflict had they both been part of an overridden function declared in C. Since the new implicit conversions are only applied to functions imported from C, this change cannot introduce any new override conflicts. + +## Effect on ABI stability + +Not applicable. Pointer conversion is entirely handled on the caller side. + +## Effect on API resilience + +Not applicable. Pointer conversion is entirely handled on the caller side. + +## Alternatives considered + +*Use C shims to make C APIs more raw-pointer-friendly.* In SwiftNIO, the pointer conversion problem was prevalent enough that it made sense to introduce a replacement C APIs taking `void *` instead of `char *`. For example: https://github.com/apple/swift-nio/blob/nio-1.14/Sources/CNIOHTTPParser/include/CNIOHTTPParser.h#L22-L27 This is not an obvious workaround, and it it impractical for most developers to introduce shims in their project for C APIs. + +*Rely on C APIs to be replaced or wrapped with Swift shims.* The rate at which programmers run into this interoperability problem is speeding up, not slowing down. Swift continues to be adopted in situations that require interoperability. There are a large number of bespoke C APIs that won't be replaced by Swift APIs in the foreseeable future. If the existing C API is wrapped with a Swift shim, then that only hides the incorrect memory binding workaround rather than fixing it. + +*Add more implicit conversions to Swift.* This would introduce C's legacy pointer aliasing rules into the Swift language. Swift's model for type pointer aliasing should remain simple and robust. Special case aliasing rules that happen to work for common cases are deeply misleading. They introduce complexity in the language definition, implementation, and tooling. These special cases are unnecessary and undesirable for well-designed Swift APIs. Implicit type punning introduces more opportunities for bugs. Special aliasing rules would also penalize performance of pure Swift code. Finally, this would not be a source-compatible change. + +*Introduce `UnsafeRawPointer.withMemoryRebound(to:capacity:)`.* This is a generally useful, although somewhat unsafe API. We also plan to introduce this API, but it isn't a sufficient fix for C interoperability. It only provides yet another ugly and confusing workaround alternative. + +## Acknowledgments + +Thank you to all the patient Swift programmers who have struggled with C interoperability and shared their experience with the Swift team. + +Thanks to @eskimo, @lukasa, @jrose, @karl, and @itaiferber for helping those programmers use unsafe pointers while waiting for the language and libraries to be improved. diff --git a/proposals/0325-swiftpm-additional-plugin-apis.md b/proposals/0325-swiftpm-additional-plugin-apis.md new file mode 100644 index 0000000000..626710d855 --- /dev/null +++ b/proposals/0325-swiftpm-additional-plugin-apis.md @@ -0,0 +1,556 @@ +# Additional Package Plugin APIs + +* Proposal: [SE-0325](0325-swiftpm-additional-plugin-apis.md) +* Authors: [Anders Bertelrud](https://github.com/abertelrud) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.6)** +* Implementation: [apple/swift-package-manager#3758](https://github.com/apple/swift-package-manager/pull/3758) +* Pitch: [Forum discussion](https://forums.swift.org/t/pitch-additional-api-available-to-swiftpm-plugins/) +* Review: [Forum discussion](https://forums.swift.org/t/se-0325-additional-package-plugin-apis/) + +## Introduction + +[SE-0303](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md) introduced the ability to define *build tool plugins* in SwiftPM, allowing custom tools to be invoked while building a package. In support of this, SE-0303 introduced a minimal initial API through which plugins can access information about the target for which they are invoked. + +This proposal extends the plugin API to provide more context, including a richer representation of the package graph. This is in preparation for supporting new kinds of plugins in the future. + +## Motivation + +The build tool plugin support introduced in SE-0303 is focused on code generation during a build of a package, for such purposes as generating Swift source files from `.proto` files or other inputs. The initial API provided to plugins was oriented toward that task, and was purposefully kept minimal in order to keep the scope of the proposal bounded. + +New kinds of plugins that are being discussed will require a richer context. In particular, providing a distilled form of the whole package graph would allow for a wide variety of new kinds of plugins. + +## Proposed Solution + +This proposal extends the plugin API that was introduced in SE-0303 by defining a generic `PluginContext` structure that supersedes `TargetBuildContext`. This new structure provides a distilled form of the resolved package graph as seen by SwiftPM, with information about all the products and targets therein. + +This is the same structure that SwiftPM’s built-in subsystems currently use, and the intent is that, over time, at least some of those subsystems can be reimplemented as plugins. This information is also expected to be useful to various kinds of other plugins, provided by external packages. + +In addition to the new information, this proposal adds API for traversing the package graph, such as being able to access topologically sorted lists of target dependencies. This will make it more convenient for build tool plugins that, for example, need to generate command line arguments that include a search paths for each dependency target. + +## Detailed Design + +This proposal defines a new `PluginContext` structure that contains: + +* a reference to the `Package` at the root of the subgraph to which the plugin is being applied +* the contextual information that was previously part of `TargetBuildContext` + +This structure factors out all the information related to the package graph, such as the package and target names and directories, leaving the context with just the top-level contextual information. + +The `BuildToolPlugin` protocol entry point defined by SE-0303 is superseded by a new entry point that takes the new `PluginContext` type and a reference to the `Target` for which build commands should be generate. The previous API remains so that existing plugins continue to work. + +### Plugin API + +The new `PluginContext` structure in the `PackagePlugin` API is defined like this: + +```swift +/// Provides information about the package for which the plugin is invoked, +/// as well as contextual information based on the plugin's stated intent +/// and requirements. +public struct PluginContext { + /// Information about the package to which the plugin is being applied, + /// and any other package reachable from it. + public let package: Package + + /// The path of a writable directory into which the plugin or the build + /// commands it constructs can write anything it wants. This could include + /// any generated source files that should be processed further, and it + /// could include any caches used by the build tool or the plugin itself. + /// The plugin is in complete control of what is written under this di- + /// rectory, and the contents are preserved between builds. + /// + /// A plugin would usually create a separate subdirectory of this directory + /// for each command it creates, and the command would be configured to + /// write its outputs to that directory. The plugin may also create other + /// directories for cache files and other file system content that either + /// it or the command will need. + public let pluginWorkDirectory: Path + + /// Looks up and returns the path of a named command line executable tool. + /// The executable must be provided by an executable target or a binary + /// target on which the package plugin target depends. This function throws + /// an error if the tool cannot be found. The lookup is case sensitive. + public func tool(named name: String) throws -> Tool + + /// Information about a particular tool that is available to a plugin. + public struct Tool { + /// Name of the tool (suitable for display purposes). + public let name: String + + /// Full path of the built or provided tool in the file system. + public let path: Path + } +} +``` + +The `package` property is a reference to the package to which the plugin is being applied. Through it, the script that implements the plugin can reach the entire subgraph of resolved packages on which it either directly or indirectly depends. Note that this might only constitute part of the package graph, if the plugin is being applied to a package other than the root package of the whole graph SwiftPM sees. + +The function and structure definition that relates to looking up tools with a particular name are unchanged from the original SE-0303 proposal. + +This has the effect of factoring out all information related to the package and target, and it puts them into its own directed acyclic graph consisting of the following types: + +```swift +/// Represents a single package in the graph (either the root or a dependency). +public protocol Package { + /// Opaque package identifier, unique among the packages in the graph. + var id: ID { get } + typealias ID = String + + /// The name of the package (for display purposes only). + var displayName: String { get } + + /// The absolute path of the package directory in the local file system, + /// regardless of the original provenance of the package. + var directory: Path { get } + + /// The origin of the package (root, local, repository, registry, etc). + var origin: PackageOrigin { get } + + /// The tools version specified by the resolved version of the package. + /// Behavior is often gated on the tools version, to make sure older + /// packages continue to work as intended. + var toolsVersion: ToolsVersion { get } + + /// Any dependencies on other packages, in the same order as they are + /// specified in the package manifest. + var dependencies: [PackageDependency] { get } + + /// Any regular products defined in this package (except plugin products), + /// in the same order as they are specified in the package manifest. + var products: [Product] { get } + + /// Any regular targets defined in this package (except plugin targets), + /// in the same order as they are specified in the package manifest. + var targets: [Target] { get } +} + +/// Represents the origin of a package in the graph. +public enum PackageOrigin { + /// A root package (unversioned). + case root + + /// A local package, referenced by path (unversioned). + case local(path: String) + + /// A package from a Git repository, with a URL and with a textual + /// description of the resolved version or branch name (for display + /// purposes only), along with the corresponding SCM revision. The + /// revision is the Git commit hash and may be useful for plugins + /// that generates source code that includes version information. + case repository(url: String, displayVersion: String, scmRevision: String) + + /// A package from a registry, with an identity and with a textual + /// description of the resolved version or branch name (for display + /// purposes only). + case registry(identity: String, displayVersion: String) +} + +/// Represents a version of SwiftPM on whose semantics a package relies. +public struct ToolsVersion: CustomStringConvertible, Comparable { + /// The major version. + public let major: Int + + /// The minor version. + public let minor: Int + + /// The patch version. + public let patch: Int +} + +/// Represents a resolved dependency of a package on another package. Other +/// information in addition to the resolved package is likely to be added +/// to this struct in the future. +public struct PackageDependency { + /// A description of the dependency as declared in the package (intended + /// for display purposes only). + public let description: String + + /// The package to which the dependency was resolved. + public let package: Package +} + +/// Represents a single product defined in a package. +public protocol Product { + /// Opaque product identifier, unique among the products in the graph. + var id: ID { get } + typealias ID = String + + /// The name of the product, as defined in the package manifest. This name + /// is unique among the products in the package in which it is defined. + var name: String { get } + + /// The targets that directly comprise the product, in the order in which + /// they are declared in the package manifest. The product will contain the + /// transitive closure of the these targets and their dependencies. + var targets: [Target] { get } +} + +/// Represents an executable product defined in a package. +public struct ExecutableProduct: Product { + /// The target that contains the main entry point of the executable. Every + /// executable product has exactly one main executable target. This target + /// will always be one of the targets that is also included in the product's + /// `targets` list. + public let mainTarget: Target +} + +/// Represents a library product defined in a package. +public struct LibraryProduct: Product { + /// Whether the library is static, dynamic, or automatically determined. + public let kind: Kind + + /// Represents a kind of library product. + public enum Kind { + /// A static library, whose code is copied into its clients. + case `static` + + /// Dynamic library, whose code is referenced by its clients. + case `dynamic` + + /// The kind of library produced is unspecified and will be determined + /// by the build system based on how the library is used. + case automatic + } +} + +/// Represents a single target defined in a package. +public protocol Target { + /// Opaque target identifier, unique among the targets in the graph. + var id: ID { get } + typealias ID = String + + /// The name of the target, as defined in the package manifest. This name + /// is unique among the targets in the package in which it is defined. + var name: String { get } + + /// The absolute path of the target directory in the local file system. + var directory: Path { get } + + /// Any other targets on which this target depends, in the same order as + /// they are specified in the package manifest. Conditional dependencies + /// that do not apply have already been filtered out. + var dependencies: [Dependency] { get } +} + +/// Represents a dependency of a target on a product or on another target. +public enum TargetDependency { + /// A dependency on a target in the same package. + case target(Target) + + /// A dependency on a product in another package. + case product(Product) +} + +/// Represents a target consisting of a source code module, containing either +/// Swift or source files in one of the C-based languages. +public protocol SourceModuleTarget: Target { + /// The name of the module produced by the target (derived from the target + /// name, though future SwiftPM versions may allow this to be customized). + public let moduleName: String + + /// The source files that are associated with this target (any files that + /// have been excluded in the manifest have already been filtered out). + public let sourceFiles: FileList + + /// Any custom linked libraries required by the module, as specified in + /// the package manifest. + public let linkedLibraries: [String] + + /// Any custom linked frameworks required by the module, as specified in + /// the package manifest. + public let linkedFrameworks: [String] +} + +/// Represents a target consisting of a source code module compiled using Swift. +public struct SwiftSourceModuleTarget: SourceModuleTarget { + /// Any custom compilation conditions specified for the Swift target in + /// the package manifest. + public let compilationConditions: [String] +} + +/// Represents a target consisting of a source code module compiled using Clang. +public struct ClangSourceModuleTarget: SourceModuleTarget { + /// Any preprocessor definitions specified for the Clang target. + public let preprocessorDefinitions: [String] + + /// Any custom header search paths specified for the Clang target. + public let headerSearchPaths: [Path] + + /// The directory containing public C headers, if applicable. This will + /// only be set for targets that have a directory of a public headers. + public let publicHeadersDirectory: Path? +} + +/// Represents a target describing an artifact (e.g. a library or executable) +/// that is distributed as a binary. +public struct BinaryArtifactTarget: Target { + /// The kind of binary artifact. + public let kind: Kind + + /// The original source of the binary artifact. + public let origin: Origin + + /// The location of the binary artifact in the local file system. + public let artifact: Path + + /// Represents a kind of binary artifact. + public enum Kind { + /// Represents a .xcframework directory containing frameworks for + /// one or more platforms. + case xcframework + + /// Represents a .artifactsarchive directory containing SwiftPM + /// multiplatform artifacts. + case artifactsArchive + } + + // Represents the original location of a binary artifact. + public enum Origin: Equatable { + /// Represents an artifact that was available locally. + case local + + /// Represents an artifact that was downloaded from a remote URL. + case remote(url: String) + } +} + +/// Represents a target describing a system library that is expected to be +/// present on the host system. +public struct SystemLibraryTarget: Target { + /// The name of the `pkg-config` file, if any, describing the library. + public let pkgConfig: String? + + /// Flags from `pkg-config` to pass to Clang (and to SwiftC via `-Xcc`). + public let compilerFlags: [String] + + /// Flags from `pkg-config` to pass to the platform linker. + public let linkerFlags: [String] +} + +/// Provides information about a list of files. The order is not defined +/// but is guaranteed to be stable. This allows the implementation to be +/// more efficient than a static file list. +public protocol FileList: Sequence { + func makeIterator() -> FileListIterator +} +public struct FileListIterator: IteratorProtocol { + mutating public func next() -> File? +} + +/// Provides information about a single file in a FileList. +public struct File { + /// The path of the file. + public let path: Path + + /// File type, as determined by SwiftPM. + public let type: FileType +} + +/// Provides information about the type of a file. Any future cases will +/// use availability annotations to make sure existing plugins still work +/// until they increase their required tools version. +public enum FileType { + /// A source file. + case source + + /// A header file. + case header + + /// A resource file (either processed or copied). + case resource + + /// A file not covered by any other rule. + case unknown +} +``` + +Note that the `Target` and `Product` types are defined using protocols, with structing implementing the specific types of targets and products. Although they define unique identifiers, they do not in this proposal conform to `Identifiable`, since that would introduce `Self` requirements on the protocol that makes it difficult to have heterogeneous collections of targets and products, as the SwiftPM package model has. This should be alleviated via [SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md). The `ID` type alias and `id` property should make it easy to conform these protocols to `Identifiable` in the future without affecting existing plugins. + +The `BuildToolPlugin` is extended with the following new entry point that takes the new, more general context and a direct reference to the target for which build commands should be created: + +```swift +/// Defines functionality for all plugins having a `buildTool` capability. +public protocol BuildToolPlugin: Plugin { + /// Invoked by SwiftPM to create build commands for a particular target. + /// The context parameter contains information about the package and its + /// dependencies, as well as other environmental inputs. + /// + /// This function should create and return build commands or prebuild + /// commands, configured based on the information in the context. Note + /// that it does not directly run those commands. + func createBuildCommands( + context: PluginContext, + target: Target + ) throws -> [Command] +} +``` + +The previous entry point remains, and a default implementation of the new one calls through to the old one. Its implementation is this, which also provides an example of using the new API and how it maps to the old one: + +```swift +extension BuildToolPlugin { + /// Default implementation that invokes the old callback with an old-style + /// context, for compatibility. + public func createBuildCommands( + context: PluginContext, + target: Target + ) throws -> [Command] { + return try self.createBuildCommands(context: TargetBuildContext( + targetName: target.name, + moduleName: (target as? SourceModuleTarget)?.moduleName ?? target.name, + targetDirectory: target.directory, + packageDirectory: context.package.directory, + inputFiles: (target as? SourceModuleTarget)?.sourceFiles ?? .init([]), + dependencies: target.recursiveTargetDependencies.map { .init( + targetName: $0.name, + moduleName: ($0 as? SourceModuleTarget)?.moduleName ?? $0.name, + targetDirectory: $0.directory, + publicHeadersDirectory: ($0 as? SourceModuleTarget)?.publicHeadersDirectory) + }, + pluginWorkDirectory: context.pluginWorkDirectory, + toolNamesToPaths: context.toolNamesToPaths)) + } +} +``` + +## Additional APIs + +This proposal also adds the first of what is expected to be a toolbox of APIs to cover common things that plugins want to do: + +```swift +extension Target { + /// The transitive closure of all the targets on which the receiver depends, + /// ordered such that every dependency appears before any other target that + /// depends on it (i.e. in "topological sort order"). + public var recursiveTargetDependencies: [Target] +} + +extension SourceModuleTarget { + /// A possibly empty list of source files in the target that have the given + /// filename suffix. + public func sourceFiles(withSuffix: String) -> FileList +} +``` + +Future proposals might add other useful APIs. + +## Example 1: SwiftGen + +```swift +import PackagePlugin + +@main +struct SwiftGenPlugin: BuildToolPlugin { + /// This plugin's implementation returns a single `prebuild` command to run `swiftgen`. + func createBuildCommands(context: PluginContext, target: Target) throws -> [Command] { + // This example configures `swiftgen` to take inputs from a `swiftgen.yml` file + let swiftGenConfigFile = context.package.directory.appending("swiftgen.yml") + + // This example configures the command to write to a "GeneratedSources" directory. + let genSourcesDir = context.pluginWorkDirectory.appending("GeneratedSources") + + // Return a command to run `swiftgen` as a prebuild command. It will be run before + // every build and generates source files into an output directory provided by the + // build context. This example sets some environment variables that `swiftgen.yml` + // bases its output paths on. + return [.prebuildCommand( + displayName: "Running SwiftGen", + executable: try context.tool(named: "swiftgen").path, + arguments: [ + "config", "run", + "--config", "\(swiftGenConfigFile)" + ], + environment: [ + "PROJECT_DIR": "\(context.packageDirectory)", + "TARGET_NAME": "\(target.name)", + "DERIVED_SOURCES_DIR": "\(genSourcesDir)", + ], + outputFilesDirectory: genSourcesDir)] + } +} +``` + +## Example 2: SwiftProtobuf + +```swift +import PackagePlugin +import Foundation + +@main +struct MyPlugin: BuildToolPlugin { + /// This plugin's implementation returns multiple build commands, each of which + /// calls `protoc`. + func createBuildCommands(context: PluginContext, target: Target) throws -> [Command] { + // In this case we generate an invocation of `protoc` for each input file, + // passing it the path of the `protoc-gen-swift` generator tool. + let protocTool = try context.tool(named: "protoc") + let protocGenSwiftTool = try context.tool(named: "protoc-gen-swift") + + // Construct the search paths for the .proto files, which can include any + // of the targets in the dependency closure. Here we assume that the public + // ones are in a `protos` directory, but this can be made arbitrarily complex. + var protoSearchPaths = target.recursiveTargetDependencies.map { + $0.directory.appending("protos") + } + + // This example configures the commands to write to a "GeneratedSources" + // directory. + let genSourcesDir = context.pluginWorkDirectory.appending("GeneratedSources") + + // This example uses a different directory for other files generated by + // the plugin. + let otherFilesDir = context.pluginWorkDirectory.appending("OtherFiles") + + // Add the search path to the system proto files. This sample implementation + // assumes that they are located relative to the `protoc` compiler provided + // by the binary target, but real implementation could be more sophisticated. + protoSearchPaths.append(protocTool.path.removingLastComponent().appending("system-protos")) + + // Create a module mappings file. This is something that the Swift source + // generator `protoc` plug-in we are using requires. The details are not + // important for this proposal, except that it needs to be able to be con- + // structured from the information in the context given to the plugin, and + // to be written out to the intermediates directory. + let moduleMappingsFile = otherFilesDir.appending("module-mappings") + let outputString = ". . . module mappings file . . ." + let outputData = outputString.data(using: .utf8) + FileManager.default.createFile(atPath: moduleMappingsFile.string, contents: outputData) + + // Iterate over the .proto input files, creating a command for each. + let inputFiles = target.sourceFiles(withSuffix: ".proto") + return inputFiles.map { inputFile in + // The name of the output file is based on the name of the input file, + // in a way that's determined by the protoc source generator plug-in + // we're using. + let outputName = inputFile.path.stem + ".swift" + let outputPath = genSourcesDir.appending(outputName) + + // Specifying the input and output paths lets the build system know + // when to invoke the command. + let inputFiles = [inputFile.path] + let outputFiles = [outputPath] + + // Construct the command arguments. + var commandArgs = [ + "--plugin=protoc-gen-swift=\(protocGenSwiftTool.path)", + "--swift_out=\(genSourcesDir)", + "--swift_opt=ProtoPathModuleMappings=\(moduleMappingsFile)" + ] + commandArgs.append(contentsOf: protoSearchPaths.flatMap { ["-I", "\($0)"] }) + commandArgs.append("\(inputFile.path)") + + // Append a command containing the information we generated. + return .buildCommand( + displayName: "Generating \(outputName) from \(inputFile.path.stem)", + executable: protocTool.path, + arguments: commandArgs, + inputFiles: inputFiles, + outputFiles: outputFiles) + } + } +} +``` + +## Security Considerations + +As specified in SE-0303, plugins are invoked in a sandbox that prevents network access and file system write operations other than to a small set of predetermined locations. This proposal only extends the information that is available to plugins so that it contains information that is already defined in a package graph — it doesn’t grant any new abilities to the plugin. diff --git a/proposals/0326-extending-multi-statement-closure-inference.md b/proposals/0326-extending-multi-statement-closure-inference.md new file mode 100644 index 0000000000..0ed3efe8f0 --- /dev/null +++ b/proposals/0326-extending-multi-statement-closure-inference.md @@ -0,0 +1,229 @@ +# Enable multi-statement closure parameter/result type inference + +* Proposal: [SE-0326](0326-extending-multi-statement-closure-inference.md) +* Author: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#38577](https://github.com/apple/swift/pull/38577), [apple/swift#40397](https://github.com/apple/swift/pull/40397), [apple/swift#41730](https://github.com/apple/swift/pull/41730) +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0326-multi-statement-closure-parameter-result-type-inference/53502) + +## Introduction + +I propose to improve inference behavior of multi-statement closures by enabling parameter and result type inference from the closure body. This will make type inference less surprising for developers, and remove the existing behavior cliff where adding one more expression or statement to a closure could result in a compilation failure. + +Swift-evolution thread: [[Pitch] Enable multi-statement closure parameter/result type inference](https://forums.swift.org/t/pitch-enable-multi-statement-closure-parameter-result-type-inference/52619) + + +## Motivation + +Multi-statement closures, unlike their single-statement counterparts, currently cannot propagate information, e.g. parameter and result types from their body, back to the enclosing context, because they are type-checked separately from the expression containing the closure. Information in such closures flows strictly in one direction - from the enclosing context into the body, and statement by statement from the top to the bottom of the closure. + +Currently adding a new expression or statement to a single-statement closure could lead to surprising results. + +Let’s consider the following example: + +```swift +func map(fn: (Int) -> T) -> T { + return fn(42) +} + +func doSomething(_: U) -> Void { /* processing */ } + +let _ = map { + doSomething($0) +} +``` + +Because single-statement closures are type-checked together with enclosing context, it’s possible to infer a type for generic parameter `T` based on the result type of a call to `doSomething`. The behavior is completely different if `doSomething` is not the only call in the body of the `map` closure: + +```swift +let _ = map { + logger.info("About to call 'doSomething(\$0)'") + doSomething($0) +} +``` + +This closure is considered to a multi-statement closure, and currently the type inference behavior is different from the previous example. The body of a multi-statement closure is type-checked *after the* call to `map` has been fully resolved, which means that a type for generic parameter `T` could either a). not be determined at all (because it depends on the body of the closure); or b). be inferred to `Void` which is a type all single-expression closures could default to. + +Neither a). nor b). could be considered an expected outcome in this case from the developer’s perspective because by looking at the code it’s unclear why a). the result type of the closure couldn’t be determined because `doSomething($0)` is a valid call and b). Where did `Void` come from and/or why default type has been applied. Another angle here is compiler diagnostics - because the body of the `map` closure is type-checked separately, it’s impossible to pinpoint the precise source of an error which leads to either mis-diagnosing the issue, e.g. `‘Void’ does not conform to ‘BinaryInteger’`, or a fallback diagnostic asking to specify type of the closure explicitly. + +## Proposed solution + +I propose to allow multi-statement closures to propagate information inferred from the body back to the expression it’s contained in. Such would unify semantics with their single-statement counterparts, remove the distinction from the language, remove artificial restrictions from parameter and result type inference, and help developers to write better and more expressive code. + +## Detailed design + +Multi-statement closure inference would have the following semantics: + +* Parameter and result type inference: + * Contextual type of the closure is the primary source of type information for its parameters and return type. + * If there is no contextual information, the first reference to an anonymous parameter defines its type for the duration of the context. If subsequent statements produce a different type, such situation is considered an ambiguity, and reported as an error; + * Just like anonymous parameters, the first `return` statement defines the result type of the closure (inference is done only if there is no contextual information), if type-checker encounters `return` statements that produce different types such situation is considered an error. + * `inout` inference from the body of a closure and back-propagation to the context is not supported - `inout` must be explicit on the parameter declaration or in the contextual type. This is a status quo behavior without source compatibility implications, and the least surprising for the developers albeit inconsistent with parameter type inference in single-statement closures. Please refer to the Future Directions section for the possibility of unifying this behavior. + * `Void` is a default result type for a closure without any explicit `return` statements. + * Single-expression closures are allowed to default to `Void` if enclosing context requires it e.g. `let _: (Int) → Void = { $0 }` unless `return` is explicit `let _: (Int) -> Void = { return $0 }` results in an error. +* The body of a closure is type-checked just like before, information flows one way, from the first statement to the last, without any backward propagation of information between statements. + * This is important because in this model closure inference is consistent with other types of declarations: functions, subscripts, getters etc., and first type inferred from a declaration becomes its de facto type; + * Type-checking of a closure that contains a single expression remains unchanged in this proposal. + +Let’s go back to our example from `Motivation` section. + +```swift +func map(fn: (Int) -> T) -> T { + return fn(42) +} + +func doSomething(_: U) -> Void { /* processing */ } + +let _ = map { + logger.info("About to call 'doSomething(\$0)'") + doSomething($0) +} +``` + +According to new unified semantics, it is possible to correctly type-check the closure argument to `map` and infer: + +* Anonymous parameter `$0` to be `Int` type based on the expected argument type for the call to `doSomething`; +* Result type of the closure to be `Void` because there are no explicit `return` statements in the body. + +Let’s consider another example which would be supported under new semantic rules: + +```swift +struct Box { + let weight: UInt +} + +func heavier_than(boxies: [Box], min: UInt) -> [UInt] { + let result = boxies.map { + if $0.weight > min { + return $0.weight + } + + return 0 + } + + return result +} +``` + +Currently, because multi-statement closures are type-checked separately from the expression, it’s impossible to determine the result type of the closure. Things are made even worse because diagnostics can’t provide any useful guidance either and would only suggest to specify type of the closure explicitly. + +Under the new semantic rules, `result` would be type-checked to have a type of `[UInt]` because the first return statement `return $0.weight` is inferred as `UInt` and that type information is first propagated to the integer literal `0` in the next return statement (because information flows from the first statement to the last), and back to the expression afterwards. + +This could be extrapolated to a more complex expression, for example: + +```swift +struct Box { + let weight: UInt +} + +func precisely_between(boxies: [Box], min: UInt, max: UInt) -> [UInt] { + let result = boxies.map { + if $0.weight > min { + return $0.weight + } + + return 0 + }**.filter { + $0 < max + }** + + return result +} +``` + +Use of multi-statement closures (or simply closures) becomes less cumbersome by removing the need to constantly specify explicit closure types which sometimes could be pretty large e.g. when there are multiple parameters or a complex tuple result type. + +## Type-Checker Performance Impact + +There were attempts to improve this behavior over the years, the latest one being https://github.com/apple/swift/pull/32223 by Doug Gregor. All of them ran into technical difficulties related to internal limitations of the type-checker, because multi-statement closures, just like function/subscript/setter bodies, tend be composed of a substantial number of statements and cannot be efficiently type-checked the same way as single-expression closures are, which leads to “expression too complex” errors. These issues are resolved by new implementation that takes a more incremental approach. + +## Source compatibility + +All of the aspects of a single-statement closure type-checking are preserved as-is by this proposal, so there is no source compatibility impact for them. + +There is at least one situation where type-checker behavior differs between single- and multi-statement closures because multi-statement do not support parameter type inference from the body. Some of the expressions that used to fail type-check (due to ambiguity) with single-statement closure argument, but type-checked with multi-statement, would now become ambiguous regardless of type of the closure used. + +Let’s consider a call to an overloaded function `test` that expects a closure argument: + +```swift +func test(_: (UnsafePointer) -> R) -> R { ... } +func test(_: (UnsafeRawBufferPointer) -> ResultTy) -> ResultTy { ... } + +let _: Int = test { ptr in + return Int(ptr[0]) << 2 +} +``` + +Currently call to `test` is ambiguous because it’s possible to infer that `PtrTy` is `Int` from the body of the (single-statement closure) closure, so both overloads of `test` produce a valid solution. The situation is different if we were to introduce a new, and possibly completely unrelated, statement to this closure e.g.: + +```swift +func test(_: (UnsafePointer) -> R) -> R { ... } +func test(_: (UnsafeRawBufferPointer) -> ResultTy) -> ResultTy { ... } + +let _: Int = test { ptr in + print(ptr) // <-- shouldn't affect semantics of the body + return Int(ptr[0]) << 2 +} +``` + +This new call to `test` type-checks because the body of this multi-statement closure under current language rules doesn’t participate in the type-check, so there is no way to infer `PtrTy` from the first overload choice. Under the proposed rules type-checker would be able to determine that `PtrTy` is `Int` based on `return` statement just like it did in the previous single-statement closure example, which means that call to `test` becomes ambiguous just like it did before introduction of `print`. + +There are a couple of ways to mitigate situations like this: + +* Add a special ranking rule to type-checker that preserves current behavior by rejecting solutions where parameter type has unresolved (either completely or partially unresolved) parameter and/or result types and argument is a multi-statement closure in favor of an overload choice with “less generic” type e.g. one that only has an unresolved result type. +* Ask users to supply an explicit type for a parameter and/or result type that unambiguously determines the overload e.g. `UnsafePointer` or `UnsafeRawBufferPointer` in our example. +* Don’t do any parameter and/or result type inference from the body of the closure. This is exactly how current multi-statement closures are type-checked under existing rules, which is too restrictive. + +## Future Directions + +### `inout` inference without a contextual type + +There is an inconsistency between single- and multi-statement closures - `inout` inference from the body of a multi-statement closure and its back-propagation is unsupported and requires explicit parameter annotation e.g. `{ (x: inout Int) -> Void in ... }` . + +Currently `inout` is allowed to be inferred: + +* From contextual type for anonymous and name-only parameters e.g. `[1, 2].reduce(into: 0) { $0 += $1 }`. In this case `inout` is passed down to the body from the contextual type - `(inout Result, Self.Element) → Void`, so inference only happens one way. +* For single-statement closures it’s possible to infer `inout` of the external parameter type (visible to the expression closure is associated with) based on its use in the body - assignment, operators, in argument positions with explicit `&`. + +Back-propagation behavior, second bullet, is inconsistent across different kinds of closures (it works only for single-statement closures). This is confusing because there are no visual clues for the developers to reason about the behavior, and easily fixed by providing explicit closure type. + +To make incremental progress, I think it’s reasonable to split `inout` changes from this proposal, because of uncertainty of source compatibility impact (that might be too great for such change to be reasonable for the language) unification of this behavior between single- and multi-statement closures could be a future direction for Swift 6. Doing so would allow to improve closure ergonomics without source compatibility impact, and take advantage of the new implementation to improve result builder and code completion performance and reliability. + +### Type inference across `return` statements in the body of a closure + +It’s common to have situations where an early `guard` statement returns `nil` that doesn’t supply enough type information to be useful for inference under the proposed rules: + +```swift +func test(_: () -> T?) { ... } + +test { + guard let x = doSomething() else { + return nil // there is not enough information to infer `T` from `nil` + } + + ... +} +``` + +Only way to get this closure to type-check is to supply explicit type e.g. `() -> Int? in ...`. To improve this situation type-checker could allow type inference across `return` statements in the body of a closure. That would mean that the actual type of the result would be a join between all of the types encountered in `return` statements, which is going to be semantically unique for the language. + + +## Effect on ABI stability + +No ABI impact since only type-checker handling of closures is going to change but outcomes should not. + + +## Effect on API resilience + +This is not an API-level change and would not impact resilience. + + +## Alternatives considered + +Keep current behavior. + + +## Acknowledgments + +Holly Borla - for helping with content and wording of this proposal. diff --git a/proposals/0327-actor-initializers.md b/proposals/0327-actor-initializers.md new file mode 100644 index 0000000000..f5bfe3feda --- /dev/null +++ b/proposals/0327-actor-initializers.md @@ -0,0 +1,900 @@ +# On Actors and Initialization + +* Proposal: [SE-0327](0327-actor-initializers.md) +* Authors: [Kavon Farvardin](https://github.com/kavon), [John McCall](https://github.com/rjmccall), [Konrad Malawski](https://github.com/ktoso) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.10)** +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0327-on-actors-and-initialization/54587) +* Previous Discussions: + * [On Actor Initializers](https://forums.swift.org/t/on-actor-initializers/49001) + * [Deinit and MainActor](https://forums.swift.org/t/deinit-and-mainactor/50132) + * [First review](https://forums.swift.org/t/se-0327-on-actors-and-initialization/53053) + * [Result of first review](https://forums.swift.org/t/returned-for-revision-se-0327-on-actors-and-initialization/53447) +* Implementation: **Partially implemented in `main`.** + +**Table of Contents** + +- [On Actors and Initialization](#on-actors-and-initialization) + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Overly restrictive non-async initializers](#overly-restrictive-non-async-initializers) + - [Data-races in deinitializers](#data-races-in-deinitializers) + - [Stored Property Isolation](#stored-property-isolation) + - [Initializer Delegation](#initializer-delegation) + - [Proposed functionality](#proposed-functionality) + - [Non-delegating Initializers](#non-delegating-initializers) + - [Flow-sensitive Actor Isolation](#flow-sensitive-actor-isolation) + - [Initializers with `isolated self`](#initializers-with-isolated-self) + - [Initializers with `nonisolated self`](#initializers-with-nonisolated-self) + - [Global-actor isolated types](#global-actor-isolated-types) + - [Delegating Initializers](#delegating-initializers) + - [Syntactic Form](#syntactic-form) + - [Isolation](#isolation) + - [Sendability](#sendability) + - [Deinitializers](#deinitializers) + - [Global-actor isolation and instance members](#global-actor-isolation-and-instance-members) + - [Removing Redundant Isolation](#removing-redundant-isolation) + - [Source compatibility](#source-compatibility) + - [Alternatives considered](#alternatives-considered) + - [Introducing `nonisolation` after `self` is fully-initialized](#introducing-nonisolation-after-self-is-fully-initialized) + - [Permitting `await` for property access in `nonisolated self` initializers](#permitting-await-for-property-access-in-nonisolated-self-initializers) + - [Async Actor Deinitializers](#async-actor-deinitializers) + - [Requiring Sendable arguments only for delegating initializers](#requiring-sendable-arguments-only-for-delegating-initializers) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Acknowledgments](#acknowledgments) + +## Introduction + +Actors are a relatively new nominal type in Swift that provides data-race safety for its mutable state. +The protection is achieved by _isolating_ the mutable state of each actor instance to at most one task at a time. +The proposal that introduced actors ([SE-0306](0306-actors.md)) is quite large and detailed, but misses some of the subtle aspects of creating and destroying an actor's isolated state. +This proposal aims to shore up the definition of an actor, to clarify *when* the isolation of the data begins and ends for an actor instance, along with *what* can be done inside the body of an actor's `init` and `deinit` declarations. + +## Motivation + +While there is no existing specification for how actor initialization and deinitialization *should* work, that in itself is not the only motivation for this proposal. +The *de facto* expected behavior, as induced by the existing implementation in Swift 5.5, is also problematic. In summary, the issues include: + + 1. Non-async initializers are overly strict about what can be done with `self`. + 2. Actor deinitializers can exhibit data races. + 3. Global-actor isolation for default values of stored properties cannot always be respected during initialization. + 4. Initializer delegation requires the use of the `convenience` keyword like classes, even though actors do not support inheritance. + +It's important to keep in mind that these is not an exhaustive list. In particular, global-actor isolated types are effectively actors themselves, so many of the same protections should apply to them, too. + +The following subsections will discuss these high-level problems in more detail. + +### Overly restrictive non-async initializers + +An actor's executor serves as the arbiter for race-free access to the actor's stored properties, analogous to a lock. A task can access an actor's isolated state if it is running on the actor's executor. The process of gaining access to an executor can only be done asynchronously from a task, as blocking a thread to wait for access is against the ethos of Swift Concurrency. This is why invoking a non-async method of an actor instance, from outside of the actor's isolation domain, requires an `await` to mark the possible suspension. The process of gaining access to an actor's executor will be referred to as "hopping" onto the executor throughout this proposal. + +Non-async initializers and all deinitializers of an actor cannot hop to an actor's executor, which would protect its state from concurrent access by other tasks. Without performing a hop, a race between a new task and the code appearing in an `init` can happen: + +```swift +actor Clicker { + var count: Int + func click() { self.count += 1 } + + init(bad: Void) { + self.count = 0 + // no actor hop happens, because non-async init. + + Task { await self.click() } + + self.click() // 💥 this mutation races with the task! + + print(self.count) // 💥 Can print 1 or 2! + } +} +``` + +To prevent the race above in `init(bad:)`, Swift 5.5 imposed a restriction on what can be done with `self` in a non-async initializer. In particular, having `self` escape in a closure capture, or be passed (implicitly) in the method call to `click`, triggers a warning that such uses of `self` will be an error in Swift 6. But, these restrictions are overly broad, because they would also reject initializers that are race-free, such as `init(ok:)` below: + +```swift +actor Clicker { + var count: Int + func click() { self.count += 1 } + nonisolated func announce() { print("performing a click!") } + + init(ok: Void) { + self.count = 0 + Task { await self.click() } + self.announce() // rejected in Swift 5.5, but is race-free. + } +} +``` + +Leveraging the actor isolation model, we know `announce` cannot touch the stored property `count` to observe that `click` happened concurrently with the initializer. That's because `announce` is not isolated to the actor instance. In fact, a race can only happen in the initializer if an access to `count` appears after the creation of the task. This proposal aims to generalize that idea into something we refer to _flow-sensitive isolation_, which uses [data-flow analysis](https://en.wikipedia.org/wiki/Data-flow_analysis) to prove statically that non-async initializers of an actor are race-free. + +### Data-races in deinitializers + +While non-async initializers gained restrictions to prevent data races in Swift 5.5, deinitializers did not. Yet, `deinit`s can still exhibit races and illegal lifetime extensions of `self`, which means that `self` outlives the `deinit`'s invocation. One possible kind of race in a `deinit` is conceptually the same as the one described earlier for non-async initializers: + +```swift +actor Clicker { + var count: Int = 0 + + func click(_ times: Int) { + for _ in 0.. + +### Stored Property Isolation + +The classes, structs, and enums are currently permitted to have global-actor isolation independently applied to each of their stored properties. When programmers specify a default value for a stored property, those default values are computed by each of the non-delegating initializers of the type. The problem is that the expressions for those default values are treated as though they are running on that global-actor's executor. So, it is possible to create impossible constraints on those initializers: + +```swift +@MainActor func getStatus() -> Int { /* ... */ } +@PIDActor func genPID() -> ProcessID { /* ... */ } + +class Process { + @MainActor var status: Int = getStatus() + @PIDActor var pid: ProcessID = genPID() + + init() {} // Problem: what is the isolation of this init? +} +``` + +The example above is accepted in Swift 5.5, but is impossible to implement. Because `status` and `pid` are isolated to two different global-actors, there is no single actor-isolation that can be specified for the non-async `init`. In fact, it's not possible to perform the appropriate actor hops within the non-async initializer. As a result, `getStatus` and `genPID` are being called without hopping to the appropriate executor. + +### Initializer Delegation + +All nominal types in Swift support initializer delegation, which is when one initializer calls another one to perform the rest of the initialization. +For classes, initializer [delegation rules](https://docs.swift.org/swift-book/LanguageGuide/Initialization.html#ID216) are complex due to the presence of inheritance. +So, classes have a required and explicit `convenience` modifier to make a distinction between initializers that delegate. +In contrast, value types do *not* support inheritance, so [the rules](https://docs.swift.org/swift-book/LanguageGuide/Initialization.html#ID215) are much simpler: any `init` can delegate, but if it does, then it must delegate or assign to `self` in all cases: + +```swift +struct S { + var x: Int + init(_ v: Int) { self.x = v } + init(b: Bool) { + if b { + self.init(1) + } else { + self.x = 0 // error: 'self' used before 'self.init' call or assignment to 'self' + } + } +} +``` + +Unlike classes, actors do not support inheritance. But, the proposal for actors did not specify whether `convenience` is required or not in order to have a delegating initializer. Yet, Swift 5.5 requires the use of a `convenience` modifier to mark actor initializers that perform delegation. + + + + + + + + + + + + + + + + + + + + +## Proposed functionality + +The previous sections briefly described some problems with the current state of initialization and deinitialization in Swift. +The remainder of this section aims to fix those problems while defining how actor and global-actor isolated type (GAIT) initializers and deinitializers differ from those belonging to an ordinary class. While doing so, this proposal will highlight how the problems above are resolved. + +### Non-delegating Initializers + +A non-delegating initializer of an actor or a global-actor isolated type (GAIT) is required to initialize all of the stored properties of that type. + +#### Flow-sensitive Actor Isolation + +The focus of this section is on non-delegating initializers for `actor` types, not GAITs. +In Swift 5.5, an actor's initializer that obeys the _escaping-use restriction_ means that the following are rejected throughout the entire initializer: + +- Capturing `self` in a closure. +- Calling a method or computed property on `self`. +- Passing `self` as any kind of argument, whether by-value, `autoclosure`, or `inout`. + +But, those rules are an over-approximation of the restrictions needed to prevent the races described earlier. This proposal removes the escaping-use restriction for initializers. Instead, we propose a simpler set of rules. First we define two categories of initializers, distinguished by their isolation: + +- An initializer has a `nonisolated self` reference if it is: + - non-async + - or global-actor isolated + - or `nonisolated` +- Asynchronous actor initializers have an `isolated self` reference. + +The remainder of this section discusses how these two classes of initializers work. + +##### Initializers with `isolated self` + +For an asynchronous initializer, a hop to the actor's executor will be performed immediately after `self` becomes fully-initialized, in order to ascribe the isolation to `self`. Choosing this location for performing the executor hop preserves the concept of `self` being isolated throughout the entire async initializer. That is, before any escaping uses of `self` can happen in an initializer, the executor hop has been performed. + +It's important to recognize that an executor hop is a suspension point. There are many possible points in an initializer where these suspensions can happen, since there are multiple places where a store to `self` cause it to become initialized. Consider this example of `Bob`: + +```swift +actor Bob { + var x: Int + var y: Int = 2 + func f() {} + init(_ cond: Bool) async { + if cond { + self.x = 1 // initializing store + } + self.x = 2 // initializing store + + f() // this is ok, since we're on the executor here. + } +} +``` + +The problem with trying to explicitly mark the suspension points in `Bob.init` is that they are not easy for programmers to track, nor are they consistent enough to stay the same under simple refactorings. Adding or removing a default value for a stored property, or changing the number of stored properties, can greatly influence where the hops may occur. Consider this slightly modified example from before: + +```swift +actor EvolvedBob { + var x: Int + var y: Int + func f() {} + init(_ cond: Bool) async { + if cond { + self.x = 1 + } + self.x = 2 + self.y = 2 // initializing store + + f() // this is ok, since we're on the executor here. + } +} +``` + +Relative to `Bob`, the only change made to `EvolvedBob` is that its default value for `y` was converted into an unconditional store in the body of the initializer. From an observational point of view, `Bob.init` and `EvolvedBob.init` are identical. But from an implementation perspective, the suspension points for performing an executor hop differ dramatically. If those points required some sort of annotation in Swift, such as with `await`, then the reason why those suspension points moved is hard to explain to programmers. + +In summary, we propose to _implicitly_ perform suspensions to hop to the actors executor once `self` is initialized, instead of having programmers mark those points explicitly, for the following reasons: + +- The finding and continually updating the suspension points is annoying for programmers. +- The reason _why_ some simple stores to a property can trigger a suspension is an implementation detail that is hard to explain to programmers. +- The benefits of marking these suspensions is very low. The reference to `self` is known to be unique by the time the suspension will happen, so it is impossible to create an [actor reentrancy](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md#actor-reentrancy) situation. +- There is [already precedent](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0317-async-let.md#requiring-an-awaiton-any-execution-path-that-waits-for-an-async-let) in the language for performing implicit suspensions, namely for `async let`, when the benefits outweigh the negatives. + +The net effect of these implicit executor-hops is that, for programmers, an `async` initializer does not appear to have any additional rules added to it! That is, programmers can simply view the initializer as being isolated throughout, like any ordinary `async` method would be! The flow-sensitive points where the hop is inserted into the initializer can be safely ignored as an implementation detail for all but the most rare situations. For example: + +```swift +actor OddActor { + var x: Int + init() async { + let name = Thread.current.name + self.x = 0 // initializing store + assert(name == Thread.current.name) // may fail + } +} +``` + +Note that the callers of `OddActor.init` cannot assume that the callee hasn't performed a suspension, just as with any `async` method, because an `await` is required to enter the initializer. Thus, this ability to observe an unmarked suspension is extremely limited. + +**In-depth discussions** + +The remainder of this subsection covers some technical details that are not required to understand this proposal and may be safely skipped. + +**Compiler Implementation Notes:** Identifying the assignment that fully-initializes `self` _does_ require a non-trivial data-flow analysis. Such an analysis is not feasible to do early in the compiler, during type checking. Does acceptance of this proposal mean that the actor-isolation checker, which is run as part of type-checking, will require additional analysis or significant changes? Nope! We can rely on existing restrictions on uses of `self`, prior to initialization, to exclude all places where `self` could be considered only `nonisolated`: + +```swift +func isolatedFunc(_ a: isolated Alice) {} + +actor Alice { + var x: Int + var y: Task + + nonisolated func nonisolatedMethod() {} + func isolatedMethod() {} + + init() async { + self.x = self.nonisolatedMethod() // error: illegal use of `self` before initialization. + self.y = Task { self.isolatedMethod() } // error: illegal capture of `self` before initialization + Task { + self.isolatedMethod() // no await needed, since `self` is isolated. + } + self.isolatedMethod() // OK + isolatedFunc(self) // OK + } +} +``` + +This means that the actor-isolation checker, run prior to converting the program to SIL, can uniformly view the parameter `self` as having type `isolated Self` for the async initializer above. Later in SIL, the defined-before-use verification (i.e., "definite initialization") will find and emit the errors above. As a bonus, that same analysis can be leveraged to find the initializing assignment and introduce the suspension to hop to the actor's executor. + +**Data-race Safety:** In terms of correctness, the proposed `isolated self` initializers are race-free because a hop to the actor's executor happens immediately after the initializing store to `self`, but before the next statement begins executing. Gaining access to the executor at this exact point prevents races, because escaping `self` to another task is only possible _after_ that point. In the `Alice` example above, we can see this in action, where the rejected assignment to `self.y` is due to an illegal capture of `self`. + +**Only one suspension is performed:** It is possible to construct an initializer with control-flow that crosses an implicit suspension points multiple times, as seen in `Bob` above and loops such as: + +```swift +actor LoopyBob { + var x: Int + init(_ counter: Int) async { + var i = 0 + repeat { + self.x = 0 // initializing store + i += 1 + } while i < counter + } +} +``` + +Once gaining access to an executor by crossing the first suspension point, crossing another suspension point does not change the executor, nor will that actually perform a suspension. Avoiding these unnecessary executor hops is an optimization that is done throughout Swift (e.g., self-recursive `async` and `isolated` functions). + + + +##### Initializers with `nonisolated self` + +The category of actor initializers that have a `nonisolated self` contain those which are non-async, or have an isolation that differs from being isolated to `self`. Unlike its methods, an actor's non-async initializer does _not_ require an `await` to be invoked, because there is no actor-instance to synchronize with. In addition, an initializer with a `nonisolated self` can access the instance's stored properties without synchronization, when it is safe to do so. + +Accesses to the stored properties of `self` is required to bootstrap an instance of an actor. Such accesses are considered to be a weaker form of isolation that relies on having exclusive access to the reference `self`. If `self` escapes the initializer, such uniqueness can no-longer be guaranteed without time-consuming analysis. Thus, the isolation of `self` decays (or changes) to `nonisolated` during any use of `self` that is not a direct stored-property access. That change happens once on a given control-flow path and persists through the end of the initializer. Here are some example uses of `self` within an initializer that cause it to decay to a `nonisolated` reference: + +1. Passing `self` as an argument in any procedure call. This includes: + - Invoking a method of `self`. + - Accessing a computed property of `self`, including ones using a property wrapper. + - Triggering an observed property (i.e., one with a `didSet` and/or `willSet`). +2. Capturing `self` in a closure (or autoclosure). +3. Storing `self` to memory. + +Consider the following example that helps demonstrate how this isolation decay works: + +```swift +class NotSendableString { /* ... */ } +class Address: Sendable { /* ... */ } +func greetCharlie(_ charlie: Charlie) {} + +actor Charlie { + var score: Int + let fixedNonSendable: NotSendableString + let fixedSendable: Address + var me: Self? = nil + + func incrementScore() { self.score += 1 } + nonisolated func nonisolatedMethod() {} + + init(_ initialScore: Int) { + self.score = initialScore + self.fixedNonSendable = NotSendableString("Charlie") + self.fixedSendable = NotSendableString("123 Main St.") + + if score > 50 { + nonisolatedMethod() // ✅ a nonisolated use of `self` + greetCharlie(self) // ✅ a nonisolated use of `self` + self.me = self // ✅ a nonisolated use of `self` + } else if score < 50 { + score = 50 + } + + assert(score >= 50) // ❌ error: cannot access mutable isolated storage after `nonisolated` use of `self` + + _ = self.fixedNonSendable // ❌ error: cannot access non-Sendable property after `nonisolated` use of `self` + _ = self.fixedSendable + + Task { await self.incrementScore() } // ✅ a nonisolated use of `self` + } +} +``` + +The central piece of this example is the `if-else` statement chain, which introduces multiple control-flow paths in the initializer. In the body of one of the first conditional block, several different `nonisolated` uses of `self` appear. In the other conditional cases (the `else-if`'s block and the implicitly empty `else`), it is still OK for reads and writes of `score` to appear. But, once control-flow meets-up after the `if-else` statement at the `assert`, `self` is considered `nonisolated` because one of the blocks that can reach that point introduces non-isolation. + +As a consequence, the only stored properties that are accessible after `self` becomes `nonisolated` are let-bound properties whose type is `Sendable`. +The diagnostics emitted for illegal accesses to other stored properties will point to one of the earlier uses of `self` that caused the isolation to change. The sense of "earlier" here is in terms of control-flow and not in terms of where the statements appear in the program. To see how this can happen in practice, consider this alternative definition of `Charlie.init` that uses `defer`: + +```swift +init(hasADefer: Void) { + self.score = 0 + defer { + print(self.score) // ❌ error: cannot access mutable isolated storage after `nonisolated` use of `self` + } + Task { await self.incrementScore() } // note: a nonisolated use of `self` +} +``` + +Here, we defer the printing of `self.score` until the end of the initializer. But, because `self` is captured in a closure before the `defer` is executed, that read of `self.score` is not always safe from data-races, so it is flagged as an error. Another scenario where an illegal property access can visually precede the decaying use is for loops: + +```swift +init(hasALoop: Void) { + self.score = 0 + for i in 0..<10 { + self.score += i // error: cannot access mutable isolated storage after `nonisolated` use of `self` + greetCharlie(self) // note: a nonisolated use of `self` + } +} +``` + +In this for-loop example, we must still flag the mutation of `self.score` in a loop as an error, because it is only safe on the first loop iteration. On subsequent loop iterations, it will not be safe because `self` may be concurrently accessed after being escaped in a procedure call. + +**Other Examples** + +Other than non-async inits, a global-actor isolated initializer or one that is marked with `nonisolated` will have a `nonisolated self`. Consider this example of such an initializer: + +```swift +func printStatus(_ s: Status) { /* ... */} + +actor Status { + var valid: Bool + + // an isolated method + func exchange(with new: Bool) { + let old = valid + valid = new + return old + } + + // an isolated method + func isValid() { return self.valid } + + // A `nonisolated self` initializer that calls isolated methods with `await`. + @MainActor init(_ val: Bool) async { + self.valid = val + + let old = await self.exchange(with: false) // note: a non-isolated use + assert(old == val) + + _ = self.valid // ❌ error: cannot access mutable isolated storage after non-isolated use of `self` + + let isValid = await self.isValid() // ✅ OK + + assert(isValid == false) + } +} +``` + +Notice that calling an isolated method from an initializer with a `nonisolated self` is permitted, provided that you can `await` the call. That call is considered a nonisolated use, i.e., it's the first use of `self` other than to access a stored property. Afterwards, access to most stored properties within the `init` is lost, just like for the non-async case. Because this initializer is `async`, it could technically `await` to read the `Sendable` value of `self.valid`. But, we have chosen to forbid awaited access to stored properties in this situation. See the [discussion](#permitting-await-for-property-access-in-nonisolated-self-initializers) in the Alternatives Considerred section for more details. + + +**In-depth discussions** + +The remainder of this subsection covers some technical details that are not required to understand this proposal and may be safely skipped. + +**Limitations of Static Analysis** +Not all loops iterate more than once, or even at all. The Swift compiler will be free to reject programs that may never exhibit a race dynamically, based on the static assumption that loops can iterate more than once and conditional blocks can be executed. To make this more concrete, consider these two silly loops: + +```swift +init(hasASillyLoop1: Void) { + self.score = 0 + while false { + self.score += i // error: cannot access isolated storage after `nonisolated` use of `self` + greetCharlie(self) // note: a nonisolated use of `self` + } +} + +init(hasASillyLoop2: Void) { + self.score = 0 + repeat { + self.score += i // error: cannot access isolated storage after `nonisolated` use of `self` + greetCharlie(self) // note: a nonisolated use of `self` + } while false +} +``` + +In both loops above, it is clear to the programmer that no race will happen, because control-flow will not dynamically reach the statement incrementing `score` _after_ passing `self` in a procedure call. For these trivial examples, the compiler _may_ be able to prove that these loops do not execute more than once, but that is not guaranteed due to the [limitations of static analysis](https://en.wikipedia.org/wiki/Halting_problem). + +**Data-race Safety** + +In effect, the concept of isolation decay prevents data-races by disallowing access to stored properties once the compiler can no-longer prove that the reference to `self` will not be concurrently accessed. For efficiency reasons, the compiler might not perform interprocedural analysis to prove that passing `self` to another function is safe from concurrent access by another task. Interprocedural analysis is inherently limited due to the nature of modules in Swift (i.e., separate compilation). Immediately after `self` has escaped the initializer, the treatment of `self` in the initializer changes to match the unacquired status of the actor's executor. + +#### Global-actor isolated types + +A non-isolated initializer of a global-actor isolated type (GAIT) is in the same situation as a non-async actor initializer, in that it must bootstrap the instance without the executor's protection. Thus, we can construct a data-race just like before: + +```swift +@MainActor +class RequiresFlowIsolation + where T: Sendable, T: Equatable { + + var item: T + + func mutateItem() { /* ... */ } + + nonisolated init(with t: T) { + self.item = t + Task { await self.mutateItem() } + self.item = t // 💥 races with the task! + } +} +``` + +To solve this race, we propose to apply flow-sensitive actor isolation to the initializers of GAITs that are marked as non-isolated. + +For isolated initializers, GAITs have the ability to gain actor-isolation prior to calling the initializer itself. That's because its executor is a static instance, existing prior to even allocating uninitialized memory for a GAIT instance. Thus, all isolated initializers of a GAIT require callers to `await`, which will gain access to the right executor before starting initialization. That executor is held until the initializer returns. Thus for isolated initializers of GAITs, there is no danger of race among the isolated stored properties: + +```swift +@MainActor +class ProtectedByExecutor { + var item: T + + func mutateItem() { /* ... */ } + + init(with t: T) { + self.item = t + Task { self.mutateItem() } // ✅ we're on the executor when creating this task. + assert(self.item == t) // ✅ always true, since we hold the executor here. + } +} +``` + +GAITs that have `nonisolated` stored properties rely on Swift's existing `Sendable` restrictions to help prevent data races. + + +### Delegating Initializers + +This section defines the syntactic form and rules about delegating initializers for `actor` types and global-actor isolated types (GAITs). + +#### Syntactic Form + +While `actor`s are a reference type, their delegating initializers will follow the same basic rules that exist for value types, namely: + +1. If an initializer body contains a call to some `self.init`, then it's a delegating initializer. No `convenience` keyword is required. +2. For delegating initializers, `self.init` must always be called on all paths, before `self` can be used. + +The reason for this difference between `actor` and `class` types is that `actor`s do not support inheritance, so they can shed the complexity of `class` initializer delegation. GAITs use the same syntactic form as ordinary classes to define delegating initializers. + +#### Isolation + +Much like their non-delegating counterparts, an actor's delegating initializer either has an `isolated self` or a `nonisolated self` reference. The decision procedure for categorizing these initializers are exactly the same: non-async delegating initializers have a `nonisolated self`, *etc*. + +But, the delegating initializers of an actor have simpler rules about what can appear in their body, because they are not required to initialize the instance's stored properties. Thus, instead of using flow-sensitive actor isolation, delegating initializers have a uniform isolation for `self`, much like an ordinary function. + +### Sendability + +When passing values to any of an `actor`'s initializers, from outside of that actor, those values must be `Sendable`. +Thus, during the initialization of a new instance, the actor's "boundary" in terms of Sendability begins at the initial call-site to one of its initializers. +This rule forces programmers to correctly deal with `Sendable` values when creating a new actor instance. Fundamentally, programmers will have only two options for initializing a non-`Sendable` stored property of an actor: + +```swift +class NotSendableType { /* ... */ } +struct Piece: Sendable { /* ... */ } + +actor Greg { + var ns: NotSendableType + + // Option 1: an initializer that can be called from anywhere, + // because its arguments are Sendable. + init(fromPieces ps: (Piece, Piece)) { + self.ns = NotSendableType(ps) + } + + // Option 2: an initializer that can only be delegated to, + // because its arguments are not Sendable. + init(with ns: NotSendableType) { + self.init(fromPieces: ns.getPieces()) + } +} +``` + +As shown in the example above, you _can_ construct an actor that has a non-`Sendable` stored property. But, you should create a new instance of that type from `Sendable` pieces of data in order to store it in the actor instance. Once inside an actor's initializer, non-Sendable values can be freely passed when delegating to another initializer, or calling its methods, *etc*. The following example illustrates this rule: + +```swift +class NotSendableType { /* ... */ } +struct Piece: Sendable { /* ... */ } + +actor Gene { + var ns: NotSendableType + + init(_ ns: NotSendableType) { + self.ns = ns + } + + init(with ns: NotSendableType) async { + self.init(ns) // ✅ non-Sendable is OK during initializer delegation... + someMethod(ns) // ✅ and when calling a method from an initializer, etc. + } + + init(fromPieces ps: (Piece, Piece)) async { + let ns = NotSendableType(ps) + await self.init(with: ns) // ✅ non-Sendable is OK during initializer delegation + } + + func someMethod(_: NotSendableType) { /* ... */ } +} + +func someFunc() async { + let ns = NotSendableType() + + _ = Gene(ns) // ❌ error: cannot pass non-Sendable value across actor boundary + _ = await Gene(with: ns) // ❌ error: cannot pass non-Sendable value across actor boundary + _ = await Gene(fromPieces: ns.getPieces()) // ✅ OK because (Piece, Piece) is Sendable + + _ = await SomeGAIT(isolated: ns) // ❌ error: cannot pass non-Sendable value across actor boundary + _ = await SomeGAIT(secondNonIso: ns) // ❌ error: cannot pass non-Sendable value across actor boundary +} +``` + +For a global-actor isolated type (GAIT), the same rule applies to its `nonisolated` initializers. Thus, upon entering such an initializer from outside of the actor, the values must be `Sendable`. The differences from an `actor` are that: + +1. The caller of the first initializer may already be isolated to the global-actor, so there is no `Sendable` barrier (as usual). +2. When delegating from a `nonisolated` initializer to one that is isolated to the global actor, the value must be `Sendable`. + +The second difference only manifests when a `nonisolated` and `async` initializer delegates to an isolated initializer of the GAIT: + +```swift +@MainActor +class SomeGAIT { + var ns: NotSendableType + + init(isolated ns: NotSendableType) { + self.ns = ns + } + + nonisolated init(firstNonIso ns: NotSendableType) async { + await self.init(isolated: ns) // ❌ error: cannot pass non-Sendable value across actor boundary + } + + nonisolated init(secondNonIso ns: NotSendableType) async { + await self.init(firstNonIso: ns) // ✅ + } +} +``` + +The barrier in the example above can be resolved by removing the `nonisolated` attribute, so that the initializer has a matching isolation. + +### Deinitializers + +In Swift 5.5, two different kinds of data races with an actor or global-actor isolated type (GAIT) can be created within a `deinit`, as shown in an earlier section. The first one involves a reference to `self` being shared with another task, and the second one with actors having shared executors. + +To solve the first kind of race, we propose having the same flow-sensitive actor isolation rules discussed earlier for a `nonisolated self` apply to an actor's `deinit`. A `deinit` falls under the `nonisolated self` category, because it is effectively a non-async, non-delegating initializer whose purpose is to clean-up or tear-down, instead of bootstrap. In particular, a `deinit` starts with a unique reference to `self`, so the rules for decaying to a `nonisolated self` match up perfectly. This solution will apply to the `deinit` of both actor types and GAITs. + +To solve the second race, we propose that a `deinit` can only access the stored properties of `self` that are `Sendable`. This means that, even when `self` is a unique reference and has not decayed to being `nonisolated`, only the `Sendable` stored properties of an actor or GAIT can be accessed. This restriction is not needed for an `init`, because the initializer has known call-sites that are checked for isolation and `Sendable` arguments. The lack of knowledge about when and where a `deinit` will be invoked is why `deinit`s must carry this extra burden. In effect, non-`Sendable` actor-isolated state can only be deinitialized by an actor by invoking that state's `deinit`. + +Here is an example to help illustrate the new rules for `deinit`: + +```swift +actor A { + let immutableSendable = SendableType() + var mutableSendable = SendableType() + let nonSendable = NonSendableType() + + init() { + _ = self.immutableSendable // ✅ ok + _ = self.mutableSendable // ✅ ok + _ = self.nonSendable // ✅ ok + + f(self) // trigger a decay to `nonisolated self` + + _ = self.immutableSendable // ✅ ok + _ = self.mutableSendable // ❌ error: must be immutable + _ = self.nonSendable // ❌ error: must be sendable + } + + + deinit { + _ = self.immutableSendable // ✅ ok + _ = self.mutableSendable // ✅ ok + _ = self.nonSendable // ❌ error: must be sendable + + f(self) // trigger a decay to `nonisolated self` + + _ = self.immutableSendable // ✅ ok + _ = self.mutableSendable // ❌ error: must be immutable + _ = self.nonSendable // ❌ error: must be sendable + } +} +``` + +In the above, the only difference between the `init` and the `deinit` is that the `deinit` can only access `Sendable` properties, whereas the `init` can access non-`Sendable` properties prior to the isolation decay. + + +### Global-actor isolation and instance members + +**Note:** The isolation rules in this section for stored property initial values was never implemented because it was too onerous in existing code patterns that make use of `@MainActor`-isolated types. These rules have been subsumed by [SE-0411: Isolated default values](/proposals/0411-isolated-default-values.md). + +The main problem with global-actor isolation on the stored properties of a type is that, if the property is isolated to a global actor, then its default-value expression is also isolated to that actor. Since global-actor isolation can be applied independently to each stored property, an impossible isolation requirement can be constructed. The isolation needed for a type's non-delegating *and* non-async initializers would be the union of all isolation applied to its stored properties that have a default value. That's because a non-async initializer cannot hop to any executor, and a function cannot be isolated to two global actors. Currently, Swift 5.5 accepts programs with these impossible requirements. + +To fix this problem, we propose to remove any isolation applied to the default-value expressions of stored properties that are a member of a nominal type. Instead, those expressions will be treated by the type system as being `nonisolated`. If isolation is required to initialize those properties, then an `init` can always be defined and given the appropriate isolation. + +For global or static stored properties, the isolation of the default-value expression will continue to match the isolation applied to the property. This isolation is needed to support declarations such as: + +```swift +@MainActor +var x = 20 + +@MainActor +var y = x + 2 +``` + + +#### Removing Redundant Isolation + +Global-actor isolation on a stored property provides safe concurrent access to the storage occupied by that stored property in the type's instances. +For example, if `pid` is an actor-isolated stored property (i.e., one without an observer or property wrapper), then the access `p.pid.reset()` only protects the memory read of `pid` from `p`, and not the call to `reset` afterwards. Thus, for value types (enums and structs), global-actor isolation on those stored properties fundamentally serves no use: mutations of the storage occupied by the stored property in a value type are concurrency-safe by default, because mutable variables cannot be shared between tasks. For example, it is error when trying to capture a mutable var in a Sendable closure: + +```swift +@MainActor +struct StatTracker { + var count = 0 + + mutating func update() { + count += 1 + } +} + +var st = StatTracker() +Task { await st.update() } // error: mutation of captured var 'st' in concurrently-executing code +``` + +As a result, there is no way to concurrently mutate the memory of a struct, regardless of whether the stored properties of the struct are isolated to a global actor. Whether the instance can be shared only depends on whether it's var-bound or not, and the only kind of sharing permitted is via copying. Any mutations of reference types stored _within_ the struct require the usual actor-isolation applied to that reference type itself. In other words, applying global-actor isolation to a stored property containing a class type does _not_ protect the members of that class instance from concurrent access. +So, we propose to remove the requirement that access to those properties are protected by isolation. That is, accessing those stored properties do not require an `await`. + +The [global actors](0316-global-actors.md) proposal explicitly excludes actor types from having stored properties that are global-actor isolated. But in Swift 5.5, that is not enforced by the compiler. We feel that the rule should be enforced, i.e., the storage of an actor should uniformly be isolated to the actor instance. One benefit of this rule is that it reduces the possibility of [false sharing](https://en.wikipedia.org/wiki/False_sharing) among threads. Specifically, only one thread will have write access the memory occupied by an actor instance at any given time. + + +## Source compatibility +There are some changes in this proposal that are backwards compatible or easy to migrate: + +- The set of `init` declarations accepted by the compiler in Swift 5.5 (without emitted warnings) is a strict subset of the ones that will be permitted if this proposal is accepted, i.e., flow-sensitive isolation broadens the set of permitted programs. +- Appearances of `convenience` on an actor's initializer can be ignored and/or have a fix-it emitted. +- Appearances of superfluous global-actor isolation annotations on ordinary stored properties (say, in value types) can be ignored and/or have a fix-it emitted. + +But, there are others which will cause a non-trivial source break to patch holes in the concurrency model of Swift 5.5, for example: + +- The set of `deinit`s accepted by the compiler for actors and GAITs will be narrowed. +- GAITs will have data-race protections applied to their non-isolated `init`s, which slightly narrows the set of acceptable `init` declarations. +- Global-actor isolation on stored-property members of an actor type are prohibited. +- Stored-property members that are still permitted to have actor isolation applied to them will have a `nonisolated` default-value expression. + +Note that these changes to GAITs will only apply to classes defined in Swift. Classes imported from Objective-C with MainActor-isolation applied will be assumed to not have data races. + + +## Alternatives considered + +This section explains alternate approaches that were ultimately not chosen for this proposal. + +### Introducing `nonisolation` after `self` is fully-initialized + +It is tempting to say that, to avoid introducing another concept into the language, `nonisolation` should begin at the point where `self` becomes fully-initialized. But, because control-flow can cross from a scope where `self` is fully-initialized, to another scope where `self` _might_ be fully-initialized, this rule is not enough to determine whether an initializer has a race. Here are two examples of initializers where this simplistic rule breaks down: + +```swift +actor CounterExampleActor { + var x: Int + + func mutate() { self.x += 1 } + + nonisolated func f() { + Task { await self.mutate() } + } + + init(ex1 cond: Bool) { + if cond { + self.x = 0 + f() + } + self.x = 1 // if cond is true, this might race! + } + + init(ex2 max: Int) { + var i = 0 + repeat { + self.x = i // after first loop iteration, this might race! + f() + i += 1 + } while i < max + } +} +``` + +In Swift, `self` can be freely used, _immediately_ after becoming fully-initialized. Thus, if we tie `nonisolation` to whether `self` is fully-initialized _at each use_, both initializers above should be accepted, even though they permit data races: `f` can escape `self` into a task that mutates the actor, yet the initializer will continue after returning from `f` with unsynchronized access to its stored properties. + +With the flow-sensitive isolation rules in this proposal, both property accesses above that can race are rejected because of a flow-isolation error. The source of `nonisolation` would be identified as the calls to `f()`, so that programmers can correct their code. + +Now, consider what would happen if the calls to `f` above were removed. With the proposed isolation rules, the programs would now be accepted because they are safe: there is no source of `nonisolation`. If we had said that `nonisolation` _always_ starts immediately after `self` is fully-initialized, and _persists until the end of the initializer_, then even without the calls to `f`, the initializers above would be would be needlessly rejected. + + +### Permitting `await` for property access in `nonisolated self` initializers + +In an `nonisolated self` initializer, we reject stored property accesses after the first non-isolated use. For a non-async initializer, there is no alternative to rejecting the program, since one cannot hop to the actor's executor in that context. But an `async` initializer that is not isolated to `self` _could_ perform that hop: + +```swift +actor AwkwardActor { + var x: SomeClass + nonisolated func f() { /* ... */ } + + nonisolated init() async { + self.x = SomeClass() + let a = self.x + f() + let b = await self.x // SomeClass would need to be Sendable for this access. + print(a + b) + } +} +``` + +From an implementation perspective, it _is_ feasible to support the program above, where property accesses can become `async` expressions based on flow-sensitive isolation. But, this proposal takes the subjective position that such code should be rejected. + +The expressiveness gained by supporting such a flow-sensitive `async` property access is not worth the confusion they might create. For programmers who simply _read_ this valid code in a project, the `await` might look unnecessary and challenge their understanding of isolation applying to entire functions. But, this specific kind of `nonisolated self` _and_ `async` initializer would be the only place where one could demonstrate to _readers_ that isolation can change mid-function in valid Swift code. + +The ability to observe an isolation-change mid-function in _valid_ Swift code is the reason for rejecting the program above. This proposal says that, for a non-async and `nonisolated self` initializer, some property accesses are _rejected_ for violations of the same conceptual isolation-change. The valid formulation of those kinds of initializers have no observable isolation change, so casual readers notice nothing unusual. Only when modifying that code does the isolation-decay concept become relevant. But, isolation "decay" is just a tool used to explain the concept in this porposal. Programmers only need to keep in mind that accesses to stored properties are lost after you escape `self` in the initializer. + + +### Async Actor Deinitializers + +One idea for working around the inability to synchronize from a `deinit` with the actor or GAIT's executor prior to destruction is to wrap the body of the `deinit` in a task. This would effectively allow the non-async `deinit` to act as though it were `async` in its body. There is no other way to define an asynchronous `deinit`, since the callers of a deinit are never guaranteed to be in an asynchronous context. + +The primary danger here is that it is currently undefined behavior in Swift for a reference to `self` to escape a `deinit` and persist after the `deinit` has completed, which must be possible if the `deinit` were asynchronous. The only other option would be to have `deinit` be blocking, but Swift concurrency is designed to avoid blocking. + +### Requiring Sendable arguments only for delegating initializers + +Delegating initializers are conceptually a good place to construct a fresh +instance of a non-Sendable value to pass-along to the actor during initialization. +This could only work by saying that only `nonisolated self` delegating initializers +can accept a non-Sendable value from any context. But also, an initializer's +delegation status must now be published in the interface of a type, i.e., +some annotation like `convenience` is required. Eliminating the need for +`convenience` was chosen over non-Sendable values for a specific kind of delegating +initializer for a few reasons: + +1. The rules for Sendable values and initializers would become complex, being dependent on three factors: delegation status, isolation of `self`, and the caller's context. The usual Sendable rules only depend on two. +2. Requiring `convenience` for just one narrow use-case is not worth it. +3. Static functions, using a factory pattern, can replace the need for initializers with non-Sendable arguments callable from anywhere. + +## Effect on ABI stability + +This proposal does not affect ABI stability. + +## Effect on API resilience + +This proposal does not affect API resilience. + +## Acknowledgments + +Thank you to the members of the Swift Forums for their time spent reading this proposal and its prior versions and providing comments. diff --git a/proposals/0328-structural-opaque-result-types.md b/proposals/0328-structural-opaque-result-types.md new file mode 100644 index 0000000000..ab57b9a362 --- /dev/null +++ b/proposals/0328-structural-opaque-result-types.md @@ -0,0 +1,162 @@ +# Structural opaque result types + +* Proposal: [SE-0328](0328-structural-opaque-result-types.md) +* Authors: [Benjamin Driscoll](https://github.com/willtunnels), [Holly Borla](https://github.com/hborla) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0328-structural-opaque-result-type/53789) +* Implementation: [apple/swift#38392](https://github.com/apple/swift/pull/38392) +* Toolchain: Any recent [nightly main snapshot](https://swift.org/download/#snapshots) + +## Introduction + +An [opaque result type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md) may be used as the result type of a function, the type of a variable, or the result type of a subscript. In all cases, the opaque result type must be the entire type. This proposal recommends lifting that restriction and allowing opaque result types in "structural" positions. + +Swift-evolution thread: [Structural opaque result types](https://forums.swift.org/t/structural-opaque-result-types/50998) + +## Motivation + +The current restriction on opaque result types prevents them from being used in many common API patterns. Some examples are as follows: + +```swift +// we cannot express a function that might fail to produce an opaque result type +func f0() -> (some P)? { /* ... */ } + +// we cannot use an opaque result type as one of several return values +func f1() -> (some P, some Q) { /* ... */ } + +// we cannot return a lazily computed opaque result type +func f2() -> () -> some P { /* ... */ } + +// more generally, we cannot embed an opaque result type into a larger structure +func f3() -> S { /* ... */ } +``` + +These restrictions are artificial, and lifting them enables more APIs to be expressed using opaque result types. + +## Proposed solution + +We should allow opaque result types in structural positions in the result type of a function, the type of a variable, or the result type of a subscript. + +## Detailed design + +### Syntax for Optionals + +The `some` keyword binds more loosely than `?` or `!`. An optional of an opaque result type must be written `(some P)?`, and an optional of an unwrapped opaque result type must be written `(some P)!`. + +`some P?` gets interpreted as `some Optional

` and therefore produces an error because an opaque type must be constrained to `Any`, `AnyObject`, a protocol composition, and/or a base class. The analogous thing is true of `some P!`. + +### Higher order functions + +If the result type of a function, the type of a variable, or the result type of a subscript is a function type, that function type can only contain structural opaque types in return position. For example, `func f() -> () -> some P` is valid, and `func g() -> (some P) -> ()` produces an error: + +```swift +protocol P {} + +func g() -> (some P) -> () { ... } // error: 'some' cannot appear in parameter position in result type '(some P) -> ()' +``` + +### Constraint inference + +When a generic parameter type is used in a structural position in the signature of a function, the compiler implicitly constrains the generic parameter based on the context is which it is used. E.g., + +```swift +struct H { init(_ t: T) {} } +struct S{ init(_ t: T) {} } + +// same as 'f' because 'H' implies 'T: Hashable' +func f(_ t: T) -> H { + var h = Hasher() + h.combine(t) // OK - we know 'T: Hashable' + let _ = h.finalize() + return H(0) +} + +// 'S' doesn't imply anything about 'T' +func g(_ t: T) -> S { + var h = Hasher() + h.combine(t) // ERROR - instance method 'combine' requires that 'T' conform to 'Hashable' + let _ = h.finalize() + return S(0) +} +``` + +Opaque result types do not feature such inference. E.g., + +```swift +// ERROR - type 'some P' does not conform to protocol 'Hashable' +func f(_ t: T) -> H { /* ... */ } +``` + +## Source compatibility + +This change is purely additive so has no source compatibility consequences. + +As discussed in [SE-0244](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#source-compatibility): + +> If opaque result types are retroactively adopted in a library, it would initially break source compatibility [...] but could provide longer-term benefits for both source and ABI stability because fewer details would be exposed to clients. There are some mitigations for source compatibility, e.g., a longer deprecation cycle for the types or overloading the old signature (that returns the named types) with the new signature (that returns an opaque result type). + +## Effect on ABI stability + +This change is purely additive so has no ABI stability consequences. + +## Effect on API resilience + +This change is purely additive so has no API resilience consequences. Adopting opaque types in structural positions in a resilient library has the same implications as top-level opaque result types. From [SE-0244](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#effect-on-api-resilience): + +> Opaque result types are part of the result type of a function/type of a variable/element type of a subscript. The requirements that describe the opaque result type cannot change without breaking the API/ABI. However, the underlying concrete type can change from one version to the next without breaking ABI, because that type is not known to clients of the API. + +## Rust's `impl Trait` + +As discussed in [SE-0244](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#rusts-impl-trait), Swift's opaque result types were inspired by `impl Trait` in Rust, which is described in [RFC-1522](https://github.com/rust-lang/rfcs/blob/master/text/1522-conservative-impl-trait.md) and extended in [RFC-1951](https://github.com/rust-lang/rfcs/blob/master/text/1951-expand-impl-trait.md). + +Though SE-0244 lists several differences between `some` and `impl Trait`, one difference it does not make explicit is that `impl Trait` is allowed in structural positions, in similar to the manner to that suggested by this proposal. One difference between this proposal and `impl Trait` is that `impl Trait` may not appear in the return type of closure traits or function pointers. + +## Alternatives considered + +### Syntax for optionals + +This proposal recommends that an optional of an opaque result type with a conformance constraint to a protocol `P` be notated `(some P)?`. However, a user's first instinct might be to write `some P?`. This latter syntax is moderately less verbose, and is, in fact, unambiguous since `Optional

` is not a valid opaque result type constraint. It would be possible to add a special case that expands `some P?` into `(some P)?`. The analogous thing would be done with `some P!` and `(some P)!`. + +However, this is inconsistent with other parts of the language, e.g. the interpretation of `() -> P?` as `() -> Optional

` or the fact that `P & Q?` is an invalid construction which is properly written as `(P & Q)?`. Adding special cases to the language can decrease its learnability. + +Furthermore, since `P?` is never a correct constraint, it would be possible to (and in fact this proposal's implementation does) provide a "fix it" to the user which suggests that they change `some P?` to `(some P)?`. + +### Higher order functions + +Consider the function `func f() -> (some P) -> ()`. If this were a valid structural opaque result type, the closure value produced by calling `f` has type `(some P) -> ()`, meaning it takes an opaque result type as an argument. That argument has some concrete type, `T`, determined by the body of the closure. Assuming no special structure on `P`, such as `ExpressibleByIntegerLiteral`, the user cannot call the closure. If they were able to, then they would be depending at the source level on the concrete type of `T` to remain fixed, which is one of the things opaque result types are designed to prevent. + +Another reason to disallow returning functions that take opaque result types is that [SE-0341: Opaque Parameter Declarations](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md) proposes a different meaning for `some` in parameter position in function declarations, which would cause confusion if opaque parameter types mean something different within a function type. + +### Constraint inference + +We could infer additional constraints on opaque result types based on context, but this would likely be confusing for the user. Whereas the syntax for generic parameters draws the user's attention to the underlying type itself, i.e. the `T` in `f`, opaque result type syntax draws the user's attention to an explicit list of the protocols which the underlying type satisfies, i.e. the `P` in `some P`. At least one constraining protocol must be specified, unlike with generic parameters. The closest thing to `` one can write is `some Any`. + +The decision about what to do in this case seems pretty clear, which is why this section was not included in the original version of this proposal. The main utility of this discussion is in teasing out why opaque result types should function differently than generic parameters because of the implications this has for [named opaque result types](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--reverse-generics), which will likely be proposed in the future. + +**Though this is outside the scope of this proposal**, named opaque result types have a similar syntactic quality to generic parameters, and therefore should probably be subject to constraint inference in the result of a function, as well as the type of variable or the result type of a subscript. + +## Future directions + +This proposal is a natural stepping stone to fully generalized reverse generics, as demonstrated by the following code snippet from the [generics UI design document](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--reverse-generics): + +```swift +func groupedValues(in collection: C) -> (even: Output, odd: Output) + where C.Element == Int, Output.Element == Int +{ + return (even: collection.lazy.filter { $0 % 2 == 0 }, + odd: collection.lazy.filter { $0 % 2 != 0 }) +} +``` + +This syntax is powerful, but it's unnecessarily verbose for many cases, such as naming an opaque result type with a conformance requirement to `Collection` in order to constrain the `Element` type. We can introduce a more natural syntax for simple associated type constraints, such as writing constraints on associated types names in angle-brackets: + +```swift +func concatenate(a: some Collection<.Element == T>, b: some Collection<.Element == T>) -> some Collection<.Element == T> +``` + +or using a more [light-weight same-type constraint syntax](https://forums.swift.org/t/pitch-light-weight-same-type-constraint-syntax/52889): + +```swift +func concatenate(a: some Collection, b: some Collection) -> some Collection +``` diff --git a/proposals/0329-clock-instant-duration.md b/proposals/0329-clock-instant-duration.md new file mode 100644 index 0000000000..743e21a61d --- /dev/null +++ b/proposals/0329-clock-instant-duration.md @@ -0,0 +1,564 @@ +# Clock, Instant, and Duration + +* Proposal: [SE-0329](0329-clock-instant-duration.md) +* Author: [Philippe Hausler](https://github.com/phausler) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#40609](https://github.com/apple/swift/pull/40609) +* Review: ([first review](https://forums.swift.org/t/se-0329-clock-instant-date-and-duration/53309)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0329-clock-instant-date-and-duration/53635)) ([second review](https://forums.swift.org/t/se-0329-second-review-clock-instant-and-duration/54509)) ([third review](https://forums.swift.org/t/se-0329-third-review-clock-instant-and-duration/54727)) ([acceptance](https://forums.swift.org/t/accepted-se-0329-clock-instant-and-duration/55324)) + +

+Revision history + +* **v1** Initial pre-pitch +* **v1.1** Refinement to clock, deadline and duration types + * Expanded to include a Deadline type + * Posed a Clock defined type protocol grouping instead of Duration based protocol grouping (Clock -> Deadline -> Duration rather than Duration + Clock) +* **v1.2** + * Removed the DurationProtocol concept to aide ease of use and simplify implementations + * Introduced WallClock.Duration as the lowered Date type +* **v1.3** + * Rename Deadline to Instant since that makes a bit more sense generally (especially for Date) + * Add the requirement of a referencePoint to ClockProtocol +* **v1.4** + * Move the concept of `now` to the protocol requirement for clocks as an instance method + * Move the `duration(from:to:)` to an instance method on `InstantProtocol` + * Add a number of really useful operators + * Concrete clock types now have `.now` on their `Instant` types + * Added an example ManualClock +* **v1.4.1** + * Clarify the concrete clock types to show their conformances +* **v1.4.2** + * Move the measurement function to clock itself to prevent conflicts with existing APIs +* **v1.4.3** + * Re-added hours and minutes construction + * added a base requirement for `ClockProtocol` to require a `minimumResolution` +* **v1.4.4** + * Rename `ClockProtocol` to `Clock` to better adhere to naming guidelines + * Adjusted measurement to have both a required method (plus default implementation) as well as a free floating function for standardized measurement. +* **v2.0** + * Remove `Date` lowering + * Remove `WallClock` + * Add tolerances + * Remove `.hours` and `.minutes` + * Proposal reorganization + * Added `DurationProtocol` and per instant interval association + * Rename Monotonic and Uptime clocks to Continuous and Suspending to avoid platform ambiguity and perhaps add more clarity of uses. +* **v2.1** + * Refined `DurationProtocol` to only use `Int` instead of `BinaryInteger` for arithmetic. + * Added some additional alternatives considered for `DurationProtocol` and naming associated with it. + * Renamed the associated type on `InstantProtocol` to be `Duration`. + * Added back in task based sleep methods. Added a shorthand for sleeping tasks given a Duration. +* **v3.0** + * Moved `measure` into a category from a protocol requirement + * Renamed the `nanoseconds` and `seconds` property of `Duration` to `nanosecondsPortion` and `secondsPortion` to indicate their fractional composition to types like `timespec` +* **v3.1** + * Adjust the portion accessors to one singular `components` based accessor and add an initializer for raw value construction from components. +* **v3.2** + * Add `Duration` as an associated type requirement of `Clock`, so that it can be marked as the primary associated type. + +
+ +## Introduction + +The concepts of time can be broken down into three distinct parts: + +1. An item to provide a concept of now plus a way to wake up after a given point in time +2. A concept of a point in time +3. A concept of a measurement of elapsed time. + +These three items are respectively a **clock**, an **instant** and a **duration**. The measurement of time can be used for many types of APIs, all the way from the high levels of a concept of a timeout on a network connection, to the amount of time to sleep a task. Currently, the APIs that take measurement of time types take `NSTimeInterval` (aka `TimeInterval`), `DispatchTimeInterval`, and even types like `timespec`. + +## Motivation + +To define a standard way of interacting with time, we need to ensure that in the cases where it is important to limit clock measurement to a specific concept, that ability is preserved. For example, if an API can only accept realtime deadlines as instants, that API cannot be passed to a monotonic instant. This specificity needs to be balanced with the ergonomics of being able to use high-level APIs with little encumbrance of needing to know exactly the time type that is needed; in UI, it might not be welcoming to new developers learning Swift to force them to understand the differential between the myriad of clock concepts available for the operating system. Likewise, any implementation must be robust and performant enough to support multiple operating system back ends (Linux, Darwin, Windows, etc.), but also be easy enough to get right for the common use cases. Practically speaking, durations should be a progressive disclosure to instants and clocks. + +From a performance standpoint, a distinct requirement is that any duration type (or clock type) must be reasonably performant enough to do tasks like measuring the execution performance of a function, without incurring a large overhead to the execution of the measurement. This means that any type that is expressing a duration should be small, and likely backed by some sort of (or group of) PoD type(s). + +Time itself is always measured in a manner that is in reference to a certain frame of analysis. For example, uptime is measured in relative perspective to how long the machine has been booted, whereas other clocks may be relative to a specific epoch. Any instants expressed in terms of a specific reference point may be converted in potentially a lossy manner whereas others may not be convertible at all; so these conversions cannot be uniformly expressed as a general protocol requirement. + +The primary motivation for clocks is to offer a way to schedule work to be done at a later time. Instants are intended to serve a temporal reference point for that scheduling. Durations are specifically designed to be a high precision integral time representing an elapsed duration between two points in time. + +As it stands today, there are a number of APIs and types to represent clocks, instants, and durations. Foundation, for example, defines instant as `Date`, which is constructed from a UTC reference point using an epoch of Jan 1 2001, and `TimeInterval` which is defined as a `Double` representing the number of seconds between two points in time. Dispatch defines `DispatchTime`, `DispatchWallTime`, and `DispatchTimeInterval`; these, respectively, work in relation to a reference of uptime, a wall clock time, and a value of seconds/milliseconds/microseconds/nanoseconds. These obviously are not the only definitions, but when dealing with concurrency, a uniform accessor to all of these concepts is helpful to build the primitives needed for sleep and other temporal concepts. + +## Prior Art + +This proposal focuses on time as used for scheduling work in a process. The most useful clocks for this purpose are simple and local ones that calculate the time since the machine running the process was started. Time can also be expressed in human terms by using calendars, like "April 1, 2021" in the Gregorian calendar. To align with the different responsibilities of the standard library and Foundation, we aim to leave the definition of calendars and the math related to moving between dates in a calendar to Foundation's `Calendar`, `DateComponents`, `TimeZone` and `Date` types. + +For brevity three other languages were chosen to represent an analysis of how time is handled for other languages; Go, Rust, Kotlin. These by no means are the only examples in other languages. Python and C++ also have notable implementations that share some similarities with the proposed implementation. + +### Go +https://pkg.go.dev/time +https://golang.org/src/time/time.go + +Go stores time as a structure of a wall clock reference point (uint64), an 'ext' additional nanoseconds field (int64), and a location (pointer). +Go stores duration as an alias to int64 (nanoseconds). + +There is no control over the reference points in Go to specify a given clock; either monotonic or wall clock. The base implementation attempts to encapsulate both monotonic and wall clock values together in Go. For common use case this likely has little to no impact, however it lacks the specificity needed to identify a progressive disclosure of use. + +### Rust +https://doc.rust-lang.org/stable/std/time/struct.Duration.html + +Rust stores duration as a u64 seconds and a u32 nanoseconds. +The measurement of time in Rust uses Instant, which seems to use a monotonic clock for most platforms. + +### Kotlin +https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.time/-duration/ + +Kotlin stores Duration as a Long plus a unit discriminator comprised of either milliseconds or nanoseconds. Kotlin's measurement functions do not return duration (yet?) but instead rely on conversion functions from Long values in milliseconds etc and those currently measurement functions use system uptime to determine reference points. + +## Detailed Design + +Swift can take this to another level of both accuracy of intent and ease of use beyond any of the other examples given. Following in the themes of other Swift APIs, we can embrace the concept of progressive disclosure and leverage the existing frameworks that define time concepts. + +The given requirements are that we must have a way of expressing the frame of reference of time. This needs to be able to express a concept of now, and a concept of waking up after a given instant has passed. Instants must be able to be compared among each other but are specific to the clock they were obtained. Instants also must be able to be advanced by a given duration or a distance between two instants must be able to emit a duration. Durations must be comparable and also must have some intrinsic unit of time that can suffice for broad application. + +### Clock + +The base protocol for defining a clock requires two primitives; a way to wake up after a given instant, and a way to produce a concept of now. Clocks can also be defined in terms of a potential resolution of access; some clocks may offer resolution at the nanosecond scale, other clocks may offer only microsecond scale. Any values of elapsed time may be considered to be 0 if they are below the minimum resolution. + +```swift +public protocol Clock: Sendable { + associatedtype Duration: DurationProtocol + associatedtype Instant: InstantProtocol where Instant.Duration == Duration + + var now: Instant { get } + + func sleep(until deadline: Instant, tolerance: Instant.Duration?) async throws + + var minResolution: Instant.Duration { get } +} + +extension Clock { + func measure(_ work: () async throws -> Void) reasync rethrows -> Instant.Duration +} +``` + +This means that given an instant, it is intrinsically linked to the clock; e.g., a specific clock's instant is not meaningfully comparable to all other clock instants. However, as an ease of use concession, the duration between two instants can be compared. However, doing this across clocks is potentially considered a programmer error, unless handled very carefully. By making the protocol hierarchy just clocks and instants, it means that we can easily express a compact form of a duration that is usable in all cases; particularly for APIs that might adopt Duration as a replacement to an existing type. + +The clock minimum resolution will have a default implementation that returns `.nanosecond(1)`. This property serves to inform users of a clock the potential minimum granularity of what to invocations to now may return but also indicate the minimum variance between two instants that are significant. Practically speaking, this becomes relevant when measuring work - execution of a small work load may be executed in under the minimum resolution and not provide accurate information. + +Clocks can then be used to measure a given amount of work. This means that clock should have the extensions to allow for the affordance of measuring workloads for metrics, but also measure them for performance benchmarks. This means that making benchmarks is quite easy to do: + +```swift +let elapsed = someClock.measure { + someWorkToBenchmark() +} +``` + +The primary use for a clock beyond vending `now` is to wake up after a given deadline. This affords the possibility to schedule work to be done after that given instant. Wake-ups for scheduled work can incur power implications. Specifically waking up the CPU too often can cause undue power drain. By indicating a tolerance to the deadline it allows the underlying scheduling mechanisms from the kernel to potentially offer a slightly adjusted deadline to wake up by which means that work along with other work being scheduled can be grouped together for more power efficient execution. Not specifying a tolerance infers to the implementor of the clock that the tolerance is up to the implementation details of that clock to choose an appropriate value. The tolerance is a maximum duration after deadline by which the system may delay sleep by. + +``` +func delayedHello() async throws { + try await someClock.sleep(until: .now.advanced(by: .seconds(3)) + print("hello delayed world") +} +``` + +In the above example a clock is slept until 3 seconds from the instant it was called and then prints. The sleep function should throw if the task was cancelled while the sleep function is suspended. In this example the tolerance value is defaulted to nil by the clock and left as a "dealers choice" of how much tolerance may be applied to the deadline. + +### Instant + +As previously stated, instants need to be compared, and might be stored as a key, but only need to define a concept of now, and a way to advance them given a duration. By utilizing a protocol to define an instant, it provides a mechanism in which to use the right storage for the type, but also be type safe with regards to the clock they are intended for. + +The primary reasoning that instants are useful is that they can be composed. Given a function with a deadline as an instant, if it calls another function that takes a deadline as an instant, the original can just be passed without mutation to the next function. That means that the instant in which that deadline elapses does not have interference with the pre-existing calls or execution time in-between functions. One common example of this is the timeout associated with url requests; a timeout does not fully encapsulate how the execution deadline occurs; there is a deadline to meet for the connection to be established, data to be sent, and a response to be received; a timeout spanning all of those must then have measurement to account for each step, whereas a deadline is static throughout. + +```swift +public protocol InstantProtocol: Comparable, Hashable, Sendable { + associatedtype Duration: DurationProtocol + func advanced(by duration: Duration) -> Self + func duration(to other: Self) -> Duration +} + +extension InstantProtocol { + public static func + (_ lhs: Self, _ rhs: Duration) -> Self + public static func - (_ lhs: Self, _ rhs: Duration) -> Self + + public static func += (_ lhs: inout Self, _ rhs: Duration) + public static func -= (_ lhs: inout Self, _ rhs: Duration) + + public static func - (_ lhs: Self, _ rhs: Self) -> Duration +} +``` + +`InstantProtocol`, in addition to the `advance(by:)` and `duration(to:)` methods, has operators to add and subtract durations. However, it does not adhere to `AdditiveArithmetic`. That protocol would require adding two instant values together and defining a zero value (which comes from the clock, and cannot be statically know for all `InstantProtocol` types). Furthermore, InstantProtocol does not require `Strideable` because that requires the stride to be `SignedNumeric` which means that `Duration` would be required to be multiplied by another `Duration` which is inappropriate for two durations. + +If at such time that `Strideable` no longer requires `SignedNumeric` strides, or that `SignedNumeric` no longer requires the multiplication of self; this or adopting types should be considered for adjustment. + +### DurationProtocol + +Specific clocks may have concepts of durations that may express durations outside of temporal concepts. For example a clock tied to the GPU may express durations as a number of frames, whereas a manual clock may express them as steps. Most clocks however will express their duration type as a `Duration` represented by an integral measuring seconds/nanoseconds etc. We feel that it is not an incredibly common task to implement a clock and using the extended name of `Swift.Duration` is reasonable to expect and does not impact normal interactions with clocks. This duration has a few basic requirements; it must be comparable, and able to be added (similar to the concept previously stated with `InstantProtocol` they cannot be `Stridable` since it would mean that two `DurationProtocol` adopting types would then be allowed to be multiplied together). + +```swift +public protocol DurationProtocol: Comparable, AdditiveArithmetic, Sendable { + static func / (_ lhs: Self, _ rhs: Int) -> Self + static func /= (_ lhs: inout Self, _ rhs: Int) + static func * (_ lhs: Self, _ rhs: Int) -> Self + static func *= (_ lhs: inout Self, _ rhs: Int) + + static func / (_ lhs: Self, _ rhs: Self) -> Double +} +``` + +In order to ensure efficient calculations for durations there must be a few additional methods beyond just additive arithmetic that types conforming to `DurationProtocol` must implement - these are the division and multiplication by binary integers and a division creating a double value. This provides the most minimal set of functions to accomplish concepts like the scheduling of a timer, or back-off algorithms. This protocol definition is very close to a concept of `VectorSpace`; if at such time that a more refined protocol definition for a composition of `Comparable` and `AdditiveArithmetic` comes to be - this protocol should be considered as part of any potential improvement in that area. + +The naming of `DurationProtocol` was chosen because we feel that the canonical definition of durations is a temporal duration. All clocks being proposed here have an interval type of `Swift.Duration`; but other more specialized clocks may offer duration types that provide their own custom durations. + +### Duration + +Meaningful durations can always be expressed in terms of nanoseconds plus a number of seconds, either a duration before a reference point or after. They can be constructed from meaningful human measured (or machine measured precision) but should not account for any calendrical calculations (e.g., a measure of days, months or years distinctly need a calendar to be meaningful). Durations should able to be serialized, compared, and stored as keys, but also should be able to be added and subtracted (and zero is meaningful). They are distinctly NOT `Numeric` due to the aforementioned issue with regards to multiplying two `TimeInterval` variables. That being said, there is utility for ad-hoc division and multiplication to calculate back-offs. + +The `Duration` must be able to account for high scale resolution of calculation; the storage will under the hood ensure proper rounding for division (by likely storing higher precision than exposed) and enough range to span the full range of potential reasonable instants. This means that spanning the full range of +/- thousands of years at a non lossy scale can be accomplished by storing the seconds and nanoseconds. Not all systems will need that full range, however in order to properly represent nanosecond precision across the full range of times expressed in the operating systems that Swift works on a full 128 bit storage is needed to represent these values. That in turn necessitates exposing the conversion to existing types as breaking the duration into two components. These components of a duration are exposed for interoperability with existing APIs such as `timespec` as a seconds portion and an attoseconds portion (used to ensure full precision is not lost). If the Swift language gains a signed integer type that can support 128 bits of storage then `Duration` should be considered to replace the components accessor and initializer with a direct access and initialization to that stored attoseconds value. + +```swift +public struct Duration: Sendable { + public var components: (seconds: Int64, attoseconds: Int64) { get } + public init(secondsComponent: Int64, attosecondsComponent: Int64) +} + + +extension Duration { + public static func seconds(_ seconds: T) -> Duration + public static func seconds(_ seconds: Double) -> Duration + public static func milliseconds(_ milliseconds: T) -> Duration + public static func milliseconds(_ milliseconds: Double) -> Duration + public static func microseconds(_ microseconds: T) -> Duration + public static func microseconds(_ microseconds: Double) -> Duration + public static func nanoseconds(_ value: T) -> Duration +} + +extension Duration: Codable { } +extension Duration: Hashable { } +extension Duration: Equatable { } +extension Duration: Comparable { } +extension Duration: AdditiveArithmetic { } + +extension Duration { + public static func / (_ lhs: Duration, _ rhs: Double) -> Duration + public static func /= (_ lhs: inout Duration, _ rhs: Double) + public static func / (_ lhs: Duration, _ rhs: Int) -> Duration + public static func /= (_ lhs: inout Duration, _ rhs: Int) + public static func / (_ lhs: Duration, _ rhs: Duration) -> Double + public static func * (_ lhs: Duration, _ rhs: Double) -> Duration + public static func *= (_ lhs: inout Duration, _ rhs: Double) + public static func * (_ lhs: Duration, _ rhs: Int) -> Duration + public static func *= (_ lhs: inout Duration, _ rhs: Int) +} + +extension Duration: DurationProtocol { } +``` + +### ContinuousClock + +When instants are for local processing only and need to be high resolution without the encumbrance of suspension while the machine is asleep, `ContinuousClock` is the tool for the job. On Darwin platforms this refers to time derived from the monotonic clock, for linux platforms this is in reference to the uptime clock; being that those two are the closest in behavioral meaning. This clock also offers an extension to access the clock instance as the inferred base type property. + +```swift +public struct ContinuousClock { + public init() + + public static var now: Instant { get } +} + +extension ContinuousClock: Clock { + public struct Instant { + public static var now: ContinuousClock.Instant { get } + } + + public var now: Instant { get } + public var minimumResolution: Duration { get } + public func sleep(until deadline: Instant, tolerance: Duration? = nil) async throws +} + +extension ContinuousClock.Instant: InstantProtocol { + public func advanced(by duration: Duration) -> ContinuousClock.Instant + public func duration(to other: ContinuousClock.Instant) -> Duration +} + +extension Clock where Self == ContinuousClock { + public static var continuous: ContinuousClock { get } +} +``` + +### SuspendingClock + +Where local process scoped or cross machine scoped instants are not suitable: uptime serves the purpose of a clock that does not increment while the machine is asleep but is a time that is referenced to the boot time of the machine. This allows for the affordance of cross process communication in the scope of that machine. Similar to the other clocks there is an extension to access the clock instance as the inferred base type property. For Darwin based platforms this is derived from the uptime clock whereas for linux based platforms this is derived from the monotonic clock since those most closely represent the concept for not incrementing while the machine is asleep. + +```swift +public struct SuspendingClock { + public init() + + public static var now: Instant { get } +} + +extension SuspendingClock: Clock { + public struct Instant { + public static var now: SuspendingClock.Instant { get } + } + + public var minimumResolution: Duration { get } + public func sleep(until deadline: Instant, tolerance: Duration? = nil) async throws +} + +extension SuspendingClock.Instant: InstantProtocol { + public func advanced(by duration: Duration) -> SuspendingClock.Instant + public func duration(to other: SuspendingClock.Instant) -> Duration +} + +extension Clock where Self == SuspendingClock { + public static var suspending: SuspendingClock { get } +} +``` + +### Clocks Outside of the Standard Library + +In previous iterations of this proposal we offered a concept of a WallClock, however, after some compelling feedback we feel that this type may not be the most generally useful without the context of calendrical calculations. Since Foundation is the home of these types of calculations we feel that a clock based upon UTC more suitably belongs in that layer. This clock will adjust the fire time based upon the current UTC time; this means that that if a bit of work is scheduled by a specific time of day made by calculation via `Calendar` this clock can wake up from the sleep when the system time hits that deadline. + +Foundation will provide a type `UTCClock` that encompasses this behavior and use `Date` as the instant type. Additionally Foundation will provide conversions to and from `Date` to the other instant types in this proposal. + +```swift +public struct UTCClock { + public init() + + public static var now: Date { get } +} + +extension UTCClock: Clock { + public var minimumResolution: Duration { get } + public func sleep(until deadline: Date, tolerance: Duration? = nil) async throws +} + +extension Date { + public func leapSeconds(to other: Date) -> Duration + public init(_ instant: ContinuousClock.Instant) + public init(_ instant: SuspendingClock.Instant) +} + +extension ContinuousClock.Instant { + public init?(_ instant: Date) +} + +extension SuspendingClock.Instant { + public init?(_ instant: Date) +} + +extension Date: InstantProtocol { + public func advanced(by duration: Duration) -> Date + public func duration(to other: Date) -> Duration +} + +extension Clock where Self == UTCClock { + public static var utc: UTCClock { get } +} +``` + +The `UTCClock` will allow for a method in which to wake up after a deadline defined by a `Date`. The implementation of `Date` transacts upon the number of seconds since Jan 1 2001 as defined by the system clock so any network time (or manual) updates may shift that point of now either forward or backward depending on the skew the system clock may undergo. The value being stored is not dependent upon timezone, daylight savings, or calendrical representation but the current NTP updates do represent any applied leap seconds that may have occurred. In light of this particular edge case that previously was not exposed, `Date` will now offer a new method to determine the leap second duration that may have elapsed between a given data and another date. This provides a method in which to account for these leap seconds in a historical sense. Similar to timezone databases the leap seconds will be updated (if there is any additional planned leap seconds) along with software updates. + +Previous revisions of this proposal moved `Date` to the standard library along with a new wall clock that uses it. After feedback from the community, we now believe the utility of this clock is very specialized and more closely related to the calendar types in Foundation. Therefore, `Date` will remain in Foundation alongside them. + +`Date` is best used as the storage for point in time to be interpreted using a `Calendar`, `TimeZone`, and with formatting functions for display to people. A survey of the existing `Date` API in the macOS and iOS SDKs shows this to already be the case for the vast majority of properties and functions that use it. The discussion around the appropriateness of the `Date` name was mostly focused on its uses in *non*-calendrical contexts. We hope this combination of `Date` and `UTCClock` will help reinforce the relationship between those types and add clarity to when it should be used. + +This approach preserves compatibility with those APIs while still providing the capability to use `Date` for scheduling in the rare cases that it is needed. + +### Task + +The existing `Task` API has methods in which to sleep. These existing methods do not have any specified behavior of sleeping; however under the hood it uses a continuous clock on Darwin and a suspending clock on Linux. + +The existing API for sleeping will be deprecated, and the existing deprecation will be updated accordingly to point to the new APIs. + +```swift +extension Task { + @available(*, deprecated, renamed: "Task.sleep(for:)") + public static func sleep(_ duration: UInt64) async + + @available(*, deprecated, renamed: "Task.sleep(for:)") + public static func sleep(nanoseconds duration: UInt64) async throws + + public static func sleep(for: Duration) async throws + + public static func sleep(until deadline: C.Instant, tolerance: C.Instant.Duration? = nil, clock: C) async throws +} +``` + +### Example Custom Clock + +One example for adopting `Clock` is a manual clock. This could be a useful item for testing (but not currently part of this proposal as an API to add). It allows for the manual advancement of time in a deterministic manner. The general intent is to allow the manual clock type to be advanced from one thread and the sleep function can then be used to act as if it was a standard clock in generic APIs. + +```swift +public final class ManualClock: Clock, @unchecked Sendable { + public struct Instant: InstantProtocol { + var offset: Duration = .zero + + public func advanced(by duration: Duration) -> ManualClock.Instant { + Instant(offset: offset + duration) + } + + public func duration(to other: ManualClock.Instant) -> Duration { + other.offset - offset + } + + public static func < (_ lhs: ManualClock.Instant, _ rhs: ManualClock.Instant) -> Bool { + lhs.offset < rhs.offset + } + } + + struct WakeUp { + var when: Instant + var continuation: UnsafeContinuation + } + + public private(set) var now = Instant() + + // General storage for the sleep points we want to wake-up for + // this could be optimized to be a more efficient data structure + // as well as enforced for generation stability for ordering + var wakeUps = [WakeUp]() + + // adjusting now or the wake-ups can be done from different threads/tasks + // so they need to be treated as critical mutations + let lock = os_unfair_lock_t.allocate(capacity: 1) + + deinit { + lock.deallocate() + } + + public func sleep(until deadline: Instant, tolerance: Duration? = nil) async throws { + // Enqueue a pending wake-up into the list such that when + return await withUnsafeContinuation { + if deadline <= now { + $0.resume() + } else { + os_unfair_lock_lock(lock) + wakeUps.append(WakeUp(when: deadline, continuation: $0)) + os_unfair_lock_unlock(lock) + } + } + } + + public func advance(by amount: Duration) { + // step the now forward and gather all of the pending + // wake-ups that are in need of execution + os_unfair_lock_lock(lock) + now += amount + var toService = [WakeUp]() + for index in (0..<(wakeUps.count)).reversed() { + let wakeUp = wakeUps[index] + if wakeUp.when <= now { + toService.insert(wakeUp, at: 0) + wakeUps.remove(at: index) + } + } + os_unfair_lock_unlock(lock) + + // make sure to service them outside of the lock + toService.sort { lhs, rhs -> Bool in + lhs.when < rhs.when + } + for item in toService { + item.continuation.resume() + } + } +} +``` + +## Existing Application Code + +This proposal is purely additive and has no direct impact to existing application code. + +## Impact on ABI + +The proposed implementation will introduce three runtime functions; a way of obtaining time, a way of sleeping given a standard clock, and a way of obtaining the minimum resolution given a standard clock. + +## Alternatives Considered + +### Singular Instant Representation + +It was considered to have a singular type to represent monotonic, uptime, and wall clock instants similar to Go. However this approach causes a problem with comparability; an instant may be greater in one respect but less or equal in some other respect. In order to properly adhere to `Comparable` as a requisite to `InstantProtocol` we feel that combining the instants into one unified type is not ideal. + +### Inverted Protocol Hierarchy + +Another exploration was to have an inverted scheme of instant and clock however this means that the generic signatures of functions that use specific clocks or instants become much more difficult to write. + +### Lowering of Date/UTCClock + +Originally the proposal included a concept of lowering `Date` to the standard library in addition to altering its storage from `Double` to a `Duration`. There were strong objections on a few fronts with this move which ultimately had convincing merit. The primary objection was to the name `Date`; given that there was no additional contextual API within the standard library or concurrency library this meant that `Date` could easily get confused with the concept of a calendrical date (which that type definitively is not). Additionally it was rightfully brought up that `Date` is missing concepts of leap seconds (which has since been accepted and proposed as an alteration to Foundation) because we see the utility of that as an additional functionality to `Date`. + +Also in the original revisions of the proposal we had a concept of `WallClock`. After much discussion we feel that the name wall clock is misleading since the type really represents a clock based on UTC (once `Date` has a historical accounting of leap seconds). But furthermore, we feel that the general utility of scheduling via a UTC clock is not a common task and that a vast majority of clocks for scheduling are really things that transact either via a clock that time passes while the machine is asleep or a clock that time does not pass while the machine is asleep. That accounting means that we feel that the right home for `UTCClock` is in a higher level framework for that specialized task along side the calendrical calculation APIs; which is Foundation. + +### DurationProtocol Generalized Arithmetics and Protocol Definition + +It was considered to have a more general form of the arithmetics for `DurationProtocol`. This poses a potential pitfall for adopters that may inadvertently implement some truncation of values. Since most values passed around that are integral types are spelled as `Int` it means that this interface is better served as just using multiplication and division via `Int`. In that vein; it was also considered to use `Double` instead, this however does not work nicely for types that define durations like "steps" or "frames"; e.g. things that are not distinctly divisible beyond 1 unit. It is still under the domain of that `DurationProtocol` adopting type to define that behavior and how it rounds or asserts etc. + +Similarly to the arithmetics; it was also considered to have the associated type to `InstantProtocol` as just a glob of protocols `Comparable & AdditiveArithmetic & Sendable`, however this lacks the capability of fast-paths for things like back-offs (ala Zeno's algorithm) or debounce, or timer coalescing. Some of them could be re-written in terms of loops of addition, however it would likely result in hot-looping over missed intervals in some cases, or in others not even being able to implement them (e.g. division for back-offs). + +### Clock and Task Sleep Tolerance Optionality + +It was raised that the hint from IDEs such as Xcode for the `.none` autocomplete do exist and those nomenclatures are perhaps misleading for the `tolerance` parameter to the sleep functions. We agree that this is perhaps a less than ideal name to expose as an autocomplete, however it was decided that code using `.none` instead of not passing a parameter or passing `nil` is stylistically problematic and left-overs from earlier versions of swift. It was concluded that the solutions in this space should be applicable to any other method that has an optional parameter and not just `Clock` and `Task`; moreover it seems like this is perhaps a bug in Xcode's autocomplete than an issue with the API as proposed since the `ContinuousClock`, `SuspendingClock` and `UTCClock` being proposed are most meaningful of the lack of a parameter value than to introduce any sort of enumeration mirroring `Optional` without any sort of direct type passing capability. In short - a more general solution should be approached with this problem and the optional duration type should remain. + +### Alternative Names + +There have been a number of names that have been considered during this proposal (these are a few highlights): + +The protocol `Clock` has been considered to be named: +* `ClockProtocol` - The protocol suffix was considered superfluous and a violation of the naming guidelines. + +The protocol `InstantProtocol` has been considered to be named: +* `ReferencePoint` - This ended up being too vague and did not capture the concept of time +* `Deadline`/`DeadlineProtocol` - Not all instant types are actually deadlines, so the nomenclature became confusing. +* The associated type of `InstantProtocol.Duration` was considered for a few other names; `TimeSpan` and `Interval`. These names lack symmetry; `Clock` has an `Instant` which is an `InstantProtocol`, `InstantProtocol` has a `Duration` which is a `DurationProtocol`. + +The protocol `DurationProtocol` has been considered to be named: +* Not having it has been considered but ultimately rejected to ensure flexibility of the API for other clock types that transact in concept like "frames" or "steps". + +The clock `ContinuousClock` has been considered to be named: +* `MonotonicClock` - Unfortunately Darwin and Linux differ on the definition of monotonic. +* `UniformClock` - This does not disambiguate the behavioral difference between this clock and the `SuspendingClock` since both are uniform in their incrementing while the machine is not asleep. + +The clock `SuspendingClock` has been considered to be named: +* `UptimeClock` - Just as `MonotonicClock` has ambiguity with regards to Linux and Darwin behaviors. +* `AbsoluteClock` - Very vague when not immediately steeped in mach-isms. +* `ExecutionClock` - The name more infers the concept of `CLOCK_PROCESS_CPUTIME_ID` than `CLOCK_UPTIME_RAW` (on Darwin). +* `DiscontinuousClock` - Has its roots in the mathematical concept of discontinuous functions but perhaps is not immediately obvious that it is the clock that does not advance while the machine is asleep + +The type `Duration` has been considered to be named: +* `Interval` - This is quite ambiguous and could refer to numerous other concepts other than time. +* The `nanosecondsPortion` and `secondsPortion` were considered to be named `nanoseconds` and `seconds` however those names posed ambiguity of rounding; naming them with the term portion infers their fractional composition rather than just a rounded/truncated value. + +The type `Date` has been considered to be named: +* `Timestamp` - A decent alternative but still comes at a slight ambiguity with regards to being tied to a calendar. Also has string like connotations (with how it is used in logs) +* `Timepoint`/`TimePoint` - A reasonable alternative with less ambiguity but ultimately not compelling enough to churn thousands of APIs that already exist (just counting the ones included in the iOS and macOS SDKs, not to mention the other use sites that may exist). +* `WallClock.Instant`/`UTCClock.Instant` - This is a very wordy way of spelling the same idea as `Date` represents today. + +The `Task.sleep(for:tolerance:clock:)` API has been considered to be named: +* `Task.sleep(_:tolerance:clock:)` - even though this is still grammatically correct and omits potentially a needless word of "for", having this extra word still reads well but also offers a better fix-it for migration from deprecated APIs. That migration was considered worth it to keep the "for". + +## Appendix + +Time is relative, temporal types doubly so. In this document, there will be some discussion with regards to the categorization of temporal types that readers should be distinctly aware of. + +**Calendar:** A human locale based system in which to measure time. + +**Clock:** The mechanism in which to measure time, and understand how that time flows. + +**Continuous Time:** Time that always increments but does not stop incrementing while the system is asleep. This is useful to consider as a stopwatch style time; the reference point at which this starts and are most definitely different per machine. + +**Date:** A Date value encapsulates a single point in time, independent of any particular calendrical system or time zone. Date values represent a time interval relative to an absolute reference date. + +**Deadline:** In common parlance, it is a limit defined as an instant in time: a narrow field of time by which an objective must be accomplished. + +**Duration:** A measurement of how much time has elapsed between two deadlines or reference points. + +**Instant:** A precise moment in time. + +**Monotonic Time:** Darwin and BSD define this as continuous time. Linux, however, defines this as a time that always increments, but does stop incrementing while the system is asleep. + +**Network Update Time:** A value of wall clock time that is transmitted via ntp; used to synchronize the wall clocks of machines connected to a network. + +**Temporal:** Related to the concept of time. + +**Time Zone:** An arbitrary political defined system in which to normalize time in a quasi-geospatial delineation intended to keep the apex of the solar day around 12:00. + +**Tolerance:** The duration around a given point in time is accepted as accurate. + +**Uptime:** Darwin and BSD define this as absolute time suspending when asleep. Linux, however, defines this as time that does not suspend while asleep but is relative to the boot. + +**Wall Clock Time:** Time like reading from a clock. This may be adjusted forwards or backwards for numerous reasons; in this context, it is time that is not specific to a time zone or locale, but measured from an absolute reference date. Network updates may adjust the drift on the clock either backwards or forwards depending on the relativistic drift, clock skew from inaccuracies with the processor, or from hardware power characteristics. diff --git a/proposals/0330-collection-conditionals.md b/proposals/0330-collection-conditionals.md new file mode 100644 index 0000000000..ea6892efaf --- /dev/null +++ b/proposals/0330-collection-conditionals.md @@ -0,0 +1,61 @@ +# Conditionals in Collections + +* Proposal: [SE-0330](0330-collection-conditionals.md) +* Authors: [John Holdsworth](https://github.com/johnno1962) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Returned for revision** +* Decision notes: [Rationale](https://forums.swift.org/t/se-0330-conditionals-in-collections/53375/22) +* Implementation: [apple/swift#19347](https://github.com/apple/swift/pull/19347) +* Bugs: [SR-8743](https://bugs.swift.org/browse/SR-8743) + +## Introduction + +This is a lightning proposal to extend the existing Swift language slightly to allow `#if` conditional inclusion of elements in array and dictionary literals. For example: + +```swift +let array = [ + 1, + #if os(Linux) + 2, + #endif + 3] +let dictionary = [ + #if DEBUG + "a": 1, + #if swift(>=5.0) + "b": 2, + #endif + #endif + "c": 3] +``` +Swift-evolution thread: [Allow conditional inclusion of elements in array/dictionary literals?](https://forums.swift.org/t/allow-conditional-inclusion-of-elements-in-array-dictionary-literals/16171) + +## Motivation + +The most notable use case for this is conditional inclusion of tests for the Swift version of XCTest though it is certain to have other applications in practice allowing data to be tailored to production/development environments, architecture or build configuration. + +## Proposed solution + +The solution proposed is to allow #if conditionals using their exiting syntax inside collection literals surrounding sublists of elements. These elements would be either included or not included in the resulting array or dictionary instance dependant on the truth of the `#if`, `#elseif` or `#else` i.e. whether they where "active". One new syntactic requirement is the trailing comma in sublists before or inside conditional clauses is not optional as it would normally be at the end of the collection. + +## Detailed design + +The implementation involves a slight modification to `Parser::parseList` to detect `#if` "statements" if they are present and call `Parser::parseIfConfig` recursively call `parseList` to gather the elements in the clauses of the conditionals. Only the elements in the "active" clause of the conditional are included in the elements of the final `CollectionExpr` AST instance after parsing. + +As conditionals themselves and inactive elements are not included in the parser AST representation, a new data structure, the "Conditionals Map" is maintained on the `CollectionExpr` which is used to support features such as the AST dump and stripping conditionals from module interfaces. The syntax model of libSyntax also required minor modification. + +## Source compatibility + +N/A. This is an purely additive proposal for syntax that is not currently valid in Swift. + +## Effect on ABI stability + +N/A. This is a compile time alteration of a collections's elements. The resulting collection is a conventional container as it would have been without the conditional though exactly which elements are included can affect the collection's type. + +## Effect on API resilience + +N/A. This is not an API. + +## Alternatives considered + +It was decided to tackle this limited scope for the introduction of conditional syntax first, as specific use cases can be thought of and to be honest this has always seemed like a bit of an omission. Other areas where conditionals could be introduced abound but can be discussed with reference to their own particular subtleties of implementation separately at a later date. diff --git a/proposals/0331-remove-sendable-from-unsafepointer.md b/proposals/0331-remove-sendable-from-unsafepointer.md new file mode 100644 index 0000000000..c392413e48 --- /dev/null +++ b/proposals/0331-remove-sendable-from-unsafepointer.md @@ -0,0 +1,60 @@ +# Remove Sendable conformance from unsafe pointer types + +* Proposal: [SE-0331](0331-remove-sendable-from-unsafepointer.md) +* Authors: [Andrew Trick](https://github.com/atrick) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.6)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0331-remove-sendable-conformance-from-unsafe-pointer-types/53979) +* Implementation: [apple/swift#39218](https://github.com/apple/swift/pull/39218) + +## Introduction + +[SE-0302](0302-concurrent-value-and-concurrent-closures.md) introduced the `Sendable` protocol, including `Sendable` requirements for various language constructs, conformances of various standard library types to `Sendable`, and inference rules for non-public types to implicitly conform to `Sendable`. SE-0302 states that the unsafe pointer types conform to `Sendable`: + +> `Unsafe(Mutable)(Buffer)Pointer`: these generic types _unconditionally_ conform to the `Sendable` protocol. This means that an unsafe pointer to a non-Sendable value can potentially be used to share such values between concurrency domains. Unsafe pointer types provide fundamentally unsafe access to memory, and the programmer must be trusted to use them correctly; enforcing a strict safety rule for one narrow dimension of their otherwise completely unsafe use seems inconsistent with that design. + +Experience with `Sendable` shows that this formulation is unnecessarily dangerous and has unexpected negative consequences for implicit conformance. + +Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/unsafepointer-sendable-should-be-revoked/51926) + +## Motivation + +The role of `Sendable` is to prevent sharing reference-semantic types across actor or task boundaries. Unsafe pointers have reference semantics, and therefore should not be be `Sendable`. + +Unsafe pointers are unsafe in one primary way: it is the developer's responsibility to guarantee the lifetime of the memory referenced by the unsafe pointer. This is an intentional and explicit hole in Swift's memory safety story that has been around since the beginning. That well-understood form of unsafety should not be implicitly extended to allow pointers to be unsafely used in concurrent code. The concurrency diagnostics that prevent race conditions in other types with reference semantics should provide the same protection for pointers. + +Another problem with making the unsafe pointers `Sendable` is the second-order effect it has on value types that store unsafe pointers. Consider a wrapper struct around a resource: + +```swift +struct FileHandle { // implicitly Sendable + var stored: UnsafeMutablePointer +} +``` + +The `FileHandle` type will be inferred to be `Sendable` because all of its instance storage is `Sendable`. Even if we accept that an `UnsafeMutablePointer` by itself can be `Sendable ` because the "unsafe" can now apply to concurrency safety as well (as was argued in SE-0302), that same argument does not hold for the `FileHandle` type. Removing the conformance of the unsafe pointer types to `Sendable` eliminates the potential for it to propagate out to otherwise-safe wrappers. + +## Proposed solution + +Remove the `Sendable` conformance introduced by SE-0302 for the following types: + +* `AutoreleasingUnsafeMutablePointer` +* `OpaquePointer` +* `CVaListPointer` +* `Unsafe(Mutable)?(Raw)?(Buffer)?Pointer` +* `Unsafe(Mutable)?(Raw)?BufferPointer.Iterator` + +## Source compatibility + +The removal of `Sendable` conformances from unsafe pointer types can break code that depends on those conformances. There are two mitigating factors here that make us feel comfortable doing so at this time. The first mitigating factor is that `Sendable` is only very recently introduced in Swift 5.5, and `Sendable` conformances aren't enforced in the Swift Concurrency model just yet. The second is that the staging in of `Sendable` checking in the compiler implies that missing `Sendable` conformances are treated as warnings, not errors, so there is a smooth transition path for any code that depended on this now-removed conformances. + +## Effect on ABI stability + +`Sendable` is a marker protocol, which was designed to have zero ABI impact. Therefore, this proposal itself should have no ABI impact, because adding and removing `Sendable` conformances is invisible to the ABI. + +## Effect on API resilience + +This proposal does not affect API resilience. + +## Alternatives considered + +Keep SE-0302 behavior. This would require explicit "non-Sendable" annotation of many important aggregate types that embed unsafe pointers. diff --git a/proposals/0332-swiftpm-command-plugins.md b/proposals/0332-swiftpm-command-plugins.md new file mode 100644 index 0000000000..3e30dc8957 --- /dev/null +++ b/proposals/0332-swiftpm-command-plugins.md @@ -0,0 +1,814 @@ +# Package Manager Command Plugins + +* Proposal: [SE-0332](0332-swiftpm-command-plugins.md) +* Author: [Anders Bertelrud](https://github.com/abertelrud) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.6)** +* Implementation: [apple/swift-package-manager#3855](https://github.com/apple/swift-package-manager/pull/3855) +* Pitch: [Forum discussion](https://forums.swift.org/t/pitch-package-manager-command-plugins/) +* Review: [Forum discussion](https://forums.swift.org/t/se-0332-package-manager-command-plugins/) + +## Introduction + +[SE-0303](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md) introduced the ability to define *build tool plugins* in SwiftPM, allowing custom tools to be automatically invoked during a build. This proposal extends that plugin support to allow the definition of custom *command plugins* — plugins that users can invoke directly from the SwiftPM CLI, or from an IDE that supports Swift Packages, in order to perform custom actions on their packages. + +## Motivation + +The *build tool plugins* that were introduced in SE-0303 are focused on code generation during the build of a package, for such purposes as generating Swift source files from `.proto` files or from other inputs. In order to allow build tools to be incorporated into the build graph and to run automatically in a safe manner, there are restrictions on what such plugins can do. For example, build tool plugins are prevented from modifying any files inside a package directory. + +It would be useful to support a different kind of plugin that users can invoke directly, and that can be allowed to have more flexibility than a build tool that's invoked automatically during the build. Such custom command plugins could be used for documentation generation, source code reformatting, unit test report generation, build artifact post-processing, and other uses that don't fit the definition of a typical build tool. Rather than extending the build system, such plugins could extend and improve the workflow for package authors and users, whether or not those workflows have anything to do with the build system. + +One key tension in this proposal is between providing functionality that is rich enough to be useful for plugins, while still presenting that functionality in a way that's general enough to be implemented in both the SwiftPM CLI and in any IDE that supports packages. To that end, this proposal provides a minimal initial API, with the intention of adding more functionality in future proposals. + +Separately to this proposal, it would also be useful to define custom actions that could run as a side effect of operations such as building and testing, and which would be called in response to various events that can happen during a build or test run — but that is not what this proposal is about, and that kind of plugin would be the subject of a future proposal. Rather, this proposal focuses on the direct invocation of a custom command by a user, independently of whether the plugin that implements the command then decides to ask SwiftPM to perform a build as part of its implementation. + +## Proposed Solution + +This proposal defines a new plugin capability called `command` that allows packages to augment the set of package-related commands available in the SwiftPM CLI and in IDEs that support packages. A command plugin specifies the semantic intent of the command — this might be one of the predefined intents such “documentation generation” or “source code formatting”, or it might be a custom intent with a specialized verb that can be passed to the `swift` `package` command. A command plugin can also specify any special permissions it needs (such as the permission to modify the files under the package directory). + +The command's intent declaration provides a way of grouping command plugins by their functional categories, so that SwiftPM — or an IDE that supports SwiftPM packages — can show the commands that are available for a particular purpose. For example, this approach supports having different command plugins for generating documentation for a package, while still allowing those different commands to be grouped and discovered by intent. + +As with build tool plugins, command plugins are made available to a package by declaring dependencies on the packages that provide the plugins. Unlike build tool plugins, which are applied on a target-by-target basis using a declaration in the package manifest, custom command plugins are not invoked automatically — instead they can be invoked directly by the user after the package graph has been resolved. This proposal adds options to the `swift` `package` CLI that allow users to invoke a plugin-provided command and to control the set of targets to which the command should apply. It is expected that IDEs that support SwiftPM packages should provide a way to invoke the command plugins thorugh their user interfaces. + +Command plugins are implemented similarly to build tool plugins: each plugin is a Swift script that has access to API in the `PackagePlugin` module and that is invoked with parameters describing its inputs. The Swift script contains the logic to carry out the functionality of the plugin, usually by invoking other subprocesses, but potentially also by asking the package manager to perform certain actions such as building package products or running unit tests. + +Unlike build tool plugins, which operate indirectly by defining build commands for SwiftPM to run at a later point in time (and only when needed), custom command plugins directly carry out the functionality of the plugin at the time they are invoked. This usually involves invoking tools that are in the toolchain or that are provided by dependencies, but it could also involve logic that is implemented completely inside the plugin itself (using Foundation APIs, for example). The plugin does not return until the command is complete. + +Command plugins are provided with a read-only snapshot of the package, but they can also call into SwiftPM's build system to have it produce or update certain artifacts if it needs to. For example, a command that post-processes a release build can ask SwiftPM to build the release artifacts. + +In this initial proposal there is a fairly modest set of build parameters that can be controlled by the plugin. The intent is to extend this over time, although it will only be possible to support functionality that is common enough to be available in the build systems of any IDE that supports SwiftPM command plugins (the "host" of the plugin). + +As with all kinds of plugins, command plugins can emit diagnostics if it encounters any problems. Console output emitted by the command plugin is shown to users. + +## Detailed Design + +This proposal extends both the package manifest API and the package plugin API. + +### Manifest API + +This proposal defines a new `command` plugin capability in `PackageDescription`: + +```swift +extension PluginCapability { + /// Plugins that specify a `command` capability define commands that can be run + /// using the SwiftPM CLI (`swift package `), or in an IDE that supports + /// Swift Packages. + public static func command( + /// The semantic intent of the plugin (either one of the predefined intents, + /// or a custom intent). + intent: PluginCommandIntent, + + /// Any permissions needed by the command plugin. This affects what the + /// sandbox in which the plugin is run allows. Some permissions may require + /// approval by the user. + permissions: [PluginPermission] = [] + ) -> PluginCapability +} +``` + +The plugin specifies the intent of the command as either one of a set of predefined intents or as a custom intent with an custom verb and help description. + +In this proposal, the intent is expressed as an opaque struct with enum semantics in `PackageDescription`: + +```swift +public struct PluginCommandIntent { + /// The intent of the command is to generate documentation, either by parsing the + /// package contents directly or by using the build system support for generating + /// symbol graphs. Invoked by a `generate-documentation` verb to `swift package`. + public static func documentationGeneration() -> PluginCommandIntent + + /// The intent of the command is to modify the source code in the package based + /// on a set of rules. Invoked by a `format-source-code` verb to `swift package`. + public static func sourceCodeFormatting() -> PluginCommandIntent + + /// An intent that doesn't fit into any of the other categories, with a custom + /// verb through which it can be invoked. + public static func custom(verb: String, description: String) -> PluginCommandIntent +} +``` + +Future proposals will almost certainly add to this set of possible intents, using availability annotations gated on the tools version to conditionally make new types of intent available. + +If multiple command plugins in the dependency graph of a package specify the same intent, or specify a custom intent with the same verb, then the user will need to specify which plugin to invoke by qualifying the verb with the name of the plugin target followed by a `:` character, e.g. `MyPlugin:do-something`. Because plugin names are target names, they are already known to be unique within the package graph, so the combination of plugin name and verb is known to be unique. + +A command plugin can also specify the permissions it needs, which affect the ways in which the plugin can access external resources such as the file system or network. By default, command plugins have only read-only access to the file system (except for temporary-files locations) and cannot access the network. + +A command plugin that wants to modify the package source code (as for example a source code formatter might want to) needs to request the `writeToPackageDirectory` permission. This modifies the sandbox in which the plugin is invoked to let it write inside the package directory in the file system, after notifying the user about what is going to happen and getting approval in a way that is appropriate for the IDE in question. + +The permissions needed by the command are expressed as an opaque static struct with enum semantics in `PackageDescription`: + +```swift +public struct PluginPermission { + /// The command plugin wants permission to modify the files under the package + /// directory. The `reason` string is shown to the user at the time of request + /// for approval, explaining why the plugin is requesting this access. + public static func writeToPackageDirectory(reason: String) -> PluginPermission +} +``` + +Future proposals will almost certainly add to this set of possible permissions, using availability annotations gated on the tools version to conditionally make new types of permission available. + +In particular, it is likely that future proposals will want to provide a way for a plugin to ask for permission to access the network. In the interest of keeping this proposal bounded, we note that as a possible future need here, but do not initially allow any network access. + +### Plugin API + +This proposal extends the PackagePlugin API to: + +* define a new kind of plugin entry point specific to command plugins +* allow the plugin to ask the Swift Package Manager to perform actions such as building or testing +* allow the plugin to ask the Swift Package Manager for specialized information such as Swift symbol graphs + + +#### Plugin Entry Point + +This proposal extends `PluginAPI` with an entry point for command plugins: + +```swift +/// Defines functionality for all plugins that have a `command` capability. +public protocol CommandPlugin: Plugin { + /// Invoked by SwiftPM to perform the custom actions of the command. + func performCommand( + /// The context in which the plugin is invoked. This is the same for all + /// kinds of plugins, and provides access to the package graph, to cache + /// directories, etc. + context: PluginContext, + + /// Any literal arguments passed after the verb in the command invocation. + arguments: [String], + ) async throws + + /// A proxy to the Swift Package Manager or IDE hosting the command plugin, + /// through which the plugin can ask for specialized information or actions. + var packageManager: PackageManager { get } +} +``` + +This defines a basic entry point for a command plugin, passing it information about the context in which the plugin is invoked (including information about the package graph) and the arguments passed by the user after the verb in the `swift` `package` invocation. + +The `context` parameter provides access to the package to which the user applies the plugin, including any dependencies, and it also provides access to a working directory that the plugin can use for any purposes, as well as a way to look up command line tools with a given name. This is the same as the support that is available to all plugins via SE-0325. + +An opaque reference to a proxy for the Package Manager services in SwiftPM or the host IDE is also made available to the plugin, for use in accessing derived information and for carrying out more specialized actions. This is described in more detail below. + +Many command plugins will invoke tools using subprocesses in order to do the actual work. A plugin can use the Foundation module’s `Process` API to invoke executables, after using the PackagePlugin module's `PluginContext.tool(named:)` API to obtain the full path of the command line tool in the local file system. + +Plugins can also use Foundation APIs for reading and writing files, encoding and decoding JSON, and other actions. + +The arguments are a literal array of strings that the user specified when invoking the plugin. Plugins that operate on individual targets or products would typically support a `--target` or `--product` option that allows users to specify the names of targets or products to operate on in the package to which the plugin command is applied. + +#### Accessing Package Manager Services + +In addition to invoking invoking tool executables and using Foundation APIs, command plugins can use the `packageManager` property to obtain more specialized information and to invoke certain SwiftPM services. This is a proxy to SwiftPM or to the IDE that is hosting the plugin, and provides access to some of its functionality. The set of services provided in this API is expected to grow over time, and would ideally, over time, comprise most of the SwiftPM functionality available in its CLI. + +```swift +/// Provides specialized information and services from the Swift Package Manager or +/// an IDE that supports Swift Packages. Different plugin hosts will implement this +/// functionality in whatever way is appropriate for them, but should preserve the +/// same semantics described here. +public struct PackageManager { + // + // Building + // + + /// Performs a build of all or a subset of products and targets in a package. + /// + /// Any errors encountered during the build are reported in the build result, + /// as is the log of the build commands that were run. This method throws an + /// error if the input parameters are invalid or in case the build cannot be + /// started. + /// + /// The SwiftPM CLI or any IDE that supports packages may show the progress + /// of the build as it happens. + /// + /// Future proposals should consider adding ways for the plugin to receive + /// incremental progress during the build. + public func build( + _ subset: BuildSubset, + parameters: BuildParameters + ) async throws -> BuildResult + + /// Specifies a subset of products and targets of a package to build. + public enum BuildSubset { + /// Represents the subset consisting of all products and of either all + /// targets or (if `includingTests` is false) just non-test targets. + case all(includingTests: Bool) + + /// Represents the product with the specified name. + case product(String) + + /// Represents the target with the specified name. + case target(String) + } + + /// Parameters and options to apply during the build. + public struct BuildParameters { + /// Whether to build for debug or release. + public var configuration: BuildConfiguration = .debug + + /// Controls the amount of detail to include in the build log. + public var logging: BuildLogVerbosity = .concise + + /// Additional flags to pass to all C compiler invocations. + public var otherCFlags: [String] = [] + + /// Additional flags to pass to all C++ compiler invocations. + public var otherCxxFlags: [String] = [] + + /// Additional flags to pass to all Swift compiler invocations. + public var otherSwiftcFlags: [String] = [] + + /// Additional flags to pass to all linker invocations. + public var otherLinkerFlags: [String] = [] + + /// Future proposals should add more controls over the build. + } + + /// Represents an overall purpose of the build, which affects such things as + /// optimization and generation of debug symbols. + public enum BuildConfiguration { + case debug, release + } + + /// Represents the amount of detail in a build log (corresponding to the `-v` + /// and `-vv` options to `swift build`). + public enum BuildLogVerbosity { + case concise, verbose, debug + } + + /// Represents the results of running a build. + public struct BuildResult { + /// Whether the build succeeded or failed. + public var succeeded: Bool + + /// Log output (in this proposal just a long text string; future proposals + /// should consider returning structured build log information). + public var logText: String + + /// The artifacts built from the products in the package. Intermediates + /// such as object files produced from individual targets are not listed. + public var builtArtifacts: [BuiltArtifact] + + /// Represents a single artifact produced during a build. + public struct BuiltArtifact { + /// Full path of the built artifact in the local file system. + public var path: Path + + /// The kind of artifact that was built. + public var kind: Kind + + /// Represents the kind of artifact that was built. The specific file + /// formats may vary from platform to platform — for example, on macOS + /// a dynamic library may in fact be built as a framework. + public enum Kind { + case executable, dynamicLibrary, staticLibrary + } + } + } + + // + // Testing + // + + /// Runs all or a specified subset of the unit tests of the package, after + /// an incremental build if necessary (the same as `swift test` does). + /// + /// Any test failures are reported in the test result. This method throws an + /// error if the input parameters are invalid or in case the test cannot be + /// started. + /// + /// The SwiftPM CLI or any IDE that supports packages may show the progress + /// of the tests as they happen. + /// + /// Future proposals should consider adding ways for the plugin to receive + /// incremental progress during the tests. + public func test( + _ subset: TestSubset, + parameters: TestParameters + ) async throws -> TestResult + + /// Specifies what tests in a package to run. + public enum TestSubset { + /// Represents all tests in the package. + case all + + /// Represents one or more tests filtered by regular expression, with the + /// format . or ./. + /// This is the same as the `--filter` option of `swift test`. + case filtered([String]) + } + + /// Parameters that control how the tests are run. + public struct TestParameters { + /// Whether to collect code coverage information while running the tests. + public var enableCodeCoverage: Bool = false + + /// Future proposals should add more controls over running the tests. + } + + /// Represents the result of running unit tests. + public struct TestResult { + /// Whether the test run succeeded or failed. + public var succeeded: Bool + + /// Results for all the test targets that were run (filtered based on + /// the input subset passed when running the test). + public var testTargets: [TestTarget] + + /// Represents the results of running some or all of the tests in a + /// single test target. + public struct TestTarget { + public var name: String + public var testCases: [TestCase] + + /// Represents the results of running some or all of the tests in + /// a single test case. + public struct TestCase { + public var name: String + public var tests: [Test] + + /// Represents the results of running a single test. + public struct Test { + public var name: String + public var result: Result + public var duration: Double + + /// Represents the result of running a single test. + public enum Result { + case succeeded, skipped, failed + } + } + } + } + + /// Path of a generated `.profdata` file suitable for processing using + /// `llvm-cov`, if `enableCodeCoverage` was set in the test parameters. + public var codeCoverageDataFile: Path? + } + + // + // Accessing Specialized Information + // + + /// Return a directory containing symbol graph files for the given target + /// and options. If the symbol graphs need to be created or updated first, + /// they will be. SwiftPM or an IDE may generate these symbol graph files + /// in any way it sees fit. + public func getSymbolGraph( + for target: Target, + options: SymbolGraphOptions + ) async throws -> SymbolGraphResult + + /// Represents options for symbol graph generation. These options are taken + /// into account when determining whether generated information is already + /// up-to-date. + public struct SymbolGraphOptions { + /// The symbol graph will include symbols at this access level and higher. + public var minimumAccessLevel: AccessLevel = .public + + /// Represents a Swift access level. + public enum AccessLevel { + case `private`, `fileprivate`, `internal`, `public`, `open` + } + + /// Whether to include synthesized members. + public var includeSynthesized: Bool = false + + /// Whether to include symbols marked as SPI. + public var includeSPI: Bool = false + } + + /// Represents the result of symbol graph generation. + public struct SymbolGraphResult { + /// The directory that contains the symbol graph files for the target. + public var directoryPath: Path + } +} +``` + +### Permissions + +Like other plugins, command plugins are run in a sandbox on platforms that support it. By default this sandbox does not allow the plugin to modify the file system (except in special temporary-files paths) and it blocks any network access. + +Some commands, such as source code formatters, might need to modify the file system in order to be useful. Such plugins can specify the permissions they need, and this will: + +* notify the user about the need for the additional permission and provide a way to approve or reject it +* if the user approves, cause the sandbox to be modified in an appropriate manner + +The exact form of the notification and approval will depend on the CLI or IDE that runs the plugin. SwiftPM’s CLI is expected to ask the user for permission using a console prompt (if connected to TTY), and to provide options for approving or rejecting the request when not connected to a TTY. + +Note that this approval needs to be obtained before running the plugin, which is why it is declared in the package manifest. There is currently no provision for a plugin to ask for more permissions while it runs. + +An IDE might present user interface affordances providing the notification and allowing the choice. In order to avoid having to request permission every time the plugin is invoked, some kind of caching of the response could be implemented. + +SwiftPM or IDEs may also provide options to allow users to specify additional writable file system locations for the plugin, but that would not affect the API described in this proposal. + +### Invoking Command Plugins + +In the SwiftPM CLI, command plugins provided by a package or its direct dependencies are available as verbs that can be specified in a `swift` `package` invocation. For example, if the root package defines a command plugin with a `do-something` verb — or if it has a dependency on a package that defines such a plugin — a user can run it using the invocation: + +```shell +❯ swift package do-something +``` + +This will invoke the plugin and only return when it completes. Since no other options were provided, this will pass all regular targets in the package to the plugin ("special" targets such as those that define plugins will be excluded). + +Any parameters passed after the name of the plugin command are passed verbatim to the entry point of the plugin. For example, if a plugin accepts a `--target` option, a subset of the targets to operate on can be passed on the command line that invokes the plugin: + +```shell +❯ swift package do-something --target Foo --target Bar --someOtherFlag +``` + +It is the responsibility of the plugin to interpret any command line arguments passed to it. + +Arguments are currently passed to the plugin exactly as they are written after the command’s verb. A future proposal could allow the plugin to define parameters (using SwiftArgumentParser) that SwiftPM could interpret and that would integrate better with SwiftPM’s own command line arguments. + +As mentioned in the *Permissions* section, command plugins are by default blocked from modifying the files inside the package directory on platforms that support sandboxing. If a command plugin that requires file system writability is invoked, `swift` `package` will ask for approval — this is done using console input if stdin is connected to a TTY, or if not, an error will be reported without invoking the plugin. An `--allow-writing-to-package-directory` option can be used to bypass the request to approve the file system access, which is useful in CI and other automation. + +```shell +❯ swift package --allow-writing-to-package-directory do-something +``` + +Asking for permission from the user helps to prevent unexpected modification of the package by command plugins. + +In IDEs that support Swift packages, command plugins could be provided through context menus or other user interface affordances that allow the commands to be invoked on a package or possibly on a selection of targets in a package. A plugin itself does not need to know, and should not make assumptions about, how or in what context it is being invoked. + +If a future proposal introduces a way of declaring parameters in a manner similar to SwiftArgumentParser, then an IDE could possibly also show a more targeted user interface for those parameters, since the types and optionality will be known. + +### Discovering Command Plugins + +Any plugins defined by a package are included in the `swift` `package` `describe` output for that package. + +Because the command plugins that are available to a package also include those that are defined as plugin products by any package dependencies, it is also useful to have a convenient way of listing all commands that are visible to a particular package. This is provided by the `swift` `package` `plugin` `--list` option, which defaults to text output but also supports a `--json` option. A `--capability` option can be used to filter plugins to only those supporting a particular capability. + +For example: + +```shell +❯ swift package plugin --list --capability=buildTool +``` + +would produce textual output of any plugins with a build tool capacity available to the package, while: + +```shell +❯ swift package plugin --list --capability=command --json +``` + +would produce JSON output of an plugins with a command capacity available to the package. + +## Example 1: Generating Documentation + +Here's a brief example of a hypothetical command plugin that uses `docc` to generate documentation for one or more targets in a package. This example calls back to the plugin host (SwiftPM or an IDE) to generate symbol graphs. + +The package manifest contains the `.plugin()` declaration: + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "MyDocCPlugin", + products: [ + // Declaring the plugin product vends the plugin to clients of the package. + .plugin( + name: "MyDocCPlugin", + targets: ["MyDocCPlugin"] + ), + ], + targets: [ + // This is the target that implements the command plugin. + .plugin( + name: "MyDocCPlugin", + capability: .command( + intent: .documentationGeneration() + ) + ) + ] +) +``` + +The implementation of the package plugin itself: + +```swift +import PackagePlugin +import Foundation + +@main +struct MyDocCPlugin: CommandPlugin { + func performCommand( + context: PluginContext, + arguments: [String] + ) async throws { + // We'll be creating commands that invoke `docc`, so start by locating it. + let doccTool = try context.tool(named: "docc") + + // Construct the path of the directory in which to emit documentation. + let outputDir = context.pluginWorkDirectory.appending("Outputs") + + // Iterate over the targets in the package. + for target in context.package.targets { + // Only consider those kinds of targets that can have source files. + guard let target = target as? SourceModuleTarget else { continue } + + // Find the first DocC catalog in the target, if there is one (a more + // robust example would handle the presence of multiple catalogs). + let doccCatalog = target.sourceFiles.first { $0.path.extension == "docc" } + + // Ask SwiftPM to generate or update symbol graph files for the target. + let symbolGraphInfo = try await packageManager.getSymbolGraph(for: target, + options: .init( + minimumAccessLevel: .public, + includeSynthesized: false, + includeSPI: false)) + + // Invoke `docc` with arguments and the optional catalog. + let doccExec = URL(fileURLWithPath: doccTool.path.string) + var doccArgs = ["convert"] + if let doccCatalog = doccCatalog { + doccArgs += ["\(doccCatalog.path)"] + } + doccArgs += [ + "--fallback-display-name", target.name, + "--fallback-bundle-identifier", target.name, + "--fallback-bundle-version", "0", + "--additional-symbol-graph-dir", "\(symbolGraphInfo.directoryPath)", + "--output-dir", "\(outputDir)", + ] + let process = try Process.run(doccExec, arguments: doccArgs) + process.waitUntilExit() + + // Check whether the subprocess invocation was successful. + if process.terminationReason == .exit && process.terminationStatus == 0 { + print("Generated documentation at \(outputDir).") + } + else { + let problem = "\(process.terminationReason):\(process.terminationStatus)" + Diagnostics.error("docc invocation failed: \(problem)") + } + } + } +} +``` + +In order to use this plugin from another package, a dependency would be used on the package that declares the plugin: + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "MyLibrary", + dependencies: [ + .package(url: "https://url/of/docc/plugin/package", from: "1.0.0"), + ], + targets: [ + .target(name: "MyLibrary") + ] +) +``` + +Note, that, unlike with built tool plugins, there is no `plugins` clause for command plugins — this is because they are applied explicitly by user action and not implicitly when building targets. + +Users can then invoke this command plugin using the `swift` `package` invocation: + +```shell +❯ swift package generate-documentation +``` + +The plugin would usually print the path at which it generated the documentation. + +## Example 2: Formatting Source Code + +This example uses `swift-format` to reformat the code in a package, which requires the plugin to have `.writeToPackageDirectory` permission. + +Note that this package depends on the executable provided by the *swift-format* package. + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "MyFormatterPlugin", + dependencies: [ + .package(url: "https://github.com/apple/swift-format.git", from: "0.50500.0"), + ], + targets: [ + .plugin( + name: "MyFormatterPlugin", + capability: .command( + intent: .sourceCodeFormatting(), + permissions: [ + .writeToPackageDirectory(reason: "This command reformats source files") + ] + ), + dependencies: [ + .product(name: "swift-format", package: "swift-format"), + ] + ) + ] +) +``` + +The implementation of the package plugin itself: + +```swift +import PackagePlugin +import Foundation + +@main +struct MyFormatterPlugin: CommandPlugin { + func performCommand( + context: PluginContext, + arguments: [String] + ) async throws { + // We'll be invoking `swift-format`, so start by locating it. + let swiftFormatTool = try context.tool(named: "swift-format") + + // By convention, use a configuration file in the package directory. + let configFile = context.package.directory.appending(".swift-format.json") + + // Iterate over the targets in the package. + for target in context.package.targets { + // Skip any type of target that doesn't have source files. + // Note: We could choose to instead emit a warning or error here. + guard let target = target as? SourceModuleTarget else { continue } + + // Invoke `swift-format` on the target directory, passing a configuration + // file from the package directory. + let swiftFormatExec = URL(fileURLWithPath: swiftFormatTool.path.string) + let swiftFormatArgs = [ + "--configuration", "\(configFile)", + "--in-place", + "--recursive", + "\(target.directory)" + ] + let process = try Process.run(swiftFormatExec, arguments: swiftFormatArgs) + process.waitUntilExit() + + // Check whether the subprocess invocation was successful. + if process.terminationReason == .exit && process.terminationStatus == 0 { + print("Formatted the source code in \(target.directory).") + } + else { + let problem = "\(process.terminationReason):\(process.terminationStatus)" + Diagnostics.error("swift-format invocation failed: \(problem)") + } + } + } +} +``` + +Users can then invoke this command using the `swift` `package` invocation: + +```shell +❯ swift package format-source-code +``` + +Since `--allow-writing-to-package-directory` is not passed, `swift` `package` will ask the user for permission if its stdin is attached to a TTY, or will fail with an error if not. If `--allow-writing-to-package-directory` were passed, it would just allow the plugin to run (with package directory writability allowed by the sandbox profile) without asking for permission. + +## Example 3: Building Deployment Artifacts + +The final example of a command plugin uses the `PackageManager` service provider’s build functionality to do a release build of a hypothetical product and then to create a distribution archive from it. + +This example shows use of a local plugin target, so no package dependency is needed. This is mostly appropriate for custom commands that are unlikely to be useful outside the package. + +```swift +// swift-tools-version: 5.6 +import PackageDescription + +let package = Package( + name: "MyExecutable", + products: [ + .executable(name: "MyExec", targets: ["MyExec"]) + ], + targets: [ + // This is the hypothetical executable we want to distribute. + .executableTarget( + name: "MyExec" + ), + // This is the plugin that defines a custom command to distribute the executable. + .plugin( + name: "MyDistributionArchiveCreator", + capability: .command( + intent: .custom( + verb: "create-distribution-archive", + description: "Creates a .zip containing release builds of products" + ) + ) + ) + ] +) +``` + +The implementation of the package plugin itself: + +```swift +import PackagePlugin +import Foundation + +@main +struct MyDistributionArchiveCreator: CommandPlugin { + func performCommand( + context: PluginContext, + arguments: [String] + ) async throws { + // Check that we were given the name of a product as the first argument + // and the name of an archive as the second. + guard arguments.count == 2 else { + throw Error("Expected two arguments: product name and archive name") + } + let productName = arguments[0] + let archiveName = arguments[1] + + // Ask the plugin host (SwiftPM or an IDE) to build our product. + let result = try await packageManager.build( + .product(productName), + parameters: .init(configuration: .release, logging: .concise) + ) + + // Check the result. Ideally this would report more details. + guard result.succeeded else { throw Error("couldn't build product") } + + // Get the list of built executables from the build result. + let builtExecutables = result.builtArtifacts.filter{ $0.kind == .executable } + + // Decide on the output path for the archive. + let outputPath = context.pluginWorkDirectory.appending("\(archiveName).zip") + + // Use Foundation to run `zip`. The exact details of using the Foundation + // API aren't relevant; the point is that the built artifacts can be used + // by the script. + let zipTool = try context.tool(named: "zip") + let zipArgs = ["-j", outputPath.string] + builtExecutables.map{ $0.path.string } + let zipToolURL = URL(fileURLWithPath: zipTool.path.string) + let process = try Process.run(zipToolURL, arguments: zipArgs) + process.waitUntilExit() + + // Check whether the subprocess invocation was successful. + if process.terminationReason == .exit && process.terminationStatus == 0 { + print("Created distribution archive at \(outputPath).") + } + else { + let problem = "\(process.terminationReason):\(process.terminationStatus)" + Diagnostics.error("zip invocation failed: \(problem)") + } + } +} +``` + +Users can then invoke this custom command using the `swift` `package` invocation: + +```shell +❯ swift package create-distribution-archive MyExec MyDistributionArchive-1.0 +``` + +This example does not need to ask for permission to write to the package directory since it only writes to the temporary directory provided by the context. A future proposal could allow the plugin to also get permission to write to output directories provided by the user. + +## Security Considerations + +On platforms where SwiftPM supports sandboxing, all plugins are sandboxed in a way that restricts their access to certain system resources. By default, plugins are prevented from writing to the file system (other than to temporary directories and cache directories), and are prevented from accessing the network. + +Custom command plugins that need special permissions — such as writing to the package source directory — can specify a requirement for this permission in the declaration of the plugin. This may cause user interaction to approve the plugin’s request, and if granted, the sandbox is modified to allow this access. + +The form that this request for approval will take depends on whether the plugin is invoked from the SwiftPM CLI or from an IDE that supports Swift Packages. The CLI may implement an option that needs to be passed at the time the plugin is invoked, while an IDE should ideally cache the response in some way that prevents the user from being prompted every time they invoke the plugin. + +On platforms where SwiftPM does not support sandboxing, the user should be notified that invoking the command plugin will result in running code that might perform any action, and should be given the location of the Swift script that implements the plugin so it can be examined by the user. + +## Alternatives Considered + +### Package Manager services + +Most of the alternatives that were considered for this proposal center around what kinds of services the Package Manager provides to plugins. This proposal chooses to expose to the plugin some of the most common actions, such as building and testing, that are also available in SwiftPM CLI commands such as `swift` `build` and `swift` `test`. + +There was an intentional choice to keep the set of options provided as simple as possible in order to keep this proposal bounded. Future proposals should be able to extend this API to provide more options and additional functionality. + +### Declaring prerequisites in the manifest + +An alternative to having the plugin use the `PackageManager` APIs to call back to the host to get specialized information and to perform builds or run tests would be to declare these prerequisites in the manifest of the package that provides the plugin. Under such an approach, SwiftPM would first make sure that the prerequisites are satisfied before invoking the plugin at all. + +The major problem with such an approach is that it’s difficult to express conditional dependencies in the manifest without adding greatly to the complexity of the manifest API, and any approach relying on such up-front prerequisites would necessarily be less flexible than letting the plugin perform package actions when and if it needs them. It would also be contrary to the goal of keeping the package manifest as clear and simple as possible. The implementation of the plugin seems like a much more appropriate place for any non-trivial logic regarding its prerequisites. + +## Future Directions + +### Better support for plugin options + +In this initial proposal, the command plugin is passed all the command line options that the user provided after the command verb in the `swift` `package` invocation. It is then up to the plugin logic to interpret these options. + +Since SwiftPM currently has only a single-layered package dependency graph, it isn't feasible in today's SwiftPM to allow a plugin to define its own dependencies on packages such as SwiftArgumentParser. + +Once this is possible, a future direction might be to have a command plugin use SwiftArgumentParser to declare a supported set of input parameters. This could allow SwiftPM (or possibly an IDE) to present an interface for those plugin options — IDEs, in particular, could construct user interfaces for well-defined options (possibly in the manner of the archaic MPW `Commando` tool). + +Another direction might be for the PackagePlugin API to define its own facility for a plugin to declare externally visible properties. This might include considerations particular to plugins, such as whether or not a particular path property is intended to be writable (requiring permission from the user before the plugin runs). As with SwiftArgumentParser, a natural approach would be to declare such properties on the type that implements the plugin, with their values having been set by the plugin host at the time the plugin is invoked. + +### Additional access to Package Manager services + +The API in the `PackageManager` type that this proposal defines is just a start. The idea is to, over time, offer plugins a variety of functionality and derivable information that they can request and then further process. + +Currently, the only specialized information that a user command plugin can request from SwiftPM is the directory of symbol graph files for a particular target. The intent is to provide a menu of useful information that might or might not require computation in order to provide, and to allow the plugin to request this information from SwiftPM whenever it needs it. + +Extending the `PackageManager` API does need to be done in a way that is possible to implement in various IDEs that support Swift packages but that use a different build system than SwiftPM's. + +### Providing access to build and test progress and structured results + +The initial proposed API for having plugins run builds and tests is fairly minimal. In particular, the build log is returned at the end of the build as a single text string, and the plugin has no way to cancel the build. Future proposals should extend this, ideally to the point at which `swift` `build` and `swift` `test` could themselves be implemented using the same API as for custom commands. + +### Allowing a plugin to report progress + +While a plugin can emit diagnostics using the `Diagnostics` type, there is currently no way for a plugin to report progress while it is running. This would be very useful for long-running plugins, and should be addressed in a future proposal. diff --git a/proposals/0333-with-memory-rebound.md b/proposals/0333-with-memory-rebound.md new file mode 100644 index 0000000000..850529c3a5 --- /dev/null +++ b/proposals/0333-with-memory-rebound.md @@ -0,0 +1,834 @@ +# Expand usability of `withMemoryRebound` + +* Proposal: [SE-0333](0333-with-memory-rebound.md) +* Authors: [Guillaume Lessard](https://github.com/glessard), [Andrew Trick](https://github.com/atrick) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Acceptance](https://forums.swift.org/t/54699) +* Implementation: [apple/swift#39529](https://github.com/apple/swift/pull/39529) +* Bugs: [SR-11082](https://bugs.swift.org/browse/SR-11082), [SR-11087](https://bugs.swift.org/browse/SR-11087) + +## Introduction + +The function `withMemoryRebound(to:capacity:_ body:)` +executes a closure while temporarily binding a range of memory to a different type than the callee is bound to. +We propose to lift some notable limitations of `withMemoryRebound` and enable rebinding to a larger set of types, +as well as rebinding the memory pointed to by raw memory pointers and buffers. + +Swift-evolution threads: [Pitch thread](https://forums.swift.org/t/52500), [Review thread](https://forums.swift.org/t/53799) + +## Motivation + +When using Swift in a systems programming context or using Swift with libraries written in C, +we occasionally need to temporarily access a range of memory as instances of a different type than has been declared +(the pointer's `Pointee` type parameter). +In those cases, `withMemoryRebound` is the tool to reach for, +allowing scoped access to the range of memory as another type. + +As a reminder, the function is declared as follows on the type `UnsafePointer`: +```swift +func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (UnsafePointer) throws -> Result +) rethrows -> Result +``` + +In its current incarnation, this function is more limited than necessary. +It requires that the stride of `Pointee` and `T` be equal, +and that requirement makes many legitimate use cases technically illegal, +even though they could be supported by the compiler. + +We propose to expand and better define the rules by which the function can be used, +including to allow temporarily binding to a type `T` that is a homogeneous aggregate of `Pointee`, +or a type `T` of which `Pointee` is a homogeneous aggregate. +For instance, the tuple `(Int, Int, Int)` is a homogeneous aggregate. + +As an example of rebinding, suppose that a buffer of `Double` consisting of a series of (x,y) pairs is returned from data analysis code written in C. +The next step might be to display it in a preview graph, which needs to read `CGPoint` values. +We need to copy pairs of `Double` values to values of type `CGPoint` (when executing on a 64-bit platform): + +```swift +var count = 0 +let pointer: UnsafePointer = calculation(&count) + +var points = Array(unsafeUninitializedCapacity: count/2) { + buffer, initializedCount in + var p = pointer + for i in buffer.indices where p+1 < pointer+count { + buffer.baseAddress!.advanced(by: i).initialize(to: CGPoint(x: p[0], y: p[1])) + p += 2 + } + initializedCount = pointer.distance(to: p)/2 +} +``` + +We could do better with an improved version of `withMemoryRebound`. +Since `CGPoint` values consist of a pair of `CGFloat` values, +and `CGFloat` values are themselves layout-equivalent with `Double` (when executing on a 64-bit platform): +```swift +var points = Array(unsafeUninitializedCapacity: data.count/2) { + buffer, initializedCount in + pointer.withMemoryRebound(to: CGPoint.self, capacity: buffer.count) { + buffer.baseAddress!.initialize(from: $0, count: buffer.count) + } + initializedCount = buffer.count +} +``` + +Alternately, the data could have been received as bytes from a network request, wrapped in a `Data` instance. +Previously we would have needed to do: +```swift +let data: Data = ... + +var points = Array(unsafeUninitializedCapacity: data.count/MemoryLayout.stride) { + buffer, initializedCount in + data.withUnsafeBytes { data in + var read = 0 + for i in buffer.indices where (read+2*MemoryLayout.stride)<=data.count { + let x = data.load(fromByteOffset: read, as: CGFloat.self) + read += MemoryLayout.stride + let y = data.load(fromByteOffset: read, as: CGFloat.self) + read += MemoryLayout.stride + buffer.baseAddress!.advanced(by: i).initialize(to: CGPoint(x: x, y: y)) + } + initializedCount = read / MemoryLayout.stride + } +} +``` + +In this case having the ability to use `withMemoryRebound` with `UnsafeRawBuffer` improves readability in a similar manner as in the example above: + +```swift +var points = Array(unsafeUninitializedCapacity: data.count/MemoryLayout.stride) { + buffer, initializedCount in + data.withUnsafeBytes { + $0.withMemoryRebound(to: CGPoint.self) { + (_, initializedCount) = buffer.initialize(from: $0) + } + } +} +``` + +## Proposed solution + +`withMemoryRebound` is currently defined for `UnsafePointer`, `UnsafeMutablePointer`, +`UnsafeBufferPointer` and `UnsafeMutableBufferPointer`. +The type to which the memory is bound by the `Pointer` types is called `Pointee`, +while it is `Element` for the `BufferPointer` types. +For simplicity the following discussion calls both `Pointee`. + +In the general case, the runtime performs housekeeping tasks when initializing, deinitializing or updating a value of a type. +Initializing and deinitialization of a type that is or stores a reference type means that type-specific code is executed, +and therefore in general data cannot be accessed as another type. + +`withMemoryRebound` can be used safely with pairs of types `Pointee` and `T` that do _not_ require initialization or deinitialization. +These types do not yet have a formal name in Swift, +but are referred to as "trivial" types in some API documentation. + +In order to safely use `withMemoryRebound`, the current rule +is that the destination type, `T`, must be _layout equivalent_ with `Pointee`. +To this we add that, as an alternative, `T` can be a homogeneous aggregate of `Pointee`, or `Pointee` can be a homogeneous aggregate of `T`. + +Two types A and B are layout equivalent when they are, for example: +- identical types; +- one is a typealias for the other; +- trivial scalar types with the same size and alignment, such as floating-point, integer and pointer types; +- one is a class type, and the other is one of its superclass types, or `AnyObject`; +- optional references whose underlying types are layout equivalent; +- pointer types, such as `UnsafePointer` and `OpaquePointer`; +- optional pointer types, such as `UnsafePointer?` and `UnsafeRawPointer?`; +- one is a struct with a single stored property, the other is the type of its stored property; + +Homogeneous aggregate types (tuples, array storage, and frozen structs) are layout equivalent if they have the same number of layout-equivalent elements. + + +### Instance methods of `UnsafePointer` and `UnsafeMutablePointer` + +We propose to lift the restriction that the strides of `T` and `Pointee` must be equal when calling `withMemoryRebound`. +`T` and `Pointee` must either be layout equivalent (see above,) +or one must be a homogeneous aggregate of the other. +The function declarations remain the same on these two types, +though given the updated rules, +we must clarify the meaning of the `capacity` argument. +`capacity` shall mean the number of strides of elements of the temporary type (`T`) to be temporarily bound. +The documentation will be updated to reflect the changed behaviour. +We will also add parameter labels to the closure type declaration to benefit code completion (a source compatible change.) + +```swift +extension UnsafePointer { + public func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: UnsafePointer) throws -> Result + ) rethrows -> Result +} + +extension UnsafeMutablePointer { + public func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: UnsafeMutablePointer) throws -> Result + ) rethrows -> Result +} +``` + +### Instance methods of `UnsafeRawPointer` and `UnsafeMutableRawPointer` + +We propose adding a `withMemoryRebound` method, which currently does not exist on these types. +Since it operates on raw memory, this version of `withMemoryRebound` places no restriction on the temporary type (`T`). +It is therefore up to the program author to ensure type safety when using these methods. +When applied to memory that is initialized but viewed as raw memory, +the relation between the initialized type and `T` must be valid under the `UnsafePointer.withMemoryRebound` rules. +As in the `UnsafePointer` case, `capacity` means the number of strides of elements of the temporary type (`T`) to be temporarily bound. + +```swift +extension UnsafeRawPointer { + public func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: UnsafePointer) throws -> Result + ) rethrows -> Result +} + +extension UnsafeMutableRawPointer { + public func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: UnsafeMutablePointer) throws -> Result + ) rethrows -> Result +} +``` + +### Instance methods of `UnsafeBufferPointer` and `UnsafeMutableBufferPointer` + +We propose to lift the restriction that the strides of `T` and `Element` must be equal when calling `withMemoryRebound`. +`T` and `Element` must either be layout equivalent (see above,) +or one must be a homogeneous aggregate of the other. +The function declarations remain the same on these two types. +The capacity of the buffer to the temporary type will be calculated using the capacity of the `UnsafeBufferPointer` and the stride of the temporary type. +The documentation will be updated to reflect the changed behaviour. +We will add parameter labels to the closure type declaration to benefit code completion (a source compatible change.) + +```swift +extension UnsafeBufferPointer { + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: UnsafeBufferPointer) throws -> Result + ) rethrows -> Result +} + +extension UnsafeMutableBufferPointer { + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result +} +``` + +### Instance methods of `UnsafeRawBufferPointer` and `UnsafeMutableRawBufferPointer` + +We propose adding a `withMemoryRebound` method, which currently does not exist on these types. +Since it operates on raw memory, this version of `withMemoryRebound` places no restriction on the temporary type (`T`). +It is therefore up to the program author to ensure type safety when using these methods. +When applied to memory that is initialized but viewed as raw memory, +the relation between the initialized type and `T` must be valid under the `UnsafePointer.withMemoryRebound` rules. +The capacity of the buffer to the temporary type will be calculated using the capacity of the `UnsafeRawBufferPointer` and the stride of the temporary type. + +To complete the set, we propose to add an `assumingMemoryBound` function that calculates the capacity of the returned `UnsafeBufferPointer`. + +```swift +extension UnsafeRawBufferPointer { + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: UnsafeBufferPointer) throws -> Result + ) rethrows -> Result + + public func assumingMemoryBound(to type: T.Type) -> UnsafeBufferPointer +} + +extension UnsafeMutableRawBufferPointer { + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result + + public func assumingMemoryBound(to type: T.Type) -> UnsafeMutableBufferPointer +} +``` + + +## Detailed design + +```swift +extension UnsafePointer { + /// Executes the given closure while temporarily binding memory to + /// the specified number of instances of type `T`. + /// + /// Use this method when you have a pointer to memory bound to one type and + /// you need to access that memory as instances of another type. Accessing + /// memory as a type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// The region of memory that starts at this pointer and covers `count` + /// strides of `T` instances must be bound to `Pointee`. + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. Every instance of `Pointee` overlapping with a given + /// instance of `T` should have the same initialization state (i.e. + /// initialized or uninitialized.) Accessing a `T` whose underlying + /// `Pointee` storage is in a mixed initialization state shall be + /// undefined behaviour. + /// + /// The following example temporarily rebinds the memory of a `UInt64` + /// pointer to `Int64`, then accesses a property on the signed integer. + /// + /// let uint64Pointer: UnsafePointer = fetchValue() + /// let isNegative = uint64Pointer.withMemoryRebound(to: Int64.self, + /// capacity: 1) { + /// return $0.pointee < 0 + /// } + /// + /// Because this pointer's memory is no longer bound to its `Pointee` type + /// while the `body` closure executes, do not access memory using the + /// original pointer from within `body`. Instead, use the `body` closure's + /// pointer argument to access the values in memory as instances of type + /// `T`. + /// + /// After executing `body`, this method rebinds memory back to the original + /// `Pointee` type. + /// + /// - Note: Only use this method to rebind the pointer's memory to a type `T` + /// that is layout equivalent with the `Pointee` type, or a type `T` that + /// is an aggregate of `Pointee` instances, or a type `T` such that `Pointee` + /// is an aggregate of `T` instances. As such, the stride of the + /// temporary type (`T`) may be an integer multiple or a whole fraction + /// of `Pointee`'s stride. + /// To bind a region of memory to a type that does not match these + /// requirements, convert the pointer to a raw pointer and use its + /// `withMemoryRebound(to:)` method. + /// If `T` and `Pointee` have different alignments, this pointer + /// must be aligned with the larger of the two alignments. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This pointer must be correctly aligned for `type`. + /// - count: The number of instances of `T` in the re-bound region. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - pointer: The pointer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + @_alwaysEmitIntoClient + public func withMemoryRebound( + to type: T.Type, capacity count: Int, + _ body: (_ pointer: UnsafePointer) throws -> Result + ) rethrows -> Result +} +``` + +```swift +extension UnsafeMutablePointer { + /// Executes the given closure while temporarily binding memory to + /// the specified number of instances of the given type. + /// + /// Use this method when you have a pointer to memory bound to one type and + /// you need to access that memory as instances of another type. Accessing + /// memory as a type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// The region of memory that starts at this pointer and covers `count` + /// strides of `T` instances must be bound to `Pointee`. + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. Every instance of `Pointee` overlapping with a given + /// instance of `T` should have the same initialization state (i.e. + /// initialized or uninitialized.) Accessing a `T` whose underlying + /// `Pointee` storage is in a mixed initialization state shall be + /// undefined behaviour. + /// + /// The following example temporarily rebinds the memory of a `UInt64` + /// pointer to `Int64`, then modifies the signed integer. + /// + /// let uint64Pointer: UnsafeMutablePointer = fetchValue() + /// uint64Pointer.withMemoryRebound(to: Int64.self, capacity: 1) { ptr in + /// ptr.pointee.negate() + /// } + /// + /// Because this pointer's memory is no longer bound to its `Pointee` type + /// while the `body` closure executes, do not access memory using the + /// original pointer from within `body`. Instead, use the `body` closure's + /// pointer argument to access the values in memory as instances of type + /// `T`. + /// + /// After executing `body`, this method rebinds memory back to the original + /// `Pointee` type. + /// + /// - Note: Only use this method to rebind the pointer's memory to a type `T` + /// that is layout equivalent with the `Pointee` type, or a type `T` that + /// is an aggregate of `Pointee` instances, or a type `T` such that `Pointee` + /// is an aggregate of `T` instances. As such, the stride of the + /// temporary type (`T`) may be an integer multiple or a whole fraction + /// of `Pointee`'s stride. + /// To bind a region of memory to a type that does not match these + /// requirements, convert the pointer to a raw pointer and use its + /// `withMemoryRebound(to:)` method. + /// If `T` and `Pointee` have different alignments, this pointer + /// must be aligned with the larger of the two alignments. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This pointer must be correctly aligned for `type`. + /// - count: The number of instances of `T` in the re-bound region. + /// - body: A closure that takes a mutable typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - pointer: The pointer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + @_alwaysEmitIntoClient + public func withMemoryRebound( + to type: T.Type, capacity count: Int, + _ body: (_ pointer: UnsafeMutablePointer) throws -> Result + ) rethrows -> Result +} +``` + +```swift +extension UnsafeBufferPointer { + /// Executes the given closure while temporarily binding the memory referenced + /// by this buffer to the given type. + /// + /// Use this method when you have a buffer of memory bound to one type and + /// you need to access that memory as a buffer of another type. Accessing + /// memory as type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// The number of instances of `T` referenced by the rebound buffer may be + /// different than the number of instances of `Element` referenced by the + /// original buffer. The number of instances of `T` will be calculated + /// at runtime. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. Every instance of `Pointee` overlapping with a given + /// instance of `T` should have the same initialization state (i.e. + /// initialized or uninitialized.) Accessing a `T` whose underlying + /// `Pointee` storage is in a mixed initialization state shall be + /// undefined behaviour. + /// + /// Because this buffer's memory is no longer bound to its `Element` type + /// while the `body` closure executes, do not access memory using the + /// original buffer from within `body`. Instead, use the `body` closure's + /// buffer argument to access the values in memory as instances of type + /// `T`. + /// + /// After executing `body`, this method rebinds memory back to the original + /// `Element` type. + /// + /// - Note: Only use this method to rebind the pointer's memory to a type `T` + /// that is layout equivalent with the `Element` type, or a type `T` that + /// is an aggregate of `Element` instances, or a type `T` such that `Element` + /// is an aggregate of `T` instances. As such, the stride of the + /// temporary type (`T`) may be an integer multiple or a whole fraction + /// of `Element`'s stride. + /// To bind a region of memory to a type that does not match these + /// requirements, convert the pointer to a raw buffer and use its + /// `withMemoryRebound(to:)` method. + /// If `T` and `Element` have different alignments, this buffer's + /// `baseAddress` must be aligned with the larger of the two alignments. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this pointer. + /// This buffer's `baseAddress` must be correctly aligned for `type`. + /// - body: A closure that takes a typed buffer to the + /// same memory as this buffer, only bound to type `T`. The buffer + /// parameter contains a number of complete instances of `T` based + /// on the capacity of the original buffer and the stride of `Element`. + /// The closure's buffer argument is valid only for the duration of the + /// closure's execution. If `body` has a return value, that value + /// is also used as the return value for the `withMemoryRebound(to:_:)` + /// method. + /// - buffer: The buffer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + @_alwaysEmitIntoClient + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: UnsafeBufferPointer) throws -> Result + ) rethrows -> Result +} +``` + +```swift +extension UnsafeMutableBufferPointer { + /// Executes the given closure while temporarily binding the memory referenced + /// by this buffer to the given type. + /// + /// Use this method when you have a buffer of memory bound to one type and + /// you need to access that memory as a buffer of another type. Accessing + /// memory as type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// The number of instances of `T` referenced by the rebound buffer may be + /// different than the number of instances of `Element` referenced by the + /// original buffer. The number of instances of `T` will be calculated + /// at runtime. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. Every instance of `Pointee` overlapping with a given + /// instance of `T` should have the same initialization state (i.e. + /// initialized or uninitialized.) Accessing a `T` whose underlying + /// `Pointee` storage is in a mixed initialization state shall be + /// undefined behaviour. + /// + /// Because this buffer's memory is no longer bound to its `Element` type + /// while the `body` closure executes, do not access memory using the + /// original buffer from within `body`. Instead, use the `body` closure's + /// buffer argument to access the values in memory as instances of type + /// `T`. + /// + /// After executing `body`, this method rebinds memory back to the original + /// `Element` type. + /// + /// - Note: Only use this method to rebind the pointer's memory to a type `T` + /// that is layout equivalent with the `Element` type, or a type `T` that + /// is an aggregate of `Element` instances, or a type `T` such that `Element` + /// is an aggregate of `T` instances. As such, the stride of the + /// temporary type (`T`) may be an integer multiple or a whole fraction + /// of `Element`'s stride. + /// To bind a region of memory to a type that does not match these + /// requirements, convert the pointer to a raw buffer and use its + /// `withMemoryRebound(to:)` method. + /// If `T` and `Element` have different alignments, this buffer's + /// `baseAddress` must be aligned with the larger of the two alignments. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this pointer. + /// This buffer's `baseAddress` must be correctly aligned for `type`. + /// - body: A closure that takes a mutable typed buffer to the + /// same memory as this buffer, only bound to type `T`. The buffer + /// parameter contains a number of complete instances of `T` based + /// on the capacity of the original buffer and the stride of `Element`. + /// The closure's buffer argument is valid only for the duration of the + /// closure's execution. If `body` has a return value, that value + /// is also used as the return value for the `withMemoryRebound(to:_:)` + /// method. + /// - buffer: The buffer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + @_alwaysEmitIntoClient + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result +} +``` + +```swift +extension UnsafeRawPointer { + /// Executes the given closure while temporarily binding memory to + /// the specified number of instances of type `T`. + /// + /// Use this method when you have a pointer to raw memory and you need + /// to access that memory as instances of a given type `T`. Accessing + /// memory as a type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. The memory underlying any individual instance of `T` + /// must have the same initialization state (i.e. initialized or + /// uninitialized.) Accessing a `T` whose underlying memory + /// is in a mixed initialization state shall be undefined behaviour. + /// + /// The following example temporarily rebinds a raw memory pointer + /// to `Int64`, then accesses a property on the signed integer. + /// + /// let pointer: UnsafeRawPointer = fetchValue() + /// let isNegative = pointer.withMemoryRebound(to: Int64.self, + /// capacity: 1) { + /// return $0.pointee < 0 + /// } + /// + /// After executing `body`, this method rebinds memory back to its original + /// binding state. This can be unbound memory, or bound to a different type. + /// + /// - Note: The region of memory starting at this pointer must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Note: The region of memory starting at this pointer may have been + /// bound to a type (the prebound type). If that is the case, then `T` must be + /// layout equivalent with the prebound type, or `T` must be an aggregate of + /// the prebound type, or the the prebound type is an aggregate of `T`. + /// This requirement does not apply if the region of memory + /// has not been bound to any type. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This pointer must be correctly aligned for `type`. + /// - count: The number of instances of `T` in the re-bound region. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - pointer: The pointer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: UnsafePointer) throws -> Result + ) rethrows -> Result +} +``` + +```swift +extension UnsafeMutableRawPointer { + /// Executes the given closure while temporarily binding memory to + /// the specified number of instances of type `T`. + /// + /// Use this method when you have a pointer to raw memory and you need + /// to access that memory as instances of a given type `T`. Accessing + /// memory as a type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. The memory underlying any individual instance of `T` + /// must have the same initialization state (i.e. initialized or + /// uninitialized.) Accessing a `T` whose underlying memory + /// is in a mixed initialization state shall be undefined behaviour. + /// + /// The following example temporarily rebinds a raw memory pointer + /// to `Int64`, then modifies the signed integer. + /// + /// let pointer: UnsafeMutableRawPointer = fetchValue() + /// pointer.withMemoryRebound(to: Int64.self, capacity: 1) { + /// ptr.pointee.negate() + /// } + /// + /// After executing `body`, this method rebinds memory back to its original + /// binding state. This can be unbound memory, or bound to a different type. + /// + /// - Note: The region of memory starting at this pointer must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Note: The region of memory starting at this pointer may have been + /// bound to a type (the prebound type). If that is the case, then `T` must be + /// layout equivalent with the prebound type, or `T` must be an aggregate of + /// the prebound type, or the the prebound type is an aggregate of `T`. + /// This requirement does not apply if the region of memory + /// has not been bound to any type. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This pointer must be correctly aligned for `type`. + /// - count: The number of instances of `T` in the re-bound region. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - pointer: The pointer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: UnsafeMutablePointer) throws -> Result + ) rethrows -> Result +} +``` + +```swift +extension UnsafeRawBufferPointer { + /// Executes the given closure while temporarily binding the buffer to + /// instances of type `T`. + /// + /// Use this method when you have a buffer to raw memory and you need + /// to access that memory as instances of a given type `T`. Accessing + /// memory as a type `T` requires that the memory be bound to that type. + /// A memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. The memory underlying any individual instance of `T` + /// must have the same initialization state (i.e. initialized or + /// uninitialized.) Accessing a `T` whose underlying memory + /// is in a mixed initialization state shall be undefined behaviour. + /// + /// If the byte count of the original buffer is not a multiple of + /// the stride of `T`, then the re-bound buffer is shorter + /// than the original buffer. + /// + /// After executing `body`, this method rebinds memory back to its original + /// binding state. This can be unbound memory, or bound to a different type. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Note: A raw buffer may represent memory that has been bound to a type. + //// (the prebound type). If that is the case, then `T` must be + /// layout equivalent with the prebound type, or `T` must be an aggregate of + /// the prebound type, or the the prebound type is an aggregate of `T`. + /// This requirement does not apply if the region of memory + /// has not been bound to any type. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This buffer's `baseAddress` must be correctly aligned + /// for `type`. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - buffer: The buffer temporarily bound to instances of `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: UnsafeBufferPointer) throws -> Result + ) rethrows -> Result + + /// Returns a typed buffer to the memory referenced by this buffer, + /// assuming that the memory is already bound to the specified type. + /// + /// Use this method when you have a raw buffer to memory that has already + /// been bound to the specified type. The memory starting at this pointer + /// must be bound to the type `T`. Accessing memory through the returned + /// pointer is undefined if the memory has not been bound to `T`. To bind + /// memory to `T`, use `bindMemory(to:capacity:)` instead of this method. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Parameter to: The type `T` that the memory has already been bound to. + /// - Returns: A typed pointer to the same memory as this raw pointer. + public func assumingMemoryBound( + to: T.Type + ) -> UnsafeBufferPointer +} +``` + +```swift +extension UnsafeMutableRawBufferPointer { + /// Executes the given closure while temporarily binding the buffer to + /// instances of type `T`. + /// + /// Use this method when you have a buffer to raw memory and you need + /// to access that memory as instances of a given type `T`. Accessing + /// memory as a type `T` requires that the memory be bound to that type. + /// A memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. The memory underlying any individual instance of `T` + /// must have the same initialization state (i.e. initialized or + /// uninitialized.) Accessing a `T` whose underlying memory + /// is in a mixed initialization state shall be undefined behaviour. + /// + /// If the byte count of the original buffer is not a multiple of + /// the stride of `T`, then the re-bound buffer is shorter + /// than the original buffer. + /// + /// After executing `body`, this method rebinds memory back to its original + /// binding state. This can be unbound memory, or bound to a different type. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Note: A raw buffer may represent memory that has been bound to a type. + //// (the prebound type). If that is the case, then `T` must be + /// layout equivalent with the prebound type, or `T` must be an aggregate of + /// the prebound type, or the the prebound type is an aggregate of `T`. + /// This requirement does not apply if the region of memory + /// has not been bound to any type. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This buffer's `baseAddress` must be correctly aligned + /// for `type`. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - buffer: The buffer temporarily bound to instances of `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result + + /// Returns a typed buffer to the memory referenced by this buffer, + /// assuming that the memory is already bound to the specified type. + /// + /// Use this method when you have a raw buffer to memory that has already + /// been bound to the specified type. The memory starting at this pointer + /// must be bound to the type `T`. Accessing memory through the returned + /// pointer is undefined if the memory has not been bound to `T`. To bind + /// memory to `T`, use `bindMemory(to:capacity:)` instead of this method. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Parameter to: The type `T` that the memory has already been bound to. + /// - Returns: A typed pointer to the same memory as this raw pointer. + public func assumingMemoryBound( + to: T.Type + ) -> UnsafeMutableBufferPointer +} +``` + +## Source compatibility + +This proposal is source-compatible. +Some changes are compatible with existing correct uses of the API, +while others are additive. + + +## Effect on ABI stability + +This proposal consists of ABI-preserving changes and ABI-additive changes. + + +## Effect on API resilience + +The behaviour change for the `withMemoryRebound` is compatible with previous uses, +since restrictions were lifted. +Code that depends on the new semantics may not be compatible with old versions of these functions. +Back-deployment of new binaries will be supported by making the updated versions `@_alwaysEmitIntoClient`. +Compatibility of old binaries with a new standard library will be supported by ensuring that a compatible entry point remains. + + +## Alternatives considered + +One alternative is to implement none of this change, and leave `withMemoryRebound` as is. +The usability problems of `withMemoryRebound` would remain. + +Another alternative is to leave the type layout restrictions as they are for the typed `Pointer` and `BufferPointer` types, +but add the `withMemoryRebound` functions to the `RawPointer` and `RawBufferPointer` variants. +In that case, the stride restriction would be no more than a speedbump, +because it would be straightforward to bypass it by transiting through the appropriate `Raw` variant. diff --git a/proposals/0334-pointer-usability-improvements.md b/proposals/0334-pointer-usability-improvements.md new file mode 100644 index 0000000000..ffae2587ad --- /dev/null +++ b/proposals/0334-pointer-usability-improvements.md @@ -0,0 +1,494 @@ +# Pointer API Usability Improvements + +* Proposal: [SE-0334](0334-pointer-usability-improvements.md) +* Authors: [Guillaume Lessard](https://github.com/glessard), [Andrew Trick](https://github.com/atrick) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Decision notes: [Acceptance](https://forums.swift.org/t/54700) +* Implementation: [Draft pull request][draft-pr] +* Bugs: [rdar://64342031](rdar://64342031), [SR-11156](https://bugs.swift.org/browse/SR-11156) ([rdar://53272880](rdar://53272880)), [rdar://22541346](rdar://22541346) + +[draft-pr]: https://github.com/apple/swift/pull/39639 +[pitch-thread]: https://forums.swift.org/t/52736 + +## Introduction + +This proposal introduces some quality-of-life improvements for `UnsafePointer` and its `Mutable` and `Raw` variants. + +1. Add an API to obtain an `UnsafeRawPointer` instance that is advanced to a given alignment from its starting point. +2. Add an API to obtain a pointer to a stored property of an aggregate `T`, given an `UnsafePointer`. +3. Add the ability to compare pointers of any two types. + +Swift-evolution threads: [Discussion][pitch-thread], [Review](https://forums.swift.org/t/53800) + +## Motivation + +The everyday use of `UnsafePointer` and its variants comes with many difficulties unrelated to the unsafeness of the type. +We can improve the ergonomics of these types without hiding the unsafeness. + +For example, if one needs to advance a pointer to a given alignment, +there is no need to force the programmer to derive the proper calculation +(or consult a textbook, or copy an answer from stack overflow.) +An API that provides this utility would not take away from the fact that the type is called "unsafe". + +Similarly, it is rather difficult to pass a pointer to a property of a struct to (e.g.) a C function. +In such cases, the poor ergonomics lead to code that is less safe than it should be. + +Finally, when dealing with pointers of different types, +we can often get in situations where Swift's type system gets in the way. +Regardless of their type, pointers represent one unique storage location in memory. +As such, casting the type of a pointer in order to be able to compare it to another is not a useful exercise. + + +## Proposed solution + +#### Ability to obtain a pointer properly aligned to store a given type + +When using pointers into untyped (raw) memory, +it is often desirable to obtain another pointer that is advanced to a given alignment, +rather than advanced by a particular offset. +The current API provides no help in performing this task, +even though the calculation isn't entirely obvious. +The programmer should not need to derive the proper calculation, or to consult a textbook. + +For example, consider implementing a complex data structure whose nodes include atomic pointers to other nodes in the graph. +In order to avoid two allocations per node, we allocate a range of raw memory and manually bind subranges of the allocation. +Our example node allocates space for one atomic pointer value and one value of type `T`: +```swift +import SwiftAtomics + +struct Node: RawRepresentable, AtomicValue, AtomicOptionalWrappable { + typealias AtomicRepresentation = AtomicRawRepresentableStorage + typealias AtomicOptionalRepresentation = + AtomicOptionalRawRepresentableStorage + typealias NodeStorage = (AtomicOptionalRepresentation, T) + + let rawValue: UnsafeMutableRawPointer + + init(_ element: T) { + rawValue = .allocate(byteCount: MemoryLayout.size, + alignment: MemoryLayout.alignment) + + // bind and initialize atomic storage + rawValue.initializeMemory(as: AtomicOptionalRepresentation.self, + repeating: AtomicOptionalRepresentation(nil), + count: 1) + // bind and initialize payload storage + let tMask = MemoryLayout.alignment - 1 + let tOffset = (MemoryLayout.size + tMask) & ~tMask + let t = rawValue.advanced(by: tOffset) + .initializeMemory(as: T.self, repeating: element, count: 1) + } +} +``` + +The calculation of `tOffset` above is overly complex. +Calculating the offset between the start of the data structure to the field of type `T` should be straightforward! + +We propose to add a function to help perform this operation on raw pointer types: +```swift +extension UnsafeRawPointer { + public func alignedUp(for: T.Type) -> Self +} +``` + +This function will round the current pointer up to the next address properly aligned to access an instance of `T`. +When applied to a `self` already aligned for `T`, `UnsafeRawPointer.alignedUp(for:)` will return `self`. + +The new function would make identifying the storage location of `T` much more straightforward than in the example above: +```swift + init(_ element: T) { + rawValue = .allocate(byteCount: MemoryLayout.size, + alignment: MemoryLayout.alignment) + + // bind and initialize atomic storage + rawValue.initializeMemory(as: AtomicOptionalRepresentation.self, + repeating: AtomicOptionalRepresentation(nil), + count: 1) + // bind and initialize payload storage + rawValue.advanced(by: MemoryLayout.size) + .alignedUp(for: T.self) + .initializeMemory(as: T.self, repeating: element, count: 1) + } +``` + +Along with `alignedUp(for:)`, we also propose to add `alignedDown(for:)`, +as well as a pair of corresponding functions that take an integer argument: +`alignedUp(toMultipleOf:)` and `alignedDown(toMultipleOf:)`. + + +#### Ability to obtain a pointer to a member of an aggregate value + +When using a pointer to a struct with multiple stored properties, +it isn't obvious how to obtain pointers to more than one of the stored properties. +For example, consider using the pthreads library, a major C API. +The pthreads library uses the return value to indicate error conditions, +and modifies values through pointers it receives as parameters. +It has many APIs with multiple pointer arguments. +One would query a thread's scheduling parameters using `pthread_getschedparam`, +which has the following prototype: +```C +int pthread_getschedparam(pthread_t tid, int *policy, struct sched_param *param); +``` + +A swift user, concerned with keeping related data packaged together, +might have elected to define a struct thusly: +```swift +struct ThreadSchedulingParameters { + var policy: Int + var parameters: sched_param + var priority: Int { parameters.sched_priority } +} +``` + +Updating a `ThreadSchedulingParameters` instance using the above C function is not obvious: +```swift +var scheduling = ThreadSchedulingParameters() +var tid = pthread_create(...) +var e = withUnsafeMutableBytes(of: &scheduling) { bytes in + let o1 = MemoryLayout.offset(of: \.policy)! + let policy_p = bytes.baseAddress!.advanced(by: o1).assumingMemoryBound(to: Int32.self) + let o2 = MemoryLayout.offset(of: \.parameters)! + let params_p = bytes.baseAddress!.advanced(by: o2).assumingMemoryBound(to: sched_param.self) + return pthread_getschedparam(thread, policy_p, params_p) +} +``` + +We must first reach for the non-obvious `withUnsafeMutableBytes` rather than for `withUnsafePointer`. +In so doing, we suppress statically-known type information, +only to immediately assert the type using `assumingMemoryBound`. +We can use `KeyPath` to do better. +We shall add a new function to `UnsafePointer` and `UnsafeMutablePointer` to perform this task: +```swift +extension UnsafeMutablePointer { + func pointer(to property: WritableKeyPath) -> UnsafeMutablePointer? +} +``` + +The return value of this function must be optional, +because whether any given `KeyPath` represents a stored or computed property is not represented in its type. +If the `KeyPath` represents a computed property, +there is no corresponding pointer, and we must return `nil`. + +With this new function, a correct call to `pthread_getschedparam` becomes the much simpler: +```swift +var e = withUnsafeMutablePointer(to: &scheduling) { + pthread_getschedparam(thread, + $0.pointer(to: \.policy)!, + $0.pointer(to: \.parameters)!) +} +``` + + +#### Allow comparisons of pointers of any type + +Pointers are effectively an index into the fundamental collection that is the computer's memory. +Regardless of their type, they represent a unique storage location in memory. +As such, having to cast the type of a pointer in order to be able to compare it to another is not a useful exercise. + +It's very common to end up with a combination of `Mutable` and non-`Mutable` pointers into the same buffer, +and the programmer needs to write conversions that satisfy the compiler but have no real effect in the generated code. + +To remedy this, we propose to add the following static functions, scoped to the existing `_Pointer` protocol: +```swift +extension _Pointer { + public static func == (lhs: Self, rhs: Other) -> Bool + public static func != (lhs: Self, rhs: Other) -> Bool + + public static func < (lhs: Self, rhs: Other) -> Bool + public static func <= (lhs: Self, rhs: Other) -> Bool + public static func > (lhs: Self, rhs: Other) -> Bool + public static func >= (lhs: Self, rhs: Other) -> Bool +} +``` + +Note that it is always possible to enclose both pointers in a conversion to `UnsafeRawPointer`. +This addition simply removes the necessity to insert conversions that are _always_ legal. + + +## Detailed design + +#### API to obtain a pointer properly aligned to store a given type + +```swift +extension UnsafeRawPointer { + /// Obtain the next pointer properly aligned to store a value of type `T`. + /// + /// If `self` is properly aligned for accessing `T`, + /// this function returns `self`. + /// + /// - Parameters: + /// - type: the type to be stored at the returned address. + /// - Returns: a pointer properly aligned to store a value of type `T`. + public func alignedUp(for type: T.Type) -> UnsafeRawPointer + + /// Obtain the preceding pointer properly aligned to store a value of type `T`. + /// + /// If `self` is properly aligned for accessing `T`, + /// this function returns `self`. + /// + /// - Parameters: + /// - type: the type to be stored at the returned address. + /// - Returns: a pointer properly aligned to store a value of type `T`. + public func alignedDown(for type: T.Type) -> UnsafeRawPointer + + /// Obtain the next pointer whose bit pattern is a multiple of `alignment`. + /// + /// If the bit pattern of `self` is a multiple of `alignment`, + /// this function returns `self`. + /// + /// - Parameters: + /// - alignment: the alignment of the returned pointer, in bytes. + /// `alignment` must be a whole power of 2. + /// - Returns: a pointer aligned to `alignment`. + public func alignedUp(toMultipleOf alignment: Int) -> UnsafeRawPointer + + /// Obtain the preceding pointer whose bit pattern is a multiple of `alignment`. + /// + /// If the bit pattern of `self` is a multiple of `alignment`, + /// this function returns `self`. + /// + /// - Parameters: + /// - alignment: the alignment of the returned pointer, in bytes. + /// `alignment` must be a whole power of 2. + /// - Returns: a pointer aligned to `alignment`. + public func alignedDown(toMultipleOf alignment: Int) -> UnsafeRawPointer +} +``` + +```swift +extension UnsafeMutableRawPointer { + /// Obtain the next pointer properly aligned to store a value of type `T`. + /// + /// If `self` is properly aligned for accessing `T`, + /// this function returns `self`. + /// + /// - Parameters: + /// - type: the type to be stored at the returned address. + /// - Returns: a pointer properly aligned to store a value of type `T`. + public func alignedUp(for type: T.Type) -> UnsafeMutableRawPointer + + /// Obtain the preceding pointer properly aligned to store a value of type `T`. + /// + /// If `self` is properly aligned for accessing `T`, + /// this function returns `self`. + /// + /// - Parameters: + /// - type: the type to be stored at the returned address. + /// - Returns: a pointer properly aligned to store a value of type `T`. + public func alignedDown(for type: T.Type) -> UnsafeMutableRawPointer + + /// Obtain the next pointer whose bit pattern is a multiple of `alignment`. + /// + /// If the bit pattern of `self` is a multiple of `alignment`, + /// this function returns `self`. + /// + /// - Parameters: + /// - alignment: the alignment of the returned pointer, in bytes. + /// `alignment` must be a whole power of 2. + /// - Returns: a pointer aligned to `alignment`. + public func alignedUp(toMultipleOf alignment: Int) -> UnsafeMutableRawPointer + + /// Obtain the preceding pointer whose bit pattern is a multiple of `alignment`. + /// + /// If the bit pattern of `self` is a multiple of `alignment`, + /// this function returns `self`. + /// + /// - Parameters: + /// - alignment: the alignment of the returned pointer, in bytes. + /// `alignment` must be a whole power of 2. + /// - Returns: a pointer aligned to `alignment`. + public func alignedDown(toMultipleOf alignment: Int) -> UnsafeMutableRawPointer +} +``` + +#### API to obtain a pointer to a member of an aggregate value + +```swift +extension UnsafePointer { + /// Obtain a pointer to the stored property referred to by a key path. + /// + /// If the key path represents a computed property, + /// this function will return `nil`. + /// + /// - Parameter property: A `KeyPath` whose `Root` is `Pointee`. + /// - Returns: A pointer to the stored property represented + /// by the key path, or `nil`. + public func pointer( + to property: KeyPath + ) -> UnsafePointer? +} + +extension UnsafeMutablePointer { + /// Obtain a pointer to the stored property referred to by a key path. + /// + /// If the key path represents a computed property, + /// this function will return `nil`. + /// + /// - Parameter property: A `KeyPath` whose `Root` is `Pointee`. + /// - Returns: A pointer to the stored property represented + /// by the key path, or `nil`. + public func pointer( + to property: KeyPath + ) -> UnsafePointer? + + /// Obtain a mutable pointer to the stored property referred to by a key path. + /// + /// If the key path represents a computed property, + /// this function will return `nil`. + /// + /// - Parameter property: A `WritableKeyPath` whose `Root` is `Pointee`. + /// - Returns: A mutable pointer to the stored property represented + /// by the key path, or `nil`. + public func pointer( + to property: WritableKeyPath + ) -> UnsafeMutablePointer? +} +``` + + +#### Allow comparisons of pointers of any type + +```swift + /// Returns a Boolean value indicating whether two pointers represent + /// the same memory address. + /// + /// - Parameters: + /// - lhs: A pointer. + /// - rhs: Another pointer. + /// - Returns: `true` if `lhs` and `rhs` reference the same memory address; + /// otherwise, `false`. + public static func == (lhs: Self, rhs: Other) -> Bool + + /// Returns a Boolean value indicating whether two pointers represent + /// different memory addresses. + /// + /// - Parameters: + /// - lhs: A pointer. + /// - rhs: Another pointer. + /// - Returns: `true` if `lhs` and `rhs` reference different memory addresses; + /// otherwise, `false`. + public static func != (lhs: Self, rhs: Other) -> Bool + + /// Returns a Boolean value indicating whether the first pointer references + /// a memory location earlier than the second pointer references. + /// + /// - Parameters: + /// - lhs: A pointer. + /// - rhs: Another pointer. + /// - Returns: `true` if `lhs` references a memory address + /// earlier than `rhs`; otherwise, `false`. + public static func < (lhs: Self, rhs: Other) -> Bool + + /// Returns a Boolean value indicating whether the first pointer references + /// a memory location earlier than or same as the second pointer references. + /// + /// - Parameters: + /// - lhs: A pointer. + /// - rhs: Another pointer. + /// - Returns: `true` if `lhs` references a memory address + /// earlier than or the same as `rhs`; otherwise, `false`. + public static func <= (lhs: Self, rhs: Other) -> Bool + + /// Returns a Boolean value indicating whether the first pointer references + /// a memory location later than the second pointer references. + /// + /// - Parameters: + /// - lhs: A pointer. + /// - rhs: Another pointer. + /// - Returns: `true` if `lhs` references a memory address + /// later than `rhs`; otherwise, `false`. + public static func > (lhs: Self, rhs: Other) -> Bool + + /// Returns a Boolean value indicating whether the first pointer references + /// a memory location later than or same as the second pointer references. + /// + /// - Parameters: + /// - lhs: A pointer. + /// - rhs: Another pointer. + /// - Returns: `true` if `lhs` references a memory address + /// later than or the same as `rhs`; otherwise, `false`. + public static func >= (lhs: Self, rhs: Other) -> Bool +} +``` + + +## Source compatibility + +All of the proposed changes are additive, and do not affect existing code. + + +## Effect on ABI stability + +We can implement these changes in an ABI-neutral manner. + + +## Effect on API resilience + +The proposed additions will be public API, +and will all be marked `@_alwaysEmitIntoClient` to support back-deployability. + + +## Alternatives considered + +#### API to obtain a pointer properly aligned to store a given type + +Instead of the proposed function that takes a type argument, +we could only add an API that simply takes an integer, +and rounds the value of the pointer to a multiple of that number. +We believe that having a type parameter is the correct default. +Since it is not currently possible to define a type whose alignment is greater than 16, +we also include versions that take an integer argument. + +The name of the function could simply be `advanced(toAlignmentOf: T.type)`. +This pairs well with the existing pointer advancement functions, +but implies that it the returned value is always different from `self`. + +There is a pre-existing internal API to obtain pointers aligned with a type's alignment, +consisting of static members of `MemoryLayout` whose names start with `_roundingUp`. +We believe that the functionality is a more natural fit as methods of `Unsafe[Mutable]RawPointer`. +The name `roundedUp` was the name originally pitched for this functionality, +but it is a strange fit for an operation that is entirely about integer values. + +Ultimately we are proposing `alignedUp(for:)` and `alignedDown(for:)`. +These names have the important property of not being misleading. +There is no clear best choice for the preposition to be used as an argument label, +though we note that the argument label `for` meshes well with the parameter name `type`, +which is visible in documentation, including auto-completion. + + +#### API to obtain a pointer to a member of an aggregate value + +We originally proposed to use a subscript instead of a function to provide this functionality: +```swift +subscript(property: KeyPath) -> UnsafePointer? { get } +``` +It was pointed out that a subscript generally implies direct access to the property, +whereaes this one would only provide access to a pointer to the property. +Furthermore, since there is no need for a setter (lvalue), +the functionality can be provided just as well with a function. + +It might be possible to use the `@dynamicMemberLookup` functionality to make the subscript approach even more elegant. +This seemed to imply even more strongly a direct access to the property, +as well as being deemed "too magical". + + +#### Allow comparisons of pointers of any type + +Compiler performance is a concern, and operator overloads have been the cause of performance issues in the past. +Preliminary compiler performance testing [suggests][performance-test] that this addition does not appreciably affect performance. + +[performance-test]: https://github.com/apple/swift/pull/39635#issuecomment-966767929 + + +#### Add `unchecked` argument label to `UnsafePointer`'s integer subscript + +The original pitch for this proposal included the addition of an argument label ("unchecked") to `Unsafe[Mutable]Pointer`'s integer subscript. +The intention for this change was to begin the process of better marking the use of unsafe API at the point of use. +We are deferring this portion of the pitch because it has source compatibility implications, +and will require a staged plan for deprecation and eventual removal. + + +## Acknowledgements + +Thanks to Kyle Macomber and the Swift Standard Library team for valuable feedback. diff --git a/proposals/0335-existential-any.md b/proposals/0335-existential-any.md new file mode 100644 index 0000000000..eaf480380e --- /dev/null +++ b/proposals/0335-existential-any.md @@ -0,0 +1,303 @@ +# Introduce existential `any` + +* Proposal: [SE-0335](0335-existential-any.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.6)** +* Upcoming Feature Flag: `ExistentialAny` (implemented in Swift 5.8) +* Implementation: [apple/swift#40282](https://github.com/apple/swift/pull/40282) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0335-introduce-existential-any/54504) + +## Contents + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [Grammar of explicit existential types](#grammar-of-explicit-existential-types) + - [Semantics of explicit existential types](#semantics-of-explicit-existential-types) + - [`Any` and `AnyObject`](#any-and-anyobject) + - [Metatypes](#metatypes) + - [Type aliases and associated types](#type-aliases-and-associated-types) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Alternatives considered](#alternatives-considered) + - [Rename `Any` and `AnyObject`](#rename-any-and-anyobject) + - [Use `Any

` instead of `any P`](#use-anyp-instead-of-any-p) + - [Future Directions](#future-directions) + - [Extending existential types](#extending-existential-types) + - [Re-purposing the plain protocol name](#re-purposing-the-plain-protocol-name) + - [Revisions](#revisions) + - [Changes from the pitch discussion](#changes-from-the-pitch-discussion) + - [Acknowledgments](#acknowledgments) + +## Introduction + +Existential types in Swift have an extremely lightweight spelling: a plain protocol name in type context means an existential type. Over the years, this has risen to the level of **active harm** by causing confusion, leading programmers down the wrong path that often requires them to re-write code once they hit a fundamental [limitation of value-level abstraction](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--limits-of-existentials). This proposal makes the impact of existential types explicit in the language by annotating such types with `any`. + +Swift evolution discussion thread: [[Pitch] Introduce existential `any`](https://forums.swift.org/t/pitch-introduce-existential-any/53520). + +## Motivation + +Existential types in Swift have significant limitations and performance implications. Some of their limitations are missing language features, but many are fundamental to their type-erasing semantics. For example, given a protocol with associated type requirements, the existential type cannot conform to the protocol itself without a manual conformance implementation, because there is not an obvious concrete associated type that works for any value conforming to the protocol, as shown by the following example: + +```swift +protocol P { + associatedtype A + func test(a: A) +} + +func generic(p: ConcreteP, value: ConcreteP.A) { + p.test(a: value) +} + +func useExistential(p: P) { + generic(p: p, value: ???) // what type of value would P.A be?? +} +``` + +Existential types are also significantly more expensive than using concrete types. Because they can store any value whose type conforms to the protocol, and the type of value stored can change dynamically, existential types require dynamic memory unless the value is small enough to fit within an inline 3-word buffer. In addition to heap allocation and reference counting, code using existential types incurs pointer indirection and dynamic method dispatch that cannot be optimized away. + +Despite these significant and often undesirable implications, existential types have a minimal spelling. Syntactically, the cost of using one is hidden, and the similar spelling to generic constraints has caused many programmers to confuse existential types with generics. In reality, the need for the dynamism they provided is relatively rare compared to the need for generics, but the language makes existential types too easy to reach for, especially by mistake. The cost of using existential types should not be hidden, and programmers should explicitly opt into these semantics. + +## Proposed solution + +I propose to make existential types syntactically explicit in the language using the `any` keyword. This proposal introduces the new syntax in the Swift 5 language mode, and this syntax should be required for existential types under a future language mode. + +In Swift 5, anywhere that an existential type can be used today, the `any` keyword can be used to explicitly denote an existential type: + +```swift +// Swift 5 mode + +protocol P {} +protocol Q {} +struct S: P, Q {} + +let p1: P = S() // 'P' in this context is an existential type +let p2: any P = S() // 'any P' is an explicit existential type + +let pq1: P & Q = S() // 'P & Q' in this context is an existential type +let pq2: any P & Q = S() // 'any P & Q' is an explicit existential type +``` + +In a future language mode, existential types are required to be explicitly spelled with `any`: + +```swift +// Future language mode + +protocol P {} +protocol Q {} +struct S: P, Q {} + +let p1: P = S() // error +let p2: any P = S() // okay + +let pq1: P & Q = S() // error +let pq2: any P & Q = S() // okay +``` + +This behavior can be enabled in earlier language modes with the [upcoming feature flag](0362-piecemeal-future-features.md) `ExistentialAny`. + +## Detailed design + +### Grammar of explicit existential types + +This proposal adds the following production rules to the grammar of types: + +``` +type -> existential-type + +existential-type -> 'any' type +``` + +### Semantics of explicit existential types + +The semantics of `any` types are the same as existential types today. Explicit `any` can only be applied to protocols and protocol compositions, or metatypes thereof; `any` cannot be applied to nominal types, structural types, type parameters, and protocol metatypes: + +```swift +struct S {} + +let s: any S = S() // error: 'any' has no effect on concrete type 'S' + +func generic(t: T) { + let x: any T = t // error: 'any' has no effect on type parameter 'T' +} + +let f: any ((Int) -> Void) = generic // error: 'any' has no effect on concrete type '(Int) -> Void' +``` + +#### `Any` and `AnyObject` + +`any` is unnecessary for `Any` and `AnyObject` (unless part of a protocol composition): + +```swift +struct S {} +class C {} + +let value: any Any = S() +let values: [any Any] = [] +let object: any AnyObject = C() + +protocol P {} +extension C: P {} + +let pObject: any AnyObject & P = C() // okay +``` + +> **Rationale**: `any Any` and `any AnyObject` are redundant. `Any` and `AnyObject` are already special types in the language, and their existence isn’t nearly as harmful as existential types for regular protocols because the type-erasing semantics is already explicit in the name. + +#### Metatypes + +The existential metatype, i.e. `P.Type`, becomes `any P.Type`. The protocol metatype, i.e. `P.Protocol`, becomes `(any P).Type`. The protocol metatype value `P.self` becomes `(any P).self`: + +```swift +protocol P {} +struct S: P {} + +let existentialMetatype: any P.Type = S.self + +protocol Q {} +extension S: Q {} + +let compositionMetatype: any (P & Q).Type = S.self + +let protocolMetatype: (any P).Type = (any P).self +``` + +> **Rationale**: The existential metatype is spelled `any P.Type` because it's an existential type that is a generalization over metatypes. The protocol metatype is the singleton metatype of the existential type `any P` itself, which is naturally spelled `(any P).Type`. + +Under this model, the `any` keyword conceptually acts like an existential quantifier `∃ T`. Formally, `any P.Type` means `∃ T:P . T.Type`, i.e. for some concrete type `T` conforming to `P`, this is the metatype of that concrete type.`(any P).Type` is formally `(∃ T:P . T).Type`, i.e. the metatype of the existential type itself. + +The distinction between `any P.Type` and `(any P).Type` is syntactically very subtle. However, `(any P).Type` is rarely useful in practice, and it's helpful to explain why, given a generic context where a type parameter `T` is substituted with an existential type, `T.Type` is the singleton protocol metatype. + +##### Metatypes for `Any` and `AnyObject` + +Like their base types, `Any.Type` and `AnyObject.Type` remain valid existential metatypes; writing `any` on these metatypes in unnecessary. The protocol metatypes for `Any` and `AnyObject` are spelled `(any Any).Type` and `(any AnyObject).Type`, respectively. + +#### Type aliases and associated types + +Like plain protocol names, a type alias to a protocol `P` can be used as both a generic constraint and an existential type. Because `any` is explicitly an existential type, a type alias to `any P` can only be used as an existential type, it cannot be used as a generic conformance constraint, and `any` does not need to be written at the use-site: + +```swift +protocol P {} +typealias AnotherP = P +typealias AnyP = any P + +struct S: P {} + +let p2: any AnotherP = S() +let p1: AnyP = S() + +func generic(value: T) { ... } +func generic(value: T) { ... } // error +``` + +Once the `any` spelling is required under a future language mode, a type alias to a plain protocol name is not a valid type witness for an associated type requirement; existential type witnesses must be explicit in the `typealias` with `any`: + +```swift +protocol P {} + +protocol Requirements { + associatedtype A +} + +struct S1: Requirements { + typealias A = P // error: associated type requirement cannot be satisfied with a protocol +} + +struct S2: Requirements { + typealias A = any P // okay +} +``` + +## Source compatibility + +Enforcing that existential types use the `any` keyword will require a source change. To ease the migration, I propose to start allowing existential types to be spelled with `any` with the Swift 5.6 compiler, and require existential types to be spelled with `any` under a future language mode. The old existential type syntax will continue to be supported under the Swift 5 language mode, and the transition to the new syntax is mechanical, so it can be performed automatically by a migrator. + +[SE-0309 Unlock existentials for all protocols](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md) enables more code to be written using existential types. To minimize the amount of new code written that will become invalid under `ExistentialAny`, I propose requiring `any` immediately for protocols with `Self` and associated type requirements. This introduces an inconsistency for protocols under the Swift 5 language mode, but this inconsistency already exists today (because you cannot use certain protocols as existential types at all), and the syntax difference serves two purposes: + +1. It saves programmers time in the long run by preventing them from writing new code that will become invalid later. +2. It communicates the existence of `any` and encourages programmers to start using it for other existential types before adopting `ExistentialAny`. + +### Transitioning to `any` + +The new `any` syntax will be staged in over several major Swift releases. In the release where `any` is introduced, the compiler will not emit diagnostics for the lack of `any` on existential types, save for the aforementioned cases. After `any` is introduced, warnings will be added to guide programmers toward the new syntax. Finally, a missing `any` will become an unconditional error, or [plain protocol names may be repurposed](#re-purposing-the-plain-protocol-name) — in Swift 6 or a later language mode. + +## Effect on ABI stability + +None. + +## Effect on API resilience + +None. + +## Alternatives considered + +### Rename `Any` and `AnyObject` + +Instead of leaving `Any` and `AnyObject` in their existing spelling, an alternative is to spell these types as `any Value` and `any Object`, respectively. Though this is more consistent with the rest of the proposal, this change would have an even bigger source compatibility impact. Given that `Any` and `AnyObject` aren’t as harmful as other existential types, changing the spelling isn’t worth the churn. + +### Use `Any

` instead of `any P` + +A common suggestion is to spell existential types with angle brackets on `Any`, e.g. `Any`. However, an important aspect of the proposed design is that `any` has symmetry with `some`, where both keywords can be applied to protocol constraints. This symmetry is important for helping programmers understand and remember the syntax, and for future extensions of the `some` and `any` syntax. Opaque types and existential types would both greatly benefit from being able to specify constraints on associated types. This could naturally be done in angle brackets, e.g. `some Sequence` and `any Sequence`, or `some Sequence<.Element == Int>` and `any Sequence<.Element == Int>`. + +Using the same syntax between opaque types and exsitential types also makes it very easy to replace `any` with `some`, and it is indeed the case that many uses of existential types today could be replaced with opaque types instead. + +Finally, the `Any

` syntax is misleading because it appears that `Any` is a generic type, which is confusing to the mental model for 2 reasons: + +1. A generic type is something programmers can implement themselves. In reality, existential types are a built-in language feature that would be _very_ difficult to replicate with regular Swift code. +2. This syntax creates the misconception that the underlying concrete type is a generic argument to `Any` that is preserved statically in the existential type. The `P` in `Any

` looks like an implicit type parameter with a conformance requirement, but it's not; the underlying type conforming to `P` is erased at compile-time. + +## Future Directions + +### Extending existential types + +This proposal provides an obvious syntax for extending existential types in order to manually implement protocol conformances: + +```swift +extension any Equatable: Equatable { ... } +``` + +### Re-purposing the plain protocol name + +In other places in the language, a plain protocol name is already sugar for a type parameter conforming to the protocol. Consider a normal protocol extension: + +```swift +extension Collection { ... } +``` + +This extension is a form of universal quantification; it extends all types that conform to `Collection`. This extension introduces a generic context with a type parameter ``, which means the above syntax is effectively sugar for a parameterized extension: + +```swift +extension Self where Self: Collection { ... } +``` + +Changing the syntax of existential types creates an opportunity to expand upon this sugar. If existential types are spelled explicitly with `any`, a plain protocol name could always mean sugar for a type parameter on the enclosing context with a conformance requirement to the protocol. For example, consider the declaration of `append(contentsOf:)` from the standard library: + +```swift +extension Array { + mutating func append(contentsOf newElements: S) where S.Element == Element +} +``` + +Combined with a syntax for constraining associated types in angle brackets, such as in [[Pitch] Light-weight same-type constraint syntax](https://forums.swift.org/t/pitch-light-weight-same-type-constraint-syntax/52889), the above declaration could be simplified to: + +```swift +extension Array { + mutating func append(contentsOf newElements: Sequence) +} +``` + +This sugar eliminates a lot of noise in cases where a type parameter is only referred to once in a generic signature, and it enforces a natural model of abstraction, where programmers only need to name an entity when they need to refer to it multiple times. + +## Revisions + +### Changes from the pitch discussion + +* Spell the existential metatype as `any P.Type`, and the protocol metatype as `(any P).Type`. +* Preserve `any` through type aliases. +* Allow `any` on `Any` and `AnyObject`. + +## Acknowledgments + +Thank you to Joe Groff, who originally suggested this direction and syntax in [Improving the UI of generics](https://forums.swift.org/t/improving-the-ui-of-generics/22814), and to those who advocated for this change in the recent discussion about [easing the learning curve for generics](https://forums.swift.org/t/discussion-easing-the-learning-curve-for-introducing-generic-parameters/52891). Thank you to John McCall and Slava Pestov, who helped me figure out the implementation model. diff --git a/proposals/0336-distributed-actor-isolation.md b/proposals/0336-distributed-actor-isolation.md new file mode 100644 index 0000000000..bc7b91c704 --- /dev/null +++ b/proposals/0336-distributed-actor-isolation.md @@ -0,0 +1,1822 @@ +# Distributed Actor Isolation + +* Proposal: [SE-0336](0336-distributed-actor-isolation.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [Pavel Yaskevich](https://github.com/xedin), [Doug Gregor](https://github.com/DougGregor), [Kavon Farvardin](https://github.com/kavon) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0336-distributed-actor-isolation/54726) +* Implementation: + * Partially available in [recent `main` toolchain snapshots](https://swift.org/download/#snapshots) behind the `-enable-experimental-distributed` feature flag. + * This flag also implicitly enables `-enable-experimental-concurrency`. +* Sample app: + * A sample app, showcasing how the various "pieces" work together is available here: + [https://github.com/apple/swift-sample-distributed-actors-transport](https://github.com/apple/swift-sample-distributed-actors-transport) + +## Table of Contents + +- [Distributed Actor Isolation](#distributed-actor-isolation) + - [Table of Contents](#table-of-contents) + - [Introduction](#introduction) + - [Useful links](#useful-links) + - [Motivation](#motivation) + - [Location Transparency](#location-transparency) + - [Remote and Local Distributed Actors](#remote-and-local-distributed-actors) + - [Proposed solution](#proposed-solution) + - [Distributed Actors](#distributed-actors) + - [Complete isolation of state](#complete-isolation-of-state) + - [Distributed Methods](#distributed-methods) + - [Detailed design](#detailed-design) + - [Distributed Actors and Distributed Actor Systems](#distributed-actors-and-distributed-actor-systems) + - [Distributed Actor Initializers](#distributed-actor-initializers) + - [Distributed Actors implicitly conform to Codable](#distributed-actors-implicitly-conform-to-codable) + - [Distributed Methods](#distributed-methods-1) + - [Distributed Method Serialization Requirements](#distributed-method-serialization-requirements) + - [Distributed Methods and Generics](#distributed-methods-and-generics) + - [Distributed Methods and Existential Types](#distributed-methods-and-existential-types) + - [Implicit effects on Distributed Methods](#implicit-effects-on-distributed-methods) + - [Isolation states and Implicit effects on Distributed Methods](#isolation-states-and-implicit-effects-on-distributed-methods) + - [Distributed Actor Properties](#distributed-actor-properties) + - [Stored properties](#stored-properties) + - [Computed properties](#computed-properties) + - [Protocol Conformances](#protocol-conformances) + - [The `DistributedActor` protocol and protocols inheriting from it](#the-distributedactor-protocol-and-protocols-inheriting-from-it) + - [Breaking through Location Transparency](#breaking-through-location-transparency) + - [Future Directions](#future-directions) + - [Versioning and Evolution of Distributed Actors and Methods](#versioning-and-evolution-of-distributed-actors-and-methods) + - [Evolution of parameter values only](#evolution-of-parameter-values-only) + - [Evolution of distributed methods](#evolution-of-distributed-methods) + - [Introducing the `local` keyword](#introducing-the-local-keyword) + - [Alternatives Considered](#alternatives-considered) + - [Implicitly `distributed` methods / "opt-out of distribution"](#implicitly-distributed-methods--opt-out-of-distribution) + - [Introducing "wrapper" type for `Distributed`](#introducing-wrapper-type-for-distributedsomeactor) + - [Creating only a library and/or source-generation tool](#creating-only-a-library-andor-source-generation-tool) + - [Acknowledgments & Prior Art](#acknowledgments--prior-art) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Changelog](#changelog) + +## Introduction + +With the recent introduction of [actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md) to the language, Swift gained powerful and foundational building blocks for expressing *thread-safe* concurrent programs. This proposal is the first in a series of proposals aiming to extend Swift's actor runtime with the concept of *distributed actors*, allowing developers leverage the actor model not only in local, but also distributed settings. + +With distributed actors, we acknowledge that the world we live in is increasingly built around distributed systems, and that we should provide developers with better tools to work within those environments. We aim to simplify and push the state-of-the-art for distributed systems programming in Swift as we did with concurrent programming with local actors and Swift’s structured concurrency approach embedded in the language. + +> The distributed actor proposals will be structured similarly to how Swift Concurrency proposals were: as a series of interconnected proposals that build on top of each other. + +This proposal focuses on the extended actor isolation and type-checking aspects of distributed actors. + +#### Useful links + +Swift Evolution: + +- [Distributed Actors: Pitch #1](https://forums.swift.org/t/pitch-distributed-actors/51669) - a comprehensive, yet quite large, pitch encompassing all pieces of the distributed actor feature; It will be split out into smaller proposals going into the details of each subject, such that we can focus on, and properly review, its independent pieces step by step. + +While this pitch focuses _only_ on the actor isolation rules, we have work-in-progress transport implementations for distributed actors available as well. While they are work-in-progress and do not make use of the complete model described here, they may be useful to serve as reference for how distributed actors might be used. + +- [Swift Distributed Actors Library](https://www.swift.org/blog/distributed-actors/) - a reference implementation of a *peer-to-peer cluster* for distributed actors. Its internals depend on the work in progress language features and are dynamically changing along with these proposals. It is a realistic implementation that we can use as reference for these design discussions. +- "[Fishy Transport](https://github.com/apple/swift-sample-distributed-actors-transport)" Sample - a simplistic example transport implementation that is easier to follow the basic integration pieces than the realistic cluster implementation. Feel free to refer to it as well, while keeping in mind that it is very simplified in its implementation approach. + +## Motivation + +Distributed actors are necessary to expand Swift's actor model to distributed environments. The new `distributed` keyword offers a way for progressively disclosing the additional complexities that come with multiprocess or multi-node environments, into the local-only actor model developers are already familiar with. + +Distributed actors need stronger isolation guarantees than those that are offered by Swift's "local-only" actors. This was a conscious decision, as part of making sure actors are convenient to use in the common scenario where they are only used as concurrency isolation domains. This convenience though is too permissive for distributed programming. + +This proposal introduces the additional isolation checks necessary to allow a distributed runtime to utilize actors as its primary building block, while keeping the convenience and natural feel of such actor types. + +### Location Transparency + +The design of distributed actors intentionally does not provide facilities to easily determine whether an instance is local or remote. The programmer should not _need_ to think about where the instance is located, because Swift will make it work in either case. There are numerous benefits to embracing location transparency: + +- The programmer can write a complex distributed systems algorithm and test it locally. Running that program on a cluster becomes merely a configuration and deployment change, without any additional source code changes. +- Distributed actors can be used with multiple transports without changing the actor's implementation. +- Actor instances can be balanced between nodes once capacity of a cluster changes, or be passivated when not in use, etc. There are many more advanced patterns for allocating instances, such as the "virtual actor" style as popularized by Orleans or Akka's cluster sharding. + +Swift's take on location transparency is expressed and enforced in terms of actor isolation. The same way as actors isolate their state to protect from local race conditions, distributed actors must isolate their state because the state "might not actually be available locally" while we're dealing with a remote distributed actor reference. + +It is also possible to pass distributed actors to distributed methods, if the actor is able to conform to the serialization requirements imposed on it by the actor system. + +### Remote and Local Distributed Actors + +For the purpose of this proposal, we omit the implementation details of a remote actor reference, however as the purpose of actor isolation is to erase the observable difference between a local and remote instance (to achieve location transparency), we need to at least introduce the general concept. + +It is, by design, not possible to *statically* determine if a distributed actor instance is remote or local, therefore all programming against a distributed actor must be done as-if it was remote. This is the root reason for most of the isolation rules introduced in this proposal. For example, the following snippet illustrates location transparency in action, where in our tests we use a local instance, but in a real deployment they would be remote instances communicating: + +```swift +distributed actor TokenRange { + let range: (Token, Token) + var storage: [Token: Data] + + init(...) { ... } + + distributed func read(at loc: Token) -> Data? { + return storage[loc] + } + + distributed func write(to loc: Token, data: Data) -> Data? { + let prev = storage[loc] + storage[loc] = data + return prev + } +} +``` + +Which can be used in a local test: + +```swift +func test_distributedTokenRange() async throws {} + let range = TokenRange(...) + try await assert(range.read(at: testToken) == nil) + + try await write(to: testToken, someData) + try await assert(range.read(at: testToken) == someData) +} +``` + +Distributed functions must be marked with `try` and `await` because they imply asynchronous network calls which may fail. While the `await` rule is the same as with local-only actors, the rule about distributed methods throwing is unique to them because of the assumption that underlying transport mechanisms can fail (i.e. network or serialization errors), regardless if the called function is able to throw or not. + +Note that the even though this test is strictly local -- there are no remote actors involved here at all -- the call-sites of distributed methods have implicitly gained the async and throwing effects, which means that we must invoke them with `try await dist.` This is an important aspect of the design, as it allows us to surface any potential network issues that might occur during these calls, such as timeouts, network failures or other issues that may have caused these calls to fail. This failure is a natural consequence of the calls potentially having to cross process or network boundaries. The asynchronous effect is similar, because we might be waiting for a long time for a response to arrive, distributed calls must be potential suspension points. + +We could write the same unit-test using a distributed remote actor, and the test would remain exactly the same: + +```swift +func test_distributedTokenRange() async throws {} + // the range is actually 'remote' now + let range: TokenRange = + try await assert(range.read(at: testToken) == nil) + + try await write(to: testToken, someData) + try await assert(range.read(at: testToken) == someData) +} +``` + +During this proposal, we will be using the following phrases which have well-defined meanings, so in order to avoid confusion, let us define them explicitly up-front: + +- _distributed actor type_ - any `distributed actor` declaration, or `protocol` declaration that also conforms to `DistributedActor` because they can only be implemented by specific distributed actors, e.g. `protocol Worker: DistributedActor` as well as `distributed actor Worker`, both, can be referred to as "distributed actor type" +- _distributed actor reference_ - any variable, or parameter referring to a distributed actor instance (regardless if remote or local), +- _known-to-be-local distributed actor_, or "_distributed local actor_" for short - a specific known to be local instance of a distributed actor. A distributed actor reference can be checked at runtime if it is remote or local, but in certain situations it is also known in the type system that an actor is "definitely local" and not all isolation checks need to be applied, +- "_distributed remote actor_" - an instance of a distributed actor type, that is actually "remote" and therefore does not have any storage allocated and effectively functions like a "proxy" object. This state does not exist anywhere explicitly in the type-system explicitly, and is what we assume every distributed actor is, unless proven to be "known to be local". + +Keeping this in mind, let us proceed to discussing the specific isolation rules of distributed actors. + +## Proposed solution + +### Distributed Actors + +Distributed actors are a flavor of the `actor` type that enforces additional rules on the type and its instances in order to enable location transparency. Thanks to this, it is possible to program against a `distributed actor` without *statically* knowing if a specific instance is remote or local. All calls are made to look as-if they were remote, and in the local case simply no networking s performed and the calls execute the same as if they were a normal local-only actor. + +Distributed actors are declared by prepending `distributed` to an `actor` declaration: + +```swift +distributed actor Player { + // ... + let name: String +} +``` + +While we do not deep dive into the runtime representation in this proposal, we need to outline the general idea behind them: a `distributed actor` is used to represent an actor which may be either *local* or *remote*. + +This property of hiding away information about the location of the actual instance is called _location transparency_. Under this model, we must program against such location transparent type as-if it was remote, even when it might not be. This allows us to develop and test distributed algorithms locally, without having to resort to networking (unless we want to), vastly simplifying the testing of such systems. + +> **Note:** This is not the same as making "remote calls look like local ones" which has been a failure of many RPC systems. Instead, it is the opposite! Pessimistically assuming that all calls made cross-actor to a distributed actor may be remote, and offering specific ways to guarantee that some calls are definitely local (and thus have the usual, simpler isolation rules). + +Distributed actor isolation checks introduced by this proposal serve the purpose of enforcing the property of location transparency, and helping developers not accidentally break it. For example, the above `Player` actor could be used to represent an actor in a remote host, where the same game state is stored and references to player's devices are managed. As such, the _state_ of a distributed actor is not known locally. This brings us to the first of the additional isolation checks: properties. + +### Complete isolation of state + +Because a distributed actor, along with its actual state, may be located on a remote host, some conveniences local-only actors allow cannot be allowed for distributed ones. Let's consider the following `Player` type: + +```swift +public distributed actor Player { + public let name: String + public var score: Int +} +``` + +Such actor may be running on some remote host, meaning that if we have a "remote reference" to it we _do not_ have its state available, and any attempt to get it would involve network communication. Because of that, stored properties are not accessible across distributed actors: + +```swift +let player: Player = // ... get remote reference to Player +player.name // ❌ error: distributed actor state is only available within the actor instance +``` + +Developers should think carefully about operations that cross into the actor's isolation domain, because the cost of each operation can be very expensive (e.g., if the actor is on a machine across the internet). Properties make it very easy to accidentally make multiple round-trips: + +```swift +func example1(p: Player) async throws -> (String, Int) { + try await (p.name, p.score) // ❌ might make two slow network round-trips to `p` +} +``` + +Instead, the use of methods to perform a batched read is strongly encouraged. + +Stored properties can only be accessed when the actor is known-to-be-local, a property that is possible to check at runtime using the `whenLocal` function that we'll discuss later during this proposal. The following snippet illustrates one example of such known-to-be-local actor access, though there can be different situations where this situation occurs: + +```swift +distributed actor Counter { + var count = 0 + + func publishNextValue() { + count += 1 + Task.detached { @MainActor in + ui.countLabel.text = "Count is now \(await self.count)" + } + } +} +``` + +Stored properties cannot be declared `distributed` nor `nonisolated`. Computed properties however can be either of the two. However, computed properties can only be `distributed` if they are `get`-only due to limitations in how effectful properties work, in which case they function effectively the same as distributed methods which we'll discuss next. + +### Distributed Methods + +In order to enforce the distributed "*maybe remote*" nature of distributed actors, this proposal introduces a new flavor of method declaration called a *distributed method*. Other than a few special cases (such as `nonisolated` members), distributed methods are the only members that can be invoked cross-actor on distributed actors. + +It is necessary to give developers tight control over the distributed nature of methods they write, and it must be a conscious opt-in step. It is also possible to declared computed properties as `distributed`. A distributed method or property is defined within a distributed actor type by writing `distributed` in front of the method's declaration: + +```swift +distributed actor Player { + + distributed func yourTurn() -> Move { + return thinkOfNextMove() + } + + func thinkOfNextMove() -> Move { + // ... + } + + distributed var currentTurn: Int { + // ... + } +} +``` + +It is not possible to invoke the `thinkOfNextMove()` method cross-actor, because the target of the invocation may be remote, and it was not "exposed" for distribution using the `distributed func` keywords. This is checked at compile time and is a more restrictive form of actor-isolation checking: + +```swift +func test(p: Player) async throws { + try await p.yourTurn() + // ✅ ok, distributed func + + try await p.currentTurn + // ✅ ok, distributed computed property + + try await p.thinkOfNextMove() + // ❌ error: only 'distributed' instance methods can be called on a potentially remote distributed actor +} +``` + +Distribution must not be simply inferred from access-control, because the concept of distribution is orthogonal to access control. For example, it is very much common to have `internal distributed func` (or even `private distributed func`) declarations, which are useful for actors within a module communicating with each other (remotely), however those methods should be invoked be end-users of such library. + +Distributed methods may be subject to additional type-checking, specifically a distributed actor infers a `SerializationRequirement` from the ActorSystem it is associated with. One common serialization requirement is `Codable`. + +Such `SerializationRequirement` typealias defined on the actor system the actor is associated with causes additional type-checks to be enforced on distributed methods: all parameter types and return type of such method must be or conform to the SerializationRequirement type. This allows the compiler to fail compilation early, rather than leaving serialization crashes to the runtime, easing development and analysis of distributed actor systems: + +```swift +distributed actor Player { + typealias ActorSystem = CodableMessagingSystem + // inferred: typealias SerializationRequirement = Codable + + distributed func test(not: NotCodable) {} + // ❌ error: parameter 'not' of type 'NotCodable' in distributed instance method + // does not conform to 'Codable' +} +``` + + + +## Detailed design + +Unless otherwise specified in this proposal, the semantics of a distributed actor are the same as a regular actor, as described in [SE-0306](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md). + +### Distributed Actors + +Distributed actors can only be declared using the `distributed actor` keywords. Such types automatically conform to the `DistributedActor` protocol. The protocol is defined in the `_Distributed` module as follows: + +```swift +/// Common protocol to which all distributed actors conform. +/// +/// The `DistributedActor` protocol generalizes over all distributed actor types. +/// All distributed actor types implicitly conform to this protocol. +/// +/// It is not possible to explicitly conform to this protocol using any other declaration +/// other than a 'distributed actor', e.g. it cannot be conformed to by a plain 'actor' or 'class'. +/// +/// ### Implicit Codable conformance +/// If the 'ID' conforms to `Codable` then the concrete distributed actor adopting this protocol +/// automatically gains a synthesized Codable conformance as well. This is because the only reasonable +/// way to implement coding of a distributed actor is to encode it `ID`, and decoding can make use of +/// decoding the same ID, and resolving it using an actor system found in the Decoder's `userInfo`. +/// +/// This works well with `Codable` serialization requirements, and allows actor references to be +/// sent to other distributed actors. +protocol DistributedActor: AnyActor, Identifiable, Hashable + where ID == ActorSystem.ActorID { + + /// Type of the distributed actor system this actor is able to operate with. + /// It can be a type erased, or existential actor system (through a type-eraser wrapper type), + /// if the actor is able to work with different ones. + associatedtype ActorSystem: DistributedActorSystem + + /// The serialization requirement to apply to all distributed declarations inside the actor. + typealias SerializationRequirement = ActorSystem.SerializationRequirement + + /// Unique identity of this distributed actor, used to resolve remote references to it from other peers, + /// and also enabling the Hashable and (optional) Codable conformances of a distributed actor. + /// + /// The id may be freely shard across tasks and processes, and resolving it should return a reference + /// to the actor where it originated from. + nonisolated override var id: ID { get } + + /// Distributed Actor System responsible for managing this distributed actor. + /// + /// It is responsible for assigning and managing the actor's id, + /// as well as delivering incoming messages as distributed method invocations on the actor. + nonisolated var actorSystem: ActorSystem { get } +} +``` + +All distributed actors are *explicitly* part of some specific distributed actor system. The term "actor system" originates from both early, and current terminology relating to actor runtimes and loosely means "group of actors working together", which carries a specific meaning for distributed actors, because it implies they must be able to communicate over some (network or ipc) protocol they all understand. In Swift's local-only actor model, the system is somewhat implicit, because it simply is "the runtime", as all local objects can understand and invoke each other however they see fit. In distribution this needs to become a little more specific: there can be different network protocols and "clusters" to which actors belong, and as such, they must be explicit about their actor system use. We feel this is an expected and natural way to introduce the concept of actor systems only once we enter distribution, because previously (in local only actors) the concept would not have added much value, but in distribution it is the *core* of everything distributed actors do. + +The protocol also includes two nonisolated property requirements: `id` and `actorSystem`. Witnesses for these requirements are nonisolated computed properties that the compiler synthesizes in specific distributed actor declarations. They store the actor system the actor was created with, and its id, which is crucial to its lifecycle and messaging capabilities. We will not discuss in depth how the id is assigned in this proposal, but in short: it is created and assigned by the actor system during the actor's initialization. + +Note, that the `DistributedActor` protocol does *not* refine the `Actor` protocol, but instead it refines the `AnyActor` protocol, also introduced in this proposal. This detail is very important to upholding the soundness of distributed actor isolation. + +Sadly, just refining the Actor protocol results in the following unsound isolation behavior: + +```swift +// Illustrating isolation violation, IF 'DistributedActor' were to refine 'Actor': +extension Actor { + func f() -> SomethingSendable { ... } +} +func g(a: A) async { + print(await a.f()) +} + +// given any distributed actor: +actor MA: DistributedActor {} // : Actor implicitly (not proposed, for illustration purposes only) + +func h(ma: MA) async { + await g(ma) // 💥 would be allowed because a MA is an Actor, but can't actually work at runtime +} +``` + +The general issue here is that a distributed actor type must uphold its isolation guarantees, because the actual instance of such type may be remote, and therefore cannot be allowed to have non-distributed calls made on it. One could argue for the inverse relationship, that `Actor: DistributedActor` as the Actor is more like "`LocalActor`", however this idea also breaks down rather quickly, as one would expect "any IS-A distributed actor type, to have distributed actor isolation", however we definitely would NOT want `actor Worker {}` suddenly exhibit distributed actor isolation. In a way, this way of inheritance breaks the substitution principle in weird ways which could be hacked together to make work, but feel fragile and would lead to hard to understand isolation issues. + +In order to prevent this hole in the isolation model, we must prevent `DistributedActor` from being downcast to `Actor` and the most natural way of doing so, is introducing a shared super-type for the two Actor-like types: `AnyActor`. + +```swift +@_marker +@available(SwiftStdlib 5.6, *) +public protocol AnyActor: Sendable, AnyObject {} + +public protocol Actor: AnyActor { ... } +public protocol DistributedActor: AnyActor, ... { ... } +``` + +Thanks to this protocol we gain an understandable, and complete, type hierarchy for all actor-like behaviors, that is, types that perform a kind of isolation checking and guarantee data-race freedom to invocations on them by serializing them through an actor mailbox. This does not incur much implementation complexity in practice because functionality wise, distributed actors mirror actors exactly, however their customization of e.g. executors only applies to local instances. + +## Distributed Actor Systems + +Libraries aiming to implement distributed actor systems, and act as the runtime for distributed actors must implement the `DistributedActorSystem`. We will expand the definition of this protocol with important lifecycle functions in the runtime focused proposal, however for now let us focus on its aspects which affect type checking and isolation of distributed actors. The protocol is defined as: + +```swift +public protocol DistributedActorSystem: Sendable { + associatedtype ActorID: Hashable & Sendable // discussed below + + /// The serialization requirement that will be applied to all distributed targets used with this system. + typealias SerializationRequirement = // (simplified, actually an associatetype) + + // ... many lifecycle related functions, to be defined in follow-up proposals ... +} +``` + +Every distributed actor must declare what distributed actor system it is able to work with, this is expressed as an `associatedtype` requirement on the `DistributedActor` protocol, to which all `distributed actor` declarations conform implicitly. For example, this distributed actor works with some `ClusterSystem`: + +```swift +distributed actor Worker { + typealias ActorSystem = ClusterSystem +} +``` + +The necessity of declaring this statically will become clear as we discuss the serialization requirements and details of the typechecking mechanisms in the sections below. + +Please note that it is possible to use a protocol or type eraser as the actor system, which allows actors to swap-in completely different actor system implementations, as long as their serialization mechanisms are compatible. Using existential actor systems though comes at a slight performance penalty (as do all uses of existentials). + +It is possible to declare a module-wide `typealias DefaultDistributedActorSystem` in order to change this "default" actor system type, for all distributed actor types declared within a module: + +```swift +// in 'Cluster' module: +typealias DefaultDistributedActorSystem = ClusterSystem + +// in 'Cluster' module, clearly we want to use the 'ClusterSystem' +distributed actor Example { + // synthesized: + // typealias DistributedActorSystem = DefaultDistributedActorSystem // ClusterSystem + + // synthesized initializers (discussed below) also accept the expected type then: + // init(system: DefaultDistributedActorSystem) { ... } +} +``` + +It is also possible to declare protocols which refine the general `DistributedActor` concept to some specific transport, such as: + +```swift +protocol ClusterActor: DistributedActor where DistributedActorSystem == ClusterSystem {} + +protocol XPCActor: DistributedActor where DistributedActorSystem == XPCSystem { } +``` + +Those protocols, because they refine the `DistributedActor` protocol, can also only be conformed to by other distributed actors. It allows developers to declare specific requirements to their distributed actor's use, and even provide extensions based on the actor system type used by those actors, e.g.: + +```swift +extension DistributedActor where DistributedActorSystem == ClusterSystem { + /// Returns the node on which this distributed actor instance is located. + nonisolated var node: Cluster.Node? { ... } +} +``` + +> **Note:** We refer to `distributed actor` declarations or protocols refining the `DistributedActor` protocol as any "distributed actor type" - wherever this phrase is used, it can apply to a specific actor or such protocol. + +### Distributed Actor Initializers + +Distributed actor initializers are always _local_, therefore no special rules are applied to their isolation checking. + +Distributed actor initializers are subject to the same isolation rules as actor initializers, as outlined in [SE-0327: On Actors and Initialization](https://forums.swift.org/t/se-0327-on-actors-and-initialization/53053). Please refer to that proposal for details about when it is safe to escape `self` out of an actor initializer, as well as when it is permitted to call other functions on the actor during its initialization. + +A distributed actor's *designated initializer* must always contain exactly one `DistributedActorSystem` parameter. This is because the lifecycle and messaging of a distributed actor is managed by the system. It also assigns every newly initialized distributed actor instance an identity, that the actor then stores and makes accessible via the compiler-synthesized computed property `id`. The system is similarly available to the actor via the compiler synthesized computed property `actorSystem`. + +Similar to classes and local-only actors, a distributed actor gains an implicit default designated initializer when no user-defined initializer is found. This initializer accepts an actor system as parameter, in order to conform to the requirement stated above: + +```swift +// default system for this module: +typealias DefaultDistributedActorSystem = SomeSystem + +distributed actor Worker { + // synthesized default designated initializer: + // init(system: DefaultDistributedActorSystem) +} +``` + +if no module-wide `DefaultDistributedActorSystem` is defined, such declaration would request the developer to provide one at compile time: + +```swift +distributed actor Worker { + typealias ActorSystem = SomeSystem + + // synthesized default designated initializer: + // init(system: SomeSystem) +} +``` + +Alternatively, we can infer this typealias from a user-defined initializer, like this: + +```swift +distributed actor Worker { + // inferred typealias from explicit initializer declaration + // typealias ActorSystem = SomeSystem + + init(system: SomeSystem) { self.name = "Alice" } +} +``` + +The necessity to pass an actor system to each newly created distributed actor is because the system is the one assigning and managing identities. While we don't discuss those details in depth in this proposal, here is a short pseudocode of why passing this system is necessary: + +```swift +// Lifecycle interactions with the system during initialization +// NOT PART OF THIS PROPOSAL; These will be discussed in-depth in a forthcoming proposal focused on the runtime. +distributed actor Worker { + init(system: SomeSystem) { + // self._system = system + // the actor is assigned an unique identity as it initializes: + // self._id = system.assignID(Self.self) + self.name = "Alice" + // once fully initialized, the actor is ready to receive remote calls: + // system.actorReady(self) + } +} +``` + +Having that said, here are a few example of legal and illegal initializer declarations: + +```swift +distributed actor InitializeMe { + init() + // ❌ error: designated distributed actor initializer 'init()' is missing required 'DistributedActorSystem' parameter + + init(x: String) + // ❌ error: designated distributed actor initializer 'init(x:)' is missing required 'DistributedActorSystem' parameter + + init(system: AnyDistributedActorSystem, too many: AnyDistributedActorSystem) + // ❌ error: designated distributed actor initializer 'init(system:too:)' must accept exactly one DistributedActorSystem parameter, found 2 + + // -------- + + + init(system: AnyDistributedActorSystem) // ✅ ok + init(y: Int, system: AnyDistributedActorSystem) // ✅ ok + init(canThrow: Bool, system: AnyDistributedActorSystem) async throws // ✅ ok, effects are ok too + + // 'convenience' may or may not be necessary, depending on SE-0327 review outcome. + convenience init() { + self.init(system: SomeSystem(...)) // legal, but not recommended + } +} +``` + +*Remote* distributed actor references are not obtained via initializers, but rather through a static `resolve(_:using:)` function that is available on any distributed type: + +```swift +extension DistributedActor { + + /// Resolves the passed in `id` using the passed distributed actor `system`, + /// returning either a local or remote distributed actor reference. + /// + /// The system will be asked to `resolve` the identity and return either + /// a local instance or request a "proxy" to be created for this identity. + /// + /// A remote distributed actor reference will forward all invocations through + /// the system, allowing it to take over the remote messaging with the + /// remote actor instance. + /// + /// - Parameter id: identity uniquely identifying a, potentially remote, actor in the system + /// - Parameter system: distributed actor system which must resolve and manage the returned distributed actor reference + static func resolve(id: ID, using system: DistributedActorSystem) throws -> Self +} +``` + +The specifics of resolving, and remote actor runtime details will be discussed in a follow-up proposal focused on the runtime aspects of distributed actors. We mention it here to share a complete picture how Identities, systems, and remote references all fit into the picture. + +### Distributed Actors implicitly conform to Codable + +If a distributed actor's `ID` conforms to `Codable`, the distributed actor automatically gains a `Codable` conformance as well. + +This conformance is synthesized by the compiler, for every specific `distributed actor` declaration. It is not possible to express such conformance using the conditional conformances. + +> **Note:** It is not possible to implement such conformance semantics on the DistributedActor protocol using conditional conformances (like this `extension DistributedActor: Codable where ID: Codable`), and it is unlikely to be supported in the future. As such, we currently opt to synthesize the conformance for specific distributed actor declarations. + +```swift +distributed actor Player /*: DistributedActor, Codable */ { + // typealias ID = SomeCodableID +} +``` + +The synthesized `Codable` conformance strictly relies on the implementation of the actors' identity `Codable` conformance. When we "encode" a distributed actor, we never encode "the actor", but rather only its identity: + +```swift +// distributed actor Player: Codable, ... { + nonisolated public func encode(to encoder: Encoder) throws { + var container = encoder.singleValueContainer() + try container.encode(self.id) + } +// } +``` + +And similarly, decoding a distributed actor has the specific meaning of attempting to `resolve(_:using:)` a reference of the specific actor type, using the decoded id: + +```swift +// distributed actor Player: Codable, ... { + nonisolated public init(from decoder: Decoder) throws { + // ~~~ pseudo code for illustration purposes ~~~ + guard let system = decoder.userInfo[.distributedActorSystemKey] as? Self.ActorSystem else { + throw DistributedActorCodingError(message: + "Missing DistributedActorSystem (for key .distributedActorSystemKey) " + + "in \(decoder).userInfo, while decoding \(Self.self)!") + } + + // [1] decode the identity + let id: ID = try Self.ID(from: decoder) + // [2] resolve the identity using the current system; this usually will return a "remote reference" + self = try Self.resolve(id: id, using: system) // (!) + } +// } +``` + +The Decodable's `init(from:)` implementation is actually not possible to express in plain Swift today, because the restriction on self assignment in class initializers (and therefore also actor initializers). + +> **Note:** We could eventually generalize this more mutable `self` in class/actor initializer mechanism, however that would be done as separate Swift Evolution proposal. We are aware [this feature was requested before](https://forums.swift.org/t/allow-self-x-in-class-convenience-initializers/15924), and feels like a natural follow up to this proposal to generalize this capability. + +Note also that, realistically, there is only one correct way to implement a distributed actor's codability (as well as `Hashable` and `Equatable` conformances), because the only property that is related to its identity, and is known to both local and remote "sides" is the identity, as such implementations of those protocols must be directly derived from the `id` property of a distributed actor. + +The capability, to share actor references across to other (potentially remote) distributed actors, is crucial for location-transparency and the ability to "send actor references around" which enables developers to implement "call me later" style patterns (since we cannot do so with closures, as they are not serializable). In a way, this is similar to the delegate pattern, known to developers on Apple platforms: where we offer an instance to some other object, that will call lifecycle or other types of methods on the delegate whenever certain events happen. + +To illustrate how this capability is used in practice, let us consider the following turn-based distributed `Game` example, which waits until it has enough players gathered, and then kicks off the game by notifying all the players (regardless _where_ they are located) that the game is now starting. + +```swift +typealias DefaultDistributedActorSystem = SomeCodableDistributedActorSystem +struct SomeCodableDistributedActorSystem: DistributedActorSystem { + typealias ActorID = SomeCodableID + typealias SerializationRequirement = Codable +} + +distributed actor Player { + distributed func play(turn: Int) -> Move { ... } + distributed func opponentMoved(_ move: Move) { ... } +} + +distributed actor Game { + let minPlayers = 2 + var players: Set = [] + + distributed func join(player: Player) async throws { + guard players.count < 2 else { + throw ... + } + + players.insert(player) + + if players.count == 2 { + await play() // keep asking players for their move via 'play(turn:)' until one of them wins + } + } + + func play() async throws { + // keep asking players for their move via 'play(turn:)' until one of them wins + } + + distributed var result: GameResult { + ... + } +} + +func play(game: Game) async throws { + try await game.join(player: Player(system: ...)) + try await game.join(player: Player(system: ...)) + // the game begins, players are notified about it + + let result = try await game.result + print("Winner of \(game) was: \(result.winner)") +} +``` + +The `Player` distributed actor automatically gained a Codable conformance, because it is using the `SomeCodableDistributedActorSystem` that assigns it a `SomeCodableID`. Other serialization mechanisms are also able to implement this "encode the ID" and "decode the ID, and resolve it" pattern, so this pattern is equally achievable using Codable, or other serialization mechanisms. + +### Distributed Methods + +The primary way a distributed actor can be interacted with are distributed methods. Most notably, invoking a non-distributed method (i.e. those declared with *just* the `func` keyword by itself), is not allowed as it may be potentially violating distributed actor isolation rules, that is unless the target of the invocation is known to be a *local* distributed actor - a topic we'll explore later on in this proposal: + +```swift +distributed actor IsolationExample { + func notDistributed() {} + distributed func accessible() {} + distributed var computed: String { "" } +} + +func test(actor: IsolationExample) async throws { + try await actor.notDistributed() + // ❌ error: only 'distributed' instance methods can be called on a potentially remote distributed actor + + try await actor.accessible() + // ✅ ok, method is distributed + + try await actor.computed + // ✅ ok, distributed get-only computed property +} +``` + +Distributed methods are declared by writing the `distributed` keyword in the place of a declaration modifier, under the `actor-isolation-modifier` production rule as specified by [the grammar in TSPL](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_declaration-modifiers). Only methods can use `distributed` as a declaration modifier, and no order is specified for this modifier. + +It is also possible to declare distributed get-only properties, and they obey the same rules as a parameter-less `distributed func` would. It is not permitted to make get/set computed properties, or stored properties `distributed`. + +Distributed actor types are the only types in which a distributed method declaration is allowed. This is because, in order to implement a distributed method, an actor system and identity must be associated with the values carrying the method. Distributed methods can synchronously refer to any of the state isolated to the distributed actor instance. + +The following distributed method declarations are not allowed: + +```swift +actor/class/enum/struct NotDistributedActor { + distributed func test() {} + // ❌ error: 'distributed' function can only be declared within 'distributed actor' +} + +protocol NotDistributedActorProtocol { + distributed func test() + // ❌ error: 'distributed' function can only be declared within 'distributed actor' + // 💡 fixit: add ': DistributedActor' to protocol inheritance clause +} +``` + +While these are all proper declarations: + +```swift +distributed actor Worker { + distributed func work() { ... } +} + +extension Worker { + distributed func reportWorkedHours() -> Duration { ... } +} + +protocol TypicalGreeter: DistributedActor { + distributed func greet() +} +``` + +The last example, the `TypicalGreeter` protocol, can *only* be implemented by a `distributed actor`, because of the `DistributedActor` requirement. We will discuss distributed actors conforming to protocols in great detail below. + +It is not allowed to combine `distributed` with `nonisolated`, as a distributed function is _always_ isolated to the actor in which it is defined. + +```swift +distributed actor Charlie { + distributed nonisolated func cantDoThat() {} + // ❌ error: 'distributed' function must not be 'nonisolated' + // 💡 fixit: remove 'nonisolated' or 'distributed' +} +``` + +It is possible to declare a nonisolated method though. Such function can only access other `nonisolated` members of the instance. Two important members which are such nonisolated computed properties are the actor's identity, and associated actor system. Those are synthesized by the compiler, however they just follow the same isolation rules as laid out in this proposal: + +```swift +distributed actor Charlie: CustomStringConvertible { + // synthesized: nonisolated var id: Self.ID { get } + // synthesized: nonisolated var actorSystem: Self.ActorSystem { get } + + nonisolated var description: String { + "Charlie(\(self.id))" // ok to refer to `self.id` since also nonisolated + } +} +``` + +Distributed methods may be declared explicitly `async` or `throws` and this has the usual effect on the declaration and method body. It has no effect on cross distributed actor calls, because such calls are implicitly asynchronous and throwing to begin with. + +The `distributed` nature of a method is completely orthogonal to access control. It is even possible to declare a `private distributed func` because the following pattern may make it an useful concept to have: + +```swift +distributed actor Robot { + + nonisolated async throws isHuman(caller: Caller) -> String { + guard isTrustworthy(caller) else { + return "It is a mystery!" // no remote call needs to be performed + } + + return try await self.checkHumanity() + } + + private distributed func checkHumanity() -> String { + "Human, after all!" + } +} +``` + +Such methods allow us avoiding remote calls if some local validation already can short-circuit them. While not a common pattern, it definitely can have its uses. Note that the ability to invoke distributed methods remotely, also directly translates into such methods being "effectively public", even if access control wise they are not. This makes sense, and distributed methods must always be audited and carefully checked if they indeed should be allowed to execute when invoked remotely, e.g. they may need to perform caller authentication – a feature we do not provide out of the box yet, but are definitely interested in exploring in the future. + +It is not allowed to declare distributed function parameters as `inout` or varargs: + +```swift +distributed actor Charlie { + distributed func varargs(int: Int...) {} + // ❌ error: cannot declare variadic argument 'int' in distributed instance method 'varargs(int:)' + + distributed func noInout(inNOut burger: inout String) {} + // ❌ error: cannot declare 'inout' argument 'burger' in distributed instance method 'noInout(inNOut:)' + // 💡 fixit: remove 'inout' +} +``` + +While subscripts share many similarities with methods, they can lead to complex and potentially impossible to support invocations, meaning that they are currently also not allowed to be `distributed`. Such subscripts' usefulness would, in any case, be severely limited by both their lack of support for being `async` (e.g., could only support read-only subscripts, because no coroutine-style accessors) and their lightweight syntax can lead to the same problems as properties. + +Distributed functions _may_ be combined with property wrappers to function parameters (which were introduced by [SE-0293: Extend Property Wrappers to Function and Closure Parameters](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0293-extend-property-wrappers-to-function-and-closure-parameters.md)), and their semantics are what one would expect: they are a transformation on the syntactical level, meaning that the actual serialized parameter value is what the property wrapper has wrapped the parameter in. This is especially interesting for implementing eager validation of specific parameters, such that calls with illegal argument values can be synchronously prevented before even sending the message. Of course, the recipient should still validate the incoming arguments using the same logic, but thanks to this we are able to avoid sending wrong values in non-adversarial situations, and just validate some values on the client side eagerly. + +#### Distributed Method Serialization Requirements + +An important goal of the distributed actor design is being able to enforce some level of compile time safety onto distributed methods calls, which helps prevent unexpected runtime failures, and aides developers make conscious decisions which types should be exposed to remote peers and which not. + +This feature is applied to `distributed` methods, and configured by declaring a `SerializationRequirement` typealias on the actor system, from which specific actors infer it. This type alias informs the type-checker to ensure that all parameters, as well as return type of distributed methods must conform to the type that is provided as `SerializationRequirement`. This is in addition to the usual `Sendable` conformance requirements enforced on any values passed to/from actors). + +Another interesting capability this unlocks is being able to confine actors to sending only well-known types, if we wanted to enforce such closed-world assumptions onto the permissible messages exchanged between actors. + +Most frequently, the serialization requirement is going to be `Codable`, so for the rest of this proposal we'll focus mostly on this use-case. It is equally possible and supported to provide e.g. an external serialization systems top-level protocol as requirement here, e.g. a Protocol Buffer `Message`. The following snippet illustrates how this can work in practice: + +```swift +protocol CodableDistributedActorSystem: DistributedActorSystem { + typealias SerializationRequirement = Codable +} + +distributed actor Worker { + typealias ActorSystem = CodableDistributedActorSystem + typealias SerializationRequirement = SpecificActorSystem.SerializationRequirement + // = Codable +} +``` + +It is possible, albeit not recommended, to disable this checking by setting the `SerializationRequirement` to `Any` in which case no additional checks are performed on distributed methods. + +This section will discuss the implications of the `SerializationRequirement` on distributed method declarations. + +A serialization requirement means that all parameter types and return type of distributed method must conform to the requirement. With the `CodableDistributedActorSystem` in mind, let us write a few methods and see how this works: + +```swift +distributed actor Worker { + typealias ActorSystem = CodableDistributedActorSystem + + distributed func ok() // ✅ ok, no parameters + distributed func greet(name: String) -> String // ✅ ok, String is Codable + + struct NotCodable {} + + distributed func reject(not: NotCodable) + // ❌ error: parameter 'not' of type 'NotCodable' in distributed instance method + // does not conform to 'Codable' + // 💡 fixit: add ': Codable' to 'struct NotCodable' +} +``` + +This also naturally extends to closures without any the need of introducing any special rules, because closures do not conform to protocols (such as `Codable`), the following is naturally ill-formed and rejected: + +```swift +distributed actor Worker { + typealias ActorSystem = CodableDistributedActorSystem + + distributed func take(_ closure: (String) -> String) + // ❌ error: parameter 'closure' of type '(String) -> String' in distributed instance method + // does not conform to 'Codable' +} +``` + +Thrown errors are not enforced to be `Codable`, however a distributed actor system may detect that an error is Codable at runtime, and attempt to transfer it back entirely. For throws of non-Codable types, systems should attempt some form of best-effort description of the error, while keeping in mind privacy of error descriptions. I.e. errors should never be sent back to the caller by just getting their description, as that may leak sensitive information from the server system. A recommended approach here is to send back the type of the thrown error and throwing some generic `NotCodableError("\(type(of: error))")` or similar. + +Distributed actors may also witness protocol requirements (discussed in more detail below), however their method declarations must then also conform to the `SerializationRequirement`: + +```swift +protocol Greetings { + func greet(name: String) async throws + func synchronous() +} + +distributed actor Greeter: Greetings { + // typealias SerializationRequirement = Codable + distributed func greet(name: String) { // may or may not be async/throws, it always is when cross-actor + // ✅ ok, String is Codable + } + + nonisolated func synchronous() {} // nonisolated func may be used the same as on normal actors +} +``` + +Note that while every `distributed actor` must be associated with some specific distributed actor system, protocols need not be so strict and we are allowed to specify a distributed actor protocol like this: + +```swift +protocol Greetings: DistributedActor { + // no specific ActorSystem requirement (!) + func greet(name: String) +} +``` + +At the declaration site of such protocol the distributed functions are *not* subject to any `SerializationRequirement` checks. However once it is implemented by a distributed actor, that actor will be associated with a specific actor system, and thus also a specific SerializationRequirement, and could potentially not be able to implement such protocol because of the serializability checks, e.g.: + +```swift +protocol Greetings: DistributedActor { + // no specific ActorSystem requirement (!) + func greet(name: String) +} + +distributed actor Greeter { + // typealias SerializationRequirement = MagicMessage + distributed func greet(name: String) {} + // ❌ error: parameter 'name' of type 'String' in distributed instance method + // does not conform to 'MagicMessage' +} +``` + +A similar mechanism will exist for resolving remote actor references only based on a protocol. + +#### Distributed Methods and Generics + +It is possible to declare and use distributed methods that make use of generics. E.g. we could define an actor that picks an element out of a collection, yet does not really care about the element type: + +```swift +distributed actor Picker { + func pickOne(from items: [Item]) -> Item? { // Is this ok? It depends... + ... + } +} +``` + +This is possible to implement in general, however the `Item` parameter will be subject to the same `SerializableRequirement` checking as any other parameter. Depending on the associated distributed actor system's serialization requirement, this declaration may fail to compile, e.g. because `Item` was not guaranteed to be `Codable`: + +```swift +distributed actor Picker { + // typealias ActorSystem = CodableMessagingSystem + func pickOne(from items: [Item]) -> Item? { nil } + // ❌ error: parameter 'items' of type '[Item]' in distributed instance method + // does not conform to 'Codable' + // ❌ error: return type 'Item' in distributed instance method does not conform to 'Codable' + + func pickOneFixed(from items: [Item]) -> Item? + where Item: Codable { nil } // ✅ ok, we declared that the generic 'Item' is 'Codable' +} +``` + +This is the same rule about serialization requirements really, but spelled out explicitly. + + The runtime implementation of such calls is more complicated than non-generic calls, and does incur a slight wire envelope size increase, because it must carry the *specific type identifier* that was used to perform the call (e.g. that it was invoked using the *specific* `struct MyItem: Item` and not just some item). Generic distributed function calls will perform the deserialization using the *specific type* that was used to perform the remote invocation. + +As with any other type involved in message passing, actor systems may also perform additional inspections at run time of the types and check if they are trusted or not before proceeding to decode them (i.e. actor systems have the possibility to inspect incoming message envelopes and double-check involved types before proceeding tho decode the parameters). + +It is also allowed to make distributed actors themselves generic, and it works as one would expect: + +```swift +distributed actor Worker { // ✅ ok + func work() -> Item { ... } +} +``` + + + +#### Distributed Methods and Existential Types + +It is worth calling out that due to existential types not conforming to themselves, it is not possible to just pass a `Codable`-conforming existential as parameter to distributed functions. It will result in the following compile time error: + +```swift +protocol P: Codable {} + +distributed actor TestExistential { + typealias ActorSystem = CodableMessagingSystem + + distributed func compute(s: String, i: Int, p: P) {} + // ❌ error: parameter 'p' of type 'P' in distributed instance method does not conform to 'Codable' +} +``` + +The way to deal with this, as with usual local-only Swift programming, is to make the `P` existential generic, like this: + +```swift +protocol P: Codable {} + +distributed actor TestExistential { + typealias ActorSystem = CodableMessagingSystem + + distributed func compute(s: String, i: Int, p: Param) {} + // ✅ ok, the generic allows us getting access to the specific underlying type +} +``` + +which will compile, and work as expected. + +#### Implicit effects on Distributed Methods + +Local-only actor methods can be asynchronous , throwing or both, however invoking them cross-actor always causes them to become implicitly asynchronous: + +```swift +// Reminder about implicit async on actor functions +actor Greeter { + func greet() -> String { "Hello!" } + func inside() { + greet() // not asynchronous, we're not crossing an actor boundary + } +} + +Task { + await Greeter().hi() // implicitly asynchronous +} +``` + +The same mechanism is extended to the throwing behavior of distributed methods. Distributed cross-actor calls may fail not only because of the remote side actively throwing an error, but also because of transport errors such as network issues or serialization failures. Therefore, distributed cross-actor calls also implicitly gain the throwing effect, and must be marked with `try` when called: + +```swift +distributed actor Greeter { + distributed func greet() -> String { "Hello!" } + + func inside() { + greet() // not asynchronous or throwing, we're inside the actual local instance + } +} + +Task { + try await Greeter().greet() // cross-actor distributed function call: implicitly async throws +} +``` + +It is also possible to declare distributed functions as either `throws` or `async` (or both). The implicitly added effect is a no-op then, as the function always was, respectively, throwing or asynchronous already. + +The following snippets illustrate all cases how effects are applied to distributed actor methods: + +```swift +distributed actor Worker { + distributed func simple() {} + distributed func funcAsync() async {} + distributed func funcThrows() throws {} + distributed func funcAsyncThrows() async throws {} +} +``` + +Cross distributed-actor calls behave similar to cross actor calls, in the sense that they gain those implicit effects. This is because we don't know if the callee is remote or local, and thus assume that it might be remote, meaning that there may be transport errors involved in the call, making the function call implicitly throwing: + +```swift +func outside(worker: Worker) async throws { + // wrong invocation: + worker.simple() + // ❌ error: expression is 'async' but is not marked with 'await' + // ❌ error: call can throw but is not marked with 'try' + // 💡 note: calls to distributed instance method 'simple()' from outside of its actor context are implicitly asynchronous + + // proper invocations: + try await worker.simple() + try await worker.funcAsync() + try await worker.funcThrows() + try await worker.funcAsyncThrows() +} +``` + +These methods may be also be called from *inside* the actor, as well as on an `isolated` parameter of that actor type, without any implicit effects applied to them. This is the same idea applies that actor methods becoming implicitly asynchronous but only during cross-actor calls. + +```swift +extension Worker { + distributed func inside() async throws { + self.simple() + await self.funcAsync() + try self.funcThrows() + try await self.funcAsyncThrows() + } +} + +func isolatedFunc(worker: isolated Worker) async throws { + worker.simple() + await worker.funcAsync() + try worker.funcThrows() + try await worker.funcAsyncThrows() +} +``` + +The isolated function parameter works because the only way to offer an `isolated Worker` to a function, is for a real local actor instance to offer its `self` to `isolatedFunc`, and because of that it is known that it is a real local instance (after all, only a real local instance has access to `self`). + +It is not allowed to declare `isolated` parameters on distributed methods, because distributed methods _must_ be isolated to the actor they are declared on. This can be thought of always using an `isolated self: Self` parameter, and in combination of a func only being allowed to be isolated to a single actor instance, this means that there cannot be another isolated parameter on such functions. Following this logic a `nonisolated func` declared on a distributed actor, _is_ allowed to accept `isolated` parameters, however such call will not be crossing process boundaries. + +It is also worth calling out the interactions with `Task` and `async let`. Their context may be the same asynchronous context as the actor, in which case we also do not need to cause the implicit asynchronous effect. When it is known the invocation is performed on an `isolated` distributed actor reference, we infer the fact that it indeed is "known to be local", and do not need to apply the implicit throwing effect either: + +```swift +extension Worker { + func test(other: Philosopher) async throws { + // self -------------------------------------------------------------------- + async let alet = self.simple() // implicitly async; async let introduced concurrent context + _ = await alet // not throwing, but asynchronous! + + Task { + _ = self.hi() // no implicit effects, Task inherited the Actor's execution context + } + + Task.detached { + _ = await self.hi() // implicitly async, different Task context than the actor + // however not implicitly throwing; we know there is no networking involved in a call on self + } + + // other ------------------------------------------------------------------- + async let otherLet = other.hi() // implicitly async and throws; other may be remote + _ = try await otherLet // forced to 'try await' here, as per usual 'async let' semantics + + Task { + _ = try await other.hi() // implicitly async and throws + } + + Task.detached { + _ = try await other.hi() // implicitly async and throws + } + } +} +``` + +#### Isolation states and Implicit effects on Distributed Methods + +A distributed actor reference. such as a variable or function parameter, effectively can be in one of three states: + +- `isolated` – as defined by Swift's local-only actors. The `isolated` also implies the following "local" state, because it is not possible to pass isolated members across distributed boundaries, +- "local" – not explicitly modeled in the type-system in this proposal, though we might end up wanting to do so (see Future Directions), or +- "potentially remote" – which is the default state of any distributed actor variable. + +These states determine the implicit effects that function invocations, and general distributed actor isolation checking, need to apply when checking accesses through the distributed actor reference. + +Let us discuss the implications of these states on the effects applied to method calls on such distributed actor references, starting from the last "potentially remote" state, as it is the default and most prominent state which enables location-transparency. + +By default, any call on a ("potentially remote") distributed actor must be assumed to be crossing network boundaries. Thus, the type system pessimistically applies implicit throwing and async effects to such call-sites: + +```swift +func test(actor: Greeter) async throws { + try await actor.greet(name: "Asa") // ✅ call could be remote +} +``` + +In special circumstances, a reference may be "known to be local", even without introducing a special "local" keyword in the language this manifests itself for example in closures which capture `self`. For example, we may capture `self` in a detached task, meaning that the task's closure will be executing on some different execution context than the actor itself -- and thus `self` is *not* isolated, however we *know* that it definitely is local, because there is no way we could ever refer to `self` from a remote actor: + +```swift +distributed actor Closer { + distributed func check() -> Bool { true } + + func test() { + Task.detached { + await self.check() // ✅ call is definitely local, but it must be asynchronous + } + } +} +``` + +In the above situation, we know for sure that the `self.check()` will not be crossing any process boundaries, and therefore there cannot be any implicit errors emitted by the underlying distributed actor system transport. This manifests in the type-system by the `distributed func` call not being throwing (!), however it remains asynchronous because of the usual local-only actor isolation rules. + +The last case is `isolated` distributed actor references. This is relatively simple, because it just reverts all isolation checking to the local-only model. Instance members of actors are effectively methods which take an `isolated Self`, and in the same way functions which accept an `isolated Some(Distributed)Actor` are considered to be isolated to that actor. For the purpose of distributed actor isolation checking it effectively means there are no distributed checks at all, and we can even access stored properties synchronously on such reference: + +```swift +distributed actor Namer { + let baseName: String = ... +} + +func bad(n: Namer) { + n.baseName // ❌ error, as expected we cannot access the distributed actor-isolated state +} + +func good(n: isolated Namer) { + n.baseName // ✅ ok; we are isolated to the specific 'n' Namer instance +} +``` + +### Distributed Actor Properties + +#### Stored properties + +Distributed actors may declare any kind of stored property, and the declarations themselves are *not restricted in any way*. This is important and allows distributed actors to store any kind of state, even if it were not serializable. Access to such state from the outside though is only allowed through distributed functions, meaning that cross-network access to such non-serializable state must either be fully encapsulated or "packaged up" into some serializable format that leans itself to transporting across the network. + +One typical example of this is a distributed actor storing a live database connection, and being unable to send this connection across to other nodes, it should send the results of querying the database to its callers. This is a very natural way to think about actor storage, and will even be possible to enforce at compile time, which we'll discuss in follow-up proposals discussing serialization and runtime aspects of distributed actor messages. + +To re-state the rule once again more concisely: It is not possible to reach a distributed actors stored properties cross-actor. This is because stored properties may be located on a remote host, and we do not want to subject them to the same implicit effects, and serialization type-checking as distributed methods. + +```swift +distributed actor Properties { + let fullName: String + var age: Int +} +``` + +Trying to access those properties results in isolation errors at compile time: + +```swift +Properties().fullName +// ❌ error: distributed actor-isolated property 'fullName' can only be referenced inside the distributed actor +Properties().age +// ❌ error: distributed actor-isolated property 'age' can only be referenced inside the distributed actor +``` + +Unlike with local-only actors, it is *not* allowed to declare `nonisolated` *stored properties*, because a nonisolated stored property implies the ability to access it without any synchronization, and would force the remote "proxy" instance to have such stored property declared and initialized, however there is no meaningful good way to initialize such variable, because a remote reference is _only_ the actor's identity and associated transport (which will be explored in more depth in a separate proposal): + +```swift +distributed actor Properties { + nonisolated let fullName: String // ❌ error: distributed actor cannot declare nonisolated stored properties +} +``` + +It is allowed to declare static properties on distributed actors, and they are not isolated to the actor. This is the same as static properties on local-only actors. + +```swift +distributed actor Worker { + static let MAX_ITEMS: Int = 12 // ⚠️ static properties always refer to the value in the *local process* + var workingOnItems: Int = 0 + + distributed func work(on item: Item) throws { + guard workingOnItems < Self.MAX_ITEMS else { + throw TooMuchWork(max: Self.MAX_ITEMS) + } + + workingonItems += 1 + } +} +``` + +Be aware though that any such `static` property on a `distributed actor` always refers to whatever the property was initialized with _locally_ (in the current process). i.e. if the remote node is running a different version of the software, it may have the `MAX_ITEMS` value set to something different. So keep this in mind when debugging code while rolling out new versions across a cluster. Static properties are useful for things like constants, so feel free to use them in the same manner as you would with local-only actors. + +It is permitted, same as with local-only actors, to declare `static` methods and even `static` variables on distributed actors, although please be advised that currently static variables are equally thread-*unsafe* as global properties and Swift Concurrency currently does not perform any checks on those. + +```swift +// Currently allowed in Swift 5.x, but dangerous (for now) +[distributed] actor Glass { + var contents: String = Glass.defaultContents + + static var defaultContents: String { "water" } // ⚠️ not protected from data-races in Swift 5.x +} +``` + +As such, please be very careful with such mutable declarations. Swift Concurrency will eventually also check for shared global and static state, and devise a model preventing races in such declarations as well. Static properties declared on distributed actors will be subject to the same checks as any other static properties or globals once this has been proposed and implemented (via a separate Swift Evolution proposal). + +#### Computed properties + +Distributed _computed properties_ are possible to support in a very limited fashion because of the effectful nature of the distributed keyword. It is only possible to make *read-only* properties distributed, because only such properties may be effectful (as introduced by [SE-0310: Effectful Read-only Properties](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0310-effectful-readonly-properties.md)). + +```swift +distributed actor Chunk { + let chunk: NotSerializableDataChunk + + distributed var size: Int { self.chunk.size } +} +``` + +A distributed computed property is similar to a method accepting zero arguments, and returning a value. + +Distributed computed properties are subject to the same isolation rules, and implicit async and throwing effects. As such, accessing such variable (even across the network) is fairly explicitly telling the developer something is going on here, and they should re-consider if e.g. doing this in a loop truly is a good idea: + +```swift +var i = 0 +while i < (try await chunk.size) { // very bad idea, don't do this + // logic here + i += 1 +} + +// better, only check the size once: +var i = 0 +let max = try await chunk.size // implicitly 'async throws', same as distributed methods +while i < max { + // logic here + i += 1 +} +``` + +Because distributed methods and properties are statically known, we could envision IDEs giving explicit warnings, and even do some introspection and analysis detecting such patterns if they really wanted to. + +Any value returned by such computed property needs to be able to be serialized, similarly to distributed method parameters and return values, and would be subject to the same checks. + +It is not possible to declare read/write computed properties, because of underlying limitations of effectful properties. + +### Protocol Conformances + +Distributed actors can conform to protocols in the same manner as local-only actors can. + +As calls "through" protocols are always cross-actor, requirements that are possible to witness by a `distributed actor` must be `async throws`. The following protocol shows a few examples of protocol requirements, and whether they are possible to witness using a distributed actor's distributed function: + +```swift +protocol Example { + func synchronous() + func justAsync() async -> Int + func justThrows() throws -> Int + func asyncThrows() async throws -> String +} +``` + +We can attempt to conform to this protocol using a distributed actor: + +```swift +distributed actor ExampleActor: Example { + distributed func synchronous() {} + // ❌ error: actor-isolated instance method 'synchronous()' cannot be used to satisfy a protocol requirement + // cross-actor calls to 'justThrows()' are 'async throws' yet protocol requirement is synchronous + + distributed func justAsync() async -> Int { 2 } + // ❌ error: actor-isolated instance method 'justAsync()' cannot be used to satisfy a protocol requirement + // cross-actor calls to 'justAsync()' are 'async throws' yet protocol requirement is only 'async' + + distributed func justThrows() throws -> Int { 2 } + // ❌ error: actor-isolated instance method 'justThrows()' cannot be used to satisfy a protocol requirement}} + // cross-actor calls to 'justThrows()' are 'async throws' yet protocol requirement is only 'throws' + + distributed func asyncThrows() async throws -> String { "two" } // ✅ +} +``` + +Let us focus on the last example, `asyncThrows()` which is declared as a throwing and asynchronous protocol requirement, and returns a `String`. We are able to witness this requirement, but we should mention the future direction of compile time serialization checking while discussing this function as well. + +If we recall the previously mentioned serialization conformance checking mechanism, we could imagine that the `ExampleActor` configured itself to use e.g. `Codable` for its message serialization. This means that the method declarations are subject to `Codable` checking: + +```swift +distributed actor CodableExampleActor: Example { + typealias SerializationRequirement = Codable + + distributed func asyncThrows() async throws -> String { "two" } // ✅ ok, String is Codable +} +``` + +As we can see, we were still able to successfully witness the `asyncThrows` protocol requirement, since the signature matches our serialization requirement. This allows us to conform to existing protocol requirements with distributed actors, without having to invent complicated wrappers. + +If we used a different serialization mechanism, we may have to provide a `nonisolated` witness, that converts the types expected by the protocol, to whichever types we are able to serialize (e.g. protocol buffer messages, or anything else, including custom serialization formats). Either way, we are able to work our way through and conform to protocols if necessary. + +It is possible to utilize `nonisolated` functions to conform to synchronous protocol requirements, however those have limited use in practice on distributed actors since they cannot access any isolated state. In practice such functions are implementable by accessing the actor's identity or actor system it belongs to, but not much else. + +```swift +protocol CustomStringConvertible { + var description: String { get } +} + +distributed actor Example: CustomStringConvertible { + nonisolated var description: String { + "distributed actor Example: \(self.identity)" + } +} +``` + +The above example conforms a distributed actor to the well-known `CustomStringConvertible` protocol, and we can use similar techniques to implement protocols like `Hashable`, `Identifiable`, and even `Codable`. We will discuss these in the following proposals about distributed actor runtime details though. + +#### The `DistributedActor` protocol and protocols inheriting from it + +This proposal mentioned the `DistributedActor` protocol a few times, however without going into much more depth about its design. We will leave this to the *actor runtime* focused proposals, however in regard to isolation we would like do discuss its relation to protocols and protocol conformances: + +The `DistributedActor` protocol cannot be conformed to explicitly by any other type other than a `distributed actor` declaration. This is similar to the `Actor` protocol and `actor` declarations. + +It is possible however to express protocols that inherit from the `DistributedActor` protocol, like this: + +```swift +protocol Worker: DistributedActor { + distributed func work(on: Item) -> Int + + nonisolated func same(as other: Worker) -> Bool + + static func isHardWorking(_ worker: Worker) -> Bool +} +``` + +Methods definitions inside distributed actor inheriting protocols must be declared either:`distributed`, `static`or `nonisolated`. Again, we value the explicitness of the definitions, and the compiler will guide and help you decide how the method shall be isolated. + +Note that it is always possible to conform to a distributed protocol requirement with a witness with "more" effects, since the cross-actor API remains the same - thanks to the implicit effects caused by the distributed keyword. + +```swift +protocol Arnold: Worker { + distributed func work(on: Item) async -> Int { + // turns out we need this to be async internally, this is okay + } +} +``` + +This witness works properly, because the `distributed func` requirement in the protocol is always going to be `async throws` due to the `distributed func`'s effect on the declaration. Therefore the declaration "inside the actor" can make use of `async` or `throws` without changing how the protocol can be used. + +### Breaking through Location Transparency + +Programs based on distributed actors should always be written to respect location transparency, but sometimes it is useful to break through that abstraction. The most common situation where breaking through location transparency can be useful is when writing unit tests. Such tests may need to inspect state, or call non-distributed methods, of a distributed actor instance that is known to be local. + +To support this kind of niche circumstance, all distributed actors offer a `whenLocal` method, which executes a provided closure based on whether it is a local instance: + +```swift +extension DistributedActor { + /// Runs the 'body' closure if and only if the passed 'actor' is a local instance. + /// + /// Returns `nil` if the actor was remote. + @discardableResult + nonisolated func whenLocal( + _ body: (isolated Self) async throws -> T + ) async rethrows -> T? + + /// Runs the 'body' closure if and only if the passed 'actor' is a local instance. + /// + /// Invokes the 'else' closure if the actor instance was remote. + @discardableResult + nonisolated func whenLocal( + _ body: (isolated Self) async throws -> T, + else whenRemote: (Self) async throws -> T + ) async rethrows -> T +``` + +When the instance is local, the `whenLocal` method exposes the distributed actor instance to the provided closure, as if it were a regular actor instance. This means you can invoke non-distributed methods when the actor instance is local, without relying on hacks that would trigger a crash if invoked on a remote instance. + +> **Note:** We would like to explore a slightly different shape of the `whenLocal` functions, that would allow _not_ hopping to the actor unless necessary, however we are currently lacking the implementation ability to do so. So this proposal for now shows the simple, `isolated` based approach. The alternate API we are considering would have the following shape: +> +> ```swift +> @discardableResult +> nonisolated func whenLocal( +> _ body: (local Self) async throws -> T +> ) reasync rethrows -> T? +> ``` +> +> This API could enable us to treat such `local DistActor` exactly the same as a local-only actor type; We could even consider allowing nonisolated stored properties, and allow accessing them synchronously like that: +> +> ```swift +> // NOT part of this proposal, but a potential future direction +> distributed actor FamousActor { +> let name: String = "Emma" +> } +> +> FamousActor().whenLocal { fa /*: local FamousActor*/ in +> fa.name // OK, known to be local, distributed-isolation does not apply +> } +> ``` + +## Future Directions + +### Versioning and Evolution of Distributed Actors and Methods + +Versioning and evolution of exposed `distributed` functionality is a very important, and quite vast topic to tackle. This proposal by itself does not include new capabilities - we are aware this might be limiting adoption in certain use-cases. + +#### Evolution of parameter values only + +In today's proposal, it is possible to evolve data models *inside* parameters passed through distributed method calls. This completely relies on the serialization mechanism used for the individual parameters. Most frequently, we expect Codable, or some similar mechanism, to be used here and this evolution of those values relies entirely on what the underlying encoders/decoders can do. As an example, we can define a `Message` struct like this: + +```swift +struct Message: Codable { + let oldVersion: String + let onlyInNewVersion: String +} + +distributed func accept(_: Message) { ... } +``` + +and the usual backwards / forwards evolution techniques used with `Codable` can be applied here. Most coders are able to easily ignore new unrecognized fields when decoding. It is also possible to improve or implement a different decoder that would also store unrecognized fields in some other container, e.g. like this: + +```swift +struct Message: Codable { + let oldVersion: String + let unknownFields: [String: ...] +} + +JSONDecoderAwareOfUnknownFields().decode(Message.self, from: ...) +``` + +and the decoder could populate the `unknownFields` if necessary. There are various techniques to perform schema evolution here, and we won't be explaining them in more depth here. We are aware of limitations and challenges related to `Codable` and might revisit it for improvements. + +#### Evolution of distributed methods + +The above-mentioned techniques apply only for the parameter values themselves though. With distributed methods we need to also take care of the method signatures being versioned, this is because when we declare + +```swift +distributed actor Greeter { + distributed func greet(name: String) +} +``` + +we exposed the ability to invoke `greet(name:)` to other peers. Such normal, non-generic signature will *not* cause the transmission of `String`, over the wire. They may be attempting to invoke this method, even as we roll out a new version of the "greeter server" which now has a new signature: + +```swift +distributed actor Greeter { + distributed func greet(name: String, in language: Language) +} +``` + +This is a breaking change as much in API/ABI and of course also a break in the declared wire protocol (message) that the actor is willing to accept. + +Today, Swift does not have great facilities to move between such definitions without manually having to keep around the forwarder methods, so we'd do the following: + +```swift +distributed actor Greeter { + + @available(*, deprecated, renamed: "greet(name:in:)") + distributed func greet(name: String) { + self.greet(name: name, in: .defaultLanguage) + } + + distributed func greet(name: String, in language: Language) { + print("\(language.greeting), name!") + } +} +``` + +This manual pattern is used frequently today for plain old ABI-compatible library evolution, however is fairly manual and increasingly annoying to use as more and more APIs become deprecated and parameters are added. It also means we are unable to use Swift's default argument values, and have to manually provide the default values at call-sites instead. + +Instead, we are interested in extending the `@available` annotation's capabilities to be able to apply to method arguments, like this: + +```swift +distributed func greet( + name: String, + @available(macOS 12.1, *) in language: Language = .defaultLanguage) { + print("\(language.greeting), name!") +} + +// compiler synthesized: +// // "Old" API, delegating to `greet(name:in:)` +// distributed func greet(name: String) { +// self.greet(name: name, in: .defaultLanguage) +// } +``` + +This functionality would address both ABI stable library development, and `distributed` method evolution, because effectively they share the same concern -- the need to introduce new parameters, without breaking old API. For distributed methods specifically, this would cause the emission of metadata and thunks, such that the method `greet(name:)` can be resolved from an incoming message from an "old" peer, while the actual local invocation is performed on `greet(name:in:)`. + +Similar to many other runtimes, removing parameters is not going to be supported, however we could look into automatically handling optional parameters, defaulting them to `nil` if not present incoming messages. + +In order to serve distribution well, we might have to extend what notion of "platform" is allowed in the available annotation, because these may not necessarily be specific to "OS versions" but rather "version of the distributed system cluster", which can be simply sem-ver numbers that are known to the cluster runtime: + +```swift +distributed func greet( + name: String, + @available(distributed(cluster) 1.2.3, *) in language: Language = .defaultLanguage) { + print("\(language.greeting), name!") +} +``` + +During the initial handshake peers in a distributed system exchange information about their runtime version, and this can be used to inform method lookups, or even reject "too old" clients. + +## Introducing the `local` keyword + +It would be possible to expand the way distributed actors can conform to protocols which are intended only for the actor's "local side" if we introduced a `local` keyword. It would be used to taint distributed actor variables as well as functions in protocols with a local bias. + +For example, `local` marked distributed actor variables could simplify the following (surprisingly common in some situations!) pattern: + +```swift +distributed actor GameHost { + let myself: local Player + let others: [Player] + + init(system: GameSystem) { + self.myself = Player(system: GameSystem) + self.others = [] + } + + distributed func playerJoined(_ player: Player) { + others.append(player) + if others.count >= 2 { // we need 2 other players to start a game + self.start() + } + } + + func start() { + // start the game somehow, inform the local and all remote players + // ... + // Since we know `myself` is local, we can send it a closure with some logic + // (or other non-serializable data, like a connection etc), without having to use the whenLocal trick. + myself.onReceiveMessage { ... game logic here ... } + } +} +``` + +The above example makes use of the `myself: local Player` stored property, which propagates the knowledge that the player instance stored in this property *definitely* is local, and therefore we can call non-distributed methods on it, which is useful when we need to pass it closures or other non-serializable state -- as we do in the `start()` method. + +An `isolated Player` where Player is a `distributed actor` would also automatically be known to be `local`, and the `whenLocal` function could be expressed more efficiently (without needing to hop to the target actor at all): + +```swift +// WITHOUT `local`: +// extension DistributedActor { +// public nonisolated func whenLocal(_ body: @Sendable (isolated Self) async throws -> T) +// async rethrows -> T? where T: Sendable + +// WITH local, we're able to not "hop" when not necessary: +extension DistributedActor { + public nonisolated func whenLocal(_ body: @Sendable (local Self) async throws -> T) + reasync rethrows -> T? where T: Sendable // note the reasync (!) +} +``` + +This version of the `whenLocal` API is more powerful, since it would allow accessing actor state without hops, if we extended the model to allow this. This would allow treating `local AnyDistributedActor` the same way as we treat any local-only actor, and can be very useful in testing. + +We would not have to wrap APIs in `whenLocal` or provide wrapper APIs that are `nonisolated` but actually invoke things on self, like this real problem example, from implementing a Cluster "receptionist" actor where certain calls shall only be made by the "local side", however the entire actor is accessible remotely for other peers to communicate with: + +```swift +distributed actor Receptionist { + distributed func receiveGossip(...) { ... } + + // only to be invoked by "local" actors + func registerLocalActor(actor: Act) where Act: DistributedActor { ... } +} +``` + +Since it is too annoying to tell end-users to "always use `whenLocal` to invoke the local receptionist", library developers are forced to provide the following wrapper: + +```swift +extension Receptionist { + + // annoying forwarder/wrapper func; potentially unsafe, intended only for local use. + nonisolated func register(actor: Act) async where Act: DistributedActor { + await self.whenLocal { myself in + myself.registerLocalActor(actor: actor) + } else: { + fatalError("\(#function) must only be called on the local receptionist!") + } + } +} + +// ------------------------------------ +final class System: DistributedActorSystem { + // ... + let receptionist: Receptionist +} + +distributed actor Worker { + init(system: System) async { + receptionist.register(self) // ✅ OK + } +} +``` + +This mostly works, but the implementation of the `nonisolated func register` leaves much to be desired. Rather, we want to express the following: + +```swift +final class System: DistributedActorSystem { + // ... + let receptionist: local Receptionist +} + +distributed actor Worker { + init(system: System) async { + await receptionist.registerLocalActor(self) // ✅ OK + } +} +``` + +Without the need of manually implementing the "discard the distributed nature" of such actors. + +We see this as a natural follow up and future direction, which may take a while to implement, but would vastly improve the ergonomics of distributed actors in those special yet common enough few cases where such actors make an appearance. + +## Alternatives Considered + +This section summarizes various points in the design space for this proposal that have been considered, but ultimately rejected from this proposal. + +### Implicitly `distributed` methods / "opt-out of distribution" + +After initial feedback that `distributed func` seems to be "noisy", we actively explored the idea of alternative approaches which would reduce this perceived noise. We are convinced that implicitly distributed functions are a bad idea for the overall design, understandability, footprint and auditability of systems expressed using distributed actors. + +A promising idea, described by Pavel Yaskevich in the [Pitch #1](https://forums.swift.org/t/pitch-distributed-actors/51669/129) thread, was to inverse the rule, and say that _all_ functions declared on distributed actors are `distributed` by default (except `private` functions), and introduce a `local` keyword to opt-out from the distributed nature of actors. This listing exemplifies the idea: + +```swift +distributed actor Worker { + func work(on: Item) {} // "implicitly distributed" + private func actualWork() {} // not distributed + + local func shouldWork(on item: Item) -> Bool { ... } // NOT distributed +} +``` + +However, this turns out to complicate the understanding of such a system rather than simplify it. + +[1] We performed an analysis of a real distributed actor runtime (that we [open sourced recently](https://swift.org/blog/distributed-actors/)), and noticed that complex distributed actors have by far more non-distributed functions, than distributed ones. It is typical for a single distributed function, to invoke multiple non distributed functions in the same actor - simply because good programming style causes the splitting out of small pieces of logic into small functions with good names; Special care would have to be taken to mark those methods local. It is easy to forget doing so, since it is not a natural concept anywhere else in Swift to have to mark things "local" -- everything else is local after all. + +For example, the [distributed actor cluster implementation](https://github.com/apple/swift-distributed-actors) has a few very complex actors, and their sizes are more or less as follows: + +- ClusterShell - a very complex actor, orchestrating node connections etc. + - 14 distributed methods (it's a very large and crucial actor for the actor system) + - ~25 local methods +- SWIMShell, thee actor orchestrating the SWIM failure detection mechanism, + - 5 distributed methods + - 1 public local-only methods used by local callers + - ~12 local methods + +- ClusterReceptionist, responsible for discovering and gossiping information about actors + - 2 distributed methods + - 3 public local-only methods + - ~30 internal and private methods (lots of small helpers) + +- NodeDeathWatcher, responsible for monitoring node downing, and issuing associated actor termination events, + - 5 distributed functions + - no local-only methods + +[2] We are concerned about the auditability and review-ability of implicit distributed methods. In a plain text review it is not possible to determine whether the following introduces a distributed entry point or not. Consider the following diff, that one might be reviewing when another teammate submits a pull request: + +```swift ++ extension Worker { ++ func runShell(cmd: String) { // did this add a remotely invocable endpoint? we don't know from this patch! ++ // execute in shell ++ } ++ } +``` + +Under implicit `distributed func` rules, it is impossible to know if this function is possible to be invoked remotely. And if it were so, it could be a potential exploitation vector. Of course transports do and will implement their own authentication and authorization mechanisms, however nevertheless the inability to know if we just added a remotely invokable endpoint is worrying. + +In order to know if we just introduced a scary security hole in our system, we would have to go to the `Worker` definition and check if it was an `actor` or `distributed actor`. + +The accidental exposing can have other, unintended, side effects such as the following declaration of a method which is intended only for the actor itself to invoke it when some timer tick is triggered: + +```swift +// inside some distributed actor +func onPeriodicAckTick() { ... } +``` + +The method is not declared `private`, because in tests we want to be able to trigger the ticks manually. Under the implicit `distributeed func` rule, we would have to remember to make it local, as otherwise we accidentally made a function that is only intended for our own timers as remotely invocable, which could be misunderstood and/or be abused by either mistake, or malicious callers. + +Effectively, the implicitly-distributed rule causes more cognitive overhead to developers, every time having to mark and think about local only functions, rather than only think about the few times they actively want to _expose_ methods. + +[3] We initially thought we could delay additional type checks of implicit distributed functions until their first use. This would be similar to `Sendable` checking, where one can define a function accepting not-Sendable values, and only once it is attempted to be used in a cross-actor situation, we get compile errors. + +With distribution this poses a problem though: For example, should we allow the following conformance: + +```swift +struct Item {} // NOT Codable + +protocol Builder { + func build(_: Item) async throws +} + +distributed actor Bob: Builder { + typealias SerializationRequirement = Codable + func build(_: Item) async throws { ... } +} +``` + +Under implicit distributed rules, we should treat this function as distributed, however that means we should be checking `Item` for the `Codable` conformance. We know at declaration time that this conformance is faulty. While in theory we could delay the error until someone actually invoked the build function: + +``` swift +let bob: Bob +try await bob.build(Item()) // ❌ error: parameter type 'Item' does not conform to 'Bob.SerializationRequirement' +``` + +so we have declared a method that is impossible to invoke... however if we attempted to erase `Bob` to `Builder`... + +```swift +let builder: Builder = bob +try await builder.build(Item()) +``` + +there is nothing preventing this call from happening. There is no good way for the runtime to handle this; We would have to invent some defensive throwing modes, throwing in the distributed remote thunk, if the passed parameters do not pass what the type-system should have prevented from happening. + +In other words, the Sendable-like conformance model invites problematic cases which may lead to unsoundness. + +Thus, the only type-checking model of distributed functions, implicit or not, is an eager one. Where we fail during type checking immediately as we see the illegal declaration: + +```swift +struct Item {} // NOT Codable + +protocol Builder { + func build(_: Item) async throws +} + +distributed actor Bob: Builder { + typealias SerializationRequirement = Codable + func build(_: Item) async throws { ... } + // ❌ error: function 'build(_:)' cannot be used to satisfy protocol requirement + // ❌ error: parameter type 'Item' does not conform to 'Bob.SerializationRequirement' +} +``` + +By itself this is fine, however this has a painful effect on common programming patterns in Swift, where we are encouraged to extract small meaningful functions that are re-used in places by the actor. We are forced to annotate _more_ APIs as `local` than we would have been with the _explicit_ `distributed` annotation model (see observation that real world distributed actors often have many small functions, not intended for distribution) + +[4] Since almost all functions are distributed by default in the implicit model, we need to create and store metadata for all of them, regardless if they are used or not. This may cause unnecessary binary size growth, and seems somewhat backwards to Swift's approach to be efficient and minimal in metadata produced. + +We are aware of runtimes where every byte counts, and would not want to prevent them from adopting distributed actors for fear of causing accidental binary size growth. In practice, we would force developers to always write `local func` unless proven that it needs to be distributed, then removing the keyword – this model feels backwards from the explicit distributed marking model, in which we make a conscious decision that "yes, this function is intended for distribution" and mark it as `distributed func` only once we actively need to. + +[5] While it may seem simplistic, an effective method for auditing a distributed "attack surface" of a distributed actor system is enabled by the ability search the codebase for `distributed func` and make sure all functions perform the expected authorization checks. These functions are as important as "service endpoints" and should be treated with extra care. This only works when distributed functions are explicit. + +We should also invest in transport-level authentication and authorization techniques, however some actions are going to be checked action-per-action, so this additional help of quickly locating distributed functions is a feature, not an annoyance. + +Summing up, the primary benefit of the implicit `distributed func` rule was to attempt to save developers a few keystrokes, however it fails to deliver this in practice because frequently (verified by empirical data) actors have many local methods which they do not want to expose as well. The implicit rule makes these more verbose, and results in more additional annotations. Not only that, but it causes greater mental overhead for having to remember if we're in the context of a distributed actor, and if a `func` didn't just accidentally get exposed as remotely accessible endpoint. We also noticed a few soundness and additional complexity in regard to protocol conformances that we found quite tricky. + +We gave this alternative design idea significant thought and strongly favor the explicit distributed rule. + +### Declaring actors and methods as "`distributable`" + +Naming of distributed actors has been debated and while it is true that `distributed` means "may be distributed (meaning 'remote') or not", this is not really the mindset we want to promote with distributed actors. The mental mindset should be that these are distributed and we must treat them this way, and they may happen to be local. Locality is the special case, distribution is the capability we're working with while designing location transparent actors. While we do envision the use of "known to be local" distributed actors, this is better solved with either a `worker.whenLocal { ...` API or allowing marking types with a `local` keyword - either approaches are not part of this proposal and will be pitched in dependently. + +The `distributed` keyword functions the same way as `async` on methods. Async methods are not always asynchronous. The `async` keyword merely means that such method _may suspend_. Similarly, a `distributed func` may or may not perform a remote call, as such the semantics follow the same "beware, the more expensive thing may happen" style of marking methods. + +### Unconditionally conforming `DistributedActor` to `Codable` + +This was part of an earlier design, where the distributed actor protocol was declared as: + +```swift +protocol DistributedActor: AnyActor, Sendable, Codable, ... { ... } +``` + +forcing all implementations of distributed actors to implement the Codable `init(from:)` initializer and `encode(to:)` method. + +While we indeed to expect `Codable` to play a large role in some distributed actor implementations, we have specific use-cases in mind where: + +- Codable might not be used _at all_, thus the re-design and strong focus on being serialization mechanism agnostic in the proposal, by introducing the `SerializationRequirement` associated type. +- Some distributed actor runtimes may behave more like "services" which are _not_ meant to be "passed around" to other nodes. This capability has been explicitly requested by some early adopters in IPC scenarios, where it will help to clean up vague and hacky solutions today, with a clear model where some distributed actors are Codable and thus "pass-around-able" and some are not, depending on the specifics how they were created. + +As such, we are left with no other implementation approach other than the implicit conformance, because it is not possible to add the `Codable` conformance to types managed by a distributed actor system that _wants to_ make distributed actors Codable otherwise (i.e. it is not possible to express `extension DistributedActor: Codable where ID: Codable {}` in today's Swift). Alternative approaches force implementations into casting and doing unsafe tricksy and lose out on the type-safety of only passing Codable actors to distributed methods. + +For distributed actor systems which _do not_ use `Codable`, forcing them to implement Codable methods and initializers would be quite a problem and the implementations would likely be implemented as just crashing. Implementations may force actors to conform to some other protocol, like `IPCServiceDistributedActor` which conforms to the `SerializationRequirement` and attempts to initialize an actor which does not conform to this protocol can crash eagerly, at initialization time. This way actor system authors gain the same developer experience as using `Codable` for passing distributed actors through distributed methods, but the initialization can be specialized -- as it is intended to, because libraries may require specific things from actor types after all. + +### Introducing "wrapper" type for `Distributed` + +We did consider (and have implemented, assisted by swift-syntax based source-generation) the idea of wrapping distributed actors using some "wrapper" type, that would delegate calls to all distributed functions, but prevent access to e.g. stored properties wrapped by such instance. + +This loses the benefit that a proper nominal type distributed actor offers though: the easy to incrementally move actors to distribution as it becomes necessary. The complexity of forming the "call forwarding" functions is also problematic, and extensions to such types would be confusing, would we have to do extensions like this? + +```swift +extension Distributed where Actor == SomeActor { + func hi() { ... } +} +``` + +while _also_ forwarding to functions extended on the `SomeActor` itself? + +```swift +extension SomeActor { + func hi() { ... } // conflict? +} +``` + +What would that mean for when we try to call `hi()` on a distributed actor? It also does not really simplify testing, as we want to test the actual actor, but also the distributed functions actually working correctly (i.e. enforcing serialization constraints on parameters). + +### Creating only a library and/or source-generation tool + +While this may be a highly subjective and sensitive topic, we want to tackle the question up-front, so why are distributed actors better than "just" some RPC library? + +The answer lies in the language integration and the mental model developers can work with when working with distributed actors. Swift already embraces actors for its local concurrency programming, and they will be omni-present and become a familiar and useful tool for developers. It is also important to notice that any async function may be technically performing work over network, and it is up to developers to manage such calls in order to not overwhelm the network etc. With distributed actors, such calls are more _visible_ because IDEs have the necessary information to e.g. underline or otherwise highlight that a function is likely to hit the network and one may need to consider its latency more, than if it was just a local call. IDEs and linters can even use this statically available information to write hints such as "hey, you're doing this distributed actor call in a tight loop - are you sure you want to do that?" + +Distributed actors, unlike "raw" RPC frameworks, help developers think about their distributed applications in terms of a network of collaborating actors, rather than having to think and carefully manage every single serialization call and network connection management between many connected peers - which we envision to be more and more important in the future of device and server programming et al. You may also refer to the [Swift Concurrency Manifesto; Part 4: Improving system architecture](https://gist.github.com/lattner/31ed37682ef1576b16bca1432ea9f782#part-4-improving-system-architecture) section for some other ideas on the topic. + +This does _not_ mean that we shun RPC style libraries or plain-old HTTP clients and libraries similar to them, which may rather be expressed as non-actor types with asynchronous functions. They still absolutely have their place, and we do not envision distributed actors fully replacing them - they are fantastic for cross-language communication, however distributed actors offer a vastly superior programming model, while we remain mostly within Swift and associated actor implementations (we *could*, communicate with non-swift actors over the network, however have not invested into this yet). We do mean however that extending the actor model to its natural habitat (networking) will enable developers to build some kinds of interactive multi-peer/multi-node systems far more naturally than each time having to re-invent a similar abstraction layer, never quite reaching the integration smoothness as language provided integration points such as distributed actors can offer. + +## Acknowledgments & Prior Art + +We would like to acknowledge the prior art in the space of distributed actor systems which have inspired our design and thinking over the years. Most notably we would like to thank the Akka and Orleans projects, each showing independent innovation in their respective ecosystems and implementation approaches. As these are library-only solutions, they have to rely on wrapper types to perform the hiding of information, and/or source generation; we achieve the same goal by expanding the already present in Swift actor-isolation checking mechanisms. + +We would also like to acknowledge the Erlang BEAM runtime and Elixir language for a more modern take built upon the on the same foundations, which have greatly inspired our design, however take a very different approach to actor isolation (i.e. complete isolation, including separate heaps for actors). + +## Source compatibility + +This change is purely additive to the source language. + +The additional use of the keyword `distributed` in `distributed actor` and `distributed func` applies more restrictive requirements to the use of such an actor, however this only applies to new code, as such no existing code is impacted. + +Marking an actor as distributed when it previously was not is potentially source-breaking, as it adds additional type checking requirements to the type. + +## Effect on ABI stability + +None. + +## Effect on API resilience + +None. + +## Changelog + +- 1.3.1 Minor cleanups + - Allow `private distributed func` + - Allow generic distributed actor declarations +- 1.3 More about serialization typechecking and introducing mentioned protocols explicitly + - Revisions Introduce `DistributedActor` and `DistributedActorSystem` protocols properly + - Discuss future directions for versioning and evolving APIs + - Introduce conditional Codable conformance of distributed actors, based on ID + - Discuss `SerializationRequirement` driven typechecking of distributed methods + - Discuss `DistributedActorSystem` parameter requirement in required initializers + - Discuss isolation states in depth "isolated", "known to be local", "potentially remote" and their effect on implicit effects on call-sites +- 1.2 Drop implicitly distributed methods +- 1.1 Implicitly distributed methods +- 1.0 Initial revision +- [Pitch: Distributed Actors](https://forums.swift.org/t/pitch-distributed-actors/51669) + - Which focused on the general concept of distributed actors, and will from here on be cut up in smaller, reviewable pieces that will become their own independent proposals; Similar to how Swift Concurrency is a single coherent feature, however was introduced throughout many interconnected Swift Evolution proposals. diff --git a/proposals/0337-support-incremental-migration-to-concurrency-checking.md b/proposals/0337-support-incremental-migration-to-concurrency-checking.md new file mode 100644 index 0000000000..f3804f5a8f --- /dev/null +++ b/proposals/0337-support-incremental-migration-to-concurrency-checking.md @@ -0,0 +1,286 @@ +# Incremental migration to concurrency checking + +* Proposal: [SE-0337](0337-support-incremental-migration-to-concurrency-checking.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Ben Cohen](https://github.com/AirspeedSwift) +* Status: **Implemented (Swift 5.6)** +* Upcoming Feature Flag: `StrictConcurrency` (Implemented in Swift 6.0) (Enabled in Swift 6 language mode) +* Implementation: [Pull request](https://github.com/apple/swift/pull/40680), [Linux toolchain](https://ci.swift.org/job/swift-PR-toolchain-Linux/761//artifact/branch-main/swift-PR-40680-761-ubuntu16.04.tar.gz), [macOS toolchain](https://ci.swift.org/job/swift-PR-toolchain-osx/1256//artifact/branch-main/swift-PR-40680-1256-osx.tar.gz) + +## Introduction + +Swift 5.5 introduced mechanisms to eliminate data races from the language, including the `Sendable` protocol ([SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md)) to indicate which types have values that can safely be used across task and actor boundaries, and global actors ([SE-0316](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0316-global-actors.md)) to help ensure proper synchronization with (e.g.) the main actor. However, Swift 5.5 does not fully enforce `Sendable` nor all uses of the main actor because interacting with modules which have not been updated for Swift Concurrency was found to be too onerous. We propose adding features to help developers migrate their code to support concurrency and interoperate with other modules that have not yet adopted it, providing a smooth path for the Swift ecosystem to eliminate data races. + +Swift-evolution threads: [[Pitch] Staging in `Sendable` checking](https://forums.swift.org/t/pitch-staging-in-sendable-checking/51341), [Pitch #2](https://forums.swift.org/t/pitch-2-staging-in-sendable-checking/52413), [Pitch #3](https://forums.swift.org/t/pitch-3-incremental-migration-to-concurrency-checking/53610) + +## Motivation + +Swift Concurrency seeks to provide a mechanism for isolating state in concurrent programs to eliminate data races. The primary mechanism is `Sendable` checking. APIs which send data across task or actor boundaries require their inputs to conform to the `Sendable` protocol; types which are safe to send declare conformance, and the compiler checks that these types only contain `Sendable` types, unless the type's author explicitly indicates that the type is implemented so that it uses any un-`Sendable` contents safely. + +This would all be well and good if we were writing Swift 1, a brand-new language which did not need to interoperate with any existing code. Instead, we are writing Swift 6, a new version of an existing language with millions of lines of existing libraries and deep interoperation with C and Objective-C. None of this code specifies any of its concurrency behavior in a way that `Sendable` checking can understand, but until it can be updated, we still want to use it from Swift. + +There are several areas where we wish to address adoption difficulties. + +### Adding retroactive concurrency annotations to libraries + +Many existing APIs should be updated to formally specify concurrency behavior that they have always followed, but have not been able to describe to the compiler until now. For instance, it has always been the case that most UIKit methods and properties should only be used on the main thread, but before the `@MainActor` attribute, this behavior could only be documented and asserted in the implementation, not described to the compiler. + +Thus, many modules should undertake a comprehensive audit of their APIs to decide where to add concurrency annotations. But if they try to do so with the tools they currently have, this will surely cause source breaks. For instance, if a method is marked `@MainActor`, projects which have not yet adopted Swift Concurrency will be unable to call it even if they are using it correctly, because the project does not yet have the annotations to *prove to the compiler* that the call will run in the main actor. + +In some cases, these changes can even cause ABI breaks. For instance, `@Sendable` attributes on function types and `Sendable` constraints on generic parameters are incorporated into mangled function names, even though `Sendable` conformances otherwise have no impact on the calling convention (there isn't an extra witness table parameter, for instance). A mechanism is needed to enforce these constraints during typechecking, but generate code as though they do not exist. + +Here, we need: + +* A formal specification of a "compatibility mode" for pre-concurrency code which imports post-concurrency modules + +* A way to mark declarations as needing special treatment in this "compatibility mode" because their signatures were changed for concurrency + +### Adopting `Sendable` checking before the modules you use have been updated + +The process of auditing libraries to add concurrency annotations will take a long time. We don't think it's realistic for each module to wait until all of its libraries have been updated before they can start adopting `Sendable` checking. + +This means modules need a way to work around incomplete annotations in their imports--either by tweaking the specifications of imported declarations, or by telling the compiler to ignore errors. Whatever mechanism we use, we don't want it to be too verbose, though; for example, marking every single variable of a non-`Sendable` type which we want to treat as `Sendable` would be pretty painful. + +We must also pay special attention to what happens when the library finally *does* add its concurrency annotations, and they reveal that a client has made a mistaken assumption about its concurrency behavior. For instance, suppose you import type `Point` from module `Geometry`. You enable `Sendable` checking before `Geometry`'s maintainers have added concurrency annotations, so it diagnoses a call that sends a `Point` to a different actor. Based on the publicly-known information about `Point`, you decide that this type is probably `Sendable`, so you silence this diagnostic. However, `Geometry`'s maintainers later examine the implementation of `Point` and determine that it is *not* safe to send, so they mark it as non-`Sendable`. What should happen when you get the updated version of `Geometry` and rebuild your project? + +Ideally, Swift should not continue to suppress the diagnostic about this bug. After all, the `Geometry` team has now marked the type as non-`Sendable`, and that is more definitive than your guess that it would be `Sendable`. On the other hand, it probably shouldn't *prevent* you from rebuilding your project either, because this bug is not a regression. The updated `Geometry` module did not add a bug to your code; your code was already buggy. It merely *revealed* that your code was buggy. That's an improvement on the status quo--a diagnosed bug is better than a hidden one. + +But if Swift reacts to this bug's discovery by preventing you from building a module that built fine yesterday, you might have to put off updating the `Geometry` module or even pressure `Geometry`'s maintainers to delay their update until you can fix it, slowing forward progress. So when your module assumes something about an imported declaration that is later proven to be incorrect, Swift should emit a *warning*, not an error, about the bug, so that you know about the bug but do not have to correct it just to make your project build again. + +Here, we need: + +* A mechanism to silence diagnostics about missing concurrency annotations related to a particular declaration or module + +* Rules which cause those diagnostics to return once concurrency annotations have been added, but only as warnings, not errors + +## Proposed solution + +We propose a suite of features to aid in the adoption of concurrency annotations, especially `Sendable` checking. These features are designed to enable the following workflow for adopting concurrency checking: + +1. Enable concurrency checking, by adopting concurrency features (such as `async/await` or actors), enabling Swift 6 mode, or adding the `-warn-concurrency` flag. This causes new errors or warnings to appear when concurrency constraints are violated. + +2. Start solving those problems. If they relate to types from another module, a fix-it will suggest using a special kind of import, `@preconcurrency import`, which silences these warnings. + +3. Once you've solved these problems, integrate your changes into the larger build. + +4. At some future point, a module you import may be updated to add `Sendable` conformances and other concurrency annotations. If it is, and your code violates the new constraints, you will see warnings telling you about these mistakes; these are latent concurrency bugs in your code. Correct them. + +5. Once you've fixed those bugs, or if there aren't any, you will see a warning telling you that the `@preconcurrency import` is unnecessary. Remove the `@preconcurrency` attribute. Any `Sendable`-checking failures involving that module from that point forward will not suggest using `@preconcurrency import` and, in Swift 6 mode, will be errors that prevent your project from building. + +Achieving this will require several features working in tandem: + +* In Swift 6 mode, all code will be checked completely for missing `Sendable` conformances and other concurrency violations, with mistakes generally diagnosed as errors. The `-warn-concurrency` flag will diagnose these violations as warnings in older language versions. + +* When applied to a nominal declaration, the `@preconcurrency` attribute specifies that a declaration was modified to update it for concurrency checking, so the compiler should allow some uses in Swift 5 mode that violate concurrency checking, and generate code that interoperates with pre-concurrency binaries. + +* When applied to an `import` statement, the `@preconcurrency` attribute tells the compiler that it should only diagnose `Sendable`-requiring uses of non-`Sendable` types from that module if the type explicitly declares a `Sendable` conformance that is unavailable or has constraints that are not satisfied; even then, this will only be a warning, not an error. + + +## Detailed design + +### Recovery behavior + +When this proposal speaks of an error being emitted as a warning or suppressed, it means that the compiler will recover by behaving as though (in order of preference): + +* A nominal type that does not conform to `Sendable` does. + +* A function type with an `@Sendable` or global actor attribute doesn't have it. + +### Concurrency checking modes + +Every scope in Swift can be described as having one of two "concurrency checking modes": + +* **Strict concurrency checking**: Missing `Sendable` conformances or global-actor annotations are diagnosed. In Swift 6, these will generally be errors; in Swift 5 mode and with nominal declarations visible via `@preconcurrency import` (defined below), these diagnostics will be warnings. + +* **Minimal concurrency checking**: Missing `Sendable` conformances or global-actor annotations are diagnosed as warnings; on nominal declarations, `@preconcurrency` (defined below) has special effects in this mode which suppress many diagnostics. + +The top level scope's concurrency checking mode is: + +* **Strict** when the module is being compiled in Swift 6 mode or later, when the `-warn-concurrency` flag is used with an earlier language mode, or when the file being parsed is a module interface. + +* **Minimal** otherwise. + +A child scope's concurrency checking mode is: + +* **Strict** if the parent's concurrency checking mode is **Minimal** and any of the following conditions is true of the child scope: + + * It is a closure with an explicit global actor attribute. + + * It is a closure or autoclosure whose type is `async` or `@Sendable`. (Note that the fact that the parent scope is in Minimal mode may affect whether the closure's type is inferred to be `@Sendable`.) + + * It is a declaration with an explicit `nonisolated` or global actor attribute. + + * It is a function, method, initializer, accessor, variable, or subscript which is marked `async` or `@Sendable`. + + * It is an `actor` declaration. + +* Otherwise, the same as the parent scope's. + +> Implementation note: The logic for determining whether a child scope is in Minimal or Strict mode is currently implemented in `swift::contextRequiresStrictConcurrencyChecking()`. + +Imported C declarations belong to a scope with Minimal concurrency checking. + +### `@preconcurrency` attribute on nominal declarations + +To describe their concurrency behavior, maintainers must change some existing declarations in ways which, by themselves, could be source-breaking in pre-concurrency code or ABI-breaking when interoperating with previously-compiled binaries. In particular, they may need to: + +* Add `@Sendable` or global actor attributes to function types +* Add `Sendable` constraints to generic signatures +* Add global actor attributes to declarations + +When applied to a nominal declaration, the `@preconcurrency` attribute indicates that a declaration existed before the module it belongs to fully adopted concurrency, so the compiler should take steps to avoid these source and ABI breaks. It can be applied to any `enum`, enum `case`, `struct`, `class`, `actor`, `protocol`, `var`, `let`, `subscript`, `init` or `func` declaration. + +When a nominal declaration uses `@preconcurrency`: + +* Its name is mangled as though it does not use any of the listed features. + +* At use sites whose enclosing scope uses Minimal concurrency checking, the compiler will suppress any diagnostics about mismatches in these traits. + +* The ABI checker will remove any use of these features when it produces its digests. + +Objective-C declarations are always imported as though they were annotated with `@preconcurrency`. + +For example, consider a function that can only be called on the main actor, then runs the provided closure on a different task: + +```swift +@MainActor func doSomethingThenFollowUp(_ body: @Sendable () -> Void) { + // do something + Task.detached { + // do something else + body() + } +} +``` + +This function could have existed before concurrency, without the `@MainActor` and `@Sendable` annotations. After adding these concurrency annotations, code that worked previously would start producing errors: + +```swift +class MyButton { + var clickedCount = 0 + + func onClicked() { // always called on the main thread by the system + doSomethingThenFollowUp { // ERROR: cannot call @MainActor function outside the main actor + clickedCount += 1 // ERROR: captured 'self' with non-Sendable type `MyButton` in @Sendable closure + } + } +} +``` + +However, if we add `@preconcurrency` to the declaration of `doSomethingThenFollowUp`, its type is adjusted to remove both the `@MainActor` and the `@Sendable`, eliminating the errors and providing the same type inference from before concurrency was adopted by `doSomethingThenFollowUp`. The difference is visible in the type of `doSomethingThenFollowUp` in a minimal vs. a strict context: + +```swift +func minimal() { + let fn = doSomethingThenFollowUp // type is (( )-> Void) -> Void +} + +func strict() async { + let fn = doSomethingThenFollowUp // type is @MainActor (@Sendable ( )-> Void) -> Void +} +``` + +### `Sendable` conformance status + +A type can be described as having one of the following three `Sendable` conformance statuses: + +* **Explicitly `Sendable`** if it actually conforms to `Sendable`, whether via explicit declaration or because the `Sendable` conformance was inferred based on the rules specified in [SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md). + +* **Explicitly non-`Sendable`** if a `Sendable` conformance has been declared for the type, but it is not available or has constraints the type does not satisfy, *or* if the type was declared in a scope that uses Strict concurrency checking.[2] + +* **Implicitly non-`Sendable`** if no `Sendable` conformance has been declared on this type at all. + +> [2] This means that, if a module is compiled with Swift 6 mode or the `-warn-concurrency` flag, all of its types are either explicitly `Sendable` or explicitly non-`Sendable`. + +A type can be made explicitly non-`Sendable` by creating an unavailable conformance to `Sendable`, e.g., + +```swift +@available(*, unavailable) +extension Point: Sendable { } +``` + +Such a conformance suppresses the implicit conformance of a type to `Sendable`. + +### `@preconcurrency` on `Sendable` protocols + +Some number of existing protocols describe types that should all be `Sendable`. When such protocols are updated for concurrency, they will likely inherit from the `Sendable` protocol. However, doing so will break existing types that conform to the protocol and are now assumed to be `Sendable`. This problem was [described in SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md#thrown-errors) because it affects the `Error` and `CodingKey` protocols from the standard library: + +```swift +protocol Error: /* newly added */ Sendable { ... } + +class MutableStorage { + var counter: Int +} +struct ProblematicError: Error { + var storage: MutableStorage // error: Sendable struct ProblematicError has non-Sendable stored property of type MutableStorage +} +``` + +To address this, SE-0302 says the following about the additional of `Sendable` to the `Error` protocol: + +> To ease the transition, errors about types that get their `Sendable` conformances through `Error` will be downgraded to warnings in Swift < 6. + +We propose to replace this bespoke rule for `Error` and `CodingKey` to apply to every protocol that is annotated with `@preconcurrency` and inherits from `Sendable`. These two standard-library protocols will use `@preconcurrency`: + +```swift +@preconcurrency protocol Error: Sendable { ... } +@preconcurrency protocol CodingKey: Sendable { ... } +``` + +### `@preconcurrency` attribute on `import` declarations + +The `@preconcurrency` attribute can be applied to an `import` declaration to indicate that the compiler should reduce the strength of some concurrency-checking violations caused by types imported from that module. You can use it to import a module which has not yet been updated with concurrency annotations; if you do, the compiler will tell you when all of the types you need to be `Sendable` have been annotated. It also serves as a temporary escape hatch to keep your project compiling until any mistaken assumptions you had about that module are fixed. + +When an import is marked `@preconcurrency`, the following rules are in effect: + +* If an implicitly non-`Sendable` type is used where a `Sendable` type is needed: + + * If the type is visible through a `@preconcurrency import`, the diagnostic is suppressed (prior to Swift 6) or emitted as a warning (in Swift 6 and later). + + * Otherwise, the diagnostic is emitted normally, but a separate diagnostic is provided recommending that `@preconcurrency import` be used to work around the issue. + +* If an explicitly non-`Sendable` type is used where a `Sendable` type is needed: + + * If the type is visible through an `@preconcurrency import`, a warning is emitted instead of an error, even in Swift 6. + + * Otherwise, the diagnostic is emitted normally. + +* If the `@preconcurrency` attribute is unused[3], a warning will be emitted recommending that it be removed. + +> [3] We don't define "unused" more specifically because we aren't sure if we can refine it enough to, for instance, recommend removing one of a pair of `@preconcurrency` imports which both import an affected type. + +## Source compatibility + +This proposal is largely motivated by source compatibility concerns. Correct use of `@preconcurrency` should prevent source breaks in code built with Minimal concurrency checking, and `@preconcurrency import` temporarily weakens concurrency-checking rules to preserve source compatibility if a project adopts Full or Strict concurrency checking before its dependencies have finished adding concurrency annotations. + +## Effect on ABI stability + +By itself, `@preconcurrency` does not change the ABI of a declaration. If it is applied to declarations which have already adopted one of the features it affects, that will create an ABI break. However, if those features are added at the same time or after `@preconcurrency` is added, adding those features will *not* break ABI. + +`@preconcurrency`'s tactic of disabling `Sendable` conformance errors is compatible with the current ABI because `Sendable` was designed to not emit additional metadata, have a witness table that needs to be passed, or otherwise impact the calling convention or most other parts of the ABI. It only affects the name mangling. + +This proposal should not otherwise affect ABI. + +## Effect on API resilience + +`@preconcurrency` on nominal declarations will need to be printed into module interfaces. It is effectively a feature to allow the evolution of APIs in ways that would otherwise break resilience. + +`@preconcurrency` on `import` statements will not need to be printed into module interfaces; since module interfaces use the Strict concurrency checking mode, where concurrency diagnostics are warnings, they have enough "wiggle room" to tolerate the missing conformances. (As usual, compiling a module interface silences warnings by default.) + +## Alternatives considered + +### A "concurrency epoch" + +If the evolution of a given module is tied to a version that can be expressed in `@available`, it is likely that there will be some specific version where it retroactively adds concurrency annotations to its public APIs, and that thereafter any new APIs will be "born" with correct concurrency annotations. We could take advantage of this by allowing the module to specify a particular version when it started ensuring that new APIs were annotated and automatically applying `@preconcurrency` to APIs available before this cutoff. + +This would save maintainers from having to manually add `@preconcurrency` to many of the APIs they are retroactively updating. However, it would have a number of limitations: + +1. It would only be useful for modules used exclusively on Darwin. Non-Darwin or cross-platform modules would still need to add `@preconcurrency` manually. + +2. It would only be useful for modules which are version-locked with either Swift itself or a Darwin OS. Modules in the package ecosystem, for instance, would have little use for it. + +3. In practice, version numbers may be insufficiently granular for this task. For instance, if a new API is added at the beginning of a development cycle and it is updated for concurrency later in that cycle, you might mistakenly assume that it will automatically get `@preconcurrency` when in fact you will need to add it by hand. + +Since these shortcomings significantly reduce its applicability, and you only need to add `@preconcurrency` to declarations you are explicitly editing (so you are already very close to the place where you need to add it), we think a concurrency epoch is not worth the trouble. + +### Objective-C and `@preconcurrency` + +Because all Objective-C declarations are implicitly `@preconcurrency`, there is no way to force concurrency APIs to be checked in Minimal-mode code, even if they are new enough that there should be no violating uses. We think this limitation is acceptable to simplify the process of auditing large, existing Objective-C libraries. diff --git a/proposals/0338-clarify-execution-non-actor-async.md b/proposals/0338-clarify-execution-non-actor-async.md new file mode 100644 index 0000000000..d7d8d29aae --- /dev/null +++ b/proposals/0338-clarify-execution-non-actor-async.md @@ -0,0 +1,224 @@ +# Clarify the Execution of Non-Actor-Isolated Async Functions + +* Proposal: [SE-0338](0338-clarify-execution-non-actor-async.md) +* Author: [John McCall](https://github.com/rjmccall) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.7)** ([Decision notes](https://forums.swift.org/t/accepted-se-0338-clarify-the-execution-of-non-actor-isolated-async-functions/54929)) + +## Introduction + +[SE-0306](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md), which introduced actors to Swift, states that `async` functions may be actor-isolated, meaning that they formally run on some actor's executor. Nothing in either SE-0306 or [SE-0296](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md) (`async`/`await`) ever specifies where asynchronous functions that *aren't* actor-isolated run. This proposal clarifies that they do not run on any actor's executor, and it tightens up the rules for [sendability checking](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md) to avoid a potential data race. + +## Motivation + +It is sometimes important that programmers be able to understand which executor is formally responsible for running a particular piece of code. A function that does a large amount of computation on an actor's executor will prevent other tasks from making progress on that actor. The proper isolation of a value may also depend on only accessing it from a particular executor (see note below). Furthermore, in some situations the current executor has other semantic impacts, such as being "inherited" by tasks created with the `Task` initializer. Therefore, Swift needs to provide an intuitive and comprehensible rule for which executors are responsible for running which code. + +> Note: Swift will enforce the correct isolation of data by default with `Sendable` checking. However, this will not be fully enabled until code adopts a future language mode (probably Swift 6). Even under that mode, it will be possible to opt out using `@unsafe Sendable`, making safety the programmer's responsibility. And even if neither of those caveats were true, it would still be important for programmers to be able to understand the execution rules in order to understand how best to fix an isolation error. + +In the current implementation of Swift, `async` functions that aren't actor-isolated never intentionally change the current executor. That is, whenever execution enters such an `async` function, it will continue on whatever the current executor is, with no specific preference for any particular executor. + +To be slightly more precise, we can identify three principle ways that execution can enter an `async` function: + +- It's called by some other `async` function. +- It calls some other `async` function which then returns, resuming the caller. +- It needs to suspend for internal reasons (perhaps it uses `withContinuation` or calls a runtime function that suspends), and it is resumed after that suspension. + +In the current implementation, calls and returns from actor-isolated functions will continue running on that actor's executor. As a result, actors are effectively "sticky": once a task switches to an actor's executor, they will remain there until either the task suspends or it needs to run on a different actor. But if a task suspends within a non-actor-isolated function for a different reason than a call or return, it will generally resume on a non-actor executor. + +This rule perhaps makes sense from the perspective of minimizing switches between executors, but it has several unfortunate consequences. It can lead to unexpected "overhang", where an actor's executor continues to be tied up long after it was last truly needed. An actor's executor can be surprisingly inherited by tasks created during this overhang, leading to unnecessary serialization and contention for the actor. It also becomes unclear how to properly isolate data in such a function: some data accesses may be safe because of the executor the function happens to run on dynamically, but it is unlikely that this is guaranteed by the system. All told, it is a very dynamic rule which interacts poorly with how the rest of concurrency is generally understood, both by Swift programmers and statically by the Swift implementation. + +## Proposed solution + +`async` functions that are not actor-isolated should formally run on a generic executor associated with no actor. Such functions will formally switch executors exactly like an actor-isolated function would: on any entry to the function, including calls, returns from calls, and resumption from suspension, they will switch to a generic, non-actor executor. If they were previously running on some actor's executor, that executor will become free to execute other tasks. + +```swift +extension MyActor { + func update() async { + // This function is actor-isolated, so formally we switch to the actor. + // as soon as it is called. + + // Here we call a function which is not actor-isolated. + let update = await session.readConsistentUpdate() + + // Now we resume executing the function, so formally we switch back to + // the actor. + name = update.name + age = update.age + } +} + +extension MyNetworkSession { + func readConsistentUpdate() async -> Update { + // This function is not actor-isolated, so formally we switch to a + // generic executor when it's called. So if we happen to be called + // from an actor-isolated function, we will immediately switch off the + // actor here. + + // This code runs without any special isolation. + + // Keep calling readUpdate until it returns the same thing twice in a + // row. If that never happens in 1000 different calls, just return the + // last update. This code is just for explanatory purposes; please don't + // expect too much from it. + var update: Update? + for i in 0..<1000 { + // Here we make an async call. + let newUpdate = await readUpdateOnce() + + // Formally, we will switch back to the generic executor after the + // call, so if we happen to have called an actor-isolated function, + // we will immediately switch off of the actor here. + + if update == newUpdate { break } + update = newUpdate + } + return update! + } +} +``` + +## Detailed design + +This proposal changes the semantics of non-actor-isolated `async` functions by specifying that they behave as if they were running on a generic executor not associated with any actor. Technically, the current rule was never written down, so you could say that this proposal *sets* the semantics of these functions; in practice, though, this is an observable change in behavior. + +As a result of this change, the formal executor of an `async` function is always known statically: +- actor-isolated `async` functions always formally run on the actor's executor +- non-actor-isolated `async` functions never formally run on any actor's executor + +This change calls for tasks to switch executors at certain points: +- when the function is called +- when a call made by the function returns +- when the function returns from an internal suspension (e.g. due to a continuation) +As usual, these switches are subject to static and dynamic optimization. These optimizations are the same as are already done with switches to actor executors. + +Statically, if a non-actor-isolated async function doesn't do any significant work before returning, suspending, or making an async call, it can simply remain on the current executor and allow its caller, resumer, or callee to make whatever switches it feels are advisable. This is why this proposal is careful to talk about what executor is *formally* running the task: the actual executor is permitted to be different. Typically, this difference will not be observable, but there are some exceptions. For example, if a function makes two consecutive calls to the same actor, it's possible (but not guaranteed) that the actor will not be given up between them, preventing other work from interleaving. It is outside the scope of this proposal to define what work is "significant". + +Dynamically, a switch will not suspend the task if the task is already on an appropriate executor. Furthermore, some executor changes can be done cheaply without fully suspending the task by giving up the current thread. + +### Sendability + +The `Sendable` rule for calls to non-actor-isolated `async` functions is currently broken. This rule is closely tied to the execution semantics of these functions because of the role of sendability checking in proving the absence of data races. The `Sendable` rule is broken even under the current semantics, but it's arguably even more broken under the proposed rule, so we really do need to fix it as part of this proposal. (There is an alternative which would make the current rule correct, but it doesn't seem advisable; see "Alternatives Considered".) + +It is a basic goal of Swift concurrency that programs should be free of basic data races. In order to achieve this, we must be able to prove that all uses of certain values and memory are totally ordered. All of the code that runs on a particular task is totally ordered with respect to itself. Similarly, all of the code that runs on a particular actor is totally ordered with respect to itself. So, if we can restrict a value/memory to only be used by a single task or actor, we've proven that all of its uses are totally ordered. This is the immediate goal of sendability checking: it prevents non-`Sendable` values from being shared between different concurrent contexts and thus potentially being accessed in non-totally-ordered ways. + +For the purposes of sendability, the concurrent context of an actor-isolated `async` function is the actor. An actor can have non-`Sendable` values in its actor-isolated storage. Actor-isolated functions can read values from that storage into their local state, and similarly they can write values from their local state into actor-isolated storage. Therefore, such functions must strictly separate their "internal" local state from the "external" local state of the task. (It would be possible to be more lenient here, but that is outside the scope of this proposal.) + +The current sendability rule for `async` calls is that the arguments and results of calls to actor-isolated `async` functions must be `Sendable` unless the callee is known to be isolated to the same actor as the caller. Unfortunately, no such restriction is placed on calls to non-isolated `async` functions. That is incorrect under both the current and the proposed execution semantics of such functions because the local state of such functions is not strictly isolated to the actor. + +As a result, the following is allowed: + +```swift +actor MyActor { + var isolated: NonSendableValue + + // Imagine that there are two different tasks calling these two + // functions, and the actor runs the task for `inside_one()` first. + + func inside_one() async { + await outside(argument: isolated) + } + + func inside_two() async { + isolated.operate() + } +} + +// This is a non-actor-isolated async function. +func outside(argument: NonSendableValue) async { + // Under the current execution semantics, when we resume from this + // sleep, we will not be on the actor's executor anymore. + // Under the proposed execution semantics, we will leave the actor's + // executor even before sleeping. + await Task.sleep(nanoseconds: 1_000) + + // In either case, this use of the non-Sendable value can now happen + // concurrently with a use of it on the actor. + argument.operate() +} +``` + +The sendability rule for `async` calls must be changed: the arguments and results of *all* `async` calls must be `Sendable` unless: +- the caller and callee are both known to be isolated to the same actor, or +- the caller and callee are both known to be non-actor-isolated. + +## Source compatibility + +The change to the execution semantics will not break source compatibility. However, it's possible that recompiling code under this proposal will introduce a data race if that code was previously relying on an actor-isolated value passed as an argument to a non-actor-isolation function only being accessed on the actor's executor. There should at least be a warning in this case. + +The change to the sendability rule may break source compatibility for code that has already adopted concurrency. + +In both cases, since Swift's current behavior is clearly undesirable, these seem like necessary changes. There will not be any attempt to maintain compatibility for existing code. + +## Effect on ABI stability + +The change in execution semantics does not require additional runtime support; the compiler will simply emit a different pattern of calls. + +The change in the sendability rule is compile-time and has no ABI impact. + +## Effect on API resilience + +This proposal does not introduce a new feature. + +It may become more difficult to use `async` APIs that take non-`Sendable` arguments. Such APIs are rare and usually aren't a good idea. + +## Alternatives considered + +### Full inheritance of the caller's executor + +One alternative to this would be for `async` functions that aren't actor-isolated to "inherit" the executors of their callers. Essentially, they would record the current executor when they are called, and they would return to that executor whenever they're resumed. + +There are several benefits to this approach: + +- It can be seen as consistent with the behavior of calls to synchronous functions, which of course "inherit" their executor because they have no ability to change it. + +- It significantly improves the overhang problem relative to the current execution semantics. Overhang would be bounded by the end of the call to an actor function, since upon return the caller would resume its own executor. + +- It is the only alternative which would make the current sendability rule for calls to `async` functions correct. + +However, it has three significant drawbacks: + +- While the overhang would be bounded, it would still cover potentially a large amount of code. Everything called by an actor-isolated async function would resume to the actor, which could include a large amount of work that really doesn't need to be actor-isolated. Actors could become heavily contended for artificial and perhaps surprising reasons. + +- It would make it difficult to write code that does leave the actor, since the inheritance would be implicit and recursive. There could be an attribute which avoids the inheritance, but programmers would have to explicitly remember to use it. This is the opposite of Swift's usual language design approach (e.g. with `mutating` methods); it's better to be less permissive by default so that the places which need stronger guarantees are explicit about it. + +- It would substantially impede optimization. Since the current executor would be semantically observable by inheritance, optimizations that remove executor switches would still have to dynamically record the correct executor that should be inherited. Since they currently do not do this, and since there is no efficient support in the runtime for doing this, this would come at a substantial runtime cost. + +### Initial inheritance of the caller's executor + +Another alternative would be to only inherit the executor of the caller for the initial period of execution, from the call to the first suspension. Later resumptions would resume to a generic, non-actor executor. + +This would permit the current sendability rule for arguments, but only if we enforce that the parameters are not used after a suspension in the callee. This is more flexible, but in ways that are highly likely to prove extremely brittle and limiting; a programmer relying on this flexibility is likely to come to regret it. It would also still not permit return values to be non-`Sendable`, so the rule would still need changing. + +The overhang problem would be further improved relative to full inheritance. The only real overhang risk would be a function that does a lot of synchronous work before returning or suspending. + +Sophisticated programmers might be able to use these semantics to avoid some needless switching. It is common for `async` functions to begin with an `async` call, but if Swift has trouble analyzing the code that sets up that call, then under the proposed semantics, Swift might be unable to avoid the initial switch. However, this optimization deficiency equally affects actor-isolated `async` functions, and arguably it ought to have a consistent solution. + +This would still significantly inhibit optimization prior to `async` calls, since the current executor would be observable when (e.g.) creating new tasks with the `Task` initializer. Other situations would be able to optimize freely. + +Using a sendability rule that's sensitive to both data flow and control flow seems like a non-starter; it is far too complex for its rather weak benefits. However, using such a rule is unnecessary, and these execution semantics could instead be combined with the proposed sendability rule. Non-`Sendable` values that are isolated to the actor would not be shareable with the non-actor-isolated function, and uses of non-`Sendable` values created during the initial segment would be totally ordered by virtue of being isolated to the task. + +Overall, while this approach has some benefits over the proposal, it seems better to go with a consistent and wholly static rule for which executor is running any particular `async` function. Allowing a certain amount of inheritance of executors is an interesting future direction. + +## Future directions + +### Explicit inheritance of executors + +There is still room under this proposal for `async` functions to dynamically inherit their executor from their caller. It simply needs to be opt-in rather than opt-out. This does not seem like such an urgent need that it needs to be part of this proposal. + +While `reasync` functions have not yet been proposed, it would probably be reasonable for them to inherit executors, since they deliberately blur the lines between synchronous and asynchronous operation. + +To allow the caller to use a stronger sendability rule, to avoid over-constraining static optimization of switching, and to support a more efficient ABI, this kind of inheritance should be part of the function signature of the callee. + +### Control over executor-switching optimization + +By adding potential switches in non-actor-isolated `async` functions, this proposal puts more pressure on Swift's optimizer to eliminate unnecessary switches. It may be valuable to add a way for programmers to explicitly inform the optimizer that none of the code prior to a suspension is sensitive to the current executor. + +### Distinguishing actor-isolated from task-isolated values + +As discussed above, uses of a non-`Sendable` value may be totally ordered by being restricted to either a consistent task or a consistent actor. The current sendability rules do not distinguish between these cases; instead, all non-`Sendable` values in a function are subject to uniform restrictions. This forces the creation of hard walls between actor-isolated functions and other functions on the same task. A more expressive sendability rule would distinguish these in actor-isolated `async` functions. This would significantly decrease the degree to which this proposal infringes on reasonable expressivity in such functions. + +The default for parameters and return values should probably be task-isolation rather than actor-isolation, so if we're going to consider this, we need to do it soon for optimal results. + +## Acknowledgments + +Many people contributed to the development of this proposal, but I'd like to especially thank Kavon Farvardin for his part in the investigation. diff --git a/proposals/0339-module-aliasing-for-disambiguation.md b/proposals/0339-module-aliasing-for-disambiguation.md new file mode 100644 index 0000000000..a8e1dfdcee --- /dev/null +++ b/proposals/0339-module-aliasing-for-disambiguation.md @@ -0,0 +1,261 @@ +# Module Aliasing For Disambiguation + +* Proposal: [SE-0339](0339-module-aliasing-for-disambiguation.md) +* Authors: [Ellie Shin](https://github.com/elsh) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Pitch: [Module Aliasing](https://forums.swift.org/t/pitch-module-aliasing/51737) +* Implementation: ([toolchain](https://github.com/apple/swift/pull/40899)), +[apple/swift-package-manager#4023](https://github.com/apple/swift-package-manager/pull/4023), others +* Review: ([review](https://forums.swift.org/t/se-0339-module-aliasing-for-disambiguation/54730)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0339-module-aliasing-for-disambiguation/55032)) + +## Introduction + +Swift does not allow multiple modules in a program to share the same name, and attempts to do so will fail to build. These name collisions can happen in a reasonable program when using multiple packages developed independently from each other. This proposal introduces a way to resolve these conflicts without making major, invasive changes to a package's source by turning a module name in source into an alias, a different unique name. + +## Motivation + +As the Swift package ecosystem has grown, programmers have begun to frequently encounter module name clashes, as seen in several forum discussions including [module name 'Logging' clash in Vapor](https://forums.swift.org/t/logging-module-name-clash-in-vapor-3/25466) and [namespacing packages/modules regarding SwiftNIO](https://forums.swift.org/t/namespacing-of-packages-modules-especially-regarding-swiftnio/24726). There are two main use cases where these arise: + +* Two different packages include logically different modules that happen to have the same name. Often, these modules are "internal" dependencies of the package, which would be submodules if Swift supported submodules; for example, it's common to put common utilities into a `Utils` module, which will then collide if more than one package does it. Programmers often run into this problem when adding a new dependency or upgrading an existing one. +* Two different versions of the same package need to be included in the same program. Programmers often run into this problem when trying to upgrade a dependency that another library has pinned to a specific version. Being unable to resolve this collision makes it difficult to gradually update dependencies, forcing migration to be done all at once later. + +In both cases, it is important to be able to resolve the conflict without making invasive changes to the conflicting packages. While submodules might be a better long-term solution for the first case, they are not currently supported by Swift. Even if submodules were supported, they might not always be correctly adopted by packages, and it would not be reasonable for package clients to have to rewrite the package to properly use them. Submodules and other namespacing features would not completely eliminate the need to "retroactively" resolve module name conflicts. + +## Proposed solution + +We believe that module aliasing provides a systematic method for addressing module name collisions. The conflicting modules can be given unique names while still allowing the source code that depends on them to compile. There's already a way to set a module name to a different name, but we need a new aliasing technique that will allow source files referencing the original module names to compile without making source changes. This will be done via new build settings which will then translate to new compiler flags described below. Together, these low-level tools will allow conflicts to be resolved by giving modules a unique name while using aliases to avoid the need to change any source code. + +We propose to introduce the following new settings in SwiftPM. To illustrate the flow, let's go over an example. Consider the following scenario: `App` imports the module `Game`, which imports a module `Utils` from the same package. `App` also imports another module called `Utils` from a different package. This collision might have been introduced when updating to a new version of `Game`'s package, which introduced an "internal" `Utils` module for the first time. + +``` +App + |— Module Game (from package ‘swift-game’) + |— Module Utils (from package ‘swift-game’) + |— Module Utils (from package ‘swift-draw’) +``` + +The modules from each package have the following code: + +```swift +[Module Game] // swift-game + +import Utils // swift-game +public func start(level: Utils.Level) { ... } +``` + +```swift +[Module Utils] // swift-game + +public struct Level { ... } +public var currentLevel: Utils.Level { ... } +``` + +```swift +[Module Utils] // swift-draw + +public protocol Drawable { ... } +public class Canvas: Utils.Drawable { ... } +``` + +Since `App` depends on these two `Utils` modules, we have a conflict, thus we need to rename one. We will introduce a new setting in SwiftPM called `moduleAliases` that will allow setting unique names for dependencies, like so: +```swift + targets: [ + .executableTarget( + name: "App", + dependencies: [ + .product(name: "Game", package: "swift-game", moduleAliases: ["Utils": "GameUtils"]), + .product(name: "Utils", package: "swift-draw"), + ]) + ] +``` + +The setting `moduleAliases` will rename `Utils` from the `swift-game` package as `GameUtils` and alias all its references in the source code to be compiled as `GameUtils`. Since renaming one of the `Utils` modules will resolve the conflict, it is not necessary to rename the other `Utils` module. The references to `Utils` in the `Game` module will be built as `GameUtils` without requiring any source changes. If `App` needs to reference both `Utils` modules in its source code, it can do so by directly including the aliased name: +```swift +[App] + +import GameUtils +import Utils +``` + +Module aliasing relies on being able to change the namespace of all declarations in a module, so initially only pure Swift modules will be supported and users will be required to opt in. Support for languages that give declarations names outside of the control of Swift, such as Objective-C, C, and C++, would be limited as it will require special handling; see the **Requirements / Limitations** section for more details. + + +## Detailed design + +### Changes to Swift Frontend + +Most use cases should just require setting `moduleAliases` in a package manifest. However, it may be helpful to understand how that setting changes the compiler invocations under the hood. In our example scenario, those invocations will change as follows: + +1. First, we need to take the `Utils` module from `swift-game` and rename it `GameUtils`. To do this, we will compile the module as if it were actually named `GameUtils`, while treating any references to `Utils` in its source files as references to `GameUtils`. + 1. The first part (renaming) can be achieved by passing the new module name (`GameUtils`) to `-module-name`. The new module name will also need to be used in any flags specifying output paths, such as `-o`, `-emit-module-path`, or `-emit-module-interface-path`. For example, the binary module file should be built as `GameUtils.swiftmodule` instead of `Utils.swiftmodule`. + 2. The second part (treating references to `Utils` in source files as `GameUtils`) can be achieved with a new compiler flag `-module-alias [name]=[new_name]`. Here, `name` is the module name that appears in source files (`Utils`), while `new_name` is the new, unique name (`GameUtils`). So in our example, we will pass `-module-alias Utils=GameUtils`. + + Putting these steps together, the compiler invocation command would be `swiftc -module-name GameUtils -emit-module-path /path/to/GameUtils.swiftmodule -module-alias Utils=GameUtils ...`. + + For all intents and purposes, the true name of the module is now `GameUtils`. The name `Utils` is no longer associated with it. Module aliases can be used in specific parts of the build to allow source code that still uses the name `Utils` (possibly including the module itself) to continue to compile. +2. Next, we need to build the module `Game`. `Game` contains references to `Utils`, which we need to treat as references to `GameUtils`. We can do this by just passing `-module-alias Utils=GameUtils` without any other changes. The overall compiler invocation command to build `Game` is `swiftc -module-name Game -module-alias Utils=GameUtils ...`. +3. We don't need any build changes when building `App` because the source code in `App` does not expect to use the `Utils` module from `swift-game` under its original name. If `App` tries to import a module named `Utils`, that will refer to the `Utils` module from `swift-draw`, which has not been renamed. If `App` does need to import the `Utils` module from `swift-game`, it must use `import GameUtils`. + + +The arguments to the `-module-alias` flag will be validated against reserved names, invalid identifiers, a wrong format or ordering (`-module-alias Utils=GameUtils` is correct but `-module-alias GameUtils=Utils` is not). The flag can be repeated to allow multiple aliases, e.g. `-module-alias Utils=GameUtils -module-alias Logging=GameLogging`, and will be checked against duplicates. Diagnostics and fix-its will contain the name `Utils` in the error messages as opposed to `GameUtils` to be consistent with the names appearing to users. + +The validated map of aliases will be stored in the AST context and used for dependency scanning/resolution and module loading; from the above scenario, if `Game` is built with `-module-alias Utils=GameUtils` and has `import Utils` in source code, `GameUtils.swiftmodule` should be loaded instead of `Utils.swiftmodule` during import resolution. + +While the name `Utils` appears in source files, the actual binary name `GameUtils` will be used for name lookup, semantic analysis, symbol mangling (e.g. `$s9GameUtils5Level`), and serialization. Since the binary names will be stored during serialization, the aliasing flag will only be needed to build the conflicting modules and their immediate consuming modules; building non-immediate consuming modules will not require the flag. + +Direct references to the renamed modules should only be allowed in source code if multiple conflicting modules need to be imported; in such case, a direct reference in an import statement, e.g. `import GameUtils`, is allowed. Otherwise, the original name `Utils` should be used in source code instead of the binary name `GameUtils`. The module alias map will be used to track when to disallow direct references to the binary module names in source files, and an attempt to use the binary name will result in an error along with a fix-it. This restriction is useful as it can make it easier to rename the module again later if needed, e.g. from `GameUtils` to `SwiftGameUtils`. + +Unlike source files, the generated interface (.swiftinterface) will contain the binary module name in all its references. The binary module name will also be stored for indexing and debugging, and treated as the source of truth. + +### Changes to Code Assistance / Indexing + +The compiler arguments including the new flag `-module-alias` will be available to SourceKit and indexing. The aliases will be stored in the AST context and used to fetch the right results for code completion and other code assistance features. They will also be stored for indexing so features such as `Jump to Definition` can navigate to declarations scoped to the binary module names. + +Generated documentation, quick help, and other assistance features will contain the binary module names, which will be treated as the source of truth. + +### Changes to Swift Driver + +The module aliasing arguments will be used during the dependency scan for both implicit and explicit build modes; the resolved dependency graph will contain the binary module names. In case of the explicit build mode, the dependency input passed to the frontend will contain the binary module names in its json file. Similar to the frontend, validation of the aliasing arguments will be performed at the driver. + +### Changes to SwiftPM + +To make module aliasing more accessible, we will introduce new build configs which can map to the compiler flags for aliasing described above. Let’s go over how they can be adopted by SwiftPM with the above scenario (copied here). +``` +App + |— Module Game (from package ‘swift-game’) + |— Module Utils (from package ‘swift-game’) + |— Module Utils (from package ‘swift-draw’) +``` + +Here are the manifest examples for `swift-game` and `swift-draw`. + +```swift +let package = Package( + name: "swift-game", + dependencies: [], + products: [ + .library(name: "Game", targets: ["Game"]), + .library(name: "Utils", targets: ["Utils"]), + ], + targets: [ + .target(name: "Game", dependencies: ["Utils"]), + .target(name: "Utils", dependencies: []) + ] +) +``` + +```swift +let package = Package( + name: "swift-draw", + dependencies: [], + products: [ + .library(name: "Utils", targets: ["Utils"]), + ], + targets: [ + .target(name: "Utils", dependencies: []) + ] +) +``` + +The `App` manifest needs to explicitly define unique names for the conflicting modules via a new parameter called `moduleAliases`. +```swift +let package = Package( + name: "App", + dependencies: [ + .package(url: https://.../swift-game.git), + .package(url: https://.../swift-draw.git) + ], + products: [ + .executable(name: "App", targets: ["App"]) + ] + targets: [ + .executableTarget( + name: "App", + dependencies: [ + .product(name: "Game", package: "swift-game", moduleAliases: ["Utils": "GameUtils"]), + .product(name: "Utils", package: "swift-draw"), + ]) + ] +) +``` + +SwiftPM will perform validations when it parses `moduleAliases`; for each entry, it will check whether the given alias is a unique name, whether there is a conflict among aliases, whether the specified module is built from source (pre-compiled modules cannot be rebuilt to respect the rename), and whether the module is a pure Swift module (see **Requirements/Limitations** section for more details). + +It will also check if any aliases are defined in upstream packages and override them if necessary. For example, if the `swift-game` package were modified per below and defined its own alias `SwiftUtils` for module `Utils` from a dependency package, the alias defined in `App` will override it, thus the `Utils` module from `swift-utils` will be built as `GameUtils`. + +```swift +let package = Package( + name: "swift-game", + dependencies: [ + .package(url: https://.../swift-utils.git), + ], + products: [ + .library(name: "Game", targets: ["Game"]), + ], + targets: [ + .target(name: "Game", + dependencies: [ + .product(name: "UtilsProduct", + package: "swift-utils", + moduleAliases: ["Utils": "SwiftUtils"]), + ]) + ] +) +``` + +Once the validation and alias overriding steps pass, dependency resolution will take place using the new module names, and the `-module-alias [name]=[new_name]` flag will be passed to the build execution. + + +### Resources + +Tools invoked by a build system to compile resources should be modified to handle the module aliasing. The module name entry should get the renamed value and any references to aliased modules in the resources should correctly map to the corresponding binary names. The resources likely impacted by this are IB, CoreData, and anything that explicitly requires module names. We will initially only support asset catalogs and localized strings as module names are not required for those resources. + +### Debugging + +When module aliasing is used, the binary module name will be stored in mangled symbols, e.g. `$s9GameUtils5Level` instead of `$s5Utils5Level`, which will be stored in Debuginfo. + +For evaluating an expression, the name `Utils` can be used as it appears in source files (which were already compiled with module aliasing); however, the result of the evaluation will contain the binary module name. + +If a module were to be loaded directly into lldb, the binary module name should be used, i.e. `import GameUtils` instead of `import Utils`, since it does not have access to the aliasing flag. + +In REPL, binary module names should be used for importing or referencing; support for aliasing in that mode may be added in the future. + +## Requirements / Limitations + +To allow module aliasing, the following requirements need to be met, which come with some limitations. + +* Only pure Swift modules allowed for aliasing: no ObjC/C/C++/Asm due to potential symbol collision. Similarly, `@objc(name)` is discouraged. +* Building from source only: aliasing distributed binaries is not possible due to the impact on mangling and serialization. +* Runtime: calls to convert String to types in module, i.e direct or indirect calls to `NSClassFromString(...)`, will fail and should be avoided. +* For resources, only asset catalogs and localized strings are allowed. +* Higher chance of running into the following existing issues: + * [Retroactive conformance](https://forums.swift.org/t/retroactive-conformances-vs-swift-in-the-os/14393): this is already not a recommended practice and should be avoided. + * Extension member “leaks”: this is [considered a bug](https://bugs.swift.org/browse/SR-3908) which hasn’t been fixed yet. More discussions [here](https://forums.swift.org/t/pre-pitch-import-access-control-a-modest-proposal/50087). +* Code size increase will be more implicit thus requires a caution, although module aliasing will be opt-in and a size threshold could be added to provide a warning. + +## Source compatibility +This is an additive feature. Currently when there are duplicate module names, it does not compile at all. This feature requires explicitly opting in to allow and use module aliaisng via package manifests or compiler invocation commands and does not require source code changes. + +## Effect on ABI stability +The feature in this proposal does not have impact on the ABI. + +## Effect on API resilience +This proposal does not introduce features that would be part of a public API. + +## Future Directions + +* Currently when a module contains a type with the same name, fully qualifying a decl in the module results in an error; it treats the left most qualifier as a type instead of the module ([SR-14195](https://bugs.swift.org/browse/SR-14195), [pitch](https://forums.swift.org/t/fixing-modules-that-contain-a-type-with-the-same-name/3025), [pitch](https://forums.swift.org/t/pitch-fully-qualified-name-syntax/28482)); `XCTest` is a good example as it contains a class called `XCTest`. Trying to access a top level function `XCTAssertEqual` via `XCTest.XCTAssertEqual(...)` results in `Type 'XCTest' has no member 'XCTAssertEqual'` error. Module aliasing could mitigate this issue by renaming `XCTest` as `XCTestFramework` without requiring source changes in the `XCTest` module and allowing the function access via `XCTestFramework.XCTAssertEqual(...)` in the user code. + +* Introducing new import syntax such as `import Utils as GameUtils` has been discussed in forums to improve module disambiguation. The module aliasing infrastructure described in this proposal paves the way towards using such syntax that could allow more explicit (in source code) aliasing. + +* Visibility change to import decl access level (from public to internal) pitched [here](https://forums.swift.org/t/pre-pitch-import-access-control-a-modest-proposal/50087) could help address the extension leaks issues mentioned in **Requirements / Limitations** section. + +* Swift modules that have C target dependencies could, in a limited capacity, be supported by changing visibility to C symbols. + +* C++ interop support could potentially allow C++ modules to be aliased besides pure Swift modules. + +* Nested namespacing or submodules might be a better long-term solution for some of the collision issues described in **Motivation**. However, it would not completely eliminate the need to "retroactively" resolve module name conflicts. Module aliasing does not introduce any lexical or structural changes that might have an impact on potential future submodules support; it's an orthogonal feature and can be used in conjunction if needed. + +## Acknowledgments +This proposal was improved with feedback and helpful suggestions along with code reviews by Becca Royal-Gordon, Alexis Laferriere, John McCall, Joe Groff, Mike Ash, Pavel Yaskevich, Adrian Prantl, Artem Chikin, Boris Buegling, Anders Bertelrud, Tom Doron, and Johannes Weiss, and others. diff --git a/proposals/0340-swift-noasync.md b/proposals/0340-swift-noasync.md new file mode 100644 index 0000000000..3a95229752 --- /dev/null +++ b/proposals/0340-swift-noasync.md @@ -0,0 +1,375 @@ +# Unavailable From Async Attribute + +* Proposal: [SE-0340](0340-swift-noasync.md) +* Authors: [Evan Wilde](https://github.com/etcwilde) +* Review manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Implementation: [noasync availability](https://github.com/apple/swift/pull/40769) +* Discussion: [Discussion: Unavailability from asynchronous contexts](https://forums.swift.org/t/discussion-unavailability-from-asynchronous-contexts/53088) +* Pitch: [Pitch: Unavailability from asynchronous contexts](https://forums.swift.org/t/pitch-unavailability-from-asynchronous-contexts/53877) +* Review: [SE-0340: Unavailable from Async Attribute](https://forums.swift.org/t/se-0340-unavailable-from-async-attribute/54852) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0340-unavailable-from-async-attribute/55356) + +## Introduction + +The Swift concurrency model allows tasks to resume on different threads from the +one they were suspended on. For this reason, API that relies on thread-local +storage, locks, mutexes, and semaphores, should not be used across suspension +points. + +```swift +func badAsyncFunc(_ mutex: UnsafeMutablePointer, _ op : () async -> ()) async { + // ... + pthread_mutex_lock(mutex) + await op() + pthread_mutex_unlock(mutex) // Bad! May unlock on a different thread! + // ... +} +``` + +The example above exhibits undefined behaviour if `badAsyncFunc` resumes on a +different thread than the one it started on after running `op` since +`pthread_mutex_unlock` must be called from the same thread that locked the +mutex. + +We propose extending `@available` with a new `noasync` availability kind to +indicate API that may not be used directly from asynchronous contexts. + +Swift evolution thread: [Pitch: Unavailability from asynchronous contexts](https://forums.swift.org/t/pitch-unavailability-from-asynchronous-contexts/53877) + +## Motivation + +The Swift concurrency model allows tasks to suspend and resume on different +threads. While this behaviour allows higher utility of computational resources, +there are some nasty pitfalls that can spring on an unsuspecting programmer. One +such pitfall is the undefined behaviour from unlocking a `pthread_mutex_t` from +a different thread than the thread that holds the lock, locking threads may +easily cause unexpected deadlocks, and reading from and writing to thread-local +storage across suspension points may result in unintended behaviour that is +difficult to debug. + +## Proposed Solution + +We propose extending `@available` to accept a `noasync` availability kind. +The `noasync` availability kind is applicable to most declarations, but is not +allowed on destructors as those are not explicitly called and must be callable +from anywhere. + +```swift +@available(*, noasync) +func doSomethingNefariousWithNoOtherOptions() { } + +@available(*, noasync, message: "use our other shnazzy API instead!") +func doSomethingNefariousWithLocks() { } + +func asyncFun() async { + // Error: doSomethingNefariousWithNoOtherOptions is unavailable from + // asynchronous contexts + doSomethingNefariousWithNoOtherOptions() + + // Error: doSomethingNefariousWithLocks is unavailable from asynchronous + // contexts; use our other shanzzy API instead! + doSomethingNefariousWithLocks() +} +``` + +The `noasync` availability attribute only prevents API usage in the immediate +asynchronous context; wrapping a call to an unavailable API in a synchronous +context and calling the wrapper will not emit an error. This allows for cases +where it is possible to use the API safely within an asynchronous context, but +in specific ways. The example below demonstrates this with an example of using a +pthread mutex to wrap a critical section. The function ensures that there cannot +be a suspension point between obtaining and releasing the lock, and therefore is +safe for consumption by asynchronous contexts. + +```swift +func goodAsyncFunc(_ mutex: UnsafeMutablePointer, _ op : () -> ()) async { + // not an error, pthread_mutex_lock is wrapped in another function + with_pthread_mutex_lock(mutex, do: op) +} + +func with_pthread_mutex_lock( + _ mutex: UnsafeMutablePointer, + do op: () throws -> R) rethrows -> R { + switch pthread_mutex_lock(mutex) { + case 0: + defer { pthread_mutex_unlock(mutex) } + return try op() + case EINVAL: + preconditionFailure("Invalid Mutex") + case EDEADLK: + fatalError("Locking would cause a deadlock") + case let value: + fatalError("Unknown pthread_mutex_lock() return value: '\(value)'") + } +} +``` + +The above snippet is a safe wrapper for `pthread_mutex_lock` and +`pthread_mutex_unlock`, since the lock is not held across suspension points. The +critical section operation must be synchronous for this to hold true though. +The following snippet uses a synchronous closure to call the unavailable +function, circumventing the protection provided by the attribute. + +```swift +@available(*, noasync) +func pthread_mutex_lock(_ lock: UnsafeMutablePointer) {} + +func asyncFun(_ mutex : UnsafeMutablePointer) async { + // Error! pthread_mutex_lock is unavailable from async contexts + pthread_mutex_lock(mutex) + + // Ok! pthread_mutex_lock is not called from an async context + _ = { unavailableFun(mutex) }() + + await someAsyncOp() +} +``` + +### Replacement API + +In some cases, it is possible to provide an alternative that is safe. The +`with_pthread_mutex_lock` is an example of a way to provide a safe way to wrap +locking and unlocking pthread mutexes. + +In other cases, it may be safe to use an API from a specific actor. For +example, API that uses thread-local storage isn't safe for consumption by +asynchronous functions in general, but is safe for functions on the MainActor +since it will only use the main thread. + +The unavailable API should still be annotated as such, but an alternative +function can be implemented as an extension of the actors that support the +operation. + +```swift +@available(*, noasync, renamed: "mainactorReadID()", message: "use mainactorReadID instead") +func readIDFromThreadLocal() -> Int { } + +@MainActor +func readIDFromMainActor() -> Int { readIDFromThreadLocal() } + +func asyncFunc() async { + // Bad, we don't know what thread we're on + let id = readIDFromThreadLocal() + + // Good, we know it's coming from the main actor on the main thread. + // Note the suspension due to the jump to the main actor. + let id = await readIDFromMainActor() +} +``` + +Restricting a synchronous API to an actor is done similarly, as demonstrated in +the example below. The synchronous `save` function is part of a public API, so +it can't just be pulled into the `DataStore` actor without causing a source +break. Instead, it is annotated with a `noasync` available attribute. +`DataStore.save` is a thin wrapper around the original synchronous save +function. Calls from an asynchronous context to `save` may only be done through +the `DataStore` actor, ensuring that the cooperative pool isn't tied up with the +save function. The original save function is still available to synchronous code +as it was before. + +```swift +@available(*, noasync, renamed: "DataStore.save()") +public func save(_ line: String) { } + +public actor DataStore { } + +public extension DataStore { + func save(_ line: String) { + save(line) + } +} +``` + +## Additional design details + +Verifying that unavailable functions are not used from asynchronous contexts is +done weakly; only unavailable functions called directly from asynchronous +contexts are diagnosed. This avoids the need to recursively typecheck the bodies +of synchronous functions to determine whether they are implicitly available from +asynchronous contexts, or to verify that they are appropriately annotated. + +While the typechecker doesn't need to emit diagnostics from synchronous +functions, they cannot be omitted entirely. It is possible to declare +asynchronous contexts inside of synchronous contexts, wherein diagnostics should +be emitted. + +```swift +@available(*, noasync) +func bad2TheBone() {} + +func makeABadAsyncClosure() -> () async -> Void { + return { () async -> Void in + bad2TheBone() // Error: Unavailable from asynchronous contexts + } +} +``` + +## Source Compatibility + +Swift 3 and Swift 4 do not have this attribute, so code coming from Swift 3 and +Swift 4 won't be affected. + +The attribute will affect any current asynchronous code that currently contains +use of API that are modified with this attribute later. To ease the transition, +we propose that this attribute emits a warning in Swift 5.6, and becomes a full +error in Swift 6. In cases where someone really wants unsafe behavior and enjoys +living on the edge, the diagnostic is easily circumventable by wrapping the API +in a synchronous closure, noted above. + +## Effect on ABI stability + +This feature has no effect on ABI. + +## Effect on API resilience + +The presence of the attribute has no effect on the ABI. + +## Alternatives Considered + +### Propagation + +The initial discussion focused on how unavailability propagated, including the +following three designs; + - implicitly-inherited unavailability + - explicit unavailability + - thin unavailability + +The ultimate decision is to go with the thin checking; both the implicit and +explicit checking have high performance costs and require far more consideration +as they are adding another color to functions. + +The attribute is expected to be used for a fairly limited set of specialized +use-cases. The goal is to provide some protection without dramatically impacting +the performance of the compiler. + +#### Implicitly inherited unavailability + +Implicitly inheriting unavailability would transitively apply the unavailability +to functions that called an unavailable function. This would have the lowest +developer overhead while ensuring that one could not accidentally use the +unavailable functions indirectly. + +```swift +@unavailableFromAsync +func blarp1() {} + +func blarp2() { + // implicitly makes blarp2 unavailable + blarp1() +} + +func asyncFun() async { + // Error: blarp2 is impicitly unavailable from async because of call to blarp1 + blarp2() +} +``` + +Unfortunately, computing this is very expensive, requiring type-checking the +bodies of every function a given function calls in order to determine if the +declaration is available from an async context. Requiring even partial +type-checking of the function bodies to determine the function declaration is +prohibitively expensive, and is especially detrimental to the performance of +incremental compilation. + +We would need an additional attribute to disable the checking for certain +functions that are known to be usable from an async context, even though they +use contain unavailable functions. An example of a safe, but "unavailable" +function is `with_pthread_mutex_lock` above. + +#### Explicit unavailability + +This design behaves much like availability does today. In order to use an +unavailable function, the calling function must be explicitly annotated with the +unavailability attribute or an error is emitted. + +Like the implicit unavailability propagation, we still need an additional +attribute to indicate that, while a function may contain unsafe API, it uses +them in a way that is safe for use in asynchronous contexts. + +The benefits of this design are that it both ensures that unsafe API are explicitly +handled correctly, avoiding bugs. Additionally, typechecking asynchronous +functions is reasonably performant and does not require recursively +type-checking the bodies of every synchronous function called by the +asynchronous function. + +Unfortunately, we would need to walk the bodies of every synchronous function to +ensure that every synchronous function is correctly annotated. This reverses the +benefits of the implicit availability checking, while having a high developer +overhead. + +### Separate Attribute + +We considered using a separate attribute, spelled `@unavailableFromAsync`, to +annotate the unavailable API. After more consideration, it became apparent that +we would likely need to reimplement much of the functionality of the +`@available` attribute. + +Some thoughts that prompted the move from `@unavailableFromAsync` to an +availability kind include: + + - A given API may have different implementations on different platforms, and + therefore may be implemented in a way that is safe for consumption in + asynchronous contexts in some cases but not others. + - An API may be currently implemented in a way that is unsafe for consumption + in asynchronous contexts, but may be safe in the future. + - We get `message`, `renamed`, and company, with serializations, for free by + merging this with `@available`. + +Challenges to the merge mostly focus on the difference in the verification model +between this and the other availability modes. The `noasync`, as discussed +above, is a weaker check and does not require API that is using the unavailable +function to also be annotated. The other availability checks do require that the +availability information be propagated. + +## Future Directions + +[Custom executors](https://forums.swift.org/t/support-custom-executors-in-swift-concurrency/44425) +are pitched to become part of the language as a future feature. +Restricting an API to a custom executor is the same as restricting that API to +an actor. The difference is that the actor providing the replacement API has +it's `unownedExecutor` overridden with the desired custom executor. + +Hand-waving around some of the syntax, this protection could look something like +the following example: + +```swift +protocol IOActor : Actor { } + +extension IOActor { + nonisolated var unownedExecutor: UnownedSerialExecutor { + return getMyCustomIOExecutor() + } +} + +@available(*, noasync, renamed: "IOActor.readInt()") +func readIntFromIO() -> String { } + +extension IOActor { + // IOActor replacement API goes here + func readInt() -> String { readIntFromIO() } +} + +actor MyIOActor : IOActor { + func printInt() { + // Okay! It's synchronous on the IOActor + print(readInt()) + } +} + +func print(myActor : MyIOActor) async { + // Okay! We only call `readIntFromIO` on the IOActor's executor + print(await myActor.readInt()) +} +``` + +The `IOActor` overrides it's `unownedExecutor` with a specific custom IO +executor and provides a synchronous `readInt` function wrapping a call to the +`readIntFromIO` function. The `noasync` availability attribute ensures that +`readIntFromIO` cannot generally be used from asynchronous contexts. +When `readInt` is called, there will be a hop to the `MyIOActor`, which uses the +custom IO executor. + +## Acknowledgments + +Thank you Becca and Doug for you feedback and help shaping the proposal. diff --git a/proposals/0341-opaque-parameters.md b/proposals/0341-opaque-parameters.md new file mode 100644 index 0000000000..e324461490 --- /dev/null +++ b/proposals/0341-opaque-parameters.md @@ -0,0 +1,270 @@ +# Opaque Parameter Declarations + +* Proposal: [SE-0341](0341-opaque-parameters.md) +* Author: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Ben Cohen](https://github.com/AirspeedSwift) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#40993](https://github.com/apple/swift/pull/40993) + +## Introduction + +Swift's syntax for generics is designed for generality, allowing one to express complicated sets of constraints amongst the different inputs and outputs of a function. For example, consider an eager concatenation operation that builds an array from two sequences: + +```swift +func eagerConcatenate( + _ sequence1: Sequence1, _ sequence2: Sequence2 +) -> [Sequence1.Element] where Sequence1.Element == Sequence2.Element +``` + +There is a lot going on in that function declaration: the two function parameters are of different types determined by the caller, which are captured by `Sequence1` and `Sequence2`, respectively. Both of these types must conform to the `Sequence` protocol and, moreover, the element types of the two sequences must be equivalent. Finally, the result of this operation is an array of the sequence's element type. One can use this operation with many different inputs, so long as the constraints are met: + +```swift +eagerConcatenate([1, 2, 3], Set([4, 5, 6])) // okay, produces an [Int] +eagerConcatenate([1: "Hello", 2: "World"], [(3, "Swift"), (4, "!")]) // okay, produces an [(Int, String)] +eagerConcatenate([1, 2, 3], ["Hello", "World"]) // error: sequence element types do not match +``` + +However, when one does not need to introduce a complex set of constraints, the syntax starts to feel quite heavyweight. For example, consider a function that composes two SwiftUI views horizontally: + +```swift +func horizontal(_ v1: V1, _ v2: V2) -> some View { + HStack { + v1 + v2 + } +} +``` + +There is a lot of boilerplate to declare the generic parameters `V1` and `V2` that are only used once, making this function look far more complex than it really is. The result, on the other hand, is able to use an [opaque result type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md) to hide the specific returned type (which would be complicated to describe), describing it only by the protocols to which it conforms. + +This proposal extends the syntax of opaque result types to parameters, allowing one to specify function parameters that are generic without the boilerplate associated with generic parameter lists. The `horizontal` function above can then be expressed as: + +```swift +func horizontal(_ v1: some View, _ v2: some View) -> some View { + HStack { + v1 + v2 + } +} +``` + +Semantically, this formulation is identical to the prior one, but is simpler to read and understand because the inessential complexity from the generic parameter lists has been removed. It takes two views (the concrete type does not matter) and returns a view (the concrete type does not matter). + +Swift-evolution threads: [Pitch for this proposal](https://forums.swift.org/t/pitch-opaque-parameter-types/54914), [Easing the learning curve for introducing generic parameters](https://forums.swift.org/t/discussion-easing-the-learning-curve-for-introducing-generic-parameters/52891), [Improving UI of generics pitch](https://forums.swift.org/t/improving-the-ui-of-generics/22814) + +## Proposed solution + +This proposal extends the use of the `some` keyword to parameter types for function, initializer, and subscript declarations. As with opaque result types, `some P` indicates a type that is unnamed and is only known by its constraint: it conforms to the protocol `P`. When an opaque type occurs within a parameter type, it is replaced by an (unnamed) generic parameter. For example, the given function: + +```swift +func f(_ p: some P) { } +``` + +is equivalent to a generic function described as follows, with a synthesized (unnamable) type parameter `_T`: + +```swift +func f<_T: P>(_ p: _T) +``` + +Note that, unlike with opaque result types, the caller determines the type of the opaque type via type inference. For example, if we assume that both `Int` and `String` conform to `P`, one can call or reference the function with either `Int` or `String`: + +```swift +f(17) // okay, opaque type inferred to Int +f("Hello") // okay, opaque type inferred to String + +let fInt: (Int) -> Void = f // okay, opaque type inferred to Int +let fString: (String) -> Void = f // okay, opaque type inferred to String +let fAmbiguous = f // error: cannot infer parameter for `some P` parameter +``` + +[SE-0328](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0328-structural-opaque-result-types.md) extended opaque result types to allow multiple uses of `some P` types within the result type, in any structural position. Opaque types in parameters permit the same structural uses, e.g., + +```swift +func encodeAnyDictionaryOfPairs(_ dict: [some Hashable & Codable: Pair]) -> Data +``` + +This is equivalent to: + +```swift +func encodeAnyDictionaryOfPairs<_T1: Hashable & Codable, _T2: Codable, _T3: Codable>(_ dict: [_T1: Pair<_T2, _T3>]) -> Data +``` + +Each instance of `some` within the declaration represents a different implicit generic parameter. + +## Detailed design + +Opaque parameter types can only be used in parameters of a function, initializer, or subscript declaration. They cannot be used in (e.g.) a typealias or any value of function type. For example: + +```swift +typealias Fn = (some P) -> Void // error: cannot use opaque types in a typealias +let g: (some P) -> Void = f // error: cannot use opaque types in a value of function type +``` + +There are additional restrictions on the use of opaque types in parameters where they may conflict with future language features. + +### Variadic generics + +An opaque type cannot be used in a variadic parameter: + +```swift +func acceptLots(_: some P...) +``` + +This restriction is in place because the semantics implied by this proposal might not be the appropriate semantics if Swift gains variadic generics. Specifically, the semantics implied by this proposal itself (without variadic generics) would be equivalent to: + +```swift +func acceptLots<_T: P>(_: _T...) +``` + +where `acceptLots` requires that all of the arguments have the same type: + +```swift +acceptLots(1, 1, 2, 3, 5, 8) // okay +acceptLots("Hello", "Swift", "World") // okay +acceptLots("Swift", 6) // error: argument for `some P` could be either String or Int +``` + +With variadic generics, one might instead make the implicit generic parameter a generic parameter pack, as follows: + +```swift +func acceptLots<_Ts: P...>(_: _Ts...) +``` + +In this case, `acceptLots` accepts any number of arguments, all of which might have different types: + +```swift +acceptLots(1, 1, 2, 3, 5, 8) // okay, Ts contains six Int types +acceptLots("Hello", "Swift", "World") // okay, Ts contains three String types +acceptLots(Swift, 6) // okay, Ts contains String and Int +``` + +### Opaque parameters in "consuming" positions of function types + +The resolution of [SE-0328](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0328-structural-opaque-result-types.md) prohibited the use of opaque parameters in "consuming" positions of function types. For example: + +```swift +func f() -> (some P) -> Void { ... } // error: cannot use opaque type in parameter of function type +``` + +The result of function `f` is fairly hard to use, because there is no way for the caller to easily create a value of an unknown, unnamed type: + +```swift +let fn = f() +fn(/* how do I create a value here? */) +``` + +The same prohibition applies to opaque types that occur within parameters of function type, e.g., + +```swift +func g(fn: (some P) -> Void) { ... } // error: cannot use opaque type in parameter of function type +``` + +The reasoning for this prohibition is similar. In the implementation of `g`, it's hard to produce a value of the type `some P` when that type isn't named anywhere else. + +## Source compatibility + +This is a pure language extension with no backward-compatibility concerns, because all uses of `some` in parameter position are currently errors. + +## Effect on ABI stability + +This proposal has no effect on the ABI or runtime because it is syntactic sugar for generic parameters. + +## Effect on API resilience + +This feature is purely syntactic sugar, and one can switch between using opaque parameter types and the equivalent formulation with explicit generic parameters without breaking either the ABI or API. However, the complete set of constraints must be the same in such cases. + +## Future Directions + +### Constraining the associated types of a protocol + +This proposal composes well with an idea that allows the use of generic syntax to specify the associated type of a protocol, e.g., where `Collection`is "a `Collection` whose `Element` type is `String`". Combined with this proposal, one can more easily express a function that takes an arbitrary collection of strings: + +```swift +func takeStrings(_: some Collection) { ... } +``` + +Recall the complicated `eagerConcatenate` example from the introduction: + +```swift +func eagerConcatenate( + _ sequence1: Sequence1, _ sequence2: Sequence2 +) -> [Sequence1.Element] where Sequence1.Element == Sequence2.Element +``` + +With opaque parameter types and generic syntax on protocol types, one can express this in a simpler form with a single generic parameter representing the element type: + +```swift +func eagerConcatenate( + _ sequence1: some Sequence, _ sequence2: some Sequence +) -> [T] +``` + +And in conjunction with opaque result types, we can hide the representation of the result, e.g., + +```swift +func lazyConcatenate( + _ sequence1: some Sequence, _ sequence2: some Sequence +) -> some Sequence +``` + +### Enabling opaque types in consuming positions + +The prohibition on opaque types in "consuming" positions could be lifted for opaque types both in parameters and in return types, but they wouldn't be useful with their current semantics because in both cases the wrong code (caller vs. callee) gets to choose the parameter. We could enable opaque types in consuming positions by "flipping" who gets to choose the parameter. To understand this, think of opaque result types as a form of "reverse generics", where there is a generic parameter list after a function's `->` and for which the function itself (the callee) gets to choose the type. For example: + +```swift +func f1() -> some P { ... } +// translates to "reverse generics" version... +func f1() -> T { /* callee implementation here picks concrete type for T */ } +``` + +The problem with opaque types in consuming positions of the return type is that the callee picks the concrete type, and the caller can't reason about it. We can see this issue by translating to the reverse-generics formulation: + +```swift +func f2() -> (some P) -> Void { ... } +// translates to "reverse generics" version... +func f2() -> (T) -> Void { /* callee implementation here picks concrete type for T */} +``` + +We could "flip" the caller/callee choice here by translating opaque types in consuming positions to the other side of the `->`. For example, `f2` would be translated into + +```swift +// if we "flip" opaque types in consuming positions +func f2() -> (some P) -> Void { ... } +// translates to +func f2() -> (T) -> Void { ... } +``` + +This is a more useful translation, because the caller picks the type for `T` using type context, and the callee provides a closure that can work with whatever type the caller picks, generically. For example: + +```swift +let fn1: (Int) -> Void == f2 // okay, T == Int +let fn2: (String) -> Void = f2 // okay, T == String +``` + +Similar logic applies to opaque types in consuming positions within parameters. Consider this function: + +```swift +func g2(fn: (some P) -> Void) { ... } +``` + +If this translates to "normal" generics, i.e., then the parameter isn't readily usable: + +```swift +// if we translated to "normal" generics +func g2(fn: (T) -> Void) { /* how do we come up with a T to call fn with? */} +``` + +Again, the problem here is that the caller gets to choose what `T` is, but then the callee cannot use it effectively. We could again "flip" the generics, moving the implicit type parameter for an opaque type in consuming position to the other side of the function's `->`: + +```swift +// if we "flip" opaque types in consuming positions +func g2(fn: (some P) -> Void) { ... } +// translates to +func g2(fn: (T) -> Void) -> Void { ... } +``` + +Now, the implementation of `g2` (the callee) gets to choose the type of `T`, which is appropriate because it will be providing values of type `T` to `fn`. The caller will need to provide a closure or generic function that's able to accept any `T` that conforms to `P`. It cannot write the type out, but it can certainly make use of it via type inference, e.g.: + +```swift +g2 { x in x.doSomethingSpecifiedInP() } +``` diff --git a/proposals/0342-static-link-runtime-libraries-by-default-on-supported-platforms.md b/proposals/0342-static-link-runtime-libraries-by-default-on-supported-platforms.md new file mode 100644 index 0000000000..788c935013 --- /dev/null +++ b/proposals/0342-static-link-runtime-libraries-by-default-on-supported-platforms.md @@ -0,0 +1,232 @@ +# Statically link Swift runtime libraries by default on supported platforms + +* Proposal: [SE-0342](0342-static-link-runtime-libraries-by-default-on-supported-platforms.md) +* Authors: [neonichu](https://github.com/neonichu) [tomerd](https://github.com/tomerd) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Accepted** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0342-statically-link-swift-runtime-libraries-by-default-on-supported-platforms/56517) +* Implementation: [apple/swift-package-manager#3905](https://github.com/apple/swift-package-manager/pull/3905) +* Initial discussion: [Forum Thread](https://forums.swift.org/t/pre-pitch-statically-linking-the-swift-runtime-libraries-by-default-on-linux) +* Pitch: [Forum Thread](https://forums.swift.org/t/pitch-package-manager-statically-link-swift-runtime-libraries-by-default-on-supported-platforms) + +## Introduction + +Swift 5.3.1 introduced [statically linking the Swift runtime libraries on Linux](https://forums.swift.org/t/static-linking-on-linux-in-swift-5-3-1/). +With this feature, users can set the `--static-swift-stdlib` flag when invoking SwiftPM commands (or the long form `-Xswiftc -static-stdlib`) in order to statically link the Swift runtime libraries into the program. + +On some platforms, such as Linux, this is often the preferred way to link programs, since the program is easier to deploy to the target server or otherwise share. + +This proposal explores making it SwiftPM's default behavior when building executable programs on such platforms. + +## Motivation + +Darwin based platform ship with the Swift runtime libraries in the _dyld shared cache_. +This allows building smaller Swift programs by dynamically linking the Swift runtime libraries. +The shared cache keeps the cost of loading these libraries low. + +Other platforms, such as conventional Linux distributions, do not ship with the Swift runtime libraries. +Hence, a deployment of a program built with Swift (e.g. a web-service or CLI tool) on such platform requires one of three options: + +1. Package the application with a "bag of shared objects" (the `libswift*.so` files making up Swift's runtime libraries) alongside the program. +2. Statically link the runtime libraries using the `--static-swift-stdlib` flag described above. +3. Use a "runtime" docker image that contain the _correct version_ of the the runtime libraries (matching the compiler version exactly). + +Out of the three options, the most convenient is #2 given that #1 requires manual intervention and/or additional wrapper scripts that use `ldd`, `readelf` or similar tools to deduce the correct list of runtime libraries. +#3 is convenient but version sensitive. +#2 also has cold start performance advantage because there is less dynamic library loading. +However #2 comes at a cost of bigger binaries. + +Stepping outside the Swift ecosystem, deployment of statically linked programs is often the preferred way on server centric platforms such as Linux, as it dramatically simplifies deployment of server workloads. +For reference, Go and Rust both chose to statically link programs by default for this reason. + +### Policy for system shipping with the Swift runtime libraries + +At this time, Darwin based systems are the only ones that ship with the Swift runtime libraries. +In the future, other operating systems may choose to ship with the Swift runtime libraries. +As pointed out by [John_McCall](https://forums.swift.org/u/John_McCall), in such cases the operating system vendors are likely to choose a default that is optimized for their software distribution model. +For example, they may choose to distribute a Swift toolchain defaulting to dynamic linking (against the version of the Swift runtime libraries shipped with that system) to optimize binary size and memory consumption on that system. +To support this, the Swift toolchain build script could include a preset to controls the default linking strategy. + +As pointed out by [Joe_Groff](https://forums.swift.org/u/Joe_Groff), even in such cases there is still an argument for defaulting Swift.org distributed Swift toolchains to static linking for distribution, since we want to preserve the freedom to pursue ABI-breaking improvements in top-of-tree for platforms that aren't ABI-constrained. This situation wouldn't be different from other ecosystems: The Python or Ruby that comes with macOS are set up to provide a stable runtime environment compatible with the versions of various Python- and Ruby-based tools and libraries packaged by the distribution, but developers can and do often install their own local Python/Ruby interpreter if they need a language version different from the vendor's, or a package that's incompatible with the vendor's interpreter. + +## Proposed solution + +Given that at this time Darwin based systems are the only ones that ship with the Swift runtime libraries, +we propose to make statically linking of the Swift runtime libraries SwiftPM's default behavior when building executables in release mode on non-Darwin platforms that support such linking, +with an opt-out way to disable this default behavior. + +Building in debug mode will continue to dynamically link the Swift runtime libraries, given that debugging requires the Swift toolchain to be installed and as such the Swift runtime libraries are by definition present. + +Note that SwiftPM's default behavior could change over time as its designed to reflect the state and available options of Swift runtime distribution on that platform. In other words, whether or not a SwiftPM defaults to static linking of the Swift runtime libraries is a by-product of the state of Swift runtime distribution on a platform. + +Also note this does not mean the resulting program is fully statically linked - only the Swift runtime libraries (stdlib, Foundation, Dispatch, etc) would be statically linked into the program, while external dependencies will continue to be dynamically linked and would remain a concern left to the user when deploying Swift based programs. +Such external dependencies include: +1. Glibc (including `libc.so`, `libm.so`, `libdl.so`, `libutil.so`): On Linux, Swift relies on Glibc to interact with the system and its not possible to fully statically link programs based on Glibc. In practice this is usually not a problem since most/all Linux systems ship with a compatible Glibc. +2. `libstdc++` and `libgcc_s.so`: Swift on Linux also relies on GNU's C++ standard library as well as GCC's runtime library which much like Glibc is usually not a problem because a compatible version is often already installed on the target system. +3. At this time, non-Darwin version of Foundation (aka libCoreFoundation) has two modules that rely on system dependencies (`FoundationXML` on `libxml2` and `FoundationNetworking` on `libcurl`) which cannot be statically linked at this time and require to be installed on the target system. +4. Any system dependencies the program itself brings (e.g. `libsqlite`, `zlib`) would not be statically linked and must be installed on the target system. + + +## Detailed design + +The changes proposed are focused on the behavior of SwiftPM. +We propose to change SwiftPM's default linking of the Swift runtime libraries when building executables as follows: + +### Default behavior + +* Darwin-based platforms used to support statically linking the Swift runtime libraries in the past (and never supported fully static binaries). + Today however, the Swift runtime library is shipped with the operating system and can therefore not easily be included statically in the binary. + Naturally, dynamically linking the Swift runtime libraries will remain the default on Darwin. + +* Linux and WASI support static linking and would benefit from it for the reasons highlighted above. + We propose to change the default on these platforms to statically link the Swift runtime libraries. + +* Windows may benefit from statically linking for the reasons highlighted above but it is not technically supported at this time. + As such the default on Windows will remain dynamically linking the Swift runtime libraries. + +* The default behavior on platforms not listed above will remain dynamically linking the Swift runtime libraries. + +### Opt-in vs. Opt-out + +SwiftPM's `--static-swift-stdlib` CLI flag is designed as an opt-in way to achieve static linking of the Swift runtime libraries. + +We propose to deprecate `--static-swift-stdlib` and introduce a new flag `--disable-static-swift-runtime` which is designed as an opt-out from the default behavior described above. + +Users that want to force static linking (as with `--static-swift-stdlib`) can use the long form `-Xswiftc -static-stdlib`. + +### Example + +Consider the following simple program: + +```bash +$ swift package init --type executable +Creating executable package: test +Creating Package.swift +Creating README.md +Creating .gitignore +Creating Sources/ +Creating Sources/test/main.swift +Creating Tests/ +Creating Tests/testTests/ +Creating Tests/testTests/testTests.swift + +$ cat Sources/test/main.swift +print("Hello, world!") +``` +Building the program with default dynamic linking yields the following: + +```bash +$ swift build -c release +[3/3] Build complete! + +$ ls -la --block-size=K .build/release/test +-rwxr-xr-x 1 root root 17K Dec 3 22:48 .build/release/test* + +$ ldd .build/release/test + linux-vdso.so.1 (0x00007ffc82be4000) + libswift_Concurrency.so => /usr/lib/swift/linux/libswift_Concurrency.so (0x00007f0f5cfb5000) + libswiftCore.so => /usr/lib/swift/linux/libswiftCore.so (0x00007f0f5ca55000) + libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f5c85e000) + libdispatch.so => /usr/lib/swift/linux/libdispatch.so (0x00007f0f5c7fd000) + libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0f5c7da000) + libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0f5c7d4000) + libswiftGlibc.so => /usr/lib/swift/linux/libswiftGlibc.so (0x00007f0f5c7be000) + libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0f5c5dc000) + libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0f5c48d000) + libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0f5c472000) + /lib64/ld-linux-x86-64.so.2 (0x00007f0f5d013000) + libicui18nswift.so.65 => /usr/lib/swift/linux/libicui18nswift.so.65 (0x00007f0f5c158000) + libicuucswift.so.65 => /usr/lib/swift/linux/libicuucswift.so.65 (0x00007f0f5bf55000) + libicudataswift.so.65 => /usr/lib/swift/linux/libicudataswift.so.65 (0x00007f0f5a4a2000) + librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0f5a497000) + libBlocksRuntime.so => /usr/lib/swift/linux/libBlocksRuntime.so (0x00007f0f5a492000) +``` + +Building the program with static linking of the Swift runtime libraries yields the following: + +```bash +$ swift build -c release --static-swift-stdlib +[3/3] Build complete! + +$ ls -la --block-size=K .build/release/test +-rwxr-xr-x 1 root root 35360K Dec 3 22:50 .build/release/test* + +$ ldd .build/release/test + linux-vdso.so.1 (0x00007fffdaafa000) + libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdd521c5000) + libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdd521a2000) + libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdd51fc0000) + libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdd51e71000) + libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdd51e56000) + libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdd51c64000) + /lib64/ld-linux-x86-64.so.2 (0x00007fdd54211000) +``` + +These snippets demonstrates the following: +1. Statically linking of the Swift runtime libraries increases the binary size from 17K to ~35M. +2. Statically linking of the Swift runtime libraries reduces the dependencies reported by `ldd` to core Linux libraries. + +This jump in binary size may be alarming at first sight, but since the program is not usable without the Swift runtime libraries, the actual size of the deployable unit is similar. + +```bash +$ mkdir deps + +$ ldd ".build/release/test" | grep swift | awk '{print $3}' | xargs cp -Lv -t ./deps +'/usr/lib/swift/linux/libswift_Concurrency.so' -> './deps/libswift_Concurrency.so' +'/usr/lib/swift/linux/libswiftCore.so' -> './deps/libswiftCore.so' +'/usr/lib/swift/linux/libdispatch.so' -> './deps/libdispatch.so' +'/usr/lib/swift/linux/libswiftGlibc.so' -> './deps/libswiftGlibc.so' +'/usr/lib/swift/linux/libicui18nswift.so.65' -> './deps/libicui18nswift.so.65' +'/usr/lib/swift/linux/libicuucswift.so.65' -> './deps/libicuucswift.so.65' +'/usr/lib/swift/linux/libicudataswift.so.65' -> './deps/libicudataswift.so.65' +'/usr/lib/swift/linux/libBlocksRuntime.so' -> './deps/libBlocksRuntime.so' + +$ ls -la --block-size=K deps/ +total 42480K +drwxr-xr-x 2 root root 4K Dec 3 22:59 ./ +drwxr-xr-x 6 root root 4K Dec 3 22:58 ../ +-rw-r--r-- 1 root root 17K Dec 3 22:59 libBlocksRuntime.so +-rw-r--r-- 1 root root 432K Dec 3 22:59 libdispatch.so +-rwxr-xr-x 1 root root 27330K Dec 3 22:59 libicudataswift.so.65* +-rwxr-xr-x 1 root root 4030K Dec 3 22:59 libicui18nswift.so.65* +-rwxr-xr-x 1 root root 2403K Dec 3 22:59 libicuucswift.so.65* +-rwxr-xr-x 1 root root 520K Dec 3 22:59 libswift_Concurrency.so* +-rwxr-xr-x 1 root root 7622K Dec 3 22:59 libswiftCore.so* +-rwxr-xr-x 1 root root 106K Dec 3 22:59 libswiftGlibc.so* +``` + +## Impact on existing packages + +The new behavior will take effect with a new version of SwiftPM, and packages build with that version will be linked accordingly. + +* Deployment of applications using "bag of shared objects" technique (#1 above) will continue to work as before (though would be potentially redundant). +* Deployment of applications using explicit static linking (#2 above) will continue to work and emit a warning that its redundant. +* Deployment of applications using docker "runtime" images (#3 above) will continue to work as before (though would be redundant). + +### Additional validation when linking libraries + +SwiftPM currently performs no validation when linking libraries into an executable that statically links the Swift runtime libraries. +This means that users can mistakenly link a library that already has the Swift runtime libraries statically linked into the executable that will also statically link the Swift runtime libraries, which could lead to runtime errors if the versions of the Swift runtime libraries do not match. +As part of this proposal, SwiftPM will gain a new post build validation checking for this condition and warning the user accordingly. + +## Alternatives considered and future directions + +The most obvious question this proposal brings is why not fully statically link the program instead of statically linking only the runtime libraries. +Go is a good example for creating fully statically linked programs, contributing to its success in the server ecosystem at large. +Swift already offers a flag for this linking mode: `-Xswiftc -static-executable`, but in reality Swift's ability to create fully statically linked programs is constrained. +This is mostly because today, Swift on Linux only supports GNU's libc (Glibc) and GNU's C++ standard library which do not support producing fully static binaries. +A future direction could be to look into supporting the `musl libc` and LLVM's `libc++` which should be able to produce fully static binaries. + +Further, Swift has good support to interoperate with C libraries installed on the system. +Whilst that is a nice feature, it does make it difficult to create fully statically linked programs because it would be necessary to make sure each and every of these dependencies is available in a fully statically linked form with all the common dependencies being compatible. +For example, it is not possible to link a binary that uses the `musl libc` with libraries that expect to be statically linked with Glibc. +As Swift's ability to create fully statically linked programs improves, we should consider changing the default from `-Xswiftc -static-stdlib` to `-Xswiftc -static-executable`. + +A more immediate future direction which would improve programs that need to use of FoundationXML and FoundationNetworking is to replace the system dependencies of these modules with native implementation. +This is outside the scope of this proposal which focuses on SwiftPM's behavior. + +Another alternative is to do nothing. +In practice, this proposal does not add new features, it only changes default behavior which is already achievable with the right knowledge of build flags. +That said, we believe that changing the default will make using Swift on non-Darwin platforms easier, saving time and costs to Swift users on such platforms. + +The spelling of the new flag `--disable-static-swift-runtime` is open to alternative ideas, e.g. `--disable-static-swift-runtime-libraries`. diff --git a/proposals/0343-top-level-concurrency.md b/proposals/0343-top-level-concurrency.md new file mode 100644 index 0000000000..e11e040563 --- /dev/null +++ b/proposals/0343-top-level-concurrency.md @@ -0,0 +1,213 @@ +# Concurrency in Top-level Code + +* Proposal: [SE-0343](0343-top-level-concurrency.md) +* Authors: [Evan Wilde](https://github.com/etcwilde) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.7)** +* Implementation: [Fix top-level global-actor isolation crash](https://github.com/apple/swift/pull/40963), [Add `@MainActor @preconcurrency` to top-level variables](https://github.com/apple/swift/pull/40998), [Concurrent top-level inference](https://github.com/apple/swift/pull/41061) + +## Introduction + +Bringing concurrency to top-level code is an expected continuation of the +concurrency work in Swift. This pitch looks to iron out the details of how +concurrency will work in top-level code, specifically focusing on how top-level +variables are protected from data races, and how a top-level code context goes +from a synchronous context to an asynchronous context. + +Swift-evolution thread: [Discussion thread topic for concurrency in top-level code](https://forums.swift.org/t/concurrency-in-top-level-code/55001) + +## Motivation + +The top-level code declaration context works differently than other declaration +spaces. As such, adding concurrency features to this spaces results in questions +that have not yet been addressed. + +Variables in top-level code behave as a global-local hybrid variable; they exist +in the global scope and are accessible as global variables within the module, +but are initialized sequentially like local variables. Global variables are +dangerous, especially with concurrency. There are no isolation guarantees made, +and are therefore subject to race conditions. + +As top-level code is intended as a safe space for testing out features and +writing pleasant little scripts, this simply will not do. + +In addition to the strange and dangerous behavior of variables, changing whether +a context is synchronous or asynchronous has an impact on how function overloads +are resolved, so simply flipping a switch could result in some nasty hidden +semantic changes, potentially breaking scripts that already exist. + +## Proposed solution + +The solutions will only apply when the top-level code is an asynchronous +context. As a synchronous context, the behavior of top-level code does not +change. In order to trigger making the top-level context an asynchronous context, I +propose using the presence of an `await` in one of the top-level expressions. + +An await nested within a function declaration or a closure will not trigger the +behavior. + +```swift +func doAsyncStuff() async { + // ... +} + +let countCall = 0 + +let myClosure = { + await doAsyncStuff() // `await` does not trigger async top-level + countCall += 1 +} + +await myClosure() // This `await` will trigger an async top-level +``` + +Top-level global variables are implicitly assigned a `@MainActor` global actor +isolation to prevent data races. To avoid breaking sources, the variable is +implicitly marked as pre-concurrency up to Swift 6. + +```swift +var a = 10 + +func bar() { + print(a) +} + +bar() + +await something() // make top-level code an asynchronous context +``` + +After Swift 6, full actor-isolation checking will take place. The usage of `a` +in `bar` will result in an error due to `bar` not being isolated to the +`MainActor`. In Swift 5, this will compile without errors. + +## Detailed design + +### Asynchronous top-level context inference + +The rules for inferring whether the top-level context is an asynchronous context +are the same for anonymous closures, specified in [SE-0296 Async/Await](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md#closures). + +The top-level code is inferred to be an asynchronous context if it contains a +suspension point in the immediate top-level context. + +```swift +func theAnswer() async -> Int { 42 } + +async let a = theAnswer() // implicit await, top-level is async + +await theAnswer() // explicit await, top-level is async + +let numbers = AsyncStream(Int.self) { continuation in + Task { + for number in 0 .. < 10 { + continuation.yield(number) + } + continuation.finish() + } +} + +for await number in numbers { // explicit await, top-level is asnyc + print(number) +} +``` + +The above example demonstrates each kind of suspension point, triggering an +asynchronous top-level context. Specifically, `async let a = theAnswer()` +involves an implicit suspension, `await theAnswer()` involves an explicit +suspension, as does `for await number in numbers`. Any one of these is +sufficient to trigger the switch to an asynchronous top-level context. + +Not that the inference of `async` in the top-level does not propagate to +function and closure bodies, because those contexts are separably asynchronous +or synchronous. + +```swift +func theAnswer() async -> Int { 42 } + +let closure1 = { @MainActor in print(42) } +let closure2 = { () async -> Int in await theAnswer() } +``` + +The top-level code in the above example is not an asynchronous context because +the top-level does not contain a suspension point, either explicit or implicit. + +The mechanism for inferring whether a closure body is an asynchronous context +lives in the `FindInnerAsync` ASTWalker. With minimal effort, the +`FindInnerAsync` walker can be generalized to handle top-level code bodies, +maintaining the nice parallel inference behaviour between top-level code and +closure body asynchronous detection. + +### Variables + +Variables in top-level code are initialized sequentially like a local variable, +but are in the global scope and are otherwise treated as global variables. To +prevent data races, variables should implicitly be isolated to the main actor. +It would be a shame if every top-level variable access had to go through an +`await` though. Luckily, like the other entrypoints, top-level code runs on the +main thread, so we can make the top-level code space implicitly main-actor +isolated so the variables can be accessed and modified directly. This is still +source-breaking though; a synchronous global function written in the top-level +code will emit an error because the function is not isolated to the main actor +when the variable is. While the diagnostic is correct in stating that there is a +potential data-race, the source-breaking effect is also unfortunate. To +alleviate the source break, the variable is implicitly annotated with the +`@preconcurrency` attribute. The attribute only applies to Swift 5 code, and +once the language mode is updated to Swift 6, these data races will become hard +errors. + +If `-warn-concurrency` is passed to the compiler and there is an `await` in +top-level code, the warnings are hard errors in Swift 5, as they would in any +other asynchronous context. If there is no `await` and the flag is passed, +variables are implicitly protected by the main actor and concurrency checking is +strictly enforced, even though the top-level is not an asynchronous context. +Since the top-level is not an asynchronous context, no run-loops are created +implicitly and the overload resolution behavior does not change. + +In summary, top-level variable declarations behave as though they were declared +with `@MainActor @preconcurrency` in order to strike a nice balance between +data-race safety and reducing source breaks. + +Going back to the global behaviour variables, there are some additional design +details that I should point out. + +I would like to propose removing the ability to explicitly specify a global +actor on top-level variables. Top-level variables are treated like a hybrid of +global and local variables, which has some nasty consequences. The variables are +declared in the global scope, so they are assumed to be available anywhere. This +results in some nasty memory safety issues, like the following example: + +```swift +print(a) +let a = 10 +``` + +The example compiles and prints "0" when executed. The declaration `a` is +available at the `print` statement because it is a global variable, but it is +not yet initialized because initialization happens sequentially. Integer types +and other primitives are implicitly zero-initialized; however, classes are +referential types, initialized to zero, so this results in a segmentation fault +if the variable is a class type. + +Eventually, we would like to plug this hole in the memory model. The design for +that is still in development, but will likely move toward making top-level +variables local variables of the implicit main function. I am proposing that we +disallow explicit global actors to facilitate that change and reduce the source +breakage caused by that change. + +## Source compatibility + +The `await` expression cannot appear in top-level code today since the top-level +is not an asynchronous context. As the features proposed herein are enabled by +the presence of an `await` expression in the top level, there are no scripts +today that will be affected by the changes proposed in this proposal. + +## Effect on ABI stability + +This proposal has no impact on ABI. Functions and variables have the same +signature as before. + +## Acknowledgments + +Thank you, Doug, for lots of discussion on how to break this down into something +that minimizes source breakage to a level where we can introduce this to Swift 5. diff --git a/proposals/0344-distributed-actor-runtime.md b/proposals/0344-distributed-actor-runtime.md new file mode 100644 index 0000000000..b052a3531c --- /dev/null +++ b/proposals/0344-distributed-actor-runtime.md @@ -0,0 +1,1888 @@ +# Distributed Actor Runtime + +* Proposal: [SE-0344](0344-distributed-actor-runtime.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [Pavel Yaskevich](https://github.com/xedin), [Doug Gregor](https://github.com/DougGregor), [Kavon Farvardin](https://github.com/kavon), [Dario Rexin](https://github.com/drexin), [Tomer Doron](https://github.com/tomerd) +* Review Manager: [Joe Groff](https://github.com/jckarter/) +* Status: **Implemented (Swift 5.7)** +* Implementation: + * Partially available in [recent `main` toolchain snapshots](https://swift.org/download/#snapshots) behind the `-enable-experimental-distributed` feature flag. + * This flag also implicitly enables `-enable-experimental-concurrency`. +* Review threads + * [First Review](https://forums.swift.org/t/se-0344-distributed-actor-runtime/55525) ([summary](https://forums.swift.org/t/returned-for-revision-se-0344-distributed-actor-runtime/55836)) + * [Second Review](https://forums.swift.org/t/se-0344-second-review-distributed-actor-runtime/56002) ([summary](https://forums.swift.org/t/accepted-se-0344-distributed-actor-runtime/56416)) + + +## Table of Contents + +- [Distributed Actor Runtime](#distributed-actor-runtime) + - [Table of Contents](#table-of-contents) + - [Introduction](#introduction) + - [Useful links](#useful-links) + - [Motivation](#motivation) + - [Example scenario](#example-scenario) + - [Caveat: Low-level implementation details](#caveat-low-level-implementation-details) + - [Detailed design](#detailed-design) + - [The `DistributedActorSystem` protocol](#the-distributedactorsystem-protocol) + - [Implicit distributed actor properties](#implicit-distributed-actor-properties) + - [Initializing distributed actors](#initializing-distributed-actors) + - [Distributed actor initializers](#distributed-actor-initializers) + - [Initializing `actorSystem` and `id` properties](#initializing-actorsystem-and-id-properties) + - [Ready-ing distributed actors](#ready-ing-distributed-actors) + - [Ready-ing distributed actors, exactly once](#ready-ing-distributed-actors-exactly-once) + - [Resigning distributed actor IDs](#resigning-distributed-actor-ids) + - [Resolving distributed actors](#resolving-distributed-actors) + - [Invoking distributed methods](#invoking-distributed-methods) + - [Sender: Invoking a distributed method](#sender-invoking-a-distributed-method) + - [Sender: Serializing and sending invocations](#sender-serializing-and-sending-invocations) + - [Recipient: Receiving invocations](#recipient-receiving-invocations) + - [Recipient: Deserializing incoming invocations](#recipient-deserializing-incoming-invocations) + - [Recipient: Resolving the recipient actor instance](#recipient-resolving-the-recipient-actor-instance) + - [Recipient: The `executeDistributedTarget` method](#recipient-the-executedistributedtarget-method) + - [Recipient: Executing the distributed target](#recipient-executing-the-distributed-target) + - [Recipient: Collecting result/error from invocations](#recipient-collecting-resulterror-from-invocations) + - [Amendments](#amendments) + - [Initializers no longer need to accept a single DistributedActorSystem](#initializers-no-longer-need-to-accept-a-single-distributedactorsystem) + - [Future work](#future-work) + - [Variadic generics removing the need for `remoteCallVoid`](#variadic-generics-removing-the-need-for-remotecallvoid) + - [Identifying, evolving and versioning remote calls](#identifying-evolving-and-versioning-remote-calls) + - [Default distributed call target identification scheme](#default-distributed-call-target-identification-scheme) + - [Compression techniques to avoid repeatedly sending large identifiers](#compression-techniques-to-avoid-repeatedly-sending-large-identifiers) + - [Overlap with general ABI and versioning needs in normal Swift code](#overlap-with-general-abi-and-versioning-needs-in-normal-swift-code) + - [Discussion: User provided target identities](#discussion-user-provided-target-identities) + - [Resolving `DistributedActor` protocols](#resolving-distributedactor-protocols) + - [Passing parameters to `assignID`](#passing-parameters-to-assignid) + - [Alternatives considered](#alternatives-considered) + - [Define `remoteCall` as protocol requirement, and accept `[Any]` arguments](#define-remotecall-as-protocol-requirement-and-accept-any-arguments) + - [Constraining arguments, and return type with of `remoteCall` with `SerializationRequirement`](#constraining-arguments-and-return-type-with-of-remotecall-with-serializationrequirement) + - [Hardcoding the distributed runtime to make use of `Codable`](#hardcoding-the-distributed-runtime-to-make-use-of-codable) + - [Acknowledgments & Prior art](#acknowledgments--prior-art) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Changelog](#changelog) + +## Introduction + +With the recent introduction of [actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md) to the language, Swift gained powerful and foundational building blocks for expressing *thread-safe* concurrent programs. Actors guarantee thread-safety thanks to actor-isolation of mutable state they encapsulate. + +In [SE-0336: Distributed Actor Isolation][isolation] we took it a step further, guaranteeing complete isolation of state with distributed actor-isolation, and setting the stage for `distributed` method calls to be performed across process and node boundaries. + +This proposal focuses on the runtime aspects of making such remote calls possible, their exact semantics and how developers can provide their own `DistributedActorSystem` implementations to hook into the same language mechanisms, extending Swift's distributed actor model to various environments (such as cross-process communication, clustering, or even client/server communication). + +#### Useful links + +It is recommended, though not required, to familiarize yourself with the prior proposals before reading this one: + +- [SE-0336: Distributed Actor Isolation][isolation] — a detailed proposal +- Distributed Actor Runtime (this proposal) + +Feel free to reference the following library implementations which implement this proposal's library side of things: + +- [Swift Distributed Actors Library](https://www.swift.org/blog/distributed-actors/) — a reference implementation of a *peer-to-peer cluster* for distributed actors. Its internals depend on the work in progress language features and are dynamically changing along with these proposals. It is a realistic implementation that we can use as reference for these design discussions. + +## Motivation + +With distributed actor-isolation checking laid out in [SE-0336: Distributed Actor Isolation][isolation], we took the first step towards enabling remote calls being made by invoking `distributed func` declarations on distributed actors. The isolation model and serialization requirement checks in that proposal outline how we can guarantee the soundness of such distributed actor model at compile time. + +Distributed actors enable developers to build their applications and systems using the concept of actors that may be "local" or "remote", and communicate with them regardless of their location. Our goal is to set developers free from having to re-invent ad-hoc approaches to networking, serialization and error handling every time they need to embrace distributed computing. + +Instead, we aim to embrace a co-operative approach to the problem, in which: + +1. the Swift language, compiler, and runtime provide the necessary isolation checks and runtime hooks for distributed actor lifecycle management, and distributed method calls that can be turned into "messages" that can be sent to remote peers, +2. `DistributedActorSystem` library implementations, hook into the language provided cut-points, take care of the actual message interactions, e.g. by sending messages representing remote distributed method calls over the network, +3. `distributed actor` authors, who want to focus on getting things done, express their distributed API boundaries and communicate using them. They may have opinions about serialization and specifics of message handling, and should be able to configure and use the `DistributedActorSystem` of their choice to get things done. + +In general, we propose to embrace the actor style of communication for typical distributed system development, and aim to provide the necessary tools in the language, and runtime to make this a pleasant and nice default go-to experience for developers. + +Distributed actors may not serve *all* possible use-cases where networking is involved, but we believe a large group of applications and systems will benefit from them, as the ecosystem gains mature `DistributedActorSystem` implementations. + +#### Example scenario + +In this proposal we will focus only on the runtime aspects of distributed actors and methods, i.e. what happens in order to create, send, and receive messages formed when a distributed method is called on a remote actor. For more details on distributed actor isolation and other compile-time checks, please refer to [SE-0336: Distributed Actor Isolation][isolation]. + +We need to pass around distributed actors in order to invoke methods on them at some later point in time. We need those actors to declare `distributed` methods such that we have something we can message them with, and there must be some lifecycle and registration mechanisms related to them. + +One example use case we can keep in mind is a simple turn-based `Game` which showcases most of the capabilities we come to expect of distributed actors: + +```swift +distributed actor Player { + // ... + + distributed func makeMove() -> Move { ... } + + distributed func gameFinished(result: GameResult) { + if result.winner == self { + print("I WON!") + } else { + print("Player \(result.winner) won the game.") + } + } +} + +distributed actor Game { + var state: GameState = ... + + // players can be located on different nodes + var players: Set = [] + + distributed func playerJoined(_ player: Player) { + others.append(player) + if others.count >= 2 { // we need 2 other players to start a game + Task { try await self.start() } + } + } + + func start() async throws { + state = .makeNewGameState(with: players) + while !state.finished { + for player in players { + let move = try await p.makeMove() // TODO: handle failures, e.g. "move timed-out" + state.apply(move, by: player) + } + } + + let winner = state.winner + try await game.finishedResult + } +} +``` + +This code snippet showcases what kind of distributed actors one might want to implement – they represent addressable identities in a system where players may be hosted on different hosts or devices, and we'd like to communicate with any of them from the `Game` actor which manages the entire game's state. Players may be on the same host as the `Game` actor, or on different ones, but we never have to change the implementation of `Game` to deal with this – thanks to distributed actors and the concept of location transparency, we can implement this piece of code once, and run it all locally, or distributed without changing the code specifically for either of those cases. + +### Caveat: Low-level implementation details + +This proposal includes low-level implementation details in order to showcase how one can use to build a real, efficient, and extensible distributed actor system using the proposed language runtime. It is primarily written for distributed actor system authors, which need to understand the underlying mechanisms which distributed actors use. + +End users, who just want to use _distributed actors_, and not necessarily _implement_ a distributed actor system runtime, do not need to dive deep as deep into this proposal, and may be better served by reading [SE-0366: Distributed Actor Isolation][isolation] which focuses on how distributed actors are used. Reading this — runtime — proposal, however, will provide additional insights as to why distributed actors are isolated the way they are. + +This proposal focuses on how a distributed actor system runtime can be implemented. Because this language feature is extensible, library authors may step in and build their own distributed actor runtimes. It is expected that there will be relatively few, but solid actor system implementations eventually, yet their use would apply to many many more end-users than actor system developers. + +## Detailed design + +This section is going to deep dive into the runtime details and its interaction with user provided `DistributedActorSystem` implementations. Many of these aspects are not strictly necessary to internalize by end-user/developer, who only wants to write some distributed actors and have them communicate using *some* distributed actor system. + +### The `DistributedActorSystem` protocol + +At the core of everything distributed actors do, is the `DistributedActorSystem` protocol. This protocol is open to be implemented by anyone, and can be used to extend the functionality of distributed actors to various environments. + +Building a solid actor system implementation is not a trivial task, and we only expect a handful of mature implementations to take the stage eventually. + +> At the time of writing, we–the proposal authors–have released a work in progress [peer-to-peer cluster actor system implementation](https://www.swift.org/blog/distributed-actors/) that is tracking this evolving language feature. It can be viewed as a reference implementation for the language features and `DistributedActorSystem` protocol discussed in this proposal. + +Below we present the full listing of the `DistributedActorSystem` protocol, and we'll be explaining the specific methods one by one as we go: + +```swift +// Module: _Distributed + +protocol DistributedActorSystem: Sendable { + /// The type of `ID` assigned to a distributed actor while initializing with this actor system. + /// The identity should be meaningfully unique, in the sense that ID equality should mean referring to the + /// same distributed actor. + /// + /// A distributed actor created using a specific actor system will use the system's `ActorID` as + /// the `ID` type it stores and for its `Hashable` implementation. + /// + /// ### Implicit distribute actor `Codable` conformance + /// If the `ActorID` (and therefore also the `DistributedActor.ID`) conforms to `Codable`, + /// the `distributed actor` will gain an automatically synthesized conformance to `Codable` as well. + associatedtype ActorID: Sendable & Hashable + + /// The specific type of the invocation encoder that will be created and populated + /// with details about the invocation when a remote call is about to be made. + /// + /// The populated instance will be passed to the `remoteCall` from where it can be + /// used to serialize into a message format in order to perform the remote invocation. + associatedtype InvocationEncoder: DistributedTargetInvocationEncoder + + /// The specific type of invocation decoder used by this actor system. + /// + /// An instance of this type must be passed to `executeDistributedTarget` which + /// extracts arguments and applies them to the local target of the invocation. + associatedtype InvocationDecoder: DistributedTargetInvocationDecoder + + /// The serialization requirement that will be applied to all distributed targets used with this system. + /// + /// An actor system is still allowed to throw serialization errors if a specific value passed to a distributed + /// func violates some other restrictions that can only be checked at runtime, e.g. checking specific types + /// against an "allow-list" or similar. The primary purpose of the serialization requirement is to provide + /// compile time hints to developers, that they must carefully consider evolution and serialization of + /// values passed to and from distributed methods and computed properties. + associatedtype SerializationRequirement + where SerializationRequirement == InvocationEncoder.SerializationRequirement, + SerializationRequirement == InvocationDecoder.SerializationRequirement + + // ==== --------------------------------------------------------------------- + // - MARK: Actor Lifecycle + + /// Called by a distributed when it begins its initialization (in a non-delegating init). + /// + /// The system should take special care to not assign two actors the same `ID`, and the `ID` + /// must remain valid until it is resigned (see `resignID(_:)`). + func assignID(_ actorType: Actor.Type) -> ActorID + where Actor: DistributedActor, + Actor.ID == ActorID + + /// Automatically called by in every distributed actor's non-delegating initializer. + /// + /// The call is made specifically before the `self` of such distributed actor is about to + /// escape, e.g. via a function call, closure or otherwise. If no such event occurs the + /// call is made at the end of the initializer. + /// + /// The passed `actor` is the `self` of the initialized actor, and its `actor.id` is expected + /// to be of the same value that was assigned to it in `assignID`. + /// + /// After the ready call returns, it must be possible to resolve it using the 'resolve(_:as:)' + /// method on the system. + func actorReady(_ actor: Actor) + where Actor: DistributedActor, + Actor.ID == ActorID + + /// Called when the distributed actor is deinitialized (or has failed to finish initializing). + /// + /// The system may release any resources associated with this actor id, and should not make + /// further attempts to deliver messages to the actor identified by this identity. + func resignID(_ id: ActorID) + + // ==== --------------------------------------------------------------------- + // - MARK: Resolving distributed actors + + /// Resolve a local or remote actor address to a real actor instance, or throw if unable to. + /// The returned value is either a local actor or proxy to a remote actor. + /// + /// Resolving an actor is called when a specific distributed actors `init(from:)` + /// decoding initializer is invoked. Once the actor's identity is deserialized + /// using the `decodeID(from:)` call, it is fed into this function, which + /// is responsible for resolving the identity to a remote or local actor reference. + /// + /// If the resolve fails, meaning that it cannot locate a local actor managed for + /// this identity, managed by this transport, nor can a remote actor reference + /// be created for this identity on this transport, then this function must throw. + /// + /// If this function returns correctly, the returned actor reference is immediately + /// usable. It may not necessarily imply the strict *existence* of a remote actor + /// the identity was pointing towards, e.g. when a remote system allocates actors + /// lazily as they are first time messaged to, however this should not be a concern + /// of the sending side. + /// + /// Detecting liveness of such remote actors shall be offered / by transport libraries + /// by other means, such as "watching an actor for termination" or similar. + func resolve(_ id: ActorID, as actorType: Actor.Type) throws -> Actor? + where Actor: DistributedActor, + Actor.ID: ActorID, + Actor.SerializationRequirement == Self.SerializationRequirement + + // ==== --------------------------------------------------------------------- + // - MARK: Remote Target Invocations + + /// Invoked by the Swift runtime when a distributed remote call is about to be made. + /// + /// The returned `InvocationEncoder` will be populated with all + /// generic substitutions, arguments, and specific error and return types + /// that are associated with this specific invocation. + func makeInvocationEncoder() -> InvocationEncoder + + // We'll discuss the remoteCall method in detail in this proposal. + // It cannot be declared as protocol requirement, and remains an ad-hoc + // requirement like this: + /// Invoked by the Swift runtime when making a remote call. + /// + /// The `invocation` is the arguments container that was previously created + /// by `makeInvocationEncoder` and has been populated with all arguments. + /// + /// This method should perform the actual remote function call, and await for its response. + /// + /// ## Errors + /// This method is allowed to throw because of underlying transport or serialization errors, + /// as well as by re-throwing the error received from the remote callee (if able to). + /// + /// Ad-hoc protocol requirement. + func remoteCall( + on actor: Actor, + target: RemoteCallTarget, + invocation: inout InvocationEncoder, + throwing: Failure.Type, + returning: Success.Type + ) async throws -> Success + where Actor: DistributedActor, + Actor.ID == ActorID, + Failure: Error, + Success: Self.SerializationRequirement + + /// Invoked by the Swift runtime when making a remote call to a `Void` returning function. + /// + /// ( ... Same as `remoteCall` ... ) + /// + /// Ad-hoc protocol requirement. + func remoteCallVoid( + on actor: Actor, + target: RemoteCallTarget, + invocation: inout InvocationEncoder, + throwing: Failure.Type + ) async throws + where Actor: DistributedActor, + Actor.ID == ActorID, + Failure: Error +} + +/// A distributed 'target' can be a `distributed func` or `distributed` computed property. +/// +/// The actor system should encode the identifier however it sees fit, +/// and transmit it to the remote peer in order to invoke identify the target of an invocation. +public struct RemoteCallTarget: Hashable { + /// The mangled name of the invoked distributed method. + /// + /// It contains all information necessary to lookup the method using `executeDistributedActorMethod(...)` + var mangledName: String { ... } + + /// The human-readable "full name" of the invoked method, e.g. 'Greeter.hello(name:)'. + var fullName: String { ... } +} +``` + +In the following sections, we will be explaining how the various methods of a distributed system are invoked by the Swift runtime. + +### Implicit distributed actor properties + +Distributed actors have two properties that are crucial for the inner workings of actors that we'll explore during this proposal: the `id` and `actorSystem`. + +These properties are synthesized by the compiler, in every `distributed actor` instance, and they witness the `nonisolated` property requirements defined on the `DistributedActor` protocol. + +The `DistributedActor` protocol (defined in [SE-0336][isolation]), defines those requirements: + +```swift +protocol DistributedActor { + associatedtype ActorSystem: DistributedActorSystem + + typealias ID = ActorSystem.ActorID + typealias SerializationRequirement: ActorSystem.SerializationRequirement + + nonisolated var id: ID { get } + nonisolated var actorSystem: ActorSystem { get } + + // ... +} +``` + +which are witnessed by *synthesized properties* in every specific distributed actor instance. + +Next, we will discuss how those properties get initialized, and used in effectively all aspects of a distributed actor's lifecycle. + +### Initializing distributed actors + +At runtime, a *local* `distributed actor` is effectively the same as a local-only `actor`. The allocated `actor` instance is a normal `actor`. However, its initialization is a little special, because it must interact with its associated actor system to make itself available for remote calls. + +We will focus on non-delegating initializers, as they are the ones where distributed actors cause additional things to happen. + +> Please note that **initializing** a distributed actor with its `init` always returns a **local** reference to a new actor. The only way to obtain a a remote reference is by using the `resolve(id:using:)` method, which is discussed in [Resolving Distributed Actors](#resolving-distributed-actors). + +Distributed actor initializers inject a number of calls into specific places of the initializer's body. These calls allow for the associated actor system to manage the actor's identity, and availability to remote calls. Before we dive into the details, the following diagram outlines the various calls that will be explained in this section: + +``` +┌────────────────────────────┐ ┌──────────────────────────┐ +│ distributed actor MyActor │ │ MyDistributedActorSystem │ +└────────────────────────────┘ └──────────────────────────┘ + │ │ + init(...) │ + │ │ + │── // self.id = actorSystem.assignID(Self.self) ───▶│ Generate and reserve ID + │ │ + │ // self.actorSystem = system │ + │ │ + │ │ + │ │ + │── // actorSystem.actorReady(self) ────────────────▶│ Store a mapping (ID -> some DistributedActor) + │ │ + ... ... + │ │ + ◌ deinit ─ // actorSystem.resignID(self.id) ────────▶│ Remove (ID -> some DistributedActor) mapping + │ + ... +``` + +### Distributed actor initializers + +A non-delegating initializer of a type must *fully initialize* it. The place in code where an actor becomes fully initialized has important and specific meaning to actor isolation which is defined in depth in [SE-0327: On Actors and Initialization](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md). Not only that, but once fully initialized it is possible to escape `self` out of a (distributed) actor's initializer. This aspect is especially important for distributed actors, because it means that once fully initialized they _must_ be registered with the actor system as they may be sent to other distributed actors and even sent messages to. + +All non-delegating initializers must store an instance of a `DistributedActorSystem` conforming type into their `self.actorSystem` property. + +This is an improvement over the initially proposed semantics in [SE-0336: Distributed Actor Isolation][isolation], where the initializers must have accepted a _single_ distributed actor system argument and would initialize the synthesized stored property automatically. + +This proposal amends this behavior to the following semantics: + +- the default initializer gains a `actorSystem: Self.ActorSystem` parameter. +- other non-delegating initializers must initialize the `self.actorSystem` property explicitly. + - it is recommended to accept an actor system argument and store it + - technically it is also possible to store a global actor system to this property, however this is generally an anti-pattern as it hurts testability of such actor (i.e. it becomes impossible to swap the actor system for a "for testing" one during test execution). + +```swift +distributed actor DA { + // synthesized: + // init(actorSystem: Self.ActorSystem) {} // ✅ no user defined init, so we synthesize a default one +} + +distributed actor DA2 { + init(system: Self.ActorSystem) { // ⚠️ ok declaration, but self.actorSystem was not initialized + // ❌ error: implicit stored property 'actorSystem' was not initialized + } + + init(other: Int, actorSystem system: Self.ActorSystem) { // ✅ ok + self.actorSystem = system + } +} +``` + +Now in the next sections, we will explore in depth why this parameter was necessary to enforce to begin with. + +#### Initializing `actorSystem` and `id` properties + +The two properties (`actorSystem` and `id`) are synthesized _stored_ properties in the body of every actor. + +Users must initialize the `actorSystem`, however management of the `id` is left up to the compiler to synthesize. All properties of an actor have to be initialized for the type to become fully initialized, same as with any other type. However, the initialization of `id` is left to the compiler in order to streamline the initialization, as well as keep the _strict_ contract between the stored actor system and that the `ID` _must_ be assigned by that exact actor system. It would be illegal to just assign an `id` from some other source because of how tightly it relates to the actors lifecycle. + +The compiler synthesizes code that does this in any designated initializer of distributed actors: + +```swift +distributed actor DA { + // let id: ID + // let actorSystem: ActorSystem + + init(actorSystem: Self.ActorSystem) { + self.actorSystem = actorSystem + // ~~~ synthesized ~~~ + // ... + // ~~~ end of synthesized ~~~ + } +} +``` + +Synthesizing the id assignment means that we need to communicate with the `system` used to initialize the distributed actor, for it is the `ActorSystem` that allocates and manages identifiers. In order to obtain a fresh `ID` for the actor being initialized, we need to call `system`'s `assignID` method. This is done before any user-defined code is allowed to run in the actors designated initializer, like this: + +```swift +distributed actor DA { + let number: Int + + init(system: ActorSystem) { + self.actorSystem = system + // ~~ injected property initialization ~~ + // self.id = system.assignID(Self.self) + // ~~ end of injected property initialization ~~ + + // user-defined code follows... + self.number = 42 + + // ... + } +} +``` + +### Ready-ing distributed actors + +So far, the initialization process was fairly straightforward. We only needed to find a way to initialize the stored properties, and that's it. There is one more step though that is necessary to make distributed actors work: "ready-ing" the actor. + +As the actor becomes fully initialized, the type system allows escaping its `self` through method calls or closures. There are a number of rules which govern the isolation state of `self` of any actor during its initializer, which are fully explained in: [SE-0327: On Actor Initialization](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md). Distributed actor initializers are subject to the same rules, and in addition to that they inject an `actorReady(self)` at the point where self would have become nonisolated under the rules explained in SE-0327. + +This call is necessary in order for the distributed actor system to be able to resolve an incoming `ID` (that it knows, since it assigned it) to a specific distributed actor instance (which it does not know, until `actorReady` is called on the system). This means that there is a state between the `assignID` and `actorReady` calls, during which the actor system cannot yet properly resolve the actor. + +A distributed actor becomes "ready", and transparently invokes `actorSystem.ready(self)`, during its non-delegating initializer just _before_ the actor's `self` first escaping use, or at the end of the initializer if no explicit escape is found. + +This rule is not only simple to remember, but also consistent between synchronous and asynchronous initializers. The rule plays also very well with the flow-isolation treatment of self in plain actors. + +The following snippets illustrate where the ready call is emitted by the compiler: + +```swift +distributed actor DA { + let number: Int + + init(sync system: ActorSystem) { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + self.number = 42 + // << system.actorReady(self) + } + + init(sync system: ActorSystem) { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + self.number = 42 + // << system.actorReady(self) + Task.detached { // escaping use of `self` + await self.hello() + } + } +} +``` + +If the self of the actor were to be escaped on multiple execution paths, the ready call is injected in all appropriate paths, like this: + +```swift +distributed actor DA { + let number: Int + init(number: Int, system: ActorSystem) async { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + if number % 2 == 0 { + print("even") + self.number = number + // << system.actorReady(self) + something(self) + } else { + print("odd") + self.number = number + // << system.actorReady(self) + something(self) + } + } +} +``` + +Special care needs to be taken about the distributed actor and actor system interaction in the time between the `assignID` and `actorReady` calls, because during this time the system is unable to *deliver* an invocation to the target actor. However, it is always able to recognize that an ID is known, but just not ready yet – the system did create and assign the ID after all. + +This should not be an issue for developers using distributed actors, but actor system authors need to be aware of this interval between the actor ID being reserved and readies. We suggest "reserving" the ID immediately in `assignID` in order to avoid issuing the same ID to multiple actors which can yield unexpected behavior when handling incoming messages. + +Another thing to be aware of is "long" initializers, which take a long time to complete which may sometimes be the case with asynchronous initializers. For example, consider this initializer which performs a lot of work on the passed in items before returning: + +```swift +init(items: [Item], system: ActorSystem) async { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + for await item in items { + await compute(item) + } + // ... + // ... + // ?? what if init "never" returns" ?? + // ... + // ... + // << system.actorReady(self) +} +``` + +This is arguably problematic for any class, struct or actor, however for distributed actors this also means that the period of time during an ID was assigned and will finally be readied can be potentially quite long. In general, we discourage such "long running" initializers as they make use of the actor in distribution impossible until it is readied. On the other hand, though, it can only be used in distribution once the initializer returns in any case so this is a similar problem to any long running initializer. + +#### Ready-ing distributed actors, exactly once + +Another interesting case the `actorReady` synthesis in initializers needs to take care of is triggering the `actorReady` call only *once*, as the actor first becomes fully initialized. The following snippet does a good job showing an example of where it can manifest: + +```swift +distributed actor DA { + var int: Int + init(system: ActorSystem) async { + self.actorSystem = system + var loops = 10 + while loops > 0 { + self.int = loops + // ~ AT THE FIRST ITERATION ~ + // ~ become fully initialized ~ + // ... + escape(self) + + loops -= 1 + } + } +} +``` + +This actor performs a loop during which it assigns values to `self.int`, the actor becomes fully initialized the first time this loop runs. + +We need to emit the `actorReady(self)` call, only once, and we should not repeatedly call the actor system's `actorReady` method which would force system developers into weirdly defensive implementations of this method. Thankfully, this is possible to track in the compiler, and we can emit the ready call only once, based on internal initialization marking mechanisms (that store specific bits for every initialized field). + +The synthesized (pseudo-)code therefore is something like this: + +```swift +distributed actor DA { + var int: Int + init(system: ActorSystem) { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + // MARK INITMAP[id] = INITIALIZED + + var loops = 10 + while loops > 0 { + self.int = loops + // MARK INITMAP[int] = INITIALIZED + // INITMAP: FULLY INITIALIZED + // + // IF INITMAP[IMPLICIT_HOP_TO_SELF] != DONE { + // << + // MARK INITMAP[IMPLICIT_HOP_TO_SELF] = DONE + // } + // + // IF INITMAP[ACTOR_READY] != INITIALIZED { + // << system.actorReady(self) + // MARK INITMAP[ACTOR_READY] = INITIALIZED + // } + escape(self) + + loops -= 1 + } + } +} +``` + +Using this technique we are able to emit the ready call only once, and put off the complexity of dealing with repeated ready calls from distributed actor system library authors. + +> The same technique is used to avoid hopping to the self executor 10 times, and the implicit hop-to-self is only performed once, on the initial iteration where the actor became fully initialized. + +Things get more complex in the face of failable as well as throwing initializers. Specifically, because we not only have to assign identities, we also need to ensure that they are resigned when the distributed actor is deallocated. In the simple, non-throwing initialization case this is simply done in the distributed actor's `deinit`. However, some initialization semantics make this more complicated. + +### Resigning distributed actor IDs + +In addition to assigning `ID` instances to specific actors as they get created, we must also *always* ensure the `ID`s assigned are resigned as their owning actors get destroyed. + +Resigning an `ID` allows the actor system to release any resources it might have held in association with this actor. Most often this means removing it from some internal lookup table that was used to implement the `resolve(ID) -> Self` method of a distributed actor, but it could also imply tearing down connections, clearing caches, or even dropping any in-flight messages addressed to the now terminated actor. + +In the simple case this is trivially solved by deinitialization: we completely initialize the actor, and once it deinitializes, we invoke `resignID` in the actor's deinitializer: + +```swift +deinit { + // << self.actorSystem.resignID(self.id) +} +``` + +This also works with user defined deinitializers, where the resign call is injected as the *first* operation in the deinitializer: + +```swift +// user-defined deinit +deinit { + // << self.actorSystem.resignID(self.id) + print("deinit \(self.id)") +} +``` + +Things get more complicated once we take into account the existence of *failable* and *throwing* initializers. Existing Swift semantics around those types of initializers, and their effect on if and when `deinit` is invoked mean that we need to take special care of them. + +Let us first discuss [failable initializers](https://docs.swift.org/swift-book/LanguageGuide/Initialization.html#ID224), i.e. initializers which are allowed to assign `nil` during their initialization. As actors allow such initializers, distributed actors should too, in order to make the friction of moving from local-only to distributed actors as small as possible. + +```swift +distributed actor DA { + var int: Int + + init?(int: Int, system: ActorSystem) { + self.actorSystem = system + // << self.id = actorSystem.assignID(Self.self) + // ... + if int < 10 { + // ... + // << self.actorSystem.resignID(self.id) + return nil + } + self.int = int + // << self.actorSystem.actorReady(self) + } + + // deinit { + // << self.actorSystem.resignID(self.id) + // } +} +``` + +Due to rules about actor and class init/deinit, when we `return nil` from a failable initializer, its deinitializer *does not run* (!). Because of this, we cannot rely on the deinit to resign the ID as we'd leave an un-used, but still registered identity hanging in the actor system, and the `resignID` is injected just before the "failing return" from such an initializer. This is done transparently, and neither distributed actor developers nor actor system developers need to worry about this: the ID is always resigned properly. + +> This does mean that `resignID` may be called without `actorReady` having ever been called! The system should react to this as it would to any usual resignID and free any resources associated with the identifier. + +Next, we need to discuss *throwing* initializers, and their multiple paths of execution. Again, rules about class and actor deinitialization, are tightly related to whether a type's `deinit` will be executed or not, so let us analyze the following example: + +```swift +distributed actor DA { + var int: Int + + init(int: Int, system: ActorSystem) throws { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + // ... + if int <= 1 { + // ... + // << self.actorSystem.resignID(self.id) + throw Boom() // [1] + } + + if int <= 2 { + self.int = int + // ~ become fully initialized ~ + throw Boom() // [2] + } + + throw Boom() // Boom for good measure... (same as [2] though) + // theoretically, the ready call is inserted at the end of the init: + // << system.actorReady(self) + } + + init(int: Int, system: ActorSystem) async throws { + // << self.id = system.assignID(Self.self) + // ... + if int <= 1 { + // ... + // << self.actorSystem.resignID(self.id) + throw Boom() // [1] + } + + if int <= 2 { + self.int = int + // ~ become fully initialized ~ + // << system.actorReady(self) + throw Boom() // [2] + } + + throw Boom() // Boom for good measure... (same as [2] though) + } + + // deinit { + // << self.actorSystem.resignID(self.id) + // } +} +``` + +The actor shown above both has state that it needs to initialize, and it is going to throw. It will throw either before becoming fully initialized `[1]`, or after it has initialized all of its stored properties `[2]`. Swift handles those two executions differently. Only a fully initialized reference type's `deinit` is going to be executed. This means that if the `init` throws at `[1]` we need to inject a `resignID` call there, while if it throws after becoming fully initialized, e.g. on line `[2]` we do not need to inject the `resignID` call, because the actor's `deinit` along with the injected-there `resignID` will be executed instead. + +Both the synchronous and asynchronous initializers deal with this situation well, because the resign call must be paired with the assign, and if the actor was called ready before it calls `resignID` it does not really impact the resignation logic. + +To summarize, the following are rules that distributed actor system implementors can rely on: + +- `assignID(_:)` will be called exactly-once, at the very beginning of the initialization of a distributed actor associated with the system. +- `actorReady(_:)` will be called exactly-once, after all other properties of the distributed actor have been initialized, and it is ready to receive messages from other peers. By construction, it will also always be called after `assignID(_:)`. +- `resignID(_:)` will be called exactly-once as the actor becomes deinitialized, or fails to finish its initialization. This call will always be made after an `assignID(_:)` call. While there may be ongoing racy calls to the transport as the actor invokes this method, any such calls after `resignID(_:)` was invoked should be handled as if actor never existed to begin with. + +Note that the system usually should not hold the actor with a strong reference, as doing so inhibits its ability to deinit until the system lets go of it. + +### Resolving distributed actors + +Every distributed actor type has a static "resolve" method with the following signature: + +```swift +extension DistributedActor { + public static func resolve(id: Self.ID, using system: Self.ActorSystem) throws -> Self { + ... + } +} +``` + +This method will return a distributed actor reference, or throw when the actor system is unable to resolve the reference. + +The `resolve(id:using:)` method on distributed actors is an interesting case of the Swift runtime collaborating with the `DistributedActorSystem`. The Swift runtime implements this method as calling the passed-in actor system to resolve the ID, and if the system claims that this is a _remote_ reference, the Swift runtime will allocate a _remote_ distributed actor reference, sometimes called a "proxy" instance. + +Its implementation can be thought of as follows: + +```swift +extension DistributedActor { + // simplified implementation + static func resolve(id: Self.ID, using system: ActorSystem) throws -> Self { + switch try system.resolve(id: id, as: Self.self) { + case .some(let localInstance): + return localInstance + case nil: + return <>(id: id, system: system) + } + } +} +``` + +Specifically, this calls into the `ActorSystem`'s `resolve(id:as:)` method which has a slightly different signature than the one defined on actors, specifically it can return `nil` to signal the instance is not found in this actor system, but we're able to proxy it. + +> The **result** of `resolve(id:using:)` may be a **local instance** of a distributed actor, or a **reference to a remote** distributed actor. + +The resolve implementation should be fast, and should be non-blocking. Specifically, it should *not* attempt to contact the remote peer to confirm whether this actor really exists or not. Systems should blindly resolve remote identifiers assuming the remote peer will be able to handle them. Some systems may after all spin up actor instances lazily, upon the first message sent to them etc. + +Allocating the remote reference is implemented by the Swift runtime, by creating a fixed-size object that serves only the purpose of proxying calls into the `system.remoteCall`. The `_isDistributedRemoteActor()` function always returns `true` for such a reference. + +If the system entirely fails to resolve the ID, e.g. because it was ill-formed or the system is unable to handle proxies for the given ID, it must throw with an error conforming to `DistribtuedActorSystemError`, rather than returning `nil`. An example implementation could look something like this: + +```swift +final class ClusterSystem: DistributedActorSystem { + private let lock: Lock + private var localActors: [ActorID: AnyWeaklyHeldDistributedActor] // stored into during actorReady + + // example implementation; more sophisticated ones can exist, but boil down to the same idea + func resolve(id: ID, as actorType: Actor.Type) + throws -> Actor? where Actor: DistributedActor, + Actor.ID == Self.ActorID, + Actor.SerializationRequirement == Self.SerializationRequirement { + if validate(id) == .illegal { + throw IllegalActorIDError(id) + } + + return lock.synchronized { + guard let known = self.localActors[id] else { + return nil // not local actor, but we can allocate a remote reference for it + } + + return try known.as(Actor.self) // known managed local instance + } + } +} +``` + +The types work out correctly since it is the specific actor system that has _assigned_ the `ID`, and stored the specific distributed actor instance for the specific `ID`. + +> **Note:** Errors thrown by actor systems should conform to the `protocol DistributedActorSystemError: Error {}` protocol. While it is just a marker protocol, it helps end users understand where an error originated. + +Attempting to ready using one type, and resolve using another will cause a throw to happen during the resolve, e.g. like this: + +```swift +distributed actor One {} +distributed actor Two {} + +let one = One(system: cluster) +try Two.resolve(id: one.id, using: cluster) +// throws: DistributedActorResolveError.wrongType(found: One) // system specific error +``` + +This is only the case for local instances, though. For remote instances, by design, the local actor system does not track any information about them and as any remote call can fail anyway, the failures surface at call-site (as the remote recipient will fail to be resolved). + +### Invoking distributed methods + +Invoking a distributed method (or distributed computed property) involves a number of steps that occur on two "sides" of the call. + +The local/remote wording which works well with actors in general can get slightly confusing here, because every call made "locally" on a "remote reference", actually results in a "local" invocation execution on the "remote system". Instead, we will be using the terms "**sender**" and "**recipient**" to better explain which side of a distributed call we are focusing on. + +As was shown earlier, invoking a `distributed func` essentially can follow one of two execution paths: + +- if the distributed actor instance was **local**: + - the call is made directly, as if it was a plain-old local-only `actor` +- if the distributed actor was **remote**: + - the call must be transformed into an invocation that will be offered to the `system.remoteCall(...)` method to execute + +The first case is governed by normal actor execution rules. There might be a context switch onto the actor's executor, and the actor will receive and execute the method call as usual. + +In this section, we will explain all the steps involved in the second, remote, case of a distributed method call. The invocations will be using two very important types that represent the encoding and decoding side of such distributed method invocations. + +The full listing of those types is presented below: + +```swift +protocol DistributedActorSystem: ... { + // ... + associatedtype InvocationEncoder: DistributedTargetInvocationEncoder + associatedtype InvocationDecoder: DistributedTargetInvocationDecoder + + func makeInvocationEncoder() -> InvocationEncoder +} +``` + + + +```swift +public protocol DistributedTargetInvocationEncoder { + associatedtype SerializationRequirement + + /// Record a type of generic substitution which is necessary to invoke a generic distributed invocation target. + /// + /// The arguments must be encoded order-preserving, and once `decodeGenericSubstitutions` + /// is called, the substitutions must be returned in the same order in which they were recorded. + mutating func recordGenericSubstitution(_ type: T.Type) throws + + /// Record an argument of `Argument` type in this arguments storage. + /// + /// The argument name is provided to the `name` argument and can either be stored or ignored. + /// The value of the argument is passed as the `argument`. + /// + /// Ad-hoc protocol requirement. + mutating func recordArgument( + _ argument: RemoteCallArgument + ) throws + + /// Record the error type thrown by the distributed invocation target. + /// If the target does not throw, this method will not be called and the error type can be assumed `Never`. + mutating func recordErrorType(_ type: E.Type) throws + + /// Record the return type of the distributed method. + /// If the target does not return any specific value, this method will not be called and the return type can be assumed `Void`. + /// + /// Ad-hoc protocol requirement. + mutating func recordReturnType(_ type: R.Type) throws + + /// All values and types have been recorded. + /// Optionally "finalize" the recording, if necessary. + mutating func doneRecording() throws +} + +/// Represents an argument passed to a distributed call target. +public struct RemoteCallArgument { + /// The "argument label" of the argument. + /// The label is the name visible name used in external calls made to this + /// target, e.g. for `func hello(label name: String)` it is `label`. + /// + /// If no label is specified (i.e. `func hi(name: String)`), the `label`, + /// value is empty, however `effectiveLabel` is equal to the `name`. + /// + /// In most situations, using `effectiveLabel` is more useful to identify + /// the user-visible name of this argument. + let label: String? + var effectiveLabel: String { + return label ?? name + } + + /// The internal name of parameter this argument is accessible as in the + /// function body. It is not part of the functions API and may change without + /// breaking the target identifier. + /// + /// If the method did not declare an explicit `label`, it is used as the + /// `effectiveLabel`. + let name: String + + /// The value of the argument being passed to the call. + /// As `RemoteCallArgument` is always used in conjunction with + /// `recordArgument` and populated by the compiler, this Value will generally + /// conform to a distributed actor system's `SerializationRequirement`. + let value: Value +} +``` + + + +```swift +public protocol DistributedTargetInvocationDecoder { + associatedtype SerializationRequirement + + mutating func decodeGenericSubstitutions() throws -> [Any.Type] + + /// Attempt to decode the next argument from the underlying buffers into pre-allocated storage + /// pointed at by 'pointer'. + /// + /// This method should throw if it has no more arguments available, if decoding the argument failed, + /// or, optionally, if the argument type we're trying to decode does not match the stored type. + /// + /// Ad-hoc protocol requirement. + mutating func decodeNextArgument() throws -> Argument + + mutating func decodeErrorType() throws -> Any.Type? + + mutating func decodeReturnType() throws -> Any.Type? +} +``` + +#### Sender: Invoking a distributed method + +A call to a distributed method (or computed property) on a remote distributed actor reference needs to be turned into a runtime introspectable representation which will be passed to the `remoteCall` method of a specific distributed actor system implementation. + +In this section, we'll see what happens for the following `greet(name:)` distributed method call: + +```swift +// distributed func greet(name: String) -> String { ... } + +try await greeter.greet(name: "Alice") +``` + +Such invocation is calling the method via a "distributed thunk" rather than directly. The "distributed thunk" is synthesized by the compiler for every `distributed func`, and can be illustrated by the following snippet: + +```swift +// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SYNTHESIZED ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +extension Greeter { + // synthesized; not user-accessible thunk for: greet(name: String) -> String + nonisolated func greet_$distributedThunk(name: String) async throws -> String { + guard _isDistributedRemoteActor(self) else { + // the local func was not throwing, but since we're nonisolated in the thunk, + // we must hop to the target actor here, meaning the 'await' is always necessary. + return await self.greet(name: name) + } + + // [1] prepare the invocation object: + var invocation = self.actorSystem.makeInvocationEncoder() + + // [1.1] if method has generic parameters, record substitutions + // e.g. for func generic(a: A, b: B) we would get two substitutions, + // for the generic parameters A and B: + // + // << invocation.recordGenericSubstitution() + // << invocation.recordGenericSubstitution() + + // [1.2] for each argument, synthesize a specialized recordArgument call: + try invocation.recordArgument(RemoteCallArgument(label: nil, name: "name", value: name)) + + // [1.3] if the target was throwing, record Error.self, + // otherwise do not invoke recordErrorType at all. + // + // << try invocation.recordErrorType(Error.self) + + // [1.4] we also record the return type; it may or may not be necessary to + // transmit over the wire but if necessary, the system may choose to do so. + // + // This call is not made when the return type is Void. + try invocation.recordReturnType(String.self) + + // [1.5] done recording arguments + try invocation.doneRecording() + + // [2] invoke the `remoteCall` method of the actor system + return try await self.actorSystem.remoteCall( + on: self, + target: RemoteCallTarget(...), + invocation: invocation, + throwing: Never.self, // the target func was not throwing + returning: String.self + ) + } +} +``` + +The synthesized thunk is always throwing and asynchronous. This is correct because it is only invoked in situations where we might end up calling the `actorSystem.remoteCall(...)` method, which by necessity is asynchronous and throwing. + +The thunk is `nonisolated` because it is a method that can actually run on a *remote* instance, and as such is not allowed to touch any other state than other nonisolated stored properties. This is specifically designed such that the thunk and actor system are able to access the `id` of the actor (and the `actorSystem` property itself) which is necessary to perform the actual remote message send. + +The `nonisolated` aspect of the method has another important role to play: if this invocation happens to be on a local distributed actor, we do not want to "hop" executors twice. If this invocation were on a local actor, only accessing `nonisolated` state, or for other reasons the hop could be optimized away, we want to keep this ability for the optimizer to do as good of a job as it would for local only actors. If the instance was remote, we don't need to suspend early at all, and we leave it to the `ActorSystem` to decide when exactly the task will suspend. For example, the system may only suspend the call after it has sent the bytes synchronously over some IPC channel, etc. The semantics of to suspend are highly dependent on the specific underlying transport, and thanks to this approach we allow system implementations to do the right thing, whatever that might be: they can suspend early, late, or even not at all if the call is known to be impossible to succeed. + +Note that the compiler will pass the `self` of the distributed *known-to-be-remote* actor to the `remoteCall` method on the actor system. This allows the system to check the passed type for any potential, future, customization points that the actor may declare as static properties, and/or conformances affecting how a message shall be serialized or delivered. It is impossible for the system to access any of that actor's state, because it is remote after all. The one piece of state it will need to access though is the actor's `id` because that is signifying the *recipient* of the call. + +The thunk creates the `invocation` container `[1]` into which it records all arguments. Note that all these APIs are using only concrete types, so we never pay for any existential wrapping or other indirections. Arguments are wrapped in a `RemoteCallArgument` which also provides additional information about the argument, such as the labels used in its declaration. This can be useful for transports which wish to encode values in named dictionaries, e.g. such as JSON dictionaries. It is important to recognize that storing names or labels is _not_ required or recommended for the majority of transports, however for those which need it, they are free to use this additional information. + +Since the strings created are string literals, known at compile time, there is no allocation impact for this argument wrapper type and this additional label information. + +> It may seem tempting to simply record arguments into a dictionary using their effective labels, however this can lead to issues due to labels being allowed to be reused. For example, `func sum(a a1: Int, a a2: Int)` is a legal, although not very readable, function declaration. A naive encoding scheme could risk overriding the "first `a`" value with the "second `a` if it strongly relied on the labels only. A distributed actor system implementation should decide how to deal with such situations and may choose to throw at runtime if such a risky signature is detected, or apply some mangling, e.g. use the parameter names to clarify which value was encoded. + +The `record...` calls are expected to serialize the values, using any mechanism they want to, and thanks to the fact that the type performing the recording is being provided by the specific `ActorSystem`, it also knows that it can rely on the arguments to conform to the system's `SerializationRequirement`. + +The first step in the thunk is to record any "generic substitutions" `[1.1]` if they are necessary. This makes it possible for remote calls to support generic arguments, and even generic distributed actors. The substitutions are not recorded for call where the generic context is not necessary for the invocation. For a generic method, however, the runtime will invoke the `recordGenericTypeSubstitution` with _concrete_ generic arguments that are necessary to perform the call. For example, if we declared a generic `echo` method like this: + +```swift +distributed func echo(_ value: T) -> T +``` + +and call it like this: + +```swift +try await greeter.echo("Echo!") // typechecks ok; String: SerializationRequirement +``` + +The Swift runtime would generate the following call: + +```swift +try invocation.recordGenericTypeSubstitution(String.self) +``` + +This method is implemented by a distributed actor system library, and can use this information to double-check this type against an allow-list of types allowed to be transmitted over the wire, and then store and send it over to the recipient such that it knows what type to decode the argument as. + +Next, the runtime will record all arguments of the invocation `[1.2]`. This is done in a series of `recordArgument` calls. If the type of actor the target is declared on also includes a generic parameter that is used by the invocation, this also is recorded. + +As the `recordArgument(_:)` method is generic over the argument type (``), and requires the argument to conform to `SerializationRequirement` (which in turn was enforced at compile time by [SE-0336][isolation]), the actor system implementation will have an easy time to serialize or validate this argument. For example, if the `SerializationRequirement` was codable — this is where one could invoke `SomeEncoder().encode(argument)` because `Argument` is a concrete type conforming to `Codable`! + +Finally, the specific error `[1.3]` and return types `[1.4]` are also recorded. If the function is not throwing, `recordErrorType` is not called. Likewise, if the return type is `Void` the `recordReturnType` is not called. + +Recording the error type is mostly future-proofing and currently will only ever be invoked with the `Error.self` or not at all. It allows informing the system if a throw from the remote side is to be expected, and technically, if Swift were to gain typed throws this method could record specific expected error types as well — although we have no plans with regards to typed throws at this point in time. + +The last encoder call `doneRecording()` is made, to signal to the invocation encoder that no further record calls will be made. This is useful since with the optional nature of some of the calls, it would be difficult to know for a system implementation when the invocation is fully constructed. Operations which may want to be delayed until completion could include serialization, de-duplicating values or similar operations which benefit from seeing the whole constructed invocation state in the encoder. + +Lastly, the populated encoder, along with additional type and function identifying information is passed to the `remoteCall`, or `remoteCallVoid` method on the actor system which should actually perform the message request/response interaction with the remote actor. + +#### Sender: Serializing and sending invocations + +The next step in making a remote call is serializing a representation of the distributed method (or computed property) invocation. This is done through a series of compiler, runtime, and distributed actor system interactions. These interactions are designed to be highly efficient and customizable. Thanks to the `DistributedTargetInvocationEncoder`, we are able to never resort to existential boxing of values, allow serializers to manage and directly write into their destination buffers (i.e. allowing for zero copies to be performed between the message serialization and the underlying networking layer), and more. + +Let us consider a `ClusterSystem` that will use `Codable` and send messages over the network. Most systems will need to form some kind of "Envelope" (easy to remember as: "the thing that contains the **message** and also has knowledge of the **recipient**"). For the purpose of this proposal, we'll define a a `WireEnvelope` and use it in the next snippets to showcase how a typical actor system would work with it. This type is not pre-defined or required by this proposal, but it is something implementations will frequently do on their own: + +```swift +// !! ClusterSystem or WireEnvelope are NOT part of the proposal, but serves as illustration how actor systems might !! +// !! implement the necessary pieces of the DistributedActorSystem protocol. !! + +final struct ClusterSystem: DistributedActorSystem { + // ... + typealias SerializationRequirement = Codable + typealias InvocationEncoder = ClusterTargetInvocationEncoder + typealias InvocationDecoder = ClusterTargetInvocationDecoder + + // Just an example, we can implement this more efficiently if we wanted to. + private struct WireEnvelope: Codable, Sendable { + var recipientID: ClusterSystem.ActorID // is Codable + + /// Mangled method/property identifier, e.g. in a mangled format + var identifier: String + + // Type substitutions matter only for distributed methods which use generics: + var genericSubstitutions: [String] + + // For illustration purposes and simplicity of code snippets we use `[Data]` here, + // but real implementations can be much more efficient here — packing all the data into exact + // byte buffer that will be passed to the networking layer, etc. + var arguments: [Data] // example is using Data, because that's what Codable coders use + + // Metadata can be used by swift-distributed-tracing, or other instrumentations to carry extra information: + var metadata: [String: [Data]] // additional metadata, such as trace-ids + } +} +``` + +Note that `method` property is enough to identify the target of the call, we do not need to carry any extra type information explicitly in the call. The method identifier is sufficient to resolve the target method on the recipient, however in order to support generic distributed methods, we need to carry additional (mangled) type information for any of the generic parameters of this specific method invocation. Thankfully, these are readily provided to us by the Swift runtime, so we'll only need to store and send them over. + +> **Note:** An implementation may choose to define any shape of "envelope" (or none at all) that suits its needs. It may choose to transport mangled names of involved types for validation purposes, or choose to not transfer them at all and impose other limitations on the system and its users for the sake of efficiency. +> +> While advanced implementations may apply compression and other techniques to minimize the overhead of these envelopes — this is a deep topic by itself, and we won't be going in depth on it in this proposal — rest assured though, we have focused on making different kinds of implementations possible with this approach. + +Next, we will discuss how the `InvocationEncoder` can be implemented in order to create such `WireEnvelope`. + +> Note on ad-hoc requirements: Some of the protocol requirements on the encoder, as well as actor system protocols, are so-called "ad-hoc" requirements. This means that they are not directly expressed in Swift source, but instead the compiler is aware of the signatures and specifically enforces that a type conforming to such protocol implements these special methods. +> +> Specifically, methods which fall into this category are functions which use the `SerializationRequirement` as generic type requirement. This is currently not expressible in plain Swift, due to limitations in the type system which are difficult to resolve immediately, but in time as this could become implementable these requirements could become normal protocol requirements. +> +> This tradeoff was discussed at length and we believe it is worth taking, because it allows us to avoid numerous un-necessary type-casts, both inside the runtime and actor system implementations. It also allows us to avoid any existential boxing and thus lessens the allocation footprint of making remote calls which is an important aspect of the design and use cases we are targeting. + +The following listing illustrates how one _could_ implement a `DistributedTargetInvocationEncoder`: + +```swift +extension ClusterSystem { + typealias InvocationEncoder = ClusterTargetInvocationEncoder + + func makeInvocationEncoder() -> Self.InvocationEncoder { + return ClusterTargetInvocation(system: system) + } +} + +struct ClusterTargetInvocationEncoder: DistributedTargetInvocationEncoder { + typealias SerializationRequirement = ClusterSystem.SerializationRequirement + + let system: ClusterSystem + var envelope: Envelope + + init(system: ClusterSystem) { + self.system = system + self.envelope = .init() // new "empty" envelope + } + + /// The arguments must be encoded order-preserving, and once `decodeGenericSubstitutions` + /// is called, the substitutions must be returned in the same order in which they were recorded. + mutating func recordGenericSubstitution(_ type: T.Type) throws { + // NOTE: we are showcasing a pretty simple implementation here... + // advanced systems could use mangled type names or registered type IDs. + envelope.genericSubstitutions.append(String(reflecting: T.self)) + } + + mutating func recordArgument(_ argument: RemoteCallArgument) throws { + // in this implementation, we just encode the values one-by-one as we receive them: + let argData = try system.encoder.encode(argument) // using whichever Encoder the system has configured + envelope.arguments.append(argData) + } + + mutating func recordErrorType(_ errorType: E.Type) throws { + envelope.errorType = String(reflecting: errorType) + } + + mutating func recordReturnType(_ returnType: R.Type) throws { + envelope.returnType = String(reflecting: returnType) + } + + /// Invoked when all the `record...` calls have been completed and the `DistributedTargetInvocation` + /// will be passed off to the `remoteCall` to perform the remote call using this invocation representation. + mutating func doneRecording() throws { + // our impl does not need to do anything here + } +} +``` + +The above encoder is going to be called by the Swift runtime as was explained in the previous section. + +Once that is complete, the runtime will pass the constructed `InvocationEncoder` to the `remoteCall`: + +```swift + extension ClusterSystem { + // 'remoteCall' is not a protocol requirement, however its signature is well known to the compiler, + // and it will invoke the method. We also are guaranteed that the 'Success: Codable' requirement is correct, + // since the type-system will enforce this conformance thanks to the type-level checks on distributed funcs. + func remoteCall( + on actor: Actor, + target: RemoteCallTarget, + invocation: Self.InvocationEncoder, + throwing: Failure.Type, + returning: Success.Type + ) async throws -> Success + where Actor: DistributedActor, + Actor.ID == ActorID. + Failure: Error, + Success: Self.SerializationRequirement { + var envelope = invocation.envelope + + // [1] the recipient is transferred over the wire as its id + envelope.recipient = recipient.id + + // [2] the method is a mangled identifier of the 'distributed func' (or var). + // In this system, we just use the mangled name, but we could do much better in the future. + envelope.target = target.identifier + + // [3] send the envelope over the wire and await the reply: + let responseData = try await self.underlyingTransport.send(envelope, to: actor.id) + + // [4] decode the response from the response bytes + // in our example system, we're using Codable as SerializationRequirement, + // so we can decode the response like this (and never need to cast `as? Codable` etc.): + return try self.someDecoder.decode(as: Success.self, from: responseData) + } +} +``` + +The overall purpose of this `remoteCall` implementation is to create some form of message representation of the invocation and send it off to the remote node (or process) to receive and invoke the target method on the remote actor. + +In our example implementation, the `Invocation` already serialized the arguments and stored them in the `Envelope`, so the `remoteCall` only needs to add the information about the call recipient `[1]`, and the target (method or computed property) of the call `[2]`. In our example implementation, we just store the target's mangled name `[2]`, which is simple, but it has its challenges in regard to protocol evolution. + +One notable issue that mangled names have is that any change in the method signature will result in not being able to resolve the target method anymore. We are very much aware of the issues this may cause to protocol evolution, and we lay out plans in [Future Work](#future-work) to improve the lookup mechanisms in ways that will even allow adding parameters (with default values), in wire (and ABI) compatible ways. + +The final step is handing over the envelope containing the encoded arguments, recipient information, etc., to the underlying transport mechanism `[3]`. The transport does not really have to concern itself with any of the specifics of the call, other than transmitting the bytes to the callee and the response data back. As we get the response data back, we have the concrete type of the expected response and can attempt to decode it `[4]`. + +> Note on `remoteCallVoid`: One limitation in the current implementation approach is that a remote call signature cannot handle void returning methods, because of the `Success: SerializationRequirement` requirement on the method. +> +> This will be possible to solve using the incoming [Variadic Generics](https://forums.swift.org/t/variadic-generics/54511) language feature that is being currently worked on and pitched. With this feature, the return type could be represented as variadic generic and the `Void` return type would be modeled as "empty" tuple, whereas a value return would contain the specific type of the return, this way we would not violate the `Success: SerializationRequirement` when we needed to model `Void` calls. + +#### Recipient: Receiving invocations + +On the remote side, there usually will be some receive loop or similar mechanism that is implemented in the transport layer of the actor system. In practice this often means binding a port and receiving TCP (or UDP) packets, applying some form of framing and eventually decoding the incoming message envelope. + +Since the communication of the sending and receiving side is going to be implemented by the same type of transport and actor system, receiving the envelopes is straightforward: we know the wire protocol, and follow it to receive enough bytes to decode the `Envelope` which we sent a few sections above. + +This part does not have anything specific prescribed in the `DistributedActorSystem` protocol. It is up to every system to implement whichever transport mechanism works for it. While not a "real" snippet, this can be thought of a simple loop over incoming connections, like this: + +```swift +// simplified pseudo code for illustration purposes +func receiveLoop(with node: Node) async throws { + for try await envelopeData in connection(node).receiveEnvelope { + await self.receive(envelopeData) + } +} +``` + +In a real server implementation we'd likely use a [Swift NIO](https://github.com/apple/swift-nio) `ChannelPipeline` to perform this networking, framing and emitting of `Envelope`s, but this is beyond the scope of what we need to explain in this proposal to get the general idea of how this is going to work. + +#### Recipient: Deserializing incoming invocations + +Now that we have received all the bytes for one specific envelope, we need to perform a two-step deserialization on it. + +First, we'll need to decode the target identifier (e.g. method name, mangled method name, or some other form of target identifier), and the actor `ID` of the recipient. These are necessary to decode always, as we need to locate both the method and actor we're trying to invoke. + +Next, the deserialization of the actual message representation of our invocation will take place. However, this is done lazily. Rather than just decoding the values and storing them somewhere in our system implementation, these will be requested by the Swift runtime when it is about to perform the method call. + +Before we dive deeper into this, let us visualize how this two-step process is intended to work, by looking at what might be a typical envelope format on the wire: + +```c ++------------------------------- ENVELOPE --------------------------------------+ +| +---------- HEADER --------++-------------------- MESSAGE ------------------+ | +| | target | recipient | ... || [ ... lazy decoded section: types, args ... ] | | +| +--------------------------++-----------------------------------------------+ | ++-------------------------------------------------------------------------------+ +``` + +We see that as we decode our wire envelope, we are able to get the header section, and all values contained within it eagerly and leave the remaining slice of the buffer untouched. It will be consumed during performing of the invocation soon enough. The nice thing about this design is that we're still able to hold onto the actual buffer handed us from the networking library, and we never had to copy the buffer to our own local copies. + +Next, we need to prepare for the decoding of the message section. This is done by implementing the remaining protocol requirements on the `ClusterTargetInvocation` type we defined earlier, as well as implementing a decoding iterator of type `DistributedTargetInvocationArgumentDecoder`, as shown below: + +```swift +struct ClusterTargetInvocationDecoder: DistributedTargetInvocationDecoder { + typealias SerializationRequirement = Codable + + let system: ClusterSystem + var bytes: ByteBuffer + + mutating func decodeGenericSubstitutions() throws -> [Any.Type] { + let subCount = try self.bytes.readInt() + + var subTypes: [Any.Type] = [] + for _ in 0..() throws -> Argument { + try nextDataLength = try bytes.readInt() + let nextData = try bytes.readData(bytes: nextDataLength) + // since we are guaranteed the values are Codable, so we can just invoke it: + return try system.decoder.decode(as: Argument.self, from: bytes) + } + + mutating func decodeErrorType() throws -> Any.Type? { + let length = try self.bytes.readInt() // read the length of the type + guard length > 0 { + return nil // we don't always transmit it, 0 length means "none" + } + let typeName = try self.bytes.readString(length: length) + return try self.system.summonType(byName: typeName) + } + + mutating func decodeReturnType() throws -> Any.Type? { + let length = try self.bytes.readInt() // read the length of the type + guard length > 0 { + return nil // we don't always transmit it, 0 length means "none" + } + let typeName = try self.bytes.readString(length: length) + return try self.system.summonType(byName: typeName) + } +} +``` + +The general idea here is that the `InvocationDecoder` is *lazy* in its decoding and just stores the remaining bytes of the envelope. All we need to do for now is to implement the Invocation in such way that it expects the decoding methods be invoked in the following order (which is the same as the order on the sending side): + +- 0...1 invocation of `decodeGenericArguments`, +- 0...n invocation(s) of `decoder.decodeNextArgument`, +- 0...1 invocation of `decodeReturnType`, +- 0...1 invocation of `decodeErrorType`. + +Decoding arguments is the most interesting here. This is another case where the compiler and Swift runtime enable us to implement things more easily. Since the `Argument` generic type of the `decodeNextArgument` is ensured to conform to the `SerializationRequirement`, actor system implementations can rely on this fact and have a simpler time implementing the decoding steps. For example, with `Codable` the decoding steps becomes a rather simple task of invoking the usual `Decoder` APIs. + +This decoder must be prepared by the actor system and eventually passed to the `executeDistributedTarget` method which we'll discuss next. That, Swift runtime provided, function is the one which will be calling the `decode...` methods and will is able to ensure all the type requirements are actually met and form the correct generic method invocations. + +> **Note:** This proposal does not include an implementation for the mentioned `summonType(byName:)` function, it is just one way systems may choose to implement these functions. Possible implementations include: registering all "trusted" types, using mangled names, or something else entirely. This proposal has no opinion about how these types are recovered from the transmitted values. + +#### Recipient: Resolving the recipient actor instance + +Now that we have prepared our `InvocationDecoder` we are ready to make the next step, and resolve the recipient actor which the invocation shall be made on. + +We already discussed how resolving actors works in [Resolving distributed actors](#resolving-distributed-actors), however in this section we can tie it into the real process of invoking the target function as well. + +In the example we're following so far, the recipient resolution is simple because we have the recipient ID available in the `Envelope.recipientID`, so we only need to resolve that using the system that is receiving the message: + +```swift +guard let actor: any DistributedActor = try self.knownActors[envelope.recipientID] else { + throw ClusterSystemError.unknownRecipient(envelope.recipientID) +} +``` + +This logic is the same as the internal implementation of the `resolve(id:as:)` method only that we don't have a need to validate the specific type of the actor — this will be handled by the Swift runtime in `executeDistributedTarget`'s implementation the target of the call which we'll explain in the next section. + +#### Recipient: The `executeDistributedTarget` method + +Invoking a distributed method is a tricky task, and involves a lot of type demangling, opening existential types, forming specific generic invocations and tightly managing all of that in order to avoid un-necessary heap allocations to pass the decoded arguments to the target function, etc. After iterating over multiple designs, we decided to expose a single `DistributedActorSystem.executeDistributedTarget` entry point which efficiently performs all the above operations. + +Thanks to abstracting the decoding logic into the `DistributedTargetInvocationDecoder` type, all deserialization can be made directly from the buffers that were received from the underlying network transport. The `executeDistributedTarget` method has no opinion about what serialization mechanism is used either, and any mechanism — be it `Codable` or other external serialization systems — can be used, allowing distributed actor systems developers to implement whichever coding strategy they choose, potentially directly from the buffers obtained from the transport layer. + +The `executeDistributedTarget` method is defined as: + +```swift +extension DistributedActorSystem { + /// Prepare and execute a call to the distributed function identified by the passed arguments, + /// on the passed `actor`, and collect its results using the `ResultHandler`. + /// + /// This method encapsulates multiple steps that are invoked in executing a distributed function, + /// into one very efficient implementation. The steps involved are: + /// + /// - looking up the distributed function based on its name; + /// - decoding, in an efficient manner, all arguments from the `Args` container into a well-typed representation; + /// - using that representation to perform the call on the target method. + /// + /// The reason for this API using a `ResultHandler` rather than returning values directly, + /// is that thanks to this approach it can avoid any existential boxing, and can serve the most + /// latency sensitive-use-cases. + func executeDistributedTarget( + on actor: Actor, + mangledName: String, + invocation: inout Self.InvocationDecoder, + handler: ResultHandler + ) async throws where Actor: DistributedActor, + Actor.ID == ActorID, + ResultHandler: DistributedTargetInvocationResultHandler { + // implemented by the _Distributed library + } +} +``` + +This method encapsulates all the difficult and hard to implement pieces of the target invocation, and it accepts the base actor the call should be performed on, along with a `DistributedTargetInvocationResultHandler`. + +Rather than having the `executeDistributedTarget` method return an `Any` result, we use the result handler in order to efficiently, and type-safely provide the result value to the actor system library implementation. This technique is the same as we did with the `recordArgument` method before, and it allows us to provide the _specific_ type including its `SerializationRequirement` conformance making handling results much simpler, and without having to resort to any casts which can be unsafe if used wrongly, or have impact on runtime performance. + +The `DistributedTargetInvocationResultHandler` is defined as follows: + +```swift +protocol DistributedTargetInvocationResultHandler { + associatedtype SerializationRequirement + + func onReturn(value: Success) async throws + where Success: SerializationRequirement + func onThrow(error: Error) async throws + where Failure: Error +} +``` + +In a way, the `onReturn`/`onThrow` methods can be thought of as the counterparts of the `recordArgument` calls on the sender side. We need to encode the result and send it _back_ to the sender after all. This is why providing the result value along with the appropriate SerializationRequirement conforming type is so important — it makes sending back the reply to a call, as simple as encoding the argument of the call. + +Errors must be handled by informing the sender about the failed call. This is in order to avoid senders waiting and waiting for a reply, and eventually triggering a timeout; rather, they should be informed as soon as possible that a call has failed. Treat an error the same way as you would a valid return in terms of sending the reply back. However, it is not required to actually send back the actual error, as it may not be safe, or a good idea from a security and information exposure perspective, to send back entire errors. Instead, systems are encouraged to send back a reasonable amount of information about a failure, and e.g. optionally, only if the thrown error type is `Codable` and allow-listed to be sent over the wire, transport it directly. + +#### Recipient: Executing the distributed target + +Now that we have completed all the above steps, all building up to actually invoking the target of a remote call: it is finally time to do so, by calling the `executeDistributedTarget` method: + +```swift +// inside recipient actor system +let envelope: IncomingEnvelope = // receive & decode ... +let recipient: DistributedActor = // resolve ... + +let invocationDecoder = InvocationDecoder(system: self, bytes: envelope.bytes) + +try await executeDistributedTarget( + on: recipient, // target instance for the call + mangledName: envelope.targetName, // target func/var for the call + invocation: invocationDecoder // will be used to perform decoding arguments, + handler: ClusterTargetInvocationResultHandler(system, envelope) // handles replying to the caller (omitted in proposal) +) +``` + +This call triggers all the decoding that we discussed earlier, and if any of the decoding, or distributed func/var resolution fails this call will throw. Otherwise, once all decoding has successfully been completed, the arguments are passed through the buffer to a distributed method accessor that actually performs the local method invocation. Once the method returns, its results are moved into the handler where the actor system takes over in order to send a reply to the remote caller — completing the remote call! + +Internally, the execute distributed thunk heavily relies on the lookup and code generated by the compiler for every `distributed func` which we refer to as **distributed method accessor thunk**. This thunk is able to decode incoming arguments using the `InvocationDecoder` and directly apply the target function, all while properly handling generics and other important aspects of function invocations. It is the distributed method accessor thunk that must be located using the "target identifier" when we handle an incoming the remote call, the thunk then calls the actual target function. + +For sake of completeness, the listing below shows the distributed method accessor thunk that is synthesized by the compiler. The thunk contains compiler synthesized logic specific to every distributed function to locate the target function, obtain the expected parameter types and use the passed in decoder to decode the arguments to finally pass them to the final function application. + +The thunk can be thought of in terms of this abstract example. However, it cannot be implemented like this because of various interactions with the generic system as well as how emissions (function calls) actually work. Distributed method accessor thunks are implemented directly in IR as it would not be possible to synthesize the necessary emissions in any higher level part of the compiler (!). Thankfully, the logic contained in those accessors is fairly straightforward and can be imagined as: + +```swift +distributed actor DA { + func myCompute(_ i: Int, _ s: String, _ d: Double) async throws -> String { + "i:\(i), s:\(s), d:\(d)" + } +} + +extension DA { + // Distributed accessor thunk" for 'myCompute(_:_:_:) -> String' + // + // PSEUDO CODE FOR ILLUSTRATION PURPOSES; NOT IMPLEMENTABLE IN PLAIN SWIFT; + // Implemented in directly in IR for expressibility reasons, and not user-accessible. + nonisolated func $distributedFuncAccessor_myCompute( + decoder: UnsafeMutableRawPointer, + argumentTypes: UnsafeRawPointer, + resultBuffer: UnsafeRawPointer, + genericSubstitutions: UnsafeRawPointer, + witnessTables: UnsafeRawPointer, + numWitnessTables: Int, + actorSelf: UnsafeRawPointer) async { + + // - get generic signature of 'myCompute' + // - create storage 'args' for all the parameters; it will be used directly + // - for every argument, get the argumentType + // - invoke 'decoder.decodeArgument()' + // - store in 'args' + // - deal with the generic substitutions, witness tables and prepare the call + // invoke 'myCompute' with 'args', and the prepared 'result' and 'error' buffers + } +} +``` + +As we can see, this thunk is "just" taking care of converting the heterogeneous parameters into the well typed counterparts, and finally performing a plain-old method invocation using those parameters. The actual code emission and handling of generics for all this to work is rather complex and can only be implemented in the IR layer of the compiler. The good part about it is that the compiler is able to prepare and emit good errors in case the types or witness tables seem to be mismatched with the target or other issues are found. Allocations are also kept to a minimum, as no intermediate allocations need to be made for the arguments and they are stored and directly emitted into the call emission of the target. + +The thunk again uses the indirect return, so we can avoid any kind of implicit existential boxing even on those layers. Errors are always returned indirectly, so we do not need to do it explicitly. + +#### Recipient: Collecting result/error from invocations + +Now that the distributed method has been invoked, it eventually returns or throws an error. + +Collecting the return (or error) value is also implemented using the `DistributedMethodInvocationHandler` we passed to the `executeDistributedTarget(...)` method. This is done for the same reason as parameters: we need a concrete type in order to efficiently pass the values to the actor system, so it can encode them without going through existential wrappers. As we cannot implement the `invoke()` method to be codable over the expected types — we don't know them until we've looked up the actual method we were about to invoke (and apply generic substitutions to them). + +The implementation could look as follows: + +```swift +extension ExampleDistributedMethodInvocationHandler { + func onReturn(result: Success) throws { + do { + let replyData = system.encoder.encode(result) + self.reply(replyData) + } catch { + self.replyError("Failed to encode reply: \(type(of: error))") + } + } + + func onError(error: Failure) { + guard Failure is Encodable else { + // best effort error reporting just sends back the type string + // we don't want to send back string representation since it could leak sensitive information + self.replyError("\(Failure.self)") + } + + // ... if possible, `as?` cast to Encodable and return an actual error, + // but only if it is allow-listed, as we don't want to send arbitrary errors back. + } +} +``` + +We omit the implementations of `replyError` and `reply` because they are more of the same patterns that we have already discussed here, and this is a proposal focused on illustrating the language feature, not a complete system implementation after all. + +The general pattern here is the same as with decoding parameters, however in the opposite direction. + +Once the `onError` or `onReturn` methods complete, the `executeDistributedTarget` method returns, and its caller knows the distributed request/response has completed – at least, as far as this peer is concerned. We omit the implementation of the `reply` and `replyError` methods that the actor system would implement here, because they are pretty much the same process as sending the request, except that the message must be sent as a response to a specific request, rather than target a specific actor and method. How this is achieved can differ wildly between transport implementations: some have built-in request/reply mechanisms, while others are uni-directional and rely on tagging replies with identifiers such as "this is a reply for request 123456". + +## Amendments + +### Initializers no longer need to accept a single DistributedActorSystem + +During this review, feedback was received and addressed with regards to the previous implementation limitation that user defined designated actor initializers must have always accepted a single `DistributedActorSystem` conforming parameter. This was defined in [SE-0336: Distributed Actor Isolation](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0336-distributed-actor-isolation.md). + +The limitation used to be that one had to accept such parameter or the system would fail to synthesize the initializer an offer a compile-time error: + +```swift +// PREVIOUSLY, as defined by SE-0336 +init(x: String) {} +// ❌ error: designated distributed actor initializer 'init(x:)' is missing required 'DistributedActorSystem' parameter + +init(system: AnyDistributedActorSystem) {} +// ✅ used to be ok; though the "unused" parameter looks confusing and is actually necessary +``` + +This system argument, picked by type, would have been picked up by the compiler and used in synthesis of necessary calls during a distributed actor's initialization. This led to a weird "dangling" parameter that is not used by visible code, but is crucial for the correctness of the program. + +This implementation restriction is now lifted, and is rephrased as requiring an assignment to the `self.actorSystem` property in non-delegating initializers. This is in line with how normal Swift works, and can be explained to users more intuitively, because every `distributed actor` has a synthesized `let actorSystem: ActorSystem` property, it clearly must be initialized to something as implied by normal initializer rules. + +The checks therefore are now plain property assignment checks, and the compiler is able to pick up the assigned property and synthesize the necessary `assignID` and `actorReady` calls based on the property, e.g.: + +```swift +distributed actor Example { + init(x: String) { + // ❌ error: property 'actorSystem' was not initialized + // more educational notes to be produced here (e.g. what type it expects etc) + } + + init (x: String, cluster: ClusterSystem) { + self.actorSystem = cluster // ✅ ok + } +} +``` + +This means that it is also possible to assign a "global" or static actor system, however it is **strongly discouraged** to do so because it hurts reusability and testability, e.g. by swapping a test instance of an actor system during unit tests. + +```swift +// Possible, but considered an anti-pattern as it hurts testability of such actor +// DON'T DO THIS. +distributed actor NotGoodIdea { + init() { + self.actorSystem = SomeSystem.global // anti-pattern, always accept an actor system via initializer. + } +} +``` + +## Future work + +### Variadic generics removing the need for `remoteCallVoid` + +Once [variadic generics](https://forums.swift.org/t/variadic-generics/54511/2) are fully implemented, we will be able to remove the limitation that we cannot express the `remoteCall<..., Success: SerializationRequirement>(..., returning returnType: Success.Type)` function for the `Void` type, since it cannot always conform to `SerializationRequirement`. + +With variadic generics, it would be natural to conform an "empty tuple" to the `SerializationRequirement` and we'd this way be able to implement only a single method (`remoteCall`) rather than having to provide an additional special case implementation for `Void` return types. + +### Identifying, evolving and versioning remote calls + +At present remote calls use an opaque string identifier (wrapped as `RemoteCallTarget`) to refer to, and invoke remote call targets. This identifier is populated by default using the mangled name of the distributed thunk of the target function. + +#### Default distributed call target identification scheme + +Distributed actor system implementations shall treat the identifier as opaque value that they shall ship from the sender to the recipient node, without worrying too much about their contents. The identification scheme though has a large and important impact on the versioning story of distributed calls, so in this section we'd like to discuss and outline the general plans we foresee here as we will introduce proper and more versioning friendly schemes in the future. + +Swift's default mangling scheme is used for distributed methods is problematic for API evolution, however, it is a great "default" to pick until we devise a different scheme in the future. The mangling scheme is a good default because: + +- it allows callers to use all of the richness of Swift's calling semantics, including: + - overloads by type (e.g. we can call `receive(_:Int)` as well as `receive(_:String)`) and invoke the correct target based on the type. +- it allows us to evolve APIs manually by adding overloads, deprecate methods etc, and we will continue to work towards a feature rich versioning story while adopting limited use-cases where these limitations are not a problem. + +This is also avoids a well-known problem from objective-c, where selectors must not accidentally be the same, otherwise bad-things-happen™. + +One potential alternative would be to use the full names of methods, i.e. `hello(name:surname:)`, similar to objective-c selectors to identify methods. However, this means the loss of any and all type-safety and _intent_ of target methods being used according to their appropriate types. It also means identifiers must be unique and overloads must be banned at compile time, or we risk reusing identifiers and not knowing which function to invoke. This again could be solved by separately shipping type identifiers, but this would mean reinventing what Swift's mangling scheme already does, but in a worse way. Instead, we propose to start with the simple mangling scheme, and in a subsequent proposal, address the versioning story more holistically, in a way that addresses ABI concerns of normal libraries, as well as distributed calls. + +We are aware that the mangling scheme makes the following, what should be possible to evolve without breaking APIs situations not work, that a different scheme could handle well: + +- adding parameters even with default parameters, +- changing a type of argument from struct to class (or enum etc), is also breaking though it need not be. + - We could address this by omitting the kind information from the mangled names. + +The robust versioning and evolution scheme we have in mind for the future must be able to handle these cases, and we will be able to roll out a new identification scheme that handles those in the future, without breaking API or breaking existing remote calls. The metadata to lookup distributed method accessors would still be available using the "most precise" mangled format, even as we introduce a lossy, more versioning friendly format. APIs would remain unchanged, because from the perspective of `DistributedActorSystem` all it does is ship around an opaque String `identifier` of a remote call target. + +There are numerous other versioning and rollout topics to cover, such as "rolling deployments", "green/blue deployments" and other techniques that both the versioning scheme _and_ the specific actor system implementation must be able to handle eventually. However, based on experience implementing other actor systems in the past, we are confident that those are typical to add in later phases of such an endeavor, not as the first thing in the first iteration of the project. + +#### Compression techniques to avoid repeatedly sending large identifiers + +One of the worries brought up with regards to mangled names and string identifiers in general during the review has been that they can impose a large overhead when the actual messages are small. Specifically, as the `RemoteCallTarget.identifier` often can be quite large, and for distributed methods which e.g. accept only a few integers, or even no values at all, the identifiers often can dominate the entire message payload. + +This is not a novel problem – we have seen and solved such issues in the past in other actor runtimes (i.e. Akka). One potential solution is to establish a compression mechanism between peers where message exchanges establish shared knowledge about "long identifiers" and mapping them to unique numbers. The sender in such a system at first pessimistically sends the "long" String-based identifier, and in return may get additional metadata in the response "the next time you want to call this target, send the ID 12345". The sender system then stores in a cache that when invoking the "X...Z" target it does not need to send the long string identifier, but instead can send the number 12345. This solves the long string identifiers problem, as they need not be sent repeatedly over the wire. (There are a lot of details to how such schemes can be implemented that we do not need to dive into here, however we are confident those work well, because we have seen them solve this exact issue before). + +Given the need, and assuming other shared knowledge, we could even implement other identification schemes, which can also avoid that "initial" long string identifier sending. We will be exploring those as specific use-cases and requirements from adopters as they arise. We are confident in our ability to fit such techniques into this design because of the flexibility that APIs based around `RemoteCallTarget` gives us, without requiring us to actually send the entire target object over the wire. + +Such optimization techniques, are entirely implementable in concrete distributed actor system implementations. However, if necessary, we would be able to even extend the capabilities of `executeDistributedTarget` to accommodate other concrete needs and designs. + +#### Overlap with general ABI and versioning needs in normal Swift code + +The general concept binary/wire-compatible evolution of APIs by _adding_ new parameters to methods is not unique to distribution. In fact, this is a feature that would benefit ABI-stable libraries like the ones shipping with the SDK. It is often the case that new APIs are introduced that accept _more_ arguments, perhaps to enable or extend functionality of an existing feature. Developers today have to add new functions with "one more argument", like this: + +```swift +public func f() { + f(x: 0) +} + +@available(macOS 9999) +public func f(x: Int) { + // new implementation +} +``` + +Where the new function is going to ship with a new OS, yet actually contains an implementation that is compatible with the old definition of `f()`. In such situations, developers are forced to manually write forwarding methods. While this seems simple on small APIs, it can easily become rather complex and easy to introduce subtle bugs in the forwarding logic. + +Instead, developers would want to be able to introduce new parameters, with a default value that would be used when the function is invoked on older platforms, like this: + +```swift +public func f(@available(macOS 9999) x: Int = 0) { + // new implementation +} + +// compiler synthesizes: +// +// public func f() { self.f(x: 0) } +// +// @available(macOS 9999) +// public func f( x: Int = 0) { +``` + +Where the compiler would synthesize versions of the methods for the various availabilities, and delegate, in an ABI-compatible way to the new implementation. This is very similar to what would be necessary to help wire-compatible evolution of distributed methods. So the same mechanisms could be used for distributed calls, if instead of OS versions, we could also allow application/node versions in the availability perhaps? This is not completely designed, but definitely a direction we are interested in exploring. + +#### Discussion: User provided target identities + +We also are considering, though are not yet convinced that these would be the optimal way to address some of the evolution concerns, the possibility to use "stable names". This mechanism would allow users to give full control over target identity to developers, where the `RemoteCallTarget.identifier` offered to the `remoteCall` implemented by a distributed actor system library, could be controlled by end users of the library, i.e. those who define distributed methods. + +On one hand, this gives great power to end-users, as they may use specific and minimal identifiers, even using `"A"` and other short names to minimize the impact of the identifiers to the payload size. On the other hand though, it opens up developers to a lot of risks, including conflicts in identifier use – a problem all to well known to objective-c developers where selectors could end up in similar problematic situations. + +We want to take time to design a proper system rather than just open up full control over the remote call target `identifier` to developers. We will aim to provide a flexible, and powerful mechanism, that does not risk giving developers easy ways to get shoot themselves in the foot. The design proposed here, is flexible enough to evolve. + +### Resolving `DistributedActor` protocols + +We want to be able to publish only protocols that contain distributed methods, and allow clients to resolve remote actors based on protocols alone, without having any knowledge about the specific `distributed actor` type implementing the protocol. This allows binary, closed-source frameworks to offer distributed actors as way of communication. Of course, for this to be viable we also need to solve the above ABI and wire-compatible evolution of distributed methods, assuming we solve those though, publishing distributed actor protocols is very useful and interesting for client/server scenarios, where the peers of a communication are not exact mirrors of the same process, but exhibit some asymmetry. + +Currently, we resolve distributed actors using the static method defined on the `DistributedActor` protocol, sadly this method is not possible to invoke on just a protocol: + +```swift +protocol Greeter: DistributedActor { + func greet() -> String +} + +let greeter: any Greeter = try Greeter.resolve(id: ..., using: websocketActorSystem) +// ❌ error: static member 'resolve' cannot be used on protocol metatype 'Greeter.Protocol' +``` + +A "client" peer does not have to know what distributed actor exactly implements this protocol, just that we're able to send a "greet" message to it, we should be able to obtain an existential `any Greeter` and be able to invoke `greet()` on it. + +In order to facilitate this capability, we need to: + +- implement ad-hoc synthesis of a type that effectively works like a "stub" that other RPC systems generally source-generate, yet thanks to our actor model we're able to synthesize it in the compiler on demand; +- find a way to invoke `resolve` on such protocol, for example we could offer a global function `resolveDistributedActorProtocol(Greeter.self, using: websocketActorSystem)` + +The `resolveDistributedActorProtocol` method has to be able to check the serialization requirement at compile-time where we invoke the resolve, because the distributed actor protocols don't have to declare a serialization requirement — they can, but they don't have to (and this is by design). + +It should be possible to resolve the following examples: + +```swift +protocol Greeter: DistributedActor { + distribute func greet() -> String +} + +final class WebsocketActorSystem: DistributedActorSystem { + typealias ActorID: WebsocketID // : Codable + typealias SerializationRequirement: Codable +} + +... = try resolveDistributedActorProtocol(id: ..., as: Greeter.self, using: websocketActorSystem) +``` + +The resolve call would use the types defined for the `ActorID` and `SerializationRequirement` to see if this `Greeter` protocol is even implementable using these, i.e. if its distributed method parameters/return types do indeed conform to `Codable`, and if there isn't a conflict with regards to the actor ID. + +This means that we should reject at compile-time any attempts to resolve a protocol that clearly cannot be implemented over the actor system in question, for example: + +```swift +protocol Greeter: DistributedActor { + distributed func greet() -> NotCodableResponse +} + +final class WebsocketActorSystem: DistributedActorSystem { + typealias ActorID: WebsocketID // : Codable + typealias SerializationRequirement: Codable +} + +... = try resolveDistributedActorProtocol(id: ..., as: Greeter.self, using: websocketActorSystem) +// ❌ error: 'Greeter' cannot be resolved using 'WebsocketActorSystem' +// ❌ error: result type 'NotCodableResponse' of distributed instance method does not conform to 'Codable' +``` + +These are the same checks that are performed on `distributed actor` declarations, but they are performed on the type. We can think of these checks running whenever distributed methods are "combined" with a specific actor system: this is the case in `distributed actor` declarations, as well as this protocol resolution time, because we're effectively creating a not-user-visible actor declaration that combines the given `DistributedActorSystem` with the synthesized "stub" distributed actor, so we need to run the checks here. Thankfully, we can run them at compile time, disallowing any ill-formed and impossible to implement combinations. + +### Passing parameters to `assignID` + +Sometimes, transports may need to get a little of configuration for a specific actor being initialized. + +Since `DistributedActorSystem.assignID` accepts the actor *type* it can easily access any configuration that is static for some specific actor type, e.g. like this: + +```swift +protocol ConfiguredDistributedActor: DistributedActor { + static var globalServiceName: String { get } +} + +distributed actor Cook: DistributedActorConfiguration { + static var globalServiceName: String { "com.apple.example.CookingService" } +} +``` + +This way the `assignID` can detect the static property and e.g. ensure this actor is possible to look up by this static name: + +```swift +extension SpecificDistributedActorSystem { + func assignID(_ type: Actor.Type) -> Actor.ID where Actor: DistributedActor { + let id = <> + if let C = type as ConfiguredDistributedActor.Type { + // for example, we could make sure the actor is discoverable using the service name: + let globalServiceName = C.globalServiceName + self.ensureAccessibleAs(id: id, as: globalServiceName) + } + + return id + } +} +``` + +Or similar configuration patterns. However, it is hard to implement a per instance configuration to be passed to the system. + +One way we could solve this is by introducing an `assignID` overload that accepts the `ActorConfiguration` that may be +passed to the actor initializer, and would be passed along to the actor system like this: + +```swift +extension SpecificDistributedActorSystem { + // associatedtype ActorConfiguration + func assignID(_ type: Actor.Type, _ properties: Self.ActorConfiguration) -> Actor.ID where Actor: DistributedActor { + if let name = properties.name { + return makeID(withName: name) + } + + return makeRandomID() + } +} +``` + +The creation of the actor would then be able to be passed at-most one `ActorConfiguration` instance, and that would be then passed down to this method: + +```swift +distributed actor Worker {...} + +Worker(actorSystem: system, actorProperties: .name("worker-1234")) +``` + +Which can be *very* helpful since now IDs can have user provided information that are meaningful in the user's domain. + +## Alternatives considered + +This section summarizes various points in the design space for this proposal that have been considered, but ultimately rejected from this proposal. + +### Define `remoteCall` as protocol requirement, and accept `[Any]` arguments + +The proposal includes the fairly special `remoteCall` method that is expected to be present on a distributed actor system, however is not part of the protocol requirements because it cannot be nicely expressed in today's Swift, and it suffers from the lack of variadic generics (which are being worked on, see: [Pitching The Start of Variadic Generics](https://forums.swift.org/t/pitching-the-start-of-variadic-generics/51467)), however until they are complete, expressing `remoteCall` in the type-system is fairly painful, and we resort to providing multiple overloads of the method: + +```swift + func remoteCall( + on recipient: Actor, + method: DistributedMethodName, + _ arg1: P1, + throwing errorType: Failure.Type, + returning returnType: Success.Type + ) async throws -> Success where Actor: DistributedActor, Actor.ID = ActorID { ... } + + func remoteCall( + on recipient: Actor, + method: DistributedMethodName, + _ arg1: P1, _ arg2: P2, + throwing errorType: Failure.Type, + returning returnType: Success.Type + ) async throws -> Success where Actor: DistributedActor, Actor.ID = ActorID { ... } + + // ... +``` + +This is annoying for the few distributed actor system developers. However, it allows us to completely avoid any existential boxing that shuttling values through `Any` would imply. We are deeply interested in offering this system to systems that are very concerned about allocations, and runtime overheads, and believe this is the right tradeoff to make, while we await the arrival of variadic generics which will solve this system implementation annoyance. + +We are also able to avoid any heap allocations during the `remoteCall` thanks to this approach, as we do not have to construct type erased `arguments: [Any]` which would have been the alternative: + +```swift + func remoteCall( + on recipient: Actor, + method: DistributedMethodIdentifier, + _ args: [Any], // BAD + throwing errorType: Failure.Type, + returning returnType: Success.Type + ) async throws -> Success where Actor: DistributedActor, Actor.ID = ActorID { ... } +``` + +Not only that, but passing arguments as `[Any]` would force developers into using internal machinery to open the existentials (the not officially supported `_openExistential` feature), in order to obtain their specific types, and e.g. use `Codable` with them. + +### Constraining arguments, and return type with of `remoteCall` with `SerializationRequirement` + +Looking at the signature, one might be tempted to also include a `where` clause to statically enforce that all parameters and return type, conform to the `Self.SerializationRequirement`, like so: + +```swift + func remoteCall( + on recipient: Actor, + method: DistributedMethodName, + _ arg1: P1, + throwing errorType: Failure.Type, + returning returnType: Success.Type + ) async throws -> Success where Actor: DistributedActor, + Actor.ID = ActorID, + P1: SerializationRequirement { ... } +// ❌ error: type 'P1' constrained to non-protocol, non-class type 'Self.R' +``` + +However, this is not expressible today in Swift, because we cannot prove the `associatedtype SerializationRequirement` can be used as constraint. + +Fixing this would require introducing new very advanced type system features, and after consultation with the core team we decided to accept this as a current implementation limitation. + +In practice this is not a problem, because the parameters are guaranteed to succeed being cast to `SerializationRequirement` at runtime thanks to the compile-time guarantee about parameters of distributed methods. + +### Hardcoding the distributed runtime to make use of `Codable` + +`Codable` is a great, useful, and relatively flexible protocol allowing for serialization of Swift native types. However, it may not always be the best serialization system available. For example, we currently do not have a great binary serialization format that works with `Codable`, or perhaps some developers just really want to use a 3rd party serialization format such as protocol buffers, SBE or something entirely custom. + +The additional complexity of the configurable `SerializationRequirement` is pulling its weight, and we are not interested in closing down the system to just use Codable. + +## Acknowledgments & Prior art + +We would like to acknowledge the prior art in the space of distributed actor systems which have inspired our design and thinking over the years. Most notably we would like to thank the Akka and Orleans projects, each showing independent innovation in their respective ecosystems and implementation approaches. As these are library-only solutions, they have to rely on wrapper types to perform the hiding of information, and/or source generation; we achieve the same goal by expanding the actor-isolation checking mechanisms already present in Swift. + +We would also like to acknowledge the Erlang BEAM runtime and Elixir language for a more modern take built upon the on the same foundations, which have greatly inspired our design, however take a very different approach to actor isolation (i.e. complete isolation, including separate heaps for actors). + +## Source compatibility + +This change is purely additive to the source language. + +The language impact has been mostly described in the Distributed Actor Isolation proposal + +## Effect on ABI stability + +TODO + +## Effect on API resilience + +None. + +## Changelog + +- 2.0 Adjustments after first round of review + - Expanded discussion on current semantics, and **future directions for versioning** and wire compatibility of remote calls. + - Amendment to **non-delegating distributed actor initializer semantics** + - Non-delegating Initializers no longer require a single `DistributedActorSystem` conforming argument to be passed, and automatically store and initialize the `self.id` with it. + - Instead, users must initialize the `self.actorSystem` property themselves. + - This should generally be done using the same pattern, by passing _in_ an actor system from the outside which helps in testing and stubbing out systems, however if one wanted to. + - However the actorSystem property is initialized, from an initializer parameter or some global value, task-local etc, the general initialization logic remains the same, and the compiler will inject the assignment of the `self.id` property. + - Useful error messages explaining this decision are reported if users attempt to assign to the `id` property directly. + + - Thanks to YR Chen for recognizing this limitation and helping arrive at a better design here. + + - **Removal of explicit use of mangled names** in the APIs, and instead all APIs use an opaque `RemoteCallTarget` that carries an opaque String `identifier` + - The `RemoteCallTarget` is populated by the compiler using mangled names by default, but other schemes can be used in the future + - The `RemoteCallTarget` now pretty prints as "`hello(name:surname:)`" if able to decode the target from the identifier, otherwise it prints the identifier directly. This can be used to include pretty names of call targets in tracing systems etc. + - This design future-proofs the APIs towards potential new encoding schemed we might come up in the future as we tackle a proper and feature complete versioning story for distributed calls. + + - `recordArgument` is passed a **`RemoteCallArgument`** which carries additional information about the argument in question + - This parameter can be either ignored, or stored along the serialized format which may be useful for non swift targets of invocations. E.g. it is possible to store an invocation as JSON object where the argument names are used as labels or similar patterns. + - Thanks to Slava Pestov for suggesting this improvement. + +- 1.3 Larger revision to match the latest runtime developments + - recording arguments does not need to write into provided pointers; thanks to the calls being made in IRGen, we're able to handle things properly even without the heterogenous buffer approach. Thank you, Pavel Yaskevich + - simplify rules of readying actors across synchronous and asynchronous initializers, we can always ready "just before `self` is escaped", in either situation; This is thanks to the latest developments in actor initializer semantics. Thank you, Kavon Farvardin + - express recording arguments and remote calls as "ad-hoc" requirements which are invoked directly by the compiler + - various small cleanups to reflect the latest implementation state +- 1.2 Drop implicitly distributed methods +- 1.1 Implicitly distributed methods +- 1.0 Initial revision +- [Pitch: Distributed Actors](https://forums.swift.org/t/pitch-distributed-actors/51669) + - Which focused on the general concept of distributed actors, and will from here on be cut up in smaller, reviewable pieces that will become their own independent proposals. Similar to how Swift Concurrency is a single coherent feature, however was introduced throughout many interconnected Swift Evolution proposals. + +[isolation]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0336-distributed-actor-isolation.md diff --git a/proposals/0345-if-let-shorthand.md b/proposals/0345-if-let-shorthand.md new file mode 100644 index 0000000000..265d89d873 --- /dev/null +++ b/proposals/0345-if-let-shorthand.md @@ -0,0 +1,382 @@ +# `if let` shorthand for shadowing an existing optional variable + +* Proposal: [SE-0345](0345-if-let-shorthand.md) +* Author: [Cal Stephens](https://github.com/calda) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0345-if-let-shorthand-for-shadowing-an-existing-optional-variable/56364) +* Implementation: [apple/swift#40694](https://github.com/apple/swift/pull/40694) + +## Introduction + +Optional binding using `if let foo = foo { ... }`, to create an unwrapped variable that shadows an existing optional variable, is an extremely common pattern. This pattern requires the author to repeat the referenced identifier twice, which can cause these optional binding conditions to be verbose, especially when using lengthy variable names. We should introduce a shorthand syntax for optional binding when shadowing an existing variable: + +```swift +let foo: Foo? = ... + +if let foo { + // `foo` is of type `Foo` +} +``` + +Swift-evolution thread: [`if let` shorthand](https://forums.swift.org/t/if-let-shorthand/54230) + +## Motivation + +Reducing duplication, especially of lengthy variable names, makes code both easier to write _and_ easier to read. + +For example, this statement that unwraps `someLengthyVariableName` and `anotherImportantVariable` is rather arduous to read (and was without a doubt arduous to write): + +```swift +let someLengthyVariableName: Foo? = ... +let anotherImportantVariable: Bar? = ... + +if let someLengthyVariableName = someLengthyVariableName, let anotherImportantVariable = anotherImportantVariable { + ... +} +``` + +One approach for dealing with this is to use shorter, less descriptive, names for the unwrapped variables: + +```swift +if let a = someLengthyVariableName, let b = anotherImportantVariable { + ... +} +``` + +This approach, however, reduces clarity at the point of use for the unwrapped variables. Instead of encouraging short variable names, we should allow for the ergonomic use of descriptive variable names. + +## Proposed solution + +If we instead omit the right-hand expression, and allow the compiler to automatically shadow the existing variable with that name, these optional bindings are much less verbose, and noticeably easier to read / write: + +```swift +let someLengthyVariableName: Foo? = ... +let anotherImportantVariable: Bar? = ... + +if let someLengthyVariableName, let anotherImportantVariable { + ... +} +``` + +This is a fairly natural extension to the existing syntax for optional binding conditions. + +## Detailed design + +Specifically, this proposal extends the Swift grammar for [`optional-binding-condition`](https://docs.swift.org/swift-book/ReferenceManual/Statements.html#grammar_optional-binding-condition)s. + +This is currently defined as: + +> optional-binding-condition → **let** [pattern](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_pattern) [initializer](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_initializer) | **var** [pattern](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_pattern) [initializer](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_initializer) + +and would be updated to: + +> optional-binding-condition → **let** [pattern](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_pattern) [initializer](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_initializer)opt | **var** [pattern](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_pattern) [initializer](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_initializer)opt + +This would apply to all conditional control flow statements: + +```swift +if let foo { ... } +if var foo { ... } + +else if let foo { ... } +else if var foo { ... } + +guard let foo else { ... } +guard var foo else { ... } + +while let foo { ... } +while var foo { ... } +``` + +The compiler would synthesize an initializer expression that references the variable being shadowed. + +For example: + +```swift +if let foo { ... } +``` + +is transformed into: + +```swift +if let foo = foo { ... } +``` + +Explicit type annotations are permitted, like with standard optional binding conditions. + +For example: + +```swift +if let foo: Foo { ... } +``` + +is transformed into: + +```swift +if let foo: Foo = foo { ... } +``` + +The pattern following the introducer serves as both an evaluated expression _and_ an identifier for the newly-defined non-optional variable. Existing precedent for this type of syntax includes closure capture lists, which work the same way: + +```swift +let foo: Foo +let closure = { [foo] in // `foo` is both an expression and the identifier + ... // for a new variable defined within the closure +} +``` + +Because of this, only valid identifiers would be permitted with this syntax. For example, this example would not be valid: + +```swift +if let foo.bar { ... } // 🛑 unwrap condition requires a valid identifier + ^ // fix-it: insert `<#identifier#> = ` +``` + +### Interaction with implicit self + +Like with existing optional bindings, this new syntax would support implifict self references to unwrap optional members of `self`. For example, the usage in this example would be permitted: + +```swift +struct UserView: View { + let name: String + let emailAddress: String? + + var body: some View { + VStack { + Text(user.name) + + // Equivalent to `if let emailAddress = emailAddress { ... }`, + // unwraps `self.emailAddress`. + if let emailAddress { + Text(emailAddress) + } + } + } +} +``` + +## Source compatibility + +This change is purely additive and does not break source compatibility of any valid existing Swift code. + +## Effect on ABI stability + +This change is purely additive, and is a syntactic transformation to existing valid code, so has no effect on ABI stability. + +## Effect on API resilience + +This change is purely additive, and is a syntactic transformation to existing valid code, so has no effect on ABI stability. + +## Future directions + +### Optional casting + +A natural extension of this new syntax could be to support shorthand for optional casting. For example: + +`if let foo as? Bar { ... }` + +could be equivalent to: + +`if let foo = foo as? Bar { ... }` + +This is not included in this proposal, but is a reasonable feature that could be added in the future. + +### Interaction with future borrow introducers + +["A roadmap for improving Swift performance predictability"](https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206#borrow-variables-7) discusses potential new `ref` and `inout` introducers for creating variables that "borrow" existing variables without making a copy (by enforcing exclusive access). For consistency with `let` / `var`, it will likely make sense to support optional binding conditions for these new introducers: + +```swift +if ref foo = foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, immutable variable +} + +if inout foo = &foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, mutable variable +} +``` + +The shorthand syntax for `let` / `var` optional bindings would extend fairly naturally to these new introducers: + +```swift +if ref foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, immutable variable +} + +if inout &foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, mutable variable +} +``` + +### Unwrapping nested members of objects + +This proposal doesn't permit shorthand unwrapping for members nested in other objects. For example: + +`if let foo.bar { ... } // 🛑` + +There are a few different options that could allow us to support this type of syntax in the future. + +One approach could be to automatically synthesize the identifier name for the unwrapped variable in the inner scope. For example. `if let foo.bar` could introduce a new non-optional variable named `bar` or `fooBar`. + +Another approach could be to permit this for potential future borrow introducers `ref` and `inout` (from ["A roadmap for improving Swift performance predictability"](https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206#borrow-variables-7)). These borrows would have compiler-enforced exclusive access to the underlying storage, so they technically do not require a unique identifier name for the inner scope. This could allow us to unwrap members of objects without any new variables or copies. For example: + +```swift +// `mother.father.sister` is optional + +if ref mother.father.sister { + // `mother.father.sister` is non-optional and immutable +} + +if inout &mother.father.sister { + // `mother.father.sister` is non-optional and mutable +} +``` + +## Alternatives considered + +There have been many other proposed spellings for this feature: + +### `if foo` + +The briefest possible spelling for this feature would just be a bare `if foo` condition. This spelling, however, would create ambiguity between optional unwrapping conditions and boolean conditions, and could lead to confusing / conter-intuitive situations: + +```swift +let foo: Bool = true +let bar: Bool? = false + +if foo, bar { + // would succeed +} + +if foo == true, bar == true { + // would fail +} +``` + +To avoid this ambiguity, we need some sort of distinct syntax for optional bindings. + +### `if unwrap foo` + +Another option is to introduce a new keyword or sigil for this purpose, like `if unwrap foo`, `if foo?` or `if have foo`. + +A key benefit of introducing a completely new syntax like `if unwrap foo` is that it gives us the opportunity to also revisit the _semantics_ of how optional binding conditions actually work. Today, optional binding conditions always make a copy of the value. From a performance perspective, it would be more efficient to perform a _borrow_ instead of a copy. + +["A roadmap for improving Swift performance predictability"](https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206#borrow-variables-7) discusses potential future introducers `ref` (to perform an immutable borrow) and `inout` (to perform a mutable borrow). For consistency with `let` / `var`, it will likely make sense to support optional binding conditions for these new introducers: + +```swift +if ref foo = foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, immutable variable +} + +if inout foo = &foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, mutable variable +} +``` + +Instead of being shorthand for `if let`, this new shorthand syntax could instead be shorthand for `if ref`. This would improve performance in general, and could nudge users towards using borrows instead of copies (since only the borrow form would receive shorthand sugar). + +A key downside of borrows, however, is that they require exclusive access to the borrowed variable. Memory exclusivity violations will result in compiler errors in some cases, but can also manifest as runtime errors in more complex cases. For example: + +```swift +var x: Int? = 1 + +func increment(by number: Int) { + x? += number +} + +if ref x = x { + increment(by: x) +} +``` + +This would trap at runtime, because `increment(by:)` would attempt to modify the value of `x` while it is already being borrowed by the `if ref x = x` optional binding condition. + +Once borrow introducers are added to the language, seeing `ref x` or `inout x` anywhere in Swift will serve as an important visual marker about the exclusivity requirements of the code. On the other hand, a new syntax like `if unwrap x` doesn't explicitly indicate that the variable is being borrowed. This could lead to users being surprised by unexpected exclusivity violations, which could cause confusing compile-time errors or runtime crashes. + +Borrow introducers will be very useful, but adopting them is a tradeoff between performance and conceptual overhead. Borrows are cheap but come with high conceptual overhead. Copies can be expensive but always work as expected without much extra thought. Given this tradeoff, it likely makes sense for this shorthand syntax to provide a way for users to choose between performing a copy or performing a borrow, rather than limiting users to one or the other. + +Additionally, for consistency with existing optional binding conditions, this new shorthand should support the distinction between immutable and mutable variables. Combined with the distinction between copies and borrows, that would give us the same set of options as normal variables: + +```swift +// Included in this proposal: +if let foo { /* foo is an immutable copy */ } +if var foo { /* foo is a mutable copy */ } + +// Potentially added in the future: +if ref foo { /* foo is an immutable borrow */ } +if inout &foo { /* foo is a mutable borrow */ } +``` + +Since we already have syntax for these concepts, we should reuse that syntax in this shorthand rather than create a new syntax that is less expressive (e.g. only supports a subset of the available options) and less explicit (e.g. that users would have to memorize whether this new shorthand performs a copy or a borrow). + +### `if let foo?` + +Another option is to include a `?` to explicitly indicate that this is unwrapping an optional, using `if let foo?`. This is indicative of the existing `case let foo?` pattern matching syntax. + +`if let foo = foo` (the most common existing syntax for this) unwraps optionals without an explicit `?`. This implies that a conditional optional binding is sufficiently clear without a `?` to indicate the presence of an optional. If this is the case, then an additional `?` is likely not strictly necessary in the shorthand `if let foo` case. + +While the symmetry of `if let foo?` with `case let foo?` is nice, consistency with `if let foo = foo` is even more important condiering they will more frequently appear within the same statement: + +```swift +// Consistent +if let user, let defaultAddress = user.shippingAddresses.first { ... } + +// Inconsistent +if let user?, let defaultAddress = user.shippingAddresses.first { ... } +``` + +Additionally, the `?` symbol makes it trickier to support explicit type annotations like in `if let foo: Foo = foo`. `if let foo: Foo` is a natural consequence of the existing grammar. It's less clear how this would work with an additional `?`. `if let foo?: Foo` likely makes the most sense, but doesn't match any existing language constructs. + +### `if foo != nil` + +One somewhat common proposal is to permit `nil`-checks (like `if foo != nil`) to unwrap the variable in the inner scope. Kotlin supports this type of syntax: + +```kt +var foo: String? = "foo" +print(foo?.length) // "3" + +if (foo != null) { + // `foo` is non-optional + print(foo.length) // "3" +} +``` + +This pattern in Kotlin _does not_ define a new variable -- it merely changes the type of the existing variable within the inner scope. So mutations that affect the inner scope also affect the outer scope: + +```kt +var foo: String? = "foo" + +if (foo != null) { + print(foo) // "foo" + foo = "bar" + print(foo) // "bar" +} + +print(foo) // "bar" +``` + +This is different from Swift's optional binding conditions (`if let foo = foo`), which define a new, _separate_ variable in the inner scope. This is a defining characteristic of optional binding conditions in Swift, so any shorthand syntax must make it abundantly clear that a new variable is being declared. + +### Don't permit `if var foo` + +Since `if var foo = foo` is significantly less common than `if let foo = foo`, we could potentially choose to _not_ support `var` in this shorthand syntax. + +`var` shadowing has the potential to be more confusing than `let` shadowing -- `var` introduces a new _mutable_ variable, and any mutations to the new variable are not shared with the original optional variable. On the other hand, `if var foo = foo` already exists, and it seems unlikely that `if var foo` would be more confusing / less clear than the existing syntax. + +Since `let` and `var` are interchangeable elsewhere in the language, that should also be the case here -- disallowing `if var foo` would be inconsistent with existing optional binding condition syntax. If we were using an alternative spelling that _did not_ use `let`, it may be reasonable to exclude `var` -- but since we are using `let` here, `var` should also be allowed. + +## Acknowledgments + +Many thanks to Craig Hockenberry, who recently wrote about this topic in [Let’s fix `if let` syntax](https://forums.swift.org/t/lets-fix-if-let-syntax/48188) which directly informed this proposal. + +Thanks to Ben Cohen for suggesting the alternative `if let foo?` spelling, and for providing valuable feedback on this proposal during the pitch phase. + +Thanks to Chris Lattner for suggesting to consider how this proposal should interact with upcoming language features like potential `ref` and `inout` borrow introducers. + +Thanks to [tera](https://forums.swift.org/u/tera/summary) for suggesting the alternative `if foo` spelling. + +Thanks to Jon Shier for providing the SwiftUI optional binding example. + +Thanks to James Dempsey for providing the "consistency with existing optional binding conditions" example. + +Thanks to Frederick Kellison-Linn for pointing out that variables in closure capture lists are an existing precedent for this type of syntax. diff --git a/proposals/0346-light-weight-same-type-syntax.md b/proposals/0346-light-weight-same-type-syntax.md new file mode 100644 index 0000000000..7199446d35 --- /dev/null +++ b/proposals/0346-light-weight-same-type-syntax.md @@ -0,0 +1,496 @@ +# Lightweight same-type requirements for primary associated types + +* Proposal: [SE-0346](0346-light-weight-same-type-syntax.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin), [Holly Borla](https://github.com/hborla), [Slava Pestov](https://github.com/slavapestov) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Previous Revisions: [1st](https://github.com/swiftlang/swift-evolution/blob/5d86d57cfd6d803df4da90b196682d495e5de9b9/proposals/0346-light-weight-same-type-syntax.md) +* Review: ([first pitch](https://forums.swift.org/t/pitch-light-weight-same-type-constraint-syntax/52889)) ([second pitch](https://forums.swift.org/t/pitch-2-light-weight-same-type-requirement-syntax/55081)) ([first review](https://forums.swift.org/t/se-0346-lightweight-same-type-requirements-for-primary-associated-types/55869)) ([second review](https://forums.swift.org/t/se-0346-second-review-lightweight-same-type-requirements-for-primary-associated-types/56414)) ([acceptance](https://forums.swift.org/t/accepted-se-0346-lightweight-same-type-requirements-for-primary-associated-types/56747)) + +## Introduction + +As a step toward the goal of improving the UI of generics outlined in [Improving the UI of Generics](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--directly-expressing-constraints), this proposal introduces a new syntax for conforming a generic parameter and constraining an associated type via a same-type requirement. + +## Motivation + +Consider a function that returns a `Sequence` of lines in a source file: + +```swift +struct LineSequence : Sequence { + struct Iterator : IteratorProtocol { + mutating func next() -> String? { ... } + } + + func makeIterator() -> Iterator { + return Iterator() + } +} + +func readLines(_ file: String) -> LineSequence { ... } +``` + +Suppose you are implementing a syntax highlighting library. You might define another function which wraps the result in a `SyntaxTokenSequence`, whose element type is `[Token]`, representing an array of syntax-highlighted tokens on each line: + +```swift +func readSyntaxHighlightedLines(_ file: String) + -> SyntaxTokenSequence { + ... +} +``` + +At this point, the concrete result type is rather complex, and we might wish to hide it behind an opaque result type using the `some` keyword: + +```swift +func readSyntaxHighlightedLines(_ file: String) -> some Sequence { + ... +} +``` + +However, the resulting definition of `readSyntaxHighlightedLines()` is not as useful as the original, because the requirement that the `Element` associated type of the resulting `Sequence` is equal to `[Token]` cannot be expressed. + +As another example, consider a global function `concatenate` that operates on two arrays of `String`: + +```swift +func concatenate(_ lhs: Array, _ rhs: Array) -> Array { + ... +} +``` + +To generalize this function to arbitrary sequences, one might write: + +```swift +func concatenate(_ lhs: S, _ rhs: S) -> S where S.Element == String { + ... +} +``` + +However, while `where` clauses are very general and allow complex generic requirements to be expressed, they also introduce cognitive overhead when reading and writing the declaration, and looks quite different than the concrete implementation where the type was simply written as `Array`. It would be nice to have a simpler solution for cases where there is only a single same-type requirement, as above. + +## Proposed solution + +We’d like to propose a new syntax for declaring a protocol conformance requirement together with one or more same-type requirements on the protocol's _primary associated types_. This new syntax looks like the application of a concrete generic type to a list of type arguments, allowing you to write `Sequence` or `Sequence<[Token]>`. This builds on the user's previous intuition and understanding of generic types and is analogous to `Array` and `Array<[Token]>`. + +Protocols can declare one or more primary associated types using a syntax similar to a generic parameter list of a concrete type: + +```swift +protocol Sequence { + associatedtype Element + associatedtype Iterator : IteratorProtocol + where Element == Iterator.Element + ... +} + +protocol DictionaryProtocol { + associatedtype Key : Hashable + associatedtype Value + ... +} +``` + +A protocol with primary associated types can be written with a list of type arguments in angle brackets, from any position where a protocol conformance requirement was previously allowed. + +For example, an opaque result type can now constrain the primary associated type: + +```swift +func readSyntaxHighlightedLines(_ file: String) -> some Sequence<[Token]> { + ... +} +``` + +The `concatenate()` function shown earlier can now be written like this: + +```swift +func concatenate>(_ lhs: S, _ rhs: S) -> S { + ... +} +``` + +Primary associated types are intended to be used for associated types which are usually provided by the caller. These associated types are often witnessed by generic parameters of the conforming type. For example, `Element` is a natural candidate for the primary associated type of `Sequence`, since `Array` and `Set` both conform to `Sequence`, with the `Element` associated type witnessed by a generic parameter in the corresponding concrete types. This introduces a clear correspondence between the constrained protocol type `Sequence` on one hand and the concrete types `Array`, `Set` on the other hand. + +## Detailed design + +At the protocol declaration, an optional _primary associated types list_ delimited by angle brackets can follow the protocol name. When present, at least one primary associated type must be named. Multiple primary associated types are separated by commas. Each entry in the primary associated type list must name an existing associated type declared in the body of the protocol or one of its inherited protocols. The formal grammar is amended as follows, adding an optional **primary-associated-type-list** production to **protocol-declaration**: + +- **protocol-declaration** → attributesopt access-level-modifieropt `protocol` protocol-name primary-associated-type-listopt type-inheritance-clauseopt generic-where-clauseopt protocol-body +- **primary-associated-type-list** → `<` primary-associated-type-entry `>` +- **primary-associated-type-entry** → primary-associated-type | primary-associated-type `,` primary-associated-type-entry +- **primary-associated-type** → type-name + +Some examples: + +```swift +// Primary associated type 'Element' is declared inside the protocol +protocol SetProtocol { + associatedtype Element : Hashable + ... +} + +protocol SortedMap { + associatedtype Key + associatedtype Value +} + +// Primary associated types 'Key' and 'Value' are declared inside +// the inherited 'SortedMap' protocol +protocol PersistentSortedMap : SortedMap { + ... +} +``` + +At the usage site, a _constrained protocol type_ may be written with one or more type arguments, like `P`. Omitting the list of type arguments altogether is permitted, and leaves the protocol unconstrained. Specifying fewer or more type arguments than the number of primary associated types is an error. Adding a primary associated type list to a protocol is a source-compatible change; the protocol can still be referenced without angle brackets as before. + +### Constrained protocols in desugared positions + +An exhaustive list of positions where the constrained protocol syntax may appear follows. In the first set of cases, the new syntax is equivalent to the existing `where` clause syntax with a same-type requirement constraining the primary associated types. + +- The extended type of an extension, for example: + + ```swift + extension Collection { ... } + + // Equivalent to: + extension Collection where Element == String { ... } + ``` + +- The inheritance clause of another protocol, for example: + + ```swift + protocol TextBuffer : Collection { ... } + + // Equivalent to: + protocol TextBuffer : Collection where Element == String { ... } + ``` + +- The inheritance clause of a generic parameter, for example: + + ```swift + func sortLines>(_ lines: S) -> S + + // Equivalent to: + func sortLines(_ lines: S) -> S + where S.Element == String + ``` + +- The inheritance clause of an associated type, for example: + + ```swift + protocol Document { + associatedtype Lines : Collection + } + + // Equivalent to: + protocol Document { + associatedtype Lines : Collection + where Lines.Element == String + } + ``` + +- The right-hand side of a conformance requirement in a `where` clause, for example: + + ```swift + func merge(_ sequences: S) + where S.Element : Sequence + + // Equivalent to: + func merge(_ sequences: S) + where S.Element : Sequence, S.Element.Element == String + ``` + +- An opaque parameter declaration (see [SE-0341 Opaque Parameter Declarations](0341-opaque-parameters.md)): + + ```swift + func sortLines(_ lines: some Collection) + + // Equivalent to: + func sortLines>(_ lines: C) + + // In turn equivalent to: + func sortLines(_ lines: C) + where C.Element == String + ``` + +- The protocol arguments can contain nested opaque parameter declarations. For example, + + ```swift + func sort(elements: inout some Collection) {} + + // Equivalent to: + func sort(elements: inout C) {} + where C.Element == E + ``` + +When referenced from one of the above positions, a conformance requirement `T : P` desugars to a conformance requirement `T : P` followed by one or more same-type requirements: + +```swift +T : P +T.PrimaryType1 == Arg1 +T.PrimaryType2 == Arg2 +... +``` + +If the right hand side `Arg1` is itself an opaque parameter type, a fresh generic parameter is introduced for use as the right-hand side of the same-type requirement. See [SE-0341 Opaque Parameter Declarations](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md) for details. + +### Constrained protocols in opaque result types + +- A constrained protocol may appear in an opaque result type specified by the `some` keyword. In this case, the syntax actually allows you to express something that was previously not possible to write, since we do not allow `where` clauses on opaque result types: + + ```swift + func transformElements, E>(_ lines: S) -> some Sequence + ``` + + This example also demonstrates that the argument can itself depend on generic parameters from the outer scope. + + The [SE-0328 Structural Opaque Result Types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0328-structural-opaque-result-types.md) pitch allows multiple occurrences of `some` in a return type. This generalizes to constrained protocol types, whose constraint can be another opaque result type: + + ```swift + func transform(_: some Sequence) -> some Sequence + ``` + + Note that in the above, the opaque result type `some Sequence` is unrelated to the opaque _parameter_ type `some Sequence`. The parameter type is provided by the caller. The opaque result type is a (possibly different) homogeneous sequence of elements, where the element type is known to conform to `some Equatable` but is otherwise opaque to the caller. + +### Other positions + +There are three more places where constrained protocols may appear: + +- In the inheritance clause of a concrete type, for example: + + ```swift + struct Lines : Collection { ... } + ``` + + In this position it is sugar for specifying the associated type witness, similar to explicitly declaring a typealias: + + ```swift + struct Lines : Collection { + typealias Element = String + } + ``` + +- As the underlying type of a typealias: + + ```swift + typealias SequenceOfInt = Sequence + ``` + + The typealias may be used in any position where the constrained protocol type itself would be used. + +- As a member of a protocol composition, when the protocol composition appears in any position where a constrained protocol type would be valid: + + ```swift + func takeEquatableSequence(_ seqs: some Sequence & Equatable) {} + ``` + +### Unsupported positions + +A natural generalization is to enable this syntax for existential types, e.g. `any Collection`. This is a larger feature that needs careful consideration of type conversion behaviors. It will also require runtime support for metadata and dynamic casts. For this reason it will be covered by a separate proposal. + +## Alternatives considered + +### Treat primary associated type list entries as declarations + +In an earlier revision of this proposal, the associated type list entries would declare new associated types, instead of naming associated types declared in the body. That is, you would write + +```swift +protocol SetProtocol { + ... +} +``` + +instead of + +```swift +protocol SetProtocol { + associatedtype Key : Hashable + ... +} +``` + +We felt that allowing declaration of associated types in the primary associated type list promotes confusion that what’s actually happening here is a generic declaration, when it is semantically different in important ways. Another potential source of confusion would be if a primary associated type declared a default type witness: + +```swift +protocol SetProtocol { + ... +} +``` + +This makes it look like a "defaulted generic parameter", which it is not; writing `SetProtocol` means leaving `Key` unconstrained, and is not the same as `SetProtocol`. The new form makes it clearer that what is going on here is that a default is being declared for conformances to the protocol, and not for the usage site of the generic constraint: + +```swift +protocol SetProtocol { + associatedtype Key : Hashable = String + ... +} +``` + +### Require associated type names, e.g. `Collection<.Element == String>` + +Explicitly writing associated type names to constrain them in angle brackets has a number of benefits: + +* Doesn’t require any special syntax at the protocol declaration. +* Explicit associated type names allows constraining arbitrary associated types. + +There are also a number of drawbacks to this approach: + +* No visual clues at the protocol declaration about what associated types are useful. +* The use-site may become onerous. For protocols with only one primary associated type, having to specify the name of it is unnecessarily repetitive. +* The syntax can be confusing when the constrained associated type has the same name as a generic parameter of the declaration. For example, the following: + + ```swift + func adjacentPairs(_: some Sequence, + _: some Sequence) + -> some Sequence<(Element, Element)> {} + ``` + + reads better than the hypothetical alternative: + + ```swift + func adjacentPairs(_: some Sequence<.Element == Element>, + _: some Sequence<.Element == Element>) + -> some Sequence<.Element == (Element, Element)> {} + ``` + +* This more verbose syntax is not as clear of an improvement over the existing syntax today, because most of the where clause is still explicitly written. This may also encourage users to specify most or all generic constraints in angle brackets at the front of a generic signature instead of in the `where` clause, violates a core tenet of [SE-0081 Move where clause to end of declaration](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0081-move-where-expression.md). + +* Finally, this syntax lacks the symmetry between concrete types and generic types; generalizing from `Array` requires learning and writing the novel syntax `some Collection<.Element == Int>` instead of simply `some Collection`. + +Note that nothing in this proposal _precludes_ adding the above syntax in the future; the presence of a leading dot (or some other signifier) should allow unambiguous parsing in either case. + +### Implement more general syntax for opaque result type requirements first + +As previously mentioned, in the case of opaque result types, this proposal introduces new expressive power, since opaque result types cannot have a `where` clause where a same-type requirement on a primary associated type could otherwise be written. + +It would be possible to first introduce a language feature allowing general requirements on opaque result types. One such possibility is "named opaque result types", which can have requirements imposed upon them in a `where` clause: + +```swift +func readLines(_ file: String) -> some Sequence { ... } + +// Equivalent to: +func readLines(_ file: String) -> S + where S : Sequence, S.Element == String { ... } +``` + +However, the goal of this proposal is to make generics more approachable by introducing a symmetry between concrete types and generics, and make generics feel more like a generalization of what programmers coming from other languages are already familiar with. + +A more general syntax for opaque result types can be considered on its own merits, and as with the `some Collection<.Element == Int>` syntax discussed in the previous section, nothing in this proposal precludes opaque result types from being generalized further in the future. + +### Annotate regular `associatedtype` declarations with `primary` + +Adding some kind of modifier to `associatedtype` declaration shifts complexity to the users of an API because it’s still distinct from how generic types declare their parameters, which goes against the progressive disclosure principle, and, if we choose to generalize this proposal to multiple primary associated types in the future, requires an understanding of ordering on the use-site. + +This would also make declaration order significant, in a way that is not currently true for the members of a protocol definition. + +Annotation of associated type declarations could make it easier to conditionally declare a protocol which defines primary associated types in newer compiler versions only. The syntax described in this proposal applies to the protocol declaration itself. As a consequence, a library wishing to adopt this feature in a backwards-compatible manner must duplicate entire protocol definitions behind `#if` blocks: + +```swift +#if swift(>=5.7) +protocol SetProtocol { + associatedtype Element : Hashable + + var count: Int { get } + ... +} +#else +protocol SetProtocol { + associatedtype Element : Hashable + + var count: Int { get } + ... +} +#endif +``` + +With a hypothetical `primary` keyword, only the primary associated types must be duplicated: + +```swift +protocol SetProtocol { +#if swift(>=5.7) + primary associatedtype Element : Hashable +#else + associatedtype Element : Hashable +#if + + var count: Int { get } + ... +} +``` + +However, duplicating the associated type declaration in this manner is still an error-prone form of code duplication, and it makes the code harder to read. We feel that this use case should not unnecessarily hinder the evolution of the language syntax. The concerns of libraries adopting new language features while remaining compatible with older compilers is not unique to this proposal, and would be best addressed with a third-party pre-processor tool. + +With the current proposed syntax, it is sufficient for the pre-processor to strip out everything between angle brackets after the protocol name to produce a backward-compatible declaration. A minimal implementation of such a pre-processor is a simple sed invocation, like `sed -e 's/\(protocol .*\)<.*> {/\1 {/'`. + +### Generic protocols + +This proposal uses the angle-bracket syntax for constraining primary associated types, instead of a hypothetical "generic protocols" feature modeled after Haskell's multi-parameter typeclasses or Rust's generic traits. The idea is that such a "generic protocol" can be parametrized over multiple types, not just a single `Self` conforming type: + +```swift +protocol ConvertibleTo { + static func convert(_: Self) -> Other +} + +extension String : ConvertibleTo { + static func convert(_: String) -> Int +} + +extension String : ConvertibleTo { + static func convert(_: String) -> Double +} +``` + +We believe that constraining primary associated types is a more generally useful feature than generic protocols, and using angle-bracket syntax for constraining primary associated types gives users what they generally expect, with the clear analogy between `Array` and `Collection`. + +Nothing in this proposal precludes introducing generic protocols in the future under a different syntax, perhaps something that does not privilege the `Self` type over other types to make it clear there is no functional dependency between the type parameters like there is with associated types: + +```swift +protocol Convertible(from: Self, to: Other) { + static func convert(_: Self) -> Other +} + +extension Convertible(from: String, to: Int) { + static func convert(_: String) -> Int +} + +extension Convertible(from: String, to: Double) { + static func convert(_: String) -> Double +} +``` + +## Source compatibility + +This proposal does not impact source compatibility for existing code. + +Adding a primary associated type list to an existing protocol is a source-compatible change. + +The following are **source-breaking** changes: + +- Removing the primary associated type list from an existing protocol. +- Changing the order or contents of a primary associated type list of an existing protocol. + +## Effect on ABI stability + +This proposal does not impact ABI stability for existing code. The new feature does not require runtime support and can be backward-deployed to existing Swift runtimes. + +The primary associated type list is not part of the ABI, so all of the following are binary-compatible changes: + +- Adding a primary associated type list to an existing protocol. +- Removing the primary associated type list from a existing protocol. +- Changing the primary associated type list of an existing protocol. + +The last two are source-breaking however, so are not recommended. + +## Effect on API resilience + +This change does not impact API resilience for existing code. + +## Future Directions + +### Standard library adoption + +Actually adopting primary associated types in the standard library is outside of the scope of this proposal. There are the obvious candidates such as `Sequence` and `Collection`, and no doubt others that will require additional discussion. + +### Constrained existentials + +As stated above, this proposal alone does not enable constrained protocol existential types, such as `any Collection`. + +## Acknowledgments + +Thank you to Joe Groff for writing out the original vision for improving generics ergonomics — which included the initial idea for this feature — and to Alejandro Alonso for implementing the lightweight same-type constraint syntax for extensions on generic types which prompted us to think about this feature again for protocols. diff --git a/proposals/0347-type-inference-from-default-exprs.md b/proposals/0347-type-inference-from-default-exprs.md new file mode 100644 index 0000000000..db105e1d92 --- /dev/null +++ b/proposals/0347-type-inference-from-default-exprs.md @@ -0,0 +1,264 @@ +# Type inference from default expressions + +* Proposal: [SE-0347](0347-type-inference-from-default-exprs.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0347-type-inference-from-default-expressions/56558) +* Implementation: [apple/swift#41436](https://github.com/apple/swift/pull/41436) + +## Introduction + +It's currently impossible to use a default value expression with a generic parameter type to default the argument and its type: + +```swift +func compute(_ values: C = [0, 1, 2]) { ❌ + ... +} +``` + +An attempt to compile this declaration results in the following compiler error - `default argument value of type '[Int]' cannot be converted to type 'C'` because, under the current semantic rules, the type of a default expression has to work for every possible concrete type replacement of `C` inferred at a call site. There are couple of ways to work around this expressivity limitation, but all of them require overloading which complicates APIs: + +``` +func compute(_ values: C) { // original declaration without default + ... +} + +func compute(_ values: [Int] = [0, 1, 2]) { // concretely typed overload of `compute` with default value + ... +} +``` + +I propose to allow type inference for generic parameters from concretely-typed default parameter values (referred to as default expressions in the proposal) when the call-site omits an explicit argument. Concretely-typed default expressions would still be rejected by the compiler if generic parameters associated with a defaulted parameter could be inferred _at a call site_ from any other location in a parameter list by an implicit or explicit argument. For example, declaration `func compute(_: T = 42, _: U) where U: Collection, U.Element == T` is going to be rejected by the compiler because it's possible to infer a type of `T` from the second argument, but declaration `func compute(_: T = 42, _: U = []) where U: Collection, U.Element == Int` is going to be accepted because `T` and `U` are independent. + +Under the proposed rules, the original `compute` declaration becomes well formed and doesn't require any additional overloads: + +```swift +func compute(_ values: C = [0, 1, 2]) { ✅ + ... +} +``` + +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/pitch-type-inference-from-default-expressions/55585) + + +## Motivation + +Interaction between generic parameters and default expressions is confusing when default expression only works for a concrete specialization of a generic parameter. It's possible to spell it in the language (in some circumstances) but requires boiler-plate code and knowledge about nuances of constrained extensions. + +For example, let's define a `Flags` protocol and a container type for default set of flags: + +```swift +protocol Flags { + ... +} + +struct DefaultFlags : Flags { + ... +} +``` + +Now, let's declare a type that accepts a set of flags to act upon during initialization. + +```swift +struct Box { + init(dimensions: ..., flags: F) { + ... + } +} +``` + + +To create a `Box` , the caller would have to pass an instance of type conforming to `Flags` to its initializer call. If the majority of `Box`es doesn’t require any special flags, this makes for subpar API experience, because although there is a `DefaultFlags` type, it’s not currently possible to provide a concretely typed default value for the `flags` parameter, e.g. (`flags: F = DefaultFlags()`). Attempting to do so results in the following error: + +``` +error: default argument value of type 'DefaultFlags' cannot be converted to type 'F' +``` + +This happens because even though `DefaultFlags` does conform to protocol `Flags` the default value cannot be used for _every possible_ `F` that can be inferred at a call site, only when `F` is `DefaultFlags`. + +To avoid having to pass flags, it's possible to "specialize" the initializer over a concrete type of `F` via combination of conditional extension and overloading. + +Let’s start with a direct `where` clause: + +```swift +struct Box { + init(dimensions: ..., flags: F = DefaultFlags()) where F == DefaultFlags { + ... + } +} +``` + +This `init` declaration results in a loss of memberwise initializers for `Box`. + +Another possibility is a constrained extension which makes `F` concrete `DefaultFlags` like so: + +```swift +extension Box where F == DefaultFlags { + init(dimensions: ..., flags: F = DefaultFlags()) { + ... + } +} +``` + +Initialization of `Box` without `flags:` is now well-formed and implicit memberwise initializers are preserved, albeit with `init` now being overloaded, but this approach doesn’t work in situations where generic parameters belong to the member itself. + +Let’s consider that there is an operation on our `Box` type that requires passing a different set of flags: + +```swift +extension Box { + func ship(_ flags: F) { + ... + } +} +``` + + +The aforementioned approach that employs constrained extension doesn’t work in this case because generic parameter `F` is associated with the method `ship` instead of the `Box` type. There is another trick that works in this case - overloading. + + New method would have to have a concrete type for `flags:` like so: + +```swift +extension Box { + func ship(_ flags: DefaultShippingFlags = DefaultShippingFlags()) { + ... + } +} +``` + +This is a usability pitfall - what works for some generic parameters, doesn’t work for others, depending on whether the parameter is declared. This inconsistency sometimes leads to API authors reaching for existential types, potentially without realizing all of the consequences that might entail, because a declaration like this would be accepted by the compiler: + +```swift +extension Box { + func ship(_ flags: any Flags = DefaultShippingFlags()) { + ... + } +} +``` + + Also, there is no other way to associate default value `flags:` parameter without using existential types for enum declarations: + +```swift +enum Box { +} + +extension Box where F == DefaultFlags { + case flatRate(dimensions: ..., flags: F = DefaultFlags()) ❌ // error: enum 'case' is not allowed outside of an enum +} +``` + + +To summarize, there is a expressivity limitation related to default expressions which could be, only in some circumstances, mitigated via constrained extensions feature, its other issues include: + +1. Doesn’t work for generic parameters associated with function, subscript, or case declarations because constrained extensions could only be declared for types i.e. `init(..., flags: F = F()) where F == DefaultFlags` is not allowed. +2. Methods have to be overloaded, which increases API surface of the `Box` , and creates a matrix of overloads if there are more than combination of parameters with default values required i.e. if `dimensions` parameter was to be made generic and defaulted for some box sides. +3. Doesn’t work for `enum` declarations at all because Swift does not support overloading cases or declaring them in extensions. +4. Requires know-how related to constrained extensions and their ability to bind generic parameters to concrete types. + +## Proposed solution + +To address the aforementioned short-comings of the language, I propose to support a more concise and intuitive syntax - to allow concretely typed default expressions to be associated with parameters that refer to generic parameters. + +```swift +struct Box { + init(flags: F = DefaultFlags()) { + ... + } +} + +Box() // F is inferred to be DefaultFlags +Box(flags: CustomFlags()) // F is inferred to be CustomFlags +``` + +This syntax could be achieved by amending the type-checking semantics associated with default expressions to allow type inference from them at call sites in cases where such inference doesn’t interfere with explicitly passed arguments. + +## Detailed design + +Type inference from default expressions would be allowed if: + +1. The generic parameter represents either a direct type of a parameter i.e. `(_: T = ...)` or used in a nested position i.e. `(_: [T?] = ...)` +2. The generic parameter is used only in a single location in the parameter list. For example, `(_: T, _: T = ...)` or `(_: [T]?, _: T? = ...)` are *not* allowed because only an explicit argument is permitted to resolve a type conflict to avoid any surprising behavior related to implicit joining of the types. + 1. Note: A result type is allowed to reference generic parameter types inferable from default expressions to make it possible to use the feature while declaring initializers of generic types or `case`s of generic enums. +3. There are no same-type generic constraints that relate a generic parameter that could be inferred from a default expression with any other parameter that couldn’t be inferred from the same expression. For example, `(_: T = [...], _: U) where T.Element == U` is not allowed because `U` is not associated with defaulted parameter where `T` is used, but `(_: [(K, V?)] = ...) where K.Element == V` is permitted because both generic parameters are associated with one expression. +4. The default expression produces a type that satisfies all of the conformance, layout and other generic requirements placed on each generic parameter it would be used to infer at a call site. + + +With these semantic updates, both the initializer and `ship` method of the `Box` type could be expressed in a concise and easily understandable way that doesn’t require any constrained extensions or overloading: + +```swift +struct Box { + init(dimensions: ..., flags: F = DefaultFlags()) { + ... + } + + func ship(_ flags: F = DefaultShippingFlags()) { + ... + } +} +``` + +`Box` could also be converted to an enum without any loss of expressivity: + +```swift +enum Box { +case flatRate(dimensions: D = [...], flags: F = DefaultFlags()) +case overnight(dimensions: D = [...], flags: F = DefaultFlags()) +... +} +``` + +At the call site, if the defaulted parameter doesn’t have an argument, the type-checker will form an argument conversion constraint from the default expression type to the parameter type, which guarantees that all of the generic parameter types are always inferred. + +```swift +let myBox = Box(dimensions: ...) // F is inferred as DefaultFlags + +myBox.ship() // F is inferred as DefaultShippingFlags +``` + +Note that it is important to establish association between the type of a default expression and a corresponding parameter type not just for inference sake, but to guarantee that there are not generic parameter type clashes with a result type (which is allowed to mention the same generic parameters): + +```swift +func compute(initialValues: T = [0, 1, 2, 3]) -> T { + // A complex computation that uses initial values +} + +let result: Array = compute() ✅ +// Ok both `initialValues` and result type are the same type - `Array` + +let result: Array = compute() ❌ +// This is an error because type of default expression is `Array` and result +// type is `Array` +``` + +## Source compatibility + +Proposed changes to default expression handling do not break source compatibility. + + +## Effect on ABI stability + +No ABI impact since this is an additive change to the type-checker. + + +## Effect on API resilience + +All of the resilience rules associated with adding and removing of default expressions are left unchanged, see https://github.com/apple/swift/blob/main/docs/LibraryEvolution.rst#id12 for more details. + + +## Alternatives considered + +[Default generic arguments](https://github.com/apple/swift/blob/main/docs/GenericsManifesto.md#default-generic-arguments) feature mentioned in the Generics Manifesto should not be confused with type inference rules proposed here. Having an ability to default generic arguments alone is not enough to provide a consistent way to use default expressions when generic parameters are involved. The type inference described in this proposal would still be necessary allow default expressions with concrete type to be used when the parameter references a type parameter, and to determine whether the default expression works with a default generic argument type, which means that default generic arguments feature could be considered an enhancement instead of an alternative approach. + +A number of similar approaches has been discussed on Swift Forums, one of them being [[Pre-pitch] Conditional default arguments - #4 by Douglas_Gregor - Dis...](https://forums.swift.org/t/pre-pitch-conditional-default-arguments/7122/4) which relies on overloading, constrained extensions, and/or custom attributes and therefore has all of the issues outlined in the Motivation section. Allowing type inference from default expressions in this regard is a much cleaner approach that works for all situations without having to introduce any new syntax or custom attributes. + + +## Future Directions + +This proposal limits use of inferable generic parameters to a single location in a parameter list because all default expressions are type-checked independently. It is possible to lift this restriction and type-check all of the default expressions together which means that if generic parameters is inferable from different default expressions its type is going to be a common type that fits all locations (action of obtaining such a type is called type-join). It’s not immediately clear whether lifting this restriction would always adhere to the principle of the least surprise for the users, so it would require a separate discussion if this proposal is accepted. + +The simplest example that illustrates the problem is `test(a: T = 42, b: T = 4.2)-> T` , this declaration creates a matrix of possible calls each of which could be typed differently: + +1. `test()` — T = Double because the only type that fits both `42` and `4.2` is `Double` +2. `test(a: 0.0)` — T = `Double` +3. `test(b: 0)` — T = `Int` +4. `let _: Int = test()` - fails because `T` cannot be `Int` and `Double` at the same time. diff --git a/proposals/0348-buildpartialblock.md b/proposals/0348-buildpartialblock.md new file mode 100644 index 0000000000..4a196e68c6 --- /dev/null +++ b/proposals/0348-buildpartialblock.md @@ -0,0 +1,306 @@ +# `buildPartialBlock` for result builders + +* Proposal: [SE-0348](0348-buildpartialblock.md) +* Author: [Richard Wei](https://github.com/rxwei) +* Implementation: [apple/swift#41576](https://github.com/apple/swift/pull/41576) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** + +## Overview + +We introduce a new result builder customization point that allows components of a block to be combined pairwise. + +```swift +@resultBuilder +enum Builder { + /// Builds a partial result component from the first component. + static func buildPartialBlock(first: Component) -> Component + + /// Builds a partial result component by combining an accumulated component + /// and a new component. + /// - Parameter accumulated: A component representing the accumulated result + /// thus far. + /// - Parameter next: A component representing the next component after the + /// accumulated ones in the block. + static func buildPartialBlock(accumulated: Component, next: Component) -> Component +} +``` + +When `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` are both provided, the [result builder transform](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md#the-result-builder-transform) will transform components in a block into a series of calls to `buildPartialBlock`, combining one subsequent line into the result at a time. + +```swift +// Original +{ + expr1 + expr2 + expr3 +} + +// Transformed +// Note: `buildFinalResult` and `buildExpression` are called only when they are defined, just like how they behave today. +{ + let e1 = Builder.buildExpression(expr1) + let e2 = Builder.buildExpression(expr2) + let e3 = Builder.buildExpression(expr3) + let v1 = Builder.buildPartialBlock(first: e1) + let v2 = Builder.buildPartialBlock(accumulated: v1, next: e2) + let v3 = Builder.buildPartialBlock(accumulated: v2, next: e3) + return Builder.buildFinalResult(v3) +} +``` + +The primary goal of this feature is to reduce the code bloat caused by overloading `buildBlock` for multiple arities, allowing libraries to define builder-based generic DSLs with joy and ease. + +## Motivation + +Among DSLs powered by result builders, it is a common pattern to combine values with generic types in a block to produce a new type that contains the generic parameters of the components. For example, [`ViewBuilder`](https://developer.apple.com/documentation/swiftui/viewbuilder) and [`SceneBuilder`](https://developer.apple.com/documentation/swiftui/scenebuilder) in SwiftUI use `buildBlock` to combine views and scenes without losing strong types. + +```swift +extension SceneBuilder { + static func buildBlock(Content) -> Content + static func buildBlock(_ c0: C0, _ c1: C1) -> some Scene where C0: Scene, C1: Scene + ... + static func buildBlock(_ c0: C0, _ c1: C1, _ c2: C2, _ c3: C3, _ c4: C4, _ c5: C5, _ c6: C6, _ c7: C7, _ c8: C8, _ c9: C9) -> some Scene where C0: Scene, C1: Scene, C2: Scene, C3: Scene, C4: Scene, C5: Scene, C6: Scene, C7: Scene, C8: Scene, C9: Scene +} +``` + +Due to the lack of variadic generics, `buildBlock` needs to be overloaded for any supported block arity. This unfortunately increases code size, causes significant code bloat in the implementation and documentation, and it is often painful to write and maintain the boiletplate. + +While this approach works for types like `ViewBuilder` and `SceneBuilder`, some builders need to define type combination rules that are far too complex to implement with overloads. One such example is [`RegexComponentBuilder`](https://github.com/apple/swift-experimental-string-processing/blob/85c7d906dd871364357156126278d9d427936ca4/Sources/_StringProcessing/RegexDSL/Builder.swift#L13) in [Declarative String Processing](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/DeclarativeStringProcessing.md). + +The regex builder DSL is designed to allow developers to easily compose regex patterns. [Strongly typed captures](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md#strongly-typed-regex-captures) are represented as part of the `Match` generic parameter in the `Regex` type, which has a builder-based initializer. + +```swift +struct Regex { + init(@RegexComponentBuilder _ builder: () -> Self) +} +``` + +> #### Recap: Regular expression capturing basics +> +> When a regular expression does not contain any capturing groups, its `Match` type is `Substring`, which represents the whole matched portion of the input. +> +> ```swift +> let noCaptures = #/a/# // => Regex +> ``` +> +> When a regular expression contains capturing groups, i.e. `(...)`, the `Match` type is extended as a tuple to also contain *capture types*. Capture types are tuple elements after the first element. +> +> ```swift +> // ________________________________ +> // .0 | .0 | +> // ____________________ _________ +> let yesCaptures = #/a(?:(b+)c(d+))+e(f)?/# // => Regex<(Substring, Substring, Substring, Substring?)> +> // ---- ---- --- --------- --------- ---------- +> // .1 | .2 | .3 | .1 | .2 | .3 | +> // | | | | | | +> // | | |_______________________________ | ______ | ________| +> // | | | | +> // | |______________________________________ | ______ | +> // | | +> // |_____________________________________________| +> // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +> // Capture types +> ``` + +Using the result builder syntax, the regular expression above becomes: + +```swift +let regex = Regex { + "a" // => Regex + OneOrMore { // { + Capture { OneOrMore("b") } // => Regex<(Substring, Substring)> + "c" // => Regex + Capture { OneOrMore("d") } // => Regex<(Substring, Substring)> + } // } => Regex<(Substring, Substring, Substring)> + "e" // => Regex + Optionally { Capture("f") } // => Regex<(Substring, Substring?)> +} // => Regex<(Substring, Substring, Substring, Substring?)> + // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + // Capture types + +let result = "abbcddbbcddef".firstMatch(of: regex) +// => MatchResult<(Substring, Substring, Substring, Substring?)> +``` + +`RegexComponentBuilder` concatenates the capture types of all components as a flat tuple, forming a new `Regex` whose `Match` type is `(Substring, CaptureType...)`. We can define the following `RegexComponentBuilder`: + +```swift +@resultBuilder +enum RegexComponentBuilder { + static func buildBlock() -> Regex + static func buildBlock(_: Regex) -> Regex + // Overloads for non-tuples: + static func buildBlock(_: Regex, _: Regex) -> Regex + static func buildBlock(_: Regex, _: Regex, _: Regex) -> Regex + ... + static func buildBlock(_: Regex, _: Regex, _: Regex, ..., _: Regex) -> Regex + // Overloads for tuples: + static func buildBlock(_: Regex<(W0, C0)>, _: Regex) -> Regex<(Substring, C0)> + static func buildBlock(_: Regex, _: Regex<(W1, C0)>) -> Regex<(Substring, C0)> + static func buildBlock(_: Regex<(W0, C0, C1)>, _: Regex) -> Regex<(Substring, C0, C1)> + static func buildBlock(_: Regex<(W0, C0)>, _: Regex<(W1, C1)>) -> Regex<(Substring, C0, C1)> + static func buildBlock(_: Regex, _: Regex<(W1, C0, C1)>) -> Regex<(Substring, C0, C1)> + ... + static func buildBlock( + _: Regex<(W0, C0, C1)>, _: Regex<(W1, C2)>, _: Regex<(W3, C3, C4, C5)>, _: Regex<(W4, C6)> + ) -> Regex<(Substring, C0, C1, C2, C3, C4, C5, C6)> + ... +} +``` + +Here we just need to overload for all tuple combinations for each arity... Oh my! That is an `O(arity!)` combinatorial explosion of `buildBlock` overloads; compiling these methods alone could take hours. + +## Proposed solution + +This proposal introduces a new block-building approach similar to building heterogeneous lists. Instead of calling a single method to build an entire block wholesale, this approach recursively builds a partial block by taking one new component at a time, and thus significantly reduces the number of overloads. + +We introduce a new customization point to result builders via two user-defined static methods: + +```swift +@resultBuilder +enum Builder { + static func buildPartialBlock(first: Component) -> Component + static func buildPartialBlock(accumulated: Component, next: Component) -> Component +} +``` + +When `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` are both defined, the result builder transform will turn components in a block into a series of calls to `buildPartialBlock`, combining components from top to bottom. + +With this approach, many result builder types with overloaded `buildBlock` can be simplified. For example, the `buildBlock` overloads in SwiftUI's `SceneBuilder` could be simplified as the following: + +```swift +extension SceneBuilder { + static func buildPartialBlock(first: some Scene) -> some Scene + static func buildPartialBlock(accumulated: some Scene, next: some Scene) -> some Scene +} +``` + +Similarly, the overloads of `buildBlock` in `RegexComponentBuilder` can be vastly reduced from `O(arity!)`, down to `O(arity^2)` overloads of `buildPartialBlock(accumulated:next:)`. For an arity of 10, 100 overloads are trivial compared to over 3 million ones. + +```swift +extension RegexComponentBuilder { + static func buildPartialBlock(first regex: Regex) -> Regex + static func buildPartialBlock(accumulated: Regex, next: Regex) -> Regex + static func buildPartialBlock(accumulated: Regex<(W0, C0)>, next: Regex) -> Regex<(Substring, C0)> + static func buildPartialBlock(accumulated: Regex, next: Regex<(W1, C0)>) -> Regex<(Substring, C0)> + static func buildPartialBlock(accumulated: Regex, next: Regex<(W1, C0, C1)>) -> Regex<(Substring, C0, C1)> + static func buildPartialBlock(accumulated: Regex<(W0, C0, C1)>, next: Regex) -> Regex<(Substring, C0, C1)> + static func buildPartialBlock(accumulated: Regex<(W0, C0)>, next: Regex<(W1, C1)>) -> Regex<(Substring, C0, C1)> + ... +} +``` + +### Early adoption feedback + +- In the [regex builder DSL](https://forums.swift.org/t/pitch-regex-builder-dsl/56007), `buildPartialBlock` reduced the number of required overloads from millions (`O(arity!)`) down to hundreds (`O(arity^2)`). +- In the pitch thread, [pointfreeco/swift-parsing reported](https://forums.swift.org/t/pitch-buildpartialblock-for-result-builders/55561/10) that `buildPartialBlock` enabled the deletion of 21K lines of generated code, increased arity support, and reduced compile times from 20 seconds to <2 seconds in debug mode. + +## Detailed design + +When a type is marked with `@resultBuilder`, the type was previously required to have at least one static `buildBlock` method. With this proposal, such a type is now required to have either at least one static `buildBlock` method, or both `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)`. + +In the result builder transform, the compiler will look for static members `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` in the builder type. If the following conditions are met: + +* Both methods `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` exist. +* The availability of the enclosing declaration is greater than or equal to the availability of `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)`. + +Then, a non-empty block will be transformed to the following: + +```swift +// Original +{ + expr1 + expr2 + expr3 +} + +// Transformed +// Note: `buildFinalResult` and `buildExpression` are called only when they are defined, just like how they behave today. +{ + let e1 = Builder.buildExpression(expr1) + let e2 = Builder.buildExpression(expr2) + let e3 = Builder.buildExpression(expr3) + let v1 = Builder.buildPartialBlock(first: e1) + let v2 = Builder.buildPartialBlock(accumulated: v1, next: e2) + let v3 = Builder.buildPartialBlock(accumulated: v2, next: e3) + return Builder.buildFinalResult(v3) +} +``` + +Otherwise, the result builder transform will transform the block to call `buildBlock` instead as proposed in [SE-0289](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md). + +## Source compatibility + +This proposal does not intend to introduce source-breaking changes. Although, if an existing result builder type happens to have static methods named `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)`, the result builder transform will be creating calls to those methods instead and may cause errors depending on `buildPartialBlock`'s type signature and implementation. Nevertheless, such cases should be extremely rare. + +## Effect on ABI stability + +This proposal does not contain ABI changes. + +## Effect on API resilience + +This proposal does not contain API changes. + +## Alternatives considered + +### Prefer viable `buildBlock` overloads to `buildPartialBlock` + +As proposed, the result builder transform will always prefer `buildPartialBlock` to `buildBlock` when they are both defined. One could argue that making a single call to a viable overload of `buildBlock` would be more efficient and more customizable. However, because the result builder transform currently operates before type inference has completed, it would increase the type checking complexity to decide to call `buildBlock` or `buildPairwiseBlock` based on argument types. None of the informal requirements (`buildBlock`, `buildOptional`, etc) of result builders depend on argument types when being transformed to by the result builder transform. + +### Use nullary `buildPartialBlock()` to form the initial value + +As proposed, the result builder transform calls unary `buildPartialBlock(first:)` on the first component in a block before calling `buildPartialBlock(accumulated:next:)` on the rest. While it is possible to require the user to define a nullary `buildPartialBlock()` method to form the initial result, this behavior may be suboptimal for result builders that do not intend to support empty blocks, e.g. SwiftUI's `SceneBuilder`. Plus, the proposed approach does allow the user to define an nullary `buildBlock()` to support building an empty block. + +### Rely on variadic generics + +It can be argued that variadic generics would resolve the motivations presented. However, to achieve the concatenating behavior needed for `RegexComponentBuilder`, we would need to be able to express nested type sequences, perform collection-like transformations on generic parameter packs such as dropping elements and splatting. + +```swift +extension RegexComponentBuilder { + static func buildBlock<(W, (C...))..., R...>(_ components: Regex<(W, C...)>) -> Regex<(Substring, (R.Match.dropFirst()...).splat())> +} +``` + +Such features would greatly complicate the type system. + +### Alternative names + +#### Overload `buildBlock` method name + +Because the proposed feature overlaps `buildBlock`, one could argue for reusing `buildBlock` as the method base name instead of `buildPartialBlock` and using argument labels to distinguish whether it is the pairwise version, e.g. `buildBlock(partiallyAccumulated:next:)` or `buildBlock(combining:into:)`. + +```swift +extension Builder { + static func buildBlock(_: Component) -> Component + static func buildBlock(partiallyAccumulated: Component, next: Component) -> Component +} +``` + +However, the phrase "build block" does not have a clear indication that the method is in fact building a partial block, and argument labels do not have the prominence to carry such indication. + +#### Use `buildBlock(_:)` instead of `buildPartialBlock(first:)` + +The unary base case method `buildPartialBlock(first:)` and `buildBlock(_:)` can be viewed as being functionally equivalent, so one could argue for reusing `buildBlock`. However, as mentioned in [Overload `buildBlock` method name](#overload-buildblock-method-name), the phrase "build block" lacks clarity. + +A more important reason is that `buildPartialBlock(first:)`, especially with its argument label `first:`, leaves space for a customization point where the developer can specify the direction of combination. As a future direction, we could allow `buildPartialBlock(last:)` to be defined instead of `buildPartialBlock(first:)`, and in this scenario the result builder transform will first call `buildPartialBlock(last:)` on the _last_ component and then call `buildPartialBlock(accumulated:next:)` on each preceeding component. + +#### Different argument labels + +The proposed argument labels `accumulated:` and `next:` took inspirations from some precedents in the standard library: +- "accumulated" is used as an argument name of [`Array.reduce(into:_:)`](https://developer.apple.com/documentation/swift/array/3126956-reduce). +- "next" is used as an argument name of [`Array.reduce(_:_:)`](https://developer.apple.com/documentation/swift/array/2298686-reduce) + +Meanwhile, there are a number of alternative argument labels considered in the place of `accumulated:` and `next:`. + +Possible replacements for `accumulated:`: +- `partialResult:` +- `existing:` +- `upper:` +- `_:` + +Possible replacements for `next:`: +- `new:` +- `_:` + +We believe that "accumulated" and "next" have the best overall clarity. diff --git a/proposals/0349-unaligned-loads-and-stores.md b/proposals/0349-unaligned-loads-and-stores.md new file mode 100644 index 0000000000..b542077621 --- /dev/null +++ b/proposals/0349-unaligned-loads-and-stores.md @@ -0,0 +1,277 @@ +# Unaligned Loads and Stores from Raw Memory + +* Proposal: [SE-0349](0349-unaligned-loads-and-stores.md) +* Authors: [Guillaume Lessard](https://github.com/glessard), [Andrew Trick](https://github.com/atrick) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#41033](https://github.com/apple/swift/pull/41033) +* Review: ([pitch](https://forums.swift.org/t/55036/)) ([review](https://forums.swift.org/t/se-0349-unaligned-loads-and-stores-from-raw-memory/56423)) ([acceptance](https://forums.swift.org/t/accepted-se-0349-unaligned-loads-and-stores-from-raw-memory/56748)) + +## Introduction + +Swift does not currently provide a clear way to load data from an arbitrary source of bytes, such as a binary file, in which data may be stored without respect for in-memory alignment. This proposal aims to rectify the situation, making workarounds unnecessary. + +## Motivation + +The method `UnsafeRawPointer.load(fromByteOffset offset: Int, as type: T.Type) -> T` requires the address at `self+offset` to be properly aligned to access an instance of type `T`. Attempts to use a combination of pointer and byte offset that is not aligned for `T` results in a runtime crash. Unfortunately, in general, data saved to files or network streams does not adhere to the same restrictions as in-memory layouts do, and tends to not be properly aligned. When copying data from such sources to memory, Swift users therefore frequently encounter alignment mismatches that require using a workaround. This is a longstanding issue reported in e.g. [SR-10273](https://bugs.swift.org/browse/SR-10273). + +For example, given an arbitrary data stream in which a 4-byte value is encoded between byte offsets 3 through 7: + +```swift +let data = Data([0x0, 0x0, 0x0, 0xff, 0xff, 0xff, 0xff, 0x0]) +``` + +In order to extract all the `0xff` bytes of this stream to an `UInt32`, we would like to be able to use `load(as:)`, as follows: + +```swift +let result = data.dropFirst(3).withUnsafeBytes { $0.load(as: UInt32.self) } +``` + +However, that will currently crash at runtime, because in this case `load` requires the base pointer to be correctly aligned for accessing `UInt32`. A workaround is required, such as the following: + +```swift +let result = data.dropFirst(3).withUnsafeBytes { buffer -> UInt32 in + var storage = UInt32.zero + withUnsafeMutableBytes(of: &storage) { + $0.copyBytes(from: buffer.prefix(MemoryLayout.size)) + } + return storage +} +``` + +The necessity of this workaround (or of others that produce the same outcome) is unsatisfactory for two reasons; firstly it is tremendously non-obvious. Secondly, it requires two copies instead of the expected single copy: the first to a correctly-aligned raw buffer, and then to the final, correctly-typed variable. We should be able to do this with a single copy. + +The kinds of types for which it is important to improve loads from arbitrary alignments are types whose values can be copied bit for bit, without reference counting operations. These types are commonly referred to as "POD" (plain old data) or "trivial" types. We propose to restrict the use of the unaligned loading operation to those types. + +## Proposed solution + +We propose to add an API `UnsafeRawPointer.loadUnaligned(fromByteOffset:as:)` to support unaligned loads from `UnsafeRawPointer`, `UnsafeRawBufferPointer` and their mutable counterparts. These will be explicitly restricted to POD types. Loading a non-POD type remains meaningful only when the source memory is another live object where the memory is, by construction, already correctly aligned. The original API (`load`) will continue to support this case. The new API (`loadUnaligned`) will assert that the return type is POD when run in debug mode. + +`UnsafeMutableRawPointer.storeBytes(of:toByteOffset:)` is documented to only be meaningful for POD types. However, at runtime it enforces storage to an offset correctly aligned to the source type. We propose to remove that alignment restriction and instead enforce the documented POD restriction. The API will otherwise be unchanged, though its documentation will be updated. Please see the ABI stability section for a discussion of binary compatibility with this approach. + +The `UnsafeRawBufferPointer` and `UnsafeMutableRawBufferPointer` types will receive matching changes. + +## Detailed design + +```swift +extension UnsafeRawPointer { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// This function only supports loading trivial types, + /// and will trap if this precondition is not met. + /// A trivial type does not contain any reference-counted property + /// within its in-memory representation. + /// The memory at this pointer plus `offset` must be laid out + /// identically to the in-memory representation of `T`. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance isn't associated + /// with the value in the range of memory referenced by this pointer. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T +} +``` + +```swift +extension UnsafeMutableRawPointer { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// This function only supports loading trivial types, + /// and will trap if this precondition is not met. + /// A trivial type does not contain any reference-counted property + /// within its in-memory representation. + /// The memory at this pointer plus `offset` must be laid out + /// identically to the in-memory representation of `T`. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance isn't associated + /// with the value in the range of memory referenced by this pointer. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T + + /// Stores the given value's bytes into raw memory at the specified offset. + /// + /// The type `T` to be stored must be a trivial type. The memory + /// must also be uninitialized, initialized to `T`, or initialized to + /// another trivial type that is layout compatible with `T`. + /// + /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is + /// initialized to the raw bytes of `value`. If the memory is bound to a + /// type `U` that is layout compatible with `T`, then it contains a value of + /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the + /// bound type of the memory. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// If you need to store a copy of a value of a type that isn't trivial into memory, + /// you cannot use the `storeBytes(of:toByteOffset:as:)` method. Instead, you must know + /// the type of value previously in memory and initialize or assign the + /// memory. For example, to replace a value stored in a raw pointer `p`, + /// where `U` is the current type and `T` is the new type, use a typed + /// pointer to access and deinitialize the current value before initializing + /// the memory with a new value. + /// + /// let typedPointer = p.bindMemory(to: U.self, capacity: 1) + /// typedPointer.deinitialize(count: 1) + /// p.initializeMemory(as: T.self, repeating: newValue, count: 1) + /// + /// - Parameters: + /// - value: The value to store as raw bytes. + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of `value`. + public func storeBytes(of value: T, toByteOffset offset: Int = 0, as type: T.Type) +} +``` + + + +`UnsafeRawBufferPointer` and `UnsafeMutableRawBufferPointer` receive a similar addition of a `loadUnaligned` function. It enables loading from an arbitrary offset with the buffer, subject to the usual index validation rules of `BufferPointer` types: indexes are checked when client code is compiled in debug mode, while indexes are unchecked when client code is compiled in release mode. + +```swift +extension Unsafe{Mutable}RawBufferPointer { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// This function only supports loading trivial types. + /// A trivial type does not contain any reference-counted property + /// within its in-memory stored representation. + /// The memory at `offset` bytes into the buffer must be laid out + /// identically to the in-memory representation of `T`. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// You can use this method to create new values from the buffer pointer's + /// underlying bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the next four bytes. + /// + /// let a = someBytes.load(as: Int32.self) + /// let b = someBytes.load(fromByteOffset: 4, as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the buffer + /// pointer's memory region---that is, `offset + MemoryLayout.size` must + /// be less than or equal to the buffer pointer's `count`. + /// + /// - Parameters: + /// - offset: The offset, in bytes, into the buffer pointer's memory at + /// which to begin reading data for the new instance. The buffer pointer + /// plus `offset` must be properly aligned for accessing an instance of + /// type `T`. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer's + /// memory. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T +} +``` + +Additionally, the semantics of `UnsafeMutableBufferPointer.storeBytes(of:toByteOffset)` will be changed in the same way as its counterpart `UnsafeMutablePointer.storeBytes(of:toByteOffset)`, no longer enforcing alignment at runtime. Again, the index validation behaviour is unchanged: indexes are checked when client code is compiled in debug mode, while indexes are unchecked when client code is compiled in release mode. + +```swift +extension UnsafeMutableRawBufferPointer { + /// Stores a value's bytes into the buffer pointer's raw memory at the + /// specified byte offset. + /// + /// The type `T` to be stored must be a trivial type. The memory must also be + /// uninitialized, initialized to `T`, or initialized to another trivial + /// type that is layout compatible with `T`. + /// + /// The memory written to must not extend beyond the buffer pointer's memory + /// region---that is, `offset + MemoryLayout.size` must be less than or + /// equal to the buffer pointer's `count`. + /// + /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is + /// initialized to the raw bytes of `value`. If the memory is bound to a + /// type `U` that is layout compatible with `T`, then it contains a value of + /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the + /// bound type of the memory. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// If you need to store a copy of a value of a type that isn't trivial into memory, + /// you cannot use the `storeBytes(of:toByteOffset:as:)` method. Instead, you must know + /// the type of value previously in memory and initialize or assign the memory. + /// + /// - Parameters: + /// - offset: The offset in bytes into the buffer pointer's memory to begin + /// reading data for the new instance. The buffer pointer plus `offset` + /// must be properly aligned for accessing an instance of type `T`. The + /// default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + public func storeBytes(of value: T, toByteOffset offset: Int = 0, as: T.Type) +} +``` + + + +## Source compatibility + +This proposal is source compatible. The proposed API modifications relax existing restrictions and keep the same signatures, therefore the changes are compatible. The API additions are source compatible by definition. + +## Effect on ABI stability + +Existing binaries that expect the old behaviour of `storeBytes` will not be affected by the relaxed behaviour proposed here, as we will ensure that the old symbol (with its existing semantics) will remain. + +New binaries that require the new behaviour will correctly backwards deploy by the use of the `@_alwaysEmitIntoClient` attribute. The new API will likewise use the `@_alwaysEmitIntoClient` attribute. + +## Effect on API resilience + +If the added API were removed in a future release, the change would be source-breaking but not ABI-breaking, because the proposed additions will always be inlined. + +## Alternatives considered + +#### Use a marker protocol to restrict unaligned loads to trivial types + +We could enforce the use of unaligned loads at compile time by declaring a new marker protocol for trivial types, and require conformance to this protocol for types loaded through a function that can load from unaligned offsets. While this may be the ideal outcome, we believe this option would take too long to be realized. The approach proposed here can be a stepping stone on the way there. + +#### Relax the alignment restriction on the existing `load` API + +Arguably, user expectations are that the `load` API supports unaligned loads, but since that is not the case with the existing API, source-compatibility considerations dictate that the behaviour of the existing API should not change. If its preconditions were relaxed, a developer would encounter runtime crashes when deploying to a server using Swift 5.5, having tested using a newer toolchain. + +For that reason, we chose to leave the existing API untouched. + +Other programming languages have chosen whether loading from bytes is aligned or unaligned by default depending on their focus. For example, Go's [binary](https://pkg.go.dev/encoding/binary@go1.18) package privileges decoding data from a stream, and accordingly its various `Read` functions perform unaligned loads. [binary](https://pkg.go.dev/encoding/binary@go1.18)'s package documentation acknowledges privileging simplicity over efficiency. On the other hand, Rust's [raw pointer](https://doc.rust-lang.org/core/primitive.pointer.html) primitive type includes both [`read`](https://doc.rust-lang.org/core/ptr/fn.read.html) and [`read_unaligned`](https://doc.rust-lang.org/core/ptr/fn.read_unaligned.html) functions, where the default (with the "good" name) is more strict and more efficient. We believe that Swift's goals align well with having the more performant function (aligned load) be the default one. + +#### Add a separate unaligned store API + +Adding a separate unaligned store API would avoid ABI stability concerns, but the old API would become redundant. The risk of removing the restriction on `storeBytes` is less than it is for `load`, as the restriction is implemented using `_debugPrecondition`, which is compiled away in release mode. + +#### Rename `storeBytes` to `storeUnaligned`, or call `loadUnaligned` `loadFromBytes` instead. + +The idea of making the "load" and the "store" operations have more symmetric names is compelling, however there is a fundamental asymmetry in the operation itself. When a `load` operation completes, a new value is created to be managed by the Swift runtime. On the other hand the `storeBytes` operation is completely transparent to the runtime: the destination is a container of bytes that is _not_ managed by the Swift runtime. For this reason, the "store" operation has the word "bytes" in its name. + +## Acknowledgments + +Thanks to the Swift Standard Library team for valuable feedback and discussion. diff --git a/proposals/0350-regex-type-overview.md b/proposals/0350-regex-type-overview.md new file mode 100644 index 0000000000..2eef59ffb0 --- /dev/null +++ b/proposals/0350-regex-type-overview.md @@ -0,0 +1,576 @@ +# Regex Type and Overview + +* Proposal: [SE-0350](0350-regex-type-overview.md) +* Authors: [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: https://github.com/apple/swift-experimental-string-processing + * Available in nightly toolchain snapshots with `import _StringProcessing` + +## Introduction + +Swift strings provide an obsessively Unicode-forward model of programming with strings. String processing with `Collection`'s algorithms is woefully inadequate for many day-to-day tasks compared to other popular programming and scripting languages. + +We propose addressing this basic shortcoming through an effort we are calling regex. What we propose is more powerful, extensible, and maintainable than what is traditionally thought of as regular expressions from other programming languages. This effort is presented as 6 interrelated proposals: + +1. `Regex` and `Regex.Match` types with support for typed captures, both static and dynamic. +2. A best-in-class treatment of traditional, familiar regular expression syntax for run-time construction of regex. +3. A literal for compile-time construction of a regex with statically-typed captures, enabling powerful source tools. +4. An expressive and composable result-builder DSL, with support for capturing strongly-typed values. +5. A modern treatment of Unicode semantics and string processing. +6. A slew of regex-powered string processing algorithms, along with library-extensible protocols enabling industrial-strength parsers to be used seamlessly as regex components. + +This proposal provides details on \#1, the `Regex` type and captures, and gives an overview of how each of the other proposals fit into regex in Swift. + +At the time of writing, these related proposals are in various states of being drafted, pitched, or proposed. For the current status, see [Pitch and Proposal Status][pitches]. + +

Obligatory differentiation from formal regular expressions + +Regular expressions originated in formal language theory as a way to answer yes-or-no whether a string is in a given [regular language](https://en.wikipedia.org/wiki/Regular_language). They are more powerful (and less composable) than [star-free languages](https://en.wikipedia.org/wiki/Star-free_language) and less powerful than [context-free languages](https://en.wikipedia.org/wiki/Context-free_language). Because they just answer a yes-or-no question, _how_ that answer is determined is irrelevant; i.e. their execution model is ambiguous. + +Regular expressions were brought into practical applications for text processing and compiler lexers. For searching within text, where the result (including captures) is a portion of the searched text, _how_ a match happened affects the result. Over time, more and more power was needed and "regular expressions" diverged from their formal roots. + +For compiler lexers, especially when implemented as a [discrete compilation phase](https://en.wikipedia.org/wiki/Lexical_analysis), regular expressions were often ingested by a [separate tool](https://en.wikipedia.org/wiki/Flex_(lexical_analyser_generator)) from the rest of the compiler. Understanding formal regular expressions can help clarify the separation of concerns between lexical analysis and parsing. Beyond that, they are less relevant for structuring modern parsers, which interweave error handling and recovery, debuggability, and fine-grained source location tracking across this traditional separation-of-tools. + +The closest formal analogue to what we are proposing are [Parsing Expression Grammars](https://en.wikipedia.org/wiki/Parsing_expression_grammar) ("PEGs"), which describe a recursive descent parser. Our alternation is ordered choice and we support possessive quantification, recursive subpattern calls, and lookahead. However, we are first and foremost providing a regexy presentation: quantification, by default, is non-possessive. + +
+ + +## Motivation + +Imagine processing a bank statement in order to extract transaction details for further scrutiny. Fields are separated by 2-or-more spaces: + +```swift +struct Transaction { + enum Kind: String { + case credit = "CREDIT" + case debit = "DEBIT" + } + + let kind: Kind + let date: Date + let accountName: String + let amount: Decimal +} + +let statement = """ + CREDIT 03/02/2022 Payroll $200.23 + CREDIT 03/03/2022 Sanctioned Individual A $2,000,000.00 + DEBIT 03/03/2022 Totally Legit Shell Corp $2,000,000.00 + DEBIT 03/05/2022 Beanie Babies Forever $57.33 + """ +``` + +One option is to `split()` around whitespace, hard-coding field offsets for everything except the account name, and `join()`ing the account name fields together to restore their spaces. This carries a lot of downsides such as hard-coded offsets, many unnecessary allocations, and this pattern would not easily expand to supporting other representations. + +Another option is to process an entry in a single pass from left-to-right, but this can get unwieldy: + +```swift +// Parse dates using a simple (localized) numeric strategy +let dateParser = Date.FormatStyle(date: .numeric).parseStrategy + +// Parse currencies as US dollars +let decimalParser = Decimal.FormatStyle.Currency(code: "USD") + +func processEntry(_ s: String) -> Transaction? { + var slice = s[...] + guard let kindEndIdx = slice.firstIndex(of: " "), + let kind = Transaction.Kind(slice[.. Transaction? { + let range = NSRange(line.startIndex..` describes a string processing algorithm. Captures surface the portions of the input that were matched by subpatterns. By convention, capture `0` is the entire match. + +### Creating Regex + +Regexes can be created at run time from a string containing familiar regex syntax. If no output type signature is specified, the regex has type `Regex`, in which captures are existentials and the number of captures is queryable at run time. Alternatively, providing an output type signature produces strongly-typed outputs, where captures are concrete types embedded in a tuple, providing safety and enabling source tools such as code completion. + +```swift +let pattern = #"(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)"# +let regex = try! Regex(pattern) +// regex: Regex + +let regex: Regex<(Substring, Substring, Substring, Substring, Substring)> = + try! Regex(pattern) +``` + +*Note*: The syntax accepted and further details on run-time compilation, including `AnyRegexOutput` and extended syntaxes, are discussed in [Run-time Regex Construction][pitches]. + +Type mismatches and invalid regex syntax are diagnosed at construction time by `throw`ing errors. + +When the pattern is known at compile time, regexes can be created from a literal containing the same regex syntax, allowing the compiler to infer the output type. Regex literals enable source tools, e.g. syntax highlighting and actions to refactor into a result builder equivalent. + +```swift +let regex = /(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)/ +// regex: Regex<(Substring, Substring, Substring, Substring, Substring)> +``` + +*Note*: Regex literals, most notably the choice of delimiter, are discussed in [Regex Literals][pitches]. + +This same regex can be created from a result builder, a refactoring-friendly representation: + +```swift +let fieldSeparator = Regex { + CharacterClass.whitespace + OneOrMore(.whitespace) +} + +let regex = Regex { + Capture(OneOrMore(.word)) + fieldSeparator + + Capture(OneOrMore(.whitespace.inverted)) + fieldSeparator + + Capture { + OneOrMore { + NegativeLookahead(fieldSeparator) + CharacterClass.any + } + } + fieldSeparator + + Capture { OneOrMore(.any) } +} +// regex: Regex<(Substring, Substring, Substring, Substring, Substring)> +``` + +*Note*: The result builder API is discussed in [Regex Builders][pitches]. Character classes and other Unicode concerns are discussed in [Unicode for String Processing][pitches]. + +`Regex` itself is a valid component for use inside a result builder, meaning that embedded literals can be used for concision. + +### Using Regex + +A `Regex.Match` contains the result of a match, surfacing captures by number, name, and reference. + +```swift +func processEntry(_ line: String) -> Transaction? { + // Multiline literal implies `(?x)`, i.e. non-semantic whitespace with line-ending comments + let regex = #/ + (? \w+) \s\s+ + (? \S+) \s\s+ + (? (?: (?!\s\s) . )+) \s\s+ + (? .*) + /# + // regex: Regex<( + // Substring, + // kind: Substring, + // date: Substring, + // account: Substring, + // amount: Substring + // )> + + guard let match = line.wholeMatch(of: regex), + let kind = Transaction.Kind(match.kind), + let date = try? Date(String(match.date), strategy: dateParser), + let amount = try? Decimal(String(match.amount), format: decimalParser) + else { + return nil + } + + return Transaction( + kind: kind, date: date, account: String(match.account), amount: amount) +} +``` + +*Note*: Details on typed captures using tuple labels are covered in [Regex Literals][pitches]. + +The result builder allows for inline failable value construction, which participates in the overall string processing algorithm: returning `nil` signals a local failure and the engine backtracks to try an alternative. This not only relieves the use site from post-processing, it enables new kinds of processing algorithms, allows for search-space pruning, and enhances debuggability. + +Swift regexes describe an unambiguous algorithm, where choice is ordered and effects can be reliably observed. For example, a `print()` statement inside the `TryCapture`'s transform function will run whenever the overall algorithm naturally dictates an attempt should be made. Optimizations can only elide such calls if they can prove it is behavior-preserving (e.g. "pure"). + +`CustomMatchingRegexComponent`, discussed in [String Processing Algorithms][pitches], allows industrial-strength parsers to be used a regex components. This allows us to drop the overly-permissive pre-parsing step: + +```swift +func processEntry(_ line: String) -> Transaction? { + let fieldSeparator = Regex { + CharacterClass.whitespace + OneOrMore(.whitespace) + } + + // Declare strongly-typed references to store captured values into + let kind = Reference() + let date = Reference() + let account = Reference() + let amount = Reference() + + let regex = Regex { + TryCapture(as: kind) { + OneOrMore(.word) + } transform: { + Transaction.Kind($0) + } + fieldSeparator + + TryCapture(as: date) { dateParser } + fieldSeparator + + Capture(as: account) { + OneOrMore { + NegativeLookahead(fieldSeparator) + CharacterClass.any + } + } + fieldSeparator + + TryCapture(as: amount) { decimalParser } + } + // regex: Regex<(Substring, Transaction.Kind, Date, Substring, Decimal)> + + guard let match = line.wholeMatch(of: regex) else { return nil } + + return Transaction( + kind: match[kind], + date: match[date], + account: String(match[account]), + amount: match[amount]) +} +``` + +*Note*: Details on how references work is discussed in [Regex Builders][pitches]. `Regex.Match` supports referring to _all_ captures by position (`match.1`, etc.) whether named or referenced or neither. Due to compiler limitations, result builders do not support forming labeled tuples for named captures. + + +### Regex-powered algorithms + +Regexes can be used right out of the box with a variety of powerful and convenient algorithms, including trimming, splitting, and finding/replacing all matches within a string. + +These algorithms are discussed in [String Processing Algorithms][pitches]. + + +### Unicode handling + +A regex describes an algorithm to be ran over some model of string, and Swift's `String` has a rather unique Unicode-forward model. `Character` is an [extended grapheme cluster](https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) and equality is determined under [canonical equivalence](https://www.unicode.org/reports/tr15/#Canon_Compat_Equivalence). + +Calling `dropFirst()` will not drop a leading byte or `Unicode.Scalar`, but rather a full `Character`. Similarly, a `.` in a regex will match any extended grapheme cluster. A regex will match canonical equivalents by default, strengthening the connection between regex and the equivalent `String` operations. + +Additionally, word boundaries (`\b`) follow [UTS\#29 Word Boundaries](https://www.unicode.org/reports/tr29/#Word_Boundaries). Contractions ("don't") are correctly detected and script changes are separated, without incurring significant binary size costs associated with language dictionaries. + +Regex targets [UTS\#18 Level 2](https://www.unicode.org/reports/tr18/#Extended_Unicode_Support) by default, but provides options to switch to scalar-level processing as well as compatibility character classes. Detailed rules on how we infer necessary grapheme cluster breaks inside regexes, as well as options and other concerns, are discussed in [Unicode for String Processing][pitches]. + + +## Detailed design + +```swift +/// A regex represents a string processing algorithm. +/// +/// let regex = try Regex("a(.*)b") +/// let match = "cbaxb".firstMatch(of: regex) +/// print(match.0) // "axb" +/// print(match.1) // "x" +/// +public struct Regex { + /// Match a string in its entirety. + /// + /// Returns `nil` if no match and throws on abort + public func wholeMatch(in s: String) throws -> Regex.Match? + + /// Match part of the string, starting at the beginning. + /// + /// Returns `nil` if no match and throws on abort + public func prefixMatch(in s: String) throws -> Regex.Match? + + /// Find the first match in a string + /// + /// Returns `nil` if no match is found and throws on abort + public func firstMatch(in s: String) throws -> Regex.Match? + + /// Match a substring in its entirety. + /// + /// Returns `nil` if no match and throws on abort + public func wholeMatch(in s: Substring) throws -> Regex.Match? + + /// Match part of the string, starting at the beginning. + /// + /// Returns `nil` if no match and throws on abort + public func prefixMatch(in s: Substring) throws -> Regex.Match? + + /// Find the first match in a substring + /// + /// Returns `nil` if no match is found and throws on abort + public func firstMatch(in s: Substring) throws -> Regex.Match? + + /// The result of matching a regex against a string. + /// + /// A `Match` forwards API to the `Output` generic parameter, + /// providing direct access to captures. + @dynamicMemberLookup + public struct Match { + /// The range of the overall match + public var range: Range { get } + + /// The produced output from the match operation + public var output: Output { get } + + /// Lookup a capture by name or number + public subscript(dynamicMember keyPath: KeyPath) -> T { get } + + /// Lookup a capture by number + @_disfavoredOverload + public subscript( + dynamicMember keyPath: KeyPath<(Output, _doNotUse: ()), Output> + ) -> Output { get } + // Note: this allows `.0` when `Match` is not a tuple. + + } +} +``` + +*Note*: The below are covered by other proposals, but listed here to help round out intuition. + +```swift + +// Result builder interfaces +extension Regex: RegexComponent { + public var regex: Regex { self } + + /// Result builder interface + public init( + @RegexComponentBuilder _ content: () -> Content + ) where Content.Output == Output + +} +extension Regex.Match { + /// Lookup a capture by reference + public subscript(_ reference: Reference) -> Capture +} + +// Run-time compilation interfaces +extension Regex { + /// Parse and compile `pattern`, resulting in a strongly-typed capture list. + public init(_ pattern: String, as: Output.Type = Output.self) throws +} +extension Regex where Output == AnyRegexOutput { + /// Parse and compile `pattern`, resulting in an existentially-typed capture list. + public init(_ pattern: String) throws +} +``` + +### Cancellation + +Regex is somewhat different from existing standard library operations in that regex processing can be a long-running task. +For this reason regex algorithms may check if the parent task has been cancelled and end execution. + +### On severability and related proposals + +The proposal split presented is meant to aid focused discussion, while acknowledging that each is interconnected. The boundaries between them are not completely cut-and-dry and could be refined as they enter proposal phase. + +Accepting this proposal in no way implies that all related proposals must be accepted. They are severable and each should stand on their own merit. + +## Source compatibility + +Everything in this proposal is additive. Regex delimiters may have their own source compatibility impact, which is discussed in that proposal. + +## Effect on ABI stability + +Everything in this proposal is additive. Run-time strings containing regex syntax are represented in the ABI as strings. For this initial release, literals are strings in the ABI as well (they get re-parsed at run time), which avoids baking an intermediate representation into Swift's ABI as we await better static compilation support (see future work). + +## Effect on API resilience + +N/A + +## Alternatives considered + + +### Regular expressions are a blight upon computing! + +"I had one problem so I wrote a regular expression, now I have two problems!" + +Regular expressions have a deservedly mixed reputation, owing to their historical baggage and treatment as a completely separate tool or subsystem. Despite this, they still occupy an important place in string processing. We are proposing the "regexiest regex", allowing them to shine at what they're good at and providing mitigations and off-ramps for their downsides. + +* "Regular expressions are bad because you should use a real parser" + - In other systems, you're either in or you're out, leading to a gravitational pull to stay in when... you should get out + - Our remedy is interoperability with real parsers via `CustomMatchingRegexComponent` + - Literals with refactoring actions provide an incremental off-ramp from regex syntax to result builders and real parsers +* "Regular expressions are bad because ugly unmaintainable syntax" + - We propose literals with source tools support, allowing for better syntax highlighting and analysis + - We propose result builders and refactoring actions from literals into result builders +* "Regular expressions are bad because Unicode" + - We propose a modern Unicode take on regexes + - We treat regexes as algorithms to be ran over some model of String, like's Swift's default Character-based view. +* "Regular expressions are bad because they're not powerful enough" + - Engine is general-purpose enough to support recursive descent parsers with captures, back-references, and lookahead + - We're proposing a regexy presentation on top of more powerful functionality +* "Regular expressions are bad because they're too powerful" + - We provide possessive quantifications, atomic groups, etc., all the normal ways to prune backtracking + - We provide clear semantics of how alternation works as ordered-choice, allowing for understandable execution + - Pathological behavior is ultimately a run-time concern, better handled by engine limiters (future work) + - Optimization is better done as a compiler problem, e.g. static compilation to DFAs (future work) + - Formal treatment of power is better done by other presentations, like PEGs and linear automata (future work) + + + +### Alternative names + +The generic parameter to `Regex` is `Output` and the erased version is `AnyRegexOutput`. This is... fairly generic sounding. + +An alternative could be `Captures`, doubling down on the idea that the entire match is implicitly capture `0`, but that can make describing and understanding how captures combine in the result builder harder to reason through (i.e. a formal distinction between explicit and implicit captures). + +An earlier prototype used the name `Match` for the generic parameter, but that quickly got confusing with all the match methods and was confusing with the result of a match operation (which produces the output, but isn't itself the generic parameter). We think `Match` works better as the result of a match operation. + + +### What's with all the `String(...)` initializer calls at use sites? + +We're working on how to eliminate these, likely by having API to access ranges, slices, or copies of the captured text. + +We're also looking for more community discussion on what the default type system and API presentation should be. As pitched, `Substring` emphasizes that we're referring to slices of the original input, with strong sharing connotations. + +The actual `Match` struct just stores ranges: the `Substrings` are lazily created on demand. This avoids unnecessary ARC traffic and memory usage. + + +### `Regex` instead of `Regex` + +The generic parameter `Output` is proposed to contain both the whole match (the `.0` element if `Output` is a tuple) and captures. One alternative we have considered is separating `Output` into the entire match and the captures, i.e. `Regex`, and using `Void` for for `Captures` when there are no captures. + +The biggest issue with this alternative design is that the numbering of `Captures` elements misaligns with the numbering of captures in textual regexes, where backreference `\0` refers to the entire match and captures start at `\1`. This design would sacrifice familarity and have the pitfall of introducing off-by-one errors. + +### Encoding `Regex`es into the type system + +During the initial review period the following comment was made: + +> I think the goal should be that, at least for regex literals (and hopefully for the DSL to some extent), one day we might not even need a bytecode or interpreter. I think the ideal case is if each literal was its own function or type that gets generated and optimised as if you wrote it in Swift. + +This is an approach that has been tried a few times in a few different languages (including by a few members of the Swift Standard Library and Core teams), and while it can produce attractive microbenchmarks, it has almost always proved to be a bad idea at the macro scale. In particular, even if we set aside witness tables and other associated swift generics overhead, optimizing a fixed pipeline for each pattern you want to match causes significant codesize expansion when there are multiple patterns in use, as compared to a more flexible byte code interpreter. A bytecode interpreter makes better use of instruction caches and memory, and can also benefit from micro architectural resources that are shared across different patterns. There is a tradeoff w.r.t. branch prediction resources, where separately compiled patterns may have more decisive branch history data, but a shared bytecode engine has much more data to use; this tradeoff tends to fall on the side of a bytecode engine, but it does not always do so. + +It should also be noted that nothing prevents AOT or JIT compiling of the bytecode if we believe it will be advantageous, but compiling or interpreting arbitrary Swift code at runtime is rather more unattractive, since both the type system and language are undecidable. Even absent this rationale, we would probably not encode regex programs directly into the type system simply because it is unnecessarily complex. + +### Future work: static optimization and compilation + +Swift's support for static compilation is still developing, and future work here is leveraging that to compile regex when profitable. Many regex describe simple [DFAs](https://en.wikipedia.org/wiki/Deterministic_finite_automaton) and can be statically compiled into very efficient programs. Full static compilation needs to be balanced with code size concerns, as a matching-specific bytecode is typically far smaller than a corresponding program (especially since the bytecode interpreter is shared). + +Regex are compiled into an intermediary representation and fairly simple analysis and optimizations are high-value. This compilation currently happens at run time (as such the IR is not ABI), but more of this could happen at compile time to save load/compilation time of the regex itself. Ideally, this representation would be shared along the fully-static compilation path and can be encoded in the ABI as a compact bytecode. + + +### Future work: parser combinators + +What we propose here is an incremental step towards better parsing support in Swift using parser-combinator style libraries. The underlying execution engine supports recursive function calls and mechanisms for library extensibility. `CustomMatchingRegexComponent`'s protocol requirement is effectively a [monadic parser](https://homepages.inf.ed.ac.uk/wadler/papers/marktoberdorf/baastad.pdf), meaning `Regex` provides a regex-flavored combinator-like system. + +An issues with traditional parser combinator libraries are the compilation barriers between call-site and definition, resulting in excessive and overly-cautious backtracking traffic. These can be eliminated through better [compilation techniques](https://core.ac.uk/download/pdf/148008325.pdf). As mentioned above, Swift's support for custom static compilation is still under development. + +Future work is a parser combinator system which leverages tiered static compilation and presents a parser-flavored approach, such as limited backtracking by default and more heavily interwoven recursive calls. + + +### Future work: `Regex`-backed enums + +Regexes are often used for tokenization and tokens can be represented with Swift enums. Future language integration could include `Regex` backing somewhat analogous to `RawRepresentable` enums. A Regex-backed enum could conform to `RegexComponent` producing itself upon a match by forming an ordered choice of its cases. + + + +[pitches]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md diff --git a/proposals/0351-regex-builder.md b/proposals/0351-regex-builder.md new file mode 100644 index 0000000000..41a39eae1d --- /dev/null +++ b/proposals/0351-regex-builder.md @@ -0,0 +1,1936 @@ +# Regex builder DSL + +* Proposal: [SE-0351](0351-regex-builder.md) +* Authors: [Richard Wei](https://github.com/rxwei), [Michael Ilseman](https://github.com/milseman), [Nate Cook](https://github.com/natecook1000), [Alejandro Alonso](https://github.com/azoy) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Implementation: [apple/swift-experimental-string-processing](https://github.com/apple/swift-experimental-string-processing/tree/main/Sources/RegexBuilder) + * Available in nightly toolchain snapshots with `import _StringProcessing` +* Status: **Implemented (Swift 5.7)** +* Review: ([pitch](https://forums.swift.org/t/pitch-regex-builder-dsl/56007)) + ([first review](https://forums.swift.org/t/se-0351-regex-builder-dsl/56531)) + ([revision](https://forums.swift.org/t/returned-for-revision-se-0351-regex-builder-dsl/57224)) + ([second review](https://forums.swift.org/t/se-0351-second-review-regex-builder-dsl/58721)) + ([acceptance](https://forums.swift.org/t/accepted-se-0351-regex-builder-dsl/58972)) + +**Table of Contents** +- [Regex builder DSL](#regex-builder-dsl) + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [`RegexComponent` protocol](#regexcomponent-protocol) + - [Concatenation](#concatenation) + - [Capture](#capture) + - [Mapping Output](#mapping-output) + - [Reference](#reference) + - [Alternation](#alternation) + - [Repetition](#repetition) + - [Repetition behavior](#repetition-behavior) + - [Anchors and Lookaheads](#anchors-and-lookaheads) + - [Subpattern](#subpattern) + - [Scoping](#scoping) + - [Composability](#composability) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Future directions](#future-directions) + - [Conversion to textual regex](#conversion-to-textual-regex) + - [Recursive subpatterns](#recursive-subpatterns) + - [Alternatives considered](#alternatives-considered) + - [Operators for quantification and alternation](#operators-for-quantification-and-alternation) + - [Postfix `capture` and `tryCapture` methods](#postfix-capture-and-trycapture-methods) + - [Unify quantifiers under `Repeat`](#unify-quantifiers-under-repeat) + - [Free functions instead of types](#free-functions-instead-of-types) + - [Support `buildOptional` and `buildEither`](#support-buildoptional-and-buildeither) + - [Flatten optionals](#flatten-optionals) + - [Structured rather than flat captures](#structured-rather-than-flat-captures) + - [Unify `Capture` with `TryCapture`](#unify-capture-with-trycapture) + +## Introduction + +[Declarative string processing] aims to offer powerful pattern matching capabilities with expressivity, clarity, type safety, and ease of use. To achieve this, we propose to introduce a result-builder-based DSL, **regex builder**, for creating and composing regular expressions (**regex**es). + +Regex builder is part of the Swift Standard Library but resides in a standalone module named `RegexBuilder`. By importing `RegexBuilder`, you get all necessary API for building a regex. + +```swift +import RegexBuilder + +let emailPattern = Regex { + let word = OneOrMore(.word) + Capture { + ZeroOrMore { + word + "." + } + word + } + "@" + Capture { + word + OneOrMore { + "." + word + } + } +} // => Regex<(Substring, Substring, Substring)> + +let email = "My email is my.name@mail.swift.org." +if let match = try emailPattern.firstMatch(in: email) { + let (wholeMatch, name, domain) = match.output + // wholeMatch: "my.name@mail.swift.org" + // name: "my.name" + // domain: "mail.swift.org" +} +``` + +This proposal introduces all core API for creating and composing regexes that echos the textual [regex syntax] and [strongly typed regex captures], but does not formally specify the matching semantics or define character classes. + +## Motivation + +Regex is a fundamental and powerful tool for textual pattern matching. It is a domain-specific language often expressed as text. For example, given the following bank statement: + +``` +CREDIT 04062020 PayPal transfer $4.99 +CREDIT 04032020 Payroll $69.73 +DEBIT 04022020 ACH transfer $38.25 +DEBIT 03242020 IRS tax payment $52249.98 +``` + +One can write the follow textual regex to match each line: + +``` +(CREDIT|DEBIT)\s+(\d{2}\d{2}\d{4})\s+([\w\s]+\w)\s+(\$\d+\.\d{2}) +``` + +While a regex like this is very compact and expressive, it is very difficult read, write and use: + +1. Syntactic special characters, e.g. `\`, `(`, `[`, `{`, are too dense to be readable. +2. It contains a hierarchy of subpatterns fit into a single line of text. +3. No code completion when typing syntactic components. +4. Capturing groups produce raw data (i.e. a range or a substring) and can only be converted to other data structures after matching. +5. While comments `(?#...)` can be added inline, it only complicates readability. + +## Proposed solution + +We introduce regex builder, a result-builder-based API for creating and composing regexes. This API resides in a new module named `RegexBuilder` that is to be shipped as part of the Swift toolchain. + +With regex builder, the regex for matching a bank statement can be written as the following: + +```swift +import RegexBuilder + +enum TransactionKind: String { + case credit = "CREDIT" + case debit = "DEBIT" +} + +struct Date { + var month, day, year: Int + init?(mmddyyyy: String) { ... } +} + +struct Amount { + var valueTimes100: Int + init?(twoDecimalPlaces text: Substring) { ... } +} + +let statementPattern = Regex { + // Parse the transaction kind. + TryCapture { + ChoiceOf { + "CREDIT" + "DEBIT" + } + } transform: { + TransactionKind(rawValue: String($0)) + } + OneOrMore(.whitespace) + // Parse the date, e.g. "01012021". + TryCapture { + Repeat(.digit, count: 2) + Repeat(.digit, count: 2) + Repeat(.digit, count: 4) + } transform: { Date(mmddyyyy: $0) } + OneOrMore(.whitespace) + // Parse the transaction description, e.g. "ACH transfer". + Capture { + OneOrMore(CharacterClass(.word, .whitespace)) + CharacterClass.word + } transform: { String($0) } + OneOrMore(.whitespace) + "$" + // Parse the amount, e.g. `$100.00`. + TryCapture { + OneOrMore(.digit) + "." + Repeat(.digit, count: 2) + } transform: { Amount(twoDecimalPlaces: $0) } +} // => Regex<(Substring, TransactionKind, Date, String, Amount)> + + +let statement = """ + CREDIT 04062020 PayPal transfer $4.99 + CREDIT 04032020 Payroll $69.73 + DEBIT 04022020 ACH transfer $38.25 + DEBIT 03242020 IRS tax payment $52249.98 + """ +for match in statement.matches(of: statementPattern) { + let (line, kind, date, description, amount) = match.output + ... +} +``` + +Regex builder addresses all of textual regexes' shortcomings presented in the [Motivation](#motivation) section: +1. Capture groups and quantifiers are expressed as API calls that are easy to read. +2. Scoping and indentations clearly distinguish subpatterns in the hierarchy. +3. Code completion is available when the developer types an API call. +4. Capturing groups can be transformed into structured data at the regex declaration site. +5. Normal code comments can be written within a regex declaration to further improve readability. + +## Detailed design + +### `RegexComponent` protocol + +One of the goals of the regex builder DSL is allowing the developers to easily compose regexes from common currency types and literals, or even define custom patterns to use for matching. We introduce `RegexComponent` in the implicitly-imported `Swift` module, a protocol that unifies all types that can represent a component of a regex. Since regexes are composable, the `Regex` type itself conforms to `RegexComponent`. + +```swift +public protocol RegexComponent { + associatedtype RegexOutput + var regex: Regex { get } +} + +extension Regex: RegexComponent { + public typealias RegexOutput = Output + public var regex: Regex { self } +} +``` + +Note: +- `RegexComponent` and `Regex`'s conformance to `RegexComponent` are available without importing `RegexBuilder`. All other types and conformances introduced in this proposal are in the `RegexBuilder` module. +- The associated type `RegexOutput` intentionally has a `Regex` prefix. `Output` would cause confusion in standard library conforming types such as `String`, i.e. `String.Output`. + +By conforming standard library types to `RegexComponent`, we allow them to be used inside the regex builder DSL as a match target. These conformances are available in the `RegexBuilder` module. + +```swift +// A string represents a regex that matches the string. +extension String: RegexComponent { + public var regex: Regex { get } +} + +// A substring represents a regex that matches the substring. +extension Substring: RegexComponent { + public var regex: Regex { get } +} + +// A character represents a regex that matches the character. +extension Character: RegexComponent { + public var regex: Regex { get } +} + +// A unicode scalar represents a regex that matches the scalar. +extension UnicodeScalar: RegexComponent { + public var regex: Regex { get } +} + +// To be introduced in a future pitch. +extension CharacterClass: RegexComponent { + public var regex: Regex { get } +} +``` + +All of the regex builder DSL in the rest of this pitch will accept generic components that conform to `RegexComponent`. + +### Concatenation + +A regex can be viewed as a concatenation of smaller regexes. In the regex builder DSL, `RegexComponentBuilder` is the basic facility to allow developers to compose regexes by concatenation. + +```swift +@resultBuilder +public enum RegexComponentBuilder { ... } +``` + +A closure marked with `@RegexComponentBuilder` will be transformed to produce a `Regex` by concatenating all of its components, where the result type's `Output` type will be a `Substring` followed by concatenated captures (tuple when plural). + +> #### Recap: Regex capturing basics +> +> `Regex` is a generic type with generic parameter `Output`. +> +> ```swift +> struct Regex { ... } +> ``` +> +> When a regex does not contain any capturing groups, its `Output` type is `Substring`, which represents the whole matched portion of the input. +> +> ```swift +> let noCaptures = #/a/# // => Regex +> ``` +> +> When a regex contains capturing groups, i.e. `(...)`, the `Output` type is extended as a tuple to also contain *capture types*. Capture types are tuple elements after the first element. +> +> ```swift +> // ________________________________ +> // .0 | .0 | +> // ____________________ _________ +> let yesCaptures = #/a(?:(b+)c(d+))+e(f)?/# // => Regex<(Substring, Substring, Substring, Substring?)> +> // ---- ---- --- --------- --------- ---------- +> // .1 | .2 | .3 | .1 | .2 | .3 | +> // | | | | | | +> // | | |_______________________________ | ______ | ________| +> // | | | | +> // | |______________________________________ | ______ | +> // | | +> // |_____________________________________________| +> // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +> // Capture types +> ``` + +We introduce a new initializer `Regex.init(_:)` which accepts a `@RegexComponentBuilder` closure. This initializer is the entry point for creating a regex using the regex builder DSL. + +```swift +extension Regex { + public init( + @RegexComponentBuilder _ content: () -> R + ) where R.RegexOutput == Output +} +``` + +Example: + +```swift +Regex { + regex0 // Regex + regex1 // Regex<(Substring, Int)> + regex2 // Regex<(Substring, Float)> + regex3 // Regex<(Substring, Substring)> +} // Regex<(Substring, Int, Float, Substring)> +``` + +This above regex will be transformed to: + +```swift +Regex { + let e0 = RegexComponentBuilder.buildExpression(regex0) // Regex + let e1 = RegexComponentBuilder.buildExpression(regex1) // Regex<(Substring, Int)> + let e2 = RegexComponentBuilder.buildExpression(regex2) // Regex<(Substring, Float)> + let e3 = RegexComponentBuilder.buildExpression(regex3) // Regex<(Substring, Substring)> + let r0 = RegexComponentBuilder.buildPartialBlock(first: e0) + let r1 = RegexComponentBuilder.buildPartialBlock(accumulated: r0, next: e1) + let r2 = RegexComponentBuilder.buildPartialBlock(accumulated: r1, next: e2) + let r3 = RegexComponentBuilder.buildPartialBlock(accumulated: r2, next: e3) + return r3 +} // Regex<(Substring, Int, Float, Substring)> +``` + +The following example creates a regex by concatenating subpatterns. + +```swift +let regex = Regex { + "regex builder " + "is " + "so easy" +} +let match = try regex.prefixMatch(in: "regex builder is so easy!") +match?.0 // => "regex builder is so easy" +``` + +
+API definition + +Basic methods in `RegexComponentBuilder`, e.g. `buildBlock()`, provides support for creating the most fundamental blocks. The `buildExpression` method wraps a user-provided component in a `RegexComponentBuilder.Component` structure, before passing the component to other builder methods. This is used for saving the source location of the component so that runtime errors can be reported with an accurate location. + +```swift +@resultBuilder +public enum RegexComponentBuilder { + /// Returns an empty regex. + public static func buildBlock() -> Regex + + /// A builder component that stores a regex component and its source location + /// for debugging purposes. + public struct Component { + public var value: Value + public var file: String + public var function: String + public var line: Int + public var column: Int + } + + /// Returns a component by wrapping the component regex in `Component` and + /// recording its source location. + public static func buildExpression( + _ regex: R, + file: String = #file, + function: String = #function, + line: Int = #line, + column: Int = #column + ) -> Component +} +``` + +`RegexComponentBuilder` utilizes `buildPartialBlock` to be able to concatenate all components' capture types to a single result tuple. `buildPartialBlock(first:)` provides support for creating a regex from a single component, and `buildPartialBlock(accumulated:next:)` support for creating a regex from multiple results. + +Before Swift supports variadic generics, `buildPartialBlock(accumulated:next:)` must be overloaded to support concatenating regexes of supported capture quantities (arities). It is overloaded up to `arity^2` times to account for all possible pairs of regexes that make up 10 captures. + +In the initial version of the DSL, we plan to support regexes with up to 10 captures, as 10 captures are sufficient for most use cases. These overloads can be superseded by variadic versions of `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` in a future release. + +```swift +extension RegexComponentBuilder { + @_disfavoredOverload + public static func buildPartialBlock( + first r: Component + ) -> Regex + + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildPartialBlock< + // AccumulatedWholeMatch, NextWholeMatch, + // AccumulatedCapture..., NextCapture..., + // Accumulated: RegexComponent, Next: RegexComponent + // >( + // accumulated: Accumulated, next: Component + // ) -> Regex<(Substring, AccumulatedCapture..., NextCapture...)> + // where Accumulated.RegexOutput == (AccumulatedWholeMatch, AccumulatedCapture...), + // Next.RegexOutput == (NextWholeMatch, NextCapture...) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0, C1)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0, C1) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0, C1, C2)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0, C1, C2) + + // ... `O(arity^2)` overloads of `buildPartialBlock(accumulated:next:)` +} +``` + +To support `if #available(...)` statements, `buildLimitedAvailability(_:)` is defined with overloads to support up to 10 captures. The overload for non-capturing regexes, due to the lack of generic constraints, must be annotated with `@_disfavoredOverload` in order not shadow other overloads. We expect that a variadic-generic version of this method will eventually superseded all of these overloads. + +```swift +extension RegexComponentBuilder { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildLimitedAvailability< + // Component, WholeMatch, Capture... + // >( + // _ component: Component + // ) where Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?)> + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?, C1?)> + + // ... `O(arity)` overloads of `buildLimitedAvailability(_:)` +} +``` + +`buildOptional` and `buildEither` are intentionally not supported due to ergonomic issues and fundamental semantic differences between regex conditionals and result builder conditionals. Please refer to the [alternatives considered](#support-buildoptional-and-buildeither) section for detailed rationale. + +
+ +### Capture + +Capture is a common regex feature that saves a portion of the input upon match. In regex builder, `Capture` and `TryCapture` are regex components that produce a new regex by inserting the captured pattern's whole match (`.0`) to the `.1` position of `RegexOutput`. When a transform closure is provided, the whole match (`.0`) of the captured content will be transformed to using the closure. + +```swift +public struct Capture: RegexComponent { ... } +public struct TryCapture: RegexComponent { ... } +``` + +To do a simple capture, you provide `Capture` with a regex component or a regex component builder closure. + +```swift +// Equivalent: '(CREDIT|DEBIT)' +Capture { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} // `.RegexOutput == (Substring, Substring)` +``` + +A capture will be represented in the type signature as a slice of the input, i.e. `Substring`. To transform the captured substring into another value during matching, specify a `transform:` closure. + +```swift +// This example is similar to the one above, however in this example we +// transform the result of the capture into: +// "Transaction Kind: CREDIT" or "Transaction Kind: DEBIT" +Capture { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} transform: { + "Transaction Kind: \($0)" +} // `.RegexOutput == (Substring, String)` +``` + +The transform closure can throw. When a transform closure throws during matching, the matching will abort and the error will be propagated directly to the top-level matching API that's being called, e.g. `Regex.wholeMatch(in:)` and `Regex.prefixMatch(in:)`. Aborting is useful for cases where you know that matching can never succeed or when you detect that an important invariant has been violated and the matching procedure needs to be aborted. + +An alternative version of capture is called `TryCapture`, which works in cases where you want to transform the capture, but the transformation may return nil. When a nil is returned, the regex engine backtracks and tries an alternative. For example, `TryCapture` makes it easy to directly transform a capture by calling a failable initializer during matching. + +```swift +enum TransactionKind: String { + case credit = "CREDIT" + case debit = "DEBIT" +} + +TryCapture { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} transform: { + // This initializer may return nil which is why we used TryCapture. + TransactionKind(rawValue: String($0)) +} +``` + +
+API definition + +```swift +public struct Capture: RegexComponent { + public var regex: Regex { get } +} + +public struct TryCapture: RegexComponent { + public var regex: Regex { get } +} +``` + +Below are `Capture` and `TryCapture` initializer variants on capture arity 0. Higher capture arities are omitted for simplicity. + +```swift +extension Capture { + public init( + _ component: R + ) where Output == (Substring, W), R.RegexOutput == W + + public init( + _ component: R, as reference: Reference + ) where Output == (Substring, W), R.RegexOutput == W + + public init( + _ component: R, + transform: @Sendable @escaping (W) throws -> NewCapture + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + @RegexComponentBuilder _ component: () -> R + ) where Output == (Substring, W), R.RegexOutput == W + + // ... `O(arity)` overloads +} + +extension TryCapture { + public init( + _ component: R, + transform: @Sendable @escaping (W) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + @RegexComponentBuilder _ component: () -> R, + transform: @Sendable @escaping (W) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + // ... `O(arity)` overloads +} +``` + +
+ +### Mapping Output + +In addition to transforming individual captures within a regex, you can also map the output of an entire regex to a different output type. You can use the `mapOutput(_:)` methods to reorder captures, flatten nested optionals, or create instances of a custom type. + +This example shows how you can transform the output of a regex with three capture groups into an instance of a custom `SemanticVersion` type, matching strings such as `"1.0.0"` or `"1.0"`: + +```swift +struct SemanticVersion: Hashable { + var major, minor, patch: Int +} + +let semverRegex = Regex { + TryCapture(OneOrMore(.digit)) { Int($0) } + "." + TryCapture(OneOrMore(.digit)) { Int($0) } + Optionally { + "." + TryCapture(OneOrMore(.digit)) { Int($0) } + } +}.mapOutput { _, c1, c2, c3 in + SemanticVersion(major: c1, minor: c2, patch: c3 ?? 0) +} + +let semver1 = "1.11.4".firstMatch(of: semverRegex)?.output +// semver1 == SemanticVersion(major: 1, minor: 11, patch: 4) +let semver2 = "0.6".firstMatch(of: semverRegex)?.output +// semver2 == SemanticVersion(major: 0, minor: 6, patch: 0) +``` + +
+API definition + +Note: This extension is defined in the standard library, not the `RegexBuilder` module. + +```swift +extension Regex { + /// Returns a regex that transforms its matches using the given closure. + /// + /// When you call `mapOutput(_:)` on a regex, you change the type of + /// output available on each match result. The `body` closure is called + /// when each match is found to transform the result of the match. + /// + /// - Parameter body: A closure for transforming the output of this + /// regex. + /// - Returns: A regex that has `NewOutput` as its output type. + func mapOutput(_ body: @escaping (Output) -> NewOutput) -> Regex +} +``` +
+ +### Reference + +Reference is a feature that can be used to achieve named captures and named backreferences from textual regexes. Simply state what type the reference will hold on to and you can use it later once you've matched a string to get back a specific capture. Note the type you pass to reference will be whatever the result of a capture's transform is. A capture with no transform always has a reference type of `Substring`. + +```swift +let kind = Reference(Substring.self) + +let regex = Capture(as: kind) { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} + +let input = "CREDIT" +if let result = try regex.firstMatch(in: input) { + print(result[kind]) // Optional("CREDIT") +} +``` + +Capturing stores the most recently captured content, and references can be used as a name to look up the result of matching. The reference itself can also be used within a regex (commonly called a "backreference") to match the most recently captured content during matching. + +```swift +let a = Reference(Substring.self) +let b = Reference(Substring.self) +let c = Reference(Substring.self) +let regex = Regex { + Capture("abc", as: a) + Capture("def", as: b) + ZeroOrMore { + Capture("hij", as: c) + } + a + Capture(b) +} + +if let result = try regex.firstMatch(in: "abcdefabcdef") { + print(result[a]) // => Optional("abc") + print(result[b]) // => Optional("def") + print(result[c]) // => nil +} +``` + +A regex is considered invalid when it contains a use of reference without it ever being used as the `as:` argument to an initializer of `Capture` or `TryCapture` in the regex. When this occurs in the regex builder DSL, a runtime error will be reported. + +Similarly, the argument to a `Regex.Match.subscript(_:)` must have been used as the `as:` argument to an initializer of `Capture` or `TryCapture` in the regex that produced the match. + +
+API definition + +```swift +/// A reference to a regex capture. +public struct Reference: RegexComponent { + public init(_ captureType: Capture.Type = Capture.self) + public var regex: Regex +} + +extension Capture { + public init( + _ component: R, + as reference: Reference, + transform: @escaping (Substring) throws -> NewCapture + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + as reference: Reference, + @RegexComponentBuilder _ component: () -> R + ) where Output == (Substring, W), R.RegexOutput == W + + // ... `O(arity)` overloads +} + +extension TryCapture { + public init( + _ component: R, + as reference: Reference, + transform: @escaping (Substring) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + as reference: Reference, + @RegexComponentBuilder _ component: () -> R, + transform: @escaping (Substring) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + // ... `O(arity)` overloads +} + +extension Regex.Match { + /// Returns the capture referenced by the given reference. + /// + /// - Precondition: The reference must have been captured in the regex that produced this match. + public subscript(_ reference: Reference) -> Capture? { get } +} +``` + +
+ +### Alternation + +An alternation is used to match one of multiple patterns. When one pattern in an alternation does not match successfully, the regex engine tries the next pattern until there's a successful match. An alternation wraps its underlying patterns' capture types in an `Optional` and concatenates them together, first to last. + +```swift +let choice = ChoiceOf { + regex0 // Regex + regex1 // Regex<(Substring, Int)> + regex2 // Regex<(Substring, Float)> + regex3 // Regex<(Substring, Substring)> +} // => Regex<(Substring, Int?, Float?, Substring?)> +``` + +`AlternationBuilder` is a result builder type for creating alternations from components of a block. + +```swift +@resultBuilder +public struct AlternationBuilder { ... } +``` + +To the developer, the top-level API is a type named `ChoiceOf`. This type has an initializer that accepts an `@AlternationBuilder` closure. + +```swift +public struct ChoiceOf: RegexComponent { + ... + public init( + @AlternationBuilder builder: () -> R + ) where R.RegexOutput == Output +} +``` + +For example, the following code creates an alternation of two subpatterns. + +```swift +let regex = Regex { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} +let match = try regex.prefixMatch(in: "DEBIT 04032020 Payroll $69.73") +match?.0 // => "DEBIT" +``` + +
+API definition + +`AlternationBuilder` is mostly similar to `RegexComponent` with the following distinctions: +- Empty blocks are not supported. +- Capture types are wrapped in a layer of `Optional` before being concatenated in the resulting `Output` type. +- `buildEither(first:)` and `buildEither(second:)` are overloaded for each supported capture arity because they need to wrap capture types in `Optional`. + +```swift +public struct ChoiceOf: RegexComponent { + public var regex: Regex { get } + public init( + @AlternationBuilder builder: () -> R + ) where R.RegexOutput == Output +} + +@resultBuilder +public enum AlternationBuilder { + public typealias Component = RegexComponentBuilder.Component + + /// Returns a component by wrapping the component regex in `Component` and + /// recording its source location. + public static func buildExpression( + _ regex: R, + file: String = #file, + function: String = #function, + line: Int = #line, + column: Int = #column + ) -> Component + + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildPartialBlock< + // R, WholeMatch, Capture... + // >( + // first component: Component + // ) -> Regex<(Substring, Capture?...)> + // where Component.RegexOutput == (WholeMatch, Capture...), + + @_disfavoredOverload + public static func buildPartialBlock( + first r: Component + ) -> Regex + + public static func buildPartialBlock( + first r: Component + ) -> Regex<(Substring, C0?)> where R.RegexOutput == (W, C0) + + public static func buildPartialBlock( + first r: Component + ) -> Regex<(Substring, C0?, C1?)> where R.RegexOutput == (W, C0, C1) + + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildPartialBlock< + // AccumulatedWholeMatch, NextWholeMatch, + // AccumulatedCapture..., NextCapture..., + // Accumulated: RegexComponent, Next: RegexComponent + // >( + // accumulated: Accumulated, next: Component + // ) -> Regex<(Substring, AccumulatedCapture..., NextCapture...)> + // where Accumulated.RegexOutput == (AccumulatedWholeMatch, AccumulatedCapture...), + // Next.RegexOutput == (NextWholeMatch, NextCapture...) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0?)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0?, C1?)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0, C1) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0?, C1?, C2?)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0, C1, C2) + + // ... `O(arity^2)` overloads of `buildPartialBlock(accumulated:next:)` +} + +extension AlternationBuilder { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildLimitedAvailability< + // Component, WholeMatch, Capture... + // >( + // _ component: Component + // ) -> Regex<(Substring, Capture?...)> + // where Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?)> + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?, C1?)> + + // ... `O(arity)` overloads of `buildLimitedAvailability(_:)` + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?, C1?, C2?, C3?, C4?, C5?, C6?, C7?, C8, C9?)> where R.RegexOutput == (W, C0, C1, C2, C3, C4, C5, C6, C7, C8, C9) +} +``` + +
+ +### Repetition + +One of the most useful features of regex is repetition, aka. quantification, as it allows you to match a specific range of number of occurrences of a subpattern. Regex builder provides 5 repetition components: `One`, `OneOrMore`, `ZeroOrMore`, `Optionally`, and `Repeat`. + +```swift +public struct One: RegexComponent { ... } +public struct OneOrMore: RegexComponent { ... } +public struct ZeroOrMore: RegexComponent { ... } +public struct Optionally: RegexComponent { ... } +public struct Repeat: RegexComponent { ... } +``` + +| Repetition in regex builder | Textual regex equivalent | +|-----------------------------|--------------------------| +| `One(...)` | `...` | +| `OneOrMore(...)` | `...+` | +| `ZeroOrMore(...)` | `...*` | +| `Optionally(...)` | `...?` | +| `Repeat(..., count: n)` | `...{n}` | +| `Repeat(..., n...)` | `...{n,}` | +| `Repeat(..., n...m)` | `...{n,m}` | + +`One`, `OneOrMore` and count-based `Repeat` are quantifiers that produce a new regex with the original capture types. Their `Output` type is `Substring` followed by the component's capture types. `ZeroOrMore`, `Optionally`, and range-based `Repeat` are quantifiers that produce a new regex with optional capture types. Their `Output` type is `Substring` followed by the component's capture types wrapped in `Optional`. + +| Quantifier | Component `Output` | Result `Output` | +|------------------------------------------------------|----------------------------|----------------------------| +| `One`
`OneOrMore`
`Repeat(..., count: ...)` | `(WholeMatch, Capture...)` | `(Substring, Capture...)` | +| `One`
`OneOrMore`
`Repeat(..., count: ...)` | `WholeMatch` (non-tuple) | `Substring` | +| `ZeroOrMore`
`Optionally`
`Repeat(..., n...m)` | `(WholeMatch, Capture...)` | `(Substring, Capture?...)` | +| `ZeroOrMore`
`Optionally`
`Repeat(..., n...m)` | `WholeMatch` (non-tuple) | `Substring` | + +
+API definition + +```swift +public struct One: RegexComponent { + public var regex: Regex { get } +} + +public struct OneOrMore: RegexComponent { + public var regex: Regex { get } +} + +public struct ZeroOrMore: RegexComponent { + public var regex: Regex { get } +} + +public struct Optionally: RegexComponent { + public var regex: Regex { get } +} + +public struct Repeat: RegexComponent { + public var regex: Regex { get } +} +``` + +Due to the lack of variadic generics, initializers must be overloaded for every supported capture arity. + +```swift +extension One { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // _ behavior: RegexRepetitionBehavior = .eager + // ) + // where Output == (Substring, Capture...)>, + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ behavior: RegexRepetitionBehavior = .eager, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture...), + // Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring + + @_disfavoredOverload + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + // ... `O(arity)` overloads +} + +extension OneOrMore { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // _ behavior: RegexRepetitionBehavior = .eager + // ) + // where Output == (Substring, Capture...)>, + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ behavior: RegexRepetitionBehavior = .eager, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture...), + // Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring + + @_disfavoredOverload + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + // ... `O(arity)` overloads +} + +extension ZeroOrMore { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // _ behavior: RegexRepetitionBehavior = nil + // ) + // where Output == (Substring, Capture?...)>, + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ behavior: RegexRepetitionBehavior? = nil, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture?...), + // Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring + + @_disfavoredOverload + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == (Substring, C0?), Component.RegexOutput == (W, C0) + + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0?), Component.RegexOutput == (W, C0) + + // ... `O(arity)` overloads +} + +extension Optionally { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // _ behavior: RegexRepetitionBehavior? = nil + // ) + // where Output == (Substring, Capture?...), + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ behavior: RegexRepetitionBehavior? = nil, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture?...)>, + // Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring + + @_disfavoredOverload + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == (Substring, C0?), Component.RegexOutput == (W, C0) + + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0?), Component.RegexOutput == (W, C0) + + // ... `O(arity)` overloads +} + +extension Repeat { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // count: Int, + // _ behavior: RegexRepetitionBehavior? = nil + // ) + // where Output == (Substring, Capture...), + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // count: Int, + // _ behavior: RegexRepetitionBehavior? = nil, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture...), + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture..., RE: RangeExpression + // >( + // _ component: Component, + // _ expression: RE, + // _ behavior: RegexRepetitionBehavior? = nil + // ) + // where Output == (Substring, Capture?...), + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture..., RE: RangeExpression + // >( + // _ expression: RE, + // _ behavior: RegexRepetitionBehavior? = nil, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture?...), + // Component.RegexOutput == (WholeMatch, Capture...) + + // Nullary + + @_disfavoredOverload + public init( + _ component: Component, + count: Int, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring, R.Bound == Int + + @_disfavoredOverload + public init( + count: Int, + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring, R.Bound == Int + + @_disfavoredOverload + public init( + _ component: Component, + _ expression: RE, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring, R.Bound == Int + + @_disfavoredOverload + public init( + _ expression: RE, + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring, R.Bound == Int + + + // Unary + + public init( + _ component: Component, + count: Int, + _ behavior: RegexRepetitionBehavior? = nil + ) + where Output == (Substring, C0), + Component.RegexOutput == (Substring, C0), + R.Bound == Int + + public init( + count: Int, + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) + where Output == (Substring, C0), + Component.RegexOutput == (Substring, C0), + R.Bound == Int + + public init( + _ component: Component, + _ expression: RE, + _ behavior: RegexRepetitionBehavior? = nil + ) + where Output == (Substring, C0?), + Component.RegexOutput == (W, C0), + R.Bound == Int + + public init( + _ expression: RE, + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) + where Output == (Substring, C0?), + Component.RegexOutput == (W, C0), + R.Bound == Int + + // ... `O(arity)` overloads +} +``` + +
+ +#### Repetition behavior + +Repetition behavior defines how eagerly a repetition component should match the input. Behavior can be unspecified, in which case it will default to `.eager` unless an option is provided to change the default (see [Unicode for String Processing](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md#unicode-for-string-processing)). + +```swift +/// Specifies how much to attempt to match when using a quantifier. +public struct RegexRepetitionBehavior { + /// Match as much of the input string as possible, backtracking when + /// necessary. + public static var eager: RegexRepetitionBehavior { get } + + /// Match as little of the input string as possible, expanding the matched + /// region as necessary to complete a match. + public static var reluctant: RegexRepetitionBehavior { get } + + /// Match as much of the input string as possible, performing no backtracking. + public static var possessive: RegexRepetitionBehavior { get } +} +``` + +| Repetition behavior in regex builder | Textual regex equivalent | +|--------------------------------------|--------------------------| +| `.eager` | no suffix | +| `.reluctant` | suffix `?` | +| `.possessive` | suffix `+` | + +To demonstrate how each repetition behavior works, let's look at the following +example. Suppose we want to make a regex that wants to capture an html tag, e.g. +``. We might start with something like the following: + + +```swift +let tag = Reference(Substring.self) + +let htmlRegex = Regex { + "<" + Capture(as: tag) { + // Remember, the default behavior is .eager here! + OneOrMore(.any) + } + ">" +} + +let input = #"print("hello world!")"# + +if let result = htmlRegex.firstMatch(in: input) { + print(result[tag]) +} +``` + +The code above prints `code>print("hello world!")"` because our string was out of characters. This is intended for `.possessive` because it doesn't backtrack the string to find a match for the ending `">"`. + +The desired behavior in this case is `.reluctant`, where the repetition will match as little of the input string as possible. If we use `OneOrMore(.any, .reluctant)`, the code prints expected output ``. + +### Anchors and Lookaheads + +Anchors are a way to constrain a regex, or part of a regex, to matching particular locations within an input string. Regex builder provides anchors that correspond to regex syntax anchors. Regex builder also provides two types that represent look-ahead assertions — essentially a non-consuming sub-regex that has to match (or not match) before the regex can proceed. + +```swift +/// A regex component that matches a specific condition at a particular position +/// in an input string. +/// +/// You can use anchors to guarantee that a match only occurs at certain points +/// in an input string, such as at the beginning of the string or at the end of +/// a line. +public struct Anchor: RegexComponent { + /// An anchor that matches at the start of a line, including the start of + /// the input string. + /// + /// This anchor is equivalent to `^` in regex syntax when the `m` option + /// has been enabled or `anchorsMatchLineEndings(true)` has been called. + public static var startOfLine: Anchor { get } + + /// An anchor that matches at the end of a line, including at the end of + /// the input string. + /// + /// This anchor is equivalent to `$` in regex syntax when the `m` option + /// has been enabled or `anchorsMatchLineEndings(true)` has been called. + public static var endOfLine: Anchor { get } + + /// An anchor that matches at a word boundary. + /// + /// Word boundaries are identified using the Unicode default word boundary + /// algorithm by default. To specify a different word boundary algorithm, + /// see the `RegexComponent.wordBoundaryKind(_:)` method. + /// + /// This anchor is equivalent to `\b` in regex syntax. + public static var wordBoundary: Anchor { get } + + /// An anchor that matches at the start of the input string. + /// + /// This anchor is equivalent to `\A` in regex syntax. + public static var startOfSubject: Anchor { get } + + /// An anchor that matches at the end of the input string. + /// + /// This anchor is equivalent to `\z` in regex syntax. + public static var endOfSubject: Anchor { get } + + /// An anchor that matches at the end of the input string or at the end of + /// the line immediately before the the end of the string. + /// + /// This anchor is equivalent to `\Z` in regex syntax. + public static var endOfSubjectBeforeNewline: Anchor { get } + + /// An anchor that matches at a grapheme cluster boundary. + /// + /// This anchor is equivalent to `\y` in regex syntax. + public static var textSegmentBoundary: Anchor { get } + + /// An anchor that matches at the first position of a match in the input + /// string. + /// + /// This anchor is equivalent to `\y` in regex syntax. + public static var firstMatchingPositionInSubject: Anchor { get } + + /// The inverse of this anchor, which matches at every position that this + /// anchor does not. + /// + /// For the `wordBoundary` and `textSegmentBoundary` anchors, the inverted + /// version corresponds to `\B` and `\Y`, respectively. + public var inverted: Anchor { get } +} + +/// A regex component that allows a match to continue only if its contents +/// match at the given location. +/// +/// A lookahead is a zero-length assertion that its included regex matches at +/// a particular position. Lookaheads do not advance the overall matching +/// position in the input string — once a lookahead succeeds, matching continues +/// in the regex from the same position. +public struct Lookahead: RegexComponent { + /// Creates a lookahead from the given regex component. + public init(_ component: some RegexComponent) + + /// Creates a lookahead from the regex generated by the given builder closure. + public init(@RegexComponentBuilder _ component: () -> some RegexComponent) +} + +/// A regex component that allows a match to continue only if its contents +/// do not match at the given location. +/// +/// A negative lookahead is a zero-length assertion that its included regex +/// does not match at a particular position. Lookaheads do not advance the +/// overall matching position in the input string — once a lookahead succeeds, +/// matching continues in the regex from the same position. +public struct NegativeLookahead: RegexComponent { + /// Creates a negative lookahead from the given regex component. + public init(_ component: some RegexComponent) + + /// Creates a negative lookahead from the regex generated by the given builder + /// closure. + public init(@RegexComponentBuilder _ component: () -> some RegexComponent) +} +``` + +### Subpattern + +In textual regex, one can refer to a subpattern to avoid duplicating the subpattern, for example: + +``` +(you|I) say (goodbye|hello); (?1) say (?2) +``` + +The above regex is equivalent to + +``` +(you|I) say (goodbye|hello); (you|I) say (goodbye|hello) +``` + +With regex builder, there is no special API required to reuse existing subpatterns, as a subpattern can be defined modularly using a `let` binding inside or outside a regex builder closure. + +```swift +Regex { + let subject = ChoiceOf { + "I" + "you" + } + let object = ChoiceOf { + "goodbye" + "hello" + } + subject + "say" + object + ";" + subject + "say" + object +} +``` + +### Scoping + +Because the regex engine backtracks by default when trying to match on a string, sometimes this backtracking can be wasted performance because we don't want to try various possibilities to eventually (maybe) find a match. + +In textual regexes, atomic groups (`(?>...)`) solve this problem by informing the regex engine to actually discard the backtrack location of a group, that is, defining a scope for backtracking. In regex builder, the `Local` type serves this purpose. + +```swift +public struct Local: RegexComponent { ... } +``` + +For example, the following regex matches string `abcc` but not `abc`. + +```swift +Regex { + "a" + Local { + ChoiceOf { + "bc" + "b" + } + } + "c" +} +``` + +If our input is `abcc`, we'll successfully find a match, however if we try to match against `abc` we won't get a match. The reason behind this is that in the `ChoiceOf` we actually matched the "bc" case first, but due to the local group we immediately disregard the backtracking location and continue to try and the rest of the regex. Since we matched the "bc", we don't have anymore string left to match the "c" and our local group will not try and attempt to match the other option, "b". + +
+API definition + +```swift +public struct Local: RegexComponent { + public var regex: Regex + + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init( + // @RegexComponentBuilder _ component: () -> Component + // ) where Output == (Substring, Capture...), Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + public init( + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0, C1), Component.RegexOutput == (W, C0, C1) + + // ... `O(arity)` overloads +} +``` + +
+ +### Composability + +Let's put everything together now and parse this example bank statement. + +``` +CREDIT 04062020 PayPal transfer $4.99 +CREDIT 04032020 Payroll $69.73 +DEBIT 04022020 ACH transfer $38.25 +DEBIT 03242020 IRS tax payment $52249.98 +``` + +Here we have 2 types of transaction kinds, CREDIT and DEBIT, we have a date +denoted by mmddyyyy, a description, and the amount paid. + +```swift +enum TransactionKind: String { + case credit = "CREDIT" + case debit = "DEBIT" +} + +struct Date { + var month: Int + var day: Int + var year: Int + + init?(mmddyyyy: String) { + ... + } +} + +let statementRegex = Regex { + // First, let's capture the transaction kind by wrapping our `ChoiceOf` in a + // `TryCapture` because our initializer can return nil on failure. + TryCapture { + ChoiceOf { + "CREDIT" + "DEBIT" + } + } transform: { + TransactionKind(rawValue: String($0)) + } + + OneOrMore(.whitespace) + + // Next, lets represent our date as 3 separate repeat quantifiers. The first + // two will require 2 digit characters, and the last will require 4. Then + // we'll take the entire substring and try to parse a date out. + TryCapture { + Repeat(.digit, count: 2) + Repeat(.digit, count: 2) + Repeat(.digit, count: 4) + } transform: { + Date(mmddyyyy: String($0)) + } + + OneOrMore(.whitespace) + + // Next, grab the description which can be any combination of word characters, + // digits, etc. + Capture { + OneOrMore(.any, .reluctant) + } + + OneOrMore(.whitespace) + + "$" + + // Finally, we'll grab one or more digits which will represent the whole + // dollars, match the decimal point, and finally get 2 digits which will be + // our cents. + TryCapture { + OneOrMore(.digit) + "." + Repeat(.digit, count: 2) + } transform: { + Double($0) + } +} + +for match in statement.matches(of: statementRegex) { + let (line, kind, date, description, amount) = match.output + ... +} +``` + +## Source compatibility + +Regex builder will be shipped in a new module named `RegexBuilder`, and thus will not affect the source compatibility of the existing code. + +## Effect on ABI stability + +The proposed feature does not change the ABI of existing features. + +## Effect on API resilience + +The proposed feature relies heavily upon overloads of `buildBlock` and `buildPartialBlock(accumulated:next:)` to work for different capture arities. In the fullness of time, we are hoping for variadic generics to supersede existing overloads. Such a change should not involve ABI-breaking modifications as it is merely a change of overload resolution. + +## Future directions + +### Conversion to textual regex + +Sometimes it may be useful to convert a regex created using regex builder to textual regex. This may be achieved in the future by extending `RegexComponent` with a computed property. + +```swift +extension RegexComponent { + public func makeTextualRegex() -> String? +} +``` + +It is worth noting that the internal representation of a `Regex` is _not_ textual regex, but an efficient pattern matching bytecode compiled from an abstract syntax tree. Moreover, not every `Regex` can be converted to textual regex. Regex builder supports arbitrary types that conform to the `RegexComponent` protocol, including `CustomMatchingRegexComponent` (pitched in [String Processing Algorithms]) which can be implemented with arbitrary code. If a `Regex` contains a `CustomMatchingRegexComponent`, it cannot be converted to textual regex. + +### Recursive subpatterns + +Sometimes, a textual regex may also use `(?R)` or `(?0)` to recusively evaluate the entire regex. For example, the following textual regex matches "I say you say I say you say hello". + +``` +(you|I) say (goodbye|hello|(?R)) +``` + +For this, `Regex` offers a special initializer that allows its pattern to recursively reference itself. This is somewhat akin to a fixed-point combinator. + +```swift +extension Regex { + public init( + @RegexComponentBuilder _ content: (Regex) -> R + ) where R.RegexOutput == Match +} +``` + +With this initializer, the above regex can be expressed as the following using regex builder. + +```swift +Regex { wholeSentence in + ChoiceOf { + "I" + "you" + } + "say" + ChoiceOf { + "goodbye" + "hello" + wholeSentence + } +} +``` + +There are some concerns with this design which we need to consider: +- Due to the lack of labeling, the argument to the builder closure can be arbitrarily named and cause confusion. +- When there is an initializer that accepts a result builder closure, overloading that initializer with the same argument labels could lead to bad error messages upon interor type errors. + +## Alternatives considered + +### Semicolons or parentheses instead of `One` + +In the DSL syntax as described in the first version of this proposal, there was a problem with the use of leading-dot syntax for character classes and other "atoms" and the builder syntax: +```swift +Regex { + .digit + OneOrMore(.whitespace) +} +``` +worked as expected, but: +```swift +Regex { + OneOrMore(.whitespace) + .digit +} +``` +did not, because `.digit` parses as a property on `OneOrMore` rather than a regex component. This could have been resolved by making people use either semicolons: +```swift +Regex { + OneOrMore(.whitespace); + .digit +} +``` +or parentheses: +```swift +Regex { + OneOrMore(.whitespace) + (.digit) +} +``` + +Instead we decided to introduce the quantifier `One` to resolve the ambiguity: +```swift +Regex { + OneOrMore(.whitespace) + One(.digit) +} +``` + +This increase the API surface, which is mildly undesirable, but feels much more stylistically consistent with the rest of the DSL and with Swift as whole. We also considered a "two protocol" approach that would force the use of `One` in these cases by making it impossible to use the dot-prefixed "atoms" within builder blocks, but this seems like too much heavy machinery to resolve the problem. + +### Operators for quantification and alternation + +While `ChoiceOf` and quantifier types provide a general way of creating alternations and quantifications, we recognize that some synctactic sugar can be useful for creating one-liners like in textual regexes, e.g. infix operator `|`, postfix operator `*`, etc. + +```swift +// The following functions implement what would be possible with variadic +// generics (using imaginary syntax) as a single function: +// +// public func | < +// R0: RegexComponent, R1: RegexComponent, +// WholeMatch0, WholeMatch1, +// Capture0..., Capture1... +// >( +// _ r0: RegexComponent, +// _ r1: RegexComponent +// ) -> Regex<(Substring, Capture0?..., Capture1?...)> +// where R0.RegexOutput == (WholeMatch0, Capture0...), +// R1.RegexOutput == (WholeMatch1, Capture1...) + +@_disfavoredOverload +public func | (lhs: R0, rhs: R1) -> Regex where R0: RegexComponent, R1: RegexComponent { + +public func | (lhs: R0, rhs: R1) -> Regex<(Substring, C0?)> where R0: RegexComponent, R1: RegexComponent, R1.RegexOutput == (W1, C0) + +public func | (lhs: R0, rhs: R1) -> Regex<(Substring, C0?, C1?)> where R0: RegexComponent, R1: RegexComponent, R1.RegexOutput == (W1, C0, C1) + +// ... `O(arity^2)` overloads. +``` + +However, like `RegexComponentBuilder.buildPartialBlock(accumulated:next:)`, operators such as `|`, `+`, `*`, `.?` require a large number of overloads to work with regexes of every capture arity, compounded by the fact that operator type checking is prone to performance issues in Swift. Here is a list of + +| Opreator | Meaning | Required number of overloads | +|---------------|---------------------------|------------------------------| +| Infix `\|` | Choice of two | `O(arity^2)` | +| Postfix `*` | Zero or more eagerly | `O(arity)` | +| Postfix `*?` | Zero or more reluctantly | `O(arity)` | +| Postfix `*+` | Zero or more possessively | `O(arity)` | +| Postfix `+` | One or more eagerly | `O(arity)` | +| Postfix `+?` | One or more reluctantly | `O(arity)` | +| Postfix `++` | One or more possessively | `O(arity)` | +| Postfix `.?` | Optionally eagerly | `O(arity)` | +| Postfix `.??` | Optionally reluctantly | `O(arity)` | +| Postfix `.?+` | Optionally possessively | `O(arity)` | + + When variadic generics are supported in the future, we may be able to define one function per operator and reduce type checking burdens. + +### Postfix `capture` and `tryCapture` methods + +An earlier iteration of regex builder declared `capture` and `tryCapture` as methods on `RegexComponent`, meaning that you can append `.capture(...)` to any subpattern within a regex to capture it. For example: + +```swift +Regex { + OneOrMore { + r0.capture() + r1 + }.capture() +} // => Regex<(Substring, Substring, Substring)> +``` + +However, there are two shortcomings of this design: + +1. When a subpattern to be captured contains multiple components, the developer has to explicitly group them using a `Regex { ... }` block. + + ```swift + let emailPattern = Regex { + let word = OneOrMore(.word) + Regex { // <= Had to explicitly group multiple components + ZeroOrMore { + word + "." + } + word + }.capture() + "@" + Regex { + word + OneOrMore { + "." + word + } + }.capture() + } // => Regex<(Substring, Substring, Substring)> + ``` + +2. When there are nested captures, it is harder to number the captures visually because the order `capture()` appears is flipped in the postfix (method) notation. + + ```swift + let emailSuffixPattern = Regex { + "@" + Regex { + word + OneOrMore { + "." + word.capture() // top-level domain (.0) + } + }.capture() // full domain (.1) + } // => Regex<(Substring, Substring, Substring)> + // + // full domain ^~~~~~~~~ + // top-level domain ^~~~~~~~~ + ``` + + In comparison, prefix notation (`Capture` and `TryCapture` as a types) makes it easier to visually capture captures as you can number captures in the order they appear from top to bottom. This is consistent with textual regexes where capturing groups are numbered by the left parenthesis of the group from left to right. + + ```swift + let emailSuffixPattern = Regex { + Capture { // full domain (.0) + word + OneOrMore { + "." + Capture(word) // top-level domain (.1) + } + } + } // => Regex<(Substring, Substring, Substring)> + // + // full domain ^~~~~~~~~ + // top-level domain ^~~~~~~~~ + ``` + +### Unify quantifiers under `Repeat` + +Since `Repeat` is the most general version of quantifiers, one could argue for all quantifiers to be unified under the type `Repeat`, for example: + +```swift +Repeat(oneOrMore: r) +Repeat(zeroOrMore: r) +Repeat(optionally: r) +``` + +However, given that one-or-more (`+`), zero-or-more (`*`) and optional (`?`) are the most common quantifiers in textual regexes, we believe that these quantifiers deserve their own type and should be written as a single word instead of two. This can also reduce visual clutter when the quantification is used in multiple places of a regex. + +### Free functions instead of types + +One could argue that type such as `OneOrMore` could be defined as a top-level function that returns `Regex`. While it is entirely possible to do so, it would lose the name scoping benefits of a type and pollute the top-level namespace with `O(arity^2)` overloads of quantifiers, `capture`, `tryCapture`, etc. This could be detrimental to the usefulness of code completion. + +Another reason to use types instead of free functions is consistency with existing result-builder-based DSLs such as SwiftUI. + +### Support `buildOptional` and `buildEither` + +To support `if` statements, an earlier iteration of this proposal defined `buildEither(first:)`, `buildEither(second:)` and `buildOptional(_:)` as the following: + +```swift +extension RegexComponentBuilder { + public static func buildEither< + Component, WholeMatch, Capture... + >( + first component: Component + ) -> Regex<(Substring, Capture...)> + where Component.RegexOutput == (WholeMatch, Capture...) + + public static func buildEither< + Component, WholeMatch, Capture... + >( + second component: Component + ) -> Regex<(Substring, Capture...)> + where Component.RegexOutput == (WholeMatch, Capture...) + + public static func buildOptional< + Component, WholeMatch, Capture... + >( + _ component: Component? + ) where Component.RegexOutput == (WholeMatch, Capture...) +} +``` + +However, multiple-branch control flow statements (e.g. `if`-`else` and `switch`) would need to be required to produce either the same regex type, which is limiting, or an "either-like" type, which can be difficult to work with when nested. Unlike `ChoiceOf`, producing a tuple of optionals is not an option, because the branch taken would be decided when the builder closure is executed, and it would cause capture numbering to be inconsistent with conventional regex. + +Moreover, result builder conditionals does not work the same way as regex conditionals. In regex conditionals, the conditions are themselves regexes and are evaluated by the regex engine during matching, whereas result builder conditionals are evaluated as part of the builder closure. We hope that a future result builder feature will support "lifting" control flow conditions into the DSL domain, e.g. supporting `Regex` as a condition. + +### Flatten optionals + +With the proposed design, `ChoiceOf` with `AlternationBuilder` wraps every component's capture type with an `Optional`. This means that any `ChoiceOf` with optional-capturing components would lead to a doubly-nested optional captures. This could make the result of matching harder to use. + +```swift +ChoiceOf { + OneOrMore(Capture(.digit)) // RegexOutput == (Substring, Substring) + Optionally { + ZeroOrMore(Capture(.word)) // RegexOutput == (Substring, Substring?) + "a" + } // RegexOutput == (Substring, Substring??) +} // RegexOutput == (Substring, Substring?, Substring???) +``` + +One way to improve this could be overloading quantifier initializers (e.g. `ZeroOrMore.init(_:)`) and `AlternationBuilder.buildPartialBlock` to flatten any optionals upon composition. However, this would be non-trivial. Quantifier initializers would need to be overloaded `O(2^arity)` times to account for all possible positions of `Optional` that may appear in the `Output` tuple. Even worse, `AlternationBuilder.buildPartialBlock` would need to be overloaded `O(arity!)` times to account for all possible combinations of two `Output` tuples with all possible positions of `Optional` that may appear in one of the `Output` tuples. + +### Structured rather than flat captures + +We propose inferring capture types in such a way as to align with the traditional numbering of backreferences. This is because much of the motivation behind providing regex in Swift is their familiarity. + +If we decided to deprioritize this motivation, there are opportunities to infer safer, more ergonomic, and arguably more intuitive types for captures. For example, to be consistent with traditional regex backreferences quantifications of multiple or nested captures had to produce parallel arrays rather than an array of tuples. + +```swift +OneOrMore { + Capture { + OneOrMore(.hexDigit) + } + ".." + Capture { + OneOrMore(.hexDigit) + } +} + +// Flat capture types: +// => `RegexOutput == (Substring, Substring, Substring)>` + +// Structured capture types: +// => `RegexOutput == (Substring, (Substring, Substring))` +``` + +Similarly, an alternation of multiple or nested captures could produce a structured alternation type (or an anonymous sum type) rather than flat optionals. + +This is cool, but it adds extra complexity to regex builder and it isn't as clear because the generic type no longer aligns with the traditional regex backreference numbering. We think the consistency of the flat capture types trumps the added safety and ergonomics of the structured capture types. + +### Unify `Capture` with `TryCapture` + +The primary difference between `Capture` and `TryCapture` at the API level is that `TryCapture`'s transform closure returns an `Optional` of the target type, whereas `Capture`'s transform closure returns the target type. `TryCapture` would cause the regex engine to backtrack when the transform closure returns nil, whereas `Capture` does not backtrack. + +It has been argued in the review thread that the distinction between `Capture` and `TryCapture` need not be reflected at the type name level, but could be differentiated by argument label, e.g. `transform:`/`tryTransform:` or `map:`/`compactMap:`. However, doing so may cause ambiguity in cases where the transform closure is not the second, but the first, trailing closure in the initializer. + +```swift +extension Capture { + public init( + _ component: R, + map: @escaping (Substring) throws -> NewCapture + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + _ component: R, + compactMap: @escaping (Substring) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W +} +``` + +In this case, since the argument label will not be specified for the first trailing closure, using `Capture` where the component is a non-builder-closure may cause type-checking ambiguity. + +```swift +Regex { + Capture(OneOrMore(.digit)) { + Int($0) + } // Which output type, `(Substring, Substring)` or `(Substring, Substring?)`? +} +``` + +Spelling out `TryCapture` also has the benefit of clarity, as it makes clear that a capture's transform closure can cause the regex engine to backtrack. Since backtracking can be expensive, one could choose to throw errors instead and use a normal `Capture`. + +```swift +Regex { + Capture(OneOrMore(.digit)) { + guard let number = Int($0) else { + throw MyCustomParsingError.invalidNumber($0) + } + return number + } +} +``` + +[Declarative String Processing]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/DeclarativeStringProcessing.md +[Strongly Typed Regex Captures]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md +[Regex Syntax]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md +[String Processing Algorithms]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0357-regex-string-processing-algorithms.md diff --git a/proposals/0352-implicit-open-existentials.md b/proposals/0352-implicit-open-existentials.md new file mode 100644 index 0000000000..ccf49c1942 --- /dev/null +++ b/proposals/0352-implicit-open-existentials.md @@ -0,0 +1,684 @@ +# Implicitly Opened Existentials + +* Proposal: [SE-0352](0352-implicit-open-existentials.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Upcoming Feature Flag: `ImplicitOpenExistentials` (Implemented in Swift 6.0) (Enabled in Swift 6 language mode) +* Implementation: [apple/swift#41996](https://github.com/apple/swift/pull/41996), [macOS toolchain](https://ci.swift.org/job/swift-PR-toolchain-macos/120/artifact/branch-main/swift-PR-41996-120-osx.tar.gz) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0352-implicitly-opened-existentials/57553) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/77374319a7d70c866bd197faada46ecfce461645/proposals/0352-implicit-open-existentials.md) +* Previous Review: [First review](https://forums.swift.org/t/se-0352-implicitly-opened-existentials/56557/52) + +## Table of Contents + + * [Introduction](#introduction) + * [Proposed solution](#proposed-solution) + * [Moving between any and some](#moving-between-any-and-some) + * [Detailed design](#detailed-design) + * [When can we open an existential?](#when-can-we-open-an-existential) + * [Type-erasing resulting values](#type-erasing-resulting-values) + * ["Losing" constraints when type-erasing resulting values](#losing-constraints-when-type-erasing-resulting-values) + * [Contravariant erasure for parameters of function type](#contravariant-erasure-for-parameters-of-function-type) + * [Order of evaluation restrictions](#order-of-evaluation-restrictions) + * [Avoid opening when the existential type satisfies requirements (in Swift 5)](#avoid-opening-when-the-existential-type-satisfies-requirements-in-swift-5) + * [Suppressing explicit opening with as any P / as! any P](#suppressing-explicit-opening-with-as-any-p--as-any-p) + * [Source compatibility](#source-compatibility) + * [Effect on ABI stability](#effect-on-abi-stability) + * [Effect on API resilience](#effect-on-api-resilience) + * [Alternatives considered](#alternatives-considered) + * [Explicitly opening existentials](#explicitly-opening-existentials) + * [Value-dependent opening of existentials](#value-dependent-opening-of-existentials) + * [Revisions](#revisions) + * [Acknowledgments](#acknowledgments) + +## Introduction + +Existential types in Swift allow one to store a value whose specific type is unknown and may change at runtime. The dynamic type of that stored value, which we refer to as the existential's *underlying type*, is known only by the set of protocols it conforms to and, potentially, its superclass. While existential types are useful for expressing values of dynamic type, they are necessarily restricted because of their dynamic nature. Recent proposals have made [existential types more explicit](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md) to help developers understand this dynamic nature, as well as [making existential types more expressive](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md) by removing a number of limitations. However, a fundamental issue with existential types remains, that once you have a value of existential type it is *very* hard to use generics with it. Developers usually encounter this via the error message "protocol 'P' as a type cannot conform to itself": + +```swift +protocol P { + associatedtype A + func getA() -> A +} + +func takeP(_ value: T) { } + +func test(p: any P) { + takeP(p) // error: protocol 'P' as a type cannot conform to itself +} +``` + +This interaction with the generics system makes existentials a bit of a trap in Swift: it's easy to go from generics to existentials, but once you have an existential it is very hard to go back to using it generically. At worst, you need to go back through many levels of functions, changing their parameters or results from `any P` to being generic over `P`, or writing a custom [type eraser](https://www.swiftbysundell.com/articles/different-flavors-of-type-erasure-in-swift/). + +This proposal addresses this existential trap by allowing one to "open" an existential value, binding a generic parameter to its underlying type. Doing so allows us to call a generic function with an existential value, such that the generic function operates on the underlying value of the existential rather than on the existential box itself, making it possible to get out of the existential trap without major refactoring. This capability already exists in the language when accessing a member of an existential (e.g., `p.getA()`), and this proposal extends that behavior to all call arguments in a manner that is meant to be largely invisible: calls to generic functions that would have failed (like `takeP(p)` above) will now succeed. Smoothing out this interaction between existentials and generics can simplify Swift code and make the language more approachable. + +Swift-evolution thread: [Pitch #1](https://forums.swift.org/t/pitch-implicitly-opening-existentials/55412), [Pitch #2](https://forums.swift.org/t/pitch-2-implicitly-opening-existentials/56360) + +## Proposed solution + +To make it easier to move from existentials back to the more strongly-typed generics, we propose to implicitly *open* an existential value when it is passed to a parameter of generic type. In such cases, the generic argument refers to the *underlying* type of the existential value rather than the existential "box". Let's start with a protocol `Costume` that involves `Self` requirements, and write a generic function that checks some property of a costume: + +```swift +protocol Costume { + func withBells() -> Self + func hasSameAdornments(as other: Self) -> Bool +} + +// Okay: generic function to check whether adding bells changes anything +func hasBells(_ costume: C) -> Bool { + return costume.hasSameAdornments(as: costume.withBells()) +} +``` + +This is fine. However, let's write a function that makes sure every costume has bells for the big finale. We run into problems at the boundary between the array of existential values and our generic function: + +```swift +func checkFinaleReadiness(costumes: [any Costume]) -> Bool { + for costume in costumes { + if !hasBells(costume) { // error: protocol 'Costume' as a type cannot conform to the protocol itself + return false + } + } + + return true +} +``` + +In the call to `hasBells`, the generic parameter `C` is getting bound to the type `any Costume`, i.e., a box that contains a value of some unknown underlying type. Each instance of that box type might have a different type at runtime, so even though the underlying type conforms to `Costume`, the box does not. That box itself does not conform to `Costume` because it does not meet the requirement for `hasSameAdornments`., i.e., two boxes aren't guaranteed to store the same the same underlying type. + +This proposal introduces implicitly opened existentials, which allow one to use a value of existential type (e.g., `any Costume`) where its underlying type can be captured in a generic parameter. For example, the call `hasBells(costume)` above would succeed, binding the generic parameter `C` to the underlying type of that particular instance of `costume`. Each iteration of the loop could have a different underlying type bound to `C`: + +```swift +func checkFinaleReadiness(costumes: [any Costume]) -> Bool { + for costume in costumes { + if !hasBells(costume) { // okay with this proposal: C is bound to the type stored inside the 'any' box, known only at runtime + return false + } + } + + return true +} +``` + +Implicitly opening existentials allows one to take a dynamically-typed value and give its underlying type a name by binding it to a generic parameter, effectively moving from a dynamically-typed value to a more statically-typed one. This notion isn't actually new: calling a member of a protocol on a value of existential type implicitly "opens" the `Self` type. In the existing language, one could implement a shim for `hasBells` as a member of a protocol extension: + +```swift +extension Costume { + var hasBellsMember: Bool { + hasBells(self) + } +} + +func checkFinaleReadinessMember(costumes: [any Costume]) -> Bool { + for costume in costumes { + if !costume.hasBellsMember { // okay today: 'Self' is bound to the type stored inside the 'any' box, known only at runtime + return false + } + } + + return true +} +``` + +In that sense, implicitly opening existentials for calls to generic functions is a generalization of this existing behavior to all generic parameters. It isn't strictly more expressive: as the `hasBellsMember` example shows, one *can* always write a member in a protocol extension to get this opening behavior. This proposal aims to make implicit opening of existentials more uniform and more ergonomic, by making it more general. + +Let's consider one last implementation of our "readiness" check, where want to "open code" the check for bells without putting the logic into a separate generic function `hasBells`: + +```swift +func checkFinaleReadinessOpenCoded(costumes: [any Costume]) -> Bool { + for costume in costumes { + let costumeWithBells = costume.withBells() // returned type is 'any Costume' + if !costume.hasSameAdornments(costumeWithBells) { // error: 'any Costume' isn't necessarily the same type as 'any Costume' + return false + } + } + + return true +} +``` + +There are two things to notice here. First, the method `withBells()` returns type `Self`. When calling that method on a value of type `any Costume`, the concrete result type is not known, so it is type-erased to `any Costume` (which becomes the type of `costumeWithBells`). Second, on the next line, the call to `hasSameAdornments` produces a type error because the function expects a value of type `Self`, but there is no statically-typed link between `costume` and `costumeWithBells`: both are of type `any Costume`. Implicit opening of existential arguments only occurs in calls, so that its effects can be type-erased at the end of the call. To have the effects of opening persist over multiple statements, factor that code out into a generic function that gives a name to the generic parameter, as with `hasBells`. + +### Moving between `any` and `some` + +One of the interesting aspects of this proposal is that it allows one to refactor `any` parameters into `some` parameters (as introduced by [SE-0341](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md)) without a significant effect on client code. Let's rewrite our generic `hasBells` function using `some`: + +```swift +func hasBells(_ costume: some Costume) -> Bool { + return costume.hasSameAdornments(as: costume.withBells()) +} +``` + +With this proposal, we can now call `hasBells` given a value of type `any Costume`: + +```swift +func isReadyForFinale(_ costume: any Costume) -> Bool { + return hasBells(costume) // implicit opening of the existential value +} +``` + +It's always the case that one can go from a statically-typed `some Costume` to an `any Costume`. This proposal also allows one to go the other way, opening up an `any Costume` into a `some Costume` parameter. Therefore, with this proposal, we could refactor `isReadyForFinale` to make it generic via `some`: + +```swift +func isReadyForFinale(_ costume: some Costume) -> Bool { + return hasBells(costume) // okay, `T` binds to the generic argument +} +``` + +Any callers to `isReadyForFinale` that provided concrete types now avoid the overhead of "boxing" their type in an `any Costume`, and any callers that provided an `any Costume` will now implicitly open up that existential in the call to `isReadyForFinale`. This allows existential operations to be migrated to generic ones without having to also make all clients generic at the same time, offering an incremental way out of the "existential trap". + +## Detailed design + +Fundamentally, opening an existential means looking into the existential box to find the dynamic type stored within the box, then giving a "name" to that dynamic type. That dynamic type name needs to be captured in a generic parameter somewhere, so it can be reasoned about statically, and the value with that type can be passed along to the generic function being called. The result of such a call might also refer to that dynamic type name, in which case it has to be erased back to an existential type. The After the call, any values described in terms of that dynamic type opened existential type has to be type-erased back to an existential so that the opened type name doesn't escape into the user-visible type system. This both matches the existing language feature (opening an existential value when accessing one of its members) and also prevents this feature from constituting a major extension to the type system itself. + +This section describes the details of opening an existential and then type-erasing back to an existential. These details of this change should be invisible to the user, and manifest only as the ability to use existentials with generics in places where the code would currently be rejected. However, there are a *lot* of details, because moving from dynamically-typed existential boxes to statically-typed generic values must be carefully done to maintain type identity and the expected evaluation semantics. + +### When can we open an existential? + +To open an existential, the argument (or source) must be of existential type (e.g., `any P`) or existential metatype (e.g., `any P.Type`) and must be provided to a parameter (or target) whose type involves a generic parameter that can bind directly to the underlying type of the existential. This means that, for example, we can open an existential when its underlying type would directly bind to a generic parameter: + +```swift +protocol P { + associatedtype A + + func getA() -> A +} + +func openSimple(_ value: T) { } + +func testOpenSimple(p: any P) { + openSimple(p) // okay, opens 'p' and binds 'T' to its underlying type +} +``` + +It's also possible to open an `inout` parameter. The generic function will operate on the underlying type, and can (e.g.) call `mutating` methods on it, but cannot change its *dynamic* type because it doesn't have access to the existential box: + +```swift +func openInOut(_ value: inout T) { } +func testOpenInOut(p: any P) { + var mutableP: any P = p + openInOut(&mutableP) // okay, opens to 'mutableP' and binds 'T' to its underlying type +} +``` + +However, we cannot open when there might be more than one value of existential type or no values at all, because we need to be guaranteed to have a single underlying type to infer. Here are several such examples where the generic parameter is used in multiple places in a manner that prevents opening the existential argument: + +```swift +func cannotOpen1(_ array: [T]) { .. } +func cannotOpen2(_ a: T, _ b: T) { ... } +func cannotOpen3(_ values: T...) { ... } + +struct X { } +func cannotOpen4(_ x: X) { } + +func cannotOpen5(_ x: T, _ a: T.A) { } + +func cannotOpen6(_ x: T?) { } + +func testCannotOpenMultiple(array: [any P], p1: any P, p2: any P, xp: X, pOpt: (any P)?) { + cannotOpen1(array) // each element in the array can have a different underlying type, so we cannot open + cannotOpen2(p1, p2) // p1 and p2 can have different underlying types, so there is no consistent binding for 'T' + cannotOpen3(p1, p2) // similar to the case above, p1 and p2 have different types, so we cannot open them + cannotOpen4(xp) // cannot open the existential in 'X' there isn't a specific value there. + cannotOpen5(p1, p2.getA()) // cannot open either argument because 'T' is used in both parameters + cannotOpen6(pOpt) // cannot open the existential in '(any P)?' because it might be nil, so there would not be an underlying type +} +``` + +The case of optionals is somewhat interesting. It's clear that the call `cannotOpen6(pOpt)` cannot work because `pOpt` could be `nil`, in which case there is no type to bind `T` to. We *could* choose to allow opening a non-optional existential argument when the parameter is optional, e.g., + +```swift +cannotOpen6(p1) // we *could* open here, binding T to the underlying type of p1, but choose not to +``` + +but this proposal doesn't allow this because it would be odd to allow this call but not the `cannotOpen6(pOpt)` call. + +A value of existential metatype can also be opened, with the same limitations as above. + +```swift +func openMeta(_ type: T.Type) { } + +func testOpenMeta(pType: any P.Type) { + openMeta(pType) // okay, opens 'pType' and binds 'T' to its underlying type +} +``` + +### Type-erasing resulting values + +The result type of a generic function can involve generic parameters and their associated types. For example, here's a generic function that returns the original value and some values of its associated types: + +```swift +protocol Q { + associatedtype B: P + func getB() -> B +} + +func decomposeQ(_ value: T) -> (T, T.B, T.B.A) { + (value, value.getB(), value.getB().getA()) +} +``` + +When calling `decomposeQ` with an existential value, the existential is opened and `T` will bind to its underlying type. `T.B` and `T.B.A` are types derived from that underlying type. Once the call completes, however, the types `T`, `T.B`, and `T.B.A` are *type-erased* to their upper bounds, i.e., the existential type that captures all of their requirements. For example: + +```swift +func testDecomposeQ(q: any Q) { + let (a, b, c) = decomposeQ(q) // a is any Q, b is any P, c is Any +} +``` + +This is identical to the [covariant erasure of associated types described in SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md#covariant-erasure-for-associated-types), and the rules specified there apply equally here. We can restate those requirements more generally for an arbitrary generic parameter as: + +When binding a generic parameter `T` to an opened existential, `T`, `T` and `T`-rooted associated types that + +- are **not** bound to a concrete type, and +- appear in covariant position within the result type of the generic function + +will be type-erased to their upper bounds as per the generic signature of the existential that is used to access the member. The upper bounds can be either a class, protocol, protocol composition, or `Any`, depending on the *presence* and *kind* of generic constraints on the associated type. + +When `T` or a `T`-rooted associated type appears in a non-covariant position in the result type, `T` cannot be bound to the underlying type of an existential value because there would be no way to represent the type-erased result. This is essentially the same property as described for the parameter types that prevents opening of existentials, as described above. For example: + +```swift +func cannotOpen7(_ value: T) -> X { /*...*/ } +``` + +However, because the return value is permitted a conversion to erase to an existential type, optionals, tuples, and even arrays *are* permitted: + +```swift +func openWithCovariantReturn1(_ value: T) -> T.B? { /*...*/ } +func openWithCovariantReturn2(_ value: T) -> [T.B] { /*...*/ } + +func covariantReturns(q: any Q){ + let r1 = openWithCovariantReturn1(q) // okay, 'T' is bound to the underlying type of 'q', resulting type is 'any P' + let r2 = openWithCovariantReturn2(q) // okay, 'T' is bound to the underlying type of 'q', resulting type is '[any Q]' +} +``` + +### "Losing" constraints when type-erasing resulting values + +When the result of a call involving an opened existential is type-erased, it is possible that some information about the returned type cannot be expressed in an existential type, so the "upper bound" described above will lose information. For example, consider the type of `b` in this example: + +```swift +protocol P { + associatedtype A +} + +protocol Q { + associatedtype B: P where B.A == Int +} + +func getBFromQ(_ q: T) -> T.B { ... } + +func eraseQAssoc(q: any Q) { + let b = getBFromQ(q) +} +``` + +When type-erasing `T.B`, the most specific upper bound would be "a type that conforms to `P` where the associated type `A` is known to be `Int`". However, Swift's existential types cannot express such a type, so the type of `b` will be the less-specific `any P`. + +It is likely that Swift's existentials will grow in expressivity over time. For example, [SE-0353 "Constrained Existential Types"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0353-constrained-existential-types.md) allows one to express existential types that involve bindings for [primary associated types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md). If we were to adopt that feature for protocol `P`, the most specific upper bound would be expressible: + +```swift +// Assuming SE-0353... +protocol P
{ + associatedtype A +} + +// ... same as above ... +``` + +Now, `b` would be expected to have the type `any P`. Future extensions of existential types might make the most-specific upper bound expressible even without any source code changes, and one would expect that the type-erasure after calling a function with an implicitly-opened existential would become more precise when those features are added. + +However, this kind of change presents a problem for source compatibility, because code might have come to depend on the type of `b` being the less-precise `any P` due to, e.g., overloading: + +```swift +func f(_: T) -> Int { 17 } +func f(_: T) -> Double where T.A == Int { 3.14159 } + +// ... +func eraseQAssoc(q: any Q) { + let b = getBFromQ(q) + f(b) +} +``` + +With the less-specific upper bound (`any P`), the call `f(b)` would choose the first overload that returns an `Int`. With the more-specific upper bound (`any P` where `A` is known to be `Int`), the call `f(b)` would choose the second overload that returns a `Double`. + +Due to overloading, the source-compatibility impacts of improving the upper bound cannot be completely eliminated without (for example) holding the upper bound constant until a new major language version. However, we propose to mitigate the effects by requiring a specific type coercion on any call where the upper bound is unable to express some requirements due to limitations on existentials. Specifically, the call `getBFromQ(q)` would need to be written as: + +```swift +getBFromQ(q) as any P +``` + +This way, if the upper bound changes due to increased expressiveness of existential types in the language, the overall expression will still produce a value of the same type---`any P`---as it always has. A developer would be free to remove the `as any P` at the point where Swift can fully capture all of the information known about the type in an existential. + +Note that this requirement for an explicit type coercion also applies to all type erasure due to existential opening, including ones that existed prior to this proposal. For example, `getBFromQ` could be written as a member of a protocol extension. The code below has the same issues (and the same resolution) as our example, as was first made well-formed with [SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md): + +```swift +extension Q { + func getBFromQ() -> B { ... } +} + +func eraseQAssocWithSE0309(q: any Q) { + let b = q.getBFromQ() +} +``` + +### Contravariant erasure for parameters of function type + +While covariant erasure applies to the result type of a generic function, the opposite applies to other parameters of the generic function. This affects parameters of function type that reference the generic parameter binding to the opened existential, which will be type-erased to their upper bounds. For example: + +```swift +func acceptValueAndFunction(_ value: T, body: (T) -> Void) { ... } + +func testContravariantErasure(p: any P) { + acceptValueAndFunction(p) { innerValue in // innerValue has type 'any P' + // ... + } +} +``` + +Like the covariant type erasure applied to result types, this type erasure ensures that the "name" assigned to the dynamic type doesn't escape into the user-visible type system through the inferred closure parameter. It effectively maintains the illusion that the generic type parameter `T` is binding to `any P`, while in fact it is binding to the underlying type of that specific value. + +There is one exception to this rule: if the argument to such a parameter is a reference to a generic function, the type erasure does not occur. In such cases, the dynamic type name is bound directly to the generic parameter of this second generic function, effectively doing the same implicit opening of existentials again. This is best explained by example: + +```swift +func takeP(_: U) -> Void { ... } + +func implicitOpeningArguments(p: any P) { + acceptValueAndFunction(p, body: takeP) // okay: T and U both bind to the underlying type of p +} +``` + +This behavior subsumes most of the behavior of the hidden `_openExistential` operation, which specifically only supports opening one existential value and passing it to a generic function. `_openExistential` might still have a few scattered use cases when opening an existential that doesn't have conformance requirements on it. + + ### Order of evaluation restrictions + +Opening an existential box requires evaluating that the expression that produces that box and then peering inside it to extract its underlying type. The evaluation of the expression might have side effects, for example, if one calls the following `getP()` function to produce a value of existential box type `any P`: + +```swift +extension Int: P { } + +func getP() -> any P { + print("getP()") + return 17 +} +``` + +Now consider a generic function for which we want open an existential argument: + +```swift +func acceptFunctionStringAndValue(body: (T) -> Void, string: String, value: T) { ... } + +func hello() -> String { + print("hello()") +} + +func implicitOpeningArgumentsBackwards() { + acceptFunctionStringAndValue(body: takeP, string: hello(), value: getP()) // will be an error, see later +} +``` + +Opening the argument to the `value` parameter requires performing the call to `getP()`. This has to occur *before* the argument to the `body` parameter can be formed, because `takeP`'s generic type parameter `U` is bound to the underlying type of that existential box. Doing so means that the program would produce side effects in the following order: + +``` +getP() +hello() +``` + +However, this would contradict Swift's longstanding left-to-right evaluation order. Rather than do this, we instead place another limitation on the implicit opening of existentials: an existential argument cannot be opened if the generic type parameter bound to its underlying type is used in any function parameter preceding the one corresponding to the existential argument. In the `implicitOpeningArgumentsBackwards` above, the call to `acceptFunctionStringAndValue` does not permit opening the existential argument to the `value` parameter because its generic type parameter, `T`, is also used in the `body` parameter that precedes `value`. This ensures that the underlying type is not needed for any argument prior to the opened existential argument, so the left-to-right evaluation order is maintained. + +### Avoid opening when the existential type satisfies requirements (in Swift 5) + +As presented thus far, opening of existential values can change the behavior of existing programs that relied on passing the existential box to a generic function. For example, consider the effect of passing an existential box to an unconstrained generic function that puts the parameter into the returned array: + +```swift +func acceptsBox(_ value: T) -> Any { [value] } + +func passBox(p: any P) { + let result = acceptsBox(p) // currently infers 'T' to be 'any P', returns [any P] + // unrestricted existential opening would infer 'T' to be the underlying type of 'p', returns [T] +} +``` + +Here, the dynamic type of the result of `acceptsBox` would change if the existential box is opened as part of the call. The change itself is subtle, and would not be detected until runtime, which could cause problems for existing Swift programs that rely on binding generic parameters. Therefore, in Swift 5, this proposal prevents opening of existential values when the existential types themselves would satisfy the conformance requirements of the corresponding generic parameter, making it a strictly additive change: calls to generic functions with existential values that previously worked will continue to work with the same semantics, but calls that didn't work before will open the existential and can therefore succeed. + +Most of the cases in today's Swift where a generic parameter binds to an existential type succeed because there are no conformance requirements on the generic parameter, as with the `T` generic parameter to `acceptsBox`. For most protocols, an existential referencing the corresponding type does not conform to that protocol, i.e., `any Q` does not conform to `Q`. However, there are a small number of exceptions: + +* The existential type `any Error` conforms to the `Error` protocol, as specified in [SE-0235](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0235-add-result.md#adding-swifterror-self-conformance). +* An existential type `any Q` of an `@objc` protocol `Q`, where `Q` contains no `static` requirements, conforms to `Q`. + +For example, consider an operation that takes an error. Passing a value of type `any Error` to it succeeds without opening the existential: + +```swift +func takeError(_ error: E) { } + +func passError(error: any Error) { + takeError(error) // okay without opening: 'E' binds to 'any Error' because 'any Error' conforms to 'Error' +} +``` + +This proposal preserves the semantics of the call above by not opening the existential argument in cases where the existential type satisfies the corresponding generic parameter's conformance requirements according to the results above. Should Swift eventually grow a mechanism to make existential types conform to protocols (e.g., so that `any Hashable` conforms to `Hashable`), then such conformances will **not** suppress implicit opening, because any code that made use of these conformances would be newly-valid code and would start with implicit-opening semantics. + +Swift 6 will be a major language version change that can incorporate some semantics- and source-breaking changes. In Swift 6, the suppression mechanism described in this section will *not* apply, so the `passBox` example above would open the value of `p` and bind `T` to that opened existential type. This provides a more consistent semantics that, additionally, subsumes all of the behavior of `type(of:)` and the hidden `_openExistential` operation. + +### Suppressing explicit opening with `as any P` / `as! any P` + +If for some reason one wants to suppress the implicit opening of an existential value, one can explicitly write a coercion or forced cast to an existential type directly on the call argument. For example: + +```swift +func f1(_: T) { } // #1 +func f1(_: T) { } // #2 + +func test(p: any P) { + f1(p) // opens p and calls #1, which is more specific + f1(p as any P) // suppresses opening of 'p', calls #2 which is the only valid candidate + f1((p as any P)) // parentheses disable this suppression mechanism, so this opens p and calls #1 +} +``` + +Given that implicit opening of existentials is defined to occur in those cases where a generic function would not otherwise be callable, this suppression mechanism should not be required often in Swift 5. In Swift 6, where implicit opening will be more eagerly performed, it can be used to provide the Swift 5 semantics. + +An extra set of parentheses will disable this suppression mechanism, which can be important when `as any P` is required for some other reason. For example, because it acknowledges when information is lost from the result type due to type erasure. This can help break ambiguities when both meanings of `as` could apply: + +```swift +protocol P { + associatedtype A +} +protocol Q { + associatedtype B: P where B.A == Int +} + +func getP(_ p: T) +func getBFromQ(_ q: T) -> T.B { ... } + +func eraseQAssoc(q: any Q) { + getP(getBFromQ(q)) // error, must specify "as any P" due to loss of constraint T.B.A == Int + getP(getBFromQ(q) as any P) // suppresses error above, but also suppresses opening, so it produces + // error: now "any P does not conform to P" and op + getP((getBFromQ(q) as any P)) // okay! original error message should suggest this +} + +``` + +## Source compatibility + +This proposal is defined specifically to avoid most impacts on source compatibility, especially in Swift 5. Some calls to generic functions that would previously have been ill-formed (e.g., they would fail because `any P` does not conform to `P`) will now become well-formed, and existing code will behavior in the same manner as before. As with any such change, it's possible that overload resolution that would have succeeded before will continue to succeed but will now pick a different function. For example: + +```swift +protocol P { } + +func overloaded1(_: T, _: U) { } // A +func overloaded1(_: Any, _: U) { } // B + +func changeInResolution(p: any P) { + overloaded1(p, 1) // used to choose B, will choose A with this proposal +} +``` + +Such examples are easy to construct in the abstract for any feature that makes ill-formed code well-formed, but these examples rarely cause problems in practice. + +## Effect on ABI stability + +This proposal changes the type system but has no ABI impact whatsoever. + +## Effect on API resilience + +This proposal changes the use of APIs, but not the APIs themselves, so it doesn't impact API resilience per se. + +## Alternatives considered + +This proposal opts to open existentials implicitly and locally, type-erasing back to existentials after the immediate call, as a generalization of opening when using a member of an existential value. There are alternative designs that are explicit or open the existential more broadly, with different tradeoffs. + +### Explicitly opening existentials + +This proposal implicitly opens existentials at call sites. Instead, we could provide an explicit syntax for opening an existential, e.g., via [an `as` coercion to `some P`](https://forums.swift.org/t/pitch-implicitly-opening-existentials/55412/8). For example, + +```swift +protocol P { + associatedtype A +} + +func takesP(_ value: T) { } + +func hasExistentialP(p: any P) { + takesP(p) // error today ('any P' does not conform to 'P'), would be well-formed with implicit opening +} +``` + +could be written to explicitly open the existential, e.g., + +```swift +func hasExistentialP(p: any P) { + takesP(p) // error today ('any P' does not conform to 'P'), would still be an error + takesP(p as some P) // explicitly open the existential +} +``` + +There are two advantages to this approach over the implicit opening in this proposal. The first is that it is a purely additive feature and completely opt-in feature, which one can read and reason about when it is encountered in source code. The second is that the opened existential could persist throughout the body of the function. This would allow one to write the "open-coded" finale check from earlier in the proposal without having to factor the code into a separate (generic) function: + +```swift +func checkFinaleReadinessOpenCoded(costumes: [any Costume]) -> Bool { + for costume in costumes { + let openedCostume = costume as some Costume // type is "opened type of costume at this point" + let costumeWithBells = openedCostume.withBells() // returned type is the same as openedCostume + if !openedCostume.hasSameAdornments(costumeWithBells) { // okay, both types are known to be the same + return false + } + } + + return true +} +``` + +The type of `openedCostume` is based on the dynamic type of the the value in the variable `costume` at the point where the `as some Costume` expression occurred. That type must not be allowed to "escape" the scope where the value is created, which implies several restrictions: + +* Only non-`static` local variables can have opened existential type. Any other kind of variable can be referenced at some later point in time where the dynamic type might have changed. +* A value of opened existential type cannot be returned from a function that has an opaque result type (e.g., `some P`), because then the underlying type of the opaque type would be dependent on runtime values provided to the function. + +Additionally, having an explicit opening expression means that opened existential types become part of the user-visible type system: the type of `openedCostume` can only be reasoned about based on its constraints (`P`) and the location in the source code where the expression occurred. Two subsequent openings of the same variable would produce two different types: + +```swift +func f(eq: any Equatable) { + let x1 = eq as some Equatable + if x1 == x1 { ... } // okay + + let x2 = eq as some Equatable + if x1 == x2 { ... } // error: "eq as some Equatable" produces different types in x1 and x2 +} +``` + +An explicit opening syntax is more expressive within a single function than the proposed implicit opening, because one can work with different values that are statically known to be derived from the same opened existential without having to introduce a new generic function to do so. However, this explicitness comes with a corresponding increase in the surface area of the language: not only the expression that performs the explicit opening (`as some P`), but the notion of opened types in the type system, which has heretofore been an implementation detail of the compiler not exposed to users. + +In contrast, the proposed implicit opening improves the expressivity of the language without increasing it's effective surface area. The opening is implicit, and the opened types remain an implementation detail. + +This "alternative Considered" could perhaps be expressed as a potential future direction. Nothing in this proposal prevents us from adding explicitly opened existentials in the future, should they prove to be useful, and we would still want the implicitly opening with type erasure as described in this proposal. Should that happen, the implicit behavior in this proposal could be retroactively understood as inferring something that could be written in the explicit syntax: + +```swift +protocol Q { } + +protocol P { + associatedtype A: Q +} + +func getA(_ value: T) -> T.A { ... } + +func unwrap(p: any P) { + let a = getA(p) // implicitly the same as "getA(p as some P) as any Q" +} +``` + +### Value-dependent opening of existentials + +Implicit opening in this proposal is always scoped to a particular binding of a specific generic parameter (`T`) and is erased thereafter. For example, this means that two invocations of the same generic function on the same existential value will return values of existential type that are not (statically) known to be equivalent: + +```swift +func identity(_ value: T) -> T { value } +func testIdentity(p: any Equatable) { + let p1 = identity(p) // p1 gets type-erased type 'any Equatable' + let p2 = identity(p) // p2 gets type-erased type 'any Equatable' + if p1 == p2 { ... } // error: p1 and p2 aren't known to have the same concrete type + + let openedP1: some P = identity(p) // openedP1 has an opaque type binding to the underlying type of the call + let openedP2: some P = identity(p) // openedP2 has an opaque type binding to the underlying type of the call + if openedP1 == openedP2 { ... } // error: openedP1 and openedP2 aren't known to have the same concrete type +} +``` + +One could imagine tying the identity of the opened existential type to the *value* of the existential. For example, the two calls to `identity(p)` could produce opaque types that are identical because they are based on the underlying type of the value `p`. This is a form of dependent typing, because the (static) types of some entities are determined by their values. It begins to break down if there is any way in which the value can change, e.g., + +```swift +func identityTricks(p: any Equatable) { + let openedP1 = identity(p) // openedP1 has the underlying type of 'p' + let openedP2 = identity(p) // openedP2 has the underlying type of 'p' + if openedP1 == openedP2 { ... } // okay because both values have the underlying type of 'p' + + var q = p // q has the underlying type of 'p' + let openedQ1: some P = identity(q) // openedQ1 has the underlying type of 'q' and therefore 'p' + if openedP1 == openedQ1 { ... } // okay because both values have the underlying type of 'p' + + if condition { + q = 17 // different underlying type for 'q' + } + + let openedQ2: some P = identity(q) + if openedQ1 == openedQ2 { } // error: openedQ1 has the underlying type of 'p', but + // openedQ2 has the underlying type of 'q', which now might be different from 'p' +} +``` + +This approach is much more complex because it introduces value tracking into the type system (where was this existential value produced?), at which point mutations to variables can affect the static types in the system. + +## Revisions + +Fifth revision: + +* Note that parentheses disable the `as any P` suppression mechanism, avoiding the problem where `as any P` is both required (because type erasure lost information from the return type) and also has semantic effect (suppressing opening). + +Fourth revision: + +* Add discussion about type erasure losing constraints and the new requirement to introduce an explicit `as` coercion when the upper bound loses information. + +Third revision: + +* Only apply the source-compatibility rule, which avoids opening an existential argument when the existential box would have sufficed, in Swift 5. In Swift 6, we will open the existential argument whenever we can, providing a consistent and desirable semantics. +* Re-introduce `as any P` and `as! any P` , now that they will be useful in Swift 6. +* Clarify more about the relationship to the explicit opening syntax, which could also be a future direction. + +Second revision: + +* Remove the discussion about `type(of:)`, whose special behavior is no longer subsumed by this proposal. Weaken statements about fully subsuming `_openExistential`. +* Removed `as any P` and `as! any P` as syntaxes to suppress the implicit opening of an existential value. It isn't needed given that we only open when the existential type doesn't meet the generic function's constraints. + +First revision: + +* Describe contravariant erasure for parameters +* Describe the limitation on implicit existential opening to maintain order of evaluation +* Avoid opening an existential argument when the existential type already satisfies the conformance requirements of the corresponding generic parameter, to better maintain source compatibility +* Introduce `as any P` and `as! any P` as syntaxes to suppress the implicit opening of an existential value. +* Added discussion on the relationship with `some` parameters ([SE-0341](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md)). +* Expand discussion of an explicit opening syntax. + +## Acknowledgments + +This proposal builds on the difficult design work of [SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md), which charted most of the detailed semantics for working with values of existential type and dealing with (e.g.) covariant erasure and the restrictions that must be placed on opening existentials. Moreover, the implementation work from one of SE-0309's authors, [Anthony Latsis](https://github.com/AnthonyLatsis), formed the foundation of the implementation work for this feature, requiring only a small amount of generalization. Ensan highlighted the issue with losing information in upper bounds and [suggested an approach](https://forums.swift.org/t/se-0352-implicitly-opened-existentials/56557/7) similar to what is used here. diff --git a/proposals/0353-constrained-existential-types.md b/proposals/0353-constrained-existential-types.md new file mode 100644 index 0000000000..44a784e86e --- /dev/null +++ b/proposals/0353-constrained-existential-types.md @@ -0,0 +1,231 @@ +# Constrained Existential Types + +* Proposal: [SE-0353](0353-constrained-existential-types.md) +* Authors: [Robert Widmann](https://github.com/codafi) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Implementation: implemented in `main` branch, under flag `-enable-parameterized-existential-types` +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0353-constrained-existential-types/57560) + +## Introduction + +Existential types complement the Swift type system’s facilities for abstraction. Like generics, they enable a function to take and return multiple possible types. Unlike generic parameter types, existential types need not be known up front when passed as inputs to a function. Further, concrete types can be *erased* (hidden behind the interface of a protocol) when returned from a function. There has been a flurry of activity in this space with[SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md#covariant-erasure-for-associated-types) unblocking the remaining restrictions on using protocols with associated types as existential types, and [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md) paving the way for a lightweight constraint syntax for the associated types of protocols. Building directly upon those ideas, this proposal seeks to re-use the syntax of lightweight associated type constraints in the context of existential types. + +```swift +any Collection +``` + +In essence, this proposal seeks to provide the same expressive power that [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md) gives to `some` types to `any` types. + +Swift-evolution pitch thread: https://forums.swift.org/t/pitch-constrained-existential-types/56361 + +## Motivation + +Though [SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md#covariant-erasure-for-associated-types) provides the ability to use protocols with associated types freely, it does not leave any room for authors to further constrain the associated types of those protocols, creating a gap in expressiveness between generics and existentials. Consider the implementation of a type-erased stack of event producers and consumers: + +```swift +protocol Producer { + associatedtype Event + + func poll() -> Self.Event? +} + +protocol Consumer { + associatedtype Event + + func respond(to event: Self.Event) +} +``` + +If a hypothetical event system type wishes to accept an arbitrary mix of `Producer`s and an arbitrary mix of `Consumer`s, it is free to do so with existential types: + +```swift +struct EventSystem { + var producers: [any Producer] + var consumers: [any Consumer] + + mutating func add(_ producer: any Producer) { + self.producers.append(producer) + } +} +``` + +However, we run into trouble when trying to compose producers and consumers with one another. As any given `Producer` yields data of an unspecified and unrelated `Event` type when `poll`’ed, Swift will (rightly) tell us that none of our consumers can safely accept any events. One solution would be to make `EventSystem` generic over the type of events and require `Producer` and `Consumer` instances to only return those events. As it stands, this also means restricting the producers and consumers to be concrete, with the added downside of requiring us to homogenize their types - ad-hoc type erasure strikes again: + +```swift +struct EventSystem { + var producers: [AnyProducer] + var consumers: [AnyConsumer] + + mutating func add(_ producer: P) + where P.Event == Event + { + self.producers.append(AnyProducer(erasing: producer)) + } +} +``` + +In this example, we have sacrificed quite a lot for type safety - and also have to maintain two extra type erasing wrappers for producers and consumers. Really, what is missing is the ability to express the fact that the producer and consumer types don’t matter (existential types) but the data they operate on *does* (generic constraints). This is where constrained existential types shine. When combined with the power of primary associated types from [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md), it allows us to write the code we wanted to in the first place: + +```swift +struct EventSystem { + var producers: [any Producer] + var consumers: [any Consumer] + + mutating func add(_ producer: any Producer) { + self.producers.append(producer) + } +} +``` + +## Proposed solution + +Existential types will be augmented with the ability to specify constraints on their primary associated types. When an existential type appears with such constraints, they will be converted into same-type requirements. + +```swift +protocol P { } + +var xs: [any P] // "Equivalent" to [any P] where P.T == B, P.U == N, P.V == J +``` + +## Detailed design + +The syntax of existential types will be updated to accept constraint clauses. Type inference procedures will be updated to apply inference rules to generic parameters appearing as part of parameterized existential types. + +The Swift type system and runtime will accept casts from parameterized existential types to non-parameterized existential types and vice versa, as well as casts that refine any constrained primary associated types. Upcasts and downcasts to, from, and between existential types will be updated to take these additional constraints into account: + +```swift +var x: any Sequence +_ = x as any Sequence // trivially true +_ = x as! any Sequence // requires examining Sequence.Element at runtime +``` + +### Equality of constrained protocol types + +The language must define when two types that are derived differently in code are in fact the same type. In principle, it would make sense to say that two constrained protocol types are the same if and only if they have exactly the same set of possible conforming types. Unfortunately, this rule is impractical in Swift’s type system for complex technical reasons. This means that some constrained protocol types which are logically equivalent to each other will be considered different types in Swift. + +The exact rule is still being determined, but for example, it is possible that the type `any P & Q` might be considered different from the type `any P & Q` even if the associated types of these protocols are known to be equal. Because these types have equivalent logical content, however, there will be an implicit conversion between them in both directions. As a result, this is not expected to pose a large practical difficulty. + +Substitutions of constrained protocol types written with the same basic “shape”, such as `any P`and `any P` in a generic context where `T == Int`, will always be the same type. + +### Variance + +One primary use-case for constrained existential types is their the Swift Standard Library’s Collection types. The Standard Library’s *concrete* collection types have built-in support for covariant coercions. For example, + +```swift +func up(from values: [NSView]) -> [Any] { return values } +``` + +At first blush, it would seem like constrained existential types should support variance as well: + +```swift +func up(from values: any Collection) -> any Collection { return values } +``` + +But this turns out to be quite a technical feat. There is a naive implementation of this coercion that recasts the input collection as an `Array` of the appropriate type, but this would be deeply surprising and would bake the fact that `Array` is always returned into the ABI of the standard library forever. + +Constrained existential types will behave as normal generic types with respect to variance - that is, they are *invariant -* and the code above will be rejected. + +### Covariant Erasure with Constrained Existentials + +[SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md) specifies that one can use a member on a value of existential type only when references to `Self` or its associated types are in *covariant* positions, such as the return type of a method defined a protocol. In such positions, its associated type is *erased* to its upper bound, which is the type that most closely describes the capabilities of that associated type. For example, consider a use of the `first` property on a collection. + +```swift +extension Collection {} + var first: Element? { get } +} + +func test(collection: any Collection, stringCollection: any Collection) { + let x = collection.first // Previously an error. With SE-0309, erases to 'Any?' + let y = stringCollection.first // With SE-0309, relies on Element == String to produce 'String?' +} +``` + +However, when `Self` or its associated type occurs in an *invariant* position (defined in SE-0309), one cannot use the member of an existential type unless the concrete type is known. SE-0309 provides the following example: + +```swift +var collection: any RangeReplaceableCollection = [1, 2, 3] +// error: member 'append' cannot be used on value of protocol type 'RangeReplaceableCollection' +// because it references associated type 'Element' in contravariant position; use a conformance +// constraint instead. +collection.append(4) +``` + +With constrained existentials, one could append to a `RangeReplaceableCollection`: + +```swift +var intCollection: any RangeReplaceableCollection = [1, 2, 3] +collection.append(4) // okay: the Element type is concrete (Int) within the existential +``` + +The principle here is that a use of an associated type in the member is not considered invariant if that associated type has been made concrete by the existential type. This allows the use of `append` above, because `Element` has been made concrete by the type `any RangeReplaceableCollection`. Additionally, this means that the existential type that results from erasing an associated type can make use of constrained existentials. For example: + +```swift +extension Sequence { + func eagerFilter(_ isIncluded: @escaping (Element) -> Bool) -> [Element] { ... } + func lazyFilter(_ isIncluded: @escaping (Element) -> Bool) -> some Sequence { ... } +} + +func doFilter(sequence: any Sequence, intSequence: any Sequence) { + let e1 = sequence.eagerFilter { _ in true } // error: 'Element' is used in an invariant position + let e2 = intSequence.eagerFilter { _ in true } // okay, returns '[Int]' + let l1 = sequence.lazyFilter { _ in true } // error: 'Element' is used in an invariant position + let l2 = intSequence.lazyFilter { _ in true } // okay, returns 'any Sequence' +} +``` + +The same erasure effects with the implicitly opened existentials introduced in [SE-0352](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md). + +## Effect on ABI stability + +As constrained existential types are an entirely additive concept, there is no impact upon ABI stability. + +It is worth noting that this feature requires revisions to the Swift runtime and ABI that are not backwards-compatible nor backwards-deployable to existing OS releases. + +## Alternatives considered + +Aside from the obvious of not accepting this proposal, we could imagine many different kinds of spellings to introduce same-type requirements on associated types. For example, a where-clause based approach as in: + +```swift +any (Collection where Self.Element == Int) +``` + +Syntax like this is hard to read and use in context and the problem becomes worse as it is made to compose with other existential types and constraints. Further it would conflict with the overall direction that generic constraints in Swift are taking as of [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md). Generalized constraint syntaxes are out of scope for this proposal and are mentioned later as future directions. + +## Future directions + +#### Generalized Constraints + +This proposal intentionally does not take a position on the generalized constraint syntax considered during the review of [SE-0341](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md#constraining-the-associated-types-of-a-protocol). To take one spelling: + +```swift +any Collection<.Index == String.Index> +``` + +Though when and if such a syntax is available we expect it to apply to constrained existential types. Possible designs for generalized constraints on existential types are discussed in https://forums.swift.org/t/generalized-opaque-and-existential-type-constraints/55494. + +#### Opaque Constraints + +One particularly interesting construction is the composition of opaque types and constrained existential types. This combo allows for a particularly powerful form of type abstraction: + +```swift +any Collection +``` + +This type describes any value that implements the `Collection` protocol but whose element type is an opaque instance of the `View` protocol. Today, Swift’s generics system lacks the ability to express same-type constraints with opaque types as an operand. + +#### Even More Generalized Existentials + +Constraints on existing primary associated types are hardly the only thing existential types can express. Swift’s type system can be given the ability to open arbitrary (constrained) type parameters into scope via an existential. This enables not just top-level usages as in + +```swift +any Collection +``` + +But also nested usages as in + +```swift +any Collection Collection> +``` + +Essentially enabling ad-hoc abstraction over generic types of *any shape* at any point in the program. diff --git a/proposals/0354-regex-literals.md b/proposals/0354-regex-literals.md new file mode 100644 index 0000000000..752e22f085 --- /dev/null +++ b/proposals/0354-regex-literals.md @@ -0,0 +1,668 @@ +# Regex Literals + +* Proposal: [SE-0354](0354-regex-literals.md) +* Authors: [Hamish Knight](https://github.com/hamishknight), [Michael Ilseman](https://github.com/milseman), [David Ewing](https://github.com/DaveEwing) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Upcoming Feature Flag: `BareSlashRegexLiterals` (implemented in Swift 5.8) +* Implementation: [apple/swift#42119](https://github.com/apple/swift/pull/42119), [apple/swift#58835](https://github.com/apple/swift/pull/58835) + * Bare slash syntax `/.../` available with `-enable-bare-slash-regex` +* Review: ([first pitch](https://forums.swift.org/t/pitch-regular-expression-literals/52820)) + ([second pitch](https://forums.swift.org/t/pitch-2-regex-literals/56736)) + ([first review](https://forums.swift.org/t/se-0354-regex-literals/57037)) + ([revision](https://forums.swift.org/t/returned-for-revision-se-0354-regex-literals/57366)) + ([second review](https://forums.swift.org/t/se-0354-second-review-regex-literals/57367)) + ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0354-regex-literals/58537)) + +## Introduction + +We propose the introduction of regex literals to Swift source code, providing compile-time checks and typed-capture inference. Regex literals help complete the story told in *[Regex Type and Overview][regex-type]*. + +## Motivation + +In *[Regex Type and Overview][regex-type]* we introduced the `Regex` type, which is able to dynamically compile a regex pattern: + +```swift +let pattern = #"(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)"# +let regex = try! Regex(pattern) +// regex: Regex +``` + +The ability to compile regex patterns at run time is useful for cases where it is e.g provided as user input, however it is suboptimal when the pattern is statically known for a number of reasons: + +- Regex syntax errors aren't detected until run time, and explicit error handling (e.g `try!`) is required to deal with these errors. +- No special source tooling support, such as syntactic highlighting, code completion, and refactoring support, is available. +- Capture types aren't known until run time, and as such a dynamic `AnyRegexOutput` capture type must be used. +- The syntax is overly verbose, especially for e.g an argument to a matching function. + +## Proposed solution + +A regex literal may be written using `/.../` delimiters: + +```swift +// Matches " = ", extracting the identifier and hex number +let regex = /(?[[:alpha:]]\w*) = (?[0-9A-F]+)/ +// regex: Regex<(Substring, identifier: Substring, hex: Substring)> +``` + +Forward slashes are a regex term of art. The association between forward slashes and regexes dates back to 1969's ed, the first Unix editor, and it was inherited by subsequent interactive text tools like less and vim. The syntax was also adopted by the sed language; from there it passed to Perl, and then to Ruby and Javascript. Forward slash is instantly recognizable as a regex; the only common alternative is an ordinary string literal passed to a library API, which usually has extra overhead, requires more escaping, and defers regex syntax errors to runtime. The proposed Swift regex literals do not have these limitations, so forward slash provides the right behavioral cues to developers. There are over fifty years of precedents for forward slash and very little for anything else. + +Perl and Ruby additionally allow for [user-selected delimiters](https://perldoc.perl.org/perlop#Quote-and-Quote-like-Operators) to avoid having to escape any slashes inside a regex. For that purpose, we propose the extended literal `#/.../#`. + +An extended literal, `#/.../#`, avoids the need to escape forward slashes within the regex. It allows an arbitrary number of balanced `#` characters around the literal and escape. When the opening delimiter is followed by a new line, it supports a multi-line literal where whitespace is non-semantic and line-ending comments are ignored. + +The compiler will parse the contents of a regex literal using regex syntax outlined in *[Regex Construction][internal-syntax]*, diagnosing any errors at compile time. The capture types and labels are automatically inferred based on the capture groups present in the regex. Regex literals allows editors and source tools to support features such as syntax coloring inside the literal, highlighting sub-structure of the regex, and conversion of the literal to an equivalent result builder DSL (see *[Regex builder DSL][regex-dsl]*). + +A regex literal also allows for seamless composition with the Regex DSL, enabling lightweight intermixing of a regex syntax with other elements of the builder: + +```swift +// A regex for extracting a currency (dollars or pounds) and amount from input +// with precisely the form /[$£]\d+\.\d{2}/ +let regex = Regex { + Capture { /[$£]/ } + TryCapture { + /\d+/ + "." + /\d{2}/ + } transform: { + Amount(twoDecimalPlaces: $0) + } +} +``` + +This flexibility allows for terse matching syntax to be used when it's suitable, and more explicit syntax where clarity and strong types are required. + +Due to the existing use of `/` in comment syntax and operators, there are some syntactic ambiguities to consider. While there are quite a few cases to consider, we do not feel that the impact of any individual case is sufficient to disqualify the syntax. Some of these ambiguities require a couple of source breaking language changes, and as such the `/.../` syntax requires upgrading to a new language mode in order to use. + +## Detailed design + +### Typed captures + +Regex literals have their capture types statically determined by the capture groups present. This follows a similar inference behavior to [the DSL][regex-dsl], and is explored in more detail in *[Strongly Typed Captures][strongly-typed-captures]*. We are proposing the following inference behavior for regex literals: + +- A `Substring` is always present for the entire match. +- If any captures are present, a tuple is formed with the `Substring`, with subsequent elements representing the capture types. Captures are ordered according to [their numbering][capture-numbering]. + +The type of a capture is `Substring` by default, however it gets wrapped in an optional if it is not guaranteed to have a value on a successful match. This occurs when it is nested within a quantification that may be zero, which includes `?`, `*`, and any range quantifier with a `0` lower bound, e.g `{0,n}`. It also occurs when it appears in a branch of an alternation. For example: + +```swift +let regex1 = /([ab])?/ +// regex1: Regex<(Substring, Substring?)> + +let regex2 = /([ab])|\d+/ +// regex2: Regex<(Substring, Substring?)> +``` + +A zero quantifier or alternation nested within a capture do not produce an optional capture, unless the capture itself is inside a zero quantifier or alternation: + +```swift +let regex = /([ab]*)cd/ +// regex: Regex<(Substring, Substring)> +``` + +In this case, if the `*` quantifier is matched zero times, the resulting capture will be an empty string. + +The optional wrapping does not become nested, at most one layer of optionality is applied. For example: + +```swift +let regex = /(.)*|\d/ +// regex: Regex<(Substring, Substring?)> +``` + +This behavior differs from that of the DSL, which does apply multiple layers of optionality in such cases due to a current limitation of result builders. + +### Named captures + +One additional feature of typed captures that is currently unique to the literal is the ability to infer labeled tuple elements for named capture groups. For example: + +```swift +func matchHexAssignment(_ input: String) -> (String, Int)? { + let regex = /(?[[:alpha:]]\w*) = (?[0-9A-F]+)/ + // regex: Regex<(Substring, identifier: Substring, hex: Substring)> + + guard let match = input.wholeMatch(of: regex), + let hex = Int(match.hex, radix: 16) + else { return nil } + + return (String(match.identifier), hex) +} +``` + +This allows the captures to be referenced as `match.identifier` and `match.hex`, in addition to numerically (like unnamed capture groups) as `match.1` and `match.2`. This label inference behavior is not available in the DSL, however users are able to [bind captures to named variables instead][dsl-captures]. + +### Extended delimiters `#/.../#`, `##/.../##` + +Backslashes may be used to write forward slashes within the regex literal, e.g `/foo\/bar/`. However, this can be quite syntactically noisy and confusing. To avoid this, a regex literal may be surrounded by an arbitrary number of balanced number signs. This changes the delimiter of the literal, and therefore allows the use of forward slashes without escaping. For example: + +```swift +let regex = #/usr/lib/modules/([^/]+)/vmlinuz/# +// regex: Regex<(Substring, Substring)> +``` + +The number of `#` characters may be further increased to allow the use of e.g `/#` within the literal. This is similar in style to the raw string literal syntax introduced by [SE-0200], however it has a couple of key differences. Backslashes do not become literal characters. Additionally, a multi-line literal, where whitespace and line-ending comments are ignored, is supported when the opening delimiter is followed by a newline. + +```swift +let regex = #/ + usr/lib/modules/ # Prefix + (? [^/]+) + /vmlinuz # The kernel +/# +// regex: Regex<(Substring, subpath: Substring)> +``` + +#### Escaping of backslashes + +This syntax differs from raw string literals `#"..."#` in that it does not treat backslashes as literal within the regex. A string literal `#"\n"#` represents the literal characters `\n`. However a regex literal `#/\n/#` remains a newline escape sequence. + +One of the primary motivations behind this escaping behavior in raw string literals is that it allows the contents to be easily transportable to/from e.g external files where escaping is unnecessary. For string literals, this suggests that backslashes be treated as literal by default. For regex literals however, it instead suggests that backslashes should retain their semantic meaning. This enables interoperability with regexes taken from outside your code without having to adjust escape sequences to match the delimiters used. + +With string literals, escaping can be tricky without the use of raw syntax, as backslashes may have semantic meaning to the consumer, rather than the compiler. For example: + +```swift +// Matches '\' * '=' * + +let regex = try NSRegularExpression(pattern: "\\\\\\w\\s*=\\s*\\d+", options: []) +``` + +In this case, the intent is not for the compiler to recognize any of these sequences as string literal escapes, it is instead for `NSRegularExpression` to interpret them as regex escape sequences. As such, a raw string may be used to treat the backslashes literally, allowing `NSRegularExpression` to directly process the escapes, e.g `#"\\\w\s*=\s*\d+"#`. + +However this is not an issue for regex literals, as the regex parser is the only possible consumer of such escape sequences. Such a regex can be directly spelled as: + +```swift +let regex = /\\\w\s*=\s*\d+/ +// regex: Regex +``` + +Backslashes still require escaping to be treated as literal, however we don't expect this to be as common of an occurrence as needing to write a regex escape sequence such as `\s`, `\w`, or `\p{...}`, within a regex literal with extended delimiters `#/.../#`. + +#### Multi-line literals + +Extended regex delimiters additionally support a multi-line literal when the opening delimiter is followed by a new line. For example: + +```swift +let regex = #/ + # Match a line of the format e.g "DEBIT 03/03/2022 Totally Legit Shell Corp $2,000,000.00" + (? \w+) \s\s+ + (? \S+) \s\s+ + (? (?: (?!\s\s) . )+) \s\s+ # Note that account names may contain spaces. + (? .*) +/# +``` + +In such a literal, [extended regex syntax][extended-regex-syntax] `(?x)` is enabled. This means that whitespace in the regex becomes non-semantic (including within character classes), and end-of-line comments are supported with `# comment` syntax. + +This mode is supported with any (non-zero) number of `#` characters in the delimiter. Similar to multi-line strings introduced by [SE-0168], the closing delimiter must appear on a new line. To avoid parsing confusion, such a literal will not be parsed if a closing delimiter is not present. This avoids inadvertently treating the rest of the file as regex if you only type the opening. + +Extended syntax in such a literal may not be disabled with `(?-x)`, however it may be disabled for the contents of a group `(?-x:...)` or quoted sequence `\Q...\E`, as long as they do not span multiple lines. Supporting semantic whitespace over multiple lines would require stripping leading and trailing whitespace while maintaining the verbatim newlines. This could feasibly be supported, however we feel that its behavior could potentially be confusing. + +If desired, newlines may be written using `\n`, or by using a backslash to escape the literal newline character: + +```swift +let regex = #/ + a\ + b\ + c +/# +// regex = /a\nb\nc/ +``` + +### Ambiguities of `/.../` with comment syntax + +Line comment syntax `//` and block comment syntax `/*` will continue to be parsed as comments. An empty regex literal is not a particularly useful thing to express, but can be written as `#//#` if desired. `*` would be an invalid starting character of a regex, and therefore does not pose an issue. + +A parsing conflict does however arise when a block comment surrounds a regex literal ending with `*`, for example: + + ```swift + /* + let regex = /[0-9]*/ + */ + ``` + +In this case, the block comment prematurely ends on the second line, rather than extending all the way to the third line as the user would expect. This is already an issue today with `*/` in a string literal, though it is more likely to occur in a regex given the prevalence of the `*` quantifier. This issue can be avoided in many cases by using line comment syntax `//` instead, which it should be noted is the syntax that Xcode uses when commenting out multiple lines. + + +### Ambiguity of `/.../` with infix operators + +There is a minor ambiguity when infix operators are used with regex literals. When used without whitespace, e.g `x+/y/`, the expression will be treated as using an infix operator `+/`. Whitespace is therefore required for regex literal interpretation, e.g `x + /y/`. Alternatively, extended literals may be used, e.g `x+#/y/#`. + +### Regex syntax limitations in `/.../` + +In order to help avoid a parsing ambiguity, a `/.../` regex literal will not be parsed if it starts or ends with a space or tab character. This restriction may be avoided by using the extended `#/.../#` literal. + +#### Rationale + +The restriction on the ending character helps avoid breaking source compatibility with prefix and infix `/` operators in certain cases. Such cases are explored in the next section. The restriction on the starting character is due to a parsing ambiguity that arises when a `/.../` regex literal starts a new line. This is particularly problematic for result builders, where we expect it to be frequently used, in particular within a `Regex` builder: + +```swift +let digit = Regex { + TryCapture(OneOrMore(.digit)) { Int($0) } +} +// Matches against + (' + ' | ' - ') + +let regex = Regex { + digit + / [+-] / + digit +} +``` + +Instead of being parsed as 3 result builder elements, the second of which being a regex literal, this is instead parsed as a single operator chain with the operands `digit`, `[+-]`, and `digit`. This will therefore be diagnosed as semantically invalid. + +To avoid this issue, a regex literal may not start with a space or tab character. If a space or tab is needed as the first character, it must be either escaped, e.g: + +```swift +let regex = Regex { + digit + /\ [+-] / + digit +} +``` + +or an extended literal must be used, e.g: + +```swift +let regex = Regex { + digit + #/ [+-] /# + digit +} +``` + +This restriction takes advantage of the fact that infix operators require consistent spacing on either side. This includes both space characters as well as newlines. For example: + +```swift +let a = 0 + 1 // Valid +let b = 0+1 // Also valid +let c = 0 ++ 1 // Valid operator chain because the newline before '+' is whitespace. + +let d = 0 +1 // Not valid, '+' is treated as prefix, which cannot then appear next to '0'. +let e = 0+ 1 // Same but postfix +let f = 0 ++1 // Not a valid operator chain, same as 'd', except '+1' is no longer sequenced with '0'. +``` + +In much the same way as `f`, by requiring the first character of a regex literal not to be space or tab, we ensure it cannot be treated as an operator chain: + +```swift +let g = 0 +/1 + 2/ // Must be a regex +``` + +### How `/.../` is parsed + +A `/.../` regex literal will be parsed when an opening `/` is encountered in expression position, and there is a closing `/` present. As such, the following will continue to parse as normal: + +```swift +// Infix '/' is never in an expression position in valid code (unless unapplied). +let a = 1 / 2 / 3 + +// None of these '/^/' cases are in expression position. +infix operator /^/ +func /^/ (lhs: Int, rhs: Int) -> Int { 0 } +let b = 0 /^/ 1 + +// Also fine. +prefix operator / +prefix func / (_ x: Int) -> Int { x } +let c = /0 // No closing '/', so not a regex literal. The '//' of this comment doesn't count either. +``` + +But `let r = /^/` will be parsed as a regex. + +A regex literal may be used with a prefix operator, e.g `let r = ^^/x/` is parsed as `let r = ^^(/x/)`. In this case, when encountering operator characters containing `/` in an expression position, the characters up to the first `/` are split into a prefix operator, and regex literal parsing continues as normal. + +As already discussed, a regex literal may not start or end with a space or tab. This means that the following will continue to be parsed as normal: + +```swift +// Unapplied '/' in a call to 'reduce': +let x = array.reduce(1, /) / 5 +let y = array.reduce(1, /) + otherArray.reduce(1, /) + +// Prefix '/' with another '/' on the same line: +foo(/a, /b) +bar(/x) / 2 + +// Unapplied operators: +baz(!/, 1) / 2 +qux(/, /) +qux(/^, /) +qux(!/, /) + +let d = hasSubscript[/] / 2 // Unapplied infix '/' and infix '/' + +let e = !/y / .foo() // Prefix '!/' with infix '/' and operand '.foo()' +``` + +However this is not sufficient to disambiguate cases such as: + +```swift +// Prefix '/' used multiple times on the same line without trailing whitespace: +(/x).foo(/y) +bar(/x) + bar(/y) + +// Cases where the closing '/' is not used with whitespace: +bar(/x)/2 +baz(!/, 1)/2 + +// Prefix '/^' with postfix '/': +let f = (/^x)/ +``` + +In all of these cases, the opening `/` appears in expression position, and there is a potential closing `/` that is used without whitespace. To avoid source breakage for such cases, one further heuristic is employed. A regex literal will not be parsed if it contains an unbalanced `)`. This takes both escapes and custom character classes into consideration, and therefore only applies to syntax that would already be invalid for a regex. As such, all of the above cases will continue to be parsed as normal. + +This additional heuristic also allows for straightforward disambiguation in source breaking cases where the regex is valid. For example, the following cases will become regex literals: + +```swift +foo(/a, b/) // Will become regex literal '/a, b/' +qux(/, !/) // Will become regex literal '/, !/' +qux(/,/) // Will become regex literal '/,/' + +let g = hasSubscript[/]/2 // Will become regex literal '/]/' + +let h = /0; let f = 1/ // Will become the regex literal '/0; let y = 1/' +let i = /^x/ // Will become the regex literal '/^x/' +``` + +However they can be readily disambiguated by inserting parentheses: + +```swift +// Now a prefix and postfix '/': +foo((/a), b/) + +// Now unapplied operators: +qux((/), !/) +qux((/),/) +let g = hasSubscript[(/)]/2 + +let h = (/0); let f = 1/ // Now prefix '/' and postfix '/' +let i = (/^x)/ // Now prefix '/^' and postfix '/' +``` + +or, in some cases, by inserting whitespace: + +```swift +qux(/, /) +let g = hasSubscript[/] / 2 +``` + +We however expect these cases will be fairly uncommon. A similar case is the use of an unapplied infix operator with two `/` characters, for example: + +```swift +baz(/^/) // Will become the regex literal '/^/' rather than an unapplied operator +``` + +This cannot be disambiguated with parentheses or whitespace, however it can be disambiguated using a closure. For example: + +```swift +baz({ $0 /^/ $1 }) // Is now infix '/^/' +``` + +This takes advantage of the fact that a regex literal will not be parsed in an infix operator position. + +## Source Compatibility + +As explored above, the parsing of `/.../` does have potential to break source in cases where all of the following hold: + +- `/` appears in an expression position. +- There is a closing `/` on the same line. +- The first and last character of the literal is not space or tab. +- There are no unbalanced `)` characters within the literal. + +However we expect these cases will be uncommon, and can be disambiguated with parentheses or closures if needed. + +To accommodate the cases where source may be broken, `/.../` regex literals will be introduced in Swift 6 mode. However, projects may adopt the syntax earlier by passing the compiler flag `-enable-bare-slash-regex` or the [upcoming feature flag](0362-piecemeal-future-features.md) `BareSlashRegexLiterals`. Note this does not affect the extended delimiter syntax `#/.../#`, which will be usable immediately. + +## Future Directions + +### Modern literal syntax + +We could support a more modern Swift-like syntax in regex literals. For example, comments could be done with `//` and `/* ... */`, and quoted sequences could be done with `"..."`. This would however be incompatible with the syntactic superset of regex syntax we intend to parse, and as such may need to be introduced using a new literal kind, with no obvious choice of delimiter. + +However, such a syntax would lose out on the familiarity benefits of standard regex, and as such may lead to an "uncanny valley" effect. It's also possible that the ability to use regex literals in the DSL lessens the benefit that this syntax would bring. + +### Typed captures for duplicate named group + +PCRE allows duplicate capture group names when `(?J)` is set. However this would be incompatible with labeled tuple elements for the captures, as tuples may not have duplicate names. Given we do not currently support `(?J)` in regex literals, the handling of typed captures here is left as future work. + +### Typed captures for branch reset alternations + +PCRE and Perl support a branch reset construct `(?|(a)|(b))` where a child alternation resets the capture numbering for each branch, allowing `(a)` and `(b)` to share the same capture number. This would require unifying their types for the purposes of typed captures. Given we do not currently support this construct, the handling of typed captures here is left as future work. + +### Library-extensible protocol support + +A regex literal describes a string processing algorithm which can be ran over some model of String. The precise semantics of running over extended grapheme clusters vs Unicode scalar values is part of [Unicode for String Processing][regex-unicode]. Libraries may wish to extend this behavior, but the approach presented by various `ExpressibleBy*` protocols is underpowered as libraries would need access to the structure of the algorithm itself. + +A better (and future) approach is to open up the regex parser's AST, API, and AST actions to libraries. Here's some examples of why a library might want to customize regex: + +A library may wish to provide support for a different or higher level model of string. For example, using localized comparison or tailored grapheme-cluster breaks. Such a use case would need access to the structure of the string processing algorithm literal. + +A library may wish to provide support for running over another engine, such as ICU, PCRE, or Javascript. Such a use case would want to pretty-print Swift's regex syntax into one of these syntax variants. + +A library may wish to provide their own higher-level structure around which regex literals can be embedded for the purpose of multi-tier processing. For example, processing URLs where regex literal-character portions would be converted into percent-encoded equivalents (with some kind of character class customization/mapping as well). Additionally, a library may have the desire to explicitly delineate patterns that evaluate within a component vs patterns spanning multiple components. Such an approach would benefit from access to the real AST and rich semantic API. + +## Alternatives Considered + +### Alternative delimiter to `/.../` + +Given the fact that `/.../` is an existing term of art for regular expressions, we feel it should be the preferred delimiter syntax. It should be noted that the syntax has become less popular in some communities such as Perl, however we still feel that it is a compelling choice, especially with extended delimiters `#/.../#`. Additionally, while there are some syntactic ambiguities, we do not feel they are sufficient to disqualify the syntax. To evaluate this trade-off, below is a list of alternative delimiters that would not have the same ambiguities, and would not therefore require source breaking changes. + +#### Extended literal delimiters only `#/.../#` + +We could choose to avoid adding the bare forward slash syntax, and instead require at least one `#` character to be present in the delimiter. This would retain some of the familiarity of `/.../` while avoiding the parsing ambiguities and source breaking changes. + +However we feel that `/.../` is the better choice of default syntax, especially for simple regex where the additional noise of the `#` characters would be undesirable. While there are some parsing ambiguities to contend with, we do not feel they outweigh the benefits of having a lightweight and instantly recognizable syntax for regex. + +#### Prefixed quote `re'...'` + +We could choose to use `re'...'` delimiters, for example: + +```swift +// Matches " = ", extracting the identifier and hex number +let regex = re'([[:alpha:]]\w*) = ([0-9A-F]+)' +``` + +The use of two letter prefix could potentially be used as a namespace for future literal types. It would also have obvious extensions to extended and multi-line literals using `re#'...'#` and `re'''...'''` respectively. However, it is unusual for a Swift literal to be prefixed in this way. We also feel that its similarity to a string literal might have users confuse it with a raw string literal. + +Also, there are a few items of regex grammar that use the single quote character as a metacharacter. These include named group definitions and references such as `(?'name')`, `(?('name'))`, `\g'name'`, `\k'name'`, as well as callout syntax `(?C'arg')`. The use of a single quote conflicts with the `re'...'` delimiter as it will be considered the end of the literal. However, alternative syntax exists for all of these constructs, e.g `(?)`, `\k`, and `(?C"arg")`. Those could be required instead. An extended regex literal syntax e.g `re#'...'#` would also avoid this issue. + +#### Prefixed double quote `re"...."` + +This would be a double quoted version of `re'...'`, more similar to string literal syntax. This has the advantage that single quote regex syntax e.g `(?'name')` would continue to work without requiring the use of the alternative syntax or extended literal syntax. However it could be argued that regex literals are distinct from string literals in that they introduce their own specific language to parse. As such, regex literals are more like "program literals" than "data literals", and the use of single quote instead of double quote may be useful in expressing this difference. + +#### Single letter prefixed quote `r'...'` + +This would be a slightly shorter version of `re'...'`. While it's more concise, it could potentially be confused to mean "raw", especially as Python uses this syntax for raw strings. + +#### Single quotes `'...'` + +This would be an even more concise version of `re'...'` that drops the prefix entirely. However, given how close it is to string literal syntax, it may not be entirely clear to users that `'...'` denotes a regex as opposed to some different form of string literal (e.g some form of character literal, or a string literal with different escaping rules). + +We could help distinguish it from a string literal by requiring e.g `'/.../'`, though it may not be clear that the `/` characters are part of the delimiters rather than part of the literal. Additionally, this would potentially rule out the use of `'...'` as a future literal kind. + +#### Magic literal `#regex(...)` + +We could opt for for a more explicitly spelled out literal syntax such as `#regex(...)`. This is a more heavyweight option, similar to `#selector(...)`. As such, it may be considered syntactically noisy as e.g a function argument `str.match(#regex([abc]+))` vs `str.match(/[abc]+/)`. + +Such a syntax would require the containing regex to correctly balance parentheses for groups, otherwise the rest of the line might be incorrectly considered a regex. This could place additional cognitive burden on the user, and may lead to an awkward typing experience. For example, if the user is editing a previously written regex, the syntax highlighting for the rest of the line may change, and unhelpful spurious errors may be reported. With a different delimiter, the compiler would be able to detect and better diagnose unbalanced parentheses in the regex. + +We could avoid the parenthesis balancing issue by requiring an additional internal delimiter such as `#regex(/.../)`. However this is even more heavyweight, and it may be unclear that `/` is part of the delimiter rather than part of an argument. Alternatively, we could replace the internal delimiter with another character such as ```#regex`...` ```, `#regex{...}`, or `#regex/.../`. However those would be inconsistent with the existing `#literal(...)` syntax and the first two would overload the existing meanings for the ``` `` ``` and `{}` delimiters. + +It should also be noted that `#regex(...)` would introduce a syntactic inconsistency where the argument of a `#literal(...)` is no longer necessarily valid Swift syntax, despite being written in the form of an argument. + +##### On future extensibility to other foreign language snippets + +One of the benefits of `#regex(...)` or `re'...'` is the extensibility to other kinds of foreign language snippets, such as SQL. Nothing in this proposal precludes a scalable approach to foreign language snippets using `#lang(...)` or `lang'...'`. If or when that happens, regex could participate as well, but the proposed syntax would still be valuable as regex literals *are* unique in their prevalence as fragments passed directly to API, as well as components of a result builder DSL. + + +#### Shortened magic literal `#(...)` + +We could reduce the visual weight of `#regex(...)` by only requiring `#(...)`. However it would still retain the same issues, such as still looking potentially visually noisy as an argument, and having suboptimal behavior for parenthesis balancing. It is also not clear why regex literals would deserve such privileged syntax. + +#### Double slash `// ... //` + +Rather than using single forward slash delimiters `/.../`, we could use double slash delimiters. This would have previously been comment syntax, and would therefore be potentially source breaking. In particular, file header comments frequently use this style. Even if they successfully parse as a regex, they would receive different syntax highlighting, and emit a spurious error about being unused. + +This would also significantly impact a variety of commonly occurring comments, some examples from the Swift repository include: + +```swift +// rdar://41219750 + +// Please submit a bug report (https://swift.org/contributing/#reporting-bugs) + +// let pt = CGPoint(x: 1.0, y: 2.0) // Here we query for CGFloat. +``` + +This syntax also means the editor would not be able to automatically complete the closing delimiter, as it would initially appear to be a regular comment. This further means that typing the literal would receive comment syntax highlighting until the closing delimiter is written. + +#### Reusing string literal syntax + +Instead of supporting a first-class literal kind for regex, we could instead allow users to write a regex in a string literal, and parse, diagnose, and generate the appropriate code when it's coerced to the `Regex` type. + +```swift +let regex: Regex = #"([[:alpha:]]\w*) = ([0-9A-F]+)"# +``` + +However we decided against this because: + +- We would not be able to easily apply custom syntax highlighting and other editor features for the regex syntax. +- It would require a `Regex` contextual type to be treated as a regex, otherwise it would be defaulted to `String`, which may be undesired. +- In an overloaded context it may be ambiguous or unclear whether a string literal is meant to be interpreted as a literal string or regex. +- Regex-specific escape sequences such as `\w` would likely require the use of raw string syntax `#"..."#`, as they are otherwise invalid in a string literal. +- It wouldn't be compatible with other string literal features such as interpolations. + +### No custom literal + +Instead of adding a custom regex literal, we could require users to explicitly write `try! Regex("[abc]+")`. This would be similar to `NSRegularExpression`, and loses all the benefits of parsing the literal at compile time. This would mean: + +- No source tooling support (e.g syntax highlighting, refactoring actions) would be available. +- Parse errors would be diagnosed at run time rather than at compile time. +- We would lose the type safety of typed captures. +- More verbose syntax is required. + +We therefore feel this would be a much less compelling feature without first class literal support. + +### Non-semantic whitespace by default for single-line literals + +We could choose to enable non-semantic whitespace by default for single-line literals, matching the behavior of multi-line literals. While this is quite compelling for better readability, we feel that it would lose out on the familiarity and compatibility of the single-line literal. + +Non-semantic whitespace can always be enabled explicitly with `(?x)`: + +```swift +let r = /(?x) abc | def/ +``` + +or by writing a multi-line literal: + +```swift +let r = #/ + abc | def +/# +``` + +### Multi-line literal with semantic whitespace by default + +We could choose semantic whitespace by default within a multi-line regex literal. Such a literal would require a whitespace stripping rule, while keeping newlines of the contents verbatim. To enable non-semantic whitespace in such a literal, you would either have to explicitly write `(?x)` at the very start of the literal: + +```swift +let regex = #/ +(?x) abc | def +/# +``` + +Or we could support an explicit specifier as part of the delimiter syntax. For example: + +```swift +let regex = #/x + abc | def +/# +``` + +However, we don't find either of these options particularly compelling. The former is somewhat verbose considering we expect it to be a common mode for multi-line literals, and it would change meaning if indented at all. The latter wouldn't extend to other matching options, and wouldn't be usable within a single-line literal. + +We ultimately feel that non-semantic whitespace is a much more useful default for a multi-line regex literal, and unlike the single-line case, does not lose out on compatibility or familiarity. We could still enforce the specification of `(?x)` or `x`, however they would retain the same drawbacks. We are therefore not convinced they would be beneficial, and feel that the literal being split over multiple lines provides enough signal to indicate different semantics. + +#### Supporting the full matching option syntax as part of the delimiter + +Rather than supporting a specifier such as `x` on the delimiter, we could support the full range of matching option syntax on the delimiter. For example: + +```swift +let regex = #/(?xi) + abc | def +/# +``` + +However this would be more verbose, and would add additional complexity to the lexing logic which needs to be able to distinguish between an unterminated single-line literal, and a multi-line literal. It would also be limited to the isolated syntax, and e.g wouldn't support `(?xi:...)`. As we expect non-semantic whitespace to be the frequently desired mode in such a literal, we are not convinced the extra complexity or verbosity is beneficial. + +### Allow matching option flags on the literal `/.../x` + +We could choose to support Perl-style specification of matching options on the literal. This could feasibly be supported without introducing source compatibility issues, as identifiers cannot normally be sequenced with a regex literal. However it is unusual for a Swift literal to be suffixed like that. For matching options that affect runtime matching, e.g `i`, we intend on exposing API such as `/.../.ignoresCase()`. The only remaining options that affect parsing instead of matching are `x`, `xx`, `n`, and `J`. These cannot be exposed as API, however the multi-line literal already provides a way to enter extended syntax mode, and we feel writing `(?n)` or `(?J)` at the start of the literal is a suitable alternative to `/.../n` and `/.../J`. + +For the multi-line literal, we could require the specification of the `x` flag to enable extended syntax mode. However this would still require the `#/` delimiter. As such, it would lose out on the familiarity of the `/.../x` syntax, and wouldn't provide much visual signal for non-trivial literals. For example: + +```swift +let regex = #/ + # Match a line of the format e.g "DEBIT 03/03/2022 Totally Legit Shell Corp $2,000,000.00" + (? \w+) \s\s+ + (? \S+) \s\s+ + (? (?: (?!\s\s) . )+) \s\s+ # Note that account names may contain spaces. + (? .*) +/x# +``` + +### Using `///` or `#///` for multi-line + +Instead of re-using the extended delimiter syntax `#/.../#` for multi-line regex literals, we could choose a delimiter that more closely parallels the multi-line string delimiter `"""`. `///` would be the obvious choice, but unfortunately already signifies a documentation comment. As such, it would likely not be viable without further heuristics and regex syntax limitations. A possible alternative is to require at least one `#` character, e.g `#///`. This would be more syntactically viable at the cost of being more verbose. However it may seem odd that a `///` delimiter does not exist for such a literal. + +In either case, we are not convinced that drawing a parallel to multi-line string literals is particularly desirable, as multi-line regex literals have considerably different semantics. For example, whitespace is non-semantic and backslashes treat newlines as literal, rather than eliding them: + +```swift +let str = """ + a\ + b\ + c +""" +// str = " a b c" + +let re = #/ + a\ + b\ + c +/# +// re = /a\nb\nc/ +``` + +For multi-line string literals, the two main reasons for choosing `"""` over `"` were: + +1. Editing: It was felt that typing `"` and temporarily messing up the source highlighting of the rest of the file was a bad experience. +2. Visual weight: It was felt that a single `"` written after potentially paragraphs of text would be difficult to notice. + +However we do not feel that these are serious issues for regex literals. The `#/` delimiter has plenty of visual weight, and we require a closing `/#` before the literal is treated as multi-line. While it may be possible for the closing `/#` of an existing multi-line regex literal to be treated as a closing delimiter when typing `#/` above, we feel such cases will be quite a bit less common than the string literal `"` case. + +### No multi-line literal + +We could choose to only support single-line regex literals, with more complex multi-line cases requiring the DSL. However we feel that the ability to write non-semantic whitespace multi-line regex literals is quite a compelling feature that is not covered by the DSL. We feel that confining the literal's ability to work with non-semantic whitespace to the single-line case would lose a lot of the benefits of the extended syntax. + +### Restrict feature set to that of the builder DSL + +The regex builder DSL is unable to provide some of the features presented such as named captures as tuble labels. An alternative could be to cut those features from the literal out of concern they may lead to an over-use of the literals. However, to do so would remove the clearest demonstration of the need for better type-level operations including working with labeled tuples. + +Similarly, there is no literal equivalent for some of the regex builder features, but that isn't an argument against them. The regex builder DSL has references which serves this role (though not as concisely) and they are useful beyond just naming captures. + +Regex literals should not be outright avoided, they should be used well. Artificially hampering their usage doesn't provide any benefit and we wouldn't want to lock these limitations into Swift's ABI. + + + +[SE-0168]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0168-multi-line-string-literals.md +[SE-0200]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0200-raw-string-escaping.md + +[pitch-status]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md +[regex-type]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md +[strongly-typed-captures]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md +[regex-unicode]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md#unicode-for-string-processing + +[internal-syntax]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md +[extended-regex-syntax]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#extended-syntax-modes + +[capture-numbering]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#group-numbering + +[regex-dsl]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0351-regex-builder.md +[dsl-captures]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0351-regex-builder.md#capture-and-reference diff --git a/proposals/0355-regex-syntax-run-time-construction.md b/proposals/0355-regex-syntax-run-time-construction.md new file mode 100644 index 0000000000..a3ed0c09ce --- /dev/null +++ b/proposals/0355-regex-syntax-run-time-construction.md @@ -0,0 +1,1075 @@ +# Regex Syntax and Run-time Construction + +* Proposal: [SE-0355](0355-regex-syntax-run-time-construction.md) +* Authors: [Hamish Knight](https://github.com/hamishknight), [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: https://github.com/apple/swift-experimental-string-processing + * Available in nightly toolchain snapshots with `import _StringProcessing` +* Review: ([first pitch](https://forums.swift.org/t/pitch-regex-syntax/55711)) + ([second pitch](https://forums.swift.org/t/pitch-2-regex-syntax-and-run-time-construction/56624)) + ([review](https://forums.swift.org/t/se-0355-regex-syntax-and-runtime-construction/57038)) + ([acceptance](https://forums.swift.org/t/accepted-se-0355-regex-syntax-and-runtime-construction/59232)) + +## Introduction + +A regex declares a string processing algorithm using syntax familiar across a variety of languages and tools throughout programming history. We propose the ability to create a regex at run time from a string containing regex syntax (detailed here), API for accessing the match and captures, and a means to convert between an existential capture representation and concrete types. + +The overall story is laid out in [SE-0350 Regex Type and Overview][overview] and each individual component is tracked in [Pitch and Proposal Status][pitches]. + +## Motivation + +Swift aims to be a pragmatic programming language, striking a balance between familiarity, interoperability, and advancing the art. Swift's `String` presents a uniquely Unicode-forward model of string, but currently suffers from limited processing facilities. + +`NSRegularExpression` can construct a processing pipeline from a string containing [ICU regular expression syntax][icu-syntax]. However, it is inherently tied to ICU's engine and thus it operates over a fundamentally different model of string than Swift's `String`. It is also limited in features and carries a fair amount of Objective-C baggage, such as the need to translate between `NSRange` and `Range`. + +```swift +let pattern = #"(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)"# +let nsRegEx = try! NSRegularExpression(pattern: pattern) + +func processEntry(_ line: String) -> Transaction? { + let range = NSRange(line.startIndex.. + +let regex: Regex<(Substring, Substring, Substring, Substring, Substring)> = + try! Regex(pattern) +``` + +### Syntax + +We propose accepting a syntactic "superset" of the following existing regular expression engines: + +- [PCRE 2][pcre2-syntax], an "industry standard" and a rough superset of Perl, Python, etc. +- [Oniguruma][oniguruma-syntax], a modern engine with additional features. +- [ICU][icu-syntax], used by NSRegularExpression, a Unicode-focused engine. +- [.NET][.net-syntax], which adds delimiter-balancing and some interesting minor details around conditional patterns. + +To our knowledge, all other popular regex engines support a subset of the above syntaxes. + +We also support [UTS#18][uts18]'s full set of character class operators (to our knowledge no other engine does). Beyond that, UTS#18 deals with semantics rather than syntax, and what syntax it uses is covered by the above list. We also parse Java's properties (e.g. `\p{javaLowerCase}`), meaning we support a superset of Java 8 as well. + +Note that there are minor syntactic incompatibilities and ambiguities involved in this approach. Each is addressed in the relevant sections below. + +Regex syntax will be part of Swift's source-compatibility story as well as its binary-compatibility story. Thus, we present a detailed and comprehensive design. + +## Detailed Design + +We propose initializers to declare and compile a regex from syntax. Upon failure, these initializers throw compilation errors, such as for syntax or type errors. API for retrieving error information is future work. + +```swift +extension Regex { + /// Parse and compile `pattern`, resulting in a strongly-typed capture list. + public init(_ pattern: String, as: Output.Type = Output.self) throws +} +extension Regex where Output == AnyRegexOutput { + /// Parse and compile `pattern`, resulting in a type-erased capture list. + public init(_ pattern: String) throws +} +``` + +We propose `AnyRegexOutput` for capture types not known at compilation time, alongside casting API to convert to a strongly-typed capture list. + +```swift +/// A type-erased regex output +public struct AnyRegexOutput { + /// Creates a type-erased regex output from an existing match. + /// + /// Use this initializer to fit a strongly-typed regex match into the + /// use site of a type-erased regex output. + public init(_ match: Regex.Match) + + /// Returns a strongly-typed output by converting type-erased values to the specified type. + /// + /// - Parameter type: The expected output type. + /// - Returns: The output, if the underlying value can be converted to the + /// output type; otherwise `nil`. + public func extractValues( + as type: Output.Type = Output.self + ) -> Output? +} + +extension AnyRegexOutput: RandomAccessCollection { + /// An individual type-erased output value. + public struct Element { + /// The range over which a value was captured. `nil` for no-capture. + public var range: Range? { get } + + /// The slice of the input over which a value was captured. `nil` for no-capture. + public var substring: Substring? { get } + + /// The captured value. `nil` for no-capture. + public var value: Any? { get } + + /// The name of this capture, if it has one, otherwise `nil`. + public var name: String? + } + + // Trivial collection conformance requirements + + public var startIndex: Int { get } + + public var endIndex: Int { get } + + public var count: Int { get } + + public func index(after i: Int) -> Int + + public func index(before i: Int) -> Int + + public subscript(position: Int) -> Element { get } +} +``` + +We propose adding an API to `Regex` and `Regex.Match` to cast the output type to a concrete one. A regex match will lazily create a `Substring` on demand, so casting the match itself saves ARC traffic vs extracting and casting the output. + +```swift +extension Regex.Match where Output == AnyRegexOutput { + /// Creates a type-erased regex match from an existing match. + /// + /// Use this initializer to fit a regex match with strongly-typed captures into the + /// use site of a type-erased regex match. + public init(_ match: Regex.Match) +} + +extension Regex where Output == AnyRegexOutput { + /// Creates a type-erased regex from an existing regex. + /// + /// Use this initializer to fit a regex with strongly-typed captures into the + /// use site of a type-erased regex, i.e. one that was created from a string. + public init(_ regex: Regex) +} + +extension Regex { + /// Creates a strongly-typed regex from a type-erased regex. + /// + /// Use this initializer to create a strongly-typed regex from + /// one that was created from a string. Returns `nil` if the types + /// don't match. + public init?(_ erased: Regex, as: Output.Type = Output.self) +} +``` + +We propose adding API to query and access captures by name in an existentially typed regex and match: + +```swift +extension Regex where Output == AnyRegexOutput { + /// Returns whether a named-capture with `name` exists. + public func contains(captureNamed name: String) -> Bool +} + +extension Regex.Match where Output == AnyRegexOutput { + /// Access a capture by name. Returns `nil` if there's no capture with that name. + public subscript(_ name: String) -> AnyRegexOutput.Element? { get } +} + +extension AnyRegexOutput { + /// Access a capture by name. Returns `nil` if no capture with that name was present in the Regex. + public subscript(_ name: String) -> AnyRegexOutput.Element? { get } +} +``` + +Finally, we propose API for creating a regex containing literal string content. This produces an equivalent regex to a string literal embedded in the result builder DSL. As this is much less common than run-time compilation or an embedded literal in the DSL, it has an explicit argument label. + +```swift +extension Regex { + /// Produces a regex that matches `verbatim` exactly, as though every + /// metacharacter in it was escaped. + public init(verbatim: String) +} +``` + +The rest of this proposal will be a detailed and exhaustive definition of our proposed regex syntax. + +
Grammar Notation + +For the grammar sections, we use a modified PEG-like notation, in which the grammar also describes an unambiguous top-down parsing algorithm. + +- ` -> ` gives the definition of `Element` +- The `|` operator specifies a choice of alternatives +- `'x'` is the literal character `x`, otherwise it's a reference to x + + A literal `'` is spelled `"'"` +- Postfix `*` `+` and `?` denote zero-or-more, one-or-more, and zero-or-one +- Range quantifiers, like `{1...4}`, use Swift range syntax as convention. +- Basic custom character classes are written like `[0-9a-zA-Z]` +- Prefix `!` operator means the next element must not appear (a zero-width assertion) +- Parenthesis group for the purposes of quantification +- Builtins use angle brackets: + - `` refers to an integer, `` a character, etc. + - `` is any whitespace character + - `` is the end-of-line anchor (e.g. `$` in regex). + +For example, `(!'|' !')' ConcatComponent)*` means any number (zero or more) occurrences of `ConcatComponent` so long as the initial character is neither a literal `|` nor a literal `)`. + +
+ +### Top-level regular expression + +``` +Regex -> GlobalMatchingOptionSequence? RegexNode +RegexNode -> '' | Alternation +Alternation -> Concatenation ('|' Concatenation)* +Concatenation -> (!'|' !')' ConcatComponent)* +``` + +A regex may be prefixed with a sequence of [global matching options](#pcre-global-matching-options). Its contents can be empty or a sequence of alternatives separated by `|`. + +Alternatives are a series of expressions concatenated together. The concatenation ends with either a `|` denoting the end of the alternative or a `)` denoting the end of a recursively parsed group. + +Alternation has a lower precedence than concatenation or other operations, so e.g `abc|def` matches against `abc` or `def`. + +### Concatenated subexpressions + +``` +ConcatComponent -> Trivia | Quote | Interpolation | Quantification + +Trivia -> Comment | NonSemanticWhitespace +Comment -> '(?#' (!')')* ')' | EndOfLineComment +Interpolation -> '<{' (!'}>')* '}>' + +(extended syntax only) EndOfLineComment -> '#' (! .)* +(extended syntax only) NonSemanticWhitespace -> + + +Quote -> '\Q' (!'\E' .)* '\E'? +``` + +Each component of a concatenation may be "trivia" (comments and non-semantic whitespace, if applicable), a quoted run of literal content, or a potentially-quantified subexpression. + +In-line comments, similarly to C, are lexical and are not recursively nested like normal groups are. A closing `)` cannot be escaped. + +Quotes are similarly lexical, non-nested, and the `\` before a `\E` cannot be escaped. For example, `\Q^[xy]+$\E`, is treated as the literal characters `^[xy]+$` rather than an anchored quantified character class. `\Q\\E` is a literal `\`. A quoted sequence `\Q` may not have a closing `\E`, in which case it extends to the end of the regex. A quote may appear in a custom character class, but such a quote may not be empty. + +An interpolation sequence `<{...}>` is syntax that is reserved for a potential future interpolation feature. As such, the details surrounding it are future work, and it will currently be rejected for both literals and run-time compiled patterns. It may however be made available in the future as the literal characters. + +### Quantified subexpressions + +``` +Quantification -> QuantOperand Quantifier? +Quantifier -> QuantAmount QuantKind? +QuantAmount -> '?' | '*' | '+' | '{' Range '}' +QuantKind -> '?' | '+' +Range -> ',' | ',' ? | + +QuantOperand -> AbsentFunction | Atom | Conditional | CustomCharClass | Group +``` + +Subexpressions can be quantified, meaning they will be repeated some number of times: + +- `?`: 0 or 1 times. +- `*`: 0 or more times. +- `+`: 1 or more times. +- `{n,m}`: Between `n` and `m` (inclusive) times. +- `{n,}`: `n` or more times. +- `{,m}`: Up to `m` times. +- `{n}`: Exactly `n` times. + +Behavior can further be refined by a subsequent `?` or `+`: + +- `x*` _eager_: consume as much of input as possible. +- `x*?` _reluctant_: consume as little of the input as possible. +- `x*+`: _possessive_: eager and never relinquishes any input consumed. + +### Atoms + +``` +Atom -> Anchor + | Backreference + | BacktrackingDirective + | BuiltinCharacterClass + | Callout + | CharacterProperty + | EscapeSequence + | NamedScalar + | Subpattern + | UnicodeScalar + | '\K' + | '\'? +``` + +Atoms are the smallest units of regex syntax. They include escape sequences, metacharacters, backreferences, etc. The most basic form of atom is a literal character. A metacharacter may be treated as literal by preceding it with a backslash. Other literal characters may also be preceded by a backslash, in which case it has no effect, e.g `\%` is literal `%`. However this does not apply to either non-whitespace Unicode characters, or to unknown ASCII letter and number character escapes, e.g `\I` is invalid and would produce an error. `(...)[\1]` is similarly invalid, as a backreference may not appear in a custom character class. + +#### Anchors + +``` +Anchor -> '^' | '$' | '\A' | '\b' | '\B' | '\G' | '\y' | '\Y' | '\z' | '\Z' +``` + +Anchors match against a certain position in the input rather than on a particular character of the input. + +- `^`: Matches at the very start of the input string, or the start of a line when in multi-line mode. +- `$`: Matches at the very end of the input string, or the end of a line when in multi-line mode. +- `\A`: Matches at the very start of the input string. +- `\Z`: Matches at the very end of the input string, in addition to before a newline at the very end of the input string. +- `\z`: Like `\Z`, but only matches at the very end of the input string. +- `\G`: Like `\A`, but also matches against the start position of where matching resumes in global matching mode (e.g `\Gab` matches twice in `abab`, `\Aab` would only match once). +- `\b` matches a boundary between a word character and a non-word character. The definitions of which vary depending on matching engine. +- `\B` matches a non-word-boundary. +- `\y` matches a text segment boundary, the definition of which varies based on the `y{w}` and `y{g}` matching option. +- `\Y` matches a non-text-segment-boundary. + +#### Escape sequences + +``` +EscapeSequence -> '\a' | '\b' | '\c' | '\e' | '\f' | '\n' | '\r' | '\t' +``` + +These escape sequences each denote a specific scalar value. + +- `\a`: The alert (bell) character `U+7`. +- `\b`: The backspace character `U+8`. Note this may only be used in a custom character class, otherwise it represents a word boundary. +- `\c `: A control character sequence, which denotes a scalar from `U+00` - `U+7F` depending on the ASCII character provided. +- `\e`: The escape character `U+1B`. +- `\f`: The form-feed character `U+C`. +- `\n`: The newline character `U+A`. +- `\r`: The carriage return character `U+D`. +- `\t`: The tab character `U+9`. + +#### Builtin character classes + +``` +BuiltinCharClass -> '.' | '\C' | '\d' | '\D' | '\h' | '\H' | '\N' | '\O' | '\R' | '\s' | '\S' | '\v' | '\V' | '\w' | '\W' | '\X' +``` + +- `.`: Any character excluding newlines. +- `\C`: A single UTF code unit. +- `\d`: Digit character. +- `\D`: Non-digit character. +- `\h`: Horizontal space character. +- `\H`: Non-horizontal-space character. +- `\N`: Non-newline character. +- `\O`: Any character (including newlines). This is syntax from Oniguruma. +- `\R`: Newline sequence. +- `\s`: Whitespace character. +- `\S`: Non-whitespace character. +- `\v`: Vertical space character. +- `\V`: Non-vertical-space character. +- `\w`: Word character. +- `\W`: Non-word character. +- `\X`: Any extended grapheme cluster. + +Precise definitions of character classes is discussed in [Unicode for String Processing][pitches]. + +#### Unicode scalars + +``` +UnicodeScalar -> '\u{' UnicodeScalarSequence '}' + | '\u' HexDigit{4} + | '\x{' HexDigit{1...} '}' + | '\x' HexDigit{0...2} + | '\U' HexDigit{8} + | '\o{' OctalDigit{1...} '}' + | '\0' OctalDigit{0...3} + +UnicodeScalarSequence -> * UnicodeScalarSequencElt+ +UnicodeScalarSequencElt -> HexDigit{1...} * + +HexDigit -> [0-9a-fA-F] +OctalDigit -> [0-7] + +NamedScalar -> '\N{' ScalarName '}' +ScalarName -> 'U+' HexDigit{1...8} | [\s\w-]+ +``` + +These sequences define a unicode scalar value using hexadecimal or octal notation. + +In addition to a regular scalar literal e.g `\u{65}`, `\u{...}` also supports a scalar sequence syntax. This is syntactic sugar that implicitly expands a whitespace separated list of scalars e.g `\u{A B C}` into `\u{A}\u{B}\u{C}`. Such a sequence is currently only valid outside of a custom character class, their behavior within a custom character class is left as future work. + +`\x`, when not followed by any hexadecimal digit characters, is treated as `\0`, matching PCRE's behavior. + +`\N{...}` allows a specific Unicode scalar to be specified by name or hexadecimal code point. + +#### Character properties + +``` +CharacterProperty -> '\' ('p' | 'P') '{' PropertyContents '}' +POSIXCharacterProperty -> '[:' PropertyContents ':]' + +PropertyContents -> PropertyName ('=' PropertyName)? +PropertyName -> [\s\w-]+ +``` + +A character property specifies a particular Unicode, POSIX, or PCRE property to match against. We propose supporting: + +- The full range of Unicode character properties. +- The POSIX properties `alnum`, `blank`, `graph`, `print`, `word`, `xdigit` (note that `alpha`, `lower`, `upper`, `space`, `punct`, `digit`, and `cntrl` are covered by Unicode properties). +- The UTS#18 special properties `any`, `assigned`, `ascii`. +- The special PCRE2 properties `Xan`, `Xps`, `Xsp`, `Xuc`, `Xwd`. +- The special Java properties, including e.g `javaLowerCase`, `javaUpperCase`, `javaWhitespace`, `javaMirrored`. + +We follow [UTS#18][uts18]'s guidance for character properties, including fuzzy matching for property name parsing, according to rules set out by [UAX44-LM3]. The following property names are equivalent: + +- `whitespace` +- `isWhitespace` +- `is-White_Space` +- `iSwHiTeSpaCe` +- `i s w h i t e s p a c e` + +Unicode properties consist of both a key and a value, e.g `General_Category=Whitespace`. Each component follows the fuzzy matching rule, and additionally may have an alternative alias spelling, as defined by Unicode in [PropertyAliases.txt][unicode-prop-key-aliases] and [PropertyValueAliases.txt][unicode-prop-value-aliases]. + +There are some Unicode properties where the key or value may be inferred. These include: + +- General category properties e.g `\p{Whitespace}` is inferred as `\p{General_Category=Whitespace}`. +- Script properties e.g `\p{Greek}` is inferred as `\p{Script_Extensions=Greek}`. +- Boolean properties that are inferred to have a `True` value, e.g `\p{Lowercase}` is inferred as `\p{Lowercase=True}`. +- Block properties that begin with the prefix `in`, e.g `\p{inBasicLatin}` is inferred to be `\p{Block=Basic_Latin}`. + +Other Unicode properties however must specify both a key and value. + +For non-Unicode properties, only a value is required. These include: + +- The UTS#18 special properties `any`, `assigned`, `ascii`. +- The POSIX compatibility properties `alnum`, `blank`, `graph`, `print`, `word`, `xdigit`. The remaining POSIX properties are already covered by boolean Unicode property spellings. +- The special PCRE2 properties `Xan`, `Xps`, `Xsp`, `Xuc`, `Xwd`. +- The special Java properties `javaLowerCase`, `javaUpperCase`, `javaWhitespace`, `javaMirrored`. + +Note that the internal `PropertyContents` syntax is shared by both the `\p{...}` and POSIX-style `[:...:]` syntax, allowing e.g `[:script=Latin:]` as well as `\p{alnum}`. Both spellings may be used inside and outside of a custom character class. + +#### `\K` + +The `\K` escape sequence is used to drop any previously matched characters from the final matching result. It does not affect captures, e.g `a(b)\Kc` when matching against `abc` will return a match of `c`, but with a capture of `b`. + +### Groups + +``` +Group -> GroupStart RegexNode ')' +GroupStart -> '(' GroupKind | '(' +GroupKind -> '' | '?' BasicGroupKind | '*' PCRE2GroupKind ':' + +BasicGroupKind -> ':' | '|' | '>' | '=' | '!' | '*' | '<=' | ' 'atomic' + | 'pla' | 'positive_lookahead' + | 'nla' | 'negative_lookahead' + | 'plb' | 'positive_lookbehind' + | 'nlb' | 'negative_lookbehind' + | 'napla' | 'non_atomic_positive_lookahead' + | 'naplb' | 'non_atomic_positive_lookbehind' + | 'sr' | 'script_run' + | 'asr' | 'atomic_script_run' + +NamedGroup -> 'P<' GroupNameBody '>' + | '<' GroupNameBody '>' + | "'" GroupNameBody "'" + +GroupNameBody -> Identifier | BalancingGroupBody + +Identifier -> [\w--\d] \w* +``` + +Groups define a new scope that contains a recursively nested regex. Groups have different semantics depending on how they are introduced. + +Note there are additional constructs that may syntactically appear similar to groups, such as backreferences and PCRE backtracking directives, but are distinct. + +#### Basic group kinds + +- `()`: A capturing group. +- `(?:)`: A non-capturing group. +- `(?|)`: A group that, for a direct child alternation, resets the numbering of groups at each branch of that alternation. See [Group Numbering](#group-numbering). + +Capturing groups produce captures, which remember the range of input matched for the scope of that group. + +A capturing group may be named using any of the `NamedGroup` syntax. The characters of the group name may be any letter or number characters or the character `_`. However the name must not start with a number. This restriction follows the behavior of other regex engines and avoids ambiguities when it comes to named and numeric group references. Duplicate group names are only permitted when either `(?J)` is set, or when the captures share the same numbering, e.g within a branch reset alternation `(?|)`. Otherwise, they are considered invalid. + +#### Atomic groups + +An atomic group e.g `(?>...)` specifies that its contents should not be re-evaluated for backtracking. This has the same semantics as a possessive quantifier, but applies more generally to any regex pattern. + +#### Lookahead and lookbehind + +These groups evaluate the input ahead or behind the current matching position, without advancing the input. + +- `(?=`: A lookahead, which matches against the input following the current matching position. +- `(?!`: A negative lookahead, which ensures a negative match against the input following the current matching position. +- `(?<=`: A lookbehind, which matches against the input prior to the current matching position. +- `(? Identifier? '-' Identifier +``` + +Introduced by .NET, [balancing groups][balancing-groups] extend the `GroupNameBody` syntax to support the ability to refer to a prior group. Upon matching, the prior group is deleted, and any intermediate matched input becomes the capture of the current group. + +#### Group numbering + +Capturing groups are implicitly numbered according to the position of their opening `(` in the regex. For example: + +``` +(a((?:b)(?c)d)(e)f) +^ ^ ^ ^ +1 2 3 4 +``` + +Non-capturing groups are skipped over when counting. + +Branch reset groups can alter this numbering, as they reset the numbering in the branches of an alternation child. Outside the alternation, numbering resumes at the next available number not used in one of the branches. For example: + +``` +(a()(?|(b)(c)|(?:d)|(e)))(f) +^ ^ ^ ^ ^ ^ +1 2 3 4 3 5 +``` + +Because this construct allows multiple capture groups to share the same number, it allows a capture to share the same name in both branches. For example: + +``` +(?|(?a)|(?b)) +``` + +which produces a single capture result named `x`. This would be otherwise be invalid in a regular alternation, as the captures would have distinct numberings. + +### Custom character classes + +``` +CustomCharClass -> Start Set (SetOp Set)* ']' +Start -> '[' '^'? +Set -> Member+ +Member -> CustomCharClass | Quote | Range | Atom +Range -> RangeElt `-` RangeElt +RangeElt -> | UnicodeScalar | EscapeSequence +SetOp -> '&&' | '--' | '~~' +``` + +Custom characters classes introduce their own sublanguage, in which most regular expression metacharacters become literal. The basic element in a custom character class is an `Atom`, though only some atoms are considered valid: + +- Builtin character classes, except for `.`, `\R`, `\O`, `\X`, `\C`, and `\N`. +- Escape sequences, including `\b` which becomes the backspace character (rather than a word boundary). +- Unicode scalars. +- Named scalars. +- Character properties. +- Plain literal characters. + +Atoms may be used to compose other character class members, including ranges, quoted sequences, and even nested custom character classes `[[ab]c\d]`. Adjacent members form an implicit union of character classes, e.g `[[ab]c\d]` is the union of the characters `a`, `b`, `c`, and digit characters. + +Custom character classes may not be empty, e.g `[]` is forbidden. + +Quoted sequences may be used to escape the contained characters, e.g `[a\Q]\E]` is the character class of `]` and `a`. + +Ranges of characters may be specified with `-`, e.g `[a-z]` matches against the letters from `a` to `z`. Only unicode scalars and literal characters are valid range operands. If `-` cannot be used to form a range, it is interpreted as literal, e.g `[-a-]` is the character class of `-` and `a`. `[a-c-d]` is the character class of `a`...`c`, `-`, and `d`. + +Operators may be used to apply set operations to character class members. The operators supported are: + +- `&&`: Intersection of the LHS and RHS. +- `--`: Subtraction of the RHS from the LHS. +- `~~`: Symmetric difference of the RHS and LHS. + +These operators have a lower precedence than the implicit union of members, e.g `[ac-d&&a[d]]` is an intersection of the character classes `[ac-d]` and `[ad]`. + +Note that a custom character class may begin with the `:` character, and only becomes a POSIX character property if a closing `:]` is present. For example, `[:a]` is the character class of `:` and `a`. + +### Matching options + +``` +MatchingOptionSeq -> '^' MatchingOption* + | MatchingOption+ + | MatchingOption* '-' MatchingOption* + +MatchingOption -> 'i' | 'J' | 'm' | 'n' | 's' | 'U' | 'x' | 'xx' | 'w' | 'D' | 'P' | 'S' | 'W' | 'y{' ('g' | 'w') '}' +``` + +A matching option sequence may be used as a group specifier, and denotes a change in matching options for the scope of that group. For example `(?x:a b c)` enables extended syntax for `a b c`. A matching option sequence may be part of an "isolated group" which has an implicit scope that wraps the remaining elements of the current group. For example, `(?x)a b c` also enables extended syntax for `a b c`. + +If used in the branch of an alternation, an isolated group affects all the following branches of that alternation. For example, `a(?i)b|c|d` is treated as `a(?i:b)|(?i:c)|(?i:d)`. + +We support all the matching options accepted by PCRE, ICU, and Oniguruma. In addition, we accept some matching options unique to our matching engine. + +#### PCRE options + +- `i`: Case insensitive matching. +- `J`: Allows multiple groups to share the same name, which is otherwise forbidden. +- `m`: Enables `^` and `$` to match against the start and end of a line rather than only the start and end of the entire string. +- `n`: Disables the capturing behavior of `(...)` groups. Named capture groups must be used instead. +- `s`: Changes `.` to match any character, including newlines. +- `U`: Changes quantifiers to be reluctant by default, with the `?` specifier changing to mean greedy. +- `x`, `xx`: Enables extended syntax mode, which allows non-semantic whitespace and end-of-line comments. See [Extended Syntax Modes](#extended-syntax-modes) for more info. + +#### ICU options + +- `w`: Enables the Unicode interpretation of word boundaries `\b`. + +#### Oniguruma options + +- `D`: Enables ASCII-only digit matching for `\d`, `\p{Digit}`, `[:digit:]`. +- `S`: Enables ASCII-only space matching for `\s`, `\p{Space}`, `[:space:]`. +- `W`: Enables ASCII-only word matching for `\w`, `\p{Word}`, `[:word:]`, and `\b`. +- `P`: Enables ASCII-only for all POSIX properties (including `digit`, `space`, and `word`). +- `y{g}`, `y{w}`: Changes the meaning of `\X`, `\y`, `\Y`. These are mutually exclusive options, with `y{g}` specifying extended grapheme cluster mode, and `y{w}` specifying word mode. + +#### Swift options + +These options are specific to the Swift regex matching engine and control the semantic level at which matching takes place. + +- `X`: Grapheme cluster matching. +- `u`: Unicode scalar matching. +- `b`: Byte matching. + +Further details on these are TBD and outside the scope of this pitch. + +### References + +``` +NamedOrNumberRef -> NamedRef | NumberRef +NamedRef -> Identifier RecursionLevel? +NumberRef -> ('+' | '-')? RecursionLevel? +RecursionLevel -> '+' | '-' +``` + +A reference is an abstract identifier for a particular capturing group in a regular expression. It can either be named or numbered, and in the latter case may be specified relative to the current group. For example `-2` refers to the capture group `N - 2` where `N` is the number of the next capture group. References may refer to groups ahead of the current position e.g `+3`, or the name of a future group. These may be useful in recursive cases where the group being referenced has been matched in a prior iteration. If a referenced capture does not exist anywhere in the regular expression, the reference is diagnosed as invalid. + +A backreference may optionally include a recursion level in certain cases, which is a syntactic element inherited [from Oniguruma][oniguruma-syntax] that allows the reference to specify a capture relative to a given recursion level. + +#### Backreferences + +``` +Backreference -> '\g{' NamedOrNumberRef '}' + | '\g' NumberRef + | '\k<' NamedOrNumberRef '>' + | "\k'" NamedOrNumberRef "'" + | '\k{' NamedRef '}' + | '\' [1-9] [0-9]+ + | '(?P=' NamedRef ')' +``` + +A backreference evaluates to the value last captured by the referenced capturing group. If the referenced capture has not been evaluated yet, the match fails. + +#### Subpatterns + +``` +Subpattern -> '\g<' NamedOrNumberRef '>' + | "\g'" NamedOrNumberRef "'" + | '(?' GroupLikeSubpatternBody ')' + +GroupLikeSubpatternBody -> 'P>' NamedRef + | '&' NamedRef + | 'R' + | NumberRef +``` + +A subpattern causes the referenced capture group to be re-evaluated at the current position. The syntax `(?R)` is equivalent to `(?0)`, and causes the entire pattern to be recursed. + +### Conditionals + +``` +Conditional -> ConditionalStart Concatenation ('|' Concatenation)? ')' +ConditionalStart -> KnownConditionalStart | GroupConditionalStart + +KnownConditionalStart -> '(?(' KnownCondition ')' +GroupConditionalStart -> '(?' GroupStart + +KnownCondition -> 'R' + | 'R' NumberRef + | 'R&' NamedRef + | '<' NamedOrNumberRef '>' + | "'" NamedOrNumberRef "'" + | 'DEFINE' + | 'VERSION' VersionCheck + | NumberRef + +PCREVersionCheck -> '>'? '=' PCREVersionNumber +PCREVersionNumber -> '.' +``` + +A conditional evaluates a particular condition, and chooses a branch to match against accordingly. 1 or 2 branches may be specified. If 1 branch is specified e.g `(?(...)x)`, it is treated as the true branch. Note this includes an empty true branch, e.g `(?(...))` which is the null pattern as described in the [Top-Level Regular Expression](#top-level-regular-expression) section. If 2 branches are specified, e.g `(?(...)x|y)`, the first is treated as the true branch, the second being the false branch. + +A condition may be: + +- A numeric or delimited named reference to a capture group, which checks whether the group matched successfully. +- A recursion check on either a particular group or the entire regex. In the former case, this checks to see if the last recursive call is through that group. In the latter case, it checks if the match is currently taking place in any kind of recursive call. +- A PCRE version check. + +If the condition does not syntactically match any of the above, it is treated as an arbitrary recursive regular expression. This will be matched against, and evaluates to true if the match is successful. It may contain capture groups that add captures to the match. + +The `DEFINE` keyword is not used as a condition, but rather a way in which to define a group which is not evaluated, but may be referenced by a subpattern. + +### PCRE backtracking directives + +``` +BacktrackingDirective -> '(*' BacktrackingDirectiveKind (':' )? ')' +BacktrackingDirectiveKind -> 'ACCEPT' | 'FAIL' | 'F' | 'MARK' | '' | 'COMMIT' | 'PRUNE' | 'SKIP' | 'THEN' +``` + +This is syntax specific to PCRE, and is used to control backtracking behavior. Any of the directives may include an optional tag, however `MARK` must have a tag. The empty directive is treated as `MARK`. Only the `ACCEPT` directive may be quantified, as it can use the backtracking behavior of the engine to be evaluated only if needed by a reluctant quantification. + +- `ACCEPT`: Causes matching to terminate immediately as a successful match. If used within a subpattern, only that level of recursion is terminated. +- `FAIL`, `F`: Causes matching to fail, forcing backtracking to occur if possible. +- `MARK`: Assigns a label to the current matching path, which is passed back to the caller on success. Subsequent `MARK` directives overwrite the label assigned, so only the last is passed back. +- `COMMIT`: Prevents backtracking from reaching any point prior to this directive, causing the match to fail. This does not allow advancing the input to try a different starting match position. +- `PRUNE`: Similar to `COMMIT`, but allows advancing the input to try and find a different starting match position. +- `SKIP`: Similar to `PRUNE`, but skips ahead to the position of `SKIP` to try again as the starting position. +- `THEN`: Similar to `PRUNE`, but when used inside an alternation will try to match in the subsequent branch before attempting to advance the input to find a different starting position. + +### PCRE global matching options + +``` +GlobalMatchingOptionSequence -> GlobalMatchingOption+ +GlobalMatchingOption -> '(*' GlobalMatchingOptionKind ')' + +GlobalMatchingOptionKind -> LimitOptionKind '=' + | NewlineKind | NewlineSequenceKind + | 'NOTEMPTY_ATSTART' | 'NOTEMPTY' + | 'NO_AUTO_POSSESS' | 'NO_DOTSTAR_ANCHOR' + | 'NO_JIT' | 'NO_START_OPT' | 'UTF' | 'UCP' + +LimitOptionKind -> 'LIMIT_DEPTH' | 'LIMIT_HEAP' | 'LIMIT_MATCH' +NewlineKind -> 'CRLF' | 'CR' | 'ANYCRLF' | 'ANY' | 'LF' | 'NUL' +NewlineSequenceKind -> 'BSR_ANYCRLF' | 'BSR_UNICODE' +``` + +This is syntax specific to PCRE, and allows a set of global options to appear at the start of a regular expression. They may not appear at any other position. + +- `LIMIT_DEPTH`, `LIMIT_HEAP`, `LIMIT_MATCH`: These place certain limits on the resources the matching engine may consume, and matches it may make. +- `CRLF`, `CR`, `ANYCRLF`, `ANY`, `LF`, `NUL`: These control the definition of a newline character, which is used when matching e.g the `.` character class, and evaluating where a line ends in multi-line mode. +- `BSR_ANYCRLF`, `BSR_UNICODE`: These change the definition of `\R`. +- `NOTEMPTY`: Does not consider the empty string to be a valid match. +- `NOTEMPTY_ATSTART`: Like `NOT_EMPTY`, but only applies to the first matching position in the input. +- `NO_AUTO_POSSESS`: Disables an optimization that treats a quantifier as possessive if the following construct clearly cannot be part of the match. In other words, disables the short-circuiting of backtracks in cases where the engine knows it will not produce a match. This is useful for debugging, or for ensuring a callout gets invoked. +- `NO_DOTSTAR_ANCHOR`: Disables an optimization that tries to automatically anchor `.*` at the start of a regex. Like `NO_AUTO_POSSESS`, this is mainly used for debugging or ensuring a callout gets invoked. +- `NO_JIT`: Disables JIT compilation +- `NO_START_OPT`: Disables various optimizations performed at the start of matching. Like `NO_DOTSTAR_ANCHOR`, is mainly used for debugging or ensuring a callout gets invoked. +- `UTF`: Enables UTF pattern support. +- `UCP`: Enables Unicode property support. + +### Callouts + +``` +Callout -> PCRECallout | NamedCallout | InterpolatedCallout + +PCRECallout -> '(?C' CalloutBody ')' +PCRECalloutBody -> '' | + | '`' '`' + | "'" "'" + | '"' '"' + | '^' '^' + | '%' '%' + | '#' '#' + | '$' '$' + | '{' '}' + +NamedCallout -> '(*' Identifier CalloutTag? CalloutArgs? ')' +CalloutArgs -> '{' CalloutArgList '}' +CalloutArgList -> CalloutArg (',' CalloutArgList)* +CalloutArg -> [^,}]+ +CalloutTag -> '[' Identifier ']' + +InterpolatedCallout -> '(?' '{' Interpolation '}' CalloutTag? CalloutDirection? ')' +Interpolation -> | '{' Interpolation '}' +CalloutDirection -> 'X' | '<' | '>' +``` + +A callout is a feature that allows a user-supplied function to be called when matching reaches that point in the pattern. We supported parsing 3 types of callout: + +- PCRE callout syntax, which accepts a string or numeric argument that is passed to the function. +- Oniguruma named callout syntax, which accepts an identifier with an optional tag and argument list. +- Interpolated callout syntax, which is equivalent to Oniguruma's "callout of contents". This callout accepts an arbitrary interpolated program. This is an expanded version of Perl's interpolation syntax, and allows an arbitrary nesting of delimiters in addition to an optional tag and direction. + +While we propose parsing these for the purposes of issuing helpful diagnostics, we are deferring full support for the interpolated syntax for the future. + +### Absent functions + +``` +AbsentFunction -> '(?~' RegexNode ')' + | '(?~|' Concatenation '|' Concatenation ')' + | '(?~|' Concatenation ')' + | '(?~|)' +``` + +An absent function is an [Oniguruma][oniguruma-syntax] feature that allows for the easy inversion of a given pattern. There are 4 variants of the syntax: + +- `(?~|absent|expr)`: Absent expression, which attempts to match against `expr`, but is limited by the range that is not matched by `absent`. +- `(?~absent)`: Absent repeater, which matches against any input not matched by `absent`. Equivalent to `(?~|absent|\O*)`. +- `(?~|absent)`: Absent stopper, which limits any subsequent matching to not include `absent`. +- `(?~|)`: Absent clearer, which undoes the effects of the absent stopper. + + +## Syntactic differences between engines + +The proposed "syntactic superset" introduces some minor ambiguities, as each engine supports a slightly different set of features. When a particular engine's parser sees a feature it doesn't support, it typically has a fall-back behavior, such as treating the unknown feature as literal contents. + +Explicit compatibility modes, i.e. precisely mimicking emergent behavior from a specific engine's parser, is deferred as future work from this proposal. Conversion from this "syntactic superset" to a particular engine's syntax (e.g. as an AST "pretty printer") is deferred as future work from this proposal. + +Below is an exhaustive treatment of every syntactic ambiguity we have encountered. + +### Character class set operations + +In a custom character class, some engines allow for binary set operations that take two character class inputs, and produce a new character class output. However which set operations are supported and the spellings used differ by engine. + +| PCRE | ICU | UTS#18 | Oniguruma | .NET | Java | +|------|-----|--------|-----------|------|------| +| ❌ | Intersection `&&`, Subtraction `--` | Intersection, Subtraction | Intersection `&&` | Subtraction via `-` | Intersection `&&` | + + +[UTS#18][uts18] requires intersection and subtraction, and uses the operation spellings `&&` and `--` in its examples, though it doesn't mandate a particular spelling. In particular, conforming implementations could spell the subtraction `[[x]--[y]]` as `[[x]&&[^y]]`. UTS#18 also suggests a symmetric difference operator `~~`, and uses an explicit `||` operator in examples, though doesn't require either. + +Engines that don't support a particular operator fallback to treating it as literal, e.g `[x&&y]` in PCRE is the character class of `["x", "&", "y"]` rather than an intersection. + +Unlike other engines, .NET supports the use of `-` to denote both a range as well as a set subtraction. .NET disambiguates this by only permitting its use as a subtraction if the right hand operand is a nested custom character class, otherwise it is a range operator. This conflicts with e.g ICU where `[x-[y]]`, in which the `-` is treated as literal. + +We propose supporting the operators `&&`, `--`, and `~~`. This means that any regex literal containing these sequences in a custom character class while being written for an engine not supporting that operation will have a different semantic meaning in our engine. However this ought not to be a common occurrence, as specifying a character multiple times in a custom character class is redundant. + +In order to help avoid confusion between engines, we will reject the use of .NET style `-` for subtraction. Users will be required to write `--` instead, or escape with `\-`. + +### Nested custom character classes + +This allows e.g `[[a]b[c]]`, which is interpreted the same as `[abc]`. It also allows for more complex set operations with custom character classes as the operands. + +| PCRE | ICU | UTS#18 | Oniguruma | .NET | Java | +|------|-----|--------|-----------|------|------| +| ❌ | ✅ | 💡 | ✅ | ❌ | ✅ | + + +UTS#18 doesn't require this, though it does suggest it as a way to clarify precedence for chains of character class set operations e.g `[\w--\d&&\s]`, which the user could write as `[[\w--\d]&&\s]`. + +PCRE does not support this feature, and as such treats `]` as the closing character of the custom character class. Therefore `[[a]b[c]]` is interpreted as the character class `["[", "a"]`, followed by literal `b`, and then the character class `["c"]`, followed by literal `]`. + +.NET does not support nested character classes in general, although allows them as the right-hand side of a subtraction operation. + +We propose allowing nested custom character classes. + +### `\U` + +In PCRE, if `PCRE2_ALT_BSUX` or `PCRE2_EXTRA_ALT_BSUX` are specified, `\U` matches literal `U`. However in ICU, `\Uhhhhhhhh` matches a hex sequence. We propose following the ICU behavior. + +### `{,n}` + +This quantifier is supported by Oniguruma, but in PCRE it matches the literal characters `{`, `,`, `n`, and `}` in sequence. We propose supporting it as a quantifier. + +### `\DDD` + +This syntax is implemented in a variety of different ways depending on the engine. In ICU and Java, it is always a backreference unless prefixed with `0`, in which case it is an octal sequence. + +In PCRE, Oniguruma, and .NET, it is also always an octal sequence if prefixed with `0`, however there are other cases where it may be treated as octal. These cases vary slightly between the engines. In PCRE, it will be treated as backreference if any of the following hold: + +- Its value is `0 < n < 10`. +- Its first digit is `8` or `9`. +- Its value corresponds to a valid *prior* group number. + +Otherwise it is treated as an octal sequence. + +Oniguruma follows all of these except the second. If the first digit is `8` or `9`, it is instead treated as the literal number, e.g `\81` is `81`. .NET also follows this behavior, but additionally has the last condition consider *all* groups, not just prior ones (as backreferences can refer to future groups in recursive cases). + +We propose a simpler behavior more inline with ICU and Java. A `\DDD` sequence that does not start with a `0` will be treated as a backreference, otherwise it will be treated as an octal sequence. If an invalid backreference is formed with this syntax, we will suggest prefixing with a `0` if an octal sequence is desired. + +One further difference exists between engines in the octal sequence case. In ICU, up to 3 additional digits are read after the `0`. In PCRE, only 2 additional digits may be interpreted as octal, the last is literal. We will follow the ICU behavior, as it is necessary when requiring a `0` prefix. + +### `\x` + +In PCRE, a bare `\x` denotes the NUL character (`U+00`). In Oniguruma, it denotes literal `x`. We propose following the PCRE behavior. + +### Whitespace in ranges + +In PCRE, `x{2,4}` is a range quantifier meaning that `x` can be matched from 2 to 4 times. However if any whitespace is introduced within the braces e.g `x{2, 4}`, it becomes an invalid range and is then treated as the literal characters instead. We find this behavior to be unintuitive, and therefore propose parsing any intermixed whitespace in the range. + +### Implicitly-scoped matching option scopes + +PCRE and Oniguruma both support changing the active matching options through an isolated group e.g `(?i)`. However, they have differing semantics when it comes to their scoping. In Oniguruma, it is treated as an implicit new scope that wraps everything until the end of the current group. In PCRE, it is treated as changing the matching option for all the following expressions until the end of the group. + +These sound similar, but have different semantics around alternations, e.g for `a(?i)b|c|d`, in Oniguruma this becomes `a(?i:b|c|d)`, where `a` is no longer part of the alternation. However in PCRE it becomes `a(?i:b)|(?i:c)|(?i:d)`, where `a` remains a child of the alternation. + +We propose matching the PCRE behavior. + +### Backreference condition kinds + +PCRE and .NET allow for conditional patterns to reference a group by its name without any form of delimiter, e.g: + +``` +(?x)?(?(group1)y) +``` + +where `y` will only be matched if `(?x)` was matched. PCRE will always treat such syntax as a backreference condition, however .NET will only treat it as such if a group with that name exists somewhere in the regex (including after the conditional). Otherwise, .NET interprets `group1` as an arbitrary regular expression condition to try match against. Oniguruma on the other hand will always treat `group1` as an regex condition to match against. + +We propose parsing such conditions as an arbitrary regular expression condition, as long as they do not conflict with other known condition spellings such as `R&name`. If the condition has a name that matches a named group in the regex, we will emit a warning asking users to explicitly use the syntax `(?()y)` if they want a backreference condition. This more explicit syntax is supported by both PCRE and Oniguruma. + +### `\N` + +PCRE supports `\N` meaning "not a newline", however there are engines that treat it as a literal `N`. We propose supporting the PCRE behavior. + +### Extended character property syntax + +ICU unifies the character property syntax `\p{...}` with the syntax for POSIX character classes `[:...:]`. This has two effects: + +- They share the same internal grammar, which allows the use of any Unicode character properties in addition to the POSIX properties. +- The POSIX syntax may be used outside of custom character classes, unlike in PCRE and Oniguruma. + +We propose following both of these rules. The former is purely additive, and therefore should not conflict with regex engines that implement a more limited POSIX syntax. The latter does conflict with other engines, but we feel it is much more likely that a user would expect e.g `[:space:]` to be a character property rather than the character class `[:aceps]`. We do however feel that a warning might be warranted in order to avoid confusion. + +### POSIX character property disambiguation + +PCRE, Oniguruma and ICU allow `[:` to be part of a custom character class if a closing `:]` is not present. For example, `[:a]` is the character class of `:` and `a`. However they each have different rules for detecting the closing `:]`: + +- PCRE will scan ahead until it hits either `:]`, `]`, or `[:`. +- Oniguruma will scan ahead until it hits either `:]`, `]`, or the length exceeds 20 characters. +- ICU will scan ahead until it hits a known escape sequence (e.g `\a`, `\e`, `\Q`, ...), or `:]`. Note this excludes character class escapes e.g `\d`. It also excludes `]`, meaning that even `[:a][:]` is parsed as a POSIX character property. + +We propose unifying these behaviors by scanning ahead until we hit either `[`, `]`, `:]`, or `\`. Additionally, we will stop on encountering `}` or a second occurrence of `=`. These fall out the fact that they would be invalid contents of the alternative `\p{...}` syntax. + + +### Script properties + +Shorthand script property syntax e.g `\p{Latin}` is treated as `\p{Script=Latin}` by PCRE, ICU, Oniguruma, and Java. These use [the Unicode Script property][unicode-scripts], which assigns each scalar a particular script value. However, there are scalars that may appear in multiple scripts, e.g U+3003 DITTO MARK. These are often assigned to the `Common` script to reflect this fact, which is not particularly useful for matching purposes. To provide more fine-grained script matching, Unicode provides [the Script Extension property][unicode-script-extensions], which exposes the set of scripts that a scalar appears in. + +As such we feel that the more desirable default behavior of shorthand script property syntax e.g `\p{Latin}` is for it to be treated as `\p{Script_Extension=Latin}`. This matches Perl's default behavior. Plain script properties may still be written using the more explicit syntax e.g `\p{Script=Latin}` and `\p{sc=Latin}`. + +### Extended syntax modes + +Various regex engines offer an "extended syntax" where whitespace is treated as non-semantic (e.g `a b c` is equivalent to `abc`), in addition to allowing end-of-line comments `# comment`. In both PCRE and Perl, this is enabled through the `(?x)`, and in later versions, `(?xx)` matching options. The former allows non-semantic whitespace outside of character classes, and the latter also allows non-semantic whitespace in custom character classes. + +ICU and Java however enable the more broad behavior under `(?x)`. We propose following this behavior, with `(?x)` and `(?xx)` being treated the same. + +Different regex engines also have different rules around what characters are considered non-semantic whitespace. When compiled with Unicode support, PCRE follows the `Pattern_White_Space` Unicode property, which consists of the following scalars: + +- The space character `U+20` +- Whitespace characters `U+9...U+D` +- Next line `U+85` +- Left-to-right mark `U+200E` +- Right-to-left mark `U+200F` +- Line separator `U+2028` +- Paragraph separator `U+2029` + +This is the same set of scalars matched by `UnicodeScalar.Properties.isPatternWhitespace`. Additionally, in a custom character class, PCRE only considers the space and tab characters as whitespace. Other engines do not differentiate between whitespace characters inside and outside custom character classes, and appear to follow a subset of this list. Therefore we propose supporting exactly the characters in this list for the purposes of non-semantic whitespace parsing. + +### Group numbering + +In PCRE, groups are numbered according to the position of their opening parenthesis. .NET also follows this rule, with the exception that named groups are numbered after unnamed groups. For example: + +``` +(a(?x)b)(?y)(z) +^ ^ ^ ^ +1 3 4 2 +``` + +The `(z)` group gets numbered before the named groups get numbered. + +We propose matching the PCRE behavior where groups are numbered purely based on order. + +### Duplicate group names + +By default, Oniguruma, Perl, and .NET allow duplicate capture group names for differently numbered captures. PCRE also allows this when `(?J)` is set. However, each engine has a different backreference behavior to such captures: + +- PCRE and Perl refer to the first matched group with that name. +- .NET refers to the last matched group with that name. +- Oniguruma allows a reference to any of the previously matched values of the groups with that name. + +We feel that this behavior can be unintuitive, and therefore intend to make duplicate group names invalid by default for differently numbered captures. This follows the behavior of ICU, Java, and PCRE's default behavior. + +## Swift canonical syntax + +The proposed syntactic superset means there will be multiple ways to write the same thing. Below we discuss what Swift's preferred spelling could be, a "Swift canonical syntax". + +We are not formally proposing this as a distinct syntax or concept, rather it is useful for considering compiler features such as fixits, pretty-printing, and refactoring actions. We're hoping for further discussion with the community here. Useful criteria include how well the choice fits in with the rest of Swift, whether there's an existing common practice, and whether one choice is less confusing in the context of others. + +[Unicode scalar literals](#unicode-scalars) can be spelled in many ways. We propose treating Swift's string literal syntax of `\u{HexDigit{1...}}` as the preferred spelling. + +Character properties can be spelled `\p{...}` or `[:...:]`. We recommend preferring `\p{...}` as the bracket syntax historically meant POSIX-defined character classes, and still has that connotation in some engines. The [spelling of properties themselves can be fuzzy](#character-properties) and we (weakly) recommend the shortest spelling (no opinion on casing yet). For script extensions, we (weakly) recommend e.g. `\p{Greek}` instead of `\p{Script_Extensions=Greek}`. We would like more discussion with the community here. + +[Lookaround assertions](#lookahead-and-lookbehind) have common shorthand spellings, while PCRE2 introduced longer more explicit spellings. We are (very weakly) recommending the common short-hand syntax of e.g. `(?=...)` as that's wider spread. We are interested in more discussion with the community here. + +Named groups may be specified with a few different delimiters: `(?...)`, `(?P...)`, `(?'name'...)`. We (weakly) recommend `(?...)`, but the final preference may be influenced by choice of delimiter for the regex itself. We'd appreciate any insight from the community. + +[Backreferences](#backreferences) have multiple spellings. For absolute numeric references, `\DDD` seems to be a strong candidate for the preferred syntax due to its familiarity. For relative numbered references, as well as named references, either `\k<...>` or `\k'...'` seem like the better choice, depending on the syntax chosen for named groups. This avoids the confusion between `\g{...}` and `\g<...>` referring to a backreferences and subpatterns respectively, as well as any confusion with group syntax. + +For [subpatterns](#subpatterns), we recommend either `\g<...>` or `\g'...'` depending on the choice for named group syntax. We're unsure if we should prefer `(?R)` as a spelling for e.g. `\g<0>` or not, as it is more widely used and understood, but less consistent with other subpatterns. + +[Conditional references](#conditionals) have a choice between `(?('name'))` and `(?())`. The preferred syntax in this case would likely reflect the syntax chosen for named groups. + +We are deferring runtime support for callouts from regex literals as future work, though we will correctly parse their contents. We have no current recommendation for a preference of PCRE-style [callout syntax](#callouts), and would like to discuss with the community whether we should have one. + +## Alternatives Considered + +### Failable inits + +There are many ways for compilation to fail, from syntactic errors to unsupported features to type mismatches. In the general case, run-time compilation errors are not recoverable by a tool without modifying the user's input. Even then, the thrown errors contain valuable information as to why compilation failed. For example, swiftpm presents any errors directly to the user. + +As proposed, the errors thrown will be the same errors presented to the Swift compiler, tracking fine-grained source locations with specific reasons why compilation failed. Defining a rich error API is future work, as these errors are rapidly evolving and it is too early to lock in the ABI. + + +### Skip the syntax + +The top alternative is to just skip regex syntax altogether by only shipping the result builder DSL and forbidding run-time regex construction from strings. However, doing so would miss out on the familiarity benefits of existing regex syntax. Additionally, without support for run-time strings containing regex syntax, important domains would be closed off from better string processing, such as command-line tools and user-input searches. This would land us in a confusing world where NSRegularExpression, even though it operates over a fundamentally different model of string than Swift's `String` and exhibits different behavior than Swift regexes, is still used for these purposes. + +We consider our proposed direction to be more compelling, especially when coupled with refactoring actions to convert literals into regex DSLs. + +### Introduce a novel regex syntax + +Another alternative is to invent a new syntax for regex. This would similarly lose out on the familiarity benefit, though a few simple adjustments could aid readability. + +We are prototyping an "experimental" Swift extended syntax, which is future work and outside the scope of this proposal. Every syntactic extension, while individually compelling, does introduce incompatibilities and can lead to an "uncanny valley" effect. Further investigation is needed and such support can be built on top of what is presented here. + +### Support a minimal syntactic subset + +Regex syntax will become part of Swift's source and binary-compatibility story, so a reasonable alternative is to support the absolute minimal syntactic subset available. However, we would need to ensure that such a minimal approach is extensible far into the future. Because syntax decisions can impact each other, we would want to consider the ramifications of this full syntactic superset ahead of time anyways. + +Even though it is more work up-front and creates a longer proposal, it is less risky to support the full intended syntax. The proposed superset maximizes the familiarity benefit of regex syntax. + +### Future: Capture descriptions on Regex + +Future API could include a description of the capture list that a regex contains, provided as a collection of optionally-named captures and their types. This would further enhance dynamic regexes. + + + + + +[pcre2-syntax]: https://www.pcre.org/current/doc/html/pcre2syntax.html +[oniguruma-syntax]: https://github.com/kkos/oniguruma/blob/master/doc/RE +[icu-syntax]: https://unicode-org.github.io/icu/userguide/strings/regexp.html +[uts18]: https://www.unicode.org/reports/tr18/ +[.net-syntax]: https://docs.microsoft.com/en-us/dotnet/standard/base-types/regular-expressions +[UAX44-LM3]: https://www.unicode.org/reports/tr44/#UAX44-LM3 +[unicode-prop-key-aliases]: https://www.unicode.org/Public/UCD/latest/ucd/PropertyAliases.txt +[unicode-prop-value-aliases]: https://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt +[unicode-scripts]: https://www.unicode.org/reports/tr24/#Script +[unicode-script-extensions]: https://www.unicode.org/reports/tr24/#Script_Extensions +[balancing-groups]: https://docs.microsoft.com/en-us/dotnet/standard/base-types/grouping-constructs-in-regular-expressions#balancing-group-definitions +[overview]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md +[pitches]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md + + + diff --git a/proposals/0356-swift-snippets.md b/proposals/0356-swift-snippets.md new file mode 100644 index 0000000000..e1ab8692c2 --- /dev/null +++ b/proposals/0356-swift-snippets.md @@ -0,0 +1,427 @@ +# Swift Snippets + +* Proposal: [SE-0356](0356-swift-snippets.md) +* Authors: [Ashley Garland](https://github.com/bitjammer) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.7)** +* Implementation: + Available in [recent nightly](https://swift.org/download/#snapshots) snapshots. Requires `--enable-experimental-snippet-support` feature flag when using the [Swift DocC Plugin](https://github.com/apple/swift-docc-plugin). Related pull requests: + * Swift DocC + * [Add snippet support](https://github.com/apple/swift-docc/pull/61) + * Swift Package Manager: + * [Introduce the snippet target type](https://github.com/apple/swift-package-manager/pull/3694) + * [Rename _main symbol when linking snippets](https://github.com/apple/swift-package-manager/pull/3732) + * SymbolKit: + * [Add snippet mixin for symbols](https://github.com/apple/swift-docc-symbolkit/pull/10) + * [Add .snippet and .snippetGroup kind](https://github.com/apple/swift-docc-symbolkit/pull/15) + * Swift DocC Plugin + * [Swift DocC Plugin Snippets](https://github.com/apple/swift-docc-plugin/pull/7) +* Review threads + * [Pitch](https://forums.swift.org/t/pitch-swift-snippets/56348) + * [Review](https://forums.swift.org/t/se-0356-swift-snippets/57097) + + +## Introduction + +This proposal describes a convention for writing a new form of sample code called *snippets*. Snippets are short, single-file examples that can build and run from within a Swift package, with access to other code within that package, and can be used in a variety of ways. + +## Motivation + +There are two main vehicles people employ when they want to use code to demonstrate an idea or API: + +* Complete sample projects +* Bits of code displayed inline within documentation + +Both of these are critical tools, and snippets aren’t intended to replace either of them. Instead, snippets offer an additional method for teaching with code, that fits somewhere between both ideas. First, let’s look at the current options available to most developers. + +### Sample code projects + +Sample code is often created as a full project with build configuration files, resources, and multiple source files to produce a finished “app”. These sample projects are useful when you want to show code in a specific, complete scenario. However, these projects also tend to be a lot of effort to create and maintain. For this reason, developers often simply don’t build great samples. + +Because sample code projects require more time and effort, they tend to become "kitchen sink" examples that show anything and everything around a particular topic, or grow to exemplify multiple topics and libraries. Not only does this make the project increasingly difficult to maintain, it also makes it more difficult for a reader to navigate and find the gems that may be hidden in a sample code project. + +### Code listings within documentation + +Code listings are generally presented as a few lines of code printed inline within larger bits of documentation prose, most often carved out in a small “code box” area. This code is generally authored right along side the prose by a writer, in their favorite word processor. While these bits of code are incredibly helpful while reading the documentation — often this is the best sort of documentation — there are downsides, too. + +**Code listings tend to go stale.** Code listings, once put into the documentation, tend to be treated as regular text. The code isn’t built regularly, and may not be revisited by the author for a long period of time. Over time, changes to the programming language or APIs can easily make code listings go stale and stop compiling successfully. A larger documentation team may build bespoke systems to extract and test this code — time better spent writing great new documentation. Making it easy to add code listings to documentation that can also be built and run (and validated) is one of the main goals of snippets. + +**Code listings don't get good editor support.** As most code listings are typed into an editor focused on writing prose, the author misses out on the coding features typically available in a code editor or IDE. Missing inline error checking and syntax highlighting means it is much more likely the code sample will have an error. + +**Code listings tend to be more like pseudocode.** This happens because the author knows they aren’t actually building running code, and all the explanation for the code happens in the surrounding prose. This results in code that is much less useful for the reader to copy into their own projects. + +### Snippets combine the best of both + +Snippets are designed to provide some of the best features of each of the above approaches. Each snippet is a single file, making it easy to think about as an author and as a reader, but it is also a fully valid program. Each snippet can access the full code and features of the rest of the package, so behavior can be powerful, while the code in each snippet remains simple. This means the code should also be small enough to present inline within documentation — perfect to act as a code listing. This code is able to be tested and maintained as fully-functional code ready to be copied and used by the reader. + +Snippets fill a gap in the spectrum of example-oriented documentation, shown below roughly in decreasing granularity: + +* **API reference and inline code fragments.** Not typically compilable, these usually occur in lists, an index, or perhaps a link to a symbol page. These are not compositional in nature. +* **Snippets.** Here, one file demonstrates one task, composing one or more APIs, across no more than a handful of modules. A snippet should basically be something a reader can copy and paste or use as a starting point. Some examples of a snippet might be: + * An implementation of a sort algorithm + * An interesting SwiftUI view hierarchy + * A quick recipe for displaying a 3D model in a view + * A struct that demonstrates a Swift Argument Parser option + * An example of Swift Coding with custom coding keys + * Many of the examples in [Swift Algorithms’s “Guides”](https://github.com/apple/swift-algorithms/tree/main/Guides) + * Many StackOverflow answers +* **Full sample projects and tutorials.** Here, a project demonstrates a full application, composing not just one or more APIs, but potentially many technologies or modules. A sample project also demonstrates multiple, potentially independent scenarios. Most developers are already familiar with these. + +**Once written, snippets are useful in many contexts.** Both sample projects and inline code listings are written once and meant to be consumed in one particular manner. In contrast, snippets are meant to be written once but read (or even run) anywhere. Snippets are just simple bits of code (but with access to the full package), great for importing within documentation prose, runnable from the command line, or copied and edited within an IDE. + +**Short, focused, single files.** This versatility comes with constraints — snippets should be small and focused, for instance. They should also stand on their own and not require a complex scenario to be understood. With these constraints, snippets can then be easily shuttled around, shown inline in docs, used in interactive code tutorials, run from the command line, quickly gleaned, and provide useful code for a developer to take on the spot. As soon as a snippet feels like it needs multiple files or resources, a traditional sample project starts to become appropriate. But with full access to the rest of the package, it may make sense to group “big” functionality elsewhere in the package, allowing each snippet to remain small, focused, and easily understood. + +**The possibility of snippet-only packages.** While sample code projects' strength is their depth in a specific scenario (often application development), packages consisting mostly or entirely of snippets provide breadth. Examples of snippet-only packages might be a collection of recipes for composing UI elements in new and interesting ways, teaching the Swift language snippet by snippet, or providing exercises for a textbook. Again, since snippets get access to the package's shared code libraries, it is possible to demonstrate even powerful concepts in an easy-to-read snippet. + +## Proposed Solution + +This proposal is a definition of a sample code convention that offers a bite-sized hybrid of sample code and API reference, with a sprinkle of prose: snippets. Snippets are individual `.swift` files that can build and run as executable targets in a Swift package. Each snippet is a complete program, that stands on its own as a token of documentation. + +### Writing a snippet + +A snippet file might look like the following: + +```swift +// The first contiguous line comments +// serve as the snippet's short description. + +func someCodeToShow() { + print("Hello, world!") +} + +// snippet.hide + +func someCodeToHide() { + print("Some demo message") +} + +// Still hidden +someCodeToHide() + +// snippet.show + +someCodeToShow() +``` + +At the top is the snippet's description written in Markdown, typically a short paragraph that may appear with the snippet. `// snippet.hide` and `// snippet.show` toggle hiding and showing code when displaying a snippet in documentation or other tools that may support snippets in the future. This lets the author add some additional demo logic when running the snippet while still keeping its presentation clean when it shows up the finished documentation. + +The above snippet would end up looking something like this within the docs: + +> The first contiguous line comments serve as the snippet's short description. +> +> ```swift +> func someCodeToShow() { +> print("Hello, world!") +> } +> someCodeToShow() +> ``` + +This code extracted after resolving hiding markers is known as a snippet’s *presentation code*. + +### Slices + +When snippets exist in documentation, each code block often continues from the previous one in a sequential narrative, alternating between code and prose. For example: + +> First, call `setup()` to initialize the context: +> +> ```swift +> let context = setup() +> ``` +> +> Then, call `request(_:)` with the desired mode: +> +> ```swift +> context.request(.immediate) +> ``` + +The second code block refers to `context` defined in the first so, for the purposes of compilation, the snippet is comprised of two code blocks. To support this, an author can write the code in a single file and "slice" it with an identifiers, referring to them in the documentation. Here is what the snippet for the above might look like in the Swift source file: + +```swift +// snippet.setup +let context = setup() + +// snippet.request +context.request(.immediate) +``` + +The special comment marker takes the form `// snippet.IDENTIFIER`, where `IDENTIFIER` is a URL-compatible path component in order to be compatible with DocC link resolution logic. Starting a new slice automatically terminates the previous slice. For slices that aren't adjacent, one can use `// snippet.end` to end the current slice: + +```swift +// snippet.setup +let context = setup() +// snippet.end + +// More code here... + +// snippet.request +context.request(.immediate) +``` + +You can also mix show/hide markers with slices: + +```swift +// snippet.setup +let context = setup() + +// snippet.hide +// More code here... +// snippet.show + +// snippet.request +context.request(.immediate) +``` + +### Getting started + +To start adding snippets to a package, first create a `Snippets` directory alongside a package's familiar `Sources` and `Tests` directories. From here, you can start dropping in `.swift` files. Base filenames must be unique from one another. SwiftPM will assume each of these comprise their own executable targets, so it can build and run them for the host platform as it would any other executable. + +After getting started, a package might start to look like the following: + + +``` +📁 MyPackage + 📁 Package.swift + 📁 Sources + 📁 Tests + 📂 Snippets + 📄 Snippet1.swift + 📄 Snippet2.swift + 📄 Snippet3.swift + +``` + +### Grouping + +To help organizing a growing number of snippets, you can also create one additional level of subdirectories under `Snippets` . This does not affect snippet links as shown below. + + +``` +📁 MyPackage + 📁 Package.swift + 📁 Sources + 📁 Tests + 📂 Snippets + 📁 Group1 + 📄 Snippet1.swift + 📄 Snippet2.swift + 📄 Snippet3.swift + 📁 Group2 + 📄 Snippet4.swift + 📄 Snippet5.swift +``` + +### Overriding the location of snippets + +Similar to the `./Sources` and `./Tests` directories, a user may want to override the location of the `./Snippets` directory with a new, optional `snippetsDirectory` argument to the `Package` initializer. Since snippet targets aren't declared individually in the manifest, the setting exists at the package level. + +```swift +let package = Package( + name: "MyPackage", + snippetsDirectory: "Examples", + products: [ + // ... + ], + dependencies: [ + // ... + ], + targets: [ + // ... + ] +) +``` + +> For the remainder of the document, examples will assume the default `Snippets` directory. + +### Using snippets in Swift-DocC documentation + +Swift-DocC (or other documentation tools) can then import snippets within prose Markdown files. For DocC, a snippet's description and code will appear anywhere you use a new block directive called `@Snippet`, with a single required `path` argument: + + + +```swift +@Snippet(path: "my-package/Snippets/Snippet1") +``` + +The `path` argument consists of the following three components: + + +* `my-package` : The package name, as taken from the `Package.swift` manifest. +* `Snippets`: An informal namespace to differentiate snippets from symbols and articles. This is the same regardless of the `snippetsDirectory` override mentioned above. +* `Snippet1`: The snippet name taken from the snippet file basename without extension. + +To insert a snippet slice, add the optional `slice` argument with the matching identifier in the source: + +```swift +@Snippet(path: "my-package/Snippets/Snippet1", slice: "setup") +``` + +### Building and running snippets + +After creating snippets, the Swift Package Manager can build and run them in the same way as executable targets. + +Snippet targets will be built by default when running `swift build --build-snippets`. This is consistent with how building tests is an explicit choice, like when running `swift test` or `swift build --build-tests`. It’s recommended to build snippets in all CI build processes or at least when building documentation. + +Example usage: + +```bash +swift build # Builds source targets as usual, + # but excluding tests and snippets. + +swift build --build-snippets # Builds source targets, including snippets. + +swift build Snippet1 # Build the snippet, Snippet1.swift, as an executable. + +swift run Snippet1 # Run the snippet, Snippet1.swift. +``` + +### Testing snippets + +While the code exemplified in snippets should already be covered by tests, an author may want to assert specific behavior when running a snippet. While we could use a test library like XCTest, it comes with platform-specific considerations and difficulties with execution–XCTest assertions can’t be collected and logged without a platform-specific test harness to execute the tests. Again, thinking about snippets as a kind of executable, how does one assert behavior in an executable? With asserts and preconditions. These should be enough for a majority of use cases, while letting interactive and non-interactive snippets to live side-by-side and treated the same for now. It is important that snippets are testable within CI and external testing solutions to provide additional automation to make this happen in one step. + +**Example:** + +```swift +let numbers = [20, 19, 7, 12] +let numbersMap = numbers.map({ (number: Int) -> Int in + return 3 * number +}) + +// snippet.hide +print(numbersMap) +precondition(numbersMap == [60, 57, 21, 36]) +``` + +## Detailed Design + +### Swift Package Manager + +When constructing the set of available targets for a package, SwiftPM will automatically find snippet files with the following pattern: + +* `./Snippets/**.swift*` +* `./Snippets/*/*.swift` + +These will each become a new kind of `.snippet` target behaving more or less as existing executable targets. A single level of subdirectories is allowed to balance filesystem organization and further subdirectories for snippet-related resources, which are expected to be found informally using relative paths. + +Snippet targets automatically depend on any library targets declared in their host package, so snippets are free to import those modules. In the future, in order to support snippet-only packages, packages that illustrate combining two independenct packages, or packages that require helper libraries for snippets, snippets will be able to import libraries from dependent packages declared in the manifest as well (see Future Directions below). + +### SymbolKit + +Snippets will be communicated to DocC via Symbol Graph JSON, with each snippet becoming a kind of symbol. + +Snippet symbols will include two primary pieces of information: a description carried as the symbol’s “documentation comment”, and presentation code via new mix-in called `Snippet`: + +```swift +public struct Snippet: Mixin, Codable { + public struct Slice: Codable { + public var name: String? + public var language: String? + public var code: String + } + public var slices: [Slice] +} +``` + +When a snippet doesn't have any slice comments, the above snippet model will consist of one slice containing all of the visible code. + +### Swift-DocC + +Swift-DocC will need to do the following to support snippets: + +**Look for and register occurrences of the new snippet mix-ins in symbol graph JSON.** By treating snippets as symbols, this mostly comes for free with the SymbolKit data model. + +**Add support for the new `@Snippet` directive**, checking the `path` and `slice` arguments with the same logic as symbol links. This comes in the form of a new `Semantic` instance: + +```swift +public final class Snippet: Semantic, DirectiveConvertible { + public static let directiveName = "Snippet" + // etc. +} +``` + +**Convert** `@Snippet` **occurrences to paragraphs and code blocks** as needed in the `RenderContentCompiler`, resulting in the following content for each occurrence: + +If the `@Snippet` is a slice, only: +* The slice code as a `CodeBlock`. + +If the `@Snippet` is not a slice: +* The documentation comment Markdown processed as normal for a symbol, a list of block elements. +* For each snippet slice: + * The slice code as a `CodeBlock`. + +### Swift-DocC Plugin + +The recently added [Swift DocC Plugin](https://github.com/apple/swift-docc-plugin) is a new [SwiftPM command plugin](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md) that builds documentation for SwiftPM libraries and executables. + +In order to forward a package's snippet information to DocC, a new tool, `snippet-build`, is added to convert `.swift` files into Symbol Graph JSON, which the plugin will run before `docc`. + +The `snippet-build` tool crawls the `Snippets` directory structure in the same way as SwiftPM, looking for `.swift` files. For each file, a snippet symbol entry is created in a Symbol Graph, and emitted into an output directory. The tool’s usage looks like the following: + +``` +USAGE: snippet-build + +ARGUMENTS: + - The directory containing Swift snippets + - The directory in which to place Symbol Graph JSON file(s) representing the snippets + - The module name to use for the Symbol Graph (typically should be the package name) +``` + +It’s not expected that a person will run this command manually. + +#### A note on Swift plugin dependencies + +Because SwiftPM plugins fold their dependencies into the plugin client’s dependency graph, some useful but minor dependencies were dropped to prevent the possibility for dependency cycles or conflicts: + +* **Swift Argument Parser.** This is a common dependency for lots of packages so the `snippet-build` tool implements argument parsing manually using positional arguments. It’s not expected that the usage will change over time. +* **Swift Syntax.** This could be useful for tokenizing code blocks, but DocC implements syntactic highlighting in the `Swift-DocC-Render` project. + +This current restriction on dependencies is one motivating factor for investigating moving Symbol Graph generation from `.swift` files down to the compiler. This would have nearly identical usage to [existing functionality to emit library and executable symbol graphs](https://github.com/apple/swift/tree/main/lib/SymbolGraphGen) today. More on this below. + +## Source compatibility + +Proposed changes to enable snippets do not break source compatibility. + +## Effect on ABI stability + +Proposed changes to enable snippets do not break ABI stability. + +## Effect on API resilience + +Proposed changes to enable snippets do not impact API resilience. + +## Alternatives considered + +### Literate approach: snippets within Markdown + +Another option was to support something like “[literate programming](http://www.literateprogramming.com/)” where source code is embedded in Markdown documentation files. In this approach, new tools and workflows would be created to extract code from the documentation, assemble that code into valid Swift files or packages, then build, run, and test that code. That tooling would likely use a custom file format with the ability to hide setup and test code, control imports, and more. The goal is to let documentation authors write bits of code inline, but to add tooling to validate the code. Literate programming is very interesting, and may be a good project for Swift, but it is not a small undertaking, and not likely to integrate well with existing tooling. + +Snippets, in contrast, are intended to primarily act as small sample programs that work with existing tooling. It should be super easy for anyone to look at a snippet as just source code, see how it works, remix it, and run it. Snippets should be easy to share, and even paste into a StackOverflow answer. + +At their core, snippets are simply `.swift` files, with conventions in place to make them really easy to fit into existing documentation tools, editors, IDEs, CLI commands, and CI systems. Code conforming to the snippets convention is straight forward to support within [Swift-DocC](https://github.com/apple/swift-docc) documentation tooling, as well as to build a nice CLI to discover, view, and quickly run snippets within a package. + +### Snippets in documentation comments + +Writing snippets exclusively in documentation comments limits their utility, putting too much focus on only documenting APIs within a module. More interesting uses for snippets would be left behind, such as composing functionality across multiple modules, or packages of just snippets for educational purposes. + +### Snippets as playgrounds + +Why aren’t these just playgrounds? While playgrounds started out very similarly to snippets, they have evolved into something more powerful, more tied to custom tooling, and a bit more complex. Playgrounds tend to tell a story, and are stand-alone entities with their own supporting files and sources. + +For open source Swift, packages already have a model for building targets that have multiple files and resources, and in fact, we’re seeing playgrounds migrating more toward looking like packages. + +Snippets are intentionally small programs written as a simple `.swift` file, integrated closely with the Swift Package Manager approach, as is Swift-DocC. + +### Tests acting as snippets + +Snippets are not meant to be tests or come directly from tests, although they may include their own testing and assertions to validate behavior. While tests may use public API in similar ways, the context in which one writes and thinks about tests is usually different from writing example code. For those tests that do match common use cases very well, it may be possible in the future to extract snippets from multiple sources (see below). + +## Future Directions + +**Multiple snippets per file.** In the future there is the option to create multiple snippets per file, where each snippet’s identifier is expressed as a kind of start/end marker in source code. + +**Multi-file snippets.** This could manifest in a couple ways. First, requiring several files to build a snippet already exists in the form sample target or project, so this is probably not a future goal. However, for snippets embedded within existing multi-file projects, it may be possible extract those snippets during build time. This will likely require that the snippet extraction move down to the compiler. + +**Extract snippets while building.** To facilitate some of the above future possibilities and others, the `snippet-build` tool may move down to the `SymbolGraphGen` library that coverts modules into Symbol Graph JSON. Since snippets are communicated with the same Symbol Graph format, moving the implementation down to the compiler will allow utilizing shared implementation and semantic information for future enhancements. This would allow snippets to be pulled from different kinds of sources: from libraries, unit tests, larger sample projects, etc. + +**Build snippets when building documentation.** The current Swift-DocC implementation only requires reading snippet source files when rendering documentation, so building is not required. Depending on whether the implementation is moved down to the compiler, this could be implemented by having the Swift-DocC plugin request snippet builds before generating documentation, or implicitly as the compiler builds snippets to generate symbol graphs. + +**Snippet dependencies.** While snippets automatically depend on any libraries defined in their host package, there may be packages that exist solely to illustrate using one or more libraries from other packages. In the future, we can add the ability to declare external dependencies to which snippets have access. diff --git a/proposals/0357-regex-string-processing-algorithms.md b/proposals/0357-regex-string-processing-algorithms.md new file mode 100644 index 0000000000..1cbd60d3f3 --- /dev/null +++ b/proposals/0357-regex-string-processing-algorithms.md @@ -0,0 +1,1153 @@ +# Regex-powered string processing algorithms + +* Proposal: [SE-0357](0357-regex-string-processing-algorithms.md) +* Authors: [Tina Liu](https://github.com/itingliu), [Michael Ilseman](https://github.com/milseman), [Nate Cook](https://github.com/natecook1000), [Tim Vermeulen](https://github.com/timvermeulen) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift-experimental-string-processing](https://github.com/apple/swift-experimental-string-processing/) + * Available in nightly toolchain snapshots with `import _StringProcessing` +* Review: ([pitch](https://forums.swift.org/t/pitch-regex-powered-string-processing-algorithms/55969)) + ([review](https://forums.swift.org/t/se-0357-regex-string-processing-algorithms/57225)) + ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0357-regex-string-processing-algorithms/58706)) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7741017763f528dfbdfa54c6d11f559918ab53e4/proposals/0357-regex-string-processing-algorithms.md) + +## Introduction + +The Swift standard library's string processing algorithms are underpowered compared to other popular programming and scripting languages. Some of these omissions can be found in `NSString`, but these fundamental algorithms should have a place in the standard library. + +We propose: + +1. New regex-powered algorithms over strings, bringing the standard library up to parity with scripting languages +2. Generic `Collection` equivalents of these algorithms in terms of subsequences +3. `protocol CustomConsumingRegexComponent`, which allows 3rd party libraries to provide their industrial-strength parsers as intermixable components of regexes + +This proposal is part of a larger [regex-powered string processing initiative](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md), the status of each proposal is tracked [here](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md). Further discussion of regex specifics is out of scope of this proposal and better discussed in their relevant reviews. + +## Motivation + +A number of common string processing APIs are missing from the Swift standard library. While most of the desired functionalities can be accomplished through a series of API calls, every gap adds a burden to developers doing frequent or complex string processing. For example, here's one approach to find the number of occurrences of a substring ("banana") within a string: + +```swift +let str = "A banana a day keeps the doctor away. I love bananas; banana are my favorite fruit." + +var idx = str.startIndex +var ranges = [Range]() +while let r = str.range(of: "banana", options: [], range: idx.. + Comparison of how Swift's APIs stack up with Python's. + +Note: Only a subset of Python's string processing API are included in this table for the following reasons: + +- Functions to query if all characters in the string are of a specified category, such as `isalnum()` and `isalpha()`, are omitted. These are achievable in Swift by passing in the corresponding character set to `allSatisfy(_:)`, so they're omitted in this table for simplicity. +- String formatting functions such as `center(length, character)` and `ljust(width, fillchar)` are also excluded here as this proposal focuses on matching and searching functionalities. + +##### Search and replace + +|Python |Swift | +|--- |--- | +| `count(sub, start, end)` | | +| `find(sub, start, end)`, `index(sub, start, end)` | `firstIndex(where:)` | +| `rfind(sub, start, end)`, `rindex(sub, start, end)` | `lastIndex(where:)` | +| `expandtabs(tabsize)`, `replace(old, new, count)` | `Foundation.replacingOccurrences(of:with:)` | +| `maketrans(x, y, z)` + `translate(table)` | + +##### Prefix and suffix matching + +|Python |Swift | +|--- |--- | +| `startswith(prefix, start, end)` | `starts(with:)` or `hasPrefix(:)`| +| `endswith(suffix, start, end)` | `hasSuffix(:)` | +| `removeprefix(prefix)` | Test if string has prefix with `hasPrefix(:)`, then drop the prefix with `dropFirst(:)`| +| `removesuffix(suffix)` | Test if string has suffix with `hasSuffix(:)`, then drop the suffix with `dropLast(:)` | + +##### Strip / trim + +|Python |Swift | +|--- |--- | +| `strip([chars])`| `Foundation.trimmingCharacters(in:)` | +| `lstrip([chars])` | `drop(while:)` | +| `rstrip([chars])` | Test character equality, then `dropLast()` iteratively | + +##### Split + +|Python |Swift | +|--- |--- | +| `partition(sep)` | `Foundation.components(separatedBy:)` | +| `rpartition(sep)` | | +| `split(sep, maxsplit)` | `split(separator:maxSplits:...)` | +| `splitlines(keepends)` | `split(separator:maxSplits:...)` | +| `rsplit(sep, maxsplit)` | | + +
+ + + +### Complex string processing + +Even with the API additions, more complex string processing quickly becomes unwieldy. String processing in the modern world involves dealing with localization, standards-conforming validation, and other concerns for which a dedicated parser is required. + +Consider parsing the date field `"Date: Wed, 16 Feb 2022 23:53:19 GMT"` in an HTTP header as a `Date` type. The naive approach is to search for a substring that looks like a date string (`16 Feb 2022`), and attempt to post-process it as a `Date` with a date parser: + +```swift +let regex = Regex { + Capture { + OneOrMore(.digit) + " " + OneOrMore(.word) + " " + OneOrMore(.digit) + } +} + +let dateParser = Date.ParseStrategy(format: "\(day: .twoDigits) \(month: .abbreviated) \(year: .padded(4))" +if let dateMatch = header.firstMatch(of: regex)?.0 { + let date = try? Date(dateMatch, strategy: dateParser) +} +``` + +This requires writing a simplistic pre-parser before invoking the real parser. The pre-parser will suffer from being out-of-sync and less featureful than what the real parser can do. + +Or consider parsing a bank statement to record all the monetary values in the last column: + +```swift +let statement = """ +CREDIT 04/06/2020 Paypal transfer $4.99 +CREDIT 04/03/2020 Payroll $69.73 +DEBIT 04/02/2020 ACH transfer ($38.25) +DEBIT 03/24/2020 IRX tax payment ($52,249.98) +""" +``` + +Parsing a currency string such as `$3,020.85` with regex is also tricky, as it can contain localized and currency symbols in addition to accounting conventions. This is why Foundation provides industrial-strength parsers for localized strings. + + +## Proposed solution + +### Complex string processing + +We propose a `CustomConsumingRegexComponent` protocol which allows types from outside the standard library participate in regex builders and `RegexComponent` algorithms. This allows types, such as `Date.ParseStrategy` and `FloatingPointFormatStyle.Currency`, to be used directly within a regex: + +```swift +let dateRegex = Regex { + Capture(dateParser) +} + +let date: Date = header.firstMatch(of: dateRegex).map(\.result.1) + +let currencyRegex = Regex { + Capture(.localizedCurrency(code: "USD").sign(strategy: .accounting)) +} + +let amount: [Decimal] = statement.matches(of: currencyRegex).map(\.result.1) +``` + +### String algorithm additions + +We also propose the following regex-powered algorithms as well as their generic `Collection` equivalents. See the Detailed design section for a complete list of variation and overloads . + +|Function | Description | +|--- |--- | +|`contains(_:) -> Bool` | Returns whether the collection contains the given sequence or `RegexComponent` | +|`starts(with:) -> Bool` | Returns whether the collection contains the same prefix as the specified `RegexComponent` | +|`trimPrefix(_:)`| Removes the prefix if it matches the given `RegexComponent` or collection | +|`firstRange(of:) -> Range?` | Finds the range of the first occurrence of a given sequence or `RegexComponent`| +|`ranges(of:) -> some Collection` | Finds the ranges of the all occurrences of a given sequence or `RegexComponent` within the collection | +|`replace(:with:subrange:maxReplacements)`| Replaces all occurrences of the sequence matching the given `RegexComponent` or sequence with a given collection | +|`split(by:)`| Returns the longest possible subsequences of the collection around elements equal to the given separator | +|`firstMatch(of:)`| Returns the first match of the specified `RegexComponent` within the collection | +|`wholeMatch(of:)`| Matches the specified `RegexComponent` in the collection as a whole | +|`prefixMatch(of:)`| Matches the specified `RegexComponent` against the collection at the beginning | +|`matches(of:)`| Returns a collection containing all matches of the specified `RegexComponent` | + +## Detailed design + +### `CustomConsumingRegexComponent` + +`CustomConsumingRegexComponent` inherits from `RegexComponent` and satisfies its sole requirement. Conformers can be used with all of the string algorithms generic over `RegexComponent`. + +```swift +/// A protocol allowing custom types to function as regex components by +/// providing the raw functionality backing `prefixMatch`. +public protocol CustomConsumingRegexComponent: RegexComponent { + /// Process the input string within the specified bounds, beginning at the given index, and return + /// the end position (upper bound) of the match and the produced output. + /// - Parameters: + /// - input: The string in which the match is performed. + /// - index: An index of `input` at which to begin matching. + /// - bounds: The bounds in `input` in which the match is performed. + /// - Returns: The upper bound where the match terminates and a matched instance, or `nil` if + /// there isn't a match. + func consuming( + _ input: String, + startingAt index: String.Index, + in bounds: Range + ) throws -> (upperBound: String.Index, output: RegexOutput)? +} +``` + +
+Example for protocol conformance + +We use Foundation `FloatingPointFormatStyle.Currency` as an example for protocol conformance. It would implement the `match` function with `Match` being a `Decimal`. It could also add a static function `.localizedCurrency(code:)` as a member of `RegexComponent`, so it can be referred as `.localizedCurrency(code:)` in the `Regex` result builder: + +```swift +extension FloatingPointFormatStyle.Currency : CustomConsumingRegexComponent { + public func consuming( + _ input: String, + startingAt index: String.Index, + in bounds: Range + ) -> (upperBound: String.Index, match: Decimal)? +} + +extension RegexComponent where Self == FloatingPointFormatStyle.Currency { + public static func localizedCurrency(code: Locale.Currency) -> Self +} +``` + +Matching and extracting a localized currency amount, such as `"$3,020.85"`, can be done directly within a regex: + +```swift +let regex = Regex { + Capture(.localizedCurrency(code: "USD")) +} +``` + +
+ + +### String and Collection algorithm additions + +#### Contains + +We propose a `contains` variant over collections that tests for subsequence membership. The second algorithm allows for specialization using e.g. the [two way search algorithm](https://en.wikipedia.org/wiki/Two-way_string-matching_algorithm). + +```swift +extension Collection where Element: Equatable { + /// Returns a Boolean value indicating whether the collection contains the + /// given sequence. + /// - Parameter other: A sequence to search for within this collection. + /// - Returns: `true` if the collection contains the specified sequence, + /// otherwise `false`. + public func contains(_ other: C) -> Bool + where S.Element == Element +} +extension BidirectionalCollection where Element: Comparable { + /// Returns a Boolean value indicating whether the collection contains the + /// given sequence. + /// - Parameter other: A sequence to search for within this collection. + /// - Returns: `true` if the collection contains the specified sequence, + /// otherwise `false`. + public func contains(_ other: C) -> Bool + where S.Element == Element +} +``` + +We propose a regex-taking variant over string types (those that produce a `Substring` upon slicing). + +```swift +extension Collection where SubSequence == Substring { + /// Returns a Boolean value indicating whether the collection contains the + /// given regex. + /// - Parameter regex: A regex to search for within this collection. + /// - Returns: `true` if the regex was found in the collection, otherwise + /// `false`. + public func contains(_ regex: some RegexComponent) -> Bool +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a Boolean value indicating whether this collection contains a + /// match for the regex, where the regex is created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to search for within + /// this collection. + /// - Returns: `true` if the regex returned by `content` matched anywhere in + /// this collection, otherwise `false`. + public func contains( + @RegexComponentBuilder _ content: () -> some RegexComponent + ) -> Bool +} +``` + +#### Starts with + +We propose a regex-taking `starts(with:)` variant for string types: + +```swift +extension Collection where SubSequence == Substring { + /// Returns a Boolean value indicating whether the initial elements of the + /// sequence are the same as the elements in the specified regex. + /// - Parameter regex: A regex to compare to this sequence. + /// - Returns: `true` if the initial elements of the sequence matches the + /// beginning of `regex`; otherwise, `false`. + public func starts(with regex: some RegexComponent) -> Bool +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a Boolean value indicating whether the initial elements of this + /// collection are a match for the regex created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to match at + /// the beginning of this collection. + /// - Returns: `true` if the initial elements of this collection match + /// regex returned by `content`; otherwise, `false`. + public func starts( + @RegexComponentBuilder with content: () -> some RegexComponent + ) -> Bool +} +``` + +#### Trim prefix + +We propose generic `trimmingPrefix` and `trimPrefix` methods for collections that trim elements matching a predicate or a possible prefix sequence. + +```swift +extension Collection { + /// Returns a new collection of the same type by removing initial elements + /// that satisfy the given predicate from the start. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + /// - Returns: A collection containing the elements of the collection that are + /// not removed by `predicate`. + public func trimmingPrefix(while predicate: (Element) throws -> Bool) rethrows -> SubSequence +} + +extension Collection where SubSequence == Self { + /// Removes the initial elements that satisfy the given predicate from the + /// start of the sequence. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + public mutating func trimPrefix(while predicate: (Element) throws -> Bool) rethrows +} + +extension RangeReplaceableCollection { + /// Removes the initial elements that satisfy the given predicate from the + /// start of the sequence. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + public mutating func trimPrefix(while predicate: (Element) throws -> Bool) rethrows +} + +extension Collection where Element: Equatable { + /// Returns a new collection of the same type by removing `prefix` from the + /// start. + /// - Parameter prefix: The collection to remove from this collection. + /// - Returns: A collection containing the elements that does not match + /// `prefix` from the start. + public func trimmingPrefix(_ prefix: Prefix) -> SubSequence + where Prefix.Element == Element +} + +extension Collection where SubSequence == Self, Element: Equatable { + /// Removes the initial elements that matches `prefix` from the start. + /// - Parameter prefix: The collection to remove from this collection. + public mutating func trimPrefix(_ prefix: Prefix) + where Prefix.Element == Element +} + +extension RangeReplaceableCollection where Element: Equatable { + /// Removes the initial elements that matches `prefix` from the start. + /// - Parameter prefix: The collection to remove from this collection. + public mutating func trimPrefix(_ prefix: Prefix) + where Prefix.Element == Element +} +``` + +We propose regex-taking variants for string types: + +```swift +extension Collection where SubSequence == Substring { + /// Returns a new subsequence by removing the initial elements that matches + /// the given regex. + /// - Parameter regex: The regex to remove from this collection. + /// - Returns: A new subsequence containing the elements of the collection + /// that does not match `prefix` from the start. + public func trimmingPrefix(_ regex: some RegexComponent) -> SubSequence +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a subsequence of this collection by removing the elements + /// matching the regex from the start, where the regex is created by + /// the given closure. + /// + /// - Parameter content: A closure that returns the regex to search for at + /// the start of this collection. + /// - Returns: A collection containing the elements after those that match + /// the regex returned by `content`. If the regex does not match at + /// the start of the collection, the entire contents of this collection + /// are returned. + public func trimmingPrefix( + @RegexComponentBuilder _ content: () -> some RegexComponent + ) -> SubSequence +} + +extension RangeReplaceableCollection where SubSequence == Substring { + /// Removes the initial elements that matches the given regex. + /// - Parameter regex: The regex to remove from this collection. + public mutating func trimPrefix(_ regex: some RegexComponent) +} + +// In RegexBuilder module +extension RangeReplaceableCollection where SubSequence == Substring { + /// Removes the initial elements matching the regex from the start of + /// this collection, if the initial elements match, using the given closure + /// to create the regex. + /// + /// - Parameter content: A closure that returns the regex to search for + /// at the start of this collection. + public mutating func trimPrefix( + @RegexComponentBuilder _ content: () -> some RegexComponent + ) +} +``` + +#### First range + +We propose a generic collection algorithm for finding the first range of a given subsequence: + +```swift +extension Collection where Element: Equatable { + /// Finds and returns the range of the first occurrence of a given sequence + /// within the collection. + /// - Parameter sequence: The sequence to search for. + /// - Returns: A range in the collection of the first occurrence of `sequence`. + /// Returns nil if `sequence` is not found. + public func firstRange(of other: C) -> Range? + where C.Element == Element +} + +extension BidirectionalCollection where Element: Comparable { + /// Finds and returns the range of the first occurrence of a given sequence + /// within the collection. + /// - Parameter other: The sequence to search for. + /// - Returns: A range in the collection of the first occurrence of `sequence`. + /// Returns `nil` if `sequence` is not found. + public func firstRange(of other: C) -> Range? + where C.Element == Element +} +``` + +We propose a regex-taking variant for string types. + +```swift +extension Collection where SubSequence == Substring { + /// Finds and returns the range of the first occurrence of a given regex + /// within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: A range in the collection of the first occurrence of `regex`. + /// Returns `nil` if `regex` is not found. + public func firstRange(of regex: some RegexComponent) -> Range? +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns the range of the first match for the regex within this collection, + /// where the regex is created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to search for. + /// - Returns: A range in the collection of the first occurrence of the first + /// match of if the regex returned by `content`. Returns `nil` if no match + /// for the regex is found. + public func firstRange( + @RegexComponentBuilder of content: () -> some RegexComponent + ) -> Range? +} +``` + +#### Ranges + +We propose a generic collection algorithm for iterating over all (non-overlapping) ranges of a given subsequence. + +```swift +extension Collection where Element: Equatable { + /// Finds and returns the ranges of the all occurrences of a given sequence + /// within the collection. + /// - Parameter other: The sequence to search for. + /// - Returns: A collection of ranges of all occurrences of `other`. Returns + /// an empty collection if `other` is not found. + public func ranges(of other: C) -> some Collection> + where C.Element == Element +} + +extension BidirectionalCollection where Element: Comparable { + /// Finds and returns the ranges of the all occurrences of a given sequence + /// within the collection. + /// - Parameter other: The sequence to search for. + /// - Returns: A collection of ranges of all occurrences of `other`. Returns + /// an empty collection if `other` is not found. + public func ranges(of other: C) -> some Collection> + where C.Element == Element +} +``` + +And of course regex-taking versions for string types: + +```swift +extension Collection where SubSequence == Substring { + /// Finds and returns the ranges of the all occurrences of a given sequence + /// within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: A collection or ranges in the receiver of all occurrences of + /// `regex`. Returns an empty collection if `regex` is not found. + public func ranges(of regex: some RegexComponent) -> some Collection> +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns the ranges of the all non-overlapping matches for the regex + /// within this collection, where the regex is created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to search for. + /// - Returns: A collection of ranges of all matches for the regex returned by + /// `content`. Returns an empty collection if no match for the regex + /// is found. + public func ranges( + @RegexComponentBuilder of content: () -> some RegexComponent + ) -> some Collection> +} +``` + +#### Match + +We propose algorithms for extracting a `Match` instance from a given regex from the start, anywhere in the middle, or over the entire `self`. + +```swift +extension Collection where SubSequence == Substring { + /// Returns the first match of the specified regex within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: The first match of `regex` in the collection, or `nil` if + /// there isn't a match. + public func firstMatch(of regex: R) -> Regex.Match? + + /// Match a regex in its entirety. + /// - Parameter regex: The regex to match against. + /// - Returns: The match if there is one, or `nil` if none. + public func wholeMatch(of regex: R) -> Regex.Match? + + /// Match part of the regex, starting at the beginning. + /// - Parameter regex: The regex to match against. + /// - Returns: The match if there is one, or `nil` if none. + public func prefixMatch(of regex: R) -> Regex.Match? +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns the first match for the regex within this collection, where + /// the regex is created by the given closure. + /// + /// - Parameter content: A closure that returns the regex to search for. + /// - Returns: The first match for the regex created by `content` in this + /// collection, or `nil` if no match is found. + public func firstMatch( + @RegexComponentBuilder of content: () -> R + ) -> Regex.Match? + + /// Matches a regex in its entirety, where the regex is created by + /// the given closure. + /// + /// - Parameter content: A closure that returns a regex to match against. + /// - Returns: The match if there is one, or `nil` if none. + public func wholeMatch( + @RegexComponentBuilder of content: () -> R + ) -> Regex.Match? + + /// Matches part of the regex, starting at the beginning, where the regex + /// is created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to match against. + /// - Returns: The match if there is one, or `nil` if none. + public func prefixMatch( + @RegexComponentBuilder of content: () -> R + ) -> Regex.Match? +} +``` + +#### Matches + +We propose an algorithm for iterating over all (non-overlapping) matches of a given regex: + +```swift +extension Collection where SubSequence == Substring { + /// Returns a collection containing all matches of the specified regex. + /// - Parameter regex: The regex to search for. + /// - Returns: A collection of matches of `regex`. + public func matches(of regex: R) -> some Collection.Match> +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a collection containing all non-overlapping matches of + /// the regex, created by the given closure. + /// + /// - Parameter content: A closure that returns the regex to search for. + /// - Returns: A collection of matches for the regex returned by `content`. + /// If no matches are found, the returned collection is empty. + public func matches( + @RegexComponentBuilder of content: () -> R + ) -> some Collection.Match> +} +``` + +#### Replace + +We propose generic collection algorithms that will replace all occurrences of a given subsequence: + +```swift +extension RangeReplaceableCollection where Element: Equatable { + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - subrange: The range in the collection in which to search for `other`. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: C, + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max + ) -> Self where C.Element == Element, Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: C, + with replacement: Replacement, + maxReplacements: Int = .max + ) -> Self where C.Element == Element, Replacement.Element == Element + + /// Replaces all occurrences of a target sequence with a given collection + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + public mutating func replace( + _ other: C, + with replacement: Replacement, + maxReplacements: Int = .max + ) where C.Element == Element, Replacement.Element == Element +} +extension RangeReplaceableCollection where Self: BidirectionalCollection, Element: Comparable { + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - subrange: The range in the collection in which to search for `other`. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: C, + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max + ) -> Self where C.Element == Element, Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: C, + with replacement: Replacement, + maxReplacements: Int = .max + ) -> Self where C.Element == Element, Replacement.Element == Element + + /// Replaces all occurrences of a target sequence with a given collection + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + public mutating func replace( + _ other: C, + with replacement: Replacement, + maxReplacements: Int = .max + ) where C.Element == Element, Replacement.Element == Element +} +``` + +We propose regex-taking variants for string types as well as variants that take a closure which will generate the replacement portion from a regex match (e.g. by reading captures). + +```swift +extension RangeReplaceableCollection where SubSequence == Substring { + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - subrange: The range in the collection in which to search for `regex`. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` in `subrange` are replaced by `replacement`. + public func replacing( + _ r: some RegexComponent, + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max + ) -> Self where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ r: some RegexComponent, + with replacement: Replacement, + maxReplacements: Int = .max + ) -> Self where Replacement.Element == Element + + /// Replaces all occurrences of the sequence matching the given regex with + /// a given collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + public mutating func replace( + _ r: some RegexComponent, + with replacement: Replacement, + maxReplacements: Int = .max + ) where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another regex match. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - subrange: The range in the collection in which to search for `regex`. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ regex: R, + subrange: Range, + maxReplacements: Int = .max, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows -> Self where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ regex: R, + maxReplacements: Int = .max, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows -> Self where Replacement.Element == Element + + /// Replaces all occurrences of the sequence matching the given regex with + /// a given collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + public mutating func replace( + _ regex: R, + maxReplacements: Int = .max, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows where Replacement.Element == Element +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a new collection in which all matches for the regex + /// are replaced, using the given closure to create the regex. + /// + /// - Parameters: + /// - replacement: The new elements to add to the collection in place of + /// each match for the regex, using `content` to create the regex. + /// - subrange: The range in the collection in which to search for + /// the regex. + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - Returns: A new collection in which all matches for regex in `subrange` + /// are replaced by `replacement`, using `content` to create the regex. + public func replacing( + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> some RegexComponent + ) -> Self where Replacement.Element == Element + + /// Returns a new collection in which all matches for the regex + /// are replaced, using the given closure to create the regex. + /// + /// - Parameters: + /// - replacement: The new elements to add to the collection in place of + /// each match for the regex, using `content` to create the regex. + /// - maxReplacements: A number specifying how many occurrences of regex + /// to replace. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - Returns: A new collection in which all matches for regex in `subrange` + /// are replaced by `replacement`, using `content` to create the regex. + public func replacing( + with replacement: Replacement, + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> some RegexComponent + ) -> Self where Replacement.Element == Element + + /// Replaces all matches for the regex in this collection, using the given + /// closure to create the regex. + /// + /// - Parameters: + /// - replacement: The new elements to add to the collection in place of + /// each match for the regex, using `content` to create the regex. + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace. + /// - content: A closure that returns the collection to search for + /// and replace. + public mutating func replace( + with replacement: Replacement, + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> some RegexComponent + ) where Replacement.Element == Element + + /// Returns a new collection in which all matches for the regex + /// are replaced, using the given closures to create the replacement + /// and the regex. + /// + /// - Parameters: + /// - subrange: The range in the collection in which to search for the + /// regex, using `content` to create the regex. + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - Returns: A new collection in which all matches for regex in `subrange` + /// are replaced by the result of calling `replacement`, where regex + /// is the result of calling `content`. + public func replacing( + subrange: Range, + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> R, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows -> Self where Replacement.Element == Element + + /// Returns a new collection in which all matches for the regex + /// are replaced, using the given closures to create the replacement + /// and the regex. + /// + /// - Parameters: + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace, using `content` to create the regex. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - Returns: A new collection in which all matches for regex in `subrange` + /// are replaced by the result of calling `replacement`, where regex is + /// the result of calling `content`. + public func replacing( + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> R, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows -> Self where Replacement.Element == Element + + /// Replaces all matches for the regex in this collection, using the + /// given closures to create the replacement and the regex. + /// + /// - Parameters: + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace, using `content` to create the regex. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + public mutating func replace( + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> R, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows where Replacement.Element == Element +} +``` + +#### Split + +We propose a generic collection `split` that can take a subsequence separator: + +```swift +extension Collection where Element: Equatable { + /// Returns the longest possible subsequences of the collection, in order, + /// around elements equal to the given separator collection. + /// + /// - Parameters: + /// - separator: A collection of elements to be split upon. + /// - maxSplits: The maximum number of times to split the collection, + /// or one less than the number of subsequences to return. + /// - omittingEmptySubsequences: If `false`, an empty subsequence is + /// returned in the result for each consecutive pair of separator + /// sequences in the collection and for each instance of separator + /// sequences at the start or end of the collection. If `true`, only + /// nonempty subsequences are returned. + /// - Returns: A collection of subsequences, split from this collection's + /// elements. + public func split( + separator: C, + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true + ) -> some Collection where C.Element == Element +} +extension BidirectionalCollection where Element: Comparable { + /// Returns the longest possible subsequences of the collection, in order, + /// around elements equal to the given separator collection. + /// + /// - Parameters: + /// - separator: A collection of elements to be split upon. + /// - maxSplits: The maximum number of times to split the collection, + /// or one less than the number of subsequences to return. + /// - omittingEmptySubsequences: If `false`, an empty subsequence is + /// returned in the result for each consecutive pair of separator + /// sequences in the collection and for each instance of separator + /// sequences at the start or end of the collection. If `true`, only + /// nonempty subsequences are returned. + /// - Returns: A collection of subsequences, split from this collection's + /// elements. + public func split( + separator: C, + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true + ) -> some Collection where C.Element == Element +} +``` + +And a regex-taking variant for string types: + +```swift +extension Collection where SubSequence == Substring { + /// Returns the longest possible subsequences of the collection, in order, + /// around subsequence that match the given separator regex. + /// + /// - Parameters: + /// - separator: A regex to be split upon. + /// - maxSplits: The maximum number of times to split the collection, + /// or one less than the number of subsequences to return. + /// - omittingEmptySubsequences: If `false`, an empty subsequence is + /// returned in the result for each consecutive pair of matches + /// and for each match at the start or end of the collection. If + /// `true`, only nonempty subsequences are returned. + /// - Returns: A collection of substrings, split from this collection's + /// elements. + public func split( + separator: some RegexComponent, + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true + ) -> some Collection +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns the longest possible subsequences of the collection, in order, + /// around subsequence that match the regex created by the given closure. + /// + /// - Parameters: + /// - maxSplits: The maximum number of times to split the collection, + /// or one less than the number of subsequences to return. + /// - omittingEmptySubsequences: If `false`, an empty subsequence is + /// returned in the result for each consecutive pair of matches + /// and for each match at the start or end of the collection. If + /// `true`, only nonempty subsequences are returned. + /// - separator: A closure that returns a regex to be split upon. + /// - Returns: A collection of substrings, split from this collection's + /// elements. + public func split( + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true, + @RegexComponentBuilder separator: () -> some RegexComponent + ) -> some Collection +} +``` + +**Note:** We plan to adopt the new generics features enabled by [SE-0346][] for these proposed methods when the standard library adopts primary associated types, [pending a forthcoming proposal][stdlib-pitch]. For example, the first method in the _Replacement_ section above would instead be: + +```swift +extension RangeReplaceableCollection where Element: Equatable { + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + public func replacing( + _ other: some Collection, + with replacement: some Collection, + subrange: Range, + maxReplacements: Int = .max + ) -> Self +} +``` + +#### Searching for empty strings and matches + +Empty matches and inputs are an important edge case for several of the algorithms proposed above. For example, what is the result of `"123.firstRange(of: /[a-z]*/)`? How do you split a collection separated by an empty collection, as in `"1234".split(separator: "")`? For the Swift standard library, this is a new consideration, as current algorithms are `Element`-based and cannot be passed an empty input. + +Languages and libraries are nearly unanimous about finding the location of an empty string, with Ruby, Python, C#, Java, Javascript, etc, finding an empty string at each index in the target. Notably, Foundation's `NSString.range(of:)` does _not_ find an empty string at all. + +The methods proposed here follow the consensus behavior, which makes sense if you think of `a.firstRange(of: b)` as returning the first subrange `r` where `a[r] == b`. If a regex can match an empty substring, like `/[a-z]*/`, the behavior is the same. + +```swift +let hello = "Hello" +let emptyRange = hello.firstRange(of: "") +// emptyRange is equivalent to '0..<0' (integer ranges shown for readability) +``` + +Because searching again at the same index would yield that same empty string, we advance one position after finding an empty string or matching an empty pattern when finding all ranges. This yields the position of every valid index in the string. + +```swift +let allRanges = hello.ranges(of: "") +// allRanges is equivalent to '[0..<0, 1..<1, 2..<2, 3..<3, 4..<4, 5..<5]' +``` + +Splitting with an empty separator (or a pattern that matches empty string), uses this same behavior, resulting in a collection of single-element substrings. Interestingly, a couple languages make different choices here. C# returns the original string instead of its parts, and Python rejects an empty separator (though it permits regexes that match empty strings). + +```swift +let parts = hello.split(separator: "") +// parts == ["h", "e", "l", "l", "o"] + +let moreParts = hello.split(separator: "", omittingEmptySubsequences: false) +// parts == ["", "h", "e", "l", "l", "o", ""] +``` + +Finally, searching for an empty string within an empty string yields, as you might imagine, the empty string: + +```swift +let empty = "" +let range = empty.firstRange(of: empty) +// empty == empty[range] +``` + +[SE-0346]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md +[stdlib-pitch]: https://forums.swift.org/t/pitch-primary-associated-types-in-the-standard-library/56426 + +## Alternatives considered + +### Extend `Sequence` instead of `Collection` + +Most of the proposed algorithms are necessarily on `Collection` due to the use of indices or mutation. `Sequence` does not support multi-pass iteration, so even `trimmingPrefix` would problematic on `Sequence` because it needs to look one `Element` ahead to know when to stop trimming and would need to return a wrapper for the in-progress iterator instead of a subsequence. + +### Cross-proposal API naming consistency + +The regex work is broken down into 6 proposals based on technical domain, which is advantageous for deeper technical discussions and makes reviewing the large body of work manageable. The disadvantage of this approach is that relatively-shallow cross-cutting concerns, such as API naming consistency, are harder to evaluate until we've built up intuition from multiple proposals. + +We've seen the [Regex type and overview](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md), the [Regex builder DSL](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0351-regex-builder.md), and here we present lots of ways to use regex. Now's a good time to go over API naming consistency. + +(The other proposal with a significant amount of API is [Unicode for String Processing](https://forums.swift.org/t/pitch-unicode-for-string-processing/56907), which is in the pitch phase. It is a technical niche and less impactful on these naming discussions. We'll still want to design those names for consistency, of course.) + + +```swift +protocol RegexComponent { + associatedtype RegexOutput +} +``` + +The associatedtype name is "RegexOutput" to help libraries conform their parsers to this protocol (e.g. via `CustomConsumingRegexComponent`). Regex's capture representation is regexy: it has the overall matched portion as the first capture and the regex builders know how to combine these kinds of capture lists together. This could be different than how e.g. a parser combinator library's output types might be represented. Thus, we chose a more specific name to avoid any potential conflicts. + +The name "RegexComponent" accentuates that any conformer can be used as part of a larger regex, while it de-emphasizes that `Regex` instances themselves can be used directly. We propose methods that are generic over `RegexComponent` and developers will be considering whether they should make their functions that otherwise take a `Regex` also be generic over `RegexComponent`. + +It's possible there might be some initial confusion around the word "component", i.e. a developer may have a regex and not be sure how to make it into a component or how to get the component out. The word "component" carries a lot of value in the context of the regex DSL. An alternative name might be `RegexProtocol`, which implies that a Regex can be used at the site and would be clearly the way to make a function taking a concrete `Regex` generic. But, it's otherwise a naming workaround that doesn't carry the additional regex builder connotations. + +The protocol requirement is `var regex: Regex`, i.e. any type that can produce a regex or hook into the engine's customization hooks (this is what `consuming` does) can be used as a component of the DSL and with these generic API. An alternative name could be "CustomRegexConvertible", but we don't feel that communicates component composability very well, nor is it particularly enlightening when encountering these generic API. + +Another alternative is to have a second protocol just for generic API. But without a compelling semantic distinction or practical utility, we'd prefer to avoid adding protocols just for names. If a clearly superior name exists, we should just choose that. + + +```swift +protocol CustomConsumingRegexComponent { + func consuming(...) +} +``` + +This is not a normal developer-facing protocol or concept; it's an advanced library-extensibility feature. Explicit, descriptive, and careful names are more important than concise names. The "custom" implies that we're not just vending a regex directly ourselves, we're instead customizing behavior by hooking into the run-time engine directly. + +Older versions of the pitch had `func match(...) -> (String.Index, T)?` as the protocol requirement. As [Regex type and overview](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md) went through review, naming convention settled on using the word "match" as a noun and in context with operations that produce a `Match` instance. Since this is the engine's customization hook, it produces the value and position to resume execution from directly, and hence different terminology is apt and avoids confusion or future ambiguities. "Consuming" is the nomenclature we're going with for something that chews off the front of its input in order to produces a value. + +This protocol customizes the basic consume-from-the-front functionality. A protocol for customizing search is future work and involves accommodating different kinds of state and ways that a searcher may wish to speed up subsequent searches. Alternative names for the protocol include `CustomRegexComponent`, `CustomConsumingRegex`, etc., but we don't feel brevity is the key consideration here. + + +### Why `where SubSequence == Substring`? + +A `Substring` slice requirement allows the regex engine to produce indices in the original collection by operating over a portion of the input. Unfortunately, this is not one of the requirements of `StringProtocol`. + +A new protocol for types that can produce a `Substring` on request (e.g. from UTF-8 contents) would have to eagerly produce a `String` copy first and would need requirements to translate indices. When higher-level algorithms are implemented via multiple calls to the lower-level algorithms, these copies could happen many times. Shared strings are future work but a much better solution to this. + +## Future directions + +### Backward algorithms + +It would be useful to have algorithms that operate from the back of a collection, including ability to find the last non-overlapping range of a pattern in a string, and/or that to find the first range of a pattern when searching from the back, and trimming a string from both sides. They are deferred from this proposal as the API that could clarify the nuances of backward algorithms are still being explored. + +
+ Nuances of backward algorithms + +There is a subtle difference between finding the last non-overlapping range of a pattern in a string, and finding the first range of this pattern when searching from the back. + +The currently proposed algorithm that finds a pattern from the front, e.g. `"aaaaa".ranges(of: "aa")`, produces two non-overlapping ranges, splitting the string in the chunks `aa|aa|a`. It would not be completely unreasonable to expect to introduce a counterpart, such as `"aaaaa".lastRange(of: "aa")`, to return the range that contains the third and fourth characters of the string. This would be a shorthand for `"aaaaa".ranges(of: "aa").last`. Yet, it would also be reasonable to expect the function to return the first range of `"aa"` when searching from the back of the string, i.e. the range that contains the fourth and fifth characters. + +Trimming a string from both sides shares a similar story. For example, `"ababa".trimming("aba")` can return either `"ba"` or `"ab"`, depending on whether the prefix or the suffix was trimmed first. +
+ +### Split preserving the separator + +Future work is a split variant that interweaves the separator with the separated portions. For example, when splitting over `\p{punctuation}` it might be useful to be able to preserve the punctionation as a separate entry in the returned collection. + +### Future API + +Some common string processing functions are not currently included in this proposal, such as trimming the suffix from a string/collection, and finding overlapping ranges of matched substrings. This pitch aims to establish a pattern for using `RegexComponent` with string processing algorithms, so that further enhancement can to be introduced to the standard library easily in the future, and eventually close the gap between Swift and other popular scripting languages. diff --git a/proposals/0358-primary-associated-types-in-stdlib.md b/proposals/0358-primary-associated-types-in-stdlib.md new file mode 100644 index 0000000000..a95f941c09 --- /dev/null +++ b/proposals/0358-primary-associated-types-in-stdlib.md @@ -0,0 +1,218 @@ +# Primary Associated Types in the Standard Library + +* Proposal: [SE-0358](0358-primary-associated-types-in-stdlib.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#41843](https://github.com/apple/swift/pull/41843) +* Review: ([pitch](https://forums.swift.org/t/pitch-primary-associated-types-in-the-standard-library/56426/)) ([review](https://forums.swift.org/t/se-0358-primary-associated-types-in-the-standard-library/57432)) ([partial acceptance](https://forums.swift.org/t/se-0358-primary-associated-types-in-the-standard-library/57432/14)) ([revision and extension](https://forums.swift.org/t/se-0358-primary-associated-types-in-the-standard-library/57432/32)) ([acceptance](https://forums.swift.org/t/accepted-se-0358-primary-associated-types-in-the-standard-library/58547)) +* Related Proposals: + - [SE-0023] API Design Guidelines + - [SE-0346] Lightweight same-type requirements for primary associated types + +[SE-0023]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0023-api-guidelines.md +[SE-0346]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md + +## Introduction + +[SE-0346] introduced the concept of primary associated types to the language. This document proposes to adopt this feature in the Swift Standard Library, adding primary associated types to select existing protocols. Additionally, we provide some general API design recommendations that protocol authors may find helpful when adding support for this language feature. + +## Motivation + +In order for the lightweight constraint syntax introduced in [SE-0346] to be actually usable, protocol definitions inside and outside the Standard Library need to be extended with primary associated type declarations. + +See [SE-0346] for several motivating examples for these changes. + +## API Design Guidelines + +Primary associated types add a new facet to the design of protocols. For every public protocol with associated type requirements, we need to carefully consider which of them (if any) we want to mark as primary. On the one hand, we want to allow people to use the shorthand syntax whenever possible; on the other hand, we only get one chance to decide this: once a protocol gains a primary associated type annotation, most subsequent changes would be source-breaking. + +We've found the following guidelines helpful when considering the adoption of primary associated types within the Standard Library. We haven't had enough real-life experience with this new feature to propose these guidelines for general use -- however, the recommendations below can still serve as a useful starting point. + +(Aside: If you decide to follow these guidelines when annotating your own protocols, and they lead you to a choice that you later regret, please post a note on the Swift forums! Negative examples are going to be extremely helpful while revising the guidelines for general use. We're also looking for (positive or negative) examples for multiple primary associated types on a single protocol.) + +1. **Let usage inform your design.** + + If you are considering adding a primary associated type declaration to a preexisting protocol, then look at its existing clients to discover which associated types get typically constrained. Is there one particular type that is used overwhelmingly more than any other? If so, then it will probably be a good choice for the primary. + + For example, in the case of `Sequence`, use sites overwhelmingly tend to constrain `Element` -- `Iterator` is almost never mentioned in `where` clauses. This makes it fairly clear that `Element` is the right choice for the primary type. + + If you're designing a new protocol, think about which type people will most likely want to constrain. Sometimes it may not even be one you planned to have as an associated type! + + For example, protocol `Clock` in [SE-0329](0329-clock-instant-duration.md) initially only had `Instant` as an associated type. As it turns out, in actual use cases, people are far more likely to want to constrain `Instant.Duration` rather than `Instant` itself. Clocks tend to be far too closely coupled to their instants for it to serve as a useful constraint target -- `some Clock` is effectively just a circuitous way of spelling `ContinuousClock`. On the other hand, `some Clock` captures all clocks that measure elapsed time in physical seconds -- a far more useful abstraction. Therefore, we decided to add `Clock.Duration` for the express purpose to serve as the primary associated type. + +2. **Consider clarity at the point of use.** To prevent persistent confusion, _people familiar with the protocol_ ought to be able to correctly intuit the meaning of a same-type constraint such as `some Sequence`. + + Lightweight constraint specifications share the same angle-bracketed syntax as generic type arguments, including the same limitations. In particular, the language does not support argument labels in such lists, which prevents us from clarifying the role of the type names provided. A type name such as `Foo` on its own provides no hints about the role of its generic arguments `Int` and `String`; likewise, it isn't possible to decipher the role of `Character` in a same-type requirement such as `some Bar`, unless the reader is already somewhat familiar with the protocol `Bar`. + + The best candidates for primary associated types tend to be those that have a simple, obvious relationship to the protocol itself. A good heuristic is that if the relationship can be described using a simple preposition, then the associated type will probably make a viable primary: + + - `Collection` *of* `Int` + - `Identifiable` *by* `String` + - `SIMD` *of* `Float` + - `RawRepresentable` *by* `Int32` + + Associated types that don't support this tend to have a more complex / idiosyncratic role in their protocol, and often make poor choices for a primary associated type. + + For example, `Numeric` has an associated type called `Magnitude` that does sometimes appear in associated type constraints. However, its role seems too subtle and non-obvious to consider marking it as primary. The meaning of `Int` in `some Numeric` is unlikely to be clear to readers, even if they are deeply familiar with Swift's numeric protocol hierarchy. + +3. **Not every protocol needs primary associated types.** Don't feel obligated to add a primary associated type just because it is possible to do so. If you don't expect people will want to constrain an associated type in practice, there is little reason to mark it as a primary. Similarly, if there are multiple possible choices that seem equally useful, it might be best not to select one. (See point 2 above.) + + For example, `ExpressibleByIntegerLiteral` is not expected to be mentioned in generic function declarations, so there is no reason to mark its sole associated type (`IntegerLiteral`) as the primary. + +4. **Limit yourself to just one primary associated type.** In most cases, it's best not to declare more than one primary associated type on any protocol. + + While the language does allow this, [SE-0346] requires clients using the lightweight syntax to always explicitly constrain all primary associated types, which may become an obstacle. Clients don't have an easy way to indicate that they want to leave one of the types unconstrained -- to do that, they need to revert to classic generic syntax, partially or entirely giving up on the lightweight variant: + + ```swift + protocol MyDictionaryProtocol { + associatedtype Key: Equatable + associatedtype Value + ... + } + + // This function is happy to work on any dictionary-like thing + // as long as it has string keys. + func twiddle(_ items: some MyDictionaryProtocol) -> Int { ... } + + // Possible approaches: + func twiddle(_ items: some MyDictionaryProtocol) -> Int { ... } + func twiddle(_ items: T) -> Int where T.Key == String { ... } + ``` + + Of course, if the majority of clients actually do want to constrain both `Key` and `Value`, then having them both marked primary can be an appropriate choice. + + +## Proposed solution + +The table below lists all public protocols in the Standard Library with associated type requirements, along with their proposed primary associated type, as well as a list of other associated types. + +[note]: #alternatives-considered + +| Protocol | Primary | Others | +|------------------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Sequence` | `Element` | `Iterator` | +| `IteratorProtocol` | `Element` | -- | +| `Collection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `MutableCollection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `BidirectionalCollection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `RandomAccessCollection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `RangeReplaceableCollection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `LazySequenceProtocol` | -- [(1)][note] | `Element`, `Iterator`, `Elements` | +| `LazyCollectionProtocol` | -- [(1)][note] | `Element`, `Index`, `Iterator`, `SubSequence`, `Indices`, `Elements` | +| `Identifiable` | `ID` | -- | +| `RawRepresentable` | `RawValue` | -- | +| `RangeExpression` | `Bound` | -- | +| `Strideable` | `Stride` | -- | +| `SetAlgebra` | `Element` | `ArrayLiteralElement` | +| `OptionSet` | -- [(2)][note] | `Element`, `ArrayLiteralElement`, `RawValue` | +| `Numeric` | -- | `IntegerLiteralType`, `Magnitude` | +| `SignedNumeric` | -- | `IntegerLiteralType`, `Magnitude` | +| `BinaryInteger` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Words` | +| `UnsignedInteger` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Words` | +| `SignedInteger` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Words` | +| `FixedWidthInteger` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Words` | +| `FloatingPoint` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Exponent` | +| `BinaryFloatingPoint` | -- | `IntegerLiteralType`, `FloatLiteralType`, `Magnitude`, `Stride`, `Exponent`, `RawSignificand`, `RawExponent` | +| `SIMD` | `Scalar` | `ArrayLiteralElement`, `MaskStorage` | +| `SIMDStorage` | -- | `Scalar` | +| `SIMDScalar` | -- | `SIMDMaskScalar`, `SIMD2Storage`, `SIMD4Storage`, ..., `SIMD64Storage` | +| `KeyedEncodingContainerProtocol` | -- | `Key` | +| `KeyedDecodingContainerProtocol` | -- | `Key` | +| `ExpressibleByIntegerLiteral` | -- | `IntegerLiteralType` | +| `ExpressibleByFloatLiteral` | -- | `FloatLiteralType` | +| `ExpressibleByBooleanLiteral` | -- | `BooleanLiteralType` | +| `ExpressibleByUnicodeScalarLiteral` | -- | `UnicodeScalarLiteralType` | +| `ExpressibleByExtended-`
`GraphemeClusterLiteral` | -- | `UnicodeScalarLiteralType`, `ExtendedGraphemeClusterLiteralType` | +| `ExpressibleByStringLiteral` | -- | `UnicodeScalarLiteralType`, `ExtendedGraphemeClusterLiteralType`, `StringLiteralType` | +| `ExpressibleByStringInterpolation` | -- | `UnicodeScalarLiteralType`, `ExtendedGraphemeClusterLiteralType`, `StringLiteralType`, `StringInterPolation` | +| `ExpressibleByArrayLiteral` | -- | `ArrayLiteralElement` | +| `ExpressibleByDictionaryLiteral` | -- | `Key`, `Value` | +| `StringInterpolationProtocol` | -- | `StringLiteralType` | +| `Unicode.Encoding` | -- | `CodeUnit`, `EncodedScalar`, `ForwardParser`, `ReverseParser` | +| `UnicodeCodec` | -- | `CodeUnit`, `EncodedScalar`, `ForwardParser`, `ReverseParser` | +| `Unicode.Parser` | -- | `Encoding` | +| `StringProtocol` | -- | `Element`, `Index`, `Iterator`, `SubSequence`, `Indices`, `UnicodeScalarLiteralType`, `ExtendedGraphemeClusterLiteralType`, `StringLiteralType`, `StringInterPolation`, `UTF8View`, `UTF16View`, `UnicodeScalarView` | +| `CaseIterable` | -- | `AllCases` | +| `Clock` | `Duration` | `Instant` | +| `InstantProtocol` | `Duration` | -- | +| `AsyncIteratorProtocol` | -- [(3)][note] | `Element` | +| `AsyncSequence` | -- [(3)][note] | `AsyncIterator`, `Element` | +| `GlobalActor` | -- | `ActorType` | +| `DistributedActor` | -- [(4)][note] | `ID`, `ActorSystem`, `SerializationRequirement` | +| `DistributedActorSystem` | -- [(4)][note] | `ActorID`, `SerializationRequirement`, `InvocationEncoder`, `InvocationDecoder`, `ResultHandler` | +| `DistributedTargetInvocationEncoder` | -- [(4)][note] | `SerializationRequirement` | +| `DistributedTargetInvocationDecoder` | -- [(4)][note] | `SerializationRequirement` | +| `DistributedTargetInvocationResultHandler` | -- [(4)][note] | `SerializationRequirement` | + +As of Swift 5.6, the following public protocols don't have associated type requirements, so they are outside of the scope of this proposal. + +```swift +Equatable, Hashable, Comparable, Error, AdditiveArithmetic, +DurationProtocol, Encodable, Decodable, Encoder, Decoder, +UnkeyedEncodingContainer, UnkeyedDecodingContainer, +SingleValueEncodingContainer, SingleValueDecodingContainer, +ExpressibleByNilLiteral, CodingKeyRepresentable, +CustomStringConvertible, LosslessStringConvertible, TextOutputStream, +TextOutputStreamable, CustomPlaygroundDisplayConvertible, +CustomReflectable, CustomLeafReflectable, MirrorPath, +RandomNumberGenerator, CVarArg, Sendable, UnsafeSendable, Actor, +AnyActor, Executor, SerialExecutor, DistributedActorSystemError +``` + +## Detailed design + +```swift +public protocol Sequence +public protocol IteratorProtocol +public protocol Collection: Sequence +public protocol MutableCollection: Collection +public protocol BidirectionalCollection: Collection +public protocol RandomAccessCollection: BidirectionalCollection +public protocol RangeReplaceableCollection: Collection + +public protocol Identifiable +public protocol RawRepresentable +public protocol RangeExpression +public protocol Strideable: Comparable + +public prococol SetAlgebra: Equatable, ExpressibleByArrayLiteral + +public protocol SIMD: ... + +public protocol Clock: Sendable +public protocol InstantProtocol: Comparable, Hashable, Sendable +``` + +## Source compatibility + +None. The new annotations enable new ways to use these protocols, but they are tied to new syntax, and they do not affect existing code. + +## Effect on ABI stability + +None. The annotations aren't ABI impacting, and the new capabilities deploy back to any previous Swift Standard Library release. + +## Effect on API resilience + +Once introduced, primary associated types cannot be removed from a protocol or reordered without breaking source compatibility. + +[SE-0346] requires usage sites to always list every primary associated type defined by a protocol. Until/unless this restriction is lifted, adding a new primary associated type to a protocol that already has some will also be a source breaking change. + +Therefore, we will not be able to make any changes to the list of primary associated types of any of the protocols that are affected by this proposal once this ships in a Standard Library release. + +## Alternatives considered + +(1) It is tempting to declare `Element` as the primary associated type for `LazySequenceProtocol` and `LazyCollectionProtocol`, for consistency with other protocols in the collection hierarchy. However, in actual use, `Elements` seems just as useful (if not more) to be easily constrained. We left the matter of selecting one of these as primary unresolved for now; as we get more experience with the lightweight constraint syntax, we may revisit these protocols. + +(2) In the `OptionSet` protocol, the `Element` type is designed to always be `Self`, so `RawValue` would be the most practical choice for the primary associated type. However, to avoid potential confusion, we left `OptionSet` without a primary associated type annotation. + +(3) `AsyncSequence` and `AsyncIteratorProtocol` logically ought to have `Element` as their primary associated type. However, we have [ongoing evolution discussions][rethrows] about adding a precise error type to these. If those discussions bear fruit, then it's possible we may want to _also_ mark the potential new `Error` associated type as primary. To prevent source compatibility complications, adding primary associated types to these two protocols is deferred to a future proposal. + +[rethrows]: https://forums.swift.org/t/se-0346-lightweight-same-type-requirements-for-primary-associated-types/55869/70 + +(4) Declaring primary associated types on the distributed actor protocols would be desirable, but it was [deferred to a future proposal](https://forums.swift.org/t/pitch-primary-associated-types-in-the-standard-library/56426/47), to prevent interfering with potential future language improvements that would make them more useful in this use case. + +## Revisions + +- [2022-05-28](https://github.com/swiftlang/swift-evolution/blob/716db41ccefde348ac38bd2fd1eb5bd7842be7b6/proposals/0358-primary-associated-types-in-stdlib.md): Initial proposal version. +- 2022-06-22: Removed the primary associated type declaration from the `OptionSet` protocol. The API guidelines section has revised wording; it no longer proposes the new guidelines for inclusion in the official Swift API Guidelines document. Adjusted wording to prefer the term "lightweight constraint syntax" to "lightweight same-type requirements", as the new syntax can be used for more than just to express same-type constraints. diff --git a/proposals/0359-build-time-constant-values.md b/proposals/0359-build-time-constant-values.md new file mode 100644 index 0000000000..5778ef2da4 --- /dev/null +++ b/proposals/0359-build-time-constant-values.md @@ -0,0 +1,328 @@ +# Build-Time Constant Values + +* Proposal: [SE-0359](0359-build-time-constant-values.md) +* Authors: [Artem Chikin](https://github.com/artemcm), [Ben Cohen](https://github.com/airspeedswift), [Xi Ge](https://github.com/nkcsgexi) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Returned for revision** +* Decision notes: [First review rationale](https://forums.swift.org/t/returned-for-revision-se-0359-build-time-constant-values/58976) +* Implementation: Implemented on `main` as `_const` + +## Introduction + +A Swift language feature for requiring certain values to be knowable at compile-time. This is achieved through an attribute, `@const`, constraining properties and function parameters to have compile-time knowable values. Such information forms a foundation for richer compile-time features in the future, such as extraction and validation of values at compile time. + +Related forum threads: + +* [[Pitch] Compile-Time Constant Values](https://forums.swift.org/t/pitch-compile-time-constant-values/53606) +* [[Pitch #2] Build-Time Constant Values](https://forums.swift.org/t/pitch-2-build-time-constant-values/56762) + +## Motivation + +Compile-time constant values are values that can be known or computed during compilation and are guaranteed to not change after compilation. Use of such values can have many purposes, from enforcing desirable invariants and safety guarantees to enabling users to express arbitrarily-complex compile-time algorithms. + +The first step towards building out support for compile-time constructs in Swift is a basic primitives consisting of an attribute to declare function parameters and properties to require being known at *compile-time*. While this requirement explicitly calls out the compiler as having additional guaranteed information about such declarations, it can be naturally extended into a broader sense of *build-time* known values - with the compiler being able to inform other tools with type-contextual value information. For an example of the latter, see the “Declarative Package Manifest” motivating use-case below. + +## Proposed Solution + +The Swift compiler will recognize declarations of properties, local variables, and function parameters declared with a `@const` attribute as having an additional requirement to be known at compile-time. If a `@const` property or variable is initialized with a runtime value, the compiler will emit an error. Similarly, if a runtime value is passed as an argument to a `@const` function parameter, the compiler will emit an error. Aside from participating in name mangling, the attribute has no runtime effect. + +For example, a `@const` property can provide the compiler and relevant tooling build-time knowledge of a type-specific value: + +```swift +struct DatabaseParams { + @const let encoding: String = "utf-8" + @const let capacity: Int = 256 +} +``` + +A `@const` parameter provides a guarantee of a build-time-known argument being specified at all function call-sites, allowing future APIs to enforce invariants and provide build-time correctness guarantees: + +```swift +func acceptingURLString(@const _ url: String) +``` + +And a `@const static let` protocol property requirement allows protocol authors to enforce get the benefits of build-time known properties for all conforming types: + +```swift +protocol DatabaseSerializableWithKey { + @const static let key: String +} +``` + +## Detailed Design + +### Property `@const` attribute + +A stored property on a `struct` or a `class` can be marked with a `@const` attribute to indicate that its value is known at compile-time. +```swift +struct Foo { + @const let title: String = "foo" +} +``` +The value of such a property must be default-initialized with a compile-time-known value, unlike a plain `let` property, which can also be assigned a value in the type's initializer. + +```swift +struct Foo { + // 👍 + @const let superTitle: String = "Encyclopedia" + // ❌ error: `title` must be initialized with a const value + @const let title: String + // ❌ error: `subTitle` must be initialized with a const value + @const let subTitle: String = bar() +} +``` + +Similarly to Implicitly-Unwrapped Optionals, the mental model for semantics of this attribute is that it is a flag on the declaration that guarantees that the compiler is able to know its value as shared by all instance of the type. For now, `@const let` and `@const static let` are equivalent in what information the `@const` attribute conveys to the compiler. + +### Parameter `@const` attribute + +A function parameter can be marked with a `@const` keyword to indicate that values passed to this parameter at the call-site must be compile-time-known values. + +```swift +func foo(@const input: Int) {...} +``` + +Passing in a runtime value as an argument to `foo` will result in a compilation error: +```swift +foo(11) // 👍 + +let x: Int = computeRuntimeCount() +foo(x) // ❌ error: 'greeting' must be initialized with a const value +``` + +### Protocol `@const` property requirement + +A protocol author may require conforming types to default initialize a given property with a compile-time-known value by specifying it as `@const static let` in the protocol definition. For example: + +```swift +protocol NeedsConstGreeting { + @const static let greeting: String +} +``` +Unlike other property declarations on protocols that require the use of `var` and explicit specification of whether the property must provide a getter `{ get }` or also a setter `{ get set }`, using `var` for build-time-known properties whose values are known to be fixed at runtime is counter-intuitive. Moreover, `@const` implies the lack of a run-time setter and an implicit presence of the value getter. + +If a conforming type initializes `greeting` with something other than a compile-time-known value, a compilation error is produced: + +```swift +struct Foo: NeedsConstGreeting { + // 👍 + static let greeting = "Hello, Foo" +} +struct Bar: NeedsConstGreeting { + // error: ❌ 'greeting' must be initialized with a const value + static let greeting = "\(Bool.random ? "Hello" : "Goodbye"), Bar" +} +``` + +### Supported Types + +The requirement that values of `@const` properties and parameters be known at compile-time restricts the allowable types for such declarations. The current scope of the proposal includes: + +* Enum cases with no associated values +* Certain standard library types that are expressible with literal values +* Integer and Floating-Point types (`(U)?Int(\d*)`, `Float`, `Double`, `Half`), `String` (excluding interpolated strings), `Bool`. +* `Array` and `Dictionary` literals consisting of literal values of above types. +* Tuple literals consisting of the above list items. + +This list will expand in the future to include more literal-value kinds or potential new compile-time valued constructs. + +## Motivating Example Use-Cases + +### Enforcement of Compile-Time Attribute Parameters + +Attribute definitions can benefit from additional guarantees of compile-time constant values. +For example, a `@const` version of the [@Clamping](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md#clamping-a-value-within-bounds) property wrapper that requires that lower and upper bounds be compile-time-known values can ensure that the clamp values are fixed and cannot change for different instantiations of wrapped properties which may occur if runtime values are used to specify the bounds: + +```swift +@propertyWrapper +struct Clamping { + var value: V + let min: V + let max: V + + init(wrappedValue: V, @const min: V, @const max: V) { + value = wrappedValue + self.min = min + self.max = max + assert(value >= min && value <= max) + } + ... +``` +It could also allow the compiler to generate more-efficient comparison and clamping code, in the future. + +Or imagine a property wrapper that declares a property is to be serialized and that it must be stored/retrieved using a specific string key. `Codable` requires users to provide a `CodingKeys` `enum` boilerplate, relying on the `enum`’s `String` raw values. Alternatively, such key can be specified on the property wrapper itself: + +```swift +struct Foo { + @SpecialSerializationSauce(key: "title") + var someSpecialTitleProperty: String +} + +@propertyWrapper +struct SpecialSerializationSauce { + init(@const key: String) {...} +} +``` + +Having the compiler enforce the compile-time constant property of the `key` parameter eliminates the possibility of an error where a run-time value is specified which can cause serialized data to not be able to be deserialized, for example. + +Enforcing compile-time constant nature of the parameters is also the first step to allowing attribute/library authors to be able to check uses by performing compile-time sanity checking and having the capability to emit custom build-time error messages. + +### Enforcement of Non-Failable Initializers + +Ergonomics of the recently-pitched [Foundation.URL](https://forums.swift.org/t/foundation-url-improvements/54057) would benefit greatly from the ability to require the string argument to be compile-time constant. With evolving compile-time evaluation facilities, Swift may even gain an ability to perform compile-time validation of such URLs even though the user may never be able to express a fully compile-time constant `Foundation.URL` type because this type is a part of an ABI-stable SDK. While a type like `StaticString` may be used to require that the argument string must be static, which string is chosen can still be determined at runtime, e.g.: + +```swift +URL(Bool.random() ? "https://valid.url.com" : "invalid url . com") +``` + +### Facilitate Compile-time Extraction of Values + +The [Result Builder-based SwiftPM Manifest](https://forums.swift.org/t/pre-pitch-swiftpm-manifest-based-on-result-builders/53457) pre-pitch outlines a proposal for a manifest format that encodes package model/structure using Swift’s type system via Result Builders. Extending the idea to use the builder pattern throughout can result in a declarative specification that exposes the entire package structure to build-time tools, for example: + +```swift +let package = Package { + Modules { + Executable("MyExecutable", public: true, include: { + Internal("MyDataModel") + }) + Library("MyLibrary", public: true, include: { + Internal("MyDataModel", public: true) + }) + Library("MyDataModel") + Library("MyTestUtilities") + Test("MyExecutableTests", for: "MyExecutable", include: { + Internal("MyTestUtilities") + External("SomeModule", from: "some-package") + }) + Test("MyLibraryTests", for: "MyLibrary") + } + Dependencies { + SourceControl(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0") + } +} +``` + +A key property of this specification is that all the information required to know how to build this package is encoded using compile-time-known concepts: types and literal (and therefore compile-time-known) values. This means that for a category of simple packages where such expression of the package’s model is possible, the manifest does not need to be executed in a sandbox by the Package Manager - the required information can be extracted at manifest *build* time. + +To *ensure* build-time extractability of the relevant manifest structure, a form of the above API can be provided that guarantees the compile-time known properties. For example, the following snippet can guarantee the ability to extract complete required knowledge at build time: + +```swift +Test("MyExecutableTests", for: "MyExecutable", include: { + Internal("MyTestUtilities") + External("SomeModule", from: "some-package") + }) +``` +By providing a specialized version of the relevant types (`Test`, `Internal`, `External`) that rely on parameters relevant to extracting the package structure being `const`: + +```swift +struct Test { + init(@const _ title: String, @const for: String, @DependencyBuilder include: ...) {...} +} +struct Internal { + init(@const _ title: String) +} +struct External { + init(@const _ title: String, @const from: String) +} +``` +This could, in theory, allow SwiftPM to build such packages without executing their manifest. Some packages, of course, could still require run-time (execution at package build-time) Swift constructs. More-generally, providing the possibility of declarative APIs that can express build-time-knowable abstractions can both eliminate (in some cases) the need for code execution - reducing the security surface area - and allow for further novel use-cases of Swift’s DSL capabilities (e.g. build-time extractable database schema, etc.). + +### Guaranteed Optimization Hints + +Similarly, ergonomics of numeric intrinsics can benefit from allowing only certain function parameters to be required to be compile-time known. For example, requiring a given numeric operation to specify a `@const` parameter for the rounding mode of an operation as an enum case, while allowing the operands of the operation be runtime values, allowing the compiler to generate more-efficient code. + +## Source compatibility + +This is a purely additive change and has no source compatibility impacts. + +## Effect on ABI stability and API resilience + +The new function parameter attribute is a part of name mangling. The *value* of `public @const` properties is a part of a module's ABI. See discussion on [*Memory placement*](#memory-placement-and-runtime-initialization) for details. + +## Effect on SwiftPM packages + +There is no impact on SwiftPM packages. + +## Alternatives Considered + +### Using a keyword or an introducer instead of an attribute +`@const` being an attribute, as opposed to a keyword or a new introducer (such as `const` instead of `let`), is an approach that is more amenable to applying to a greater variety of constructs in the futures, in addition to property and parameter declarations, such as `@const func`. In addition, as described in comparison to Implicitly-Unwrapped Optionals above, this attribute does not fundamentally change the behavior of the declaration, rather it restricts its handling by the compiler, similar to `@objc`. + +### Difference to `StaticString`-like types +As described in the **Enforcement of Non-Failable Initializers**, the key difference to types like `StaticString` that require a literal value is the `@const` attribute's requirement that the exact value be known at compile-time. `StaticString` allows for a runtime selection of multiple compile-time known values. + +### Placing `@const` on the declaration type +One alternative to declaring compile-time known values as proposed here with the declaration attribute: + +```swift +@const let x = 11 +``` +Is to instead shift the annotation to declared property's type: + +```swift +let x: @const Int = 11 +``` +This shifts the information conveyed to the compiler about this declaration to be carried by the declaration's type. Semantically, this departs from, and widely broadens the scope from what we intend to capture: the knowability of the declared *value*. Encoding the compile-time property into the type system would force us to reckon with a great deal of complexity and unintended consequences. Consider the following example: + +```swift +typealias CI = @const Int +let x: CI? +``` +What is the type of `x`? It appears to be Optional<@const Int>, which is not a meaningful or useful type, and the programmer most likely intended to have a @const Optional. And although today Implicitly-Unwrapped optional syntax conveys an additional bit of information about the declared value using a syntactic indicator on the declared type, without affecting the declaration's type, the [historical context](https://www.swift.org/blog/iuo/) of that feature makes it a poor example to justify requiring consistency with it. + +### Alternative attribute names +More-explicit spellings of the attribute's intent were proposed in the form of `@buildTime`/`@compileTime`/`@comptime`, and the use of `const` was also examined as a holdover from its use in C++. + +While build-time knowability of values this attribute covers is one of the intended semantic takeaways, the potential use of this attribute for various optimization purposes also lends itself to indicate the additional immutability guarantees on top of a plain `let` (which can be initialized with a dynamic value), as well as capturing the build-time evaluation/knowledge signal. For example, in the case of global variables, thread-safe lazy initialization of `@const` variables may no longer be necessary, in which case the meaning of the term `const` becomes even more explicit. + +Similarly with the comparison to C++, where the language uses the term `const` to describe a runtime behaviour concept, rather than convey information about the actual value. The use of the term `const` is more in line with the mathematical meaning of having the value be a **defined** constant. + +## Forward-Looking Design Aspects and Future Directions + +### Future Inference/Propagation Rules +Though this proposal does **not** itself introduce rules of `@const` inference, their future existence is worth discussing in this design. Consider the example: + +```swift +@const let i = 1 +let j = i +``` +While not valid under this proposal, our intent is to allow the use of `i` where `@const` values are expected in the future, for example `@const let k = i` or `f(i)` where `f` is `func f(@const _: Int)`. It is therefore important to consider whether `@const` is propagated to values like `j` in the above example, which determines whether or not statements like `f(j)` and `@const let k = j` are valid code. While it is desirable to allow such uses of the value within the same compilation unit, if `j` is `public`, automatically inferring it to be `@const` is problematic at the module boundary: it creates a contract with the module's clients that the programmer may not have intended. Therefore, `public` properties must explicitly be marked `@const` in order to be accessible as such outside the defining module. This is similar in nature to `Sendable` inference - `internal` or `private` entities can automatically be inferred by the compiler as `Sendable`, while `public` types must explicitly opt-in. + +### Memory placement and runtime initialization +Effect on runtime placement of `@const` values is an implementation detail that this proposal does not cover beyond indicating that today this attribute has no effect on memory layout of such values at runtime. It is however a highly desirable future direction for the implementation of this feature to allow the use of read-only memory for `@const` values. With this in mind, it is important to allow semantics of this attribute to allow such implementation in the future. For example, a global `@const let`, by being placed into read-only memory removes the need for synchronization on access to such data. Moreover, using read-only memory reduces memory pressure that comes from having to maintain all mutable state in-memory at a given program point - read-only data can be evicted on-demand to be read back later. These are desirable traits for optimization of existing programs which become increasingly important for enabling of low-level system programs to be written in Swift. + +In order to allow such implementation in the future, this proposal makes the *value* of `public` `@const` values/properties a part of a module's ABI. That is, a resilient library that vends `@const let x = 11` changing the value of `x` is considered an ABI break. This treatment allows `public` `@const` data to exist in a single read-only location shared by all library clients, without each client having to copy the value or being concerned with possible inconsistency in behavior across library versions. + +## Future Directions + +### Constant-propagation +Allow default-initialization of `@const` properties using other `@const` values and allow passing `@const` values to `@const` parameters. The [Future Inference/Propagation Rules](#future-inferencepropagation-rules) section discusses a direction for enabling inference of the attribute on values. This is a necessary next building-block to generalizing the use of compile-time values. + +```swift +func foo(@const i: Int) { + @const let j = i +} +``` + +### Toolchain support for extracting compile-time values at build time. +The current proposal covers an attribute that allows clients to build an API surface that is capable of carrying semantic build-time information that may be very useful to build-time tooling, such as [SwiftPM plugins](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md). The next step towards this goal would include toolchain support for tooling that extracts such information in a client-agnostic fashion so that it can be adopted equally by use-cases like the manifest example in [Facilitate Compile-time Extraction of Values](#facilitate-compile-time-extraction-of-values) and others. + +### Compile-time expressions and functions +Building on propagation and inference of `@const`, some of the most interesting use-cases for compile-time-known values emerge with the ability to perform operations on them that result in other compile-time-known values. For example, the [Compiler Diagnostic Directives](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md) could be expanded to trigger conditionally based on a value of a compile-time-known input expression: + +```swift +func foo(@const input: Int) { + #const_assert(input <= 0, "'foo()' expects a positive input") +} +``` +Which would require that it is possible to evaluate the `input <= 0` expression at compile-time, which would also require that certain functions can be evaluated at compile-time if their arguments are known at compile-time. For example: + + +```swift +func <=(@const lhs: Int, @const rhs: Int) -> Bool +``` +Inference on which functions can or cannot be evaluated at compile time will be defined in a future proposal and can follow similar ideas to those described in [Future Inference/Propagation Rules](#future-inferencepropagation-rules) section. + +### Compile-time types +Finally, the flexibility of build-time values and code that operates on them can be greatly expanded by allowing entire user-defined types to form compile-time-known values via either custom literal syntax or having a `@const` initializer. \ No newline at end of file diff --git a/proposals/0360-opaque-result-types-with-availability.md b/proposals/0360-opaque-result-types-with-availability.md new file mode 100644 index 0000000000..d326954662 --- /dev/null +++ b/proposals/0360-opaque-result-types-with-availability.md @@ -0,0 +1,245 @@ +# Opaque result types with limited availability + +* Proposal: [SE-0360](0360-opaque-result-types-with-availability.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Implementation: [apple/swift#42072](https://github.com/apple/swift/pull/42072), [apple/swift#42104](https://github.com/apple/swift/pull/42104), [apple/swift#42167](https://github.com/apple/swift/pull/42167), [apple/swift#42456](https://github.com/apple/swift/pull/42456) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0360-opaque-result-types-with-limited-availability/58712) + +## Introduction + +Since their introduction in [SE-0244](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md), opaque result types have become a powerful tool of type-level abstraction that allows library authors to hide implementation details of their APIs. + +Under the rules described in SE-0244 - a function returning an opaque result type *must return a value of the same concrete type `T`* from each `return` statement, and `T` must meet all of the constraints stated on the opaque type. + +The same-type `return` requirement is unnecessarily strict when it comes to availability conditions. SE-0244 states that it should be possible to change the underlying type in the future version of the library, but that would only work with pre-existing types. In other words, the same-type condition does not have to apply across executions of the same program, the same way that `Hashable` must produce the same output for the same value during program execution, but may produce a different value in the next execution. `#available` is special because it's a checkable form of that: dynamic availability will not change while the program is running, but may be different the next time the program runs. + +Current model and implementation limit usefulness of opaque result types as an abstraction mechanism, because it prevents frameworks from introducing new types and using them as underlying types in existing APIs. To bridge this usability gap, I propose to relax same-type restriction for `return`s inside of availability conditions. + +Swift-evolution thread: [ +[Pitch] Opaque result types with limited availability](https://forums.swift.org/t/pitch-opaque-result-types-with-limited-availability/57286) + +## Motivation + +To illustrate the problematic interaction between opaque result types and availability conditions, let's consider a framework that already has a `Shape` protocol and a `Square` type that conforms to the `Shape` protocol. + +``` +protocol Shape { + func draw(to: Surface) +} + +struct Square : Shape { + ... +} +``` + +In a new version of the framework, the library authors decided to introduce a new shape - `Rectangle` with limited availability: + +``` +@available(macOS 100, *) +struct Rectangle : Shape { + ... +} +``` + +Since a `Rectangle` is generalization of a `Square` it makes sense to allow transforming a `Square` into a `Rectangle` but that currently requires extension with limited availability: + +``` +@available(macOS 100, *) +extension Square { + func asRectangle() -> some Shape { + return Rectangle(...) + } +} +``` + +The fact that the new method has to be declared in availability context to return `Rectangle` limits its usefulness because all uses of `asRectangle` would have to be encapsulated into `if #available` blocks. + +If `asRectangle` already existed in the original version of the framework, it wouldn’t be possible to use a new type at all without declaring `if #available` block in its body: + +``` +struct Square { + func asRectangle() -> some Shape { + if #available(macOS 100, *) { + return Rectangle(...) + } + + return self + } +} +``` + +But doing so is not allowed because all of the `return` statements in the body of the `asRectangle` function have to return the same concrete type: + +``` + error: function declares an opaque return type 'some Shape', but the return statements in its body do not have matching underlying types + func asRectangle() -> some Shape { + ^ ~~~~~~~~~~ +note: return statement has underlying type 'Rectangle' + return Rectangle() + ^ +note: return statement has underlying type 'Square' + return Square() + ^ +``` + +This is a dead-end for the library author although SE-0244 states that it should be possible to change underlying result type in the future version of the library/framework but that assumes that the type already exists so it could be used in all `return` statements. + +## Proposed solution + +To bridge this usability gap, I propose to relax the same-type restriction for functions with `if #available` conditions: if an `if #available` condition is always executed, it can return a different type than the type returned by the rest of the function. + +The proposed changes allow functions to: + +* use multiple `if #available` conditions to return different types based on their dynamic availability and +* safely fall back to a return type with no availability restrictions if none of the availability conditions are met. + +Because the return type must be decidable without running the code in the function, mixing availability conditions with other conditions (such as `if`, `guard`, or `switch`) removes this special power and requires `return`s in the `if #available` to return the same type as the rest of the function. + +This example satisfies these rules: + +```swift +func test() -> some Shape { + if #available(macOS 100, *) { ✅ + return Rectangle() + } + + return self +} +``` + +## Detailed design + +An *unconditional availability clause* is an `if` or `else if` clause that satisfies the following conditions: + + - The clause is part of an `if` statement at the top level of the containing function. + - There are no `return` statements in the containing function prior to the `if` statement. + - The condition of the clause is an `#available` condition. + - The clause is either the initial `if` clause or an `else if` clause immediately following an unconditional availability clause. + - The clause contains at least one `return` statement. + - All paths through the block controlled by the clause terminate by either returning or throwing. + +All `return` statements outside of unconditional availability clauses must return the same type as each other, and this type must be as available as the containing function. + +All `return` statements within a given unconditional availability clause must return the same type as each other, and this type must be as available as the `#available` condition of the clause. This type need not be the same type returned by any `return` statement outside of the clause. + +There must be at least one `return` statement in the containing function. If there are no `return` statements outside of unconditional availability clauses, then at least one of the return types within unconditional availability clauses must be as available as the containing function. + +Dynamically, the return type of the containing function is: + - the return type of `return` statements in the first unconditional availability clause whose condition is dynamically satisfied, or if none are satisfied then + - the return type of `return` statements outside of all unconditional availability clauses, or if there are no such statements then + - the return type of `return` statements in the first unconditional availability clause that is as available as the containing function. + +Now let's consider a couple of examples to better demonstrate the difference between well-formed and invalid functions under the proposed rules. + +The following example is well-formed because the first `if #available` statement terminates with a `return` and the second one is associated with a valid `if #available` and also terminates with a `return`. + + ```swift + func test() -> some Shape { + if #available(macOS 100, *) { ✅ + return Rectangle() + } else if #available(macOS 99, *) { ✅ + return Square() + } + return self + } + ``` + + But + + ```swift + func test() -> some Shape { + if cond { + ... + } else if #available(macOS 100, *) { ❌ + return Rectangle() + } + return self + } + ``` + +is not accepted by the compiler because `if #available` associated with a dynamic condition. + +The following is incorrect because `if #available` is preceded by a dynamic condition that returns: + +```swift +func test() -> some Shape { + guard let x = else { + return ... + } + + if #available(macOS 100, *) { ❌ + return Rectangle() + } + + return self +} +``` + +Similarly, the following is incorrect because `if #available` appears inside of a loop: + +```swift +func test() -> some Shape { + for ... { + if #available(macOS 100, *) { ❌ + return Rectangle() + } + } + return self +} +``` + +The following `test()` function is well-formed because `if` statement produces the same result in both of its branches and it's statically known that the `if #available` always terminates with a `return` + + ```swift + func test() -> some Shape { + if #available(macOS 100, *) { + if cond { ✅ + return Rectangle(...) + } else { + return Rectangle(...) + } + } + return self + } + ``` + + But: + + ```swift + func test() -> some Shape { + if #available(macOS 100, *) { + if cond { ❌ + return Rectangle() + } else { + return Square() + } + } + return self + } + ``` + +is not going to be accepted by the compiler because return types are different: `Rectangle` vs. `Square`. + +This semantic adjustment fits well into the existing model because it makes sure that there is always a single underlying type per platform and universally. + +## Source compatibility + +Proposed changes do not break source compatibility and allow previously incorrect code to compile. + +## Effect on ABI stability + +No ABI impact since this is an additive change. + +## Effect on API resilience + +All of the resilience rules associated with opaque result types are preserved. + +## Alternatives considered + +* Only alternative is to change the API patterns used in the library, e.g. by exposing the underlying result type and overloading the method. + +## Acknowledgments + +[John McCall](https://forums.swift.org/u/john_mccall) for the help with `Proposed Solution` and `Detail Design` improvements. diff --git a/proposals/0361-bound-generic-extensions.md b/proposals/0361-bound-generic-extensions.md new file mode 100644 index 0000000000..8b91826d08 --- /dev/null +++ b/proposals/0361-bound-generic-extensions.md @@ -0,0 +1,196 @@ +# Extensions on bound generic types + +* Proposal: [SE-0361](0361-bound-generic-extensions.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#41172](https://github.com/apple/swift/pull/41172), gated behind the frontend flag `-enable-experimental-bound-generic-extensions` +* Review: ([pitch](https://forums.swift.org/t/pitch-extensions-on-bound-generic-types/57535/)) ([review](https://forums.swift.org/t/se-0361-extensions-on-bound-generic-types/58366)) ([acceptance](https://forums.swift.org/t/accepted-se-0361-extensions-on-bound-generic-types/58716)) + +## Contents + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Alternatives considered](#alternatives-considered) + - [Reserving syntax for parameterized extensions](#reserving-syntax-for-parameterized-extensions) + - [Future directions](#future-directions) + - [Parameterized extensions](#parameterized-extensions) + +## Introduction + +Specifying the type arguments to a generic type in Swift is almost always written in angle brackets, such as `Array`. Extensions are a notable exception, and if you attempt to extend `Array`, the compiler reports the following error message: + +```swift +extension Array { ... } // error: Constrained extension must be declared on the unspecialized generic type 'Array' with constraints specified by a 'where' clause +``` + +As the error message suggests, this extension must instead be written using a `where` clause: + +```swift +extension Array where Element == String { ... } +``` + +This proposal removes this limitation on extensions, allowing you to write bound generic extensions the same way you write bound generic types everywhere else in the language. + +Swift evolution discussion thread: [[Pitch] Extensions on bound generic types](https://forums.swift.org/t/pitch-extensions-on-bound-generic-types/57535). + +## Motivation + +Nearly everywhere in the language, you write bound generic types using angle brackets after the generic type name. For example, you can write a typealias to an array of strings using angle brackets, and extend that type using the typealias: + +```swift +typealias StringArray = Array + +extension StringArray { ... } +``` + +With [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md), we can also declare a primary associated type, and bind it in an extension using angle-brackets: + +```swift +protocol Collection { + associatedtype Element +} + +extension Collection { ... } +``` + +Not allowing this syntax directly on generic type extensions is clearly an artificial limitation, and even the error message produced by the compiler suggests that the compiler understood what the programmer was trying to do: + +```swift +extension Array { ... } // error: Constrained extension must be declared on the unspecialized generic type 'Array' with constraints specified by a 'where' clause +``` + +This limitation is confusing, because programmers don’t understand why they can write `Array` everywhere *except* to extend `Array`, as evidenced by the numerous questions about this limitation here on the forums, such as [this thread](https://forums.swift.org/t/why-doesnt-eg-extension-array-int-compile-even-though-using-a-typealias-does/56049). + +## Proposed solution + +I propose to allow extending bound generic types using angle-brackets for binding type arguments, or using sugared types such as `[String]` and `Int?`. + +The following declarations all express an extension over the same type: + +```swift +extension Array where Element == String { ... } + +extension Array { ... } + +extension [String] { ... } +``` + +## Detailed design + +A generic type name in an extension can be followed by a comma-separated type argument list in angle brackets. The type argument list binds the type parameters of the generic type to each of the specified type arguments. This is equivalent to writing same-type requirements in a `where` clause. For example: + +```swift +struct Pair {} + +extension Pair {} +``` + +is equivalent to + +```swift +extension Pair where T == Int, U == String {} +``` + +A type argument list applied to an extended type name must specify all type arguments; constraining only a subset of the type parameters in an extension must still use a `where` clause: + +```swift +struct Pair {} + +extension Pair {} // error: Generic type 'Pair' specialized with too few type parameters (got 1, but expected 2) + +extension Pair where T == Int {} // okay +``` + +The types specified in the type argument list must be concrete types. For example, you cannot extend a generic type with placeholders as type arguments: + +```swift +extension Pair {} // error: Cannot extend a type that contains placeholders +``` + +> **Rationale**: When `_` is used as a type placeholder, it directs the compiler to infer the type at the position of the underscore. Using `_` in a bound generic extension would introduce a subtly different meaning of `_`, which is to leave the type at that position unconstrained, so `Pair` would mean different things in different contexts. + +Name lookup of the type arguments is performed outside the extension context, so the type parameters of the generic type cannot appear in the type argument list: + +```swift +extension Array {} // error: Cannot find type 'Element' in scope +``` + +If a generic type has a sugared spelling, the sugared type can also be used to extend the generic type: + +```swift +extension [String] { ... } // Extends Array + +extension String? { ... } // Extends Optional +``` + +## Source compatibility + +This change has no impact on source compatibility. + +## Effect on ABI stability + +This is a syntactic sugar change with no impact on ABI. + +## Effect on API resilience + +This change has no impact on API resilience. Changing an existing bound generic extension using a where clause to the sugared syntax and vice versa is a resilient change. + +## Alternatives considered + +### Reserving syntax for parameterized extensions + +Using angle brackets after an extended type name as sugar for same-type requirements prevents this syntax from being used to declare a parameterized extension. Alternatively, `extension Array { ... }` could mean an extension that declares two new type parameters `T` and `U`, rather than an (invalid) application of type arguments to `Array`'s type parameters. However, SE-0346 already introduced this syntax as sugar for same-type requirements on associated types: + +```swift +protocol Collection { + associatedtype Element +} + +// Already sugar for `extension Collection where Element == String` +extension Collection { ... } +``` + +Instead of reserving this syntax for parameterized extensions, type parameters could be declared in angle brackets after the `extension` keyword, which will help indicate that the type parameters belong to the extension: + +```swift +// Introduces new type parameters `T` and `U` for the APIs +// in this extension. +extension Array { ... } +``` + +## Future directions + +### Parameterized extensions + +This proposal does not provide parameterized extensions, but a separate proposal could build upon this proposal to allow extending a generic type with more sophisticated constraints on the type parameters: + +```swift +extension Array> { ... } + +extension [Wrapped?] { ... } +``` + +Parameterized extensions could also allow using the shorthand `some` syntax to write generic extensions where a type parameter has a conformance requirement: + +```swift +extension Array { ... } + +extension [some Equatable] { ... } +``` + +Writing the type parameter list after the `extension` keyword applies more naturally to extensions over structural types. With this syntax, an extension over all two-element tuples could be spelled + +```swift +extension (T, U) { ... } +``` + +This syntax also generalizes to variadic type parameters, e.g. to extend all tuple types to provide a protocol conformance: + +```swift +extension (T...): Hashable where T: Hashable { ... } +``` diff --git a/proposals/0362-piecemeal-future-features.md b/proposals/0362-piecemeal-future-features.md new file mode 100644 index 0000000000..d52071ddd2 --- /dev/null +++ b/proposals/0362-piecemeal-future-features.md @@ -0,0 +1,171 @@ +# Piecemeal adoption of upcoming language improvements + +* Proposal: [SE-0362](0362-piecemeal-future-features.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#59055](https://github.com/apple/swift/pull/59055), [apple/swift-package-manager#5632](https://github.com/apple/swift-package-manager/pull/5632) +* Review: ([pitch](https://forums.swift.org/t/piecemeal-adoption-of-swift-6-improvements-in-swift-5-x/57184)) ([review](https://forums.swift.org/t/se-0362-piecemeal-adoption-of-future-language-improvements/58384)) ([acceptance](https://forums.swift.org/t/accepted-se-0362-piecemeal-adoption-of-future-language-improvements/59076)) + +## Introduction + +Swift 6 is accumulating a number of improvements to the language that have enough source-compatibility impact that they could not be enabled by default in prior language modes (Swift 4.x and Swift 5.x). These improvements are already implemented in the Swift compiler behind the Swift 6 language mode, but they are inaccessible to users, and will remain so until Swift 6 becomes available as a language mode. There are several reasons why we should consider making these improvements available sooner: + +* Developers would like to get the benefits from these improvements soon, rather than wait until Swift 6 is available. +* Making these changes available to developers prior to Swift 6 provides more experience, allowing us to tune them further for Swift 6 if necessary. +* The sum of all changes made in Swift 6 might make migration onerous for some modules, and adopting these language changes one-by-one while in Swift 4.x/5.x can smooth that transition path. + +A few proposals have already introduced bespoke solutions to provide a migration path: [SE-0337](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md) adds `-warn-concurrency` to enable warnings for `Sendable`-related checks in Swift 4.x/5.x. [SE-0354](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0354-regex-literals.md) adds the flag `-enable-bare-slash-regex` to enable the bare `/.../` regular expression syntax. And although it wasn't part of the proposal, the discussion of [SE-0335](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md) included requests for a compiler flag to require `any` on all existentials. These all have the same flavor, of opting existing Swift 4.x/5.x code into improvements that will come in Swift 6. + +This proposal explicitly embraces the piecemeal, intentional adoption of features that were held until Swift 6 for source-compatibility reasons. It establishes a direct path to incrementally adopt Swift 6 features, one-by-one, to gain their benefits in a Swift 4.x/5.x code base and smooth the migration path to a Swift 6 language mode. Developers can use a new compiler flag, `-enable-upcoming-feature X` to enable the specific feature named `X` for that module, and multiple features can be specified in this manner. When the developer moves to the next major language version, `X` will be implied by that language version and the compiler flag will be rejected. This way, upcoming feature flags only accumulate up to the next major Swift language version and are then cleared away, so we don't fork the language into incompatible dialects. + +Swift-evolution thread: [Pitch #1](https://forums.swift.org/t/piecemeal-adoption-of-swift-6-improvements-in-swift-5-x/57184) + +## Language version and tools version + +There are two related kinds of "Swift version" that are distinct, but we often conflate them for convenience. However, both kinds of version have a bearing on this proposal: + +- *Swift tools version*: the version number of the compiler itself. For example, the Swift 5.6 compiler was introduced in March 2022. +- *Swift language version*: the language version with which we are providing source compatibility. For example, Swift version 5 is the most current language version supported by Swift tools version 5.6. + +The Swift tools support multiple Swift language versions. All recent versions (since Swift tools version 5.0) have supported multiple Swift language versions, of which there are currently only three: 4, 4.2, and 5. As the tools evolve, they try to avoid making source-incompatible changes within a Swift language version, and this is also reflected in the evolution process itself: proposals that change the meaning of existing source code, or make it invalid, are generally not accepted for existing language modes. Many proposals do *extend* the Swift language within an existing language mode. For example, `async`/`await` became available in Swift tools version 5.5, and is available in all language versions (4, 4.2, 5). + +This proposal involves source-incompatible changes that are waiting for the introduction of a new Swift language version, e.g., 6. Swift tools version 6.0 will be the first tools to officially allow the use of Swift language version 6. Those tools will continue to support Swift language versions 4, 4.2, and 5. Code does not need to move to Swift language version 6 to use Swift tools version 6.0, or 6.1, and so on, and code written to Swift language version 6 will interoperate with code written to Swift language version 4, 4.2, or 5. + +## Proposed solution + +Introduce a compiler flag `-enable-upcoming-feature X`, where `X` is a name for the feature to enable. Each proposal will document what `X` is, so it's clear how to enable that feature. For example, SE-0274 could use `ConciseMagicFile`, so that `-enable-upcoming-feature ConciseMagicFile` will enable that change in semantics. One can of course pass multiple `-enable-upcoming-feature` flags to the compiler to enable multiple features. + +Unrecognized upcoming features will be ignored by the compiler. This allows older tools to use the same command lines as newer tools for Swift code that has started adopting new features, but has appropriate workarounds to still work with older tools. Sometimes this is possible because older compilers will still have a reasonable interpretation of the code, other times one will need a way to [detect features in source code](#feature-detection-in-source-code), the subject of a later section. + +All "upcoming" features are enabled by default in some language version. The compiler will produce an error if `-enable-upcoming-feature X` is provided and the language version enables the feature `X` by default. This will make it clear to developers when their expectations about when a feature is available, and clean up projects and manifests that have evolved from from earlier language versions, adopted features piecemeal, and then moved to later language versions. + +### Proposals define their own feature identifier + +Amend the [Swift proposal template](https://github.com/swiftlang/swift-evolution/blob/main/proposal-templates/0000-swift-template.md) with a new, optional field that defines the feature identifier: + +* **Feature identifier**: `UpperCamelCaseFeatureName` + +Amend the following proposals, which are partially or wholly delayed until Swift 6, with the following feature identifiers: + +* [SE-0274 "Concise magic file names"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0274-magic-file.md) (`ConciseMagicFile`) delayed the semantic change to `#file` until Swift 6. Enabling this feature changes `#file` to mean `#fileID` rather than `#filePath`. +* [SE-0286 "Forward-scan matching for trailing closures"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0286-forward-scan-trailing-closures.md) (`ForwardTrailingClosures`) delays the removal of the "backward-scan matching" rule of trailing closures until Swift 6. Enabling this feature removes the backward-scan matching rule. +* [SE-0335 "Introduce existential `any`"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md) (`ExistentialAny`) delays the requirement to use `any` for all existentials until Swift 6. Enabling this feature requires `any` for existential types. +* [SE-0337 "Incremental migration to concurrency checking"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md) (`StrictConcurrency`) delays some checking of the concurrency model to Swift 6 (with a flag to opt in to warnings about it in Swift 5.x). Enabling this feature is equivalent to `-warn-concurrency`, performing complete concurrency checking. +* [SE-0352 "Implicitly Opened Existentials"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md) (`ImplicitOpenExistentials`) expands implicit opening to more cases in Swift 6, because we didn't want to change the semantics of well-formed code in Swift 5.x. Enabling this feature performs implicit opening in these additional cases. +* [SE-0354 "Regex Literals"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0354-regex-literals.md) (`BareSlashRegexLiterals`) delays the introduction of the `/.../` regex literal syntax until Swift 6. Enabling this feature is equivalent to `-enable-bare-regex-syntax`, making the `/.../` regex literal syntax available. If this proposal and SE-0354 are accepted in the same release, `-enable-bare-regex-syntax` can be completely removed in favor of this approach. + +### Swift Package Manager support for upcoming features + +SwiftPM targets should be able to specify the upcoming language features they require. Extend `SwiftSetting` with an API to enable an upcoming feature: + +```swift +extension SwiftSetting { + public static func enableUpcomingFeature( + _ name: String, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting +} +``` + +SwiftPM would then pass each of the upcoming features listed there to the compiler via the `-enable-upcoming-feature` flag when building a module using this setting. Other targets that depend on this one do not need to pass the features when they build, because the effect of upcoming features does not cross module boundaries. + +The features are provided as strings here so that SwiftPM's manifest format doesn't need to change each time a new feature is added to the compiler. Package authors can add upcoming features while still supporting older tools without creating a new, versioned manifest. + +### Feature detection in source code + +When adopting a new feature, it's common to want code to still compile with older tools where that feature is not available. Doing so requires a way to check whether the feature is enabled, either by `-enable-upcoming-feature` or by enabling a suitable language version. + +We should extend Swift's `#if` with explicit support for a `hasFeature(X)` check, which evaluates true whenever the feature with identifier `X` is available. Code that needs to check for a specific feature can use `#if hasFeature` like this: + +```swift +#if hasFeature(ImplicitOpenExistentials) + f(aCollectionOfInts) +#else + f(AnyCollection(aCollectionOfInts)) +#endif +``` + +The `hasFeature(X)` check indicates the presence of features, but by itself an older compiler will still attempt to parse the `#if` branch even if the feature isn't known. That's fine for this feature (implicitly opened existentials) because it doesn't add any syntax, but other features that add syntax might require something more. `hasFeature` can be composed with the `compiler` directive introduced by [SE-0212](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0212-compiler-version-directive.md), e.g., + +```swift +#if compiler(>=5.7) && hasFeature(BareSlashRegexLiterals) +let regex = /.../ +#else +let regex = try NSRegularExpression(pattern: "...") +#endif +``` + +There is an issue with the above, because `hasFeature` *itself* is not understood by tools that predate this proposal, so the code above will fail to compile with any Swift compiler that predates the introduction of `hasFeature`. It is possible to avoid this problem by nesting the `hasFeature` check like this (assuming that Swift 5.7 introduced `hasFeature`): + +```swift +#if compiler(>=5.7) + #if hasFeature(BareSlashRegexLiterals) + let regex = /.../ + #else + let regex = #/.../# + #endif +#else +let regex = try NSRegularExpression(pattern: "...") +#endif +``` + +In the worst case, this does involve some code duplication for libraries that need to work on Swift versions that predate the introduction of `hasFeature`, but it is possible to handle those compilers, and over time that limitation will go away. + +To prevent this issue for any upcoming extensions to the `#if` syntax, the compiler should not attempt to interpret any "call" syntax on the right-hand side of a `&&` or `||` whose left-hand side disables parsing of the `#if` body, such as `compiler(>=5.7)` or `swift(>=6.0)`, and where the right-hand term is not required to determine the result of the whole expression. For example, if we invent something like `#if hasAttribute(Y)` in the future, one can use this formulation: + +```swift +#if compiler(>=5.8) && hasAttribute(Sendable) +... +#endif +``` + +On Swift 5.8 or newer compilers (which we assume will support `hasAttribute`), the full condition will be evaluated. On prior Swift compilers (i.e., ones that support this proposal but not something newer like `hasAttribute`), the code after the `&&` or `||` will be parsed as an expression, but will not be evaluated, so such compilers will not reject this `#if` condition. + +### Embracing experimental features + +It is common for language features in the compiler to be staged in behind an "experimental" flag as they are developed. This is usually done in an ad hoc manner, and the flag is removed before the feature finally ships. However, we should embrace the experimental feature model further: when a feature is under development, provide it with a feature identifier that allows it to be enabled with a new flag, `-enable-experimental-feature X`, or its SwiftPM counterpart `enableExperimentalFeature`. + +Experimental features are still to be considered unstable, and should not be available in released compilers. However, by unifying the manner in which experimental and upcoming features are introduced, we can rely on the same staging mechanisms: a way to enable the feature and to check for its presence in source code, making it easier to experiment with these features. If a feature then "graduates" to a complete, supported language feature, `hasFeature` can return true for it and, if part of it was delayed until the next major language version, `-enable-upcoming-feature` will work with it, too. + +## Source compatibility + +For the language itself, `hasFeature` is the only addition, and it occurs in a constrained syntactic space (`#if`) where there are no user-definable functions. Therefore, there is no source-compatibility issue in the traditional sense, where a newer compiler rejects existing, well-formed code. + +For SwiftPM, the addition of the `enableUpcomingFeature` and `enableExperimentalFeature` functions to `SwiftSetting` represents a one-time break in the manifest file format. Packages that wish to adopt these functions and support tools versions that predate the introduction of `enableUpcomingFeature` and `enableExperimentalFeature` can use versioned manifest, e.g., `Package@swift-5.6.swift`, to adopt the feature for newer tools versions. Once `enableUpcomingFeature` and `enableExperimentalFeature` have been added, adopting additional features this way won't require another copy of the manifest file. + +## Alternatives considered + +### `$X` instead of `hasFeature(X)` + +The original pitch for this proposal used special identifiers `$X` for feature detection instead of `hasFeature(X)`. `$X` has been used in the compiler implementation to help stage in Swift's concurrency features, especially when producing Swift interface files that might need to be understood by older tools versions. The compiler still defines `$AsyncAwait`, for example, which can be used with `#if` to check for async/await support: + +```swift +#if compiler(>=5.3) && $AsyncAwait +func f() async -> String +#endif +``` + +The primary advantage to the `$` syntax is that all Swift compilers already treat `$` as an acceptable leading character for an identifier. The compiler can define names with a leading `$`, but developers aren't technically supposed to, so it's effectively a reserved space for "magic" names. This means that, unlike the `hasFeature` formulation of the above, older compilers can process the code above without producing an error. + +However, this proposal introduces `hasFeature` because it's clearer in the code, and makes the forward-looking changes to the way `#if` conditions are processed to make it easier for additional `hasFeature`-like features to be introduced in the future without having this problem with older compilers. + +### Enabling optional features + +This proposal narrowly introduces `-enable-upcoming-feature` to only describe accepted features that will be enabled with a newer language version, but that were held back (partially or in full) due to source compatibility concerns. It is not meant to be used to enable "optional" features, which would create permanent dialects, and is designed to be somewhat self-healing: as folks move to newer language modes (e.g., Swift 6), the upcoming feature flags are eliminated with the new baseline. + +### Enabling all upcoming features + +The set of upcoming features will expand over time, as Swift introduces new features with source-compatibility impact that are staged in via a new major language version. For developers who want to be on the leading edge, it would be more convenient to have a single flag that enables all upcoming features, rather than having to specify each upcoming feature as they get added. However, the introduction of such a flag would create a shifting dialect of Swift: features are only "upcoming" features if they have source-compatibility impact, so code that adopted this flag could break with every new Swift release. That would directly cut against our source-compatibility goals for Swift, so we do not propose such a flag. Instead, we should find a central place to document all upcoming features on swift.org, updated with each release, so that developers know where to go to learn about the new upcoming features they want to enable. + +## Revision History + +* Changes from first reviewed version: + * Changed the SwiftPM manifest API to be based on `SwiftSettings` rather than the target. + * Use the term "upcoming feature" rather than "future feature" to reduce confusion. + * Don't parse the right-hand side of a `&&` or `||` that doesn't affect the result. + * Add some discussion of language and tools versions. + +## Acknowledgments + +Becca Royal-Gordon designed the original `#if compiler(>=5.5) && $AsyncAwait` approach to adopting features without breaking compatibility with older tools, and helped shape this design. Ben Rimmington provided the design for the SwiftPM API, replacing the less-flexible design from the original reviewed proposal. diff --git a/proposals/0363-unicode-for-string-processing.md b/proposals/0363-unicode-for-string-processing.md new file mode 100644 index 0000000000..a56e074ddf --- /dev/null +++ b/proposals/0363-unicode-for-string-processing.md @@ -0,0 +1,1361 @@ +# Unicode for String Processing + +* Proposal: [SE-0363](0363-unicode-for-string-processing.md) +* Authors: [Nate Cook](https://github.com/natecook1000), [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift-experimental-string-processing][repo] +* Review: ([pitch](https://forums.swift.org/t/pitch-unicode-for-string-processing/56907)), ([review](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520)), ([acceptance](https://forums.swift.org/t/accepted-se-0363-unicode-for-string-processing/59998)) + +### Version History + +- Version 1: Initial version +- Version 2: + - Improved option method API names + - Added Unicode property APIs to match regex syntax + - Added `CharacterClass.noneOf(_:)` and sequence-based `init` + - Clarified default state of options + - Added detail around switching semantic modes + - Added detail about Unicode property matching in character mode + - Revised details of custom character class matching + - Removed `\O`/`.anyUnicodeScalar` + +### Table of Contents + + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [Options](#options) + - [Case insensitivity](#case-insensitivity) + - [Single line mode (`.` matches newlines)](#single-line-mode--matches-newlines) + - [Multiline mode](#multiline-mode) + - [ASCII-only character classes](#ascii-only-character-classes) + - [Unicode word boundaries](#unicode-word-boundaries) + - [Matching semantic level](#matching-semantic-level) + - [Default repetition behavior](#default-repetition-behavior) + - [Character Classes](#character-classes) + - [“Any”](#any) + - [Digits](#digits) + - ["Word" characters](#word-characters) + - [Whitespace and newlines](#whitespace-and-newlines) + - [Unicode properties](#unicode-properties) + - [POSIX character classes: `[:NAME:]`](#posix-character-classes-name) + - [Custom classes](#custom-classes) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Future directions](#future-directions) + - [Alternatives considered](#alternatives-considered) + +## Introduction + +This proposal describes `Regex`'s rich Unicode support during regex matching, along with the character classes and options that define and modify that behavior. + +This proposal is one component of a larger [regex-powered string processing initiative](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md). For the status of each proposal, [see this document](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md) — discussion of other facets of the overall regex design is out of scope of this proposal and better discussed in the most relevant review. + +## Motivation + +Swift's `String` type provides, by default, a view of `Character`s or [extended grapheme clusters][graphemes] whose comparison honors [Unicode canonical equivalence][canoneq]. Each character in a string can be composed of one or more Unicode scalar values, while still being treated as a single unit, equivalent to other ways of formulating the equivalent character: + +```swift +let str = "Cafe\u{301}" // "Café" +str == "Café" // true +str.dropLast() // "Caf" +str.last == "é" // true (precomposed e with acute accent) +str.last == "e\u{301}" // true (e followed by composing acute accent) +``` + +This default view is fairly novel. Most languages that support Unicode strings generally operate at the Unicode scalar level, and don't provide the same affordance for operating on a string as a collection of grapheme clusters. In Python, for example, Unicode strings report their length as the number of scalar values, and don't use canonical equivalence in comparisons: + +```python +cafe = u"Cafe\u0301" +len(cafe) # 5 +cafe == u"Café" # False +``` + +Existing regex engines follow this same model of operating at the Unicode scalar level. To match canonically equivalent characters, or have equivalent behavior between equivalent strings, you must normalize your string and regex to the same canonical format. + +```python +# Matches a four-element string +re.match(u"^.{4}$", cafe) # None +# Matches a string ending with 'é' +re.match(u".+é$", cafe) # None + +cafeComp = unicodedata.normalize("NFC", cafe) +re.match(u"^.{4}$", cafeComp) # +re.match(u".+é$", cafeComp) # +``` + +With Swift's string model, this behavior would surprising and undesirable — Swift's default regex semantics must match the semantics of a `String`. + +
Other engines + +Other regex engines match character classes (such as `\w` or `.`) at the Unicode scalar value level, or even the code unit level, instead of recognizing grapheme clusters as characters. When matching the `.` character class, other languages will only match the first part of an `"e\u{301}"` grapheme cluster. Some languages, like Perl, Ruby, and Java, support an additional `\X` metacharacter, which explicitly represents a single grapheme cluster. + +| Matching `"Cafe\u{301}"` | Pattern: `^Caf.` | Remaining | Pattern: `^Caf\X` | Remaining | +|---|---|---|---|---| +| C#, Rust, Go, Python | `"Cafe"` | `"´"` | n/a | n/a | +| NSString, Java, Ruby, Perl | `"Cafe"` | `"´"` | `"Café"` | `""` | + +Other than Java's `CANON_EQ` option, the vast majority of other languages and engines are not capable of comparing with canonical equivalence. + +
+ +## Proposed solution + +In a regex's simplest form, without metacharacters or special features, matching behaves like a test for equality. A string always matches a regex that simply contains the same characters. + +```swift +let str = "Cafe\u{301}" // "Café" +str.contains(/Café/) // true +``` + +From that point, small changes continue to comport with the element counting and comparison expectations set by `String`: + +```swift +str.contains(/Caf./) // true +str.contains(/.+é/) // true +str.contains(/.+e\u{301}/) // true +str.contains(/\w+é/) // true +``` + +For compatibility with other regex engines and the flexibility to match at both `Character` and Unicode scalar level, you can switch between matching levels for an entire regex or within select portions. This powerful capability provides the expected default behavior when working with strings, while allowing you to drop down for Unicode scalar-specific matching. + +By default, literal characters and Unicode scalar values (e.g. `\u{301}`) are coalesced into characters the same way as a normal string, as shown above. Metacharacters, like `.` and `\w`, and custom character classes each match a single element at the current matching level. + +For example, these matches fail, because by the time the engine attempts to match the "`\u{301}`" Unicode scalar literal in the regex, the full `"é"` character in `str` has been matched, even though that character is made up of two Unicode scalar values: + +```swift +str.contains(/Caf.\u{301}/) // false - `.` matches "é" character +str.contains(/Caf\w\u{301}/) // false - `\w` matches "é" character +str.contains(/.+\u{301}/) // false - `.+` matches each character +``` + +Alternatively, we can drop down to use Unicode scalar semantics if we want to match specific Unicode sequences. For example, these regexes match an `"e"` followed by any modifier with the specified parameters: + +```swift +str.contains(/e[\u{300}-\u{314}]/.matchingSemantics(.unicodeScalar)) +// true - matches an "e" followed by a Unicode scalar in the range U+0300 - U+0314 +str.contains(/e\p{Nonspacing Mark}/.matchingSemantics(.unicodeScalar)) +// true - matches an "e" followed by a Unicode scalar with general category "Nonspacing Mark" +``` + +Matching in Unicode scalar mode is analogous to comparing against a string's `UnicodeScalarView` — individual Unicode scalars are matched without combining them into characters or testing for canonical equivalence. + +```swift +str.contains(/Café/.matchingSemantics(.unicodeScalar)) +// false - "e\u{301}" doesn't match with /é/ +str.contains(/Cafe\u{301}/.matchingSemantics(.unicodeScalar)) +// true - "e\u{301}" matches with /e\u{301}/ +``` + +Swift's `Regex` follows the level 2 guidelines for Unicode support in regular expressions described in [Unicode Technical Standard #18][uts18], with support for Unicode character classes, canonical equivalence, grapheme cluster matching semantics, and level 2 word boundaries enabled by default. In addition to selecting the matching semantics, `Regex` provides options for selecting different matching behaviors, such as ASCII character classes or Unicode scalar semantics, which corresponds more closely with other regex engines. + +## Detailed design + +First, we'll discuss the options that let you control a regex's behavior, and then explore the character classes that define the your pattern. + +As detailed below, there are a few differences in defaults between Swift's `Regex` and the typical regex engine. In particular: + +- `Regex` matches at the Swift `Character` level, instead of matching Unicode scalars, UTF-16 code units, or bytes. A regex that deliberately matches multi-scalar characters may need to switch to Unicode scalar semantics. +- `Regex` uses "default" word boundaries, instead of "simple" word boundaries. A regex that expects `\b` to always match the boundary between a word character (`\w`) and a non-word character (`\W`) may need to switch to simple word boundaries. +- For multi-line regex literals, extended syntax is automatically enabled, which ignores whitespace both in patterns and within custom character classes. To use semantic whitespace, you can temporarily disable extended mode (`(?-x:...)`), quote a section of your pattern (`\Q...\E`), or escape a space explicitly (`a\ b`). + +### Options + +Options can be enabled and disabled in two different ways: as part of [regex internal syntax][internals], or applied as methods when declaring a `Regex`. For example, both of these `Regex`es are declared with case insensitivity: + +```swift +let regex1 = /(?i)banana/ +let regex2 = Regex { + "banana" +}.ignoresCase() +``` + +Because the `ignoresCase()` option method is defined on `Regex`, you can always use the more readable option-setting interface in conjunction with regex literals or run-time compiled `Regex`es: + +```swift +let regex3 = /banana/.ignoresCase() +``` + +Calling an option-setting method like `ignoresCase()` acts like wrapping the callee in an option-setting group `(?:...)`. That is, while it sets the behavior for the callee, it doesn’t override options that are applied to more specific regions. In this example, the middle `"na"` in `"banana"` matches case-sensitively, despite the outer call to `ignoresCase()`: + +```swift +let regex4 = Regex { + "ba" + Regex { "na" }.ignoresCase(false) + "na" +}.ignoresCase() + +"banana".contains(regex4) // true +"BAnaNA".contains(regex4) // true +"BANANA".contains(regex4) // false + +// Equivalent to: +let regex5 = /(?i)ba(?-i:na)na/ +``` + +The options that `Regex` supports are shown in the table below, in three groups: Options that affect matching behavior for _both regex syntax and APIs_, options that affect the matching behavior of _regex syntax only_, and options with _structural_ or _syntactic_ effects that are only supported through regex syntax. + +| **Matching Behavior** | | | Default | +|------------------------------|----------------|---------------------------------|--------------------| +| Case insensitivity | `(?i)` | `ignoresCase()` | disabled | +| ASCII-only character classes | `(?DSWP)` | `asciiOnlyClasses(_:)` | `.none` | +| Unicode word boundaries | `(?w)` | `wordBoundaryKind(_:)` | `.default` | +| Semantic level | n/a | `matchingSemantics(_:)` | `.graphemeCluster` | +| Default repetition behavior | n/a | `defaultRepetitionBehavior(_:)` | `.eager` | +| **Regex Syntax Only** | | | | +| Single-line mode | `(?s)` | `dotMatchesNewlines()` | disabled | +| Multi-line mode | `(?m)` | `anchorsMatchNewlines()` | disabled | +| Swap eager/reluctant | `(?U)` | n/a | disabled | +| **Structural/Syntactic** | | | | +| Extended syntax | `(?x)`,`(?xx)` | n/a | `xx` enabled in multi-line regex literals; otherwise, off | +| Named captures only | `(?n)` | n/a | disabled | + +#### Case insensitivity + +Regexes perform case sensitive comparisons by default. The `i` option or the `ignoresCase(_:)` method enables case insensitive comparison. + +```swift +let str = "Café" + +str.firstMatch(of: /CAFÉ/) // nil +str.firstMatch(of: /(?i)CAFÉ/) // "Café" +str.firstMatch(of: /cAfÉ/.ignoresCase()) // "Café" +``` + +Case insensitive matching uses case folding to ensure that canonical equivalence continues to operate as expected. + +**Regex syntax:** `(?i)...` or `(?i:...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression that ignores casing when matching. + public func ignoresCase(_ ignoresCase: Bool = true) -> Regex +} +``` + +#### ASCII-only character classes + +With one or more of these options enabled, the default character classes match only ASCII values instead of the full Unicode range of characters. Four options are included in this group: + +* Regex syntax `(?D)`: Match only ASCII members for `\d`, `[:digit:]`, and `CharacterClass.digit`. +* Regex syntax `(?S)`: Match only ASCII members for `\s`, `[:space:]`, and any of the whitespace-representing `CharacterClass` members. +* Regex syntax `(?W)`: Match only ASCII members for `\w`, `[:word:]`, and `CharacterClass.word`. Also only considers ASCII characters for `\b`, `\B`, and `Anchor.wordBoundary`. +* Regex syntax `(?D)`: Match only ASCII members for all POSIX properties (including `digit`, `space`, and `word`). + +This option affects the built-in character classes listed in the "Character Classes" section below. When one or more of these options is enabled, the set of characters matched by those character classes is constrained to the ASCII character set. For example, `CharacterClass.hexDigit` usually matches `0...9`, `a-f`, and `A-F`, in either the ASCII or half-width variants. When the `(?D)` or `.asciiOnlyClasses(.digit)` options are enabled, only the ASCII characters are matched. + +```swift +let str = "0x35AB" +str.contains(/0x(\d+)/.asciiOnlyClasses()) +``` + +**Regex syntax:** `(?DSWP)...` or `(?DSWP...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression that only matches ASCII characters as digits. + public func asciiOnlyClasses(_ kinds: RegexCharacterClassKind = .all) -> Regex +} + +/// A built-in regex character class kind. +/// +/// Pass one or more `RegexCharacterClassKind` classes to `asciiOnlyClasses(_:)` +/// to control whether character classes match any character or only members +/// of the ASCII character set. +public struct RegexCharacterClassKind: OptionSet, Hashable { + public var rawValue: Int { get } + + /// Regex digit-matching character classes, like `\d`, `[:digit:]`, and + /// `\p{HexDigit}`. + public static var digit: RegexCharacterClassKind { get } + + /// Regex whitespace-matching character classes, like `\s`, `[:space:]`, + /// and `\p{Whitespace}`. + public static var whitespace: RegexCharacterClassKind { get } + + /// Regex word character-matching character classes, like `\w`. + public static var wordCharacter: RegexCharacterClassKind { get } + + /// All built-in regex character classes. + public static var all: RegexCharacterClassKind { get } + + /// No built-in regex character classes. + public static var none: RegexCharacterClassKind { get } +} +``` + +#### Unicode word boundaries + +By default, matching word boundaries with the `\b` and `Anchor.wordBoundary` anchors uses Unicode _default word boundaries,_ specified as [Unicode level 2 regular expression support][level2-word-boundaries]. + +Disabling the `w` option switches to _[simple word boundaries][level1-word-boundaries],_ finding word boundaries at points in the input where `\b\B` or `\B\b` match. Depending on the other matching options that are enabled, this may be more compatible with the behavior other regex engines. + +As shown in this example, the default matching behavior finds the whole first word of the string, while the match with simple word boundaries stops at the apostrophe: + +```swift +let str = "Don't look down!" + +str.firstMatch(of: /D\S+\b/) +// "Don't" +str.firstMatch(of: /D\S+\b/.wordBoundaryKind(.simple)) +// "Don" +``` + +You can see more differences between level 1 (simple) and level 2 (default) word boundaries in the following table, generated by calling `matches(of: /\b.+\b/)` on the strings in the first column: + +| Example | Level 1 | Level 2 | +|---------------------|---------------------------------|-------------------------------------------| +| I can't do that. | ["I", "can", "t", "do", "that"] | ["I", "can't", "do", "that", "."] | +| 🔥😊👍 | ["🔥😊👍"] | ["🔥", "😊", "👍"] | +| 👩🏻👶🏿👨🏽🧑🏾👩🏼 | ["👩🏻👶🏿👨🏽🧑🏾👩🏼"] | ["👩🏻", "👶🏿", "👨🏽", "🧑🏾", "👩🏼"] | +| 🇨🇦🇺🇸🇲🇽 | ["🇨🇦🇺🇸🇲🇽"] | ["🇨🇦", "🇺🇸", "🇲🇽"] | +| 〱㋞ツ | ["〱", "㋞", "ツ"] | ["〱㋞ツ"] | +| hello〱㋞ツ | ["hello〱", "㋞", "ツ"] | ["hello", "〱㋞ツ"] | +| 나는 Chicago에 산다 | ["나는", "Chicago에", "산다"] | ["나", "는", "Chicago", "에", "산", "다"] | +| 眼睛love食物 | ["眼睛love食物"] | ["眼", "睛", "love", "食", "物"] | +| 아니ㅋㅋㅋ네 | ["아니ㅋㅋㅋ네"] | ["아", "니", "ㅋㅋㅋ", "네"] | +| Re:Zero | ["Re", "Zero"] | ["Re:Zero"] | +| \u{d}\u{a} | ["\u{d}", "\u{a}"] | ["\u{d}\u{a}"] | +| €1 234,56 | ["1", "234", "56"] | ["€", "1", "234,56"] | + + +**Regex syntax:** `(?-w)...` or `(?-w...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression that uses the specified word boundary algorithm. + /// + /// A simple word boundary is a position in the input between two characters + /// that match `/\w\W/` or `/\W\w/`, or between the start or end of the input + /// and `\w` character. Word boundaries therefore depend on the option-defined + /// behavior of `\w`. + /// + /// The default word boundaries use a Unicode algorithm that handles some cases + /// better than simple word boundaries, such as words with internal + /// punctuation, changes in script, and Emoji. + public func wordBoundaryKind(_ wordBoundaryKind: RegexWordBoundaryKind) -> Regex +} + +public struct RegexWordBoundaryKind: Hashable { + /// A word boundary algorithm that implements the "simple word boundary" + /// Unicode recommendation. + /// + /// A simple word boundary is a position in the input between two characters + /// that match `/\w\W/` or `/\W\w/`, or between the start or end of the input + /// and a `\w` character. Word boundaries therefore depend on the option- + /// defined behavior of `\w`. + public static var simple: Self { get } + + /// A word boundary algorithm that implements the "default word boundary" + /// Unicode recommendation. + /// + /// Default word boundaries use a Unicode algorithm that handles some cases + /// better than simple word boundaries, such as words with internal + /// punctuation, changes in script, and Emoji. + public static var default: Self { get } +} +``` + +#### Matching semantic level + +To support both matching on `String`'s default character-by-character view and more broadly-compatible Unicode scalar-based matching, you can select a matching level for an entire regex or a portion of a regex constructed with the `RegexBuilder` API. + +When matching with *grapheme cluster semantics* (the default), metacharacters like `.` and `\w`, custom character classes, and character class instances like `.any` match a grapheme cluster, corresponding with the default string representation. In addition, matching with grapheme cluster semantics compares characters using their canonical representation, corresponding with the way comparing strings for equality works. + +When matching with *Unicode scalar semantics*, metacharacters and character classes match a single Unicode scalar value, even if that scalar comprises part of a grapheme cluster. Canonical representations are _not_ used, corresponding with the way comparison would work when using a string's `UnicodeScalarView`. + +These specific levels of matching, and the options to switch between them, are unique to Swift, but not unprecedented in other regular expression engines. Several engines, including Perl, Java, and ICU-based engines like `NSRegularExpression`, support the `\X` metacharacter for matching a grapheme cluster within otherwise Unicode scalar semantic matching. Rust has a related concept in its [`regex::bytes` type][regexbytes], which matches over arbitrary bytes by default but allows switching into Unicode mode for segments of the regular expression. + +These semantic levels lead to different results when working with strings that have characters made up of multiple Unicode scalar values, such as Emoji or decomposed characters. In the following example, `queRegex` matches any 3-character string that begins with `"q"`. + +```swift +let composed = "qué" +let decomposed = "que\u{301}" + +let queRegex = /^q..$/ + +print(composed.contains(queRegex)) +// Prints "true" +print(decomposed.contains(queRegex)) +// Prints "true" +``` + +When using Unicode scalar semantics, however, the regex only matches the composed version of the string, because each `.` matches a single Unicode scalar value. + +```swift +let queRegexScalar = queRegex.matchingSemantics(.unicodeScalar) +print(composed.contains(queRegexScalar)) +// Prints "true" +print(decomposed.contains(queRegexScalar)) +// Prints "false" +``` + +The index boundaries of the overall match and capture groups are affected by the matching semantic level. With grapheme cluster semantics, the start and end index of the overall match and each capture is `Character`-aligned. Matching with Unicode scalar semantics, on the other hand, can yield string indices that aren't aligned to character boundaries. Take care when using indices that aren't aligned with grapheme cluster boundaries, as they may have to be rounded to a boundary if used in a `String` instance. + +```swift +let family = "👨‍👨‍👧‍👦 is a family" + +// Grapheme-cluster mode: Yields a character +let firstCharacter = /^./ +let characterMatch = family.firstMatch(of: firstCharacter)!.output +print(characterMatch) +// Prints "👨‍👨‍👧‍👦" + +// Unicode-scalar mode: Yields only part of a character +let firstUnicodeScalar = /^./.matchingSemantics(.unicodeScalar) +let unicodeScalarMatch = family.firstMatch(of: firstUnicodeScalar)!.output +print(unicodeScalarMatch) +// Prints "👨" + +// The end of `unicodeScalarMatch` is not aligned on a character boundary +print(unicodeScalarMatch.endIndex == family.index(after: family.startIndex)) +// Prints "false" +``` + +When there is a boundary between Unicode scalar semantic and grapheme scalar semantic matching in the middle of a regex, an implicit grapheme cluster boundary assertion is added at the start of the grapheme scalar semantic section. That is, the two regexes in the following example are equivalent; each matches a single "word" scalar, followed by a combining mark scalar, followed by one or more grapheme clusters. + +```swift +let explicit = Regex { + Regex { + CharacterClass.word + CharacterClass.generalCategory(.combiningMark) + }.matchingSemantics(.unicodeScalar) + Anchor.graphemeClusterBoundary // explicit grapheme cluster boundary + OneOrMore(.any) +} + +let implicit = Regex { + Regex { + CharacterClass.word + CharacterClass.generalCategory(.combiningMark) + }.matchingSemantics(.unicodeScalar) + OneOrMore(.any) +} + +try implicit.wholeMatch(in: "e\u{301} abc") // match +try implicit.wholeMatch(in: "e\u{301}\u{327} abc") // no match +``` + +The second call to `wholeMatch(in:)` fails because at the point the matching engine exits the inner regex, the matching position is still in the middle of the `"e\u{301}\u{327}` character. This implicit grapheme cluster boundary assertion maintains the guarantee that capture groups over grapheme cluster semantic sections will have valid character-aligned indices. + +If a regex starts or ends with a Unicode scalar semantic section, there is no assertion added at the start or end of the pattern. Consider the following regex, which has Unicode scalars for the entire pattern except for a section in the middle that matches a purple heart emoji. When applied to a string with a multi-scalar character before or after the `"💜"`, the resulting match includes a partial character at its beginning and end. + +```swift +let regex = Regex { + CharacterClass.any + Regex { // <-- Implicit grapheme cluster boundary assertion, as above + CharacterClass.binaryProperty(\.isEmoji) + }.matchingSemantics(.graphemeCluster) + CharacterClass.any +}.matchingSemantics(.unicodeScalar) + +let borahae = "태형💜아미" // Note: These hangeul characters are decomposed +if let match = borahae.firstMatch(of: regex) { + print(match.0) +} +// Prints "ᆼ💜ᄋ" +``` + +Boundaries from a grapheme cluster section into a Unicode scalar also imply a grapheme cluster boundary, but in this case no assertion is needed. This boundary is an emergent property of the fact that under grapheme cluster semantics, matching always happens one character at a time. + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression that matches with the specified semantic + /// level. + public func matchingSemantics(_ semanticLevel: RegexSemanticLevel) -> Regex +} + +public struct RegexSemanticLevel: Hashable { + /// Match at the default semantic level of a string, where each matched + /// element is a `Character`. + public static var graphemeCluster: RegexSemanticLevel + + /// Match at the semantic level of a string's `UnicodeScalarView`, where each + /// matched element is a `UnicodeScalar` value. + public static var unicodeScalar: RegexSemanticLevel +} +``` + +#### Default repetition behavior + +The `defaultRepetitionBehavior(_:)` method lets you set the default behavior for all quantifiers that don't explicitly provide their own behavior. For example, you can make all quantifiers behave possessively, eliminating any quantification-caused backtracking. This option applies both to quanitifiers in regex syntax that don't include an additional `?` or `+` (indicating reluctant or possessive quantification, respectively) and quantifiers in `RegexBuilder` syntax without an explicit behavior parameter. + +In the following example, both regexes use possessive quantification: + +```swift +let regex1 = /[0-9a-f]+\s*$/.defaultRepetitionBehavior(.possessive) + +let regex2 = Regex { + OneOrMore { + CharacterClass.anyOf( + "0"..."9", + "a"..."f" + ) + } + ZeroOrMore(.whitespace) + Anchor.endOfInput +}.defaultRepetitionBehavior(.possessive) +``` + +This option is related to, but independent from, the regex syntax option `(?U)`. See below for more about that regex-syntax-only option. + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression where quantifiers use the specified behavior + /// by default. + /// + /// You can call this method to change the default repetition behavior for + /// quantifier operators in regex syntax and `RegexBuilder` quantifier + /// methods. For example, in the following example, both regexes use + /// possessive quantification when matching a quotation surround by `"` + /// quote marks: + /// + /// let regex1 = /"[^"]*"/.defaultRepetitionBehavior(.possessive) + /// + /// let quoteMark = "\"" + /// let regex2 = Regex { + /// quoteMark + /// ZeroOrMore(.noneOf(quoteMark)) + /// quoteMark + /// }.defaultRepetitionBehavior(.possessive) + /// + /// This setting only changes the default behavior of quantifiers, and does + /// not affect regex syntax operators with an explicit behavior indicator, + /// such as `*?` or `++`. Likewise, calls to quantifier methods such as + /// `OneOrMore` always use the explicit `behavior`, when given. + /// + /// - Parameter behavior: The default behavior to use for quantifiers. + public func defaultRepetitionBehavior(_ behavior: RegexRepetitionBehavior) -> Regex +} + +public struct RegexRepetitionBehavior { + /// Match as much of the input string as possible, backtracking when + /// necessary. + public static var eager: RegexRepetitionBehavior { get } + + /// Match as little of the input string as possible, expanding the matched + /// region as necessary to complete a match. + public static var reluctant: RegexRepetitionBehavior { get } + + /// Match as much of the input string as possible, performing no backtracking. + public static var possessive: RegexRepetitionBehavior { get } +} +``` + +As described in the [Regex Builder proposal][regexbuilder], `RegexBuilder` quantifier APIs include a `nil`-defaulted optional `behavior` parameter. When you pass `nil`, the quantifier uses the default behavior as set by this option. If an explicit behavior is passed, that behavior is used regardless of the default. + +```swift +// Example `OneOrMore` initializer +extension OneOrMore { + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0), Component.Output == (W, C0) +} +``` + +#### Single line mode (`.` matches newlines) + +The "any" metacharacter (`.`) matches any character in a string *except* newlines by default. With the `s` option enabled, `.` matches any character including newlines. + +```swift +let str = """ + <> + """ + +str.firstMatch(of: /<<.+>>/) +// nil +str.firstMatch(of: /<<.+>>/.dotMatchesNewLines()) +// "This string\nuses double-angle-brackets\nto group text." +``` + +This option applies only to `.` used in regex syntax and does _not_ affect the behavior of `CharacterClass.any`, which always matches any character or Unicode scalar. To get the default `.` behavior when using `RegexBuilder` syntax, use `CharacterClass.anyNonNewline`. + +**Regex syntax:** `(?s)...` or `(?s...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression where the start and end of input + /// anchors (`^` and `$`) also match against the start and end of a line. + public func dotMatchesNewlines(_ dotMatchesNewlines: Bool = true) -> Regex +} +``` + +#### Multiline mode + +By default, the start and end anchors (`^` and `$`) match only the beginning and end of a string. With the `m` or the option, they also match the beginning and end of each line. + +```swift +let str = """ + abc + def + ghi + """ + +str.firstMatch(of: /^abc/) +// "abc" +str.firstMatch(of: /^abc$/) +// nil +str.firstMatch(of: /^abc$/.anchorsMatchLineEndings()) +// "abc" + +str.firstMatch(of: /^def/) +// nil +str.firstMatch(of: /^def$/.anchorsMatchLineEndings()) +// "def" +``` + +This option applies only to anchors used in regex syntax. The anchors defined in `RegexBuilder` are specific about matching at the start/end of the input or the line, and therefore are not affected by this option. + +```swift +str.firstMatch(of: Regex { Anchor.startOfInput ; "def" }) // nil +str.firstMatch(of: Regex { Anchor.startOfLine ; "def" }) // "def" +``` + +**Regex syntax:** `(?m)...` or `(?m...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression where the start and end of input + /// anchors (`^` and `$`) also match against the start and end of a line. + public func anchorsMatchLineEndings(_ matchLineEndings: Bool = true) -> Regex +} +``` + +#### Eager/reluctant toggle + +Regex quantifiers (`+`, `*`, and `?`) match eagerly by default when they repeat, such that they match the longest possible substring. Appending `?` to a quantifier makes it reluctant, instead, so that it matches the shortest possible substring. + +```swift +let str = "A value." + +// By default, the '+' quantifier is eager, and consumes as much as possible. +str.firstMatch(of: /<.+>/) // "A value." + +// Adding '?' makes the '+' quantifier reluctant, so that it consumes as little as possible. +str.firstMatch(of: /<.+?>/) // "" +``` + +The `U` option toggles the "eagerness" of quantifiers, so that quantifiers are reluctant by default, and only become eager when `?` is added to the quantifier. This change only applies within regex syntax. See the `defaultRepetitionBehavior(_:)` method, described above, for broader control over repetition behavior, including setting the default for `RegexBuilder` syntax. + +```swift +// '(?U)' toggles the eagerness of quantifiers: +str.firstMatch(of: /(?U)<.+>/) // "" +str.firstMatch(of: /(?U)<.+?>/) // "A value." +``` + +**Regex syntax:** `(?U)...` or `(?U...)` + + +--- + +### Character Classes + +We propose the following definitions for regex character classes, along with a `CharacterClass` type as part of the `RegexBuilder` module, to encapsulate and simplify character class usage within builder-style regexes. + +The two regexes defined in this example will match the same inputs, looking for one or more word characters followed by up to three digits, optionally separated by a space: + +```swift +let regex1 = /\w+\s?\d{,3}/ +let regex2 = Regex { + OneOrMore(.word) + Optionally(.whitespace) + Repeat(.digit, ...3) +} +``` + +You can build custom character classes by combining regex-defined classes with individual characters or ranges, or by performing common set operations such as subtracting or negating a character class. + + +#### “Any” + +The simplest character class, representing **any character**, is written as `.` and is sometimes referred to as the "dot" metacharacter. This class always matches a single `Character` or Unicode scalar value, depending on the matching semantic level. This class excludes newlines, unless "single line mode" is enabled (see section above). + +When using the `CharacterClass` type in a `RegexBuilder`-defined regex, the `.any` and `.anyNonNewline` provide separate APIs for the two behaviors of `.`, and are therefore unaffected by the current "single line mode" setting. + +```swift +"Cafe\u{301}".contains(/C.../) +// true +``` + +For this example, using Unicode scalar semantics, a dot matches only a single Unicode scalar value, so the combining marks don't get grouped with the commas before them: + +```swift +let data = "\u{300},\u{301},\u{302},\u{303},..." +for match in data.matches(of: /(.),/.matchingSemantics(.unicodeScalar)) { + print(match.1) +} +// Prints: +// ̀ +// ́ +// ̂ +// ... +``` + +#### Any grapheme cluster + +`Regex` also provides a way to match a single grapheme cluster, regardless of the current semantic level. The **any grapheme cluster** character class is written as `\X` or `CharacterClass.anyGraphemeCluster`, and matches from the current location up to the next grapheme cluster boundary. This includes matching newlines, regardless of any option settings. This metacharacter is equivalent to the regex syntax `(?Xs:.)`. + + +#### Digits + +The **decimal digit** character class is matched by `\d` or `CharacterClass.digit`. Both regexes in this example match one or more decimal digits followed by a colon: + +```swift +let regex1 = /\d+:/ +let regex2 = Regex { + OneOrMore(.digit) + ":" +} +``` + +_Unicode scalar semantics:_ Matches a Unicode scalar that has a `numericType` property equal to `.decimal`. This includes the digits from the ASCII range, from the _Halfwidth and Fullwidth Forms_ Unicode block, as well as digits in some scripts, like `DEVANAGARI DIGIT NINE` (U+096F). This corresponds to the general category `Decimal_Number`. + +_Grapheme cluster semantics:_ Matches a character made up of a single Unicode scalar that fits the decimal digit criteria above. + + +To invert the decimal digit character class, use `\D` or `CharacterClass.digit.inverted`. + + +The **hexadecimal digit** character class is matched by `CharacterClass.hexDigit`. + +_Unicode scalar semantics:_ Matches a decimal digit, as described above, or an uppercase or small `A` through `F` from the _Halfwidth and Fullwidth Forms_ Unicode block. Note that this is a broader class than described by the `UnicodeScalar.properties.isHexDigit` property, as that property only include ASCII and fullwidth decimal digits. + +_Grapheme cluster semantics:_ Matches a character made up of a single Unicode scalar that fits the hex digit criteria above. + +To invert the hexadecimal digit character class, use `CharacterClass.hexDigit.inverted`. + +*
Rationale* + +Unicode's recommended definition for `\d` is its [numeric type][numerictype] of "Decimal" in contrast to "Digit". It is specifically restricted to sets of ascending contiguously-encoded scalars in a decimal radix positional numeral system. Thus, it excludes "digits" such as superscript numerals from its [definition][derivednumeric] and is a proper subset of `Character.isWholeNumber`. + +We interpret Unicode's definition of the set of scalars, especially its requirement that scalars be encoded in ascending chains, to imply that this class is restricted to scalars which meaningfully encode base-10 digits. Thus, we choose to make the grapheme cluster interpretation *restrictive*. + +
+ + +#### "Word" characters + +The **word** character class is matched by `\w` or `CharacterClass.word`. This character class and its name are essentially terms of art within regexes, and represents part of a notional "word". Note that, by default, this is distinct from the algorithm for identifying word boundaries. + +_Unicode scalar semantics:_ Matches a Unicode scalar that has one of the Unicode properties `Alphabetic`, `Digit`, or `Join_Control`, or is in the general category `Mark` or `Connector_Punctuation`. + +_Grapheme cluster semantics:_ Matches a character that begins with a Unicode scalar value that fits the criteria above. + +To invert the word character class, use `\W` or `CharacterClass.word.inverted`. + +*
Rationale* + +Word characters include more than letters, and we went with Unicode's recommended scalar semantics. Following the Unicode recommendation that nonspacing marks remain with their base characters, we extend to grapheme clusters similarly to `Character.isLetter`. That is, combining scalars do not change the word-character-ness of the grapheme cluster. + +
+ + +#### Whitespace and newlines + +The **whitespace** character class is matched by `\s` and `CharacterClass.whitespace`. + +_Unicode scalar semantics:_ Matches a Unicode scalar that has the Unicode properties `Whitespace`, including a space, a horizontal tab (U+0009), `LINE FEED (LF)` (U+000A), `LINE TABULATION` (U+000B), `FORM FEED (FF)` (U+000C), `CARRIAGE RETURN (CR)` (U+000D), and `NEWLINE (NEL)` (U+0085). Note that under Unicode scalar semantics, `\s` only matches the first scalar in a `CR`+`LF` pair. + +_Grapheme cluster semantics:_ Matches a character that begins with a `Whitespace` Unicode scalar value. This includes matching a `CR`+`LF` pair. + +The **horizontal whitespace** character class is matched by `\h` and `CharacterClass.horizontalWhitespace`. + +_Unicode scalar semantics:_ Matches a Unicode scalar that has the Unicode general category `Zs`/`Space_Separator` as well as a horizontal tab (U+0009). + +_Grapheme cluster semantics:_ Matches a character that begins with a Unicode scalar value that fits the criteria above. + +The **vertical whitespace** character class is matched by `\v` and `CharacterClass.verticalWhitespace`. Additionally, `\R` and `CharacterClass.newline` provide a way to include the `CR`+`LF` pair, even when matching with Unicode scalar semantics. + +_Unicode scalar semantics:_ Matches a Unicode scalar that has the Unicode general category `Zl`/`Line_Separator` or `Zp`/`Paragraph_Separator`, as well as any of the following control characters: `LINE FEED (LF)` (U+000A), `LINE TABULATION` (U+000B), `FORM FEED (FF)` (U+000C), `CARRIAGE RETURN (CR)` (U+000D), and `NEWLINE (NEL)` (U+0085). Only when specified as `\R` or `CharacterClass.newline` does this match the whole `CR`+`LF` pair. + +_Grapheme cluster semantics:_ Matches a character that begins with a Unicode scalar value that fits the criteria above. + +To invert these character classes, use `\S`, `\H`, and `\V`, respectively, or the `inverted` property on a `CharacterClass` instance. + +
Rationale + +Note that "whitespace" is a term-of-art and is not correlated with visibility, which is a completely separate concept. + +We use Unicode's recommended scalar semantics for horizontal and vertical whitespace, extended to grapheme clusters as in the existing `Character.isWhitespace` property. + +
+ + +#### Unicode properties + +Character classes that match **Unicode properties** are written as `\p{PROPERTY}` or `\p{PROPERTY=VALUE}`, as described in the [Run-time Regex Construction proposal][internals-properties]. + +While most Unicode properties are only defined at the scalar level, some are defined to match an extended grapheme cluster. For example, `\p{RGI_Emoji_Flag_Sequence}` will match any flag emoji character, which are composed of two Unicode scalar values. Such property classes will match multiple scalars, even when matching with Unicode scalar semantics. + +Unicode property matching is extended to `Character`s with a goal of consistency with other regex character classes, and as dictated by prior standard library additions to the `Character` type. For example, for `\p{Decimal}` and `\p{Hex_Digit}`, only single-scalar `Character`s can match, for the reasons described in that section, above. For other Unicode property classes, like `\p{Whitespace}`, the character matches when the first scalar has that Unicode property. Open the following disclosure area to see the full list of properties, along with the rubric for extending them to grapheme clusters. + +
Unicode properties + +We can choose to extend a Unicode property to a grapheme cluster in one of several ways: + +- *single-scalar*: Only a character that comprises a single Unicode scalar value can match +- *first-scalar*: If the first Unicode scalar in a character matches, then the character matches +- *any-scalar*: If any Unicode scalar in a character matches, then the character matches +- *all-scalars*: A character matches if and only if all its Unicode scalar members match + +With a few guidelines, we can make headway on classifying Unicode properties: + +- Numeric-related properties, like `Numeric_Value`, should only apply to single-scalar characters, for the reasons described in the "Digit" character class section, above. +- Any other properties that directly or approximately correspond to regex or POSIX character classes should use the first-scalar rule. This corresponds with the way `Character.isWhitespace` is implemented and generally matches the perceived categorization of characters. +- Properties that resolve to a unique Unicode scalar, such as `Name`, should only apply to single-scalar characters. +- Properties that govern the way Unicode scalars combine into characters, such as `Canonical_Combining_Class`, or are otherwise only relevant when examining specific Unicode data, such as `Age`, should only apply to single-scalar characters. +- Properties that can naturally apply to a sequence of Unicode scalars, such as `Lowercase_Mapping`, should use an all-scalars approach. This corresponds with the way `Character.isLowercased` and other casing properties are implemented. + +In many cases, properties with a *single-scalar* treatment won't match any characters at all, and will only be useful when matching with Unicode scalar semantics. For example, `/\p{Emoji_Modifier}/` matches the five Fitzpatrick skin tone modifier Unicode scalar values that affect the appearance of emoji within a grapheme cluster. When matching with grapheme cluster semantics, no match for the pattern will be found. Using Unicode scalar semantics, however, you can search for all characters that include such a modifier: + +``` +let regex = /(?u)\y.+?\p{Emoji_Modifier}.+?\y/ +for ch in "👩🏾‍🚀🚀 👨🏻‍🎤🎸 🧑🏻‍💻📲".matches(of: regex) { + print(ch) +} +// Prints: +// 👩🏾‍🚀 +// 👨🏻‍🎤 +// 🧑🏻‍💻 +``` + +The table below shows our best effort at choosing the right manner of extending. + +| Property | Extension | Notes | +|-------------------------------------|-------------------------------|-----------------------------------| +| **General** | | | +| `Name` | single-scalar | | +| `Name_Alias` | single-scalar | | +| `Age` | single-scalar | | +| `General_Category` | first-scalar | Numeric categories: single-scalar | +| `Script` | first-scalar | | +| `White_Space` | first-scalar | Existing `Character` API | +| `Alphabetic` | first-scalar | | +| `Noncharacter_Code_Point` | single-scalar | | +| `Default_Ignorable_Code_Point` | single-scalar | | +| `Deprecated` | single-scalar | | +| `Logical_Order_Exception` | single-scalar | | +| `Variation_Selector` | single-scalar | | +|
**Numeric** | | | +| `Numeric_Value` | single-scalar | | +| `Numeric_Type` | single-scalar | | +| `Hex_Digit` | single-scalar | Existing `Character` API | +| `ASCII_Hex_Digit` | single-scalar | | +|
**Identifiers** | | | +| `ID_Start` | single-scalar | | +| `ID_Continue` | single-scalar | | +| `XID_Start` | single-scalar | | +| `XID_Continue` | single-scalar | | +| `Pattern_Syntax` | single-scalar | | +| `Pattern_White_Space` | single-scalar | | +|
**CJK** | | | +| `Ideographic` | first-scalar | | +| `Unified_Ideograph` | first-scalar | | +| `Radical` | first-scalar | | +| `IDS_Binary_Operator` | single-scalar | | +| `IDS_Trinary_Operator` | single-scalar | | +|
**Case** | | | +| `Lowercase` | first-scalar | | +| `Uppercase` | first-scalar | | +| `Lowercase_Mapping` | all-scalars | | +| `Titlecase_Mapping` | all-scalars | | +| `Uppercase_Mapping` | all-scalars | | +| `Soft_Dotted` | first-scalar | | +| `Cased` | any-scalar | | +| `Case_Ignorable` | all-scalars | | +| `Changes_When_Lowercased` | all-scalars | | +| `Changes_When_Uppercased` | all-scalars | | +| `Changes_When_Titlecased` | all-scalars | | +| `Changes_When_Casefolded` | all-scalars | | +| `Changes_When_Casemapped` | all-scalars | | +|
**Normalization** | | | +| `Canonical_Combining_Class` | single-scalar | | +| `Full_Composition_Exclusion` | single-scalar | | +| `Changes_When_NFKC_Casefolded` | all-scalars | | +|
**Emoji** | | | +| `Emoji` | first-scalar | | +| `Emoji_Presentation` | any-scalar | | +| `Emoji_Modifier` | single-scalar | | +| `Emoji_Modifier_Base` | single-scalar | | +|
**Shaping and Rendering** | | | +| `Join_Control` | single-scalar | | +|
**Bidirectional** | | | +| `Bidi_Control` | single-scalar | | +| `Bidi_Mirrored` | first-scalar | | +|
**Miscellaneous** | | | +| `Math` | first-scalar | | +| `Quotation_Mark` | first-scalar | | +| `Dash` | first-scalar | | +| `Sentence_Terminal` | first-scalar | | +| `Terminal_Punctuation` | first-scalar | | +| `Diacritic` | single-scalar | | +| `Extender` | single-scalar | | +| `Grapheme_Base` | single-scalar | | +| `Grapheme_Extend` | single-scalar | | + +
+ +To invert a Unicode property character class, use `\P{...}`. + +When using `RegexBuilder` syntax, Unicode property classes are available through the following methods on `CharacterClass`: + +- `static func generalCategory(_: Unicode.GeneralCategory) -> CharacterClass` +- `static func binaryProperty(_: KeyPath, value: Bool = true) -> CharacterClass` +- `static func named(_: String) -> CharacterClass` +- `static func age(_: Unicode.Version) -> CharacterClass` +- `static func numericType(_: Unicode.NumericType) -> CharacterClass` +- `static func numericValue(_: Double) -> CharacterClass` +- `static func lowercaseMapping(_: String) -> CharacterClass` +- `static func uppercaseMapping(_: String) -> CharacterClass` +- `static func titlecaseMapping(_: String) -> CharacterClass` +- `static func canonicalCombiningClass(_: Unicode.CanonicalCombiningClass) -> CharacterClass` + +You can see the full `CharacterClass` API with documentation comments in the **Custom Classes** section, below. + +#### POSIX character classes: `[:NAME:]` or `\p{NAME}` + +**POSIX character classes** represent concepts that we'd like to define at all semantic levels. We propose the following definitions, some of which have been described above. When matching with grapheme cluster semantics, Unicode properties are extended to `Character`s as described in the rationale above, and as shown in the table below. That is, for POSIX class `[:word:]`, any `Character` that starts with a matching scalar is a match, while for `[:digit:]`, a matching `Character` must only comprise a single Unicode scalar value. + +| POSIX class | Unicode property class | Character behavior | ASCII mode value | +|--------------|-----------------------------------|----------------------|-------------------------------| +| `[:lower:]` | `\p{Lowercase}` | starts-with | `[a-z]` | +| `[:upper:]` | `\p{Uppercase}` | starts-with | `[A-Z]` | +| `[:alpha:]` | `\p{Alphabetic}` | starts-with | `[A-Za-z]` | +| `[:alnum:]` | `[\p{Alphabetic}\p{Decimal}]` | starts-with | `[A-Za-z0-9]` | +| `[:word:]` | See \* below | starts-with | `[[:alnum:]_]` | +| `[:digit:]` | `\p{DecimalNumber}` | single-scalar | `[0-9]` | +| `[:xdigit:]` | `\p{Hex_Digit}` | single-scalar | `[0-9A-Fa-f]` | +| `[:punct:]` | `\p{Punctuation}` | starts-with | `[-!"#%&'()*,./:;?@[\\\]{}]` | +| `[:blank:]` | `[\p{Space_Separator}\u{09}]` | starts-with | `[ \t]` | +| `[:space:]` | `\p{Whitespace}` | starts-with | `[ \t\n\r\f\v]` | +| `[:cntrl:]` | `\p{Control}` | starts-with | `[\x00-\x1f\x7f]` | +| `[:graph:]` | See \*\* below | starts-with | `[^ [:cntrl:]]` | +| `[:print:]` | `[[:graph:][:blank:]--[:cntrl:]]` | starts-with | `[[:graph:] ]` | + +\* The Unicode scalar property definition for `[:word:]` is `[\p{Alphanumeric}\p{Mark}\p{Join_Control}\p{Connector_Punctuation}]`. +\*\* The Unicode scalar property definition for `[:cntrl:]` is `[^\p{Space}\p{Control}\p{Surrogate}\p{Unassigned}]`. + +#### Custom classes + +Custom classes function as the set union of their individual components, whether those parts are individual characters, individual Unicode scalar values, ranges, Unicode property classes or POSIX classes, or other custom classes. + +- Individual characters and scalars will be tested using the same behavior as if they were listed in an alternation. That is, a custom character class like `[abc]` is equivalent to `(a|b|c)` under the same options and modes. +- Metacharacters that represent built-in character classes keep their same function inside custom character classes. For example, in `[abc\d]+`, the `\d` matches any digit, so the regex matches the entirety of the string `"0a1b2c3"`, and `[\t\R]` matches a tab or any newline character or newline sequence. +- Metacharacters that represent zero-width assertions have their literal meaning in custom character classes, if one exists. For example, `[\b^]` matches either the BEL control character or a literal carat (`^`), while `\B` is an invalid member of a custom character class. + +Ranges in a custom character class require special consideration to avoid unexpected or dangerous results. Using simple lexicographical ordering for comparison is unintuitive when working with multi-scalar characters. For example, +the custom character class `[0-9]` is intended to match only the ten ASCII digits, but because of lexicographical ordering, complex characters like `"3̠̄"` and `"5️⃣"` would fall into that range. Ranges in custom character classes therefore having the following requirements: + +- Range endpoints must be single Unicode scalar values. When parsing a regex, endpoints will be converted to their canonical composed form, so that characters that have a multi-Unicode scalar form in source but a single-scalar canonical representation will still be permitted. +- When matching with grapheme cluster semantics, only single-scalar characters will match a range. The same conversion to canonical composed form will be used to support the expectation of matching with canonical equivalence. + +```swift +let allDigits = /^[0-9]+$/ +"1230".contains(allDigits) // true +"123̠̄0".contains(allDigits) // false +"5️⃣".contains(allDigits) // false + +let cafeExtended = /Caf[à-ÿ]/ +"Café".contains(cafeExtended) // true +"Cafe\u{301}".contains(cafeExtended) // true +``` + +Inside regexes, custom classes are enclosed in square brackets `[...]`, and can be nested or combined using set operators like `&&`. For more detail, see the [Run-time Regex Construction proposal][internals-charclass]. + +With `RegexBuilder`'s `CharacterClass` type, you can use built-in character classes with ranges and groups of characters. For example, to parse a valid octodecimal number, you could define a custom character class that combines `.digit` with a range of characters. + +```swift +let octoDecimalRegex: Regex<(Substring, Int?)> = Regex { + let charClass = CharacterClass(.digit, "a"..."h").ignoresCase() + Capture { + OneOrMore(charClass) + } transform: { Int($0, radix: 18) } +} +``` + +The full `CharacterClass` API is as follows: + +```swift +/// A class of characters that match in a regex. +/// +/// A character class can represent individual characters, a group of +/// characters, the set of character that match some set of criteria, or +/// a set algebraic combination of all of the above. +public struct CharacterClass: RegexComponent { + public var regex: Regex { get } + + /// A character class that matches any character that does not match this + /// character class. + public var inverted: CharacterClass { get } +} + +// MARK: Built-in character classes + +extension RegexComponent where Self == CharacterClass { + /// A character class that matches any element. + /// + /// This character class is unaffected by the `dotMatchesNewlines()` method. + /// To match any character that isn't a newline, see + /// ``CharacterClass.anyNonNewline``. + /// + /// This character class is equivalent to the regex syntax "dot" + /// metacharacter in single-line mode: `(?s:.)`. + public static var any: CharacterClass { get } + + /// A character class that matches any element that isn't a newline. + /// + /// This character class is unaffected by the `dotMatchesNewlines()` method. + /// To match any character, including newlines, see ``CharacterClass.any``. + /// + /// This character class is equivalent to the regex syntax "dot" + /// metacharacter with single-line mode disabled: `(?-s:.)`. + public static var anyNonNewline: CharacterClass { get } + + /// A character class that matches any single `Character`, or extended + /// grapheme cluster, regardless of the current semantic level. + /// + /// This character class is equivalent to `\X` in regex syntax. + public static var anyGraphemeCluster: CharacterClass { get } + + /// A character class that matches any digit. + /// + /// This character class is equivalent to `\d` in regex syntax. + public static var digit: CharacterClass { get } + + /// A character class that matches any hexadecimal digit. + public static var hexDigit: CharacterClass { get } + + /// A character class that matches any element that is a "word character". + /// + /// This character class is equivalent to `\w` in regex syntax. + public static var word: CharacterClass { get } + + /// A character class that matches any element that is classified as + /// whitespace. + /// + /// This character class is equivalent to `\s` in regex syntax. + public static var whitespace: CharacterClass { get } + + /// A character class that matches any element that is classified as + /// horizontal whitespace. + /// + /// This character class is equivalent to `\h` in regex syntax. + public static var horizontalWhitespace: CharacterClass { get } + + /// A character class that matches any element that is classified as + /// vertical whitespace. + /// + /// This character class is equivalent to `\v` in regex syntax. + public static var verticalWhitespace: CharacterClass { get } + + /// A character class that matches any newline sequence. + /// + /// This character class is equivalent to `\R` or `\n` in regex syntax. + public static var newlineSequence: CharacterClass { get } +} + +// MARK: anyOf(_:) / noneOf(_:) + +extension RegexComponent where Self == CharacterClass { + /// Returns a character class that matches any character in the given string + /// or sequence. + /// + /// Calling this method with a group of characters is equivalent to listing + /// those characters in a custom character class in regex syntax. For example, + /// the two regexes in this example are equivalent: + /// + /// let regex1 = /[abcd]+/ + /// let regex2 = OneOrMore(.anyOf("abcd")) + public static func anyOf(_ s: S) -> CharacterClass + where S.Element == Character + + /// Returns a character class that matches any Unicode scalar in the given + /// sequence. + /// + /// Calling this method with a group of Unicode scalars is equivalent to + /// listing them in a custom character class in regex syntax. + public static func anyOf(_ s: S) -> CharacterClass + where S.Element == UnicodeScalar + + /// Returns a character class that matches none of the characters in the given + /// string or sequence. + /// + /// Calling this method with a group of characters is equivalent to listing + /// those characters in a negated custom character class in regex syntax. For + /// example, the two regexes in this example are equivalent: + /// + /// let regex1 = /[^abcd]+/ + /// let regex2 = OneOrMore(.noneOf("abcd")) + public static func noneOf(_ s: S) -> CharacterClass + where S.Element == Character + + /// Returns a character class that matches none of the Unicode scalars in the + /// given sequence. + /// + /// Calling this method with a group of Unicode scalars is equivalent to + /// listing them in a negated custom character class in regex syntax. + public static func noneOf(_ s: S) -> CharacterClass + where S.Element == UnicodeScalar +} + +// MARK: Unicode properties + +extension CharacterClass { + /// Returns a character class that matches any element with the given Unicode + /// general category. + /// + /// For example, when passed `.uppercaseLetter`, this method is equivalent to + /// `/\p{Uppercase_Letter}/` or `/\p{Lu}/`. + public static func generalCategory(_ category: Unicode.GeneralCategory) -> CharacterClass + + /// Returns a character class that matches any element with the given Unicode + /// binary property. + /// + /// For example, when passed `\.isAlphabetic`, this method is equivalent to + /// `/\p{Alphabetic}/` or `/\p{Is_Alphabetic=true}/`. + public static func binaryProperty( + _ property: KeyPath, + value: Bool = true + ) -> CharacterClass + + /// Returns a character class that matches any element with the given Unicode + /// name. + /// + /// This method is equivalent to `/\p{Name=name}/`. + public static func name(_ name: String) -> CharacterClass + + /// Returns a character class that matches any element that was included in + /// the specified Unicode version. + /// + /// This method is equivalent to `/\p{Age=version}/`. + public static func age(_ version: Unicode.Version) -> CharacterClass + + /// Returns a character class that matches any element with the given Unicode + /// numeric type. + /// + /// This method is equivalent to `/\p{Numeric_Type=type}/`. + public static func numericType(_ type: Unicode.NumericType) -> CharacterClass + + /// Returns a character class that matches any element with the given numeric + /// value. + /// + /// This method is equivalent to `/\p{Numeric_Value=value}/`. + public static func numericValue(_ value: Double) -> CharacterClass + + /// Returns a character class that matches any element with the given Unicode + /// canonical combining class. + /// + /// This method is equivalent to + /// `/\p{Canonical_Combining_Class=combiningClass}/`. + public static func canonicalCombiningClass( + _ combiningClass: Unicode.CanonicalCombiningClass + ) -> CharacterClass + + /// Returns a character class that matches any element with the given + /// lowercase mapping. + /// + /// This method is equivalent to `/\p{Lowercase_Mapping=value}/`. + public static func lowercaseMapping(_ value: String) -> CharacterClass + + /// Returns a character class that matches any element with the given + /// uppercase mapping. + /// + /// This method is equivalent to `/\p{Uppercase_Mapping=value}/`. + public static func uppercaseMapping(_ value: String) -> CharacterClass + + /// Returns a character class that matches any element with the given + /// titlecase mapping. + /// + /// This method is equivalent to `/\p{Titlecase_Mapping=value}/`. + public static func titlecaseMapping(_ value: String) -> CharacterClass +} + +// MARK: Set algebra methods + +extension CharacterClass { + /// Returns a character class that combines the characters classes in the + /// given sequence or collection via union. + public init(_ characterClasses: some Sequence) + + /// Creates a character class that combines the given classes in a union. + public init(_ first: CharacterClass, _ rest: CharacterClass...) + + /// Returns a character class from the union of this class and the given class. + public func union(_ other: CharacterClass) -> CharacterClass + + /// Returns a character class from the intersection of this class and the given class. + public func intersection(_ other: CharacterClass) -> CharacterClass + + /// Returns a character class by subtracting the given class from this class. + public func subtracting(_ other: CharacterClass) -> CharacterClass + + /// Returns a character class matching elements in one or the other, but not both, + /// of this class and the given class. + public func symmetricDifference(_ other: CharacterClass) -> CharacterClass +} + +// MARK: Range syntax + +public func ...(lhs: Character, rhs: Character) -> CharacterClass + +@_disfavoredOverload +public func ...(lhs: UnicodeScalar, rhs: UnicodeScalar) -> CharacterClass +``` + +## Source compatibility + +Everything in this proposal is additive, and has no compatibility effect on existing source code. + +## Effect on ABI stability + +Everything in this proposal is additive, and has no effect on existing stable ABI. + +## Effect on API resilience + +N/A + +## Future directions + +### Expanded options and modifiers + +The initial version of `Regex` includes only the options described above. Filling out the remainder of options described in the [Run-time Regex Construction proposal][internals] could be completed as future work, as well as additional improvements, such as adding an option that makes a regex match only at the start of a string. + +### Extensions to Character and Unicode Scalar APIs + +An earlier version of this pitch described adding standard library APIs to `Character` and `UnicodeScalar` for each of the supported character classes, as well as convenient static members for control characters. In addition, regex literals support Unicode property features that don’t currently exist in the standard library, such as a scalar’s script or extended category, or creating a scalar by its Unicode name instead of its scalar value. These kinds of additions have value outside of just their relationship to the `Regex` additions, so they can be pitched and considered in a future proposal. + +### Byte semantic mode + +A future `Regex` version could support a byte-level semantic mode in addition to grapheme cluster and Unicode scalar semantics. Byte-level semantics would allow matching individual bytes, potentially providing the capability of parsing string and non-string data together. + +### More general `CharacterSet` replacement + +Foundation's `CharacterSet` type is in some ways similar to the `CharacterClass` type defined in this proposal. `CharacterSet` is primarily a set type that is defined over Unicode scalars, and can therefore sometimes be awkward to use in conjunction with Swift `String`s. The proposed `CharacterClass` type is a `RegexBuilder`-specific type, and as such isn't intended to be a full general purpose replacement. Future work could involve expanding upon the `CharacterClass` API or introducing a different type to fill that role. + +## Alternatives considered + +### Operate on String.UnicodeScalarView instead of using semantic modes + +Instead of providing APIs to select whether `Regex` matching is `Character`-based vs. `UnicodeScalar`-based, we could instead provide methods to match against the different views of a string. This different approach has multiple drawbacks: + +* As the scalar level used when matching changes the behavior of individual components of a `Regex`, it’s more appropriate to specify the semantic level at the declaration site than the call site. +* With the proposed options model, you can define a Regex that includes different semantic levels for different portions of the match, which would be impossible with a call site-based approach. + +### Binary word boundary option method + +A prior version of this proposal used a binary method for setting the word boundary algorithm, called `usingSimpleWordBoundaries()`. A method taking a `RegexWordBoundaryKind` instance is included in the proposal instead, to leave room for implementing other word boundary algorithms in the future. + +### More "Swifty" default option settings + +Swift's `Regex` includes some default behaviors that don't match other regex engines — in particular, matching characters with `.` and using Unicode's default word boundary algorithm. For other option-based behaviors, `Regex` adheres to the general standard set by other regular expression engines, like having `.` not match newlines and `^` and `$` only match the start and end of the input instead of the beginning and end of each line. This is to ease the process of bringing existing regular expressions and existing knowledge into Swift. + +Instead, we could use this opportunity to choose default options that are more ergonomic or intuitive, and provide a `compatibilityOptions()` API that reverts back to the typical settings, including matching based on Unicode scalars instead of characters. This method could additionally be a point of documentation for Swift's choices of default behaviors. + +### Include `\O` and `CharacterClass.anyUnicodeScalar` + +An earlier draft of this proposal included a metacharacter and `CharacterClass` API for matching an individual Unicode scalar value, regardless of the current matching level, as a counterpart to `\X`/`.anyGraphemeCluster`. The behavior of this character class, particularly when matching with grapheme cluster semantics, is still unclear at this time, however. For example, when matching the expression `\O*`, does the implicit grapheme boundary assertion apply between the `\O` and the quantification operator, or should we treat the two as a single unit and apply the assertion after the `*`? + +At the present time, we prefer to allow authors to write regexes that explicitly shift into and out of Unicode scalar mode, where those kinds of decisions are handled by the explicit scope of the setting. If common patterns emerge that indicate some version of `\O` would be useful, we can add it in the future. + +## Future Work + +### Additional protocol to limit option methods + +The option-setting methods, like `ignoresCase()`, are implemented as extensions of the `Regex` type instead of on the `RegexComponent` protocol. This makes sure that nonsensical formulations like `"abc".defaultRepetitionBehavior(.possessive)"` are impossible to write, but is somewhat inconvenient when working with `RegexBuilder` syntax, as you need to add an additional `Regex { ... }` block around a quantifier or other grouping scope that you want to have a particular behavior. + +One possible future addition would be to add another protocol that refines `RegexComponent`, with a name like `RegexCompoundComponent`, representing types that can hold or more other regex components. Types like `OneOrMore`, `CharacterClass`, and `Regex` itself would all conform, and the option-setting methods would move to an extension on that new protocol, permitting more convenient usage where appropriate. + +### API for current options + +As we gather information about how regexes are used and extended, we may find it useful to query an existing regex instance for the set of options that are present globally, or at the start of the regex. Likewise, if `RegexBuilder` gains the ability to use a predicate or other call out to other code, that may require providing the current set of options at the time of execution. + +Such an options type could have a simple read-write, property accessor interface: + +```swift +/// A set of options that affect matching behavior and semantics. +struct RegexOptions { + /// A Boolean value indicating whether casing is ignored while matching. + var ignoresCase: Bool + /// An option set representing any character classes that are matched as ASCII-only. + var asciiOnlyClasses: RegexCharacterClassKind + /// The current matching semantics. + var matchingSemantics: RegexMatchingSemantics + // etc... +} +``` + +### Regex syntax for matching level + +An earlier draft of this proposal included options within the regex syntax that are equivalent to calling the `matchingSemantics(_:)` method: `(?X)` for switching to grapheme cluster more and `(?u)` for switching to Unicode scalar mode. As these are new additions to regex syntax, and their exclusive behavior has yet to be determined, they are not included in the proposed functionality at this time. + +### API for overriding Unicode property mapping + +We could add API in the future to change how individual Unicode scalar properties are extended to characters. One such approach could be to provide a modifier method that takes a key path and a strategy: + +```swift +// Matches only the character "a" +let regex1 = /\p{name=latin lowercase a}/ + +// Matches any character with "a" as its first scalar +let regex1 = /\p{name=latin lowercase a}/.extendUnicodeProperty(\.name, by: .firstScalar)`. +``` + +[repo]: https://github.com/apple/swift-experimental-string-processing/ +[option-scoping]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#matching-options +[internals]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md +[internals-properties]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#character-properties +[internals-charclass]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#custom-character-classes +[level1-word-boundaries]:https://unicode.org/reports/tr18/#Simple_Word_Boundaries +[level2-word-boundaries]:https://unicode.org/reports/tr18/#RL2.3 + +[overview]: https://forums.swift.org/t/declarative-string-processing-overview/52459 +[charprops]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0221-character-properties.md +[regexbuilder]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0351-regex-builder.md +[charpropsrationale]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0221-character-properties.md#detailed-semantics-and-rationale +[canoneq]: https://www.unicode.org/reports/tr15/#Canon_Compat_Equivalence +[graphemes]: https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries +[meaningless]: https://forums.swift.org/t/declarative-string-processing-overview/52459/121 +[scalarprops]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0211-unicode-scalar-properties.md +[ucd]: https://www.unicode.org/reports/tr44/tr44-28.html +[numerictype]: https://www.unicode.org/reports/tr44/#Numeric_Type +[derivednumeric]: https://www.unicode.org/Public/UCD/latest/ucd/extracted/DerivedNumericType.txt + + +[uts18]: https://unicode.org/reports/tr18/ +[proplist]: https://www.unicode.org/Public/UCD/latest/ucd/PropList.txt +[pcre]: https://www.pcre.org/current/doc/html/pcre2pattern.html +[perl]: https://perldoc.perl.org/perlre +[raku]: https://docs.raku.org/language/regexes +[rust]: https://docs.rs/regex/1.5.4/regex/ +[regexbytes]: https://docs.rs/regex/1.5.4/regex/bytes/ +[python]: https://docs.python.org/3/library/re.html +[ruby]: https://ruby-doc.org/core-2.4.0/Regexp.html +[csharp]: https://docs.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-language-quick-reference +[icu]: https://unicode-org.github.io/icu/userguide/strings/regexp.html +[posix]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html +[oniguruma]: https://www.cuminas.jp/sdk/regularExpression.html +[go]: https://pkg.go.dev/regexp/syntax@go1.17.2 +[cplusplus]: https://www.cplusplus.com/reference/regex/ECMAScript/ +[ecmascript]: https://262.ecma-international.org/12.0/#sec-pattern-semantics +[re2]: https://github.com/google/re2/wiki/Syntax +[java]: https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html diff --git a/proposals/0364-retroactive-conformance-warning.md b/proposals/0364-retroactive-conformance-warning.md new file mode 100644 index 0000000000..6300d01b9e --- /dev/null +++ b/proposals/0364-retroactive-conformance-warning.md @@ -0,0 +1,150 @@ +# Warning for Retroactive Conformances of External Types + +* Proposal: [SE-0364](0364-retroactive-conformance-warning.md) +* Author: [Harlan Haskins](https://github.com/harlanhaskins) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.0)** +* Review: ([first pitch](https://forums.swift.org/t/warning-for-retroactive-conformances-if-library-evolution-is-enabled/45321)) + ([second pitch](https://forums.swift.org/t/pitch-warning-for-retroactive-conformances-of-external-types-in-resilient-libraries/56243)) + ([first review](https://forums.swift.org/t/se-0364-warning-for-retroactive-conformances-of-external-types/58922)) + ([second review](https://forums.swift.org/t/second-review-se-0364-warning-for-retroactive-conformances-of-external-types/64615)) + ([acceptance](https://forums.swift.org/t/accepted-se-0364-warning-for-retroactive-conformances-of-external-types/65015)) + ([amendment](https://forums.swift.org/t/amendment-se-0364-allow-same-package-conformances/71877)) + ([acceptance](https://forums.swift.org/t/accepted-amendment-se-0364-allow-same-package-conformances/72880)) + +## Introduction + +Many Swift libraries vend currency protocols, like Equatable, Hashable, Codable, +among others, that unlock worlds of common functionality for types that conform +to them. Sometimes, if a type from another module does not conform to a common +currency protocols, developers will declare a conformance of that type to that +protocol within their module. However, protocol conformances are globally unique +within a process in the Swift runtime, and if multiple modules declare the same +conformance, it can cause major problems for library clients and hinder the +ability to evolve libraries over time. + +## Motivation + +Consider a library that, for one of its core APIs, declares a conformance of +`Date` to `Identifiable`, in order to use it with an API that diffs elements +of a collection by their identity. + +```swift +// Not a great implementation, but I suppose it could be useful. +extension Date: Identifiable { + public var id: TimeInterval { timeIntervalSince1970 } +} +``` + +Now that this client has declared this conformance, if Foundation decides to +add this conformance in a later revision, this client will fail to build. +Before the client removes their conformance and rebuilds, however, their +application will exhibit undefined behavior, as it is indeterminate which +definition of this conformance will "win". Foundation may well have defined +it to use `Date.timeIntervalSinceReferenceDate`, and if the client had persisted +these IDs to a database or some persistent storage beyond the lifetime of the process, +then their dates will have completely different IDs. + +Worse, if this is a library target, this conformance propagates down to every +client that imports the library. This is especially bad for frameworks that +are built with library evolution enabled, as their clients link against +binary frameworks and usually are not aware these conformances don't come from +the actual owning module. + +## Proposed solution + +This proposal adds a warning that explicitly calls out this pattern as +problematic and unsupported. + +```swift +/tmp/retro.swift:3:1: warning: extension declares a conformance of imported type +'Date' to imported protocol 'Identifiable'; this will not behave correctly if +the owners of 'Foundation' introduce this conformance in the future +extension Date: Identifiable { +^ +``` + +If absolutely necessary, clients can silence this warning by adding a new attribute, +`@retroactive`, to the protocol in question. + +The compiler will enforce that there is an explicit `@retroactive` conformance +for each protocol included up the hierarchy. If needed, it will emit a fix-it to +generate extensions for each retroactive conformance in the hierarchy. + +```swift +extension Date: @retroactive Identifiable { + // ... +} +``` + +## Detailed design + +This warning will appear only if all of the following conditions are met, with a few exceptions. + +- The type being extended was declared in a different module from the extension. +- The protocol for which the extension introduces the conformance is declared in a different + module from the extension. + +The following exceptions apply to either the conforming type or the protocol: + +- If it is declared in a Clang module, and the extension in question is declared + in a Swift overlay of that module, this is not considered a retroactive conformance. +- If it is declared or transitively imported in a bridging header or through the + `-import-objc-header` flag, and the type does not belong to any other module, the warning is not + emitted. This could be a retroactive conformance, but since these are added to an implicit module + called `__ObjC`, we have to assume the client takes responsibility for these declaration. +- If it is declared in one module, but uses the `@_originallyDefined(in:)` attribute to + signify that it has moved from a different module, then this will not warn. +- If it is declared in a module that is part of the same package as the conformance, + this is not retroactive. Duplicated same-package conformances will be detected at link or load + time. + +For clarification, the following are still valid, safe, and allowed: +- Conformances of external types to protocols defined within the current module. +- Extensions of external types that do not introduce a conformance. These do not introduce runtime conflicts, since the + module name is mangled into the symbol. + +The `@retroactive` attribute may only be used in extensions, and only when used +to introduce a conformance that requires its existence. It will be an error to +use `@retroactive` outside of the declaration of a retroactive conformance. + +## Source compatibility + +`@retroactive` is a new attribute, but it is purely additive; it can be accepted +by all language versions. It does mean projects building with an older Swift +will not have access to this syntax, so as a source compatible fallback, +a client can silence this warning by fully-qualifying all types in the extension. +As an example, the above conformance can also be written as + +```swift +extension Foundation.Date: Swift.Identifiable { + // ... +} +``` + +This will allow projects that need to build with multiple versions of Swift, and +which have valid reason to declare such conformances, to declare them without +tying their project to a newer compiler build. + +## Effect on ABI stability + +This proposal has no effect on ABI stability. + +## Effect on API resilience + +This proposal has no direct effect on API resilience, but has the indirect effect of reducing +the possible surface of client changes introduced by the standard library adding new conformances. + +## Alternatives considered + +#### Enabling this warning only for resilient libraries + +A previous version of this proposal proposed enabling this warning only for resilient libraries, as those +are meant to be widely distributed and such a conformance is much more difficult to remove from clients +who expect ABI stability. However, review feedback showed a clear preference to enable this warning always, +to give library authors more freedom to introduce conformances. + +#### Putting it behind a flag + +This warning could very well be enabled by a flag, but there's not much +precedent in Swift for flags to disable individual warnings. diff --git a/proposals/0365-implicit-self-weak-capture.md b/proposals/0365-implicit-self-weak-capture.md new file mode 100644 index 0000000000..bb86d69bcb --- /dev/null +++ b/proposals/0365-implicit-self-weak-capture.md @@ -0,0 +1,231 @@ +# Allow implicit `self` for `weak self` captures, after `self` is unwrapped + +* Proposal: [SE-0365](0365-implicit-self-weak-capture.md) +* Author: [Cal Stephens](https://github.com/calda) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#40702](https://github.com/apple/swift/pull/40702), [apple/swift#61520](https://github.com/apple/swift/pull/61520) + +## Introduction + +As of [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md), implicit `self` is permitted in closures when `self` is written explicitly in the capture list. We should extend this support to `weak self` captures, and permit implicit `self` as long as `self` has been unwrapped. + +```swift +class ViewController { + let button: Button + + func setup() { + button.tapHandler = { [weak self] in + guard let self else { return } + dismiss() + } + } + + func dismiss() { ... } +} +``` + +Swift-evolution thread: [Allow implicit `self` for `weak self` captures, after `self` is unwrapped](https://forums.swift.org/t/allow-implicit-self-for-weak-self-captures-after-self-is-unwrapped/54262) + +## Motivation + +Explicit `self` has historically been required in closures, in order to help prevent users from inadvertently creating retain cycles. [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md) relaxed these rules in cases where implicit `self` is unlikely to introduce a hidden retain cycle, such as when `self` is explicitly captured in the closure's capture list: + +```swift +button.tapHandler = { [self] in + dismiss() +} +``` + +SE-0269 left the handling of `weak self` captures as a future direction, so explicit `self` is currently required in this case: + +```swift +button.tapHandler = { [weak self] in + guard let self else { return } + self.dismiss() +} +``` + +Since `self` has already been captured explicitly, there is limited value in requiring authors to use explicit `self`. This is inconsistent, and adds unnecessary visual noise to the body of closures using `weak self` captures. + +## Proposed solution + +We should permit implicit `self` for `weak self` captures, once `self` has been unwrapped. + +This code would now be allowed to compile: + +```swift +class ViewController { + let button: Button + + func setup() { + button.tapHandler = { [weak self] in + guard let self else { return } + dismiss() + } + } + + func dismiss() { ... } +} +``` + +## Detailed design + +### Enabling implicit `self` + +All of the following forms of optional unwrapping are supported, and enable implicit self for the following scope where `self` is non-optional: + +```swift +button.tapHandler = { [weak self] in + guard let self else { return } + dismiss() +} + +button.tapHandler = { [weak self] in + guard let self = self else { return } + dismiss() +} + +button.tapHandler = { [weak self] in + if let self { + dismiss() + } +} + +button.tapHandler = { [weak self] in + if let self = self { + dismiss() + } +} + +button.tapHandler = { [weak self] in + while let self { + dismiss() + } +} + +button.tapHandler = { [weak self] in + while let self = self { + dismiss() + } +} +``` + +Like with implicit `self` for `strong` and `unowned` captures, the compiler will synthesize an implicit `self.` for calls to properties / methods on `self` inside a closure that uses `weak self`. + +If `self` has not been unwrapped yet, the following error will be emitted: + +```swift +button.tapHandler = { [weak self] in + // error: explicit use of 'self' is required when 'self' is optional, + // to make control flow explicit + // fix-it: reference 'self?.' explicitly + dismiss() +} +``` + +### Nested closures + +Nested closures can be a source of subtle retain cycles, so have to be handled more carefully. For example, if the following code was allowed to compile, then the implicit `self.bar()` call could introduce a hidden retain cycle: + +```swift +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { + bar() + } +} +``` + +Following the precedent of [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md), additional closures nested inside the `[weak self]` closure must capture `self` explicitly in order to use implicit `self`. + +```swift +// Not allowed: +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { + // error: call to method 'method' in closure requires + // explicit use of 'self' to make capture semantics explicit + bar() + } +} + +// Allowed: +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { [weak self] in + guard let self else { return } + bar() + } +} + +// Also allowed: +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { + self.bar() + } +} + +// Also allowed: +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { [self] in + bar() + } +} +``` + +Also following [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md), implicit `self` will only be permitted if the `self` optional binding specifically, and exclusively, refers the closure's `self` capture: + +```swift +button.tapHandler = { [weak self] in + guard let self = self ?? someOptionalWithSameTypeOfSelf else { return } + + // error: call to method 'method' in closure requires explicit use of 'self' + // to make capture semantics explicit + method() +} +``` + +## Source compatibility + +This change is purely additive and does not break source compatibility of any valid existing Swift code. + +## Effect on ABI stability + +This change is purely additive, and is a syntactic transformation to existing valid code, so has no effect on ABI stability. + +## Effect on API resilience + +This change is purely additive, and is a syntactic transformation to existing valid code, so has no effect on API resilience. + +## Alternatives considered + +It is technically possible to also support implicit `self` _before_ `self` has been unwrapped, like: + +```swift +button.tapHandler = { [weak self] in + dismiss() // as in `self?.dismiss()` +} +``` + +That would effectively add implicit control flow, however. `dismiss()` would only be executed when `self` is not nil, without any indication that it may not run. We could create a new way to spell this that still implies optional chaining, like `?.dismiss()`, but that is not meaningfully better than the existing `self?.dismiss()` spelling. + +## Acknowledgments + +Thanks to the authors of [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md) for laying the foundation for this proposal. + +Thanks to Kyle Sluder for [the suggestion](https://forums.swift.org/t/allow-implicit-self-for-weak-self-captures-after-self-is-unwrapped/54262/2) to not permit implicit `self` in cases where the unwrapped `self` value doesn't necessarily refer to the closure's `self` capture, like in `let self = self ?? C.global`. + +Many thanks to Pavel Yaskevich, John McCall, and Xiaodi Wu for providing significant feedback and advice regarding the implementation of this proposal. \ No newline at end of file diff --git a/proposals/0366-move-function.md b/proposals/0366-move-function.md new file mode 100644 index 0000000000..fa899f24c7 --- /dev/null +++ b/proposals/0366-move-function.md @@ -0,0 +1,758 @@ +# `consume` operator to end the lifetime of a variable binding + +* Proposal: [SE-0366](0366-move-function.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm), [Andrew Trick](https://github.com/atrick), [Joe Groff](https://github.com/jckarter) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.9)** +* Implementation: in main branch of compiler +* Review: ([pitch](https://forums.swift.org/t/pitch-move-function-use-after-move-diagnostic/53983)) ([first review](https://forums.swift.org/t/se-0366-move-function-use-after-move-diagnostic/59202)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0366-move-operation-use-after-move-diagnostic/59687)) ([second review](https://forums.swift.org/t/se-0366-second-review-take-operator-to-end-the-lifetime-of-a-variable-binding/61021)) ([third review](https://forums.swift.org/t/combined-se-0366-third-review-and-se-0377-second-review-rename-take-taking-to-consume-consuming/61904)), ([acceptance](https://forums.swift.org/t/accepted-se-0366-consume-operator-to-end-the-lifetime-of-a-variable-binding/62758)) +* Previous Revisions: + * [1](https://github.com/swiftlang/swift-evolution/blob/567fb1a66c784bcc5394491d24f72a3cb393674f/proposals/0366-move-function.md) + * [2](https://github.com/swiftlang/swift-evolution/blob/43849aa9ae3e87c434866c5a5e389af67537ca26/proposals/0366-move-function.md) + * [3](https://github.com/swiftlang/swift-evolution/blob/7af91127d693bffcb01aa87978d75d5a3170c4d1/proposals/0366-move-function.md) + +## Introduction + +In this document, we propose adding a new operator, marked by the +context-sensitive keyword `consume`, to the +language. `consume` ends the lifetime of a specific local `let`, +local `var`, or function parameter, and enforces this +by causing the compiler to emit a diagnostic upon any use after the +consume. This allows for code that relies on **forwarding ownership** +of values for performance or correctness to communicate that requirement to +the compiler and to human readers. As an example: + +```swift +useX(x) // do some stuff with local variable x + +// Ends lifetime of x, y's lifetime begins. +let y = consume x // [1] + +useY(y) // do some stuff with local variable y +useX(x) // error, x's lifetime was ended at [1] + +// Ends lifetime of y, destroying the current value. +_ = consume y // [2] +useX(x) // error, x's lifetime was ended at [1] +useY(y) // error, y's lifetime was ended at [2] +``` + +## Motivation + +Swift uses reference counting and copy-on-write to allow developers to +write code with value semantics and not normally worry too much +about performance or memory management. However, in performance sensitive code, +developers want to be able to control the uniqueness of COW data structures and +reduce retain/release calls in a way that is future-proof against changes to +the language implementation or source code. Consider the following +example: + +```swift +func test() { + var x: [Int] = getArray() + + // x is appended to. After this point, we know that x is unique. We want to + // preserve that property. + x.append(5) + + // Pass the current value of x off to another function, that could take + // advantage of its uniqueness to efficiently mutate it further. + doStuffUniquely(with: x) + + // Reset x to a new value. Since we don't use the old value anymore, + // it could've been uniquely referenced by the callee. + x = [] + doMoreStuff(with: &x) +} + +func doStuffUniquely(with value: [Int]) { + // If we received the last remaining reference to `value`, we'd like + // to be able to efficiently update it without incurring more copies. + var newValue = value + newValue.append(42) + + process(newValue) +} +``` + +In the example above, a value is built up in the variable `x` and then +handed off to `doStuffUniquely(with:)`, which makes further modifications to +the value it receives. `x` is then set to a new value. It should be possible for +the caller to **forward ownership** of the value of `x` to `doStuffUniquely`, +since it no longer uses the value as is, to avoid unnecessary retains or +releases of the array buffer and unnecessary copy-on-writes of the array +contents. `doStuffUniquely` should in turn be able to move its parameter into +a local mutable variable and modify the unique buffer in place. The compiler +could make these optimizations automatically, but a number of analyses have to +align to get the optimal result. The programmer may want to guarantee that this +series of optimizations occurs, and receive diagnostics if they wrote the code +in a way that would interfere with these optimizations being possible. + +Swift-evolution pitch threads: + +- [https://forums.swift.org/t/pitch-move-function-use-after-move-diagnostic](https://forums.swift.org/t/pitch-move-function-use-after-move-diagnostic) +- [https://forums.swift.org/t/selective-control-of-implicit-copying-behavior-take-borrow-and-copy-operators-noimplicitcopy/60168](https://forums.swift.org/t/selective-control-of-implicit-copying-behavior-take-borrow-and-copy-operators-noimplicitcopy/60168) + +## Proposed solution: `consume` operator + +That is where the `consume` operator comes into play. `consume` consumes +the current value of a **binding with static lifetime**, which is either +an unescaped local `let`, unescaped local `var`, or function parameter, with +no property wrappers or get/set/read/modify/etc. accessors applied. It then + provides a compiler guarantee that the current value will +be unable to be used again locally. If such a use occurs, the compiler will +emit an error diagnostic. We can modify the previous example to use `consume` to +explicitly end the lifetime of `x`'s current value when we pass it off to +`doStuffUniquely(with:)`: + +```swift +func test() { + var x: [Int] = getArray() + + // x is appended to. After this point, we know that x is unique. We want to + // preserve that property. + x.append(5) + + // Pass the current value of x off to another function, that + doStuffUniquely(with: consume x) + + // Reset x to a new value. Since we don't use the old value anymore, + x = [] + doMoreStuff(with: &x) +} +``` + +The `consume x` operator syntax deliberately mirrors the +proposed [ownership modifier](https://forums.swift.org/t/borrow-and-take-parameter-ownership-modifiers/59581) +parameter syntax, `(x: consuming T)`, because the caller-side behavior of `consume` +operator is analogous to a callee’s behavior receiving a `consuming` parameter. +`doStuffUniquely(with:)` can use the `consume` operator, combined with +the `consuming` parameter modifier, to preserve the uniqueness of the parameter +as it moves it into its own local variable for mutation: + +```swift +func doStuffUniquely(with value: consuming [Int]) { + // If we received the last remaining reference to `value`, we'd like + // to be able to efficiently update it without incurring more copies. + var newValue = consume value + newValue.append(42) + + process(newValue) +} +``` + +This takes the guesswork out of the optimizations discussed above: in the +`test` function, the final value of `x` before reassignment is explicitly +handed off to `doStuffUniquely(with:)`, ensuring that the callee receives +unique ownership of the value at that time, and that the caller can't +use the old value again. Inside `doStuffUniquely(with:)`, the lifetime of the +immutable `value` parameter is ended to initialize the local variable `newValue`, +ensuring that the assignment doesn't cause a copy. +Furthermore, if a future maintainer modifies the code in a way that breaks +this transfer of ownership chain, then the compiler will raise an +error. For instance, if a maintainer later introduces an additional use of +`x` after it's consumed, but before it's reassigned, they will see an error: + +```swift +func test() { + var x: [Int] = getArray() + x.append(5) + + doStuffUniquely(with: consume x) + + // ERROR: x used after being consumed + doStuffInvalidly(with: x) + + x = [] + doMoreStuff(with: &x) +} +``` + +Likewise, if the maintainer tries to access the original `value` parameter inside +of `doStuffUniquely` after being consumed to initialize `newValue`, they will +get an error: + +```swift +func doStuffUniquely(with value: consuming [Int]) { + // If we received the last remaining reference to `value`, we'd like + // to be able to efficiently update it without incurring more copies. + var newValue = consume value + newValue.append(42) + + process(newValue) +} +``` + +`consume` can also end the lifetime of local immutable `let` bindings, which become +unavailable after their value is consumed since they cannot be reassigned. +Also note that `consume` operates on bindings, not values. If we declare a +constant `x` and another local constant `other` with the same value, +we can still use `other` after we consume the value from `x`, as in: + +```swift +func useX(_ x: SomeClassType) -> () {} + +func f() { + let x = ... + useX(x) + let other = x // other is a new binding used to extend the lifetime of x + _ = consume x // x's lifetime ends + useX(other) // other is used here... no problem. + useX(other) // other is used here... no problem. +} +``` + +We can also consume `other` independently of `x`, and get separate diagnostics for both +variables: + +```swift +func useX(_ x: SomeClassType) -> () {} + +func f() { + let x = ... + useX(x) + let other = x + _ = consume x + useX(consume other) + useX(other) // error: 'other' used after being consumed + useX(x) // error: 'x' used after being consumed +} +``` + +`inout` function parameters can also be used with `consume`. Like a `var`, an +`inout` parameter can be reassigned after being consumed from and used again; +however, since the final value of an `inout` parameter is passed back to the +caller, it *must* be reassigned by the callee before it +returns. So this will raise an error because `buffer` doesn't have a value at +the point of return: + +```swift +func f(_ buffer: inout Buffer) { // error: 'buffer' not reinitialized after consume! + let b = consume buffer // note: consume was here + b.deinitialize() + ... write code ... +} // note: return without reassigning inout argument `buffer` +``` + +But we can reinitialize `buffer` by writing the following code: + +```swift +func f(_ buffer: inout Buffer) { + let b = consume buffer + b.deinitialize() + // ... write code ... + // We re-initialized buffer before end of function so the checker is satisfied + buffer = getNewInstance() +} +``` + +`defer` can also be used to reinitialize an `inout` or `var` after a consume, +in order to ensure that reassignment happens on any exit from scope, including +thrown errors or breaks out of loops. So we can also write: + +```swift +func f(_ buffer: inout Buffer) { + let b = consume buffer + // Ensure the buffer is reinitialized before we exit. + defer { buffer = getNewInstance() } + try b.deinitializeOrError() + // ... write code ... +} +``` + +## Detailed design + +At runtime, `consume x` evaluates to the current value bound to `x`, just like the +expression `x` does. However, at compile time, the presence of a `consume` forces +ownership of the argument to be transferred out of the binding at the given +point so. Any ensuing use of the binding that's reachable from the `consume` +is an error. The operand to `consume` is required to be a reference +to a *binding with static lifetime*. The following kinds of declarations can +currently be referenced as bindings with static lifetime: + +- a local `let` constant in the immediately-enclosing function, +- a local `var` variable in the immediately-enclosing function, +- one of the immediately-enclosing function's parameters, or +- the `self` parameter in a `mutating` or `__consuming` method. + +A binding with static lifetime also must satisfy the following requirements: + +- it cannot be captured by an `@escaping` closure or nested function, +- it cannot have any property wrappers applied, +- it cannot have any accessors attached, such as `get`, `set`, + `didSet`, `willSet`, `_read`, or `_modify`, +- it cannot be an `async let`. + +Possible extensions to the set of operands that can be used with `consume` are +discussed under Future Directions. It is an error to use `consume` with an operand +that doesn't reference a binding with static lifetime. + +Given a valid operand, `consume` enforces that there are no other +references to the binding after it is consumed. The analysis is +flow sensitive, so one is able to end the lifetime of a value conditionally: + +```swift +if condition { + let y = consume x + // I can't use x anymore here! + useX(x) // !! ERROR! Use after consume. +} else { + // I can still use x here! + useX(x) // OK +} +// But I can't use x here. +useX(x) // !! ERROR! Use after consume. +``` + +If the binding is a `var`, the analysis additionally allows for code to +conditionally reinitialize the var and thus use it in positions +that are dominated by the reinitialization. Consider the +following example: + +```swift +if condition { + _ = consume x + // I can't use x anymore here! + useX(x) // !! ERROR! Use after consume. + x = newValue + // But now that I have re-assigned into x a new value, I can use the var + // again. + useX(x) // OK +} else { + // I can still use x here, since it wasn't consumed on this path! + useX(x) // OK +} +// Since I reinitialized x along the `if` branch, and it was never consumed +// from on the `else` branch, I can use it here too. +useX(x) // OK +``` + +Notice how in the above, we are able to use `x` both in the true block AND the +code after the `if` block, since over both paths through the `if`, `x` ends up +with a valid value before proceeding. + +For an `inout` parameter, the analysis behaves the same as for a `var`, except +that all exits from the function (whether by `return` or by `throw`) are +considered to be uses of the parameter. Correct code therefore *must* reassign +inout parameters after they are consumed from. + +Using `consume` on a binding without using the result raises a warning, +just like a function call that returns an unused non-`Void` result. +To "drop" a value without using it, the `consume` can be assigned to +`_` explicitly. + +## Source compatibility + +`consume` behaves as a contextual keyword. In order to avoid interfering +with existing code that calls functions named `consume`, the operand to +`consume` must begin with another identifier, and must consist of an +identifier or postfix expression: + +```swift +consume x // OK +consume [1, 2, 3] // Subscript access into property named `consume`, not a consume operation +consume (x) // Call to function `consume`, not a consume operation +consume x.y.z // Syntactically OK (although x.y.z is not currently semantically valid) +consume x[0] // Syntactically OK (although x[0] is not currently semantically valid +consume x + y // Parses as (consume x) + y +``` + +## Effect on ABI stability + +`consume` requires no ABI additions. + +## Effect on API resilience + +None, this is additive. + +## Alternatives considered + +### Alternative spellings + +The [first reviewed revision](https://github.com/swiftlang/swift-evolution/blob/567fb1a66c784bcc5394491d24f72a3cb393674f/proposals/0366-move-function.md) +of this proposal offered `move(x)` as a special +function with semantics recognized by the compiler. Based on initial feedback, +we pivoted to the contextual keyword spelling. As a function, this operation +would be rather unusual, since it only accepts certain forms of expression as +its argument, and it doesn't really have any runtime behavior of its own, +acting more as a marker for the compiler to perform additional analysis. + +The community reviewed the contextual keyword syntax, using the name `move x`, +and through further discussion the alternative name `take` arose. This name +aligns with the term used in the Swift compiler internals, and also reads well +as the analogous parameter ownership modifier, `(x: take T)`, so the authors +now favor this name. However, during review of SE-0377, reviewers found that +`take` was too generic, and could be confused with the common colloquial language +of talking about function calls as "taking their arguments". Of alternative +names reviewers offered, the language workgroup favors `consume`. + +Many have suggested alternative spellings that also make `consume`'s special +nature more syntactically distinct, including: + +- an expression attribute, like `useX(@consume x)` +- a compiler directive, like `useX(#consume x)` +- an operator, like `useX(<-x)` + +### Use of scoping to end lifetimes + +It is possible in the language today to foreshorten the lifetime of local +variables using lexical scoping: + +```swift +func test() { + var x: [Int] = getArray() + + // x is appended to. After this point, we know that x is unique. We want to + // preserve that property. + x.append(5) + + // We create a new variable y so we can write an algorithm where we may + // change the value of y (causing a COW copy of the buffer shared with x). + do { + var y = x + longAlgorithmUsing(&y) + consumeFinalY(y) + } + + // We no longer use y after this point. Ideally, x would be guaranteed + // unique so we know we can append again without copying. + x.append(7) +} +``` + +However, there are a number of reasons not to rely solely on lexical scoping +to end value lifetimes: + +- Adding lexical scopes requires nesting, which can lead to "pyramid of doom" + situations when managing the lifetimes of multiple variables. +- Value lifetimes don't necessarily need to nest, and may overlap or interleave + with control flow. This should be valid: + + ```swift + let x = foo() + let y = bar() + // end x's lifetime before y's + consume(consume x) + consume(consume y) + ``` + +- Lexical scoping cannot be used by itself to shorten the lifetime of function + parameters, which are in scope for the duration of the function body. +- Lexical scoping cannot be used to allow for taking from and reinitializing + mutable variables or `inout` parameters. + +Looking outside of Swift, the Rust programming language originally only had +strictly scoped value lifetimes, and this was a significant ergonomic +problem until "non-lexical lifetimes" were added later, which allowed for +value lifetimes to shrinkwrap to their actual duration of use. + +## Future directions + +### Dynamic enforcement of `consume` for other kinds of bindings + +In the future, we may want to accommodate the ability to dynamically +consume bindings with dynamic lifetime, such as escaped local +variables, and class stored properties, although doing so in full +generality would require dynamic enforcement in addition to static +checking, similar to how we need to dynamically enforce exclusivity +when accessing globals and class stored properties. Since this +dynamic enforcement turns misuse of `consume`s into runtime errors +rather than compile-time guarantees, we might want to make those +dynamic cases syntactically distinct, to make the possibility of +runtime errors clear. + +`Optional` and other types with a canonical "no value" or "empty" +state can use the static `consume` operator to provide API that +dynamically takes ownership of the current value inside of them +while leaving them in their empty state: + +```swift +extension Optional { + mutating func take() -> Wrapped { + switch consume self { + case .some(let x): + self = .none + return x + case .none: + fatalError("trying to consume an empty Optional") + } + } +} +``` + +### Piecewise `consume` of frozen structs and tuples + +For frozen structs and tuples, both aggregates that the compiler can statically +know the layout of, we could do finer-grained analysis and allow their +individual fields to be consumed independently: + +```swift +struct TwoStrings { + var first: String + var second: String +} + +func foo(x: TwoStrings) { + use(consume x.first) + // ERROR! part of x was consumed + use(x) + // OK, this part wasn't + use(x.second) +} +``` + +### Destructuring methods for move-only types with `deinit` + +Move-only types would allow for the possibility of value types with custom +`deinit` logic that runs at the end of a value of the type's lifetime. +Typically, this logic would run when the final owner of the value is finished +with it, which means that a function which `consume`s an instance, or a +`taking func` method on the type itself, would run the deinit if it does not +forward ownership anywhere else: + +```swift +moveonly struct FileHandle { + var fd: Int32 + + // close the fd on deinit + deinit { close(fd) } +} + +func dumpAndClose(to fh: consume FileHandle, contents: Data) { + write(fh.fd, contents) + // fh implicitly deinit-ed here, closing it +} +``` + +However, this may not always be desirable, either because the function performs +an operation that invalidates the value some other way, making it unnecessary +or incorrect for `deinit` to run on it, or because the function wants to be able +to take ownership of parts away from the value: + +```swift +extension FileHandle { + // Return the file descriptor back to the user for manual management + // and disable automatic management with the FileHandle value. + + taking func giveUp() -> Int32 { + return fd + // How do we stop the deinit from running here? + } +} +``` + +Rust has the magic function `mem::forget` to suppress destruction of a value, +though `forget` in Rust still does not allow for the value to be destructured +into parts. We could come up with a mechanism in Swift that both suppresses +implicit deinitialization, and allows for piecewise taking of its components. +This doesn't require a new parameter convention (since it fits within the +ABI of a `consume T` parameter), but could be spelled as a new operator +inside of a `taking func`: + +```swift +extension FileHandle { + // Return the file descriptor back to the user for manual management + // and disable automatic management with the FileHandle value. + + taking func giveUp() -> Int32 { + // `deinit fd` is strawman syntax for consuming a value without running + // its deinitializer. it is only allowed inside of `taking func` methods + // that have visibility into the type layout. + return (deinit self).fd + } +} + +moveonly struct SocketPair { + var input, output: FileHandle + + deinit { /* ... */ } + + // Break the pair up into separately-managed FileHandles + taking func split() -> (input: FileHandle, output: FileHandle) { + // Break apart the value without running the standard deinit + return ((deinit self).input, (deinit self).output) + } +} +``` + +Suppressing the normal deinit logic for a type should be done carefully. +Although Rust allows `mem::forget` to be used anywhere, and considers it "safe" +based on the observation that destructors may not always run if a program +crashes or leaks a value anyway, that observation is less safe in the case +of values whose lifetime depends on a parent value, and where the child value's +deinit must be sequenced with operations on the parent. As such, I think +the ability should be limited to methods defined on the type itself. Also, +within the constraints of Swift's stable ABI and library evolution model, +code that imports a module may not always have access to the concrete layout +of a type, which would make partial destructuring impossible, which further +limits the contexts in which such an operator can be used. + +### `consume` from computed properties, property wrappers, properties with accessors, etc. + +It would potentially be useful to be able to `consume` variables +and properties with modified access behavior, such as computed +properties, properties with didSet/willSet observers, property +wrappers, and so on. Although we could do lifetime analysis on these +properties, we wouldn't be able to get the full performance benefits +from consuming a computed variable without allowing for some +additional accessors to be defined, such as a "consuming getter" +that can consume its `self` in order to produce the property value, +and an initializer to reinitialize `self` on reassignment after a +`move`. + +### Additional selective controls for implicit copying behavior + +Pitch: [Selective control of implicit copying behavior](https://forums.swift.org/t/selective-control-of-implicit-copying-behavior-take-borrow-and-copy-operators-noimplicitcopy/60168) + +The `consume` operator is one of a number of implicit copy controls +we're considering: + +- A value that isn't modified can generally be "borrowed" and + shared in-place by multiple bindings, or between a caller and + callee, without copying. However, the compiler will + pass shared mutable state by copying the current value, and + passing that copy to a callee. We do this to avoid + potential rule-of-exclusivity violations, since it is difficult to + know for sure whether a callee will go back and try to mutate the + same global variable, object, or other bit of shared mutable + state: + + ```swift + var global = Foo() + + func useFoo(x: Foo) { + // We would need exclusive access to `global` to do this: + + /* + global = Foo() + */ + } + + func callUseFoo() { + // callUseFoo doesn't know whether `useFoo` accesses global, + // so we want to avoid imposing shared access to it for longer + // than necessary. So by default the compiler will + // pass a copy instead, and this: + useFoo(x: global) + + // will compile more like: + + /* + let copyOfGlobal = copy(global) + useFoo(x: copyOfGlobal) + destroy(copyOfGlobal) + */ + } + ``` + + Although the compiler is allowed to eliminate the defensive copy + inside callUseFoo if it proves that useFoo doesn't try to write + to the global variable, it is unlikely to do so in practice. The + developer however knows that useFoo doesn't modify global, and + may want to suppress this copy in the call site. An explicit + `borrow` operator would let the developer communicate this to the + compiler: + + ```swift + var global = Foo() + + func useFoo(x: Foo) { + /* global not used here */ + } + + func callUseFoo() { + // The programmer knows that `useFoo` won't + // touch `global`, so we'd like to pass it without copying + useFoo(x: borrow global) + } + ``` + +- `consume` and `borrow` operators can eliminate copying in + common localized situations, but it is also useful to be able to + suppress implicit copying altogether for certain variables, types, + and scopes. We could define an attribute to specify that bindings + with static lifetime, types, or scopes should not admit implicit + copies: + + ```swift + // we're not allowed to implicitly copy `x` + func foo(@noImplicitCopy x: String) { + } + + // we're not allowed to implicitly copy values (statically) of + // type Gigantic + @noImplicitCopy struct Gigantic { + var fee, fie, fo, fum: String + } + + // we're not allowed to implicitly copy inside this hot loop + for item in items { + @noImplicitCopy do { + } + } + ``` + +### `borrow` and `consume` argument modifiers + +Pitch: [`borrow` and `consume` parameter ownership modifiers](https://forums.swift.org/t/borrow-and-take-parameter-ownership-modifiers/59581) + +Swift currently only makes an explicit distinction between +pass-by-value and pass-by-`inout` parameters, leaving the mechanism +for pass-by-value up to the implementation. But there are two broad +conventions that the compiler uses to pass by value: + +- The callee can **borrow** the parameter. The caller guarantees that + its argument object will stay alive for the duration of the call, + and the callee does not need to release it (except to balance any + additional retains it performs itself). +- The callee can **consume** the parameter. The callee becomes + responsible for either releasing the parameter or passing + ownership of it along somewhere else. If a caller doesn't want to + give up its own ownership of its argument, it must retain the + argument so that the callee can consume the extra reference count. + +In order to allow for manual optimization of code, and to support +move-only types where this distinction becomes semantically +significant, we plan to introduce explicit parameter modifiers to +let developers specify explicitly which convention a parameter +should use. + +## Acknowledgments + +Thanks to Nate Chandler, Tim Kientzle, and Holly Borla for their help with this! + +## Revision history + +Changes from the [third revision](https://github.com/swiftlang/swift-evolution/blob/7af91127d693bffcb01aa87978d75d5a3170c4d1/proposals/0366-move-function.md): + +- `take` is renamed again to `consume`. + +Changes from the [second revision](https://github.com/swiftlang/swift-evolution/blob/43849aa9ae3e87c434866c5a5e389af67537ca26/proposals/0366-move-function.md): + +- `move` is renamed to `take`. +- Dropping a value without using it now requires an explicit + `_ = take x` assignment again. +- "Movable bindings" are referred to as "bindings with static lifetime", + since this term is useful and relevant to other language features. +- Additional "alternatives considered" raised during review + and pitch discussions were added. +- Expansion of "related directions" section contextualizes the + `take` operator among other planned features for selective copy + control. +- Now that [ownership modifiers for parameters](https://forums.swift.org/t/borrow-and-take-parameter-ownership-modifiers/59581) + are being pitched, this proposal ties into that one. Based on + feedback during the first review, we have gone back to only allowing + parameters to be used with the `take` operator if the parameter + declaration is `take` or `inout`. + +Changes from the [first revision](https://github.com/swiftlang/swift-evolution/blob/567fb1a66c784bcc5394491d24f72a3cb393674f/proposals/0366-move-function.md): + +- `move x` is now proposed as a contextual keyword, instead of a magic function + `move(x)`. +- The proposal no longer mentions `__owned` or `__shared` parameters, which + are currently an experimental language feature, and leaves discussion of them + as a future direction. `move x` is allowed to be used on all function + parameters. +- `move x` is allowed as a statement on its own, ignoring the return value, + to release the current value of `x` without forwarding ownership without + explicitly assigning `_ = move x`. diff --git a/proposals/0367-conditional-attributes.md b/proposals/0367-conditional-attributes.md new file mode 100644 index 0000000000..09b915db61 --- /dev/null +++ b/proposals/0367-conditional-attributes.md @@ -0,0 +1,101 @@ +# Conditional compilation for attributes + +* Proposal: [SE-0367](0367-conditional-attributes.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.8)** + +* Implementation: [apple/swift#60208](https://github.com/apple/swift/pull/60208) +* Swift-evolution thread: [Pitch](https://forums.swift.org/t/pitch-conditional-compilation-for-attributes-and-modifiers/58339) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0367-conditional-compilation-for-attributes/59756) + +## Introduction + +Over time, Swift has introduced a number of new attributes to communicate additional information in source code. Existing code can then be updated to take advantage of these new constructs to improve its behavior, providing more expressive capabilities, better compile-time checking, better performance, and so on. + +However, adopting a new attribute in existing source code means that source code won't compile with an older compiler. Conditional compilation can be used to address this problem, but the result is verbose and unsatisfactory. For example, one could use `#if` to check the compiler version to determine whether to use the `@preconcurrency` attribute: + +```swift +#if compiler(>=5.6) +@preconcurrency protocol P: Sendable { + func f() + func g() +} +#else +protocol P: Sendable { + func f() + func g() +} +#endif +``` + +This is unsatisfactory for at least two reasons. First, it's a lot of code duplication, because the entire protocol `P` needs to be duplicated just to provide the attribute. Second, the Swift 5.6 compiler is the first to contain the `@preconcurrency` attribute, but that is somewhat incidental and not self-documenting: the attribute could have been enabled by a compiler flag or partway through the development of Swift 5.6, making that check incorrect. Moreover, the availability of some attributes can depend not on compiler version, but on platform and configuration flags: for example, `@objc` is only available when the Swift runtime has been compiled for interoperability with Objective-C. Although these are small issues in isolation, they make adopting new attributes in existing code harder than it needs to be. + +## Proposed solution + +I propose two related changes to make it easier to adopt new attributes in existing code: + +* Allow `#if` checks to surround attributes on a declaration wherever they appear, eliminating the need to clone a declaration just to adopt a new attribute. +* Add a conditional directive `hasAttribute(AttributeName)` that evaluates `true` when the compiler has support for the attribute with the name `AttributeName` in the current language mode. + +The first two of these can be combined to make the initial example less repetitive and more descriptive: + +```swift +#if hasAttribute(preconcurrency) +@preconcurrency +#endif +protocol P: Sendable { + func f() + func g() +} +``` + +## Detailed design + +The design of these features is relatively straightforward, but there are a few details to cover. + +### Grammar changes + +The current production for an attribute list: + +``` +attributes → attribute attributes[opt] +``` + +will be augmented with an additional production for a conditional attribute: + +``` +attributes → conditional-compilation-attributes attributes[opt] + +conditional-compilation-attributes → if-directive-attributes elseif-directive-attributes[opt] else-directive-attributes[opt] endif-directive +if-directive-attributes → if-directive compilation-condition attributes[opt] +elseif-directive-attributes → elseif-directive-attributes elseif-directive-attributes[opt] +elseif-directive-attributes → elseif-directive compilation-condition attributes[opt] +else-directive-attributes → else-directive attributes[opt] +``` + +i.e., within an attribute list one can have a conditional clause `#if...#endif` that wraps another attribute list. + +### `hasAttribute` only considers attributes that are part of the language + +A number of Swift language features, including property wrappers, result builders, and global actors, all introduce forms of custom attributes. For example, a type `MyWrapper` that has been marked with the `@propertyWrapper` attribute (and meets the other requirements for a property wrapper type) can be used with the attribute syntax `@MyWrapper`. While the built-in attribute that enables the feature will be recognized by `hasAttribute` (e.g., `hasAttribute(propertyWrapper)` will evaluate `true`), the custom attribute will not (e.g., `hasAttribute(MyWrapper)` will evaluate `false`). + +### Parsing the conditionally-compiled branches not taken + +Due to support for custom attributes, attributes have a very general grammar that should suffice for any new attributes we introduce into Swift: + +``` +attribute → @ attribute-name attribute-argument-clause[opt] +attribute-name → identifier +attribute-argument-clause → ( balanced-tokens[opt] ) +``` + +Therefore, a conditionally-compiled branch based on `#if hasAttribute(UnknownAttributeName)` can still be parsed by an existing compiler, even though it will not be applied to the declaration because it isn't understood: + +```swift +#if hasAttribute(UnknownAttributeName) +@UnknownAttributeName(something we do not understand) // okay, we parse this but don't reject it +#endif +func f() +``` + diff --git a/proposals/0368-staticbigint.md b/proposals/0368-staticbigint.md new file mode 100644 index 0000000000..4c86fb5ff0 --- /dev/null +++ b/proposals/0368-staticbigint.md @@ -0,0 +1,179 @@ +# StaticBigInt + +* Proposal: [SE-0368](0368-staticbigint.md) +* Author: [Ben Rimmington](https://github.com/benrimmington) +* Review Manager: [Doug Gregor](https://github.com/DougGregor), [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#40722](https://github.com/apple/swift/pull/40722), [apple/swift#62733](https://github.com/apple/swift/pull/62733) +* Review: ([pitch](https://forums.swift.org/t/staticbigint/54545)) ([review](https://forums.swift.org/t/se-0368-staticbigint/59421)) ([acceptance](https://forums.swift.org/t/accepted-se-0368-staticbigint/59962)) ([amendment](https://forums.swift.org/t/pitch-amend-se-0368-to-remove-prefix-operator/62173)), ([amendment review](https://forums.swift.org/t/amendment-se-0368-staticbigint/62992)), ([amendment acceptance](https://forums.swift.org/t/accepted-amendment-se-0368-staticbigint/63246)) + +
+Revision history + +| | | +| ---------- | ------------------------------------------------- | +| 2022-01-10 | Initial pitch. | +| 2022-02-01 | Updated with an "ABI-neutral" abstraction. | +| 2022-04-23 | Updated with an "infinitely-sign-extended" model. | +| 2022-08-18 | Updated with a "non-generic" subscript. | +| 2023-02-03 | Amended to remove the prefix `+` operator. | + +
+ +## Introduction + +Integer literals in Swift source code can express an arbitrarily large value. However, types outside of the standard library which conform to `ExpressibleByIntegerLiteral` are restricted in practice in how large of a literal value they can be built with, because the value passed to `init(integerLiteral:)` must be of a type supported by the standard library. This makes it difficult to write new integer types outside of the standard library. + +## Motivation + +Types in Swift that want to be buildable with an integer literal can conform to the following protocol: + +```swift +public protocol ExpressibleByIntegerLiteral { + associatedtype IntegerLiteralType: _ExpressibleByBuiltinIntegerLiteral + init(integerLiteral value: IntegerLiteralType) +} +``` + +The value passed to `init(integerLiteral:)` must have a type that knows how to manage the primitive interaction with the Swift compiler so that it can be built from an arbitrary literal value. That constraint is expressed with the `_ExpressibleByBuiltinIntegerLiteral` protocol, which cannot be implemented outside of the standard library. All of the integer types in the standard library conform to `_ExpressibleByBuiltinIntegerLiteral` as well as `ExpressibleByIntegerLiteral`. A type outside of the standard library must select one of those types as the type it takes in `init(integerLiteral:)`. As a result, such types cannot be built from an integer literal if there isn't a type in the standard library big enough to express that integer. + +For example, if larger fixed-width integers (such as `UInt256`) were added to the [Swift Numerics][] package, they would currently have to use smaller literals (such as `UInt64`). + +```swift +let value: UInt256 = 0x1_0000_0000_0000_0000 +// ^ +// error: integer literal '18446744073709551616' overflows when stored into 'UInt256' +``` + +## Proposed solution + +We propose adding a new type to the standard library called `StaticBigInt` which is capable of expressing any integer value. This can be used as the associated type of an `ExpressibleByIntegerLiteral` conformance. For example: + +```swift +extension UInt256: ExpressibleByIntegerLiteral { + + public init(integerLiteral value: StaticBigInt) { + precondition( + value.signum() >= 0 && value.bitWidth <= Self.bitWidth + 1, + "integer literal '\(value)' overflows when stored into '\(Self.self)'" + ) + self.words = Words() + for wordIndex in 0.. Int + + /// Returns the minimal number of bits in this value's binary representation, + /// including the sign bit, and excluding the sign extension. + /// + /// The following examples show the least significant byte of each value's + /// binary representation, separated (by an underscore) into excluded and + /// included bits. Negative values are in two's complement. + /// + /// * `-4` (`0b11111_100`) is 3 bits. + /// * `-3` (`0b11111_101`) is 3 bits. + /// * `-2` (`0b111111_10`) is 2 bits. + /// * `-1` (`0b1111111_1`) is 1 bit. + /// * `+0` (`0b0000000_0`) is 1 bit. + /// * `+1` (`0b000000_01`) is 2 bits. + /// * `+2` (`0b00000_010`) is 3 bits. + /// * `+3` (`0b00000_011`) is 3 bits. + public var bitWidth: Int { get } + + /// Returns a 32-bit or 64-bit word of this value's binary representation. + /// + /// The words are ordered from least significant to most significant, with + /// an infinite sign extension. Negative values are in two's complement. + /// + /// let negative: StaticBigInt = -0x0011223344556677_8899AABBCCDDEEFF + /// negative.signum() //-> -1 + /// negative.bitWidth //-> 118 + /// negative[0] //-> 0x7766554433221101 + /// negative[1] //-> 0xFFEEDDCCBBAA9988 + /// negative[2] //-> 0xFFFFFFFFFFFFFFFF + /// + /// let positive: StaticBigInt = 0x0011223344556677_8899AABBCCDDEEFF + /// positive.signum() //-> +1 + /// positive.bitWidth //-> 118 + /// positive[0] //-> 0x8899AABBCCDDEEFF + /// positive[1] //-> 0x0011223344556677 + /// positive[2] //-> 0x0000000000000000 + /// + /// - Parameter wordIndex: A nonnegative zero-based offset. + public subscript(_ wordIndex: Int) -> UInt { get } +} +``` + +## Effect on ABI stability + +This feature adds to the ABI of the standard library, and it won't back-deploy (by default). + +The integer literal type has to be selected statically as the associated type. There is currently no way to conditionally use a different integer literal type depending on the execution environment. Types will not be able to adopt this and use the most flexible possible literal type dynamically available. + +## Alternatives considered + +- Modeling the original source text instead of a mathematical value would allow this type to support a wide range of use cases, such as fractional values, decimal values, and other things such as arbitrary binary strings expressed in hexadecimal. However, it is not a goal of Swift's integer literals design to support these use cases. Supporting them would burden integer types with significant code size, dynamic performance, and complexity overheads. For example, either the emitted code would need to contain both the original source text and a more optimized representation used by ordinary integer types, or ordinary integer types would need to fall back on parsing numeric values from source at runtime. + +- Along similar lines, it is intentional that `StaticBigInt` cannot represent fractional values. Integer types should not be constructible with fractional literals, and allowing that simply adds unnecessary costs and introduces a new way for construction to fail. It is still a language goal for Swift to someday support dynamically flexible floating-point literals the way it does for integer literals, but that is a separable project from introducing `StaticBigInt`. + +- A prior design had a `words` property, initially as a contiguous buffer, subsequently as a custom collection. John McCall requested an "ABI-neutral" abstraction, and suggested the current "infinitely-sign-extended" model. Xiaodi Wu convincingly argued for a "non-generic" subscript, rather than over-engineering a choice of element type. + +- Xiaodi Wu [suggested](https://forums.swift.org/t/staticbigint/54545/23) that a different naming scheme and API design be chosen to accommodate other similar types, such as IEEE 754 interchange formats. However, specific alternatives weren't put forward for consideration. Using non-arithmetic types for interchange formats would seem to be a deliberate choice; whereas for `StaticBigInt` it's because of an inherent limitation. + +- A previously accepted version of this proposal included the following operator, for symmetry between negative and positive literals. + + ```swift + extension StaticBigInt { + /// Returns the given value unchanged. + public static prefix func + (_ rhs: Self) -> Self + } + ``` + + It was later discovered to be a source-breaking change. For example: + + ```swift + let a = -7 // inferred as `a: Int` + let b = +6 // inferred as `b: StaticBigInt` + let c = a * b + // ^ + // error: Cannot convert value of type 'StaticBigInt' to expected argument type 'Int' + ``` + + The prefix `+` operator on [`AdditiveArithmetic`][numeric protocols] was no longer chosen, because concrete overloads are preferred over generic overloads. + +## Acknowledgments + +John McCall made significant improvements to this proposal; and (in Swift 5.0) implemented arbitrary-precision integer literals. `StaticBigInt` is a thin wrapper around the existing [`Builtin.IntLiteral`][] type. + +Stephen Canon proposed an amendment to remove the prefix `+` operator. + + + +[`Builtin.IntLiteral`]: + +[numeric protocols]: + +[Swift Numerics]: diff --git a/proposals/0369-add-customdebugdescription-conformance-to-anykeypath.md b/proposals/0369-add-customdebugdescription-conformance-to-anykeypath.md new file mode 100644 index 0000000000..6f58d7aff8 --- /dev/null +++ b/proposals/0369-add-customdebugdescription-conformance-to-anykeypath.md @@ -0,0 +1,160 @@ +# Add CustomDebugStringConvertible conformance to AnyKeyPath + +* Proposal: [SE-0369](0369-add-customdebugdescription-conformance-to-anykeypath.md) +* Author: [Ben Pious](https://github.com/benpious) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#60133](https://github.com/apple/swift/pull/60133) +* Review: ([pitch](https://forums.swift.org/t/pitch-add-customdebugdescription-conformance-to-anykeypath/58705)) ([review](https://forums.swift.org/t/se-0369-add-customdebugstringconvertible-conformance-to-anykeypath/59704)) ([acceptance](https://forums.swift.org/t/accepted-se-0369-add-customdebugstringconvertible-conformance-to-anykeypath/60001)) + +## Introduction + +This proposal is to add conformance to the protocol `CustomDebugStringConvertible` to `AnyKeyPath`. + +## Motivation + +Currently, passing a keypath to `print()`, or to the `po` command in LLDB, yields the standard output for a Swift class. This is not very useful. For example, given +```swift +struct Theme { + + var backgroundColor: Color + var foregroundColor: Color + + var overlay: Color { + backgroundColor.withAlpha(0.8) + } +} +``` +`print(\Theme.backgroundColor)` would have an output of roughly +``` +Swift.KeyPath +``` +which doesn't allow `foregroundColor` to be distinguished from any other property on `Theme`. + +Ideally, the output would be +``` +\Theme.backgroundColor +``` +exactly as it was written in the program. + +## Proposed solution + +Take advantage of whatever information is available in the binary to implement the `debugDescription` requirement of `CustomDebugStringConvertible`. In the best case, roughly the output above will be produced, in the worst cases, other, potentially useful information will be output instead. + +## Detailed design + +### Implementation of `CustomDebugStringConvertible` + +Much like the `_project` functions currently implemented in `KeyPath.swift`, this function would loop through the keypath's buffer, handling each segment as follows: + +For offset segments, the implementation is simple: use `_getRecursiveChildCount`, `_getChildOffset`, and `_getChildMetadata` to get the string name of the property. I believe these are the same mechanisms used by `Mirror` today. + +For optional chain, force-unwrap, etc. the function appends a hard coded "?" or "!", as appropriate. + +For computed segments, call `swift::lookupSymbol()` on the result of `getter()` in the `ComputedAccessorsPtr`. Demangle the result to get the property name. + +### Changes to the Swift Runtime + +To implement descriptions for computed segments, it is necessary to make two changes to the runtime: + +1. Expose a Swift calling-convention function to call `swift::lookupSymbol()`. +2. Implement and expose a function to demangle keypath functions without the ornamentation the existing demangling functions produce. + +### Dealing with missing data + +There are two known cases where data might not be available: + +1. type metadata has not been emitted because the target was built using the `swift-disable-reflection-metadata` flag +2. the linker has stripped the symbol names we're trying to look up + +In these cases, we would print the following: + +#### Offset case +`` where `x` is the memory offset we read from the reflection metadata, and `typename` is the type that will be returned. +So +``` +print(\Theme.backgroundColor) // outputs "\Theme." +``` + +#### `lookupSymbol` failure case + +In this case we'll print the address-in-memory as hex, plus the type name: +``` +print(\Theme.overlay) // outputs \Theme. +``` + +As it might be difficult to correlate a memory address with the name of the function, the type name may be useful here to provide extra context. + +## Source compatibility + +Programs that extend `AnyKeyPath` to implement `CustomDebugStringConvertible` themselves will no longer compile and the authors of such code will have to delete the conformance. Based on a search of Github, there are currently no publicly available Swift projects that do this. + +Calling `print` on a KeyPath will, of course, produce different results than before. + +It is unlikely that any existing Swift program is depending on the existing behavior in a production context. While it is likely that someone, somewhere has written code in unit tests that depends on the output of this function, any issues that result will be easy for the authors of such code to identify and fix, and will likely represent an improvement in the readability of those tests. + +## Effect on ABI stability + +This proposal will add a new var & protocol conformance to the Standard Library's ABI. It will be availability guarded appropriately. + +The new debugging output will not be backdeployed, so Swift programs running on older ABI stable versions of the OS won't be able to rely on the new output. + +## Effect on API resilience + +The implementation of `debugDescription` might change after the initial work to implement this proposal is done. In particular, the output format will not be guaranteed to be stable. Here are a few different changes we might anticipate making: + +- As new features are added to the compiler, there may be new metadata available in the binary to draw from. One example would be lookup tables of KeyPath segment to human-readable-name or some other unique, stable identifier +- Whenever a new feature is added to `KeyPath`, it will need to be reflected in the output of this function. For example, the `KeyPath`s produced by [\_forEachFieldWithKeyPath](https://github.com/apple/swift/blob/main/stdlib/public/core/ReflectionMirror.swift#L324) are incomplete, in the sense that they merely set a value at an offset in memory and do not call `didSet` observers. If this function were ever publicly exposed, it would be useful if this was surfaced in the debugging information. +- The behavior of subscript printing might be changed: for example, we might always print out the value of the argument to the subscript, or we might do so only in cases where the output is short. We might also change from `.subscript()` to `[]` +- The Swift language workgroup may create new policies around debug descriptions and the output of this function might need to be updated to conform to them + +## Alternatives considered + +### Print fully qualified names or otherwise add more information to the output + +ex. `\ModuleName.MyType.myField`, `> \ModuleName.MyType.myField` `(writable) \Theme.backgroundColor` + +As this is just for debugging, it seems likely that the information currently being provided would be enough to resolve any ambiguities. If ambiguities arose during a debugging session, in most cases the user could figure out exactly which keypath they were dealing with simply by running `po myKeyPath == \MyType.myField` till they found the right one. + +### Modify KeyPaths to include a string description + +This is an obvious solution to this problem, and would likely be very easy to implement, as the compiler already produces `_kvcString`. + +It has the additional advantage of being 100% reliable, to the point where it arguably could be the basis for implementing `description` rather than `debugDescription`. + +However, it would add to the code size of the compiled code, perhaps unacceptably so. Furthermore, it precludes the possibility of someday printing out the arguments of subscript based keypaths, as +these can be created dynamically. It would also add overhead to appending keypaths, as the string would also have to be appended. + +An alternative implementation of this idea would be the output of additional metadata: a lookup table of function -> name. However, this would require a lot of additional work on the compiler for a relatively small benefit. + +I think that most users who might want this _really_ want to use it to build something else, like encodable keypaths. Those features should be provided opt-in on a per-keypath or per-type basis, which will make it much more useful (and in the context of encodable keypaths specifically, eliminate major potential security issues). Such a feature should also include the option to let the user configure this string, so that it can remain backwards compatible with older versions of the program. + +### Make Keypath functions global to prevent the linker from stripping them + +This would also potentially make it feasible to change this proposal from implementing `debugDescription` to implementing `description`. + +This would also potentially bloat the binary and would increase linker times. It could also be a security issue, as `dlysm` would now be able to find these functions. + +I am not very knowledgeable about linkers or how typical Swift builds strip symbols, but I think it might be useful to have this as an option in some IDEs that build Swift programs. But that is beyond the scope of this proposal. + +## Future Directions + +### Add LLDB formatters/summaries + +This would be a good augmentation to this proposal, and might improve the developer experience, as there [might be debug metadata available to the debugger](https://forums.swift.org/t/pitch-add-customdebugdescription-conformance-to-anykeypath/58705/2) that is not available in the binary itself. + +However, I think it might be very difficult to implement this. I see two options: + +1. Implement a public reflection API for KeyPaths in the Swift Standard Library that the formatter can interact with from Python. +2. The formatter parses the raw memory of the KeyPath, essentially duplicating the code in `debugDescription`. + +I think (1) is overkill, especially considering the limited potential applications of this API beyond its use by the formatter. If it's possible to implement this as an `internal` function in the Swift stdlib then this is a much more attractive option. +From personal experience trying to parse KeyPath memory from outside the Standard Library, I think (2) would be extremely difficult to implement, and unsustainable to maintain considering that the [memory layout of KeyPaths](https://github.com/apple/swift/blob/main/docs/ABI/KeyPaths.md) is not ABI stable. + +### Make Keypath functions global in DEBUG builds only + +This may be necessary to allow `swift::lookupSymbol` to function correctly on Windows, Linux and other platforms that use COFF or ELF-like formats. + +## Acknowledgments + +Thanks to Joe Groff for answering several questions on the initial pitch document, and Slava Pestov for answering some questions about the logistics of pitching this. diff --git a/proposals/0370-pointer-family-initialization-improvements.md b/proposals/0370-pointer-family-initialization-improvements.md new file mode 100644 index 0000000000..9786754117 --- /dev/null +++ b/proposals/0370-pointer-family-initialization-improvements.md @@ -0,0 +1,1763 @@ +# Pointer Family Initialization Improvements and Better Buffer Slices + +* Proposal: [SE-0370](0370-pointer-family-initialization-improvements.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#41608](https://github.com/apple/swift/pull/41608) +* Review: ([first pitch](https://forums.swift.org/t/pitch-pointer-family-initialization-improvements/53168)) ([second pitch](https://forums.swift.org/t/pitch-buffer-partial-initialization-better-buffer-slices/53795)) ([third pitch](https://forums.swift.org/t/pitch-pointer-family-initialization-improvements-better-buffer-slices/55689)) ([review](https://forums.swift.org/t/se-0370-pointer-family-initialization-improvements-and-better-buffer-slices/59724)) ([acceptance](https://forums.swift.org/t/accepted-se-0370-pointer-family-initialization-improvements-and-better-buffer-slices/60007)) + +## Introduction + +The types in the `UnsafeMutablePointer` family typically require manual management of memory allocations, including the management of their initialization state. Unfortunately, not every relevant type in the family has the necessary functionality to fully manage the initialization state of the memory it represents. The states involved are, after allocation: + +1. Unbound and uninitialized (as returned from `UnsafeMutableRawPointer.allocate()`) +2. Bound to a type, and uninitialized (as returned from `UnsafeMutablePointer.allocate()`) +3. Bound to a type, and initialized + +Memory can be safely deallocated whenever it is uninitialized. + +We intend to round out initialization functionality for every relevant member of that family: `UnsafeMutablePointer`, `UnsafeMutableRawPointer`, `UnsafeMutableBufferPointer`, `UnsafeMutableRawBufferPointer`, `Slice` and `Slice`. The functionality will allow managing initialization state in a much greater variety of situations, including easier handling of partially-initialized buffers. + +## Motivation + +Memory allocated using `UnsafeMutablePointer`, `UnsafeMutableRawPointer`, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer` is passed to the user in an uninitialized state. In the general case, such memory needs to be initialized before it is used in Swift. Memory can be "initialized" or "uninitialized". We hereafter refer to this as a memory region's "initialization state". + +The methods of `UnsafeMutablePointer` that interact with initialization state are: + +- `func initialize(to value: Pointee)` +- `func initialize(repeating repeatedValue: Pointee, count: Int)` +- `func initialize(from source: UnsafePointer, count: Int)` +- `func assign(repeating repeatedValue: Pointee, count: Int)` +- `func assign(from source: UnsafePointer, count: Int)` +- `func move() -> Pointee` +- `func moveInitialize(from source: UnsafeMutablePointer, count: Int)` +- `func moveAssign(from source: UnsafeMutablePointer, count: Int)` +- `func deinitialize(count: Int) -> UnsafeMutableRawPointer` + +This is a fairly complete set. + +- The `initialize` functions change the state of memory locations from uninitialized to initialized, + then assign the corresponding value(s). +- The `assign` functions update the values stored at memory locations that have previously been initialized. +- `deinitialize` changes the state of a range of memory from initialized to uninitialized. +- The `move()` function deinitializes a memory location, then returns its current contents. +- The `move` prefix means that the `source` range of memory will be deinitialized after the function returns. + +Unfortunately, `UnsafeMutablePointer` is the only one of the list of types listed in the introduction to allow full control of initialization state, and this means that complex use cases such as partial initialization of a buffer become needlessly difficult. + +An example of partial initialization is the insertion of elements in the middle of a collection. This is one of the possible operations needed in an implementation of `RangeReplaceableCollection.replaceSubrange(_:with:)`. Given a `RangeReplaceableCollection` whose unique storage can be represented by a partially-initialized `UnsafeMutableBufferPointer`: + +```swift +mutating func replaceSubrange(_ subrange: Range, with newElements: C) + where C: Collection, Element == C.Element { + + // obtain unique storage as UnsafeMutableBufferPointer + let buffer: UnsafeMutableBufferPointer = self.myUniqueStorage() + let oldCount = self.count + let growth = newElements.count - subrange.count + let newCount = oldCount + growth + if growth > 0 { + assert(newCount < buffer.count) + let oldTail = subrange.upperBound..(_ subrange: Range, with newElements: C) + where C: Collection, Element == C.Element { + + // obtain unique storage as UnsafeMutableBufferPointer + let buffer: UnsafeMutableBufferPointer = self.myUniqueStorage() + let oldCount = self.count + let growth = newElements.count - subrange.count + let newCount = oldCount + growth + if growth > 0 { + assert(newCount < buffer.count) + let oldTail = subrange.upperBound..` and `Slice`. + + +## Proposed solution + +Note: the pseudo-diffs presented in this section denotes added functions with `+++` and renamed functions with `---`. Unmarked functions are unchanged. + +##### `UnsafeMutableBufferPointer` + +We propose to modify `UnsafeMutableBufferPointer` as follows: + +```swift +extension UnsafeMutableBufferPointer { + func initialize(repeating repeatedValue: Element) ++++ func initialize(from source: S) -> (unwritten: S.Iterator, index: Index) where S: Sequence, S.Element == Element +--- func initialize(from source: S) -> (S.Iterator, Index) where S: Sequence, S.Element == Element ++++ func initialize(fromContentsOf source: C) -> Index where C: Collection, C.Element == Element +--- func assign(repeating repeatedValue: Element) ++++ func update(repeating repeatedValue: Element) ++++ func update(from source: S) -> (unwritten: S.Iterator, index: Index) where S: Sequence, S.Element == Element ++++ func update(fromContentsOf source: C) -> Index where C: Collection, C.Element == Element ++++ func moveInitialize(fromContentsOf source: UnsafeMutableBufferPointer) -> Index ++++ func moveInitialize(fromContentsOf source: Slice) -> Index ++++ func moveUpdate(fromContentsOf source: Self) -> Index ++++ func moveUpdate(fromContentsOf source: Slice) -> Index ++++ func deinitialize() -> UnsafeMutableRawBufferPointer + ++++ func initializeElement(at index: Index, to value: Element) ++++ func moveElement(from index: Index) -> Element ++++ func deinitializeElement(at index: Index) +} +``` + + + +We would like to use the verb `update` instead of `assign`, in order to better communicate the intent of the API. It is currently a common programmer error to use one of the existing `assign` functions for uninitialized memory; using the verb `update` instead would express the precondition in the API name itself. + +The methods that initialize or update from a `Collection` will have consistent (and strict) semantics, and require the destination buffer (or slice) to have enough storage to copy the entire source collection, and then return the index in the destination that follows the last element copied. The storage available precondition will be strictly enforced, resulting in a consistent behaviour. The existing `Sequence` functions intended such a precondition, but in practice cannot enforce it. + +We also add functions to manipulate the initialization state for single elements of the buffer. There is no `buffer.updateElement(at index: Index, to value: Element)`, because it can already be expressed as `buffer[index] = value`. + +##### `UnsafeMutablePointer` + +The proposed modifications to `UnsafeMutablePointer` are renamings: + +```swift +extension UnsafeMutablePointer { + func initialize(to value: Pointee) + func initialize(repeating repeatedValue: Pointee, count: Int) + func initialize(from source: UnsafePointer, count: Int) +--- func assign(repeating repeatedValue: Pointee, count: Int) ++++ func update(repeating repeatedValue: Pointee, count: Int) +--- func assign(from source: UnsafePointer, count: Int) ++++ func update(from source: UnsafePointer, count: Int) + func move() -> Pointee + func moveInitialize(from source: UnsafeMutablePointer, count: Int) +--- func moveAssign(from source: UnsafeMutablePointer, count: Int) ++++ func moveUpdate(from source: UnsafeMutablePointer, count: Int) + func deinitialize(count: Int) -> UnsafeMutableRawPointer +} +``` + +The motivation for these renamings are explained above. + +##### `UnsafeMutableRawBufferPointer` + +We propose to add new functions to initialize memory referenced by `UnsafeMutableRawBufferPointer` instances. + +```swift +extension UnsafeMutableRawBufferPointer { + func initializeMemory( + as type: T.Type, repeating repeatedValue: T + ) -> UnsafeMutableBufferPointer + + func initializeMemory( + as type: S.Element.Type, from source: S + ) -> (unwritten: S.Iterator, initialized: UnsafeMutableBufferPointer) where S: Sequence + ++++ func initializeMemory( + as type: C.Element.Type, fromContentsOf source: C + ) -> UnsafeMutableBufferPointer where C: Collection + ++++ func moveInitializeMemory( + as type: T.Type, fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + ++++ func moveInitializeMemory( + as type: T.Type, fromContentsOf source: Slice> + ) -> UnsafeMutableBufferPointer +} +``` + +The first addition will initialize raw memory from a `Collection` and have similar behaviour as `UnsafeMutableBufferPointer.initialize(fromContentsOf:)`, described above. The other two initialize raw memory by moving data from another range of memory, leaving that other range of memory deinitialized. + +##### `UnsafeMutableRawPointer` + +```swift +extension UnsafeMutableRawPointer { ++++ func initializeMemory(as type: T.Type, to value: T) -> UnsafeMutablePointer + + func initializeMemory( + as type: T.Type, repeating repeatedValue: T, count: Int + ) -> UnsafeMutablePointer + + func initializeMemory( + as type: T.Type, from source: UnsafePointer, count: Int + ) -> UnsafeMutablePointer + + func moveInitializeMemory( + as type: T.Type, from source: UnsafeMutablePointer, count: Int + ) -> UnsafeMutablePointer +} +``` + +The addition here initializes a single value. + +##### Slices of `BufferPointer` + +We propose to extend slices of `Unsafe[Mutable][Raw]BufferPointer` with all the `BufferPointer`-specific methods of their `Base`. The following declarations detail the additions, which are all intended to behave exactly as the functions on the base BufferPointer types: + +```swift +extension Slice> { + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeBufferPointer) throws -> Result + ) rethrows -> Result +} +``` + +```swift +extension Slice> { + func initialize(repeating repeatedValue: Element) + + func initialize(from source: S) -> (unwritten: S.Iterator, index: Index) + where S.Element == Element + + func initialize(fromContentsOf source: C) -> Index + where C.Element == Element + + func update(repeating repeatedValue: Element) + + func update( + from source: S + ) -> (unwritten: S.Iterator, index: Index) where S.Element == Element + + func update( + fromContentsOf source: C + ) -> Index where C.Element == Element + + func moveInitialize(fromContentsOf source: UnsafeMutableBufferPointer) -> Index + func moveInitialize(fromContentsOf source: Slice>) -> Index + func moveUpdate(fromContentsOf source: UnsafeMutableBufferPointer) -> Index + func moveUpdate(fromContentsOf source: Slice>) -> Index + + func deinitialize() -> UnsafeMutableRawBufferPointer + + func initializeElement(at index: Index, to value: Element) + func moveElement(at index: Index) -> Element + func deinitializeElement(at index: Index) + + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result +} +``` + +Slices of `Unsafe[Mutable]RawBufferPointer` will add memory binding functions, memory initialization functions, and variants of `load`, `loadUnaligned` and `storeBytes`. +```swift +extension Slice { + func bindMemory(to type: T.Type) -> UnsafeBufferPointer + func assumingMemoryBound(to type: T.Type) -> UnsafeBufferPointer + + func withMemoryRebound( + to type: T.Type, _ body: (UnsafeBufferPointer) throws -> Result + ) rethrows -> Result + + func load(fromByteOffset offset: Int = 0, as type: T.Type) -> T + func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T +} +``` + +```swift +extension Slice { + func copyMemory(from source: UnsafeRawBufferPointer) + func copyBytes(from source: C) where C.Element == UInt8 + + func initializeMemory( + as type: T.Type, repeating repeatedValue: T + ) -> UnsafeMutableBufferPointer + + func initializeMemory( + as type: S.Element.Type, from source: S + ) -> (unwritten: S.Iterator, initialized: UnsafeMutableBufferPointer) + + func initializeMemory( + as type: C.Element.Type, fromContentsOf source: C + ) -> UnsafeMutableBufferPointer + + func moveInitializeMemory( + as type: T.Type, fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + + func moveInitializeMemory( + as type: T.Type, fromContentsOf source: Slice> + ) -> UnsafeMutableBufferPointer + + func bindMemory(to type: T.Type) -> UnsafeMutableBufferPointer + func assumingMemoryBound(to type: T.Type) -> UnsafeMutableBufferPointer + + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result + + func load(fromByteOffset offset: Int = 0, as type: T.Type) -> T + func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T + func storeBytes(of value: T, toByteOffset offset: Int = 0, as type: T.Type) +} +``` + +## Detailed design + +##### `UnsafeMutableBufferPointer` + +```swift +extension UnsafeMutableBufferPointer { + /// Initializes the buffer's memory with the given elements. + /// + /// Prior to calling the `initialize(from:)` method on a buffer, + /// the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer up to, but not including, + /// the returned index is initialized. + /// The buffer must contain sufficient memory to accommodate + /// `source.underestimatedCount`. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer, which is one past the last element written. + /// If `source` contains no elements, the returned index is equal to + /// the buffer's `startIndex`. If `source` contains an equal or greater number + /// of elements than the buffer can hold, the returned index is equal to + /// the buffer's `endIndex`. + /// + /// - Parameter source: A sequence of elements with which to initialize the + /// buffer. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and an index to the next uninitialized element in the buffer. + public func initialize( + from source: S + ) -> (unwritten: S.Iterator, index: Index) where S.Element == Element + + /// Initializes the buffer's memory with every element of the source. + /// + /// Prior to calling the `initialize(fromContentsOf:)` method on a buffer, + /// the memory referenced by the buffer must be uninitialized, + /// or the `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer up to, but not including, + /// the returned index is initialized. + /// The buffer must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A collection of elements to be used to + /// initialize the buffer's storage. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + func initialize(fromContentsOf source: C) -> Index + where C: Collection, C.Element == Element + + /// Updates every element of this buffer's initialized memory. + /// + /// The buffer’s memory must be initialized or the buffer's `Element` + /// must be a trivial type. + /// + /// - Note: All buffer elements must already be initialized. + /// + /// - Parameters: + /// - repeatedValue: The value used when updating this pointer's memory. + public func update(repeating repeatedValue: Element) + + /// Updates the buffer's initialized memory with the given elements. + /// + /// The buffer's memory must be initialized or the buffer's `Element` type + /// must be a trivial type. + /// + /// - Parameter source: A sequence of elements to be used to update + /// the buffer's contents. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and the index one past the last updated element in the buffer. + public func update(from source: S) -> (unwritten: S.Iterator, index: Index) + where S: Sequence, S.Element == Element + + /// Updates the buffer's initialized memory with every element of the source. + /// + /// Prior to calling the `update(fromContentsOf:)` method on a buffer, + /// the first `source.count` elements of the buffer's memory must be + /// initialized, or the buffer's `Element` type must be a trivial type. + /// The buffer must reference enough initialized memory to accommodate + /// `source.count` elements. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A collection of elements to be used to update + /// the buffer's contents. + /// - Returns: An index one past the index of the last element updated. + public func update(fromContentsOf source: C) -> Index + where C: Collection, C.Element == Element + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer, leaving the source memory + /// uninitialized and this buffer's memory initialized. + /// + /// Prior to calling the `moveInitialize(fromContentsOf:)` method on a buffer, + /// the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer up to, but not including, + /// the returned index is initialized. The memory referenced by + /// `source` is uninitialized after the function returns. + /// The buffer must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to copy. The memory + /// region underlying `source` must be initialized. The memory regions + /// referenced by `source` and this buffer may overlap. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + public func moveInitialize(fromContentsOf source: Self) -> Index + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer, leaving the source memory + /// uninitialized and this buffer's memory initialized. + /// + /// Prior to calling the `moveInitialize(fromContentsOf:)` method on a buffer, + /// the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer up to, but not including, + /// the returned index is initialized. The memory referenced by + /// `source` is uninitialized after the function returns. + /// The buffer must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to copy. The memory + /// region underlying `source` must be initialized. The memory regions + /// referenced by `source` and this buffer may overlap. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + public func moveInitialize(fromContentsOf source: Slice) -> Index + + /// Updates this buffer's initialized memory initialized memory by + /// moving every element from the source buffer slice, + /// leaving the source memory uninitialized. + /// + /// Prior to calling the `moveUpdate(fromContentsOf:)` method on a buffer, + /// the first `source.count` elements of the buffer's memory must be + /// initialized, or the buffer's `Element` type must be a trivial type. + /// The memory referenced by `source` is uninitialized after the function + /// returns. The buffer must reference enough initialized memory + /// to accommodate `source.count` elements. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to move. + /// The memory region underlying `source` must be initialized. The + /// memory regions referenced by `source` and this pointer must not overlap. + /// - Returns: An index one past the index of the last element updated. + public func moveUpdate(fromContentsOf source: Self) -> Index + + /// Updates this buffer's initialized memory initialized memory by + /// moving every element from the source buffer slice, + /// leaving the source memory uninitialized. + /// + /// Prior to calling the `moveUpdate(fromContentsOf:)` method on a buffer, + /// the first `source.count` elements of the buffer's memory must be + /// initialized, or the buffer's `Element` type must be a trivial type. + /// The memory referenced by `source` is uninitialized after the function + /// returns. The buffer must reference enough initialized memory + /// to accommodate `source.count` elements. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer slice containing the values to move. + /// The memory region underlying `source` must be initialized. The + /// memory regions referenced by `source` and this pointer must not overlap. + /// - Returns: An index one past the index of the last element updated. + public func moveUpdate(fromContentsOf source: Slice) -> Index + + /// Deinitializes every instance in this buffer. + /// + /// The region of memory underlying this buffer must be fully initialized. + /// After calling `deinitialize(count:)`, the memory is uninitialized, + /// but still bound to the `Element` type. + /// + /// - Note: All buffer elements must already be initialized. + /// + /// - Returns: A raw buffer to the same range of memory as this buffer. + /// The range of memory is still bound to `Element`. + public func deinitialize() -> UnsafeMutableRawBufferPointer + + /// Initializes the buffer element at `index` to the given value. + /// + /// The destination element must be uninitialized or the buffer's `Element` + /// must be a trivial type. After a call to `initialize(to:)`, the + /// memory underlying this element of the buffer is initialized. + /// + /// - Parameters: + /// - value: The value used to initialize the buffer element's memory. + /// - index: The index of the element to initialize + public func initializeElement(at index: Index, to value: Element) + + /// Retrieves and returns the buffer element at `index`, + /// leaving that element's memory uninitialized. + /// + /// The memory underlying buffer the element at `index` must be initialized. + /// After calling `moveElement(from:)`, the memory underlying the buffer + /// element at `index` is uninitialized, and still bound to type `Element`. + /// + /// - Parameters: + /// - index: The index of the buffer element to retrieve and deinitialize. + /// - Returns: The instance referenced by this index in this buffer. + public func moveElement(from index: Index) -> Element + + /// Deinitializes the buffer element at `index`. + /// + /// The memory underlying the buffer element at `index` must be initialized. + /// After calling `deinitializeElement()`, the memory underlying the buffer + /// element at `index` is uninitialized, and still bound to type `Element`. + /// + /// - Parameters: + /// - index: The index of the buffer element to deinitialize. + public func deinitializeElement(at index: Index) +} +``` + +##### `UnsafeMutablePointer` + +```swift +extension UnsafeMutablePointer { + /// Update this pointer's initialized memory with the specified number of + /// instances, copied from the given pointer's memory. + /// + /// The region of memory starting at this pointer and covering `count` + /// instances of the pointer's `Pointee` type must be initialized or + /// `Pointee` must be a trivial type. After calling + /// `update(from:count:)`, the region is initialized. + /// + /// - Note: Returns without performing work if `self` and `source` are equal. + /// + /// - Parameters: + /// - source: A pointer to at least `count` initialized instances of type + /// `Pointee`. The memory regions referenced by `source` and this + /// pointer may overlap. + /// - count: The number of instances to copy from the memory referenced by + /// `source` to this pointer's memory. `count` must not be negative. + public func update(from source: UnsafePointer, count: Int) + + /// Update this pointer's initialized memory by moving the specified number + /// of instances the source pointer's memory, leaving the source memory + /// uninitialized. + /// + /// The region of memory starting at this pointer and covering `count` + /// instances of the pointer's `Pointee` type must be initialized or + /// `Pointee` must be a trivial type. After calling + /// `moveUpdate(from:count:)`, the region is initialized and the memory + /// region `source..<(source + count)` is uninitialized. + /// + /// - Note: The source and destination memory regions must not overlap. + /// + /// - Parameters: + /// - source: A pointer to the values to be moved. The memory region + /// `source..<(source + count)` must be initialized. The memory regions + /// referenced by `source` and this pointer must not overlap. + /// - count: The number of instances to move from `source` to this + /// pointer's memory. `count` must not be negative. + public func moveUpdate(from source: UnsafeMutablePointer, count: Int) +``` + +##### `UnsafeMutableRawPointer` + +```swift +extension UnsafeMutableRawPointer { + /// Initializes the memory referenced by this pointer with the given value, + /// binds the memory to the value's type, and returns a typed pointer to the + /// initialized memory. + /// + /// The memory referenced by this pointer must be uninitialized or + /// initialized to a trivial type, and must be properly aligned for + /// accessing `T`. + /// + /// The following example allocates raw memory for one instance of `UInt`, + /// and then uses the `initializeMemory(as:to:)` method + /// to initialize the allocated memory. + /// + /// let bytePointer = UnsafeMutableRawPointer.allocate( + /// byteCount: MemoryLayout.stride, + /// alignment: MemoryLayout.alignment) + /// let int8Pointer = bytePointer.initializeMemory(as: UInt.self, to: 0) + /// + /// // After using 'int8Pointer': + /// int8Pointer.deallocate() + /// + /// After calling this method on a raw pointer `p`, the region starting at + /// `self` and continuing up to `p + MemoryLayout.stride` is bound + /// to type `T` and initialized. If `T` is a nontrivial type, you must + /// eventually deinitialize the memory in this region to avoid memory leaks. + /// + /// - Parameters: + /// - type: The type to which this memory will be bound. + /// - value: The value used to initialize this memory. + /// - Returns: A typed pointer to the memory referenced by this raw pointer. + public func initializeMemory(as type: T.Type, to value: T) -> UnsafeMutablePointer +} +``` + +##### `UnsafeMutableRawBufferPointer` + +```swift +extension UnsafeMutableRawBufferPointer { + /// Initializes the buffer's memory with every element of the source, + /// binding the initialized memory to the elements' type. + /// + /// When calling the `initializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer must be uninitialized, or initialized + /// to a trivial type. The buffer must reference enough memory to store + /// `source.count` elements, and its `baseAddress` must be properly aligned + /// for accessing `C.Element`. + /// + /// This method initializes the buffer with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A collection of elements to be used to + /// initialize the buffer's storage. + /// - Returns: A typed buffer containing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this buffer, and its count is equal to `source.count` + public func initializeMemory( + as: C.Element.Type, fromContentsOf source: C + ) -> UnsafeMutableBufferPointer + where C: Collection + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer, leaving the source memory + /// uninitialized and this buffer's memory initialized. + /// + /// When calling the `moveInitializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer must be uninitialized, or initialized + /// to a trivial type. The buffer must reference enough memory to store + /// `source.count` elements, and its `baseAddress` must be properly aligned + /// for accessing `C.Element`. After the method returns, + /// the memory referenced by the returned buffer is initialized and the + /// memory region underlying `source` is uninitialized. + /// + /// This method initializes the buffer with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A buffer containing the values to copy. + /// The memory region underlying `source` must be initialized. + /// The memory regions referenced by `source` and this buffer may overlap. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this buffer, and its count is equal to `source.count`. + public func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + + /// Moves every element of an initialized source buffer slice into the + /// uninitialized memory referenced by this buffer, leaving the source memory + /// uninitialized and this buffer's memory initialized. + /// + /// When calling the `moveInitializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer must be uninitialized, or initialized + /// to a trivial type. The buffer must reference enough memory to store + /// `source.count` elements, and its `baseAddress` must be properly aligned + /// for accessing `C.Element`. After the method returns, + /// the memory referenced by the returned buffer is initialized and the + /// memory region underlying `source` is uninitialized. + /// + /// This method initializes the buffer with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A buffer containing the values to copy. + /// The memory region underlying `source` must be initialized. + /// The memory regions referenced by `source` and this buffer may overlap. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this buffer, and its count is equal to `source.count`. + public func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: Slice> + ) -> UnsafeMutableBufferPointer +} +``` + + + +For `Slice` of typed buffers, the functions need to add an additional generic parameter, which is immediately restricted in the `where` clause. This is necessary because "parameterized extensions" are not yet a Swift feature. Eventually, these functions should be able to have exactly the same generic signatures as the counterpart function on their `UnsafeBufferPointer`-family base. This change will be neither source-breaking nor ABI-breaking. + +##### `Slice` + +```swift +extension Slice { + /// Executes the given closure while temporarily binding the memory referenced + /// by this buffer slice to the given type. + /// + /// Use this method when you have a buffer of memory bound to one type and + /// you need to access that memory as a buffer of another type. Accessing + /// memory as type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// The number of instances of `T` referenced by the rebound buffer may be + /// different than the number of instances of `Element` referenced by the + /// original buffer slice. The number of instances of `T` will be calculated + /// at runtime. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. Every instance of `Pointee` overlapping with a given + /// instance of `T` should have the same initialization state (i.e. + /// initialized or uninitialized.) Accessing a `T` whose underlying + /// `Pointee` storage is in a mixed initialization state shall be + /// undefined behaviour. + /// + /// Because this range of memory is no longer bound to its `Element` type + /// while the `body` closure executes, do not access memory using the + /// original buffer slice from within `body`. Instead, + /// use the `body` closure's buffer argument to access the values + /// in memory as instances of type `T`. + /// + /// After executing `body`, this method rebinds memory back to the original + /// `Element` type. + /// + /// - Note: Only use this method to rebind the buffer's memory to a type + /// that is layout compatible with the currently bound `Element` type. + /// The stride of the temporary type (`T`) may be an integer multiple + /// or a whole fraction of `Element`'s stride. + /// To bind a region of memory to a type that does not match these + /// requirements, convert the buffer to a raw buffer and use the + /// `bindMemory(to:)` method. + /// If `T` and `Element` have different alignments, this buffer's + /// `baseAddress` must be aligned with the larger of the two alignments. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// buffer. The type `T` must be layout compatible + /// with the pointer's `Element` type. + /// - body: A closure that takes a typed buffer to the + /// same memory as this buffer, only bound to type `T`. The buffer + /// parameter contains a number of complete instances of `T` based + /// on the capacity of the original buffer and the stride of `Element`. + /// The closure's buffer argument is valid only for the duration of the + /// closure's execution. If `body` has a return value, that value + /// is also used as the return value for the `withMemoryRebound(to:_:)` + /// method. + /// - buffer: The buffer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, _ body: (UnsafeBufferPointer) throws -> Result + ) rethrows -> Result + where Base == UnsafeBufferPointer +} +``` + +##### `Slice>` + +```swift +extension Slice { + /// Initializes every element in this buffer slice's memory to + /// a copy of the given value. + /// + /// The destination memory must be uninitialized or the buffer's `Element` + /// must be a trivial type. After a call to `initialize(repeating:)`, the + /// entire region of memory referenced by this buffer slice is initialized. + /// + /// - Parameter repeatedValue: The value with which to initialize this + /// buffer slice's memory. + public func initialize(repeating repeatedValue: Element) + where Base == UnsafeMutableBufferPointer + + /// Initializes the buffer slice's memory with the given elements. + /// + /// Prior to calling the `initialize(from:)` method on a buffer slice, + /// the memory it references must be uninitialized, + /// or the `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer slice up to, but not including, + /// the returned index is initialized. + /// The buffer must contain sufficient memory to accommodate + /// `source.underestimatedCount`. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer slice, which is one past the last element written. + /// If `source` contains no elements, the returned index is equal to + /// the buffer's `startIndex`. If `source` contains an equal or greater number + /// of elements than the buffer slice can hold, the returned index is equal to + /// the buffer's `endIndex`. + /// + /// - Parameter source: A sequence of elements with which to initialize the + /// buffer. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and an index to the next uninitialized element in the buffer. + public func initialize( + from source: S + ) -> (unwritten: S.Iterator, index: Index) + where S: Sequence, Base == UnsafeMutableBufferPointer + + /// Initializes the buffer slice's memory with with every element of the source. + /// + /// Prior to calling the `initialize(fromContentsOf:)` method on a buffer slice, + /// the memory it references must be uninitialized, + /// or the `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer slice up to, but not including, + /// the returned index is initialized. + /// + /// The returned index is the index of the next uninitialized element + /// in the buffer slice, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to + /// the buffer's `startIndex`. If `source` contains as many elements + /// as the buffer slice can hold, the returned index is equal to + /// to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A collection of elements to be used to + /// initialize the buffer's storage. + /// - Returns: An index to the next uninitialized element in the + /// buffer slice, or `endIndex`. + public func initialize( + fromContentsOf source: C + ) -> Index + where C : Collection, Base == UnsafeMutableBufferPointer + + /// Updates every element of this buffer slice's initialized memory. + /// + /// The buffer slice’s memory must be initialized or its `Element` + /// must be a trivial type. + /// + /// - Note: All buffer elements must already be initialized. + /// + /// - Parameters: + /// - repeatedValue: The value used when updating this pointer's memory. + public func update(repeating repeatedValue: Element) + where Base == UnsafeMutableBufferPointer + + /// Updates the buffer slice's initialized memory with the given elements. + /// + /// The buffer slice's memory must be initialized or its `Element` type + /// must be a trivial type. + /// + /// - Parameter source: A sequence of elements to be used to update + /// the buffer's contents. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and the index one past the last updated element in the buffer. + public func update( + from source: S + ) -> (unwritten: S.Iterator, index: Index) + where S: Sequence, Base == UnsafeMutableBufferPointer + + /// Updates the buffer slice's initialized memory with every element of the source. + /// + /// Prior to calling the `update(fromContentsOf:)` method on a buffer + /// slice, the first `source.count` elements of the referenced memory must be + /// initialized, or the buffer's `Element` type must be a trivial type. + /// The buffer must reference enough initialized memory to accommodate + /// `source.count` elements. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// slice can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A collection of elements to be used to update + /// the buffer's contents. + /// - Returns: An index one past the index of the last element updated. + public func update( + fromContentsOf source: C + ) -> Index + where C: Collection, Base == UnsafeMutableBufferPointer + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer slice, leaving the + /// source memory uninitialized and this buffer slice's memory initialized. + /// + /// Prior to calling the `moveInitialize(fromContentsOf:)` method on a + /// buffer slice, the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer slice up to, but not including, + /// the returned index is initialized. The memory referenced by + /// `source` is uninitialized after the function returns. + /// The buffer slice must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer slice, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// slice's `startIndex`. If `source` contains as many elements as the slice + /// can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to copy. + /// The memory region underlying `source` must be initialized. The memory + /// regions referenced by `source` and this buffer may overlap. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + public func moveInitialize( + fromContentsOf source: UnsafeMutableBufferPointer + ) -> Index + where Base == UnsafeMutableBufferPointer + + /// Moves every element of an initialized source buffer slice into the + /// uninitialized memory referenced by this buffer slice, leaving the + /// source memory uninitialized and this buffer slice's memory initialized. + /// + /// Prior to calling the `moveInitialize(fromContentsOf:)` method on a + /// buffer slice, the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer slice up to, but not including, + /// the returned index is initialized. The memory referenced by + /// `source` is uninitialized after the function returns. + /// The buffer slice must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer slice, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// slice's `startIndex`. If `source` contains as many elements as the slice + /// can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer slice containing the values to copy. + /// The memory region underlying `source` must be initialized. The memory + /// regions referenced by `source` and this buffer slice may overlap. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + public func moveInitialize( + fromContentsOf source: Slice> + ) -> Index + where Base == UnsafeMutableBufferPointer + + /// Updates this buffer slice's initialized memory initialized memory by + /// moving every element from the source buffer, + /// leaving the source memory uninitialized. + /// + /// The region of memory starting at the beginning of this buffer slice and + /// covering `source.count` instances of its `Element` type must be + /// initialized, or `Element` must be a trivial type. After calling + /// `moveUpdate(fromContentsOf:)`, + /// the region of memory underlying `source` is uninitialized. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// slice can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to move. + /// The memory region underlying `source` must be initialized. The memory + /// regions referenced by `source` and this buffer slice must not overlap. + /// - Returns: An index one past the index of the last element updated. + public func moveUpdate( + fromContentsOf source: UnsafeMutableBufferPointer + ) -> Index + where Base == UnsafeMutableBufferPointer + + /// Updates this buffer slice's initialized memory initialized memory by + /// moving every element from the source buffer slice, + /// leaving the source memory uninitialized. + /// + /// The region of memory starting at the beginning of this buffer slice and + /// covering `source.count` instances of its `Element` type must be + /// initialized, or `Element` must be a trivial type. After calling + /// `moveUpdate(fromContentsOf:)`, + /// the region of memory underlying `source` is uninitialized. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// slice can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer slice containing the values to move. + /// The memory region underlying `source` must be initialized. The memory + /// regions referenced by `source` and this buffer slice must not overlap. + /// - Returns: An index one past the index of the last element updated. + public func moveUpdate( + fromContentsOf source: Slice> + ) -> Index + where Base == UnsafeMutableBufferPointer + + /// Deinitializes every instance in this buffer slice. + /// + /// The region of memory underlying this buffer slice must be fully + /// initialized. After calling `deinitialize(count:)`, the memory + /// is uninitialized, but still bound to the `Element` type. + /// + /// - Note: All buffer elements must already be initialized. + /// + /// - Returns: A raw buffer to the same range of memory as this buffer. + /// The range of memory is still bound to `Element`. + public func deinitialize() -> UnsafeMutableRawBufferPointer + where Base == UnsafeMutableBufferPointer + + /// Initializes the element at `index` to the given value. + /// + /// The memory underlying the destination element must be uninitialized, + /// or `Element` must be a trivial type. After a call to `initialize(to:)`, + /// the memory underlying this element of the buffer slice is initialized. + /// + /// - Parameters: + /// - value: The value used to initialize the buffer element's memory. + /// - index: The index of the element to initialize + public func initializeElement(at index: Int, to value: Element) + where Base == UnsafeMutableBufferPointer + + /// Updates the initialized element at `index` to the given value. + /// + /// The memory underlying the destination element must be initialized, + /// or `Element` must be a trivial type. This method is equivalent to: + /// + /// self[index] = value + /// + /// - Parameters: + /// - value: The value used to update the buffer element's memory. + /// - index: The index of the element to update + public func updateElement(at index: Index, to value: Element) + where Base == UnsafeMutableBufferPointer + + /// Retrieves and returns the element at `index`, + /// leaving that element's underlying memory uninitialized. + /// + /// The memory underlying the element at `index` must be initialized. + /// After calling `moveElement(from:)`, the memory underlying this element + /// of the buffer slice is uninitialized, and still bound to type `Element`. + /// + /// - Parameters: + /// - index: The index of the buffer element to retrieve and deinitialize. + /// - Returns: The instance referenced by this index in this buffer. + public func moveElement(from index: Index) -> Element + where Base == UnsafeMutableBufferPointer + + /// Deinitializes the memory underlying the element at `index`. + /// + /// The memory underlying the element at `index` must be initialized. + /// After calling `deinitializeElement()`, the memory underlying this element + /// of the buffer slice is uninitialized, and still bound to type `Element`. + /// + /// - Parameters: + /// - index: The index of the buffer element to deinitialize. + public func deinitializeElement(at index: Base.Index) + where Base == UnsafeMutableBufferPointer + + /// Executes the given closure while temporarily binding the memory referenced + /// by this buffer slice to the given type. + /// + /// Use this method when you have a buffer of memory bound to one type and + /// you need to access that memory as a buffer of another type. Accessing + /// memory as type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// The number of instances of `T` referenced by the rebound buffer may be + /// different than the number of instances of `Element` referenced by the + /// original buffer slice. The number of instances of `T` will be calculated + /// at runtime. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. Every instance of `Pointee` overlapping with a given + /// instance of `T` should have the same initialization state (i.e. + /// initialized or uninitialized.) Accessing a `T` whose underlying + /// `Pointee` storage is in a mixed initialization state shall be + /// undefined behaviour. + /// + /// Because this range of memory is no longer bound to its `Element` type + /// while the `body` closure executes, do not access memory using the + /// original buffer slice from within `body`. Instead, + /// use the `body` closure's buffer argument to access the values + /// in memory as instances of type `T`. + /// + /// After executing `body`, this method rebinds memory back to the original + /// `Element` type. + /// + /// - Note: Only use this method to rebind the buffer's memory to a type + /// that is layout compatible with the currently bound `Element` type. + /// The stride of the temporary type (`T`) may be an integer multiple + /// or a whole fraction of `Element`'s stride. + /// To bind a region of memory to a type that does not match these + /// requirements, convert the buffer to a raw buffer and use the + /// `bindMemory(to:)` method. + /// If `T` and `Element` have different alignments, this buffer's + /// `baseAddress` must be aligned with the larger of the two alignments. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// buffer. The type `T` must be layout compatible + /// with the pointer's `Element` type. + /// - body: A closure that takes a ${Mutable.lower()} typed buffer to the + /// same memory as this buffer, only bound to type `T`. The buffer + /// parameter contains a number of complete instances of `T` based + /// on the capacity of the original buffer and the stride of `Element`. + /// The closure's buffer argument is valid only for the duration of the + /// closure's execution. If `body` has a return value, that value + /// is also used as the return value for the `withMemoryRebound(to:_:)` + /// method. + /// - buffer: The buffer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, _ body: (UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result + where Base == UnsafeMutableBufferPointer +} +``` + +##### `Slice` + +```swift +extension Slice where Base: UnsafeRawBufferPointer { + + /// Binds this buffer’s memory to the specified type and returns a typed buffer + /// of the bound memory. + /// + /// Use the `bindMemory(to:)` method to bind the memory referenced + /// by this buffer to the type `T`. The memory must be uninitialized or + /// initialized to a type that is layout compatible with `T`. If the memory + /// is uninitialized, it is still uninitialized after being bound to `T`. + /// + /// - Warning: A memory location may only be bound to one type at a time. The + /// behavior of accessing memory as a type unrelated to its bound type is + /// undefined. + /// + /// - Parameters: + /// - type: The type `T` to bind the memory to. + /// - Returns: A typed buffer of the newly bound memory. The memory in this + /// region is bound to `T`, but has not been modified in any other way. + /// The typed buffer references `self.count / MemoryLayout.stride` instances of `T`. + public func bindMemory(to type: T.Type) -> UnsafeBufferPointer + + /// Executes the given closure while temporarily binding the buffer to + /// instances of type `T`. + /// + /// Use this method when you have a buffer to raw memory and you need + /// to access that memory as instances of a given type `T`. Accessing + /// memory as a type `T` requires that the memory be bound to that type. + /// A memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. The memory underlying any individual instance of `T` + /// must have the same initialization state (i.e. initialized or + /// uninitialized.) Accessing a `T` whose underlying memory + /// is in a mixed initialization state shall be undefined behaviour. + /// + /// If the byte count of the original buffer is not a multiple of + /// the stride of `T`, then the re-bound buffer is shorter + /// than the original buffer. + /// + /// After executing `body`, this method rebinds memory back to its original + /// binding state. This can be unbound memory, or bound to a different type. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Note: A raw buffer may represent memory that has been bound to a type. + /// If that is the case, then `T` must be layout compatible with the + /// type to which the memory has been bound. This requirement does not + /// apply if the raw buffer represents memory that has not been bound + /// to any type. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This pointer must be a multiple of this type's alignment. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - buffer: The buffer temporarily bound to instances of `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, _ body: (UnsafeBufferPointer) throws -> Result + ) rethrows -> Result + + /// Returns a typed buffer to the memory referenced by this buffer, + /// assuming that the memory is already bound to the specified type. + /// + /// Use this method when you have a raw buffer to memory that has already + /// been bound to the specified type. The memory starting at this pointer + /// must be bound to the type `T`. Accessing memory through the returned + /// pointer is undefined if the memory has not been bound to `T`. To bind + /// memory to `T`, use `bindMemory(to:capacity:)` instead of this method. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Parameter to: The type `T` that the memory has already been bound to. + /// - Returns: A typed pointer to the same memory as this raw pointer. + public func assumingMemoryBound(to type: T.Type) -> UnsafeBufferPointer + + /// Returns a new instance of the given type, read from the + /// specified offset into the buffer pointer slice's raw memory. + /// + /// The memory at `offset` bytes into this buffer pointer slice + /// must be properly aligned for accessing `T` and initialized to `T` or + /// another type that is layout compatible with `T`. + /// + /// You can use this method to create new values from the underlying + /// buffer pointer's bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the next four bytes. + /// + /// let a = someBytes[0..<4].load(as: Int32.self) + /// let b = someBytes[4..<8].load(as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the + /// memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// - Parameters: + /// - offset: The offset into the slice's memory, in bytes, at + /// which to begin reading data for the new instance. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer + /// slice's memory. + public func load(fromByteOffset offset: Int = 0, as type: T.Type) -> T + + /// Returns a new instance of the given type, read from the + /// specified offset into the buffer pointer slice's raw memory. + /// + /// This function only supports loading trivial types. + /// A trivial type does not contain any reference-counted property + /// within its in-memory stored representation. + /// The memory at `offset` bytes into the buffer slice must be laid out + /// identically to the in-memory representation of `T`. + /// + /// You can use this method to create new values from the buffer pointer's + /// underlying bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the fourth through seventh bytes. + /// + /// let a = someBytes[..<4].loadUnaligned(as: Int32.self) + /// let b = someBytes[3...].loadUnaligned(as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the + /// memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// - Parameters: + /// - offset: The offset into the slice's memory, in bytes, at + /// which to begin reading data for the new instance. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer's + /// memory. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T +} +``` + +##### `Slice` + +```swift +extension Slice where Base == UnsafeMutableRawBufferPointer { + + /// Copies the bytes from the given buffer to this buffer slice's memory. + /// + /// If the `source.count` bytes of memory referenced by this buffer are bound + /// to a type `T`, then `T` must be a trivial type, the underlying pointer + /// must be properly aligned for accessing `T`, and `source.count` must be a + /// multiple of `MemoryLayout.stride`. + /// + /// The memory referenced by `source` may overlap with the memory referenced + /// by this buffer. + /// + /// After calling `copyMemory(from:)`, the first `source.count` bytes of + /// memory referenced by this buffer are initialized to raw bytes. If the + /// memory is bound to type `T`, then it contains values of type `T`. + /// + /// - Parameter source: A buffer of raw bytes. `source.count` must + /// be less than or equal to this buffer slice's `count`. + public func copyMemory(from source: UnsafeRawBufferPointer) + + /// Copies from a collection of `UInt8` into this buffer slice's memory. + /// + /// If the `source.count` bytes of memory referenced by this buffer are bound + /// to a type `T`, then `T` must be a trivial type, the underlying pointer + /// must be properly aligned for accessing `T`, and `source.count` must be a + /// multiple of `MemoryLayout.stride`. + /// + /// After calling `copyBytes(from:)`, the first `source.count` bytes of memory + /// referenced by this buffer are initialized to raw bytes. If the memory is + /// bound to type `T`, then it contains values of type `T`. + /// + /// - Parameter source: A collection of `UInt8` elements. `source.count` must + /// be less than or equal to this buffer slice's `count`. + public func copyBytes(from source: C) where C.Element == UInt8 + + /// Initializes the memory referenced by this buffer with the given value, + /// binds the memory to the value's type, and returns a typed buffer of the + /// initialized memory. + /// + /// The memory referenced by this buffer must be uninitialized or + /// initialized to a trivial type, and must be properly aligned for + /// accessing `T`. + /// + /// After calling this method on a raw buffer with non-nil `baseAddress` `b`, + /// the region starting at `b` and continuing up to + /// `b + self.count - self.count % MemoryLayout.stride` is bound to type `T` and + /// initialized. If `T` is a nontrivial type, you must eventually deinitialize + /// or move the values in this region to avoid leaks. If `baseAddress` is + /// `nil`, this function does nothing and returns an empty buffer pointer. + /// + /// - Parameters: + /// - type: The type to bind this buffer’s memory to. + /// - repeatedValue: The instance to copy into memory. + /// - Returns: A typed buffer of the memory referenced by this raw buffer. + /// The typed buffer contains `self.count / MemoryLayout.stride` + /// instances of `T`. + func initializeMemory( + as type: T.Type, repeating repeatedValue: T + ) -> UnsafeMutableBufferPointer + + /// Initializes the buffer's memory with the given elements, binding the + /// initialized memory to the elements' type. + /// + /// When calling the `initializeMemory(as:from:)` method on a buffer `b`, + /// the memory referenced by `b` must be uninitialized or initialized to a + /// trivial type, and must be properly aligned for accessing `S.Element`. + /// The buffer must contain sufficient memory to accommodate + /// `source.underestimatedCount`. + /// + /// This method initializes the buffer with elements from `source` until + /// `source` is exhausted or, if `source` is a sequence but not a + /// collection, the buffer has no more room for its elements. After calling + /// `initializeMemory(as:from:)`, the memory referenced by the returned + /// `UnsafeMutableBufferPointer` instance is bound and initialized to type + /// `S.Element`. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A sequence of elements with which to initialize the buffer. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and a typed buffer of the written elements. The returned + /// buffer references memory starting at the same base address as this + /// buffer. + public func initializeMemory( + as type: S.Element.Type, from source: S + ) -> (unwritten: S.Iterator, initialized: UnsafeMutableBufferPointer) + + /// Initializes the buffer slice's memory with every element of the source, + /// binding the initialized memory to the elements' type. + /// + /// When calling the `initializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer slice must be uninitialized, + /// or initialized to a trivial type. The buffer slice must reference + /// enough memory to store `source.count` elements, and it + /// must be properly aligned for accessing `C.Element`. + /// + /// This method initializes the buffer with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A collection of elements to be used to + /// initialize the buffer's storage. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this slice, and its count is equal to `source.count` + public func initializeMemory( + as type: C.Element.Type, + fromContentsOf source: C + ) -> UnsafeMutableBufferPointer + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer slice, leaving + /// the source memory uninitialized and this slice's memory initialized. + /// + /// When calling the `moveInitializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer slice must be uninitialized, + /// or initialized to a trivial type. The buffer slice must reference + /// enough memory to store `source.count` elements, and it must be properly + /// aligned for accessing `C.Element`. After the method returns, + /// the memory referenced by the returned buffer is initialized and the + /// memory region underlying `source` is uninitialized. + /// + /// This method initializes the buffer slice with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer slice, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A buffer referencing the values to copy. + /// The memory region underlying `source` must be initialized. + /// The memory regions referenced by `source` and this slice may overlap. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this slice, and its count is equal to `source.count`. + public func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + + /// Moves every element from an initialized source buffer slice into the + /// uninitialized memory referenced by this buffer slice, leaving + /// the source memory uninitialized and this slice's memory initialized. + /// + /// When calling the `moveInitializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer slice must be uninitialized, + /// or initialized to a trivial type. The buffer slice must reference + /// enough memory to store `source.count` elements, and it must be properly + /// aligned for accessing `C.Element`. After the method returns, + /// the memory referenced by the returned buffer is initialized and the + /// memory region underlying `source` is uninitialized. + /// + /// This method initializes the buffer slice with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type of `T` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer slice, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A buffer referencing the values to copy. + /// The memory region underlying `source` must be initialized. + /// The memory regions referenced by `source` and this buffer may overlap. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this slice, and its count is equal to `source.count`. + public func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: Slice> + ) -> UnsafeMutableBufferPointer + + /// Binds this buffer’s memory to the specified type and returns a typed buffer + /// of the bound memory. + /// + /// Use the `bindMemory(to:)` method to bind the memory referenced + /// by this buffer to the type `T`. The memory must be uninitialized or + /// initialized to a type that is layout compatible with `T`. If the memory + /// is uninitialized, it is still uninitialized after being bound to `T`. + /// + /// - Warning: A memory location may only be bound to one type at a time. The + /// behavior of accessing memory as a type unrelated to its bound type is + /// undefined. + /// + /// - Parameters: + /// - type: The type `T` to bind the memory to. + /// - Returns: A typed buffer of the newly bound memory. The memory in this + /// region is bound to `T`, but has not been modified in any other way. + /// The typed buffer references `self.count / MemoryLayout.stride` instances of `T`. + public func bindMemory(to type: T.Type) -> UnsafeMutableBufferPointer + + /// Executes the given closure while temporarily binding the buffer to + /// instances of type `T`. + /// + /// Use this method when you have a buffer to raw memory and you need + /// to access that memory as instances of a given type `T`. Accessing + /// memory as a type `T` requires that the memory be bound to that type. + /// A memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. The memory underlying any individual instance of `T` + /// must have the same initialization state (i.e. initialized or + /// uninitialized.) Accessing a `T` whose underlying memory + /// is in a mixed initialization state shall be undefined behaviour. + /// + /// If the byte count of the original buffer is not a multiple of + /// the stride of `T`, then the re-bound buffer is shorter + /// than the original buffer. + /// + /// After executing `body`, this method rebinds memory back to its original + /// binding state. This can be unbound memory, or bound to a different type. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Note: A raw buffer may represent memory that has been bound to a type. + /// If that is the case, then `T` must be layout compatible with the + /// type to which the memory has been bound. This requirement does not + /// apply if the raw buffer represents memory that has not been bound + /// to any type. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This pointer must be a multiple of this type's alignment. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - buffer: The buffer temporarily bound to instances of `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, _ body: (UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result + + /// Returns a typed buffer to the memory referenced by this buffer, + /// assuming that the memory is already bound to the specified type. + /// + /// Use this method when you have a raw buffer to memory that has already + /// been bound to the specified type. The memory starting at this pointer + /// must be bound to the type `T`. Accessing memory through the returned + /// pointer is undefined if the memory has not been bound to `T`. To bind + /// memory to `T`, use `bindMemory(to:capacity:)` instead of this method. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Parameter to: The type `T` that the memory has already been bound to. + /// - Returns: A typed pointer to the same memory as this raw pointer. + public func assumingMemoryBound(to type: T.Type) -> UnsafeMutableBufferPointer + + /// Returns a new instance of the given type, read from the + /// specified offset into the buffer pointer slice's raw memory. + /// + /// The memory at `offset` bytes into this buffer pointer slice + /// must be properly aligned for accessing `T` and initialized to `T` or + /// another type that is layout compatible with `T`. + /// + /// You can use this method to create new values from the underlying + /// buffer pointer's bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the next four bytes. + /// + /// let a = someBytes[0..<4].load(as: Int32.self) + /// let b = someBytes[4..<8].load(as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the + /// memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// - Parameters: + /// - offset: The offset into the slice's memory, in bytes, at + /// which to begin reading data for the new instance. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer + /// slice's memory. + public func load(fromByteOffset offset: Int = 0, as type: T.Type) -> T + + /// Returns a new instance of the given type, read from the + /// specified offset into the buffer pointer slice's raw memory. + /// + /// This function only supports loading trivial types. + /// A trivial type does not contain any reference-counted property + /// within its in-memory stored representation. + /// The memory at `offset` bytes into the buffer slice must be laid out + /// identically to the in-memory representation of `T`. + /// + /// You can use this method to create new values from the buffer pointer's + /// underlying bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the fourth through seventh bytes. + /// + /// let a = someBytes[..<4].loadUnaligned(as: Int32.self) + /// let b = someBytes[3...].loadUnaligned(as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the + /// memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// - Parameters: + /// - offset: The offset into the slice's memory, in bytes, at + /// which to begin reading data for the new instance. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer's + /// memory. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T + + /// Stores a value's bytes into the buffer pointer slice's raw memory at the + /// specified byte offset. + /// + /// The type `T` to be stored must be a trivial type. The memory must also be + /// uninitialized, initialized to `T`, or initialized to another trivial + /// type that is layout compatible with `T`. + /// + /// The memory written to must not extend beyond + /// the memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is + /// initialized to the raw bytes of `value`. If the memory is bound to a + /// type `U` that is layout compatible with `T`, then it contains a value of + /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the + /// bound type of the memory. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// If you need to store into memory a copy of a value of a type that isn't + /// trivial, you cannot use the `storeBytes(of:toByteOffset:as:)` method. + /// Instead, you must know either initialize the memory or, + /// if you know the memory was already bound to `type`, assign to the memory. + /// + /// - Parameters: + /// - value: The value to store as raw bytes. + /// - offset: The offset in bytes into the buffer pointer slice's memory + /// to begin writing bytes from the value. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + public func storeBytes(of value: T, toByteOffset offset: Int = 0, as type: T.Type) +} +``` + +## Source compatibility + +This proposal consists mostly of additions, which are by definition source compatible. + +The proposal includes the renaming of four existing functions from `assign` to `update`. The existing function names would be deprecated, producing a warning. A fixit will support an easy transition to the renamed versions of these functions. + +The proposal also includes the addition of labels to the tuple return value of one function. This is technically source breaking. We have tested the change against the source compatibility suite and found no issue. + +## Effect on ABI stability + +The functions proposed here are generally small wrappers around existing functionality. They are implemented as `@_alwaysEmitIntoClient` functions, which means they have no ABI impact. + +The renamed functions can reuse the existing symbol, while the deprecated functions can forward using an `@_alwaysEmitIntoClient` stub to support the functionality under its previous name. The renamings would therefore have no ABI impact. + +In the case of the added tuple labels, we can avoid an ABI impact by reusing the mangled symbol of the previously-existing function that had an unlabeled return tuple type. + +## Effect on API resilience + +All functionality implemented as `@_alwaysEmitIntoClient` will back-deploy. Renamed functions that reuse a previous symbol will also back-deploy. + + +## Alternatives considered + +##### Single element update functions + +An earlier version of this proposal included single-element update functions, `UnsafeMutablePointer.update(to:)` and `UnsafeMutableBufferPointer.updateElement(at:to:)`. These are synonyms for the setters of `UnsafeMutablePointer.pointee` and `UnsafeMutableBufferPointer.subscript(_ i: Index)`, respectively. They were intended to improve the documentation for that operation, in particular the often overlooked initialization requirement. + +##### Renaming `assign` to `update` + +The renaming of `assign` to `update` could be omitted entirely, although we believe the word "update" communicates the API's intent much better than does the word "assign". In _The Swift Programming Language_ (_TSPL_,) the `=` symbol is named "the assignment operator", and its function is described as to either "initialize" or "update" a value. The current name (`assign`) seemingly conflates the two roles of `=` as described in *TSPL*, while the proposed name (`update`) builds on _TSPL_. + +There are only four current symbols to be renamed by this proposal, and their replacements are easily migrated by a fixit. For context, this renaming would change only 6 lines of code in the standard library, outside of the function definitions. If the renaming is omitted, the four new functions proposed in the family should use the name `assign` as well. The two single-element versions would be `assign(_ value:)` and `assignElement(at:_ value:)`. + +##### Element-by-element copies from `Collection` inputs + +The initialization and updating functions that copy from `Collection` inputs use the argument label `fromContentsOf`. This is a different label than used by the pre-existing functions that copy from `Sequence` inputs. We could use the same argument label (`from`) as with the `Sequence` inputs, but that would mean that we must return the `Iterator` for the `Collection` versions, and that is not necessarily desirable, especially if a particular `Iterator` cannot be copied cheaply. If we used the same argument label (`from`) and did not return `Iterator`, then the `Sequence` and `Collection` versions of the `initialize(from:)` would be overloaded by their return type, and that would be source-breaking: +an existing use of the current function that doesn't destructure the returned tuple on assignment could now pick up the `Collection` overload, which would have a return value incompatible with the subsequent code which assumes that the return value is of type `(Iterator, Index)`. + +##### Returned tuple labels + +One of the pre-existing returned tuples does not have element labels, and the original version of the proposal did not change that. New labels are now proposed for this case. This is technically source-breaking, in that if existing source uses exactly the proposed labels in a different position, then the returned tuple value would be shuffled. The chosen labels have sufficiently pointed names that the risk is very small. + +## Acknowledgments + +[Diana Ma](https://github.com/tayloraswift) (aka [Taylor Swift](https://forums.swift.org/u/taylorswift/summary))'s initial versions of the pitch that became SE-0184 included more functions to manipulate initialization state. These were deferred, but much of the deferred functionality has not been pitched again until now. + +Members of the Swift Standard Library team for valuable discussions. diff --git a/proposals/0371-isolated-synchronous-deinit.md b/proposals/0371-isolated-synchronous-deinit.md new file mode 100644 index 0000000000..00b88aca31 --- /dev/null +++ b/proposals/0371-isolated-synchronous-deinit.md @@ -0,0 +1,531 @@ +# Isolated synchronous deinit + +* Proposal: [SE-0371](0371-isolated-synchronous-deinit.md) +* Author: [Mykola Pokhylets](https://github.com/nickolas-pohilets) +* Review Manager: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Status: **Implemented (Swift 6.2)** +* Implementation: [apple/swift#60057](https://github.com/apple/swift/pull/60057) +* Review: ([first pitch](https://forums.swift.org/t/isolated-synchronous-deinit/58177)) ([first review](https://forums.swift.org/t/se-0371-isolated-synchronous-deinit/59754)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0371-isolated-synchronous-deinit/60060)) ([discussion](https://forums.swift.org/t/isolated-synchronous-deinit-2/62565)) ([second pitch](https://forums.swift.org/t/pitch-2-se-0371-isolated-async-deinit/64836)) ([sub-pitch](https://forums.swift.org/t/sub-pitch-task-local-values-in-isolated-synchronous-deinit-and-async-deinit/70060)) ([second review](https://forums.swift.org/t/second-review-se-0371-isolated-synchronous-deinit/73406)) ([accepted with modifications](https://forums.swift.org/t/accepted-with-modifications-se-0371-isolated-synchronous-deinit/74042)) + +## Introduction + +This feature allows `deinit`'s of actors and global-actor isolated classes to access non-sendable isolated state, lifting restrictions imposed by [SE-0327](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md). This is achieved by providing runtime support for hopping onto the relevant actor's executor before executing the `deinit` body. + +## Motivation + +Restrictions imposed by [SE-0327](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md) reduce the usefulness of explicit `deinit`s in actors and global actor isolated types. Workarounds for these limitations may involve creation of `close()`-like methods, or even manual reference counting if the API should be able to serve several clients. + +In cases when `deinit` belongs to a subclass of `UIView` or `UIViewController` which are known to call `dealloc` on the main thread, developers may be tempted to silence the diagnostic by adopting `@unchecked Sendable` in types that are not actually sendable. This undermines concurrency checking by the compiler, and may lead to data races when using incorrectly marked types in other places. + +While this proposal introduces additional control over deinit execution semantics that are necessary in some situations, it also introduces a certain amount of non-determinism to deinitializer execution. Because isolated deinits must potentially be enqueued and executed "later" rather than directly inline, this can cause subtle timing issues in resource reclamation. For example, APIs which require quick and predictable resource cleanup, such as scarce resources such as e.g. file descriptors or connections, should not be managed using isolated deinitializers, as the exact timing of when the resource would be released is non-deterministic, which can lead to subtle timing and resource starvation issues. Instead, for types which require tight control over lifetime and cleanup, one should still prefer using "with-style" APIs (`await withResource { resource }`), explicit `await resource.close()` or non-copyable & non-escapable types. + +## Proposed solution + +Allow users to specify a `deinit` as `isolated deinit`, indicating that the execution of the `deinit` body, destruction of the stored properties and object deallocation should be scheduled on the executor (either that of the actor itself or that of the relevant global actor), if needed. + +Let's consider [examples from SE-0327](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md#data-races-in-deinitializers): + +In the case of several instances with shared data isolated on a common actor, the problem is completely eliminated: + +```swift +class NonSendableAhmed { + var state: Int = 0 +} + +@MainActor +class Maria { + let friend: NonSendableAhmed + + init() { + self.friend = NonSendableAhmed() + } + + init(sharingFriendOf otherMaria: Maria) { + // While the friend is non-Sendable, this initializer and + // and the otherMaria are isolated to the MainActor. That is, + // they share the same executor. So, it's OK for the non-Sendable value + // to cross between otherMaria and self. + self.friend = otherMaria.friend + } + + isolated deinit { + // Used to be a potential data race. Now, deinit is also + // isolated on the MainActor, so this code is perfectly + // correct. + friend.state += 1 + } +} + +func example() async { + let m1 = await Maria() + let m2 = await Maria(sharingFriendOf: m1) + doSomething(m1, m2) +} +``` + +In the case of escaping `self`, the race condition is eliminated but the problem of dangling reference remains. + +```swift +actor Clicker { + var count: Int = 0 + + func click(_ times: Int) { + for _ in 0..Objective-C boundary in either direction. + +This is still sufficient to create ObjC subclasses in runtime, as KVO does. Or swizzle `dealloc` of the most derived class of an instance. But swizzling `dealloc` of intermediate Swift classes might not work as expected. + +### Isolated synchronous deinit of default actors + +When deinitializing an instance of default actor, runtime attempts to take actor's lock and execute deinit on the current thread. If previous executor was another default actor, it remains locked. So potentially multiple actors can be locked at the same time. This does not lead to deadlocks, because (1) lock is acquired conditionally, without waiting; and (2) object cannot be deinitializer twice, so graph of the deinit calls has no cycles. + +### Interaction with distributed actors + +A `deinit` declared in the code of the distributed actor applies only to the local actor and can be isolated as described above. Remote proxy has an implicit compiler-generated synchronous `deinit` which is never isolated. + +### Interaction with task executor preference + +Ad-hoc job created for isolated synchronous deinit is executed outside a task, so task executor preference does not apply. + +### Task-local values + +Task-local values set by the task/thread that performed last release are blocked inside isolated deinit. +Attempting to read such task-local inside isolated deinit will return a default value without any runtime warnings. +Task-local values set inside the body of the isolated deinit are visible for the corresponding scope. + +This behavior ensures that isolated deinit behaves the same way both when running inline and when hopping, without high runtime costs for copying task-local values. + +The point of the last release of the object can be hard to predict and can be changed by optimizations, leading to different behavior between debug and release builds. +Because of this, developers are discouraged from depending on the set of task-local values available at the point of the last release. +Instead of using task-local values, developers are advised to inject dependencies into deinit using the object's stored properties. +This advice applies to non-isolated deinit as well, but this proposal does not change the behavior of the non-isolated deinit. + +Note that any existing hopping in overridden `retain`/`release` for UIKit classes is unlikely to be aware of task-local values. + +## Source compatibility + +This proposal makes previously invalid code valid. + +## Effect on ABI stability + +This proposal does not change the ABI of existing language features, but does introduce new runtime functions. + +## Effect on API resilience + +Isolation attributes of the `deinit` become part of the public API, but they matter only when inheriting from the class. + +Any changes to the isolation of `deinit` of non-open classes are allowed. + +For open non-@objc classes, it is allowed to change synchronous `deinit` from isolated to nonisolated. Any non-recompiled subclasses will keep calling `deinit` of the superclass on the original actor. Changing `deinit` from nonisolated to isolated or changing identity of the isolating actor is a breaking change. + +For open @objc classes, any change in isolation of the synchronous `deinit` is a breaking change, even changing from isolated to nonisolated. This removes symbol for `__isolated_deallocating_deinit` and clients will fail to link with new framework version. See also [Interaction with ObjC runtime](#interaction-with-objc-runtime). + + + + + + + + + + + + + + + + + + + + + + +
Change|opennon-open
| + @objc + + non-@objc +
remove isolation|breakingokok
add isolation|breakingbreakingok
change actor|breakingbreakingok
+ +Adding an isolation annotation to a `dealloc` method of an imported Objective-C class is a breaking change. Existing Swift subclasses may have `deinit` isolated on different actor, and without recompilation will be calling `[super deinit]` on that actor. When recompiled, subclasses isolating on a different actor will produce a compilation error. Subclasses that had a non-isolated `deinit` (or a `deinit` isolated on the same actor) remain ABI compatible. It is possible to add isolation annotation to UIKit classes now, because currently all their subclasses have nonisolated `deinit`s. + +Removing an isolation annotation from the `dealloc` method (together with `retain`/`release` overrides) is a breaking change. Any existing subclasses would be type-checked as isolated but compiled without isolation thunks. After changes in the base class, subclass `deinit`s could be called on the wrong executor. + +If isolated deinit need to be suppressed in `.swiftinterface` for compatibility with older compilers, then `open` classes are emitted as `public` to prevent subclassing. + +## Future Directions + +### Implicit asynchronous `deinit` + +Currently, if users need to initiate an asynchronous operation from `deinit`, they need to manually start a task. This requires copying all the needed data from the object, which can be tedious and error-prone. If some data is not copied explicitly, `self` will be captured implicitly, leading to a fatal error in runtime. + +```swift +actor Service { + func shutdown() {} +} + +@MainActor +class ViewModel { + let service: Service + + deinit { + // Incorrect: + _ = Task { await service.shutdown() } + + // Corrected version: + _ = Task { [service] in await service.shutdown() } + } +} +``` + +If almost every instance property is copied, then it would be more efficient to reuse original object as a task closure context and make `deinit` asynchronous: + +```swift + ... + deinit async { + await service.shutdown() + + // Destroy stored properties and deallocate memory + // after asynchronous shutdown is complete + } +} +``` + +Similarly to this proposal, `__deallocating_deinit` can be used as a thunk that starts an unstructured task for executing async deinit. But this is out of scope of this proposal. + +### Linear types + +Invoking sequential async cleanup is a suspension point, and needs to be marked with `await`. Explicit method calls fit this role better than implicitly invoked `deinit`. But using such methods can be error-prone without compiler checks that cleanup method is called exactly once on all code paths. Move-only types help to ensure that cleanup method is called **at most once**. Linear types help to ensure that cleanup method is called **exactly once**. + +```swift +@linear // like @moveonly, but consumption is mandatory +struct Connection { + // acts as a named explicit async deinit + consuming func close() async { + ... + } +} + +func communicate() async { + let c = Connection(...) + // error: value of linear type is not consumed +} +``` + +### Improving de-virtualization and inlining of the executor access. + +Consider the following example: + +```swift +import Foundation + +@_silgen_name("do_it") +@MainActor func doIt() + +public class Foo { + @MainActor + public func foo() async { + doIt() + } + @MainActor + deinit {} +} +``` + +Currently both the `foo()` and `deinit` entry points produce two calls to access the `MainActor.shared.unownedExecutor`, with the second one even using dynamic dispatch. These two calls could be replaced with a single call to the statically referenced `swift_task_getMainExecutor()`. + +```llvm +%1 = tail call swiftcc %swift.metadata_response @"type metadata accessor for Swift.MainActor"(i64 0) #6 +%2 = extractvalue %swift.metadata_response %1, 0 +%3 = tail call swiftcc %TScM* @"static Swift.MainActor.shared.getter : Swift.MainActor"(%swift.type* swiftself %2) +%4 = tail call i8** @"lazy protocol witness table accessor for type Swift.MainActor and conformance Swift.MainActor : Swift.Actor in Swift"() #6 +%5 = bitcast %TScM* %3 to %objc_object* +%6 = tail call swiftcc { i64, i64 } @"dispatch thunk of Swift.Actor.unownedExecutor.getter : Swift.UnownedSerialExecutor"(%objc_object* swiftself %5, %swift.type* %2, i8** %4) +``` + +### Improving extended stack trace support + +Developers who put breakpoints in the isolated deinit might want to see the call stack that led to the last release of the object. Currently, if switching of executors was involved, the release call stack won't be shown in the debugger. + +### Implementing API for synchronously scheduling arbitrary work on the actor + +Added runtime function has calling convention optimized for the `deinit` use case, but using a similar runtime function with a slightly different signature, one could implement an API for synchronously scheduling arbitrary work on the actor: + +```swift +extension Actor { + /// Adds a job to the actor queue that calls `work` passing `self` as an argument. + nonisolated func enqueue(_ work: __owned @Sendable @escaping (isolated Self) -> Void) + + /// If actor's executor is already the current one - executes work immediately + /// Otherwise adds a job to the actor's queue. + nonisolated func executeOrEnqueue(_ work: __owned @Sendable @escaping (isolated Self) -> Void) +} + +actor MyActor { + var k: Int = 0 + func inc() { k += 1 } +} + +let a = MyActor() +a.enqueue { aIsolated in + aIsolated.inc() // no await +} +``` + +## Alternatives considered + +### Placing hopping logic in `swift_release()` instead. + +`UIView` and `UIViewController` implement hopping to the main thread by overriding the `release` method. But in Swift there are no vtable/wvtable slots for releasing, and adding them would also affect a lot of code that does not need isolated deinit. + +### Copy task-local values when hopping by default + +This comes with a performance cost, which is unlikely to be beneficial to most of the users. +Leaving behavior of the task-locals undefined allows to potentially change it in the future, after getting more feedback from the users. + +### Keeping behavior of task-local values undefined + +Approach of 'make no promises' is likely to result in users inadvertently relying on implementation details which would turn out to be difficult to change later. + +### Implicitly propagate isolation to synchronous `deinit`. + +This would be a source-breaking change. + +Majority of the `deinit`'s are implicitly synthesized by the compiler and only release stored properties. Global open source search in Sourcegraph, gives is 77.5k deinit declarations for 2.2m classes - 3.5%. Release can happen from any executor/thread and does not need isolation. Isolating implicit `deinit`s would come with a major performance cost. Providing special rules for propagating isolation to synchronous `deinit` unless it is implicit, would complicate propagation rules. diff --git a/proposals/0372-document-sorting-as-stable.md b/proposals/0372-document-sorting-as-stable.md new file mode 100644 index 0000000000..2449d0ae3c --- /dev/null +++ b/proposals/0372-document-sorting-as-stable.md @@ -0,0 +1,77 @@ +# Document Sorting as Stable + +* Proposal: [SE-0372](0372-document-sorting-as-stable.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift PR #60936](https://github.com/apple/swift/pull/60936) +* Review: ([pitch](https://forums.swift.org/t/pitch-document-sorting-as-stable/59880)) ([review](https://forums.swift.org/t/se-0372-document-sorting-as-stable/60165)) ([acceptance](https://forums.swift.org/t/accepted-se-0372-document-sorting-as-stable/60425)) + +## Introduction + +Swift's sorting algorithm was changed to be stable before Swift 5, but we've never updated the documentation to provide that guarantee. Let's commit to the sorting algorithm being stable so that people can rely on that behavior. + +## Motivation + +A *stable sort* is a sort that keeps the original relative order for any elements that compare as equal or unordered. For example, given this list of players that are already sorted by last name, a sort by first name preserves the original order of the two players named "Ashley": + +```swift +var roster = [ + Player(first: "Sam", last: "Coffey"), + Player(first: "Ashley", last: "Hatch"), + Player(first: "Kristie", last: "Mewis"), + Player(first: "Ashley", last: "Sanchez"), + Player(first: "Sophia", last: "Smith"), +] + +roster.sort(by: { $0.first < $1.first }) +// roster == [ +// Player(first: "Ashley", last: "Hatch"), +// Player(first: "Ashley", last: "Sanchez"), +// Player(first: "Kristie", last: "Mewis"), +// Player(first: "Sam", last: "Coffey"), +// Player(first: "Sophia", last: "Smith"), +// ] +``` + +For users who are unaware that many sorting algorithms aren't stable, an unstable sort can be surprising. Preserving relative order is an expectation set by software like spreadsheets, where sorting by one column, and then another, is a way to complete a sort based on multiple properties. + +Sort stability isn't always observable. When a collection is sorted based on the elements' `Comparable` conformance, like sorting an array of integers, "unordered" elements are typically indistinguishable. In general, sort stability is important when elements are sorted based on a subset of their properties. + +The standard library `sort()` has long been stable, but the documentation explicitly [doesn't make this guarantee](https://github.com/apple/swift/blob/release/5.7/stdlib/public/core/Sort.swift#L40-L41): + +> The sorting algorithm is not guaranteed to be stable. A stable sort preserves the relative order of elements that compare as equal. + +This status quo is a problem — developers who are aware of what stability is cannot rely on the current behavior, and developers who are unaware of stability could be surprised by unexpected bugs if stability were to disappear. Guaranteeing stability would resolve both of these issues. + +## Proposed solution + +Let's change the documentation! Since all current versions of the Swift runtime include a stable sort (which was introduced before ABI stability), this change can be made to the standard library documentation only: + +```diff +- /// The sorting algorithm is not guaranteed to be stable. A stable sort ++ /// The sorting algorithm is guaranteed to be stable. A stable sort + /// preserves the relative order of elements that compare as equal. +``` + +## Source compatibility + +This change codifies the existing standard library behavior, so it is compatible with all existing source code. + +## Effect on ABI stability + +The change to make sorting stable was implemented before ABI stability, so all ABI-stable versions of Swift already provide this behavior. + +## Effect on API resilience + +Making this guarantee explicit requires that any changes to the sort algorithm maintain stability. + +## Alternatives considered + +### Providing an `unstableSort()` + +Discussing the *stability* of the current sort naturally brings up the question of providing an alternative sort that is *unstable*. An unstable sort by itself, however, doesn't provide any specific benefit to users — no one is asking for a sort that mixes up equivalent elements! Instead, users could be interested in sort algorithms that have other characteristics, such as using only an array's existing allocation, that are much faster to implement without guaranteeing stability. If and when proposals for those sort algorithms are introduced, the lack of stability can be addressed through documentation and/or API naming, and having the default sort be stable is still valuable for the reasons listed above. + +## Future directions + +There are a variety of other sorting-related improvements that could be interesting to pursue, including key-path or function-based sorting, sorted collection types or protocols, sort descriptors, and more. These ideas can be explored in future pitches and proposals. diff --git a/proposals/0373-vars-without-limits-in-result-builders.md b/proposals/0373-vars-without-limits-in-result-builders.md new file mode 100644 index 0000000000..9beffe1bde --- /dev/null +++ b/proposals/0373-vars-without-limits-in-result-builders.md @@ -0,0 +1,113 @@ +# Lift all limitations on variables in result builders + +* Proposal: [SE-0373](0373-vars-without-limits-in-result-builders.md) +* Author: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#60839](https://github.com/apple/swift/pull/60839) +* Review: ([pitch](https://forums.swift.org/t/pitch-lift-all-limitations-on-variables-in-result-builders/60460)) ([review](https://forums.swift.org/t/se-0373-lift-all-limitations-on-variables-in-result-builders/60592)) ([acceptance](https://forums.swift.org/t/accepted-se-0373-lift-all-limitations-on-variables-in-result-builders/61041)) + +## Introduction + +The implementation of the result builder transform (introduced by [SE-0289](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md)) places a number of limitations on local variable declarations in the transformed function. Specifically, local variables need to have an initializer expression, they cannot be computed, they cannot have observers, and they cannot have attached property wrappers. None of these restrictions were explicit in the SE-0289 proposal, but they are a *de facto* part of the current feature. + +## Motivation + +The result builder proposal [describes how the result builder transform handles each individual component in a function body](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md#the-result-builder-transform). It states that local declarations are unaffected by the transformation, which implies that any declaration allowed in that context should be supported. That is not the case under the current implementation, which requires that local variables declarations must have a simple name, storage, and an initializing expression. + +In certain circumstances, it's useful to be able to declare a local variable that, for example, declares multiple variables, has default initialization, or has an attached property wrapper (with or without an initializer). Let's take a look at a simple example: + +```swift +func compute() -> (String, Error?) { ... } + +func test(@MyBuilder builder: () -> Int?) { + ... +} + +test { + let (result, error) = compute() + + let outcome: Outcome + + if let error { + // error specific logic + outcome = .failure + } else { + // complex computation + outcome = .success + } + + switch outcome { + ... + } +} +``` + +Both declarations are currently rejected because result builders only allow simple (with just one name) stored variables with an explicit initializer expression. + +Local variable declarations with property wrappers (with or without an explicit initializer) can be utilized for a variety of use-cases, including but not limited to: + +* Verification and/or formatting of the user-provided input + +```swift +import SwiftUI + +struct ContentView: View { + var body: some View { + GeometryReader { proxy in + @Clamped(10...100) var width = proxy.size.width + Text("\(width)") + } + } +} +``` + +* Interacting with user defaults + +```swift +import SwiftUI + +struct AppIntroView: View { + var body: some View { + @UserDefault(key: "user_has_ever_interacted") var hasInteracted: Bool + ... + Button("Browse Features") { + ... + hasInteracted = true + } + Button("Create Account") { + ... + hasInteracted = true + } + } +} +``` + + +## Proposed solution + +I propose to treat local variable declarations in functions transformed by result builders as if they appear in an ordinary function without any additional restrictions. + +## Detailed design + +The change is purely semantic, without any new syntax. It allows variables of all of these kinds to be declared in a function that will be transformed by a result builder: + +* uninitialized variables (only if supported by the builder, see below for more details) +* default-initialized variables (e.g. variables with optional type) +* computed variables +* observed variables +* variables with property wrappers +* `lazy` variables + +These variables will be treated just like they are treated in regular functions. All of the ordinary semantic checks to verify their validity will still be performed, and invalid declarations (based on the standard rules) will still be rejected by the compiler. + +There is one notable exception to this general rule. Initializing a variable after its declaration requires writing an assignment to it, and assignments require the result builder to support `Void` results, as described in [SE-0289](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md#assignments). If the result builder does not support `Void` results (whether with an explicit `buildExpression` or just by handling them in `buildBlock`), transformed functions will not be allowed to contain uninitialized declarations. + + +## Source compatibility + +This is an additive change which should not affect existing source code. + +## Effect on ABI stability and API resilience + +These changes do not require support from the language runtime or standard library, and they do not change anything about the external interface to the transformed function. diff --git a/proposals/0374-clock-sleep-for.md b/proposals/0374-clock-sleep-for.md new file mode 100644 index 0000000000..f9653cc4e0 --- /dev/null +++ b/proposals/0374-clock-sleep-for.md @@ -0,0 +1,200 @@ +# Add sleep(for:) to Clock + +* Proposal: [SE-0374](0374-clock-sleep-for.md) +* Authors: [Brandon Williams](https://github.com/mbrandonw), [Stephen Celis](https://github.com/stephencelis) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#61222](https://github.com/apple/swift/pull/61222) +* Review: ([pitch](https://forums.swift.org/t/pitch-clock-sleep-for/60376)) ([review](https://forums.swift.org/t/se-0374-add-sleep-for-to-clock/60787)) ([acceptance](https://forums.swift.org/t/accepted-se-0374-add-sleep-for-to-clock/62148)) + +## Introduction + +The `Clock` protocol introduced in Swift 5.7 provides a way to suspend until a future instant, but +does not provide a way to sleep for a duration. This differs from the static `sleep` methods on +`Task`, which provide both a way to sleep until an instant or for a duration. + +This imbalance in APIs might be reason enough to add a `sleep(for:)` method to all clocks, but the +real problem occurs when dealing with `Clock` existentials. Because the `Instant` associated type +is fully erased, and only the `Duration` is preserved via the primary associated type, any API +that deals with instants is inaccessible to an existential. This means one cannot invoke +`sleep(until:)` on an existential clock, and hence you can't really do anything with an existential +clock. + +## Motivation + +Existentials provide a convenient way to inject dependencies into features so that you can use one +kind of dependency in production, and another kind in tests. The most prototypical version of this +is API clients. When you run your feature in production you want the API client to make real life +network requests, but when run in tests you may want it to just return some mock data. + +Due to the current design of `Clock`, it is not possible to inject a clock existential into a +feature so that you can use a `ContinuousClock` in production, but some other kind of controllable +clock in tests. + +For example, suppose you have an observable object for the logic of some feature that wants to show +a welcoming message after waiting 5 seconds. That might look like this: + +```swift +class FeatureModel: ObservableObject { + @Published var message: String? + func onAppear() async { + do { + try await Task.sleep(until: .now.advanced(by: .seconds(5))) + self.message = "Welcome!" + } catch {} + } +} +``` + +If you wrote a test for this, your test suite would have no choice but to wait for 5 real life +seconds to pass before it could make an assertion: + +```swift +let model = FeatureModel() + +XCTAssertEqual(model.message, nil) +await model.onAppear() // Waits for 5 seconds +XCTAssertEqual(model.message, "Welcome!") +``` + +This affects people who don't even write tests. If you put your feature into an Xcode preview, then +you would have to wait for 5 full seconds to pass before you get to see the welcome message. That +means you can't quickly iterate on the styling of that message. + +The solution to these problems is to not reach out to the global, uncontrollable `Task.sleep`, and +instead inject a clock into the feature. And that is typically done using an existential, but +unfortunately that does not work: + +```swift +class FeatureModel: ObservableObject { + @Published var message: String? + let clock: any Clock + + func onAppear() async { + do { + try await self.clock.sleep(until: self.clock.now.advanced(by: .seconds(5))) // 🛑 + self.message = "Welcome!" + } catch {} + } +} +``` + +One cannot invoke `sleep(until:)` on a clock existential because the `Instant` has been fully +erased, and so there is no way to access `.now` and advance it. + +For similar reasons, one cannot invoke `Task.sleep(until:clock:)` with a clock existential: + +```swift +try await Task.sleep(until: self.clock.now.advanced(by: .seconds(5)), clock: self.clock) // 🛑 +``` + +What we need instead is the `sleep(for:)` method on clocks that allow you to sleep for a duration +rather than sleeping until an instant: + +```swift +class FeatureModel: ObservableObject { + @Published var message: String? + let clock: any Clock + + func onAppear() async { + do { + try await self.clock.sleep(for: .seconds(5)) // ✅ + self.message = "Welcome!" + } catch {} + } +} +``` + +Without a `sleep(for:)` method on clocks, one cannot use a clock existential in the feature, and +that forces you to introduce a generic: + +```swift +class FeatureModel>: ObservableObject { + @Published var message: String? + let clock: C + + func onAppear() async { + do { + try await self.clock.sleep(until: self.clock.now.advanced(by: .seconds(5))) + self.message = "Welcome!" + } catch {} + } +} +``` + +But this is problematic. This will force any code that touches `FeatureModel` to also introduce a +generic if you want that code to be testable and controllable. And it's strange that the class +is statically announcing its dependence on a clock when its mostly just an internal detail of the +class. + +By adding a `sleep(for:)` method to `Clock` we can fix all of these problems, and give Swift users +the ability to control time-based asynchrony in their applications. + +## Proposed solution + +A single extension method will be added to the `Clock` protocol: + +```swift +extension Clock { + /// Suspends for the given duration. + /// + /// Prefer to use the `sleep(until:tolerance:)` method on `Clock` if you have access to an + /// absolute instant. + public func sleep( + for duration: Duration, + tolerance: Duration? = nil + ) async throws { + try await self.sleep(until: self.now.advanced(by: duration), tolerance: tolerance) + } +} +``` + +This will allow one to sleep for a duration with a clock rather than sleeping until an instant. + +Further, to make the APIs between `clock.sleep(for:)` and `Task.sleep(for:)` similar, we will also add a `clock` and `tolerance` argument to `Task.sleep(for:)`: + +```swift +extension Task where Success == Never, Failure == Never { + /// Suspends the current task for the given duration on a continuous clock. + /// + /// If the task is cancelled before the time ends, this function throws + /// `CancellationError`. + /// + /// This function doesn't block the underlying thread. + /// + /// try await Task.sleep(for: .seconds(3)) + /// + /// - Parameter duration: The duration to wait. + public static func sleep( + for duration: C.Duration, + tolerance: C.Duration? = nil, + clock: C = ContinuousClock() + ) async throws { + try await sleep(until: clock.now.advanced(by: duration), tolerance: tolerance, clock: clock) + } +} +``` + +And we will add a default value for the `clock` argument of `Task.sleep(until:)`: + +```swift +extension Task where Success == Never, Failure == Never { + public static func sleep( + until deadline: C.Instant, + tolerance: C.Instant.Duration? = nil, + clock: C = ContinuousClock() + ) async throws { + try await clock.sleep(until: deadline, tolerance: tolerance) + } +} +``` + +## Source compatibility, effect on ABI stability, effect on API resilience + +As this is an additive change, it should not have any compatibility, stability or resilience +problems. + +## Alternatives considered + +We could leave things as is, and not add this method to the standard library, as it is possible for +people to define it themselves. diff --git a/proposals/0375-opening-existential-optional.md b/proposals/0375-opening-existential-optional.md new file mode 100644 index 0000000000..e2e5272d39 --- /dev/null +++ b/proposals/0375-opening-existential-optional.md @@ -0,0 +1,66 @@ +# Opening existential arguments to optional parameters + +* Proposal: [SE-0375](0375-opening-existential-optional.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#61321](https://github.com/apple/swift/pull/61321) +* Review: ([pitch](https://forums.swift.org/t/mini-pitch-for-se-0352-amendment-allow-opening-an-existential-argument-to-an-optional-parameter/60501)) ([review](https://forums.swift.org/t/se-0375-opening-existential-arguments-to-optional-parameters/60802)) ([acceptance](https://forums.swift.org/t/accepted-se-0375-opening-existential-arguments-to-optional-parameters/61045)) + +## Introduction + +[SE-0352 "Implicitly Opened Existentials"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md) has a limitation that prevents the opening of an existential argument when the corresponding parameter is optional. This proposal changes that behavior, so that such a call will succeed when a (non-optional) existential argument is passed to a parameter of optional type: + +```swift +func acceptOptional(_ x: T?) { } +func test(p: any P, pOpt: (any P)?) { + acceptOptional(p) // SE-0352 does not open "p"; this proposal will open "p" and bind "T" to its underlying type + acceptOptional(pOpt) // does not open "pOpt", because there is no "T" to bind to when "pOpt" is "nil" +} +``` + +The rationale for not opening the existential `p` in the first call was to ensure consistent behavior with the second call, in an effort to avoid confusion. SE-0352 says: + +> The case of optionals is somewhat interesting. It's clear that the call `cannotOpen6(pOpt)` cannot work because `pOpt` could be `nil`, in which case there is no type to bind `T` to. We *could* choose to allow opening a non-optional existential argument when the parameter is optional, e.g., +> +> ``` +> cannotOpen6(p1) // we *could* open here, binding T to the underlying type of p1, but choose not to +> ``` +> +> but this proposal doesn't allow this because it would be odd to allow this call but not the `cannotOpen6(pOpt)` call. + +However, experience with implicitly-opened existentials has shown that opening an existential argument in the first case is important, because many functions accept optional parameters. It is possible to work around this limitation, but doing so requires a bit of boilerplate, using a generic function that takes a non-optional parameter as a trampoline to the one that takes an optional parameter: + +```swift +func acceptNonOptionalThunk(_ x: T) { + acceptOptional(x) +} + +func test(p: any P) { + acceptNonOptionalThunk(p) // workaround for SE-0352 to get a call to acceptOptional with opened existential +} +``` + +## Proposed solution + +Allow an argument of (non-optional) existential type to be opened to be passed to an optional parameter: + +```swift +func openOptional(_ value: T?) { } + +func testOpenToOptional(p: any P) { + openOptional(p) // okay, opens 'p' and binds 'T' to its underlying type +} +``` + +## Source compatibility + +Generally speaking, opening an existential argument in one more case will make code that would have been rejected by the compiler (e.g., with an error like "`P` does not conform to `P`") into code that is accepted, because the existential is opened. This can change the behavior of overload resolution, in the same manner as was [discussed in SE-0352](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md#source-compatibility). Experience with SE-0352's integration into Swift 5.7 implies that the practical effect of these changes is quite small. + +## Effect on ABI stability + +This proposal changes the type system but has no ABI impact whatsoever. + +## Effect on API resilience + +This proposal changes the use of APIs, but not the APIs themselves, so it doesn't impact API resilience per se. diff --git a/proposals/0376-function-back-deployment.md b/proposals/0376-function-back-deployment.md new file mode 100644 index 0000000000..396dcc93da --- /dev/null +++ b/proposals/0376-function-back-deployment.md @@ -0,0 +1,213 @@ +# Function Back Deployment + +* Proposal: [SE-0376](0376-function-back-deployment.md) +* Author: [Allan Shortlidge](https://github.com/tshortli) +* Implementation: [apple/swift#41271](https://github.com/apple/swift/pull/41271), [apple/swift#41348](https://github.com/apple/swift/pull/41348), [apple/swift#41416](https://github.com/apple/swift/pull/41416), [apple/swift#41612](https://github.com/apple/swift/pull/41612) as the underscored attribute `@_backDeploy` +* Review Manager: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Review: ([pitch](https://forums.swift.org/t/pitch-function-back-deployment/55769)) ([review](https://forums.swift.org/t/se-0376-function-back-deployment/61015)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0376-function-back-deployment/61507)) ([second review](https://forums.swift.org/t/se-0376-second-review-function-back-deployment/61671)) ([returned for revision (second review)](https://forums.swift.org/t/returned-for-revision-se-0376-second-review-function-back-deployment/62374))([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0376-function-back-deployment/62905)) +* Status: **Implemented (Swift 5.8)** + +## Introduction + +This proposal introduces a `@backDeployed` attribute to allow ABI-stable libraries to make their own public APIs available on older OSes. When a `@backDeployed` API isn't present in the library that ships with an older OS, a client running on that OS can still use the API because a fallback copy of its implementation has been emitted into the client. + +With `@backDeployed`, a function may be emitted into clients as a fallback copy of _itself_. Note that the attribute doesn't mark a function as a fallback implementation of some _other_ function, and therefore it doesn't help one module to extend the availability of APIs declared in some _other_ module. + +## Motivation + +Resilient Swift libraries, such as the ones present in the SDKs for Apple's platforms, are distributed as dynamic libraries. Authors of these libraries use `@available` annotations to indicate the operating system version that a declaration was introduced in. For example, suppose this were the interface of ToastKit, a library that is part of the toasterOS SDK: + +```swift +@available(toasterOS 1.0, *) +public struct BreadSlice { ... } + +@available(toasterOS 1.0, *) +public struct Toast { ... } + +@available(toasterOS 1.0, *) +public struct Toaster { + public func makeToast(_ slice: BreadSlice) -> Toast +} +``` + +In response to developer feedback, the ToastKit authors enhance `Toaster` in toasterOS 2.0 with the capability to make toast in batches: + +```swift +extension Toaster { + @available(toasterOS 2.0, *) + public func makeBatchOfToast(_ slices: [BreadSlice]) -> [Toast] { + var toast: [Toast] = [] + for slice in slices { + toast.append(makeToast(slice)) + } + return toast + } +} +``` + +Unfortunately, developers who wish to both distribute an app compatible with toasterOS 1.0 and also adopt `makeBatchOfToast(_:)` must call the API conditionally to account for its potential unavailability: + +```swift +let slices: [BreadSlice] = ... +if #available(toasterOS 2.0, *) { + let toast = toaster.makeBatchOfToast(slices) + // ... +} else { + // ... do something else, like reimplement makeBatchOfToast(_:) +} +``` + +Considering that the implementation of `makeBatchOfToast(_:)` is self contained and could run unmodified on toasterOS 1.0, it would be ideal if the ToastKit authors had the option to back deploy this new API to older OSes and allow clients to adopt it unconditionally. + +The `@_alwaysEmitIntoClient` attribute is an unofficial Swift language feature that can be used to solve this problem. The bodies of functions with this attribute are emitted into the library's `.swiftinterface` (similarly to `@inlinable` functions) and the compiler makes a local copy of the annotated function in the client module. References to these functions _always_ resolve to a copy in the same module so the function is effectively not a part of the library's ABI. + +While `@_alwaysEmitIntoClient` can be used to back deploy APIs, there are some drawbacks to using it. Since a copy of the function is always emitted, there is code size overhead for every client even if the client's deployment target is new enough that the library API would always be available at runtime. Additionally, if the implementation of the API were to change in order to improve performance, fix a bug, or close a security hole then the client would need to be recompiled against a new SDK before users benefit from those changes. An attribute designed specifically to support back deployment should avoid these drawbacks by ensuring that: + +1. The API implementation from the original library is preferred at runtime when it is available. +2. Fallback copies of the API implementation are absent from clients binaries when they would never be used. + +## Proposed solution + +Add a `@backDeployed(before: ...)` attribute to Swift that can be used to indicate that a copy of the function should be emitted into the client to be used at runtime when executing on an OS prior to the version identified with the `before:` argument. The attribute can be adopted by ToastKit's authors like this: + +```swift +extension Toaster { + @available(toasterOS 1.0, *) + @backDeployed(before: toasterOS 2.0) + public func makeBatchOfToast(_ breadSlices: [BreadSlice]) -> [Toast] { ... } +} +``` + +The API is now available on toasterOS 1.0 and later so clients may now reference `makeBatchOfToast(_:)` unconditionally. The compiler detects applications of `makeBatchOfToast(_:)` and generates code to automatically handle the potentially runtime unavailability of the API. + +## Detailed design + +The `@backDeployed` attribute may apply to functions, methods, and subscripts. Properties may also have the attribute as long as the they do not have storage. The attribute takes a comma separated list of one or more platform versions, so declarations that are available on more than one platform can be back deployed on multiple platforms with a single attribute. The following are examples of legal uses of the attribute: + +```swift +extension Temperature { + @available(toasterOS 1.0, ovenOS 1.0, *) + @backDeployed(before: toasterOS 2.0, ovenOS 2.0) + public var degreesFahrenheit: Double { + return (degreesCelsius * 9 / 5) + 32 + } +} + +extension Toaster { + /// Returns whether the slot at the given index can fit a bagel. + @available(toasterOS 1.0, *) + @backDeployed(before: toasterOS 2.0) + public subscript(fitsBagelsAt index: Int) -> Bool { + get { return index < 2 } + } +} +``` + +### Behavior of back deployed APIs + +When the compiler encounters a call to a back deployed function, it generates and calls a thunk instead that forwards the arguments to either the library copy of the function or a fallback copy of the function. For instance, suppose the client's code looks like this: + +```swift +let toast = toaster.makeBatchOfToast(slices) +``` + +The transformation done by the compiler would effectively result in this: + +```swift +let toast = toaster.makeBatchOfToast_thunk(slices) + +// Compiler generated +extension Toaster { + func makeBatchOfToast_thunk(_ breadSlices: [BreadSlice]) -> [Toast] { + if #available(toasterOS 2.0, *) { + return makeBatchOfToast(breadSlices) // call the original + } else { + return makeBatchOfToast_fallback(breadSlices) // call local copy + } + } + + func makeBatchOfToast_fallback(_ breadSlices: [BreadSlice]) -> [Toast] { + // ... copy of function body from ToastKit + } +} +``` + +When the deployment target of the client app is at least toasterOS 2.0, the compiler can eliminate the branch in `makeBatchOfToast_thunk(_:)` and therefore make `makeBatchOfToast_fallback(_:)` an unused function, which reduces the unnecessary bloat that could otherwise result from referencing a back deployed API. + +### Restrictions on declarations that may be back deployed + +There are rules that limit which declarations may have a `@backDeployed` attribute: + +* The declaration must be `public` or `@usableFromInline` since it only makes sense to offer back deployment for declarations that would be used by other modules. +* Only functions that can be invoked with static dispatch are eligible to back deploy, so back deployed instance and class methods must be `final`. The `@objc` attribute also implies dynamic dispatch and therefore is incompatible with `@backDeployed`. +* The declaration should be available earlier than the platform versions specified in `@backDeployed` (otherwise the fallback functions would never be called). +* The `@_alwaysEmitIntoClient` and `@_transparent` attributes are incompatible with `@backDeployed` because they require the function body to always be emitted into the client, defeating the purpose of `@backDeployed`. +* Declarations with `@inlinable` _may_ use `@backDeployed`. As usual with `@inlinable`, the bodies of these functions may be emitted into the client at the discretion of the optimizer. The copy of the function in the client may therefore be used even when a copy of the function is available in the library. + +### Requirements for the bodies of back deployed functions + +The restrictions on the bodies of back deployed functions are the same as `@inlinable` functions. The body may only reference declarations that are accessible to the client, such as `public` and `@usableFromInline` declarations. Similarly, those referenced declarations must also be at least as available the back deployed function, or `if #available` must be used to handle potential unavailability. Type checking in `@backDeployed` function bodies must ignore the library's deployment target since the body will be copied into clients with unknown deployment targets. + +## Source compatibility + +The introduction of this attribute to the language is an additive change and therefore doesn't affect existing Swift code. + +## Effect on ABI stability + +The `@backDeployed` attribute has no effect on the ABI of Swift libraries. A Swift function with and without a `@backDeployed` attribute has the same ABI; the attribute simply controls whether the compiler automatically generates additional logic in the client module. The thunk and fallback functions that are emitted into the client do have a special mangling to disambiguate them from the original function in the library, but these symbols are never referenced across separately compiled modules. + +## Effect on API resilience + +By itself, adding a `@backDeployed` attribute to a declaration does not affect source compatibility for clients of a library, and neither does removing the attribute. However, adding a `@backDeployed` attribute would typically be done simultaneously with expanding the availability of the declaration by lowering the `introduced:` version in the `@available` attribute. Expansion of the availability of an API is source compatible for clients, but reversing that expansion would not be. + +## Alternatives considered + +### Use a different argument label name + +A few alternative spellings of the argument label `before:` were considered including `upTo:`, `until:`, and `implemented:`. The choice of label is significant because it influences the reader's intuitive understanding of the semantics of the attribute. The label should ideally make the directionality of the effect clear as well as the exclusivity of the OS version range. It also helps if the attribute as a whole reads fluently when expanded into an English sentence like this: + +> The function is back deployed for all minimum deployment targets _before_ iOS 13. + +Reviewers did not consistently agree that any of the labels that were considered successfully clarified the directionality of the effect or the exclusivity of the range but the label `before:` was ultimately deemed the clearest option. + +### Use a different attribute name + +One way to frame the proposed attribute is that it indicates which OS versions the function became ABI stable in. From that perspective, naming the attribute something like `@abi(introduced:)` could make sense. However, by default every public function in an SDK library is already implicitly ABI stable at the `introduced:` version of its availability so it would be reasonable to ask what distinction this attribute is making and why it is not present on every API that is ABI stable. This naming choice would obfuscate the essential effect of the attribute, requiring unfamiliar readers to read the documentation to learn that the purpose of the attribute is to extend the function's availability to earlier deployment targets. + +### Extend @available + +Another possible design for this feature would be to augment the existing `@available` attribute instead of introducing a new attribute. In the following example, a `backDeployBefore:` label is added to the `@available` attribute: + +```swift +extension Toaster { + @available(toasterOS, introduced: 1.0, backDeployBefore: 2.0) + public func makeBatchOfToast(_ breadSlices: [BreadSlice]) -> [Toast] +} +``` + +This design has the advantage of grouping the introduction and back deployment versions together in a single attribute, which may be easier to understand for library authors who want to adopt this capability. However, there are drawbacks: + +- The `@available` attribute's existing responsibilities relate to constraining the contexts in which a declaration can be used. The version in which the declaration became ABI is not an availability constraint, but rather information that the library author provides to the compiler in order to give the declaration extended availability. A client of the library does not need this information in order to understand where the API may be used. It seems wise to avoid further complicating the already complex `@available` attribute with additional responsibilities that do not relate to its core purpose. +- This design would require library authors to use the long form of `@available`, which would lead to increased verbosity for APIs that are available on many different OSes. + +A variant of this alternative design would be to add a `backDeployTo:` label instead and change the meaning of the `introduced:` label to indicate the version of OS that the declaration became ABI stable: + +```swift +extension Toaster { + @available(toasterOS, backDeployTo: 1.0, introduced: 2.0) + public func makeBatchOfToast(_ breadSlices: [BreadSlice]) -> [Toast] +} +``` + +This has the same drawbacks documented above and also further contradicts the principle of progressive disclosure by making it necessary to learn about back deployment as a concept in order to understand where an API declaration may be used. + + +## Future directions + +### Back deployment for other kinds of declarations + +It would also be useful to be able to back deploy the implementations of other types of declarations, such as entire enums, structs, or even protocol conformances. Exploring the feasibility of such a feature is out of scope for this proposal, but whether or not the design can accommodate being extended to other kinds of declarations is important to consider. + +## Acknowledgments + +Thank you to Alexis Laferriere, Ben Cohen, and Xi Ge for their help designing the feature and to Slava Pestov for his assistance with SILGen. diff --git a/proposals/0377-parameter-ownership-modifiers.md b/proposals/0377-parameter-ownership-modifiers.md new file mode 100644 index 0000000000..136860120e --- /dev/null +++ b/proposals/0377-parameter-ownership-modifiers.md @@ -0,0 +1,677 @@ +# `borrowing` and `consuming` parameter ownership modifiers + +* Proposal: [SE-0377](0377-parameter-ownership-modifiers.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm), [Joe Groff](https://github.com/jckarter) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9)** +* Implementation: in main branch of compiler +* Review: ([first pitch](https://forums.swift.org/t/pitch-formally-defining-consuming-and-nonconsuming-argument-type-modifiers/54313)) ([second pitch](https://forums.swift.org/t/borrow-and-take-parameter-ownership-modifiers/59581)) ([first review](https://forums.swift.org/t/se-0377-borrow-and-take-parameter-ownership-modifiers/61020)) ([second review](https://forums.swift.org/t/combined-se-0366-third-review-and-se-0377-second-review-rename-take-taking-to-consume-consuming/61904)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0377-borrowing-and-consuming-parameter-ownership-modifiers/62759)) ([revision and third review](https://forums.swift.org/t/se-0377-revision-make-borrowing-and-consuming-parameters-require-explicit-copying-with-the-copy-operator/64996)) ([revision acceptance](https://forums.swift.org/t/accepted-se-0377-revision-make-borrowing-and-consuming-parameters-require-explicit-copying-with-the-copy-operator/65293)) +* Previous Revisions: ([as of first review](https://github.com/swiftlang/swift-evolution/blob/3f984e6183ce832307bb73ec72c842f6cb0aab86/proposals/0377-parameter-ownership-modifiers.md)) ([as of second review](https://github.com/swiftlang/swift-evolution/blob/7e1d16316e5f68eb94546df9241aa6b4cacb9411/proposals/0377-parameter-ownership-modifiers.md)) + +## Introduction + +We propose new `borrowing` and `consuming` parameter modifiers to allow developers to +explicitly choose the ownership convention that a function uses to receive +immutable parameters. Applying one of these modifiers to a parameter causes +that parameter binding to no longer be implicitly copyable, and potential +copies need to be marked with the new `copy x` operator. This allows for +fine-tuning of performance by reducing the number of ARC calls or copies needed +to call a function, and provides a necessary prerequisite feature for +noncopyable types to specify whether a function consumes a noncopyable value or +not. + +## Motivation + +Swift uses automatic reference counting to manage the lifetimes of reference- +counted objects. There are two broad conventions that the compiler uses to +maintain memory safety when passing an object by value from a caller to a +callee in a function call: + +* The callee can **borrow** the parameter. The caller + guarantees that its argument object will stay alive for the duration of the + call, and the callee does not need to release it (except to balance any + additional retains it performs itself). +* The callee can **consume** the parameter. The callee + becomes responsible for either releasing the parameter or passing ownership + of it along somewhere else. If a caller doesn't want to give up its own + ownership of its argument, it must retain the argument so that the callee + can consume the extra reference count. + +These two conventions generalize to value types, where a "retain" +becomes an independent copy of the value, and "release" the destruction and +deallocation of the copy. By default Swift chooses which convention to use +based on some rules informed by the typical behavior of Swift code: +initializers and property setters are more likely to use their parameters to +construct or update another value, so it is likely more efficient for them to +*consume* their parameters and forward ownership to the new value they construct. +Other functions default to *borrowing* their parameters, since we have found +this to be more efficient in most situations. + +These choices typically work well, but aren't always optimal. +Although the optimizer supports "function signature optimization" that can +change the convention used by a function when it sees an opportunity to reduce +overall ARC traffic, the circumstances in which we can automate this are +limited. The ownership convention becomes part of the ABI for public API, so +cannot be changed once established for ABI-stable libraries. The optimizer +also does not try to optimize polymorphic interfaces, such as non-final class +methods or protocol requirements. If a programmer wants behavior different +from the default in these circumstances, there is currently no way to do so. + +[SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md) +introduces noncopyable types into Swift. Since noncopyable types do not have +the ability to be copied, the distinction between these two conventions becomes +an important part of the API contract: functions that *borrow* noncopyable +values make temporary use of the value and leave it valid for further use, like +reading from a file handle, whereas functions that *consume* a noncopyable +value consume it and prevent its further use, like closing a file handle. +Relying on implicit selection of the parameter convention will not suffice for +these types. + +## Proposed solution + +We give developers direct control over the ownership convention of +parameters by introducing two new parameter modifiers, `borrowing` and +`consuming`. + +## Detailed design + +### Syntax of parameter ownership modifiers + +`borrowing` and `consuming` become contextual keywords inside parameter type +declarations. They can appear in the same places as the `inout` modifier, and +are mutually exclusive with each other and with `inout`. In a `func`, +`subscript`, or `init` declaration, they appear as follows: + +```swift +func foo(_: borrowing Foo) +func foo(_: consuming Foo) +func foo(_: inout Foo) +``` + +In a closure: + +```swift +bar { (a: borrowing Foo) in a.foo() } +bar { (a: consuming Foo) in a.foo() } +bar { (a: inout Foo) in a.foo() } +``` + +In a function type: + +```swift +let f: (borrowing Foo) -> Void = { a in a.foo() } +let f: (consuming Foo) -> Void = { a in a.foo() } +let f: (inout Foo) -> Void = { a in a.foo() } +``` + +Methods can also use the `consuming` or `borrowing` modifier to indicate +respectively that they consume ownership of their `self` parameter or that they +borrow it. These modifiers are mutually exclusive with each other and with the +existing `mutating` modifier: + +```swift +struct Foo { + consuming func foo() // `consuming` self + borrowing func foo() // `borrowing` self + mutating func foo() // modify self with `inout` semantics +} +``` + +`consuming` cannot be applied to parameters of nonescaping closure type, which by +their nature are always borrowed: + +```swift +// ERROR: cannot `consume` a nonescaping closure +func foo(f: consuming () -> ()) { +} +``` + +`consuming` or `borrowing` on a parameter do not affect the caller-side syntax for +passing an argument to the affected declaration, nor do `consuming` or +`borrowing` affect the application of `self` in a method call. For typical +Swift code, adding, removing, or changing these modifiers does not have any +source-breaking effects. (See "related directions" below for interactions with +other language features being considered currently or in the near future which +might interact with these modifiers in ways that cause them to break source.) + +### Ownership convention conversions in protocols and function types + +Protocol requirements can also use `consuming` and `borrowing`, and the modifiers will +affect the convention used by the generic interface to call the requirement. +The requirement may still be satisfied by an implementation that uses different +conventions for parameters of copyable types: + +```swift +protocol P { + func foo(x: consuming Foo, y: borrowing Foo) +} + +// These are valid conformances: + +struct A: P { + func foo(x: Foo, y: Foo) +} + +struct B: P { + func foo(x: borrowing Foo, y: consuming Foo) +} + +struct C: P { + func foo(x: consuming Foo, y: borrowing Foo) +} +``` + +Function values can also be implicitly converted to function types that change +the convention of parameters of copyable types among unspecified, `borrowing`, +or `consuming`: + +```swift +let f = { (a: Foo) in print(a) } + +let g: (borrowing Foo) -> Void = f +let h: (consuming Foo) -> Void = f + +let f2: (Foo) -> Void = h +``` + +These implicit conversions for protocol conformances and function values +are not available for parameter types that are noncopyable, in which case +the convention must match exactly. + +### Using parameter bindings with ownership modifiers + +Inside of a function or closure body, `consuming` parameters may be mutated, as can +the `self` parameter of a `consuming func` method. These +mutations are performed on the value that the function itself took ownership of, +and will not be evident in any copies of the value that might still exist in +the caller. This makes it easy to take advantage of the uniqueness of values +after ownership transfer to do efficient local mutations of the value: + +```swift +extension String { + // Append `self` to another String, using in-place modification if + // possible + consuming func plus(_ other: String) -> String { + // Modify our owned copy of `self` in-place, taking advantage of + // uniqueness if possible + self += other + return self + } +} + +// This is amortized O(n) instead of O(n^2)! +let helloWorld = "hello ".plus("cruel ").plus("world") +``` + +`borrowing` and `consuming` parameter values are also **not implicitly copyable** +inside of the function or closure body: + +```swift +func foo(x: borrowing String) -> (String, String) { + return (x, x) // ERROR: needs to copy `x` +} +func bar(x: consuming String) -> (String, String) { + return (x, x) // ERROR: needs to copy `x` +} +``` + +And so is the `self` parameter within a method that has the method-level +`borrowing` or `consuming` modifier: + +```swift +extension String { + borrowing func foo() -> (String, String) { + return (self, self) // ERROR: needs to copy `self` + } + consuming func bar() -> (String, String) { + return (self, self) // ERROR: needs to copy `self` + } +} +``` + +A value would need to be implicitly copied if: + +- a *consuming operation* is applied to a `borrowing` binding, or +- a *consuming operation* is applied to a `consuming` binding after it has + already been consumed, or while a *borrowing* or *mutating operation* is simultaneously + being performed on the same binding + +where *consuming*, *borrowing*, and *mutating operations* are as described for +values of noncopyable type in +[SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md#using-noncopyable-values). +In essence, disabling implicit copying for a binding makes the binding behave +as if it were of some noncopyable type. + +To allow a copy to occur, the `copy x` operator may be used: + +```swift +func dup(_ x: borrowing String) -> (String, String) { + return (copy x, copy x) // OK, copies explicitly allowed here +} +``` + +`copy x` is a *borrowing operation* on `x` that returns an independently +owned copy of the current value of `x`. The copy may then be independently +consumed or modified without affecting the original `x`. Note that, while +`copy` allows for a copy to occur, it is not a strict +obligation for the compiler to do so; the copy may still be optimized away +if it is deemed semantically unnecessary. + +`copy` is a contextual keyword, parsed as an operator if it is immediately +followed by an identifier on the same line, like the `consume x` operator before +it. In all other cases, `copy` is still treated as a reference to a +declaration named `copy`, as it would have been prior to this proposal. + +The constraint on implicit copies only affects the parameter binding itself. +The value of the parameter may be passed to other functions, or assigned to +other variables (if the convention allows), at which point the value may +be implicitly copied through those other parameter or variable bindings. + +```swift +func foo(x: borrowing String) { + let y = x // ERROR: attempt to copy `x` + bar(z: x) // OK, invoking `bar(z:)` does not require copying `x` +} + +func bar(z: String) { + let w = z // OK, z is implicitly copyable here +} + +func baz(a: consuming String) { + // let aa = (a, a) // ERROR: attempt to copy `a` + + let b = a + let bb = (b, b) // OK, b is implicitly copyable +} +``` + +To clarify the boundary within which the no-implicit-copy constraint applies, a +parameter binding's value *is* noncopyable as part of the *call expression* in +the caller, so if forming the call requires copying, that will raise an error, +even if the parameter would be implicitly copyable in the callee. The function +body serves as the boundary for the no-implicit-copy constraint: + +```swift +struct Bar { + var a: String + var b: String + init(ab: String) { + // OK, ab is implicitly copyable here + a = ab + b = ab + } +} + +func foo(x: borrowing String) { + _ = Bar(ab: x) // ERROR: would need to copy `x` to let `Bar.init` consume it +} +``` + +## Source compatibility + +Adding `consuming` or `borrowing` to a parameter in the language today does not +affect source compatibility with existing code outside of that function. +Callers can continue to call the function as normal, and the function body can +use the parameter as it already does. A method with `consuming` or `borrowing` +modifiers on its parameters can still be used to satisfy a protocol requirement +with different modifiers. Although `consuming` parameter bindings become +mutable, and parameters with either of the `borrowing` or `consuming` modifiers +are not implicitly copyable, the effects are localized to the function +adopting the modifiers. This allows for API authors to use +`consuming` and `borrowing` annotations to fine-tune the copying behavior of +their implementations, without forcing clients to be aware of ownership to use +the annotated APIs. Source-only packages can add, remove, or adjust these +annotations on copyable types over time without breaking their clients. + +Changing parameter modifiers from `borrowing` to `consuming` may however break +source of any client code that also adopts those parameter modifiers, since the +change may affect where copies need to occur in the caller. Going from +`consuming` to `borrowing` however should generally not be source-breaking +for a copyable type. A change in either direction is source-breaking if the +parameter type is noncopyable. + +## Effect on ABI stability + +`consuming` or `borrowing` affects the ABI-level calling convention and cannot be +changed without breaking ABI-stable libraries (except on "trivial types" +for which copying is equivalent to `memcpy` and destroying is a no-op; however, +`consuming` or `borrowing` also has no practical effect on parameters of trivial type). + +## Effect on API resilience + +`consuming` or `borrowing` break ABI for ABI-stable libraries, but are intended to have +minimal impact on source-level API. When using copyable types, adding or +changing these annotations to an API should not affect its existing clients, +except where those clients have also adopted the not-implicitly-copyable +conventions. + +## Alternatives considered + +### Leaving `consuming` parameter bindings immutable inside the callee + +We propose that `consuming` parameters should be mutable inside of the callee, +because it is likely that the callee will want to perform mutations using +the value it has ownership of. There is a concern that some users may find this +behavior unintuitive, since those mutations would not be visible in copies +of the value in the caller. This was the motivation behind +[SE-0003](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0003-remove-var-parameters.md), +which explicitly removed the former ability to declare parameters as `var` +because of this potential for confusion. However, whereas `var` and `inout` +both suggest mutability, and `var` does not provide explicit directionality as +to where mutations become visible, `consuming` on the other hand does not +suggest any kind of mutability to the caller, and it explicitly states the +directionality of ownership transfer. Furthermore, with noncopyable types, the +chance for confusion is moot, because the transfer of ownership means the +caller cannot even use the value after the callee takes ownership anyway. + +Another argument for `consuming` parameters to remain immutable is to serve the +proposal's stated goal of minimizing the source-breaking impact of +parameter ownership modifiers. When `consuming` parameters are mutable, +changing a `consuming` parameter to `borrowing`, or removing the +`consuming` annotation altogether, is potentially source-breaking. However, +any such breakage is purely localized to the callee; callers are still +unaffected (as long as copyable arguments are involved). If a developer wants +to change a `consuming` parameter back into a `borrowing`, they can still assign the +borrowed value to a local variable and use that local variable for local +mutation. + +### Naming + +We have considered several alternative naming schemes for these modifiers: + +- The current implementation in the compiler uses `__shared` and `__owned`, + and we could remove the underscores to make these simply `shared` and + `owned`. These names refer to the way a borrowed parameter receives a + "shared" borrow (as opposed to the "exclusive" borrow on an `inout` + parameter), whereas a consumed parameter becomes "owned" by the callee. + found that the "shared" versus "exclusive" language for discussing borrows, + while technically correct, is unnecessarily confusing for explaining the + model. +- A previous pitch used the names `nonconsuming` and `consuming`. The current + implementation also uses `__consuming func` to notate a method that takes + ownership of its `self` parameter. We think it is better to describe + `borrowing` in terms of what it means, rather than as the opposite of + the other convention. +- The first reviewed revision used `take` instead of `consume`. Along with + `borrow`, `take` arose during [the first review of +SE-0366](https://forums.swift.org/t/se-0366-move-function-use-after-move-diagnostic/59202). + These names also work well as names for operators that explicitly + transfer ownership of a variable or borrow it in place. However, + reviewers observed that `take` is possibly confusing, since it conflicts with + colloquial discussion of function calls "taking their arguments". `consume` + reads about as well while being more specific. +- Reviewers offered `use`, `own`, or `sink` as alternatives to `consume`. + +We think it is helpful to align the naming of these parameter modifiers with +the corresponding `consume` and `borrow` operators (discussed below under +Future Directions), since it helps reinforce the relationship between the +calling conventions and the expression operators: to explicitly transfer +ownership of an argument in a call site to a parameter in a function, use +`foo(consuming x)` at the call site, and use `func foo(_: consuming T)` in the +function declaration. Similarly, to explicitly pass an argument by borrow +without copying, use `foo(borrow x)` at the call site, and `func foo(_: borrowing T)` +in the function declaration. + +### `@noImplicitCopy` attribute + +Instead of having no-implicit-copy behavior be tied to the ownership-related +binding forms and parameter modifiers, we could have an attribute that can +be applied to any binding to say that it should not be implicitly copyable: + +```swift +@noImplicitCopy(self) +func foo(x: @noImplicitCopy String) { + @noImplicitCopy let y = copy x +} +``` + +We had [pitched this possibility](https://forums.swift.org/t/pitch-noimplicitcopy-attribute-for-local-variables-and-function-parameters/61506), +but community feedback rightly pointed out the syntactic weight and noise +of this approach, as well as the fact that, as an attribute, it makes the +ability to control copies feel like an afterthought not well integrated +with the rest of the language. We've decided not to continue in this direction, +since we think that attaching no-implicit-copy behavior to the ownership +modifiers themselves leads to a more coherent design. + +### `copy` as a regular function + +Unlike the `consume x` or `borrow x` operator, copying doesn't have any specific +semantic needs that couldn't be done by a regular function. Instead of an +operator, `copy` could be defined as a regular standard library function: + +```swift +func copy(_ value: T) -> T { + return value +} +``` + +We propose `copy x` as an operator, because it makes the relation to +`consume x` and `borrow x`, and it avoids the issues of polluting the +global identifier namespace and occasionally needing to be qualified as +`Swift.copy` if it was a standard library function. + +### Transitive no-implicit-copy constraint + +The no-implicit-copy constraint for a `borrowing` and `consuming` parameter +only applies to that binding, and is not carried over to other variables +or function call arguments receiving the binding's value. We could also +say that the parameter can only be passed as an argument to another function +if that function's parameter uses the `borrowing` or `consuming` modifier to +keep implicit copies suppressed, or that it cannot be bound to `let` or `var` +bindings and must be bound using one of the borrowing bindings once we have +those. However, we think those additional restrictions would only make the +`borrowing` and `consuming` modifiers harder to adopt, since developers would +only be able to use them in cases where they can introduce them bottom-up from +leaf functions. + +The transitivity restriction also would not really improve +local reasoning; since the restriction is only on *implicit* copies, but +explicit copies are still possible, calling into another function may lead +to that other function performing copies, whether they're implicit or not. +The only way to be sure would be to inspect the callee's implementation. +One of the goals of SE-0377 is to introduce the parameter ownership modifiers +in a way that minimizes disruption to the the rest of a codebase, allowing +for the modifiers to be easily adopted in spots where the added control is +necessary, and a transitivity requirement would interfere with that goal for +little benefit. + +## Related directions + +#### `consume` operator + +[SE-0366](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0366-move-function.md) +introduced an operator that explicitly ends the lifetime of a +variable before the end of its scope. This allows the compiler to reliably +destroy the value of the variable, or transfer ownership, at the point of its +last use, without depending on optimization and vague ARC optimizer rules. +When the lifetime of the variable ends in an argument to a `consume` parameter, +then we can transfer ownership to the callee without any copies: + +```swift +func consume(x: consuming Foo) + +func produce() { + let x = Foo() + consume(x: consume x) + doOtherStuffNotInvolvingX() +} +``` + +#### `borrow` operator + +Relatedly, there are circumstances where the compiler defaults to copying +when it is theoretically possible to borrow, particularly when working with +shared mutable state such as global or static variables, escaped closure +captures, and class stored properties. The compiler does +this to avoid running afoul of the law of exclusivity with mutations. In +the example below, if `callUseFoo()` passed `global` to `useFoo` by borrow +instead of passing a copy, then the mutation of `global` inside of `useFoo` +would trigger a dynamic exclusivity failure (or UB if exclusivity checks +are disabled): + +```swift +var global = Foo() + +func useFoo(x: borrowing Foo) { + // We need exclusive access to `global` here + global = Foo() +} + +func callUseFoo() { + // callUseFoo doesn't know whether `useFoo` accesses global, + // so we want to avoid imposing shared access to it for longer + // than necessary, and we'll pass a copy of the value. This: + useFoo(x: global) + + // will compile more like: + + /* + let globalCopy = copy(global) + useFoo(x: globalCopy) + destroy(globalCopy) + */ +} +``` + +It is difficult for the compiler to conclusively prove that there aren't +potential interfering writes to shared mutable state, so although it may +in theory eliminate the defensive copy if it proves that `useFoo`, it is +unlikely to do so in practice. The developer may know that the program will +not attempt to modify the same object or global variable during a call, +and want to suppress this copy. An explicit `borrow` operator could allow for +this: + +```swift +var global = Foo() + +func useFooWithoutTouchingGlobal(x: borrowing Foo) { + /* global not used here */ +} + +func callUseFoo() { + // The programmer knows that `useFooWithoutTouchingGlobal` won't + // touch `global`, so we'd like to pass it without copying + useFooWithoutTouchingGlobal(x: borrow global) +} +``` + +If `useFooWithoutTouchingGlobal` did in fact attempt to mutate `global` +while the caller is borrowing it, an exclusivity failure would be raised. + +#### Noncopyable types + +The `consuming` versus `borrowing` distinction becomes much more important and +prominent for values that cannot be implicitly copied. +[SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md) +introduces noncopyable types, whose values are never copyable, as well as +attributes that suppress the compiler's implicit copying behavior selectively +for particular variables or scopes. Operations that borrow +a value allow the same value to continue being used, whereas operations that +consume a value destroy it and prevent its continued use. This makes the +convention used for noncopyable parameters a much more important part of their +API contract, since it directly affects whether the value is still available +after the operation: + +```swift +struct FileHandle: ~Copyable { ... } + +// Operations that open a file handle return new FileHandle values +func open(path: FilePath) throws -> FileHandle + +// Operations that operate on an open file handle and leave it open +// borrow the FileHandle +func read(from: borrowing FileHandle) throws -> Data + +// Operations that close the file handle and make it unusable consume +// the FileHandle +func close(file: consuming FileHandle) + +func hackPasswords() throws -> HackedPasswords { + let fd = try open(path: "/etc/passwd") + // `read` borrows fd, so we can continue using it after + let contents = try read(from: fd) + // `close` consumes fd, so we can't use it again + close(fd) + + let moreContents = try read(from: fd) // compiler error: use after consume + + return hackPasswordData(contents) +} +``` + +As such, SE-0390 requires parameters of noncopyable type to explicitly state +whether they are `borrowing` or `consuming`, since there isn't a clear +default that is always safe to assume. + +### `set`/`out` parameter convention + +By making the `borrowing` and `consuming` conventions explicit, we mostly round out +the set of possibilities for how to handle a parameter. `inout` parameters get +**exclusive access** to their argument, allowing them to mutate or replace the +current value without concern for other code. By contrast, `borrowing` parameters +get **shared access** to their argument, allowing multiple pieces of code to +share the same value without copying, so long as none of them mutate the +shared value. A `consuming` parameter consumes a value, leaving nothing behind, but +there still isn't a parameter analog to the opposite convention, which would +be to take an uninitialized argument and populate it with a new value. Many +languages, including C# and Objective-C when used with the "Distributed +Objects" feature, have `out` parameter conventions for this, and the Val +programming language calls this `set`. + +In Swift up to this point, return values have been the preferred mechanism for +functions to pass values back to their callers. This proposal does not propose +to add some kind of `out` parameter, but a future proposal could. + +### `borrowing`, `mutating`, and `consuming` local variables + +Swift currently lacks the ability to form local bindings to part of an +aggregate without copying that part, other than by passing the part as +an argument to a function call. We plan to introduce [`borrow` and `inout` +bindings](https://forums.swift.org/t/pitch-borrow-and-inout-declaration-keywords/62366) +that will provide this functionality, with the same no-implicit-copy constraint +described by this proposal applied to these bindings. + +### Consistency for `inout` parameters and the `self` parameter of `mutating` methods + +`inout` parameters and `mutating` methods have been part of Swift since before +version 1.0, and their existing behavior allows for implicit copying of the +current value of the binding. We can't change the existing language +behavior in Swift 5, but accepting this proposal would leave `inout` parameters +and `mutating self` inconsistent with the new modifiers. There are a few things +we could potentially do about that: + +- We could change the behavior of `inout` and `mutating self` parameters to + make them not implicitly copyable in Swift 6 language mode. +- `inout` is also conspicuous now in not following the `-ing` convention we've + settled on for `consuming`/`borrowing`/`mutating` modifiers. We could introduce + `mutating` as a new parameter modifier spelling, with no-implicit-copy + behavior. + +One consideration is that, whereas `borrowing` and `consuming` are strictly +optional for code that works only with copyable types, and is OK with letting +the compiler manage copies automatically, there is no way to get in-place +mutation through function parameters except via `inout`. Tying +no-implicit-copy behavior to mutating parameters could be seen as a violation +of the "progressive disclosure" goal of these ownership features, since +developers would not be able to avoid interacting with the ownership model when +using `inout` parameters anymore. + +## Acknowledgments + +Thanks to Robert Widmann for the original underscored implementation of +`__owned` and `__shared`: [https://forums.swift.org/t/ownership-annotations/11276](https://forums.swift.org/t/ownership-annotations/11276). + +## Revision history + +The [first reviewed revision](https://github.com/swiftlang/swift-evolution/blob/3f984e6183ce832307bb73ec72c842f6cb0aab86/proposals/0377-parameter-ownership-modifiers.md) +of this proposal used `take` and `taking` as the name of the callee-destroy convention. + +The [second reviewed revision](https://github.com/swiftlang/swift-evolution/blob/e3966645cf07d6103561454574ab3e2cc2b48ee9/proposals/0377-parameter-ownership-modifiers.md) +used the imperative forms, `consume` and `borrow`, as parameter modifiers, +which were changed to the gerunds `consuming` and `borrowing` in review. The +proposal was originally accepted after these revisions. + +The current revision alters the originally-accepted proposal to make it so that +`borrowing` and `consuming` parameter bindings are not implicitly copyable, +and introduces a `copy x` operator that can be used to explicitly allow copies +where needed. diff --git a/proposals/0378-package-registry-auth.md b/proposals/0378-package-registry-auth.md new file mode 100644 index 0000000000..7a9f9f825f --- /dev/null +++ b/proposals/0378-package-registry-auth.md @@ -0,0 +1,449 @@ +# Package Registry Authentication + +* Proposal: [SE-0378](0378-package-registry-auth.md) +* Author: [Yim Lee](https://github.com/yim-lee) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift-package-manager#5838](https://github.com/apple/swift-package-manager/pull/5838) +* Review: ([pitch](https://forums.swift.org/t/pitch-package-registry-authentication/61047)), ([review](https://forums.swift.org/t/se-0378-swift-package-registry-authentication/61436)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0378-swift-package-registry-authentication/62556)) + +## Introduction + +A package registry may require authentication for some or all of +its API in order to identify user performing the action and authorize +the request accordingly. + +## Motivation + +Common authentication methods used by web services include basic +authentication, access token, and OAuth. SwiftPM supports only basic +authentication today, which limits its abilities to interact with +package registry services. + +## Proposed solution + +We propose to modify the `swift package-registry` command and registry +configuration to add token authentication support. The changes should also +ensure there is flexibility to add other authentication methods in the future. + +The design draws inspiration from [`docker login`](https://docs.docker.com/engine/reference/commandline/login/) and [`npm login`](https://docs.npmjs.com/cli/v8/commands/npm-adduser), +in that there will be a single command for user to verify and persist +registry credentials. + +## Detailed design + +### Changes to `swift package-registry` command + +Instead of the `swift package-registry set` subcommand and the `--login` +and `--password` options as proposed in [SE-0292](0292-package-registry-service.md) originally, +we propose the new `login` and `logout` subcommands for adding/removing +registry credentials. + +#### New `login` subcommand + +Log in to a package registry. SwiftPM will verify the credentials using +the registry service's [`login` API](#login-api). If it returns a successful +response, credentials will be persisted to the operating system's +credential store if supported, or the user-level netrc file otherwise. +The user-level configuration file located at `~/.swiftpm/configuration/registries.json` +will also be updated. + +```manpage +SYNOPSIS + swift package-registry login [options] +OPTIONS: + --username Username + --password Password + + --token Access token + + --no-confirm Allow writing to netrc file without confirmation + --netrc-file Specify the netrc file path + --netrc Use netrc file even in cases where other credential stores are preferred +``` + +`url` should be the registry's base URL (e.g., `https://example-registry.com`). +In case the location of the `login` API is something other than `/login` +(e.g., `https://example-registry.com/api/v1/login`), provide the full URL. + +The URL must be HTTPS. + +The table below shows the supported authentication types and their +required option(s): + +| Authentication Method | Required Option(s) | +| --------------------- | -------------------------- | +| Basic | `--username`, `--password` | +| Token | `--token` | + +The tool will analyze the provided options to determine the authentication +type and prompt (i.e., interactive mode) for the password/token if it +is missing. For example, if only `--username` is present, the tool +assumes basic authentication and prompts for the password. + +For non-interactive mode, simply provide the `--password` or `--token` +option as required or make sure the secret is present in credential storage. + +If the operating system's credential store is not supported, the +tool will prompt user for confirmation before writing credentials +to the less secured netrc file. Use `--no-confirm` to disable +this confirmation. + +To force usage of netrc file instead of the operating system's +credential store, pass the `--netrc` flag. + +##### Example: basic authentication (macOS, interactive) + +```console +> swift package-registry login https://example-registry.com \ + --username jappleseed +Enter password for 'jappleseed': + +Login successful. +Credentials have been saved to the operating system's secure credential store. +``` + +An entry for `example-registry.com` would be added to Keychain. + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic" + }, + ... + }, + ... +} +``` + +##### Example: basic authentication (operating system's credential store not supported, interactive) + +```console +> swift package-registry login https://example-registry.com \ + --username jappleseed +Enter password for 'jappleseed': + +Login successful. + +WARNING: Secure credential store is not supported on this platform. +Your credentials will be written out to netrc file. +Continue? (Yes/No): Yes + +Credentials have been saved to netrc file. +``` + +An entry for `example-registry.com` would be added to the netrc file: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic" + }, + ... + }, + ... +} +``` + +##### Example: basic authentication (use netrc file instead of operating system's credential store, interactive) + +```console +> swift package-registry login https://example-registry.com \ + --username jappleseed + --netrc +Enter password for 'jappleseed': + +Login successful. + +WARNING: You choose to use netrc file instead of the operating system's secure credential store. +Your credentials will be written out to netrc file. +Continue? (Yes/No): Yes + +Credentials have been saved to netrc file. +``` + +An entry for `example-registry.com` would be added to the netrc file: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic" + }, + ... + }, + ... +} +``` + +##### Example: basic authentication (operating system's credential store not supported, non-interactive) + +```console +> swift package-registry login https://example-registry.com \ + --username jappleseed \ + --password alpine \ + --no-confirm + +Login successful. +Credentials have been saved to netrc file. +``` + +An entry for `example-registry.com` would be added to the netrc file: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic" + }, + ... + }, + ... +} +``` + +
+ +##### Example: basic authentication (operating system's credential store not supported, non-interactive, non-default `login` URL) + +```console +> swift package-registry login https://example-registry.com/api/v1/login \ + --username jappleseed \ + --password alpine \ + --no-confirm + +Login successful. +Credentials have been saved to netrc file. +``` + +An entry for `example-registry.com` would be added to the netrc file: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic", + "loginAPIPath": "/api/v1/login" + }, + ... + }, + ... +} +``` + +##### Example: token authentication + +```console +> swift package-registry login https://example-registry.com \ + --token jappleseedstoken +``` + +An entry for `example-registry.com` would be added to the operating +system's credential store if supported, or the user-level netrc +file otherwise: + +``` +machine example-registry.com +login token +password jappleseedstoken +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires token authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "token" + }, + ... + }, + ... +} +``` + +#### New `logout` subcommand + +Log out from a registry. Credentials are removed from the operating system's +credential store if supported, and the user-level configuration file +(`registries.json`). + +To avoid accidental removal of sensitive data, netrc file needs to be +updated manually by the user. + +```manpage +SYNOPSIS + swift package-registry logout +``` + +### Changes to registry configuration + +We will introduce a new `authentication` key to the user-level +`registries.json` file, which by default is located at +`~/.swiftpm/configuration/registries.json`. Any package +registry that requires authentication must have a corresponding +entry in this dictionary. + +```json +{ + "registries": { + "[default]": { + "url": "https://example-registry.com" + } + }, + "authentication": { + "example-registry.com": { + "type": , // One of: "basic", "token" + "loginAPIPath": // Optional. Overrides the default API path (i.e., /login). + } + }, + "version": 1 +} +``` + +`type` must be one of the following: +* `basic`: username and password +* `token`: access token + +Credentials are to be specified in the native credential store +of the operating system if supported, otherwise in the user-level +netrc file. (Only macOS Keychain will be supported in the +initial feature release; more might be added in the future.) + +See [credential storage](#credential-storage) for more details on configuring +credentials for each authentication type. + +### Credential storage + +SwiftPM will always use the most secure way to handle credentials +on the platform. In general, this would mean using the operating +system's credential store (e.g., Keychain on macOS). It falls +back to netrc file only if there is no other solution available. + +#### Basic Authentication + +##### macOS Keychain + +Registry credentials should be stored as "Internet password" +items in the macOS Keychain. The "item name" should be the +registry URL, including `https://` (e.g., `https://example-registry.com`). + +##### netrc file (if operating system's credential store is not supported) + +A netrc entry for basic authentication looks as follows: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +By default, SwiftPM looks for netrc file in the user's +home directory. A custom netrc file can be specified using +the `--netrc-file` option. + +#### Token Authentication + +User can configure access token for a registry as similarly +done for basic authentication, but with `token` as the login/username +and the access token as the password. + +For example, a netrc entry would look like: + +``` +machine example-registry.com +login token +password jappleseedstoken +``` + +### Additional changes in SwiftPM + +1. Only the user-level netrc file will be used. Project-level netrc file will not be supported. +2. SwiftPM will perform lookups in one credential store only. For macOS, it will be Keychain. For all other platforms, it will be the user-level netrc file. +3. The `--disable-keychain` and `--disable-netrc` options will be removed. + +### New package registry service API + +A package registry that requires authentication must implement +the new API endpoint(s) covered in this section. + +#### `login` API + +SwiftPM will send a HTTP `POST` request to this API to validate +user credentials provided by the `login` subcommand. + +The default API path is `/login`, but this [can be overridden](#override-login-url) +by providing the full API URL to the `login` subcommand. + +The API request will include an `Authorization` HTTP header +constructed as follows: + +* Basic authentication: `Authorization: Basic ` +* Token authentication: `Authorization: Bearer ` + +The registry service must return HTTP status `200` in the +response if login is successful, and `401` otherwise. + +In case the registry service does not support an authentication method, +it should return HTTP status `501`. + +SwiftPM will persist user credentials to local credential store +if login is successful. + +## Security + +This proposal moves SwiftPM to use operating system's native credential +store (e.g., macOS Keychain) on supported platforms, which should yield +better security. + +We are also eliminating the use of project-level netrc file. This should +prevent accidental checkin of netrc file and thus leakage of sensitive +information. + +## Impact on existing packages + +This proposal eliminates the project-level netrc file. There should be +no other impact on existing packages. + diff --git a/proposals/0379-opt-in-reflection-metadata.md b/proposals/0379-opt-in-reflection-metadata.md new file mode 100644 index 0000000000..ddbcb87a6c --- /dev/null +++ b/proposals/0379-opt-in-reflection-metadata.md @@ -0,0 +1,244 @@ +# Swift Opt-In Reflection Metadata + +* Proposal: [SE-0379](0379-opt-in-reflection-metadata.md) +* Authors: [Max Ovtsin](https://github.com/maxovtsin) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Returned for revision** +* Implementation: [apple/swift#34199](https://github.com/apple/swift/pull/34199) +* Review: ([first pitch](https://forums.swift.org/t/proposal-opt-in-reflection-metadata/40981)) ([second pitch](https://forums.swift.org/t/pitch-2-opt-in-reflection-metadata/41696)) ([third pitch](https://forums.swift.org/t/pitch-3-opt-in-reflection-metadata/58852)) ([review](https://forums.swift.org/t/se-0379-opt-in-reflection-metadata/61714)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0379-opt-in-reflection-metadata/62390)) + +## Introduction + +This proposal seeks to increase the safety, efficiency, and secrecy of Swift Reflection Metadata by improving the existing mechanism and providing the opportunity to express a requirement on Reflection Metadata in APIs that consume it. + + +## Motivation + +There are two main kinds of Swift metadata emitted by the compiler: + +1. Core Metadata (type metadata records, nominal type descriptors, etc). +2. Reflection metadata (reflection metadata field descriptors). + +Core metadata must constantly be emitted and may only be stripped if provenly not used. (This kind of metadata isn't affected by this proposal.) +On the other hand, reflection metadata contains optional information about declaration fields - their names and references to their types. +The language's runtime features don't use this metadata, and the emission may be skipped if such types aren't passed to reflection-consuming APIs. + + +Currently, there is no way to selectively enable the emission of reflectable metadata for a type or understand if an API consumes reflection metadata under the hood. +Moreover, compiler's flags exist that allow to completely disable emission. + +A developer has two ways right now - either +1. To enable Reflection in full just in case. +2. To try to guess which used APIs consume Reflection, and enable it only for modules that use such APIs. + +Both of those options have flaws. The first one leads to excessive contribution of reflection metadata to binary size and might affects the secrecy of generated code. +The second one isn't safe because many APIs are black boxes. If the developer's guess is wrong, an app might behave not as expected at runtime. + +Furthermore, APIs can use Reflection Metadata differently. Some like `print`, `debugPrint`, and `dump` will still work with disabled reflection, but the output will be limited. +Others, like SwiftUI, rely on it and won't work correctly if the reflection metadata is missing. +While the former can benefit as well, the main focus of this proposal is on the latter. + +A developer can mistakenly turn off Reflection Metadata for a Swift module and won't be warned at compile-time if APIs that consume reflection are used by that module. +An app with such a module won't behave as expected at runtime which may be challenging to notice and track down such bugs back to Reflection. +For instance, SwiftUI implementation uses reflection metadata from user modules to trigger the re-rendering of the view hierarchy when a state has changed. +If for some reason a user module was compiled with metadata generation disabled, changing the state won't trigger that behavior and will cause inconsistency +between state and representation which will make such API less safe since it becomes a runtime issue rather than a compile-time one. + +On the other hand, excessive Reflection metadata may be preserved in a binary even if not used, because there is currently no way to statically determine its usage. +There was an attempt to limit the amount of unused reflection metadata by improving the Dead Code Elimination LLVM pass, but in many cases +it’s still preserved in the binary because it’s referenced by Full Type Metadata. This prevents Reflection Metadata from being stripped. +This unnecessarily increases the binary size and may simplify reverse-engineering. + +Introducing a static compilation check can help to solve both of mentioned issues by adding to the language a way to express the requirement to have Reflection metadata available at runtime. + + +## Proposed solution + +Teaching the Type-checker and IRGen to ensure Reflection metadata is preserved in a binary if reflection-consuming APIs are used, will help to move the problem from runtime to compile time. + +To achieve that, a new marker protocol `Reflectable` will be introduced. Firstly, APIs developers will gain an opportunity to express a dependency on Reflection Metadata through a generic requirement of their functions, which will make such APIs safer. +Secondly, during IRGen, the compiler will be able to selectively emit Reflection symbols for the types that explicitly conform to the `Reflectable` protocol, which will reduce the overhead from reflection symbols for cases when reflection is emitted but not consumed. + +### Case Study 1: + +SwiftUI: +```swift +protocol SwiftUI.View: Reflectable {} +class NSHostingView where Content : View { + init(rootView: Content) { ... } +} +``` +User module: +```swift +import SwiftUI + +struct SomeModel {} + +struct SomeView: SwiftUI.View { + var body: some View { + Text("Hello, World!") + .frame(...) + } +} + +window.contentView = NSHostingView(rootView: SomeView()) +``` +Reflection metadata for `SomeView` will be emitted because it implicitly conforms to `Reflectable` protocol, while for `SomeModel` Reflection metadata won't be emitted. If the user module gets compiled with the reflection metadata disabled, the compiler will emit an error. + + +### Case Study 2: + +Framework: +```swift +public func foo(_ t: T) { ... } +``` +User module: +```swift +struct Bar: Reflectable {} +foo(Bar()) +``` +Reflection metadata for `Bar` will be emitted because it explicitly conforms to Reflectable protocol. Without conformance to Reflectable, an instance of type Bar can't be used on function `foo`. If the user module gets compiled with the reflection metadata disabled, the compiler will emit an error. + + +### Conditional and Force casts (`as? Reflectable`, `as! Reflectable`, `is Reflectable`) + +We also propose to allow conditional and force casts to the `Reflectable` protocol, which would succeed only if Reflection Metadata related to a type is available at runtime. This would allow developers to explicitly check if reflection metadata is present and based on that fact branch the code accordingly. + +```swift +public func conditionalUse(_ t: T) { + if let _t = t as? Reflectable { // Consume Reflection metadata + } else { // Back to default implementation } +} + +public func forceUse(_ t: T) { + debugPrint(t as! Reflectable) // Will crash if reflection metadata isn't available +} + +public func testIsReflectable(_ t: T) -> Bool { + return t is Reflectable // returns True if reflection is available +} +``` + +### Behavior change for Swift 6 + +Starting with Swift 6, we propose to enable Opt-In mode by default, to make the user experience consistent and safe. +However, if full reflection isn't enabled with a new flag (`-enable-full-reflection-metadata`), the emission of reflection metadata will be skipped for all types that don't conform to the `Reflectable` protocol. +This may cause changes in the behavior of the code that wasn't audited to conform to Reflectable and uses reflection-consuming APIs. + +For instance, stdlib's APIs like `dump`, `debugPrint`, `String(describing:)` will be returning limited output. +Library authors will have to prepare their APIs for Swift 6 and introduce generic requirements on `Reflectable` in their APIs. + +We also propose to deprecate the compiler's options that can lead to missing reflection - `-reflection-metadata-for-debugger-only` and `-disable-reflection-metadata` and starting with Swift 6, ignore these arguments in favor of the default opt-in mode. + + +### No stdlib behavior changes + +In Swift `Mirror(reflecting:)` is the only official way to access Reflection metadata, all other APIs are using it under the hood. +We intentionally do not propose adding a Reflectable constraint on Mirror type, because it would impose restrictions on those developers who still don't want to require it and consume Reflection optionally. +If the presence of reflection metadata is mandatory, the requirement on Reflectable protocol should be expressed in the signatures of calling functions. + + +## Detailed design + +To decide when to emit reflection metadata IRGen will check the conformance of a type to the `Reflectable` protocol and if the type conforms, IRGen will emit reflection symbols. + +Conformance to Reflectable should be only allowed at type declarations level, to avoid confusing behavior, when a developer adds conformance on an imported from another module type that doesn't have reflection enabled. + +Transitive conformance to Reflectable should be allowed to give API authors an opportunity to hide reflection logic from APIs users as implementation details. + +```swift +// Library +public protocol Foo: Reflectable {} +public func consume(_ t: T) {} + +// User +struct Bar: Foo {} // Reflection is emitted +consume(Bar()) +``` + +### Changes for debugging + +Since Reflection metadata might be used by the debugger, we propose to always keep that metadata +if full emission of debugging information is enabled (with `-gdwarf-types` or `-g` flags). +However, such Reflection metadata won't be accessible through the nominal type descriptor +which will avoid inconsistencies in API behavior between Release and Debug modes. + +### Changes in flags + +To handle behavior change between Swift pre-6 and 6, we can introduce a new upcoming feature, +which will allow to enable Opt-In mode explicitly for Swift pre-6 with `-enable-upcoming-feature OptInReflection` and will set this mode by default in Swift 6. + +A new flag `-enable-full-reflection-metadata` will also have to be introduced to allow developers to enable reflection in full if they desire in Swift 6 and later. + +For Swift 6, flags `-disable-reflection-metadata` and `-emit-reflection-for-debugger` will be a no-op, to ensure the reflection metadata is always available when needed. + +1. Reflection Disabled (`-disable-reflection-metadata` and `-reflection-metadata-for-debugger-only`) +- For Swift pre-6 emit Reflection metadata only if full debugging information is enabled. +- If there is a type in a module conforming to `Reflectable`, the compiler will emit an error. +- A no-op in Swift 6 and later (Opt-in mode is enabled by default). + +2. Opt-In Reflection (`-enable-upcoming-feature OptInReflection`) +- If debugging is disabled, emit only for types that conform to `Reflectable`. +- Will emit reflection in full for all types if debugging is enabled. +- For Swift pre-6 will require an explicit flag, for Swift 6 will be enabled by default. + +3. Fully enabled (`-enable-full-reflection-metadata`) +- Always emit reflection metadata for all types regardless of debugging information level. +- Conformance to Reflectable will be synthesized for all types to allow usage of reflection-consuming APIs. +- Current default level for Swift pre-6. + +Introducing a new flag to control the feature will allow us to safely roll it out and avoid breakages of the existing code. For those modules that get compiled with fully enabled metadata, nothing will change (all symbols will stay present). For modules that have the metadata disabled, but are consumers of reflectable API, the compiler will emit the error enforcing the guarantee. + +### Casts implementation + +Casting might be a good way to improve the feature's ergonomics because currently there is no way to check if reflection is available at runtime. (`Mirror.children.count` doesn't really help because it doesn't distinguish between the absence of reflection metadata and the absence of fields on a type) + +To implement this feature, we propose to introduce a new runtime function `swift_reflectableCast`, and emit a call to it instead of `swift_dynamicCast`during IRGen if Reflectable is a target type. + +Because of the fact that the compiler emits a call to that function at compile-time, all casts must be statically visible. +All other cases like implicit conversion to `Reflectable` must be banned. +This could be done at CSSimplify, when a new conversion constraint is introduced between a type variable and `Reflectable` type, the compiler will emit an error. + +```swift +func cast(_ x: U) -> T { + return x as! T +} +let a = cast(1) as Reflectable // expression can't be implicitly converted to Reflectable; use 'as? Reflectable' or 'as! Reflectable' instead +let b: Reflectable = cast(1) // expression can't be implicitly converted to Reflectable; use 'as? Reflectable' or 'as! Reflectable' instead +``` +Some diagnostics and optimizations will also have to be disabled even if conformance is statically visible to the compiler because all casts will have to go through the runtime call. + +**Availability checks** +Since reflectable casting will require a new runtime function, it should be gated by availability checks. If a deployment target is lower than supported, an error will be emitted. However, it might be possible to embed new runtime functions into a compatibility library for back deployment. + + +## Source compatibility + +The change won’t break source compatibility in versions prior to Swift 6, because of the gating by the new flag. +If as proposed, it’s enabled by default in Swift 6, the code with types that has not been audited to conform to the `Reflectable` protocol will fail to compile if used with APIs that consume the reflection metadata. + + +## Effect on ABI stability + +`Reflectable` is a marker protocol, which doesn't have a runtime representation, has no requirements and doesn't affect ABI. + +## Effect on API resilience + +This proposal has no effect on API resilience. + +## Alternatives considered + +Dead Code Elimination and linker optimisations were also considered as a way to reduce the amount of present Reflection metadata in release builds. +The optimiser could use a conformance to a `Reflectable` protocol as a hint about what reflection metadata should be preserved. +However, turned out it was quite challenging to statically determine all usages of Reflection metadata even with hints. + +It was also considered to use an attribute `@reflectable` on nominal type declaration to express the requirement to have reflection metadata, however, a lot of logic had to be re-implemented outside of type-checker to ensure all guarantees are fulfilled. + +## Future directions + +Currently, there is only one kind of Reflection Metadata - Field Descriptor Metadata. In the future, it is possible that other kinds will be added (e.g methods, computed properties, etc) `Reflectable` should be able to cover all of them. +If this proposal is approved, it will become easier and more native to migrate Codable to the usage of Reflection metadata for encoding/decoding logic instead of autogenerating code at compile time. + +## Acknowledgments + +Thanks to [Joe Groff](https://github.com/jckarter) for various useful pieces of advice, general help along the way, and for suggesting several useful features like Reflectable casts! diff --git a/proposals/0380-if-switch-expressions.md b/proposals/0380-if-switch-expressions.md new file mode 100644 index 0000000000..a4ad2fd3db --- /dev/null +++ b/proposals/0380-if-switch-expressions.md @@ -0,0 +1,478 @@ +# `if` and `switch` expressions + +* Proposal: [SE-0380](0380-if-switch-expressions.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Hamish Knight](https://github.com/hamishknight) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#612178](https://github.com/apple/swift/pull/62178), including a downloadable toolchain. +* Review: ([pitch](https://forums.swift.org/t/pitch-if-and-switch-expressions/61149)), ([review](https://forums.swift.org/t/se-0380-if-and-switch-expressions/61899)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0380-if-and-switch-expressions/62695)) + +## Introduction + +This proposal introduces the ability to use `if` and `switch` statements as expressions, for the purpose of: +- Returning values from functions, properties, and closures; +- Assigning values to variables; and +- Declaring variables. + +## Motivation + +Swift has always had a terse but readable syntax for closures, which allows the `return` to be omitted when the body is a single expression. [SE-0255: Implicit Returns from Single-Expression Functions](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0255-omit-return.md) extended this to functions and properties with a single expression as their body. + +This omission of the `return` keyword is in keeping with Swift's low-ceremony approach, and is in common with many of Swift's peer "modern" languages. However, Swift differs from its peers in the lack of support for `if` and `switch` expressions. + +In some cases, this causes ceremony to make a return (ahem), for example: + +```swift +public static func width(_ x: Unicode.Scalar) -> Int { + switch x.value { + case 0..<0x80: return 1 + case 0x80..<0x0800: return 2 + case 0x0800..<0x1_0000: return 3 + default: return 4 + } +} +``` + +In other cases, a user might be tempted to lean heavily on the harder-to-read ternary syntax: + +```swift +let bullet = isRoot && (count == 0 || !willExpand) ? "" + : count == 0 ? "- " + : maxDepth <= 0 ? "▹ " : "▿ " +``` + +Opinions vary on this kind of code, from enthusiasm to horror, but it's accepted that it's reasonable to find this syntax _too_ terse. + +Another option is to use Swift's definite initialization feature: + +```swift +let bullet: String +if isRoot && (count == 0 || !willExpand) { bullet = "" } +else if count == 0 { bullet = "- " } +else if maxDepth <= 0 { bullet = "▹ " } +else { bullet = "▿ " } +``` + +Not only does this add the ceremony of explicit typing and assignment on each branch (perhaps tempting overly terse variable names), it is only practical when the type is easily known. It cannot be used with opaque types, and is very inconvenient and ugly if the type is a complex nested generic. + +Programmers less familiar with Swift might not know this technique, so they may be tempted to take the approach of `var bullet = ""`. This is more bug prone where the default value may not be desired in _any_ circumstances, but definitive initialization won't ensure that it's overridden. + +Finally, a closure can be used to simulate an `if` expression: + +```swift +let bullet = { + if isRoot && (count == 0 || !willExpand) { return "" } + else if count == 0 { return "- " } + else if maxDepth <= 0 { return "▹ " } + else { return "▿ " } +}() +``` + +This also requires `return`s, plus some closure ceremony. But here the `return`s are more than ceremony – they require extra cognitive load to understand they are returning from a closure, not the outer function. + +This proposal introduces a new syntax that avoids all of these problems: + +```swift +let bullet = + if isRoot && (count == 0 || !willExpand) { "" } + else if count == 0 { "- " } + else if maxDepth <= 0 { "▹ " } + else { "▿ " } +``` + +Similarly, the `return` ceremony could be dropped from the earlier example: + +```swift +public static func width(_ x: Unicode.Scalar) -> Int { + switch x.value { + case 0..<0x80: 1 + case 0x80..<0x0800: 2 + case 0x0800..<0x1_0000: 3 + default: 4 + } +} +``` + +Both these examples come from posts by [Nate Cook](https://forums.swift.org/t/if-else-expressions/22366/48) and [Michael Ilseman](https://forums.swift.org/t/omitting-returns-in-string-case-study-of-se-0255/24283), documenting many more examples where the standard library code would be much improved by this feature. + + +## Detailed Design + +`if` and `switch` statements will be usable as expressions, for the purpose of: + +- Returning values from functions, properties, and closures (either with implicit or explicit `return`); +- Assigning values to variables; and +- Declaring variables. + +There are of course many other places where an expression can appear, including as a sub-expression, or as an argument to a function. This is not being proposed at this time, and is discussed in the future directions section. + +For an `if` or `switch` to be used as an expression, it would need to meet these criteria: + +**Each branch of the `if`, or each `case` of the `switch`, must be a single expression.** + +Each of these expressions becomes the value of the overall expression if the branch is chosen. + +This does have the downside of requiring fallback to the existing techniques when, for example, a single expression has a log line above it. This is in keeping with the current behavior of `return` omission. + +An exception to this rule is if a branch either explicitly throws, or terminates the program (e.g. with `fatalError`), in which case no value for the overall expression needs to be produced. In these cases, multiple expressions could be executed on that branch prior to that point. + +In the case where a branch throws, either because a call in the expression throws (which requires a `try`) or with an explicit `throw`, there is no requirement to mark the overall expression with an additional `try` (e.g. before the `if`). + +Within a branch, further `if` or `switch` expressions may be nested. + +**Each of those expressions, when type checked independently, must produce the same type.** + +This has two benefits: it dramatically simplifies the compiler's work in type checking the expression, and it makes it easier to reason about both individual branches and the overall expression. + +It has the effect of requiring more type context in ambiguous cases. The following code would _not_ compile: + +```swift +let x = if p { 0 } else { 1.0 } +``` + +since when type checked individually, `0` is of type `Int`, and `1.0` is of type `Double`. The fix would be to disambiguate each branch. In this case, either by rewriting `0` as `0.0`, or by providing type context e.g. `0 as Double`. + +This can be resolved by providing type context to each of the branches: + +```swift +let y: Float = switch x.value { + case 0..<0x80: 1 + case 0x80..<0x0800: 2.0 + case 0x0800..<0x1_0000: 3.0 + default: 4.5 +} +``` + +This decision is in keeping with other recent proposals such as [SE-0244: Opaque Result Types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md): + +```swift +// Error: Function declares an opaque return type 'some Numeric', but the +// return statements in its body do not have matching underlying types +func f() -> some Numeric { + if Bool.random() { + return 0 + } else { + return 1.0 + } +} +``` + +This rule will require explicit type context for declarations in order to determine the type of `nil` literals: + +```swift +// invalid: +let x = if p { nil } else { 2.0 } +// valid with required type context: +let x: Double? = if p { nil } else { 2.0 } +``` + +Of course, when returning from a function or assigning to an existing variable, this type context is always provided. + +It is also in keeping with [SE-0326: Enable multi-statement closure parameter/result type inference]( https://github.com/swiftlang/swift-evolution/blob/main/proposals/0326-extending-multi-statement-closure-inference.md): + +```swift +func test(_: (Int?) -> T) {} + +// invalid +test { x in + guard let x { return nil } + return x +} + +// valid with required type context: +test { x -> Int? in + guard let x { return nil } + return x +} +``` + +It differs from the behavior of the ternary operator (`let x = p ? 0 : 1.0` compiles, with `x: Double`). + +However, the impact of bidirectional inference on the performance of the type checker would likely prohibit this feature from being implemented today, even if it were considered preferable. This is especially true in cases where there are many branches. This decision could be revisited in future: switching to full bidirectional type inference may be source breaking in theory, but probably not in practice (the proposal authors can't think of any examples where it would be). + +Bidirectional inference also makes it very difficult to reason about each of the branches individually, leading to sometimes unexpected results: + +```swift +let x = if p { + [1] +} else { + [1].lazy.map(expensiveOperation) +} +``` + +With full bidirectional inference, the `Array` in the `if` branch would force the `.lazy.map` in the `else` branch to be unexpectedly eager. + +The one exception to this rule is that some branches could produce a `Never` type. This would be allowed, so long as all non-`Never` branches are of the same type: + +```swift +// x is of type Int, discounting the type of the second branch +let x = if .random() { + 1 +} else { + fatalError() +} +``` + +**In the case of `if` statements, the branches must include an `else`** + +This rule is consistent with the current rules for definitive initialization and return statements with `if` e.g. + +```swift +func f() -> String { + let b = Bool.random() + if b == true { + return "true" + } else if b == false { // } else { here would compile + return "false" + } +} // Missing return in global function expected to return 'String' +``` + +This could be revisited in the future across the board (to DI, return values, and `if` expressions) if logic similar to that of exhaustive switches were applied, but this would be a separate proposal. + +**The expression is not part of a result builder expression** + +`if` and `switch` statements are already expressions when used in the context of a result builder, via the `buildEither` function. This proposal does not change this feature. + +The variable declaration form of an `if` will be allowed in result builders. + +**Pattern matching bindings may occur within an `if` or `case`** + +For example, returns could be dropped from + +```swift + private func balance() -> Tree { + switch self { + case let .node(.B, .node(.R, .node(.R, a, x, b), y, c), z, d): + .node(.R, .node(.B,a,x,b),y,.node(.B,c,z,d)) + case let .node(.B, .node(.R, a, x, .node(.R, b, y, c)), z, d): + .node(.R, .node(.B,a,x,b),y,.node(.B,c,z,d)) + case let .node(.B, a, x, .node(.R, .node(.R, b, y, c), z, d)): + .node(.R, .node(.B,a,x,b),y,.node(.B,c,z,d)) + case let .node(.B, a, x, .node(.R, b, y, .node(.R, c, z, d))): + .node(.R, .node(.B,a,x,b),y,.node(.B,c,z,d)) + default: + self + } + } +``` + +and optional unwrapping could be used with `if let`: + +```swift +// equivalent to let x = foo.map(process) ?? someDefaultValue +let x = if let foo { process(foo) } else { someDefaultValue } +``` + +## Future Directions + +This proposal chooses a narrow path of only enabling expressions in the 3 cases laid out at the start. This is intended to cover the vast majority of use cases, but could be followed up by expanded functionality covering many other use cases. Further cases could be added in later proposals once the community has had a chance to use this feature in practice – including source breaking versions introduced under a language variant. + +### Full Expressions + +A feel for the kind of expressions this could produce can be found in [this commit](https://github.com/apple/swift/compare/main...hamishknight:express-yourself#diff-7db38bc4b6f7872e5a631989c2925f5fac21199e221aa9112afbbc9aae66a2de) which adds this functionality to the parser. + +Full expressions would include various fairly conventional examples not proposed here: + +```swift +let x = 1 + if .random() { 3 } else { 4 } +``` + +but also some pretty strange ones such as + +```swift +for b in [true] where switch b { case true: true case false: false } {} +``` + +The strange examples can mostly be considered "weird but harmless" but there are some source breaking edge cases, in particular in result builders: + +```swift +var body: some View { + VStack { + if showButton { + Button("Click me!", action: clicked) + } else { + Text("No button") + } + .someStaticProperty + } +} +``` + +In this case, if `if` expressions were allowed to have postfix member expressions (which they aren't today, even in result builders), it would be ambiguous whether this should be parsed as a modifier on the `if` expression, or as a new expression. This could only be an issue for result builders, but the parser does not have the ability to specialize behavior for result builders. Note, this issue can happen today (and is why `One` exists for Regex Builders) but could introduce a new ambiguity for code that works this way today. + +### `do` Expressions + +`do` blocks could similarly be transformed into expressions, for example: + +```swift +let foo: String = do { + try bar() +} catch { + "Error \(error)" +} +``` + +### Guard + +Often enthusiasm for `guard` leads to requests for `guard` to have parity with `if`. Returning a value from a `guard`'s else is very common, and could potentially be sugared as + +```swift +guard hasNativeStorage else { nil } +``` + +This is appealing, but is really a different proposal, of allowing omission `return` in `guard` statements. + +### Multi-statement branches + +The requirement that every branch be just a single expression leads to an unfortunate usability cliff: + +```swift +let decoded = + if isFastUTF8 { + Log("Taking the fast path") + withFastUTF8 { _decodeScalar($0, startingAt: i) } + } else + Log("Running error-correcting slow-path") + foreignErrorCorrectedScalar( + startingAt: String.Index(_encodedOffset: i)) + } +``` + +This is consistent with other cases, like multi-statement closures. But unlike in that case, where all that is needed is a `return` from the closure, this requires the user refactor the code back to the old mechanisms. + +The trouble is, there is no great solution here. The approach taken by some other languages such as rust is to allow a bare expression at the end of the scope to be the expression value for that scope. There are stylistic preferences for and against this. More importantly, this would be a fairly radical new direction for Swift, and if proposed should probably be considered for all such cases (like function and closure return values too). + +Alternatively, a new keyword could be introduced to make explicit that an expression value is being used as the value for this branch (Java uses `yield` for this in `switch` expressions). + +### Either + +As mentioned above, in result builders an `if` can be used to construct an `Either` type, which means the expressions in the branches could be of different types. + +This could be done with `if` expressions outside result builders too, and would be a powerful new feature for Swift. However, it is large in scope (including the introduction of a language-integrated `Either` type) and should be considered in a separate proposal, probably after the community has adjusted to the more vanilla version proposed here. + +## Alternatives Considered + +### Sticking with the Status Quo + +The list of [commonly rejected proposals](https://github.com/swiftlang/swift-evolution/blob/main/commonly_proposed.md) includes the subject of this proposal: + +> **if/else and switch as expressions**: These are conceptually interesting things to support, but many of the problems solved by making these into expressions are already solved in Swift in other ways. Making them expressions introduces significant tradeoffs, and on balance, we haven't found a design that is clearly better than what we have so far. + +The motivation section above outlines why the alternatives that exist today fall short. One of the reasons this proposal is narrow in scope is to bring the majority of value while avoiding resolving some of these more difficult trade-offs. + +The lack of this feature puts Swift's [claim](https://www.swift.org/about/) to be a modern programming language under some strain. It is one of the few modern languages (Go being the other notable exception) not to support something along these lines. + +### Alternative syntax + +Instead of extending the current implicit return mechanism, where a single expression is treated as the returned value, this proposal could introduce a new syntax for expression versions of `if`/`switch`. For example, in Java: + +```java +var response = switch (utterance) { + case "thank you" -> "you’re welcome"; + case "atchoo" -> "gesundheit"; + case "fire!" -> { + log.warn("fire detected"); + yield "everybody out!"; // yield = value of multi-statement branch + }; + default -> { + throw new IllegalStateException(utterance); + }; +}; +``` + +A similar suggestion was made during [SE-0255: Implicit Returns from Single-Expression Functions](https://forums.swift.org/t/se-0255-implicit-returns-from-single-expression-functions/), where an alternate syntax for single-expression functions was discussed e.g. `func sum() -> Element = reduce(0, +)`. In that case, the core team did not consider introduction of a separate syntax for functions to be sufficiently motivated. + +The main benefit to the alternate `->` syntax is to make it more explicit, but comes at the cost of needing to know about two different kinds of switch syntax. Note that this is orthogonal to, and does not solve, the separate goal of providing a way of explicitly "yielding" an expression value in the case of multi-statement branches (also shown here in this java example) versus taking the "last expression" approach. + +Java did not introduce this syntax for `if` expressions. Since this is a goal for Swift, this implies: + +```swift +let x = + if .random() -> 1 + else -> fatalError() +``` + +However, this then poses an issue when evolving to multi-statement branches. Unlike with `switch`, these would require introducing braces, leaving a combination of both braces _and_ a "this is an expression" sigil: + +```swift +let x = + if .random() -> { + let y = someComputation() + y * 2 + } else -> fatalError() +``` + +Unlike Java and C, this "braces for 2+ arguments" style of `if` is out of keeping in Swift. + +It is also not clear if the `->` would work well if expression status is brought to more kinds of statement e.g. + +```swift +let foo: String = + do -> + try bar() + catch ns as NSError -> + "Error \(error)" +``` + +or mixed branches with expressions and a return: + +```swift +let x = + if .random() -> 1 + else -> return 2 +``` + +If a future direction of full expressions is considered, the `->` form may not work so well, especially when single-line expressions are desired e.g. + +```swift +// is this (p ? 1 : 2) + 3 +// or p ? 1 : (2 + 3) +let x = if p -> 1 else -> 2 + 3 +``` + +### Support for control flow + +An earlier version of this proposal allowed use of `return` in a branch. Similar to `return`, statements that `break` or `continue` to a label, were considered a future direction. + +Allowing new control flow out of expressions could be unexpected and error-prone. Swift currently only allows control flow out of expressions through thrown errors, which must be explicitly marked with `try` (or, in the case of `if` or `switch` branches, with `throw`) as an indication of the control flow to the programmer. Allowing other control flow out of expressions would undermine this principle. The control flow impact of nested return statements would become more difficult to reason about if we extend SE-0380 to support multiple-statement branches in the future. The use-cases for this functionality presented in the review thread were also fairly niche. Given the weak motivation and the potential problems introduced, the Language Workgroup accepts SE-0380 without this functionality. + +## Source compatibility + +As proposed, this addition has one source incompatibility, related to unreachable code. The following currently compiles, albeit with a warning that the `if` statement is unreachable (and the values in the branches unused): + +```swift +func foo() { + return + if .random() { 0 } else { 0 } +} +``` + +but under this proposal, it would fail to compile with an error of "Unexpected non-void return value in void function" because it now parses as returning the `Int` expression from the `if`. This could be fixed in various ways (with `return;` or `return ()` or by `#if`-ing out the dead code explicitly). + +Another similar case can occur if constant evaluation leads the compiler to ignore dead code: + +```swift +func foo() -> Int { + switch true { + case true: + return 0 + case false: + print("unreachable") + } +} +``` + +This currently _doesn't_ warn that the `false` case is unreachable (though probably should), but without special handling would after this proposal result in a type error that `()` does not match expected type `Int`. + +Given these examples all require dead code, it seems reasonable to accept this source break rather than gate this change under a language version or add special handling to avoid the break. + +## Effect on ABI stability + +This proposal has no impact on ABI stability. + +## Acknowledgments + +Much of this implementation layers on top of ground work done by [Pavel Yaskevich](https://github.com/xedin), particularly the work done to allow [multi-statement closure type inference](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0326-extending-multi-statement-closure-inference.md). + +Both [Nate Cook](https://forums.swift.org/t/if-else-expressions/22366/48) and [Michael Ilseman](https://forums.swift.org/t/omitting-returns-in-string-case-study-of-se-0255/24283) provided analysis of use cases in the standard library and elsewhere. Many community members have made a strong case for this change, most recently [Dave Abrahams](https://forums.swift.org/t/if-else-expressions/22366). diff --git a/proposals/0381-task-group-discard-results.md b/proposals/0381-task-group-discard-results.md new file mode 100644 index 0000000000..27ec7607b0 --- /dev/null +++ b/proposals/0381-task-group-discard-results.md @@ -0,0 +1,267 @@ +# DiscardingTaskGroups + +* Proposal: [SE-0381](0381-task-group-discard-results.md) +* Authors: [Cory Benfield](https://github.com/Lukasa), [Konrad Malawski](https://github.com/ktoso) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#62361](https://github.com/apple/swift/pull/62361) +* Review: ([pitch](https://forums.swift.org/t/pitch-task-pools/61703)) ([review](https://forums.swift.org/t/se-0381-discardresults-for-taskgroups/62072)) ([acceptance](https://forums.swift.org/t/accepted-se-0381-discardingtaskgroups/62615)) + +### Introduction + +We propose to introduce a new type of structured concurrency task group: `Discarding[Throwing]TaskGroup`. This type of group is similar to `TaskGroup` however it discards results of its child tasks immediately. It is specialized for potentially never-ending task groups, such as top-level loops of http or other kinds of rpc servers. + +## Motivation + +Task groups are the building block of structured concurrency, allowing for the Swift runtime to relate groups of tasks together. This enables powerful features such as automatic cancellation propagation, correctly propagating errors, and ensuring well-defined lifetimes, as well as providing diagnostic information to programming tools. + +The version of Task Groups introduced in [SE-0304](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md) provides all of these features. However, it also provides the ability to propagate return values to the user of the task group. This capability provides an unexpected limitation in some use-cases. + +As users of Task Groups are able to retrieve the return values of child tasks, it implicitly follows that the Task Group preserves at least the `Result` of any completed child task. As a practical matter, the task group actually preserves the entire `Task` object. This data is preserved until the user consumes it via one of the Task Group consumption APIs, whether that is `next()` or by iterating the Task Group. + +The result of this is that Task Groups are ill-suited to running for a potentially unbounded amount of time. An example of such a use-case is managing connections accepted from a listening socket. A simplified example of such a workload might be: + +```swift +try await withThrowingTaskGroup(of: Void.self) { group in + while let newConnection = try await listeningSocket.accept() { + group.addTask { + handleConnection(newConnection) + } + } +} +``` + +As written, this task group will leak all the child `Task` objects until the listening socket either terminates or throws. If this was written for a long-running server, it is entirely possible for this Task Group to survive for a period of days, leaking thousands of Task objects. For stable servers, this will eventually drive the process into memory exhaustion, forcing it to be killed by the OS. + +The current implementation of Task Groups do not provide a practical way to avoid this issue. Task Groups are (correctly) not `Sendable`, so neither the consumption of completed `Task` results nor the submission of new work can be moved to a separate `Task`. + +The most natural attempt to avoid this unbounded memory consumption would be to attempt to occasionally purge the completed task results. An example might be: + +```swift +try await withThrowingTaskGroup(of: Void.self) { group in + while let newConnection = try await listeningSocket.accept() { + group.addTask { + handleConnection(newConnection) + } + try await group.next() + } +} +``` + +Unfortunately, all of the methods for attempting to pop the queue of completed `Task`s will suspend if all currently live child `Task`s are executing. This means that the above pattern (or any similar pattern) is at risk of occasional livelocks, where pending connections could be accepted, but the `Task` is blocked waiting for existing work to complete. + +There is only one design pattern to avoid this issue, which involves forcibly bounding the maximum concurrency of the Task Group. This pattern looks something like the below: + +```swift +try await withThrowingTaskGroup(of: Void.self) { group in + // Fill the task group up to maxConcurrency + for _ in 0..( + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout DiscardingTaskGroup) async -> GroupResult +) async -> GroupResult { ... } + +public func withThrowingDiscardingTaskGroup( + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout ThrowingDiscardingTaskGroup) async throws -> GroupResult +) async throws -> GroupResult { ... } +``` + +And the types themselves, mostly mirroring the APIs of `TaskGroup`, except that they're missing `next()` and related functionality: + +```swift +public struct DiscardingTaskGroup { + + public mutating func addTask( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Void + ) + + public mutating func addTaskUnlessCancelled( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Void + ) -> Bool + + public var isEmpty: Bool + + public func cancelAll() + public var isCancelled: Bool +} +@available(*, unavailable) +extension DiscardingTaskGroup: Sendable { } + +public struct ThrowingDiscardingTaskGroup { + + public mutating func addTask( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Void + ) + + public mutating func addTaskUnlessCancelled( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Void + ) -> Bool + + public var isEmpty: Bool + + public func cancelAll() + public var isCancelled: Bool +} +@available(*, unavailable) +extension DiscardingThrowingTaskGroup: Sendable { } +``` + +## Detailed Design + +### Discarding results + +As indicated by the name a `[Throwing]DiscardingTaskGroup` will discard results of its child tasks _immediately_ and release the child task that produced the result. This allows for efficient and "running forever" request accepting loops such as HTTP or RPC servers. + +Specifically, the first example shown in the Motivation section of this proposal, _is_ safe to be expressed using a discarding task group, as follows: + +```swift +// GOOD, no leaks! +try await withThrowingDiscardingTaskGroup() { group in + while let newConnection = try await listeningSocket.accept() { + group.addTask { + handleConnection(newConnection) + } + } +} +``` + +This code–unlike the `withThrowingTaskGroup` version shown earlier–does not leak tasks and therefore is safe and the recommended way to express such handler loops. + +### Error propagation and group cancellation + +Throwing task groups rely on the `next()` (or `waitForAll()`) being throwing and end users consuming the child tasks this way in order to surface any error that the child tasks may have thrown. It is possible for a `ThrowingTaskGroup` to explicitly collect results (and failures), and react to them, like this: + +```swift +try await withThrowingTaskGroup(of: Void.self) { group in + group.addTask { try boom() } + group.addTask { try boom() } + group.addTask { try boom() } + + try await group.next() // re-throws whichever error happened first +} // since body threw, the group and remaining tasks are immediately cancelled +``` + +The above snippet illustrates a simple case of the error propagation out of a child task, through `try await next()` (or `try await group.waitForAll()`) out of the `withThrowingTaskGroup` closure body. As soon as an error is thrown out of the closure body, the group cancels itself and all remaining tasks implicitly, finally proceeding to await all the pending tasks. + +This pattern is not possible with `ThrowingDiscardingTaskGroup` because the the results collecting methods are not available on discarding groups. In order to properly support the common use-case of discarding groups, the failure of a single task, should implicitly and _immediately_ cancel the group and all of its siblings. + +This can be seen as the implicit immediate consumption of the child tasks inspecting the task for failures, and "re-throwing" the failure automatically. The error is then also re-thrown out of the `withThrowingDiscardingTaskGroup` method, like this: + +```swift +try await withThrowingDiscardingTaskGroup() { group in + group.addTask { try boom(1) } + group.addTask { try boom(2) } + group.addTask { try boom(3) } + // whichever failure happened first, is collected, stored, and re-thrown out of the method when exiting. +} +``` + +In other words, discarding task groups follow the "one for all, and all for one" pattern for failure handling. A failure of a single child task, _immediately_ causes cancellation of the group and its siblings. + +Preventing this behavior can be done in two ways: + +- using `withDiscardingTaskGroup`, since the child tasks won't be allowed to throw, and must handle their errors in some other way, +- including normal `do {} catch {}` error handling logic inside the child-tasks, which only re-throws. + +We feel this is the right approach for this structured concurrency primitive, as we should be leaning on normal swift code patterns, rather than introduce special one-off ways to handle and deal with errors. Although, if it were necessary, we could introduce a "failure reducer" in the future. + +## Alternatives Considered + +### Introducing new "TaskPool" type (initial pitch) + +The [original pitch](https://forums.swift.org/t/pitch-task-pools/61703) introduced two new types, `TaskPool` and `ThrowingTaskPool`. These types were introduced in order to expose at the type system level the inability to iterate the pool for new tasks. This would avoid the `next()` behaviour introduced in this pitch, where `next()` always returns `nil`. This was judged a worthwhile change to justify introducing new types. + +Several reviewers of the pitch felt that this was not a sufficiently useful capability to justify the introduction of the new types, and that the pitched behaviour more properly belonged as a "mode" of operation on `TaskGroup`. In line with that feedback, this proposal has moved to using the `discardResults` option. + +### Extending [Throwing]TaskGroup with discardResults flag + +After feedback on the the initial pitch, we attempted to avoid introducing a new type, and instead handle it using a `discardResults: Bool` flag on `with[Throwing]TaskGroup()` this was fairly problematic because: + +- the group would have the `next()` method as well as `AsyncSequence` conformance present, but non-functional, i.e. always returning `nil` from `next()` which could lead to subtle bugs and confusion. +- we'd end up constraining this new option only to child task result types of `Void`, making access to this functionality a bit hard to discover + +The group would also have very different implicit cancellation behavior, ultimately leading us to conclude during the Swift Evolution review that these two behaviors should not be conflated into one type. + +### Alternate Error throwing behaviour + +The pitch proposes that `ThrowingDiscardingTaskGroup` will throw only the _first_ error thrown by a child `Task`. This means that all subsequent errors will be discarded, which is an unfortunate loss of information. Two alternative behaviours could be chosen: we could not provide `ThrowingDiscardingTaskGroup` at all, or we could throw an aggregate error that contains *all* errors thrown by the child `Task`s. + +Not allowing offering `ThrowingDiscardingTaskGroup` at all is a substantial ergonomic headache. Automatic error propagation is one of the great features of structured concurrency, and not being able to use it in servers or other long-running processes is an unnecessary limitation, especially as it's not particularly technically challenging to propagate errors. For this reason, we do not think it's wise to omit `discardResults` on `ThrowingDiscardingTaskGroup`. + +The other alternative is to throw an aggregate error. This would require that `ThrowingDiscardingTaskGroup` persist all (or almost all) errors thrown by child tasks and merge them together into a single error `struct` that is thrown. This idea is a mixed bag. + +The main advantage of throwing an aggregate error is that no information is lost. Programs can compute on all errors that were thrown, and at the very least can log or provide other metrics based on those errors. Avoiding data loss in this way is valuable, and gives programmers more flexibility. + +Throwing an aggregate error has two principal disadvantages. The first is that aggregate errors do not behave gracefully in `catch` statements. If a child task has thrown `MyModuleError`, programmers would like to write `catch MyModuleError` in order to handle it. Aggregate errors break this situation, even if only one error is thrown: programmers have to write `catch let error = error as? MyAggregateError where error.baseErrors.contains(where: { $0 is MyModuleError })`, or something else equally painful. + +The other main disadvantage is the storage bloat from `CancellationError`. The first thrown error will auto-cancel all child `Task`s. This is great, but that cancellation will likely manifest in as series of `CancellationError`s, which will presumably bubble to the top and be handled by the `ThrowingDiscardingTaskGroup`. This means that a `ThrowingDiscardingTaskGroup` will likely store a substantial collection of errors, where all but the first are `CancellationError`. This is a substantial regression in convenience for the mainline case, with additional costs in storage, without providing any more meaningful information. + +For these reasons we've chosen the middle behaviour, where only one error is thrown. We think there is merit in throwing an aggregate error, however, and we'd like community feedback on this alternative. + +### Child Task for reaping + +An alternative would be to have Task Group spin up a child `Task` that can be used to consume tasks from the group. The API surface would look something like this: + +```swift +withTaskGroupWithChildTask(of: Void.self) { group in + group.addTask { + handleConnection(newConnection) + } +} +consumer: { group in + for task in group { } +} +``` + +The advantage of this variant is that it is substantially more flexible, and allows non-`Void`-returning tasks. The downside of this variant is that it muddies the water on the question of whether Task Groups are `Sendable` (requiring a specific-exemption for this use-case) and forces users to understand the lifetime of a pair of different closures. + +## Future Directions + +### Error Handling + +A number of concerns were raised during the pitch process that the "throw the first error only" pattern may be insufficiently flexible. Community members were particularly interested in having some sort of error filter function that could be used to filter, accumulate, or discard errors as needed. + +The proposers feel that introducing this API surface in the first version of this feature adds significant complexity to this type. This requires us to be confident that the API surface proposed is going to serve the necessary use-cases, without adding unnecessary cognitive load. It's also not entirely clear where the line is between features that can be handled using `try`/`catch` and features that require a new error filter function. + +As a result, the proposal authors have elected to defer implementing anything here until there are real-world examples to generalise from. Having some sort of error filter is likely to be valuable, and the implementation will preserve the capability to implement such a function, but for now the proposal is going to be kept relatively small. diff --git a/proposals/0382-expression-macros.md b/proposals/0382-expression-macros.md new file mode 100644 index 0000000000..e3a33eb558 --- /dev/null +++ b/proposals/0382-expression-macros.md @@ -0,0 +1,602 @@ +# Expression Macros + +* Proposal: [SE-0382](0382-expression-macros.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.9)** +* Implementation: Partial implementation is available in `main` under the experimental feature flag `Macros`. An [example macro repository](https://github.com/DougGregor/swift-macro-examples) provides a way to experiment with this feature. +* Review: ([pitch](https://forums.swift.org/t/pitch-expression-macros/61499)) ([pitch #2](https://forums.swift.org/t/pitch-2-expression-macros/61861)) ([review #1](https://forums.swift.org/t/se-0382-expression-macros/62090)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0382-expression-macros/62898)) ([pitch #3](https://forums.swift.org/t/se-0382-expression-macros-mini-pitch-for-updates/62810)) ([review #2](https://forums.swift.org/t/se-0382-second-review-expression-macros/63064)) ([acceptance](https://forums.swift.org/t/accepted-se-0382-expression-macros/63495)) ([post-acceptance update](https://forums.swift.org/t/update-on-se-0382-and-se-0389-expression-macros-and-attached-macros/74094)) + +## Introduction + +Expression macros provide a way to extend Swift with new kinds of expressions, which can perform arbitrary syntactic transformations on their arguments to produce new code. Expression macros make it possible to extend Swift in ways that were only previously possible by introducing new language features, helping developers build more expressive libraries and eliminate extraneous boilerplate. + +## Motivation + +Expression macros are one part of the [vision for macros in Swift](https://github.com/swiftlang/swift-evolution/pull/1927), which lays out general motivation for introducing macros into the language. Expressions in particular are an area where the language already provides decent abstractions for factoring out runtime behavior, because one can create a function that you call as an expression from anywhere. However, with a few hard-coded exceptions like `#file` and `#line`, an expression cannot reason about or modify the source code of the program being compiled. Such use cases will require external source-generating tools, which don't often integrate cleanly with other tooling. + +## Proposed solution + +This proposal introduces the notion of expression macros, which are used as expressions in the source code (marked with `#`) and are expanded into expressions. Expression macros can have parameters and a result type, much like a function, which describes the effect of the macro expansion on the expression without requiring the macro to actually be expanded first. + +The actual macro expansion is implemented with source-to-source translation on the syntax tree: the expression macro is provided with the syntax tree for the macro expansion itself (e.g., starting with the `#` and ending with the last argument), which it can rewrite into the expanded syntax tree. That expanded syntax tree will be type-checked against the result type of the macro. + +As a simple example, let's consider a `stringify` macro that takes a single argument as input and produces a tuple containing both the original argument and also a string literal containing the source code for that argument. This macro could be used in source code as, for example: + +```swift +#stringify(x + y) +``` + +and would be expanded into + +```swift +(x + y, "x + y") +``` + +The type signature of a macro is part of its declaration, which looks a lot like a function: + +```swift +@freestanding(expression) macro stringify(_: T) -> (T, String) +``` + +### Type-checked macro arguments and results + +Macro arguments are type-checked against the parameter types of the macro prior to instantiating the macro. For example, the macro argument `x + y` will be type-checked; if it is ill-formed (for example, if `x` is an `Int` and `y` is a `String`), the macro will never be expanded. If it is well-formed, the generic parameter `T` will be inferred to the result of `x + y`, and that type is carried through to the result type of the macro. There are several benefits to this type-checked model: + +* Macro implementations are guaranteed to have well-typed arguments as inputs, so they don't need to be concerned about incorrect code being passed into the macro. +* Tools can treat macros much like functions, providing the same affordances for code completion, syntax highlighting, and so on, because the macro arguments follow the same rules as other Swift code. +* A macro expansion expression can be partially type-checked without having to expand the macro. This allows tools to still have reasonable results without performing macro expansion, as well as improving compile-time performance because the same macro will not be expanded repeatedly during type inference. + +When the macro is expanded, the expanded syntax tree is type-checked against the result type of the macro. In the `#stringify(x + y)` case, this means that if `x + y` had type `Int`, the expanded syntax tree (`(x + y, "x + y")`) is type-checked with a contextual type of `(Int, String)`. + +The type checking of macro expressions is similar to type-checking a call, allowing type inference information to flow from the macro arguments to the result type and vice-versa. For example, given: + +```swift +let (a, b): (Double, String) = #stringify(1 + 2) +``` + +the integer literals `1` and `2` would be assigned the type `Double`. + +### Syntactic translation + +Macro expansion is a syntactic operation, which takes as input a well-formed syntax tree consisting of the full macro expansion expression (e.g., `#stringify(x + y)`) and produces a syntax tree as output. The resulting syntax tree is then type-checked based on the macro result type. + +Syntactic translation has a number of benefits over more structured approaches such as direct manipulation of a compiler's Abstract Syntax Tree (AST) or internal representation (IR): + +* A macro expansion can use the full Swift language to express its effect. If it can be written as Swift source code at that position in the grammar, a macro can expand to it. +* Swift programmers understand Swift source code, so they can reason about the output of a macro when applied to their source code. This helps both when authoring and using macros. +* Source code that uses macros can be "expanded" to eliminate the use of the macro, for example to make it easier to reason about or debug, or make it work with an older Swift compiler that doesn't support macros. +* The compiler's AST and internal representation need not be exposed to clients, which would limit the ability of the compiler to evolve and improve due to backward-compatibility concerns. + +On the other hand, purely syntactic translations have a number of downsides, too: + +* Syntactic macro expansions are prone to compile-time failures, because we're effectively working with source code as strings, and it's easy to introduce (e.g.) syntax errors or type errors in the macro implementation. +* Syntactic macro expansions are re-parsed and re-type-checked, which incurs more compile-time overhead than an approach that (say) manipulated the AST or IR directly. +* Syntactic macros are not *hygienic*, meaning that the way in which a macro expansion is processed depends on the environment in which it is expanded, and can affect that environment. + +The proposed macro design attempts to mitigate these problems, but they are somewhat fundamental to the use of syntactic macros. On balance, the ease-of-use and easy-of-interpretation of syntactic macros outweighs these problems. + +### Macros defined as separate programs + +Macro definitions operate on syntax trees. Broadly speaking, there are two different ways in which a macro's expansion operation can be defined: + +* *A declarative set of transformations*: this involves extending the language with a special syntax that allows macros to define how the macro is expanded given the macro inputs, and the compiler applies those rules for each macro expansion. The C preprocessor employs a simplistic form of this, but Racket's [pattern-based macros](https://docs.racket-lang.org/guide/pattern-macros.html) and Rust's [declarative macros](https://doc.rust-lang.org/book/ch19-06-macros.html#declarative-macros-with-macro_rules-for-general-metaprogramming) offer more advanced rules that match the macro arguments to a pattern and then perform a rewrite to new syntax as described in the macro. For Swift to adopt this approach, we would likely need to invent a pattern language for matching and rewriting syntax trees. +* *An executable program that transforms the source*: this involves running a program that manipulates the program syntax directly. How the program is executed depends a lot on the environment: [Scala 3 macros](https://docs.scala-lang.org/scala3/guides/macros/macros.html) benefit from the use of the JVM so they can intertwine target code (the program being generated) and host code (where the compiler is running), whereas Rust's [procedural macros](https://doc.rust-lang.org/reference/procedural-macros.html) are built into a separate crate that the compiler interacts with. For Swift to adopt this approach, we would either need to build a complete interpreter for Swift code in the compiler, or take an approach similar to Rust's and build macro definitions as separate programs that the compiler can interact with. + +We propose the latter approach, where a macro definition is a separate program that operates on Swift syntax trees using the [swift-syntax](https://github.com/apple/swift-syntax/) package. Expression macros are defined as types that conform to the `ExpressionMacro` protocol: + +```swift +public protocol ExpressionMacro: FreestandingMacro { + /// Expand a macro described by the given freestanding macro expansion + /// within the given context to produce a replacement expression. + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> ExprSyntax +} +``` + +The `expansion(of:in:)` method takes as arguments the syntax node for the macro expansion expression (e.g., `#stringify(x + y)`) and a "context" that provides more information about the compilation context in which the macro is being expanded. It produces a macro result that includes the rewritten syntax tree. + +The specifics of `Macro`, `ExpressionMacro`, and `MacroExpansionContext` will follow in the Detailed Design section. + +### The `stringify` macro implementation + +Let's continue with the implementation of the `stringify` macro. It's a new type `StringifyMacro` that conforms to `ExpressionMacro`: + +```swift +import SwiftSyntax +import SwiftSyntaxBuilder +import SwiftSyntaxMacros + +public struct StringifyMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + guard let argument = node.argumentList.first?.expression else { + fatalError("compiler bug: the macro does not have any arguments") + } + + return "(\(argument), \(literal: argument.description))" + } +} +``` + +The `expansion(of:in:)` function is fairly small, because the `stringify` macro is relatively simple. It extracts the macro argument from the syntax tree (the `x + y` in `#stringify(x + y)`) and then forms the resulting tuple expression by interpolating in the original argument both as a value and then as source code in [a string literal](https://github.com/apple/swift-syntax/blob/main/Sources/SwiftSyntaxBuilder/ConvenienceInitializers.swift#L259-L265). That string is then parsed as an expression (producing an `ExprSyntax` node) and returned as the result of the macro expansion. This is a simple form of quasi-quoting provided by the `SwiftSyntaxBuilder` module, implemented by making the major syntax nodes (`ExprSyntax` in this case, for expressions) conform to [`ExpressibleByStringInterpolation`](https://developer.apple.com/documentation/swift/expressiblebystringinterpolation), where existing syntax nodes can be interpolated into string literals containing the expanded Swift code. + +The `StringifyMacro` struct is the implementation for the `stringify` macro declared earlier. We will need to tie these together in the source code via some mechanism. We propose to provide a builtin macro that names the module and the `ExpressionMacro` type name within the macro declaration following an `=`, e.g., + +```swift +@freestanding(expression) +macro stringify(_: T) -> (T, String) = + #externalMacro(module: "ExampleMacros", type: "StringifyMacro") +``` + +## Detailed design + +There are two major pieces to the macro design: how macros are declared and expanded within a program, and how they are implemented as separate programs. The following sections provide additional details. + +### Macro declarations + +A macro declaration is described by the following grammar: + +``` +declaration -> macro-declaration + +macro-declaration -> macro-head identifier generic-parameter-clause[opt] macro-signature macro-definition[opt] generic-where-clause[opt] + +macro-head -> attributes[opt] declaration-modifiers[opt] 'macro' + +macro-signature -> parameter-clause macro-function-signature-result[opt] + +macro-function-signature-result -> '->' type + +macro-definition -> '=' expression +``` + +The `@freestanding(expression)` attribute applies only to macros. It indicates that the macro is an expression macro. The "freestanding" terminology comes from the [macros vision document](https://github.com/swiftlang/swift-evolution/pull/1927), and is used to describe macros that are expanded with the leading `#` syntax. + +Macro signatures are function-like, with a parameter clause (that may be empty) and an optional result type. + +Macros can only be declared at file scope. They can be overloaded in the same way as functions, so long as the argument labels, parameter types, or result type differ. + +The `macro-definition` provides the implementation used to expand the macro. It is parsed as a general expression, but must always be a `macro-expansion-expression`, so all non-builtin macros are defined in terms of other macros, terminating in a builtin macro whose definition is provided by the compiler. The arguments provided within the `macro-expansion-expression` of the macro definition must either be direct references to the parameters of the enclosing macro or must be literals. The `macro-expansion-expression` is type-checked (to ensure that the argument and result types make sense), but no expansion is performed at the time of definition. Rather, expansion of the macro referenced by the `macro-definition` occurs when the macro being declared is expanded. See the following section on macro expansion for more information. + +Macro parameters may have default arguments, but those default arguments can only consist of literal expressions and other macro expansions. + +Macros can have opaque result types. The rules for uniqueness of opaque result types for macros are somewhat different from opaque result types of functions, because each macro expansion can easily produce a different type. Therefore, each macro expansion producing an opaque result type will be considered to have a distinct type, e.g., the following is ill-formed: + +```swift +@freestanding(expression) macro someMacroWithOpaqueResult() -> some Collection + +var a = #someMacroWithOpaqueResult +a = #someMacroWithOpaqueResult // cannot assign value with type of macro expansion here to opaque type from macro expansion above +``` + +### Macro expansion + +A macro expansion expression is described by the following grammar: + +``` +primary-expression -> macro-expansion-expression +macro-expansion-expression -> '#' identifier generic-argument-clause[opt] function-call-argument-clause[opt] trailing-closures[opt] +``` + +The `#` syntax for macro expansion expressions was specifically chosen because Swift already contains a number of a `#`-prefixed expressions that are macro-like in nature, some of which could be implemented directly as expression macros. The macro referenced by the `identifier` must be an expression macro, as indicated by `@freestanding(expression)` on the corresponding macro declaration. + +Both `function-call-argument-clause` and `trailing-closures` are optional. When both are omitted, the macro is expanded as-if the empty argument list `()` was provided. Macros are not first-class entities in the way functions are, so they cannot be passed around as values and do not need an "unapplied macro" syntax. This allows `#line` et al to be macros without requiring them to be written as `#line()`. There is some precedent for this with property wrappers, which will also be used for attached macros. + +When a macro expansion is encountered in the source code, its expansion occurs in two phases. The first phase is the type-check phase, where the arguments to the macro are type-checked against the parameters of the named macro, and the result type of the named macro is checked against the context in which the macro expansion occurs. This type-checking is equivalent to that performed for a function call, and does not involve the macro definition. + +The second phase is the macro expansion phase, during which the syntax of the macro arguments is provided to the macro definition. For builtin-macro definitions, the behavior at this point depends on the semantics of the macro, e.g., the `externalMacro` macro invokes the external program and provides it with the source code of the macro expansion. For other macros, the arguments are substituted into the `macro-expansion-expression` of the definition. For example: + +```swift +@freestanding(expression) macro prohibitBinaryOperators(_ value: T, operators: [String]) -> T = + #externalMacro(module: "ExampleMacros", type: "ProhibitBinaryOperators") +@freestanding(expression) macro addBlocker(_ value: T) -> T = #prohibitBinaryOperators(value, operators: ["+"]) + +#addBlocker(x + y * z) +``` + +Here, the macro expansion of `#addBlocker(x + y * z)` will first expand to `#prohibitBinaryOperators(x + y * z, operators: ["+"])`. Then that expansion will be processed by the `ExampleMacros.ProhibitBinaryOperators`, which would be defined as a struct conforming to `ExpressionMacro`. + +Macro expansion produces new source code (in a syntax tree), which is then type-checked using the original macro result type as its contextual type. For example, the `stringify` example macro returned a `(T, String)`, so when given an argument of type `Int`, the result of expanding the macro would be type-checked as if it were on the right-hand side of + +```swift +let _: (Int, String) = +``` + +Macro expansion expressions can occur within the arguments to a macro. For example, consider: + +```swift +#addBlocker(#stringify(1 + 2)) +``` + +The first phase of the macro type-check does not perform any macro expansion: the macro expansion expression `#stringify(1 + 2)` will infer that its `T` is `Int`, and will produce a value of type `(Int, String)`. The `addBlocker` macro expansion expression will infer that its `T` is `(Int, String)`, and the result is the same. + +The second phase of macro expansions occurs outside-in. First, the `addBlocker` macro is expanded, to `#prohibitBinaryOperators(#stringify(1 + 2), operators: ["+"])`. Then, the `prohibitBinaryOperators` macro is expanded given those (textual) arguments. The expansion result it produces will be type-checked, which will end up type-checking `#stringify(1 + 2)` again and, finally, expanding `#stringify(1 + 2)`. + +From an implementation perspective, the compiler reserves the right to avoid performing repeated type checking of the same macro arguments. For example, we type-checked `#stringify(1 + 2)` in the first phase of the expansion of `prohibitBinaryOperators`, and then again on the expanded result. When the compiler recognizes that the same syntax node is being re-used unmodified, it can re-use the types computed in the first phase. This is an important performance optimization for the type checker. + +Macro expansion cannot be recursive: if the expansion of a given macro produces source code that expands that same macro, the program is ill-formed. This prevents unbounded macro expansion. + +With the exception of the built-in macro declarations for source locations (e.g., `#fileID`, `#line`), a macro cannot be used as the default argument of a parameter. The existing features for source locations have special behavior when they appear as a default argument, wherein they are expanded by the caller using the source-location information at the call site rather than in the function declaration where they appear. This is useful, existing behavior that we cannot change, but it might not make sense for all macros, and could be surprising. Therefore, we prohibit such default argument that are (non-built-in) macros to avoid confusion, and are open to revisiting this restriction in the future. + +### Macro implementation library + +Macro definitions will make use of the [swift-syntax](https://github.com/apple/swift-syntax) package, which provides the Swift syntax tree manipulation and parsing capabilities for Swift tools. The `SwiftSyntaxMacros` module will provide the functionality required to define macros. + +#### `Macro` protocols + +The `Macro` protocol is the root protocol for all kinds of macro definitions. At present, it does not have any requirements: + +```swift +public protocol Macro { } +``` + +All "freestanding" macros conform to the `FreestandingMacro` protocol: + +```swift +public protocol FreestandingMacro: Macro { } +``` + +The `ExpressionMacro` protocol is used to describe expression macros, and is a form of freestanding macro: + +```swift +public protocol ExpressionMacro: FreestandingMacro { + /// Expand a macro described by the given freestanding macro expansion syntax node + /// within the given context to produce a replacement expression. + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> ExprSyntax +} +``` + +The `FreestandingMacroExpansionSyntax` protocol is the `swift-syntax` node describing the `macro-expansion-expression` grammar term from above, so it carries the complete syntax tree (including all whitespace and comments) of the macro expansion as it appears in the source code. + +Macro definitions should conform to the `ExpressionMacro` protocol and implement their syntactic transformation via `expansion(of:in:)`, returning the new expression as a syntax node. + +If the macro expansion cannot proceed for some reason, the `expansion(of:in:)` operation can throw an error rather than try to produce a new syntax node. The compiler will then report the error to the user. More detailed diagnostics can be provided via the macro expansion context. + +#### `MacroExpansionContext` + +The macro expansion context provides additional information about the environment in which the macro is being expanded. This context can be queried as part of the macro expansion: + +```swift +/// Protocol whose conforming types provide information about the context in +/// which a given macro is being expanded. +public protocol MacroExpansionContext: AnyObject { + /// Generate a unique name for use in the macro. + public func makeUniqueName(_ name: String) -> TokenSyntax + + /// Emit a diagnostic (i.e., warning or error) that indicates a problem with the macro + /// expansion. + public func diagnose(_ diagnostic: Diagnostic) + + /// Retrieve a source location for the given syntax node. + /// + /// - Parameters: + /// - node: The syntax node whose source location to produce. + /// - position: The position within the syntax node for the resulting + /// location. + /// - filePathMode: How the file name contained in the source location is + /// formed. + /// + /// - Returns: the source location within the given node, or `nil` if the + /// given syntax node is not rooted in a source file that the macro + /// expansion context knows about. + func location( + of node: some SyntaxProtocol, + at position: PositionInSyntaxNode, + filePathMode: SourceLocationFilePathMode + ) -> AbstractSourceLocation? +} +``` + +The `makeUniqueName()` function allows one to create new, unique names so that the macro expansion can produce new declarations that won't conflict with any other declarations in the same scope. It produces an identifier token containing the unique name, which will also incorporate the `name` identifier for better debuggability. This allows macros to be more hygienic, by not introducing new names that could affect the way that the code provided via macro expansion arguments is type-checked. + +It is intended that `MacroExpansionContext` will grow over time to include more information about the build environment in which the macro is being expanded. For example, information about the target platform (such as OS, architecture, and deployment version) and any compile-time definitions passed via `-D`, should be included as part of the context. + +The `diagnose` method allows a macro implementation to provide diagnostics as part of macro expansion. The [`Diagnostic`](https://github.com/apple/swift-syntax/blob/main/Sources/SwiftDiagnostics/Diagnostic.swift) type used in the parameter is part of the swift-syntax library, and its form is likely to change over time, but it is able to express the different kinds of diagnostics a compiler or other tool might produce, such as warnings and errors, along with range highlights, Fix-Its, and attached notes to provide more clarity. A macro definition can introduce diagnostics if, for example, the macro argument successfully type-checked but used some Swift syntax that the macro implementation does not understand. The diagnostics will be presented by whatever tool is expanding the macro, such as the compiler. A macro that emits diagnostics is still expected to produce an expansion result unless it also throws an error, in which case both emitted diagnostics and the error will be reported. + +The `location` operation allows one to determine source location information for a syntax node. The resulting source location contains the file, line, and column for the corresponding syntax node. The `position` and `filePathMode` can be used to customize the resulting output, e.g., which part of the syntax node to point at and how to render the file name. + +```swift +/// Describe the position within a syntax node that can be used to compute +/// source locations. +public enum PositionInSyntaxNode { + /// Refers to the start of the syntax node's leading trivia, which is + /// the first source location covered by the syntax node. + case beforeLeadingTrivia + + /// Refers to the start of the syntax node's first token, which + /// immediately follows the leading trivia. + case afterLeadingTrivia + + /// Refers to the end of the syntax node's last token, right before the + /// trailing trivia. + case beforeTrailingTrivia + + /// Refers just past the end of the source text that is covered by the + /// syntax node, after all trailing trivia. + case afterTrailingTrivia +} + +/// Describes how a source location file path will be formed. +public enum SourceLocationFilePathMode { + /// A file ID consisting of the module name and file name (without full path), + /// as would be generated by the macro expansion `#fileID`. + case fileID + + /// A full path name as would be generated by the macro expansion `#filePath`, + /// e.g., `/home/taylor/alison.swift`. + case filePath +} +``` + +Source locations are described in an abstract form that can be interpolated into source code (they are expressions) in places that expect a string literal (for the file name) or integer literal (for line and column). As with `makeUniqueName` returning a `TokenSyntax` rather than a `String`, this abstraction allows the compiler to introduce a different kind of syntax node (that might not even be expressible in normal Swift) to represent these values. + +```swift +/// Abstractly represents a source location in the macro. +public struct AbstractSourceLocation { + /// A primary expression that represents the file and is `ExpressibleByStringLiteral`. + public let file: ExprSyntax + + /// A primary expression that represents the line and is `ExpressibleByIntegerLiteral`. + public let line: ExprSyntax + + /// A primary expression that represents the column and is `ExpressibleByIntegerLiteral`. + public let column: ExprSyntax +} +``` + +### Macros in the Standard Library + +#### `externalMacro` definition + +The builtin `externalMacro` macro is declared as follows: + +```swift +macro externalMacro(module: String, type: String) -> T +``` + +The arguments identify the module name and type name of the type that provides an external macro definition. Note that the `externalMacro` macro is special in that it can only be expanded to define another macro. It is an error to use it anywhere else, which is why it does not include an `@freestanding(expression)` attribute. + +#### Builtin macro declarations + +As previously noted, expression macros use the same leading `#` syntax as a number of built-in expressions like `#line`. With the introduction of expression macros, we propose to subsume those built-in expressions into macros that come as part of the Swift standard library. The actual macro implementations are provided by the compiler, and may even involve things that aren't necessarily implementable with the pure syntactic macro. However, by providing macro declarations we remove special cases from the language and benefit from all of the tooling affordances provided for macros. + +We propose to introduce a number of macro declarations into the Swift standard library. There are several different kinds of such macros. + +##### Source-location macros + +```swift +// File and path-related information +@freestanding(expression) macro fileID() -> T +@freestanding(expression) macro file() -> T +@freestanding(expression) macro filePath() -> T + +// Current function +@freestanding(expression) macro function() -> T + +// Source-location information +@freestanding(expression) macro line() -> T +@freestanding(expression) macro column() -> T + +// Current shared object handle. +@freestanding(expression) macro dsohandle() -> UnsafeRawPointer +``` + +The operations that provide information about the current location in source code are mostly implementable as `ExpressionMacro`-conforming types, using the `location` operation on the `MacroExpansionContext`. The exceptions are `#file`, which would need an extension to `MacroExpansionContext` to determine whether we are in a compilation mode where `#file` behaves like `#fileID` vs. behaving like [`#filePath`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0285-ease-pound-file-transition.md); `dsohandle`, which requires specific compiler support; and `#function`, which would require contextual information that is not available in the `MacroExpansionContext`. + +The type signatures of these macros capture most of the type system behavior of the existing `#file`, `#line`, etc., because they are treated like literals and therefore can pick up any contextual type that implements the proper `ExpressibleBy*` protocol. However, the implementations above would fail to type-check code like this: + +```swift +let x = #file +``` + +with an error such as + +``` +error: generic parameter 'T' could not be inferred +``` + +To match the existing behavior of the built-in `#file`, `#line`, etc. would require a defaulting rule that matches what we get for literal types. At present, this requires special handling in the compiler, but a future extension to the language to enable default generic arguments would likely allow us to express this notion directly in the type system. + +##### Objective-C helper macros + +The Swift `#selector` and `#keyPath` expressions can have their syntax and type-checking behavior expressed in terms of macro declarations: + +```swift +@freestanding(expression) macro selector(_ method: T) -> Selector +@freestanding(expression) macro selector(getter property: T) -> Selector +@freestanding(expression) macro selector(setter property: T) -> Selector +@freestanding(expression) macro keyPath(_ property: T) -> String +``` + +These macros cannot be implemented in terms of `ExpressionMacro` based on the facilities in this proposal, because one would need to determine which declarations are referenced within the argument of a macro expansion such as `#selector(getter: Person.name)`. However, providing them with macro declarations that have built-in implementations makes them less special, removing some special cases from more of the language. + +##### Object literals + +```swift +@freestanding(expression) macro colorLiteral(red: Float, green: Float, blue: Float, alpha: Float) -> T +@freestanding(expression) macro imageLiteral(resourceName: String) -> T +@freestanding(expression) macro fileLiteral(resourceName: String) -> T +``` + +The object literals allow one to reference a resource in a program of various kinds. The three kinds of object literals (color, image, and file) can be described as expression macros. The type signatures provided above are not exactly how type checking currently works for object literals, because they aren't necessarily generic. Rather, when they are used, the compiler currently looks for a specially-named type (e.g., `_ColorLiteralType`) in the current module and uses that as the type of the corresponding color literal. To maintain that behavior, we propose to type-check macro expansions for object literals by performing the same lookup that is done today (e.g., for `_ColorLiteralType`) and then using that type as the generic argument for the corresponding macro. That way, the type checking behavior is unchanged when moving from special object literal expressions in the language to macro declarations with built-in implementations. + +### Sandboxing macro implementations + +The details of how macro implementation modules are built and provided to the compiler will be left to a separate proposal. However, it's important to call out here that macro implementations will be executed in a sandbox [like SwiftPM plugins](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md#security), preventing file system and network access. This is both a security precaution and a practical way of encouraging macros to not depend on any state other than the specific macro expansion node they are given to expand and its child nodes (but not its parent nodes), and the information specifically provided by the macro expansion context. If in the future macros need access to other information, this will be accomplished by extending the macro expansion context, which also provides a mechanism for the compiler to track what information the macro actually queried. + +## Tools for using and developing macros + +One of the primary concerns with macros is their ease of use and development: how do we know what a macro does to a program? How does one develop and debug a new macro? + +With the right tool support, the syntactic model of macro expansion makes it easy to answer the first question. The tools will need to be able to show the developer what the expansion of any use of a macro is. At a minimum, this should include flags that can be passed to the compiler to expand macros (the prototype provides `-Xfrontend -dump-macro-expansions` for this), and possibly include a mode to write out a "macro-expanded" source file akin to how C compilers can emit a preprocessed source file. Other tools such as IDEs should be able to show the expansion of a given use of a macro so that developers can inspect what a macro is doing. Because the result is always Swift source code, one can reason about it more easily than (say) inspecting the implementation of a macro that manipulates an AST or IR. + +The fact that macro implementations are separate programs actually makes it easier to develop macros. One can write unit tests for a macro implementation that provides the input source code for the macro (say, `#stringify(x + y)`), expands that macro using facilities from swift-syntax, and verifies that the resulting code is free of syntax errors and matches the expected result. Most of the "builtin" macro examples were developed this way in the [syntax macro test file](https://github.com/apple/swift-syntax/blob/main/Tests/SwiftSyntaxMacrosTest/MacroSystemTests.swift). + +## Example expression macros + +There are many uses for expression macros beyond what has been presented here. This section will collect several examples of macro implementations based on existing built-in `#` expressions as well as ones that come from the Swift forums and other sources of inspiration. Prototype implementations of a number of these macros are [available in the swift-syntax repository](https://github.com/apple/swift-syntax/blob/main/Tests/SwiftSyntaxMacrosTest/MacroSystemTests.swift). + +* The `#colorLiteral` macro provides syntax for declaring a color with a given red, green, blue, and alpha values. It can be declared and implemented as follows + + ```swift + // Declaration of #colorLiteral + @freestanding(expression) macro colorLiteral(red: Float, green: Float, blue: Float, alpha: Float) -> _ColorLiteralType + = SwiftBuiltinMacros.ColorLiteralMacro + + // Implementation of #colorLiteral + struct ColorLiteralMacro: ExpressionMacro { + /// Replace the label of the first element in the tuple with the given + /// new label. + func replaceFirstLabel( + of tuple: TupleExprElementListSyntax, with newLabel: String + ) -> TupleExprElementListSyntax{ + guard let firstElement = tuple.first else { + return tuple + } + + return tuple.replacing( + childAt: 0, with: firstElement.withLabel(.identifier(newLabel))) + } + + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + let argList = replaceFirstLabel( + of: node.argumentList, with: "_colorLiteralRed" + ) + let initSyntax: ExprSyntax = ".init(\(argList))" + if let leadingTrivia = node.leadingTrivia { + return MacroResult(initSyntax.withLeadingTrivia(leadingTrivia)) + } + return initSyntax + } + } + ``` + + The same approach can be used for file and image literals. + +* [Power assertions](https://forums.swift.org/t/a-possible-vision-for-macros-in-swift/60900/87) by Kishikawa Katsumi: this assertion macro captures intermediate values within the assertion expression so that when the assertion fails, those values are displayed. The results are exciting! + + ```swift + #powerAssert(mike.isTeenager && john.age < mike.age) + | | | | | | | | + | true | | 42 | | 13 + | | | | Person(name: "Mike", age: 13) + | | | false + | | Person(name: "John", age: 42) + | false + Person(name: "Mike", age: 13) + ``` + +## Source compatibility + +Macros are a pure extension to the language, utilizing new syntax, so they don't have an impact on source compatibility. + +## Effect on ABI stability + +Macros are a source-to-source transformation tool that have no ABI impact. + +## Effect on API resilience + +Macros are a source-to-source transformation tool that have no effect on API resilience. + +## Future Directions + +There are a number of potential directions one could take macros, both by providing additional information to the macro implementations themselves and expanding the scope of macros. + +### Macro argument type information + +The arguments to a macro are fully type-checked before the macro implementation is invoked. However, information produced while performing that type-check is not provided to the macro, which only gets the original source code. In some cases, it would be useful to also have information determined during type checking, such as the types of the arguments and their subexpressions, the full names of the declarations referenced within those expressions, and any implicit conversions performed as part of type checking. For example, consider a use of a macro like the [power assertions](https://forums.swift.org/t/a-possible-vision-for-macros-in-swift/60900/87) mentioned earlier: + +```swift +#assert(Color(parsing: "red") == .red) +``` + +The implementation would likely want to separate the two operands to `==` into local variables (with fresh names generated by `createUniqueName`) to capture the values, so they can be printed later. For example, the assertion could be translated into code like the following: + +```swift +{ + let _unique1 = Color(parsing: "red") + let _unique2 = .red + if !(_unique1 == _unique2) { + fatalError("assertion failed: \(_unique1) != \(_unique2)") + } +}() +``` + +Note, however, that this code will not type check, because initializer for `_unique2` requires context information to determine how to resolve `.red`. If the macro implementation were provided with the types of the two subexpressions, `Color(parsing: "red")` and `.red`, it could have been translated into a something that will type-check properly: + +```swift +{ + let _unique1: Color = Color(parsing: "red") + let _unique2: Color = .red + if !(_unique1 == _unique2) { + fatalError("assertion failed: \(_unique1) != \(_unique2)") + } +}() +``` + +The macro expansion context could be extended with an operation to produce the type of a given syntax node, e.g., + +```swift +extension MacroExpansionContext { + func type(of node: ExprSyntax) -> Type? +} +``` + +When given one of the expression syntax nodes that is part of the macro expansion expression, this operation would produce a representation of the type of that expression. The `Type` would need to be able to represent the breadth of the Swift type system, including structural types like tuple and function types, and nominal types like struct, enum, actor, and protocol names. + +Additional information could be provided about the actual resolved declarations. For example, the syntax node for `.red` could be queried to produce a full declaration name `Color.red`, and the syntax node for `==` could resolve to the full name of the declaration of the `==` operator that compares two `Color` values. A macro could then distinguish between different `==` operator implementations. + +The main complexity of this future direction is in defining the APIs to be used by macro implementations to describe the Swift type system and related information. It would likely be a simplified form of a type checker's internal representation of types, but would need to remain stable. Therefore, while we feel that the addition of type information is a highly valuable extension for expression macros, the scope of the addition means it would best be introduced as a follow-on proposal. + +### Additional kinds of macros + +Expressions are just one place in the language where macros could be valuable. Other places could include function or closure bodies (e.g., to add tracing or logging), within type or extension definitions (e.g., to add new members), or on protocol conformances (e.g., to synthesize a protocol conformance). A number of potential ideas are presented in the [vision for macros in Swift](https://forums.swift.org/t/a-possible-vision-for-macros-in-swift/60900). For each of them, we assume that the basic `macro` declaration will stay roughly the same, but the contexts in which the macro can be used would be different, as might the spelling of the expansion (e.g., `@` might be more appropriate if the macro expansion occurs on a declaration), there would be an attribute on the `macro` declaration that indicates what type of macro it is, and there would be a corresponding protocol that inherits from `Macro` in the `SwiftSyntaxMacros` module. + +## Revision History + +* Revision after acceptance: + * Make the `ExpressionMacro.expansion(of:in:)` requirement non-`async`. +* Revisions based on review feedback: + * Switch `@expression` to `@freestanding(expression)` to align with the other macros proposals and vision document. + * Make the `ExpressionMacro.expansion(of:in:)` requirement `async`. + * Allow macro declarations to have opaque result types, and define the uniqueness rules. + * Simplify the grammar of macro declarations to be more function-like: they always require a parameter list, and if they have a return value, its type is specified following `->`. To account for macros that take no arguments, omitting both an argument list and trailing closures from a macro expansion expression will implicitly add `()`. + * Make `MacroExpansionContext` a class-bound protocol, because the state involving diagnostics and unique names needs to be shared, and the implementations could vary significantly between (e.g.) the compiler and a test harness. + * Introduce a general `location` operation on `MacroExpansionContext` to get the source location of any syntax node from a macro input. Remove the `moduleName` and `fileName`, which were always too limited to be useful. + * Allow macro parameters to have default arguments, with restrictions on what can occur within a default argument. + * Clarify that macro expansion cannot be recursive. + * Rename `createUniqueLocalName` to `makeUniqueName`; the names might not always be local in scope. Also add a parameter to it so developers can provide a partial name that will show up in the unique name. + * Prohibit the use of non-builtin macros as default arguments of parameters. +* Revisions from the second pitch: + * Moved SwiftPM manifest changes to a separate proposal that can explore the building of macros in depth. This proposal will focus only on the language aspects. + * Simplified the type signature of the `#externalMacro` built-in macro. + * Added `@expression` to the macro to distinguish it from other kinds of macros that could come in the future. + * Make `expansion(of:in:)` throwing, and have that error be reported back to the user. + * Expand on how the various builtin standard library macros will work. +* Revisions from the first pitch: + * Rename `MacroEvaluationContext` to `MacroExpansionContext`. + * Remove `MacroResult` and instead allow macros to emit diagnostics via the macro expansion context. + * Remove `sourceLocationConverter` from the macro expansion context; it provides access to the whole source file, which interferes with incremental builds. + * Rename `ExpressionMacro.apply` to `expansion(of:in)` to make it clear that it's producing the expansion of a syntax node within a given context. + * Remove the implementations of `#column`, as well as the implication that things like `#line` can be implemented with macros. Based on the above changes, they cannot. + * Introduce a new section providing declarations of macros for the various `#` expressions that exist in the language, but will be replaced with (built-in) macros. + * Replace the `external-macro-name` production for defining macros with the more-general `macro-expansion-expression`, and a builtin macro `externalMacro` that makes it far more explicit that we're dealing with external types that are looked up by name. This also provides additional capabilities for defining macros in terms of other macros. + * Add much more detail about how macro expansion works in practice. + * Introduce SwiftPM manifest extensions to define macro plugins. + * Added some future directions and alternatives considered. + + +## Acknowledgments + +Richard Wei implemented the compiler plugin mechanism on which the prototype implementation depends, as well as helping identify and explore additional use cases for macros. John McCall and Becca Royal-Gordon provided numerous insights into the design and practical implementation of macros. Tony Allevato provided additional feedback on building and sandboxing. diff --git a/proposals/0383-deprecate-uiapplicationmain-and-nsapplicationmain.md b/proposals/0383-deprecate-uiapplicationmain-and-nsapplicationmain.md new file mode 100644 index 0000000000..a2ea5c6da7 --- /dev/null +++ b/proposals/0383-deprecate-uiapplicationmain-and-nsapplicationmain.md @@ -0,0 +1,117 @@ +# Deprecate @UIApplicationMain and @NSApplicationMain + +* Proposal: [SE-0383](0383-deprecate-uiapplicationmain-and-nsapplicationmain.md) +* Authors: [Robert Widmann](https://github.com/codafi) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.10)** +* Upcoming Feature Flag: `DeprecateApplicationMain` +* Implementation: [PR 62151](https://github.com/apple/swift/pull/62151) +* Review: ([pitch](https://forums.swift.org/t/deprecate-uiapplicationmain-and-nsapplicationmain/61493)) ([review](https://forums.swift.org/t/se-0383-deprecate-uiapplicationmain-and-nsapplicationmain/62375)) ([acceptance](https://forums.swift.org/t/accepted-se-0383-deprecate-uiapplicationmain-and-nsapplicationmain/62645)) + +## Introduction + +`@UIApplicationMain` and `@NSApplicationMain` used to be the standard way for +iOS and macOS apps respectively to declare a synthesized platform-specific +entrypoint for an app. These functions have since been obsoleted by +[SE-0281](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0281-main-attribute.md)'s +introduction of the `@main` attribute, and they now represent a confusing bit of +duplication in the language. This proposal seeks to deprecate these alternative +entrypoint attributes in favor of `@main` in pre-Swift 6, and it makes their use +in Swift 6 a hard error. + +## Motivation + +UIKit and AppKit have fully embraced the `@main` attribute and have made +adoption by applications as simple as conforming to the `UIApplicationDelegate` +and `NSApplicationDelegate` protocols. This now means that an author of an +application is presented with two different, but ultimately needless, choices +for an entrypoint: + +* use one of the hard coded framework-specific attributes `@UIApplicationMain` or `@NSApplicationMain`, or +* use the more general `@main` attribute. + +At runtime, the behavior of the `@main` attribute on classes that conform to +one of the application delegate protocols above is identical to the corresponding +framework-specific attribute. Having two functionally identical ways to express the +concept of an app-specific entrypoint is clutter at best and confusing at worst. +This proposal seeks to complete the migration work implied by +[SE-0281](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0281-main-attribute.md) +by having the compiler push Swift authors towards the more general, unified +solution. + +## Proposed solution + +Using either `@UIApplicationMain` and `@NSApplicationMain` in a pre-Swift 6 +language mode will unconditionally warn and offer to replace these attributes +with the appropriate conformances. In Swift 6 language mode (and later), using +these attributes will result in a hard error. + +## Detailed design + +> Because `@UIApplicationMain` and `@NSApplicationMain` are used in identical +> ways, this portion of the document will only discuss `@UIApplicationMain`. +> The design for `@NSApplicationMain` follows the exact same pattern. + +Framework-specific attributes were added to the language to automate the +boilerplate involved in declaring a standard application entrypoint. In UIKit +code, the entrypoint always ends with a call to `UIApplicationMain`. The last +parameter of this call is the name of a subclass of `UIApplicationDelegate`. +UIKit will search for and instantiate this delegate class so it can issue +application lifecycle callbacks. Swift, therefore, requires this attribute to +appear on a class that conforms to the `UIApplicationDelegate` protocol so it +can provide the name of that class to UIKit. + +But a conformance to `UIApplicationDelegate` comes with more than just lifecycle +callbacks. A default implementation of a `main` entrypoint is [provided for +free](https://developer.apple.com/documentation/uikit/uiapplicationdelegate/3656306-main) +to a conforming type, but the `@UIApplicationMain` attribute suppresses it. This +fact is key to the migration path for existing users of the framework-specific +attribute. + +Under this proposal, when the compiler sees a use of `@UIApplicationMain`, it +will emit a diagnostic including a suggestion to replace the attribute with +`@main`. In Swift 6 and later language modes, this diagnostic will be an error; +otherwise it will be a warning. + +```swift +@UIApplicationMain // warning: '@UIApplicationMain' is deprecated in Swift 5 + // fixit: Change `@UIApplicationMain` to `@main` +final class MyApplication: UIResponder, UIApplicationDelegate { + /**/ +} +``` + +Once the fixit has been applied, the result will be + +```swift +@main +final class MyApplication: UIResponder, UIApplicationDelegate { + /**/ +} +``` + +This simple migration causes the compiler to select the `main` entrypoint +inherited by the conformance to `UIApplicationDelegate`. No further source +changes are required. + +## Source compatibility + +Current Swift libraries will continue to build because they compile under +pre-Swift 6 language modes. Under such language modes this proposal adds only an +unconditional warning when framework-specific entrypoints are used, and provides +diagnostics to avoid the warning by automatically migrating user code. + +In Swift 6 and later modes, this proposal is intentionally source-breaking as the +compiler will issue an unconditional error upon encountering a framework-specific +attribute. This source break will occur primarily in older application code, as +most libraries and packages do not use framework-specific attributes to define a main +entrypoint. Newer code, including templates for applications provided by Xcode 14 +and later, already use the `@main` attribute. + +## Effect on ABI stability + +This proposal has no impact on ABI. + +## Effect on API resilience + +None. diff --git a/proposals/0384-importing-forward-declared-objc-interfaces-and-protocols.md b/proposals/0384-importing-forward-declared-objc-interfaces-and-protocols.md new file mode 100644 index 0000000000..e7da438da2 --- /dev/null +++ b/proposals/0384-importing-forward-declared-objc-interfaces-and-protocols.md @@ -0,0 +1,244 @@ +# Importing Forward Declared Objective-C Interfaces and Protocols + +* Proposal: [SE-0384](0384-importing-forward-declared-objc-interfaces-and-protocols.md) +* Author: [Nuri Amari](https://github.com/NuriAmari) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 5.9)** +* Implementation:[apple/swift#61606]( https://github.com/apple/swift/pull/61606) +* Upcoming Feature Flag: `ImportObjcForwardDeclarations` +* Review: ([pitch](https://forums.swift.org/t/pitch-importing-forward-declared-objective-c-classes-and-protocols/61926)) ([review](https://forums.swift.org/t/se-0384-importing-forward-declared-objective-c-interfaces-and-protocols/62392)) ([acceptance](https://forums.swift.org/t/accepted-se-0384-importing-forward-declared-objective-c-interfaces-and-protocols/62670)) + +## Introduction + +This proposal seeks to improve the usability of existing Objective-C libraries from Swift by reducing the negative +impact forward declarations have on API visibility from Swift. We wish to start synthesizing placeholder types to +represent forward declared Objective-C interfaces and protocols in Swift. + +## Motivation + +Forward declarations are very common in many existing Objective-C code bases, used often to break cyclic dependencies or to improve build performance. +Unfortunately, when it comes to "importability" into Swift they are quite detrimental. + +As it stands, the ClangImporter will fail to import any declaration that references a forward declared type in many common cases. This means a single +forward declared type can render larger portions of an Objective-C API unusable from Swift. For example, the following Objective-C API from the implementation PR +is empty from the Swift perspective (Swift textual interface generated via swift-ide-test): + +_Objective-C_ +```objective-c +#import + +@class ForwardDeclaredInterface; +@protocol ForwardDeclaredProtocol; + +@interface IncompleteTypeConsumer1 : NSObject +@property id propertyUsingAForwardDeclaredProtocol1; +@property ForwardDeclaredInterface *propertyUsingAForwardDeclaredInterface1; +- (id)init; +- (NSObject *)methodReturningForwardDeclaredProtocol1; +- (ForwardDeclaredInterface *)methodReturningForwardDeclaredInterface1; +- (void)methodTakingAForwardDeclaredProtocol1: + (id)param; +- (void)methodTakingAForwardDeclaredInterface1: + (ForwardDeclaredInterface *)param; +@end + +ForwardDeclaredInterface *CFunctionReturningAForwardDeclaredInterface1(); +void CFunctionTakingAForwardDeclaredInterface1( + ForwardDeclaredInterface *param); + +NSObject *CFunctionReturningAForwardDeclaredProtocol1(); +void CFunctionTakingAForwardDeclaredProtocol1( + id param); +``` + +_Swift_ +```swift +class IncompleteTypeConsumer1 : NSObject { + init!() +} +``` +It is possible to fix such issues by importing the definition of the type, either in the Objective-C header or in the consuming Swift file. However, such manual changes make consuming existing libraries more difficult and needlessly increase build times. We wish to make the experience of consuming an Objective-C API with forward declarations from Swift more consistent with the experience of consuming the API from Objective-C. + +We have done some work in the past to help diagnose such failures in the ClangImporter: + +https://forums.swift.org/t/pitch-improved-clangimporter-diagnostics/52687/17 + +https://forums.swift.org/t/pitch-lazy-clangimporter-diagnostics-enabled-by-default/54651 + +We want to build on this work, specifically with respect to forward declarations, this time +fixing a shortcoming of the ClangImporter, not just diagnosing it. + +## Proposed solution + +We propose the following representation for forward declared Objective-C interfaces and protocols in Swift: + +```swift +// @class Foo turns into +@available(*, unavailable, message: “This Objective-C class has only been forward declared; import its owning module to use it”) +class Foo : NSObject {} + +// @protocol Bar turns into +@available(*, unavailable, message: “This Objective-C protocol has only been forward declared; import its owning module to use it”) +protocol Bar : NSObjectProtocol {} +``` + +The idea is to introduce the minimal change that will make Objective-C APIs usable in a predictable safe manner. + +The aforementioned Objective-C API with this change looks like this from Swift: + +```swift +@available(*, unavailable, message: "This Objective-C class has only been forward-declared; import its owning module to use it") +class ForwardDeclaredInterface { +} +@available(*, unavailable, message: "This Objective-C protocol has only been forward-declared; import its owning module to use it") +protocol ForwardDeclaredProtocol : NSObjectProtocol { +} +class IncompleteTypeConsumer1 : NSObject { + var propertyUsingAForwardDeclaredProtocol1: ForwardDeclaredProtocol! + var propertyUsingAForwardDeclaredInterface1: ForwardDeclaredInterface! + init!() + func methodReturningForwardDeclaredProtocol1() -> ForwardDeclaredProtocol! + func methodReturningForwardDeclaredInterface1() -> ForwardDeclaredInterface! + func methodTakingAForwardDeclaredProtocol1(_ param: ForwardDeclaredProtocol!) + func methodTakingAForwardDeclaredInterface1(_ param: ForwardDeclaredInterface!) +} +func CFunctionReturningAForwardDeclaredInterface1() -> ForwardDeclaredInterface! +func CFunctionTakingAForwardDeclaredInterface1(_ param: ForwardDeclaredInterface!) +func CFunctionReturningAForwardDeclaredProtocol1() -> ForwardDeclaredProtocol! +func CFunctionTakingAForwardDeclaredProtocol1(_ param: ForwardDeclaredProtocol!) +``` + +More usage examples can be found in these tests introduced here: https://github.com/apple/swift/pull/61606 + +## Detailed design + +Modifications lie almost exclusively in `ClangImporter` -- specifically in `SwiftDeclConverter`. If asked to convert a +`clang::ObjCInterfaceDecl` or `clang::ObjCProtocolDecl` with no definition `SwiftDeclConverter` will now return a placeholder type instead of +bailing. These placeholder types are as described: + +```swift +// @class Foo turns into +@available(*, unavailable, message: “This Objective-C class has only been forward declared; import its owning module to use it”) +class Foo : NSObject {} + +// @protocol Bar turns into +@available(*, unavailable, message: “This Objective-C protocol has only been forward declared; import its owning module to use it”) +protocol Bar : NSObjectProtocol {} +``` + +Permitted usages of these types are intentionally limited. You will be able to use Objective-C and C declarations that refer to these types without issue. +You will be able to pass around instances of these incomplete types from Swift to Objective-C and vice versa. + +You’ll also be able to call the set of methods provided by `NSObject` or `NSObjectProtocol` on instances. Notably this assumes +that the backing Objective-C implementation does indeed inherit from / conform to `NSObject`. We are under the impression that +inheriting from `NSObject` is a requirement for Swift - Objective-C interop, but are not sure about protocols conforming to `NSObject`. +We could use some input on these assumptions, and will remove this feature if they turn out to be unsound. + +Using the type itself directly in Swift is forbidden by the attached unavailable attribute. This is to keep the impact of the change small and to prevent unsound declarations, +such as declaring a new Swift class that inherits from or conforms to such a type. You will also not be able to create new instances of these types in Swift. + +To make these limitations more intuitive to users, a new diagnostic has been added. This diagnostic helps guide users when they attempt to access a member on the incomplete synthesized +type, expecting the complete type. The new diagnostics are shown in the last 4 lines of this example: + +```swift +swift-client-fwd-declared.swift:6:7: error: value of type 'Foo' has no member 'sayHello' +myFoo.sayHello() +~~~~~ ^~~~~~~~ +:0: note: class 'Foo' is a placeholder for a forward declared Objective-C interface and may be missing members; import the definition to access the complete interface +foo-bar-consumer.h:3:1: note: interface 'Foo' forward declared here +@class Foo; +^ +``` + +In Swift 5.x, the feature is gated behind the upcoming feature flag `ImportObjcForwardDeclarations`. This flag is on by default for Swift version 6 and onwards. + +The flag is always disabled in the REPL, as allowing it currently leads to confusing behavior. In the REPL, the environment in terms of whether a complete definition of some type Foo is available can change during execution. These examples show some of the confusing behavior: + +```swift +import IncompleteFoo +let foo = FunctionReturningFoo() +FunctingTakingFoo(foo) +import CompleteFoo // This has no effect as far as Swift's view of type 'Foo' is concerned +let completeFoo = Foo() // error type 'IncompleteFoo.Foo' has no initializer +foo.methodOnCompleteFoo() // error no member 'methodOnCompleteFoo' on type 'IncompleteFoo.Foo' +``` + +```swift +import IncompleteFoo +import CompleteFoo +let completeFoo = Foo() // This time it works because the cache was not populated until after CompleteFoo was imported +foo.methodOnCompleteFoo() // Works as well +``` + +This happens because `ClangImporter` is lazy and caches imports, once the placeholder is synthesized (when only the incomplete definition is +available), the complete definition will never be imported. Without an understanding of `ClangImporter`, it is impossible to understand why +the statements between the two imports have an effect on those after. We have made attempts at solving these issues, but no satisfactory solution +was found. The details of these attempts are listed in the alternatives considered section. + +The [existing ClangImporter diagnostics](https://forums.swift.org/t/pitch-improved-clangimporter-diagnostics/52687) will still properly warn users about forward declaration limitations, with or with out the feature enabled. + +## Source compatibility + +This change does have the potential to break source compatibility, since +`ClangImporter` will now import declarations that were previously dropped. +That said, the impact should rare. The current implementation already passes +the source compatibility suite and the source kit stress test. + +As mentioned, to address source compatibility issues, the feature is gated behind +a flag for Swift 5 and before, and enabled by default in Swift 6. + +## Effect on ABI stability + +This change has no affect on ABI. + +## Effect on API resilience + +This change does not influence the ABI characteristics +of public APIs. + +## Alternatives considered + +The unavailable attribute attached to synthesized declarations might be more restrictive than necessary. +We considered the introduction of a new attribute to denote an incomplete type. This attribute would be used to prevent +operations that are not suitable for an incomplete type (ie. declaring a Swift type that conforms to an incomplete protocol). + +Allowing more use of the type however increases the complexity of the problem, and the likelihood of source compatibility issues. +For example, if the synthesized type is not file private, issues could arise with different files having different representations +of the type depending on which modules they import. + +We also attempted to make this implementation usable from the REPL. As mentioned, we had issues with `ClangImporter` caching +imports. We must invalidate the cache, at the very least to allow the new definition of type 'Foo' to be imported. This creates issues +where two incompatible representations of the type (`IncompleteFoo.Foo` and `CompleteFoo.Foo`) are in context at the same time. +To solve these issues we attempted the following: + +1. Invalidate all direct or indirect references to `IncompleteFoo.Foo` and replace them with `CompleteFoo.Foo`: + +Issues: + +- We do not know how to safely patch the type of existing instances at runtime (ex: local variable `foo` in the snippet above) +- Any imported declarations that depended upon a type, whose definition was incomplete at the time of import but is now complete, need reimporting + - We need new machinery in `ClangImporter` to support smart cache invalidation like this + - For all cached declarations, we need to keep track of the "completeness" of all types we depend on at the time of caching + - A simple cache invalidation heuristic such as "don't cache anything that depends upon an incomplete type" won't work either + - Besides the significant performance cost, the typechecker compares type declarations using pointer equality, expansion of the typechecker to recognize two imports of the same Clang declaration as equivalent would be required + +2. Convince the typechecker that `IncompleteFoo.Foo` and `CompleteFoo.Foo` are the same, *in all usages*: + +Issues: + +- It seems achievable to teach the typechecker that `IncompleteFoo.Foo` and `CompleteFoo.Foo` can be implicitly converted between one another, but that is not enough + - Any typechecking constraint that `CompleteFoo.Foo` satisfies must also be satisfied by `IncompleteFoo.Foo`. + - It might be possible to "inject" an extension into the REPL's context that adds any required methods, computed properties and protocol conformances to `IncompleteFoo.Foo`. However, it is not possible to add new inheritances. + +We believe that given these issues, it is better to disable the feature in the REPL entirely rather than provide a confusing experience. We believe that this +proposal should advance in some form regardless. Forward declarations are a huge problem for consumers of large Objective-C codebases looking to transition to Swift. +The LLDB / REPL experience already diverges from the compiled experience, is relatively niche, and we are not making the REPL experience any worse. Furthermore, +unlike in large scale projects, the existing solution of "import the definition" is perfectly reasonable for small REPL sessions. If this divergence between the +REPL and compiled experience is unacceptable, we think it could be reasonable to gate the feature behind a flag. This flag could potentially also be enabled in +the REPL, but with appropriate warnings and "at your own risk". + +That said, it would be great if this feature could come to the REPL eventually, but the REPL should not stand in the way of progress in the compiled experience. + +## Acknowledgments + +Thank you to @drodriguez and [@rmaz](https://github.com/rmaz) for helping develop the idea. diff --git a/proposals/0385-custom-reflection-metadata.md b/proposals/0385-custom-reflection-metadata.md new file mode 100644 index 0000000000..8c7d571cf9 --- /dev/null +++ b/proposals/0385-custom-reflection-metadata.md @@ -0,0 +1,443 @@ +# Custom Reflection Metadata + +* Proposal: [SE-0385](0385-custom-reflection-metadata.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin), [Holly Borla](https://github.com/hborla), [Alejandro Alonso](https://github.com/Azoy), [Stuart Montgomery](https://github.com/stmontgomery) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Returned for revision** ([Rationale](https://forums.swift.org/t/returned-for-revision-se-0385-custom-reflection-metadata/63758)) +* Implementation: [PR#1](https://github.com/apple/swift/pull/62426), [PR#2](https://github.com/apple/swift/pull/62738), [PR#3](https://github.com/apple/swift/pull/62818), [PR#4](https://github.com/apple/swift/pull/62850), [PR#5](https://github.com/apple/swift/pull/62920), [PR#6](https://github.com/apple/swift/pull/63057) + +## Introduction + +In Swift, declarations are annotated with attributes to opt into both built-in language features (e.g. `@available`) and library functionality (e.g. `@RegexComponentBuilder`). This proposal introduces the ability to attach library-defined reflection metadata to declarations using custom attributes, which can then be queried by the library to opt client code into library functionality. + +Previous Swift Forum discussions + +* [Custom attributes](https://forums.swift.org/t/custom-attributes/13976) +* [Pitch: introduce custom attributes](https://forums.swift.org/t/pitch-introduce-custom-attributes/21335) + +## Motivation + +There are some problem domains in which it can be beneficial for a library author to let a client annotate within their own code certain declarations that the library should be made aware of, since requiring the client call an explicit API instead would be too onerous, repetitive, or easy to forget. + +One classic example is **testing**: there is a common pattern in unit testing libraries where users define a type that extends one of the library's types, they annotate some of its methods as tests, and the library locates, initializes, and runs all tests automatically. There is no official mechanism in Swift to implement this test discovery pattern today, however XCTest—the language's current de-facto standard test library—has longstanding workarounds: + +* On Apple platforms, XCTest relies on the Objective-C runtime to enumerate all subclasses of a known base class and their methods, and it considers all instance methods with a supported signature and name prefixed with "test" a test method. +* On other platforms, XCTest is typically used from a package, and the Swift Package Manager has special logic to introspect build-time indexer data, locate test methods, and explicitly pass the list of discovered tests to XCTest to run. + +XCTest's current approach has some drawbacks and limitations: + +* Users must adhere to a strict naming convention by prefixing all test methods with the word "test". This prefix can be redundant since all tests include it, and may surprise users if they accidentally use that prefix on a non-test method since the behavior is implicit. +* Since tests are declared implicitly, there is no way for a user to provide additional details about an individual test or group of tests. It would be useful to have a way to indicate whether a test is enabled, its requirements, or other metadata, for example, so that the testing library could use this information to inform how it executes tests and offer more powerful features. +* The lack of a built-in runtime discovery mechanism means that related tools (such as Swift Package Manager) require specialized discovery logic for each test library they support. This makes adding support for alternate test libraries to those tools very difficult and increases their implementation complexity. + +Registering code to be discovered by a framework is a common pattern across Swift programs. For example, a program that uses a plugin architecture commonly uses a protocol for the interface of the plugin, which is then implemented on concrete types in clients. This pattern imposes error-prone registration boilerplate, where clients must explicitly supply a list of concrete plugin types or explicitly register individual plugin types to be used by the framework before the framework needs them. + +More generally, annotating parts of a program with metadata to be used by other parts of the program has use-cases beyond registration patterns. Consider the `Persisted` property wrapper from the [Realm](https://github.com/realm/realm-swift) package: + +```swift +@propertyWrapper +public struct Persisted { ... } + +class Dog: Object { + @Persisted var name: String + @Persisted var age: Int +} +``` + +To support [advanced schema customization](https://forums.swift.org/t/se-0385-custom-reflection-metadata/62777/19), the property wrapper could store a string that provides a custom name for the underlying database column, specified in the attribute arguments, e.g. `@Persisted(named: "CustomName")`. However, storing this metadata in the property wrapper requires additional storage for each _instance_ of the containing type, even though the metadata value is fixed for the declaration the property wrapper is attached to. In addition to higher memory overload, the metadata values are evaluated eagerly, and for each instantiation of the containing type, rendering property-wrapper instance metadata too expensive for this use case. + + +## Proposed solution + +* A new built-in attribute `@reflectionMetadata` that can be applied to structs, enums, classes, and actors. +* Types annotated with this built-in attribute can be used as custom attributes on declarations that can be used as values. + * The custom attribute can have additional arguments; the custom attribute application will turn into an initializer call on the attribute type, passing in the declaration value as the first argument. +* A reflection API that can gather all declarations with a given custom attribute attached, which lazily constructs the metadata values when invoked. + +Combined with [attached macros](https://forums.swift.org/t/pitch-attached-macros/62812), the `@Persisted` property wrapper in Realm can evolve into a macro attached to persistent types, combined with custom metadata attributes that provide schema customization for specific declarations: + +```swift +@reflectionMetadata +struct Named { + let name: String + + init(attachedTo: T.Type, _ name: String) { + self.name = name + } +} + +@Persisted +class Dog: Object { + var name: String + @Named("CustomName") var age: Int +} +``` + +This approach completely eliminates initialization overhead of using property wrappers, provides separate storage of custom metadata values, and enables lazy initialization of metadata values that is only invoked when the framework requests the metadata. + +## Detailed design + +### Declaring reflection metadata attributes + +Reflection metadata custom attributes are declared by attaching the built-in `@reflectionMetadata` attribute to a nominal type, i.e. a struct, enum, class, or actor: + +```swift +@reflectionMetadata +struct Example { ... } +``` + +A reflection metadata type must have a synchronous initializer of the form `init(attachedTo:)`. The type of the `attachedTo:` parameter dictates which types of declarations the custom attribute can be applied to, as described in the following section. + +### Applications of reflection metadata types + +Reflection metadata custom attributes can be applied to any declaration that can be used as a first-class value in Swift, including: + +* Types +* Global functions +* Static methods +* Instance methods, both non-mutating and mutating +* Instance properties + +Reflection metadata types opt into which kinds of declarations are supported based on their initializer overloads which begin with a parameter labeled `attachedTo:`. For an application of a reflection metadata attribute to be well-formed, the reflection metadata type must declare an initializer that accepts the appropriate value as the first argument. Applications of a reflection metadata type to a declaration will synthesize an initializer call with the attribute arguments, and the declaration value passed as the first initializer argument: + +* Types will pass a metatype. +* Global functions will pass an unapplied function reference. +* Static methods on a type `T` will pass a function which calls the method on the metatype `T.Type` passed as the first parameter. +* Instance methods on a type `T` will pass a function which calls the method on an instance `T` passed as the first parameter. The function will support `mutating` instance methods when the first parameter is declared `inout T`. +* Instance properties will pass a key-path. + +```swift +@reflectionMetadata +struct Flag { + // Initializer that accepts a metatype of a nominal type + init(attachedTo: T.Type) { + // ... + } + + // Initializer that accepts an unapplied reference to a global function + init(attachedTo: (Args) -> Result) { + // ... + } + + // Initializer that accepts a function which calls a static method + init(attachedTo: (T.Type, Args) -> Result) { + // ... + } + + // Initializer that accepts a function which calls an instance method + init(attachedTo: (T, Args) -> Result) { + // ... + } + + // Initializer that accepts a function which calls a mutating instance method + init(attachedTo: (inout T, Args) -> Result) { + // ... + } + + // Initializer that accepts a reference to an instance property + init(attachedTo: KeyPath, custom: Int) { + // ... + } +} + +// The compiler will synthesize the following initializer call +// -> Flag.init(attachedTo: doSomething) +@Flag func doSomething(_: Int, other: String) {} + +// The compiler will synthesize the following initializer call +// -> Flag.init(attachedTo: Test.self) +@Flag +struct Test { + // The compiler will synthesize the following initializer call + // -> Flag.init(attachedTo: { metatype in metatype.computeStateless() }) + @Flag static func computeStateless() {} + + // The compiler will synthesize the following initializer call + // -> Flag.init(attachedTo: { instance, values in instance.compute(values: values) }) + @Flag func compute(values: [Int]) {} + + var state = 1 + + // The compiler will synthesize the following initializer call + // -> Flag.init(attachedTo: { (instance: inout Test) in instance.incrementState() }) + @Flag mutating func incrementState() { + state += 1 + } + + // The compiler will synthesize the following initializer call + // -> Flag.init(attachedTo: \Test.answer, custom: 42) + @Flag(custom: 42) var answer: Int = 42 +} +``` + +#### Restrictions on custom reflection metadata application + +A given declaration can have multiple reflection metadata attributes as long as a given reflection metadata type only appears once: + +```swift +@Flag @Ignore func ignored() { 🟢 + // ... +} + +@Flag @Flag func specialFunction() { 🔴 + ^ error: duplicate reflection metadata attribute + // ... +} +``` + +Reflection metadata attributes must be applied at either the primary declaration of a type or in an unavailable unconstrained extension of the type within the same module as the type’s primary declaration. Unavailable extensions are supported to allow API implementers a way to opt-out from an attribute. Applying the attribute to a type in an available/constrained extension or in extension outside its module is prohibited to prevent the same type from having multiple reflection metadata annotations of the same type. + +```swift +@available(*, unavailable) +@Flag extension MyType { 🟢 if extension is in the same module +} +``` + +```swift +@Flag extension MyType { 🔴 + ^ error: cannot associate reflection metadata @Flag with MyType in extension +} +``` + +```swift +@Flag extension MyType where ... { 🔴 + ^ error: cannot associate reflection metadata @Flag with MyType in constrained extension +} +``` + +Declarations with custom reflection metadata attributes must be fully concrete: + +```swift +struct GenericType { + @Flag + var genericValue: T 🔴 + ^ error +} + +extension GenericType where T == Int { + @Flag + var concreteValue: Int // okay +} +``` + +Generic declarations cannot be discovered through the Reflection query that gathers all instances of reflection metadata, because generic values cannot be represented in a higher-kinded way in Swift; generic values must always have substitutions at runtime. Generic declarations could be supported in the future by adding reflection queries for the other direction, e.g. a query to return the custom reflection metadata for a given key-path `\Generic.value`. + + +### Inference of reflection metadata attributes + +A reflection metadata attribute can be applied to a protocol: + +```swift +@EditorCommandRecord +protocol EditorCommand { /* ... */ } +``` + +Conceptually, the reflection metadata attribute is applied to the generic `Self` type that represents the concrete conforming type. When a protocol conformance is written at the primary declaration of a concrete type, the reflection metadata attribute is inferred: + +```swift +// @EditorCommandRecord is inferred +struct SelectWordCommand: EditorCommand { /* ... */ } +``` + +If the protocol conformance is written in an extension on the conforming type, attribute inference is prohibited. A reflection metadata attribute applied to a protocol is a form of requirement, so such conformances declared in extensions are invalid unless the primary declaration already has the explicit reflection metadata attribute: + +```swift +// Error unless the primary declaration of 'SelectWordCommand' has '@EditorCommandRecord' +extension SelectWordCommand : EditorCommand { 🔴 + // ... +} +``` + +Reflection metadata attributes applied to protocols cannot have additional attribute arguments; attribute arguments must be explicitly written on the conforming type. + +A type which conforms to a protocol that has a reflection metadata attribute may specify the attribute explicitly. This can be useful if the reflection metadata type includes additional parameters in its `init(attachedTo: ...)` overload, since it allows the conforming type to pass arguments for those parameters: + +```swift +// Overrides the inferred `@EditorCommandRecord` attribute from `EditorCommand` +@EditorCommandRecord(keyboardShortcut: "j", modifier: .command) +struct SelectWordCommand: EditorCommand { /* ... */ } +``` + +### Accessing metadata through Reflection + +With the introduction of the new [Reflection](https://forums.swift.org/t/pitch-reflection/61438) module, we feel a natural place to reflectively retrieve these attributes is there. The following Reflection APIs provide the runtime query for custom reflection metadata: + +```swift +/// Get all the instances of a custom reflection attribute wherever it's attached to. +/// +/// - Parameters: +/// - type: The type of the attribute that is attached to various sources. +/// - Returns: A sequence of attribute instances of `type` in no particular +/// order. +public enum Attribute { + public static func allInstances(of type: T.Type) -> AttributeInstances +} + +/// A sequence wrapper over some runtime attribute instances. +/// +/// Instances of `AttributeInstances` are created with the +/// `Attribute.allInstances(of:)` function. +public struct AttributeInstances {} + +extension AttributeInstances: IteratorProtocol { + @inlinable + public mutating func next() -> T? +} + +extension AttributeInstances: Sequence {} +``` + +This API will retrieve all of the instances of your reflection attribute across all modules. Instances of metadata types are initialized in the Reflection query to gather the metadata. Attributes who are not available in the current running OS, i.e. because the `attachedTo` declaration is not available as described in the following section, will be excluded from the results. + +### Magic literals in custom reflection metadata attributes + +When custom reflection metadata type is accessed through the Reflection APIs, magic literals - `#function`, `#file`, `#line`, and `#column` associated with `init(attachedTo:)` would behave in a special way. Even though in such cases `init(attachedTo:)` is called from a special generator function `#function` literal is still going to point to the declaration attribute is attached to, and `#file`, `#line`, and `#column` are going to point to the attribute itself at the point of use or to the declaration if the attribute has been inferred. + +**test.swift** + +```swift + 1: @reflectionMetadata + 2: struct Flag { + 3: init(attachedTo: T.Type, + 4: func: String = #function, + 5: file: String = #file, + 6: line: Int = #line, + 7: column: Int = #column) {} + 8: + 9: init(attachedTo: KeyPath, +10: func: String = #function, +11: file: String = #file, +12: line: Int = #line, +13: column: Int = #column) {} +14: } +15: +16: struct Test { +17: @Flag var value: Int = 42 +18: } +19: +20: @Flag +21: protocol Flagged {} +22: +23: struct InferredTest : Flagged {} +``` + +**other.swift** + +```swift +1: let flags = Attribute.allInstances(of: Flag.self) +``` + +`Flag.init(attachedTo:)` associated with `Test.value` in this case is going to receive the following information: + +* `#function` = `"value"` +* `#file` = `"test.swift"` +* `#line` = `17` +* `#column` = `4` + +`Flag.init(attachedTo:)` for implicitly inferred attribute on `InferredTest` in this case is going to receive the following information: + +* `#function` = `"InferredTest"` +* `#file` = `"test.swift"` +* `#line` = `23` +* `#column` = `1` + +We think that this behavior provides the most benefit to the users because it preserves all of the information about attribute locations. + + +### API Availability + +Custom metadata attributes can be attached to declarations with limited availability. The Reflection query for an individual instance of the metadata attribute type will be gated on a matching availability condition and will return `nil` for instances which are unavailable at runtime. For example: + +```swift +@available(macOS 12, *) +@Flag +struct NewType { /* ... */ } +``` + +The Reflection query that produces the `Flag` instance attached to `NewType` will effectively execute the following code: + +```swift +if #available(macOS 12, *) { + return Flag(attachedTo: NewType.self) +} else { + return nil +} +``` + +and if `nil` is returned, there will not be a `Flag` instance representing `NewType` included in the collection returned by `Attribute.allInstances(of:)`. + +## Alternatives considered + +### Extend other language features + +Some reviewers of the original pitch suggested that the motivating use cases could be addressed through a combination of improved Reflection capabilities and enhancing existing language features. For example: + +* We could use existing protocol conformance metadata to allow discovering all types conforming to a protocol. +* We could allow property wrappers to be used to discover properties via reflection. + +These suggestions have some notable downsides, however. Supporting discovery of all types that conform to *any* protocol would be very expensive, and the majority of protocols do not need this reflection capability. Opting-in to this capability via an attribute on protocols which require it is an intentional aspect of this feature’s design intended to mitigate this cost. + +It’s also important to note that a reflection API which *only* allows discovering types that conform to a protocol would be insufficient to satisfy some of the use cases which motivate this feature because it would not allow including additional, custom values in the reflection metadata. For example, the `@EditorCommandRecord(keyboardShortcut: "j", modifier: .command)` example shown above includes custom values on a type conforming to a protocol, and the design of this feature includes a way for the reflection query to retrieve these values in addition to the declaration the attribute was attached to. For types conforming to a protocol, similar functionality could be provided through protocol requirements, but this strategy does not generalize to enable providing custom metadata on functions or computed properties. + +Regarding the use of property wrappers to represent metadata on properties: We feel that property wrappers are not an ideal tool for reflection metadata because they require an instance of the backing property to be stored for each instance, even though the wrapper is constant per-declaration. Property wrappers that are *only* used for reflection metadata don’t need to introduce any access indirection of the wrapped value, either. The value itself can simply be stored inline in the type, rather than synthesizing computed properties. + +### Using reflection types in the `init(attachedTo:)` signature + +We considered using types from the [Reflection](https://forums.swift.org/t/pitch-reflection/61438) module to represent declarations which have reflection attributes. For example, Reflection’s `Field` could be used as the type of the first parameter in `init(attachedTo:)` when a reflection attribute is attached to a property declaration. + +But this design would not allow constraining the types of the declaration(s) the reflection attribute can be attached to using techniques like generic requirements or additional parameters after `attachedTo:` in an initializer, since Reflection types do not expose the interface type of the declaration they represent. For example, `Field` is not parameterized on the field’s type, which would prevent compile-time enforcement of requirements. + +### Use static methods instead of `init(attachedTo:)` overloads + +We considered using static methods such as `buildMetadata(attachedTo:)` instead of overloads of `init(attachedTo:)` on reflection metadata types to generate metadata instances. This could potentially allow the overloads of `buildMetadata` to return a different type than `Self`, or even an associated type from some protocol. For example: + +```swift +// Defined in either the standard library or Reflection +protocol Attribute { + associatedtype Metadata +} + +// Example usage +@reflectionMetadata +struct Flag: Attribute { + static func buildMetadata(attachedTo: ...) -> Metadata { /* ... */ } +} +``` + +This alternative has a potential advantage of making it easier for `@propertyWrapper` types to also act as `@reflectionMetadata` types, because it would mean that the storage for any additional, custom values used for metadata purposes only (which are constant for every instance of the declared property) could be stored separately rather than having those values be stored redundantly in every instance of a property wrapper. + +### Alternative attribute names + +We considered several alternative spellings of the attribute used to declare a reflection metadata type: + +* `@runtimeMetadata` +* `@dynamicMetadata` +* `@metadata` +* `@runtimeAnnotation` +* `@runtimeAttribute` +* `@reflectionAnnotation` + +### Bespoke `@test` attribute + +A previous Swift Evolution discussion suggested [adding a built-in `@test` attribute](https://forums.swift.org/t/rfc-in-line-tests/12111) to the language. However, registration is a general code pattern that is also used outside of testing, so allowing libraries to declare their own domain-specific attributes is a more general approach that supports a wider set of use cases. + +## Acknowledgments + +Thank you to [Thomas Goyne](https://forums.swift.org/u/tgoyne) for surfacing use cases in the Realm Swift project and insights into alternative design directions. + +## Revision history + +### Changes after first pitch + +* Changed the proposed function signature for reflection metadata type initializer overloads for instance methods to accept `T` as the first parameter, instead of an unapplied function reference, and allow `inout T` to support `mutating` instance methods. +* Changed the proposed function signature for reflection metadata type initializer overloads for static methods to accept `T.Type` as the first parameter, instead of an unapplied function reference. +* Changed the spelling of the proposed attribute from `@runtimeMetadata` to `@reflectionMetadata`. +* Added `@reflectionAnnotation` (suggested by @xedin) to the list of alternative attribute spellings considered. +* Updated the list of supported use cases in the "Applications of reflection metadata types" section by separating global functions and static methods into separate bullets, to describe their differing type signatures. In particular, the function parameter for static methods now has type `(T.Type, Args) -> Result`. +* Clarified paragraph describing where reflection metadata attribute can be applied, to mention it is allowed in extensions of a type within the same module as the type's primary declaration, just not in extensions outside the module. +* Mentioned the ability to explicitly specify a reflection attribute on a type conforming to a protocol with that attribute, and described how that can be useful for specifying additional custom values. Added a code example of this. +* Changed the proposed name and return type of the Reflection API to `func allInstances(of type: T.Type) -> AttributeInstances`, returning a custom `Sequence` type whose type is `T`. Clarified that the returned sequence will omit values which do not satisfy the API availability conditions at runtime, rather than including `nil` values for them. +* Added discussion of some alternatives that were considered involving extending Reflection capabilities and other existing language features. +* Added discussion of an alternative that was considered about using Reflection types as the parameters to `init(attachedTo:)`. +* Added discussion of an alternative that was considered about using static methods instead of `init(attachedTo:)` overloads. +* Clarified interaction between extensions and custom reflection metadata attributes. diff --git a/proposals/0386-package-access-modifier.md b/proposals/0386-package-access-modifier.md new file mode 100644 index 0000000000..da88d20016 --- /dev/null +++ b/proposals/0386-package-access-modifier.md @@ -0,0 +1,334 @@ +# New access modifier: `package` + +* Proposal: [SE-0386](0386-package-access-modifier.md) +* Authors: [Ellie Shin](https://github.com/elsh), [Alexis Laferriere](https://github.com/xymus) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#61546](https://github.com/apple/swift/pull/62700), [apple/swift#62704](https://github.com/apple/swift/pull/62704), [apple/swift#62652](https://github.com/apple/swift/pull/62652), [apple/swift#62652](https://github.com/apple/swift/pull/62652) +* Review: ([pitch](https://forums.swift.org/t/new-access-modifier-package/61459)) ([first review](https://forums.swift.org/t/se-0386-package-access-modifier/62808)) ([second review](https://forums.swift.org/t/second-review-se-0386-package-access-modifier/64086)) ([acceptance](https://forums.swift.org/t/accepted-se-0386-package-access-modifier/64904)) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/28fd2fb9b7258117f912cec5e5f7eb178520fbf2/proposals/NNNN-package-access-modifier.md), [2](https://github.com/swiftlang/swift-evolution/blob/32e51946296f67be79a58a8c23eb9d7460a06232/proposals/0386-package-access-modifier.md), [3](https://github.com/swiftlang/swift-evolution/blob/4a3a11b18037526cf8d83a9d10b22b94890727e8/proposals/0386-package-access-modifier.md) + +## Introduction + +This proposal introduces `package` as a new access modifier. Currently, to access a symbol in another module, that symbol needs to be declared `public`. However, a symbol being `public` allows it to be accessed from any module at all, both within a package and from outside of a package, which is sometimes undesirable. We need a new access modifier to enable more control over the visibility scope of such symbols. + +## Motivation + +At the most basic level, every Swift program is just a collection of declarations: functions, types, variables, and so on. In principle, every level of organization above this is arbitrary; all of those declarations could be piled into a single file, compiled, and run. In reality, Swift programs are organized into separate files, directories, libraries, and so on. At each level, this organization reflects programmer judgment about relationships, both in the code and in how it is developed. + +As a language, Swift recognizes some of these levels. Modules are the smallest unit of library structure, with an independent interface and non-cyclic dependencies, and it makes sense for Swift to recognize that in both namespacing and access control. Files are the smallest grouping beneath that and are often used to collect tightly-related declarations, so they also make sense to respect in access control. + +Packages, as expressed by the Swift Package Manager, are a unit of code distribution. Some packages contain just a single module, but it's frequently useful to split a package's code into multiple modules. For example, when a module contains some `internal` helper APIs, those APIs can be split out into a utility module and maybe reused by other modules or packages. + +However, because Swift does not recognize organizations of code above the module level, it is not possible to create APIs like this that are purely internal to the package. To be usable from other modules within the package, the API must be public, but this means it can also be used outside of the package. This allows clients to form unwanted source dependencies on the API. It also means the built module has to export the API, which has negative implications for code size and performance. + +For example, here’s a scenario where a client has access to a utility API from a package it depends on. The client `App` could be an executable or an Xcode project. It depends on a package called `gamePkg`, which contains two modules, `Game` and `Engine`. + + +Module `Engine` in `gamePkg`: +```swift +public struct MainEngine { + public init() { ... } + // Intended to be public + public var stats: String { ... } + // A helper function made public only to be accessed by Game + public func run() { ... } +} +``` + +Module `Game` in `gamePkg`: +```swift +import Engine + +public func play() { + MainEngine().run() // Can access `run` as intended since it's within the same package +} +``` + +Client `App` in `appPkg`: +```swift +import Game +import Engine + +let engine = MainEngine() +engine.run() // Can access `run` from App even if it's not an intended behavior +Game.play() +print(engine.stats) // Can access `stats` as intended +``` + +In the above scenario, `App` can import `Engine` (a utility module in `gamePkg`) and access its helper API directly, even though the API is not intended to be used outside of its package. + +Allowing this kind of unintended public access to package APIs is especially bad because packages are a unit of code distribution. Swift wants to encourage programs to be divided into modules with well-defined interfaces, so it enforces the boundaries between modules with access control. Despite being divided this way, it's not uncommon for closely-related modules to be written by closely-related (or even the same) people. Access control between such modules still serves a purpose — it promotes the separation of concerns — but if a module's interface needs to be fixed, that's usually easy to coordinate, maybe even as simple as a single commit. However, packages allow code to be shared much more broadly than a single small organization. The boundaries between packages often represent significant differences between programmers, making coordination around API changes much more difficult. For example, the developers of an open source package generally don't know most of their clients, and the standard recommendation is for such packages to only ever remove existing APIs in major-version releases. It's therefore particularly important to allow programmers to enforce these boundaries between packages. + +## Proposed solution + +Our goal is to introduce a mechanism to Swift to recognize a package as a unit in the aspect of access control. We propose to do so by introducing a new access modifier called `package`. The `package` access modifier allows symbols to be accessed from outside of their defining module, but only from other modules in the same package. This helps to set clear boundaries between packages. + +## Detailed design + +### `package` Keyword + +`package` is introduced as an access modifier. It cannot be combined with other access modifiers. +`package` is a contextual keyword, so existing declarations named `package` will continue to work. This follows the precedent of `open`, which was also added as a contextual keyword. For example, the following is allowed: + +```swift +package var package: String {...} +``` + +### Declaration Site + +The `package` keyword is added at the declaration site. Using the scenario above, the helper API `run` can be declared with the new access modifier like so: + +Module `Engine`: +```swift +public struct MainEngine { + public init() { ... } + public var stats: String { ... } + package func run() { ... } +} +``` + +The `package` access modifier can be used anywhere that the existing access modifiers can be used, e.g. `class`, `struct`, `enum`, `func`, `var`, `protocol`, etc. + +Swift requires that the declarations used in certain places (such as the signature of a function) be at least as accessible as the containing declaration. For the purposes of this rule, `package` is less accessible than `open` and `public` and more accessible than `internal`, `fileprivate`, and `private`. For example, a `public` function cannot use a `package` type in its parameters or return type, and a `package` function cannot use an `internal` type in its parameters or return type. Similarly, an `@inlinable` `public` function cannot use a `package` declaration in its implementation, and an `@inlinable` `package` function cannot use an `internal` declaration in its implementation. + +### Use Site + +The `Game` module can access the helper API `run` since it is in the same package as `Engine`. + +Module `Game`: +```swift +import Engine + +public func play() { + MainEngine().run() // Can access `run` as it is a package symbol in the same package +} +``` + +However, if a client outside of the package tries to access the helper API, it will not be allowed. + +Client `App`: +```swift +import Game +import Engine + +let engine = MainEngine() +engine.run() // Error: cannot find `run` in scope +``` + +### Package Names + +Swift as a language leaves it up to the build system to define the boundaries of a package. The compiler considers two modules to belong to the same package if they were built with the same package name, which is just a Unicode string. The package name is not exposed in the source language, so its exact contents are not significant as long as it is unique to a "package". + +A new flag `-package-name` is passed down to a commandline invocation, as follows. + +```sh +swiftc -module-name Engine -package-name gamePkg ... +swiftc -module-name Game -package-name gamePkg ... +swiftc -module-name App -package-name appPkg ... +``` + +When building the `Engine` module, the package name `gamePkg` is recorded in the built interface to the module. When building `Game`, its package name `gamePkg` is compared with the package name recorded in `Engine`'s built interface; since they match, `Game` is allowed to access `Engine`'s `package` declarations. When building `App`, its package name `appPkg` is different from `gamePkg`, so it is not allowed to access `package` symbols in either `Engine` or `Game`, which is what we want. + +If `-package-name` is not given, the `package` access modifier is disallowed. Swift code that does not use `package` access will continue to build without needing to pass in `-package-name`. Modules built without a package name are never considered to be in the same package as any other module. + +The build system should make a best effort to ensure that package names are unique. The Swift Package Manager already has a concept of a package identity string for every package. This string is verified to be unique, and it already works as a package name, so SwiftPM will pass it down automatically. Other build systems such as Bazel may need to introduce a new build setting for a package name. Since it needs to be unique, a reverse-DNS name may be used to avoid clashing. + +If a target needs to be excluded from the package boundary, that can be done with a new `packageAccess` setting in the manifest, like so: + +```swift + .target(name: "Game", dependencies: ["Engine"], packageAccess: false) +``` + +The `packageAccess` setting is set to `true` by default, and the target is built with `-package-name PACKAGE_ID` where `PACKAGE_ID` is the manifest's package identifier. If `packageAccess` is set to `false`, `-package-name` is not passed when building the target, thus the target has no access to any package symbols; it essentially acts as if it's a client outside of the package. This would be useful for an example app or a black-box test target in the package. + +### Package Symbols Distribution + +When the Swift frontend builds a `.swiftmodule` file directly from source, the file will include the package name and all of the `package` declarations in the module. When the Swift frontend builds a `.swiftinterface` file from source, the file will include the package name, but it will put `package` declarations in a secondary `.package.swiftinterface` file. When the Swift frontend builds a `.swiftmodule` file from a `.swiftinterface` file that includes a package name, but it does not have the corresponding `.package.swiftinterface` file, it will record this in the `.swiftmodule`, and it will prevent this file from being used to build other modules in the same package. + +### Package Symbols and `@inlinable` + +`package` types can be made `@inlinable`. Just as with `@inlinable public`, not all symbols are usable within the body of `@inlinable package`: they must be `open`, `public`, or `@usableFromInline`. The `@usableFromInline` attribute can be applied to `package` besides `internal` declarations. These attributed symbols are allowed in the bodies of `@inlinable public` or `@inlinable package` declarations (that are defined anywhere in the same package). Just as with `internal` symbols, the `package` declarations with `@usableFromInline` or `@inlinable` are stored in the public `.swiftinterface` for a module. + +Here's an example. + +```swift +func internalFuncA() {} +@usableFromInline func internalFuncB() {} + +package func packageFuncA() {} +@usableFromInline package func packageFuncB() {} + +public func publicFunc() {} + +@inlinable package func pkgUse() { + internalFuncA() // Error + internalFuncB() // OK + packageFuncA() // Error + packageFuncB() // OK + publicFunc() // OK +} + +@inlinable public func publicUse() { + internalFuncA() // Error + internalFuncB() // OK + packageFuncA() // Error + packageFuncB() // OK + publicFunc() // OK +} +``` + +### Subclassing and Overrides + +Access control in Swift usually doesn't distinguish between different kinds of use. If a program has access to a type, for example, that gives the programmer a broad set of privileges: the type name can be used in most places, values of the type can be borrowed, copied, and destroyed, members of the type can be accessed (up to the limits of their own access control), and so on. This is because access control is a tool for enforcing encapsulation and allowing the future evolution of code. Broad privileges are granted because restricting them more precisely usually doesn't serve that goal. + +However, there are two exceptions. The first is that Swift allows `var` and `subscript` to restrict mutating accesses more tightly than read-only accesses; this is done by writing a separate access modifier for the setter, e.g. `private(set)`. The second is that Swift allows classes and class members to restrict subclassing and overriding more tightly than normal references; this is done by writing `public` instead of `open`. Allowing these privileges to be separately restricted serves the goals of promoting encapsulation and evolution. + +Because setter access levels are controlled by writing a separate modifier from the primary access, the syntax naturally extends to allow `package(set)`. However, subclassing and overriding are controlled by choosing a specific keyword (`public` or `open`) as the primary access modifier, so the syntax does not extend to `package` the same way. This proposal has to decide what `package` by itself means for classes and class members. It also has to decide whether to support the options not covered by `package` alone or to leave them as a possible future direction. + +Here is a matrix showing where symbols with each current access level can be used or overridden: + + + + + + + + + + + + + + + + + + + + + + + + + + +
Accessible AnywhereAccessible in Module
Subclassable Anywhereopen(illegal)
Subclassable in Modulepublicinternal
Subclassable Nowherepublic finalinternal final
+ +With `package` as a new access modifier, the matrix is modified like so: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Accessible AnywhereAccessible in PackageAccessible in Module
Subclassable Anywhereopen(illegal)(illegal)
Subclassable in Package?(a)?(b)(illegal)
Subclassable in Modulepublicpackageinternal
Subclassable Nowherepublic finalpackage finalinternal final
+ + +This proposal takes the position that `package` alone should not allow subclassing or overriding outside of the defining module. This is consistent with the behavior of `public` and makes `package` fit into a simple continuum of ever-expanding privileges. It also allows the normal optimization model of `public` classes and methods to still be applied to `package` classes and methods, implicitly making them `final` when they aren't subclassed or overridden, without requiring a new "whole package optimization" build mode. + +However, this choice leaves no way to spell the two combinations marked in the table above with `?`. These are more complicated to design and implement and are discussed in Future Directions. + +## Future Directions + +### Subclassing and Overrides + +The entities marked with `?(a)` and `?(b)` from the matrix above both require accessing and subclassing cross-modules in a package (`open` within a package). The only difference is that (b) hides the symbol from outside of the package and (a) makes it visible outside. Use cases involving (a) should be rare but its underlying flow should be the same as (b) except its symbol visibility. + +Potential solutions include introducing new keywords for specific access combinations (e.g. `packageopen`), allowing `open` to be access-qualified (e.g. `open(package)`), and allowing access modifiers to be qualified with specific purposes (e.g. `package(override)`). + +### Package-Private Modules + +Sometimes entire modules are meant to be private to the package that provides them. Allowing this to be expressed directly would allow these utility modules to be completely hidden outside of the package, avoiding unwanted dependencies on the existence of the module. It would also allow the build system to automatically namespace the module within the package, reducing the need for [explicit module aliases](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0339-module-aliasing-for-disambiguation.md) when utility modules of different packages share a name (such as `Utility`) or when multiple versions of a package need to be built into the same program. + +### Grouping Within A Package + +The basic language design of this proposal can work for any group of related modules, but the application of that design in SPM allows only a single such group per SPM package. Developers with complex SPM packages sometimes find that they have multiple architectural "layers" within a single package and may wish to make `package` apply only within a layer. Logically, it makes some sense to put each layer in its own package. Pragmatically, because different SPM packages must currently live in separate repositories and be independently versioned, splitting a package that way introduces a huge amount of extra complexity to the development process, and it is not something that should be done casually. + +There are several reasonable ways that SPM could evolve to support multiple layers within a single package repository. One would be to allow targets to be grouped within a manifest, such as by adding a `group` parameter to `.target`. An earlier version of this proposal suggested this and even designed the `packageAccess:` exclusion feature around it. However, this would tend to lead to large, complex manifests that mingled the details of all the layers together. A very different approach would be to allow the creation of sub-packages within a repository, each with its own manifest. SPM would treat these sub-packages as logically separate units that happen to share a single repository and version. Because they would be described in independent manifests, they would feel like different packages, and it would make sense for `package` access to be scoped within them. + +### Optimizations + +* A package can be treated as a resilience domain, even with library evolution enabled which makes modules resilient. The Swift frontend will assume that modules defined in the same package will always be rebuilt together and do not require a resilient ABI boundary between them. This removes the need for performance and code size overhead introduced by ABI-resilient code generation, and it also eliminates language requirements such as `@unknown default` for a non-`frozen enum`. + +* By default, `package` symbols are exported in the final libraries/executables. It would be useful to introduce a build setting that allows users to hide package symbols for statically linked libraries; this would help with code size and build time optimizations. + +## Source Compatibility + +The new `package` access modifier is a contextual keyword. Existing symbols that are named `package` should not require renaming, and existing source code should continue to work. + +## Effect on ABI stability + +Boundaries between separately-built modules within a package are still potentially ABI boundaries. The ABI for package symbols is not different from the ABI for public symbols, although it might be considered in the future to add an option to not export package symbols that can be resolved within an image. + +## Alternatives considered + +### `@_spi` + +One workaround for the scenario in the Motivation would be to use the `@_spi(groupName)` attribute, which allows part of the API of a module to be hidden unless it is imported in a special way that explicitly requests access to it. This is an unsatisfying alternative to package-level access control because it is designed around a very different situation. An SPI is a "hole" in the normal public interface, one meant for the use of a specific client. That client is typically outside of the module's normal code-distribution boundary, but the module authors still have a cooperative working relationship. This relationship is reflected in the design of `@_spi` in multiple ways: + +* First, access to the SPI is granted to a specific client by name. This is a clear and unavoidable communication of intent about who is meant to be using the SPI. Other clients can still pose as this client and use the SPI, but that would be a clear breach of trust with predictable consequences. + +* Second, clients must explicitly request the SPI by name. This means that clients must opt in to using the SPI in every file, which works to limit its accidental over-use even by the intended client. It also means that SPI use is obvious in the code, which code reviewers can see and raise questions about, and which SPI authors can easily find with a code search. + +The level of care implied by these properties is appropriate for a carefully-targeted hole in an API that must cross a code-distribution boundary and will therefore require equal amounts of care to ever modify or close. That rarely applies to two modules within the same package, where a package-level interface can ideally be changed with just a quick edit to a few different parts of a repository. The `@_spi` attribute is intentionally designed to not be as lightweight as a package-local change should be. + +* `@_spi` would also not be easy to optimize. By design, clients of an SPI can be anywhere, making it effectively part of the public ABI of a module. To avoid exporting an SPI, the build system would have to know about that specific SPI group and promise the compiler that it was only used in the current built image. Recognizing that all of the modules in a package are being linked into the same image and can be optimized together is comparatively easy for a build system and so is a much more feasible future direction. + +### `@_implementationOnly` + +Another workaround for the scenario in the Motivation is to use the `@_implementationOnly` attribute on the import of a module. This attribute causes the module to be imported only for the use of the current module; clients of the current module don't implicitly transitively import the target module, and the symbols of the target module are restricted from appearing in the `public` API of the current module. This would prevent clients from accidentally using APIs from the target module. However, this is a very incomplete workaround for the lack of package-level access control. For one, it doesn't actually prevent access to the module, which can still be explicitly imported and used. For another, it only works on an entire module at a time, so a module cannot restrict some of its APIs to the package while making others available publicly. Taming transitive import would be a good future direction for Swift, but it does not solve the problems of package-level APIs. + +### Other Workarounds + +There are a few other workarounds to the absence of package-level access control, such as using `@testable` or the `-disable-access-control` flag. These are hacky subversions of Swift's language design, and they severely undermine the use of module boundaries for encapsulation. `-disable-access-control` is also an unstable and unsupported feature that can introduce build failures by causing symbol name collisions. + +### Introduce Submodules + +Instead of adding a new package access level above modules, we could allow modules to contain other modules as components. This is an idea often called "submodules". Packages would then define an "umbrella" module that contains the package's modules as components. However, there are several weaknesses in this approach: + +* It doesn't actually solve the problem by itself. Submodule APIs would still need to be able to declare whether they're usable outside of the umbrella or not, and that would require an access modifier. It might be written in a more general way, like `internal(MyPackage)`, but that generality would also make it more verbose. + +* Submodule structure would be part of the source language, so it would naturally be source- and ABI-affecting. For example, programmers could use the parent module's name to qualify identifiers, and symbols exported by a submodule would include the parent module's name. This means that splitting a module into submodules or adding an umbrella parent module would be much more impactful than desired; ideally, those changes would be purely internal and not change a module's public interface. It also means that these changes would end up permanently encoding package structure. + +* The "umbrella" submodule structure doesn't work for all packages. Some packages include multiple "top-level" modules which share common dependencies. Forcing these to share a common umbrella in order to use package-private dependencies is not desirable. + +* In a few cases, the ABI and source impact above would be desirable. For example, many packages contain internal Utility modules; if these were declared as submodules, they would naturally be namespaced to the containing package, eliminating spurious collisions. However, such modules are generally not meant to be usable outside of the package at all. It is a reasonable future direction to allow whole modules to be made package-private, which would also make it reasonable to automatically namespace them. + +### `@usableFromPackageInline` + +An earlier version of this proposal included a new attribute `@usableFromPackageInline`, which would have allowed an `internal` declaration to be used in the body of an `@inlinable package` declaration, but not in an `@inlinable public` declaration. Under the logic of this proposal, there is no good reason to make a declaration `@usableFromPackageInline internal` instead of simply `package`: the uses of the latter will be restricted to the package and therefore by assumption can still be easily found and reviewed. Furthermore, it is a goal of the Swift project to not require extensive `@inlinable` annotations just to enable basic optimizations between modules: there should be little reason in the long run to have an `@inlinable package` declaration at all. Therefore this attribute has been removed from the proposal. + +## Acknowledgments + +Doug Gregor, Becca Royal-Gordon, Allan Shortlidge, Artem Chikin, and Xi Ge provided helpful feedback and analysis as well as code reviews on the implementation. diff --git a/proposals/0387-cross-compilation-destinations.md b/proposals/0387-cross-compilation-destinations.md new file mode 100644 index 0000000000..40367a6464 --- /dev/null +++ b/proposals/0387-cross-compilation-destinations.md @@ -0,0 +1,605 @@ +# Swift SDKs for Cross-Compilation + +* Proposal: [SE-0387](0387-cross-compilation-destinations.md) +* Authors: [Max Desiatov](https://github.com/MaxDesiatov), [Saleem Abdulrasool](https://github.com/compnerd), [Evan Wilde](https://github.com/etcwilde) +* Review Manager: [Mishal Shah](https://github.com/shahmishal) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift-package-manager#5911](https://github.com/apple/swift-package-manager/pull/5911), + [apple/swift-package-manager#5922](https://github.com/apple/swift-package-manager/pull/5922), + [apple/swift-package-manager#6023](https://github.com/apple/swift-package-manager/pull/6023), + [apple/swift-package-manager#6186](https://github.com/apple/swift-package-manager/pull/6186) +* Review: ([pitch](https://forums.swift.org/t/pitch-cross-compilation-destination-bundles/61777)) + ([first review](https://forums.swift.org/t/se-0387-cross-compilation-destination-bundles/62875)) + ([second review](https://forums.swift.org/t/second-review-se-0387-cross-compilation-destination-bundles/64660)) + +## Table of Contents + +- [Introduction](#introduction) +- [Motivation](#motivation) +- [Proposed Solution](#proposed-solution) +- [Detailed Design](#detailed-design) + - [Swift SDK Bundles](#swift-sdk-bundles) + - [`toolset.json` Files](#toolsetjson-files) + - [`swift-sdk.json` Files](#swift-sdkjson-files) + - [Swift SDK Installation and Configuration](#swift-sdk-installation-and-configuration) + - [Using a Swift SDK](#using-a-swift-sdk) + - [Swift SDK Bundle Generation](#swift-sdk-bundle-generation) +- [Security](#security) +- [Impact on Existing Packages](#impact-on-existing-packages) +- [Prior Art](#prior-art) + - [Rust](#rust) + - [Go](#go) +- [Alternatives Considered](#alternatives-considered) + - [Extensions Other Than `.artifactbundle`](#extensions-other-than-artifactbundle) + - [Building Applications in Docker Containers](#building-applications-in-docker-containers) + - [Alternative Bundle Formats](#alternative-bundle-formats) +- [Making Swift SDK Bundles Fully Self-Contained](#making-swift-sdk-bundles-fully-self-contained) +- [Future Directions](#future-directions) + - [Identifying Platforms with Dictionaries of Properties](#identifying-platforms-with-dictionaries-of-properties) + - [SwiftPM Plugins for Remote Running, Testing, Deployment, and Debugging](#swiftpm-plugins-for-remote-running-testing-deployment-and-debugging) + - [`swift sdk select` Subcommand](#swift-sdk-select-subcommand) + - [SwiftPM and SourceKit-LSP Improvements](#swiftpm-and-sourcekit-lsp-improvements) + - [Source-Based Swift SDKs](#source-based-swift-sdks) + - [Swift SDK Bundles and Package Registries](#swift-sdk-bundles-and-package-registries) + +## Introduction + +Cross-compilation is a common development use case. When cross-compiling, we need to refer to these concepts: + +- a **toolchain** is a set of tools used to build an application or a library; +- a **triple** describes features of a given machine such as CPU architecture, vendor, OS etc, corresponding to LLVM's + triple; +- a **host triple** describes a machine where application or library code is built; +- a **target triple** describes a machine where application or library code is running; +- an **SDK** is a set of dynamic and/or static libraries, headers, and other resources required to generate code for the + target triple. + +When a triple of a machine on which the toolchain is built is different from the host triple, we'll call it a **build triple**. +The cross-compilation configuration itself that involves three different triples is called +[the Canadian Cross](https://en.wikipedia.org/wiki/Cross_compiler#Canadian_Cross). + +Let’s call a Swift toolchain and an SDK bundled together in an artifact bundle a **Swift SDK**. + +## Motivation + +In Swift 5.8 and earlier versions users can cross-compile their code with so called "destination files" passed to +SwiftPM invocations. These destination files are produced on an ad-hoc basis for different combinations of +host and target triples. For example, scripts that produce macOS → Linux destinations were created by both +[the Swift +team](https://github.com/apple/swift-package-manager/blob/swift-5.8-RELEASE/Utilities/build_ubuntu_cross_compilation_toolchain) +and [the Swift community](https://github.com/SPMDestinations/homebrew-tap). At the same time, the distribution process +of assets required for cross-compiling is cumbersome. After building a destination tree on the file system, required +metadata files rely on hardcoded absolute paths. Adding support for relative paths in destination's metadata and +providing a unified way to distribute and install required assets as archives would clearly be an improvement for the +multi-platform Swift ecosystem. + +The primary audience of this pitch are people who cross-compile from macOS to Linux. When deploying to single-board +computers supporting Linux (e.g. Raspberry Pi), building on such hardware may be too slow or run out of available +memory. Quite naturally, users would prefer to cross-compile on a different machine when developing for these platforms. + +In other cases, building in a Docker container is not always the best solution for certain development workflows. For +example, when working with Swift AWS Lambda Runtime, some developers may find that installing Docker just for building a +project is a daunting step that shouldn’t be required. + +The solution described below is general enough to scale for any host/target triple combination. + +## Proposed Solution + +Since a Swift SDK is a collection of binaries arranged in a certain directory hierarchy, it makes sense to distribute +it as an archive. We'd like to build on top of +[SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md) and +extend the `.artifactbundle` format to support this. + +Additionally, we propose introducing a new `swift sdk` CLI command for installation and removal of Swift SDKs on the +local filesystem. + +We introduce a notion of a top-level toolchain, which is the toolchain that handles user’s `swift sdk` +invocations. Parts of this top-level toolchain (linker, C/C++ compilers, and even the Swift compiler) can be overridden +with tools supplied in `.artifactbundle` s installed by `swift sdk` invocations. + +When the user runs `swift build` with the selected Swift SDK, the overriding tools from the corresponding bundle +are invoked by `swift build` instead of tools from the top-level toolchain. + +The proposal is intentionally limited in scope to build-time experience and specifies only configuration metadata, basic +directory layout for proposed artifact bundles, and some CLI helpers to operate on those. + +## Detailed Design + +### Swift SDK Bundles + +As a quick reminder for a concept introduced in +[SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md), an +**artifact bundle** is a directory that has the filename suffix `.artifactbundle` and has a predefined structure with +`.json` manifest files provided as metadata. + +The proposed structure of artifact bundles containing Swift SDKs looks like: + +``` +.artifactbundle +├ info.json +├ +│ ├ swift-sdk.json +│ ├ toolset.json +│ └ +├ +│ ├ swift-sdk.json +│ ├ toolset.json +│ └ +├ +┆ └┄ +``` + +For example, a Swift SDK bundle allowing to cross-compile Swift 5.9 source code to recent versions of Ubuntu from +macOS would look like this: + +``` +swift-5.9_ubuntu.artifactbundle +├ info.json +├ toolset.json +├ ubuntu_jammy +│ ├ swift-sdk.json +│ ├ toolset.json +│ ├ +│ ├ aarch64-unknown-linux-gnu +│ │ ├ toolset.json +│ │ └ +│ └ x86_64-unknown-linux-gnu +│ ├ toolset.json +│ └ +├ ubuntu_focal +│ ├ swift-sdk.json +│ └ x86_64-unknown-linux-gnu +│ ├ toolset.json +│ └ +├ ubuntu_bionic +┆ └┄ +``` + +Here each artifact directory is dedicated to a specific Swift SDK, while files specific to each triple are placed +in `aarch64-unknown-linux-gnu` and `x86_64-unknown-linux-gnu` subdirectories. + +`info.json` bundle manifests at the root of artifact bundles should specify `"type": "swiftSDK"` for +corresponding artifacts. Artifact identifiers in this manifest file uniquely identify a Swift SDK, and +`supportedTriples` property in `info.json` should contain host triples that a given Swift SDK supports. The rest +of the properties of bundle manifests introduced in SE-0305 are preserved. + +Here's how `info.json` file could look like for `swift-5.9_ubuntu.artifactbundle` introduced in the example +above: + +```json5 +{ + "artifacts" : { + "swift-5.9_ubuntu22.04" : { + "type" : "swiftSDK", + "version" : "0.0.1", + "variants" : [ + { + "path" : "ubuntu_jammy", + "supportedTriples" : [ + "arm64-apple-darwin", + "x86_64-apple-darwin" + ] + } + ] + }, + "swift-5.9_ubuntu20.04" : { + "type" : "swiftSDK", + "version" : "0.0.1", + "variants" : [ + { + "path" : "ubuntu_focal", + "supportedTriples" : [ + "arm64-apple-darwin", + "x86_64-apple-darwin" + ] + } + ] + } + }, + "schemaVersion" : "1.0" +} +``` + +### `toolset.json` Files + +We find that properties dedicated to tools configuration are useful outside of the cross-compilation context. Due to +that, separate toolset configuration files are introduced: + +```json5 +{ + "schemaVersion": "1.0", + "rootPath": "optional path to a root directory containing toolchain executables", + // If `rootPath` is specified, all relative paths below will be resolved relative to `rootPath`. + "swiftCompiler": { + "path": "", + "extraCLIOptions": [""] + }, + "cCompiler": { + "path": "", + "extraCLIOptions": [""] + }, + "cxxCompiler": { + "path": "", + "extraCLIOptions": [""] + }, + "linker": { + "path": "", + "extraCLIOptions": [""] + }, + "librarian": { + "path": "", + "extraCLIOptions": [""] + }, + "debugger": { + "path": "", + "extraCLIOptions": [""] + }, + "testRunner": { + "path": "", + "extraCLIOptions": [""] + }, +} +``` + +More types of tools may be enabled in toolset files in the future in addition to those listed above. + +Users familiar with CMake can draw an analogy between toolset files and CMake toolchain files. Toolset files are +designed to supplant previous ad-hoc ways of specifying paths and flags in SwiftPM, such as `SWIFT_EXEC` and `CC` +environment variables, which were applied in use cases unrelated to cross-compilation. We propose that +users also should be able to pass `--toolset ` option to `swift build`, `swift test`, and +`swift run`. + +We'd like to allow using multiple toolset files at once. With this users can "assemble" toolchains on the fly out of +tools that in certain scenarios may even come from different vendors. A toolset file can have an arbitrary name, and +each file should be passed with a separate `--toolset` option, i.e. `swift build --toolset t1.json --toolset t2.json`. + +All of the properties related to names of the tools are optional, which allows merging configuration from multiple +toolset files. For example, consider `toolset1.json`: + +```json5 +{ + "schemaVersion": "1.0", + "swiftCompiler": { + "path": "/usr/bin/swiftc", + "extraCLIOptions": ["-Xfrontend", "-enable-cxx-interop"] + }, + "cCompiler": { + "path": "/usr/bin/clang", + "extraCLIOptions": ["-pedantic"] + } +} +``` + +and `toolset2.json`: + +```json5 +{ + "schemaVersion": "1.0", + "swiftCompiler": { + "path": "/custom/swiftc" + } +} +``` + +With multiple `--toolset` options, passing both of those files will merge them into a single configuration. Tools passed +in subsequent `--toolset` options will shadow tools from previous options with the same names. That is, +`swift build --toolset toolset1.json --toolset toolset2.json` will build with `/custom/swiftc` and no extra flags, as +specified in `toolset2.json`, but `/usr/bin/clang -pedantic` from `toolset1.json` will still be used. + +Tools not specified in any of the supplied toolset files will be looked up in existing implied search paths that are +used without toolsets, even when `rootPath` is present. We'd like toolsets to be explicit in this regard: if a +tool would like to participate in toolset path lookups, it must provide either a relative or an absolute path in a +toolset. + +Tools that don't have `path` property but have `extraCLIOptions` present will append options from that property to a +tool with the same name specified in a preceding toolset file. If no other toolset files were provided, these options +will be appended to the default tool invocation. For example `pedanticCCompiler.json` that looks like this + +```json5 +{ + "schemaVersion": "1.0", + "cCompiler": { + "extraCLIOptions": ["-pedantic"] + } +} +``` + +in `swift build --toolset pedanticCCompiler.json` will pass `-pedantic` to the C compiler located at a default path. + +When cross-compiling, paths in `toolset.json` files supplied in Swift SDK bundles should be self-contained: +no absolute paths and no escaping symlinks are allowed. Users are still able to provide their own `toolset.json` files +outside of artifact bundles to specify additional developer tools for which no relative "non-escaping" path can be +provided within the bundle. + +### `swift-sdk.json` Files + +Note the presence of `swift-sdk.json` files in each `` subdirectory. These files should contain +a JSON dictionary with an evolved version of the schema of [existing `destination.json` files that SwiftPM already +supports](https://github.com/apple/swift-package-manager/pull/1098) and `destination.json` files presented in the pitch +version of this proposal, hence `"schemaVersion": "4.0"`. We'll keep parsing `"version": 1`, `"version": 2`, +and `"version": "3.0"` for backward compatibility, but for consistency with `info.json` this field is renamed to +`"schemaVersion"`. Here's an informally defined schema for these files: + +```json5 +{ + "schemaVersion": "4.0", + "targetTriples": { + "": { + "sdkRootPath": "", + // all of the properties listed below are optional: + "swiftResourcesPath": "", + "swiftStaticResourcesPath": "", + "includeSearchPaths": [""], + "librarySearchPaths": [""], + "toolsetPaths": [""] + }, + // a Swift SDK can support more than one target triple: + "": { + "sdkRootPath": "", + // all of the properties listed below are optional: + "swiftResourcesPath": "", + "swiftStaticResourcesPath": "", + "includeSearchPaths": [""], + "librarySearchPaths": [""], + "toolsetPaths": [""] + } + // more triples can be supported by a single Swift SDK if needed, primarily for sharing files between them. + } +} +``` + +We propose that all relative paths in `swift-sdk.json` files should be validated not to "escape" the Swift SDK +bundle for security reasons, in the same way that `toolset.json` files are validated when contained in Swift SDK +bundles. That is, `../` components, if present in paths, will not be allowed to reference files and +directories outside of a corresponding Swift SDK bundle. Symlinks will also be validated to prevent them from escaping +out of the bundle. + +If `sdkRootPath` is specified and `swiftResourcesPath` is not, the latter is inferred to be +`"\(sdkRootPath)/usr/lib/swift"` when linking the Swift standard library dynamically, `"swiftStaticResourcesPath"` is +inferred to be `"\(sdkRootPath)/usr/lib/swift_static"` when linking it statically. Similarly, `includeSearchPaths` is +inferred as `["\(sdkRootPath)/usr/include"]`, `librarySearchPaths` as `["\(sdkRootPath)/usr/lib"]`. + +Here's `swift-sdk.json` file for the `ubuntu_jammy` artifact previously introduced as an example: + +```json5 +{ + "schemaVersion": "4.0", + "targetTriples": { + "aarch64-unknown-linux-gnu": { + "sdkRootPath": "aarch64-unknown-linux-gnu/ubuntu-jammy.sdk", + "toolsetPaths": ["aarch64-unknown-linux-gnu/toolset.json"] + }, + "x86_64-unknown-linux-gnu": { + "sdkRootPath": "x86_64-unknown-linux-gnu/ubuntu-jammy.sdk", + "toolsetPaths": ["x86_64-unknown-linux-gnu/toolset.json"] + } + } +} +``` + +Since not all platforms can support self-contained Swift SDK bundles, users will be able to provide their own +additional paths on the filesystem outside of bundles after a Swift SDK is installed. The exact options for specifying +paths are proposed in a subsequent section for a newly introduced `swift sdk configure` command. + +### Swift SDK Installation and Configuration + +To manage Swift SDKs, we'd like to introduce a new `swift sdk` command with three subcommands: + +- `swift sdk install `, which downloads a given bundle if needed and + installs it in a location discoverable by SwiftPM. For Swift SDKs installed from remote URLs an additional + `--checksum` option is required, through which users of a Swift SDK can specify a checksum provided by a publisher of + the SDK. The latter can produce a checksum by running `swift package compute-checksum` command (introduced in + [SE-0272](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md)) with the + Swift SDK bundle archive as an argument. + + If a Swift SDK with a given artifact ID has already been installed and its version is equal or higher to a version + of a new Swift SDK, an error message will be printed. If the new version is higher, users should invoke the + `install` subcommand with `--update` flag to allow updating an already installed Swift SDK artifact to a new + version. +- `swift sdk list`, which prints a list of already installed Swift SDKs with their identifiers. +- `swift sdk configure `, which allows users to provide additional search paths and toolsets to be +used subsequently when building with a given Swift SDK. Specifically, multiple `--swift-resources-path`, +`--include-search-path`, `--library-search-path`, and `--toolset` options with corresponding paths can be provided, +which then will be stored as configuration for this Swift SDK. +`swift sdk configure --show-configuration` will print currently set paths, while +`swift sdk configure --reset` will reset all of those at once. +- `swift sdk remove ` will remove a given Swift SDK from the filesystem. + +### Using a Swift SDK + +After a Swift SDK is installed, users can refer to it via its identifier passed to the `--swift-sdk` option, e.g. + +``` +swift build --swift-sdk ubuntu_focal +``` + +We'd also like to make `--swift-sdk` option flexible enough to recognize target triples when there's only a single +Swift SDK installed for such triple: + +``` +swift build --swift-sdk x86_64-unknown-linux-gnu +``` + +When multiple Swift SDKs support the same triple, an error message will be printed listing these Swift SDKs and +asking the user to select a single one via its identifier instead. + +### Swift SDK Bundle Generation + +Swift SDKs can be generated quite differently, depending on host and target triple combinations and user's +needs. We intentionally don't specify in this proposal how exactly Swift SDK bundles should be generated. + +Authors of this document intend to publish source code for a macOS → Linux Swift SDK generator, which community is +welcome to fork and reuse for their specific needs. As a configurable option, this generator will use Docker for setting +up the build environment locally before copying it to a Swift SDK tree. Relying on Docker in this generator makes it +easier to reuse and customize existing build environments. Important to clarify, that Docker is only used for bundle +generation, and users of Swift SDK bundles do not need to have Docker installed on their machine to use these bundles. + +As an example, Swift SDK publishers looking to add a library to an Ubuntu 22.04 target environment would modify a +`Dockerfile` similar to this one in their Swift SDK generator source code: + +```dockerfile +FROM swift:5.9-jammy + +apt-get install -y \ + # PostgreSQL library provided as an example. + libpq-dev + # Add more libraries as arguments to `apt-get install`. +``` + +Then to generate a new Swift SDK, a generator executable delegates to Docker for downloading and installing +required tools and libraries, including the newly added ones. After a Docker image with Swift SDK environment is +ready, the generator copies files from the image to a corresponding `.artifactbundle` Swift SDK tree. + +## Security + +The proposed `--checksum` flag provides basic means of verifying Swift SDK bundle's validity. As a future direction, +we'd like to consider sandboxed and codesigned toolchains included in Swift SDKs running on macOS. + +## Impact on Existing Packages + +This is an additive change with no impact on existing packages. + +## Prior Art + +### Rust + +In the Rust ecosystem, its toolchain and standard library built for a target triple are managed by [the `rustup` +tool](https://github.com/rust-lang/rustup). For example, artifacts required for cross-compilation to +`aarch64-linux-unknown-gnu` are installed with +[`rustup target add aarch64-linux-unknown-gnu`](https://rust-lang.github.io/rustup/cross-compilation.html). Then +building for this target with Rust’s package manager looks like `cargo build --target=aarch64-linux-unknown-gnu` . + +Mainstream Rust tools don’t provide an easy way to create your own targets. You’re only limited to the list +of targets provided by Rust maintainers. This likely isn’t a big problem per se for Rust users, as Rust doesn’t provide +C/C++ interop on the same level as Swift. It means that Rust packages much more rarely than Swift expect certain +system-provided packages to be available in the same way that SwiftPM allows with `systemLibrary`. + +Currently, Rust doesn’t supply all of the required tools when running `rustup target add`. It’s left to a user to +specify paths to a linker that’s suitable for their host/target triple combination manually in a config file. We +feel that this should be unnecessary, which is why Swift SDK bundles proposed for Swift can provide their own tools +via toolset configuration files. + +### Go + +Go’s standard library is famously self-contained and has no dependencies on C or C++ standard libraries. Because of this +there’s no need to install artifacts. Cross-compiling in Go works out of the box by passing `GOARCH` and `GOOS` +environment variables with chosen values, an example of this is `GOARCH=arm64 GOOS=linux go build` invocation. + +This would be a great experience for Swift, but it isn’t easily achievable as long as Swift standard library depends on +C and C++ standard libraries. Any code interoperating with C and/or C++ would have to link with those libraries as well. +When compared to Go, our proposed solution allows both dynamic and, at least on Linux when Musl is supported, full +static linking. We’d like Swift to allow as much customization as needed for users to prepare their own Swift SDK +bundles. + +## Alternatives Considered + +### Extensions Other Than `.artifactbundle` + +Some members of the community suggested that Swift SDK bundles should use a more specific filepath extension. Since +we're relying on the existing `.artifactbundle` format and extension, which is already used for binary targets, we think +a specialized extension only for Swift SDKs would introduce an inconsistency. On the other hand, we think that +specific extensions could make sense with a change applied at once. For example, we could consider `.binarytarget` and +`.swiftsdk` extensions for respective artifact types. But that would require a migration strategy for existing +`.artifactbundle`s containing binary targets. + +### Building Applications in Docker Containers + +Instead of coming up with a specialized bundle format for Swift SDKs, users of Swift on macOS building for Linux could +continue to use Docker. But, as discussed in the [Motivation](#motivation) section, building applications in Docker +doesn’t cover all of the possible use cases and complicates onboarding for new users. It also only supports Linux, while +we’re looking for a solution that can be generalized for all possible platforms. + +### Alternative Bundle Formats + +One alternative is to allow only a single host/target combination per bundle, but this may complicate +distribution of Swift SDK bundles in some scenarios. The existing `.artifactbundle` format is flexible enough to +support bundles with a single or multiple combinations. + +Different formats of Swift SDK bundles can be considered, but we don't think those would be significantly different +from the proposed one. If they were different, this would complicate bundle distribution scenarios for users who want to +publish their own artifact bundles with executables, as defined in SE-0305. + +### Triples nomenclature + +Authors of the proposal considered alternative nomenclature to the established "build/host/target platform" naming convention, +but felt that preserving consistency with other ecosystems is more important. + +While "target" already has a different meaning within the build systems nomenclature, users are most likely to stumble upon +targets when working with SwiftPM package manifests. To avoid this ambiguity, as a future direction SwiftPM can consider renaming +`target` declarations used in `Package.swift` to a different unambiguous term. + +### Making Swift SDK Bundles Fully Self-Contained + +Some users expressed interest in self-contained Swift SDK bundles that ignore the value of `PATH` environment variable +and prevent launching any executables from outside of a bundle. So far in our practice we haven't seen any problems +caused by the use of executables from `PATH`. Quite the opposite, we think most Swift SDKs would want to reuse as many +tools from `PATH` as possible, which would allow making Swift SDK bundles much smaller. For example as of Swift 5.7, +on macOS `clang-13` binary takes ~360 MB, `clangd` ~150 MB, and `swift-frontend` ~420 MB. Keeping copies of these +binaries in every Swift SDK bundle seems quite redundant when existing binaries from `PATH` can be easily reused. +Additionally, we find that preventing tools from being launched from arbitrary paths can't be technically enforced +without sandboxing, and there's no cross-platform sandboxing solution available for SwiftPM. Until such sandboxing +solution is available, we'd like to keep the existing approach where setting `PATH` environment variable behaves in a +predictable way and is consistent with established CLI conventions. + +## Future Directions + +### Identifying Platforms with Dictionaries of Properties + +Platform triples are not specific enough in certain cases. For example, `aarch64-unknown-linux` host triple can’t +prevent a user from installing a Swift SDK bundle on an unsupported Linux distribution. In the future we could +deprecate `supportedTriples` and `targetTriples` JSON properties in favor of dictionaries with keys and values that +describe aspects of platforms that are important for Swift SDKs. Such dictionaries could look like this: + +```json5 +"platform": { + "kernel": "Linux", + "libcFlavor": "Glibc", + "libcMinVersion": "2.36", + "cpuArchitecture": "aarch64" + // more platform capabilities defined here... +} +``` + +A toolchain providing this information could allow users to refer to these properties in their code for conditional +compilation and potentially even runtime checks. + +### SwiftPM Plugins for Remote Running, Testing, Deployment, and Debugging + +After an application is built with a Swift SDK, there are other development workflow steps to be improved. We could +introduce new types of plugins invoked by `swift run` and `swift test` for purposes of remote running, debugging, and +testing. For Linux target triples, these plugins could delegate to Docker for running produced executables. + +### `swift sdk select` Subcommand + +While `swift sdk select` subcommand or a similar one make sense for selecting a Swift SDK instead of +passing `--swift-sdk` to `swift build` every time, users will expect `swift run` and `swift test` to also work for any +Swift SDK previously passed to `swift sdk select`. That’s out of scope for this proposal on its own and +depends on making plugins (from the previous subsection) or some other remote running and testing implementation to +fully work. + +### SwiftPM and SourceKit-LSP Improvements + +It is a known issue that SwiftPM can’t run multiple concurrent builds for different target triples. This may cause +issues when SourceKit-LSP is building a project for indexing purposes (for a host platform by default), while a user may +be trying to build for a target for testing, for example. One of these build processes will fail due to the +process locking the build database. A potential solution would be to maintain separate build databases per platform. + +Another issue related to SourceKit-LSP is that [it always build and indexes source code for the host +platform](https://github.com/apple/sourcekit-lsp/issues/601). Ideally, we want it to maintain indices for multiple +platforms at the same time. Users should be able to select target triples and corresponding indices to enable +semantic syntax highlighting, auto-complete, and other features for areas of code that are conditionally compiled with +`#if` directives. + +### Source-Based Swift SDKs + +One interesting solution is distributing source code of a minimal base SDK, as explored by [Zig programming +language](https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html). In this scenario, Swift SDK +binaries are produced on the fly when needed. We don't consider this option to be mutually exclusive with solutions +proposed in this document, and so it could be explored in the future for Swift as well. However, this requires reducing +the number of dependencies that Swift runtime and core libraries have. + +### Swift SDK Bundles and Package Registries + +Since `info.json` manifest files contained within bundles contain versions, it would make sense to host Swift SDK +bundles at package registries. Although, it remains to be seen whether it makes sense for an arbitrary SwiftPM package +to specify a Swift SDK bundle within its list of dependencies. diff --git a/proposals/0388-async-stream-factory.md b/proposals/0388-async-stream-factory.md new file mode 100644 index 0000000000..cda799d450 --- /dev/null +++ b/proposals/0388-async-stream-factory.md @@ -0,0 +1,165 @@ +# Convenience Async[Throwing]Stream.makeStream methods + +* Proposal: [SE-0388](0388-async-stream-factory.md) +* Authors: [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#62968](https://github.com/apple/swift/pull/62968) +* Review: ([pitch](https://forums.swift.org/t/pitch-convenience-async-throwing-stream-makestream-methods/61030)) ([review](https://forums.swift.org/t/se-0388-convenience-async-throwing-stream-makestream-methods/63139)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0388-convenience-async-throwing-stream-makestream-methods/63568)) + +## Introduction + +We propose introducing helper methods for creating `AsyncStream` and `AsyncThrowingStream` +instances which make the stream's continuation easier to access. + +## Motivation + +With [SE-0314](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0314-async-stream.md) +we introduced `AsyncStream` and `AsyncThrowingStream` which act as a root +`AsyncSequence` that the standard library offers. + +After having used `Async[Throwing]Stream` for some time, a common usage +is to pass the continuation and the `Async[Throwing]Stream` to different places. +This requires escaping the `Async[Throwing]Stream.Continuation` out of +the closure that is passed to the initialiser. +Escaping the continuation is slightly inconvenient since it requires a dance +around an implicitly unwrapped optional. Furthermore, the closure implies +that the continuation lifetime is scoped to the closure which it isn't. This is how +an example usage of the current `AsyncStream` API looks like. + +```swift +var cont: AsyncStream.Continuation! +let stream = AsyncStream { cont = $0 } +// We have to assign the continuation to a let to avoid sendability warnings +let continuation = cont + +await withTaskGroup(of: Void.self) { group in + group.addTask { + for i in 0...9 { + continuation.yield(i) + } + continuation.finish() + } + + group.addTask { + for await i in stream { + print(i) + } + } +} +``` + +## Proposed solution + +In order to fill this gap, I propose to add a new static method `makeStream` on +`AsyncStream` and `AsyncThrowingStream` that returns both the stream +and the continuation. An example of using the new proposed convenience methods looks like this: + +```swift +let (stream, continuation) = AsyncStream.makeStream(of: Int.self) + +await withTaskGroup(of: Void.self) { group in + group.addTask { + for i in 0...9 { + continuation.yield(i) + } + continuation.finish() + } + + group.addTask { + for await i in stream { + print(i) + } + } +} +``` + +## Detailed design + +I propose to add the following code to `AsyncStream` and `AsyncThrowingStream` +respectively. These methods are also marked as backdeployed to previous Swift versions. + +```swift +@available(SwiftStdlib 5.1, *) +extension AsyncStream { + /// Initializes a new ``AsyncStream`` and an ``AsyncStream/Continuation``. + /// + /// - Parameters: + /// - elementType: The element type of the stream. + /// - limit: The buffering policy that the stream should use. + /// - Returns: A tuple containing the stream and its continuation. The continuation should be passed to the + /// producer while the stream should be passed to the consumer. + @backDeployed(before: SwiftStdlib 5.9) + public static func makeStream( + of elementType: Element.Type = Element.self, + bufferingPolicy limit: Continuation.BufferingPolicy = .unbounded + ) -> (stream: AsyncStream, continuation: AsyncStream.Continuation) { + var continuation: AsyncStream.Continuation! + let stream = AsyncStream(bufferingPolicy: limit) { continuation = $0 } + return (stream: stream, continuation: continuation!) + } +} + +@available(SwiftStdlib 5.1, *) +extension AsyncThrowingStream { + /// Initializes a new ``AsyncThrowingStream`` and an ``AsyncThrowingStream/Continuation``. + /// + /// - Parameters: + /// - elementType: The element type of the stream. + /// - failureType: The failure type of the stream. + /// - limit: The buffering policy that the stream should use. + /// - Returns: A tuple containing the stream and its continuation. The continuation should be passed to the + /// producer while the stream should be passed to the consumer. + @backDeployed(before: SwiftStdlib 5.9) + public static func makeStream( + of elementType: Element.Type = Element.self, + throwing failureType: Failure.Type = Failure.self, + bufferingPolicy limit: Continuation.BufferingPolicy = .unbounded + ) -> (stream: AsyncThrowingStream, continuation: AsyncThrowingStream.Continuation) where Failure == Error { + var continuation: AsyncThrowingStream.Continuation! + let stream = AsyncThrowingStream(bufferingPolicy: limit) { continuation = $0 } + return (stream: stream, continuation: continuation!) + } +} +``` + +## Source compatibility +This change is additive and does not affect source compatibility. + +## Effect on ABI stability +This change introduces new concurrency library ABI in the form of the `makeStream` methods, but it does not affect the ABI of existing declarations. + +## Effect on API resilience +None; adding static methods is permitted by the existing resilience model. + +## Alternatives considered + +### Return a concrete type instead of a tuple +My initial pitch was using a tuple as the result type of the factory; +however, I walked back on it before the review since I think we can provide better documentation on +the concrete type. However during the review the majority of the feedback was leaning towards the tuple based approach. +After comparing the two approaches, I agree with the review feedback. The tuple based approach has two major benefits: + +1. It nudges the user to destructure the returned typed which we want since the continuation and stream should be retained by the +producer and consumer respectively. +2. It allows us to back deploy the method. + +### Pass a continuation to the `AsyncStream.init()` +During the pitch it was brought up that we could let users pass a continuation to the +`AsyncStream.init()`; however, this opens up a few problems: +1. A continuation could be passed to multiple streams +2. A continuation which is not passed to a stream is useless + +In the end, the `AsyncStream.Continuation` is deeply coupled to one instance of an +`AsyncStream` hence we should create an API that conveys this coupling and prevents +users from misuse. + +### Do nothing alternative +We could just leave the current creation of `Async[Throwing]Stream` as is; +however, since it is part of the standard library we should provide +a better method to create a stream and its continuation. + +## Revision history + +- After review: Changed the return type from a concrete type to a tuple + diff --git a/proposals/0389-attached-macros.md b/proposals/0389-attached-macros.md new file mode 100644 index 0000000000..fe01f3495f --- /dev/null +++ b/proposals/0389-attached-macros.md @@ -0,0 +1,795 @@ +# Attached Macros + +* Proposal: [SE-0389](0389-attached-macros.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Holly Borla](https://github.com/hborla), [Richard Wei](https://github.com/rxwei) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 5.9)** +* Implementation: Implemented on GitHub `main` behind the experimental flag `Macros`. See the [example repository](https://github.com/DougGregor/swift-macro-examples) for more macros. +* Review: ([pitch #1, under the name "declaration macros"](https://forums.swift.org/t/pitch-declaration-macros/62373)) ([pitch #2](https://forums.swift.org/t/pitch-attached-macros/62812)) ([review](https://forums.swift.org/t/se-0389-attached-macros/63165)) ([acceptance](https://forums.swift.org/t/accepted-se-0389-attached-macros/63593)) ([post-acceptance update](https://forums.swift.org/t/update-on-se-0382-and-se-0389-expression-macros-and-attached-macros/74094)) + +## Introduction + +Attached macros provide a way to extend Swift by creating and extending declarations based on arbitrary syntactic transformations on their arguments. They make it possible to extend Swift in ways that were only previously possible by introducing new language features, helping developers build more expressive libraries and eliminate extraneous boilerplate. + +Attached macros are one part of the [vision for macros in Swift](https://github.com/swiftlang/swift-evolution/pull/1927), which lays out general motivation for introducing macros into the language. They build on the ideas and motivation of [SE-0382 "Expression macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) to cover a large new set of use cases; we will refer to that proposal for the basic model of how macros integrate into the language. While expression macros are designed as standalone entities introduced by `#`, attached macros are associated with a specific declaration in the program that they can augment and extend. This supports many new use cases, greatly expanding the expressiveness of the macro system: + +* Creating trampoline or wrapper functions, such as automatically creating a completion-handler version of an `async` function or vice-versa. +* Creating members of a type based on its definition, such as forming an [`OptionSet`](https://developer.apple.com/documentation/swift/optionset) from an enum containing flags and conforming it to the `OptionSet` protocol or adding a memberwise initializer. +* Creating accessors for a stored property or subscript, subsuming some of the behavior of [SE-0258 "Property Wrappers"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md). +* Augmenting members of a type with a new attribute, such as applying a property wrapper to all stored properties of a type. + +There is an [example repository](https://github.com/DougGregor/swift-macro-examples) containing a number of macros that have been implemented using the prototype of this feature. + +## Proposed solution + +The proposal adds *attached macros*, so-called because they are attached to a particular declaration. They are written using the custom attribute syntax (e.g., `@AddCompletionHandler`) that already provides extensibility for declarations through property wrappers, result builders, and global actors. Attached macros can reason about the declaration to which they are attached, and provide additions and changes based on one or more different macro *roles*. Each role has a specific purpose, such as adding members, creating accessors, or adding peers alongside the declaration. A given attached macro can inhabit several different roles, and as such will be expanded multiple times corresponding to the different roles, which allows the various roles to be composed. For example, an attached macro emulating property wrappers might inhabit both the "peer" and "accessor" roles, allowing it to introduce a backing storage property and also synthesize a getter/setter that go through that backing storage property. Composition of macro roles will be discussed in more depth once the basic macro roles have been established. + +As with expression macros, attached declaration macros are declared with `macro`, and have [type-checked macro arguments](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#type-checked-macro-arguments-and-results) that allow their behavior to be customized. Attached macros are identified with the `@attached` attribute, which also provides the specific role as well as any names they introduce. For example, the aforemented macro to add a completion handler would be declared as follows: + +```swift +@attached(peer, names: overloaded) +macro AddCompletionHandler(parameterName: String = "completionHandler") +``` + +The macro can be used as follows: + +```swift +@AddCompletionHandler(parameterName: "onCompletion") +func fetchAvatar(_ username: String) async -> Image? { ... } +``` + +The use of the macro is attached to `fetchAvatar`, and generates a *peer* declaration alongside `fetchAvatar` whose name is "overloaded" with `fetchAvatar`. The generated declaration is: + +```swift +/// Expansion of the macro produces the following. +func fetchAvatar(_ username: String, onCompletion: @escaping (Image?) -> Void) { + Task.detached { + onCompletion(await fetchAvatar(username)) + } +} +``` + +### Implementing attached macros + +All attached macros are implemented as types that conform to one of the protocols that inherits from the `AttachedMacro` protocol. Like the [`Macro` protocol](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-protocols), the `AttachedMacro` protocol has no requirements, but is used to organize macro implementations. Each attached macro role will have its own protocol that inherits `AttachedMacro`. + +```swift +public protocol AttachedMacro: Macro { } +``` + +The biggest difference from expression macros is that there is more than one relevant piece of syntax for the macro to consider: each attached macro implementation receives the syntax node for both the attribute (e.g., `@AddCompletionHandler(parameterName: "onCompletion")`) and the declaration to which the macro is attached (`func fetchAvatar`...), and can return new code that's appropriate to the macro role: peer macros return new declarations, accessor macros return getters/getters, and so on. For example, `PeerMacro` is defined as follows: + +```swift +public PeerMacro: AttachedMacro { + /// Expand a macro described by the given attribute to + /// produce "peer" declarations of the declaration to which it + /// is attached. + /// + /// The macro expansion can introduce "peer" declarations that + /// go alongside the given declaration. + static func expansion( + of node: AttributeSyntax, + providingPeersOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] +} +``` + +### Naming macro-produced declarations + +Unlike expression macros, attached macros can introduce new declarations. These declarations can have an impact on code elsewhere in the program, for example if a macro provides a declaration of a function named `hello` and that function is called from another source file. Our design requires macros to document which names they can introduce: this provides more information up-front to developers and tools alike to understand the impact that a macro can have on the surrounding program. For developers, this can mean fewer surprises; for tools, this can be used to improve compilation times by avoiding unnecessary macro expansions. + +The `@AddCompletionHandler` macro notes that it introduces an *overloaded* name, meaning that it produces a declaration with the same base name as the declaration to which it is attached. A macro that emulated a property wrapper would specify the storage name via `prefixed(_)`, meaning that `_` will be added as a prefix to the name of the declaration to which that macro is attached. Other ways in which macro-generated names are communicated are discussed in the Detailed Design. + +### Kinds of attached macros + +#### Peer macros + +Peer macros produce new declarations alongside the declaration to which they are attached. The `AddCompletionHandler` macro from earlier was a peer macro. Peer macros are implemented via types that conform to the `PeerMacro` protocol shown earlier. The implementation of `AddCompletionHandlerMacro` looks like the following: + +```swift +public struct AddCompletionHandlerMacro: PeerDeclarationMacro { + public static func expansion( + of node: CustomAttributeSyntax, + providingPeersOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] { + // make sure we have an async function to start with + // form a new function "completionHandlerFunc" by starting with that async function and + // - remove async + // - remove result type + // - add a completion-handler parameter + // - add a body that forwards arguments + // return the new peer function + return [completionHandlerFunc] + } +} +``` + +The details of the implementation are left to an Appendix, with a complete version in the [example repository](https://github.com/DougGregor/swift-macro-examples). + +#### Member macros + +Member macros allow one to introduce new members into the type or extension to which the macro is attached. For example, we can write a macro that defines static members to ease the definition of an [`OptionSet`](https://developer.apple.com/documentation/swift/optionset). Given: + +```swift +@OptionSetMembers +struct MyOptions: OptionSet { + enum Option: Int { + case a + case b + case c + } +} +``` + +This struct should be expanded to contain both a `rawValue` field and static properties for each of the options, e.g., + +```swift +// Expands to... +struct MyOptions: OptionSet { + enum Option: Int { + case a + case b + case c + } + + // Synthesized code below... + var rawValue: Int = 0 + + static var a = MyOptions(rawValue: 1 << Option.a.rawValue) + static var b = MyOptions(rawValue: 1 << Option.b.rawValue) + static var c = MyOptions(rawValue: 1 << Option.c.rawValue) +} +``` + +The macro itself will be declared as a member macro that defines an arbitrary set of members: + +```swift +/// Create the necessary members to turn a struct into an option set. +@attached(member, names: named(rawValue), arbitrary) macro OptionSetMembers() +``` + +The `member` role specifies that this macro will be defining new members of the declaration to which it is attached. In this case, while the macro knows it will define a member named `rawValue`, there is no way for the macro to predict the names of the static properties it is defining, so it also specifies `arbitrary` to indicate that it will introduce members with arbitrarily-determined names. + +Member macros are implemented with types that conform to the `MemberMacro` protocol: + +```swift +protocol MemberMacro: AttachedMacro { + /// Expand a macro described by the given attribute to + /// produce additional members of the given declaration to which + /// the attribute is attached. + static func expansion( + of node: AttributeSyntax, + providingMembersOf declaration: some DeclGroupSyntax, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] +} +``` + +#### Accessor macros + +Accessor macros allow a macro to add accessors to a property, turning a stored property into a computed property. For example, consider a macro that can be applied to a stored property to instead access a dictionary keyed by the property name. Such a macro could be used like this: + +```swift +struct MyStruct { + var storage: [AnyHashable: Any] = [:] + + @DictionaryStorage + var name: String + + @DictionaryStorage(key: "birth_date") + var birthDate: Date? +} +``` + +The `DictionaryStorage` attached macro would alter the properties of `MyStruct` as follows, adding accessors to each: + +```swift +struct MyStruct { + var storage: [String: Any] = [:] + + var name: String { + get { + storage["name"]! as! String + } + + set { + storage["name"] = newValue + } + } + + var birthDate: Date? { + get { + storage["birth_date"] as Date? + } + + set { + if let newValue { + storage["birth_date"] = newValue + } else { + storage.removeValue(forKey: "birth_date") + } + } + } +} +``` + +The macro can be declared as follows: + +```swift +@attached(accessor) macro DictionaryStorage(key: String? = nil) +``` + +Implementations of accessor macros conform to the `AccessorMacro` protocol, which is defined as follows: + +```swift +protocol AccessorMacro: AttachedMacro { + /// Expand a macro described by the given attribute to + /// produce accessors for the given declaration to which + /// the attribute is attached. + static func expansion( + of node: AttributeSyntax, + providingAccessorsOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [AccessorDeclSyntax] +} +``` + +The implementation of the `DictionaryStorage` macro would create the accessor declarations shown above, using either the `key` argument (if present) or deriving the key name from the property name. The effect of this macro isn't something that can be done with a property wrapper, because the property wrapper wouldn't have access to `self.storage`. + +An accessor macro can specify that it produces observers by listing at least one of `willSet` or `didSet ` in the names, e.g., `@attached(accessors, names: named(willSet))`. Such a macro can only produce observers; it cannot change a stored property into a computed property. + +The expansion of an accessor macro that does not specify one of `willSet` or `didSet` in its list of names must result in a computed property. A side effect of the expansion is to remove any initializer from the stored property itself; it is up to the implementation of the accessor macro to either diagnose the presence of the initializer (if it cannot be used) or incorporate it in the result. + +#### Member attribute macros + +Member declaration macros allow one to introduce new member declarations within the type or extension to which they apply. In contrast, member *attribute* macros allow one to modify the member declarations that were explicitly written within the type or extension by adding new attributes to them. Those new attributes could then refer to other macros, such as peer or accessor macros, or builtin attributes. As such, they are primarily a means of *composition*, since they have fairly little effect on their own. + +Member attribute macros allow a macro to provide similar behavior to how many built-in attributes work, where declaring the attribute on a type or extension will apply that attribute to each of the members. For example, a global actor `@MainActor` written on an extension applies to each of the members of that extension (unless they specify another global actor or `nonisolated`), an access specifier on an extension applies to each of the members of that extension, and so on. + +As an example, we'll define a macro that partially mimics the behavior of the `@objcMembers` attributes introduced long ago in [SE-0160](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0160-objc-inference.md#re-enabling-objc-inference-within-a-class-hierarchy). Our `myObjCMembers` macro is a member-attribute macro: + +```swift +@attached(memberAttribute) +macro MyObjCMembers() +``` + +The implementation of this macro will add the `@objc` attribute to every member of the type or extension, unless that member either already has an `@objc` macro or has `@nonobjc` on it. For example, this: + +```swift +@MyObjCMembers extension MyClass { + func f() { } + + var answer: Int { 42 } + + @objc(doG) func g() { } + + @nonobjc func h() { } +} +``` + +would result in: + +```swift +extension MyClass { + @objc func f() { } + + @objc var answer: Int { 42 } + + @objc(doG) func g() { } + + @nonobjc func h() { } +} +``` + +Member-attribute macro implementations should conform to the `MemberAttributeMacro` protocol: + +```swift +protocol MemberAttributeMacro: AttachedMacro { + /// Expand a macro described by the given custom attribute to + /// produce additional attributes for the members of the type. + static func expansion( + of node: AttributeSyntax, + attachedTo declaration: some DeclGroupSyntax, + providingAttributesOf member: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [AttributeSyntax] +} +``` + +The `expansion` operation accepts the attribute syntax `node` for the spelling of the member attribute macro and the declaration to which that attribute is attached (i.e., the type or extension). The implementation of the macro can walk the members of the `declaration` to determine which members require additional attributes. The returned dictionary will map from those members to the additional attributes that should be added to each of the members. + +#### Conformance macros + +Conformance macros allow one to introduce new protocol conformances to a type. This would often be paired with other macros whose purpose is to help satisfy the protocol conformance. For example, one could imagine an extended version of the `OptionSetMembers` attributed shown earlier that also adds the `OptionSet` conformance. With it, the minimal implementation of an option set could be: + +```swift +@OptionSet +struct MyOptions { + enum Option: Int { + case a + case b + case c + } +} +``` + +where `OptionSet` is both a member and a conformance macro, providing members (as in `OptionSetMembers`) and the conformance to `OptionSet`. The macro would be declared as, e.g., + +```swift +/// Create the necessary members to turn a struct into an option set. +@attached(member, names: named(rawValue), arbitrary) +@attached(conformance) +macro OptionSet() +``` + +Conformance macro implementations should conform to the `ConformanceMacro` protocol: + +```swift +/// Describes a macro that can add conformances to the declaration it's +/// attached to. +public protocol ConformanceMacro: AttachedMacro { + /// Expand an attached conformance macro to produce a set of conformances. + /// + /// - Parameters: + /// - node: The custom attribute describing the attached macro. + /// - declaration: The declaration the macro attribute is attached to. + /// - context: The context in which to perform the macro expansion. + /// + /// - Returns: the set of `(type, where-clause?)` pairs that each provide the + /// protocol type to which the declared type conforms, along with + /// an optional where clause. + static func expansion( + of node: AttributeSyntax, + providingConformancesOf declaration: some DeclGroupSyntax, + in context: some MacroExpansionContext + ) throws -> [(TypeSyntax, WhereClauseSyntax?)] +} +``` + +The returned array contains the conformances. The `TypeSyntax` describes the protocol to which the enclosing type conforms, and the optional `where` clause provides any additional constraints that would make the conformance conditional. + +## Detailed design + +### Composing macro roles + +A given macro can have several different roles, allowing the various macro features to be composed. Each of the roles is considered independently, so a single use of a macro in source code can result in different macro expansion functions being called. These calls are independent, and could even happen concurrently. As an example, let's define a macro that emulates property wrappers fairly closely. The property wrappers proposal has an example for a [clamping property wrapper](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md#clamping-a-value-within-bounds): + +```swift +@propertyWrapper +struct Clamping { + var value: V + let min: V + let max: V + + init(wrappedValue: V, min: V, max: V) { + value = wrappedValue + self.min = min + self.max = max + assert(value >= min && value <= max) + } + + var wrappedValue: V { + get { return value } + set { + if newValue < min { + value = min + } else if newValue > max { + value = max + } else { + value = newValue + } + } + } +} + +struct Color { + @Clamping(min: 0, max: 255) var red: Int = 127 + @Clamping(min: 0, max: 255) var green: Int = 127 + @Clamping(min: 0, max: 255) var blue: Int = 127 + @Clamping(min: 0, max: 255) var alpha: Int = 255 +} +``` + +Instead, let's implement this as a macro: + +```swift +@attached(peer, prefixed(_)) +@attached(accessor) +macro Clamping(min: T, max: T) = #externalMacro(module: "MyMacros", type: "ClampingMacro") +``` + +The usage syntax is the same in both cases. As a macro, `Clamping` both defines a peer (a backing storage property with an `_` prefix) and also defines accessors (to check min/max). The peer declaration macro is responsible for defining the backing storage, e.g., + +```swift +private var _red: Int = { + let newValue = 127 + let minValue = 0 + let maxValue = 255 + if newValue < minValue { + return minValue + } + if newValue > maxValue { + return maxValue + } + return newValue +}() +``` + +Which is implemented by having `ClampingMacro` conform to `PeerMacro`: + +```swift +enum ClampingMacro { } + +extension ClampingMacro: PeerDeclarationMacro { + static func expansion( + of node: CustomAttributeSyntax, + providingPeersOf declaration: DeclSyntax, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] { + // create a new variable declaration that is the same as the original, but... + // - prepend an underscore to the name + // - make it private + } +} +``` + +And introduces accessors such as + +```swift + get { return _red } + + set(__newValue) { + let __minValue = 0 + let __maxValue = 255 + if __newValue < __minValue { + _red = __minValue + } else if __newValue > __maxValue { + _red = __maxValue + } else { + _red = __newValue + } + } +``` + +by conforming to `AccessorMacro`: + +```swift +extension ClampingMacro: AccessorMacro { + static func expansion( + of node: CustomAttributeSyntax, + providingAccessorsOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [AccessorDeclSyntax] { + let originalName = /* get from declaration */, + minValue = /* get from custom attribute node */, + maxValue = /* get from custom attribute node */ + let storageName = "_\(originalName)", + newValueName = context.getUniqueName(), + maxValueName = context.getUniqueName(), + minValueName = context.getUniqueName() + return [ + """ + get { \(storageName) } + """, + """ + set(\(newValueName)) { + let \(minValueName) = \(minValue) + let \(maxValueName) = \(maxValue) + if \(newValueName) < \(minValueName) { + \(storageName) = \(minValueName) + } else if \(newValueName) > maxValue { + \(storageName) = \(maxValueName) + } else { + \(storageName) = \(newValueName) + } + } + """ + ] + } +} +``` + +This formulation of `@Clamping` offers some benefits over the property-wrapper version: we don't need to store the min and max values as part of the backing storage (so the presence of `@Clamping` doesn't add any storage), nor do we need to define a new type. More importantly, it demonstrates how the composition of different macro roles together can produce interesting effects. + +Not every macro role applies to every kind of declaration: an `accessor` macro doesn't make sense of a function, nor does a `member` member make sense on a property. If a macro is attached to a declaration, the macro will only be expanded for those roles that are applicable to a declaration. For example, if the `Clamping` macro is applied to a function, it will only be expanded as a peer macro; it's role as an accessor macro is ignored. If a macro is attached to a declaration and none of its roles apply to that kind of declaration, the program is ill-formed. + +### Specifying newly-introduced names + +Whenever a macro produces declarations that are visible to other Swift code, it is required to declare the names in advance. All of the names need to be specified within the attribute declaring the macro role, using the following forms: + +- Declarations with a specific fixed name: `named()`. +- Declarations whose names cannot be described statically, for example because they are derived from other inputs: `arbitrary`. + +* Declarations that have the same base name as the declaration to which the macro is attached, and are therefore overloaded with it: `overloaded`. +* Declarations whose name is formed by adding a prefix to the name of the declaration to which the macro is attached: `prefixed(_)`. As a special consideration, `$` is permissible as a prefix, allowing macros to produce names with a leading `$` that are derived from the name of the declaration to which the macro is attached. This carve-out enables macros that behavior similarly to property wrappers that introduce a projected value. +* Declarations whose name is formed by adding a suffix to the name of the declaration to which the macro is attached: `suffixed(_docinfo)`. + +A macro can only introduce new declarations whose names are covered by the kinds provided, or have their names generated via `MacroExpansionContext.createUniqueName`. This ensures that, in most cases (where `arbitrary` is not specified), the Swift compiler and related tools can reason about the set of names that will be introduced by a given use of a macro without having to expand the macro, which can reduce the compile-time cost of macros and improve incremental builds. For example, when the compiler performs name lookup for a name `_x`, it can avoid expanding any macros that are unable to produce a declaration named `_x`. Macros that can produce arbitrary names must always be expanded, as would a macro with `prefixed(_)` that is attached to a declaration named `x`. + +The macro is not required to provide a declaration for every name it describes: for example, `OptionSetMembers` will state that it produces a declaration named `rawValue`, but the implementation may opt not to do so if it sees that such a property already exists. + +### Visibility of names used and introduced by macros + +Macros can introduce new names into the program. Whether and when those names are visible to other parts of the program, and in particular other macros, is a subtle design problem that has a number of interesting constraints. + +First, the arguments to a macro are type-checked before it can be determined which macro is being expanded, so they cannot depend on the expansion of that macro. For example, consider an attached macro use spelled `@myMacro(x)`: if it introduced a declaration named `x`, it would potentially invalidate the type-check of its own argument, or cause that macro expansion to choose a different `myMacro` that doesn't produce an `x`! + +Second, if the output of one macro expansion (say, `@myMacro1(x)`) introduces a name (say, `y`) that is then used in the argument to another macro expansion (say, `@myMacro2(y)`), then the order in which the macros are expanded can affect the semantics of the resulting program. It is not generally possible to introduce an ordering on macro expansions, nor would it be desirable, because Swift generally tries not to have order dependencies among declarations in a program. Incidental orderings based on the names introduced by the macro don't help, either, because names of declaration macros can be specified as `arbitrary` and therefore cannot be predicated. + +Third, macro expansion is a relatively expensive compile-time operation, involving serialization overhead to transfer parts of the program from the compiler/IDE to another program that expands the macro. Therefore, macros are [only expanded once per use site](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion), so their expansions cannot participate in the type checking of their arguments or of other surrounding statements or expressions. For example, consider a this (intentionally obtuse) Swift code: + +```swift +@attached(peer) macro myMacro(_: Int) +@attached(peer, names: named(y)) macro myMacro(_: Double) + +func f(_: Int, body: (Int) -> Void) +func f(_: Double, body: (Double) -> Void) + +f(1) { x in + @myMacro(x) func g() { } + print(y) +} +``` + +The macro use `@myMacro(x)` provides different names depending on whether the argument is an `Int` or ` Double`. From the perspective of the call to `f`, both overloads of `f` are possible, and the only way to check beyond that macro use to the `print` expression that follows is to try to expand the macro... for the first `myMacro`, the `print(y)` expression will fail to type-check (there is no `y`) whereas the second would find the `y` generated by the second `myMacro` and will succeed. However, this approach does not scale, because it could involve expanding macros a large number of times during type checking. Moreover, the macro expansions might end up getting incomplete information if, for example, macros are someday provided with type information and that type information isn't known yet (because the expansion is happening during type inference). + +To address these issues, a name introduced by a macro expansion is not visible within macro arguments for macros the same scope as the macro expansion or any of its enclosing scopes. This is conceptually similar to [outside-in expansion of expression macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion), where one can imagine that all of the macros at a particular scope are type-checked and expanded simultaneously at the top-level scope. Then, macros within each type definition and its extensions are type-checked and expanded simultaneously: they can see the names introduced by top-level macro expansions, but not names introduced at other levels. The following annotated code example shows which names are visible: + +```swift +import OtherModule // declares names x, y, z + +@macro1(x) // uses OtherModule.x, introduces name y +func f() { } + +@macro2(y) // uses OtherModule.y, introduce name x +func g() { } + +struct S1 { + @macro3(x) // uses the name x introduced by #@acro2, introduces the name z + func h() +} + +extension S1 { + @macro4(z) // uses OtherModule.z + func g() { } + + func f() { + print(z) // uses z introduced by @macro3 + } +} +``` + +These name lookup rules, while important for providing a reasonable scheme for macros to be expanded without interfering with each other, do introduce a new kind of name shadowing to the language. If `@macro2` were manually expanded into source code, the declaration `x` it produces would then become visible to the macro argument in `@macro1(x)`, changing the behavior of the program. The compiler might be able to detect such shadowing in practice, when a macro-introduced name at a particular scope would change the meaning of a lookup from that particular scope. + +Within function and closure bodies, the fact that names introduced by the macro expansion are not visible within the current scope means that our earlier example will never find a declaration `y` introduced by `@myMacro`: + +```swift +f(1) { x in + @myMacro(x) func g() { } + print(y) // does not consider any names introduced by `@myMacro` +} +``` + +Therefore, a macro used within a closure or function body can only introduce declarations using names produced by `createUniqueName`. This maintains the [two-phase of checking macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion) where type checking and inference is performed without expanding the macro, then the macro is expanded and its result type-checked independently, with no ability to influence type inference further. + +### Restrictions on `arbitrary` names + +Attached macros that specify `arbitrary` names require expanding the macro in order to determine the exact set of names that the macro generates. This means that name lookup must expand all macros that can produce `arbitrary` names in the given scope in order to determine name lookup results. This is a problem for macros that introduce names at global scope, because any unqualified or module qualified lookup must expand those macros. However, because macro attributes can have arguments that are type-checked prior to macro expansion, type checking those arguments may require unqualified or module qualified name lookup that requires expanding that same macro, which is fundamentally circular. Though macro arguments do not have visibility of macro-generated names in the same scope, macro argument type checking can invoke type checking of other declarations that can use macro-generated names. For example: + +```swift +@attached(peer, names: arbitrary) +macro IntroduceArbitraryPeers(_: T) = #externalMacro(...) + +@IntroduceArbitraryPeers(MyType().x) +struct S {} + +struct MyType { + var x: AnotherType +} +``` + +Resolving the macro attribute `@IntroduceArbitraryPeers(MyType().x)` can invoke type-checking of the `var x: AnotherType` property of the `MyType` struct. Unqualified lookup of `AnotherType` must expand `@IntroduceArbitraryPeers` because the macro can introduce arbitrary names; it might introduce a top-level type called `AnotherType`. Resolving the `@IntroduceArbitraryPeers` attribute depends on type checking the `var x: AnotherType` property, and type checking the property depends on expanding the `@IntroduceArbitraryPeers` macro, which results in a circular reference error. + +Because `arbitrary` names introduced at global scope are extremely prone to circularity, peer macros attached to top-level declarations cannot introduce `arbitrary` names. + + +### Ordering of macro expansions + +When multiple macros are applied to a single declaration, the order in which macros are expanded could have a significant impact on the resulting program if the the outputs of one macro expansion are treated as inputs to the others. This form of macro composition could be fairly powerful, but it also can have a number of surprising side effects: + +1. The order in which attributes are written on a declaration could affect the result. +2. There is no obvious single source of truth for what the inputs to the macro would be. +3. Independently-developed macros could conflict in ways that break the program. + +The fine-grained nature of attached macros means that many attached macros have independent effects, e.g., macros can generate distinct members, conformances, and peers. We consider this independence of different macros applying to the same definition to be a feature, because it makes the application of macros more predictable as well as enabling more efficient implementations. + +To ensure the independence of macro invocations, each macro expansion is provided with syntax nodes as they were originally written, and not updated with the effects of other macros. For example, consider the `OptionSet` macro earlier: + +```swift +@OptionSet +struct MyOptions { + enum Option: Int { + case a + case b + case c + } +} +``` + +The `OptionSet` macro is both a member macro (which generates the instance property `rawValue` as well as static properties `a`, `b`, and `c`) and a conformance macro (to add the `OptionSet` conformance). Two different `expansion` functions will be called on the option set macro implementation: `expansion(of:providingMembersOf:in:)` and `expansion(of:providingConformancesOf:in:)`, respectively. In both cases, the syntax tree for `MyOptions` is passed as the middle argument, and both functions provide a result that augments the definition of `MyOptions`, either by adding members or by adding a conformance. + +In both cases, the expansion operation is provided with the original definition of `MyOptions`, without the new conformance or added members. That way, each expansion operation operates independently of the other---whether it's different roles for the same macro, or different macros entirely---and the order of expansion does not matter. + +### Permitted declaration kinds + +A macro can expand to any declaration that is syntactically and semantically well-formed within the context where the macro is expanded, with a few notable exceptions: + +* `import` declarations can never be produced by a macro. Swift tooling depends on the ability to resolve import declarations based on a simple scan of the original source files. Allowing a macro expansion to introduce an import declaration would complicate import resolution considerably. +* A type with the `@main` attribute cannot be produced by a macro. Swift tooling should be able to determine the presence of a main entry point in a source file based on a syntactic scan of the source file without expanding macros. +* `extension` declarations can never be produced by a macro. The effect of an extension declaration is wide-ranging, with the ability to add conformances, members, and so on. These capabilities are meant to be introduced in a more fine-grained manner. +* `operator` and `precedencegroup` declarations can never be produced by a macro, because they could allow one to reshape the precedence graph for existing code causing subtle differences in the semantics of code that sees the macro expansion vs. code that does not. +* `macro` declarations can never be produced by a macro, because allowing this would allow a macro to trivially produce infinitely recursive macro expansion. +* Top-level default literal type overrides, including `IntegerLiteralType`, + `FloatLiteralType`, `BooleanLiteralType`, + `ExtendedGraphemeClusterLiteralType`, `UnicodeScalarLiteralType`, and + `StringLiteralType`, can never be produced by a macro. + +## Source compatibility + +Attached declaration macros use the same syntax introduced for custom attributes (such as property wrappers), and therefore do not have an impact on source compatibility. + +## Effect on ABI stability + +Macros are a source-to-source transformation tool that have no ABI impact. + +## Effect on API resilience + +Macros are a source-to-source transformation tool that have no effect on API resilience. + +## Alternatives considered + +### Mutating declarations rather than augmenting them + +Attached declaration macro implementations are provided with information about the macro expansion itself and the declaration to which they are attached. The implementation cannot directly make changes to the syntax of the declaration to which it is attached; rather, it must specify the additions or changes by packaging them in `AttachedDeclarationExpansion`. + +An alternative approach would be to allow the macro implementation to directly alter the declaration to which the macro is attached. This would provide a macro implementation with greater flexibility to affect the declarations it is attached to. However, it means that the resulting declaration could vary drastically from what the user wrote, and would preventing the compiler from making any determinations about what the declaration means before expanding the macro. This could have detrimental effects on compile-time performance (one cannot determine anything about a declaration until the macros have been run on it) and might also prevent macros from accessing information about the actual declaration in the future, such as the types of parameters. + +It might be possible to provide a macro implementation API that is expressed in terms of mutation on the original declaration, but restrict the permissible changes to those that can be handled by the implementation. For example, one could "diff" the syntax tree provided to the macro implementation and the syntax tree produced by the macro implementation to identify changes, and return those changes that are acceptable to the compiler. + +## Revision History + +* Revision after acceptance: + * Make the `expansion` requirements non-`async`. +* After the first pitch: + * Added conformance macros, to produce conformances + * Moved the discussion of macro-introduced names from the freestanding macros proposal here. + * Added a carve-out to allow a `$` prefix on names generated from macros, allowing them to match the behavior of property wrappers. + * Revisited the design around the ordering of macro expansions, forcing them to be independent. + * Clarify when accessor macros need to produce observers for a stored property vs. turning it into a computed property. + * Moved the discussion of "permitted declaration kinds" from the freestanding macros proposal here. +* Originally pitched as "declaration macros"; attached macros were separated into their own proposal after the initial discussion. + +## Future directions + +### Additional attached macro roles + +There are numerous ways in which this proposal could be extended to provide new macro roles. Each new macro role would introduce a new role kind to the `@attached` attribute, along with a corresponding protocol. The macro vision document has a number of such suggestions. + +## Appendix + +### Implementation of `AddCompletionHandler` + +```swift +public struct AddCompletionHandler: PeerDeclarationMacro { + public static func expansion( + of node: CustomAttributeSyntax, + providingPeersOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] { + // Only on functions at the moment. We could handle initializers as well + // with a little bit of work. + guard let funcDecl = declaration.as(FunctionDeclSyntax.self) else { + throw CustomError.message("@AddCompletionHandler only works on functions") + } + + // This only makes sense for async functions. + if funcDecl.signature.asyncOrReasyncKeyword == nil { + throw CustomError.message( + "@AddCompletionHandler requires an async function" + ) + } + + // Form the completion handler parameter. + let resultType: TypeSyntax? = funcDecl.signature.output?.returnType.withoutTrivia() + + let completionHandlerParam = + FunctionParameterSyntax( + firstName: .identifier("completionHandler"), + colon: .colonToken(trailingTrivia: .space), + type: "(\(resultType ?? "")) -> Void" as TypeSyntax + ) + + // Add the completion handler parameter to the parameter list. + let parameterList = funcDecl.signature.input.parameterList + let newParameterList: FunctionParameterListSyntax + if let lastParam = parameterList.last { + // We need to add a trailing comma to the preceding list. + newParameterList = parameterList.removingLast() + .appending( + lastParam.withTrailingComma( + .commaToken(trailingTrivia: .space) + ) + ) + .appending(completionHandlerParam) + } else { + newParameterList = parameterList.appending(completionHandlerParam) + } + + let callArguments: [String] = try parameterList.map { param in + guard let argName = param.secondName ?? param.firstName else { + throw CustomError.message( + "@AddCompletionHandler argument must have a name" + ) + } + + if let paramName = param.firstName, paramName.text != "_" { + return "\(paramName.withoutTrivia()): \(argName.withoutTrivia())" + } + + return "\(argName.withoutTrivia())" + } + + let call: ExprSyntax = + "\(funcDecl.identifier)(\(raw: callArguments.joined(separator: ", ")))" + + // FIXME: We should make CodeBlockSyntax ExpressibleByStringInterpolation, + // so that the full body could go here. + let newBody: ExprSyntax = + """ + + Task.detached { + completionHandler(await \(call)) + } + + """ + + // Drop the @addCompletionHandler attribute from the new declaration. + let newAttributeList = AttributeListSyntax( + funcDecl.attributes?.filter { + guard case let .customAttribute(customAttr) = $0 else { + return true + } + + return customAttr != node + } ?? [] + ) + + let newFunc = + funcDecl + .withSignature( + funcDecl.signature + .withAsyncOrReasyncKeyword(nil) // drop async + .withOutput(nil) // drop result type + .withInput( // add completion handler parameter + funcDecl.signature.input.withParameterList(newParameterList) + .withoutTrailingTrivia() + ) + ) + .withBody( + CodeBlockSyntax( + leftBrace: .leftBraceToken(leadingTrivia: .space), + statements: CodeBlockItemListSyntax( + [CodeBlockItemSyntax(item: .expr(newBody))] + ), + rightBrace: .rightBraceToken(leadingTrivia: .newline) + ) + ) + .withAttributes(newAttributeList) + .withLeadingTrivia(.newlines(2)) + + return [DeclSyntax(newFunc)] + } +} +``` diff --git a/proposals/0390-noncopyable-structs-and-enums.md b/proposals/0390-noncopyable-structs-and-enums.md new file mode 100644 index 0000000000..79549cd230 --- /dev/null +++ b/proposals/0390-noncopyable-structs-and-enums.md @@ -0,0 +1,1906 @@ +# Noncopyable structs and enums + +* Proposal: [SE-0390](0390-noncopyable-structs-and-enums.md) +* Authors: [Joe Groff](https://github.com/jckarter), [Michael Gottesman](https://github.com/gottesmm), [Andrew Trick](https://github.com/atrick), [Kavon Farvardin](https://github.com/kavon) +* Review Manager: [Stephen Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 5.9)** +* Implementation: in main branch of compiler +* Review: ([pitch](https://forums.swift.org/t/pitch-noncopyable-or-move-only-structs-and-enums/61903)) ([first review](https://forums.swift.org/t/se-0390-noncopyable-structs-and-enums/63258)) ([second review](https://forums.swift.org/t/second-review-se-0390-noncopyable-structs-and-enums/63866)) ([acceptance](https://forums.swift.org/t/accepted-se-0390-noncopyable-structs-and-enums/65157)) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/5d075b86d57e3436b223199bd314b2642e30045f/proposals/0390-noncopyable-structs-and-enums.md) + +## Introduction + +This proposal introduces the concept of **noncopyable** types (also known +as "move-only" types). An instance of a noncopyable type always has unique +ownership, unlike normal Swift types which can be freely copied. + +## Motivation + +All currently existing types in Swift are **copyable**, meaning it is possible +to create multiple identical, interchangeable representations of any value of +the type. However, copyable structs and enums are not a great model for +unique resources. Classes by contrast *can* represent a unique resource, +since an object has a unique identity once initialized, and only references to +that unique object get copied. However, because the references to the object are +still copyable, classes always demand *shared ownership* of the resource. This +imposes overhead in the form of heap allocation (since the overall lifetime of +the object is indefinite) and reference counting (to keep track of the number +of co-owners currently accessing the object), and shared access often +complicates or introduces unsafety or additional overhead into an object's +APIs. Swift does not yet have a mechanism for defining types that +represent unique resources with *unique ownership*. + +## Proposed solution + +We propose to allow for `struct` and `enum` types to declare themselves as +*noncopyable*, using a new syntax for suppressing implied generic constraints, +`~Copyable`. Values of noncopyable type always have unique ownership, and +can never be copied (at least, not using Swift's implicit copy mechanism). +Since values of noncopyable structs and enums have unique identities, they can +also have `deinit` declarations, like classes, which run automatically at the +end of the unique instance's lifetime. + +For example, a basic file descriptor type could be defined as: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + init(fd: Int32) { self.fd = fd } + + func write(buffer: Data) { + buffer.withUnsafeBytes { + write(fd, $0.baseAddress!, $0.count) + } + } + + deinit { + close(fd) + } +} +``` + +Like a class, instances of this type can provide managed access to a file +handle, automatically closing the handle once the value's lifetime ends. Unlike +a class, no object needs to be allocated; only a simple struct containing the +file descriptor ID needs to be stored in the stack frame or aggregate value +that uniquely owns the instance. + +## Detailed design + +### The `Copyable` generic constraint + +Before this proposal, almost every type in Swift was automatically copyable. +The standard library provides a new generic constraint `Copyable` to make +this capability explicit. All existing first-class types (excluding nonescaping +closures) implicitly satisfy this constraint, and all generic type parameters, +existential types, protocols, and associated type requirements implicitly +require it. Types may explicitly declare that they are `Copyable`, and generic +types may explicitly require `Copyable`, but this currently has no effect. + +```swift +struct Foo: Copyable {} +``` + +### Declaring noncopyable types + +A `struct` or `enum` type can be declared as noncopyable by suppressing the +`Copyable` requirement on their declaration, by combining the new `Copyable` +constraint with the new requirement suppression syntax `~Copyable`: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 +} +``` + +If a `struct` has a stored property of noncopyable type, or an `enum` has +a case with an associated value of noncopyable type, then the containing type +must also suppress its `Copyable` capability: + +```swift +struct SocketPair: ~Copyable { + var in, out: FileDescriptor +} + +enum FileOrMemory: ~Copyable { + // write to an OS file + case file(FileDescriptor) + // write to an array in memory + case memory([UInt8]) +} + +// ERROR: copyable value type cannot contain noncopyable members +struct FileWithPath { + var file: FileDescriptor + var path: String +} +``` + +Classes, on the other hand, may contain noncopyable stored properties without +themselves becoming noncopyable: + +```swift +class SharedFile { + var file: FileDescriptor +} +``` + +A class type declaration may not use `~Copyable`; all class types remain copyable +by retaining and releasing references to the object. + +```swift +// ERROR: classes must be `Copyable` +class SharedFile: ~Copyable { + var file: FileDescriptor +} +``` + +It is also not yet allowed to suppress the `Copyable` requirement on generic +parameters, associated type requirements in protocols, or the `Self` type +in a protocol declaration, or in extensions: + +```swift +// ERROR: generic parameter types must be `Copyable` +func foo(x: T) {} + +// ERROR: types that conform to protocols must be `Copyable` +protocol Foo where Self: ~Copyable { + // ERROR: associated type requirements must be `Copyable` + associatedtype Bar: ~Copyable +} + +// ERROR: cannot suppress `Copyable` in extension of `FileWithPath` +extension FileWithPath: ~Copyable {} +``` + +`Copyable` also cannot be suppressed in existential type declarations: + +```swift +// ERROR: `any` types must be `Copyable` +let foo: any ~Copyable = FileDescriptor() +``` + +### Restrictions on use in generics + +Noncopyable types may have generic type parameters: + +```swift +// A type that reads from a file descriptor consisting of binary values of type T +// in sequence. +struct TypedFile: ~Copyable { + var rawFile: FileDescriptor + + func read() -> T { ... } +} + +let byteFile: TypedFile // OK +``` + +At this time, as noted above, generic types are still always required to be +`Copyable`, so noncopyable types themselves are not allowed to be used as a +generic type argument. This means a noncopyable type _cannot_: + +- conform to any protocols, except for `Sendable`. +- serve as a type witness for an `associatedtype` requirement. +- be used as a type argument when instantiating generic types or calling generic functions. +- be cast to (or from) `Any` or any other existential. +- be accessed through reflection. +- appear in a tuple. + +The reasons for these restrictions and ways of lifting them are discussed under +Future Directions. The key implication of these restrictions is that a +noncopyable struct or enum is only a subtype of itself, because all other types +it might be compatible with for conversion would also permit copying. + +#### The `Sendable` exception + +The need for preventing noncopyable types from conforming to +protocols is rooted in the fact that all existing constrained generic types +(like `some P` types) and existentials (`any P` types) are assumed to be +copyable. Recording any conformances to these protocols would be invalid for +noncopyable types. + +But, an exception is made where noncopyable types can conform to `Sendable`. +Unlike other protocols, the `Sendable` marker protocol leaves no conformance +record in the output program. Thus, there will be no ABI impact if a future +noncopyable version of the `Sendable` protocol is created. + +The big benefit of allowing `Sendable` conformances is that noncopyable types +are compatible with concurrency. Keep in mind that despite their ability to +conform to `Sendable`, noncopyable structs and enums are still only a subtype +of themselves. That means when the noncopyable type conforms to `Sendable`, you +still cannot convert it to `any Sendable`, because copying that existential +would copy its underlying value: + +```swift +extension FileDescriptor: Sendable {} // OK + +struct RefHolder: ~Copyable, Sendable { + var ref: Ref // ERROR: stored property 'ref' of 'Sendable'-conforming struct 'RefHolder' has non-sendable type 'Ref' +} + +func openAsync(_ path: String) async throws -> FileDescriptor {/* ... */} +func sendToSpace(_ s: some Sendable) {/* ... */} + +@MainActor func example() async throws { + // OK. FileDescriptor can cross actors because it is Sendable + var fd: FileDescriptor = try await openAsync("/dev/null") + + // ERROR: noncopyable types cannot be conditionally cast + // WARNING: cast from 'FileDescriptor' to unrelated type 'any Sendable' always fails + if let sendy: Sendable = fd as? Sendable { + + // ERROR: noncopyable types cannot be conditionally cast + // WARNING: cast from 'any Sendable' to unrelated type 'FileDescriptor' always fails + fd = sendy as! FileDescriptor + } + + // ERROR: noncopyable type 'FileDescriptor' cannot be used with generics + sendToSpace(fd) +} +``` + +#### Working around the generics restrictions + +Since a good portion of Swift's standard library rely on generics, there are a +a number of common types and functions that will not work with today's +noncopyable types: + +```swift +// ERROR: Cannot use noncopyable type FileDescriptor in generic type Optional +let x = Optional(FileDescriptor(open("/etc/passwd", O_RDONLY))) + +// ERROR: Cannot use noncopyable type FileDescriptor in generic type Array +let fds: [FileDescriptor] = [] + +// ERROR: Cannot use noncopyable type FileDescriptor in generic type Any +print(FileDescriptor(-1)) + +// ERROR: Noncopyable struct SocketEvent cannot conform to Error +enum SocketEvent: ~Copyable, Error { + case requestedDisconnect(SocketPair) +} +``` + +For example, the `print` function expects to be able to convert its argument to +`Any`, which is a copyable value. Internally, it also relies on either +reflection or conformance to `CustomStringConvertible`. Since a noncopyable type +can't do any of those, a suggested workaround is to explicitly define a +conversion to `String`: + +```swift +extension FileDescriptor /*: CustomStringConvertible */ { + var description: String { + return "file descriptor #\(fd)" + } +} + +let fd = FileDescriptor(-1) +print(fd.description) +``` + +A more general kind of workaround to mix generics and noncopyable types +is to wrap the value in an ordinary class instance, which itself can participate +in generics. To transfer the noncopyable value in or out of the wrapper class +instance, using `Optional` for the class's field would be +ideal. But until that is supported, a concrete noncopyable enum can represent +the case where the value of interest was taken out of the instance: + +```swift +enum MaybeFileDescriptor: ~Copyable { + case some(FileDescriptor) + case none + + // Returns this MaybeFileDescriptor by consuming it + // and leaving .none in its place. + mutating func take() -> MaybeFileDescriptor { + let old = self // consume self + self = .none // reinitialize self + return old + } +} + +class WrappedFile { + var file: MaybeFileDescriptor + + enum Err: Error { case noFile } + + init(_ fd: consuming FileDescriptor) { + file = .some(fd) + } + + func consume() throws -> FileDescriptor { + if case let .some(fd) = file.take() { + return fd + } + throw Err.noFile + } +} + +func example(_ fd1: consuming FileDescriptor, + _ fd2: consuming FileDescriptor) -> [WrappedFile] { + // create an array of descriptors + return [WrappedFile(fd1), WrappedFile(fd2)] +} +``` + +All of this boilerplate melts away once noncopyable types support generics. +Even before then, one major improvement would be to eliminate the need to define +types like `MaybeFileDescriptor` through a noncopyable `Optional` +(see Future Directions). + + +### Using noncopyable values + +As the name suggests, values of noncopyable type cannot be copied, a major break +from most other types in Swift. Many operations are currently defined as +working as pass-by-value, and use copying as an implementation technique +to give that semantics, but these operations now need to be defined more +precisely in terms of how they *borrow* or *consume* their operands in order to +define their effects on values that cannot be copied. + +We use the term **consume** to refer to an operation +that invalidates the value that it operates on. It may do this by directly +destroying the value, freeing its resources such as memory and file handles, +or forwarding ownership of the value to yet another owner who takes +responsibility for keeping it alive. Performing a consuming operation on +a noncopyable value generally requires having ownership of the value to begin +with, and invalidates the value the operation was performed on after it is +completed. + +We use the term **borrow** to refer to +a shared borrow of a single instance of a value; the operation that borrows +the value allows other operations to borrow the same value simultaneously, and +it does not take ownership of the value away from its current owner. This +generally means that borrowers are not allowed to mutate the value, since doing +so would invalidate the value as seen by the owner or other simultaneous +borrowers. Borrowers also cannot *consume* the value. They can, however, +initiate arbitrarily many additional borrowing operations on all or part of +the value they borrow. + +Both of these conventions stand in contrast to **mutating** (or **inout**) +operations, which take an *exclusive* borrow of their operands. The behavior +of mutating operations on noncopyable values is much the same as `inout` +parameters of copyable type today, which are already subject to the +"law of exclusivity". A mutating operation has exclusive access to its operand +for the duration of the operation, allowing it to freely mutate the value +without concern for aliasing or data races, since not even the owner may +access the value simultaneously. A mutating operation may pass its operand +to another mutating operation, but transfers exclusivity to that other operation +until it completes. A mutating operation may also pass its operand to +any number of borrowing operations, but cannot assume exclusivity while those +borrows are enacted; when the borrowing operations complete, the mutating +operation may assume exclusivity again. Unlike having true ownership of a +value, mutating operations give ownership back to the owner at the end of an +operation. A mutating operation therefore may consume the current value of its +operand, but if it does, it must replace it with a new value before completing. + +For copyable types, the distinction between borrowing and consuming operations +is largely hidden from the programmer, since Swift will implicitly insert +copies as needed to maintain the apparent value semantics of operations; a +consuming operation can be turned into a borrowing one by copying the value and +giving the operation the copy to consume, allowing the program to continue +using the original. This of course becomes impossible for values that cannot +be copied, forcing the distinction. + +Many code patterns that are allowed for copyable types also become errors for +noncopyable values because they would lead to conflicting uses of the same +value, without the ability to insert copies to avoid the conflict. For example, +a copyable value can normally be passed as an argument to the same function +multiple times, even to a `borrowing` and `consuming` parameter of the same +call, and the compiler will copy as necessary to make all of the function's +parameters valid according to their ownership specifiers: + +```swift +func borrow(_: borrowing Value, and _: borrowing Value) {} +func consume(_: consuming Value, butBorrow _: borrowing Value) {} +let x = Value() +borrow(x, and: x) // this is fine, multiple borrows can share +consume(x, butBorrow: x) // also fine, we'll copy x to let a copy be consumed + // while the other is borrowed +``` + +By contrast, a noncopyable value *must* be passed by borrow or consumed, +without copying. This makes the second call above impossible for a noncopyable +`x`, since attempting to consume `x` would end the binding's lifetime while +it also needs to be borrowed: + +```swift +func borrow(_: borrowing FileDescriptor, and _: borrowing FileDescriptor) {} +func consume(_: consuming FileDescriptor, butBorrow _: borrowing FileDescriptor) {} +let x = FileDescriptor() +borrow(x, and: x) // still OK to borrow multiple times +consume(x, butBorrow: x) // ERROR: consuming use of `x` would end its lifetime + // while being borrowed +``` + +A similar effect happens when `inout` parameters take noncopyable arguments. +Swift will copy the value of a variable if it is passed both by value and +`inout`, so that the by-value parameter receives a copy of the current value +while leaving the original binding available for the `inout` parameter to +exclusively access: + +```swift +func update(_: inout Value, butBorrow _: borrow Value) {} +func update(_: inout Value, butConsume _: consume Value) {} +var x = Value() +update(&x, butBorrow: x) // this is fine, we'll copy `x` in the second parameter +update(&x, butConsume: x) // also fine, we'll also copy +``` + +But again, for a noncopyable value, this implicit copy is impossible, so +these sorts of calls become exclusivity errors: + +```swift +func update(_: inout FileDescriptor, butBorrow _: borrow FileDescriptor) {} +func update(_: inout FileDescriptor, butConsume _: consume FileDescriptor) {} + +var y = FileDescriptor() +update(&y, butBorrow: y) // ERROR: cannot borrow `y` while exclusively accessed +update(&y, butConsume: y) // ERROR: cannot consume `y` while exclusively accessed +``` + +The following sections attempt to classify existing language operations +according to what ownership semantics they have when performed on noncopyable +values. + +### Consuming operations + +The following operations are consuming: + +- assigning a value to a new `let` or `var` binding, or setting an existing + variable or property to the binding: + + ```swift + let x = FileDescriptor() + let y = x + use(x) // ERROR: x consumed by assignment to `y` + ``` + + ```swift + var y = FileDescriptor() + let x = FileDescriptor() + y = x + use(x) // ERROR: x consumed by assignment to `y` + ``` + + ```swift + class C { + var property = FileDescriptor() + } + let c = C() + let x = FileDescriptor() + c.property = x + use(x) // ERROR: x consumed by assignment to `c.property` + ``` + + The one exception is assigning to the "black hole" `_ = x`, which is + a borrowing operation, as noted below. This allows for the + `_ = x` idiom to still be used to prevent warnings about a borrowed + binding that is otherwise unused. + +- passing an argument to a `consuming` parameter of a function: + + ```swift + func consume(_: consuming FileDescriptor) {} + let x1 = FileDescriptor() + consume(x1) + use(x1) // ERROR: x1 consumed by call to `consume` + ``` + +- passing an argument to an `init` parameter that is not explicitly + `borrowing`: + + ```swift + struct S: ~Copyable { + var x: FileDescriptor, y: Int + } + let x = FileDescriptor() + let s = S(x: x, y: 219) + use(x) // ERROR: x consumed by `init` of struct `S` + ``` + +- invoking a `consuming` method on a value, or accessing a property of the + value through a `consuming get` or `consuming set` accessor: + + ```swift + extension FileDescriptor { + consuming func consume() {} + } + let x = FileDescriptor() + x.consume() + use(x) // ERROR: x consumed by method `consume` + ``` + +- explicitly consuming a value with the `consume` operator: + + ```swift + let x = FileDescriptor() + _ = consume x + use(x) // ERROR: x consumed by explicit `consume` + ``` + +- `return`-ing a value; + +- pattern-matching a value with `switch`, `if let`, or `if case`: + + ```swift + let x: Optional = getValue() + if let y = consume x { ... } + use(x) // ERROR: x consumed by `if let` + + enum FileDescriptorOrBuffer: ~Copyable { + case file(FileDescriptor) + case buffer(String) + } + + let x = FileDescriptorOrBuffer.file(FileDescriptor()) + + switch consume x { + case .file(let f): + break + case .buffer(let b): + break + } + + use(x) // ERROR: x consumed by `switch` + ``` + + In order to allow for borrowing pattern matching to potentially become + the default later, when it's supported, the operand to `switch` or + the right-hand side of a `case` condition in an `if` or `while` must + use the `consume` operator in order to indicate that it is consumed. + We may want `switch x` to borrow by default in the future. + +- iterating a `Sequence` with a `for` loop: + + ```swift + let xs = [1, 2, 3] + for x in consume xs {} + use(xs) // ERROR: xs consumed by `for` loop + ``` + +(Although noncopyable types are not currently allowed to conform to +protocols, preventing them from implementing the `Sequence` protocol, +and cannot be used as generic parameters, preventing the formation of +`Optional` noncopyable types, these last two cases are listed for completeness, +since they would affect the behavior of other language features that +suppress implicit copying when applied to copyable types.) + +The `consume` operator can always transfer ownership of its operand when the +`consume` expression is itself the operand of a consuming operation. + +Consuming is flow-sensitive, so if one branch of an `if` or other control flow +consumes a noncopyable value, then other branches where the value +is not consumed may continue using it: + +```swift +let x = FileDescriptor() +guard let condition = getCondition() else { + consume(x) + return +} +// We can continue using x here, since only the exit branch of the guard +// consumed it +use(x) +``` + +For the purposes of the following discussion, a closure is considered nonescaping +in the following cases: + +- if the closure literal appears as an argument to a function parameter of + non-`@escaping` function type, or +- if the closure literal is assigned to a local `let` variable, that does not + itself get captured by an escaping closure. + +These cases correspond to the cases where a closure is allowed to capture an +`inout` parameter from its surrounding scope, before this proposal. + +### Borrowing operations + +The following operations are borrowing: + +- Passing an argument to a `func` or `subscript` parameter that does not + have an ownership modifier, or an argument to any `func`, `subscript`, or + `init` parameter which is explicitly marked `borrow`. The + argument is borrowed for the duration of the callee's execution. +- Borrowing a stored property of a struct or tuple borrows the struct or tuple + for the duration of the access to the stored property. This means that one + field of a struct cannot be borrowed while another is being mutated, as in + `call(struc.fieldA, &struc.fieldB)`. Allowing for fine-grained subelement + borrows in some circumstances is discussed as a Future Direction below. +- A stored property of a class may be borrowed using a dynamic exclusivity + check, to assert that there are no aliasing mutations attempted during the + borrow, as discussed under "Noncopyable stored properties in classes" below. +- Invoking a `borrowing` method on a value, or a method which is not annotated + as any of `borrowing`, `consuming` or `mutating`, borrows the `self` parameter + for the duration of the callee's execution. +- Accessing a computed property or subscript through `borrowing` or + `nonmutating` getter or setter borrows the `self` parameter for the duration + of the accessor's execution. +- Capturing an immutable local binding into a nonescaping closure borrows the + binding for the duration of the callee that receives the nonescaping closure. +- Assigning into the "black hole" `_ = x` borrows the right-hand side of the + assignment. + +### Mutating operations + +The following operations are mutating uses: + +- Passing an argument to a `func` parameter that is `inout`. The argument is + exclusively accessed for the duration of the call. +- Projecting a stored property of a struct for mutation is a mutating use of + the entire struct. +- A stored property of a class may be mutated using a dynamic exclusivity + check, to assert that there are no aliasing mutations, as happens today. + For noncopyable properties, the assertion also enforces that no borrows + are attempted during the mutation, as discussed under "Noncopyable stored + properties in classes" below. +- Invoking a `mutating` method on a value is a mutating use of the `self` + parameter for the duration of the callee's execution. +- Accessing a computed property or subscript through a `mutating` getter and/or + setter is a mutating use of `self` for the duration of the accessor's + execution. +- Capturing a mutable local binding into a nonescaping closure is a mutating + use of the binding for the duration of the callee that receives the + nonescaping closure. + +### Declaring functions and methods with noncopyable parameters + +When noncopyable types are used as function parameters, the ownership +convention becomes a much more important part of the API contract. +As such, when a function parameter is declared with a noncopyable type, it +**must** declare whether the parameter uses the `borrowing`, `consuming`, or +`inout` convention: + +```swift +// Redirect a file descriptor +// Require exclusive access to the FileDescriptor to replace it +func redirect(_ file: inout FileDescriptor, to otherFile: borrowing FileDescriptor) { + dup2(otherFile.fd, file.fd) +} + +// Write to a file descriptor +// Only needs shared access +func write(_ data: [UInt8], to file: borrowing FileDescriptor) { + data.withUnsafeBytes { + write(file.fd, $0.baseAddress, $0.count) + } +} + +// Close a file descriptor +// Consumes the file descriptor +func close(file: consuming FileDescriptor) { + close(file.fd) +} +``` + +Methods of the noncopyable type are considered to be `borrowing` unless +declared `mutating` or `consuming`: + +```swift +extension FileDescriptor { + mutating func replace(with otherFile: borrowing FileDescriptor) { + dup2(otherFile.fd, self.fd) + } + + // borrowing by default + func write(_ data: [UInt8]) { + data.withUnsafeBytes { + write(file.fd, $0.baseAddress, $0.count) + } + } + + consuming func close() { + close(fd) + } +} +``` + +Static casts or coercions of function types that change the ownership modifier +of a noncopyable parameter are currently invalid. One reason is that it is +impossible to convert a function with a noncopyable `consuming` parameter, into +one where that parameter is `borrowed`, without inducing a copy of the borrowed +parameter. See Future Directions for details. + +### Declaring properties of noncopyable type + +A class or noncopyable struct may declare stored `let` or `var` properties of +noncopyable type. A noncopyable `let` stored property may only be borrowed, +whereas a `var` stored property may be both borrowed and mutated. Stored +properties cannot generally be consumed because doing so would leave the +containing aggregate in an invalid state. + +Any type may also declare computed properties of noncopyable type. The `get` +accessor returns an owned value that the caller may consume, like a function +would. The `set` accessor receives its `newValue` as a `consuming` parameter, +so the setter may consume the parameter value to update the containing +aggregate. + +Accessors may use the `consuming` and `borrowing` declaration modifiers to +affect the ownership of `self` while the accessor executes. `consuming get` +is particularly useful as a way of forwarding ownership of part of an aggregate, +such as to take ownership away from a wrapper type: + +```swift +struct FileDescriptorWrapper: ~Copyable { + private var _value: FileDescriptor + + var value: FileDescriptor { + consuming get { return _value } + } +} +``` + +However, a `consuming get` cannot be paired with a setter when the containing +type is `~Copyable`, because invoking the getter consumes the aggregate, +leaving nothing to write a modified value back to. + +Because getters return owned values, non-`consuming` getters generally cannot +be used to wrap noncopyable stored properties, since doing so would require +copying the value out of the aggregate: + +```swift +class File { + private var _descriptor: FileDescriptor + + var descriptor: FileDescriptor { + return _descriptor // ERROR: attempt to copy `_descriptor` + } +} +``` + +These limitations could be addressed in the future by exposing the ability for +computed properties to also provide "read" and "modify" coroutines, which would +have the ability to yield borrowing or mutating access to properties without +copying them. + +### Using stored properties and enum cases of noncopyable type + +When classes or noncopyable types contain members that are of noncopyable +type, then the container is the unique owner of the member value. Outside of +the type's definition, client code cannot perform consuming operations on +the value, since it would need to take away the container's ownership to do +so: + +```swift +struct Inner: ~Copyable {} + +struct Outer: ~Copyable { + var inner = Inner() +} + +let outer = Outer() +let i = outer.inner // ERROR: can't take `inner` away from `outer` +``` + +However, when code has the ability to mutate the member, it may freely modify, +reassign, or replace the value in the field: + +```swift +var outer = Outer() +let newInner = Inner() +// OK, transfers ownership of `newInner` to `outer`, destroying its previous +// value +outer.inner = newInner +``` + +Note that, as currently defined, `switch` to pattern-match an `enum` is a +consuming operation, so it can only be performed inside `consuming` methods +on the type's original definition: + +```swift +enum OuterEnum: ~Copyable { + case inner(Inner) + case file(FileDescriptor) +} + +// Error, can't partially consume a value outside of its definition +let enum = OuterEnum.inner(Inner()) +switch enum { +case .inner(let inner): + break +default: + break +} +``` + +Being able to borrow in pattern matches would address this shortcoming. + +### Noncopyable stored properties in classes + +Since objects may have any number of simultaneous references, Swift uses +dynamic exclusivity checking to prevent simultaneous writes of the same +stored property. This dynamic checking extends to borrows of noncopyable +stored properties; the compiler will attempt to diagnose obvious borrowing +failures, as it will for local variables and value types, but a runtime error +will occur if an uncaught exclusivity error occurs, such as an attempt to mutate +an object's stored property while it is being borrowed: + +```swift +class Foo { + var fd: FileDescriptor + + init(fd: FileDescriptor) { self.fd = fd } +} + +func update(_: inout FileDescriptor, butBorrow _: borrow FileDescriptor) {} + +func updateFoo(_ a: Foo, butBorrowFoo b: Foo) { + update(&a.fd, butBorrow: b.fd) +} + +let foo = Foo(fd: FileDescriptor()) + +// Will trap at runtime when foo.fd is borrowed and mutated at the same time +updateFoo(foo, butBorrowFoo: foo) +``` + +`let` properties do not allow mutating accesses, and this continues to hold for +noncopyable types. The value of a `let` property in a class therefore does not +need dynamic checking, even if the value is noncopyable; the value behaves as +if it is always borrowed, since there may potentially be a borrow through +some reference to the object at any point in the program. Such values can +thus never be consumed or mutated. + +The dynamic borrow state of properties is tracked independently for every +stored property in the class, so it is safe to mutate one property while other +properties of the same object are also being mutated or borrowed: + +```swift +class SocketTriple { + var in, middle, out: FileDescriptor +} + +func update(_: inout FileDescriptor, and _: inout FileDescriptor, + whileBorrowing _: borrowing FileDescriptor) {} + +// This is OK +let object = SocketTriple(...) +update(&object.in, and: &object.out, whileBorrowing: object.middle) +``` + +This dynamic tracking, however, cannot track accesses at finer resolution +than properties, so in circumstances where we might otherwise eventually be +able to support independent borrowing of fields in structs, tuples, and enums, +that support will not extend to fields within class properties, since the +entire property must be in the borrowing or mutating state. + +Dynamic borrowing or mutating accesses require that the enclosing object be +kept alive for the duration of the assertion of the access. Normally, this +is transparent to the developer, as the compiler will keep a copy of a +reference to the object retained while these accesses occur. However, if +we introduce noncopyable bindings to class references, such as [the `borrow` +and `inout` bindings](https://forums.swift.org/t/pitch-borrow-and-inout-declaration-keywords/62366) +currently being pitched, this would manifest as a borrow of the noncopyable +reference, preventing mutation or consumption of the reference during +dynamically-asserted accesses to its properties: + +```swift +class SocketTriple { + var in, middle, out: FileDescriptor +} + +func borrow(_: borrowing FileDescriptor, + whileReplacingObject _: inout SocketTriple) {} + +var object = SocketTriple(...) + +// This is OK, since ARC will keep a copy of the `object` reference retained +// while `object.in` is borrowed +borrow(object.in, whileReplacingObject: &object) + +inout objectAlias = &object + +// This is an error, since we aren't allowed to implicitly copy through +// an `inout` binding, and replacing `objectAlias` without keeping a copy +// retained might invalidate the object while we're accessing it. +borrow(objectAlias.in, whileReplacingObject: &objectAlias) +``` + +### Noncopyable variables captured by escaping closures + +Nonescaping closures have scoped lifetimes, so they can borrow their captures, +as noted in the "borrowing operations" and "consuming operations" sections +above. Escaping closures, on the other hand, have indefinite lifetimes, since +they can be copied and passed around arbitrarily, and multiple escaping closures +can capture and access the same local variables alongside the local context +from which those captures were taken. Variables captured by escaping closures +thus behave like class properties; immutable captures are treated as always +borrowed both inside the closure body and in the capture's original context. + +```swift +func escape(_: @escaping () -> ()) {...} + +func borrow(_: borrowing FileDescriptor) {} +func consume(_: consuming FileDescriptor) {} + +func foo() { + let x = FileDescriptor() + + // ERROR: cannot consume variable before it's been captured + consume(x) + + escape { + borrow(x) // OK + consume(x) // ERROR: cannot consume captured variable + } + + // OK + borrow(x) + + // ERROR: cannot consume variable after it's been captured by an escaping + // closure + consume(x) +} +``` + +Mutable captures are subject to dynamic exclusivity checking like class +properties are, and similarly cannot be consumed and reinitialized. When +a closure escapes, the compiler isn't able to statically know when the closure +is invoked, and it may even be invoked multiple overlapping times, or +simultaneously on different threads if the closure is `@Sendable`, so the +captures must always remain in a valid state for memory safety, and exclusivity +of mutations can only be enforced dynamically. + +```swift +var escapedClosure: (@escaping (inout FileDescriptor) -> ())? + +func foo() { + var x = FileDescriptor() + + // ERROR: cannot consume variable before it's been captured. + // (We could potentially support local consumption before the variable + // capture occurs as a future direction.) + consume(x) + x = FileDescriptor() + + escapedClosure = { _ in borrow(x) } + + // Runtime error when exclusive access to `x` dynamically conflicts + // with attempted borrow of `x` during `escapedClosure`'s execution + escapedClosure!(&x) +} +``` + +### Deinitializers + +A noncopyable struct or enum may declare a `deinit`, which will run +implicitly when the lifetime of the value ends (unless explicitly suppressed +with `discard` as explained below): + +```swift +struct File: ~Copyable { + var descriptor: Int32 + + func write(_ values: S) { /*..*/ } + + consuming func close() { + print("closing file") + } + + deinit { + print("deinitializing file") + closeFile(rawDescriptor: descriptor) + } +} +``` + +Like a class `deinit`, a struct or enum `deinit` may not propagate any +uncaught errors. Within the body of the `deinit`, `self` behaves as in +a `borrowing` method; it may not be modified or consumed inside the +`deinit`. (Allowing for mutation and partial invalidation inside a +`deinit` is explored as a future direction.) + +A value's lifetime ends, and its `deinit` runs if present, in the following +circumstances: + +- For a local `var` or `let` binding, or `consuming` function parameter, that is + not itself consumed, `deinit` runs at the end of the binding's lexical + scope. If, on the other hand, the binding is consumed, then responsibility + for deinitialization gets forwarded to the consumer (which may in turn forward + it somewhere else). As explained later, a `_ = consume` operator with no + destination immediately runs the `deinit`. + + ```swift + do { + let file = File(descriptor: 42) + file.close() // consuming use + // file's deinit runs inside `close` + print("done writing") + } + // Output: + // closing file + // deinitializing file + // done writing + + do { + let file = File(descriptor: 42) + file.write([1,2,3]) // borrowing use + print("done writing") + // file's deinit runs here + } + // Output: + // done writing + // deinitializing file + ``` + + If a noncopyable value is conditionally consumed, then the deinitializer + runs as late as possible on any nonconsumed paths: + + ```swift + let condition = false + do { + let file = File(descriptor: 42) + file.write([1,2,3]) // borrowing use + if condition { + file.close() + } else { + print("not closed") + // file's deinit runs here + } + print("done writing") + } + // Output: + // not closed + // deinitializing file + // done writing + ``` + +- When a struct, enum, or class contains a member of noncopyable type, the member is destroyed, and its deinit is +run, after the container's deinit runs. For example: + +```swift +struct Inner: ~Copyable { + deinit { print("destroying inner") } +} + +struct Outer: ~Copyable { + var inner = Inner() + deinit { print("destroying outer") } +} + +do { + _ = Outer() +} +``` + +will print: +``` +destroying outer +destroying inner +``` + +### Suppressing `deinit` in a `consuming` method + +It is often useful for noncopyable types to provide alternative ways to consume +the resource represented by the value besides `deinit`. However, +under normal circumstances, a `consuming` method will still invoke the type's +`deinit` after the last use of `self`, which is undesirable when the method's +own logic already invalidates the value: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + deinit { + close(fd) + } + + consuming func close() { + close(fd) + + // The lifetime of `self` ends here, triggering `deinit` (and another call to `close`)! + } +} +``` + +In the above example, the double-close could be avoided by having the +`close()` method do nothing on its own and just allow the `deinit` to +implicitly run. However, we may want the method to have different behavior +from the deinit; for example, it could raise an error (which a normal `deinit` +is unable to do) if the `close` system call triggers an OS error : + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + consuming func close() throws { + // POSIX close may raise an error (which leaves the file descriptor in an + // unspecified state, so we can't really try to close it again, but the + // error may nonetheless indicate a condition worth handling) + if close(fd) != 0 { + throw CloseError(errno) + } + + // We don't want to trigger another close here! + } +} +``` + +or it could be useful to take manual control of the file descriptor back from +the type, such as to pass to a C API that will take care of closing it: + +```swift +struct FileDescriptor: ~Copyable { + // Take ownership of the C file descriptor away from this type, + // returning the file descriptor without closing it + consuming func take() -> Int32 { + return fd + + // We don't want to trigger close here! + } +} +``` + +We propose to introduce a special operator, `discard self`, which ends the +lifetime of `self` without running its `deinit`: + +```swift +struct FileDescriptor: ~Copyable { + // Take ownership of the C file descriptor away from this type, + // returning the file descriptor without closing it + consuming func take() -> Int32 { + let fd = self.fd + discard self + return fd + } +} +``` + +`discard self` can only be applied to `self`, in a consuming method +defined in the same file as the type's original definition. (This is in +contrast to Rust's similar special function, +[`mem::forget`](https://doc.rust-lang.org/std/mem/fn.forget.html), which is a +standalone function that can be applied to any value, anywhere. Although the +Rust documentation notes that this operation is "safe" on the principle that +destructors may not run at all, due to reference cycles, process termination, +etc., in practice the ability to forget arbitrary values creates semantic +issues for many Rust APIs, particularly when there are destructors on types +with lifetime dependence on each other like `Mutex` and `LockGuard`. As such, +we think it is safer to restrict the ability to suppress the standard `deinit` +for a value to the core API of its type. We can relax this restriction if +experience shows a need to.) + +For the extent of this proposal, we also propose that `discard self` can only +be applied in types whose components include no reference-counted, generic, +or existential fields, nor do they include any types that transitively include +any fields of those types or that have `deinit`s defined of their own. (Such +a type might be called "POD" or "trivial" following C++ terminology). We explore +lifting this restriction as a future direction. + + +Even with the ability to `discard self`, care would still need be taken when +writing destructive operations to avoid triggering the deinit on alternative +exit paths, such as early `return`s, `throw`s, or implicit propagation of +errors from `try` operations. For instance, if we write: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + consuming func close() throws { + // POSIX close may raise an error (which still invalidates the + // file descriptor, but may indicate a condition worth handling) + if close(fd) != 0 { + throw CloseError(errno) + // !!! Oops, we didn't suppress deinit on this path, so we'll double close! + } + + // We don't need to deinit self anymore + discard self + } +} +``` + +then the `throw` path exits the method without `discard`, and +`deinit` will still execute if an error occurs. To avoid this mistake, we +propose that if any path through a method uses `discard self`, then +**every** path must choose either to `discard` or to explicitly `consume self`, +which triggers the standard `deinit`. This will make the above code an error, +alerting that the code should be rewritten to ensure `discard self` +always executes: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + consuming func close() throws { + // Save the file descriptor and give up ownership of it + let fd = self.fd + discard self + + // We can now use `fd` below without worrying about `deinit`: + + // POSIX close may raise an error (which still invalidates the + // file descriptor, but may indicate a condition worth handling) + if close(fd) != 0 { + throw CloseError(errno) + } + } +} +``` + +The [consume operator](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md) +must be used to explicitly end the value's lifetime using its `deinit` if +`discard` is used to conditionally destroy the value on other paths +through the method. + +```swift +struct MemoryBuffer: ~Copyable { + private var address: UnsafeRawPointer + + init(size: Int) throws { + guard let address = malloc(size) else { + throw MallocError() + } + self.address = address + } + + deinit { + free(address) + } + + consuming func takeOwnership(if condition: Bool) -> UnsafeRawPointer? { + if condition { + // Save the memory buffer and give it to the caller, who + // is promising to free it when they're done. + let address = self.address + discard self + return address + } else { + // We still want to free the memory if we aren't giving it away. + _ = consume self + return nil + } + } +} +``` + +## Source compatibility + +For existing Swift code, this proposal is additive. + +## Effect on ABI stability + +### Adding or removing `Copyable` breaks ABI + +An existing copyable struct or enum cannot have its `Copyable` capability +taken away without breaking ABI, since existing clients may copy values of the +type. + +Ideally, we would allow noncopyable types to become `Copyable` without breaking +ABI; however, we cannot promise this, due to existing implementation choices we +have made in the ABI that cause the copyability of a type to have unavoidable +knock-on effects. In particular, when properties are declared in classes, +protocols, or public non-`@frozen` structs, we define the property's ABI to use +accessors even if the property is stored, with the idea that it should be +possible to change a property's implementation to change it from a stored to +computed property, or vice versa, without breaking ABI. + +The accessors used as ABI today are the traditional `get` and `set` +computed accessors, as well as a `_modify` coroutine which can optimize `inout` +operations and projections into stored properties. `_modify` and `set` are +not problematic for noncopyable types. However, `get` behaves like a +function, producing the property's value by returning it like a function would, +and returning requires *consuming* the return value to transfer it to the +caller. This is not possible for noncopyable stored properties, since the +value of the property cannot be copied in order to return a copy without +invalidating the entire containing struct or object. + +Therefore, properties of noncopyable type need a different ABI in order to +properly abstract them. In particular, instead of exposing a `get` accessor +through abstract interfaces, they must use a `_read` coroutine, which is the +read-only analog to `_modify`, allowing the implementation to yield a borrow of +the property value in-place instead of returning by value. This allows for +noncopyable stored properties to be exposed while still being abstracted enough +that they can be replaced by a computed implementation, since a `get`-based +implementation could still work underneath the `read` coroutine by evaluating +the getter, yielding a borrow of the returned value, then disposing of the +temporary value. + +As such, we cannot simply say that making a noncopyable type copyable is an +ABI-safe change, since doing so will have knock-on effects on the ABI of any +properties of the type. We could potentially provide a "born noncopyable" +attribute to indicate that a copyable type should use the noncopyable ABI +for any properties, as a way to enable the evolution into a copyable type +while preserving existing ABI. However, it also seems unlikely to us that many +types would need to evolve between being copyable or not frequently. + +### Adding, removing, or changing `deinit` in a struct or enum + +An noncopyable type that is not `@frozen` can add or remove its deinit without +affecting the type's ABI. If `@frozen`, a deinit cannot be added or removed, +but the deinit implementation may change (if the deinit is not additionally +`@inlinable`). + +### Adding noncopyable fields to classes + +A class may add fields of noncopyable type without changing ABI. + +## Effect on API resilience + +Introducing new APIs using noncopyable types is an additive change. APIs that +adopt noncopyable types have some notable restrictions on how they can further +evolve while maintaining source compatibility. + +A noncopyable type can be made copyable while generally maintaining source +compatibility. Values in client source would acquire normal ARC lifetime +semantics instead of eager-move semantics when those clients are recompiled +with the type as copyable, and that could affect the observable order of +destruction and cleanup. Since copyable value types cannot directly define +`deinit`s, being able to observe these order differences is unlikely, but not +impossible when references to classes are involved. + +A `consuming` parameter of noncopyable type can be changed into a `borrowing` +parameter without breaking source for clients (and likewise, a `consuming` +method can be made `borrowing`). Conversely, changing +a `borrowing` parameter to `consuming` may break client source. (Either direction +is an ABI breaking change.) This is because a consuming use is required to +be the final use of a noncopyable value, whereas a borrowing use may or may not +be. + +Adding or removing a `deinit` to a noncopyable type does not affect source +for clients. + +## Alternatives considered + +### Naming the attribute "move-only" + +We have frequently referred to these types as "move-only types" in various +vision documents. However, as we've evolved related proposals like the +`consume` operator and parameter modifiers, the community has drifted away +from exposing the term "move" in the language elsewhere. When explaining these +types to potential users, we've also found that the name "move-only" incorrectly +suggests that being noncopyable is a new capability of types, and that there +should be generic functions that only operate on "move-only" types, when really +the opposite is the case: all existing types in Swift today conform to +effectively an implicit "Copyable" requirement, and what this feature does is +allow types not to fulfill that requirement. When generics grow support for +move-only types, then generic functions and types that accept noncopyable +type parameters will also work with copyable types, since copyable types +are strictly more capable. This proposal prefers the term "noncopyable" to make +the relationship to an eventual `Copyable` constraint, and the fact that annotated +types lack the ability to satisfy this constraint, more explicit. + +### Spelling as a generic constraint + +It's a reasonable question why declaring a type as noncopyable isn't spelled +like a regular protocol constraint, instead of as the removal of an existing +constraint: + +```swift +struct Foo: NonCopyable {} +``` + +As noted in the previous discussion, an issue with this notation is that it +implies that `NonCopyable` is a new capability or requirement, rather than +really being the lack of a `Copyable` capability. For an example of why +this might be misleading, consider what would happen if we expand +standard library collection types to support noncopyable elements. Value types +like `Array` and `Dictionary` would become copyable only when the elements they +contain are copyable. However, we cannot write this in terms of `NonCopyable` +conditional requirements, since if we write: + +```swift +extension Dictionary: NonCopyable where Key: NonCopyable, Value: NonCopyable {} +``` + +this says that the dictionary is noncopyable only when both the key and value +are noncopyable, which is wrong because we can't copy the dictionary even if +only the keys or only the values are noncopyable. If we flip the constraint to +`Copyable`, the correct thing would fall out naturally: + +```swift +extension Dictionary: Copyable where Key: Copyable, Value: Copyable {} +``` + +However, for progressive disclosure and source compatibility reasons, we still +want the majority of types to be `Copyable` by default without making them +explicitly declare it; noncopyable types are likely to remain the exception +rather than the rule, with automatic lifetime management via ARC by the +compiler being sufficient for most code like it is today. + +### English language bikeshedding + +Some dictionaries specify that "copiable" is the standard spelling for "able to +copy", although the Oxford English Dictionary and Merriam-Webster both also +list "copyable" as an accepted alternative. We prefer the more regular "copyable" +spelling. + +## Future directions + +### Noncopyable tuples + +It should be possible for a tuple to contain noncopyable elements, rendering +the tuple noncopyable if any of its elements are. Since tuples' structure is +always known, it would be reasonable to allow for the elements within a tuple +to be independently borrowed, mutated, and consumed, as the language allows +today for the elements of a tuple to be independently mutated via `inout` +accesses. (Due to the limitations of dynamic exclusivity checking, this would +not be possible for class properties, globals, and escaping closure captures.) + +### Noncopyable `Optional` + +This proposal initiates support for noncopyable types without any support for +generics at all, which precludes their use in most standard library types, +including `Optional`. We expect the lack of `Optional` support in particular +to be extremely limiting, since `Optional` can be used to manage dynamic +consumption of noncopyable values in situations where the language's static +rules cannot soundly support consumption. For instance, the static rules above +state that a stored property of a class can never be consumed, because it is +not knowable if other references to an object exist that expect the property +to be inhabited. This could be avoided using `Optional` with `mutating` +operation that forwards ownership of the `Optional` value's payload, if any, +writing `nil` back. Eventually this could be written as an extension method +on `Optional`: + +```swift +extension Optional where Self: ~Copyable { + mutating func take() -> Wrapped { + switch self { + case .some(let wrapped): + self = nil + return wrapped + case .none: + fatalError("trying to take from an Optional that's already empty") + } + } +} + +class Foo { + var fd: FileDescriptor? + + func close() { + // We normally would not be able to close `fd` except via the + // object's `deinit` destroying the stored property. But using + // `Optional` assignment, we can dynamically end the value's lifetime + // here. + fd = nil + } + + func takeFD() -> FileDescriptor { + // We normally would not be able to forward `fd`'s ownership to + // anyone else. But using + // `Optional.take`, we can dynamically end the value's lifetime + // here. + return fd.take() + } +} +``` + +Without `Optional` support, the alternative would be for every noncopyable type +to provide its own ad-hoc `nil`-like state, which would be very unfortunate, +and go against Swift's general desire to encourage structural code correctness +by making invalid states unrepresentable. Therefore, `Optional` is likely to +be worth considering as a special case for noncopyable support, ahead of full +generics support for noncopyable types. + +### Generics support for noncopyable types + +This proposal comes with an admittedly severe restriction that noncopyable types +cannot conform to protocols or be used at all as type arguments to generic +functions or types, including common standard library types like `Optional` +and `Array`. All generic parameters in Swift today carry an implicit assumption +that the type is copyable, and it is another large language design project to +integrate the concept of noncopyable types into the generics system. Full +integration will very likely also involve changes to the Swift runtime and +standard library to accommodate noncopyable types in APIs that weren't +originally designed for them, and this integration might then have backward +deployment restrictions. We believe that, even with these restrictions, +noncopyable types are a useful self-contained addition to the language for +safely and efficiently modeling unique resources, and this subset of the feature +also has the benefit of being adoptable without additional runtime requirements, +so developers can begin making use of the feature without giving up backward +compatibility with existing Swift runtime deployments. + +### Conditionally copyable types + +This proposal states that a type, including one with generic parameters, is +currently always copyable or always noncopyable. However, some types may +eventually be generic over copyable and non-copyable types, with the ability +to be copyable for some generic arguments but not all. A simple case might be +a tuple-like `Pair` struct: + +```swift +struct Pair: ~Copyable { + var first: T + var second: U +} +``` + +We will need a way to express this conditional copyability, perhaps using +conditional conformance style declarations: + +```swift +extension Pair: Copyable where T: Copyable, U: Copyable {} +``` + +### Suppressing implicitly derived conformances with `~Constraint` + +There are situations where a type's conformance to a protocol is implicitly +derived because of aspects of its declaration or usage. For instance, enums that +don't have any associated values are implicitly made `Hashable` (and, +by refinement, `Equatable`): + +```swift +enum Foo { + case a, b, c +} + +// OK to compare with `==` because `Foo` is automatically Equatable, +// through an implementation of `==` synthesized by the compiler for you. +print(Foo.a == Foo.b) +``` + +and internal structs and enums are implicitly `Sendable` if all of their +components are `Sendable`: + +```swift +struct Bar { + var x: Int, y: Int +} + +func foo() async { + let x = Bar(x: 17, y: 38) + + // OK to use x in an async task because it's implicitly Sendable + async let y = x +} +``` + +However, this isn't always desirable; an enum may want to reserve the right to +add associated values in the future that aren't `Equatable`, or a type may be +made up of `Sendable` components that represent resources that are not safe +to share across threads. There is currently no direct way to suppress these +automatically derived conformances. We propose to introduce the `~Constraint` +syntax as a way to explicitly suppress automatic derivation of a conformance +that would otherwise be performed for a declaration: + +```swift +enum Candy: ~Equatable { + case redVimes, twisslers, smickers +} + +// ERROR: `Candy` does not conform to `Equatable` +print(Candy.redVimes == Candy.twisslers) + +struct ThreadUnsafeHandle: ~Sendable { + // although this is an integer, it represents a system resource that + // can only be accessed from a specific thread, and should not be shared + // across threads + var handle: Int32 +} + +func foo(handle: ThreadUnsafeHandle) async { + // ERROR: `ThreadUnsafeHandle` is not `Sendable` + async let y = handle +} +``` + +It is important to note that `~Constraint` only avoids the implicit, automatic +derivation of conformance. It does **not** mean that the type strictly does +not conform to the protocol. Extensions may add the conformance back separately, +possibly conditionally: + +```swift +struct ResourceHandle: ~Sendable { + // although this is an integer, it represents a system resource that + // gives access to values of type `T`, which may not be thread safe + // across threads + var handle: Int32 +} + +// It is safe to share the handle when the resource type is thread safe +extension ResourceHandle: Sendable where T: Sendable {} + +// Suppress the implicit Equatable (and Hashable) derivation... +enum Candy: ~Equatable { + case redVimes, twisslers, smickers +} + +// ... and still add an Equatable conformance. +extension Candy: Equatable { + static func ==(a: Candy, b: Candy) -> Bool { + switch (a, b) { + // RedVimes are considered equal to Twisslers + case (.redVimes, .redVimes), (.twisslers, .twisslers), + (.smickers, .smickers), (.twisslers, .redVimes) + (.redVimes, .twisslers): + return true + default: + return false + } + } +} +``` + +Keep in mind that `~Constraint` is not required to suppress Swift's synthesized implementations of protocol requirements. For example, if you only want to +provide your own implementation of `==` for an enum, but are fine with Equatable +(and Hashable, etc) being derived for you, then the derivation of `Equatable` +already will use your version of `==`. + +```swift +enum Soda { + case mxPepper, drPibb, doogh + + // This is used instead of a synthesized `==` when + // implicitly deriving the Equatable conformance + static func ==(a: Soda, b: Soda) -> Bool { + switch (a, b) { + case (.doogh, .doogh): return true + case (_, .doogh), (.doogh, _): return false + default: return true + } + } +} +``` + +### Allowing `deinit` to mutate or consume `self`, while avoiding accidental recursion + +During destruction, `deinit` formally has sole ownership of `self`, so it +is possible to allow `deinit` to mutate or consume `self` as part of +deinitialization. However, inside of other `mutating` or `consuming` methods, +it's easy to inadvertently trigger implicit destruction of the value and +reenter `deinit` again: + +```swift +struct Foo: ~Copyable { + init() { ... } + + consuming func consumingHelper() { + // If a consuming method does nothing else, it will run `deinit` + } + + mutating func mutatingHelper() { + // A mutating method may consume and reassign self, indirectly triggering + // an implicit deinit + consumingHelper() + self = .init() + } + + deinit { + // mutatingHelper calls consumingHelper, which calls deinit again, leading to an infinite loop + mutatingHelper() + } +} +``` + +Since this is an easy trap to fall into, before we allow `deinit` to mutate +or consume `self`, it's worth considering whether there are any constraints we +could impose to make it less likely to get into an infinite +`deinit` loop situation when doing so. Some possibilities include: + +* We could say that the value remains immutable during `deinit`. Many types + don't need to modify their internal state for cleanup, especially if they + only store a pointer or handle to some resource. This seems overly + restrictive for other kinds of types that have direct ownership of resources, + though. +* We could say that individual *fields* of the value inside of `deinit` are + mutable and consumable, but that the value as a whole is not. This would + allow for `deinit` to individually mutate and/or forward ownership of + elements of the value, but not pass off the entire value to be mutated or + consumed (and potentially re-deinited). This would allow for `deinit`s to + implement logic that modifies or consumes part of the value, but they + wouldn't be allowed to use any methods of the type, other than maybe + `borrowing` methods, to share implementation logic with other members of the + type. +* Since `deinit` must be declared as part of the original type declaration, any + nongeneric methods that it can possibly call on the type must be defined in + the same module as the `deinit`, so we could potentially do some local + analysis of those methods. We could raise a warning or error if a method + called from the deinit either visibly contains any implicit deinit calls + itself, or cannot be analyzed because it's generic, from a protocol + extension, etc. +* We could do nothing and leave it in developers' hands to understand why + deinit loops happen when they do. + +### Finer-grained destructuring in `consuming` methods and `deinit` + +As currently specified, noncopyable types are (outside of `init` implementations) +always either fully initialized or fully destroyed, without any support +for incremental destruction even inside of `consuming` methods or deinits. A +`deinit` may modify, but not invalidate, `self`, and a `consuming` method may +`discard self`, forward ownership of all of `self`, or destroy `self`, but +cannot yet partially consume parts of `self`. This would be particularly useful +for types that contain other noncopyable types, which may want to relinquish +ownership of some or all of the resources owned by those members. In the +current proposal, this isn't possible without allowing for an intermediate +invalid state: + +```swift +struct SocketPair: ~Copyable { + let input, output: FileDescriptor + + // Gives up ownership of the output end, closing the input end + consuming func takeOutput() -> FileDescriptor { + // We would like to do something like this, taking ownership of + // `self.output` while leaving `self.input` to be destroyed. + // However, we can't do this without being able to either copy + // `self.output` or partially invalidate `self`. + return self.output + } +} +``` + +Analogously to how `init` implementations use a "definite initialization" +pass to allow the value to initialized field-by-field, we can implement the +inverse dataflow pass to allow `deinit` implementations to partially +invalidate `self`. This analysis would also enable `consuming` methods to +partially invalidate `self` in cases where either the type has no `deinit` or, +as discussed in the following section, `discard self` is used to disable the +`deinit` in cases when the value is partially invalidated. + +### Generalizing `discard self` for types with component cleanups + +The current proposal limits the use of `discard self` to types that don't have +any fields that require additional cleanup, meaning that it cannot be used in +a type that has class, generic, existential, or other noncopyable type fields. +Allowing this would be an obvious generalization; however, allowing it requires +answering some design questions: + +- When `self` is discarded, are its fields still destroyed? +- Is access to `self`'s fields still allowed after `discard self`? In other + words, does `discard self` immediately consume all of `self`, running + the cleanups for its elements at the point where the `discard` is executed, + or does it only disable the `deinit` on `self`, allowing the fields to + still be individually borrowed, mutated, and/or consumed, and leaving them + to be cleaned up when their individual lifetimes end? + +Although Rust's `mem::forget` completely leaks its operand, including its fields, +the authors of this proposal generally believe that is undesirable, so we expect +that `discard self` should only disable the type's own `deinit` while still +leaving the components of `self` to be cleaned up. + +The choice of what effect `discard` has on the lifetime of the fields affects +the observed order in which field deinits occurs, but also affects how code +would be expressed that performs destructuring or partial invalidation: + +```swift +struct SocketPair: ~Copyable { + let input, output: FileDescriptor + + deinit { ... } + + enum End { case input, output } + + // Give up ownership of one end and closes the other end + consuming func takeOneEnd(which: End) -> FileDescriptor { + // If a consuming method could partially invalidate self, would it do it + // like this... +#if discard_immediately_consumes_whats_left_of_self + switch which { + case .input: + // Move out the field we want + let result = self.input + // Destroy the rest of self + discard self + return result + + case .output: + let result = self.output + discard self + return result + } + + // ...or like this +#elseif discard_only_disables_deinit + // Disable deinit on self, which subsequently allows individual consumption + // of its fields + discard self + + switch which { + case .input: + return self.input + case .output: + return self.output + } +#endif + } +} +``` + +### `read` and `modify` accessor coroutines for computed properties + +The current computed property model allows for properties to provide a getter, +which returns the value of the property on read to the caller as an owned value, +and optionally a setter, which receives the `newValue` of the property as +a parameter with which to update the containing type's state. This is +sometimes inefficient for value types, since the get/set pattern requires +returning a copy, modifying the copy, then passing the copy back to the setter +in order to model an in-place update, but it also limits what computed +properties can express for noncopyable types. Because a getter has to return +by value, it cannot pass along the value of a stored noncopyable property +without also destroying the enclosing aggregate, so `get`/`set` cannot be used +to wrap logic around access to a stored noncopyable property. + +The Swift stable ABI for properties internally uses **accessor coroutines** +to allow for efficient access to stored properties, while still providing +abstraction that allows library evolution to change stored properties into +computed and back. These coroutines **yield** access to a value in-place for +borrowing or mutating, instead of passing copies of values back and forth. +We can expose the ability for code to implement these coroutines directly, +which is a good optimization for copyable value types, but also allows for +more expressivity with noncopyable properties. + +### Static casts of functions with ownership modifiers + +The rule for casting function values via `as` or some other static, implicit +coercion is that a noncopyable parameter's ownership modifier must remain the +same. But there are some cases where static conversions of functions +with noncopyable parameters are safe. It's not safe in general to do any dynamic +casts of function values, so `as?` and `as!` are excluded. + +One reason behind the currently restrictive rule for static casts is a matter of +scope for this proposal. There may be a broader demand to support such casts +even for copyable types. For example, it should be safe to allow a cast to +change a `borrowing` parameter into one that is `inout`, as it only adds a +capability (mutation) that is not actually used by the underlying function: +```swift +// This could be possible, but currently is not. +{ (x: borrowing SomeType) in () } as (inout SomeType) -> () +``` +The second reason is that some casts are _only_ valid for copyable types. +In particular, a cast that changes a `consuming` parameter into one that is +`borrowing` is only valid for copyable types, because a copy of the borrowed +value is required to provide a non-borrowed value to the underlying function. +```swift +// String is copyable, so both are OK and currently permitted. +{ (x: borrowing String) in () } as (consuming String) -> () +{ (x: consuming String) in () } as (borrowing String) -> () +// FileDescriptor is noncopyable, so it cannot go from consuming to borrowing: +{ (x: consuming FileDescriptor) in () } as (borrowing String) -> () +// but the reverse could be permitted in the future: +{ (x: borrowing FileDescriptor) in () } as (consuming String) -> () +``` + +## Revision history + +This revision makes the following changes from the [second reviewed revision](https://github.com/swiftlang/swift-evolution/blob/a9e21e3a4eb9526f998915c6554c7c72e5885a91/proposals/0390-noncopyable-structs-and-enums.md) +in response to Language Steering Group review and implementation experience: + +- `_ = x` is now a borrowing operation. +- `switch` and `if/while case` require the subject of a pattern match to use + the `consume x` operator. The fact that they are consuming operations now + is an artifact of the implementation, and with further development, we may + want to make the default semantics of `switch x` without explicit consumption + to be borrowing. +- Escaped closure captures are constrained from being consumed for their + entire lifetime, even before the closure that escapes it is formed. This + analysis was not practical to implement using our current analysis, and the + added expressivity is unlikely to be worth the implementation complexity. +- `self` in a deinit is currently constrained to be immutable, since there + is [ongoing discussion](https://forums.swift.org/t/se-0390-noncopyable-type-deinit-s-mutation-and-accidental-recursion/64767) + about how best to manage mutation or consumption during deinits while + managing the possibility to accidentally cause recursion into `deinit` + by implicit destruction. + +The [second reviewed revision](https://github.com/swiftlang/swift-evolution/blob/a9e21e3a4eb9526f998915c6554c7c72e5885a91/proposals/0390-noncopyable-structs-and-enums.md) +of the proposal made the following changes from the +[first reviewed revision](https://github.com/swiftlang/swift-evolution/blob/5d075b86d57e3436b223199bd314b2642e30045f/proposals/0390-noncopyable-structs-and-enums.md): + +- The original revision did not provide a `Copyable` generic constraint, and + declared types as noncopyable using a `@noncopyable` attribute. The + language workgroup believes that it is a good idea to build toward a future + where noncopyable types are integrated with the language's generics system, + and that the syntax for suppressing generic constraints is a good general + notation to have for suppressing implicit conformances or assumptions about + generic capabilities we may take away in the future, so it makes sense to + provide a syntax that allows for growth in those directions. + +- The original revision suppressed implicit `deinit` within methods using the + spelling `forget self`. Although the term `forget` has a precedent in Rust, + the behavior of `mem::forget` in Rust doesn't correspond to the semantics of + the operation proposed here, and the language workgroup doesn't find the + term clear enough on its own. This revision of the proposal chooses the + `discard` as a starting point for further review discussion. Furthermore, + we limit its use to types whose contents are otherwise trivial, in order to + avoid committing to interactions with elementwise consumption of fields + that we may want to refine later. + +- The original revision allowed for a `consuming` method declared anywhere in + the type's original module to suppress `deinit`. This revision narrows the + capability to only methods declared in the same file as the type, for + consistency with other language features that depend on having visibility into + a type's entire layout, such as implicit `Sendable` inference. diff --git a/proposals/0391-package-registry-publish.md b/proposals/0391-package-registry-publish.md new file mode 100644 index 0000000000..6aba3942c2 --- /dev/null +++ b/proposals/0391-package-registry-publish.md @@ -0,0 +1,705 @@ +# Package Registry Publish + +* Proposal: [SE-0391](0391-package-registry-publish.md) +* Author: [Yim Lee](https://github.com/yim-lee) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.9)** +* Implementation: + * [apple/swift-package-manager#6101](https://github.com/apple/swift-package-manager/pull/6101) + * [apple/swift-package-manager#6146](https://github.com/apple/swift-package-manager/pull/6146) + * [apple/swift-package-manager#6159](https://github.com/apple/swift-package-manager/pull/6159) + * [apple/swift-package-manager#6169](https://github.com/apple/swift-package-manager/pull/6169) + * [apple/swift-package-manager#6188](https://github.com/apple/swift-package-manager/pull/6188) + * [apple/swift-package-manager#6189](https://github.com/apple/swift-package-manager/pull/6189) + * [apple/swift-package-manager#6215](https://github.com/apple/swift-package-manager/pull/6215) + * [apple/swift-package-manager#6217](https://github.com/apple/swift-package-manager/pull/6217) + * [apple/swift-package-manager#6220](https://github.com/apple/swift-package-manager/pull/6220) + * [apple/swift-package-manager#6229](https://github.com/apple/swift-package-manager/pull/6229) + * [apple/swift-package-manager#6237](https://github.com/apple/swift-package-manager/pull/6237) +* Review: ([pitch](https://forums.swift.org/t/pitch-package-registry-publish/62828)), ([review](https://forums.swift.org/t/se-0391-package-registry-publish/63405)), ([acceptance](https://forums.swift.org/t/accepted-se-0391-swift-package-registry-authentication/64088)) + +## Introduction + +A package registry makes packages available to consumers. Starting with Swift 5.7, +SwiftPM supports dependency resolution and package download using any registry that +implements the [service specification](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md) proposed alongside with [SE-0292](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0292-package-registry-service.md). +SwiftPM does not yet provide any tooling for publishing packages, so package authors +must manually prepare the contents (e.g., source archive) and interact +with the registry on their own to publish a package release. This proposal +aims to standardize package publishing such that SwiftPM can offer a complete and +well-rounded experience for using package registries. + +## Motivation + +Publishing package release to a Swift package registry generally involves these steps: + 1. Gather package release metadata. + 1. Prepare package source archive by using the [`swift package archive-source` subcommand](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0292-package-registry-service.md#archive-source-subcommand). + 1. Sign the metadata and archive (if needed). + 1. [Authenticate](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0378-package-registry-auth.md) (if required by the registry). + 1. Send the archive and metadata (and their signatures if any) by calling the ["create a package release" API](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-6). + 1. Check registry server response to determine if publication has succeeded or failed (if the registry processes request synchronously), or is pending (if the registry processes request asynchronously). + +SwiftPM can streamline the workflow by combining all of these steps into a single +`publish` command. + +## Proposed solution + +We propose to introduce a new `swift package-registry publish` subcommand to SwiftPM +as well as standardization on package release metadata and package signing to ensure a +consistent user experience for publishing packages. + +## Detailed design + +### Package release metadata + +Typically a package release has metadata associated with it, such as URL of the source +code repository, license, etc. In general, metadata gets set when a package release is +being published, but a registry service may allow modifications of the metadata afterwards. + +The current [registry service specification](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md) states that: + - A client (e.g., package author, publishing tool) may provide metadata for a package release by including it in the ["create a package release" request](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#462-package-release-metadata). The registry server will store the metadata and include it in the ["fetch information about a package release" response](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-2). + - If a client does not include metadata, the registry server may populate it unless the client specifies otherwise (i.e., by sending an empty JSON object `{}` in the "create a package release" request). + +It does not, however, define any requirements or server-client API contract on the +metadata contents. We would like to change that by proposing the following: + - Package release metadata will continue to be sent as a JSON object. + - Package release metadata must adhere to the [schema](#package-release-metadata-standards). + - Package release metadata will continue to be included in the "create a package release" request as a multipart section named `metadata` in the request body. + - Registry server may allow and/or populate additional metadata by expanding the schema, but it must not alter any of the predefined properties. + - Registry server may make any properties in the schema and additional metadata it defines required. Registry server may fail the "create a package release" request if any required metadata is missing. + - Client cannot change how registry server handles package release metadata. In other words, client will no longer be able to instruct registry server not to populate metadata by sending an empty JSON object `{}`. + - Registry server will continue to include metadata in the "fetch information about a package release" response. + +#### Package release metadata standards + +Package release metadata submitted to a registry must be a JSON object of type +[`PackageRelease`](#packagerelease-type), the schema of which is defined below. + +
+ +Expand to view JSON schema + +```json +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md", + "title": "Package Release Metadata", + "description": "Metadata of a package release.", + "type": "object", + "properties": { + "author": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "Name of the author." + }, + "email": { + "type": "string", + "description": "Email address of the author." + }, + "description": { + "type": "string", + "description": "A description of the author." + }, + "organization": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "Name of the organization." + }, + "email": { + "type": "string", + "description": "Email address of the organization." + }, + "description": { + "type": "string", + "description": "A description of the organization." + }, + "url": { + "type": "string", + "description": "URL of the organization." + }, + }, + "required": ["name"] + }, + "url": { + "type": "string", + "description": "URL of the author." + }, + }, + "required": ["name"] + }, + "description": { + "type": "string", + "description": "A description of the package release." + }, + "licenseURL": { + "type": "string", + "description": "URL of the package release's license document." + }, + "readmeURL": { + "type": "string", + "description": "URL of the README specifically for the package release or broadly for the package." + }, + "repositoryURLs": { + "type": "array", + "description": "Code repository URL(s) of the package release.", + "items": { + "type": "string", + "description": "Code repository URL." + } + } + } +} +``` + +
+ +##### `PackageRelease` type + +| Property | Type | Description | Required | +| ----------------- | :-----------------: | ------------------------------------------------ | :------: | +| `author` | [Author](#author-type) | Author of the package release. | | +| `description` | String | A description of the package release. | | +| `licenseURL` | String | URL of the package release's license document. | | +| `readmeURL` | String | URL of the README specifically for the package release or broadly for the package. | | +| `repositoryURLs` | Array | Code repository URL(s) of the package. It is recommended to include all URL variations (e.g., SSH, HTTPS) for the same repository. This can be an empty array if the package does not have source control representation.
Setting this property is one way through which a registry can obtain repository URL to package identifier mappings for the ["lookup package identifiers registered for a URL" API](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#45-lookup-package-identifiers-registered-for-a-url). A registry may choose other mechanism(s) for package authors to specify such mappings. | | + +##### `Author` type + +| Property | Type | Description | Required | +| ----------------- | :-----------------: | ------------------------------------------------ | :------: | +| `name` | String | Name of the author. | ✓ | +| `email` | String | Email address of the author. | | +| `description` | String | A description of the author. | | +| `organization` | [Organization](#organization-type) | Organization that the author belongs to. | | +| `url` | String | URL of the author. | | + +##### `Organization` type + +| Property | Type | Description | Required | +| ----------------- | :-----------------: | ------------------------------------------------ | :------: | +| `name` | String | Name of the organization. | ✓ | +| `email` | String | Email address of the organization. | | +| `description` | String | A description of the organization. | | +| `url` | String | URL of the organization. | | + +### Package signing + +A registry may require packages to be signed. In order for SwiftPM to be able to +download and handle signed packages from a registry, we propose to standardize +package signature format and establish server-client API contract on package +signing. + +#### Package signature + +Package signature format will be identified by the underlying standard/technology +(e.g., Cryptographic Message Syntax (CMS), JSON Web Signature (JWS), etc.) and +version number. In the initial release, all signatures will be in [CMS](https://www.rfc-editor.org/rfc/rfc5652.html). + +| Signature format ID | Description | +| ------------------- | --------------------------------------------------------- | +| `cms-1.0.0` | Version 1.0.0 of package signature in CMS | + +##### Package signature format `cms-1.0.0` + +Package signature format `cms-1.0.0` uses CMS. + +| CMS Attribute | Details | +| ------------------------------ | --------------------------------------------------------- | +| Content type | [Signed-Data](https://www.rfc-editor.org/rfc/rfc5652.html#section-5) | +| Encapsulated data | The content being signed ([`EncapsulatedContentInfo.eContent`](https://www.rfc-editor.org/rfc/rfc5652.html#section-5.2)) is omitted since we are constructing an external signature. | +| Message digest algorithm | SHA-256, computed on the package source archive. | +| Signature algorithm | ECDSA P-256 | +| Number of signatures | 1 | +| Certificate | Certificate that contains the signing key. It is up to the registry to define the certificate policy (e.g., trusted root(s)). | + +The signature, represented in CMS, will be included as part +of the "create a package release" API request. + +A registry receiving such signed package will: + - Check if the signature format (`cms-1.0.0`) is accepted. + - Validate the signature is well-formed according to the signature format. + - Validate the certificate chain meets registry policy. + - Extract public key from the certificate and use it to verify the signature. + +Then the registry will process the package and save it for client downloads if publishing is successful. + +The registry must include signature information in the "fetch information about a package release" API +response to indicate the package is signed and the signature format (`cms-1.0.0`). + +After downloading a signed package SwiftPM will: + - Check if the signature format (`cms-1.0.0`) is supported. + - Validate the signature is well-formed according to the signature format. + - Validate that the signed package complies with the locally-configured signing policy. + - Extract public key from the certificate and use it to verify the signature. + +#### Server-side requirements for package signing + +A registry that requires package signing should provide documentations +on the signing requirements (e.g., any requirements for certificates +used in signing). + +A registry must also modify the ["create package release" API](#create-package-release-api) to allow +signature in the request, as well as the response for the ["fetch package release metadata"](#fetch-package-release-metadata-api) +and ["download package source archive"](#download-package-source-archive-api) API to include signature information. + +#### SwiftPM's handling of registry packages + +##### SwiftPM configuration + +Users will be able to configure how SwiftPM handles packages downloaded from a +registry. In the user-level `registries.json` file, which by default is located at +`~/.swiftpm/configuration/registries.json`, we will introduce a new `security` key: + +```json5 +{ + "security": { + "default": { + "signing": { + "onUnsigned": "prompt", // One of: "error", "prompt", "warn", "silentAllow" + "onUntrustedCertificate": "prompt", // One of: "error", "prompt", "warn", "silentAllow" + "trustedRootCertificatesPath": "~/.swiftpm/security/trusted-root-certs/", + "includeDefaultTrustedRootCertificates": true, + "validationChecks": { + "certificateExpiration": "disabled", // One of: "enabled", "disabled" + "certificateRevocation": "disabled" // One of: "strict", "allowSoftFail", "disabled" + } + } + }, + "registryOverrides": { + // The example shows all configuration overridable at registry level + "packages.example.com": { + "signing": { + "onUnsigned": "warn", + "onUntrustedCertificate": "warn", + "trustedRootCertificatesPath": , + "includeDefaultTrustedRootCertificates": , + "validationChecks": { + "certificateExpiration": "enabled", + "certificateRevocation": "allowSoftFail" + } + } + } + }, + "scopeOverrides": { + // The example shows all configuration overridable at scope level + "mona": { + "signing": { + "trustedRootCertificatesPath": , + "includeDefaultTrustedRootCertificates": + } + } + }, + "packageOverrides": { + // The example shows all configuration overridable at package level + "mona.LinkedList": { + "signing": { + "trustedRootCertificatesPath": , + "includeDefaultTrustedRootCertificates": + } + } + } + }, + ... +} +``` + +Security configuration for a package is computed using values from +the following (in descending precedence): +1. `packageOverrides` (if any) +1. `scopeOverrides` (if any) +1. `registryOverrides` (if any) +1. `default` + +The `default` JSON object contains all configurable security options +and their default value when there is no override. + +- `signing.onUnsigned`: Indicates how SwiftPM will handle an unsigned package. + + | Option | Description | + | ------------- | --------------------------------------------------------- | + | `error` | SwiftPM will reject the package and fail the build. | + | `prompt` | SwiftPM will prompt user to see if the unsigned package should be allowed.
  • If no, SwiftPM will reject the package and fail the build.
  • If yes and the package has never been downloaded, its checksum will be stored for [local TOFU](#local-tofu). Otherwise, if the package has been downloaded before, its checksum must match the previous value or else SwiftPM will reject the package and fail the build.
SwiftPM will record user's response to prevent repetitive prompting. | + | `warn` | SwiftPM will not prompt user but will emit a warning before proceeding. | + | `silentAllow` | SwiftPM will allow the unsigned package without prompting user or emitting warning. | + +- `signing.onUntrustedCertificate`: Indicates how SwiftPM will handle a package signed with an [untrusted certificate](#trusted-vs-untrusted-certificate). + + | Option | Description | + | ------------- | --------------------------------------------------------- | + | `error` | SwiftPM will reject the package and fail the build. | + | `prompt` | SwiftPM will prompt user to see if the package signed with an untrusted certificate should be allowed.
  • If no, SwiftPM will reject the package and fail the build.
  • If yes, SwiftPM will proceed with the package as if it were an unsigned package.
SwiftPM will record user's response to prevent repetitive prompting. | + | `warn` | SwiftPM will not prompt user but will emit a warning before proceeding. | + | `silentAllow` | SwiftPM will allow the package signed with an untrusted certificate without prompting user or emitting warning. | + +- `signing.trustedRootCertificatesPath`: Absolute path to the directory containing custom trusted roots. SwiftPM will include these roots in its [trust store](#trusted-vs-untrusted-certificate), and certificates used for package signing must chain to roots found in this store. This configuration allows override at the package, scope, and registry levels. +- `signing.includeDefaultTrustedRootCertificates`: Indicates if SwiftPM should include default trusted roots in its [trust store](#trusted-vs-untrusted-certificate). This configuration allows override at the package, scope, and registry levels. +- `signing.validationChecks`: Validation check settings for the package signature. + + | Validation | Description | + | ------------------------ | --------------------------------------------------------------- | + | `certificateExpiration` |
  • `enabled`: SwiftPM will check that the current timestamp when downloading falls within the signing certificate's validity period. If it doesn't, SwiftPM will reject the package and fail the build.
  • `disabled`: SwiftPM will not perform this check.
| + | `certificateRevocation` | With the exception of `disabled`, SwiftPM will check revocation status of the signing certificate. SwiftPM will only support revocation check done through [OCSP](https://www.rfc-editor.org/rfc/rfc6960) in the first feature release.
  • `strict`: Revocation check must complete successfully and the certificate must be in good status. SwiftPM will reject the package and fail the build if the revocation status is revoked or unknown (including revocation check not supported or failed).
  • `allowSoftFail`: SwiftPM will reject the package and fail the build iff the certificate has been revoked. SwiftPM will allow the certificate's revocation status to be unknown (including revocation check not supported or failed).
  • `disabled`: SwiftPM will not perform this check.
| + +##### Trusted vs. untrusted certificate + +A certificate is **trusted** if it is chained to any roots in SwiftPM's +trust store, which is a combination of: + - SwiftPM's default trust store, if `signing.includeDefaultTrustedRootCertificates` is `true`. + - Custom root(s) in the configured trusted roots directory at `signing.trustedRootCertificatesPath`. + +Otherwise, a certificate is **untrusted** and handled according to the `signing.onUntrustedCertificate` setting. + +Both `signing.includeDefaultTrustedRootCertificates` and `signing.trustedRootCertificatesPath` +support multiple levels of overrides. SwiftPM will choose the configuration value that has the +highest specificity. + + +For example, when evaluating the value of `signing.includeDefaultTrustedRootCertificates` +or `signing.trustedRootCertificatesPath` for package `mona.LinkedList`: + 1. SwiftPM will use the package override from `packageOverrides` (i.e., `packageOverrides["mona.LinkedList"]`), if any. + 1. Otherwise, SwiftPM will use the scope override from `scopeOverrides` (i.e., `scopeOverrides["mona"]`), if any. + 1. Next, depending on the registry the package is downloaded from, SwiftPM will look for and use the registry override in `registryOverrides`, if any. + 1. Finally, if no override is found, SwiftPM will use the value from `default`. + +##### Local TOFU + +When SwiftPM downloads a package release from registry via the +["download source archive" API](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-4), it will: + 1. Search local fingerprints storage, which by default is located at `~/.swiftpm/security/fingerprints/`, to see if the package release has been downloaded before and its recorded checksum. The checksum of the downloaded source archive must match the previous value or else [trust on first use (TOFU)](https://en.wikipedia.org/wiki/Trust_on_first_use) check would fail. + 1. Fetch package release metadata from the registry to get: +
    +
  • Checksum for TOFU if the package release is downloaded for the first time.
  • +
  • Signature information if the package release is signed.
  • +
+ 1. Retrieve security settings from the user-level `registries.json`. + 1. Check if the package is allowed based on security settings. + 1. Validate the signature according to the signature format if package is signed. + 1. Some certificates allow SwiftPM to extract additional information that can drive additional security features. For packages signed with these certificates, SwiftPM will apply additional, publisher-level TOFU by extracting signing identity from the certificate and enforcing the same signing identity across all signed versions of a package. + +### New `package-registry publish` subcommand + +The new `package-registry publish` subcommand will create a package +source archive, sign it if needed, and publish it to a registry. + +```manpage +> swift package-registry publish --help +OVERVIEW: Publish a package release to registry + +USAGE: package-registry publish + +ARGUMENTS: + The package identifier. + The package release version being created. + +OPTIONS: + --url The registry URL. + --scratch-directory The path of the directory where working file(s) will be written. + + --metadata-path The path to the package metadata JSON file if it's not 'package-metadata.json' in the package directory. + + --signing-identity The label of the signing identity to be retrieved from the system's secrets store if supported. + + --private-key-path The path to the certificate's PKCS#8 private key (DER-encoded). + --cert-chain-paths Path(s) to the signing certificate (DER-encoded) and optionally the rest of the certificate chain. The signing certificate must be listed first. + + --dry-run Dry run only; prepare the archive and sign it but do not publish to the registry. +``` + +- `id`: The package identifier in the `.` notation as defined in [SE-0292](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0292-package-registry-service.md#package-identity). It is the package author's responsibility to register the package identifier with the registry beforehand. +- `version`: The package release version in [SemVer 2.0](https://semver.org) notation. +- `url`: The URL of the registry to publish to. SwiftPM will try to determine the registry URL by searching for a scope-to-registry mapping or use the `[default]` URL in `registries.json`. The command will fail if this value is missing. +- `scratch-directory`: The path of the working directory. SwiftPM will write to the package directory by default. + +The following may be required depending on registry support and/or requirements: + - `metadata-path`: The path to the JSON file containing [package release metadata](#package-release-metadata). By default, SwiftPM will look for a file named `package-metadata.json` in the package directory if this is not specified. SwiftPM will include the content of the metadata file in the request body if present. If the package source archive is being signed, the metadata will be signed as well. + - `signing-identity`: The label that identifies the signing identity to use for package signing in the system's secrets store if supported. + - `private-key-path`: Required for package signing unless `signing-identity` is specified, this is the path to the private key used for signing. + - `cert-chain-paths`: Required for package signing unless `signing-identity` is specified, this is the signing certificate chain. + +A signing identity encompasses a private key and a certificate. On +systems where it is supported, SwiftPM can look for a signing identity +using the query string given via the `--signing-identity` option. This +feature will be available on macOS through Keychain in the initial +release, so a certificate and its private key can be located by the +certificate label alone. + +Otherwise, both `--private-key-path` and `--cert-chain-paths` must be +provided to locate the signing key and certificate chain. + +SwiftPM will sign the package source archive and package release metadata if `signing-identity` +or both `private-key-path` and `cert-chain-paths` are set. + +All signatures in the initial release will be in the [`cms-1.0.0`](#package-signature-format-cms-100) format. + +Using these inputs, SwiftPM will: + - Generate source archive for the package release. + - Sign the source archive and metadata if the required parameters are provided. + - Make HTTP request to the "create a package release" API. + - Check server response for any errors. + +Prerequisites: +- Run [`swift package-registry login`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0378-package-registry-auth.md#new-login-subcommand) to authenticate registry user if needed. +- The user has the necessary permissions to call the ["create a package release" API](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-6) for the package identifier. + +### Changes to the registry service specification + +#### Create package release API + +A registry must update [this existing endpoint](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-6) to handle package release +metadata as described in a [previous section](#package-release-metadata) of this document. + +If the package being published is signed, the client must identify the signature format +in the `X-Swift-Package-Signature-Format` HTTP request header so that the +server can process the signature accordingly. + +Signatures of the source-archive and metadata are sent as part of the request body +(`source-archive-signature` and `metadata-signature`, respectively): + +``` +PUT /mona/LinkedList?version=1.1.1 HTTP/1.1 +Host: packages.example.com +Accept: application/vnd.swift.registry.v1+json +Content-Type: multipart/form-data;boundary="boundary" +Content-Length: 336 +Expect: 100-continue +X-Swift-Package-Signature-Format: cms-1.0.0 + +--boundary +Content-Disposition: form-data; name="source-archive" +Content-Type: application/zip +Content-Length: 32 +Content-Transfer-Encoding: base64 + +gHUFBgAAAAAAAAAAAAAAAAAAAAAAAA== + +--boundary +Content-Disposition: form-data; name="source-archive-signature" +Content-Type: application/octet-stream +Content-Length: 88 +Content-Transfer-Encoding: base64 + +l1TdTeIuGdNsO1FQ0ptD64F5nSSOsQ5WzhM6/7KsHRuLHfTsggnyIWr0DxMcBj5F40zfplwntXAgS0ynlqvlFw== + +--boundary +Content-Disposition: form-data; name="metadata" +Content-Type: application/json +Content-Transfer-Encoding: quoted-printable +Content-Length: 25 + +{ "repositoryURLs": [] } + +--boundary +Content-Disposition: form-data; name="metadata-signature" +Content-Type: application/octet-stream +Content-Length: 88 +Content-Transfer-Encoding: base64 + +M6TdTeIuGdNsO1FQ0ptD64F5nSSOsQ5WzhM6/7KsHRuLHfTsggnyIWr0DxMcBj5F40zfplwntXAgS0ynlqvlFw== +``` + +#### Fetch package release metadata API + +A registry may update [this existing endpoint](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-2) for the [metadata changes](#package-release-metadata) +described in this document. + +If the package release is signed, the registry must include a `signing` JSON +object in the response: + +```json5 +{ + "id": "mona.LinkedList", + "version": "1.1.1", + "resources": [ + { + "name": "source-archive", + "type": "application/zip", + "checksum": "a2ac54cf25fbc1ad0028f03f0aa4b96833b83bb05a14e510892bb27dea4dc812", + "signing": { + "signatureBase64Encoded": "l1TdTeIuGdNsO1FQ0ptD64F5nSSOsQ5WzhM6/7KsHRuLHfTsggnyIWr0DxMcBj5F40zfplwntXAgS0ynlqvlFw==", + "signatureFormat": "cms-1.0.0" + } + } + ], + "metadata": { ... } +} +``` + +#### Download package source archive API + +If a registry supports signing, it must update [this existing endpoint](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-4) +to include the `X-Swift-Package-Signature-Format` and `X-Swift-Package-Signature` headers in +the HTTP response for a signed package source archive. + +``` +HTTP/1.1 200 OK +Accept-Ranges: bytes +Cache-Control: public, immutable +Content-Type: application/zip +Content-Disposition: attachment; filename="LinkedList-1.1.1.zip" +Content-Length: 2048 +Content-Version: 1 +Digest: sha-256=oqxUzyX7wa0AKPA/CqS5aDO4O7BaFOUQiSuyfepNyBI= +Link: ; rel=duplicate; geo=jp; pri=10; type="application/zip" +X-Swift-Package-Signature-Format: cms-1.0.0 +X-Swift-Package-Signature: l1TdTeIuGdNsO1FQ0ptD64F5nSSOsQ5WzhM6/7KsHRuLHfTsggnyIWr0DxMcBj5F40zfplwntXAgS0ynlqvlFw== +``` + +## Security + +This proposal introduces the framework for package signing, allowing package +authors the ability to provide additional authenticity guarantees by signing their +source archives before publishing them to the registry. Package users will be +able to control the kind(s) of packages they trust by specifying a local validation +policy. This can include a trust on first use approach, or by validating against a +pre-configured set of trusted roots. + +While this proposal introduces the package signature format, it does not validate +that a package is published by a specific entity. Instead, it validates that a package +is published by an entity who can obtain a signing certificate that meets the +requirements defined by the registry, which could be anybody. As such, it does +not provide any protection against malware, and it would be wrong to assumed that +signed packages can be trusted unconditionally. + +In this proposal, package signing is primarily intended to provide additional security +controls for package registries. By requiring packages be signed, and by the +registry limiting what keys or identities are allowed to publish packages, a registry +can provide additional security in the event the package author's registry credentials +are compromised. + +Although this proposal introduces policy controls for package users, they are limited +in scope, and do not yet allow SwiftPM to validate that multiple package versions are +from the same entity--recording signing identities for [TOFU](#local-tofu) provides some +protection against a compromised registry, but it is not for all packages and +[more work needs to be done](#local-signing-identity-checks) before it can be so. As such, SwiftPM continues to trust +the registry to provide authentic packages and accurate information about the +signature status of the package. + +### Privacy implications of certificate revocation check + +Revocation checking via OCSP implicitly discloses to the certificate +authority and anyone on the network the packages that a user may be +downloading. If this is a concern, revocation check can be disabled +in [SwiftPM configuration](#swiftpm-configuration). + +## Impact on existing packages + +Current packages won't be affected by changes in this proposal. + +## Alternatives considered + +### Signing package source archive vs. manifest + +A package manifest is a reference list of files that are present in the +source archive. We considered an approach where SwiftPM would produce +such manifest, sign the manifest instead of the source archive, +then create a new archive containing the source archive, manifest, and +signature file. This way the archive and its signature can be distributed +by the registry as a single file. + +However, given the potential complications with extracting files from the +archive and verifying manifest contents, moreover there is no restriction +that would require single-file download (i.e., SwiftPM can download the +source archive and signature separately), we have decided to take the approach +covered in previous sections of this proposal. + +### Use key in certificate as signing identity for local publisher-level TOFU + +We considered using the key in a certificate as signing identity for +[local publisher-level TOFU](#local-tofu) (i.e., different versions of a package must +have the same signing identity). However, since key can change easily +(e.g., lost key, key rotation, etc.), all users of the package must reset data +used for local TOFU each time or else TOFU check would fail, which can introduce +significant overhead and confusion. + +## Future directions + +### Support encrypted private keys + +Private keys are encrypted typically. SwiftPM commands that have private key +as input, such as `package sign` and `package-registry publish`, should support +reading encrypted private key. This could mean modifying the command to prompt +user for the passphrase if needed, and adding a `--private-key-passphrase` +option to the command for non-interactive/automation use-cases. + +### Auto-populate package release metadata + +Parts of the [package release metadata](#package-release-metadata-standards) can be populated by SwiftPM using +information found in the package directory. The auto-generated metadata can +serve as a default or starting point which package authors may optionally edit, +and ensure every package release to have metadata. + +### Support additional certificate revocation checks + +SwiftPM may support alternative mechanisms to check revocation besides OCSP. + +### Local signing identity checks + +In the current proposal, signing identity is left for the package registry to define, implement, and +enforce at publication time. However, this requires SwiftPM to rely on the package +registry to correctly implement these checks, and a compromise of the registry, or +SwiftPM's connection to the registry, would potentially allow for unauthorized packages +to be published. Performing additional checks in SwiftPM can mitigate this risk, but +requires defining a consistent identity that can be extracted and relied upon, and +determining how those identities are provisioned and authorized. + +A future Swift evolution proposal can provide specification of a certificate +from which signing identity can be extracted, such that more certificates can +be used for [local publisher-level TOFU](#local-tofu), which provides an extra layer of +trust on top of checksum TOFU done at the package release level. + +### Timestamping and Countersignatures + +In the proposed implementation, the signing certificate associated with +the package may expire, and this can prevent SwiftPM from validating +the information and revocation status of the certificate. Using +approaches such as [Time Stamping Authority](https://www.rfc-editor.org/rfc/rfc3161) or having the registry +itself perform a [countersignature](https://www.rfc-editor.org/rfc/rfc5652#section-11.4), information about when a +package was first published can be provided, +even after the signing certificate has expired. This can avoid the need +for package authors to re-sign packages when the signing certificate +expires. + +### Transitive trust + +SwiftPM's TOFU mitigation could be further improved by +including checksum and signing identity in `Package.resolved` +(or another similar file), which then gets included in the package content. +Including such security metadata would allow distributing information about +direct and transitive dependencies across the ecosystem much faster than a +local-only TOFU without requiring a centralized database/service to vend +this information. + +```json5 +{ + "pins": [ + { + "identity": "mona.LinkedList", + "kind": "registry", + "location": "https://packages.example.com/mona/LinkedList", + "state": { + "checksum": "ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d", + "version": "0.12.0" + }, + "signingBy": { + "identityType": , + "name": , + ... + } + }, + { + "identity": "Foo", + "kind": "remoteSourceControl", + "location": "https://github.com/something/Foo.git", + "state": { + "revision": "90a9574276f0fd17f02f58979423c3fd4d73b59e", + "version": "1.0.2", + } + } + ], + "version": 2 +} +``` diff --git a/proposals/0392-custom-actor-executors.md b/proposals/0392-custom-actor-executors.md new file mode 100644 index 0000000000..13497cff8d --- /dev/null +++ b/proposals/0392-custom-actor-executors.md @@ -0,0 +1,993 @@ +# Custom Actor Executors + +* Proposal: [SE-0392](0392-custom-actor-executors.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [John McCall](https://github.com/rjmccall), [Kavon Farvardin](https://github.com/kavon) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.9)** +* Previous threads: + - Original pitch thread from around Swift 5.5: [Support custom executors in Swift Concurrency](https://forums.swift.org/t/support-custom-executors-in-swift-concurrency/44425) + - Original "assume..." proposal which was subsumed into this proposal, as it relates closely to asserting on executors: [Pitch: Unsafe Assume on MainActor](https://forums.swift.org/t/pitch-unsafe-assume-on-mainactor/63074/) +* Reviews: + - First review thread: https://forums.swift.org/t/returned-for-revision-se-0392-custom-actor-executors/64172 + - Revisions: + - Rename `Job` to `ExecutorJob`, making it less likely to conflict with existing type names, and typealias `UnownedJob` with `UnownedExecutorJob` (however the old type remains for backwards compatibility). + - Move assert/precondition/assume APIs to extensions on actor types, e.g. `Actor/assertIsolated`, `DistributedActor/preconditionIsolated`, `MainActor/assumeIsolated { ... }` + - Distributed actor executor customization `unownedExecutor` invoked on a remote distributed actor, to return an executor that fatal errors only once attempts are made to enqueue work onto it, rather than crashing immediately upon attempting to obtain the executor. + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Proposed solution](#proposed-solution) +* [Detailed design](#detailed-design) + + [A low-level design](#a-low-level-design) + + [Executors](#executors) + + [Serial Executors](#serial-executors) + + [ExecutorJobs](#executorjobs) + + [Actors with custom SerialExecutors](#actors-with-custom-serialexecutors) + + [Asserting on executors](#asserting-on-executors) + + [Assuming actor executors](#asserting-actor-executors) + + [Default Swift Runtime Executors](#default-swift-runtime-executors) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) +* [Alternatives considered](#alternatives-considered) +* [Future Directions](#future-directions) + + [Overriding the MainActor Executor](#overriding-the-mainactor-executor) + + [Executor Switching Optimizations](#executor-switching) + + [Specifying Task executors](#specifying-task-executors) + + [DelegateActor property](#delegateactor-property) + +## Introduction + +As Swift Concurrency continues to mature it is becoming increasingly important to offer adopters tighter control over where exactly asynchronous work is actually executed. + +This proposal introduces a basic mechanism for customizing actor executors. By providing an instance of an executor, actors can influence "where" they will be executing any task they are running, while upholding the mutual exclusion and actor isolation guaranteed by the actor model. + +> **Note:** This proposal defines only a set of APIs to customize actor executors, and other kinds of executor control is out of scope for this specific proposal. + +## Motivation + +Swift's concurrency design is intentionally vague about the details of how code is actually run. Most code does not rely on specific properties of the execution environment, such as being run to a specific operating system thread, and instead needs only high-level semantic properties, expressed in terms of actor isolation, such as that no other code will be accessing certain variables concurrently. Maintaining flexibility about how work is scheduled onto threads allows Swift to avoid certain performance pitfalls by default. + +Nonetheless, it is sometimes useful to more finely control how code is executed: + +- The code may need to cooperate with an existing system that expects to run code in a certain way. + + For example, the system might expect certain kinds of work to be scheduled in special ways, like how some platforms require UI code to be run on the main thread, or single-threaded event-loop based runtimes assume all calls will be made from the same thread that is owned and managed by the runtime itself. + + For another example, a project might have a large amount of existing code which protects some state with a shared queue. In principle, this is the actor pattern, and the code could be rewritten to use Swift's actor support. However, it may be impossible to do that, or at least impractical to do it immediately. Using the existing queue as the executor for an actor allows code to adopt actors more incrementally. + +- The code may depend on being run on a specific system thread. + + For example, some libraries maintain state in thread-local variables, and running code on the wrong thread will lead to broken assumptions in the library. + + For another example, not all execution environments are homogeneous; some threads may be pinned to processors with extra capabilities. + +- The code's performance may benefit from the programmer being more explicit about where code should run. + + For example, if one actor frequently makes requests of another, and the actors rarely benefit from running concurrently, configuring them to use the same executor may decrease the runtime costs of switching between them. + + For another example, if an asynchronous function makes many calls to the same actor without any intervening suspensions, running the function explicitly on that actor's executor may eventually allow Swift to avoid a lot of switching overhead (or may even be necessary to perform those calls "atomically"). + +This is the first proposal discussing custom executors and customization points in the Swift Concurrency runtime, and while it introduces only the most basic customization points, we are certain that it already provides significant value to users seeking tighter control over their actor's execution semantics. + +Along with introducing ways to customize where code executes, this proposal also introduces ways to assert and assume the appropriate executor is used. This allows for more confidence when migrating away from other concurrency models to Swift Concurrency. + +## Proposed solution + +We propose to give developers the ability to implement simple serial executors, which then can be used with actors in order to ensure that any code executoring on such "actor with custom serial executor" runs on the appropriate thread or context. Implementing a naive executor takes the shape of: + +```swift +final class SpecificThreadExecutor: SerialExecutor { + let someThread: SomeThread // simplified handle to some specific thread + + func enqueue(_ job: consuming ExecutorJob) { + let unownedJob = UnownedExecutorJob(job) // in order to escape it to the run{} closure + someThread.run { + unownedJob.runSynchronously(on: self) + } + } + + func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(ordinary: self) + } +} + +extension SpecificThreadExecutor { + static var sharedUnownedExecutor: UnownedSerialExecutor { + // ... use some shared configured instance and return it ... + } +} +``` + +Such executor can then be used with an actor declaration by implementing its `unownedExecutor` property: + +```swift +actor Worker { + nonisolated var unownedExecutor: UnownedSerialExecutor { + // use the shared specific thread executor mentioned above. + // alternatively, we can pass specific executors to this actors init() and store and use them this way. + SpecificThreadExecutor.sharedUnownedExecutor + } +} +``` + +And lastly, in order to increase the confidence during moves from other concurrency models to Swift Concurrency with custom executors, we also provide ways to assert that a piece of code is executing on the appropriate executor. These methods should be used only if there is not better way to express the requirement statically. For example by expressing the code as a method on a specific actor, or annotating it with a `@GlobalActor`, should be preferred to asserting when possible, however sometimes this is not possible due to the old code fulfilling synchronous protocol requirements that still have these threading requirements. + +Asserting the appropriate executor is used in a synchronous piece of code looks like this: + +````swift +func synchronousButNeedsMainActorContext() { + // check if we're executing on the main actor context (or crash if we're not) + MainActor.preconditionIsolated() + + // same as precondition, however only in DEBUG builds + MainActor.assertIsolated() +} +```` + +Furthermore, we also offer a new API to safely "assume" an actor's execution context. For example, a synchronous function may know that it always will be invoked by the `MainActor` however for some reason it cannot be marked using `@MainActor`, this new API allows to assume (or crash if called from another execution context) the appropriate execution context, including the safety of synchronously accessing any state protected by the main actor executor: + +```swift +@MainActor func example() {} + +func alwaysOnMainActor() /* must be synchronous! */ { + MainActor.assumeIsolated { // will crash if NOT invoked from the MainActor's executor + example() // ok to safely, synchronously, call + } +} + +// Always prefer annotating the method using a global actor, rather than assuming it though. +@MainActor func alwaysOnMainActor() /* must be synchronous! */ { } // better, but not always possible +``` + +## Detailed design + +### A low-level design + +The API design of executors is intended to support high-performance implementations, with an expectation that custom executors will be primarily implemented by experts. Therefore, the following design heavily prioritizes the reliable elimination of abstraction costs over most other conceivable goals. In particular, the primitive operations specified by protocols are generally expressed in terms of opaque, unsafe types which implementations are required to use correctly. These operations are then used to implement more convenient APIs as well as the high-level language operations of Swift Concurrency. + +### Executors + +First, we introduce an `Executor` protocol, that serves as the parent protocol of all the specific kinds of executors we'll discuss next. It is the simplest kind of executor that does not provide any ordering guarantees about the submitted work. It could decide to run the submitted jobs in parallel, or sequentially. + +This protocol has existed in Swift ever since the introduction of Swift Concurrency, however, in this proposal we revise its API to make use of the newly introduced move-only capabilities in the language. The existing `UnownedExecutorJob` API will be deprecated in favor of one accepting a move-only `ExecutorJob`. The `UnownedExecutorJob` type remains available (and equally unsafe), because today still some usage patterns are not supported by the initial revision of move-only types. + +The concurrency runtime uses the `enqueue(_:)` method of an executor to schedule some work onto given executor. + +```swift +/// A service that can execute jobs. +public protocol Executor: AnyObject, Sendable { + + // This requirement is repeated here as a non-override so that we + // get a redundant witness-table entry for it. This allows us to + // avoid drilling down to the base conformance just for the basic + // work-scheduling operation. + func enqueue(_ job: consuming ExecutorJob) + + @available(*, deprecated, message: "Implement the enqueue(_:ExecutorJob) method instead") + func enqueue(_ job: UnownedExecutorJob) +} +``` + +In order to aid this transition, the compiler will offer assistance similar to how the transition from `Hashable.hashValue` to `Hashable.hash(into:)` was handled. Existing executor implementations which implemented `enqueue(UnownedExecutorJob)` will still work, but print a deprecation warning: + +```swift +final class MyOldExecutor: SerialExecutor { + // WARNING: 'Executor.enqueue(UnownedExecutorJob)' is deprecated as a protocol requirement; + // conform type 'MyOldExecutor' to 'Executor' by implementing 'enqueue(ExecutorJob)' instead + func enqueue(_ job: UnownedExecutorJob) { + // ... + } +} +``` + +Executors are required to follow certain ordering rules when executing their jobs: + +- The call to `ExecutorJob.runSynchronously(on:)` must happen-after the call to `enqueue(_:)`. +- If the executor is a serial executor, then the execution of all jobs must be *totally ordered*: for any two different jobs *A* and *B* submitted to the same executor with `enqueue(_:)`, it must be true that either all events in *A* happen-before all events in *B* or all events in *B* happen-before all events in *A*. + - Do note that this allows the executor to reorder `A` and `B`–for example, if one job had a higher priority than the other–however they each independently must run to completion before the other one is allowed to run. + + +### Serial Executors + +We also define a `SerialExecutor` protocol, which is what actors use to guarantee their serial execution of tasks (jobs). + +```swift +/// A service that executes jobs one-by-one, and specifically, +/// guarantees mutual exclusion between job executions. +/// +/// A serial executor can be provided to an actor (or distributed actor), +/// to guarantee all work performed on that actor should be enqueued to this executor. +/// +/// Serial executors do not, in general, guarantee specific run-order of jobs, +/// and are free to re-order them e.g. using task priority, or any other mechanism. +public protocol SerialExecutor: Executor { + /// Convert this executor value to the optimized form of borrowed + /// executor references. + func asUnownedSerialExecutor() -> UnownedSerialExecutor + + // Discussed in depth in "Details of 'same executor' checking" of this proposal. + func isSameExclusiveExecutionContext(other executor: Self) -> Bool +} + +extension SerialExecutor { + // default implementation is sufficient for most implementations + func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(ordinary: self) + } + + func isSameExclusiveExecutionContext(other: Self) -> Bool { + self === other + } +} +``` + +A `SerialExecutor` does not introduce new API, other than the wrapping itself in an `UnownedSerialExecutor` which is used by the Swift runtime to pass executors without incurring reference counting overhead. + +```swift +/// An unowned reference to a serial executor (a `SerialExecutor` +/// value). +/// +/// This is an optimized type used internally by the core scheduling +/// operations. It is an unowned reference to avoid unnecessary +/// reference-counting work even when working with actors abstractly. +/// Generally there are extra constraints imposed on core operations +/// in order to allow this. For example, keeping an actor alive must +/// also keep the actor's associated executor alive; if they are +/// different objects, the executor must be referenced strongly by the +/// actor. +public struct UnownedSerialExecutor: Sendable { + /// The default and ordinary way to expose an unowned serial executor. + public init(ordinary executor: E) + + /// Discussed in depth in "Details of same-executor checking" of this proposal. + public init(complexEquality executor: E) +} +``` + +`SerialExecutors` will potentially be extended to support "switching" which can lessen the amount of thread switches incured when using custom executors. Please refer to the Future Directions for a discussion of this extension. + +### ExecutorJobs + +An `ExecutorJob` is a representation of a chunk of of work that an executor should execute. For example, a `Task` effectively consists of a series of jobs that are enqueued onto executors, in order to run them. The name "job" was selected because we do not want to constrain this API to just "partial tasks", or tie them too closely to tasks, even though the most common type of job created by Swift concurrency are "partial tasks". + +Whenever the Swift Concurrency runtime needs to execute some piece of work, it enqueues an `UnownedExecutorJob`s on a specific executor the job should be executed on. The `UnownedExecutorJob` type is an opaque wrapper around Swift's low-level representation of such job. It cannot be meaningfully inspected, copied and must never be executed more than once. + +```swift +@noncopyable +public struct ExecutorJob: Sendable { + /// The priority of this job. + public var priority: JobPriority { get } +} +``` + +```swift +/// The priority of this job. +/// +/// The executor determines how priority information affects the way tasks are scheduled. +/// The behavior varies depending on the executor currently being used. +/// Typically, executors attempt to run tasks with a higher priority +/// before tasks with a lower priority. +/// However, the semantics of how priority is treated are left up to each +/// platform and `Executor` implementation. +/// +/// A ExecutorJob's priority is roughly equivalent to a `TaskPriority`, +/// however, since not all jobs are tasks, represented as separate type. +/// +/// Conversions between the two priorities are available as initializers on the respective types. +public struct JobPriority { + public typealias RawValue = UInt8 + + /// The raw priority value. + public var rawValue: RawValue +} + +extension TaskPriority { + /// Convert a job priority to a task priority. + /// + /// Most values are directly interchangeable, but this initializer reserves the right to fail for certain values. + public init?(_ p: JobPriority) { ... } +} +``` + +Because move-only types in the first early iteration of this language feature still have a number of limitations, we also offer an `UnownedExecutorJob` type, that is an unsafe "unowned" version of a `ExecutorJob`. One reason one might need to reach for an `UnownedExecutorJob` is whenever a `ExecutorJob` were to be used in a generic context, because in the initial version of move-only types that is available today, such types cannot appear in a generic context. For example, a naive queue implementation using an `[ExecutorJob]` would be rejected by the compiler, but it is possible to express using an `UnownedExecutorJob` (i.e.`[UnownedExecutorJob]`). + +```swift +public struct UnownedExecutorJob: Sendable, CustomStringConvertible { + + /// Create an unsafe, unowned, job by consuming a move-only ExecutorJob. + /// + /// This may be necessary currently when intending to store a job in collections, + /// or otherwise intreracting with generics due to initial implementation + /// limitations of move-only types. + @usableFromInline + internal init(_ job: consuming ExecutorJob) { ... } + + public var priority: JobPriority { ... } + + public var description: String { ... } +} +``` + +A job's description includes its job or task ID, that can be used to correlate it with task dumps as well as task lists in Instruments and other debugging tools (e.g. `swift-inspect`'s ). A task ID is an unique number assigned to a task, and can be useful when debugging scheduling issues, this is the same ID that is currently exposed in tools like Instruments when inspecting tasks, allowing to correlate debug logs with observations from profiling tools. + +Eventually, an executor will want to actually run a job. It may do so right away when it is enqueued, or on some different thread, this is entirely left up to the executor to decide. Running a job is done by calling the `runSynchronously` on a `ExecutorJob` which consumes it. The same method is provided on the `UnownedExecutorJob` type, however that API is not as safe, since it cannot consume the job, and is open to running the same job multiple times accidentally, which is undefined behavior. Generally, we urge developers to stick to using `ExecutorJob` APIs whenever possible, and only move to the unowned API if the noncopyable `ExecutorJob`s restrictions prove too strong to do the necessary operations on it. + +```swift +extension ExecutorJob { + /// Run the job synchronously. + /// + /// This operation consumes the job. + public consuming func runSynchronously(on executor: UnownedSerialExecutor) { + _swiftJobRun(UnownedExecutorJob(job), executor) + } +} + +extension UnownedExecutorJob { + /// Run the job synchronously. + /// + /// A job can only be run *once*. Accessing the job after it has been run is undefined behavior. + public func runSynchronously(on executor: UnownedSerialExecutor) { + _swiftJobRun(job, executor) + } +} +``` + +### Actors with custom SerialExecutors + +All actors implicitly conform to the `Actor` (or `DistributedActor`) protocols, and those protocols include the customization point for the executor they are required to run on in form of the the `unownedExecutor` property. + +An actor's executor must conform to the `SerialExecutor` protocol, which refines the Executor protocol, and provides enough guarantees to implement the actor's mutual exclusion guarantees. In the future, `SerialExecutors` may also be extended to support "switching", which is a technique to avoid thread-switching in calls between actors whose executors are compatible to "lending" each other the currently running thread. This proposal does not cover switching semantics. + +Actors select which serial executor they should use to run jobs by implementing the `unownedExecutor` protocol requirement on the `Actor` and `DistributedActor` protocols: + +```swift +public protocol Actor: AnyActor { + /// Retrieve the executor for this actor as an optimized, unowned + /// reference. + /// + /// This property must always evaluate to the same executor for a + /// given actor instance, and holding on to the actor must keep the + /// executor alive. + /// + /// This property will be implicitly accessed when work needs to be + /// scheduled onto this actor. These accesses may be merged, + /// eliminated, and rearranged with other work, and they may even + /// be introduced when not strictly required. Visible side effects + /// are therefore strongly discouraged within this property. + nonisolated var unownedExecutor: UnownedSerialExecutor { get } +} + +public protocol DistributedActor: AnyActor { + /// Retrieve the executor for this distributed actor as an optimized, + /// unowned reference. This API is equivalent to ``Actor/unownedExecutor``. + /// + /// ## Executor of remote distributed actor reference + /// + /// The default implementation of the `unownedExecutor` uses a special "crash if enqueued on" + /// executor, that can be obtained using `buildDefaultDistributedRemoteActorExecutor(any DistributedActor)` + /// method. If implementing a custom executor of a distributed actor, the implementation may derive + /// its executor value from the `nonisolated var id` every actor possesses (e.g. by means of the `ID` + /// indicating some "executor preference"), however if the actor is remote, the implementation SHOULD + /// return the default remote distributed actor executor, same as the default implementation does. + /// + /// Even if a remote distributed actor reference were to return some shared executor, + /// the Swift runtime will never actively make use of it, because code in this process + /// never runs methods which can be called cross-actor isolated "on" such distributed actor, + /// but merely delegates to the ``DistributedActorSystem/remoteCall` to perform the remote call. + /// This call is performed on the actor system, and is not isolated to the actor. + /// + /// Returning a shared executor for a remote distributed actor reference will not "trick" the + /// swift runtime into wrongly allowing one to `assumeIsolated()` and run code isolated on a + /// remote actor, because a remote actor reference cannot ever be `isolated` with. + /// + /// ## Availability + /// + /// Distributed actors can only use custom executors if their availability requires + /// a platform with Swift 5.9 (or higher) present. On platforms without availability + /// annotations, a distributed actor may always + /// + /// ## Custom implementation requirements + /// + /// This property must always evaluate to the same executor for a + /// given actor instance, and holding on to the actor must keep the + /// executor alive. + /// + /// This property will be implicitly accessed when work needs to be + /// scheduled onto this actor. These accesses may be merged, + /// eliminated, and rearranged with other work, and they may even + /// be introduced when not strictly required. Visible side effects + /// are therefore strongly discouraged within this property. + nonisolated var unownedExecutor: UnownedSerialExecutor { get } +} +``` + +> Note: It is not possible to express this protocol requirement on `AnyActor` directly because `AnyActor` is a "marker protocol" which are not present at runtime, and cannot have protocol requirements. + +The compiler synthesizes an implementation for this requirement for every `(distributed) actor` declaration, unless an explicit implementation is provided. The default implementation synthesized by the compiler uses the default `SerialExecutor`, which uses the appropriate mechanism for the platform (e.g. Dispatch). Actors using this default synthesized implementation are referred to as "Default Actors", i.e. actors using the default serial executor implementation. + +Developers can customize the executor used by an actor on a declaration-by-declaration basis, by implementing this protocol requirement in an actor. For example, thanks to the `sharedUnownedExecutor` static property on `MainActor` it is possible to declare other actors which are also guaranteed to use the same serial executor (i.e. "the main thread"). + +```swift +(distributed) actor MainActorsBestFriend { + nonisolated var unownedExecutor: UnownedSerialExecutor { + MainActor.sharedUnownedExecutor + } + func greet() { + print("Main-friendly...") + try? await Task.sleep(for: .seconds(3)) + } +} + +@MainActor +func mainGreet() { + print("Main hello!") +} + +func test() { + Task { await mainGreet() } + Task { await MainActorsBestFriend().greet() } +} +``` + +The snippet above illustrates that while the `MainActor` and the `MainActorsBestFriend` are different actors, and thus are generally allowed to execute concurrently, because they *share* the same main actor serial executor, they will never execute concurrently. A serial executor can only run one task at any given time, which enforces the mutual exclusive execution of those two actors. + +It is also possible for libraries to offer protocols where a default, library specific, executor is already defined, like this: + +```swift +protocol WithSpecifiedExecutor: Actor { + nonisolated var executor: LibrarySpecificExecutor { get } +} + +protocol LibrarySpecificExecutor: SerialExecutor {} + +extension LibrarySpecificActor { + /// Establishes the WithSpecifiedExecutorExecutor as the serial + /// executor that will coordinate execution for the actor. + nonisolated var unownedExecutor: UnownedSerialExecutor { + executor.asUnownedSerialExecutor() + } +} + +/// A naive "run on calling thread" job executor. +/// Generally executors should enqueue and process the job on another thread instead. +/// Ways to efficiently avoid hops when not necessary, will be offered as part of the +/// "executor switching" feature, that is not part of this proposal. +final class InlineExecutor: SpecifiedExecutor, CustomStringConvertible { + public func enqueue(_ job: __owned ExecutorJob) { + runJobSynchronously(job) + } +} +``` + +Which ensures that users of such library implementing such actors provide the library specific `SpecificExecutor` for their actors: + +```swift +actor MyActor: WithSpecifiedExecutor { + + nonisolated let executor: SpecifiedExecutor + + init(executor: SpecifiedExecutor) { + self.executor = executor + } +} +``` + +A library could also provide a default implementation of such executor as well. + +### Asserting on executors + +A common pattern in event-loop heavy code–not yet using Swift Concurrency–is to ensure/verify that a synchronous piece of code is executed on the exected event-loop. Since one of the goals of making executors customizable is to allow such libraries to adopt Swift Concurrency by making such event-loops conform to `SerialExecutor`, it is useful to allow the checking if code is indeed executing on the appropriate executor, for the library to gain confidence while it is moving towards fully embracing actors and Swift concurrency. + +For example, SwiftNIO intentionally avoids synchronization checks in some synchronous methods, in order to avoid the overhead of doing so, however in DEBUG mode it performs assertions that given code is running on the expected event-loop: + +```swift +// SwiftNIO +private var _channel: Channel +internal var channel: Channel { + self.eventLoop.assertInEventLoop() + assert(self._channel != nil || self.destroyed) + return self._channel ?? DeadChannel(pipeline: self) +} +``` + +Dispatch based systems also have similar functionality, with the `dispatchPrecondition` API: + +```swift +// Dispatch +func checkIfMainQueue() { + dispatchPrecondition(condition: .onQueue(DispatchQueue.main)) +} +``` + +While, generally, in Swift Concurrency such preconditions are not necessary, because we can *statically* ensure to be on the right execution context by putting methods on specific actors, or using global actor annotations: + +```swift +@MainActor +func definitelyOnMainActor() {} + +actor Worker {} +extension Worker { + func definitelyOnWorker() {} +} +``` + +Sometimes, especially when porting existing codebases _to_ Swift Concurrency we recognize the ability to assert in synchronous code if it is running on the expected executor can bring developers more confidence during their migration to Swift Concurrency. In order to support these migrations, we propose the following method: + +```swift +extension SerialExecutor { + /// Checks if the current task is running on the expected executor. + /// + /// Do note that if multiple actors share the same serial executor, + /// this assertion checks for the executor, not specific actor instance. + /// + /// Generally, Swift programs should be constructed such that it is statically + /// known that a specific executor is used, for example by using global actors or + /// custom executors. However, in some APIs it may be useful to provide an + /// additional runtime check for this, especially when moving towards Swift + /// concurrency from other runtimes which frequently use such assertions. + public func preconditionIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} + +extension Actor { + public nonisolated func preconditionIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} + +extension DistributedActor { + public nonisolated func preconditionIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} +``` + +as well as an `assert...` version of this API, which triggers only in `debug` builds: + +```swift +extension SerialExecutor { + // Same as ``SerialExecutor/preconditionIsolated(_:file:line)`` however only in DEBUG mode. + public func assertIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} + +extension Actor { + // Same as ``Actor/preconditionIsolated(_:file:line)`` however only in DEBUG mode. + public nonisolated func assertIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} + +extension DistributedActor { + // Same as ``DistributedActor/preconditionIsolated(_:file:line)`` however only in DEBUG mode. + public nonisolated func assertIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} +``` + +The versions of the APIs offered on `Actor` and `DistributedActor` offer better diagnostics than would be possible to implement using a plain `precondition()` implemented by developers using some `precondition(isOnExpectedExecutor(someExecutor))` because they offer a description of the actually active executor when mismatched: + +````swift +MainActor.preconditionIsolated() +// Precondition failed: Incorrect actor executor assumption; Expected 'MainActorExecutor' executor, but was executing on 'Sample.InlineExecutor'. +```` + +It should be noted that this API will return true whenever two actors share an executor. Semantically sharing a serial executor means running in the same isolation domain, however this is only known dynamically and `await`s are still necessary for calls between such actors: + +```swift +actor A { + nonisolated var unownedExecutor: UnownedSerialExecutor { MainActor.sharedUnownedExecutor } + + func test() {} +} + +actor B { + nonisolated var unownedExecutor: UnownedSerialExecutor { MainActor.sharedUnownedExecutor } + + func test(a: A) { + await a.test() // await is necessary, since we do not statically know about them being on the same executor + } +} +``` + +Potential future work could enable static checking where a relationship between actors is expressed statically (a specific instance of `B` declaring that it is on the same serial executor as a specific instance of `A`), and therefore awaits would not be necessary between such two specific actor instances. Such work is not within the scope of this initial proposal though, and only the dynamic aspect is proposed right now. + +At this point, similar to Dispatch, these APIs only offer an "assert" / "precondition" version. And currently the way to dynamically get a boolean answer about being on a specific executor is not exposed. + +### Assuming actor executors + +> Note: This API was initially pitched separately from custom executors, but as we worked on the feature we realized how closely it is related to custom executors and asserting on executors. The initial pitch thread is located here: [Pitch: Unsafe Assume on MainActor](https://forums.swift.org/t/pitch-unsafe-assume-on-mainactor/63074/). + +This revision of the proposal introduces the `MainActor.assumeIsolated(_:)` method, which allows synchronous code to safely assume that they are called within the context of the main actor's executor. This is only available in synchronous functions, because the right way to spell this requirement in asynchronous code is to annotate the function using `@MainActor` which statically ensures this requirement. + +Synchronous code can assume that it is running on the main actor executor by using this assume method: + +```swift +extension MainActor { + /// A safe way to synchronously assume that the current execution context belongs to the MainActor. + /// + /// This API should only be used as last resort, when it is not possible to express the current + /// execution context definitely belongs to the main actor in other ways. E.g. one may need to use + /// this in a delegate style API, where a synchronous method is guaranteed to be called by the + /// main actor, however it is not possible to annotate this legacy API with `@MainActor`. + /// + /// This method cannot be used in an asynchronous context. Instead, prefer implementing + /// a method annotated with `@MainActor` and calling it from your asynchronous context. + /// + /// - Warning: If the current executor is *not* the MainActor's serial executor, this function will crash. + /// + /// Note that this check is performed against the MainActor's serial executor, meaning that + /// if another actor uses the same serial executor--by using ``MainActor/sharedUnownedExecutor`` + /// as its own ``Actor/unownedExecutor``--this check will succeed, as from a concurrency safety + /// perspective, the serial executor guarantees mutual exclusion of those two actors. + @available(*, noasync) + func assumeIsolated( + _ operation: @MainActor () throws -> T, + file: StaticString = #fileID, line: UInt = #line + ) rethrows -> T +} +``` + +Similarly to the `preconditionIsolated` API, the executor check is performed against the target actor's executor, so if multiple actors are run on the same executor, this check will succeed in synchronous code invoked by such actors as well. In other words, the following code is also correct: + +```swift +func check(values: MainActorValues) /* synchronous! */ { + // values.get("any") // error: main actor isolated, cannot perform async call here + MainActor.assumeIsolated { + values.get("any") // correct & safe + } +} + +actor Friend { + var unownedExecutor: UnownedSerialExecutor { + MainActor.sharedUnownedExecutor + } + + func callCheck(values: MainActorValues) { + check(values) // correct + } +} + +actor Unknown { + func callCheck(values: MainActorValues) { + check(values) // will crash, we're not on the MainActor executor + } +} + +@MainActor +final class MainActorValues { + func get(_: String) -> String { ... } +} +``` + +> Note: Because it is not possible to abstract over the `@SomeGlobalActor () -> T` function type's global actor isolation, we currently do not offer a version of this API for _any_ global actor, however it would be possible to implement such API today using macros, which could be expored in a follow-up proposal if seen as important enough. Such API would have to be spelled `SomeGlobalActor.assumeIsolated() { @SomeGlobalActor in ... }`. + +In addition to the `MainActor` specialized API, the same shape of API is offered for instance actors and allows obtaining an `isolated` actor reference if we are guaranteed to be executing on the same serial executor as the given actor, and thus no concurrent access violations are possible. + +```swift +extension Actor { + /// A safe way to synchronously assume that the current execution context belongs to the passed in `actor`. + /// + /// If currently executing in the context of the actor's serial executor, safely execute the `operation` + /// isolated to the actor. Otherwise, crash reporting the difference in expected and actual executor. + /// + /// This method cannot be used in an asynchronous context. Instead, prefer implementing + /// a method on the actor and calling it from your asynchronous context. + /// + /// This API should only be used as last resort, when it is not possible to express the current + /// execution context definitely belongs to the main actor in other ways. E.g. one may need to use + /// this in a delegate style API, where a synchronous method is guaranteed to be called by the + /// main actor, however it is not possible to move some function implementation onto the target + /// `actor` for some reason. + /// + /// - Warning: If the current executor is *not* the actor's serial executor this function will crash. + /// + /// - Parameters: + /// - operation: the operation that will run if the executor checks pass + /// - Returns: the result of the operation + /// - Throws: the error the operation has thrown + @available(*, noasync) + func assumeIsolated( + _ operation: (isolated Self) throws -> T, + file: StaticString = #fileID, line: UInt = #line + ) rethrows -> T +} +``` + +These assume methods have the same semantics as the just explained `MainActor.assumeIsolated` in the sense that the check is performed about the actor's _executor_ and not specific instance. In other words, if many instance actors share the same serial executor, this check would pass for each of them, as long as the same executor is found to be the current one. + +The same method is offered for distributed actors, where code can only ever be isolated to an instance if the reference is to a _local_ distributed actor, as well as the same serial executor as the checked actor is running the current task: + +```swift +extension DistributedActor { + /// A safe way to synchronously assume that the current execution context belongs to the passed in `actor`. + /// + /// If currently executing in the context of the actor's serial executor, safely execute the `operation` + /// isolated to the actor. If the actor is local, or the current and expected executors are not compatible, + /// crash reporting the difference in expected and actual executor. + /// + /// This method cannot be used in an asynchronous context. Instead, prefer implementing + /// a method on the distributed actor and calling it from your asynchronous context. + /// + /// The actor must be a local distributed actor reference, as isolating execution to a remote reference + /// would not be memory safe, since a distributed remote actor reference is allowed to not allocate any + /// memory for its storage, and thus, any attempts to access it are illegal. If the actor is remote, + /// this method will terminate with a fatal error. + /// + /// This API should only be used as last resort, when it is not possible to express the current + /// execution context definitely belongs to the main actor in other ways. E.g. one may need to use + /// this in a delegate style API, where a synchronous method is guaranteed to be called by the + /// main actor, however it is not possible to move some function implementation onto the target + /// `distributed actor` for some reason. + /// + /// - Warning: If the current executor is *not* compatible with the expected serial executor, + /// or the distributed actor is a remote reference, this function will crash. + /// + /// - Parameters: + /// - operation: the operation that will run if the executor checks pass + /// - Returns: the result of the operation + /// - Throws: the error the operation has thrown + @available(*, noasync) + func assumeIsolated( + _ operation: (isolated Self) throws -> T, + file: StaticString = #fileID, line: UInt = #line + ) rethrows -> T +} +``` + +### Details of "same executor" checking + +The previous two sections described the various `assert`, `precondition` and `assume` APIs all of which depend on the notion of "the same serial execution context". By default, every actor gets its own serial executor instance, and each such instance is unique. Therefore without sharing executors, each actor's serial executor is unique to itself, and thus the `precondition` APIs would effectively check "are we on this _specific_ actor" even though the check is performed against the executor identity. + +#### Unique executors delegating to the same SerialExecutor + +There are two cases of checking "the same executor" that we'd like to discuss in this proposal. Firstly, even though some actors may want to share the a serial executor, sometimes developers may not want to receive this "different actors on same serial executor are in the same execution context" semantic for the various precondition checks. + +The solution here is in the way an executor may be implemented, and specifically, it is always possible to provide a _wrapper_ executor around another existing executor. This way we are able to assign unique executor identities, even if they would end up scheduling onto the same serial executor. As an example, this might look like this: + +```swift +final class SpecificThreadExecutor: SerialExecutor { ... } + +final class UniqueSpecificThreadExecutor: SerialExecutor { + let delegate: SpecificThreadExecutor + init(delegate: SpecificThreadExecutor) { + self.delegate = delegate + } + + func enqueue(_ job: consuming ExecutorJob) { + delegate.enqueue(job) + } + + func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(ordinary: self) + } +} + +actor Worker { + let unownedExecutor: UnownedSerialExecutor + + init(executor: SpecificThreadExecutor) { + let uniqueExecutor = UniqueSpecificThreadExecutor(delegate: executor) + self.unownedExecutor = uniqueExecutor.asUnownedSerialExecutor() + } + + func test(other: Worker) { + assert(self !== other) + assertOnActorExecutor(other) // expected crash. + // `other` has different unique executor, + // even through they both eventually delegate to the same + } +} +``` + +#### Different executors offering the same execution context + +We also introduce an optional extension to serial executor identity compatibility checking, which allows an executor to _participate_ in the check. This is in order to handle the inverse situation to what we just discussed: when different executors _are_ in fact the same exclusive serial execution context and _want to_ inform Swift runtime about this for the purpose of these assertion APIs. + +One example of an executor which may have different unique instances of executors, however they should behave as the same exclusive serial execution context are dispatch queues which have the ability to "target" a different queue. In other words, it is possible to have a dispatch queue `Q1` and `Q2` target the same queue `Qx` (or even the "main" dispatch queue). + +In order to facilitate this capability, when exposing the `UnownedSerialExecutor` for itself, the executor must use the `init(complexEquality:)` initializer: + +```swift +extension MyQueueExecutor { + public func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(complexEquality: self) + } +} +``` + +The unique initializer keeps the current semantics of "*if the executor pointers are the same, it is the same executor and exclusive execution context*" fast path of executor equality checking, however it adds a "deep check" code-path if the equality has failed. + +> The word "complex" was selected due to its meaning "consisting of many different and connected parts", which describes this feature very well. The various executors are able to form a complex network that may be necessary to be inspected in order to answer the "*is this the same context?*" question. + +When performing the "is this the same (or a compatible) serial execution context" checks, the Swift runtime first compares the raw pointers to the executor objects. If those are not equal and the executors in question have `complexEquality`, following some additional type-checks, the following `isSameExclusiveExecutionContext(other:)` method will be invoked: + +```swift +protocol SerialExecutor { + + // ... previously discussed protocol requirements ... + + /// If this executor has complex equality semantics, and the runtime needs to compare + /// two executors, it will first attempt the usual pointer-based equality check, + /// and if it fails it will compare the types of both executors, if they are the same, + /// it will finally invoke this method, in an attempt to let the executor itself decide + /// if this and the `other` executor represent the same serial, exclusive, isolation context. + /// + /// This method must be implemented with great care, as wrongly returning `true` would allow + /// code from a different execution context (e.g. thread) to execute code which was intended + /// to be isolated by another actor. + /// + /// This check is not used when performing executor switching. + /// + /// This check is used when performing `preconditionTaskOnActorExecutor`, `preconditionTaskOnActorExecutor`, + /// `assumeOnActorExecutor` and similar APIs which assert about the same "exclusive serial execution context". + /// + /// - Parameter other: + /// - Returns: true, if `self` and the `other` executor actually are mutually exclusive and it is safe–from a concurrency perspective–to execute code assuming one on the other. + func isSameExclusiveExecutionContext(other: Self) -> Bool +} + +extension SerialExecutor { + func isSameExclusiveExecutionContext(other: Self) -> Bool { + self === other + } +} +``` + +This API allows for executor, like for example dispatch queues in the future, to perform the "deep" check and e.g. return true if both executors are actually targeting the same thread or queue, and therefore guaranteeing a properly isolated mutually exclusive serial execution context. + +The API explicitly enforces that both executors must be of the same type, in order to avoid comparing completely unrelated executors using this rather expensive call into user code. The concrete logic for comparing executors for the purpose of the above described APIs is as follows: + +We inspect at the type of the executor (the bit we store in the ExecutorRef, specifically in the Implementation / witness table field), and if both are: + +- "**ordinary**" (or otherwise known as "unique"), which can be be thought of as definitely a "root" executor + - creation: + - using today's `UnownedSerialExecutor.init(ordinary:)` + - comparison: + - compare the two executors pointers directly + - return the result +- **complexEquality**, may be thought of as "**inner**" executor, i.e. one that's exact identity may need deeper introspection + - creation: + - `UnownedSerialExecutor(complexEquality:)` which sets specific bits that the runtime can recognize and enter the complex comparison code-path when necessary + - comparison: + - compare the two executor pointers directly, + - if they are the same, return true (same as in the "ordinary" case) + - check if the *target* executor has `complexEquality`, we check if the current executors have compatible witness tables + - if not, we return false + - invoke the executor implemented comparison the `currentExecutor.isSameExclusiveExecutionContext(expectedExecutor)` + - return the result + +These checks are likely *not* enough to to completely optimize task switching, and other mechanisms will be provided for optimized task switching in the future (see Future Directions). + +### Default Swift Runtime Executors + +Swift Concurrency provides a number of default executors already, such as: + +- the main actor executor, which services any code annotated using @MainActor, and +- the default global concurrent executor, which all (default) actors target by their own per-actor instantiated serial executor instances. + +The `MainActor`'s executor is available via the `sharedUnownedExecutor` static property on the `MainActor`: + +```swift +@globalActor public final actor MainActor: GlobalActor { + public nonisolated var unownedExecutor: UnownedSerialExecutor { get { ... } } + public static var sharedUnownedExecutor: UnownedSerialExecutor { get { ... } } +} +``` + +So putting other actors onto the same executor as the MainActor is executing on, is possible using the following pattern: + +```swift +actor Friend { + nonisolated var unownedExecutor: UnownedSerialExecutor { + MainActor.sharedUnownedExecutor + } +} +``` + +Note that the raw type of the MainActor executor is never exposed, but we merely get unowned wrappers for it. This allows the Swift runtime to pick various specific implementations depending on the runtime environment. + +The default global concurrent executor is not accessible directly from code, however it is the executor that handles all the tasks which do not have a specific executor requirement, or are explicitly required to run on that executor, e.g. like top-level async functions. + +## Source compatibility + +Many of these APIs are existing public types since the first introduction of Swift Concurrency (and are included in back-deployment libraries). As all types and pieces of this proposal are designed in a way that allows to keep source and behavioral compatibility with already existing executor APIs. + +Special affordances are taken to introduce the move-only ExecutorJob based enqueue API in an source compatible way, while deprecating the previously existing ("unowned") API. + +## Effect on ABI stability + +Swift's concurrency runtime has already been using executors, jobs and tasks since its first introduction, as such, this proposal remains ABI compatible with all existing runtime entry points and types. + +The design of `SerialExecutor` currently does not support non-reentrant actors, and it does not support executors for which dispatch is always synchronous (e.g. that just acquire a traditional mutex). + +Some of the APIs discussed in this proposal existed from the first introduction of Swift Concurrency, so making any breaking changes to them is not possible. Some APIs were carefully renamed and polished up though. We encourage discussion of all the types and methods present in this proposal, however changing some of them may prove to be challenging or impossible due to ABI impact. + +## Effect on API resilience + +While some APIs may depend on being executed on particular executors, this proposal makes no effort to formalize that in interfaces, as opposed to being an implementation detail of implementations, and so has no API resilience implications. + +If this is extended in the future to automatic, declaration-driven executor switching, as actors do, that would have API resilience implications. + +## Alternatives considered + +The proposed ways for actors to opt in to custom executors are brittle, in the sense that a typo or some similar error could accidentally leave the actor using the default executor. This could be fully mitigated by requiring actors to explicitly opt in to using the default executor; however, that would be an unacceptable burden on the common case. Short of that, it would be possible to have a modifier that marks a declaration as having special significance, and then complain if the compiler doesn't recognize that significance. However, there are a number of existing features that use a name-sensitive design like this, such as dynamic member lookup ([SE-0195](https://github.com/rjmccall/swift-evolution/blob/custom-executors/proposals/0195-dynamic-member-lookup.md)). A "special significance" modifier should be designed and considered more holistically. + +## Future Directions + +### Overriding the MainActor executor + +Because of the special semantics of `MainActor` as well as its interaction with an asynchronous `main` function, customizing its serial executor is slightly more tricky than customizing any other executor. We must both guarantee that the main function of a program runs on the main thread, and that any `MainActor` code also gets to run on the main thread. This also introduces interesting complications with the main function actually returning an exit code. + +It should be possible to override the serial executor used by the the asynchronous `main` method, as well as the `MainActor`. While the exact semantics remain to be designed, we envision an API that allows replacing the main executor before any asynchronous work has happened, and this way uphold the serial execution guarantees expected from the main actor. + +```swift +// DRAFT; Names of protocols or exact shape of such replacement API are non-final. + +protocol MainActorSerialExecutor: [...]SerialExecutor { ... } +func setMainActorExecutor(_ executor: some MainActorSerialExecutor) { ... } + +@main struct Boot { + func main() async { + // + + // The following call must be made: + // - before any suspension point is encountered + // - before + setMainActorExecutor(...) + + // + await hello() // give control of the "raw" main thread to the RunLoopSerialExecutor + // still main thread, but executing on the selected MainActorExecutor + } +} + +@MainActor +func hello() { + // guaranteed to be MainActor (main thread), + // executed on the selected main actor executor + print("Hello") +} +``` + +### Executor Switching + +Executor switching is the capability to avoid un-necessary thread hops, when attempting to hop between actors/executors, where the target executor is compatible with "taking over" the calling thread. This allows Swift to optimize for less thread hops and scheduling calls. E.g. if actors are scheduled on the same executor identity and they are compatible with switching, it is possible to avoid thread-hops entirely and execution can "follow the Task" through multiple executors. + +The early sketch of switching focused around adding the following methods to the executor protocols: + +```swift +// DRAFT; Names and specific APIs mentioned in this snippet are non-final. + +protocol SerialExecutor: Executor { + // .... existing APIs ... + + /// Is it possible for this executor to give up the current thread + /// and allow it to start running a different actor? + var canGiveUpThread: Bool { get } + + /// Given that canGiveUpThread() previously returned true, give up + /// the current thread. + func giveUpThread() + + /// Attempt to start running a task on the current actor. Returns + /// true if this succeeds. + func tryClaimThread() -> Bool +} +``` + +We will consider adding these, or similar, APIs to enable custom executors to participate in efficient switching, when we are certain these API shapes are "enough" to support all potential use-cases for this feature. + +### Specifying Task executors + +Specifying executors to tasks has a surprising number of tricky questions it has to answer, so for the time being we are not introducing such capability. Specifically, passing an executor to `Task(startingOn: someExecutor) { ... }` would make the Task _start_ on the specified executor, but detailed semantics about if the _all_ of this Task's body is expected to execute on `someExecutor` (i.e. we have to hop-back to it every time after an `await`), or if it is enough to just start on it and then continue avoiding scheduling more jobs if possible (i.e. allow for aggressive switching). + +### DelegateActor property + +The previous pitch of custom executors included a concept of a `delegateActor` which allowed an actor to declare a `delegateActor: Actor` property which would allow given actor to execute on the same executor as another actor instance. At the same time, this would provide enough information to the compiler at compile time, that both actors can be assumed to be within the same isolation domain, and `await`s between those actors could be skipped (!). A property that with custom executors holds dynamically, would this way be reinforced statically by the compiler and type-system. diff --git a/proposals/0393-parameter-packs.md b/proposals/0393-parameter-packs.md new file mode 100644 index 0000000000..3102304563 --- /dev/null +++ b/proposals/0393-parameter-packs.md @@ -0,0 +1,878 @@ +# Value and Type Parameter Packs + +* Proposal: [SE-0393](0393-parameter-packs.md) +* Authors: [Holly Borla](https://github.com/hborla), [John McCall](https://github.com/rjmccall), [Slava Pestov](https://github.com/slavapestov) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.9)** +* Review: ([pitch 1](https://forums.swift.org/t/pitch-parameter-packs/60543)) ([pitch 2](https://forums.swift.org/t/pitch-2-value-and-type-parameter-packs/60830)) ([review](https://forums.swift.org/t/se-0393-value-and-type-parameter-packs/63859)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0393-value-and-type-parameter-packs/64382)) + +## Introduction + +Many modern Swift libraries include ad-hoc variadic APIs with an arbitrary upper bound, typically achieved with overloads that each have a different fixed number of type parameters and corresponding arguments. Without variadic generic programming support in the language, these ad-hoc variadic APIs have a significant cost on library maintenance and the developer experience of using these APIs. + +This proposal adds _type parameter packs_ and _value parameter packs_ to enable abstracting over the number of types and values with distinct type. This is the first step toward variadic generics in Swift. + +## Contents + +- [Value and Type Parameter Packs](#value-and-type-parameter-packs) + - [Introduction](#introduction) + - [Contents](#contents) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [Type parameter packs](#type-parameter-packs) + - [Pack expansion type](#pack-expansion-type) + - [Type substitution](#type-substitution) + - [Single-element pack substitution](#single-element-pack-substitution) + - [Type matching](#type-matching) + - [Label matching](#label-matching) + - [Trailing closure matching](#trailing-closure-matching) + - [Type list matching](#type-list-matching) + - [Member type parameter packs](#member-type-parameter-packs) + - [Generic requirements](#generic-requirements) + - [Same-shape requirements](#same-shape-requirements) + - [Restrictions on same-shape requirements](#restrictions-on-same-shape-requirements) + - [Value parameter packs](#value-parameter-packs) + - [Overload resolution](#overload-resolution) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Alternatives considered](#alternatives-considered) + - [Modeling packs as tuples with abstract elements](#modeling-packs-as-tuples-with-abstract-elements) + - [Syntax alternatives to `repeat each`](#syntax-alternatives-to-repeat-each) + - [The `...` operator](#the--operator) + - [Another operator](#another-operator) + - [Magic builtin `map` method](#magic-builtin-map-method) + - [Future directions](#future-directions) + - [Variadic generic types](#variadic-generic-types) + - [Local value packs](#local-value-packs) + - [Explicit type pack syntax](#explicit-type-pack-syntax) + - [Pack iteration](#pack-iteration) + - [Pack element projection](#pack-element-projection) + - [Dynamic pack indexing with `Int`](#dynamic-pack-indexing-with-int) + - [Typed pack element projection using key-paths](#typed-pack-element-projection-using-key-paths) + - [Value expansion operator](#value-expansion-operator) + - [Pack destructuring operations](#pack-destructuring-operations) + - [Tuple conformances](#tuple-conformances) + - [Revision history](#revision-history) + - [Acknowledgments](#acknowledgments) + +## Motivation + +Generic functions currently require a fixed number of type parameters. It is not possible to write a generic function that accepts an arbitrary number of arguments with distinct types, instead requiring one of the following workarounds: + +* Erasing all of the types involved, e.g. using `Any...` +* Using a single tuple type argument instead of separate type arguments +* Overloading for each argument length with an artificial limit + +One example in the Swift Standard Library is the 6 overloads for each tuple comparison operator: + +```swift +func < (lhs: (), rhs: ()) -> Bool + +func < (lhs: (A, B), rhs: (A, B)) -> Bool where A: Comparable, B: Comparable + +func < (lhs: (A, B, C), rhs: (A, B, C)) -> Bool where A: Comparable, B: Comparable, C: Comparable + +// and so on, up to 6-element tuples +``` + +With language support for a variable number of type parameters, this API could be expressed more naturally and concisely as a single function declaration: + +```swift +func < (lhs: (repeat each Element), rhs: (repeat each Element)) -> Bool +``` + +## Proposed solution + +This proposal adds support for generic functions which abstract over a variable number of type parameters. While this proposal is useful on its own, there are many future directions that build upon this concept. This is the first step toward equipping Swift programmers with a set of tools that enable variadic generic programming. + +Parameter packs are the core concept that facilitates abstracting over a variable number of parameters. A pack is a new kind of type-level and value-level entity that represents a list of types or values, and it has an abstract length. A type parameter pack stores a list of zero or more type parameters, and a value parameter pack stores a list of zero or more value parameters. A type parameter pack is declared in angle brackets using the `each` contextual keyword: + +```swift +// 'S' is a type parameter pack where each pack element conforms to 'Sequence'. +func zip(...) +``` + +A parameter pack itself is not a first-class value or type, but the elements of a parameter pack can be used anywhere that naturally accepts a list of values or types using _pack expansions_, including top-level expressions. + +A pack expansion consists of the `repeat` keyword followed by a type or an expression. The type or expression that `repeat` is applied to is called the _repetition pattern_. The repetition pattern must contain at least one pack reference, spelled with the `each` keyword. At runtime, the pattern is repeated for each element in the substituted pack, and the resulting types or values are _expanded_ into the list provided by the surrounding context. + +Similarly, pack references can only appear inside repetition patterns and generic requirements: + +```swift +func zip(_ sequence: repeat each S) where repeat each S: Sequence +``` + +Given a concrete pack substitution, the pattern is repeated for each element in the substituted pack. If `S` is substituted with `Array, Set`, then `repeat Optional` will repeat the pattern `Optional` for each element in the substitution to produce `Optional>, Optional>`. + +Here are the key concepts introduced by this proposal: + +- Under the new model, all existing types and values in the language become _scalar types_ and _scalar values_. +- A _type pack_ is a new kind of type-level entity which represents a list of scalar types. Type packs do not have syntax in the surface language, but we will write them as `{T1, ..., Tn}` where each `Ti` is a scalar type. Type packs cannot be nested; type substitution is defined to always flatten type packs. +- A _type parameter pack_ is a list of zero or more scalar type parameters. These are declared in a generic parameter list using the syntax `each T`, and referenced with `each T`. +- A _value pack_ is a list of scalar values. The type of a value pack is a type pack, where each element of the type pack is the scalar type of the corresponding scalar value. Value packs do not have syntax in the surface language, but we will write them as `{x1, ..., xn}` where each `xi` is a scalar value. Value packs cannot be nested; evaluation is always defined to flatten value packs. +- A _value parameter pack_ is a list of zero or more scalar function or macro parameters. +- A _pack expansion_ is a new kind of type-level and value-level construct that expands a type or value pack into a list of types or values, respectively. Written as `repeat P`, where `P` is the _repetition pattern_ that captures at least one type parameter pack (spelled with the `each` keyword). At runtime, the pattern is repeated for each element in the substituted pack. + +The following example demonstrates these concepts together: + +```swift +struct Pair { + init(_ first: First, _ second: Second) +} + +func makePairs( + firsts first: repeat each First, + seconds second: repeat each Second +) -> (repeat Pair) { + return (repeat Pair(each first, each second)) +} + +let pairs = makePairs(firsts: 1, "hello", seconds: true, 1.0) +// 'pairs' is '(Pair(1, true), Pair("hello", 2.0))' +``` + +The `makePairs` function declares two type parameter packs, `First` and `Second`. The value parameter packs `first` and `second` have the pack expansion types `repeat each First` and `repeat each Second`, respectively. The return type `(repeat Pair)` is a tuple type where each element is a `Pair` of elements from the `First` and `Second` parameter packs at the given tuple position. + +Inside the body of `makePairs()`, `repeat Pair(each first, each second)` is a pack expansion expression referencing the value parameter packs `first` and `second`. + +The call to `makePairs()` substitutes the type pack `{Int, Bool}` for `First`, and the type pack `{String, Double}` for `Second`. These substitutions are deduced by the _type matching rules_, described below. The function is called with four arguments; `first` is the value pack `{1, "hello"}`, and `second` is the value pack `{true, 2.0}`. + +The substituted return type is the tuple type with two elements `(Pair, Pair)`, and the returned value is the tuple value with two elements `(Pair(1, true), Pair("hello", 2.0))`. + +## Detailed design + +**Note:** While this proposal talks about "generic functions", everything also applies to initializers and subscripts nested inside types. With closure expressions, the situation is slightly more limited. Closure expressions support value parameter packs, however since closure expressions do not have polymorphic types in Swift, they're limited to referencing type parameter packs from outer scopes and cannot declare type parameter packs of their own. Also, the value parameter packs of closures cannot have argument labels, because as usual only named declarations have argument labels in Swift. + +### Type parameter packs + +The generic parameter list of a generic function can contain one or more _type parameter pack declarations_, written as an identifier preceded by `each`: + +```swift +func variadic() {} +``` + +When referenced from type context, this identifier resolves to a _type parameter pack_. References to type parameter packs can only appear in the following positions: + +* The base type of a member type parameter pack, which is again subject to these rules +* The pattern type of a pack expansion type, where it stands for the corresponding scalar element type +* The pattern expression of a pack expansion expression, where it stands for the metatype of the corresponding scalar element type and can be used like any other scalar metatype, e.g. to call a static method, call an initializer, or reify the metatype value +* The subject type of a conformance, superclass, layout, or same-type requirement +* The constraint type of a same-type requirement + +### Pack expansion type + +A pack expansion type, written as `repeat P`, has a *pattern type* `P` and a non-empty set of _captured_ type parameter packs spelled with the `each` keyword. For example, the pack expansion type `repeat Array` has a pattern type `Array` that captures the type parameter pack `T`. + +**Syntactic validity:** Pack expansion types can appear in the following positions: + +* The type of a parameter in a function declaration, e.g. `func foo(values: repeat each T) -> Bool` +* The type of a parameter in a function type, e.g. `(repeat each T) -> Bool` +* The type of an unlabeled element in a tuple type, e.g. `(repeat each T)` + +Because pack expansions can only appear in positions that accept a list of types or values, pack expansion patterns are naturally delimited by a comma, the next statement in top-level code, or an end-of-list delimiter, e.g. `)` for call argument lists or `>` for generic argument lists. + +The restriction where only unlabeled elements of a tuple type may have a pack expansion type is motivated by ergonomics. If you could write `(t: repeat each T)`, then after a substitution `T := {Int, String}`, the substituted type would be `(t: Int, String)`. This would be strange, because projecting the member `t` would only produce the first element. When an unlabeled element has a pack expansion type, like `(repeat each T)`, then after the above substitution you would get `(Int, String)`. You can still write `0` to project the first element, but this is less surprising to the Swift programmer. + +**Capture:** A type _captures_ a type parameter pack if the type parameter pack appears inside the pattern type, without any intervening pack expansion type. For example, if `T` and `U` are type parameter packs, then `repeat Array<(each T) -> each U>` captures both `T` and `U`. However, `repeat Array<(each T) -> (repeat each U)>` captures `T`, but *not* `U`. Only the inner pack expansion type `repeat each U` captures `U`. (Indeed, in a valid program, every reference to a type parameter pack is captured by exactly one pack expansion type.) + +The captures of the pattern type are a subset of the captures of the pack expansion type itself. In some situations (described in the next section), the pack expansion type might capture a type parameter pack that does not appear in the pattern type. + +**Typing rules:** A pack expansion type is _well-typed_ if replacing the captured type parameter packs in the pattern type with scalar type parameters of the same constraints produces a well-typed scalar type. + +For example, if `each T` is a type parameter pack subject to the conformance requirement `each T: Hashable`, then `repeat Set` is well-typed, because `Set` is well-typed given `T: Hashable`. + +However, if `each T` were not subject to this conformance requirement, then `repeat Set` would not be well-typed; the user might substitute `T` with a type pack containing types that do not conform to `Hashable`, like `T := {AnyObject, Int}`, and the expanded substitution `Set, Set` is not well-typed because `Set` is not well-typed. + +### Type substitution + +Recall that a reference to a generic function from expression context always provides an implicit list of *generic arguments* which map each of the function's type parameters to a *replacement type*. The type of the expression referencing a generic declaration is derived by substituting each type parameter in the declaration's type with the corresponding replacement type. + +The replacement type of a type parameter pack is always a type pack. Since type parameter packs always occur inside the pattern type of a pack expansion type, we need to define what it means to perform a substitution on a type that contains pack expansion types. + +Recall that pack expansion types appear in function parameter types and tuple types. Substitution replaces each pack expansion type with an expanded type list, which is flattened into the outer type list. + +**Intuition:** The substituted type list is formed by replacing the captured type parameter pack references with the corresponding elements of each replacement type pack. + +For example, consider the declaration: + +```swift +func variadic( + t: repeat each T, + u: repeat each U +) -> (Int, repeat ((each T) -> each U)) +``` + +Suppose we reference it with the following substitutions: + +```swift +T := {String, repeat each V, Float} +U := {NSObject, repeat Array, NSString} +``` + +The substituted return type of `variadic` becomes a tuple type with 4 elements: + +```swift +(Int, (String) -> NSObject, repeat ((each V) -> Array), (Float) -> NSString) +``` + +**Formal algorithm:** Suppose `repeat P` is a pack expansion type with pattern type `P`, that captures a list of type parameter packs `Ti`, and let `S[Ti]` be the replacement type pack for `Ti`. We require that each `S[Ti]` has the same length; call this length `N`. If the lengths do not match, the substitution is malformed. Let `S[Ti][j]` be the `j`th element of `S[Ti]`, where `0 ≤ j < N`. + +The `j`th element of the replacement type list is derived as follows: + +1. If each `S[Ti][j]` is a scalar type, the element type is obtained by substituting each `Ti` with `S[Ti][j]` in the pattern type `P`. +2. If each `S[Ti][j]` is a pack expansion type, then `S[Ti][j]` = `repeat Pij` for some pattern type `Pij`. The element type is the pack expansion type `repeat Qij`, where `Qij` is obtained by substituting each `Ti` with `Pij` in the pattern type `P`. +3. Any other combination means the substitution is malformed. + +When the lengths or structure of the replacement type packs do not match, the substitution is malformed. This situation is diagnosed with an error by checking generic requirements, as discussed below. + +For example, the following substitutions are malformed because the lengths do not match: + +```swift +T := {String, Float} +U := {NSObject} +``` + +The following substitutions are malformed because the replacement type packs have incompatible structure, hitting Case 3 above: + +```swift +T := {repeat each V, Float} +U := {NSObject, repeat each W} +``` + +To clarify what it means for a type to capture a type parameter pack, consider the following: + +```swift +func variadic(t: repeat each T, u: repeat each U) -> (repeat (each T) -> (repeat each U)) +``` + +The pack expansion type `repeat (each T) -> (repeat each U)` captures `T`, but not `U`. If we apply the following substitutions: + +```swift +T := {Int, String} +U := {Float, Double, Character} +``` + +Then the substituted return type becomes a pair of function types: + +```swift +((Int) -> (Float, Double, Character), (String) -> (Float, Double, Character)) +``` + +Note that the entire replacement type pack for `U` was flattened in each repetition of the pattern type; we did not expand "across" `U`. + +**Concrete pattern type:** It is possible to construct an expression with a pack expansion type whose pattern type does not capture any type parameter packs. This is called a pack expansion type with a _concrete_ pattern type. For example, consider this declaration: + +```swift +func counts(_ t: repeat each T) { + let x = (repeat (each t).count) +} +``` + +The `count` property on the `Collection` protocol returns `Int`, so the type of the expression `(repeat (each t).count)` is written as the one-element tuple type `(repeat Int)` whose element is the pack expansion type `repeat Int`. While the pattern type `Int` does not capture any type parameter packs, the pack expansion type must still capture `T` to represent the fact that after expansion, the resulting tuple type has the same length as `T`. This kind of pack expansion type can arise during type inference, but it cannot be written in source. + +#### Single-element pack substitution + +If a parameter pack `each T` is substituted with a single element, the parenthesis around `(repeat each T)` are unwrapped to produce the element type as a scalar instead of a one-element tuple type. + +For example, the following substitutions both produce the element type `Int`: +- Substituting `each T := {Int}` into `(repeat each T)`. +- Substituting `each T := {}` into `(Int, repeat each T)`. + +Though unwrapping single-element tuples complicates type matching, surfacing single-element tuples in the programming model would increase the surface area of the language. One-element tuples would need to be manually unwrapped with `.0` or pattern matching in order to make use of their contents. This unwrapping would clutter up code. + + +### Type matching + +Recall that the substitutions for a reference to a generic function are derived from the types of call argument expressions together with the contextual return type of the call, and are not explicitly written in source. This necessitates introducing new rules for _matching_ types containing pack expansions. + +There are two separate rules: + +- For call expressions where the callee is a named function declaration, _label matching_ is performed. +- For everything else, _type list matching_ is performed. + +#### Label matching + +Here, we use the same rule as the "legacy" variadic parameters that exist today. If a function declaration parameter has a pack expansion type, the parameter must either be the last parameter, or followed by a parameter with a label. A diagnostic is produced if the function declaration violates this rule. + +Given a function declaration that is well-formed under this rule, type matching then uses the labels to delimit type packs. For example, the following is valid: + + ```swift + func concat(t: repeat each T, u: repeat each U) -> (repeat each T, repeat each U) + + // T := {Int, Double} + // U := {String, Array} + concat(t: 1, 2.0, u: "hi", [3]) + + // substituted return type is (Int, Double, String, Array) + ``` + + while the following is not: + + ```swift + func bad(t: repeat each T, repeat each U) -> (repeat each T, repeat each U) + // error: 'repeat each T' followed by an unlabeled parameter + + bad(1, 2.0, "hi", [3]) // ambiguous; where does 'each T' end and 'each U' start? + ``` + +#### Trailing closure matching + +Argument-to-parameter matching for parameter pack always uses a forward-scan for trailing closures. For example, the following code is valid: + +```swift +func trailing(t: repeat each T, u: repeat each U) {} + +// T := {() -> Int} +// U := {} +trailing { 0 } +``` + +while the following produces an error: + +```swift +func trailing(t: repeat each T, u: repeat each U) {} + +// error: type '() -> Int' cannot conform to 'Sequence' +trailing { 0 } +``` + +#### Type list matching + +In all other cases, we're matching two comma-separated lists of types. If either list contains two or more pack expansion types, the match remains _unsolved_, and the type checker attempts to derive substitutions by matching other types before giving up. (This allows a call to `concat()` as defined above to succeed, for example; the match between the contextual return type and `(repeat each T, repeat each U)` remains unsolved, but we are able to derive the substitutions for `T` and `U` from the call argument expressions.) + +Otherwise, we match the common prefix and suffix as long as no pack expansion types appear on either side. After this has been done, there are three possibilities: + +1. Left hand side contains a single pack expansion type, right hand size contains zero or more types. +2. Left hand side contains zero or more types, right hand side contains a single pack expansion type. +3. Any other combination, in which case the match fails. + +For example: + +```swift +func variadic(_: repeat each T) -> (Int, repeat each T, String) {} + +let fn = { x, y in variadic(x, y) as (Int, Double, Float, String) } +``` + +Case 3 covers the case where one of the lists has a pack expansion, but the other one is too short; for example, matching `(Int, repeat each T, String, Float)` against `(Int, Float)` leaves you with `(repeat each T, String)` vs `()`, which is invalid. + +If neither side contains a pack expansion type, Case 3 also subsumes the current behavior as implemented without this proposal, where type list matching always requires the two type lists to have the same length. For example, when matching `(Int, String)` against `(Int, Float, String)`, we end up with `()` vs `(Float)`, which is invalid. + +The type checker derives the replacement type for `T` in the call to `variadic()` by matching the contextual return type `(Int, Double, Float, String)` against the declared return type `(Int, repeat each T, String)`. The common prefix `Int` and common suffix `String` successfully match. What remains is the pack expansion type `repeat each T` and the type list `Double, Float`. This successfully matches, deriving the substitution `T := {Double, Float}`. + +While type list matching is positional, the type lists may still contain labels if we're matching two tuple types. We require the labels to match exactly when dropping the common prefix and suffix, and then we only allow Case 1 and 2 to succeed if the remaining type lists do not contain any labels. + +For example, matching `(x: Int, repeat each T, z: String)` against `(x: Int, Double, y: Float, z: String)` drops the common prefix and suffix, and leaves you with the pack expansion type `repeat each T` vs the type list `Double, y: Float`, which fails because `Double: y: Float` contains a label. + +However, matching `(x: Int, repeat each T, z: String)` against `(x: Int, Double, Float, z: String)` leaves you with `repeat each T` vs `Double, Float`, which succeeds with `T := {Double, Float}`, because the labels match exactly in the common prefix and suffix, and no labels remain once we get to Case 1 above. + +### Member type parameter packs + +If a type parameter pack `each T` is subject to a protocol conformance requirement `P`, and `P` declares an associated type `A`, then `(each T).A` is a valid pattern type for a pack expansion type, called a _member type parameter pack_. + +Under substitution, a member type parameter pack projects the associated type from each element of the replacement type pack. + +For example: + +```swift +func variadic(_: repeat each T) -> (repeat (each T).Element) +``` + +After the substitution `T := {Array, Set}`, the substituted return type of this function becomes the tuple type `(Int, String)`. + +We will refer to `each T` as the _root type parameter pack_ of the member type parameter packs `(each T).A` and `(each T).A.B`. + +### Generic requirements + +All existing kinds of generic requirements can be used inside _requirement expansions_, which represent a list of zero or more requirements. Requirement expansions are spelled with the `repeat` keyword followed by a generic requirement pattern that captures at least one type parameter pack reference spelled with the `each` keyword. Same-type requirements generalize in multiple different ways, depending on whether one or both sides involve a type parameter pack. + +1. Conformance, superclass, and layout requirements where the subject type is a type parameter pack are interpreted as constraining each element of the replacement type pack: + + ```swift + func variadic(_: repeat each S) where repeat each S: Sequence { ... } + ``` + + A valid substitution for the above might replace `S` with `{Array, Set}`. Expanding the substitution into the requirement `each S: Sequence` conceptually produces the following conformance requirements: `Array: Sequence, Set: Sequence`. + +1. A same-type requirement where one side is a type parameter pack and the other type is a scalar type that does not capture any type parameter packs is interpreted as constraining each element of the replacement type pack to _the same_ scalar type: + + ```swift + func variadic(_: repeat each S) where repeat (each S).Element == T {} + ``` + + This is called a _same-element requirement_. + + A valid substitution for the above might replace `S` with `{Array, Set}`, and `T` with `Int`. + + +3. A same-type requirement where each side is a pattern type that captures at least one type parameter pack is interpreted as expanding the type packs on each side of the requirement, equating each element pair-wise. + + ```swift + func variadic(_: repeat each S) where repeat (each S).Element == Array {} + ``` + + This is called a _same-type-pack requirement_. + + A valid substitution for the above might replace `S` with `{Array>, Set>}`, and `T` with `{Int, String}`. Expanding `(each S).Element == Array` will produce the following list of same-type requirements: `Array.Element == Array, Set>.Element == String`. + +There is an additional kind of requirement called a _same-shape requirement_. There is no surface syntax for spelling a same-shape requirement; they are always inferred, as described in the next section. + +**Symmetry:** Recall that same-type requirements are symmetrical, so `T == U` is equivalent to `U == T`. Therefore some of the possible cases above are not listed, but the behavior can be understood by first transposing the same-type requirement. + +**Constrained protocol types:** A conformance requirement where the right hand side is a constrained protocol type `P` may reference type parameter packs from the generic arguments `Ti` of the constrained protocol type. In this case, the semantics are defined in terms of the standard desugaring. Independent of the presence of type parameter packs, a conformance requirement to a constrained protocol type is equivalent to a conformance requirement to `P` together with one or more same-type requirements that constrain the primary associated types of `P` to the corresponding generic arguments `Ti`. After this desugaring step, the induced same-type requirements can then be understood by Case 2, 3, 4 or 5 above. + +#### Same-shape requirements + +A same-shape requirement states that two type parameter packs have the same number of elements, with pack expansion types occurring at identical positions. + +This proposal does not include a spelling for same-shape requirements in the surface language; same-shape requirements are always inferred, and an explicit same-shape requirement syntax is a future direction. However, we will use the notation `shape(T) == shape(U)` to denote same-shape requirements in this proposal. + +A same-shape requirement always relates two root type parameter packs. Member types always have the same shape as the root type parameter pack, so `shape(T.A) == shape(U.B)` reduces to `shape(T) == shape(U)`. + +**Inference:** Same-shape requirements are inferred in the following ways: + +1. A same-type-pack requirement implies a same-shape requirement between all type parameter packs captured by the pattern types on each side of the requirement. + + For example, given the parameter packs ``, the same-type-pack requirement `Pair == (each S).Element` implies `shape(First) == shape(Second), shape(First) == shape(S), shape(Second) == shape(S)`. + +2. A same-shape requirement is inferred between each pair of type parameter packs captured by a pack expansion type appearing in the following positions + * all types appearing in the requirements of a trailing `where` clause of a generic function + * the parameter types and the return type of a generic function + +Recall that if the pattern of a pack expansion type contains more than one type parameter pack, all type parameter packs must be known to have the same shape, as outlined in the [Type substitution](#type-substitution) section. Same-shape requirement inference ensures that these invariants are satisfied when the pack expansion type occurs in one of the two above positions. + +If a pack expansion type appears in any other position, all type parameter packs captured by the pattern type must already be known to have the same shape, otherwise an error is diagnosed. + +For example, `zip` is a generic function, and the return type `(repeat (each T, each U))` is a pack expansion type, therefore the same-shape requirement `shape(T) == shape(U)` is automatically inferred: + +```swift +// Return type infers 'where length(T) == length(U)' +func zip(firsts: repeat each T, seconds: repeat each U) -> (repeat (each T, each U)) { + return (repeat (each firsts, each seconds)) +} + +zip(firsts: 1, 2, seconds: "hi", "bye") // okay +zip(firsts: 1, 2, seconds: "hi") // error; length requirement is unsatisfied +``` + +Here is an example where the same-shape requirement is not inferred: + +```swift +func foo(t: repeat each T, u: repeat each U) { + let tup: (repeat (each T, each U)) = /* whatever */ +} +``` + +The type annotation of `tup` contains a pack expansion type `repeat (each T, each U)`, which is malformed because the requirement `shape(T) == shape(U)` is unsatisfied. This pack expansion type is not subject to requirement inference because it does not occur in one of the above positions. + +#### Restrictions on same-shape requirements + +Type packs cannot be written directly, but requirements involving pack expansions where both sides are concrete types are desugared using the type matching algorithm. This means it is possible to write down a requirement that constrains a type parameter pack to a concrete type pack, unless some kind of restriction is imposed: + +```swift +func constrain(_: repeat each S) where (repeat (each S).Element) == (Int, String) {} +``` + +Furthermore, since the same-type requirement implies a same-shape requirement, we've implicitly constrained `S` to having a length of 2 elements, without knowing what those elements are. + +This introduces theoretical complications. In the general case, same-type requirements on type parameter packs allows encoding arbitrary systems of integer linear equations: + +```swift +// shape(Q) = 2 * shape(R) + 1 +// shape(Q) = shape(S) + 2 +func solve(q: repeat each Q, r: repeat each R, s: repeat each S) + where (repeat each Q) == (Int, repeat each R, repeat each R), + (repeat each Q) == (repeat each S, String, Bool) { } +``` + +While type-level linear algebra is interesting, we may not ever want to allow this in the language to avoid significant implementation complexity, and we definitely want to disallow this expressivity in this proposal. + +To impose restrictions on same-shape and same-type requirements, we will formalize the concept of the “shape” of a pack, where a shape is one of: + +* A single scalar type element; all scalar types have a singleton ``scalar shape'' +* An abstract shape that is specific to a pack parameter +* A concrete shape that is composed of the scalar shape and abstract shapes + +For example, the pack `{Int, repeat each T, U}` has a concrete shape that consists of two single elements and one abstract shape. + +This proposal only enables abstract shapes. Each type parameter pack has an abstract shape, and same-shape requirements merge equivalence classes of abstract shapes. Any same-type requirement that imposes a concrete shape on a type parameter pack will be diagnosed as a *conflict*, much like other conflicting requirements such as `where T == Int, T == String` today. + +This aspect of the language can evolve in a forward-compatible manner. Over time, some restrictions can be lifted, while others remain, as different use-cases for type parameter packs are revealed. + +### Value parameter packs + +A _value parameter pack_ represents zero or more function or macro parameters, and it is declared with a function parameter that has a pack expansion type. In the following declaration, the function parameter `value` is a value parameter pack that receives a _value pack_ consisting of zero or more argument values from the call site: + +```swift +func tuplify(_ value: repeat each T) -> (repeat each T) + +_ = tuplify() // T := {}, value := {} +_ = tuplify(1) // T := {Int}, value := {1} +_ = tuplify(1, "hello", [Foo()]) // T := {Int, String, [Foo]}, value := {1, "hello", [Foo()]} +``` + +**Syntactic validity:** A value parameter pack can only be referenced from a pack expansion expression. A pack expansion expression is written as `repeat expr`, where `expr` is an expression containing one or more value parameter packs or type parameter packs spelled with the `each` keyword. Pack expansion expressions can appear in any position that naturally accepts a list of expressions, including comma-separated lists and top-level expressions. This includes the following: + +* Call arguments, e.g. `generic(repeat each value)` +* Subscript arguments, e.g. `subscriptable[repeat each index]` +* The elements of a tuple value, e.g. `(repeat each value)` +* The elements of an array literal, e.g. `[repeat each value]` + +Pack expansion expressions can also appear in an expression statement at the top level of a brace statement. In this case, the semantics are the same as scalar expression statements; the expression is evaluated for its side effect and the results discarded. + +**Capture:** A pack expansion expression _captures_ a value (or type) pack parameter the value (or type) pack parameter appears as a sub-expression without any intervening pack expansion expression. + +Furthermore, a pack expansion expression also captures all type parameter packs captured by the types of its captured value parameter packs. + +For example, say that `x` and `y` are both value parameter packs and `T` is a type parameter pack, and consider the pack expansion expression `repeat foo(each x, (each T).self, (repeat each y))`. This expression captures both the value parameter pack `x` and type parameter pack `T`, but it does not capture `y`, because `y` is captured by the inner pack expansion expression `repeat each y`. Additionally, if `x` has the type `Foo`, then our expression captures `U`, but not `V`, because again, `V` is captured by the inner pack expansion type `repeat each V`. + +**Typing rules:** For a pack expansion expression to be well-typed, two conditions must hold: + +1. The types of all value parameter packs captured by a pack expansion expression must be related via same-shape requirements. + +2. After replacing all value parameter packs with non-pack parameters that have equivalent types, the pattern expression must be well-typed. + +Assuming the above hold, the type of a pack expansion expression is defined to be the pack expansion type whose pattern type is the type of the pattern expression. + +**Evaluation semantics:** At runtime, each value (or type) parameter pack receives a value (or type) pack, which is a concrete list of values (or types). The same-shape requirements guarantee that all value (and type) packs have the same length, call it `N`. The evaluation semantics are that for each successive `i` such that `0 ≤ i < N`, the pattern expression is evaluated after substituting each occurrence of a value (or type) parameter pack with the `i`th element of the value (or type) pack. The evaluation proceeds from left to right according to the usual evaluation order, and the list of results from each evaluation forms the argument list for the parent expression. + +For example, pack expansion expressions can be used to forward value parameter packs to other functions: + +```swift +func tuplify(_ t: repeat each T) -> (repeat each T) { + return (repeat each t) +} + +func forward(u: repeat each U) { + let _ = tuplify(repeat each u) // T := {repeat each U} + let _ = tuplify(repeat each u, 10) // T := {repeat each U, Int} + let _ = tuplify(repeat each u, repeat each u) // T := {repeat each U, repeat each U} + let _ = tuplify(repeat [each u]) // T := {repeat Array} +} +``` + +### Overload resolution + +Generic functions can be overloaded by the "pack-ness" of their type parameters. For example, a function can have two overloads where one accepts a scalar type parameter and the other accepts a type parameter pack: + +```swift +func overload(_: T) {} +func overload(_: repeat each T) {} +``` + +If the parameters of the scalar overload have the same or refined requirements as the parameter pack overload, the scalar overload is considered a subtype of the parameter pack overload, because the parameters of the scalar overload can be forwarded to the parameter pack overload. Currently, if a function call successfully type checks with two different overloads, the subtype is preferred. This overload ranking rule generalizes to overloads with parameter packs, which effectively means that scalar overloads are preferred over parameter pack overloads when the scalar requirements meet the requirements of the parameter pack: + +```swift +func overload() {} +func overload(_: T) {} +func overload(_: repeat each T) {} + +overload() // calls the no-parameter overload + +overload(1) // calls the scalar overload + +overload(1, "") // calls the parameter pack overload +``` + +The general overload subtype ranking rule applies after localized ranking, such as implicit conversions and optional promotions. That remains unchanged with this proposal. For example: + +```swift +func overload(_: T, _: Any) {} +func overload(_: repeat each T) {} + +overload(1, "") // prefers the parameter pack overload because the scalar overload would require an existential conversion +``` + +More complex scenarios can still result in ambiguities. For example, if multiple overloads match a function call, but each parameter list can be forwarded to the other, the call is ambiguous: + +```swift +func overload(_: repeat each T) {} +func overload(vals: repeat each T) {} + +overload() // error: ambiguous +``` + +Similarly, if neither overload can forward their parameter lists to the other, the call is ambiguous: + +```swift +func overload(str: String, _: repeat each T) {} +func overload(str: repeat each U) {} + +func test(_ z: Z) { + overload(str: "Hello, world!", z, z) // error: ambiguous +} +``` + +Generalizing the existing overload resolution ranking rules to parameter packs enables library authors to introduce new function overloads using parameter packs that generalize existing fixed-arity overloads while preserving the overload resolution behavior of existing code. + +## Effect on ABI stability + +This is still an area of open discussion, but we anticipate that generic functions with type parameter packs will not require runtime support, and thus will backward deploy. As work proceeds on the implementation, the above is subject to change. + +## Alternatives considered + +### Modeling packs as tuples with abstract elements + +Under this alternative design, packs are just tuples with abstract elements. This model is attractive because it adds expressivity to all tuple types, but there are some significant disadvantages that make packs hard to work with: + +* There is a fundamental ambiguity between forwarding a tuple with its elements flattened and passing a tuple as a single tuple value. This could be resolved by requiring a splat operator to forward the flattened elements, but it would still be valid in many cases to pass the tuple without flattening the elements. This may become a footgun, because you can easily forget to splat a tuple, which will become more problematic when tuples can conform to protocols. +* Because of the above issue, there is no clear way to zip tuples. This could be solved by having an explicit builtin to treat a tuple as a pack, which leads us back to needing a distinction between packs and tuples. + +The pack parameter design where packs are distinct from tuples also does not preclude adding flexibility to all tuple types. Converting tuples to packs and expanding tuple values are both useful features and are detailed in the future directions. + +### Syntax alternatives to `repeat each` + +The `repeat each` syntax produces fairly verbose variadic generic code. However, the `repeat` keyword is explicit signal that the pattern is repeated under substitution, and requiring the `each` keyword for pack references indicates which types or values will be substituted in the expansion. This syntax design helps enforce the mental model that pack expansions result in iteration over each element in the parameter pack at runtime. + +The following syntax alternatives were also considered. + +#### The `...` operator + +A previous version of this proposal used `...` as the pack expansion operator with no explicit syntax for pack elements in pattern types. This syntax choice follows precedent from C++ variadic templates and non-pack variadic parameters in Swift. However, there are some serious downsides of this choice, because ... is already a postfix unary operator in the Swift standard library that is commonly used across existing Swift code bases, which lead to the following ambiguities: + +1. **Pack expansion vs non-pack variadic parameter.** Using `...` for pack expansions in parameter lists introduces an ambiguity with the use of `...` to indicate a non-pack variadic parameter. This ambiguity can arise when expanding a type parameter pack into the parameter list of a function type. For example: + +```swift +struct X { } + +struct Ambiguity { + struct Inner { + typealias A = X<((T...) -> U)...> + } +} +``` + +Here, the `...` within the function type `(T...) -> U` could mean one of two things: + +* The `...` defines a (non-pack) variadic parameter, so for each element `Ti` in the parameter pack, the function type has a single (non-pack) variadic parameter of type `Ti`, i.e., `(Ti...) -> Ui`. So, `Ambiguity.Inner.A` would be equivalent to `X<(String...) -> Float, (Character...) -> Double>`. +* The `...` expands the parameter pack `T` into individual parameters for the function type, and no pack parameters remain after expansion. Only `U` is expanded by the outer `...`. So, `Ambiguity.Inner.A` would be equivalent to `X<(String, Character) -> Float, (String, Character) -> Double>`. + +2. **Pack expansion vs postfix closed-range operator.** Using `...` as the value expansion operator introduces an ambiguity with the postfix closed-range operator. This ambiguity can arise when `...` is applied to a value pack in the pattern of a value pack expansion, and the values in the pack are known to have a postfix closed-range operator, such as in the following code which passes a list of tuple arguments to `acceptAnything`: + +```swift +func acceptAnything(_: T...) {} + +func ranges(values: T..., otherValues: U...) where T: Comparable, shape(T...) == shape(U...) { + acceptAnything((values..., otherValues)...) +} +``` + +In the above code, `values...` in the expansion pattern could mean either: + +* The postfix `...` operator is called on each element in `values`, and the result is expanded pairwise with `otherValues` such that each argument has type `(PartialRangeFrom, U)` +* `values` is expanded into each tuple passed to `acceptAnything`, with each element of `otherValues` appended onto the end of the tuple, and each argument has type `(T... U)` + + +3. **Pack expansion vs operator `...>`.** Another ambiguity arises when a pack expansion type `T...` appears as the final generic argument in the generic argument list of a generic type in expression context: + +```swift +let foo = Foo() +``` + +Here, the ambiguous parse is with the token `...>`, which would necessitate changing the grammar so that `...>` is no longer considered as a single token, and instead parses as the token `...` followed by the token `>`. + + +#### Another operator + +One alternative is to use a different operator, such as `*`, instead of `...` + +```swift +func zip(firsts: T*, seconds: U*) -> ((T, U)*) { + return ((firsts, seconds)*) +} +``` + +The downsides to postfix `*` include: + +* `*` is extremely subtle +* `*` evokes pointer types / a dereferencing operator to programmers familiar with other languages including C/C++, Go, Rust, etc. +* Choosing another operator does not alleviate the ambiguities in expressions, because values could also have a postfix `*` operator or any other operator symbol, leading to the same ambiguity. + +#### Magic builtin `map` method + +The prevalence of `map` and `zip` in Swift makes this syntax an attractive option for variadic generics: + +```swift +func wrap(_ values: repeat each T) -> (repeat Wrapped) { + return values.map { Wrapped($0) } +} +``` + +The downsides of a magic `map` method are: + +* `.map` isn't only used for mapping elements to a new element, it's also used for direct forwarding, which is very different to the way `.map` is used today. An old design exploration for variadic generics used a map-style builtin, but allowed exact forwarding to omit the `.map { $0 }`. Privileging exact forwarding would be pretty frustrating, because you would need to add `.map { $0 }` as soon as you want to append other elements to either side of the pack expansion, and it wouldn't work for other values that you might want to turn into packs such as tuples or arrays. +* There are two very different models for working with packs; the same conceptual expansion has very different spellings at the type and value level, `repeat Wrapped` vs `values.map { Wrapped($0) }`. +* Magic map can only be applied to one pack at a time, leaving no clear way to zip packs without adding other builtins. A `zip` builtin would also be misleading, because expanding two packs in parallel does not need to iterate over the packs twice, but using `zip(t, u).map { ... }` looks that way. +* The closure-like syntax is misleading because it’s not a normal closure that you can write in the language. This operation is also very complex over packs with any structure, including concrete types, because the compiler either needs to infer a common generic signature for the closure that works for all elements, or it needs to separately type check the closure once for each element type. +* `map` would still need to be resolved via overload resolution amongst the existing overloads, so this approach doesn't help much with the type checking issues that `...` has. + +## Future directions + +### Variadic generic types + +This proposal only supports type parameter packs on functions. A complementary proposal will describe type parameter packs on generic structs, enums and classes. + +### Local value packs + +This proposal only supports value packs for function parameters. The notion of a value parameter pack readily generalizes to a local variable of pack expansion type, for example: + +```swift +func variadic(t: repeat each T) { + let tt: repeat each T = repeat each t +} +``` + +References to `tt` have the same semantics as references to `t`, and must only appear inside other pack expansion expressions. + +### Explicit type pack syntax + +In this proposal, type packs do not have an explicit syntax, and a type pack is always inferred through the type matching rules. However, we could explore adding an explicit pack syntax in the future: + +```swift +struct Variadic {} + +extension Variadic where T == {Int, String} {} // {Int, String} is a concrete pack +``` + +### Pack iteration + +All list operations can be expressed using pack expansion expressions by factoring code involving statements into a function or closure. However, this approach does not allow for short-circuiting, because the pattern expression will always be evaluated once for every element in the pack. Further, requiring a function or closure for code involving statements is unnatural. Allowing `for-in` loops to iterate over packs solves both of these problems. + +Value packs could be expanded into the source of a `for-in` loop, allowing you to iterate over each element in the pack and bind each value to a local variable: + +```swift +func allEmpty(_ array: repeat [each T]) -> Bool { + for a in repeat each array { + guard a.isEmpty else { return false } + } + + return true +} +``` + +The type of the local variable `a` in the above example is an `Array` of an opaque element type with the requirements that are written on `each T`. For the *i*th iteration, the element type is the *i*th type parameter in the type parameter pack `T`. + +### Pack element projection + +Use cases for variadic generics that break up pack iteration across function calls, require random access, or operate over concrete packs can be supported in the future by projecting individual elements out from a parameter pack. Because elements of the pack have different types, there are two approaches to pack element projection; using an `Int` index which will return the dynamic type of the element, and using a statically typed index which is parameterized over the requested pack element type. + +#### Dynamic pack indexing with `Int` + +Dynamic pack indexing is useful when the specific type of the element is not known, or when all indices must have the same type, such as for index manipulation or storing an index value. Packs could support subscript calls with an `Int` index, which would return the dynamic type of the pack element directly as the opened underlying type that can be assigned to a local variable with opaque type. Values of this type need to be erased or cast to another type to return an element value from the function: + +```swift +func element(at index: Int, in t: repeat each T) -> any P { + // The subscript returns 'some P', which is erased to 'any P' + // based on the function return type. + let value: some P = t[index] + return value +} +``` + +#### Typed pack element projection using key-paths + +Some use cases for pack element projection know upfront which type within the pack will be projected, and can use a statically typed pack index. A statically typed pack index could be represented with `KeyPath` or a new `PackIndex` type, which is parameterized over the base type for access (i.e. the pack), and the resulting value type (i.e. the element within the pack to project). Pack element projection via key-paths falls out of 1) positional tuple key-paths, and 2) expanding packs into tuple values: + +```swift +struct Tuple { + var elements: (repeat each Elements) + + subscript(keyPath: KeyPath<(repeat each Elements), Value>) -> Value { + return elements[keyPath: keyPath] + } +} +``` + +The same positional key-path application could be supported directly on value packs. + +### Value expansion operator + +This proposal only supports the expansion operator on type parameter packs and value parameter packs, but there are other values that represent a list of zero or more values that the expansion operator would be useful for, including tuples and arrays. It would be desirable to introduce a new kind of expression that receives a scalar value and produces a value of pack expansion type. + +Here, we use the straw-man syntax `.element` for accessing tuple elements as a pack: + +```swift +func foo( + _ t: repeat each T, + _ u: repeat each U +) {} + +func bar1( + t: (repeat each T), + u: (repeat each U) +) { + repeat foo(each t.element, each u.element) +} + +func bar2( + t: (repeat each T), + u: (repeat each U) +) { + repeat foo(each t.element, repeat each u.element) +} +``` + +Here, `bar1(t: (1, 2), u: ("a", "b"))` will evaluate: + +```swift +foo(1, "a") +foo(2, "b") +``` + +While `bar2(t: (1, 2), u: ("a", "b"))` will evaluate: + +```swift +foo(1, "a", "b") +foo(2, "a", "b") +``` + +The distinction can be understood in terms of our notion of _captures_ in pack expansion expressions. + +### Pack destructuring operations + +The statically-known shape of a pack can enable destructing packs with concrete shape into the component elements: + +```swift +struct List { + let element: repeat each Element +} + +extension List { + func firstRemoved() -> List where (repeat each Element) == (First, repeat each Rest) { + let (first, rest) = (repeat each element) + return List(repeat each rest) + } +} + +let list = List(1, "Hello", true) +let firstRemoved = list.firstRemoved() // 'List("Hello", true)' +``` + +The body of `firstRemoved` decomposes `Element` into the components of its shape -- one value of type `First` and a value pack of type `repeat each Rest` -- effectively removing the first element from the list. + +### Tuple conformances + +Parameter packs, the above future directions, and a syntax for declaring tuple conformances based on [parameterized extensions](https://github.com/apple/swift/blob/main/docs/GenericsManifesto.md#parameterized-extensions) over non-nominal types enable implementing custom tuple conformances: + +```swift +extension (repeat each T): Equatable { + public static func ==(lhs: Self, rhs: Self) -> Bool { + for (l, r) in repeat (each lhs.element, each rhs.element) { + guard l == r else { return false } + } + return true + } +} +``` + +## Revision history + +Changes to the [first reviewed revision](https://github.com/swiftlang/swift-evolution/blob/b6ca38b9eee79650dce925e7aa8443a6a9e5e6ea/proposals/0393-parameter-packs.md): + +* The `repeat` keyword is required for generic requirement expansions to distinguish requirement expansions from single requirements on an individual pack element nested inside of a pack expansion expression. +* Overload resolution prefers scalar overloads when the scalar overload is considered a subtype of a parameter pack overload. + + +## Acknowledgments + +Thank you to Robert Widmann for exploring the design space of modeling packs as tuples, and to everyone who participated in earlier design discussions about variadic generics in Swift. Thank you to the many engineers who contributed to the implementation, including Sophia Poirier, Pavel Yaskevich, Nate Chandler, Hamish Knight, and Adrian Prantl. diff --git a/proposals/0394-swiftpm-expression-macros.md b/proposals/0394-swiftpm-expression-macros.md new file mode 100644 index 0000000000..1698611842 --- /dev/null +++ b/proposals/0394-swiftpm-expression-macros.md @@ -0,0 +1,203 @@ +# Package Manager Support for Custom Macros + +* Proposal: [SE-0394](0394-swiftpm-expression-macros.md) +* Authors: [Boris Buegling](https://github.com/neonichu), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 5.9)** +* Implementation: **Available behind pre-release tools-version** ([apple/swift-package-manager#6185](https://github.com/apple/swift-package-manager/pull/6185), [apple/swift-package-manager#6200](https://github.com/apple/swift-package-manager/pull/6200)) +* Review: ([pitch 1](https://forums.swift.org/t/pitch-package-manager-support-for-custom-macros/63482)) ([pitch 2](https://forums.swift.org/t/pitch-2-package-manager-support-for-custom-macros/63868)) ([review](https://forums.swift.org/t/se-0394-package-manager-support-for-custom-macros/64170)) ([acceptance](https://forums.swift.org/t/accepted-se-0394-package-manager-support-for-custom-macros/64589)) + +## Introduction + +Macros provide a way to extend Swift by performing arbitrary syntactic transformations on input source code to produce new code. One example for this are expression macros which were previously proposed in [SE-0382](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md). This proposal covers how custom macros are defined, built and distributed as part of a Swift package. + +## Motivation + +[SE-0382](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) and [A Possible Vision for Macros in Swift](https://gist.github.com/DougGregor/4f3ba5f4eadac474ae62eae836328b71) covered the motivation for macros themselves, defining them as part of a package will offer a straightforward way to reuse and distribute macros as source code. + +## Proposed solution + +Macros implemented in an external program can be declared as part of a package via a new macro target type, defined in +the `CompilerPluginSupport` library: + +```swift +public extension Target { + /// Creates a macro target. + /// + /// - Parameters: + /// - name: The name of the macro. + /// - dependencies: The macro's dependencies. + /// - path: The path of the macro, relative to the package root. + /// - exclude: The paths to source and resource files you want to exclude from the macro. + /// - sources: The source files in the macro. + /// - swiftSettings: The Swift settings for this macro. + /// - linkerSettings: The linker settings for this macro. + /// - plugins: The plugins used by this macro. + static func macro( + name: String, + dependencies: [Dependency] = [], + path: String? = nil, + exclude: [String] = [], + sources: [String]? = nil, + swiftSettings: [SwiftSetting]? = nil, + linkerSettings: [LinkerSetting]? = nil, + plugins: [PluginUsage]? = nil + ) -> Target { ... } +} +``` + +Similar to package plugins ([SE-0303 "Package Manager Extensible Build Tools"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md)), macro plugins are built as executables for the host (i.e, where the compiler is run). The compiler receives the paths to these executables from the build system and will run them on demand as part of the compilation process. Macro executables are automatically available for any target that transitively depends on them via the package manifest. + +A minimal package containing the implementation, definition and client of a macro would look like this: + +```swift +import PackageDescription +import CompilerPluginSupport + +let package = Package( + name: "MacroPackage", + dependencies: [ + .package(url: "https://github.com/apple/swift-syntax", from: "509.0.0"), + ], + targets: [ + .macro(name: "MacroImpl", + dependencies: [ + .product(name: "SwiftSyntaxMacros", package: "swift-syntax"), + .product(name: "SwiftCompilerPlugin", package: "swift-syntax") + ]), + .target(name: "MacroDef", dependencies: ["MacroImpl"]), + .executableTarget(name: "MacroClient", dependencies: ["MacroDef"]), + .testTarget(name: "MacroTests", dependencies: ["MacroImpl"]), + ] +) +``` + +Macro implementations will be executed in a sandbox [similar to package plugins](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md#security), preventing file system and network access. This is a practical way of encouraging macros to not depend on any state other than the specific macro expansion node they are given to expand and its child nodes (but not its parent nodes), and the information specifically provided by the macro expansion context. If in the future macros need access to other information, this will be accomplished by extending the macro expansion context, which also provides a mechanism for the compiler to track what information the macro actually queried. + +Any code from macro implementations can be tested by declaring a dependency on the macro target from a test, this works similarly to the [testing of executable targets](https://github.com/apple/swift-package-manager/pull/3316). + +## Detailed Design + +SwiftPM builds each macro as an executable for the host platform, applying certain additional compiler flags. Macros are expected to depend on SwiftSyntax using a versioned dependency that corresponds to a particular major Swift release. Note that SwiftPM's dependency resolution is workspace-wide, so all macros (and potentially other clients) will end up consolidating on one particular version of SwiftSyntax. Each target that transitively depends on a macro will have access to it, concretely this happens by SwiftPM passing `-load-plugin-executable` to the compiler to specify which executable contains the implementation of a certain macro module (e.g. `-load-plugin-executable /path/to/package/.build/debug/MacroImpl#MacroImpl` where the argument after the hash symbol is a comma separated list of module names which can be referenced by the `module` parameter of external macro declarations). The macro definition refers to the module and concrete type via an `#externalMacro` declaration which allows any dependency of the defining target to have access to the concrete macro. If any target of a library product depends on a macro, clients of said library will also get access to any public macros. Macros can have dependencies like any other target, but product dependencies of macros need to be statically linked, so explicitly dynamic library products cannot be used by a macro target. + +Concretely, the code for the macro package shown earlier would contain a macro implementation looking like this: + +```swift +import SwiftSyntax +import SwiftCompilerPlugin +import SwiftSyntaxBuilder +import SwiftSyntaxMacros + +@main +struct MyPlugin: CompilerPlugin { + var providingMacros: [Macro.Type] = [FontLiteralMacro.self] +} + +/// Implementation of the `#fontLiteral` macro, which is similar in spirit +/// to the built-in expressions `#colorLiteral`, `#imageLiteral`, etc., but in +/// a small macro. +public struct FontLiteralMacro: ExpressionMacro { + public static func expansion( + of macro: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + let argList = replaceFirstLabel( + of: macro.argumentList, + with: "fontLiteralName" + ) + let initSyntax: ExprSyntax = ".init(\(argList))" + if let leadingTrivia = macro.leadingTrivia { + return initSyntax.with(\.leadingTrivia, leadingTrivia) + } + return initSyntax + } +} + +/// Replace the label of the first element in the tuple with the given +/// new label. +private func replaceFirstLabel( + of tuple: TupleExprElementListSyntax, + with newLabel: String +) -> TupleExprElementListSyntax { + guard let firstElement = tuple.first else { + return tuple + } + + return tuple.replacing( + childAt: 0, + with: firstElement.with(\.label, .identifier(newLabel)) + ) +} +``` + +The macro definition would look like this: + +```swift +public enum FontWeight { + case thin + case normal + case medium + case semiBold + case bold +} + +public protocol ExpressibleByFontLiteral { + init(fontLiteralName: String, size: Int, weight: FontWeight) +} + +/// Font literal similar to, e.g., #colorLiteral. +@freestanding(expression) public macro fontLiteral(name: String, size: Int, weight: FontWeight) -> T = #externalMacro(module: "MacroImpl", type: "FontLiteralMacro") + where T: ExpressibleByFontLiteral +``` + +And the client of the macro would look like this: + +```swift +import MacroDef + +struct Font: ExpressibleByFontLiteral { + init(fontLiteralName: String, size: Int, weight: MacroDef.FontWeight) { + } +} + +let _: Font = #fontLiteral(name: "Comic Sans", size: 14, weight: .thin) +``` + +SwiftSyntax's versioning scheme is based on Swift major versions (e.g. 509.0.0 for Swift 5.9). + +If a package depends on two macros using the `from` version dependency and minor versions of a macro use different versions of SwiftSyntax, users should automatically get a version that's compatible with all macros. For example consider the following where a package depends on both Macro 1 and Macro 2 using `from: "1.0.0"` + +``` +Macro 1 SwiftSyntax Macro 2 + +1.0 --------------> 509.0.0 <-------------- 1.0 + 509.0.1 <-------------- 1.1 + 510.0.0 <-------------- 1.2 +``` + +In this case, SwiftPM would choose version 1.0 for Macro 1, version 1.1 for Macro 2 and end up with version 509.0.1 for SwiftSyntax. We're going to monitor how the versioning story plays out in practice and may take further action in SwiftSyntax or SwiftPM's dependency resolution if the concrete need arises. + + +## Impact on existing packages + +Since macro plugins are entirely additive, there's no impact on existing packages. + +## Alternatives considered + +### Package plugins + +The original pitch of expression macros considered declaring macros by introducing a new capability to [package plugins](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md), but since the execution model is significantly different and the APIs used for macros are external to SwiftPM, this idea was discarded. + +### `.macroTarget()` + +We're (slowly) trying to move away from having the target suffix since it is implied by the context. This is already the case for plugin targets and eventually we'd like to have e.g. `.test()` as well. This would also make the target APIs be more in line with the product ones, where e.g. we don't use `.libraryProduct()`. + +### Dependencies on macro targets + +In [SE-0303](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md), we introduced the `plugins` parameter for build order dependencies on plugins, so it could have make sense to use a `macros` parameter for dependencies on macros. However, introducing bespoke API for each type of host-side content used during the build does not seem scalable. We also already have precedence of executables being part of `dependencies` even though that dependency is strictly for build ordering (with the exception of tests, which also applies to macros). Because of this, dependencies on macros are declared via the `dependencies` parameter, however it could be interesting to revisit a separation of build order and linked dependencies in the future. + +## Future Directions + +### Generalized support for additional manifest API + +The macro target type is provided by a new library `CompilerPluginSupport` as a starting point for making package manifests themselves more extensible. Support for product and target type plugins should eventually be generalized to allow other types of externally defined specialized target types, such as, for example, a Windows application. diff --git a/proposals/0395-observability.md b/proposals/0395-observability.md new file mode 100644 index 0000000000..1a3acd1c9f --- /dev/null +++ b/proposals/0395-observability.md @@ -0,0 +1,523 @@ +# Observation + +* Proposal: [SE-0395](0395-observability.md) +* Authors: [Philippe Hausler](https://github.com/phausler), [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.9)** +* Review: ([pitch](https://forums.swift.org/t/pitch-observation-revised/63757)), ([first review](https://forums.swift.org/t/se-0395-observability/64342/)), ([second review](https://forums.swift.org/t/second-review-se-0395-observability/65261/)), ([acceptance](https://forums.swift.org/t/accepted-with-revision-se-0395-observability/66760)) + +#### Changes + +* Version 1: [Initial pitch](https://forums.swift.org/t/pitch-observation/62051) +* Version 2: Previously Observation registered observers directly to `Observable`, the new approach registers observers to an `Observable` via a `ObservationTransactionModel`. These models control the "edge" of where the change is emitted. They are the responsible component for notifying the observers of events. This allows the observers to focus on just the event and not worry about "leading" or "trailing" (will/did) "edges" of the signal. Additionally the pitch was shifted from the type wrapper feature over to the more appropriate macro features. +* Version 3: The `Observer` protocol and `addObserver(_:)` method are gone in favor of providing async sequences of changes and transactions. +* Version 4: In order to support observation for subclasses and to provide space to address design question around the asynchronous `values(for:)` and `changes(for:)` methods, the proposal now focuses on an `Observable` marker protocol and the `withTracking(_:changes:)` function. + +#### Suggested Reading + +* [Expression Macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) +* [Attached Macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md) + +## Introduction + +Making responsive apps often requires the ability to update the presentation when underlying data changes. The _observer pattern_ allows a subject to maintain a list of observers and notify them of specific or general state changes. This has the advantages of not directly coupling objects together and allowing implicit distribution of updates across potential multiple observers. An observable object needs no specific information about its observers. + +This design pattern is a well-traveled path by many languages, and Swift has an opportunity to provide a robust, type-safe, and performant implementation. This proposal defines what an observable reference is, what an observer needs to conform to, and the connection between a type and its observers. + +## Motivation + +There are already a few mechanisms for observation in Swift. These include key-value observing (KVO) and `ObservableObject`, but each of those have limitations. KVO can only be used with `NSObject` descendants, and `ObservableObject` requires using Combine, which is restricted to Darwin platforms and does not use current Swift concurrency features. By taking experience from those existing systems, we can build a more generally useful feature that applies to all Swift reference types, not just those that inherit from `NSObject`, and have it work cross-platform with the advantages from language features like `async`/`await`. + +The existing systems get a number of behaviors and characteristics right. However, there are a number of areas that can provide a better balance of safety, performance, and expressiveness. For example, grouping dependent changes into an independent transaction is a common task, but this is complex when using Combine and unsupported when using KVO. In practice, observers want access to transactions, with the ability to specify how transactions are interpreted. + +Annotations clarify what is observable, but can also be cumbersome. For example, Combine requires not just that a type conform to `ObservableObject`, but also requires each property that is being observed to be marked as `@Published`. Furthermore, computed properties cannot be directly observed. In reality, having non-observed fields in a type that is observable is uncommon. + +Throughout this document, references to both KVO and Combine will illustrate what capabilities are benefits and can be incorporated into the new approach, and what drawbacks are possible to solve in a more robust manner. + +### Prior Art + +#### KVO + +Key-value observing has served the Cocoa/Objective-C programming model well, but is limited to class hierarchies that inherit from `NSObject`. The APIs only offer the intercepting of events, meaning that the notification of changes is between the `willSet` and `didSet` events. KVO has great flexibility with granularity of events, but lacks in composability. KVO observers must also inherit from `NSObject`, and rely on the Objective-C runtime to track the changes that occur. Even though the interface for KVO has been updated to utilize the more modern Swift strongly-typed key paths, under the hood its events are still stringly typed. + +#### Combine + +Combine's `ObservableObject` produces changes at the beginning of a change event, so all values are delivered before the new value is set. While this serves SwiftUI well, it is restrictive for non-SwiftUI usage and can be surprising to developers first encountering that behavior. `ObservableObject` also requires all observed properties to be marked as `@Published` to interact with change events. In most cases, this requirement is applied to every single property and becomes redundant to the developer; folks writing an `ObservableObject` conforming type must repeatedly (with little to no true gained clarity) annotate each property. In the end, this results in meaning fatigue of what is or isn't a participating item. + +## Proposed solution + +A formalized observer pattern needs to support the following capabilities: + +* Marking a type as observable +* Tracking changes within an instance of an observable type +* Observing and utilizing those changes from somewhere else + +In addition, the design and implementation should meet these criteria: + +* Observable types are easy to annotate (without fatigue of meaning) +* Access control should be respected +* Adopting the features for observability should require minimal effort to get started +* Using advanced features should progressively disclose to more complex systems +* Observation should be able to handle more than one observed member at once +* Observation should be able to work with computed properties that reference other properties +* Observation should be able to work with computed properties that store their values in external storage +* Integration of observation should work in transactions of graphs and not just singular objects + +We propose a new standard library module named `Observation` that includes the required functionality to implement such a pattern. + +Primarily, a type can declare itself as observable simply by using the `@Observable` macro annotation: + +```swift +@Observable class Car { + var name: String + var awards: [Award] +} +``` + +The `@Observable` macro implements conformance to the `Observable` marker protocol and tracking for each stored property. Unlike `ObservableObject` and `@Published`, the properties of an `@Observable` type do not need to be individually marked as observable. Instead, all stored properties are implicitly observable. + +The `Observation` module also provides the top-level function `withObservationTracking`, which detects accesses to tracked properties within a specific scope. Once those properties are identified, any changes to the tracked properties triggers a call to the provided `onChange` closure. + +```swift +let cars: [Car] = ... + +@MainActor +func renderCars() { + withObservationTracking { + for car in cars { + print(car.name) + } + } onChange: { + Task { @MainActor in + renderCars() + } + } +} +``` + +In the example above, the `render` function accesses each car's `name` property. When any of the cars change `name`, the `onChange` closure is then called on the first change. However, if a car has an award added, the `onChange` call won't happen. This design supports uses that require implicit observation tracking, ensuring that updates are only performed in response to relevant changes. + +## Detailed Design + +The `Observable` protocol, `@Observable` macro, and a handful of supporting types comprise the `Observation` module. As described below, this design allows adopters to use a straightforward syntax for simple cases, while allowing full control over the details the implementation when necessary. + +### `Observable` protocol + +Observable types conform to the `Observable` marker protocol. While the `Observable` protocol doesn't have formal requirements, it includes a semantic requirement that conforming types must implement tracking for each stored property using an `ObservationRegistrar`. Most types can meet that requirement simply by using the `@Observable` macro: + +```swift +@Observable public final class MyObject { + public var someProperty = "" + public var someOtherProperty = 0 + fileprivate var somePrivateProperty = 1 +} +``` + +### `@Observable` Macro + +In order to make implementation as simple as possible, the `@Observable` macro automatically synthesizes conformance to the `Observable` protocol, transforming annotated types into a type that can be observed. When fully expanded, the `@Observable` macro does the following: + +- declares conformance to the `Observable` protocol, +- adds a property for the registrar, +- and adds internal helper methods for tracking accesses and mutations. + +Additionally, for each stored property, the macro: + +- annotates each stored property with the `@ObservationTracked` macro, +- converts each stored property to a computed property, +- and adds an underscored, `@ObservationIgnored` version of each stored property. + +Since all of the code generated by the macro could be manually written, developers can write or customize their own implementation when they need more fine-grained control. + +As an example of the `@Observable` macro expansion, consider the following `Model` type: + +```swift +@Observable class Model { + var order: Order? + var account: Account? + + var alternateIconsUnlocked: Bool = false + var allRecipesUnlocked: Bool = false + + func purchase(alternateIcons: Bool, allRecipes: Bool) { + alternateIconsUnlocked = alternateIcons + allRecipesUnlocked = allRecipes + } +} +``` + +Expanding the `@Observable` macro, as well as the generated macros, results in the following declaration: + +```swift +class Model: Observable { + internal let _$observationRegistrar = ObservationRegistrar() + + internal func access( + keyPath: KeyPath + ) { + _$observationRegistrar.access(self, keyPath: keyPath) + } + + internal func withMutation( + keyPath: KeyPath, + _ mutation: () throws -> T + ) rethrows -> T { + try _$observationRegistrar.withMutation(of: self, keyPath: keyPath, mutation) + } + + var order: Order? { + get { + self.access(keyPath: \.order) + return _order + } + set { + self.withMutation(keyPath: \.order) { + _order = newValue + } + } + } + + var account: Account? { + get { + self.access(keyPath: \.account) + return _account + } + set { + self.withMutation(keyPath: \.account) { + _account = newValue + } + } + } + + var alternateIconsUnlocked: Bool { + get { + self.access(keyPath: \.alternateIconsUnlocked) + return _alternateIconsUnlocked + } + set { + self.withMutation(keyPath: \.alternateIconsUnlocked) { + _alternateIconsUnlocked = newValue + } + } + } + + var allRecipesUnlocked: Bool { + get { + self.access(keyPath: \.allRecipesUnlocked) + return _allRecipesUnlocked + } + set { + self.withMutation(keyPath: \.allRecipesUnlocked) { + _allRecipesUnlocked = newValue + } + } + } + + var _order: Order? + var _account: Account? + + var _alternateIconsUnlocked: Bool = false + var _allRecipesUnlocked: Bool = false +} +``` + +### `@ObservationTracked` and `@ObservationIgnored` macros + +The `Observation` module includes two additional macros that can annotate properties of observable types. The `@ObservationTracked` macro is added to stored properties by the `@Observable` macro expansion, and, when expanded, converts a stored property to a computed one with access and mutation tracking. Developers generally won't use `@ObservationTracked` themselves. + +The `@ObservationIgnored` macro, on the other hand, doesn't add anything to a source file when expanded. Instead, it acts as a marker for properties that shouldn't be tracked. The `@Observable` macro expansion adds `@ObservationIgnored` to the underscored stored properties it creates. Developers can also apply `@ObservationIgnored` to stored properties that shouldn't be included in observation tracking. + +### Computed properties + +Computed properties that derive their values from stored properties are automatically tracked due to their reliance on tracked properties. Computed properties that source their value from remote storage or via indirection, however, must manually add tracking using the generated `access(keyPath:)` and `withMutation(keyPath:)` methods. + +For example, consider the `AtomicModel` in the following code sample. `AtomicModel` stores a score in an `AtomicInt`, with a computed property providing an `Int` interface. The atomic property is annotated with the `@ObservationIgnored` macro because it isn't useful to track the constant value for observation. For the computed `score` property, which is the public interface of the type, the getter and setter include manually-written calls to track accesses and mutations. + +```swift +@Observable +public class AtomicModel { + @ObservationIgnored + fileprivate let _scoreStorage = AtomicInt(initialValue: 0) + + public var score: Int { + get { + self.access(keyPath: \.score) + return _scoreStorage.value + } + set { + self.withMutation(keyPath: \.score) { + _scoreStorage.value = newValue + } + } + } +} +``` + +### `willSet`/`didSet` + +Observation is supported for properties with `willSet` and `didSet` property observers. For example, the `@Observable` macro on the `PropertyExample` type here: + +```swift +@Observable class PropertyExample { + var a = 0 { + willSet { print("will set triggered") } + didSet { print("did set triggered") } + } + var b = 0 + var c = "" +} +``` + +...transforms the `a` property as follows, preserving the `willSet` and `didSet` behavior: + +```swift +var a: Int { + get { + self.access(keyPath: \.a) + return _a + } + set { + self.withMutation(keyPath: \.a) { + _a = newValue + } + } +} + +var _a = 0 { + willSet { print("will set triggered") } + didSet { print("did set triggered") } +} +``` + +### Initializers + +Because observable types generally use the implicitly generated initializers, the `@Observable` macro requires that all stored properties have a default value. This guarantees definitive initialization, so that additional initializers can be added to observable types in an extension. + +The default value requirement could be relaxed in a future version; see the Future Directions section for more. + +### Subclasses + +Developers can create `Observable` subclasses of either observable or non-observable types. Only the properties of a type that implements the `Observable` tracking requirements will be observed. That is, when working with an observable subclass of a non-observable type, the superclass's stored properties will not be tracked under observation. + +### `withObservationTracking(_:onChange:)` + +In order to provide automatically scoped observation, the `ObservationModule` provides a function to capture accesses to properties within a given scope, and then call out upon the first change to any of those properties. This can be used by user interface libraries, such as SwiftUI, to provide updates to the specific properties which are accessed within a particular scope, limiting interface updates or renders to only the relevant changes. For more detail, see the SDK Impact section below. + +```swift +public func withObservationTracking( + _ apply: () -> T, + onChange: @autoclosure () -> @Sendable () -> Void +) -> T +``` + +The `withObservationTracking` function takes two closures. Any access to a tracked property within the `apply` closure will flag the property; any change to a flagged property will trigger a call to the `onChange` closure. + +Accesses are recognized for: +- tracked properties on observable objects +- tracked properties of properties that have observable type +- properties that are accessed via computed property accesses + +For example, this `Person` class has multiple tracked properties, some of which are internal: + +```swift +@Observable public class Person: Sendable { + internal var firstName = "" + internal var lastName = "" + public var age: Int? + + public var fullName: String { + "\(firstName) \(lastName)" + } + + public var friends: [Person] = [] +} +``` + +Accessing the `fullName` and `friends` properties will result in the `firstName`, `lastName`, and `friends` properties being tracked for changes: + +```swift +@MainActor +func renderPerson(_ person: Person) { + withObservationTracking { + print("\(person.fullName) has \(person.friends.count) friends.") + } onChange: { + Task { @MainActor in + renderPerson(person) + } + } +} +``` + +Whenever the person's `firstName` or `lastName` properties are updated, the `onChange` closure will be called, even though those properties are internal, since their accesses are linked to a public computed property. Mutations to the `friends` array will also cause a call to `onChange`, though changes to individual members of the array are not tracked. + +### `ObservationRegistrar` + +`ObservationRegistrar` is the required storage for tracking accesses and mutations. The `@Observable` macro synthesizes a registrar to handle these mechanisms as a generalized feature. By default, the registrar is thread safe and must be as `Sendable` as containers could potentially be; therefore it must be designed to handle independent isolation for all actions. + +```swift +public struct ObservationRegistrar: Sendable { + public init() + + public func access( + _ subject: Subject, + keyPath: KeyPath + ) + + public func willSet( + _ subject: Subject, + keyPath: KeyPath + ) + + public func didSet( + _ subject: Subject, + keyPath: KeyPath + ) + + public func withMutation( + of subject: Subject, + keyPath: KeyPath, + _ mutation: () throws -> T + ) rethrows -> T +} +``` + +The `access` and `withMutation` methods identify transactional accesses. These methods register access to the underlying tracking system for access and identify mutations to the transactions registered for observers. + +## SDK Impact (a preview of SwiftUI interaction) + +When using the existing `ObservableObject`-based observation, there are a number of edge cases that can be surprising unless developers have an in-depth understanding of SwiftUI. Formalizing observation can make these edge cases considerably more approachable by reducing the complexity of the different systems needed to be understood. + +The following is adapted from the [Fruta sample app](https://developer.apple.com/documentation/swiftui/fruta_building_a_feature-rich_app_with_swiftui), modified for clarity: + +```swift +class Model: ObservableObject { + @Published var order: Order? + @Published var account: Account? + + var hasAccount: Bool { + return userCredential != nil && account != nil + } + + @Published var favoriteSmoothieIDs = Set() + @Published var selectedSmoothieID: Smoothie.ID? + + @Published var searchString = "" + + @Published var isApplePayEnabled = true + @Published var allRecipesUnlocked = false + @Published var unlockAllRecipesProduct: Product? +} + +struct SmoothieList: View { + var smoothies: [Smoothie] + @ObservedObject var model: Model + + var listedSmoothies: [Smoothie] { + smoothies + .filter { $0.matches(model.searchString) } + .sorted(by: { $0.title.localizedCompare($1.title) == .orderedAscending }) + } + + var body: some View { + List(listedSmoothies) { smoothie in + ... + } + } +} +``` + +The `@Published` attribute identifies each field that participates in changes in the object, but it does not provide any differentiation or distinction as to the source of changes. This unfortunately results in additional layouts, rendering, and updates. + +The proposed API not only reduces the `@Published` repetition, but also simplifies the SwiftUI view code too! With the proposed `@Observable` macro, the previous example can instead be written as the following: + +```swift +@Observable class Model { + var order: Order? + var account: Account? + + var hasAccount: Bool { + userCredential != nil && account != nil + } + + var favoriteSmoothieIDs: Set = [] + var selectedSmoothieID: Smoothie.ID? + + var searchString = "" + + var isApplePayEnabled = true + var allRecipesUnlocked = false + var unlockAllRecipesProduct: Product? +} + +struct SmoothieList: View { + var smoothies: [Smoothie] + var model: Model + + var listedSmoothies: [Smoothie] { + smoothies + .filter { $0.matches(model.searchString) } + .sorted(by: { $0.title.localizedCompare($1.title) == .orderedAscending }) + } + + var body: some View { + List(listedSmoothies) { smoothie in + ... + } + } +} +``` + +There are some other interesting differences that follow from using the proposed observation system. For example, tracking observation of access within a view can be applied to an array, an optional, or even a custom type. This opens up new and interesting ways that developers can utilize SwiftUI more easily. + +This is a potential future direction for SwiftUI, but is not part of this proposal. + +## Source compatibility + +This proposal is additive and provides no impact to existing source code. + +## Effect on ABI stability + +This proposal is additive and no impact is made upon existing ABI stability. This does have implication to the marking of inline to functions and back-porting of this feature. In the cases where it is determined to be performance critical to the distribution of change events the methods will be marked as inlineable. + +Changing a type from not observable to `@Observable` has the same ABI impact as changing a property from stored to computed (which is not ABI breaking). Removing `@Observable` not only transitions from computed to stored properties but also removes a conformance (which is ABI breaking). + +## Effect on API resilience + +This proposal is additive and no impact is made upon existing API resilience. The types that adopt `@Observable` cannot remove it without breaking API contract. + +## Location of API + +This API will be housed in a module that is part of the Swift language but outside of the standard library. To use this module `import Observation` must be used (and provisionally using the preview `import _Observation`). + +## Future Directions + +The requirement that all stored properties of an observable type have initial values could be relaxed in the future, if language features are added that would support that. For example, property wrappers have a feature that allows their underlying wrapped value to be provided in an initializer rather than as a default value. Generalizing that feature to all properties could allow the `@Observable` macro to enable a more typical initialization implementation. + +Another area of focus for future enhancements is support for observable `actor` types. This would require specific handling for key paths that currently does not exist for actors. + +An earlier version of this proposal included asynchronous sequences of coalesced transactions and individual property changes, named `values(for:)` and `changes(for:)`. Similar invariant-preserving asynchronous sequences could be added in a future proposal. + +## Alternatives considered + +An earlier consideration instead of defining transactions used direct will/did events to the observer. This, albeit being more direct, promoted mechanisms that did not offer the correct granularity for supporting the required synchronization between dependent members. It was determined that building transactions are worth the extra complexity to encourage developers using the API to consider what models for transactionality they need, instead of thinking just in terms of will/did events. + +Another design included an `Observer` protocol that could be used to build callback-style observer types. This has been eliminated in favor of the `AsyncSequence` approach. + +The `ObservedChange` type could have the `Sendable` requirement relaxed by making the type only conditionally `Sendable` and then allowing access to the subject in all cases; however this poses some restriction to the internal implementations and may have a hole in the sendable nature of the type. Since it is viewed that accessing values is most commonly by one property the values `AsyncSequence` fills most of that role and for cases where more than one field is needed to be accessed on a given actor the iteration can be done with a weak reference to the observable subject. + +## Acknowledgments + +* [Holly Borla](https://github.com/hborla) - For providing fantastic ideas on how to implement supporting infrastructure to this pitch +* [Pavel Yaskevich](https://github.com/xedin) - For tirelessly iterating on prototypes for supporting compiler features +* Rishi Verma - For bouncing ideas and helping with the design of integrating this idea into other work +* [Kyle Macomber](https://github.com/kylemacomber) - For connecting resources and providing useful feedback +* Matt Ricketson - For helping highlight some of the inner guts of SwiftUI + +## Related systems + +* [Swift `Combine.ObservableObject`](https://developer.apple.com/documentation/combine/observableobject/) +* [Objective-C Key Value Observing](https://developer.apple.com/documentation/objectivec/nsobject/nskeyvalueobserving?language=objc) +* [C# `IObservable`](https://learn.microsoft.com/en-us/dotnet/api/system.iobservable-1?view=net-6.0) +* [Rust `Trait rx::Observable`](https://docs.rs/rx/latest/rx/trait.Observable.html) +* [Java `Observable`](https://docs.oracle.com/javase/7/docs/api/java/util/Observable.html) +* [Kotlin `observable`](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.properties/-delegates/observable.html) diff --git a/proposals/0396-never-codable.md b/proposals/0396-never-codable.md new file mode 100644 index 0000000000..be3ecd86ea --- /dev/null +++ b/proposals/0396-never-codable.md @@ -0,0 +1,59 @@ +# Conform `Never` to `Codable` + +* Proposal: [SE-0396](0396-never-codable.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#64899](https://github.com/apple/swift/pull/64899) +* Review: ([pitch](https://forums.swift.org/t/pitch-conform-never-to-codable/64056)) ([review](https://forums.swift.org/t/se-0396-conform-never-to-codable/64469)) ([acceptance](https://forums.swift.org/t/accepted-se-0396-conform-never-to-codable/64848)) + +## Introduction + +Extend `Never` so that it conforms to the `Encodable` and `Decodable` protocols, together known as `Codable`. + +## Motivation + +Swift can synthesize `Codable` conformance for any type that has `Codable` members. Generic types often participate in this synthesized conformance by constraining their generic parameters, like this `Either` type: + +```swift +enum Either { + case left(A) + case right(B) +} + +extension Either: Codable where A: Codable, B: Codable {} +``` + +In this way, `Either` instances where both generic parameters are `Codable` are `Codable` themselves, such as an `Either`. However, since `Never` isn't `Codable`, using `Never` as one of the parameters blocks the conditional conformance, even though it would be perfectly fine to encode or decode a type like `Either`. + +## Proposed solution + +The standard library should add `Encodable` and `Decodable` conformance to the `Never` type. + +## Detailed design + +The `Encodable` conformance is simple — since it's impossible to have a `Never` instance, the `encode(to:)` method can simply be empty. + +The `Decodable` protocol requires the `init(from:)` initializer, which clearly can't create a `Never` instance. Because trying to decode invalid input isn't a programmer error, a fatal error would be inappropriate. Instead, the implementation throws a `DecodingError.typeMismatch` error if decoding is attempted. + +## Source compatibility + +If existing code already declares `Codable` conformance, that code will begin to emit a warning: e.g. `Conformance of 'Never' to protocol 'Encodable' was already stated in the type's module 'Swift'`. + +The new conformance shouldn't differ from existing conformances, since it isn't possible to construct an instance of `Never`. + +## ABI compatibility + +The proposed change is additive and does not change any existing ABI. + +## Implications on adoption + +The new conformance will have availability annotations. + +## Future directions + +None. + +## Alternatives considered + +A previous iteration of this proposal used `DecodingError.dataCorrupted` as the error thrown from the `Decodable` initializer. Since that error is typically used for syntactical errors, such as malformed JSON, the `typeMismatch` error is now used instead. A custom error type could be created for this purpose, but as `typeMismatch` already exists as documented API, and provides the necessary information for a developer to understand the error, its use is appropriate here. diff --git a/proposals/0397-freestanding-declaration-macros.md b/proposals/0397-freestanding-declaration-macros.md new file mode 100644 index 0000000000..ab59886f02 --- /dev/null +++ b/proposals/0397-freestanding-declaration-macros.md @@ -0,0 +1,341 @@ +# Freestanding Declaration Macros + +* Proposal: [SE-0397](0397-freestanding-declaration-macros.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Richard Wei](https://github.com/rxwei), [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9)** +* Vision: [Macros](https://github.com/swiftlang/swift-evolution/blob/main/visions/macros.md) +* Implementation: On `main` behind the experimental flag `FreestandingMacros` +* Review: ([review](https://forums.swift.org/t/se-0397-freestanding-declaration-macros/64655)) ([partial acceptance and second review](https://forums.swift.org/t/se-0397-second-review-freestanding-declaration-macros/64997)) ([acceptance](https://forums.swift.org/t/accepted-se-0397-freestanding-declaration-macros/65167)) +* Previous revisions: ([1](https://github.com/swiftlang/swift-evolution/blob/c0f1e6729b6ca1a4fc2367efe68612fde175afe4/proposals/0397-freestanding-declaration-macros.md)) + +## Introduction + +[SE-0382 "Expression macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) introduced macros into Swift. The approach involves an explicit syntax for uses of macros (prefixed by `#`), type checking for macro arguments prior to macro expansion, and macro expansion implemented via separate programs that operate on the syntax tree of the arguments. + +This proposal generalizes the `#`-prefixed macro expansion syntax introduced for expression macros to also allow macros to generate declarations, enabling a number of other use cases, including: + +* Generating data structures from a template or other data format (e.g., JSON). +* Subsuming the `#warning` and `#error` directives introduced in [SE-0196](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md) as macros. + +## Proposed solution + +The proposal extends the notion of "freestanding" macros introduced in [SE-0382 "Expression macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) to also allow macros to introduce new declarations. Like expression macros, freestanding declaration macros are expanded using the `#` syntax, and have type-checked macro arguments. However, freestanding declaration macros can be used any place that a declaration is provided, and never produce a value. + +As with other macros, freestanding declaration macros are declared with the `macro` introducer. They will use the `@freestanding` attribute with the new `declaration` role and, optionally, a set of *introduced names* as described in [SE-0389 "Attached macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#specifying-newly-introduced-names). For example, a freestanding declaration macro would have an attribute like this: + +```swift +@freestanding(declaration) +``` + +whereas a freestanding declaration macro that introduced an enum named `CodingKeys` would have an attribute like this: + +```swift +@freestanding(declaration, names: named(CodingKeys)) +``` + +Implementations of freestanding declaration macros are types that conform to the `DeclarationMacro` protocol, which is defined as follows: + +```swift +public protocol DeclarationMacro: FreestandingMacro { + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] +} +``` + +Declaration macros can be used anywhere that a declaration is permitted, e.g., in a function or closure body, at the top level, or within a type definition or extension thereof. Declaration macros produce zero or more declarations. The `warning` directive introduced by [SE-0196](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md) can be described as a freestanding declaration macro as follows: + +```swift +/// Emits the given message as a warning, as in SE-0196. +@freestanding(declaration) +macro warning(_ message: String) = #externalMacro(module: "MyMacros", type: "WarningMacro") +``` + +Given this macro declaration, the syntax + +```swift +#warning("unsupported configuration") +``` + +can be used anywhere a declaration can occur. + +The implementation of a `warning` declaration macro extracts the string literal argument (producing an error if there wasn't one) and emits a warning. It returns an empty list of declarations: + +```swift +public struct WarningMacro: DeclarationMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] { + guard let messageExpr = node.argumentList.first?.expression.as(StringLiteralExprSyntax.self), + messageExpr.segments.count == 1, + let firstSegment = messageExpr.segments.first, + case let .stringSegment(message) = firstSegment else { + throw SimpleError(node, "warning macro requires a non-interpolated string literal") + } + + context.diagnose(Diagnostic(node: Syntax(node), message: SimpleDiagnosticMessage( + message: message.description, + diagnosticID: .init(domain: "test", id: "error"), + severity: .warning))) + return [] + } +} +``` + +## Detailed design + +### Syntax + +The syntactic representation of a freestanding macro expansion site is a macro expansion declaration. A macro expansion declaration is described by the following grammar. It is based on the production rule as a [macro expansion expression](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion), but with the addition of attributes and modifiers: + +``` +declaration -> macro-expansion-declaration +macro-expansion-declaration -> attributes? declaration-modifiers? '#' identifier generic-argument-clause[opt] function-call-argument-clause[opt] trailing-closures[opt] +``` + +At top level and function scope where both expressions and declarations are allowed, a freestanding macro expansion site is first parsed as a macro expansion expression. It will be replaced by a macro expansion declaration later during type checking, if the macro resolves to a declaration macro. It is ill-formed if a macro expansion expression resolves to a declaration macro but isn't the outermost expression. This parsing rule is required in case an expression starts with a macro expansion expression, such as in the following infix expression: + +```swift +#line + 1 +#line as Int? +``` + +#### Attributes and modifiers + +Any attributes and modifiers written on a freestanding macro declaration are implicitly applied to each declaration produced by the macro expansion. For example: + +```swift +@available(toasterOS 2.0, *) +public #gyb( + """ + struct Int${0} { ... } + struct UInt${0} { ... } + """, + [8, 16, 32, 64] +) +``` + +would expand to: + +```swift +@available(toasterOS 2.0, *) +public struct Int8 { ... } + +@available(toasterOS 2.0, *) +public struct UInt8 { ... } + +@available(toasterOS 2.0, *) +public struct Int16 { ... } + +@available(toasterOS 2.0, *) +public struct UInt16 { ... } + +@available(toasterOS 2.0, *) +public struct Int32 { ... } + +@available(toasterOS 2.0, *) +public struct UInt32 { ... } + +@available(toasterOS 2.0, *) +public struct Int64 { ... } + +@available(toasterOS 2.0, *) +public struct UInt64 { ... } +``` + +### Restrictions + +Like attached peer macros, a freestanding declaration macro can expand to any declaration that is syntactically and semantically well-formed within the context where the macro is expanded. It shares the same requirements and restrictions: + +- [**Specifying newly-introduced names**](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#specifying-newly-introduced-names) + - Note that only `named(...)` and `arbitrary` are allowed as macro-introduced names for a declaration macro. `overloaded`, `prefixed`, and `suffixed` do not make sense when there is no declaration from which to derive names. +- [**Visibility of names used and introduced by macros**](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#visibility-of-names-used-and-introduced-by-macros) +- [**Restrictions on `arbitrary` names**](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#restrictions-on-arbitrary-names) +- [**Permitted declaration kinds**](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#permitted-declaration-kinds) + +One additional restriction is that a macro declaration can have at most one freestanding macro role. This is because top level and function scopes allow a combination of expressions, statements, and declarations, which would be ambiguous to a freestanding macro expansion with multiple roles. + +```swift +@freestanding(expression) +@freestanding(declaration) // error: a macro cannot have multiple freestanding macro roles +macro foo() +``` + +### Examples + +#### SE-0196 `warning` and `error` + +The `#warning` and `#error` directives introduced in [SE-0196](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md): can be implemented directly as declaration macros: + +```swift +/// Emit a warning containing the given message. +@freestanding(declaration) macro warning(_ message: String) + +/// Emit an error containing the given message +@freestanding(declaration) macro error(_ message: String) +``` + +### Template code generation + +The Swift Standard Library makes extensive use of the [gyb](https://github.com/apple/swift/blob/main/utils/gyb.py) tool to generate boilerplate-y Swift code such as [tgmath.swift.gyb](https://github.com/apple/swift/blob/main/stdlib/public/Platform/tgmath.swift.gyb). The template code is written in `.gyb` files, which are processed by the gyb tool separately before Swift compilation. With freestanding declaration macros, one could write a macro to accept a string as a template and a list of replacement values, allowing templates to be defined inline and eliminating the need to set up a separate build phase. + +```swift +@freestanding(declaration, names: arbitrary) +macro gyb(String, [Any]) = #externalMacro(module: "MyMacros", type: "GYBMacro") + +#gyb( + """ + public struct Int${0} { ... } + public struct UInt${0} { ... } + """, + [8, 16, 32, 64] +) +``` + +This expands to: + +```swift + public struct Int8 { ... } + public struct UInt8 { ... } + + public struct Int16 { ... } + public struct UInt16 { ... } + + public struct Int32 { ... } + public struct UInt32 { ... } + + public struct Int64 { ... } + public struct UInt64 { ... } +``` + +### Data model generation + +Declaring a data model for an existing textual serialization may need some amount of eyeballing and is prone to errors. A freestanding declaration macro can be used to analyze a template textual serialization, e.g. JSON, and declare a model data structure against the template. + +```swift +@freestanding(declaration, names: arbitrary) +macro jsonModel(String) = #externalMacro(module: "MyMacros", type: "JSONModelMacro") + +struct JSONValue: Codable { + #jsonModel(""" + "name": "Produce", + "shelves": [ + { + "name": "Discount Produce", + "product": { + "name": "Banana", + "points": 200, + "description": "A banana that's perfectly ripe." + } + } + ] + """) +} +``` + +This expands to: + +```swift +struct JSONValue: Codable { + var name: String + var shelves: [Shelves] + + struct Shelves: Codable { + var name: String + var product: Product + + struct Product: Codable { + var description: String + var name: String + var points: Double + } + } +} +``` + +## Source compatibility + +Freestanding macros use the same syntax introduced for [expression macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md), which were themselves a pure extension without an impact on source compatibility. Because a given macro can only have a single freestanding role, and we retain the parsing rules for macro expansion expressions, this proposal introduces no new ambiguities with SE-0382. + +## Effect on ABI stability + +Macros are a source-to-source transformation tool that have no ABI impact. + +## Effect on API resilience + +Macros are a source-to-source transformation tool that have no effect on API resilience. + +## Alternatives considered + +### Multiple freestanding macro roles on a single macro + +The proposed feature bans declaring a macro as having multiple freestanding macro roles such as being both `@freestanding(expression)` and `@freestanding(declaration)`. But such a scenario could be allowed with proper rules. + +One possible solution would be to expand such a macro based on its expansion context. If it's being expanded where a declaration is allowed, it will always be expanded as a declaration. Otherwise, it's expanded as an expression. + +```swift +@freestanding(expression) +@freestanding(declaration) +macro dualRoleMacro() + +// File scope +#dualRoleMacro // expanded as a declaration + +func foo() { + #dualRoleMacro // expanded as a declaration + + _ = #dualRoleMacro // expanded as an expression + + bar(#dualRoleMacro) // expanded as an expression + + takesClosure { + #dualRoleMacro // expanded as a declaration + } +} +``` + +If a future use case deems this feature necessary, this restriction can be lifted following its own proposal. + +## Revision History + +- Scoped code item macros out as a future direction. + +## Future directions + +### Code item macros + +A code item macro is another kind of freestanding macro that can produce any mix of declarations, statements, and expressions, which are collectively called "code items" in the grammar. Code item macros can be used for top-level code and within the bodies of functions and closures. They are declaration with `@freestanding(codeItem)`. For example, we could declare a macro that logs when we are entering and exiting a function: + +```swift +@freestanding(codeItem) macro logEntryExit(arguments: Any...) +``` + +Code item macros are implemented as types conforming to the `CodeItemMacro` protocol: + +```swift +public protocol CodeItemMacro: FreestandingMacro { + /// Expand a macro described by the given freestanding macro expansion declaration + /// within the given context to produce a set of code items, which can be any mix of + /// expressions, statements, and declarations. + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> [CodeBlockItemSyntax] +} +``` + +The `logEntryExit` macro could introduce code such as: + +```swift +print("- Entering \(#function)(\(arguments))") +defer { + print("- Exiting \(#function)(\(arguments))") +} +``` + +Code item macros can only introduce new declarations that have unique names, created with `makeUniqueName(_:)`. They cannot introduce named declarations, because doing so affects the ability to type-check without repeatedly expanding the macro with potentially complete information. See the section on the visibility of names used and introduced by macros. + +Code item macros are currently under both `FreestandingMacros` and `CodeItemMacros` experimental feature flags. diff --git a/proposals/0398-variadic-types.md b/proposals/0398-variadic-types.md new file mode 100644 index 0000000000..a138841f25 --- /dev/null +++ b/proposals/0398-variadic-types.md @@ -0,0 +1,252 @@ +# Allow Generic Types to Abstract Over Packs + +* Proposal: [SE-0398](0398-variadic-types.md) +* Authors: [Slava Pestov](https://github.com/slavapestov), [Holly Borla](https://github.com/hborla) +* Review Manager: [Frederick Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 5.9)** +* Previous Proposal: [SE-0393](0393-parameter-packs.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-variadic-generic-types-abstracting-over-packs/64377)) ([review](https://forums.swift.org/t/se-0398-allow-generic-types-to-abstract-over-packs/64661)) ([acceptance](https://forums.swift.org/t/accepted-se-0398-allow-generic-types-to-abstract-over-packs/64998)) + +## Introduction + +Previously [SE-0393](0393-parameter-packs.md) introduced type parameter packs and several related concepts, allowing generic function declarations to abstract over a variable number of types. This proposal generalizes these ideas to generic type declarations. + +## Motivation + +Generic type declarations that abstract over a variable number of types arise naturally when attempting to generalize common algorithms on collections. For example, the current `zip` function returns a `Zip2Sequence`, but it's not possible from SE-0393 alone to define an equivalent variadic `zip` function because the return type would need an arbitrary number of type parameters—one for each input sequence: + +```swift +func zip(_ seq: repeat each S) -> ??? + where repeat each S: Sequence +``` + +## Proposed solution + +In the generic parameter list of a generic type, the `each` keyword declares a generic parameter pack, just like it does in the generic parameter list of a generic function. The types of stored properties can contain pack expansion types, as in `let seq` and `var iter` below. + +This lets us define the return type of the variadic `zip` function as follows: + +```swift +struct ZipSequence: Sequence { + typealias Element = (repeat (each S).Element) + + let seq: (repeat each S) + + func makeIterator() -> Iterator { + return Iterator(iter: (repeat (each seq).makeIterator())) + } + + struct Iterator: IteratorProtocol { + typealias Element = (repeat (each S).Element) + + var iter: (repeat (each S).Iterator) + + mutating func next() -> Element? { + return ... + } + } +} + +func zip(_ seq: repeat each S) -> ZipSequence + where repeat each S: Sequence +``` + +## Detailed design + +Swift has the following kinds of generic type declarations: + +- Structs +- Enums +- Classes (and actors) +- Type aliases + +A generic type is _variadic_ if it directly declares a type parameter pack with `each`, or if it is nested inside of another variadic type. In this proposal, structs, classes, actors and type aliases can be variadic. Enums will be addressed in a follow-up proposal. + +### Single parameter + +A generic type is limited to declaring at most one type parameter pack. The following are allowed: + +```swift +struct S1 {} +struct S2 {} +struct S3 {} +``` + +But this is not: + +```swift +struct S4 {} +``` + +However, by virtue of nesting, a variadic type can still abstract over multiple type parameter packs: + +```swift +struct Outer { + struct Inner { + var fn: (repeat each T) -> (repeat each U) + } +} +``` + +### Referencing a variadic type + +When used with a variadic type, the generic argument syntax `S<...>` allows a variable number of arguments to be specified. Since there can only be one generic parameter pack, the non-pack parameters are always specified with a fixed prefix and suffix of the generic argument list. + +```swift +struct S {} + +S.self // T := Int, U := Pack{}, V := Float +S.self // T := Int, U := Pack{Bool}, V := Float +S.self // T := Int, U := Pack{Bool, String}, V := Float +``` + +Note that `S` substitutes U with the empty pack type, which is allowed. The minimum number of generic arguments is equal to the number of non-pack generic parameters. In our above example, the minimum argument count is 2, because `T` and `V` must always be specified: + +```swift +S.self // error: expected at least 2 generic arguments +``` + +If the generic parameter list of a variadic type consists of a single generic parameter pack and nothing else, it is possible to reference it with an empty generic argument list: + +```swift +struct V {} + +V< >.self +``` +Note that `V< >` is not the same as `V`. The former substitutes the generic parameter pack `T` with the empty pack. The latter does not constrain the pack at all and is only permitted in contexts where the generic argument can be inferred (or within the body of `V` or an extension thereof, where it is considered identical to `Self`). + +A placeholder type in the generic argument list of a variadic generic type is always understood as a single pack element. For example: + +```swift +struct V {} + +let x: V<_> = V() // okay +let x: V<_, _> = V() // okay +let x: V<_> = V() // error +``` + +### Stored properties + +In a variadic type, the type of a stored property can contain pack expansion types. The type of a stored property cannot _itself_ be a pack expansion type. Stored properties are limited to having pack expansions nested inside tuple types, function types and other named variadic types: + +```swift +struct S { + var a: (repeat each Array) + var b: (repeat each T) -> (Int) + var c: Other +} +``` +This is in contrast with the parameters of generic function declarations, which can have a pack expansion type. A [future proposal](#future-directions) might lift this restriction and introduce true "stored property packs." + +### Requirements + +The behavior of generic requirements on type parameter packs is mostly unchanged between generic functions and generic types. However, allowing types to abstract over parameter packs introduces _requirement inference_ of [generic requirement expansions](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0393-parameter-packs.md#generic-requirements). Requirement expansion inference follows these rules: + +1. If a generic type that imposes an inferred scalar requirement is applied to a pack element inside a pack expansion, the inferred requirement is a requirement expansion. +2. If a generic type imposes an inferred requirement expansion, the requirement is expanded for each of the concrete generic arguments. +3. If a generic type that imposes an inferred requirement expansion is applied to a pack element inside a pack expansion: + + 1. The inferred requirement is invalid if it contains multiple pack elements captured by expansions at different depths. + 2. Otherwise, the nested requirement expansion is semantically equivalent to the innermost requirement expansion. + + The below code demonstrates each of the above rules: + + ```swift + protocol P { + associatedtype A + } + struct ImposeRequirement where T: P {} + struct ImposeRepeatedRequirement where repeat each T: P {} + struct ImposeRepeatedSameType where repeat T1.A == each T2 {} + + // Infers 'repeat each U: P' + func demonstrate1(_: repeat ImposeRequirement) + + // Infers 'Int: P, V: P, repeat each U: P' + func demonstrate2(_: ImposeRepeatedRequirement) + + // Error. Would attempt to infer 'repeat repeat U'.A == V' which is not a supported requirement in the language + func demonstrate3a(_: repeat ImposeRepeatedSameType)) + + // Infers 'Int: P, repeat each V: P' + func demonstrate3b(_: repeat (each U, ImposeRepeatedRequirement)) +``` + +### Conformances + +Variadic structs, classes and actors can conform to protocols. The associated type requirements of the protocol may be fulfilled by a type alias whose underlying type contains pack expansions. + +### Type aliases + +As with the other variadic types, a variadic type alias either has a generic parameter pack of its own, or can be nested inside of another variadic generic type. + +The underlying type of a variadic type alias can reference pack expansion types in the same manner as the type of a stored property. That is, the pack expansions must appear in nested positions, but not at the top level. + +```swift +typealias Element = (repeat (each S).Element) +typealias Callback = (repeat each S) -> () +typealias Factory = Other +``` + +Like other type aliases, variadic type aliases can be nested inside of generic functions (and like other structs and classes, variadic structs and classes cannot). + +### Classes + +While there are no restrictions on non-final classes adopting type parameter packs, for the time being the proposal restricts such classes from being the superclass of another class. + +An attempt to inherit from a variadic generic class outputs an error. The correct behavior of override checking and initializer inheritance in variadic generic classes will be dealt with in a follow-up proposal: + +```swift +class Base { + func foo(t: repeat each T) {} +} + +// error: cannot inherit from a class with a type parameter pack +class Derived: Base { + override func foo(t: U, _: V) {} +} +``` + +## Source compatibility + +Variadic generic types are a new language feature which does not impact source compatibility with existing code. + +## ABI compatibility + +Variadic type aliases are not part of the binary interface of a module and do not require runtime support. + +All other variadic types make use of new entry points and other behaviors being added to the Swift runtime. Since the runtime support requires extensive changes to the type metadata logic, backward deployment to older runtimes is not supported. + +Replacing a non-variadic generic type with a variadic generic type is **not** binary-compatible in either direction. When adopting variadic generic types, binary-stable frameworks must introduce them as wholly-new symbols. + +## Future directions + +A future proposal will address variadic generic enums, and complete support for variadic generic classes. + +Another possible future direction is stored property packs, which would eliminate the need to wrap a pack expansion in a tuple type in order to store a variable number of values inside of a variadic type: + +```swift +struct S { + var a: repeat each Array +} +``` + +However, there is no expressivity lost in requiring the tuple today, since the contents of a tuple can be converted into a value pack. + +## Alternatives considered + +The one-parameter limitation could be lifted if we introduced labeled generic parameters at the same time. The choice to allow only a single (unlabeled) generic parameter pack does not preclude this possibility from being explored in the future. + +Another alternative is to not enforce the one-parameter limitation at all. There would then exist variadic generic types which cannot be spelled explicitly, but can still be constructed by type inference: + +```swift +struct S { + init(t: repeat each T, u: repeat each U) {} +} + +S(t: 1, "hi", u: false) +``` + +It was felt that in the end, the single-parameter model is the simplest. + +We could require that variadic classes are declared `final`, instead of rejecting subclassing at the point of use. However, since adding or removing `final` on a class is an ABI break, this would preclude the possibility of publishing APIs which work with the existing compiler but can allow subclassing in the future. diff --git a/proposals/0399-tuple-of-value-pack-expansion.md b/proposals/0399-tuple-of-value-pack-expansion.md new file mode 100644 index 0000000000..8918268825 --- /dev/null +++ b/proposals/0399-tuple-of-value-pack-expansion.md @@ -0,0 +1,127 @@ +# Tuple of value pack expansion + +* Proposal: [SE-0399](0399-tuple-of-value-pack-expansion.md) +* Authors: [Sophia Poirier](https://github.com/sophiapoirier), [Holly Borla](https://github.com/hborla) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.9)** +* Implementation: On `main` gated behind `-enable-experimental-feature VariadicGenerics` +* Previous Proposals: [SE-0393](0393-parameter-packs.md), [SE-0398](0398-variadic-types.md) +* Review: ([pitch](https://forums.swift.org/t/tuple-of-value-pack-expansion/64269)) ([review](https://forums.swift.org/t/se-0399-tuple-of-value-pack-expansion/65017)) ([acceptance](https://forums.swift.org/t/accepted-se-0399-tuple-of-value-pack-expansion/65271)) + +## Introduction + +Building upon the **Value and Type Parameter Packs** proposal [SE-0393](https://forums.swift.org/t/se-0393-value-and-type-parameter-packs/63859), this proposal enables referencing a tuple value that contains a value pack inside a pack repetition pattern. + +## Motivation + +When a tuple value contains a value pack, there is no way to reference those pack elements or pass them as a value pack function argument. + +Additionally, type parameter packs are only permitted within a function parameter list, tuple element, or generic argument list. This precludes declaring a type parameter pack as a function return type, type alias, or as a local variable type to permit storage. The available solution to these restrictions is to contain the value pack in a tuple, which makes it important to provide full functional parity between value packs and tuple values containing them. This proposal fills that functionality gap by providing a method to reference individual value pack elements contained within a tuple. + +## Proposed solution + +This proposal extends the functionality of pack repetition patterns to values of _abstract tuple type_, which enables an implicit conversion of an _abstract tuple value_ to its contained value pack. An _abstract tuple type_ is a tuple that has an unknown length and elements of unknown types. Its elements are that of a single type parameter pack and no additional elements, and no label. In other words, the elements of the type parameter pack are the elements of the tuple. An _abstract tuple value_ is a value of _abstract tuple type_. This proposal provides methods to individually access the dynamic pack elements of an abstract tuple value inside of a repetition pattern. + +The following example demonstrates how, with this proposal, we can individually reference and make use of the elements in an abstract tuple value that was returned from another function. The example also highlights some constructs that are not permitted under this proposal: + +```swift +func tuplify(_ value: repeat each T) -> (repeat each T) { + return (repeat each value) +} + +func example(_ value: repeat each T) { + let abstractTuple = tuplify(repeat each value) + repeat print(each abstractTuple) // okay as of this proposal + + let concreteTuple = (true, "two", 3) + repeat print(each concreteTuple) // invalid + + let mixedConcreteAndAbstractTuple = (1, repeat each value) + repeat print(each mixedConcreteAndAbstractTuple) // invalid + + let labeledAbstractTuple = (label: repeat each value) + repeat print(each labeledAbstractTuple) // invalid +} +``` + +### Distinction between tuple values and value packs + +The following example demonstrates a pack repetition pattern on a value pack and an abstract tuple value separately first, then together but with the repetition pattern operating only on the value pack, and finally with the repetition pattern operating on both the value pack and the tuple value's contained value pack together interleaved. Note that, because the standard library function `print` does not currently accept parameter packs but instead only a single value parameter, all of the calls to it wrap the value argument pack in a tuple (hence all of those parentheses). + +```swift +func example(packElements value: repeat each T, tuple: (repeat each T)) { + print((repeat each value)) + print((repeat each tuple)) + + print((repeat (each value, tuple))) + + print((repeat (each value, each tuple))) +} + +example(packElements: 1, 2, 3, tuple: (4, 5, 6)) + +// Prints the following output: +// (1, 2, 3) +// (4, 5, 6) +// ((1, (4, 5, 6)), (2, (4, 5, 6)), (3, (4, 5, 6))) +// ((1, 4), (2, 5), (3, 6)) +``` + +## Detailed design + +Pack reference expressions inside a repetition pattern can have abstract tuple type. The outer structure of the tuple is removed, leaving the elements of a value pack: + +```swift +func expand(value: (repeat each T)) -> (repeat (each T)?) { + return (repeat each value) +} +``` + +Applying the pack repetition pattern effectively removes the outer structure of the tuple leaving just the value pack. In the repetition expression, the base tuple is evaluated once before iterating over its elements. + +```swift +repeat each + +// the above is evaluated like this +let tempTuple = +repeat each tempTuple +``` + +## Source compatibility + +There is no source compatibility impact given that this is an additive change. It enables compiling code that previously would not compile. + +## ABI compatibility + +This proposal does not add or affect ABI as its impact is only on expressions. It does not change external declarations or types. It rests atop the ABI introduced in the **Value and Type Parameter Packs** proposal. + +## Implications on adoption + +Given that this change rests atop the ABI introduced in the **Value and Type Parameter Packs** proposal, this shares with it the same runtime back-deployment story. + +## Alternatives considered + +An earlier design required the use of an abstract tuple value expansion operator, in the form of `.element` (effectively a synthesized label for the value pack contained within the abstract tuple value). This proposal already requires a tuple with a single element that is a value pack, so it is unnecessary to explicitly call out that the expansion is occurring on that element. Requiring `.element` could also introduce potential source breakage in the case of existing code that contains a tuple using the label "element". Dropping the `.element` requirement averts the language inconsistency of designating a reserved tuple label that functions differently than any other tuple label. + +## Future directions + +### Repetition patterns for concrete tuples + +It could help unify language features to extend the repetition pattern syntax to tuples of concrete type. + +```swift +func example(_ value: repeat each T) { + let abstractTuple = (repeat each value) + let concreteTuple = (true, "two", 3) + repeat print(each abstractTuple) + repeat print(each concreteTuple) // currently invalid +} +``` + +### Pack repetition patterns for arrays + +If all elements in a type argument pack are the same or share a conformance, then it should be possible to declare an Array value using a value pack. + +### Lift the single value pack with no label restriction + +This would be required to enable pack repetition patterns for a contained value pack amongst arbitrary other tuple elements that could be addressable via their labels. diff --git a/proposals/0400-init-accessors.md b/proposals/0400-init-accessors.md new file mode 100644 index 0000000000..852421b0fd --- /dev/null +++ b/proposals/0400-init-accessors.md @@ -0,0 +1,657 @@ +# Init Accessors + +* Proposal: [SE-0400](0400-init-accessors.md) +* Authors: [Holly Borla](https://github.com/hborla), [Doug Gregor](https://github.com/douggregor) +* Review Manager: [Frederick Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 5.9)** +* Implementation: On `main` behind experimental feature flag `InitAccessors` +* Review: ([pitch](https://forums.swift.org/t/pitch-init-accessors/64881)) ([review](https://forums.swift.org/t/se-0400-init-accessors/65583)) ([acceptance](https://forums.swift.org/t/accepted-se-0400-init-accessors/66212)) + +## Introduction + +Init accessors generalize the out-of-line initialization feature of property wrappers to allow any computed property on types to opt into definite initialization analysis, and subsume initialization of a set of stored properties with custom initialization code. + +## Motivation + +Swift applies [definite initialization analysis](https://en.wikipedia.org/wiki/Definite_assignment_analysis) to stored properties, stored local variables, and variables with property wrappers. Definite initialization ensures that memory is initialized on all paths before it is accessed. A common pattern in Swift code is to use one property as backing storage for one or more computed properties, and abstractions like [property wrappers](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md) and [attached macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md) help facilitate this pattern. Under this pattern, the backing storage is an implementation detail, and most code works with the computed property, including initializers. + +Property wrappers support bespoke definite initialization that allows initializing the backing property wrapper storage via the computed property, always re-writing initialization-via-wrapped-property in the form `self.value = value` to initialization of the backing storage in the form of `_value = Wrapper(wrappedValue: value)`: + +```swift +@propertyWrapper +struct Wrapper { + var wrappedValue: T +} + +struct S { + @Wrapper var value: Int + + init(value: Int) { + self.value = value // Re-written to self._value = Wrapper(wrappedValue: value) + } + + init(other: Int) { + self._value = Wrapper(wrappedValue: other) // Okay, initializes storage '_value' directly + } +} +``` + +The ad-hoc nature of property wrapper initializers mixed with an exact definite initialization pattern prevent property wrappers with additional arguments from being initialized out-of-line. Furthermore, property-wrapper-like macros cannot achieve the same initializer usability, because any backing storage variables added must be initialized directly instead of supporting initialization through computed properties. For example, the [`@Observable` macro](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0395-observability.md) applies a property-wrapper-like transform that turns stored properties into computed properties backed by the observation APIs, but it provides no way to write an initializer using the original property names like the programmer expects: + +```swift +@Observable +struct Proposal { + var title: String + var text: String + + init(title: String, text: String) { + self.title = title // error: 'self' used before all stored properties are initialized + self.text = text // error: 'self' used before all stored properties are initialized + } // error: Return from initializer without initializing all stored properties +} +``` + +## Proposed solution + +This proposal adds _`init` accessors_ to opt computed properties on types into definite initialization that subsumes initialization of a set of zero or more specified stored properties, which allows assigning to computed properties in the body of a type's initializer: + +```swift +struct Angle { + var degrees: Double + var radians: Double { + @storageRestrictions(initializes: degrees) + init(initialValue) { + degrees = initialValue * 180 / .pi + } + + get { degrees * .pi / 180 } + set { degrees = newValue * 180 / .pi } + } + + init(degrees: Double) { + self.degrees = degrees // initializes 'self.degrees' directly + } + + init(radiansParam: Double) { + self.radians = radiansParam // calls init accessor for 'self.radians', passing 'radiansParam' as the argument + } +} +``` + +The signature of an `init` accessor specifies up to two sets of stored properties: the properties that are accessed (via `accesses`) and the properties that are initialized (via `initializes`) by the accessor. `initializes` and `accesses` are side-effects of the `init` accessor. Access effects specify the other stored properties that can be accessed from within the `init` accessor (no other uses of `self` are allowed), and therefore must be initialized before the computed property's `init` accessor is invoked. The `init` accessor must initialize each of the initialized stored properties on all control flow paths. The `radians` property in the example above specifies no access effect, but initializes the `degrees` property, so it specifies only `initializes: degrees`. + +Access effects allow a computed property to be initialized by placing its contents into another stored property: + +```swift +struct ProposalViaDictionary { + private var dictionary: [String: String] + + var title: String { + @storageRestrictions(accesses: dictionary) + init(newValue) { + dictionary["title"] = newValue + } + + get { dictionary["title"]! } + set { dictionary["title"] = newValue } + } + + var text: String { + @storageRestrictions(accesses: dictionary) + init(newValue) { + dictionary["text"] = newValue + } + + get { dictionary["text"]! } + set { dictionary["text"] = newValue } + } + + init(title: String, text: String) { + self.dictionary = [:] // 'dictionary' must be initialized before init accessors access it + self.title = title // calls init accessor to insert title into the dictionary + self.text = text // calls init accessor to insert text into the dictionary + + // it is an error to omit either initialization above + } +} +``` + +Both `init` accessors document that they access `dictionary`, which allows them to insert the new values into the dictionary with the appropriate key as part of initialization. This allows one to fully abstract away the storage mechanism used in the type. + +Finally, computed properties with `init` accessors are privileged in the synthesized member-wise initializer. With this proposal, property wrappers have no bespoke definite and member-wise initialization support. Instead, the desugaring for property wrappers with an `init(wrappedValue:)` includes an `init` accessor for wrapped properties and a member-wise initializer including wrapped values instead of the respective backing storage. The property wrapper code in the Motivation section will desugar to the following code: + +```swift +@propertyWrapper +struct Wrapper { + var wrappedValue: T +} + +struct S { + private var _value: Wrapper + var value: Int { + @storageRestrictions(initializes: _value) + init(newValue) { + self._value = Wrapper(wrappedValue: newValue) + } + + get { _value.wrappedValue } + set { _value.wrappedValue = newValue } + } + + // This initializer is the same as the generated member-wise initializer. + init(value: Int) { + self.value = value // Calls 'init' accessor on 'self.value' + } +} + +S(value: 10) +``` + +This proposal allows macros to model the following property-wrapper-like patterns including out-of-line initialization of the computed property: +* A wrapped property with attribute arguments +* A wrapped property that is backed by an explicit stored property +* A set of wrapped properties that are backed by a single stored property + +## Detailed design + +### Syntax + +The proposal adds a new kind of accessor, an `init` accessor, which can be written in the accessor list of a computed property. Init accessors add the following production rules to the grammar: + +``` +init-accessor -> 'init' init-accessor-parameter[opt] function-body + +init-accessor-parameter -> '(' identifier ')' + +accessor-block -> init-accessor +``` + +The `identifier` in an `init-accessor-parameter`, if provided, is the name of the parameter that contains the initial value. If not provided, a parameter with the name `newValue` is automatically created. The minimal init accessor has no parameter list and no initialization effects: + +```swift +struct Minimal { + var value: Int { + init { + print("init accessor called with \(newValue)") + } + + get { 0 } + } +} +``` + +This proposal also adds a new `storageRestrictions` attribute to describe the storage restrictions for `init` accessor blocks. The attribute can only be used on `init` accessors. The attribute is described by the following production rules in the grammar: + +``` +attribute ::= storage-restrictions-attribute + +storage-restrictions-attribute ::= '@' storageRestrictions '(' storage-restrictions[opt] ')' + +storage-restrictions-initializes ::= 'initializes' ':' identifier-list +storage-restrictions-accesses ::= 'accesses' ':' identifier-list + +storage-restrictions ::= storage-restrictions-accesses +storage-restrictions ::= storage-restrictions-initializes +storage-restrictions ::= storage-restrictions-initializes ',' storage-restrictions-accesses +``` + +The storage restriction attribute can include a list of stored properties that are initialized by this accessor (the identifier list in `storage-restrictions-initializes`), and a list of stored properties that are accessed by this accessor (the identifier list in `storage-restrictions-accesses`), each of which are optional: + +```swift +struct S { + var readMe: String + + var _x: Int + + var x: Int { + @storageRestrictions(initializes: _x, accesses: readMe) + init(newValue) { + print(readMe) + _x = newValue + } + + get { _x } + set { _x = newValue } + } +} +``` + +If the accessor uses the default parameter name `newValue` and neither initializes nor accesses any stored property, the signature is not required. + +Init accessors can subsume the initialization of a set of stored properties. Subsumed stored properties are specified through the `initializes` argument to the attribute. The body of an `init` accessor is required to initialize the subsumed stored properties on all control flow paths. + +Init accessors can also require a set of stored properties to already be initialized when the body is evaluated, which are specified through the `accesses` argument to the attribute. These stored properties can be accessed in the accessor body; no other properties or methods on `self` are available inside the accessor body, nor is `self` available as a whole object (i.e., to call methods on it). + +### Definite initialization of properties on `self` + +The semantics of an assignment inside of a type's initializer depend on whether or not all of `self` is initialized on all paths at the point of assignment. Before all of `self` is initialized, assignment to a computed property with an `init` accessor is re-written to an `init` accessor call; after `self` has been initialized, assignment to a computed property is re-written to a setter call. + +With this proposal, all of `self` is initialized if: +* All stored properties are initialized on all paths, and +* All computed properties with `init` accessors are initialized on all paths. + +An assignment to a computed property with an `init` accessor before all of `self` is initialized will call the computed property's `init` accessor and initialize all of the stored properties specified in its `initializes` clause: + +```swift +struct S { + var x1: Int + var x2: Int + + var computed: Int { + @storageRestrictions(initializes: x1, x2) + init(newValue) { ... } + } + + init() { + self.computed = 1 // initializes 'computed', 'x1', and 'x2'; 'self' is now fully initialized + } +} +``` + +An assignment to a computed property that has not been initialized on all paths will be re-written to an `init` accessor call: + +```swift +struct S { + var x: Int + var y: Int + + var point: (Int, Int) { + @storageRestrictions(initializes: x, y) + init(newValue) { + (self.x, self.y) = newValue + } + get { (x, y) } + set { (x, y) = newValue } + } + + init(x: Int, y: Int) { + if (x == y) { + self.point = (x, x) // calls 'init' accessor + } + + // 'self.point' is not initialized on all paths here + + self.point = (x, y) // calls 'init' accessor + + // 'self.point' is initialized on all paths here + } +} +``` + +An assignment to a stored property before all of `self` is initialized will initialize that stored property. When all of the stored properties listed in the `initializes` clause of a computed property with an `init` accessor have been initialized, that computed property is considered initialized: + +```swift +struct S { + var x1: Int + var x2: Int + var x3: Int + + var computed: Int { + @storageRestrictions(initializes: x1, x2) + init(newValue) { ... } + } + + init() { + self.x1 = 1 // initializes 'x1'; neither 'x2' or 'computed' is initialized + self.x2 = 1 // initializes 'x2' and 'computed' + self.x3 = 1 // initializes 'x3'; 'self' is now fully initialized + } +} +``` + +An assignment to a computed property where at least one of the stored properties listed in `initializes` is initialized, but `self` is not initialized, is an error. This prevents double-initialization of the underlying stored properties: + +```swift +struct S { + var x: Int + var y: Int + + var point: (Int, Int) { + @storageRestrictions(initializes: x, y) + init(newValue) { + (self.x, self.y) = newValue + } + get { (x, y) } + set { (x, y) = newValue } + } + + init(x: Int, y: Int) { + self.x = x // Only initializes 'x' + self.point = (x, y) // error: neither the `init` accessor nor the setter can be called here + } +} +``` + +### Memberwise initializers + +If a struct does not declare its own initializers, it receives an implicit memberwise initializer based on the stored properties of the struct, because the storage is what needs to be initialized. Because many use-cases for `init` accessors are fully abstracting a single computed property to be backed by a single stored property, such as in the property-wrapper use case, an `init` accessor provides a preferred mechanism for initializing storage because the programmer will primarily interact with that storage through the computed property. As such, the memberwise initializer parameter list will include computed properties that have init accessors along with only those stored properties that have not been subsumed by an init accessor. + +```swift +struct S { + var _x: Int + + var x: Int { + @storageRestrictions(initializes: _x) + init(newValue) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} + +S(x: 10, y: 100) +``` + +The above struct `S` receives a synthesized initializer: + +```swift +init(x: Int, y: Int) { + self.x = x + self.y = y +} +``` + +The parameters of the memberwise initializer follow source order. However, if an init accessor `accesses` a stored property that precedes it in the memberwise initializer, then the properties cannot be initialized in the same order as the parameters occur in the memberwise initializer. For example: + +```swift +struct S { + var _x: Int + + var x: Int { + @storageRestrictions(initializes: _x, accesses: y) + init(newValue) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} +``` + +If the memberwise initializer of the above struct were written to initialize the properties in the same order as the parameters, it would produce an error: + +```swift +init(x: Int, y: Int) { + self.x = x // error + self.y = y +} +``` + +Therefore, the compiler will order the initializations in the synthesized memberwise initializer to respect the `accesses` clauses: + +```swift +init(x: Int, y: Int) { + self.y = y + self.x = x +} +``` + +The initial review of this proposal suppressed the memberwise initializer in such cases, based on a concern that out-of-order initialization would cause surprises. However, given the fact that the fields are initialized independently (or have `accessses` relationships that define their relative ordering), and that side effects here are limited to those of the `init` accessors themselves, one has to introduce global side effects during initialization to observe any difference. + +There remain cases where a memberwise initializer cannot be synthesized. For example, if a type contains several computed properties with `init` accessors that initialize the same stored property, it is not clear which computed property should be used within the member-wise initializer. In such cases, a member-wise initializer will not be synthesized. + +### Init accessors on computed properties + +An init accessor can be provided on a computed property, in which case it is used for initialization and as a default argument in the memberwise initializer. For example, given the following: + +```swift +struct Angle { + var degrees: Double + + var radians: Double { + @storageRestrictions(initializes: degrees) + init(initialValue) { + degrees = initialValue * 180 / .pi + } + + get { degrees * .pi / 180 } + set { degrees = newValue * 180 / .pi } + } +} +``` + +The implicit memberwise initializer will contain `radians`, but not the `degrees` stored property that it subsumes: + +```swift +init(radians: Double) { + self.radians = radians // calls init accessor, subsumes initialization of 'degrees' +} +``` + +### Init accessors for read-only properties + +Init accessors can be provided for properties that lack a setter. Such properties act much like a `let` property, able to be initialized (exactly) once and not set thereafter: + +```swift +struct S { + var _x: Int + + var x: Int { + @storageRestrictions(initializes: _x) + init(initialValue) { + self._x = x + } + + get { _x } + } + + init(halfOf y: Int) { + self.x = y / 2 // okay, calls init accessor for x + self.x = y / 2 // error, 'x' cannot be set + } +} + +``` + +### Initial values on properties with an init accessor + +A property with an init accessor can have an initial value, e.g., + +```swift +struct WithInitialValues { + var _x: Int + + var x: Int = 0 { + @storageRestrictions(initializes: _x) + init(initialValue) { + _x = initialValue + } + + get { ... } + set { ... } + } + + var y: Int +} +``` + +The synthesized memberwise initializer will use the initial value as a default argument, so it will look like the following: + +```swift +init(x: Int = 0, y: Int) { + self.x = x // calls init accessor, which initializes _x + self.y = y +} +``` + +In a manually written initializer, the initial value will be used to initialize the property with the init accessor prior to any user-written code: + +```swift +init() { + // implicitly initializes self.x = 0 + self.y = 10 + self.x = 20 // calls setter +} +``` + +### Restrictions + +A property with an `init` accessor can only be declared in the primary +declaration of a type. + +## Source compatibility + +`init` accessors are an additive capability with new syntax; there is no impact on existing source code. + +## ABI compatibility + +`init` accessors are an ABI-additive change; they are at most `internal` but can +be ABI-public. +Calling an `init` accessor from an `inlinable` type initializer requires that +the `init` accessor is ABI-public. + +## Implications on adoption + +Because `init` accessors are always called from within the defining module, adopting `init` accessors is an ABI-compatible change. Adding an `init` accessor to an existing property also cannot have any source compatibility impact outside of the defining module; the only possible source incompatibilities are on the generated memberwise initializer (if new entries are added), or on the type's `init` implementation (if new initialization effects are added). + +## Alternatives considered + +### Syntax for "initializes" and "accesses" + +A number of different syntaxes have been considered for specifying the set of stored properties that are initialized or accessed by a property that has an `init` accessor. The original pitch specified them in the parameter list using special labels: + +```swift +struct S { + var _x: Int + var x: Int { + init(newValue, initializes: _x, accesses: y) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} +``` + +This syntax choice is misleading because the effects look like function parameters, while `initializes` behaves more like the output of an init accessor, and `accesses` are not explicitly provided at the call-site. Conceptually, `initializes` and `accesses` are side effects of an `init` accessor, so the proposal was revised to place these modifiers in the effects clause. + +The first reviewed version of this proposal placed `initializes` and `accesses` along with other *effects*, e.g., + +```swift +struct S { + var _x: Int + var x: Int { + init(newValue) initializes(_x), accesses(y) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} +``` + +However, `initializes` and `effects` don't behave in the same manner as other effects in Swift, such as `throws` and `async`, for several reasons. First, there's no annotation like `try` or `await` at the call site. Second, these aren't part of the type of the entity (e.g., there is not function type that has an `initializes` clause). Therefore, using the effects clause is not a good match for Swift's semantic model. + +The current proposal uses an attribute. With attributes, there is question of whether we can remove the `@` to turn it into a declaration modifier: + +```swift +struct S { + var _x: Int + var x: Int { + storageRestrictions(initializes: _x, accesses: y) + init(newValue) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} +``` + +This is doable within the confines of this proposal's init accessors, but would prevent further extensions of this proposal that would allow the use of `initializes` or `accesses` on arbitrary functions. For example, such an extension might allow the following + +```swift +var _x, _y: Double + +storageRestrictions(initializes: _x, _y) +func initCoordinates(radius: Double, angle: Double) { ... } + +if let (r, theta) = decodeAsPolar() { + initCoordinates(radius: r, angle: theta) +} else { + // ... +} +``` + +However, there is a parsing ambiguity in the above because `storageRestrictions(initializes: _x, _y)` could be a call to a function names `storageRestrictions(initializes:)` or it could be a declaration modifier specifying that `initCoordinates` initializes `_x` and `_y`. + +Other syntax suggestions from pitch reviewers included: + +* Using a capture-list-style clause, e.g. `init { [&x, y] in ... }` +* Using more concise effect names, e.g. `writes` and `reads` instead of `initializes` and `accesses` +* And more! + +However, the current syntax in this proposal, which uses an attribute, most accurately models the semantics of initialization effects. An `init` accessor is a function -- not a closure -- that has side-effects related to initialization. _Only_ the `init` accessor has these effects; though the `set` accessor often contains code that looks the same as the code in the `init` accessor, the effects of these accessors are different. Because `init` accessors are called before all of `self` is initialized, they do not receive a fully-initialized `self` as a parameter like `set` accessors do, and assignments to `initializes` stored properties in `init` accessors have the same semantics as that of a standard initializer, such as suppressing `willSet` and `didSet` observers. + +## Future directions + +### `init` accessors for local variables + +`init` accessors for local variables have different implications on definite initialization, because re-writing assignment to `init` or `set` is not based on the initialization state of `self`. Local variable getters and setters can also capture any other local variables in scope, which raises more challenges for diagnosing escaping uses before initialization during the same pass where assignments may be re-written to `init` or `set`. As such, local variables with `init` accessors are a future direction. + +### Generalization of storage restrictions to other functions + +In the future, the `storageRestrictions` attribute could be be generalized to apply to other functions. For example, this could allow one to implement a common initialization function within a class: + +```swift +class C { + var id: String + var state: State + + @storageRestrictions(initializes: state, accesses: id) + func initState() { + self.state = /* initialization code here */ + } + + init(id: String) { + self.id = id + initState() // okay, accesses id and initializes state + } +} +``` + +The principles are the same as with `init` accessors: a function's implementation can be restricted to only access certain stored properties, and to initialize others along all paths. A call to the function then participates in definite initialization. + +This generalization comes with limitations that were not relevant to `init` accessors, because the functions are more akin to fragments of an initializer. For example, the `initState` function cannot be called after `state` is initialized (because it would re-initialize `state`), nor can it be used as a "first-class" function: + +```swift + init(id: String) { + self.id = id + initState() // okay, accesses id and initializes state + + initState() // error, 'state' is already initialized + let fn = self.initState // error: can't treat it like a function value + } +``` + +These limitations are severe enough that this future direction would require a significant amount of justification on its own to pursue, and therefore is not part of the `init` accessors proposal. + +## Revision history + +* Following the initial review: + * Replaced the "effects" syntax with the `@storageRestrictions` attribute. + * Add section on init accessors for computed properties. + * Add section on init accessors for read-only properties. + * Allow reordering of the initializations in the synthesized memberwise initializer to respect `accesses` restrictions. + * Add a potential future direction for the generalization of storage restrictions to other functions. + * Clarify the behavior of properties that have init accessors and initial values. + +## Acknowledgments + +Thank you to TJ Usiyan, Michel Fortin, and others for suggesting alternative syntax ideas for `init` accessor effects; thank you to Pavel Yaskevich for helping with the implementation. diff --git a/proposals/0401-remove-property-wrapper-isolation.md b/proposals/0401-remove-property-wrapper-isolation.md new file mode 100644 index 0000000000..48ba56235d --- /dev/null +++ b/proposals/0401-remove-property-wrapper-isolation.md @@ -0,0 +1,212 @@ +# Remove Actor Isolation Inference caused by Property Wrappers + +* Proposal: [SE-0401](0401-remove-property-wrapper-isolation.md) +* Authors: [BJ Homer](https://github.com/bjhomer) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#63884](https://github.com/apple/swift/pull/63884) +* Upcoming Feature Flag: `DisableOutwardActorInference` +* Review: ([pitch](https://forums.swift.org/t/pitch-stop-inferring-actor-isolation-based-on-property-wrapper-usage/63262)) ([review](https://forums.swift.org/t/se-0401-remove-actor-isolation-inference-caused-by-property-wrappers/65618)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0401-remove-actor-isolation-inference-caused-by-property-wrappers/66241)) + +## Introduction + +[SE-0316: Global Actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0316-global-actors.md) introduced annotations like `@MainActor` to isolate a type, function, or property to a particular global actor. It also introduced various rules for how that global actor isolation could be inferred. One of those rules was: + +> Declarations that are not explicitly annotated with either a global actor or `nonisolated` can infer global actor isolation from several different places: +> +> [...] +> +> - A struct or class containing a wrapped instance property with a global actor-qualified wrappedValue infers actor isolation from that property wrapper: +> +> ```swift +> @propertyWrapper +> struct UIUpdating { +> @MainActor var wrappedValue: Wrapped +> } +> +> struct CounterView { // infers @MainActor from use of @UIUpdating +> @UIUpdating var intValue: Int = 0 +> } +> ``` + + +This proposal advocates for **removing this inference rule** when compiling in the Swift 6 language mode. Given the example above, CounterView would no longer infer `@MainActor` isolation in Swift 6. + +## Motivation + +This particular inference rule is surprising and nonobvious to many users. Some developers have trouble understanding the Swift Concurrency model because it's not obvious to them when actor isolation applies. When something is inferred, it is not visible to the user, and that makes it harder to understand. This frequently arises when using the property wrappers introduced by Apple's SwiftUI framework, although it is not limited to that framework. For example: + +### An example using SwiftUI + +```swift +struct MyView: View { + // Note that `StateObject` has a MainActor-isolated `wrappedValue` + @StateObject private var model = Model() + + var body: some View { + Text("Hello, \(model.name)") + .onAppear { viewAppeared() } + } + + // This function is inferred to be `@MainActor` + func viewAppeared() { + updateUI() + } +} + +@MainActor func updateUI() { /* do stuff here */ } +``` + +The above code compiles just fine. But if we change `@StateObject` to `@State`, we get an error: + +```diff +- @StateObject private var model = Model() ++ @State private var model = Model() +``` + +```swift + func viewAppeared() { + // error: Call to main actor-isolated global function + // 'updateUI()' in a synchronous nonisolated context + updateUI() + } +``` + +Changing `@StateObject var model` to `@State var model` caused `viewAppeared()` to stop compiling, even though that function didn't use `model` at all. It feels non-obvious that changing the declaration of one property should cause a _sibling_ function to stop compiling. In fact, we also changed the isolation of the entire `MyView` type by changing one property wrapper. + +### An example not using SwiftUI + +This problem is not isolated to SwiftUI. For example: + + +```swift +// A property wrapper for use with our database library +@propertyWrapper +struct DBParameter { + @DatabaseActor public var wrappedValue: T +} + +// Inferred `@DatabaseActor` isolation because of use of `@DBParameter` +struct DBConnection { + @DBParameter private var connectionID: Int + + func executeQuery(_ query: String) -> [DBRow] { /* implementation here */ } +} + + +// In some other file... + +@DatabaseActor +func fetchOrdersFromDatabase() async -> [Order] { + let connection = DBConnection() + + // No 'await' needed here, because 'connection' is also isolated to `DatabaseActor`. + connection.executeQuery("...") +} +``` + +Removing the property wrapper on `DBConnection.connectionID` would remove the inferred actor isolation of `DBConnection`, which would in turn cause `fetchOrdersFromDatabase` to fail to compile. **It's unprecedented in Swift that changes to a _private_ property should cause compilation errors in some entirely separate file**. Upward inference of actor isolation (from property wrappers to their containing type) means that we can no longer locally reason about the effects of even *private* properties within a type. Instead, we get "spooky action at a distance". + +### Does this cause actual problems? + +This behavior has caused quite a bit of confusion in the community. For example, see [this tweet](https://twitter.com/teilweise/status/1580105376913297409?s=61&t=hwuO4NDJK1aIxSntRwDuZw), [this blog post](https://oleb.net/2022/swiftui-task-mainactor/), and [this entire Swift Forums thread](https://forums.swift.org/t/reconsider-inference-of-global-actor-based-on-property-wrappers/60821). One particular callout comes from [this post](https://forums.swift.org/t/reconsider-inference-of-global-actor-based-on-property-wrappers/60821/6/), where this inference made it hard to adopt Swift Concurrency in some cases, because the actor isolation goes "viral" beyond the intended scope: + +```swift +class MyContainer { + let contained = Contained() // error: Call to main actor-isolated initializer 'init()' in a synchronous nonisolated context +} + +class Contained { + @OnMainThread var i = 1 +} +``` + +The author created an `@OnMainThread` property wrapper, intended to declare that a particular property was isolated to the main thread. However, they cannot enforce that by using `@MainActor` within the property wrapper, because doing so causes the entire contained type to become unexpectedly isolated. + +The [original motivation](https://forums.swift.org/t/se-0401-remove-actor-isolation-inference-caused-by-property-wrappers/65618/10) for this inference rule was to reduce the annotation burden when using property wrappers like SwiftUI's `@ObservedObject`. But it's not clear it actually makes anything significantly easier; it only saves us from writing a single annotation on the type, and the loss of that annotation introduces violations of the [principle of least surprise](https://en.wikipedia.org/wiki/Principle_of_least_astonishment). + + +## Proposed solution + +The proposal is simple: In the Swift 6 language mode, property wrappers used within a type will not affect the type's actor isolation. We simply disable this inference step entirely. + +In the Swift 5 language mode, isolation will continue to be inferred as it currently is. The new behavior can be requested using the **`-enable-upcoming-feature DisableOutwardActorInference`** compiler flag. + +## Detailed design + +[`ActorIsolationRequest.getIsolationFromWrappers()`](https://github.com/apple/swift/blob/85d59d2e55e5e063c552c15f12a8abe933d8438a/lib/Sema/TypeCheckConcurrency.cpp#L3618) implements the actor isolation inference described in this proposal. That function will be adjusted to avoid producing any inference when running in the Swift 6 language mode or when the compiler flag described above is passed. + +## Source compatibility + +This change _does_ introduce potential for source incompatibility, because there may be code which was relying on the inferred actor isolation. That code can be explicitly annotated with the desired global actor in a source-compatible way right now. For example, if a type is currently inferred to have `@MainActor` isolation, you could explicitly declare that isolation on the type right now to avoid source compatibility. (See note about warnings in Alternatives Considered.) + +There may be cases where the source incompatibility could be mitigated by library authors in a source-compatible way. For example, if Apple chose to make SwiftUI's `View` protocol `@MainActor`-isolated, then all conforming types would consistently be isolated to the Main Actor, rather than being inconsistently isolated based on the usage of certain property wrappers. This proposal only notes that this mitigation may be _possible_, but does not make any recommendation as to whether that is necessary. + +### Source compatibility evaluation + +In an effort to determine the practical impact of this change, I used a macOS toolchain containing these changes and evaluated various open-source Swift projects (from the Swift Source Compatibility Library and elsewhere). I found no instances of actual source incompatibility as a result of the proposed changes. Most open-source projects are libraries that use no property wrappers at all, but I tried to specifically seek out a few projects that *do* use property wrappers and may be affected by this change. The results are as follows: + +Project | Outcome | Notes +---|---|--- +[ACNHBrowserUI](https://github.com/Dimillian/ACHNBrowserUI) | Fully Compatible | Uses SwiftUI property wrappers +[AlamoFire](https://github.com/Alamofire/Alamofire) | Fully Compatible | Uses custom property wrappers, but none are actor isolated +[Day One (Mac)](https://dayoneapp.com) | Fully Compatible | Uses SwiftUI property wrappers. (Not open source) +[Eureka](https://github.com/xmartlabs/Eureka) | Fully Compatible | Does not use property wrappers at all +[NetNewsWire](https://github.com/Ranchero-Software/NetNewsWire) | Fully Compatible | Uses SwiftUI property wrappers +[swift-nio](https://github.com/apple/swift-nio) | Fully Compatible | Does not use property wrappers at all +[SwiftyJSON](https://github.com/SwiftyJSON/SwiftyJSON) | Fully Compatible | Does not use property wrappers at all +[XcodesApp](https://github.com/RobotsAndPencils/XcodesApp) | Fully Compatible | Uses SwiftUI property wrappers + +All of the above had a `Swift Concurrency Checking` setting of **Minimal** by default. When I changed the concurrency checking level to **Targeted**, all of the above continued to compile with no errors, both with and without the proposed changes. + +When I changed the concurrency checking level to **Complete**, most of the above projects had compilation errors, _even without the changes proposed here_. The changes proposed here likely contributed a few _additional_ errors under "Complete" checking, but they did not break source compatibility in projects that would have otherwise been source compatible. + +## Effect on ABI stability + +This change is ABI stable, as the actor isolation of a type is not reflected in its runtime calling convention in any way. + +## Effect on API resilience + +This proposal has no effect on API resilience. + +## Alternatives considered + +#### Warn about Property Wrapper-based inference in Swift 5 + +In certain cases, we produce a warning that code will become invalid in a future Swift release. (For example, this has been done with the planned changes to Swift Concurrency in Swift 6.) I considered adding a warning to the Swift 5 language mode along these lines: + +```swift + +// ⚠️ Warning: `MyView` is inferred to use '@MainActor' isolation because +// it uses `@StateObject`. This inference will go away in Swift 6. +// +// Add `@MainActor` to the type to silence this warning. + +struct MyView: View { + @StateObject private var model = Model() + + var body: some View { + Text("Hello") + } +} +``` + +However, I found two problems: + +1. This would produce a _lot_ of warnings, even in code that will not break under the Swift 6 language mode. + +2. There's no way to silence this warning _without_ isolating the type. If I actually _didn't_ want the type to be isolated, there's no way to express that. You can't declare a non-isolated type: + +```swift +nonisolated // 🛑 Error: 'nonisolated' modifier cannot be applied to this declaration +struct MyView: View { + /* ... */ +} +``` + +Given that users cannot silence the warning in a way that matches the new Swift 6 behavior, it seems inappropriate to produce a warning here. + + +## Acknowledgments + +Thanks to Dave DeLong for reviewing this proposal, and to the many members of the Swift community who have engaged in discussion on this topic. diff --git a/proposals/0402-extension-macros.md b/proposals/0402-extension-macros.md new file mode 100644 index 0000000000..80513bae90 --- /dev/null +++ b/proposals/0402-extension-macros.md @@ -0,0 +1,200 @@ +# Generalize `conformance` macros as `extension` macros + +* Proposal: [SE-0402](0402-extension-macros.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#66967](https://github.com/apple/swift/pull/66967), [apple/swift-syntax#1859](https://github.com/apple/swift-syntax/pull/1859) +* Review: ([pitch](https://forums.swift.org/t/pitch-generalize-conformance-macros-as-extension-macros/65653)) ([review](https://forums.swift.org/t/se-0402-generalize-conformance-macros-as-extension-macros/65965)) ([acceptance](https://forums.swift.org/t/accepted-se-0402-generalize-conformance-macros-as-extension-macros/66276)) + +## Introduction + +This proposal generalizes the `conformance` macro role as an `extension` macro role that can add a member list to an extension in addition to a protocol and `where` clause. + +## Motivation + +[SE-0389: Attached Macros](0389-attached-macros.md) introduced conformance macros, which expand to a conformance with a `where` clause written in an extension on the type the macro is attached to: + +```swift +@attached(conformance) +macro AddEquatable() = #externalMacro(...) + +@AddEquatable +struct S {} + +// expands to +extension S: Equatable {} +``` + +However, the `conformance` macro role is extremely limited on its own. A conformance macro _only_ has the ability to return a protocol name and the syntax for a `where` clause. If the protocol conformance requires members --- as most protocol conformances do --- those must be added through a separate `member` macro role. + +More importantly, conformance macros are the only way for a macro to expand to an extension on the annotated type. The inability to add members in an extension of a type rather than the primary declaration is a serious limitation of the macro system, because extensions have several important semantic implications, including (but not limited to): + +* Protocols can only provide default implementations of requirements in extensions +* Initializers added in an extension of a type do not suppress the compiler-synthesized initializers +* Computed properties and methods in protocol and class extensions do not participate in dynamic dispatch + +Extensions also have stylistic benefits. Code inside an extension will share the generic requirements on the extension itself rather than repeating the generic requirement on every method, and implementing conformance requirements in an extension is a common practice in Swift. + +## Proposed solution + +This proposal removes the `conformance` macro role in favor of an `extension` macro role. An `extension` macro role can be used with the `@attached` macro attribute, and it can add a conformance, a `where` clause, and a member list in an extension on the type the macro is attached to: + +```swift +protocol MyProtocol { + func requirement +} + +@attached(extension, conformances: MyProtocol, names: named(requirement)) +macro MyProtocol = #externalMacro(...) + +@MyProtocol +struct S {} + +// expands to + +extension S: MyProtocol where T: MyProtocol { + func requirement() { ... } +} +``` + +The generated extensions of the macro must only extend the type the macro is attached to. Any conformances or members must also be specified upfront by the `@attached(extension)` attribute. + +## Detailed design + +### Specifying macro-introduced protocol conformances and member names + +SE-0389 states that whenever a macro produces declarations that are visible to other Swift code, it is required to declare the names in advance. This rule also applies to extension macros, which must specify: + +* Declarations inside the extension, which can be specified using `named`, `prefixed`, `suffixed`, and `arbitrary`. +* The names of protocols that are listed in the extension's conformance clause. These protocols are specified in the `conformances:` list of the `@attached(conformances:)` attribute. Each name that appears in this list must be a conformance constraint, where a conformance constraint is one of: + * A protocol name + * A typealias whose underlying type is a conformance constraint + * A protocol composition whose entries are each a conformance constraint + +The following restrictions apply to generated conformances and names listed in `@attached(extension)`: + +* An extension macro cannot add a conformance to a protocol that is not covered by the `conformances:` list in `@attached(extension, conformances:)`. +* An extension macro cannot add a member that is not covered by the `names:` list in `@attached(extension, names:)`. +* An extension macro cannot introduce an extension with an attached `peer` macro, because the peer-macro-generated names are not covered by the original `@attached(extension)` attribute. + +### Extension macro application + +Extension macros can only be attached to the primary declaration of a nominal type; they cannot be attached to typealias or extension declarations. + + +Swift only allows `extension` declarations at the top level in a file. Despite this, extension macros can be applied to a nested type: + +```swift +@attached(extension, conformances: MyProtocol, names: named(requirement)) +macro MyProtocol = #externalMacro(...) + +struct Outer { + @MyProtocol + struct Inner {} +} +``` + +In this situation, the macro expansion containing the `extension` is inserted at the top level of the file, instead of immediately where the macro is invoked, where the `extension` would be invalid. The above code expands to: + +```swift +struct Outer { + struct Inner {} +} + +extension Outer.Inner: MyProtocol { + func requirement() { ... } +} +``` + +It is an error to apply an extension macro to a local type, because there is no way to write an extension on a local type in Swift: + +```swift +func test() { + @MyProtocol // error + struct LocalType {} +} +``` + +### Implementing extension macros + +Extension macro implementations should conform to the `ExtensionMacro` protocol: + +```swift +/// Describes a macro that can add extensions of the declaration it's +/// attached to. +public protocol ExtensionMacro: AttachedMacro { + /// Expand an attached extension macro to produce the contents that will + /// create a set of extensions. + /// + /// - Parameters: + /// - node: The custom attribute describing the attached macro. + /// - declaration: The declaration the macro attribute is attached to. + /// - type: The type to provide extensions of. + /// - protocols: The list of protocols to add conformances to. These will + /// always be protocols that `type` does not already state a conformance + /// to. + /// - context: The context in which to perform the macro expansion. + /// + /// - Returns: the set of extension declarations introduced by the macro, + /// which are always inserted at top-level scope. Each extension must extend + /// the `type` parameter. + static func expansion( + of node: AttributeSyntax, + attachedTo declaration: some DeclGroupSyntax, + providingExtensionsOf type: some TypeSyntaxProtocol, + conformingTo protocols: [TypeSyntax], + in context: some MacroExpansionContext + ) throws -> [ExtensionDeclSyntax] +} +``` + +Each `ExtensionDeclSyntax` in the resulting array must use the `providingExtensionsOf` parameter as the extended type, which is a qualified type name. For example, for the following code: + +```swift +struct Outer { + @MyProtocol + struct Inner {} +} +``` + +The type syntax passed to `ExtensionMacro.expansion` for `providingExtensionsOf` is `Outer.Inner`. + +#### Suppressing redundant conformances + +The `conformingTo:` parameter of `ExtensionMacro.expansion` allows extension macros to suppress generating conformances that are already stated in the original source code. The `conformingTo:` argument array will contain only the protocols from the `conformances:` list in `@attached(extension conformances:)` that the type does not already conform to in the original source code, including through implied conformances or class inheritance. + +For example, consider the following code which contains an attached extension macro: + +```swift +protocol Encodable {} +protocol Decodable {} + +typealias Codable = Encodable & Decodable + +@attached(extension, conformances: Codable) +macro MyMacro() = #externalMacro(...) + +@MyMacro +struct S { ... } + +extension S: Encodable { ... } +``` + +The extension macro can add conformances to `Codable`, aka `Encodable & Decodable`. Because the struct `S` already conforms to `Encodable` in the original source, the `ExtensionMacro.expansion` method will receive the argument `[TypeSyntax(Encodable)]` for the `conformingTo:` parameter. Using this information, the macro implementation can decide to only add an extension with a conformance to `Decodable`. + +## Source compatibility + +This proposal removes the `conformance` macro role from SE-0389, which is accepted and implemented in Swift 5.9. If this proposal is accepted after 5.9, the `conformance` macro role will remain in the language as sugar for an `extension` macro that adds only a conformance. + +## ABI compatibility + +Extensions macros are expanded to regular Swift code at compile-time and have no ABI impact. + +## Implications on adoption + +The adoption implications for using extensions macros are the same as writing the expanded code manually in the project. + +## Acknowledgments + +Thank you to Gwendal Roué for inspiring the idea of `extension` macros by suggesting combining `member` macros and `conformance` macros. diff --git a/proposals/0403-swiftpm-mixed-language-targets.md b/proposals/0403-swiftpm-mixed-language-targets.md new file mode 100644 index 0000000000..c339f2e424 --- /dev/null +++ b/proposals/0403-swiftpm-mixed-language-targets.md @@ -0,0 +1,576 @@ +# Package Manager Mixed Language Target Support + +* Proposal: [SE-0403](0403-swiftpm-mixed-language-targets.md) +* Authors: [Nick Cooke](https://github.com/ncooke3) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Returned for Revision** +* Implementation: [apple/swift-package-manager#5919](https://github.com/apple/swift-package-manager/pull/5919) +* Review: ([pitch](https://forums.swift.org/t/61564)), ([review](https://forums.swift.org/t/66202)), ([returned for revision](https://forums.swift.org/t/66975)) + +## Introduction + +This is a proposal for adding package manager support for targets containing +both Swift and [C based language sources][SE-0038] (henceforth, referred to as +mixed language sources). Currently, a target’s source can be either Swift or a +C based language ([SE-0038]), but not both. + +Swift-evolution thread: [Discussion thread topic for that +proposal](https://forums.swift.org/) + +## Motivation + +This proposal enables Swift Package Manager support for multi-language targets. + +Packages may need to contain mixed language sources for both legacy or +technical reasons. For developers building or maintaining packages with mixed +languages (e.g. Swift and Objective-C), there are two workarounds for doing so +with Swift Package Manager, but they have drawbacks that degrade the developer +experience, and sometimes are not even an option: +- Distribute binary frameworks via binary targets. Drawbacks include that the + package will be less portable as it can only support platforms that the + binaries support, binary dependencies are only available on Apple platforms, + customers cannot view or easily debug the source in their project workspace, + and tooling is required to generate the binaries for release. +- Separate a target’s implementation into sub-targets based on language type, + adding dependencies where necessary. For example, a target `Foo` may have + Swift-only sources that can call into an underlying target `FooObjc` that + contains Clang-only sources. Drawbacks include needing to depend on the + public API surfaces between the targets, increasing the complexity of the + package’s manifest and organization for both maintainers and clients, and + preventing package developers from incrementally migrating internal + implementation from one language to another (e.g. Objective-C to Swift) since + there is still a separation across targets based on language. + +Package manager support for mixed language targets addresses both of the above +drawbacks by enabling developers to mix sources of supported languages within a +single target without complicating their package’s structure or developer +experience. + +## Proposed solution + +Package authors can create a mixed target by mixing language sources in their +target's source directory. When mixing some languages, like C++, authors have +the option of opting in to advanced interoperability features by configuring +the target with an interoperability mode [`SwiftSetting.InteroperabilityMode`]. + +When building a mixed language target, the package manager will build the +public API into a single module for use by clients. + +At a high level, the build process is split into two parts based on +the language of the sources. The Swift sources are built by the Swift compiler +and the C/Objective-C/C++ sources are built by the Clang compiler. + +1. The Swift compiler is made aware of the Clang part of the package when + building the Swift sources into a `swiftmodule`. +1. The Clang part of the package is built with knowledge of the + interoperability Swift header. The contents of this header will vary + depending on if/what language-specific interoperability mode is configured + on the target. The interoperability header is modularized as part of the + mixed target's public interface. + + +The [following example][mixed-package] defines a package containing mixed +language sources. + +``` +MixedPackage +├── Package.swift +├── Sources +│   └── MixedPackage +│ ├── Jedi.swift ⎤-- Swift sources +│ ├── Lightsaber.swift ⎦ +│ ├── Sith.m ⎤-- Implementations & internal headers +│ ├── SithRegistry.h ⎟ +│ ├── SithRegistry.m ⎟ +│   ├── droid_debug.c ⎦ +│   ├── hello_there.txt ]-- Resources +│   └── include ⎤-- Public headers +│   ├── MixedPackage.h ⎟ +│   ├── Sith.h ⎟ +│   └── droid_debug.h ⎦ +└── Tests + └── MixedPackageTests + ├── JediTests.swift ]-- Swift tests + ├── SithTests.m        ]-- Objective-C tests + ├── ObjcTestConstants.h ⎤-- Mixed language test utils + ├── ObjcTestConstants.m ⎟ + └── SwiftTestConstants.swift ⎦ +``` + +The proposed solution would enable the above targets to do the following: +1. Export their public API, if any, from across the mixed language sources. +1. Use C/Objective-C/C++ compatible Swift API from target’s Swift sources + within the target’s C/Objective-C/C++ sources. +1. Use Swift compatible C/Objective-C/C++ API from target’s C/Objective-C/C++ + sources within the target’s Swift sources. +1. Access target resources from Swift and Objective-C contexts. + +### Limitations + +Initial support for targets containing mixed language sources will have the +following limitations: +1. The target must be either a library or test target. Support for other types + of targets is deferred until the use cases become clear. +1. If the target contains a custom module map, it cannot contain a submodule of + the form `$(ModuleName).Swift`. This is because the package manager will + synthesize an _extended_ module map that includes a submodule that + modularizes the generated Swift interop header. + +### Importing a mixed target + +Mixed targets can be imported into a client target in several ways. The +following examples will reference `MixedPackage`, a package containing mixed +language target(s). + +#### Importing within a **Swift** context + +The public API of a mixed target, `MixedPackage`, can be imported into a +**Swift** file via an `import` statement: + +```swift +// MyClientTarget.swift + +import MixedPackage +``` + +Testing targets can import the mixed target via +`@testable import MixedPackage`. As expected, this will expose internal Swift +types within the module. It will not expose any non-public C language types. + +#### Importing within an **C/Objective-C/C++** context + +How a mixed target, `MixedPackage`, is imported into an **C/Objective-C/C++** +file will vary depending on the language it is being imported in. + +When Clang modules are supported, clients can import the module. Textual +imports are also an option. + + +For this example, consider `MixedPackage` being organized as such: + +``` +MixedPackage +├── Package.swift +└── Sources +    ├── NewCar.swift +   └── include ]-- Public headers directory +    ├── OldCar.h + └── MixedPackage-Swift.h ]-- This header is generated + during the build. +``` + + +Like Clang targets, `MixedPackage`'s public headers directory (`include` in the +above example) is added a header search path to client targets. The following +example demonstrates all the possible public headers that can be imported from +`MixedPackage`. + +```objc +// MyClientTarget.m + +// If module imports are supported, the public API (including API in the +// generated Swift header) can be imported via a module import. +@import MixedPackage; +// Imports types defined in `OldCar.h`. +#import "OldCar.h" +// Imports Objective-C compatible Swift types defined in `MixedPackage`. +#import "MixedPackage-Swift.h" +``` + +## Plugin Support + +Package manager plugins should be able to process mixed language source +targets. The following type will be added to the `PackagePlugin` module +to represent a mixed language target in a plugin's context. + +This API was created by joining together the properties of the existing +`SwiftSourceModuleTarget` and `ClangSourceModuleTarget` types +([source][Swift-Clang-SourceModuleTarget]). + +```swift +/// Represents a target consisting of a source code module compiled using both the Clang and Swift compiler. +public struct MixedSourceModuleTarget: SourceModuleTarget { + /// Unique identifier for the target. + public let id: ID + + /// The name of the target, as defined in the package manifest. This name + /// is unique among the targets of the package in which it is defined. + public let name: String + + /// The kind of module, describing whether it contains unit tests, contains + /// the main entry point of an executable, or neither. + public let kind: ModuleKind + + /// The absolute path of the target directory in the local file system. + public let directory: Path + + /// Any other targets on which this target depends, in the same order as + /// they are specified in the package manifest. Conditional dependencies + /// that do not apply have already been filtered out. + public let dependencies: [TargetDependency] + + /// The name of the module produced by the target (derived from the target + /// name, though future SwiftPM versions may allow this to be customized). + public let moduleName: String + + /// The source files that are associated with this target (any files that + /// have been excluded in the manifest have already been filtered out). + public let sourceFiles: FileList + + /// Any custom compilation conditions specified for the target's Swift sources. + public let swiftCompilationConditions: [String] + + /// Any preprocessor definitions specified for the target's Clang sources. + public let clangPreprocessorDefinitions: [String] + + /// Any custom header search paths specified for the Clang target. + public let headerSearchPaths: [String] + + /// The directory containing public C headers, if applicable. This will + /// only be set for targets that have a directory of a public headers. + public let publicHeadersDirectory: Path? + + /// Any custom linked libraries required by the module, as specified in the + /// package manifest. + public let linkedLibraries: [String] + + /// Any custom linked frameworks required by the module, as specified in the + /// package manifest. + public let linkedFrameworks: [String] +} +``` + + +## Detailed design + +### Modeling a mixed language target + +Up until this proposal, when a package was loading, each target was represented +programmatically as either a [`SwiftTarget`] or [`ClangTarget`]. Which of these +types to use was informed by the sources found in the target. For targets with +mixed language sources, an error was thrown and surfaced to the client. During +the build process, each of those types mapped to another type +([`SwiftTargetBuildDescription`] or [`ClangTargetBuildDescription`]) that +described how the target should be built. + +This proposal adds two new types, `MixedTarget` and `MixedTargetDescription`, +that represent targets with mixed language sources during the package loading +and building phases, respectively. + +While an implementation detail, it’s worth noting that in this approach, a +`MixedTarget` is a wrapper type around an underlying `SwiftTarget` and +`ClangTarget`. Initializing a `MixedTarget` will internally initialize a +`SwiftTarget` from the given Swift sources and a `ClangTarget` from the given +Clang sources. This extends to the `MixedTargetDescription` type in that it +wraps a `SwiftTargetDescription` and `ClangTargetDescription`. + +Using this approach allows for greater code-reuse, and reduces the chance of +introducing a regression from changing existing sub-target types like +`SwiftTarget` and `ClangTarget`. + +The role of the `MixedTargetBuildDescription` is to generate auxiliary +artifacts needed for the build and pass specific build flags to the underlying +`SwiftTargetBuildDescription` and `ClangTargetBuildDescription`. + +The following diagram shows the relationship between the various types. +```mermaid +flowchart LR + A>Swift sources] --> B[SwiftTarget] --> C[SwiftTargetBuildDescription] + D>Clang sources] --> E[ClangTarget] --> F[ClangTargetBuildDescription] + + subgraph MixedTarget + SwiftTarget + ClangTarget + end + + subgraph MixedTargetBuildDescription + SwiftTargetBuildDescription + ClangTargetBuildDescription + end + + G>Mixed sources] --> MixedTarget --> MixedTargetBuildDescription +``` + +### Building a mixed language target + + + + + + + +The Swift part of the target is built before the Clang part. This is because +the C language sources may require resolving a textual import of the generated +interop header, and that header is emitted alongside the Swift module when the +Swift part of the target is built. This relationship is enforced in that the +generated interop header is listed as an input to the compilation commands for +the target’s C language sources. This is specified in the llbuild manifest +(`debug.yaml` in the package's `.build` directory). + +##### Additional Swift build flags +The following flags are additionally used when compiling the Swift sub-target: +1. `-import-underlying-module` This flag triggers a partial build of the + underlying C language sources when building the Swift module. This critical + flag enables the Swift sources to use C language types defined in the Clang + part of the target. +1. `-I /path/to/modulemap_dir` The above `-import-underlying-module` flag + will look for a module map in the given header search path. The module + map used here cannot modularize the generated interop header as will be + created from building the Swift sub-target and therefore does not exist + yet. If a custom module map is provided, the public headers directory + will be used as that is where the custom module map is enforced to be + located. It's also enforced that this module map does not expose an + interop header. If a custom module map is _not_ provided, the package + manager will pass the target's build directory as that is where a module + map will be synthesized. This module map will be _un-extended_, in that + it does not modularize the generated interop header. +1. _If a custom module is NOT provided,_ the package manager will synthesize + two module maps. One is _extended_ in that it modualrizes the generated + interop header. The other is _un-extended_ in that it does not modularize + the generated interop header. A VFS Overlay file is created to swap the + extended one (named `module.modulemap`) for the unextended one + (`unextended-module.modulemap`) for the build. +1. `-Xcc -I -Xcc $(TARGET_SRC_PATH)` Adding the target's [path] allows for + importing headers using paths relative to the root of the target. Because + passing `-import-underlying-module` triggers a partial build of the Clang + sources, this is needed for resolving possible header imports. +1. `-Xcc -I -Xcc $(TARGET_PUBLIC_HDRS)` Adding the target's public header's + path allows for importing headers using paths relative to the public + header's directory. Because passing `-import-underlying-module` triggers + a partial build of the Clang sources, this is needed for resolving + possible header imports. + +##### Additional Clang build flags +The following flags are additionally used when compiling the Clang sub-target: +1. `-I $(target’s path)` Adding the target's [path] allows for importing + headers using paths relative to the root of the target. +1. `-I /path/to/generated_swift_header_dir/` The generated Swift header may be + needed when compiling the Clang sources. + +#### Performing the build + +To actually build a package, the package manager creates a llbuild manifest and +passes it to the llbuild system. Adding support for mixed targets involved +modifying [LLBuildManifestBuilder.swift] to convert a +`MixedTargetBuildDescription` into llbuild build nodes. +`MixedTargetBuildDescription` intentionally wraps and configures an underlying +`SwiftTargetBuildDescription` and `ClangTargetBuildDescription`. This means +that creating a llbuild build node for a mixed target is really just creating +build nodes for the its `SwiftTargetBuildDescription` and +`ClangTargetBuildDescription`, respectively. + +#### Build artifacts for client targets + + + + + + + +##### Module Maps + +The client-facing module map’s purpose is to define the public API of the mixed +language module. It has two parts, a primary module declaration and a secondary +submodule declaration. The former of which exposes the public C language +headers and the latter of which exposes the generated interop header. + +There are two cases when creating the client-facing module map: +- If a custom module map exists in the target, its contents are copied and + extended to modularize the generated interop header. These contents are + written to the build directory as `extended-custom-module.modulemap`. + Since the public header directory and build directory are passed as import + paths to the build invocations, a different name is needed for this module + map as the `-import-underlying-module` should only be able to find one + `module.modulemap` file from the given import paths. +- Else, the module map’s contents will be generated via the same + generation rules established in [SE-0038] with an added step to generate the + `.Swift` submodule. This file is called `module.modulemap` and lives in the + build directory. + +Clients will use an _extended_ module map that includes the modularized interop +header. Building the target will use _unextended_ module map. + +> Note: It’s possible that the Clang part of the module exports no public API. +> This could be the case for a target whose public API surface is written in +> Swift but whose implementation is written in Objective-C. In this case, the +> primary module declaration will expose no headers. + +Below is an example of a module map for a target that has an umbrella +header in its public headers directory (`include`). + +``` +// extended-custom-module.modulemap + +// This declaration is either copied from the custom module map or generated +// via the rules from SE-0038. +module MixedTarget { + umbrella header "/Users/crusty/Developer/MixedTarget/Sources/MixedTarget/include/MixedTarget.h" + export * +} +// This is added on by the package manager as part of this proposal. +module MixedTarget.Swift { + header "MixedTarget-Swift.h" + requires objc +} +``` + +##### all-product-headers.yaml + +An `all-product-headers.yaml` VFS overlay file will adjust the public headers +directory to expose the interop header as a relative path, and, if a custom +module map exists, swap it out for the extended one that modualrizes the interop +header. + +In either case, it will be passed alongside the module map as a compilation +argument to clients: +``` +-fmodule-map-file=/Users/crusty/Developer/MixedTarget/Sources/MixedTarget/include/module.modulemap +-ivfsoverlay /Users/crusty/Developer/MixedTarget/.build/.../MixedTarget.build/Product/all-product-headers.yaml +``` + +### Additional changes to the package manager + +It is the goal for mixed language targets to work on all platforms supported +by the package manager. One obstacle to that is that the package manager, +at the time of this proposal, does not invoke the build system with the +flag needed to emit the interoperability header +([code][should-emit-header]). This limitation is outdated and will be +removed as part of this proposal. + +See the related discussion [thread][swift-emit-header-fr] from the initial +formal review. + +### Related change to the Swift compiler + +When the Swift compiler creates the generated interop header (via +`-emit-objc-header`), any Objective-C symbol referenced in the Swift API that +cannot be forward declared (e.g. superclass, protocol, etc.) will attempt to +be imported via an umbrella header. Since the compiler evaluates +the target as a framework (as opposed to an app), the compiler assumes an +umbrella header exists in a subdirectory (named after the module) within +the public headers directory: + + #import <$(ModuleName)/$(ModuleName).h> + +The compiler assumes that the above path can be resolved relative to the public +header directory. Instead of forcing package authors to structure their +packages around that constraint, the Swift compiler's interop header generation +logic will be amended to do the following in such cases where the target +does not have the public headers directory structure of an xcframework: + +- If an umbrella header that is modularized by the Clang module exists, the + interop header emit a reference directly to that umbrella header instead. +- Else, the interop header will import all textual includes from the Clang + module map. + +See the related discussion [thread][swift-compiler-thread-fr] from the initial +formal review. + +### Mixed language Test Targets + +To complement library targets with mixed languages, mixed test targets are +also supported as part of this proposal. + +Using the [example package][mixed-package] from before, consider the following +layout of the package's `Tests` directory. + +``` +MixedPackage +├── ... +└── Tests + └── MixedPackageTests + ├── JediTests.swift ]-- Swift tests + ├── SithTests.m        ]-- Objective-C tests + ├── ObjcTestConstants.h ⎤-- Mixed language test utils + ├── ObjcTestConstants.m ⎟ + └── TestConstants.swift ⎦ +``` + +The types defined in `ObjcTestConstants.h` are visible in `SithTests.m` (via +importing the header). + +The Objective-C compatible types defined in `TestConstants.swift` are visible +in `JediTests.swift` (via importing the header). + +This design should give package authors flexibility in designing test suites +for their mixed targets. + + +### Failure cases + +There are several failure cases that may surface to end users: +- Attempting to build a mixed target using a tools version that does not + include this proposal’s implementation. + ``` + target at '\(path)' contains mixed language source files; feature not supported + ``` +- Attempting to build a mixed target that is neither a library target + or test target. + ``` + Target with mixed sources at '\(path)' is a \(type) target; targets + with mixed language sources are only supported for library and test + targets. + ``` +- Attempting to build a mixed target containing a custom module map + that contains a `$(MixedTargetName).Swift` submodule. + ``` + The target's module map may not contain a Swift submodule for the + module \(target name). + ``` + +## Security + +This has no impact on security, safety, or privacy. + +## Impact on existing packages + +This proposal will not affect the behavior of existing packages. In the +proposed solution, the code path to build a mixed language package is separate +from the existing code paths to build packages with Swift sources and C +Language sources, respectively. + +Additionally, this feature will be gated on a tools minor version update, so +mixed language targets building on older toolchains that do not support this +feature will continue to [throw an error][mixed-target-error]. + +## Future Directions + +- Enable package authors to expose non-public headers to their mixed + target's Swift implementation. +- Extend mixed language target support to currently unsupported types of + targets (e.g. executables). +- Extend this solution so that all targets are mixed language targets by + default. This could simplify the implementation as language-specific types + like `ClangTarget`, `SwiftTarget`, and `MixedTarget` could be consolidated + into a single type. This approach was avoided in the initial implementation + of this feature to reduce the risk of introducing a regression. + + + +[SE-0038]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0038-swiftpm-c-language-targets.md + +[mixed-package]: https://github.com/ncooke3/MixedPackage + +[`SwiftTarget`]: https://github.com/apple/swift-package-manager/blob/ce099264a187759c2f587393bd209d317a0352b4/Sources/PackageModel/Target.swift#L313 + +[`ClangTarget`]: https://github.com/apple/swift-package-manager/blob/ce099264a187759c2f587393bd209d317a0352b4/Sources/PackageModel/Target.swift#L470 + +[`SwiftTargetBuildDescription`]: https://github.com/apple/swift-package-manager/blob/main/Sources/Build/BuildPlan.swift#L549 + +[`ClangTargetBuildDescription`]: https://github.com/apple/swift-package-manager/blob/ce099264a187759c2f587393bd209d317a0352b4/Sources/Build/BuildPlan.swift#L232 + +[path]: https://developer.apple.com/documentation/packagedescription/target/path + +[LLBuildManifestBuilder.swift]: https://github.com/apple/swift-package-manager/blob/14d05ccaa13b768449cd405fff81d630a520e04a/Sources/Build/LLBuildManifestBuilder.swift + +[mixed-target-error]: https://github.com/apple/swift-package-manager/blob/ce099264a187759c2f587393bd209d317a0352b4/Sources/PackageLoading/TargetSourcesBuilder.swift#L183-L189 + +[`SwiftSetting.InteroperabilityMode`]: https://developer.apple.com/documentation/packagedescription/swiftsetting/interoperabilitymode + +[swift-compiler-thread-fr]: https://forums.swift.org/t/se-0403-package-manager-mixed-language-target-support/66202/32 + +[should-emit-header]: https://github.com/apple/swift-package-manager/blob/6478e2724b8bf77856ff358cba5f59a4a62978bf/Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift#L732-L735 + +[swift-emit-header-fr]: https://forums.swift.org/t/se-0403-package-manager-mixed-language-target-support/66202/31 + +[Swift-Clang-SourceModuleTarget]: https://github.com/apple/swift-package-manager/blob/8e512308530f808e9ef0cd149f4f632339c65bc4/Sources/PackagePlugin/PackageModel.swift#L231-L319 diff --git a/proposals/0404-nested-protocols.md b/proposals/0404-nested-protocols.md new file mode 100644 index 0000000000..9722c8b280 --- /dev/null +++ b/proposals/0404-nested-protocols.md @@ -0,0 +1,168 @@ +# Allow Protocols to be Nested in Non-Generic Contexts + +* Proposal: [SE-0404](0404-nested-protocols.md) +* Authors: [Karl Wagner](https://github.com/karwa) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.10)** +* Implementation: [apple/swift#66247](https://github.com/apple/swift/pull/66247) (gated behind flag `-enable-experimental-feature NestedProtocols`) +* Review: ([pitch](https://forums.swift.org/t/pitch-allow-protocols-to-be-nested-in-non-generic-contexts/65285)), ([review](https://forums.swift.org/t/se-0404-allow-protocols-to-be-nested-in-non-generic-contexts/66332)), ([acceptance](https://forums.swift.org/t/accepted-se-0404-allow-protocols-to-be-nested-in-non-generic-contexts/66668)) + +## Introduction + +Allows protocols to be nested in non-generic `struct/class/enum/actor`s, and functions. + +## Motivation + +Nesting nominal types inside other nominal types allows developers to express a natural scope for the inner type -- for example, `String.UTF8View` is `struct UTF8View` nested within `struct String`, and its name clearly communicates its purpose as an interface to the UTF-8 code-units of a String value. + +However, nesting is currently restricted to struct/class/enum/actors within other struct/class/enum/actors; protocols cannot be nested at all, and so must always be top-level types within a module. This is unfortunate, and we should relax this restriction so that developers can express protocols which are naturally scoped to some outer type. + +## Proposed solution + +We would allow nesting protocols within non-generic struct/class/enum/actors, and also within functions that do not belong to a generic context. + +For example, `TableView.Delegate` is naturally a delegate protocol pertaining to table-views. Developers should be declare it as such - nested within their `TableView` class: + +```swift +class TableView { + protocol Delegate: AnyObject { + func tableView(_: TableView, didSelectRowAtIndex: Int) + } +} + +class DelegateConformer: TableView.Delegate { + func tableView(_: TableView, didSelectRowAtIndex: Int) { + // ... + } +} +``` + +Currently, developers resort to giving things compound names, such as `TableViewDelegate`, to express the same natural scoping that could otherwise be expressed via nesting. + +As an additional benefit, within the context of `TableView`, the nested protocol `Delegate` can be referred to by a shorter name (as is the case with all other nested types): + +```swift +class TableView { + weak var delegate: Delegate? + + protocol Delegate { /* ... */ } +} +``` + +Protocols can also be nested within non-generic functions and closures. Admittedly, this is of somewhat limited utility, as all conformances to such protocols must also be within the same function. However, there is also no reason to artificially limit the complexity of the models which developers create within a function. Some codebases (of note, the Swift compiler itself) make use of large closures with nested types, and they benefit from abstractions using protocols. + +```swift +func doSomething() { + + protocol Abstraction { + associatedtype ResultType + func requirement() -> ResultType + } + struct SomeConformance: Abstraction { + func requirement() -> Int { ... } + } + struct AnotherConformance: Abstraction { + func requirement() -> String { ... } + } + + func impl(_ input: T) -> T.ResultType { + // ... + } + + let _: Int = impl(SomeConformance()) + let _: String = impl(AnotherConformance()) +} +``` + +## Detailed design + +Protocols may be nested anywhere that a struct/class/enum/actor may be nested, with the exception of generic contexts. For example, the following remains forbidden: + +```swift +class TableView { + + protocol Delegate { // Error: protocol 'Delegate' cannot be nested within a generic context. + func didSelect(_: Element) + } +} +``` + +The same applies to generic functions: + +```swift +func genericFunc(_: T) { + protocol Abstraction { // Error: protocol 'Abstraction' cannot be nested within a generic context. + } +} +``` + +And to other functions within generic contexts: + +```swift +class TableView { + func doSomething() { + protocol MyProtocol { // Error: protocol 'Abstraction' cannot be nested within a generic context. + } + } +} +``` + +Supporting this would require either: + +- Introducing generic protocols, or +- Mapping generic parameters to associated types. + +Neither is in in-scope for this proposal, but this author feels there is enough benefit here even without supporting generic contexts. Either of these would certainly make for interesting future directions. + +### Associated Type matching + +When nested in a concrete type, protocols do not witness associated type requirements. + +```swift +protocol Widget { + associatedtype Delegate +} + +struct TableWidget: Widget { + // Does NOT witness Widget.Delegate + protocol Delegate { ... } +} +``` + +Associated types associate one concrete type with one conforming type. Protocols are constraint types which many concrete types may conform to, so there is no obvious meaning to having a protocol witness an associated type requirement. + +There have been discussions in the past about whether protocols could gain an "associated protocol" feature, which would allow these networks of constraint types to be expressed. If such a feature were ever introduced, it may be reasonable for associated protocol requirements to be witnessed by nested protocols, the same way associated type requirements can be witnessed by nested concrete types today. + +## Source compatibility + +This feature is additive. + +## ABI compatibility + +This proposal is purely an extension of the language's ABI and does not change any existing features. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints. + +In general, moving a protocol in/out of a parent context is a source-breaking change. However, this breakage can mitigated by providing a `typealias` to the new name. + +As with other nested types, the parent context forms part of the mangled name of a nested protocol. Therefore, moving a protocol in/out of a parent context is an ABI-incompatible change. + +## Future directions + +- Allow nesting other (non-protocol) types in protocols + + Protocols themselves sometimes wish to define types which are naturally scoped to that protocol. For example, the standard library's [`FloatingPointRoundingRule`](https://developer.apple.com/documentation/swift/FloatingPointRoundingRule) enum is used by a [requirement of the `FloatingPoint` protocol](https://developer.apple.com/documentation/swift/floatingpoint/round(_:)) and defined for that purpose. + +- Allow nesting protocols in generic types + + As mentioned in the Detailed Design section, there are potentially strategies that would allow nesting protocols within generic types, and one could certainly imagine ways to use that expressive capability. The community is invited to discuss potential approaches in a separate topic. + +## Alternatives considered + +None. This is a straightforward extension of the language's existing nesting functionality. + +## Acknowledgments + +Thanks to [`@jumhyn`](https://forums.swift.org/u/jumhyn/) for reminding me about this, and to [`@suyashsrijan`](https://forums.swift.org/u/suyashsrijan/). diff --git a/proposals/0405-string-validating-initializers.md b/proposals/0405-string-validating-initializers.md new file mode 100644 index 0000000000..636b4092cc --- /dev/null +++ b/proposals/0405-string-validating-initializers.md @@ -0,0 +1,223 @@ +# String Initializers with Encoding Validation + +* Proposal: [SE-0405](0405-string-validating-initializers.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 6.0)** +* Bugs: rdar://99276048, rdar://99832858 +* Implementation: [Swift PR 68419](https://github.com/apple/swift/pull/68419), [Swift PR 68423](https://github.com/apple/swift/pull/68423) +* Review: ([pitch](https://forums.swift.org/t/66206)), ([review](https://forums.swift.org/t/se-0405-string-initializers-with-encoding-validation/66655)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0405-string-initializers-with-encoding-validation/67134)) +* Previous Revisions: [0](https://gist.github.com/glessard/d1ed79b7968b4ad2115462b3d1eba805), [1](https://github.com/swiftlang/swift-evolution/blob/37531427931a57ff2a76225741c99de8fa8b8c59/proposals/0405-string-validating-initializers.md) + +## Introduction + +We propose adding new `String` failable initializers that validate encoded input, and return `nil` when the input contains any invalid elements. + +## Motivation + +The `String` type guarantees that it represents well-formed Unicode text. When data representing text is received from a file, the network, or some other source, it may be relevant to store it in a `String`, but that data must be validated first. `String` already provides a way to transform data to valid Unicode by repairing invalid elements, but such a transformation is often not desirable, especially when dealing with untrusted sources. For example a JSON decoder cannot transform its input; it must fail if a span representing text contains any invalid UTF-8. + +This functionality has not been available directly from the standard library. It is possible to compose it using existing public API, but only at the cost of extra memory copies and allocations. The standard library is uniquely positioned to implement this functionality in a performant way. + +## Proposed Solution + +We will add a new `String` initializer that can fail, returning `nil`, when its input is found to be invalid according the encoding represented by a type parameter that conforms to `Unicode.Encoding`. + +```swift +extension String { + public init?( + validating codeUnits: some Sequence, + as: Encoding.Type + ) +} +``` + +When processing data obtained from C, it is frequently the case that UTF-8 data is represented by `Int8` (typically as `CChar`) rather than `UInt8`. We will provide a convenience initializer for this use case: + +```swift +extension String { + public init?( + validating codeUnits: some Sequence, + as: Encoding.Type + ) where Encoding.CodeUnit == UInt8 +} +``` + +`String` already features a validating initializer for UTF-8 input, intended for C interoperability. Its argument label does not convey the expectation that its input is a null-terminated C string, and this has caused errors. We propose to change the labels in order to clarify the preconditions: + +```swift +extension String { + public init?(validatingCString nullTerminatedUTF8: UnsafePointer) + + @available(Swift 5.XLIX, deprecated, renamed: "String.init(validatingCString:)") + public init?(validatingUTF8 cString: UnsafePointer) +} +``` + +Note that unlike `String.init?(validatingCString:)`, the `String.init?(validating:as:)` initializers convert their whole input, including any embedded `\0` code units. + +## Detailed Design + +We want these new initializers to be performant. As such, their implementation should minimize the number of memory allocations and copies required. We achieve this performance with `@inlinable` implementations that leverage `withContiguousStorageIfAvailable` to provide a concrete (`internal`) code path for the validation cases. The concrete `internal` initializer itself calls a number of functions internal to the standard library. + +```swift +extension String { + /// Creates a new `String` by copying and validating the sequence of + /// code units passed in, according to the specified encoding. + /// + /// This initializer does not try to repair ill-formed code unit sequences. + /// If any are found, the result of the initializer is `nil`. + /// + /// The following example calls this initializer with the contents of two + /// different arrays---first with a well-formed UTF-8 code unit sequence and + /// then with an ill-formed UTF-16 code unit sequence. + /// + /// let validUTF8: [UInt8] = [67, 97, 0, 102, 195, 169] + /// let valid = String(validating: validUTF8, as: UTF8.self) + /// print(valid) + /// // Prints "Optional("Café")" + /// + /// let invalidUTF16: [UInt16] = [0x41, 0x42, 0xd801] + /// let invalid = String(validating: invalidUTF16, as: UTF16.self) + /// print(invalid) + /// // Prints "nil" + /// + /// - Parameters: + /// - codeUnits: A sequence of code units that encode a `String` + /// - encoding: A conformer to `Unicode.Encoding` to be used + /// to decode `codeUnits`. + @inlinable + public init?( + validating codeUnits: some Sequence, + as encoding: Encoding.Type + ) where Encoding: Unicode.Encoding + + /// Creates a new `String` by copying and validating the sequence of + /// `Int8` passed in, according to the specified encoding. + /// + /// This initializer does not try to repair ill-formed code unit sequences. + /// If any are found, the result of the initializer is `nil`. + /// + /// The following example calls this initializer with the contents of two + /// different arrays---first with a well-formed UTF-8 code unit sequence and + /// then with an ill-formed ASCII code unit sequence. + /// + /// let validUTF8: [Int8] = [67, 97, 0, 102, -61, -87] + /// let valid = String(validating: validUTF8, as: UTF8.self) + /// print(valid) + /// // Prints "Optional("Café")" + /// + /// let invalidASCII: [Int8] = [67, 97, -5] + /// let invalid = String(validating: invalidASCII, as: Unicode.ASCII.self) + /// print(invalid) + /// // Prints "nil" + /// + /// - Parameters: + /// - codeUnits: A sequence of code units that encode a `String` + /// - encoding: A conformer to `Unicode.Encoding` that can decode + /// `codeUnits` as `UInt8` + @inlinable + public init?( + validating codeUnits: some Sequence, + as encoding: Encoding.Type + ) where Encoding: Unicode.Encoding, Encoding.CodeUnit == UInt8 +} +``` + +```swift +extension String { + /// Creates a new string by copying and validating the null-terminated UTF-8 + /// data referenced by the given pointer. + /// + /// This initializer does not try to repair ill-formed UTF-8 code unit + /// sequences. If any are found, the result of the initializer is `nil`. + /// + /// The following example calls this initializer with pointers to the + /// contents of two different `CChar` arrays---first with well-formed + /// UTF-8 code unit sequences and the second with an ill-formed sequence at + /// the end. + /// + /// let validUTF8: [CChar] = [67, 97, 102, -61, -87, 0] + /// validUTF8.withUnsafeBufferPointer { ptr in + /// let s = String(validatingCString: ptr.baseAddress!) + /// print(s) + /// } + /// // Prints "Optional("Café")" + /// + /// let invalidUTF8: [CChar] = [67, 97, 102, -61, 0] + /// invalidUTF8.withUnsafeBufferPointer { ptr in + /// let s = String(validatingCString: ptr.baseAddress!) + /// print(s) + /// } + /// // Prints "nil" + /// + /// - Parameter nullTerminatedUTF8: A pointer to a null-terminated UTF-8 code sequence. + @_silgen_name("sSS14validatingUTF8SSSgSPys4Int8VG_tcfC") + public init?(validatingCString nullTerminatedUTF8: UnsafePointer) + + @available(*, deprecated, renamed: "String.init(validatingCString:)") + @_silgen_name("_swift_stdlib_legacy_String_validatingUTF8") + @_alwaysEmitIntoClient + public init?(validatingUTF8 cString: UnsafePointer) +} +``` + +## Source Compatibility + +This proposal consists mostly of additions, which are by definition source compatible. + +The proposal includes the renaming of one function from `String.init?(validatingUTF8:)` to `String.init?(validatingCString:)`. The existing function name will be deprecated, producing a warning. A fixit will support an easy transition to the renamed version of the function. + +## ABI Compatibility + +This proposal adds new functions to the ABI. + +The renamed function reuses the existing ABI entry point, making the change ABI-compatible. + +## Implications on adoption + +This feature requires a new version of the standard library. + +## Alternatives considered + +#### Initializers specifying the encoding by their argument label + +For convenience and discoverability for the most common case, we originally proposed an initializer that specifies the UTF-8 input encoding as part of its argument label: + +```swift +extension String { + public init?(validatingAsUTF8 codeUnits: some Sequence) +} +``` + +Reviewers and the Language Steering Group believed that this initializer does not carry its weight, and that the discoverability issues it sought to alleviate would best be solved by improved tooling. + +#### Have `String.init?(validating: some Sequence)` take a parameter typed as `some Sequence`, or as a specific `Collection` of `CChar` + +Defining this validating initializer in terms of `some Sequence` would produce a compile-time ambiguity on platforms where `CChar` is typealiased to `UInt8` rather than `Int8`. The reviewed proposal suggested defining it in terms of `UnsafeBufferPointer`, since this parameter type would avoid such a compile-time ambiguity. The actual root of the problem is that `CChar` is a typealias instead of a separate type. Given this, discussions during the review period and by the Language Steering Group led to this initializer to be re-defined using `some Sequence`. This solves the `CChar`-vs-`UInt8` interoperability issue at source-code level, and preserves as much flexibility as possible without ambiguities. + +## Future directions + +#### Throw an error containing information about a validation failure + +When decoding a byte stream, obtaining the details of a validation failure would be useful in order to diagnose issues. We would like to provide this functionality, but the current input validation functionality is not well-suited for it. This is left as a future improvement. + +#### Improve input-repairing initialization + +There is only one initializer in the standard library for input-repairing initialization, and it suffers from a discoverability issue. We can add a more discoverable version specifically for the UTF-8 encoding, similarly to one of the additions proposed here. + +#### Add normalization options + +It is often desirable to normalize strings, but the standard library does not expose public API for doing so. We could add initializers that perform normalization, as well as mutating functions that perform normalization. + +#### Other + +- Add a (non-failable) initializer to create a `String` from `some Sequence`. +- Add API devoted to input validation specifically. + +## Acknowledgements + +Thanks to Michael Ilseman, Tina Liu and Quinn Quinn for discussions about input validation issues. + +[SE-0027](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0027-string-from-code-units.md) by [Zachary Waldowski](https://github.com/zwaldowski) was reviewed in February 2016, covering similar ground. It was rejected at the time because the design of `String` had not been finalized. The name `String.init(validatingCString:)` was suggested as part of SE-0027. Lily Ballard later [pitched](https://forums.swift.org/t/21538) a renaming of `String.init(validatingUTF8:)`, citing consistency with other `String` API involving C strings. + diff --git a/proposals/0406-async-stream-backpressure.md b/proposals/0406-async-stream-backpressure.md new file mode 100644 index 0000000000..58817b3ede --- /dev/null +++ b/proposals/0406-async-stream-backpressure.md @@ -0,0 +1,806 @@ +# Backpressure support for AsyncStream + +* Proposal: [SE-0406](0406-async-stream-backpressure.md) +* Author: [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Returned for revision** +* Implementation: [apple/swift#66488](https://github.com/apple/swift/pull/66488) +* Review: ([pitch](https://forums.swift.org/t/pitch-new-apis-for-async-throwing-stream-with-backpressure-support/65449)) ([review](https://forums.swift.org/t/se-0406-backpressure-support-for-asyncstream/66771)) ([return for revision](https://forums.swift.org/t/returned-for-revision-se-0406-backpressure-support-for-asyncstream/67248)) + +## Introduction + +[SE-0314](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0314-async-stream.md) +introduced new `Async[Throwing]Stream` types which act as root asynchronous +sequences. These two types allow bridging from synchronous callbacks such as +delegates to an asynchronous sequence. This proposal adds a new way of +constructing asynchronous streams with the goal to bridge backpressured systems +into an asynchronous sequence. Furthermore, this proposal aims to clarify the +cancellation behaviour both when the consuming task is cancelled and when +the production side indicates termination. + +## Motivation + +After using the `AsyncSequence` protocol and the `Async[Throwing]Stream` types +extensively over the past years, we learned that there are a few important +behavioral details that any `AsyncSequence` implementation needs to support. +These behaviors are: + +1. Backpressure +2. Multi/single consumer support +3. Downstream consumer termination +4. Upstream producer termination + +In general, `AsyncSequence` implementations can be divided into two kinds: Root +asynchronous sequences that are the source of values such as +`Async[Throwing]Stream` and transformational asynchronous sequences such as +`AsyncMapSequence`. Most transformational asynchronous sequences implicitly +fulfill the above behaviors since they forward any demand to a base asynchronous +sequence that should implement the behaviors. On the other hand, root +asynchronous sequences need to make sure that all of the above behaviors are +correctly implemented. Let's look at the current behavior of +`Async[Throwing]Stream` to see if and how it achieves these behaviors. + +### Backpressure + +Root asynchronous sequences need to relay the backpressure to the producing +system. `Async[Throwing]Stream` aims to support backpressure by providing a +configurable buffer and returning +`Async[Throwing]Stream.Continuation.YieldResult` which contains the current +buffer depth from the `yield()` method. However, only providing the current +buffer depth on `yield()` is not enough to bridge a backpressured system into +an asynchronous sequence since this can only be used as a "stop" signal but we +are missing a signal to indicate resuming the production. The only viable +backpressure strategy that can be implemented with the current API is a timed +backoff where we stop producing for some period of time and then speculatively +produce again. This is a very inefficient pattern that produces high latencies +and inefficient use of resources. + +### Multi/single consumer support + +The `AsyncSequence` protocol itself makes no assumptions about whether the +implementation supports multiple consumers or not. This allows the creation of +unicast and multicast asynchronous sequences. The difference between a unicast +and multicast asynchronous sequence is if they allow multiple iterators to be +created. `AsyncStream` does support the creation of multiple iterators and it +does handle multiple consumers correctly. On the other hand, +`AsyncThrowingStream` also supports multiple iterators but does `fatalError` +when more than one iterator has to suspend. The original proposal states: + +> As with any sequence, iterating over an AsyncStream multiple times, or +creating multiple iterators and iterating over them separately, may produce an +unexpected series of values. + +While that statement leaves room for any behavior we learned that a clear distinction +of behavior for root asynchronous sequences is beneficial; especially, when it comes to +how transformation algorithms are applied on top. + +### Downstream consumer termination + +Downstream consumer termination allows the producer to notify the consumer that +no more values are going to be produced. `Async[Throwing]Stream` does support +this by calling the `finish()` or `finish(throwing:)` methods of the +`Async[Throwing]Stream.Continuation`. However, `Async[Throwing]Stream` does not +handle the case that the `Continuation` may be `deinit`ed before one of the +finish methods is called. This currently leads to async streams that never +terminate. The behavior could be changed but it could result in semantically +breaking code. + +### Upstream producer termination + +Upstream producer termination is the inverse of downstream consumer termination +where the producer is notified once the consumption has terminated. Currently, +`Async[Throwing]Stream` does expose the `onTermination` property on the +`Continuation`. The `onTermination` closure is invoked once the consumer has +terminated. The consumer can terminate in four separate cases: + +1. The asynchronous sequence was `deinit`ed and no iterator was created +2. The iterator was `deinit`ed and the asynchronous sequence is unicast +3. The consuming task is canceled +4. The asynchronous sequence returned `nil` or threw + +`Async[Throwing]Stream` currently invokes `onTermination` in all cases; however, +since `Async[Throwing]Stream` supports multiple consumers (as discussed in the +`Multi/single consumer support` section), a single consumer task being canceled +leads to the termination of all consumers. This is not expected from multicast +asynchronous sequences in general. + +## Proposed solution + +The above motivation lays out the expected behaviors from a root asynchronous +sequence and compares them to the behaviors of `Async[Throwing]Stream`. These +are the behaviors where `Async[Throwing]Stream` diverges from the expectations. + +- Backpressure: Doesn't expose a "resumption" signal to the producer +- Multi/single consumer: + - Divergent implementation between throwing and non-throwing variant + - Supports multiple consumers even though proposal positions it as a unicast + asynchronous sequence +- Consumer termination: Doesn't handle the `Continuation` being `deinit`ed +- Producer termination: Happens on first consumer termination + +This section proposes new APIs for `Async[Throwing]Stream` that implement all of +the above-mentioned behaviors. + +### Creating an AsyncStream with backpressure support + +You can create an `Async[Throwing]Stream` instance using the new `makeStream(of: +backpressureStrategy:)` method. This method returns you the stream and the +source. The source can be used to write new values to the asynchronous stream. +The new API specifically provides a multi-producer/single-consumer pattern. + +```swift +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) +``` + +The new proposed APIs offer three different ways to bridge a backpressured +system. The foundation is the multi-step synchronous interface. Below is an +example of how it can be used: + +```swift +do { + let writeResult = try source.write(contentsOf: sequence) + + switch writeResult { + case .produceMore: + // Trigger more production + + case .enqueueCallback(let callbackToken): + source.enqueueCallback(token: callbackToken, onProduceMore: { result in + switch result { + case .success: + // Trigger more production + case .failure(let error): + // Terminate the underlying producer + } + }) + } +} catch { + // `write(contentsOf:)` throws if the asynchronous stream already terminated +} +``` + +The above API offers the most control and highest performance when bridging a +synchronous producer to an asynchronous sequence. First, you have to write +values using the `write(contentsOf:)` which returns a `WriteResult`. The result +either indicates that more values should be produced or that a callback should +be enqueued by calling the `enqueueCallback(callbackToken: onProduceMore:)` +method. This callback is invoked once the backpressure strategy decided that +more values should be produced. This API aims to offer the most flexibility with +the greatest performance. The callback only has to be allocated in the case +where the producer needs to be suspended. + +Additionally, the above API is the building block for some higher-level and +easier-to-use APIs to write values to the asynchronous stream. Below is an +example of the two higher-level APIs. + +```swift +// Writing new values and providing a callback when to produce more +try source.write(contentsOf: sequence, onProduceMore: { result in + switch result { + case .success: + // Trigger more production + case .failure(let error): + // Terminate the underlying producer + } +}) + +// This method suspends until more values should be produced +try await source.write(contentsOf: sequence) +``` + +With the above APIs, we should be able to effectively bridge any system into an +asynchronous stream regardless if the system is callback-based, blocking or +asynchronous. + +### Downstream consumer termination + +> When reading the next two examples around termination behaviour keep in mind +that the newly proposed APIs are providing a strict unicast asynchronous sequence. + +Calling `finish()` terminates the downstream consumer. Below is an example of +this: + +```swift +// Termination through calling finish +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) + +_ = try await source.write(1) +source.finish() + +for await element in stream { + print(element) +} +print("Finished") + +// Prints +// 1 +// Finished +``` + +The other way to terminate the consumer is by deiniting the source. This has the +same effect as calling `finish()` and makes sure that no consumer is stuck +indefinitely. + +```swift +// Termination through deiniting the source +let (stream, _) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) + +for await element in stream { + print(element) +} +print("Finished") + +// Prints +// Finished +``` + +Trying to write more elements after the source has been finish will result in an +error thrown from the write methods. + +### Upstream producer termination + +The producer will get notified about termination through the `onTerminate` +callback. Termination of the producer happens in the following scenarios: + +```swift +// Termination through task cancellation +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) + +let task = Task { + for await element in stream { + + } +} +task.cancel() +``` + +```swift +// Termination through deiniting the sequence +let (_, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) +``` + +```swift +// Termination through deiniting the iterator +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) +_ = stream.makeAsyncIterator() +``` + +```swift +// Termination through calling finish +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) + +_ = try source.write(1) +source.finish() + +for await element in stream {} + +// onTerminate will be called after all elements have been consumed +``` + +Similar to the downstream consumer termination, trying to write more elements after the +producer has been terminated will result in an error thrown from the write methods. + +## Detailed design + +All new APIs on `AsyncStream` and `AsyncThrowingStream` are as follows: + +```swift +/// Error that is thrown from the various `write` methods of the +/// ``AsyncStream.Source`` and ``AsyncThrowingStream.Source``. +/// +/// This error is thrown when the asynchronous stream is already finished when +/// trying to write new elements. +public struct AsyncStreamAlreadyFinishedError: Error {} + +extension AsyncStream { + /// A mechanism to interface between producer code and an asynchronous stream. + /// + /// Use this source to provide elements to the stream by calling one of the `write` methods, then terminate the stream normally + /// by calling the `finish()` method. + public struct Source: Sendable { + /// A strategy that handles the backpressure of the asynchronous stream. + public struct BackpressureStrategy: Sendable { + /// When the high watermark is reached producers will be suspended. All producers will be resumed again once + /// the low watermark is reached. + public static func watermark(low: Int, high: Int) -> BackpressureStrategy {} + } + + /// A type that indicates the result of writing elements to the source. + @frozen + public enum WriteResult: Sendable { + /// A token that is returned when the asynchronous stream's backpressure strategy indicated that production should + /// be suspended. Use this token to enqueue a callback by calling the ``enqueueCallback(_:)`` method. + public struct CallbackToken: Sendable {} + + /// Indicates that more elements should be produced and written to the source. + case produceMore + + /// Indicates that a callback should be enqueued. + /// + /// The associated token should be passed to the ``enqueueCallback(_:)`` method. + case enqueueCallback(CallbackToken) + } + + /// A callback to invoke when the stream finished. + /// + /// The stream finishes and calls this closure in the following cases: + /// - No iterator was created and the sequence was deinited + /// - An iterator was created and deinited + /// - After ``finish(throwing:)`` was called and all elements have been consumed + /// - The consuming task got cancelled + public var onTermination: (@Sendable () -> Void)? + + /// Writes new elements to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// - Parameter sequence: The elements to write to the asynchronous stream. + /// - Returns: The result that indicates if more elements should be produced at this time. + public func write(contentsOf sequence: S) throws -> WriteResult where Element == S.Element, S: Sequence {} + + /// Write the element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// - Parameter element: The element to write to the asynchronous stream. + /// - Returns: The result that indicates if more elements should be produced at this time. + public func write(_ element: Element) throws -> WriteResult {} + + /// Enqueues a callback that will be invoked once more elements should be produced. + /// + /// Call this method after ``write(contentsOf:)`` or ``write(_:)`` returned ``WriteResult/enqueueCallback(_:)``. + /// + /// - Important: Enqueueing the same token multiple times is not allowed. + /// + /// - Parameters: + /// - token: The callback token. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. + public func enqueueCallback(token: WriteResult.CallbackToken, onProduceMore: @escaping @Sendable (Result) -> Void) {} + + /// Cancel an enqueued callback. + /// + /// Call this method to cancel a callback enqueued by the ``enqueueCallback(callbackToken:onProduceMore:)`` method. + /// + /// - Note: This method supports being called before ``enqueueCallback(callbackToken:onProduceMore:)`` is called and + /// will mark the passed `token` as cancelled. + /// + /// - Parameter token: The callback token. + public func cancelCallback(token: WriteResult.CallbackToken) {} + + /// Write new elements to the asynchronous stream and provide a callback which will be invoked once more elements should be produced. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then `onProduceMore` will be invoked with + /// a `Result.failure`. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. This callback might be + /// invoked during the call to ``write(contentsOf:onProduceMore:)``. + public func write(contentsOf sequence: S, onProduceMore: @escaping @Sendable (Result) -> Void) where Element == S.Element, S: Sequence {} + + /// Writes the element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then `onProduceMore` will be invoked with + /// a `Result.failure`. + /// + /// - Parameters: + /// - sequence: The element to write to the asynchronous stream. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. This callback might be + /// invoked during the call to ``write(_:onProduceMore:)``. + public func write(_ element: Element, onProduceMore: @escaping @Sendable (Result) -> Void) {} + + /// Write new elements to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// This method returns once more elements should be produced. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + public func write(contentsOf sequence: S) async throws where Element == S.Element, S: Sequence {} + + /// Write new element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// This method returns once more elements should be produced. + /// + /// - Parameters: + /// - sequence: The element to write to the asynchronous stream. + public func write(_ element: Element) async throws {} + + /// Write the elements of the asynchronous sequence to the asynchronous stream. + /// + /// This method returns once the provided asynchronous sequence or the asynchronous stream finished. + /// + /// - Important: This method does not finish the source if consuming the upstream sequence terminated. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + public func write(contentsOf sequence: S) async throws where Element == S.Element, S: AsyncSequence {} + + /// Indicates that the production terminated. + /// + /// After all buffered elements are consumed the next iteration point will return `nil`. + /// + /// Calling this function more than once has no effect. After calling finish, the stream enters a terminal state and doesn't accept + /// new elements. + public func finish() {} + } + + /// Initializes a new ``AsyncStream`` and an ``AsyncStream/Source``. + /// + /// - Parameters: + /// - elementType: The element type of the stream. + /// - backpressureStrategy: The backpressure strategy that the stream should use. + /// - Returns: A tuple containing the stream and its source. The source should be passed to the + /// producer while the stream should be passed to the consumer. + public static func makeStream( + of elementType: Element.Type = Element.self, + backpressureStrategy: Source.BackpressureStrategy + ) -> (`Self`, Source) {} +} + +extension AsyncThrowingStream { + /// A mechanism to interface between producer code and an asynchronous stream. + /// + /// Use this source to provide elements to the stream by calling one of the `write` methods, then terminate the stream normally + /// by calling the `finish()` method. You can also use the source's `finish(throwing:)` method to terminate the stream by + /// throwing an error. + public struct Source: Sendable { + /// A strategy that handles the backpressure of the asynchronous stream. + public struct BackpressureStrategy: Sendable { + /// When the high watermark is reached, producers will be suspended. All producers will be resumed again once + /// the low watermark is reached. + public static func watermark(low: Int, high: Int) -> BackpressureStrategy {} + } + + /// A type that indicates the result of writing elements to the source. + @frozen + public enum WriteResult: Sendable { + /// A token that is returned when the asynchronous stream's backpressure strategy indicated that production should + /// be suspended. Use this token to enqueue a callback by calling the ``enqueueCallback(_:)`` method. + public struct CallbackToken: Sendable {} + + /// Indicates that more elements should be produced and written to the source. + case produceMore + + /// Indicates that a callback should be enqueued. + /// + /// The associated token should be passed to the ``enqueueCallback(_:)`` method. + case enqueueCallback(CallbackToken) + } + + /// A callback to invoke when the stream finished. + /// + /// The stream finishes and calls this closure in the following cases: + /// - No iterator was created and the sequence was deinited + /// - An iterator was created and deinited + /// - After ``finish(throwing:)`` was called and all elements have been consumed + /// - The consuming task got cancelled + public var onTermination: (@Sendable () -> Void)? {} + + /// Writes new elements to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// - Parameter sequence: The elements to write to the asynchronous stream. + /// - Returns: The result that indicates if more elements should be produced at this time. + public func write(contentsOf sequence: S) throws -> WriteResult where Element == S.Element, S: Sequence {} + + /// Write the element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// - Parameter element: The element to write to the asynchronous stream. + /// - Returns: The result that indicates if more elements should be produced at this time. + public func write(_ element: Element) throws -> WriteResult {} + + /// Enqueues a callback that will be invoked once more elements should be produced. + /// + /// Call this method after ``write(contentsOf:)`` or ``write(_:)`` returned ``WriteResult/enqueueCallback(_:)``. + /// + /// - Important: Enqueueing the same token multiple times is not allowed. + /// + /// - Parameters: + /// - token: The callback token. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. + public func enqueueCallback(token: WriteResult.CallbackToken, onProduceMore: @escaping @Sendable (Result) -> Void) {} + + /// Cancel an enqueued callback. + /// + /// Call this method to cancel a callback enqueued by the ``enqueueCallback(callbackToken:onProduceMore:)`` method. + /// + /// - Note: This method supports being called before ``enqueueCallback(callbackToken:onProduceMore:)`` is called and + /// will mark the passed `token` as cancelled. + /// + /// - Parameter token: The callback token. + public func cancelCallback(token: WriteResult.CallbackToken) {} + + /// Write new elements to the asynchronous stream and provide a callback which will be invoked once more elements should be produced. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then `onProduceMore` will be invoked with + /// a `Result.failure`. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. This callback might be + /// invoked during the call to ``write(contentsOf:onProduceMore:)``. + public func write(contentsOf sequence: S, onProduceMore: @escaping @Sendable (Result) -> Void) where Element == S.Element, S: Sequence {} + + /// Writes the element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then `onProduceMore` will be invoked with + /// a `Result.failure`. + /// + /// - Parameters: + /// - sequence: The element to write to the asynchronous stream. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. This callback might be + /// invoked during the call to ``write(_:onProduceMore:)``. + public func write(_ element: Element, onProduceMore: @escaping @Sendable (Result) -> Void) {} + + /// Write new elements to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// This method returns once more elements should be produced. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + public func write(contentsOf sequence: S) async throws where Element == S.Element, S: Sequence {} + + /// Write new element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// This method returns once more elements should be produced. + /// + /// - Parameters: + /// - sequence: The element to write to the asynchronous stream. + public func write(_ element: Element) async throws {} + + /// Write the elements of the asynchronous sequence to the asynchronous stream. + /// + /// This method returns once the provided asynchronous sequence or the the asynchronous stream finished. + /// + /// - Important: This method does not finish the source if consuming the upstream sequence terminated. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + public func write(contentsOf sequence: S) async throws where Element == S.Element, S: AsyncSequence {} + + /// Indicates that the production terminated. + /// + /// After all buffered elements are consumed the next iteration point will return `nil` or throw an error. + /// + /// Calling this function more than once has no effect. After calling finish, the stream enters a terminal state and doesn't accept + /// new elements. + /// + /// - Parameters: + /// - error: The error to throw, or `nil`, to finish normally. + public func finish(throwing error: Failure?) {} + } + + /// Initializes a new ``AsyncThrowingStream`` and an ``AsyncThrowingStream/Source``. + /// + /// - Parameters: + /// - elementType: The element type of the stream. + /// - failureType: The failure type of the stream. + /// - backpressureStrategy: The backpressure strategy that the stream should use. + /// - Returns: A tuple containing the stream and its source. The source should be passed to the + /// producer while the stream should be passed to the consumer. + public static func makeStream( + of elementType: Element.Type = Element.self, + throwing failureType: Failure.Type = Failure.self, + backpressureStrategy: Source.BackpressureStrategy + ) -> (`Self`, Source) where Failure == Error {} +} +``` + +## Comparison to other root asynchronous sequences + +### swift-async-algorithm: AsyncChannel + +The `AsyncChannel` is a multi-consumer/multi-producer root asynchronous sequence +which can be used to communicate between two tasks. It only offers asynchronous +production APIs and has no internal buffer. This means that any producer will be +suspended until its value has been consumed. `AsyncChannel` can handle multiple +consumers and resumes them in FIFO order. + +### swift-nio: NIOAsyncSequenceProducer + +The NIO team have created their own root asynchronous sequence with the goal to +provide a high performance sequence that can be used to bridge a NIO `Channel` +inbound stream into Concurrency. The `NIOAsyncSequenceProducer` is a highly +generic and fully inlinable type and quite unwieldy to use. This proposal is +heavily inspired by the learnings from this type but tries to create a more +flexible and easier to use API that fits into the standard library. + +## Source compatibility + +This change is additive and does not affect source compatibility. + +## ABI compatibility + +This change is additive and does not affect ABI compatibility. All new methods +are non-inlineable leaving us flexibility to change the implementation in the +future. + +## Future directions + +### Adaptive backpressure strategy + +The high/low watermark strategy is common in networking code; however, there are +other strategies such as an adaptive strategy that we could offer in the future. +An adaptive strategy regulates the backpressure based on the rate of +consumption and production. With the proposed new APIs we can easily add further +strategies. + +### Element size dependent strategy + +When the stream's element is a collection type then the proposed high/low +watermark backpressure strategy might lead to unexpected results since each +element can vary in actual memory size. In the future, we could provide a new +backpressure strategy that supports inspecting the size of the collection. + +### Deprecate `Async[Throwing]Stream.Continuation` + +In the future, we could deprecate the current continuation based APIs since the +new proposed APIs are also capable of bridging non-backpressured producers by +just discarding the `WriteResult`. The only use-case that the new APIs do not +cover is the _anycast_ behaviour of the current `AsyncStream` where one can +create multiple iterators to the stream as long as no two iterators are +consuming the stream at the same time. This can be solved via additional +algorithms such as `broadcast` in the `swift-async-algorithms` package. + +To give developers more time to adopt the new APIs the deprecation of the +current APIs should be deferred to a future version. Especially since those new +APIs are not backdeployed like the current Concurrency runtime. + +### Introduce a `Writer` and an `AsyncWriter` protocol + +The newly introduced `Source` type offers a bunch of different write methods. We +have seen similar types used in other places such as file abstraction or +networking APIs. We could introduce a new `Writer` and `AsyncWriter` protocol in +the future to enable writing generic algorithms on top of writers. The `Source` +type could then conform to these new protocols. + +## Alternatives considered + +### Providing an `Async[Throwing]Stream.Continuation.onConsume` + +We could add a new closure property to the `Async[Throwing]Stream.Continuation` +which is invoked once an element has been consumed to implement a backpressure +strategy; however, this requires the usage of a synchronization mechanism since +the consumption and production often happen on separate threads. The +added complexity and performance impact led to avoiding this approach. + +### Provide a getter for the current buffer depth + +We could provide a getter for the current buffer depth on the +`Async[Throwing]Stream.Continuation`. This could be used to query the buffer +depth at an arbitrary time; however, it wouldn't allow us to implement +backpressure strategies such as high/low watermarks without continuously asking +what the buffer depth is. That would result in a very inefficient +implementation. + +### Extending `Async[Throwing]Stream.Continuation` + +Extending the current APIs to support all expected behaviors is problematic +since it would change the semantics and might lead to currently working code +misbehaving. Furthermore, extending the current APIs to support backpressure +turns out to be problematic without compromising performance or usability. + +### Introducing a new type + +We could introduce a new type such as `AsyncBackpressured[Throwing]Stream`; +however, one of the original intentions of `Async[Throwing]Stream` was to be +able to bridge backpressured systems. Furthermore, `Async[Throwing]Stream` is +the best name. Therefore, this proposal decided to provide new interfaces to +`Async[Throwing]Stream`. + +### Stick with the current `Continuation` and `yield` naming + +The proposal decided against sticking to the current names since the existing +names caused confusion to them being used in multiple places. Continuation was +both used by the `AsyncStream` but also by Swift Concurrency via +`CheckedContinuation` and `UnsafeContinuation`. Similarly, yield was used by +both `AsyncStream.Continuation.yield()`, `Task.yield()` and the `yield` keyword. +Having different names for these different concepts makes it easier to explain +their usage. The currently proposed `write` names were chosen to align with the +future direction of adding an `AsyncWriter` protocol. `Source` is a common name +in flow based systems such as Akka. Other names that were considered: + +- `enqueue` +- `send` + +### Provide the `onTermination` callback to the factory method + +During development of the new APIs, I first tried to provide the `onTermination` +callback in the `makeStream` method. However, that showed significant usability +problems in scenarios where one wants to store the source in a type and +reference `self` in the `onTermination` closure at the same time; hence, I kept +the current pattern of setting the `onTermination` closure on the source. + +### Provide a `onConsumerCancellation` callback + +During the pitch phase, it was raised that we should provide a +`onConsumerCancellation` callback which gets invoked once the asynchronous +stream notices that the consuming task got cancelled. This callback could be +used to customize how cancellation is handled by the stream e.g. one could +imagine writing a few more elements to the stream before finishing it. Right now +the stream immediately returns `nil` or throws a `CancellationError` when it +notices cancellation. This proposal decided to not provide this customization +because it opens up the possibility that asynchronous streams are not terminating +when implemented incorrectly. Additionally, asynchronous sequences are not the +only place where task cancellation leads to an immediate error being thrown i.e. +`Task.sleep()` does the same. Hence, the value of the asynchronous not +terminating immediately brings little value when the next call in the iterating +task might throw. However, the implementation is flexible enough to add this in +the future and we can just default it to the current behaviour. + +### Create a custom type for the `Result` of the `onProduceMore` callback + +The `onProduceMore` callback takes a `Result` which is used to +indicate if the producer should produce more or if the asynchronous stream +finished. We could introduce a new type for this but the proposal decided +against it since it effectively is a result type. + +### Use an initializer instead of factory methods + +Instead of providing a `makeStream` factory method we could use an initializer +approach that takes a closure which gets the `Source` passed into. A similar API +has been offered with the `Continuation` based approach and +[SE-0388](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0388-async-stream-factory.md) +introduced new factory methods to solve some of the usability ergonomics with +the initializer based APIs. + +## Acknowledgements + +- [Johannes Weiss](https://github.com/weissi) - For making me aware how +important this problem is and providing great ideas on how to shape the API. +- [Philippe Hausler](https://github.com/phausler) - For helping me designing the +APIs and continuously providing feedback +- [George Barnett](https://github.com/glbrntt) - For providing extensive code +reviews and testing the implementation. diff --git a/proposals/0407-member-macro-conformances.md b/proposals/0407-member-macro-conformances.md new file mode 100644 index 0000000000..6e31c39d1f --- /dev/null +++ b/proposals/0407-member-macro-conformances.md @@ -0,0 +1,116 @@ +# Member Macro Conformances + +* Proposal: [SE-0407](0407-member-macro-conformances.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9.2)** +* Vision: [Macros](https://github.com/swiftlang/swift-evolution/blob/main/visions/macros.md) +* Implementation: [apple/swift#67758](https://github.com/apple/swift/pull/67758) +* Review: ([pitch](https://forums.swift.org/t/pitch-member-macros-that-know-what-conformances-are-missing/66590)) ([review](https://forums.swift.org/t/se-0407-member-macro-conformances/66951)) ([acceptance](https://forums.swift.org/t/accepted-se-0407-member-macro-conformances/67345)) + +## Introduction + +The move from conformance macros to extension macros in [SE-0402](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0402-extension-macros.md) included the ability for extension macros to learn about which protocols the type already conformed to (e.g., because a superclass conformed or an explicit conformance was stated somewhere), so that the macro could avoid adding declarations and conformances that aren't needed. It also meant that any new declarations added are part of an extension---not the original type definition---which is generally beneficial, because it means that (e.g.) a new initializer doesn't suppress the memberwise initializer. It's also usually considered good form to split protocol conformances out into their own extensions. + +However, there are some times when the member used for the conformance really needs to be part of the original type definition. For example: + +- An initializer in a non-final class needs to be a `required init` to satisfy a protocol requirement. +- An overridable member of a non-final class. +- A stored property or case can only be in the primary type definition. + +For these cases, a member macro can produce the declarations. However, member macros aren't provided with any information about which protocol conformances they should provide members for, so a macro might erroneously try to add conforming members to a type that already conforms to the protocol (e.g., through a superclass). This can make certain macros---such as macros that implement the `Encodable` or `Decodable` protocols---unimplemented. + +## Proposed solution + +To make it possible for a member macro to provide the right members for a conformance, we propose to extend member macros with the same ability that extension macros have to reason about conformances. Specifically: + +* The `attached` attribute specifying a `member` role gains the ability to specify the set of protocol conformances it is interested in, the same way an `extension` macro specifies the conformances it can provide. +* The `expansion` operation for a `MemberMacro` -conforming implementation receives the set of protocols that were stated (as above) and which the type does not already conform to. + +This information allows a macro to reason about which members it should produce to satisfy conformances. Member macros that are interested in conformances are often going to also be extension macros, which work along with the member macro to provide complete conformance information. + +As an example, consider a `Codable` macro that provides the `init(from:)` and `encode(to:)` operations required by the `Decodable` and `Encodable` protocols, respectively. Such a macro could be defined as follows: + +```swift +@attached(member, conformances: Decodable, Encodable, names: named(init(from:), encode(to:))) +@attached(extension, conformances: Decodable, Encodable, names: named(init(from:), encode(to:))) +macro Codable() = #externalMacro(module: "MyMacros", type: "CodableMacro") +``` + +This macro has several important decisions to make about where and how to generate `init(from:)` and `encode(to:)`: + +* For a struct, enum, actor, or final class, `init(from:)` and `encode(to:)` should be emitted into an extension (via the member role) along with the conformance. This is both good style and, for structs, ensures that the initializer doesn't inhibit the memberwise initializer. +* For a non-final class, `init(from:)` and `encode(to:)` should be emitted into the main class definition (via the member role) so that they can be overridden by subclasses. +* For a class that inherits `Encodable` or `Decodable` conformances from a superclass, the implementations of `init(from:)` and `encode(to:)` need to call the superclass's initializer and method, respectively, to decode/encode the entire class hierarchy. + +Given existing syntactic information about the type (including the presence or absence of `final`), and providing both the member and extension roles with information about which conformances the type needs (as proposed here), all of the above decisions can be made in the macro implementation, allowing a flexible implementation of a `Codable` macro that accounts for all manner of types. + +## Detailed design + +The specification of the `conformances` argument for the `@attached(member, ...)` attribute matches that of the corresponding argument for extension macros documented in [SE-0402](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0402-extension-macros.md). + +For macro implementations, the `expansion` requirement in the `MemberMacro` protocol is augmented with a `conformingTo:` argument that receives the same set of protocols as for extension macros. The `MemberMacro` protocol is now defined as follows: + +```swift +protocol MemberMacro: AttachedMacro { + /// Expand an attached declaration macro to produce a set of members. + /// + /// - Parameters: + /// - node: The custom attribute describing the attached macro. + /// - declaration: The declaration the macro attribute is attached to. + /// - missingConformancesTo: The set of protocols that were declared + /// in the set of conformances for the macro and to which the declaration + /// does not explicitly conform. The member macro itself cannot declare + /// conformances to these protocols (only an extension macro can do that), + /// but can provide supporting declarations, such as a required + /// initializer or stored property, that cannot be written in an + /// extension. + /// - context: The context in which to perform the macro expansion. + /// + /// - Returns: the set of member declarations introduced by this macro, which + /// are nested inside the `attachedTo` declaration. + static func expansion( + of node: AttributeSyntax, + providingMembersOf declaration: some DeclGroupSyntax, + conformingTo protocols: [TypeSyntax], + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] +} +``` + +Note that member macro definitions don't provide the conformances themselves; that is still part of the extension macro role. + +## Source compatibility + +This proposal uses the existing syntactic space for the `@attached` attribute and is a pure extension; it does not have any source compatibility impact. + +## ABI compatibility + +As a macro feature, this proposal does not affect ABI in any way. + +## Implications on adoption + +This feature can be freely adopted in source code with no deployment constraints or affecting source or ABI compatibility. Uses of any macro that employs this feature can also be removed from the source code by expanding the macro in place. + +## Alternatives considered + +### Extensions that affect the primary type definition + +A completely different approach to the stated problem would be to introduce a form of extension that adds members to the type as-if they were written directly in the type definition. For example: + +```swift +class MyClass { ... } + +@implementation extension MyClass: Codable { + required init(from decoder: Decoder) throws { ... } + func encode(to coder: Coder) throws { ... } +} +``` + +The members of `@implementation` extensions would follow the same rules as members in the main type definition. For example, stored properties could be defined in the `@implementation` extension, as could `required` initializers, and any overridable methods, properties, or subscripts. The deinitializer and enum cases could also be defined in `@implementation` extensions if that were deemed useful. + +There would be some limitations on `@implementation` extensions: they could only be defined in the same source file as the original type, and these extensions might not be permitted to have any additional generic constraints. Protocols don't have implementations per se, and therefore might not support implementation extensions. + +Given the presence of `@implementation` extensions, the extension to member macros in this proposal would no longer be needed, because one could achieve the desired effect using an extension macro that produces an `@implementation` extension for cases where it needs to extend the implementation itself. + +The primary drawback to this notion of implementation extensions is that it would no longer be possible to look at the primary definition of a type to find its full "shape": its stored properties, designated/required initializers, overridable methods, and so on. Instead, that information could be scattered amongst the original type definition and any implementation extensions, requiring readers to stitch together a view of the whole type. This would have a particularly negative effect on macros that want to reason about the shape of the type, because macros only see a single entity (such as a class or struct definition) and not extensions to that entity. The `Codable` macro discussed in this proposal, for example, would not be able to encode or decode stored properties that are written in an implementation extension, and the [`Observable`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0395-observability.md) macro would silently fail to observe any properties written in an implementation extension. By trying to use implementation extensions to address the shortcoming of macros described in this proposal, we would end up creating a larger problem for those same macros. diff --git a/proposals/0408-pack-iteration.md b/proposals/0408-pack-iteration.md new file mode 100644 index 0000000000..b625affbc9 --- /dev/null +++ b/proposals/0408-pack-iteration.md @@ -0,0 +1,203 @@ +# Pack Iteration + +* Proposal: [SE-0408](0408-pack-iteration.md) +* Authors: [Sima Nerush](https://github.com/simanerush), [Holly Borla](https://github.com/hborla) +* Review Manager: [Doug Gregor](https://github.com/DougGregor/) +* Status: **Implemented (Swift 6.0)** +* Implementation: [apple/swift#67594](https://github.com/apple/swift/pull/67594) +* Review: ([pitch](https://forums.swift.org/t/pitch-enable-pack-iteration/66168), [review](https://forums.swift.org/t/review-se-0408-pack-iteration/67152), [acceptance](https://forums.swift.org/t/accepted-se-0408-pack-iteration/67598)) + +## Introduction + +Building upon the Value and Type Parameter Packs proposal [SE-0393](https://forums.swift.org/t/se-0393-value-and-type-parameter-packs/63859), this proposal enables iterating over each element in a value pack and bind each value to a local variable using a `for-in` syntax. + + +## Motivation + +Currently, it is possible to express list operations on value packs using pack expansion expressions. This approach requires putting code involving statements into a function or closure. For example, limiting repetition patterns to expressions does not allow for short-circuiting with `break` or `continue` statements, so the pattern expression will always be evaluated once for every element in the pack. The only way to stop evaluation would be to mark the function/closure containing the pattern expression throwing, and catch the error in a do/catch block to return, which is unnatural for Swift users. + +The following implementation of `==` over tuples of arbitrary length demonstrates these workarounds: + +```swift +struct NotEqual: Error {} + +func == (lhs: (repeat each Element), rhs: (repeat each Element)) -> Bool { + // Local throwing function for operating over each element of a pack expansion. + func isEqual(_ left: T, _ right: T) throws { + if left == right { + return + } + + throw NotEqual() + } + + // Do-catch statement for short-circuiting as soon as two tuple elements are not equal. + do { + repeat try isEqual(each lhs, each rhs) + } catch { + return false + } + + return true +} +``` + +Here, the programmer can only return `false` when the `NotEqual` error was thrown. The `isEqual` function performs the comparison and throws if the sides are not equal. + +## Proposed Solution + +We propose allowing iteration over value packs using `for-in` loops. With the adoption of pack iteration, the implementation of the standard library methods like `==` operator for tuples of any number of elements will become straightforward. Instead of throwing a `NotEqual` error, the function can simply iterate over each respective element of the tuple and `return false` in the body of the loop if the elements are not equal: + +```swift +func == (lhs: (repeat each Element), rhs: (repeat each Element)) -> Bool { + + for (left, right) in repeat (each lhs, each rhs) { + guard left == right else { return false } + } + return true +} +``` + +The above code iterates pairwise over two tuples `lhs` and `rhs` using a `for-in` loop syntax. At each iteration, the pack elements of `lhs` and `rhs` are bound to the local variables `left` and `right`, respectively. Because `lhs` and `rhs` have the same type, so do `left` and `right`, and the `Equatable` requirement allows comparing `left` and `right` with `==`. + +## Detailed Design + +In addition to expressions that conform to `Sequence`, the source of a `for-in` loop can be a pack expansion expression. + +```swift +func iterate(over element: repeat each Element) { + for element in repeat each element { + + } +} +``` + +On the *i*th iteration, the type of `element` is the *i*th type parameter in the `Element` type parameter pack, and the value of `element` is the *i*th value parameter in the shadowed `element` value parameter pack. Conceptually, the type of `element` is the pattern type of `repeat each element` with each captured type parameter pack replaced with an implicit scalar type parameter with matching requirements. In this case, the pattern type is `each Element`, so the type of `element` is a scalar type parameter with no requirements. Let’s call the scalar type parameter `Element'`. For example, if `iterate` is called with `each Element` bound to the type pack `{Int, String, Bool}` the body of the `for-in` loop will first substitute `Element'` for `Int`, then `String`, then `Bool`. + +If the type parameter packs captured by the pack expansion pattern contain requirements, the scalar type parameter in the loop body will have the same requirements: + +```swift +struct Generic {} + +protocol P {} + +func iterate() { + for x in repeat Generic() { + // the type of 'x' is Generic + } +} +``` + +In the above code, the pattern type of the pack expansion is `Generic` where `each Element: P`, so the type of `x` is a scalar type parameter ` where Element': P`. + +Like regular `for-in` loops, `for-in` loops over pack expansions can pattern match over each element of the value pack: + +```swift +enum E { + case one(T) + case two +} + +func iterate(over element: repeat E) { + for case .one(let value) in repeat each element { + // 'value' has type Element' + } +} +``` +The pattern expression in the source of a `for-in repeat` loop is evaluated once at each iteration, instead of `n` times eagerly where `n` is the length of the packs captured by the pattern. If `p_i` is the pattern expression at the `i`th iteration and control flow exits the loop at iteration `i`, then `p_j` is not evaluated for `i < j < n`. For example: + +```swift +func printAndReturn(_ value: Value) -> Value { + print("Evaluated pack element value \(value)") + return value +} + +func iterate(_ t: repeat each T) { + var i = 0 + for value in repeat printAndReturn(each t) { + print("Evaluating loop iteration \(i)") + if i == 1 { + break + } else { + i += 1 + } + } + + print("Done iterating") +} + +iterate(1, "hello", true) +``` + +The above code has the following output + +``` +Evaluated pack element value 1 +Evaluating loop iteration 0 +Evaluated pack element value "hello" +Evaluating loop iteration 1 +Done iterating +``` + +## Source Compatibility + +There is no source compatibility impact, since this is an additive change. + + +## ABI Compatibility + +This proposal does not affect ABI, since its impact is only in expressions. + + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. + +## Alternatives Considered + +The only alternative to allowing pack iteration would be placing code with statements into functions/closures, however, that approach is unnatural and over-complicated. For example, this is a version of the `==` operator for tuples implementation mentioned earlier. + +## Future directions + +### Enabling `guard let` + +Another familiar pattern to Swift programmers is `guard let`. + +For example, consider the `zip` function from the standard library. `guard let` can be used for enabling this function to support any number of sequences, instead of just 2: + +```swift +public func zip(_ sequences: repeat each S) -> ZipSequence { + .init(sequences: repeat each sequences) +} + +public struct ZipSequence { + let sequences: (repeat each S) +} + +extension ZipSequence: Sequence { + public typealias Element = (repeat (each S).Element) + + public struct Iterator: IteratorProtocol { + var iterators: (repeat (each S).Iterator) + var reachedEnd = false + + public mutating func next() -> Element? { + if reachedEnd { + return nil + } + + // Using guard let for checking that the next element is not nil. + guard let element = repeat (each iterators).next() else { + return nil + } + + return (repeat each element) + } + } + + public func makeIterator() -> Iterator { + return Iterator(iter: (repeat (each sequences).makeIterator())) + } +} +``` + diff --git a/proposals/0409-access-level-on-imports.md b/proposals/0409-access-level-on-imports.md new file mode 100644 index 0000000000..4eec8c9df6 --- /dev/null +++ b/proposals/0409-access-level-on-imports.md @@ -0,0 +1,342 @@ +# Access-level modifiers on import declarations + +* Proposal: [SE-0409](0409-access-level-on-imports.md) +* Author: [Alexis Laferrière](https://github.com/xymus) +* Review Manager: [Frederick Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.0)** +* Implementation: On main and release/5.9 gated behind the frontend flag `-enable-experimental-feature AccessLevelOnImport` +* Upcoming Feature Flag: `InternalImportsByDefault` +* Review: ([pitch](https://forums.swift.org/t/pitch-access-level-on-import-statements/66657)) ([review](https://forums.swift.org/t/se-0409-access-level-modifiers-on-import-declarations/67290)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0409-access-level-modifiers-on-import-declarations/67666)) + +## Introduction + +Declaring the visibility of a dependency with an access-level modifier on import declarations enables enforcing which declarations can reference the imported module. +A dependency can be marked as being visible only to the source file, module, package, or to all clients. +This brings the familiar behavior of the access level of declarations to dependencies and imported declarations. +This feature can hide implementation details from clients and helps to manage dependency creep. + +## Motivation + +Good practices guide us to separate public and internal services to avoid having external clients rely on internal details. +Swift already offers access levels with their respective modifiers to declarations and enforcement during type-checking, +but there is currently no equivalent official feature for dependencies. + +The author of a library may have a different intent for each of the library dependencies; +some are expected to be known to the library clients while others are for implementation details internal to the package, module, or source file. +Without a way to enforce the intended access level of dependencies +it is easy to make a mistake and expose a dependency of the library to the library clients by referencing it from a public declaration even if it's intended to remain an implementation detail. + +All the library dependencies being visible to the library clients also requires the compiler to do more work than necessary. +The compiler must load all of the library dependencies when building a client of the library, +even the dependencies that are not actually required to build the client. + +## Proposed solution + +The core of this proposal consists of extending the current access level logic to support declaring the existing modifiers (excluding `open`) on import declarations and +applying the access level to the imported declarations. + +Here's an example case where a module `DatabaseAdapter` is an implementation detail of the local module. +We don't want to expose it to clients so we mark the import as `internal`. +The compiler then allows references to it from internal functions but diagnoses references from the signature of public functions. +```swift +internal import DatabaseAdapter + +internal func internalFunc() -> DatabaseAdapter.Entry {...} // Ok +public func publicFunc() -> DatabaseAdapter.Entry {...} // error: function cannot be declared public because its result uses an internal type +``` + +Additionally, this proposal uses the access level declared on each import declaration in all source files composing a module to determine when clients of a library need to load the library's dependencies or when they can be skipped. +To balance source compatibility and best practices, an import without explicit access level has an implicit access level of `public` in Swift 5 and Swift 6. It will be `internal` in a future language mode. +The attribute `@usableFromInline` on an import allows references from inlinable code. + +## Detailed design + +In this section we discuss the three main language changes of this proposal: +accept access-level modifiers on import declarations to declare the visibility of the imported module, +apply that information when type-checking the source file, +and determine when indirect clients can skip loading transitive dependencies. +We then cover other concerns addressed by this proposal: +the different default access levels of imports in different language modes, +and the relationship with other attributes on imports. + +### Declaring the access level of an imported module + +The access level is declared in front of the import declaration using some of the +modifiers used for a declaration: `public`, `package`, `internal`, `fileprivate`, and `private`. + +A public dependency can be referenced from any declaration and will be visible to all clients. +It is declared with the `public` modifier. + +```swift +public import PublicDependency +``` + +A dependency visible only to the modules of the same package is declared with the `package` modifier. +Only the signature of `package`, `internal`, `fileprivate` and `private` declarations can reference the imported module. + +```swift +package import PackageDependency +``` + +A dependency internal to the module is declared with the `internal` modifier. +Only the signature of `internal`, `fileprivate` and `private` declarations can reference the imported module. + +```swift +internal import InternalDependency +``` + +A dependency private to this source file is declared with either the `fileprivate` or the `private` modifier. +In both cases the access is scoped to the source file declaring the import. +Only the signature of `fileprivate` and `private` declarations can reference the imported module. + +```swift +fileprivate import DependencyPrivateToThisFile +private import OtherDependencyPrivateToThisFile +``` + +The `open` access-level modifier is rejected on import declarations. + +The `@usableFromInline` attribute can be applied to an import declaration to allow referencing a dependency from inlinable code +while limiting which declarations signatures can reference it. +The attribute `@usableFromInline` can be used only on `package` and `internal` imports. +It marks the dependency as visible to clients. +```swift +@usableFromInline package import UsableFromInlinePackageDependency +@usableFromInline internal import UsableFromInlineInternalDependency +``` + +*Note: Support for @usableFromInline on imports has yet to be implemented.* + +### Type-checking references to imported modules + +Current type-checking enforces that declaration respect their respective access levels. +It reports as errors when a more visible declaration refers to a less visible declaration. +For example, it raises an error if a `public` function signature uses an `internal` type. + +This proposal extends the existing logic by using the access level on the import declaration as an upper bound to the visibility of imported declarations within the source file with the import. +For example, when type-checking a source file with an `internal import SomeModule`, +we consider all declarations imported from `SomeModule` to have an access level of `internal` in the context of the file. +In this case, type-checking will enforce that declarations imported as `internal` are only referenced from `internal` or lower declaration signatures and in regular function bodies. +They cannot appear in public declaration signatures, `@usableFromInline` declaration signatures, or inlinable code. +This will be reported by the familiar diagnostics currently applied to access-level modifiers on declarations and to inlinable code. + +We apply the same logic for `package`, `fileprivate` and `private` import declarations. +In the case of a `public` import, there is no restriction on how the imported declarations can be referenced +beyond the existing restrictions on imported `package` declarations which cannot be referenced from public declaration signatures. + +The attribute `@usableFromInline` on an import takes effect for inlinable code: +`@inlinable` and `@backDeployed` function bodies, default initializers of arguments, and properties of `@frozen` structs. +The `@usableFromInline` imported dependency can be referenced from inlinable code +but doesn't affect type-checking of declaration signatures where only the access level is taken into account. + +Here is an example of the approximate diagnostics produced from type-checking in a typical case with a `fileprivate` import. +```swift +fileprivate import DatabaseAdapter + +fileprivate func fileprivateFunc() -> DatabaseAdapter.Entry { ... } // Ok + +internal func internalFunc() -> DatabaseAdapter.Entry { ... } // error: function cannot be declared internal because its return uses a fileprivate type + +public func publicFunc(entry: DatabaseAdapter.Entry) { ... } // error: function cannot be declared public because its parameter uses a fileprivate type + +public func useInBody() { + DatabaseAdapter.create() // Ok +} + +@inlinable +public func useInInlinableBody() { + DatabaseAdapter.create() // error: global function 'create()' is fileprivate and cannot be referenced from an '@inlinable' function +} +``` + +### Transitive dependency loading + +When using this access level information at the module level, +if a dependency is never imported publicly and other requirements are met, +it becomes possible to hide the dependency from clients. +The clients can then be built without loading the transitive dependency. +This can speed up build times and +avoid the need to distribute modules that are implementation details. + +The same dependency can be imported with different access levels by different files of a same module. +At the module level, we only take into account the most permissive access level. +For example, if a dependency is imported as `package` and `internal` from two different files, +we consider the dependency to be of `package` visibility at the module level. + +The module level information implies different behaviors for transitive clients. +Transitive clients are modules that have an indirect dependency on the module. +For example, in the following scenario, `TransitiveClient` is a transitive client +of `IndirectDependency` via the import of `MiddleModule`. + +``` +module IndirectDependency + ↑ +module MiddleModule + ↑ +module TransitiveClient +``` + +Depending on how the indirect dependency is imported from the middle module, +the transitive client may or may not need to load it at compile time. +There are four factors requiring a transitive dependency to be loaded; +if none of these apply, the dependency can be hidden. + +1. `public` or `@usableFromInline` dependencies must always be loaded by transitive clients. + +2. All dependencies of a non-resilient module must be loaded by transitive clients. + This is because types in the module can use types from those dependencies in their storage, + and the compiler needs complete information about the storage of non-resilient types + in order to emit code correctly. + This restriction is discussed further in the Future Directions section. + +3. `package` dependencies of a module must be loaded by its transitive clients if the module and the transitive client are part of the same package. + This is because `package` declarations in the module may use types from that dependency in their signatures. + We consider two modules to be in the same package when their package name matches, + applying the same logic used for package declarations. + +4. All dependencies of a module must be loaded if the transitive client has a `@testable` import of it. + This is because testable clients can use `internal` declarations, which may rely on dependencies with any level of import visibility. + Even `private` and `fileprivate` dependencies must be loaded. + +In all other cases, the dependency is hidden, and it doesn't have to be loaded by transitive clients. +Note that a dependency hidden on one import path may still need to be loaded because of a different import path. + +The module interface associated with a hidden dependency doesn't need to be distributed to clients. +However, the binary associated to the module still needs to be distributed to execute the resulting program. + +### Default import access level + +The access level of a default import declaration without an explicit access-level modifier depends on the language version. +We list here the implicit access levels and reasoning behind this choice. + +In language modes up to Swift 6, an import is `public` by default. +This choice preserves source compatibility. +The only official import previously available in Swift 5 behaves like the public import proposed in this document. + +In a future language mode, an import will be `internal` by default. +This will align the behavior of imports with declarations where the implicit access level is internal. +It should help limit unintentional dependency creep as marking a dependency public will require an explicit modifier. + +As a result, the following import is `public` in language modes up to Swift 6, but it will be `internal` in a future language mode: +```swift +import ADependency +``` + +The future language change will likely require source changes in code that adopts the new language mode. It will not break source compatibility for code that remains on current language modes. +A migration tool could automatically insert the `public` modifier where required. +Where the tool is unavailable, a simple script can insert a `public` modifier in front of all imports to preserve the Swift 5 behavior. + +The upcoming feature flag `InternalImportsByDefault` will enable the future language behavior even when using Swift 5 or 6. + +### Interactions with other modifiers on imports + +The `@_exported` attribute is a step above a `public` import, +as clients see the imported module declarations is if they were part of the local module. +With this proposal, `@_exported` is accepted only on public import declarations, +both with the modifier or the default `public` visibility in current language modes. + +The `@testable` attribute allows the local module to reference the internal declarations of the imported module. +The current design even allows to use an imported internal or package type in a public declaration. +The access level behavior applies in the same way as a normal import, +all imported declarations have as upper-bound the access level on the import declaration. +In the case of a `@testable` import, even the imported internal declarations are affected by the bound. + +Current uses of `@_implementationOnly import` should be replaced with an internal import or lower. +In comparison, this new feature enables stricter type-checking and shows fewer superfluous warnings. +After replacing with an internal import, the transitive dependency loading requirements will remain the same for resilient modules, +but will change for non-resilient modules where transitive dependencies must always be loaded. +In all cases, updating modules relying on `@_implementationOnly` to instead use internal imports is strongly encouraged. + +The scoped imports feature is independent from the access level declared on the same import. +In the example below, the module `Foo` is a public dependency at the module level and can be referenced from public declaration signatures in the local source file. +The scoped part, `struct Foo.Bar`, limits lookup so only `Bar` can be referenced from this file; it also prioritizes resolving references to this `Bar` if there are other `Bar` declarations in other imports. +Scoped imports cannot be used to restrict the access level of a single declaration. +```swift +public import struct Foo.Bar +``` + +## Source compatibility + +To preserve source compatibility, imports are public by default in current language modes, including Swift 6. +This will preserve the current behavior of imports in Swift 5. +As discussed previously, the future language mode behavior changes the default value and will require code changes. + +## ABI compatibility + +This proposal doesn't affect ABI compatibility, +it is a compile time change enforced by type-checking. + +## Implications on adoption + +Adopting or reverting the adoption of this feature should not affect clients if used with care. + +In the case of adoption in a non-resilient module, the change is in type-checking of the module source files only. +In this case changing the access level of different dependencies won't affect clients. + +For adoption in a resilient module, +marking an existing import as less than public will affect how clients build. +The compiler can build the clients by loading fewer transitive dependencies. +In theory, this shouldn't affect the clients but it may still lead to different compilation behaviors. + +In theory, these transitive dependencies couldn't be used by the clients, +so hiding them doesn't affect the clients. +In practice, there are leaks allowing use of extension members from transitive dependencies. +Adopting this feature may skip loading transitive dependencies and prevent those leaks, +it can break source compatibility in code relying of those behaviors. + +## Future directions + +### Hiding dependencies for non-resilient modules + +Hiding dependencies on non-resilient modules would be possible in theory but requires rethinking a few restrictions in the compilation process. +The main restriction is the need of the compiler to know the memory layout of imported types, which can depend on transitive dependencies. +Resilient modules can provide this information at run time so the transitive module isn't required at build time. +Non-resilient modules do not provide this information at run time, so the compiler must load the transitive dependencies at build time to access it. +Solutions could involve copying the required information in each modules, +or restricting further how a dependency can be referenced. +In all cases, it's a feature in itself and distinct from this proposal. + +## Alternatives considered + +### `@_implementationOnly import` + +The unofficial `@_implementationOnly` attribute offers a similar feature with both type-checking and hiding transitive dependencies. +This attribute has lead to instability and run time crashes when used from a non-resilient module or combined with an `@testable` import. +It applies a slightly different semantic than this proposal and its type-checking isn't as strict as it could be. +It relied on its own type-checking logic to report references to the implementation-only imported module from public declarations. +In contrast, this proposal uses the existing access level checking logic and semantics, +this should make it easier to learn. +Plus this proposal introduces whole new features with `package` imports and file-scoped imports with `private` and `fileprivate`. + +### Use `open import` as an official `@_exported import` + +The access-level modifier `open` remains available for use on imports as this proposal doesn't assign it a specific meaning. +It has been suggested to use it as an official `@_exported`. +That is, mark an import that is visible from all source files of the module and shown to clients as if it was part of the same module. +We usually use `@_exported` for Swift overlays to clang module +where two modules share the same name and the intention is to show them as unified to clients. + +Two main reasons keep me from incorporating this change to this proposal: + +1. A declaration marked as `open` can be overridden from outside the module. + This meaning has no relation with the behavior of `@_exported`. + The other access levels have a corresponding meaning between their use on a declaration and on an import declaration. +2. A motivation for this proposal is to hide implementation details and limit dependency creep. + Encouraging the use of `open import` or `@_exported` goes against this motivation and addresses a different set of problems. + It should be discussed in a distinct proposal with related motivations. + +### Infer the visibility of a dependency from its use in API + +By analyzing a module the compiler could determine which dependencies are used by public declarations and need to be visible to clients. +We could then automatically consider all other dependencies as internal and hide them from indirect clients if the other criteria are met. + +This approach lacks the duplication of information offered by the access-level modifier on the import declaration and the references from declaration signatures. +This duplication enables the type-checking behavior described in this proposal by +allowing the compiler to compare the intent marked on the import with the use in declaration signatures. +This check is important when the dependency is not distributed, +a change from a hidden dependency to a public dependency may break the distributed module on a dependency that is not available to third parties. + +## Acknowledgments + +Becca Royal-Gordon contributed to the design and wrote the pre-pitch of this proposal. + diff --git a/proposals/0410-atomics.md b/proposals/0410-atomics.md new file mode 100644 index 0000000000..21d348b3a2 --- /dev/null +++ b/proposals/0410-atomics.md @@ -0,0 +1,1844 @@ +# Low-Level Atomic Operations ⚛︎ + +* Proposal: [SE-0410](0410-atomics.md) +* Author: [Karoy Lorentey](https://github.com/lorentey), [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Bug: [SR-9144](https://github.com/apple/swift/issues/51640) +* Implementation: [apple/swift#68857](https://github.com/apple/swift/pull/68857) +* Version: 2023-12-04 +* Status: **Implemented (Swift 6.0)** +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/d35d6566fe2297f4782bdfac4d5253e0ca96b353/proposals/0410-atomics.md) +* Decision Notes: [pitch](https://forums.swift.org/t/atomics/67350), [first review](https://forums.swift.org/t/se-0410-atomics/68007), [first return for revision](https://forums.swift.org/t/returned-for-revision-se-0410-atomics/68522), [second review](https://forums.swift.org/t/second-review-se-0410-atomics/68810), [acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0410-atomics/69244) + +## Introduction + +This proposal adds a limited set of low-level atomic operations to the Standard Library, including native spellings for C++-style memory orderings. Our goal is to enable intrepid library authors and developers writing system level code to start building synchronization constructs directly in Swift. + +Previous Swift-evolution thread: [Low-Level Atomic Operations](https://forums.swift.org/t/low-level-atomic-operations/34683) + +New Swift-evolution thread: [Atomics](https://forums.swift.org/t/atomics/67350) + +## Revision History + +- 2020-04-13: Initial proposal version. +- 2020-06-05: Second revision. + - Removed all new APIs; the proposal is now focused solely on C interoperability. +- 2023-09-18: Third revision. + - Introduced new APIs to the standard library. +- 2023-12-04: Fourth revision. + - Response to language steering group [review decision notes](https://forums.swift.org/t/returned-for-revision-se-0410-atomics/68522). + - New APIs are now in a `Synchronization` module instead of the default `Swift` module. + - Declaring a `var` of `Atomic` type is now an error. + +## Table of Contents + + * [Motivation](#motivation) + * [Proposed Solution](#proposed-solution) + * [The Synchronization Module](#the-synchronization-module) + * [Atomic Memory Orderings](#atomic-memory-orderings) + * [The Atomic Protocol Hierarchy](#the-atomic-protocol-hierarchy) + * [Optional Atomics](#optional-atomics) + * [Custom Atomic Types](#custom-atomic-types) + * [Atomic Storage Types](#atomic-storage-types) + * [WordPair](#wordpair) + * [The Atomic type](#the-atomic-type) + * [Basic Atomic Operations](#basic-atomic-operations) + * [Specialized Integer Operations](#specialized-integer-operations) + * [Specialized Boolean Operations](#specialized-boolean-operations) + * [Atomic Lazy References](#atomic-lazy-references) + * [Restricting Ordering Arguments to Compile\-Time Constants](#restricting-ordering-arguments-to-compile-time-constants) + * [Interaction with Existing Language Features](#interaction-with-existing-language-features) + * [Interaction with Swift Concurrency](#interaction-with-swift-concurrency) + * [Detailed Design](#detailed-design) + * [Atomic Memory Orderings](#atomic-memory-orderings-1) + * [Atomic Protocols](#atomic-protocols) + * [AtomicRepresentable](#atomicrepresentable) + * [AtomicOptionalRepresentable](#atomicoptionalrepresentable) + * [WordPair](#wordpair-1) + * [Atomic Types](#atomic-types) + * [Atomic<Value>](#atomicvalue) + * [AtomicLazyReference<Instance>](#atomiclazyreferenceinstance) + * [Source Compatibility](#source-compatibility) + * [Effect on ABI Stability](#effect-on-abi-stability) + * [Effect on API Resilience](#effect-on-api-resilience) + * [Potential Future Directions](#potential-future-directions) + * [Atomic Strong References and The Problem of Memory Reclamation](#atomic-strong-references-and-the-problem-of-memory-reclamation) + * [Additional Low\-Level Atomic Features](#additional-low-level-atomic-features) + * [Alternatives Considered](#alternatives-considered) + * [Default Orderings](#default-orderings) + * [A Truly Universal Generic Atomic Type](#a-truly-universal-generic-atomic-type) + * [Providing a value Property](#providing-a-value-property) + * [Alternative Designs for Memory Orderings](#alternative-designs-for-memory-orderings) + * [Encode Orderings in Method Names](#encode-orderings-in-method-names) + * [Orderings As Generic Type Parameters](#orderings-as-generic-type-parameters) + * [Ordering Views](#ordering-views) + * [Directly bring over `swift-atomics`'s API](#directly-bring-over-swift-atomicss-api) + * [References](#references) + +## Motivation + +In Swift today, application developers use Swift's recently accepted concurrency features including async/await, structured concurrency with Task and TaskGroup, AsyncSequence/AsyncStream, etc. as well as dispatch queues and Foundation's NSLocking protocol to synchronize access to mutable state across concurrent threads of execution. + +However, for Swift to be successful as a systems programming language, it needs to also provide low-level primitives that can be used to implement such synchronization constructs (and many more!) directly within Swift. Such low-level synchronization primitives allow developers more flexible ways to synchronize access to specific properties or storage allowing them to opt their types into Swift conconcurrency by declaring their types `@unchecked Sendable`. Of course these low-level primitives also allow library authors to build more high level synchronization structures that are both easier and safer to use that developers can also utilize to synchronize memory access. + +One such low-level primitive is the concept of an atomic value, which (in the form we propose here) has two equally important roles: + +- First, atomics introduce a limited set of types whose values provide well-defined semantics for certain kinds of concurrent access. This includes explicit support for concurrent mutations -- a concept that Swift never supported before. + +- Second, atomic operations come with explicit memory ordering arguments, which provide guarantees on how/when the effects of earlier or later memory accesses become visible to other threads. Such guarantees are crucial for building higher-level synchronization abstractions. + +These new primitives are intended for people who wish to implement synchronization constructs or concurrent data structures in pure Swift code. Note that this is a hazardous area that is full of pitfalls. While a well-designed atomics facility can help simplify building such tools, the goal here is merely to make it *possible* to build them, not necessarily to make it *easy* to do so. We expect that the higher-level synchronization tools that can be built on top of these atomic primitives will provide a nicer abstraction layer. + +We want to limit this proposal to constructs that satisfy the following requirements: + +1. All atomic operations need to be explicit in Swift source, and it must be possible to easily distinguish them from regular non-atomic operations on the underlying values. + +2. The atomic type we provide must come with a lock-free implementation on every platform that implements them. (Platforms that are unable to provide lock-free implementations must not provide the affected constructs at all.) + +3. Every atomic operation must compile down to the corresponding CPU instruction (when one is available), with minimal overhead. (Ideally even if the code is compiled without optimizations.) Wait-freedom isn't a requirement -- if no direct instruction is available for an operation, then it must still be implemented, e.g. by mapping it to a compare-exchange loop. + +Following the acceptance of [Clarify the Swift memory consistency model (SE-0282)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md), the [swift-atomics package](https://github.com/apple/swift-atomics) was shortly created to experiment and design what a standard atomic API would look like. This proposal is relying heavily on some of the ideas that package has spent years developing and designing. + +## Proposed Solution + +We propose to introduce new low-level atomic APIs to the standard library via a new module. These atomic APIs will serve as the foundation for building higher-level concurrent code directly in Swift. + +As a quick taste, this is how atomics will work: + +```swift +import Synchronization +import Dispatch + +let counter = Atomic(0) + +DispatchQueue.concurrentPerform(iterations: 10) { _ in + for _ in 0 ..< 1_000_000 { + counter.wrappingAdd(1, ordering: .relaxed) + } +} + +print(counter.load(ordering: .relaxed)) +``` + +### The Synchronization Module + +While most Swift programs won't directly use the new atomic primitives, we still consider the new constructs to be an integral part of the core Standard Library. + +That said, it seems highly undesirable to add low-level atomics to the default namespace of every Swift program, so we propose to place the atomic constructs in a new Standard Library module called `Synchronization`. Code that needs to use low-level atomics will need to explicitly import the new module: + +```swift +import Synchronization +``` + +We expect that most Swift projects will use atomic operations only indirectly, through higher-level synchronization constructs. Therefore, importing the `Synchronization` module will be a relatively rare occurrence, mostly limited to projects that implement such tools. + +### Atomic Memory Orderings + +The atomic constructs later in this proposal implement concurrent read/write access by mapping to atomic instructions in the underlying architecture. All accesses of a particular atomic value get serialized into some global sequential timeline, no matter what thread executed them. + +However, this alone does not give us a way to synchronize accesses to regular variables, or between atomic accesses to different memory locations. To support such synchronization, each atomic operation can be configured to also act as a synchronization point for other variable accesses within the same thread, preventing previous accesses from getting executed after the atomic operation, and/or vice versa. Atomic operations on another thread can then synchronize with the same point, establishing a strict (although partial) timeline between accesses performed by both threads. This way, we can reason about the possible ordering of operations across threads, even if we know nothing about how those operations are implemented. (This is how locks or dispatch queues can be used to serialize the execution of arbitrary blocks containing regular accesses to shared variables.) For more details, see \[[C++17], [N2153], [Boehm 2008]]. + +In order to enable atomic synchronization within Swift, we must first introduce memory orderings that will give us control of the timeline of these operations across threads. Luckily, with the acceptance of [Clarify the Swift memory consistency model (SE-0282)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md), Swift already adopts the C/C++ concurrency memory model. In this model, concurrent access to shared state remains undefined behavior unless all such access is forced into a conflict-free timeline through explicit synchronization operations. + +This proposal introduces five distinct memory orderings, organized into three logical groups, from loosest to strictest: + +* `.relaxed` +* `.acquiring`, `.releasing`, `.acquiringAndReleasing` +* `.sequentiallyConsistent` + +These align with select members of the standard `std::memory_order` enumeration in C++, and are intended to carry the same semantic meaning: + +| C++ | Swift | +| :---: | :---: | +| `std::memory_order_relaxed` | `.relaxed` | +| `std::memory_order_consume` | *not yet adopted [[P0735]]* | +| `std::memory_order_acquire` | `.acquiring` | +| `std::memory_order_release` | `.releasing` | +| `std::memory_order_acq_rel` | `.acquiringAndReleasing` | +| `std::memory_order_seq_cst` | `.sequentiallyConsistent` | + +Atomic orderings are grouped into three frozen structs based on the kind of operation to which they are attached, as listed below. By modeling these as separate types, we can ensure that unsupported operation/ordering combinations (such as an atomic "releasing load") will lead to clear compile-time errors: + +```swift +/// Specifies the memory ordering semantics of an atomic load operation. +public struct AtomicLoadOrdering { + public static var relaxed: Self { get } + public static var acquiring: Self { get } + public static var sequentiallyConsistent: Self { get } +} + +/// Specifies the memory ordering semantics of an atomic store operation. +public struct AtomicStoreOrdering { + public static var relaxed: Self { get } + public static var releasing: Self { get } + public static var sequentiallyConsistent: Self { get } +} + +/// Specifies the memory ordering semantics of an atomic read-modify-write +/// operation. +public struct AtomicUpdateOrdering { + public static var relaxed: Self { get } + public static var acquiring: Self { get } + public static var releasing: Self { get } + public static var acquiringAndReleasing: Self { get } + public static var sequentiallyConsistent: Self { get } +} +``` + +These structs behave like non-frozen enums with a known (non-public) raw representation. This allows us to define additional memory orderings in the future (if and when they become necessary, specifically `std::memory_order_consume`) while making use of the known representation to optimize existing cases. (These cannot be frozen enums because that would prevent us from adding more orderings, but regular resilient enums can't freeze their representation, and the layout indirection interferes with guaranteed optimizations, especially in -Onone.) + +Every atomic operation introduced later in this proposal requires an ordering argument. We consider these ordering arguments to be an essential part of these low-level atomic APIs, and we require an explicit `ordering` argument on all atomic operations. The intention here is to force developers to carefully think about what ordering they need to use, each time they use one of these primitives. (Perhaps more importantly, this also makes it obvious to readers of the code what ordering is used -- making it far less likely that an unintended default `.sequentiallyConsistent` ordering slips through code review.) + +Projects that prefer to default to sequentially consistent ordering are welcome to add non-public `Atomic` extensions that implement that. However, we expect that providing an implicit default ordering would be highly undesirable in most production uses of atomics. + +We also provide a top-level function called `atomicMemoryFence` that allows issuing a memory ordering constraint without directly associating it with a particular atomic operation. This corresponds to `std::atomic_thread_fence` in C++ [[C++17]]. + +```swift +/// Establishes a memory ordering without associating it with a +/// particular atomic operation. +/// +/// - A relaxed fence has no effect. +/// - An acquiring fence ties to any preceding atomic operation that +/// reads a value, and synchronizes with any releasing operation whose +/// value was read. +/// - A releasing fence ties to any subsequent atomic operation that +/// modifies a value, and synchronizes with any acquiring operation +/// that reads the result. +/// - An acquiring and releasing fence is a combination of an +/// acquiring and a releasing fence. +/// - A sequentially consistent fence behaves like an acquiring and +/// releasing fence, and ensures that the fence itself is part of +/// the single, total ordering for all sequentially consistent +/// operations. +/// +/// This operation corresponds to `std::atomic_thread_fence` in C++. +/// +/// Be aware that Thread Sanitizer does not support fences and may report +/// false-positive races for data protected by a fence. +public func atomicMemoryFence(ordering: AtomicUpdateOrdering) +``` + +Fences are slightly more powerful (but even more difficult to use) than orderings tied to specific atomic operations [[N2153]]; we expect their use will be limited to the most performance-sensitive synchronization constructs. + +### The Atomic Protocol Hierarchy + +The notion of an atomic type is captured by the `AtomicRepresentable` protocol. + +```swift +/// A type that supports atomic operations through a separate atomic storage +/// representation. +public protocol AtomicRepresentable { + associatedtype AtomicRepresentation + + static func encodeAtomicRepresentation( + _ value: consuming Self + ) -> AtomicRepresentation + + static func decodeAtomicRepresentation( + _ representation: consuming AtomicRepresentation + ) -> Self +} +``` + +The requirements in `AtomicRepresentable` set up a bidirectional mapping between values of the atomic type and an associated storage representation that implements the actual primitive atomic operations. + +`AtomicRepresentation` is intentionally left unconstrained because as you'll see later in the proposal, atomic operations are only available when `AtomicRepresentation` is one of the core atomic storage types found here: [Atomic Storage Types](#atomic-storage-types). + +The full set of standard types implementing `AtomicRepresentable` is listed below: + +```swift +extension Int: AtomicRepresentable {...} +extension Int64: AtomicRepresentable {...} +extension Int32: AtomicRepresentable {...} +extension Int16: AtomicRepresentable {...} +extension Int8: AtomicRepresentable {...} +extension UInt: AtomicRepresentable {...} +extension UInt64: AtomicRepresentable {...} +extension UInt32: AtomicRepresentable {...} +extension UInt16: AtomicRepresentable {...} +extension UInt8: AtomicRepresentable {...} + +extension Bool: AtomicRepresentable {...} + +extension Float16: AtomicRepresentable {...} +extension Float: AtomicRepresentable {...} +extension Double: AtomicRepresentable {...} + +/// New type in the standard library discussed +/// shortly after this. +extension WordPair: AtomicRepresentable {...} + +extension Duration: AtomicRepresentable {...} + +extension Never: AtomicRepresentable {...} + +extension UnsafeRawPointer: AtomicRepresentable {...} +extension UnsafeMutableRawPointer: AtomicRepresentable {...} +extension UnsafePointer: AtomicRepresentable {...} +extension UnsafeMutablePointer: AtomicRepresentable {...} +extension Unmanaged: AtomicRepresentable {...} +extension OpaquePointer: AtomicRepresentable {...} +extension ObjectIdentifier: AtomicRepresentable {...} + +extension UnsafeBufferPointer: AtomicRepresentable {...} +extension UnsafeMutableBufferPointer: AtomicRepresentable {...} +extension UnsafeRawBufferPointer: AtomicRepresentable {...} +extension UnsafeMutableRawBufferPointer: AtomicRepresentable {...} + +extension Optional: AtomicRepresentable where Wrapped: AtomicOptionalRepresentable {...} +``` + +* On 32 bit platforms that do not support double-word atomics, the following conformances are not available: + * `UInt64` + * `Int64` + * `Double` + * `UnsafeBufferPointer` + * `UnsafeMutableBufferPointer` + * `UnsafeRawBufferPointer` + * `UnsafeMutableRawBufferPointer` +* On 64 bit platforms that do not support double-word atomics, the following conformances are not available: + * `Duration` + * `UnsafeBufferPointer` + * `UnsafeMutableBufferPointer` + * `UnsafeRawBufferPointer` + * `UnsafeMutableRawBufferPointer` + +This proposal does not conform `Duration` to `AtomicRepresentable` on any currently supported 32 bit platform. (Not even those where quad-word atomics are technically available, like arm64_32.) + +#### Optional Atomics + +The standard atomic pointer types and unmanaged references also support atomic operations on their optional-wrapped form. To spell out this optional wrapped, we introduce a new protocol: + +```swift +public protocol AtomicOptionalRepresentable: AtomicRepresentable { + associatedtype AtomicOptionalRepresentation + + static func encodeAtomicOptionalRepresentation( + _ value: consuming Self? + ) -> AtomicOptionalRepresentation + + static func decodeAtomicOptionalRepresentation( + _ representation: consuming AtomicOptionalRepresentation + ) -> Self? +} +``` + +Similar to `AtomicRepresentable`, `AtomicOptionalRepresentable`'s requirements create a bidirectional mapping between an optional value of `Self` to some atomic optional storage representation and vice versa. + +`Optional` implements `AtomicRepresentable` through a conditional conformance to this new `AtomicOptionalRepresentable` protocol. + +```swift +extension Optional: AtomicRepresentable where Wrapped: AtomicOptionalRepresentable { + ... +} +``` + +This proposal enables optional-atomics support for the following types: + +```swift +extension UnsafeRawPointer: AtomicOptionalRepresentable {} +extension UnsafeMutableRawPointer: AtomicOptionalRepresentable {} +extension UnsafePointer: AtomicOptionalRepresentable {} +extension UnsafeMutablePointer: AtomicOptionalRepresentable {} +extension Unmanaged: AtomicOptionalRepresentable {} +extension OpaquePointer: AtomicOptionalRepresentable {} +extension ObjectIdentifier: AtomicOptionalRepresentable {} +``` + +Atomic optional pointers and references are helpful when building lock-free data structures. (Although this initial set of reference types considerably limits the scope of what can be built; for more details, see the discussion on the [ABA problem](#wordpair) and [memory reclamation](#atomic-strong-references-and-the-problem-of-memory-reclamation).) + +For example, consider the lock-free, single-consumer stack implementation below. (It supports an arbitrary number of concurrently pushing threads, but it only allows a single pop at a time.) + +```swift +class LockFreeSingleConsumerStack { + struct Node { + let value: Element + var next: UnsafeMutablePointer? + } + typealias NodePtr = UnsafeMutablePointer + + private let _last = Atomic(nil) + private let _consumerCount = Atomic(0) + + deinit { + // Discard remaining nodes + while let _ = pop() {} + } + + // Push the given element to the top of the stack. + // It is okay to concurrently call this in an arbitrary number of threads. + func push(_ value: Element) { + let new = NodePtr.allocate(capacity: 1) + new.initialize(to: Node(value: value, next: nil)) + + var done = false + var current = _last.load(ordering: .relaxed) + while !done { + new.pointee.next = current + (done, current) = _last.compareExchange( + expected: current, + desired: new, + ordering: .releasing + ) + } + } + + // Pop and return the topmost element from the stack. + // This method does not support multiple overlapping concurrent calls. + func pop() -> Element? { + precondition( + _consumerCount.wrappingAdd(1, ordering: .acquiring).oldValue == 0, + "Multiple consumers detected") + defer { _consumerCount.wrappingSubtract(1, ordering: .releasing) } + var done = false + var current = _last.load(ordering: .acquiring) + while let c = current { + (done, current) = _last.compareExchange( + expected: c, + desired: c.pointee.next, + ordering: .acquiring + ) + + if done { + let result = c.move() + c.deallocate() + return result.value + } + } + return nil + } +} +``` + +#### Custom Atomic Types + +To enable a limited set of user-defined atomic types, `AtomicRepresentable` also provides a full set of default implementations for `RawRepresentable` types whose raw value is itself atomic: + +```swift +extension RawRepresentable where Self: AtomicRepresentable, RawValue: AtomicRepresentable { + ... +} +``` + +The default implementations work by forwarding all atomic operations to the raw value's implementation, converting to/from as needed. + +This enables code outside of the Standard Library to add new `AtomicRepresentable` conformances without manually implementing any of the requirements. This is especially handy for trivial raw-representable enumerations, such as in simple atomic state machines: + +```swift +enum MyState: Int, AtomicRepresentable { + case starting + case running + case stopped +} + +let currentState = Atomic(.starting) +... +if currentState.compareExchange( + expected: .starting, + desired: .running, + ordering: .sequentiallyConsistent +).exchanged { + ... +} +... +currentState.store(.stopped, ordering: .sequentiallyConsistent) +``` + +We also support the `AtomicOptionalRepresentable` defaults for `RawRepresentable` as well: + +```swift +extension RawRepresentable where Self: AtomicOptionalRepresentable, RawValue: AtomicOptionalRepresentable { + ... +} +``` + +For example, we can use this to add atomic operations over optionals of types whose raw value is a pointer: + +```swift +struct MyPointer: RawRepresentable, AtomicOptionalRepresentable { + var rawValue: UnsafeRawPointer + + init(rawValue: UnsafeRawPointer) { + self.rawValue = rawValue + } +} + +let myAtomicPointer = Atomic(nil) +... +if myAtomicPointer.compareExchange( + expected: nil, + desired: MyPointer(rawValue: somePointer), + ordering: .relaxed +).exchanged { + ... +} +... +myAtomicPointer.store(nil, ordering: .releasing) +``` + +(This gets you an `AtomicRepresentable` conformance for free as well because `AtomicOptionalRepresentable` refines `AtomicRepresentable`. So this also allows non-optional use with `Atomic`.) + +### Atomic Storage Types + +Fundamental to working with atomics is knowing that CPUs can only do atomic operations on integers. While we could theoretically do atomic operations with our current list of standard library integer types (`Int8`, `Int16`, ...), some platforms don't ensure that these types have the same alignment as their size. For example, `Int64` and `UInt64` have 4 byte alignment on i386. Atomic operations must occur on correctly aligned types. To ensure this, we need to introduce helper types that all atomic operations will be trafficked through. These types will serve as the `AtomicRepresentation` for all of the standard integer types: + +```swift +extension Int8: AtomicRepresentable { + public typealias AtomicRepresentation = ... +} + +... + +extension UInt64: AtomicRepresentable { + public typealias AtomicRepresentation = ... +} + +... +``` + +The actual underlying type is an implementation detail of the standard library. While we generally don't prefer to propose such API, the underlying types themselves are quite useless and only useful for the primitive integers. One can still access the underlying type by using the public name, `Int8.AtomicRepresentation`, for example. An example conformance to `AtomicRepresentable` may look something like the following: + +```swift +struct MyCoolInt { + var x: Int +} + +extension MyCoolInt: AtomicRepresentable { + typealias AtomicRepresentation = Int.AtomicRepresentation + + static func encodeAtomicRepresentation( + _ value: consuming MyCoolInt + ) -> AtomicRepresentation { + Int.encodeAtomicRepresentation(value.x) + } + + static func decodeAtomicRepresentation( + _ representation: consuming AtomicRepresentation + ) -> MyCoolInt { + MyCoolInt( + x:Int.decodeAtomicRepresentation(representation) + ) + } +} +``` + +This works by going through `Int`'s `AtomicRepresentable` conformance and converting our `MyCoolInt` -> `Int` -> `Int.AtomicRepresentation` . + +### `WordPair` + +In their current single-word form, atomic pointer and reference types are susceptible to a class of race condition called the *ABA problem*. A freshly allocated object often happens to be placed at the same memory location as a recently deallocated one. Therefore, two successive `load`s of a simple atomic pointer may return the exact same value, even though the pointer may have received an arbitrary number of updates between the two loads, and the pointee may have been completely replaced. This can be a subtle, but deadly source of race conditions in naive implementations of many concurrent data structures. + +While the single-word atomic primitives introduced in this document are already useful for some applications, it would be helpful to also provide a set of additional atomic operations that operate on two consecutive `Int`-sized values in the same transaction. All currently supported architectures provide direct hardware support for such double-word atomic operations. + +We propose a new separate type that provides an abstraction over the layout of what a double-word is for a platform. + +```swift +public struct WordPair { + public var first: UInt { get } + public var second: UInt { get } + + public init(first: UInt, second: UInt) +} + +// Not a real compilation conditional +#if hasDoubleWideAtomics +extension WordPair: AtomicRepresentable { +// Not a real compilation conditional +#if 64 bit + public typealias AtomicRepresentaton = ... 128 bit 16 aligned storage +#elseif 32 bit + public typealias AtomicRepresentation = ... 64 bit 8 aligned storage +#else +#error("Not a supported platform") +#endif + + ... +} +#endif +``` + +For example, the second word can be used to augment atomic values with a version counter (sometimes called a "stamp" or a "tag"), which can help resolve the ABA problem by allowing code to reliably verify if a value remained unchanged between two successive loads. + +Note that not all CPUs support double-word atomic operations and so if Swift starts supporting such processors, this type's conformance to `AtomicRepresentable` may not always be available. Platforms that cannot support double-word atomics must not make `WordPair`'s `AtomicRepresentable` conformance available for use. + +(If this becomes a real concern, a future proposal could introduce something like a `#if hasDoubleWordAtomics` compile-time condition to let code adapt to more limited environments. However, this is deferred until Swift actually starts supporting such platforms.) + +### The Atomic type + +So far, we've introduced memory orderings, giving us control of memory access around atomic operations; the atomic protocol hierarchy, which give us the initial list of standard types that can be as atomic values; and the `WordPair` type, providing an abstraction over a platform's double-word type. However, we haven't yet introduced a way to actually _use_ atomics. Here we introduce the single Atomic type that exposes atomic operations for us: + +```swift +/// An atomic value. +public struct Atomic: ~Copyable { + public init(_ initialValue: consuming Value) +} +``` + +A value of `Atomic` shares the same layout as `Value.AtomicRepresentation`. + +Now that we know how to create an atomic value, it's time to introduce some actual atomic operations. + +### Basic Atomic Operations + +`Atomic` provides seven basic atomic operations when `Value.AtomicRepresentation` is one of the fundamental atomic storage types on the standard integer types: + +```swift +extension Atomic where Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation { + /// Atomically loads and returns the current value, applying the specified + /// memory ordering. + /// + /// - Parameter ordering: The memory ordering to apply on this operation. + /// - Returns: The current value. + public borrowing func load(ordering: AtomicLoadOrdering) -> Value + + /// Atomically sets the current value to `desired`, applying the specified + /// memory ordering. + /// + /// - Parameter desired: The desired new value. + /// - Parameter ordering: The memory ordering to apply on this operation. + public borrowing func store( + _ desired: consuming Value, + ordering: AtomicStoreOrdering + ) + + /// Atomically sets the current value to `desired` and returns the original + /// value, applying the specified memory ordering. + /// + /// - Parameter desired: The desired new value. + /// - Parameter ordering: The memory ordering to apply on this operation. + /// - Returns: The original value. + public borrowing func exchange( + _ desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> Value + + /// Perform an atomic compare and exchange operation on the current value, + /// applying the specified memory ordering. + /// + /// This operation performs the following algorithm as a single atomic + /// transaction: + /// + /// ``` + /// atomic(self) { currentValue in + /// let original = currentValue + /// guard original == expected else { return (false, original) } + /// currentValue = desired + /// return (true, original) + /// } + /// ``` + /// + /// This method implements a "strong" compare and exchange operation + /// that does not permit spurious failures. + /// + /// - Parameter expected: The expected current value. + /// - Parameter desired: The desired new value. + /// - Parameter ordering: The memory ordering to apply on this operation. + /// - Returns: A tuple `(exchanged, original)`, where `exchanged` is true if + /// the exchange was successful, and `original` is the original value. + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Value) + + /// Perform an atomic compare and exchange operation on the current value, + /// applying the specified success/failure memory orderings. + /// + /// This operation performs the following algorithm as a single atomic + /// transaction: + /// + /// ``` + /// atomic(self) { currentValue in + /// let original = currentValue + /// guard original == expected else { return (false, original) } + /// currentValue = desired + /// return (true, original) + /// } + /// ``` + /// + /// The `successOrdering` argument specifies the memory ordering to use when + /// the operation manages to update the current value, while `failureOrdering` + /// will be used when the operation leaves the value intact. + /// + /// This method implements a "strong" compare and exchange operation + /// that does not permit spurious failures. + /// + /// - Parameter expected: The expected current value. + /// - Parameter desired: The desired new value. + /// - Parameter successOrdering: The memory ordering to apply if this + /// operation performs the exchange. + /// - Parameter failureOrdering: The memory ordering to apply on this + /// operation if it does not perform the exchange. + /// - Returns: A tuple `(exchanged, original)`, where `exchanged` is true if + /// the exchange was successful, and `original` is the original value. + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + successOrdering: AtomicUpdateOrdering, + failureOrdering: AtomicLoadOrdering + ) -> (exchanged: Bool, original: Value) + + /// Perform an atomic weak compare and exchange operation on the current + /// value, applying the memory ordering. This compare-exchange variant is + /// allowed to spuriously fail; it is designed to be called in a loop until + /// it indicates a successful exchange has happened. + /// + /// This operation performs the following algorithm as a single atomic + /// transaction: + /// + /// ``` + /// atomic(self) { currentValue in + /// let original = currentValue + /// guard original == expected else { return (false, original) } + /// currentValue = desired + /// return (true, original) + /// } + /// ``` + /// + /// (In this weak form, transient conditions may cause the `original == + /// expected` check to sometimes return false when the two values are in fact + /// the same.) + /// + /// - Parameter expected: The expected current value. + /// - Parameter desired: The desired new value. + /// - Parameter ordering: The memory ordering to apply on this operation. + /// - Returns: A tuple `(exchanged, original)`, where `exchanged` is true if + /// the exchange was successful, and `original` is the original value. + public borrowing func weakCompareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Value) + + /// Perform an atomic weak compare and exchange operation on the current + /// value, applying the specified success/failure memory orderings. This + /// compare-exchange variant is allowed to spuriously fail; it is designed to + /// be called in a loop until it indicates a successful exchange has happened. + /// + /// This operation performs the following algorithm as a single atomic + /// transaction: + /// + /// ``` + /// atomic(self) { currentValue in + /// let original = currentValue + /// guard original == expected else { return (false, original) } + /// currentValue = desired + /// return (true, original) + /// } + /// ``` + /// + /// (In this weak form, transient conditions may cause the `original == + /// expected` check to sometimes return false when the two values are in fact + /// the same.) + /// + /// The `ordering` argument specifies the memory ordering to use when the + /// operation manages to update the current value, while `failureOrdering` + /// will be used when the operation leaves the value intact. + /// + /// - Parameter expected: The expected current value. + /// - Parameter desired: The desired new value. + /// - Parameter successOrdering: The memory ordering to apply if this + /// operation performs the exchange. + /// - Parameter failureOrdering: The memory ordering to apply on this + /// operation does not perform the exchange. + /// - Returns: A tuple `(exchanged, original)`, where `exchanged` is true if + /// the exchange was successful, and `original` is the original value. + public borrowing func weakCompareExchange( + expected: consuming Value, + desired: consuming Value, + successOrdering: AtomicUpdateOrdering, + failureOrdering: AtomicLoadOrdering + ) -> (exchanged: Bool, original: Value) +} +``` + +Because these are only available when `Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation`, some atomic specializations may not support atomic operations at all. + +The first three operations are relatively simple: + +- `load` returns the current value. +- `store` updates it. +- `exchange` is a combination of `load` and `store`; it updates the + current value and returns the previous one as a single atomic + operation. + +The three `compareExchange` variants are somewhat more complicated: they implement a version of `exchange` that only performs the update if the original value is the same as a supplied expected value. To be specific, they execute the following algorithm as a single atomic transaction: + +```swift + guard currentValue == expected else { + return (exchanged: false, original: currentValue) + } + currentValue = desired + return (exchanged: true, original: expected) +``` + +All four variants implement the same algorithm. The single ordering variants use the same memory ordering whether or not the exchange succeeds, while the others allow callers to specify two distinct memory orderings for the success and failure cases. The two orderings are independent from each other -- all combinations of update/load orderings are supported [[P0418]]. (Of course, the implementation may need to "round up" to the nearest ordering combination that is supported by the underlying code generation layer and the targeted CPU architecture.) + +The `weakCompareExchange` form may sometimes return false even when the original and expected values are equal. (Such failures may happen when some transient condition prevents the underlying operation from succeeding -- such as an incoming interrupt during a load-link/store-conditional instruction sequence.) This variant is designed to be called in a loop that only exits when the exchange is successful; in such loops, using `weakCompareExchange` may lead to a performance improvement by eliminating a nested loop in the regular, "strong", `compareExchange` variants. + +The compare-exchange primitive is special: it is a universal operation that can be used to implement all other atomic operations, and more. For example, here is how we could use `compareExchange` to implement a wrapping add operation over `Atomic` values: + +```swift +extension Atomic where Value == Int { + func wrappingAdd( + _ operand: Int, + ordering: AtomicUpdateOrdering + ) { + var done = false + var current = load(ordering: .relaxed) + while !done { + (done, current) = compareExchange( + expected: current, + desired: current &+ operand, + ordering: ordering + ) + } + } +} +``` + +### Specialized Integer Operations + +Most CPU architectures provide dedicated atomic instructions for certain integer operations, and these are generally more efficient than implementations using `compareExchange`. Therefore, it makes sense to expose a set of dedicated methods for common integer operations so that these will always get compiled into the most efficient implementation available. + +| Method Name | Returns | Implements | +| --- | --- | --- | +| `wrappingAdd(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a &+= b` | +| `wrappingSubtract(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a &-= b` | +| `add(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a += b` (checks for overflow) | +| `subtract(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a -= b` (checks for overflow) | +| `bitwiseAnd(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a &= b` | +| `bitwiseOr(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a \|= b` | +| `bitwiseXor(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a ^= b` | +| `min(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a = Swift.min(a, b)` | +| `max(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a = Swift.max(a, b)` | + +All operations are also marked as `@discardableResult` in the case where one doesn't care about the old value or new value. The `add` and `subtract` operations explicitly check for overflow and will trap at runtime if one occurs, except in `-Ounchecked` builds. + +While we require all atomic operations to be free of locks, we don't require wait-freedom. Therefore, on architectures that don't provide direct hardware support for some or all of these operations, we still require them to be implemented using `compareExchange` loops like the one for `wrappingAdd` above. + +`Atomic` exposes these operations when `Value` is one of the standard fixed-width integer types. + +```swift +extension Atomic where Value == Int {...} +extension Atomic where Value == UInt8 {...} +... + +let counter = Atomic(0) +counter.wrappingAdd(42, ordering: .relaxed) + +let oldMax = counter.max(82, ordering: .relaxed).oldValue +``` + +### Specialized Boolean Operations + +Similar to the specialized integer operations, we can provide similar ones for booleans: + +| Method Name | Returns | Implements | +| ---------------------------- | --------------------- | ------------ | +| `logicalAnd(_: Bool, ordering: AtomicUpdateOrdering)` | `(oldValue: Bool, newValue: Bool)` | `a = a && b` | +| `logicalOr(_: Bool, ordering: AtomicUpdateOrdering)` | `(oldValue: Bool, newValue: Bool)` | `a = a \|\| b` | +| `logicalXor(_: Bool, ordering: AtomicUpdateOrdering)` | `(oldValue: Bool, newValue: Bool)` | `a = a != b` | + +Like the integer operations, all of these boolean operations are marked as `@discardableResult`. + +`Atomic` exposes these operations when `Value` is `Bool`. + +```swift +extension Atomic where Value == Bool {...} + +let tracker = Atomic(false) +let newOr = tracker.logicalOr(true, ordering: .relaxed).newValue +``` + +### Atomic Lazy References + +The operations provided by `Atomic>` only operate on the unmanaged reference itself. They don't allow us to directly access the referenced object -- we need to manually invoke the methods `Unmanaged` provides for this purpose (usually, `takeUnretainedValue`). + +Note that loading the atomic unmanaged reference and converting it to a strong reference are two distinct operations that won't execute as a single atomic transaction. This can easily lead to race conditions when a thread releases an object while another is busy loading it: + +```swift +// BROKEN CODE. DO NOT EMULATE IN PRODUCTION. +let myAtomicRef = Atomic>(...) + +// Thread A: Load the unmanaged value and then convert it to a regular +// strong reference. +let ref = myAtomicRef.load(ordering: .acquiring).takeUnretainedValue() +... + +// Thread B: Store a new reference in the atomic unmanaged value and +// release the previous reference. +let new = Unmanaged.passRetained(...) +let old = myAtomicRef.exchange(new, ordering: .acquiringAndReleasing) +old.release() // RACE CONDITION +``` + +If thread B happens to release the same object that thread A is in the process of loading, then thread A's `takeUnretainedValue` may attempt to retain a deallocated object. + +Such problems make `Atomic>` exceedingly difficult to use in all but the simplest situations. The section on [*Atomic Strong References*](#atomic-strong-references-and-the-problem-of-memory-reclamation) below describes some new constructs we may introduce in future proposals to assist with this issue. + +For now, we provide the standalone type `AtomicLazyReference`; this is an example of a useful construct that could be built on top of `Atomic>` operations. (Of the atomic constructs introduced in this proposal, only `AtomicLazyReference` represents a regular strong reference to a class instance -- the other pointer/reference types leave memory management entirely up to the user.) + +An `AtomicLazyReference` holds an optional reference that is initially set to `nil`. The value can be set exactly once, but it can be read an arbitrary number of times. Attempts to change the value after the first `storeIfNil` call are ignored, and return the current value instead. + +```swift +/// A lazily initializable atomic strong reference. +/// +/// These values can be set (initialized) exactly once, but read many +/// times. +public struct AtomicLazyReference: ~Copyable { + /// The value logically stored in an atomic lazy reference value. + public typealias Value = Instance? + + /// Initializes a new managed atomic lazy reference with a nil value. + public init() +} + +extension AtomicLazyReference { + /// Atomically initializes this reference if its current value is nil, then + /// returns the initialized value. If this reference is already initialized, + /// then `storeIfNil(_:)` discards its supplied argument and returns + /// the current value without updating it. + /// + /// The following example demonstrates how this can be used to implement a + /// thread-safe lazily initialized reference: + /// + /// ``` + /// class Image { + /// var _histogram: AtomicLazyReference = .init() + /// + /// // This is safe to call concurrently from multiple threads. + /// var atomicLazyHistogram: Histogram { + /// if let histogram = _histogram.load() { return histogram } + /// // Note that code here may run concurrently on + /// // multiple threads, but only one of them will get to + /// // succeed setting the reference. + /// let histogram = ... + /// return _histogram.storeIfNil(histogram) + /// } + /// ``` + /// + /// This operation uses acquiring-and-releasing memory ordering. + public borrowing func storeIfNil( + _ desired: consuming Instance + ) -> Instance + + /// Atomically loads and returns the current value of this reference. + /// + /// The load operation is performed with the memory ordering + /// `AtomicLoadOrdering.acquiring`. + public borrowing func load() -> Instance? +} +``` + +This is the only atomic type in this proposal that doesn't provide the usual `load`/`store`/`exchange`/`compareExchange` operations. + +This construct allows library authors to implement a thread-safe lazy initialization pattern: + +```swift +let _foo: AtomicLazyReference = ... + +// This is safe to call concurrently from multiple threads. +nonisolated var atomicLazyFoo: Foo { + if let foo = _foo.load() { return foo } + // Note: the code here may run concurrently on multiple threads. + // All but one of the resulting values will be discarded. + let foo = Foo() + return _foo.storeIfNil(foo) +} +``` + +The Standard Library has been internally using such a pattern to implement deferred bridging for `Array`, `Dictionary` and `Set`. + +Note that unlike the rest of the atomic types, `load` and `storeIfNil(_:)` do not expose `ordering` parameters. (Internally, they map to acquiring/releasing operations to guarantee correct synchronization.) + +### Restricting Ordering Arguments to Compile-Time Constants + +Modeling orderings as regular function parameters allows us to specify them using syntax that's familiar to all Swift programmers. Unfortunately, it means that in the implementation of atomic operations we're forced to switch over the ordering argument: + +```swift +extension Atomic where Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation { + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Int) { + // Note: This is a simplified version of the actual implementation + let won: Bool + let oldValue: Value + + switch ordering { + case .relaxed: + (oldValue, won) = Builtin.cmpxchg_monotonic_monotonic_IntNN( + address, expected, desired + ) + + case .acquiring: + (oldValue, won) = Builtin.cmpxchg_acquire_acquire_IntNN( + address, expected, desired + ) + + case .releasing: + (oldValue, won) = Builtin.cmpxchg_release_monotonic_IntNN( + address, expected, desired + ) + + case .acquiringAndReleasing: + (oldValue, won) = Builtin.cmpxchg_acqrel_acquire_IntNN( + address, expected, desired + ) + + case .sequentiallyConsistent: + (oldValue, won) = Builtin.cmpxchg_seqcst_seqcst_IntNN( + address, expected, desired + ) + + default: + fatalError("Unknown atomic memory ordering") + } + + return (exchanged: won, original: oldValue) + } +} +``` + +Given our requirement that primitive atomics must always compile down to the actual atomic instructions with minimal additional overhead, we must guarantee that these switch statements always get optimized away into the single case we need; they must never actually be evaluated at runtime. + +Luckily, configuring these special functions to always get force-inlined into all callers guarantees that constant folding will get rid of the switch statement *as long as the supplied ordering is a compile-time constant*. Unfortunately, it's all too easy to accidentally violate this latter requirement, with dire consequences to the expected performance of the atomic operation. + +Consider the following well-meaning attempt at using `compareExchange` to define an atomic integer addition operation that traps on overflow rather than allowing the result to wrap around: + +```swift +extension Atomic where Value == Int { + // Non-inlinable + public func add(_ operand: Int, ordering: AtomicUpdateOrdering) { + var done = false + var current = load(ordering: .relaxed) + + while !done { + (done, current) = compareExchange( + expected: current, + desired: current + operand, // Traps on overflow + ordering: ordering + ) + } + } +} + +// Elsewhere: +counter.add(1, ordering: .relaxed) +``` + +If for whatever reason the Swift compiler isn't able (or willing) to inline the `add` call, then the value of `ordering` won't be known at compile time to the body of the function, so even though `compareExchange` will still get inlined, its switch statement won't be eliminated. This leads to a potentially significant performance regression that could interfere with the scalability of the operation. + +The big issue here is that if `add` is in another module, then callers of this function have no visibility inside this function's body. If callers can't see this function's implementation, then the switch statement will be executed at runtime regardless of the compiler optimization mode. However, another issue is that the ordering argument may still be dynamic in which case the compiler still can't eliminate the switch statement even though the caller may be able to see the entire implementation. + +To prevent the last issue, the memory ordering arguments of all atomic operations must be compile-time constants. Any attempt to pass a dynamic ordering value (such as in the `compareExchange` call above) will result in a compile-time error. + +An ordering expression will be considered constant-evaluable if it's either (1) a direct call to one of the `Atomic*Ordering` factory methods (`.relaxed`, `.acquiring`, etc.), or (2) it is a direct reference to a variable that is in turn constrained to be constant-evaluable. + +## Interaction with Existing Language Features + +Please refer to the [Clarify the Swift memory consistency model (SE-0282)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md#interaction-with-non-instantaneous-accesses) proposal which goes over how atomics interact with the Law of Exclusivity, Non-Instantaneous Accesses, and Implicit Pointer Conversions. + +An additional note with regards to the Law of Exclusivity, atomic values should never be declared with a `var` binding, always prefer a `let` one. Consider the following: + +```swift +class Counter { + var value: Atomic +} +``` + +By declaring this variable as a `var`, we opt into Swift's dynamic exclusivity checking for this property, so all non-exclusive accesses incur a runtime check to see if there is an active exclusive (e.g. mutating) access. This inherently means that atomic operations through such a variable will incur undesirable runtime overhead -- they are no longer purely atomic. (Even if the check never actually triggers a trap.) + +To prevent users from accidentally falling into this trap, `Atomic` (and `AtomicLazyReference`) will not support `var` bindings. It is a compile-time error to have a `var` that has an explicit or inferred type of `Atomic`. + +```swift +// error: variable of type 'Atomic' must be declared with a 'let' +var myAtomic = Atomic(123) +``` + +By making this a compiler error, we can safely assume that atomic accesses will never incur an unexpected dynamic exclusivity check. It is forbidden to create mutable variables of type `struct Atomic`. + +Similarly, it is an error to declare a computed property that returns an `Atomic`, as its getter would need to create and return a new instance each time it is accessed. Instead, you can return the actual value that would be the initial value for the atomic: + +```swift +var computedInt: Int { + 123 +} + +let myAtomic = Atomic(computedInt) +``` + +Alternatively, you can choose to convert the property to a function. This makes it clear that a new instance is being returned every time the function is called: + +```swift +func makeAnAtomic() -> Atomic { + Atomic(123) +} + +let myAtomic = makeAnAtomic() +``` + + + +In the same vein, these types must never be passed as `inout` parameters as that declares that the callee has exclusive access to the atomic, which would make the access no longer atomic. Attempting to create an `inout` binding for an atomic variable is also a compile-time error. Parameters that are used to pass `Atomic` values must either be `borrowing` or `consuming`. (Passing a variable as `consuming` is also an exclusive access, but it's destroying the original variable, so we no longer need to care for its atomicity.) + +```swift +// error: parameter of type 'Atomic' must be declared as either 'borrowing' or 'consuming' +func passAtomic(_: inout Atomic) +``` + +Mutating methods on atomic types are also forbidden, as they introduce `inout` bindings on `self`. For example, trying to extend `Atomic` with our own `mutating` method results in a compile-time error: + +```swift +extension Atomic { + // error: type `Atomic` cannot have mutating function 'greet()' + mutating func greet() { + print("Hello! From: Atomic") + } +} +``` + +These conditions for `Atomic` and `AtomicLazyReference` are important to prevent users from accidentally introducing dynamic exclusivity for these low-level performance sensitive concurrency primitives. + +### Interaction with Swift Concurrency + +The `Atomic` type is `Sendable` where `Value: Sendable`. One can pass a value of an `Atomic` to an actor or any other async context by using a borrow reference. + +```swift +actor Updater { + ... + + func update(_ counter: borrowing Atomic) { + ... + } + + func doOtherWork() {} +} + +func version1() async { + let counter = Atomic(0) + + // |--- There are no suspension points in this function, so + // | this atomic value will be allocated on whatever + // | thread's stack that decides to run this async function. + // | + // v + let updatedCount = counter.load(ordering: .relaxed) +} + +func version2() async { + let counter = Atomic(0) + + let updater = Updater() + await updater.update(counter) // <--------- Potential suspension point that + // uses the atomic value directly. + // The atomic value will get + // promoted to the async stack frame + // meaning it will be available until + // this async function has ended. + + // |----- Atomic value used after suspension point. + // | Because of the fact that we're passing it to a + // | suspension point, it's already been allocated on + // | the async stack frame, and accessing it later will + // | access that same resource, preserving atomicity. + // | + // v + let updatedCount = counter.load(ordering: .relaxed) +} + +func version3() async { + let counter = Atomic(0) + + let updater = Updater() + await updater.doOtherWork() // <--------- Potential suspension point that + // doesn't use the atomic value directly. + + + // |----- Atomic value used after suspension point, so it is + // | promoted to the async stack frame which makes this + // | value's lifetime persist even after the await. + // | The compiler could in theory also reorder the + // | atomic value's initialization after the suspension + // | because it isn't used before nor during meaning it + // | could be allocated on whatever thread's stack frame. + // | + // v + let updatedCount = counter.load(ordering: .relaxed) +} +``` + +Variables of type `struct Atomic` are always located at a single, stable memory location, no matter its nature (be that a stored property in a class type or a noncopyable struct, an associated value in a noncopyable enum, a local variable that got promoted to the heap through a closure capture, or any other kind of variable.) + +Considering these factors, we can safely declare that `struct Atomic` is `Sendable` whenever its value is `Sendable`. By analogue reasoning, `struct AtomicLazyReference` is declared `Sendable` whenever its instance is `Sendable`. + +## Detailed Design + +In the interest of keeping this document (relatively) short, the following API synopsis does not include API documentation, inlinable method bodies, or `@usableFromInline` declarations, and omits most attributes (`@available`, `@inlinable`, etc.). + +To allow atomic operations to compile down to their corresponding CPU instructions, most entry points listed here will be defined `@inlinable`. + +For the full API definition, please refer to the [implementation](https://github.com/apple/swift/pull/68857). + +### Atomic Memory Orderings + +```swift +public struct AtomicLoadOrdering: Equatable, Hashable, CustomStringConvertible { + public static var relaxed: Self { get } + public static var acquiring: Self { get } + public static var sequentiallyConsistent: Self { get } + + public static func ==(left: Self, right: Self) -> Bool + public func hash(into hasher: inout Hasher) + public var description: String { get } +} + +public struct AtomicStoreOrdering: Equatable, Hashable, CustomStringConvertible { + public static var relaxed: Self { get } + public static var releasing: Self { get } + public static var sequentiallyConsistent: Self { get } + + public static func ==(left: Self, right: Self) -> Bool + public func hash(into hasher: inout Hasher) + public var description: String { get } +} + +public struct AtomicUpdateOrdering: Equatable, Hashable, CustomStringConvertible { + public static var relaxed: Self { get } + public static var acquiring: Self { get } + public static var releasing: Self { get } + public static var acquiringAndReleasing: Self { get } + public static var sequentiallyConsistent: Self { get } + + public static func ==(left: Self, right: Self) -> Bool + public func hash(into hasher: inout Hasher) + public var description: String { get } +} + +public func atomicMemoryFence(ordering: AtomicUpdateOrdering) +``` + +### Atomic Protocols + +#### `AtomicRepresentable` + +```swift +public protocol AtomicRepresentable { + associatedtype AtomicRepresentation + + static func encodeAtomicRepresentation( + _ value: consuming Self + ) -> AtomicRepresentation + + static func decodeAtomicRepresentation( + _ representation: consuming AtomicRepresentation + ) -> Self +} +``` + +The requirements set up a bidirectional mapping between values of the atomic type and an associated storage representation that supplies the actual primitive atomic operations. + +Conforming types: + +```swift +extension Int: AtomicRepresentable {...} +extension Int64: AtomicRepresentable {...} +extension Int32: AtomicRepresentable {...} +extension Int16: AtomicRepresentable {...} +extension Int8: AtomicRepresentable {...} +extension UInt: AtomicRepresentable {...} +extension UInt64: AtomicRepresentable {...} +extension UInt32: AtomicRepresentable {...} +extension UInt16: AtomicRepresentable {...} +extension UInt8: AtomicRepresentable {...} + +extension Bool: AtomicRepresentable {...} + +extension Float16: AtomicRepresentable {...} +extension Float: AtomicRepresentable {...} +extension Double: AtomicRepresentable {...} + +extension WordPair: AtomicRepresentable {...} +extension Duration: AtomicRepresentable {...} + +extension Never: AtomicRepresentable {...} + +extension UnsafeRawPointer: AtomicRepresentable {...} +extension UnsafeMutableRawPointer: AtomicRepresentable {...} +extension UnsafePointer: AtomicRepresentable {...} +extension UnsafeMutablePointer: AtomicRepresentable {...} +extension Unmanaged: AtomicRepresentable {...} +extension OpaquePointer: AtomicRepresentable {...} +extension ObjectIdentifier: AtomicRepresentable {...} + +extension UnsafeBufferPointer: AtomicRepresentable {...} +extension UnsafeMutableBufferPointer: AtomicRepresentable {...} +extension UnsafeRawBufferPointer: AtomicRepresentable {...} +extension UnsafeMutableRawBufferPointer: AtomicRepresentable {...} + +extension Optional: AtomicRepresentable where Wrapped: AtomicOptionalRepresentable {...} +``` + +To support custom "atomic-representable" types, `AtomicRepresentable` also comes with default implementations for all its requirements for `RawRepresentable` types whose `RawValue` is also atomic: + +```swift +extension RawRepresentable where Self: AtomicRepresentable, RawValue: AtomicRepresentable { + // Implementations for all requirements. +} +``` + +The default implementations work by converting values to their `rawValue` form, and forwarding all atomic operations to it. + +#### `AtomicOptionalRepresentable` + +```swift +public protocol AtomicOptionalRepresentable: AtomicRepresentable { + associatedtype AtomicOptionalRepresentation + + static func encodeAtomicOptionalRepresentation( + _ value: consuming Self? + ) -> AtomicOptionalRepresentation + + static func decodeAtomicOptionalRepresentation( + _ representation: consuming AtomicOptionalRepresentation + ) -> Self? +} +``` + +Atomic `Optional` operations are currently enabled for the following `Wrapped` types: + +```swift +extension UnsafeRawPointer: AtomicOptionalRepresentable {} +extension UnsafeMutableRawPointer: AtomicOptionalRepresentable {} +extension UnsafePointer: AtomicOptionalRepresentable {} +extension UnsafeMutablePointer: AtomicOptionalRepresentable {} +extension Unmanaged: AtomicOptionalRepresentable {} +extension OpaquePointer: AtomicOptionalRepresentable {} +extension ObjectIdentifier: AtomicOptionalRepresentable {} +``` + +### `WordPair` + +```swift +public struct WordPair { + public var first: UInt { get } + public var second: UInt { get } + + public init(first: UInt, second: UInt) +} + +extension WordPair: AtomicRepresentable {...} +extension WordPair: Equatable {...} +extension WordPair: Hashable {...} + +// NOTE: WordPair is semantically a (UInt, UInt). Tuple comparability +// works based of lexicographical ordering, so WordPair will do +// the same. It will compare 'first' first, and 'second' second. +extension WordPair: Comparable {...} + +extension WordPair: CustomStringConvertible {...} +extension WordPair: CustomDebugStringConvertible {...} +extension WordPair: Sendable {} +``` + +### Atomic Types + +#### `Atomic` + +```swift +public struct Atomic: ~Copyable { + public init(_ initialValue: consuming Value) +} + +extension Atomic where Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation { + // Atomic operations: + + public borrowing func load( + ordering: AtomicLoadOrdering + ) -> Value + + public borrowing func store( + _ desired: consuming Value, + ordering: AtomicStoreOrdering + ) + + public borrowing func exchange( + _ desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> Value + + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Value) + + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + successOrdering: AtomicUpdateOrdering, + failureOrdering: AtomicLoadOrdering + ) -> (exchanged: Bool, original: Value) + + public borrowing func weakCompareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Value) + + public borrowing func weakCompareExchange( + expected: consuming Value, + desired: consuming Value, + successOrdering: AtomicUpdateOrdering, + failureOrdering: AtomicLoadOrdering + ) -> (exchanged: Bool, original: Value) +} + +extension Atomic: @unchecked Sendable where Value: Sendable {} +``` + +`Atomic` also provides a handful of integer operations for the standard fixed-width integer types. This is implemented via same type requirements: + +```swift +extension Atomic where Value == Int { + @discardableResult + public borrowing func wrappingAdd( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func wrappingSubtract( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func add( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func subtract( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func bitwiseAnd( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func bitwiseOr( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func bitwiseXor( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func min( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func max( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) +} + +extension Atomic where Value == Int8 {...} +... +``` + +as well as providing convenience functions for boolean operations: + +```swift +extension Atomic where Value == Bool { + @discardableResult + public borrowing func logicalAnd( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func logicalOr( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func logicalXor( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) +} +``` + +#### `AtomicLazyReference` + +```swift +public struct AtomicLazyReference: ~Copyable { + public typealias Value = Instance? + + public init(_ initialValue: consuming Instance) + + // Atomic operations: + + public borrowing func storeIfNil( + _ desired: consuming Instance + ) -> Instance + + public borrowing func load() -> Instance? +} + +extension AtomicLazyReference: @unchecked Sendable where Instance: Sendable {} +``` + +## Source Compatibility + +This is a purely additive change with no source compatibility impact. + +## Effect on ABI Stability + +This proposal introduces new entry points to the Standard Library ABI in a standalone `Synchronization` module, but otherwise it has no effect on ABI stability. + +On ABI-stable platforms, the struct types and protocols introduced here will become part of the stdlib's ABI with availability matching the first OS releases that include them. + +Most of the atomic methods introduced in this document will be force-inlined directly into client code at every call site. As such, there is no reason to bake them into the stdlib's ABI -- the stdlib binary will not export symbols for them. + +## Effect on API Resilience + +This is an additive change; it has no effect on the API of existing code. + +For the new constructs introduced here, the proposed design allows us to make the following changes in future versions of the Swift Standard Library: + +- Addition of new atomic types (and higher-level constructs built around them). (These new types would not directly back-deploy to OS versions that predate their introduction.) + +- Addition of new memory orderings. Because all atomic operations compile directly into user code, new memory orderings that we decide to introduce later could potentially back-deploy to any OS release that includes this proposal. + +- Addition of new atomic operations on the types introduced here. These would be also be back deployable. + +- Introducing a default memory ordering for atomic operations (either by adding a default value to `ordering`, or by adding new overloads that lack that parameter). This too would be a back-deployable change. + +- Change the memory ordering model as long as the changes preserve source compatibility. + +(We don't necessarily plan to actually perform any of these changes; we merely leave the door open to doing them.) + +## Potential Future Directions + +### Atomic Strong References and The Problem of Memory Reclamation + +Perhaps counter-intuitively, implementing a high-performance, *lock-free* atomic version of regular everyday strong references is not a trivial task. This proposal doesn't attempt to provide such a construct beyond the limited use-case of `AtomicLazyReference`. + +Under the hood, Swift's strong references have always been using atomic operations to implement reference counting. This allows references to be read (but not mutated) from multiple, concurrent threads of execution, while also ensuring that each object still gets deallocated as soon as its last outstanding reference disappears. However, atomic reference counts on their own do not allow threads to safely share a single *mutable* reference without additional synchronization. + +The difficulty is in the implementation of the atomic load operation, which boils down to two separate sub-operations, both of which need to be part of the *same atomic transaction*: + +1. Load the value of the reference. +2. Increment the reference count of the corresponding object. + +If an intervening store operation were allowed to release the reference between steps 1 and 2, then the loaded reference could already be deallocated by the time `load` tries to increment its refcount. + +Without an efficient way to implement these two steps as a single atomic transaction, the implementation of `store` needs to delay releasing the overwritten value until it can guarantee that every outstanding load operation is completed. Exactly how to implement this is the problem of *memory reclamation* in concurrent data structures. + +There are a variety of approaches to tackle this problem, but the one we think would be the best fit is the implementation of [`AtomicReference`][https://swiftpackageindex.com/apple/swift-atomics/1.2.0/documentation/atomics/atomicreference] in the [swift-atomics package](https://github.com/apple/swift-atomics). + +### Additional Low-Level Atomic Features + +To enable use cases that require even more fine-grained control over atomic operations, it may be useful to introduce additional low-level atomics features: + +* support for additional kinds of atomic values (such as floating-point atomics [[P0020]]), +* new memory orderings, such as a consuming load ordering [[P0750]] or tearable atomics [[P0690]], +* "volatile" atomics that prevent certain compiler optimizations +* and more + +We defer these for future proposals. + +## Alternatives Considered + +### Default Orderings + +We considered defaulting all atomic operations to sequentially consistent ordering. While we concede that doing so would make atomics slightly more approachable, implicit ordering values tend to interfere with highly performance-sensitive use cases of atomics (which is *most* use cases of atomics). Sequential consistency tends to be relatively rarely used in these contexts, and implicitly defaulting to it would allow accidental use to easily slip through code review. + +Users who wish for default orderings are welcome to define their own overloads for atomic operations: + +```swift +extension Atomic where Value.AtomicRepresentation == UInt8.AtomicRepresentation { + func load() -> Value { + load(ordering: .sequentiallyConsistent) + } + + func store(_ desired: consuming Value) { + store(desired, ordering: .sequentiallyConsistent) + } + + func exchange(_ desired: consuming Value) -> Value { + exchange(desired, ordering: .sequentiallyConsistent) + } + + func compareExchange( + expected: consuming Value, + desired: consuming Value + ) -> (exchanged: Bool, original: Value) { + compareExchange( + expected: expected, + desired: desired, + ordering: .sequentiallyConsistent + ) + } + + func weakCompareExchange( + expected: consuming Value, + desired: consuming Value + ) -> (exchanged: Bool, original: Value) { + weakCompareExchange( + expected: expected, + desired: desired, + successOrdering: .sequentiallyConsistent, + failureOrdering: .sequentiallyConsistent + ) + } +} + +... + +extension Atomic where Value == Int { + func wrappingAdd(_ operand: Value) { + wrappingAdd(operand, ordering: .sequentiallyConsistent) + } + + etc. +} + +... +``` + +### A Truly Universal Generic Atomic Type + +While future proposals may add a variety of other atomic types, we do not expect to ever provide a truly universal generic `Atomic` construct. The `Atomic` type is designed to provide high-performance lock-free primitives, and these are heavily constrained by the atomic instruction sets of the CPU architectures Swift targets. + +A universal `Atomic` type that can hold *any* value is unlikely to be implementable without locks, so it is outside the scope of this proposal. We may eventually consider adding such a construct in a future concurrency proposal: + +```swift +struct Serialized: ~Copyable { + private let _lock = UnfairLock() + private var _value: Value + + func withLock(_ body: (inout Value) throws -> T) rethrows -> T { + _lock.lock() + defer { _lock.unlock() } + + return try body(&_value) + } +} +``` + +### Providing a `value` Property + +Our atomic constructs are unusual because even though semantically they behave like containers holding a value, they do not provide direct access to it. Instead of exposing a getter and a setter on a handy `value` property, they expose cumbersome `load` and `store` methods. There are two reasons for this curious inconsistency: + +First, there is the obvious issue that property getter/setters have no room for an ordering parameter. + +Second, there is a deep underlying problem with the property syntax: it encourages silent race conditions. For example, consider the code below: + +```swift +let counter = Atomic(0) +... +counter.value += 1 +``` + +Even though this increment looks like it may be a single atomic operation, it gets executed as two separate atomic transactions: + +```swift +var temp = counter.value // atomic load +temp += 1 +counter.value = temp // atomic store +``` + +If some other thread happens to update the value after the atomic load, then that update gets overwritten by the subsequent store, resulting in data loss. + +To prevent this gotcha, none of the proposed atomic types provide a property for accessing their value, and we don't foresee adding such a property in the future, either. + +(Note that this problem cannot be mitigated by implementing [modify accessors]. Lock-free updates cannot be implemented without the ability to retry the update multiple times, and modify accessors can only yield once.) + +[modify accessors]: https://forums.swift.org/t/modify-accessors/31872 + +### Alternative Designs for Memory Orderings + +Modeling memory orderings with enumeration(-like) values fits well into the Standard Library's existing API design practice, but `ordering` arguments aren't without problems. Most importantly, the quality of code generation depends greatly on the compiler's ability to constant-fold switch statements over these ordering values into a single instruction. This can be fragile -- especially in unoptimized builds. We think [constraining these arguments to compile-time constants](#restricting-ordering-arguments-to-compile-time-constants) strikes a good balance between readability and performance, but it's instructive to look at some of the approaches we considered before settling on this choice. + +#### Encode Orderings in Method Names + +One obvious idea is to put the ordering values directly in the method name for every atomic operation. This would be easy to implement but it leads to practically unusable API names. Consider the two-ordering compare/exchange variant below: + +```swift +flag.sequentiallyConsistentButAcquiringAndReleasingOnFailureCompareExchange( + expected: 0, + desired: 1 +) +``` + +We could find shorter names for the orderings (`Serialized`, `Barrier` etc.), but ultimately the problem is that this approach tries to cram too much information into the method name, and the resulting multitude of similar-but-not-exactly-the-same methods become an ill-structured mess. + +#### Orderings As Generic Type Parameters + +A second idea is model the orderings as generic type parameters on the atomic types themselves. + +```swift +struct Atomic { + ... +} + +let counter = Atomic(0) +counter.wrappingAdd(1) +``` + +This simplifies the typical case where all operations on a certain atomic value use the same "level" of ordering (relaxed, acquire/release, or sequentially consistent). However, there are considerable drawbacks: + +* This design puts the ordering specification far away from the actual operations -- obfuscating their meaning. +* It makes it a lot more difficult to use custom orderings for specific operations (like the speculative relaxed load in the `wrappingAdd` example in the section on [Atomic Operations](#atomic-operations) above). +* We wouldn't be able to provide a default value for a generic type parameter. +* Finally, there is also the risk of unspecialized generics interfering with runtime performance. + +#### Ordering Views + +The most promising alternative idea to represent memory orderings was to model them like `String`'s encoding views: + +```swift +let counter = Atomic(0) + +counter.relaxed.wrappingAdd(1) + +let current = counter.acquiring.load() +``` + +There are some things that we really like about this "ordering view" approach: + +- It eliminates the need to ever switch over orderings, preventing any and all constant folding issues. +- It makes it obvious that memory orderings are supposed to be compile-time parameters. +- The syntax is arguably more attractive. + +However, we ultimately decided against going down this route, for the following reasons: + + - **Composability.** Such ordering views are unwieldy for the variant of `compareExchange` that takes separate success/failure orderings. Ordering views don't nest very well at all: + + ```swift + counter.acquiringAndReleasing.butAcquiringOnFailure.compareExchange(...) + ``` + + - **API surface area and complexity.** Ordering views act like a multiplier for API entry points. In our prototype implementation, introducing ordering views increased the API surface area of atomics by 3×: we went from 6 public structs with 53 public methods to 27 structs with 175 methods. While clever use of protocols and generics could reduce this factor, the increased complexity seems undesirable. (E.g., generic ordering views would reintroduce potential performance problems in the form of unspecialized generics.) + + API surface area is not necessarily the most important statistic, but public methods do have some cost. (In e.g. the size of the stdlib module, API documentation etc.) + + - **Unintuitive syntax.** While the syntax is indeed superficially attractive, it feels backward to put the memory ordering *before* the actual operation. While memory orderings are important, I suspect most people would consider them secondary to the operations themselves. + + - **Limited Reuse.** Implementing ordering views takes a rather large amount of (error-prone) boilerplate-heavy code that is not directly reusable. Every new atomic type would need to implement a new set of ordering views, tailor-fit to its own use-case. + +#### Memory Orderings as Overloads + +Another promising alternative was the idea to model each ordering as a separate type and have overloads for the various atomic operations. + +```swift +struct AtomicMemoryOrdering { + struct Relaxed { + static var relaxed: Self { get } + } + + struct Acquiring { + static var acquiring: Self { get } + } + + ... +} + +extension Atomic where Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation { + func load(ordering: AtomicMemoryOrdering.Relaxed) -> Value {...} + func load(ordering: AtomicMemoryOrdering.Acquiring) -> Value {...} + ... +} +``` + +This approach shares a lot of the same benefits of views, but the biggest reason for this alternative was the fact that the switch statement problem we described earlier just doesn't exist anymore. There is no switch statement! The overload always gets resolved to a single atomic operation + ordering + storage meaning there's no question about what to compile the operation down to. However, this is just another type of flavor of views in that the API surface explodes especially with double ordering operations. + +There are 5 storage types and we define the primitive atomic operations on extensions of all of these. For the constant expression case for single ordering operations that's `5 (storage) * 1 (ordering) = 5` number of overloads and `5 (storage) * 1 (update ordering) * 1 (load ordering) = 5` for the double ordering case. The overload solution is now dependent on the number of orderings supported for a specific operation. So for single ordering loads it's `5 (storage) * 3 (orderings) = 15` different load orderings and for the double ordering compare and exchange it's `5 (storage) * 5 (update orderings) * 3 (load orderings) = 75` overloads. + +| | Overloads | Constant Expressions | +| ----------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| Overload Resolution | Very bad | Not so bad | +| API Documentation | Very bad (but can be fixed!) | Not so bad (but can be fixed!) | +| Custom Atomic Operations | Requires users to define multiple overloads for their operations. | Allows users to define a single entrypoint that takes a constant ordering and passes that to the primitive atomic operations. | +| Back Deployable New Orderings | Almost impossible unless we defined the ordering types in C because types in Swift must come with availability. | Can easily be done because the orderings are static property getters that we can back deploy. | + +The same argument for views creating a very vast API surface can be said about the overloads which helped us determine that the constant expression approach is still superior. + +### Directly bring over `swift-atomics`'s API + +The `swift-atomics` package has many years of experience using their APIs to interface with atomic values and it would be beneficial to simply bring over the same API. However, once we designed the general purpose `Atomic` type, we noticied a few deficiencies with the atomic protocol hierarchy that made using this type awkward for users. We've redesigned these protocols to make using the atomic type easier to use and easier to reason about. + +While there are some API differences between this proposal and the package, most of the atomic operations are the same and should feel very familiar to those who have used the package before. We don't plan on drastically renaming any core atomic operation because we believe `swift-atomics` already got those names correct. + +### A different name for `WordPair` + +Previous revisions of this proposal named this type `DoubleWord`. This is a good name and is in fact the name used in the `swift-atomics` package. We felt the prefix `Double*` could cause confusion with the pre-existing type in the standard library `Double`. The name `WordPair` has a couple of advantages: + +1. Code completion. Because this name starts with an less common letter in the English alphabet, the likelihood of seeing this type at the top level in code completion is very unlikely and not generally a type used for newer programmers of Swift. +2. Directly conveys the semantic meaning of the type. This type is not semantically equivalent to something like `{U}Int128` (on 64 bit platforms). While although its layout shares the same size, the meaning we want to drive home with this type is quite simply that it's a pair of `UInt` words. If and when the standard library proposes a `{U}Int128` type, that will add a conformance to `AtomicRepresentable` on 64 bit platforms who support double-words as well. That itself wouldn't deprecate uses of `WordPair` however, because it's much easier to grab both words independently with `WordPair` as well as being a portable name for such semantics on both 32 bit and 64 bit platforms. + +### A different name for the `Synchronization` module + +In its [notes returning the initial version of this proposal for revision](https://forums.swift.org/t/returned-for-revision-se-0410-atomics/68522), the Swift Language Steering Group suggested the strawman name `Atomics` as a for this module. I think this name is far too restrictive because it prevents other similar low-level concurrency primitives or somewhat related features like volatile loads/stores from sharing a module. It would also be extremely source breaking for folks that upgrade their Swift SDK to a version that may include this proposed new module while depending on the existing [swift-atomics](https://github.com/apple/swift-atomics) whose module is also named `Atomics`. + +We shouldn't be afraid of conflicting module names causing spurious source breaks when introducing a new module to the standard libraries; however, in this case, the direct precursor is prominently using this name, and reusing the same module name would cause significant breakage. We expect this package will need to remain in active use for a number of years, as it will be able to provide reimplementations of the constructs proposed here without the ABI availability constraints that come with Standard Library additions. + +## References + +[Clarify the Swift memory consistency model (SE-0282)]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md +**\[Clarify the Swift memory consistency model (SE-0282)]** Karoy Lorenty. "Clarify the Swift memory consistency model."*Swift Evolution Proposal*, 2020. https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md + +[C++17]: https://isocpp.org/std/the-standard +**\[C++17]** ISO/IEC. *ISO International Standard ISO/IEC 14882:2017(E) – Programming Language C++.* 2017. + https://isocpp.org/std/the-standard + +[Boehm 2008]: https://doi.org/10.1145/1375581.1375591 +**\[Boehm 2008]** Hans-J. Boehm, Sarita V. Adve. "Foundations of the C++ Concurrency Memory Model." In *PLDI '08: Proc. of the 29th ACM SIGPLAN Conf. on Programming Language Design and Implementation*, pages 68–78, June 2008. + https://doi.org/10.1145/1375581.1375591 + +[N2153]: http://wg21.link/N2153 +**\[N2153]** Raúl Silvera, Michael Wong, Paul McKenney, Bob Blainey. *A simple and efficient memory model for weakly-ordered architectures.* WG21/N2153, January 12, 2007. http://wg21.link/N2153 + +[P0020]: http://wg21.link/P0020 +**\[P0020]** H. Carter Edwards, Hans Boehm, Olivier Giroux, JF Bastien, James Reus. *Floating Point Atomic.* WG21/P0020r6, November 10, 2017. http://wg21.link/P0020 + +[P0418]: http://wg21.link/P0418 +**\[P0418]** JF Bastien, Hans-J. Boehm. *Fail or succeed: there is no atomic lattice.* WG21/P0417r2, November 9, 2016. http://wg21.link/P0418 + +[P0690]: http://wg21.link/P0690 +**\[P0690]** JF Bastien, Billy Robert O'Neal III, Andrew Hunter. *Tearable Atomics.* WG21/P0690, February 10, 2018. http://wg21.link/P0690 + +[P0735]: http://wg21.link/P0735 +**\[P0735]**: Will Deacon, Jade Alglave. *Interaction of `memory_order_consume` with release sequences.* WG21/P0735r1, June 17, 2019. http://wg21.link/P0735 + +[P0750]: http://wg21.link/P0750 +**\[P0750]** JF Bastien, Paul E. McKinney. *Consume*. WG21/P0750, February 11, 2018. http://wg21.link/P0750 + +⚛︎︎ + + + + + + + + diff --git a/proposals/0411-isolated-default-values.md b/proposals/0411-isolated-default-values.md new file mode 100644 index 0000000000..02280a41be --- /dev/null +++ b/proposals/0411-isolated-default-values.md @@ -0,0 +1,330 @@ +# Isolated default value expressions + +* Proposal: [SE-0411](0411-isolated-default-values.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.10)** +* Bug: *if applicable* [apple/swift#58177](https://github.com/apple/swift/issues/58177) +* Implementation: [apple/swift#68794](https://github.com/apple/swift/pull/68794) +* Upcoming Feature Flag: `IsolatedDefaultValues` +* Review: ([pitch](https://forums.swift.org/t/pitch-isolated-default-value-expressions/67714)), ([review](https://forums.swift.org/t/se-0411/68065)), ([acceptance](https://forums.swift.org/t/accepted-se-0411-isolated-default-value-expressions/68806)) + +## Introduction + +Default value expressions are permitted for default arguments and default stored property values. There are several issues with the current actor isolation rules for default value expressions: the rules for stored properties admit data races, the rules for default argument values are overly restrictive, and the rules between the different places you can use default value expressions are inconsistent with each other, making the actor isolation model harder to understand. This proposal unifies the actor isolation rules for default value expressions, eliminates data races, and improves expressivity by safely allowing isolation for default values. + +## Motivation + +The current actor isolation rules for initial values of stored properties admit data races. For example, the following code is currently valid: + +```swift +@MainActor func requiresMainActor() -> Int { ... } +@AnotherActor func requiresAnotherActor() -> Int { ... } + +class C { + @MainActor var x1 = requiresMainActor() + @AnotherActor var x2 = requiresAnotherActor() + + nonisolated init() {} // okay??? +} +``` + +The above code allows any context to initialize an instance of `C()` through a synchronous, nonisolated `init`. The initializer synchronously calls both `requiresMainActor()` and `requiresAnotherActor()`, which are `@MainActor`-isolated and `@AnotherActor`-isolated, respectively. This violates actor isolation checking because `requiresMainActor()` and `requiresAnotherActor()` may run concurrently with other code on their respective global actors. + +The current actor isolation rules for default argument values do not admit data races, but default argument values are always `nonisolated` which is overly restrictive. This rule prohibits programmers from making `@MainActor`-isolated calls in default argument values of `@MainActor`-isolated functions. For example, the following code is not valid even though it is perfectly safe: + +```swift +@MainActor class C {} + +@MainActor func f(c: C = C()) {} // error: Call to main actor-isolated initializer 'init()' in a synchronous nonisolated context + +@MainActor func useFromMainActor() { + f() +} +``` + +## Proposed solution + +I propose allowing default value expressions to have the same isolation as the enclosing function or the corresponding stored property. As usual, if the caller is not already in the isolation domain of the callee, then the call must be made asynchronously and must be explicitly marked with `await`. For isolated default values of stored properties, the implicit initialization only happens in the body of an `init` with the same isolation. + +These rules make the stored property example above invalid at the `nonisolated` initializer: + +```swift +@MainActor func requiresMainActor() -> Int { ... } +@AnotherActor func requiresAnotherActor() -> Int { ... } + +class C { + @MainActor var x1 = requiresMainActor() + @AnotherActor var x2 = requiresAnotherActor() + + nonisolated init() {} // error: 'self.x1' and 'self.x2' are not initialized +} +``` + +Calling `requiresMainActor()` and `requiresAnotherActor()` explicitly with `await` resolves the issue: + +```swift +class C { + @MainActor var x1 = requiresMainActor() + @AnotherActor var x2 = requiresAnotherActor() + + nonisolated init() async { + self.x1 = await requiresMainActor() + self.x2 = await requiresAnotherActor() + } +} +``` + +This rule also makes the default argument example above valid, because the default argument and the enclosing function are both `@MainActor`-isolated. + +## Detailed design + +### Inference of default value isolation requirements + +Default value expressions are always evaluated in a synchronous context, so all calls that are made during the evaluation of the expression must also be synchronous. If the callee is isolated, then the default value expression must already be in the same isolation domain in order to make the call synchronously. So, for a given default value expression, the inferred isolation is the required isolation of its subexpressions. For example: + +```swift +@MainActor func requiresMainActor() -> Int { ... } + +@MainActor func useDefault(value: Int = requiresMainActor()) { ... } +``` + +In the above code, the default argument for `value` requires `@MainActor` isolation, because the default value calls `requiresMainActor()` which is isolated to `@MainActor`. + +#### Closures + +Evaluating a closure literal itself can happen in any isolation domain; the actor isolation of a closure only applies when calling the closure. An actor-isolated closure enables the closure body to make calls within that isolation domain synchronously. For a closure literal in a default value expression that is not explicitly annotated with actor isolation, the inferred isolation of the closure is the union of the isolation of all callees in the closure body for synchronous calls. For example: + +```swift +@MainActor func requiresMainActor() -> Int { ... } + +@MainActor func useDefaultClosure( + closure: () -> Void = { + requiresMainActor() + } +) {} +``` + +The above `useDefaultClosure` function has a default argument value that is a closure literal. The closure body calls a `@MainActor`-isolated function synchronously, therefore the closure itself must be `@MainActor` isolated. + +Note that the only way for a closure literal in a default argument to be isolated to an actor instance is for the isolation to be written explicitly with an isolated parameter. The inference algorithm will never determine the isolation to be an actor instance based on the following two properties: + +1. To be isolated to an actor instance, a closure must either have its own (explicit) isolated parameter or capture an isolated parameter from its enclosing context. +2. Closure literals in default arguments cannot capture values. + +#### Restrictions + +* If a function or type itself has actor isolation, the required isolation of its default value expressions must share the same actor isolation. For example, a `@MainActor`-isolated function cannot have a default argument that is isolated to `@AnotherActor`. Note that it's always okay to mix isolated default values with `nonisolated` default values. +* If a function or type is `nonisolated`, then the required isolation of its default value expressions must be `nonisolated`. + +### Enforcing default value isolation requirements + +#### Default argument values + +Isolation requirements for default argument expressions are enforced at the caller. If the caller is not in the required isolation domain, the default arguments must be evaluated asynchronously and explicitly marked with `await`. For example: + +```swift +@MainActor func requiresMainActor() -> Int { ... } + +@MainActor func useDefault(value: Int = requiresMainActor()) { ... } + +@MainActor func mainActorCaller() { + useDefault() // okay +} + +func nonisolatedCaller() async { + await useDefault() // okay + + useDefault() // error: call is implicitly async and must be marked with 'await' +} +``` + +In the above example, `useDefault` has default arguments that are isolated to `@MainActor`. The default arguments can be evaluated synchronously from a `@MainActor`-isolated caller, but the call must be marked with `await` from outside the `@MainActor`. Note that these rules already fall out of the semantics of calling actor isolated functions. + +#### Argument evaluation + +For a given call, argument evaluation happens in the following order: + +1. Left-to-right evaluation of explicit r-value arguments +2. Left-to-right evaluation of default arguments and formal access arguments + +For example: + +```swift +nonisolated var defaultVal: Int { print("defaultVal"); return 0 } +nonisolated var explicitVal: Int { print("explicitVal"); return 0 } +nonisolated var explicitFormalVal: Int { + get { print("explicitFormalVal"); return 0 } + set {} +} + +func evaluate(x: Int = defaultVal, y: Int = defaultVal, z: inout Int) {} + +evaluate(y: explicitVal, z: &explicitFormalVal) +``` + +The output of the above program is + +``` +explicitVal +defaultVal +explicitFormalVal +``` + +Unlike the explicit argument list, isolated default arguments must be evaluated in the isolation domain of the callee. As such, if any of the argument values require the isolation of the callee, argument evaluation happens in the following order: + +1. Left-to-right evaluation of explicit r-value arguments +2. Left-to-right evaluation of formal access arguments +3. Hop to the callee's isolation domain +4. Left-to-right evaluation of default arguments + +For example: + +```swift +@MainActor var defaultVal: Int { print("defaultVal"); return 0 } +nonisolated var explicitVal: Int { print("explicitVal"); return 0 } +nonisolated var explicitFormalVal: Int { + get { print("explicitFormalVal"); return 0 } + set {} +} + +@MainActor func evaluate(x: Int = defaultVal, y: Int = defaultVal, z: inout Int) {} + +nonisolated func nonisolatedCaller() { + await evaluate(y: explicitVal, z: &explicitFormalVal) +} +``` + +The output of calling `nonisolatedCaller()` is: + +``` +explicitVal +explicitFormalVal +defaultVal +``` + +#### Stored property initial values + +Isolation requirements for default initializer expressions for stored properties apply in the body of initializers. If an `init` does not match the isolation of the initializer expression, the initialization of that stored property is not emitted at the beginning of the `init`. Instead, the stored property must be explicitly initialized in the body of the `init`. For example: + +```swift +@MainActor func requiresMainActor() -> Int { ... } +@AnotherActor func requiresAnotherActor() -> Int { ... } + +class C { + @MainActor var x1: Int = requiresMainActor() + @AnotherActor var x2: Int = requiresAnotherActor() + + nonisolated init() {} // error: 'self.x1' and 'self.x2' aren't initialized + + nonisolated init(x1: Int, x2: Int) { // okay + self.x1 = x1 + self.x2 = x2 + } + + @MainActor init(x2: Int) { // okay + // 'self.x1' gets assigned to the default value 'requiresMainActor()' + self.x2 = x2 + } +} +``` + +In the above example, the no-parameter `nonisolated init()` is invalid, because it does not initialize `self.x1` and `self.x2`. Because the default initializer expressions require different actor isolation, those values are not used in the `nonisolated` initializer. The other two initializers are valid. + +### Stored property isolation in initializers + +#### Initializing isolated stored properties from across isolation boundaries + +It is invalid to initialize an isolated stored property from across isolation boundaries: + +```swift +class NonSendable {} + +class C { + @MainActor var ns: NonSendable + + init(ns: NonSendable) { + self.ns = ns // error: passing non-Sendable value 'ns' to a MainActor-isolated context. + } +} +``` + +The above code violates `Sendable` guarantees because the initialization of the `MainActor`-isolated property `self.ns` from a `nonisolated` context is effectively passing a non-`Sendable` value across isolation boundaries. To prevent this class of data races, this proposal requires that any `init` that initializes a global actor isolated stored property must also be isolated to that global actor. + +Note that this rule is not specific to default values, but it's necessary to specify the behavior of default values in compiler-synthesized initializers. + +#### Default value isolation in synthesized initializers + +For structs, default initializer expressions for stored properties are used as default argument values to the compiler-generated memberwise initializer. For structs and classes that have a compiler-generated no-parameter initializer, the default initializer expressions are also used in the synthesized `init()` body. + +If any of the type's stored properties with non-`Sendable` type are actor isolated, or if any of the isolated default initializer expressions are actor isolated, then the compiler-synthesized initializer(s) must also be actor isolated. For example: + +```swift +class NonSendable {} + +@MainActor struct MyModel { + // @MainActor inferred from annotation on enclosing struct + var value: NonSendable = .init() + + /* compiler-synthesized memberwise init is @MainActor + @MainActor + init(value: NonSendable = .init()) { + self.value = value + } + */ +} +``` + +If none of the type's stored properties are non-`Sendable` and actor isolated, and none of the default initializer expressions require actor isolation, then the compiler-synthesized initializer is `nonisolated`. For example: + +```swift +@MainActor struct MyView { + // @MainActor inferred from annotation on enclosing struct + var value: Int = 0 + + /* compiler-synthesized 'init's are 'nonisolated' + + nonisolated init() { + self.value = 0 + } + + nonisolated init(value: Int = 0) { + self.value = value + } + */ + + // @MainActor inferred from the annotation on the enclosing struct + var body: some View { ... } +} +``` + +These rules ensure that the default value expressions in compiler-synthesized initializers are always valid. If a default value expression requires actor isolation, then the enclosing initializer always shares the same actor isolation. It is an error for two different default values to require different actor isolation, because it's not possible to ever use those default values. Initializing an instance of a type using two different initial value expressions with different actor isolation must be done in an `async` initializer, with suspension points explicitly marked with `await`. + +## Source compatibility + +The actor isolation rules for initial values of stored properties are stricter than what is currently accepted in Swift 5 mode in order to eliminate data races. The isolation rules for stored properties will be staged in under the `IsolatedDefaultValues` upcoming feature identifier. + +## ABI compatibility + +This is a change to actor isolation checking with no impact on ABI. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. + +## Alternatives considered + +### Remove isolation from all default initializer expressions + +SE-0327 originally proposed changing default initializer expressions for stored properties to always be `nonisolated`, matching the current default argument value rules. However, this change was implemented and later reverted because it impacted a lot of code that followed a common pattern: a `@MainActor`-isolated type with stored properties that have default values that call the initializers of other `@MainActor`-isolated types. In some cases, it's possible to make the initializer of a `@MainActor` type `nonisolated`, but many of these cases do access `@MainActor`-isolated properties and functions in the body of the initializer. + +## Acknowledgments + +Thank you to Kavon Farvardin for implementing the default initializer expression rules originally proposed by SE-0327 and discovering the usability issues outlined in this proposal. Thank you to John McCall for the observation that memberwise initializers can and should be `nonisolated` when possible. + +## Revision history + +* Changes from the first pitch + * Require that isolated default arguments share the same isolation as their enclosing function or type. + * Specify the semantic restrictions on initializing actor isolated properties from across isolation boundaries. + * Enable using isolated default arguments from across isolation boundaries by changing the argument evaluation between formal access and default arguments. diff --git a/proposals/0412-strict-concurrency-for-global-variables.md b/proposals/0412-strict-concurrency-for-global-variables.md new file mode 100644 index 0000000000..d045827598 --- /dev/null +++ b/proposals/0412-strict-concurrency-for-global-variables.md @@ -0,0 +1,109 @@ +# Strict concurrency for global variables + +* Proposal: [SE-0412](0412-strict-concurrency-for-global-variables.md) +* Authors: [John McCall](https://github.com/rjmccall), [Sophia Poirier](https://github.com/sophiapoirier) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.10)** +* Upcoming Feature Flag: `GlobalConcurrency` (Enabled in Swift 6 language mode) +* Implementation: On `main` gated behind `-enable-experimental-feature GlobalConcurrency` +* Previous Proposals: [SE-0302](0302-concurrent-value-and-concurrent-closures.md), [SE-0306](0306-actors.md), [SE-0316](0316-global-actors.md), [SE-0337](0337-support-incremental-migration-to-concurrency-checking.md), [SE-0343](0343-top-level-concurrency.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-strict-concurrency-for-global-variables/66908)), ([review](https://forums.swift.org/t/se-0412-strict-concurrency-for-global-variables/68352)), ([acceptance](https://forums.swift.org/t/accepted-se-0412-strict-concurrency-for-global-variables/69004)) + +## Introduction + +This proposal defines options for the usage of global variables free of data races. Within this proposal, global variables encompass any storage of static duration: `let`s and stored `var`s that are either declared at global scope or as static member variables. + +## Motivation + +Global state poses a challenge within concurrency because it is memory that can be accessed from any program context. Global variables are of particular concern in data isolation checking because they defy other attempts to enforce isolation. Variables that are local and un-captured can only be accessed from that local context, which implicitly isolates them. Stored properties of value types are already isolated by the exclusivity rules. Stored properties of reference types can be isolated by isolating their containing object with sendability enforcement or using actor restrictions. But global variables can be accessed from anywhere, so these tools do not work. + +```swift +var value = 1 + +func f() { + value = 2 // warning: reference to var 'value' is not concurrency-safe because it involves shared mutable state +} +``` + +## Proposed solution + +Under strict concurrency checking, require every global variable to either be isolated to a global actor or be both: + +1. immutable +2. of `Sendable` type + +Global variables that are immutable and `Sendable` can be safely accessed from any context, and otherwise, isolation is required. + +Top-level global variables are already implicitly isolated to `@MainActor` and therefore automatically meet these proposed requirements. + +## Detailed design + +These requirements can be enforced in the type checker at declaration time. + +Although global variables are lazily initialized, the initialization is already guaranteed to be thread-safe and therefore requires no further specification under strict concurrency checking. + +There may be need in some circumstances to opt out of static checking to enable the developer to rely upon their own data isolation management, such as with an associated global lock serializing data access. The attribute `nonisolated(unsafe)` can be used to annotate the global variable (or any form of storage). Though this will disable static checking of data isolation for the global variable, note that without correct implementation of a synchronization mechanism to achieve data isolation, dynamic run-time analysis from exclusivity enforcement or tools such as Thread Sanitizer could still identify failures. + +```swift +nonisolated(unsafe) var global: String +``` + +The same annotation on a local variable can be used to suppress a static diagnostic being generated when the local variable is referenced asynchronously: + +```swift +func f() async { + nonisolated(unsafe) var value = 1 + let task = Task { + value = 2 + return value + } + print(await task.value) +} +``` + +Because `nonisolated` is a contextual keyword, there is ambiguity when using `nonisolated(unsafe)` on a separate line immediately preceding a top-level variable declaration in script mode as it could also be the invocation of a function named `nonisolated` with argument `unsafe`. This ambiguity can be resolved by favoring the interpretation of `nonisolated` as a keyword if it has a single unlabeled argument of `unsafe` and precedes a variable declaration. + +Importing a module via `@preconcurrency import` suppresses any potential errors resulting from data isolation checking of imported global variables that lack explicit concurrency annotations. Any use of a `@preconcurrency import`ed concurrency-unsafe global variable will produce a warning at the use site. + +Note that imports from other languages are implicitly `@preconcurrency`. There remain tools for enforcing safety for imported global variables from other languages, such as isolating to a global actor using for example `__attribute__((swift_attr("@MainActor")))` in C or Obj-C, or wrapping access within a safer API that declares the correct isolation or locks appropriately. + +## Source compatibility + +Due to the addition of restrictions, this could require changes to some type declaration when strict concurrency checking is in use. Such source changes however would still be backwards compatible to any version of Swift with concurrency features. + +Resolving the ambiguity of `nonisolated(unsafe)` in a top-level variable declaration would break existing top-level script code that invokes a function named `nonisolated` with a single unlabeled argument `unsafe` when immediately preceding a variable declaration by eliminating that function invocation in favor of its interpretation as an isolation specification. + +## ABI compatibility + +This proposal does not add or affect ABI in and of itself, however type declaration changes that it may instigate upon an adopting project could impact that project's ABI. + +## Implications on adoption + +Some global variable types may need to be modified in a project adopting strict concurrency checking. + +## Alternatives considered + +For isolation, rather than requiring a global actor, we could implicitly lock around accesses of the variable. While providing memory safety, this can be problematic for thread safety, because developers can easily write non-atomic use patterns: + +```swift +// value of global may concurrently change between +// the read for the multiplication expression +// and the write for the assignment +global = global * 2 +``` + +Though we could consider implicit locking if we needed to do something source-compatible in old language modes, generally our approach has just been to say that old language modes are concurrency-unsafe. It also would not work for non-`Sendable` types unless we force the value to remain isolated while accessing it. We potentially could accomplish that with the proposed [Safely sending non-Sendable values across isolation domains](https://forums.swift.org/t/pitch-safely-sending-non-sendable-values-across-isolation-domains/66566) feature, but that is probably too advanced a feature to push as a solution for such a basic problem. + +We could default all global variables that require isolation to `@MainActor`. It is arguably better to make developers think about the choice (e.g. perhaps it should just be a `let` constant). + +Access control is theoretically useful here: for example, we could know that a global variable is concurrency-safe because it is private to a file and all of the accesses in that file are from within a single global actor context, or because it is never mutated. That is a more global analysis than we usually want to do in the compiler, though; we would have to check everything in the context, and then it might be hard for the developer to understand why it works. + +## Future directions + +We do not necessarily need to require isolation to a global actor to be _explicit_; there is room for inferring the right global actor. A global mutable variable of global-actor-constrained type could be inferred to be constrained to that global actor (though unnecessary if the variable is immutable, since global-actor-constrained class types are `Sendable`). + +## Revision history + +Post-review changes: +* removed implicit `nonisolated(unsafe)` import of C global variables in favor of `@preconcurrency import` as the mechanism to suppress static isolation checking of global variables +* clarifed `nonisolated(unsafe)` for local variables diff --git a/proposals/0413-typed-throws.md b/proposals/0413-typed-throws.md new file mode 100644 index 0000000000..9ee9502297 --- /dev/null +++ b/proposals/0413-typed-throws.md @@ -0,0 +1,1401 @@ +# Typed throws + +* Proposal: [SE-0413](0413-typed-throws.md) +* Authors: [Jorge Revuelta (@minuscorp)](https://github.com/minuscorp), [Torsten Lehmann](https://github.com/torstenlehmann), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.0)** +* Review: [latest pitch](https://forums.swift.org/t/pitch-n-1-typed-throws/67496), [review](https://forums.swift.org/t/se-0413-typed-throws/68507), [acceptance](https://forums.swift.org/t/accepted-se-0413-typed-throws/69099) + +## Introduction + +Swift's error handling model allows functions and closures marked `throws` to note that they can exit by throwing an error. The error values themselves are always type-erased to `any Error`. This approach encourages errors to be handled generically, and remains a good default for most code. However, there are some places where the type erasure is unfortunate, because it doesn't allow for more precise error typing in narrow places where it is possible and desirable to handle all errors, or where the costs of type erasure are prohibitive. + +This proposal introduces the ability to specify that functions and closures only throw errors of a particular concrete type. + +> Note: the [originally accepted version](https://github.com/swiftlang/swift-evolution/blob/821970ae986219f88eb3f950ed787a55ce31d512/proposals/0413-typed-throws.md) of this proposal included type inference changes intended for Swift 6.0 that were behind the upcoming feature flag `FullTypedThrows`. These type inference changes did not get implemented in Swift 6.0, and have therefore been removed from this proposal and placed into "Future Directions" so they can be revisited once implemented. + +## Table of Contents + +[Typed throws](#typed-throws) + + * [Introduction](#introduction) + * [Motivation](#motivation) + * [Communicates less error information than Result or Task](#communicates-less-error-information-than-result-or-task) + * [Inability to interconvert throws with Result or Task](#inability-to-interconvert-throws-with-result-or-task) + * [Approach 1: Chaining Results](#approach-1-chaining-results) + * [Approach 2: Unwrap/switch/wrap on every chaining/mapping point](#approach-2-unwrapswitchwrap-on-every-chainingmapping-point) + * [Existential error types incur overhead](#existential-error-types-incur-overhead) + * [Proposed solution](#proposed-solution) + * [Specific types in catch blocks](#specific-types-in-catch-blocks) + * [Throwing any Error or Never](#throwing-any-error-or-never) + * [An alternative to rethrows](#an-alternative-to-rethrows) + * [When to use typed throws](#when-to-use-typed-throws) + * [Detailed design](#detailed-design) + * [Syntax adjustments](#syntax-adjustments) + * [Function type](#function-type) + * [Closure expression](#closure-expression) + * [Function, initializer, and accessor declarations](#function-initializer-and-accessor-declarations) + * [Examples](#examples) + * [Throwing and catching with typed throws](#throwing-and-catching-with-typed-throws) + * [Throwing within a function that declares a typed error](#throwing-within-a-function-that-declares-a-typed-error) + * [Catching typed thrown errors](#catching-typed-thrown-errors) + * [rethrows](#rethrows) + * [Opaque thrown error types](#opaque-thrown-error-types) + * [async let](#async-let) + * [Subtyping rules](#subtyping-rules) + * [Function conversions](#function-conversions) + * [Protocol conformance](#protocol-conformance) + * [Override checking](#override-checking) + * [Type inference](#type-inference) + * [Associated type inference](#associated-type-inference) + * [Standard library adoption](#standard-library-adoption) + * [Converting between throws and Result](#converting-between-throws-and-result) + * [Standard library operations that rethrow](#standard-library-operations-that-rethrow) + * [Source compatibility](#source-compatibility) + * [Effect on API resilience](#effect-on-api-resilience) + * [Effect on ABI stability](#effect-on-abi-stability) + * [Future directions](#future-directions) + * [Closure thrown type inference](#closure-thrown-type-inference) + * [Standard library operations that rethrow](#standard-library-operations-that-rethrow) + * [Concurrency library adoption](#concurrency-library-adoption) + * [Specific thrown error types for distributed actors](#specific-thrown-error-types-for-distributed-actors) + * [Alternatives considered](#alternatives-considered) + * [Thrown error type syntax](#thrown-error-type-syntax) + * [Multiple thrown error types](#multiple-thrown-error-types) + * [Treat all uninhabited thrown error types as nonthrowing](#treat-all-uninhabited-thrown-error-types-as-nonthrowing) + * [Typed rethrows](#typed-rethrows) + * [Revision history](#revision-history) + +## Motivation + +Swift is known for being explicit about semantics and using types to communicate constraints that apply to specific APIs. From that perspective, the fact that all thrown errors are of type `any Error` feels like an outlier. However, it reflects the view laid out in the original [error handling rationale](https://github.com/apple/swift/blob/main/docs/ErrorHandlingRationale.md) that errors are generally propagated and rendered, but rarely handled exhaustively, and are prone to changing over time in a way that types are not. + +The desire to provide specific thrown error types has come up repeatedly on the Swift forums. Here are just a few of the forum threads calling for some form of typed throws: + +* [[Pitch N+1] Typed throws](https://forums.swift.org/t/pitch-n-1-typed-throws/67496) +* [Typed throw functions](https://forums.swift.org/t/typed-throw-functions/38860) +* [Status check: typed throws](https://forums.swift.org/t/status-check-typed-throws/66637) +* [Precise error typing in Swift](https://forums.swift.org/t/precise-error-typing-in-swift/52045) +* [Typed throws](https://forums.swift.org/t/typed-throws/6501) +* [[Pitch\] Typed throws](https://forums.swift.org/t/pitch-typed-throws/5233) +* [Type-annotated throws](https://forums.swift.org/t/type-annotated-throws/3875) +* [Proposal: Allow Type Annotations on Throws](https://forums.swift.org/t/proposal-allow-type-annotations-on-throws/1149) +* [Proposal: Allow Type Annotations on Throws](https://forums.swift.org/t/proposal-allow-type-annotations-on-throws/623) +* [Proposal: Typed throws](https://forums.swift.org/t/proposal-typed-throws/268) +* [Type Inferencing For Error Handling (try catch blocks)](https://forums.swift.org/t/type-inferencing-for-error-handling-try-catch-blocks/117) + +In a sense, Swift started down the path toward typed throws with the introduction of the [`Result`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0235-add-result.md) type in the standard library, which captured a specific thrown error type in its `Failure` parameter. That pattern was replicated in the [`Task` type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md) and other concurrency APIs. The loss of information between types like `Result` and `Task` and the language's error-handling system provides partial motivation for the introduction of typed throws, and is discussed further below. + +Typed throws also provides benefits in places where clients need to exhaustively handle errors. For this to make sense, the set of potential failure conditions must be relatively fixed, either because they come from the same module or package as the clients, or because they come from a library that is effectively standalone and unlikely to evolve to (e.g.) pass through an error from another lower-level library. Typed throws also provides benefits in generic code that will propagate errors from its arguments, but never generate errors itself, as a more flexible alternative to the existing `rethrows`. Finally, typed throws also open up the potential for more efficient code, because they avoid the overhead associated with existential types (`any Error`). + +Even with the introduction of typed throws into Swift, the existing (untyped) `throws` remains the better default error-handling mechanism for most Swift code. The section ["When to use typed throws"](#when-to-use-typed-throws) describes the circumstances in which typed throws should be used. + +### Communicates less error information than `Result` or `Task` + +Assume you have this Error type + +```swift +enum CatError: Error { + case sleeps + case sitsAtATree +} +``` + +Compare + +```swift +func callCat() -> Result +``` + +or + +```swift +func callFutureCat() -> Task +``` + +with + +```swift +func callCatOrThrow() throws -> Cat +``` + +`throws` communicates less information about why the cat is not about to come to you. + +### Inability to interconvert `throws` with `Result` or `Task` + +The fact that`throws` carries less information than `Result` or `Task` means that conversions to `throws` loses type information, which can only be recovered by explicit casting: + +```swift +func callAndFeedCat1() -> Result { + do { + return Result.success(try callCatOrThrow()) + } catch { + // won't compile, because error type guarantee is missing in the first place + return Result.failure(error) + } +} +``` + +```swift +func callAndFeedCat2() -> Result { + do { + return Result.success(try callCatOrThrow()) + } catch let error as CatError { + // compiles + return Result.failure(error) + } catch { + // won't compile, because exhaustiveness can't be checked by the compiler + // so what should we return here? + return Result.failure(error) + } +} +``` + +### `Result` is not the go to replacement for `throws` in imperative languages + +Using explicit errors with `Result` has major implications for a code base. Because the exception handling mechanism ("goto catch") is not built into the language (like `throws`), you need to do that on your own, mixing the exception handling mechanism with domain logic. + +#### Approach 1: Chaining Results + +If you use `Result` in a functional (i.e. monadic) way, you need extensive use of `map`, `flatMap` and similar operators. + +Example is taken from [Question/Idea: Improving explicit error handling in Swift (with enum operations) - Using Swift - Swift Forums](https://forums.swift.org/t/question-idea-improving-explicit-error-handling-in-swift-with-enum-operations/35335). + +```swift +struct SimpleError: Error { + let message: String +} + +struct User { + let firstName: String + let lastName: String +} + +func stringResultFromArray(_ array: [String], at index: Int, errorMessage: String) -> Result { + guard array.indices.contains(index) else { return Result.failure(SimpleError(message: errorMessage)) } + return Result.success(array[index]) +} + +func userResultFromStrings(strings: [String]) -> Result { + return stringResultFromArray(strings, at: 0, errorMessage: "Missing first name") + .flatMap { firstName in + stringResultFromArray(strings, at: 1, errorMessage: "Missing last name") + .flatMap { lastName in + return Result.success(User(firstName: firstName, lastName: lastName)) + } + } +} +``` + +That's the functional way of writing exceptions, but Swift does not provide enough functional constructs to handle that comfortably (compare with [Haskell/do notation](https://en.wikibooks.org/wiki/Haskell/do_notation)). + +#### Approach 2: Unwrap/switch/wrap on every chaining/mapping point + +We can also just unwrap every result by switching over it and wrapping the value or error into a result again. + +```swift +func userResultFromStrings(strings: [String]) -> Result { + let firstNameResult = stringResultFromArray(strings, at: 0, errorMessage: "Missing first name") + + switch firstNameResult { + case .success(let firstName): + let lastNameResult = stringResultFromArray(strings, at: 1, errorMessage: "Missing last name") + + switch lastNameResult { + case .success(let lastName): + return Result.success(User(firstName: firstName, lastName: lastName)) + case .failure(let simpleError): + return Result.failure(simpleError) + } + + case .failure(let simpleError): + return Result.failure(simpleError) + } +} +``` + +This is even more boilerplate than the first approach, because now we are writing the implementation of the `flatMap` operator over and over again. + +### Existential error types incur overhead + +Untyped errors have the existential type `any Error`, which incurs some [necessary overhead](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md), in code size, heap allocation overhead, and execution performance, due to the need to support values of unknown type. In constrained environments such as those supported by [Embedded Swift](https://forums.swift.org/t/embedded-swift/67057), existential types may not be permitted due to these overheads, making the existing untyped throws mechanism unusable in those environments. + + +## Proposed solution + +In general, we want to add the possibility of using `throws` with a single, specific error type. + +```swift +func callCat() throws(CatError) -> Cat { + if Int.random(in: 0..<24) < 20 { + throw .sleeps + } + // ... +} +``` + +The function can only throw instances of `CatError`. This provides contextual type information for all throw sites, so we can write `.sleeps` instead of the more verbose `CatError.sleeps` that's needed with untyped throws. Any attempt to throw any other kind of error out of the function will be an error: + +```swift +func callCatBadly() throws(CatError) -> Cat { + throw SimpleError(message: "sleeping") // error: SimpleError cannot be converted to CatError +} +``` + +Maintaining specific error types throughout a function is much easier than when using `Result`, because one can use `try` consistently: + +```swift +func stringFromArray(_ array: [String], at index: Int, errorMessage: String) throws(SimpleError) -> String { + guard array.indices.contains(index) else { throw SimpleError(message: errorMessage) } + return array[index] +} + +func userResultFromStrings(strings: [String]) throws(SimpleError) -> User { + let firstName = try stringFromArray(strings, at: 0, errorMessage: "Missing first name") + let lastName = try stringFromArray(strings, at: 1, errorMessage: "Missing last name") + return User(firstName: firstName, lastName: lastName) +} +``` + +The error handling mechanism is pushed aside and you can see the domain logic more clearly. + +### Specific types in catch blocks + +With typed throws, a throwing function contains the same information about the error type as `Result`, making it easier to convert between the two: + +```swift +func callAndFeedCat1() -> Result { + do { + return Result.success(try callCat()) + } catch { + // would compile now, because error is `CatError` + return Result.failure(error) + } +} +``` + +Note that the implicit `error` variable within the catch block is inferred to the concrete type `CatError`; there is no need for the existential `any Error`. + +When a `do` statement can throw errors with different concrete types, or involves any calls to functions using untyped throws, the `catch` block will receive a thrown error type of an `any Error` type: + +```swift +func callKids() throws(KidError) -> [Kid] { ... } + +do { + try callCat() + try callKids() +} catch { + // error has type 'any Error', as it does today +} +``` + +The caught error type for a `do..catch` statement will be inferred from the various throwing sites within the body of the `do` block. One can explicitly specify this type with a `throws` clause on ` do` block itself, i.e., + +```swift +do throws(CatError) { + if isDaylight && foodBowl.isEmpty { + throw .sleeps // equivalent to CatError.sleeps + } + try callCat() +} catch let myError { + // myError is of type CatError +} +``` + +When one needs to translate errors of one concrete type to another, use a `do...catch` block around each sequence of calls that produce the same kind of error : + +```swift +func firstNameResultFromArray(_ array: [String]) throws(FirstNameError) -> String { + guard array.indices.contains(0) else { throw FirstNameError() } + return array[0] +} + +func userResultFromStrings(strings: [String]) throws(SimpleError) -> User { + do { + let firstName = try firstNameResultFromArray(strings) + return User(firstName: firstName, lastName: "") + } catch { + // error is a `FirstNameError`, map it to a `SimpleError`. + throw SimpleError(message: "Missing first name") + } +} +``` + +### Throwing `any Error` or `Never` + +Typed throws generalizes over both untyped throws and non-throwing functions. A function specified with `any Error` as its thrown type: + +```swift +func throwsAnything() throws(any Error) { ... } +``` + +is equivalent to untyped throws: + +```swift +func throwsAnything() throws { ... } +``` + +Similarly, a function specified with `Never` as its thrown type: + +```swift +func throwsNothing() throws(Never) { ... } +``` + +is equivalent to a non-throwing function: + +```swift +func throwsNothing() { } +``` + +There is a more general subtyping rule here that says that you can loosen the thrown type, i.e., converting a non-throwing function to a throwing one, or a function that throws a concrete type to one that throws `any Error`. + +### An alternative to `rethrows` + +The ability to throw a generic error parameter that might be `Never` allows one to safely express some rethrowing patterns that are otherwise not possible with rethrows. For example, consider a function that semantically rethrows, but needs to do so by going through some code that doesn't throw: + +```swift +/// Count number of nodes in the tree that match a particular predicate +func countNodes(in tree: Node, matching predicate: (Node) throws -> Bool) rethrows -> Int { + class MyNodeVisitor: NodeVisitor { + var error: (any Error)? = nil + var count: Int = 0 + var predicate: (Node) throws -> Bool + + init(predicate: @escaping (Node) throws -> Bool) { + self.predicate = predicate + } + + override func visit(node: Node) { + do { + if try predicate(node) { + count = count + 1 + } + } catch let localError { + error = error ?? localError + } + } + } + + return try withoutActuallyEscaping(predicate) { predicate in + let visitor = MyNodeVisitor(predicate: predicate) + visitor.visitTree(node) + if let error = visitor.error { + throw error // error: is not throwing as a consequence of 'predicate' throwing. + } else { + return visitor.count + } + } +} +``` + +Walking through the code, we can convince ourselves that `MyNodeVisitor.error` will only ever be set as a result of the predicate throwing an error, so this code semantically fulfills the contract of `rethrows`. However, the Swift compiler's rethrows checking cannot perform such an analysis, so it will reject this function. The limitation on `rethrows` has prompted at least [two](https://forums.swift.org/t/pitch-rethrows-unchecked/10078) [pitches](https://forums.swift.org/t/pitch-fix-rethrows-checking-and-add-rethrows-unsafe/44863) to add an "unsafe" or "unchecked" rethrows variant, turning this into a runtime-checked contract. + +Typed throws offer a compelling alternative: one can capture the error type of the closure argument in a generic parameter, and use that consistently throughout. This is immediately useful for maintaining precise typed error information in generic code that only rethrows the error from its closure arguments, like `map`: + +```swift +extension Collection { + func map(body: (Element) throws(E) -> U) throws(E) -> [U] { + var result: [U] = [] + for element in self { + result.append(try body(element)) + } + return result + } +} +``` + +When given a closure that throws `CatError`, this formulation of `map` will throw `CatError`. When given a closure that doesn't throw, `E` will be `Never`, so `map` is non-throwing. + +This approach extends to our `countNodes` example: + +```swift +/// Count number of nodes in the tree that match a particular predicate +func countNodes(in tree: Node, matching predicate: (Node) throws(E) -> Bool) throws(E) -> Int { + class MyNodeVisitor: NodeVisitor { + var error: E? = nil + var count: Int = 0 + var predicate: (Node) throws(E) -> Bool + + init(predicate: @escaping (Node) throws(E) -> Bool) { + self.predicate = predicate + } + + override func visit(node: Node) { + do { + if try predicate(node) { + count = count + 1 + } + } catch let localError { + error = error ?? localError // okay, error has type E?, localError has type E + } + } + } + + return try withoutActuallyEscaping(predicate) { predicate in + let visitor = MyNodeVisitor(predicate: predicate) + visitor.visitTree(node) + if let error = visitor.error { + throw error // okay! error has type E, which can be thrown out of this function + } else { + return visitor.count + } + } +} +``` + +Note that typed throws has elegantly solved our problem, because any throwing site that throws a value of type `E` is accepted. When the closure argument doesn't throw, `E` is inferred to `Never`, and (dynamically) no instance of it will ever be created. + +### When to use typed throws + +Typed throws makes it possible to strictly specify the thrown error type of a function, but doing so constrains the evolution of that function's implementation. Additionally, errors are usually propagated or rendered, but not exhaustively handled, so even with the addition of typed throws to Swift, untyped `throws` is better for most scenarios. Consider typed throws only in the following circumstances: + +1. In code that stays within a module or package where you always want to handle the error, so it's purely an implementation detail and it is plausible to handle the error. +2. In generic code that never produces its own errors, but only passes through errors that come from user components. The standard library contains a number of constructs like this, whether they are `rethrows` functions like `map` or are capturing a `Failure` type like in `Task` or `Result`. +3. In dependency-free code that is meant to be used in a constrained environment (e.g., Embedded Swift) or cannot allocate memory, and will only ever produce its own errors. + +Resist the temptation to use typed throws because there is only a single kind of error that the implementation can throw. For example, consider an operation that loads bytes from a specified file: + +```swift +public func loadBytes(from file: String) async throws(FileSystemError) -> [UInt8] // should use untyped throws +``` + +Internally, it is using some file system library that throws a `FileSystemError`, which it then republishes directly. However, the fact that the error was specified to always be a `FileSystemError` may hamper further evolution of this API: for example, it might be reasonable for this API to start supporting loading bytes from other sources (say, a network connection or database) when the file name matches some other schema. However, errors from those other libraries will not be `FileSystemError` instances, which poses a problem for `loadBytes(from:)`: it either needs to translate the errors from other libraries into `FileSystemError` (if that's even possible), or it needs to break its API contract by adopting a more general error type (or untyped `throws`). + +This section will be added to the [Swift API Design Guidelines](https://www.swift.org/documentation/api-design-guidelines/). + +## Detailed design + +### Syntax adjustments + +The [Swift grammar](https://docs.swift.org/swift-book/ReferenceManual/zzSummaryOfTheGrammar.html) is updated wherever there is either `throws` or `rethrows`, to optionally include a thrown type, e.g., + +``` +throws-clause -> throws thrown-type(opt) + +thrown-type -> '(' type ')' +``` + +#### Function type + +Changing from + +``` +function-type → attributes(opt) function-type-argument-clause async(opt) throws(opt) -> type +``` + +to + +``` +function-type → attributes(opt) function-type-argument-clause async(opt) throws-clause(opt) -> type +``` + +Examples + +```swift +() -> Bool +() throws -> Bool +() throws(CatError) -> Bool +``` + +#### Closure expression + +Changing from + +``` +closure-signature → capture-list(opt) closure-parameter-clause async(opt) throws(opt) function-result opt in +``` + +to + +``` +closure-signature → capture-list(opt) closure-parameter-clause async(opt) throws-clause(opt) function-result opt in +``` + +Examples + +```swift +{ () -> Bool in true } +{ () throws -> Bool in true } +{ () throws(CatError) -> Bool in true } +``` + + +#### Function, initializer, and accessor declarations + +Changing from + +``` +function-signature → parameter-clause async(opt) throws(opt) function-result(opt) +function-signature → parameter-clause async(opt) rethrows(opt) function-result(opt) +initializer-declaration → initializer-head generic-parameter-clause(opt) parameter-clause async(opt) throws(opt) +initializer-declaration → initializer-head generic-parameter-clause(opt) parameter-clause async(opt) throws(opt) +``` + +to + +``` +function-signature → parameter-clause async(opt) throws-clause(opt) function-result(opt) +initializer-declaration → initializer-head generic-parameter-clause(opt) parameter-clause async(opt) throws-clause(opt) +``` + +Note that the current grammar does not account for throwing accessors, although they should receive the same transformation. + +#### `do..catch` blocks + +The syntax of a `do..catch` block is extended with an optional throw clause: + +``` +do-statement → do throws-clause(opt) code-block catch-clauses? +``` + +If a `throws-clause` is present, then there must be at least one `catch-clause`. + +#### Examples + +```swift +func callCat() -> Cat +func callCat() throws -> Cat +func callCat() throws(CatError) -> Cat + +init() +init() throws +init() throws(CatError) + +var value: Success { + get throws(Failure) { ... } +} +``` + +### Throwing and catching with typed throws + +#### Throwing within a function that declares a typed error + +Any function, closure or function type that is marked as `throws` can declare which type the function throws. That type, which is called the *thrown error type*, must conform to the `Error` protocol. + +Every uncaught error that can be thrown from the body of the function must be convertible to the thrown error type. This applies to both explicit `throw` statements and any errors thrown by other calls (as indicated by a `try`). For example: + +```swift +func throwingTypedErrors() throws(CatError) { + throw CatError.asleep // okay, type matches + throw .asleep // okay, can infer contextual type from the thrown error type + throw KidError() // error: KidError is not convertible to CatError + + try callCat() // okay + try callKids() // error: throws KidError, which is not convertible to CatError + + do { + try callKids() // okay, because this error is caught and suppressed below + } catch { + // eat the error + } +} +``` + +Because a value of any `Error`-conforming type implicitly converts to `any Error`, this implies that an function declared with untyped `throws` can throw anything: + +```swift +func untypedThrows() throws { + throw CatError.asleep // okay, CatError converts to any Error + throw KidError() // okay, KidError converts to any Error + try callCat() // okay, thrown CatError converts to any Error + try callKids() // okay, thrown KidError converts to any Error +} +``` + +Therefore, these rules subsume those of untyped throws, and no existing code will change behavior. + +Note that the constraint that the thrown error type must conform to `Error` means that one cannot use an existential type such as `any Error & Codable` as the thrown error type: + +```swift +// error: any Error & Codable does not conform to Error +func remoteCall(function: String) async throws(any Error & Codable) -> String { ... } +``` + +The `any Error` existential has [special semantics](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0235-add-result.md#adding-swifterror-self-conformance) that allow it to conform to the `Error` protocol, introduced along with `Result`. A separate language change would be required to allow other existential types to conform to the `Error` protocol. + +#### Catching typed thrown errors + +A `do...catch` block is used to catch and process thrown errors. With only untyped errors, the type of the error thrown from inside the `do` block is always `any Error`. In the presence of typed throws, the type of the error thrown from inside the `do` block can either be explicitly specified with a `throws` clause following the `do`, or inferred from the specific throwing sites. + +When the `do` block specifies a thrown error type, that error type can be used for inferring the contextual type of `throw` statements. For example: + +```swift +do throws(CatError) { + if isDaytime && foodBowl.isEmpty { + throw .sleep + } +} catch { + // implicit 'error' value has type CatError +} +``` + +As with other uses of untyped throws, `do throws` is equivalent to `do throws(any Error)`. + +When there is no throws clause, the thrown error type is inferred from the body of the `do` block. When all throwing sites within a `do` block produce the same error type (ignoring any that throw `Never`), that error type is used as the type of the thrown error. For example: + +```swift +do /*infers throws(CatError)*/ { + try callCat() // throws CatError + if something { + throw CatError.asleep // throws CatError + } +} catch { + // implicit 'error' value has type CatError + if error == .asleep { + openFoodCan() + } +} +``` + +This also implies that one can use the thrown type context to perform type-specific checks in the catch clauses, e.g., + +```swift +do /*infers throws(CatError)*/ { + try callCat() // throws CatError + if something { + throw CatError.asleep // throws CatError + } +} catch .asleep { + openFoodCan() +} // note: CatError can be thrown out of this do...catch block when the cat isn't asleep +``` + +> **Rationale**: By inferring a concrete result type for the thrown error type, we can entirely avoid having to reason about existential error types within `catch` blocks, leading to a simpler syntax. Additionally, it preserves the notion that a `do...catch` block that has a `catch` site accepting anything (i.e., one with no conditions) can exhaustively suppress all errors. + +When throw sites within the `do` block throw different (non-`Never`) error types, the inferred error type is `any Error`. For example: + +```swift +do /*infers throws(any Error)*/ { + try callCat() // throws CatError + try callKids() // throw KidError +} catch { + // implicit 'error' variable has type 'any Error' +} +``` + +In essence, when there are multiple possible thrown error types, we immediately resolve to the untyped equivalent of `any Error`. We will refer to this notion as a type function `errorUnion(E1, E2, ..., EN)`, which takes `N` different error types (e.g., for throwing sites within a `do` block) and produces the union error type of those types. Our definition and use of `errorUnion` for typed throws subsumes the existing rule for untyped throws, in which every throw site produces an error of type `any Error`. + +> **Rationale**: While it would be possible to compute a more precise "union" type of different error types, doing so is potentially an expensive operation at compile time and run time, as well as being harder for the programmer to reason about. If in the future it becomes important to tighten up the error types, that could be done in a mostly source-compatible manner. + +The semantics specified here are not fully source compatible with existing Swift code. A `do...catch` block that contains `throw` statements of a single concrete type (and no other throwing sites) might depend on the error being caught as `any Error`. Here is a contrived example: + +```swift +do /*infers throws(CatError) in Swift 6 */ { + throw CatError.asleep +} catch { + var e = error // currently has type any Error, will have type CatError + e = KidsError() // currently well-formed, will become an error +} +``` + +To prevent this source compatibility issue, we refine the rule slightly to specify that any `throw` statement always throws a value of type `any Error`. That way, one can only get a caught error type more specific than `any Error` when the both of the `do..catch` contains no `throw` statements and all of the `try` operations are using functions that make use of typed throws. + +Note that the only way to write an exhaustive `do...catch` statement is to have an unconditional `catch` block. The dynamic checking provided by `is` or `as` patterns in the `catch` block cannot be used to make a catch exhaustive, even if the type specified is the same as the type thrown from the body of the `do`: + +```swift +func f() { + do /*infers throws(CatError)*/ { + try callCat() + } catch let ce as CatError { + + } // error: do...catch is not exhaustive, so this code rethrows CatError and is ill-formed +} +``` + +> **Note**: Exhaustiveness checking in the general is expensive at compile time, and the existing language uses the presence of an unconditional `catch` block as the indicator for an exhaustive `do...catch`. See the section on closure thrown type inference for more details about inferring throwing closures. + +#### `rethrows` + +A function marked `rethrows` throws only when one of its closure parameters throws. It is typically used with higher-order functions, such as the `map` operation on a collection: + +```swift +extension Collection { + func map(body: (Element) throws -> U) rethrows -> [U] { + var result: [U] = [] + for element in self { + result.append(try body(element)) + } + return result + } +} +``` + +When provided with a throwing closure, `map` can throw, and it chooses to directly throw the same error as the body. This contract can be more precisely modeled using typed throws: + +```swift +extension Collection { + func map(body: (Element) throws(E) -> U) throws(E) -> [U] { + var result: [U] = [] + for element in self { + result.append(try body(element)) + } + return result + } +} +``` + +Now, when `map` is provided with a closure that throws `E`, it can only throw an `E`. For a non-throwing closure, `E` will be `Never` and `map` is non-throwing. For an untyped throwing closure, `E` will be `any Error` and we get the same type-level behavior as the `rethrows` version of `map`. + +However, because `rethrows` uses untyped errors, `map` would be permitted to substitute a different error type that, for example, provides more information about the failing element: + +```swift +struct MapError: Error { + var failedElement: Element + var underlyingError: any Error +} + +extension Collection { + func map(body: (Element) throws -> U) rethrows -> [U] { + var result: [U] = [] + for element in self { + do { + result.append(try body(element)) + } catch { + // Provide more information about the failure + throw MapError(failedElement: element, underlyingError: error) + } + } + return result + } +} +``` + +Typed throws, as presented here, is not able to express the contract of this function. + +The Swift standard library does not perform error substitution of this form, and its contract for operations like `map` is best expressed by typed throws as shown above. It is likely that many existing `rethrows` functions are better expressed with typed throws. However, not *all* `rethrows` functions can be expressed by typed throws, if they are performing error substitution like this last `map`. + +Therefore, this proposal does not change the primary semantics of `rethrows`: it remains untyped, and it is ill-formed to attempt to provide a thrown error type to a `rethrows` function. The Alternatives Considered section provides several options for `rethrows`, which can become the subject of a future proposal. + +However, there is a small change in the type checking behavior of a `rethrows` function to improve source compatibility in certain cases. Specifically, consider a `rethrows` function that calls into a function with typed throws: + +```swift +extension Collection { + func filter(_ isIncluded: (Element) throws(E) -> Bool) throws(E) -> [Element] { ... } + + func filterOdds(_ isIncluded: (Element) throws -> Bool) rethrows -> [Element { + var onOdd = true + return try filter { element in + defer { onOdd = !onOdd } + return onOdd && isIncluded(element) + } // error: call to filter isn't "rethrows" + } +} +``` + +The standard `rethrows` checking rejects the call to `filter` because, technically, it could throw `any Error` under any circumstances. Unfortunately, this behavior is a source compatibility problem for the standard library's adoption of typed throws, because an existing `rethrows` function calling into something like `map` or `filter` would be rejected once those introduce typed throws. This proposal introduces a small compatibility feature that considers a function that + +1. Has a thrown error type that is a generic parameter (call it `E`) of the function itself, +2. Has no protocol requirements on `E` other than that it conform to the `Error` protocol, and +3. Any parameters of throwing function type throw the specific error type `E`. + +to be a rethrowing function for the purposes of `rethrows` checking in its caller. This compatibility feature introduces a small soundness hole in `rethrows` functions that can only be removed with improvements to type inference behavior. + +#### Opaque thrown error types + +The thrown error type of a function can be specified with an [opaque result type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md). For example: + +```swift +func doSomething() throws(some Error) { ... } +``` + +The opaque thrown error type is like a result type, so the concrete type of the error is chosen by the `doSomething` function itself, and could change from one version to the next. The caller only knows that the error type conforms to the `Error` protocol; the concrete type won't be knowable until runtime. + +Opaque result types can be used as an alternative to existentials (`any Error`) when there is a fixed number of potential error types that might be thrown , and we either can't (due to being in an embedded environment) or don't want to (for performance or code-evolution reasons) expose the precise error type. For example, one could use a suitable `Either` type under the hood: + +```swift +func doSomething() throws(some Error) { + do { + try callCat() + } catch { + throw Either.left(error) + } + + do { + try callKids() + } catch { + throw Either.right(error) + } +} +``` + +Due to the contravariance of parameters, an opaque thrown error type that occurs within a function parameter will be an [opaque parameter](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md). This means that the closure argument itself will choose the type, so + +```swift +func map(_ transform: (Element) throws(some Error) -> T) rethrows -> [T] +``` + +is equivalent to + +```swift +func map(_ transform: (Element) throws(E) -> T) rethrows -> [T] +``` + +#### `async let` + +An `async let` initializer can throw an error, and that error is effectively rethrown at any point where one of the variables defined in the `async let` is referenced. For example: + +```swift +async let answer = callCat() +// ... +try await answer // could rethrow the result from the initializer here +``` + +The type thrown by the variables of an `async let` is determined using the same rules as for the `do` part of a `do...catch` block. In the example above, accesses to `answer` can throw an error of type `CatError`. + +### Subtyping rules + +A function type that throws an error of type `A` is a subtype of a function type that differs only in that it throws an error of type `B` when `A` is a subtype of `B`. As previously noted, a `throws` function that does not specify the thrown error type will have a thrown type of `any Error`, and a non-throwing function has a thrown error type of `Never`. For subtyping purposes, `Never` is assumed to be a subtype of all error types. + +The subtyping rule manifests in a number of places, including function conversions, protocol conformance checking and refinements, and override checking, all of which are described below. + +#### Function conversions + +Having related errors and a non-throwing function + +```swift +class BaseError: Error {} +class SubError: BaseError {} + +let f1: () -> Void +``` + +Converting a non-throwing function to a throwing one is allowed + +```swift +let f2: () throws(SubError) -> Void = f1 +``` + +It's also allowed to assign a subtype of a thrown error, though the subtype information is erased and the error of f2 will be casted up. + +```swift +let f3: () throws(BaseError) -> Void = f2 +``` + +Erasing the specific error type is possible + +```swift +let f4: () throws -> Void = f3 +``` + +#### Protocol conformance + +Protocols should have the possibility to conform and refine other protocols containing throwing functions based on the subtype relationship of their functions. This way it would be possible to throw a more specialised error or don't throw an error at all. + +```swift +protocol Throwing { + func f() throws +} + +struct ConcreteNotThrowing: Throwing { + func f() { } // okay, doesn't have to throw +} + +enum SpecificError: Error { ... } + +struct ConcreteThrowingSpecific: Throwing { + func f() throws(SpecificError) { } // okay, throws a specific error +} +``` + +#### Override checking + +A declaration in a subclass that overrides a superclass declaration can be a subtype of the superclass declaration, for example: + +```swift +class BlueError: Error { ... } +class DeepBlueError: BlueError { ... } + +class Superclass { + func f() throws { } + func g() throws(BlueError) { } +} + +class Subclass: Superclass { + override func f() throws(BlueError) { } // okay + override func g() throws(DeepBlueError) { } // okay +} + +class Subsubclass: Subclass { + override func f() { } // okay + override func g() { } // okay +} +``` + +### Type inference + +The type checker can infer thrown error types in a number of different places, making it easier to carry specific thrown type information through a program without additional annotation. This section covers the various ways in which thrown errors interact with type inference. + +#### Associated type inference + +An associated type can be used as the thrown error type in other protocol requirements. For example: + +```swift +protocol CatFeeder { + associatedtype FeedError: Error + + func feedCat() throws(FeedError) -> CatStatus +} +``` + +When a concrete type conforms to such a protocol, the associated type can be inferred from the declarations that satisfy requirements that mention the associated type in a typed throws clause. For the purposes of this inference, a non-throwing function has `Never` as its error type and an untyped `throws` function has `any Error` as its error type. For example: + +```swift +struct Tabby: CatFeeder { + func feedCat() throws(CatError) -> CatStatus { ... } // okay, FeedError is inferred to CatError +} + +struct Sphynx: CatFeeder { + func feedCat() throws -> CatStatus { ... } // okay, FeedError is inferred to any Error +} + +struct Ragdoll: CatFeeder { + func feedCat() -> CatStatus { ... } // okay, FeedError is inferred to Never +} +``` + +#### `Error` requirement inference + +When a function signature uses a generic parameter or associated type as a thrown type, that generic parameter or associated type is implicitly inferred to conform to the `Error` type. For example, given this declaration for `map`: + +```swift +func map(body: (Element) throws(E) -> T) throws(E) { ... } +``` + +the function has an inferred requirement `E: Error`. + +### Standard library adoption + +#### Converting between `throws` and `Result` + +`Result`'s [init(catching:)](https://developer.apple.com/documentation/swift/result/3139399-init) operation translates a throwing closure into a `Result` instance. It's currently defined only when the `Failure` type is `any Error`, i.e., + +```swift +init(catching body: () throws -> Success) where Failure == any Error { ... } +``` + +Replace this with an initializer that uses typed throws: + +```swift +init(catching body: () throws(Failure) -> Success) +``` + +The new initializer is more flexible: in addition to retaining the error type from typed throws, it also supports non-throwing closure arguments by inferring `Failure` to be equal to `Never`. + +Additionally, `Result`'s `get()` operation: + +```swift +func get() throws -> Success +``` + +should use `Failure` as the thrown error type: + +```swift +func get() throws(Failure) -> Success +``` + +#### Standard library operations that `rethrow` + +The standard library contains a large number of operations that `rethrow`. In all cases, the standard library will only throw from a call to one of the closure arguments: it will never substitute a different thrown error. Therefore, each `rethrows` operation in the standard library should be replaced with one that uses typed throws to propagate the same error type. For example, the `Optional.map` operation would change from: + +```swift +public func map( + _ transform: (Wrapped) throws -> U +) rethrows -> U? +``` + +to + +```swift +public func map( + _ transform: (Wrapped) throws(E) -> U +) throws(E) -> U? +``` + +This is a mechanical transformation that is applied throughout the standard library. + +## Source compatibility + +This proposal has called out a few specific places where the introduction of typed throws into the language could affect source compatibility. However, in those places, we have opted for semantics that ensure that existing Swift code that does not change behavior, to make this proposal act as a purely additive change to the language. Once a function adopts typed throws, the effect of typed throws can then ripple to its callers. + +## Effect on API resilience + +An API that uses typed throws cannot make its thrown error type more general (or untyped) without breaking existing clients that depend on the specific thrown error type: + +```swift +// Library +public enum DataLoaderError { + case missing +} + +public class DataLoader { + func load() throws(DataLoaderError) -> Data { ... } +} + +// Client code +func processError(_ error: DataLoaderError) { ... } + +func load(from dataLoader: dataLoader) { + do { + try dataLoader.load() + } catch { + processError(error) + } +} +``` + +Any attempt to generalize the thrown type of `DataLoader.load()` will break the client code, which depends on getting a `DataLoaderError` in the `catch` block. + +Going in the other direction, of making the thrown error type *more* specific than it used to be (or adopting typed throws in an API that previously used untyped throws) can also break clients, but in much more limited cases. For example, let's consider the same API above, but in reverse: + +```swift +// Library +public enum DataLoaderError { + case missing +} + +public class DataLoader { + func load() throws -> Data { ... } +} + +// Client +func processError(_ error: any Error) { ... } + +func load(from dataLoader: dataLoader) { + do { + try dataLoader.load() + } catch { + processError(error) + } +} +``` + +Here, the `DataLoader.load()` function could be updated to throw `DataLoaderError` and this particular client code would still work, because `DataLoaderError` is convertible to `any Error`. Note that clients could still be broken by this kind of change, for example overrides of an `open` function, declarations that satisfy a protocol requirement, or code that relies on the precide error type (say, by overloading). However, such a change is far less likely to break clients of an API than loosening thrown type informance. + +A `rethrows` function can generally be replaced with a function that is generic over the thrown error type of its closure argument and propagates that thrown error. For example, one can replace this API: + +```swift +public func last( + where predicate: (Element) throws -> Bool +) rethrows -> Element? +``` + +with + +```swift +public func last( + where predicate: (Element) throws(E) -> Bool +) throws(E) -> Element? +``` + +When calling this function, the closure argument supplies the thrown error type (`E`), which can also be inferred to `any Error` (for untyped `throws`) or `Never` (for non-throwing functions). Existing clients of this new function therefore see the same behavior as with the `rethrows` version. + +There is one difference between the two functions that could break client code that is referring to such functions without calling them. For example, consider the following code: + +```swift +let primes = [2, 3, 5, 7] +let getLast = primes.last(where:) +``` + +With the `rethrows` formulation of the `last(where:)` function, `getLast` will have the type `((Int) throws -> Bool) throws -> Int?`. With the typed-errors formulation, this code will result in an error because the an argument for the generic parameter `E` cannot be inferred without context. Note that this is only a problem when there is no context type for `getLast`, and can be fixed by providing it with a type: + +```swift +let getLast: ((Int) -> Bool) -> Int? = primes.last(where:) // okay, E is inferred to Never +``` + +Note that one would have to do the same thing with the `rethrows` formulation to produce a non-throwing `getLast`, because `rethrows` is not a part of the formal type system. Given that most `rethrows` operations are already generic in other parameters (unlike `last(where:)`), and most uses of such APIs are either calls or have type context, it is expected that the actual source compatibility impact of replacing `rethrows` with typed errors will be small. + +## Effect on ABI stability + +The ABI between a function with an untyped throws and one that uses typed throws will be different, so that typed throws can benefit from knowing the precise type. + +Replacing a `rethrows` function with one that uses typed throws, as proposed for the standard library, is an ABI-breaking change. However, it can be done in a manner that doesn't break ABI by retaining the `rethrows` function only for binary-compatibility purposes. The existing `rethrows` functions will be renamed at the source level (so they don't conflict with the new ones) and made `@usableFromInline internal`, which retains the ABI while making the function invisible to clients of the standard library: + +```swift +@usableFromInline +@_silgen_name() +internal func _oldRethrowingMap( + _ transform: (Wrapped) throws -> U +) rethrows -> U? +``` + +Then, the new typed-throws version will be introduced with [back-deployment support](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0376-function-back-deployment.md): + +```swift +@backDeploy(...) +public func map( + _ transform: (Wrapped) throws(E) -> U +) throws(E) -> U? +``` + +This way, clients compiled against the updated standard library will always use the typed-throws version. Note that many of these functions are quite small and will be generic, so implementers may opt to use `@_alwaysEmitIntoClient` rather than `@backDeploy`. + +## Future directions + +### Closure thrown type inference + +Function declarations must always explicitly specify whether they throw, optionally providing a specific thrown error type. For closures, whether they throw or not is inferred by the Swift compiler. Specifically, the Swift compiler looks at the structure of body of the closure. If the body of the closure contains a throwing site (either a `throw` statement or a `try` expression) that is not within an exhaustive `do...catch` (i.e., one that has an unconditional `catch` clause), then the closure is inferred to be `throws`. Otherwise, it is non-throwing. Here are some examples: + +```swift +{ throw E() } // throws + +{ try call() } // throws + +{ + do { + try call() + } catch let e as CatError { + // ... + } +} // throws, the do...catch is not exhaustive + +{ + do { + try call() + } catch e {} + // ... + } +} // does not throw, the do...catch is exhaustive +``` + +With typed throws, the closure type could be inferred to have a typed error by considering all of the throwing sites that aren't caught (let each have a thrown type `Ei`) and then inferring the closure's thrown error type to be `errorUnion(E1, E2, ... EN)`. + +This inference rule will change the thrown error types of existing closures that throw concrete types. For example, the following closure: + +```swift +{ + if Int.random(in: 0..<24) < 20 { + throw CatError.asleep + } +} +``` + +will currently be inferred as `throws`. With the rule specified here, it will be inferred as `throws(CatError)`. This could break some code that depends on the precisely inferred type. To prevent this from becoming a source compatibility problem, we apply the same rule as for `do...catch` statements to limit inference: `throw` statements within the closure body are treated as having the type `any Error` in Swift 5. This way, one can only infer a more specific thrown error type in a closure when the `try` operations are calling functions that make use of typed errors. + +Note that one can explicitly specify the thrown error type of a closure to disable this type inference, which has the nice effect of also providing a contextual type for throw statements: + +```swift +{ () throws(CatError) in + if Int.random(in: 0..<24) < 20 { + throw .asleep + } +} +``` + +Such a change would need to be under an upcoming feature flag (e.g., `FullTypedThrows`) and should also involve inference from the actual thrown error type of `throw` statements as well as closing the minor semantic hole introduced for compatibility with `rethrows` functions. + +### Concurrency library adoption + +The concurrency library has a number of places that could benefit from the adoption of typed throws, including `Task` creation and completion, continuations, task cancellation, task groups, and async sequences and streams. + +`Task` is similar to `Result` because it also carries a `Failure` type that could benefit from typed throws. Continuations and task groups could propagate typed throws information from closures to make more of the library usable with precise thrown type information. + +`AsyncSequence`, and the asynchronous `for..in` loop that depends on it, could be improved by using typed throws. Both `AsyncIteratorProtocol` and `AsyncSequence` could be augmented with a `Failure` associated type that is used for the thrown error type of `next()`, and will be used by the asynchronous `for..in` loop to determine whether the sequence can throw. This can be combined with [primary associated types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md) to make it possible to use existentials such as `any AsyncSequence`: + +```swift +public protocol AsyncIteratorProtocol { + associatedtype Element + associatedtype Failure: Error = any Error + mutating func next() async throws(Failure) -> Element? +} + +public protocol AsyncSequence { + associatedtype AsyncIterator: AsyncIteratorProtocol + associatedtype Element where AsyncIterator.Element == Element + associatedtype Failure where AsyncIterator.Failure == Failure + __consuming func makeAsyncIterator() -> AsyncIterator +} +``` + +The scope of potential changes to the concurrency library to make full use of typed throws is large. Unlike with the standard library, the adoption of typed throws in the concurrency library requires some interesting design. Therefore, we leave it to a follow-on proposal, noting only that whatever form `AsyncSequence` takes with typed throws, the language support for asynchronous `for..in` will need to adjust. + +### Specific thrown error types for distributed actors + +The transport mechanism for [distributed actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0344-distributed-actor-runtime.md), `DistributedActorSystem`, can throw an error due to transport failures. This error is currently untyped, but it should be possible to adopt typed throws (with a `Failure` associated type in `DistributedActorSystem` and mirrored in `DistributedActor`) so that the distributed actor system can be more specific about the kind of error it throws. Calls to a distributed actor from outside the actor (i.e., that could be on a different node) would then throw `errorUnion(Failure, E)` where the `E` is the type that the function normally throws. + +## Alternatives considered + +### Thrown error type syntax + +There have been several alternatives to the `throws(E)` syntax proposed here. The `throws(E)` syntax was chosen because it is syntactically unambiguous, allows arbitrary types for `E`, and is consistent with the way in which attributes (like property wrappers or macros with arguments) and modifiers (like `unowned(unsafe)`) are written. + +The most commonly proposed syntax omits the parentheses, i.e., `throws E`. However, this syntax introduces some syntactic ambiguities that would need to be addressed and might cause problems for future evolution of the language: + +* The following code is syntactically ambiguous if `E` is parsed with the arbitrary `type` grammar: + + ```swift + func f() throws (E) -> Int { ... } + ``` + + because the error type would parse as either `(E)` or `(E) -> Int`. One could parse a subset of the `type` grammar that doesn't include function types to dodge this ambiguity, with a more complicated grammar. + +* The identifier following `throws` could end up conflicting with a future effect: + + ```swift + func f() throws E { ... } + ``` + + If `E` were an effect name in some later Swift version, then there is an ambiguity between typed throws and that effect that we would need to resolve. Future effect modifiers might require more than one argument (and therefore need parentheses), which would make them inconsistent with `throws E`. + +Another suggestion uses angle brackets around the thrown type, i.e., + +```swift +func f() throws -> Int { ... } +``` + +This follows more closely with generic syntax, and highlights the type nature of the arguments more clearly. It's inconsistent with the use of parentheses in modifiers, but has some precedent in attached macros where one can explicitly specify the generic arguments to the macro, e.g., `@OptionSet`. + +### Multiple thrown error types + +This proposal specifies that a function may throw at most one error type, and if there is any reason to throw more than one error type, one should use `any Error` (or the equivalent untyped `throws` spelling). It would be possible to support multiple error types, e.g., + +```swift +func fetchData() throws(FileSystemError, NetworkError) -> Data +``` + +However, this change would introduce a significant amount of complexity in the type system, because everywhere that deals with thrown errors would have to deal with an arbitrary set of thrown errors. + +A more reasonable direction to support this use case would be to introduce a form of anonymous enum (often called a *sum* type) into the language itself, where the type `A | B` can be either an `A` or ` B`. With such a feature in place, one could express the function above as: + +```swift +func fetchData() throws(FileSystemError | NetworkError) -> Data +``` + +Trying to introduce multiple thrown error types directly into the language would introduce nearly all of the complexity of sum types, but without the generality, so this proposal only considers a single thrown error type. + +### Treat all uninhabited thrown error types as nonthrowing + +This proposal specifies that a function type whose thrown error type is `Never` is equivalent to a function type that does not throw. This rule could be generalized from `Never` to any *uninhabited* type, i.e., any type for which we can structurally determine that there is no runtime value. The simplest uninhabited type is a frozen enum with no cases, which is how `Never` itself is defined: + +```swift +@frozen public enum Never {} +``` + +However, there are other forms of uninhabited type: a `struct` or `class` with a stored property of uninhabited type is uninhabited, as is an enum where all cases have an associated value containing an uninhabited type (a generalization of the "no cases" rule mentioned above). This can happen generically. For example, a simple `Pair` struct: + +```swift +struct Pair { + var first: First + var second Second +} +``` + +will be uninhabited when either `First` or `Second` is uninhabited. The `Either` enum will be uninhabited when both of its generic arguments are uninhabited. `Optional` is never uninhabited, because it's always possible to create a `nil` value. + +It is possible to generalize the rule about non-throwing function types to consider any function type with an uninhabited thrown error type to be equivalent to a non-throwing function type (all other things remaining equal). However, we do not do so due to implementation concerns: the check for a type being uninhabited is nontrivial, requiring one to walk all of the storage of the type, and (in the presence of indirect enum cases and reference types) is recursive, making it a potentially expensive computation. Crucially, this computation will need to be performed at runtime, to produce proper function type metadata within generic functions: + +```swift +func f(_: E.Type)) { + typealias Fn = () throws(E) -> Void + let meta = Fn.self +} + +f(Never.self) // Fn should be equivalent to () -> Void +f(Either.self) // Fn should be equivalent to () -> Void +f(Pair.self) // Fn should be equivalent to () -> Void +``` + +The runtime computation of "uninhabited" therefore carries significant cost in terms of the metadata required (one may need to walk all of the storage of the type) as well as the execution time to evaluate that metadata during runtime type formation. + +The most plausible route here involves the introduction of an `Uninhabited` protocol, which could then be used with conditional conformances to propagate the "uninhabited" type information. For example, `Never` would conform to `Uninhabited`, and one could conditionally conform a generic error type. For example: + +```swift +struct WrappedError: Error { + var wrapped: E +} + +extension WrappedError: Uninhabited where E: Uninhabited { } +``` + +With this, one can express "rethrowing" behavior that wraps the underlying error via typed throws: + +```swift +func translatesError(f: () throws(E) -> Void) throws(WrappedError) { ... } +``` + +Here, when give a non-throwing closure for `f` (which infers `E = Never`), `translatesError` is known not to throw because `WrappedError` is known to be uninhabited (via the conditional conformance). This approach extends to the use of an `Either` type to capture errors: + +```swift +extension Either: Uninhabited when Left: Uninhabited, Right: Uninhabited { } +``` + +However, it breaks down when there are two such generic error parameters for something like `WrappedError`, because having either one of them be `Uninhabited` makes the struct uninhabited, and the generics system does not permit disjunctive constraints like that. + +Extending from `Never` to arbitrary uninhabited types has some benefits, but requires enough additional design work and complexity that it should constitute a separate proposal. Therefore, we stick with the simpler rule where `Never` is the only uninhabited type considered to be special. + +### Typed `rethrows` + +A function marked `rethrows` throws only when one or more of its closure arguments throws. As note previously, typed throws allows one to more precisely express when the function only rethrows exactly the error from its closure, without translation, as demonstrated with `map`: + +```swift +func map(_ transform: (Element) throws(E) -> T) throws(E) -> [T] +``` + +However, it cannot express rethrowing behavior when the function is performing translation of errors. For example, consider the following: + +```swift +func translateErrors( + f: () throws(E1) -> Void, + g: () throws(E2) -> Void +) ??? { + do { + try f() + } catch { + throw SimpleError(message: "E1: \(error)") + } + + do { + try g() + } catch { + throw SimpleError(message: "E2: \(error)") + } +} +``` + +This function will only throw when `f` or `g` throw, and in both cases will translate the errors into `SimpleError`. With this proposal, there are two options for specifying the error-handling behavior of `translateErrors`, neither of which is precise: + +* `rethrows` correctly communicates that this function throws only when the arguments for `f` or `g` do, but the thrown error type is treated as `any Error`. +* `throws(SimpleError)` correctly communicates that this function throws errors of type `SimpleError`, but not that it throws when the argument for `f` or `g` do. + +One way to address this would be to allow `rethrows` to specify the thrown error type, e.g., `rethrows(SimpleError)`, which captures both of the aspects of how this function behaves---when it throws, and what specific error type it `throws`. + +With typed `rethrows`, a bare `rethrows` could be treated as syntactic sugar for `rethrows(any Error)`, similarly to how `throws` is syntactic sugar for `throws(any Error)`. This extension is source-compatible and allows one to express more specific error types with throwing behavior. + +However, this definition of `rethrows` is somewhat unfortunate in a typed-throws world, because it is likely the wrong default. Many use cases for `rethrows` do not involve error translation, and would be better served by using typed throws in the manner that `map` does. If `rethrows` were not already part of the Swift language prior to this proposal, it's likely that we either would not introduce the feature at all, or would treat it as syntactic sugar for typed throws that introduces a generic parameter for the error type that is used for the thrown type of the closure parameters and the function itself. For example: + +```swift +// rethrows could try rethrows as syntactic sugar.. +func map(_ transform: (Element) throws -> T) rethrows -> [T] +// for typed errors: +func map(_ transform: (Element) throws(E) -> T) throws(E) -> [T] +``` + +Removing or changing the semantics of `rethrows` would be a source-incompatible change, so we leave such concerns to a later proposal. + +## Revision history + +* Revision 6 (post-review): + * Closure type inference did not get implemented in Swift 6.0, so this proposal has been "shrunk" down to what actually got implemented in Swift 6.0. +* Revision 5 (first review): + * Add `do throws(MyError)` { ... } syntax to allow explicit specification of the thrown error type within the body of a `do..catch` block, suppressing type inference of the thrown error type. Thank you to Becca Royal-Gordon for the idea! +* Revision 4: + * Update the introduction, motivation, and "when to use typed throws" to be more direct. + * Re-incorporate the replacement of `rethrows` functions in the standard library with generic typed throws into the actual proposal. It's so mechanical and straightforward that it doesn't need a separate proposal. + * Extend the discussion on API resilience to talk through the source compatibility impacts of replacing a `rethrows` function with one that uses typed throws, since it is quite relevant to this proposal. + * Explain that one cannot currently have a thrown error type of `any Error & Codable` or similar because it doesn't conform to `Error`. + * Introduce a compatibility feature to `rethrows` functions to cope with their callees moving to typed throws. +* Revision 3: + * Move the the typed `rethrows` feature out of this proposal, and into Alternatives Considered. Once we gain more experience with typed throws, we can decide what to do with `rethrows`. + * Expand the discussion on allowing all uninhabited error types to mean "non-throwing". + * Provide a better example for inferring `Error` conformance on generic parameters. + * Move the replacement of `rethrows` in the standard library with typed throws into "Future Directions", because it is large enough that it needs a separate proposal. + * Move the concurrency library changes for typed throws into "Future Directions", because it is large enough that it needs a separate proposal. + * Add an extended example of replacing the need for `rethrows(unsafe)` with typed throws. + * Provide a more significant example of opaque thrown errors that makes use of `Either` internally. +* Revision 2: + * Add a short section on when to use typed throws + * Add an Alternatives Considered section for other syntaxes + * Make it clear that only unconditional catches make `do...catch` exhaustive + * Update continuation APIs with typed throws + * Add an example of an existential thrown error type + * Describe semantics of `async let` with respect to thrown errors + * Add updates to task cancellation APIs diff --git a/proposals/0414-region-based-isolation.md b/proposals/0414-region-based-isolation.md new file mode 100644 index 0000000000..715b245ccc --- /dev/null +++ b/proposals/0414-region-based-isolation.md @@ -0,0 +1,2074 @@ +# Region based Isolation + +* Proposal: [SE-0414](0414-region-based-isolation.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm) [Joshua Turcotti](https://github.com/jturcotti) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 6.0)** +* Upcoming Feature Flag: `RegionBasedIsolation` +* Review: ([first pitch](https://forums.swift.org/t/pitch-safely-sending-non-sendable-values-across-isolation-domains/66566)), ([second pitch](https://forums.swift.org/t/pitch-region-based-isolation/67888)), ([first review](https://forums.swift.org/t/se-0414-region-based-isolation/68805)), ([revision](https://forums.swift.org/t/returned-for-revision-se-0414-region-based-isolation/69123)), ([second review](https://forums.swift.org/t/se-0414-second-review-region-based-isolation/69740)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0414-region-based-isolation/70051)) + +## Introduction + +Swift Concurrency assigns values to *isolation domains* determined by actor and +task boundaries. Code running in distinct isolation domains can execute +concurrently, and `Sendable` checking defines away concurrent access to +shared mutable state by preventing non-`Sendable` values from being passed +across isolation boundaries full stop. In practice, this is a significant +semantic restriction, because it forbids natural programming patterns that are +free of data races. + +In this document, we propose loosening these rules by introducing a +new control flow sensitive diagnostic that determines whether a non-`Sendable` +value can safely be transferred over an isolation boundary. This is done by +introducing the concept of *isolation regions* that allows the compiler to +reason conservatively if two values can affect each other. Through the usage of +isolation regions, the language can prove that transferring a non-`Sendable` +value over an isolation boundary cannot result in races because the value (and +any other value that might reference it) is not used in the caller after the +point of transfer. + +## Motivation + +[SE-0302](0302-concurrent-value-and-concurrent-closures.md) states +that non-`Sendable` values cannot be passed across isolation boundaries. The +following code demonstrates a `Sendable` violation when passing a +newly-initialized value into an actor-isolated function: + +```swift +// Not Sendable +class Client { + init(name: String, initialBalance: Double) { ... } +} + +actor ClientStore { + var clients: [Client] = [] + + static let shared = ClientStore() + + func addClient(_ c: Client) { + clients.append(c) + } +} + +func openNewAccount(name: String, initialBalance: Double) async { + let client = Client(name: name, initialBalance: initialBalance) + await ClientStore.shared.addClient(client) // Error! 'Client' is non-`Sendable`! +} +``` + +This is overly conservative; the program is safe because: + +* `client` does not have access to any non-`Sendable` state from its initializer + parameters since Strings and Doubles are `Sendable`. +* `client` just being initialized implies that `client` cannot have any uses + outside of `openNewAccount`. +* `client` is not used within `openNewAccount` beyond `addClient`. + +The simple example above shows the expressivity limitations of Swift's strict +concurrency checking. Programmers are required to use unsafe escape hatches, +such as `@unchecked Sendable` conformances, for common patterns that are already +free of data races. + +## Proposed solution + +We propose the introduction of a new control flow sensitive diagnostic that +enables transferring non-`Sendable` values across isolation boundaries and emits +errors at use sites of non-`Sendable` values that have already been transferred +to a different isolation domain. + +This change makes the motivating example valid code, because the `client` +variable does not have any further uses after it's transferred to the +`ClientStore.shared` actor through the call to `addClient`. If we were to modify +`openNewAccount` to call a method on `client` after the call to `addClient`, the +code would be invalid since a non-`Sendable` value that had already been +transferred from a non-isolated context to an actor-isolated context could be +accessed concurrently: + +```swift +func openNewAccount(name: String, initialBalance: Double) async { + let client = Client(name: name, initialBalance: initialBalance) + await ClientStore.shared.addClient(client) + client.logToAuditStream() // Error! Already transferred into clientStore's isolation domain... this could race! +} +``` + +After the call to `addClient`, any other non-`Sendable` value that is statically +proven to be impossible to reference from `client` can still be used safely. We +can prove this property using the concept of *isolation regions*. An isolation +region is a set of values that can only ever be referenced through other values +within that set. Formally, two values $x$ and $y$ are defined to be within the +same isolation region at a program point $p$ if: + +1. $x$ may alias $y$ at $p$. +2. $x$ or a property of $x$ might be referenceable from $y$ via chained access of $y$'s properties at $p$. + +This definition ensures that non-`Sendable` values in different isolation +regions can be used concurrently, because any code that uses $x$ cannot affect +$y$. Lets consider a further example: + +```swift +let john = Client(name: "John", initialBalance: 0) +let joanna = Client(name: "Joanna", initialBalance: 0) + +await ClientStore.shared.addClient(john) +await ClientStore.shared.addClient(joanna) // (1) +``` + +The above code creates two new `Client` instances. It's impossible for +`john` to reference `joanna` and vice versa, so these two values belong to +different isolation regions. Values in different isolation regions can be +used concurrently, so the use of `joanna` at `(1)`, which may be executing +concurrently with some code inside `ClientStore.shared` that accesses `john`, +is safe from data races. + +In contrast, if we add a `friend` property to `Client` and assign `joanna` to +`john.friend`: + +```swift +let john = Client(name: "John", initialBalance: 0) +let joanna = Client(name: "Joanna", initialBalance: 0) + +john.friend = joanna // (1) + +await ClientStore.shared.addClient(john) +await ClientStore.shared.addClient(joanna) // (2) +``` + +After the assignment at point `(1)`, `joanna` can be referenced through +`john.friend`, so `john` and `joanna` must be in the same isolation region at +`(1)`. The access to `joanna` at point `(2)` can be executing concurrently with +code inside `ClientStore.shared` that accesses `john.friend`. Using `joanna` at +point `(2)` is diagnosed as a potential data race. + +## Detailed Design + +NOTE: While this proposal contains rigorous details that enable the compiler to +prove the absence of data races, programmers will not have to reason about +regions at this level of detail. The compiler will allow transfers of non-`Sendable` values between +isolation domains where it can prove they are safe and will emit diagnostics +when it cannot at potential concurrent access points so that programmers don't +have to reason through the data flow themselves. + +### Isolation Regions + +#### Definitions + +An *isolation region* is a set of non-`Sendable` values that can only be aliased +or reachable from values that are within the isolation region. An isolation +region can be associated with a specific *isolation domain* associated with a +task, protected by an actor instance or a global actor, or disconnected from any +specific isolation domain. As the program executes, each isolation region can be +merged with other isolation regions as new values begin to alias or be reachable +from each other. + +Isolation regions and isolation domains are not concepts that are explicitly +denoted in source code. To help explain the concepts throughout this proposal, +isolation regions and their isolation domains will be written in comments in +the following notation: + +* `[(a)]`: A single disconnected region with a single value. + +* `[{(a), actorInstance}]`: A single region that is isolated to actorInstance. + +* `[(a), {(b), actorInstance}]`: Two values in separate isolation regions. a's + region is disconnected but b's region is assigned to the isolation domain of + the actor instance `actorInstance`. + +* `[{(x, y), @OtherActor}, (z), (w, t)]`: Five values in three separate + isolation regions. `x` and `y` are within one isolation region that is + isolated to the global actor `@OtherActor`. `z` is within its own + disconnected isolation region. `w` and `t` are within the same disconnected + region. + +* `[{(a), Task1}]`: A single region that is part of `Task1`'s + isolation domain. + +#### Rules for Merging Isolation Regions + +Isolation regions are merged together when the program introduces a potential +alias or access path to another value. This can happen through function calls, +and assignments. Many expression forms are sugar for a function application, +including property accesses. + +Given a function $f$ with arguments $a_{i}$ and result that is assigned to +variable $y$: + +$$ +y = f(a_{0}, ..., a_{n}) +$$ + +1. All regions of non-`Sendable` arguments $a_{i}$ are merged into one larger + region after $f$ executes. +2. If any of $a_{i}$ are non-`Sendable` and $y$ is non-`Sendable`, then $y$ is in + the same merged region as $a_{i}$. If all of the $a_{i}$ are `Sendable`, + then $y$ is within a new disconnected region that consists only of $y$. +3. If $y$ is not a new variable, i.e. it's mutable, then + + a) If $y$ was previously captured by reference in a closure, then the assignment + to $y$ merges $y$'s new region into its old region. + + b) If $y$ was not captured by reference, then $y$'s old region is + forgotten. + +The above rules are conservative; without any further annotations, we must assume: +* In the implementation of $f$, any $a_{i}$ could become reachable from $a_{j}$. +* $y$ could be one of the $a_{i}$ values or alias contents of $a_{i}$. +* If $y$ was captured by reference in a closure and then assigned a new value, + calling the closure could reference $y$'s new value. + +See the future directions section for additional annotations that enable more +precise regions. + +##### Examples + +Now lets apply these rules to some specific examples in Swift code: + +* **Initializing a `let` or `var` binding**. ``let y = x, var y = x``. Initializing + a let or var binding `y` with `x` results in `y` being in the same region as + `x`. This follows from rule `(2)` since formally a copy is equivalent to calling a + function that accepts `x` and returns a copy of `x`. + + ```swift + func bindingInitialization() { + let x = NonSendable() + // Regions: [(x)] + let y = x + // Regions: [(x, y)] + let z = consume x + // Regions: [(x, y, z)] + } + ``` + + Note that whether or not `x` is in the region after `consume x` does not + change program semantics. A valid program must still obey the no-reuse + constraints of `consume`. + +* **Assigning a `var` binding**. ``y = x``. Assigning a var binding `y` with `x` + results in `y` being in the same region as `x`. If `y` is not captured by + reference in a closure, then `y`'s previous assigned region is forgotten due + to `(3)(b)`: + + ```swift + func mutableBindingAssignmentSimple() { + var x = NonSendable() + // Regions: [(x)] + let y = NonSendable() + // Regions: [(x), (y)] + x = y + // Regions: [(x, y)] + let z = NonSendable() + // Regions: [(x, y), (z)] + x = z + // Regions: [(y), (x, z)] + } + ``` + + In contrast if `y` was captured in a closure by reference, then `y`'s former + region is merged with the region of `x` due to `(3)(a)`. + + ```swift + // Since we pass x as inout in the closure, the closure has to capture x by + // reference. + func mutableBindingAssignmentClosure() { + var x = NonSendable() + // Regions: [(x)] + let closure = { useInOut(&x) } + // Regions: [(x, closure)] + let y = NonSendable() + // Regions: [(x, closure), (y)] + x = y + // Regions: [(x, closure, y)] + } + ``` + +* **Accessing a non-`Sendable` property of a non-`Sendable` value**. + ``let y = x.f``. Accessing a property `f` on a non-`Sendable` value `x` + results in a value `y` that must be in the same region as `x`. This follows + from `(2)` since formally a property access is equivalent to calling a getter + passing `x` as `self`. Importantly, this property forces all non-`Sendable` + types to form one large region containing their non-`Sendable` state: + + ```swift + func assignFieldToValue() { + let x = NonSendableStruct() + // Regions: [(x)] + let y = x.field + // Regions: [(x, y)] + } + ``` + +* **Setting a non-`Sendable` property of a non-`Sendable` value**. ``y.f = x`` + Assigning `x` into a property `y.f` results in `y` and `y.f` being in the + same region as `x`. This again follows from `(2)`: + + ```swift + func assignValueToField() { + let x = NonSendableStruct() + // Regions: [(x)] + let y = NonSendable() + // Regions: [(x), (y)] + x.field = y + // Regions: [(x, y)] + } + ``` + +* **Capturing non-`Sendable` values by reference in a closure**. ``closure = { + useX(x); useY(y) }``. Capturing non-`Sendable` values `x` and `y` results in + `x` and `y` being in the same region. This is a consequence of `(2)` since + `x` and `y` are formally arguments to the closure formation. This + also means that the closure must be part of that same region: + + ```swift + func captureInClosure() { + let x = NonSendable() + let y = NonSendable() + // Regions: [(x), (y)] + let closure = { print(x); print(y) } + // Regions: [(x, y, closure)] + } + ``` + +* **Function arguments in the body of a function**. Given a function `func + transfer(x: NonSendable, y: NonSendable) async`, in the body of + `transfer`, `x` and `y` are considered to be within the same region. Since + `self` is a function argument to methods, this implies that when `self` is + non-`Sendable` all method arguments must be in the same region as `self`: + + ```swift + func transfer(x: NonSendable, y: NonSendable) { + // Regions: [(x, y)] + let z = NonSendable() + // Regions: [(x, y), (z)] + f(x, z) + // Regions: [(x, y, z)] + } + ``` + +#### Control Flow + +Isolation regions are also affected by control flow. Let $x$ and $y$ +be two values that are used in a control flow statement. After the +control flow statement, the regions of $x$ and $y$ are merged if any +of the blocks within the statement merge the regions of $x$ and $y$. +For example: + +```swift +// Regions: [(x), (y)] +var x: NonSendable? = NonSendable() +var y: NonSendable? = NonSendable() +if ... { + // Regions: [(x), (y)] + x = y + // Regions: [(x, y)] +} else { + // Regions: [(x), (y)] +} + +// Regions: [(x, y)] +``` + +Because the first block of the `if` statement assigns `x` to `y`, causing +their regions to be merged within that block, `x` and `y` are in the +same region after the `if` statement. + +This rule is conservative since it is always safe to consider two values +that are disconnected from each other as if they are isolated together. The +only effect would be the rejection of programs that we otherwise could accept. + +The above description of regions naturally allows the definition of an +optimistic forward dataflow problem that allows us to determine at every point +of the program the isolation region that a value belongs to. We outline this +dataflow in more detail in an [appendix](#isolation-region-dataflow) to this proposal. + +### Transferring Values and Isolation Regions + +As defined above, all non-`Sendable` values in a Swift program belong to some +isolation region. An isolation region is isolated to an actor's isolation +domain, a task's isolation domain, or disconnected from any specific isolation +domain: + +```swift +actor Actor { + // 'field' is in an isolation region that is isolated to the actor instance. + var field: NonSendable + + func method() { + // 'ns' is in a disconnected isolation region. + let ns = NonSendable() + } +} + +func nonisolatedFunction() async { + // 'ns' is in a disconnected isolation region. + let ns = NonSendable() +} + +// 'globalVariable' is in a region that is isolated to @GlobalActor. +@GlobalActor var globalVariable: NonSendable + +// 'x' is isolated to the task that calls taskIsolatedArgument. +func taskIsolatedArgument(_ x: NonSendable) async { ... } +``` + +As the program executes, an isolation region can be passed across isolation +boundaries, but an isolation region can never be accessed by multiple +isolation domains at once. When a region $R_{1}$ is merged into another region +$R_{2}$ that is isolated to an actor, $R_{1}$ becomes protected by +that isolation domain and cannot be passed or accessed across isolation +boundaries again. + +The following code example demonstrates merging a disconnected region into a +region that is `@MainActor` isolated: + +```swift +@MainActor func transferToMainActor(_ t: T) async { ... } + +func assigningIsolationDomainsToIsolationRegions() async { + // Regions: [] + + let x = NonSendable() + // Regions: [(x)] + + let y = x + // Regions: [(x, y)] + + await transferToMainActor(x) + // Regions: [{(x, y), @MainActor}] + + print(y) // Error! +} +``` + +Passing `x` into `transferToMainActor` introduces a potential alias to `x` +from any `@MainActor`-isolated state, because the implementation of +`transferToMainActor` can store `x` into any state within that isolation +domain. So, the region containing `x` must be merged into the `@MainActor`'s +region. Accessing `y` after that merge is an error because `x` and `y` are now +both effectively `@MainActor` isolated, and the access occurs from outside the +`@MainActor`. + +Formally, when we pass a non-`Sendable` value $v$ into a function $f$ and the +call to $f$ crosses an isolation boundary, then we say that $v$ and $v$'s +region are *transferred* into $f$. During the execution of $f$, the only way to +reference $v$ or any value in the same region as $v$ is through the parameter +bound to $v$ in the implementation of $f$. This deep structural isolation +guarantees that values in a region cannot be accessed concurrently. + +In this proposal, we are defining the default convention for passing +non-`Sendable` values across isolation boundaries as being a transfer +operation. This does not apply when calling async functions from within the same +isolation domain. To do so would require an explicit transferring modifier which +is described in the [Future Directions](#transferring-parameters) section below. + +### Taxonomy of Isolation Regions + +There are four types of isolation regions that a non-`Sendable` value can belong +to that determine the rules for transferring value over an isolation boundary. + +#### Disconnected Isolation Regions + +A *disconnected isolation region* is a region that consists only of +non-`Sendable` values and is not associated with a specific isolation +domain. A value in a disconnected region can be transferred to another +isolation domain as long as the value is used uniquely by said isolation +domain and never used later outside of that isolation domain lest we introduce +races: + +```swift +@MainActor func transferToMainActor(_ t: T) async { ... } + +actor Actor { + func method() async { + let x = NonSendable() + // Regions: [(x)] + + await transferToMainActor(x) + // Regions: [{(x), @MainActor}] + + print(x) // Error! x being used outside of @MainActor isolated code. + } +} +``` + +#### Actor Isolated Regions + +An *actor isolated region* is a region that is strongly bound to a specific +actor's isolation domain. Since the region is tied to an actor's isolation +domain, the values of the region can *never* be transferred into another +isolation domain since that would cause the non-`Sendable` value to be used by +code both inside and outside the actor's isolation domain allowing for races: + +```swift +actor Actor { + var nonSendable: NonSendable +} + +@MainActor func actorRegionExample() async { + let a = Actor() + // Regions: [{(a.nonSendable), a}] + + let x = await a.nonSendable // Error! + + await transferToMainActor(a.nonSendable) // Error! +} +``` + +In the above code example, `x` must be in the actor `a`'s region because it +aliases actor-isolated state, making `x` effectively isolated to `a`. The +initialization is invalid, because `x` is not usable from a `@MainActor` +context. Similarly, attempting to transfer actor-isolated state into another +isolation domain is invalid. + +The parameters of an actor method or a global actor isolated function are +considered to be within the actor's region. This is since a caller can pass +actor isolated state as an argument to such a method or function. This implies +that parameters of actor isolated methods and functions can not be transferred +like other values in actor isolation regions. + +The objects that make up an actor region varies depending on the kind of actor: + +* **Actor**. An actor region for an actor contains the actor's non-`Sendable` + fields and any values derived from the actor's fields. + + ```swift + class NonSendableLinkedList { + var next: NonSendableLinkedList? + } + + actor Actor { + var listHead: NonSendableLinkedList + + func method() async { + // Regions: [{(self.listHead, self.listHead.next, ...), self}] + + let x = self.listHead + // Regions: [{(x, self.listHead, self.listHead.next, ...), self}] + + let z = self.listHead.next! + // Regions: [{(x, z, self.listHead, self.listHead.next, ...), self}] + ... + } + } + ``` + + In the above example, `x` is in `self`'s region because it aliases + non-`Sendable` state isolated to `self`, and `z` is in `self`'s region + because the value of `next` is reachable from `self.listHead`. + +* **Global Actor**. An actor region for a global actor contains any global + variables isolated to the global actor, all instances of nominal types + isolated to the global actor, and all values derived from the fields of the + isolated global variable or nominal types. + + ```swift + @GlobalActor var firstList: NonSendableLinkedList + @GlobalActor var secondList: NonSendableLinkedList + + @GlobalActor func useGlobalActor() async { + // Regions: [{(firstList, secondList), @GlobalActor}] + + let x = firstList + // Regions: [{(x, firstList, secondList), @GlobalActor}] + + let y = secondList.listHead.next! + // Regions: [{(x, firstList, secondList, y), @GlobalActor}] + ... + } + ``` + + In the above code example `x` is in `@GlobalActor`'s region because it + aliases `@GlobalActor`-isolated state, and `y` is in `@GlobalActor`'s region + because it aliases a value that's reachable from `@GlobalActor`-isolated + state. + +An operation to disconnect a value from an actor region in order to transfer +it to another isolation domain is out of the scope of this proposal. A +potential extension to enable this is described in the [Future Directions](disconnected-fields-and-the-disconnect-operator). + +#### Task Isolated Regions + +A task isolated isolation region consists of values that are isolated to a +specific task. This can only occur today in the form of the parameters of +nonisolated asynchronous functions since unlike actors, tasks do not have +non-`Sendable` state that can be isolated to them. Similarly to actor isolated +regions, a task isolated region is strongly tied to the task so values within +the task isolated region cannot be transferred out of the task: + +```swift +@MainActor func transferToMainActor(_ x: NonSendable) async { ... } + +func nonIsolatedCallee(_ x: NonSendable) async { ... } + +func nonIsolatedCaller(_ x: NonSendable) async { + // Regions: [{(x), Task1}] + + // Not a transfer! Same Task! + await nonIsolatedCallee(x) + + // Error! + await transferToMainActor(x) +} +``` + +In the example above, `x` is in a task isolated region. Since +`nonIsolatedCallee` will execute on the same task as `nonIsolatedCaller`, they +are in the same isolation domain and a transfer does not occur. In contrast, +`transferToMainActor` is in a different isolation domain so passing `x` to it is +a transfer resulting in an error. + +#### Invalid Isolation Regions + +An invalid isolation region is a region that results from conditional control +flow causing the merging of regions that can never be merged together due to +isolation properties. It is an error to use a value that is in an invalid +isolation region since statically the specific region that the value belongs to +can not be determined: + +```swift +func mergeTwoActorRegions() async { + let a1 = Actor() + // Regions: [{(), a1}] + let a2 = Actor() + // Regions: [{(), a1}, {(), a2}] + let x = NonSendable() + // Regions: [{(), a1}, {(), a2}, (x)] + + if await boolean { + await a1.useNS(x) + // Regions: [{(x), a1}, {(), a2}] + } else { + await a2.useNS(x) + // Regions: [{(), a1}, {(x), a2}] + } + + // Regions: [{(x), invalid}, {(), a1}, {(), a2}] +} +``` + +#### Merging Isolation Regions + +The behavior of merging two isolation regions depends on the kind of each +region. + +* **Disconnected and Disconnected**. Given two non-`Sendable` values in separate + disconnected regions, merging the regions produces one large disconnected + region. + + ```swift + let x = NonSendable() + // Regions: [(x)] + let y = NonSendable() + // Regions: [(x), (y)] + useValue(x, y) + // Regions: [(x, y)] + ``` + +* **Disconnected and Actor Isolated**. Merging a disconnected region and an + actor-isolated region expands the actor-isolated region with the values in + the disconnected region. This forces all values in the disconnected region + to be treated as if they are isolated to the actor. This can only occur when + calling a method on an actor or assigning into an actor's field: + + ```swift + func example1() async { + let x = NonSendable() + // Regions : [(x)] + + let a = Actor() + // Regions: [(x), {(a.field), a}] + + await a.useNonSendable(x) + // Regions: [{(x, a.field), a}] + + useValue(x) // Error! 'x' is effectively isolated to 'a' + + let y = NonSendable() + // Regions: [{(x, a.field), a}, (y)] + + a.field = y + // Regions: [{(x, a.field, y), a}] + + useValue(y) // Error! 'y' is effectively isolated to 'a' + } + ``` + +* **Disconnected and Task isolated**. Merging a disconnected region and a + task-isolated region expands the task-isolated region with the values in the + disconnected region. This forces all values in the disconnected region to be + treated like they are isolated to the task: + + ```swift + func nonIsolated(_ arg: NonSendable) async { + // Regions: [{(arg), Task1}] + let x = NonSendable() + // Regions: [{(arg), Task1}, (x)] + arg.doSomething(x) + // Regions: [{(arg, x), Task1}] + await transferToMainActor(x) // Error! 'x' is isolated to 'Task1' + } + ``` + +* **Actor isolated and Actor isolated**. Merging two actor-isolated regions + results in an invalid region. This can only occur via conditional control flow + since an actor isolated region cannot be transferred into another actor's + isolation region: + + ```swift + func test() async { + let a1 = Actor() + // Regions: [{(), a1}] + let a2 = Actor() + // Regions: [{(), a1}, {(), a2}] + let x = NonSendable() + // Regions: [{(), a1}, {(), a2}, (x)] + + if await boolean { + await a1.useNS(x) + // Regions: [{(x), a1}, {(), a2}] + } else { + await a2.useNS(x) + // Regions: [{(), a1}, {(x), a2}] + } + + // Regions: [{(x), invalid}, {(), a1}, {(), a2}] + } + ``` + + In the above example, `x` cannot be accessed from `test` after the `if` + statement since `x` is now in an invalid isolation domain. + +* **Actor Isolated and Task Isolated**. Merging an actor isolated region and + task isolated region results in an invalid isolation region. This occurs since + an actor isolated region and a task isolated region can run concurrently from + each other. Since values in either type of region cannot be transferred, this + can only occur through conditional control flow: + + ```swift + func nonIsolated(_ arg: NonSendable) async { + // Regions: [{(arg), Task1}] + let a = Actor() + // Regions: [{(), a}, {(arg), Task1}] + let x = NonSendable() + // Regions: [(x), {(), a}, {(arg), Task1}] + + if await boolean { + await a.useNS(x) + // Regions: [{(x), a}, {(arg), Task1}] + } else { + arg.useNS(x) + // Regions: [{(), a}, {(arg, x), Task1}] + } + + // Regions: [{(arg, x), invalid}, {(), a}, {(), Task1}] + } + ``` + +* **Task Isolated and Task Isolated**. Since task isolated isolation regions are + only introduced due to function arguments, it is impossible to have two + separate task isolated regions that could be merged. + +### Weak Transfers, `nonisolated` functions, and disconnected isolation regions + +When we transfer a value over an isolation boundary, the caller according to the +ownership conventions of Swift may still own the value despite it being illegal +for the caller to use the value due to region based isolation: + +```swift +class NonSendable { + deinit { print("deinit was called") } +} + +@MainActor func transferToMainActor(_ t: T) async { } + +actor MyActor { + func example() async { + // Regions: [{(), self}] + let x = NonSendable() + + // Regions: [(x), {(), self}] + await transferToMainActor(x) + // Regions: [{(x), @MainActor}, {(), self}] + + // Error! Since 'x' was transferred to @MainActor, we cannot use 'x' + // directly here. + useValue(x) // (1) + + print("After nonisolated callee") + + // But since example still owns 'x', the lifetime of 'x' ends here. (2) + } +} + +let a = MyActor() +await a.example() +``` + +In the above example, the program will first print out "After nonisolated +callee" and then "deinit was called". This is because even though +`nonIsolatedCallee` is transferred `x`'s region, `x` is still passed to +`nonIsolatedCallee` using Swift's default guaranteed ownership convention. This +implies that the caller from an ownership perspective still owns the memory of +the class implying the lifetime of `x` actually ends at `(1)` despite the caller +not being able to use `x` directly at that point. + +This illustrates how the transfer convention used when passing a value over an +isolation boundary is a *weak transfer* convention. A weak transfer convention +implies that one can still reference a value within the transferred region from +the original isolation domain, but one cannot access the value through the +reference. In contrast, a *strong transfer* convention would require that the +caller isolation domain cannot maintain even references to values in the +transferred isolation region. This would require transferring to always be a +1 +operation since to preserve this property we would always need to pass off +ownership from the caller to the callee to ensure that the callee cleans up the +region as shown in the example above. + +Requiring our transfer convention to be a strong convention would have several +unfortunate side-effects: + +* All async functions would by default take their parameters as owned. This + would be an ABI break and would also have the unfortunate consequence that the + bodies of asynchronous functions could never be marked as readonly or readnone + since they may need to invoke a deinit to end ownership of a value and deinits + may have unknown side-effects. + +* This would hurt the performance of asynchronous functions by increasing the + amount of ARC overhead required since unless we inline, there will be a cross + function call boundary copy that can not be eliminated. This in turn would + cause hits to code-size since to remedy this performance problem the inliner + would need to be more aggressive about inlining code. + +To achieve a *strong transfer* convention, one can use the *transferring* function +parameter annotation. Please see extensions below for more information about +*transferring*. + +Since our transfer convention is weak, a disconnected isolation region that +was transferred into an isolation domain can be used again if the isolation +domain no longer maintains any references to the region. This occurs with +`nonisolated` asynchronous functions. When we transfer a disconnected value into +a `nonisolated` asynchronous functions, the value becomes part of the function's +task isolated isolation domain for the duration of the function's +execution. Once the function finishes executing, we know that the value is no +longer isolated to the function since: + +* A `nonisolated` function does not have any non-temporary isolated state of its + own that the non-`Sendable` value could escape into. + +* Parameters in a task isolated isolation region cannot be transferred into a + different isolation domain that does have persistent isolated state. + +Thus the value in the caller's region again becomes disconnected once more and +thus can be used after the function returns and be transferred again: + +```swift +func nonIsolatedCallee(_ x: NonSendable) async { ... } +func useValue(_ x: NonSendable) { ... } +@MainActor func transferToMainActor(_ t: T) { ... } + +actor MyActor { + var state: NonSendable + + func example() async { + // Regions: [{(), self}] + + let x = NonSendable() + // Regions: [(x), {(), self}] + + // While nonIsolatedCallee executes the regions are: + // Regions: [{(x), Task}, {(), self}] + await nonIsolatedCallee(x) + // Once it has finished executing, 'x' is disconnected again + // Regions: [(x), {(), self}] + + // 'x' can be used since it is disconnected again. + useValue(x) // (1) + + // 'x' can be transferred since it is disconnected again. + await transferToMainActor(x) // (2) + + // Error! After transferring to main actor, permanently + // in main actor, so we can't use it. + useValue(x) // (3) + } +} +``` + +In the example above, we transfer `x` into `nonIsolatedCallee` and while +`nonIsolatedCallee` is executing are not allowed to access `x` in the +caller. Since `nonIsolatedCallee`'s execution ends immediately after it is +called, we are then allowed to use `x` again. + +### non-`Sendable` Closures + +Currently non-`Sendable` closures like other non-`Sendable` values are not +allowed to be passed over isolation boundaries since they may have captured +state from within the isolation domain in which the closure is defined. We would +like to loosen these rules. + +#### Captures + +A non-`Sendable` closure's region is the merge of its non-`Sendable` captured +parameters. As such a nonisolated non-`Sendable` closure that only captures +values that are in disconnected regions must itself be in a disconnected region +and can be transferred: + +```swift +let x = NonSendable() +// Regions: [(x)] +let y = NonSendable() +// Regions: [(x), (y)] +let closure = { useValues(x, y) } +// Regions: [(x, y, closure)] +await transferToMain(closure) // Ok to transfer! +// Regions: [{(x, y, closure), @MainActor}] +``` + +A non-`Sendable` closure that captures an actor-isolated value is considered to +be within the actor-isolated region of the value: + +```swift +actor MyActor { + var ns = NonSendable() + + func doSomething() { + let closure = { print(self.ns) } + // Regions: [{(closure, self.ns), self}] + await transferToMain(closure) // Error! Cannot transfer value in actor region. + } +} +``` + +When a non-`Sendable` value is captured by an actor-isolated non-`Sendable` +closure, we treat the value as being transferred into the actor isolation domain +since the value is now able to merged into actor-isolated state: + +```swift +@MainActor var nonSendableGlobal = NonSendable() + +func globalActorIsolatedClosureTransfersExample() { + let x = NonSendable() + // Regions: [(x), {(nonSendableGlobal), MainActor}] + let closure = { @MainActor in + nonSendableGlobal = x // Error! x is transferred into @MainActor and then accessed later. + } + // Regions: [{(nonSendableGlobal, x, closure), MainActor}] + useValue(x) // Later access is here +} + +actor MyActor { + var field = NonSendable() + + func closureThatCapturesActorIsolatedStateTransfersExample() { + let x = NonSendable() + // Regions: [(x), {(nonSendableGlobal), MainActor}] + let closure = { + self.field.doSomething() + x.doSomething() // Error! x is transferred into @MainActor and then accessed later. + } + // Regions: [{(nonSendableGlobal, x, closure), MainActor}] + useValue(x) // Later access is here + } +} +``` + +Importantly this ensures that APIs like `assumeIsolated` that take an +actor-isolated closure argument cannot introduce races by transferring function +parameters of nonisolated functions into an isolated closure: + +```swift +actor ContainsNonSendable { + var ns: NonSendableType = .init() + + nonisolated func unsafeSet(_ ns: NonSendableType) { + self.assumeIsolated { isolatedSelf in + isolatedSelf.ns = ns // Error! Cannot transfer a parameter! + } + } +} + +func assumeIsolatedError(actor: ContainsNonSendable) async { + let x = NonSendableType() + actor.unsafeSet(x) + useValue(x) // Race is here +} +``` + +Within the body of a non-`Sendable` closure, the closure and its non-`Sendable` +captures are treated as being Task isolated since just like a parameter, both +the closure and the captures may have uses in their caller: + +```swift +var x = NonSendable() +var closure = {} +closure = { + await transferToMain(x) // Error! Cannot transfer Task isolated value! + await transferToMain(closure) // Error! Cannot transfer Task isolated value! +} +``` + +#### Transferring + +A nonisolated non-`Sendable` synchronous or asynchronous closure that is in a +disconnected region can be transferred into another isolation domain if the +closure's region is never used again locally: + +```swift +extension MyActor { + func synchronousNonIsolatedNonSendableClosure() async { + // This is non-Sendable and nonisolated since it does not capture MyActor or + // any field of my actor. + let nonSendable = NonSendable() + let closure: () -> () = { + print("I am in a closure: \(nonSendable.name)") + } + + // We can safely transfer closure. + await transferClosure(closure) + + // If we were to invoke closure again, an error diagnostic would be + // emitted. + closure() // Error! + + // If we were to access nonSendable, an error diagnostic would be + // emitted. + nonSendable.doSomething() // Error! + } +} +``` + +An actor-isolated synchronous non-`Sendable` closure cannot be transferred to a +callsite that expects a synchronous closure. This is because as part of +transferring the closure, we have erased the specific isolation domain that the +closure was isolated to, so we cannot guarantee that we will invoke the value in +the actor's isolation domain: + +```swift +@MainActor func transferClosure(_ f: () -> ()) async { ... } + +extension Actor { + func isolatedClosure() async { + // This closure is isolated to actor since it captures self. + let closure: () -> () = { + self.doSomething() + } + + // When we transfer the closure, we have lost the specific actor that + // the closure belongs to so an error must be emitted! + await transferClosure(closure) // Error! + } +} +``` + +We may be able to accept this code in the future if we allowed for isolated +synchronous closures to propagate around the specific isolation domain that they +belonged to and dynamically swap to it. We discuss *dynamic isolation domains* +as an extension below. + +In contrast, one can transfer an actor-isolated synchronous non-`Sendable` +closure at a call site that expects an asynchronous function argument. This is +because the closure will be wrapped into an asynchronous thunk that will hop +onto the defining isolation domain of the closure: + +```swift +@MainActor func transferClosure(_ f: () async -> ()) async { ... } + +extension Actor { + func isolatedClosure() async { + // This closure is isolated to actor since it captures self. + let closure: () -> () = { + self.doSomething() + } + + // As part of transferring the closure, the closure is wrapped into an + // asynchronous thunk that will hop onto the Actor's executor. + await transferClosure(closure) + } +} +``` + +In the example above, since the closure is wrapped in the asynchronous thunk and +that thunk hops onto the Actor's executor before calling the closure, we know +that isolation to the actor is preserved when we call the synchronous closure. + +An actor-isolated asynchronous non-`Sendable` closure can be transferred since +upon the closure's invocation, we will always hop into the actor's isolation +domain: + +```swift +extension Actor { + func isolatedClosure() async { + // This async closure is isolated to actor since it captures self. + let closure: () async -> () = { + self.doSomething() + } + + // Since the closure is async, we can transfer it as much as we want + // since we will always invoke the closure within the actor's isolation + // domain... + await transferClosure(closure) + + // ... so this is safe as well. + await transferClosure(closure) + } +} +``` + +#### Closures and Global Actors + +If a closure uses values that are isolated from a global actor in any way, we +assume that the closure must also be isolated to that global actor: + +```swift +@MainActor func mainActorUtility() {} + +@MainActor func mainActorIsolatedClosure() async { + let closure = { + mainActorUtility() + } + // Regions: [{(closure), @MainActor}] + await transferToCustomActor(closure) // Error! +} +``` + +If `mainActorUtility` was not called within `closure`'s body then `closure` +would be disconnected and could be transferred: + +```swift +@MainActor func mainActorUtility() {} + +@MainActor func mainActorIsolatedClosure() async { + let closure = { + ... + } + // Regions: [(closure)] + await transferToCustomActor(closure) // Ok! +} +``` + +### KeyPath + +A non-`Sendable` keypath that is not actor-isolated is considered to be +disconnected and can be transferred into an isolation domain as long as the +value's region is not reused again locally: + +```swift +class Person { + var name = "John Smith" +} + +class Wrapper { + var root: Root + init(root: Root) { self.root = root } + func setKeyPath(_ keyPath: ReferenceWritableKeyPath, to value: T) { + root[keyPath: keyPath] = value + } +} + +func useNonIsolatedKeyPath() async { + let nonIsolated = Person() + // Regions: [(nonIsolated)] + let wrapper = Wrapper(root: nonIsolated) + // Regions: [(nonIsolated, wrapper)] + let keyPath = \Person.name + // Regions: [(nonIsolated, wrapper, keyPath)] + await transferToMain(keyPath) // Ok! + await wrapper.setKeyPath(keyPath, to: "Jenny Smith") // Error! +} +``` + +A non-`Sendable` keypath that is actor-isolated is considered to be in the +actor's isolation domain and as such cannot be transferred out of the actor's +isolation domain: + +```swift +@MainActor +final class MainActorIsolatedKlass { + var name = "John Smith" +} + +@MainActor +func useKeyPath() async { + let actorIsolatedKlass = MainActorIsolatedKlass() + // Regions: [{(actorIsolatedKlass.name), @MainActor}] + let wrapper = Wrapper(root: actorIsolatedKlass) + // Regions: [{(actorIsolatedKlass.name), @MainActor}] + let keyPath = \MainActorIsolatedKlass.name + // Regions: [{(actorIsolatedKlass.name, keyPath), @MainActor}] + await wrapper.setKeyPath(keyPath, to: "value") // Error! Cannot pass non-`Sendable` + // keypath out of actor isolated domain. +} +``` + +If a KeyPath captures any values then the KeyPath's region consists of a merge +of the captured values regions combined with the actor-isolation region of the +KeyPath if the KeyPath is isolated to an actor: + +```swift +class NonSendableType { + subscript(_ t: T) -> Bool { ... } +} + +func keyPathInActorIsolatedRegionDueToCapture() async { + let mainActorKlass = MainActorIsolatedKlass() + // Regions: [{(mainActorKlass), @MainActor}] + let keyPath = \NonSendableType.[mainActorKlass] + // Regions: [{(mainActorKlass, keyPath), @MainActor}] + await transferToMainActor(keyPath) // Error! Cannot transfer keypath in actor isolated region! +} + +func keyPathInDisconnectedRegionDueToCapture() async { + let ns = NonSendableType() + // Regions: [(ns)] + let keyPath = \NonSendableType.[ns] + // Regions: [(ns, keyPath)] + await transferToMainActor(ns) + useValue(keyPath) // Error! Use of keyPath after transferring ns +} +``` + +### Async Let + +When an async let binding is initialized with an expression that uses a +disconnected non-`Sendable` value, the value is treated as being transferred +into a `nonisolated` asynchronous callee that additionally allows for the value +to be transferred. If the value is used only by synchronous code and +`nonisolated` asynchronous functions, we allow for the value to be reused again +once the async let binding has been awaited upon: + +```swift +func nonIsolatedCallee(_ x: NonSendable) async -> Int { 5 } + +actor MyActor { + func example() async { + // Regions: [{(), self}] + let x = NonSendable() + // Regions: [(x), {(), self}] + async let value = nonIsolatedCallee(x) + x.integerField + // Regions: [{(x), Task}, {(), self}] + useValue(x) // Error! Illegal to use x here. + await value + // Regions: [(x), {(), self}] + useValue(x) // Ok! x is disconnected again so it can be used... + await transferToMainActor(x) // and even transferred to another actor. + } +} +``` + +If the disconnected value is transferred into an actor region, the value is +treated as if the value was transferred into the actor region at the point where +the async let is declared and is considered transferred even after the async let +has been awaited upon: + +```swift +// Regions: [] +let x = NonSendable() +// Regions: [(x)] +async let y = transferToMainActor(x) // Transferred here. +// Regions: [{(x), @MainActor}] +_ = await y +// Regions: [{(x), @MainActor}] +useValue(x) // Error! x is used after it has been transferred! +``` + +If a disconnected value is reused later in an async let initializer after +transferring it into an actor region, a use after transfer error diagnostic will +be emitted: + +```swift +// Regions: [] +let x = NonSendable() +// Regions: [(x)] +async let y = + transferToMainActorAndReturnInt(x) + + useValueAndReturnInt(x) // Error! Cannot use x after it has been transferred! +``` + +Since a disconnected value can only be transferred into one async let binding at +a time, a use after transfer diagnostic will be emitted if one initializes +multiple async let bindings in one statement with the same non-`Sendable` +disconnected value: + +```swift +// Regions: [] +let x = NonSendable() +// Regions: [(x)] +async let y = x, + z = x // Error! Cannot use x after it has been transferred! +``` + +A non-`Sendable` value that is in an actor isolation region is never allowed to +be used to initialize an async let binding since values in an async let +binding's initializer are allowed to be transferred into further callees: + +```swift +actor MyActor { + var field = NonSendable() + + func example() async { + // Regions: [{(self.field), self}] + async let value = transferToMainActor(field) // Error! Cannot transfer actor + // isolated field to + // @MainActor! + _ = await value + } +} +``` + +### Using transferring to simplify `nonisolated` actor initializers and actor deinitializers + +In [SE-0327](0327-actor-initializers.md), a flow sensitive diagnostic +was introduced to ensure that one can directly access stored properties of `self` +in `nonisolated` actor designated initializers and actor deinitializers despite +the methods not being isolated to self. The diagnostic set out a model where +initially `nonisolated` self is stated to have a weaker form of isolation that +relies on having exclusive access to self. While self is in that state, one is +allowed to access stored properties of self, but once self has escaped that +property is lost and self becomes nonisolated preventing one from accessing its +stored properties without using synchronization. In this proposal, we subsume +that proposal into the region based isolation model and eliminate the need for a +separate flow sensitive diagnostic. + +In Swift's concurrency model, an actor is Sendable since one can only access the +actor's internal state from the actor's executor. If the actor is nonisolated to +the current function this implies one must hop on to the actor's executor to +safely access state. In the case of an initializer or deinitializer with +nonisolated self, this creates a conundrum since we explicitly want to +initialize or deinitialize self's stored fields without synchronizing by hopping +onto the actor's executor. + +In order to implement these semantics, we model self as entering these methods +as a non-`Sendable` value that is strongly transferred into the method. Since +self is strongly transferred, we know that there cannot be any other references +in the program to self when the method begins executing and thus it is safe to +initially access the internal state of the actor directly. Self must initially +be a non-`Sendable` value since if self's storage can be accessed directly, then +passing self to another task could lead to a race on self's storage. To prevent +this possibility, when self escapes self becomes instantaneously +`Sendable`. Once self is `Sendable`, it is no longer safe to access self's +storage directly: + +```swift +actor Actor { + var nonSendableField: NonSendableType + + // self is passed into init using a strongly transferred convention. This means + // that it is unique and safe to access without worrying about concurrency. + init() { + // At this point, self is non-Sendable and we can access its fields directly. + self.nonSendableField = NonSendableType() + + // self is Sendable once callMethod is executed. This includes in callMethod itself. + self.callMethod() + + // Error! Cannot directly access storage of a Sendable actor. + self.nonSendableField.useValue() + } +} +``` + +In the example above, self starts as a unique non-`Sendable` typed value. Thus +it is safe for us to initialize `self.nonSendableField`. When self is passed +into `callMethod`, self becomes `Sendable`. Since self could have been +transferred to another task by callMethod, it is no longer safe to directly +access self's memory and thus we emit an error when we access +`self.nonSendableField`. + +Deinits work just like inits with one additional rule. Just like with initializers, +self is considered initially to be strongly transferred and non-`Sendable`. One +is allowed to access the `Sendable` stored properties of self while self is +non-`Sendable`. One can access the non-`Sendable` fields of self if one knows +statically that the non-`Sendable` fields are uniquely isolated to the self +instance. For the case of actors, this means that since the actor's state is +completely isolated only to that one actor instance we can touch non-`Sendable` +fields. But in the case of global actor isolated classes this is not true since +other global actor isolated class instances could also have a reference to the +same non-`Sendable` value since all global actor isolated instances are part of +the same isolation region: + +```swift +actor Actor { + var mutableNonSendableField: NonSendableType + let immutableNonSendableField: NonSendableType + var mutableSendableField: SendableType + let immutableSendableField: SendableType + + deinit { + _ = self.immutableSendableField // Ok + _ = self.mutableSendableField // Ok + // Safe to access since no other actor instances + _ = self.mutableNonSendableField // Ok + _ = self.immutableNonSendableField // Ok + + escapeSelfIntoNonIsolated(self) + + _ = self.immutableSendableField // Ok + _ = self.mutableSendableField // Error! Must be immutable. + _ = self.mutableNonSendableField // Error! Must be sendable + _ = self.immutableNonSendableField // Error! Must be sendable + } +} + +@MainActor class GlobalActorIsolatedClass { + var mutableNonSendableField: NonSendableType + let immutableNonSendableField: NonSendableType + var mutableSendableField: SendableType + let immutableSendableField: SendableType + + deinit { + _ = self.immutableSendableField // Ok + _ = self.mutableSendableField // Ok + _ = self.mutableNonSendableField // Error! Must be sendable! + _ = self.immutableNonSendableField // Error! Must be sendable! + + escapeSelfIntoNonIsolated(self) + + _ = self.immutableSendableField // Ok + _ = self.mutableSendableField // Error! Must be immutable! + _ = self.mutableNonSendableField // Error! Must be sendable! + _ = self.immutableNonSendableField // Error! Must be sendable! + } +} +``` + +### Using transferring to pass non-Sendable values to async isolated actor initializers + +In [SE-0327](0327-actor-initializers.md), all initializers with non-`Sendable` +arguments were only allowed to be called by delegating initializers: + +```swift +actor MyActor { + var x: NonSendableType + + // Can call this from anywhere. + init(_ arg: SendableType) { + self.init(NonSendableType(arg)) + } + + // Since this has a non-Sendable type, this designated initializer can only + // be called by other initializers like the delegating init above. + init(_ arg: NonSendableType) { + x = arg + } +} + +func constructActor() { + // Error! Cannot call init with non-`Sendable` argument from outside of + // MyActor. + let a = Actor(NonSendableType()) +} +``` + +Using isolation regions we can loosen this restriction and allow for +non-`Sendable` types to be passed to asynchronous initializers since our region +isolation rules guarantee that the caller will have transferred the value into +the initializer due to the isolation boundary: + +```swift +actor MyActor { + var x: NonSendableType + + init(_ arg: NonSendableType) async { + self.x = arg + } +} + +func makeActor() async -> MyActor { + // Regions: [] + let x = NonSendableType() + // Regions: [(x)] + let a = await MyActor(x) // Ok! + // Regions: [{(x), a}] + return a +} +``` + +In the above example, it is safe to pass `x` into `MyActor` despite `x` being +non-`Sendable` since if we were to use `x` afterwards, the compiler would error +since we would be using `x` from multiple isolation domains: + +```swift +func makeActor() async -> MyActor { + // Regions: [] + let x = NonSendableType() + // Regions: [(x)] + let a = await MyActor(x) // Ok! + // Regions: [{(x), a}] + x.doSomething() // Error! 'x' was transferred to a's isolation domain! + return a +} +``` + +Sadly synchronous initializers without additional work can still only take +`Sendable` types since there is not a guarantee that the non-`Sendable` types +that are passed to it is in its own region. In order to pass a non-`Sendable` +type to a synchronous initializer, one must mark the parameter with the +`transferring` function parameter modifier which is described below in [Future +Directions](#transferring-parameters). + +### Regions Merge when assigning to Struct and Tuple type var like bindings + +In this proposal, regions are not computed in a field sensitive manner. This +means that if we assign into a struct with multiple stored fields or a tuple +with multiple fields then assigning to one field affects the region of the +entire struct and requires us to merge into such types rather than assign since +otherwise we would lose the regions associated with the other fields: + +```swift +struct NonSendableBox { + var s1 = NonSendable() + var s2 = NonSendable() +} + +func mergeWhenAssignIntoMultiFieldStructField() async { + var box = NonSendableBox() + // Regions: [(box.s1, box.s1)] + let x = NonSendable() + // Regions: [(box.s1, box.s2), (x)] + let y = NonSendable() + // Regions: [(box.s1, box.s2), (x), (y)] + box.s1 = x + // Regions: [(box.s1, box.s2, x), (y)] + // If we used an assignment operation instead of a merge operation, + // this would cause us to lose that x was still in box.s1 and thus + // in box's region. + box.s2 = y + // Regions: [(box.s1, box.s2, x, y)] +} +``` + +In the above example, if we were to treat ``box.s2 = y`` as an assignment +instead of merge then we would be removing ``x`` from ``box``'s region which +would be unsound since ``x`` and ``box.s1`` still point at the same +reference. Unfortunately this has the affect that when we overwrite an element +of a var like struct, the previous region assigned to that field would have to +remain in the overall struct/tuple's region: + +```swift +func mergeWhenAssignIntoMultiFieldTupleField() async { + var box = (NonSendable(), NonSendable()) + // Regions: [(box.0, box.1)] + let x = NonSendable() + // Regions: [(box.0, box.1), (x)] + let y = NonSendable() + // Regions: [(box.0, box.1), (x), (y)] + box.0 = x + // Regions: [(box.0, box.1, x), (y)] + box.0 = y (1) + // Regions: [(box.0, box.1, x, y)] +} +``` + +In the above, even though we reassign ``box.0`` from ``x`` to ``y``, since we +must perform a merge, we must have that ``x`` is still in ``box``'s region. If +one assigns over the entire box though, one can still get an assign instead of a +region: + +```swift +func mergeWhenAssignIntoMultiFieldTupleField2() async { + var box = (NonSendable(), NonSendable()) + // Regions: [(box.0, box.1)] + let x = NonSendable() + // Regions: [(box.0, box.1), (x)] + let y = NonSendable() + // Regions: [(box.0, box.1), (x), (y)] + box.0 = x + // Regions: [(box.0, box.1, x), (y)] + box = (y, NonSendable()) + // Regions: [(box.0, box.1, y), (x)] +} +``` + +In order to mitigate this, we are able to be stricter with structs and tuples +that store a single field. In such a case, since the struct/tuple does not have +multiple fields updating the single field does not cause us to lose the region +of any other values: + +```swift +func assignWhenAssignIntoSingleFieldStruct() async { + var box = SingleFieldBox() + // Regions: [(box.field)] + let x = NonSendable() + // Regions: [(box.field), (x)] + let y = NonSendable() + // Regions: [(box.field), (x), (y)] + box.field = x + // Regions: [(box.field, x), (y)] + box.field = y + // Regions: [(box.field, y), (x)] +} +``` + +### Accessing `Sendable` fields of non-`Sendable` types after weak transferring + +Given a non-`Sendable` value `x` that has been weakly transferred, a `Sendable` +field `x.f` can be accessed in the caller after `x`'s transferring if the +compiler can statically prove that there cannot be any writes to `x.f` from +another concurrency domain. This is necessary since although `x.f` is +`Sendable`, if code from another concurrency domain can reference `x` in a +manner that allows for `x.f` to be written to, our initial access to `x.f` could +result in a race. Of course once the access is over, we are safe against races +due to the Sendability of `x.f`'s underlying type. The situations where this +occurs varies in between reference types and value types. We go through the +individual cases below. + +#### Classes + +If `x` is a reference type like a class, we only allow for `Sendable` let fields +of `x` to be accessed. This is safe since a let field can never be modified +after initialization implying that we cannot race on assignment to the field +when attempting to read from the field. We cannot allow for `Sendable` var +fields to be accessed due to the aforementioned possible race caused by another +concurrency domain writing to the `Sendable` field as we attempt to access it: + +```swift +class NonSendable { + let letSendable: SendableType + var varSendable: SendableType + let ns: NonSendable +} + +@MainActor func modifyOnMainActor(_ x: NonSendable) async { + x.varSendable = SendableType() +} + +func example() async { + let x = NonSendable() + await modifyOnMainActor(x) + _ = x.letSendable // This is safe. + _ = x.varSendable // Error! Use after transfer of mutable field that could + // race with a write to x.varSendable in modifyOnMainActor. +} +``` + +#### Immutable Bindings to Value Types + +If `x` is an immutable binding (e.x.: let) to a value type (e.x.: struct, tuple, +enum) then we allow for access to all of `x`'s `Sendable` subtypes. This is safe +because: + +1. `x` will be initialized by copying its initial value. This means that even if + `x`'s initial value is a field of a larger value, any modifications to the + other value will not cause `x`'s fields to point to different values. + +2. When `x` is transferred to a callee, `x` will be passed by value. Thus the + callee will receive a completely new value type albeit with copied + fields. This means that if the callee attempts to modify the value, it will + be modifying the new value instead of our caller value implying that we + cannot race against any assignment when accessing the field in our + caller. + + ```swift + struct NonSendableStruct { + let letSendableField: Sendable + var varSendableField: Sendable + let ns: NonSendable + } + + @MainActor func modifyOnMainActor(_ y: consuming NonSendableStruct) async { + // These assignments only affect our parameter, not x in the callee. + y.varSendableField = Sendable() + y = NonSendableStruct() + } + + func letExample() async { + let x = NonSendableStruct() + + await modifyOnMainActor(x) // Transfer x, giving useValueOnMainActor a + // shallow copy of x. + + // We do not race with the assignment in modifyOnMainActor since the + // assignment is to y, not to x. Since the fields are sendable, once + // we avoid the race on accessing the field, we are safe. + print(x.letSendableField) + print(x.varSendableField) + } + ``` + +3. If `x` is captured by reference, since `x` is a let it will be captured + immutably implying that we cannot write to `x.f`. + +#### Mutable Bindings to Value Types + +If `x` is a mutable binding (e.x.: `var`), then we can follow the same logic as +with our immutable bindings except in the case where `x` is captured by +reference. If `x` is captured by reference, it is captured mutably implying that +when accessing `x.f`, we could race against an assignment to `x.f` in the +closure: + +```swift +struct NonSendableStruct { + let letSendableField: Sendable + var varSendableField: Sendable + let ns: NonSendable +} + +@MainActor func invokeOnMain(_ f: () -> ()) async { + f() +} + +func unsafeMutableReferenceCaptureExample() async { + var x = NonSendableStruct() + let closure = { + x = NonSendableStruct(otherInit: ()) + } + await invokeOnMain(closure) + + _ = x.letSendableField // Error! Could race against write in closure! + _ = x.varSendableField // Error! Could race against write in closure! +} +``` + +This also implies that one cannot access `Sendable` computed properties or +functions later since those routines could perform a read like the above +resulting in a race against a write in the closure. + +## Source compatibility + +Region-based isolation opens up a new data-race safety hole when using APIs +change the static isolation in the implementation of a `nonisolated` function, +such as `assumeIsolated`, because values can become referenced by actor-isolated +state without any indication in the function signature: + +```swift +class NonSendable {} + +@MainActor var globalNonSendable: NonSendable = .init() + +nonisolated func stashIntoMainActor(ns: NonSendable) { + MainActor.assumeIsolated { + globalNonSendable = ns + } +} + +func stashAndTransfer() -> NonSendable { + let ns = NonSendable() + stashIntoMainActor(ns) + Task.detached { + print(ns) + } +} + +@MainActor func transfer() async { + let ns = stashAndTransfer() + await sendSomewhereElse(ns) +} +``` + +Without additional restrictions, the above code would be valid under this proposal, +but it risks a runtime data-race because the value returned from `stashAndTransfer` +is stored in `MainActor`-isolated state and send to another isolation domain to +be accessed concurrently. To close this hole, values must be sent into and out of +`assumeIsolated`. The base region-isolation rules accomplish this by treating +captures of isolated closures as a region merge, and the standard library annotates +`assumeIsolated` as requiring the result type `T` to conform to `Sendable`. This +impacts existing uses of `assumeIsolated`, so the change is staged in as warnings +under complete concurrency checking, which enables `RegionBasedIsolation` by default, +and an error in Swift 6 mode. + +## ABI compatibility + +This has no affect on ABI. + +## Future directions + +### Transferring Parameters + +In the above, we mentioned that the transferring of non-`Sendable` values as +discussed above is a callee side property since when analyzing an async callee, +we do not know if the callee's caller is from a different isolation domain or +not. This means that we must be conservative and treat all function parameters as +being in the same region and prevent transferring of function parameters. + +We could introduce a stronger form of transferring that is applied to a function +argument in the callee's signature and forces all callers to transfer the +parameter even if the caller is synchronous or is async but in the same +isolation domain. + +The transferred parameter is guaranteed to be strongly transferred so we know +that once the callee is called there are no other program visible references to +the value outside of the callee's parameter. The implications of this are: + +* Since the value is strongly isolated, it will be within its own disconnected + region separate from the regions of the other parameters: + + ```swift + actor Actor { + func method(_ x: transferring NonSendable, + _ y : NonSendable, + _ z : NonSendable) async { + // Regions: [(x), {(y, z), self}] + // Safe to transfer x since x is marked as transferring. + await transferToMainActor(x) + } + } + ``` + + +* Regardless of if the callee is synchronous or asynchronous, a non-`Sendable` + value that is passed as a transferring parameter cannot be used again locally. + + ```swift + actor Actor { + func transfer(_ t: transferring T) async {} + func method() async { + let a = NonSendable() + + // Pass a into transfer. Even though we are in the same + // isolation domain as transfer... + await transfer(a) + + // Since we transferred a, we are no longer allowed to use a here. Error! + useValue(a) + } + } + ``` + +* Given an asynchronous function, one can safely transfer the non-`Sendable` + parameter to another asynchronous function with a different isolation domain: + + ```swift + @MainActor func transferToMainActor(_ t: T) async {} + + actor Actor { + func method(_ x: transferring NonSendable) async { + // Regions: [(x)] + // Safe to transfer x since x is marked as transferring. + await transferToMainActor(x) + } + } + ``` + +* Given a transferring parameter of a synchronous function, the parameter's + strongly isolated implies that we can transfer it into `Task.init` or + `Task.detach`. + + ```swift + func someSynchronousFunction(_ x: transferring NonSendable) { + Task { + doSomething(x) + } + } + ``` + + if we did not have the strong isolation, then `x` could still be used in the + caller of `someSynchronousFunction`. + +* Due to the isolation of a transferring parameter, it is legal to have a + non-`Sendable` transferring parameter of a synchronous actor designated + initializer: + + ```swift + actor Actor { + var field: NonSendable + + init(_ x: transferring NonSendable) { + self.field = x + } + } + ``` + + Without the transferring argument modifier on `x`, it would not be safe to + store `x` into `self.field` since it may be introducing a value into the + actor's state that could be raced upon. + +#### Returns Isolated + +As discussed above, if a function takes non-`Sendable` parameters and has a +non-`Sendable` result, then the result is part of the merged region of the +function's parameters. This is not always the appropriate semantics since there +are APIs whose results will be in different regions than their parameters. As an +example of this, consider a function that performs control flow based off of +non-`Sendable` state and then returns a result: + +```swift +func example(_ x: NonSendable) async -> NonSendable? { + if x.boolean { + return NonSendable() + } + return nil +} +``` + +In the above, the result of `example` is a newly initialized value that has no +data dependence on the parameter `x`, but as laid out in this proposal, we +cannot express this. We propose the addition of a new function parameter +modifier called `returnsIsolated` that causes callers to treat the result of a +function as being in a disconnected region regardless of the inputs. As part of +this annotation, we would only allow for the callee to return a value that is in +a disconnected region preventing the returning of function arguments or in the +case of an actor any state related internally to the actor: + +```swift +actor Actor { + var field: NonSendableType + + func getValue() -> @returnsIsolated NonSendableType { + // Regions: [{(self.field), self}] + let x = NonSendableType() + // Regions: [(x), {(self.field), self}] + + if await booleanValue { + // Safe to do since 'x' is in a disconnected region. + return x + } + + // Error! Cannot return a value from the actor's region! + return field + } +} +``` + +Since the value returned is always in its own disconnected region, it can be +used in the caller isolation domain without triggering races: + +```swift +func getValueFromActor(_ a: Actor) async { + // Regions: [{(a.field), a}] + + // This is safe since we know that 'x' is independent of the actor. + let x = await a.getValue() + // Regions: [(x), {(a.field), a}] + + // So we could transfer it to another function if we wanted to. + await transferToMainActor(x) +} +``` + +> NOTE: @returnsIsolated is just a strawman syntax introduced for the purpose of +> expositing this extension. It is not an actual proposed or final syntax. + +#### Disconnected Fields and the Disconnect Operator + +Even though we can use `@returnsIsolated` to return a value from the Actor's +isolation domain, we have not specified a manner to safely return non-`Sendable` +values from the internal state of an Actor or GAIT. To do so, we introduce a new +type of field called a *disconnected field*. A disconnected field of an actor is +an actor isolated region that is separate from the normal actor's region. Since +it is separate from the other region of the actor, it cannot be reachable by the +other fields of the actor... but since it is an actor field, it cannot be +escaped from the actor without doing additional work. In order to escape such a +field, we introduce a new `disconnect` operation that consumes the disconnected +field and returns the field's value as a new disconnected region which is safe +to use as a `@returnsIsolated` result: + +```swift +actor MyActor { + disconnected var x: NonSendableType + + /// Reinitialize a field, returning the old value. + func reinitField() -> @returnsIsolated NonSendableType { + let result = disconnect x + x = NonSendableType() + return result + } +} +``` + +In the above example, we disconnect `x`'s value into `result`, reinitialize `x` +with a fresh value, and return the result. + +> NOTE: We may be able to reuse the `consume` operator for this purpose, but for +> the purposes of framing this as an extension, we introduce a new operator for +> simplicity. + +If the author forgets to update the disconnected field with a new value, a +control flow sensitive error will be emitted: + +```swift +actor MyActor { + disconnected var x: NonSendableType + + func reinitField() -> @returnsIsolated NonSendableType { + let result = disconnect x + + if booleanTest { + x = newValue + } else { + ... + } + + return result + } // Error! Must update disconnected field 'x' along all program paths after disconnecting! +} +``` + +In the above example, we emit an error since along the else path we do not +provide a new value for `x`. + +Since a disconnected field can only be initialized with a value from a +disconnected region implying that a field cannot be assigned to by a parameter +of an actor method unless the parameter is transferred: + +```swift +actor MyActor { + disconnected var x: NonSendableType + + /// Update the internal state to use a new value, returning the old value + func updateValue(_ newValue: transferring NonSendableType) -> @returnsIsolated NonSendableType { + let result = disconnect x + x = newValue + return result + } +} +``` + +since the parameter in the above example is transferred, it has a disconnected +region and thus can be assigned into the disconnected region. + +## Alternatives considered + +### Require users to audit all types for sendability + +We could require users to audit all of their non-`Sendable` types for +Sendability. This would create a large annotation burden on users that this +approach avoids. + +### Force weak transferring to be explicitly marked + +We could require transferred arguments to be explicitly marked with an operator +like consume or transfer. This is not needed since the APIs in question are +already explicitly marked as being a point of concurrency via `async`, `await`, +or `Task` implying that whether or not an API can result in transferring is +already explicitly marked. The only information that requiring an additional +explicit marker would provide the user is that the programmer can know without +reading the API surface that a transfer will occur here, information that can +also be ascertained by just reading the source. + +## Acknowledgments + +This proposal is based on work from the PLDI 2022 paper [A Flexible Type System for Fearless Concurrency](https://www.cs.cornell.edu/andru/papers/gallifrey-types/). + +Thanks to Doug Gregor, Kavon Farvardin for early assistance to Joshua during his +internship. + +Thanks to Doug Gregor and Holly Borla for our stimulating discussions and to +Holly for her help with editing! + +## Appendix + +### Isolation Region Dataflow + +The dataflow for computing *isolation regions* is defined as follows: + +1. The lattice of the dataflow consists of graphs where each value is a node and + each edge represents a statement that causes two values to be apart of the + same region. We partially order our lattice by stating that given a graph + `g1` and a graph `g2` then `g1 <= g2` only if `g1 U g2 = g1` where `U` is a + graph union operation. + +2. Control flow merges are defined by unions of graphs meaning that if there is + an edge in between two nodes in any predecessor control flow blocks, there is + an edge in the successor control flow block. + +3. We consider the top of the dataflow to be the empty graph consisting of + values that are all in their own independent regions and the bottom of our + dataflow to be a completely connected graph where all values are in the same + region. + +4. Since the dataflow is a forward optimistic dataflow, we initially treat + backedges as propagating the top graph. + +5. We can prove that our dataflow always converges since our transfer function + can be proven as monotonic since given two sets `g1`, `g2` with `g1 <= g2`, + we know that `F(g1) <= F(g2)` since any edges that we remove from `g1` must + also be removed from `g2` and any edges that we add will be added identically + to `g1` and `g2` since `g1` is a subset of `g2`. diff --git a/proposals/0415-function-body-macros.md b/proposals/0415-function-body-macros.md new file mode 100644 index 0000000000..12797b0a2f --- /dev/null +++ b/proposals/0415-function-body-macros.md @@ -0,0 +1,312 @@ +# Function Body Macros + +* Proposal: [SE-0415](0415-function-body-macros.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 6.0)** +* Feature Flag: `BodyMacros` +* Review: [pitch](https://forums.swift.org/t/function-body-macros/66471), [review](https://forums.swift.org/t/se-0415-function-body-macros/68847), [returned for revision](https://forums.swift.org/t/returned-for-revision-se-0415-function-body-macros/69114), [second review](https://forums.swift.org/t/se-0415-second-review-function-body-macros/71644), [acceptance](https://forums.swift.org/t/accepted-se-0415-function-body-macros/72013) + +## Table of contents + +* [Introduction](#introduction) +* [Proposed solution](#proposed-solution) +* [Detailed design](#detailed-design) + * [Declaring function body macros](#declaring-function-body-macros) + * [Implementing function body macros](#implementing-function-body-macros) + * [Composing function body macros](#composing-function-body-macros) + * [Type checking of functions involving function body macros](#type-checking-of-functions-involving-function-body-macros) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) +* [Future directions](#future-directions) + * [Function body macros on closures](#function-body-macros-on-closures) +* [Alternatives considered](#alternatives-considered) + * [Eliminating preamble macros](#eliminating-preamble-macros) + * [Capturing the withSpan pattern in another macro role](#capturing-the-withspan-pattern-in-another-macro-role) + * [Type-checking bodies as they were written](#type-checking-bodies-as-they-were-written) +* [Revision history](#revision-history) + +## Introduction + +Macros augment Swift programs with additional code, which can include new declarations, expressions, and statements. One of the key ways in which one might want to augment code---synthesizing or updating the body of a function---is not currently supported by the macro system. One can create new functions that have their own function bodies, but not provide, augment, or replace function bodies for a function declared by the user. + +This proposal introduces *function body macros*, which do exactly that: allow the wholesale synthesis of function bodies given a declaration, as well as augmenting an existing function body with more functionality. This opens up a number of new use cases for macros, including: + +* Synthesizing function bodies given the function declaration and some metadata, such as automatically synthesizing remote procedure calls that pass along the provided arguments. +* Augmenting function bodies to perform logging/tracing, check preconditions, or establish invariants. +* Replacing function bodies with a new implementation based on the one provided. For example, moving the body into a closure that is executed somewhere else, or treating the body as written as a domain specific language that the macro "lowers" to executable code. + +## Proposed solution + +This proposal introduces *function body macros*, which are [attached macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md) that can augment a function (including initializers, deinitializers, and accessors) with a new body. For example, one could introduce a `Remote` macro that packages up arguments for a remote procedure call: + +```swift + @Remote + func f(a: Int, b: String) async throws -> String +``` + +which could expand the function to provide a body, e.g.: + +```swift +func f(a: Int, b: String) async throws -> String { + return try await remoteCall(function: "f", arguments: ["a": a, "b": b]) +} +``` + +One could also use a macro to introduce logging code on entry and exit to a function, expanding the following + +```swift +@Logged +func g(a: Int, b: Int) -> Int { + return a + b +} +``` + +into + +```swift +func g(a: Int, b: Int) -> Int { + log("Entering g(a: \(a), b: \(b))") + defer { + log("Exiting g") + } + return a + b +} +``` + +Or one could provide a macro that makes it easier to assume that a function that cannot be marked as `@MainActor` using [`assumeIsolated`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md): + +```swift +extension MyView: SomeDelegate { + @AssumeMainActor + nonisolated func onSomethingHappened(event: Event) { + myView.title = newTitle(processing: event) + } +} +``` + +which could expand to: + +```swift +extension MyView: SomeDelegate { + nonisolated func onSomethingHappened(event: Event) { + MainActor.assumeIsolated { + myView.title = newTitle(processing: event) + } + } +} +``` + +Function body macros can be applied to accessors as well, in which case they go on the accessor itself, e.g., + +```swift +var area: Double { + @Logged get { + return length * width + } +} +``` + +When using the shorthand syntax for get-only properties, a function body macro can be applied to the property itself: + +```swift +@Logged var area: Double { + return length * width +} +``` + +## Detailed design + +### Declaring function body macros + +Function body macros are declared with the `body` role, which indicate that they can be attached to any kind of function, and can produce the contents of a function body. For example, here are declarations for the macros used above: + +```swift +@attached(body) macro Remote() = #externalMacro(...) + +@attached(body) macro Logged() = #externalMacro(...) + +@attached(body) macro AssumeMainActor() = #externalMacro(...) +``` + +Like other attached macros, function body macros have no return type. + +### Implementing function body macros + +Body macros are implemented with a type that conforms to the `BodyMacro` protocol: + +```swift +/// Describes a macro that can create the body for a function. +public protocol BodyMacro: AttachedMacro { + /// Expand a macro described by the given custom attribute and + /// attached to the given declaration and evaluated within a + /// particular expansion context. + /// + /// The macro expansion introduces code block items that will become the body for the + /// given function. Any existing body will be implicitly ignored. + static func expansion( + of node: AttributeSyntax, + providingBodyFor declaration: some DeclSyntaxProtocol & WithOptionalCodeBlockSyntax, + in context: some MacroExpansionContext + ) throws -> [CodeBlockItemSyntax] +} +``` + +That function may have a function body, which will be replaced by the code items produced from the macro implementation. + +### Composing function body macros + +At most one `body` macro can be applied to a given function. It receives the function declaration to which it is attached as it was written in the source code and produces a new function body. + +### Type checking of functions involving function body macros + +When a function body macro is applied, the macro-expanded function body will need to be type checked when it is incorporated into the program. However, the function might already have a body that was written by the developer, which can be inspected by the macro implementation. The function body as written must be syntactically well-formed (i.e., it must conform to the [Swift grammar](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/summaryofthegrammar/)) but will *not* be type-checked, so it need not be semantically well-formed. + +This approach follows what other attached macros do: they operate on the syntax of the declaration to which they are attached, and the declaration itself need not have been type-checked before the macro is expanded. However, this approach does lend itself to potential abuse. For example, one could create a `SQL` macro that expects the function body to be a SQL statement, then rewrites that into code that executes the query. For example, the input could be: + +```swift +@SQL +func employees(hiredIn year: Int) -> [String] { + SELECT + name + FROM + employees + WHERE + YEAR(hire_date) = year; +} +``` + +However, this would only work for places where the SQL grammar is a subset of the Swift grammar. Collapsing the same function into two lines would produce an error because it is not syntactically well-formed Swift: + +```swift +@SQL +func employees(hiredIn year: Int) -> [String] { + SELECT name FROM employees // error: consecutive statements on a line must be separated by ';' + WHERE YEAR(hire_date) = year; +} +``` + +The requirement for syntactic wellformedness should help rein in the more outlandish uses of function body macros, as well as making sure that existing tools that operate on source code will continue to work well even in the presence of body macros. + +## Source compatibility + +Function body macros introduce a new macro role into the existing attached macro syntax, and therefore does not have an impact on source compatibility. + +## Effect on ABI stability + +Macros are a source-to-source transformation tool that have no ABI impact. + +## Effect on API resilience + +Macros are a source-to-source transformation tool that have no effect on API resilience. + +## Future directions + +### Function body macros on closures + +Function body macros as presented in this proposal are limited to declared functions, initializers, deinitializers, and accessors. In the future, they could be expanded to apply to closures as well, e.g., + +```swift +@Traced(z) { (x, y) in + x + y +} +``` + +This extension would involve extending the `BodyMacro` protocol with another `expansion` method that accepts closure syntax. The primary challenge with applying function body macros to closures is the interaction with type inference, because closures generally occur within an expression and some of the macro arguments themselves might be part of the expression. In the example above, the `z` value could come from an outer scope and be the subject of type inference: + +```swift +f(0) { z in + @Traced(z) { (x, y) in + x + y + } +} +``` + +Macros are designed to avoid [multiply instantiating the same macro](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion), and have existing limitations in place to prevent the type checker from getting into a position where it is not obvious which macro to expand or the same macro needs to be expanded multiple times. To extend function body macros to closures will require a solution to this type-checking issue, and might be paired with lifting other restrictions on (e.g.) freestanding declaration macros. + +### Preamble macros + +The first reviewed revision of this proposal contained *preamble* macros, which let a macro introduce code at the beginning of a function without changing the rest of the function body. Preamble macros aren't technically necessary, because one could always write a function body macro that injects the preamble code into an existing body. However, preamble macros provide several end-user benefits over function body macros for the cases where they apply: + +* Preamble macros can be composed, whereas function body macros cannot. +* Preamble macros don't change the code as written by the user, so they provide a better user experience (e.g., for diagnostics, code completion, and so on). + +Preamble macros would be expressed as its own attached macro role (`preamble`), implemented with a type that conforms to the `PreambleMacro` protocol. Details are available in [the prior revision](https://github.com/swiftlang/swift-evolution/blob/f1b9da80315578666352a7d6d40a9f6cc936f69a/proposals/0415-function-body-macros.md). + +Preamble macros have been moved out to Future Directions because they represent a possible future, but not an obviously right one: preamble macros might not add sufficient expressivity to cover the cost of the complexity they introduce, and another kind of macro (like the "wrapper" macro below) might provide a more reasonable tradeoff between expressivity and complexity. + +### Wrapper macros + +A number of use cases for body macros involve "wrapping" the existing body in additional logic. For example, consider an alternative formulation of the `Traced` macro (let's call it `@TracedWithSpan`) could make use of the [`withSpan` API](https://swiftpackageindex.com/apple/swift-distributed-tracing/1.0.1/documentation/tracing) such that a function such as: + +```swift +@TracedWithSpan("Doing complicated math") +func h(a: Int, b: Int) -> Int { + return a + b +} +``` + +will expand to: + +```swift +func h(a: Int, b: Int) -> Int { + withSpan("Doing complicated math") { + return a + b + } +} +``` + +This `withSpan` function used here is one instance of a fairly general pattern in Swift, where a function accepts a closure argument and runs it with some extra contextual parameters. As we with the `preamble` macro role mentioned above, we could introduce a special macro role that describes this pattern: the macro would not see the function body that was written by the developer at all, but would instead have a function value representing the body that it could call opaquely. For example, the `TracedWithSpan` example function `h` would expand to: + +```swift +func h(a: Int, b: Int) -> Int { + withSpan("Doing complicated math", body: h-impl) +} +``` + +With this approach, the original function body for `h` would be type-checked prior to macro expansion, and then would be handed off to the macro as an opaque value `h-impl` to be called by `withSpan`. The macro could introduce its own closure wrapping that body as needed, e.g., + +```swift +@TracedWithSpan("Doing complicated math", { span in + span.attributes["operation"] = "addition" +}) +func myMath(a: Int, b: Int) -> Int { + return a + b +} +``` + +could expand to: + +```swift +func myMath(a: Int, b: Int) -> Int { + return withSpan("Doing complicated math") { span in + span.attributes["operation"] = "addition" + return myMath-impl() + } +} +``` + +The advantage of this approach over allowing a `body` macro to replace a body is that we can type-check the function body as it was written, and only need to do so once---then it becomes a value of function type that's passed along to the underlying macro. Also like preamble macros, this approach can compose, because the result of one macro could produce another value of function type that can be passed along to another macro. [Python decorators](https://www.datacamp.com/tutorial/decorators-python) have been successful in that language for customizing the behavior of functions in a similar manner. + +## Alternatives considered + +### Type-checking bodies as they were written + +As noted previously, not checking the body of functions that was written by the user and then replaced by a `body` macro has some down sides. For one, it allows some abuse, where code that wouldn't make sense in Swift is permitted to be written by the user and then significantly altered by the `body` macro. Moreover, wherever the macro is performing some modification that makes ill-formed code into well-formed code (even by something as simple as introducing a `span` variable like `@Traced` does), tools that cannot reason about the macro expansion might be less useful: code completion won't know to provide `span` as a possible completion, nor will it know what type `span` would have. Therefore, the experience of writing code that makes use of `body` macros could be significantly worse than that for normal Swift code. + +On the other hand, type-checking the function bodies before macro expansion has other issues. Type checking is a significant part of compilation time, and having to type-check the body of a function twice---once before macro expansion, once after---could be prohibitively expensive. Type-checking the function body before macro expansion also limits what can be expressed by body macros, including making some use cases (like the `@Traced` macro described earlier) impossible to express without more extensions to the model. + +## Revision history + +* Revision 3: + * Narrowed the focus down to `body` macros. + * Moved preamble macros into Future Directions, added discussion of wrapper macros. +* Revision 2: + * Clarify that preamble macro-introduced local names can shadow names from outer scopes + * Clarify the effect of function body macros on single-expression functions and implicit returns +* Revision 1: + * Allow preamble macros to introduce names. + * Introduce `@AssumeMainActor` example macro for body macros that perform replacement. + * Switch `@Traced` example over to be a preamble macro with push/pop operations, so it can nicely introduce `span`. + * Allow function body macros to be applied to properties that use the shorthand getter syntax. diff --git a/proposals/0416-keypath-function-subtyping.md b/proposals/0416-keypath-function-subtyping.md new file mode 100644 index 0000000000..4816effb16 --- /dev/null +++ b/proposals/0416-keypath-function-subtyping.md @@ -0,0 +1,99 @@ +# Subtyping for keypath literals as functions + +* Proposal: [SE-0416](0416-keypath-function-subtyping.md) +* Authors: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Implementation: [apple/swift#39612](https://github.com/apple/swift/pull/39612) +* Review: ([pitch](https://forums.swift.org/t/pitch-generalize-keypath-to-function-conversions/52681)) ([review](https://forums.swift.org/t/se-0416-subtyping-for-keypath-literals-as-functions/68984)) ([acceptance](https://forums.swift.org/t/accepted-se-0416-subtyping-for-keypath-literals-as-functions/69241)) + +## Introduction + +Today, keypath literals can only be narrowly converted to a function which exactly matches the argument and return type. This proposal allows key path literals to partake in the full generality of the conversions we allow between arbitrary function types, so that the following code compiles without error: + +```swift +let _: (String) -> Int? = \.count +``` + +## Motivation + +[SE-0249](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0249-key-path-literal-function-expressions.md) introduced a conversion between key path literals and function types, which allowed users to write code like the following: + +```swift +let strings = ["Hello", "world", "!"] +let counts = strings.map(\.count) // [5, 5, 1] +``` + +However, SE-0249 does not quite live up to its promise of allowing the equivalent key path construction "wherever it allows (Root) -> Value functions." Function types permit conversions that are covariant in the result type and contravariant in the parameter types, but key path literals require exact type matches. This can lead to some potentially confusing behavior from the compiler: + +```swift +struct S { + var x: Int +} + +// All of the following are okay... +let f1: (S) -> Int = \.x +let f2: (S) -> Int? = f1 +let f3: (S) -> Int? = { $0.x } +let f4: (S) -> Int? = { kp in { root in root[keyPath: kp] } }(\S.x) +let f5: (S) -> Int? = \.x as (S) -> Int + +// But the direct conversion fails! +let f6: (S) -> Int? = \.x // <------------------- Error! +``` + +## Proposed solution + +Allow key path literals to be converted freely in the same manner as functions are converted today. This would allow the definition `f6` above to compile without error, in addition to allowing constructions like: + +```swift +class Base { + var derived: Derived { Derived() } +} +class Derived: Base {} + +let g1: (Derived) -> Base = \Base.derived +``` + +## Detailed design + +Rather than permitting a key path literal with root type `Root` and value type `Value` to only be converted to a function type `(Root) -> Value`, key path literals will be permitted to be converted to any function type which `(Root) -> Value` may be converted to. + +The actual key-path-to-function conversion transformation proceeds exactly as before, generating code with the following semantics (adapting an example from SE-0249): + +```swift +// You write this: +let f: (User) -> String? = \User.email + +// The compiler generates something like this: +let f: (User) -> String? = { kp in { root in root[keyPath: kp] } }(\User.email) +``` + +## Source compatibility + +This proposal allows conversions in some situations that were previously impossible. This can affect source compatibility because overloaded function calls may gain new viable overload candidates. + +In typical scenarios, these new candidates will be strictly worse than previous candidates because the new conversion is strictly less favorable. In situations such as: + +```swift +func evil(_: (T) -> U) { print("generic") } +func evil(_ x: (String) -> Bool?) { print("concrete") } + +evil(\String.isEmpty) +``` + +Swift will (without this proposal) prefer to call the generic function because the conversion necessary for the concrete function is invalid. With this proposal, Swift will still prefer to call the generic function because the concrete function requires an extra conversion (not only does the keypath need to be converted to a function, but the 'natural' type of the keypath function is `(String) -> Bool`, which requires another conversion to get to `(String) -> Bool?`). + +However, this is not always true. A newly-viable overload candidate may be disfavored for the key path conversion but favored for other reasons. This should be uncommon, and so the author expects this proposal will have a very small impact in practice, but this will need to be demonstrated as part of landing the proposal in a Swift release. + +## Effect on ABI stability + +N/A + +## Effect on API resilience + +N/A + +## Acknowledgements + +Thanks to [@ChrisOffner](https://forums.swift.org/u/chrisoffner) for kicking off this discussion on the forums to point out the inconsistency here, and to [@jrose](https://forums.swift.org/u/jrose) for assistance in exploring some strange edge cases in the existing behavior of this feature. diff --git a/proposals/0417-task-executor-preference.md b/proposals/0417-task-executor-preference.md new file mode 100644 index 0000000000..9f0d5848fd --- /dev/null +++ b/proposals/0417-task-executor-preference.md @@ -0,0 +1,798 @@ +# Task Executor Preference + +* Proposal: [SE-0417](0417-task-executor-preference.md) +* Author: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [John McCall](https://github.com/rjmccall), [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-task-executor-preference/68191)), ([review](https://forums.swift.org/t/se-0417-task-executor-preference/68958)), ([acceptance](https://forums.swift.org/t/accepted-se-0417-task-executor-preference/69705)) + +## Introduction + +Swift Concurrency uses tasks and actors to model concurrency and primarily relies on actor isolation to determine where a specific piece of code shall execute. + +The recent introduction of custom actor executors in [SE-0392](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md) allows specifying a `SerialExecutor` implementation code should be running on while isolated to a specific actor. This allows developers to gain some control over exact threading semantics of actors, by e.g. making sure all work made by a specific actor is made on a dedicated queue or thread. + +Today, the same flexibility is not available to tasks in general, and nonisolated asynchronous functions are always executed on the default global concurrent thread pool managed by Swift concurrency. + +## Motivation + +Custom actor executors allow developers to customize where execution of a task “on” an actor must happen (e.g. on a specific queue or thread, represented by a `SerialExecutor`), the same capability is currently missing for code that is not isolated to an actor. + +Notably, since Swift 5.7’s [SE-0338: Clarify the Execution of Non-Actor-Isolated Async Functions](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0338-clarify-execution-non-actor-async.md), functions which are not isolated to an actor will always hop to the default global concurrent executor, which is great for correctness and understanding the code and avoids “hanging onto” actors longer than necessary. This is also a desirable semantic for code running on the `MainActor` calling into `nonisolated` functions, since it allows the main actor to be quickly freed up to proceed with other work, however it has a detrimental effect on applications which want to *avoid* hops in order to maximize request processing throughput. This is especially common with event-loop based systems, such as network servers or other kinds of tight request handling loops. + +As Swift concurrency is getting adopted in a wider variety of performance sensitive codebases, it has become clear that the lack of control over where nonisolated functions execute is a noticeable problem. +At the same time, the defensive "hop-off" semantics introduced by [SE-0338](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0338-clarify-execution-non-actor-async.md) are still valuable, but sometimes too restrictive and some use-cases might even say that the exact opposite behavior might be desirable instead. + +This proposal acknowledges the different needs of various use-cases, and provides a new flexible mechanism for developers to tune their applications and avoid potentially unnecessary context switching when possible. + +## Proposed solution + +We propose to introduce an additional layer of control over where a task can be executed, and have this executor setting be “sticky” to the task. + +**Currently** the decision where an async function or closure is going to execute is binary: + +``` +// `func` execution semantics before this proposal + +[ func / closure ] - /* where should it execute? */ + | + +--------------+ +==========================+ + +- no - | is isolated? | - yes -> | default (actor) executor | + | +--------------+ +==========================+ + | + | +==========================+ + +-------------------------------> | on global conc. executor | + +==========================+ +``` + +This proposal introduces a way to control hopping off to the global concurrent pool for `nonisolated` functions and closures. This is expressed as **task executor preference** and is sticky to the task and entire *structured task hierarchy* created from a task with a specified preference. This changes the current decision diagram to the following: + +``` +// `func` execution semantics with this proposal + +[ func / closure ] - /* where should it execute? */ + | + +--------------+ +===========================+ + +-------- | is isolated? | - yes -> | actor has unownedExecutor | + | +--------------+ +===========================+ + | | | + | yes no + | | | + | v v + | +=======================+ /* task executor preference? */ + | | on specified executor | | | + | +=======================+ yes no + | | | + | | v + | | +==========================+ + | | | default (actor) executor | + | v +==========================+ + v +==============================+ +/* task executor preference? */ ---- yes ----> | on Task's preferred executor | + | +==============================+ + no + | + v + +===============================+ + | on global concurrent executor | + +===============================+ +``` + +In other words, this proposal introduces the ability to specify where code may execute from a Task, and not just by using a custom actor executor, +and even influence the thread use of default actors. + +With this proposal a **`nonisolated` function** will execute, as follows: + +* if task preference **is not** set: + * it is equivalent to current semantics, and will execute on the global concurrent executor, + +* if a task preference **is** set, + * **(new)** nonisolated functions will execute on the selected executor. + + +The preferred executor also may influence where **actor-isolated code** may execute, specifically: + +- if task preference **is** set: + - **(new)** default actors will use the task's preferred executor + - actors with a custom executor execute on that specified executor (i.e. "preference" has no effect), and are not influenced by the task's preference + +The task executor preference can be specified either at task creation time: + +```swift +Task(executorPreference: executor) { + // starts and runs on the 'executor' + await nonisolatedAsyncFunc() +} + +Task.detached(executorPreference: executor) { + // starts and runs on the 'executor' + await nonisolatedAsyncFunc() +} + +await withDiscardingTaskGroup { group in + group.addTask(executorPreference: executor) { + // starts and runs on the 'executor' + await nonisolatedAsyncFunc() + } +} + +func nonisolatedAsyncFunc() async -> Int { + // if the Task has a specific executor preference, + // runs on that 'executor' rather than on the default global concurrent executor + return 42 +} +``` + +or, for a specific scope using the `withTaskExecutorPreference` method. Notably, the task executor preference is in effect for the entire structured task hierarchy while running in a task or scope where a task executor preference is set. For example, the following snippet illustrates child tasks created inside of a `withTaskExecutorPreference`: + +```swift +await withTaskExecutorPreference(executor) { + // if not already running on specified 'executor' + // the withTaskExecutorPreference would hop to it, and run this closure on it. + + // task groups + await withDiscardingTaskGroup { group in + group.addTask { + // starts and runs on the 'executor' + await nonisolatedAsyncFunc() // also runs on 'executor' + } + } + + // async let + async let number = nonisolatedAsyncFunc() // starts and runs on 'executor' + await number +} +``` + +If a task with such executor preference encounters code which is `isolated` to some specific actor, the isolation properties of the actor still are upheld, however, unless that actor has a custom executor configured, the source of the thread actually running the actor's functions will be from the preferred executor: + +```swift +let capy: Capybara = Capybara() +actor Capybara { func eat() {} } + +Task(executorPreference: executor) { + // starts on 'executor' + try await capy.eat() // execution is isolated to the 'capy' actor, however execution happens on the 'executor' TaskExecutor +} +``` + +In a way, one should think of the `SerialExecutor` of the actor and `TaskExecutor` both being tracked and providing different semantics. +The `SerialExecutor` guarantees mutual exclusion, and the `TaskExecutor` provides a source of threads. + +## Detailed design + +### Setting task executor preference + +A new concept of task executor preference is added to Swift Concurrency tasks. This preference is stored in a task and propagated throughout child tasks (such as ones created by TaskGroups and async let). + +The preference can be set using various APIs that will be discussed in detail in their respective sections. The first of those APIs is `withTaskExecutorPreference` which can be called inside an asynchronous context to both ensure we’re executing on the expected executor, as well as set the task executor preference for the duration of the operation closure: + +```swift +await withTaskExecutorPreference(someExecutor) { + // guaranteed to be executing on someExecutor +} +``` + +Once set, the effect of an executor preference is such that a nonisolated func instead of immediately hopping to the global pool, it may hop to the preferred executor, e.g.: + +```swift +nonisolated func doSomething() async { + // ... +} + +let preferredExecutor: SomeConcreteTaskExecutor = ... +Task(executorPreference: preferredExecutor) { + // executes on 'preferredExecutor' + await doSomething() // doSomething body would execute on 'preferredExecutor' +} + + await doSomething() // doSomething body would execute on 'default global concurrent executor' +``` + +### The `TaskExecutor` protocol + +In order to fulfil the requirement that we'd like default actors to run on a task executor, if it was set, we need to introduce a new kind of executor. + +This stems from the fact that `SerialExecutor` and how a default actor effectively acts as an executor "for itself" in Swift. +A default actor (so an actor which does not use a custom executor), has a "default" serial executor that is created by the Swift runtime and uses the actor is the executor's identity. +This means that the runtime executor tracking necessarily needs to track that some code is executing on a specific serial executor in order for things like `assumeIsolated` or the built-in runtime thread-safety checks can utilize them. + +The new protocol mirrors `Executor` and `SerialExecutor` in API, however it provides different semantics, and is tracked using a different mechanism at runtime -- by obtaining it from a task's executor preference record. + +The `TaskExecutor` is defined as: + +```swift +public protocol TaskExecutor: Executor { + func enqueue(_ job: consuming ExecutorJob) + + func asUnownedTaskExecutor() -> UnownedTaskExecutor +} +``` + +As an intuitive way to think about `TaskExecutor` and `SerialExecutor`, one can think of the prior as being a "source of threads" to execute work on, +and the latter being something that "provides serial isolation" and is a crucial part of Swift actors. The two share similarities, however the task executor has a more varied application space. + +### Task executor preference inheritance in Structured Concurrency + +Task executor preference is inherited by child tasks and actors which do not declare an explicit executor (so-called "default actors"), and is *not* inherited by un-structured tasks. + +Specifically: + +* **Do** inherit task executor preference + * TaskGroup’s `addTask()`, unless overridden with explicit parameter + * `async let` + * methods on default actors (actors which do not use a custom executor) +* **Do not** inherit task executor preference + * Unstructured tasks: `Task {}` and `Task.detached {}` + * methods on actors which **do** use a custom executor (including e.g. the `MainActor`) + +This also means that an entire tree can be made to execute their nonisolated work on a specific executor, just by means of setting the preference on the top-level task. + +### Task executor preference and async let + +Since `async let` are the simplest form of structured concurrency, they do not offer much in the way of customization. + +An async let currently always executes on the global concurrent executor, and with the inclusion of this proposal, it does take into account task executor preference. In other words, if an executor preference is set, it will be used by async let to enqueue its underlying task: + +```swift +func test() async -> Int { + return 42 +} + +await withTaskExecutorPreference(someExecutor) { + async let value = test() // async let's "body" and target function execute on 'someExecutor' + // ... + await value +} +``` + +### Task executor preference and TaskGroups + +A `TaskGroup` and its various friends (`ThrowingTaskGroup`, `DiscardingTaskGroup`, ...) are the most powerful, but also most explicit and verbose API for structured concurrency. A group allows creating multiple child tasks using the `addTask` method, and always awaits all child tasks to complete before returning. + +This proposal adds overloads to the `addTask` method, which changes the executor the child tasks will be enqueued on: + +```swift +extension (Discarding)(Throwing)TaskGroup { + mutating func addTask( + on executor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async (throws) -> Void + ) +} +``` + +Which allows users to require child tasks be enqueued and run on specific executors: + +```swift +Task(executorPreference: specialExecutor) { + _ = await withTaskGroup(of: Int.self) { group in + group.addTask { + // using 'specialExecutor' (inherited preference) + return 12 + } + group.addTask(executorPreference: differentExecutor) { + // using 'differentExecutor', overridden preference + return 42 + } + group.addTask(executorPreference: nil) { + // using 'specialExecutor' (inherited preference) + // + // explicitly documents that this task has "no task executor preference". + // this is semantically equivalent to the addTask() call without specifying + // an executor, and therefore since the surrounding scope has a specialExecutor preference, + // that's the executor used. + return 84 + } + group.addTask(executorPreference: globalConcurrentExecutor) { + // using 'globalConcurrentExecutor', overridden preference + // + // using the global concurrent executor -- effectively overriding + // the task executor preference set by the outer scope back to the + // default semantics of child tasks -- to execute on the global concurrent executor. + return 84 + } + return await group.next()! + } +``` + +This gives developers explicit control over where a task group child task shall be executed. Notably, this gives callers of libraries more control over where work should be performed. Do note that code executing on an actor will always hop to that actor; and task executor preference has no impact on code which *requires* to be running in some specific isolation. + +If a library really wants to ensure that hops to the global concurrent executor *are* made by child tasks it may use the newly introduced `globalConcurrentExecutor` global variable. + +### Task executor preference and Unstructured Tasks + +We propose adding new APIs and necessary runtime changes to allow a Task to be enqueued directly on a specific `Executor`, by using a new `Task(executorPreference:)` initializer: + +```swift +extension Task where Failure == Never { + @discardableResult + public init( + executorPreference taskExecutor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Success + ) + + @discardableResult + static func detached( + executorPreference taskExecutor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Success + ) +} + +extension Task where Failure == Error { + @discardableResult + public init( + executorPreference taskExecutor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Success + ) + + @discardableResult + static func detached( + executorPreference taskExecutor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Success + ) +} +``` + +Tasks created this way are **immediately enqueued** on given executor. + +It is possible to pass `nil` to all task executor accepting APIs introduced in this proposal. Passing `nil` to an `executorPreference:` parameter means "no preference", and for structured tasks means to inherit the surrounding context's executor preference; and for unstructured tasks (`Task.init`, `Task.detached`) it serves as a way of documenting no specific executor preference was selected for this task. In both cases, passing `nil` is equivalent to calling the methods which do not accept an executor preference. + +By default, serial executors are not task executors, and therefore cannot be directly used with these APIs. +This is because it would cause confusion in the runtime about having two "mutual exclusion" contexts at the same time, which could result in difficult to understand behaviors. + +It is possible however to write a custom `SerialExecutor` and conform to the `TaskExecutor` protocol at the same time, if indeed one intended to use it for both purposes. +The serial executor conformance can be used for purposes of isolation (including the asserting and "assuming" of isolation), and the task executor conformance allows +using a type to provide a hint where tasks should execute although cannot be used to fulfil isolation requirements. + +#### Task executor preference and default actor isolated methods + +It is also worth explaining the interaction with actors which do not use a custom executor -- which is the majority of actors usually defined in a typical codebase. +Such actors are referred to as "default actors" and are the default way of how actors are declared: + +```swift +actor RunsAnywhere { // a "default" actor == without an executor requirement + func hello() { + return "Hello" + } +} +``` + +Such actor has no requirement as to where it wants to execute. This means that if we were to call the `hello()` isolated +actor method from a task that has defined an executor preference -- the hello() method would still execute on a thread owned by that executor (!), +however isolation is still guaranteed by the actor's semantics: + +```swift +let anywhere = RunsAnywhere() +Task { await anywhere.hello() } // runs on "default executor", using a thread from the global pool + +Task(executorPreference: myExecutor) { + // runs on preferred executor, using a thread owned by that executor + await anywhere.hello() +} +``` + +Methods which assert isolation, such as `Actor/assumeIsolated` and similar still function as expected. + +The task executor can be seen as a "source of threads" for the execution, while the actor's serial executor is used to ensure the serial and isolated execution of the code. + +## Inspecting task executor preference + +It is possible to inspect the current preferred task executor of a task, however because doing so is inherently unsafe -- due to lack of guarantees surrounding the lifetime of an executor referred to using an `UnownedTaskExecutor`, +this operation is only exposed on the `UnsafeCurrentTask`. + +Furthermore, the API purposefully does not expose an `any TaskExecutor` because this would risk incurring atomic ref-counting on an executor object that may have been already deallocated. + +This API is intended only for fine-tuning and checking if we are executing on the "expected" task executor, and therefore the `UnownedTaskExecutor` also implements the `Equatable` protocol, +and implements it using pointer equality. This comparison is not strictly safe, in case if an executor was deallocated, and a new executor was allocated in the same memory location, +however for purposes of executors -- especially long-lived ones, we believe this is not going to prove to be a problem in practical uses of task executors. + +An example use of this API might be something like this: + +``` swift +struct MyEventLoopTaskExecutor: TaskExecutor {} + +func test(expected eventLoop: MyEventLoopTaskExecutor) { + withUnsafeCurrentTask { task in + guard let task else { + fatalError("Missing task?") + } + guard let currentTaskExecutor = task.unownedTaskExecutor else { + fatalError("Expected to have task executor") + } + + precondition(currentTaskExecutor == eventLoop.asUnownedTaskExecutor()) + + // perform action that is required to run on the expected executor + } +} +``` + +This may be useful in synchronous functions; however should be used sparingly, and with caution. +Asynchronous functions, or functions on actors should instead rely on the usual ways to statically ensure to be running on an expected executor: +by providing the right annotations or custom executors to their enclosing actors. + +Instead, functions which have strict execution requirements may be better served as declaring them inside of an actor +that has the required specific executor specified (by using custom actor executors), or by using an asynchronous function +and wrapping the code that is required to run on a specific executor in an `withTaskExecutorPreference(eventLoop) { ... }` block. + +Nevertheless, because we understand there may be situations where synchronous code may want to compare task executors, this capability is exposed for advanced use cases. + +### TaskExecutor ownership + +Task executors, unlike serial executors, are explicitly owned by tasks as long as they are running on the given task executor. + +This is achieved in two ways. The `withTaskExecutorPreference` APIs by their construction as `with...`-style APIs, +naturally retain and keep alive the task executor for as long as the `with... { ... }` body is executing. +This also naturally extends to other structured concurrency constructs like `async let` and task groups, which can +rely on the task executor to remain alive while these constructs are running within such `withTaskExecutorPreference(...) { ... }` closure body. + +Unstructured tasks which are started with a task executor preference (e.g. `Task(executorPreference: someTaskExecutor)`), +take ownership of the executor for as long as the task is running. + +In other words, it is safe to rely on a task, structured or not, to keep alive the task executor it may be running on. +This makes it possible to write code like the following snippet, without having to worry about manually keeping the +executor alive until "all tasks which may be executing on it have finished": + +```swift +func computeThings() async { + let eventLoop: any TaskExecutor = MyCoolEventLoop() + defer { eventLoop.shutdown() } + + let computed = withTaskExecutorPreference(eventLoop) { + async let first = computation(1) + async let second = computation(2) + return await first + second + } + + return computed // event loop will be shutdown and the executor destroyed(!) +} + +func computation(_ int: Int) -> Int { + withUnsafeCurrentTask { task in + let unownedExecutor: UnownedTaskExecutor? = task?.unownedTaskExecutor + let eventLoop: MyCoolEventLoop = EventLoops.find(unownedExecutor) + // we need to start an unstructured task for some reason (try to avoid this if possible) + // and we have located the `MyCoolEventLoop` in our "cache". + // + // Since we have a real MyCoolEventLoop reference, this is safe to forward + // to the unstructured task which will retain it. + Task(executorPreference: eventLoop) { + async let something = ... // inherits the executor preference + } + } +} +``` + +Same as with `SerialExecutor`'s `UnownedSerialExecutor` type, the `UnownedTaskExecutor` does _not_ retain the executor, +so you have to be extra careful when relying on unowned task executor references for any kind of operations. If you +were to write some form of "lookup" function, which takes an unowned executor and returns an `any TaskExecutor`, +please make sure that the returned references are alive (i.e. by keeping them alive in the "cache" using strong references). + +## Combining `SerialExecutor` and `TaskExecutor` + +It is possible to declare a single executor type and have it conform to *both* the `SerialExecutor` (introduced in the custom actor executors proposal), +as well as the `TaskExecutor` (introduce in this proposal). + +If declaring an executor which conforms to both protocols, it truly **must** adhere to the `SerialExecutor` +semantics of not running work concurrently, as it may be used as an *isolation context* by an actor. + +```swift +// naive executor for illustration purposes; we'll assert on the dispatch queue and isolation. +final class NaiveQueueExecutor: TaskExecutor, SerialExecutor { + let queue: DispatchQueue + + init(_ queue: DispatchQueue) { + self.queue = queue + } + + public func enqueue(_ _job: consuming ExecutorJob) { + let job = UnownedJob(_job) + queue.async { + job.runSynchronously( + isolatedOn: self.asUnownedSerialExecutor(), + taskExecutor: self.asUnownedTaskExecutor()) + } + } + + @inlinable + public func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(ordinary: self) + } + + @inlinable + public func asUnownedTaskExecutor() -> UnownedTaskExecutor { + UnownedTaskExecutor(ordinary: self) + } +} +``` + +Since the enqueue method shares the same signature between the two protocols it is possible to just implement it once. +It is of crucial importance to run the job using the new `runSynchronously(isolatedOn:taskExecutor:)` overload +of the `runSynchronously` method. This will set up all the required thread-local state for both isolation assertions +and task-executor preference semantics to be handled properly. + +Given such an executor, we are able to have it both be used by an actor (thanks to being a `SerialExecutor`), and have +any structured tasks or nonisolated async functions execute on it (thanks to it being a `TaskExecutor`): + +```swift +nonisolated func nonisolatedFunc(expectedExecutor: NaiveQueueExecutor) async { + dispatchPrecondition(condition: .onQueue(expectedExecutor.queue)) + expectedExecutor.assertIsolated() +} + +actor Worker { + let executor: NaiveQueueExecutor + + init(on executor: NaiveQueueExecutor) { + self.executor = executor + } + + func test(_ expectedExecutor: NaiveQueueExecutor) async { + // we are isolated to the serial-executor (!) + dispatchPrecondition(condition: .onQueue(expectedExecutor.queue)) + expectedExecutor.preconditionIsolated() + + // the nonisolated async func properly executes on the task-executor + await nonisolatedFunc(expectedExecutor: expectedExecutor) + + /// the task-executor preference is inherited properly: + async let val = { + dispatchPrecondition(condition: .onQueue(expectedExecutor.queue)) + expectedExecutor.preconditionIsolated() + return 12 + }() + _ = await val + + // as expected not-inheriting + _ = await Task.detached { + dispatchPrecondition(condition: .notOnQueue(expectedExecutor.queue)) + }.value + + // we properly came back to the serial executor, just to make sure + dispatchPrecondition(condition: .onQueue(expectedExecutor.queue)) + expectedExecutor.preconditionIsolated() + } +} +``` + +### The `globalConcurrentExecutor` + +This proposal also introduces a way to obtain a reference to the global concurrent executor which is used by default by all tasks and asynchronous functions unless they require some specific executor. + +The implementation of this executor is not exposed as a type, however it is accessible through the `globalConcurrentExecutor` global variable: + +```swift +nonisolated(unsafe) +public var globalConcurrentExecutor: any _TaskExecutor { get } +``` + +Accessing this global computed property is thread-safe and can be done without additional synchronization. + +At present, it is not possible to customize the returned executor from this property, however customizing it is something we are interested in exploring in the future (as well as the main actor's executor). + +This executor does not introduce new functionality to Swift Concurrency per se, as it was always there since the beginning of the concurrency runtime, however it is the first time it is possible to obtain a reference to the global concurrent executor in pure Swift. Generally just creating tasks and calling nonisolated asynchronous functions would automatically enqueue them onto this underlying global thread-pool. + +This proposal introduces the `globalConcurrentExecutor` variable in order to be able to effectively "disable" a task executor preference, because setting a task's executor preference to the default executor is equivalent to the task having the default behavior, as if no executor preference was set. This matters particularly which child tasks, which do want to execute on the default executor under any circumstances: + +```swift +async let noPreference = computation() // child task executes on the global concurrent executor + +await withTaskExecutorPreference(specific) { + async let compute = computation() // child task executes on 'specific' executor + + await withTaskGroup(of: Int.self) { group in + // child task executes on 'specific' executor + group.addTask { computation() } + + // child task executes on global concurrent executor + group.addTask(executorPreference: globalConcurrentExecutor) { + async let compute = computation() // child task executes on the global concurrent executor + + computation() // executed on the global concurrent executor + } + } +} +``` + +## Execution semantics discussion + +### Not a Golden Hammer + +As with many new capabilities in libraries and languages, one may be tempted to use task executors to solve various problems. + +We advise care when doing so with task executors, because while they do minimize the "hopping off" of executors and the associated context switching, +this is also a behavior that may be entirely _undesirable_ in some situations. For example, over-hanging on the MainActor's executor is one of the main reasons +earlier Swift versions moved to make `nonisolated` asynchronous functions always hop off their calling execution context; and this proposal brings back this behavior +for specific executors. + +Applying task executors to solve a performance problem should be done after thoroughly understanding the problem an application is facing, +and only then determining the right "sticky"-ness behavior and specific pieces of code which might benefit from it. + +Examples of good candidates for task executor usage would be systems which utilize some form of "specific thread" to minimize synchronization overhead, +like for example event-loop based systems (often, network applications), or IO systems which willingly perform blocking operations and need to perform them off the global concurrency pool. + +### Analysis of use-cases and the "sticky" preference semantics + +The semantics explained in this proposal may at first seem tricky, however in reality the rule is quite straightforward: + +- when there is a strict requirement for code to run on some specific executor, *it will* (and therefore disregard the "preference"), +- when there is no requirement where asynchronous code should execute, this proposal allows to specify a preference and therefore avoid hopping and context switches, leading to more efficient programs. + +It is worth discussing how user-control is retained with this proposal. Most notably, we believe this proposal follows Swift's core principle of progressive disclosure. + +When developing an application at first one does not have to optimize for fewer context switches, however if as applications grow performance analysis diagnoses context switching being a problem this proposal gives developers the tools to, selectively, in specific parts of a code-base introduce sticky task executor behavior. + +### Separating blocking code off the global shared pools + +This proposal gives control to developers who know that they'd like to isolate their code off from callers. For example, imagine an IO library which wraps blocking IO primitives like read/write system calls. You may not want to perform those on the width-limited default pool of Swift Concurrency, but instead wrap APIs which will be calling such APIs with the executor preference of some "`DedicatedIOExecutor`" (not part of this proposal): + +```swift +// MyCoolIOLibrary.swift + +func blockingRead() -> Bytes { ... } + +public func callRead() async -> Bytes { + await withTaskExecutorPreference(DedicatedIOExecutor.shared) { // sample executor + blockingRead() // OK, we're on our dedicated thread + } +} + +public func callBulk() async -> Bytes { + // The same executor is used for both public functions + await withTaskExecutorPreference(DedicatedIOExecutor.shared) { // sample executor + await callRead() + await callRead() + } +} +``` + +This way we won't be blocking threads inside the shared pool, and are not risking thread starving the entire application. + +We can call `callRead` from inside `callBulk` and avoid unnecessary context switching as the same thread servicing the IO operation may be used for those asynchronous functions -- and no actual context switch may need to be performed when `callBulk` calls into `callRead` either. + +End-users of this library don't need to worry about any of this, but the author of such a library is in full control over where execution will happen -- be it using task executor preference, or custom actor executors. + +This also works the other way around, when a user of a library notices that it is doing blocking work which they would rather separate out onto a different executor. This is true even if the library has declared asynchronous methods but still is taking too long to yield the thread for some reason, causing issues to the shared pool. + +```swift +// SomeLibrary +nonisolated func slowSlow() async { ... } // causes us issues by blocking +``` + + In such situation, we, as users of given library can notice and work around this issue by wrapping it with an executor preference: + +```swift +// our code +func caller() async { + // on shared global pool... + // let's make sure to run slowSlow on a dedicated IO thread: + await withTaskExecutorPreference(DedicatedIOExecutor.shared) { // sample executor + await slowSlow() // will not hop to global pool, but stay on our IOExecutor + } +} +``` + +In other words, task executor preference gives control to developers when and where care needs to be taken. + +The default of hop-avoiding when a preference is set has the benefit of optimizing for less context switching and can lead to better performance. + +It is possible to effectively restore the default behavior as-if no task executor preference was present, by setting the preference to the `globalConcurrentExecutor` which is the executor used by default actors, tasks, and free async functions when no task executor preference is set: + +```swift +func function() async { + // make sure to ignore caller's task executor preference, + // and always use the global concurrent executor. + await withTaskExecutorPreference(globalConcurrentExecutor) { ... } +} +``` + +## Prior-Art + +It is worth comparing with other concurrency runtimes with similar concepts to make sure if there are some common ideas or something different other projects have researched. + +For example, in Kotlin, a `launch` which equivalent to Swift’s creation of a new task, takes a coroutine context which can contain an executor preference. The official documentation showcases the following example: + +```kotlin +launch { // context of the parent, main runBlocking coroutine + println("main runBlocking : I'm working in thread ${Thread.currentThread().name}") +} +launch(Dispatchers.Unconfined) { // not confined -- will work with main thread + println("Unconfined : I'm working in thread ${Thread.currentThread().name}") +} +launch(Dispatchers.Default) { // will get dispatched to DefaultDispatcher + println("Default : I'm working in thread ${Thread.currentThread().name}") +} +launch(newSingleThreadContext("MyOwnThread")) { // will get its own new thread + println("newSingleThreadContext: I'm working in thread ${Thread.currentThread().name}") +} +``` + +Which is similar to the here proposed semantics of passing a specific executor preference. Notably though, because Swift has the concept of actor `isolation` the executor semantics introduced in this proposal are only a preference and will never override the executor requirements of actually strongly `isolated` code. + +Kotlin jobs also inherit the coroutine context from their parent, which is similar to the here proposed executor inheritance works. + +## Future directions + +### Task executor preference and global actors + +Thanks to the improvements to treating @SomeGlobalActor isolation proposed in [SE-NNNN: Improved control over closure actor isolation](https://github.com/swiftlang/swift-evolution/pull/2174) we would be able to that a Task may prefer to run on a specific global actor’s executor, and shall be isolated to that actor. + +Thanks to the equivalence between `SomeGlobalActor.shared` instance and `@SomeGlobalActor` annotation isolations (introduced in the linked proposal), this does not require a new API, but uses the previously described API that accepts an actor as parameter, to which we can pass a global actor’s `shared` instance. + +```swift +@MainActor +var example: Int = 0 + +Task(executorPreference: MainActor.shared) { + example = 12 // not crossing actor-boundary +} +``` + +It is more efficient to write `Task(executorPreference: MainActor.shared) {}` than it is to `Task { @MainActor in }` because the latter will first launch the task on the inferred context (either enclosing actor, or global concurrent executor), and then hop to the main actor. The `executorPreference: MainActor.shared` spelling allows Swift to immediately enqueue on the actor itself. + +### Static closure isolation + +It would be interesting to allow starting a task on a specific actor's executor, and have this infer the specific isolation. + +Today the proposal does not allow using serial executors, which are strictly associated with actors to start a Task "on" such executor. +We could consider adding some form of such ability, and then be able to infer that the closure of a Task is isolated to the actor passed to `Task(executorPreference: some Actor)`. + +The upcoming [SE-NNNN: Improved control over closure actor isolation](https://github.com/swiftlang/swift-evolution/pull/2174) proposal includes a future direction which would allow isolating a closure to a known other value. + +This could be utilized to spell the `Task` initializer like this: + +```swift +extension Task where ... { + init( + executorPreference target: TargetActor, + // ..., + operation: @isolated(target) () async -> () + ) where TargetActor: Actor +} +``` + +This would allow us to cut down on the noise of passing the isolated-on parameter explicitly and avoid a hop to the global executor before the task eventually hops back to the intended actor. +Today, if we were to allow a default actor's executor to be used as `TaskExecutor`, a similar API could be made that would look like this: + +```swift +actor Worker { func work() {} } +let worker: Worker = Worker() + +Task(executorPreference: worker) { worker in // noisy parameter; though required for isolation purposes + worker.work() +} +``` + +However, it would be noisy in the sense of having to repeat the `worker` parameter for purposes of isolation. + + + +## Alternatives considered + +### Do not provide any control over task executors + +We considered if not introducing this feature could be beneficial and forcing developers to always pass explicit `isolated` parameters instead. We worry that this becomes a) very tedious and b) impossibly ties threading semantics with public API and ABI of methods. We are concerned that the lack of executor “preference” which only affects the nonisolated functions in a task hierarchy would cause developers to defensively and proactively create multiple versions of APIs. It would also only allow passing actors as the executors, because isolation is an actor concept, and therefore we’d only be able to isolate using serial executors, while we may want to isolate using general purpose `Executor` types. + + +## Revisions + +- 1.6 + - introduce the global `var defaultConcurrentExecutor: any TaskExecutor` we we can express a task specifically wanting to run on the default global concurrency pool. +- 1.5 + - document that an executor may be both SerialExecutor and TaskExecutor at the same time +- 1.4 + - added `unownedTaskExecutor` to UnsafeCurrentTask +- 1.3 + - introduce TaskExecutor in order to be able to implement actor isolation properly and still use a different thread for running default actors + - wording cleanups + - removal of the `Task(executorPreference: Actor)` APIs; we could perhaps revisit this if we made default actors' executors somehow aware of being a thread source as well etc. +- 1.2 + - preference also has effect on default actors +- 1.1 + - added future direction about simplifying the isolation of closures without explicit parameter passing + - removed ability to observe current executor preference of a task diff --git a/proposals/0418-inferring-sendable-for-methods.md b/proposals/0418-inferring-sendable-for-methods.md new file mode 100644 index 0000000000..9e030a6872 --- /dev/null +++ b/proposals/0418-inferring-sendable-for-methods.md @@ -0,0 +1,421 @@ +# Inferring `Sendable` for methods and key path literals + +* Proposal: [SE-0418](0418-inferring-sendable-for-methods.md) +* Authors: [Angela Laar](https://github.com/angela-laar), [Kavon Farvardin](https://github.com/kavon), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 6.0)** +* Upcoming Feature Flag: `InferSendableFromCaptures` +* Review: ([pitch](https://forums.swift.org/t/pitch-inferring-sendable-for-methods/66565)) ([review](https://forums.swift.org/t/se-0418-inferring-sendable-for-methods-and-key-path-literals/68999)) ([acceptance](https://forums.swift.org/t/accepted-se-0418-inferring-sendable-for-methods-and-key-path-literals/69242)) + +## Introduction + +This proposal is focused on a few corner cases in the language surrounding functions as values and key path literals when using concurrency. We propose Sendability should be inferred for partial and unapplied methods. We also propose to lift a Sendability restriction placed on key path literals in [SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md#key-path-literals) by allowing the developers to control whether key path literal is Sendable or not. The goal is to improve flexibility, simplicity, and ergonomics without significant changes to Swift. + +## Motivation + +The partial application of methods and other first-class uses of functions have a few rough edges when combined with concurrency. + +Let’s look at partial application on its own before we combine it with concurrency. In Swift, you can create a function-value representing a method by writing an expression that only accesses (but does not call) a method using one of its instances. This access is referred to as a "partial application" of a method to one of its (curried) arguments - the object instance. + +```swift +struct S { + func f() { ... } +} + +let partial: (() -> Void) = S().f +``` + + +When referencing a method *without* partially applying it to the object instance, using the expression NominalType.method, we call it "unapplied." + + +```swift +let unapplied: (S) -> (() -> Void) = S.f +``` + + +Suppose we want to create a generic method that expects an unapplied function method conforming to Sendable as a parameter. We can create a protocol `P` that conforms to the `Sendable` protocol and tell our generic function to expect some generic type that conforms to `P`. We can also use the `@Sendable` attribute, introduced for closures and functions in [SE-302](https://github.com/kavon/swift-evolution/blob/sendable-functions/proposals/0302-concurrent-value-and-concurrent-closures.md), to annotate the closure parameter. + + +```swift +protocol P: Sendable { + init() +} + +func g(_ f: @escaping @Sendable (T) -> (() -> Void)) where T: P { + Task { + let instance = T() + f(instance)() + } +} +``` + +Now let’s call our method and pass our struct type `S` . First we should make `S` conform to Sendable, which we can do by making `S` conform to our new Sendable type `P` . + +This should make `S` and its methods Sendable as well. However, when we pass our unapplied function `S.f` to our generic function `g`, we get a warning that `S.f` is not Sendable as `g()` is expecting. + + +```swift +struct S: P { + func f() { ... } +} + +g(S.f) // Converting non-sendable function value to '@Sendable (S) -> (() -> Void)' may introduce data races +``` + + +We can work around this by wrapping our unapplied function in a Sendable closure. + +```swift +// S.f($0) == S.f() +g({ @Sendable in S.f($0) }) +``` + + +However, this is a lot of churn to get the expected behavior. The compiler should preserve `@Sendable` in the type signature instead. + +**Key Paths** + +[SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md#key-path-literals) makes an explicit mention that all key path literals are treated as implicitly `Sendable` which means that they are not allowed to capture any non-`Sendable` values. This behavior is justified when key path values are passed across concurrency domains or otherwise involved in concurrently executed code but is too restrictive for non-concurrency related code. + +```swift +class Info : Hashable { + // some information about the user +} + +public struct Entry {} + +public struct User { + public subscript(info: Info) -> Entry { + // find entry based on the given info + } +} + +let entry: KeyPath = \.[Info()] +``` + +With sendability checking enabled this example is going to produce the following warning: + +``` +warning: cannot form key path that captures non-sendable type 'Info' +let entry: KeyPath = \.[Info()] + ^ +``` + +Use of the key path literal is currently being diagnosed because all key path literals should be Sendable. In actuality, this code is concurrency-safe, there are no data races here because key path doesn’t actually cross any isolation boundary. The compiler should instead verify and diagnose situations when key path is actually passed across an isolation boundary otherwise a warning like that would be confusing for the developers unfamiliar with Swift concurrency, might not always be actionable when type is declared in a different module, and goes against the progressive disclosure principle of the language. + +## Proposed solution + +We propose the compiler should automatically employ `Sendable` on functions and key paths that cannot capture non-Sendable values. This includes partially-applied and unapplied instance methods of `Sendable` types, as well as non-local functions. Additionally, it should be disallowed to utilize `@Sendable` on instance methods of non-`Sendable` types. + +**Functions** + +For a function, the `@Sendable` attribute primarily influences the kinds of values that can be captured by the function. But methods of a nominal type do not capture anything but the object instance itself. Semantically, a method can be thought of as being represented by the following functions: + + +```swift +// Pseudo-code declaration of a Nominal Type: +type NominalType { + func method(ArgType) -> ReturnType { /* body of method */ } +} + +// Can desugar to these two global functions: +func NominalType_method_partiallyAppliedTo(_ obj: NominalType) -> ((ArgType) -> ReturnType) { + let inner = { [obj] (_ arg1: ArgType) -> ReturnType in + return NominalType_method(obj, arg1) + } + return inner +} +// The actual method call +func NominalType_method(_ self: NominalType, _ arg1: ArgType) -> ReturnType { + /* body of method */ +} +``` + +Thus, the only way a partially-applied method can be `@Sendable` is if the `inner` closure were `@Sendable`, which is true if and only if the nominal type conforms to `Sendable`. + + +```swift +type NominalType : Sendable { + func method(ArgType) -> ReturnType { /* body of method */ } +} +``` + +For example, by declaring the following type `Sendable`, the partial and unapplied function values of the type would have implied Sendability and the following code would compile with no errors. + +```swift +struct User : Sendable { + func updatePassword (new: String, old: String) -> Bool { + /* update password*/ + return true + } +} + +let unapplied: @Sendable (User) -> ((String, String) → Bool) = User.updatePassword // no error + +let partial: @Sendable (String, String) -> Bool = User().updatePassword // no error +``` + +**Key paths** + +Key path literals are very similar to functions, their sendability could be influenced by sendability of the values they capture in their arguments and isolation of the referenced properties and subscripts. Instead of requiring key path literals to always be sendable and warning about cases where key path literals capture non-Sendable types, let’s flip that requirement and allow the developers to explicitly state when a key path is required to be Sendable via `& Sendable` type composition and employ type inference to infer sendability in the same fashion as functions when no contextual type is specified. [The key path hierarchy of types is non-Sendable]. + +Let’s extend our original example type `User` with a new property and a subscript to showcase the change in behavior: + +```swift +struct User { + var name: String + + @MainActor var age: Int + + subscript(_ info: Info) -> Entry { ... } +} +``` + +A key path to reference a property `name` does not capture any non-Sendable types which means the type of such key path literal could either be inferred as `WritableKeyPath & Sendable` or stated to have a sendable type via `& Sendable` composition: + +```swift +let name = \User.name // WritableKeyPath **& Sendable** +let name: KeyPath & Sendable = \.name // 🟢 +``` + +It is also allowed to use `@Sendable` function type and `& Sendable` key path interchangeably: + +```swift +let name: @Sendable (User) -> String = \.name 🟢 +``` + +It is important to note that **under the proposed rule all of the declarations that do not explicitly specify a Sendable requirement alongside key path type are treated as non-Sendable** (see Source Compatibility section for further discussion): + +```swift +let name: KeyPath = \.name // 🟢 but key path is **non-Sendable** +``` + +Since Sendable is a marker protocol it should be possible to adjust all declarations where `& Sendable` is desirable without any ABI impact. + +Existing APIs that use key path in their parameter types or default values can add `Sendable` requirement in a non-ABI breaking way by marking existing declarations as @preconcurrency and adding `& Sendable` at appropriate positions: + +```swift +public func getValue(_: KeyPath) { ... } +``` + +becomes + +```swift +@preconcurrency public func getValue(_: KeyPath & Sendable) { ... } +``` + +Explicit sendability annotation does not override sendability checking and it would still be incorrect to state that the key path literal is Sendable when it captures non-Sendable values: + +```swift +let entry: KeyPath & Sendable = \.[Info()] 🔴 Info is a non-Sendable type +``` + +Such `entry` declaration would be diagnosed by the sendability checker: + +```swift +warning: cannot form key path that captures non-sendable type 'Info' +``` + +In the same fashion key path that references `age` (i.e. `\User.age`), which is a global actor isolated property, is non-Sendable. + +## Detailed design + +This proposal includes five changes to `Sendable` behavior. + +The first two are what we just discussed regarding partial and unapplied methods. + +```swift +struct User : Sendable { + var address: String + var password: String + + func changeAddress (new: String, old: String) {/*do work*/ } +} +``` + +1. The inference of `@Sendable` for unapplied references to methods of a Sendable type. + +```swift +let unapplied : @Sendable (User)-> ((String, String) -> Void) = User.changeAddress // no error +``` + +2. The inference of `@Sendable` for partially-applied methods of a Sendable type. + +```swift +let partial : @Sendable (String, String) -> Void = User().changeAddress // no error +``` + + +These two rules include partially applied and unapplied static methods but do not include partially applied or unapplied mutable methods. Unapplied references to mutable methods are not allowed in the language because they can lead to undefined behavior. More details about this can be found in [SE-0042](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0042-flatten-method-types.md). + + +3. A key path literal without non-Sendable type captures and references to actor-isolated properties and/or subscripts is going to be inferred as key path type with a `& Sendable` requirement or a function type with `@Sendable` attribute. + +```swift +extension User { + @MainActor var age: Int { get { 0 } } +} + +let ageKP = \User.age +let infoKP = \User.[Info()] +``` + +The type of `ageKP` is `KeyPath` because `age` is isolated to a global actor. Similarly `infoKP` is a non-Sendable key path because `Info()` argument to a subscript reference has a non-Sendable type. + +Key path types respect all of the existing sub-typing rules related to Sendable protocol which means a key path that is not marked as Sendable cannot be assigned to a value that is Sendable. + +```swift +let name: KeyPath = \.name +let otherName: KeyPath & Sendable = \.name 🔴 +``` + +The conversion between key path and a `@Sendable` function doesn’t actually require the key path itself to be `Sendable` because it’s not captured by the closure but wrapped by it. + +```swift +let name: @Sendable (User) -> String = \.name 🟢 +``` + + The example above is accepted and is transformed by the compiler into: + +```swift +let name: @Sendable (User) -> String = { $0[keyPath: \.name] } +``` + +But any subscript arguments that are non-Sendable would preclude the conversion because they’d be captured by the implicitly synthesized closure which makes the closure non-Sendable: + +```swift +let value: NonSendable = NonSendable() +let _: @Sendable (User) -> String = \.[value] 🔴 +``` + +This is an error because `value` has a non-Sendable type and the compiler synthesized closure that wraps the key path - `{ $0[keyPath: \.[value]] }` is going to be inferred as non-Sendable (because it captures `value`) hence non-convertible to a `@Sendable` function type. + +Similarly if the conversion captures a key path that has a reference to an isolated property or subscript the implicitly generated closure is not inferred to be non-Sendable. + +Key path literals are allowed to infer Sendability requirements from the context i.e. when a key path literal is passed as an argument to a parameter that requires a Sendable type: + +```swift +func getValue(_: KeyPath & Sendable) -> T {} + +getValue(name) // 🟢 both parameter & argument match on sendability requirement +getValue(\.name) // 🟢 use of '& Sendable' by the parameter transfers to the key path literal +getValue(\.[NonSendable()]) // 🔴 This is invalid because key path captures a non-Sendable type + +func filter(_: @Sendable (User) -> T) {} +filter(name) // 🟢 use of @Sendable applies a sendable key path +``` + +Next is: + +4. The inference of `@Sendable` when referencing non-local functions. + +Unlike closures, which retain the captured value, global functions can't capture any variables - because global variables are just referenced by the function without any ownership. With this in mind there is no reason not to make these `Sendable` by default. This change will also include static global functions. + +```swift +func doWork() -> Int { + Int.random(in: 1..<42) +} + +Task.detached(priority: nil, operation: doWork) // Converting non-sendable function value to '@Sendable () async -> Void' may introduce data races +``` + +Currently, trying to start a `Task` with the global function `doWork` will cause an error complaining that the function is not `Sendable`. This should compile with no issue. + +5. Prohibition of marking methods `@Sendable` when the type they belong to is not `@Sendable`. + +```swift +class C { + var random: Int = 0 // random is mutable so `C` can't be checked sendable + + @Sendable func generateN() async -> Int { //error: adding @Sendable to function of non-Senable type prohibited + random = Int.random(in: 1..<100) + return random + } +} + +func test(x: C) { x.generateN() } + +let num = C() +Task.detached { + test(num) +} +test(num) // data-race +``` + +If we move the previous work we wanted to do into a class that stores the random number we generate as a mutable value, we could be introducing a data race by marking the function responsible for this work `@Sendable` . Doing this should be prohibited by the compiler. + +Since `@Sendable` attribute will be automatically determined with this proposal, you will no longer have to explicitly write it on function and method declarations. + +### Extending key path merging functionality to preserve sendability + +Existing Key path API provides a way to join two key paths together via using instance method `appending(...)` . Overloads of this method take key path types of varying mutability as their parameters and produce a new “joined” key path of a desired mutability (read-only, writable, or reference writable). + +Under the proposed semantics all overloads of this method become non-Sendable but it is possible and desirable to alleviate that and support/propagate sendability if both “base” and “appended” key paths are `Sendable`. + +Such could be archived by introducing new overloads to `func appending(...)` that utilize `& Sendable` for their parameter and result in an extension of `Sendable` protocol. For example: + +```swift +extension Sendable where Self: AnyKeyPath { + @inlinable + public func appending( + path: KeyPath & Sendable + ) -> KeyPath & Sendable where Self : KeyPath { + ... + } +} +``` + +This overload would be selected if both “base” key path and the argument are `Sendable` and would produce a new `Sendable` key path: + +```swift +func makeUTF8CountKeyPath(from base: KeyPath & Sendable) -> KeyPath & Sendable { + // Both `base` and `\String.utf8.count` are Sendable key paths, + // so `appending(path:)` returns a Sendable key path too. + return base.appending(path: \.utf8.count) 🟢 +} +``` + +Standard library would have to introduce a variety of new overloads to keep `Sendable` capable `appending(...)` on par with existing non-Sendable functionality. + +## Source compatibility + +As described in the Proposed Solution section, some of the existing property and variable declarations **without explicit types** could change their type but the impact of the inference change should be very limited. For example, it would only be possible to observe it when a function or key path value which is inferred as Sendable is passed to an API which is overloaded on Sendable capability: + +```swift +func callback(_: @Sendable () -> Void) {} +func callback(_: () -> Void) {} + +callback(MyType.f) // if `f` is inferred as @Sendable first `callback` is preferred + +func getValue(_: KeyPath & Sendable) {} +func getValue(_: KeyPath) {} + +getValue(\.utf8.count) // prefers first overload of `getValue` if key path is `& Sendable` +``` + +Such calls to `callback` and `getValue` are currently ambiguous but under the proposed rules the type-checker would pick the first overload of `callback` and `getValue` as a solution if `f` is inferred as `@Sendable` and `\String.utf8.count` would be inferred as having a type of `KeyPath & Sendable` instead of just `KeyPath`. + +## Effect on ABI stability + +When you remove an explicit `@Sendable` from a method, the mangling of that method will change. Since `@Sendable` will now be inferred, if you choose to remove the explicit annotation to "adopt" the inference, you may need to consider the mangling change. + +Adding or removing `& Sendable` from type doesn’t have any ABI impact because `Sendable` is a marker protocol that can be added transparently. + +## Effect on API resilience + +N/A + +## Future Directions + +Accessors are not currently allowed to participate with the `@Sendable` system in this proposal. It would be straightforward to allow getters to do so in a future proposal if there was demand for this. + +## Alternatives Considered + +Swift could forbid explicitly marking function declarations with the `@Sendable` attribute, since under this proposal there’s no longer any reason to do this. + +```swift +/*@Sendable*/ func alwaysSendable() {} +``` + +However, since these attributes are allowed today, this would be a source breaking change. Swift 6 could potentially include fix-its to remove `@Sendable` attributes to ease migration, but it’d still be disruptive. The attributes are harmless under this proposal, and they’re still sometimes useful for code that needs to compile with older tools, so we have chosen not to make this change in this proposal. We can consider deprecation at a later time if we find a good reason to do so. diff --git a/proposals/0419-backtrace-api.md b/proposals/0419-backtrace-api.md new file mode 100644 index 0000000000..d185267a86 --- /dev/null +++ b/proposals/0419-backtrace-api.md @@ -0,0 +1,472 @@ +# Swift Backtrace API + +* Proposal: [SE-0419](0419-backtrace-api.md) +* Authors: [Alastair Houghton](https://github.com/al45tair) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Accepted** +* Implementation: Implemented on main, requires explicit `_Backtracing` import. +* Review: ([pitch](https://forums.swift.org/t/pitch-swift-backtracing-api/62741)) ([review](https://forums.swift.org/t/se-0419-swift-backtracing-api/69595)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0419-swift-backtracing-api/70318)) + +## Introduction + +This year we are improving the usability of Swift for command line and +server-side development by adding first-class support for backtraces +to Swift. + +The backtrace support consists of two parts; the first is the actual +backtracing implementation, and the second is the new API surface in +the Swift standard library. This proposal concerns the latter. + +## Motivation + +In addition to the runtime providing backtraces when programs crash or +terminate abnormally, it is often useful for testing frameworks and +sometimes even library or application code to capture details of the +call stack at a point in time. + +This functionality is somewhat tricky to implement correctly and any +implementation tends, of necessity, to be non-portable. Existing +third-party packages that provide backtrace support have various +downsides, including lack of support for tracing through async frames, +and add additional dependencies to client packages and applications. + +## Proposed solution + +We will add a `Backtrace` struct to the standard library, with methods +to capture a backtrace from the current location, and support for +symbolication and symbol demangling. All of the backtracing types will +exist in a new `Runtime` module. + +Note, importantly, that **the API presented here is not async-signal-safe**, +and **it is not an appropriate tool with which to build a general purpose +crash reporter**. The intended use case for this functionality is the +programmatic capture of backtraces during normal execution. + +## Detailed design + +The `Backtrace` struct will capture an `Array` of `Frame` objects, +each of which will represent a stack frame or a `Task` activation +context. + +```swift +/// Holds a backtrace. +public struct Backtrace: CustomStringConvertible, Codable, Sendable { + /// The type of an address. + /// + /// This is used as an opaque type; if you have some Address, you + /// can ask if it's NULL, and you can attempt to convert it to a + /// FixedWidthInteger. + /// + /// This is intentionally _not_ a pointer, because you shouldn't be + /// dereferencing them; they may refer to some other process, for + /// example. + public struct Address: Comparable, Hashable, Codable, Sendable, + LosslessStringConvertible, + ExpressibleByIntegerLiteral { + var bitWidth: Int { get } + var isNull: Bool { get } + } + + /// The unwind algorithm to use. + public enum UnwindAlgorithm { + /// Choose the most appropriate for the platform. + case auto + + /// Use the fastest viable method. + /// + /// Typically this means walking the frame pointers. + case fast + + /// Use the most precise available method. + /// + /// On Darwin and on ELF platforms, this will use EH unwind + /// information. On Windows, it will use Win32 API functions. + case precise + } + + /// Represents an individual frame in a backtrace. + public enum Frame: CustomStringConvertible, Codable, Sendable { + /// An accurate program counter. + /// + /// This might come from a signal handler, or an exception or some + /// other situation in which we have captured the actual program counter. + case programCounter(Address) + + /// A return address. + /// + /// Corresponds to a call from a normal function. + case returnAddress(Address) + + /// An async resume point. + /// + /// Corresponds to an `await` in an async task. + case asyncResumePoint(Address) + + /// Indicates a discontinuity in the backtrace. + /// + /// This occurs when you set a limit and a minimum number of frames at + /// the top. For example, if you set a limit of 10 frames and a minimum + /// of 4 top frames, but the backtrace generated 100 frames, you will see + /// + /// 0: frame 100 <----- bottom of call stack + /// 1: frame 99 + /// 2: frame 98 + /// 3: frame 97 + /// 4: frame 96 + /// 5: ... <----- omittedFrames(92) + /// 6: frame 3 + /// 7: frame 2 + /// 8: frame 1 + /// 9: frame 0 <----- top of call stack + /// + /// Note that the limit *includes* the discontinuity. + /// + /// This is good for handling cases involving deep recursion. + case omittedFrames(Int) + + /// Indicates a discontinuity of unknown length. + /// + /// This can only be present at the end of a backtrace; in other cases + /// we will know how many frames we have omitted. For instance, + /// + /// 0: frame 100 <----- bottom of call stack + /// 1: frame 99 + /// 2: frame 98 + /// 3: frame 97 + /// 4: frame 96 + /// 5: ... <----- truncated + case truncated + + /// The original program counter, with no adjustment. + /// + /// The value returned from this property is undefined if the frame + /// is a discontinuity. + public var originalProgramCounter: Address { get } + + /// The adjusted program counter to use for symbolication. + /// + /// The value returned from this property is undefined if the frame + /// is a discontinuity. + public var adjustedProgramCounter: Address { get } + + /// A textual description of this frame. + public var description: String { get } + } + + /// Represents an image loaded in the process's address space + public struct Image: CustomStringConvertible, Codable, Identifiable, Sendable { + /// The name of the image (e.g. libswiftCore.dylib). + public var name: String? { get } + + /// The full path to the image (e.g. /usr/lib/swift/libswiftCore.dylib). + public var path: String? { get } + + /// The unique ID of the image, as a byte array (note that the exact number + /// of bytes may vary, and that some images may not have a unique ID). + /// + /// On Darwin systems, this is the LC_UUID value; on Linux this is the + /// build ID, which may take one of a number of forms or may not even + /// be present. + public var uniqueID: [UInt8]? { get } + + /// The base address of the image. + public var baseAddress: Address { get } + + /// The end of the text segment in this image. + public var endOfText: Address { get } + + /// Provide a textual description of an Image. + public var description: String { get } + } + + /// The architecture of the process to which this backtrace refers. + public var architecture: String + + /// A `Sequence` of captured frame information. + /// + /// The underlying storage is intentionally not exposed, because there may + /// be cases where it's desirable to use a more compact form (for instance + /// delta compression). + public var frames: some Sequence { get } + + /// A list of captured images. + /// + /// Some backtracing algorithms may require this information, in which case + /// it will be filled in by the `capture()` method. Other algorithms may + /// not, in which case it will be `nil` and you can capture an image list + /// separately yourself using `captureImages()`. + public var images: [Image]? + + /// Capture a backtrace from the current program location. + /// + /// The `capture()` method itself will not be included in the backtrace; + /// i.e. the first frame will be the one in which `capture()` was called, + /// and its programCounter value will be the return address for the + /// `capture()` method call. + /// + /// @param algorithm Specifies which unwind mechanism to use. If this + /// is set to `.auto`, we will use the platform default. + /// @param limit The backtrace will include at most this number of + /// frames; you can set this to `nil` to remove the + /// limit completely if required. + /// @param offset Says how many frames to skip; this makes it easy to + /// wrap this API without having to inline things and + /// without including unnecessary frames in the backtrace. + /// @param top Sets the minimum number of frames to capture at the + /// top of the stack. + /// + /// @returns A new `Backtrace` struct. + @inline(never) + public static func capture(algorithm: UnwindAlgorithm = .auto, + limit: Int? = 64, + offset: Int = 0, + top: Int = 16) throws -> Backtrace + + /// Capture a list of the images currently mapped into the calling + /// process. + /// + /// @returns A list of `Image`s. + public static func captureImages() -> [Image] + + /// Specifies options for the `symbolicated` method. + public struct SymbolicationOptions: OptionSet { + public let rawValue: Int + + /// Add virtual frames to show inline function calls. + public static let showInlineFrames: SymbolicationOptions + + /// Look up source locations. + /// + /// This may be expensive in some cases; it may be desirable to turn + /// this off e.g. in Kubernetes so that pods restart promptly on crash. + public static let showSourceLocations: SymbolicationOptions + + /// Use a symbol cache, if one is available. + public static let useSymbolCache: SymbolicationOptions + + public static let default: SymbolicationOptions = [.showInlineFrames, + .showSourceLocations, + .useSymbolCache] + } + + /// Return a symbolicated version of the backtrace. + /// + /// @param images Specifies the set of images to use for symbolication. + /// If `nil`, the function will look to see if the `Backtrace` + /// has already captured images. If it has, those will be + /// used; otherwise we will capture images at this point. + /// + /// @param options Symbolication options; see `SymbolicationOptions`. + /// + /// @returns A new `SymbolicatedBacktrace`. + public func symbolicated(with images: [Image]? = nil, + options: SymbolicationOptions = .default) + -> SymbolicatedBacktrace? + + /// Provide a textual version of the backtrace. + public var description: String { get } +} +``` + +We allow `Address` to be converted to a `FixedWidthInteger` by means of an +extension on `FixedWidthInteger`: + +```swift +extension FixedWidthInteger { + /// Convert from a Backtrace.Address. + /// + /// This initializer will return nil if the address width is larger than the + /// type you are attempting to convert into. + /// + /// @param address The `Address` to convert. + init?(_ address: Backtrace.Address) +} +``` + +_Symbolication_, by which we mean the process of looking up the symbols +associated with addresses in a backtrace, is in general an expensive +process, and for efficiency reasons is normally performed for a backtrace +as a whole, rather than for individual frames. It therefore makes sense +to provide a separate `SymbolicatedBacktrace` type and to provide a +method on a `Backtrace` +to symbolicate. + +```swift +/// A symbolicated backtrace +public struct SymbolicatedBacktrace: CustomStringConvertible, Codable, Sendable { + /// The `Backtrace` from which this was constructed + public var backtrace: Backtrace + + /// Represents a location in source code. + /// + /// The information in this structure comes from compiler-generated + /// debug information and may not correspond to the current state of + /// the filesystem --- it might even hold a path that only works + /// from an entirely different machine. + public struct SourceLocation: CustomStringConvertible, Codable, Sendable { + /// The path of the source file. + var path: String { get } + + /// The line number. + var line: Int { get } + + /// The column number. + var column: Int { get } + + /// Provide a textual description. + public var description: String { get } + } + + /// Represents an individual frame in the backtrace. + public struct Frame: CustomStringConvertible, Codable, Sendable { + /// The captured frame from the `Backtrace`. + public var captured: Backtrace.Frame { get } + + /// The result of doing a symbol lookup for this frame. + public var symbolInfo: SymbolInfo? { get } + + /// If `true`, then this frame was inlined. + public var isInline: Bool { get } + + /// `true` if this frame represents a Swift runtime failure. + public var isSwiftRuntimeFailure: Bool { get } + + /// `true` if this frame represents a Swift thunk function. + public var isSwiftThunk: Bool { get } + + /// `true` if this frame is a system frame. + public var isSystem: Bool { get } + + /// A textual description of this frame. + public var description: String { get } + } + + /// Represents a symbol we've located + public struct SymbolInfo: CustomStringConvertible, Codable, Sendable { + /// The image in which the symbol for this address is located. + public var image: Backtrace.Image { get } + + /// The raw symbol name, before demangling. + public var rawName: String { get } + + /// The demangled symbol name. + public var name: String { get } + + /// The offset from the symbol. + public var offset: Int { get } + + /// The source location, if available. + public var sourceLocation: SourceLocation? { get } + + /// True if this symbol represents a Swift runtime failure. + /// + /// These are things that are trapped by Swift itself at runtime, for + /// example divide by zero or arithmetic overflow. + public var isSwiftRuntimeFailure: Bool { get } + + /// True if this symbol is a Swift thunk function. + public var isSwiftThunk: Bool { get } + + /// True if this symbol represents a system function. + /// + /// System frames are generally things that people not involved in + /// compiler or runtime development would not be interested in, for + /// instance runtime initialisation routines that happen before + /// the Swift program is started, or runtime support code for the + /// Swift Concurrency system. + public var isSystem: Bool { get } + + /// Construct a new Symbol. + public init(image: Backtrace.Image, rawName: String, offset: Int, + sourceLocation: SourceLocation?) + + /// A textual description of this symbol. + public var description: String { get } + } + + /// A list of captured frame information. + public var frames: some Sequence { get } + + /// A list of images found in the process. + public var images: [Backtrace.Image] + + /// True if this backtrace is a Swift runtime failure. + public var isSwiftRuntimeFailure: Bool { get } + + /// Provide a textual version of the backtrace. + public var description: String { get } +} +``` + +Example usage: + +```swift +import Runtime + +var backtrace = Backtrace.capture() + +print(backtrace) + +var symbolicated = backtrace.symbolicated() + +print(symbolicated) +``` + +## Source compatibility + +This proposal is entirely additive. There are no source compatibility +concerns. + +## Effect on ABI stability + +The addition of this API will not be ABI-breaking, although as with any +new additions to the standard library it will constrain future versions +of Swift to some extent. + +## Effect on API resilience + +Once added, some changes to this API will be ABI and source-breaking +changes. Changes to the new structs/classes will be restricted as +described in the [library evolution +document](https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst) +in the Swift repository. + +## Alternatives considered + +This could have been addressed by creating a separate Swift package, +or by updating the existing [swift-server/swift-backtrace +package](https://github.com/swift-server/swift-backtrace). + +The latter focuses explicitly on Linux and Windows, and has +significant limitations, in addition to which we would like for this +functionality to be built in to Swift---just as it is built into +competing languages. This is why we felt it should be built into the +Swift runtime itself. + +The `Address` type could have been a fixed width integer, but that +loses some flexibility, both in terms of backtrace storage, and in our +ability to cope with backtraces from a platform other than the host. +It could also have been a protocol, but that then necessitates the use +of existentials; or it could have been a generic parameter, but doing +that makes it difficult to cope with a backtrace unless you already +know what kind of addresses it contains at compile time. + +The `frames` member variables could have been arrays, but implementing +them instead as a sequence means that we have the flexibility to use +a different backing store where doing so makes sense. An example where +we might want that is where we're capturing very large numbers of +backtraces, in which case doing some kind of delta compression on the +frame addresses might enable us to save significant amounts of memory. + +Some desirable features are intentionally left out of this proposal; +the intent is that while some of these may even be implemented, they +will remain SPI and may be promoted to API at a later date. Examples +include the ability to construct a `Backtrace` from an array of +addresses that have been gathered through some other mechanism; +provision for advanced formatting of backtraces; and features to +allow backtraces to be captured from another thread or process. + +## Acknowledgments + +Thanks to Jonathan Grynspan and Mike Ash for their helpful comments +on this proposal. diff --git a/proposals/0420-inheritance-of-actor-isolation.md b/proposals/0420-inheritance-of-actor-isolation.md new file mode 100644 index 0000000000..14f83d2603 --- /dev/null +++ b/proposals/0420-inheritance-of-actor-isolation.md @@ -0,0 +1,580 @@ +# Inheritance of actor isolation + +* Proposal: [SE-0420](0420-inheritance-of-actor-isolation.md) +* Authors: [John McCall](https://github.com/rjmccall), [Holly Borla](https://github.com/hborla), [Doug Gregor](https://github.com/douggregor) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-inheriting-the-callers-actor-isolation/68391)) ([review](https://forums.swift.org/t/se-0420-inheritance-of-actor-isolation/69638)) ([acceptance](https://forums.swift.org/t/accepted-se-0420-inheritance-of-actor-isolation/69913)) + +[SE-0302]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md +[SE-0304-propagation]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md#actor-context-propagation +[SE-0306]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md +[SE-0313]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0313-actor-isolation-control.md +[SE-0316]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0316-global-actors.md +[SE-0336]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0336-distributed-actor-isolation.md +[SE-0338]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0338-clarify-execution-non-actor-async.md +[SE-0392]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md + +## Introduction + +Under Swift's [actors design][SE-0306], every function in Swift has +an actor isolation: it is either isolated to some specific actor or +non-isolated. It is sometimes useful to be able to give a function +the same actor isolation as its caller, either to give it access to +actor-isolated data or just to avoid unnecessary suspensions. This +proposal allows `async` functions to opt in to this behavior. + +## Motivation + +The actor isolation of a function controls whether and how the +function can access actor-isolated data. An isolated function +can synchronously access the isolated storage of its actor, such +as the isolated properties of an [`actor` declaration][SE-0306] +or a global variable annotated with a [global actor attribute][SE-0316]. +When called from another function with the same isolation, it can +also pass and return non-[`Sendable`][SE-0302] values that are +isolated to the actor. A non-isolated function cannot do these +things, so making sure that functions share the same formal actor +isolation is sometimes important in order to safely express certain +patterns. + +Actor isolation also affects how the function is executed. Calls +and returns between functions with different actor isolations +may require the task to be suspended and then enqueued on a +different executor.[^1] Even when this is not required, there is +typically some overhead associated with the switch. Programmers +trying to optimize `async` code often find that avoiding these +overheads is important. Avoiding extra suspensions from +actor-isolated code can also be semantically important because +code from other tasks can interleave on the actor during suspensions, +potentially changing the values stored in isolated storage; +this is guaranteed not to happen at the moments of call and return +between functions with the same isolation. + +[^1]: This always happens when one of the functions is isolated +to an actor with a [custom actor executor][SE-0392], such as the +main actor (which uses a custom executor to ensure that execution +always happens on the main thread). For other actors, it typically +only happens when the actor is contended. + +Non-isolated synchronous functions dynamically inherit the isolation +of their caller. For example, an `actor` method can call a non-isolated +synchronous function, and the function will behave dynamically as if it +is isolated to the actor. While the function cannot directly access +actor-isolated storage --- it would need to be statically isolated to +the actor to do that --- it can be passed and return non-`Sendable` +values that are isolated to the actor. Among other things, this means +that you can call a function like `map` on an actor-isolated `Array` +of non-`Sendable` values; you can even pass it an actor-isolated +function, and everything will run synchronously and without suspension. + +However, there is currently no way to get this same effect from an +asynchronous function. [SE-0338][] clarified that non-isolated +`async` functions do not inherit isolation in this same way; instead, +they reset isolation.[^2] This means that these functions cannot +get passed and return non-`Sendable` data when called from an isolated +context, which can be a serious expressivity restriction, especially +for higher-order functions. It may also cause unwanted suspensions. + +[^2]: Prior to SE-0338, non-isolated asynchronous functions still +didn't properly inherit their caller's isolation: they just didn't +actively switch away. As a result, they ran with whatever isolation +they happened to the called or resumed with. That is not good enough +to allow them to safely be passed actor-isolated data or to make +strong guarantees of a lack of suspensions. This is now an ABI +constraint: even if we wanted to change the language to make these +functions inherit their caller's isolation by default, they aren't +passed that information reliably and have no way to implement those +semantics. + +For example, consider the following code that calls `next` on an instance +of `AsyncStream.Iterator` from the `@MainActor`: + +```swift +@MainActor func iterate(over stream: AsyncStream) async { + var iterator = stream.makeAsyncIterator() + while let element = await iterator.next() { + // do something with 'element' + } +} +``` + +The above code produces a warning: + +``` +warning: passing argument of non-sendable type 'inout AsyncStream.Iterator' outside of main actor-isolated context may introduce data races + while let element = await iterator.next() { + ^ +``` + +This happens because `AsyncIteratorProtocol.next()` is a non-isolated +asynchronous function, and most concrete `AsyncIteratorProtocol` types +including `AsyncStream.Iterator` are not `Sendable`. If `next()` is called +from another non-isolated asynchronous function, everything's okay: +it can be passed an arbitrary function and work with arbitrary types. +But if it's called from an *isolated* asynchronous function, Swift will +treat the call as crossing an isolation barrier and enforce three restrictions: + +- First, the result of the call must be `Sendable`. This restriction prevents + `next()` from being used from an actor to produce non-`Sendable` element + values. + +- Second, the `self` argument to the call (the iterator) must + be `Sendable`. This restriction prevents `next()` from being + used from an actor for concrete async iterator types that are not + `Sendable`. + +- Finally, any other arguments to the function must be `Sendable`. + This particular example doesn't have other function arguments, but + this restriction prevents non-isolated `async` functions from using + any other data that's isolated to the actor in the general case. + +In summary, these restrictions unnecessarily limit the capability of +the API when used from an isolated context. Furthermore, even +if the API is usable (e.g. because all the types involved are +`Sendable`), it may be unexpectedly inefficient if, say, the +element-producing closure is actor isolated, because `next()` will +hop to the generic executor only to immediately hop to the isolation +domain of the closure. + +This proposal addresses this problem by giving programmers better +tools for formally inheriting isolation from their caller, allowing +non-`Sendable` data to be safely passed back and forth and +avoiding unnecessary suspensions. + +## Proposed solution + +This proposal makes two changes to the language: + +- First, [SE-0313][]'s `isolated` parameters can now have optional + type. This is required in order for them to express that the + function should be dynamically non-isolated. + +- Second, default argument expressions can now have the special form + `#isolation`, which will be filled in with the actor isolation of + the caller. If the default argument is for an `isolated` parameter, + this allows isolation to be implicitly passed down. + +## Detailed design + +### Design approach + +The basic design approach of this proposal is to first enable +polymorphism over actor isolation, so that a function can declare +itself to have an arbitrary dynamic isolation, then add features +to allow that to be implicitly propagated in calls to the function. +The isolation logic can then recognize calls that propagate the +caller's isolation in sufficiently obvious ways and know that the +callee will share the current context's isolation. + +A function can be non-isolated, isolated to a specific actor instance, +or isolated to a global actor type. Dynamically, however, global actor +isolation is really just isolation to the `shared` instance of the +global actor, so a function's isolation can actually be dynamically +represented as just an optional actor reference, with `nil` +representing non-isolation. + +Since isolation is unavoidably value-dependent (an actor method is +isolated to a *specific* actor reference, not just any actor of that +type), polymorphism over it can't be expressed with just generics. +The natural next choice is to just use a parameter of polymorphic +type, such as `(any Actor)?`. This matches [SE-0313][]'s `isolated` +parameter feature, except that `isolated` parameters are currently +required to be non-optional actor types: either a concrete `actor` +type or a protocol type which implies `Actor`. Generalizing this +is straightforward and gives us the ability to make functions +explicitly polymorphic over an arbitrary isolation. + +Allowing arbitrary isolation to implicitly propagate from caller to +callee is a little trickier. If isolation is specified as a parameter, +then the caller must implicitly provide an argument to it; the most +obvious way to do that is to create a new special form for default +arguments, like `#line`, which expands to an expression that +evaluates to the isolation of the caller. + +### Generalized `isolated` parameters + +The type of an `isolated` parameter must be an *isolation type*. +Currently, the only kind of isolation is a possibly-optional actor type, +which is to say, either `T` or `Optional`, where `T` either conforms +to `Actor` or is a protocol type that implies `Actor`. + +If a function's `isolated` parameter has an optional actor type, then +the dynamic isolation of the function depends on whether the argument +value is `nil`. If it is `nil`, then the function behaves dynamically +as it were non-isolated; for example, if the function is `async`, it +resets isolation on entry under [SE-0338][] just as a non-isolated +function would. Otherwise, the function behaves dynamically as it +were isolated to the unwrapped actor reference. + +According to [SE-0304][SE-0304-propagation], closures passed directly +to the `Task` initializer (i.e. `Task { /*here*/ }`) inherit the +statically-specified isolation of the current context if: + +- the current context is non-isolated, +- the current context is isolated to a global actor, or +- the current context has an `isolated` parameter (including the + implicit `self` of an actor method) and that parameter is strongly + captured by the closure. + +The third clause is modified by this proposal to say that isolation +is also inherited if a non-optional binding of an isolated parameter +is captured by the closure. A non-optional binding of an isolated +parameter is defined in the +[generalized isolation checking](#generalized-isolation-checking) section. + +### Isolated distributed actors + +There is currently no type or protocol that enables abstracting over both +actor and distributed actor isolation using isolated parameters. The +[Distributed actor isolation][SE-0336] proposal introduced the +`DistributedActor` protocol as a separate protocol from `Actor` because +distributed actors only behave like actors when they are known to be +local. An `isolated` distributed actor parameter is known to be local, so +it has the capabilities of an actor. The following local API on +`DistributedActor` is provided to return a local actor instance from a +distributed actor, enabling distributed actors to be used with isolated +parameters of type `isolated any Actor` and `isolated (any Actor)?`: + +```swift +@available(SwiftStdlib 5.7, *) +extension DistributedActor { + /// Produces an erased `any Actor` reference to this known to be local distributed actor. + /// + /// Since this method is not distributed, it can only be invoked when the underlying + /// distributed actor is known to be local, e.g. from a context that is isolated + /// to this actor. + /// + /// Such reference can be used to work with APIs accepting `isolated any Actor`, + /// as only a local distributed actor can be isolated on and may be automatically + /// erased to such `any Actor` when calling methods implicitly accepting the + /// caller's actor isolation, e.g. by using the `#isolation` macro. + @backDeployed(before: SwiftStdlib 6.0) + public var asLocalActor: any Actor { +} +``` + +### Generalized isolation checking + +When calling a function with an `isolated` parameter, the function +shares the same isolation as the current context if: + +- the current context is non-isolated, the parameter type is optional, + and the argument expression is `nil` or a reference to `Optional.none`; + +- the current context has an `isolated` parameter (including the + implicitly-`isolated` `self` parameter of an actor function) and + the argument expression is a reference to that parameter, a + non-optional derivation of it (see below), or a local actor derivation + from a distributed actor using `DistributedActor.asAnyActor`; or + +- the current context is isolated to a global actor type `T` and the + argument expression is `T.shared`, where `shared` is `GlobalActor`'s + protocol requirement or the concrete declaration which provides it + in `T`'s conformance to `GlobalActor`. + +An expression is a non-optional derivation of an isolated parameter +`param` if it is: +- `param?` (the optional-chaining operator); +- `param!` (the force-unwrapping operator); or +- a reference to a *non-optional binding* of `param`, i.e. a `let` + constant initialized by a successful pattern-match which removes + the optionality from `param`, such as `ref` in `if let ref = param`. + +When analyzing an argument expression in all cases above, certain +non-instrumental differences in expression syntax and behavior must +be ignored: +- parentheses; +- the effect-marking operators `try`, `try?`, `try!`, and `await`;[^5] +- the type coercion operator `as` (in the cases where it doesn't + perform a dynamic bridging conversion); and +- implicit type conversions such as promotion to `Optional` type. + +[^5]: The restrictions on the underlying expression should make it +pointless to use these operators, but they must be ignored anyway. + +Note that the special `#isolation` default argument form should +always be replaced by something matching the rule above, so calls +using this default argument for an isolated parameter will always be +to a context that shares isolation. + +For example: + +```swift +/// This class type is not Sendable. +class Counter { + var count = 0 +} + +extension Counter { + /// Since this is an async function, if it were just declared + /// non-isolated, calling it from an isolated context would be + /// forbidden because it requires sharing a non-Sendable value + /// between concurrency domains. Inheriting isolation makes it + /// okay. This is a contrived example chosen for its simplicity. + func incrementAndSleep(isolation: isolated (any Actor)?) async { + count += 1 + await Task.sleep(nanoseconds: 1_000_000) + } +} + +actor MyActor { + var counter = Counter() +} + +extension MyActor { + func testActor(other: MyActor) { + // allowed + await counter.incrementAndSleep(isolation: self) + + // not allowed + await counter.incrementAndSleep(isolation: other) + + // not allowed + await counter.incrementAndSleep(isolation: MainActor.shared) + + // not allowed + await counter.incrementAndSleep(isolation: nil) + } +} + +@MainActor func testMainActor(counter: Counter) { + // allowed + await counter.incrementAndSleep(isolation: MainActor.shared) + + // not allowed + await counter.incrementAndSleep(isolation: nil) +} + +func testNonIsolated(counter: Counter) { + // allowed + await counter.incrementAndSleep(isolation: nil) + + // not allowed + await counter.incrementAndSleep(isolation: MainActor.shared) +} +``` + +### `#isolation` default argument + +The special expression form `#isolation` can be used in arbitrary +expression position: + +```swift +extension AsyncIteratorProtocol { + func next(isolation: isolated (any Actor)? = #isolation) async -> Element { + ... + } +} +``` + +When a call uses `#isolation` as the argument to an isolated parameter, +it behaves as if the argument was an expression representing the static +actor isolation of the current context: + +- if the current context is statically non-isolated, the parameter + must have optional type, and the argument is `nil`; +- if the current context is isolated to a global actor `T`, the argument + is `T.shared`; +- if the current context has an `isolated` actor parameter (including the + implicit `self` parameter of an actor method), the argument is a + reference to that parameter; +- if the current context has an `isolated` distributed actor parameter + `d` (including the implicit `self` parameter of a distributed actor + method), the argument is `d.asAnyActor`; +- otherwise, the current context must be a closure which captures + an `isolated` parameter or a non-optional binding of it, and the + argument is a reference to that capture. + +The type of `#isolation` depends on the type annotation provided in the +context of the expression, similar to other builtin macros such as `#file` +and `#line`, with a default type of `(any Actor)?` if no contextual type +is provided. When type-checking considers a candidate function for a call +that would use `#isolation` as an argument for a parameter, +it assumes that the notional argument expression above can be coerced +to the parameter type. If the call is actually resolved to use that +candidate, the coercion must succeed or the call is ill-formed. +This rule is necessary in order to avoid the need to decide the isolation +of the calling context before resolving calls from it. + +The parameter does not have to be an `isolated` parameter. + +## Source compatibility + +This proposal is largely additive and should not affect the behavior +of existing code. + +The new rules for isolation checking permit more calls to be +recognized as sharing isolation. This should strictly allow +more code to be compiled; it cannot cause source-compatibility +regressions by allowing different overloads to be picked because +isolation checking is performed separately from type-checking. + +## ABI compatibility + +This proposal does not change how any existing code is compiled. + +## Implications for adoption + +This proposal does not add any new types and does not require new +runtime or library support. It can be implemented purely in the compiler. + +Adding `#isolation` as a default argument to an existing parameter is not +ABI-breaking, but this is probably an uncommon situation. Adding a new +parameter to an existing declaration is ABI-breaking, of course. + +Making a library function inherit isolation is effectively a promise that +it can work when called from any isolated context. While this might seem +superficially like a pretty strong guarantee, it's not very different +in practice from just making the library function non-isolated: in both +cases, the function does not have any isolation preconditions that it can +rely on. Library authors should not be reserved about adopting this +proposal on that account. + +A better reason to be cautious about adopting this feature is that it +can cause more work to be done while actor-isolated, potentially creating +significant "hangover" on the actor lock and a less effective use of +concurrency. It may be better for the whole system if functions that do +significant computational work, including doing a lot of object +allocation and initialization, stay non-isolated rather than +isolation-inheriting. On the other hand, `async` functions with "fast +paths" --- functions that usually return quickly and only occasionally +need to set up more expensive work --- may see real benefits from +extracting the fast path into a function that inherits isolation and +then leaving the slow path in a non-isolated function. + +## Future directions + +### Syntax sugar for inheriting actor isolation + +Isolated parameters have three downsides. + +The first downside is that the use pattern we expect to dominate --- +declaring a function to inherit its caller's isolation --- is pretty +cumbersome: + +```swift +func foo(isolation: isolated (any Actor)? = #isolation) +``` + +The second downside is we can only do this if we can add formal +parameters to a function. Unfortunately, there are several places +in the language where we really can't do that, most importantly +accessors for computed properties: + +```swift +var count: Int { + get { // How do we add an isolated parameter here? + ... + } +} +``` + +The third downside is minor in comparison, but this pattern naturally +turns into passing an actor reference, which isn't the most efficient +way of passing down actor isolation because it still requires dynamic +dispatch in order to extract the executor. It would be better for the +implementation if we could pass down the exact `UnownedSerialExecutor?` +value that's needed at runtime. While we do not currently want to +encourage programmers to work with values of this type directly because +of its tricky lifetime semantics, the compiler can manage it fairly +easily. + +All of these downsides could be addressed by adding an attribute +which causes an entity (including an accessor) to inherit its caller's +isolation. This would be equivalent to receiving an `isolated` parameter +with the same value as would be produced by `#isolated`, but it's easier +to write, can be used in a few places that can't add arbitrary parameters, +and may be more efficiently implementable. + +### Isolated function types + +This proposal is focused on propagating isolation information *into* +functions, but it's also interesting to look at propagating isolation +*out* of functions. Currently, the Swift type system only allows +function isolation to be expressed in limited ways: functions can be +declared as isolated to a global actor (e.g. `@MainActor () -> ()`), but +all other kinds of isolation must be "type-erased", leaving a value +whose type appears to be non-isolated. + +One way to solve this would be to introduce value-dependent isolated +function types. With such a feature, you could declare a value to have +type, say, `@isolated(myActor) () -> ()`, where `myActor` is a `let` +constant in the local scope. This kind of value dependence, however, +is a large step in complexity for a type system, and it's not a likely +path for Swift in the foreseeable future. + +A more promising approach would be to allow the isolation to be +statically erased but still make it dynamically recoverable by carrying +it along in the function value, essentially as an extra value of type +`(any Actor)?`. A function type that supports dynamically recovering +the isolation would look something like `@isolated () -> ()`, +and it could be used to e.g. dynamically propagate the isolation of +a function into something like the `Task` initializer so that the task +can immediately start on the right executor. This would compose well +with the features in this proposal because it would be natural to allow +such functions to be used as `isolated` parameters. This would be very +nice for functions like `sequentialMap` that should probably be isolated +not to their *caller* but to the *function they've been passed*: + +```swift +extension Collection { + func sequentialMap(transform: (Element) async -> R) async -> [R] { + var results: [R] = [] + for elt in self { + results.append(await transform(elt)) + } + return results + } +} +``` + +## Alternatives Considered + +### Allowing isolation to `SerialExecutor` types + +This proposal observes that it is more efficient to pass down an +`UnownedSerialExecutor` value instead of an actor reference. However, a +function cannot use this more efficient pattern because an `isolated` +parameter must be an actor type. This is an intentional decision. + +Philosophically, Swift programmers should be encouraged to think about +actors in terms of isolation rather than execution policy. There are many +ways for actors to provide isolation, many of which don't require taking +over execution; in fact, Swift's actors use one such approach by default. +Keeping the focus on actors rather than executors supports this. + +Putting that aside, there also just isn't a reasonable type that could +be used here: + +- `UnownedSerialExecutor` is an unsafe type that requires the compiler +to implicitly manage a dependency on the underlying actor or executor +reference in order to safely use. While this is not difficult for the +compiler, we do not want to encourage programmers to use this type +directly. If Swift introduces a safe replacement in the future, possibly +using future language support for value dependencies, we can consider +allowing that to be used as an `isolated` parameter type at that time. + +- A managed serial executor reference such as `any SerialExecutor` +would be a safe alternative, but it's a surprisingly complex one. +For one, normal isolated contexts would not to be able to implement +`#isolation` forwarding to such a parameter, because there's currently no +way to get a managed serial executor reference from an actor, only an +`UnownedSerialExecutor`. For another, actor types can (and often do) +also conform to `SerialExecutor`, but there's nothing in the language +requiring those actors to always use `self` as their executor. This +greatly complicates the logic for both establishing and forwarding +isolation; e.g. an isolated *actor* parameter must not be forwarded +directly as an isolated *executor* (as opposed to extracting the +correct executor reference) even if the actor's type would normally +implicitly convert. Furthermore, the decision logic for whether a call +crosses isolation would have to recognize expressions that extract serial +executors, as well as appropriately reasoning about actor/executor +differences. And finally, getting an `UnownedSerialExecutor` from an +`any SerialExecutor` still requires calling a protocol method, so it's +not really enabling any sort of optimization. + +## Acknowledgments + +I'd like to especially thank Konrad Malawski, and Doug Gregor +for their help in developing the ideas in this proposal. diff --git a/proposals/0421-generalize-async-sequence.md b/proposals/0421-generalize-async-sequence.md new file mode 100644 index 0000000000..c0c19c4550 --- /dev/null +++ b/proposals/0421-generalize-async-sequence.md @@ -0,0 +1,271 @@ +# Generalize effect polymorphism for `AsyncSequence` and `AsyncIteratorProtocol` + +* Proposal: [SE-0421](0421-generalize-async-sequence.md) +* Authors: [Doug Gregor](https://github.com/douggregor), [Holly Borla](https://github.com/hborla) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-generalize-asyncsequence-and-asynciteratorprotocol/69283))([review](https://forums.swift.org/t/se-0421-generalize-effect-polymorphism-for-asyncsequence-and-asynciteratorprotocol/69662)) ([acceptance](https://forums.swift.org/t/accepted-se-0421-generalize-effect-polymorphism-for-asyncsequence-and-asynciteratorprotocol/69973)) + +## Introduction + +This proposal generalizes `AsyncSequence` in two ways: +1. Proper `throws` polymorphism is accomplished with adoption of typed throws. +2. A new overload of the `next` requirement on `AsyncIteratorProtocol` includes an isolated parameter to abstract over actor isolation. + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Proposed solution](#proposed-solution) +* [Detailed design](#detailed-design) + + [Adopting typed throws](#adopting-typed-throws) + - [Error type inference from `for try await` loops](#error-type-inference-from-for-try-await-loops) + + [Adopting primary associated types](#adopting-primary-associated-types) + + [Adopting isolated parameters](#adopting-isolated-parameters) + + [Default implementations of `next()` and `next(isolation:)`](#default-implementations-of-next-and-nextisolation) + + [Associated type inference for `AsyncIteratorProtocol` conformances](#associated-type-inference-for-asynciteratorprotocol-conformances) +* [Source compatibility](#source-compatibility) +* [ABI compatibility](#abi-compatibility) +* [Implications on adoption](#implications-on-adoption) +* [Future directions](#future-directions) + + [Add a default argument to `next(isolation:)`](#add-a-default-argument-to-nextisolation) +* [Alternatives considered](#alternatives-considered) + + [Avoiding an existential parameter in `next(isolation:)`](#avoiding-an-existential-parameter-in-nextisolation) +* [Acknowledgments](#acknowledgments) + +## Motivation + +`AsyncSequence` and `AsyncIteratorProtocol` were intended to be polymorphic over the `throws` effect and actor isolation. However, the current API design has serious limitations that impact expressivity in generic code, `Sendable` checking, and runtime performance. + +Some `AsyncSequence`s can throw during iteration, and others never throw. To enable callers to only require `try` when the given sequence can throw, `AsyncSequence` and `AsyncIteratorProtocol` used an experimental feature to try to capture the throwing behavior of a protocol. However, this approach was insufficiently general, which has also [prevented `AsyncSequence` from adopting primary associated types](https://forums.swift.org/t/se-0346-lightweight-same-type-requirements-for-primary-associated-types/55869/70). Primary associated types on `AsyncSequence` would enable hiding concrete implementation details behind constrained opaque or existential types, such as in transformation APIs on `AsyncSequence`: + +```swift +extension AsyncSequence { + // 'AsyncThrowingMapSequence' is an implementation detail hidden from callers. + public func map( + _ transform: @Sendable @escaping (Element) async throws -> Transformed + ) -> some AsyncSequence { ... } +} +``` + +Additionally, `AsyncSequence` types are designed to work with `Sendable` and non-`Sendable` element types, but it's currently impossible to use an `AsyncSequence` with non-`Sendable` elements in an actor-isolated context: + +```swift +class NotSendable { ... } + +@MainActor +func iterate(over stream: AsyncStream) { + for await element in stream { // warning: non-sendable type 'NotSendable?' returned by implicitly asynchronous call to nonisolated function cannot cross actor boundary + + } +} +``` + +Because `AsyncIteratorProtocol.next()` is `nonisolated async`, it always runs on the generic executor, so calling it from an actor-isolated context crosses an isolation boundary. If the result is non-`Sendable`, the call is invalid under strict concurrency checking. + +More fundamentally, calls to `AsyncIteratorProtocol.next()` from an actor-isolated context are nearly always invalid in practice today. Most concrete `AsyncIteratorProtocol` types are not `Sendable`; concurrent iteration using `AsyncIteratorProtocol` is a programmer error, and the iterator is intended to be used/mutated from the isolation domain that formed it. However, when an iterator is formed in an actor-isolated context and `next()` is called, the non-`Sendable` iterator is passed across isolation boundaries, resulting in a diagnostic under strict concurrency checking. + +Finally, `next()` always running on the generic executor is the source of unnecessary hops between an actor and the generic executor. + +## Proposed solution + +This proposal introduces a new associated type `Failure` to `AsyncSequence` and `AsyncIteratorProtocol`, adopts both `Element` and `Failure` as primary associated types, adds a new protocol requirement to `AsyncIteratorProtocol` that generalizes the existing `next()` requirement by throwing the `Failure` type, and adds an `isolated` parameter to the new requirement to abstract over actor isolation: + +```swift +@available(SwiftStdlib 5.1, *) +protocol AsyncIteratorProtocol { + associatedtype Element + + mutating func next() async throws -> Element? + + @available(SwiftStdlib 6.0, *) + associatedtype Failure: Error = any Error + + @available(SwiftStdlib 6.0, *) + mutating func next(isolation actor: isolated (any Actor)?) async throws(Failure) -> Element? +} + +@available(SwiftStdlib 5.1, *) +public protocol AsyncSequence { + associatedtype AsyncIterator: AsyncIteratorProtocol + associatedtype Element where AsyncIterator.Element == Element + + @available(SwiftStdlib 6.0, *) + associatedtype Failure = AsyncIterator.Failure where AsyncIterator.Failure == Failure + + func makeAsyncIterator() -> AsyncIterator +} +``` + +The new `next(isolation:)` has a default implementation so that conformances will continue to behave as they do today. Code generation for `for-in` loops will switch over to calling `next(isolation:)` instead of `next()` when the context has appropriate availability. + +## Detailed design + +### Adopting typed throws + +Concrete `AsyncSequence` and `AsyncIteratorProtocol` types determine whether calling `next()` can `throw`. This can be described in each protocol with a `Failure` associated type that is thrown by the `AsyncIteratorProtocol.next(isolation:)` requirement. Describing the thrown error with an associated type allows conformances to fulfill the requirement with a type parameter, which means that libraries do not need to expose separate throwing and non-throwing concrete types that otherwise have the same async iteration functionality. + +#### Error type inference from `for try await` loops + +The `Failure` associated type is only accessible at runtime in the Swift 6.0 standard library; code running against older standard library versions does not include the `Failure` requirement in the witness tables for `AsyncSequence` and `AsyncIteratorProtocol` conformances. This impacts error type inference from `for try await` loops. + +When the thrown error type of an `AsyncIteratorProtocol` is available, either through the associated type witness (because the context has appropriate availability) or because the iterator type is concrete, iteration over an async sequence throws its `Failure` type: + +```swift +struct MyAsyncIterator: AsyncIteratorProtocol { + typealias Failure = MyError + ... +} + +func iterate(over s: S) where S.AsyncIterator == MyAsyncIterator { + let closure = { + for try await element in s { + print(element) + } + } +} +``` + +In the above code, the type of `closure` is `() async throws(MyError) -> Void`. + +When the thrown error type of an `AsyncIteratorProtocol` is not available, iteration over an async sequence throws `any Error`: + +```swift +@available(SwiftStdlib 5.1, *) +func iterate(over s: some AsyncSequence) { + let closure = { + for try await element in s { + print(element) + } + } +} +``` + +In the above code, the type of `closure` is `() async throws(any Error) -> Void`. + +When the `Failure` type of the given async sequence is constrained to `Never`, `try` is not required in the `for-in` loop: + +```swift +struct MyAsyncIterator: AsyncIteratorProtocol { + typealias Failure = Never + ... +} + +func iterate(over s: S) where S.AsyncIterator == MyAsyncIterator { + let closure = { + for await element in s { + print(element) + } + } +} +``` + +In the above code, the type of `closure` is `() async -> Void`. + +### Adopting primary associated types + +The `Element` and `Failure` associated types are promoted to primary associated types. This enables using constrained existential and opaque `AsyncSequence` and `AsyncIteratorProtocol` types, e.g. `some AsyncSequence` or `any AsyncSequence`. + +### Adopting isolated parameters + +The `next(isolation:)` requirement abstracts over actor isolation using [isolated parameters](/proposals/0313-actor-isolation-control.md). For callers to `next(isolation:)` that pass an iterator value that cannot be transferred across isolation boundaries under [SE-0414: Region based isolation](/proposals/0414-region-based-isolation.md), the call is only valid if it does not cross an isolation boundary. Explicit callers can pass in a value of `#isolation` to use the isolation of the caller, or `nil` to evaluate `next(isolation:)` on the generic executor. + +Desugared async `for-in` loops will call `AsyncIteratorProtocol.next(isolation:)` instead of `next()` when the context has appropriate availability, and pass in an isolated argument value of `#isolation` of type `(any Actor)?`. The `#isolation` macro always expands to the isolation of the caller so that the call does not cross an isolation boundary. + +### Default implementations of `next()` and `next(isolation:)` + +Because existing `AsyncIteratorProtocol`-conforming types only implement `next()`, the standard library provides a default implementation of `next(isolation:)`: + +```swift +extension AsyncIteratorProtocol { + /// Default implementation of `next(isolation:)` in terms of `next()`, which is + /// required to maintain backward compatibility with existing async iterators. + @available(SwiftStdlib 6.0, *) + @available(*, deprecated, message: "Provide an implementation of 'next(isolation:)'") + public mutating func next(isolation actor: isolated (any Actor)?) async throws(Failure) -> Element? { + nonisolated(unsafe) var unsafeIterator = self + do { + let element = try await unsafeIterator.next() + self = unsafeIterator + return element + } catch { + throw error as! Failure + } + } +} +``` + +Note that the default implementation of `next(isolation:)` necessarily violates `Sendable` checking in order to pass `self` from a possibly-isolated context to a `nonisolated` one. Though this is generally unsafe, this is how calls to `next()` behave today, so existing conformances will maintain the behavior they already have. Implementing `next(isolation:)` directly will eliminate the unsafety. + +To enable conformances of `AsyncIteratorProtocol` to only implement `next(isolation:)`, a default implementation is also provided for `next()`: + +```swift +extension AsyncIteratorProtocol { + @available(SwiftStdlib 6.0, *) + public mutating func next() async throws -> Element? { + // Callers to `next()` will always run `next(isolation:)` on the generic executor. + try await next(isolation: nil) + } +} +``` + +Both function requirements of `AsyncIteratorProtocol` have default implementations that are written in terms of each other, meaning that it is a programmer error to implement neither of them. Types that are available prior to the Swift 6.0 standard library must provide an implementation of `next()`, because the default implementation is only available with the Swift 6.0 standard library. + +To avoid silently allowing conformances that implement neither requirement, and to facilitate the transition of conformances from `next()` to `next(isolation:)`, we add a new availability rule where the witness checker diagnoses a protocol conformance that uses an deprecated, obsoleted, or unavailable default witness implementation. Deprecated implementations will produce a warning, while obsoleted and unavailable implementations will produce an error. + +Because the default implementation of `next(isolation:)` is deprecated, conformances that do not provide a direct implementation will produce a warning. This is desirable because the default implementation of `next(isolation:)` violates `Sendable` checking, so while it's necessary for source compatibility, it's important to aggressively suggest that conforming types implement the new method. + +### Associated type inference for `AsyncIteratorProtocol` conformances + +When an `AsyncIteratorProtocol`-conforming type provides a `next(isolation:)` function, the `Failure` type is inferred based on whether (and what) `next(isolation:)` throws using the rules described in [SE-0413](/proposals/0413-typed-throws.md). + +If the `AsyncIteratorProtocol`-conforming type uses the default implementation of `next(isolation:)`, then the `Failure` associated type is inferred from the `next` function instead. Whatever type is thrown from the `next` function (including `Never` if it is non-throwing) is inferred as the `Failure` type. + +## Source compatibility + +The new requirements to `AsyncSequence` and `AsyncIteratorProtocol` are additive, with default implementations and `Failure` associated type inference heuristics that ensure that existing types that conform to these protocols will continue to work. + +The experimental "rethrowing conformances" feature used by `AsyncSequence` and `AsyncIteratorProtocol` presents some challenges for source compatibility. Namely, one can declare a `rethrows` function that considers conformance to these rethrowing protocols as sources of errors for rethrowing. For example, the following `rethrows` function is currently valid: + +```swift +extension AsyncSequence { + func contains(_ value: Element) rethrows -> Bool where Element: Hashable { ... } +} +``` + +With the removal of the experimental "rethrowing conformances" feature, this function becomes ill-formed because there is no closure argument that can throw. To preserve source compatibility for such functions, this proposal introduces a specific rule that allows requirements on `AsyncSequence` and `AsyncIteratorProtocol` to be involved in `rethrows` checking: a `rethrows` function is considered to be able to throw `T.Failure` for every `T: AsyncSequence` or `T: AsyncIteratorProtocol` conformance requirement. In the case of this `contains` operation, that means it can throw `Self.Failure`. The rule permitting the definition of these `rethrows` functions will only be permitted prior to Swift 6. + +## ABI compatibility + +This proposal is purely an extension of the ABI of the standard library and does not change any existing features. Note that the addition of a new `next(isolation:)` requirement, rather than modifying the existing `next()` requirement, is necessary to maintain ABI compatibility, because changing `next()` to abstract over actor isolation requires passing the actor as a parameter in order to hop back to that actor after any `async` calls in the implementation. The typed throws ABI is also different from the rethrows ABI, so the adoption of typed throws alone necessitates a new requirement. + +## Implications on adoption + +The associated `Failure` types of `AsyncSequence` and `AsyncIteratorProtocol` are only available at runtime with the Swift 6.0 standard library, because code that runs against prior standard library versions does not have a witness table entry for `Failure`. Code that needs to access the `Failure` type through the associated type, e.g. to dynamic cast to it or constrain it in a generic signature, must be availability constrained. For this reason, the default implementations of `next()` and `next(isolation:)` have the same availability as the Swift 6.0 standard library. + +This means that concrete `AsyncIteratorProtocol` conformances cannot switch over to implementing `next(isolation:)` only (without providing an implementation of `next()`) if they are available earlier than the Swift 6.0 standard library. + +Similarly, primary associated types of `AsyncSequence` and `AsyncIteratorProtocol` must be gated behind Swift 6.0 availability. + +Once the concrete `AsyncIteratorProtocol` types in the standard library, such as `Async{Throwing}Stream.Iterator`, implement `next(isolation:)` directly, code that iterates over those concrete `AsyncSequence` types in an actor-isolated context may exhibit fewer hops to the generic executor at runtime. + +## Future directions + +### Add a default argument to `next(isolation:)` + +Most calls to `next(isolation:)` will pass the isolation of the enclosing context. We could consider lifting the restriction that protocol requirements cannot have default arguments, and adding a default argument value of `#isolated` as described in the [pitch for actor isolation inheritance](https://forums.swift.org/t/pitch-inheriting-the-callers-actor-isolation/68391). + +## Alternatives considered + +### Avoiding an existential parameter in `next(isolation:)` + +The isolated parameter to `next(isolation:)` has existential type `(any Actor)?` because a `nil` value is used to represent `nonisolated`. There is no concrete `Actor` type that describes a `nonisolated` context, which necessitates using `(any Actor)?` instead of `some Actor` or `(some Actor)?`. Potential alternatives to this are: + +1. Represent `nonisolated` with some other value than `nil`, or a specific declaration in the standard library that has a concrete optional actor type to enable `(some Actor)?`. Any solution in this category requires the compiler to have special knowledge of the value that represents `nonisolated` for actor isolation checking of the call. +2. Introduce a separate entrypoint for `next(isolation:)` that is always `nonisolated`. This defeats the purpose of having a single implementation of `next(isolation:)` that abstracts over actor isolation. + +Note that the use of an existential type `(any Actor)?` means that [embedded Swift](/visions/embedded-swift.md) would need to support class existentials in order to use `next(isolation:)`. + +## Acknowledgments + +Thank you to Franz Busch and Konrad Malawski for starting the discussions about typed throws and primary associated type adoption for `AsyncSequence` and `AsyncIteratorProtocol` in the [Typed throws in the Concurrency module](https://forums.swift.org/t/pitch-typed-throws-in-the-concurrency-module/68210/1) pitch. Thank you to John McCall for specifying the rules for generalized isolated parameters in the [pitch for inheriting the caller's actor isolation](https://forums.swift.org/t/pitch-inheriting-the-callers-actor-isolation/68391). diff --git a/proposals/0422-caller-side-default-argument-macro-expression.md b/proposals/0422-caller-side-default-argument-macro-expression.md new file mode 100644 index 0000000000..92b9f6f836 --- /dev/null +++ b/proposals/0422-caller-side-default-argument-macro-expression.md @@ -0,0 +1,196 @@ +# Expression macro as caller-side default argument + +* Proposal: [SE-0422](0422-caller-side-default-argument-macro-expression.md) +* Authors: [Apollo Zhu](https://github.com/ApolloZhu) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-expression-macro-as-caller-side-default-argument/69019)), ([review](https://forums.swift.org/t/se-0422-expression-macro-as-caller-side-default-argument/69730)), ([acceptance](https://forums.swift.org/t/accepted-se-0422-expression-macro-as-caller-side-default-argument/70050)) + +## Introduction + +This proposal aims to lift the restriction afore set in [SE-0382 "Expression macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) to allow non-built-in expression macros as caller-side default argument expressions. + +## Motivation + +Built-in magic identifiers like [#line](https://developer.apple.com/documentation/swift/line()) and [#fileID](https://developer.apple.com/documentation/swift/fileID()) are documented as expression macros in the official documentation, but if Swift developers try to implement a similar macro themselves and use it as the default argument for some function, the code will not compile: + +```swift +public struct MakeLabeledPrinterMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> ExprSyntax { + return "{ value in print(\"\\(#fileID):\\(#line): \\(value)\") }" + } +} + +public macro LabeledPrinter() -> (T) -> Void += #externalMacro(module: ..., type: "MakeLabeledPrinterMacro") + +public func greet( + _ thing: T, + print: (T) -> Void = #LabeledPrinter +// error: ^ non-built-in macro cannot be used as default argument +) { + print("Hello, \(thing)") +} +``` + +This is because built-in expression macros/magic identifiers have a special behavior: when used as default arguments, instead of been expanded at where the expressions are written like all other macros, they are expanded by the caller using the source-location information of the call site: + +```swift +// in MyLibrary.swift +public func greet(_ thing: T, file: String = #fileID) { + print("\(fileID): Hello, \(thing)" +} + +// in main.swift +greet("World") +// prints "main.swift: Hello, World" instead of "MyLibrary.swift: ... +``` + +This a useful existing behavior that should be supported, but could be surprising as it differs from all other macro expansions, and might not be desired for all expression macros. + +## Proposed solution + +The proposal lifts the restriction and makes non-built-in expression macros behave consistently as built-in magic identifier expression macros: + +* if expression macros are used as default arguments, they’ll be expanded with caller side source location information and context; +* if they are used as sub-expressions of default arguments, they’ll be expanded at where they are written + +```swift +// in MyLibrary.swift ======= +@freestanding(expression) +macro MyFileID() -> T = ... + +public func callSiteFile(_ file: String = #MyFileID) { file } + +public func declarationSiteFile(_ file: String = (#MyFileID)) { file } + +public func alsoDeclarationSiteFile( + file: String = callSiteFile(#MyFileID) +) { file } + +// in main.swift ============ +print(callSiteFile()) // print main.swift, the current file +print(declarationSiteFile()) // always prints MyLibrary.swift +print(alsoDeclarationSiteFile()) // always prints MyLibrary.swift +``` + +Macro author can inquire the source location information using `context.location(of:)` just like before and implement `#fileID`, `#line`, and `#column` as shown below: + +```swift +public struct MyFileIDMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + context.location( + of: node, at: .afterLeadingTrivia, filePathMode: .fileID + )!.file + } +} + +public struct MyLineMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + context.location(of: node)!.line + } +} + +public struct MyColumnMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + context.location(of: node)!.column + } +} +``` + +## Detailed design + +### Type-checking default argument macro expressions + +Since the macro expanded expression might reference declarations that are not available in the scope where the function is declared, macro expressions are not expanded at the primary function declaration. However, macro expression used as a default argument is type checked without expansion to make sure that + +1. it is at least as visible as the function using it, +2. its return type matches what that parameter expects, and +3. its arguments, if any, are literals without string interpolation. + +### Type-checking macro expanded expressions + +For each call to a function that has an expression macro default argument, the macro will be expanded with each call-site’s source location and type-checked in the corresponding caller-side context, as if the macro expression is written at where it is expanded: + +```swift +@freestanding(expression) +// expands to `foo + bar` +public macro VariableReferences() -> String = ... + +public func preferVariablesFromCallerSide( + param: String = #VariableReferences +) { + print(param) +} + +// in another file ========== +var foo = "hi " +var bar = "caller" +preferVariablesFromCallerSide() // prints: hi caller +// ^ same as #VariableReferences written here +``` + +## Source compatibility + +As non-built-in macro expressions aren’t allowed as default argument, this change is purely additive and has no impact on existing code. + +## ABI compatibility + +This feature does not affect the ABI. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. + +## Future directions + +### Allow arguments to default argument macro expressions to be arbitrary expressions + +If these arguments can be arbitrary expressions, type-checking the macro expression at function declaration will require any declarations referenced in these expressions to be also in scope: + +```swift +@freestanding(expression) +// expands to: "Hello " + string +public macro PrependHello(_ string: String) -> String = ... + +// this is needed so it can be referenced in the default argument +public var shadowedVariable: String = "World" + +public func preferVariablesFromCallerSide( + param: String = #PrependHello(shadowedVariable) +) { + print(param) +} +``` + +However, as the expanded expression is type-checked in the caller-side context, it’s rather unintuitive that one must add the public variable in the example above, yet it might not be what the macro expanded expressions use. For example, if there's a variable with the same name in scope on the caller side, that variable will be used, and the call to the function might fail to type-check: + +```swift +// in another file ========== +var shadowedVariable: Int = 42 +preferVariablesFromCallerSide() +// #PrependHello(shadowedVariable) expands to "Hello " + 42 +// error: binary operator '+' cannot be applied to operands of type 'String' and 'Int' +``` + +## Alternatives considered + +### Expand non-built-in expression macro default arguments at the primary declaration + +While this allows all macro expansions to be expanded at where they are written, it creates an inconsistency for expression macros where they behave differently depending on whether they are built-in or not. Therefore, this alternative won’t be a solution for addressing the surprising behavior of built-in expression macros as caller-side default arguments, while the proposed solution unifies, and clarifies how to make expression macro default arguments expand at caller-side vs. at function declaration. + +## Acknowledgments + +Thanks to Doug Gregor, Richard Wei, and Holly Borla for early feedback and suggestions on design and implementation. diff --git a/proposals/0423-dynamic-actor-isolation.md b/proposals/0423-dynamic-actor-isolation.md new file mode 100644 index 0000000000..fb7215418e --- /dev/null +++ b/proposals/0423-dynamic-actor-isolation.md @@ -0,0 +1,182 @@ +# Dynamic actor isolation enforcement from non-strict-concurrency contexts + +* Proposal: [SE-0423](0423-dynamic-actor-isolation.md) +* Authors: [Holly Borla](https://github.com/hborla), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.0)** +* Upcoming Feature Flag: `DynamicActorIsolation` +* Review: ([pitch](https://forums.swift.org/t/pitch-dynamic-actor-isolation-enforcement/68354)) ([first review](https://forums.swift.org/t/se-0423-dynamic-actor-isolation-enforcement-from-non-strict-concurrency-contexts/70155)) ([second review](https://forums.swift.org/t/se-0423-second-review-dynamic-actor-isolation-enforcement-from-non-strict-concurrency-contexts/71159)) ([acceptance](https://forums.swift.org/t/accepted-se-0423-dynamic-actor-isolation-enforcement-from-non-strict-concurrency-contexts/71540)) + +## Introduction + +Many Swift programs need to interoperate with frameworks written in C/C++/Objective-C whose implementations cannot participate in static data race safety. Similarly, many Swift programs have dependencies that have not yet adopted strict concurrency checking. A `@preconcurrency import` statement downgrades concurrency-related error messages that the programmer cannot resolve because the fundamental issue is in one of the dependencies. To strengthen Swift's data-race safety guarantees while working with preconcurrency dependencies, this proposals adds actor isolation checking at runtime for synchronous isolated functions. + +## Motivation + +The ecosystem of Swift libraries has a vast surface area of APIs that predate strict concurrency checking, relying on carefully calling APIs from the appropriate thread or dispatch queue to avoid data races. Migrating all of these libraries to strict concurrency checking will happen incrementally, motivating [SE-0337: Incremental migration to concurrency checking](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md) which introduced the `@preconcurrency import` statement to suppress concurrency warnings from APIs that programmers do not control. + +If an actor isolation violation exists in the implementation of a preconcurrency library, the bug is only surfaced to clients as hard-to-debug data races on isolated state. `@preconcurrency` also does not apply to protocol conformances; there is no way to suppress concurrency diagnostics when conforming to a protocol from a preconcurrency library. This is unfortunate, because it's common for protocols to have a dynamic invariant that all requirements are called on the main thread or a specific dispatch queue provided by the client. + +For example, consider the following protocol in a library called `NotMyLibrary`, which provides a guarantee that its requirements are always called from the main thread: + +```swift +public protocol ViewDelegateProtocol { + func respondToUIEvent() +} +``` + +and a client of `NotMyLibrary` that contains a conformance to `ViewDelegateProtocol`: + +```swift +import NotMyLibrary + +@MainActor +class MyViewController: ViewDelegateProtocol { + func respondToUIEvent() { // error: @MainActor function cannot satisfy a nonisolated requirement + // implementation... + } +} +``` + +The above code is invalid because `MyViewController.respondToUIEvent()` is `@MainActor`-isolated, but it satisfies a `nonisolated` protocol requirement that can be called from generic code off the main actor. If the library provides a dynamic guarantee that the requirement is always called on the main actor, a sensible workaround is to resort to dynamic actor isolation checking by marking the function as `nonisolated` and wrapping the implementation in `MainActor.assumeIsolated`: + +```swift +import NotMyLibrary + +@MainActor +class MyViewController: ViewDelegateProtocol { + nonisolated func respondToUIEvent() { + MainActor.assumeIsolated { + // implementation... + } + } +} +``` + +With this workaround, the programmer must annotate every witness with `nonisolated` and wrap the implementation in `MainActor.assumeIsolated`. More importantly, the programmer loses static data-race safety in their own code, because internal callers of `respondToUIEvent()` are free to invoke it from any isolation domain without compiler errors. + +## Proposed solution + +This proposal adds dynamic actor isolation checking to: + + - Witnesses of synchronous `nonisolated` protocol requirements when the witness is isolated and the protocol conformance is annotated as `@preconcurrency`. For example: + + If `respondToUIEvent` is a witness to a synchronous `nonisolated` protocol requirement, the protocol conformance error can be suppressed using a `@preconcurrency` annotation on the protocol to indicate that the protocol itself predates concurrency: + + ```swift + import NotMyLibrary + + @MainActor + class MyViewController: @preconcurrency ViewDelegateProtocol { + func respondToUIEvent() { + // implementation... + } + } + ``` + + The witness checker diagnostic will be suppressed, the actor isolation assertion will fail if `respondToUIEvent()` is called inside `NonMyLibrary` from off the main actor, and the compiler will continue to emit diagnostics inside the module when called from off the main actor. + + These dynamic checks apply to any situation where a synchronous `nonisolated` requirement is implemented by an isolated method, including synchronous actor methods. + + - `@objc` thunks of synchronous actor-isolated members of classes. + + Similarly to the previous case if a class or its individual synchronous members are actor-isolated and marked as either `@objc` or `@objcMembers`, the thunks, synthesized by the compiler to make them available from Objective-C, would have a new precondition check to make sure that use always happens on the right actor. + + - Synchronous actor-isolated function values passed to APIs that erase actor isolation and haven't yet adopted strict concurrency checking. + + When API comes from a module that doesn't have strict concurrency checking enabled it's possible that it could introduce actor isolation violations that would not be surfaced to a client. In such cases actor isolation erasure should be handled defensively by introducing a runtime check at each position for granular protection. + + ```swift + @MainActor + func updateUI(view: MyViewController) { + NotMyLibrary.track(view.renderToUIEvent) + } + ``` + + The use of `track` here would be considered unsafe if it accepts a synchronous nonisolated function type due to loss of `@MainActor` from `renderToUIEvent` and compiler would transform the call site into a function equivalent of: + + ```swift + @MainActor + func updateUI(view: MyViewController) { + NotMyLibrary.track({ + MainActor.assumeIsolated { + view.renderToUIEvent() + } + }) + } + ``` + + - Call-sites of synchronous actor-isolated functions imported from Swift 6 libraries. + + When importing a module that was compiled with the Swift 6 language mode into code that is not, it's possible to call actor-isolated functions from outside the actor using `@preconcurrency`. For example: + + ```swift + // ModuleA built with -swift-version 6 + @MainActor public func onMain() { ... } + + // ModuleB built with -swift-version 5 -strict-concurrency=minimal + import ModuleA + + @preconcurrency @MainActor func callOnMain() { + onMain() + } + + func notIsolated() { + callOnMain() + } + ``` + + In the above code, `onMain` from ModuleA can be called from outside the main actor via a call to `notIsolated()`. To close this safety hole, a dynamic check is inserted at the call-site of `onMain()` when ModuleB is recompiled against ModuleA after ModuleA has migrated to the Swift 6 language mode. + +These are the most common circumstances when losing actor isolation could be problematic and restricting runtime checking to them significantly limits negative performance impact of the new checks. The strategy of only emitting runtime checks when there’s potential for the function to be called from unchecked code is desirable, because it means the dynamic checks will be eliminated as more of the Swift ecosystem transitions to Swift 6. + + +## Detailed design + +### Runtime actor isolation checking + +For all of the situations described in the previous section the compiler will emit a runtime check to assert that the current executor matches the expected executor of the isolated actor. Calling an isolated synchronous function from outside the isolation domain will result in a runtime error that halts program execution. + +Runtime checking for actor isolation is not necessary for `async` functions, because switching to the callee's actor is always performed by the callee. `async` functions cannot be unsafely called from non-Swift code because they are not available directly in C/C++/Objective-C. + +### `@preconcurrency` conformances + +A `@preconcurrency` protocol conformance is scoped to the implementation of the protocol requirements in the conforming type. A `@preconcurrency` conformance can be written at the primary declaration or in an extension, and witness checker diagnostics about actor isolation will be suppressed. Like other `@preconcurrency` annotations, if no diagnotsics are suppressed, a warning will be emitted at the `@preconcurrency` annotation stating that the annotation has no effect and it should be removed. + +### Disabling dynamic actor isolation checking + +The dynamic actor isolation checks can be disabled using the flag `-disable-dynamic-actor-isolation`. Disabling dynamic actor isolation is discouraged, but it may be necessary if code that you don't control violates actor isolation in a way that causes the program to crash, such as by passing a non-`Sendable` function argument outside of a main actor context. `-disable-dynamic-actor-isolation` is similar to the `-enforce-exclusivity=unchecked` flag, which was a tool provided when staging in dynamic memory exclusivity enforcement under the Swift 5 language mode. + +## Source compatibility + +Dynamic actor isolation checking can introduce new runtime assertions for existing programs. Therefore, dynamic actor isolation is only performed for synchronous functions that are witnesses to an explicitly annotated `@preconcurrency` protocol conformance, or that are compiled under the Swift 6 language mode. + +## ABI compatibility + +This proposal has no impact on ABI compatibility of existing code. There are runtime implications for code that explicitly adopts this feature; see the following section. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. However, as noted in the Source compatibility section, adoption of this feature has runtime implications, because actor-isolated code called incorrectly from preconcurrency code will crash instead of race. + +## Alternatives considered + +### Always emit dynamic checks upon entry to synchronous isolated functions + +A previous iteration of this proposal specified that dynamic actor isolation checks are always emitted upon entry to a synchronous isolated function. This approach is foolproof; there's little possibility for missing a dynamic check for code that can be called from another module that does not have strict concurrency checking at compile time. However, the major downside of this approach is that code will be paying the price of runtime overhead for actor isolation checking even when actor isolation is fully enforced at compile time in Swift 6. + +The current approach in this proposal has a very desirable property of eliminated more runtime overhead as more of the Swift ecosystem transitions to Swift 6 at the cost of introducing the potential for missing dynamic checks where synchronous functions can be called from not-statically-checked code. We believe this is the right tradeoff for the long term arc of data race safety in Swift 6 and beyond, but it may require more special cases when we discover code patterns that are not covered by the specific set of rules in this proposal. + +### `@preconcurrency(unsafe)` to downgrade dynamic actor isolation violations to warnings + +If adoption of this feature exposes a bug in existing binaries because actor-isolated code is run outside the actor, a `@preconcurrency(unsafe)` annotation (or similar) could be provided to downgrade assertion failures to warnings. However, it's not clear whether allowing a known data race exhibited at runtime is the right approach to solving such a problem. + +## Revision history + +* Changes from the first review + * Insert dynamic checks at direct calls to synchronous actor-isolated functions imported from Swift 6 libraries. + * Add a flag to disable all dynamic actor isolation checking. + +## Acknowledgments + +Thank you to Doug Gregor for implementing the existing dynamic actor isolation checking gated behind `-enable-actor-data-race-checks`. diff --git a/proposals/0424-custom-isolation-checking-for-serialexecutor.md b/proposals/0424-custom-isolation-checking-for-serialexecutor.md new file mode 100644 index 0000000000..cc7a2d0d29 --- /dev/null +++ b/proposals/0424-custom-isolation-checking-for-serialexecutor.md @@ -0,0 +1,206 @@ +# Custom isolation checking for SerialExecutor + +* Proposal: [SE-0424](0424-custom-isolation-checking-for-serialexecutor.md) +* Author: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-custom-isolation-checking-for-serialexecutor/69786)) ([review](https://forums.swift.org/t/se-0424-custom-isolation-checking-for-serialexecutor/70195)) ([acceptance](https://forums.swift.org/t/accepted-se-0424-custom-isolation-checking-for-serialexecutor/70480)) + +## Introduction + +[SE-0392 (Custom Actor Executors)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md) added support for custom actor executors, but its support is incomplete. Safety checks like [`Actor.assumeIsolated`](https://developer.apple.com/documentation/swift/actor/assumeisolated(_:file:line:)) work correctly when code is running on the actor through a task, but they don't work when code is scheduled to run on the actor's executor through some other mechanism. For example, if an actor uses a serial `DispatchQueue` as its executor, a function dispatched _directly_ to the queue with DispatchQueue.async cannot use `assumeIsolated` to assert that the actor is currently isolated. This proposal fixes this by allowing custom actor executors to provide their own logic for these safety checks. + +## Motivation + +The Swift concurrency runtime dynamically tracks the current executor of a running task in thread-local storage. To run code on behalf of a task, an executor must call into the runtime, and the runtime will set up the tracking appropriately. APIs like `assertIsolated` and `assumeIsolated` are built on top of that functionality and perform their checks by comparing the expected executor with the current executor tracked by the runtime. If the current thread is not running a task, the runtime treats it as if it were running a non-isolated function, and the comparison will fail. + +This logic is not sufficient to handle the situation in which code is running on an actor's serial executor, but the code is not associated with a task. Swift's default actor executors currently do not provide any way to enqueue work on them that is not associated with a task, so this situation does not apply to them. However, many custom executors do provide other APIs for enqueuing work, such as the `async` method on `DispatchSerialQueue`. These APIs are not required to inform the Swift concurrency runtime before running the code. As a result, the runtime will be unaware that the current thread is associated with an actor's executor, and checks like `assumeIsolated` will fail. This is undesirable because, as long as the executor still acts like a serial executor for any non-task code it runs this way, the code will still be effectively actor-isolated: no code that accesses the actor's isolated state can run concurrently with it. + +The following example demonstrates such a situation: + +```swift +import Dispatch + +actor Caplin { + let queue: DispatchSerialQueue = .init(label: "CoolQueue") + + var num: Int // actor isolated state + + // use the queue as this actor's `SerialExecutor` + nonisolated var unownedExecutor: UnownedSerialExecutor { + queue.asUnownedSerialExecutor() + } + + nonisolated func connect() { + queue.async { + // guaranteed to execute on `queue` + // which is the same as self's serial executor + self.queue.assertIsolated() // CRASH: Incorrect actor executor assumption + self.assumeIsolated { caplin in // CRASH: Incorrect actor executor assumption + caplin.num += 1 + } + } + } +} +``` + +Even though the code is executing on the correct Dispatch**Serial**Queue, the assertions trigger and we're left unable to access the actor's state, even though isolation-wise it would be safe and correct to do so. + +Being able to assert isolation for non-task code this way is important enough that the Swift runtime actually already has a special case for it: even if the current thread is not running a task, isolation checking will succeed if the target actor is the `MainActor` and the current thread is the *main thread*. This problem is more general than the main actor, however; it exists for all kinds of threads which may be used as actor executors. The most important example of this is `DispatchSerialQueue`, especially because it is so commonly used in pre-concurrency code bases to provide actor-like isolation. Allowing types like `DispatchSerialQueue` to hook into isolation checking makes it much easier to gradually migrate code to actors: if an actor uses a queue as its executor, existing code that uses the queue don't have to be completely rewritten in order to access the actor's state. + +One way to think of this proposal is that gives all `SerialExecutor`s the power to provide a "fallback" check like this, rather than keeping it special-cased to `MainActor`. + +## Proposed solution + +We propose to add a new last-resort mechanism to executor comparison, which will be used by all the isolation-checking APIs in the concurrency library. + +This will be done by providing a new `checkIsolated()` protocol requirement on `SerialExecutor`: + +```swift +protocol SerialExecutor: Executor { + // ... + + /// Invoked as last resort when the Swift concurrency runtime is performing an isolation + /// assertion and could not confirm that the current execution context belongs to the + /// expected executor. + /// + /// This function MUST crash the program with a fatal error if it is unable + /// to prove that this thread can currently be safely treated as isolated + /// to this ``SerialExecutor``. That is, if a synchronous function calls + /// this method, and the method does not crash with a fatal error, + /// then the execution of the entire function must be well-ordered + /// with any other job enqueued on this executor, as if it were part of + /// a job itself. + /// + /// A default implementation is provided that unconditionally causes a fatal error. + func checkIsolated() +} + +extension SerialExecutor { + public func checkIsolated() { + fatalError("Incorrect actor executor assumption, expected: \(self)") + } +} +``` + +## Detailed design + +This proposal adds another customization point to the Swift concurrency runtime that hooks into isolation context comparison mechanisms used by `assertIsolated`, `preconditionIsolated`, and `assumeIsolated`, as well as any implicitly injected assertions used in `@preconcurrency` code. + +### Extended executor comparison mechanism + +With this proposal, the logic for checking if the current executor is the same as an expected executor changes, and can be expressed using the following pseudo-code: + +```swift +// !!!! PSEUDO-CODE !!!! Simplified for readability. + +let current = Task.current.executor + +guard let current else { + // no current executor, last effort check performed by the expected executor: + expected.checkIsolated() + + // e.g. MainActor: + // MainActorExecutor.checkIsolated() { + // guard Thread.isMain else { fatalError("Expected main thread!") + // return // ok! + // } +} + +if isSameSerialExecutor(current, expected) { + // comparison takes into account "complex equality" as introduced by 'SE-0392 + return // ok! +} else { + // executor comparisons failed... + + // give the expected executor a last chance to check isolation by itself: + expected.checkIsolated() + + // as the default implementation of checkIsolated is to unconditionally crash, + // this call usually will result in crashing -- as expected. +} + +return // ok, it seems the expected executor was able to prove isolation +``` + +This pseudo code snippet explains the flow of the executor comparisons. There are two situations in which the new `checkIsolated` method may be invoked: when there is no current executor present, or if all other comparisons have failed. +For more details on the executor comparison logic, you can refer to [SE-0392: Custom Actor Executors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md). + +Specific use-cases of this API include `DispatchSerialQueue`, which would be able to implement the requirement as follows: + +```swift +// Dispatch + +extension DispatchSerialQueue { + public func checkIsolated() { + dispatchPrecondition(condition: .onQueue(self)) // existing Dispatch API + } +} +``` + +An executor that wishes to take advantage of this proposal will need to have some mechanism to identity its active worker thread. If that's not possible or desired, the executor should leave the default implementation (that unconditionally crashes) in place. + +### Impact on async code and isolation assumptions + +The `assumeIsolated(_:file:line:)` APIs purposefully only accept a **synchronous** closure. This is correct, and it remains correct with these proposed additions. An isolation check on an executor ensures that any actor using the executor is synchronously isolated, and the closure provided to `assumeIsolated` will execute prior to any possible async suspension. This is what makes it safe to access actor-isolated state within the closure. + +This means that the following code snippet, while a bit unusual remains correct isolation-wise: + +```swift +actor Worker { + var number: Int + + nonisolated func canOnlyCallMeWhileIsolatedOnThisInstance() -> Int { + self.preconditionIsolated("This method must be called while isolated to \(self)") + + return self.assumeIsolated { // () throws -> Int + // suspensions are not allowed in this closure. + + self.number // we are guaranteed to be isolated on this actor; read is safe + } + } + +``` + +As such, there is no negative impact on the correctness of these APIs. + +Asynchronous functions should not use dynamic isolation checking. Isolation checking is useful in synchronous functions because they naturally inherit execution properties like their caller's isolation without disturbing it. A synchronous function may be formally non-isolated and yet actually run in an isolated context dynamically. This is not true for asynchronous functions, which switch to their formal isolation on entry without regard to their caller's isolation. If an asynchronous function is not formally isolated to an actor, its execution will never be dynamically in an isolated context, so there's no point in checking for it. + +## Future directions + +### Introduce `globalMainExecutor` global property and utilize `checkIsolated` on it + +This proposal also paves the way to clean up this hard-coded aspect of the runtime, and it would be possible to change these heurystics to instead invoke the `checkIsolated()` method on a "main actor executor" SerialExecutor reference if it were available. + +This proposal does not introduce a `globalMainActorExecutor`, however, similar how how [SE-0417: Task ExecutorPreference](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0417-task-executor-preference.md) introduced a: + +```swift +nonisolated(unsafe) +public var globalConcurrentExecutor: any TaskExecutor { get } +``` + +the same could be done to the MainActor's executor: + +```swift +nonisolated(unsafe) +public var globalMainExecutor: any SerialExecutor { get } +``` + +The custom heurystics that are today part of the Swift Concurrency runtime to detect the "main thread" and "main actor executor", could instead be delegated to this global property, and function correctly even if the MainActor's executor is NOT using the main thread (which can happen on some platforms): + +```swift +// concurrency runtime pseudo-code +if expectedExecutor.isMainActor() { + expectedExecutor.checkIsolated() +} +``` + +This would allow the isolation model to support different kinds of main executor and properly assert their isolation, using custom logic, rather than hardcoding the main thread assumptions into the Swift runtime. + +## Alternatives considered + +### Do not provide customization points, and just hardcode DispatchQueue handling + +Alternatively, we could hardcode detecting dispatch queues and triggering `dispatchPrecondition` from within the Swift runtime. + +This is not a good direction though, as our goal is to have the concurrency runtime be less attached to Dispatch and allow Swift to handle each and every execution environment equally well. As such, introducing necessary hooks as official and public API is the way to go here. diff --git a/proposals/0425-int128.md b/proposals/0425-int128.md new file mode 100644 index 0000000000..809464b3f8 --- /dev/null +++ b/proposals/0425-int128.md @@ -0,0 +1,193 @@ +# 128-bit Integer Types + +* Proposal: [SE-0425](0425-int128.md) +* Author: [Stephen Canon](https://github.com/stephentyrone) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.0)** +* Review: ([Pitch](https://forums.swift.org/t/pitch-128-bit-integer-types/70188)) ([Review](https://forums.swift.org/t/se-0425-128-bit-integer-types/70456)), ([Acceptance](https://forums.swift.org/t/accepted/71063)) + +## Motivation + +128b integers are the largest fixed-size type that is currently commonly +used in "general-purpose" code. They are much less common than 64b types, +but common enough that adding them to the standard library makes sense. +We use them internally in the standard library already (e.g. as an +implementation detail of Duration). + +## Proposed solution + +Introduce two new structs, `UInt128` and `Int128`, conforming to all of the +usual fixed-width integer protocols. + +## Detailed design + +The `[U]Int128` types are 16B aligned on 64b targets¹ and have the same +alignment as `[U]Int64` on 32b targets. They will match the endianness of +all other integer types. + +The clang importer will be updated to bridge `__uint128_t` to `UInt128` and +`__int128_t` to `Int128`. We will not bridge `_BitInt()` types until +the ABI problems with those types have been clearly resolved (see Alternatives +Considered for sordid history). + +The `[U]Int128` types conform to `AtomicRepresentable` on targets with +`_hasAtomicBitWidth(_128)` set (notably x86\_64, arm64, and arm64\_32). + +The actual API of the types is uninteresting; they are entirely constrained by +their protocol conformances. Notably, these types conform to the following +protocols, and hence to any protocol that they refine: + +- Hashable +- Equatable +- Comparable +- Codable +- Sendable +- LosslessStringConvertible +- ExpressibleByIntegerLiteral +- AdditiveArithmetic +- [Signed]Numeric +- BinaryInteger +- FixedWidthInteger +- [Unsigned|Signed]Integer + +------- +¹ For the purposes of this discussion, arm64\_32 and similar architectures +are "64b targets." + +### Codable details + +An earlier version of this proposal conformed `[U]Int128` to `Codable` +with a representation as a pair of `[U]Int64` values. During the first review +period several people made excellent points: + +- Making it possible for encoders to customize how they represent these types +is desirable. Some cannot represent all 64b values or might prefer to use a +string representation, others might prefer to treat 128b integers as native +values to encode. + +- If we make it customizable but provide a default behavior, some decoders +would have to support that default as well as their desired representation, +for compatibility with any encodings created between when we added support +and when they defined their preferred encoding. + +For this reason, the proposal has been updated to add new protocol requirements +for encoders and decoders to support `[U]Int128`, but with default +implementations that throw an EncodingError or DecodingError unconditionally, +allowing implementations to choose their preferred behavior when they add +support without worrying about compatibility with a defaulted implementation. + +Thus, the following requirements will be added: +```swift +protocol KeyedEncodingContainerProtocol { + mutating func encode(_ value: Int128, forKey key: Key) throws + mutating func encode(_ value: UInt128, forKey key: Key) throws + mutating func encodeIfPresent(_ value: Int128?, forKey key: Key) throws + mutating func encodeIfPresent(_ value: UInt128?, forKey key: Key) throws + // And matching changes to KeyedEncodingContainer +} + +protocol KeyedDecodingContainerProtocol { + func decode(_ type: Int128.Type, forKey key: Key) throws -> Int128 + func decode(_ type: UInt128.Type, forKey key: Key) throws -> UInt128 + func decodeIfPresent(_ type: Int128.Type, forKey key: Key) throws -> Int128? + func decodeIfPresent(_ type: UInt128.Type, forKey key: Key) throws -> UInt128? + // And matching changes to KeyedDecodingContainer +} + +protocol UnkeyedEncodingContainer { + mutating func encode(_ value: Int128) throws + mutating func encode(_ value: UInt128) throws + mutating func encode( + contentsOf sequence: T + ) throws where T.Element == Int128 + mutating func encode( + contentsOf sequence: T + ) throws where T.Element == UInt128 +} + +protocol UnkeyedDecodingContainer { + mutating func decode(_ type: Int128.Type) throws -> Int128 + mutating func decode(_ type: UInt128.Type) throws -> UInt128 + mutating func decodeIfPresent(_ type: Int128.Type) throws -> Int128? + mutating func decodeIfPresent(_ type: UInt128.Type) throws -> UInt128? +} + +protocol SingleValueEncodingContainer { + mutating func encode(_ value: Int128) throws + mutating func encode(_ value: UInt128) throws +} + +protocol SingleValueDecodingContainer { + func decode(_ type: Int128.Type) throws -> Int128 + func decode(_ type: UInt128.Type) throws -> UInt128 +} +``` +and given default implementations. The default encode implementations throw +`EncodingError.invalidValue`, and the default decode implementations throw +`DecodingError.typeMismatch`. + +## Source compatibility + +This proposal has no effect on source compatibility. + +## ABI compatibility + +This proposal has no effect on ABI compatibility. + +## Implications on adoption + +Adopting this feature will require a target with runtime support. + +## Future directions + +Implement clang importer support for `_BitInt(128)` on any platforms where +the finalized ABI is compatible with our layout. + +## Alternatives considered + +### Alignment and `_BitInt()` types +Clang and GCC have historically exposed the extension types `__uint128_t` and +`__int128_t` on 64b platforms only. These types basically behave like C +builtin integer types--their size and alignment are 16B. + +The C23 standard introduces `_BitInt(N)` as a means to spell arbitrary-width +integer types, but these still have some warts. In particular, `_BitInt(128)` +as implemented in clang has 8B alignment on x86\_64 and arm64. For arm64, +this is clearly a bug; the AAPCS specifies that it should have 16B alignment. +For x86\_64, the situation is less clear. The x86\_64 psABI document specifies +that it should have 8B alignment, but the authors of the proposal that added +the feature tell me that it _should_ be 16B aligned and that they are +attempting to change the psABI. + +We would like to be layout-compatible with `_BitInt(128)` on all platforms, +but given the currently-murky state of the layout of those types, it makes +the most sense to guarantee compatibility with the widely-used but non- +standard `__[u]int128_t` and find mechanisms to make `_BitInt(128)` work +once its ABI has been finalized on Swift's targeted platforms. + +### Generic-sized fixed width integers +Rather than adding `[U]Int128`, we could implement some form of generic- +sized fixed-width integer (like `\_BitInt()` in C). Given both the lack +of consensus around what integer generic parameters ought to look like in +Swift (or if they ought to exist at all), and the growing pains that +`\_BitInt()` is currently going through, such a design would be premature. +While other fixed-width integer types are interesting, 128 bits is a couple +orders of magnitude more useful than all the others for general-purpose +software at this point in time. + +### NSNumber bridging +`[U]Int128` will not bridge to `NSNumber`. In the future, Swift will need +a careful rethinking of how best to handle type-erased numbers, but we don't +want to pile on the debt by including ever more types in an existing system +that isn't supported on all platforms. In addition, the most common use for +such bridging, unpacking type-erased fields from encoded dictionaries, is +somewhat moot since most existing coders do not support 128b integers. We +will likely revisit this more holistically in the future. + +### Codable errors +It would be nice to introduce new `unsupportedType` error cases for the +default Codable conformances, but we cannot add new cases with associated +values and constrained availability to existing enums, which prevents +attaching context or a useful debug description. It's more useful for users +if we use existing error cases `invalidValue` and `typeMismatch` but put an +actionable message in that description field. diff --git a/proposals/0426-bitwise-copyable.md b/proposals/0426-bitwise-copyable.md new file mode 100644 index 0000000000..f057c4dd5e --- /dev/null +++ b/proposals/0426-bitwise-copyable.md @@ -0,0 +1,493 @@ +# BitwiseCopyable + +* Proposal: [SE-0426](0426-bitwise-copyable.md) +* Authors: [Kavon Farvardin](https://github.com/kavon), [Guillaume Lessard](https://github.com/glessard), [Nate Chandler](https://github.com/nate-chandler), [Tim Kientzle](https://github.com/tbkka) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Implementation: in main branch of compiler (https://github.com/apple/swift/pull/73235) +* Status: **Implemented (Swift 6.0)** +* Review: ([Pitch](https://forums.swift.org/t/pitch-bitwisecopyable-marker-protocol/69943)) ([First review](https://forums.swift.org/t/se-0426-bitwisecopyable/70479)) ([Returned for revision](https://forums.swift.org/t/returned-for-revision-se-0426-bitwisecopyable/70892)) ([Second review](https://forums.swift.org/t/se-0426-second-review-bitwisecopyable/71316)) ([Acceptance](https://forums.swift.org/t/accepted-se-0426-bitwisecopyable/71600)) + + + +## Introduction + +We propose a new, [limited](#limitations) protocol `BitwiseCopyable` that _can_ be conformed to by types that are "bitwise-copyable"[^1]--that is, that can be moved or copied with direct calls to `memcpy` and which require no special destroy operation. +When compiling generic code with such constraints, the compiler can emit these efficient operations directly, only requiring minimal overhead to look up the size of the value at runtime. +Alternatively, developers can use this constraint to selectively provide high-performance variations of specific operations, such as bulk copying of a container. + +[^1]: The term "trivial" is used in [SE-138](0138-unsaferawbufferpointer.md) and [SE-0370](0370-pointer-family-initialization-improvements.md) to refer to types with this property. The discussion below will explain why certain generic or exported types that are bitwise-copyable will not in fact be `BitwiseCopyable`. + +## Motivation + +Swift can compile generic code into an unspecialized form in which the compiled function receives a value and type information about that value. +Basic operations are implemented by the compiler as calls to a table of "value witness functions." + +This approach is flexible, but can represent significant overhead. +For example, using this approach to copy a buffer with a large number of `Int` values requires a function call for each value. + +Constraining the types in generic functions to `BitwiseCopyable` allows the compiler (and in some cases, the developer) to instead use highly efficient direct memory operations in such cases. + +The standard library already contains many examples of functions that could benefit from such a concept, and more are being proposed: + +The `UnsafeMutablePointer.initialize(to:count:)` function introduced in [SE-0370](0370-pointer-family-initialization-improvements.md) could use a bulk memory copy whenever it statically knew that its argument was `BitwiseCopyable`. + +The proposal for [`StorageView`](nnnn-safe-shared-contiguous-storage.md) includes the ability to copy items to or from potentially-unaligned storage, which requires that it be safe to use bulk memory operations: +```swift +public func loadUnaligned( + fromByteOffset: Int = 0, as: T.Type +) -> T + +public func loadUnaligned( + from index: Index, as: T.Type +) -> T +``` + +And this proposal includes the addition of three overloads of existing standard library functions. + +## Proposed solution + +We add a new protocol `BitwiseCopyable` to the standard library: +```swift +@_marker public protocol BitwiseCopyable {} +``` + +That a type conforms to the protocol [implies](#transient-and-permanent) that the type is bitwise-copyable; the reverse is _not_ true. + +Many basic types in the standard library will conformed to this protocol. + +Developer's own types may be conformed to the protocol, as well. +The compiler will check any such conformance and emit a diagnostic if the type contains elements that are not `BitwiseCopyable`. + +Furthermore, when building a module, the compiler will infer conformance to `BitwiseCopyable` for any non-exported struct or enum defined within the module whose stored members are all `BitwiseCopyable`, +except those for which conformance is explicitly [suppressed](#suppression). + +Developers cannot conform types defined in other modules to the protocol. + +## Detailed design + +Our design first conforms a number of core types to `BitwiseCopyable`, and then extends that to aggregate types. + +### Standard library changes + +Many types and a few key protocols are constrained to `BitwiseCopyable`. +A few highlights: + +* Integer types +* Floating point types +* SIMD types +* Pointer types +* `Unmanaged` +* `Optional` + +For an exhaustive list, see the [appendix](#all-stdlib-conformers). + +### Additional BitwiseCopyable types + +In addition to the standard library types marked above, the compiler will recognize several other types as `BitwiseCopyable`: + +* Tuples of `BitwiseCopyable` elements. + +* `unowned(unsafe)` references. + Such references can be copied without reference counting operations. + +* `@convention(c)` and `@convention(thin)` function types do not carry a reference-counted capture context, unlike other Swift function types, and are therefore `BitwiseCopyable`. + +### Explicit conformance to `BitwiseCopyable` + +Enum and struct types can be explicitly declared to conform to `BitwiseCopyable`. +When a type is declared to conform, the compiler will check that its elements are all `BitwiseCopyable` and emit an error otherwise. + +For example, the following struct can conform to `BitwiseCopayble` +```swift +public struct Coordinate : BitwiseCopyable { + var x: Int + var y: Int +} +``` +because `Int` is `BitwiseCopyable`. + +Similarly, the following enum can conform to `BitwiseCopyable` +```swift +public enum PositionUpdate : BitwiseCopyable { + case begin(Coordinate) + case move(x_change: Int, y_change: Int) + case end +} +``` +because both `Coordinate` and `(x_change: Int, y_change: Int)` are `BitwiseCopyable`. + +The same applies to generic types. For example, the following struct can conform to `BitwiseCopyable` +```swift +struct BittyBox : BitwiseCopyable { + var first: Value +} +``` +because its field `first` is a of type `Value` which is `BitwiseCopyable`. + +Generic types may be `BitwiseCopyable` only some of the time. +For example, +```swift +struct RegularBox { + var first: Value +} +``` +cannot conform unconditionally because `Value` needn't conform to `BitwiseCopyable`. +In this case, a conditional conformance may be written: + +```swift +extension Box : BitwiseCopyable where Value : BitwiseCopyable {} +``` + +### Automatic inference for aggregates + +As a convenience, unconditional conformances will be inferred for structs and enums[^2] much of the time. +When the module containing the type is built, if all of the type's fields are `BitwiseCopyable`, the compiler will generate a conformance for it to `BitwiseCopyable`. + +For generic types, a conformance will only be inferred if its fields unconditionally conform to `BitwiseCopyable`. +In the `RegularBox` example above, a conditional conformance will not be inferred. +If such a conformance is desired, the developer must explicitly write the conditional conformance. + +[^2]: This includes raw-value enums. While such enums do include a conformance to `RawRepresentable` where `RawValue` could be a non-conforming type (`String`), the instances of the enums themselves are `BitwiseCopyable`. + +### Inference for imported types + +The same inference will be done on imported C and C++ types. + +For an imported C or C++ enum, the compiler will always generate a conformance to to `BitwiseCopyable`. + +For an imported C struct, if all its fields are `BitwiseCopyable`, the compiler will generate a conformance to `BitwiseCopyable`. +The same is true for an imported C++ struct or class, unless the type is non-trivial[^3]. + +For an imported C or C++ struct, if any of its fields cannot be represented in Swift, the compiler will not generate a conformance. +This can be overridden, however, by annotating the type `__attribute__((__swift_attr__("BitwiseCopyable")))`. + +[^3]: A C++ type is considered non-trivial (for the purpose of calls, as defined by the Itanium ABI) if any of the following is non-default: its constructor; its copy-constructor; its destructor. + +### Inference for exported types + +This does not apply to exported (`public`, `package`, or `@usableFromInline`) types. +In the case of a library built with library evolution, while all the type's fields may be `BitwiseCopyable` at the moment, the compiler can't predict that they will always be. +If this is the developer's intent, they can explicitly conform the type. +To avoid having semantics that vary based on library evolution, the same applies to all exported (`public`, `package`, or `@usableFromInline`) types. + +For `@frozen` types, however, `BitwiseCopyable` conformance will be inferred. +That's allowed, even in the case of a library built with library evolution, because the compiler can see that the type's fields are all `BitwiseCopyable` and knows that they will remain that way. + +For example, the compiler will infer a conformance of the following struct +```swift +@frozen +public struct Coordinate3 { + public var x: Int + public var y: Int +} +``` +to `BitwiseCopyable`. + +### Suppressing inferred conformance + +To suppress the inference of `BitwiseCopyable`, `~BitwiseCopyable` can be added to the type's inheritance list. + +```swift +struct Coordinate4 : ~BitwiseCopyable {...} +``` + +Suppression must be declared on the type declaration itself, not on an extension. + +### Transient and permanent notions + +The Swift runtime already describes[^4] whether a type is bitwise-copyable. +It is surfaced, among other places, in the standard library function `_isPOD`[^5]. + +[^4]: The `IsNonPOD` value witness flag is set for every type that is _not_ bitwise-copyable. + +[^5]: "POD" here is an acronym for "plain old data" which is yet another name for the notion of bitwise-copyable or trivial. + +If a type conforms to `BitwiseCopyable`, then `_isPOD` must be true for the type. +The converse is not true, however. + +As a type evolves, it may [both gain _and_ lose bitwise-copyability](#fluctuating-bitwise-copyability). +A type may only _gain_ a conformance to `BitwiseCopyable`, however; +it cannot _lose_ its conformance without breaking source and ABI. + +The two notions are related, but distinct: +That a type `_isPOD` is a statement that the type is currently bitwise-copyable. +That a type conforms to `BitwiseCopyable` is a promise that the type is now and will remain bitwise-copyable as the library evolves. +In other words returning true from `_isPOD` is a transient property, and conformance to `BitwiseCopyable` is a permanent one. + +For this reason, conformance to `BitwiseCopyable` is not inherent. +Its declaration on a public type provides a guarantee that the compiler cannot infer. + +### Limitations of BitwiseCopyable + +Being declared with `@_marker`, `BitwiseCopyable` is a limited protocol. +Its limited nature allows the protocol's runtime behavior to be defined later, as needed. + +1. `BitwiseCopyable` cannot be extended. +This limitation is similar to that on `Sendable` and `Any`: +it prevents polluting the namespace of conforming types, especially types whose conformance is inferred. + +2. Because conformance to `BitwiseCopyable` is distinct from being bitwise-copyable, +the runtime cannot use the `IsNonPOD` bit as a proxy for conformance (although actual [conformance could be ignored](#casting-by-duck-typing)). +A separate mechanism would be necessary. +Until such a mechanism is added, `is`, `as?` and usage as a generic constraint to enable conditional conformance to another protocol is not possible. + +### Standard library API improvements + +The standard library includes a load method on both `UnsafeRawPointer` and `UnsafeMutableRawPointer` + +```swift +@inlinable +@_alwaysEmitIntoClient +public func loadUnaligned( + fromByteOffset offset: Int = 0, + as type: T.Type +) -> T +``` + +and a corresponding write method on `UnsafeMutableRawPointer` + +```swift +@inlinable +@_alwaysEmitIntoClient +public func storeBytes( + of value: T, toByteOffset offset: Int = 0, as type: T.Type +) +``` + +that must be called with a trivial `T`. + +We propose adding overloads of these methods to constrain the value to `BitwiseCopyable`: + +```swift +// on both UnsafeRawPointer and UnsafeMutableRawPointer +@inlinable +@_alwaysEmitIntoClient +public func loadUnaligned( + fromByteOffset offset: Int = 0, + as type: T.Type +) -> T + +// on UnsafeMutableRawPointer +@inlinable +@_alwaysEmitIntoClient +public func storeBytes( + of value: T, toByteOffset offset: Int = 0, as type: T.Type +) +``` + +This allows for optimal code generation because `memcpy` instead of value witnesses can be used. + +The existing methods that use a runtime assert instead of a type constraint will still be available (see [alternatives considered](#deprecation)). + +## Effect on ABI stability + +The addition of the `BitwiseCopyable` constraint to either a type or a protocol in a library will not cause an ABI break for users. + +## Source compatibility + +This addition of a new protocol will not impact existing source code that does not use it. + +Removing the `BitwiseCopyable` conformance from a type is source-breaking. +As a result, future versions of Swift may conform additional existing types to `BitwiseCopyable`, but will not remove it from any type already conforming to `BitwiseCopyable`. + +## Effect on API resilience + +Adding a `BitwiseCopyable` constraint on a generic type will not cause an ABI break. +As with any protocol, the additional constraint can cause a source break for users. + +## Future Directions + +### Automatic derivation of conditional conformances + +The wrapper type mentioned above +```swift +struct RegularBox { + var first: Value +} +``` +cannot conform to `BitwiseCopyable` unconditionally. +It can, however, so long as `Value` is `BitwiseCopyable`. + +With this proposal, such a conditional conformance can be added manually: + +```swift +extension Box : BitwiseCopyable where Value : BitwiseCopyable {} +``` + +In the future we may in some cases be able to derive it automatically. + +### Dynamic casting + +Being a [limited](#limitations) protocol, `BitwiseCopyable` does not currently have any runtime representation. +While a type's [transient](#transient-and-permanent) bitwise-copyability has a preexisting runtime representation, that is different from the type conforming to `BitwiseCopyable`. + +Being a low-level, performance-enabling feature, it is not clear that dynamic casting should be allowed at all. +If it were to be allowed at some point, a few different approaches can already be foreseen: + +#### Explicitly record a type's conformance + +The standard way to support dynamic casting would be to represent a type's conformance to the protocol and query the type at runtime. + +This approach has the virtue that dynamic casting behaves as usual. +A type could only be cast to `BitwiseCopyable` if it actually conformed to the protocol. +For example, casting a type which suppressed a conformance to `BitwiseCopyable` would fail. + +If this approach were taken, such casting could be back-deployed as far as the oldest OS in which this runtime representation was added. +Further back deployment would be possible by adding conformance records to back deployed binaries. + +#### Duck typing for BitwiseCopyable + +An alternative would be to dynamically treat any type that's bitwise-copyable as if it conformed to `BitwiseCopyable`. + +This is quite different from typical Swift casting behavior. +Rather than relying on a permanent characteristic of the type, it would rely on a [transient](#transient-and-permanent) one. +This would be visible to the programmer in several ways: +- different overloads would be selected for a value of concrete type from those selected for a value dynamically cast to `BitwiseCopyable` +- dynamic casts to `BitwiseCopyable` could fail, then succeed, then fail again in successive OS versions + +On the other hand, these behavioral differences may be desirable. + +Considering that this approach would just ignore the existence of conformances to `BitwiseCopyable`, +it would be reasonable to ignore the existence of a suppressed conformance as well. + +This approach also has the virtue of being completely back-deployable[^6]. +[^6]: All runtimes have had the `IsNonPOD` bit. + +### BitwiseMovable + +Most Swift types have the property that their representation can be relocated in memory with direct memory operations. +This could be represented with a `BitwiseMovable` protocol that would be handled similarly to `BitwiseCopyable`. + +### BitwiseCopyable as a composition + +Some discussion in the pitch thread discussed how `BitwiseCopyable` could be defined as the composition of several protocols. +For example, +```swift +typealias BitwiseCopyable = Bitwise & Copyable & DefaultDeinit +``` +Such a definition remains possible after this proposal. + +Because `BitwiseCopyable` is annotated `@_marker`, its ABI is rather limited. +Specifically, it only affects name mangling. +If, in a subsequent proposal, the protocol were redefined as a composition, symbols into which `BitwiseCopyable` was mangled could still be mangled in the same way, ensuring ABI compatibility. + +## Alternatives considered + +### Alternate Spellings + +**Trivial** is widely used within the compiler and Swift evolution discussions to refer to the property of bitwise copyability. `BitwiseCopyable`, on the other hand, is more self-documenting. + +### Deprecation of unconstrained functions dependent on `isPOD` + +The standard library has a few pre-existing functions that receive a generic bitwise-copyable value as a parameter. These functions work with types for which the `_isPOD()` function returns true, even though they do not have a `BitwiseCopyable` conformance. If we were to deprecate these unconstrained versions, we would add unresolvable warnings to some of the codebases that use them. For example, they might use types that could be conditionally `BitwiseCopyable`, but come from a module whose types have not been conformed to `BitwiseCopyable` by their author. Furthermore, as explained [above](#transient-and-permanent), it is not necessarily the case that a transiently bitwise-copyable type can be permanently annotated as `BitwiseCopyable`. + +At present, the unconstrained versions check that `_isPOD()` returns true in debug mode only. We may in the future consider changing them to check at all times, since in general their use in critical sections will have been updated to use the `BitwiseCopyable`-constrained overloads. + +## Acknowledgments + +This proposal has benefitted from discussions with John McCall, Joe Groff, Andrew Trick, Michael Gottesman, and Arnold Schwaigofer. + +## Appendix: Standard library conformers + +The following protocols in the standard library will gain the `BitwiseCopyable` constraint: + +- `_Pointer` +- `SIMDStorage`, `SIMDScalar`, `SIMD` + + +The following types in the standard library will gain the `BitwiseCopyable` constraint: + +- `Optional` when `T` is `BitwiseCopyable` +- The fixed-precision integer types: + - `Bool` + - `Int8`, `Int16`, `Int32`, `Int64`, `Int` + - `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt` + - `StaticBigInt` + - `UInt8.Words`, `UInt16.Words`, `UInt32.Words`, `UInt64.Words`, `UInt.Words` + - `Int8.Words`, `Int16.Words`, `Int32.Words`, `Int64.Words`, `Int.Words` +- The fixed-precision floating-point types: + - `Float`, `Double`, `Float16`, `Float80` + - `FloatingPointSign`, `FloatingPointClassification` +- The family of `SIMDx` types +- The family of unmanaged pointer types: + - `OpaquePointer` + - `UnsafeRawPointer`, `UnsafeMutableRawPointer` + - `UnsafePointer`, `UnsafeMutablePointer`, `AutoreleasingUnsafeMutablePointer` + - `UnsafeBufferPointer`, `UnsafeMutableBufferPointer` + - `UnsafeRawBufferPointer`, `UnsafeMutableRawBufferPointer` + - `Unmanaged` + - `CVaListPointer` +- Some types related to collections + - `EmptyCollection` + - `UnsafeBufferPointer.Iterator`, `UnsafeRawBufferPointer.Iterator`, `EmptyCollection.Iterator` + - `String.Index`, `CollectionDifference.Index` +- Some types related to unicode + - `Unicode.ASCII`, `Unicode.UTF8`, `Unicode.UTF16`, `Unicode.UTF32`, `Unicode.Scalar` + - `Unicode.ASCII.Parser`, `Unicode.UTF8.ForwardParser`, `Unicode.UTF8.ReverseParser`, `Unicode.UTF16.ForwardParser`, `Unicode.UTF16.ReverseParser`, `Unicode.UTF32.Parser` + - `Unicode.Scalar.UTF8View`, `Unicode.Scalar.UTF16View` + - `UnicodeDecodingResult` +- Some fieldless types + - `Never`, `SystemRandomNumberGenerator` +- `StaticString` +- `Hasher` +- `ObjectIdentifier` +- `Duration` +- Atomic changes + - `AtomicRepresentable.AtomicRepresentation` + - `AtomicOptionalRepresentable.AtomicOptionalRepresentation` + +## Appendix: Fluctuating bitwise-copyability + +Let's say the following type is defined in a framework built with library evolution. + +```swift +public struct Dish {...} +``` + +In the first version of the framework, the type only contains bitwise-copyable fields: + +```swift +/// NoodleKit v1.0 + +public struct Dish { + public let substrate: Noodle + public let isTopped: Bool +} +``` + +So in version `1.0`, the type is bitwise-copyable. + +In the next version of the framework, to expose more information to its clients, the stored `Bool` is replaced with a stored `Array`: + +```swift +/// NoodleKit v1.1 + +public struct Dish { + public let substrate: Noodle + public let toppings: [Topping] + public var isTopped: Bool { toppings.count > 0 } +} +``` + +As a result, in version `1.1`, the type is _not_ bitwise-copyable. + +In a subsequent version, as an optimization, the stored `Array` is replaced with an `OptionSet` + +```swift +/// NoodleKit v2.0 + +public struct Dish { + public let substrate: Noodle + private let toppingOptions: Topping + public let toppings: [Topping] { ... } + public var isTopped: Bool { toppings.count > 0 } +} +``` + +In release `2.0` the type is once again bitwise-copyable. diff --git a/proposals/0427-noncopyable-generics.md b/proposals/0427-noncopyable-generics.md new file mode 100644 index 0000000000..321c9090e8 --- /dev/null +++ b/proposals/0427-noncopyable-generics.md @@ -0,0 +1,700 @@ +# Noncopyable Generics + +* Proposal: [SE-0427](0427-noncopyable-generics.md) +* Authors: [Kavon Farvardin](https://github.com/kavon), [Tim Kientzle](https://github.com/tbkka), [Slava Pestov](https://github.com/slavapestov) +* Review Manager: [Holly Borla](https://github.com/hborla), [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.0)** +* Implementation: On `main` gated behind `-enable-experimental-feature NoncopyableGenerics` +* Previous Proposal: [SE-0390: Noncopyable structs and enums](0390-noncopyable-structs-and-enums.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-noncopyable-generics/68180)) ([first review](https://forums.swift.org/t/se-0427-noncopyable-generics/70525)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0427-noncopyable-generics/72039)) ([second review](https://forums.swift.org/t/second-review-se-0427-noncopyable-generics/72881)) ([acceptance](https://forums.swift.org/t/accepted-se-0427-noncopyable-generics/73560)) + + +**Table of Contents** + +- [Noncopyable Generics](#noncopyable-generics) + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed Solution](#proposed-solution) + - [The `Copyable` protocol](#the-copyable-protocol) + - [Default conformance to `Copyable`](#default-conformance-to-copyable) + - [Suppression of `Copyable`](#suppression-of-copyable) + - [Detailed Design](#detailed-design) + - [The `Copyable` protocol](#the-copyable-protocol-1) + - [Default conformances and suppression](#default-conformances-and-suppression) + - [Struct, enum and class extensions](#struct-enum-and-class-extensions) + - [Protocol extensions](#protocol-extensions) + - [Protocol inheritance](#protocol-inheritance) + - [Conformance to `Copyable`](#conformance-to-copyable) + - [Classes](#classes) + - [Existential types](#existential-types) + - [Source Compatibility](#source-compatibility) + - [ABI Compatibility](#abi-compatibility) + - [Alternatives Considered](#alternatives-considered) + - [Alternative spellings](#alternative-spellings) + - [Associated types without defaulting behavior](#associated-types-without-defaulting-behavior) + - [Inferred conditional copyability](#inferred-conditional-copyability) + - [Extension defaults](#extension-defaults) + - [Recursive `Copyable`](#recursive-copyable) + - [`~Copyable` as logical negation](#copyable-as-logical-negation) + - [Future Directions](#future-directions) + - [Suppressed associated types](#suppressed-associated-types) + - [Standard library adoption](#standard-library-adoption) + - [Tuples and parameter packs](#tuples-and-parameter-packs) + - [`~Escapable`](#escapable) + - [Acknowledgments](#acknowledgments) + + + +## Introduction + +The noncopyable types introduced in +[SE-0390: Noncopyable structs and enums](0390-noncopyable-structs-and-enums.md) +cannot be used with generics, protocols, or existentials, +leaving an expressivity gap in the language. This proposal extends Swift's +type system to fill this gap. + +## Motivation + +Noncopyable structs and enums are intended to express value types for which +it is not meaningful to have multiple copies of the same value. + +Support for noncopyable generic types was omitted from SE-0390. For example, +`Optional` could not be instantiated with a noncopyable type, +which prevented declaration of a failable initializer: +```swift +struct FileDescriptor: ~Copyable { + init?(filename: String) { // error: cannot form a Optional + ... + } +} +``` + +Practical use of generics also requires conformance to protocols, however +noncopyable types could not conform to protocols. + +In order to broaden the utility of noncopyable types in the language, we need +a consistent and sound way to relax the fundamental assumption of copyability +that permeates Swift's generics system. + +## Proposed Solution + +We begin by recalling the restrictions from SE-0390: + +1. A noncopyable type could not appear in the generic argument of some other generic type. +2. A noncopyable type could not conform to protocols. +3. A noncopyable type could not be boxed as an existential. + +This proposal builds on the `~Copyable` notation introduced in SE-0390, and +introduces three fundamental concepts that together eliminate these +restrictions: + +1. A new `Copyable` protocol abstracts over types whose values can be copied. +2. Every struct, enum, class, generic parameter, protocol and associated type +now conforms to `Copyable` _by default_. +3. The `~Copyable` notation is used to _suppress_ this default conformance +requirement anywhere it would otherwise be inferred. + +**Note**: The adoption of noncopyable generics in the standard library will be +covered in a subsequent proposal. + +### The `Copyable` protocol + +The notion of copyability of a value is now expressed as a special kind of +protocol. The existing `~Copyable` notation is re-interpreted as _suppressing_ +a conformance to this protocol, as we detail below. This protocol has no +explicit requirements, and it has some special behaviors. For example, +metatypes and tuples cannot normally conform to other protocols, +but they do conform to `Copyable`. + +A key goal of the design is _progressive disclosure_. The idea of _default_ conformance to +`Copyable` means that a user never interacts with noncopyable generics unless +they choose to do so, using the `~Copyable` notation to _suppress_ +the default conformance. + +The meaning of existing code remains the same; all generic parameters and +protocols now require conformance to `Copyable`, but all existing concrete +types do in fact conform. + +### Default conformance to `Copyable` + +Every struct and enum now has a default conformance to `Copyable`, unless the +conformance is suppressed by writing `~Copyable` in the inheritance clause. In +this proposal, we will show these inferred requirements in comments. For example, +a definition of a copyable struct is understood as if the user wrote the +conformance to `Copyable`: +```swift +struct Polygon /* : Copyable */ {...} +``` + +Furthermore, generic parameters now conform to `Copyable` by +default, so the following generic function can only be called with `Copyable` types: +```swift +func identity(x: T) /* where T: Copyable */ { return x } +``` + +Finally, protocols also have a default conformance to `Copyable`, thus +only `Copyable` types can conform to `Shape` below: +```swift +protocol Shape /*: Copyable */ {} +``` + +### Suppression of `Copyable` + +So far, we haven't described anything new, just formalized existing behavior with +a protocol. Now, we allow writing `~Copyable` in some new positions. + +For example, to generalize our identity function to also allow noncopyable types, we +suppress the default `Copyable` conformance on `T` as follows: +```swift +func identity(x: consuming T) { return x } +``` +This function imposes _no_ requirements on the generic parameter `T`. All possible +types, both `Copyable` and noncopyable, can be substituted for `T`. +This is the reason why we refer to `~Copyable` as _suppressing_ the conformance +rather than _inverting_ or _negating_ it. + +As with a concrete noncopyable type, a noncopyable generic parameter type must +be prefixed with one of the ownership modifiers `borrowing`, +`consuming`, or `inout`, when it appears as the type of a function's parameter. +For details on these parameter ownership modifiers, +see [SE-377](0377-parameter-ownership-modifiers.md). + +A protocol can allow noncopyable conforming types by suppressing its inherited +conformance to `Copyable`: +```swift +protocol Resource: ~Copyable { + consuming func dispose() +} + +extension FileDescriptor: Resource {...} +``` +A `Copyable` type can still conform to a `~Copyable` protocol. + +What it means to write `~Copyable` in each position will be fully explained in +the **Detailed Design** section. + +## Detailed Design + +This proposal does not fundamentally change the abstract theory of Swift +generics, with its four fundamental kinds of requirements that can appear in a +`where` clause; namely conformance, superclass, `AnyObject`, and same-type +requirements. + +The proposed mechanism of default conformance to `Copyable`, and its suppression by +writing `~Copyable`, is essentially a new form of syntax sugar; the transformation +is purely syntactic and local. + +### The `Copyable` protocol + +While `Copyable` is a protocol in the current implementation, it is unlike a +protocol in some ways. In particular, protocol extensions of `Copyable` are not +allowed: +```swift +extension Copyable { // error + func f() {} +} +``` +Such a protocol extension would effectively add new members to _every_ +copyable type, which would complicate overload resolution and possibly lead to +user confusion. + +### Default conformances and suppression + +Default conformance to `Copyable` is inferred in each position below, +unless explicitly suppressed: + +1. A struct, enum or class declaration. +2. A generic parameter declaration. +3. A protocol declaration. +4. An associated type declaration; does not support suppression (see Future Directions). +5. The `Self` type of a protocol extension. +6. The generic parameters of a concrete extension. + +The `~Copyable` notation is also permitted to appear as the _member_ of +a protocol composition type. This ensures that the following three declarations +have the same meaning, as one might expect: +```swift +func f(_: T) {} +func f(_: T) where T: Resource & ~Copyable {} +func f(_: T) where T: Resource, T: ~Copyable {} +``` + +A conformance to `Copyable` cannot be suppressed if it must hold for +some _other_ reason. In the above declaration of `f()`, we can suppress +`Copyable` on `T` because `Resource` suppresses its own `Copyable` requirement +on `Self`: +```swift +protocol Resource: ~Copyable {...} +``` +Thus, nothing else forces `f()`'s generic parameter `T` to be `Copyable`. On the +other hand, let's look at a copyable protocol like `Shape` below: +```swift +protocol Shape /*: Copyable */ {...} +``` +If we try to suppress the `Copyable` conformance on a generic parameter that also +conforms to `Shape`, we get an error: +```swift +func f(_: T) {...} // error +``` +The reason being that the conformance `T: Copyable` is _implied_ by `T: Shape`, and +cannot be suppressed. + +Furthermore, a `Copyable` conformance can only be suppressed if the subject type +is a generic parameter declared in the innermost scope. That is, the following +is an error: +```swift +struct S { + func f(_: T, _: U) where T: ~Copyable // error! +} +``` +The rationale here is that since `S` must be instantiated with a copyable type, +it does not make sense for a method of `S` to operate on an `S` where `T` +might be noncopyable. For a similar reason the same rule applies to nested +generic types. + +### Struct, enum and class extensions + +We wish to allow existing types to adopt noncopyability without changing the +meaning of existing code. Thus, an extension of a concrete type must introduce +a default `T: Copyable` requirement on every generic parameter of the +extended type: +```swift +struct Pair: ~Copyable {...} + +extension Pair /* where T: Copyable */ {...} +``` +The conformance can be suppressed to get an unconstrained extension of `Pair`: +```swift +extension Pair where T: ~Copyable {...} +``` + +An extension presents a copyable view of the world by default, behaving as if +`Pair` were declared like so: +```swift +struct Pair /* : Copyable */ {...} +``` + +An extension of a nested type introduces default conformance requirements for +all outer generic parameters of the extended type, and each conformance +can be individually suppressed: +```swift +struct Outer { + struct Inner {} +} + +extension Outer.Inner /* where T: Copyable, U: Copyable */ {} +extension Outer.Inner where T: ~Copyable /* , U: Copyable */ {} +extension Outer.Inner where /* T: Copyable, */ U: ~Copyable {} +``` + +An extension of a type whose generic parameters must be copyable cannot +suppress conformances: +```swift +struct Horse {...} +extension Horse where Hay: ~Copyable {...} // error +``` + +### Protocol extensions + +Where possible, we wish to allow the user to change an existing protocol to +accommodate noncopyable conforming types, without changing the meaning of existing +code. + +For this reason, an extension of a `~Copyable` protocol also introduces a default +`Self: Copyable` requirement, because this is the behavior expected from +existing clients: +```swift +protocol EventLog: ~Copyable { + ... +} + +extension EventLog /* where Self: Copyable */ { + func duplicate() -> Self { + return copy self // OK + } +} +``` + +To write an unconstrained protocol extension, suppress the conformance on `Self`: +```swift +extension EventLog where Self: ~Copyable { + ... +} +``` +Associated types cannot have their `Copyable` requirement suppressed +(see Future Directions). + +### Protocol inheritance + +Another consequence that immediately follows from the rules as explained so far +is that protocol inheritance must re-state `~Copyable` if needed: +```swift +protocol Token: ~Copyable {} +protocol ArcadeToken: Token /* , Copyable */ {} +protocol CasinoToken: Token, ~Copyable {} +``` +Again, because `~Copyable` suppresses a default conformance instead of introducing +a new kind of requirement, it is not propagated through protocol inheritance. + +### Conformance to `Copyable` + +Structs and enums conform to `Copyable` unconditionally by default, but a +conditional conformance can also be defined. For example, take this +noncopyable generic type: +```swift +enum List: ~Copyable { + case empty + indirect case element(T, List) +} +``` +We would like `List` to be `Copyable` since `Int` is, while still being +able to use a noncopyable element type, like `List`. We do +this by declaring a _conditional conformance_: +```swift +extension List: Copyable where T: Copyable {} +``` +Note that the `where` clause needs to be written, because a conformance to +`Copyable` declared in an extension does _not_ automatically add any other +requirements, unlike other extensions. + +A conditional `Copyable` conformance is not permitted if the +struct or enum declares a `deinit`. Deterministic destruction requires the +type to be unconditionally noncopyable. + +A conformance to `Copyable` is checked by verifying that every stored property +(of a struct) or associated value (of an enum) itself conforms to `Copyable`. +For a conditional `Copyable` conformance, the conditional requirements must be +sufficient to ensure this is the case. For example, the following is rejected, +because the struct cannot unconditionally conform to `Copyable`, having a +stored property of the noncopyable type `T`: +```swift +struct Holder /* : Copyable */ { + var value: T // error +} +``` + +There are two situations when it is permissible for a copyable type to +have a noncopyable generic parameter. The first is when the generic parameter +is not stored inside the type itself: +```swift +struct Factory /* : Copyable */ { + let fn: () -> T // ok +} +``` +The above is permitted, because a _function_ of type `() -> T` is still copyable, +even if a _value_ of type `T` is not copyable. + +The second case is when the type is a class. The contents of a class is never +copied, so noncopyable types can appear in the stored properties of a class: +```swift +class Box { + let value: T // ok + + init(value: consuming T) { self.value = value } +} +``` + +For a conditional `Copyable` conformance, the conditional requirements must be +of the form `T: Copyable` where `T` is a generic parameter of the type. It is +not permitted to make `Copyable` conditional on any other kind of requirement: +```swift +extension Pair: Copyable where T == Array {} // error +``` + +Conditional `Copyable` conformance must be declared in the same source +file as the struct or enum itself. Unlike conformance to other protocols, +copyability is a deep, inherent property of the type itself. + +### Classes + +This proposal supports classes with noncopyable generic parameters, +but it does not permit classes to themselves be `~Copyable`. +Similarly, an `AnyObject` or superclass requirement cannot be combined with +`~Copyable`: +```swift +func f(_ t: T) where T: AnyObject, T: ~Copyable { ... } // error +``` + +### Existential types + +The type `Any` is no longer the supertype of all types in the type system's +implicit conversion rules. + +The constraint type of an existential type is now understood as being a +protocol composition, with a default `Copyable` _member_. So +the empty protocol composition type `Any` is really `any Copyable`, and the +supertype of all types is now `any ~Copyable`: + +``` + any ~Copyable + / \ + / \ + Any == any Copyable + | + +``` + +This default conformance is suppressed by writing `~Copyable` as a member of a +protocol composition: + +```swift +protocol Pizza: ~Copyable {} +struct UniquePizza: Pizza, ~Copyable {} + +let t: any Pizza /* & Copyable */ = UniquePizza() // error +let _: any Pizza & ~Copyable = UniquePizza() // ok +``` + +## Source Compatibility + +The default conformance to `Copyable` is inferred anywhere it is not explicitly +suppressed with `~Copyable`, so this proposal does not change the interpretation +of existing code. + +Similarly, the re-interpretation of the SE-0390 restrictions in terms of +conformance to `Copyable` preserves the meaning of existing code that makes use of +noncopyable structs and enums. + +## ABI Compatibility + +This proposal does not change the ABI of existing code. + +Adding `~Copyable` to +an existing generic parameter is generally an ABI-breaking change, even when +source-compatible. + +Targeted mechanisms are being developed to preserve ABI compatibility when +adopting `~Copyable` on previously-shipped generic code. This will enable adoption +of this feature by standard library types such as `Optional`. Such mechanisms will +require extreme care to use correctly. + +## Alternatives Considered + +### Alternative spellings + +The spelling of `~Copyable` generalizes the existing syntax introduced in +SE-0390, and changing it is out of scope for this proposal. + +### Associated types without defaulting behavior + +A simple design for suppressed associated types was considered, where the +default conformance in a protocol extension applies only to `Self`, and not +the associated types of `Self`. For example, we first declare a protocol with a +`~Copyable` associated type: +```swift +protocol Manager { + associatedtype Resource: ~Copyable +} +``` +Now, a protocol extension of `Manager` does _not_ carry an implicit +`Self.Resource: Copyable` requirement: +```swift +extension Manager { + func f(resource: Resource) { + // `resource' cannot be copied here! + } +} +``` +For this reason, while adding `~Copyable` to the inheritance clause of a protocol +is a source-compatible change, the same with an _associated type_ is not +source compatible. The designer of a new protocol must decide which associated +types are `~Copyable` up-front. + +Requirements on associated types can be written in the associated type's +inheritance clause, or in a `where` clause, or on the protocol itself. As +with ordinary requirements, all three of the following forms define the same +protocol: +```swift +protocol P { associatedtype A: ~Copyable } +protocol P { associatedtype A where A: ~Copyable } +protocol P where A: ~Copyable { associatedtype A } +``` + +If a base protocol declares an associated type with a suppressed conformance +to `Copyable`, and a derived protocol re-states the associated type, a +default conformance is introduced in the derived protocol, unless it is again +suppressed: +```swift +protocol Base { + associatedtype A: ~Copyable + func f() -> A +} + +protocol Derived: Base { + associatedtype A /* : Copyable */ + func g() -> A +} +``` + +Finally, conformance to `Copyable` cannot be conditional on the copyability of +an associated type: +```swift +struct ManagerManager: ~Copyable {} +extension ManagerManager: Copyable where T.Resource: Copyable {} // error +``` + +This design for associated types was initially implemented but ultimately +removed from this proposal, because of the source compatibility issues. A more +comprehensive design that allows for some way of preserving source compatibility +requires a separate proposal due to the open design issues. + +### Inferred conditional copyability + +A struct or enum can opt out of copyability with `~Copyable`, and then possibly +declare a conditional conformance. It would be possible to automatically infer +this conditional conformance. For example, in the below, +```swift +struct MaybeCopyable { + var t: T +} +``` +The only way this _could_ be valid is if we had inferred the conditional +conformance: +``` +extension MaybeCopyable: Copyable /* where T: Copyable */ {} +``` +Feedback from early attempts at implementing this form of inference suggested +it was more confusing than helpful, so it was removed. + +### Extension defaults + +One possible downside is that extensions of types with noncopyable generic +parameters must suppress the conformance on each generic parameter. + +It would be possible to allow library authors to explicitly control this +behavior, with a new syntax allowing the default `where` clause of an +extension to be written inside of a type declaration. For example, +```swift +public enum Either { + case a(T) + case b(U) + + // Hypothetical syntax: + default extension where T: Copyable, U: ~Copyable +} + +// `T` is copyable, but `U` is not, because of the defaults above: +extension Either /* where T: Copyable */ { ... } + +``` + +This becomes much more complex for protocols that impose conformance +requirements on their own associated types: +```swift +protocol P: ~Copyable { + associatedtype A: P, ~Copyable + + // Hypothetical syntax: + default extension where A: Copyable +} + +extension P { + // A is Copyable. What about A.A? A.A.A? ... +} +``` + +Besides the unclear semantics with associated types, it was also felt this +approach could lead to user confusion about the meaning of a particular +extension. As a result, we feel that explicitly suppressing `Copyable` on +every extension is the best approach. + +### Recursive `Copyable` + +The behavior of default `Copyable` conformance on associated types prevents +existing protocols from adopting `~Copyable` on their associated types in a +source compatible way. + +For example, suppose we attempt to change `IteratorProtocol` to accommodate +noncopyable element types: +```swift +protocol IteratorProtocol: ~Copyable { + associatedtype Element: ~Copyable + mutating func next() -> Element? +} +``` +An existing program might declare a generic function that assumes `T.Element` is +`Copyable`: +```swift +func f(iter: inout T) { + let value = iter.next()! + let copy = value // error +} +``` +Since `IteratorProtocol` suppresses its `Copyable` conformance, the generic +parameter `T` defaults to `Copyable`. However, `T.Element` is no longer +`Copyable`, thus the above code would not compile. + +One can imagine a design where instead of a single default conformance +requirement `T: Copyable` being introduced above, we also add a requirement +`T.Element: Copyable`. This would preserve source compatibility and our +function `f()` would continue to work as before. + +However, this approach introduces major complications, if we again consider +protocols that impose conformance requirements on their associated types. + +Consider this simple protocol and function that uses it: +```swift +protocol P: ~Copyable { + associatedtype A: P, ~Copyable +} + +func f(_: T) {} +``` +Our hypothetical design would actually introduce an infinite sequence of +requirements here unless suppressed: +```swift +func f(_: T) /* where T: Copyable, T.A: Copyable, T.A.A: Copyable, ... */ {} +``` +Of course, it seems natural to represent this infinite sequence of requirements +as a new kind of "recursive conformance" requirement instead. + +Swift generics are based on the mathematical theory of +_string rewriting_, and requirements and associated types define certain _rewrite +rules_ which operate on a set of terms. In this formalism, a +hypothetical "recursive conformance" requirement corresponds to a rewrite +rule that can match an infinite set of terms given by a _regular expression_. +We would then need to generalize the algorithms for deciding term equivalence to +handle regular expressions. While there has been research in this area, +the design for such a system is far beyond the scope of this proposal. + +### `~Copyable` as logical negation + +Instead of the syntactic desugaring presented in this proposal, one can attempt to +formalize `T: ~Copyable` as the _logical negation_ +of a conformance, extending the theory of Swift generics with a fifth requirement kind to +represent this negation. It is not apparent how this leads to a sound and +usable model and we have not explored this further. + +## Future Directions + +### Suppressed associated types + +Supporting the full generality of associated types with suppressed Copyable +requirements, while providing a mechanism to preserve source compatibility is +a highly desirable goal. At the same time, it is a large, open design problem. +A few ideas were considered (see Alternatives Considered) but it was ultimately +determined to be too complex to tackle in this proposal. + +### Standard library adoption + +The `Optional` and `UnsafePointer` family of types can support noncopyable types +in a straightforward way. In the future, we will also explore noncopyable +collections, and so on. All of this requires significant design work and is out +of scope for this proposal. + +### Tuples and parameter packs + +Noncopyable tuples and parameter packs are a straightforward generalization +which will be discussed in a separate proposal. + +### `~Escapable` + +The ability to "escape" the current context is another implicit capability +of all current Swift types. +Suppressing this requirement provides an alternative way to control object lifetimes. +A companion proposal will provide details. + +## Acknowledgments + +Thank you to Joe Groff and Ben Cohen for their feedback throughout the +development of this proposal. diff --git a/proposals/0428-resolve-distributed-actor-protocols.md b/proposals/0428-resolve-distributed-actor-protocols.md new file mode 100644 index 0000000000..e7b4128e3d --- /dev/null +++ b/proposals/0428-resolve-distributed-actor-protocols.md @@ -0,0 +1,472 @@ +# Resolve DistributedActor protocols + +* Proposal: [SE-0428](0428-resolve-distributed-actor-protocols.md) +* Author: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-resolve-distributedactor-protocols-for-server-client-apps/69933)) ([review](https://forums.swift.org/t/se-0428-resolve-distributedactor-protocols/70669)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0428-resolve-distributedactor-protocols/71366)) + +## Introduction + +Swift's distributed actors offer developers a flexible bring-your-own-runtime approach to building distributed systems using the actor paradigm. The initial design of the feature aimed for systems where all nodes of a distributed actor system (such as nodes in a [cluster](https://github.com/apple/swift-distributed-actors)) share the same binary, and therefore all have access to the concrete `distributed actor` declarations which may be resolved and made remote calls on. + +Although this works well for peer-to-peer systems, distributed actors are also useful for systems where a client/server split is necessary. Examples of such use-cases include isolating some failure prone logic into another process, or split between a client without the necessary libraries or knowledge how to implement an API and delegating this work to a backend service. + +## Motivation + +Distributed actors allow to abstract over the location (i.e. in process or not) of an actor -- often referred to as "location transparency". Currently, Swift's distributed actors have a practical limitation in how well this property can be applied to server/client split applications because the only way to obtain a *remote reference* on which a *remote call* can be made, is by invoking the resolve method on a concrete distributed actor type, like this: + +```swift +import Distributed +import DistributedCluster // github.com/apple/swift-distributed-actors + +protocol Greeter: DistributedActor { + distributed func greet(name: String) -> String +} + +distributed actor EnglishGreeter: Greeter { + typealias ActorSystem = ClusterSystem + + func greet(name: String) -> String { + "Hello, \(name)!" + } +} + +let system: ClusterSystem = ... + +let knownID: EnglishGreeter.ID = /* obtained ID using discovery mechanisms, see ClusterSystem */ +let remote: EnglishGreeter = try EnglishGreeter.resolve(id: knownID, using: system) + +// remote call (implicitly async and throwing, due to e.g. network errors) +let greeting = try await remote.greet(name: "Caplin") +assert(greeting == "Hello, Caplin!") +``` + +This is common and acceptable in a **peer-to-peer system**, where all nodes of a cluster share the same types -- or at least a "client side" of a connection can only discover types it knows about, e.g. during version upgrade rollouts. However, this pattern is problematic in a **client/server deployment**, where the two applications do not share the concrete implementation of the `Greeter` type. It is also worth calling out that typical inter-process communications (IPC) use-cases often fall into the category of a client/server setup, where e.g. a daemon process serves as a "server" and an application "client" calls into it. + +The goal of this proposal is to allow the following module approach to sharing distributed actor APIs: +- **API** module: allow sharing a `DistributedActor` constrained protocol, which only declares the API surface the server is going to expose +- **Server** module: which depends on API module, and implements the API description using a concrete distributed actor +- **Client** module: which depends on API module, and `$Greeter` type (synthesized by the `@Resolvable` macro) to resolve a remote actor reference it can then invoke distributed methods on + +```swift + ┌────────────────────────────────────────┐ + │ API Module │ + │========================================│ + │ @Resolvable │ + │ protocol Greeter: DistributedActor { │ + ┌───────┤ distributed func greet(name: String) ├───────┐ + │ │ } │ │ + │ └────────────────────────────────────────┘ │ + │ │ + ▼ ▼ +┌────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────┐ +│ Client Module │ │ Server Module │ +│================================================│ │==============================================│ +│ let g = try $Greeter.resolve(...) /*new*/ │ │ distributed actor EnglishGreeter: Greeter { │ +│ try await greeter.hello(name: ...) │ │ distributed func greet(name: String) { │ +└────────────────────────────────────────────────┘ │ "Greeting in english, for \(name)!" │ +/* Client cannot know about EnglishGreeter type */ │ } │ + │ } │ + └──────────────────────────────────────────────┘ +``` + +In this scenario the client module has no knowledge of the concrete distributed actor type or implementation. + +In order to achieve this, this proposal improves upon _three_ aspects of distributed actors: +- introduce the `@Resolvable` macro that can be attached to distributed actor protocols, and enable the use of `resolve(id:using:)` with such types, +- allow distributed actors to be generic over their `ActorSystem` +- extend the distributed metadata section such that a distributed method which is witness to a distributed requirement, is also recorded using the protocol method's `RemoteCallTarget.identifier` and not only the concrete method's + +## Proposed solution + +### The `@Resolvable` macro + +At the core of this proposal is the `@Resolvable` macro. It is an attached declaration macro, which introduces a number of declarations which allow the protocol, or rather, a "stub" type for the protocol, to be used on a client without knowledge about the server implementation's concrete distributed actor type. + +The macro must be attached to the a `protocol` declaration that is a `DistributedActor` constrained protocol, like this: + +```swift +import Distributed + +@Resolvable +protocol Greeter: DistributedActor where ActorSystem: DistributedActorSystem { + distributed func greet(name: String) -> String +} +``` + +The protocol must specify a constraint on the `ActorSystem` which specifies the kind of `SerializationRequirement` it is able to work with. This serialization requirement must be a protocol, and existing distributed actor functionality already will be verifying this. + +Checking of distributed functions works as before, and the compiler will check that the `distributed` declarations all fulfill the `SerializationRequirement` constraint. E.g. in the example above, the parameter type `String` and return type `String` both conform to the `Codable` protocol, so this distributed protocol is well formed. + +It is possible to for a distributed actor protocol to contain non-distributed requirements. However in practice, it will be impossible to ever invoke such methods on a remote distributed actor reference. It is possible to call such methods if one were to obtain a local distributed actor reference implementing such protocol, and use the existing `whenLocal(operation:)` method on it. + +The `@Resolvable` macro generates a number of internal declarations necessary for the distributed machinery to work, however the only type users should care about is always a `$`-prefixed concrete distributed actor declaration, that is the "stub" type. This stub type can be used to resolve a remote actor using this protocol stub: + +E.g. if we knew a remote has a `Greeter` instance for a specific ID we have discovered using some external mechanism, this is how we'd resolve it: + +```swift +let clusterSystem: ClusterSystem // example system + +let greeter = try $Greeter.resolve(id: id, using: clusterSystem) +``` + +As the `ClusterSystem` is using `Codable` as it's serialization requirement, the resolve compiles and produces a valid reference. + +### Distributed actors generic over their `ActorSystem` + +The previous section made use of a distributed actor that abstracted over a generic `ActorSystem`. Today (in Swift 5.10) this is not possible, and would result in a compile time error. + +Previously, the compiler would require that the `ActorSystem` typealias refer to a specific distributed actor system type (not a protocol, a concrete nominal type). For example, the following actor can only be used with the `ClusterSystem`: + +```swift +import Distributed +import DistributedCluster // github.com/apple/swift-distributed-actors provides `ClusterSystem` + +distributed actor DistributedAsyncSequence where Element: Sendable & Codable { + typealias ActorSystem = ClusterSystem + + // not real implementation; simplified method to showcase introduced capabilities + distributed func gimmeNextElement() async throws -> Element? { ... } +} +``` + +And while such generally useful "distributed async sequence" actor can be written generically, to work with the vast majority of actor systems, today's language did not allow to write such generic actor, and the `ActorSystem` type always was forced to be a _concrete_ type. + +To support this new pattern, the `DistributedActorSystem` protocol gains a *primary associated type* for the `SerializationRequirement` associated type: + +```swift +// before: +// protocol DistributedActorSystem: Sendable { +// associatedtype SerializationRequirement where ... +// // ... +// } + +// now: +protocol DistributedActorSystem: Sendable { + /// The serialization requirement that will be applied to all distributed targets used with this system. + associatedtype SerializationRequirement + where SerializationRequirement == InvocationEncoder.SerializationRequirement, + SerializationRequirement == InvocationDecoder.SerializationRequirement, + SerializationRequirement == ResultHandler.SerializationRequirement + // ... +} + +// +``` + +The `SerializationRequirement` must be specified for all actors and protocols attempting to abstract over an actor system, because it is necessary to compile-time guarantee the correctness of values passed to such distributed actor methods. The compiler uses this associated type to verify all argument types and returned values are able to be serialized when performing remote calls, and will refuse to compile invocations would otherwise would have failed at runtime. + +Thanks to this new primary associated type, it is now possible to spell our `DistributedAsyncSequence` as a generic actor, implement it once, and re-use it across any compatible actor system implementation: + +```swift +distributed actor DistributedAsyncSequence + where Element: Sendable & Codable, + ActorSystem: DistributedActorSystem { + + // not real implementation; simplified method to showcase introduced capabilities + distributed func exampleNextElement() async throws -> Element? { ... } +} +``` + +Note that since the `ActorSystem` specifies a concrete `SerializationRequirement` the compiler is still able to check that all types invoked in a distributed function call conform to this protocol, i.e. we're guaranteed to be able to serialize `Element` because the ActorSystem provided must be able to handle this serialization mechanism. + +This also extends to distributed protocols, which are now able to abstract over an actor system, while specifying what serialization requirement they support: + +```swift +protocol DistributedAsyncSequence: DistributedActor + where ActorSystem: DistributedActorSystem { + associatedtype Element: Sendable & Codable + + distributed func exampleNextElement() async throws -> Element? { ... } +} +``` + +Failing to specify the serialization requirement is a compile time error: + +```swift +protocol DistributedAsyncSequence: DistributedActor + where ActorSystem: DistributedActorSystem { + // error: distributed actor protocol must specify `ActorSystem.SerializationRequirement`, + // you can provide it like this: DistributedActorSystem + associatedtype Element: Sendable & Codable + + distributed func exampleNextElement() async throws -> Element? { ... } +} +``` + +The serialization requirement must be a `protocol` type; This was previously enforced, and remains so after this proposal. The important part is that the macro is able to synthesize an stub implementation type, that the existing resolution mechanisms can be invoked on. + +## Detailed design + +The `@Resolvable` macro generates a concrete `distributed actor` declaration as well as an extension which implements the protocol's method requirements with "stub" implementations. + +> **NOTE:** The exact details of the macro synthesized code are not guaranteed to remain the same, and may change without notice. The existence of the $-prefixed generated type is guaranteed however, as it is the public API how developers resolve and obtain remote references, I.e. for a `protocol Greeter` annotated with the `@Resolvable` macro, developers may rely on the existence of the `distributed actor $Greeter` with the same access level as the protocol. + +This proposal also introduces an empty `DistributedActorStub` protocol: + +```swift +public protocol DistributedActorStub where Self: DistributedActor {} +``` + +The `@Resolvable` macro synthesizes a concrete distributed actor which accepts a generic `ActorSystem`. The generated actor declaration matches access level with the original declaration, and implements the protocol as well as the `DistributedActorStub` protocol: + +```swift +protocol Greeter: DistributedActor where ActorSystem: DistributedActorSystem { + distributed func greet(name: String) -> String +} + +// "stub" type +distributed actor $Greeter: Greeter, DistributedStubActor + where DistributedActorSystem { + private init() {} // cannot initialize, can only resolve(id:using:) +} + +extension Greeter where Self: DistributedActorStub { + // ... stub implementations for protocol requirements ... +} +``` + +Default implementations for all the protocol's requirements (including non-distributed requirements) are provided by extensions utilizing the `DistributedActorStub` protocol. + +It is possible for a protocol type to inherit other protocols, in that case the parent protocol must either have default implementations for all its requirements, or it must also apply the `@Resolvable` protocol which generates such default implementations. + +The default method "stub" implementations provided by the `@Resolvable` simply fatal error if they were to ever be invoked. In practice, invoking those methods is not possible, because resolving a stub will always return a remote reference, and therefore calls on these methods are redirected to `DistributedActorSystem`'s `remoteCall` rather than invoking the "local" methods. + +It is recommended to use `some Greeter` or `any Greeter` rather than `$Greeter` when passing around resolved instances of a distributed actor protocol. This way none of your code is tied to the fact of using a specific type of proxy, but rather, can accept any greeter, be it local or resolved through a proxy reference. This can come in handy when refactoring a codebase, and merging modules in such way where the greeter may actually be a local instance in some situations. + +### Interaction with the `DefaultDistributedActorSystem` + +Since the introduction of distributed actors, it is possible to declare a module-wide `DefaultDistributedActorSystem` type alias, like this: + +```swift +typealias DefaultDistributedActorSystem = ClusterSystem +``` + +This makes it easier to declare distributed actors as the `ActorSystem` type requirement is witnessed by an implicit type alias generated in every concrete distributed actor, like this: + +```swift +distributed actor Worker { + // synthesized: + // typealias ActorSystem = ClusterSystem // because 'DefaultDistributedActorSystem = ClusterSystem' + + distributed func work() +} +``` + +The newly introduced ability to abstract over the `ActorSystem` in concrete distributed actors _wins_ over the synthesized typealias, causing the typealias to not be emitted: + +```swift +distributed actor Worker where ActorSystem: DistributedActorSystem { + distributed func work() +} +``` + +The `ActorSystem` type requirement of the `DistributedActorProtocol` in this case is witnessed by the generic parameter, and not by the "default" fallback type. + +This is the right behavior because this generic type works with _any_ distributed actor system where the `SerializationRequirement` is Codable, and not only on the `ClusterSystem`. + +### Extend Distributed metadata for protocol method identifier lookups + +The way distributed method invocations work on the recipient node is that a message is parsed from some incoming transport, and a `RemoteCallTarget` is recovered. The remote call target in currently is a mangled encoding of the concrete distributed method the call was made for, like this: + +A shared module introducing the `Capybara` protocol: + +```swift +// Module "Shared" +@Resolvable +public protocol Capybara where ActorSystem: DistributedActorSystem { + distributed var name: String { get } + distributed func eat() +} +``` + +The distributed actor runtime stores static metadata about distributed methods, such that the `executeDistributedTarget(on:target:invocationDecoder:handler:)` method is able to turn the mangled `RemoteCallIdentifier` into a concrete method handle that it then invokes. This proposal introduces a way to mangle calls on distributed protocol requirements, in such a way that they refer to the `$`-prefixed name, and invocations on such accessor are performed on the recipient side using the concrete actor's witness tables. + +We can illustrate the new `remoteCall` flow like this: + +**Caller, i.e. client side:** + +```swift +// Module MyClientApp + +let discoveredCaplinID = ... +let capybara: some Capybara = try $Caybara.resolve(id: discoveredCaplinID, using: system) + +// make the remote call, without knowing the concrete +// capybara type that we'll invoke on the remote side +let name = try await capybara.name + +// invokes selected actor system's remoteCall: +// DistributedActorSystem.remoteCall( +// on: someCapybara, +// target: RemoteCallIdentifier("Shared.$Capybara.name"), <<< PROTOCOL REQUIREMENT MANGLING +// invocation: , +// throwing: Never.self, +// returnType: String.self) +// ------- + +assert("Hello, \(name)!" == + "Hello, Caplin!") +``` + +The caller just performed a normal remote call as usual, however the Distributed runtime offered a protocol based remote call identifier (`RemoteCallIdentifier("Shared.$Capybara.name")`) rather than one based on some underlying type that is hidden under the `any Capybara` existential. + +The client-side does not need to know or care about what concrete implementation type is used to implement this call on the recipient system, as it will only ever be performing such distributed protocol based calls. + +**Recipient, i.e. server side:** + +```swift +// Module MyActorSystem + +final class MySampleDistributedActorSystem: DistributedActorSystem, ... { + // ... + + func findById(_ id: ActorID) -> (any DistributedActor)? { ... } + + func receiveMessage() async throws { + let envelope: MyTransportEnvelope = try await readFromNetwork() + + guard let actor = findById(envelope.id) else { + throw TargetActorNotFound(envelope.id) + } + + // RemoteCallIdentifier("Shared.Capybara.name") <<< + let target: RemoteCallTarget = envelope.target + try await executeDistributedTarget( + on: actor, + target: target, // the protocol method identifier + invocationDecoder: invocation.makeDecoder(), + handler: resultHandler + } +) +``` + +This logic is exactly the same as any existing `DistributedActorSystem` implementation -- however, changes in the Distributed runtime will handle the protocol method invocation and be able to route it to the concrete resolved actor (returned by `findByID` in this snippet, e.g. the `Caplin` concrete type). + +## Source compatibility + +The changes proposed are purely additive. + +The introduced macros do not introduce any new capabilities to the `DistributedActorSystem` protocol itself, but rather introduce new source generation techniques. + +## ABI compatibility + +This proposal is purely ABI additive. + +We introduce new static, accessible at runtime, metadata necessary for the identification and location of distributed protocol methods. + +## Wire compatibility + +> Since distributed actors are used across processes, an additional kind of compatibility is necessary to discuss in proposals which may impact how messages are sent or methods identified and invoked. + +This proposal is additive and provides additional metadata such that "distributed protocol" methods may be invoked across process. Such calls were previously not supported. + +Remote calls are identified using the `RemoteCallTarget` struct, which contains an `identifier` of the target method. In today's distributed actors these identifiers are the mangled name of the target method. + +This proposal introduces a special way to mangle calls made on default implementations of distributed protocol requirements, in such a way that the target type identifier of the protocol (e.g. `Greeter`) is replaced with the stub type (e.g. `$Greeter`), and the server performs the invocation on a specific target actor using the concrete types witness and generic accessor thunk when such calls are made. + +## Future directions + +### Improve tools for non-breaking protocol evolution + +While this approach allows sharing protocols as source of "truth" for APIs vended by a server, their capability to evolve over time is limited. + +We find that the needs of distributed protocol evolution overlap in some parts with protocol evolution in binary stable libraries. While removing an API would always be breaking, it should be possible to automatically deprecate one method, and delegate to another by adding a new parameter and defaulting it in the deprecated version. + +This could be handled with declaration macros, introducing a peer method with the expected new API: + +```swift +@Resolvable(deprecatedName: "Greeter") +protocol DeprecatedGreeter: DistributedActor { + @Distributed.Deprecated(newVersion: greet(name:), defaults: [name: nil]) + distributed func greet() -> String + + // whoops, we forgot about a name parameter and need to add it... + // We can delegate from greet() to greet(name: nil) automatically though! +} +``` + +The deprecation macro could generate the necessary delegation code, like this: + +```swift +protocol Greeter: DistributedActor { + @Distributed.Deprecated(newVersion: greet(name:), defaults: [name: nil]) + distributed func greet() -> String + +/*** + distributed func greet(name: String?) -> String + ***/ +} + +/*** +extension Greeter { + /// Default implementation for deprecated ``greet()`` + /// Delegates to ``greet(name:)`` + distributed func greet() -> String { + self.greet(name: nil) + } +} + ***/ +``` + +This simplified snippet does not solve the problem about introducing the new protocol requirement in a binary compatible way, and we'd have to come up with some pattern for it -- however the general direction of allowing introducing new versions of APIs with easier deprecation of old ones is something we'd like to explore in the future. + +Support for renaming methods could also be provided, such that the legacy method can be called `__deprecated_greet()` for example, while maintaining the "legacy name" of "`greet()`". Overall, we believe that the protocol evolution story here is something we will have to flesh out in the near future, anf feel we have the tools to do so. + +### Consider customization points for distributed call target metadata assignment + +Currently the metadata used for remote call targets is based on Swift's mangling scheme. This is sub-optimal as it includes slightly "too much" information, such as the parameters being classes or structs, un-necessarily wire causing wire-incompatible changes when one could handle them more gracefully. + +Another downside of using the mangling for the keys of the distributed method accessor identifiers is that the names can be rather long as mangling is pretty verbose. It is possible to avoid sending then complete metadata by using compression schemes such that each identifier can only be sent at-most-once over the wire, and later on a numeric representation is used between the peers. Such scheme would need to be implemented dynamically at runtime and involves some tricky logic. It would be interesting to provide a hook in the actor system to allow for consistent remote call target identifier assignment, such that the identifiers could be both small, and predictable on both sides of a system. + +This would require introducing a dynamic lookup table in the runtime and it would need to interoperate with any given distributed actor system... It remains unclear if this is a net win, or un-necessary complexity since each actor system may handle this slightly differently... + +### Utilize distributed method metadata for auditing + +Given the information in distributed metadata, we could provide a command line application, or rather extend `swift-inspect` to be able to inspect a binary or running application for the distributed entry points to the application. + +This is useful as it allows auditors to quickly scan for all potential distributed entry points into an application, making auditing easier and more reliable than source scanning as it can be performed on the final artifact of a build. + +## Alternatives considered + +### Restrict distributed actors only to peer-to-peer systems + +Since this feature expands the usefulness of distributed actors to client/server settings, we should discuss wether or not this is a good idea to begin with. + +This topic has been one of heated discussions in various ecosystems every time distributed actor systems are compared to source-generation based RPC systems (such as gRPC, or OpenAPI, and others like SOAP and others in earlier days). + +Distributed actors in Swift have the unique position of being placed in a language that deeply embraces the actor model. There are valid reasons to NOT use distributed actors in some situations, and e.g. prefer exposing your APIs over OpenAPI or other source generation tools, especially if one wants to treat these as the "source of truth." + +At the same time though, Swift is used in many exciting domains where the use of OpenAPI, or gRPC would be deemed problematic. We are interested in supporting IPC and other low-level systems which are tightly integrated with eachother, and frequently even maintained by the same teams or organizations. Deeply integrating the language, with auditing capabilities and control over distributed process boundaries, without having to step out to secondary source generation and tools is a very valuable goal in these scenarios. + +### Handle stub synthesis in the compiler + +An earlier attempt at implementation of this feature attempted to handle synthesis in the compiler, and emit ad-hoc distributed actor declaration types as triggered by the _call site_ of `resolve(id:using:)` - this is problematic in being a very custom and special path in the compiler, complicating the language and giving distributed actors more "privileges" than normal code. + +The idea was as follows: + +```swift +protocol Greeter: DistributedActor { + distributed func greet(name: String) -> String +} + +let someSystem: some DistributedActorSystem = ... +let g: any Greeter = try .resolve(id: id, using: someSystem) +``` + +This would have to synthesize an ad-hoc created anonymous declaration for a `$Greeter` and at the site of the `resolve` type-check if the declaration can be used with the `someSystem`'s serialization requirement. We would have to check if the distributed greet method's parameters and return type conform to `Codable` etc, and all this would have to happen lazily -- triggered by the existence of a `.resolve` method combining a protocol with a specific actor system. + +The only possible spelling of such API would have been this: `let g: any Greeter = try .resolve(...)` as the concrete type that is used to implement this `any Greeter` is not user visible, and cannot be. This is a lot of complexity, to what amounts to just a simple stub type. + +We believe that the macro stub approach is a good balance between convenience and lack of "magic" compiler support, as for this specific piece of the design no deep integration in the type system is necessary. + + +## Revisions + +- 1.2 + - Change implementation to not need `#resolve` macro, but rely on generic distributed actors + - General cleanup + - Change implementation approach to macros, introduce `#resolve` macro +- 1.0 + - Initial revision diff --git a/proposals/0429-partial-consumption.md b/proposals/0429-partial-consumption.md new file mode 100644 index 0000000000..d760ce11b9 --- /dev/null +++ b/proposals/0429-partial-consumption.md @@ -0,0 +1,334 @@ +# Partial consumption of noncopyable values + +* Proposal: [SE-0429](0429-partial-consumption.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm), [Nate Chandler](https://github.com/nate-chandler) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch #1](https://forums.swift.org/t/request-for-feedback-partial-consumption-of-fields-of-noncopyable-types/65884)) ([pitch #2](https://forums.swift.org/t/pitch-piecewise-consumption-of-noncopyable-values/70045)) ([review](https://forums.swift.org/t/se-0429-partial-consumption-of-noncopyable-values/70675)) ([acceptance](https://forums.swift.org/t/accepted-se-0429-partial-consumption-of-noncopyable-values/70972)) + +## Introduction + +We propose allowing noncopyable fields in deinit-less aggregates to be consumed individually, +so long as they are defined in the current module or frozen. +Additionally, we propose allowing fields of such an aggregate with a deinit to be consumed individually _within that deinit_. +This permits common patterns to be used with many noncopyable values. + +## Motivation + +In Swift today, it can be challenging to manipulate noncopyable fields of an aggregate. + +For example, consider a `Pair` of noncopyable values: + +```swift +struct Unique : ~Copyable {...} +struct Pair : ~Copyable { + let first: Unique + let second: Unique +} +``` + +It is currently not straightforward to write a function that forms a new `Pair` with the values reversed. +For example, the following code is not currently allowed: + +```swift +extension Pair { + consuming func swap() -> Pair { + return Pair( + first: second, // error: cannot partially consume 'self' + second: first // error: cannot partially consume 'self' + ) + } +} +``` + +There are various workarounds for this, but they are not ideal. + +## Proposed solution + +We allow noncopyable aggregates without deinits to be consumed field-by-field, if they are defined in the current module or frozen. +That makes `swap` above legal as written. + +This initial proposal is deliberately minimal: +- We do not allow partial consumption of [noncopyable aggregates that have deinits](#future-direction-discard). +- We do not support [reinitializing](#future-direction-partial-reinitialization) fields after they are consumed. + +[Imported aggregates](#imported-aggregates) can never be partially consumed, unless they are frozen. + +## Detailed design + +We relax the requirement that a noncopyable aggregate be consumed at most once on each path. +Instead we require only that each of its noncopyable fields be consumed at most once on each path. +Imported aggregates (i.e. those defined in another module and marked either `public` or `package`), however, cannot be partially consumed unless they are marked `@frozen`. + +Extending the `Pair` example above, the following becomes legal: + +```swift +func takeUnique(_ elt: consuming Unique) {} +extension Pair { + consuming func passUniques(_ forward: Bool) { + if forward { + takeUnique(first) + takeUnique(second) + } else { + takeUnique(second) + takeUnique(first) + } + } +} +``` + +The struct `Pair` has two noncopyable fields, `first` and `second`. +And there are two paths through the function: the paths taken when `forward` is `true` and when it is `false`. +On both paths, `first` and `second` are both consumed exactly once. + +It's not necessary to consume every field on every path, however. +For example, the following is allowed as well: + +```swift +extension Pair { + consuming func passUnique(_ front: Bool) { + if front { + takeUnique(first) + } else { + takeUnique(second) + } + } +} +``` + +Here, only `first` is consumed on the path taken when `front` is `true` and only `second` on that taken when `front` is `false`. + +### Field lifetime extension + +When a field is _not_ consumed on some path, its destruction is deferred as long as possible. +Here, that looks like this: + +```swift +extension Pair { + consuming func passUnique(_ front: Bool) { + if front { + takeUnique(first) + // second is destroyed + } else { + takeUnique(second) + // first is destroyed + } + } +} +``` + +Neither `first` nor `second` can be destroyed _after_ the `if`/`else` blocks because that would require a copy. + +### Explicit field consumption + +Fields can also be consumed explicitly via the `consume` keyword. +This enables overriding the [extension of a field's lifetime](#lifetime-extension). + +Continuing the example, if it were necessary that `first` always be destroyed before `second`, the following could be written: + +```swift +extension Pair { + consuming func passUnique(_ front: Bool) { + if front { + takeUnique(first) + // second is destroyed + } else { + _ = consume first + takeUnique(second) + } + } +} +``` + +### Imported aggregates + +Partial consumption of a non-copyable type is always allowed when the type is defined in the module where it is consumed. +If the type is defined in another module, partial consumption is only permitted if the type is marked `@frozen`. + +The reason for this limitation is that as the module defining a type changes, +the type itself may change, adding or removing fields, changing fields to computed properties, and so on. +A partial consumption of the type's fields that makes sense as the type is defined by one version of the module +may not make sense as the type is defined in another version. +That consideration does not apply to frozen types, however, +because by marking them `@frozen`, the module's author promises not to change their layouts. + +These rules are unavoidable for libraries built with library evolution +and are applied universally to avoid having language rules differ based on the build mode. + +### Copyable fields + +It is currently legal to have multiple consuming uses of a copyable field of a noncopyable aggregate. +For example: + +```swift +func takeString(_ name: consuming String) {} +struct Named : ~Copyable { + let unique: Unique + let name: String + consuming func use() { + takeString(name) + takeString(name) + takeString(name) + takeString(name) + // unique is consumed + } +} +``` + +This remains true when a value is partially consumed: + +```swift +extension Named { + consuming func unpack() { + takeString(name) + takeString(name) + takeUnique(unique) + takeString(name) + takeString(name) + } +} +``` + +### Partial consumption within deinits + +There are two related reasons to limit partial consumption to fields of types without deinits: +First, the deinit of such types can't be run if it is partially consumed. +Second, no proposed mechanism to indicate that the deinit should not be run has been accepted. + +Neither applies when partially consuming a value within its own deinit. +We propose allowing a value to be partially consumed there. + +```swift +struct Pair2 : ~Copyable { + let first: Unique + let second: Unique + + deinit { + takeUnique(first) // partially consumes self + takeUnique(second) // partially consumes self + } +} +``` + +This enables noncopyable structs to dispose of any resources they own on destruction. + +## Source compatibility + +No effect. +The proposal makes more code legal. + +## ABI compatibility + +No effect. + +## Implications on adoption + +This proposal makes more code legal. +And the code it makes legal is code written in a style familiar to Swift developers used to working with copyable values. +It alleviates some pain points associated with writing noncopyable code, easing further adoption. + +## Future directions + +### Discard + +This document proposes limiting partial consumption to aggregates without deinit. +In the future, another proposal could lift that restriction. +The trouble with lifting it is that the deinit can no longer be run, which may be surprising. +That trouble could be mitigated by requiring the value be `discard`'d prior to partial consumption, +indicating that the deinit should not be run. + +```swift +struct Box : ~Copyable { + var unique: Unique + deinit {...} + + consuming func unpack() -> Unique { + discard self + return unique + } +} +``` + +### Partial reinitialization + +This document only proposes allowing the fields of an aggregate to be consumed individually. +It does not allow for those fields to be _reinitialized_ in order to return the aggregate to a legal state. +In the future, though, another proposal could lift that restriction. + +That would enable further code patterns--already legal with copyable values--to be written in noncopyable contexts +For example: + +```swift +struct Unique : ~Copyable {} +struct Pair : ~Copyable { + var first: Unique + var second: Unique +} + +extension Pair { + mutating func swap() { + let tmp = first + first = second + second = tmp + } +} +``` + +### Partial consumption of copyable fields + +This document only proposes allowing the noncopyable fields of a noncopyable aggregate to be consumed individually. +In the future, the ability to explicitly consume (via the `consume` keyword) the copyable fields of a copyable aggregate could be added. + +```swift +class C {} +func takeC(_ c: consuming C) +struct PairPlusC : ~Copyable { + let first: Unique + let second: Unique + let c: C +} + +func disaggregate(_ p: consuming PairPlusC) { + takeUnique(p.first) + takeC(consume p.c) // p.c's lifetime ends + takeUnique(p.second) +} +``` + +That would provide the ability to specify the point at which the lifetime of a copyable field should end. + +### Partial consumption of copyable aggregates + +This document only proposes allowing noncopyable aggregates to be partially consumed. +There is a natural extension of this to copyable aggregates: + +```swift +class C {} +struct CopyablePairOfCs { + let c1: C + let c2: C +} +func tearDownInOrder(_ p: consuming CopyablePairOfCs) { + takeC(consume p.c2) + takeC(consume p.c1) +} +``` + +## Alternatives considered + +### Explicit destructuring + +Instead of consuming the fields of a struct piecewise, an alternative would be to simultaneously bind every field to a variable: + +```swift +let (a, b) = destructure s +``` + +Something like this might be desirable eventually, but it would be best introduced as part of support for pattern matching for structs. +Even with such a feature, the behavior proposed here would remain desirable: +fields of a copyable aggregate can be consumed field-by-field, +so consuming fields of a noncopyable aggregate should be supported as much as possible too. + +## Acknowledgments + +Thanks to Andrew Trick for extensive design conversations and implementation review. diff --git a/proposals/0430-transferring-parameters-and-results.md b/proposals/0430-transferring-parameters-and-results.md new file mode 100644 index 0000000000..031685c0ee --- /dev/null +++ b/proposals/0430-transferring-parameters-and-results.md @@ -0,0 +1,539 @@ +# `sending` parameter and result values + +* Proposal: [SE-0430](0430-transferring-parameters-and-results.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm), [Holly Borla](https://github.com/hborla), [John McCall](https://github.com/rjmccall) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 6.0)** +* Previous Proposal: [SE-0414: Region-based isolation](/proposals/0414-region-based-isolation.md) +* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/87943205551af43682ef50260816f3ff2ef9b7ea/proposals/0430-transferring-parameters-and-results.md) [2](https://github.com/apple/swift-evolution/blob/4dded8ed382b526a5a301c225a1d45018f8d556b/proposals/0430-transferring-parameters-and-results.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-transferring-isolation-regions-of-parameter-and-result-values/70240)) ([first review](https://forums.swift.org/t/se-0430-transferring-isolation-regions-of-parameter-and-result-values/70830)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0430-transferring-isolation-regions-of-parameter-and-result-values/71297)) ([second review](https://forums.swift.org/t/se-0430-second-review-sendable-parameter-and-result-values/71685)) ([acceptance with modifications](https://forums.swift.org/t/accepted-with-modifications-se-0430-second-review-sendable-parameter-and-result-values/71850)) ([amendment pitch](https://forums.swift.org/t/pitch-revise-se-0430-to-adopt-sending-on-unsafecontinuation/72289)) ([amendment review](https://forums.swift.org/t/amendment-se-0430-sending-parameter-and-result-values/72653)) + + +## Introduction + +This proposal extends region isolation to enable the application of an explicit +`sending` annotation to function parameters and results. A function parameter +or result that is annotated with `sending` is required to be disconnected at +the function boundary and thus possesses the capability of being safely sent +across an isolation domain or merged into an actor-isolated region in the +function's body or the function's caller respectively. + +## Motivation + +SE-0414 introduced region isolation to enable non-`Sendable` typed values to be +safely sent over isolation boundaries. In most cases, function argument and +result values are merged together into the same region for any given call. This +means that non-`Sendable` typed parameter values can never be sent: + +```swift +// Compiled with -swift-version 6 + +class NonSendable {} + +@MainActor func main(ns: NonSendable) {} + +func trySend(ns: NonSendable) async { + // error: sending 'ns' can result in data races. + // note: sending task-isolated 'ns' to main actor-isolated + // 'main' could cause races between main actor-isolated + // and task-isolated uses + await main(ns: ns) +} +``` + +Actor initializers have a special rule that requires their parameter values to be +sent into the actor instance's isolation region. Actor initializers are +`nonisolated`, so a call to an actor initializer does not cross an isolation +boundary, meaning the argument values would be usable in the caller after the +initializer returns under the standard region isolation rules. SE-0414 consider +actor initializer parameters as being sent into the actor's region to allow +initializing actor-isolated state with those values: + +```swift +class NonSendable {} + +actor MyActor { + let ns: NonSendable + init(ns: NonSendable) { + self.ns = ns + } +} + +func send() { + let ns = NonSendable() + let myActor = MyActor(ns: ns) // okay; 'ns' is sent into the 'myActor' region +} + +func invalidSend() { + let ns = NonSendable() + + // error: sending 'ns' may cause a data race + // note: sending 'ns' from nonisolated caller to actor-isolated + // 'init'. Later uses in caller could race with uses on the actor. + let myActor = MyActor(ns: ns) + + print(ns) // note: note: access here could race +} +``` + +In the above code, if the local variable `ns` in the function `send` was instead +a function parameter, it would be invalid to send `ns` into `myActor`'s region +because the caller of `send()` may use the argument value after `send()` +returns: + +```swift +func send(ns: NonSendable) { + // error: sending 'ns' may cause a data race + // note: task-isolated 'ns' to actor-isolated 'init' could cause races between + // actor-isolated and task-isolated uses. + let myActor = MyActor(ns: ns) +} + +func callSend() { + let ns = NonSendable() + send(ns: ns) + print(ns) +} +``` + +The "sending parameter" behavior of actor initializers is a generally +useful concept, but it is not possible to explicitly specify that functions +and methods can send away specific parameter values. Consider the following +code that uses `CheckedContinuation`: + +```swift +@MainActor var mainActorState: NonSendable? + +nonisolated func test() async { + let ns = await withCheckedContinuation { continuation in + Task { @MainActor in + let ns = NonSendable() + // Oh no! 'NonSendable' is passed from the main actor to a + // nonisolated context here! + continuation.resume(returning: ns) + + // Save 'ns' to main actor state for concurrent access later on + mainActorState = ns + } + } + + // 'ns' and 'mainActorState' are now the same non-Sendable value; + // concurrent access is possible! + ns.mutate() +} +``` + +In the above code, the closure argument to `withCheckedContinuation` crosses an +isolation boundary to get onto the main actor, creates a non-`Sendable` typed +value, then resumes the continuation with that non-`Sendable` typed value. The +non-`Sendable` typed value is then returned to the original `nonisolated` context, +thus crossing an isolation boundary. Because `resume(returning:)` does not +impose a `Sendable` requirement on its argument, this code does not produce any +data-race safety diagnostics, even under `-strict-concurrency=complete`. + +Requiring `Sendable` on the parameter type of `resume(returning:)` is a harsh +restriction, and it's safe to pass a non-`Sendable` typed value as long as the value +is in a disconnected region and all values in that disconnected region are not +used again after the call to `resume(returning:)`. + +## Proposed solution + +This proposal enables explicitly specifying parameter and result values as +possessing the capability of being sent over an isolation boundary by annotating +the value with a contextual `sending` keyword: + +```swift +public struct CheckedContinuation: Sendable { + public func resume(returning value: sending T) +} + +public func withCheckedContinuation( + function: String = #function, + _ body: (CheckedContinuation) -> Void +) async -> sending T +``` + +## Detailed design + +### Sendable Values and Sendable Types + +A type that conforms to the `Sendable` protocol is a thread-safe type: values of +that type can be shared with and used safely from multiple concurrent contexts +at once without causing data races. If a value does not conform to `Sendable`, +Swift must ensure that the value is never used concurrently. The value can still +be sent between concurrent contexts, but the send must be a complete transfer of +the value's entire region implying that all uses of the value (and anything +non-`Sendable` typed that can be reached from the value) must end in the source +concurrency context before any uses can begin in the destination concurrency +context. Swift achieves this property by requiring that the value is in a +disconnected region and we say that such a value is a `sending` value. + +Thus a newly-created value with no connections to existing regions is always a +`sending` value: + +```swift +func f() async { + // This is a `sending` value since we can transfer it safely... + let ns = NonSendable() + + // ... here by calling 'sendToMain'. + await sendToMain(ns) +} +``` + +Once defined, a `sending` value can be merged into other isolation +regions. Once merged, such regions, if not disconnected, will prevent the value +from being sent to another isolation domain implying that the value is no longer +a `sending` value: + +```swift +actor MyActor { + var myNS: NonSendable + + func g() async { + // 'ns' is initially a `sending` value since it is in a disconnected region... + let ns = NonSendable() + + // ... but once we assign 'ns' into 'myNS', 'ns' is no longer a sending + // value... + myNS = ns + + // ... causing calling 'sendToMain' to be an error. + await sendToMain(ns) + } +} +``` + +If a `sending` value's isolation region is merged into another disconnected +isolation region, then the value is still considered to be `sending` since two +disconnected regions when merged form a new disconnected region: + +```swift +func h() async { + // This is a `sending` value. + let ns = Nonsending() + + // This also a `sending` value. + let ns2 = NonSendable() + + // Since both ns and ns2 are disconnected, the region associated with + // tuple is also disconnected and thus 't' is a `sending` value... + let t = (ns, ns2) + + // ... that can be sent across a concurrency boundary safely. + await sendToMain(t) +} +``` + +### sending Parameters and Results + +A `sending` function parameter requires that the argument value be in a +disconnected region. At the point of the call, the disconnected region is no +longer in the caller's isolation domain, allowing the callee to send the +parameter value to a region that is opaque to the caller: + +```swift +@MainActor +func acceptSend(_: sending NonSendable) {} + +func sendToMain() async { + let ns = NonSendable() + + // error: sending 'ns' may cause a race + // note: 'ns' is passed as a 'sending' parameter to 'acceptSend'. Local uses could race with + // later uses in 'acceptSend'. + await acceptSend(ns) + + // note: access here could race + print(ns) +} +``` + +What the callee does with the argument value is opaque to the caller; the callee +may send the value away, or it may merge the value to the isolation region of +one of the other parameters. + +A `sending` result requires that the function implementation returns a value in +a disconnected region: + +```swift +@MainActor +struct S { + let ns: NonSendable + + func getNonSendableInvalid() -> sending NonSendable { + // error: sending 'self.ns' may cause a data race + // note: main actor-isolated 'self.ns' is returned as a 'sending' result. + // Caller uses could race against main actor-isolated uses. + return ns + } + + func getNonSendable() -> sending NonSendable { + return NonSendable() // okay + } +} +``` + +The caller of a function returning a `sending` result can assume the value is +in a disconnected region, enabling non-`Sendable` typed result values to cross +an actor isolation boundary: + +```swift +@MainActor func onMain(_: NonSendable) { ... } + +nonisolated func f(s: S) async { + let ns = s.getNonSendable() // okay; 'ns' is in a disconnected region + + await onMain(ns) // 'ns' can be sent away to the main actor +} +``` + +### Function subtyping + +For a given type `T`, `sending T` is a subtype of `T`. `sending` is +contravariant in parameter position; if a function type is expecting a regular +parameter of type `T`, it's perfectly valid to pass a `sending T` value +that is known to be in a disconnected region. If a function is expecting a +parameter of type `sending T`, it is not valid to pass a value that is not +in a disconnected region: + +```swift +func sendingParameterConversions( + f1: (sending NonSendable) -> Void, + f2: (NonSendable) -> Void +) { + let _: (sending NonSendable) -> Void = f1 // okay + let _: (sending NonSendable) -> Void = f2 // okay + let _: (NonSendable) -> Void = f1 // error +} +``` + +`sending` is covariant in result position. If a function returns a value +of type `sending T`, it's valid to instead treat the result as if it were +merged with the other parameters. If a function returns a regular value of type +`T`, it is not valid to assume the value is in a disconnected region: + +```swift +func sendingResultConversions( + f1: () -> sending NonSendable, + f2: () -> NonSendable +) { + let _: () -> sending NonSendable = f1 // okay + let _: () -> sending NonSendable = f2 // error + let _: () -> NonSendable = f1 // okay +} +``` + +### Protocol conformances + +A protocol requirement may include `sending` parameter or result annotations: + +```swift +protocol P1 { + func requirement(_: sending NonSendable) +} + +protocol P2 { + func requirement() -> sending NonSendable +} +``` + +Following the function subtyping rules in the previous section, a protocol +requirement with a `sending` parameter may be witnessed by a function with a +non-`sending` parameter: + +```swift +struct X1: P1 { + func requirement(_: sending NonSendable) {} +} + +struct X2: P1 { + func requirement(_: NonSendable) {} +} +``` + +A protocol requirement with a `sending` result must be witnessed by a function +with a `sending` result, and a requirement with a plain result of type `T` may +be witnessed by a function returning a `sending T`: + +```swift +struct Y1: P1 { + func requirement() -> sending NonSendable { + return NonSendable() + } +} + +struct Y2: P1 { + let ns: NonSendable + func requirement() -> NonSendable { // error + return ns + } +} +``` + +### `inout sending` parameters + +A `sending` parameter can also be marked as `inout`, meaning that the argument +value must be in a disconnected region when passed to the function, and the +parameter value must be in a disconnected region when the function +returns. Inside the function, the `inout sending` parameter can be merged with +actor-isolated callees or further sent as long as the parameter is +re-assigned a value in a disconnected region upon function exit. + +### Ownership convention for `sending` parameters + +When a call passes an argument to a `sending` parameter, the caller cannot +use the argument value again after the callee returns. By default `sending` +on a function parameter implies that the callee consumes the parameter. Like +`consuming` parameters, a `sending` parameter can be re-assigned inside +the callee. Unlike `consuming` parameters, `sending` parameters do not +have no-implicit-copying semantics. + +To opt into no-implicit-copying semantics or to change the default ownership +convention, `sending` may also be paired with an explicit `consuming` ownership modifier: + +```swift +func sendingConsuming(_ x: consuming sending T) { ... } +``` + +### Adoption in the Concurrency library + +There are several APIs in the concurrency library that send a parameter across +isolation boundaries and don't need the full guarantees of `Sendable`. These +APIs will instead adopt `sending` parameters: + +* `CheckedContinuation.resume(returning:)` +* `UnsafeContinuation.resume(returning:)` +* `Async{Throwing}Stream.Continuation.yield(_:)` +* `Async{Throwing}Stream.Continuation.yield(with:)` +* The `Task` creation APIs + +## Source compatibility + +In the Swift 5 language mode, `sending` diagnostics are suppressed under +minimal concurrency checking, and diagnosed as warnings under strict concurrency +checking. The diagnostics are errors in the Swift 6 language mode, as shown in +the code examples in this proposal. This diagnostic behavior based on language +mode allows `sending` to be adopted in existing Concurrency APIs including +`CheckedContinuation`. + +## ABI compatibility + +This proposal does not change how any existing code is compiled. + +## Implications on adoption + +Adding `sending` to a parameter is more restrictive at the caller, and +more expressive in the callee. Adding `sending` to a result type is more +restrictive in the callee, and more expressive in the caller. + +For libraries with library evolution, `sending` changes name mangling, so +any adoption must preserve the mangling using `@_silgen_name`. Adoping +`sending` must preserve the ownership convention of parameters; no +additional annotation is necessary if the parameter is already (implicitly or +explicitly) `consuming`. + +## Future directions + +### `Disconnected` types + +`sending` requires parameter and result values to be in a disconnected +region at the function boundary, but there is no way to preserve that a value +is in a disconnected region through stored properties, collections, function +calls, etc. To preserve that a value is in a disconnected region through the +type system, we could introduce a `Disconnected` type into the Concurrency +library. The `Disconnected` type would suppress copying via `~Copyable`, it +would conform to `Sendable`, constructing a `Disconnected` instance would +require the value it wraps to be in a disconnected region, and a value of type +`Disconnected` can never be merged into another isolation region. + +This would enable important patterns that take a `sending T` parameter, store +the value in a collection of `Disconnected`, and later remove values from the +collection and return them as `sending T` results. This would allow some +`AsyncSequence` types to return non-`Sendable` typed buffered elements as +`sending` without resorting to unsafe opt-outs in the implementation. + +## Alternatives considered + +### Use `transferring` or `sending` instead of `sendable` + +This proposal originally used the word `transferring` for `sendable`. The idea +was that this would superficially match parameter modifiers like `consuming` and +`borrowing`. But, this ignored that we are not actually `transferring` the +parameter into another isolation domain at the function boundary point. Instead, +we are requiring that the value at that point be in a disconnected region and +thus have the _capability_ to be sent to another isolation domain or merged into +actor isolated state. This is in contrast to `consuming` and `borrowing` which +actively affect the value at the function boundary point by consuming or +borrowing the value. Additionally, by using `transferring` would introduce a new +term of art into the language unnecessarily and contrasts with already +introduced terms like `@Sendable` and the `Sendable` protocol. + +It was also suggested that perhaps instead of renaming `transferring` to +`sendable`, it should have been renamed to `sending`. This was rejected by the +authors since it runs into the same problem as `transferring` namely that it is +suggesting that the value is actively being moved to another isolation domain, +when we are expressing a latent capability of the value. + +### Exclude `UnsafeContinuation` + +An earlier version of this proposal excluded +`UnsafeContinuation.resume(returning:)` from the list of standard library +APIs that adopt `sending`. This meant that `UnsafeContinuation` didn't +require either the return type to be `Sendable` or the return value to be +`sending`. Since `UnsafeContinuation` is unconditionally `Sendable`, this +effectively made it a major hole in sendability checking. + +This was an intentional choice. The reasoning was that `UnsafeContinuation` +was already an explicitly unsafe type, and so it's not illogical for it +to also work as an unsafe opt-out from sendability checks. There are some +uses of continuations that do need an opt-out like this. For example, it +is not uncommon for a continuation to be resumed from a context that's +isolated to the same actor as the context that called `withUnsafeContinuation`. +In this situation, the data flow through the continuation is essentially +internal to the actor. This means there's no need for any sendability +restrictions; both `sending` and `Sendable` would be over-conservative. + +However, the nature of the unsafety introduced by this exclusion is very +different from the normal unsafety of an `UnsafeContinuation`. +Continuations must be resumed exactly once; that's the condition that +`CheckedContinuation` checks. If a programmer can prove that +they will do that to their own satisfaction, they should be able to use +`UnsafeContinuation` instead of `CheckedContinuation` in full confidence. +Making `UnsafeContinuation` *also* a potential source of concurrency-safety +holes is likely to be surprising to programmers. + +Conversely, if a programmer needs to opt out of sendability checks but +is *not* confident about how many times their continuation will be +resumed --- for example, if it's resumed from an arbitrary callback --- +forcing them to adopt `UnsafeContinuation` in order to achieve their +goal is actively undesirable. + +Not requiring `sending` in `UnsafeContinuation` also makes the high-level +interfaces of `UnsafeContinuation` and `CheckedContinuation` inconsistent. +This means that programmers cannot always easily move from an unsafe to a +checked continuation. That is a common need, for example when fixing +a bug and trying to prove that the unsafe continuation is not implicated. + +Swift has generally resisted adding new dimensions of unsafety to unsafe +types this way. For example, `UnsafePointer` was originally specified as +unconditionally `Sendable` in [SE-0302][], but that conformance was +removed in [SE-0331][], and pointers are now unconditionally +non-`Sendable`. The logic in both of those proposals closely parallels +this one: at first, `UnsafePointer` was seen as an unsafe type that should +not be burdened with partial safety checks, and then the community +recognized that this was actually adding a new dimension of unsafety to +how the type interacted with concurrency. + +Finally, there is already a general unsafe opt-out from sendability +checking: `nonisolated(unsafe)`. It is better for Swift to encourage +consistent use of a single unsafe opt-out than to build *ad hoc* +opt-outs into many different APIs, because it is much easier to find, +recognize, and audit uses of the former. + +For these reasons, `UnsafeContinuation.resume(returning:)` now requires +its argument to be `sending`, and the result of `withUnsafeContinuation` +is correspondingly now marked as `sending`. + +[SE-0302]: https://github.com/apple/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md +[SE-0331]: https://github.com/apple/swift-evolution/blob/main/proposals/0331-remove-sendable-from-unsafepointer.md diff --git a/proposals/0431-isolated-any-functions.md b/proposals/0431-isolated-any-functions.md new file mode 100644 index 0000000000..1797df6616 --- /dev/null +++ b/proposals/0431-isolated-any-functions.md @@ -0,0 +1,863 @@ +# `@isolated(any)` Function Types + +* Proposal: [SE-0431](0431-isolated-any-functions.md) +* Authors: [John McCall](https://github.com/rjmccall) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.0)** +* Previous revision: [1](https://github.com/swiftlang/swift-evolution/blob/b35498bf6f198477be50809c0fec3944259e86d0/proposals/0431-isolated-any-functions.md) +* Review: ([pitch](https://forums.swift.org/t/isolated-any-function-types/70562))([review](https://forums.swift.org/t/se-0431-isolated-any-function-types/70939))([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0431-isolated-any-function-types/71611)) + +[SE-0316]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0316-global-actors.md +[SE-0392]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md +[isolated-captures]: https://forums.swift.org/t/closure-isolation-control/70378 +[generalized-isolation]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0420-inheritance-of-actor-isolation.md#generalized-isolation-checking +[regions]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0414-region-based-isolation.md +[region-transfers]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0430-transferring-parameters-and-results.md + +## Introduction + +The actor isolation of a function is an important part of how it's +used. Swift can reason precisely about the isolation of a specific +function *declaration*, but when functions are passed around as +*values*, Swift's function types are not expressive enough to keep up. +This proposal adds a new kind of function type that carries its function's +actor isolation dynamically. This solves a variety of expressivity +problems in the language. It also allows features such as the standard +library's task-creation APIs to be implemented more efficiently and +with stronger semantic guarantees. + +## Motivation + +The safety of Swift concurrency relies on understanding the isolation +requirements of functions. The caller of an isolated synchronous +function must run it in an appropriately-isolated context or else the +function will almost certainly introduce data races. + +Function declarations and closures in Swift support three different +forms of actor isolation: + +- They can be non-isolated. +- They can be isolated to a specific [global actor][SE-0316] type. +- They can be isolated to a specific parameter or captured value. + +A function's isolation can be specified or inferred in many ways. +Non-isolation is the default if no other rules apply, and it can also +be specified explicitly with the `nonisolated` modifier. Global actor +isolation can be expressed explicitly with a global actor attribute, +such as `@MainActor`, but it can also be inferred from context, such +as in the methods of main-actor-isolated types. A function can +explicitly declare one of its parameters as `isolated` to isolate +itself to the value of that parameter; this is also done implicitly to +the `self` parameter of an actor method if the method doesn't explicitly +use some other isolation. Closure expressions can be declared with a +global actor attribute, and there is a [proposal currently being +developed][isolated-captures] to also allow them to have an explicit +`isolated` capture or to be explicitly non-isolated. Additionally, +when you pass a closure expression directly to the `Task` initializer, +that closure is inferred to have the isolation of the enclosing context.[^1] +These rules are fairly complex, but at the end of the day, they all +boil down to this: every function is assigned one of the three kinds of +actor isolation above. + +[^1]: Currently, if the enclosing context is isolated to a value, the +closure is only isolated to it if it actually captures that value (by +using it somewhere in its body). This is often seen as confusing, and +the `isolated` captures proposal is considering lifting this restriction +by unconditionally capturing the value. + +When a function is called directly, Swift's isolation checker can +analyze its isolation precisely and compare that to the isolation of the +calling context. However, when a call expression calls an opaque value +of function type, Swift is limited by what can be expressed in the type +system: + +- A function type with no isolation specifiers, such as `() -> Int`, + represents a non-isolated function. + +- A function type with a global actor attribute, such as + `@MainActor () -> Int`, represents a function that's isolated to that + global actor. + +- A function type with an `isolated` parameter, such as + `(isolated MyActor) - > Int`, represents a function that's isolated to + that parameter. + +But there's a very important case that can't be expressed in the type +system like this: a closure can be isolated to one of its captures. In +the following example, the closure is isolated to its captured `self` +value: + +```swift +actor WorldModelObject { + var position: Point3D + + func linearMove(to finalPosition: Point3D, over time: Duration) { + let originalPosition = self.position + let motion = finalPosition - originalPosition + + gradually(over: time) { [isolated self] progressProportion in + self.position = originalPosition + progressProportion * motion + } + } + + func updateLater() { + Task { + // This closure doesn't have an explicit isolation + // specification, and it's being passed to the `Task` + // initializer, so it will be inferred to have the same + // isolation as its enclosing context. The enclosing + // context is isolated to its `self` parameter, which this + // closure captures, so this closure will also be isolated + // that value. + self.update() + } + } +} +``` + +This inexpressible case also arises with a partial application of an +actor method, such as `myActor.methodName`: the resulting function +value captures `myActor` and is isolated to it. For now, these are +the only two cases of isolated captures. However, the upcoming +[closure isolation control][isolated-captures] proposal is expected +to give this significantly greater prominence and importance. Under +that proposal, isolated captures will become a powerful general tool +for controlling the isolation of a specific piece of code. But there +will still not be a way to express the isolation of that closure in +the type system.[^2] + +[^2]: Expressing this exactly would require the use of value-dependent +types. Value dependence is an advanced type system feature that we +cannot easily add to Swift. This is discussed in greater depth in +the Future Directions section. + +This is a very unfortunate limitation, because it actually means that +there's no way for a function to accept a function argument with +arbitrary isolation without completely erasing that isolation. Swift +does allow functions with arbitrary isolation to be converted to a +non-isolated function type, but this comes with three severe drawbacks. +The first is that the resulting function type must be `async` so that +it can switch to the right isolation internally. The second is that, +because the function changes isolation internally, it is limited in its +ability to work with non-`Sendable` values because any argument or return +value must cross an isolation boundary. And the third is that the +isolation is completely dynamically erased: there is no way for the +recipient of the function value to recover what isolation the function +actually wants, which often puts the recipient in the position of doing +unnecessary work. + +Here's an example of that last problem. The `Task` initializer receives +an opaque value of type `() async throws -> ()`. Because it cannot +dynamically recover the isolation from this value, the initializer has +no choice but to start the task on the global concurrent executor. If +the function passed to the initializer is actually isolated to an actor, +it will immediately switch to that actor on entry. This requires +additional synchronization and may require re-suspending the task. +Perhaps more importantly, it means that the order in which tasks are +actually enqueued on the actor is not necessarily the same as the order +in which they were created. It would be much better --- both semantically +and for performance --- if the initializer could immediately enqueue the +task on the right executor to begin with. + +The straightforward solution to these problems is to add a type which +is capable of expressing a function with an arbitrary (but statically +unknown) isolation. That is what we propose to do. + +## Proposed solution + +This proposal adds a new attribute that can be placed on function types: + +```swift +func gradually(over: Duration, operation: @isolated(any) (Double) -> ()) +``` + +A function value with this type dynamically carries the isolation of +the function that was used to initialize it. + +When such a function is called from an arbitrary context, it must be +assumed to always cross an isolation boundary. This means, among other +things, that the call is effectively asynchronous and must be `await`ed. + +```swift +await operation(timePassed / overallDuration) +``` + +The isolation can be read using the special `isolation` property +of these types: + +```swift +func traverse(operation: @isolated(any) (Node) -> ()) { + let isolation = operation.isolation +} +``` + +The isolation checker knows that the value of this special property +matches the isolation of the function, so calls to the function from +contexts that are isolated to the `isolation` value do not cross +an isolation boundary. + +Finally, every task-creation API in the standard library will be updated +to take a `@isolated(any)` function value and synchronously enqueue the +new task on the appropriate executor. + +## Detailed design + +### Grammar and structural rules + +`@isolated(any)` is a new type attribute that can only be applied to +function types. It is an isolation specification, and it is an error +to combine it with other isolation specifications such as a global +actor attribute or an `isolated` parameter. + +`@isolated(any)` is not a *concrete* isolation specification and cannot +be directly applied to a declaration or a closure. That is, you cannot +declare a function *entity* as having `@isolated(any)` isolation, +because Swift needs to know what the actual isolation is, and +`@isolated(any)` does not provide a rule for that. + +### Conversions + +Let `F` and `G` be function types, and let `F'` and `G'` be the corresponding +function types with any isolation specifier removed (including but not +limited to `@isolated(any)`. If either `F` or `G` specifies +`@isolated(any)` then a value of type `F` can be converted to type `G` +if a value of type `F'` could be converted to type `G'` *and* the +following conditions apply: + +- If `F` and `G` both specify `@isolated(any)`, there are no further + conditions. The resulting function is dynamically isolated to the + same value as the original function. + +- If only `G` specifies `@isolated(any)`, then the behavior depends on the + specified isolation of `F`: + + - If `F` has an `isolated` parameter, the conversion is invalid. + - Otherwise, the conversion is valid, and the dynamic isolation of + the resulting function is determined as follows: + - If the converted value is the result of an expression that is a + closure expression (including an implicit autoclosure), a function + reference, or a partial application of a method reference, the + resulting function is dynamically isolated to the isolation of the + function or closure. This looks through syntax that has no impact + on the value produced by the expression, such as parentheses; the + list is the same as in [SE-0420][generalized-isolation]. + - Otherwise, if `F` is non-isolated, the resulting function is + dynamically non-isolated. + - Otherwise, `F` must be isolated to a global actor, and the resulting + function is dynamically isolated to that global actor. + +- If only `F` specifies `@isolated(any)`, then `G` must be an `async` function + type. `G` may have any isolation specifier, but it will be ignored and the + function will run with the isolation of the original value. The arguments + and result must be sendable across an isolation boundary. It is unspecified + whether the task will dynamically suspend when calling or returning from + the resulting value. + +### Effects of intermediate conversions + +In general, all of the isolation semantics and runtime behaviors laid out +here are affected by intermediate conversions to non-`@isolated(any)` +function types. For example, if you coerce a non-isolated function or +closure to the type `@MainActor () -> ()`, the resulting function will +thereafter be treated as a `MainActor`-isolated function; the fact that +it was originally a non-isolated function is both statically and +dynamically erased. + +### Runtime behavior + +Calling a `@isolated(any)` function value behaves the same way as a direct +call to a function with that isolation would: + +- If the function is `async`, it will run with its formal isolation. This + includes leaving isolated contexts if the function is dynamically + non-isolated, as specified by [SE-0338](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0338-clarify-execution-non-actor-async.md). + +- If the function is synchronous, it will run with its formal isolation + only if it is dynamically isolated. If it is dynamically non-isolated, + it will simply run synchronously in the current context, even if that + is isolated, just like an ordinary call to a non-isolated synchronous + function would. + +### `isolation` property + +Values of `@isolated(any)` function type have a special `isolation` +property. The property is read-only and has type `(any Actor)?`. The +value of the property is determined by the dynamic isolation of the +function value: + +- If the function is dynamically non-isolated, the value of `isolation` + is `nil`. +- If the function is dynamically isolated to a global actor type `G`, + the value of `isolation` is `G.shared`. +- If the function is dynamically isolated to a specific actor reference, + the value of `isolation` is that actor reference. + +### Distributed actors + +Function values cannot generally be isolated to a distributed actor +unless the actor is known to be local. When a distributed actor *is* +local, function values isolated to the actor can be converted to +`@isolated(any)` type as above. The `isolation` property presents +the distributed actor as an `(any Actor)?` using the same mechanism +as [`#isolation`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0420-inheritance-of-actor-isolation.md#isolated-distributed-actors). + +### Isolation checking + +Since the isolation of an `@isolated(any)` function value is +statically unknown, calls to it typically cross an isolation boundary. +This means that the call must be `await`ed even if the function is +synchronous, and the arguments and result must satisfy the usual +sendability restrictions for cross-isolation calls. The function +value itself must satisfy a slightly less restrictive rule: it must +be a sendable value only if it is `async` and the current +context is not statically known to be non-isolated.[^4] + +[^4]: The reasoning here is as follows. All actor-isolated functions +are inherently `Sendable` because they will only use their captures from +an isolated context.[^5] There is only a data-race risk for the +captures of a non-`Sendable` `@isolated(any)` function in the case +where the function is dynamically non-isolated. The sendability +restrictions therefore boil down to the same restrictions we would +impose on calling a non-isolated function. A call to a non-isolated +function never crosses an isolation boundary if the function is +synchronous or if the current context is non-isolated. + +[^5]: Sending an isolated function value may cause its captures to be +*destroyed* in a different context from the function's formal isolation. +Swift pervasively assumes this is okay: copies of non-`Sendable` values +must still be managed in a thread-safe manner. This is a significant +departure from Rust, where non-`Send` values cannot necessarily be safely +managed concurrently, and it means that `Sendable` is not sufficient +to enable optimizations like non-atomic reference counting. Swift +accepts this in exchange for being more permissive, as long as the code +avoids "user-visible" data races. Note that this assumption is not new +to this proposal. + +In order for a call to an `@isolated(any)` function to be treated as +not crossing an isolation boundary, the caller must be known to have +the same isolation as the function. Since the isolation of an +`@isolated(any)` parameter is necessarily an opaque value, this would +require the caller to be declared with value-specific isolation. It +is currently not possible for a local function or closure to be +isolated to a specific value that isn't already the isolation of the +current context.[^6] The following rules lay out how `@isolated(any)` +should interact with possible future language support for functions +that are explicitly isolated to a captured value. In order to +present these rules, this proposal uses the syntax currently proposed +by the [closure isolation control pitch][isolated-captures], where +putting `isolated` before a capture makes the closure isolated to +that value. This should not be construed as accepting the terms of +that pitch. Accepting this proposal will leave most of this +section "suspended" until a feature with a similar effect is added +to the language. + +[^6]: Technically, it is possible to achieve this effect in Swift +today in a way that Swift could conceivably look through: the caller +could be a closure with an `isolated` parameter, and that closure +could be called with an expression like `fn.isolation` as the argument. +Swift could analyze this to see that the parameter has the value of +`fn.isolation` and then understand the connection between the caller's +isolation and `fn`. This would be very cumbersome, though, and it +would have significant expressivity gaps vs. an isolated-captures +feature. + +If `f` is an immutable binding of `@isolated(any)` function type, +then a call to `f` does not cross an isolation boundary if the +current context is isolated to a *derivation* of the expression +`f.isolation`. + +In the isolated captures pitch, a closure can be isolated to a specific +value by using the `isolated` modifier on an entry in its capture list. +So this question would reduce to whether that capture was initialized +to a derivation of `f.isolation`. + +An expression is a derivation of some expression form `E` if: + +- it has the exact form required by `E`; +- it is a reference to a capture or immutable binding immediately + initialized with a derivation of `E`; +- it is the result of `?` (the optional-chaining operator) or `!` + (the optional-forcing operator) applied to a derivation of `E`; or +- it is a reference to a non-optional binding (an immutable binding + initialized by a successful pattern-match which removes optionality, + such as `x` in `if let x = E`) of a derivation of `E`. + +The term *immutable binding* in the rules above means a `let` constant +or immutable (non-`inout`) parameter that is neither `weak` nor +`unowned`. The analysis ignores syntax that has no effect on the +value of an expression, such as parentheses; the exact set of cases +are the same as described in [SE-0420][generalized-isolation]. + +For example: + +```swift +func delay(operation: @isolated(any) () -> ()) { + let isolation = operation.isolation + Task { [isolated isolation] in // <-- tentative syntax from the isolated captures pitch + print("waking") + operation() // <-- does not cross an isolation barrier and so is synchronous + print("finished") + } +} +``` + +In this example, the expression `operation()` calls `operation`, +which is an immutable binding (a parameter) of `@isolated(any)` +function type. The call therefore does not cross an isolation +boundary if the calling context is isolated to a derivation of +`operation.isolation`. The calling context is the closure passed +to `Task.init`, which has an explicit `isolated` capture named +`isolation` and so is isolated to that value of that capture. +The capture is initialized with the value of the enclosing +variable `isolation`, which is an immutable binding (a `let` +constant) initialized to `operation.isolation`. As such, the +calling context is isolated to a derivation of `operation.isolation`, +so the call does not cross an isolation boundary. + +The primary intent of the rules above is simply to extend the +generalized isolation checking rules laid out in +[SE-0420][generalized-isolation] to work with an underlying +expression like `fn.isolation`. However, the rules above go +beyond the SE-0420 rules in some ways, most importantly by looking +through local `let`s. Looking through such bindings was not especially +important for SE-0420, but it is important for this proposal. In +order to keep the rules consistent, the isolation checking rules from +SE-0420 will be "rebased" on top of the rules in this proposal, +as follows: + +- When calling a function with an `isolated` parameter `calleeParam`, + if the current context also has an `isolated` parameter or capture + `callerIsolation`, the function has the same isolation as the current + context if the argument expression corresponding to `calleeParam` is + a derivation of either: + + - a reference to `callerIsolation` or + - a call to `DistributedActor.asAnyActor` applied to a derivation of + `calleeIsolation`. + +As a result, the following code is now well-formed: + +```swift +func operate(actor1: isolated MyActor) { + let actor2 = actor1 + actor2.isolatedMethod() // Swift now knows that actor2 is isolated +} +``` + +There is no reason to write this code instead of just using `actor1`, +but it's good to have consistent rules. + +### Adoption in task-creation routines + +There are a large number of functions in the standard library that create +tasks: +- `Task.init` +- `Task.detached` +- `TaskGroup.addTask` +- `TaskGroup.addTaskUnlessCancelled` +- `ThrowingTaskGroup.addTask` +- `ThrowingTaskGroup.addTaskUnlessCancelled` +- `DiscardingTaskGroup.addTask` +- `DiscardingTaskGroup.addTaskUnlessCancelled` +- `ThrowingDiscardingTaskGroup.addTask` +- `ThrowingDiscardingTaskGroup.addUnlessCancelled` + +This proposal modifies all of these APIs so that the task function has +`@isolated(any)` function type. These APIs now all synchronously enqueue +the new task directly on the appropriate executor for the task function's +dynamic isolation. + +Swift reserves the right to optimize the execution of tasks to avoid +"unnecessary" isolation changes, such as when an isolated `async` function +starts by calling a function with different isolation.[^3] In general, this +includes optimizing where the task initially starts executing: + +```swift +@MainActor class MyViewController: UIViewController { + @IBAction func buttonTapped(_ sender : UIButton) { + Task { + // This closure is implicitly isolated to the main actor, but Swift + // is free to recognize that it doesn't actually need to start there. + let image = await downloadImage() + display.showImage(image) + } + } +} +``` + +[^3]: This optimization doesn't change the formal isolation of the functions +involved and so has no effect on the value of either `#isolation` or +`.isolation`. + +As an exception, in order to provide a primitive scheduling operation with +stronger guarantees, Swift will always start a task function on the +appropriate executor for its formal dynamic isolation unless: +- it is non-isolated or +- it comes from a closure expression that is only *implicitly* isolated + to an actor (that is, it has neither an explicit `isolated` capture + nor a global actor attribute). This can currently only happen with + `Task {}`. + +As a result, in the following code, these two tasks are guaranteed +to start executing on the main actor in the order in which they were +created, even if they immediately switch away from the main actor without +having done anything that requires isolation:[^4] + +```swift +func process() async { + Task { @MainActor in + ... + } + + // do some work + + Task { @MainActor in + ... + } +} +``` + + +[^4]: This sort of guarantee is important when working with a FIFO +"pipeline", which is a common pattern when working with explicit queues. +In a pipeline, code responds to an event by performing work on a series +of queues, like so: + + ```swift + func handleEvent(event: Event) {} + queue1.async { + let x = makeX(event) + queue2.async { + let y = makeY(event) + queue3.async { + handle(x, y) + } + } + } + } + ``` + + As long as execution always goes through the exact same sequence of FIFO + queues, each queue will execute its stage of the overall pipeline in + the same order as the events were originally received. This can be a + difficult property to maintain --- concurrency at any stage will destroy + it, as will skipping any stages of the pipeline --- but it's not uncommon + for systems to be architected around it. + +The exception here to allow more optimization for implicitly-isolated +closures is an effort to avoid turning `Task {}` into a surprising +performance bottleneck. Programmers often reach for `Task {}` just to +do something concurrently with the current context, such as downloading +a file from the internet and then storing it somewhere. However, if +`Task {}` is used from an isolated context (such as from a `@MainActor` +event handler), the closure passed to `Task` will implicitly formally +inherit that isolation. A strict interpretation of the scheduling +guarantee in this proposal would require the closure to run briefly +on the current actor before it could do anything else. That would mean +that the task could never begin the download immediately; it would have +to wait, not just for the current operation on the actor to finish, but +for the actor to finish processing everything else currently in its +queue. If this is needed, it is not unreasonable to ask programmers +to state it explicitly, just as they would have to from a non-isolated +context. + +## Source compatibility + +Most of this proposal is additive. The exception is the adoption +in the standard library, which changes the types of certain API +parameters. Calls to these APIs should continue to work, as any +function that could be passed to the current parameter should also +be convertible to an `@isolated(any)` type. The observed type of +the API will change, however, if anyone does an abstract reference +such as `Task.init`. Contravariant conversion should allow these +unapplied references to work in any concrete type context that +would accept the current function, but references in other contexts +can lead to source breaks (such as `var fn = Task.init`). This is +unlikely to be an issue in practice. More importantly, I believe +Swift has a general policy of declining to guarantee stable types +for unapplied function references in the standard library this way. +Doing so would prevent a wide variety of reasonable code evolution +for the library, such as generalizing the type of a parameter (as +this proposal does) or adding a new defaulted parameter. + +## ABI compatibility + +This feature does not change the ABI of any existing code. + +## Implications on adoption + +The basic functionality of `@isolated(any)` function types is +implemented directly in generated code and does not require runtime +support. + +Using a type as a generic argument generally requires runtime type +metadata support for the type. For `@isolated(any)` function types, +that metadata support requires a new Swift runtime. It will therefore +not possible to use a type such as `[@isolated(any) () -> ()]` when +back-deploying code on a platform with ABI stability. However, +wrapping the function in a `struct` with a single field will generally +work around this problem. (It also generally allows the function to +be stored more efficiently.) + +The task-creation APIs in the standard library have been implemented +in a way that allows their signatures to be changed without ABI +considerations. Direct enqueuing on the isolated actor does require +runtime support, but fortunately that support has present in the +concurrency runtime since the first release. Therefore, there should +not be any back-deployment problems supporting the proposed changes. + +Adopters of `@isolated(any)` function types will generally face the +same source-compatibility considerations as this proposal does with +the task-creation APIs: it requires generalizing some parameter types, +which generally should not cause incompatibilities with direct callers +but can introduce problems in the somewhat unlikely case that anyone +is using those function as values. + +### When to use `@isolated(any)` + +It is recommended that APIs which take functions that are likely to run +concurrently and don't have a predetermined isolation take those functions +as `@isolated(any)`. This allows the API to make more intelligent +scheduling decisions about the function. + +Examples that should usually use `@isolated(any)` include: +- functions that wrap the creation of a task +- algorithms that call a function multiple times in parallel, such as a + parallel `map` + +Examples that should usually not use `@isolated(any)` include: +- algorithms that preserve the current isolation, such as a non-parallel + `map`; these functions should usually take a non-`Sendable` function + instead +- APIs that intend to call the function with a specific isolation, such + as UI frameworks that expect their event handlers to be `@MainActor` + or actor functions that run an operation on the actor + +## Future directions + +### Interaction with `assumeIsolated` + +It would be convenient in some cases to be able to assert that the +current synchronous context is already isolated to the isolation of +an `@isolated(any)` function, allowing the function to be called without +crossing isolation. Similar functionality is provided by the +`assumeIsolated` function introduced by [SE-0392][SE-0392]. +Unfortunately, the current `assumeIsolated` function is inadequate +for this purpose for several reasons. + +The first problem is that `assumeIsolated` only works on a +non-optional actor reference. We could add a version of this API +which does work on optional actors, but it's not clear what it +should actually do if given a `nil` reference. A `nil` isolation +represents non-isolation, which of course does not actually isolate +anything. Should `assumeIsolated` check that the current context +has exactly the given isolation, or should it check that it is safe +to use something with the given isolation requirement from the current +context? The first rule is probably the one that most people would +assume when they first heard about the feature. However, it implies +that `assumeIsolated(nil)` should check that no actors are currently +isolated, and that is not something we can feasibly check in general: +Swift's concurrency runtime does track the current isolation of a task, +but outside of a task, arbitrary things can be isolated without Swift +knowing about them. It is also needlessly restrictive, because there +is nothing that is unsafe to do in an isolated context that would be +safe if done in a non-isolated context.[^7] The second rule is less +intuitive but more closely matches the safety properties that static +isolation checking tests for. It implies that `assumeIsolated(nil)` +should always succeed. This is notably good enough for `@isolated(any)`: +since `assumeIsolated` is a synchronous function, only synchronous +`@isolated(any)` functions can be called within it, and calling a +synchronous non-isolated function always runs immediately without +changing the current isolation. + +[^7]: As far as data-race safety goes, at least. A specific actor +could conceivably have important semantic restrictions against doing +certain operations in its isolated code. Of course, such an actor should +generally not be calling arbitrary functions that are handed to it. + +The second problem is that `assumeIsolated` does not currently establish +a link back to the original expression passed to it. Code such as +the following is invalid: + +```swift +myActor.assumeIsolated { + myActor.property += 1 // invalid: Swift doesn't know that myActor is isolated +} +``` + +The callback passed to `assumeIsolated` is isolated because it takes +an `isolated` parameter, and while this parameter is always bound to +the actor that `assumeIsolated` was called on, Swift's isolation checking +doesn't know that. As a result, it is necessary to use the parameter +instead of the original actor reference, which is a persistent annoyance +when using this API: + +```swift +myActor.assumeIsolated { myActor2 in + myActor2.property += 1 +} +``` + +For `@isolated(any)`, we would naturally want to write this: + +```swift +myFn.isolation.assumeIsolated { + myFn() +} +``` + +However, since Swift doesn't understand the connection between the +closure's `isolated` parameter and `myFn`, this call will not work, +and there is no way to make it work. + +One way to fix this would be to add some new way to assert that an +`@isolated(any)` function is currently isolated. This could even +destructure the function value, giving the program access it to as +a non-`@isolated(any)` function. But it seems like a better approach +to allow isolation checking to understand that the `isolated` parameter +and the `self` argument of `assumeIsolated` are the same value. +That would fix both the usability problem with actors and the +expressivity problem with `@isolated(any)`. Decomposition could +be done as a general rule that permits isolation to be removed from +a function value as long as that isolation matches the current +context and the resulting function is non-`Sendable`. + +This is all sufficiently complex that it seems best to leave it for +a future direction. However, it should be relatively approachable. + +### Statically-isolated function types + +`@isolated(any)` function types are effectively an "existential +erasure" of the isolation of the function, removing the type system's +static knowledge of the isolation while dynamically preserving it. +This is directly analogous to how `Any` erases the type of the value +you store into it: the type system no longer knows statically what +type is stored there, but it's still possible to recover it dynamically. +This analogy is why this proposal uses the keyword `any` in the +attribute name. + +Where there's an existential, there's also a generic. The generic +analogue to `@isolated(any)` would be a type that expressed that it +was isolated to a specific value, like so: + +```swift +func delay(on operationActor: A, + operation: @isolated(to: operationActor) () async -> ()) +``` + +This is a kind of value-dependent type. Value-dependent types add a +lot of complexity to a type system. Consider how the arguments interact +in the example above: both value and type information from the first +argument flows into the second. This is not something to do lightly, +and we think Swift is relatively unlikely to ever add such a feature +as `@isolated(to:)`. + +Fortunately, it is unlikely to be necessary. We believe that +`@isolated(any)` function types are superior from a usability perspective +for all the dominant patterns of higher-order APIs. The main thing that +`@isolated(to:)` can express in an API signature that `@isolated(any)` +cannot is multiple functions that share a common isolation. It is +quite uncommon for APIs to take multiple closely-related functions +this way, especially `@Sendable` functions where there's an expected +isolation change from the current context. When only a single function +is required in an API, `@isolated(any)` allows its isolation to bound +up with it in a single value, which is both more convenient and likely +to have a more performant representation. + +If Swift ever does explore in the direction of `@isolated(to:)`, +nothing in this proposal would interfere with it. In fact, the +features would support each other well. Erasing the isolation of +an `@isolated(to:)` function into an `@isolated(any)` type would +be straightforward, much like erasing an `Int` into an `Any`. +Similarly, an `@isolated(any)` function could be "opened" into a +pair of an `@isolated(to:)` function and its known isolation. +Since the common cases will still be more convenient to express +with `@isolated(any)`, the community is unlikely to regret having +added this proposal first. + +## Alternatives considered + +### Other spellings + +`isolated` and `nonisolated` are used as bare-word modifiers in several +places already in Swift: you can declare a parameter as `isolated`, and +you can declare methods and properties as `nonisolated`. Using `@isolated` +as a function type attribute therefore risks confusion about whether +`isolated` should be written with an `@` sign. + +One alternative would be to drop the `@` sign and spell these function +types as e.g. `isolated(any) () -> ()`. However, this comes with its own +problems. Modifiers typically affect a specific entity without changing +its type; for example, the `weak` modifier makes a variable or property +a weak reference, but the type of that reference is unchanged (although +it is required to be optional). This wouldn't be too confusing if +modifiers and types were written in fundamentally different places, but +it's expected that `@isolated(any)` will usually be used on parameter +functions, and parameter modifiers are written immediately adjacent to +the parameter type. As a result, removing the `@` would create this +unfortunate situation: + +```swift +// This means `foo` is isolated to the actor passed in as `actor`. +func foo(actor: isolated MyActor) {} + +// This means `operation` is a value of isolated(any) function type; +// it has no impact on the isolation of `bar`. +func bar(operation: isolated(any) () -> ()) +``` + +It is better to preserve the current rule that type modifiers are +written with an `@` sign. + +Another alternative would be to not spell the attribute `@isolated(any)`. +For example, it could be spelled `@anyIsolated` or `@dynamicallyIsolated`. +The spelling `@isolated(any)` was chosen because there's an expectation +that this will be one of a family of related isolation-specifying +attributes. For example, if Swift wanted to make it easier to inherit +actor isolation from one's caller, it could add an `@isolated(caller)` +attribute. Another example is the `@isolated(to:)` future direction +listed above. There's merit in having these attributes be closely +related in spelling. Using a common `Isolated` suffix could serve as +that connection, but in the author's opinion, `@isolated` is much +clearer. + +If programmers do end up confused about when to use `@` with `isolated`, +it should be relatively straightforward to provide a good compiler +experience that corrects misuses. + +### Implying `@Sendable` + +An earlier version of this proposal made `@isolated(any)` imply `@Sendable`. +The logic behind this implication was that `@isolated(any)` is only +really useful if the function is going to be passed to a different +concurrent context. If a function cannot be passed to a different +concurrent context, the reasoning goes, there's really no point in +it carrying its isolation dynamically, because it can only be used +if that isolation is compatible with the current context. There's +therefore no reason not to eliminate the redundant `@Sendable` attribute. + +However, this logic subtly misunderstands the meaning of `Sendable` +in a world with [region-based isolation][regions]. A type conforming +to `Sendable` means that its values are intrinsically thread-safe and +can be used from multiple concurrent contexts *concurrently*. +Values of non-`Sendable` type are still safe to use from different +concurrent contexts as long as those uses are well-ordered: if the +value is properly [transferred][region-transfers] between contexts, +everything is fine. Given that, it is sensible for a non-`Sendable` +function to be `@isolated(any)`: if the function can be transferred +to a different concurrent context, it's still useful for it to carry +its isolation dynamically. + +In particular, something like a task-creation function ought to declare +the initial task function as a non-`@Sendable` but still transferrable +`@isolated(any)` function. This permits closures passed in to capture +non-`Sendable` state as long as that state can be transferred into the +closure. (Ideally, the initial task function would then be able to +transfer that captured state out of the closure. However, this would +require the compiler to understand that the task function is only +called once.) + +## Acknowledgments + +I'd like to thank Holly Borla and Konrad Malawski for many long +conversations about the design and implementation of this feature. diff --git a/proposals/0432-noncopyable-switch.md b/proposals/0432-noncopyable-switch.md new file mode 100644 index 0000000000..5eb6171845 --- /dev/null +++ b/proposals/0432-noncopyable-switch.md @@ -0,0 +1,324 @@ +# Borrowing and consuming pattern matching for noncopyable types + +* Proposal: [SE-0432](0432-noncopyable-switch.md) +* Authors: [Joe Groff](https://github.com/jckarter) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.0)** +* Experimental Feature Flag: `BorrowingSwitch` +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/86cf6eadcdb35a09eb03330bf5d4f31f2599da02/proposals/ABCD-noncopyable-switch.md) +* Review: ([review](https://forums.swift.org/t/se-0432-borrowing-and-consuming-pattern-matching-for-noncopyable-types/71158)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0432-borrowing-and-consuming-pattern-matching-for-noncopyable-types/71656)) + +## Introduction + +Pattern matching over noncopyable types, particularly noncopyable enums, can +be generalized to allow for pattern matches that borrow their subject, in +addition to the existing support for consuming pattern matches. + +## Motivation + +[SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md) +introduced noncopyable types, allowing for programs to define +structs and enums whose values cannot be copied. However, it restricted +`switch` over noncopyable values to be a `consuming` operation, meaning that +nothing can be done with a value after it's been matched against. This +severely limits the expressivity of noncopyable enums in particular, +since switching over them is the only way to access their associated values. + +## Proposed solution + +We lift the restriction that noncopyable pattern matches must consume their +subject value, and formalize the ownership behavior of patterns during +matching and dispatch to case blocks. `switch` statements **infer their +ownership behavior** based on a combination of whether the subject expression +refers to storage or a temporary value, in addition to the necessary ownership +behavior of the patterns in the `switch`. We also introduce **borrowing +bindings** into patterns, as a way of explicitly declaring a binding as a +borrow that doesn't allow for implicit copies. + +## Detailed design + +### Determining the ownership behavior of a `switch` operation + +Whether a `switch` borrows or consumes its subject can be determined from +the subject expression and the patterns involved in the switch. Based on +the criteria below, a switch may be one of: + +- **copying**, meaning that the subject is semantically copied, and additional + copies of some or all of the subject value may be formed to execute the + pattern match. +- **borrowing**, meaning that the subject is borrowed for the duration of the + `switch` block. +- **consuming**, meaning that the subject is consumed by the `switch` block. + +These modes can be thought of as being increasing in strictness. The compiler +looks recursively through the patterns in the `switch` and increases the +strictness of the `switch` behavior when it sees a pattern requiring stricter +ownership behavior. For copyable subjects, *copying* is the baseline mode, +whereas for noncopyable subjects, the baseline mode depends on the subject +expression: + +- If the expression refers to a variable or stored property, and is not + explicitly consumed using the `consume` operator, then the baseline + mode is *borrowing*. (Properties and subscripts which use the experimental + `_read`, `_modify`, or `unsafeAddress` accessors also get a baseline mode + of borrowing.) +- Otherwise, the baseline mode is *consuming*. + +For example, given the following copyable definition: + +```swift +enum CopyableEnum { + case foo(Int) + case bar(Int, String) +} +``` + +then the following patterns have ownership behavior as indicated below: + +```swift +case let x: // copying +case .foo(let x): // copying +case .bar(let x, let y): // copying +``` + +And for a noncopyable enum definition: + +```swift +struct NC: ~Copyable {} + +enum NoncopyableEnum: ~Copyable { + case copyable(Int) + case noncopyable(NC) +} +``` + +then the following patterns have ownership behavior as indicated below: + +```swift +var foo: NoncopyableEnum // stored variable + +switch foo { +case let x: // borrowing + +case .copyable(let x): // borrowing (because `x: Int` is copyable) + +case .noncopyable(let x): // borrowing +} + +func bar() -> NoncopyableEnum {...} // function returning a temporary + +switch bar() { +case let x: // consuming +case .copyable(let x): // borrowing (because `x: Int` is copyable) +case .noncopyable(let x): // consuming +} +``` + +### Refining the ownership behavior of `switch` + +The order in which `switch` patterns are evaluated is unspecified in Swift, +aside from the property that when multiple patterns can match a value, +the earliest matching `case` condition takes priority. Therefore, it is +important that matching dispatch **cannot mutate or consume the subject** +until a final match has been chosen. For copyable values, this means that +pattern matching operations can't mutate the subject, but they can be copied +as necessary to keep an instance of the subject available throughout the +pattern match even if a match operation wants to consume an instance of +part of the value. + +Copying isn't an option for noncopyable types, so +**noncopyable types strictly cannot undergo `consuming` operations until +the pattern match is complete**. For many kinds of pattern matches, this +doesn't need to affect their expressivity, since checking whether a type +matches the pattern criteria can be done nondestructively separate from +consuming the value to form variable bindings. Matching enum cases and tuples +(when noncopyable tuples are supported) for instance is still possible +even if they contain consuming `let` or `var` bindings as subpatterns: + +```swift +extension Handle { + var isReady: Bool { ... } +} + +let x: MyNCEnum = ... +switch consume x { +// OK to have `let y` in multiple patterns because we can delay consuming +// `x` to form bindings until we establish a match +case .foo(let y) where y.isReady: + y.close() +case .foo(let y): + y.close() +} +``` + +However, when a pattern has a `where` clause, variable bindings cannot be +consumed in the `where` clause even if the binding is consumable in the `case` +body: + +```swift +extension Handle { + consuming func tryClose() -> Bool { ... } +} + +let x: MyNCEnum = ... +switch consume x { +// error: cannot consume `y` in a "where" clause +case .foo(let y) where y.tryClose(): + // OK to consume in the case body + y.close() +case .foo(let y): + y.close() +} +``` + +Similarly, an expression subpattern whose `~=` operator consumes the subject +cannot be used to test a noncopyable subpattern. + +```swift +extension Handle { + static func ~=(identifier: Int, handle: consuming Handle) -> Bool { ... } +} + +switch consume x { +// error: uses a `~=` operator that would consume the subject before +// a match is chosen +case .foo(42): + .... +case .foo(let y): + ... +} +``` + +Noncopyable types do not yet support dynamic casting, but it is worth +anticipating how `is` and `as` patterns will work given this restriction. +An `is T` pattern only needs to determine whether the value being matched can +be cast to `T` or not, which can generally be answered nondestructively. +However, in order to form the value of type `T`, many kinds of casting, +including casts that bridge or which wrap the value in an existential +container, need to consume or copy parts of the input value in order to form +the result. The cast can still be separated into a check whether the type +matches, using a borrowing access, followed by constructing the actual cast +result by consuming if necessary. To do this, the switch would have already +be a consuming switch. But also, for a consuming `as T` pattern to work, the +subpattern `p` of the `p as T` pattern would need to be irrefutable, and the +pattern could not have an associated `where` clause, since we would be unable +to back out of the pattern match once a consuming cast is performed. + +### `case` conditions in `if`, `while`, `for`, and `guard` + +Patterns can also appear in `if`, `while`, `for`, and `guard` forms as part +of `case` conditions, such as `if case = { }`. These behave +just like `switch`es with one `case` containing the pattern, corresponding +to a true condition result with bindings, and a `default` branch corresponding +to a false condition result. Therefore, the ownership behavior of the `case` +condition on the subject follows the behavior of that one pattern. + +## Source compatibility + +SE-0390 explicitly required that a `switch` over a noncopyable variable +use the `consume` operator. This will continue to work in most cases, forcing +the lifetime of the binding to end regardless of whether the `switch` actually +consumes it or not. In some cases, the formal lifetime of the value or parts +of it may end up different than the previous implementation, but because +enums cannot yet have `deinit`s, noncopyable tuples are not yet supported, +and structs with `deinit`s cannot be partially destructured and must be +consumed as a whole, it is unlikely that this will be noticeable in real +world code. + +Previously, it was theoretically legal for noncopyable `switch`es to use +consuming `~=` operators, or to consume pattern bindings in the `where` +clause of a pattern. This proposal now expressly forbids these formulations. +We believe it is impossible to exploit these capabilities in practice under the +old implementation, since doing so would leave the value partially or fully +consumed on the failure path where the `~=` match or `where` clause fails, +leading to either mysterious ownership error messages, compiler crashes, or +both. + +## ABI compatibility + +This proposal has no effect on ABI. + +## Future directions + +### `inout` pattern matches + +With this proposal, pattern matches are able to *borrow* and *consume* their +subjects, but they still aren't able to take exclusive `inout` access to a +value and bind parts of it for in-place mutation. This proposal lays the +groundwork for supporting this in the future; we could introduce `inout` +bindings in patterns, and introducing **mutating** switch behavior as a level +of ownership strictness between *borrowing* and *consuming*. + +### Automatic borrow deduction for `let` bindings, and explicitly `consuming` bindings + +When working with copyable types, although `let` and `var` bindings formally +bind independent copies of their values, in cases where it's semantically +equivalent, the compiler optimizes aways the copy and borrows the original +value in place, with the idea that developers do not need to think about +ownership if the compiler does an acceptable job of optimizing their code. +By similar means, we could say that `let` pattern bindings for noncopyable types +borrow rather than consume their binding automatically if the binding is +not used in a way that requires it to consume the binding. This would +give developers a "do what I mean" model for noncopyable types closer to the +convenience of copyable types. This should be a backward compatible change +since it would allow for strictly more code to compile than does currently +when `let` bindings are always consuming. + +Conversely, performance-minded developers would also like to have explicit +control over ownership behavior and copying, while working with either +copyable or noncopyable types. To that end, we could add explicitly `consuming` +bindings to patterns as well, which would not be implicitly copyable, and +which would force the switch behavior mode on the subject to become *consuming* +even if the subject is copyable. + +### enum `deinit` + +SE-0390 left `enum`s without the ability to have a `deinit`, based on the fact +that the initial implementation of noncopyable types only supported consuming +`switch`es. Noncopyable types with `deinit`s generally cannot be decomposed, +since doing so would bypass the `deinit` and potentially violate invariants +maintained by `init` and `deinit` on the type, so an `enum` with a `deinit` +would be completely unusable when the only primitive operation supported on it +is consuming `switch`. Now that this proposal allows for `borrowing` switches, +we could allow `enum`s to have `deinit`s, with the restriction that such +enums cannot be decomposed by a consuming `switch`. + +### Explicit `borrow` operator + +The [`borrow` operator](https://forums.swift.org/t/selective-control-of-implicit-copying-behavior-take-borrow-and-copy-operators-noimplicitcopy/60168) +could be used in the future to explicitly mark the subject of a switch as +being borrowed, even if it is normally copyable or would be a consumable +temporary, as in: + +```swift +let x: String? = "hello" + +switch borrow x { +case .some(let y): // ensure y is bound from a borrow of x, no copies + ... +} +``` + +### `borrowing` bindings in patterns + +In the future, we want to support `borrowing` and `inout` local bindings +in functions and potentially even as fields in nonescapable types. It might +also be useful to specify explicitly `borrowing` bindings within patterns. +Although the default behavior for a `let` binding within a noncopyable +borrowing `switch` pattern is to borrow the matched value, an explicitly +`borrowing` binding could be used to indicate that a copyable binding should +have its local implicit copyability suppressed, like a `borrowing` parameter +binding. + +## Alternatives considered + +### Determining pattern match ownership wholly from patterns + +The [first pitched revision](https://github.com/swiftlang/swift-evolution/blob/86cf6eadcdb35a09eb03330bf5d4f31f2599da02/proposals/ABCD-noncopyable-switch.md) +of this proposal kept `let` bindings in patterns as always being consuming +bindings, and required the use of `borrowing` bindings in every pattern in order +for a `switch` to act as a borrow. Early feedback using the feature found this +tedious; `borrowing` is more often a better default for accessing values +stored in variables and stored properties. This led us to the design now +proposed, where `let` behaves as a copying, consuming, or borrowing binding +based on the subject expression. diff --git a/proposals/0433-mutex.md b/proposals/0433-mutex.md new file mode 100644 index 0000000000..cdb7946574 --- /dev/null +++ b/proposals/0433-mutex.md @@ -0,0 +1,351 @@ +# Synchronous Mutual Exclusion Lock 🔒 + +* Proposal: [SE-0433](0433-mutex.md) +* Author: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Stephen Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-synchronous-mutual-exclusion-lock/69889)), ([review](https://forums.swift.org/t/se-0433-synchronous-mutual-exclusion-lock/71174)), ([acceptance](https://forums.swift.org/t/accepted-se-0433-synchronous-mutual-exclusion-lock/71463)) + +## Introduction + +This proposal introduces a mutual exclusion lock, or a mutex, to the standard library. `Mutex` will be a new synchronization primitive in the synchronization module. + +## Motivation + +In concurrent programs, protecting shared mutable state is one of the core fundamental problems to ensuring reading and writing data is done in an explainable fashion. Synchronizing access to shared mutable state is not a new problem in Swift. We've introduced many features to help protect mutable data. Actors are a good default go-to solution for protecting mutable state because it isolates the stored data in its own domain. At any given point in time, only one task will be executing "on" the actor, and have exclusive access to it. Multiple tasks cannot access state protected by the actor at the same time, although they may interleave execution at potential suspension points (indicated by `await`). In general, the actor approach also lends itself well to code organization, since the actor's state, and operations on this state are logically declared in the same place: inside the actor. + +Not all code may be able (or want) to adopt actors. Reasons for this can be very varied, for example code may have to execute synchronously without any potential for other tasks interleaving with it. Or the `async` effect introduced on methods may prevent legacy code which cannot use Swift Concurrency from interacting with the protected state. + +Whatever the reason may be, it may not be feasible to use an actor. In such cases, Swift currently is missing standard tools for developers to ensure proper synchronization in their concurrent data-structures. Many Swift programs opt to use ad-hoc implementations of a mutual exclusion lock, or a mutex. A mutex is a simple to use synchronization primitive to help protect shared mutable data by ensuring that a single execution context has exclusive access to the related data. The main issue is that there isn't a single standardized implementation for this synchronization primitive resulting in everyone needing to roll their own. + +## Proposed solution + +We propose a new type in the Standard Library Synchronization module: `Mutex`. This type will be a wrapper over a platform-specific mutex primitive, along with a user-defined mutable state to protect. Below is an example use of `Mutex` protecting some internal data in a class usable simultaneously by many threads: + +```swift +class FancyManagerOfSorts { + let cache = Mutex<[String: Resource]>([:]) + + func save(_ resource: Resource, as key: String) { + cache.withLock { + $0[key] = resource + } + } +} +``` + +Use cases for such a synchronized type are common. Another common need is a global cache, such as a dictionary: + +```swift +let globalCache = Mutex<[MyKey: MyValue]>([:]) +``` + +### Toolchains + +You can try out `Mutex` using one of the following toolchains: + +macOS: https://ci.swift.org/job/swift-PR-toolchain-macos/1207/artifact/branch-main/swift-PR-71383-1207-osx.tar.gz + +Linux (x86_64): https://download.swift.org/tmp/pull-request/71383/779/ubuntu2004/PR-ubuntu2004.tar.gz + +Windows: `https://ci-external.swift.org/job/swift-PR-build-toolchain-windows/1200/artifact/*zip*/archive.zip` + +Note that these toolchains don't currently have the `transferring inout` implemented, but the functions are marked `@Sendable` to at least enforce sendability. + +## Detailed design + +### Underlying System Mutex Implementation + +The `Mutex` type proposed is a wrapper around a platform's implementation. + +* macOS, iOS, watchOS, tvOS, visionOS: + * `os_unfair_lock` +* Linux: + * `futex` +* Windows: + * `SRWLOCK` + +These mutex implementations all have different capabilities and guarantee different levels of fairness. Our proposed `Mutex` type does not guarantee fairness, and therefore it's okay to have different behavior from platform to platform. We only guarantee that only one execution context at a time will have access to the critical section, via mutual exclusion. + +### API Design + +Below is the complete API design for the new `Mutex` type: + +```swift +/// A synchronization primitive that protects shared mutable state via +/// mutual exclusion. +/// +/// The `Mutex` type offers non-recursive exclusive access to the state +/// it is protecting by blocking threads attempting to acquire the lock. +/// At any one time, only one execution context at a time has access to +/// the value stored within the `Mutex` allowing for exclusive access. +/// +/// An example use of `Mutex` in a class used simultaneously by many +/// threads protecting a `Dictionary` value: +/// +/// class Manager { +/// let cache = Mutex<[Key: Resource]>([:]) +/// +/// func saveResource(_ resource: Resource, as key: Key) { +/// cache.withLock { +/// $0[key] = resource +/// } +/// } +/// } +/// +/// - Warning: Instances of this type are not recursive. Calls +/// to `withLock(_:)` (and related functions) within their +/// closure parameters will have platform-dependent behavior. +/// Some platforms may choose to panic the process, deadlock, +/// or leave this behavior unspecified. +/// +public struct Mutex: ~Copyable { + /// Initializes an instance of this mutex with the given initial state. + /// + /// - Parameter state: The initial state to give to the mutex. + public init(_ state: transferring consuming State) +} + +extension Mutex: Sendable where State: ~Copyable {} + +extension Mutex where State: ~Copyable { + /// Calls the given closure after acquiring the lock and then releases + /// ownership. + /// + /// This method is equivalent to the following sequence of code: + /// + /// mutex.lock() + /// defer { + /// mutex.unlock() + /// } + /// return try body(&value) + /// + /// - Warning: Recursive calls to `withLock` within the + /// closure parameter has behavior that is platform dependent. + /// Some platforms may choose to panic the process, deadlock, + /// or leave this behavior unspecified. This will never + /// reacquire the lock however. + /// + /// - Parameter body: A closure with a parameter of `State` + /// that has exclusive access to the value being stored within + /// this mutex. This closure is considered the critical section + /// as it will only be executed once the calling thread has + /// acquired the lock. + /// + /// - Returns: The return value, if any, of the `body` closure parameter. + public borrowing func withLock( + _ body: (transferring inout State) throws(E) -> transferring Result + ) throws(E) -> transferring Result + + /// Attempts to acquire the lock and then calls the given closure if + /// successful. + /// + /// If the calling thread was successful in acquiring the lock, the + /// closure will be executed and then immediately after it will + /// release ownership of the lock. If we were unable to acquire the + /// lock, this will return `nil`. + /// + /// This method is equivalent to the following sequence of code: + /// + /// guard mutex.tryLock() else { + /// return nil + /// } + /// defer { + /// mutex.unlock() + /// } + /// return try body(&value) + /// + /// - Warning: Recursive calls to `withLockIfAvailable` within the + /// closure parameter has behavior that is platform dependent. + /// Some platforms may choose to panic the process, deadlock, + /// or leave this behavior unspecified. This will never + /// reacquire the lock however. + /// + /// - Parameter body: A closure with a parameter of `State` + /// that has exclusive access to the value being stored within + /// this mutex. This closure is considered the critical section + /// as it will only be executed if the calling thread acquires + /// the lock. + /// + /// - Returns: The return value, if any, of the `body` closure parameter + /// or nil if the lock couldn't be acquired. + public borrowing func withLockIfAvailable( + _ body: (transferring inout State) throws(E) -> transferring Result? + ) throws(E) -> transferring Result? +} +``` + +## Interaction with Existing Language Features + +`Mutex` will be decorated with the `@_staticExclusiveOnly` attribute, meaning you will not be able to declare a variable of type `Mutex` as `var`. These are the same restrictions imposed on the recently accepted `Atomic` and `AtomicLazyReference` types. Please refer to the [Atomics proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0410-atomics.md) for a more in-depth discussion on what is allowed and not allowed. These restrictions are enabled for `Mutex` for all of the same reasons why it was restricted for `Atomic`. We do not want to introduce dynamic exclusivity checking when accessing a value of `Mutex` as a class stored property for instance. + +### Interactions with Swift Concurrency + +`Mutex` is unconditionally `Sendable` regardless of the value it's protecting. We can ensure the safetyness of this value due to the `transferring` marked parameters of both the initializer and the closure `inout` argument. (Please refer to [SE-0430 `transferring` isolation regions of parameter and result values](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0430-transferring-parameters-and-results.md)) This allows us to statically determine that the non-sendable value we're initializing the mutex with will have no other uses after initialization. Within the closure body, it ensures that if we tried to escape the protected non-sendable value, it would require us to replace the stored value for the notion of transferring out and then transferring back in. + +Consider the following example of a mutex to a non-sendable class. + +```swift +class NonSendableReference { + var prop: UnsafeMutablePointer +} + +// Some non-sendable class reference somewhere, perhaps a global. +let nonSendableRef = NonSendableReference(...) + +let lockedPointer = Mutex>(...) + +func something() { + lockedPointer.withLock { + // error: isolated parameter transferred out + // but hasn't had a value transferred back in. + nonSendableRef.prop = $0 + } +} +``` + +Had this closure not been marked `transferring inout` or perhaps `@Sendable`, then `Mutex` would not have protected this class references or any underlying memory referenced by pointers. Transferring inout allows us to plug this safety hole of shared mutable state by statically requiring that if we need to escape the closure or `Mutex` isolation domain as a whole, that we transfer in a new value in this domain (or it could be the same value perhaps). + +```swift +func something() { + lockedPointer.withLock { + // OK because we assign '$0' to a new value + nonSendableRef.prop = $0 + + // OK because we're transferring a new value in + $0 = ... + } +} +``` + +By marking the closure as such, we've effectively declared that the mutex is in itself its own isolation domain. We must not let non-sendable values it holds onto be unsafely sent across isolation domains to prevent these holes of shared mutable state. + +### Differences between mutexes and actors + +The mutex type we're proposing is a synchronous lock. This means when other participants want to acquire the lock to access the protected shared data, they will halt execution until they are able to do so. Threads that are waiting to acquire the lock will not be able to make forward progress until their request to acquire the lock has completed. This can lead to thread contention if the acquired thread's critical section is not able to be executed relatively quickly exhausting resources for the rest of the system to continue making forward progress. Synchronous locks are also prone to deadlocks (which Swift's actors cannot currently encounter due to their re-entrant nature) and live-locks which can leave a process in an unrecoverable state. These scenarios can occur when there is a complex hierarchy of different locks that manage to depend on the acquisition of each other. + +Actors work very differently. Typical use of an actor doesn't request access to underlying shared data, but rather instruct the actor to perform some operation or service that has exclusive access to that data. An execution context making this request may need to await on the return value of that operation, but with Swift's `async`/`await` model it can immediately start doing other work allowing it to make forward progress on other tasks. The actor executes requests in a serial fashion in the order they are made. This ensures that the shared mutable state is only accessed by the actor. Deadlocks are not possible with the actor model. Asynchronous code that is dependent on a specific operation and resource from an actor can be later resumed once the actor has serviced that request. While deadlocking is not possible, there are other problems actors have such as the actor reentrancy problem where the state of the actor has changed when the executing operation got resumed after a suspension point. + +Mutexes and actors are very different synchronization tools that help protect shared mutable state. While they can both achieve synchronization of that data access, they do so in varying ways that may be desirable for some and undesirable for others. The proposed `Mutex` is yet another primitive that Swift should expose to help those achieve concurrency safe programs in cases where actors aren't suitable. + +## Source compatibility + +Source compatibility is preserved with the proposed API design as it is all additive as well as being hidden behind an explicit `import Synchronization`. Users who have not already imported the Synchronization module will not see this type, so there's no possibility of potential name conflicts with existing `Mutex` named types for instance. Of course, the standard library already has the rule that any type names that collide will disfavor the standard library's variant in favor of the user's defined type anyway. + +## ABI compatibility + +The API proposed here is fully addative and does not change or alter any of the existing ABI. + +`Mutex` as proposed will be a new `@frozen` struct which means we cannot change its layout in the future on ABI stable platforms, namely the Darwin family. Because we cannot change the layout, we will most likely not be able to change to a hypothetical new and improved system mutex implementation on those platforms. If said new system mutex were to share the layout of the currently proposed underlying implementation, then we _may_ be able to migrate over to that implementation. Keep in mind that Linux and Windows are non-ABI stable platforms, so we can freely change the underlying implementation if the platform ever supports something better. + +## Future directions + +There are quite a few potential future directions this new type can take as well as new future similar types. + +### Mutex Guard API + +A token based approach for locking and unlocking may also be highly desirable for mutex API. This is similar to C++'s `std::lock_guard` or Rust's `MutexGuard`: + +```swift +extension Mutex { + public struct Guard: ~Copyable, ~Escapable { + // Hand waving some syntax to borrow Mutex, or perhaps + // we just store a pointer to it. + let mutex: borrow Mutex + + public var value: Value {...} + + deinit { + mutex.unlock() + } + } +} + +extension Mutex { + public borrowing func lock() -> borrow(self) Guard {...} + + public borrowing func tryLock() -> borrow(self) Guard? {...} +} + +func add(_ i: Int, to mutex: Mutex) { + // This acquires the lock by calling the platform's + // underlying 'lock()' primitive. + let mGuard = mutex.lock() + + mGuard.value += 1 + + // At the end of the scope, mGuard is immediately deinitialized + // and releases the mutex by calling the platform's + // 'unlock()' primitive. +} +``` + +The above example shows an API similar to Rust's `MutexGuard` which allows access to the protected state in the mutex. C++'s guard on the other hand just performs `lock()` and `unlock()` for the user (because `std::mutex` doesn't protect any state). Of course the immediate issue with this approach right now is that we don't have access to non-escapable types. When one were to call `lock()`, there's nothing preventing the user from taking the guard value and escaping it from the scope that the caller is in. Rust resolves this issue with lifetimes, but C++ doesn't solve this at all and just declares: + +> The behavior is undefined if m is destroyed before the `lock_guard` object is. + +Which is not something we want to introduce in Swift if it's something we can eventually prevent. If we had this feature today, the primitive `lock()`/`unlock()` operations would be better suited in the form of the guard API. I don't believe we'd have those methods if we had guards. + +### Reader-Writer Locks, Recursive Locks, etc. + +Another interesting future direction is the introduction of new kinds of locks to be added to the standard library, such as a reader-writer lock. One of the core issues with the proposal mutual exclusion lock is that anyone who takes the lock, either a reader and/or writer, must be the only person with exclusive access to the protected state. This is somewhat unfortunate for models where there are infinitely more readers than there will be writers to the state. A reader-writer lock resolves this issue by allowing multiple readers to take the lock and enforces that writers who need to mutate the state have exclusive access to the value. Another potential lock is a recursive lock who allows the lock to be acquired multiple times by the acquired thread. In the same vein, the acquired thread needs to be the one to release the lock and needs to release X amount of times equal to the number of times it acquired it. + +## Alternatives considered + +### Implement `lock()`, `unlock()`, and `tryLock()` + +Seemingly missing from the `Mutex` type are the primitive locking and unlocking functions. These functions are fraught with peril in both Swift's concurrency model and in its ownership model. + +In the face of `async`/`await`, these primitives are very dangerous. The example below highlights incorrect usage of these operations in an `async` function: + +```swift +func test() async { + mutex.lock() // Called on Thread A + await downloadImage(...) // <--- Potential suspension point + mutex.unlock() // May be called on Thread A, B, C, etc. +} +``` + +The potential suspension point may cause the proceeding code to be called on a different thread than the one that initiated the `await` call. We can make these primitives safe in asynchronous contexts though by disallowing their use altogether by marking them `@available(*, noasync)`. Calling `withLock` in an asynchronous function is \_okay\_ because the same thread that calls `lock()` will be the same one that calls `unlock()` because there will not be any suspension points between the calls. + +Another bigger issue is how these functions interact with the ownership model. + +```swift +// borrow access begins +mutex.lock() +// borrow access ends +... +// borrow access begins +mutex.unlock() +// borrow access ends +``` + +In the above, I've modeled where the borrowing accesses occur when calling these functions. This is an important distinction to make because unlike C++'s similar synchronization primitives, `Mutex` (and similarly `Atomic`) can be _moved_. These types guarantee a stable address "for the duration of a borrow access", but as you can see there's nothing guaranteeing that the unlock is occurring on the same address as the call to lock. As opposed to the closure based API (and hopefully in the future the guard based API): + +```swift +// borrow access begins +mutex.withLock { + ... +} +// borrow access ends + +do { + // borrow access begins + let locked = mutex.lock() + ... + + // borrow access ends when 'locked' gets deinitialized +} +``` + +In the first example with the closure, we syntactically define our borrow access with the closure because the entire closure will be executed during the borrow access of the mutex. In the second example, the guarded value we get back from a guard based `lock()` will extend the duration of the borrow access for as long as the `locked` binding is available. + +Providing these APIs on `Mutex` would be incredibly unsafe. We feel that the proposed `withLock` closure based API is much safer and sufficient for most use cases of a mutex. A guard based `lock()` API should cover most of the remaining use cases of needing these bare primitive operations in a much more safer fashion. + +### Rename to `Lock` or similar + +A very common name for this type in various codebases is simply `Lock`. This is a decent name because many people know immediately what the purpose of the type is, but the issue is that it doesn't describe _how_ it's implemented. I believe this is a very important aspect of not only this specific type, but of synchronization primitives in general. Understanding that this particular lock is implemented via mutual exclusion conveys to developers who have used something similar in other languages that they cannot have multiple readers and cannot call `lock()` again on the acquired thread for instance. Many languages similar to Swift have opted into a similar design, naming this type mutex instead of a vague lock. In C++ we have `std::mutex` and in Rust we have `Mutex`. + +### Include `Mutex` in the default Swift namespace (either in `Swift` or in `_Concurrency`) + +This is another intriguing idea because on one hand misusing this type is significantly harder than misusing something like `Atomic`. Generally speaking, we do want folks to reach for this when they just need a simple traditional lock. However, by including it in the default namespace we also unintentionally discouraging folks from reaching for the language features and APIs they we've already built like `async/await`, `actors`, and so much more in this space. Gating the presence of this type behind `import Synchronization` is also an important marker for anyone reading code that the file deals with managing their own synchronization through the use of synchronization primitives such as `Atomic` and `Mutex`. diff --git a/proposals/0434-global-actor-isolated-types-usability.md b/proposals/0434-global-actor-isolated-types-usability.md new file mode 100644 index 0000000000..da254db685 --- /dev/null +++ b/proposals/0434-global-actor-isolated-types-usability.md @@ -0,0 +1,257 @@ +# Usability of global-actor-isolated types + +* Proposal: [SE-0434](0434-global-actor-isolated-types-usability.md) +* Authors: [Sima Nerush](https://github.com/simanerush), [Matt Massicotte](https://github.com/mattmassicotte), [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Upcoming Feature Flag: `GlobalActorIsolatedTypesUsability` +* Review: ([pitch](https://forums.swift.org/t/pitch-usability-of-global-actor-isolated-types/70799)) ([review](https://forums.swift.org/t/se-0434-usability-of-global-actor-isolated-types/71187)) ([acceptance](https://forums.swift.org/t/accepted-se-0434-usability-of-global-actor-isolated-types/72743)) + +## Introduction + +This proposal encompasses a collection of changes to concurrency rules concerning global-actor-isolated types to improve their usability. + +## Motivation + +Currently, there exist limitations in the concurrency model around types that are isolated to global actors. + +First, let's consider the stored properties of `struct`s isolated to global actors. `let` properties of such types are implicitly treated as `nonisolated` within the current module if they have `Sendable` type, but `var` properties are not. This poses a number of problems, such as when implementing a protocol conformance. Currently, the only solution is to declare the property `nonisolated(unsafe)`: + +```swift +@MainActor struct S { + nonisolated(unsafe) var x: Int = 0 +} + +extension S: Equatable { + static nonisolated func ==(lhs: S, rhs: S) -> Bool { + return lhs.x == rhs.x + } +} +``` + +However, there is nothing unsafe about treating `x` as `nonisolated`. The general rule is that concurrency is safe as long as there aren't data races. The type of `x` conforms to `Sendable`, and using a value of `Sendable` type from multiple concurrent contexts shouldn't ever introduce a data race, so any data race involved with an access to `x` would have to be on memory in which `x` is stored. But `x` is part of a value type, which means any access to it is always also an access to the containing `S` value. As long as Swift is properly preventing data races on that larger access, it's always safe to access the `x` part of it. So, first off, there's no reason for Swift to require `(unsafe)` when marking `x` `nonisolated`. + +We can do better than that, though. It should be possible to treat a `var` stored property of a global-actor-isolated value type as *implicitly* `nonisolated` under the same conditions that a `let` property can be. A stored property from a different module can be changed to a computed property in the future, and those future computed accessors may need to be isolated to the global actor, so allowing access across module boundaries would not be okay for source or binary compatibility without an explicit `nonisolated` annotation. But within the module that defines the property, we know that hasn't happened, so it's fine to use a more relaxed rule. + +Next, under the current concurrency rules, it is possible for a function type to be both isolated to a global actor and yet not required to be `Sendable`: + +```swift +func test(globallyIsolated: @escaping @MainActor () -> Void) { + Task { + // error: capture of 'globallyIsolated' with non-sendable type '@MainActor () -> Void' in a `@Sendable` closure + await globallyIsolated() + } +} +``` + +This is not a useful combination: such a function can only be used if the current context is isolated to the global actor, and in that case the global actor annotation is unnecessary because *all* non-`Sendable` functions will run with global actor isolation. It would be better for a global actor attribute to always imply `@Sendable`. + +Because a globally-isolated closure cannot be called concurrently, it's safe for it to capture non-`Sendable` values even if it's implicitly `@Sendable`. Such values just need to be transferred to the global actor's region (if they aren't there already). The same logic also applies to closures that are isolated to a specific actor reference, although it isn't currently possible to write such a closure in a context that isn't isolated to that actor. + + +Finally, the current diagnostic for a global-actor-isolated subclass of a non-isolated superclass is too restrictive: + +```swift +class NotSendable {} + + +@MainActor +class Subclass: NotSendable {} // error: main actor-isolated class 'Subclass' has different actor isolation from nonisolated superclass 'NotSendable' +``` + +Because global actor isolation on a class implies a `Sendable` conformance, adding isolation to a subclass of a non-`Sendable` superclass can circumvent `Sendable` checking: + +```swift +func computeCount() async -> Int { ... } + +class NotSendable { + var mutableState = 0 + func mutate() async { + let count = await computeCount() + mutableState += count + } +} + +@MainActor +class Subclass: NotSendable {} + +func test() async { + let c = Subclass() + await withDiscardingTaskGroup { group in + group.addTask { + await c.mutate() + } + + group.addTask { @MainActor in + await c.mutate() + } + } +} +``` + +In the above code, an instance of `Subclass` can be passed across isolation boundaries because `@MainActor` implies that the type is `Sendable`. However, `Subclass` inherits non-isolated, mutable state from the superclass, so this `Sendable` conformance allows smuggling unprotected shared mutable state across isolation boundaries to create potential for concurrent access. For this reason, the warning about adding isolation to a subclass was added in Swift 5.10, but this restriction could be lifted by instead preventing the subclass from being `Sendable`. + +## Proposed solution + +We propose that: + +- Stored properties of `Sendable` type in a global-actor-isolated value type can be declared as `nonisolated` without using `(unsafe)`. +- Stored properties of `Sendable` type in a global-actor-isolated value type are treated as `nonisolated` when used within the module that defines the property. +- `@Sendable` is inferred for global-actor-isolated functions and closures. +- Global-actor-isolated closures are allowed to capture non-`Sendable` values despite being `@Sendable`. +- A global-actor-isolated subclass of a non-isolated, non-`Sendable` class is allowed, but it must be non-`Sendable`. + + +## Detailed design + + +### Inference of `nonisolated` for `var` properties of globally isolated value types + +Let's look at the first problem with usability of a `var` property of a main-actor-isolated struct: + +```swift +@MainActor +struct S { + var x: Int = 0 // okay ('nonisolated' is inferred within the module) +} + +extension S: Equatable { + static nonisolated func ==(lhs: S, rhs: S) -> Bool { + return lhs.x == rhs.x // okay + } +} +``` + +In the above code, `x` is implicitly `nonisolated` within the module. Under this proposal, `nonisolated` is inferred for in-module access to `Sendable` properties of a global-actor-isolated value type. A `var` with `Sendable` type within a value type can also have an explicit `nonisolated` modifier to allow synchronous access from outside the module. Once added, `nonisolated` cannot later be removed without potentially breaking clients. The programmer can still convert the property to a computed property, but it has to be a `nonisolated` computed property. + +Because `nonisolated` access only applies to stored properties, wrapped properties and `lazy`-initialized properties with `Sendable` type still must be isolated because they are computed properties: + +```swift +@propertyWrapper +struct MyWrapper { ... } + +@MainActor +struct S { + @MyWrapper var x: Int = 0 +} + +extension S: Equatable { + static nonisolated func ==(lhs: S, rhs: S) -> Bool { + return lhs.x == rhs.x // error + } +} +``` + +### `@Sendable` inference for global-actor-isolated functions and closures + +To improve usability of globally-isolated functions and closures, under this proposal `@Sendable` is inferred: + +```swift +func test(globallyIsolated: @escaping @MainActor () -> Void) { + Task { + await globallyIsolated() //okay + } +} +``` + +The `globallyIsolated` closure in the above code is global-actor isolated because it has the `@MainActor` attribute. Because it will always run isolated, it's fine for it to capture and use values that are isolated the same way. It's also safe to share it with other isolation domains because the captured values are never directly exposed to those isolation domains. This means that there's no reason not to always treat these functions as `@Sendable`. + +#### Non-`Sendable` captures in isolated closures + +Under this proposal, globally-isolated closures are allowed to capture non-`Sendable` values: + +```swift +class NonSendable {} + +func test() { + let ns = NonSendable() + + let closure = { @MainActor in + print(ns) + } + + Task { + await closure() // okay + } +} +``` + +The above code is data-race safe, since a globally-isolated closure will never operate on the same instance of `NonSendable` concurrently. + +Note that under region isolation in SE-0414, capturing a non-`Sendable` value in an actor-isolated closure will transfer the region into the actor, so it is impossible to have concurrent access on non-`Sendable` captures even if the isolated closure is formed outside the actor: + +```swift +class NonSendable {} + +func test(ns: NonSendable) async { + let closure = { @MainActor in + print(ns) // error: task-isolated value 'ns' can't become isolated to the main actor + } + + await closure() +} +``` + +### Global actor isolation and inheritance + +Subclasses may add global actor isolation when inheriting from a nonisolated, non-`Sendable` superclass. In this case, an implicit conformance to `Sendable` will not be added, and explicitly specifying a `Sendable` conformance is an error: + +```swift +class NonSendable { + func test() {} +} + +@MainActor +class IsolatedSubclass: NonSendable { + func trySendableCapture() { + Task.detached { + self.test() // error: Capture of 'self' with non-sendable type 'IsolatedSubclass' in a `@Sendable` closure + } + } +} +``` + +Inherited and overridden methods still must respect the isolation of the superclass method: + +```swift +class NonSendable { + func test() { ... } +} + +@MainActor +class IsolatedSubclass: NonSendable { + var mutable = 0 + override func test() { + super.test() + mutable += 0 // error: Main actor-isolated property 'mutable' can not be referenced from a non-isolated context + } +} +``` + +Matching the isolation of the superclass method is necessary because the superclass method implementation may internally rely on the static isolation, such as when hopping back to the isolation after any asynchronous calls, and because there are a variety of ways to call the subclass method that don't preserve its isolation, including: + +* Upcasting to the superclass type +* Erasing to an existential type based on conformances of the superclass type +* Passing the isolated subclass as a generic argument to a type parameter that requires a conformance implemented by the superclass + +## Source compatibility + +This proposal changes the interpretation of existing code that uses global-actor-isolated function types that are not already marked with `@Sendable`. This can cause minor changes in type inference and overload resolution. However, the proposal authors have not encountered any such issues in source compatibility testing, so this proposal does not gate the inference change behind an upcoming feature flag. + +An alternative choice would be to introduce an upcoming feature flag that's enabled by default in the Swift 6 language mode, but this flag could not be enabled by default under `-strict-concurrency=complete` without risk of changing behavior in existing projects that adopt complete concurrency checking. Gating the `@Sendable` inference change behind a separate upcoming feature flag may lead to more code churn than necessary when migrating to complete concurrency checking unless the programmer knows to enable the flags in a specific order. + +## ABI compatibility + +`@Sendable` is included in name mangling, so treating global-actor-isolated function types as implicitly `@Sendable` changes mangling. This change only impacts resilient libraries that use global-actor-isolated-but-not-`Sendable` function types in effectively-public APIs. However, as noted in this proposal, such a function type is not useful, and the proposal authors expect that any API that uses a global-actor-isolated function type either already has `@Sendable`, or should add `@Sendable`. Because the only ABI impact of `@Sendable` is mangling, `@_silgen_name` can be used to preserve ABI in cases where `@Sendable` should be added, and the API is not already `@preconcurrency` (in which case the mangling will strip both the global actor and `@Sendable`). + +## Implications on adoption + +The existing adoption implications of `@Sendable` and global actor isolation adoption apply when making use of the rules in this proposal. For example, `@Sendable` and `@MainActor` can be staged into existing APIs using `@preconcurrency`. See [SE-0337: Incremental migration to concurrency checking](/proposals/0337-support-incremental-migration-to-concurrency-checking.md) for more information. + +## Alternatives considered + +Instead of implicitly suppressing a `Sendable` conformance on isolated subclasses of non-`Sendable`, non-isolated superclasses, the compiler could instead require an explicit opt-out, such as `~Sendable` in the conformance clause. This would make it obvious that the subclass does not have a `Sendable` conformance. However, the programmer does not need to understand that the class does not conform to `Sendable` until they use the type in a way that requires `Sendable`, and the reason for the class not conforming to `Sendable` can be explained with notes attached to the diagnostic. It is also not always the case that global actor isolation implies `Sendable`. Notably, `@MainActor` on a protocol does not imply that the protocol refines `Sendable`, so requiring more boilerplate for programmers in the isolated subclass case does not leave the programmer with a simple rule to remember about when `@MainActor` implies a conformance to `Sendable`. + +## Acknowledgments + +Thank you to Frederick Kellison-Linn for surfacing the problem with global-actor-isolated function types, and to Kabir Oberai for exploring the implications more deeply. diff --git a/proposals/0435-swiftpm-per-target-swift-language-version-setting.md b/proposals/0435-swiftpm-per-target-swift-language-version-setting.md new file mode 100644 index 0000000000..c9c62e089a --- /dev/null +++ b/proposals/0435-swiftpm-per-target-swift-language-version-setting.md @@ -0,0 +1,54 @@ +# Swift Language Version Per Target + +* Proposal: [SE-0435](0435-swiftpm-per-target-swift-language-version-setting.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-swiftpm-swift-language-version-per-target/71067)) ([review](https://forums.swift.org/t/se-0435-swift-language-version-per-target/71546)) ([acceptance](https://forums.swift.org/t/accepted-se-0435-swift-language-version-per-target/71846)) + +## Introduction + +The current Swift Package Manager manifest API for specifying Swift language version(s) applies to an entire package which is limiting when adopting new language versions that have implications for source compatibility. + +Swift-evolution thread: [Pitch: [SwiftPM] Swift Language Version Per Target](https://forums.swift.org/t/pitch-swiftpm-swift-language-version-per-target/71067) + +## Motivation + +Adopting new language versions at the target granularity allows for gradual migration to prevent possible disruptions. Swift 6, for example, turns on strict concurrency by default, which can have major implications for the project in the form of new errors that were previously downgraded to warnings. SwiftPM should allow to specify a language version per target so that package authors can incrementally transition their project to the newer version. + +## Proposed solution + +Add a new Swift target setting API, similar to `enable{Upcoming, Experimental}Feature`, to specify a Swift language version that should be used to build the target, if such version is not specified, fallback to the current language version determination logic. + +## Detailed design + +Add a new `swiftLanguageVersion` API to `SwiftSetting` limited to manifests >= 6.0: + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.0) + public static func swiftLanguageVersion( + _ version: SwiftVersion, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting { + ... + } +} +``` + +## Security + +New language version setting has no implications on security, safety or privacy. + +## Impact on existing packages + +Since this is a new API, all existing packages will use the default behavior - version specified at the package level when set or determined based on the tools version. + +## Alternatives considered + +- Add a new setting for 'known-safe' flags, of which `-swift-version` could be the first. This seems less user-friendly and error prone than re-using `SwiftLanguageVersion`, which has known language versions as its cases (with a plaintext escape hatch when necessary). + +- Add new initializer parameter to `Target` API and all of the convenience static functions, this is less flexible because it would require a default value. + diff --git a/proposals/0436-objc-implementation.md b/proposals/0436-objc-implementation.md new file mode 100644 index 0000000000..517599b667 --- /dev/null +++ b/proposals/0436-objc-implementation.md @@ -0,0 +1,348 @@ +# Objective-C implementations in Swift + +* Proposal: [SE-0436](0436-objc-implementation.md) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.1)** +* Implementation: [swiftlang/swift#73309](https://github.com/swiftlang/swift/pull/73309), [swiftlang/swift#74801](https://github.com/swiftlang/swift/pull/74801) +* Review: ([first pitch](https://forums.swift.org/t/pitch-objective-c-implementations-in-swift/61907)) ([second pitch](https://forums.swift.org/t/pitch-2-objective-c-implementations-in-swift/68090)) ([third pitch](https://forums.swift.org/t/pitch-3-objective-c-implementations-in-swift/71315)) ([review](https://forums.swift.org/t/se-0436-objective-c-implementations-in-swift/71712)) ([acceptance](https://forums.swift.org/t/accepted-se-0436-objective-c-implementations-in-swift/72053)) + +## Introduction + +We propose an alternative to `@objc` classes where Objective-C header `@interface` declarations are implemented by Swift `extension`s marked with `@objc @implementation`. The resulting classes will be implemented in Swift, but will be indistinguishable from Objective-C classes, fully supporting Objective-C subclassing and runtime trickery. + +Swift-evolution thread: [first pitch](https://forums.swift.org/t/pitch-objective-c-implementations-in-swift/61907), [second pitch](https://forums.swift.org/t/pitch-2-objective-c-implementations-in-swift/68090), [third pitch](https://forums.swift.org/t/pitch-3-objective-c-implementations-in-swift/71315) + +## Motivation + +Swift has always had a mechanism that allows Objective-C code to use Swift types: the `@objc` attribute. When a class is marked with `@objc` (or, more typically, inherits from an `@objc` or imported Objective-C class), Swift generates sufficient Objective-C metadata to allow it to be used through the Objective-C runtime, and prints a translated Objective-C declaration into a generated header file that can be imported into Objective-C code. The same goes for members of the class. + +This feature works really well for mixed-language apps and project-internal frameworks, but it's poorly suited to exposing private and especially public APIs to Objective-C. There are three key issues: + +1. To avoid circularity while building the Swift half of the module, the generated header cannot be included into other headers in the same module, which can make it difficult to use the Swift-implemented parts of the API in the Objective-C-implemented parts. Worse, some build systems install the headers for all modules and then build binaries for them out of order; generated headers can't really be used across modules in these systems. + +2. Objective-C programmers expect API headers to serve as a second source of documentation on the APIs, but generated headers are disorganized, unreadable messes because Swift cannot mechanically produce the formatting that a human engineer would add to a handwritten header. + +3. While `@objc` classes can be *used* from Objective-C, they are not truly Objective-C types. They still contain Swift vtables and other Swift-specific data that the Objective-C compiler and runtime don't fully understand. This limits their capabilities—for instance, Objective-C code cannot subclass an `@objc` class or reliably swizzle its methods. + +Together, these issues make it very hard for frameworks with a lot of Objective-C clients to implement their functionality in Swift. If they have classes that are meant to be subclassed, it's actually impossible to fully port them to Swift, because it would break existing Objective-C subclasses. And yet the trade-offs made by `@objc` are really good for the things it's designed for, like writing custom views and view controllers in Swift app targets. We don't want to radically change the existing `@objc` feature. + +Swift also quietly supports a hacky pseudo-feature that allows a different model for Objective-C interop: It will not diagnose a selector conflict if a Swift extension redeclares members already imported from Objective-C, so you can declare a method or property in a header and then implement it in a Swift extension. However, this feature has not really been designed to work properly; it doesn't check that your implementation's name and signature match the header, there's no protection against forgetting to implement a method, and you still need an `@implementation` for the class metadata itself. Nevertheless, a few projects use this and find it helpful because it avoids the issues with normal interop. Formalizing and improving this pattern seems like a promising direction for Objective-C interop. + +## Proposed solution + +We propose adding a new attribute, `@implementation`, which, when paired with an interop attribute like `@objc`, tells Swift that it is to implement a declaration it has imported from another language, rather than creating a new declaration and exporting it *to* that language. + +Specifically, in this proposal, `@objc @implementation` allows a Swift `extension` to replace an Objective-C `@implementation` block. You write headers as normal for an Objective-C class, but instead of writing an `@implementation` in an Objective-C file, you write an `@objc @implementation extension` in a Swift file. You can even port an existing class’s implementation to Swift one category at a time without breaking backwards compatibility. + +For instance, if you were adding a new class, you would start by writing a normal Objective-C header, as though you were planning to implement the class in an Objective-C .m file: + +```objc +#import + +NS_HEADER_AUDIT_BEGIN(nullability, sendability) + +@interface MYFlippableViewController : UIViewController + +@property (strong) UIViewController *frontViewController; +@property (strong) UIViewController *backViewController; +@property (assign,getter=isShowingFront) BOOL showingFront; + +- (instancetype)initWithFrontViewController:(UIViewController *)front backViewController:(UIVIewController *)back; + +@end + +@interface MYFlippableViewController (Animation) + +- (void)setShowingFront:(BOOL)isShowingFront animated:(BOOL)animated NS_SWIFT_NAME(setIsShowingFront(_:animated:)); +- (void)setFrontViewController:(UIViewController *)front animated:(BOOL)animated; +- (void)setBackViewController:(UIViewController *)back animated:(BOOL)animated; + +@end + +@interface MYFlippableViewController (Actions) + +- (IBAction)flip:(id)sender; + +@end + +NS_HEADER_AUDIT_END(nullability, sendability) +``` + +And you would arrange for Swift to import it through an umbrella or bridging header. You would then write an `extension` for each `@interface` you wish to implement in Swift. For example, you could implement the main `@interface` (plus any visible class extensions) in Swift by writing: + +```swift +@objc @implementation extension MYFlippableViewController { + ... +} +``` + +And the `Animation` category by writing: + +```swift +@objc(Animation) @implementation extension MYFlippableViewController { + ... +} +``` + +Note that there is nothing special in the header which indicates that a given `@interface` is implemented in Swift. The header can use all of the usual Swift annotations—like `NS_SWIFT_NAME`, `NS_NOESCAPE` etc.—but they simply affect how the member is imported. Swift does not even require you to implement every declared `@interface` in Swift, so you can implement some parts of a class in Objective-C and others in Swift. But if you choose to implement a particular `@interface` in Swift, each Objective-C member in that `@interface` must be matched by a Swift declaration in the extension that has the same Swift name; these special members are called "member implementations". + +Specifically, member implementations must be non-`final`, not be overrides, and have an `open`, `public`, `package`, or `internal` access level. Every member implementation must match a member declared in the Objective-C header, and every member declared in the Objective-C header must have a matching member implementation. This ensures that everything declared by the header is correctly implemented without any accidental misspellings or type signature mismatches. + +In addition to member implementations, an `@objc @implementation` extension can also contain three other kinds of members: + +1. **Fileprivate or private non-`final` members** are helper methods (think `@IBAction`s or callback selectors). They must *not* match a member from the imported headers, but they are accessible from Objective-C by performing a selector or declaring them in a place that is not visible to Swift. (Objective-C `@implementation` blocks often declare private members like this, so it's helpful to allow them in `@objc @implementation` extensions too.) + +2. **Members with an `override` modifier** override superclass members and function normally. (Again, Objective-C `@implementation` blocks often override superclass members without declaring the override in their headers, so it's helpful to allow `@objc @implementation` extensions to do the same.) + +3. **Members with a `final` modifier (or `@nonobjc` on an initializer)** are Swift-only and can use Swift-only types or features. These may be Swift-only implementation details (if `internal` or `private`) or Swift-only APIs (if `public` or `package`). (This feature is necessary to support stored properties containing Swift-only types because ordinary extensions cannot declare stored properties; in theory, we could require other `final` members to be declared in a separate ordinary extension, but we also allow them in an `@objc @implementation` extension as a convenience.) + +Within an `@objc @implementation` extension, `final` members are `@nonobjc` by default. + +As a special exception to the usual rule, a non-category `@objc @implementation extension` can declare stored properties and other members that are normally only allowed in the main `class` body. They can be (perhaps implicitly) `@objc` or they can also be `final`; in the latter case they are only accessible from Swift. Note that `@implementation` does not have an equivalent to Objective-C's implicit `@synthesize`—you must declare a `var` explicitly for each `@property` in the header that you want to be backed by a stored property. + +## Detailed design + +### Custom category names + +As an enabling step, we propose that using `@objc(CustomName)` on an `extension` should cause the Objective-C category created for that extension to be named `CustomName`, rather than using a category name generated by the compiler. This name should appear in both generated headers and Objective-C metadata. The compiler should enforce that there is only one extension per class/category-name combination. + +`@objc(CustomName) extension` also has the effect of `@objc extension`, namely, making members of the extension `@objc` by default. + +### `@implementation` + +The compiler will accept a new attribute, `@implementation`, which turns a declaration that would normally be exported to another language into a declaration that implements something imported from another language. This attribute takes no arguments; it should be used alongside another attribute specifying the foreign language, such as `@objc`. + +In general, `@implementation` will cause the affected declarations to be matched against imported declarations, and will emit errors if a good enough match cannot be found. It cannot necessarily be applied to all declarations that can be exported to a foreign language; for instance, `@objc @implementation` is only supported on whole extensions, not on individual members. + +### `@objc @implementation extension`s + +`@objc @implementation extension` causes an extension to be used as the implementation of an Objective-C `@interface` declaration for the class it extends. If the `@objc` attribute specifies a custom category name, it will implement a category with that name; otherwise it will implement the main `@interface` for the class (plus any class extensions). + +```swift +@objc @implementation extension SomeClass { + // Equivalent to `@implementation SomeClass`; + // implements everything in `@interface SomeClass` and + // all `@interface SomeClass ()` extensions. +} + +@objc(SomeCategory) @implementation extension SomeClass { + // Equivalent to `@implementation SomeClass (SomeCategory)`; + // implements everything in `@interface SomeClass (SomeCategory)`. +} +``` + +As in any `@objc extension`, all members are implicitly `@objc` by default, and all `@objc` members are implicitly `dynamic`; unlike other `@objc extension`s, adding the `final` keyword makes the declaration *not* `@objc` by default. + +As a special exception to the usual rule, an `@objc @implementation extension` which implements the main Objective-C interface of a class can declare stored properties, designated and required `init`s, and `deinit`s. + +#### Rules + +An `@objc @implementation extension` must: + +* Extend a non-root class imported from Objective-C which does not use lightweight generics. +* If a category name is present, match a category by that name for that class (if no category name is present, the extension matches the main interface). +* Provide a member implementation (see below) for each member of the `@interface` it implements. +* Not declare conformances. (Conformances should be declared in the header if they are for Objective-C protocols, or in an ordinary extension otherwise.) +* Contain only `@objc`, `override`, `final`, or (for initializers) `@nonobjc` members. (Note that member implementations are implicitly `@objc`, as mentioned below, so this effectively means that non-`override`, non-`final`, non-`@nonobjc` members *must* be member implementations.) +* `@nonobjc` initializers must be convenience initializers, not designated or required initializers. + +> **Note**: `@objc @implementation` cannot support Swift-only designated and required initializers because subclasses with additional stored properties must be able to override designated and required initializers, but `@implementation` only supports overriding of `@objc` members. The Swift-only metadata that would be used for dynamic dispatch in an ordinary `@objc` class is not present in an `@implementation` class. + +### Member implementations + +Any non-`override` open, public, package, or internal `@objc` member of an `@objc @implementation extension` is a “member implementation”; that is, it implements some imported Objective-C member of the class it is extending. Member implementations are special because much of the compiler completely ignores them: + +* Access control denies access to member implementations in most contexts. +* Module interfaces and generated interfaces do not include member implementations. +* Objective-C generated headers do not include member implementations. + +This means that calls in expressions will *always* ignore the member implementation and use the imported Objective-C member instead. In other words, even other Swift code in the same module will behave as though the member is implemented in Objective-C. + +Some members cannot be implemented in an `@objc @implementation` extension because `@objc` does not support some of the features they use; see "Future Directions" for more specific discussion of this. These members will have to be moved to a separate category and implemented in Objective-C. + +#### Rules + +A member implementation must: + +* Have the same Swift name as the member it implements. +* Have the same selector as the member it implements. +* Have the same foreign error convention and foreign async convention as the member it implements. +* Not have other traits, like an overload signature, `@nonobjc`/`final` attribute, `class` modifier, or mutability, which is incompatible with the member it implements. +* Not have `@_spi` attributes (they would be pointless since the visibility of the imported Objective-C attribute is what will make the member usable or not). + +Both the Swift name and the Objective-C selector of a member implementation must match the corresponding Objective-C declaration; Swift will diagnose an error if one matches but the other doesn't. This checking respects both the `@objc(custom:selector:)` in Swift implementations and the Swift name (`NS_SWIFT_NAME(custom(_:name:))`) attribute in Objective-C headers. + +Member implementations must have an overload signature that closely matches the Objective-C declaration’s. However, types that are non-optional in the Objective-C declaration may be implicitly unwrapped optionals in the member implementation if this is ABI-compatible; this is because Objective-C does not prevent clients from passing `nil` or implementations from returning `nil` when `nonnull` is used, and member implementations may need to implement backwards compatibility logic for this situation. + +### Objective-C metadata generation + +When Swift generates metadata for an `@objc @implementation extension`, it will generate metadata that matches what clang would have generated for a similar `@implementation`. That is: + +* `@objc` members will only have Objective-C metadata, not Swift metadata. (`final` members may require Swift metadata.) +* If the extension is for the main class interface, it will generate complete Objective-C class metadata with an ivar for each stored property, and without setting the Swift bit or using any features incompatible with clang subclasses or categories. + +## Source compatibility + +These changes are additive and don't affect existing code. Replacing an Objective-C `@implementation` declaration with a Swift `@objc @implementation extension` is invisible to the library's Objective-C and Swift clients, so it should not be source-breaking unless the implementations have observable behavior differences. + +Previous versions of Swift have accepted the `@objc(CustomName) extension` syntax and simply ignored the custom name, so merely adding a custom category name won't break source compatibility. + +## Effect on ABI stability + +All `@objc` members of an `@implementation extension`—member implementation or otherwise—have the ABI of an `@objc dynamic` member, so turning one into the other is not ABI-breaking. `@implementation extension` classes generate only Objective-C metadata, not Swift metadata, so existing Objective-C subclasses will continue to function as normal. + +Because `@implementation` attributes and member implementations are not printed into module interfaces, this proposal has no direct effect on Swift ABI stability. + +## Implications on adoption + +`@implementation` extensions that implement categories are back-deployable to Swift 5.0 runtimes and later, and many `@implementation` extensions that implement classes are too. However, if a class's ivar layout cannot be computed at compile time, that class will require new runtime support and will not be back-deployable to old platforms. + +Affected classes are ones whose stored properties contain a non-frozen enum or struct imported from another module that has library evolution enabled. (This property is transitive—if your stored properties contain a struct in your own module, but that struct has a stored property of an affected type, that also limits back deployment.) In practice, it is usually possible to work around this problem by boxing affected values in a class or existential, at the cost of some overhead. + +> **Note**: Some of the required runtime changes are in the Objective-C runtime, so even a development toolchain will not be sufficient to actually run modules with affected classes. However, you can test the diagnostics and code generation by compiling with the experimental feature flag `ObjCImplementationWithResilientStorage`; OS version 99.99 will be treated as high enough to have the necessary runtime support. + +## Future directions + +### Extending `@objc` capabilities to extend `@objc @implementation` capabilities + +`@objc @implementation` extensions cannot implement `@interface` members that cannot be created by `@objc`. The most notable limitations include: + +* Factory convenience initializers (those implemented as class methods, like `+[NSString stringWithCharacters:length:]`). +* `__attribute__((objc_direct))` methods and `@property (direct)` properties. +* Members with nonstandard memory management behavior, even if it is correctly annotated. +* Members which deviate from the Objective-C error convention in certain subtle ways, such as by having the `NSError**` parameter in the wrong place. + +Additionally, `@objc @implementation` cannot implement global declarations that cannot be created by `@objc`, such as: + +* Free functions, global variables, cases of `NS_TYPED_ENUM` typedefs, and other non-member Objective-C declarations. +* Global declarations imported as members of a type using `NS_SWIFT_NAME`'s import-as-member capabilities. + +`@objc @implementation` heavily piggybacks on `@objc`'s code emission, so in most cases, the best approach to expanding `@implementation`'s support would be to extend `@objc` to support the feature and then make sure `@objc @implementation` supports it too. + +### `@implementation` for plain C declarations + +Many of the capabilities mentioned as future directions for `@objc` would also be useful for plain-C clients, including those on non-Darwin platforms. Once again, the best approach here would probably be to stabilize and extend something like `@_cdecl` to support creating these with a generated header, and then make sure `@implementation` supports this attribute too. + +The compiler currently has experimental support for `@_cdecl @implementation` for global functions; it's behind a separate experimental feature flag because it's not part of this proposal. + +### `@implementation` for C++ declarations + +One could similarly imagine a C++ version of this feature: + +```cpp +// C++ header file + +class CppClass { +private: + int myStorage = 0; +public: + int someMethod() { ... } + int swiftMethod(); +}; +``` + +```swift +// Swift implementation file + +@_expose(Cxx) @implementation extension CppClass { + func swiftMethod() -> Int32 { return self.myStorage } +} +``` + +This would be tricker than Objective-C interop because for Objective-C interop, Swift already generates machine code thunks directly in the binary, whereas for C++ interop, Swift generates C++ source code thunks in a generated header. Swift could either compile this generated code internally, or it could emit it to a file and expect the build system to build and link it. + +We believe that there wouldn't be a problem with sharing the `@implementation` attribute with this feature because `@implementation` is always paired with a language-specific attribute. + +### Supporting lightweight generics + +Classes using Objective-C lightweight generics have type-erased generic parameters; this imposes a lot of tricky limitations on Swift extensions of these classes. Since very few classes use lightweight generics, we have chosen to ban the combination for now. If there turns out to be a lot of demand for implementing Objective-C generic classes in Swift, we can lift the ban after we've figured out how to make the combination more usable. + +### `@objc @implementation(unchecked)` + +One could imagine an option that disables `@objc @implementation`'s exhaustiveness checking so that Swift implementations can use dynamic mechanisms like `+instanceMethodForSelector:` to create methods at runtime. This change would be purely additive, so we can consider it if there's demand. + +### Implementation-only bridging header + +This feature would work extremely well with a feature that allowed Swift to import an implementation-only bridging header alongside the umbrella header when building a public framework. This would not only give the Swift module access to internal Objective-C declarations, but also allow it to implement those declarations. However, the two features are fully orthogonal, so I’ll leave that to a different proposal. + +### Improvements to private Objective-C modules + +This feature would also work very well with some improvements to private Objective-C modules: + +1. The Swift half of a mixed-source framework could implicitly import the private Clang module with `internal`; this would allow you to easily provide implementations for Objective-C-compatible SPI. +2. We could perhaps set up some kind of equivalence between `@_spi` and private Clang modules so that `final` Swift members could be made public. + +Again, that’s something we can flesh out over time. + +## Alternatives considered + +### A different attribute spelling + +We've chosen the proposed spelling—`@objc(CategoryName) @implementation`—because it makes `@implementation` orthogonal to the specific language being implemented. Everything Objective-C-specific about it is tied to the `@objc` attribute, and it's pretty clear how it would be expanded in the future to support other languages. Many alternatives—such as the original pitch's `@objcImplementation(CategoryName)` and suggestions like `@implementation(objc, category: CategoryName)`—do not have this property. + +We chose the word "implementation" rather than "extern" because the attribute marks an implementation of something that was declared elsewhere; in most languages, "extern" works in the opposite direction, marking a declaration of something that is implemented elsewhere. Also, the name suggests a relationship to the `@implementation` keyword in Objective-C, which is appropriate since an `@objc @implementation extension` is a one-to-one drop-in replacement. + +`@implementation` has a similar spelling to the compiler-internal `@_implements` attribute, but there has been little impetus to stabilize `@_implements` in the seven years since it was added, and if it ever *is* stabilized it wouldn't be unreasonable to make it a variant of `@implementation` (e.g. `@implementation(forRequirement: SomeProto.someMethod)`). + +### Allow/require `@objc @implementation` extensions to declare conformances + +Rather than banning `@objc @implementation` extensions from declaring conformances, we could require them to re-declare conformances listed in the header and/or allow them to add additional conformances. This would more closely align with our design decisions about members, where we allow private members and overrides because Objective-C `@implementation` allows them. + +Unfortunately, every design here has at least one wart or inconsistency. There are actually four different alternatives here; let's tackle them separately: + +1. **Re-declares header conformances** + **Cannot declare extra conformances**: The conformances here are pure boilerplate; they duplicate what's in the header without any opportunity to add more information. By contrast, we force the extension to re-declare members because the redeclarations *add more information* (like the body), or at least they have the opportunity to add more information. It seems wasteful to require the developer to keep two conformance lists exactly in sync. + +2. **Doesn't re-declare header conformances** + **Can declare extra conformances**: The conformances behave very differently from the members. With members, you are *required* to declare what's in the header but also *allowed* to add certain kinds of extra members. Allowing a conformance list but requiring it to *not* include what's in the header is unintuitive. + +3. **Re-declares header conformances** + **Can declare extra conformances**: Unlike with members—where the additional members are made textually obvious by a modifier like `private` or `final`—there would be no visual distinction between re-declared conformances and extra conformances. That's unfortunate because their visibility is very different. The extra conformances would only be visible to clients which have imported the .swiftinterface, .swiftmodule, or -Swift.h files, and clients often don't know exactly which files they are importing from a module. Having a single, combined list of things which have invisible distinctions between them would be confusing. + +4. **Doesn't re-declare header conformances** + **Cannot declare extra conformances** (the design we selected): Does not map 1:1 to the behavior of `@implementation` in Objective-C (which can declare additional conformances). + +Since all of these options have unappealing aspects, we have chosen the simplest one with the smallest potential for mistakes, which is banning all use of the conformance list. It is still possible to write an ordinary extension which adds conformances to an `@objc @implementation` class; these conformances will have the same visibility behavior as extra conformances would, but putting them in a separate, ordinary extension creates a visible distinction to make this obvious. + +### `@implementation class` + +We considered using a `class` declaration, not an `extension`, to implement the main body of a class: + +```swift +// Header as above + +#if canImport(UIKit) +import UIKit +#else +import SwiftUIKitClone // hypothetical pure-Swift UIKit for non-Darwin platforms +#endif + +#if OBJC +@objc @implementation +#endif +class MYFlippableViewController: UIViewController { + var frontViewController: UIViewController { + didSet { ... } + } + var backViewController: UIViewController { + didSet { ... } + } + var isShowingFront: Bool { + didSet { ... } + } + + init(frontViewController: UIViewController, backViewController: UIViewController) { + ... + } +} + +#if OBJC +@objc(Animation) @implementation +#endif +extension MYFlippableViewController { + ... +} +``` + +This might be able to reduce code duplication for adopters who want to write cross-platform classes that only use `@implementation` on platforms which support Objective-C, and it would also mean we could remove the stored-property exception. However, it is a significantly more complicated model—there is much more we'd need to hide and a lot more scope for the `class` to have important mismatches with the `@interface`. And the reduction to code duplication would be limited because pure-Swift extension methods are non-overridable, so all methods you wanted clients to be able to override would have to be listed in the `class`. This means that in practice, mechanically generating pure-Swift code from the `@implementation`s might be a better approach. + +## Acknowledgments + +Doug Gregor gave a *ton* of input into this design; Allan Shortlidge reviewed much of the code; and Mike Ash provided timely help with a tricky Objective-C metadata issue. Thanks to all of them for their contributions, and thanks to all of the engineers who have provided feedback and bug reports on this feature in its experimental state. diff --git a/proposals/0437-noncopyable-stdlib-primitives.md b/proposals/0437-noncopyable-stdlib-primitives.md new file mode 100644 index 0000000000..ddef3d2c7c --- /dev/null +++ b/proposals/0437-noncopyable-stdlib-primitives.md @@ -0,0 +1,1760 @@ +# Noncopyable Standard Library Primitives + +* Proposal: [SE-0437](0437-noncopyable-stdlib-primitives.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Roadmap: [Improving Swift performance predictability: ARC improvements and ownership control][Roadmap] +* Review: ([pitch](https://forums.swift.org/t/pitch-noncopyable-standard-library-primitives/71566)) ([review](https://forums.swift.org/t/se-0437-generalizing-standard-library-primitives-for-non-copyable-types/72020)) ([acceptance](https://forums.swift.org/t/accepted-se-0437-generalizing-standard-library-primitives-for-non-copyable-types/72275)) + +[Roadmap]: https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206 + +Related proposals: + +- [SE-0377] `borrowing` and `consuming` parameter ownership modifiers +- [SE-0390] Noncopyable structs and enums +- [SE-0426] BitwiseCopyable +- [SE-0427] Noncopyable generics +- [SE-0429] Partial consumption of noncopyable values +- [SE-0432] Borrowing and consuming pattern matching for noncopyable types + +[SE-0377]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md +[SE-0390]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md +[SE-0426]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0426-bitwise-copyable.md +[SE-0427]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md +[SE-0429]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0429-partial-consumption.md +[SE-0432]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0432-noncopyable-switch.md + +### Table of Contents + + * [Motivation](#motivation) + * [Low\-level memory management](#low-level-memory-management) + * [Generalized optional types](#generalized-optional-types) + * [Proposed Solution](#proposed-solution) + * [Generalizing function members](#generalizing-function-members) + * [Generalizing higher\-order functions](#generalizing-higher-order-functions) + * [Generalizing caller\-provided return types](#generalizing-caller-provided-return-types) + * [(Lack of) protocol generalizations](#lack-of-protocol-generalizations) + * [Unblocking basic construction work](#unblocking-basic-construction-work) + * [Detailed Design](#detailed-design) + * [protocol ExpressibleByNilLiteral](#protocol-expressiblebynilliteral) + * [enum Optional](#enum-optional) + * [enum Result](#enum-result) + * [enum MemoryLayout](#enum-memorylayout) + * [Unsafe Pointer Types](#unsafe-pointer-types) + * [Member generalizations](#member-generalizations) + * [Related enhancements in other types](#related-enhancements-in-other-types) + * [Setting temporary pointers on arbitrary entities](#setting-temporary-pointers-on-arbitrary-entities) + * [Unsafe Buffer Pointers](#unsafe-buffer-pointers) + * [Member generalizations](#member-generalizations-1) + * [Protocol conformances](#protocol-conformances) + * [Extracting parts of a buffer pointer](#extracting-parts-of-a-buffer-pointer) + * [Exceptions](#exceptions) + * [Related enhancements in other types](#related-enhancements-in-other-types-1) + * [Temporary buffer pointers over arbitrary entities](#temporary-buffer-pointers-over-arbitrary-entities) + * [Temporary Allocation Facility](#temporary-allocation-facility) + * [Managed Buffers](#managed-buffers) + * [Lifetime Management](#lifetime-management) + * [Swapping and exchanging items](#swapping-and-exchanging-items) + * [Source compatibility](#source-compatibility) + * [ABI compatibility](#abi-compatibility) + * [Alternatives Considered](#alternatives-considered) + * [Omitting UnsafeBufferPointer](#omitting-unsafebufferpointer) + * [Alternatives to UnsafeBufferPointer\.extracting()](#alternatives-to-unsafebufferpointerextracting) + * [Future Work](#future-work) + * [Non\-escapable Optional and Result](#non-escapable-optional-and-result) + * [Generalizing higher\-order functions](#generalizing-higher-order-functions-1) + * [Generalizing Optional\.unsafelyUnwrapped](#generalizing-optionalunsafelyunwrapped) + * [Generalized managed buffer headers](#generalized-managed-buffer-headers) + * [Additional raw pointer operations](#additional-raw-pointer-operations) + * [Protocol generalizations](#protocol-generalizations) + * [Additional future work](#additional-future-work) + * [Appendix: struct Hypoarray](#appendix-struct-hypoarray) + +## Motivation + +[SE-0427] allowed noncopyable types to participate in Swift generics, and introduced the protocol `Copyable` to the Standard Library. However, it stopped short of adapting the Standard Library to support using such constructs. + +The expectation that everything is copyable has been a crucial simplifying assumption throughout all previous API design work in Swift. It allowed and encouraged us to define and use interfaces without having to think too deeply about who is responsible for owning the entities we pass between functions; it let us define convenient container types with implicit copy-on-write value semantics; it has been a constant, familiar, friendly companion of every Swift programmer for almost a decade. Fully rethinking the Standard Library to facilitate working with noncopyable types is not going to happen overnight: it is going to take a series of proposals. This document takes the first step by focusing on an initial set of core changes that will enable building simple generic abstractions using noncopyable types. + +To achieve this, we need to tweak some core parts of the Standard Library to start eliminating the assumption of copyability. The changes proposed here only affect a small subset of the Standard Library's API surface; much more work remains to be done. But these changes are intended to be enough to let us start using Swift's new ownership control features in earnest, so that we can use them to solve real problems, but also so that we can gain crucial experience that will inform subsequent Standard Library work. + +This proposal concentrates on two particular areas: low-level memory management and generalized optional types. We propose to modify some preexisting generic constructs in the Standard Library to eliminate the assumption of copyability. Such a retroactive generalization is unlikely to be the right approach for every construct (especially not for copy-on-write container types), but it is the appropriate choice for these particular abstractions. + +### Low-level memory management + +First, we need to extend the existing low-level unsafe pointer operations to allow managing memory that holds noncopyable entities. + +- We need to teach `MemoryLayout` how to provide basic information on the memory layout of noncopyable types. +- `UnsafePointer` and `UnsafeMutablePointer` need to support noncopyable pointees. The existing pointer operations need to support working with such instances. This includes heap allocations, pointer conversions and comparisons, operations that bind or rebind raw memory, that initialize/deinitialize memory, etc. +- Similarly, `UnsafeBufferPointer` and `UnsafeMutableBufferPointer` must learn to support noncopyable elements. +- We need the standard low-level memory management facilities to allow working with noncopyable types: + - Scoped pointer-based access to arbitrary entities (`func withUnsafePointer(to:)`) + - Unmanaged heap memory allocations (`UnsafeMutablePointer.allocate`, `UnsafeMutableBufferPointer.allocate`) + - Managed tail-allocated storage allocations (`class ManagedBuffer`, `struct ManagedBufferPointer`). + - Allocating a temporary buffer (`func withUnsafeTemporaryAllocation`) + +Generalizing these constructs for noncopyable types does not fundamentally change their nature -- an `UnsafePointer` to a noncopyable pointee is still a regular, copyable pointer, conforming to much the same protocols as before, and providing many of the same operations. For example, given this simple noncopyable type `Foo`: + +```swift +struct Foo: ~Copyable { + var value: Int + mutating func increment() { value += 1 } +} +``` + +We want to be able to dynamically allocate memory for instances of `Foo`, and use the familiar pointer operations we've already learned while working with copyable types: + +```swift +let p = UnsafeMutablePointer.allocate(capacity: 2) +let q = p + 1 +p.initialize(to: Foo(value: 42)) +q.initialize(to: Foo(value: 23)) +p.pointee.increment() +print(p.pointee.value) // Prints "43" +print(p[0].value, p[1].value) // Prints "43 23" +print(p < q) // Prints "true" +let foo = p.move() +q.deinitialize(count: 1) +p.deallocate() +``` + +Most of the core pointer operations were already (implicitly) defined in terms of ownership control, and so they readily translate into noncopyable use. + +Of course, not all operations can be generalized: for example, `p.initialize(repeating: Foo(7), count: 2)` cannot possibly work, as repeating an item inherently requires making copies of it. That's not a problem though: we can continue to have such operations require a copyable pointee. + + +### Generalized optional types + +The second area that requires immediate attention is the `Optional` enumeration and its close sibling, `Result`. `Optional` is particularly frequently used in the definition of programming interfaces: it is the standard way a Swift function can take or return a potentially absent value. It is also deeply integrated into the language itself: features such as optional chaining, failable initializers and `try?` statements all rely on it, and we need these features to work even in noncopyable contexts. + +It is therefore very much desirable for optionals to start supporting noncopyable payloads, so that Swift functions can continue to use these well-known types in their interface definitions. + +Instances of `Optional` and `Result` directly contain the items they wrap. Therefore, an optional wrapping a noncopyable type will necessarily need to become noncopyable itself. This is a far more radical change than generalizing a pointer type: it means these enumerations must turn into conditionally copyable types. + +```swift +enum Optional: ~Copyable { + case none + case some(Wrapped) +} + +extension Optional: Copyable where Wrapped: Copyable {} +``` + +This is no small matter -- every existing use of `Optional` implicitly assumes its copyability, including all its protocol conformances. We need to lift this assumption without breaking source- and (on some platforms) binary compatibility with existing code that relies on it. + +Furthermore, compatibility expectations also go in the reverse direction, as `Optional` has been an unavoidable part of Swift since its initial release. On ABI stable platforms, we therefore also expect that code freshly built with this newly generalized `Optional` type will continue to be able to run on older versions of the Swift Standard Library. At minimum, we expect that all code that uses copyable types would be directly back-deployable. + +Allowing noncopyable use will necessarily involve defining new operations to help dealing with problems that are specific to noncopyable types. However, when doing that, we need to balance the need to help developers who embrace ownership control with the desire to avoid confusing folks when they continue relying on copyability. (These aren't necessarily different groups of people -- we expect developers will often find it useful to generally stay with copyable abstractions, only reaching for ownership control in specific parts of their code.) Retrofitting noncopyable support on existing types risks muddling up their semantics, hurting our desire for progressive disclosure and potentially overwhelming newcomers. + +However, in the particular case of `Optional`, the benefits of making copyability conditional greatly outweigh these drawbacks. Indeed, we don't have much choice but to retrofit `Optional`: we need a common idiom for representing a potentially absent item, shared across all contexts throughout the language. + +For example, introducing a separate version of `Optional` that's dedicated to noncopyable use would not be workable. This would prevent generic functions that want to support noncopyable type arguments from taking or returning "classic" optionals, so generic code would quickly standardize on using the new type, while existing interfaces would be stuck with the original -- causing universal confusion. + +The case for `Result` is less pressing, as it isn't tied as deeply into the language as `Optional` is. However, `Result` serves a similar purpose as `Optional`: it "merely" expands the nil case to explain the absence with an error value, to implement manual error propagation. It therefore makes sense to propose `Result`'s generalization alongside `Optional`: it involves solving effectively the same problems, and applying the same solutions. + +## Proposed Solution + +In this proposal, we extend the following generic types in the Standard Library with support for noncopyable type arguments: + +- `enum Optional` +- `enum Result` +- `struct MemoryLayout` +- `struct UnsafePointer` +- `struct UnsafeMutablePointer` +- `struct UnsafeBufferPointer` +- `struct UnsafeMutableBufferPointer` +- `class ManagedBuffer` +- `struct ManagedBufferPointer` + +`Optional` and `Result` become conditionally copyable, inheriting their copyability from their type arguments. All other types above remain unconditionally copyable, independent of the copyability of their type argument. + +We also update a single standard protocol to allow noncopyable conforming types: + +- `protocol ExpressibleByNilLiteral: ~Copyable` + +Additionally, we generalize the following top-level function families: + +- `func swap(_:_:)` +- `func withExtendedLifetime(_:_:)` +- `func withUnsafeTemporaryAllocation(byteCount:alignment:_:)` +- `func withUnsafeTemporaryAllocation(of:capacity:_:)` +- `func withUnsafePointer(to:_:)` +- `func withUnsafeMutablePointer(to:_:)` +- `func withUnsafeBytes(of:_:)` +- `func withUnsafeMutableBytes(of:_:)` + +We also generalize some low-level generic functions elsewhere in the stdlib that take or return the types above -- such as pointer conversion or rebinding operations. (See below for details.) + +In several example snippets, we'll be using the following (nonexistent) type to illustrate the use of noncopyable types: + +```swift +struct File: ~Copyable, Sendable { + init(opening path: FilePath) throws {...} + mutating func readByte() throws -> UInt8 {...} + consuming func close() throws {...} + deinit {...} +} +``` + +This is for illustrative purposes only -- we're not proposing to add an I/O facility to the stdlib in this document, and we do not expect a hypothetical future I/O feature would actually use this exact API surface. + +The rest of this section presents the principles this proposal follows in generalizing the constructs above. For a detailed list of changes, please see the [Detailed Design](#detailed-design) below. + +### Generalizing function members + +In past Swift versions, the difference between an operation consuming and borrowing its input argument was merely a subtle implementation detail: it was relevant for some performance optimization work, but generally there was little reason to learn or care about it. With noncopyable inputs, the distinction between consuming vs borrowing an argument rises to upmost importance -- it is one of the first things we need to learn when we write code that needs to use noncopyable types, whether we want to design our own APIs or to understand and use APIs provided by others. + +Retrofitting existing generic APIs for noncopyable use involves determining what ownership semantics to apply on their potentially noncopyable input arguments (including the special `self` argument). If the result of a function is noncopyable, then that inherently means the function is passing ownership of its output to its caller. This means that the function cannot keep hold of that value. + +When we cannot assume copyability, we need to carefully distinguish between consuming and borrowing use: functions need to declare this choice up front, for every parameter that isn't guaranteed to be copyable. + +For example, take the existing pointer operation that initializes the addressed location: + +```swift +extension UnsafeMutablePointer { + func initialize(to value: Pointee) { ... } +} +``` + +Semantically, this used to take a _copy_ of its input value, as that's how copyable values get passed to functions. (In this case, the implementation of the function is exposed to clients---it is marked `@inlinable`---so the copying can often be optimized away when the call happens to be the last use of the original instance. However, this depends on optimization heuristics; there is no strict guarantee that it will always happen.) + +With a noncopyable pointee, this will no longer work! The calling convention no longer makes sense. + +To support potentially non-copyable `Pointee` types, this function must now explicitly specify ownership semantics for its input argument. In this case, the operation wants to _take ownership_ of its input, because it needs to move it into its new place at the addressed location in memory. Therefore the function must explicitly consume its input argument: + +```swift +extension UnsafeMutablePointer where Pointee: ~Copyable { + func initialize(to value: consuming Pointee) { ... } +} +``` + +This new function remains source-compatible with the classic definition, except now it can work with noncopyable types. If a copyable value is passed as a `consuming` argument, Swift can pass an implicit copy of it as needed, based on whether or not the caller will need to continue using the original. Importantly, the explicit `consuming` keyword now _guarantees_ that the code will avoid making unnecessary copies when possible, even if `Pointee` happens to be copyable: we no longer rely on optimizer heuristics to avoid unnecessary copying overhead. + +By distinguishing between consuming and borrowing use, we gain more precise control over how our code behaves. The downside, of course, is that we pay for this control by having to think about it -- not only while defining these operations, but also while using them. To call this function with a noncopyable entity, we need to provide it with an item we own, and we need to be willing to let the function consume it. For example, we cannot call it on an instance that we're only borrowing from someone else. + +Preserving source compatibility is great, but unfortunately entirely replacing the old entry point with this new definition would be an ABI breaking change: the new function follows a new calling convention and it is exposed under a different linkage name. Existing binaries will keep linking with the original entry point, and we need to ensure they continue working. To allow this, on ABI stable platforms we continue to expose the old definition as an obsoleted `@usableFromInline internal` function. To allow newly built code to run on previously shipped Standard Library versions, the replacement needs to be defined in a back-deployable manner, such as by using the [`@backDeploy` attribute][SE-0376]. + +[SE-0376]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0376-function-back-deployment.md + +```swift +// Non-normative illustration of an implementation technique. +extension UnsafeMutablePointer where Pointee: ~Copyable { + @backDeployed(before: ...) + public func initialize(to value: consuming Pointee) { ... } +} +extension UnsafeMutablePointer /*where Pointee: Copyable*/ { + @available(swift, obsoleted: 1) + @usableFromInline + internal func initialize(to value: Pointee) { ... } +} +``` + +This way, existing binaries can continue to link with the original entry point, while newly built code will smoothly transition to the new definition.) + +The new function needs to be marked back-deployable, as it is replacing the original copyable version, and as such it needs to have matching availability. The function's implementation is directly embedded into binaries, so this means that in this particular case, newly introduced support for noncopyable use is also expected to "magically" work on older releases. (This will not necessarily extend to all noncopyable generalizations, as not every operation can retroactively learn to deal with noncopyable entities. In particular, older runtimes aren't expected to understand how to perform dynamic operations on noncopyable types (such as looking up metatype instances, performing conformance checks, dealing with existentials, downcasting or reflection); any operation that requires such features is not expected to deploy back without limits.) + +Declaring a function's input parameters `consuming` or `borrowing` can generally be done without tweaking the implementation, as long as the function does not happen to implicitly copy such arguments. If the implementation does rely on implicit copying, then it needs to be corrected to avoid doing that. + +### Generalizing higher-order functions + +For most operations, introducing support for noncopyable use is simply a matter of deciding what ownership rules they should follow and then adding the corresponding annotations. Sometimes the choice between consuming/borrowing use isn't obvious though -- the function may make sense in both flavors. + +Such is often the case with operations that take function arguments. For, example, take `Optional.map`: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws(E) -> U + ) throws(E) -> U? +} +``` + +Is this function supposed to consume or borrow the optional? Looking through existing (copyable) use cases, the answer seems to be both! + +In many cases, `map` is used to _transform_ the wrapped value into some other type, logically consuming it in the process. In some others, `map` is used to _project_ the wrapped value into some other entity, for example by copying a component or some computed property of it. + +The existing `map` name cannot be used to name both flavors, as consuming/borrowing annotations are not involved in overload resolution, so trying to do that would make the `map` name ambiguous. Of course, we could use the above distinction between consuming transformations and borrowing projections to replace `map` with two functions named `transform` and `project`. However, this terminology would be way too subtle, and it would not apply to similar cases elsewhere, such as `map`'s close sibling, `Optional.flatMap`. + +To resolve the ambiguity, we'll probably need to introduce a naming convention, such as to use `consuming` and/or `borrowing` as naming prefixes (as in `consumingMap` or `borrowingFlatMap`), or to invent `consuming` or `borrowing` views and move these operations there (as in `consuming.map` or `borrowing.flatMap`). Some of these choices depend on language features (non-escapable types, stored borrows, read accessors, consuming getters) that do not exist yet. Accordingly, we defer introducing consuming/borrowing higher-order functions until we can gain enough practical experience to make an informed decision. + +Functions like `Optional.map` will therefore keep requiring copyability for now. However, we can and should immediately generalize such functions in a different direction. + +### Generalizing caller-provided return types + +Many of the higher-order functions in the Standard Library are designed to return whatever value is returned by their function arguments. This includes the `Optional.map` function we saw above: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws(E) -> U + ) throws(E) -> U? +} +``` + +The generic return type `U` is implicitly required to be copyable here, which prevents the transformation from returning a noncopyable type: + +```swift +let name: FilePath? = ... +let file: File? = try name.map { try File(opening: $0) } +// error: noncopyable type 'File' cannot be substituted for copyable generic parameter 'U' in 'map' +``` + +Code that uses noncopyable types would hit this limitation surprisingly frequently, and working around it requires annoying and error-prone acrobatics, such as using an inout capture of an optional: + +```swift +let name: FilePath? = ... +var file: File? = nil +try name.map { file = try File(opening: $0) } // OK +``` + +To avoid forcing developers to use such workarounds, we systematically generalize such closure-taking APIs to allow noncopyable result types: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws(E) -> U + ) throws(E) -> U? +} + +let name: FilePath? = ... +let file: File? = try name.map { try File(opening: $0) } // OK +``` + +This is particularly important for interfaces like `ManagedBuffer.withUnsafeMutablePointers`, which are commonly used to access some container type's backing storage. A typical example is inside the implementation of a removal operation where the closure wants to return the item it removed from the container. + +(This generalization of outputs can be safely shipped separate from the consuming/borrowing generalizations of the input side. Doing these generalizations in two separate phases is not expected to cause any issues: it does not prevent getting us to whatever final APIs we want, and it does not introduce any unique compatibility problems that wouldn't also occur if we did both generalizations at the same time.) + +### (Lack of) protocol generalizations + +It would be very much desirable to generalize some of the existing Standard Library protocols for noncopyable use. However, each protocol needs careful consideration that is best deferred to subsequent proposals; therefore, in this particular document, we limit ourselves to generalizing just a single protocol, `ExpressibleByNilLiteral`. + +Generalizing `ExpressibleByNilLiteral` allows our newly noncopyable `Optional` to unconditionally conform to it, so that we can continue to use the `nil` keyword to refer to an empty optional instance, even if it happens to be wrapping a noncopyable type. + +```swift +var document: File? = nil // OK +``` + +All other public protocols in the Standard Library continue to require their conforming types as well as all their associated types to be copyable for now. This includes (but isn't limited to) such basic protocols as `Equatable`, `CustomStringConvertible`, `ExpressibleByArrayLiteral`, `Codable`, and `Sequence`. Some of these are directly generalizable, but most will require considerable design work, which we defer to future proposals. + +Therefore, all other conformances on `Optional` and `Result` will remain conditional on `Wrapped`'s copyability until future proposals. For example, while we'll be able to use the `== nil` form to check if an optional wraps no entity, the full `==` function is leaning on `Equatable` and thus it will only work for copyable types for now: + +```swift +let file: File? = try File(opening: "noncopyable-stdlib-primitives.md") +print(c == nil) // OK, prints "false" +let d: File? = nil +print(c == d) // error: operator function '==' requires that 'File' conform to 'Equatable' +``` + +On the other hand, unsafe pointer types remain unconditionally copyable, so some of their conformances can continue to remain unconditional. `UnsafePointer` and `UnsafeMutablePointer` can remain `Equatable`, `Comparable`, `Strideable` etc. even if their pointee happens to be noncopyable. + +```swift +let p: UnsafePointer = ... +var q: UnsafePointer = ... +print(p == q) // OK +q += 1 // OK +``` + +Unfortunately, the same isn't true for `UnsafeBufferPointer`, whose conformances to the `Collection` protocol hierarchy do not translate at all -- all our current container protocols require a copyable `Element`. + +Unsafe buffer pointers gain much of their core functionality from the `Collection` protocol: for instance, even the idea of accessing an item by subscripting with an integer index comes from that protocol. However, all buffer pointers need a way identify positions in themselves, and so we must nevertheless generalize the `Index` type and its associated index navigation methods and the crucial indexing subscript operation. + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + typealias Index = Int + var startIndex: Int { get } + var endIndex: Int { get } + var isEmpty: Bool { get } + var count: Int { get } + func index(after i: Int) -> Int + func index(before i: Int) -> Int + ... + subscript(i: Int) -> Element // (unstable accessor not shown) +} +``` + +Note that the generalized indexing subscript cannot provide a regular getter, as that would work by returning a copy of the item -- so the Standard Library currently has to resort to an unstable/unsafe language feature to provide direct borrowing access. (This isn't new, as we previously relied on this scheme to optimize performance; but its use now becomes unavoidable. Defining a stable language feature to implement such accessors is expected to be a topic of a future proposal.) + +While we can generalize the basic container primitives, sadly we need to leave the actual sequence & collection conformances conditional on copyability. Some of the protocol requirements also need to stick with copyability: + +```swift +extension UnsafeBufferPointer: Sequence /* where Element: Copyable */ { + struct Iterator: IteratorProtocol {...} + func makeIterator() -> Iterator {...} +} + +extension UnsafeBufferPointer: RandomAccessCollection /* where Element: Copyable */ { + typealias Indices = Range + typealias SubSequence = Slice> + var indices: Indices { get } + subscript(bounds: Range) -> Slice +} +``` + +For-in loops currently require a `Sequence` conformance, which means it will not yet be possible to iterate over the contents of an unsafe buffer pointer of noncopyable elements using a direct for-in loop. For now, we will need to write manual loops such as this one: + +```swift +let buffer: UnsafeBufferPointer> = ... +for i in buffer.startIndex ..< buffer.endIndex { + buffer[i].add(1, ordering: .sequentiallyConsistent) +} +``` + +This will have to bide us over until we invent new protocols for noncopyable containers. (Of course, that is expected to be the subject of subsequent work; however, we'll first need to introduce nonescapable types and use them to build some fundamental Standard Library constructs.) + +Another item of particular note is the loss of slicing subscript for noncopyable buffer pointers. The original slicing subscript returns a `Slice`, which requires a `Base` that conforms to `Collection`. Therefore, we can only provide the slicing subscript if `Element` happens to be copyable. Slicing buffer pointers is a very common operation, quite crucial for their usability. To make up for this, we propose to add an operation that returns a new standalone buffer over the supplied range of elements: + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + func extracting(_ bounds: Range) -> Self + func extracting(_ bounds: some RangeExpression) -> Self +} +``` + +The returned buffer does not share indices with the original; its indices start at zero. + +For buffer pointers with noncopyable elements, this operation will be the only (easy) way to split a buffer into small parts: + +```swift +import Synchronization + +// A bank of atomic integers +let bank = UnsafeMutableBufferPointer>.allocate(capacity: 4) +for i in 0 ..< 4 { + bank.initializeElement(at: i, to: Atomic(i)) +} + +let part = bank.extracting(2 ..< 4) +print(part[0].load(ordering: .sequentiallyConsistent)) // Prints "2" +print(part[1].load(ordering: .sequentiallyConsistent)) // Prints "3" + +bank.deinitialize().deallocate() +``` + +For copyable elements, the `extracting` operation is not crucial, but it is still useful: it is effectively a shorthand for slicing the buffer and immediately passing the returned slice to the `UnsafeBufferPointer.init(rebasing:)` initializer. This too is a common idiom, so it makes sense to provide a universally available shorter spelling for it. + +### Unblocking basic construction work + +The changes proposed here are enough to start constructing noncopyable containers, such as this illustrative noncopyable array variant, built entirely around an unsafe buffer pointer: + +```swift +struct Hypoarray: ~Copyable { + private var _storage: UnsafeMutableBufferPointer + private var _count: Int + + init() { + _storage = .init(start: nil, count: 0) + _count = 0 + } + + init(_ element: consuming Element) { + _storage = .allocate(capacity: 1) + _storage.initializeElement(at: 0, to: element) + _count = 1 + } + + deinit { + _storage.extracting(0 ..< count).deinitialize() + _storage.deallocate() + } +} +``` + +See the appendix for a full(er) definition of this sample type, including some of the fundamental array operations. + +Note that this type is presented here only to illustrate the use of the newly enhanced Standard Library; we do not propose to add such a type to the library as part of this proposal. On the other hand, building up a suite of basic noncopyable data structure implementations is naturally expected to be the subject of subsequent future work. + +## Detailed Design + +### `protocol ExpressibleByNilLiteral` + +In this proposal, we limit ourselves to generalizing just one standard protocol, `ExpressibleByNilLiteral`. We lift the requirement that conforming types must be copyable: + +```swift +protocol ExpressibleByNilLiteral: ~Copyable { + init(nilLiteral: ()) +} +``` + +This lets us continue to support the use of `nil` with noncopyable `Optional` types. + +(We do need to eventually generalize additional protocols, of course; as we mentioned above, such work is deferred to future proposals.) + +### `enum Optional` + +The `Optional` enum needs to be generalized to allow wrapping non-copyable types. This requires `Optional` to itself become conditionally copyable. + +```swift +@frozen +enum Optional: ~Copyable { + case none + case some(Wrapped) +} + +extension Optional: Copyable /* where Wrapped: Copyable */ {} +extension Optional: Sendable where Wrapped: ~Copyable & Sendable { } + +extension Optional: ExpressibleByNilLiteral where Wrapped: ~Copyable { + init(nilLiteral: ()) +} + +extension Optional where Wrapped: ~Copyable { + init(_ some: consuming Wrapped) +} +``` + +`Optional`'s `map` and `flatMap` members can be generalized to relax the copyability requirement on their return type: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws(E) -> U + ) throws(E) -> U? + + func flatMap( + _ transform: (Wrapped) throws(E) -> U? + ) throws(E) -> U? +} +``` + +However, these members cannot work on noncopyable optionals, as we have to distinguish between consuming and borrowing variants in that context. Choosing which of these two variants to generalize the existing names, and precisely what notation to use for the remaining variant is deferred to a future proposal. + +The current `unsafelyUnwrapped` property cannot currently be generalized for noncopyable types, so it is also kept restricted to the copyable case. + +[As foreshadowed in SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md#noncopyable-optional), it is commonly useful to use noncopyable optionals to allow partial consumption of stored properties in [contexts where that isn't normally allowed][SE-0429]. To support such use, we introduce a brand new mutating member `Optional.take()` that resets `self` to nil, returning its original value: + +```swift +extension Optional where Wrapped: ~Copyable { + mutating func take() -> Self { + let result = consume self + self = nil + return result + } +} +``` + +Having a named operation for this common need establishes it as a universal idiom. + +The standard nil-coalescing `??` operator is updated to explicitly consume its first operand: + +``` +func ?? ( + optional: consuming T?, + defaultValue: @autoclosure () throws -> T +) rethrows -> T + +func ?? ( + optional: consuming T?, + defaultValue: @autoclosure () throws -> T? +) rethrows -> T? +``` + +This matches the behavior of the second argument, where ownership of the default value is passed to the `??` implementation. + +In this initial phase, `Equatable` continues to require conforming types to be copyable, so optionals containing noncopyable types cannot yet be compared for equality. However, `Optional` also provides special support for `== nil` and `!= nil` comparisons whether or not its wrapped type is Equatable. We do generalize this support to allow noncopyable wrapped types: + +```swift +extension Optional where Wrapped: ~Copyable { + static func ==( + lhs: borrowing Wrapped?, + rhs: _OptionalNilComparisonType + ) -> Bool + static func !=( + lhs: borrowing Wrapped?, + rhs: _OptionalNilComparisonType + ) -> Bool + static func ==( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool + static func !=( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool +} +``` + +We are also generalizing the standard `~=` operators to support pattern matching `nil` on noncopyable Optionals. + +```swift +extension Optional where Wrapped: ~Copyable { + static func ~=( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool +} +``` + +(The implementations above currently rely on an unstable `_OptionalNilComparisonType` type to represent a type-agnostic `nil` value. This type and this particular way of implementing `== nil` is an internal implementation detail of the Standard Library that remains subject to change in future versions. These signatures are listed to illustrate the changes we're making; they aren't intended to stabilize this particular implementation.) + +### `enum Result` + +The standard `Result` type similarly needs to be generalized to allow noncopyable `Success` values, itself becoming conditionally noncopyable. + +```swift +@frozen +enum Result: ~Copyable { + case success(Success) + case failure(Failure) +} + +extension Result: Copyable where Success: Copyable {} +extension Result: Sendable where Success: Sendable & ~Copyable {} +``` + +Like with `Optional`, some of `Result`'s existing members can be directly generalized not to require a copyable success type. + +```swift +extension Result where Success: ~Copyable { + init(catching body: () throws(Failure) -> Success) + + consuming func get() throws(Failure) -> Success + + consuming func mapError( + _ transform: (Failure) -> NewFailure + ) -> Result( + _ transform: (Failure) -> Result + ) -> Result +} +``` + +The `mapError` members need to potentially return the `Success` value they originally stored in `self`, so they need to become consuming functions -- we cannot provide any borrowing variants. + +Like we saw with `Optional`, unfortunately this does not apply to members that transform the success value. We can still generalize the type of the result, but not the input: + +```swift +extension Result { + func map( + _ transform: (Success) -> NewSuccess + ) -> Result + + func flatMap( + _ transform: (Success) -> Result + ) -> Result +} +``` + +We defer generalizing the "input side" into borrowing/consuming map/flatMap variants until a future proposal; until then, `map` and `flatMap` continue to require a copyable `Success`. + +### `enum MemoryLayout` + +We extend `MemoryLayout` to allow querying the layout properties of noncopyable types: + +```swift +enum MemoryLayout: Copyable {} + +extension MemoryLayout where T: ~Copyable { + static var size: Int { get } + static var stride: Int { get } + static var alignment: Int { get } + + static func size(ofValue value: borrowing T) -> Int + static func stride(ofValue value: borrowing T) -> Int + static func alignment(ofValue value: borrowing T) -> Int +} +``` + +Note that the current `offset(of:)` member continues to require `T` to be copyable, as key paths do not (currently) support noncopyable targets. + +### Unsafe Pointer Types + +We have two typed unsafe pointer types, `UnsafePointer` and `UnsafeMutablePointer`. To allow building noncopyable constructs, these types need to start supporting noncopyable pointee types. + +```swift +struct UnsafePointer: Copyable +struct UnsafeMutablePointer: Copyable +``` + +Pointers to noncopyable types still need to work like pointers -- in particular, the pointers themselves must always remain copyable. Unlike with `Optional` and `Result`, pointers can therefore continue to unconditionally conform to the `Equatable`, `Hashable`, `Comparable`, `Strideable` and `CVarArg` protocols, as well as the new `AtomicRepresentable`, `AtomicOptionalRepresentable` protocols, regardless of the copyability of their pointee type. + +```swift +extension Unsafe[Mutable]Pointer: Equatable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: Hashable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: Comparable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: Strideable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: CustomDebugStringConvertible where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: CustomReflectable where Pointee: ~Copyable {...} + +// module Synchronization: +extension Unsafe[Mutable]Pointer: AtomicRepresentable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: AtomicOptionalRepresentable where Pointee: ~Copyable {...} +``` + +#### Member generalizations + +Most existing members of unsafe pointers adapt directly into the noncopyable world, with some notable exceptions that inherently require copyability: + +- Some operations rely on duplicating or copying pointee values: + - `func initialize(repeating: Pointee, count: Int)` + - `func update(from source: UnsafePointer, count: Int)` + - `func initialize(from source: UnsafePointer, count: Int)` +- Others depend on key paths that have not been generalized for noncopyable types yet: + - `func pointer(to: KeyPath) -> Unsafe[Mutable]Pointer?` + - `func pointer(to: WritableKeyPath) -> UnsafeMutablePointer?` + +These members will continue to require that `Pointee` be copyable. + +All other standard pointer operations lift the copyability requirement: + +- In Swift 5.x, the `pointee` property and the standard offsetting subscript have already been defined with special accessors that provide in-place borrowing or mutating access to instances addressed by the pointer. These translate directly for noncopyable use. + + ```swift + extension Unsafe[Mutable]Pointer where Pointee: ~Copyable { + var pointee: Pointee // (unstable accessors not shown) + subscript(i: Int) -> Pointee // (unstable accessors not shown) + } + ``` + +- Of special note is the `withMemoryRebound` function, which needs to generalized not just for noncopyable pointees, but also for potentially noncopyable target and result types: + + ```swift + extension Unsafe[Mutable]Pointer where Pointee: ~Copyable { + func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: Unsafe[Mutable]Pointer) throws(E) -> Result + ) throws(E) -> Result + } + ``` + +- In previous Swift releases, the existing `UnsafeMutablePointer.initialize(to:)` member used to be defined to (effectively) borrow, rather than consume, its argument. This used to be a minor performance wrinkle, but with noncopyable pointees, it has now become a correctness problem. Therefore, in its newly generalized form, `initialize(to:)` now consumes its argument: + + ```swift + extension UnsafeMutablePointer where Pointee: ~Copyable { + func initialize(to value: consuming Pointee) + } + ``` + + This change does not affect source compatibility with existing copyable call sites, and its ABI impact is mitigated by continuing to expose the original borrowing entry point as an obsolete `@usableFromInline` function. + +- All other pointer members generalize in a straightforward way: + + ```swift + extension Unsafe[Mutable]Pointer where Pointee: ~Copyable { + init(_ other: Self) + init?(_ other: Self?) + init(_ from: OpaquePointer) + init?(_ from: OpaquePointer?) + init?(bitPattern: Int) + init?(bitPattern: UInt) + + func deallocate() + } + + extension UnsafeMutablePointer where Pointee: ~Copyable { + init(mutating other: UnsafePointer) + init?(mutating other: UnsafePointer?) + init(_ other: UnsafeMutablePointer) + init?(_ other: UnsafeMutablePointer?) + + static func allocate(capacity count: Int) -> UnsafeMutablePointer + + func move() -> Pointee + + func moveInitialize(from source: UnsafeMutablePointer, count: Int) + func moveUpdate(from source: UnsafeMutablePointer, count: Int) + + func deinitialize(count: Int) -> UnsafeMutableRawPointer + } + ``` + +#### Related enhancements in other types + +To keep the Standard Library's family of pointer types coherent, we also need to ensure that pointers to noncopyable types continue to interact well with other pointer types in the language, including `UnsafeRawPointer` and `OpaquePointer`: + +- The Swift Standard Library provides heterogeneous pointer comparison operators (`==`, `!=`, `<`, `<=`, `>`, `>=`) that allow comparing any two pointer values, no matter their type. We generalize these to extend their support to comparing pointers with noncopyable pointees. + +- Similarly, the `init(bitPattern:)` initializers on `Int` and `UInt` can work with any pointer type. These initializers must now also extend support to the newly generalized pointer types. + + (Note: We do not list interface updates for the last two enhancements, as they are currently implemented by generalizing the source-unstable `_Pointer` protocol, an implementation detail of the Standard Library.) + +- We need to generalize all generic conversion operations on raw and opaque pointers: + + ```swift + extension OpaquePointer { + init(_ from: Unsafe[Mutable]Pointer) + init?(_ from: Unsafe[Mutable]Pointer?) + } + + extension Unsafe[Mutable]RawPointer { + init(_ other: Unsafe[Mutable]Pointer) + init?(_ other: Unsafe[Mutable]Pointer?) + } + ``` + +- Operations that bind and initialize raw memory to arbitrary types also need to relax their copyability requirements: + + ```swift + extension Unsafe[Mutable]RawPointer { + func bindMemory( + to type: T.Type, capacity count: Int + ) -> Unsafe[Mutable]Pointer + + func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: Unsafe[Mutable]Pointer) throws(E) -> Result + ) throws(E) -> Result + + func assumingMemoryBound( + to: T.Type + ) -> Unsafe[Mutable]Pointer + + func moveInitializeMemory( + as type: T.Type, from source: UnsafeMutablePointer, count: Int + ) -> UnsafeMutablePointer + } + ``` + +- As well as raw pointer operations that deal with a generic type's memory layout: + + ```swift + extension Unsafe[Mutable]RawPointer { + func alignedUp(for type: T.Type) -> Self + func alignedDown(for type: T.Type) -> Self + } + ``` + +#### Setting temporary pointers on arbitrary entities + +The standard `withUnsafe[Mutable]Pointer` top-level functions allow temporary pointer access to any inout value. These now need to be extended to support noncopyable types: + +```swift +func withUnsafeMutablePointer( + to value: inout T, + _ body: (UnsafeMutablePointer) throws(E) -> Result +) throws(E) -> Result + +func withUnsafePointer( + to value: inout T, + _ body: (UnsafePointer) throws(E) -> Result +) throws(E) -> Result +``` + +Beware that the pointer argument to `body` continues to be valid only during the execution of the function, even if `T` happens to be noncopyable. There is also no guarantee that the address will remain unchanged across repeated calls to `withUnsafe[Mutable]Pointer`. + +This also emphatically applies to the third `withUnsafePointer` variant that provides a temporary pointer to a borrowed instance. This one also gets generalized: + +```swift +func withUnsafePointer( + to value: borrowing T, + _ body: (UnsafePointer) throws(E) -> Result +) throws(E) -> Result +``` + +Borrows aren't exclusive, so it is possible to reentrantly call this function multiple times on the same noncopyable instance. When we do so, it may sometimes appear that the same (ostensibly noncopyable) entity is concurrently occupying multiple different locations in memory: + +```swift +struct Ghost: ~Copyable { + var value: Int +} + +let ghost = Ghost(value: 42) +withUnsafePointer(to: ghost) { p1 in + withUnsafePointer(to: ghost) { p2 in + print(p1 == p2) // Can print false! + } +} +``` + +Do not adjust your set -- this curiosity is inherent in the call-by-value calling convention that Swift normally uses for passing borrowed instances. (Semantically, there is still only a single extant copy, although it can sometimes be smeared over multiple locations.) + +### Unsafe Buffer Pointers + +Like pointers, typed buffer pointers need to start supporting noncopyable elements, without themselves becoming noncopyable. + +```swift +struct UnsafeBufferPointer: Copyable {} +struct UnsafeMutableBufferPointer: Copyable {} +``` + +#### Member generalizations + +Most existing buffer pointer operations directly translate to the noncopyable world: + +- Initializers adapt with no changes: + + ```swift + extension UnsafeBufferPointer where Element: ~Copyable { + init(start: UnsafePointer?, count: Int) + init(_ other: UnsafeMutableBufferPointer) + } + extension UnsafeMutableBufferPointer where Element: ~Copyable { + init(start: UnsafeMutablePointer?, count: Int) + init(mutating other: UnsafeBufferPointer) + } + ``` + +- So do the properties for accessing the components of a buffer pointer: + + ```swift + extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + var baseAddress: Unsafe[Mutable]Pointer? { get } + var count: Int { get } + } + ``` + +- As well as mutable/immutable deallocation: + + ```swift + extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + func deallocate() + } + ``` + +- And most mutating operations: + + ```swift + extension UnsafeMutableBufferPointer where Element: ~Copyable { + static func allocate(capacity count: Int) -> UnsafeMutableBufferPointer + + func moveInitialize(fromContentsOf source: Self) -> Index + + func moveUpdate(fromContentsOf source: Self) -> Index + + func deinitialize() -> UnsafeMutableRawBufferPointer + func deinitializeElement(at index: Index) + func moveElement(from index: Index) -> Element + } + ``` + +Like we saw with `UnsafeMutablePointer`, some operations need to be adjusted: + +- In Swift 5.10, `initializeElement(at:to:)` has an issue where it borrows, rather than consumes, its argument. We need to replace it with a source compatible variant that resolves this: + + ```swift + extension UnsafeMutableBufferPointer where Element: ~Copyable { + func initializeElement(at index: Index, to value: consuming Element) + } + ``` + + To ensure compatibility with current binaries, we also keep providing the old function as an obsolete entry point, like we did for `UnsafeMutablePointer.initialize(to:)`. + +- Memory rebinding operations again need to be generalized along multiple axes: + + ```swift + extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: Unsafe[Mutable]BufferPointer) throws(E) -> Result + ) throws(E) -> Result + } + ``` + +#### Protocol conformances + +The buffer pointer types also conform to the `Sequence`, `Collection`, `BidirectionalCollection`, `RandomAccessCollection` and `MutableCollection` protocols. We aren't generalizing these protocols in this proposal -- they continue to require copyable `Element` types. Therefore, buffer pointer conformances to these protocols must remain restricted to the pre-existing copyable cases. + +This also affects some related typealiases and nested types: the `UnsafeBufferPointer.Iterator` type and its `SubSequence` and `Indices` typealiases will only exist when `Element` is copyable. A buffer pointer of noncopyable elements is not a sequence, so as of this proposal it cannot be iterated over by a for-in loop. It also doesn't get any of the standard Sequence/Collection algorithms. (We expect to reintroduce these features in the future.) + +However, we do propose to generalize most of the core collection operations, even without carrying the actual conformance: + +```swift +extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + typealias Index = Int + var isEmpty: Bool { get } + var startIndex: Int { get } + var endIndex: Int { get } + func index(after i: Int) -> Int + func formIndex(after i: inout Int) + func index(before i: Int) -> Int + func formIndex(before i: inout Int) + func index(_ i: Int, offsetBy n: Int) -> Int + func index(_ i: Int, offsetBy n: Int, limitedBy limit: Int) -> Int? + func distance(from start: Int, to end: Int) -> Int +} + +extension UnsafeMutableBufferPointer where Element: ~Copyable { + func swapAt(_ i: Int, _ j: Int) +} +``` + +In Swift 5.x, the indexing subscript was already defined with special accessors that support in-place mutating access. To support in-place borrowing access, we can adapt the unstable/unsafe accessors from the unsafe pointers types, to define a subscript with direct support for use with noncopyable elements: + +```swift +extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + subscript(i: Int) -> Element // (special accessors not shown) +} +``` + +#### Extracting parts of a buffer pointer + +Unfortunately, the slicing subscript cannot be generalized, as its `Slice` return type requires a base container that conforms to `Collection`. + +We therefore propose to add the following new member methods for extracting a standalone buffer that covers a range of indices: + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + func extracting(_ bounds: Range) -> Self + func extracting(_ bounds: some RangeExpression) -> Self + func extracting(_ bounds: UnboundedRange) -> Self +} + +extension UnsafeMutableBufferPointer where Element: ~Copyable { + func extracting(_ bounds: Range) -> Self + func extracting(_ bounds: some RangeExpression) -> Self + func extracting(_ bounds: UnboundedRange) -> Self +} +``` + +Unlike with slicing, the returned buffer does not share indices with the original -- the result is a regular buffer that has its own 0-based indices. This operation is effectively equivalent to slicing the buffer and then immediately rebasing the slice into a standalone buffer pointer: `buffer.extracting(i ..< j)` produces the same result as the expression `UnsafeBufferPointer(rebasing: buffer[i ..< j])` did in Swift 5.x. + + +#### Exceptions + +There are also some buffer pointer operations that inherently cannot be generalized for noncopyable cases. These include: + +- Operations that require copying elements: + - `func initialize(repeating: Element)` + - `func update(repeating: Element)` +- Operations that operate on sequences or collections of items: + - `func initialize>(from: S) -> (unwritten: S.Iterator, index: Index)` + - `func initialize(fromContentsOf source: some Collection)` + - `func update>(from: S) -> (unwritten: S.Iterator, index: Index)` + - `func update(fromContentsOf: some Collection) -> Index` +- Members that involve buffer pointer slices: + - `init(rebasing slice: Slice>)` + - `func moveInitialize(fromContentsOf source: Slice) -> Index` + - `func moveUpdate(fromContentsOf source: Slice) -> Index` + +These operations are not in any way deprecated; they just continue requiring `Element` to be copyable. (We do expect to introduce noncopyable alternatives for the sequence/collection operations in a subsequent proposal.) + +#### Related enhancements in other types + +To keep the Standard Library's family of pointer types coherent, we also need to generalize some conversion/rebinding/initialization operations on raw buffer pointers: + +```swift +extension Unsafe[Mutable]RawBufferPointer { + init(_ buffer: UnsafeMutableBufferPointer) + init(_ buffer: UnsafeBufferPointer) + + func bindMemory( + to type: T.Type + ) -> Unsafe[Mutable]BufferPointer + + func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: Unsafe[Mutable]BufferPointer) throws(E) -> Result + ) throws(E) -> Result + + func assumingMemoryBound( + to: T.Type + ) -> Unsafe[Mutable]BufferPointer +} + +extension UnsafeMutableRawBufferPointer { + func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer +} +``` + + +#### Temporary buffer pointers over arbitrary entities + +We also need to similarly generalize the top-level `withUnsafe[Mutable]Pointer` functions that provide temporary buffer pointers over arbitrary values: + +```swift +func withUnsafeMutableBytes( + of value: inout T, + _ body: (UnsafeMutableRawBufferPointer) throws(E) -> Result +) throws(E) -> Result + +func withUnsafeBytes( + of value: inout T, + _ body: (UnsafeRawBufferPointer) throws(E) -> Result +) throws(E) -> Result + +func withUnsafeBytes( + of value: borrowing T, + _ body: (UnsafeRawBufferPointer) throws(E) -> Result +) throws(E) -> Result +``` + +All of these (and especially the borrowing variant) is subject to the same limitations as the original copyable variants: the pointers exposed are only valid for the duration of the function invocation, and multiple executions on the same instance may provide different locations for the same entity. + + +### Temporary Allocation Facility + +The [Standard Library's facility for allocating temporary uninitialozed buffers][SE-0322] needs to be generalized to support allocating storage for noncopyable types, as well as returning a potentially noncopyable type: + +[SE-0322]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0322-temporary-buffers.md + +```swift +func withUnsafeTemporaryAllocation( + byteCount: Int, + alignment: Int, + _ body: (UnsafeMutableRawBufferPointer) throws(E) -> R +) throws(E) -> R + +func withUnsafeTemporaryAllocation( + of type: T.Type, + capacity: Int, + _ body: (UnsafeMutableBufferPointer) throws(E) -> R +) throws(E) -> R + +``` + +### Managed Buffers + +Managed buffers provide a way for Swift container types to dynamically allocate storage for their contents in the form of a managed class reference. + +We generalize managed buffer types to support noncopyable element types, including all of their existing member operations. + +```swift +open class ManagedBuffer { + final var header: Header + init(_doNotCallMe: ()) +} + +@available(*, unavailable) +extension ManagedBuffer: Sendable where Element: ~Copyable {} + +extension ManagedBuffer where Element: ~Copyable { + // All existing members +} + +struct ManagedBufferPointer {...} + +extension ManagedBufferPointer where Element: ~Copyable { + // All existing members +} +``` + +The core `withUnsafeMutablePointer` interfaces are further generalized to allow noncopyable return types: + +```swift +extension ManagedBuffer where Element: ~Copyable { + final func withUnsafeMutablePointerToHeader( + _ body: (UnsafeMutablePointer
) throws(E) -> R + ) throws(E) -> R + + final func withUnsafeMutablePointerToElements( + _ body: (UnsafeMutablePointer) throws(E) -> R + ) throws(E) -> R + + final func withUnsafeMutablePointers( + _ body: ( + UnsafeMutablePointer
, UnsafeMutablePointer + ) throws(E) -> R + ) throws(E) -> R +} +``` + +```swift +extension ManagedBufferPointer where Element: ~Copyable { + func withUnsafeMutablePointerToHeader( + _ body: (UnsafeMutablePointer
) throws(E) -> R + ) throws(E) -> R + + func withUnsafeMutablePointerToElements( + _ body: (UnsafeMutablePointer) throws(E) -> R + ) throws(E) -> R + + func withUnsafeMutablePointers( + _ body: ( + UnsafeMutablePointer
, UnsafeMutablePointer + ) throws(E) -> R + ) throws(E) -> R +} +``` + +Notably, we preserve the requirement that the `Header` type must be copyable for now. It would be desirable to allow noncopyable `Header` types, but preserving compatibility with the stored property `ManagedBuffer.header` requires further work, so it is deferred. (We do not believe this to be a significant obstacle in practice.) + +### Lifetime Management + +The Standard Library offers the `withExtendedLifetime` family of functions to explicitly extend the lifetime of an entity to cover the entire duration of a closure. To support ownership control, we lift the copyability requirement on both the item whose lifetime is being extended, and for the return type of the function argument: + +```swift +func withExtendedLifetime( + _ x: borrowing T, + _ body: () throws(E) -> Result +) throws(E) -> Result +``` + +There exists a second variant of `withExtendedLifetime` whose function argument is passed the entity whose lifetime is being extended. This variant is less frequently used, but it still makes sense to generalize this to pass a borrowed instance: + +```swift +func withExtendedLifetime( + _ x: borrowing T, + _ body: (borrowing T) throws(E) -> Result +) throws(E) -> Result +``` + +### Swapping and exchanging items + +We have a standalone `swap` function that swaps the values of two `inout` values. We propose to generalize this operation to lift its copyability requirement. This is a good opportunity to make use of the new ownership control features to greatly simplify its implementation: + +```swift +func swap(_ a: inout T, _ b: inout T) { + let tmp = consume a + a = consume b + b = consume tmp +} +``` + +We also propose to add a new variant of this same operation that takes a single `inout` value, setting it to a given value and returning the original: + +```swift +public func exchange( + _ value: inout T, + with newValue: consuming T +) -> T { + var oldValue = consume value + value = consume newValue + return oldValue +} +``` + +This is a nonatomic analogue of the `exchange` operation on `struct Atomic`. This is a commonly invoked idiom, and having a standard operation for it will reduce the need to reinvent it from scratch with each use. (Thereby eliminating a potential source of errors, and improving readability.) By using `exchange`, we can avoid the need to manually introduce a second `inout` binding just to be able to invoke `swap`. + +## Source compatibility + +This proposal is heavily built on the assumption that removing the assumption of copyability on these constructs will not break existing code that relies on it. This is largely the case, although there are subtle cases where these generalizations break code that relies on shadowing standard declarations. + +For instance, code that used to substitute their own definition of `Optional.map` (or any other newly generalized function) in place of the stdlib's official definition may find that their declaration is no longer considered to shadow the original: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws -> U + ) rethrows -> U? { + print("Hello from map!") + switch self { + case .some(let y): + return .some(try transform(y)) + case .none: + return .none + } + } +} + +let foo: Int? = 42 +foo.map { $0 + 1 } // error: ambiguous use of 'map' +``` + +The new `map` uses typed throws and it allows noncopyable return types, rendering it different enough to make this substitution no longer shadow the original. This makes such generalizations technically source breaking; however the breakage is similar in nature and severity as a source break that can arise from new API additions that happen to clash with preexisting extensions of similar names defined outside the Standard Library. If such issues prove harmful in practice, we can subsequently amend Swift's shadowing rules to ignore differences in throwing and noncopyability. + +## ABI compatibility + +We limited the changes proposed so that we allow maintaining full backward compatibility with existing binaries. + +Adding support for noncopyable type parameters generally changes linker-level mangled symbol names in emitted code, which would break ABI -- we avoid this either by continuing to ship the original function definitions as obsoleted `@usableFromInline internal` functions, or by overriding mangling to ignore `~Copyable` (using an unstable `@_preInverseGenerics` attribute). + +We also provide a measure of forward compatibility -- newly built code that calls newly generalized functions will continue to remain compatible with previously shipped versions of the Standard Library. This naturally must apply to the preexisting copyable cases, but it also extends to noncopyable use: the newly generalized generic operations are generally expected to work on older Swift runtime environments. Of course, older runtimes do not understand noncopyable generics (or even noncopyable types in general), so features that rely on runtime dynamism will come with a stricter deployment limit. (The feature set we propose in this document is not expected to hit this.) + +The `Optional` and `Result` types that shipped in previous versions of the Standard Library were naturally built with the assumption of copyability, but they tended to avoid making unnecessary copies, which means they are mostly expected to be also "magically" compatible with noncopyable use. (It is okay to break an assumption that was never actually relied on.) The places where we preserved mangling are the places where we think this applies -- we expect newly built code that invokes the old implementations will still run fine. (If we missed a case where an earlier implementation did rely on copying or runtime dynamism, we can correct it at any point by switching to the `@backDeplpoyed`/`@usableFromInline` implementation pattern.) + +## Alternatives Considered + +The primary alternative is to delay this work until it becomes possible to express more of the functionality that is deferred by this proposal. However, this would leave noncopyable types in a limbo state, where the language ships with rich functionality to support them, but the core Standard Library continues to treat them as second class entities. + +The inability to apply unsafe pointer APIs to noncopyable types would be a particularly severe obstacle to practical adoption -- it is tricky to fully embrace ownership control if we have no way to dynamically allocate storage for noncopyable entities. + +Avoiding the use of `Optional` is a similarly severe API design issue, with no elegant solutions. Forcing adopters of ownership control to define custom `Optional` types has proved impractical beyond simple throwaway prototypes; it's better to have a standard solution. + +We do not consider the generalization of `Result` to be anywhere near as important as `Optional`, although it does provide a standard way to implement manual error propagation. However, as it is a close relative to `Optional`, it seems undesirable to defer its generalization. + +### Omitting `UnsafeBufferPointer` + +`UnsafeBufferPointer` conforms to `Collection`, and it relies on the standard `Slice` type for its `SubSequence` concept. Neither `Collection` nor `Slice` can be directly generalized for noncopyable elements, and so these conformances need to continue require copyable elements. + +Given that buffer pointers are essentially useless without an idea of an index (which comes from `Collection`), we considered omitting them from this proposal, deferring their generalization until we have protocols for noncopyable container types. + +However, in practice, this would not be acceptable: the buffer pointer is Swift's native way to represent a region of direct memory, and we urgently need to enable dealing with memory regions that contain noncopyable instances. Leaving buffer pointers ungeneralized would strongly encourage Swift code to start passing around base pointers and counts as distinct items, which would be a significant step backwards -- we must avoid training Swift developers to do that. (We'd also lose the ability to generalize the `withUnsafeTemporaryAllocation` function, which is built on top of buffer pointers.) + +Therefore, this proposal generalizes buffer pointers, including the parts of `Collection` that we strongly believe will directly translate to noncopyable containers (the basic concept of an index, the index navigation members and the indexing subscript). + +### Alternatives to `UnsafeBufferPointer.extracting()` + +A different concern arises with buffer pointer slices. Regrettably, it seems we have to give up on the `buffer[i.., in buffer: UnsafeBufferPointer) +} + +// Usage: +UnsafeMutableBufferPointer(rebasing: i ..< j, in: buffer) +``` + +This easily fits into Swift API design conventions, but it doesn't feel like a good enough solution in practice. Specifically, it suffers from two distinct (but related) problems: + +1. It remains just as verbose, inconvenient and non-intuitive as the original rebasing initializer; and we have considered that a significant problem even in the copyable case. + + + + (Indeed, a large part of [SE-0370][SE-370-Slice] was dedicated to reducing the need to directly invoke this initializer, by cleverly extending the `Slice` type with direct methods that [hide the `init(rebasing:)` call](https://github.com/apple/swift/blob/swift-5.10-RELEASE/stdlib/public/core/UnsafeBufferPointerSlice.swift#L699-L702 +). This is very helpful, but in exchange for simplifying use sites, we've made it more difficult to define custom operations: each operation has to be defined on both the buffer pointer and the slice type, and the latter requires advanced generics trickery. Of course, none of this work helps the noncopyable case, as `Slice` does not translate there -- so we get back to where we started.) + +[SE-370-Slice]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0370-pointer-family-initialization-improvements.md#slices-of-bufferpointer + + + +2. The new initializer would also apply to the copyable case, but it would serve no discernible purpose in that context, other than to increase confusion. + +The solution we propose is to make the new operation a regular member function. This solves the first problem: `buffer.extracting(i..: ~Copyable, ~Escapable { + case none + case some(Wrapped) +} + +extension Optional: Copyable where Wrapped: ~Escapable {} +extension Optional: Escapable where Wrapped: ~Copyable {} +extension Optional: Sendable where Wrapped: ~Copyable & ~Escapable & Sendable {} + +extension Optional where Wrapped: ~Copyable & ~Escapable { + public init(_ some: consuming Wrapped) dependsOn(some) -> Self { + self = .some(some) + } +} +``` + +It is likely that we will want to generalize `MemoryLayout` as well. Allowing unsafe pointers to address non-escapable types is not nearly as straightforward, but it's possible we'll need to tackle that, too. + +### Generalizing higher-order functions + +This proposal does not allow `map` or `flatMap` to be called on noncopyable `Optional` or `Result` types yet, to avoid prematurely establishing a pattern before it becomes possible to express better solutions. + +As detailed in the [Proposed Solution]((#generalizing-higher-order-functions)) section, this is mostly a naming/presentation problem: we need distinct notations for the `map` that consumes `self` vs. the variant that merely borrows it. + +One straightforward idea is to simply use `consuming` and `borrowing` as naming prefixes: + +```swift +extension Optional where Wrapped: ~Copyable { + consuming func consumingMap( + _ transform: (consuming Wrapped) throws(E) -> U + ) throws(E) -> U? + + borrowing func borrowingMap( + _ transform: (borrowing Wrapped) throws(E) -> U + ) throws(E) -> U? +} +``` + +This is a somewhat verbose choice, but it makes the choice eminently clear at point of use: + +```swift +struct Wrapper: ~Copyable { + var value: T +} + +let v: Wrapper? +print(v.borrowingMap { $0.value }) + +let w: Wrapper? +let file = try v.consumingMap { try File(opening: $0) } +``` + +The primary drawback of this simple solution is that developers working with classic (i.e. copyable) `Optional` values would now be faced with three separate APIs for what is (from their viewpoint) the same operation. Making a distinction between guaranteed-consuming and guaranteed-borrowing transformations is not entirely pointless even in the copyable case, but it is mostly a nitpicky performance detail that wouldn't otherwise merit any new API additions. However, the distinction is crucial for noncopyable use, and that may excuse the new variations even if they mean additional noise for the classic copyable cases. + +A similar idea is to introduce `consuming` and `borrowing` views, and to move the ownership-aware operations into them, leaving us with the notations `v.borrowing.map { $0.value }` or `v.consuming.map { try File(opening: $0) }`. These are also eminently readable, and they would also be a good spiritual fit with the `.lazy` sequence view we already have. They also help with the noise issue, as the nitpicky variants with explicit ownership annotations would all get hidden away in views dedicated to ownership control. + +The idea of a "consuming view" is a bit of a stretch, as it doesn't seem particularly useful outside of this context; but a "borrowing view" certainly would have merit on its own -- it would be a type that consists of a "borrow" of an instance of some other type, which would be an independently useful construct. (E.g., it would allow us to generalize `Slice` into a `BorrowingSlice` while keeping it generic over the base container.) + +Therefore, the best choice may be to introduce the idea of a `borrowing` view (returning a standard `Borrow` (or `Ref`) type), but to avoid introducing a `consuming` view, preferring to instead generalize the existing `map`/`flatMap`/`filter`/`reduce` etc functions in the consuming sense. So `v.map { ... }` would be (implicitly) consuming, while `v.borrowing.map { ... }` would be explicitly borrowing. + +It isn't currently possible to implement generic borrowing views, as structs can only contain owned instances of another type, not borrowed ones. Therefore, we need to delay work on consuming/borrowing higher-order functions until it becomes possible to express such a thing. (We could implement the `consumingMap` and `borrowingMap` naming convention right now, but it seems likely that we'd regret that when it becomes possible to express the borrowing view concept.) + +### Generalizing `Optional.unsafelyUnwrapped` + +The `unsafelyUnwrapped` property of `Optional` implements an unsafe variant of the safe force-unwrap operation that is built into Swift (denoted `!`). (This property is _unsafe_ because it does not guarantee to check if the optional is empty before attempting to extract its wrapped value. Trying to access a value that isn't there is undefined behavior.) + +This proposal keeps this property in its original form, so it will be only available if `Wrapped` is copyable. + +Ideally, `unsafelyUnwrapped` would be generalized to follow the same adaptive behavior as the force-unwrap form, allowing both consuming and borrowing use. + +To achieve this, Swift would need to implement the following three enhancements: + +1. Provide a way to define a (coroutine based) borrowing accessor on a computed property +2. Provide a way to define an accessor on a computed property that consumes `self` (i.e., a consuming getter). +3. Allow these two accessors to coexist within the same property, with the language inferring which one to use based on usage context. + +Generalizing `unsafelyUnwrapped` needs to be deferred either until these become possible or until we decide not to do them. + +```swift +// Illustration; this is not real Swift +extension Optional where Wrapped: ~Copyable { + var unsafelyUnwrapped: Wrapped { + consuming get { ... } + read { ... } + modify { ... } // Let's throw this in the mix as well + } +} +``` + +In the meantime, we considered adding a separate `unsafeUnwrap()` member to provide a separate solution for point 2 above: + +```swift +extension Optional where Wrapped: ~Copyable { + consuming func unsafeUnwrap() -> Wrapped +} +``` + +However, if we do end up getting these enhancements, then this new function would become an unnecessary addition. As this is a rather obscure/niche operation, it doesn't seem worth this trouble. + +### Generalized managed buffer headers + +This proposal lifts the copyability requirement on `ManagedBuffer`'s `Element` type, but it continues to require `Header` to be copyable. + +Of course, it would be desirable to lift this requirement, too. Unfortunately, `ManagedBuffer` exposes the public (stored) property `header`, and lifting the copyability requirement would break this property's (implicit) ABI for low-level access. Until we find a way to mitigate this problem, we cannot generalize stored properties to remove the assumption of copyability; therefore, we need to postpone generalizing `Header`. + +Requiring a copyable `Header` does not appear to be a significant hurdle in most use cases, so it seems preferable to leave time to design a proper solution rather than attempting to ship a quick stopgap fix that may prove to be incomplete. + +### Additional raw pointer operations + +`Unsafe[Mutable]RawPointer` includes the `load(fromByteOffset:as:)` operation that directly returns a copy an instance of an arbitrary type at the indicated location. We kept this restricted to copyable types, and we refrained from providing noncopyable equivalents, such as the closure-based member below: + +```swift +extension Unsafe[Mutable]RawPointer { + func withValue( + atByteOffset offset: Int = 0, + as type: T.Type, + _ body: (borrowing T) throws(E) -> Result + ) throws(E) -> Result +} +``` + +We also do not provide a mutating operation that consumes an instance at a particular offset: + +```swift +extension UnsafeMutableRawPointer { + func move( + fromByteOffset offset: Int = 0, + as type: T.Type + ) -> T +} +``` + +We omitted these, as it is unclear if these would be the best ways to express these. For now, we instead recommend explicitly binding memory and using `Unsafe[Mutable]Pointer` operations. + +### Protocol generalizations + +[As noted above](#lack-of-protocol-generalizations), this proposal leaves most standard protocols as is, deferring their generalizations to subsequent future work. The single protocol we do generalize is `ExpressibleByNilLiteral` -- the `nil` syntax is so closely associated with the `Optional` type that it would not have been reasonable to omit it. + +This of course is not tenable; we expect that many (or even most) of our standard protocols will need to eventually get generalized for noncopyable use. + +For some protocols, this work is relatively straightforward. For example, we expect that generalizing `Equatable`, `Hashable` and `Comparable` would not be much of a technical challenge -- however, it will involve overhauling/refining `Equatable`'s semantic requirements, which I do not expect to be an easy process. (`Equatable` currently requires that "equality implies substitutability"; if the two equal instances happen to be noncopyable, such unqualified, absolute statements no longer seem tenable.) The `RawRepresentable` protocol is also in this category. + +In other cases, the generalization fundamentally requires additional language enhancements. For example, we may want to consider allowing noncopyable `Error` types -- but that implies that we'll also want to throw and catch noncopyable errors, and that will require a bit more work than adding a `~Copyable` clause on the protocol. It makes sense to defer generalizing the protocol until we decide to do this; if/when we do, the generalizations of `Result` can and should be part of the associated discussion and proposal. Another example is `ExpressibleByArrayLiteral`, which is currently built around an initializer with a variadic parameter -- to generalize it, we need to either figure out how to generalize those, or we need to design some alternative interface. + +In a third category of cases, the existing protocols make heavy use of copyability to (implicitly) unify concerns that need stay distinct when we introduce ownership control. Retroactively untangling these concerns is going to be difficult at best -- and sometimes it may in fact prove impractical. For instance, the current `Sequence` protocol is shaped like a consuming construct: `makeIterator` semantically consumes the sequence, and `Iterator.next()` passes ownership of the elements to its caller. However, the documentation of `Sequence` explicitly allows conforming types to implement multipass/nondestructive behavior, and it in fact it _requires_ `Collection` types to do precisely that. By definition, a consuming sequence cannot be multipass; such sequences are borrowing by nature. To support noncopyable elements, we'll need to introduce distinct abstractions for borrowing and consuming sequences. Generalizing the existing `Sequence` in either of these directions seems fraught with peril. + +Each of these protocol generalizations will require effort that's _at least_ comparable in complexity to this proposal; so it makes sense to consider them separately, in a series of future proposals. + +### Additional future work + +Fully supporting ownership control and noncopyable types will require overhauling much of the existing Standard Library. + +This includes generalizing dynamic runtime operations -- a huge area that includes facilities such as isa checks, downcasts, existentials, reflection, key paths, etc. (For instance, updating `print()` to fully support printing noncopyable types is likely to require many of these dynamic features to work.) + +On the way to generalizing the Standard Library's current sequence and collection abstractions, we'll also need to implement a variety of alternatives to the existing copy-on-write collection types, `Array`, `Set`, `Dictionary`, `String`, etc, providing clients direct control over (runtime and memory) performance: consider a fixed-capacity array type, or a stack-allocated dictionary construct. + +Many of these depend on future language enhancements, and as such they will be developed alongside those. + +## Appendix: `struct Hypoarray` + +Hypoarray is a simple noncopyable generic struct that is a very thin, safe wrapper around a piece of directly allocated memory. It is presented here as an illustration of the pointer improvements introduced in this document. + +This section is not normative: we are not proposing to add a `Hypoarray` type to the Standard Library. However, it illustrates the use of the proposed Standard Library extensions, and it does serve as a first prototype for a potential future addition. + +This type operates on a lower level of abstraction than the standard `Array` type. "Hypo" is greek for "under", so "hypoarray" is an apt name for such a construct. (In fact, if we started anew, the existing `Array` would potentially be built on top of such a construct.) + +A hypoarray is like an `Array` without the implicit copy-on-write machinery: it is still dynamically allocated, and it can still implicitly resize itself as needed, but it replaces copy-on-write behavior with strict ownership control. Its storage is always uniquely held, so every mutation can be done in place, resulting in more predictable performance. (Although implicit reallocations can still result in unexpected spikes of latency! To get rid of those, we'd need to introduce an even lower-level array variant that has a fixed capacity. We'll leave that as an exercise to the reader for now.) + +A hypoarray consists of a dynamically allocated storage buffer (of variable capacity) and an integer count that specifies how many initialized elements it contains. The elements of the array are all compacted at the beginning of storage, with any remaining slots serving as free capacity for future additions. + +```swift +struct Hypoarray: ~Copyable { + private var _storage: UnsafeMutableBufferPointer + private var _count: Int +``` + +The buffer's count is the current capacity of the hypoarray. We'll need to keep referring to it elsewhere, so it makes sense to introduce a name for it early on: + +```swift + var capacity: Int { _storage.count } +``` + +Initializing an empty array can be done by simply setting up an empty buffer, and setting the count to zero. + +```swift + init() { + _storage = .init(start: nil, count: 0) + _count = 0 + } +``` + +That wasn't very interesting, so to spruce things up, we can also provide a single-element initializer that needs to actually allocate and initialize some memory: + +```swift + init(_ element: consuming Element) { + _storage = .allocate(capacity: 1) + _storage.initializeElement(at: 0, to: element) + _count = 1 + } +``` + +This nontrivial initializer takes ownership of the element it is given, so naturally it has to be declared to consume its argument. + +(Of course, we will eventually also want to have an initializer that can take any sequence of elements; however, this needs the idea a sequence type that produces consumable items, and we do not yet have a protocol that could express that. The `Sequence` we currently have requires its `Element` to be copyable, and it inherently combines borrowing and consuming iteration into a single, convenient abstraction. Sadly it does not directly translate to noncopyable use.) + +When the array is destroyed, we need to properly deinitialize its elements and deallocate its storage. To do this, we need to define a deinitializer: + +```swift + deinit { + _storage.extracting(0 ..< count).deinitialize() + _storage.deallocate() + } +} +``` + +Note the use of the new `extracting` operation to get a buffer pointer that consists of just the slots that have been populated. We cannot call `_storage.deinitialize()` as it isn't necessarily fully initialized; and we also cannot use the classic slicing operation `_storage[.. Int { i + 1 } + func index(before i: Int) -> Int { i - 1 } + func distance(from start: Int, to end: Int) -> Int { end - start } + // etc. +} +``` + +The most fundamental `Collection` operation is probably its indexing subscript for accessing a particular element. Obviously, we need hypoarray to provide this functionality, too. + +Unfortunately, subscripts (and computed properties) cannot currently return noncopyable results without transferring ownership of the result to the caller. + +```swift +// Illustration: an array of atomic integers +import Synchronization +let array = Hypoarray(Atomic(42)) +print(array[0]) // This cannot work! +``` + +The subscript getter would need to move the item out of the array to give ownership to the caller, which we do not want. Getter accessors will need to be generalized into a coroutine-based read accessor that supports in-place borrowing access. (And setters need to be generalized to allow in-place mutating access.) [Introducing such accessors is still in progress][_modify], so for now, the best we can do is to provide closure-based access methods: + +[_modify]: https://forums.swift.org/t/modify-accessors/31872 + +```swift +extension Hypoarray where Element: ~Copyable { + func borrowElement ( + at index: Int, + by body: (borrowing Element) throws(E) -> R + ) throws(E) -> R { + precondition(index >= 0 && index < _count) + return try body(_storage[index]) + } + + mutating func updateElement ( + at index: Int, + by body: (inout Element) throws(E) -> R + ) throws(E) -> R { + precondition(index >= 0 && index < _count) + return try body(&_storage[index]) + } +} +``` + +These are quite clumsy, but they do work safely, and they provide in-place borrowing and mutating access to any element in a hypoarray, without having to change its ownership. + +```swift +// Example usage: +var array = Hypoarray(42) +array.updateElement(at: 0) { $0 += 1 } +array.borrowElement(at: 0) { print($0) } // Prints "43" +``` + +[[Aside: A future language extension will hopefully allow us to replace these with the subscript we actually want to write, along the lines of this hypothetical example: + +```swift +// This isn't real Swift yet: +extension Hypoarray where Element: ~Copyable { + subscript(position: Int) -> Element { + read { + precondition(position >= 0 && position < _count) + try yield _storage[position] + } + modify { + precondition(position >= 0 && position < _count) + try yield &_storage[position] + } + } +} +``` + +```swift +// Example usage: +var array = Hypoarray(42) +array[0] += 1 +print(array[0]) // Prints "43" +``` + +Note that the proposed `UnsafeMutableBufferPointer` changes already include a subscript that allows in-place borrowing and mutating use. However, the solution used there is tied to low-level unsafe pointer semantics that would not directly translate to a higher-level type like `Hypoarray`.]] + +It would be desirable to allow iteration over `Hypoarray` instances. Unfortunately, Swift's `for in` construct currently relies on `protocol Sequence`, and that protocol doesn't support noncopyable use. (Not only does it require copyable conforming types and copyable `Element`s, but its iterator is also defined to give ownership of returned elements to the caller; that is to say, it is shaped like a _consuming_ construct, not a _borrowing_ one.) Introducing a mechanism for borrowing iteration, and retooling `for in` loops to allow such use is future work. While that work is in progress, we can of course still manually iterate over the contents of a hypoarray by using its indices: + +```swift +// Example usage: +var array: Hypoarray = ... +for i in array.startIndex ..< array.endIndex { // a.k.a. 0 ..< array.count + array.borrowElement(at: i) { print($0) } +} +``` + +Not having noncopyable container protocols also means that `Hypoarray` cannot conform to any, so subsequently it will not get any of the standard generic container algorithms for free: there is no `firstIndex(of:)`, there is no `map`, no `filter`, no slicing, no `sort`, no `reverse`. Indeed, many of these standard algorithms expect to work on `Equatable` or `Comparable` items, and those protocols are also yet to be generalized. + +Okay, so all we have is `borrowElement` and `updateElement`, for borrowing and mutating access. What about consuming access, though? + +Consuming an item of an array at a particular index would require either removing the item from the array, or destroying and discarding the rest of the array. Neither of these looks desirable as a primitive operation for accessing an element. However, we do expect arrays to provide a named operation for removing items, `remove(at:)`. This operation is easily implementable on `Hypoarray`: + +```swift +extension Hypoarray where Element: ~Copyable { + @discardableResult + mutating func remove(at index: Int) -> Element { + precondition(index >= 0 && index < count) + let old = _storage.moveElement(from: index) + let source = _storage.extracting(index + 1 ..< count) + let target = _storage.extracting(index ..< count - 1) + let i = target.moveInitialize(fromContentsOf: source) + assert(i == target.endIndex) + _count -= 1 + return old + } +} +``` + +Note how this moves the removed element out of the array, so it can legitimately give ownership of it to the caller. Following our preexisting convention, the result of `remove(at:)` is marked discardable; if the caller decides to discard it, then the removed item immediately gets destroyed, as expected. + +(Implementing the classic `removeSubrange(_: some RangeExpression)` operation is left as an exercise for the reader.) + +Okay, so now we know how to create simple single-element hypoarrays, how to access their contents, and we are even able to remove elements from them. How do we add new elements, though? + +Hypoarray is supposed to implement a dynamically resizing array, so insertions generally need to be able to expand storage. Let's tackle this sub-problem first, by implementing `reserveCapacity`: + +```swift +extension Hypoarray where Element: ~Copyable { + mutating func reserveCapacity(_ n: Int) { + guard capacity < n else { return } + let newStorage: UnsafeMutableBufferPointer = .allocate(capacity: n) + let source = _storage.extracting(0 ..< count) + let i = newStorage.moveInitialize(fromContentsOf: source) + assert(i == count) + _storage.deallocate() + _storage = newStorage + } +} +``` + +Note again the use of `extracting` to operate on parts of a buffer -- in this case, we use it to move initialized items between the two allocations. + +We want insertions to have amortized O(1) complexity, so they need to be careful about the rate at which they grow the array's storage. In this simple illustration, we'll use a geometric growth factor of 2, so that each reallocation will at least double the capacity of the array: + +```swift +extension Hypoarray where Element: ~Copyable { + mutating func _ensureFreeCapacity(_ minimumCapacity: Int) { + guard capacity < _count + minimumCapacity else { return } + reserveCapacity(max(_count + minimumCapacity, 2 * capacity)) + } +} +``` + +With that done, we can finally implement insertions, starting with the `append` operation. Its implementation is fairly straightforward: + +```swift +extension Hypoarray where Element: ~Copyable { + mutating func append(_ item: consuming Element) { + _ensureFreeCapacity(1) + _storage.initializeElement(at: _count, to: item) + _count += 1 + } +} +``` + +Inserting at a particular index is complicated by the need to make room for the new item, but it's not that tricky, either: + +```swift +extension Hypoarray where Element: ~Copyable { + mutating func insert(_ item: consuming Element, at index: Int) { + precondition(index >= 0 && index <= count) + _ensureFreeCapacity(1) + if index < count { + let source = _storage.extracting(index ..< count) + let target = _storage.extracting(index + 1 ..< count + 1) + target.moveInitialize(fromContentsOf: source) + } + _storage.initializeElement(at: index, to: item) + _count += 1 + } +} +``` + +```swift +// Example usage: +var array = Hypoarray() +for i in 0 ..< 10 { + array.insert(i, at: 0) +} +// array now consists of 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 +``` + +Without noncopyable container protocols, we cannot yet implement `append(contentsOf:)`, `insert(contentsOf:)`, `replaceSubrange` operations. But we can still provide classic `Sequence`/`Collection`-based operations in cases where `Element` happens to copyable: + +```swift +extension Hypoarray { + mutating func append(contentsOf items: some Sequence) { + for item in items { + append(item) + } + } +} +``` + +Note how this extension omits the suppression of element copyability -- it does not have a `where Element: ~Copyable` clause. This means that the extension only applies if `Element` is copyable. + +These operations give us all primitive operations we expect an array type to provide. Of course, the `Hypoarray` we have now created is just the very first draft of a future dynamically sized noncopyable array type. There is plenty of work left: we need to add more operations; we need to implement noncopyable variants of more data structures; we need to define the general shape of a noncopyable container; we need to populate that shape with a family of standard generic algorithms. Implicit resizing is not always appropriate in memory-starved or low-latency applications, so for those use cases we also need to design data structure variants that work within some fixed storage capacity (or even a fixed count). We may want the backing store to be allocated dynamically, like we've seen, or we may want it to become part of the construct's representation ("inline storage"); perhaps we want to allocate storage on the stack, or statically reserve space for a global variable at compile time. We expect future work will tackle all these tasks, and plenty more. diff --git a/proposals/0438-metatype-keypath.md b/proposals/0438-metatype-keypath.md new file mode 100644 index 0000000000..e12bd4afb6 --- /dev/null +++ b/proposals/0438-metatype-keypath.md @@ -0,0 +1,136 @@ +# Metatype Keypaths + +* Proposal: [SE-0438](0438-metatype-keypath.md) +* Authors: [Amritpan Kaur](https://github.com/amritpan), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift#73242](https://github.com/apple/swift/pull/73242) +* Review: ([pitch](https://forums.swift.org/t/pitch-metatype-keypaths/70767)) ([review](https://forums.swift.org/t/se-0438-metatype-keypaths/72172)) ([acceptance](https://forums.swift.org/t/accepted-se-0438-metatype-keypaths/72878)) + +## Introduction + +Key path expressions access properties dynamically. They are declared with a concrete root type and one or more key path components that define a path to a resulting value via the type’s properties, subscripts, optional-chaining expressions, forced unwrapped expressions, or self. This proposal expands key path expression access to include static properties of a type, i.e., metatype keypaths. + +## Motivation + +Metatype keypaths were briefly explored in the pitch for [SE-0254](https://forums.swift.org/t/pitch-static-and-class-subscripts/21850) and the [proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0254-static-subscripts.md#metatype-key-paths) later recommended them as a future direction. Allowing key path expressions to directly refer to static properties has also been discussed on the Swift Forums for database lookups when used [in conjunction with @dynamicMemberLookup](https://forums.swift.org/t/dynamic-key-path-member-lookup-cannot-refer-to-static-member/30212) and as a way to avoid verbose hacks like [referring to a static property through another computed property](https://forums.swift.org/t/key-path-cannot-refer-to-static-member/28055). Supporting metatype keypaths in the Swift language will address these challenges and improve language semantics. + +## Proposed solution + +We propose to allow keypath expressions to define a reference to static properties. The following usage, which currently generates a compiler error, will be allowed as valid Swift code. + +```swift +struct Bee { + static let name = "honeybee" +} + +let kp = \Bee.Type.name +``` + +## Detailed design + +### Metatype syntax + +Keypath expressions where the first component refers to a static property will include `.Type` on their root types stated in the key path contextual type or in the key path literal. For example: + +```swift +struct Bee { + static let name = "honeybee" +} + +let kpWithContextualType: KeyPath = \.name // key path contextual root type of Bee.Type +let kpWithLiteral = \Bee.Type.name // key path literal \Bee.Type +``` + +Attempting to write the above metatype keypath without including `.Type will trigger an error diagnostic: + +```swift +let kpWithLiteral = \Bee.name // error: static member 'name' cannot be used on instance of type 'Bee' +``` + +Keypath expressions where the component referencing a static property is not the first component do not require `.Type`: +```swift +struct Species { + static let isNative = true +} + +struct Wasp { + var species: Species.Type {Species.self} +} + +let kpSecondComponentIsStatic = \Wasp.species.isNative +``` +### Access semantics + +Immutable static properties will form the read-only keypaths just like immutable instance properties. +```swift +struct Tip { + static let isIncluded = True + let isVoluntary = False +} + +let kpStaticImmutable: KeyPath = \.isIncluded +let kpInstanceImmutable: KeyPath = \.isVoluntary +``` +However, unlike instance members, keypaths to mutable static properties will always conform to `ReferenceWritableKeyPath` because metatypes are reference types. +```swift +struct Tip { + static var total = 0 + var flatRate = 20 +} + +let kpStaticMutable: ReferenceWriteableKeyPath = \.total +let kpInstanceMutable: WriteableKeyPath = \.flatRate +``` +## Effect on source compatibility + +This feature breaks source compatibility for key path expressions that reference static properties after subscript overloads. For example, the compiler cannot differentiate between subscript keypath components by return type in the following: + +```swift +struct S { + static var count: Int { 42 } +} + +struct Test { + subscript(x: Int) -> String { "" } + subscript(y: Int) -> S.Type { S.self } +} + +let kpViaSubscript = \Test.[42] // fails to typecheck +``` + +This keypath does not specify a contextual type, without which the key path value type is unknown. To form a keypath to the metatype subscript and return an `Int`, we can specify a contextual type with a value type of `S.Type` and chain the metatype keypath: + +```swift +let kpViaSubscript: KeyPath = \Test.[42] +let kpAppended = kpViaSubscript.appending(path: \.count) +``` + +## ABI compatibility + +This feature does not affect ABI compatibility. + +## Implications on adoption + +This feature is back-deployable but it requires emission of new (property descriptors) symbols for static properties. + +The type-checker wouldn't allow to form key paths to static properties of types that come from modules that are built by an older compiler that don't support the feature because dynamic or static library produced for such module won't have all of the required symbols. + +Attempting to form a key path to a static property of a type from a module compiled with a compiler that doesn't yet support the feature will result in the following error with a note to help the developers: + +```swift +error: cannot form a keypath to a static property of type +note: rebuild to enable the feature +``` + +## Future directions + +### Key Paths to Enum cases + +Adding language support for read-only key paths to enum cases has been widely discussed on the [Swift Forums](https://forums.swift.org/t/enum-case-key-paths-an-update/68436) but has been left out of this proposal as this merits a separate discussion around [syntax design and implementation concerns](https://forums.swift.org/t/enum-case-keypaths/60899/32). + +Since references to enum cases must be metatypes, extending keypath expressions to include references to metatypes will hopefully bring the Swift language closer to adopting keypaths to enum cases in a future pitch. + +## Acknowledgments + +Thank you to Joe Groff for providing pivotal feedback on this pitch and its possible implementation and to Becca Royal-Gordon for an insightful discussion around the anticipated hurdles in implementing this feature. diff --git a/proposals/0439-trailing-comma-lists.md b/proposals/0439-trailing-comma-lists.md new file mode 100644 index 0000000000..1de9044bbb --- /dev/null +++ b/proposals/0439-trailing-comma-lists.md @@ -0,0 +1,304 @@ +# Allow trailing comma in comma-separated lists + +* Proposal: [SE-0439](0439-trailing-comma-lists.md) +* Author: [Mateus Rodrigues](https://github.com/mateusrodriguesxyz) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.1)** +* Implementation: [swiftlang/swift#74522](https://github.com/swiftlang/swift/pull/74522) +* Previous Proposal: [SE-0084](0084-trailing-commas.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-allow-trailing-comma-in-tuples-arguments-and-if-guard-while-conditions/70170)), ([review](https://forums.swift.org/t/se-0439-allow-trailing-comma-in-comma-separated-lists/72876)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0439-allow-trailing-comma-in-comma-separated-lists/73216)) +* Previous Revision: ([1](https://github.com/swiftlang/swift-evolution/blob/7864fa20cfb3a43aa6874feedb5aedb8be02da2c/proposals/0439-trailing-comma-lists.md)) + +## Introduction + +This proposal aims to allow the use of trailing commas, currently restricted to array and dictionary literals, in symmetrically delimited comma-separated lists. + +## Motivation + +### Development Quality of Life Improvement + +A trailing comma is an optional comma after the last item in a list of elements: + +```swift +let rank = [ + "Player 1", + "Player 3", + "Player 2", +] +``` + +Swift's support for trailing commas in array and dictionary literals makes it as easy to append, remove, reorder, or comment out the last element as any other element. + +Other comma-separated lists in the language could also benefit from the flexibility enabled by trailing commas. Consider the function [`split(separator:maxSplits:omittingEmptySubsequences:)`](https://swiftpackageindex.com/apple/swift-algorithms/1.2.0/documentation/algorithms/swift/lazysequenceprotocol/split(separator:maxsplits:omittingemptysubsequences:)-4q4x8) from the [Algorithms](https://github.com/apple/swift-algorithms) package, which has a few parameters with default values. + + +```swift +let numbers = [1, 2, 0, 3, 4, 0, 0, 5] + +let subsequences = numbers.split( + separator: 0, +// maxSplits: 1 +) ❌ Unexpected ',' separator +``` + +### The Language Evolved + +Back in 2016, a similar [proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0084-trailing-commas.md) with a narrower scope was reviewed and rejected for Swift 3. Since that time, the language has evolved substantially that challenges the basis for rejection. The code style that "puts the terminating right parenthesis on a line following the arguments to that call" has been widely adopted by community, Swift standard library codebase, swift-format, DocC documentation and Xcode. Therefore, not encouraging or endorsing this code style doesn't hold true anymore. + +The language has also seen the introduction of [parameter packs](https://github.com/apple/swift-evolution/blob/main/proposals/0393-parameter-packs.md), which enables APIs that are generic over variable numbers of type parameters, and code generation tools like plugins and macros that, with trailing comma support, wouldn't have to worry about a special condition for the last element when generating comma-separated lists. + +## Proposed solution + +This proposal adds support for trailing commas in symmetrically delimited comma-separated lists, which are the following: + +- Tuples and tuple patterns. + + ```swift + let velocity = ( + 1.66007664274403694e-03, + 7.69901118419740425e-03, + 6.90460016972063023e-05, + ) + + let ( + velocityX, + velocityY, + velocityZ, + ) = velocity + ``` + +- Parameter and argument lists of initializers, functions, enum associated values, expression macros and attributes. + + ```swift + func foo( + input1: Int = 0, + input2: Int = 0, + ) { } + + foo( + input1: 1, + input2: 1, + ) + + struct S { + init( + input1: Int = 0, + input2: Int = 0, + ) { } + } + + enum E { + case foo( + input1: Int = 0, + input2: Int = 0, + ) + } + + @Foo( + "input 1", + "input 2", + "input 3", + ) + struct S { } + + #foo( + "input 1", + "input 2", + "input 3", + ) + + struct S { + #foo( + "input 1", + "input 2", + "input 3", + ) + } + + ``` + +- Subscripts, including key path subscripts. + + ```swift + let value = m[ + x, + y, + ] + + let keyPath = \Foo.bar[ + x, + y, + ] + + f(\.[ + x, + y, + ]) + ``` + +- Closure capture lists. + + ```swift + { [ + capturedValue1, + capturedValue2, + ] in + } + ``` + +- Generic parameters. + + ```swift + struct S< + T1, + T2, + T3, + > { } + ``` + +- String interpolation. + + ```swift + let s = "\(1, 2,)" + ``` + +## Detailed Design + +Trailing commas will be supported in comma-separated lists when symmetric delimiters (including `(...)`, `[...]`, and `<...>`) enable unambiguous parsing. + +Note that the requirement for symmetric delimiters means that the following cases will not support trailing comma: + +- `if`, `guard` and `while` condition lists. + + ```swift + if + condition1, + condition2, ❌ + { } + + while + condition1, + condition2, ❌ + { } + + guard + condition1, + condition2, ❌ + else { } + ``` + +- Enum case label lists. + + ```swift + enum E { + case + a, + b, + c, ❌ + } + ``` + +- `switch` case labels. + + ```swift + switch number { + case + 1, + 2, ❌ + : + ... + default: + .. + } + ``` + +- Inheritance clauses. + + ```swift + struct S: + P1, + P2, + P3, ❌ + { } + ``` + +- Generic `where` clauses. + + ```swift + struct S< + T1, + T2, + T3, + > where + T1: P1, + T2: P2, ❌ + { } + ``` + +Trailing commas will be allowed in single-element lists but not in zero-element lists, since the trailing comma is actually attached to the last element. +Supporting a zero-element list would require supporting _leading_ commas, which isn't what this proposal is about. + +```swift +(1,) // OK +(,) ❌ expected value in tuple +``` + + +## Source compatibility + +This is a purely additive change with no source compatibility impact. + +## Alternatives considered + +### Allow trailing comma in all comma-separated lists + +Comma-separated lists that are not symmetrically delimited could also benefit from trailing comma support; for example, condition lists, in which reordering is fairly common. +However, these lists currently rely on the comma after the penultimate element to determine that what comes next is the last element, and some of them present challenges if relying on opening/closing delimiters instead. + +At first sight, `{` may seem a reasonable closing delimiter for `if` and `while` condition lists, but conditions can have a `{` themselves. + +```swift +if + condition1, + condition2, + { true }(), +{ } +``` + +This particular case can be handled but, given how complex conditions can be, it's hard to conclude that there's absolutely no corner case where ambiguity can arise in currently valid code. + +Inheritance lists and generic `where` clauses can appear in protocol definitions where there's no clear delimiter, making it harder to disambiguate where the list ends. + +```swift +protocol Foo { + associatedtype T: + P1, + P2, ❌ Expected type + ... +} +``` + +Although some comma-separated lists without symmetric delimiters may have a clear terminator in some cases, this proposal restricts trailing comma support to symmetrically delimited ones where it's clear that the presence of a trailing comma will not cause parsing ambiguity. + +### Eliding commas + +A different approach to address similar motivations is to allow the comma between two expressions to be elided when they are separated by a newline. + +```swift +print( + "red" + "green" + "blue" +) +``` +This was even [proposed](https://forums.swift.org/t/se-0257-eliding-commas-from-multiline-expression-lists/22889/188) and returned for revision back in 2019. + +The two approaches are not mutually exclusive. There remain unresolved questions about how the language can accommodate elided commas, and adopting this proposal does not prevent that approach from being considered in the future. + +## Revision History + +- Update to address acceptance decision of restricting trailing comma to lists with symmetric delimiters. + +## Acknowledgments + +Thanks to Alex Hoppen, Xiaodi Wu and others for their help on the proposal text and implementation. diff --git a/proposals/0440-debug-description-macro.md b/proposals/0440-debug-description-macro.md new file mode 100644 index 0000000000..e374da0f85 --- /dev/null +++ b/proposals/0440-debug-description-macro.md @@ -0,0 +1,242 @@ +# DebugDescription Macro + +* Proposal: [SE-0440](0440-debug-description-macro.md) +* Authors: [Dave Lee](https://github.com/kastiglione) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.0)** +* Implementation: Present in `main` under experimental feature `DebugDescriptionMacro` [apple/swift#69626](https://github.com/apple/swift/pull/69626) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/fda6746506368c8c6d2933ee6d71c87e6ed92f94/proposals/0440-debug-description-macro.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-debug-description-macro/67711)) ([review](https://forums.swift.org/t/se-0440-debugdescription-macro/72958)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0440-debugdescription-macro/73270)) ([second review](https://forums.swift.org/t/second-review-se-0440-debugdescription-macro/73325))([acceptance](https://forums.swift.org/t/accepted-se-0440-debugdescription-macro/73741)) + +## Introduction + +This proposal introduces `@DebugDescription`, a new debugging macro to the standard library, which lets data types specify a custom summary to be presented by the debugger. This macro brings improvements to the debugging experience, and simplifies the maintenance and delivery of debugger type summaries. It can be used in place of `CustomDebugStringConvertible` conformance, or in addition to, for custom use cases. + +## Motivation + +Displaying data is a fundamental part of software development. Both the standard library and the debugger offer multiple ways of printing values - Swift's print and dump, and LLDB's `p` and `po` commands. These all share the ability to render an arbitrary value into human readable text. Out of the box, both the standard library and the debugger present data as a nested tree of property-value pairs. The similarities run deep, for example the standard library and the debugger provide control over how much of the tree is shown. This functionality requires no action from the developer. + +The utility of displaying a complete value depends on the size and complexity of the data type(s), or depends on the context the data is being presented. Displaying the entirety of a small/shallow structure is sufficient, but some data types reach sizes/complexities where the complete tree of data is too large to be useful. + +For types that are too large or complex, the standard library and debugger again both provide tools giving us control over how our data is displayed. In the standard library, Swift has the `CustomDebugStringConvertible` protocol, which allows types to represented not as the aforementioned property tree, but as an arbitrary string. Relatedly, Swift has `CustomReflectable`, which lets developers control the contents and structure of the rendered property tree. For brevity and convention, from this point on this document will refer to the `CustomDebugStringConvertible` and `CustomReflectable` protocols via their single properties: `debugDescription` and `customMirror` respectively. + +LLDB has analogous features, which are called Type Summaries (\~`debugDescription`) and Synthetic Children (\\~`customMirror`) respectively. However, Swift and the debugger don't share or interoperate these definitions. Implementing these customizing protocols provides limited benefit inside the debugger. Likewise, defining Type Summaries or Synthetic Children in LLDB will have no benefit to Swift. + +While LLDB’s `po` command provides a convenient way to evaluate a `debugDescription` property defined in Swift, there are downsides to expression evaluation: Running arbitrary code can have side effects, be unstable to the application, and is slower. Expression evaluation happens by JIT compiling code, pushing it to the device the application is running on, and executing it in the context of the application, which involves a lot of work. As such, LLDB only does expression evaluation when explicitly requested by a user, most commonly with the `po` command in the console. Debugger UIs (IDEs) often provide a variable view which is populated using LLDB’s variable inspection which does not perform expression evaluation and is built on top of reflection. In some cases, such as when viewing crashlogs, or working with a core file, expression evaluation is not even possible. For these reasons, rendering values is ideally done without expression evaluation. + +This proposal introduces the ability to share a `debugDescription` definition between Swift and the debugger. This has benefits for developers, and for the debugger. + +LLDB Type Summaries can be defined using LLDB’s own (non Turing-complete) string interpolation syntax, called [Summary Strings](https://lldb.llvm.org/use/variable.html#summary-strings). While similar to Swift string interpolation, LLDB Summary Strings have restrictions that Swift string interpolation does not have. The primary restriction is that it allows data/property access, but not computation. LLDB Summary Strings cannot evaluate function calls, which includes computed properties. For the purpose of definition sharing, LLDB Type Summaries can be viewed as a lower common denominator of the two. As a result, definition sharing can be achieved only when a `debugDescription` definition meets the criteria imposed by LLDB Summary Strings. The criteria is not overly limiting, LLDB Summary Strings have been in for some time. + +Swift macros provide a convenient means to implement automatic translation of compatible `debugDescription` definitions into LLDB Summary Strings. A macro provides benefits that LLDB Summary Strings do not currently offer, including the ability to do compile time static validation to produce typo-free LLDB Summary Strings. The previously mentioned criteria that `debugDescription` must meet in order to be converted to an LLDB Summary String will loosen over time. This will be achieved first through the macro implementation becoming more sophisticated, and second as LLDB’s Summary Strings gain advancements. + +## Proposed solution + +Consider this simple example data type: + +```swift +struct Organization: CustomDebugStringConvertible { + var id: String + var name: String + var manager: Person + var members: [Person] + // ... and more + + var debugDescription: String { + "#\(id) \(name) (\(manager.name))" + } +} +``` + +To see the results of `debugDescription` in the debugger, the user has to run `po team` in the console. + +``` +(lldb) po team +"#15 Shipping (Francis Carlson) +``` + +Running the `p` command, or viewing the value in the Debugger UI (IDE), will show the value’s property tree, which may have arbitrary size/nesting: + +``` +(lldb) p team +(Organization) { + id = "..." + name = "Shipping" + manager = { + name = "Francis Carlson" + ... + } + members = { + [0] = ... + } + ... +} +``` + +However, by introducing the `@DebugDescription` macro, we can teach the debugger how to generate a summary without expression evaluation. + +```swift +@DebugDescription +struct Organization: CustomDebugStringConvertible { + var id: String + var name: String + var manager: Person + var members: [Person] + var officeAddress: [Address] + // ... and more + + var debugDescription: String { + "#\(id) \(name) (\(manager.name))" + } +} +``` + +The macro expands the body of `debugDescription` into the following LLDB Summary String: + +``` +#${var.id} ${var.name} (${var.manage.name}) +``` + +This summary string is emitted into the binary, where LLDB will load it automatically. Using this definition, LLDB can now present this description in contexts it previously could not, including the variable view and other parts of the debugger UI. + +A notable difference between the debugger console and debugger UI is that that UI displays one level at a time. When viewing an Array for example, its children are not expanded. To distinguish between elements of an Array (or any other collection), a user must expand each child. By employing `@DebugDescription`, LLDB will show a summary for each element of a collection, so that users may know – at a glance – exactly which element(s) to expand. + +## Detailed design + +```swift +/// Converts description definitions to a debugger Type Summary. +/// +/// This macro converts compatible description implementations written in Swift +/// to an LLDB format known as a Type Summary. A Type Summary is LLDB's +/// equivalent to debugDescription, with the distinction that it does not +/// execute code inside the debugged process. By avoiding code execution, +/// descriptions can be produced faster, without potential side effects, and +/// shown in situations where code execution is not performed, such as the +/// variable list of an IDE. +/// +/// Consider this an example. This Team struct has a debugDescription which +/// summarizes some key details, such as the team's name. The debugger only +/// computes this string on demand - typically via the po command. By applying +/// the DebugDescription macro, a matching Type Summary is constructed. This +/// allows the user to show a string like "Rams [11-2]", without executing +/// debugDescription. This improves the usability, performance, and +/// reliability of the debugging experience. +/// +/// @DebugDescription +/// struct Team: CustomDebugStringConvertible { +/// var name: String +/// var wins, losses: Int +/// +/// var debugDescription: String { +/// "\(name) [\(wins)-\(losses)]" +/// } +/// } +/// +/// The DebugDescription macro supports both debugDescription, description, +/// as well as a third option: a property named lldbDescription. The first +/// two are implemented when conforming to the CustomDebugStringConvertible +/// and CustomStringConvertible protocols. The additional lldbDescription +/// property is useful when both debugDescription and description are +/// implemented, but don't meet the requirements of the DebugDescription +/// macro. If lldbDescription is implemented, DebugDescription choose it +/// over debugDescription and description. Likewise, debugDescription is +/// preferred over description. +/// +/// ### Description Requirements +/// +/// The description implementation has the following requirements: +/// +/// * The body of the description implementation must a single string +/// expression. String concatenation is not supported, use string interpolation +/// instead. +/// * String interpolation can reference stored properties only, functions calls +/// and other arbitrary computation are not supported. Of note, conditional +/// logic and computed properties are not supported. +/// * Overloaded string interpolation cannot be used. +@attached(member) +@attached(memberAttribute) +public macro DebugDescription() = + #externalMacro(module: "SwiftMacros", type: "DebugDescriptionMacro") + +/// Internal-only macro. See @DebugDescription. +@attached(peer, names: named(_lldb_summary)) +public macro _DebugDescriptionProperty( debugIdentifier: String, _ computedProperties: [String]) = + #externalMacro(module: "SwiftMacros", type: "_DebugDescriptionPropertyMacro") +``` + +Of note, the work is split between two macros `@DebugDescription` and `@_DebugDescriptionProperty`. By design, `@DebugDescription` is attached to the type, where it gathers type-level information, including gather a list of stored properties. This macro also determines which description property to attach @_DebugDescriptionProperty to. + +`@_DebugDescriptionProperty` is not intended for direct use by users. This macro is scoped to the inspect a single description property, not the entire type. This approach of splitting the work allows the compiler to avoid unnecessary work. + +The additional supported property, `lldbDescription`, is to support two different use cases. + +First, in some cases existing `debugDescription`/`description` cannot be changed, because doing so would be a breaking change to either `String(reflecting:)` or `String(describing:)`. In these circumstances, developers can define `lldbDescription` instead. + +Second, in some cases developer may want to use LLDB Summary String syntax directly. Since `lldbDescription` is not coupled to API, developers are free to include Summary String elements in their `lldbDescription`. This is expected to be uncommon, but lets developers have more control over Summary Strings. Since LLDB syntax has no meaning inside `debugDescription`/`description`, the macro performs escaping when translating those definitions. + +Using both `debugDescription` and `lldbDescription` is an intended use case. The design of this macro allows developers to have both an LLDB compatible `lldbDescription`, and a more complex `debugDescription`. This allows the debugger to show a simple summary, while providing enabling a more detailed or dynamic `debugDescription`. + +## Source compatibility + +This proposal adds a new macro to the standard library. There are no source compatibility concerns. + +## ABI compatibility + +The macro implementation emits metadata for the debugger, and does not affect ABI. + +## Implications on adoption + +The macro can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. + +## Future directions + +### Support for Generics + +The `@DebugDescription` macro cannot currently be attached to generic type definitions. This is because the macro's implementation emits a static property into the type, which is a use case not currently supported by the Swift language. To support generic types, Swift will need a separate proposal to enable code such as the following: + +```swift +struct Generic { + static let _lldb_summary = (23 as UInt8, 30 as UInt8, ...) +} +``` + +The goal is to allow static properties in generic types, specifically when the property's type is concrete and independent of the generic signature. In the above code, the property's type is a tuple of `UInt8`, which does not depend on `T`. + +### Beyond Summary Strings + +Future directions include generating Python instead of LLDB Summary Strings. This has the benefit of having fewer restrictions on the `debugDescription` definition. It has the downside of needing security scrutiny not required by LLDB Summary Strings. + +A similar future direction is to support sharing Swift `customMirror` definitions into LLDB Synthetic Children definitions. Unlike LLDB Type Summaries, LLDB has no "DSL" to expression LLDB Synthetic Children, currently the main option is Python. Given that there are two uses solved by generating Python, it's an approach worth considering in the future. While `customMirror` implementations are less common in Swift than their `debugDescription` counterpart, in LLDB, Synthetic Children are as important, or even more important than Summary Strings. The reason is that Synthetic Children allow data types to express their data "interface" rather than their implementation. Consider types like Array and Dictionary, which often have implementation complexity that provides optimal performance, not for data simplicity. + +## Alternatives considered + +### Explicit LLDB Summary Strings + +The simplest macro implementation is one that performs no Swift-to-LLDB translation and directly accepts an LLDB Summary String. This approach requires users to know LLDB Summary String syntax, which while not complex, still presents a hindrance to adoption. Such a macro would could create redundancy: `debugDescription` and the separate LLDB Summary String. These would need to be manually kept in sync. + +### Independent Property (No `debugDescription` Reuse) + +Instead of leveraging existing `debugDescription`/`description` implementations, the `@DebugDescription` macro could use a completely separate property. + +Reusing existing `debugDescription` implementations makes a tradeoff that may not be obvious to developers. The benefit is a single definition, and getting more out of a well known idiom. The risk comes from the requirements imposed by `@DebugDescription`. The requirements could lead to developers changing their existing implementation. Any changes to `debugDescription` will impact `String(reflecting:)`, and similarly changes to `description` will impact `String(describing:)`. + +The risk involving String conversion would be avoided by having the macro use an independent property. The macro would not support `debugDescription`/`description`. In this scenario, developers would be required to implement `lldbDescription`, even if the implementation is identical to the existing `debugDescription`. + +Our expectation is that most code, particularly application code, will not depend on String conversion (especially `String(reflecting:)`). For code that does depend on String conversion, it should have testing in place to catch breaking changes. Inside of an application, authors of code which has behavior that depends on String conversion initializers should already be aware of the consequences of changing `debugDescription`/`description`. Frameworks are a more challenging situation, where its authors are not always aware of if/how its clients depend on String conversion. + +The belief is that the benefits of reusing `debugDescription` will outweigh the downsides. Framework authors can make it a policy of their own to not reuse `debugDescription`, if they believe that presents a risk to clients of their framework. + +### Contextual Diagnostics + +To help address the potential risk around reuse of `debugDescription`, the macro could emit diagnostics that vary by the property being used. Specifically, if the developer implements `lldbDescription`, they will get the full diagnostics available, indicating how to fix its implementation. Conversely, when `debugDescription` is being reused, the diagnostics will not contain details of which requirements were not met, instead the diagnostics would tell the user that `debugDescription` is not compatible, and to define `lldbDescription` instead. This should make it less likely that the macro leads to changes affecting String conversion. + +## Revision history + +* Changes from the first review + * Document support for generic types as a future direction + * Rename `_debugDescription` to `lldbDescription` + * Direct use of LLDB Summary String syntax is supported only by `lldbDescription` + +## Acknowledgments + +Thank you to Doug Gregor and Alex Hoppen for their generous guidance, and PR reviews. Adrian Prantl, for the many productive discussions and implementation ideas. To Kuba Mracek, for implementing linkage macros which support this work. Thank you to Tony Parker and Steve Canon for their adoption feedback. To Holly Borla, for timely technical and process support. diff --git a/proposals/0441-formalize-language-mode-terminology.md b/proposals/0441-formalize-language-mode-terminology.md new file mode 100644 index 0000000000..d34b6eeacf --- /dev/null +++ b/proposals/0441-formalize-language-mode-terminology.md @@ -0,0 +1,279 @@ +# Formalize ‘language mode’ terminology + +* Proposal: [SE-0441](0441-formalize-language-mode-terminology.md) +* Author: [James Dempsey](https://github.com/dempseyatgithub) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.1)** +* Implementation: [swiftlang/swift-package-manager#7620](https://github.com/swiftlang/swift-package-manager/pull/7620), [swiftlang/swift#75564](https://github.com/swiftlang/swift/pull/75564) +* Review: ([first pitch](https://forums.swift.org/t/pitch-formalize-swift-language-mode-naming-in-tools-and-api/71733)) ([second pitch](https://forums.swift.org/t/pitch-2-formalize-language-mode-naming-in-tools-and-api/72136)) ([review](https://forums.swift.org/t/se-0441-formalize-language-mode-terminology/73182)) ([acceptance](https://forums.swift.org/t/accepted-se-0441-formalize-language-mode-terminology/73716)) + +## Introduction +The term "Swift version” can refer to either the toolchain/compiler version or the language mode. This ambiguity is a consistent source of confusion. This proposal formalizes the term _language mode_ in tool options and APIs. + +## Proposed Solution +The proposed solution is to use the term _language mode_ for the appropriate Swift compiler option and Swift Package Manager APIs. Use of "Swift version" to refer to language mode will be deprecated. + +### Terminology +The term _language mode_ has been consistently used to describe this compiler feature since it was introduced with Swift 4.0 and is an established term of art in the Swift community. + +The **Alternatives Considered** section contains a more detailed discussion of the term's history and usage. + +### Swift compiler option +Introduce a `-language-mode` option that has the same behavior as the existing `-swift-version` option, while de-emphasizing the `-swift-version` option in help and documentation. + +#### Naming note +The proposed compiler option uses the term 'language mode' instead of 'Swift language mode' because the context strongly implies a Swift language mode. The intent is that the `languageMode()` compiler condition described in **Future directions** would also use that naming convention for the same reason. + +### Swift Package Manager +Introduce four Swift Package Manager API changes limited to manifests \>= 6.0: + +#### 1. A new Package init method that uses the language mode terminology + +```swift +@available(_PackageDescription, introduced: 6) +Package( + name: String, + defaultLocalization: [LanguageTag]? = nil. + platforms: [SupportedPlatform]? = nil, + products: [Product] = [], + dependencies: [Package.Dependency] = [], + targets: [Target] = [], + swiftLanguageModes: [SwiftLanguageMode]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil +) +``` + +Add a new `init` method to `Package` with the following changes from the current `init` method: + +- The parameter `swiftLanguageVersions` is renamed to `swiftLanguageModes` +- The parameter type is now an optional array of `SwiftLanguageMode` values instead of `SwiftVersion` values + +The existing init method will be marked as `deprecated` and `renamed` allowing the compiler to provide a fix-it. + +#### 2. Rename `swiftLanguageVersions` property to `swiftLanguageModes` +Rename the public `Package` property `swiftLanguageVersions` to `swiftLanguageModes`. Add a `swiftLanguageVersions` computed property that accesses `swiftLanguageModes` for backwards compatibility. + + +#### 3. Rename `SwiftVersion` enum to `SwiftLanguageMode` +Rename the `SwiftVersion` enum to `SwiftLanguageMode`. Add `SwiftVersion` as a type alias for backwards compatibility. + + +#### 4. Add `swiftLanguageMode()` to `SwiftSetting` + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.0) + public static func swiftLanguageMode( + _ mode: SwiftLanguageMode, + _ condition: BuildSettingCondition? = nil + ) +``` + +The changes in [SE-0435: Swift Language Version Per Target](https://github.com/apple/swift-evolution/blob/main/proposals/0435-swiftpm-per-target-swift-language-version-setting.md) have been implemented and released in pre-release versions of Swift 6.0. This proposal will add `swiftLanguageMode()` as a `SwiftSetting` and deprecate the `swiftLanguageVersion()` setting added by SE-0435 with a `renamed` annotation. + +#### Naming note + +In Swift PM manifests, multiple languages are supported. For clarity, there is existing precedent for parameter and enum type names to have a language name prefix. + +For example the Package `init` method currently includes: + +```swift + ... + swiftLanguageVersions: [SwiftVersion]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil + ... +``` + +For clarity and to follow the existing precedent, the proposed Swift PM APIs will be appropriately capitalized versions of "Swift language mode". + +## Detailed design + +### New swift compiler option +A new `-language-mode` option will be added with the same behavior as the existing `-swift-version` option. + +The `-swift-version` option will continue to work as it currently does, preserving backwards compatibility. + +The `-language-mode` option will be presented in the compiler help. + +The `-swift-version` option will be suppressed from the top-level help of the compiler. + +### Swift Package Manager +Proposed Swift Package Manager API changes are limited to manifests \>= 6.0: + +### New Package init method and deprecated init method +A new `init` method will be added to `Package` that renames the `swiftLanguageVersions` parameter to `swiftLanguageModes` with the type of the parameter being an optional array of `SwiftLanguageMode` values instead of `SwiftVersion` values: + +```swift +@available(_PackageDescription, introduced: 6) +Package( + name: String, + defaultLocalization: [LanguageTag]? = nil. + platforms: [SupportedPlatform]? = nil, + products: [Product] = [], + dependencies: [Package.Dependency] = [], + targets: [Target] = [], + swiftLanguageModes: [SwiftLanguageMode]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil +) +``` + + +The existing init method will be marked as `deprecated` and `renamed`, allowing the compiler to provide a fix-it: + +``` +@_disfavoredOverload +@available(_PackageDescription, introduced: 5.3, deprecated: 6, renamed: +"init(name:defaultLocalization:platforms:pkgConfig:providers:products: +dependencies:targets:swiftLanguageModes:cLanguageStandard: +cxxLanguageStandard:)") + public init( + name: String, + ... + swiftLanguageVersions: [SwiftVersion]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil + ) { +``` + +See the **Source compatibility** section for more details about this change. + +### Rename `swiftLanguageVersions` property to `swiftLanguageModes` +Rename the `Package` public property `swiftLanguageVersions` to `swiftLanguageModes`. Introduce a computed property named `swiftLanguageModes` that accesses the renamed stored property for backwards compatibility. + +The computed property will be annotated as `deprecated` in Swift 6 and `renamed` to `swiftLanguageModes`. + +For packages with swift tools version less than 6.0, accessing the `swiftLanguageModes` property will continue to work. + +For 6.0 and later, that access will be a warning with a fix-it to use the new property name. + +```swift + @available(_PackageDescription, deprecated: 6.0, renamed: "swiftLanguageModes") + public var swiftLanguageVersions: [SwiftVersion]? { + get { swiftLanguageModes } + set { swiftLanguageModes = newValue } + } +``` + +See the **Source compatibility** section for more details about this change. + +### Rename `SwiftVersion` enum to `SwiftLanguageMode` +Rename the existing `SwiftVersion` enum to `SwiftLanguageMode` with `SwiftVersion` added back as a type alias for backwards compatibility. The type alias will be deprecated in 6.0 with a `renamed` annotation to `SwiftLanguageMode`. + +This change will not affect serialization of PackageDescription types. Serialization is handled by converting PackageDescription types into separate, corresponding Codable types. The existing serialization types will remain as-is. + + +### Add `swiftLanguageMode()` to `SwiftSetting` +Add a new `swiftLanguageMode()` static function to `SwiftSetting` with the same behavior of `swiftLanguageBehavior()` added by [SE-0435](https://github.com/apple/swift-evolution/blob/main/proposals/0435-swiftpm-per-target-swift-language-version-setting.md): + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.0) + public static func swiftLanguageMode( + _ mode: SwiftLanguageMode, + _ condition: BuildSettingCondition? = nil + ) +``` + +The name of the function is `swiftLanguageMode()` instead of `languageMode()` to keep naming consistent with the `swiftLanguageModes` parameter of the Package init method. The parameter label `mode` is used to follow the precedent set by the existing `interoperabilityMode()` method in `SwiftSetting`. + +Deprecate the `swiftLanguageVersion()` setting added by SE-0435 with a `renamed` annotation to provide a fix-it for developers who have adopted this API in pre-release versions of Swift 6.0.: + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.0, deprecated: 6.0, renamed: "swiftLanguageMode(_:_:)") + public static func swiftLanguageVersion( + _ version: SwiftVersion, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting { + ... + } +} +``` + +## Source compatibility +The new Package `init` method and deprecating the existing `init` method will cause a deprecation warning for package manifests that specify the existing `swiftLanguageVersions` parameter when updating to swift tools version 6.0 + +A search of manifest files in public repositories suggests that about 10% of manifest files will encounter this breakage. + +Because the deprecated `init` method is annotated as `renamed` the compiler will automatically provide a fix-it to update to the new `init` method. + +Renaming the public `swiftLanguageVersions` property of `Package` preserves backwards compatibility by introducing a computed property with that name. The computed property will be marked as `deprecated` in 6.0 and annotated as `renamed` to provide a fix-it. + +Searching manifest files in public repositories suggests that accessing the `swiftLanguageVersions` property directly is not common. + +The `SwiftVersion` type alias will be deprecated in favor of the `SwiftLanguageMode` enum. This also will provide a fix-it. + +Finally the `swiftLanguageVersion()` method in `SwiftSetting` added as part of SE-0435 will be deprecated in favor of the `swiftLanguageMode()` method with a fix-it. + +## ABI compatibility +This proposal has no effect on ABI stability. + +## Future directions +This proposal originally included the proposed addition of a `languageMode()` compilation condition to further standardize on the terminology and allow the compiler to check for valid language mode values. + +That functionality has been removed from this proposal with the intent to pitch it separately. Doing so keeps this proposal focused on the tools, including the source breaking API changes. The future direction is purely additive and would focus on the language change. + +## Alternatives considered + +### Alternate terminology + +In the pitch phase, a number of terms were suggested as alternatives for _language mode_. Some concerns were also expressed that the term _language mode_ may be too broad and cause future ambiguity. + +The intent of this proposal is to formalize established terminology in tool options and APIs. + +The term _language mode_ is a long-established term of art in the Swift community to describe this functionality in the compiler. + +This includes the [blog post](https://www.swift.org/blog/swift-4.0-released/) announcing the functionality as part of the release of Swift 4 in 2017 (emphasis added): + +> With Swift 4, you may not need to modify your code to use the new version of the compiler. The compiler supports two _language modes_… + +> The _language mode_ is specified to the compiler by the -swift-version flag, which is automatically handled by the Swift Package Manager and Xcode. +> +> One advantage of these _language modes_ is that you can start using the new Swift 4 compiler and migrate fully to Swift 4 at your own pace, taking advantage of new Swift 4 features, one module at a time. + +Usage also includes posts in the last year from LSG members about Swift 6 language mode: + +- [Design Priorities for the Swift 6 Language Mode](https://forums.swift.org/t/design-priorities-for-the-swift-6-language-mode/62408/27) +- [Progress toward the Swift 6 language mode](https://forums.swift.org/t/progress-toward-the-swift-6-language-mode/68315) + +Finally, searching for "language modes" and "language mode" in the Swift forums found that at least 90% of the posts use the term in this way. Many of the remaining posts use the term in the context of Clang. + +#### Alternatives mentioned + +Alternate terms raised as possibilities were: + - _Edition_: a term used by Rust for a similar concept + - _Standard_: similar to C or C++ standards + Language standards tend to be associated with a written specification, which Swift does not currently have. + Using the term _standard_ would preclude using the term in the future to describe a formal standard. + + +#### Potential overload of _language mode_ +Some reviewers raised concern that Embedded Swift could be considered a language mode and lead to future ambiguity. + +On consideration, this concern is mitigated in two ways: + +1. As noted above, the term _language mode_ is a well-established term of art in the Swift community. + +2. The term _Embedded Swift_ already provides an unambiguous, concise name that can be discussed without requiring a reference to modes. + + This is demonstrated by the following hypothetical FAQ based on the Embedded Swift vision document: +> _What is Embedded Swift?_ +> Embedded Swift is a subset of Swift suitable for restricted environments such as embedded and low-level environments. +> +> _How do you enable Embedded Swift?_ +> Pass the `-embedded` compiler flag to compile Embedded Swift. + +Considering these alternatives, it seems likely that introducing a new term to replace the long-established term _language mode_ and potentially giving the existing term a new meaning would lead to more ambiguity than keeping and formalizing the existing meaning of _language mode_. + +## Acknowledgments + +Thank you to Pavel Yaskevich for providing guidance and assistance in the implementation of this proposal. diff --git a/proposals/0442-allow-taskgroup-childtaskresult-type-to-be-inferred.md b/proposals/0442-allow-taskgroup-childtaskresult-type-to-be-inferred.md new file mode 100644 index 0000000000..8d64c12134 --- /dev/null +++ b/proposals/0442-allow-taskgroup-childtaskresult-type-to-be-inferred.md @@ -0,0 +1,191 @@ +# Allow TaskGroup's ChildTaskResult Type To Be Inferred + +* Proposal: [SE-0442](0442-allow-taskgroup-childtaskresult-type-to-be-inferred.md) +* Author: [Richard L Zarth III](https://github.com/rlziii) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift#74517](https://github.com/apple/swift/pull/74517) +* Review: ([pitch](https://forums.swift.org/t/allow-taskgroups-childtaskresult-type-to-be-inferred/72175))([review](https://forums.swift.org/t/se-0442-allow-taskgroups-childtaskresult-type-to-be-inferred/73397))([acceptance](https://forums.swift.org/t/accepted-se-0422-allow-taskgroups-childtaskresult-type-to-be-inferred/73747)) + +## Introduction + +`TaskGroup` and `ThrowingTaskGroup` currently require that one of their two generics (`ChildTaskResult`) always be specified upon creation. Due to improvements in closure parameter/result type inference introduced by [SE-0326](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0326-extending-multi-statement-closure-inference.md) this can be simplified by allowing the compiler to infer both of the generics in most cases. + +## Motivation + +Currently to create a new task group, there are two generics involved: `ChildTaskResult` and `GroupResult`. The latter can often be inferred in many cases, but the former must always be supplied as part of either the `withTaskGroup(of:returning:body:)` or `withThrowingTaskGroup(of:returning:body:)` function. For example: + +```swift +let messages = await withTaskGroup(of: Message.self) { group in + for id in ids { + group.addTask { await downloadMessage(for: id) } + } + + var messages: [Message] = [] + for await message in group { + messages.append(message) + } + return messages +} +``` + +The type of `messages` (which is the `GroupResult` type) is correctly inferred as `[Message]`. However, the return value of the `addTask(...)` closures is not inferred and currently must be supplied to the `of:` parameter of the `withTaskGroup(of:returning:body:)` function (e.g. `Message`). The correct value of the generic can be non-intuitive for new users to the task group APIs. + +Note that `withDiscardingTaskGroup(returning:body:)` and `withThrowingDiscardingTaskGroup(returning:body:)` do not have `ChildTaskResult` generics since their child tasks must always be of type `Void`. + +## Proposed solution + +Adding a default `ChildTaskResult.self` argument for `of childTaskResultType: ChildTaskResult.Type` will allow `withTaskGroup(of:returning:body:)` to infer the type of `ChildTaskResult` in most cases. The currently signature of `withTaskGroup(of:returning:body:)` looks like: + +```swift +public func withTaskGroup( + of childTaskResultType: ChildTaskResult.Type, + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout TaskGroup) async -> GroupResult +) async -> GroupResult where ChildTaskResult : Sendable +``` + +The function signature of `withThrowingTaskGroup(of:returning:body:)` is nearly identical, so only `withTaskGroup(of:returning:body:)` will be used as an example throughout this proposal. + +Note that the `GroupResult` generic is inferable via the `= GroupResult.self` default argument. This can also be applied to `ChildTaskResult` as of [SE-0326](0326-extending-multi-statement-closure-inference.md). As in: + +```swift +public func withTaskGroup( + of childTaskResultType: ChildTaskResult.Type = ChildTaskResult.self, // <- Updated. + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout TaskGroup) async -> GroupResult +) async -> GroupResult where ChildTaskResult : Sendable +``` + +This allows the original example above to be simplified: + +```swift +// No need for `(of: Message.self)` like before. +let messages = await withTaskGroup { group in + for id in ids { + group.addTask { await downloadMessage(for: id) } + } + + var messages: [Message] = [] + for await message in group { + messages.append(message) + } + return messages +} +``` + +In the above snippet, `ChildTaskResult` is inferred as `Message` and `GroupResult` is inferred as `[Message]`. Not needing to specify the generics explicitly will simplify the API design for these functions and make it easier for new users of these APIs, as it can currently be confusing to understand the differences between `ChildTaskResult` and `GroupResult`. This can be especially true when one or both of those is `Void`. For example: + +```swift +let logCount = await withTaskGroup(of: Void.self) { group in + for id in ids { + group.addTask { await logMessageReceived(for: id) } + } + + return ids.count +} +``` + +In the above example, it can be confusing (and not intuitive) to know that `Void.self` is needed for `ChildTaskResult` and the compiler does not currently give great hints for what that type should be or steering the user into fixing the generic argument if it is mismatched (for example, if the user swaps `Int.self` for `Void.self` in the above example). With the proposed solution, the above can become the following example with type inference used for both generic arguments: + +```swift +let logCount = await withTaskGroup { group in + for id in ids { + group.addTask { await logMessageReceived(for: id) } + } + + return ids.count +} +``` + +## Detailed design + +Because type inference is top-down, it relies on the first statement that uses `group` to infer the generic arguments for `ChildTaskResult`. Therefore, it is possible to get a compiler error by creating a task group where the first use of `group` does not use `addTask(...)`, like so: + +```swift +// Expect `ChildTaskResult` to be `Void`... +await withTaskGroup { group in // Generic parameter 'ChildTaskResult' could not be inferred + // Since `addTask(...)` wasn't the first statement, this fails to compile. + group.cancelAll() + + for id in ids { + group.addTask { await logMessageReceived(for: id) } + } +} +``` + +This can be fixed by going back to specifying the generic like before: + +```swift +// Expect `ChildTaskResult` to be `Void`... +await withTaskGroup(of: Void.self) { group in + group.cancelAll() + + for id in ids { + group.addTask { await logMessageReceived(for: id) } + } +} +``` + +However, this is a rare case in general since `addTask(...)` is generally the first `TaskGroup`/`ThrowingTaskGroup` statement in a task group body. + +It is also possible to create a compiler error by returning two different values from an `addTask(...)` closure: + +```swift +await withTaskGroup { group in + group.addTask { await downloadMessage(for: id) } + group.addTask { await logMessageReceived(for: id) } // Cannot convert value of type 'Void' to closure result type 'Message' +} +``` + +The compiler will already give a good error message here, since the first `addTask(...)` statement is what determined (in this case) that the `ChildTaskResult` generic was set to `Message`. If this needs to be made more clear (instead of being inferred), the user can always specify the generic directly as before: + +```swift +await withTaskGroup(of: Void.self) { group in + // Now the error has moved here since the generic was specified up front... + group.addTask { await downloadMessage(for: id) } // Cannot convert value of type 'Message' to closure result type 'Void' + group.addTask { await logMessageReceived(for: id) } +} +``` + +## Source compatibility + +Omitting the `of childTaskResultType: ChildTaskResult.Type` parameter for both `withTaskGroup(of:returning:body:)` and `withThrowingTaskGroup(of:returning:body:)` is new, and therefore the inference of `ChildTaskResult` is opt-in and does not break source compatibility. + +## ABI compatibility + +No ABI impact since adding a default argument value is binary compatible change. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source +code with no deployment constraints and without affecting source or ABI +compatibility. + +## Future directions + +### TaskGroup APIs Without the "with..." Closures + +While not possible without more compiler features to enforce the safety of a task group not escaping a context, and having to await all of its results at the end of a "scope..." it is an interesting future direction to explore a `TaskGroup` API that does not need to resort to "with..." methods, like this: + +```swift +// Potential long-term direction that might be possible: +func test() async { + let group = TaskGroup() + group.addTask { /* ... */ } + + // Going out of scope would have to imply `group.waitForAll()`... +} +``` + +If we were to explore such API, the type inference rules would be somewhat different, and a `TaskGroup` would likely be initialized more similarly to a collection: `TaskGroup`. + +This proposal has no impact on this future direction, and can be accepted as is, without precluding future developments in API ergonomics like this. + +## Alternatives considered + +The main alternative is to do nothing; as in, leave the `withTaskGroup(of:returning:body:)` and `withThrowingTaskGroup(of:returning:body:)` APIs like they are and require the `ChildTaskResult` generic to always be specified. + +## Acknowledgments + +Thank you to both Konrad Malawski ([@ktoso](https://github.com/ktoso)) and Pavel Yaskevich ([@xedin](https://github.com/xedin)) for confirming the viability of this proposal/idea, and for Konrad Malawski ([@ktoso](https://github.com/ktoso)) for helping to review the proposal and implementation. diff --git a/proposals/0443-warning-control-flags.md b/proposals/0443-warning-control-flags.md new file mode 100644 index 0000000000..eb24169971 --- /dev/null +++ b/proposals/0443-warning-control-flags.md @@ -0,0 +1,242 @@ + +# Precise Control Flags over Compiler Warnings + +* Proposal: [SE-0443](0443-warning-control-flags.md) +* Authors: [Doug Gregor](https://github.com/douggregor), [Dmitrii Galimzianov](https://github.com/DmT021) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift#74466](https://github.com/swiftlang/swift/pull/74466) +* Review: ([pitch](https://forums.swift.org/t/warnings-as-errors-exceptions/72925)) ([review](https://forums.swift.org/t/se-0443-precise-control-flags-over-compiler-warnings/74116)) ([acceptance](https://forums.swift.org/t/accepted-se-0443-precise-control-flags-over-compiler-warnings/74377)) +* Previous revisions: [1](https://github.com/swiftlang/swift-evolution/blob/57fe29d5d55edb85b14c153b7f4cbead6b6539eb/proposals/0443-warning-control-flags.md), [2](https://github.com/swiftlang/swift-evolution/blob/7b12899ad0d96002c793d33ef8109ec47c5d256f/proposals/0443-warning-control-flags.md) + +## Introduction + +This proposal introduces new compiler options that allow fine-grained control over how the compiler emits certain warnings: as warnings or as errors. + +## Motivation + +The current compiler options for controlling how warnings are emitted are very inflexible. Currently, the following options exist: +- `-warnings-as-errors` - upgrades all warnings to errors +- `-no-warnings-as-errors` - cancels the upgrade of warnings to errors +- `-suppress-warnings` - disables the emission of all warnings + +This lack of flexibility leads to situations where users who want to use `-warnings-as-errors` find themselves unable to do so, or unable to upgrade to a new version of the compiler or SDK until all newly diagnosed warnings are resolved. The most striking example of this is deprecation warnings for certain APIs, though they are not limited to them. + +## Proposed solution + +This proposal suggests adding new options that will allow the behavior of warnings to be controlled based on their diagnostic group. +- `-Werror ` - upgrades warnings in the specified group to errors +- `-Wwarning ` - indicates that warnings in the specified group should remain warnings, even if they were previously suppressed or upgraded to errors + +The `` parameter is a string identifier of the diagnostic group. + +A diagnostic group is a stable identifier for an error or warning. It is an abstraction layer over the diagnostic identifiers used within the compiler. This is necessary because diagnostics within the compiler may change, but we need to provide stable user-facing identifiers for them. + +A diagnostic group may include errors, warnings, or other diagnostic groups. For example, the `DeprecatedDeclaration` diagnostic group includes warnings related to the use of an API marked with the `@available(..., deprecated: ...)` attribute. The `Deprecated` diagnostic group includes the `DeprecatedDeclaration` group and other groups related to deprecation. + +Diagnostic groups may expand over time, but they can never become narrower. When a new diagnostic is added to the compiler, it is either included in an existing group or a new group is created for it, which in turn can also be included in one of the broader groups, if appropriate. + +The order in which these flags are specified when invoking the compiler is important. If two or more options change the behavior of the same warning, we follow the rule "the last one wins." + +We also retain the existing compiler options but modify their handling algorithm so that they are considered in the general list with the new options and follow the "last one wins" rule as well. + +Thus, for example, you can use the combination `-warnings-as-errors -Wwarning Deprecated`, which will upgrade all warnings to errors except for those in the `Deprecated` group. However, if these flags are specified in the reverse order(`-Wwarning Deprecated -warnings-as-errors`) it will be interpreted as upgrading all warnings to errors, as the `-warnings-as-errors` flag is the last one. + +We are also introducing a new compiler flag, `-print-diagnostic-groups`, to display the names of diagnostic groups along with the textual representation of the warnings. When used, the warning message will be followed by the name of the narrowest group that includes that warning, enclosed in square brackets. For example: +``` +main.swift:33:1: warning: 'f()' is deprecated [#DeprecatedDeclaration] +``` + +## Detailed design + +### Diagnostic groups + +Diagnostic groups form an acyclic graph with the following properties: + +- A warning or error can only be included in one diagnostic group. This artificial restriction is introduced to solve two main problems: + - When using the `-print-diagnostic-groups` flag, it would be inconvenient if a warning corresponded to multiple groups. + - Documentation lookup will also be easier for the user if a diagnostic has only one identifier. + +- A diagnostic group may include any number of other diagnostic groups. This will allow organizing groups into sets with similar meanings but different specific diagnostics. For example, the warnings `DeprecatedDeclaration` and `UnsafeGlobalActorDeprecated` are part of the supergroup `Deprecated`. + +- A diagnostic group can be included in any number of diagnostic groups. This allows expressing the membership of a group in multiple supergroups, where appropriate. For example, the group `UnsafeGlobalActorDeprecated` is part of both the `Deprecated` and `Concurrency` groups. + +The internal structure of the graph may change to some extent. However, the set of diagnostics included in a diagnostic group (directly or transitively) should not shrink. There are two typical situations where the graph structure may change: +- When adding a new diagnostic to the compiler, consider creating a new group corresponding to that diagnostic. If the new group is created it can also be included in one or more existing groups if it belongs to them. For example, it is expected that the `Deprecated` group will continuously include new subgroups. +- If an existing diagnostic is split into more specific versions, and we want to allow users to use the more specific version in compiler options, a separate group is created for it, which **must** be included in the group of the original diagnostic. + + For example, suppose we split the `DeprecatedDeclaration` warning into a general version and a specialized version `DeprecatedDeclarationSameModule`, which the compiler emits if the deprecated symbol is declared in the same module. In this case, the `DeprecatedDeclarationSameModule` group must be added to the `DeprecatedDeclaration` group to ensure that the overall composition of the `DeprecatedDeclaration` group does not change. The final structure should look like this: + ``` + DeprecatedDeclaration (group) + ├─ DeprecatedDeclaration (internal diag id) + └─ DeprecatedDeclarationSameModule (group) + └─ DeprecatedDeclarationSameModule (internal diag id) + ``` + Thus, invoking the compiler with the `-Werror DeprecatedDeclaration` parameter will cover both versions of the warning, and the behavior will remain unchanged. At the same time, the user can control the behavior of the narrower `DeprecatedDeclarationSameModule` group if they want to. + +### Compiler options evaluation + +Each warning in the compiler is assigned one of three behaviors: `warning`, `error`, or `suppressed`. +Compiler options for controlling the behavior of groups are now processed as a single list. These options include: +``` +-Werror +-Wwarning +-warnings-as-errors +-no-warnings-as-errors +``` +When these options are passed to the compiler, we sequentially apply the specified behavior to all warnings within the specified group from left to right. For `-warnings-as-errors` and `-no-warnings-as-errors`, we apply the behavior to all warnings. + +Examples of option combinations: +- `-warnings-as-errors -Wwarning Deprecated` + + Warnings from the `Deprecated` group will be kept as warnings, but all the rest will be upgraded to errors. + +- `-Werror Deprecated -Wwarning DeprecatedDeclaration` + + Warnings from the `DeprecatedDeclaration` group will remain as warnings. Other warnings from the `Deprecated` group will be upgraded to errors. All others will be kept as warnings. + +It’s crucial to understand that the order in which these flags are applied can significantly affect the behavior of diagnostics. The rule is "the last one wins", meaning that if multiple flags apply to the same diagnostic group, the last one specified on the command line will determine the final behavior. + +It is also important to note that the order matters even if the specified groups are not explicitly related but have a common subgroup. +For example, as mentioned above, the `UnsafeGlobalActorDeprecated` group is part of both the `Deprecated` and `Concurrency` groups. So the order in which options for the `Deprecated` and `Concurrency` groups are applied will change the final behavior of the `UnsafeGlobalActorDeprecated` group. Specifically: + +- `-Wwarning Deprecated -Werror Concurrency` will make it an error, +- `-Werror Concurrency -Wwarning Deprecated` will keep it as a warning. + +#### Interaction with `-suppress-warnings` + +This proposal deliberately excludes `-suppress-warnings` and its group-based counterpart from the new unified model. We retain the behavior of the existing `-suppress-warnings` flag but forbid its usage with the new options. The following rules will be applied: + +- It is forbidden to combine `-suppress-warnings` with `-Wwarning` or `-Werror`. The compiler will produce an error if these options are present in the command line together. +- It is allowed to be combined with `-no-warnings-as-errors`. The current compiler behavior permits the usage of `-no-warnings-as-errors` or `-warnings-as-errors -no-warnings-as-errors` with `-suppress-warnings`. We will maintain this behavior. +- It remains position-independent. Whenever `-no-warnings-as-errors` and `-suppress-warnings` are combined, `-suppress-warnings` will always take precedence over `-no-warnings-as-errors`, regardless of the order in which they are specified. + +### Usage of `-print-diagnostic-groups` and `-debug-diagnostic-names` + +As mentioned earlier, we are adding support for the `-print-diagnostic-groups` compiler option, which outputs the group name in square brackets. + +A similar behavior already exists in the compiler and is enabled by the `-debug-diagnostic-names` option, but it prints the internal diagnostic identifiers used in the compiler. For example: +```swift +@available(iOS, deprecated: 10.0, renamed: "newFunction") +func oldFunction() { ... } + +oldFunction() +``` +When compiled with the `-debug-diagnostic-names` option, the following message will be displayed: +``` +'oldFunction()' is deprecated: renamed to 'newFunction' [#RenamedDeprecatedDeclaration] +``` +The string `RenamedDeprecatedDeclaration` is the internal identifier of this warning, not the group. Accordingly, it is not supported by the new compiler options. + +When compiling the same code with the `-print-diagnostic-groups` option, the following message will be displayed: +``` +'oldFunction()' is deprecated: renamed to 'newFunction' [#DeprecatedDeclaration] +``` +Here, the string `DeprecatedDeclaration` is the diagnostic group. + +Often, group names and internal diagnostic identifiers coincide, but this is not always the case. + +We retain support for `-debug-diagnostic-names` in its current form. However, to avoid confusion between diagnostic IDs and diagnostic groups, we prohibit the simultaneous use of these two options. + +## Source compatibility + +This proposal has no effect on source compatibility. + +## ABI compatibility + +This proposal has no effect on ABI compatibility. + +## Implications on adoption + +The adoption of diagnostic groups and the new compiler options will provide a foundation for flexible and precise control over warning behavior. However, to make this useful to end-users, significant work will be needed to mark existing diagnostics in diagnostic groups. It will also be necessary to develop a process for maintaining the relevance of diagnostic groups when new diagnostics are introduced in the compiler. + +## Future directions + +### Support in the language + +While diagnostic groups are introduced to support the compiler options, it may be possible in the future to standardize the structure of the group graph itself. This could open up the possibility of using these same identifiers in the language, implementing something analogous to `#pragma diagnostic` or `[[attribute]]` in C++. It could also address suppressing warnings entirely, which isn't covered by this proposal. However, such standardization and the design of new language constructs go far beyond the scope of this proposal, and we need to gain more experience with diagnostic groups before proceeding with this. + +### Support in SwiftPM + +If this proposal is accepted, it would make sense to support these parameters in SwiftPM as well, allowing the behavior of warnings to be conveniently specified in SwiftSetting. + +## Alternatives considered + +### Alternatives to diagnostic groups +#### Status quo +The lack of control over the behavior of specific diagnostics forces users to abandon the `-warnings-as-errors` compiler option and create ad-hoc compiler wrappers that filter its output. + +#### Using existing diagnostic identifiers +Warnings and errors in Swift can change as the compiler evolves. +For example, one error might be renamed or split into two that are applied in different situations to improve the clarity of the text message depending on the context. Such a change would result in a new ID for the new error variant. + +The example of `DeprecatedDeclarationSameModule` illustrates this well. If we used the warning ID, the behavior of the compiler with the `-Wwarning DeprecatedDeclaration` option would change when a new version of the warning is introduced, as this warning would no longer be triggered for the specific case of the same module. + +Therefore, we need a solution that allows us to modify errors and warnings within the compiler while providing a reliable mechanism for identifying diagnostics that can be used by the user. + +#### Flat list instead of a graph + +To solve this problem, we could use an additional alias-ID for diagnostics that does not change when the main identifier changes. + +Suppose we split the `DeprecatedDeclaration` diagnostic into a generic variant and `DeprecatedDeclarationSameModule`. To retain the existing name for the new variant, we could describe these two groups as +``` +DeprecatedDeclaration (alias: DeprecatedDeclaration) +DeprecatedDeclarationSameModule (alias: DeprecatedDeclaration) +``` +However, this solution would not allow specifying the narrower `DeprecatedDeclarationSameModule` or the broader group `Deprecated`. + +#### Using multiple alias IDs for diagnostics +To express a diagnostic's membership in multiple groups, we could allow multiple alias-IDs to be listed. +``` +DeprecatedDeclaration aliases: + DeprecatedDeclaration + Deprecated +DeprecatedDeclarationSameModule aliases: + DeprecatedDeclarationSameModule + DeprecatedDeclaration + Deprecated +``` +However, such a declaration lacks structure and makes it difficult to understand which alias-ID is the most specific. + +### Alternative names for the compiler options + +During the design process, other names for the compiler options were considered, which were formed as the singular form of the existing ones: +| Plural | Singular | +|--------------------------|--------------------------------| +| `-warnings-as-errors` | `-warning-as-error ` | +| `-no-warnings-as-errors` | `-no-warning-as-error ` | + +In Clang, diagnostic behavior is controlled through `-W...` options, but the format suffers from inconsistency. We adopt the `-W` prefix while making the format consistent. +| Clang | Swift | +|-------------------|----------------------| +| `-W` | `-Wwarning ` | +| `-Wno-` | | +| `-Werror=` | `-Werror ` | + +The option name `-Wwarning` is much better suited when it comes to enabling suppressed-by-default warnings. Today we have several of them behind dedicated flags like `-driver-warn-unused-options` and `-warn-concurrency`. It might be worth having a common infrastructure for warnings that are suppressed by default. + +### Alternative format for `-print-diagnostic-groups` + +Theoretically, we could allow the simultaneous use of `-debug-diagnostic-names` and `-print-diagnostic-groups`, but this would require choosing a different format for printing diagnostic groups. + +Since `-debug-diagnostic-names` has been available in the compiler for a long time, we proceed from the fact that there are people who rely on this option and its format with square brackets. + +To avoid overlap, we would need to use a different format, for example: +``` +'foo()' is deprecated [#DeprecatedDeclaration] [group:#DeprecatedDeclaration] +``` + +However, even this does not eliminate the possibility of breaking code that parses the compiler's output. + +Moreover, `-print-diagnostic-groups` provides a formalized version of the same functionality using identifiers suitable for user use. And thus it should supersede the usages of `-debug-diagnostic-names`. Therefore, we believe the best solution would be to use the same format for `-print-diagnostic-groups` and prohibit the simultaneous use of these two options. + +## Revision History + +- Revisions based on review feedback: + - `-Wsuppress` was excluded from the proposal. + - `-suppress-warnings` was excluded from the unified model and addressed separately by forbidding its usage with the new flags. + - The guideline in the "Diagnostic Groups" subsection for adding a new diagnostic has been softened to a consideration. + +## Acknowledgments + +Thank you to [Frederick Kellison-Linn](https://forums.swift.org/u/Jumhyn) for the idea of addressing the `-suppress-warnings` behavior without incorporating it into the new model. diff --git a/proposals/0444-member-import-visibility.md b/proposals/0444-member-import-visibility.md new file mode 100644 index 0000000000..7904ebc4d5 --- /dev/null +++ b/proposals/0444-member-import-visibility.md @@ -0,0 +1,157 @@ +# Member import visibility + +* Proposal: [SE-0444](0444-member-import-visibility.md) +* Authors: [Allan Shortlidge](https://github.com/tshortli) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 6.1)** +* Bug: [apple/swift#46493](https://github.com/apple/swift/issues/46493) +* Implementation: [apple/swift#72974](https://github.com/apple/swift/pull/72974), [apple/swift#73063](https://github.com/apple/swift/pull/73063) +* Upcoming Feature Flag: `MemberImportVisibility` +* Review: ([pitch](https://forums.swift.org/t/pitch-fixing-member-import-visibility/71432)) ([review](https://forums.swift.org/t/se-0444-member-import-visibility/74519)) ([acceptance](https://forums.swift.org/t/accepted-se-0444-member-import-visibility/74966)) + +## Introduction + +In Swift, there are rules dictating whether the name of a declaration in another module is considered in scope. For example, if you have a program that uses the `swift-algorithms` package and you want to use the global function [chain()](https://github.com/apple/swift-algorithms/blob/33abb694280321a84aa7dc9806de284afb8ca226/Sources/Algorithms/Chain.swift#L287) then you must write `import Algorithms` in the file that references that function or the compiler will consider it out of scope: + +``` swift +// Missing 'import Algorithms' +let chained = chain([1], [2]) // error: Cannot find 'chain' in scope +``` + +The visibility rules for a member declaration, such as a method declared inside of a struct, are different though. When resolving a name to a member declaration, the member is in scope even if the module introducing the member is only *transitively* imported. A transitively imported module could be imported directly in another source file, or it could be a dependency of some direct dependency of your program. This inconsistency may be best understood as a subtle bug rather than an intentional design decision, and in a lot of Swift code it goes unnoticed. However, the import rules for members become more surprising when you consider the members of extensions, since an extension and its nominal type can be declared in different modules. + +This proposal unifies the behavior of name lookup by changing the rules to bring both top-level declarations and members into scope using the same criteria. + +## Motivation + +Suppose you have an app that depends on an external library named `RecipeKit`. To start, your app contains a single source file `main.swift` that imports `RecipeKit`: + +```swift +// main.swift +import RecipeKit + +let recipe = "2 slices of bread, 1.5 tbs peanut butter".parse() +``` + +The interface of `RecipeKit` looks like this: + +```swift +// RecipeKit interface + +public struct Recipe { /*...*/ } + +extension String { + /// Returns the recipe represented by this string. + public func parse() -> Recipe? +} +``` + +Later, you decide to integrate with a new library named `GroceryKit`. You add a second file to your app that imports `GroceryKit`: + +```swift +// Groceries.swift +import GroceryKit + +var groceries = GroceryList() +// ... +``` + +Surprisingly, after adding the second file there's now a compilation error in `main.swift`: + +```swift +// main.swift +import RecipeKit + +let recipe = "2 slices of bread, 1.5 tbs peanut butter".parse() +// error: Ambiguous use of 'parse()' +``` + +The call to `parse()` is now ambiguous because `GroceryKit` happens to also declares its own `parse()` method in an extension on `String`: + +```swift +// GroceryKit interface +public struct GroceryList { /*...*/ } + +extension String { + /// Returns the grocery list represented by this string. + public func parse() -> GroceryList? +} + +``` + +Even though `GroceryKit` was not imported in `main.swift`, its `parse()` method is now a candidate in that file. To resolve the ambiguity, you can add a type annotation to the declaration of the variable `recipe` to give the compiler the additional context it needs to disambiguate the call: + +```swift +let recipe: Recipe? = "2 slices of bread, 1.5 tbs peanut butter".parse() // OK +``` + +This example demonstrates why Swift's existing "leaky" member visibility is undesirable. Although the fix for the new error is relatively simple in this code, providing disambiguation context to the compiler is not always so straightforward. Additionally, the fact that some declarations from `GroceryKit` are now visible in `main.swift` contradicts developer expectations, since visibility rules for top level declarations do not behave this way. This idiosyncrasy in Swift's import visibility rules harms local reasoning and results in confusing errors. + +## Proposed solution + +In a future language version, or whenever the `MemberImportVisibility` feature is enabled, both member declarations and top level declarations should be resolved from the same set of visible modules in a given source file. + +## Detailed design + +A reference to a member in a source file will only be accepted if that member is declared in a module that is contained in the set of visible modules for that source file. According to the existing rules for top level name lookup, a module is visible if any of the following statements are true: + +- The module is directly imported. In other words, some import statement in the source file names the module explicitly. +- The module is directly imported from the bridging header. +- The module is in the set of modules that is re-exported by any module that is either directly imported in the file or directly imported in the bridging header. + +A re-exported module is one that another module makes visible to any client that imports it. Clang modules list the modules that they re-export in their modulemap files, and it is common for a Clang module to re-export every module it imports using `export *`. There is no officially supported way to re-export a module from a Swift module, but the compiler-internal `@_exported` attribute is sometimes used for this purpose. Re-exports are also transitive, so if module `A` re-exports module `B`, and module `B` re-exports module `C`, then declarations from `A`, `B`, and `C` are all in scope in a file that only imports `A` directly. + +Note that there are some imports that are added to every source file implicitly by the compiler for normal programs. The implicitly imported modules include the standard library and the module being compiled. As a subtle consequence implicitly importing the current module, any module that the current module re-exports in any of its source files will be considered visible in every other source file. + +## Source compatibility + +The proposed change in behavior is source breaking because it adds stricter requirements to name lookup. There is much existing Swift code that will need to be updated to adhere to these new requirements, either by introducing additional import statements in some source files or by reorganizing code among files. This change in behavior therefore must be opt-in, which is why it should be limited to a future language mode with an upcoming feature identifier that allows opt-in with previous language modes. + +## ABI compatibility + +This change does not affect ABI. + +## Implications on adoption + +To make it easier to migrate to the new language mode, the compiler can attempt to identify whether a member reference would resolve to a member declared in a transitively imported module and emit a fix-it to suggest adding a direct import to resolve the errors caused by the stricter look up rules: + +```swift +// In this example, RecipeKit is imported in another file + +// note: add import of module 'RecipeKit' + +let recipe = "1 scoop ice cream, 1 tbs chocolate syrup".parse() +// error: instance method 'parse()' is inaccessible due to missing import of defining module 'RecipeKit' +``` + +With these fix-its, the burden of updating source code to be compatible with the new language mode should be significantly reduced. + +This feature will have some impact on source compatibility with older compilers and previous language modes. Adding new direct imports of modules that were previously only transitively imported is a backward compatible change syntactically. However, it's possible that source code could depend on the new language mode in order to make the code unambiguous. In these cases additional measures, like adding explicit type annotations, might be required to maintain backward compatibility with previous language modes. + +## Future directions + +#### Add module qualification syntax for extension members + +This proposal seeks to give developers explicit control over which members are visible in a source file because this control can be used to prevent and resolve ambiguities that arise when different modules declare conflicting members in extensions. With this proposal implemented, if an extension member ambiguity still arises in a source file then the developer has the option of curating the imports in that file to resolve the ambiguity. This may work in some situations, but in others it may be awkward to refactor code in order to avoid importing a module that introduces a conflict. For these cases it would be useful to have a syntax that unambiguously identifies the desired extension member at the use site. For example, here's a hypothetical syntax for explicitly calling the `parse()` method declared in the module `RecipeKit`: + +```swift +let recipe = "...".RecipeKit::parse() +``` + +#### Implement rules for retroactive conformance visibility + +A typical protocol conformance in Swift is declared in either the module that declares the protocol or the module of the conforming type and this allows the compiler to enforce that conformances are globally unique. However, Swift also allows conformances to be "retroactive", which means that the conformance is declared in some third module and cannot be guaranteed to be unique. Retroactive conformances make it possible for multiple declarations of the same protocol conformance from different dependency modules to be simultaneously visible to the compiler. Currently, there is no defined way to influence which conformance declaration gets chosen when such an ambiguity arises, but a rule similar to the one in this proposal could be implemented for conformance lookup as well. For instance, a direct import of a module could be required in order to bring the conformances from the module into scope in a source file. + +#### Improve lookup for operators and precedence groups + +Swift has bespoke rules for looking up operators and operator precedence groups that differ in important ways from the rules for both top-level names and members. Documenting the existing rules and reforming them to bring more consistency to name lookup in Swift would be worthy of a future proposal. + +## Alternatives considered + +#### Introduce module qualification syntax for extension members instead + +One alternative approach to the problem would be to rely exclusively on a new syntax for disambiguation of extension members (as discussed in Future directions). The limitation of that approach is that alone it only empowers the developer to solve conflicts *reactively*. On the other hand, the solution provided by this proposal is preventative because it stops unnecessary conflicts from arising in the first place. In the fullness of time, it would be best for both solutions to be available simultaneously. + +## Acknowledgments + +I would like to thank Doug Gregor for providing a proof-of-concept implementation of this pitch. diff --git a/proposals/0445-string-index-printing.md b/proposals/0445-string-index-printing.md new file mode 100644 index 0000000000..f1c2beaa89 --- /dev/null +++ b/proposals/0445-string-index-printing.md @@ -0,0 +1,180 @@ +# Improving `String.Index`'s printed descriptions + +* Proposal: [SE-0445](0445-string-index-printing.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift#75433](https://github.com/swiftlang/swift/pull/75433) +* Review: ([pitch](https://forums.swift.org/t/improving-string-index-s-printed-descriptions/57027)) ([review](https://forums.swift.org/t/se-0445-improving-string-indexs-printed-descriptions/74643)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0445-improving-string-index-s-printed-descriptions/75108)) +* Previous Revision: [v1](https://github.com/swiftlang/swift-evolution/blob/682f7c293a3a05bff3e619c3b479bfb68541fb6e/proposals/0445-string-index-printing.md) + +## Introduction + +This proposal conforms `String.Index` to `CustomDebugStringConvertible`. + +## Motivation + +String indices represent offsets from the start of the string's underlying storage representation, referencing a particular UTF-8 or UTF-16 code unit, depending on the string's encoding. (Most Swift strings are UTF-8 encoded, but strings bridged over from Objective-C may remain in their original UTF-16 encoded form.) + +If you ever tried printing a string index, you probably noticed that the output is gobbledygook: + +```swift +let string = "👋🏼 Helló" + +print(string.startIndex) // ⟹ Index(_rawBits: 15) +print(string.endIndex) // ⟹ Index(_rawBits: 983047) +print(string.utf16.index(after: string.startIndex)) // ⟹ Index(_rawBits: 16388) +print(string.firstRange(of: "ell")!) // ⟹ Index(_rawBits: 655623).. + + + + + + +## Detailed design + +``` +@available(SwiftStdlib 6.1, *) +extension String.Index: CustomDebugStringConvertible {} + +extension String.Index { + @backDeployed(before: SwiftStdlib 6.1) + public var debugDescription: String {...} +} +``` + +## Source compatibility + +The new conformance changes the result of converting a `String.Index` value to a string. This changes observable behavior: code that attempts to parse the result of `String(describing:)` or `String(reflecting:)` can be mislead by the change of format. + +However, the documentation of these interfaces explicitly state that when the input type conforms to none of the standard string conversion protocols, then the result of these operations is unspecified. + +Changing the value of an unspecified result is not considered to be a source incompatible change. + +## ABI compatibility + +The proposal retroactively conforms a previously existing standard type to a previously existing standard protocol. This is technically an ABI breaking change -- on ABI stable platforms, we may have preexisting Swift binaries that assume that `String.Index is CustomDebugStringConvertible` returns `false`, or ones that are implementing this conformance on their own. + +We do not expect this to be an issue in practice. + +## Implications on adoption + +The `String.Index.debugDescription` property is defined to be backdeployable, but the conformance itself is not. (It cannot be.) + +Code that runs on ABI stable platforms will not get the nicer displays when running on earlier versions of the Swift Standard Library. + +```swift +let str = "🐕 Doggo" +print(str.firstRange(of: "Dog")!) +// older stdlib: Index(_rawBits: 327943).. NotEscapable { + let ne = NotEscapable() + borrowingFunc(ne) // OK to pass to borrowing function + let another = ne // OK to make local copies + globalVar = ne // 🛑 Cannot assign ~Escapable type to a global var + return ne // 🛑 Cannot return ~Escapable type +} +``` + +**Note**: +The section ["Returned nonescapable values require lifetime dependency"](#Returns) explains the implications for how you must write initializers. + +Without `~Escapable`, the default for any type is to be escapable. Since `~Escapable` suppresses a capability, you cannot declare it with an extension. + +```swift +// Example: Escapable by default +struct Ordinary { } +extension Ordinary: ~Escapable // 🛑 Extensions cannot remove a capability +``` + +Classes cannot be declared `~Escapable`. + +#### In generic contexts, `~Escapable` suppresses the default Escapable requirement + +When used in a generic context, `~Escapable` allows you to define functions or types that can work with values that might or might not be escapable. +That is, `~Escapable` indicates the default escapable requirement has been suppressed. +Since the values might not be escapable, the compiler must conservatively prevent the values from escaping: + +```swift +func f(_ value: MaybeEscapable) { + // `value` might or might not be Escapable + globalVar = value // 🛑 Cannot assign possibly-nonescapable type to a global var +} +f(NotEscapable()) // Ok to call with nonescapable argument +f(7) // Ok to call with escapable argument +``` + +[SE-0427 Noncopyable Generics](https://github.com/apple/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md) provides more detail on +how suppressible protocols such as `Escapable` are handled in the generic type system. + +**Note:** There is no relationship between `Copyable` and `Escapable`. +Copyable or noncopyable types can be escapable or nonescapable. + +#### Constraints on nonescapable local variables + +A nonescapable value can be freely copied and passed into other functions, including async and throwing functions, as long as the usage can guarantee that the value does not persist beyond the current scope: + +```swift +// Example: Local variable with nonescapable type +func borrowingFunc(_: borrowing NotEscapable) { ... } +func consumingFunc(_: consuming NotEscapable) { ... } +func inoutFunc(_: inout NotEscapable) { ... } +func asyncBorrowingFunc(_: borrowing NotEscapable) async -> ResultType { ... } + +func f() { + var value: NotEscapable + let copy = value // OK to copy as long as copy does not escape + globalVar = value // 🛑 May not assign to global + SomeType.staticVar = value // 🛑 May not assign to static var + async let r = asyncBorrowingFunc(value) // OK to pass borrowing + borrowingFunc(value) // OK to pass borrowing + inoutFunc(&value) // OK to pass inout + consumingFunc(value) // OK to pass consuming + // `value` was consumed above, but NotEscapable + // is Copyable, so the compiler can insert + // a copy to satisfy the following usage: + borrowingFunc(value) // OK +} +``` + +#### Constraints on nonescapable parameters + +A value of nonescapable type received as a parameter is subject to the same constraints as any other local variable. +In particular, a nonescapable `consuming` parameter (and all direct copies thereof) must actually be destroyed during the execution of the function. +This is in contrast to an _escapable_ `consuming` parameter which can be disposed of by being returned or stored to an instance property or global variable. + +#### Types that contain nonescapable values must be nonescapable + +Stored struct properties and enum payloads can have nonescapable types if the surrounding type is itself nonescapable. +Equivalently, an escapable struct or enum can only contain escapable values. +Nonescapable values cannot be stored as class properties, since classes are always inherently escaping. + +```swift +// Example +struct EscapableStruct { + // 🛑 Escapable struct cannot have nonescapable stored property + var nonesc: Nonescapable +} + +enum EscapableEnum { + // 🛑 Escapable enum cannot have a nonescapable payload + case nonesc(Nonescapable) +} + +struct NonescapableStruct: ~Escapable { + var nonesc: Nonescapable // OK +} + +enum NonescapableEnum: ~Escapable { + case nonesc(Nonescapable) // OK +} +``` + +#### Returned nonescapable values require lifetime dependency + +As mentioned earlier, a simple return of a nonescapable value is not permitted: +```swift +func f() -> NotEscapable { // 🛑 Cannot return a nonescapable type + var value: NotEscapable + return value // 🛑 Cannot return a nonescapable type +} +``` + +A future proposal will describe “lifetime dependency annotations” that can relax this requirement by tying the lifetime of the returned value to the lifetime of another binding. +In particular, struct and enum initializers (which build a new value and return it to the caller) cannot be written without some such mechanism. + +#### Globals and static variables cannot be nonescapable + +Nonescapable values must be constrained to some specific local execution context. +This implies that they cannot be stored in global or static variables. + +#### Closures and nonescapable values + +Escaping closures cannot capture nonescapable values. +Nonescaping closures can capture nonescapable values subject to the usual exclusivity restrictions. + +Returning a nonescapable value from a closure will only be possible with explicit lifetime dependency annotations, to be covered in a future proposal. + +#### Nonescapable values and concurrency + +All of the requirements on use of nonescapable values as function parameters and return values also apply to async functions, including those invoked via `async let`. + +The closures used in `Task.init`, `Task.detached`, or `TaskGroup.addTask` are escaping closures and therefore cannot capture nonescapable values. + +#### Conditionally `Escapable` types + +You can define types whose escapability varies depending on their generic arguments. +As with other conditional behaviors, this is expressed by using an extension to conditionally add a new capability to the type: + +```swift +// Example: Conditionally Escapable generic type +// By default, Box is itself nonescapable +struct Box: ~Escapable { + var t: T +} + +// Box gains the ability to escape whenever its +// generic argument is Escapable +extension Box: Escapable where T: Escapable { } +``` + +This can be used in conjunction with other suppressible protocols. +For example, many general library container types will need to be copyable and/or escapable depending on their contents. +Here's a compact way to declare such a type: +```swift +struct Wrapper: ~Copyable, ~Escapable { ... } +extension Wrapper: Copyable where T: Copyable, T: ~Escapable {} +extension Wrapper: Escapable where T: Escapable, T: ~Copyable {} +``` + +## Source compatibility + +The compiler will treat any type without explicit `~Escapable` as escapable. +This matches the current behavior of the language. + +Only when new types are marked as `~Escapable` does this have any impact. + +Adding `~Escapable` to an existing concrete type is generally source-breaking because existing source code may rely on being able to escape values of this type. +Removing `~Escapable` from an existing concrete type is not generally source-breaking since it effectively adds a new capability, similar to adding a new protocol conformance. + +## ABI compatibility + +As above, existing code is unaffected by this change. +Adding or removing a `~Escapable` constraint on an existing type is an ABI-breaking change. + +## Implications on adoption + +Manglings and interface files will only record the lack of escapability. +This means that existing interfaces consumed by a newer compiler will treat all types as escapable. +Similarly, an old compiler reading a new interface will have no problems as long as the new interface does not contain any `~Escapable` types. + +These same considerations ensure that escapable types can be shared between previously-compiled code and newly-compiled code. + +Retrofitting existing generic types so they can support both escapable and nonescapable type arguments is possible with care. + +## Future directions + +#### `Span` family of types + +This proposal is being driven in large part by the needs of the `Span` types that have been discussed elsewhere. +Briefly, this type would provide an efficient universal “view” of array-like data stored in contiguous memory. +Since values of this type do not own any data but only refer to data stored elsewhere, their lifetime must be limited to not exceed that of the owning storage. +We expect to publish a sample implementation and proposal for that type very soon. + +#### Initializers and Lifetime Dependencies + +Nonescapable function parameters may not outlive the function scope. +Consequently, nonescapable values can never be returned from a function. +Nonescapable values come into existence within the body of the initializer. +Naturally, the initializer must return its value, and this creates an exception to the rule. +The parameters to the initializer typically indicate a lifetime that the nonescapable value cannot outlive. +An initializer may, for example, create a nonescapable value that depends on a container variable that is bound to an object with its own lifetime: +```swift +struct Iterator: ~Escapable { + init(container: borrowing Container) { ... } +} + +let container = ... +let iterator = Iterator(container) +consume container // `container` lifetime ends here +use(iterator) // 🛑 'iterator' outlives `container` +``` + +Specifying a dependency from a function parameter to its nonescapable result currently requires an experimental lifetime dependency feature. +With lifetime dependencies, initialization of nonescapable types is safe: misuses similar to the one shown above are compile-time errors. +Adopting new syntax for lifetime dependencies merits a separate, focussed review. +Until then, initialization of nonescapable values remains experimental. + +#### Expanding standard library types + +We expect that many standard library types will need to be updated to support possibly-nonescapable types, including `Optional`, `Array`, `Set`, `Dictionary`, and the `Unsafe*Pointer` family of types. + +Some of these types will require first exploring whether it is possible for the `Collection`, `Iterator`, `Sequence`, and related protocols to adopt these concepts directly or whether we will need to introduce new protocols to complement the existing ones. + +The more basic protocols such as `Equatable`, `Comparable`, and `Hashable` should be easier to update. + +#### Refining `with*` closure-taking APIs + +The `~Escapable` types can be used to refine common `with*` closure-taking APIs by ensuring that the closures cannot save or hold their arguments beyond their own lifetime. +For example, this can greatly improve the safety of locking APIs that expect to unlock resources upon completion of the closure. + +#### Nonescapable classes + +We’ve explicitly excluded class types from being nonescapable. +In the future, we could allow class types to be declared nonescapable as a way to avoid most reference-counting operations on class objects. + +#### Concurrency + +Structured concurrency implies lifetime constraints similar to those outlined in this proposal. +It may be appropriate to incorporate `~Escapable` into the structured concurrency primitives. + +For example, the current `TaskGroup` type is supposed to never be escaped from the local context; +making it `~Escapable` would prevent this type of abuse and possibly enable other optimizations. + +#### Global nonescapable types with immortal lifetimes + +This proposal currently prohibits putting values with nonescapable types into global or static variables. +We expect to eventually allow this by explicitly annotating a “static” or “immortal” lifetime. + +## Alternatives considered + +#### Require `Escapable` to indicate escapable types without using `~Escapable` + +We could avoid using `~Escapable` to mark types that lack the `Escapable` property by requiring `Escapable` on all escapable types. +However, it is infeasible to require updating all existing types in all existing Swift code with a new explicit capability. + +Apart from that, we expect almost all types to continue to be escapable in the future, so the negative marker reduces the overall burden. +It is also consistent with progressive disclosure: +Most new Swift programmers should not need to know details of how escapable types work, since that is the common behavior of most data types in most programming languages. +When developers use existing nonescapable types, specific compiler error messages should guide them to correct usage without needing to have a detailed understanding of the underlying concepts. +With our current proposal, the only developers who will need detailed understanding of these concepts are library authors who want to publish nonescapable types. + +#### `Nonescapable` as a marker protocol + +We considered introducing `Nonescapable` as a marker protocol indicating that the values of this type required additional compiler checks. +With that approach, you would define a conditionally-escapable type such as `Box` above in this fashion: + +```swift +// Box does not normally require additional escapability checks +struct Box { + var t: T +} + +// But if T requires additional checks, so does Box +extension Box: Nonescapable where T: Nonescapable { } +``` + +However, this would imply that any `Nonescapable` type was a +subtype of `Any` and could therefore be placed within an `Any` existential box. +An `Any` existential box is both `Copyable` and `Escapable`, +so it cannot be allowed to contain a nonescapable value. + +#### Rely on `~Copyable` + +As part of the `Span` design, we considered whether it would suffice to use `~Copyable` instead of introducing a new type concept. +Andrew Trick's analysis in [Language Support for Bufferview](https://forums.swift.org/t/roadmap-language-support-for-bufferview/66211) concluded that making `Span` be non-copyable would not suffice to provide the full semantics we want for that type. +Further, introducing `Span` as `~Copyable` would actually preclude us from later expanding it to be `~Escapable`. + +The iterator example in the beginning of this document provides another motivation: +Iterators are routinely copied in order to record a particular point in a collection. +Thus we concluded that non-copyable was not the correct lifetime restriction for types of this sort, and it was worthwhile to introduce a new lifetime concept to the language. + +## Acknowledgements + +Many people discussed this proposal and gave important feedback, including: Kavon Farvardin, Meghana Gupta, John McCall, Slava Pestov, Joe Groff, Guillaume Lessard, and Franz Busch. diff --git a/proposals/0447-span-access-shared-contiguous-storage.md b/proposals/0447-span-access-shared-contiguous-storage.md new file mode 100644 index 0000000000..71b3ff8aa4 --- /dev/null +++ b/proposals/0447-span-access-shared-contiguous-storage.md @@ -0,0 +1,799 @@ +# Span: Safe Access to Contiguous Storage + +* Proposal: [SE-0447](0447-span-access-shared-contiguous-storage.md) +* Authors: [Guillaume Lessard](https://github.com/glessard), [Michael Ilseman](https://github.com/milseman), [Andrew Trick](https://github.com/atrick) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [BufferView Roadmap](https://forums.swift.org/t/66211) +* Bug: rdar://48132971, rdar://96837923 +* Implementation: [apple/swift#76406](https://github.com/swiftlang/swift/pull/76406) +* Review: ([Pitch 1](https://forums.swift.org/t/69888))([Pitch 2](https://forums.swift.org/t/72745))([Review](https://forums.swift.org/t/se-0447-span-safe-access-to-contiguous-storage/74676))([Acceptance](https://forums.swift.org/t/accepted-se-0447-span-safe-access-to-contiguous-storage/75508)) + +## Introduction + +We introduce `Span`, an abstraction for container-agnostic access to contiguous memory. It will expand the expressivity of performant Swift code without compromising on the memory safety properties we rely on: temporal safety, spatial safety, definite initialization and type safety. + +In the C family of programming languages, memory can be shared with any function by using a pointer and (ideally) a length. This allows contiguous memory to be shared with a function that doesn't know the layout of a container being used by the caller. A heap-allocated array, contiguously-stored named fields or even a single stack-allocated instance can all be accessed through a C pointer. We aim to enable a similar idiom in Swift, without compromising Swift's memory safety. + +This proposal builds on [Nonescapable types][PR-2304] (`~Escapable`,) and is a precursor to [Compile-time Lifetime Dependency Annotations][PR-2305], which will be proposed in the following weeks. The [BufferView roadmap](https://forums.swift.org/t/66211) forum thread was an antecedent to this proposal. This proposal also depends on the following proposals: + +- [SE-0426] BitwiseCopyable +- [SE-0427] Noncopyable generics +- [SE-0437] Non-copyable Standard Library Primitives +- [SE-0377] `borrowing` and `consuming` parameter ownership modifiers + +[SE-0426]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0426-bitwise-copyable.md +[SE-0427]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md +[SE-0437]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0437-noncopyable-stdlib-primitives.md +[SE-0377]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md +[PR-2304]: https://github.com/swiftlang/swift-evolution/pull/2304 +[PR-2305]: https://github.com/swiftlang/swift-evolution/pull/2305 +[PR-2305-pitch]: https://forums.swift.org/t/69865 + +## Motivation + +Swift needs safe and performant types for local processing over values in contiguous memory. Consider for example a program using multiple libraries, including one for [base64](https://datatracker.ietf.org/doc/html/rfc4648) decoding. The program would obtain encoded data from one or more of its dependencies, which could supply the data in the form of `[UInt8]`, `Foundation.Data` or even `String`, among others. None of these types is necessarily more correct than another, but the base64 decoding library must pick an input format. It could declare its input parameter type to be `some Sequence`, but such a generic function can significantly limit performance. This may force the library author to either declare its entry point as inlinable, or to implement an internal fast path using `withContiguousStorageIfAvailable()`, forcing them to use an unsafe type. The ideal interface would have a combination of the properties of both `some Sequence` and `UnsafeBufferPointer`. + +The `UnsafeBufferPointer` passed to a `withUnsafeXXX` closure-style API, while performant, is unsafe in multiple ways: + +1. The pointer itself is unsafe and unmanaged +2. `subscript` is only bounds-checked in debug builds of client code +3. It might escape the duration of the closure + +Even if the body of the `withUnsafeXXX` call does not escape the pointer, other functions called within the closure have to be written in terms of unsafe pointers. This requires programmer vigilance across a project and potentially spreads the use of unsafe types, even if the helper functions could have been written in terms of safe constructs. + +## Proposed solution + +#### `Span` + +`Span` will allow sharing the contiguous internal representation of a type, by providing access to a borrowed view of an interval of contiguous memory. `Span` does not copy the underlying data: it instead relies on a guarantee that the original container cannot be modified or destroyed while the `Span` exists. In the prototype that accompanies this first proposal, `Span`s will be constrained to closures from which they structurally cannot escape. Later, we will introduce a lifetime dependency between a `Span` and the binding of the type vending it, preventing its escape from the scope where it is valid for use. Both of these approaches guarantee temporal safety. `Span` also performs bounds-checking on every access to preserve spatial safety. Additionally `Span` always represents initialized memory, preserving the definite initialization guarantee. + +`Span` is intended as the currency type for local processing over values in contiguous memory. It is a replacement for many API currently using `Array`, `UnsafeBufferPointer`, `Foundation.Data`, etc., that do not need to escape the owning container. + +A `Span` provided by a container represents a borrow of that container. `Span` can therefore provide simultaneous access to a non-copyable container. It can also help avoid unwanted copies of copyable containers. Note that `Span` is not a replacement for a copyable container with owned storage; see [future directions](#Directions) for more details ([Resizable, contiguously-stored, untyped collection in the standard library](#Bytes).) + +In this initial proposal, no initializers are proposed for `Span`. Initializers for non-escapable types such as `Span` require a concept of lifetime dependency, which does not exist at this time. The lifetime dependency annotation will indicate to the compiler how a newly-created `Span` can be used safely. See also ["Initializers"](#Initializers) in [future directions](#Directions). + +#### `RawSpan` + +`RawSpan` allows sharing contiguous memory representing values which may be heterogeneously-typed, such as memory intended for parsing. It makes the same safety guarantees as `Span`. Since it is a fully concrete type, it can achieve great performance in debug builds of client code as well as straightforward performance in library code. + +A `RawSpan` can be obtained from containers of `BitwiseCopyable` elements, as well as be initialized directly from an instance of `Span`. + +## Detailed design + +`Span` is a simple representation of a region of initialized memory. + +```swift +@frozen +public struct Span: Copyable, ~Escapable { + internal var _start: UnsafeRawPointer? + internal var _count: Int +} + +extension Span: Sendable where Element: Sendable & ~Copyable {} +``` + +We store a `UnsafeRawPointer` value internally in order to explicitly support reinterpreted views of memory as containing different types of `BitwiseCopyable` elements. Note that the the optionality of the pointer does not affect usage of `Span`, since accesses are bounds-checked and the pointer is only dereferenced when the `Span` isn't empty, and the pointer cannot be `nil`. + +It provides a buffer-like interface to the elements stored in that span of memory: + +```swift +extension Span where Element: ~Copyable { + public var count: Int { get } + public var isEmpty: Bool { get } + + public typealias Index = Int + public var indices: Range { get } + + public subscript(_ index: Index) -> Element { _read } +} +``` + +Note that `Span` does _not_ conform to `Collection`. This is because `Collection`, as originally conceived and enshrined in existing source code, assumes pervasive copyability and escapability of the `Collection` itself as well as of element type. In particular a subsequence of a `Collection` is semantically a separate value from the instance it was derived from. In the case of `Span`, a sub-span representing a subrange of its elements _must_ have the same lifetime as the `Span` from which it originates. Another proposal will consider collection-like protocols to accommodate different combinations of `~Copyable` and `~Escapable` for the collection and its elements. + +Like `UnsafeBufferPointer`, `Span` uses a simple offset-based indexing. The first element of a given span is always at position zero, and its last element is always at position `count-1`. + +As a side-effect of not conforming to `Collection` or `Sequence`, `Span` is not directly supported by `for` loops at this time. It is, however, easy to use in a `for` loop via indexing: + +```swift +for i in mySpan.indices { + calculation(mySpan[i]) +} +``` + +### `Span` API: + +Initializers, required for library adoption, will be proposed alongside [lifetime annotations][PR-2305]; for details, see "[Initializers](#Initializers)" in the [future directions](#Directions) section. + +##### Basic API: + +The following properties, functions and subscripts have direct counterparts in the `Collection` protocol hierarchy. Their semantics shall be as described where they counterpart is declared (in `Collection` or `RandomAccessCollection`). + +```swift +extension Span where Element: ~Copyable { + /// The number of initialized elements in the span. + public var count: Int { get } + + /// A Boolean value indicating whether the span is empty. + public var isEmpty: Bool { get } + + /// The type that represents a position in `Span`. + public typealias Index = Int + + /// The range of indices valid for this `Span` + public var indices: Range { get } + + /// Accesses the element at the specified position. + public subscript(_ position: Index) -> Element { _read } +} +``` + +Note that we use a `_read` accessor for the subscript, a requirement in order to `yield` a borrowed non-copyable `Element` (see ["Coroutines"](#Coroutines).) This yields an element whose lifetime is scoped around this particular access, as opposed to matching the lifetime dependency of the `Span` itself. This is a language limitation we expect to resolve with a followup proposal introducing a new accessor model. The subscript will then be updated to use the new accessor semantics. We expect the updated accessor to be source-compatible, as it will provide a borrowed element with a wider lifetime than a `_read` accessor can provide. + +##### Unchecked access to elements: + +The `subscript` mentioned above has always-on bounds checking of its parameter, in order to prevent out-of-bounds accesses. We also want to provide unchecked variants as an alternative for cases where bounds-checking is proving costly, such as in tight loops: + +```swift +extension Span where Element: ~Copyable { + + /// Accesses the element at the specified `position`. + /// + /// This subscript does not validate `position`; this is an unsafe operation. + /// + /// - Parameter position: The offset of the element to access. `position` + /// must be greater or equal to zero, and less than `count`. + public subscript(unchecked position: Index) -> Element { _read } +} +``` + +When using the unchecked subscript, the index must be known to be valid. While we are not proposing explicit index validation API on `Span` itself, its `indices` property can be use to validate a single index, in the form of the function `Range.contains(_: Int) -> Bool`. We expect that `Range` will also add efficient containment checking of a subrange's endpoints, which should be generally useful for index range validation in this and other contexts. + +##### Identifying whether a `Span` is a subrange of another: + +When working with multiple `Span` instances, it is often desirable to know whether one is identical to or a subrange of another. We include functions to determine whether this is the case, as well as a function to obtain the valid offsets of the subrange within the larger span: + +```swift +extension Span where Element: ~Copyable { + /// Returns true if the other span represents exactly the same memory + public func isIdentical(to span: borrowing Self) -> Bool + + /// Returns the indices within `self` where the memory represented by `span` + /// is located, or `nil` if `span` is not located within `self`. + /// + /// Parameters: + /// - span: a span that may be a subrange of `self` + /// Returns: A range of offsets within `self`, or `nil` + public func indices(of span: borrowing Self) -> Range? +} +``` + +##### Interoperability with unsafe code: + +We provide two functions for interoperability with C or other legacy pointer-taking functions. + +```swift +extension Span where Element: ~Copyable { + /// Calls a closure with a pointer to the viewed contiguous storage. + /// + /// The buffer pointer passed as an argument to `body` is valid only + /// during the execution of `withUnsafeBufferPointer(_:)`. + /// Do not store or return the pointer for later use. + /// + /// - Parameter body: A closure with an `UnsafeBufferPointer` parameter + /// that points to the viewed contiguous storage. If `body` has + /// a return value, that value is also used as the return value + /// for the `withUnsafeBufferPointer(_:)` method. The closure's + /// parameter is valid only for the duration of its execution. + /// - Returns: The return value of the `body` closure parameter. + func withUnsafeBufferPointer( + _ body: (_ buffer: UnsafeBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} + +extension Span where Element: BitwiseCopyable { + /// Calls the given closure with a pointer to the underlying bytes of + /// the viewed contiguous storage. + /// + /// The buffer pointer passed as an argument to `body` is valid only + /// during the execution of `withUnsafeBytes(_:)`. + /// Do not store or return the pointer for later use. + /// + /// - Parameter body: A closure with an `UnsafeRawBufferPointer` + /// parameter that points to the viewed contiguous storage. + /// If `body` has a return value, that value is also + /// used as the return value for the `withUnsafeBytes(_:)` method. + /// The closure's parameter is valid only for the duration of + /// its execution. + /// - Returns: The return value of the `body` closure parameter. + func withUnsafeBytes( + _ body: (_ buffer: UnsafeRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} +``` + +These functions use a closure to define the scope of validity of `buffer`, ensuring that the underlying `Span` and the binding it depends on both remain valid through the end of the closure. They have the same shape as the equivalents on `Array` because they fulfill the same function, namely to keep the underlying binding alive. + +### RawSpan + +In addition to `Span`, we propose the addition of `RawSpan`, to represent heterogeneously-typed values in contiguous memory. `RawSpan` is similar to `Span`, but represents _untyped_ initialized bytes. `RawSpan` is a specialized type that is intended to support parsing and decoding applications, as well as applications where heavily-used code paths require concrete types as much as possible. Its API supports the data loading operations `unsafeLoad(as:)` and `unsafeLoadUnaligned(as:)`. + +#### `RawSpan` API: + +```swift +@frozen +public struct RawSpan: Copyable, ~Escapable { + internal var _start: UnsafeRawPointer + internal var _count: Int +} + +extension RawSpan: Sendable {} +``` + +Initializers, required for library adoption, will be proposed alongside [lifetime annotations][PR-2305]; for details, see "[Initializers](#Initializers)" in the [future directions](#Directions) section. + +##### Accessing the memory of a `RawSpan`: + +`RawSpan` has basic operations to access the contents of its memory: `unsafeLoad(as:)` and `unsafeLoadUnaligned(as:)`: + +```swift +extension RawSpan { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// The memory at this pointer plus `offset` must be properly aligned for + /// accessing `T` and initialized to `T` or another type that is layout + /// compatible with `T`. + /// + /// This is an unsafe operation. Failure to meet the preconditions + /// above may produce an invalid value of `T`. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance is memory-managed and unassociated + /// with the value in the memory referenced by this pointer. + public func unsafeLoad( + fromByteOffset offset: Int = 0, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// The memory at this pointer plus `offset` must be initialized to `T` + /// or another type that is layout compatible with `T`. + /// + /// This is an unsafe operation. Failure to meet the preconditions + /// above may produce an invalid value of `T`. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance isn't associated + /// with the value in the range of memory referenced by this pointer. + public func unsafeLoadUnaligned( + fromByteOffset offset: Int = 0, as: T.Type + ) -> T +``` + +These operations are not type-safe, in that the loaded value returned by the operation can be invalid, and violate type invariants. Some types have a property that makes the `unsafeLoad(as:)` function safe, but we don't have a way to [formally identify](#SurjectiveBitPattern) such types at this time. + +The `unsafeLoad` functions have counterparts which omit bounds-checking for cases where redundant checks affect performance: + +```swift + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// The memory at this pointer plus `offset` must be properly aligned for + /// accessing `T` and initialized to `T` or another type that is layout + /// compatible with `T`. + /// + /// This is an unsafe operation. This function does not validate the bounds + /// of the memory access, and failure to meet the preconditions + /// above may produce an invalid value of `T`. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance is memory-managed and unassociated + /// with the value in the memory referenced by this pointer. + public func unsafeLoad( + fromUncheckedByteOffset offset: Int, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// The memory at this pointer plus `offset` must be initialized to `T` + /// or another type that is layout compatible with `T`. + /// + /// This is an unsafe operation. This function does not validate the bounds + /// of the memory access, and failure to meet the preconditions + /// above may produce an invalid value of `T`. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance isn't associated + /// with the value in the range of memory referenced by this pointer. + public func unsafeLoadUnaligned( + fromUncheckedByteOffset offset: Int, as: T.Type + ) -> T +} +``` + +`RawSpan` provides `withUnsafeBytes` for interoperability with C or other legacy pointer-taking functions: + +```swift +extension RawSpan { + /// Calls the given closure with a pointer to the underlying bytes of + /// the viewed contiguous storage. + /// + /// The buffer pointer passed as an argument to `body` is valid only + /// during the execution of `withUnsafeBytes(_:)`. + /// Do not store or return the pointer for later use. + /// + /// - Parameter body: A closure with an `UnsafeRawBufferPointer` + /// parameter that points to the viewed contiguous storage. + /// If `body` has a return value, that value is also + /// used as the return value for the `withUnsafeBytes(_:)` method. + /// The closure's parameter is valid only for the duration of + /// its execution. + /// - Returns: The return value of the `body` closure parameter. + func withUnsafeBytes( + _ body: (_ buffer: UnsafeRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} +``` + +##### Examining `RawSpan` bounds: + +```swift +extension RawSpan { + /// The number of bytes in the span. + public var byteCount: Int { get } + + /// A Boolean value indicating whether the span is empty. + public var isEmpty: Bool { get } + + /// The range of valid byte offsets into this `RawSpan` + public var byteOffsets: Range { get } +} +``` + +##### Identifying whether a `RawSpan` is a subrange of another: + +When working with multiple `RawSpan` instances, it is often desirable to know whether one is identical to or a subrange of another. We include a function to determine whether this is the case, as well as a function to obtain the valid offsets of the subrange within the larger span. The documentation is omitted here, as it is substantially the same as for the equivalent functions on `Span`: + +```swift +extension RawSpan { + public func isIdentical(to span: borrowing Self) -> Bool + + public func byteOffsets(of span: borrowing Self) -> Range? +} +``` + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a new version of the standard library and runtime. + +## Alternatives considered + +##### Make `Span` a noncopyable type +Making `Span` non-copyable was in the early vision of this type. However, we found that would make `Span` a poor match to model borrowing semantics. This realization led to the initial design for non-escapable declarations. + +##### Use a non-escapable index type +A non-escapable index type implies that any indexing operation would borrow its `Span`. This would prevent using such an index for a mutation, since a mutation requires an _exclusive_ borrow. Noting that the usage pattern we desire for `Span` must also apply to `MutableSpan`(described [below](#MutableSpan),) a non-escapable index would make it impossible to also implement a mutating subscript, unless any mutating operation consumes the index. This seems untenable. + +##### Naming + +The ideas in this proposal previously used the name `BufferView`. While the use of the word "buffer" would be consistent with the `UnsafeBufferPointer` type, it is nevertheless not a great name, since "buffer" is commonly used in reference to transient storage. Another previous pitch used the term `StorageView` in reference to the `withContiguousStorageIfAvailable()` standard library function. We also considered the name `StorageSpan`, but that did not add much beyond the shorter name `Span`. `Span` clearly identifies itself as a relative of C++'s `std::span`. + +The OpenTelemetry project and its related libraries use the word "span" for a concept of a timespan. The domains of use between that and direct memory access are very distinct, and we believe that the confusability between the use cases should be low. We also note that standard library type names can always be shadowed by type names from packages, mitigating the risk of source breaks. + +##### Sendability of `RawSpan` + +This proposal makes `RawSpan` a `Sendable` type. We believe this is the right decision. The sendability of `RawSpan` could be used to unsafely transfer a pointer value across an isolation boundary, despite the non-sendability of pointers. For example, suppose a `RawSpan` were obtained from an existing `Array` variable. We could send the `RawSpan` across the isolation boundary, and there extract the pointer using `rawSpan.unsafeLoad(as: UnsafeRawPointer.self)`. While this is an unsafe outcome, a similar operation can be done encoding a pointer as an `Int`, and then using `UnsafeRawPointer(bitPattern: mySentInt)` on the other side of the isolation boundary. + +##### A more sophisticated approach to indexing + +This is discussed more fully in the [indexing appendix](#Indexing) below. + +## Future directions + +#### Initializing and returning `Span` instances + +A `Span` represents a region of memory and, as such, must be initialized using an unsafe pointer. This is an unsafe operation which will typically be performed internally to a container's implementation. In order to bridge to safe code, these initializers require new annotations that indicate to the compiler how the newly-created `Span` can be used safely. + +These annotations have been [pitched][PR-2305-pitch] and are expected to be formally [proposed][PR-2305] soon. `Span` initializers using lifetime annotations will be proposed alongside the annotations themselves. + +#### Obtaining variant `Span`s and `RawSpan`s from `Span` and `RawSpan` + +`Span`s representing subsets of consecutive elements could be extracted out of a larger `Span` with an API similar to the `extracting()` functions recently added to `UnsafeBufferPointer` in support of non-copyable elements: + +```swift +extension Span where Element: ~Copyable { + public func extracting(_ bounds: Range) -> Self +} +``` + +Each variant of such a function needs to return a `Span`, which requires a lifetime dependency. + +Similarly, a `RawSpan` should be initializable from a `Span`, and `RawSpan` should provide a function to unsafely view its content as a typed `Span`: + +```swift +extension RawSpan { + public init(_ span: Span) + + public func unsafeView(as type: T.Type) -> Span +} +``` + +We are subsetting these functions of `Span` and `RawSpan` until the lifetime annotations are proposed. + +#### Coroutine or Projection Accessors + +This proposal includes some `_read` accessors, the coroutine version of the `get` accessor. `_read` accessors are not an official part of the Swift language, but are necessary for some types to be able to provide borrowing access to their internal storage, in particular storage containing non-copyable elements. The correct solution may involve a projection of a different type than is provided by a coroutine. When correct, stable replacement for `_read` accessors is proposed and accepted, the implementation of `Span` and `RawSpan` will be adapted to the new syntax. + +#### Extensions to Standard Library and Foundation types + +The standard library and Foundation has a number of types that can in principle provide access to their internal storage as a `Span`. We could provide `withSpan()` and `withBytes()` closure-taking functions as safe replacements for the existing `withUnsafeBufferPointer()` and `withUnsafeBytes()` functions. We could also provide lifetime-dependent `span` or `bytes` properties. For example, `Array` could be extended as follows: + +```swift +extension Array { + public func withSpan( + _ body: (_ elements: Span) throws(E) -> Result + ) throws(E) -> Result + + public var span: Span { borrowing get } +} + +extension Array where Element: BitwiseCopyable { + public func withBytes( + _ body: (_ bytes: RawSpan) throws(E) -> Result + ) throws(E) -> Result where Element: BitwiseCopyable + + public var bytes: RawSpan { borrowing get } +} +``` + +Of these, the closure-taking functions can be implemented now, but it is unclear whether they are desirable. The lifetime-dependent computed properties require lifetime annotations, as initializers do. We are deferring proposing these extensions until the lifetime annotations are proposed. + +#### A `ContiguousStorage` protocol + +An earlier version of this proposal proposed a `ContiguousStorage` protocol by which a type could indicate that it can provide a `Span`. `ContiguousStorage` would form a bridge between generically-typed interfaces and a performant concrete implementation. It would supersede the rejected [SE-0256](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0256-contiguous-collection.md). + +For example, for the hypothetical base64 decoding library mentioned in the [motivation](#Motivation) section, a possible API could be: + +```swift +extension HypotheticalBase64Decoder { + public func decode(bytes: some ContiguousStorage) -> [UInt8] +} +``` + +`ContiguousStorage` would have the following definition: + +```swift +public protocol ContiguousStorage: ~Copyable, ~Escapable { + associatedtype Element: ~Copyable & ~Escapable + var storage: Span { _read } +} +``` + +Two issues prevent us from proposing it at this time: (a) the ability to suppress requirements on `associatedtype` declarations was deferred during the review of [SE-0427], and (b) we cannot declare a `_read` accessor as a protocol requirement. + +Many of the standard library collections could conform to `ContiguousStorage`. + +#### Index Validation Utilities + +This proposal originally included index validation utilities for `Span`. such as `boundsContain(_: Index) -> Bool` and `boundsContain(_: Range) -> Bool`. After review feedback, we believe that the utilities proposed would also be useful for index validation on `UnsafeBufferPointer`, `Array`, and other similar `RandomAccessCollection` types. `Range` already a single-element `contains(_: Bound) -> Bool` function which can be made even more efficient. We should add an additional function that identifies whether a `Range` contains the _endpoints_ of another `Range`. Note that this is not the same as the existing `contains(_: some Collection) -> Bool`, which is about the _elements_ of the collection. This semantic difference can lead to different results when examining empty `Range` instances. + +#### Support for `Span` in `for` loops + +This proposal does not define an `IteratorProtocol` conformance, since an iterator for `Span` would need to be non-escapable. This is not compatible with `IteratorProtocol`. As such, `Span` is not directly usable in `for` loops as currently defined. A `BorrowingIterator` protocol for non-escapable and non-copyable containers must be defined, providing a `for` loop syntax where the element is borrowed through each iteration. Ultimately we should arrive at a way to iterate through borrowed elements from a borrowed view: + +```swift +func doSomething(_ e: borrowing Element) { ... } +let span: Span = ... +for borrowing element in span { + doSomething(element) +} +``` + +In the meantime, it is possible to loop through a `Span`'s elements by direct indexing: + +```swift +let span: Span = ... +// either: +var i = 0 +while i < span.count { + doSomething(span[i]) + i += 1 +} + +// ...or: +for i in 0..Layout constraint for safe loading of bit patterns + +`RawSpan` has unsafe functions that interpret the raw bit patterns it contains as values of arbitrary `BitwiseCopyable` types. In order to have safe alternatives to these, we could add a layout constraint refining `BitwiseCopyable`, specifically for types whose mapping from bit pattern to values is a [surjective function](https://en.wikipedia.org/wiki/Surjective_function) (e.g. `SurjectiveBitPattern`). Such types would be safe to [load](#Load) from `RawSpan` instances. 1-byte examples are `Int8` (any of 256 values are valid) and `Bool` (256 bit patterns map to `true` or `false` because only one bit is considered.) + +An alternative to a layout constraint is to add a type validation step to ensure that if a given bit pattern were to be interpreted as a value of type `T`, then all the invariants of type `T` would be respected. This alternative would be more flexible, but may have a higher runtime cost. + +#### Byte parsing helpers + +We could add some API to `RawSpan` to make it better suited for binary parsers and decoders. + +```swift +extension RawSpan { + public struct Cursor: Copyable, ~Escapable { + public let base: RawSpan + + /// The current parsing position + public var position: Int + + /// Parse an instance of `T` and advance. + /// Returns `nil` if there are not enough bytes remaining for an instance of `T`. + public mutating func parse( + _ t: T.Type = T.self + ) -> T? + + /// Parse `numBytes`and advance. + /// Returns `nil` if there are fewer than `numBytes` remaining. + public mutating func parse( + numBytes: some FixedWidthInteger + ) -> RawSpan? + + /// The bytes that we've parsed so far + public var parsedBytes: RawSpan { get } + } +} +``` + +`Cursor` stores and manages a parsing subrange, which alleviates the developer from managing one layer of slicing. + +Alternatively, if some future `RawSpan.Iterator` were 3 words in size (start, current position, and end) instead of 2 (current pointer and end), making it a "resettable", it could host this API instead of introducing a new `Cursor` type or concept. + +##### Example: Parsing PNG + +The code snippet below parses a [PNG Chunk](https://www.w3.org/TR/png-3/#4Concepts.FormatChunks), using the byte parsing helpers defined above: + +```swift +// Parse a PNG chunk +let length = try cursor.parse(UInt32.self).bigEndian +let type = try cursor.parse(UInt32.self).bigEndian +let data = try cursor.parse(numBytes: length) +let crc = try cursor.parse(UInt32.self).bigEndian +``` + +#### Safe mutations of memory with `MutableSpan` + +Some data structures can delegate mutations of their owned memory. In the standard library the function `withMutableBufferPointer()` provides this functionality in an unsafe manner. + +The `UnsafeMutableBufferPointer` passed to a `withUnsafeMutableXXX` closure-style API is unsafe in multiple ways: + +1. The pointer itself is unsafe and unmanaged +2. `subscript` is only bounds-checked in debug builds of client code +3. It might escape the duration of the closure +4. Exclusivity of writes is not enforced +5. Initialization of any particular memory address is not ensured + +in other words, it is unsafe in all the same ways as `UnsafeBufferPointer`-passing closure APIs, in addition to enforcing neither exclusivity nor initialization state. + +Loading an uninitialized non-`BitwiseCopyable` value leads to undefined behavior. Loading an uninitialized `BitwiseCopyable` value does not immediately lead to undefined behavior, but it produces a garbage value which may lead to misbehavior of the program. + +A `MutableSpan` should provide a better, safer alternative to mutable memory in the same way that `Span` provides a better, safer read-only type. `MutableSpan` would apply to initialized memory and would enforce exclusivity of writes, thereby preserving the initialization state of its memory between mutations. + +#### Delegating initialization of memory with `OutputSpan` + +Some data structures can delegate initialization of their initial memory representation, and in some cases the initialization of additional memory. For example, the standard library features the initializer`Array.init(unsafeUninitializedCapacity:initializingWith:)`, which depends on `UnsafeMutableBufferPointer` and is known to be error-prone. A safer abstraction for initialization would make such initializers less dangerous, and would allow for a greater variety of them. + +We can define an `OutputSpan` type, which could support appending to the initialized portion of a data structure's underlying storage. `OutputSpan` allows for uninitialized memory beyond the last position appended. Such an `OutputSpan` would also be a useful abstraction to pass user-allocated storage to low-level API such as networking calls or file I/O. + +#### Resizable, contiguously-stored, untyped collection in the standard library + +The example in the [motivation](#Motivation) section mentions the `Foundation.Data` type. There has been some discussion of either replacing `Data` or moving it to the standard library. This document proposes neither of those. A major issue is that in the "traditional" form of `Foundation.Data`, namely `NSData` from Objective-C, it was easier to control accidental copies because the semantics of the language did not lead to implicit copying. + +Even if `Span` were to replace all uses of a constant `Data` in API, something like `Data` would still be needed, for the same reason as `Array` is needed: such a type allows for resizing mutations (e.g. `RangeReplaceableCollection` conformance.) We may want to add an untyped-element equivalent of `Array` to the standard library at a later time. + +#### Syntactic Sugar for Automatic Conversions + +Even with a `ContiguousStorage` protocol, a generic entry point in terms of `some ContiguousStorage` may add unwanted overhead to resilient libraries. As detailed above, an entry point in an evolution-enabled library requires an inlinable generic public entry point which forwards to a publicly-accessible function defined in terms of `Span`. If `Span` does become a widely-used type to interface between libraries, we could simplify these conversions with a bit of compiler help. + +We could provide an automatic way to use a `ContiguousStorage`-conforming type with a function that takes a `Span` of the appropriate element type: + +```swift +func myStrnlen(_ b: Span) -> Int { + guard let i = b.firstIndex(of: 0) else { return b.count } + return b.distance(from: b.startIndex, to: e) +} +let data = Data((0..<9).reversed()) // Data conforms to ContiguousStorage +let array = Array(data) // Array also conforms to ContiguousStorage +myStrnlen(data) // 8 +myStrnlen(array) // 8 +``` + +This would probably consist of a new type of custom conversion in the language. A type author would provide a way to convert from their type to an owned `Span`, and the compiler would insert that conversion where needed. This would enhance readability and reduce boilerplate. + +#### Interoperability with C++'s `std::span` and with llvm's `-fbounds-safety` + +The [`std::span`](https://en.cppreference.com/w/cpp/container/span) class template from the C++ standard library is a similar representation of a contiguous range of memory. LLVM may soon have a [bounds-checking mode](https://discourse.llvm.org/t/70854) for C. These are opportunities for better, safer interoperation with Swift, via a type such as `Span`. + +## Acknowledgments + +Joe Groff, John McCall, Tim Kientzle, Steve Canon and Karoy Lorentey contributed to this proposal with their clarifying questions and discussions. + +### Appendix: Index and slicing design considerations + +Early prototypes of this proposal defined an `Index` type, `Iterator` types, etc. We are proposing `Int`-based API and are deferring defining `Index` and `Iterator` until more of the non-escapable collection story is sorted out. The below is some of our research into different potential designs of an `Index` type. + +There are 3 potentially-desirable features of `Span`'s `Index` design: + +1. `Span` is its own slice type +2. Indices from a slice can be used on the base collection +3. Additional reuse-after-free checking + +Each of these introduces practical tradeoffs in the design. + +#### `Span` is its own slice type + +Collections which own their storage have the convention of separate slice types, such as `Array` and `String`. This has the advantage of clearly delineating storage ownership in the programming model and the disadvantage of introducing a second type through which to interact. + +When types do not own their storage, separate slice types can be [cumbersome](https://github.com/swiftlang/swift/blob/swift-5.10.1-RELEASE/stdlib/public/core/StringComparison.swift#L175). The reason `UnsafeBufferPointer` has a separate slice type is because it wants to allow indices to be reused across slices and its `Index` is a relative offset from the start (`Int`) rather than an absolute position (such as a pointer). + +`Span` does not own its storage and there is no concern about leaking larger allocations. It would benefit from being its own slice type. + +#### Indices from a slice can be used on the base collection + +There is very strong stdlib precedent that indices from the base collection can be used in a slice and vice-versa. + +```swift +let myCollection = [0,1,2,3,4,5,6] +let idx = myCollection.index(myCollection.startIndex, offsetBy: 4) +myCollection[idx] // 4 +let slice = myCollection[idx...] // [4, 5, 6] +slice[idx] // 4 +myCollection[slice.indices] // [4, 5, 6] +``` + +Code can be written to take advantage of this fact. For example, a simplistic parser can be written as mutating methods on a slice. The slice's indices can be saved for reference into the original collection or another slice. + +```swift +extension Slice where Base == UnsafeRawBufferPointer { + mutating func parse(numBytes: Int) -> Self { + let end = index(startIndex, offsetBy: numBytes) + defer { self = self[end...] } + return self[.. Int { + parse(numBytes: MemoryLayout.stride).loadUnaligned(as: Int.self) + } + + mutating func parseHeader() -> Self { + // Comments show what happens when ran with `myCollection` + + let copy = self + parseInt() // 0 + parseInt() // 1 + parse(numBytes: 8) // [2, 0, 0, 0, 0, 0, 0, 0] + parseInt() // 3 + parse(numBytes: 7) // [4, 0, 0, 0, 0, 0, 0] + + // self: [0, 5, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0] + parseInt() // 1280 (0x00_00_05_00 little endian) + // self: [0, 6, 0, 0, 0, 0, 0, 0, 0] + + return copy[..( + _ c: C +) -> C.Element where C.Index == Int { + c[0] +} + +getFirst(myCollection) // 0 +getFirst(slice) // Fatal error: Index out of bounds +``` + +#### Additional reuse-after-free checking + +`Span` bounds-checks its indices, which is important for safety. If the index is based around a pointer (instead of an offset), then bounds checks will also ensure that indices are not used with the wrong span in most situations. However, it is possible for a memory address to be reused after being freed and using a stale index into this reused memory may introduce safety problems. + +```swift +var idx: Span.Index + +let array1: Array = ... +let span1 = array1.span +idx = span1.startIndex.advanced(by: ...) +... +// array1 is freed + +let array2: Array = ... +let span2 = array2.span +// array2 happens to be allocated within the same memory of array1 +// but with a different base address whose offset is not an even +// multiple of `MemoryLayout.stride`. + +span2[idx] // misaligned load, what happens? +``` + +If `T` is `BitwiseCopyable`, then the misaligned load is not undefined behavior, but the value that is loaded is garbage. Whether the program is well-behaved going forwards depends on whether it is resilient to getting garbage values. + +If `T` is not `BitwiseCopyable`, then the misaligned load may introduce undefined behavior. No matter how well-written the rest of the program is, it has a critical safety and security flaw. + +When the reused allocation happens to be stride-aligned, there is no undefined behavior from undefined loads, nor are there "garbage" values in the strictest sense, but it is still reflective of a programming bug. The program may be interacting with an unexpected value. + +Bounds checks protect against critical programmer errors. It would be nice, pending engineering tradeoffs, to also protect against some reuse after free errors and invalid index reuse, especially those that may lead to undefined behavior. + +Future improvements to microarchitecture may make reuse after free checks cheaper, however we need something for the foreseeable future. Any validation we can do reduces the need to switch to other mitigation strategies or make other tradeoffs. + +#### Design approaches for indices + +##### Index is an offset (`Int` or a wrapper around `Int`) + +When `Index` is an offset, there is no undefined behavior from misaligned loads because the `Span`'s base address is advanced by `MemoryLayout.stride * offset`. + +However, there is no protection against invalidly using an index derived from a different span, provided the offset is in-bounds. + +Since `Span` is 2 words (base address and count), indices cannot be interchanged between slices and the base span. In order to do so, `Span` would need to additionally store a base offset, bringing it up to 3 words in size. + +##### Index is a pointer (wrapper around `UnsafeRawPointer`) + +When Index holds a pointer, `Span` only needs to be 2 words in size, as valid index interchange across slices falls out naturally. Additionally, invalid reuse of an index across spans will typically be caught during bounds checking. + +However, in a reuse-after-free situation, misaligned loads (i.e. undefined behavior) are possible. If stride is not a multiple of 2, then alignment checking can be expensive. Alternatively, we could choose not to detect these bugs. + +##### Index is a fat pointer (pointer and allocation ID) + +We can create a per-allocation ID (e.g. a cryptographic `UInt64`) for both `Span` and `Span.Index` to store. This would make `Span` 3 words in size and `Span.Index` 2 words in size. This provides the most protection possible against all forms of invalid index use, including reuse-after-free. However, making `Span` be 3 words and `Span.Index` 2 words for this feature is unfortunate. + +We could instead go with 2 word `Span` and 2 word `Span.Index` by storing the span's `baseAddress` in the `Index`'s second word. This will detect invalid reuse of indices across spans in addition to misaligned reuse-after-free errors. However, indices could not be interchanged without a way for the slice type to know the original span's base address (e.g. through a separate slice type or making `Span` 3 words in size). + +In either approach, making `Span.Index` be 2 words in size is unfortunate. `Range` is now 4 words in size, storing the allocation ID twice. Anything built on top of `Span` that wishes to store multiple indices is either bloated or must hand-extract the pointers and hand-manage the allocation ID. diff --git a/proposals/0448-regex-lookbehind-assertions.md b/proposals/0448-regex-lookbehind-assertions.md new file mode 100644 index 0000000000..744bdbd85c --- /dev/null +++ b/proposals/0448-regex-lookbehind-assertions.md @@ -0,0 +1,135 @@ +# Regex lookbehind assertions + +* Proposal: [SE-0448](0448-regex-lookbehind-assertions.md) +* Authors: [Jacob Hearst](https://github.com/JacobHearst) [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Accepted** +* Implementation: https://github.com/swiftlang/swift-experimental-string-processing/pull/760 +* Review: + ([pitch](https://forums.swift.org/t/pitch-regex-reverse-matching/73482)) + ([review](https://forums.swift.org/t/se-0448-regex-lookbehind-assertions/74672)) + ([acceptance](https://forums.swift.org/t/accepted-se-0448-regex-lookbehind-assertions/75111)) + + +## Introduction + +Regex supports lookahead assertions, but does not currently support lookbehind assertions. We propose adding these. + +## Motivation + +Modern regular expression engines support lookbehind assertions, whether fixed length (Perl, PCRE2, Python, Java) or arbitrary length (.NET, Javascript). + +## Proposed solution + +We propose supporting arbitrary-length lookbehind regexes which can be achieved by performing matching in reverse. + +Like lookahead assertions, lookbehind assertions are _zero-width_, meaning they do not affect the current match position. + +Examples: + + +```swift +"abc".firstMatch(of: /a(?<=a)bc/) // matches "abc" +"abc".firstMatch(of: /a(?<=b)c/) // no match +"abc".firstMatch(of: /a(?<=.)./) // matches "ab" +"abc".firstMatch(of: /ab(?<=a)c/) // no match +"abc".firstMatch(of: /ab(?<=.a)c/) // no match +"abc".firstMatch(of: /ab(?<=a.)c/) // matches "abc" +``` + +Lookbehind assertions run in reverse, i.e. right-to-left, meaning that right-most eager quantifications have the opportunity to consume more of the input than left-most. This does not affect whether an input matches, but could affect the value of captures inside of a lookbehind assertion: + +```swift +"abcdefg".wholeMatch(of: /(.+)(.+)/) +// Produces ("abcdefg", "abcdef", "g") + +"abcdefg".wholeMatch(of: /.*(?<=(.+)(.+)/)) +// Produces ("abcdefg", "a", "bcdefg") +``` + +## Detailed design + + +### Syntax + +Lookbehind assertion syntax is already supported in the existing [Regex syntax](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#lookahead-and-lookbehind). + +The engine is currently incapable of running them, so a compilation error is thrown: + +```swift +let regex = /(?<=a)b/ +// error: Cannot parse regular expression: lookbehind is not currently supported +``` + +With this proposal, this restriction is lifted and the following syntactic forms will be accepted: + +```swift +// Positive lookbehind +/a(?<=b)c/ +/a(*plb:b)c/ +/a(*positive_lookbehind:b)c/ + +// Negative lookbehind +/a(?) -> Self + + /// The trait's canonical name. + /// + /// This is used when enabling the trait or when referring to it from other modifiers in the manifest. + /// + /// The following rules are enforced on trait names: + /// - The first character must be a [Unicode XID start character](https://unicode.org/reports/tr31/#Figure_Code_Point_Categories_for_Identifier_Parsing) + /// (most letters), a digit, or `_`. + /// - Subsequent characters must be a [Unicode XID continue character](https://unicode.org/reports/tr31/#Figure_Code_Point_Categories_for_Identifier_Parsing) + /// (a digit, `_`, or most letters), `-`, or `+`. + /// - `default` and `defaults` (in any letter casing combination) are not allowed as trait names to avoid confusion with default traits. + public var name: String + + /// The trait's description. + /// + /// Use this to explain what functionality this trait enables. + public var description: String? + + /// A set of other traits of this package that this trait enables. + public var enabledTraits: Set + + /// Initializes a new trait. + /// + /// - Parameters: + /// - name: The trait's canonical name. + /// - description: The trait's description. + /// - enabledTraits: A set of other traits of this package that this trait enables. + public init( + name: String, + description: String? = nil, + enabledTraits: Set = [] + ) + + /// Initializes a new trait. + /// + /// This trait is disabled by default and enables no other trait of this package. + public init(stringLiteral value: StringLiteralType) + + /// Initializes a new trait. + /// + /// - Parameters: + /// - name: The trait's canonical name. + /// - description: The trait's description. + /// - enabledTraits: A set of other traits of this package that this trait enables. + public static func trait( + name: String, + description: String? = nil, + enabledTraits: Set = [] + ) -> Trait +} +``` + +The `Package` class is extended to define a set of traits: + +```swift +public final class Package { + // ... + + /// The set of traits of this package. + public var traits: Set + + /// Initializes a Swift package with configuration options you provide. + /// + /// - Parameters: + /// - name: The name of the Swift package, or `nil` to use the package's Git URL to deduce the name. + /// - defaultLocalization: The default localization for resources. + /// - platforms: The list of supported platforms with a custom deployment target. + /// - pkgConfig: The name to use for C modules. If present, Swift Package Manager searches for a + /// `.pc` file to get the additional flags required for a system target. + /// - providers: The package providers for a system target. + /// - products: The list of products that this package makes available for clients to use. + /// - traits: The set of traits of this package. + /// - dependencies: The list of package dependencies. + /// - targets: The list of targets that are part of this package. + /// - swiftLanguageVersions: The list of Swift versions with which this package is compatible. + /// - cLanguageStandard: The C language standard to use for all C targets in this package. + /// - cxxLanguageStandard: The C++ language standard to use for all C++ targets in this package. + public init( + name: String, + defaultLocalization: LanguageTag? = nil, + platforms: [SupportedPlatform]? = nil, + pkgConfig: String? = nil, + providers: [SystemPackageProvider]? = nil, + products: [Product] = [], + traits: Set = [], + dependencies: [Dependency] = [], + targets: [Target] = [], + swiftLanguageVersions: [SwiftVersion]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil + ) +} +``` + +Furthermore, a new `Package.Dependency.Trait` type is introduced that can be used +to configure the traits of a dependency. + +```swift +extension Package.Dependency { + /// A struct representing an enabled trait of a dependency. + public struct Trait: Hashable, Sendable, ExpressibleByStringLiteral { + /// Enables the default traits of a package. + public static let default: Self + + /// A condition that limits the application of a dependencies trait. + public struct Condition: Hashable, Sendable { + /// Creates a package dependency trait condition. + /// + /// - Parameter traits: The set of traits that enable the dependencies trait. If any of the traits are enabled on this package + /// the dependencies trait will be enabled. + public static func when( + traits: Set + ) -> Self? + } + + /// The name of the enabled trait. + public var name: String + + /// The condition under which the trait is enabled. + public var condition: Condition? + + /// Initializes a new enabled trait. + /// + /// - Parameters: + /// - name: The name of the enabled trait. + /// - condition: The condition under which the trait is enabled. + public init( + name: String, + condition: Condition? = nil + ) + + public init(stringLiteral value: StringLiteralType) + + /// Initializes a new enabled trait. + /// + /// - Parameters: + /// - name: The name of the enabled trait. + /// - condition: The condition under which the trait is enabled. + public static func trait( + name: String, + condition: Condition? = nil + ) -> Trait + } +} +``` + +The dependency APIs are then extended with new variants that take a `Trait` parameter: + +```swift +extension Package.Dependency { + // MARK: Path + + public static func package( + path: String, + traits: Set + ) -> Package.Dependency + + public static func package( + name: String, + path: String, + traits: Set + ) -> Package.Dependency + + // MARK: Source repository + + public static func package( + url: String, + from version: Version, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + branch: String, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + revision: String, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + _ range: Range, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + _ range: ClosedRange, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + exact version: Version, + traits: Set + ) -> Package.Dependency + + // MARK: Registry + + public static func package( + id: String, + from version: Version, + traits: Set + ) -> Package.Dependency + + public static func package( + id: String, + exact version: Version, + traits: Set + ) -> Package.Dependency + + public static func package( + id: String, + _ range: Range, + traits: Set + ) -> Package.Dependency + + public static func package( + id: String, + _ range: ClosedRange, + traits: Set + ) -> Package.Dependency +} +``` + +Lastly, traits can also be used to conditionalize `SwiftSettings`, `CSettings`, +`CXXSettings` and `LinkerSettings`. For this the `BuildSettingCondition` is extended. + +```swift +/// Creates a build setting condition. +/// +/// - Parameters: +/// - platforms: The applicable platforms for this build setting condition. +/// - configuration: The applicable build configuration for this build setting condition. +/// - traits: The applicable traits for this build setting condition. +public static func when( + platforms: [Platform]? = nil, + configuration: BuildConfiguration? = nil, + traits: Set? = nil +) -> BuildSettingCondition { + precondition(!(platforms == nil && configuration == nil)) + return BuildSettingCondition(platforms: platforms, config: configuration, traits: nil) +} +``` + +### Trait unification + +At this point, it is important to talk about the trait unification across the +entire dependency graph. After dependency resolution the union of enabled traits +per package is calculated. This is then used to determine both the enabled +optional dependencies and the enabled traits for the compile time checks. Since +the enabled traits of a dependency are specified on a per package level and not +from the root of the tree, any combination of enabled traits must be supported. +A consequence of this is that all traits **should** be _additive_. Enabling a trait +**should not** disable functionality i.e. remove API or lead to any other +**SemVer-incompatible** change. + +#### Mutally exclusive traits + +Some rare use-cases may want mutally exclusive traits which are incompatible to +be enabled at the same time. This should be avoided if possible because it +requires the whole dependency graph to coordinate on what trait to enable. In +the rare case where mutually exclusive traits are used consider adding a +compiler error to detect this during build time. + +```swift +#if Trait1 && Trait2 +#error("Trait1 and Trait2 are mutually exclusive") +#endif +``` + +A few options to avoid mutually exclusive traits: +- Separate the code into multiple packages +- Choose one trait over the other when possible +- Use platform checks `#if os(Windows)` when possible + +### Default traits + +Default traits allow package authors to define a set of traits that they think +cater to the majority use-cases of the package. When choosing the initial +default traits or adding a new default trait it is important to consider that +removing a default trait is a **SemVer-incompatible** change since it can potentially +remove APIs. + +### Trait specific command line options for `swift test/build/run` + +When executing one of `swift test/build/run` options can be passed to control which +traits for the root package are enabled: + +- `--traits` _TRAITS_: Enables the passed traits of the package. Multiple traits + can be specified by providing a comma separated list e.g. `--traits + Trait1,Trait2`. +- `--enable-all-traits`: Enables all traits of the package. +- `--disable-default-traits`: Disables all default traits of the package. + +### Trait namespaces + +Trait names are namespaced per package; hence, multiple packages can define the +same trait names. Moreover, it is an expected scenario that multiple packages +define the same trait name and conditionally enable the equivalent named trait +in their dependencies. + +### Trait limitations + +To prevent abuse, limit the complexity and make sure it integrates with the +compiler a few limitations are imposed. + +#### Number of traits + +[Other +ecosystems](https://blog.rust-lang.org/2023/10/26/broken-badges-and-23k-keywords.html) +have shown that a large number of traits can have significant impact on +registries and dependency managers. To avoid such a scenario an initial maximum +number of 300 defined traits per package is imposed. This can be revisited later +once traits have been used in the ecosystem extensively. + +### Allowed characters for trait names + +Since traits can show up both in the `Package.swift` and in source code when +checking if a trait is enabled, the allowed characters for a trait name are +restricted to [legal Swift +identifier](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/summaryofthegrammar/). +Additional, the following rules are enforced on trait names: + +- `default` and `defaults` (in any letter casing combination) are not allowed as + trait names to avoid confusion with default traits. + +### swift package dump-package + +The `swift package dump-package` command will include information of the trait configuration for the +package and the dependencies it uses in the JSON output. + +## Impact on existing packages + +There is no impact on existing packages. Any package can start adopting package +traits but in doing so **must not** move existing API behind new traits. Even if +the trait is a enabled by default any consumer might have already disabled all +default traits; hence, moving API behind a new default trait could potentially +break them. + +### Impact on other build systems + +The initial impact on other build systems should be minimal. Exisiting packages +must not move exisiting APIs behind a trait. For new APIs that are guarded by a +trait a build system must pass the correct `SWIFT_ACTIVE_COMPILATION_CONDITIONS` +when building the modules of the package. Other build systems might want to +consider to expose a way to model traits in their target description. + +### Impact on documentation + +Traits are used to conditionally compile code. When building documentation the +symbol graph extracter will only see the code that is actually compiled. Systems +that produce documentation for packages should default to building with all +traits enabled so that all API documentation is visible. + +## Future directions + +### Consider traits during dependency resolution + +The implementation to this proposal only considers traits **after** the +dependency resolution when constructing the module graph. This is inline with +how platform specific dependencies are currently handled. In the future, both +platform specific dependencies and traits can be taken into consideration during +dependency resolution to avoid fetching an optional dependency that is not +enabled by a trait. Changing this **doesn't** require a Swift evolution proposal +since it is just an implementation detail of how dependency resolution currently +works. + +### Integrated compiler trait checking + +The current proposal passes enabled traits via custom defines to the compiler +and code can check it using regular define checks (`#if DEFINE`). In the future, +we can extend the compiler to make it aware of package traits to allows syntax +like `#if trait(FOO)` or implement an extensible configuration macro similar to +Rust's `cfg` macro. + +### Enabled trait compile time checking + +Since trait unification is done for every package in the graph during build time +the information which module enabled which trait of its dependencies is lost. +As a consequence it might be that a package accidentally uses an API from a +dependency which is guarded by a trait that another package in the graph has +enabled. Since the traits that any one package in the graph enables on its +dependencies are not considered part of the semantic version, it can happen that +disabling a trait could result in breaking a build. In the future, we could +integrate trait checking further into the compiler where it understands if an +API is only available if a certain trait is set. + +> Cargo currently [treats this +similar](https://users.rust-lang.org/t/is-disabling-features-of-a-dependency-considered-a-breaking-change/94302/2) +and doesn't consider disabling a cargo feature a breaking change. + +### Different default traits depending on platform + +A future evolution could allow to mark traits as default depending on the +platform that the package is build on. This would allow packages such as the +`swift-openapi-generator` to default the used transport depending on the +platform which makes it even easier to offer users the best out of box +experience. This is left as a future evolution since it intersects interestingly +with the future direction "Consider traits during dependency resolution". If +default traits depend on the target build platform then this must be an input to +the dependency resolution. + +### Globally configured traits + +One use-case where mutually exclusive traits are used is to configure the +behaviour of a single package globally. This means that only the final +executable is deciding what trait to enable. In a future proposal, we could +introduce a new setting for marking a trait as globally configured and check +that only executable targets are enabling such a trait. + +### Tooling to test different trait combinations + +Since a single package should support building with any combination of traits, +it would be helpful to offer package authors tooling to build and test all +combinations. A new option `--all-trait-combinations` could be added to +`swift test/build/run` make testing all combinations easy as possible. + +### Surface traits in documentation + +If the compiler gains knowledge about package traits in the future, we could +extract information if a public API is guarded by a trait and surface this in +the documentation. + +## Alternatives considered + +### Different naming + +During the implementation and writing of the proposal different names for +_package traits_ have been considered such as: +- Package features +- Package optional features +- Package options +- Package parameters +- Package flags +- Package configuration + +A lot of the other considered names have other meanings in the language already. +For example `feature` is already used in expressing compiler feature via +`enable[Upcoming|Experimental]Feature` and the `hasFeature` check. + +_Package traits_ are also consistent with [the "traits" concept in the +`swift-testing` +library](https://github.com/apple/swift-testing/blob/25d0eed9b339de36365ff16deb9a3d9c64322f1c/Sources/Testing/Traits/Trait.swift#L22). + +### Using SPI instead + +During the investigation how to solve the optional dependency problem `@_spi` +was considered; however, the problem with `@_spi` is that the code is still +compiled and present in the final binary. Optional dependencies can't work like +this since the symbols potentially aren't present during compile time. + +### Enum based traits + +Traits could be expressed via an enum in the package description which would +make sure that they are statically typed. This proposal decided to use `String` +based trait names instead to align with the other definitions inside the package +description such as `targets` or `products`. + +## Prior art + +Other dependency managers have similar features to control optional dependencies +and conditional compilation. + +- [Cargo](https://doc.rust-lang.org/cargo/) has [optional features](https://doc.rust-lang.org/cargo/reference/features.html) that allow conditional compilation and optional dependencies. +- [Maven](https://maven.apache.org/) has [optional dependencies](https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html). +- [Gradle](https://gradle.org/) has [feature variants](https://docs.gradle.org/current/userguide/feature_variants.html) that allow conditional compilation and optional dependencies. +- [Go](https://golang.org/) has [build constraints](https://golang.org/pkg/go/build/#hdr-Build_Constraints) which can conditionally include a file. +- [pip](https://pypi.org/project/pip/) dependencies can have [optional dependencies and extras](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies). +- [Hatch](https://hatch.pypa.io/latest/) offers [optional dependencies](https://hatch.pypa.io/latest/config/metadata/#optional) and [features](https://hatch.pypa.io/latest/config/dependency/#features). diff --git a/proposals/0451-escaped-identifiers.md b/proposals/0451-escaped-identifiers.md new file mode 100644 index 0000000000..3105c97626 --- /dev/null +++ b/proposals/0451-escaped-identifiers.md @@ -0,0 +1,464 @@ +# Raw identifiers + +* Proposal: [SE-0451](0451-escaped-identifiers.md) +* Author: [Tony Allevato](https://github.com/allevato) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#76636](https://github.com/swiftlang/swift/pull/76636), [swiftlang/swift-syntax#2857](https://github.com/swiftlang/swift-syntax/pull/2857) +* Previous Proposal: [SE-0275](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-revisiting-backtick-delimited-identifiers-that-allow-more-non-identifier-characters/74432), [review](https://forums.swift.org/t/se-0451-raw-identifiers/75602), [acceptance](https://forums.swift.org/t/accepted-with-revision-se-0451-raw-identifiers/76387)) + + +## Introduction + +This proposal adds _raw identifiers_ to the Swift grammar, which are backtick-delimited identifiers that can contain characters other than the current set of identifier-allowed characters in the language. + +## Motivation + +When naming things in Swift, we use identifiers that are limited to a specific set of characters defined by the [Swift grammar](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/summaryofthegrammar/). For the vast majority of identifiers, this set is perfectly reasonable: names of APIs should be no more complex than they need to be, and many natural names do fit within these requirements. However, there do exist some use cases that would be improved by more flexibility in naming: + +* Declarations whose names serve a purely descriptive purpose, like tests. +* Externally-defined entities that already have natural names that do not fit within Swift's rules for identifiers. + +The unifying principle behind these motivating cases is that information is lost or complexity is unnecessarily increased when a developer must forego a more natural name for one that meets Swift's requirements. Indeed, Swift already admits this to a degree by defining the backtick syntax to allow reserved keywords to be used as identifiers. + +### Descriptive test naming + +When this feature was originally proposed as [SE-0275](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md), one of the reasons for rejection was that ["[t]he core team feels that this use case could be better handled by test library design that allowed strings to be associated with test cases, rather than try to encode that information entirely in symbol names."](https://forums.swift.org/t/se-0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers/32538/46) + +We now have a new testing library in the Swift project called [swift-testing](https://github/com/apple/swift-testing) that implements the feature described above: + +```swift +@Test("square returns x * x") +func squareIsXTimesX() { + #expect(square(4) == 4 * 4) +} +``` + +Unfortunately, if the user wants to provide a descriptive name for the test, they are now forced to name it **twice**: once in the `@Test` macro and then again in the function itself. As the number of tests increases (see swift-testing's own tests, for [example](https://github.com/swiftlang/swift-testing/blob/main/Tests/TestingTests/ClockTests.swift)), so, too, does the burden on the developer. + +This is not merely redundant; it also creates an inconsistency in how the tests are referenced by different tools. Test result reports that are generated by the framework will use the display name, as may user interfaces that display the test hierarchy. However, lower-level tooling such as the compiler itself, the linker, the debugger, index data used for code navigation, and backtraces, will only show the Swift language symbol. + +Re-designing the testing framework to eliminate the function name would also not be the correct solution. Consider this hypothetical alternative using a trailing closure that would remove one of the redundant names: + +```swift +// Not being proposed +@Test("square returns x * x") { + #expect(square(4) == 4 * 4) +} +``` + +This would be unsatisfactory for a few reasons: + +* The current `@Test func` syntax used by swift-testing is a strength because it provides progressive disclosure rather than introducing a new bespoke DSL for tests. +* There are subtle differences between how the compiler processes closures and regular function declarations that could result in awkward behavior for test authors. +* The testing framework must still contrive a symbol name for the test by either mangling the user-provided description into something that Swift can support or creating a completely unique name. The user now has no control over that name and no easy way to discover it. That name would still appear in the debugger, index data, and backtraces, so the inconsistency described above still remains. + +### Naturally non-alphabetic identifiers + +There are some situations where the best name for a particular symbol is numeric or at least starts with a number. As an example, consider a hypothetical color design system that lets users choose a base hue and then a variant/shade of that hue from a fixed set. These design systems often give the color variants numeric names that indicate their intensity—100, 200, and so forth. A naïve API might represent the variant as a numeric value: + +```swift +struct Color { + init(hue: Hue, variant: Int) +} +``` + +But this API can only enforce correctness at runtime. A developer should naturally reach for an `enum` type for a fixed set of variants like this, but they must transform the name or add unnecessary ceremony to make it fit into Swift's naming rules: + +```swift +struct Color { + init(hue: Hue, variant: ColorVariant) +} + +// Not ideal; leading underscore usually means "don't use this" +enum ColorVariant { + case _50 + case _100 + case _200 + // ... +} +let color = Color(hue: .red, variant: ._100) + +// Repetitive +enum ColorVariant { + case variant50 + case variant100 + case variant200 + // ... +} +let color = Color(hue: .red, variant: .variant100) + +// "v" is for... version? No, that's not right. +enum ColorVariant { + case v50 + // ... +} +``` + +### Code generators and FFI + +Generating code from an external source like a data exchange format specification or transpiling an API from another language is common in programming. When generating such code, names in the original data source may not be usable as Swift identifiers. + +Another example is generating type-safe accessors for resources bundled in a library on the filesystem or in an asset catalog. The names of such resources may not necessarily obey Swift conventions—an example is Apple's own _SF Symbols_, which has images with names like `1.circle`. + +In these situations, the generator needs to implement a fixed set of transformation rules to convert names that aren't valid identifiers into names that are. One problem with this approach is that such conversions have a cascading effect: the result of converting a Swift-incompatible name into a Swift-compatible one might produce a name that could also be a plausible name in the original source. Therefore, code generators must complicate their logic to avoid such collisions so that an unexpected input doesn't produce invalid code and block the user. This collision-breaking logic often results in arbitrary and difficult-to-understand rules that the user must nevertheless internalize in order to know what to write in the destination language when faced with one of these names in the origin language. + +### Module naming at scale + +Swift modules (and Clang modules with which Swift interoperates) occupy a single, flat[^1] namespace. The prevailing code organization pattern among many Swift projects is to treat a module as a semantic unit and to give them short, simple names. This approach has already been proven to cause difficulties in real-world builds, as evidenced by the need for the [module aliasing](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0339-module-aliasing-for-disambiguation.md) feature added in Swift 5.7 to reduce the problems that arise when modules with conflicting names occur in the same dependency graph. + +[^1]: Clang supports submodules, but they are still _contained_ by their parent modules and importing a parent module also imports all of its (implicit) submodules. This is distinct from having independent compilation units that share a hierarchical namespace. Java packages are an example of the latter. + +Experience has shown that this model does not scale well to massively large Swift codebases that use different organizational and build models; for example, companies that use monorepos and build with distributed build systems like Bazel, which we will elaborate on below. + +First, many of the projects now using Swift historically used Objective-C or are using common components that are written in Objective-C. That Objective-C code was not written with modules in mind and uses header-file-based inclusions. In order to interoperate with Swift, all of that code would need to be grouped into modules. This would be a significant undertaking to do manually. + +Additionally, requiring human developers to choose their own module names has high cognitive overhead and easily leads to conflicts. A component in one part of the codebase can add a new dependency that breaks a different project elsewhere simply by having a conflicting module name with something else the project is already using. Module aliasing has originally designed and implemented does not immediately provide a satisfying remedy here, because it still requires that the project _consuming_ the modules opt into aliasing to resolve the conflict, and it forces them to choose a _second, contrived_ name for the module. + +To work around these issues, the approach Bazel has adopted is to automatically derive a module name for a build target based on the unique identifier—a path to its location in the repository—that Bazel already uses to refer to it. This removes the burden from the developer to choose a unique name and reduces the chance of collisions. However, this process still projects a name from a hierarchical namespace onto a flat one, so the build target's natural identifier must be contorted to fit into the identifier-safe naming required by a Swift module. For example, a module defined at the path `myapp/extensions/widget/common/utils` would be given the name `myapp_extensions_widget_common_utils`. + +While this works, it is not ideal. Users often know where the code that they want to import lives before they write the line of code that imports it. In order to write the `import` declaration, they must mentally transform the path to the code into the module name, or infrastructure developers must provide tooling to do so. And like the code generation case discussed above, the transformation is not usually reversible, which makes certain kinds of tooling difficult to write and does not the possibility of collisions. For example, a module named `myapp_extensions_widget_common_utils` could live at `myapp/extensions/widget/common/utils` or at `myapp/extensions/widget/common_utils`. Designing a reversible transform would require making the encoding less friendly to humans who need to read and write those names in their code. + +## Proposed Solution + +We propose extending the backtick-based syntax used to escape keywords as identifiers to support identifiers that can contain characters other than the standard identifier-safe characters allowed by Swift. + +We will distinguish these two uses of backticks with the following terminology: + +* An **escaped identifier** is a sequence of characters surrounded by backticks that starts with `identifier-head` and is followed by zero or more `identifier-character`, as defined in the Swift [grammar](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/summaryofthegrammar/). This is what Swift supports today to treat keywords as identifiers. +* A **raw identifier** is a sequence of characters surrounded by backticks that contains characters other than those allowed in an escaped identifier. The exact contents are described in [Permitted Characters](#permitted-characters) below. + +In both cases, the backticks are **not** considered part of the identifier; they only delimit the identifier from surrounding tokens. + +Raw identifiers would provide more flexibility in naming for use cases like the ones discussed earlier in the document as we explain below. + +### Describing tests + +Test functions could use raw identifiers to describe their purpose clearly without redundancy: + +```swift +@Test func `square returns x * x`() { + #expect(square(4) == 4 * 4) +} +``` + +Here, the user need only provide a **single** description of the test function. That one name will be used anywhere that the test is referenced: test result reports, the debugger, index data, crash logs, and so forth. + +A key observation here is that when using swift-testing, or another framework like XCTest, the name of the test function serves **only to describe the test.** It is _not_ real API, and its only caller is the test framework itself that discovers the function through some dynamic or generative mechanism. Since the name will only be written once at the declaration site, it makes sense to allow for more flexible and verbose naming for the sake of expressibility and simplicity. + +Using raw identifiers for test naming fits very well with Swift's philosophy of progressive disclosure. Rather than using a bespoke API for descriptive naming, the author's journey follows a path through the following stages: + +* You learn how to write a regular Swift function. +* You learn how to make it a unit test by prepending `@Test` to it. +* You learn that raw identifiers exist and how to apply them to the names of your tests, and this knowledge applies for any identifiers in the language rather than just a specific framework. + +### Other kinds of identifiers + +Raw identifiers would provide a clean solution to the cases where the most natural name for an identifier is numeric. Considering the color design system again, we could write: + +```swift +enum ColorVariant { + case `50` + case `100` + case `200` +} +let color = Color(hue: .red, variant: .`100`) +``` + +One could claim that the backticks are just as much ceremony as using a different prefix, but we believe that raw identifiers are a better choice because they would be using **a common notation** that applies to **all** such symbols. This _reduces_ complexity across all Swift codebases, rather than requiring each developer/API to come up with their own conventions as they do today, which increases complexity for readers. + +Raw identifiers could likewise be used for other kinds of foreign identifiers introduced by FFI or code generation. For example, a tool that generates strongly-typed accessors for resources could use raw identifiers to produce a more direct mapping from the resource name to the name in code, reducing complexity by making collisions less likely. + +```swift +extension UIImage { + static var `10.circle`: UIImage { ... } +} +``` + +Raw identifiers also alleviate the naming problems for modules in very large codebases. Instead of deriving a name using a non-reversible transformation, the build system could simply name the module with the unique identifier that it already associates with that module: + +```swift +import `myapp/extensions/widget/common/utils` + +// if explicit disambiguation is needed: +`myapp/extensions/widget/common/utils`.SomeClass +``` + +Doing so greatly improves the experience of developers working in large codebases who can more easily map imports to where the code resides and vice versa, and it also trivializes writing automated tooling that manages build dependencies by scanning imports written in code and updating the build system's definition of the targets. + +### Support in other languages + +Several other modern programming languages acknowledge the occasional but important need that exists for identifiers that do not meet their standard requirements and support raw identifiers like the ones being proposed here. + +In [F#](https://fsharp.org/specs/language-spec/4.1/FSharpSpec-4.1-latest.pdf), identifiers that are surrounded by double backticks and can contain any characters excluding newlines, tabs, and double backticks. + +In [Groovy](https://groovy-lang.org/syntax.html#_quoted_identifiers), identifiers after the dot in a dot-expression can be written as quoted string literals containing characters other than those allowed in standard identifiers. + +In [Kotlin](https://kotlinlang.org/docs/reference/grammar.html#Identifier), identifiers can be surrounded by single backticks. The characters that are allowed inside backticks may differ based on the target backend (for example, the JVM places additional restrictions). The closest comparison to Swift would be Kotlin/Native, which permits any character other than carriage return, newline, or backtick. + +In [Scala](https://www.scala-lang.org/files/archive/spec/2.11/01-lexical-syntax.html), an identifier can contain an arbitrary string when surrounded by single backticks, but host systems may impose some restrictions on which strings are legal for identifiers. + +In [Zig](https://ziglang.org/documentation/master/#Identifiers), an identifier that does not meet the standard requirements may be expressed by using an `@` symbol followed by a string literal, such as `@"with a space"`. + +## Detailed Design + +### Permitted characters + +A raw identifier may contain any valid Unicode characters except for the following: + +* The backtick (`` ` ``) itself, which terminates the identifier. +* The backslash (`\`), which is reserved for potential future escape sequences. +* Carriage return (`U+000D`) or newline (`U+000A`); identifiers must be written on a single line. +* The NUL character (`U+0000`), which already emits a warning if present in Swift source but would be disallowed completely in a raw identifier. +* All other non-printable ASCII code units that are also forbidden in single-line Swift string literals (`U+0001...U+001F`, `U+007F`). + +In addition to these rules, some specific combinations that require special handling are discussed below. + +#### Whitespace + +A raw identifier may have leading, trailing, or internal whitespace; however, it may not consist of *only* whitespace. "Whitespace" is defined here to mean characters satisfying the Unicode `White_Space` property, exposed in Swift by `Unicode.Scalar.Properties.isWhitespace`. + +#### Operator characters + +A raw identifier may start with, contain, or end with operator characters, but it may not contain **only** operator characters. To avoid confusion, a raw identifier containing only operator characters is treated as a parsing error: it is neither a valid identifier nor an operator: + +```swift +func + (lhs: Int, rhs: Int) -> Int // ok +func `+` (lhs: Int, rhs: Int) -> Int // error + +let x = 1 + 2 // ok +let x = 1 `+` 2 // error +``` + +This leaves the door open for a future use of that syntax, should it be desired. There is more discussion of this in [Alternatives Considered](#alternatives-considered). + +### Symbol generation and mangling + +Backticks are required to delimit a raw identifier written in source code, but they are **not part** of the identifier as it relates to symbols generated by the compiler. For example, ``func `with a space`()`` defines a function whose name is `with a space` without backticks. This is the same as escaped identifiers today: ``func `for`()`` defines a function named `for`, not `` `for` ``. + +Fortunately, Swift's symbol mangler already handles non-ASCII-identifier code units in symbol names today by converting them using Punycode. The current algorithm already works for raw identifiers, with one change needed: it does not recognize that an identifier starting with a digit needs to be encoded. Consider the following example: + +```swift +// Module "test" +public struct FontWeight { + public func `100`() { ... } +} +``` + +Without Punycoding an identifier starting with a digit, the current implementation would mangle the function above as `$s4test10FontWeightV3X00yyF`, which round-trips incorrectly as `test.FontWeight.X00() -> ()`. This proposal would implement the necessary change so that it produces the correct mangling `$s4test10FontWeightV004_100_yyF`. + +### Property wrappers + +When a property wrapper wraps a variable named with a raw identifier, backticks must also be used to refer to its backing storage or its projected value. The prefix sigil (`_` or `$`, respectively) is part of those identifiers, so it is placed _inside_ the backticks. + +```swift +@propertyWrapper +struct Wrapper { + var wrappedValue: T + var projectedValue: Wrapper +} + +struct UsesWrapper { + @Wrapper var `with a space`: Int +} + +let x = UsesWrapper() +print(x.`_with a space`) // correct +doSomethingWith(x.`$with a space`) // correct + +print(x._`with a space`) // error +doSomethingWith(x.$`with a space`) // error +``` + +The dollar sign (`$`) has two special meanings in Swift when used at the beginning of an identifier: + +* When followed by an integer, it references an unnamed closure argument at that index (e.g., `$0`). +* When followed by an identifier, it references the projected value of a property wrapper (e.g., `$someBinding`). + +We must then decide what an identifier like `` `$0` `` means. The only correct choice is to treat `` `$0` `` as a regular identifier, not a closure argument. This is necessary to allow property wrapper projections to be accessed on properties whose names are numeric raw identifiers: + +```swift +@propertyWrapper +struct Wrapper { + var wrappedValue: T + var projectedValue: Wrapper +} + +let `$1` = "hello" // error: cannot declare entity named '$1'; the '$' prefix + // is reserved for implicitly-synthesized declarations + +struct UsesWrapper { + @Wrapper var `0`: Int + + func f() { + let closure: (Int) -> Int { + doSomethingWith(`$0`) // ok, refers to projected value of `0` + return $0 // ok, refers to unnamed closure argument + } + } +} +``` + +### Member access expressions + +When escaping a keyword as an identifer, Swift allows the backticks to be omitted in certain contexts when the compiler knows that it cannot be a keyword. Most commonly, this occurs with member access expressions: + +```swift +enum Access { + case `public` // must be escaped here + case `private` +} + +_ = Access.`public` // ok, but not necessary +_ = Access.public // also ok +``` + +Raw identifiers, on the other hand, **must be escaped by backticks in all contexts** since they can contain characters that could be treated as parsing delimiters: + +```swift +struct S { + var `with a space` = 0 +} + +S().with a space // error, the parser can't know where the member + // name is supposed to end +S().`with a space` // correct +``` + +This rule also guarantees an important invariant for tuples: raw identifiers are never confusable with tuple element indices. If a tuple member is accessed with a numeric raw identifier (i.e., `` tuple.`0` ``), that will only interpreted as a tuple element _label_ and never a tuple element _index_. Similarly, if a tuple member is accessed with an unescaped integer (i.e., `tuple.0`), the behavior has not changed: it is only interpreted as an _index_ and never as a _label_, even if there is a label with the same name. For example, + +```swift +let x = (0, 1) +_ = x.`0` // error + +let a = (5, `0`: 10) +let b = y.0 // z <- 5 +let c = y.`0` // z <- 10 +``` + +### Objective-C Compatibility + +A Swift declaration named with a raw identifier must be given an explicit Objective-C name to export it to Objective-C. The compiler will emit an error if a declaration is given an explicit name with the `@objc(...)` attribute and that name is not a valid Objective-C identifier, or if a declaration is otherwise inferred to be `@objc` and the Swift name of that declaration is not a valid Objective-C identifier. + +```swift +@objc class `Class with a Space` { // error + @objc(someFunction) func `some function`() {} // ok + @objc(`not valid`) func myFunction() {} // error +} + +@objc @objcMembers class AnotherClass { + var `some property`: Int // error +} +``` + +The Objective-C runtime can dynamically support class names and selectors that contain characters that are not expressible in source code, but we do not consider that to be something should be supported. Therefore, such names are forbidden even for symbols that would only be exposed to the runtime but not written to the generated header file. + +### Module names + +The Clang module map parser already supports modules with non-identifier characters in their names by writing their names in double quotes: + +``` +module "some/module/name" { + header "..." +} +``` + +Such a module would already import cleanly into Swift simply by allowing the module name in an `import` statement to be a raw identifier: + +```swift +import `some/module/name` +``` + +Swift modules pose a different challenge. The compiler's serialization and import search path logic assumes that compiled module artifacts have filenames that match the name of the module; for example, `SomeModule` would be named `SomeModule.swiftmodule`. Using raw identifiers as module names would limit us to only those characters supported in filenames, and these character sets differ between platforms. + +There is a feature in the compiler called an "explicit Swift module map" that uses a JSON file to explicitly list all dependencies without using search paths. This can be used to give a module a different name than its filename. However, it does not seem appropriate to restrict the use of raw identifier module names only to users of this feature; a solution should also include users of standard module resolution. + +This proposal suggests an alternative: Continue to require that the `-module-name` passed during compilation be a filesystem-compatible name, as is the case today, and then allow aliases set by `-module-alias` to be raw identifiers. This effectively separates the "physical" name of the module (its filesystem name and its ABI name) from what users write in source code. Build systems using this feature would generate the physical names behind the scenes for users and provide the aliases to the compiler, completely transparent to the user. This approach elegantly builds on top of existing features in the compiler and avoids adding more complexity to serialization. + +Finally, users who want the ABI name (the name that appears in symbol manglings) to match exactly what users are typing in source code—for example, to ensure consistency between the module names in source and what appears in the debugger and crash logs—could go one step further and pass the raw identifier using the `-module-abi-name` flag. + +### Impacts on tooling + +During the review of SE-0275, concerns were raised about whether identifiers containing whitespace (or other traditional delimiter characters) would make language tooling harder to write or use. While we cannot address all possible editors here, we believe that modern LSP-driven editors can handle raw identifiers gracefully. One specific concern raised was that double-clicking a word in a raw identifier would not be able to select the whole identifier. Since then, the Language Server Protocol has defined a `textDocument/selectionRange` request that can be used to determine an "interesting" selection range for a text position in a document. If this request is made for the text position that is inside a raw identifier, Swift's language server could return the range of that identifier in the response, allowing the host editor to select it in its entirety. + +Likewise, syntax highlighting for raw identifiers is straightforward; they are no more complicated than a single-line string literal. + +### Impacts on future reflective APIs + +During the review of SE-0275, the point was raised that allowing a name to contain common type delimiters could create ambiguity for future APIs that look up symbols via runtime reflection: + +```swift +struct X { + struct Y {} // 1 +} +struct `X.Y` {} // 2 + +// Does this return 1 or 2? +_ = hypotheticalTypeByName("X.Y") +``` + +While these APIs do not yet exist, we feel that they could be implemented unambiguously. Without designing such an API (it is certainly out of scope for this proposal), we can identify some basic principles to address that concern. + +First, we can imagine that such an API would—at least at a lower level—allow users to drill down through the module context and its descendant contexts explicitly; for example, something in the style of `myModule.type("X", genericArgs: [swiftStdlibModule.type("Int")]).type("Y")` would be clearly distinct from `myModule.type("X.Y")`. + +Then, if a simplified API like `hypotheticalTypeByName` is desired, we can observe that the type name is being passed in a form such that the library would need the capability to parse a subset of the language's type grammar in order to understand which parts are generic arguments, which are member references, and so forth. Therefore, it stands to reason that the argument to `hypotheticalTypeByName` should be **written as it would be in source**, including any raw identifier delimiters. In doing so, each case is easily distinguished: + +```swift +_ = hypotheticalTypeByName("X.Y") // returns 1 above +_ = hypotheticalTypeByName("`X.Y`") // returns 2 above +``` + +## Alternatives Considered + +[SE-0275](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md) originally proposed using backticks to support qualified references to operators, such as the following: + +```swift +let add: (Int, Int) -> Int = Int.+ // not allowed today +let add: (Int, Int) -> Int = Int.`+` // would have been allowed by SE-0275 +``` + +As mentioned above, this proposal does not allow this; it is an error to write a raw identifier containing **only** identifier characters. We agree with the Core Team's original feedback that since backticks remove the "magic" around a special character sequence, there is potential for confusion about whether writing `` `+` `` refers to the operator or to a different regular identifier named `+`. If we intended the former, there are also open questions about how one would distinguish prefix, postfix, infix operators with such a syntax. This feature, if desired, demands its own in-depth design and separate proposal. + +## Future Directions + +There are natural extensions of the raw identifier syntax that could be used to support multi-line identifiers or identifiers containing backticks by adopting a similar syntax to that of raw string literals: + +~~~swift +let x = #`contains`some`backticks`# +let y = ##`contains`#single`#pound`#delimiters`## + +func ``` + This is a function that multiplies two numbers. + It's named this way to make sure you REALLY know + what you're getting into when you multiply two + numbers. + ```(_ x: Int, _ y: Int) { x * y } + +let fifteen = ``` + This is a function that multiplies two numbers. + It's named this way to make sure you REALLY know + what you're getting into when you multiply two + numbers. + ```(3, 5) +~~~ + +At this time, however, we do not believe there are any compelling use cases for such identifiers. + +### Escape sequences inside raw identifiers + +Raw identifiers follow similar parsing rules as string literals with respect to unprintable characters, which raises the question of how to handle backslashes. The use cases served by many backslash escapes—such as writing unprintable characters—are not desirable for identifiers, so we could choose to treat backslashes as regular literal characters. For example, `` `hello\now` `` would mean the identifier `hello\now`. This could be confusing for users though, who might expect the `\n` to be interpreted the same way that it would be in a string literal. Treating backslashes as literal characters today would also close the door on a viable method of escaping characters inside raw identifiers if we decide that it is needed later. For these reasons, we currently forbid backslashes and leave their purpose to be defined in the future. + +## Source compatibility + +This proposal is purely additive; it does not affect compatibility with existing source code. + +## ABI compatibility + +This proposal has no effect on existing ABI. It only makes new valid Swift symbol manglings for symbols that were previously invalid. + +## Implications on adoption + +For codebases built entirely from source using the same version of the Swift compiler would see no negative impact from using this feature. + +For example, if a resilient library used raw identifiers in any declarations that are serialized into the textual interface (e.g., `public` or `@usableFromInline`), that interface would not be consumable by versions of the compiler prior to when this feature was introduced because the older compilers would not be able to parse the identifiers. This, however, is the case for any feature that introduces a new syntax into the language. diff --git a/proposals/0452-integer-generic-parameters.md b/proposals/0452-integer-generic-parameters.md new file mode 100644 index 0000000000..b2ca7b2083 --- /dev/null +++ b/proposals/0452-integer-generic-parameters.md @@ -0,0 +1,625 @@ +# Integer Generic Parameters + +* Proposal: [SE-0452](0452-integer-generic-parameters.md) +* Authors: [Alejandro Alonso](https://github.com/Azoy), [Joe Groff](https://github.com/jckarter) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#75518](https://github.com/swiftlang/swift/pull/75518), [swiftlang/swift#78248](https://github.com/swiftlang/swift/pull/78248) +* Review: ([pitch](https://forums.swift.org/t/integer-generic-parameters/74181)) ([first review](https://forums.swift.org/t/se-0452-integer-generic-parameters/75844)) ([second review](https://forums.swift.org/t/second-review-se-0452-integer-generic-parameters/77043)) ([acceptance](https://forums.swift.org/t/accepted-se-0452-integer-generic-parameters/77507)) + +## Introduction + +In this proposal, we introduce the ability to parameterize generic types +on literal integer parameters. + +## Motivation + +Swift does not currently support fixed-size or fixed-capacity collections +with inline storage. (Or at least, it doesn't do so *well*, not without +forming a struct with some specific number of elements and doing horrible +things with `withUnsafePointer` to handle indexing.) Most of the implementation +of something like a fixed-size array, or a fixed-capacity growable array with +a maximum size, or a hash table with a fixed number of buckets, is agnostic +to any specific size or capacity, so that implementation +would ideally be generic over size so that a library implementation can +be reused for any given size. + +Beyond inline storage sizes, there are other use cases for carrying integers +in type information, such as to represent an operation with a particular +input or output size. Carrying this information in types can allow for APIs +with stronger static guarantees that chains of operations match in the +number of elements they consume or produce. + +## Proposed solution + +Generic types can now be parameterized by integer parameters, declared using +the syntax `let : Int` inside of the generic parameter angle brackets: + +```swift +struct Vector { + /*implementation TBD*/ +} +``` + +A generic type with integer parameters can be instantiated using literal +integer arguments: + +```swift +struct Matrix4x4 { + var matrix: Vector<4, Vector<4, Double>> +} +``` + +Or it can be instantiated using integer generic parameters from the surrounding +generic environment: + +```swift +struct Matrix { + var matrix: Vector> +} +``` + +Integer generic parameters become static constant members of the type, with +the same visibility as the type itself: + +```swift +public struct Matrix { + // implicitly has these members: + // public static var columns: Int { get } + // public static var rows: Int { get } +} + +// From another module: + +import struct Matrices.Matrix + +print(Matrix<4, 3>.columns) // prints 4 +print(Matrix<4, 3>.rows) // prints 3 +``` + +Generic functions and methods can also be parameterized by integer generic +parameters. As with other generic parameters, the values of the generic +arguments for a call are inferred from the types of the argument values +provided to the call: + +```swift +func matmul( + _ l: Matrix, + _ r: Matrix +) -> Matrix { ... } + +let m1 = Matrix<4, 2>(...) +let m2 = Matrix<2, 5>(...) + +let m3 = matmul(m1, m2) // a = 4, b = 2, c = 5, result type is Matrix<4, 5> +``` + +Within an expression, a reference to an integer generic parameter evaluates +the parameter as a value of type `Int`: + +```swift +extension Vector { + subscript(i: Int) -> Element { + get { + if i < 0 || i >= count { + fatalError("index \(i) out of bounds [0, \(count))") + } + return element(i) + } + } +} +``` + +## Detailed design + +The grammar for generic parameter lists expands to include value generic +parameters: + +```swift +generic-parameter --> 'let' type-name ':' type +``` + +Correspondingly, signed integer literals can now appear as elements in +generic argument lists and as operands of generic requirements: + +```swift +generic-argument --> '-'? integer-literal +same-type-requirement --> type-identifier '==' '-'? integer-literal +``` + +Although they can appear as elements in generic parameter lists, integer +literals are still not allowed to appear as types in and of themselves, and +cannot be used as bindings for type generic parameters. + +```swift +let x: 2 // error, 2 is not a type +let y: Array<2> // error, Array's Element is a type generic parameter +``` + +Likewise, integer generic parameters cannot be used as standalone types in their +generic context. + +```swift +struct Foo { + let y: x // Error, x is not a type + let metax: x.Type // Error, x has no member `.Type` +} +``` + +The type referenced by a value generic parameter declaration must resolve to +the `Swift.Int` standard library type. (Allowing other types of value generic +parameter is a future direction.) + +```swift +struct Foo { } // OK (assuming no shadowing `Int` declaration) +struct Foo2 { } // also OK + +struct BadFoo { } // Error, generic parameters of type Float not supported + +typealias MyInt = Swift.Int +struct Bar { } // OK + +struct Baz: P { + typealias A = Int +} + +struct Zim { } // OK + +func contrived() { + struct Int { } + + struct BadFoo { } // Error, local Int not supported + + struct Foo { } // OK +} +``` + +Integer generic parameters of types become static members of that type, +with the same visibility as the type itself. It is an error to try to +declare another static property with the same name as an integer generic +parameter within the type declaration, just as it would if the property +were independently declared: + +```swift +struct Vec { + static let count: Int // error: Vec already has a static property `count` +} +``` + +In a type reference, an integer generic argument can be provided as either +a literal integer, or as a reference to an integer generic parameter from +the enclosing generic context. References to type generic parameters, +type generic parameter packs, or declarations other than integer generic +parameters is an error. (Allowing references to constants of integer type, +or more elaborate constant expressions, as generic parameters is a future +direction.) + +```swift +struct IntParam { } + +let a: IntParam<2> // OK +let b: IntParam<-2> // OK + +struct AlsoIntParam { + let c: IntParam // OK + + static let someIntegerConstant = 42 + let d: IntParam // Error, not an Int generic parameter + + let e: IntParam // Error, is a type generic parameter + let f: IntParam // Error, is a pack generic parameter +} +``` + +Conversely, using an integer generic parameter as an argument for a type +generic parameter is also an error. + +```swift +struct IntAndTypeParam { + let xs: Array // Error, x is an integer type parameter +} +``` + +An integer generic parameter can be constrained to be equal to a specific +literal value using a same-value constraint, spelled with `==` as for a +same-type constraint. Two integer generic parameters can also be constrained +to be equal to each other. + +```swift +struct TwoIntParams {} + +extension TwoIntParams where n == 2 { + func foo() { ... } +} + +extension TwoIntParams where n == m { + func bar() { ... } +} + +let x: TwoIntParams +x.foo() // OK +x.bar() // Error, doesn't match constraint + +let y: TwoIntParams +y.foo() // Error, doesn't match constraint +y.bar() // OK +``` + +Integer generic parameters cannot be constrained to be equal to type generic +parameters, concrete types, or to declarations other than generic parameters. +Integer generic parameters also cannot be constrained to conform to protocols. + +```swift +extension TwoIntParams where n == T {} // error +extension TwoIntParams where T == n {} // error +extension TwoIntParams where n == Int {} // error + +let globalConstant = 42 +extension TwoIntParams where n == globalConstant {} // error + +extension TwoIntParams where n: Collection // error +``` + +(In the same way overload resolution already works in Swift, extensions or +functions with generic constraints on integer parameters will only be chosen +for call sites at which those constraints always hold; we won't "dispatch" +based on the value of an argument from a less-constrained call site.) + +```swift +struct Foo { + func foo() { print("foo #1") } + + func bar() { + // Always prints "foo #1" + self.foo() + } +} + +extension Foo where n == 2 { + func foo() { print("foo #2") } +} + +Foo<2>().bar() // prints "foo #1" +Foo<2>().foo() // prints "foo #2" +``` + +## Source compatibility + +This proposal is a strict extension of the existing language. The `let n: Type` +syntax should ensure source compatibility if we expand the feature to allow +value generic parameters of other types in the future. + +## ABI compatibility + +This proposal does not affect the ABI of existing code. Handling integer +generic parameters in full generality requires new functionality in the +Swift runtime to be able to encode and interpret them as part of type +metadata. + +As with generic parameters in general, adding or removing +integer generic parameters, replacing value parameters of a function with +integer generic parameters, reordering an integer generic parameter relative to +other generic parameters (whether value or type), and adding or removing +same-value constraints are all ABI-breaking changes. + +## Implications on adoption + +### Back-deployment limitations + +On platforms where the vendor ships the Swift runtime with the operating +system, there may be limitations on using integer generic parameters in +programs that want to target earlier versions of those platforms that don't +have the necessary runtime support. + +### Naming conventions + +Integer generic parameters are a new kind of declaration in Swift, and +conventions need to be established as to how they should be named. This +proposal recommends that integer generic parameters follow the convention +of other value bindings and be named using `lowerCamelCase` identifiers. + +## Future directions + +This proposal aims to establish the core functionality of integer generic +parameters. There are many possible improvements that could be built upon +this base: + +### Fixed-size and fixed-capacity collection types + +This proposal provides a foundational mechanism for fixed-size array and +fixed-capacity collection types, but does not itself introduce any +new standard library types or mechanisms for defining those types. We leave +it to future proposals to explore the design of those types. + +### Use of constant bindings as generic parameters + +It would be very useful to be able to use constant bindings as generic +parameter bindings, in addition to literals and existing generic parameter +bindings: + +```swift +static let bufferSize + = MemoryLayout.size * 64 + MemoryLayout.size * 8 + +var buffer = Vector(...) +``` + +This should be possible as long as the bindings referenced are known to be +constant (like `let` bindings are). + +A likely-fundamental limitation to this feature, as well as related +constant evaluation features explored below, is that the type checker will +likely be unable to reason about the value of these bindings. +Type checking influences overload resolution and the overall meaning of +expressions, so cannot rely on the evaluation of those expressions without +creating circularities. We may be able understand that two terms spelled +exactly the same way are equivalent, but we wouldn't recognize two different +expressions with the same result are the same type statically: + +``` +let fourShorts = 4 * MemoryLayout.size +let eightBytes = 8 * MemoryLayout.size + +var v1: Vector = [...] +var v2: Vector = [...] +v1 = v2 // Error, different types +``` + +This is similar to how opaque result types for different declarations are +type-checked as if they are potentially different types even when their +underlying types dynamically resolve to the same type. + +### Arithmetic in generic parameters + +There are many operations that would benefit from being able to express basic +arithmetic relationships among values. For instance, the concatenation of two +fixed-sized arrays would give an array whose length is the sum of the input +lengths: + +```swift +func concat( + _ a: Vector, _ b: Vector +) -> Vector +``` + +Due to the bidirectional nature of Swift's type-checking, there would be +limits to the sorts of relations we would be able to express this way. + +### Relating integer generic parameters and variadic pack shapes + +The "shape" of a parameter pack ultimately compiles down to its length. +Variadic packs don't currently have a way to directly reference or constrain +their shape or length, and integer generic parameters might be one way of doing +so. Among other things, this might allow for a variadic API to express that it +takes as many arguments as one of its integer generic parameters indicates: + +```swift +struct Vector { + // the initializer for a Vector takes one argument + // for every element + init(_ values: repeat each n * T) +} +``` + +### Non-integer value generic parameters + +We may want to eventually allow generic declarations to have value parameters +of type other than `Int`. The proposal's `let Parameter: Type` declaration +syntax maintains space for this: + +```swift +struct MatrixShape { var rows: Int, columns: Int } + +struct Matrix { + var elements: Vector> +} +``` + +Although the syntactic extension is straightforward, there are a lot of +questions to answer about how type equality is determined when values of +arbitrary type are involved, and what sorts of construction and destructuring +operations can be supported at type level. There is some precedent in +other languages to look at here, particularly C++'s non-type template +parameters or Rust's similar const generics feature. However, in relation to +those other languages, Swift puts a bit stronger emphasis on being able to +abstract the layout of types, but the type-level equality of parameters would +be heavily dependent on their types' layout and how initialization and property +access works. + +### Integer parameter packs + +There are use cases for variadic packs of integer generic parameters. +For instance, it might be a way of representing arbitrary multidimensional +matrices of values: + +```swift +struct MDMatrix { ... } + +let mat2d: MDMatrix<4, 4> = ... +let mat4d: MDMatrix<120, 24, 6, 2> = ... +``` + +## Alternatives considered + +### Variable-sized types instead of integer generic parameters + +One of the primary motivators for integer generic parameters is to represent +fixed-size and fixed-capacity collections. One of the reasons this is necessary +is because every value of a Swift type has to have a uniform size; since +a four-element array has a different size from a five-element array, that +implies that they have to be different types `Vector<4, T>` and `Vector<5, T>`. + +However, one could argue that the fundamental type of such a container +doesn't really change with its size; in most cases, a function that can +accept an array of some size can just as well accept an array of any size. +Forcing a type distinction between different-sized arrays forces the majority +of APIs that want to work with arrays to either be generic over their size, +be generic over some more abstract protocol like `Collection` that all +sized arrays conform to (along with unsized `Array` and non-array collections), +or work with the arrays indirectly through some handle type like +`UnsafeBufferPointer` or `Span`. + +So it's interesting to consider an alternative design where we instead +remove the "all values of a type have the same size" constraint. One could +say that the owner of a `Vector` value has to give it some size, but then +a `borrowing` or `inout Vector` can reference a `Vector` of any size, since +the reference representation would carry that size information from the +owner. There are however a lot of open questions following this design +path—if you want to have a two-dimensional `Vector` of `Vector`s, how do you +track the size information of both levels of nesting? There also *are* +functions that want to require taking two input arrays of the same size, +or promise to return an array as the same size as an argument. These +relationships are straightforward to express through the generics system, +and if sizes aren't propagated through types but some other means, it seems +likely we would need a parallel mechanism for reasoning about sizes generically. +Variable-sized types are an interesting idea to explore, but it isn't clear +that they lead to an overall simpler language design. + +### Declaring value parameters without `let` + +One could argue that, since `Int` clearly isn't a protocol constraint, that +it should be sufficient to declare integer generic parameters with the +syntax `` without an introducer like `let`. There are at least +a couple of reasons we choose to adopt the `let` introducer: + +- It makes it clear to the reader (and the compiler) what parameters are + value parameters without needing to do name resolution first. This may not + be a huge deal for `Int`, but if we expand the feature to allow other + types of value generic parameters, then it may not be obvious in an + unfamiliar codebase whether `T: Foo` refers to a protocol constraint + `Foo` or a concrete type `Foo`. +- If we do generalize value generic parameters to allow other types in the + future, it's not entirely out of the question that that could include + existential types, which would make `T: P` potentially ambiguous as to + whether it declares a type parameter constrained to `T` or a value + parameter of type `any P`. (There are perhaps other ways of dealing with that + ambiguity, such as requiring the value parameter form to be written + explicitly with `t: any P`.) + +### Arbitrary-precision integer generic parameters + +Instead of treating integer generic parameters as values of `Int` or any +finite type, another possible design would be to treat type-level integers +as independent of any concrete type, leaving them as ideal arbitrary-precision +integers. This would have some semantic advantages if we want to allow for +type-level arithmetic relationships, since these operations could be defined +in their ideal form without having to deal with overflow and other limitations +of concrete Swift types. In such a design, a reference to an integer generic +parameter in a value expression could be treated as polymorphic, in a similar +way to how integer literals can be used with any type that's +`ExpressibleByIntegerLiteral`. + +Although this model has some appeal, it also has some practical issues. If +type-level integers are arbitrary precision, but value-level integer types are +still finite, then there is the chance for overflow any time a type-level +integer is reified to a finite integer type. This model also would not extend +very naturally to non-integer value parameters if we introduce those in the +future. + +### Generic parameters of integer types other than `Int` + +We discuss generalizing value generic parameters to types other than `Int` +as a future direction above, but a narrower expansion might be to allow all +of Swift's primitive integer types, including all of the sized and +signed/unsigned variants, as types for generic value parameters. One could +argue that `UInt` is particular is desirable to use as the type for fixed-size +and fixed-capacity collections, of which instances can never actually be +constructed for negative sizes. + +However, we would like to continue to promote the use of `Int` as the common +currency type for integers, as we have already established for the standard +library. Introducing mixed integer types as type-level generic parameters +would inevitably lead to the need to be able to perform type conversions +at type level, and the associated need to deal with overflow during these +type-level conversions. + +The established API for the `Collection` protocol, `Array`, and the other +standard library types already use `Int` for `count` and array subscripting +operations, so establishing `Int` as the type for type-level size parameters +avoids the need for type conversions when mixing type- and value-level index +and size values. Types that use integer parameters for sizing can still +refuse to initialize values of types with negative parameters, so that a +type like `Vector<-1, Int>` is uninhabited. Given the restrictions in this +initial proposal, without type-level arithmetic, it is unlikely that +developers would intentionally form such a type with a negative size explicitly. + +### Alternative naming conventions + +We propose recommending a `lowerCamelCase` naming convention for integer +generic parameters, following the recommended convention for other value +declarations such as property and function declarations. This allows for +integer generic parameters to appear consistent with other value references +in expressions. If we add the ability to instantiate generics using value +constants or expressions as integer generic arguments in the future, this will +also maintain consistency between existing generic parameters and other value +declarations used as arguments: + +``` +struct Foo { + static let nSquared = n * n + + // These both consistently use lowerCamelCase + var vector: Vector + var matrix: Vector +} +``` + +Alternatively, we could recommend that integer generic parameters use +`UpperCamelCase`, following the convention of type generic parameters. We +believe that the distinction between values and types is better to prioritize +than the distinction between type-level and value-level parameters. + +### Syntactic separation of value and type parameters + +The angle brackets that enclose generic arguments have to be syntactically +disambiguated from the `<` and `>` operators. This disambiguation relies on +parsing ahead to determine whether the source code following a `<` is parsable +as a list of types followed by a closing `>` and a member access or initializer +call. This lookahead rule has worked well up to this point, but it could impose +constraints on our ability to allow for expressions to be used as generic +arguments, since allowing more expression productions to appear in generic +argument lists will lead to more potentially ambiguous parsing situations. + +It may be worth considering a design that separates value generic parameters +by putting them in a different set of brackets separate from the type +generic parameters, like `Vector[count]`. This would avoid the need +for disambiguation if an arbitrary expression can be used as a `count` in +the future. + +### Behavior of static properties corresponding to generic parameters + +In response to the first round of review, we added static properties +corresponding to the integer generic parameters of types, based on feedback +that, in many if not most cases, the values of the generic parameters would end +up being redeclared as static parameters, and the existence and names of the +generic parameters are already essentially part of its public API, since +usage of the type must be able to provide arguments to the parameters, and +extensions can refer to the parameters by their names. + +One good argument against doing this is that we don't already do anything +analogous for type generic parameters, such as presenting them as a typealias +member of the type. We think that it would be beneficial to do so, however, +and in discussion with the LSG, there were attempts to make this happen in +the past, but they ran into source compatibility issues. If there is a chance +to take a source break that enables this functionality for type generic +parameters, the LSG thinks it is worth considering. With integer generic +parameters, we have the opportunity to do the right thing out of the gate, +and we would have similar source-breaking considerations to do it later, so +we believe it is better to do the right thing up front. + +## Acknowledgments + +We would like to thank the following people for prototyping and design +contributions that helped shape this proposal: + +- Holly Borla +- Ben Cohen +- Erik Eckstein +- Doug Gregor +- Tim Kientzle +- Karoy Lorentey +- John McCall +- Kuba Mracek +- Slava Pestov +- Andrew Trick +- Pavel Yaskevich diff --git a/proposals/0453-vector.md b/proposals/0453-vector.md new file mode 100644 index 0000000000..d755bf66f1 --- /dev/null +++ b/proposals/0453-vector.md @@ -0,0 +1,774 @@ +# InlineArray, a fixed-size array + +* Proposal: [SE-0453](0453-vector.md) +* Authors: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [Approaches for fixed-size arrays](https://forums.swift.org/t/approaches-for-fixed-size-arrays/58894) +* Implementation: [swiftlang/swift#76438](https://github.com/swiftlang/swift/pull/76438) +* Review: ([pitch](https://forums.swift.org/t/vector-a-fixed-size-array/75264)) ([first review](https://forums.swift.org/t/se-0453-vector-a-fixed-size-array/76004)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0453-vector-a-fixed-size-array/76411)) ([second review](https://forums.swift.org/t/second-review-se-0453-vector-a-fixed-size-array/76412)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0453-inlinearray-formerly-vector-a-fixed-size-array/77678)) + +## Introduction + +This proposal introduces a new type to the standard library, `InlineArray`, which is +a fixed-size array. This is analogous to the +[classical C arrays `T[N]`](https://en.cppreference.com/w/c/language/array), +[C++'s `std::array`](https://en.cppreference.com/w/cpp/container/array), +and [Rust's arrays `[T; N]`](https://doc.rust-lang.org/std/primitive.array.html). + +## Motivation + +Arrays in Swift have served as the go to choice when needing to put items in an +ordered list. They are a great data structure ranging from a variety of +different use cases from teaching new developers all the way up to sophisticated +implementation details of something like a cache. + +However, using `Array` all the time doesn't really make sense in some scenarios. +It's important to understand that `Array` is a heap allocated growable data +structure which can be expensive and unnecessary in some situations. The next +best thing is to force a known quantity of elements onto the stack, probably by +using tuples. + +```swift +func complexAlgorithm() { + let elements = (first, second, third, fourth) +} +``` + +Unfortunately, using tuples in this way is very limited. They don't allow for +dynamic indexing or iteration: + +```swift +func complexAlgorithm() { + let elements = (first, second, third, fourth) + + // Have to manually know the tuple has N elements... + for i in 0 ..< 4 { + // error: cannot access element using subscript for tuple type + // '(Int, Int, Int, Int)'; use '.' notation instead + compute(elements[i]) + } +} +``` + +It wasn't until [SE-0322 Temporary uninitialized buffers](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0322-temporary-buffers.md), which proposed the `withUnsafeTemporaryAllocation` +facilities, that made this situation a little easier to work with by giving us a +direct `UnsafeMutableBufferPointer` pointing either somewhere on the stack or to +a heap allocation. This API allows us to get the indexing and iteration we want, +but it drops down to an unsafe layer which is unfortunate because there should +be much safer ways to achieve the same while not exposing unsafety to +developers. + +While we aren't getting rid of `Array` anytime soon, more and more folks are +looking towards Swift to build safer and performant code and having `Array` be +our only solution to an ordered list of things is less than ideal. `Array` is a +very general purpose array collection that can suit almost any need, but it is +always heap allocated, automatically resizable, and introduces retain/release +traffic. These implicit allocations are becoming more and more of a bottleneck, +especially in embedded domains where there might not be a lot of memory for many +allocations or even heap allocations at all. Swift should be able to provide +developers a safe API to have an ordered list of homogeneous items on the stack, +allowing for things like indexing, iteration, and many other collection +utilities. + +## Proposed solution + +We introduce a new top level type, `InlineArray`, to the standard library which is a +fixed-size contiguously inline allocated array. We're defining "inline" as using +the most natural allocation pattern depending on the context of where this is +used. It will be stack allocated most of the time, but as a class property +member it will be inline allocated on the heap with the rest of the properties. +`InlineArray` will never introduce an implicit heap allocation just for its storage +alone. + +```swift +func complexAlgorithm() { + // This is a stack allocation, no 'malloc's or reference counting here! + let elements: InlineArray<4, Int> = [1, 2, 3, 4] + + for i in elements.indices { + compute(elements[i]) // OK + } +} +``` + +InlineArrays of noncopyable values will be possible by using any of the closure based +taking initializers or the literal initializer: + +```swift +// [Atomic(0), Atomic(1), Atomic(2), Atomic(3)] +let incrementingAtomics = InlineArray<4, Atomic> { i in + Atomic(i) +} + +// [Sprite(), Sprite(), Sprite(), Sprite()] +// Where the 2nd, 3rd, and 4th elements are all copies of their previous +// element. +let copiedSprites = InlineArray<4, _>(first: Sprite()) { $0.copy() } + +// Inferred to be InlineArray<3, Mutex> +let literalMutexes: InlineArray = [Mutex(0), Mutex(1), Mutex(2)] +``` + +These closure based initializers are not limited to noncopyable values however! + +## Detailed design + +`InlineArray` will be a simple noncopyable struct capable of storing other potentially +noncopyable elements. It will be conditionally copyable only when its elements +are. + +```swift +public struct InlineArray: ~Copyable {} + +extension InlineArray: Copyable where Element: Copyable {} +extension InlineArray: BitwiseCopyable where Element: BitwiseCopyable {} +extension InlineArray: Sendable where Element: Sendable {} +``` + +### MemoryLayout + +The memory layout of a `InlineArray` is defined by taking its `Element`'s stride and +multiplying that by its `count` for its size and stride. Its alignment is equal +to that of its `Element`: + +```swift +MemoryLayout.stride == 1 +MemoryLayout.alignment == 1 + +MemoryLayout>.size == 4 +MemoryLayout>.stride == 4 +MemoryLayout>.alignment == 1 + +struct Uneven { + let x: UInt32 + let y: Bool +} + +MemoryLayout.stride == 8 +MemoryLayout.alignment == 4 + +MemoryLayout>.size == 32 +MemoryLayout>.stride == 32 +MemoryLayout>.alignment == 4 + +struct ACoupleOfUInt8s { + let x: InlineArray<2, UInt8> +} + +MemoryLayout.stride == 2 +MemoryLayout.alignment == 1 + +MemoryLayout>.size == 4 +MemoryLayout>.stride == 4 +MemoryLayout>.alignment == 1 +``` + +### Literal Initialization + +Before discussing any of the API, we need to discuss how the array literal +syntax will be used to initialize a value of `InlineArray`. While naively we could +conform to `ExpressibleByArrayLiteral`, the shape of the initializer always +takes an actual `Array` value. This could be optimized away in the simple cases, +but fundamentally it doesn't make sense to have to do an array allocation to +initialize a stack allocated `InlineArray`. Therefore, the array literal +initialization for `InlineArray` will be a special case, at least to start out with. +A stack allocated InlineArray using a InlineArray literal will do in place initialization +of each element at its stack slot. The two below are roughly equivalent: + +```swift +let numbers: InlineArray<3, Int> = [1, 2, 3] + +// Roughly gets compiled as: + +// This is not a real 'InlineArray' initializer! +let numbers: InlineArray<3, Int> = InlineArray() +numbers[0] = 1 +numbers[1] = 2 +numbers[2] = 3 +``` + +There shouldn't be any intermediary values being copied or moved into the InlineArray. + +Note that the array literal syntax will only create a `InlineArray` value when the +compiler knows concretely that it is a `InlineArray` value. We don't want to break +source whatsoever, so whatever current rules the compiler has will still be +intact. Consider the following uses of the array literal syntax and where each +call site creates either a `Swift.Array` or a `Swift.InlineArray`. + +```swift +let a = [1, 2, 3] // Swift.Array +let b: InlineArray<3, Int> = [1, 2, 3] // Swift.InlineArray + +func generic(_: T) {} + +generic([1, 2, 3]) // passes a Swift.Array +generic([1, 2, 3] as InlineArray<3, Int>) // passes a Swift.InlineArray + +func test(_: T) {} + +test([1, 2, 3]) // passes a Swift.Array +test([1, 2, 3] as InlineArray<3, Int>) // error: 'InlineArray<3, Int>' does not conform to 'ExpressibleByArrayLiteral' + +func array(_: [T]) {} + +array([1, 2, 3]) // passes a Swift.Array +array([1, 2, 3] as InlineArray<3, Int>) // error: 'InlineArray<3, Int>' is not convertible to 'Array' + +func inlineArray(_: InlineArray<3, T>) {} + +inlineArray([1, 2, 3]) // passes a Swift.InlineArray +inlineArray([1, 2, 3] as [Int]) // error: 'Array' is not convertible to 'InlineArray<3, Int>' +``` + +I discuss later about a hypothetical `ExpressibleByInlineArrayLiteral` and the design +challenges there in [Future Directions](#expressiblebyInlineArrayliteral). + +The literal initialization allows for more type inference just like the current +literal syntax does by inferring not only the element type, but also the count +as well: + +```swift +let a: InlineArray<_, Int> = [1, 2, 3] // InlineArray<3, Int> +let b: InlineArray<3, _> = [1, 2, 3] // InlineArray<3, Int> +let c: InlineArray<_, _> = [1, 2, 3] // InlineArray<3, Int> +let d: InlineArray = [1, 2, 3] // InlineArray<3, Int> + +func takesGenericInlineArray(_: InlineArray) {} + +takesGenericInlineArray([1, 2, 3]) // Ok, N is inferred to be '3'. +``` + +A compiler diagnostic will occur if the number of elements within the literal +do not match the desired count (as well as element with the usual diagnostic): + +```swift +// error: expected '2' elements in InlineArray literal, but got '3' +let x: InlineArray<2, Int> = [1, 2, 3] + +func takesInlineArray(_: InlineArray<2, Int>) {} + +// error: expected '2' elements in InlineArray literal, but got '3' +takesInlineArray([1, 2, 3]) +``` + +### Initialization + +In addition to literal initialization, `InlineArray` offers a few others forms of +initialization: + +```swift +extension InlineArray where Element: ~Copyable { + /// Initializes every element in this InlineArray running the given closure value + /// that returns the element to emplace at the given index. + /// + /// This will call the closure `count` times, where `count` is the static + /// count of the InlineArray, to initialize every element by passing the closure + /// the index of the current element being initialized. The closure is allowed + /// to throw an error at any point during initialization at which point the + /// InlineArray will stop initialization, deinitialize every currently initialized + /// element, and throw the given error back out to the caller. + /// + /// - Parameter next: A closure that returns an owned `Element` to emplace at + /// the passed in index. + public init(_ next: (Int) throws(E) -> Element) throws(E) + + /// Initializes every element in this InlineArray by running the closure with the + /// previously initialized element. + /// + /// This will call the closure `count - 1` times, where `count` is the static + /// count of the InlineArray, to initialize every element by passing the closure + /// an immutable borrow reference to the previously initialized element. The + /// closure is allowed to throw an error at any point during initialization at + /// which point the InlineArray will stop initialization, deinitialize every + /// currently initialized element, and throw the given error back out to the + /// caller. + /// + /// - Parameter first: The first value to insert into the InlineArray which will be + /// passed to the closure as a borrow. + /// - Parameter next: A closure that passes in an immutable borrow reference + /// of the previously initialized element of the InlineArray + /// which returns an owned `Element` instance to insert into + /// the InlineArray. + public init( + first: consuming Element, + next: (borrowing Element) throws(E) -> Element + ) throws(E) +} + +extension InlineArray where Element: Copyable { + /// Initializes every element in this InlineArray to a copy of the given value. + /// + /// - Parameter value: The instance to initialize this InlineArray with. + public init(repeating: Element) +} +``` + +### Deinitialization and consumption + +Once a InlineArray is no longer used, the compiler will implicitly destroy its value. +This means that it will do an element by element deinitialization, releasing any +class references or calling any `deinit`s on noncopyable elements. + +### Generalized `Sequence` and `Collection` APIs + +While we aren't conforming `InlineArray` to `Collection` (more information in future +directions), we do want to generalize a lot of APIs that will make this a usable +collection type. + +```swift +extension InlineArray where Element: ~Copyable { + public typealias Element = Element + public typealias Index = Int + + /// Provides the count of the collection statically without an instance. + public static var count: Int { count } + + public var count: Int { count } + public var indices: Range { 0 ..< count } + public var isEmpty: Bool { count == 0 } + public var startIndex: Int { 0 } + public var endIndex: Int { count } + + public borrowing func index(after i: Int) -> Int + public borrowing func index(before i: Int) -> Int + + public mutating func swapAt( + _ i: Int, + _ j: Int + ) + + public subscript(_ index: Int) -> Element + public subscript(unchecked index: Int) -> Element +} +``` + +## Source compatibility + +`InlineArray` is a brand new type in the standard library, so source should still be +compatible. + +Given the name of this type however, we foresee this clashing with existing user +defined types named `InlineArray`. This isn't a particular issue though because the +standard library has special shadowing rules which prefer user defined types by +default. Which means in user code with a custom `InlineArray` type, that type will +always be preferred over the standard library's `Swift.InlineArray`. By always I +truly mean _always_. + +Given the following two scenarios: + +```swift +// MyLib +public struct InlineArray { + +} + +print(InlineArray.self) + +// error: generic type 'InlineArray' specialized with too many type parameters +// (got 2, but expected 1) +print(InlineArray<3, Int>.self) +``` + +Here, we're exercising the fact that this `MyLib.InlineArray` has a different generic +signature than `Swift.InlineArray`, but regardless of that we will prefer `MyLib`'s +version even if we supply more generic arguments than it supports. + +```swift +// MyLib +public struct InlineArray { + +} + +// MyExecutable main.swift +import MyLib + +print(InlineArray.self) // OK + +// error: generic type 'InlineArray' specialized with too many type parameters +// (got 2, but expected 1) +print(InlineArray<3, Int>.self) + +// MyExecutable test.swift + +// error: generic type 'InlineArray' specialized with too few type parameters +// (got 1, but expected 2) +print(InlineArray.self) +``` + +And here, we exercise that a module with its own `InlineArray`, like `MyLib`, will +always prefer its own definition within the module, but even for dependents +who import `MyLib` it will prefer `MyLib.InlineArray`. For files that don't +explicitly `MyLib`, it will prefer `Swift.InlineArray`. + +## ABI compatibility + +`InlineArray` is a brand new type in the standard library, so ABI should still be +compatible. + +## Implications on adoption + +This is a brand new type which means there will be deployment version +requirement to be able to use this type, especially considering it is using new +runtime features from integer generics. + +## Future directions + +### `Equatable`, `Hashable`, `CustomStringConvertible`, and other protocols. + +There are a wide class of protocols that this type has the ability to conform to, +but the issue is that it can only conform to them when the element conforms to +them (this is untrue for `CustomStringConvertible`, but it still requires +copyability). We could introduce these conformances but have them be conditional +right now and generalize it later when we generalize these protocols, but if we +were to ship say Swift X.Y with: + +```swift +@available(SwiftStdlib X.Y) +extension InlineArray: Equatable where Element: Equatable // & Element: Copyable +``` + +and later down the road in Swift X.(Y + 1): + +```swift +@available(SwiftStdlib X.Y) +extension InlineArray: Equatable where Element: ~Copyable & Equatable +``` + +Suddenly, this availability isn't quite right because the conformance that +shipped in Swift X.Y doesn't support noncopyable elements. To prevent the +headache of this and any potential new availability feature, we're holding off on +these conformances until they are fully generalized. + +### `Sequence` and `Collection` + +Similarly, we aren't conforming to `Sequence` or `Collection` either. +While we could conform to these protocols when the element is copyable, `InlineArray` +is unlike `Array` in that there are no copy-on-write semantics; it is eagerly +copied. Conforming to these protocols would potentially open doors to lots of +implicit copies of the underlying InlineArray instance which could be problematic +given the prevalence of generic collection algorithms and slicing behavior. To +avoid this potential performance pitfall, we're explicitly not opting into +conforming this type to `Sequence` or `Collection`. + +We do plan to propose new protocols that look like `Sequence` and `Collection` +that avoid implicit copying making them suitable for types like `InlineArray` and +containers of noncopyable elements. +[SE-0437 Noncopyable Standard Library Primitives](0437-noncopyable-stdlib-primitives.md) +goes into more depth about this rationale and mentions that creating new +protocols to support noncopyable containers with potentially noncopyable +elements are all marked as future work. + +Much of the `Collection` API that we are generalizing here for this type are all +API we feel confident will be included in any future container protocol. Even if +we find that to not be the case, they are still useful API outside of generic +collection contexts in their own right. + +Remember, one can still iterate a `InlineArray` instance with the usual `indices` +property (which is what noncopyable InlineArray instances would have had to deal with +regardless until new container protocols have been proposed): + +```swift +let atomicInts: InlineArray<3, Atomic> = [Atomic(1), Atomic(2), Atomic(3)] + +for i in atomicInts.indices { + print(atomicInts[i].load(ordering: .relaxed)) +} +``` + +### `Span` APIs + +With the recent proposal +[SE-0447 Span: Safe Access to Contiguous Storage](0447-span-access-shared-contiguous-storage.md) +who defines a safe abstraction over viewing contiguous storage, it would make +sense to define API on `InlineArray` to be able to get one of these `Span`s. However, +the proposal states that: + +> We could provide `withSpan()` and `withBytes()` closure-taking functions as +> safe replacements for the existing `withUnsafeBufferPointer()` and +> `withUnsafeBytes()` functions. We could also also provide lifetime-dependent +> `span` or `bytes` properties. +> ... +> Of these, the closure-taking functions can be implemented now, but it is +> unclear whether they are desirable. The lifetime-dependent computed properties +> require lifetime annotations, as initializers do. We are deferring proposing +> these extensions until the lifetime annotations are proposed. + +All of which is exactly true for the current `InlineArray` type. We could propose a +`withSpan` style API now, but it's unclear if that's what we truly want vs. a +computed property that returns the span which requires lifetime annotation +features. For now, we're deferring such API until a lifetime proposal is +proposed and accepted. + +### `ExpressibleByInlineArrayLiteral` + +While the proposal does propose a literal initialization for `InlineArray` that +doesn't use `ExpressibleByArrayLiteral`, we are intentionally not exposing some +`ExpressibleByInlineArrayLiteral` or similar. It's unclear what this protocol would +look like because each design has a different semantic guarantee: + +```swift +public protocol ExpressibleByInlineArrayLiteral: ~Copyable { + associatedtype Element: ~Copyable + + init(InlineArrayLiteral: consuming InlineArray) +} +``` + +This naive approach would satisfy a lot of types like `Array`, `Set`, +some hypothetical future noncopyable array, etc. These types actually want a +generic count and can allocate just enough space to hold all of those elements. + +However, this shape doesn't quite work for `InlineArray` itself because initializing +a `InlineArray<4, Int>` should require that the literal has exactly 4 elements. Note +that we wouldn't be able to impose a new constraint just for the conformer, so +`InlineArray` couldn't require that `N == count` and still have this witness the +requirement. Similarly, a `Pair` type could be InlineArray initialized, but only if +the InlineArray has exactly 2 elements. If we had the ability to define +`associatedvalue`, then this makes the conformance pretty trivial for both of +these types: + +```swift +public protocol ExpressibleByInlineArrayLiteral: ~Copyable { + associatedtype Element: ~Copyable + associatedvalue count: Int + + init(InlineArrayLiteral: consuming InlineArray) +} + +extension InlineArray: ExpressibleByInlineArrayLiteral { + init(InlineArrayLiteral: consuming InlineArray) { ... } +} + +extension Pair: ExpressibleByInlineArrayLiteral { + init(InlineArrayLiteral: consuming InlineArray<2, Element>) { ... } +} +``` + +But even with this design it's unsuitable for `Array` itself because it doesn't +want a static count for the literal, it still wants it to be generic. + +It would be nice to define something like this either on top of `InlineArray`, +parameter packs, or something else that would let us define statically the +number of elements we need for literal initialization or be dynamic if we opt to. + +### `FixedCapacityArray` and `SmallArray` + +In the same vein as this type, it may make sense to introduce some `FixedCapacityArray` +type which would support appending and removing elements given a fixed-capacity. + +```swift +var numbers: FixedCapacityArray<4, Int> = [1, 2] +print(numbers.capacity) // 4 +print(numbers.count) // 2 +numbers.append(3) +print(numbers.count) // 3 +numbers.append(4) +print(numbers.count) // 4 +numbers.append(5) // error: not enough space +``` + +This type is significantly different than the type we're proposing because +`InlineArray` defines a fixed-size meaning you cannot append or remove from it, but +it also requires that every single element is initialized. There must never be +an uninitialized element within a `InlineArray`, however for `FixedCapacityArray` this is +not true. It would act as a regular array with an initialized prefix and an +uninitialized suffix, it would be inline allocated (stack allocated for locals, +heap allocated if it's a class member, etc.), and it would not be growable. + +The difficulty in proposing such a type right now is that we have no way of +informing the compiler what parts of `FixedCapacityArray` are initialized and what +parts are not. This is critical for copy operations, move operations, and +destroy operations. Assuming that an uninitialized element is initialized and +attempting to perform any of these operations on it may lead to runtime crashes +which is definitely undesirable. + +Once we have `FixedCapacityArray` and some hypothetical noncopyable heap allocated +array type (which [SE-0437 Noncopyable Standard Library Primitives](0437-noncopyable-stdlib-primitives.md) +dons as `HypoArray` as a placeholder), it should be very trivial to define a +`SmallArray` type similar to the one found in LLVM APIs `llvm::SmallVector`. + +```swift +public enum SmallArray: ~Copyable { + case small(FixedCapacityArray) + case large(HypoArray) +} +``` + +which would act as an inline allocated array until one out grew the inline +capacity and would fall back to a dynamic heap allocation. + +### Syntax sugar + +We feel that this type will become as fundamental as `Array` and `Dictionary` +both of which have syntactic sugar for declaring a type of them, `[T]` for +`Array` and `[K: V]` for `Dictionary`. It may make sense to define something +similar for `InlineArray`, however we leave that as a future direction as the +spelling for such syntax is not critical to landing this type. + +It should be fairly trivial to propose such a syntax in the future either via a +new proposal, or as an amendment to this one. Such a change should only require +a newer compiler that supports the syntax and nothing more. + +Some syntax suggestions: + +* `[N x T]` or `[T x N]` +* `[N * T]` or `[T * N]` +* `T[N]` (from C) +* `[T; N]` (from Rust) + +Note that it may make more sense to have the length appear before the type. I +discuss this more in depth in [Reorder the generic arguments](#reorder-the-generic-arguments-InlineArrayt-n-instead-of-InlineArrayn-t). + +### C Interop changes + +With the introduction of `InlineArray`, we have a unique opportunity to fix another +pain point within the language with regards to C interop. Currently, the Swift +compiler imports a C array of type `T[24]` as a tuple of `T` with 24 elements. +Previously, this was really the only representation that the compiler could pick +to allow interfacing with C arrays. It was a real challenge working with these +fields from C in Swift. Consider the following C struct: + +```c +struct section_64 { + char sectname[16]; + char segname[16]; + uint64_t addr; + uint64_t size; + uint32_t offset; + uint32_t align; + ... +}; +``` + +Today, this gets imported as the following Swift struct: + +```swift +struct section_64 { + let sectname: (CChar, CChar, CChar, CChar, CChar, CChar, ... 10 more times) + let segname: (CChar, CChar, CChar, CChar, CChar, CChar, ... 10 more times) + let addr: UInt64 + let size: UInt64 + let offset: UInt32 + let align: UInt32 + ... +} +``` + +Using an instance of `section_64` in Swift for the most part is really easy. +Accessing things like `addr` or `size` are simple and easy to use, but using the +`sectname` property introduces a level of complexity that isn't so fun to use. + +```swift +func getSectionName(_ section: section_64) -> String { + withUnsafePointer(to: section.sectname) { + // This is unsafe! 'sectname' isn't guaranteed to have a null byte + // indicating the end of the C string! + String(cString: $0) + } +} + +func iterateSectionNameBytes(_ section: section_64) { + withUnsafeBytes(to: section.sectname) { + for byte in $0 { + ... + } + } +} +``` + +Having to resort to using very unsafe API to do anything useful with imported C +arrays is not something a memory safe language like Swift should be in the +business of. + +Ideally we could migrate the importer from using tuples to this new `InlineArray` +type, however that would be massively source breaking. A previous revision of +this proposal proposed an _upcoming_ feature flag that modules can opt into, +but this poses issues with the current importer implementation with regards to +inlinable code. + +Another idea was to import struct fields with C array types twice, one with the +existing name with a tuple type (as to not break source) and another with some +`InlineArray` suffix in the name with the `InlineArray` type. This works pretty well for +struct fields and globals, but it leaves fields and functions who have pointers +to C arrays in question as well (spelt `char (*x)[4]`). Do we import such +functions twice using a similar method of giving it a different name? Such a +solution would also incur a longer deprecation period to eventually having just +`InlineArray` be imported and no more tuples. + +We're holding off on any C interop changes here as there are still lots of open +questions as to what the best path forward is. + +## Alternatives considered + +### Reorder the generic arguments (`InlineArray` instead of `InlineArray`) + +If we directly followed existing APIs from C++, then obviously the length should +follow the element type. However we realized that when reading this type aloud, +it's "a InlineArray of 3 integers" for example instead of "a InlineArray of integers of +size 3". It gets more interesting the more dimensions you add. +Consider an MxN matrix. In C, you'd write this as `T[N][M]` but index it as +`[m][n]`. We don't want to introduce that sort of confusion (which is a good +argument against `T[N]` as a potential syntax sugar for this type), so the +length being before the underlying element makes the most sense at least for any +potential sugared form. `[M * [N * T]]` would be indexed directly as it is spelt +out in the sugared form, `[m][n]`. In light of that, we wouldn't want the sugar +form to have a different ordering than the generic type itself leading us to +believe that the length must be before the element type. + +## Revisions + +Previously, this type was named `Vector`, but has since been renamed to `InlineArray`. + +### A name other than `Vector` + +For obvious reasons, we cannot name this type `Swift.Array` to match the +"term of art" that other languages like C, C++, and Rust are using for this +exact type. However, while this name is the de facto for other languages, it +actually mischaracterizes the properties and behaviors of this type considering +existing terminology in mathematics. A. Stepanov mentions in his book, "From +Mathematics to Generic Programming", that using the name `std::vector` for their +dynamically allocated growable array type was perhaps a mistake for this same +reason: + +> If we are coming up with a name for something, or overloading an existing name, +> we should follow these three guidelines: +> +> 1. If there is an established term, use it. +> 2. Do not use an established term inconsistently with its accepted meaning. In +> particular, overload an operator or function name only when you will be +> preserving its existing semantics. +> 3. If there are conflicting usages, the much more established one wins. +> +> The name _vector_ in STL was taken from the earlier programming languages +> Scheme and Common Lisp. Unfortunately, this was inconsistent with the much +> older meaning of the term in mathematics and violates Rule 3; this data structure +> should have been called _array_. Sadly, if you make a mistake and violate these +> principles, the result might stay around for a long time. +> +> \- Stepanov, A. A., Rose, D. E. (2014). _From Mathematics to Generic Programming_. United Kingdom: Addison-Wesley. + +Indeed, the `std::vector` type goes against the definition of vector by being a +growable container, having a non-fixed magnitude. + +We fully acknowledge that the Swift types, `Swift.Array` and `Swift.Vector`, are +complete opposites of the C++ ones, `std::vector` and `std::array`. While it may +be confusing at first, ultimately we feel that our names are more in line with +the mathematical term of art. + +If there was any type we could add to the standard library whose name could be +`Vector`, it must be this one. + +## Acknowledgments + +I would like the thank the following people for helping in the design process +of this type: + +* Karoy Lorentey +* Guillaume Lessard +* Joe Groff +* Kuba Mracek +* Andrew Trick +* Erik Eckstein +* Philippe Hausler +* Tim Kientzle diff --git a/proposals/0454-memory-allocator.md b/proposals/0454-memory-allocator.md new file mode 100644 index 0000000000..c7c6f0d755 --- /dev/null +++ b/proposals/0454-memory-allocator.md @@ -0,0 +1,68 @@ +# Custom Allocator for Toolchain + +* Proposal: [SE-0454](0454-memory-allocator.md) +* Authors: [Saleem Abdulrasool](https://github.com/compnerd) +* Review Manager: [Alastair Houghton](https://github.com/al45tair) +* Status: **Accepted** +* Implementation: [swiftlang/swift#76563](https://github.com/swiftlang/swift/pull/76563) +* Review: ([review](https://forums.swift.org/t/se-454-adopt-mimalloc-for-windows-toolchain/77096)) + ([acceptance](https://forums.swift.org/t/accepted-se-0454-custom-allocator-for-toolchain-adopt-mimalloc-for-windows-toolchain/77413)) + +## Introduction + +The tools in the Swift toolchain require allocating data structures for +compiling the code. Different memory allocators have differing performance +characteristics. Changing the default memory allocator away from the default +(system) allocator can yield benefits if the allocator is better tuned to the +allocation patterns of the compiler. + +## Motivation + +A more effecient memory allocator would improve the performance of the compiler +on Windows. This allows better developer productivity by reducing compile time. + +## Proposed solution + +We propose to adopt mimalloc as the memory allocator for the Swift toolchain on +Windows. + +## Detailed design + +Building a test codebase yielded a 4% build time decrease when the toolchain was +built with mimalloc. + +## Source compatibility + +This proposal does not affect source compatibility. + +## ABI compatibility + +This proposal does not affect ABI of code. + +## Implications on adoption + +Additional files will need to be built, packaged, and shipped as part of the +toolchain. The mimalloc build is relatively light and the overall build time +impact is minimal. + +This change has no implications for the runtime, only the toolchain is changed. + +## Future directions + +None at this time. + +## Alternatives considered + +Alternative memory allocators were considered, including +[tcmalloc](https://github.com/google/tcmalloc) and +[tbb](https://github.com/intel/tbb). mimalloc is well supported, developed by +Microsoft, and has better characteristics comparatively. + +Leaving the allocator on the default system allocator leaves the compiler +without the performance improvements of an alternative allocator. + +## Acknowledgements + +Special thanks to @hjyamauchi for performing the work to integrate the mimalloc +build into the Windows build and collecting the performance numbers that showed +the improvement. diff --git a/proposals/0455-swiftpm-testable-build-setting.md b/proposals/0455-swiftpm-testable-build-setting.md new file mode 100644 index 0000000000..186eb31892 --- /dev/null +++ b/proposals/0455-swiftpm-testable-build-setting.md @@ -0,0 +1,64 @@ +# SwiftPM @testable build setting + +* Proposal: [SE-0455](0455-swiftpm-testable-build-setting.md) +* Authors: [Jake Petroules](https://github.com/jakepetroules) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Accepted** +* Implementation: [swiftlang/swift-package-manager#8004](https://github.com/swiftlang/swift-package-manager/pull/8004) +* Review: ([pitch](https://forums.swift.org/t/pitch-swiftpm-testable-build-setting/75084)) ([review](https://forums.swift.org/t/se-0455-swiftpm-testable-build-setting/77100)) ([acceptance](https://forums.swift.org/t/accepted-se-0455-swiftpm-testable-build-setting/77510)) + +## Introduction + +The current Swift Package Manager build system is currently hardcoded to pass the `-enable-testing` flag to the Swift compiler to enable `@testable import` when building in debug mode. + +Swift-evolution thread: [Pitch: [SwiftPM] @testable build setting](https://forums.swift.org/t/pitch-swiftpm-testable-build-setting/75084) + +## Motivation + +Not all targets in a given package make use of the `@testable import` feature (or wish to use it at all), but all targets are presently forced to build their code with this support enabled regardless of whether it's needed. + +Developers should be able to disable `@testable import` when it's not needed or desired, just as they're able to do so in Xcode's build system. + +On Windows in particular, where a shared library is limited to 65k exported symbols, disabling `@testable import` provides developers an option to significantly reduce the exported symbol count of a library by hiding all of the unnecessary internal APIs. It can also improve debug build performance as fewer symbols exported from a binary can result in faster linking. + +## Proposed solution + +Add a new Swift target setting API to specify whether testing should be enabled for the specified target, falling back to the current behavior by default. + +## Detailed design + +Add a new `enableTestableImport` API to `SwiftSetting` limited to manifests >= 6.1: + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.1) + public static func enableTestableImport( + _ enable: Bool, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting { + ... + } +} +``` + +The existing `--enable-testable-imports` / `--disable-testable-imports` command line flag to `swift-test` currently defaults to `--enable-testable-imports`. It will be changed to default to "unspecified" (respecting any target settings), and explicitly passing `--enable-testable-imports` or `--disable-testable-imports` will force all targets to enable or disable testing, respectively. + +Attempting to enable `@testable import` in release builds will result in a build warning. + +## Security + +New language version setting has no implications on security, safety or privacy. + +## Impact on existing packages + +Since this is a new API, all existing packages will use the default behavior - `@testable import` will be enabled when building for the debug configuration, and disabled when building for the release configuration. + +## Future directions + +In a future manifest version, we may want to change `@testable import` to be disabled by default when building for the debug configuration. + +## Alternatives considered + +None. diff --git a/proposals/0456-stdlib-span-properties.md b/proposals/0456-stdlib-span-properties.md new file mode 100644 index 0000000000..bffb608e00 --- /dev/null +++ b/proposals/0456-stdlib-span-properties.md @@ -0,0 +1,293 @@ +# Add `Span`-providing Properties to Standard Library Types + +* Proposal: [SE-0456](0456-stdlib-span-properties.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [BufferView Roadmap](https://forums.swift.org/t/66211) +* Implementation: [swift PR #78561](https://github.com/swiftlang/swift/pull/78561), [swift PR #80116](https://github.com/swiftlang/swift/pull/80116), [swift-foundation PR#1276](https://github.com/swiftlang/swift-foundation/pull/1276) +* Review: ([pitch](https://forums.swift.org/t/76138)) ([review](https://forums.swift.org/t/se-0456-add-span-providing-properties-to-standard-library-types/77233)) ([acceptance](https://forums.swift.org/t/77684)) + +[SE-0446]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md +[SE-0447]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md +[PR-2305]: https://github.com/swiftlang/swift-evolution/pull/2305 +[SE-0453]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md + +## Introduction + +We recently [introduced][SE-0447] the `Span` and `RawSpan` types, but did not provide ways to obtain instances of either from existing types. This proposal adds properties that vend a lifetime-dependent `Span` from a variety of standard library types, as well as vend a lifetime-dependent `RawSpan` when the underlying element type supports it. + +## Motivation + +Many standard library container types can provide direct access to their internal representation. Up to now, it has only been possible to do so in an unsafe way. The standard library provides this unsafe functionality with closure-taking functions such as `withUnsafeBufferPointer()`, `withContiguousStorageIfAvailable()` and `withUnsafeBytes()`. These functions have a few different drawbacks, most prominently their reliance on unsafe types, which makes them unpalatable in security-conscious environments. Closure-taking API can also be difficult to compose with new features and with one another. These issues are addressed head-on with non-escapable types in general, and `Span` in particular. With this proposal, compatible standard library types will provide access to their internal representation via computed properties of type `Span` and `RawSpan`. + +## Proposed solution + +Computed properties returning [non-escapable][SE-0446] copyable values represent a particular case of lifetime relationships between two bindings. While initializing a non-escapable value in general requires [lifetime annotations][PR-2305] in order to correctly describe the lifetime relationship, the specific case of computed properties returning non-escapable copyable values can only represent one type of relationship between the parent binding and the non-escapable instance it provides: a borrowing relationship. + +For example, in the example below we have an instance of type `A`, with a well-defined lifetime because it is non-copyable. An instance of `A` can provide access to a type `B` which borrows the instance `A`: + +```swift +struct A: ~Copyable, Escapable {} +struct B: ~Escapable, Copyable { + init(_ a: borrowing A) {} +} +extension A { + var b: B { B(self) } +} + +func function() { + var a = A() + var b = a.b // access to `a` begins here + read(b) + // `b` has ended here, ending access to `a` + modify(&a) // `modify()` can have exclusive access to `a` +} +``` +If we were to attempt using `b` again after the call to `modify(&a)`, the compiler would report an overlapping access error, due to attempting to mutate `a` (with `modify(&a)`) while it is already being accessed through `b`'s borrow. Note that the copyability of `B` means that it cannot represent a mutation of `A`; it therefore represents a non-exclusive borrowing relationship. + +Given this, we propose to enable the definition of a borrowing relationship via a computed property. With this feature we then propose to add `span` computed properties to standard library types that can share access to their internal typed memory. When a `span` has `BitwiseCopyable` elements, it will have a `bytes` computed property to share a view of the memory it represents as untyped memory. + +One of the purposes of `Span` is to provide a safer alternative to `UnsafeBufferPointer`. This proposal builds on it and allows us to rewrite code reliant on `withUnsafeBufferPointer()` to use `span` properties instead. Eventually, code that requires access to contiguous memory can be rewritten to use `Span`, gaining better composability in the process. For example: + +```swift +let result = try myArray.withUnsafeBufferPointer { buffer in + let indices = findElements(buffer) + var myResult = MyResult() + for i in indices { + try myResult.modify(buffer[i]) + } +} +``` + +This closure-based call is difficult to evolve, such as making `result` have a non-copyable type, adding a concurrent task, or adding typed throws. An alternative based on a vended `Span` property would look like this: + +```swift +let span = myArray.span +let indices = findElements(span) +var myResult = MyResult() +for i in indices { + try myResult.modify(span[i]) +} +``` + +In this version, code evolution is not constrained by a closure. Incorrect escapes of `span` will be diagnosed by the compiler, and the `modify()` function can be updated with typed throws, concurrency or other features as necessary. + +## Detailed Design + +Computed property getters returning non-escapable and copyable types (`~Escapable & Copyable`) become possible, requiring no additional annotations. The lifetime of their returned value depends on the type vending them. A `~Escapable & Copyable` value borrows another binding. In terms of the law of exclusivity, a borrow is a read-only access. Multiple borrows are allowed to overlap, but cannot overlap with any mutation. + +A computed property getter defined on an `Escapable` type and returning a `~Escapable & Copyable` value establishes a borrowing lifetime relationship of the returned value on the callee's binding. As long as the returned value exists (including local copies,) then the callee's binding remains borrowed. + +A computed property getter defined on a non-escapable and copyable (`~Escapable & Copyable`) type and returning a `~Escapable & Copyable` value copies the lifetime dependency of the callee. The returned value becomes an additional borrow of the callee's dependency, but is otherwise independent from the callee. + +A computed property getter defined on a non-escapable and non-copyable (`~Escapable & ~Copyable`) type returning a `~Escapable & Copyable` value establishes a borrowing lifetime relationship of the returned value on the callee's binding. As long as the returned value exists (including local copies,) then the callee's binding remains borrowed. + +By allowing the language to define lifetime dependencies in these limited ways, we can add `Span`-providing properties to standard library types. + +#### Extensions to Standard Library types + +The standard library and Foundation will provide `span` computed properties, returning lifetime-dependent `Span` instances. These computed properties are the safe and composable replacements for the existing `withUnsafeBufferPointer` closure-taking functions. + +```swift +extension Array { + /// Share this `Array`'s elements as a `Span` + var span: Span { get } +} + +extension ArraySlice { + /// Share this `Array`'s elements as a `Span` + var span: Span { get } +} + +extension ContiguousArray { + /// Share this `Array`'s elements as a `Span` + var span: Span { get } +} + +extension String.UTF8View { + /// Share this `UTF8View`'s code units as a `Span` + var span: Span { get } +} + +extension Substring.UTF8View { + /// Share this `UTF8View`'s code units as a `Span` + var span: Span { get } +} + +extension CollectionOfOne { + /// Share this `Collection`'s element as a `Span` + var span: Span { get } +} + +extension KeyValuePairs { + /// Share this `Collection`'s elements as a `Span` + var span: Span<(Key, Value)> { get } +} +``` + +Following the acceptance of [`InlineArray`][SE-0453], we will also add the following: + +```swift +extension InlineArray where Element: ~Copyable { + /// Share this `InlineArray`'s elements as a `Span` + var span: Span { get } +} +``` + +#### Accessing the raw bytes of a `Span` + +When a `Span`'s element is `BitwiseCopyable`, we allow viewing the underlying memory as raw bytes with `RawSpan`: + +```swift +extension Span where Element: BitwiseCopyable { + /// Share the raw bytes of this `Span`'s elements + var bytes: RawSpan { get } +} +``` + +The returned `RawSpan` instance will borrow the same binding as is borrowed by the `Span`. + +#### Extensions to unsafe buffer types + +We hope that `Span` and `RawSpan` will become the standard ways to access shared contiguous memory in Swift, but current API provide `UnsafeBufferPointer` and `UnsafeRawBufferPointer` instances to do this. We will provide ways to unsafely obtain `Span` and `RawSpan` instances from them, in order to bridge `UnsafeBufferPointer` to contexts that use `Span`, or `UnsafeRawBufferPointer` to contexts that use `RawSpan`. + +```swift +extension UnsafeBufferPointer { + /// Unsafely view this buffer as a `Span` + var span: Span { get } +} + +extension UnsafeMutableBufferPointer { + /// Unsafely view this buffer as a `Span` + var span: Span { get } +} + +extension UnsafeRawBufferPointer { + /// Unsafely view this raw buffer as a `RawSpan` + var bytes: RawSpan { get } +} + +extension UnsafeMutableRawBufferPointer { + /// Unsafely view this raw buffer as a `RawSpan` + var bytes: RawSpan { get } +} +``` + +All of these unsafe conversions return a value whose lifetime is dependent on the _binding_ of the UnsafeBufferPointer. This dependency does not keep the underlying memory alive. As is usual where the `UnsafePointer` family of types is involved, the programmer must ensure the memory remains allocated while it is in use. Additionally, the following invariants must remain true for as long as the `Span` or `RawSpan` value exists: + + - The underlying memory remains initialized. + - The underlying memory is not mutated. + +Failure to maintain these invariants results in undefined behaviour. + +#### Extensions to `Foundation.Data` + +While the `swift-foundation` package and the `Foundation` framework are not governed by the Swift evolution process, `Data` is similar in use to standard library types, and the project acknowledges that it is desirable for it to have similar API when appropriate. Accordingly, we would add the following properties to `Foundation.Data`: + +```swift +extension Foundation.Data { + // Share this `Data`'s bytes as a `Span` + var span: Span { get } + + // Share this `Data`'s bytes as a `RawSpan` + var bytes: RawSpan { get } +} +``` + +Unlike with the standard library types, we plan to have a `bytes` property on `Foundation.Data` directly. This type conceptually consists of untyped bytes, and `bytes` is likely to be the primary way to directly access its memory. As `Data`'s API presents its storage as a collection of `UInt8` elements, we provide both `bytes` and `span`. Types similar to `Data` may choose to provide both typed and untyped `Span` properties. + +#### Performance + +The `span` and `bytes` properties should be performant and return their `Span` or `RawSpan` with very little work, in O(1) time. This is the case for all native standard library types. There is a performance wrinkle for bridged `Array` and `String` instances on Darwin-based platforms, where they can be bridged to Objective-C types that may not be represented in contiguous memory. In such cases the implementation will eagerly copy the underlying data to the native Swift form, and return a `Span` or `RawSpan` pointing to that copy. + +This eager copy behaviour will be specific to the `span` and `bytes` properties, and therefore the memory usage behaviour of existing unchanged code will remain the same. New code that adopts the `span` and `bytes` properties will occasionally have higher memory usage due to the eager copies, but we believe this performance compromise is the right approach for the standard library. The alternative is to compromise the design for all platforms supported by Swift, and we consider that a non-starter. + +As a result of the eager copy behaviour for bridged `String.UTF8View` and `Array` instances, the `span` property for these types will have a documented performance characteristic of "amortized constant time performance." + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a version of the Swift standard library which include the `Span` and `RawSpan` types. + +## Alternatives considered + +#### Adding `withSpan()` and `withBytes()` closure-taking functions + +The `span` and `bytes` properties aim to be safe replacements for the `withUnsafeBufferPointer()` and `withUnsafeBytes()` closure-taking functions. We could consider `withSpan()` and `withBytes()` closure-taking functions that would provide an quicker migration away from the older unsafe functions. We do not believe the closure-taking functions are desirable in the long run. In the short run, there may be a desire to clearly mark the scope where a `Span` instance is used. The default method would be to explicitly consume a `Span` instance: +```swift +var a = ContiguousArray(0..<8) +var span = a.span +read(span) +_ = consume span +a.append(8) +``` + +In order to visually distinguish this lifetime, we could simply use a `do` block: +```swift +var a = ContiguousArray(0..<8) +do { + let span = a.span + read(span) +} +a.append(8) +``` + +A more targeted solution may be a consuming function that takes a non-escaping closure: +```swift +var a = ContiguousArray(0..<8) +var span = a.span +consuming(span) { span in + read(span) +} +a.append(8) +``` + +During the evolution of Swift, we have learned that closure-based API are difficult to compose, especially with one another. They can also require alterations to support new language features. For example, the generalization of closure-taking API for non-copyable values as well as typed throws is ongoing; adding more closure-taking API may make future feature evolution more labor-intensive. By instead relying on returned values, whether from computed properties or functions, we build for greater composability. Use cases where this approach falls short should be reported as enhancement requests or bugs. + +#### Different naming for the properties + +We originally proposed the name `storage` for the `span` properties introduced here. That name seems to imply that the returned `Span` is the storage itself, rather than a view of the storage. That would be misleading for types that own their storage, especially those that delegate their storage to another type, such as a `ContiguousArray`. In such cases, it would make sense to have a `storage` property whose type is the type that implements the storage. + +#### Disallowing the definition of non-escapable properties of non-escapable types + +The particular case of the lifetime dependence created by a property of a copyable non-escapable type is not as simple as when the parent type is escapable. There are two possible ways to define the lifetime of the new instance: it can either depend on the lifetime of the original instance, or it can acquire the lifetime of the original instance and be otherwise independent. We believe that both these cases can be useful, but that in the majority of cases the desired behaviour will be to have an independent return value, where the newly returned value borrows the same binding as the callee. Therefore we believe that is reasonable to reserve the unannotated spelling for this more common case. + +The original version of this pitch disallowed this. As a consequence, the `bytes` property had to be added on each individual type, rather than having `bytes` as a conditional property of `Span`. + +#### Omitting extensions to `UnsafeBufferPointer` and related types + +We could omit the extensions to `UnsafeBufferPointer` and related types, and rely instead of future `Span` and `RawSpan` initializers. The initializers can have the advantage of being able to communicate semantics (somewhat) through their parameter labels. However, they also have a very different shape than the `span` computed properties we are proposing. We believe that the adding the same API on both safe and unsafe types is advantageous, even if the preconditions for the properties cannot be statically enforced. + +## Future directions + +Note: The future directions stated in [SE-0447](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md#Directions) apply here as well. + +#### Safe mutations with `MutableSpan` + +Some data structures can delegate mutations of their owned memory. In the standard library the function `withMutableBufferPointer()` provides this functionality in an unsafe manner. We expect to add a `MutableSpan` type to support delegating mutations of initialized memory. Standard library types will then add a way to vend `MutableSpan` instances. This could be with a closure-taking `withMutableSpan()` function, or a new property, such as `var mutableStorage`. Note that a computed property providing mutable access needs to have a different name than the `span` properties proposed here, because we cannot overload the return type of computed properties based on whether mutation is desired. + +#### A `ContiguousStorage` protocol + +An early version of the `Span` proposal ( [SE-0447][SE-0447] ) proposed a `ContiguousStorage` protocol by which a type could indicate that it can provide a `Span`. `ContiguousStorage` would form a bridge between generically-typed interfaces and a performant concrete implementation. It would supersede the rejected [SE-0256](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0256-contiguous-collection.md), and many of the standard library collections could conform to `ContiguousStorage`. + +The properties added by this proposal are largely the concrete implementations of `ContiguousStorage`. As such, it seems like an obvious enhancement to this proposal. + +Unfortunately, a major issue prevents us from proposing it at this time: the ability to suppress requirements on `associatedtype` declarations was deferred during the review of [SE-0427](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md). Once this restriction is lifted, then we could propose a `ContiguousStorage` protocol. + +The other limitation stated in [SE-0447][SE-0447]'s section about `ContiguousStorage` is "the inability to declare a `_read` acessor as a protocol requirement." This proposal's addition to enable defining a borrowing relationship via a computed property is a solution to that, as long as we don't need to use a coroutine accessor to produce a `Span`. While allowing the return of `Span`s through coroutine accessors may be undesirable, whether it is undesirable is unclear until coroutine accessors are formalized in the language. + +`span` properties on standard library SIMD types + +This proposal as reviewed included `span` properties for the standard library `SIMD` types. We are deferring this feature at the moment, since it is difficult to define these succinctly. The primary issue is that the `SIMD`-related protocols do not explicitly require contiguous memory; assuming that they are represented in contiguous memory fails with theoretically-possible examples. We could define the `span` property systematically for each concrete SIMD type in the standard library, but that would be very repetitive (and expensive from the point of view of code size.) We could also fix the SIMD protocols to require contiguous memory, enabling a succinct definition of their `span` property. Finally, we could also rely on converting `SIMD` types to `InlineArray`, and use the `span` property defined on `InlineArray`. + +## Acknowledgements + +Thanks to Ben Rimmington for suggesting that the `bytes` property should be on `Span` rather than on every type. diff --git a/proposals/0457-duration-attosecond-represenation.md b/proposals/0457-duration-attosecond-represenation.md new file mode 100644 index 0000000000..b88984362b --- /dev/null +++ b/proposals/0457-duration-attosecond-represenation.md @@ -0,0 +1,110 @@ +# Expose attosecond representation of `Duration` + +* Proposal: [SE-0457](0457-duration-attosecond-represenation.md) +* Authors: [Philipp Gabriel](https://github.com/ph1ps) +* Review Manager: [Stephen Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#78202](https://github.com/swiftlang/swift/pull/78202) +* Review: ([pitch](https://forums.swift.org/t/pitch-adding-int128-support-to-duration))([review](https://forums.swift.org/t/se-0457-expose-attosecond-representation-of-duration/77249)) + +## Introduction +This proposal introduces public APIs to enable seamless integration of `Int128` into the `Duration` type. Specifically, it provides support for directly accessing a `Duration`'s attosecond representation via the newly available `Int128` type and simplifies the creation of `Duration` values from attoseconds. + +## Motivation +The `Duration` type currently offers two ways to construct and decompose itself: + +**Low and high bits** +```swift +public struct Duration: Sendable { + public var _low: UInt64 + public var _high: Int64 + public init(_high: Int64, low: UInt64) { ... } +} +``` +**Components** +```swift +extension Duration { + public var components: (seconds: Int64, attoseconds: Int64) { ... } + public init(secondsComponent: Int64, attosecondsComponent: Int64) { ... } +} +``` +However, both approaches have limitations when it comes to exposing `Duration`'s total attosecond representation: +- The `_low` and `_high` properties are underscored, indicating that their direct use is discouraged. +- The `components` property decomposes the value into seconds and attoseconds, requiring additional arithmetic operations for many use cases. + +This gap becomes particularly evident in scenarios like generating a random `Duration`, which currently requires verbose and potentially inefficient code: +```swift +func randomDuration(upTo maxDuration: Duration) -> Duration { + let attosecondsPerSecond: Int128 = 1_000_000_000_000_000_000 + let upperRange = Int128(maxDuration.components.seconds) * attosecondsPerSecond + Int128(maxDuration.components.attoseconds) + let (seconds, attoseconds) = Int128.random(in: 0.. Duration { + return Duration(attoseconds: Int128.random(in: 0.. Duration { ... } +} +``` + +However, this approach would introduce asymmetry to other factory methods which support both `Double` and `BinaryInteger` overloads: +```swift +extension Duration { + public static func microseconds(_ microseconds: T) -> Duration { ... } + public static func microseconds(_ microseconds: Double) -> Duration { ... } +} +``` +For attoseconds, adding these overloads would lead to practical issues: + +1. A `Double` overload is nonsensical because sub-attoseconds are not supported, meaning the method cannot represent fractional attoseconds. +2. A `BinaryInteger` overload introduces additional complexity. Since it would need to support types other than `Int128`, arithmetic operations would be necessary to ensure correct scaling and truncation, negating the simplicity and precision that the `Int128`-specific initializer aims to provide. + +Ultimately, the static func `attoseconds(_:)` would likely end up as a one-off method with only an `Int128` overload. This inconsistency diminishes the appeal of the factory method approach. The proposed `init(attoseconds:)` initializer avoids these issues, offering a direct and clear way to work with attoseconds, while remaining symmetrical with the existing `Duration` API structure. diff --git a/proposals/0458-strict-memory-safety.md b/proposals/0458-strict-memory-safety.md new file mode 100644 index 0000000000..dce70053c2 --- /dev/null +++ b/proposals/0458-strict-memory-safety.md @@ -0,0 +1,841 @@ +# Opt-in Strict Memory Safety Checking + +* Proposal: [SE-0458](0458-strict-memory-safety.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.2)** +* Feature name: `StrictMemorySafety` +* Vision: [Optional Strict Memory Safety](https://github.com/swiftlang/swift-evolution/blob/main/visions/memory-safety.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/f2cab4ddc3381d1dc7a970e813ed29e27b5ae43f/proposals/0458-strict-memory-safety.md) [2](https://github.com/swiftlang/swift-evolution/blob/9d180aea291c6b430bcc816ce12ef0174ec0237b/proposals/0458-strict-memory-safety.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-opt-in-strict-memory-safety-checking/76689)) ([review](https://forums.swift.org/t/se-0458-opt-in-strict-memory-safety-checking/77274)) ([first revision](https://forums.swift.org/t/se-0458-opt-in-strict-memory-safety-checking/77274/33)) ([second revision](https://forums.swift.org/t/se-0458-opt-in-strict-memory-safety-checking/77274/51)) ([acceptance](https://forums.swift.org/t/accepted-se-0458-opt-in-strict-memory-safety-checking/78116)) + +## Introduction + +[Memory safety](https://en.wikipedia.org/wiki/Memory_safety) is a property of programming languages and their implementations that prevents programmer errors from manifesting as [undefined behavior](https://en.wikipedia.org/wiki/Undefined_behavior) at runtime. Undefined behavior effectively breaks the semantic model of a language, with unpredictable results including crashes, data corruption, and otherwise-impossible program states. Such behavior can lead to hard-to-reproduce bugs as well as introduce security vulnerabilities. + +Swift provides memory safety with a combination of language affordances and runtime checking. However, Swift also deliberately includes some unsafe constructs, such as the `Unsafe` pointer types in the standard library, language features like `nonisolated(unsafe)`, and interoperability with unsafe languages like C. For most Swift developers, this is a pragmatic solution that provides an appropriate level of memory safety while not getting in the way. + +However, some projects want to require stronger memory-safety guarantees than Swift provides by default. These projects want to pay closer attention to uses of unsafe constructs in their code, and discourage casual use of unsafe constructs when a safe alternative exists. This proposal introduces opt-in strict memory safety checking to identify those places in Swift code that make use of unsafe language constructs and APIs. Any code written within this strictly-safe subset also works as “normal” Swift and can interoperate with existing Swift code. + +## Motivation + +Much of the recent focus on memory safety is motivated by security, because memory safety issues offer a fairly direct way to compromise a program: in fact, the lack of memory safety in C and C++ has been found to be the root cause for ~70% of reported security issues in various analyses [[1](https://msrc.microsoft.com/blog/2019/07/a-proactive-approach-to-more-secure-code/)][[2](https://www.chromium.org/Home/chromium-security/memory-safety/)]. + +### Dimensions of memory safety + +While there are a number of potential definitions for memory safety, the one provided by [this blog post](https://security.apple.com/blog/towards-the-next-generation-of-xnu-memory-safety/) breaks it down into five dimensions of safety: + +* **Lifetime safety** : all accesses to a value are guaranteed to occur during its lifetime. Violations of this property, such as accessing a value after its lifetime has ended, are often called use-after-free errors. +* **Bounds safety**: all accesses to memory are within the intended bounds of the memory allocation, such as accessing elements in an array. Violations of this property are called out-of-bounds accesses. +* **Type safety** : all accesses to a value use the type to which it was initialized, or a type that is compatible with that type. For example, one cannot access a `String` value as if it were an `Array`. Violations of this property are called type confusions. +* **Initialization safety** : all values are initialized properly prior to being used, so they cannot contain unexpected data. Violations of this property often lead to information disclosures (where data that should be invisible becomes available) or even other memory-safety issues like use-after-frees or type confusions. +* **Thread safety:** all values are accessed concurrently in a manner that is synchronized sufficiently to maintain their invariants. Violations of this property are typically called data races, and can lead to any of the other memory safety problems. + +### Memory safety in Swift + +Since its inception, Swift has provided memory safety for the first four dimensions. Lifetime safety is provided for reference types by automatic reference counting and for value types via [memory exclusivity](https://www.swift.org/blog/swift-5-exclusivity/); bounds safety is provided by bounds-checking on `Array` and other collections; type safety is provided by safe features for casting (`as?` , `is` ) and `enum` s; and initialization safety is provided by “definite initialization”, which doesn’t allow a variable to be accessed until it has been defined. Swift 6’s strict concurrency checking extends Swift’s memory safety guarantees to the last dimension. + +Swift achieves safety with a mixture of static and dynamic checks. Static checks are better when possible, because they are surfaced at compile time and carry no runtime cost. Dynamic checks are sometimes necessary and are still acceptable, so long as the failure can't escalate into a memory safety problem. Swift offers unsafe features to allow problems to be solved when neither static nor dynamic checks are sufficient. These unsafe features can still be used without compromising memory safety, but doing so requires more care because they have requirements that Swift can't automatically check. + +For example, Swift solves null references with optional types. Statically, Swift prevents you from using an optional reference without checking it first. If you're sure it's non-null, you can use the `!` operator, which is safe because Swift will dynamically check for `nil`. If you really can't afford that dynamic check, you can use [`unsafelyUnwrapped`](https://developer.apple.com/documentation/swift/optional/unsafelyunwrapped). This can still be correct if you can prove that the reference is definitely non-null for some reason that Swift doesn't know. But it is an unsafe feature because it admits violations if you're wrong. + +## Proposed solution + +This proposal introduces an opt-in strict memory safety checking mode that identifies all uses of unsafe behavior within the given module. There are several parts to this change: + +* A compiler flag `-strict-memory-safety` that enables warnings for all uses of unsafe constructs within a given module. All warnings will be in the diagnostic group `StrictMemorySafety`, enabling precise control over memory-safety-related warnings per [SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md). When strict memory safety is enabled, the `StrictMemorySafety` feature will be set: `#if hasFeature(StrictMemorySafety)` can be used to detect when Swift code is being compiled in this mode. +* An attribute `@unsafe` that indicates that a declaration is unsafe to use. Such declarations may use unsafe constructs within their signatures. +* A corresponding attribute `@safe` that indicates that a declaration whose signature contains unsafe constructs is actually safe to use. For example, the `withUnsafeBufferPointer` method on `Array` has an unsafe type in its signature (`self`), but is actually safe to use because it handles safety for the unsafe buffer pointer it vends to its closure argument. The closure itself will need to handle the unsafety when using that unsafe buffer pointer. +* An `unsafe` expression that marks any use of unsafe constructs in an expression, much like `try` and `await`. +* Standard library annotations to identify unsafe declarations. + +### Example of `unsafe` usage + +The `UnsafeBufferPointer` type will be marked with `@unsafe` in the Standard library, as will the other unsafe types (e.g., `UnsafePointer`, `UnsafeRawPointer`): + +```swift +@unsafe +public struct UnsafeBufferPointer { ... } +``` + +This indicates that use of this type is not memory-safe. Any declaration that has `UnsafeBufferPointer` as part of its type is implicitly `@unsafe`. + +```swift +// note: implicitly @unsafe due to the use of the unsafe type UnsafePointer +func sumIntBuffer(_ address: UnsafePointer?, _ count: Int) -> Int { ... } +``` + +Users of this function that enable strict safety checking will see warnings when using it. For example: + +```swift +extension Array { + func sum() -> Int { + withUnsafeBufferPointer { buffer in + // warning: use of unsafe function 'sumIntBuffer' and unsafe property 'baseAddress' + sumIntBuffer(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +Both the call to `sumIntBuffer` and access to the property `UnsafeBufferPointer.baseAddress` involve unsafe code, and therefore will produce a warning. Because `UnsafeBufferPointer` and `UnsafePointer` are `@unsafe` types, this code will get a warning regardless of whether the declarations were marked `@unsafe`, because having unsafe types in the signature of a declaration implies that they are `@unsafe`. This helps us identify more unsafe code even when the libraries we depend on haven't enabled strict safety checking themselves. + +To suppress these warnings, the expressions involving unsafe code must be marked with `unsafe` in the same manner as one would mark a throwing expression with `try` or an asynchronous expression with `async`. The warning-free version of this code is: + +```swift +extension Array { + func sum() -> Int { + withUnsafeBufferPointer { buffer in + unsafe sumIntBuffer(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +The `unsafe` keyword here indicates the presence of unsafe code within that expression. As with `try` and `await`, it can cover multiple sources of unsafety within that expression: the call to `sumIntBuffer` is unsafe, as is the use of `buffer` and `buffer.baseAddress`, yet they are all covered by one `unsafe`. It is up to the authors of an unsafe API to document the conditions under which it is safe to use that API, and the user of that API to ensure that those conditions are met within the `unsafe` expression. + +Unlike `try`, `unsafe` doesn't propagate outward: we do *not* require that the `sum` function be marked `@unsafe` just because it has unsafe code in it. Similarly, the call to `withUnsafeBufferPointer` doesn't have to be marked as `unsafe` just because it has a closure that is unsafe. The programmer may choose to indicate that `sum` is unsafe, but the assumption is that unsafe behavior is properly encapsulated when using `unsafe` if the signature doesn't contain any unsafe types. + +The function `Array.withUnsafeBufferPointer` has an unsafe type in its signature, because it passes an unsafe buffer pointer to its closure parameter. However, this function itself is addressing all of the memory-safety issues with providing such a pointer, and it's up to the closure itself to ensure that it is memory safe. Therefore, we mark `withUnsafeBufferPointer` with the `@safe` attribute to indicate that it iself is not introducing memory-safety issues: + +```swift +extension Array { + @safe func withUnsafeBufferPointer( + _ body: (UnsafeBufferPointer) throws(E) -> R + ) throws(E) -> R +} +``` + +The new attributes `@safe` and `@unsafe`, as well as the `unsafe` expression, are all available in Swift regardless of whether strict safety checking is enabled, and all code using these features retains the same semantics. Strict safety checking will *only* produce diagnostics. + +### A larger example: `swapAt` on unsafe pointers + +The operation `UnsafeMutableBufferPointer.swapAt` swaps the values at the given two indices in the buffer. Under the proposed strict safety mode, it would look like this: + +```swift +extension UnsafeMutableBufferPointer { + /*implicitly @unsafe*/ + public func swapAt(_ i: Index, _ j: Index) { + guard i != j else { return } + precondition(i >= 0 && j >= 0) + precondition(i < endIndex && j < endIndex) + let pi = unsafe (baseAddress! + i) + let pj = unsafe (baseAddress! + j) + let tmp = unsafe pi.move() + unsafe pi.moveInitialize(from: pj, count: 1) + unsafe pj.initialize(to: tmp) + } +} +``` + +The `swapAt` implementation uses a mix of safe and unsafe code. The code marked with `unsafe` identifies operations that Swift cannot verify memory safety for: + +* Performing pointer arithmetic on `baseAddress`: Swift cannot reason about the lifetime of that underlying pointer, nor whether the resulting pointer is still within the bounds of the allocation. +* Moving and initializing the actual elements. The elements need to already be initialized. + +The code itself has preconditions to ensure that the provided indices aren't out of bounds before performing the pointer arithmetic. However, there are other safety properties that cannot be checked with preconditions: that the memory associated with the pointer has been properly initialized, has a lifetime that spans the whole call, and is not being used simultaneously by any other part of the code. These safety properties are something that must be established by the *caller* of `swapAt`. Therefore, `swapAt` is considered `@unsafe` because callers of it need to reason about these properties. Because it's part of an unsafe type, it is implicitly `@unsafe`, although the author could choose to mark it as `@unsafe` explicitly. + +### Incremental adoption + +The strict memory safety checking proposed here enforces a subset of Swift. Code written within this subset must also be valid Swift code, and must interoperate with Swift code that does not use this strict checking. Compared to other efforts in Swift that introduce stricter checking or a subset, this mode is smaller and more constrained, providing better interoperability and a more gradual adoption curve: + +* Strict concurrency checking, the focus of the Swift 6 language mode, required major changes to the type system, including the propagation of `Sendable` and the understanding of what code must be run on the main actor. These are global properties that don't permit local reasoning, or even local fixes, making the interoperability problem particularly hard. In contrast, strict safety checking has little or no effect on the type system, and unsafety can be encapsulated with `unsafe` expressions or ignored by a module that doesn't enable the checking. +* [Embedded Swift](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md) is a subset of Swift that works without a runtime. Like the proposed strictly-safe subset, code written in Embedded Swift will also work as regular Swift. Embedded Swift and the strict safety checking proposed here are orthogonal and can be composed to (for example) ensure that firmware written in Swift has no runtime and provides the best memory-safety guarantees. + +A Swift module that adopts strict safety checking can address all of the resulting diagnostics by applying the `@unsafe` attribute and `unsafe` expression in the appropriate places, without changing any other code. This application of attributes can be automated through Fix-Its, making it possible to enable the mode and silence all diagnostics automatically. It would then be left to the programmer to audit those places that have used `unsafe` to encapsulate unsafe behavior, to ensure that they are indeed safe. Note that the strict safety checking does not by itself make the code more memory-safe: rather, it identifies those constructs that aren't safe, encouraging the use of safe alternatives and making it easier to audit for unsafe behavior. + +The introduction of the `@unsafe` attribute on a declaration has no effect on clients compiled without strict safety enabled. For clients that have enabled strict safety, they will start diagnosing uses of the newly-`@unsafe` API. However, these diagnostics are warnings with their own diagnostic group, so a client can ensure that they do not prevent the client from building. Therefore, modules can adopt strict safety checking at their own pace (or not) and clients of those modules are never "stuck" having to make major changes in response. + +## Detailed design + +This section describes how the primary proposed constructs, the `@unsafe` attribute, `@safe` attribute, and `unsafe` expression, interact with the strict memory safety mode, and enumerates the places in the language, standard library, and compiler that introduce non-memory-safe code. + +### Sources of unsafety + +There are a number of places in the language where one can introduce memory unsafety. This section enumerates the ways in which the language, library, and user code can introduce memory unsafety. + +#### Unsafe language constructs + +The following language constructs are always considered to be unsafe: + +* `unowned(unsafe)`: Used to store a reference without maintaining its reference count. The safe counterpart, `unowned`, uses dynamic checking to ensure that the reference isn't accessed after the corresponding object has been released. The `unsafe` variant disables that dynamic checking. Uses of `unowned(unsafe)` entities are not memory-safe. +* `unsafeAddressor`, `unsafeMutableAddressor`: These accessors vend an unsafe pointer, and are therefore unsafe to declare. Other accessors (e.g., `get` and `set`) can provide safe alternatives. The accessors are considered to be part of the signature of the property or subscript they're associated with, making the property implicitly `@unsafe` unless explicitly marked `@safe`. +* `@exclusivity(unchecked)`: Used to remove dynamic exclusivity checks from a particular variable, which can mean that dynamic exclusivity violations go undetected at run time, causing a memory safety violation. Uses of `@exclusivity(unchecked)` entities are not memory-safe. + +The following language constructs are considered to be unsafe when strict concurrency checking is enabled (i.e., in the Swift 6 language mode): + +* `nonisolated(unsafe)`: Allows a property to be accessed from concurrent code without ensuring that such accesses are done so safely. Uses of `nonisolated(unsafe)` entities are not memory-safe. +* `@preconcurrency` imports: Suppresses diagnostics related to data race safety when they relate to specific imported modules, which can introduce thread safety issues. The `@preconcurrency` import will need to be annotated with `@unsafe` in the strict safety mode. + +#### `@unsafe` attribute + +The `@unsafe` attribute can be applied to any declaration to indicate that use of that declaration can undermine memory safety. Here are some examples: + +```swift +@unsafe +public struct DataWrapper { + var buffer: UnsafeBufferPointer +} + +@unsafe +func writeToRegisterAt(integerAddress: Int, value: Int32) { ... } +``` + +Uses of `DataWrapper` and `writeToRegisterAt` within executable code will be considered to be memory-unsafe. + +When a declaration uses unsafe types within its signature, it is implicitly considered to be `@unsafe`. The signature of a declaration is the interface that the declaration presents to clients, including the parameter and result types of functions, the type of properties, and any generic parameters and requirements. For example, a method on `DataWrapper` defined as follows: + +```swift +extension DataWrapper { + public func checksum() -> Int32 { + crc32(0, buffer.baseAddress, buffer.count) + } +} +``` + +will be implicitly `@unsafe` because the type of the implicit `self` parameter is `@unsafe`. + +Generally speaking, a declaration's signature is everything that isn't within the "body" of the definition that's enclosed in braces or following the `=` of a property. One notable exception to this is default arguments, because default arguments of functions are part of the implementation of a function, not its signature. For example, the following function does not have any unsafe types in its signature, even though the default argument for `value` involves unsafe code. That unsafe code is effectively part of the body of the function, so it follows the rules for `unsafe` expressions. + +```swift +func hasDefault(value: Int = unsafe getIntegerUnsafely()) { ... } +``` + +#### Unsafe conformances + +A given type might implement a protocol in a manner that introduces unsafety, for example because the operations needed to satisfy protocol requirements cannot ensure that all uses through the protocol can maintain memory safety. For example, the `UnsafeBufferPointer` type conforms to the `Collection` protocol, but it cannot do so in a safe way because `UnsafeBufferPointer` does not provide the lifetime, bounds, or initialization safety that clients of the `Collection` protocol expect. Such conformances should be explicitly marked `@unsafe`, e.g., + +```swift +extension UnsafeBufferPointer: @unsafe Collection { ... } +``` + +Unsafe conformances are similar to unsafe types in that their presence in a declaration's signature will make that declaration implicitly `@unsafe`. For example, the use of a collection algorithm such as `firstIndex(where:)` with an `UnsafeBufferPointer` will be considered unsafe because the conformance above is `@unsafe`. + +#### Unsafe standard library APIs + +Much of the identification of unsafe Swift code that becomes available with the strict memory safety mode is due to the identification of unsafe declarations within the standard library itself, and their propagation to other types that use them. In the standard library, the following functions and types would be marked `@unsafe` : + +* `Unsafe(Mutable)(Raw)(Buffer)Pointer`, `OpaquePointer`, `CVaListPointer`: These types provide neither lifetime nor bounds safety. Over time, Swift code is likely to move toward their safe replacements, such as `(Raw)Span`. +* `(Closed)Range.init(uncheckedBounds:)`: This operation makes it possible to create a range that doesn't satisfy invariants on which other bounds safety checking (e.g., in `Array.subscript`) relies. +* `Span.subscript(unchecked:)` : An unchecked subscript whose use can introduce bounds safety problems. +* `Unmanaged`: Wrapper over reference-counted types that explicitly disables reference counting, potentially introducing lifetime safety issues. +* `unsafeBitCast`: Allows type casts that are not known to be safe, which can introduce type safety problems. +* `unsafeDowncast`: An unchecked form of an `as!` cast that can introduce type safety problems. +* `Optional.unsafelyUnwrapped`: An unchecked form of the postfix `!` operation on optionals that can introduce various type, initialization, or lifetime safety problems when `nil` is interpreted as a typed value. +* `UnsafeContinuation`, `withUnsafe(Throwing)Continuation`: An unsafe form of `withChecked(Throwing)Continuation` that does not verify that the continuation is called exactly once, which can cause various safety problems. +* `withUnsafeCurrentTask` and `UnsafeCurrentTask`: The `UnsafeCurrentTask` type does not provide lifetime safety, and must only be used within the closure passed to `withUnsafeCurrentTask`. +* `UnownedSerialExecutor`: This type is intentionally not lifetime safe. It's primary use is the `unownedExecutor` property of the `Actor` protocol, which documents the lifetime assumptions of the `UnownedSerialExecutor` instance it produces. + +All of these APIs will be marked `@unsafe`. For standard library APIs that involve unsafe types, those that are safe to use will be marked `@safe` while those that require the user to maintain some aspect of safety will be marked `@unsafe`. Unless mentioned above, standard library APIs that do not have an unsafe type in their signature, but use unsafe constructs in their implementation, will be considered to be safe. + +There are also a number of unsafe conformances in the standard library: + +* `Unsafe(Mutable)(Raw)BufferPointer`: The conformances of these types to `Sequence` and the `Collection` protocol hierarchy are all `@unsafe`. +* `Unsafe(Mutable)(Raw)Pointer`: The conformances of these types to `Strideable` are `@unsafe`. + +#### Unsafe compiler flags + +There are a number of compiler flags that intentionally disable some safety-related checking. For each of these flags, the compiler will produce a diagnostic if they are used with strict memory safety: + +* `-Ounchecked`, which disables some checking in the standard library, including (for example) bounds checking on array accesses. +* `-enforce-exclusivity=unchecked` and `-enforce-exclusivity=none`, which disables exclusivity checking that is needed for memory safety. +* `-strict-concurrency=` for anything other than "complete", because the memory safety model requires strict concurrency to eliminate thread safety issues. +* `-disable-access-control`, which allows one to break invariants of a type that can lead to memory-safety issues, such as breaking the invariant of `Range` that the lower bound not exceed the upper bound. + +#### C(++) interoperability + +The C family of languages does not provide an equivalent to the strict safety mode described in this proposal, and unlike Swift, the defaults tend to be unsafe along all of the dimensions of memory safety. C(++) libriaries used within Swift can, therefore, introduce memory safety issues into the Swift code. + +The primary issue with memory safety in C(++) concerns the presence of pointers. C(++) pointers will generally be imported into Swift as an `Unsafe*Pointer` type of some form. For C functions (and C++ member functions), that means that a potentially unsafe API such as + +```swift +char *strstr(const char * haystack, const char *needle); +``` + +will be treated as implicitly `@unsafe` in Swift because its signature contains unsafe types: + +```swift +func strstr( + _ haystack: UnsafePointer?, + _ needle: UnsafePointer? +) -> UnsafeMutablePointer? +``` + +A C function that doesn't use pointer types, on the other hand, will implicitly be considered to be safe, because there are no unsafe types in its Swift signature. For example, the following would be considered safe: + +```swift +// int getchar(void); +func getchar() -> CInt +``` + +C and C++ also have user-defined types in the form of `struct`s, `enum`s, `union`s, and (in C++) `class`es. For such types, this proposal infers them to be `@unsafe` when their non-static data contains any C pointers or C types that are explicitly marked as unsafe. For example, a `Point` struct could be considered safe: + +```cpp +struct Point { + double x, y; +}; +``` + +but a `struct` with a pointer or C++ reference in it would be implicitly `@unsafe` in Swift: + +```swift +struct ListNode { + void *element; + struct ListNode *next; +}; +``` + +Note that C `enum`s will never be inferred to be `@unsafe` because they don't carry any values other than their underlying integral type, which is always a safe type. + +### Acknowledging unsafety + +All of the features described above are available in Swift regardless of whether strict memory safety checking is enabled. When strict memory safety checking is enabled, each use of an unsafe construct of any form must be ackwnowledged in the source code with one of the forms below, which provides an in-source auditable indication of where memory unsafety issues can arise. The following section describes each of the features for acknowledging memory unsafety. + +#### `unsafe` expression + +Any time there is executable code that makes use of unsafe constructs, the compiler will produce a diagnostic that indicates the use of those unsafe constructs unless it is within an `unsafe` expression. For example, consider the `DataWrapper` example from an earlier section: + +```swift +public struct DataWrapper { + var buffer: UnsafeBufferPointer +} + +extension DataWrapper { + public func checksum() -> Int32 { + crc32(0, buffer.baseAddress, buffer.count) + } +} +``` + +The property `buffer` uses an unsafe type, `UnsafeBufferPointer`. When using that property in the implementation of `checksum`, the Swift compiler will produce a warning when strict memory safety checking is enabled: + +```swift +warning: expression uses unsafe constructs but is not marked with 'unsafe' +``` + +This warning can be suppressed using the `unsafe` expression, as follows: + +```swift +extension DataWrapper { + public func checksum() -> Int32 { + unsafe crc32(0, buffer.baseAddress, buffer.count) + } +} +``` + +The `unsafe` expression is much like `try` and `await`, in that it acknowledges that unsafe constructs (`buffer`) are used within the subexpression but otherwise does not change the type. Unlike `try` and `await`, which require the enclosing context to handle throwing or be asynchronous, respectively, the `unsafe` expression does not imply any requirements about the enclosing block: it is purely a marker to indicate the presence of unsafe code, silencing a diagnostic. + +#### `@safe` attribute + +The `@safe` attribute is used on declarations whose signatures involve unsafe types but are, nonetheless, safe to use. For example, marking `UnsafeBufferPointer` as `@unsafe` means that all operations involving an unsafe buffer pointer are implicitly considered `@unsafe`. The `@safe` attribute can be used to say that those certain operations are actually safe. For example, any operation involving buffer indices or count are safe, because they don't touch the memory itself. This can be indicated by marking these APIs `@safe`: + +```swift +extension UnsafeBufferPointer { + @safe public let count: Int + @safe public var startIndex: Int { 0 } + @safe public var endIndex: Int { count } +} +``` + +For an array, the `withUnsafeBufferPointer` operation itself also involves the unsafe type that it passes along to the closure. The array itself takes responsibility for the memory safety of the unsafe buffer pointer it vends, ensuring that the elements have been initialized (which is always the case for array elements), that the bounds are correct, and that nobody else has access to the buffer when it is provided. From that perspective, `withUnsafeBufferPointer` itself can be marked `@safe`, and any unsafety will be in the closure's use of the `UnsafeBufferPointer`. + +```swift +extension Array { + @safe func withUnsafeBufferPointer( + _ body: (UnsafeBufferPointer) throws(E) -> R + ) throws(E) -> R +} +``` + +A use of this API with the `c_library_sum_function` would look like this: + +```swift +extension Array { + func sum() -> Int { + withUnsafeBufferPointer { buffer in + unsafe c_library_sum_function(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +The `@safe` annotation on a declaration takes responsibility for any variables of unsafe type that are used as its direct arguments (including the `self`). If such a variable is used to access a `@safe` property or subscript, or in a function call to a `@safe` function, it will not be diagnosed as unsafe: + +```swift +extension Array { + func sum() -> Int { + withUnsafeBufferPointer { buffer in + let count = buffer.count // count is `@safe`, no diagnostic even though 'buffer' has unsafe type + let address = buffer.baseAddress // warning: 'buffer' and 'baseAddress' are both unsafe + c_library_sum_function(address, count, 0) // warning: 'c_library_sum_function' and 'address' are both unsafe + } + } +} +``` + +#### Types with unsafe storage + +Types that wrap unsafe types will often encapsulate the unsafe behavior to provide safe interfaces. However, this requires deliberate design and implementation, potentially involving adding specific preconditions. When strict safety checking is enabled, a type whose storage includes any unsafe types or conformances will be diagnosed as involving unsafe code. For example, the `DataWrapper` struct from the prior section + +```swift +public struct DataWrapper { + var buffer: UnsafeBufferPointer +} +``` + +contains storage of an unsafe type (`UnsafeBufferPointer`), so the Swift compiler will produce a warning when strict memory safety checking is enabled: + +```swift +warning: type `DataWrapper` that includes unsafe storage must be explicitly marked `@unsafe` or `@safe` +``` + +As the warning implies, this diagnostic can be suppressed by marking the type as `@safe` or `@unsafe`. The `DataWrapper` type doesn't appear to provide safety over its storage, so it should likely be marked `@unsafe`. In contrast, one can wrap unsafe types to provide safe types. For example: + +```swift +// @safe is required to suppress a diagnostic about the 'buffer' property's use +// of an unsafe type. +@safe +struct ImmortalBufferWrapper : Collection { + let buffer: UnsafeBufferPointer + + @unsafe init(_ withImmortalBuffer: UnsafeBufferPointer) { + self.buffer = unsafe buffer + } + + subscript(index: Index) -> Element { + precondition(index >= 0 && index < buffer.count) + return unsafe buffer[index] + } + + /* Also: Index, startIndex, endIndex, index(after:) */ +} +``` + +Much of the standard library is built up as safe abstractions over unsafe code, and it's expected that user code will do the same. + +A type has unsafe storage if: + +* Any stored instance property (for `actor`, `class`, and `struct` types) or associated value (for cases of `enum` types) have a type that involves an unsafe type or conformance. +* Any stored instance property uses one of the unsafe language features (such as `unowned(unsafe)`). + +#### Unsafe witnesses + +When a type conforms to a given protocol, it must satisfy all of the requirements of that protocol. Part of this process is determining which declaration (called the *witness*) satisfies a given protocol requirement. If a particular witness is unsafe but the corresponding requirement is safe, the compiler will produce a warning: + +```swift +protocol P { + func f() +} + +struct ConformsToP { } + +extension ConformsToP: P { + @unsafe func f() { } // warning: unsafe instance method 'f()' cannot satisfy safe requirement +} +``` + +This unsafety can be acknowledged by marking the conformance as `@unsafe`, e.g., + +```swift +extension ConformsToP: @unsafe P { + @unsafe func f() { } // okay, it's an unsafe conformance +} +``` + +#### Unsafe overrides + +Overriding a safe method within an `@unsafe` one could introduce unsafety, so it will produce a diagnostic in the strict safety mode: + +```swift +class Super { + func f() { } +} + +class Sub: Super { + @unsafe override func f() { ... } // warning: override of safe instance method with unsafe instance method +} +``` + +to suppress this warning, the `Sub` class itself can be marked as `@unsafe`, e.g., + +```swift +@unsafe +class Sub: Super { + override func f() { ... } // no more warning +} +``` + +The `@unsafe` annotation is at the class level because any use of the `Sub` type can now introduce unsafe behavior, and any indication of that unsafe behavior will be lost once that `Sub` is converted to a `Super` instance. + +#### `for..in` loops + +Swift's `for..in` loops are effectively implemented as syntactic sugar over the `Sequence` and `IteratorProtocol` protocols, where the `for..in` creates a new iterator (with `Sequence.makeIterator()`) and then calls its `next()` operation for each loop iteration. If the conformances to `Sequence` or `IteratorProtocol` are `@unsafe`, the loop will introduce a warning: + +```swift +let someUnsafeBuffer: UnsafeBufferPointer = unsafe ... +for x in someBuffer { // warning: use of unsafe conformance of 'UnsafeBufferPointer' to 'Sequence' + // and someBuffer has unsafe type 'UnsafeBufferPointer' + // ... +} +``` + +Following the precedent set by [SE-0298](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0298-asyncsequence.md), which introduced effects in loops, the `unsafe` keyword that acknowledges unsafe behavior in the iteration will follow the `for`: + +```swift +let someUnsafeBuffer: UnsafeBufferPointer = unsafe ... +for unsafe x in someBuffer { // still warns that someBuffer has unsafe type 'UnsafeBufferPointer' + // ... +} +``` + +This may not be the only unsafe behavior in the `for..in` loop. For example, the expression that produces the sequence itself (via the reference to `someBuffer`) is also unsafe, so it needs to be acknowledged: + +```swift +let someUnsafeBuffer: UnsafeBufferPointer = unsafe ... +for unsafe x in unsafe someBuffer { + // ... +} +``` + +This repeated `unsafe` also occurs with the other effects: if an `async throws` function `getAsyncSequence()` produces an `AsyncSequence` whose iteration can throw, one will end up with two `try` and `await` keywords: + +```swift +for try await x in try await getAsyncSequence() { ... } +``` + +### Strict safety mode and escalatable warnings + +The strict memory safety mode can be enabled with the new compiler flag `-strict-memory-safety`. + +All of the memory-safety diagnostics produced by the strict memory safety mode will be warnings. These warnings be in the group `StrictMemorySafety` (possibly organized into subgroups) so that one can choose to escalate them to errors or keep them as warnings using the compiler flags introduced in [SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md). For example, one can choose to enable the mode and make memory-safety issues errors using: + +``` +swiftc -strict-memory-safety -Werror StrictMemorySafety +``` + +### SwiftPM integration + +Swift package manifests will need a way to enable strict memory safety mode on a per-module and per-package basis. This proposal extends the `SwiftSetting` type in the manifest with a new option to enable that checking: + +```swift +static func strictMemorySafety( + _ condition: BuildSettingCondition? = nil +) -> SwiftSetting +``` + +## Source compatibility + +The `unsafe` keyword in this proposal will be introduced as a contextual keyword following the precedent set by `await`'s' introduction in [SE-0296](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md) and `consume`'s introduction in [SE-0366](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0366-move-function.md). This allows `unsafe` to continue to be used as an identifier, albeit with a small potential to break existing source that uses `unsafe` as a function that is then called with a trailing closure, like this: + +```swift +func unsafe(_ body: () -> Void) { } + +unsafe { + // currently calls 'unsafe(_:)', but will become an unsafe expression +} +``` + +As with those proposals, the impact of this source break in expected to be small enough that it is acceptable. If not, the parsing of the `unsafe` expression can be limited to code that has enabled strict safety checking. + +Other than the source break above, the introduction of this strict safety checking mode has no impact on source compatibility for any module that does not enable it. When enabling strict safety checking, source compatibility impact is limited to the introduction of warnings that will not break source compatibility (and can be treated as warnings even under `-warnings-as-errors` mode using the aforementioned diagnostic flags). The interoperability story is covered in detail in prior sections. + +## ABI compatibility + +The attributes, `unsafe` expression, and strict memory-safety checking model proposed here have no impact on ABI. + +## Future Directions + +### The `SerialExecutor` and `Actor` protocols + +The `SerialExecutor` protocol provides a somewhat unique challenge for the strict memory safety mode. For one, it is impossible to implement this protocol with entirely safe code due to the presence of the `unownedExecutor` requirement: + +```swift +protocol SerialExecutor: Executor { + // ... + @unsafe var unownedExecutor: UnownedSerialExecutor { get } +} +``` + +To make it possible to safely implement `SerialExecutor`, the protocol will need to be extended with a safe form of `unownedExecutor`, which itself will likely require a [non-escapable](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md) form of `UnownedSerialExecutor` to provide lifetime safety without introducing any overhead. The `Actor` protocol has the same `unownedExecutor` requirement, so it will need the corresponding safe variant. The Swift implementation will need to start using this new requirement for scheduling work on actors to eliminate the implicit use of unsafe constructs. + +The `SerialExecutor` protocol has additional semantic constraints involving the serial execution of the jobs provided to the executor. While conformance to any protocol implies that the conforming type meets the documented semantic requirements, `SerialExecutor` is unique in that the data-race safety model (and therefore the memory safety model) depends on it correctly implementing these semantics: a conforming type that currently executes two jobs will create memory-safety violations. There are a few options for addressing this: + +* The `SerialExecutor` protocol itself could be marked `@unsafe`, meaning that any use of this protocol must account for unsafety. +* Some requirements of the `SerialExecutor` protocol (such as the replacement for `unownedExecutor`) could be marked `@unsafe`, so any use of this protocol's requirements must account for unsafety. +* Conformance to the `SerialExecutor` protocol could require some attestation (such as `@safe(unchecked)`) to make it clear from the source code that there is some unsafety encapsulated in the conformance. + +The first two options are the most straightforward, but the fact that actors have implicit uses of `SerialExecutor` means that it would effectively make every actor `@unsafe`. This pushes the responsibility for acknowledging the memory unsafety to clients of `SerialExecutor`, rather than at the conforming type where the responsibility for a correct implementation lies. The third option appears best, because it provides an auditable place to assert memory safety that corresponds with where extra care must be taken to avoid introducing a problem. + +It is unclear whether `SerialExecutor` is or will be the only protocol of this nature. If there are others, it could be worth providing a special form of the `@unsafe` attribute on the protocol itself, such as `@unsafe(conforms)`, that is only considered unsafe for conforming types. + +### Handling of `@unsafe` cases + +When an enum case is explicitly marked `@unsafe`, but involves no associated data that is unsafe, this proposal doesn't have a way to suppress safety diagnostics when pattern matching that case. For example: + +```swift +enum WeirdAddress { + @unsafe case rawOffsetIntoGlobalArray(Int) +} + +func example(_ address: WeirdAddress) { + if case .rawOffsetIntoGlobalArray(let offset) = weirdAddress { // reference to @unsafe case rawOffsetIntoGlobalArray that can't be suppressed + } +} + +``` + +We have several options here: + +* We could suppress the diagnostic for this use of an `@unsafe case`. One would still get diagnostics when constructing such a case. + +* We could reject `@unsafe` on case declarations that don't involve any unsafe types. + +* We could extend the pattern grammar with an `unsafe` pattern to suppress this diagnostic, e.g., + ```swift + if case unsafe .rawOffsetIntoGlobalArray(let offset) = weirdAddress { ... } + ``` + +### Handling unsafe code in macro expansions + +A macro can expand to any code. If the macro-expanded code contains uses of unsafe constructs not properly covered by `@safe`, `@unsafe`, or an `unsafe` expression within the macro, then strict safety checking will diagnose those safety issues within the macro expansion. In this case, the client of the macro does not have any way to suppress diagnostics within the macro expansion itself without modifying the implementation of the macro. + +There are a number of possible approaches that one could use for suppression. The `unsafe` expression could be made to apply to everything in the macro expansion, which would also require some spelling for attached attributes and other places where expressions aren't permitted. Alternatively, Swift could introduce a general syntax for suppressing a class of warnings within a block of code, and that could be used to surround the macro expansion. + +Note that both of these approaches trade away some of the benefits of the strict safety mode for the convenience of suppressing safety-related diagnostics. + +## Alternatives considered + +### Prohibiting unsafe conformances and overrides entirely + +This proposal introduces two places where polymorphism interacts with unsafety: protocol conformances and overrides. In both cases, a safe abstraction (e.g., a superclass or protocol) has a specific implementation that is unsafe, and there is a way to note the unsafety: + +* When overriding a safe declaration with an unsafe one, the overriding subclass must be marked `@unsafe`. +* When implementing a safe protocol requirement with an unsafe declaration, the corresponding conformance must be marked `@unsafe`. + +In both cases, the current proposal will consider uses of the type (in the overriding case) or conformance (for that case) as unsafe, respectively. However, that unsafety is not localized, because code that's generally safe can now cause safety problems when calling through polymorphic operations. For example, consider a function that operates on a general collection: + +```swift +func parse(_ input: some Collection) -> ParseResult +``` + +Calling this function with an unsafe buffer pointer will produce a diagnostic due to the use of the unsafe conformance of `UnsafeBufferPointer` to `Collection`: + +```swift +let result = parse(unsafeBufferPointer) // warning: use of unsafe conformance +``` + +Marking the call as `unsafe` will address the diagnostic. However, because `UnsafeBufferPointer` doesn't perform bounds checking, the `parse` function itself can introduce a memory safety problem if it subscripts into the collection with an invalid index. There isn't a way to communicate how the code that is `unsafe` is addressing memory safety issues within the context of the call. + +This proposal could prohibit use of unsafe conformances and overrides entirely, for example by making it impossible to suppress the diagnostics associated with their definition and use. This would require the `parse(unsafeBufferPointer)` call to be refactored to avoid the unsafe conformance, for example by introducing a wrapper type: + +```swift +@safe struct ImmortalBufferWrapper : Collection { + let buffer: UnsafeBufferPointer + + @unsafe init(_ withImmortalBuffer: UnsafeBufferPointer) { + self.buffer = unsafe buffer + } + + subscript(index: Index) -> Element { + precondition(index >= 0 && index < buffer.count) + return unsafe buffer[index] + } + + /* Also: Index, startIndex, endIndex, index(after:) */ +} +``` + +The call would then look like this: + +```swift +let wrapper = unsafe ImmortalBufferWrapper(withImmortalBuffer: buffer) +let result = parse(wrapper) +``` + +This approach is better than the prior one: it improves bounds safety by introducing bounds checking. It clearly documents the assumptions made around lifetime safety. It is both functionally safer (due to bounds checks) and makes it easier to reason that the `unsafe` is correctly used. It does require a lot more code, and the code itself requires careful reasoning about safety (e.g., the right preconditions for bounds checking; the right naming to capture the lifetime implications). + +Unsafe conformances and overrides remain part of this proposal because prohibiting them doesn't fundamentally change the safety model. Rather, it requires the introduction of more abstractions that could be safer--or could just be boilerplate. Swift has a number of constructs that are functionally similar to unsafe conformances, where safety checking can be disabled locally despite that having wide-ranging consequences: `@unchecked Sendable`, `nonisolated(unsafe)`, `unowned(unsafe)`, and `@preconcurrency` all fall into this category. + +### `@unsafe` implying `unsafe` throughout a function body + +A function marked `@unsafe` is unsafe to use, so any clients that have enabled strict safety checking will need to put uses of the function into an `unsafe` expression. The implementation of that function is likely to use unsafe code (possibly a lot of it), which could result in a large number of annotations: + +```swift +extension UnsafeMutableBufferPointer { + @unsafe public func swapAt(_ i: Index, _ j: Index) { + guard i != j else { return } + precondition(i >= 0 && j >= 0) + precondition(unsafe i < endIndex && j < endIndex) + let pi = unsafe (baseAddress! + i) + let pj = unsafe (baseAddress! + j) + let tmp = unsafe pi.move() + unsafe pi.moveInitialize(from: pj, count: 1) + unsafe pj.initialize(to: tmp) + } +} +``` + +We could choose to make `@unsafe` on a function acknowledge all uses of unsafe code within its definition. For example, this would mean that marking `swapAt` with `@unsafe` means that one need not have any `unsafe` expressions in its body: + +```swift +extension UnsafeMutableBufferPointer { + @unsafe public func swapAt(_ i: Index, _ j: Index) { + guard i != j else { return } + precondition(i >= 0 && j >= 0) + precondition(i < endIndex && j < endIndex) + let pi = (baseAddress! + i) + let pj = (baseAddress! + j) + let tmp = pi.move() + pi.moveInitialize(from: pj, count: 1) + pj.initialize(to: tmp) + } +} +``` + +This approach reduces the annotation burden in unsafe code, but makes it much harder to tell exactly what aspects of the implementation are unsafe. Indeed, even unsafe functions should still strive to minimize the use of unsafe constructs, and benefit from having the actual unsafe behavior marked in the source. It also conflates the notion of "exposes an unsafe interface" from "has an unsafe implementation". + +Rust's `unsafe` functions have this behavior, where an `unsafe fn` in Rust implies an `unsafe { ... }` block around the entire function body. [Rust RFC #2585](https://rust-lang.github.io/rfcs/2585-unsafe-block-in-unsafe-fn.html) argues for Rust to remove this behavior; the motivation there generally applies to Swift as well. + +### Making "encapsulation" of unsafe behavior explicit + +In the proposed design, a function with no unsafe types in its signature is considered safe unless the programmer explicitly marked it `@unsafe`. The implementation may contain any amount of unsafe code, so long as it is covered by an `unsafe` expression: + +```swift +extension Array { + // this function is considered safe + func sum() -> Int { + withUnsafeBufferPointer { buffer in + unsafe sumIntBuffer(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +This differs somewhat from the way in which throwing and asynchronous functions work. A function that has a `try` or `await` in the body needs to be `throws` or `async`, respectively. Essentially, the effect from the body has to also be reflected in the signature. With unsafe code, this could mean that having `unsafe` expressions in the function body requires you to either make the function `@unsafe` or use some other suppression mechanism to acknowledge that you are using unsafe constructs to provide a safe interface. + +There are several options for such a suppression mechanism. An attribute form, `@safe(unchecked)`, is described below as an alternative to the `unsafe` expression. Another approach would be to provide an `unsafe!` form the `unsafe` expression, which (like `try!`) acknowledges the effect but doesn't propagate that effect out to the function. For the `sum` function, it would be used as follows: + +```swift +extension Array { + // this function is considered safe + func sum() -> Int { + withUnsafeBufferPointer { buffer in + unsafe! sumIntBuffer(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +This proposal chooses not to go down this path, because having a function signature involving no unsafe types is already a strong indication that the function is providing a safe interface, and there is little to be gained from requiring additional ceremony (whether an attribute like `@safe(unchecked)` or the `unsafe!` form described above). + +### `@safe(unchecked)` attribute to allow unsafe code + +Early iterations of this proposal introduced a `@safe(unchecked)` attribute as an alternative to `unsafe` expressions. The `@safe(unchecked)` attribute would be placed on a function to suppress diagnostics about use of unsafe constructs within its definition. This has all of the same downsides as having `@unsafe` imply cover for all of the uses of unsafe code within the body of a function, albeit while providing a safe interface. + +### `unsafe` blocks + +The `unsafe` expression proposed here covers unsafe constructs within a single expression. For unsafe-heavy code, this can introduce a large number of `unsafe` keywords. There is an alternative formulation for acknowledging unsafe code that is used in some peer languages like C# and Rust: an `unsafe` block, which is a statement that suppresses diagnostics about uses of unsafe code within it. For example: + +```swift +extension UnsafeMutableBufferPointer { + @unsafe public func swapAt(_ i: Index, _ j: Index) { + guard i != j else { return } + precondition(i >= 0 && j >= 0) + precondition(i < endIndex && j < endIndex) + unsafe { + let pi = (baseAddress! + i) + let pj = (baseAddress! + j) + let tmp = pi.move() + pi.moveInitialize(from: pj, count: 1) + pj.initialize(to: tmp) + } + } +} +``` + +For Swift, an `unsafe` block would be a statement that can also be used as an expression when its body is an expression, much like `if` or `switch` expressions following [SE-0380](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0380-if-switch-expressions.md). + +`unsafe` blocks are more coarse-grained than the proposed `unsafe` expressions, which represents a trade-off: `unsafe` blocks will be less noisy for unsafe-heavy code, because one `unsafe { ... }` can cover a lot of code. On the other hand, doing so hides which code within the block is actually unsafe, making it harder to audit the unsafe parts. In languages that have `unsafe` blocks, it's considered best practice to make the `unsafe` blocks as narrow as possible. The proposed `unsafe` expressions enforce that best practice at the language level. + +### Strictly-safe-by-default + +This proposal introduced strict safety checking as an opt in mode and not an [*upcoming* language feature](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md) because there is no intent to make this feature the default behavior in a future language mode. There are several reasons why this checking should remain an opt-in feature for the foreseeable future: + +* The various `Unsafe` pointer types are the only way to work with contiguous memory in Swift today, and the safe replacements (e.g., `Span`) are new constructs that will take a long time to propagate through the ecosystem. Some APIs depending on these `Unsafe` pointer types cannot be replaced because it would break existing clients (either source, binary, or both). +* Interoperability with the C family of languages is an important feature for Swift. Most C(++) APIs are unlikely to ever adopt the safety-related attributes described above, which means that enabling strict safety checking by default would undermine the usability of C(++) interoperability. +* Swift's current (non-strict) memory safety by default is likely to be good enough for the vast majority of users of Swift, so the benefit of enabling stricter checking by default is unlikely to be worth the disruption it would cause. + +### Overloading to stage in safe APIs + +When adopting the strict memory safety mode, it's likely that a Swift module will want to replace existing APIs that traffic in unsafe types (such as `UnsafeMutablePointer`) with safer equivalents (such as `Span`). To retain compatibility for older clients, the existing APIs will need to be left in place. Unfortunately, this might mean that the best name for the API is already taken. For example, perhaps we have a data packet that exposes its bytes via a property: + +```swift +public class DataPacket { + @unsafe public let bytes: UnsafeRawBufferPointer +} +``` + +The `bytes` property is necessarily unsafe. Far better would be to produce a `RawSpan`, which we can easily do with another property: + +```swift +extension DataPacket { + public var byteSpan: RawSpan +} +``` + +Clients using the existing `bytes` will continue to work, and those that care about memory safety can choose to move to `byteSpan`. All of this works, but is somewhat annoying because the good name, `bytes`, has been taken for the API we no longer want to use. + +Swift does allow type-based overloading, including on the type of properties, so one could introduce an overloaded `bytes` property, like this: + +```swift +extension DataPacket { + public var bytes: RawSpan +} +``` + +This works for code that accesses `bytes` and then uses it in a context where type inference can figure out whether we need an `UnsafeRawBufferPointer` or a `RawSpan`, but fails if that context does not exist: + +```swift +let unsafeButGoodBytes: UnsafeRawBufferPointer = dataPacket.bytes // ok, uses @unsafe bytes +let goodBytes: RawSpan = dataPacket.bytes // ok, uses safe bytes +let badBytes = dataPacket.bytes // error: ambiguous! +``` + +We could consider extending Swift's overloading rules to make this kind of evolution possible. For example, one could introduce a pair of rules into the language: + +1. When strict memory safety checking is enabled, `@unsafe` declarations are dis-favored vs. safe ones, so the unsafe `bytes: UnsafeRawBufferPointer` would be a worse solution for type inference to pick than the safe alternative, `bytes: RawSpan`. + +2. Overloads that were introduced to replace unsafe declarations could be marked with a new attribute `@safe(unsafeDisfavored)` so that they would be disfavored only when building with strict memory safety checking disabled. + +Assuming these rules, and that the safe `bytes: RawSpan` had the `@safe(unsafeDisfavored)` attribute, the example uses of `DataPacket` would resolve as follows: + +* `unsafeButGoodBytes` would always be initialized with the unsafe `bytes`. If strict memory safety were enabled, this use would produce a warning. +* `goodBytes` would always be initialized with the safe `bytes`. +* `badBytes` would be initialized differently based on whether strict memory safety was enabled: + * If enabled, `badBytes` would choose the safe version of `bytes` to produce the safest code, because the unsafe one is disfavored (rule #1). + * If disabled, `badBytes` would choose the unsafe version of `bytes` to provide source compatibility with existing code, because the safe one is disfavored (rule #2). + +There are downsides to this approach. It partially undermines the source compatibility story for the strict safety mode, because type inference now behaves differently when the mode is enabled. That means, for example, there might be errors---not warnings---because some code like `badBytes` above would change behavior, causing additional failures. Changing the behavior of type inference is also risky in an of itself, because it is not always easy to reason about all of the effects of such a change. That said, the benefit of being able to move toward a more memory-safe future might be worth it. + +### Optional `message` for the `@unsafe` attribute + +We could introduce an optional `message` argument to the `@unsafe` attribute, which would allow programmers to indicate *why* the use of a particular declaration is unsafe and, more importantly, how to safely write code that uses it. However, this argument isn't strictly necessary: a comment could provide the same information, and there is established tooling to expose comments to programmers that wouldn't be present for this attribute's message, so we have omitted this feature. + +## Revision history + +* **Revision 3 (following second review extension)** + * Do not require declarations with unsafe types in their signature to be marked `@unsafe`; it is implied. They may be marked `@safe` to indicate that they are actually safe. + * Add `unsafe` for iteration via the `for..in` syntax. + * Add C(++) interoperability section that infers `@unsafe` for C types that involve pointers. + * Document the unsafe conformances of the `UnsafeBufferPointer` family of types to the `Collection` protocol hierarchy. + * Restructure detailed design to separate out "sources of unsafety" from "acknowledging unsafety". + +* **Revision 2 (following first review extension)** + * Specified that variables of unsafe type passed in to uses of `@safe` declarations (e.g., calls, property accesses) are not diagnosed as themselves being unsafe. This makes means that expressions like `unsafeBufferePointer.count` will be considered safe. + * Require types whose storage involves an unsafe type or conformance to be marked as `@safe` or `@unsafe`, much like other declarations that have unsafe types or conformances in their signature. + * Add an Alternatives Considered section on prohibiting unsafe conformances and overrides. + * Add a Future Directions section on handling unsafe code in macro expansions. + +## Acknowledgments + +This proposal has been greatly improved by the feedback from Félix Cloutier, Geoff Garen, Gábor Horváth, Frederick Kellison-Linn, Karl Wagner, and Xiaodi Wu. diff --git a/proposals/0459-enumerated-collection.md b/proposals/0459-enumerated-collection.md new file mode 100644 index 0000000000..14071906cd --- /dev/null +++ b/proposals/0459-enumerated-collection.md @@ -0,0 +1,199 @@ +# Add `Collection` conformances for `enumerated()` + +* Proposal: [SE-0459](0459-enumerated-collection.md) +* Author: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#78092](https://github.com/swiftlang/swift/pull/78092) +* Previous Proposal: [SE-0312](0312-indexed-and-enumerated-zip-collections.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-add-collection-conformance-for-enumeratedsequence/76680)) ([review](https://forums.swift.org/t/se-0459-add-collection-conformances-for-enumerated/77509)) ([acceptance](https://forums.swift.org/t/accepted-with-modification-se-0459-add-collection-conformances-for-enumerated/78082)) + +## Introduction + +This proposal aims to fix the lack of `Collection` conformance of the sequence returned by `enumerated()`, preventing it from being used in a context that requires a `Collection`. + +## Motivation + +Currently, `EnumeratedSequence` type conforms to `Sequence`, but not to any of the collection protocols. Adding these conformances was impossible before [SE-0234 Remove `Sequence.SubSequence`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0234-remove-sequence-subsequence.md), and would have been an ABI breaking change before the language allowed `@available` annotations on protocol conformances ([PR](https://github.com/apple/swift/pull/34651)). Now we can add them! + +Conformance to the collection protocols can be beneficial in a variety of ways, for example: +* `(1000..<2000).enumerated().dropFirst(500)` becomes a constant time operation. +* `"abc".enumerated().reversed()` will return a `ReversedCollection` rather than allocating a new array. +* SwiftUI’s `List` and `ForEach` views will be able to directly take an enumerated collection as their data. + +## Detailed design + +Conditionally conform `EnumeratedSequence` to `Collection`, `BidirectionalCollection`, `RandomAccessCollection`. + +```swift +@available(SwiftStdlib 6.1, *) +extension EnumeratedSequence: Collection where Base: Collection { + // ... +} + +@available(SwiftStdlib 6.1, *) +extension EnumeratedSequence: BidirectionalCollection + where Base: BidirectionalCollection +{ + // ... +} + +@available(SwiftStdlib 6.1, *) +extension EnumeratedSequence: RandomAccessCollection + where Base: RandomAccessCollection {} +``` + +## Source compatibility + +All protocol conformances of an existing type to an existing protocol are potentially source breaking because users could have added the exact same conformances themselves. However, given that `EnumeratedSequence` do not expose their underlying sequences, there is no reasonable way anyone could have conformed to `Collection` themselves. + +## Effect on ABI stability + +These conformances are additive to the ABI, but will affect runtime casting mechanisms like `is` and `as`. On ABI stable platforms, the result of these operations will depend on the OS version of said ABI stable platforms. Similarly, APIs like `underestimatedCount` may return a different result depending on if the OS has these conformances or not. + +## Alternatives considered + +#### Add `LazyCollectionProtocol` conformance for `EnumeratedSequence`. + +Adding `LazySequenceProtocol` conformance for `EnumeratedSequence` is a breaking change for code that relies on the `enumerated()` method currently not propagating `LazySequenceProtocol` conformance in a lazy chain: + +```swift +extension Sequence { + func everyOther_v1() -> [Element] { + let x = self.lazy + .enumerated() + .filter { $0.offset.isMultiple(of: 2) } + .map(\.element) + + // error: Cannot convert return expression of type 'LazyMapSequence<...>' to return type '[Self.Element]' + return x + } + + func everyOther_v2() -> [Element] { + // will keep working, the eager overload of `map` is picked + return self.lazy + .enumerated() + .filter { $0.offset.isMultiple(of: 2) } + .map(\.element) + } +} +``` + +We chose to keep this proposal very small to prevent any such potential headaches of source breaks. + +#### Keep `EnumeratedSequence` the way it is and add an `enumerated()` overload to `Collection` that returns a `Zip2Sequence, Self>`. + +This is tempting because `enumerated()` is little more than `zip(0..., self)`, but this would cause an unacceptable amount of source breakage due to the lack of `offset` and `element` tuple labels that `EnumeratedSequence` provides. + +#### Only conform `EnumeratedSequence` to `BidirectionalCollection` when the base collection conforms to `RandomAccessCollection` rather than `BidirectionalCollection`. + +Here’s what the `Collection` conformance could look like: + +```swift +extension EnumeratedSequence: Collection where Base: Collection { + struct Index { + let base: Base.Index + let offset: Int + } + var startIndex: Index { + Index(base: _base.startIndex, offset: 0) + } + var endIndex: Index { + Index(base: _base.endIndex, offset: 0) + } + func index(after index: Index) -> Index { + Index(base: _base.index(after: index.base), offset: index.offset + 1) + } + subscript(index: Index) -> (offset: Int, element: Base.Element) { + (index.offset, _base[index.base]) + } +} + +extension EnumeratedSequence.Index: Comparable { + static func == (lhs: Self, rhs: Self) -> Bool { + return lhs.base == rhs.base + } + static func < (lhs: Self, rhs: Self) -> Bool { + return lhs.base < rhs.base + } +} +``` + +Here’s what the `Bidirectional` conformance could look like. The question is: should `Base` be required to conform to `BidirectionalCollection` or `RandomAccessCollection`? + +```swift +extension EnumeratedSequence: BidirectionalCollection where Base: ??? { + func index(before index: Index) -> Index { + let currentOffset = index.base == _base.endIndex ? _base.count : index.offset + return Index(base: _base.index(before: index.base), offset: currentOffset - 1) + } +} +``` + +Notice that calling `index(before:)` with the end index requires computing the `count` of the base collection. This is an O(1) operation if the base collection is `RandomAccessCollection`, but O(n) if it's `BidirectionalCollection`. + +##### Option 1: `where Base: BidirectionalCollection` + +A direct consequence of `index(before:)` being O(n) when passed the end index is that some operations like `last` are also O(n): + +```swift +extension BidirectionalCollection { + var last: Element? { + isEmpty ? nil : self[index(before: endIndex)] + } +} + +// A bidirectional collection that is not random-access. +let evenNumbers = (0 ... 1_000_000).lazy.filter { $0.isMultiple(of: 2) } +let enumerated = evenNumbers.enumerated() + +// This is still O(1), ... +let endIndex = enumerated.endIndex + +// ...but this is O(n). +let lastElement = enumerated.last! +print(lastElement) // (offset: 500000, element: 1000000) +``` + +However, since this performance pitfall only applies to the end index, iterating over a reversed enumerated collection stays O(n): + +```swift +// A bidirectional collection that is not random-access. +let evenNumbers = (0 ... 1_000_000).lazy.filter { $0.isMultiple(of: 2) } + +// Reaching the last element is O(n), and reaching every other element is another combined O(n). +for (offset, element) in evenNumbers.enumerated().reversed() { + // ... +} +``` + +In other words, this could make some operations unexpectedly O(n), but it’s not likely to make operations unexpectedly O(n²). + +##### Option 2: `where Base: RandomAccessCollection` + +If `EnumeratedSequence`’s conditional conformance to `BidirectionalCollection` is restricted to when `Base: RandomAccessCollection`, then operations like `last` and `last(where:)` will only be available when they’re guaranteed to be O(1): + +```swift +// A bidirectional collection that is not random-access. +let str = "Hello" + +let lastElement = str.enumerated().last! // error: value of type 'EnumeratedSequence' has no member 'last' +``` + +That said, some algorithms that can benefit from bidirectionality such as `reversed()` and `suffix(_:)` are also available on regular collections, but with a less efficient implementation. That means that the code would still compile if the enumerated sequence is not bidirectional, it would just perform worse — the most general version of `reversed()` on `Sequence` allocates an array and adds every element to that array before reversing it: + +```swift +// A bidirectional collection that is not random-access. +let str = "Hello" + +// This no longer conforms to `BidirectionalCollection`. +let enumerated = str.enumerated() + +// As a result, this now returns a `[(offset: Int, element: Character)]` instead +// of a more efficient `ReversedCollection>`. +let reversedElements = enumerated.reversed() +``` + +The base collection needs to be traversed twice either way, but the defensive approach of giving the `BidirectionalCollection` conformance a stricter bound ultimately results in an extra allocation. + +Taking all of this into account, we've gone with option 1 for the sake of giving collections access to more algorithms and more efficient overloads of some algorithms. Conforming this collection to `BidirectionalCollection` when the base collection conforms to the same protocol is less surprising. We don’t think the possible performance pitfalls pose a large enough risk in practice to negate these benefits. diff --git a/proposals/0460-specialized.md b/proposals/0460-specialized.md new file mode 100644 index 0000000000..8c5ead4bd2 --- /dev/null +++ b/proposals/0460-specialized.md @@ -0,0 +1,263 @@ +# Explicit Specialization + +* Proposal: [SE-0460](0460-specialized.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift Next)** +* Review: ([pitch](https://forums.swift.org/t/pitch-explicit-specialization/76967)) ([review](https://forums.swift.org/t/se-0460-explicit-specialization/77541)) ([acceptance](https://forums.swift.org/t/accepted-se-0460-explicit-specialization/78583)) + +## Introduction + +The Swift compiler has the ability to "specialize" a generic function at compile time. This specialization creates a custom implementation of the function, where the generic placeholders are substituted with specific types. This can unlock optimizations of that specialized function that can be dramatically faster than the unspecialized version in some circumstances. The compiler can generate this specialized version and call it when it can see at the call site with what concrete types a function is being called, and the body of the function to specialize. + +In some cases, though, this information is obscured from the compiler. This proposal introduces a new attribute, `@specialized`, which allows the author of a generic function to generate pre-specialized versions of that function for specific types. When the unspecialized version of the function is called with one of those types, the compiler will generate code that will re-dispatch to those prespecialized versions if available. + +## Motivation + +Consider the following generic function that sums an array of any binary integer type: + +```swift +extension Sequence where Element: BinaryInteger { + func sum() -> Double { + reduce(0) { $0 + Double($1) } + } +} +``` + +If you call this function directly on an array of a specific integer type (e.g. `Array`): + +``` +let arrayOfInt: [Int] = // ... +let result = arrayOfInt.sum() +``` + +in optimized builds, the Swift compiler will generate a specialized version of `sum` for that type. + +If you inspect the binary, you will see this specialized version under a symbol `_$sST3sumSz7ElementRpzrlEAASdyFSaySiG_Tg5`, which demangles to `generic specialization <[Swift.Int]> of (extension in example):Swift.Sequence< where A.Element: Swift.BinaryInteger>.sum() -> Swift.Double`, alongside the unspecialized version. + +This specialized version of `sum` will be optimized specifically for `Array`. It can move a pointer directly over the the array's buffer, loading the elements into registers directly from memory. On modern hardware, it can make use of dedicated instructions to convert the integers to floating point. It can do this because it can see the implementation of `sum`, and can see the exact types on which it is being called. What exact assembly instructions are generated will differ significantly between e.g. `Array.sum` and `Array.sum`. + +Here is that `Array`-specialized code when compiled to x86-64 with `swiftc -Osize`: + +```assembly +_$sST3sumSz7ElementRpzrlEAASdyFSaySiG_Tg5: + mov rax, qword ptr [rdi + 16] + xorpd xmm0, xmm0 + test rax, rax + je .LBB1_3 + xor ecx, ecx +.LBB1_2: + cvtsi2sd xmm1, qword ptr [rdi + 8*rcx + 32] + inc rcx + addsd xmm0, xmm1 + cmp rax, rcx + jne .LBB1_2 +.LBB1_3: + ret +``` + +Now consider some code that places an optimization barrier between the call site and the concrete type: + +```swift +protocol Summable: Sequence where Element: BinaryInteger { } +extension Array: Summable where Element: BinaryInteger { } + +var summable: any Summable + +// later, when summable has been populated with an array of some kind of integer +let result = summable.sum() +``` + +The compiler now has no way of knowing what type `Summable` is at the call site – neither the sequence type, nor the element type. So its only option is to execute the fully unspecialized code. Instead of advancing a pointer over a buffer and loading values directly from memory, it must iterate the sequence by first calling `makeIterator()` and then calling `next()`, unwrapping optional values until it reaches `nil`. And instead of using a single instruction to convert an `Int` to a `Double`, it must call the generic initializer on `Double` that uses methods on the `BinaryInteger` protocol to convert the value into floating point representation. Unsurprisingly, this code path will be about 2 orders of magnitude slower than the specialized version. This is true even if the actual type being summed ends up being `[Int]` at runtime. + +A similar situation would occur when `sum` is a generic function in a binary framework which has not provided an `@inlinable` implementation. While the Swift compiler might know the types at the call site, it has no ability to generate a specialized version because it cannot see the implementation in order to generate it. So it must call the unspecialized version of `sum`. + +The best way to avoid this situation is just to avoid type erasure in the first place. This is why concrete types and generics should be preferred whenever practical over existential types for performance-sensitive code. But sometimes type erasure, particularly for heterogenous storage, is necessary. + +Similarly, ABI-stable binary framework authors do not always want to expose their implementations to the caller, as this tends to generate a very large and complex ABI surface that cannot be easily changed later. It also prevents upgrades of binary implementations without recompiling the caller, allowing for e.g. an operating system to fix bugs or security vulnerabilities without an app needing to be recompiled. + +## Proposed solution + +A new attribute, `@specialized`, will allow the author of a function to cause the compiler to generate specializations of that function. In the body of the unspecialized version, the types are first checked to see if they are of one of the specialized types. If they are, the specialized version will be called. + +So in our example above: + +```swift +extension Sequence where Element: BinaryInteger { + @specialized(where Self == [Int]) + func sum() -> Double { + reduce(0) { $0 + Double($1) } + } +} +``` + +A specialized version of `[Int].sum` will be generated in the same way as if it had been specialized for a callsite. And inside the unspecialized generic code, the additional check-and-redispatch logic will be inserted at the start of the function. + +Doing this restores the performance of the specialized version of `sum` (less a check and branch of the calling type) even when using the existential `any Summable` type. + +## Detailed design + +The `@specialized` attribute can be placed on any generic function. It takes an argument with the same syntax as a `where` clause for a generic signature. + +Multiple specializations can be listed: + +```swift +extension Sequence where Element: BinaryInteger { + @specialized(where Self == [Int]) + @specialized(where Self == [UInt32]) + @specialized(where Self == [Int8]) + func sum() -> Double { + reduce(0) { $0 + Double($1) } + } +} +``` + +Within the unspecialized function, redispatch will be based on the _exact_ type. That is, in pseudocode: + +```swift +extension Sequence where Element: BinaryInteger { + @specialized(where Self == [Int]) + @specialized(where Self == [Int8]) + func sum() -> Double { + if Self.self == [Int].self { + + } else if Self.self == [Int8].self { + + } else { + reduce(0) { $0 + Double($1) } + } + } +} +``` + +(Note that this is just for illustrative purposes, the actual dispatch mechanism to the specific specialization may differ and change over time) + +No attempt to implicitly convert the type is made (so for example, a specialization for `Int?` would not be executed when called on `Int`), nor is a specialization for a superclass executed when called with the generic type of the subclass. + +These specializations are the same as the ones generated by the caller. They replace the dynamic dispatch of the unspecialized generic protocol with static dispatch to the methods on the concrete type, and this in turn unlocks other optimizations within the specialized function. Note that within these specializations, no different overload resolution takes place now that the concrete type is known. That is, the functions being called will be only those made available via the protocol witness table. This ensures that there is no _semantic_ effect from using `@specialized`, only a change in performance. + +As well as protocol extensions, it can also be used on extensions of generic types, on computed properties (note, it must be put on `get` explicitly, not on the shorthand where it is elided), and on free functions: + +```swift +extension Array where Element: BinaryInteger { + @specialized(where Element == Int) + func sum() -> Double { + reduce(0) { $0 + Double($1) } + } + + var product: Double { + @specialized(where Element == Int8) + get { reduce(1) { $0 * Double($1) } } + } +} + +@specialized(where T == Int) +func sum(_ numbers: T...) -> Double { + numbers.reduce(0) { $0 + Double($1) } +} +``` + +The `where` clause must fully specialize all the generic placeholders of the types in the function signature. In the case of a protocol extension, as seen above, that includes specifying `Self`. All placeholders must be fully specified even if they do not appear to be used in the function body: + +```swift +extension Dictionary where Value: BinaryInteger { + // error: Too few generic parameters are specified in 'specialize' attribute (got 1, but expected 2) + // note: Missing equality constraint for 'Key' in 'specialize' attribute + @specialized(where Value == Int) + func sum() -> Double { + values.reduce(0) { $0 + Double($1) } + } +} +``` + +Bear in mind that even when not explicitly used in the body, they may be used implicitly. For example, in calculating the location of stored properties. Depending on how `Dictionary` is laid out, it may be important to know the size of the `Key` even if key values aren't used. But even if there is absolutely no use of a generic type in the body being specialized, the type must be explicitly specified. This requirement could be loosened in future, see "Partial Specialization" in future directions. + +Where multiple placeholders need to be specified separately, they can be separated by commas, such as in the fix for the above example: + +```swift +extension Dictionary where Value: BinaryInteger { + @specialized(where Value == Int, Key == Int) + func sum() -> Double { + values.reduce(0) { $0 + Double($1) } + } +} +``` + +## Source compatibility + +The addition or removal of explicit specializations has no impact on the caller, and is opt-in. As such it has no source compatability implications. + +## ABI compatibility + +This proposal covers only _internal_ specializations, dispatched to from within the implementation of an unspecialized generic function. As such it has no impact on ABI. It can be applied to existing ABI-stable functions in libraries built for distribution, and can be removed later without ABI impact. Generic functions can continue to be inlinable to be specialized by the caller, and the code to dispatch to specialized versions will not appear in the inlinable code emitted into the swift interface file. + +A future direction where explicit specializations are exposed as ABI appears in Future Directions. + +## Implications on adoption + +Explicit specializations impose no new runtime requirements on either the caller or the called function, so use of them can be back-deployed to earlier Swift runtimes. + +## Future directions + +### Partial Specialization + +The need to fully specify every type when using `@specialized` is somewhat limiting. For example, in the `Dictionary` example above, no reference is made in the code to the `Key` type in summing the values. Having to explicitly state the concrete type for `Key` means you need separate specializations for `[String:Int]`, `[Int:Int]` and so on. + +Even in cases where dynamic dispatch through the protocol is still required for the unspecialized aspects (such as determining where to find the values in a dictionary's layout), this might not be in the hot part of the function and so partial specialization might be the better trade-off. + +Closely related to partial specialization is the ability to specialize once for all particular type layouts. In the `Dictionary.sum` example, it might only matter what the size of the key type is in order to efficiently iterate the values. + +Another example of this is `Array.append`. There is only one single implementation needed for appending an `AnyObject`-constrained to an array. The append operation needs to retain the object, but does not need any other properties. So two class instances could be appended using the same method irrespective of their actual type. Similar techniques could be used to provide shared specializations for `BitwiseCopyable` types, possibly with the addition of a supplied size argument to avoid having to use value witness calls. + +### Making Symbols for Specializations Publicly Available + +This proposal keeps entry points for explicit specializations internal only. Callers outside the module call the generic version of the function, which redispatches to the specialized version. For ABI-stable frameworks, a useful future direction would be to make these symbols publicly available and listed in the `.swiftinterface` file, for callers to link to directly. + +Doing this allows ABI-stable framework authors to expose specializations without exposing the full implementation details of a function as inlinable code. While adding a public specialized symbol to a framework is ABI, it is a much more limited ABI surface compared to providing an inlinable implementation, which requires any future changes to a type to consider the previous inlinable code's behavior to ensure it remains compatible forever in older binaries. A specialized entry point could be updated in a framework to fix a bug without recompiling the caller. Specializations in binary frameworks also have the benefit of avoiding duplication of code into the caller. + +### Requiring Specialization + +A related feature is the ability to _require_ specialization either at the call site, or on a protocol. Specialization does not always have to happen at the call site even when it could – it remains an optimization (albeit a very aggressively applied one currently). If the specializatino does not happen, it would be useful to force the compiler to override its heuristics, similar to forcing linlining of a long but critical function. + +There are also protocols that are only meant to be used in specialized form in optimized builds. Arguably `BinaryInteger` is one of them. It may be worth exploring in these cases an annotation to indicate this to the caller, either via a compile-time errror/warning, or a runtime error. + +### Marking types or extensions as `@specialized` + +It may desirable to mark a number of generic functions "en-mass" as specialized for a particular type, by annotating either the type or an extension grouping. + +This would mostly be just sugar for annotating each function individually. The exception could be annotation of classes or protocols where an entire specialized witness or vtable could then be passed around and used from other unspecialized functions. + +### Tooling Directions + +This proposal only outlines language syntax the developer can use to instruct the compiler to emit a specialized version of a function. Complimentary to this is development of tooling to identify profitable specializations to add, for example by profiling typical usage of an app. It should be noted that specialization is only required in highly performance sensitive code. In many cases, explicit specialization will have little impact on overall performance while increasing binary size. + +The current implementation requires the attribute to be attached directly to the function being specialized. Tooling to produce specializations would benefit from an additional syntax that could be added in a separate file, or even into a separately compiled binary. + +## Alternatives considered + +Many alternatives to this proposal – such as whole-program analysis to determine prespecializations automatically – are complimentary to this technique. Even in the presence of better optimizer heroics, there are still benefits to having explicit control over specialization. + +This proposal takes as a given Swift's current dispatch mechanisms. Some alternatives to this proposal tend to end up requiring fundamental changes to Swift's core generics implementation, and are therefore out of scope for this proposal. + +### Execution of custom functions based on type + +Sometimes, it is desirable to execute not a compiler-generated specialization of a function, but a very different implementation based on the type: + +``` +extension Sequence where Element: BinaryInteger { + func sum() -> Double { + if let arrayOfInt = self as? [Int] { + arrayOfInt.handVectorizedImplementation() + } else { + reduce(0) { $0 + Double($1) } + } + } +} +``` + +The specializations created by this proposal are entirely referentially transparent, using Swift's protocol dispatch semantics to ensure the only (reasonably )observable difference is how quickly the code runs. They are just optimizations where dynamic dispatch is replaced by static, code inlined, low-level optimizations applied to that code etc. + +This is distinct from "when it's this type run this code, when it's that type, run that code", where this code and that code might do very different things semantically. You might not mean for them to differ semantically, but they can. + +This is an interesting area to explore, but it's important to be clear that it's a very different feature (and hence not included in future directions). + diff --git a/proposals/0461-async-function-isolation.md b/proposals/0461-async-function-isolation.md new file mode 100644 index 0000000000..fef2459dc9 --- /dev/null +++ b/proposals/0461-async-function-isolation.md @@ -0,0 +1,1137 @@ +# Run nonisolated async functions on the caller's actor by default + +* Proposal: [SE-0461](0461-async-function-isolation.md) +* Authors: [Holly Borla](https://github.com/hborla), [John McCall](https://github.com/rjmccall) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.2)** +* Vision: [Improving the approachability of data-race safety](/visions/approachable-concurrency.md) +* Upcoming Feature Flag: `NonisolatedNonsendingByDefault` +* Previous Proposal: [SE-0338](0338-clarify-execution-non-actor-async.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-inherit-isolation-by-default-for-async-functions/74862)) ([first review](https://forums.swift.org/t/se-0461-run-nonisolated-async-functions-on-the-callers-actor-by-default/77987)) ([acceptance with focused re-review](https://forums.swift.org/t/accepted-with-modifications-and-focused-re-review-se-0461-run-nonisolated-async-functions-on-the-callers-actor-by-default/78920)) ([second review](https://forums.swift.org/t/focused-re-review-se-0461-run-nonisolated-async-functions-on-the-callers-actor-by-default/78921)) ([second acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0461-run-nonisolated-async-functions-on-the-caller-s-actor-by-default/79117)) + +## Introduction + +Swift's general philosophy is to prioritize safety and ease-of-use over +performance, while still providing tools to write more efficient code. The +current behavior of nonisolated async functions prioritizes main actor +responsiveness at the expense of usability. + +This proposal changes the behavior of nonisolated async functions to run on +the caller's actor by default, and introduces an explicit way to state that an +async function always switches off of an actor to run. + +## Table of Contents + +- [Motivation](#motivation) +- [Proposed solution](#proposed-solution) +- [Detailed design](#detailed-design) + - [`nonisolated(nonsending)` functions](#nonisolatednonsending-functions) + - [`@concurrent` functions](#concurrent-functions) + - [Task isolation inheritance](#task-isolation-inheritance) + - [`#isolation` macro expansion](#isolation-macro-expansion) + - [Isolation inference for closures](#isolation-inference-for-closures) + - [Function conversions](#function-conversions) + - [Non-`@Sendable` function conversions](#non-sendable-function-conversions) + - [Region isolation rules](#region-isolation-rules) + - [Executor switching](#executor-switching) + - [Dynamic actor isolation APIs in async contexts](#dynamic-actor-isolation-apis-in-async-contexts) + - [Import-as-async heuristic](#import-as-async-heuristic) +- [Source compatibility](#source-compatibility) +- [ABI compatibility](#abi-compatibility) +- [Implications on adoption](#implications-on-adoption) +- [Alternatives considered](#alternatives-considered) + - [Changing isolation inference behavior to implicitly capture isolated parameters](#changing-isolation-inference-behavior-to-implicitly-capture-isolated-parameters) + - [Use `nonisolated` instead of a separate `@concurrent` attribute](#use-nonisolated-instead-of-a-separate-concurrent-attribute) + - [Alternative syntax choices](#alternative-syntax-choices) + - [No explicit spelling for `nonisolated(nonsending)`](#no-explicit-spelling-for-nonisolatednonsending) + - [Justification for `@concurrent`](#justification-for-concurrent) + - [`@executor`](#executor) + - [`@isolated`](#isolated) + - [`nonisolated` argument spelling](#nonisolated-argument-spelling) + - [Deprecate `nonisolated`](#deprecate-nonisolated) + - [Don't introduce a type attribute for `@concurrent`](#dont-introduce-a-type-attribute-for-concurrent) +- [Revisions](#revisions) + +## Motivation + +[SE-0338: Clarify the Execution of Non-Actor-Isolated Async Functions][SE-0338] +specifies that nonisolated async functions never run on an actor's executor. +This design decision was made to prevent unnecessary serialization and +contention for the actor by switching off of the actor to run the nonisolated +async function, and any new tasks it creates that inherit isolation. The actor +is then free to make forward progress on other work. This behavior is +especially important for preventing unexpected overhang on the main actor. + +This decision has a number of unfortunate consequences. + +**`nonisolated` is difficult to understand.** There is a semantic difference +between the isolation behavior of nonisolated synchronous and asynchronous +functions; nonisolated synchronous functions always run on the caller's actor, +while nonisolated async functions always switch off of the caller's actor. This +means that sendable checking applies to arguments and results of nonisolated +async functions, but not nonisolated synchronous functions. + +For example: + +```swift +class NotSendable { + func performSync() { ... } + func performAsync() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + x.performSync() // okay + + await x.performAsync() // error + } +} +``` + +The call to `performAsync` from the actor results in a data-race safety error +because the call leaves the actor to run the function. This frees up the actor +to run other tasks, but those tasks can access the non-`Sendable` value `x` +concurrently with the call to `performAsync`, which risks a data race. + +It's confusing that the two calls to methods on `NotSendable` have different +isolation behavior, because both methods are `nonisolated`. + +**Async functions that run on the caller's actor are difficult to express.** +It's possible to write an async function that does not leave an actor to run +using isolated parameters and the `#isolation` macro as a default argument: + +```swift +class NotSendable { + func performAsync( + isolation: isolated (any Actor)? = #isolation + ) async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + await x.performAsync() // okay + } +} +``` + +This resolves the data-race safety error because `performAsync` now runs on the +actor. However, this isn't an obvious solution, it's onerous boilerplate to +write, and the default argument is lost if the method is used in a higher-order +manner. + +**It's easy to write invalid async APIs.** If the `performAsync` method were in +a library that the programmer doesn't own, it's not possible to workaround the +data-race safety error without using unsafe opt outs. It's common for library +authors to mistakenly vend an API like this, because the data-race safety error +only manifests when calling the API from an actor. + +The concurrency library itself has made this mistake, and many of the async +APIs in the concurrency library have since transitioned to inheriting the +isolation of the caller using isolated parameters; see [SE-0421][SE-0421] for +an example. + +**It's difficult to write higher-order async APIs.** Consider the following +async API which provides a `with`-style method for acquiring a resource and +performing a scoped operation: + +```swift +public struct Resource { + internal init() {} + internal mutating func close() async {} +} + +public func withResource( + isolation: isolated (any Actor)? = #isolation, + _ body: (inout Resource) async -> Return +) async -> Return { + var resource = Resource() + let result = await body(&resource) + await resource.close() + return result +} +``` + +Despite `withResource` explicitly running on the caller's actor by default, +there's no way to specify that the async `body` function value should also run +in the same context. The compiler treats the async function parameter as +switching off of the actor to run, so it requires sendable checking on the +arguments and results. This particular example happens to pass a value in a +disconnected region to `body`, but passing an argument in the actor's region +would be invalid. In most cases, the call doesn't cross an isolation boundary +at runtime, because the function type is not `@Sendable`, so calling the API +from an actor-isolated context and passing a trailing closure will treat the +closure as isolated to the same actor. This sendable checking is often a source +of false positives that make higher-order async APIs extremely difficult to +write. The checking can't just be eliminated, because it's valid to pass a +nonisolated async function that will switch off the actor to run, which would +lead to a data race if actor-isolated state is passed to the `body` parameter. + +Moreover, the above explanation of isolation rules for async closures is +extremely difficult to understand; the default isolation rules are too +complicated. + +## Proposed solution + +This proposal changes the execution semantics of nonisolated async functions +to always run on the caller's actor by default. This means that nonisolated +functions will have consistent execution semantics by default, regardless of +whether the function is synchronous or asynchronous. + +This change makes the following example from the motivation section +valid, because the call to `x.performAsync()` does not cross an isolation +boundary: + +```swift +class NotSendable { + func performSync() { ... } + func performAsync() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + x.performSync() // okay + + await x.performAsync() // okay + } +} +``` + +Changing the default execution semantics of async functions can change the +behavior of existing code, so the change is gated behind the +`NonisolatedNonsendingByDefault` upcoming feature flag. To help stage in the new +behavior, new syntax can be used to explicitly specify the +execution semantics of an async function in any language mode. + +A new `nonsending` argument can be written with `nonisolated` to indicate +that by default, the argument and result values are not sent over an +isolation boundary when the function is called: + +```swift +class NotSendable { + nonisolated(nonsending) + func performAsync() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + await x.performAsync() // okay + } +} +``` + +The `@concurrent` attribute is an explicit spelling for the behavior of +async functions in language modes <= Swift 6. `@concurrent` indicates +that calling the function always switches off of an actor to run, so +the function will run concurrently with other tasks on the caller's actor: + +```swift +class NotSendable { + @concurrent + func alwaysSwitch() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + await x.alwaysSwitch() // error + } +} +``` + +`@concurrent` is the current default for nonisolated async +functions. `nonisolated(nonsending)` will become the default for async functions +when the `NonisolatedNonsendingByDefault` upcoming feature is enabled. + +## Detailed design + +The sections below will explicitly use `@concurrent` and +`nonisolated(nonsending)` to demonstrate examples that will behave consistently +independent of upcoming features or language modes. However, note that the +end state under the `NonisolatedNonsendingByDefault` upcoming feature will mean +that `(nonsending)` is not necessary to explicitly write, and +`@concurrent` will likely be used sparingly because it has far +stricter data-race safety requirements. + +### `nonisolated(nonsending)` functions + +Async functions annotated with `nonisolated(nonsending)` will always run on the +caller's actor: + +```swift +class NotSendable { + func performSync() { ... } + + nonisolated(nonsending) + func performAsync() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + x.performSync() // okay + + await x.performAsync() // okay + } +} +``` + +In the above code, the call to `x.performAsync()` continues running on the +`self` actor instance. The code does not produce a data-race safety error, +because the `NotSendable` instance `x` does not leave the actor. In other +words, the arguments are not sent across an isolation boundary when calling +`performAsync` by default. + +This behavior is accomplished by implicitly passing an optional actor parameter +to the async function. The function will run on this actor's executor. See the +[Executor switching](#executor-switching) section for more details on why the +actor parameter is necessary. + +The type of an `nonisolated(nonsending)` function declaration is an +`nonisolated(nonsending)` function type. For example: + +```swift +class NotSendable { ... } + +@MainActor let global: NotSendable = .init() + +nonisolated(nonsending) +func runOnActor(ns: NotSendable) async {} + +@MainActor +func callSendableClosure() async { + // the type of 'closure' is '@Sendable nonisolated(nonsending) (NotSendable) -> Void' + let closure = runOnActor(ns:) + + let ns = NotSendable() + await closure(ns) // okay + await closure(global) // okay +} + +callSendableClosure() +``` + +In the above code, the calls to `closure` from `callSendableClosure` run on the +main actor, because `closure` is `nonisolated(nonsending)` and `callSendableClosure` +is main actor isolated. + +### `@concurrent` functions + +Async functions can be declared to always switch off of an actor to run using +the `@concurrent` attribute: + +```swift +struct S: Sendable { + @concurrent + func alwaysSwitch() async { ... } +} +``` + +Only (implicitly or explicitly) `nonisolated` functions can be marked with +`@concurrent`; it is an error to use this attribute with +an isolation other than `nonisolated`, including global actors, isolated +parameters, and `@isolated(any)`: + +```swift +actor MyActor { + var value = 0 + + // error: '@concurrent' can only be used with 'nonisolated' methods + @concurrent + func isolatedToSelf() async { + value += 1 + } + + @concurrent + nonisolated func canRunAnywhere() async { + // cannot access 'value' or other actor-isolated state + } +} +``` + +`@concurrent` can be used together with `@Sendable` or `sending`. + +`@concurrent` cannot be applied to synchronous +functions. This is an artificial limitation that could later be lifted if use +cases arise. + +The type of an `@concurrent` function declaration is an +`@concurrent` function type. Details on function conversions are +covered in a [later section](#function-conversions). + +When an `@concurrent` function is called from a context that can +run on an actor, including `nonisolated(nonsending)` functions or actor-isolated +functions, sendable checking is performed on the argument and result values. +Either the argument and result values must have a type that conforms to +`Sendable`, or the values must be in a disconnected region so they can be sent +outside of the actor: + +```swift +class NotSendable {} + +@concurrent +func alwaysSwitch(ns: NotSendable) async { ... } + +actor MyActor { + let ns: NotSendable = .init() + + func callConcurrent() async { + await alwaysSwitch(ns: ns) // error + + let disconnected = NotSendable() + await alwaysSwitch(ns: disconnected) // okay + } +} +``` + +### Task isolation inheritance + +Unstructured tasks created in nonisolated functions never run on an actor +unless explicitly specified. This behavior is consistent for all nonisolated +functions, including synchronous functions, `nonisolated(nonsending)` async +functions, and `@concurrent` async functions. + +For example: + +```swift +class NotSendable { + var value = 0 +} + +nonisolated(nonsending) +func createTask(ns: NotSendable) async { + Task { + // This task does not run on the same actor as `createTask` + + ns.value += 1 // error + } +} +``` + +Capturing `ns` in the unstructured task is an error, because the value can +be used concurrently between the caller of `createTask` and the newly +created task. + +This decision is deliberate to match the semantics of unstructured task +creation in nonisolated synchronous functions. Note that unstructured task +creation in methods with isolated parameters also do not inherit isolation +if the isolated parameter is not explicitly captured. + +### `#isolation` macro expansion + +Uses of the `#isolation` macro will expand to the implicit isolated parameter. +For example, the following program prints `Optional(Swift.MainActor)`: + +```swift +nonisolated func printIsolation() async { + let isolation = #isolation + print(isolation) +} + +@main +struct Program { + // implicitly isolated to @MainActor + static func main() async throws { + await printIsolation() + } +} +``` + +This behavior allows async function calls that use `#isolation` as a default +isolated argument to run on the same actor when called from an +`nonisolated(nonsending)` function. For example, the following code is valid because +the call to `explicitIsolationInheritance` does not cross an isolation +boundary: + +```swift +class NotSendable { ... } + +func explicitIsolationInheritance( + ns: NotSendable, + isolation: isolated (any Actor)? = #isolation +) async { ... } + +nonisolated(nonsending) +func printIsolation(ns: NotSendable) async { + await explicitIsolationInheritance(ns: ns) // okay +} +``` + +Note that this introduces a semantic difference compared to synchronous +nonisolated functions, where there is no implicit isolated parameter and +`#isolation` always expands to `nil`. For example, the following program prints +`nil`: + +```swift +func printIsolation() { + let isolation = #isolation + print(isolation) +} + +@main +struct Program { + // implicitly isolated to @MainActor + static func main() async throws { + printIsolation() + } +} +``` + +In an `@concurrent` function, the `#isolation` macro expands to +`nil`. + +### Isolation inference for closures + +Note that the rules in this section are not new with this proposal. However, +these rules have not been specified in any other proposal, and they are +necessary for understanding the execution semantics of async closures. + +The isolation of a closure can be explicitly specified with a type annotation +or in the closure signature. If no isolation is specified, the inferred +isolation for a closure depends on two factors: +1. The isolation of the context where the closure is formed. +2. Whether the contextual type of the closure is `@Sendable` or `sending`. + +If the contextual type of the closure is neither `@Sendable` nor `sending`, the +inferred isolation of the closure is the same as the enclosing context: + +```swift +class NotSendable { ... } + +@MainActor +func closureOnMain(ns: NotSendable) async { + let syncClosure: () -> Void = { + // inferred to be @MainActor-isolated + + // capturing main-actor state is okay + print(ns) + } + + // runs on the main actor + syncClosure() + + let asyncClosure: (NotSendable) async -> Void = { + // inferred to be @MainActor-isolated + + print($0) + } + + // runs on the main actor; + // passing main-actor state is okay + await asyncClosure(ns) +} +``` + +If the type of the closure is `@Sendable` or if the closure is passed to a +`sending` parameter, the closure is inferred to be `nonisolated`. + +The closure is also inferred to be `nonisolated` if the enclosing context +has an isolated parameter (including `self` in actor-isolated methods), and +the closure does not explicitly capture the isolated parameter. This is done to +avoid implicitly capturing values that are invisible to the programmer, because +this can lead to reference cycles. + +### Function conversions + +Function conversions can change isolation. You can think of this like a +closure with the new isolation that calls the original function, asynchronously +if necessary. For example, a function conversion from one global-actor-isolated +type to another can be conceptualized as an async closure that calls the +original function with `await`: + +```swift +@globalActor actor OtherActor { ... } + +func convert( + closure: @OtherActor () -> Void +) { + let mainActorFn: @MainActor () async -> Void = closure + + // The above conversion is the same as: + + let mainActorEquivalent: @MainActor () async -> Void = { + await closure() + } +} +``` + +A function conversion that crosses an isolation boundary must only +pass argument and result values that are `Sendable`; this is checked +at the point of the function conversion. For example, converting an +actor-isolated function type to a `nonisolated` function type requires +that the argument and result types conform to `Sendable`: + +```swift +class NotSendable {} +actor MyActor { + var ns = NotSendable() + + func getState() -> NotSendable { ns } +} + +func invalidResult(a: MyActor) async -> NotSendable { + let grabActorState: nonisolated(nonsending) () async -> NotSendable = a.getState // error + + return await grabActorState() +} +``` + +In the above code, the conversion from the actor-isolated method `getState` +to a `nonisolated(nonsending)` function is invalid, because the +result type does not conform to `Sendable` and the result value could be +actor-isolated state. The `nonisolated` function can be called from +anywhere, which would allow access to actor state from outside the actor. + +Not all function conversions cross an isolation boundary, and function +conversions that don't can safely pass non-`Sendable` arguments and results. +For example, a `nonisolated(nonsending)` function type can always be converted to an +actor-isolated function type, because the `nonisolated(nonsending)` function will +simply run on the actor: + +```swift +class NotSendable {} + +nonisolated(nonsending) +func performAsync(_ ns: NotSendable) async { ... } + +@MainActor +func convert(ns: NotSendable) async { + // Okay because 'performAsync' will run on the main actor + let runOnMain: @MainActor (NotSendable) async -> Void = performAsync + + await runOnMain(ns) +} +``` + +The following table enumerates each function conversion rule and specifies +which function conversions cross an isolation boundary. Function conversions +that cross an isolation boundary require `Sendable` argument and result types, +and the destination function type must be `async`. Note that the function +conversion rules for synchronous `nonisolated` functions and asynchronous +`nonisolated(nonsending)` functions are the same; they are both +represented under the "Nonisolated" category in the table: + +| Old isolation | New isolation | Crosses Boundary | +|----------------------|------------------------|------------------| +| Nonisolated | Actor isolated | No | +| Nonisolated | `@isolated(any)` | No | +| Nonisolated | `@concurrent` | Yes | +| Actor isolated | Actor isolated | Yes | +| Actor isolated | `@isolated(any)` | No | +| Actor isolated | Nonisolated | Yes | +| Actor isolated | `@concurrent` | Yes | +| `@isolated(any)` | Actor isolated | Yes | +| `@isolated(any)` | Nonisolated | Yes | +| `@isolated(any)` | `@concurrent` | Yes | +| `@concurrent` | Actor isolated | Yes | +| `@concurrent` | `@isolated(any)` | No | +| `@concurrent` | Nonisolated | Yes | + +#### Non-`@Sendable` function conversions + +If a function type is not `@Sendable`, only one isolation domain can +reference the function at a time, and calls to the function may never +happen concurrently. These rules for non-`Sendable` types are enforced +through region isolation. When a non-`@Sendable` function is converted +to an actor-isolated function, the function value itself is merged into the +actor's region, along with any non-`Sendable` function captures: + +```swift +class NotSendable { + var value = 0 +} + +nonisolated(nonsending) +func convert(closure: () -> Void) async { + let ns = NotSendable() + let disconnectedClosure = { + ns.value += 1 + } + let valid: @MainActor () -> Void = disconnectedClosure // okay + await valid() + + let invalid: @MainActor () -> Void = closure // error + await invalid() +} +``` + +The function conversion for the `invalid` variable is an error because the +non-`Sendable` captures of `closure` could be used concurrently from the caller +of `convert` and the main actor. + +Converting a non-`@Sendable` function type to an actor-isolated one is invalid +if the original function must leave the actor in order to be called: + +```swift +nonisolated(nonsending) +func convert( + fn1: @escaping @concurrent () async -> Void, +) async { + let fn2: @MainActor () async -> Void = fn1 // error + + await withDiscardingTaskGroup { group in + group.addTask { await fn2() } + group.addTask { await fn2() } + } +} +``` + +In general, a conversion from an actor-isolated function type to a +`nonisolated` function type crosses an isolation boundary, because the +`nonisolated` function type can be called from an arbitrary isolation domain. +However, if the conversion happens on the actor, and the new function type is +not `@Sendable`, then the function must only be called from the actor. In this +case, the function conversion is allowed, and the resulting function value +is merged into the actor's region: + +```swift +class NotSendable {} + +@MainActor class C { + var ns: NotSendable + + func getState() -> NotSendable { ns } +} + +func call(_ closure: () -> NotSendable) -> NotSendable { + return closure() +} + +@MainActor func onMain(c: C) { + // 'result' is in the main actor's region + let result = call(c.getState) +} +``` + +### Region isolation rules + +`nonisolated(nonsending)` functions have the same region isolation rules as +synchronous `nonisolated` functions. When calling an `nonisolated(nonsending)` +function, all non-`Sendable` parameter and result values are merged into +the same region, but they are only merged into the caller's actor region if +one of those non-`Sendable` values is already in the actor's region. + +For example: + +```swift +class NotSendable {} + +nonisolated(nonsending) +func identity(_ t: T) async -> T { + return t +} + +actor MyActor { + func isolatedToSelf() async -> sending NotSendable { + let ns = NotSendable() + return await identity(ns) + } +} +``` + +The above code is valid; the implementation of `identity` can't access the +actor's state unless isolated state is passed in via one of the parameters. +Note that this code would be invalid if `identity` accepted an isolated +parameter, because the non-`Sendable` parameters and results would always be +merged into the actor's region. + +This proposal allows you to access `#isolation` in the implementation of an +`nonisolated(nonsending)` function for the purpose of forwarding it along to a +method that accepts an `isolated (any Actor)?`. This is still safe, because +there's no way to access the actor's isolated state via the `Actor` protocol, +and dynamic casting to a concrete actor type will not result in a value that +the function is known to be isolated to. + +### Executor switching + +Async functions switch executors in the implementation when entering the +function, and after any calls to other async functions. Note that synchronous +functions do not have the ability to switch executors. If a call to a +synchronous function crosses an isolation boundary, the call must happen in an +async context and the executor switch happens at the caller. + +`@concurrent` async functions switch to the generic executor, and +all other async functions switch to the isolated actor's executor. + +```swift +@MainActor func runOnMainExecutor() async { + // switch to main actor executor + + await runOnGenericExecutor() + + // switch to main actor executor +} + +@concurrent func runOnGenericExecutor() async { + // switch to generic executor + + await Task { @MainActor in + // switch to main actor executor + + ... + }.value + + // switch to generic executor +} +``` + +`nonisolated(nonsending)` functions will switch to the executor of the implicit +actor parameter passed from the caller instead of switching to the generic +executor: + +```swift +@MainActor func runOnMainExecutor() async { + // switch to main actor executor + ... +} + +class NotSendable { + var value = 0 +} + +actor MyActor { + let ns: NotSendable = .init() + + func callNonisolatedFunction() async { + await inheritIsolation(ns) + } +} + +nonisolated func inheritIsolation(_ ns: NotSendable) async { + // switch to isolated parameter's executor + + await runOnMainExecutor() + + // switch to isolated parameter's executor + + ns.value += 1 +} +``` + +For most calls, the switch upon entering the function will have no effect, +because it's already running on the executor of the actor parameter. + +A task executor preference can still be used to configure where a nonisolated +async function runs. However, if the nonisolated async function was called from +an actor with a custom executor, the task executor preference will not apply. +Otherwise, the code will risk a data-race, because the task executor preference +does not apply to actor-isolated methods with custom executors, and the +nonisolated async method can be passed mutable state from the actor. + +### Dynamic actor isolation APIs in async contexts + +Because nonisolated async functions may now execute on a specific actor at +runtime, the APIs in the Concurrency library for enforcing actor isolation +assertions and preconditions are now useful in these contexts. As such, the +`noasync` attribute will be removed from `assertIsolated`, `assumeIsolated`, +and `preconditionIsolated` on `Actor` and `MainActor`. + +### Import-as-async heuristic + +Nonisolated functions imported from Objective-C that match the import-as-async +heuristic from [SE-0297: Concurrency Interoperability with Objective-C][SE-0297] +will implicitly be imported as `nonisolated(nonsending)`. Objective-C async +functions already have bespoke code generation that continues running on +the caller's actor to match the semantics of the original completion handler +function, so `nonisolated(nonsending)` already better matches the semantics of these +imported `async` functions. This change will eliminate many existing data-race +safety issues that happen when calling an async function on an Objective-C +class from the main actor. Because the only effect of this change is +eliminating concurrency diagnostics -- the runtime behavior of the code will +not change -- it will not be gated behind the upcoming feature. + +## Source compatibility + +This proposal changes the semantics of nonisolated async functions when the +upcoming feature flag is enabled. Without the upcoming feature flag, the default +for nonisolated async functions is `@concurrent`. When the upcoming +feature flag is enabled, the default for nonisolated async functions changes to +`nonisolated(nonsending)`. This applies to both function declarations and function +values that are nonisolated (either implicitly or explicitly). + +Changing the default execution semantics of nonisolated async functions has +minor source compatibility impact if the implementation calls an +`@concurrent` function and passes non-Sendable state in the actor's +region. In addition to the source compatibility impact, the change can also +regress performance of existing code if, for example, a specific async function +relied on running off of the main actor when called from the main actor to +maintain a responsive UI. + +To avoid breaking source compatibility or silently changing behavior of +existing code, this change must be gated behind an upcoming feature flag. +However, unlike most other changes gated behind upcoming feature flags, this +change allows writing code that is valid with and without the upcoming feature +flag, but means something different. Many programmers have internalized the +SE-0338 semantics, and making this change several years after SE-0338 was +accepted creates an unfortunate intermediate state where it's difficult to +understand the semantics of a nonisolated async function without understanding +the build settings of the module you're writing code in. + +To make it easy to discover what kind of async function you're working with, +SourceKit will surface the implicit `nonisolated(nonsending)` or `@concurrent` +attribute for IDE inspection features like Quick Help in Xcode and Hover in +VSCode. To ease the transition to the upcoming feature flag, [migration +tooling][adoption-tooling] will provide fix-its to preserve behavior by +annotating nonisolated async functions with `@concurrent`. + +## ABI compatibility + +Adopting the semantics to run on the caller's actor for an existing nonisolated +async function is an ABI change, because the caller's actor must be passed as +a parameter. However, a number of APIs in the concurrency library have staged +in similar changes using isolated parameters and `#isolation`, and it may be +possible to offer tools to do this transformation automatically for resilient +libraries that want to adopt this behavior. + +For example, if a nonisolated async function is ABI-public and is available +earlier than a version of the Swift runtime that includes this change, the +compiler could emit two separate entry points for the function: + +```swift +@_alwaysEmitIntoClient +public func myAsyncFunc() async { + // original implementation +} + +@concurrent +@_silgen_name(...) // to preserve the original symbol name +@usableFromInline +internal func abi_myAsyncFunc() async { + // existing compiled code will continue to always run calls to this function + // on the generic executor. + await myAsyncFunc() +} +``` + +This transformation only works if the original function implementation +can be made inlinable. + +## Implications on adoption + +`nonisolated(nonsending)` functions must accept an implicit actor parameter. This +means that adding `nonisolated(nonsending)` to a function that is actor-isolated, or +changing a function from `@concurrent` to `nonisolated(nonsending)`, is +not a resilient change. + +## Alternatives considered + +### Changing isolation inference behavior to implicitly capture isolated parameters + +The current isolation inference behavior in contexts with isolated parameters +is often surprising with respect to data-race safety. However, this proposal +does not suggest changing the rules, because implicitly capturing an isolated +parameter can lead to silently causing new memory leaks in existing code. One +potential compromise is to keep the current isolation inference behavior, and +offer fix-its to capture the actor if there are any data-race safety errors +from capturing state in the actor's region. + +### Use `nonisolated` instead of a separate `@concurrent` attribute + +It's tempting to not introduce a new attribute to control where an async +function executes, and instead control this behavior with an explicit +`nonisolated` annotation. However, this approach falls short for the following +reasons: + +1. It does not accomplish the goal of having consistent semantics for + `nonisolated` by default, regardless of whether it's applied to synchronous + or async functions. +2. This approach cuts off the future direction of allowing + `@concurrent` on synchronous functions. + +### Alternative syntax choices + +Several different options for the spelling of `nonisolated(nonsending)` +and `@concurrent` were explored. An earlier iteration of this proposal +used the same base attribute for both annotations. However, these two +annotations serve very different purposes. `@concurrent` is the long-term +right way to move functions and closures off of actors. +`nonisolated(nonsending)` is necessary for the transition to the new behavior, +but it's not a syntax that will stick around long term in Swift codebases; the +ideal end state is that this is expressed via the default behavior for +(explicitly or implicitly) `nonisolated` async functions. + +Note that it is well understood that there is no perfect syntax which will +explain the semantics without other context such as educational material or +documentation. This is true for all syntax design decisions. + +#### No explicit spelling for `nonisolated(nonsending)` + +It's reasonable to question whether `nonisolated(nonsending)` is necessary +at all given that its only purpose is transitioning to the new behavior +for async functions. An explicit spelling that has consistent behavior +independent of upcoming features and language modes is valuable when +undertaking a transition that changes the meaning of existing code. + +An explicit, transitory attribute is valuable because there will be a period of +time where it is not immediately clear from source what kind of async function +a programmer is working with. It's necessary to be able to discover that +information from source, such as by showing an inferred attribute explicitly +in SourceKit's cursor info request (surfaced by "Quick Help" in Xcode and +"Hover" in LSP / VSCode). An explicit spelling that has consistent behavior +independent of language mode is also valuable for code generation tools like +macros, so that they do not have to consider build settings to determine the +right code to generate, it's valuable for posting code snippets on the forums +during the transition period, etc. + +#### Justification for `@concurrent` + +This proposal was originally pitched using the `@concurrent` syntax, and many +reviewers surfaced objects about why `@concurrent` may be misleading, such as: + +* `@concurrent` is not the only source of concurrency; concurrency can arise from + many other things. +* The execution of an `@concurrent` function is not concurrent from the local + perspective of the current task. + +It's true that concurrency can only arise if there are multiple "impetuses" +(such as tasks or event sources) in the program that are running with different +isolation. But for the most part, we can assume that there are multiple +impetuses; and while those impetuses might otherwise share isolation, +`@concurrent` is the only isolation specification under this proposal that +guarantees that they do not and therefore forces concurrency. Indeed, we expect +that programmers will be reaching for `@concurrent` exactly for that reason: +they want the current function to run concurrently with whatever else might +happen in the process. So, this proposal uses `@concurrent` because out of the +other alternatives we explored, it best reflects the programmer's intent for +using the attribute. + +#### `@executor` + +A previous iteration of this proposal used the syntax `@execution(concurrent)` +instead of `@concurrent`. The review thread explored several variations of +this syntax, including `@executor(concurrent)` and `@executor(global)`. + +However, `@execution` or `@executor` encourages +thinking about async function semantics in terms of the lower level model of +executors and threads, and we should be encouraging programmers to think about +these semantics at the higher abstraction level of actor isolation and tasks. +Trying to understand the semantics in proposal in terms of executors can also +be misleading, both because isolation does not always map naively to executor +requests and because executors are used for other things than isolation. +For example, an `@executor(global)` function could end up running on some +executor other than the global executor via task executor preferences. + +#### `@isolated` + +An alternative to `nonisolated(nonsending)` is to use the "isolated" +terminology, such as `@isolated(caller)`. However, this approach has very +unsatisfying answers for how it interacts with `nonisolated`. There are +two options: + +1. `@isolated(caller)` must be written together with `nonisolated`, + + This approach leads to the verbose and oxymoronic spelling + `@isolated(caller) nonisolated`. Though there + exists a perfectly reasonable explanation about how `nonisolated` is the + static isolation while `@isolated(caller)` is the dynamic isolation, most + programmers do not have this deep of an understanding of actor isolation, + and they should not have to in order to make basic use of nonisolated async + functions. +2. `@isolated(caller)` implies `nonisolated` and can be written alone as an + alternative. + + This direction means that programmers would sometimes write + `nonisolated` and sometimes write `@isolated(caller)`, which is not a good + end state to be in because programmers have to learn a separate syntax for + `async` functions that accomplishes the same effect as a `nonisolated` + synchronous function. Or, if we view `@isolated(caller)` as only used for + the transition to the new behavior, then the assumption is that some day + people will remove `@isolated(caller)` if it is written in source. If + `@isolated(caller)` implies `nonisolated`, then the code could change + behavior if it's in a context where global or instance actor isolation would + otherwise be inferred. + +Going in the oppose direction, this proposal could effectively deprecate +`nonisolated` and allow you to use `@isolated(caller)` everywhere that +`nonisolated` is currently supported, including synchronous methods, stored +properties, type declarations, and extensions. This direction was not chosen +for the following reasons: + +1. This would lead to much more code churn than the current proposal. Part of + the goal of this proposal is to minimize the change to only what is absolutely + necessary to solve the major usability problem with async functions on + non-`Sendable` types, because it's painful both to transition code and to + re-learn parts of the model that have already been internalized. +2. `nonisolated` is nicer to write than `@isolated(caller)` + or any other alternative attribute + argument syntax. + +#### `nonisolated` argument spelling + +An argument to `nonisolated` is more compelling than a separate attribute +to specify that an async function runs on the caller's actor because it +defines away the problem of whether this annotation implies `nonisolated` when +written alone. + +A few different options for the argument to `nonisolated` were explored. + +**`nonisolated(nosend)`**. +`nonisolated(nosend)` effectively the same as `nonisolated(nonsending)` as +proposed, but it states that the call itself does not constitute a "send", +rather than stating that the call is not "sending" its argument and result +values over an isolation boundary. `nonisolated(nosend)` is shorter, but +`nonisolated(nonsending)` is more consistent with existing Swift naming +conventions. + +**`nonisolated(caller)`**. +`nonisolated(caller)` is meant to indicate that the function is statically +`nonisolated` and dynamically isolated to the caller. However, putting those +terms together into one `nonisolated(caller)` attribute is misleading, because +it appears the mean exactly the opposite of what it actually means; +`nonisolated(caller)` reads "not isolated to the caller". + +**`nonisolated(nonconcurrent)`**. +If `@concurrent` is applied to a function, then the function must run +concurrently with the caller's actor (assuming multiple isolated tasks +in the program). `nonconcurrent` conveys the inverse; if `nonconcurrent` is +applied to an async function, then the function must not run concurrently +with the caller's actor. However, this statement isn't quite true, because the +implementation of the function can perform work concurrently, though that work +cannot involve the non-`Sendable` parameter values. + +**`nonisolated(static)`**. +`nonisolated(static)` is meant to convey that a function is only `nonisolated` +statically, but it may be dynamically isolated to a specific actor at runtime. +However, we have not yet introduced "static" into the language surface to mean +"at compile time". `static` also has an existing, different meaning; +`nonisolated static func` would mean something quite different from +`nonisolated(static) func`, despite having extremely similar spelling. + +## Revisions + +The proposal was revised with the following changes after the first review: + +* Renamed `@execution(concurrent)` back to `@concurrent`. +* Renamed `@execution(caller)` to `nonisolated(nonsending)` +* Removed the unconditional warning about nonisolated async functions that + don't explicitly specify `nonisolated(nonsending)` or `@concurrent`. +* Removed `noasync` from the `assumeIsolated` API family. +* Specified the region isolation rules for `nonisolated(nonsending)` functions [as + discussed in the first review][region-isolation]. + +The proposal was revised with the following changes after the pitch discussion: + +* Gate the behavior change behind an `NonisolatedNonsendingByDefault` upcoming + feature flag. +* Change the spelling of `@concurrent` to `@execution(concurrent)`, and add an + `@execution(caller)` attribute to allow expressing the new behavior this + proposal introduces when the upcoming feature flag is not enabled. +* Apply `@execution(caller)` to nonisolated async function types by default to + make the execution semantics consistent between async function declarations + and values. +* Change the terminology in the proposal to not use the "inherits isolation" + phrase. + +[SE-0297]: /proposals/0297-concurrency-objc.md +[SE-0338]: /proposals/0338-clarify-execution-non-actor-async.md +[SE-0421]: /proposals/0421-generalize-async-sequence.md +[adoption-tooling]: https://forums.swift.org/t/pitch-adoption-tooling-for-upcoming-features/77936 +[region-isolation]: https://forums.swift.org/t/se-0461-run-nonisolated-async-functions-on-the-callers-actor-by-default/77987/36 diff --git a/proposals/0462-task-priority-escalation-apis.md b/proposals/0462-task-priority-escalation-apis.md new file mode 100644 index 0000000000..f12b6dd375 --- /dev/null +++ b/proposals/0462-task-priority-escalation-apis.md @@ -0,0 +1,237 @@ +# Task Priority Escalation APIs + +* Proposal: [SE-0462](0462-task-priority-escalation-apis.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [Freddy Kellison-Linn](https://github.com/jumhyn) +* Status: **Implemented (Swift 6.2)** +* Implementation: https://github.com/swiftlang/swift/pull/78625 +* Review: ([pitch](https://forums.swift.org/t/pitch-task-priority-escalation-apis/77702)) ([review](https://forums.swift.org/t/se-0462-task-priority-escalation-apis/77997))([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0462-task-priority-escalation-apis/78488)) + +## Introduction + +A large part of Swift Concurrency is its Structured Concurrency model, in which tasks automatically form parent-child relationships, and inherit certain traits from their parent task. For example, a task started from a medium priority task, also starts on the medium priority, and not only that – if the parent task gets awaited on from a higher priority task, the parent's as well as all of its child tasks' task priority will be escalated in order to avoid priority inversion problems. + +This feature is automatic and works transparently for any structured task hierarchy. This proposal will discuss exposing user-facing APIs which can be used to participate in task priority escalation. + +## Motivation + +Generally developers can and should rely on the automatic task priority escalation happening transparently–at least for as long as all tasks necessary to escalate are created using structured concurrency primitives (task groups and `async let`). However, sometimes it is not possible to entirely avoid creating an unstructured task. + +One such example is the async sequence [`merge`](https://github.com/apple/swift-async-algorithms/blob/4c3ea81f81f0a25d0470188459c6d4bf20cf2f97/Sources/AsyncAlgorithms/AsyncAlgorithms.docc/Guides/Merge.md) operation from the [swift-async-algorithms](https://github.com/apple/swift-async-algorithms/) project where the implementation is forced to create an unstructured task for iterating the upstream sequences, which must outlive downstream calls. These libraries would like to participate in task priority escalation to boost the priority of the upstream consuming task, however today they lack the API to do so. + +```swift +// SIMPLIFIED EXAMPLE CODE +// Complete source: https://github.com/apple/swift-async-algorithms/blob/main/Sources/AsyncAlgorithms/Merge/MergeStorage.swift + +struct AsyncMergeSequenceIterator: AsyncIterator { + struct State { + var task: Task? // unstructured upstream consumer task + var buffer: Deque + var upstreamContinuations: [UnsafeContinuation] + var downstreamContinuation: UnsafeContinuation? + } + + let state = Mutex(State()) + + func next() async throws { + self.state.withLock { state in + if state.task == nil { + state.task = Task { + // Consume from the base iterators + // ... + } + } + } + + if let element = self.state.withLock { $0.buffer.popFirst() } { + return element + } else { + // We are handling cancellation here and need to handle task escalation here as well + try await withTaskCancellationHandler { + // HERE: need to handle priority escalation and boost `state.task` + try await withCheckedContinuation { cont in + self.state.withLock { $0.consumerContinuation = cont } + } + } onCancel: { + // trigger cancellation of tasks and fail continuations + } + } + } +} +``` + +The above example showcases a common pattern: often a continuation is paired with a Task used to complete it. Around the suspension on the continuation, waiting for it to be resumed, developers often install a task cancellation handler in order to potentially break out of potentially unbounded waiting for a continuation to be resumed. Around the same suspension (marked with `HERE` in the snippet above), we might want to insert a task priority escalation handler in order to priority boost the task that is used to resume the continuation. This can be important for correctness and performance of such operations, so we should find a way to offer these libraries a mechanism to participate in task priority handling. + +Another example of libraries which may want to reach for manual task priority escalation APIs are libraries which facilitate communication across process boundaries, and would like to react to priority escalation and propagate it to a different process. Relying on the built-in priority escalation mechanisms won't work, because they are necessarily in-process, so libraries like this need to be able to participate and be notified when priority escalation happens, and also be able to efficiently cause the escalation inside the other process. + +## Proposed solution + +In order to address the above use-cases, we propose to add a pair of APIs: to react to priority escalation happening within a block of code, and an API to _cause_ a priority escalation without resorting to trickery by creating new tasks whose only purpose is to escalate the priority of some other task: + +```swift +enum State { + case initialized + case task(Task) + case priority(TaskPriority) +} +let m: Mutex = .init(.initialized) + +await withTaskPriorityEscalationHandler { + await withCheckedContinuation { cc in + let task = Task { cc.resume() } + + let newPriority: TaskPriority? = state.withLock { state -> TaskPriority? in + defer { state = .task(task) } + switch state { + case .initialized: + return nil + case .task: + preconditionFailure("unreachable") + case .priority(let priority): + return priority + } + } + // priority was escalated just before we stored the task in the mutex + if let newPriority { + Task.escalatePriority(of: task, to: newPriority) + } + } onPriorityEscalated: { newPriority in + state.withLock { state in + switch state { + case .initialized, .priority: + // priority was escalated just before we managed to store the task in the mutex + state = .priority(newPriority) + case .task(let task): + Task.escalatePriority(of: task, to: newPriority) + } + } + } +} +``` + +The above snippet handles edge various ordering situations, including the task escalation happening after +the time the handler is registered but _before_ we managed to create and store the task. + +In general, task escalation remains a slightly racy affair, we could always observe an escalation "too late" for it to matter, +and have any meaningful effect on the work's execution, however this API and associated patterns handle most situations which +we care about in practice. + +## Detailed design + +We propose the addition of a task priority escalation handler, similar to task cancellation handlers already present in the concurrency library: + +```swift +public func withTaskPriorityEscalationHandler( + operation: () async throws(E) -> T, + onPriorityEscalated handler: @Sendable (TaskPriority) -> Void, + isolation: isolated (any Actor)? = #isolation +) async throws(E) -> T +``` + +The shape of this API is similar to the `withTaskCancellationHandler` API present since initial Swift Concurrency release, however–unlike a cancellation handler–the `onPriorityEscalated` callback may be triggered multiple times. The `TaskPriority` passed to the handler is the "new priority" the surrounding task was escalated to. + +It is guaranteed that priority is ever only increasing, as Swift Concurrency does not allow for a task priority to ever be lowered after it has been escalated. If attempts are made to escalate the task priority from multiple other threads to the same priority, the handler will only trigger once. However if priority is escalated to a high and then even higher priority, the handler may be invoked twice. + +Task escalation handlers are inherently racy, and may sometimes miss an escalation, for example if it happened immediately before the handler was installed, like this: + +```swift +// priority: low +// priority: high! +await withTaskPriorityEscalationHandler { + await work() +} onPriorityEscalated: { newPriority in // may not be triggered if ->high escalation happened before handler was installed + // do something +} +``` + +This is inherent to the nature of priority escalation and even with this behavior, we believe handlers are a worthy addition. One could also check for the `Task.currentPriority` and match it against our expectations inside the `operation` wrapped by the `withTaskPriorityEscalationHandler` if that could be useful to then perform the operation at an already _immediately_ heightened priority. + +Escalation handlers work with any existing task kind (child, unstructured, unstructured detached), and trigger at every level of the hierarchy in an "outside in" order: + +```swift +let t = Task { + await withTaskPriorityEscalationHandler { + await withTaskGroup { group in + group.addTask { + await withTaskPriorityEscalationHandler { + try? await Task.sleep(for: .seconds(1)) + } onPriorityEscalated: { newPriority in print("inner: \(newPriority)") } + } + } + } onPriorityEscalated: { newPriority in print("outer: \(newPriority)") } +} + +// escalate t -> high +// "outer: high" +// "inner: high" +``` + +The API can also be freely composed with `withTaskCancellationHandler` or there may even be multiple task escalation handlers registered on the same task (but in different pieces of the code). + +### Manually propagating priority escalation + +While generally developers should not rely on manual task escalation handling, this API also does introduce a manual way to escalate a task's priority. Primarily this should be used in combination with a task escalation handler to _propagate_ an escalation to an _unstructured task_ which otherwise would miss reacting to the escalation. + +The `escalatePriority(of:to:)` API is offered as a static method on `Task` in order to slightly hide it away from using it accidentally by stumbling upon it if it were directly declared as a member method of a Task. + +```swift +extension Task { + public static func escalatePriority(of task: Task, to newPriority: TaskPriority) +} + +extension UnsafeCurrentTask { + public static func escalatePriority(of task: UnsafeCurrentTask, to newPriority: TaskPriority) +} +``` + +It is possible to escalate both a `Task` and `UnsafeCurrentTask`, however great care must be taken to not attempt to escalate an unsafe task handle if the task has already been destroyed. The `Task` accepting API is always safe. + +Currently it is not possible to escalate a specific child task (created by `async let` or a task group) because those do not return task handles. We are interested in exposing task handles to child tasks in the future, and this design could then be easily amended to gain API to support such child task handles as well. + +## Source compatibility + +This proposal is purely additive, and does not cause any source compatibility issues. + +## ABI compatibility + +This proposal is purely ABI additive. + +## Alternatives considered + +### New Continuation APIs + +We did consider if offering a new kind of continuation might be easier to work with for developers. One shape this might take is: + +```swift +struct State { + var cc = CheckedContinuation? + var task: Task? +} +let C: Mutex + +await withCheckedContinuation2 { cc in + // ... + C.withLock { $0.cc = cc } + + let t = Task { + C.withLock { + $0.cc?.resume() // maybe we'd need to add 'tryResume' + } + } + C.withLock { $0.task = t } +} onCancel: { cc in + // remember the cc can only be resumed once; we'd need to offer 'tryResume' + cc.resume(throwing: CancellationError()) +} onPriorityEscalated: { cc, newPriority in + print("new priority: \(newPriority)") + C.withLock { Task.escalatePriority(of: $0.task, to: newPriority) } +} +``` + +While at first this looks promising, we did not really remove much of the complexity -- careful locking is still necessary, and passing the continuation into the closures only makes it more error prone than not since it has become easier to accidentally multi-resume a continuation. This also does not compose well, and would only be offered around continuations, even if not all use-cases must necessarily suspend on a continuation to benefit from the priority escalation handling. + +Overall, this seems like a tightly knit API that changes current idioms of `with...Handler ` without really saving us from the inherent complexity of these handlers being invoked concurrently, and limiting the usefulness of those handlers to just "around a continuation" which may not always be the case. + +### Acknowledgements + +I'd like to thank John McCall, David Nadoba for their input on the APIs during early reviews. diff --git a/proposals/0463-sendable-completion-handlers.md b/proposals/0463-sendable-completion-handlers.md new file mode 100644 index 0000000000..85729b6ae5 --- /dev/null +++ b/proposals/0463-sendable-completion-handlers.md @@ -0,0 +1,82 @@ +# Import Objective-C completion handler parameters as `@Sendable` + +* Proposal: [SE-0463](0463-sendable-completion-handlers.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.2)** +* Vision: [Improving the approachability of data-race safety](https://github.com/swiftlang/swift-evolution/blob/main/visions/approachable-concurrency.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-import-objective-c-completion-handler-parameters-as-sendable/77904)) ([review](https://forums.swift.org/t/se-0463-import-objective-c-completion-handler-parameters-as-sendable/78169)) ([acceptance](https://forums.swift.org/t/accepted-se-0463-import-objective-c-completion-handler-parameters-as-sendable/78489)) + +## Introduction + +This proposal changes the Objective-C importing rules such that completion handler parameters are `@Sendable` by default. + +## Motivation + +Swift's data-race safety model requires function declarations to codify their concurrency invariants in the function signature with annotations. The `@Sendable` annotation indicates that closure parameters are passed over an isolation boundary before they're called. A missing `@Sendable` annotation in a library has negative effects on clients who call the function; the caller can unknowingly introduce data races, and [SE-0423: Dynamic actor isolation enforcement from non-strict-concurrency contexts][SE-0423] injects runtime assertions for non-`Sendable` closure parameters that are passed into libraries that don't have data-race safety checking. This means that a missing `@Sendable` annotation can lead to a runtime crash for any code that calls the API from an actor isolated context, which is extremely painful for projects that are migrating to the Swift 6 language mode. + +There's a large category of APIs with closure parameters that can be automatically identified as `@Sendable` functions, even if the annotation is missing: Objective-C methods with completion handler parameters. `@Sendable` is nearly always the right default for Objective-C completion handlers, and [programmers have already been searching for an automatic way for completion handlers to be `@Sendable` by default when auditing Clang headers](https://forums.swift.org/t/clang-sendability-audit-for-closures/75557). + +## Proposed solution + +I propose automatically importing completion handler parameters from Objective-C methods as `@Sendable` functions. + +## Detailed design + +If an imported method has an async variant (as described in [SE-0297: Concurrency Interoperability with Objective-C][SE-0297]) and the method is (implicitly or explicitly) `nonisolated`, the original method will be imported with a `@Sendable` annotation on its completion handler parameter. + +For example, given the following Objective-C method signature: + +```objc +- (void)performOperation:(NSString * _Nonnull)operation + completionHandler:(void (^ _Nullable)(NSString * _Nullable, NSError * _Nullable))completionHandler; +``` + +Swift will import the method with `@Sendable` on the `completionHandler` parameter: + +```swift +@preconcurrency +func perform( + operation: String, + completionHandler: @Sendable @escaping ((String?, Error?) -> Void)? +) +``` + +When calling the `perform` method from a Swift actor, the inference rules that allow non-`Sendable` closures to be isolated to the context they're formed in will no longer apply. The closure will be inferred as `nonisolated`, and warnings will be produced if any mutable state in the actor's region is accessed from the closure. Note that all APIs imported from C/C++/Objective-C are automatically `@preconcurrency`, so data-race safety violations are only ever warnings, even in the Swift 6 language mode. + +### Completion handlers of global actor isolated functions + +Functions that are isolated to a global actor will not have completion handlers imported as `@Sendable`. Main actor isolated functions with completion handlers that are always called on the main actor is a very common Objective-C pattern, and this carve out will eliminate false positive warnings in the cases where the main actor annotation is missing on the completion handler parameter. This carve out will not add any new dynamic assertions. + +### Opting out of `@Sendable` completion handlers + +If a completion handler does not cross an isolation boundary before it's called, the parameter can be annotated in the header with the `@nonSendable` attribute using `__attribute__((swift_attr(“@nonSendable”)))`. The `@nonSendable` attribute is only for Clang header annotations; it is not meant to be used from Swift code. + +## Source compatibility + +This change has no effect in language modes prior to Swift 6 when using minimal concurrency checking, and it only introduces warnings when using complete concurrency checking, even in the Swift 6 language mode. Declarations imported from C/C++/Objective-C are implicitly `@preconcurrency`, which makes all data-race safety violations warnings. + +## ABI compatibility + +This proposal has no impact on existing ABI. + +## Alternatives considered + +### Import completion handlers as `sending` instead of `@Sendable` + +The choice to import completion handlers as `@Sendable` instead of `sending` is pragmatic - the experimental `SendableCompletionHandlers` implementation has existed since 2021 and has been extensively tested for source compatibility. Similarly, `@Sendable` has been explicitly adopted in Objective-C frameworks for several years, and source compatibility issues resulting from corner cases in the compiler implementation that were intolerable to `@Sendable` mismatches have shaken out over time. `sending` is still a relatively new parameter attribute, it has not been adopted as extensively as `@Sendable`, and it does not support downgrading diagnostics in the Swift 6 language mode when combined with `@preconcurrency`. The pain caused by the dynamic actor isolation runtime assertions is enough that it's worth solving this problem now conservatively using `@Sendable`. + +Changing this proposal later to use `sending` instead will pose source compatibility issues, because it would become invalid to have a protocol requirement that is imported with a `sending` completion handler and implement the requirement with a `@Sendable` completion handler. The same source compatibility issue exists for overridden class methods. If programmers want to take advantage of region isolation, the recommended path is to modernize the code using `async`/`await`. + +## Acknowledgments + +Thank you to Becca Royal-Gordon for implementing the `SendableCompletionHandlers` experimental feature, and thank you to Pavel Yaskevich for consistently fixing compiler bugs where the implementation was intolerant to `@Sendable` mismatches. + +[SE-0297]: /proposals/0297-concurrency-objc.md +[SE-0423]: /proposals/0423-dynamic-actor-isolation.md + +## Revisions + +The proposal was revised with the following changes after the pitch discussion: + +* Add a carve out where global actor isolated functions are still imported with non-`Sendable` completion handlers. diff --git a/proposals/0464-utf8span-safe-utf8-processing.md b/proposals/0464-utf8span-safe-utf8-processing.md new file mode 100644 index 0000000000..1f366a6004 --- /dev/null +++ b/proposals/0464-utf8span-safe-utf8-processing.md @@ -0,0 +1,837 @@ +# UTF8Span: Safe UTF-8 Processing Over Contiguous Bytes + +* Proposal: [SE-0464](0464-utf8span-safe-utf8-processing.md) +* Authors: [Michael Ilseman](https://github.com/milseman), [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (6.2)** +* Bug: rdar://48132971, rdar://96837923 +* Implementation: [swiftlang/swift#78531](https://github.com/swiftlang/swift/pull/78531) +* Review: ([first pitch](https://forums.swift.org/t/pitch-utf-8-processing-over-unsafe-contiguous-bytes/69715)) ([second pitch](https://forums.swift.org/t/pitch-safe-utf-8-processing-over-contiguous-bytes/72742)) ([third pitch](https://forums.swift.org/t/pitch-utf8span-safe-utf-8-processing-over-contiguous-bytes/77483)) ([review](https://forums.swift.org/t/se-0464-utf8span-safe-utf-8-processing-over-contiguous-bytes/78307)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0464-safe-utf-8-processing-over-contiguous-bytes/79218)) + + +## Introduction + +We introduce `UTF8Span` for efficient and safe Unicode processing over contiguous storage. `UTF8Span` is a memory safe non-escapable type [similar to `Span`](0447-span-access-shared-contiguous-storage.md). + +Native `String`s are stored as validly-encoded UTF-8 bytes in an internal contiguous memory buffer. The standard library implements `String`'s API as internal methods which operate on top of this buffer, taking advantage of the validly-encoded invariant and specialized Unicode knowledge. We propose making this UTF-8 buffer and its methods public as API for more advanced libraries and developers. + +## Motivation + +Currently, if a developer wants to do `String`-like processing over UTF-8 bytes, they have to make an instance of `String`, which allocates a native storage class, copies all the bytes, and is reference counted. The developer would then need to operate within the new `String`'s views and map between `String.Index` and byte offsets in the original buffer. + +For example, if these bytes were part of a data structure, the developer would need to decide to either cache such a new `String` instance or recreate it on the fly. Caching more than doubles the size and adds caching complexity. Recreating it on the fly adds a linear time factor and class instance allocation/deallocation and potentially reference counting. + +Furthermore, `String` may not be available on tightly constrained platforms, such as those that cannot support allocations. Both `String` and `UTF8Span` have some API that require Unicode data tables and that might not be available on embedded (String via its conformance to `Comparable` and `Collection` depend on these data tables while `UTF8Span` has a couple of methods that will be unavailable). + +### UTF-8 validity and efficiency + +UTF-8 validation is a particularly common concern and the subject of a fair amount of [research](https://lemire.me/blog/2020/10/20/ridiculously-fast-unicode-utf-8-validation/). Once an input is known to be validly encoded UTF-8, subsequent operations such as decoding, grapheme breaking, comparison, etc., can be implemented much more efficiently under this assumption of validity. Swift's `String` type's native storage is guaranteed-valid-UTF8 for this reason. + +Failure to guarantee UTF-8 encoding validity creates security and safety concerns. With invalidly-encoded contents, memory safety would become more nuanced. An ill-formed leading byte can dictate a scalar length that is longer than the memory buffer. The buffer may have bounds associated with it, which differs from the bounds dictated by its contents. + +Additionally, a particular scalar value in valid UTF-8 has only one encoding, but invalid UTF-8 could have the same value encoded as an [overlong encoding](https://en.wikipedia.org/wiki/UTF-8#Overlong_encodings), which would compromise code that checks for the presence of a scalar value by looking at the encoded bytes (or that does a byte-wise comparison). + + +## Proposed solution + +We propose a non-escapable `UTF8Span` which exposes `String` functionality for validly-encoded UTF-8 code units in contiguous memory. We also propose rich API describing the kind and location of encoding errors. + +## Detailed design + +`UTF8Span` is a borrowed view into contiguous memory containing validly-encoded UTF-8 code units. + +```swift +public struct UTF8Span: Copyable, ~Escapable, BitwiseCopyable {} +``` + +`UTF8Span` is a trivial struct and is 2 words in size on 64-bit platforms. + +### UTF-8 validation + +We propose new API for identifying where and what kind of encoding errors are present in UTF-8 content. + +```swift +extension Unicode.UTF8 { + /** + + The kind and location of a UTF-8 encoding error. + + Valid UTF-8 is represented by this table: + + ╔════════════════════╦════════╦════════╦════════╦════════╗ + ║ Scalar value ║ Byte 0 ║ Byte 1 ║ Byte 2 ║ Byte 3 ║ + ╠════════════════════╬════════╬════════╬════════╬════════╣ + ║ U+0000..U+007F ║ 00..7F ║ ║ ║ ║ + ║ U+0080..U+07FF ║ C2..DF ║ 80..BF ║ ║ ║ + ║ U+0800..U+0FFF ║ E0 ║ A0..BF ║ 80..BF ║ ║ + ║ U+1000..U+CFFF ║ E1..EC ║ 80..BF ║ 80..BF ║ ║ + ║ U+D000..U+D7FF ║ ED ║ 80..9F ║ 80..BF ║ ║ + ║ U+E000..U+FFFF ║ EE..EF ║ 80..BF ║ 80..BF ║ ║ + ║ U+10000..U+3FFFF ║ F0 ║ 90..BF ║ 80..BF ║ 80..BF ║ + ║ U+40000..U+FFFFF ║ F1..F3 ║ 80..BF ║ 80..BF ║ 80..BF ║ + ║ U+100000..U+10FFFF ║ F4 ║ 80..8F ║ 80..BF ║ 80..BF ║ + ╚════════════════════╩════════╩════════╩════════╩════════╝ + + ### Classifying errors + + An *unexpected continuation* is when a continuation byte (`10xxxxxx`) occurs + in a position that should be the start of a new scalar value. Unexpected + continuations can often occur when the input contains arbitrary data + instead of textual content. An unexpected continuation at the start of + input might mean that the input was not correctly sliced along scalar + boundaries or that it does not contain UTF-8. + + A *truncated scalar* is a multi-byte sequence that is the start of a valid + multi-byte scalar but is cut off before ending correctly. A truncated + scalar at the end of the input might mean that only part of the entire + input was received. + + A *surrogate code point* (`U+D800..U+DFFF`) is invalid UTF-8. Surrogate + code points are used by UTF-16 to encode scalars in the supplementary + planes. Their presence may mean the input was encoded in a different 8-bit + encoding, such as CESU-8, WTF-8, or Java's Modified UTF-8. + + An *invalid non-surrogate code point* is any code point higher than + `U+10FFFF`. This can often occur when the input is arbitrary data instead + of textual content. + + An *overlong encoding* occurs when a scalar value that could have been + encoded using fewer bytes is encoded in a longer byte sequence. Overlong + encodings are invalid UTF-8 and can lead to security issues if not + correctly detected: + + - https://nvd.nist.gov/vuln/detail/CVE-2008-2938 + - https://nvd.nist.gov/vuln/detail/CVE-2000-0884 + + An overlong encoding of `NUL`, `0xC0 0x80`, is used in Java's Modified + UTF-8 but is invalid UTF-8. Overlong encoding errors often catch attempts + to bypass security measures. + + ### Reporting the range of the error + + The range of the error reported follows the *Maximal subpart of an + ill-formed subsequence* algorithm in which each error is either one byte + long or ends before the first byte that is disallowed. See "U+FFFD + Substitution of Maximal Subparts" in the Unicode Standard. Unicode started + recommending this algorithm in version 6 and is adopted by the W3C. + + The maximal subpart algorithm will produce a single multi-byte range for a + truncated scalar (a multi-byte sequence that is the start of a valid + multi-byte scalar but is cut off before ending correctly). For all other + errors (including overlong encodings, surrogates, and invalid code + points), it will produce an error per byte. + + Other commonly reported error ranges can be constructed from this result. + For example, PEP 383's error-per-byte can be constructed by mapping over + the reported range. Similarly, constructing a single error for the longest + invalid byte range can be constructed by joining adjacent error ranges. + + ╔═════════════════╦══════╦═════╦═════╦═════╦═════╦═════╦═════╦══════╗ + ║ ║ 61 ║ F1 ║ 80 ║ 80 ║ E1 ║ 80 ║ C2 ║ 62 ║ + ╠═════════════════╬══════╬═════╬═════╬═════╬═════╬═════╬═════╬══════╣ + ║ Longest range ║ U+61 ║ err ║ ║ ║ ║ ║ ║ U+62 ║ + ║ Maximal subpart ║ U+61 ║ err ║ ║ ║ err ║ ║ err ║ U+62 ║ + ║ Error per byte ║ U+61 ║ err ║ err ║ err ║ err ║ err ║ err ║ U+62 ║ + ╚═════════════════╩══════╩═════╩═════╩═════╩═════╩═════╩═════╩══════╝ + + */ + public struct EncodingError: Error, Sendable, Hashable, Codable { + /// The kind of encoding error + public var kind: Unicode.UTF8.EncodingError.Kind + + /// The range of offsets into our input containing the error + public var range: Range + + public init( + _ kind: Unicode.UTF8.EncodingError.Kind, + _ range: some RangeExpression + ) + + public init(_ kind: Unicode.UTF8.EncodingError.Kind, at: Int) + } +} + +extension UTF8.EncodingError { + /// The kind of encoding error encountered during validation + public struct Kind: Error, Sendable, Hashable, Codable, RawRepresentable { + public var rawValue: UInt8 + + public init(rawValue: UInt8) + + /// A continuation byte (`10xxxxxx`) outside of a multi-byte sequence + public static var unexpectedContinuationByte: Self + + /// A byte in a surrogate code point (`U+D800..U+DFFF`) sequence + public static var surrogateCodePointByte: Self + + /// A byte in an invalid, non-surrogate code point (`>U+10FFFF`) sequence + public static var invalidNonSurrogateCodePointByte: Self + + /// A byte in an overlong encoding sequence + public static var overlongEncodingByte: Self + + /// A multi-byte sequence that is the start of a valid multi-byte scalar + /// but is cut off before ending correctly + public static var truncatedScalar: Self + } +} + +extension UTF8.EncodingError.Kind: CustomStringConvertible { + public var description: String { get } +} + +extension UTF8.EncodingError: CustomStringConvertible { + public var description: String { get } +} +``` + +### Creation and validation + +`UTF8Span` is validated at initialization time and encoding errors are diagnosed and thrown. + + +```swift + +extension UTF8Span { + /// Creates a UTF8Span containing `codeUnits`. Validates that the input is + /// valid UTF-8, otherwise throws an error. + /// + /// The resulting UTF8Span has the same lifetime constraints as `codeUnits`. + public init(validating codeUnits: Span) throws(UTF8.EncodingError) + + /// Creates a UTF8Span unsafely containing `uncheckedBytes`, skipping validation. + /// + /// `uncheckedBytes` _must_ be valid UTF-8 or else undefined behavior may + /// emerge from any use of the resulting UTF8Span, including any use of a + /// `String` created by copying the resultant UTF8Span + @unsafe + public init(unsafeAssumingValidUTF8 uncheckedCodeUnits: Span) +} +``` + +Similarly, `String`s can be created from `UTF8Span`s without re-validating their contents. + +```swift +extension String { + /// Create's a String containing a copy of the UTF-8 content in `codeUnits`. + /// Skips + /// validation. + public init(copying codeUnits: UTF8Span) +} +``` + +### Scalar processing + +We propose a `UTF8Span.UnicodeScalarIterator` type that can do scalar processing forwards and backwards. Note that `UnicodeScalarIterator` itself is non-escapable, and thus cannot conform to `IteratorProtocol`, etc. + +```swift +extension UTF8Span { + /// Returns an iterator that will decode the code units into + /// `Unicode.Scalar`s. + /// + /// The resulting iterator has the same lifetime constraints as `self`. + public func makeUnicodeScalarIterator() -> UnicodeScalarIterator + + /// Iterate the `Unicode.Scalar`s contents of a `UTF8Span`. + public struct UnicodeScalarIterator: ~Escapable { + public let codeUnits: UTF8Span + + /// The byte offset of the start of the next scalar. This is + /// always scalar-aligned. + public var currentCodeUnitOffset: Int { get private(set) } + + public init(_ codeUnits: UTF8Span) + + /// Decode and return the scalar starting at `currentCodeUnitOffset`. + /// After the function returns, `currentCodeUnitOffset` holds the + /// position at the end of the returned scalar, which is also the start + /// of the next scalar. + /// + /// Returns `nil` if at the end of the `UTF8Span`. + public mutating func next() -> Unicode.Scalar? + + /// Decode and return the scalar ending at `currentCodeUnitOffset`. After + /// the function returns, `currentCodeUnitOffset` holds the position at + /// the start of the returned scalar, which is also the end of the + /// previous scalar. + /// + /// Returns `nil` if at the start of the `UTF8Span`. + public mutating func previous() -> Unicode.Scalar? + + /// Advance `codeUnitOffset` to the end of the current scalar, without + /// decoding it. + /// + /// Returns the number of `Unicode.Scalar`s skipped over, which can be 0 + /// if at the end of the UTF8Span. + public mutating func skipForward() -> Int + + /// Advance `codeUnitOffset` to the end of `n` scalars, without decoding + /// them. + /// + /// Returns the number of `Unicode.Scalar`s skipped over, which can be + /// fewer than `n` if at the end of the UTF8Span. + public mutating func skipForward(by n: Int) -> Int + + /// Move `codeUnitOffset` to the start of the previous scalar, without + /// decoding it. + /// + /// Returns the number of `Unicode.Scalar`s skipped over, which can be 0 + /// if at the start of the UTF8Span. + public mutating func skipBack() -> Int + + /// Move `codeUnitOffset` to the start of the previous `n` scalars, + /// without decoding them. + /// + /// Returns the number of `Unicode.Scalar`s skipped over, which can be + /// fewer than `n` if at the start of the UTF8Span. + public mutating func skipBack(by n: Int) -> Int + + /// Reset to the nearest scalar-aligned code unit offset `<= i`. + public mutating func reset(roundingBackwardsFrom i: Int) + + /// Reset to the nearest scalar-aligned code unit offset `>= i`. + public mutating func reset(roundingForwardsFrom i: Int) + + /// Reset this iterator to code unit offset `i`, skipping _all_ safety + /// checks (including bounds checks). + /// + /// Note: This is only for very specific, low-level use cases. If + /// `codeUnitOffset` is not properly scalar-aligned, this function can + /// result in undefined behavior when, e.g., `next()` is called. + /// + /// For example, this could be used by a regex engine to backtrack to a + /// known-valid previous position. + /// + public mutating func reset(uncheckedAssumingAlignedTo i: Int) + + /// Returns the UTF8Span containing all the content up to the iterator's + /// current position. + /// + /// The resultant `UTF8Span` has the same lifetime constraints as `self`. + public func prefix() -> UTF8Span + + /// Returns the UTF8Span containing all the content after the iterator's + /// current position. + /// + /// The resultant `UTF8Span` has the same lifetime constraints as `self`. + public func suffix() -> UTF8Span + } +} + +``` + +### Character processing + +We similarly propose a `UTF8Span.CharacterIterator` type that can do grapheme-breaking forwards and backwards. + +The `CharacterIterator` assumes that the start and end of the `UTF8Span` is the start and end of content. + +Any scalar-aligned position is a valid place to start or reset the grapheme-breaking algorithm to, though you could get different `Character` output if resetting to a position that isn't `Character`-aligned relative to the start of the `UTF8Span` (e.g. in the middle of a series of regional indicators). + +```swift + +extension UTF8Span { + /// Returns an iterator that will construct `Character`s from the underlying + /// UTF-8 content. + /// + /// The resulting iterator has the same lifetime constraints as `self`. + public func makeCharacterIterator() -> CharacterIterator + + /// Iterate the `Character` contents of a `UTF8Span`. + public struct CharacterIterator: ~Escapable { + public let codeUnits: UTF8Span + + /// The byte offset of the start of the next `Character`. This is always + /// scalar-aligned. It is always `Character`-aligned relative to the last + /// call to `reset` (or the start of the span if not called). + public var currentCodeUnitOffset: Int { get private(set) } + + public init(_ span: UTF8Span) + + /// Return the `Character` starting at `currentCodeUnitOffset`. After the + /// function returns, `currentCodeUnitOffset` holds the position at the + /// end of the `Character`, which is also the start of the next + /// `Character`. + /// + /// Returns `nil` if at the end of the `UTF8Span`. + public mutating func next() -> Character? + + /// Return the `Character` ending at `currentCodeUnitOffset`. After the + /// function returns, `currentCodeUnitOffset` holds the position at the + /// start of the returned `Character`, which is also the end of the + /// previous `Character`. + /// + /// Returns `nil` if at the start of the `UTF8Span`. + public mutating func previous() -> Character? + + /// Advance `codeUnitOffset` to the end of the current `Character`, + /// without constructing it. + /// + /// Returns the number of `Character`s skipped over, which can be 0 + /// if at the end of the UTF8Span. + public mutating func skipForward() -> Int + + /// Advance `codeUnitOffset` to the end of `n` `Characters`, without + /// constructing them. + /// + /// Returns the number of `Character`s skipped over, which can be + /// fewer than `n` if at the end of the UTF8Span. + public mutating func skipForward(by n: Int) -> Int + + /// Move `codeUnitOffset` to the start of the previous `Character`, + /// without constructing it. + /// + /// Returns the number of `Character`s skipped over, which can be 0 + /// if at the start of the UTF8Span. + public mutating func skipBack() -> Int + + /// Move `codeUnitOffset` to the start of the previous `n` `Character`s, + /// without constructing them. + /// + /// Returns the number of `Character`s skipped over, which can be + /// fewer than `n` if at the start of the UTF8Span. + public mutating func skipBack(by n: Int) -> Int + + /// Reset to the nearest character-aligned position `<= i`. + public mutating func reset(roundingBackwardsFrom i: Int) + + /// Reset to the nearest character-aligned position `>= i`. + public mutating func reset(roundingForwardsFrom i: Int) + + /// Reset this iterator to code unit offset `i`, skipping _all_ safety + /// checks (including bounds checks). + /// + /// Note: This is only for very specific, low-level use cases. If + /// `codeUnitOffset` is not properly scalar-aligned, this function can + /// result in undefined behavior when, e.g., `next()` is called. + /// + /// If `i` is scalar-aligned, but not `Character`-aligned, you may get + /// different results from running `Character` iteration. + /// + /// For example, this could be used by a regex engine to backtrack to a + /// known-valid previous position. + /// + public mutating func reset(uncheckedAssumingAlignedTo i: Int) + + /// Returns the UTF8Span containing all the content up to the iterator's + /// current position. + /// + /// The resultant `UTF8Span` has the same lifetime constraints as `self`. + public func prefix() -> UTF8Span + + /// Returns the UTF8Span containing all the content after the iterator's + /// current position. + /// + /// The resultant `UTF8Span` has the same lifetime constraints as `self`. + public func suffix() -> UTF8Span + } +} +``` + +### Comparisons + +The content of a `UTF8Span` can be compared in a number of ways, including literally (byte semantics) and Unicode canonical equivalence. + +```swift +extension UTF8Span { + /// Whether this span has the same bytes as `other`. + public func bytesEqual(to other: UTF8Span) -> Bool + + /// Whether this span has the same bytes as `other`. + public func bytesEqual(to other: some Sequence) -> Bool + + /// Whether this span has the same `Unicode.Scalar`s as `other`. + public func scalarsEqual( + to other: some Sequence + ) -> Bool + + /// Whether this span has the same `Character`s as `other`, using + /// `Character.==` (i.e. Unicode canonical equivalence). + public func charactersEqual( + to other: some Sequence + ) -> Bool +} +``` + + +#### Canonical equivalence and ordering + +`UTF8Span` can perform Unicode canonical equivalence checks (i.e. the semantics of `String.==` and `Character.==`). + +```swift +extension UTF8Span { + /// Whether `self` is equivalent to `other` under Unicode Canonical + /// Equivalence. + public func isCanonicallyEquivalent( + to other: UTF8Span + ) -> Bool + + /// Whether `self` orders less than `other` under Unicode Canonical + /// Equivalence using normalized code-unit order (in NFC). + public func canonicallyPrecedes( + _ other: UTF8Span + ) -> Bool +} +``` + +#### Extracting sub-spans + +Slicing a `UTF8Span` is nuanced and depends on the caller's desired use. They can only be sliced at scalar-aligned code unit offsets or else it will break the valid-UTF8 invariant. Furthermore, if the caller desires consistent grapheme breaking behavior without externally managing grapheme breaking state, they must be sliced along `Character` boundaries. For this reason, we have exposed slicing as `prefix` and `suffix` operations on `UTF8Span`'s iterators instead of `Span`'s' `extracting` methods. + +### Queries + +`UTF8Span` checks at construction time and remembers whether its contents are all ASCII. Additional checks can be requested and remembered. + +```swift +extension UTF8Span { + /// Returns whether contents are known to be all-ASCII. A return value of + /// `true` means that all code units are ASCII. A return value of `false` + /// means there _may_ be non-ASCII content. + /// + /// ASCII-ness is checked and remembered during UTF-8 validation, so this + /// is often equivalent to is-ASCII, but there are some situations where + /// we might return `false` even when the content happens to be all-ASCII. + /// + /// For example, a UTF-8 span generated from a `String` that at some point + /// contained non-ASCII content would report false for `isKnownASCII`, even + /// if that String had subsequent mutation operations that removed any + /// non-ASCII content. + public var isKnownASCII: Bool { get } + + /// Do a scan checking for whether the contents are all-ASCII. + /// + /// Updates the `isKnownASCII` bit if contents are all-ASCII. + public mutating func checkForASCII() -> Bool + + /// Returns whether the contents are known to be NFC. This is not + /// always checked at initialization time and is set by `checkForNFC`. + public var isKnownNFC: Bool { get } + + /// Do a scan checking for whether the contents are in Normal Form C. + /// When the contents are in NFC, canonical equivalence checks are much + /// faster. + /// + /// `quickCheck` will check for a subset of NFC contents using the + /// NFCQuickCheck algorithm, which is faster than the full normalization + /// algorithm. However, it cannot detect all NFC contents. + /// + /// Updates the `isKnownNFC` bit. + public mutating func checkForNFC( + quickCheck: Bool + ) -> Bool +} +``` + +### `UTF8Span` from `String` + +We propose adding `utf8Span` properties to `String` and `Substring`, in line with [SE-0456](0456-stdlib-span-properties.md): + +```swift +extension String { + public var utf8Span: UTF8Span { borrowing get } +} +extension Substring { + public var utf8Span: UTF8Span { borrowing get } +} +``` + + +### `Span`-like functionality + +A `UTF8Span` is similar to a `Span`, but with the valid-UTF8 invariant and additional information such as `isASCII`. We propose a way to get a `Span` from a `UTF8Span` as well as some methods directly on `UTF8Span`: + +``` +extension UTF8Span { + public var isEmpty: Bool { get } + + public var span: Span { get } +} +``` + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a new version of the standard library and runtime. + +## Future directions + +### Streaming grapheme breaking + +Grapheme-breaking, which identifies where the boundaries between `Character`s are, is more complex than scalar decoding. Grapheme breaking can be ran from any scalar-aligned position, either with a given state from having processed previous scalars, or with a "fresh" state (as though that position were the start of new content). + +While the code units in a `UTF8Span` are always scalar-aligned (in order to be validly encoded), whether a span is grapheme-cluster aligned depends on its intended use. For example, `AttributedString` stores its content using rope-like storage, in which the entire content is a sequence of spans were each individual span is scalar-aligned but not necessarily grapheme-cluster aligned. + +A potential approach to exposing this functionality is to make the stdlib's `GraphemeBreakingState` public and define API for finding grapheme-breaks. + +```swift +extension Unicode { + public struct GraphemeBreakingState: Sendable, Equatable { + public init() + } +} +``` + +One approach is to add API to the grapheme breaking state so that the state can find the next break (while updating itself). Another is to pass grapheme breaking state to an iterator on UTF8Span, like below: + +```swift +extension UTF8Span { + public struct GraphemeBreakIterator: ~Escapable { + public var codeUnits: UTF8Span + public var currentCodeUnitOffset: Int + public var state: Unicode.GraphemeBreakingState + + public init(_ span: UTF8Span) + + public init(_ span: UTF8Span, using state: Unicode.GraphemeBreakingState) + + public mutating func next() -> Bool + + public mutating func previous() -> Bool + + + public mutating func skipForward() + + public mutating func skipForward(by n: Int) + + public mutating func skipBack() + + public mutating func skipBack(by n: Int) + + public mutating func reset( + roundingBackwardsFrom i: Int, using: Unicode.GraphemeBreakingState + ) + + public mutating func reset( + roundingForwardsFrom i: Int, using: Unicode.GraphemeBreakingState + ) + + public mutating func reset( + uncheckedAssumingAlignedTo i: Int, using: Unicode.GraphemeBreakingState + ) + + public func prefix() -> UTF8Span + public func suffix() -> UTF8Span + } +``` + + +### More alignments and alignment queries + +Future API could include word iterators (either [simple](https://www.unicode.org/reports/tr18/#Simple_Word_Boundaries) or [default](https://www.unicode.org/reports/tr18/#Default_Word_Boundaries)), line iterators, etc. + +Similarly, we could add API directly to `UTF8Span` for testing whether a given code unit offset is suitably aligned (including scalar or grapheme-cluster alignment checks). + +### `~=` and other operators + +`UTF8Span` supports both binary equivalence and Unicode canonical equivalence. For example, a textual format parser using `UTF8Span` might operate in terms of binary equivalence for processing the textual format itself and then in terms of Unicode canonical equivalnce when interpreting the content of the fields. + +We are deferring making any decision on what a "default" comparison semantics should be as future work, which would include defining a `~=` operator (which would allow one to switch over a `UTF8Span` and match against literals). + +It may also be the case that it makes more sense for a library or application to define wrapper types around `UTF8Span` which can define `~=` with their preferred comparison semantics. + + +### Creating `String` copies + +We could add an initializer to `String` that makes an owned copy of a `UTF8Span`'s contents. Such an initializer can skip UTF-8 validation. + +Alternatively, we could defer adding anything until more of the `Container` protocol story is clear. + +### Normalization + +Future API could include checks for whether the content is in a particular normal form (not just NFC). + +### UnicodeScalarView and CharacterView + +Like `Span`, we are deferring adding any collection-like types to non-escapable `UTF8Span`. Future work could include adding view types that conform to a new `Container`-like protocol. + +See "Alternatives Considered" below for more rationale on not adding `Collection`-like API in this proposal. + +### More algorithms + +We propose equality checks (e.g. `scalarsEqual`), as those are incredibly common and useful operations. We have (tentatively) deferred other algorithms until non-escapable collections are figured out. + +However, we can add select high-value algorithms if motivated by the community. + +### More validation API + +Future API could include a way to find and classify UTF-8 encoding errors in arbitrary byte sequences, beyond just `Span`. + +We could propose something like: + +```swift +extension UTF8 { + public static func findFirstError( + _ s: some Sequence + ) -> UTF8.EncodingError? + + public static func findAllErrors( + _ s: some Sequence + ) -> some Sequence? +``` + +We are leaving this as future work. It also might be better formulated in line with a segemented-storage `Container`-like protocol instead of `some Sequence`. + +For now, developers can validate UTF-8 and diagnose the location and type of error using `UTF8Span`'s validating initializer, which takes a `Span`. This is similar to how developers do UTF-8 validation [in Rust](https://doc.rust-lang.org/std/str/fn.from_utf8.html). + +### Transcoded iterators, normalized iterators, case-folded iterators, etc + +We could provide lazily transcoded, normalized, case-folded, etc., iterators. If we do any of these for `UTF8Span`, we should consider adding equivalents views on `String`, `Substring`, etc. + +### Regex or regex-like support + +Future API additions would be to support `Regex`es on `UTF8Span`. We'd expose grapheme-level semantics, scalar-level semantics, and introduce byte-level semantics. + +Another future direction could be to add many routines corresponding to the underlying operations performed by the regex engine, which would be useful for parser-combinator libraries who wish to expose `String`'s model of Unicode by using the stdlib's accelerated implementation. + +### Track other bits + +Future work include tracking whether the contents are NULL-terminated (useful for C bridging), whether the contents contain any newlines or only a single newline at the end (useful for accelerating Regex `.`), etc. + +### Putting more API on String + +`String` would also benefit from the query API, such as `isKnownNFC` and corresponding scan methods. Because a string may be a lazily-bridged instance of `NSString`, we don't always have the bits available to query or set, but this may become viable pending future improvements in bridging. + +### Generalize printing and logging facilities + +Many printing and logging protocols and facilities operate in terms of `String`. They could be generalized to work in terms of UTF-8 bytes instead, which is important for embedded. + +## Alternatives considered + +### Problems arising from the unsafe init + +The combination of the unsafe init on `UTF8Span` and the copying init on `String` creates a new kind of easily-accesible backdoor to `String`'s security and safety, namely the invariant that it holds validly encoded UTF-8 when in native form. + +Currently, String is 100% safe outside of crazy custom subclass shenanigans (only on ObjC platforms) or arbitrarily scribbling over memory (which is true of all of Swift). Both are highly visible and require writing many lines of advanced-knowledge code. + +Without these two API, it is in theory possible to skip validation and produce a String instance of the [indirect contiguous UTF-8](https://forums.swift.org/t/piercing-the-string-veil/21700) flavor through a custom subclass of NSString. But, it is only available on Obj-C platforms and involves creating a custom subclass of `NSString`, having knowledge of lazy bridging internals (which can and sometimes do change from release to release of Swift), and writing very specialized code. The product would be an unsafe lazily bridged instance of `String`, which could more than offset any performance gains from the workaround itself. + +With these two API, you can get to UB via a: + +```swift +let codeUnits = unsafe UTF8Span(unsafeAssumingValidUTF8: bytes) +... +String(copying: codeUnits) +``` + +We are (very) weakly in favor of keeping the unsafe init, because there are many low-level situations in which the valid-UTF8 invariant is held by the system itself (such as a data structure using a custom allocator). + + + +### Invalid start / end of input UTF-8 encoding errors + +Earlier prototypes had `.invalidStartOfInput` and `.invalidEndOfInput` UTF8 validation errors to communicate that the input was perhaps incomplete or not slices along scalar boundaries. In this scenario, `.invalidStartOfInput` is equivalent to `.unexpectedContinuation` with the range's lower bound equal to 0 and `.invalidEndOfInput` is equivalent to `.truncatedScalar` with the range's upper bound equal to `count`. + +This was rejected so as to not have two ways to encode the same error. There is no loss of information and `.unexpectedContinuation`/`.truncatedScalar` with ranges are more semantically precise. + +### An unsafe UTF8 Buffer Pointer type + +An [earlier pitch](https://forums.swift.org/t/pitch-utf-8-processing-over-unsafe-contiguous-bytes/69715) proposed an unsafe version of `UTF8Span`. Now that we have `~Escapable`, a memory-safe `UTF8Span` is better. + +### Alternatives to Iterators + +#### Functions + +A previous version of this pitch had code unit offset taking API directly on UTF8Span instead of using iterators as proposed. This lead to a large number of unweildy API. For example, instead of: + +```swift +extension UTF8Span.UnicodeScalarIterator { + public mutating func next() -> Unicode.Scalar? { } +} +``` + +we had: + +```swift +extension UTF8Span { +/// Decode the `Unicode.Scalar` starting at `i`. Return it and the start of + /// the next scalar. + /// + /// `i` must be scalar-aligned. + public func decodeNextScalar( + _ i: Int + ) -> (Unicode.Scalar, nextScalarStart: Int) + + /// Decode the `Unicode.Scalar` starting at `i`. Return it and the start of + /// the next scalar. + /// + /// `i` must be scalar-aligned. + /// + /// This function does not validate that `i` is within the span's bounds; + /// this is an unsafe operation. + public func decodeNextScalar( + unchecked i: Int + ) -> (Unicode.Scalar, nextScalarStart: Int) + + /// Decode the `Unicode.Scalar` starting at `i`. Return it and the start of + /// the next scalar. + /// + /// `i` must be scalar-aligned. + /// + /// This function does not validate that `i` is within the span's bounds; + /// this is an unsafe operation. + /// + /// + /// This function does not validate that `i` is scalar-aligned; this is an + /// unsafe operation if `i` isn't. + public func decodeNextScalar( + uncheckedAssumingAligned i: Int + ) -> (Unicode.Scalar, nextScalarStart: Int) +} +``` + +Every operation had `unchecked:` and `uncheckedAssumingAligned:` variants, which were needed to implement higher-performance constructs such as iterators and string processing features (including the `Regex` engine). + +This API made the caller manage the scalar-alignment invariant, while the iterator-style API proposed maintains this invariant internally, allowing it to use the most efficient implementation. + +Scalar-alignment can still be checked and managed by the caller through the `reset` API, which safely round forwards or backwards as needed. And, for high performance use cases where the caller knows that a given position is appropriately aligned already (for example, revisiting a prior point in a string during `Regex` processing), there's the `reset(uncheckedAssumingAlignedTo:)` API available. + +#### View Collections + +Another forumulation of these operations could be to provide a collection-like API phrased in terms of indices. Because `Collection`s are `Escapable`, we cannot conform nested `View` types to `Collection` so these would not benefit from any `Collection`-generic code, algorithms, etc. + +A benefit of such `Collection`-like views is that it could help serve as adapter code for migration. Existing `Collection`-generic algorithms and methods could be converted to support `UTF8Span` via copy-paste-edit. That is, a developer could interact with `UTF8Span` ala: + +```swift +// view: UTF8Span.UnicodeScalarView +var curIdx = view.startIndex +while curIdx < view.endIndex { + let scalar = view[curIdx] + foo(scalar) + view.formIndex(after: &curIndex) +} +``` + +in addition to the iterator approach of: + +```swift +// iter: UTF8Span.UnicodeScalarIterator (or UTF8Span.UnicodeScalarView.Iterator) +while let scalar = iter.next() { + foo(scalar) +} +``` + +However, the iterator-based approach is the more efficient and direct way to work with a `UTF8Span`. Even if we had `Collection`-like API, we'd still implement a custom iterator type and advocate its use as the best way to interact with `UTF8Span`. The question is whether or not, for a given `FooIterator` we should additionally provide a `FooView`, `FooView.Index`, `FooView.SubSequence`, (possibly) `FooView.Slice`, etc. + +Idiomatic `Collection`-style interfaces support index interchange, even if "support" means reliably crashing after a dynamic check. Any idiomatic index-based interface would need to dynamically check for correct alignment in case the received index was derived from a different span. (There is a whole design space around smart indices and their tradeoffs, discussed in a [lengthy appendix](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md#appendix-index-and-slicing-design-considerations) in the Span proposal). + +This means that `UTF8Span.UnicodeScalarView.subscript` would have to check for scalar alignment of its given index, as it does not know whether it originally produced the passed index or not. Similarly, `index(after:)`, `index(before:)`, `index(_:offsetBy:)`, etc., would make these checks on every call. + +If we want to give the developer access to efficient formulations of index-style interfaces, we'd additionally propose `uncheckedAssumingAligned:` variants of nearly every method: `subscript(uncheckedAssumingAligned i:)`, `index(uncheckedAssumingAlignedAfter:)`, `index(uncheckedAssumingAlignedBefore:)`, `index(uncheckedAssumingAligned:offsetBy:)`, etc.. This also undermines the value of having an adapter to existing code patterns. + +If we do provide view adapter code, the API could look a little different in that `UnicodeScalarIterator` is called `UnicodeScalarView.Iterator`, `prefix/suffix` are slicing, and the `reset()` functionality is expressed by slicing the view before creating an iterator. However, this would also have the effect of scattering the efficient API use pattern across multiple types, intermingled with inefficient or ill-advised adaptor interfaces which have the more idiomatic names. + +Finally, in the future there will likely be some kind of `Container` protocol for types that can vend segments of contiguous storage. In our case, the segment type is `UTF8Span`, while the element is decoded from the underlying UTF-8. It's likely easier and more straightforward to retrofit or deprecate a single `UnicodeScalarIterator` type than a collection of types interrelated to each other. + +## Acknowledgments + +Karoy Lorentey, Karl, Geordie_J, and fclout, contributed to this proposal with their clarifying questions and discussions. + + + diff --git a/proposals/0465-nonescapable-stdlib-primitives.md b/proposals/0465-nonescapable-stdlib-primitives.md new file mode 100644 index 0000000000..798025fef9 --- /dev/null +++ b/proposals/0465-nonescapable-stdlib-primitives.md @@ -0,0 +1,886 @@ +# Standard Library Primitives for Nonescapable Types + +* Proposal: [SE-0465](0465-nonescapable-stdlib-primitives.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [Doug Gregor](https://github.com/douggregor) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [Improving Swift performance predictability: ARC improvements and ownership control][Roadmap] +* Implementation: https://github.com/swiftlang/swift/pull/73258 +* Review: ([Acceptance](https://forums.swift.org/t/accepted-se-0465-standard-library-primitives-for-nonescapable-type/78637)) ([Review](https://forums.swift.org/t/se-0465-standard-library-primitives-for-nonescapable-types/78310)) ([Pitch](https://forums.swift.org/t/pitch-nonescapable-standard-library-primitives/77253)) + +[Roadmap]: https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206 +[Pitch]: https://forums.swift.org/t/pitch-nonescapable-standard-library-primitives/77253 + + + +Related proposals: + +- [SE-0377] `borrowing` and `consuming` parameter ownership modifiers +- [SE-0390] Noncopyable structs and enums +- [SE-0426] `BitwiseCopyable` +- [SE-0427] Noncopyable generics +- [SE-0429] Partial consumption of noncopyable values +- [SE-0432] Borrowing and consuming pattern matching for noncopyable types +- [SE-0437] Noncopyable Standard Library Primitives +- [SE-0446] Nonescapable Types +- [SE-0447] `Span`: Safe Access to Contiguous Storage +- [SE-0452] Integer Generic Parameters +- [SE-0453] `InlineArray`, a fixed-size array +- [SE-0456] Add `Span`-providing Properties to Standard Library Types + + + +[SE-0370]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0370-pointer-family-initialization-improvements.md +[SE-0377]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md +[SE-0390]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md +[SE-0426]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0426-bitwise-copyable.md +[SE-0427]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md +[SE-0429]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0429-partial-consumption.md +[SE-0432]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0432-noncopyable-switch.md +[SE-0437]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0437-noncopyable-stdlib-primitives.md +[SE-0446]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md +[SE-0447]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md +[SE-0452]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0452-integer-generic-parameters.md +[SE-0453]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md +[SE-0456]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0456-stdlib-span-properties.md + +### Table of Contents + + * [Introduction](#introduction) + * [Motivation](#motivation) + * [Proposed Solution](#proposed-solution) + * [Nonescapable optionals](#nonescapable-optionals) + * [Nonescapable Result](#nonescapable-result) + * [Retrieving the memory layout of nonescapable types](#retrieving-the-memory-layout-of-nonescapable-types) + * [Lifetime management](#lifetime-management) + * [Metatype comparisons](#metatype-comparisons) + * [Object identifiers](#object-identifiers) + * [Odds and ends](#odds-and-ends) + * [Detailed Design](#detailed-design) + * [Inferred lifetime behavior of nonescapable enum types](#inferred-lifetime-behavior-of-nonescapable-enum-types) + * [Inferred lifetime behavior of Optional's notational conveniences](#inferred-lifetime-behavior-of-optionals-notational-conveniences) + * [protocol ExpressibleByNilLiteral](#protocol-expressiblebynilliteral) + * [enum Optional](#enum-optional) + * [enum Result](#enum-result) + * [enum MemoryLayout](#enum-memorylayout) + * [Lifetime Management](#lifetime-management-1) + * [Metatype equality](#metatype-equality) + * [struct ObjectIdentifier](#struct-objectidentifier) + * [ManagedBufferPointer equatability](#managedbufferpointer-equatability) + * [Making indices universally available on unsafe buffer pointers](#making-indices-universally-available-on-unsafe-buffer-pointers) + * [Buffer pointer operations on Slice](#buffer-pointer-operations-on-slice) + * [Source compatibility](#source-compatibility) + * [ABI compatibility](#abi-compatibility) + * [Note on protocol generalizations](#note-on-protocol-generalizations) + * [Alternatives Considered](#alternatives-considered) + * [Future Work](#future-work) + * [Acknowledgements](#acknowledgements) + +## Introduction + +This document proposes to allow `Optional` and `Result` to hold instances of nonescapable types, and continues the work of adding support for noncopyable and nonescapable types throughout the Swift Standard Library. + + +## Motivation + +[SE-0437] started integrating noncopyable types into our Standard Library abstractions, by generalizing existing APIs and introducing new ones. In the time since that proposal, [SE-0446] has introduced nonescapable types to Swift, adding a new direction of generalization. + +This proposal continues the work of [SE-0437] by extending some basic constructs to support nonescapable types, where it is already possible to do so. For now, we are focusing on further generalizing a subset of the constructs covered by [SE-0437]: `MemoryLayout`, `Optional`, and `Result` types. Our immediate aim is to unblock the use of nonescapable types, especially in API surfaces. We also smooth out some minor inconsistencies that [SE-0437] has left unresolved. + +Like before, our aim is to implement these generalizations with as little disruption as possible. Existing code implicitly assumes copyability and escapability, and it needs to continue working as before. + +## Proposed Solution + +This proposal is focusing on achieving the following results: + +- Allow `Optional` to wrap nonescapable types, itself becoming conditionally escapable. +- Do the same for `Result`, allowing its success case to hold a nonescapable item. +- Generalize `MemoryLayout` to allow querying basic information on the memory layout of nonescapable types. +- Continue generalizing basic lifetime management functions; introduce a new `extendLifetime()` function that avoids a closure argument. +- Allow generating `ObjectIdentifier` instances for noncopyable and/or nonescapable metatypes. +- Allow comparing noncopyable and nonescapable metatypes for equality. + +We also propose to fix a handful of minor copyability-related omissions that have been discovered since [SE-0437] was accepted: + +- Allow `ManagedBufferPointer` instances to be equatable even when `Element` is noncopyable. +- Make the `Unsafe[Mutable]BufferPointer.indices` property universally available for all `Element` types. +- Generalize select operations on unsafe buffer pointer slices, to restore consistency with the same operations on the buffer pointers themselves. + +### Nonescapable optionals + +We want `Optional` to support wrapping all Swift types, whether or not they're copyable or escapable. This means that `Optional` needs to become conditionally escapable, depending on the escapability of its `Wrapped` type. + +`Optional` must itself become nonescapable when it is wrapping a nonescapable type, and such optional values need to be subject to precisely the same lifetime constraints as their wrapped item: we cannot allow the act of wrapping a nonescapable value in an optional to allow that value to escape its intended context. + +There are many ways to construct optional values in Swift: for example, we can explicitly invoke the factory for the `.some` case, we can rely on Swift's implicit optional promotion rules, or we can invoke the default initializer. We propose to generalize all of these basic/primitive mechanisms to support nonescapable use. For instance, given a non-optional `Span` value, this code exercises these three basic ways of constructing non-nil nonescapable optionals: + +```swift +func sample(_ span: Span) { + let a = Optional.some(span) // OK, explicit case factory + let b: Optional = span // OK, implicit optional promotion + let c = Optional(span) // OK, explicit initializer invocation +} +``` + +`a`, `b`, and `c` hold the same span instance, and their lifetimes are subject to the same constraints as the original span -- they can be used within the context of the `sample` function, but they cannot escape outside of it. (At least not without explicit lifetime dependency annotations, to be introduced in the future.) + +Of course, it also needs to be possible to make empty `Optional` values that do not hold anything. We have three basic ways to do that: we can explicitly invoke the factory for the `none` case, we can reach for the special `nil` literal, or (for `var`s) we can rely on implicit optional initialization. This proposal generalizes all three mechanisms to support noncopyable wrapped types: + +```swift +func sample(_ span: Span) { + var d: Span? = .none // OK, explicit factory invocation + var e: Span? = nil // OK, nil literal expression + var f: Span? // OK, implicit nil default value +} +``` + +Empty optionals of nonescapable types are still technically nonescapable, but they aren't inherently constrained to any particular context -- empty optionals are born with "immortal" (or "static") lifetimes, i.e., they have no lifetime dependencies, and so they are allowed to stay around for the entire execution of a Swift program. Nil optionals can be passed to any operation that takes a nonescapable optional, no matter what expectations it may dictate about its lifetime dependencies; they can also be returned from any function that returns a nonescapable optional. (Note though that Swift does not yet provide a stable way to define such functions.) + +Of course, we also expect to be able to reassign variables, rebinding them to a new value. Reassignments of local variables are allowed to arbitrarily change lifetime dependencies. There is no expectation that the lifetime dependencies of the new value have any particular relation to the old: local variable reassignments can freely "narrow" or "widen" dependencies, as they see fit. + +For instance, the code below initializes an optional variable to an immortal nil value; it then assigns it a new value that has definite lifetime constraints; and finally it turns it back to an immortal nil value: + +```swift +func sample(_ span: Span) { + var maybe: Span? = nil // immortal + maybe = span // definite lifetime + maybe = nil // immortal again +} +``` + +(Assigning `span` to `maybe` is not an escape, as the local variable will be destroyed before the function returns, even without the subsequent reassignment.) + +This flexibility will not necessarily apply to other kinds of variables, like stored properties in custom nonescapable structs, global variables, or computed properties -- I expect those variables to carry specific lifetime dependencies that cannot vary through reassignment. (For example, a global variable of a nonescapable type may be required to hold immortal values only.) However, for now, we're limiting our reasoning to local variables. + +Of course, an optional is of limited use unless we are able to decide whether it contains a value, and (if so) to unwrap it and look at its contents. We need to be able to operate on nonescapable optionals using the familiar basic mechanisms: + + - `switch` and `if case`/`guard case` statements that pattern match over them: + + ```swift + // Variant 1: Full pattern matching + func count(of maybeSpan: Span?) -> Int { + switch maybeSpan { + case .none: return 0 + case .some(let span): return span.count + } + } + + // Variant 2: Pattern matching with optional sugar + func count(of maybeSpan: Span?) -> Int { + switch maybeSpan { + case nil: return 0 + case let span?: return span.count + } + } + ``` + + - The force-unwrapping `!` special form, and its unsafe cousin, the Standard Library's `unsafelyUnwrapped` property. + + ```swift + func count(of maybeSpan: Span?) -> Int { + if case .none = maybeSpan { return 0 } + return maybeSpan!.count + } + ``` + +- The optional chaining special form `?`: + + ```swift + func count(of maybeSpan: Span?) -> Int { + guard let c = maybeSpan?.count else { return 0 } + return c + } + ``` + +- Optional bindings such as `if let` or `guard let` statements: + + ```swift + func count(of maybeSpan: Span?) -> Int { + guard let span = maybeSpan else { return 0 } + return span.count + } + ``` + +These variants all work as expected. To avoid escapability violations, unwrapping the nonescapable optional results in a value with precisely the same lifetime dependencies as the original optional value. This applies to all forms of unwrapping, including pattern matching forms that bind copies of associated values to new variables, like `let span` above -- the resulting `span` value always has the same lifetime as the optional it comes from. + +The standard `Optional` type has custom support for comparing optional instances against `nil` using the traditional `==` operator, whether or not the wrapped type conforms to `Equatable`. [SE-0437] generalized this mechanism for noncopyable wrapped types, and it is reasonable to extend this to also cover the nonescapable case: + +```swift +func count(of maybeSpan: Span?) -> Int { + if maybeSpan == nil { return 0 } // OK! + return maybeSpan!.count +} +``` + +This core set of functionality makes nonescapable optionals usable, but it does not yet enable the use of more advanced APIs. Eventually, we'd also like to use the standard `Optional.map` function (and similar higher-order functions) to operate on (or to return) nonescapable optional types, as in the example below: + +```swift +func sample(_ maybeArray: Array?) { + // Assuming `Array.storage` returns a nonescapable `Span`: + let maybeSpan = maybeArray.map { $0.storage } + ... +} +``` + +These operations require precise reasoning about lifetime dependencies though, so they have to wait until we have a stable way to express lifetime annotations on their definitions. We expect lifetime semantics to become an integral part of the signatures of functions dealing with nonescapable entities -- for the simplest cases they can often remain implicit, but for something like `map` above, we'll need to explicitly describe how the lifetime of the function's result relates to the lifetime of the result of the function argument. We need to defer this work until we have the means to express such annotations in the language. + +One related omission from the list of generalizations above is the standard nil-coalescing operator `??`. This is currently defined as follows (along with another variant that returns an `Optional`): + +```swift +func ?? ( + optional: consuming T?, + defaultValue: @autoclosure () throws -> T +) rethrows -> T +``` + +To generalize this to also allow nonescapable `T` types, we'll need to specify that the returned value's lifetime is tied to the _intersection_ of the lifetime of the left argument and the lifetime of the result of the right argument (a function). We aren't currently able to express that, so this generalization has to be deferred as well until the advent of such a language feature. + +### Nonescapable `Result` + +We generalize `Result` along the same lines as `Optional`, allowing its `success` case to wrap a nonescapable value. For now, we need to mostly rely on Swift's general enum facilities to operate on nonescapable `Result` values: switch statements, case factories, pattern matching, associated value bindings etc. + +Important convenience APIs such as `Result.init(catching:)` or `Result.map` will need to require escapability until we introduce a way to formally specify lifetime dependencies. This is unfortunate, but it still enables intrepid Swift developers to experiment with defining interfaces that take (or perhaps even return!) `Result` values. + +However, we are already able to generalize a couple of methods: `get` and the error-mapping utility `mapError`. + +```swift +func sample(_ res: Result, E>) -> Int { + guard let span = try? res.get() else { return 42 } + return 3 * span.count + 9 +} +``` + +Like unwrapping an `Optional`, calling `get()` on a nonescapable `Result` returns a value whose lifetime requirements exactly match that of the original `Result` instance -- the act of unwrapping a result cannot allow its content to escape its intended context. + +### Retrieving the memory layout of nonescapable types + +This proposal generalizes `enum MemoryLayout` to support retrieving information about the layout of nonescapable types: + +```swift +print(MemoryLayout>.size) // ⟹ 16 +print(MemoryLayout>.stride) // ⟹ 16 +print(MemoryLayout>.alignment) // ⟹ 8 +``` + +(Of course, the values returned will vary depending on the target architecture.) + +The information returned is going to be of somewhat limited use until we generalize unsafe pointer types to support nonescapable pointees, which this proposal does not include -- but there is no reason to delay this work until then. + +To usefully allow pointers to nonescapable types, we'll need to assign precise lifetime semantics to their `pointee` (and pointer dereferencing in general), and we'll most likely also need a way to allow developers to unsafely override the resulting default lifetime semantics. This requires explicit lifetime annotations, and as such, that work is postponed to a future proposal. + +### Lifetime management + +We once again generalize the `withExtendedLifetime` family of functions, this time to support calling them on nonescapable values. + +```swift +let span = someArray.storage +withExtendedLifetime(span) { span in + // `someArray` is being actively borrowed while this closure is running +} +// At this point, `someArray` may be ready to be mutated +``` + +We've now run proposals to generalize `withExtendedLifetime` for (1) typed throws, (2) noncopyable inputs and results, and (3) nonescapable inputs. It is getting unwieldy to keep having to tweak these APIs, especially since in actual practice, `withExtendedLifetime` is most often called with an empty closure, to serve as a sort of fence protecting against early destruction. The closure-based design of these interfaces are no longer fitting the real-life practices of Swift developers. These functions were originally designed to be used with a non-empty closure, like in the example below: + +```swift +withExtendedLifetime(obj) { + weak var ref = obj + foo(ref!) +} +``` + +In most cases, the formulation we actually recommend these days is to use a defer statement, with the function getting passed an empty closure: + +```swift +weak var ref = obj +defer { withExtendedLifetime(obj) {} } // Ugh 😖 +foo(ref!) +``` + +These functions clearly weren't designed to accommodate this widespread practice. To acknowledge and embrace this new style, we propose to introduce a new public Standard Library function that simply extends the lifetime of whatever variable it is given: + +```swift +func extendLifetime(_ x: borrowing T) +``` + +This allows `defer` incantations like the one above to be reformulated into a more readable form: + +```swift +// Slightly improved reality +weak var ref = obj +defer { extendLifetime(obj) } +foo(ref!) +``` + +To avoid disrupting working code, this proposal does not deprecate the existing closure-based functions in favor of the new `extendLifetime` operation. (Introducing the new function will still considerably reduce the need for future Swift releases to continue repeatedly generalizing the existing functions -- for example, to allow async use, or to allow nonescapable results.) + +### Metatype comparisons + +Swift's metatypes do not currently conform to `Equatable`, but the Standard Library still provides top-level `==` and `!=` operators that implement the expected equality relation. Previously, these operators only worked on metatypes of `Copyable` and `Escapable` types; we propose to relax this requirement. + +```swift +print(Atomic.self == Span.self) // ⟹ false +``` + +The classic operators support existential metatypes `Any.Type`; the new variants also accept generalized existentials: + +```swift +let t1: any (~Copyable & ~Escapable).Type = Atomic.self +let t2: any (~Copyable & ~Escapable).Type = Span.self +print(t1 != t2) // ⟹ true +print(t1 == t1) // ⟹ true +``` + +### Object identifiers + +The `ObjectIdentifier` construct is primarily used to generate a Comparable/Hashable value that identifies a class instance. However, it is also able to identify metatypes: + +```swift +let id1 = ObjectIdentifier(Int.self) +let id2 = ObjectIdentifier(String.self) +print(id1 == id2) // ⟹ false +``` + +[SE-0437] did not generalize this initializer; we can now allow it to work with both noncopyable and nonescapable types: + +```swift +import Synchronization + +let id3 = ObjectIdentifier(Atomic.self) // OK, noncopyable input type +let id4 = ObjectIdentifier(Span.self) // OK, nonescapable input type +print(id3 == id4) // ⟹ false +``` + +The object identifier of a noncopyable/nonescapable type is still a regular copyable and escapable identifier -- for instance, it can be compared against other ids and hashed. + +### Odds and ends + +[SE-0437] omitted generalizing the `Equatable` conformance of `ManagedBufferPointer`; this proposal allows comparing `ManagedBufferPointer` instances for equality even if their `Element` happens to be noncopyable. + +[SE-0437] kept the `indices` property of unsafe buffer pointer types limited to cases where `Element` is copyable. This proposal generalizes `indices` to be also available on buffer pointers of noncopyable elements. (In the time since the original proposal, [SE-0447] has introduced a `Span` type that ships with an unconditional `indices` property, and [SE-0453] followed suit by introducing `InlineArray` with the same property. It makes sense to also provide this interface on buffer pointers, for consistency.) `indices` is useful for iterating through these collection types, especially until we ship a new iteration model that supports noncopyable/nonescapable containers. + +Finally, [SE-0437] neglected to generalize any of the buffer pointer operations that [SE-0370] introduced on the standard `Slice` type. In this proposal, we correct this omission by generalizing the handful of operations that can support noncopyable result elements: `moveInitializeMemory(as:fromContentsOf:)`, `bindMemory(to:)`, `withMemoryRebound(to:_:)`, and `assumingMemoryBound(to:)`. `Slice` itself continues to require its `Element` to be copyable (at least for now), preventing the generalization of other operations. + +## Detailed Design + +Note that Swift provides no way to define the lifetime dependencies of a function's nonescapable result, nor to set lifetime constraints on input parameters. Until the language gains an official way to express such constraints, the Swift Standard Library will define the APIs generalized in this proposal with unstable syntax that isn't generally available. In this text, we'll be using an illustrative substitute -- the hypothetical `@_lifetime` attribute. We will loosely describe its meaning as we go. + +Note: The `@_lifetime` attribute is not real; it is merely a didactic placeholder. The eventual lifetime annotations proposal may or may not propose syntax along these lines. We expect the Standard Library to immediately switch to whatever syntax Swift eventually embraces, as soon as it becomes available. + +### Inferred lifetime behavior of nonescapable enum types + +[SE-0446] has introduced the concept of a nonescapable enum type to Swift. While that proposal did not explicitly spell this out, this inherently included a set of implicit lifetime rules for the principal language features that interact with enum types: enum construction using case factories and pattern matching. To generalize `Optional` and `Result`, we need to understand how these implicit inference rules work for enum types with a single nonescapable associated value. + +1. When constructing an enum case with a single nonescapable associated value, the resulting enum value is inferred to carry precisely the same lifetime dependencies as the origional input. +2. Pattern matching over such an enum case exposes the nonescapable associated value, inferring precisely the same lifetime dependencies for it as the original enum. + +```swift +enum Foo { + case a(T) + case b +} + +func test(_ array: Array) { + let span = array.span + let foo = Foo.a(span) // (1) + switch foo { + case .a(let span2): ... // (2) + case .b: ... + } +} +``` + +In statement (1), `foo` is defined to implicitly copy the lifetime dependencies of `span`; neither variable can escape the body of the `test` function. The let binding in the pattern match on `.a` in statement (2) creates `span2` with exactly the same lifetime dependencies as `foo`. + +(We do not describe the implicit semantics of enum cases with _multiple_ nonescapable associated values here, as they are relevant to neither `Optional` nor `Result`.) + +### Inferred lifetime behavior of `Optional`'s notational conveniences + +The `Optional` enum comes with a rich set of notational conveniences that are built directly into the language. This proposal extends these conveniences to work on nonescapable optionals; therefore it inherently needs to introduce new implicit lifetime inference rules, along the same lines as the two existing once we described above: + +1. The result of implicit optional promotion of a nonescapable value is a nonescapable optional carrying precisely the same lifetime dependencies as the original input. +2. The force-unwrapping special form `!` and the optional chaining special form `?` both implicitly infer the lifetime dependencies of the wrapped value (if any) by directly copying those of the optional. + +### `protocol ExpressibleByNilLiteral` + +In order to generalize `Optional`, we need the `ExpressibleByNilLiteral` protocol to support nonescapable conforming types. By definition, the `nil` form needs to behave like a regular, escapable value; accordingly, the required initializer needs to establish "immortal" or "static" lifetime semantics on the resulting instance. + +```swift +protocol ExpressibleByNilLiteral: ~Copyable, ~Escapable { + @_lifetime(immortal) // Illustrative syntax + init(nilLiteral: ()) +} +``` + +In this illustration, `@_lifetime(immortal)` specifies that the initializer places no constraints on the lifetime of its result. We expect a future proposal to define a stable syntax for expressing such lifetime dependency constraints. + +Preexisting types that conform to `ExpressibleByNilLiteral` are all escapable, and escapable values always have immortal lifetimes, by definition. Therefore, initializer implementations in existing conformances already satisfy this new refinement of the initializer requirement -- it only makes a difference in the newly introduced `~Escapable` case. + +### `enum Optional` + +We generalize `Optional` to allow nonescapable wrapped types in addition to noncopyable ones. + +```swift +enum Optional: ~Copyable, ~Escapable { + case none + case some(Wrapped) +} + +extension Optional: Copyable where Wrapped: Copyable & ~Escapable {} +extension Optional: Escapable where Wrapped: Escapable & ~Copyable {} +extension Optional: BitwiseCopyable where Wrapped: BitwiseCopyable & ~Escapable {} +extension Optional: Sendable where Wrapped: ~Copyable & ~Escapable & Sendable {} +``` + +To allow the use of the `nil` syntax with nonescapable optional types, we generalize `Optional`'s conformance to `ExpressibleByNilLiteral`: + +```swift +extension Optional: ExpressibleByNilLiteral +where Wrapped: ~Copyable & ~Escapable { + @_lifetime(immortal) // Illustrative syntax + init(nilLiteral: ()) +} +``` + +As discussed above, `nil` optionals have no lifetime dependencies, and they continue to work like escapable values. + +We need to generalize the existing unlabeled initializer to support the nonescapable case. When passed a nonescapable entity, the initializer creates an optional that has precisely the same lifetime dependencies as the original entity. Once again, Swift has not yet provided a stable way to express this dependency; so to define such an initializer, the Standard Library needs to use some unstable mechanism. We use the hypothetical `@_lifetime(copying some)` syntax to do this -- this placeholder notation is intended to reflect that the lifetime dependencies of the result are copied verbatim from the `some` argument. + +```swift +extension Optional where Wrapped: ~Copyable & ~Escapable { + @_lifetime(copying some) // Illustrative syntax + init(_ some: consuming Wrapped) +} +``` + +As we've seen, the language also has built-in mechanisms for constructing `Optional` values that avoid invoking this initializer: it implements implicit optional promotions and explicit case factories. When given values of nonescapable types, these methods also _implicitly_ result in the result's lifetime dependencies being copied directly from the original input. + +Swift offers many built-in ways for developers to unwrap optional values: we have force unwrapping, optional chaining, pattern matching, optional bindings, etc. Many of these rely on direct compiler support that is already able to properly handle lifetime matters; but the stdlib also includes its own forms of unwrapping, and these require some API changes. + +In this proposal, we generalize `take()` to work on nonescapable optionals. It resets `self` to nil and returns its original value with precisely the same lifetime dependency as we started with. The `nil` value it leaves behind is still constrained to the same lifetime -- we do not have a way for a mutating function to affect the lifetime dependencies of its `self` argument. + +```swift +extension Optional where Wrapped: ~Copyable & ~Escapable { + @_lifetime(copying self) // Illustrative syntax + mutating func take() -> Self +} +``` + +We are also ready to generalize the `unsafelyUnwrapped` property: + +```swift +extension Optional where Wrapped: ~Escapable { + @_lifetime(copying self) // Illustrative syntax + var unsafelyUnwrapped: Wrapped { get } +} +``` + +This property continues to require copyability for now, as supporting noncopyable wrapped types requires the invention of new accessors that hasn't happened yet. + +As noted above, we defer generalizing the nil-coalescing operator `??`. We expect to tackle it when it becomes possible to express the lifetime dependency of its result as an intersection of the lifetimes of its left argument and the _result_ of the right argument (an autoclosure). We also do not attempt to generalize similar higher-order API, like `Optional.map` or `.flatMap`. + +The Standard Library provides special support for comparing arbitrary optional values against `nil`. We generalize this mechanism to support nonescapable cases: + +```swift +extension Optional where Wrapped: ~Copyable & ~Escapable { + static func ~=( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool + + static func ==( + lhs: borrowing Wrapped?, + rhs: _OptionalNilComparisonType + ) -> Bool + + static func !=( + lhs: borrowing Wrapped?, + rhs: _OptionalNilComparisonType + ) -> Bool + + static func ==( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool + + static func !=( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool +} +``` + +### `enum Result` + +For `Result`, this proposal concentrates on merely allowing the success case to contain a nonescapable value. + +```swift +enum Result { + case success(Success) + case failure(Failure) +} + +extension Result: Copyable where Success: Copyable & ~Escapable {} +extension Result: Escapable where Success: Escapable & ~Copyable {} +extension Result: Sendable where Success: Sendable & ~Copyable & ~Escapable {} +``` + +We postpone generalizing most of the higher-order functions that make `Result` convenient to use, as we currently lack the means to reason about lifetime dependencies for such functions. But we are already able to generalize the one function that does not have complicated lifetime semantics: `mapError`. + +```swift +extension Result where Success: ~Copyable & ~Escapable { + @_lifetime(copying self) // Illustrative syntax + consuming func mapError( + _ transform: (Failure) -> NewFailure + ) -> Result +} +``` + +The returned value has the same lifetime constraints as the original `Result` instance. + +We can also generalize the convenient `get()` function, which is roughly equivalent to optional unwrapping: + +```swift +extension Result where Success: ~Copyable & ~Escapable { + @_lifetime(copying self) // Illustrative syntax + consuming func get() throws(Failure) -> Success +} +``` + +In the non-escapable case, this function returns a value with a lifetime that precisely matches the original `Result`. + +### `enum MemoryLayout` + +Swift is not yet ready to introduce pointers to nonescapable values -- we currently lack the ability to assign proper lifetime semantics to the addressed items. + +However, a nonescapable type does still have a well-defined memory layout, and it makes sense to allow developers to query the size, stride, and alignment of such instances. This information is associated with the type itself, and it is independent of the lifetime constraints of its instances. Therefore, we can generalize the `MemoryLayout` enumeration to allow its subject to be a nonescapable type: + +```swift +enum MemoryLayout +: ~BitwiseCopyable, Copyable, Escapable {} + +extension MemoryLayout where T: ~Copyable & ~Escapable { + static var size: Int { get } + static var stride: Int { get } + static var alignment: Int { get } +} + +extension MemoryLayout where T: ~Copyable & ~Escapable { + static func size(ofValue value: borrowing T) -> Int + static func stride(ofValue value: borrowing T) -> Int + static func alignment(ofValue value: borrowing T) -> Int +} +``` + +### Lifetime Management + +[SE-0437] generalized the `withExtendedLifetime` family of functions to support extending the lifetime of noncopyable entities. This proposal further generalizes these to also allow operating on nonescapable entities: + +```swift +func withExtendedLifetime< + T: ~Copyable & ~Escapable, + E: Error, + Result: ~Copyable +>( + _ x: borrowing T, + _ body: () throws(E) -> Result +) throws(E) -> Result + +func withExtendedLifetime< + T: ~Copyable & ~Escapable, + E: Error, + Result: ~Copyable +>( + _ x: borrowing T, + _ body: (borrowing T) throws(E) -> Result +) throws(E) -> Result +``` + +Note that the `Result` is still required to be escapable. + +We also propose the addition of a new function variant that eliminates the closure argument, to better accommodate the current best practice of invoking these functions in `defer` blocks: + +```swift +func extendLifetime(_ x: borrowing T) +``` + +### Metatype equality + +Swift's metatypes do not conform to `Equatable`, but the Standard Library does implement the `==`/`!=` operators over them: + +```swift +func == (t0: Any.Type?, t1: Any.Type?) -> Bool { ... } +func != (t0: Any.Type?, t1: Any.Type?) -> Bool { ... } +``` + +Note how these are defined on optional metatype existentials, typically relying on implicit optional promotion. We propose to generalize these to support metatypes of noncopyable and/or nonescapable types: + +```swift +func == ( + t0: (any (~Copyable & ~Escapable).Type)?, + t1: (any (~Copyable & ~Escapable).Type)? +) -> Bool { ... } +func != ( + t0: (any (~Copyable & ~Escapable).Type)?, + t1: (any (~Copyable & ~Escapable).Type)? +) -> Bool { ... } +``` + +### `struct ObjectIdentifier` + +The `ObjectIdentifier` construct is primarily used to generate a `Comparable`/`Hashable` value that identifies a class instance. However, it is also able to generate hashable type identifiers: + +```swift +extension ObjectIdentifier { + init(_ x: Any.Type) +} +``` + +We propose to generalize this initializer to allow generating identifiers for noncopyable and nonescapable types as well, using generalized metatype existentials: + +```swift +extension ObjectIdentifier { + init(_ x: any (~Copyable & ~Escapable).Type) +} +``` + +### `ManagedBufferPointer` equatability + +The `ManagedBufferPointer` type conforms to `Equatable`; its `==` implementation works by comparing the identity of the class instances it is referencing. [SE-0437] has generalized the type to allow a noncopyable `Element` type, but it did not generalize this specific conformance. This proposal aims to correct this oversight: + +```swift +extension ManagedBufferPointer: Equatable where Element: ~Copyable { + static func ==( + lhs: ManagedBufferPointer, + rhs: ManagedBufferPointer + ) -> Bool +} +``` + +Managed buffer pointers are pointer types -- as such, they can be compared whether or not they are addressing a buffer of copyable items. + +(Note: conformance generalizations like this can cause compatibility issues when newly written code is deployed on older platforms that pre-date the generalization. We do not expect this to be an issue in this case, as the generalization is compatible with the implementations we previously shipped.) + +### Making `indices` universally available on unsafe buffer pointers + +[SE-0437] kept the `indices` property of unsafe buffer pointer types limited to cases where `Element` is copyable. In the time since that proposal, [SE-0447] has introduced a `Span` type that ships with an unconditional `indices` property, and [SE-0453] followed suit by introducing `InlineArray` with the same property. For consistency, it makes sense to also allow developers to unconditionally access `Unsafe[Mutable]BufferPointer.indices`, whether or not `Element` is copyable. + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + var indices: Range { get } +} + +extension UnsafeMutableBufferPointer where Element: ~Copyable { + var indices: Range { get } +} +``` + +This allows Swift programmers to iterate over the indices of a buffer pointer with simpler syntax, independent of what `Element` they are addressing: + +```swift +for i in buf.indices { + ... +} +``` + +We consider `indices` to be slightly more convenient than the equivalent expression `0 ..< buf.count`. + +(Of course, we are still planning to introduce direct support for for-in loops over noncopyable/nonescapable containers, which will provide a far more flexible solution. `indices` is merely a stopgap solution to bide us over until we are ready to propose that.) + +### Buffer pointer operations on `Slice` + +Finally, to address an inconsistency that was left unresolved by [SE-0437], we generalize a handful of buffer pointer operations that are defined on buffer slices. This consists of the following list, originally introduced in [SE-0370]: + +- Initializing a slice of a mutable raw buffer pointer by moving items out of a typed mutable buffer pointer: + + ```swift + extension Slice where Base == UnsafeMutableRawBufferPointer { + func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + } + ``` + +- Binding memory of raw buffer pointer slices: + + ```swift + extension Slice where Base == UnsafeMutableRawBufferPointer { + func bindMemory( + to type: T.Type + ) -> UnsafeMutableBufferPointer + } + + extension Slice where Base == UnsafeRawBufferPointer { + func bindMemory( + to type: T.Type + ) -> UnsafeBufferPointer + } + ``` + +- Temporarily rebinding memory of a (typed or untyped, mutable or immutable) buffer pointer slice for the duration of a function call: + + ```swift + extension Slice where Base == UnsafeMutableRawBufferPointer { + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeMutableBufferPointer) throws(E) -> Result + ) throws(E) -> Result + } + + extension Slice where Base == UnsafeRawBufferPointer { + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeBufferPointer) throws(E) -> Result + ) throws(E) -> Result + } + + extension Slice { + func withMemoryRebound< + T: ~Copyable, E: Error, Result: ~Copyable, Element + >( + to type: T.Type, + _ body: (UnsafeBufferPointer) throws(E) -> Result + ) throws(E) -> Result + where Base == UnsafeBufferPointer + + public func withMemoryRebound< + T: ~Copyable, E: Error, Result: ~Copyable, Element + >( + to type: T.Type, + _ body: (UnsafeMutableBufferPointer) throws(E) -> Result + ) throws(E) -> Result + where Base == UnsafeMutableBufferPointer + } + ``` + +- Finally, converting a slice of a raw buffer pointer into a typed buffer pointer, assuming its memory is already bound to the correct type: + + ```swift + extension Slice where Base == UnsafeMutableRawBufferPointer { + func assumingMemoryBound( + to type: T.Type + ) -> UnsafeMutableBufferPointer + } + + extension Slice where Base == UnsafeRawBufferPointer { + func assumingMemoryBound( + to type: T.Type + ) -> UnsafeBufferPointer + } + ``` + +All of these forward to operations on the underlying base buffer pointer that have already been generalized in [SE-0437]. These changes are simply restoring feature parity between buffer pointer and their slices, where possible. (`Slice` still requires its `Element` to be copyable, which limits generalization of other buffer pointer APIs defined on it.) + +These generalizations are limited to copyability for now. We do expect that pointer types (including buffer pointers) will need to be generalized to allow non-escapable pointees; however, we have to postpone that work until we are able to precisely reason about lifetime requirements. + + + +## Source compatibility + +Like [SE-0437], this proposal also heavily relies on the assurance that removing the assumption of escapability on these constructs will not break existing code that used to rely on the original, escaping definitions. [SE-0437] has explored a few cases where this may not be the case; these can potentially affect code that relies on substituting standard library API with its own implementations. With the original ungeneralized definitions, such custom reimplementations could have shadowed the originals. However, this may no longer be the case with the generalizations included, and this can lead to ambiguous function invocations. + +This proposal mostly touches APIs that were already changed by [SE-0437], and that reduces the likelihood of it causing new issues. That said, it does generalize some previously unchanged interfaces that may provide new opportunities for such shadowing declarations to cause trouble. + +Like previously, we do have engineering options to mitigate such issues in case we do encounter them in practice: for example, we can choose to amend Swift's shadowing rules to ignore differences in throwing, noncopyability, and nonescapability, or we can manually patch affected definitions to make the expression checker consider them to be less specific than any custom overloads. + +## ABI compatibility + +The introduction of support for nonescapable types is (in general) a compile-time matter, with minimal (or even zero) runtime impact. This greatly simplifies the task of generalizing previously shipping types for use in nonescapable contexts. Another simplifying aspect is that while it can be relatively easy for classic Swift code to accidentally copy a value, it tends to be rare for functions to accidentally _escape_ their arguments -- previous versions of a function are less likely to accidentally violate nonescapability than noncopyability. + +The implementation of this proposal adopts the same approaches as [SE-0437] to ensure forward and backward compatibility of newly compiled (and existing) binaries, including the Standard Library itself. We expect that code that exercises the new features introduced in this proposal will be able to run on earlier versions of the Swift stdlib -- to the extent that noncopyable and/or nonescapable types are allowed to backdeploy. + +[SE-0437] has already arranged ABI compatibility symbols to get exported as needed to support ABI continuity. It has also already reimplemented most of the entry points that this proposal touches, in a way that forces them to get embedded in client binaries. This allows the changes in this proposal to get backdeployed without any additional friction. + +Like its precursor, this proposal does assume that the `~Copyable`/`~Escapable` generalization of the `ExpressibleByNilLiteral` protocol will not have an ABI impact on existing conformers of it. However, it goes a step further, by also adding a lifetime annotation on the protocol's initializer requirement; this requires that such annotations must not interfere with backward/forward binary compatibility, either. (E.g., it requires that such lifetime annotations do not get mangled into exported symbol names.) + +### Note on protocol generalizations + +Like [SE-0437], this proposal mostly avoids generalizing standard protocols, with the sole exception of `ExpressibleByNilLiteral`, which has now been generalized to allow both noncopyable and nonescapable conforming types. + +As a general rule, protocol generalizations like that may not be arbitrarily backdeployable -- it seems likely that we'll at least need to support limiting the availability of _conformance_ generalizations, if not generalizations of the protocol itself. In this proposal, we follow [SE-0437] in assuming that this potential issue will not apply to the specific case of `ExpressibleByNilLiteral`, because of its particularly narrow use case. Our experience with [SE-0437] is reinforcing this assumption, but it is still possible there is an ABI back-compatibility issue that we haven't uncovered yet. In the (unlikely, but possible) case we do discover such an issue, we may need to do extra work to patch protocol conformances in earlier stdlibs, or we may decide to limit the use of `nil` with noncopyable/nonescapable optionals to recent enough runtimes. + +To illustrate the potential problem, let's consider `Optional`'s conformance to `Equatable`: + +```swift +extension Optional: Equatable where Wrapped: Equatable { + public static func ==(lhs: Wrapped?, rhs: Wrapped?) -> Bool { + switch (lhs, rhs) { + case let (l?, r?): return l == r + case (nil, nil): return true + default: return false + } + } +} +``` + +This conformance is currently limited to copyable and escapable cases, and it is using the classic, copying form of the switch statement, with `case let (l?, r?)` semantically making full copies of the two wrapped values. We do intend to soon generalize the `Equatable` protocol to support noncopyable and/or nonescapable conforming types. When that becomes possible, `Optional` will want to immediately embrace this generalization, to allow comparing two noncopyable/nonescapable instances for equality: + +```swift +extension Optional: Equatable where Wrapped: Equatable & ~Copyable & ~Escapable { + public static func ==(lhs: borrowing Wrapped?, rhs: borrowing Wrapped?) -> Bool { + switch (lhs, rhs) { + case let (l?, r?): return l == r + case (nil, nil): return true + default: return false + } + } +} +``` + +On the surface, this seems like a straightforward change. Unfortunately, switching to `borrowing` arguments changes the semantics of the implementation, converting the original copying switch statement to the borrowing form introduced by [SE-0432]. This new variant avoids copying wrapped values to compare them, enabling the use of this function on noncopyable data. However, the old implementation of `==` did assume (and exercise!) copyability, so the `Equatable` conformance cannot be allowed to dispatch to `==` implementations that shipped in Standard Library releases that predate this generalization. + +To mitigate such problems, we'll either need to retroactively patch/substitute the generic implementations in previously shipping stdlibs, or we need to somehow limit availability of the generalized conformance, without affecting the original copyable/escapable one. + +This issue is more pressing for noncopyable cases, as preexisting implementations are far more likely to perform accidental copying than to accidentally escape their arguments. + +Our hypothesis is that `ExpressibleByNilLiteral` conformances are generally free of such issues. + +## Alternatives Considered + +Most of the changes proposed here follow directly from the introduction of nonescapable types. The API generalizations follow the patterns established by [SE-0437], and are largely mechanical in nature. For the most part, the decision points aren't about the precise form of any particular change, but more about what changes we are ready to propose _right now_. + +The single exception is the `extendLifetime` function, which is a brand new API; it comes from our experience using (and maintaining) the `withExtendedLifetime` function family. + +## Future Work + +For the most part, this proposal is concentrating on resolving the first item from [SE-0437]'s wish list (nonescapable `Optional` and `Result`), and it adds minor coherency improvements to the feature set we shipped there. + +Most other items listed as future work in that proposal continue to remain on our agenda. The advent of nonescapable types extends this list with additional items, including the following topics: + +1. We need to define stable syntax for expressing lifetime dependencies as explicit annotations, and we need to define what semantics we apply by default on functions that do not explicitly specify these. + +2. We will need an unsafe mechanism to override lifetime dependencies of nonescapable entities. We also expect to eventually need to allow unsafe bit casting to and from nonescapable types. + +3. We will need to allow pointer types to address nonescapable items: `UnsafePointer`, `UnsafeBufferPointer` type families, perhaps `ManagedBuffer`. The primary design task here is to decide what lifetime semantics we want to assign to pointer dereferencing operations, including mutations. + +4. Once we have pointers, we will also need to allow the construction of generic containers of nonescapable items, with some Sequence/Collection-like capabilities (iteration, indexing, generic algorithms, etc.). We expect the noncopyable/nonescapable container model to heavily rely on the `Span` type, which we intend to use as the basic unit of iteration, providing direct access to contiguous storage chunks. For containers of nonescapables in particular, this means we'll also need to generalize `Span` to allow it to capture nonescapable elements. + +5. We'll want to generalize most of the preexisting standard library protocols to allow nonescapable conforming types and (if possible) associated types. This is in addition to supporting noncopyability. This work will require adding carefully considered lifetime annotations on protocol requirements, while also carefully maintaining seamless forward/backward compatibility with the currently shipping protocol versions. This is expected to take several proposals; in some cases, it may include carefully reworking existing semantic requirements to better match noncopyable/nonescapable use cases. Some protocols may not be generalizable without breaking existing code; in those cases, we may need to resort to replacing or augmenting them with brand-new protocols. However, protocol generalizations for nonescapables are generally expected to be a smoother process than it is for noncopyables. + +## Acknowledgements + +Many people contributed to the discussions that led to this proposal. We'd like to especially thank the following individuals for their continued, patient and helpful input: + +- Alejandro Alonso +- Steve Canon +- Ben Cohen +- Kavon Farvardin +- Doug Gregor +- Joe Groff +- Megan Gupta +- Tim Kientzle +- Guillaume Lessard +- John McCall +- Tony Parker +- Ben Rimmington +- Andrew Trick +- Rauhul Varma diff --git a/proposals/0466-control-default-actor-isolation.md b/proposals/0466-control-default-actor-isolation.md new file mode 100644 index 0000000000..5a1e975d25 --- /dev/null +++ b/proposals/0466-control-default-actor-isolation.md @@ -0,0 +1,255 @@ +# Control default actor isolation inference + +* Proposal: [SE-0466](0466-control-default-actor-isolation.md) +* Authors: [Holly Borla](https://github.com/hborla), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Active review (July 8...15, 2025)** +* Vision: [Improving the approachability of data-race safety](/visions/approachable-concurrency.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-control-default-actor-isolation-inference/77482))([review](https://forums.swift.org/t/se-0466-control-default-actor-isolation-inference/78321))([acceptance](https://forums.swift.org/t/accepted-se-0466-control-default-actor-isolation-inference/78926))([amendment pitch](https://forums.swift.org/t/pitch-amend-se-0466-se-0470-to-improve-isolation-inference/79854))([amendment review](https://forums.swift.org/t/amendment-se-0466-control-default-actor-isolation-inference/80994)) + +## Introduction + +This proposal introduces a new compiler setting for inferring `@MainActor` isolation by default within the module to mitigate false-positive data-race safety errors in sequential code. + +## Motivation + +> Note: This motivation section was adapted from the [vision for approachable data-race safety](https://github.com/hborla/swift-evolution/blob/approachable-concurrency-vision/visions/approachable-concurrency.md#mitigating-false-positive-data-race-safety-errors-in-sequential-code). Please see the vision document for extended motivation. + +A lot of code is effectively “single-threaded”. For example, most executables, such as apps, command-line tools, and scripts, start running on the main actor and stay there unless some part of the code does something concurrent (like creating a `Task`). If there isn’t any use of concurrency, the entire program will run sequentially, and there’s no risk of data races — every concurrency diagnostic is necessarily a false positive! It would be good to be able to take advantage of that in the language, both to avoid annoying programmers with unnecessary diagnostics and to reinforce progressive disclosure. Many people get into Swift by writing these kinds of programs, and if we can avoid needing to teach them about concurrency straight away, we’ll make the language much more approachable. + +The easiest and best way to model single-threaded code is with a global actor. Everything on a global actor runs sequentially, and code that isn’t isolated to that actor can’t access the data that is. All programs start running on the global actor `MainActor`, and if everything in the program is isolated to the main actor, there shouldn’t be any concurrency errors. + +Unfortunately, it’s not quite that simple right now. Writing a single-threaded program is surprisingly difficult under the Swift 6 language mode. This is because Swift 6 defaults to a presumption of concurrency: if a function or type is not annotated or inferred to be isolated, it is treated as non-isolated, meaning it can be used concurrently. This default often leads to conflicts with single-threaded code, producing false-positive diagnostics in cases such as: + +- global and static variables, +- conformances of main-actor-isolated types to non-isolated protocols, +- class deinitializers, +- overrides of non-isolated superclass methods in a main-actor-isolated subclass, and +- calls to main-actor-isolated functions from the platform SDK. + +## Proposed solution + +This proposal allows code to opt in to being “single-threaded” by default, on a module-by-module basis. A new `-default-isolation` compiler flag specifies the default isolation within the module, and a corresponding `SwiftSetting` method specifies the default isolation per target within a Swift package. + +This would change the default isolation rule for unannotated code in the module: rather than being non-isolated, and therefore having to deal with the presumption of concurrency, the code would instead be implicitly isolated to `@MainActor`. Code imported from other modules would be unaffected by the current module’s choice of default. When the programmer really wants concurrency, they can request it explicitly by marking a function or type as `nonisolated` (which can be used on any declaration as of [SE-0449](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0449-nonisolated-for-global-actor-cutoff.md)), or they can define it in a module that doesn’t default to main-actor isolation. + +## Detailed design + +### Specifying default isolation per module + +#### `-default-isolation` compiler flag + +The `-default-isolation` flag can be used to control the default actor isolation for all code in the module. The only valid arguments to `-default-isolation` are `MainActor` and `nonisolated`. It is an error to specify both `-default-isolation MainActor` and `-default-isolation nonisolated`. If no `-default-isolation` flag is specified, the default isolation for the module is `nonisolated`. + +#### `SwiftSetting.defaultIsolation` method + +The following method on `SwiftSetting` can be used to specify the default actor isolation per target in a Swift package manifest: + +```swift +extension SwiftSetting { + @available(_PackageDescription, introduced: 6.2) + public static func defaultIsolation( + _ globalActor: MainActor.Type?, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting +} +``` + +The only valid values for the `globalActor` argument are `MainActor.self` and `nil`. The `nil` argument corresponds to `nonisolated`; `.defaultIsolation(nil)` will default to `nonisolated` within the module. When no `.defaultIsolation` setting is specified, the default isolation within the module is `nonisolated`. + +### Default actor isolation inference + +When the default actor isolation is specified as `MainActor`, declarations are inferred to be `@MainActor`-isolated by default. Default isolation does not apply in the following cases: + +* Declarations with explicit actor isolation +* Declarations with inferred actor isolation from a superclass, overridden method, protocol conformance, or member propagation +* All declarations inside an `actor` type, including static variables, methods, initializers, and deinitializers +* Declarations that cannot have global actor isolation, including typealiases, import statements, enum cases, and individual accessors +* Declarations whose primary definition directly conforms to a protocol that inherits `SendableMetatype` +* Declarations that are types nested within a nonisolated type + +The following code example shows the inferred actor isolation in comments given the code is built with `-default-isolation MainActor`: + +```swift +// @MainActor +func f() {} + +// @MainActor +class C { + // @MainActor + init() { ... } + + // @MainActor + deinit { ... } + + // @MainActor + struct Nested { ... } + + // @MainActor + static var value = 10 +} + +@globalActor +actor MyActor { + // nonisolated + init() { ... } + + // nonisolated + deinit { ... } + + // nonisolated + static let shared = MyActor() +} + +@MyActor +protocol P {} + +// @MyActor +struct S: P { + // @MyActor + func f() { ... } +} + +nonisolated protocol Q: Sendable { } + +// nonisolated +struct S2: Q { + // nonisolated + struct Inner { } + + // @MyActor + struct IsolatedInner: P +} + +// @MainActor +struct S3 { } + +extension S3: Q { } +``` + +This proposal does not change the default isolation inference rules for closures. Non-Sendable closures and closures passed to `Task.init` already have the same isolation as the enclosing context by default. When specifying `MainActor` isolation by default in a module, non-`@Sendable` closures and `Task.init` closures will have inferred `@MainActor` isolation when the default `@MainActor` inference rules apply to the enclosing context: + +```swift +// Built with -default-isolation MainActor + +// @MainActor +func f() { + Task { // @MainActor in + ... + } + + Task.detached { // nonisolated in + ... + } +} + +nonisolated func g() { + Task { // nonisolated in + ... + } +} +``` + +## Source compatibility + +Changing the default actor isolation for a given module or source file is a source incompatible change. The default isolation will remain the same for existing projects unless they explicitly opt into `@MainActor` inference by default via `-default-isolation MainActor` or `defaultIsolation(MainActor.self)` in a package manifest. + +## ABI compatibility + +This proposal has no ABI impact on existing code. + +## Implications on adoption + +This proposal does not change the adoption implications of adding `@MainActor` to a declaration that was previously `nonisolated` and vice versa. The source and ABI compatibility implications of changing actor isolation are documented in the Swift migration guide's [Library Evolution](https://github.com/apple/swift-migration-guide/blob/29d6e889e3bd43c42fe38a5c3f612141c7cefdf7/Guide.docc/LibraryEvolution.md#main-actor-annotations) article. + +## Future directions + +### Specify build settings per file + +There are some build settings that are applicable on a per-file basis, including specifying default actor isolation and controlling diagnostic behavior. We could consider allowing settings in individual files which the setting should apply to by introducing a `#pragma`-like compiler directive. This idea has been [pitched separately](https://forums.swift.org/t/pitch-compilersettings-a-top-level-statement-for-enabling-compiler-flags-locally-in-a-specific-file/77994). + +## Alternatives considered + +### Allow defaulting isolation to a custom global actor + +The `-default-isolation` flag could allow a custom global actor as the argument, and the `SwiftSetting` API could be updated to accept a string that represents a custom global actor in the target. + +This proposal only supports `MainActor` because any other global actor does not help with progressive disclosure. It has the opposite effect - it forces asynchrony on any main-actor-isolated caller. However, there's nothing in this proposal that prohibits generalizing these settings to supporting arbitrary global actors in the future if a compelling use case arises. + +### Infer `MainActor` by default as an upcoming feature + +Instead of introducing a separate mode for configuring default actor isolation inference, the default isolation could be changed to be `MainActor` under an upcoming feature that is enabled by default in a future Swift language mode. The upcoming feature approach was not taken because `MainActor` isolation is the wrong default for many kinds of modules, including libraries that offer APIs that can be used from any isolation domain, and highly-concurrent server applications. + +Similarly, a future language mode could enable main actor isolation by default, and require an opt out for using `nonisolated` as the default actor isolation. However, as the Swift package ecosystem grows, it's more likely for `nonisolated` to be the more common default amongst projects. If we discover that not to be true in practice, nothing in this proposal prevents changing the default actor isolation in a future language mode. + +See the approachable data-race safety vision document for an [analysis on the risks of introducing a language dialect](https://github.com/hborla/swift-evolution/blob/approachable-concurrency-vision/visions/approachable-concurrency.md#risks-of-a-language-dialect) for default actor isolation. + +### Alternative to `SendableMetatype` for suppressing main-actor inference + +The protocols to which a type conforms can affect the isolation of the type. Conforming to a global-actor-isolated protocol can infer global-actor isolatation for the type. When the default actor isolation is `MainActor`, it is valuable for protocols to be able to push inference toward keeping conforming types `nonisolated`, for example because conforming types are meant to be usable from any isolation domain. + +In this proposal, inheritance from `SendableMetatype` (introduced in [SE-0470](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0470-isolated-conformances.md)) is used as an indication that types conforming to the protocol should be `nonisolated`. The `SendableMetatype` marker protocol indicates when a type (but not necessarily its instances) can cross isolation domains, which implies that the type generally needs to be usable from any isolation domain. Additionally, protocols that inherit from `SendableMetatype` can only be meaningfully be used with nonisolated conformances, as discussed in SE-0470. Experience using default main actor isolation uncovered a number of existing protocols that reinforce the notion of `SendableMetatype` inheritance is a reasonable heuristic to indicate that a conforming type should be nonisolated: the standard library's [`CodingKey`](https://developer.apple.com/documentation/swift/codingkey) protocol inherits `Sendable` (which in turn inherits `SendableMetatype`) so a typical conformance will fail to compile with default main actor isolation: + +```swift +struct S: Codable { + var a: Int + + // error if CodingKeys is inferred to `@MainActor`. The conformance cannot be main-actor-isolated, and + // the requirements of the (nonisolated) CodingKey cannot be satisfied by main-actor-isolated members of + // CodingKeys. + enum CodingKeys: CodingKey { + case a + } +} +``` + +Other places that have similar issues with default main actor isolation include the [`Transferable`](https://developer.apple.com/documentation/coretransferable/transferable) protocol and the uses of key paths in the [`@Model` macro](https://developer.apple.com/documentation/swiftdata/model()). + +Instead of using `SendableMetatype` inheritance, this proposal could introduce new syntax for a protocol to explicitly indicate + +```swift +@nonisolatedConformingTypes +public protocol CodingKey { + // ... +} +``` + +This would make the behavior pushing conforming types toward `nonisolated` opt-in. However, it means that existing protocols (such as the ones mentioned above) would all need to adopt this spelling before code using default main actor isolation will work well. Given the strong semantic link between `SendableMetatype` and `nonisolated` conformances and types, the proposed rule based on `SendableMetatype` inheritance is likely to make more code work well with default main actor isolation. An explicit opt-in attribute like the above could be added at a later time if needed. + +### Use an enum for the package manifest API + +An alternative to using a `MainActor` metatype for the Swift package manifest API is to use an enum, e.g. + +```swift +public enum DefaultActorIsolation { + case mainActor + case nonisolated +} + +extension SwiftSetting { + @available(_PackageDescription, introduced: 6.2) + public static func defaultIsolation( + _ isolation: DefaultActorIsolation, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting +} + +// in a package manifest + +swiftSettings: [ + .defaultIsolation(.mainActor) +] +``` + +The enum approach introduces a different way of writing main actor isolation that does not involve the `MainActor` global actor type. The proposed design matches exactly the values used for `#isolation`, i.e. `MainActor.self` for main actor isolation and `nil` for `nonisolated`, which programmers are already familiar with. + +The primary argument for using an enum is that it can be extended in the future to support custom global actor types. This proposal deliberately puts supporting custom global actors in the alternatives considered and not future directions, because defaulting a module to a different global actor does not help improve progressive disclosure for concurrency. + +## Revision history + +* Changes in amendment review: + * Disable `@MainActor` inference when type conforms to a `SendableMetatype` protocol + +## Acknowledgments + +Thank you to John McCall for providing much of the motivation for this pitch in the approachable data-race safety vision document, and to Michael Gottesman for helping with the implementation. diff --git a/proposals/0467-MutableSpan.md b/proposals/0467-MutableSpan.md new file mode 100644 index 0000000000..846697c393 --- /dev/null +++ b/proposals/0467-MutableSpan.md @@ -0,0 +1,700 @@ +# MutableSpan and MutableRawSpan: delegate mutations of contiguous memory + +* Proposal: [SE-0467](0467-MutableSpan.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [BufferView Roadmap](https://forums.swift.org/t/66211) +* Implementation: [PR #79650](https://github.com/swiftlang/swift/pull/79650), [PR #80517](https://github.com/swiftlang/swift/pull/80517) +* Review: ([Pitch](https://forums.swift.org/t/pitch-mutablespan/77790)) ([Review](https://forums.swift.org/t/se-0467-mutablespan/78454)) ([Acceptance](https://forums.swift.org/t/accepted-se-0467-mutablespan/78875)) + +[SE-0446]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md +[SE-0447]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md +[SE-0456]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0456-stdlib-span-properties.md +[PR-2305]: https://github.com/swiftlang/swift-evolution/pull/2305 +[SE-0437]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0437-noncopyable-stdlib-primitives.md +[SE-0453]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md +[SE-0223]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0223-array-uninitialized-initializer.md +[SE-0176]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0176-enforce-exclusive-access-to-memory.md + +## Introduction + +We recently [introduced][SE-0447] the `Span` and `RawSpan` types, providing shared read-only access to borrowed memory. This proposal adds helper types to delegate mutations of exclusively-borrowed memory: `MutableSpan` and `MutableRawSpan`. + +## Motivation + +Many standard library container types can provide direct access to modify their internal representation. Up to now, it has only been possible to do so in an unsafe way. The standard library provides this unsafe functionality with closure-taking functions such as `withUnsafeMutableBufferPointer()` and `withContiguousMutableStorageIfAvailable()`. + +These functions have a few different drawbacks, most prominently their reliance on unsafe types, which makes them unpalatable in security-conscious environments. We continue addressing these issues with `MutableSpan` and `MutableRawSpan`, new non-copyable and non-escapable types that manage respectively mutations of typed and untyped memory. + +In addition to the new types, we will propose adding new API some standard library types to take advantage of `MutableSpan` and `MutableRawSpan`. + +## Proposed solution + +We introduced `Span` to provide shared read-only access to containers. The natural next step is to provide a similar capability for mutable access. A library whose API provides access to its internal storage makes a decision regarding the type of access it provides; it may provide read-only access or provide the ability to mutate its storage. That decision is made by the API author. If mutations were enabled by simply binding a `Span` value to a mutable binding (`var` binding or `inout` parameter), that decision would rest with the user of the API instead of its author. This explains why mutations must be modeled by a type separate from `Span`. + +Mutability requires exclusive access, per Swift's [law of exclusivity][SE-0176]. `Span` is copyable, and must be copyable in order to properly model read access under the law of exclusivity: a value can be simultaneously accessed through multiple read-only accesses. Exclusive access cannot be modeled with a copyable type, since a copy would represent an additional access, in violation of the law of exclusivity. This explains why the type which models mutations must be non-copyable. + +#### MutableSpan + +`MutableSpan` allows delegating mutations of a type's contiguous internal representation, by providing access to an exclusively-borrowed view of a range of contiguous, initialized memory. `MutableSpan`'s memory safety relies on guarantees that: +- it has exclusive access to the range of memory it represents, providing data race safety and enforced by `~Copyable`. +- the memory it represents will remain valid for the duration of the access, providing lifetime safety and enforced by `~Escapable`. +- each access is guarded by bounds checking, providing bounds safety. + +A `MutableSpan` provided by a container represents a mutation of that container, as an extended mutation access. Mutations are implemented by mutating functions and subscripts, which let the compiler statically enforce exclusivity. + +#### MutableRawSpan + +`MutableRawSpan` allows delegating mutations to memory representing possibly heterogeneously-typed values, such as memory intended for encoding. It makes the same safety guarantees as `MutableSpan`. A `MutableRawSpan` can be obtained from a `MutableSpan` whose `Element` is `BitwiseCopyable`. + +#### Extensions to standard library types + +The standard library will provide `mutableSpan` computed properties. These return a new lifetime-dependent `MutableSpan` instance, and that `MutableSpan` represents a mutation of the instance that provided it. The `mutableSpan` computed properties are the safe and composable replacements for the existing `withUnsafeMutableBufferPointer` closure-taking functions. For example, + +```swift +func(_ array: inout Array) { + var ms = array.mutableSpan + modify(&ms) // call function that mutates a MutableSpan + // array.append(2) // attempt to modify `array` would be an error here + _ = consume ms // access to `array` via `ms` ends here + array.append(1) +} +``` + +The `mutableSpan` computed property represents a case of lifetime relationships not covered until now. The `mutableSpan` computed properties proposed here will represent mutations of their callee. This relationship will be illustrated with a hypothetical `@_lifetime` attribute, which ties the lifetime of a return value to an input parameter in a specific way. + +Note: The `@_lifetime` attribute is not real; it is a placeholder. The eventual lifetime annotations proposal may or may not propose syntax along these lines. We expect that, as soon as Swift adopts a syntax do describe lifetime dependencies, the Standard Library will be modified to adopt that new syntax. + +```swift +extension Array { + public var mutableSpan: MutableSpan { + @_lifetime(inout self) + mutating get { ... } + } +} +``` + +Here, the lifetime of the returned `MutableSpan` is tied to an `inout` access of `self` (the `Array`.) As long as the returned instance exists, the source `Array` is being mutated, and no other access to the `Array` can occur. + +This lifetime relationship will apply to all the safe `var mutableSpan: MutableSpan` and `var mutableBytes: MutableRawSpan` properties described in this proposal. + +#### Slicing `MutableSpan` or `MutableRawSpan` instances + +An important category of use cases for `MutableSpan` and `MutableRawSpan` consists of bulk copying operations. Often times, such bulk operations do not necessarily start at the beginning of the span, thus having a method to select a sub-span is necessary. This means producing an instance derived from the callee instance. We adopt the nomenclature already introduced in [SE-0437][SE-0437], with a family of `extracting()` methods. + +```swift +extension MutableSpan where Element: ~Copyable { + @_lifetime(inout self) + public mutating func extracting(_ range: Range) -> Self +} +``` + +This function returns an instance of `MutableSpan` that represents a mutation of the same memory as represented by the callee. The callee can therefore no longer be accessed (read or mutated) while the returned value exists: + +```swift +var array = [1, 2, 3, 4, 5] +var span1 = array.mutableSpan +var span2 = span1.extracting(3..<5) +// neither array nor span1 can be accessed here +span2.swapAt(0, 1) +_ = consume span2 // explicitly end scope for `span2` +span1.swapAt(0, 1) +_ = consume span1 // explicitly end scope for `span1` +print(array) // [2, 1, 3, 5, 4] +``` + +As established in [SE-0437][SE-0437], the instance returned by the `extracting()` function does not share indices with the function's callee. + +## Detailed Design + +#### MutableSpan + +`MutableSpan` is a simple representation of a region of initialized memory. It is non-copyable in order to enforce exclusive access for mutations of its memory, as required by the law of exclusivity: + +````swift +@frozen +public struct MutableSpan: ~Copyable, ~Escapable { + internal var _start: UnsafeMutableRawPointer? + internal var _count: Int +} + +extension MutableSpan: @unchecked Sendable where Element: Sendable & ~Copyable {} +```` + +We store a `UnsafeMutableRawPointer` value internally in order to explicitly support reinterpreted views of memory as containing different types of `BitwiseCopyable` elements. Note that the the optionality of the pointer does not affect usage of `MutableSpan`, since accesses are bounds-checked and the pointer is only dereferenced when the `MutableSpan` isn't empty, when the pointer cannot be `nil`. + +Initializers, required for library adoption, will be proposed alongside [lifetime annotations][PR-2305]; for details, see "[Initializers](#initializers)" in the [future directions](#Directions) section. + +```swift +extension MutableSpan where Element: ~Copyable { + /// The number of initialized elements in this `MutableSpan`. + var count: Int { get } + + /// A Boolean value indicating whether the span is empty. + var isEmpty: Bool { get } + + /// The type that represents a position in a `MutableSpan`. + typealias Index = Int + + /// The range of indices valid for this `MutableSpan`. + var indices: Range { get } + + /// Accesses the element at the specified position. + subscript(_ index: Index) -> Element { borrow; mutate } + // accessor syntax from accessors roadmap (https://forums.swift.org/t/76707) + + /// Exchange the elements at the two given offsets + mutating func swapAt(_ i: Index, _ j: Index) + + /// Borrow the underlying memory for read-only access + var span: Span { @_lifetime(borrow self) borrowing get } +} +``` + +Like `Span` before it, `MutableSpan` does not conform to `Collection` or `MutableCollection`. These two protocols assume their conformers and elements are copyable, and as such are not compatible with a non-copyable type such as `MutableSpan`. A later proposal will consider generalized containers. + +The subscript uses a borrowing accessor for read-only element access, and a mutate accessor for element mutation. The read-only borrow is a read access to the entire `MutableSpan` for the duration of the access to the element. The `mutate` accessor is an exclusive access to the entire `MutableSpan` for the duration of the mutation of the element. + +`MutableSpan` uses offset-based indexing. The first element of a given span is always at offset 0, and its last element is always at position `count-1`. + +As a side-effect of not conforming to `Collection` or `Sequence`, `MutableSpan` is not directly supported by `for` loops at this time. It is, however, easy to use in a `for` loop via indexing: + +```swift +for i in myMutableSpan.indices { + mutatingFunction(&myMutableSpan[i]) +} +``` + +##### Bulk updates of a `MutableSpan`'s elements: + +We include functions to perform bulk copies of elements into the memory represented by a `MutableSpan`. Updating a `MutableSpan` from known-sized sources (such as `Collection` or `Span`) copies every element of a source. It is an error to do so when there is the span is too short to contain every element from the source. Updating a `MutableSpan` from `Sequence` or `IteratorProtocol` instances will copy as many items as possible, either until the input is empty or until the operation has updated the item at the last index. The bulk operations return the index following the last element updated. + +```swift +extension MutableSpan where Element: Copyable { + /// Updates every element of this span to the given value. + mutating func update( + repeating repeatedValue: Element + ) + + /// Updates the span's elements with the elements from the source + mutating func update( + from source: S + ) -> (unwritten: S.Iterator, index: Index) where S.Element == Element + + /// Updates the span's elements with the elements from the source + mutating func update( + from source: inout some IteratorProtocol + ) -> Index + + /// Updates the span's elements with every element of the source. + mutating func update( + fromContentsOf source: some Collection + ) -> Index +} + +extension MutableSpan where Element: ~Copyable + /// Updates the span's elements with every element of the source. + mutating func update( + fromContentsOf source: Span + ) -> Index + + /// Updates the span's elements with every element of the source. + mutating func update( + fromContentsOf source: borrowing MutableSpan + ) -> Index + + /// Updates the span's elements with every element of the source, + /// leaving the source uninitialized. + mutating func moveUpdate( + fromContentsOf source: UnsafeMutableBufferPointer + ) -> Index +} + +extension MutableSpan where Element: Copyable { + /// Updates the span's elements with every element of the source, + /// leaving the source uninitialized. + mutating func moveUpdate( + fromContentsOf source: Slice> + ) -> Index +} +``` + +##### Extracting sub-spans +These functions extract sub-spans of the callee. The first two perform strict bounds-checking. The last four return prefixes or suffixes, where the number of elements in the returned sub-span is bounded by the number of elements in the parent `MutableSpan`. + +```swift +extension MutableSpan where Element: ~Copyable { + /// Returns a span over the items within the supplied range of + /// positions within this span. + @_lifetime(inout self) + mutating public func extracting(_ bounds: Range) -> Self + + /// Returns a span over the items within the supplied range of + /// positions within this span. + @_lifetime(inout self) + mutating public func extracting(_ bounds: some RangeExpression) -> Self + + /// Returns a span containing the initial elements of this span, + /// up to the specified maximum length. + @_lifetime(inout self) + mutating public func extracting(first maxLength: Int) -> Self + + /// Returns a span over all but the given number of trailing elements. + @_lifetime(inout self) + mutating public func extracting(droppingLast k: Int) -> Self + + /// Returns a span containing the final elements of the span, + /// up to the given maximum length. + @_lifetime(inout self) + mutating public func extracting(last maxLength: Int) -> Self + + /// Returns a span over all but the given number of initial elements. + @_lifetime(inout self) + mutating public func extracting(droppingFirst k: Int) -> Self +} +``` + +##### Unchecked access to elements or sub-spans: + +The `subscript` and index-taking functions mentioned above always check the bounds of the `MutableSpan` before allowing access to the memory, preventing out-of-bounds accesses. We also provide unchecked variants of the `subscript`, the `swapAt()` and `extracting()` functions as alternatives in situations where repeated bounds-checking is costly and has already been performed: + +```swift +extension MutableSpan where Element: ~Copyable { + /// Accesses the element at the specified `position`. + /// + /// This subscript does not validate `position`; this is an unsafe operation. + /// + /// - Parameter position: The offset of the element to access. `position` + /// must be greater or equal to zero, and less than `count`. + @unsafe + subscript(unchecked position: Index) -> Element { borrow; mutate } + + /// Exchange the elements at the two given offsets + /// + /// This function does not validate `i` or `j`; this is an unsafe operation. + @unsafe + mutating func swapAt(unchecked i: Index, unchecked j: Index) + + /// Constructs a new span over the items within the supplied range of + /// positions within this span. + /// + /// This function does not validate `bounds`; this is an unsafe operation. + @unsafe + @_lifetime(inout self) + mutating func extracting(unchecked bounds: Range) -> Self + + /// Constructs a new span over the items within the supplied range of + /// positions within this span. + /// + /// This function does not validate `bounds`; this is an unsafe operation. + @unsafe + @_lifetime(inout self) + mutating func extracting(unchecked bounds: ClosedRange) -> Self +} +``` + +##### Interoperability with unsafe code + +```swift +extension MutableSpan where Element: ~Copyable { + /// Calls a closure with a pointer to the viewed contiguous storage. + func withUnsafeBufferPointer( + _ body: (_ buffer: UnsafeBufferPointer) throws(E) -> Result + ) throws(E) -> Result + + /// Calls a closure with a pointer to the viewed mutable contiguous + /// storage. + mutating func withUnsafeMutableBufferPointer( + _ body: (_ buffer: UnsafeMutableBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} + +extension MutableSpan where Element: BitwiseCopyable { + /// Calls a closure with a pointer to the underlying bytes of + /// the viewed contiguous storage. + func withUnsafeBytes( + _ body: (_ buffer: UnsafeRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result + + /// Calls a closure with a pointer to the underlying bytes of + /// the viewed mutable contiguous storage. + /// + /// Note: mutating the bytes may result in the violation of + /// invariants in the internal representation of `Element` + @unsafe + mutating func withUnsafeMutableBytes( + _ body: (_ buffer: UnsafeMutableRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} +``` +These functions use a closure to define the scope of validity of `buffer`, ensuring that the underlying `MutableSpan` and the binding it depends on both remain valid through the end of the closure. They have the same shape as the equivalents on `Array` because they fulfill the same function, namely to keep the underlying binding alive. + +#### MutableRawSpan + +`MutableRawSpan` is similar to `MutableSpan`, but represents untyped initialized bytes. `MutableRawSpan` specifically supports encoding and decoding applications. Its API supports `unsafeLoad(as:)` and `storeBytes(of: as:)`, as well as a variety of bulk copying operations. + +##### `MutableRawSpan` API: + +```swift +@frozen +public struct MutableRawSpan: ~Copyable, ~Escapable { + internal var _start: UnsafeMutableRawPointer? + internal var _count: Int +} + +extension MutableRawSpan: @unchecked Sendable +``` + +Initializers, required for library adoption, will be proposed alongside [lifetime annotations][PR-2305]; for details, see "[Initializers](#initializers)" in the [future directions](#Directions) section. + +```swift +extension MutableRawSpan { + /// The number of bytes in the span. + var byteCount: Int { get } + + /// A Boolean value indicating whether the span is empty. + var isEmpty: Bool { get } + + /// The range of valid byte offsets into this `RawSpan` + var byteOffsets: Range { get } +} +``` + +##### Accessing and modifying the memory of a `MutableRawSpan`: + +`MutableRawSpan` supports storing the bytes of a `BitwiseCopyable` value to its underlying memory: + +```swift +extension MutableRawSpan { + /// Stores the given value's bytes into raw memory at the specified offset. + mutating func storeBytes( + of value: T, toByteOffset offset: Int = 0, as type: T.Type + ) + + /// Stores the given value's bytes into raw memory at the specified offset. + /// + /// This function does not validate `offset`; this is an unsafe operation. + @unsafe + mutating func storeBytes( + of value: T, toUncheckedByteOffset offset: Int, as type: T.Type + ) +} +``` + +Additionally, the basic loading operations available on `RawSpan` are available for `MutableRawSpan`. These operations are not type-safe, in that the loaded value returned by the operation can be invalid, and violate type invariants. Some types have a property that makes the `unsafeLoad(as:)` function safe, but we don't have a way to [formally identify](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md#SurjectiveBitPattern) such types at this time. + +```swift +extension MutableRawSpan { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + @unsafe + func unsafeLoad( + fromByteOffset offset: Int = 0, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + @unsafe + func unsafeLoadUnaligned( + fromByteOffset offset: Int = 0, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + @unsafe + func unsafeLoad( + fromUncheckedByteOffset offset: Int, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + @unsafe + func unsafeLoadUnaligned( + fromUncheckedByteOffset offset: Int, as: T.Type + ) -> T +} +``` + +We include functions to perform bulk copies into the memory represented by a `MutableRawSpan`. Updating a `MutableRawSpan` from a `Collection` or a `Span` copies every element of a source. It is an error to do so when there is are not enough bytes in the span to contain every element from the source. Updating `MutableRawSpan` from `Sequence` or `IteratorProtocol` instance copies as many items as possible, either until the input is empty or until there are not enough bytes in the span to store another element. + +```swift +extension MutableRawSpan { + /// Updates the span's bytes with the bytes of the elements from the source + mutating func update( + from source: S + ) -> (unwritten: S.Iterator, byteOffset: Int) where S.Element: BitwiseCopyable + + /// Updates the span's bytes with the bytes of the elements from the source + mutating func update( + from source: inout some IteratorProtocol + ) -> Int + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: C + ) -> Int where C.Element: BitwiseCopyable + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: Span + ) -> Int + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: borrowing MutableSpan + ) -> Int + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: RawSpan + ) -> Int + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: borrowing MutableRawSpan + ) -> Int +} +``` + +##### Extracting sub-spans + +These functions extract sub-spans of the callee. The first two perform strict bounds-checking. The last four return prefixes or suffixes, where the number of elements in the returned sub-span is bounded by the number of elements in the parent `MutableRawSpan`. + +```swift +extension MutableRawSpan { + /// Returns a span over the items within the supplied range of + /// positions within this span. + @_lifetime(inout self) + mutating public func extracting(_ byteOffsets: Range) -> Self + + /// Returns a span over the items within the supplied range of + /// positions within this span. + @_lifetime(inout self) + mutating public func extracting(_ byteOffsets: some RangeExpression) -> Self + + /// Returns a span containing the initial elements of this span, + /// up to the specified maximum length. + @_lifetime(inout self) + mutating public func extracting(first maxLength: Int) -> Self + + /// Returns a span over all but the given number of trailing elements. + @_lifetime(inout self) + mutating public func extracting(droppingLast k: Int) -> Self + + /// Returns a span containing the final elements of the span, + /// up to the given maximum length. + @_lifetime(inout self) + mutating public func extracting(last maxLegnth: Int) -> Self + + /// Returns a span over all but the given number of initial elements. + @_lifetime(inout self) + mutating public func extracting(droppingFirst k: Int) -> Self +} +``` + +We also provide unchecked variants of the `extracting()` functions as alternatives in situations where repeated bounds-checking is costly and has already been performed: + +```swift +extension MutableRawSpan { + /// Constructs a new span over the items within the supplied range of + /// positions within this span. + /// + /// This function does not validate `byteOffsets`; this is an unsafe operation. + @unsafe + @_lifetime(inout self) + mutating func extracting(unchecked byteOffsets: Range) -> Self + + /// Constructs a new span over the items within the supplied range of + /// positions within this span. + /// + /// This function does not validate `byteOffsets`; this is an unsafe operation. + @unsafe + @_lifetime(inout self) + mutating func extracting(unchecked byteOffsets: ClosedRange) -> Self +} +``` + +##### Interoperability with unsafe code: + +```swift +extension MutableRawSpan { + /// Calls a closure with a pointer to the underlying bytes of + /// the viewed contiguous storage. + func withUnsafeBytes( + _ body: (_ buffer: UnsafeRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result + + /// Calls a closure with a pointer to the underlying bytes of + /// the viewed mutable contiguous storage. + mutating func withUnsafeMutableBytes( + _ body: (_ buffer: UnsafeMutableRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} +``` +These functions use a closure to define the scope of validity of `buffer`, ensuring that the underlying `MutableSpan` and the binding it depends on both remain valid through the end of the closure. They have the same shape as the equivalents on `Array` because they fulfill the same purpose, namely to keep the underlying binding alive. + +#### Properties providing `MutableSpan` or `MutableRawSpan` instances + +##### Accessing and mutating the raw bytes of a `MutableSpan` + +When a `MutableSpan`'s element is `BitwiseCopyable`, we allow mutations of the underlying storage as raw bytes, as a `MutableRawSpan`. + +```swift +extension MutableSpan where Element: BitwiseCopyable { + /// Access the underlying raw bytes of this `MutableSpan`'s elements + /// + /// Note: mutating the bytes may result in the violation of + /// invariants in the internal representation of `Element` + @unsafe + var mutableBytes: MutableRawSpan { @_lifetime(inout self) mutating get } +} +``` + +The standard library will provide `mutating` computed properties providing lifetime-dependent `MutableSpan` instances. These `mutableSpan` computed properties are intended as the safe and composable replacements for the existing `withUnsafeMutableBufferPointer` closure-taking functions. + +##### Extensions to Standard Library types + +```swift +extension Array { + /// Access this Array's elements as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} + +extension ContiguousArray { + /// Access this Array's elements as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} + +extension ArraySlice { + /// Access this Array's elements as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} + +extension InlineArray { + /// Access this Array's elements as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} + +extension CollectionOfOne { + /// Access this Collection's element as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} +``` + +##### Extensions to unsafe buffer types + +We hope that `MutableSpan` and `MutableRawSpan` will become the standard ways to delegate mutations of shared contiguous memory in Swift. Many current API delegate mutations via closure-based functions that receive an `UnsafeMutableBufferPointer` parameter. We will provide ways to unsafely obtain `MutableSpan` instances from `UnsafeMutableBufferPointer` and `MutableRawSpan` instances from `UnsafeMutableRawBufferPointer`, in order to bridge these unsafe types to newer, safer contexts. + +```swift +extension UnsafeMutableBufferPointer { + /// Unsafely access this buffer as a MutableSpan + @unsafe + var mutableSpan: MutableSpan { @_lifetime(borrow self) get } +} + +extension UnsafeMutableRawBufferPointer { + /// Unsafely access this buffer as a MutableRawSpan + @unsafe + var mutableBytes: MutableRawSpan { @_lifetime(borrow self) get } +} +``` + +These unsafe conversions returns a value whose lifetime is dependent on the _binding_ of the `UnsafeMutable[Raw]BufferPointer`. This dependency does not keep the underlying memory alive. As is usual where the `UnsafePointer` family of types is involved, the programmer must ensure the memory remains allocated while it is in use. Additionally, the following invariants must remain true for as long as the `MutableSpan` or `MutableRawSpan` value exists: + + - The underlying memory remains initialized. + - The underlying memory is not accessed through another means. + +Failure to maintain these invariants results in undefined behaviour. + +##### Extensions to `Foundation.Data` + +While the `swift-foundation` package and the `Foundation` framework are not governed by the Swift evolution process, `Data` is similar in use to standard library types, and the project acknowledges that it is desirable for it to have similar API when appropriate. Accordingly, we plan to propose the following additions to `Foundation.Data`: + +```swift +extension Foundation.Data { + // Access this instance's bytes as mutable contiguous storage + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } + + // Access this instance's bytes as mutable contiguous bytes + var mutableBytes: MutableRawSpan { @_lifetime(inout self) mutating get } +} +``` + +#### Performance + +The `mutableSpan` and `mutableBytes` properties should be performant and return their `MutableSpan` or `MutableRawSpan` with very little work, in O(1) time. In copy-on-write types, however, obtaining a `MutableSpan` is the start of the mutation. When the backing buffer is not uniquely referenced then a full copy must be made ahead of returning the `MutableSpan`. + +Note that `MutableSpan` incurs no special behaviour for bridged types, since mutable bindings always require a defensive copy of data bridged from Objective-C data structures. + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a new version of the Swift standard library. + +## Alternatives considered + +#### Adding `withMutableSpan()` closure-taking functions + +The `mutableSpan` and `mutableBytes` properties aim to be safe replacements for the `withUnsafeMutableBufferPointer()` and `withUnsafeMutableBytes()` closure-taking functions. We could consider `withMutableSpan()` and `withMutableBytes()` closure-taking functions that would provide a quicker migration away from the older unsafe functions. We do not believe the closure-taking functions are desirable in the long run. In the short run, there may be a desire to clearly mark the scope where a `MutableSpan` instance is used. The default method would be to explicitly consume a `MutableSpan` instance: + +```swift +var a = ContiguousArray(0..<8) +var span = a.mutableSpan +modify(&span) +_ = consume span +a.append(8) +``` + +During the evolution of Swift, we have learned that closure-based API are difficult to compose, especially with one another. They can also require alterations to support new language features. For example, the generalization of closure-taking API for non-copyable values as well as typed throws is ongoing; adding more closure-taking API may make future feature evolution more labor-intensive. By instead relying on returned values, whether from computed properties or functions, we build for **greater** composability. Use cases where this approach falls short should be reported as enhancement requests or bugs. + +#### Omitting extensions to `UnsafeBufferPointer` and related types + +We could omit the extensions to `UnsafeMutableBufferPointer` and related types, and rely instead of future `MutableSpan` and `MutableRawSpan` initializers. The initializers can have the advantage of being able to communicate semantics (somewhat) through their parameter labels. However, they also have a very different shape than the `storage` computed properties we are proposing for the safe types such as `Array`. We believe that the adding the same API on both safe and unsafe types is advantageous, even if the preconditions for the properties cannot be statically enforced. + +## Future directions + +Note: The future directions stated in [SE-0447](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md#Directions) apply here as well. + +#### Initializing and returning `MutableSpan` instances + +`MutableSpan` represents a region of memory and, as such, must be initialized using an unsafe pointer. This is an unsafe operation which will typically be performed internally to a container's implementation. In order to bridge to safe code, these initializers require new annotations that indicate to the compiler how the newly-created `Span` can be used safely. + +These annotations have been [pitched][PR-2305-pitch] and, after revision, are expected to be pitched again soon. `MutableSpan` initializers using lifetime annotations will be proposed alongside the annotations themselves. + +#### Splitting `MutableSpan` instances – `MutableSpan` in divide-and-conquer algorithms + +It is desirable to have a way to split a `MutableSpan` in multiple parts, for divide-and-conquer algorithms or other reasons: + +```swift +extension MutableSpan where Element: ~Copyable { + public mutating func split(at index: Index) -> (part1: Self, part2: Self) +} +``` + +Unfortunately, tuples do not support non-copyable or non-escapable values yet. We may be able to use `InlineArray` ([SE-0453][SE-0453]), or a bespoke type, but destructuring the non-copyable constituent part remains a challenge. Solving this issue for `Span` and `MutableSpan` is a top priority. + +#### Mutating algorithms + +Algorithms defined on `MutableCollection` such as `sort(by:)` and `partition(by:)` could be defined on `MutableSpan`. We believe we will be able to define these more generally once we have a generalized container protocol hierarchy. + +#### Exclusive Access + +The `mutating` functions in this proposal generally do not represent mutations of the binding itself, but of memory being referenced. `mutating` is necessary in order to model the necessary exclusive access to the memory. We could conceive of an access level between "shared" (`let`) and "exclusive" (`var`) that would model an exclusive access while allowing the pointer and count information to be stored in registers. + +#### Harmonizing `extracting()` functions across types + +The range of `extracting()` functions proposed here expands upon the range accepted in [SE-0437][SE-0437]. If the prefix and suffix variants are accepted, we should add them to `UnsafeBufferPointer` types as well. `Span` and `RawSpan` should also have `extracting()` functions with appropriate lifetime dependencies. + +#### Delegated initialization with `OutputSpan` + +Some data structures can delegate initialization of parts of their owned memory. The standard library added the `Array` initializer `init(unsafeUninitializedCapacity:initializingWith:)` in [SE-0223][SE-0223]. This initializer relies on `UnsafeMutableBufferPointer` and correct usage of initialization primitives. We should present a simpler and safer model of initialization by leveraging non-copyability and non-escapability. + +We expect to propose an `OutputSpan` type to represent partially-initialized memory, and to support to the initialization of memory by appending to the initialized portion of the underlying storage. diff --git a/proposals/0468-async-stream-continuation-hashable-conformance.md b/proposals/0468-async-stream-continuation-hashable-conformance.md new file mode 100644 index 0000000000..5412d1734e --- /dev/null +++ b/proposals/0468-async-stream-continuation-hashable-conformance.md @@ -0,0 +1,130 @@ +# `Hashable` conformance for `Async(Throwing)Stream.Continuation` + +* Proposal: [SE-0468](0468-async-stream-continuation-hashable-conformance.md) +* Authors: [Mykola Pokhylets](https://github.com/nickolas-pohilets) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#79457](https://github.com/swiftlang/swift/pull/79457) +* Review: ([pitch](https://forums.swift.org/t/pitch-add-hashable-conformance-to-asyncstream-continuation/77897)) ([review](https://forums.swift.org/t/se-0468-hashable-conformance-for-async-throwing-stream-continuation/78487)) ([acceptance](https://forums.swift.org/t/accepted-se-0468-hashable-conformance-for-async-throwing-stream-continuation/79116)) + +## Introduction + +This proposal adds a `Hashable` conformance to `Async(Throwing)Stream.Continuation` +to simplify working with multiple streams. + +## Motivation + +Use cases operating with multiple `AsyncStream`s may need to store multiple continuations. +When handling `onTermination` callback, client code needs to remove the relevant continuation. + +To identify the relevant continuation, client code needs to be able to compare continuations. + +It is possible to associate a lookup key with each continuation, but this is inefficient. +`AsyncStream.Continuation` already stores a reference to `AsyncStream._Storage`, +whose identity can be used to provide simple and efficient `Hashable` conformance. + +Consider this simple Observer pattern with an `AsyncSequence`-based API. +To avoid implementing `AsyncSequence` from scratch it uses `AsyncStream` as a building block. +To support multiple subscribers, a new stream is returned every time. + +```swift +@MainActor private class Sender { + var value: Int = 0 { + didSet { + for c in continuations { + c.yield(value) + } + } + } + + var values: some AsyncSequence { + AsyncStream(bufferingPolicy: .bufferingNewest(1)) { continuation in + continuation.yield(value) + self.continuations.insert(continuation) + continuation.onTermination = { _ in + DispatchQueue.main.async { + self.continuations.remove(continuation) + } + } + } + } + + private var continuations: Set.Continuation> = [] +} +``` + +Without a `Hashable` conformance, each continuation needs to be associated with an artificial identifier. +E.g. wrapping continuation in a class, identity of the wrapper object can be used: + +```swift +@MainActor private class Sender { + var value: Int = 0 { + didSet { + for c in continuations { + c.value.yield(value) + } + } + } + + var values: some AsyncSequence { + AsyncStream { (continuation: AsyncStream.Continuation) -> Void in + continuation.yield(value) + let box = ContinuationBox(value: continuation) + self.continuations.insert(box) + continuation.onTermination = { _ in + DispatchQueue.main.async { + self.continuations.remove(box) + } + } + } + } + + private var continuations: Set = [] + + private final class ContinuationBox: Hashable, Sendable { + let value: AsyncStream.Continuation + + init(value: AsyncStream.Continuation) { + self.value = value + } + + static func == (lhs: Sender.ContinuationBox, rhs: Sender.ContinuationBox) -> Bool { + lhs === rhs + } + + func hash(into hasher: inout Hasher) { + hasher.combine(ObjectIdentifier(self)) + } + } +} +``` + +Note that capturing `continuation` or `box` in `onTermination` is safe, because `onTermination` is dropped after being called +(and it is _always_ called, even if `AsyncStream` is discarded without being iterated). + +## Proposed solution + +Add a `Hashable` conformance to `Async(Throwing)Stream.Continuation`. + +## Detailed design + +Every time when the `build` closure of the `Async(Throwing)Stream.init()` is called, +it receives a continuation distinct from all other continuations. +All copies of the same continuation should compare equal. +Yielding values or errors, finishing the stream, or cancelling iteration should not affect equality. +Assigning `onTermination` closures should not affect equality. + +## Source compatibility + +This is an additive change. + +Retroactive conformances are unlikely to exist, because current public API of the `Async(Throwing)Stream.Continuation` +does not provide anything that could be reasonably used to implement `Hashable` or `Equatable` conformances. + +## ABI compatibility + +This is an additive change. + +## Implications on adoption + +Adopters will need a new version of the standard library. diff --git a/proposals/0469-task-names.md b/proposals/0469-task-names.md new file mode 100644 index 0000000000..0d1faf7de7 --- /dev/null +++ b/proposals/0469-task-names.md @@ -0,0 +1,272 @@ +# Task Naming + +* Proposal: [SE-0469](0469-task-names.md) +* Authors: [Konrad Malawski](https://github.com/ktoso), [Harjas Monga](https://github.com/Harjas12) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#79600](https://github.com/swiftlang/swift/pull/79600) +* Review: ([pitch](https://forums.swift.org/t/pitch-task-naming-api/76115)) ([review](https://forums.swift.org/t/se-0469-task-naming/78509)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0469-task-naming/79438)) + +## Introduction + +In this proposal, we introduce several new APIs to allow developers to name their Swift Tasks for the purposes of identifying tasks in a human-readable way. These names can then be used to identify tasks by printing their names, programatically inspecting the name property, or by tools which dump and inspect tasks–such as debuggers, swift-inspect or others. + +## Motivation + +In previous generations of concurrency technologies, developer tools, such as debuggers, have had access to some kind of label to help describe a process’s concurrent work. Ex: Pthread names or Grand Central Dispatch queue names. These names are very helpful to provide extra context to developers when using debugging and profiling tools. + +Currently, Swift Concurrency has no affordances to allow developers to label a Task, which can be troublesome for developers trying to identify "which task" is taking a long time to process or similar questions when observing the system externally. In order to ease the debugging and profiling of Swift concurrency code, developers should be able to annotate their Swift Tasks to describe an asynchronous workload. + +## Proposed solution + +In order to allow developers to provide helpful names for Swift Tasks, the Swift Task creation APIs should be modified to *optionally* allow developers to provide a name for that task. + +Consider the example: + +```swift +let getUsers = Task { + await users.get(accountID)) +} +``` + +In order to ease debugging, a developer could create this unstructured task by passing in a name instead: + +```swift +let getUsers = Task(name: "Get Users") { + await users.get(accountID) +} +``` + +Or, if a developer has a lot of similar tasks, they can provide more contextual information using string interpolation. + +```swift +let getUsers = Task("Get Users for \(accountID)") { + await users.get(accountID) +} +``` + +By introducing this API in Swift itself, rather than developers each inventing their own task-local with a name, runtime inspection tools and debuggers can become aware of task names and show you exactly which accountID was causing the crash or a profiling tool could tell you which accountID request was slow to load. + +## Detailed design + +Naming tasks is only allowed during their creation, and modifying names is not allowed. + +Names are arbitrary user-defined strings, which may be computed at runtime because they often contain identifying information such as the request ID or similar runtime information. + +The following APIs will be provided on `Task`: + +```swift +extension Task where Failure == /* both Never and Error cases */ { + init( + name: String?, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + priority: TaskPriority? = nil, + operation: sending @escaping @isolated(any) () async /*throws */-> Success) + + static func detached( + name: String?, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + priority: TaskPriority? = nil, + operation: sending @escaping @isolated(any) () async /*throws */ -> Success) +} +``` + +In addition to these APIs to name unstructured Tasks, the following API will be added to all kinds of task groups: + +```swift +mutating func addTask( + name: String?, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + priority: TaskPriority? = nil, + operation: sending @escaping @isolated(any) () async -> ChildTaskResult + ) + + mutating func addTaskUnlessCancelled( + name: String?, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + priority: TaskPriority? = nil, + operation: sending @escaping @isolated(any) () async -> ChildTaskResult + ) +``` + +These APIs would be added to all kinds of task groups, including throwing, discarding ones. With the signature being appropriately matching the existing addTask signatures of those groups. + +> Concurrently under review with this proposal is the `Task.startSynchronously` (working name, pending changes) proposal; +> If both this and the synchronous starting tasks proposals are accepted, these APIs would also gain the additional `name: String? = nil` parameter. + +In addition to that, it will be possible to read a name off a task, similar to how the current task's priority is possible to be read: + +```swift +extension Task { + static var name: String? { get } +} + +extension UnsafeCurrentTask { + var name: String? { get } +} +``` + +### `UnsafeCurrentTask` access from `UnownedJob` + +In order to have an `Executor` be able to inspect a task name, either to print "Now running [Task A]" or for other reasons, we propose to offer the access to an `UnsafeCurrentTask` representation of a `ExecutorJob` (or `UnownedJob`): + +```swift +extension ExecutorJob / UnownedJob { + public var unsafeCurrentTask: UnsafeCurrentTask? { ... } +} +``` + +This allows executors to inspect the task name if the `job` is a task, and has a name: + +```swift +public nonisolated func enqueue(_ job: consuming ExecutorJob) { + log.trace("Running task named: \(job?.unsafeCurrentTask?.name ?? "")") +} +``` + +We use the `UnsafeCurrentTask` type because it is possible to obtain it from an `UnownedTask` and therefore it is not safe to refer to it without knowladge about the job's lifetime. +One should not refer to the unsafe current task after invoking `runSynchronously` on the job, as the job may have completed and been destroyed; therefore the use of the existing `UnsafeCurrentTask` type here is quite appropriate. + +This also allows us to expose other information off a task, such as task local values in the future, if the `UnsafeCurrentTask` were to gain such APIs, without having to replicate "the same" accessors into yet another API that would be accessible directly from an `ExecutorJob`. + +## Source compatibility + +This proposal only contains additive changes to the API surface. + +Since Swift Tasks names will be optional, there will be no source compatibility issues. + +## ABI compatibility + +This proposal is ABI additive and does not change any existing ABI. + +## Implications on adoption + +Because runtime changes are required, these new APIs will only be available on newer OSes. + +## Future directions + +This proposal does not contain a method to name Swift Tasks created using the `async let` syntax. Unlike the other methods of creating Tasks, the `async let` syntax didn’t have an obvious way to allow a developer to provide a string. A suggestion of how we may provide automatic names to Tasks created via this method will be shown below in the [Alternatives Considered section](##Alternatives-considered). + +### Task names for "startSynchronously" + +If the ["start synchronously" tasks proposal](https://github.com/swiftlang/swift-evolution/pull/2698) would be accepted, the name parameter would also be included in those APIs. + +## Alternatives considered + +### Actor & DistributedActor Identity + +#### Actor Identity + +> Note: While not really an alternative, we would like to explain why this proposal does not propose to change anything about how actors are identified. + +This proposal focuses on task names, however, another important part of Swift Concurrency is actors, so in this section we’d like to discuss how there isn’t an actual need for new API to address *actor naming* because of how actors can already conform to protocols. + +An actor can conform e.g. to the `Identifiable` protocol. This works well with constant identifiers, as an actor can have a constant let property implement the `id` requirement from this protocol: + +```swift +actor Worker: Identifiable { + let id: String + + init(id: String) { + self.id = id + } +} +``` + +It is also likely that such identity is how a developer might want to look up and identify such actor in traces or logs, so making use of `Identifiable` seems like a good pattern to follow. + +It is also worth reminding that thread-safety of an actor is ensured even if the `id` were to be implemented using a computed property, because it will be forced to be `nonisolated` because of Swift’s conformance and actor isolation rules: + +```swift +actor Worker: Identifiable { + let workCategory: String = "fetching" // "building" etc... + let workID: Int + + nonisolated var id: String { + "\(workCategory)-\(workID)" + } +} +``` + +#### Distributed Actor Identity + +Distributed actors already implicitly conform to `Identifiable` protocol and have a very useful `id` representation that is always assigned by the actor system by which an actor is managed. + +This id is the natural human readable representation of such actor identity, and tools which want to print an “actor identity” should rely on this. In other words, this simply follows the same general pattern that makes sense for other objects and actors of using Identifiable when available to identify things. + +```swift +distributed actor Worker { // implicitly Identifiable + // nonisolated var id: Self.ActorSystem.ActorID { get } +} +``` + +### AsyncLet Task Naming + +While there is no clear way on how to name Swift Task using `async let`, the following were considered. + +#### Approach 1: + +Since we effectively want to express that “the task” is some specific task, we had considered introducing some special casing where if the right hand side of an async let we want to say at creation time that this task is something specific, thus we arrive at the following: + +```swift +async let example: String = Task(name: "get-example") { "example" } +``` + +In order to make this syntax work, we need to avoid double creating tasks. When the compiler sees the `async let` syntax and the `Task {}` initializer, it would need to not create a Task to immediately create another Task inside it, but instead use that Task initializer that we explicitly wrote. + +While, this approach could in theory allow us to name Tasks created using `async let`. It has at least one major issue: + +It can cause surprising behavior and it can be unclear that this would only work when the Task initializer is visible from the async let declaration... I.e. moving the initialization into a method like this: + +```swift +async let example: String = getTask() // error String != Task + +func getTask() -> Task { Task(name: "get-example") { "example" } } +``` + +This would not only break refactoring, as the types are not the same; but also execution semantics, as this refactoring has now caused the task to become an unstructured task “by accident”. Therefore this approach is not viable because it introduces too many easy to make mistakes. + +#### Approach 2: + +Instead of attempting to adding a naming API to the `async let` syntax, we could instead take a different where if developers really want to name a structured Task they can use a `TaskGroup` and the compiler would generate a good default name for the Tasks created using the `async let` syntax. Drawing inspiration from how closures and dispatch blocks are named, we count the declaration in the scope and use that to name it. For example: + +```swift +func getUserImages() async -> [Image] { + + async let profileImg = getProfilePicture() // <- Named "getUserImages.asyncLet-1" + async let headerImg = getHeaderPicture() // <- Named "getUserImages.asyncLet-2" + + . + . + . +} +``` + +These names at the very least give some indication of what the task was created to do, and the developer can opt to use the `TaskGroup` API if more control is desired. + +A slight alternative to this suggestion, is instead of using the name of the surrounding scope, use the name of the parent task instead. For example: + +```swift +Task(name: "get user images for \(userID)") { + async let profileImg = getProfilePicture() // <- Named "getUserImages.asyncLet-1" + async let headerImg = getHeaderPicture() // <- Named "getUserImages.asyncLet-2" + + . + . + . +} +``` + +This approach doesn’t allow developers full control over naming tasks, but it is in same spirit of allowing developer tools to provide more context for a task. + +## Structured Names + +There was some thought given to the idea of allowing developers to group similar tasks (in name only). Consider programs that create hundreds of tasks for network requests; by allowing grouping a runtime analysis tool could surface that in a textual or graphical UI. The API needed would be similar to the one proposed, but with an additional optional `category` argument for the Task initializer. For example: + +```swift +Task(category: "Networking", name: "download profile image for \(userID)) { ... } +``` + +Then a debugger than wanted to print all the tasks running when a break point is hit, it could group them by this optional “Networking” category. + +This is not in the actual proposal in order to keep the API simple and doesn’t add much additional value over a simple name. diff --git a/proposals/0470-isolated-conformances.md b/proposals/0470-isolated-conformances.md new file mode 100644 index 0000000000..879bd8d05e --- /dev/null +++ b/proposals/0470-isolated-conformances.md @@ -0,0 +1,694 @@ +# Global-actor isolated conformances + +* Proposal: [SE-0470](0470-isolated-conformances.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.2)** +* Vision: [Improving the approachability of data-race safety](https://github.com/swiftlang/swift-evolution/blob/main/visions/approachable-concurrency.md) +* Implementation: On `main` with the experimental features `IsolatedConformances` and `StrictSendableMetatypes`. +* Upcoming Feature Flag: `InferIsolatedConformances` +* Review: ([pitch](https://forums.swift.org/t/pre-pitch-isolated-conformances/77726)) ([review](https://forums.swift.org/t/se-0470-global-actor-isolated-conformances/78704)) ([acceptance](https://forums.swift.org/t/accepted-se-0470-global-actor-isolated-conformances/79189)) ([amendment pitch](https://forums.swift.org/t/pitch-amend-se-0466-se-0470-to-improve-isolation-inference/79854)) ([amendment review](https://forums.swift.org/t/amendment-se-0470-global-actor-isolated-conformances/80999)) ([amendment acceptance](https://forums.swift.org/t/amendment-accepted-se-0470-global-actor-isolated-conformances/81144)) + +## Introduction + +Types isolated to a global actor (such as `@MainActor`) are useful for representing data that can only ever be used from a single concurrency context. They occur both in single-threaded programs where all code is expected to run on the main actor as well as larger applications where interaction with the UI occurs through the main actor. Unfortunately, such types are unable to conform to most protocols due to isolation mismatches: + +```swift +@MainActor +class MyModelType: Equatable { + var name: String + + init(name: String) { + self.name = name + } + + // error: main-actor-isolated static function '==' cannot satisfy non-isolated requirement 'Equatable.==' + static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { + lhs.name == rhs.name + } +} +``` + +This proposal introduces the notion of an *isolated conformance*, which is a conformance that can only be used within the isolation domain of the type. For the code above, the conformance to `Equatable` can be specified as being isolated to the main actor as follows: + +```swift +@MainActor +class MyModelType: @MainActor Equatable { + // unchanged from the above ... +} +``` + +This allows `MyModelType` to provide a conformance to `Equatable` that works like every other conformance, except that it can only be used from the main actor. + +## Motivation + +Types isolated to the global actor are common in single-threaded programs and UI applications, among others, but their inability to conform to protocols without workarounds means that they cannot integrate with any Swift code using generics, cutting them off from interacting with many libraries. The workarounds themselves can be onerous: each operation that is used to satisfy a protocol requirement must be marked as `nonisolated`, e.g., + +```swift + nonisolated static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { + lhs.name == rhs.name + } +``` + +However, this is incompatible with using types or data on the main actor, and results in an error: + +```swift + 3 | @MainActor + 4 | class MyModelType: Equatable { + 5 | var name: String + | `- note: property declared here + 6 | + 7 | init(name: String) { + : +10 | +11 | nonisolated static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { +12 | lhs.name == rhs.name + | `- error: main actor-isolated property 'name' can not be referenced from a nonisolated context +13 | } +14 | } +``` + +We can work around this issue by assuming that this function will only ever be called on the main actor using [`MainActor.assumeIsolated`](https://developer.apple.com/documentation/swift/mainactor/assumeisolated(_:file:line:)): + +```swift + nonisolated static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { + MainActor.assumeIsolated { + lhs.name == rhs.name + } + } +``` + +This is effectively saying that `MyModelType` will only ever be considered `Equatable` on the main actor. Violating this assumption will result in a run-time error detected when `==` is called from outside the main actor. There are two problems with this approach. First, it's dynamically enforcing data-race safety for something that seems like it should be statically verifiable (but can't easily be expressed). Second, this same `nonisolated`/`assumeIsolated` pattern has to be replicated for every function that satisfies a protocol requirement, creating a lot of boilerplate. + +## Proposed solution + +This proposal introduces the notion of an *isolated conformance*. Isolated conformances are conformances whose use is restricted to a particular global actor. This is the same effective restriction as the `nonisolated`/`assumeIsolated` pattern above, but enforced statically by the compiler and without any boilerplate. The following defines a main-actor-isolated conformance of `MyModelType` to `Equatable`: + +```swift +@MainActor +class MyModelType: @MainActor Equatable { + var name: String + + init(name: String) { + self.name = name + } + + static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { + lhs.name == rhs.name + } +} +``` + +Any attempt to use this conformance outside of the main actor will result in a compiler error: + +```swift +/*nonisolated*/ func hasMatching(_ value: MyModelType, in modelValues: [MyModelType]) -> Bool { + // error: cannot use main-actor-isolated conformance of 'MyModelType' to 'Equatable' in + // non-isolated function. + return modelValues.contains(value) +} +``` + +Additionally, we need to make sure that generic code cannot take the conformance and send it to another isolation domain. The [`Sequence.contains`](https://developer.apple.com/documentation/swift/sequence/contains(_:)) operation above clearly won't do that, but one could imagine a similar operation that uses concurrency to attempt the search in parallel: + +```swift +extension Sequence { + func parallelContains(_ element: Element) -> Bool where Element: Equatable & Sendable { + // ... + } +} +``` + +This `parallelContains` function can send values of type `Element` to another isolation domain, and from there call the `Equatable.==` function. If the conformance to `Equatable` is isolated, this would violate the data race safety guarantees. Therefore, this proposal specifies that an isolated conformance cannot be used in conjunction with a `Sendable` conformance: + +```swift +@MainActor +func parallelHasMatching(_ value: MyModelType, in modelValues: [MyModelType]) -> Bool { + // error: isolated conformance of 'MyModelType' to 'Equatable' cannot be used to + // satisfy conformance requirement for a `Sendable` type parameter 'Element'. + return modelValues.parallelContains(value) +} +``` + +The corresponding restriction needs to be in place within generic functions, ensuring that they don't leak (potentially) isolated conformances across isolation boundaries. For example, the following code could introduce a data race if the conformance of `T` to `GlobalLookup` were isolated: + +```swift +protocol GlobalLookup { + static func lookupByName(_ name: String) -> Self? +} + +func hasNamed(_: T.Type, name: String) async -> Bool { + return await Task.detached { + return T.lookupByName(name) != nil + }.value +} +``` + +Here, the type `T` itself is not `Sendable`, but because *all* metatypes are `Sendable` it is considered safe to use `T` from another isolation domain within the generic function. The use of `T`'s conformance to `GlobalLookup` within that other isolation domain introduces a data-race problem if the conformance were isolated. To prevent such problems in generic code, this proposal introduces a notion of *non-sendable metatypes*. Specifically, if a type parameter `T` does not conform to either `Sendable` or to a new protocol, `SendableMetatype`, then its metatype, `T.Type`, is not considered `Sendable` and cannot cross isolation boundaries. The above code, which is accepted in Swift 6 today, would be rejected by the proposed changes here with an error message like: + +```swift +error: cannot capture non-sendable type 'T.Type' in 'sending' closure +``` + +A function like `hasNamed` can indicate that its type parameter `T`'s requires non-isolated conformance by introducing a requirement `T: SendableMetatype`, e.g., + +```swift +func hasNamed(_: T.Type, name: String) async -> Bool { + return await Task.detached { + return T.lookupByName(name) != nil + }.value +} +``` + +As with `Sendable`, an isolated conformance cannot be combined with a `SendableMetatype` constraint: + +```swift +extension MyModelType: isolated GlobalLookup { + static func lookupByName(_ name: String) -> Self? { ... } +} + +// error: isolated conformance of 'MyModelType' to 'MyModelType' cannot be used to +// satisfy conformance requirement for a `SendableMetatype` type parameter 'T'. +if hasNamed(MyModelType.self, "root") { ... } +``` + +Note that `Sendable` inherits from `SendableMetatype`, so any type `T` with a `Sendable` requirement also implies a requirement `T: SendableMetatype`. + +Protocol conformances can also be discovered dynamically with the `as?` and `is` operators. For example, one could try to produce an `any Equatable` from a value of unknown type in any isolation domain: + +```swift +func tryEquatable(_ lhs: Any, rhs: Any) -> Bool { + if let eLHS = lhs as? any Equatable { + // use Equatable.== + } else { + return false + } +} +``` + +The `Any` value could contain `MyModelType`, in which case the conformance to `Equatable` will be isolated. In such cases, the `as?` operation will check whether the code is running on the executor associated with the conformance's isolation. If so, the cast can succeed; otherwise, the case will fail (and produce `nil`). + +## Detailed design + +The proposed solution describes the basic shape of isolated conformances and how they interact with the type system. This section goes into more detail on the data-race safety issues that arise from the introduction of isolated conformances into the language. Then it details three rules that, together, ensure freedom from data race safety issues in the presence of isolated conformances: + +1. An isolated conformance can only be used within its isolation domain. +2. When an isolated conformance is used to satisfy a generic constraint `T: P`, the generic signature must not include either of the following constraints: `T: Sendable` or `T: SendableMetatype`. +3. A value using a conformance isolated to a given global actor is within the same region as that global actor. + +### Data-race safety issues + +An isolated conformance must only be used within its actor's isolation domain. Here are a few examples that demonstrate the kinds of problems that need to be addressed by a design for isolated conformances to ensure that this property holds. + +First, using an isolated conformance outside of its isolation domain creates immediate problems. For example: + +```swift +protocol Q { + static func g() { } +} + +extension C: @MainActor Q { + @MainActor static func g() { } +} + +nonisolated func callQG() { + let qType: Q.Type = C.self + qType.g() // problem: called @MainActor function from nonisolated code +} +``` + +Here, a call to `C.g()` would have been rejected because it's calling a `@MainActor` function from non-isolated code and cannot `await`. However, if we're allowed to use the isolated conformance of `C: Q`, we would subvert the checking because `Q.g()` is non-isolated. + +We can address this specific issue by prohibiting the use of an isolated conformance from outside its isolation domain, i.e., the use of `C: Q` to convert `C.Type` to `Q.Type` in a non-`@MainActor` function would be an error. + +However, this is not sufficient to ensure that this conformance `C: P` will only be used from the main actor. Consider a function like this: + +```swift +@MainActor func badReturn(c: C) -> any Sendable & P { // okay so far + c // uses C: P from the main actor context (okay) + // uses C: Sendable (okay) +} + +@MainActor func useBadReturn(c: C) { + let anyP = badReturn(c: c) + Task.detached { + anyP.f() // PROBLEM: C.f is called from off the main actor + } +} +``` + +Here, the conformance `C: P` is used from within a `@MainActor` function, but a value that stores the conformance (in the `any Sendable & P`) is returned that no longer carries the isolation restriction. The caller is free to copy that value to another isolation domain, and will end up calling `@MainActor` code from outside the main actor. + +The issue is not limited to return values. For example, a generic parameter might escape to another isolation domain: + +```swift +@MainActor func sendMe(_ value: T) { + Task.detached { + value.f() + } +} + +extension C: @unchecked Sendable { } + +@MainActor func doSend(c: C) { + sendMe(c) // uses C: P from the main actor context + // uses C: Sendable +} +``` + +Here, `sendMe` ends up calling `C.f()` from outside the main actor. The combination of an isolated conformance and a `Sendable` requirement on the same type underlies this issue. To address the problem, we can prohibit the use of an isolated conformance if the corresponding type parameter (e.g, `T` in the example above) also has a `Sendable` requirement. + +However, that doesn't address all issues, because region isolation permits sending non-`Sendable` values: + +```swift +@MainActor func badSendingReturn() -> sending any P { // okay so far + C() // uses C: P from the main actor context (okay) + // returned value is in its own region +} + +@MainActor func useBadSendingReturn(c: C) { + let anyP = badSendingReturn() + Task.detached { + anyP.f() // PROBLEM: C.f is called from off the main actor + } +} +``` + +There are similar examples for `sending` parameters, but they're not conceptually different from the return case. This particular issue can be addressed by treating a value that depends on an isolated conformance as being within the region as the actor it's isolated to. So a newly-created value of type `C` is in its own region, but if it's type-erased to an `any P`, its region is merged with the region for the main actor. This would make the return expression in `badSendingReturn` ill-formed, because the returned value is not in its own region. + +Conformances can cross isolation boundaries even if no values cross the boundary: + +```swift +nonisolated func callQGElsewhere(_: T.Type) { + Task.detached { + T.g() + } +} + +@MainActor func isolationWithStatics() { + callQGElsewhere(C.self) +} +``` + +Here, the generic type `T` is used from another isolation domain inside `callQGElsewhere`. When the isolated conformance of `C: Q` is provided to this function, it opens up a data-race safety hole because `C.g()` ends up getting called through generic code. Addressing this problem either means ensuring that there are no operations on the metatype that go through a potentially-isolated protocol conformance or that the metatype is itself does not leave the isolation domain. + +One last issue concerns dynamic casting. Generic code can query a conformance at runtime with a dynamic cast like this: + +```swift +nonisolated func f(_ value: Any) { + if let p = value as? any P { + p.f() + } +} +``` + +If the provided `value` is an instance of `C` , and this code is invoked off the main actor, allowing it to enter the `if` branch would introduce a data race. Therefore, dynamic casting will have to determine when the conformance it depends on is isolated to an actor and check whether the code is running on the executor for that actor. + +Additionally, a dynamic cast that involves a `Sendable` or `SendableMetatype` constraint should not accept an isolated conformance even if the code is running on that global actor, e.g., + +```swift + if let p = value as? any Sendable & P { // never allows an isolated conformance to P + p.f() + } +``` + +### Rule 1: Isolated conformance can only be used within its isolation domain + +Rule (1) is straightforward: the conformance can only be used within a context that is also isolated to the same global actor. This applies to any use of a conformance anywhere in the language. For example: + +```swift +struct S: @MainActor P { } + +struct WrapsP: P { + var value: T + + init(_ value: T) { self.value = value } +} + +func badFunc() -> WrapsP { } // error: non-@MainActor-isolated function uses @MainActor-isolated conformance `S: P` + +func badFunc2() -> any P { + S() // error: non-@MainActor-isolated function uses @MainActor-isolated conformance `S: P` +} + +func acceptsP(_ value: T) { } + +func badFunc3() { + acceptsP(S()) // error: non-@MainActor-isolated function uses @MainActor-isolated conformance `S: P` +} + +protocol P2 { + associatedtype A: P +} + +struct S2: P2 { // error: conformance of S2: P2 depends on @MainActor-isolated conformance `S: P` + // note: fix by making conformance of S2: P2 also @MainActor-isolated + typealias A = S +} + +protocol HasName { + var name: String { get } +} + +@MainActor class Named: @MainActor HasName { + var name: String + // ... +} + +@MainActor +func useName() { + let named = Named() + Task.detached { + named[keyPath: \HasName.name] // error: uses main-actor isolated conformance Named: P + // outside of the main actor + } +} +``` + +Note that the types can have different isolation from their conformances. For example, an `actor` or non-isolated type can have a `@MainActor` conformance: + +```swift +actor MyActor: @MainActor P { + // okay, so long as the declarations satisfying the requirements to P are + // @MainActor or nonisolated +} + +/*nonisolated*/ struct MyStruct: @MainActor P { + // okay, so long as the declarations satisfying the requirements to P are + // @MainActor or nonisolated +} +``` + +### Rule 2: Isolated conformances can only be abstracted away for non-`SendableMetatype` types + +Rule (2) ensures that when information about an isolated conformance is abstracted away by the generics system, the conformance cannot leave its original isolation domain. This requires a way to determine when a given generic function is permitted to pass a conformance it receives across isolation domains. Consider the example above where a generic function uses one of its conformances in different isolation domain: + +```swift +protocol Q { + static func g() { } +} + +nonisolated func callQGElsewhere(_: T.Type) { + Task.detached { + T.g() // use of the conformance T: Q in a different isolation domain + } +} + +extension C: @MainActor Q { ... } + +@MainActor func isolationWithStatics() { + callQGElsewhere(C.self) // passing an isolated conformance +} +``` + +The above code must be rejected to prevent a data race. There are two options for diagnosing this data race: + +1. Reject the definition of `callQGElsewhere` because it is using the conformance from a different isolation domain. +2. Reject the call to `callQGElsewhere` because it does not support isolated conformances. + +This proposal takes option (1): we assume that generic code accepts isolated conformances unless it has indicated otherwise with a `SendableMetatype` constraint. Since most generic code doesn't deal with concurrency at all, it will be unaffected. And generic code that does make use of concurrency should already have `Sendable` constraints (which imply `SendableMetatype` constraints) that indicate that it will not work with isolated conformances. + +The specific requirement for option (1) is enforced both in the caller to a generic function and in the implementation of that function. The caller can use an isolated conformance to satisfy a conformance requirement `T: P` so long as the generic function does not also contain a requirement `T: SendableMetatype`. This prevents isolated conformances to be used in conjunction with types that can cross isolation domains, preventing the data race from being introduced at the call site. Here are some examples of this rule: + +```swift +func acceptsSendableMetatypeP(_ value: T) { } +func acceptsAny(_ value: T) { } +func acceptsSendableMetatype(_ value: T) { } + +@MainActor func passIsolated(s: S) { + acceptsP(s) // okay: the type parameter 'T' requires P but not SendableMetatype + acceptsSendableMetatypeP(s) // error: the type parameter 'T' requires SendableMetatype + acceptsAny(s) // okay: no isolated conformance + acceptsSendableMetatype(s) // okay: no isolated conformance +} +``` + +The same checking occurs when the type parameter is hidden, for example when dealing with `any` or `some` types: + +```swift +@MainActor func isolatedAnyGood(s: S) { + let a: any P = s // okay: the 'any P' cannot leave the isolation domain +} + +@MainActor func isolatedAnyBad(s: S) { + let a: any SendableMetatype & P = s // error: the (hidden) type parameter for the 'any' is SendableMetatype +} + +@MainActor func returnIsolatedSomeGood(s: S) -> some P { + return s // okay: the 'any P' cannot leave the isolation domain +} + +@MainActor func returnIsolatedSomeBad(s: S) -> some SendableMetatype & P { + return s // error: the (hidden) type parameter for the 'any' is Sendable +} +``` + +Within the implementation, we ensure that a conformance that could be isolated cannot cross an isolation boundary. This is done by making the a metatype `T.Type` `Sendable` only when there existing a constraint `T: SendableMetatype`. Therefore, the following program is ill-formed: + +```swift +protocol Q { + static func g() { } +} + +nonisolated func callQGElsewhere(_: T.Type) { + Task.detached { + T.g() // error: non-sendable metatype of `T` captured in 'sending' closure + } +} +``` + +To correct this function, add a constraint `T: SendableMetatype`, which allows the function to send the metatype (along with its conformances) across isolation domains. As described above, it also prevents the caller from providing an isolated conformance to satisfy the `T: Q` requirement, preventing the data race. + +`SendableMetatype` is a new marker protocol that captures the idea that values of the metatype of `T` (i.e., `T.Type`) will cross isolation domains and can take conformances with them. It is less restrictive than a `Sendable` requirement, which specifies that *values* of a type can be sent across isolation boundaries. All concrete types (structs, enums, classes, actors) conform to `SendableMetatype` implicitly, so fixing `callQGElsewhere` will not affect any non-generic code: + +```swift +nonisolated func callQGElsewhere(_: T.Type) { + Task.detached { + T.g() + } +} + +struct MyTypeThatConformsToQ: Q { ... } +callQGElsewhere(MyTypeThatConformsToQ()) // still works +``` + +The `Sendable` protocol inherits from the new `SendableMetatype` protocol: + +```swift +/*@marker*/ protocol SendableMetatype { } +/*@marker*/ protocol Sendable: SendableMetatype { } +``` + +This means that a requirement `T: Sendable` implies `T: SendableMetatype`, so a generic function that uses concurrency along with `Sendable` requirements, like this:: + +```swift +func doSomethingElsewhere(_ value: T) { + Task.detached { + value.f() // okay + } +} +``` + +will continue to work with the stricter model for generic functions in this proposal. + +The proposed change for generic functions does have an impact on source compatibility, where functions like `callQGElsewhere` will be rejected. However, the source break is limited to generic code that: + +1. Passes the metatype `T.Type` of a generic parameter `T` across isolation boundaries; +2. Does not have a corresponding constraint `T: Sendable` requirement; and +3. Is compiled with strict concurrency enabled (either as Swift 6 or with warnings). + +Experiments with the prototype implementation of this feature uncovered very little code that was affected by this change. The benefit to introducing this source break is that the vast majority of existing generic code will work unmodified with isolated conformances, or (if it's using concurrency) correctly reject the use of isolated conformances in their callers. + +### Rule 3: Isolated conformances are in their global actor's region + +With [region-based isolation](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0414-region-based-isolation.md), values of non-`Sendable` type can be transferred to another isolation domain when it can be proven that they are in their own "region" of code that separate from all other regions. Isolated conformances are considered to be within the region of their global actor, so any value formed that involves an isolated conformance will have its region merged with that of the isolated conformance. For example: + +```swift +@MainActor func acceptSending(_ value: sending any P) { } + +@MainActor func passSending() { + let c1 = C() // in its own region + let ap1: any P = c1 // merges region of c1 with region of the conformance of C: P (MainActor) + acceptSending(ap1) // error: argument to sending parameter is within the MainActor region + + let c2 = C() // in its own region + let wp2 = WrapsP(c2) // merges region of c2 with region of the conformance of C: P (MainActor) + acceptSending(c) // error: argument to sending parameter is within the MainActor region +} +``` + +### Inferring global actor isolation for global-actor-isolated types + +Types that are isolated to a global actor are very likely to want to have their conformances to be isolated to that global actor. This is especially true because the members of global-actor isolated types are implicitly isolated to that global actor, so obvious-looking code is rejected: + +```swift +@MainActor +class MyModelType: P { + func f() { } // error: implements P.f, is implicitly @MainActor + // but conformance to P is not isolated +} +``` + +With this proposal, the fix is to mark the conformance as `@MainActor`: + +```swift +@MainActor +class MyModelType: @MainActor P { + func f() { } // okay: implements P.f, is implicitly @MainActor +} +``` + +However, the inference rule feels uneven: why is the `@MainActor` in one place inferred but not in the other? + +In the future, we'd like to extend the global actor inference rule for global-actor isolated types to also infer global actor isolated on their conformances. This makes the obvious code above also correct: + +```swift +@MainActor +class MyModelType: /*inferred @MainActor*/ P { + func f() { } // implements P.f, is implicitly @MainActor +} +``` + +If this inference is not desired, one can use `nonisolated` on the conformances: + +```swift +@MainActor +class MyModelType: nonisolated Q { + nonisolated static func g() { } // implements Q.g, is non-isolated +} +``` + +There are two additional inference rules that imply `nonisolated` on a conformance of a global-actor-isolated type: + +* If the protocol inherits from `SendableMetatype` (including indirectly, e.g., from `Sendable`), then the isolated conformance could never be used, so it is inferred to be `nonisolated`. +* If all of the declarations used to satisfy protocol requirements are `nonisolated`, the conformance will be assumed to be `nonisolated`. The conformance of `MyModelType` to `Q` would be inferred to be `nonisolated` because the static method `g` used to satisfy `Q.g` is `nonisolated.` + +This proposed change is source-breaking in the cases where a conformance is currently `nonisolated`, the rules above would not infer `nonisolated`, and the conformance crosses isolation domains. There, conformance isolation inference is staged in via an upcoming feature (`InferIsolatedConformances`) that can be folded into a future language mode. Fortunately, it is mechanically migratable: existing code migrating to `InferIsolatedConformances` could introduce `nonisolated` for each conformance of a global-actor-isolated type. + +### Infer `@MainActor` conformances + +[SE-0466](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0466-control-default-actor-isolation.md) provides the ability to specify that a given module will infer `@MainActor` on any code that hasn't explicitly stated isolated (or non-isolation, via `nonisolated`). In a module that infers `@MainActor`, the upcoming feature `InferIsolatedConformances` (from the prior section) should also be enabled. This means that types will get main-actor isolation and also have their conformances main-actor isolated, extending the "mostly single-threaded" view of SE-0466 to interactions with generic code: + +```swift +/*implicit @MainActor*/ +class MyClass: /*implicit @MainActor*/P { ... } +``` + +## Source compatibility + +As discussed in the section on rule (2), this proposal introduces a source compatibility break for code that is using strict concurrency and passes uses conformances of non-`Sendable` type parameters in other isolation domains. The overall amount of such code is expected to be small, because it's likely to be rare that the conformances of generic types cross isolation boundaries but values of those types do not. + +Initial testing of an implementation of this proposal found very little code that relied on `Sendable` metatypes where the corresponding type was not also `Sendable`. Therefore, this proposal suggests to accept this as a source-breaking change with strict concurrency (as a warning in Swift 5, error in Swift 6) rather than staging the change through an upcoming feature or alternative language mode. + +## ABI compatibility + +Isolated conformances can be introduced into the Swift ABI without any breaking changes, by extending the existing runtime metadata for protocol conformances. All existing (non-isolated) protocol conformances can work with newer Swift runtimes, and isolated protocol conformances will be usable with older Swift runtimes as well. There is no technical requirement to restrict isolated conformances to newer Swift runtimes. + +However, there is one likely behavioral difference with isolated conformances between newer and older runtimes. In newer Swift runtimes, the functions that evaluate `as?` casts will check of an isolated conformance and validate that the code is running on the proper executor before the cast succeeds. Older Swift runtimes that don't know about isolated conformances will allow the cast to succeed even outside of the isolation domain of the conformance, which can lead to different behavior that potentially involves data races. It should be possible to provide (optional) warnings when running on newer Swift runtimes when a cast fails due to isolated conformances but would incorrectly succeed on older platforms. + +## Future Directions + +### Actor-instance isolated conformances + +Actor-instance isolated conformances are considerably more difficult than global-actor isolated conformances, because the conformance needs to be associated with a specific instance of that actor. Even enforcing rule (1) is nonobvious. As with `isolated` parameters, we could spell actor-instance isolation to a protocol `P` with `isolated P`. The semantics would need to be similar to what follows: + +```swift +actor A: isolated P { + func f() { } // implements P.f() +} + +func instanceActors(a1: isolated A, a2: A) { + let anyP1: any P = a1 // okay: uses isolated conformance 'A: P' only on a1, to which this function is isolated + let anyP2: any P = a2 // error: uses isolated conformance 'A: P' on a2, which is not the actor to which this function is isolated + + let a3 = a1 + let anyP3: any P = a3 // okay? requires dataflow analysis to determine that a3 and + // a1 are in the isolation domain of this function + + let wrappedA1: WrapsP // error? isolated conformance 'A: P' used without being + // anchored to the actor instance a1 + var wrappedA2: WrapsP = .init(a1) // okay? isolated conformance 'A: P' is used with a1 + wrappedA2.value = a3 // error: isolated conformance 'A: P' used in the type is + // in a different isolation domain than 'a1' +} +``` + +It's possible that these problems can be addressed by relying more heavily on region-based isolation akin to rule (3). This can be revisited in the future if the need justifies the additional complexity and we find a suitable implementation strategy. + +## Alternatives considered + +### "Non-Sendable" terminology instead of isolated conformances + +Isolated conformances are a lot like non-`Sendable` types, in that they can be freely used within the isolation domain in which they are created, but can't necessarily cross isolation domain boundaries. We could consider using "sendable" terminology instead of "isolation" terminology, e.g., all existing conformances are "Sendable" conformances (you can freely share them across isolation domain boundaries) and these new conformances are "non-Sendable" conformances. Trying to send such a conformance across an isolation domain boundary is, of course, an error. + +However, the "sendable" analogy breaks down or causes awkwardness in a few places: + +* Values of non-`Sendable` type can be sent across isolation domain boundaries due to [region-based isolation](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0414-region-based-isolation.md), but the same cannot be said of isolated conformances, so they are more non-Sendable than most non-Sendable things. + +* Global-actor-isolated types are usually `Sendable`, but their conformances would generally need to be non-`Sendable`. + +* Usually things are non-`Sendable` but have to be explicitly opt-in to being `Sendable`, whereas conformances would be the opposite. + +* Diagnostics for invalid conformance declarations that could be addressed with isolated conformances are necessarily described in terms of isolation, e.g., + ```` + error: main-actor isolated method 'f' cannot satisfy non-isolated requirement `f` of protocol P + ```` + + It wouldn't make sense to recast that diagnostic in terms of "sendable", and would also be odd for the fix to an isolation-related error message to be "add non-Sendable." + +* There is no established spelling for "not Sendable" that would work well on a conformance. + +### Isolated conformance requirements + +This proposal introduces the notion of isolated conformances, which can satisfy a conformance requirement only when the corresponding type isn't `Sendable`. There is no way for a generic function to express that some protocol requirements are intended to allow isolated conformances while others are not. That could be made explicit, for example by allowing requirements of the form `T: isolated P` (which would work with both isolated and non-isolated conformances) and `T: nonisolated P` (which only allows non-isolated conformances). One could combine these in a given generic signature: + +```swift +func mixedConformances(_ x: [T]) { + for item in x { + item.foo() // Can use requirements of P + print(x.id) // Can use requirements of Identifiable + } + + Task.detached { + for item in x { + item.foo() // error: cannot capture isolated conformance of 'T' to 'P' in a closure in a different isolation domain + print(x.id) // okay: conformance to Identifable is nonisolated + } + } +} +``` + +This is a generalization of the proposed rules that makes more explicit when conformances can cross isolation domains within generic code, as well as allowing mixing of isolated and non-isolated conformances as in the example. One can explain this proposal's rule involving `SendableMetatype` requirements and isolated conformances in terms of (non)-isolated requirements. For a given conformance requirement `T: P` : + +* If `T: SendableMetatype`, `T: P` is interpreted as `T: nonisolated P`. +* If not `T: SendableMetatype`, `T: P` is interepreted as `T: isolated P`. + +The main down side of this alternative is the additional complexity it introduces into generic requirements. It should be possible to introduce this approach later if it proves to be necessary, by treating it as a generalization of the existing rules in this proposal. + +### Require `nonisolated` rather than inferring it + +Under the upcoming feature `InferIsolatedConformances`, this proposal infers `nonisolated` for conformances when all of the declarations that satisfy requirements of a protocol are themselves `nonisolated`. For example: + +```swift +nonisolated protocol Q { + static func create() -> Self +} + +@MainActor struct MyType: /*infers nonisolated*/ Q { + nonisolated static func create() -> MyType { ... } +} +``` + +This inference is important for providing source compatibility with and without `InferIsolatedConformances`, and is especially useful useful when combined with default main-actor isolation ([SE-0466](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0466-control-default-actor-isolation.md)), where many more types will become main-actor isolated. Experience with using these features together also identified some macros (such as [`@Observable`](https://developer.apple.com/documentation/observation/observable())) that produced `nonisolated` members for a protocol conformances, but had not yet been updated to mark the conformance as `nonisolated`. Macro-generated code is much harder for users to update when a source-compatibility issue arises, which makes `nonisolated` conformance inference particularly important for source compatibility. + +However, this inference rule has downsides. It means one needs to examine a protocol and how a type conforms to that protocol to determine whether the conformance might be `nonisolated`, which can be a lot of work for the developer reading the code as well as the compiler. It can also change over time: for example, a default implementation of a protocol requirement will likely be `nonisolated`, but a user-written one within a main-actor-isolated type would be `@MainActor` and, therefore, make the conformance `@MainActor`. + +One alternative would be to introduce this inference rule for source compatibility, but treat it as a temporary measure to be disabled again in some future language mode. Introducing the inference rule in this proposal does not foreclose on that possibility: if we find that the `nonisolated` conformance inference rule here is harmful to readability, a separate proposal can deprecate it in a future language mode, providing a suitable migration timeframe. + +## Revision history + +* Changes in amendment review: + * If the protocol inherits from `SendableMetatype` (including indirectly, e.g., from `Sendable`), then the isolated conformance could never be used, so it is inferred to be `nonisolated`. + * If all of the declarations used to satisfy protocol requirements are `nonisolated`, the conformance will be assumed to be `nonisolated`. +* Changes in review: + * Within a generic function, use sendability of metatypes of generic parameters as the basis for checking, rather than treating specific conformances as potentially isolated. This model is easier to reason about and fits better with `SendableMetatype`, and was used in earlier drafts of this proposal. diff --git a/proposals/0471-SerialExecutor-isIsolated.md b/proposals/0471-SerialExecutor-isIsolated.md new file mode 100644 index 0000000000..b91e3693ad --- /dev/null +++ b/proposals/0471-SerialExecutor-isIsolated.md @@ -0,0 +1,226 @@ +# Improved Custom SerialExecutor isolation checking for Concurrency Runtime + +* Proposal: [SE-0471](0471-SerialExecutor-isIsolated.md) +* Author: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.2)** +* Implementation: https://github.com/swiftlang/swift/pull/79788 & https://github.com/swiftlang/swift/pull/79946 +* Review: [Pitch](https://forums.swift.org/t/pitch-serialexecutor-improved-custom-serialexecutor-isolation-checking/78237/), [Review](https://forums.swift.org/t/se-0471-improved-custom-serialexecutor-isolation-checking-for-concurrency-runtime/78834), [Acceptance](https://forums.swift.org/t/accepted-se-0471-improved-custom-serialexecutor-isolation-checking-for-concurrency-runtime/79894) + +## Introduction + +In [SE-0424: Custom isolation checking for SerialExecutor](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0424-custom-isolation-checking-for-serialexecutor.md) we introduced a way for custom executors implementing the `SerialExecutor` protocol to assert and assume the static isolation if the dynamic check succeeded. This proposal extends these capabilities, allowing custom executors to not only "check and crash if assumption was wrong", but also check and act on the result of the check. + +## Motivation + +The previously ([SE-0424](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0424-custom-isolation-checking-for-serialexecutor.md)) introduced family of `Actor/assertIsolated()`, `Actor/preconditionIsolated()` and `Actor/assumeIsolated(operation:)` all rely on the `SerialExecutor/checkIsolated()` API which was introduced in the same proposal. + +These APIs all follow the "*pass or crash*" pattern. Where the crash is caused in order to prevent an incorrect assumption about isolation resulting in unsafe concurrent access to some isolated state. This is frequently used by methods which are "known to be called" on some specific actor to recover the dynamic (known at runtime) isolation information, into its static equivalent and therefore safely access some isolated state, like in this example: + +```swift +@MainActor +var counter: Int = num + +protocol P { + // Always called by the main actor, + // yet protocol author forgot to annotate the method or protocol using @MainActor + func actuallyWeKnowForSureThisIsCalledFromTheMainActor() +} + +struct Impl: P { + func actuallyWeKnowForSureThisIsCalledFromTheMainActor() { + MainActor.assumeIsolated { // we know this is safe here + counter += 1 + } + } +} + +@MainActor +func call(p: some P) { + p.actuallyWeKnowForSureThisIsCalledFromTheMainActor() +} +``` + +This works fine for many situations, however some libraries may need to be more careful when rolling out strict concurrency checks like these, and instead of crashing may want to choose to issue warnings before they adopt a mode that enforces correct use. + +Currently all APIs available are using the `checkIsolated()` API which must crash if called from a context not managed by the serial executor it is invoked on. This method is often implemented using `dispatchPrecondition()` when the serial executor is backed using `Dispatch` or custom `fatalError()` messages otherwise: + +```swift +final class ExampleExecutor: SerialExecutor { + func checkIsolated() { + dispatchPrecondition(condition: .onQueue(self.queue)) + } +} +``` + +This approach is better than not being able to participate in the checks at all (i.e. this API allows for advanced thread/queue sharing between actor and non-actor code), but it has two severe limitations: + +- the crash **messages** offered by these `checkIsolated()` crashes **are often sub-optimal** and confusing + - messages often don't include crucial information about which actor/executor the calling context was _actually_ executing on. Offering only "expected [...]" messages, leading to hard to debug crashes. +- it is **impossible** for the Swift runtime to offer **isolation violation warnings** + - because the Swift runtime _must_ call into a custom executor to verify its isolation, the "pass or crash" method will crash, rather than inform the runtime that a violation occurred and we should warn about it. + +Today, it is not possible for the Swift runtime to issue _warnings_ if something is detected to be on not the expected executor, but somehow we'd still like to continue without crashing the application. + +And for existing situations when `assumeIsolated()` is called and _should_ crash, by using this new API the Swift runtime will be able to provide _more informative_ messages, including all available context about the execution environment. + +## Proposed solution + +We propose to introduce a new `isIsolatingCurrentContext` protocol requirement to the `SerialExecutor` protocol: + +```swift +protocol SerialExecutor { + + /// May be called by the Swift runtime before `checkIsolated()` + /// in order to check for isolation violations at runtime, + /// and potentially issue isolation warnings. + /// + /// [...] + func isIsolatingCurrentContext() -> Bool + + // existing API, since SE-0424 + @available(SwiftStdlib 6.0, *) + func checkIsolated() // must crash if run on a context not managed by this serial executor + + // ... +} + +extension SerialExecutor { + /// Default implementation for backwards compatibility. + func isIsolatingCurrentContext() -> Bool? { nil } +} +``` + +The Swift runtime is free to call the `isIsolated` function whenever it wants to verify if the current context is appropriately isolated by some serial executor. + +In most cases implementing this new API is preferable to implementing `checkIsolated()`, as the Swift runtime is able to offer more detailed error messages when an isolation failure detected by a call to `isIsolatingCurrentContext()` is detected. + +## Detailed design + +The newly proposed `isIsolatingCurrentContext()` function participates in the previously established runtime isolation checking flow, and happens _before_ any calls to `checkIsolated()` are attempted. The following diagram explains the order of calls issued by the runtime to dynamically verify an isolation when e.g. `assumeIsolated()` is called: + +![diagram illustrating which method is called when](0471-is-isolated-flow.png) + +There are a lot of conditions here and availability of certain features also impacts this decision flow, so it is best to refer to the diagram for detailed analysis of every situation. However the most typical situation involves executing on a task, which has a potentially different executor than the `expected` one. In such situation the runtime will: + +- check for the existence of a "current" task, +- (fast path) if a task is present: + - compare the current task's serial executor it is isolated to (if any) with the expected serial executor, + +- :new: if **`isIsolatingCurrentContext`** **is available** on the `expected` executor: + - invoke `isIsolatingCurrentContext` + - ✅ if it returned true: pass the check. + - :x: if it returned false: fail the check. + - The runtime will **not** proceed to call `checkIsolated` after `isIsolated` is invoked! + +- if **`isIsolatingCurrentContext`** is **not available** on the expected executor, but **`checkIsolated`** **is available**: + - invoke `expected.checkIsolated` which will crash :x: or pass :white_check_mark: depending on its internal checking. +- if neither `checkIsolated` or `isIsolatingCurrentContext` are available, + - :x: crash with a best effort message. + + +This proposal specifically adds the "if `isIsolatingCurrentContext` is available" branch into the existing logic for confirming the isolation expectation. + +If `isIsolatingCurrentContext` is available, effectively it replaces `checkIsolated` because it does offer a sub-par error message experience and is not able to offer a warning if Swift would be asked to check the isolation but not crash upon discovering a violation. + +### The `isIsolatingCurrentContext` checking mode + +The `isIsolatingCurrentContext` method effectively replaces the `checkIsolated` method, because it can answer the same question if it is implemented. + +Some runtimes may not be able to implement a the returning `isIsolatingCurrentContext`, and they are not required to implement the new protocol requirement. + +The default implementation returns `nil` which is to be interpreted by the runtime as "unknown" or "unable to confirm the isolation", and the runtime may proceeed to call futher isolation checking APIs when this function returned `nil`. + +The general guidance about which method to implement is to implement `isIsolatingCurrentContext` whenever possible. This method can be used by the Swift runtime in "warning mode". When running a check in this mode, the `checkIsolated` method cannot and will not be used because it would cause an unexpected crash. An executor may still want to implement the `checkIsolated` function if it truly is unable to return a true/false response to the isolation question, but can only assert on an illegal state. The `checkIsolated` function will not be used when the runtime cannot tollerate the potential of crashing while performing an isolation check (e.g. isolated conformance checks, or when issuing warnings). + +The runtime will always invoke the `isIsolatingCurrentContext` before making attempts to call `checkIsolated`, and if the prior returns either `true` or `false`, the latter (`checkIsolated`) will not be invoked at all. + +### Checking if currently isolated to some `Actor` + +We also introduce a way to obtain `SerialExecutor` from an `Actor`, which was previously not possible. + +This API needs to be scoped because the lifetime of the serial executor must be tied to the Actor's lifetime: + +```swift +extension Actor { + /// Perform an operation with the actor's ``SerialExecutor``. + /// + /// This converts the actor's ``Actor/unownedExecutor`` to a ``SerialExecutor`` while + /// retaining the actor for the duration of the operation. This is to ensure the lifetime + /// of the executor while performing the operation. + @_alwaysEmitIntoClient + @available(SwiftStdlib 5.1, *) + public nonisolated func withSerialExecutor(_ operation: (any SerialExecutor) throws(E) -> T) throws(E) -> T + + /// Perform an operation with the actor's ``SerialExecutor``. + /// + /// This converts the actor's ``Actor/unownedExecutor`` to a ``SerialExecutor`` while + /// retaining the actor for the duration of the operation. This is to ensure the lifetime + /// of the executor while performing the operation. + @_alwaysEmitIntoClient + @available(SwiftStdlib 5.1, *) + public nonisolated func withSerialExecutor(_ operation: (any SerialExecutor) async throws(E) -> T) async throws(E) -> T + +} +``` + +This allows developers to write "warn if wrong isolation" code, before moving on to enable preconditions in a future release of a library. This gives library developers, and their adopters, time to adjust their code usage before enabling more strict validation mode in the future, for example like this: + +```swift +func something(operation: @escaping @isolated(any) () -> ()) { + operation.isolation.withSerialExecutor { se in + if !se.isIsolatingCurrentContext() { + warn("'something' must be called from the same isolation as the operation closure is isolated to!" + + "This will become a runtime crash in future releases of this library.") + } + } +} +``` + + + +This API will be backdeployed and will be available independently of runtime version of the concurrency runtime. + +### Compatibility strategy for custom SerialExecutor authors + +New executor implementations should prioritize implementing `isIsolatingCurrentContext` when available, using an appropriate `#if swift(>=...)` check to ensure compatibility. Otherwise, they should fall back to implementing the crashing version of this API: `checkIsolated()`. + +For authors of custom serial executors, adopting this feature is an incremental process and they can adopt it at their own pace, properly guarding the new feature with necessary availability guards. This feature requires a new version of the Swift concurrency runtime which is aware of this new mode and therefore able to call into the new checking function, therefore libraries should implement and adopt it, however it will only manifest itself when the code is used with a new enough concurrency runtime + +As a result, this change should cause little to no disruption to Swift concurrency users, while providing an improved error reporting experience if using executors which adopt this feature. + +## Source Compatibility + +This proposal is source compatible, a default implementation of the new protocol requirement is introduced along with it, allowing existing `SerialExecutors` to compile without changes. + +## Binary Compatibility + +This proposal is ABI additive, and does not cause any binary incompatibility risks. + +## Future directions + +### Expose `isIsolated` on (Distributed)Actor and MainActor? + +We are _not_ including new public API on Actor types because of concerns of this function being abused. + +If we determine that there are significant good use-cases for this method to be exposed, we might reconsider this position. + +## Alternatives considered + +### Somehow change `checkIsolated` to return a bool value + +This would be ideal, however also problematic since changing a protocol requirements signature would be ABI breaking. + +### Deprecate `checkIsolated`? + +In order to make adoption of this new mode less painful and not cause deprecation warnings to libraries which intend to support multiple versions of Swift, the `SerialExcecutor/checkIsolated` protocol requirement remains _not_ deprecated. It may eventually become deprecated in the future, but right now we have no plans of doing so. + +### Model the SerialExecutor lifetime dependency on Actor using `~Escapable` + +It is currently not possible to express this lifetime dependency using `~Escapable` types, because combining `any SerialExecutor` which is an `AnyObject` constrained type, cannot be combined with `~Escapable`. Perhaps in a future revision it would be possible to offer a non-escapable serial executor in order to model this using non-escapable types, rather than a `with`-style API. + +## Changelog + +- added way to obtain `SerialExecutor` from `Actor` in a safe, scoped, way. This enables using the `isIsolatingCurrentContext()` API when we have an `any Actor`, e.g. from an `@isolated(any)` closure. +- changed return value of `isIsolatingCurrentContext` from `Bool` to `Bool?`, where the `nil` is to be interpreted as "unknown", and the default implementation of `isIsolatingCurrentContext` now returns `nil`. +- removed the manual need to signal to the runtime that the specific executor supports the new checking mode. It is now detected by the compiler and runtime, checking for the presence of a non-default implementation of the protocol requirement. diff --git a/proposals/0471-is-isolated-flow.graffle b/proposals/0471-is-isolated-flow.graffle new file mode 100644 index 0000000000..05f3a5aa8d Binary files /dev/null and b/proposals/0471-is-isolated-flow.graffle differ diff --git a/proposals/0471-is-isolated-flow.png b/proposals/0471-is-isolated-flow.png new file mode 100644 index 0000000000..6a9e939edf Binary files /dev/null and b/proposals/0471-is-isolated-flow.png differ diff --git a/proposals/0472-task-start-synchronously-on-caller-context.md b/proposals/0472-task-start-synchronously-on-caller-context.md new file mode 100644 index 0000000000..1d266bfba4 --- /dev/null +++ b/proposals/0472-task-start-synchronously-on-caller-context.md @@ -0,0 +1,512 @@ +# Starting tasks synchronously from caller context + +* Proposal: [SE-0472](0472-task-start-synchronously-on-caller-context.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 6.2)** +* Implementation: + * https://github.com/swiftlang/swift/pull/79608 + * https://github.com/swiftlang/swift/pull/81428 + * https://github.com/swiftlang/swift/pull/81572 +* Review: ([pitch](https://forums.swift.org/t/pitch-concurrency-starting-tasks-synchronously-from-caller-context/77960/)) ([first review](https://forums.swift.org/t/se-0472-starting-tasks-synchronously-from-caller-context/78883)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0472-starting-tasks-synchronously-from-caller-context/79311)) ([second review](https://forums.swift.org/t/second-review-se-0472-starting-tasks-synchronously-from-caller-context/79683)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0472-starting-tasks-synchronously-from-caller-context/80037)) + +## Introduction + +Swift Concurrency's primary means of entering an asynchronous context is creating a Task (structured or unstructured), and from there onwards it is possible to call asynchronous functions, and execution of the current work may _suspend_. + +Entering the asynchronous context today incurs the creating and scheduling of a task to be executed at some later point in time. This initial delay may be wasteful for tasks which perform minimal or no (!) work at all. + +This initial delay may also be problematic for some situations where it is known that we are executing on the "right actor" however are *not* in an asynchronous function and therefore in order to call some different asynchronous function we must create a new task and introduce subtle timing differences as compared to just being able to call the target function–which may be isolated to the same actor we're calling from–immediately. + +## Motivation + +Today, the only way to enter an asynchronous execution context is to create a new task which then will be scheduled on the global concurrent executor or some specific actor the task is isolated to, and only once that task is scheduled execution of it may begin. + +This initial scheduling delay can be problematic in some situations where tight control over execution is required. While for most tasks the general semantics are a good choice–not risking overhang on the calling thread–we have found through experience that some UI or performance sensitive use-cases require a new kind of semantic: immediately starting a task on the calling context. After a suspension happens the task will resume on the the executor as implied by the task operation's isolation, as would be the case normally. + +This new behavior can especially beneficial for tasks, which *may run to completion very quickly and without ever suspending.* + +A typical situation where this new API may be beneficial often shows up with @MainActor code, such as: + +```swift +@MainActor var thingsHappened: Int = 0 + +@MainActor func asyncUpdateThingsHappenedCounter() async { + // for some reason this function MUST be async + thingsHappened += 1 +} + +func synchronousFunction() { + // we know this executes on the MainActor, and can assume so: + MainActor.assumeIsolated { + // The following would error: + // await asyncUpdateThingsHappenedCounter() + // because is it is an async call; cannot call from synchronous context + } + + // Using the newly proposed Immediate Task: + let task = Task.immediate { + // Now we CAN call the asynchronous function below: + await asyncUpdateThingsHappenedCounter() + } + + // cannot await on the `task` since still in synchronous context +} +``` + +The above example showcases a typical situation where this new API can be useful. While `assumeIsolated` gives us a specific isolation, but it would not allow us to call the async functions, as we are still in a synchronous context. + +The proposed `Task.immediate` API forms an async context on the calling thread/task/executor, and therefore allows us to call into async code, at the risk of overhanging on the calling executor. + +While this should be used sparingly, it allows entering an asynchronous context *synchronously*. + +## Proposed solution + +We propose the introduction of a new family of Task creation APIs collectively called "**immediate tasks**", which create a task and use the calling execution context to run the task's immediately, before yielding control back to the calling context upon encountering the first suspension point inside the immediate task. + +Upon first suspension inside the immediate task, the calling executor is freed up and able to continue executing other work, including the code surrounding the creation of the immediate task. This happens specifically when when a real suspension happens, and not for "potential suspension point" (which are marked using the `await` keyword). + +The canonical example for using this new API is using an *unstructured immediate task* like this: + +```swift +func synchronous() { // synchronous function + // executor / thread: "T1" + let task: Task = Task.immediate { + // executor / thread: "T1" + guard keepRunning() else { return } // synchronous call (1) + + // executor / thread: "T1" + await noSuspension() // potential suspension point #1 // (2) + + // executor / thread: "T1" + await suspend() // potential suspension point #2 // (3), suspend, (5) + // executor / thread: "other" + } + + // (4) continue execution + // executor / thread: "T1" +} +``` + +The task created by the `immediate` function begins running immediately _on the calling executor (and thread)_ without any scheduling delay. This new task behaves generally the same as any other unstructured task, it gets a copy of the outer context's task locals, and uses the surrounding context's base priority as its base priority as well. + +Since the task started running immediately, we're able to perform some calls immediately inside it, and potentially return early, without any additional scheduling delays. + +If a potential suspension point does not actually suspend, we still continue running on the calling context. For example, if potential suspension point `#1` did not suspend, we still continue running synchronously until we reach potential suspension point `#2` which for the sake of discussion let's say does suspend. At this point the calling thread continues executing the code that created the unstructured task. + +> You can refer to the `(N)` numbers in the above snippet to follow the execution order of this example execution. Specifically, once the execution reaches (3) the calling thread stops executing the unstructured task, and continues executing at (4). Eventually, when the unstructured task is resumed, it gets woken up at (5) and continues running on some other executor and/or thread. + +## Detailed design + +We propose the introduction of a family of APIs that allow for the creation of *immediate tasks*. + +The most frequent use of this API is likely going to be the unstructured task one. This is because we are able to enter an asynchronous context from a synchronous function using it: + +```swift +extension Task { + + @discardableResult + public static func immediate( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: consuming (any TaskExecutor)? = nil, + @_inheritActorContext(always) operation: sending @escaping () async throws(Failure) -> Success + ) -> Task + + @discardableResult + public static func immediateDetached( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: consuming (any TaskExecutor)? = nil, + @_inheritActorContext(always) operation: sending @escaping () async throws(Failure) -> Success + ) -> Task +} +``` + +We also introduce the same API for all kinds of task groups. These create child tasks, which participate in structured concurrency as one would expect of tasks created by task groups. + +```swift +extension (Throwing)TaskGroup { + // Similar semantics as the usual 'addTask'. + func addImmediateTask( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + operation: sending @escaping () async throws -> ChildTaskResult + ) + + // Similar semantics as the usual 'addTaskUnlessCancelled'. + func addImmediateTaskUnlessCancelled( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + operation: sending @escaping () async throws -> ChildTaskResult + ) +} + +extension (Throwing)DiscardingTaskGroup { + // Similar semantics as the usual 'addTask'. + func addImmediateTask( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + operation: sending @escaping () async throws -> ChildTaskResult + ) + + // Similar semantics as the usual 'addTaskUnlessCancelled'. + func addImmediateTaskUnlessCancelled( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + operation: sending @escaping () async throws -> ChildTaskResult + ) +} +``` + +The `addImmediateTask` function mirrors the functionality of `addTask`, unconditionally adding the task to the task group, while the `addImmediateTaskUnlessCancelled` mirrors the `addTaskUnlessCancelled` which only adds the task to the group if the group (or task we're running in, and therefore the group as well) are not cancelled. + +### Isolation rules + +Due to the semantics of "starting on the caller context", the isolation rules of the closure passed to `Task.immediate` need to be carefully considered. + +The isolation rules for the `immediate` family of APIs need to account for this synchronous "first part" of the execution. We propose the following set of rules to make this API concurrency-safe: + +- The operation closure is `sending`. +- The operation closure may only specify an isolation (e.g. `{ @AnyGlobalActor in }`) + +Another significant way in which `Task.immediate` differs from the `Task.init` initializer is that the inheritance of the surrounding actor context is performed more eagerly. This is because immediate tasks always attempt to execute on the "current" executor, unlike `Task.init` which only execute on the "surrounding" actor context when the task's operation closure closes over an isolated parameter, or was formed in a global actor isolated context: + +```swift +@MainActor +func alreadyDefinitelyOnMainActor() { + Task { + // @MainActor isolated, enqueue + } + Task.immediate { + // @MainActor isolated, run immediately + } +} +``` + +```swift +actor Caplin { + var anything: Int = 0 + + func act() { + Task { + // nonisolated, enqueue on global concurrent executor + } + Task { + // self isolated, enqueue + self.anything // any capture of 'self' + } + + Task.immediate { // regardless of captures + // self isolated, run immediately + } + } +} + +func go(with caplin: isolated Caplin) async { + Task { + // nonisolated, enqueue on global concurrent executor + } + Task { + // 'caplin' isolated, enqueue + caplin.anything // any capture of 'caplin' + } + + Task.immediate { // regardless of captures + // 'caplin' isolated, run immediately + } + } +} + +func notSpecificallyIsolatedAnywhere() { + Task { + // nonisolated, enqueue on global concurrent executor + } + + Task.immediate { + // nonisolated. + // attempt to run on current executor, + // or enqueue to global as fallback + } + } +} +``` + +The `Task.immediateDetached` does not inherit isolation automatically, same as it's non-immediate `Task.detached` equivalent. + +Task group methods which create immediate child tasks do not inherit isolation automatically, although they are allowed to specify an isolation explicitly. This is the same as existing TaskGroup APIs (`addTask`). + +### Scheduling immediate tasks given matching current and requested isolation + +The Swift concurrency runtime maintains a notion of the "current executor" in order to be able to perform executor switching and isolation checking dynamically. This information is managed runtime, and is closely related to compile time isolation rules, but it is also maintained throughout nonisolated and synchronous functions. + +Immediate tasks make use of this executor tracking to determine on which executor we're asking the task to "immediately" execute. It is possible to start an immediate task in a synchronous context, and even require it to have some specific isolation. + +The following example invokes the synchronous `sayHello()` function from a `@MainActor` isolated function. The static information about this isolation is _lost_ by the synchronous function. And the compiler will assume, that the `sayHello()` function is not isolated to any specific context -- after all, the actual isolated context would depending on where we call it from, and we're not passing an `isolated` parameter to this synchronous function. + +By using an immediate task the runtime is able to notice that the requested, and current, executor are actually the same (`MainActor`) and therefore execute the task _immediately_ on the caller's executor _and_ with the expected `@MainActor` isolation, which is guaranteed to be correct: + +```swift +@MainActor var counterUsual = 0 +@MainActor var counterImmediate = 0 + +@MainActor +func sayHelloOnMain() { + sayHello() // call synchronous function from @MainActor +} + +// synchronous function +func sayHello() { + // We are "already on" the main actor + MainActor.assertIsolated() + + // Performs an enqueue and will execute task "later" + Task { @MainActor in + counterUsual += 1 + } + // At this point (in this specific example), `counterUsual` is still 0. + // We did not "give up" the main actor executor, so the new Task could not execute yet. + + // Execute the task immediately on the calling context (!) + Task.immediate { @MainActor in + counterImmediate += 1 + } + // At this point (in this specific cexample), + // `counterImmediate` is guaranteed to == 1! +} +``` + +The difference between the use of `Task.init` and `Task.immediate` is _only_ in the specific execution ordering semantics those two tasks exhibit. + +Because we are dynamically already on the expected executor, the immediate task will not need to enqueue and "run later" the new task, but instead will take over the calling executor, and run the task body immediately (up until the first suspension point). + +This can have importand implications about the observed order of effects and task execution, so it is important for developers to internalize this difference in scheduling semantics. + +If the same `sayHello()` function were to be invoked from some execution context _other than_ the main actor, both tasks–which specify the requested isolation to be `@MainActor`–will perform the usual enqueue and "run later": + +```swift +@MainActor var counterUsual = 0 +@MainActor var counterImmediate = 0 + +actor Caplin { + func sayHelloFromCaplin() { + sayHello() // call synchronous function from Caplin + } +} + +func sayHello() { + Task { @MainActor in // enqueue, "run later" + counterUsual += 1 + } + Task.immediate { @MainActor in // enqueue, "run later" + counterImmediate += 1 + } + + // at this point, no guarantees can be made ablue the values of the `counter` variables +} +``` + +This means that a `Task.immediate` can be used to opportunistically attempt to "run immediately, if the caller matches my required isolation, otherwise, enqueue and run the task later". Which is a semantic that many developers have requested in the past, most often in relation to the `MainActor`. + +The same technique of specifying a required target isolation may be used with the new TaskGroup APIs, such as `TaskGroup/addImmediateTask`. + +### Interaction with `Actor/assumeIsolated` + +In [SE-0392: Custom Actor Executors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md) we introduced the ability to dynamically recover isolation information using the `Actor/assumeIsolated` API. It can be used to dynamically recover the runtime information about whether we are executing on some specific actor. + +The `assumeIsolated` shares some ideas with `Task/immediate` however it is distinctly different. For example, while both APIs can effectively be used to "notice we are running on the expected actor, and therefore perform some work on its context". However, `assumeIsolated` does _not_ create a new asynchronous context, while `Task.immediate` does: + +```swift +@MainActor +var state: Int = 0 + +@MainActor +func asyncMainActorMethod() async { } + +func synchronous() { + // assert that we are running "on" the MainActor, + // and therefore can access its isolated state: + MainActor.assumeIsolated { + num +=1 // ✅ ok + + await asyncMainActorMethod() // ❌ error: 'async' call in a function that does not support concurrency + } + +} +``` + +We can compose `assumeIsolated` with `Task.immediate` to both assert that the current execution context must the the expected actor, and form a new asynchronous task that will immediately start on that actor: + +```swift +func alwaysCalledFromMainActor() { // we know this because e.g. documentation, but the API wasn't annotated + MainActor.assumeIsolated { // @MainActor isolated + assert(num == 0) + Task.immediate { // @MainActor isolated + num +=1 // ✅ ok + assert(num == 1) // since we are guaranteed nothing else executed since the 'num == 0' assertion + + await asyncMainActorMethod() // ✅ ok + } + } +} +``` + +The immediately started task will not suspend and context switch until any of the called async methods does. For example, we are guaranteed that there will be no interleaved code execution between the `assert(num == 0)` in our example, and the `num += 1` inside the synchronously started task. + +After the suspension point though, there may have been other tasks executed on the main actor, and we should check the value of `num` again. + +### Immediate child tasks + +Immediate child tasks tasks can be created using the various `*TaskGroup/addImmediateTask*` methods behave similarily to their normal structured child task API counterparts (`*TaskGroup/addTask*`). + +Child tasks, including immediate child tasks, do not infer their isolation from the enclosing context, and by default are `nonisolated`. + +```swift +actor Worker { + func workIt(work: Work) async { + await withDiscardingTaskGroup { + group.addImmediateTask { // nonisolated + work.synchronousWork() + } + } + } +} +``` + +While the immediate task in the above example is indeed `nonisolated` and does not inherit the Worker's explicit isolation, it will start out immediately on the Worker's executor. Since this example features _no suspension points_ in the task group child tasks, this is effectively synchronously going to execute those child tasks on the caller (`self`). In other words, this is not performing any of its work in parallel. + +If we were to modify the work to have potential suspension points like so: + +```swift +actor Worker { + func workIt(work: Work) async { + await withDiscardingTaskGroup { + group.addImmediateTask { // nonisolated + // [1] starts on caller immediately + let partialResult = await work.work() // [2] actually suspends + // [3] resumes on global executor (or task executor, if there was one set) + work.moreWork(partialResult) + } + } + } +} +``` + +The actual suspension happening in the `work()` call, means that this task group actually would exhibit some amount of concurrent execution with the calling actor -- the remainder between `[2]` and `[3]` would execute on the global concurrent pool -- concurrently to the enclosing actor. + +Cancellation, task locals, priority escalation, and any other structured concurrency semantics remain the same for structured child tasks automatically for unstructured tasks created using the `Task/immediate[Detached]` APIs. + +The only difference in behavior is where these synchronously started tasks _begin_ their execution. + +## Source compatibility + +This proposal is purely additive, and does not cause any source compatibility issues. + +## ABI compatibility + +This proposal is purely ABI additive. + +## Alternatives considered + +### Dynamically asserting isolation correctness + +An important use case of this API is to support calling into an actor isolated context when in a synchronous function that is dynamically already running on that actor. This situation can occur both with instance actors and global actors, however the most commonly requested situation where this shows up is synchronous handler methods in existing frameworks, and which often may have had assumptions about the main thread, and did not yet annotate their API surface with @MainActor annotations. + +It would be possible to create a _dynamically asserting_ version of `Task.immediate`, which does handle the happy path where indeed we "know" where we're going to be called quite well, but gives a *false sense of security* as it may crash at runtime, in the same way the `Actor/preconditionIsolated()` or `Actor/assumeIsolated` APIs do. We believe we should not add more such dynamically crashing APIs, but rather lean into the existing APIs and allow them compose well with any new APIs that should aim to complement them. + +The dynamically asserting version would be something like this: + +```swift +// Some Legacy API: documented to be invoked on main thread but NOT @MainActor annotated and NOT 'async' +func onSomethingHappenedAlwaysOnMainThread(something: Something) { + // we "know" we are on the MainActor, however this is a legacy API that is not an 'async' method + // so we cannot call any other async APIs unless we create a new task. + Task.immediate { @MainActor in + await showThingy() + } +} + +func onSomethingHappenedSometimesOnMainThread(something: Something) { + // 💥 Must assert at runtime if not on main thread + Task.immediate { @MainActor in + await showThingy() + } +} + +func showThingy() async { ... } +``` + +This implementation approach yields safe looking code which unfortunately may have to assert at runtime, rather than further improve the compile time safety properties of Swift Concurrency. + +> See *Future Directions: Dynamically "run synchronously if in right context, otherwise enqueue as usual"* for a future direction that would allow implementing somewhat related APIs in a more elegant and correct way. + +### Banning from use in async contexts (@available(*, noasync)) + +During earlier experiments with such API it was considered if this API should be restricted to only non-async contexts, by marking it `@available(*, noasync)` however it quickly became clear that this API also has specific benefits which can be used to ensure certain ordering of operations, which may be useful regardless if done from an asynchronous or synchronous context. + +## Future Directions + +### Partial not-sending closure semantics + +The isolation rules laid out in this proposal are slightly more conservative than necessary. + +Technically one could make use of the information that the part of the closure up until the first potential suspension point is definitely running synchronously, and therefore even access state that would not be able to be accessed even under region isolation analysis rules. + +We believe that most common situations will be handled well enough by region analysis, and sending closures, however this is a future direction that could be explored if it becomes more apparent that implementing these more complex semantics would be very beneficial. + +For example, such analysis could enable the following: + +```swift +actor Caplin { + var num: Int = 0 + + func check() { + Task.immediateDetached { + num += 1 // could be ok; we know we're synchronously executing on caller + + try await Task.sleep(for: .seconds(1)) + + num += 1 // not ok anymore; we're not on the caller context anymore + } + + num += 1 // always ok + } +} +``` + +### Implementation detail: Expressing closure isolation tied to function parameter: `@isolated(to:)` + +The currently proposed API is working within the limitations of what is expressible in today's isolation model. It would be beneficial to be able to express the immediate API if we could spell something like "this closure must be isolated to the same actor as the calling function" which would allow for the following code: + +```swift +@MainActor +func test() { + Task.immediate { /* inferred to be @MainActor */ + num += 1 + } +} + +@MainActor var num = 0 +``` + +The way to spell this in an API could be something like this: + +```swift +public static func immediate( + ... + isolation: isolated (any Actor)? = #isolation, + operation: @escaping @isolated(to: isolation) sending async throws(Failure) -> Success, +) -> Task +``` + +The introduction of a hypothetical `@isolated(to:)` paired with an `isolated` `#isolation` defaulted actor parameter, would allow us to express "the *operation* closure statically inherits the exact same isolation as is passed to the isolation parameter of the `immediate` method". This naturally expresses the semantics that the `immediate` is offering, and would allow to _stay_ on that isolation context after resuming from the first suspension inside the operation closure. + +Implementing this feature is a large task, and while very desirable we are not ready yet to commit to implementing it as part of this proposal. If and when this feature would become available, we would adopt it in the `immediate` APIs. + +### Changelog + +- Moved the alternative considered of "attempt to run immediately, or otherwise just enqueue as usual" into the proposal proper diff --git a/proposals/0473-clock-epochs.md b/proposals/0473-clock-epochs.md new file mode 100644 index 0000000000..c674352d3c --- /dev/null +++ b/proposals/0473-clock-epochs.md @@ -0,0 +1,60 @@ +# Clock Epochs + +* Proposal: [SE-0473](0473-clock-epochs.md) +* Authors: [Philippe Hausler](https://github.com/phausler) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Accepted** +* Implementation: [PR #80409](https://github.com/swiftlang/swift/pull/80409) +* Review: ([pitch](https://forums.swift.org/t/pitch-suspendingclock-and-continuousclock-epochs/78017)) ([review](https://forums.swift.org/t/se-0473-clock-epochs/78923)) ([acceptance](https://forums.swift.org/t/accepted-se-0473-clock-epochs/79221)) + +## Introduction + +[SE-0329: Clock, Instant, and Duration](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0329-clock-instant-duration.md) introduced three concrete clock types: `SuspendingClock`, `ContinuousClock`, and `UTCClock`. While not all clocks have a meaningful concept of a reference or zero instant, `SuspendingClock` and `ContinuousClock` do, and having access to it can be useful. + +## Motivation + +The `Instant` type of a `Clock` represents a moment in time as measured by that clock. `Clock` intentionally imposes very few requirements on `Instant` because different kinds of clocks can have very different characteristics. Just because something does not belong on the generic `Clock` protocol, however, does not mean it shouldn't be exposed in the interface of a concrete clock type. + +Many clocks have a concept of a reference instant, also called an "epoch", that has special meaning for the clock. For example, the Unix `gettimeofday` function measures the nominal elapsed time since 00:00 UTC on January 1st, 1970, an instant often called the "Unix epoch". Swift's `SuspendingClock` and `ContinuousClock` are defined using system facilities that similarly measure time relative to an epoch, and while the exact definition of the epoch is system-specific, it is at least consistent for any given system. This means that durations since the epoch can be meaningfully compared within the system, even across multiple processes or with code written in other languages (as long as they use the same system facilities). + +## Proposed solution + +Two new properties will be added, one to `SuspendingClock` and another to `ContinuousClock`. These properties define the system epoch that all `Instant` types for the clock are derived from; practically speaking, this is the "zero" point for these clocks. Since the values may be relative to the particular system they are being used on, their names reflect that they are a system-specific definition and should not be expected to be consistent (or meaningfully serializable) across systems. + +## Detailed design + +```swift +extension ContinuousClock { + public var systemEpoch: Instant { get } +} + +extension SuspendingClock { + public var systemEpoch: Instant { get } +} +``` + +On most platforms, including Apple platforms, Linux, and Windows, the system epoch of these clocks is set at boot time, and so measurements relative to the epoch can used to gather information such as the uptime or active time of a system: + +```swift +let clock = ContinousClock() +let uptime = clock.now - clock.systemEpoch +``` + +Likewise: + +```swift +let clock = SuspendingClock() +let activeTime = clock.now - clock.systemEpoch +``` + +However, this cannot be guaranteed for all possible platforms. A platform may choose to use a different instant for its system epoch, perhaps because the concept of uptime doesn't apply cleanly on the platform or because it is intentionally not exposed to the programming environment for privacy reasons. + +## ABI compatibility + +This is a purely additive change and provides no direct impact to existing ABI. It only carries the ABI impact of new properties being added to an existing type. + +## Alternatives considered + +We considered adding a constructor or static member to `SuspendingClock.Instant` and `ContinousClock.Instant` instead of on the clock. However, placing it on the clock itself provides a more discoverable and nameable location. + +As proposed, `systemEpoch` is an informal protocol that works across multiple clock implementations. We consider formalizing it as a new protocol, but ultimately we decided not to because no generic function made much sense that would not be better served with generic specialization or explicit clock parameter types. diff --git a/proposals/0474-yielding-accessors.md b/proposals/0474-yielding-accessors.md new file mode 100644 index 0000000000..a109cee548 --- /dev/null +++ b/proposals/0474-yielding-accessors.md @@ -0,0 +1,668 @@ +# Yielding accessors + +* Proposal: [SE-0474](0474-yielding-accessors.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Nate Chandler](https://github.com/nate-chandler), [Joe Groff](https://github.com/jckarter/) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Accepted** +* Vision: [A Prospective Vision for Accessors in Swift](https://github.com/rjmccall/swift-evolution/blob/accessors-vision/visions/accessors.md) +* Implementation: Partially available on main behind the frontend flag `-enable-experimental-feature CoroutineAccessors` +* Review: ([pitch 1](https://forums.swift.org/t/modify-accessors/31872)), ([pitch 2](https://forums.swift.org/t/pitch-modify-and-read-accessors/75627)), ([pitch 3](https://forums.swift.org/t/pitch-3-yielding-coroutine-accessors/77956)), ([review](https://forums.swift.org/t/se-0474-yielding-accessors/79170)), [Acceptance](https://forums.swift.org/t/accepted-se-0474-yielding-accessors/80273) + +## Introduction + +We propose the introduction of two new accessors, `yielding mutate` and `yielding borrow`, for implementing computed properties and subscripts alongside the current `get` and `set`. + +By contrast with `get` and `set`, whose bodies behave like traditional methods, the body of a `yielding` accessor will be a coroutine, using a new contextual keyword `yield` to pause the coroutine and lend access of a value to the caller. +When the caller ends its access to the lent value, the coroutine's execution will continue after `yield`. + +These `yielding` accessors enable values to be accessed and modified without requiring a copy. +This is essential for noncopyable types and often desirable for performance even with copyable types. + +This feature has been available (but not supported) since Swift 5.0 via the `_modify` and `_read` keywords. +Additionally, the feature is available via `read` and `modify` on recent builds from the compiler's `main` branch using the flag `-enable-experimental-feature CoroutineAccessors`. + +## Motivation + +### `yielding mutate` + +Swift's `get`/`set` syntax allows users to expose computed properties and subscripts that behave as l-values. +This powerful feature allows for the creation of succinct idiomatic APIs, such as this use of `Dictionary`'s defaulting subscript: + +```swift +var wordFrequencies: [String:Int] = [:] +wordFrequencies["swift", default: 0] += 1 +// wordFrequencies == ["swift":1] +``` + +While this provides the illusion of "in-place" mutation, it actually involves three separate operations: +1. a `get` of a copy of the value +2. the mutation, performed on that returned value +3. finally, a `set` replacing the original value with the mutated temporary value. + +This can be seen by performing side effects inside of the getter and setter; for example: + +```swift +struct GetSet { + var x: String = "👋🏽 Hello" + + var property: String { + get { print("Getting",x); return x } + set { print("Setting",newValue); x = newValue } + } +} + +var getSet = GetSet() +getSet.property.append(", 🌍!") +// prints: +// Getting 👋🏽 Hello +// Setting 👋🏽 Hello, 🌍! +``` + +#### Performance + +When the property or subscript is of copyable type, this simulation of in-place mutation works well for user ergonomics but has a major performance shortcoming, which can be seen even in our simple `GetSet` type above. +Strings in Swift aren't bitwise-copyable types; once they grow beyond a small fixed size, they allocate a reference-counted buffer to hold their contents. +Mutation is handled via the copy-on-write technique: +When you make a copy of a string, only the reference to the buffer is copied, not the buffer itself. +Then, when either copy of the string is mutated, the mutation operation checks if the buffer is uniquely referenced. +If it isn't (because the string has been copied), it first duplicates the buffer before mutating it, preserving the value semantics of `String` while avoiding unnecessary eager copies. + +Given this, we can see a performance problem when appending to `GetSet.property` in our example above: + +- `GetSet.property { get }` is called and returns a copy of `x`. +- Because a copy is returned, the buffer backing the string is no longer uniquely referenced. +- The append operation must therefore duplicate the buffer before mutating it. +- `GetSet.property { set }` writes this copy back over the top of `x`, destroying the original string. +- The original buffer's reference count drops to zero, and it's destroyed too. + +So, despite looking like in-place mutation, every mutating operation on `x` made through `property` is actually causing a full copy of `x`'s backing buffer. +This is a linear-time operation. +If we appended to this property in a loop, that loop would end up being quadratic in complexity. +This is likely very surprising to the developer and is a frequent performance pitfall. + +[An accessor](#design-modify) which _lends_ a value to the caller in place, without producing a temporary copy, is desirable to avoid this quadratic time complexity while preserving Swift's ability for computed properties to provide expressive and ergonomic APIs. + +#### Non-`Copyable` types + +When the value being mutated is noncopyable, the common `get`/`set` pattern often becomes impossible to implement: +the very first step of mutation makes a copy! +Thus, `get` and `set` can't be used to wrap access to a noncopyable value: + +```swift +struct UniqueString : ~Copyable {...} + +struct UniqueGetSet : ~Copyable { + var x: UniqueString + + var property: UniqueString { + get { // error: 'self' is borrowed and cannot be consumed + x + } + set { x = newValue } + } +} +``` + +`get` borrows `self` but then tries to _give_ ownership of `x` to its caller. Since `self` is only borrowed, it cannot give up ownership of its `x` value, and since `UniqueString` is not `Copyable`, `x` cannot be copied either, making `get` impossible to implement. +Therefore, to provide computed properties on noncopyable types with comparable expressivity, we again need [an accessor](#design-modify) that borrows `self` and _lends_ access to `x` to its caller without implying independent ownership. + +### `yielding borrow` + +For properties and subscripts of noncopyable type, the current official accessors are insufficient not only for mutating, but even for _inspecting_. +Even if we remove `set` from our `UniqueGetSet` type above, we still hit the same error in its getter: + +```swift +struct UniqueString : ~Copyable {...} + +struct UniqueGet : ~Copyable { + var x: UniqueString + + var property: UniqueString { + get { // error: 'self' is borrowed and cannot be consumed + return x + } + } +} +``` + +As discussed above, `UniqueGet.property { get }` borrows `self` but attempts to transfer ownership of `self.x` to the caller, which is impossible. + +This particular error could be addressed by marking the getter `consuming`: + +```swift +struct UniqueString : ~Copyable {...} + +struct UniqueConsumingGet : ~Copyable { + var x: UniqueString + + var property: UniqueString { + consuming get { + return x + } + } +} +``` + +This allows the getter to take ownership of the `UniqueConsumingGet`, +allowing it to destructively extract `x` and transfer ownership to the caller. +Here's how that looks in the caller: + +```swift +let container = UniqueConsumingGet() +let x = container.property // consumes container! +// container is no longer valid +``` + +This might sometimes be desirable, but for many typical uses of properties and subscripts, it is not. If a container holds a number of noncopyable fields, it should be possible to inspect each field in turn, but doing so wouldn't be possible if inspecting one consumes the container. + +Similar to the mutating case, what's needed here is [an accessor](#design-read) which _borrows_ `self` and which _lends_ `x`--this time immutably--to the caller. + +## Proposed solution + +We propose two new accessor kinds: +- `yielding mutate`, to enable mutating a value without first copying it +- `yielding borrow`, to enable inspecting a value without copying it. + +## Detailed design + +### `yielding borrow` + +[`UniqueGet`](#read-motivation) can now allow its clients to inspect its field non-destructively with `yielding borrow`: + +```swift +struct UniqueString : ~Copyable {...} + +struct UniqueBorrow : ~Copyable { + var x: UniqueString + + var property: UniqueString { + yielding borrow { + yield x + } + } +} +``` + +The `UniqueBorrow.property { yielding borrow }` accessor is a "yield-once coroutine". +When called, it borrows `self`, and then runs until reaching a `yield`, at which point it suspends, lending the yielded value back to the caller. +Once the caller is finished accessing the value, it resumes the accessor's execution. +The accessor continues running where it left off after the `yield`. + +If a a property or subscript provides a `yielding borrow`, it cannot also provide a `get`. + +#### `yielding borrow` as a protocol requirement + +Such accessors should be usable on values of generic and existential type. +To indicate that a protocol provides immutable access to a property or subscript via a `yielding borrow` coroutine, we propose allowing `yielding borrow` to appear where `get` does today: + +```swift +protocol Containing { + var property: UniqueString { yielding borrow } +} +``` + +A protocol requirement cannot specify both `yielding borrow` and `get`. + +A `yielding borrow` requirement can be witnessed by a stored property, a `yielding borrow` accessor, or a getter. + +#### `get` of noncopyable type as a protocol requirement + +Property and subscript requirements in protocols have up to this point only been able to express readability in terms of `get` requirements. +This becomes insufficient when the requirement has a noncopyable type: + +```swift +protocol Producing { + var property: UniqueString { get } +} +``` + +To fulfill such a requirement, the conformance must provide a getter, meaning its implementation must be able to produce a value whose ownership can be transferred to the caller. +In practical terms, this means that the requirement cannot be witnessed by a stored property or a `yielding borrow` accessor when the result is of noncopyable type,[^2] since the storage of a stored property is owned by the containing aggregate and the result of a `yielding borrow` is owned by the suspended coroutine, and it would be necessary to copy to provide ownership to the caller. +However, if the type of the `get` requirement is copyable, the compiler can synthesize the getter from the other accessor kinds by introducing copies as necessary. + +[^2]: While the compiler does currently accept such code, it does so by interpreting that `get` as a `yielding borrow`, which is a bug. + +### `yielding mutate` + +The `GetSet` type [above](#modify-motivation) could be implemented with `yielding mutate` as follows: + +```swift +struct GetMutate { + var x: String = "👋🏽 Hello" + + var property: String { + get { print("Getting", x); return x } + yielding mutate { + print("Yielding", x) + yield &x + print("Post yield", x) + } + } +} + +var getMutate = GetMutate() +getMutate.property.append(", 🌍!") +// prints: +// Yielding 👋🏽 Hello +// Post yield 👋🏽 Hello, 🌍! +``` + +Like `UniqueBorrow.property { yielding borrow }` above, `GetMutate.property { yielding mutate }` is a yield-once coroutine. +However, the `yielding mutate` accessor lends `x` to the caller _mutably_. + +Things to note about this example: +* `get` is never called — the property access is handled entirely by the `yielding mutate` accessor +* the `yield` is similar to a `return`, but control returns to the `yielding mutate` after the `append` completes +* there is no more `newValue` – the yielded value is modified by `append` +* because it's granting _mutable_ access to the caller, `yield` uses the `&` sigil, similar to passing an argument `inout` + +Unlike the `get`/`set` pair, the `yielding mutate` accessor is able to safely provide access to the yielded value without copying it. +This can be done safely because the accessor preserves ownership of the value until it has completely finished running: +When it yields the value, it only lends it to the caller. +The caller is exclusively borrowing the value yielded by the coroutine. + +`get` is still used when only fetching, not modifying, the property: + +```swift +_ = getMutate.property +// prints: +// Getting 👋🏽 Hello, 🌍! +``` + +`yielding mutate` is sufficient to allow assignment to a property: + +``` +getMutate.property = "Hi, 🌍, 'sup?" +// prints: +// Yielding 👋🏽 Hello, 🌍! +// Post yield Hi, 🌍, 'sup? +``` + +It is, however, also possible to supply _both_ a `yielding mutate` and a `set` accessor. +The `set` accessor will be favored in the case of a whole-value reassignment, which may be more efficient than preparing and yielding a value to be overwritten: + +```swift +struct GetSetMutate { + var x: String = "👋🏽 Hello" + + var property: String { + get { x } + yielding mutate { yield &x } + set { print("Setting",newValue); x = newValue } + } +} +var getSetMutate = GetSetMutate() +getSetMutate.property = "Hi 🌍, 'sup?" +// prints: +// Setting Hi 🌍, 'sup? +``` + +#### Pre- and post-processing in `yielding mutate` + +As with `set`, `yielding mutate` gives the property or subscript implementation an opportunity to perform some post-processing on the new value after assignment. Even in these cases, `yielding mutate` can often allow for a more efficient implementation than a traditional `get`/`set` pair would provide. +Consider the following implementation of an enhanced version of `Array.first` that allows the user to modify the first value of the array: + +```swift +extension Array { + var first: Element? { + get { isEmpty ? nil : self[0] } + yielding mutate { + var tmp: Optional + if isEmpty { + tmp = nil + yield &tmp + if let newValue = tmp { + self.append(newValue) + } + } else { + tmp = self[0] + yield &tmp + if let newValue = tmp { + self[0] = newValue + } else { + self.removeFirst() + } + } + } + } +} +``` + +This implementation takes a similar approach to `Swift.Dictionary`'s key-based subscript: +if there is no `first` element, the accessor appends one. +If `nil` is assigned, the element is removed. +Otherwise, the element is updated to the value from the assigned `Optional`'s payload. + +Because the fetch and update code are all contained in one block, the `isEmpty` check is not duplicated; if the same logic were implemented with a `get`/`set` pair, `isEmpty` would need to be checked independently in both accessors. With `yielding mutate`, whether the array was initially empty is part of the accessor's state, which remains present until the accessor is resumed and completed. + +#### Taking advantage of exclusive access + +The optional return value of `first` in the code above means that, despite using `yielding mutate`, we have reintroduced the problem of triggering copy-on-write when mutating through the `first` property. +That act of placing the value in the array inside of an `Optional` (namely, `tmp = self[0]`) creates a copy. + +Like a `set` accessor, a `yielding mutate` accessor in a `struct` or `enum` property is `mutating` by default (unless explicitly declared a `nonmutating yielding mutate`). +This means that the `yielding mutate` accessor has exclusive access to `self`, and that exclusive access extends for the duration of the accessor's execution, including its suspension after `yield`-ing. +Since the implementation of `Array.first { yielding mutate }` has exclusive access to its underlying buffer, it can move the value of `self[0]` directly into the `Optional`, yield it, and then move the result back after being resumed: + +```swift +extension Array { + var first: Element? { + yielding mutate { + var tmp: Optional + if isEmpty { + // Unchanged + } else { + // Illustrative code only, Array's real internals are fiddlier. + // _storage is an UnsafeMutablePointer to the Array's storage. + + // Move first element in _storage into a temporary, leaving that slot + // in the storage buffer as uninintialized memory. + tmp = _storage.move() + + // Yield that moved value to the caller + yield &tmp + + // Once the caller returns, restore the array to a valid state + if let newValue = tmp { + // Re-initialize the storage slot with the modified value + _storage.initialize(to: newValue) + } else { + // Element removed. Slide other elements down on top of the + // uninitialized first slot: + _storage.moveInitialize(from: _storage + 1, count: self.count - 1) + self.count -= 1 + } + } + } +} +``` + +While the `yielding mutate` coroutine is suspended after yielding, the `Array` is left in an invalid state: the memory location where the first element is stored is left uninitialized, and must not be accessed. +However, this is safe thanks to Swift's rules preventing conflicting access to memory. +Unlike a `get`, the `yielding mutate` is guaranteed to have an opportunity to put the element back (or to remove the invalid memory if the entry is set to `nil`) after the caller resumes it, restoring the array to a valid state in all circumstances before any other code can access it. + +### The "yield once" rule + +Notice that there are _two_ yields in this `Array.first { yielding mutate }` implementation, for the empty and non-empty branches. +Nonetheless, exactly one `yield` is executed on any path through the accessor. +In general, the rules for yields in yield-once coroutines are similar to those of deferred initialization of `let` variables: +it must be possible for the compiler to guarantee there is exactly one yield on every path. +In other words, there must not be a path through the yield-once coroutine's body with either zero[^1] or more than one yield. +The `Array.first` example is valid since there is a `yield` in both the `if` and the `else` branch. +In cases where the compiler cannot statically guarantee this, refactoring or use of `fatalError()` to assert unreachable code paths may be necessary. + +[^1]: Note that it is legal for a path without any yields to terminate in a `fatalError`. Such a path is not _through_ the function. + +### Throwing callers + +The `Array.first { yielding mutate }` implementation above is correct even if the caller throws while the coroutine is suspended. + +```swift +try? myArray.first?.throwingMutatingOp() +``` + +Thanks to Swift's rule that `inout` arguments be initialized at function exit, the element must be a valid value when `throwingMutatingOp` throws. +When `throwingMutatingOp` does throw, control returns back to the caller. +The body of `Array.first { yielding mutate }` is resumed, and `tmp` is a valid value. +Then the code after the `yield` executes. +The coroutine cleans up as usual, writing the updated temporary value in `tmp` back into the storage buffer. + +## Source compatibility + +The following code is legal today: + +```swift +func borrow(_ c : () -> T) -> T { c() } +var reader : Int { + borrow { + fatalError() + } +} +``` + +Currently, the code declares a property `reader` with an implicit getter. +The implicit getter has an implicit return. +The expression implicitly returned is a call to the function `borrow` with a trailing closure. + +An analogous situation exists for `mutate`. + +This proposal takes the identifiers `borrow` and `mutate` as contextual keywords +as part of the `yielding borrow` and `yielding mutate` accessor declarations. +By themselves, these two new accessor forms do not immediately require a source break; however, as the [accessors vision](https://github.com/rjmccall/swift-evolution/blob/accessors-vision/visions/accessors.md) lays out, we will more than likely want to introduce non-`yielding` variants of `borrow` and `mutate` in the near future as well. + +Therefore, we propose an alternate interpretation for this code: +that it declare a property `reader` with a `borrow` accessor. +Although such an accessor does not yet exist, making the syntax change now will prepare the language for subsequent accessors we introduce in the near future. + +If this presents an unacceptable source compatibility break, the change may have to be gated on a language version. + +## ABI compatibility + +Adding a `yielding mutate` accessor to an existing read-only subscript or computed property has the same ABI implications as adding a setter; it must be guarded by availability on ABI-stable platforms. +The stable ABI for mutable properties and subscripts provides `get`, `set`, and `_modify` coroutines, deriving `set` from `_modify` or vice versa if one is not explicitly implemented. +Therefore, adding or removing `yielding mutate`, adding or removing `set`, replacing a `set` with a `yielding mutate`, or replacing `yielding mutate` with `set` are all ABI-compatible changes, so long as the property or subscript keeps at least one of `set` or `yielding mutate`, and those accessors agree on whether they are `mutating` or `nonmutating` on `self`. + +By contrast, `yielding borrow` and `get` are mutually exclusive. Replacing one with the other is always an ABI-breaking change. +Replacing `get` with `yielding borrow` is also an API-breaking change for properties or subscripts of noncopyable type. + +Renaming the current `_modify` (as used by the standard library, e.g.) to `yielding mutate` is an ABI additive change: a new `yielding mutate` symbol will be added. +When the compiler sees a `yielding mutate` with an early enough availability, the compiler will synthesize a corresponding `_modify` whose body will just forward to `yielding mutate`. +This is required for ABI stability: code compiled against an older standard library which calls `_modify` will continue to do so. +Meanwhile, code compiled against a newer standard library will call the new `yielding mutate`. +The same applies to renaming `_read` to `yielding borrow`. + +## Implications on adoption + +### Runtime support + +The new ABI will require runtime support which would need to be back deployed in order to be used on older deployment targets. + +### When to favor coroutines over `get` and `set` + +Developers will need to understand when to take advantage of `yielding` accessors, and we should provide some guidance: + +#### `yielding mutate` + +`yielding mutate` is desirable when the value of a property or subscript element is likely to be subject to frequent in-place modifications, especially when the value's type uses copy-on-write and the temporary value created by a `get`-update-`set` sequence is likely to trigger full copies of the buffer that could otherwise be avoided. +Additionally, if the mutation operation involves complex setup and teardown that requires persisting state across the operation, `yielding mutate` allows for arbitrary state to be preserved across the access. + +For noncopyable types, implementing `yielding mutate` is necessary to provide in-place mutation support if the value of the property or subscript cannot be read using a `get`. + +Even in cases where `yielding mutate` is beneficial, it is often still profitable to also provide a `set` accessor for whole-value reassignment. `set` may be able to avoid setup necessary to support an arbitrary in-place mutation and can execute as a standard function call, which may be more efficient than a coroutine. + +#### `yielding borrow` + +For read-only operations on `Copyable` types, the performance tradeoffs between `get` and `yielding borrow` are more subtle. +`borrowing yield` can avoid copying a result value that is stored as part of the originating value. +On the other hand, `get` can execute as a normal function without the overhead of a coroutine. +(As part of the implementation work for this proposal, we are implementing a more efficient coroutine ABI that should be significantly faster than the implementation used by `_read` and `_modify` today; however, a plain function call is likely to remain somewhat faster.) +`yielding borrow` may nonetheless be preferable for providing access to large value types in memory that would be expensive to copy as part of returning from a `get`. + +`yielding borrow` also introduces the ability to perform both setup work before and teardown work after the caller has accessed the value, whereas `get` can at best perform setup work before returning control to the caller. +Particularly in conjunction with non-`Escapable` types, which could eventually be lifetime-bound to their access, a `yielding borrow` accessor producing a non-`Escapable` value could provide limited access to a resource in a similar manner to `withSomething { }` closure-based APIs, without the "pyramid of doom" or scoping limitations of closure-based APIs. + +For properties and subscripts with noncopyable types, the choice between `yielding borrow` and `get` will often be forced by whether the API contract specifies a borrowed or owned result. +An operation that provides temporary borrowed access to a component of an existing value without transferring ownership, when that value is expected to persist after the access is complete, must be implemented using `yielding borrow`. +An operation that computes a new value every time is best implemented using `get`. +(Such an operation can in principle also be implemented using `yielding borrow`; it would compute the temporary value, yield a borrow to the caller, and then destroy the temporary value when the coroutine is resumed. +This would be inefficient, and would furthermore prevent the caller from being able to perform consuming operations on the result.) + +For polymorphic interfaces such as protocol requirements and overridable class members that are properties or subscripts of noncopyable type, the most general requirement is `yielding borrow`, since a `yielding borrow` requirement can be satisfied by either a `yielding borrow` or a `get` implementation (in the latter case, by wrapping the getter in a synthesized `yielding borrow` that invokes the getter, yields a borrow the result, then destroys the value on resume). +Using `yielding borrow` in polymorphic or ABI-stable interfaces may thus desirable to allow for maximum evolution flexibility if being able to consume the result is never expected to be part of the API contract. + +## Future directions + +### Yield-once functions + +Further ergonomic enhancements to the language may be needed over time to make the most of this feature. +For example, coroutine accessors do not compose well with functions because functions cannot themselves currently yield values. +In the future, it may be desirable to also support yield-once functions: + +```swift +var value: C { yielding mutate { ... } } +yielding func updateValue(...) -> inout C { + yield &self.value + additionalWork(value) +} +``` + +### Forwarding both consuming and borrowing accesses to the same declaration + +When a property or subscript has a `consuming get`, a caller can take ownership of the field at the expense of also destroying the rest of object. +When a property or subscript has a `yielding borrow` accessor, a caller can borrow the field to inspect it without taking ownership of it. + +As proposed here, it's not yet possible for a single field to provide both of these behaviors to different callers. +Since both of these behaviors have their uses, and many interfaces want to provide "perfect forwarding" where any of consuming, mutating, or borrowing operations on a wrapper can be performed by passing through the same operation to an underlying value, it may be desirable in the future to allow a single field to provide both a `yielding borrow` and a `consuming get`: + +```swift +subscript(index: Int) -> Value { + consuming get {...} + yielding borrow {...} +} +``` + +The rules that Swift currently uses to determine what accessor(s) to use in order to evaluate an expression are currently only defined in terms of whether the access is mutable or not, so these rules would need further elaboration to distinguish non-mutating accesses by whether they are consuming or borrowing in order to determine which accessor to favor. + +### Transitioning between owned and borrowed accessors for API evolution + +When a noncopyable API first comes into existence, its authors may not want to commit to it producing an owned value. As discussed [above](#read-implications), providing a `yielding borrow` accessor is often the most conservative API choice for noncopyable properties and subscripts: + +```swift +subscript(index: Int) -> Value { + yielding borrow {...} +} +``` + +As their module matures, however, the authors may decide that committing to providing a consumable value is worthwhile. +To support this use-case, in the future, it may be desirable to allow for forward-compatible API and ABI evolution by promoting a `yielding borrow` accessor to `get`: + +```swift +subscript(index: Int) -> Value { + // Deprecate the `yielding borrow` + @available(*, deprecated) + yielding borrow {...} + + // Favor the `get` for new code + get {...} +} +``` + + +That would enable the module to evolve to a greater commitment while preserving ABI. + +For APIs of copyable type, the reverse evolution is also a possibility; an API could originally be published using a `get` and later decide that a `yielding borrow` accessor is overall more efficient. + +Since it would typically be ambiguous whether the `yielding borrow` or `get` should be favored at a use site, it would make sense to require that one or the other accessor be deprecated or have earlier availability than the other in order to show that one is unambiguously preferred over the other for newly compiled code with recent enough availability. + +### Non-`yielding` `borrow` and `mutate` accessors + +A `yielding` accessor lends the value it yields to its caller. +The caller only has access to that value until it resumes the coroutine. +After the coroutine is resumed, it has the opportunity to clean up. +This enables a `yielding borrow` or `mutate` to do interesting work, such as constructing a temporary aggregate from its base object's fields: + +```swift +struct Pair : ~Copyable { + var left: Left + var right: Right + + var reversed: Pair { + yielding mutate { + let result = Pair(left: right, right: left) + yield &result + self = .init(left: result.right, right: result.left) + } + } +} +``` + +That the access ends when the coroutine is resumed means that the lifetime of the lent value is strictly shorter than that of the base value. +In the example above, the lifetime of `reversed` is shorter than that of the `Pair` it is called on. + +As discussed in the [accessors vision](https://github.com/rjmccall/swift-evolution/blob/accessors-vision/visions/accessors.md), when a value is merely being projected from the base object and does not need any cleanup after being accessed, this is undesirably limiting: +a value projected from a base naturally has _the same_ lifetime as the base. + +This is especially problematic in the context of composition with non-`Escapable` types. +Consider the following wrapper type[^3]: + +[^3]: This example involves writing out a `yielding borrow` accessor. The same issue exists when the compiler synthesizes a `yielding borrow` accessor for a stored property exported from a resilient module. + +```swift +struct Wrapper : ~Copyable & ~Escapable { + var _stuffing: Stuffing + + var stuffing: Stuffing { + yielding borrow { + yield _stuffing + } + } +} +``` + +When the instance of `Wrapper` is local to a function, the strict nesting of lifetimes may not immediately be a problem: + +```swift +{ + let wrapper: Wrapper = ... + borrowStuffing(wrapper.stuffing) + // lifetime of wrapper.stuffing ends (at coroutine resumption) + // lifetime of wrapper ends +} +``` + +But when `Wrapper` is a parameter, or otherwise nonlocal to the function, it is natural to expect to return the `Stuffing` back to the source of the `Wrapper`, but the `yielding borrow` accessor artificially limits its lifetime: + +```swift +@lifetime(borrow wrapper) +func getStuffing(from wrapper: borrowing Wrapper) -> Stuffing { + return wrapper.stuffing // error +} +``` + +The issue is that the lifetime of `stuffing` ends _within_ `getStuffing`, when the `yielding borrow` coroutine is resumed, which prevents `stuffing` from being returned. + +To address use cases like this, in the future, it may be desirable to introduce a variant of `borrow` and `mutate` accessors that immediately `return`s the borrowed or mutable value without involving a coroutine: + +```swift +var stuffing: Stuffing { + borrow { + return _stuffing + } +} +``` + +A non-`yielding` `borrow` or `mutate` accessor would be limited to returning values that can be accessed immediately and which do not require any cleanup (implicit or explicit) when the access ends. As such, the `yielding` variants would still provide the most flexibility in implementation, at the cost of the additional lifetime constraint of the coroutine. Similar to the [discussion around evolving `yielding borrow` accessors into `get` accessors](#ownership-evolution), there is also likely to be a need to allow for APIs initially published in terms of `yielding` accessors to transition to their corresponding non-`yielding` accessors in order to trade stronger implementation guarantees for more flexibility in the lifetime of derived `~Escapable` values. + +## Alternatives considered + +### Unwinding the accessor when an error is thrown in the caller + +A previous version of this proposal specified that if an error is thrown in a coroutine caller while a coroutine is suspended, the coroutine is to "unwind" and the code after the `yield` is not to run. +In the [example above](#throwing-callers), the code after the `yield` would not run if `throwingMutatingOp` threw an error. + +This approach was tied up with the idea that a `yielding mutate` accessor might clean up differently if an error was thrown in the caller. +The intervening years of experience with the feature have not borne out the need for this. +If an error is thrown in a caller into which a value has been yielded, the _caller_ must put the yielded mutable value back into a consistent state. +As with `inout` function arguments, the compiler enforces this: +it is an error to consume the value yielded from a `yielding mutate` accessor without reinitializing it before resuming the `yielding mutate` accessor. +When there are higher-level invariants which the value being modified must satisfy, in general, only the caller will be in a position to ensure that they are satisfied on the throwing path. + +Once that basis has been removed, there is no longer a reason to enable a coroutine to "unwind" when an error was thrown in the caller. +It should always finish execution the same way. + +### Naming scheme + +These coroutine accessors have been a long-standing "unofficial" feature of the Swift compiler under the names `_read` and `_modify`. +Previous revisions of this proposal merely dropped the underscore and proposed the coroutine accessors be provided under the name `read` and `modify`. +However, as we have continued to build out Swift's support for ownership and lifetime dependencies with `~Copyable` and `~Escapable` types, we have since identified the need for non-coroutine accessors that can produce borrowed and/or mutable values without imposing a lifetime dependency on a coroutine access. + +In order to avoid a proliferation of unrelated-seeming accessor names, this revision of the proposal uses the name `yielding borrow` instead of `read` and `yielding mutate` instead of `modify`. +We feel these names better connect the accessors to what ownership of the result is given: +a `borrow` accessor gives `borrowing` access to its result, and a `mutate` accessor gives `mutating` (in other words, `inout`) access. +Using `yielding` as an added modifier relates these accessors to potential non-coroutine variants of the accessors that could exist in the future; a `borrow` accessor (without `yielding`) would in the future be an accessor that returns a borrow without involving a coroutine. +The same `yielding` modifier could also be used in other places in the future, such as `func` declarations, to allow for yield-once coroutines to be defined as regular functions outside of accessors. + +## Acknowledgments + +John McCall and Arnold Schwaighofer provided much of the original implementation of accessor coroutines. Tim Kientzle and John McCall authored the accessors vision document this proposal serves as part of the implementation of. + diff --git a/proposals/0475-observed.md b/proposals/0475-observed.md new file mode 100644 index 0000000000..7690394895 --- /dev/null +++ b/proposals/0475-observed.md @@ -0,0 +1,592 @@ +# Transactional Observation of Values + +* Proposal: [SE-0475](0475-observed.md) +* Authors: [Philippe Hausler](https://github.com/phausler) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Accepted** +* Implementation: https://github.com/swiftlang/swift/pull/79817 +* Review: ([pitch](https://forums.swift.org/t/pitch-transactional-observation-of-values/78315)) ([review](https://forums.swift.org/t/se-0475-transactional-observation-of-values/79224)) ([acceptance](https://forums.swift.org/t/accepted-se-0475-transactional-observation-of-values/80389)) + +## Introduction + +Observation was introduced to add the ability to observe changes in graphs of +objects. The initial tools for observation afforded seamless integration into +SwiftUI, however aiding SwiftUI is not the only intent of the module - it is +more general than that. This proposal describes a new safe, ergonomic and +composable way to observe changes to models using an AsyncSequence, starting +transactions at the first willSet and then emitting a value upon that +transaction end at the first point of consistency by interoperating with +Swift Concurrency. + +## Motivation + +Observation was designed to allow future support for providing an `AsyncSequence` +of values, as described in the initial [Observability proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0395-observability.md). +This follow-up proposal offers tools for enabling asynchronous sequences of +values, allowing non-SwiftUI systems to have the same level of "just-the-right-amount-of-magic" +as when using SwiftUI. + +Numerous frameworks in the Darwin SDKs provide APIs for accessing an +`AsyncSequence` of values emitted from changes to a property on a given model +type. For example, DockKit provides `trackingStates` and Group Activities +provides `localParticipantStates`. These are much like other APIs that provide +`AsyncSequence` from a model type; they hand crafted to provide events from when +that object changes. These manual implementations are not trivial and require +careful book-keeping to get right. In addition, library and application code +faces the same burden to use this pattern for observing changes. Each of these +uses would benefit from having a centralized and easy mechanism to implement +this kind of sequence. + +Observation was built to let developers avoid the complexity inherent when +making sure the UI is updated upon value changes. For developers using SwiftUI +and the `@Observable` macro to mark their types, this principle is already +realized; directly using values over time should mirror this ease of use, +providing the same level of power and flexibility. That model of tracking changes +by a graph allows for perhaps the most compelling part of Observation; it +can track changes by utilizing naturally written Swift code that is written just +like the logic of other plain functions. In practice that means that any solution +will also follow that same concept even for disjoint graphs that do not share +connections. The solution will allow for iterating changed values for applications +that do not use UI as seamlessly as those that do. + +## Proposed solution + +This proposal adds a straightforward new tool: a closure-initialized `Observations` +type that acts as a sequence of closure-returned values, emitting new values +when something within that closure changes. + +This new type makes it easy to write asynchronous sequences to track changes +but also ensures that access is safe with respect to concurrency. + +The simple `Person` type declared here will be used for examples in the +remainder of this proposal: + +```swift +@Observable +final class Person { + var firstName: String + var lastName: String + + var name: String { firstName + " " + lastName } + + init(firstName: String, lastName: String) { + self.firstName = firstName + self.lastName = lastName + } +} +``` + +Creating an `Observations` asynchronous sequence is straightforward. This example +creates an asynchronous sequence that yields a value every time the composed +`name` property is updated: + +```swift +let names = Observations { person.name } +``` + +However if the example was more complex and the `Person` type in the previous +example had a `var pet: Pet?` property which was also `@Observable` then the +closure can be written with a more complex expression. + +```swift +let greetings = Observations { + if let pet = person.pet { + return "Hello \(person.name) and \(pet.name)" + } else { + return "Hello \(person.name)" + } +} +``` + +In that example it would track both the assignment of a new pet and then consequently +that pet's name. + +## Detailed design + +There are a few behaviors that are prerequisites to understanding the requirements +of the actual design. These two key behaviors are how the model handles tearing +and how the model handles sharing. + +Tearing is where a value that is expected to be assigned as a singular +transactional operation can potentially be observed in an intermediate and +inconsistent state. The example `Person` type shows this when a `firstName` is +set and then the `lastName` is set. If the observation was triggered just on the +trailing edge (the `didSet` operation) then an assignment to both properties +would garner an event for both properties and potentially get an inconsistent +value emitted from `name`. Swift has a mechanism for expressing the grouping of +changes together: isolation. When an actor or an isolated type is modified it is +expected (enforced by the language itself) to be in a consistent state at the +next suspension point. This means that if we can utilize the isolation that is +safe for the type then the suspensions on that isolation should result in safe +(and non torn values). This means that the implementation must be transactional +upon that suspension; starting the transaction on the first trigger of a leading +edge (the `willSet`) and then completing the transaction on the next suspension +of that isolation. + +The simple example of tearing would work as the following: + +```swift +let person = Person(firstName: "", lastName: "") +// willSet \.firstName - start a transaction +person.firstName = "Jane" +// didSet \.firstName +// willSet \.lastName - the transaction is still dirty +person.lastName = "Appleseed" +// didSet \.lastName +// the next suspension the `name` property will be valid +``` + +Suspensions are any point where a task can be calling out to something where +they `await`. Swift concurrency enforces safety around these by making sure that +isolation is respected. Any time a function has a suspension point data +associated with the type must be ready to be read by the definitions of actor +isolation. In the previous example of the `Person` instance the `firstName` and +`lastName` properties are mutated together in the same isolation, that means +that no other access in that isolation can read those values when they are torn +without the type being `Sendable` (able to be read from multiple isolations). +That means that in the case of a non-`Sendable` type the access must be +constrained to an isolation, and in the `Sendable` cases the mutation is guarded +by some sort of mechanism like a lock, In either case it means that the next +time one can read a safe value is on that same isolation of the safe access to +start with and that happens on that isolations next suspension. + +Observing at the next suspension point means that we can also address the second +issue too; sharing. The expectation of observing a property from a type as an +AsyncSequence is that multiple iterations of the same sequence from multiple +tasks will emit the same values at the same iteration points. The following code +is expected to emit the same values in both tasks. + +```swift + +let names = Observations { person.firstName + " " + person.lastName } + +Task.detached { + for await name in names { + print("Task1: \(name)") + } +} + +Task.detached { + for await name in names { + print("Task2: \(name)") + } +} +``` + +In this case both tasks will get consistently safe accessed values. This can +be achieved without needing an extra buffer since the suspension of each side of +the iteration are continuations resuming all together upon the accessor's +execution on the specified isolation. This facilitates subject-like behavior +such that the values are sent from the isolation for access to the iteration's +continuation. + +The previous initialization using the closure is a sequence of values of the computed +properties as a `String`. This has no sense of termination locally within the +construction. Making the return value of that closure be a lifted `Optional` suffers +the potential conflation of a terminal value and a value that just happens to be nil. +This means that there is a need for a second construction mechanism that offers a +way of expressing that the `Observations` sequence iteration will run until finished. + +For the example if `Person` then has a new optional field of `homePage` which +is an optional URL it then means that the construction can disambiguate +by returning the iteration as the `next` value or the `finished` value. + +``` +@Observable +final class Person { + var firstName: String + var lastName: String + var homePage: URL? + + var name: String { firstName + " " + lastName } + + init(firstName: String, lastName: String) { + self.firstName = firstName + self.lastName = lastName + } +} + +let hosts = Observations.untilFinished { [weak person] in + if let person { + .next(person.homePage?.host) + } else { + .finished + } +} +``` + +Putting this together grants a signature as such: + +```swift +public struct Observations: AsyncSequence, Sendable { + public init( + @_inheritActorContext _ emit: @escaping @isolated(any) @Sendable () throws(Failure) -> Element + ) + + public enum Iteration: Sendable { + case next(Element) + case finished + } + + public static func untilFinished( + @_inheritActorContext _ emit: @escaping @isolated(any) @Sendable () throws(Failure) -> Iteration + ) -> Observations +} +``` + +Picking the initializer apart first captures the current isolation of the +creation of the `Observations` instance. Then it captures a `Sendable` closure that +inherits that current isolation. This means that the closure may only execute on +the captured isolation. That closure is run to determine which properties are +accessed by using Observation's `withObservationTracking`. So any access to a +tracked property of an `@Observable` type will compose for the determination of +which properties to track. + +The closure is not run immediately it is run asynchronously upon the first call +to the iterator's `next` method. This establishes the first tracking state for +Observation by invoking the closure inside a `withObservationTracking` on the +implicitly specified isolation. Then upon the first `willSet` it will enqueue on +to the isolation a new execution of the closure and finishing the transaction to +prime for the next call to the iterator's `next` method. + +The closure has two other features that are important for common usage; firstly +the closure is typed-throws such that any access to that emission closure will +potentially throw an error if the developer specifies. This allows for complex +composition of potentially failable systems. Any thrown error will mean that the +`Observations` sequence is complete and loops that are currently iterating will +terminate with that given failure. Subsequent calls then to `next` on those +iterators will return `nil` - indicating that the iteration is complete. + +The type `Observations` will conform to `AsyncSequence`. This means that it +adheres to the cancellation behavior of other `AsyncSequence` types; if the task +is cancelled then the iterator will return nil, and any time it becomes +terminal for any reason that sequence will remain terminal and continue returning nil. +Termination by cancellation however is independent for each instance. + +## Behavioral Notes + +There are a number of scenarios of iteration that can occur. These can range from production rate to iteration rate differentials to isolation differentials to concurrent iterations. Enumerating all possible combinations is of course not possible but the following explanations should illustrate some key usages. `Observations` does not make unsafe code somehow safe - the concepts of isolation protection or exclusive access are expected to be brought to the table by the types involved. It does however require the enforcements via Swift Concurrency particularly around the marking of the types and closures being required to be `Sendable`. The following examples will only illustrate well behaved types and avoid fully unsafe behavior that would lead to crashes because the types being used are circumventing that language safety. + +The most trivial case is where a single produce and single consumer are active. In this case they both are isolated to the same isolation domain. For ease of reading; this example is limited to the `@MainActor` but could just as accurately be represented in some other actor isolation. + +```swift +@MainActor +func iterate(_ names: Observations) async { + for await name in names { + print(name) + } +} + +@MainActor +func example() async throws { + let person = Person(firstName: "", lastName: "") + + // note #2 + let names = Observations { + person.name + } + + Task { + await iterate(names) + } + + for i in 0..<5 { + person.firstName = "\(i)" + person.lastName = "\(i)" + try await Task.sleep(for: .seconds(0.1)) // note #1 + } +} + +try await example() + +``` + +The result of the observation will print the following output. + +``` +0 0 +1 1 +2 2 +3 3 +4 4 +``` + +The values are by the virtue of the suspension at `note #1` are all emitted, the first name and last name are conjoined because they are both mutated before the suspension. The type `Person` does not need to be `Sendable` because `note #2` is implicitly picking up the `@MainActor` isolation of the enclosing isolation context. That isolation means that the person is always safe to access in that scope. + +Next is the case where the mutation of the properties out-paces the iteration. Again the example is isolated to the same domain. + +```swift +@MainActor +func iterate(_ names: Observations) async { + for await name in names { + print(name) + try? await Task.sleep(for: .seconds(0.095)) + } +} + +@MainActor +func example() async throws { + let person = Person(firstName: "", lastName: "") + + // @MainActor is captured here as the isolation + let names = Observations { + person.name + } + + Task { + await iterate(names) + } + + for i in 0..<5 { + person.firstName = "\(i)" + person.lastName = "\(i)" + try await Task.sleep(for: .seconds(0.1)) + } +} + +try await example() + +``` + +The result of the observation may print the following output, but the primary property is that the values are conjoined to the same consistent view. It is expected that some values may not be represented during the iteration because the transaction has not yet been handled by the iteration. + +``` +0 0 +1 1 +2 2 +3 3 +``` + +The last value is never observed because the program ends before it would be. If the program did not terminate then another value would be observed. + +Observations can be used across boundaries of concurrency. This is where the iteration is done on a different isolation than the mutations. The types however are accessed always in the isolation that the creation of the Observations closure is executed. This means that if the `Observations` instance is created on the main actor then the subsequent calls to the closure will be done on the main actor. + +```swift +@globalActor +actor ExcplicitlyAnotherActor: GlobalActor { + static let shared = ExcplicitlyAnotherActor() +} + +@ExcplicitlyAnotherActor +func iterate(_ names: Observations) async { + for await name in names { + print(name) + } +} + +@MainActor +func example() async throws { + let person = Person(firstName: "", lastName: "") + + // @MainActor is captured here as the isolation + let names = Observations { + person.name + } + + Task.detached { + await iterate(names) + } + + for i in 0..<5 { + person.firstName = "\(i)" + person.lastName = "\(i)" + try await Task.sleep(for: .seconds(0.1)) + } +} + +``` + +The values still will be conjoined as expected for their changes, however just like the out-paced case there is a potential in which an alteration may slip between the isolations and only a subsequent value is represented during the iteration. However since is particular example has no lengthy execution (greater than 0.1 seconds) it means that it does not get out paced by production and returns all values. + +``` +0 0 +1 1 +2 2 +3 3 +4 4 +``` + +If the `iterate` function was altered to have a similar `sleep` call that exceeded the production then it would result in similar behavior of the previous producer/consumer rate case. + +The next behavioral illustration is the value distribution behaviors; this is where two or more copies of an `Observations` are iterated concurrently. + +```swift + +@MainActor +func iterate1(_ names: Observations) async { + for await name in names { + print("A", name) + } +} + + +@MainActor +func iterate2(_ names: Observations) async { + for await name in names { + print("B", name) + } +} + +@MainActor +func example() async throws { + let person = Person(firstName: "", lastName: "") + + // @MainActor is captured here as the isolation + let names = Observations { + person.name + } + + Task.detached { + await iterate1(names) + } + + Task.detached { + await iterate2(names) + } + + for i in 0..<5 { + person.firstName = "\(i)" + person.lastName = "\(i)" + try await Task.sleep(for: .seconds(0.1)) + } +} + +try await example() +``` + +This situation commonly comes up when the asynchronous sequence is stored as a property of a type. By vending these as a shared instance to a singular source of truth it can provide both a consistent view and reduce overhead for design considerations. However when the sequences are then combined with other isolations the previous caveats come in to play. + +``` +A 0 0 +B 0 0 +B 1 1 +A 1 1 +A 2 2 +B 2 2 +A 3 3 +B 3 3 +B 4 4 +A 4 4 +``` + +The same rate commentary applies here as before but an additional wrinkle is that the delivery between the A and B sides is non-determinstic (in some cases it can deliver as A then B and other cases B then A). + +There is one additional clarification of expected behaviors - the iterators should have an initial state to determine if that specific iterator is active yet or not. This means that upon the first call to next the value will be obtained by calling into the isolation of the constructing closure to "prime the pump" for observation and obtain a first value. This can be encapsulated into an exaggerated test example as the following: + +```swift + +@MainActor +func example() async { + let person = Person(firstName: "0", lastName: "0") + + // @MainActor is captured here as the isolation + let names = Observations { + person.name + } + Task { + try await Task.sleep(for: .seconds(2)) + person.firstName = "1" + person.lastName = "1" + + } + Task { + for await name in names { + print("A = \(name)") + } + } + Task { + for await name in names { + print("B = \(name)") + } + } + try? await Task.sleep(for: .seconds(10)) +} + +await example() +``` + +Which results in the following output: + +``` +A = 0 0 +B = 0 0 +B = 1 1 +A = 1 1 +``` + +This ensures the first value is produced such that every sequence will always be primed with a value and will eventually come to a mutual consistency to the values no matter the isolation. + +## Effect on ABI stability & API resilience + +This provides no alteration to existing APIs and is purely additive. However it +does have a few points of interest about future source compatibility; namely +the initializer does ferry the inherited actor context as a parameter and if +in the future Swift develops a mechanism to infer this without a user +overridable parameter then there may be a source breaking ambiguity that would +need to be disambiguated. + +## Notes to API authors + +This proposal does not change the fact that the spectrum of APIs may range from +favoring `AsyncSequence` properties to purely `@Observable` models. They both +have their place. However the calculus of determining the best exposition may +be slightly more refined now with `Observations`. + +If a type is representative of a model and is either transactional in that +some properties may be linked in their meaning and would be a mistake to read +in a disjoint manner (the tearing example from previous sections), or if the +model interacts with UI systems it now more so than ever makes sense to use +`@Observable` especially with `Observations` now as an option. Some cases may have +previously favored exposing those `AsyncSequence` properties and would now +instead favor allowing the users of those APIs compose things by using `Observations`. +The other side of the spectrum will still exist but now is more strongly +relegated to types that have independent value streams that are more accurately +described as `AsyncSequence` types being exposed. The suggestion for API authors +is that now with `Observations` favoring `@Observable` perhaps should take more +of a consideration than it previously did. + +## Alternatives Considered + +Both initialization mechanisms could potentially be collapsed into an optional, +however that creates potential ambiguity of valid nil elements versus termination. + +There have been many iterations of this feature so far but these are some of the +highlights of alternative mechanisms that were considered. + +Just expose a closure with `didSet`: This misses the mark with regards to concurrency +safety but also faces a large problem with regards to transactionality. This would also +be out sync with the expected behavior of existing observation uses like SwiftUI. +The one benefit of that approach is that each setter call would have a corresponding +callback and would be more simple to implement with the existing infrastructure. It +was ultimately rejected because that would fall prey to the issue of tearing and +the general form of composition was not as ergonomic as other solutions. + +Expose an AsyncSequence based on `didSet`: This also falls to the same issues with the +closure approach except is perhaps slightly more ergonomic to compose. This was also +rejected due to the tearing problem stated in the proposal. + +Expose an AsyncSequence property extension based on `KeyPath`: This could be adapted +to the `willSet` and perhaps transactional models, but faces problems when attempting +to use `KeyPath` across concurrency domains (since by default they are not Sendable). +The implementation of that approach would require considerable improvement to handling +of `KeyPath` and concurrency (which may be an optimization path that could be considered +in the future if the API merits it). As it stands however the `KeyPath` approach in +comparison to the closure initializer is considerably less easy to compose. + +The closure type passed to the initializer does not absolutely require @Sendable in the +cases where the initialization occurs in an isolated context, if the initializer had a +parameter of an isolation that was non-nullable this could be achieved for that restriction +however up-coming changes to Swift's Concurrency will make this approach less appealing. +If this route would be taken it would restrict the potential advanced uses cases where +the construction would be in an explicitly non-isolated context. + +A name of `Observed` was considered, however that type name led to some objections that +rightfully claimed it was a bit odd as a name since it is bending the "nouning" of names +pretty strongly. This lead to the alternate name `Observations` which strongly leans +into the plurality of the name indicating that it is more than one observation - lending +to the sequence nature. + +It was seriously considered during the feedback to remove the initializer methods and only +have construction by two global functions named `observe` and `observeUntilFinished` +that would act as the current initializer methods. Since the types must still be returned +to allow for storing that return into a property it does not offer a distinct advantage. diff --git a/proposals/0476-abi-attr.md b/proposals/0476-abi-attr.md new file mode 100644 index 0000000000..22caeee1a2 --- /dev/null +++ b/proposals/0476-abi-attr.md @@ -0,0 +1,945 @@ +# Controlling the ABI of a function, initializer, property, or subscript + +* Proposal: [SE-0476](0476-abi-attr.md) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 6.2)** +* Review: ([pitch](https://forums.swift.org/t/pitch-controlling-the-abi-of-a-declaration/75123)) ([review](https://forums.swift.org/t/se-0476-controlling-the-abi-of-a-function-initializer-property-or-subscript/79233)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0476-controlling-the-abi-of-a-function-initializer-property-or-subscript/79644)) + +## Introduction + +We propose introducing the `@abi` attribute, which provides an alternate +version of the declaration used for name mangling. This feature would allow +developers of ABI-stable libraries to make minor changes, such as changing +the sendability of a parameter or renaming a declaration (so long as source +compatibility is preserved in a backwards-deployable way), without requiring +deep knowledge of compiler implementation details. + +## Motivation + +Maintainers of ABI-stable libraries sometimes need to update or correct +existing declarations for various reasons: + +1. To adopt new language features, like changing `@Sendable` to `sending`, + in an existing declaration. + +2. To replace an existing declaration with a source-compatible but ABI-breaking + equivalent, like replacing a `rethrows` method with one using typed + `throws`. + +3. To correct a mistake, like removing an unnecessary `@escaping` attribute or + adding a `Sendable` generic constraint. + +4. To rename an API whose name is felt to be catastrophically confusing. + +Many revisions will cause fundamental changes in how an API will be used at the +machine code level that clients must account for; for instance, changing `` +to `` requires callers to generate code that will pass the witness +table for `T`'s `Hashable` conformance. However, some features are designed to +have little or no impact on the code generated by the caller. For example, +these two declarations: + +```swift +// `T` must be `Sendable` +func fn(_: T) {} + +// `T` parameter must be `sending` +func fn(_: borrowing sending T) {} // note: 'borrowing sending' is currently banned, + // pending a decision on whether it should have the + // meaning we want it to have here +``` + +Have identical parameter signatures at the IR level, with one pointer to the +argument and another pointer to `T`'s value witness table, and use the same +result type and calling convention too: + +```text +define hidden swiftcc void @"$s4main2fnyyxs8SendableRzlF"(ptr noalias %0, ptr %T) + ^~~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~ +define hidden swiftcc void @"$s4main2fnyyxlF"(ptr noalias %0, ptr %T) + ^~~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~ +``` + +Other details, such as the parameter ownership conventions, also line up to +make this work; suffice it to say, the function generated when you use +`borrowing sending` is perfectly capable of handling the arguments passed by +callers that think the parameter is `Sendable`. The only differences between +them are the compile-time checks applied by the compiler and the part of their +mangled names that indicates the feature being used: + +```text +define hidden swiftcc void @"$s4main2fnyyxs8SendableRzlF"(ptr noalias %0, ptr %T) + ^~~~~~~~~~~~~ 'T: Sendable' +define hidden swiftcc void @"$s4main2fnyyxlF"(ptr noalias %0, ptr %T) + ^ 'T' ('sending' is not indicated by the mangled name) +``` + +Thus, if there was a way to tell the compiler to continue using the mangled +name for `fn(_: T)`, a library designer could actually change the +declaration to be treated like `fn(_: borrowing sending T)` when compiling +with the new version of the library *without* breaking ABI compatibility. + +This is part of how the `@preconcurrency` attribute works. `@preconcurrency` +has two effects: It instructs the type checker to permit Swift 5 code to use +the declaration in ways that would violate the rules of certain concurrency +annotations, and it causes those annotations to be omitted from the +declaration's mangled name. That makes it perfect for retrofitting sendability +checking onto APIs that were created before Swift Concurrency was introduced. +However, it is designed specifically for that exact task, which makes it +inflexible: It cannot be used to suppress some concurrency features but not +others (for instance, to amend a mistake in one parameter without affecting +other parts of the declaration), and it cannot be applied to adopt +non-concurrency features which have the same property of being ABI-compatible +except for a different mangled name. + +For everything else, there's the compiler-internal `@_silgen_name` attribute. +`@_silgen_name` is an internal hack that overrides name mangling at specific +points in the compiler, replacing the mangled name with an arbitrary string. +If you know the original mangled name, therefore, you can use this attribute +to keep that name stable even if the declaration has evolved enough that it +would normally use a different name. That makes `@_silgen_name` enormously +flexible—it can be used to handle an arbitrary set of changes, and the +standard library uses it extensively for this purpose. For example, when the +standard library introduced a new `Collection.map(_:)` that used typed +`throws` in [SE-0413][], it continued to support clients expecting the old +`rethrows`-based `map` by using a `@_silgen_name` hack and +`@usableFromInline internal`: + +```swift +extension Collection { + // New `map(_:)` using typed `throws`: + @inlinable + @backDeployed(...) // slight lie, but that's irrelevant here + public func map( + _ transform: (Element) throws(E) -> T + ) throws(E) -> [T] { + // ...actual implementation of `map` omitted... + } + + // Wrapper with the same ABI as the old `map(_:)` which used `rethrows`: + @_silgen_name("$sSlsE3mapySayqd__Gqd__7ElementQzKXEKlF") + // ^-- func map<$T>(_: (Self.Element) throws -> $T) rethrows -> Swift.Array<$T> + // in Swift.Collection extension from module Swift + @usableFromInline + func __rethrows_map( + _ transform: (Element) throws -> T + ) throws -> [T] { // 'throws' and 'rethrows' have the same ABI + try map(transform) // calls through to the new `map(_:)` + } +} +``` + +This creates a declaration which is written in Swift source code as +`__rethrows_map(_:)`, but which has the mangled name of a function named +`map(_:)`. When a module is compiled against this new version of the standard +library, calls to `map(_:)` will use the new method directly; if a module is +compiled against an older standard library, though, it will end up calling the +`__rethrows_map(_:)` compatibility wrapper instead. + +Although it is a powerful tool, `@_silgen_name` has its own set of serious +drawbacks: + +* It has absolutely no compile-time safety checking. +* It works only with functions and is incompatible with certain function + features like opaque return types and `@backDeployed`.[1] +* It requires deep knowledge of the name mangling and calling convention to use + correctly. + +In practice, you basically need to be a Swift compiler or runtime engineer to +use it correctly. For this reason `@_silgen_name` has never been proposed to +Swift Evolution or recommended for general use. + +Library maintainers need a tool that is much more flexible than +`@preconcurrency` but also much safer and more ergonomic than `@_silgen_name`. + +> *[1] This is because the name mangling has facilities to create multiple +> symbols that are all related to the same declaration, but `@_silgen_name` +> only provides an override for the name of the main symbol. Any declaration +> that requires more than one symbol—such as a type declaration, a function +> with an opaque return type, or a function with a back-deployment +> thunk—would have no way to generate a mangled name for these additional +> symbols.* + + [SE-0413]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0413-typed-throws.md#effect-on-abi-stability + +## Proposed solution + +We propose a new attribute, called `@abi`, which specifies an alternate +declaration that provides its ABI for name mangling purposes. This alternate +declaration is enclosed within the argument parentheses; it has no body or +initializer expression but is otherwise a syntactically complete declaration. + +For example, the `@_silgen_name`-using `__rethrows_map(_:)` method shown in the +Motivation section could be written much more clearly by using `@abi`: + +```swift +extension Collection { + // Wrapper with the same ABI as the old `map(_:)` which used `rethrows`: + @abi( + func map( + _ transform: (Element) throws -> T + ) rethrows -> [T] + ) + @usableFromInline + func __rethrows_map( + _ transform: (Element) throws -> T + ) throws -> [T] { // 'throws' and 'rethrows' have the same ABI + try map(transform) // calls through to the new `map(_:)` + } +} +``` + +Notice how the `@abi` attribute basically contains the original version of the +declaration. When Swift is performing name mangling, this declaration is what +it will use; for all other functions, it will use the outer `__rethrows_map` +declaration. In particular, the `map(_:)` call in the body doesn't get resolved +to the `map(_:)` function in the `@abi` attribute; it looks for other +implementations and eventually finds the new typed-throws `map(_:)`. + +What's more, the ABI declaration can be checked against the original one to +make sure they're compatible. For example, at the ABI level `throws` and +`rethrows` are interchangeable, but a non-`throws`/`rethrows` method handles +its return values differently from them. If the maintainer accidentally dropped +the `throws` effect while implementing this function, the compiler would +complain about the mismatch: + +```swift +extension Collection { + @abi( + func map( + _ transform: (Element) throws -> T + ) rethrows -> [T] // error: 'rethrows' doesn't match API + ) + @usableFromInline + func __rethrows_map( + _ transform: (Element) throws -> T + ) -> [T] { // Whoops, should be 'throws' or 'rethrows'! + try map(transform) + } +} +``` + +This checking also makes sure that the details specified in the `@abi` +attribute are actually relevant. For example, the `@abi` attribute +automatically inherits the access control, availability, and `@objc`-ness of +the API it's attached to, so these are omitted from the `@abi` attribute. +Default arguments, too, are left out because they're irrelevant to ABI. The +compiler will diagnose this unnecessary information and suggest removing it. + +All sorts of precision changes are possible. Here's another use for a +`@_silgen_name` hack in the standard library: The maintainers discovered a data +race safety bug in an API that had already shipped and needed to add an +`@Sendable` attribute to prevent it, but `@preconcurrency` alone would have +also suppressed the `@Sendable` attribute on the parameter that had been +correctly annotated. `@abi` makes it easy to fix this sort of problem: + +```swift +public struct AsyncStream { + // ...other declarations omitted... + + @abi( + init( + unfolding produce: @escaping /* not @Sendable */ () async -> Element?, + onCancel: (@Sendable () -> Void)? = nil + ) + ) + @preconcurrency + public init( + unfolding produce: @escaping @Sendable () async -> Element?, + onCancel: (@Sendable () -> Void)? = nil + ) { + // Implementation omitted + } +} +``` + +Because `@preconcurrency` is applied to the outer declaration, but not to the +one inside the `@abi` attribute, its typechecking effects will be applied +(improving source compatibility for code written before the second `@Sendable` +was added) but its name mangling effects will not (keeping the mangled name +stable to preserve ABI compatibility). + +This feature goes beyond what `@_silgen_name` could do, however. For example, +it can be applied to `var` and `let` declarations: + +```swift +@abi(var oldName: Int) +public var newName: Int +``` + +The mangled name of an accessor includes the mangled name of the variable or +subscript it belongs to; thanks to `@abi`, the accessors for this variable will +have `oldName` mangled into their names. + +### Supported changes (and unsupported uses of them) + +This feature can be used to override the mangling of a declaration's: + +* Name, argument labels, and (for unary operator functions) fixity (`prefix` + vs. `postfix`) + +* Preconcurrency status, actor isolation (where this does not affect calling + convention), and execution environment + +* Generic constraints to marker protocols (`BitwiseCopyable`, `Copyable`, + `Escapable`, `Sendable`) + +* Certain aspects of parameter and `self` behavior (variadic (vs. `Array`); + `@autoclosure`; `sending`; ownership specifiers as long as the behavior is + compatible) + +* Certain aspects of argument, result, and thrown types (marker protocols in + existentials; tuple element labels; `@escaping`, `@Sendable`, and `sending` + results on closures) + +Note that some of these changes relate to safety properties of your code, such +as data race safety and escapability. When you use `@abi` to maintain ABI +compatibility with older versions of your library while tightening safety +constraints for new clients, you must take special care to remember that +clients compiled without those changes may violate the new constraints. In +practice, this means that you should probably only use `@abi` to make +retroactive changes to safety constraints when you know that violating the +constraint was *always* unsafe and it simply wasn't enforced until now. + +For instance, the `AsyncStream.init(unfolding:onCancel:)` example above adds +`@Sendable` to a closure parameter that previously didn't have the attribute. +This is appropriate because the closure was *always* run concurrently; code +that passed a non-`@Sendable` closure was already buggy, so this change merely +made the bug easier to detect. It would have been inappropriate if the closure +was originally run synchronously and was changed to run concurrently, because +code that previously worked fine would now have new data races. + +(`@abi` can still be used to help implement behavior changes, but the pattern +is different: you make the original version `@usableFromInline internal` and +change its API name to something you won't use by accident, applying `@abi` to +keep its mangled name the same as always. Then you create a new declaration +with the old API name and the behavior changes, using `@backDeployed` to ensure +that new binaries can interoperate with old versions of your library. +`__rethrows_map(_:)` is a good example of this pattern.) + +In short: Much like when `@inlinable` is used, **it is the developer's +responsibility to ensure that the current behavior of the declaration is +compatible with clients built against older versions of it. The compiler +doesn't understand the history of your codebase and cannot detect some +mistakes.** + +## Detailed design + +### Grammar + +An `@abi` attribute's argument list must have exactly one argument, which in +this proposal must be one of the following productions: + +* *function-declaration* +* *initializer-declaration* +* *constant-declaration* +* *variable-declaration* +* *subscript-declaration* + +This argument must *not* include any of the following sub-productions: + +* *code-block* +* *getter-setter-block* +* *getter-setter-keyword-block* +* *willSet-didSet-block* +* *initializer* (initial value expression) + +To that end, we amend the following productions in the Swift grammar to make +code blocks optional: + +```diff + initializer-declaration → initializer-head generic-parameter-clause? + parameter-clause async? throws-clause? +- generic-where-clause? initializer-body ++ generic-where-clause? initializer-body? + + initializer-declaration → initializer-head generic-parameter-clause? + parameter-clause async? 'rethrows' +- generic-where-clause? initializer-body ++ generic-where-clause? initializer-body? + + subscript-declaration → subscript-head subscript-result generic-where-clause? +- code-block ++ code-block? +``` + +We don't need to worry about ambiguity in terminating these productions because +the block-less forms always occur in `@abi` attributes; its closing parenthesis serves as a terminator for the declaration. + +> **Note**: If future development of the `@abi` attribute requires additional +> information to be added to it, this can be done by adding new productions at +> the beginning of the argument list, terminated by a comma or colon to +> distinguish them from declaration modifiers: +> +> ```swift +> @abi(unchecked, func liveDangerously(_: AnyObject)) // Future direction +> func liveDangerously(_ object: AnyObject?) { ... } +> ``` + +### Terminology and basic concepts + +Syntactically, an `@abi` attribute involves two declarations. The *ABI-only +declaration* is the one in the attribute's argument list; the *API-only +declaration* is the one the attribute is attached to. + +```swift +@abi(func abiOnlyDeclaration()) +func apiOnlyDeclaration() {} +``` + +A declaration which does not involve an `@abi` attribute at all—that is, which +is neither API-only nor ABI-only—is called a *normal declaration*. + +There are two *ABI roles*: + +* An *API-providing declaration* determines the behavior of the declaration in + source code: what name developers write to address it, what constraints and + behaviors are applied at use sites, how it is implemented (its body, + accessors, or members), etc. + +* An *ABI-providing declaration* determines how the declaration affects mangled + symbol names--both its own name and any names derived from it. + +Every declaration has at least one of these roles. Every declaration also has a +*counterpart* which fulfills the roles it does not. When the compiler wants to +compute some aspect of a declaration pertaining to a role that declaration does +not have, it automatically substitutes the declaration's counterpart. + +Roles and counterparts work as follows: + +| Declaration is… | ABI-providing | API-providing | Counterpart | +| --------------- | ------------- | ------------- | ------------------------------------------- | +| Normal | ✅ | ✅ | Is its own counterpart | +| ABI-only | ✅ | | Declaration `@abi` attribute is attached to | +| API-only | | ✅ | Declaration in `@abi` attribute | + +### Declaration checking + +When you use the `@abi` attribute, Swift validates various aspects of the +ABI-providing declaration in light of its API counterpart. An *aspect* is any +way in which the external appearance of a declaration might vary. Attributes +and modifiers are aspects, but so are the declaration's name, its result or +value types, its generic signature (if it has one), its parameter list (if it +has one), its effects (if it has any), and so on. + +#### Aspects with no ABI impact must be omitted + +Many aspects of a declaration only matter for an API-providing declaration; +they're irrelevant on a declaration that's ABI-only. These include: + +* Default arguments for parameters + +* Attributes which only affect compile-time checking or behavior, such as + `@unsafe`, `@discardableResult`, or result builder attributes + +* Certain attributes and modifiers which have ABI effects, but where the + compiler has been designed to inherit the ABI-providing declaration's + behavior from its API counterpart where needed: + + * `@objc` (and its ilk) and `dynamic`, including inference behaviors + * Access control modifiers and `@usableFromInline` + * `@inlinable` and other attributes controlling inlining + * `@available` and `@backDeployed` + * `override` + +These aspects are generally *forbidden* on an ABI-providing declaration. If +they are present, the compiler will diagnose an error and suggest they be +removed. + +In practice, this means that an `@abi` attribute is often significantly shorter +than the declaration it's attached to because it doesn't need to specify as +much information: + +```swift +@abi( + // Same signature as below, except `T` is not `Sendable`. + static + func assumeIsolated( + _ operation: @MainActor () throws -> T, + file: StaticString, + line: UInt + ) rethrows -> T +) +@available(macOS 14.0, iOS 17.0, watchOS 10.0, tvOS 17.0, *) // not needed in @abi +@usableFromInline // not needed in @abi +internal // not needed in @abi +static +func assumeIsolated( + _ operation: @MainActor () throws -> T, + file: StaticString = #fileID, // default argument not needed in @abi + line: UInt = #line // default argument not needed in @abi +) rethrows -> T { + ... +} +``` + +The intended workflow is that a developer can paste the entire original +declaration into an `@abi` attribute and the compiler will then tell them which +parts of it they should remove. + +#### Call compatibility + +For aspects which *do* have ABI impact, the compiler enforces that the +ABI-providing declaration is *call-compatible* with its API-providing +counterpart. Broadly, "call compatibility" means that, other than the mangled +names, the machine code generated to call the ABI-providing declaration would +be equally able to call its API counterpart. For instance: + +* Declarations are of the same fundamental kind (a `func` for a `func`, a `var` + for a `var`, etc.), so they expose the same basic capabilities and entry + points. + +* Effects match closely enough that the caller and callee will agree on which + stack should be used and which implicit parameters will be passed. + +* Inputs and outputs are passed similarly enough to ensure type and memory + safety, including compatible memory management behavior, and including the + implicit inputs and outputs used for generic parameters, `self`, and + throwing. + +However, call compatibility does *not* require aspects of the declaration +that only change the mangled name and/or compile-time checking to match: + +* Names, argument labels, or other name-like traits of a declaration (such + as the fixity modifiers for operator functions) may vary. + +* Aspects which affect only syntax (and possibly mangling) may vary. For + instance, a regular closure may be used instead of an `@autoclosure`; tuple + types with different element labels may be used; an array parameter may be + used instead of a variadic parameter; ordinary optionals may be used instead + of implicitly-unwrapped optionals; `throws` and `rethrows` may be used + interchangeably. + +* Concurrency safety and lifetime restrictions which don't affect the ability + to call the declaration may vary. For instance, a non-escaping closure may be + used instead of an `@escaping` closure; `sending` modifiers, `Sendable` + constraints, or neither may be used interchangeably so long as memory + management isn't affected; isolation may vary so long as extra data does not + need to be passed; `~Copyable` and `~Escapable` constraints may vary. + +#### Impact on redeclaration checking + +A declaration must have a unique signature in each of its roles. That is, an +API-only declaration is checked against API-providing declarations; an +ABI-only declaration is checked against ABI-providing declarations; a normal +declaration is checked twice, first against API-providing declarations and then +against ABI-providing declarations. + +In general, name lookup will return declarations with the API-providing role +and will ignore declarations with the ABI-providing role. Even when you're +writing an ABI-only declaration, you should use the API names of other +declarations, not the ABI names. + +#### Declaring multiple variables + +When `var` or `let` is used, the ABI-providing declaration must bind the same +number of patterns, each of which has the same number of variables, as its API +counterpart. That is, the first of these is valid, while the others are not: + +```swift +// OK: +@abi(var x, y: Int) +var a, b: Int + +// Mismatched: +@abi(var x, y, z: Int) +var a, b: Int + +// Also mismatched: +@abi(var x, y: Int) +var a: Int, (b1, b2): (Int, Int) + +// Mismatched even though the total adds up: +@abi(var x, y, z: Int) +var a: Int, (b1, b2): (Int, Int) +``` + +An ABI-providing declaration does *not* infer missing types from its API +counterpart. In practice, this means that an ABI-providing declaration +may need to explicitly declare types that its API counterpart infers from an +initial value expression. + +An ABI-providing `var` or `let` does not have a list of accessors or specify +anything about them; in a sense, it can be thought of as inferring its +accessors from its API counterpart. + +### Limitations on feature scope + +#### Supported declaration kinds + +In this proposal, `@abi` may be applied only to `func`, `init`, `var`, `let`, +and `subscript` declarations. Other declarations are less straightforward to +support in various ways; see the future directions section for details. + +#### Language features with auxiliary declarations + +`@abi` can neither contain, nor be applied alongside, `lazy` or a property +wrapper. These features implicitly create auxiliary declarations, and it isn't +clear how those should interact with `@abi`. + +#### Limited support for macros + +Neither attached nor freestanding macros can be used inside an `@abi` +attribute. None of the attached macro roles would be useful since ABI-providing +declarations do not have bodies, members, accessors, or extensions; the +freestanding macro roles, on the other hand, expand to complete declarations, +while some of the future directions involve supporting special stub syntax +which would be incompatible here. + +`@abi` can still be applied *alongside* an attached macro or *to* a +freestanding macro, although in practice many macros will need to handle `@abi` +attributes specially. + +### Non-normative: Precise rules as currently implemented + +To help evaluate how these principles will work in practice, we've listed the +current implementation's rules below. **However, we do not guarantee that the +rules listed here will exactly match the final behavior of the feature.** +Basically, we don't want to put every bug fix through an amendment or every +tiny, straightforward expansion of capabilities through a proposal. + +#### Must be omitted (no ABI impact or inheritance in place) + +* Default arguments on parameters +* Result builder attributes on parameters or declarations +* `@available` +* `@inlinable`, `@inline`, `@backDeployed`, `@usableFromInline`, + `@_alwaysEmitIntoClient`, `@_transparent` +* Objective-C opt-in attributes (`@objc`, `@IBAction`, `@IBDesignable`, + `@IBInspectable`, `@IBOutlet`, `@IBSegueAction`, `@GKInspectable`, + `@NSManaged`, `@nonobjc`) +* `optional` modifier in `@objc` protocols +* `@NSCopying` +* `@_expose` and `@_cdecl` +* `@LLDBDebuggerFunction` +* `dynamic` modifier and `@_dynamicReplacement` +* `@specialize` on functions and initializers +* `override` modifier +* Access control (`open`, `public`, `package`, `internal`, `fileprivate`, + `private`) +* Setter access control (`open(set)`, `public(set)`, `package(set)`, + `internal(set)`, `fileprivate(set)`, `private(set)`) +* `@_spi` and `@_spi_available` +* Reference ownership (`weak`, `unowned`, `unowned(unsafe)`) +* `@warn_unqualified_access` +* `@discardableResult` +* `@implementation` on functions +* `@differentiable`, `@derivative`, `@transpose` +* `@noDerivative` on declarations other than parameters +* `@exclusivity` +* `@safe` and `@unsafe` +* `@abi` +* Unsupported features (`lazy`, property wrapper attributes, attached macro + attributes) + +#### Must be specified and must match + +* Declaration kind (`func`, `var`, etc.) +* `convenience` and `required` modifiers on initializers +* `distributed` modifier +* Result type of functions +* Failability of initializers +* Value type of subscripts and variables +* Number of parameters on functions, initializers, and subscripts +* Parameter types +* `inout` on parameters and `mutating`/`nonmutating` modifiers on members +* `@noDerivative` on parameters +* `@_addressable` on parameters and `@_addressableSelf` on members +* `@lifetime` attributes (NOTE: this probably ought to be "vary with + constraints", but an interaction with `@_addressableForDependencies` needs + to be worked out) +* `async` effect +* Aspects of types which are not listed elsewhere + +#### Allowed to vary, but with constraints + +* `throws` effect and thrown type (`rethrows` is equivalent to `throws`) +* Generic signature of functions, initializers, and subscripts (marker + protocols may vary) +* Variadic parameter types (`T...` and `Array` are treated as equivalent) +* Parameter ownership specifiers and `self` ownership modifiers (ones with + equivalent memory management behavior may be substituted for one another) +* `sending` on parameter and result types (so long as its ownership behavior + is preserved) +* `static`, `class`, and `final` modifiers (`class final` is equivalent to + `static`) +* Actor isolation (other than `@isolated(any)`, which is incompatible with + the others) + +#### Allowed to vary arbitrarily + +* Base names of functions and variables +* Argument labels and parameter names +* `prefix` and `postfix` modifiers on operator functions +* Whether optionals are implicitly unwrapped +* Element labels in tuple types +* `@autoclosure` on parameters +* `@escaping` on closures +* `@Sendable` on closures +* `isolated` on parameters +* `_const` on parameters and variables +* Generic parameter names +* Use of type sugar (e.g. `Optional` and `T?` are equivalent) +* Use of generic types that have a same-type constraint (e.g. in the + presence of `T == Int`, `T` and `Int` are equivalent) +* Use of marker protocols in existential types +* `@Sendable` on functions and initializers +* `@preconcurrency` +* `@execution` +* Aspects of declarations which are not listed elsewhere + +## Source compatibility + +This feature is additive and affects only the ABI. However, many of the changes +that can be effected using it can be source-breaking unless done with care. For +example: + +* When renaming a declaration, make sure there's a declaration with the + original name that can be called in the same situations, and consider using + `@backDeployed` to ensure recompiled clients don't have to raise their + minimum deployment target. + +* When a type changes, make sure that it will either become more broad, or that + you are willing to accept any breakage that results. For example, switching + from a `Sendable` constraint to a `(borrowing) sending` parameter strictly + increases the set of valid callers, so that's probably always okay; switching + from no constraint to a `Sendable` constraint, on the other hand, will break + some callers, but might be acceptable if the missing `Sendable` constraint + created an opportunity for data races. + +## ABI compatibility + +This feature is intended to give libraries additional options to evolve APIs +without breaking the corresponding ABIs. + +We are currently evaluating adoption in `stdlib/public`. So far, it looks like +we can replace all uses of `@_silgen_name` to specify mangled Swift names with +uses of `@abi`, and in some cases remove hacks; this will be roughly 75 +declarations. + +Note that this feature does not subsume the use of `@_silgen_name` with an +arbitrary, C-style symbol name to either declare a function implemented in +C++ using the Swift calling convention, or to generate a symbol that's easy to +access from C-family code or a compiler intrinsic. About 200 of the uses of +`@_silgen_name` in `stdlib/public` are of this type; we expect these to remain +as-is. + +## Implications on adoption + +This feature is intended to help ease the adoption of other new features by +allowing a declaration's ABI to be "pinned" to its original form even as it +continues to evolve. Note that there is only ever a need to specify the +*original* form of the declaration, not any revisions that may have occurred +between then and the current form; there is therefore never a reason you would +need to specify more than one `@abi` attribute, nor to tie an `@abi` attribute +to a specific platform version. + +In module interfaces, the `@abi` attribute is partially suppressible. +Specifically, for `func`s that do not use `@backDeployed` and do not have +opaque result types, the compiler emits a module interface that falls back to +using an equivalent `@_silgen_name` attribute. For other declarations, however, +the compiler falls back to an `@available(*, unavailable)` attribute instead, +with a message indicating that the developer will need a newer compiler to use +the declaration. + +## Future directions + +### Unchecked mode + +There may be situations where a skilled engineer knows that a specific use of +`@abi` is compatible, but the compiler does not know how to prove that. While +that can often be considered a compiler bug—the checker *should* be able to +tell that the code is safe—it may be useful, either as a workaround or to +handle extreme edge cases, to be able to turn off `@abi`'s compatibility +checking: + +```swift +@abi(unchecked, func liveDangerously(_: AnyObject)) +func liveDangerously(_ object: AnyObject?) { ... } +``` + +### Support for types and extensions + +It ought to be possible to use `@abi` with types: + +```swift +@abi(struct Buffer: ~Copyable) +public struct FrameBuffer: ~Copyable { ... } + +@available(*, unavailable, renamed: "FrameBuffer") +@abi(typealias FrameBuffer) // keeps `typealias Buffer` from colliding with `struct Buffer` +typealias Buffer = FrameBuffer +``` + +Here, the library maintainer discovered after shipping that the name `Buffer` +is too vague—clients didn't understand what it meant, and some of them even +had another type named `Buffer`. When they rebuild with the new version of the +library, they will get an error with a fix-it to change `Buffer` to +`FrameBuffer`, but it will still use the name `Buffer` at the ABI level so +that existing binaries don't break. This will apply not only to the type +itself, but also to its members and even to functions with a `FrameBuffer` in +their overload signature. + +Type renaming may create challenges for module interface source stability, +since a module interfaces could refer to a type by an older or newer name than +its current one. It might be possible to address this by making module +interfaces always refer to types by their ABI name. (This should be +non-breaking as long as it's introduced at the same time as `@abi` for types.) + +This could also be used to affect the inference of properties of other +declarations. Consider this example: + +```swift +@abi(protocol Component) +@preconcurrency @MainActor // Added after shipping +public protocol Component { ... } + +extension Component { + public func onEvent(_ handler: @Sendable @escaping () -> Void) -> some Component { ... } +} +``` + +The library maintainer decided after the fact that `Component`s should be +isolated to the main actor, but that broke some Swift 5 code, so they added +`@preconcurrency` for its typechecking effects. However, `@preconcurrency` then +got applied to `onEvent(_:)`, suppressing its `@Sendable` attribute, which +changed its ABI. Using `@abi` on `Component` should override the ABI effects of +`@preconcurrency` not just for `Component` itself, but for every declaration +nested inside it. + +To support this kind of inference, the compiler may need to add an inferred +`@abi` attribute when a declaration with both roles depends on one which +provides only one role. For instance, if a type conforms to a protocol whose +`@abi` attribute specifies different actor isolation or different marker +protocols, Swift may need to add an inferred `@abi` attribute so the type's ABI +will be compatible with the protocol's ABI while its API will be compatible +with the protocol's ABI. + +### Support for enum cases + +It would probably be possible to allow `@abi` to be attached to a `case` +declaration, allowing it to backwards-compatibly rename or otherwise control +the ABI of enum cases. + +### Support for auxiliary declarations + +It might be possible to allow `@abi` to be used with `lazy` and property +wrappers either by coming up with rules to derive an `@abi` attribute for those +declarations, or by creating a syntax that can specify them: + +```swift +@abi(nonisolated var currentValue: Int) +@abi(for: projection, nonisolated var $currentValue: Binding) +@abi(for: storage, nonisolated var _currentValue: Binding) +@MainActor @Binding var currentValue: Int +``` + +### Support for accessors + +It might be possible to allow `@abi` to be attached to individual accessors. + +### Support for context changes + +It might be possible to allow an ABI-providing declaration to belong to a +different context than its counterpart—for instance, turning a global variable +into a static property, or moving a method to a single-property `@frozen` +wrapper struct. + +A particularly interesting one might be allowing an extension member to +be mangled as a member of the main type declaration, or vice versa, since users +may not be aware of the ABI impact of moving a declaration from one to the +other. + +### Equivalent type attribute + +Many uses of `@abi` only change one or two types in a complicated declaration. +It might be possible to provide an `@abi` *type* attribute that can be applied +on the spot as a shorthand: + +```swift +public func runConcurrently( + _ body: @escaping @abi(() -> Void) @Sendable () -> Void +) { ... } + +// Equivalent to: +@abi(func runConcurrently(_: @escaping () -> Void)) +public func runConcurrently( + _ body: @escaping @Sendable () -> Void +) { ... } +``` + +### Support for moving declarations to a different module + +It might be possible to roll some of the functionality of the compiler-internal +`@_originallyDefinedIn` attribute into this attribute. + +## Alternatives considered + +### Many narrow features + +The original motivation for this proposal involved fairly narrow cases where +`@preconcurrency` was too blunt an instrument, such as suppressing the ABI +impact of one sendability annotation while leaving others intact. One could +imagine designing individual type or decl attributes for each specific change +one might wish to make, but this would require both a lot of effort from +compiler engineers to design a specific tool for each problem, and a lot of +effort from library maintainers to figure out which tool to apply to a given +task and how to use it. + +### An argument that describes the differences + +Rather than creating many totally separate features, one could imagine an +`@abi` declaration attribute with an argument list which somehow described the +differences between the API and ABI. However, we see use cases for changing +virtually every aspect of an API—its name; adding or removing declaration +attributes, modifiers, and effects; adding or removing inherited protocols and +generic constraints; changing parameter, result, and error types; changing type +attributes and modifiers; even changing individual sub-types within generic +types, function types, and protocol compositions at any position in the +declaration—and a mini-language to address and edit all of these different +aspects of a declaration seems difficult to design and tedious to learn. +By contrast, re-specifying the entire declaration in the argument reuses the +developer's existing knowledge of how to read and write declarations and gives +them an easy way to adopt it (just copy the existing declaration into an `@abi` +attribute before you start editing in new features). + +### Syntax where the two declarations are peers + +We could design this differently such that the API-only and ABI-only +declarations are peers in the same context: + +```swift +// This declaration provides the ABI... +@abi(for: __rethrows_map(_:)) @usableFromInline func map( + _ transform: (Element) throws -> T +) rethrows -> [T] + +// ...for this declaration +@usableFromInline func __rethrows_map( + _ transform: (Element) throws -> T +) throws -> [T] { + try map(transform) +} +``` + +In theory, this design could simplify parsing, since the `@abi` attribute's +argument might just be an ordinary expression. However, it introduces several +complications: + +1. Merely giving the name of a declaration may not be specific enough in the + presence of overloads, or even when the API and ABI have the same name but + slight type differences. We might normally tell developers to work around + this by using more specific names, but that's not really an appropriate + answer for a tool which is designed to allow fine control of API and ABI + naming. +2. A lot of compiler logic would have to be modified to filter out ABI-only or + API-only declarations when it walked through lists of top-level decls or + members. The current design, where the ABI-only declarations are tucked away + in attributes, keeps them from being accessed by accident. +3. If the future direction for `@abi` on type declarations is taken, the + productions for full type declarations will not be suitable, as they require + member blocks. + +## Acknowledgments + +Thanks to Holly Borla for recognizing the need for an `@abi` attribute. diff --git a/proposals/0477-default-interpolation-values.md b/proposals/0477-default-interpolation-values.md new file mode 100644 index 0000000000..fc024885c9 --- /dev/null +++ b/proposals/0477-default-interpolation-values.md @@ -0,0 +1,171 @@ +# Default Value in String Interpolations + +* Proposal: [SE-0477](0477-default-interpolation-values.md) +* Authors: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#80547](https://github.com/swiftlang/swift/pull/80547) +* Review: ([pitch](https://forums.swift.org/t/pitch-default-values-for-string-interpolations/69381)) ([review](https://forums.swift.org/t/se-0477-default-value-in-string-interpolations/79302)) ([acceptance](https://forums.swift.org/t/accepted-with-modification-se-0477-default-value-in-string-interpolations/79609)) + +## Introduction + +A new string interpolation syntax for providing a default string +when interpolating an optional value. + +## Motivation + +String interpolations are a streamlined and powerful way to include values within a string literal. +When one of those values is optional, however, +interpolating is not so simple; +in many cases, a developer must fall back to unpalatable code +or output that exposes type information. + +For example, +placing an optional string in an interpolation +yields an important warning and two suggested fixes, +only one of which is ideal: + +```swift +let name: String? = nil +print("Hello, \(name)!") +// warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit? +// print("Hello, \(name)!") +// ^~~~ +// note: use 'String(describing:)' to silence this warning +// print("Hello, \(name)!") +// ^~~~ +// String(describing: ) +// note: provide a default value to avoid this warning +// print("Hello, \(name)!") +// ^~~~ +// ?? <#default value#> + +``` + +The first suggestion, adding `String(describing:)`, +silences the warning but includes `nil` in the output of the string — +maybe okay for a quick shell script, +but not really appropriate result for anything user-facing. + +The second suggestion is good, +allowing us to provide whatever default string we'd like: + +```swift +let name: String? = nil +print("Hello, \(name ?? "new friend")!") +``` + +However, the nil-coalescing operator (`??`) +only works with values of the same type as the optional value, +making it awkward or impossible to use when providing a default for non-string types. +In this example, the `age` value is an optional `Int`, +and there isn't a suitable integer to use when it's `nil`: + +```swift +let age: Int? = nil +print("Your age: \(age)") +// warning, etc.... +``` + +To provide a default string when `age` is missing, +we have to write some gnarly code, +or split out the missing case altogether: + +```swift +let age: Int? = nil +// Optional.map +print("Your age: \(age.map { "\($0)" } ?? "missing")") +// Ternary expression +print("Your age: \(age != nil ? "\(age!)" : "missing")") +// if-let statement +if let age { + print("Your age: \(age)") +} else { + print("Your age: missing") +} +``` + +## Proposed solution + +The standard library should add a string interpolation overload +that lets you write the intended default as a string, +no matter what the type of value: + +```swift +let age: Int? = nil +print("Your age: \(age, default: "missing")") +// Prints "Your age: missing" +``` + +This addition will improve the clarity of code that uses string interpolations +and encourage developers to provide sensible defaults +instead of letting `nil` leak into string output. + +## Detailed design + +The implementation of this new interpolation overload looks like this, +added as an extension to the `DefaultStringInterpolation` type: + +```swift +extension DefaultStringInterpolation { + mutating func appendInterpolation( + _ value: T?, + default: @autoclosure () -> String + ) { + if let value { + self.appendInterpolation(value) + } else { + self.appendInterpolation(`default`()) + } + } +} +``` + +The new interpolation's `default:` parameter name +matches the one in the `Dictionary` subscript that has a similar purpose. + +You can try this out yourself by copy/pasting the snippet above into a project or playground, +or by experimenting with [this Swift Fiddle](https://swiftfiddle.com/nxttprythnfbvlm4hwjyt2jbjm). + +## Source compatibility + +This proposal adds one new API to the standard library, +which should not be source-breaking for any existing projects. +If a project or a dependency has added a similar overload, +it will take precedence over the new standard library API. + +## ABI compatibility + +This proposal is purely an extension of the ABI of the +standard library and does not change any existing features. + +## Implications on adoption + +The new API will be included in a new version of the Swift runtime, +and is marked as backward deployable. + +## Future directions + +There are [some cases][reflecting] where a `String(reflecting:)` conversion +is more appropriate than the `String(describing:)` normally used via string interpolation. +Additional string interpolation overloads could make it easier to use that alternative conversion, +and to provide a default when working with optional values. + +[reflecting]: https://forums.swift.org/t/pitch-default-values-for-string-interpolations/69381/58 + +## Alternatives considered + +**An interpolation like `"\(describing: value)"`** +This alternative would provide a shorthand for the first suggested fix, +using `String(describing:)`. +Unlike the solution proposed, +this kind of interpolation doesn't make it clear that you're working with an optional value, +so you could end up silently including `nil` in output without expecting it +(which is the original reason for the compiler warnings). + +**Extend `StringInterpolationProtocol` instead** +The proposed new interpolation works with _any_ optional value, +but some types only accept a limited or constrained set of types interpolations. +If the new `\(_, default:)` interpolation proves to be a useful pattern, +other types can add it as appropriate. + diff --git a/proposals/0478-default-isolation-typealias.md b/proposals/0478-default-isolation-typealias.md new file mode 100644 index 0000000000..590ea5fd7b --- /dev/null +++ b/proposals/0478-default-isolation-typealias.md @@ -0,0 +1,115 @@ +# Default actor isolation typealias + +* Proposal: [SE-0478](0478-default-isolation-typealias.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Active Review (April 21 ... May 5, 2025)** +* Vision: [Improving the approachability of data-race safety](/visions/approachable-concurrency.md) +* Implementation: [swiftlang/swift#80572](https://github.com/swiftlang/swift/pull/80572) +* Experimental Feature Flag: `DefaultIsolationTypealias` +* Previous Proposal: [SE-0466: Control default actor isolation inference][SE-0466] +* Review: ([pitch](https://forums.swift.org/t/pitch-a-typealias-for-per-file-default-actor-isolation/79150))([review](https://forums.swift.org/t/se-0478-default-actor-isolation-typealias/79436)) + +## Introduction + +[SE-0466: Control default actor isolation inference][SE-0466] introduced the ability to specify default actor isolation on a per-module basis. This proposal introduces a new typealias for specifying default actor isolation in individual source files within a module. This allows specific files to opt out of main actor isolation within a main-actor-by-default module, and opt into main actor isolation within a nonisolated-by-default module. + +## Motivation + +SE-0466 allows code to opt in to being “single-threaded” by default by isolating everything in the module to the main actor. When the programmer really wants concurrency, they can request it explicitly by marking a function or type as `nonisolated`, or they can define it in a module that does not default to main-actor isolation. However, it's very common to group multiple declarations used in concurrent code into one source file or a small set of source files. Instead of choosing between writing `nonisolated` on each individual declaration or splitting those files into a separate module, it's desirable to state that all declarations in those files default to `nonisolated`. + +## Proposed solution + +This proposal allows writing a private typealias named `DefaultIsolation` to specify the default actor isolation for a file. + +An underlying type of `MainActor` specifies that all declarations in the file default to main actor isolated: + +```swift +// main.swift + +private typealias DefaultIsolation = MainActor + +// Implicitly '@MainActor' +var global = 0 + +// Implicitly '@MainActor' +func main() { ... } + +main() +``` + +An underlying type of `nonisolated` specifies that all declarations in the file default to `nonisolated`: + +```swift +// Point.swift + +private typealias DefaultIsolation = nonisolated + +// Implicitly 'nonisolated' +struct Point { + var x: Int + var y: Int +} +``` + +## Detailed design + + A typealias named `DefaultIsolation` can specify the actor isolation to use for the source file it's written in under the following conditions: + +* The typealias is written at the top-level. +* The typealias is `private` or `fileprivate`; the `DefaultIsolation` typealias cannot be used to set the default isolation for the entire module, so its access level cannot be `internal` or above. +* The underlying type is either `MainActor` or `nonisolated`. + + It is not invalid to write a typealias called `DefaultIsolation` that does not meet the above conditions. Any typealias named `DefaultIsolation` that does not meet the above conditions will be skipped when looking up the default isolation for the source file. The compiler will emit a warning for any `DefaultIsolation` typealias that is not considered for default actor isolation along with the reason why: + +```swift +@globalActor +actor CustomGlobalActor { + static let shared = CustomGlobalActor() +} + +private typealias DefaultIsolation = CustomGlobalActor // warning: not used for default actor isolation +``` + +To allow writing `nonisolated` as the underlying type of a typealias, this proposal adds a typealias named `nonisolated` to the Concurrency library: + +```swift +public typealias nonisolated = Never +``` + +This typealias serves no purpose beyond specifying default actor isolation. To specify `nonisolated` using the `DefaultIsolation` typealias, the underlying type must be `nonisolated` exactly; it is invalid to write `private typealias DefaultIsolation = Never`. + +## Source compatibility + +Technically source breaking if someone happens to have written a private `DefaultIsolation` typealias with an underlying type of `MainActor`, which will start to infer every declaration in that file as `@MainActor`-isolated after this change. This seems extremely unlikely. + +## ABI compatibility + +This proposal has no ABI impact on existing code. + +## Implications on adoption + +This proposal does not change the adoption implications of adding `@MainActor` to a declaration that was previously nonisolated and vice versa. The source and ABI compatibility implications of changing actor isolation are documented in the Swift migration guide's [Library Evolution](https://github.com/apple/swift-migration-guide/blob/29d6e889e3bd43c42fe38a5c3f612141c7cefdf7/Guide.docc/LibraryEvolution.md#main-actor-annotations) article. + +## Alternatives considered + +Adding a typealias named `nonisolated` to `Never` to the Concurrency library to enable writing it as the underlying type of a typealias is pretty strange; this approach leverages the fact that `nonisolated` is a contextual keyword, so it's valid to use `nonisolated` as an identifier. This proposal uses a typealias instead of an empty struct or enum type to avoid the complications of having a new type be only available with the Swift 6.2 standard library. + +It's extremely valuable to have a consistent way to spell `nonisolated`. Introducing a type that follows standard naming conventions, such as `Nonisolated`, or using an existing type like `Never` is more consistent with recommended style, but overall complicates the concurrency model because it means you need to spell `nonisolated` differently when specifying it per file versus writing it on a declaration. And because the underlying type of this typealias is used to infer actor isolation, it's not used as a type in the same way that other typealiases are. + +Another alternative is to introduce a bespoke syntax such as `using MainActor` or `using nonisolated`. This approach preserves a consistent spelling for `nonisolated`, but at the cost of adding new language syntax that deviates from other defaulting rules such as the default literal types and the default actor system types. + +Having a `nonisolated` typealias may also allow us to improve the package manifest APIs for specifying default isolation, allowing us to move away from using `nil` to specify `nonisolated`: + +```swift +SwiftSetting.defaultIsolation(nonisolated.self) +``` + +We can also pursue allowing bare metatypes without `.self` to allow: + +```swift +SwiftSetting.defaultIsolation(nonisolated) +SwiftSetting.defaultIsolation(MainActor) +``` + +[SE-0466]: /proposals/0466-control-default-actor-isolation.md diff --git a/proposals/0479-method-and-initializer-keypaths.md b/proposals/0479-method-and-initializer-keypaths.md new file mode 100644 index 0000000000..bd1db9fe2e --- /dev/null +++ b/proposals/0479-method-and-initializer-keypaths.md @@ -0,0 +1,176 @@ +# Method and Initializer Key Paths + +* Proposal: [SE-0479](0479-method-and-initializer-keypaths.md) +* Authors: [Amritpan Kaur](https://github.com/amritpan), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Active Review (April 22 ... May 5, 2025)** +* Implementation: [swiftlang/swift#78823](https://github.com/swiftlang/swift/pull/78823), [swiftlang/swiftsyntax#2950](https://github.com/swiftlang/swift-syntax/pull/2950), [swiftlang/swiftfoundation#1179](https://github.com/swiftlang/swift-foundation/pull/1179) +* Experimental Feature Flag: `KeyPathWithMethodMembers` +* Review: ([pitch](https://forums.swift.org/t/pitch-method-key-paths/76678)) ([review](https://forums.swift.org/t/se-0479-method-and-initializer-key-paths/79457)) + +## Introduction + +Swift key paths can be written to properties and subscripts. This proposal extends key path usage to include references to method members, such as instance and type methods, and initializers. + +## Motivation + +Key paths to method members and their advantages have been explored in several discussions on the Swift forum, specifically to [unapplied instance methods](https://forums.swift.org/t/allow-key-paths-to-reference-unapplied-instance-methods/35582) and to [partially and applied methods](https://forums.swift.org/t/pitch-allow-keypaths-to-represent-functions/67630). Extending key paths to include reference to methods and initializers and handling them similarly to properties and subscripts will unify instance and type member access for a more consistent API. While this does not yet encompass all method kinds, particularly those with effectful or non-hashable arguments, it lays the groundwork for more expressive, type-safe APIs. In doing so, it brings many of the benefits of existing key path components to supported methods and initializers, such as abstraction, reusability via generic functions and dynamic invocation with state type safety. + +## Proposed solution + +We propose the following usage: + +```swift +struct Calculator { + func square(of number: Int) -> Int { + return number * number * multiplier + } + + func cube(of number: Int) -> Int { + return number * number * number * multiplier + } + + init(multiplier: Int) { + self.multiplier = multiplier + } + + let multiplier: Int +} + +// Key paths to Calculator methods +let squareKeyPath = \Calculator.square +let cubeKeyPath = \Calculator.cube +``` + +These key paths can then be invoked dynamically with a generic function: + +```swift +func invoke(object: T, keyPath: KeyPath U>, param: U) -> U { + return object[keyPath: keyPath](param) +} + +let calc = Calculator(multiplier: 2) + +let squareResult = invoke(object: calc, keyPath: squareKeyPath, param: 3) +let cubeResult = invoke(object: calc, keyPath: cubeKeyPath, param: 3) +``` + +Or used to dynamically create a new instance of Calculator: + +```swift +let initializerKeyPath = \Calculator.Type.init(multiplier: 5) +``` + +This proposed feature homogenizes the treatment of member declarations by extending the expressive power of key paths to method and initializer members. + +## Detailed design + +Key path expressions can refer to instance methods, type methods and initializers, and imitate the syntax of non-key path member references. + +### Argument Application + +Key paths can reference methods in two forms: + +1. Without argument application: The key path represents the unapplied method signature. +2. With argument application: The key path references the method with arguments already applied. + +Continuing our `Calculator` example, we can write either: + +```swift +let squareWithoutArgs: KeyPath Int> = \Calculator.square +let squareWithArgs: KeyPath = \Calculator.square(of: 3) +``` + +If the member is a metatype (e.g., a static method, class method, initializer, or when referring to the type of an instance), you must explicitly include `.Type` in the key path root type. + +```swift +struct Calculator { + static func add(_ a: Int, _ b: Int) -> Int { + return a + b + } +} + +let addKeyPath: KeyPath = \Calculator.Type.add(4, 5) +``` + +Here, `addKeyPath` is a key path that references the add method of `Calculator` as a metatype member. The key path’s root type is `Calculator.Type`, and the value resolves to an applied instance method result type of`Int`. + +### Overloads + +Keypaths to methods with the same base name and distinct argument labels can be disambiguated with explicit argument labels: + +```swift +struct Calculator { + var subtract: (Int, Int) -> Int { return { $0 + $1 } } + func subtract(this: Int) -> Int { this + this} + func subtract(that: Int) -> Int { that + that } +} + +let kp1 = \Calculator.subtract // KeyPath Int +let kp2 = \Calculator.subtract(this:) // KeyPath Int> +let kp3 = \Calculator.subtract(that:) // KeyPath Int> +let kp4 = \Calculator.subtract(that: 1) // KeyPath +``` + +### Implicit closure conversion + +This feature also supports implicit closure conversion of key path methods, allowing them to used in expressions where closures are expected, such as in higher order functions: + +```swift +struct Calculator { + func power(of base: Int, exponent: Int) -> Int { + return Int(pow(Double(base), Double(exponent))) + } +} + +let calculators = [Calculator(), Calculator()] +let results = calculators.map(\.power(of: 2, exponent: 3)) +``` + +### Dynamic member lookups + +`@dynamicMemberLookup` can resolve method references through key paths, allowing methods to be accessed dynamically without explicit function calls: + +```swift +@dynamicMemberLookup +struct DynamicKeyPathWrapper { + var root: Root + + subscript(dynamicMember keyPath: KeyPath) -> Member { + root[keyPath: keyPath] + } +} + +let dynamicCalculator = DynamicKeyPathWrapper(root: Calculator()) +let power = dynamicCalculator.power +print(power(10, 2)) +``` + +### Effectful value types + +Methods annotated with `nonisolated` and `consuming` are supported by this feature. However, noncopying root and value types [are not supported](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0437-noncopyable-stdlib-primitives.md#additional-future-work). `mutating`, `throws` and `async` are not supported for any other component type and will similarly not be supported for methods. Additionally keypaths cannot capture closure arguments that are not `Hashable`/`Equatable`. + +### Component chaining + +Component chaining between methods or from method to other key path types is also supported with this feature and will continue to behave as `Hashable`/`Equatable` types. + +```swift +let kp5 = \Calculator.subtract(this: 1).signum() +let kp6 = \Calculator.subtract(this: 2).description +``` + +## Source compatibility + +This feature has no effect on source compatibility. + +## ABI compatibility + +This feature does not affect ABI compatibility. + +## Implications on adoption + +This feature has no implications on adoption. + +## Future directions + +The effectful value types that are unsupported by this feature will all require new `KeyPath` types and so have been left out of this proposal. Additionally, this lack of support impacts existing key path component kinds and could be addressed in a unified proposal that resolves this gap across all key path component kinds. diff --git a/proposals/0480-swiftpm-warning-control.md b/proposals/0480-swiftpm-warning-control.md new file mode 100644 index 0000000000..c0faff43e7 --- /dev/null +++ b/proposals/0480-swiftpm-warning-control.md @@ -0,0 +1,329 @@ +# Warning Control Settings for SwiftPM + +* Proposal: [SE-0480](0480-swiftpm-warning-control.md) +* Authors: [Dmitrii Galimzianov](https://github.com/DmT021) +* Review Manager: [John McCall](https://github.com/rjmccall), [Franz Busch](https://github.com/FranzBusch) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift-package-manager#8315](https://github.com/swiftlang/swift-package-manager/pull/8315) +* Review: ([pitch](https://forums.swift.org/t/pitch-warning-control-settings-for-swiftpm/78666)) ([review](https://forums.swift.org/t/se-0480-warning-control-settings-for-swiftpm/79475)) ([returned for revision](https://forums.swift.org/t/se-0480-warning-control-settings-for-swiftpm/79475/8)) ([acceptance](https://forums.swift.org/t/accepted-se-0480-warning-control-settings-for-swiftpm/80327)) +* Previous Proposal: [SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md) + +## Introduction + +This proposal adds new settings to SwiftPM to control how the Swift, C, and C++ compilers treat warnings during the build process. It builds on [SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md), which introduced warning control flags for the Swift compiler but left SwiftPM support as a future direction. + +## Motivation + +The Swift Package Manager currently lacks a unified way to control warnings across Swift, C, and C++ compilation. This limitation forces developers to either use `unsafeFlags` or accept the default warning settings. + +## Proposed solution + +This proposal introduces new methods to SwiftPM's build settings API, allowing fine-grained control over warnings. + +### API + +#### Cross-language API (Swift, C, and C++) + +```swift +/// The level at which a compiler warning should be treated. +public enum WarningLevel: String { + /// Treat as a warning. + /// + /// Warnings will be displayed during compilation but will not cause the build to fail. + case warning + + /// Treat as an error. + /// + /// Warnings will be elevated to errors, causing the build to fail if any such warnings occur. + case error +} + +extension SwiftSetting { // Same for CSetting and CXXSetting + public static func treatAllWarnings( + as level: WarningLevel, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting // or CSetting or CXXSetting + + public static func treatWarning( + _ name: String, + as level: WarningLevel, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting // or CSetting or CXXSetting +} +``` + +#### C/C++-specific API + +In C/C++ targets, we can also enable or disable specific warning groups, in addition to controlling their severity. + +```swift +extension CSetting { // Same for CXXSetting + public static func enableWarning( + _ name: String, + _ condition: BuildSettingCondition? = nil + ) -> CSetting // or CXXSetting + + public static func disableWarning( + _ name: String, + _ condition: BuildSettingCondition? = nil + ) -> CSetting // or CXXSetting +} +``` +_The necessity of these functions is also explained below in the Alternatives considered section._ + +### Example usage + +```swift +.target( + name: "MyLib", + swiftSettings: [ + .treatAllWarnings(as: .error), + .treatWarning("DeprecatedDeclaration", as: .warning), + ], + cSettings: [ + .enableWarning("all"), + .disableWarning("unused-function"), + + .treatAllWarnings(as: .error), + .treatWarning("unused-variable", as: .warning), + ], + cxxSettings: [ + .enableWarning("all"), + .disableWarning("unused-function"), + + .treatAllWarnings(as: .error), + .treatWarning("unused-variable", as: .warning), + ] +) +``` + +## Detailed design + +### Settings and their corresponding compiler flags + +| Method | Swift | C/C++ | +|--------|-------|-------| +| `treatAllWarnings(as: .error)` | `-warnings-as-errors` | `-Werror` | +| `treatAllWarnings(as: .warning)` | `-no-warnings-as-errors` | `-Wno-error` | +| `treatWarning("XXXX", as: .error)` | `-Werror XXXX` | `-Werror=XXXX` | +| `treatWarning("XXXX", as: .warning)` | `-Wwarning XXXX` | `-Wno-error=XXXX` | +| `enableWarning("XXXX")` | N/A | `-WXXXX` | +| `disableWarning("XXXX")` | N/A | `-Wno-XXXX` | + +### Order of settings evaluation + +The order in which warning control settings are specified in a target's settings array directly affects the order of the resulting compiler flags. This is critical because when multiple flags affect the same warning group, compilers apply them sequentially with the last flag taking precedence. + +For example, consider these two different orderings for C++ settings: + +```swift +// Example 1: "unused-variable" in front of "unused" +cxxSettings: [ + .treatWarning("unused-variable", as: .error), + .treatWarning("unused", as: .warning), +] + +// Example 2: "unused" in front of "unused-variable" +cxxSettings: [ + .treatWarning("unused", as: .warning), + .treatWarning("unused-variable", as: .error), +] +``` + +In Example 1, the compiler will receive flags in this order: +``` +-Werror=unused-variable -Wno-error=unused +``` +Since "unused-variable" is a specific subgroup of the broader "unused" group, and the "unused" flag is applied last, all unused warnings (including unused-variable) will be treated as warnings. + +In Example 2, the compiler will receive flags in this order: +``` +-Wno-error=unused -Werror=unused-variable +``` +Due to the "last one wins" rule, unused-variable warnings will be treated as errors, while other unused warnings remain as warnings. + +The same principle applies when combining any of the new build settings: + +```swift +cxxSettings: [ + .enableWarning("all"), // Enable the "all" warning group + .enableWarning("extra"), // Enable the "extra" warning group + .disableWarning("unused-parameter"), // Disable the "unused-parameter" warning group + .treatAllWarnings(as: .error), // Treat all warnings as errors + .treatWarning("unused", as: .warning), // Keep warnings of the "unused" group as warnings +] +``` + +This will result in compiler flags: +``` +-Wall -Wextra -Wno-unused-parameter -Werror -Wno-error=unused +``` + +When configuring warnings, be mindful of the order to achieve the desired behavior. + +### Remote targets behavior + +When a target is remote (pulled from a package dependency rather than defined in the local package), the warning control settings specified in the manifest do not apply to it. SwiftPM will strip all of the warning control flags for remote targets and substitute them with options for suppressing warnings (`-w` for Clang and `-suppress-warnings` for Swift). + +This behavior is already in place but takes into account only `-warnings-as-errors` (for Swift) and `-Werror` (for Clang) flags. We expand this list to include the following warning-related flags: + +**For C/C++:** +* `-Wxxxx` +* `-Wno-xxxx` +* `-Werror` +* `-Werror=xxxx` +* `-Wno-error` +* `-Wno-error=xxxx` + +**For Swift:** +* `-warnings-as-errors` +* `-no-warnings-as-errors` +* `-Wwarning xxxx` +* `-Werror xxxx` + +This approach ensures that warning control settings are applied only to the targets you directly maintain in your package, while dependencies remain buildable without warnings regardless of their warning settings. + +### Interaction with command-line flags + +SwiftPM allows users to pass additional flags to the compilers using the `-Xcc`, `-Xswiftc`, and `-Xcxx` options with the `swift build` command. These flags are appended **after** the flags generated from the package manifest. + +This ordering enables users to modify or override package-defined warning settings without modifying the package manifest. + +#### Example + +```swift +let package = Package( + name: "MyExecutable", + targets: [ + // C target with warning settings + .target( + name: "cfoo", + cSettings: [ + .enableWarning("all"), + .treatAllWarnings(as: .error), + .treatWarning("unused-variable", as: .warning), + ] + ), + // Swift target with warning settings + .executableTarget( + name: "swiftfoo", + swiftSettings: [ + .treatAllWarnings(as: .error), + .treatWarning("DeprecatedDeclaration", as: .warning), + ] + ), + ] +) +``` + +When built with additional command-line flags: + +```sh +swift build -Xcc -Wno-error -Xswiftc -no-warnings-as-errors +``` + +The resulting compiler invocations will include both sets of flags: + +``` +# C compiler invocation +clang ... -Wall -Werror -Wno-error=unused-variable ... -Wno-error ... + +# Swift compiler invocation +swiftc ... -warnings-as-errors -Wwarning DeprecatedDeclaration ... -no-warnings-as-errors -Xcc -Wno-error ... +``` + +Flags are processed from left to right, and since `-no-warnings-as-errors` and `-Wno-error` apply globally to all warnings, they override the warning treating flags defined in the package manifest. + +#### Limitations + +This approach has a limitation when used with `-suppress-warnings`, which is mutually exclusive with other warning control flags: + +```sh +swift build -Xswiftc -suppress-warnings +``` + +Results in compiler errors: + +``` +error: conflicting options '-warnings-as-errors' and '-suppress-warnings' +error: conflicting options '-Wwarning' and '-suppress-warnings' +``` + + +## Security + +This change has no impact on security, safety, or privacy. + +## Impact on existing packages + +The proposed API will only be available to packages that specify a tools version equal to or later than the SwiftPM version in which this functionality is implemented. + +## Alternatives considered + +### Disabling a warning via a treat level + +Clang allows users to completely disable a specific warning, so for C/C++ settings we could implement that as a new case in the `WarningLevel` enum: + +```swift +public enum WarningLevel { + case warning + case error + case ignored +} +``` + +_(Since Swift doesn't allow selective warning suppression, we would actually have to split the enum into two: `SwiftWarningLevel` and `CFamilyWarningLevel`)_ + +But some warnings in Clang are disabled by default. If we simply pass `-Wno-error=unused-variable`, the compiler won't actually produce a warning for an unused variable. It only makes sense to use it if we have enabled the warning: `-Wunused-variable -Werror -Wno-error=unused-variable`. + +This necessitates separate functions to enable and disable warnings. Therefore, instead of `case ignored`, we propose the functions `enableWarning` and `disableWarning`. + +## Future directions + +### Package-level settings + +It has been noted that warning control settings are often similar across all targets. It makes sense to declare them at the package level while allowing target-level customizations. However, many other settings would also likely benefit from such inheritance, and SwiftPM doesn't currently provide such an option. Therefore, it was decided to factor this improvement out and look at all the settings holistically in the future. + +### Support for other C/C++ Compilers + +The C/C++ warning control settings introduced in this proposal are initially implemented with Clang's warning flag syntax as the primary target. However, the API itself is largely compiler-agnostic, and there's potential to extend support to other C/C++ compilers in the future. + +For instance, many of the proposed functions could be mapped to flags for other compilers like MSVC: + +| SwiftPM Setting | Clang | MSVC (Potential Mapping) | +| :-------------------------------- | :---------------- | :----------------------- | +| `.treatAllWarnings(as: .error)` | `-Werror` | `/WX` | +| `.treatAllWarnings(as: .warning)` | `-Wno-error` | `/WX-` | +| `.treatWarning("name", as: .error)`| `-Werror=name` | `/we####` (where `####` is MSVC warning code) | +| `.treatWarning("name", as: .warning)`| `-Wno-error=name` | No direct equivalent | +| `.enableWarning("name")` | `-Wname` | `/wL####` (e.g., `/w4####` to enable at level 4) | +| `.disableWarning("name")` | `-Wno-name` | `/wd####` | + +Where direct mappings are incomplete (like `.treatWarning(as: .warning)` for MSVC, which doesn't have a per-warning equivalent to Clang's `-Wno-error=XXXX`), SwiftPM could emit diagnostics indicating the setting is not fully supported by the current compiler. If more fine-grained control is needed for a specific compiler (e.g., MSVC's warning levels `0-4` for `enableWarning`), future enhancements could introduce compiler-specific settings or extend the existing API. + +A key consideration is the handling of warning names or codes (the `"name"` parameter in the API). SwiftPM does not maintain a comprehensive list of all possible warning identifiers and their mapping across different compilers. Instead, package authors would be responsible for providing the correct warning name or code for the intended compiler. + +To facilitate this, if support for other C/C++ compilers is added, the existing `BuildSettingCondition` API could be extended to allow settings to be applied conditionally based on the active C/C++ compiler. For example: + +```swift +cxxSettings: [ + // Clang-specific warning + .enableWarning("unused-variable", .when(cxxCompiler: .clang)), + // MSVC-specific warning (using its numeric code) + .enableWarning("4101", .when(cxxCompiler: .msvc)), + // Common setting that maps well + .treatAllWarnings(as: .error) +] +``` + +This approach, combined with the existing behavior where remote (dependency) packages have their warning control flags stripped and replaced with suppression flags, would allow projects to adopt new compilers. Even if a dependency uses Clang-specific warning flags, it would not cause build failures when the main project is built with a different compiler like MSVC, as those flags would be ignored. + +### Formalizing "Development-Only" Build Settings + +The warning control settings introduced by this proposal only apply when a package is built directly and are suppressed when the package is consumed as a remote dependency. + +During the review of this proposal, it was suggested that this "development-only" characteristic could be made more explicit, perhaps by introducing a distinct category of settings (e.g., `devSwiftSettings`). This is an interesting avenue for future exploration. SwiftPM already has a few other settings that exhibit similar behavior. A dedicated future proposal for "development-only" settings could address all such use cases holistically, providing a clearer and more general mechanism for package authors to distinguish between "dev-only" settings and those that propagate to consumers. + +## Acknowledgments + +Thank you to [Doug Gregor](https://github.com/douggregor) for the motivation, and to both [Doug Gregor](https://github.com/douggregor) and [Holly Borla](https://github.com/hborla) for their guidance during the implementation of this API. diff --git a/proposals/0481-weak-let.md b/proposals/0481-weak-let.md new file mode 100644 index 0000000000..ca885adc1c --- /dev/null +++ b/proposals/0481-weak-let.md @@ -0,0 +1,123 @@ +# `weak let` + +* Proposal: [SE-0481](0481-weak-let.md) +* Authors: [Mykola Pokhylets](https://github.com/nickolas-pohilets) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Accepted** +* Implementation: [swiftlang/swift#80440](https://github.com/swiftlang/swift/pull/80440) +* Review: ([discussion](https://forums.swift.org/t/weak-captures-in-sendable-sending-closures/78498)) ([pitch](https://forums.swift.org/t/pitch-weak-let/79271)) ([review](https://forums.swift.org/t/se-0481-weak-let/79603)) ([acceptance](https://forums.swift.org/t/accepted-se-0481-weak-let/79895)) + +[SE-0302]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md + +## Introduction + +Swift provides weak object references using the `weak` modifier on variables and stored properties. Weak references become `nil` when the object is destroyed, causing the value of the variable to seem to change. Swift has therefore always required `weak` references to be declared with the `var` keyword rather than `let`. However, that causes unnecessary friction with [sendability checking][SE-0302]: because weak references must be mutable, classes and closures with such references are unsafe to share between concurrent contexts. This proposal lifts that restriction and allows `weak` to be combined with `let`. + +## Motivation + +Currently, Swift classes with weak stored properties cannot be `Sendable`, because weak properties have to be mutable, and mutable properties are not allowed in `Sendable` classes: + +```swift +final class C: Sendable {} + +final class VarUser: Sendable { + weak var ref1: C? // error: stored property 'ref1' of 'Sendable'-conforming class 'VarUser' is mutable +} +``` + +Similarly, closures with explicit `weak` captures cannot be `@Sendable`, because such captures are implicitly *made* mutable, and `@Sendable` closures cannot capture mutable variables. This is surprising to most programmers, because every other kind of explicit capture is immutable. It is extremely rare for Swift code to directly mutate a `weak` capture. + +```swift +func makeClosure() -> @Sendable () -> Void { + let c = C() + return { [weak c] in + c?.foo() // error: reference to captured var 'c' in concurrently-executing code + + c = nil // allowed, but surprising and very rare + } +} +``` + +In both cases, allowing the weak reference to be immutable would solve the problem, but this is not currently allowed: + +```swift +final class LetUser: Sendable { + weak let ref1: C? // error: 'weak' must be a mutable variable, because it may change at runtime +} +``` + +The restriction that weak references have to be mutable is based on the idea that the reference is mutated when the referenced object is destroyed. Since it's mutated, it must be kept in mutable storage, and hence the storage must be declared with `var`. This way of thinking about weak references is problematic, however; it does not work very well to explain the behavior of weak references that are components of other values, such as `struct`s. For example, a return value is normally an immutable value, but a `struct` return value can contain a weak reference that may become `nil` at any point. + +In fact, wrapping weak references in a single-property `struct` is a viable workaround to the `var` restriction in both properties and captures: + +```swift +struct WeakRef { + weak var ref: C? +} + +final class WeakStructUser: Sendable { + let ref: WeakRef // ok +} + +func makeClosure() -> @Sendable () -> Void { + let c = C() + return { [c = WeakRef(ref: c)] in + c.ref?.foo() // ok + } +} +``` + +The existence of this simple workaround is itself an argument that the prohibition of `weak let` is not enforcing some fundamentally important rule. + +It is true that the value of a `weak` variable can be observed to change when the referenced object is destroyed. However, this does not have to be thought of as a mutation of the variable. A different way of thinking about it is that the variable continues to hold the same weak reference to the object, but that the program is simply not allowed to observe the object through that weak reference after the object is destroyed. This better explains the behavior of weak references in `struct`s: it's not that the destruction of the object changes the `struct` value, it's that the weak reference that's part of the `struct` value will now return `nil` if you try to observe it. + +Note that all of this relies on the fact that the thread-safety of observing a weak reference is fundamentally different from the thread-safety of assigning `nil` into a `weak var`. Swift's weak references are thread-safe against concurrent destruction: well-ordered reads and writes to a `weak var` or `weak let` will always behave correctly even if the referenced object is concurrently destroyed. But they are not *atomic* in the sense that writing to a `weak var` will behave correctly if another context is concurrently reading or writing to that same `var`. In this sense, a `weak var` is like any other `var`: mutations need to be well-ordered with all other accesses. + +## Proposed solution + +`weak` can now be freely combined with `let` in any position that `weak var` would be allowed. +Similar to `weak var`, `weak let` declarations also must be of `Optional` type. + +This proposal maintains the status quo regarding `weak` on function arguments and computed properties: +* There is no valid syntax to indicate that function argument is a weak reference. +* `weak` on computed properties is allowed, but has no effect. + +An explicit `weak` capture is now immutable under this proposal, like any other explicit capture. If the programmer really needs a mutable capture, they must capture a separate `weak var`: + +```swift +func makeClosure() -> @Sendable () -> Void { + let c = C() + // Closure is @Sendable + return { [weak c] in + c?.foo() + c = nil // error: cannot assign to value: 'c' is an immutable capture + } +} + +func makeNonSendableClosure() -> () -> Void { + let c = C() + weak var explicitlyMutable: C? = c + // Closure cannot be @Sendable anymore + return { + explicitlyMutable?.foo() + explicitlyMutable = nil // ok + } +} +``` + +## Source compatibility + +Allowing `weak let` bindings is an additive change that makes previously invalid code valid. It is therefore perfectly source-compatible. + +Treating weak captures as immutable is a source-breaking change. Any code that attempts to write to the capture will stop compiling. +The overall amount of such code is expected to be small. + +Since the captures of a closure are opaque and cannot be observed outside of the closure, changing the mutability of weak captures has no impact on clients of the closure. + +## ABI compatibility + +There is no ABI impact of this change. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. diff --git a/proposals/0482-swiftpm-static-library-binary-target-non-apple-platforms.md b/proposals/0482-swiftpm-static-library-binary-target-non-apple-platforms.md new file mode 100644 index 0000000000..5c7d90c92c --- /dev/null +++ b/proposals/0482-swiftpm-static-library-binary-target-non-apple-platforms.md @@ -0,0 +1,164 @@ +# Binary Static Library Dependencies + +* Proposal: [SE-0482](0482-swiftpm-static-library-binary-target-non-apple-platforms.md) +* Authors: [Daniel Grumberg](https://github.com/daniel-grumberg), [Max Desiatov](https://github.com/MaxDesiatov), [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Kuba Mracek](https://github.com/kubamracek) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift-package-manager#8639](https://github.com/swiftlang/swift-package-manager/pull/8639) [swiftlang/swift-package-manager#8741](https://github.com/swiftlang/swift-package-manager/pull/8741) +* Review: ([discussion](https://forums.swift.org/t/se-0482-binary-static-library-dependencies/79634)) ([pitch](https://forums.swift.org/t/pitch-swiftpm-support-for-binary-static-library-dependencies/78619)) ([acceptance](https://forums.swift.org/t/accepted-se-0482-binary-static-library-dependencies/80042)) +* Bugs: [Swift Package Manger Issue](https://github.com/swiftlang/swift-package-manager/issues/7035) + +## Introduction + +Swift continues to grow as a cross-platform language supporting a wide variety of use cases from [programming embedded device](https://www.swift.org/blog/embedded-swift-examples/) to [server-side development](https://www.swift.org/documentation/server/) across a multitude of [operating systems](https://www.swift.org/documentation/articles/static-linux-getting-started.html). +However, currently SwiftPM supports linking against binary dependencies on Apple platforms only. +This proposal aims to make it possible to provide static library dependencies exposing a C interface on non-Apple platforms that depend only on the standard C library. +The scope of this proposal is C libraries only, distributing Swift libraries has additional challenges (see [Future directions](#future-directions). + +## Motivation + +The Swift Package Manager’s [`binaryTarget` type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md) lets packages vend libraries that either cannot be built in Swift Package Manager for technical reasons, +or for which the source code cannot be published for legal or other reasons. + +In the current version of SwiftPM, binary targets support the following: + +* Libraries in an Xcode-oriented format called XCFramework, and only for Apple platforms, introduced in [SE-0272](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md). +* Executables through the use of artifact bundles introduced in [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md). + +We aim here to bring a subset of the XCFramework capabilities to non-Apple platforms in a safe way. + +While this proposal is specifically focused on binary static library dependencies without unexpected unresolved external symbols on non-Apple platforms, +it tries to do so in a way that will not prevent broader future support for static libraries and dynamically linked libraries. + +## Proposed solution + +This proposal extends artifact bundles introduced by [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md) to include a new kind of artifact type to represent a binary library dependency: `staticLibrary`. +The artifact manifest would encode the following information for each variant: + +* The static library to pass to the linker. + On Apple and Linux platforms, this would be `.a` files and on Windows it would be a `.lib` file. +* Enough information to be able to use the library's API in the packages source code, + i.e., headers and module maps for libraries exporting a C-based interface. + +Additionally, we propose the addition of an auditing tool that can validate the library artifact is safe to use across the Linux-based platforms supported by the Swift project. +Such a tool would ensure that people do not accidentally distribute artifacts that require dependencies that are not met on the various deployment platforms. +However when an artifact isn't widely consumed and all dependent packages are known, +artifact vendors can provide artifacts with dependencies on other C libraries provided that each client target depends explicitly on all required dependencies of the artifact. + +## Detailed design + +This section describes the changes to artifact bundle manifests in detail, the semantic impact of the changes on SwiftPM's build infrastructure, and describes the operation of the auditing tool. + +### Artifact Manifest Semantics + +The artifact manifest JSON format for a static library is described below: + +```json +{ + "schemaVersion": "1.0", + "artifacts": { + "": { + "version": "", + "type": "staticLibrary", + "variants": [ + { + "path": "", + "supportedTriples": ["", ... ], + "staticLibraryMetadata": { + "headerPaths": [", ...], + "moduleMapPath": "" + } + }, + ... + ] + }, + ... + } +} + +``` + +The additions are: + +* The `staticLibrary` artifact `type` that indicates this binary artifact is not an executable but rather a static library to link against. +* The `headerPaths` field specifies directory paths relative to the root of the artifact bundle that contain the header interfaces to the static library. + These are forwarded along to the swift compiler (or the C compiler) using the usual search path arguments. +* The optional `moduleMapPath` field specifies the path relative to the root of the artifact bundle that contains a custom module map to use if the header paths do not contain the module definitions or to provide custom overrides. + This field is required if the library's API is to be imported into Swift code. + +As with executable binary artifacts, the `path` field represents the relative path to the binary from the root of the artifact bundle, +and the `supportedTriples` field provides information about the target triples supported by this variant. + +An example artifact might look like: + +```json +{ + "schemaVersion": "1.0", + "artifacts": { + "example": { + "type": "staticLibrary", + "version": "1.0.0", + "variants": [ + { + "path": "libExample.a", + "supportedTriples": ["aarch64-unknown-linux-gnu"], + "staticLibraryMetadata": { + "headerPaths": ["include"], + "moduleMapPath": "include/example.modulemap" + } + } + ] + } + } +} +``` + +### Auditing tool + +Without proper auditing it would be very easy to provide binary static library artifacts that call into unresolved external symbols that are not available on the runtime platform, e.g., due to missing linkage to a system dynamic library. + +We propose the introduction of a new tool that can validate the "safety" of a binary library artifact across the platforms it supports and the corresponding runtime environment. + +In this proposal we restrict ourselves to static libraries that do not have any external dependencies beyond the C standard library and runtime. +To achieve this we need to be able to detect validate this property across the three object file formats used in static libraries on our supported platforms: [Mach-O](https://developer.apple.com/library/archive/documentation/Performance/Conceptual/CodeFootprint/Articles/MachOOverview.html#//apple_ref/doc/uid/20001860-BAJGJEJC) on Apple platforms, [ELF](https://refspecs.linuxfoundation.org/elf/elf.pdf) on Linux-based platforms, and [COFF](https://learn.microsoft.com/en-us/windows/win32/debug/pe-format) on Windows. +All three formats express references to external symbols as _relocations_ which reside in a single section of each object file. + +We propose adding the `llvm-objdump` to the toolchain to provide the capability to inspect relocations across all three supported object file formats. The tool would use `llvm-objdump` every object file in the static library and construct a complete list of symbols defined and referenced across the entire library. +Additionally, the tool would construct a simple C compiler invocation to derive to generate a default linker invocation. +This would be used to derive the libraries linked by default in a C program, these libraries would then be scanned to contribute to the list of defined symbols. +The tool would then check that the referenced symbols list is a subset of the set of defined symbols and emit an error otherwise. + +This would be sufficient to guarantee that all symbols from the static library would be available at runtime for statically linked executables or for ones running on the build host. +To ensure maximum runtime compatibility we would also provide a Linux-based Docker image that uses the oldest supported `glibc` for a given Swift version. +As `glibc` is backwards compatible, a container running the audit on a given static library would ensure that the version of `glibc` on any runtime platform would be compatible with the binary artifact. +This strategy as been successfully employed in the Python community with [`manylinux`](https://peps.python.org/pep-0513/). + +## Security + +This proposal brings the security implications outlined in [SE-0272](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md#security) to non-Apple platforms, +namely that a malicious attacker having access to both the server hosting the artifact and the git repository that vends the Package Manifest could provide a malicious library. +Users should exercise caution when onboarding binary dependencies. + +## Impact on existing packages + +No current package should be affected by this change since this is only an additive change in enabling SwiftPM to use binary target library dependencies on non-Apple platforms. + +## Future directions + +### Support Swift static libraries + +To do this we would extend the static library binary artifact manifest to provide a `.swiftinterface` file that can be consumed by the Swift compiler to import the Swift APIs. +Additionally we would extend the auditing tool to validate the usage of Swift standard library and runtime symbols, e.g., from `libSwiftCore`. + +### Extend binary compatibility guarantees + +This proposal limits itself to providing facilities for binary compatibility only with the C standard library and runtime. +In the future we could provide a system to allow binary artifact distributors to specify additional linkage dependencies for their binary artifacts. +These would be used to customize the operation of the audit tool and perform automatic linking of them in any client target that depends on the binary artifact, in the same way [CMake](https://cmake.org/cmake/help/v4.0/prop_tgt/INTERFACE_LINK_LIBRARIES.html) propagates link dependencies transitively. + +### Add support for dynamically linked dependencies + +On Windows dynamic linking requires an _import library_ which is a small static library that contains stubs for symbols exported by the dynamic library. +These stubs are roughly equivalent to a PLT entry in an ELF executable, but are generated during the build of the dynamic library and must be provided to clients of the library for linking purposes. +Similarly on Linux and Apple platforms binary artifact maintainers may wish to provide a dynamic library stub to improve link performance. +To support these use cases the library binary artifact manifest schema could be extended to provide facilities to provide both a link-time and runtime dependency. diff --git a/proposals/0483-inline-array-sugar.md b/proposals/0483-inline-array-sugar.md new file mode 100644 index 0000000000..507605ec34 --- /dev/null +++ b/proposals/0483-inline-array-sugar.md @@ -0,0 +1,205 @@ +# `InlineArray` Type Sugar + +* Proposal: [SE-0483](0483-inline-array-sugar.md) +* Authors: [Hamish Knight](https://github.com/hamishknight), [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 6.2)** +* Review: ([pitch](https://forums.swift.org/t/pitch-inlinearray-type-sugar/79142)) ([first review](https://forums.swift.org/t/se-0483-inlinearray-literal-syntax/79643)) ([second review](https://forums.swift.org/t/second-review-se-0483-inlinearray-type-sugar/80337)) ([acceptance](https://forums.swift.org/t/accepted-se-0483-inlinearray-type-sugar/81509)) + +## Introduction + +We propose the introduction of type sugar for the `InlineArray` type, providing more succinct syntax for declaring an inline array. + +## Motivation + +[SE-0453](/proposals/0453-vector.md) introduced a new type, `InlineArray`, which includes a size parameter as part of its type: + +``` +let fiveIntegers: InlineArray<5, Int> = .init(repeating: 99) +``` + +Declaring this type is more cumbersome than its equivalent dynamically-sized array, which has sugar for the type syntax: + +```swift +let fiveIntegers: [Int] = .init(repeating: 99, count: 5) +``` + +This becomes more pronounced when dealing with multiple dimensions: + +```swift +let fiveByFive: InlineArray<5, InlineArray<5, Int>> = .init(repeating: .init(repeating: 99)) +``` + +Almost every other language in a similar category to Swift – C, C++, Objective-C, Pascal, Go, Rust, Zig, Java, C# – has a simple syntax for their fixed-size array type. The introduction of a fixed-size array type into Swift should also introduce a shorthand syntax, in keeping with Swift's general approach of low ceremony and concise syntax. Swift further deviates from its peer languages by giving its _dynamic_ array type, `Array` (known in many other languages as `vector`) a sugared form. This can lead to an assumption that `Array` should be used under almost all circumstances, despite it having significant downsides in many uses (see further discussion in alternatives considered). + +## Proposed solution + +A new sugared version of the `InlineArray` type is proposed: + +```swift +let fiveIntegers: [5 of Int] = .init(repeating: 99) +``` + +The choice of `of` forms something close to a grammatical phrase ("an array of five ints"). A short contextual keyword is also in keeping with Swift's tradition in other areas such as `in` or `let`. + +## Detailed design + +The new syntax consists of the value for the integer generic parameter and the type of the element generic parameter, separated by `of`. + +This will be added to the grammar alongside the current type sugar: + +> **Grammar of a type** +> _type → sized-array-type_ +> +> **Grammar of a sized array type** +> _sized-array-type → [ expression `of` type ]_ + +Note that while the grammar allows for any expression, this is currently limited to only integer literals or integer type parameters, as required by the current implementation of `InlineArray`. If that restriction changes, so would the value allowed in the expression in the sugar. + +The new sugar is equivalent to declaring a type of `InlineArray`, so all rules that can be applied to the generic placeholders for the unsugared version also apply to the sugared version: + +```swift +// Nesting +let fiveByFive: InlineArray<5, InlineArray<5, Int>> = .init(repeating: .init(repeating: 99)) +let fiveByFive: [5 of [5 of Int]] = .init(repeating: .init(repeating: 99)) + +// Inference from context: +let fiveIntegers: [5 of _] = .init(repeating: 99) +let fourBytes: [_ of Int8] = [1,2,3,4] +let fourIntegers: [_ of _] = [1,2,3,4] + +// use on rhs +let fiveDoubles = [5 of _](repeating: 1.23) +``` + +The sugar can also be used in place of the unsugared type wherever it might appear: + +```swift +[5 of Int](repeating: 99) +MemoryLayout<[5 of Int]>.size +unsafeBitCast((1,2,3), to: [3 of Int].self) +``` + +There must be whitespace on either side of the separator; i.e., you cannot write `[5of Int]`. There are no requirements to balance whitespace; `[5 of Int]` is permitted. A new line can appear after the `of` but not before it, as while this is not ambiguous, this aids with the parser recovery logic, leading to better syntax error diagnostics. + +## Source Compatibility + +Since it is not currently possible to write any form of the proposed syntax in Swift today, this proposal does not alter the meaning of any existing code. + +## Impact on ABI + +This is purely compile-time sugar for the existing type. It is resolved at compile time and does not appear in the ABI nor rely on any version of the runtime. + +## Future Directions + +### Repeated value equivalent + +Analogous to arrays, there is an equivalent *value* sugar for literals of a specific size: + +```swift +// type inferred to be [5 of Int] +let fiveInts = [5 of 99] + +// type inferred to be [5 of [5 of Int]] +let fiveByFive = [5 of [5 of 99]] +``` + +Unlike the sugar for the type, this would also have applicability for existing types: + +```swift +// equivalent to .init(repeating: 99, count: 5) +let dynamic: [Int] = [5 of 99] +``` + +This is a much bigger design space, potentially requiring a new expressible-by-literal protocol and a way to map the literal to an initializer. As such, it is left for a future proposal. + +However, the choice of syntax for the type sugar has a significant impact on the viability of this future direction (see alternatives considered). Given the potential benefit of such a value syntax, any choice for the type sugar should consider its future extension to value sugar. + +[^expressions]: It could also cause confusion once expressions are allowed for declaring an `InlineArray` i.e. `[5 * 5 * Int]` would be allowed. + +### Flattened multi-dimensional arrays + +For multi-dimensional arrays, `[5 of [5 of Int]]` could be flattened to `[5 of 5 of Int]` without any additional parsing issues. This could be an alternative considered but is in future directions as it could also be introduced as sugar for the former case at a later date. + +## Alternatives Considered + +### Indication of the "inline" nature via the sugar. + +The naming of `InlineArray` incorporates important information about the nature of the type – that it includes its values inline rather than indirectly via a pointer. This name was chosen over other alternatives such as `FixedSizeArray` because the "inline-ness" was considered the more fundamental property, and so a better driver for the name. + +This has led to suggestions that this inline nature is important to include in the sugar was well. However, the current state privileges the position of `Array` as the only array type that is sugared. This implies that `Array` is the right choice in all circumstances, with inline arrays being a rare micro-optimization. This is not the case. + +For example, consider a translation of this code from the popular [Ray Tracing in One Weekend](https://raytracing.github.io/books/RayTracingInOneWeekend.html#thevec3class) tutorial: + +```cpp +class vec3 { + public: + double e[3]; + + double x() const { return e[0]; } + double y() const { return e[1]; } + double z() const { return e[2]; } + // etc +} +``` + +The way in which Swift privileges `[Double]` with sugar strongly implies you should use that in this translation. Doing so would have significant performance downsides: +- Creating new instances of `Vec3` requires a heap allocation, and destroying them require a free operation. +- `Vec3` could no longer be `BitwiseCopyable`, instead requiring a reference counting operation to make a copy. +- Access to a coordinate would require pointer chasing, and a contiguous array of `Vec3` objects would not be guaranteed to exist in contiguous memory. +- Every access to those coordinate accessors would need to check the bounds (because while the author might ensure that the value of `e` will only ever have length 3, the compiler cannot easily know this) and, in the case of mutation, a check for uniqueness of the pointer. `InlineArray<3, Double>` has none of these problems. + +Other examples include the use of nested `Array` types i.e. using `[[Double]]` to represent a matrix of known size, which would also have noticeable negative performance impact depending on the use case, compared to using an `InlineArray>` (or perhaps `[InlineArray]`) to model the same values. Today, this is possible via custom subscripts that logically represent the inner array within a single `[Double]`, but the introduction of `InlineArray` introduces other potentially more ergonomic options. + +In other cases, the "copy on write" nature of `Array` can mislead users into thinking that copies are risk-free, when actually copying an array can lead to "defeating" copy on write in subtle ways that can cause difficult-to-hunt-down performance issues. In all these cases, you need to pick the right one of two options for the performance goals you are trying to achieve. + +Swift's choice (deviating from many of its peers) to only have a dynamic array type, and to emphasize the utility of this type through sugar, has led to a shaping of the culture of writing Swift code that favors the sugared dynamic array even when this leads to otherwise-avoidable negative performance impact. This is not intended to make the case that Swift should _not_ have this sugar. Dynamic arrays are widely useful and Swift's readability goals are improved by Swift having a concise syntax for creating them. But writing performant Swift code inevitably involves having an understanding of the underlying performance characteristics of _all_ the types you are using, and syntax or type naming alone cannot solve this. + +Of course, all this only matters when you are trying to write code that maximizes performance. But that is a really important use case for Swift. The goal for Swift is a language that is as safe and enjoyable to write as many high-level non-performant languages, but also can achieve peak performance when that is your goal. And the idea is that when you are targeting that level of performance, you don't have to go into "ugly, no longer nice swift" mode to do it, with nice sugared `[Double]` replaced with less pleasant full type name of `InlineArray` – something a user coming from Go or C++ or Rust might find a downgrade. Similarly, attempting to incorporate the word "inline" into the sugar e.g. `[5 inline Int]` creates a worst of both worlds solution that many would find offputting to use, without solving the fundamental issue. + +For these reasons, we should be considering `InlineArray` a peer of `Array` (even if the need for it is less common – just not "niche"), and providing a pleasant to use sugar for both types. + +### Choice of delimiter + +The most obvious alternative here is the choice of separator. Other options include: + +- `[5 by Int]` is similar to `of`, but is less applicable to the value syntax (`[5 by 5]` doesn't read as an array of 5 instances of 5), without being clearer for the type syntax. +- `[5 x Int]`, using the ascii letter `x`, as an approximation for multiplication, reflecting common uses such as "a 4x4 vehicle". This was the choice of a previous revision of this proposal. +- `[5 * Int]`, using the standard ASCII symbol for multiplication. +- `[5 ⨉ Int]`, the Unicode n-ary times operator. This looks nice but is impractical as not keyboard-accessible. +- `[5; Int]` is what Rust uses, but appears to have little association with "times" or "many". Similarly other arbitrary punctuation e.g. `,` or `/`. `:` is of course ruled out as it is used for dictionary literals. +- `#` does have an association with counts in some areas such as set theory, but is used as a prefix operator rather than infix i.e. `[#5 Int]`. This is less expected than the infix form, and could also be read as "the fifth `Int`". It is also unclear how this would work with expressions like an array of size `5*5`. +- No delimiter at all i.e. `[5 Int]`. While this might be made to parse, the lack of any separator is found unsettling by some users and is less visually clear, especially once expressions are allowed instead of the `5`. + +Note that `*` is an existing operator, and may lead to ambiguity in future when expressions can be used to determine the size: `[5 * N * Int]`. `of` is clearer in this case: `[5 * N of Int]`. It also avoids parsing ambiguity, as the grammar does not allow two identifiers in succession. This becomes more important if the future direction of a value equivalent is pursued. `[2 * 2 * 2]` could be interpreted as `[2, 2, 2, 2]`, `[4, 4,]`, or `[8]`. + +Since `of` cannot follow another identifier today, `[of of Int]` is unambiguous,[^type] but would clearly be hard to read. This is likely a hypothetical concern rather than a practical one since `of` is very rare as a variable name. The previous proposal's use of `x` involved a variable name that was more common. + +`x` is also less clear when used for the value version: `[5 x 5]` can be parsed unambiguously, but looks similar to five times five, and so visually has the same challenges as with the `*` operator, even if this isn't a problem for the compiler. + +[^type]: or even `[of of of]`, since `of` can be a type name, albeit one that defies Swift's naming conventions. + +Another thing to consider is how that separator looks in the fully inferred version, which tend to start to look a little like ascii diagrams: + +``` +[_ of _] +[_ x _] +[_ * _] +[_; _] +``` + +Of all these, the `of` choice is less susceptible to the ascii art problem. + +### Order of size and type + +The order of size first, then type is determined by the ordering of the unsugared type, and deviating from this for the sugared version is not an option. + +### Whitespace around the delimeter + +In theory, when using integer literals or `_` the whitespace could be omitted (`[5x_]` is unambiguously `[5 x _]`). However, special casing allowing whitespace omission is not desirable. + +### Choice of brackets + +`InlineArray` has a lot in common with tuples – especially in sharing "copy on copy" behavior, unlike regular `Array`. So `(5 of Int)` may be an appropriate alternative to the square brackets, echoing this similarity. However, tuples and `InlineArray`s remain very different types, and it could be misleading to imply that `InlineArray` is "just" a tuple. + +Beyond varying the separator, there may be other dramatically different syntax that moves further from the "like Array sugar, but with a size argument". For example, dropping the brackets altogether (i.e. `let a: 5 of Int`). However, these are probably too much of a departure from the current Swift idioms, so likely to cause further confusion without any real upside. diff --git a/proposals/0484-allow-additional-args-to-dynamicmemberlookup-subscripts.md b/proposals/0484-allow-additional-args-to-dynamicmemberlookup-subscripts.md new file mode 100644 index 0000000000..744cebb58b --- /dev/null +++ b/proposals/0484-allow-additional-args-to-dynamicmemberlookup-subscripts.md @@ -0,0 +1,281 @@ +# Allow Additional Arguments to `@dynamicMemberLookup` Subscripts + +* Proposal: [SE-0484](0484-allow-additional-args-to-dynamicmemberlookup-subscripts.md) +* Authors: [Itai Ferber](https://github.com/itaiferber) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Accepted** +* Implementation: [swiftlang/swift#81148](https://github.com/swiftlang/swift/pull/81148) +* Previous Proposals: [SE-0195](0195-dynamic-member-lookup.md), [SE-0252](0252-keypath-dynamic-member-lookup.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-allow-additional-arguments-to-dynamicmemberlookup-subscripts/79558)) ([review](https://forums.swift.org/t/se-0484-allow-additional-arguments-to-dynamicmemberlookup-subscripts/79853)) ([acceptance](https://forums.swift.org/t/accepted-se-0484-allow-additional-arguments-to-dynamicmemberlookup-subscripts/80167)) + +## Introduction + +SE-0195 and SE-0252 introduced and refined `@dynamicMemberLookup` to provide type-safe "dot"-syntax access to arbitrary members of a type by reflecting the existence of certain `subscript(dynamicMember:)` methods on that type, turning + +```swift +let _ = x.member +x.member = 42 +ƒ(&x.member) +``` + +into + +```swift +let _ = x[dynamicMember: ] +x[dynamicMember: ] = 42 +ƒ(&x[dynamicMember: ]) +``` + +when `x.member` doesn't otherwise exist statically. Currently, in order to be eligible to satisfy `@dynamicMemberLookup` requirements, a subscript must: + +1. Take _exactly one_ argument with an explicit `dynamicMember` argument label, +2. Whose type is non-variadic and is either + * A `{{Reference}Writable}KeyPath`, or + * A concrete type conforming to `ExpressibleByStringLiteral` + +This proposal intends to relax the "exactly one" requirement above to allow eligible subscripts to take additional arguments after `dynamicMember` as long as they have a default value (or are variadic, and thus have an implicit default value). + +## Motivation + +Dynamic member lookup is often used to provide expressive and succinct API in wrapping some underlying data, be it a type-erased foreign language object (e.g., a Python `PyVal` or a JavaScript `JSValue`) or a native Swift type. This (and [`callAsFunction()`](0253-callable.md)) allow a generalized API interface such as + +```swift +struct Value { + subscript(_ property: String) -> Value { + get { ... } + set { ... } + } + + func invoke(_ method: String, _ args: Any...) -> Value { + ... + } +} + +let x: Value = ... +let _ = x["member"] +x["member"] = Value(42) +x.invoke("someMethod", 1, 2, 3) +``` + +to be expressed much more naturally: + +```swift +@dynamicMemberLookup +struct Value { + struct Method { + func callAsFunction(_ args: Any...) -> Value { ... } + } + + subscript(dynamicMember property: String) -> Value { + get { ... } + set { ... } + } + + subscript(dynamicMember method: String) -> Method { ... } +} + +let x: Value = ... +let _ = x.member +x.member = Value(42) +x.someMethod(1, 2, 3) +``` + +However, as wrappers for underlying data, sometimes interfaces like this need to be able to "thread through" additional information. For example, it might be helpful to provide information about call sites for debugging purposes: + +```swift +struct Value { + subscript( + _ property: String, + function: StaticString = #function, + file: StaticString = #fileID, + line: UInt = #line + ) -> Value { + ... + } + + func invokeMethod( + _ method: String, + function: StaticString = #function, + file: StaticString = #fileID, + line: UInt = #line, + _ args: Any... + ) -> Value { + ... + } +} +``` + +When additional arguments like this have default values, they don't affect the appearance of call sites at all: + +```swift +let x: Value = ... +let _ = x["member"] +x["member"] = Value(42) +x.invoke("someMethod", 1, 2, 3) +``` + +However, these are not valid for use with dynamic member lookup subscripts, since the additional arguments prevent subscripts from being eligible for dynamic member lookup: + +```swift +@dynamicMemberLookup // error: @dynamicMemberLookupAttribute requires 'Value' to have a 'subscript(dynamicMember:)' method that accepts either 'ExpressibleByStringLiteral' or a key path +struct Value { + subscript( + dynamicMember property: String, + function: StaticString = #function, + file: StaticString = #fileID, + line: UInt = #line + ) -> Value { + ... + } + + subscript( + dynamicMember method: String, + function: StaticString = #function, + file: StaticString = #fileID, + line: UInt = #line + ) -> Method { + ... + } +} +``` + +## Proposed solution + +We can amend the rules for such subscripts to make them eligible. With this proposal, in order to be eligible to satisfy `@dynamicMemberLookup` requirements, a subscript must: + +1. Take an initial argument with an explicit `dynamicMember` argument label, +2. Whose parameter type is non-variadic and is either: + * A `{{Reference}Writable}KeyPath`, or + * A concrete type conforming to `ExpressibleByStringLiteral`, +3. And whose following arguments (if any) are all either variadic or have a default value + +## Detailed design + +Since compiler support for dynamic member lookup is already robust, implementing this requires primarily: + +1. Type-checking of `@dynamicMemberLookup`-annotated declarations to also consider `subscript(dynamicMember:...)` methods following the above rules as valid, and +2. Syntactic transformation of `T.` to `T[dynamicMember:...]` in the constraint system to fill in default arguments expressions for any following arguments + +## Source compatibility + +This is largely an additive change with minimal impact to source compatibility. Types which do not opt in to `@dynamicMemberLookup` are unaffected, as are types which do opt in and only offer `subscript(dynamicMember:)` methods which take a single argument. + +However, types which opt in to `@dynamicMemberLookup` and currently offer an overload of `subscript(dynamicMember:...)`—which today is not eligible for consideration for dynamic member lookup—_may_ now select this overload when they wouldn't have before. + +### Overload resolution + +Dynamic member lookups go through regular overload resolution, with an additional disambiguation rule that prefers keypath-based subscript overloads over string-based ones. Since the `dynamicMember` argument to dynamic member subscripts is implicit, overloads of `subscript(dynamicMember:)` are primarily selected based on their return type (and typically for keypath-based subscripts, how that return type is used in forming the type of a keypath parameter). + +With this proposal, all arguments to `subscript(dynamicMember:...)` are still implicit, so overloads are still primarily selected based on return type, with the additional disambiguation rule that prefers overloads with fewer arguments over overloads with more arguments. (This rule applies "for free" since it already applies to method calls, which dynamic member lookups are transformed into.) + +This means that if a type today offers a valid `subscript(dynamicMember:) -> T` and a (currently-unconsidered) `subscript(dynamicMember:...) -> U`, + +1. If `T == U` then the former will still be the preferred overload in all circumstances +2. If `T` and `U` are compatible (and equally-specific) at a callsite then the former will still be the preferred overload +3. If `T` and `U` are incompatible, or if one is more specific than the other, then the more specific type will be preferred + +For example: + +```swift +@dynamicMemberLookup +struct A { + /* (1) */ subscript(dynamicMember member: String) -> String { ... } + /* (2) */ subscript(dynamicMember member: String, _: StaticString = #function) -> String { ... } +} + +@dynamicMemberLookup +struct B { + /* (3) */ subscript(dynamicMember member: String) -> String { ... } + /* (4) */ subscript(dynamicMember member: String, _: StaticString = #function) -> Int { ... } +} + +@dynamicMemberLookup +struct C { + /* (5) */ subscript(dynamicMember member: String) -> String { ... } + /* (6) */ subscript(dynamicMember member: String, _: StaticString = #function) -> String? { ... } +} + +// T == U +let _ = A().member // (1) preferred over (2); no ambiguity +let _: String = A().member // (1) preferred over (2); no ambiguity + +// T and U are compatible +let _: Any = A().member // (1) preferred over (2); no ambiguity +let _: Any = B().member // (3) preferred over (4); no ambiguity +let _: Any = C().member // (5) preferred over (6); no ambiguity + +// T and U are incompatible/differently-specific +let _: String = B().member // (3) +let _: Int = B().member // (4);️ would not previously compile +let _: String = C().member // (5); no ambiguity +let _: String? = C().member // (6) preferred over (5); ⚠️ previously (5) ⚠️ +``` + +This last case is the only source of behavior change: (6) was previously not considered a valid candidate, but has a return type more specific than (5), and is now picked at a callsite. + +In practice, it is expected that this situation is exceedingly rare. + +## ABI compatibility + +This feature is implemented entirely in the compiler as a syntactic transformation and has no impact on the ABI. + +## Implications on adoption + +The changes in this proposal require the adoption of a new version of the Swift compiler. + +## Alternatives considered + +The main alternative to this proposal is to not implement it, as: +1. It was noted in [the pitch thread](https://forums.swift.org/t/pitch-allow-additional-arguments-to-dynamicmemberlookup-subscripts/79558) that allowing additional arguments to dynamic member lookup widens the gap in capabilities between dynamic members and regular members — dynamic members would be able to + + 1. Have caller side effects (i.e., have access to `#function`, `#file`, `#line`, etc.), + 2. Constrain themselves via generics, and + 3. Apply isolation to themselves via `#isolation` + + where regular members cannot. However, (i) and (iii) are not considered an imbalance in functionality but instead are the raison d'être of this proposal. (ii) is also already possible today as dynamic member subscripts can be constrained via generics (and this is often used with keypath-based lookup). +2. This is possible to work around using explicit methods such as `get()` and `set(_:)`: + + ```swift + @dynamicMemberLookup + struct Value { + struct Property { + func get( + function: StaticString = #function, + file: StaticString = #file, + line: UInt = #line + ) -> Value { + ... + } + + func set( + _ value: Value, + function: StaticString = #function, + file: StaticString = #file, + line: UInt = #line + ) { + ... + } + } + + subscript(dynamicMember member: String) -> Property { ... } + } + + let x: Value = ... + let _ = x.member.get() // x.member + x.member.set(Value(42)) // x.member = Value(42) + ``` + + However, this feels non-idiomatic, and for long chains of getters and setters, can become cumbersome: + + ```swift + let x: Value = ... + let _ = x.member.get().inner.get().nested.get() // x.member.inner.nested + x.member.get().inner.get().nested.set(Value(42)) // x.member.inner.nested = Value(42) + ``` + +### Source compatibility + +It is possible to avoid the risk of the behavior change noted above by adjusting the constraint system to always prefer `subscript(dynamicMember:) -> T` overloads over `subscript(dynamicMember:...) -> U` overloads (if `T` and `U` are compatible), even if `U` is more specific than `T`. However, + +1. This would be a departure from the normal method overload resolution behavior that Swift developers are familiar with, and +2. If `T` were a supertype of `U`, it would be impossible to ever call the more specific overload except by direct subscript access diff --git a/proposals/0485-outputspan.md b/proposals/0485-outputspan.md new file mode 100644 index 0000000000..76f467f6fa --- /dev/null +++ b/proposals/0485-outputspan.md @@ -0,0 +1,861 @@ +# OutputSpan: delegate initialization of contiguous memory + +* Proposal: [SE-0485](0485-outputspan.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.2)** ([Extensions to standard library types](#extensions) pending) +* Roadmap: [BufferView Roadmap](https://forums.swift.org/t/66211) +* Implementation: [swiftlang/swift#81637](https://github.com/swiftlang/swift/pull/81637) +* Review: [Pitch](https://forums.swift.org/t/pitch-outputspan/79473), [Review](https://forums.swift.org/t/se-0485-outputspan-delegate-initialization-of-contiguous-memory/80032), [Acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0485-outputspan-delegate-initialization-of-contiguous-memory/80435) + +[SE-0446]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md +[SE-0447]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md +[SE-0456]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0456-stdlib-span-properties.md +[SE-0467]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0467-MutableSpan.md +[PR-LifetimeAnnotations]: https://github.com/swiftlang/swift-evolution/pull/2750 +[Forum-LifetimeAnnotations]: https://forums.swift.org/t/78638 +[SE-0453]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md + + +#### Table of Contents + +- [Introduction](#introduction) + +- [Motivation](#motivation) + +- [Proposed solution](#proposed-solution) + +- [Detailed Design](#design) + +- [Source compatibility](#source-compatibility) + +- [ABI compatibility](#abi-compatibility) + +- [Implications on adoption](#implications-on-adoption) + +- [Alternatives Considered](#alternatives-considered) + +- [Future directions](#future-directions) + +- [Acknowledgements](#acknowledgements) + + +## Introduction + +Following the introduction of [`Span`][SE-0447] and [`MutableSpan`][SE-0467], this proposal adds a general facility for initialization of exclusively-borrowed memory with the `OutputSpan` and `OutputRawSpan` types. The memory represented by `OutputSpan` consists of a number of initialized elements, followed by uninitialized memory. The operations of `OutputSpan` can change the number of initialized elements in memory, unlike `MutableSpan` which always represent initialized memory representing a fixed number of elements. + +## Motivation + +Some standard library container types can delegate initialization of some or all of their storage to user code. Up to now, it has only been possible to do so with explicitly unsafe functions, which have also proven error-prone. The standard library provides this unsafe functionality with the closure-taking initializers `Array.init(unsafeUninitializedCapacity:initializingWith:)` and `String.init(unsafeUninitializedCapacity:initializingUTF8With:)`. + +These functions have a few different drawbacks, most prominently their reliance on unsafe types, which makes them unpalatable in security-conscious environments. We continue addressing these issues with `OutputSpan` and `OutputRawSpan`, new non-copyable and non-escapable types that manage initialization of typed and untyped memory. + +In addition to the new types, we propose adding new API for some standard library types to take advantage of `OutputSpan` and `OutputRawSpan`. + +## Proposed solution + +#### OutputSpan + +`OutputSpan` allows delegating the initialization of a type's memory, by providing access to an exclusively-borrowed view of a range of contiguous memory. `OutputSpan`'s contiguous memory always consists of a prefix of initialized memory, followed by a suffix of uninitialized memory. `OutputSpan`'s operations manage the initialization state in order to preserve that invariant. The common usage pattern we expect to see for `OutputSpan` consists of passing it as an `inout` parameter to a function, allowing the function to produce an output by writing into a previously uninitialized region. + +Like `MutableSpan`, `OutputSpan` relies on two guarantees: (a) that it has exclusive access to the range of memory it represents, and (b) that the memory locations it represents will remain valid for the duration of the access. These guarantee data race safety and lifetime safety. `OutputSpan` performs bounds-checking on every access to preserve bounds safety. `OutputSpan` manages the initialization state of the memory in represents on behalf of the memory's owner. + +#### OutputRawSpan + +`OutputRawSpan` allows delegating the initialization of heterogeneously-typed memory, such as memory being prepared by an encoder. It makes the same safety guarantees as `OutputSpan`, but manages untyped memory. + +#### Extensions to standard library types + +The standard library will provide new container initializers that delegate to an `OutputSpan`. Delegated initialization generally requires a container to perform some operations after the initialization has happened. In the case of `Array` this is simply noting the number of initialized elements; in the case of `String` this consists of validating the input, then noting metadata about the input. This post-processing implies the need for a scope, and we believe that scope is best represented by a closure. The `Array` initializer will be as follows: + +```swift +extension Array { + public init( + capacity: Int, + initializingWith: (_ span: inout OutputSpan) throws(E) -> Void + ) throws(E) +} +``` + +We will also extend `String`, `UnicodeScalarView` and `InlineArray` with similar initializers, and add append-in-place operations where appropriate. + +#### `@_lifetime` attribute + +Some of the API presented here must establish a lifetime relationship between a non-escapable returned value and a callee binding. This relationship will be illustrated using the `@_lifetime` attribute recently [pitched][PR-LifetimeAnnotations] and [formalized][Forum-LifetimeAnnotations]. For the purposes of this proposal, the lifetime attribute ties the lifetime of a function's return value to one of its input parameters. + +Note: The eventual lifetime annotations proposal may adopt a syntax different than the syntax used here. We expect that the Standard Library will be modified to adopt an updated lifetime dependency syntax as soon as it is finalized. + +## Detailed Design + +#### OutputSpan + +`OutputSpan` is a simple representation of a partially-initialized region of memory. It is non-copyable in order to enforce exclusive access during mutations of its memory, as required by the law of exclusivity: + +````swift +@frozen +public struct OutputSpan: ~Copyable, ~Escapable { + internal let _start: UnsafeMutableRawPointer? + public let capacity: Int + internal var _count: Int +} +```` + +The memory represented by an `OutputSpan` instance consists of `count` initialized instances of `Element`, followed by uninitialized memory with storage space for `capacity - count` additional elements of `Element`. + +```swift +extension OutputSpan where Element: ~Copyable { + /// The number of initialized elements in this `OutputSpan`. + public var count: Int { get } + + /// A Boolean value indicating whether the span is empty. + public var isEmpty: Bool { get } + + /// A Boolean value indicating whether the span is full. + public var isFull: Bool { get } + + /// The number of additional elements that can be added to this `OutputSpan` + public var freeCapacity: Int { get } // capacity - count +} +``` + +##### Single-element operations + +The basic operation supported by `OutputSpan` is appending an element. When an element is appended, the correct amount of memory needed to represent it is initialized, and the `count` property is incremented by 1. If the `OutputSpan` has no available space (`capacity == count`), this operation traps. +```swift +extension OutputSpan where Element: ~Copyable { + /// Append a single element to this `OutputSpan`. + @_lifetime(self: copy self) + public mutating func append(_ value: consuming Element) +} + +extension OutputSpan { + /// Repeatedly append an element to this `OutputSpan`. + @_lifetime(self: copy self) + public mutating func append(repeating repeatedValue: Element, count: Int) +} +``` +The converse operation `removeLast()` is also supported, and returns the removed element if `count` was greater than zero. +```swift +extension OutputSpan where Element: ~Copyable { + /// Remove the last initialized element from this `OutputSpan`. + /// + /// Returns the last element. The `OutputSpan` must not be empty. + @discardableResult + @_lifetime(self: copy self) + public mutating func removeLast() -> Element +} +``` + +##### Bulk removals from an `OutputSpan`'s memory: + +Bulk operations to deinitialize some or all of an `OutputSpan`'s memory are also available: +```swift +extension OutputSpan where Element: ~Copyable { + /// Remove the last N elements, returning the memory they occupy + /// to the uninitialized state. + /// + /// `n` must not be greater than `count` + @_lifetime(self: copy self) + public mutating func removeLast(_ n: Int) + + /// Remove all this span's elements and return its memory to the uninitialized state. + @_lifetime(self: copy self) + public mutating func removeAll() +} +``` + +##### Accessing an `OutputSpan`'s initialized memory: + +The initialized elements are accessible for read-only or mutating access via the `span` and `mutableSpan` properties: + +```swift +extension OutputSpan where Element: ~Copyable { + /// Borrow the underlying initialized memory for read-only access. + public var span: Span { + @_lifetime(borrow self) borrowing get + } + + /// Exclusively borrow the underlying initialized memory for mutation. + public mutating var mutableSpan: MutableSpan { + @_lifetime(&self) mutating get + } +} +``` + +`OutputSpan` also provides the ability to access its individual initialized elements by index: +```swift +extension OutputSpan where Element: ~Copyable { + /// The type that represents an initialized position in an `OutputSpan`. + typealias Index = Int + + /// The range of initialized positions for this `OutputSpan`. + var indices: Range { get } + + /// Accesses the element at the specified initialized position. + subscript(_ index: Index) -> Element { borrow; mutate } + // accessor syntax from accessors roadmap (https://forums.swift.org/t/76707) + + /// Exchange the elements at the two given offsets + mutating func swapAt(_ i: Index, _ j: Index) +} +``` + +##### Interoperability with unsafe code + +We provide a method to process or populate an `OutputSpan` using unsafe operations, which can also be used for out-of-order initialization. + +```swift +extension OutputSpan where Element: ~Copyable { + /// Call the given closure with the unsafe buffer pointer addressed by this + /// OutputSpan and a mutable reference to its count of initialized elements. + /// + /// This method provides a way to process or populate an `OutputSpan` using + /// unsafe operations, such as dispatching to code written in legacy + /// (memory-unsafe) languages. + /// + /// The supplied closure may process the buffer in any way it wants; however, + /// when it finishes (whether by returning or throwing), it must leave the + /// buffer in a state that satisfies the invariants of the output span: + /// + /// 1. The inout integer passed in as the second argument must be the exact + /// number of initialized items in the buffer passed in as the first + /// argument. + /// 2. These initialized elements must be located in a single contiguous + /// region starting at the beginning of the buffer. The rest of the buffer + /// must hold uninitialized memory. + /// + /// This function cannot verify these two invariants, and therefore + /// this is an unsafe operation. Violating the invariants of `OutputSpan` + /// may result in undefined behavior. + @_lifetime(self: copy self) + public mutating func withUnsafeMutableBufferPointer( + _ body: ( + UnsafeMutableBufferPointer, + _ initializedCount: inout Int + ) throws(E) -> R + ) throws(E) -> R +} +``` + +##### Creating an `OutputSpan` instance: + +Creating an `OutputSpan` is an unsafe operation. It requires having knowledge of the initialization state of the range of memory being targeted. The range of memory must be in two regions: the first region contains initialized instances of `Element`, and the second region is uninitialized. The number of initialized instances is passed to the `OutputSpan` initializer through its `initializedCount` argument. + +```swift +extension OutputSpan where Element: ~Copyable { + /// Unsafely create an OutputSpan over partly-initialized memory. + /// + /// The memory in `buffer` must remain valid throughout the lifetime + /// of the newly-created `OutputSpan`. Its prefix must contain + /// `initializedCount` initialized instances, followed by uninitialized + /// memory. + /// + /// - Parameters: + /// - buffer: an `UnsafeMutableBufferPointer` to be initialized + /// - initializedCount: the number of initialized elements + /// at the beginning of `buffer`. + @unsafe + @_lifetime(borrow buffer) + public init( + buffer: UnsafeMutableBufferPointer, + initializedCount: Int + ) +} + +extension OutputSpan { + /// Unsafely create an OutputSpan over partly-initialized memory. + /// + /// The memory in `buffer` must remain valid throughout the lifetime + /// of the newly-created `OutputSpan`. Its prefix must contain + /// `initializedCount` initialized instances, followed by uninitialized + /// memory. + /// + /// - Parameters: + /// - buffer: an `UnsafeMutableBufferPointer` to be initialized + /// - initializedCount: the number of initialized elements + /// at the beginning of `buffer`. + @unsafe + @_lifetime(borrow buffer) + public init( + buffer: borrowing Slice>, + initializedCount: Int + ) +} +``` + +We also provide a default (no-parameter) initializer to create an empty, zero-capacity `OutputSpan`. Such an initializer is useful in order to be able to materialize an empty span for the `nil` case of an Optional, for example, or to exchange with another span in a mutable struct. + +```swift +extension OutputSpan where Element: ~Copyable { + /// Create an OutputSpan with zero capacity + @_lifetime(immortal) + public init() +} +``` + +Such an empty `OutputSpan` does not depend on a memory allocation, and therefore has the longest lifetime possible :`immortal`. This capability is important enough that we also propose to immediately define similar empty initializers for `Span`, `RawSpan`, `MutableSpan` and `MutableRawSpan`. + +##### Retrieving initialized memory from an `OutputSpan` + +Once memory has been initialized using `OutputSpan`, the owner of the memory must consume the `OutputSpan` in order to retake ownership of the initialized memory. The owning type must pass the memory used to initialize the `OutputSpan` to the `finalize(for:)` function. Passing the wrong buffer is a programmer error and the function traps; this requirement also ensures that user code does not wrongly replace the `OutputSpan` with an unrelated instance. The `finalize(for:)` function consumes the `OutputSpan` instance and returns the number of initialized elements. If `finalize(for:)` is not called, the initialized portion of `OutputSpan`'s memory will be deinitialized when the binding goes out of scope. + +```swift +extension OutputSpan where Element: ~Copyable { + /// Consume the OutputSpan and return the number of initialized elements. + /// + /// Parameters: + /// - buffer: The buffer being finalized. This must be the same buffer as used + /// to initialize the `OutputSpan` instance. + /// Returns: The number of elements that were initialized. + @unsafe + public consuming func finalize( + for buffer: UnsafeMutableBufferPointer + ) -> Int +} + +extension OutputSpan { + /// Consume the OutputSpan and return the number of initialized elements. + /// + /// Parameters: + /// - buffer: The buffer being finalized. This must be the same buffer as used + /// to initialize the `OutputSpan` instance. + /// Returns: The number of bytes that were initialized. + @unsafe + public consuming func finalize( + for buffer: Slice> + ) -> Int +} +``` + + +#### `OutputRawSpan` +`OutputRawSpan` is similar to `OutputSpan`, but its initialized memory is untyped. Its API supports appending the bytes of instances of `BitwiseCopyable` types, as well as a variety of bulk initialization operations. + +```swift +@frozen +public struct OutputRawSpan: ~Copyable, ~Escapable { + internal var _start: UnsafeMutableRawPointer? + public let capacity: Int + internal var _count: Int +} +``` + +The memory represented by an `OutputRawSpan` contains `byteCount` initialized bytes, followed by uninitialized memory. + +```swift +extension OutputRawSpan { + /// The number of initialized bytes in this `OutputRawSpan` + public var byteCount: Int { get } + + /// A Boolean value indicating whither the span is empty. + public var isEmpty: Bool { get } + + /// A Boolean value indicating whither the span is full. + public var isFull: Bool { get } + + /// The number of uninitialized bytes remaining in this `OutputRawSpan` + public var freeCapacity: Int { get } // capacity - byteCount +} +``` + + +##### Appending to `OutputRawSpan` + +The basic operation is to append the bytes of some value to an `OutputRawSpan`. Note that since the fundamental operation is appending bytes, `OutputRawSpan` does not concern itself with memory alignment. +```swift +extension OutputRawSpan { + /// Append a single byte to this span + @_lifetime(self: copy self) + public mutating func append(_ value: UInt8) + + /// Appends the given value's bytes to this span's initialized bytes + @_lifetime(self: copy self) + public mutating func append( + _ value: T, as type: T.Type + ) + + /// Appends the given value's bytes repeatedly to this span's initialized bytes + @_lifetime(self: copy self) + public mutating func append( + repeating repeatedValue: T, count: Int, as type: T.Type + ) +} +``` + +An `OutputRawSpan`'s initialized memory is accessible for read-only or mutating access via the `bytes` and `mutableBytes` properties: + +```swift +extension OutputRawSpan { + /// Borrow the underlying initialized memory for read-only access. + public var bytes: RawSpan { + @_lifetime(borrow self) borrowing get + } + + /// Exclusively borrow the underlying initialized memory for mutation. + public var mutableBytes: MutableRawSpan { + @_lifetime(&self) mutating get + } +} +``` + +Deinitializing memory from an `OutputRawSpan`: + +```swift +extension OutputRawSpan { + + /// Remove the last byte from this span + @_lifetime(self: copy self) + public mutating func removeLast() -> UInt8 { + + /// Remove the last N elements, returning the memory they occupy + /// to the uninitialized state. + /// + /// `n` must not be greater than `count` + @_lifetime(self: copy self) + public mutating func removeLast(_ n: Int) + + /// Remove all this span's elements and return its memory + /// to the uninitialized state. + @_lifetime(self: copy self) + public mutating func removeAll() +} +``` + +##### Interoperability with unsafe code + +We provide a method to process or populate an `OutputRawSpan` using unsafe operations, which can also be used for out-of-order initialization. + +```swift +extension OutputRawSpan { + /// Call the given closure with the unsafe buffer pointer addressed by this + /// OutputRawSpan and a mutable reference to its count of initialized bytes. + /// + /// This method provides a way to process or populate an `OutputRawSpan` using + /// unsafe operations, such as dispatching to code written in legacy + /// (memory-unsafe) languages. + /// + /// The supplied closure may process the buffer in any way it wants; however, + /// when it finishes (whether by returning or throwing), it must leave the + /// buffer in a state that satisfies the invariants of the output span: + /// + /// 1. The inout integer passed in as the second argument must be the exact + /// number of initialized bytes in the buffer passed in as the first + /// argument. + /// 2. These initialized elements must be located in a single contiguous + /// region starting at the beginning of the buffer. The rest of the buffer + /// must hold uninitialized memory. + /// + /// This function cannot verify these two invariants, and therefore + /// this is an unsafe operation. Violating the invariants of `OutputRawSpan` + /// may result in undefined behavior. + @_lifetime(self: copy self) + public mutating func withUnsafeMutableBytes( + _ body: ( + UnsafeMutableRawBufferPointer, + _ initializedCount: inout Int + ) throws(E) -> R + ) throws(E) -> R +} +``` + +##### Creating `OutputRawSpan` instances + +Creating an `OutputRawSpan` is an unsafe operation. It requires having knowledge of the initialization state of the range of memory being targeted. The range of memory must be in two regions: the first region contains initialized bytes, and the second region is uninitialized. The number of initialized bytes is passed to the `OutputRawSpan` initializer through its `initializedCount` argument. + +```swift +extension OutputRawSpan { + + /// Unsafely create an OutputRawSpan over partly-initialized memory. + /// + /// The memory in `buffer` must remain valid throughout the lifetime + /// of the newly-created `OutputRawSpan`. Its prefix must contain + /// `initializedCount` initialized bytes, followed by uninitialized + /// memory. + /// + /// - Parameters: + /// - buffer: an `UnsafeMutableBufferPointer` to be initialized + /// - initializedCount: the number of initialized elements + /// at the beginning of `buffer`. + @unsafe + @_lifetime(borrow buffer) + public init( + buffer: UnsafeMutableRawBufferPointer, + initializedCount: Int + ) + + /// Create an OutputRawSpan with zero capacity + @_lifetime(immortal) + public init() + + /// Unsafely create an OutputRawSpan over partly-initialized memory. + /// + /// The memory in `buffer` must remain valid throughout the lifetime + /// of the newly-created `OutputRawSpan`. Its prefix must contain + /// `initializedCount` initialized bytes, followed by uninitialized + /// memory. + /// + /// - Parameters: + /// - buffer: an `UnsafeMutableBufferPointer` to be initialized + /// - initializedCount: the number of initialized elements + /// at the beginning of `buffer`. + @unsafe + @_lifetime(borrow buffer) + public init( + buffer: Slice, + initializedCount: Int + ) +} +``` + +##### Retrieving initialized memory from an `OutputRawSpan` + +Once memory has been initialized using `OutputRawSpan`, the owner of the memory must consume the instance in order to retake ownership of the initialized memory. The owning type must pass the memory used to initialize the `OutputRawSpan` to the `finalize(for:)` function. Passing the wrong buffer is a programmer error and the function traps. `finalize()` consumes the `OutputRawSpan` instance and returns the number of initialized bytes. + +```swift +extension OutputRawSpan { + /// Consume the OutputRawSpan and return the number of initialized bytes. + /// + /// Parameters: + /// - buffer: The buffer being finalized. This must be the same buffer as used + /// to create the `OutputRawSpan` instance. + /// Returns: The number of initialized bytes. + @unsafe + public consuming func finalize( + for buffer: UnsafeMutableRawBufferPointer + ) -> Int + + /// Consume the OutputRawSpan and return the number of initialized bytes. + /// + /// Parameters: + /// - buffer: The buffer being finalized. This must be the same buffer as used + /// to create the `OutputRawSpan` instance. + /// Returns: The number of initialized bytes. + @unsafe + public consuming func finalize( + for buffer: Slice + ) -> Int +} +``` + + +#### Extensions to Standard Library types + +The standard library and Foundation will add a few initializers that enable initialization in place, intermediated by an `OutputSpan` instance, passed as a parameter to a closure: + +```swift +extension Array { + /// Creates an array with the specified capacity, then calls the given + /// closure with an OutputSpan to initialize the array's contents. + public init( + capacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) + + /// Grows the array to ensure capacity for the specified number of elements, + /// then calls the closure with an OutputSpan covering the array's + /// uninitialized memory. + public mutating func append( + addingCapacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) +} + +extension ContiguousArray { + /// Creates an array with the specified capacity, then calls the given + /// closure with an OutputSpan to initialize the array's contents. + public init( + capacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) + + /// Grows the array to ensure capacity for the specified number of elements, + /// then calls the closure with an OutputSpan covering the array's + /// uninitialized memory. + public mutating func append( + addingCapacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) +} + +extension ArraySlice { + /// Grows the array to ensure capacity for the specified number of elements, + /// then calls the closure with an OutputSpan covering the array's + /// uninitialized memory. + public mutating func append( + addingCapacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) +} + +extension String { + /// Creates a new string with the specified capacity in UTF-8 code units, and + /// then calls the given closure with a OutputSpan to initialize the string's + /// contents. + /// + /// This initializer replaces ill-formed UTF-8 sequences with the Unicode + /// replacement character (`"\u{FFFD}"`). This may require resizing + /// the buffer beyond its original capacity. + public init( + repairingUTF8WithCapacity capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) + + /// Creates a new string with the specified capacity in UTF-8 code units, and + /// then calls the given closure with a OutputSpan to initialize the string's + /// contents. + /// + /// This initializer does not try to repair ill-formed code unit sequences. + /// If any are found, the result of the initializer is `nil`. + public init?( + validatingUTF8WithCapacity capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) + + /// Grows the string to ensure capacity for the specified number + /// of UTF-8 code units, then calls the closure with an OutputSpan covering + /// the string's uninitialized memory. + public mutating func append( + addingUTF8Capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) +} + +extension UnicodeScalarView { + /// Creates a new string with the specified capacity in UTF-8 code units, and + /// then calls the given closure with a OutputSpan to initialize + /// the string's contents. + /// + /// This initializer replaces ill-formed UTF-8 sequences with the Unicode + /// replacement character (`"\u{FFFD}"`). This may require resizing + /// the buffer beyond its original capacity. + public init( + repairingUTF8WithCapacity capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) + + /// Creates a new string with the specified capacity in UTF-8 code units, and + /// then calls the given closure with a OutputSpan to initialize + /// the string's contents. + /// + /// This initializer does not try to repair ill-formed code unit sequences. + /// If any are found, the result of the initializer is `nil`. + public init?( + validatingUTF8WithCapacity capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) + + /// Grows the string to ensure capacity for the specified number + /// of UTF-8 code units, then calls the closure with an OutputSpan covering + /// the string's uninitialized memory. + public mutating func append( + addingUTF8Capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) +} + +extension InlineArray { + /// Creates an array, then calls the given closure with an OutputSpan + /// to initialize the array's elements. + /// + /// NOTE: The closure must initialize every element of the `OutputSpan`. + /// If the closure does not do so, the initializer will trap. + public init( + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) +} +``` + +#### Extensions to `Foundation.Data` + +While the `swift-foundation` package and the `Foundation` framework are not governed by the Swift evolution process, `Data` is similar in use to standard library types, and the project acknowledges that it is desirable for it to have similar API when appropriate. Accordingly, we plan to propose the following additions to `Foundation.Data`: +```swift +extension Data { + /// Creates a data instance with the specified capacity, then calls + /// the given closure with an OutputSpan to initialize the instances's + /// contents. + public init( + capacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) + + /// Creates a data instance with the specified capacity, then calls + /// the given closure with an OutputSpan to initialize the instances's + /// contents. + public init( + rawCapacity: Int, + initializingWith initializer: (inout OutputRawSpan) throws(E) -> Void + ) throws(E) + + /// Ensures the data instance has enough capacity for the specified + /// number of bytes, then calls the closure with an OutputSpan covering + /// the uninitialized memory. + public mutating func append( + addingCapacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) + + /// Ensures the data instance has enough capacity for the specified + /// number of bytes, then calls the closure with an OutputSpan covering + /// the uninitialized memory. + public mutating func append( + addingRawCapacity: Int, + initializingWith initializer: (inout OutputRawSpan) throws(E) -> Void + ) throws(E) +} +``` + +#### Changes to `MutableSpan` and `MutableRawSpan` + +This proposal considers the naming of `OutputSpan`'s bulk-initialization methods, and elects to defer their implementation until we have more experience with the various kinds of container we need to support. We also introduced bulk updating functions to `MutableSpan` and `MutableRawSpan` in [SE-0467][SE-0467]. It is clear that they have the same kinds of parameters as `OutputSpan`'s bulk-initialization methods, but the discussion has taken a [different direction](#contentsOf) in the latter case. We would like both of these sets of operations to match. Accordingly, we will remove the bulk `update()` functions proposedin [SE-0467][SE-0467], to be replaced with a better naming scheme later. Prototype bulk-update functionality will also be added via a package in the meantime. + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a new version of the Swift standard library and runtime. + + +## Alternatives Considered + +#### Vending `OutputSpan` as a property + +`OutputSpan` changes the number of initialized elements in a container (or collection), and this requires some operation to update the container after the `OutputSpan` is consumed. Let's call that update operation a "cleanup" operation. The cleanup operation needs to be scheduled in some way. We could associate the cleanup with the `deinit` of `OutputSpan`, or the `deinit` of a wrapper of `OutputSpan`. Neither of these seem appealing; the mechanisms would involve an arbitrary closure executed at `deinit` time, or having to write a full wrapper for each type that vends an `OutputSpan`. We could potentially schedule the cleanup operation as part of a coroutine accessor, but these are not productized yet. The pattern established by closure-taking API is well established, and that pattern fits the needs of `OutputSpan` well. + +#### Container construction pattern + +A constrained version of possible `OutputSpan` use consists of in-place container initialization. This proposal introduces a few initializers in this vein, such as `Array.init(capacity:initializingWith:)`, which rely on closure to establish a scope. A different approach would be to use intermediate types to perform such operations: + +```swift +struct ArrayConstructor: ~Copyable { + @_lifetime(&self) mutating var outputSpan: OutputSpan + private let _ArrayBuffer + + init(capacity: Int) +} + +extension Array { + init(_: consuming ArrayConstructor) +} +``` + +## Future directions + +#### Helpers to initialize memory in an arbitrary order + +Some applications may benefit from the ability to initialize a range of memory in a different order than implemented by `OutputSpan`. This may be from back-to-front or even arbitrary order. There are many possible forms such an initialization helper can take, depending on how much memory safety the application is willing to give up in the process of initializing the memory. At the unsafe end, this can be delegating to an `UnsafeMutableBufferPointer` along with a set of requirements; this option is proposed here. At the safe end, this could be delegating to a data structure which keeps track of initialized memory using a bitmap. It is unclear how much need there is for this more heavy-handed approach, so we leave it as a future enhancement if it is deemed useful. + +#### Insertions + +A use case similar to appending is insertions. Appending is simply inserting at the end. Inserting at positions other than the end is an important capability. We expect to add insertions soon in a followup proposal if `OutputSpan` is accepted. Until then, a workaround is to append, then rotate the elements to the desired position using the `mutableSpan` view. + +#### Generalized removals + +Similarly to generalized insertions (i.e. not from the end), we can think about removals of one or more elements starting at a given position. We expect to add generalized removals along with insertions in a followup proposal after `OutputSpan` is accepted. + +#### Variations on `Array.append(addingCapacity:initializingWith:)` + +The function proposed here only exposes uninitialized capacity in the `OutputSpan` parameter to its closure. A different function (perhaps named `edit()`) could also pass the initialized portion of the container, allowing an algorithm to remove or to add elements. This could be considered in addition to `append()`. + +#### Methods to initialize or update in bulk + +The `RangeReplaceableCollection` protocol has a foundational method `append(contentsOf:)` for which this document does not propose a corresponding method. We expect to first add such bulk-copying functions as part of of a package. + +`OutputSpan` lays the groundwork for new, generalized `Container` protocols that will expand upon and succeed the `Collection` hierarchy while allowing non-copyability and non-escapability to be applied to both containers and elements. We hope to find method and property names that will be generally applicable. The `append(contentsOf:)` method we refer to above always represents copyable and escapable collections with copyable and escapable elements. The definition is as follows: `mutating func append(contentsOf newElements: __owned S)`. This supports copying elements from the source, while also destroying the source if we happen to hold its only copy. This is obviously not sufficient if the elements are non-copyable, or if we only have access to a borrowed source. + +When the elements are non-copyable, we must append elements that are removed from the source. Afterwards, there are two possible dispositions of the source: destruction (`consuming`), where the source can no longer be used, or mutation (`inout`), where the source has been emptied but is still usable. + +When the elements are copyable, we can simply copy the elements from the source. Afterwards, there are two possible dispositions of the source: releasing a borrowed source, or `consuming`. The latter is approximately the same behaviour as `RangeReplaceableCollection`'s `append(contentsOf:)` function shown above. + +In an ideal world, we would like to use the same name for all of these variants: + +```swift +extension OutputSpan { + mutating func append(contentsOf: consuming some Sequence) + mutating func append(contentsOf: borrowing some Container) +} +extension OutputSpan where Element: ~Copyable { + mutating func append(contentsOf: consuming some ConsumableContainer) + mutating func append(contentsOf: inout some RangeReplaceableContainer) +} +``` + +However, this would break down in particular for `UnsafeMutableBufferPointer`, since it would make it impossible to differentiate between just copying the elements out of it, or moving its elements out (and deinitializing its memory). Once the `Container` protocols exist, we can expect that the same issue would exist for any type that conforms to more than one of the protocols involved in the list above. For example if a type conforms to `Container` as well as `Sequence`, then there would be an ambiguity. + +We could fix this by extending the syntax of the language. It is already possible to overload two functions where they differ only by whether a parameter is `inout`, for example. This is a more advanced [future direction](#contentsOf-syntax). + +Instead of the "ideal" solution, we could propose `append()` functions in the following form: + +```swift +extension OutputSpan { + mutating func append(contentsOf: consuming some Sequence) + mutating func append(copying: borrowing some Container) +} +extension OutputSpan where Element: ~Copyable { + mutating func append(consuming: consuming some ConsumableContainer) + mutating func append(moving: inout some RangeReplaceableContainer) +} +``` + +In this form, we continue to use the `contentsOf` label for `Sequence` parameters, but use different labels for the other types of containers. The `update()` methods of `MutableSpan` could be updated in a similar manner, for the same reasons. + +We note that the four variants of `append()` are required for generalized containers. We can therefore expect that the names we choose will appear later on many types of collections that interact with the future `Container` protocols. Since this nomenclature could become ubiquitous when interacting with `Collection` and `Container` instances, we defer a formal proposal until we have more experience and feedback. + +In the meantime, many applications will work efficiently with repeated calls to `append()` in a loop. The bulk initialization functions are implementable by using `withUnsafeBufferPointer` as a workaround as well, if performance is an issue before a package-based solution is released. + +#### Language syntax to distinguish between ownership modes for function arguments + +In the previous "Future Direction" subsection about [bulk initialization methods](#contentsOf), we suggest a currently unachievable naming scheme: +```swift +extension OutputSpan { + mutating func append(contentsOf: consuming some Sequence) + mutating func append(contentsOf: borrowing some Container) +} +extension OutputSpan where Element: ~Copyable { + mutating func append(contentsOf: consuming some ConsumableContainer) + mutating func append(contentsOf: inout some RangeReplaceableContainer) +} +``` + +The language partially supports disambiguating this naming scheme, in that we can already distinguish functions over the mutability of a single parameter: + +```swift +func foo(_ a: borrowing A) {} +func foo(_ a: inout A) {} + +var a = A() +foo(a) +foo(&a) +``` + +We could expand upon this ability to disambiguate by using keywords or even new sigils: + +```swift +let buffer: UnsafeMutableBufferPointer = ... +let array = Array(capacity: buffer.count*2) { + (o: inout OutputSpan) in + o.append(contentsOf: borrow buffer) + o.append(contentsOf: consume buffer) +} +``` + +## Acknowledgements + +Thanks to Karoy Lorentey, Nate Cook and Tony Parker for their feedback. diff --git a/proposals/0486-adoption-tooling-for-swift-features.md b/proposals/0486-adoption-tooling-for-swift-features.md new file mode 100644 index 0000000000..81f3424f5e --- /dev/null +++ b/proposals/0486-adoption-tooling-for-swift-features.md @@ -0,0 +1,387 @@ +# Migration tooling for Swift features + +* Proposal: [SE-0486](0486-adoption-tooling-for-swift-features.md) +* Authors: [Anthony Latsis](https://github.com/AnthonyLatsis), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Franz Busch](https://github.com/FranzBusch) +* Status: **Implemented (Swift 6.2)** +* Implementation: https://github.com/swiftlang/swift-package-manager/pull/8613 +* Review: [Pitch](https://forums.swift.org/t/pitch-adoption-tooling-for-upcoming-features/77936), [Review](https://forums.swift.org/t/se-0486-migration-tooling-for-swift-features/80121) + +## Introduction + +Swift 5.8 introduced [upcoming features][SE-0362], which enabled piecemeal +adoption of individual source-incompatible changes that are included in a +language mode. +Many upcoming features have a mechanical migration, meaning the compiler can +determine the exact source changes necessary to allow the code to compile under +the upcoming feature while preserving the behavior of the code. +This proposal seeks to improve the experience of enabling individual Swift +features by providing an integrated mechanism for producing these source code +modifications automatically. + +## Motivation + +It is the responsibility of project maintainers to preserve source (and binary) +compatibility both internally and for library clients when enabling an upcoming +feature, which can be difficult or tedious without having tools to help detect +possibly inadvertent changes or perform monotonous migration shenanigans for +you. +*Our* responsibility is to make that an easier task for everybody. + +### User intent + +A primary limiting factor in how proactively and accurately the compiler can +assist developers with adopting a feature is a lack of comprehension of user +intent. +Is the developer expecting guidance on adopting an improvement? +All the compiler knows to do when a feature is enabled is to compile code +accordingly. +If an upcoming feature supplants an existing grammatical construct or +invalidates an existing behavior, the language rules alone suffice because +Swift can consistently infer the irrefutable need to diagnose certain code +patterns just by spotting them. + +Needless to say, not all upcoming features fall under these criteria (and not +all features are source-breaking in the first place). +Consider [`DisableOutwardActorInference`][SE-0401], which changes actor +isolation inference rules with respect to wrapped properties. +There is no way for the programmer to specify that they'd like compiler fix-its +to make the existing actor isolation inference explicit. +If they enable the upcoming feature, their code will simply behave differently. +This was a point of debate in the review of [SE-0401], and the Language +Steering Group concluded that automatic migration tooling is the right way to +address this particular workflow, as +[noted in the acceptance notes][SE-0401-acceptance]: + +> the Language Steering Group believes that separate migration tooling to +> help programmers audit code whose behavior will change under Swift 6 mode +> would be beneficial for all upcoming features that can change behavior +> without necessarily emitting errors. + +### Automation + +Many existing and prospective upcoming features account for simple and reliable +migration paths to facilitate adoption: + +* [`NonfrozenEnumExhaustivity`][SE-0192]: Restore exhaustivity with + `@unknown default:`. +* [`ConciseMagicFile`][SE-0274]: `#file` → `#filePath`. +* [`ForwardTrailingClosures`][SE-0286]: Disambiguate argument matching by + de-trailing closures and/or inlining default arguments. +* [`ExistentialAny`][SE-0335]: `P` → `any P`. +* [`ImplicitOpenExistentials`][SE-0352]: Suppress opening with `as any P` + coercions. +* [`BareSlashRegexLiterals`][SE-0354]: Disambiguate using parentheses, + e.g. `foo(/a, b/)` → `foo((/a), b/)`. +* [`DeprecateApplicationMain`][SE-0383]: `@UIApplicationMain` → `@main`, + `@NSApplicationMain` → `@main`. +* [`DisableOutwardActorInference`][SE-0401]: Specify global actor isolation + explicitly. +* [`InternalImportsByDefault`][SE-0409]: `import X` → `public import X`. +* [`GlobalConcurrency`][SE-0412]: Convert the global variable to a `let`, or + `@MainActor`-isolate it, or mark it with `nonisolated(unsafe)`. +* [`MemberImportVisibility`][SE-0444]: Add explicit imports appropriately. +* [`InferSendableFromCaptures`][SE-0418]: Suppress inference with coercions + and type annotations. +* [Inherit isolation by default for async functions][async-inherit-isolation-pitch]: + Mark nonisolated functions with the proposed attribute. + +Application of these adjustments can be fully automated in favor of preserving +behavior, saving time for more important tasks, such as identifying, auditing, +and testing code where a change in behavior is preferable. + +## Proposed solution + +Introduce the notion of a migration mode for individual experimental and +upcoming features. +The core idea behind migration mode is a declaration of intent that can be +leveraged to build better supportive adoption experiences for developers. +If enabling a feature communicates an intent to *enact* rules, migration mode +communicates an intent to migrate code so as to preserve compatibility once the +feature is enabled. + +This proposal will support the set of existing upcoming features that +have mechanical migrations, as described in the [Automation](#automation) +section. +All future proposals that intend to introduce an upcoming feature and +provide for a mechanical migration should include a migration mode and detail +its behavior alongside the migration paths in the *Source compatibility* +section. + +## Detailed design + +Upcoming features that have mechanical migrations will support a migration +mode, which is a new mode of building a project that will produce compiler +warnings with attached fix-its that can be applied to preserve the behavior +of the code under the feature. + +The action of enabling a previously disabled upcoming feature in migration +mode must not cause any new compiler errors or behavioral changes, and the +fix-its produced must preserve compatibility. +Compatibility here refers to both source and binary compatibility, as well as +to behavior. +Additionally, this action will have no effect if the mode is not supported +for a given upcoming feature, i.e., because the upcoming feature does not +have a mechanical migration. +A corresponding warning will be emitted in this case to avoid the false +impression that the impacted source code is compatible with the feature. +This warning will belong to the diagnostic group `StrictLanguageFeatures`. + +### Interface + +The `-enable-*-feature` frontend and driver command line options will start +supporting an optional mode specifier with `migrate` as the only valid mode: + +``` +-enable-upcoming-feature [:] +-enable-experimental-feature [:] + + := migrate +``` + +For example: + +``` +-enable-upcoming-feature InternalImportsByDefault:migrate +``` + +If the specified mode is invalid, the option will be ignored, and a warning will +be emitted. +This warning will belong to the diagnostic group `StrictLanguageFeatures`. +In a series of either of these options applied to a given feature, only the +last option will be honored. +If a feature is both implied by the effective language mode and enabled in +migration mode, the latter option will be disregarded. + +### Diagnostics + +Diagnostics emitted in relation to a specific feature in migration mode must +belong to a diagnostic group named after the feature. +The names of diagnostic groups can be displayed alongside diagnostic messages +using `-print-diagnostic-groups` and used to associate messages with features. + +### `swift package migrate` command + +To enable seamless migration experience for Swift packages, I'd like to propose a new Swift Package Manager command - `swift package migrate` to complement the Swift compiler-side changes. + +The command would accept one or more features that have migration mode enabled and optionally a set of targets to migrate, if no targets are specified the whole package is going to be migrated to use new features. + +#### Interface + +``` +USAGE: swift package migrate [] --to-feature ... + +OPTIONS: + --target The targets to migrate to specified set of features or a new language mode. + --to-feature + The Swift language upcoming/experimental feature to migrate to. + -h, --help Show help information. +``` + +#### Use case + +``` +swift package migrate --target MyTarget,MyTest --to-feature ExistentialAny +``` + +This command would attempt to build `MyTarget` and `MyTest` targets with `ExistentialAny:migrate` feature flag, apply any fix-its associated with +the feature produced by the compiler, and update the `Package.swift` to +enable the feature(s) if both of the previous actions are successful: + +``` +.target( + name: "MyTarget", + ... + swiftSettings: [ + // ... existing settings, + .enableUpcomingFeature("ExistentialAny") + ] +) +... +.testTarget( + name: "MyTest", + ... + swiftSettings: [ + // ... existing settings, + .enableUpcomingFeature("ExistentialAny") + ] +) +``` + +In the "whole package" mode, every target is going to be updated to include +new feature flag(s). This is supported by the same functionality as `swift package add-setting` command. + +If it's, for some reason, impossible to add the setting the diagnostic message would suggest what to add and where i.e. `...; please add 'ExistentialAny' feature to 'MyTarget' target manually`. + +#### Impact on Interface + +This proposal introduces a new command but does not interfere with existing commands. It follows the same pattern as `swift build` and `swift test` in a consistent manner. + +## Source compatibility + +This proposal does not affect language rules. +The described changes to the API surface are source-compatible. + +## ABI compatibility + +This proposal does not affect binary compatibility or binary interfaces. + +## Implications on adoption + +Entering or exiting migration mode can affect behavior and is therefore a +potentially source-breaking action. + +## Future directions + +### Producing source incompatible fix-its + +For some features, a source change that alters the semantics of +the program is a more desirable approach to addressing an error that comes +from enabling the feature. +For example, programmers might want to replace cases of `any P` with `some P`. +Migration tooling could support the option to produce source incompatible +fix-its in cases where the compiler can detect that a different behavior might +be more beneficial. + +### Applications beyond mechanical migration + +The concept of migration mode could be extrapolated to additive features, such +as [typed `throws`][SE-0413] or [opaque parameter types][SE-0341], by providing +actionable adoption tips. +Additive features are hard-enabled and become an integral part of the language +as soon as they ship. +Many recent additive features are already integrated into the Swift feature +model, and their metadata is kept around either to support +[feature availability checks][SE-0362-feature-detection] in conditional +compilation blocks or because they started off as experimental features. + +Another feasible extension of migration mode is promotion of best practices. + +### Augmented diagnostic metadata + +The current serialization format for diagnostics does not include information +about diagnostic groups or whether a particular fix-it preserves semantics. +There are several reasons why this data can be valuable for users, and why it +is essential for future tools built around migration mode: +* The diagnostic group name can be used to, well, group diagnostics, as well as + to communicate relationships between diagnostics and features and filter out + relevant diagnostics. + This can prove especially handy when multiple features are simultaneously + enabled in migration mode, or when similar diagnostic messages are caused by + distinct features. +* Exposing the purpose of a fix-it can help developers make quicker decisions + when offered multiple fix-its. + Furthermore, tools can take advantage of this information by favoring and + auto-applying source-compatible fix-its. + +## Alternatives considered + +### A distinct `-migrate` option + +This direction has a questionably balanced set of advantages and downsides. +On one hand, it would provide an adequate foundation for invoking migration +for a language mode in addition to individual features. +On the other hand, an independent option is less discoverable, has a steeper +learning curve, and makes the necessary relationships between it and the +existing `-enable-*-feature` options harder to infer. +Perhaps more notably, a bespoke option by itself would not scale to any future +modes, setting what might be an unfortunate example for further decentralization +of language feature control. + +### API for package manifests + +The decision around surfacing migration mode in the `PackageDescription` +library depends on whether there is a consensus on the value of enabling it as +a persistent setting as opposed to an automated procedure in the long run. + +Here is how an API change could look like for the proposed solution: + +```swift ++extension SwiftSetting { ++ @available(_PackageDescription, introduced: 6.2) ++ public enum SwiftFeatureMode { ++ case migrate ++ case on ++ } ++} +``` +```diff + public static func enableUpcomingFeature( + _ name: String, ++ mode: SwiftFeatureMode = .on, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting + + public static func enableExperimentalFeature( + _ name: String, ++ mode: SwiftFeatureMode = .on, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting +``` + +It can be argued that both Swift modules and the volume of changes required for +migration can be large enough to justify spreading the review over several +sessions, especially if migration mode gains support for parallel +[source-incompatible fix-its][#producing-source-incompatible-fix-its]. +However, we also expect higher-level migration tooling to allow for +incremental progress. + +### Naming + +The next candidates in line per discussions are ***adopt***, ***audit***, +***stage***, and ***preview***, respectively. +* ***preview*** and ***stage*** can both be understood as to report on the + impact of a change, but are less commonly used in the sense of code + migration. +* ***audit*** best denotes a recurrent action in this context, which we believe + is more characteristic of the static analysis domain, such as enforcing a set + of custom compile-time rules on code. +* An important reservation about ***adoption*** of source-breaking features is + that it comprises both code migration and integration. + It may be more prudent to save this term for a future add-on mode that, + unlike migration mode, implies that the feature is enabled, and can be invoked + in any language mode to aid developers in making better use of new behaviors + or rules. + To illustrate, this mode could appropriately suggest switching from `any P` + to `some P` for `ExistentialAny`. + +### `swift package migrate` vs. `swift migrate` + +Rather than have migrate as a subcommand (ie. `swift package migrate`), another option is to add another top level command, ie. `swift migrate`. + +As the command applies to the current package, we feel a `swift package` sub-command fits better than a new top-level command. This also aligns with the recently added package refactorings (eg. `add-target`). + +## Acknowledgements + +This proposal was inspired by documents prepared by [Allan Shortlidge] and +[Holly Borla]. +Special thanks to Holly for her guidance throughout the draft stage. + + + +[Holly Borla]: https://github.com/hborla +[Allan Shortlidge]: https://github.com/tshortli + +[SE-0192]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0192-non-exhaustive-enums.md +[SE-0274]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0274-magic-file.md +[SE-0286]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0286-forward-scan-trailing-closures.md +[SE-0296]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md +[SE-0335]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md +[SE-0337]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md +[SE-0341]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md +[SE-0352]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md +[SE-0354]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0354-regex-literals.md +[SE-0362]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md +[SE-0362-feature-detection]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md#feature-detection-in-source-code +[SE-0383]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0383-deprecate-uiapplicationmain-and-nsapplicationmain.md +[SE-0401]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0401-remove-property-wrapper-isolation.md +[SE-0401-acceptance]: https://forums.swift.org/t/accepted-with-modifications-se-0401-remove-actor-isolation-inference-caused-by-property-wrappers/66241 +[SE-0409]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md +[SE-0411]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0411-isolated-default-values.md +[SE-0413]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0413-typed-throws.md +[SE-0412]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0412-strict-concurrency-for-global-variables.md +[SE-0418]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0418-inferring-sendable-for-methods.md +[SE-0423]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0423-dynamic-actor-isolation.md +[SE-0434]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0434-global-actor-isolated-types-usability.md +[SE-0444]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0444-member-import-visibility.md +[async-inherit-isolation-pitch]: https://forums.swift.org/t/pitch-inherit-isolation-by-default-for-async-functions/74862 diff --git a/proposals/0487-extensible-enums.md b/proposals/0487-extensible-enums.md new file mode 100644 index 0000000000..a8e226c95b --- /dev/null +++ b/proposals/0487-extensible-enums.md @@ -0,0 +1,319 @@ +# Nonexhaustive enums + +* Proposal: [SE-0487](0487-extensible-enums.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin), [Franz Busch](https://github.com/FranzBusch), [Cory Benfield](https://github.com/lukasa) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Accepted** +* Bug: [apple/swift#55110](https://github.com/swiftlang/swift/issues/55110) +* Implementation: [apple/swift#80503](https://github.com/swiftlang/swift/pull/80503) +* Upcoming Feature Flag: `ExtensibleAttribute` +* Review: ([pitch](https://forums.swift.org/t/pitch-extensible-enums-for-non-resilient-modules/77649)) ([first review](https://forums.swift.org/t/se-0487-extensible-enums/80114)) ([second review](https://forums.swift.org/t/second-review-se-0487-extensible-enums/80837)) ([acceptance](https://forums.swift.org/t/accepted-se-0487-nonexhaustive-enums/81508)) + +Previously pitched in: + +- https://forums.swift.org/t/extensible-enumerations-for-non-resilient-libraries/35900 +- https://forums.swift.org/t/pitch-non-frozen-enumerations/68373 + +Revisions: +- Renamed the attribute to `@nonexhaustive` and `@nonexhaustive(warn)` respectively +- Re-focused this proposal on introducing a new `@extensible` attribute and + moved the language feature to a future direction +- Introduced a second annotation `@nonExtensible` to allow a migration path into + both directions +- Added future directions for adding additional associated values +- Removed both the `@extensible` and `@nonExtensible` annotation in favour of + re-using the existing `@frozen` annotation +- Added the high level goals that this proposal aims to achieve +- Expanded on the proposed migration path for packages with regards to their + willingness to break API +- Added future directions for exhaustive matching for larger compilation units +- Added alternatives considered section for a hypothetical + `@preEnumExtensibility` +- Added a section for `swift package diagnose-api-breaking-changes` + +## Introduction + +This proposal provides developers the capabilities to mark public enums in +non-resilient Swift libraries as extensible. This makes Swift `enum`s vastly +more useful in public API of such libraries. + +## Motivation + +When Swift was enhanced to add support for ABI-stable libraries that were built with +"library evolution" enabled ("resilient" libraries as we call them in this proposal), +the Swift language had to support these libraries vending enums that might have cases +added to them in a later version. Swift supports exhaustive switching over cases. +When binaries are compiled against a ABI-stable library they need to be able to handle the +addition of a new case by that library later on, without needing to be rebuilt. + +Consider the following simple library to your favorite pizza place: + +```swift +public enum PizzaFlavor { + case hawaiian + case pepperoni + case cheese +} +``` + +In the standard "non-resilient" mode, users of the library can write exhaustive switch +statements over the enum `PizzaFlavor`: + +```swift +switch pizzaFlavor { +case .hawaiian: + throw BadFlavorError() +case .pepperoni: + try validateNoVegetariansEating() + return .delicious +case .cheese: + return .delicious +} +``` + +Swift requires switches to be exhaustive i.e. the must handle every possibility. +If the author of the above switch statement was missing a case (perhaps they forgot +`.hawaiian` is a flavor), the compiler will error, and force the user to either add a +`default:` clause, or to add the missing case. + +If later a new case is added to the enum (maybe `.veggieSupreme`), exhaustive switches +over that enum might no longer be exhaustive. This is often _desirable_ within a single +codebase (even one split up into multiple modules). A case is added, and the compiler will +assist in finding all the places where this new case must be handled. + +But it presents a problem for authors of both resilient and non-resilient libraries: + +- For non-resilient libraries, adding a case is a source-breaking API change: clients +exhaustively switching over the enum will no longer compile. So can only be done with +a major semantic version bump. +- For resilient libraries, even that is not an option. An ABI-stable library cannot allow +a situation where a binary that has not yet been recompiled can no longer rely on its +switches over an enum are exhaustive. + +Because of the implications on ABI and the requirement to be able to evolve +libraries with public enumerations in their API, the resilient language dialect introduced +"non-exhaustive enums" in [SE-0192](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0192-non-exhaustive-enums.md). + +If the library was compiled with `-enable-library-evolution`, when a user attempts to +exhaustively switch over the `PizzaFlavor` enum the compiler will emit an error +(when in Swift 6 language mode, a warning in prior language modes), requiring users +to add an `@unknown default:` clause: + +```swift +switch pizzaFlavor { +case .hawaiian: + throw BadFlavorError() +case .pepperoni: + try validateNoVegetariansEating() + return .delicious +case .cheese: + return .delicious +@unknown default: + try validateNoVegetariansEating() + return .delicious +} +``` + +The user is forced to specify how cases are handled if they are introduced later. This +allows ABI-stable libraries to add cases without risking undefined behavior in client +binaries that haven't yet been recompiled. + +When a resilient library knows that an enumeration will never be extended, the author +can annotate the enum with `@frozen`, which in the case of enums is a guarantee that no +further cases can be added. For example, the `Optional` type in the standard library is +frozen, as no third option beyond `some` and `none` will ever be added. This brings +performance benefits, and also the convenience of not requiring an `@unknown default` case. + +`@frozen` is a powerful attribute that can be applied to both structs and enums. It has a +wide ranging number of effects, including exposing their size directly as part of the ABI +and providing direct access to stored properties. However, on enums it happens to +have source-level effects on the behavior of switch statements by clients of a library. +This difference was introduced late in the process of reviewing SE-0192. + +Extensibility of enums is also desirable for non-resilient libraries. Without it, there is no +way for a Swift package to be able to evolve a public enumeration without breaking the API. +However, in Swift today it is not possible for the default, "non-resilient" dialect to opt-in +to the extensible enumeration behavior. This is a substantial limitation, and greatly reduces +the utility of enumerations in non-resilient Swift. + +Over the past years, many packages have run into this limitation when trying to express APIs +using enums. As a non-exhaustive list of problems this can cause: + +- Using enumerations to represent `Error`s is inadvisable, as if new errors need + to be introduced they cannot be added to existing enumerations. This leads to + a proliferation of `Error` enumerations. "Fake" enumerations can be made using + `struct`s and `static let`s, but these do not work with the nice `Error` + pattern-match logic in catch blocks, requiring type casts. +- Using an enumeration to refer to a group of possible ideas without entirely + exhaustively evaluating the set is potentially dangerous, requiring a + deprecate-and-replace if any new elements appear. +- Using an enumeration to represent any concept that is inherently extensible is + tricky. For example, `SwiftNIO` uses an enumeration to represent HTTP status + codes. If new status codes are added, SwiftNIO needs to either mint new + enumerations and do a deprecate-and-replace, or it needs to force these new + status codes through the .custom enum case. + +This proposal plans to address these limitations on enumerations in +non-resilient Swift. + +## Proposed solution + +We propose to introduce a new `@nonexhaustive` attribute that can be applied to +enumerations to mark them as extensible. Such enums will behave the same way as +non-frozen enums from resilient Swift libraries. + +An example of using the new attribute is below: + +```swift +/// Module A +@nonexhaustive +public enum PizzaFlavor { + case hawaiian + case pepperoni + case cheese +} + +/// Module B +switch pizzaFlavor { // error: Switch covers known cases, but 'MyEnum' may have additional unknown values, possibly added in future versions +case .hawaiian: + throw BadFlavorError() +case .pepperoni: + try validateNoVegetariansEating() + return .delicious +case .cheese: + return .delicious +} +``` + +### Exhaustive switching inside same module/package + +Code inside the same module or package can be thought of as one co-developed +unit of code. Inside the same module or package, switching exhaustively over an +`@nonexhaustive` enum inside will not require an`@unknown default`, and using +one will generate a warning. + +### `@nonexhaustive` and `@frozen` + +An enum cannot be `@frozen` and `@nonexhaustive` at the same time. Thus, marking an +enum both `@nonexhaustive` and `@frozen` is not allowed and will result in a +compiler error. + +### API breaking checker + +The behavior of `swift package diagnose-api-breaking-changes` is also updated +to understand the new `@nonexhaustive` attribute. + +### Staging in using `@nonexhaustive(warn)` + +We also propose adding a new `@nonexhaustive(warn)` attribute that can be used +to mark enumerations as pre-existing to when they became extensible.This is +useful for developers that want to stage in changing an existing non-extensible +enum to be extensible over multiple releases. Below is an example of how this +can be used: + +```swift +// Package A +public enum Foo { + case foo +} + +// Package B +switch foo { +case .foo: break +} + +// Package A wants to make the existing enum extensible +@nonexhaustive(warn) +public enum Foo { + case foo +} + +// Package B now emits a warning downgraded from an error +switch foo { // warning: Enum might be extended later. Add an @unknown default case. +case .foo: break +} + +// Later Package A decides to extend the enum and releases a new major version +@nonexhaustive(warn) +public enum Foo { + case foo + case bar +} + +// Package B didn't add the @unknown default case yet. So now we we emit a warning and an error +switch foo { // error: Unhandled case bar & warning: Enum might be extended later. Add an @unknown default case. +case .foo: break +} +``` + +While the `@nonexhaustive(warn)` attribute doesn't solve the need of requiring +a new major when a new case is added it allows developers to stage in changing +an existing non-extensible enum to become extensible in a future release by +surfacing a warning about this upcoming break early. + +## Source compatibility + +### Resilient modules + +- Adding or removing the `@nonexhaustive` attribute has no-effect since it is the default in this language dialect. +- Adding the `@nonexhaustive(warn)` attribute has no-effect since it only downgrades the error to a warning. +- Removing the `@nonexhaustive(warn)` attribute is an API breaking since it upgrades the warning to an error again. + +### Non-resilient modules + +- Adding the `@nonexhaustive` attribute is an API breaking change. +- Removing the `@nonexhaustive` attribute is an API stable change. +- Adding the `@nonexhaustive(warn)` attribute has no-effect since it only downgrades the error to a warning. +- Removing the `@nonexhaustive(warn)` attribute is an API breaking since it upgrades the warning to an error again. + +## ABI compatibility + +The new attribute does not affect the ABI of an enum since it is already the +default in resilient modules. + +## Future directions + +### Aligning the language dialects + +In a previous iteration of this proposal, we proposed to add a new language +feature to align the language dialects in a future language mode. The main +motivation behind this is that the current default of non-extensible enums is a +common pitfall and results in tremendous amounts of unnoticed API breaks in the +Swift package ecosystem. We still believe that a future proposal should try +aligning the language dialects. This proposal is focused on providing a first +step to allow extensible enums in non-resilient modules. + +Regardless of whether a future language mode changes the default for non-resilient +libraries, a way of staging in this change will be required (similar to how the +`@preconcurency` attribute facilitated incremental adoption of Swift concurrency). + +### `@unknown catch` + +Enums can be used for errors. Catching and pattern matching enums could add +support for an `@unknown catch` to make pattern matching of typed throws align +with `switch` pattern matching. + +### Allow adding additional associated values + +Adding additional associated values to an enum can also be seen as extending it +and we agree that this is interesting to explore in the future. However, this +proposal focuses on solving the primary problem of the usability of public +enumerations in non-resilient modules. + +### Larger compilation units than packages + +During the pitch it was brought up that a common pattern for application +developers is to split an application into multiple smaller packages. Those +packages are versioned together and want to have the same exhaustive matching +behavior as code within a single package. As a future direction, build and +package tooling could allow to define larger compilation units to express this. +Until then developers are encouraged to use `@frozen` attributes on their +enumerations to achieve the same effect. + +## Alternatives considered + +### Different names for the attribute + +We considered different names for the attribute such as `@nonFrozen` or +`@extensible`; however, we felt that `@nonexhaustive` communicates the idea of +an extensible enum more clearly. diff --git a/proposals/0488-extracting.md b/proposals/0488-extracting.md new file mode 100644 index 0000000000..529b1d7e53 --- /dev/null +++ b/proposals/0488-extracting.md @@ -0,0 +1,146 @@ +# Apply the extracting() slicing pattern more widely + +* Proposal: [SE-0488](0488-extracting.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 6.2)** +* Implementation: underscored `_extracting()` members of `Span` and `RawSpan`, pending elsewhere. +* Review: ([pitch](https://forums.swift.org/t/pitch-apply-the-extracting-slicing-pattern-to-span-and-rawspan/80322)) ([review](https://forums.swift.org/t/se-0488-apply-the-extracting-slicing-pattern-more-widely/80854)) ([acceptance](https://forums.swift.org/t/accepted-se-0488-apply-the-extracting-slicing-pattern-more-widely/81235)) + +[SE-0437]: 0437-noncopyable-stdlib-primitives.md +[SE-0447]: 0447-span-access-shared-contiguous-storage.md +[SE-0467]: 0467-MutableSpan.md +[Forum-LifetimeAnnotations]: https://forums.swift.org/t/78638 + + +## Introduction and Motivation + +Slicing containers is an important operation, and non-copyable values have introduced a significant change in the spelling of that operation. When we [introduced][SE-0437] non-copyable primitives to the standard library, we allowed slicing `UnsafeBufferPointer` and related types via a family of `extracting()` methods. We expanded upon these when introducing [`MutableSpan`][SE-0467]. + +Now that we have a [supported spelling][Forum-LifetimeAnnotations] for lifetime dependencies, we propose adding the `extracting()` methods to `Span` and `RawSpan`, as well as members of the `UnsafeBufferPointer` family that were missed in [SE-0437][SE-0437]. + + +## Proposed solution + +As previously discussed in [SE-0437][SE-0437], the slicing pattern established by the `Collection` protocol cannot be generalized for either non-copyable elements or non-escapable containers. The solution is a family of functions named `extracting()`, with appropriate argument labels. + +The family of `extracting()` methods established by the [`MutableSpan` proposal][SE-0467] is as follows: +```swift +public func extracting(_ bounds: Range) -> Self +public func extracting(_ bounds: some RangeExpression) -> Self +public func extracting(_: UnboundedRange) -> Self +@unsafe public func extracting(unchecked bounds: Range) -> Self +@unsafe public func extracting(unchecked bounds: ClosedRange) -> Self + +public func extracting(first maxLength: Int) -> Self +public func extracting(droppingLast k: Int) -> Self +public func extracting(last maxLength: Int) -> Self +public func extracting(droppingFirst k: Int) -> Self +``` + +These will be provided for the following standard library types: +```swift +Span +RawSpan +UnsafeBufferPointer +UnsafeMutableBufferPointer +Slice> +Slice> +UnsafeRawBufferPointer +UnsafeMutableRawBufferPointer +Slice +Slice +``` +Some of the types in the list above already have a subset of the `extracting()` functions; their support will be rounded out to the full set. + + +## Detailed design + +The general declarations for these functions is as follows: +```swift +/// Returns an extracted slice over the items within +/// the supplied range of positions. +/// +/// Traps if any position within the range is invalid. +@_lifetime(copy self) +public func extracting(_ bounds: Range) -> Self + +/// Returns an extracted slice over the items within +/// the supplied range of positions. +/// +/// Traps if any position within the range is invalid. +@_lifetime(copy self) +public func extracting(_ bounds: some RangeExpression) -> Self + +/// Returns an extracted slice over all items of this container. +@_lifetime(copy self) +public func extracting(_: UnboundedRange) -> Self + +/// Returns an extracted slice over the items within +/// the supplied range of positions. +/// +/// This function does not validate `bounds`; this is an unsafe operation. +@unsafe @_lifetime(copy self) +public func extracting(unchecked bounds: Range) -> Self + +/// Returns an extracted slice over the items within +/// the supplied range of positions. +/// +/// This function does not validate `bounds`; this is an unsafe operation. +@unsafe @_lifetime(copy self) +public func extracting(unchecked bounds: ClosedRange) -> Self + +/// Returns an extracted slice over the initial elements +/// of this container, up to the specified maximum length. +@_lifetime(copy self) +public func extracting(first maxLength: Int) -> Self + +/// Returns an extracted slice excluding +/// the given number of trailing elements. +@_lifetime(copy self) +public func extracting(droppingLast k: Int) -> Self + +/// Returns an extracted slice containing the final elements +/// of this container, up to the given maximum length. +@_lifetime(copy self) +public func extracting(last maxLength: Int) -> Self + +/// Returns an extracted slice excluding +/// the given number of initial elements. +@_lifetime(copy self) +public func extracting(droppingFirst k: Int) -> Self +``` +For escapable types, the `@_lifetime` attribute is not applied. + + +### Usage hints + +The `extracting()` pattern, while not completely new, is still a departure over the slice pattern established by the `Collection` protocol. For `Span`, `RawSpan`, `MutableSpan` and `MutableRawSpan`, we can add unavailable subscripts and function with hints towards the corresponding `extracting()` function: + +```swift +@available(*, unavailable, renamed: "extracting(_ bounds:)") +public subscript(bounds: Range) -> Self { extracting(bounds) } + +@available(*, unavailable, renamed: "extracting(first:)") +public func droppingFirst(_ k: Int) -> Self { extracting(first: k) } +``` + +## Source compatibility +This proposal is additive and source-compatible with existing code. + +## ABI compatibility +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption +The additions described in this proposal require a new version of the Swift standard library. + +## Alternatives considered +This is an extension of an existing pattern. We are not considering a different pattern at this time. + +## Future directions +#### Disambiguation over ownership type +The `extracting()` functions proposed here are borrowing. `MutableSpan` has versions defined as mutating, but it could benefit from consuming ones as well. In general there could be a need for all three ownership variants of a given operation (`borrowing`, `consuming`, or `mutating`.) In order to handle these variants, we could establish a pattern for disambiguation by name, or we could invent new syntax to disambiguate by ownership type. This is a complex topic left to future proposals. + +## Acknowledgements +Thanks to Karoy Lorentey and Tony Parker. + diff --git a/proposals/0489-codable-error-printing.md b/proposals/0489-codable-error-printing.md new file mode 100644 index 0000000000..6dd70e71a5 --- /dev/null +++ b/proposals/0489-codable-error-printing.md @@ -0,0 +1,154 @@ +# Improve `EncodingError` and `DecodingError`'s printed descriptions + +* Proposal: [SE-0489](0489-codable-error-printing.md) +* Authors: [Zev Eisenberg](https://github.com/ZevEisenberg) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Accepted** +* Implementation: https://github.com/swiftlang/swift/pull/80941 +* Review: ([pitch](https://forums.swift.org/t/pitch-improve-encodingerror-and-decodingerror-s-printed-descriptions/79872)) ([review](https://forums.swift.org/t/se-0489-improve-encodingerror-and-decodingerrors-printed-descriptions/81021)) ([acceptance](https://forums.swift.org/t/accepted-se-0489-improve-encodingerror-and-decodingerrors-printed-descriptions/81380)) + +## Introduction + +`EncodingError` and `DecodingError` do not specify any custom debug description. The default descriptions bury the useful information in a format that is difficult to read. Less experienced developers may assume they are not human-readable at all, even though they contain useful information. The proposal is to conform `EncodingError` and `DecodingError` to `CustomDebugStringConvertible` and provide nicely formatted debug output. + +## Motivation + +Consider the following example model structs: + +```swift +struct Person: Codable { + var name: String + var home: Home +} + +struct Home: Codable { + var city: String + var country: Country +} + +struct Country: Codable { + var name: String + var population: Int +} +``` + +Now let us attempt to decode some invalid JSON. In this case, it is missing a field in a deeply nested struct. + +```swift +// Note missing "population" field +let jsonData = Data(""" +[ + { + "name": "Ada Lovelace", + "home": { + "city": "London", + "country": { + "name": "England" + } + } + } +] +""".utf8) + +do { + _ = try JSONDecoder().decode([Person].self, from: jsonData) +} catch { + print(error) +} +``` + +This outputs the following: + +`keyNotFound(CodingKeys(stringValue: "population", intValue: nil), Swift.DecodingError.Context(codingPath: [_CodingKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "home", intValue: nil), CodingKeys(stringValue: "country", intValue: nil)], debugDescription: "No value associated with key CodingKeys(stringValue: \"population\", intValue: nil) (\"population\").", underlyingError: nil))` + +All the information you need is there: +- The kind of error: a missing key +- Which key was missing: `"population"` +- The path of the value that had a missing key: index 0, then key `"home"`, then key `"country"` +- The underlying error: none, in this case + +However, it is not easy or pleasant to read such an error, particularly when dealing with large structures or long type names. It is common for newer developers to assume the above output is some kind of log spam and not even realize it contains exactly the information they are looking for. + +## Proposed solution + +Conform `EncodingError` and `DecodingError` to `CustomDebugStringConvertible` and provide a clean, readable debug description for each. + +Complete examples of the before/after diffs are available in the description of the [implementation pull request](https://github.com/swiftlang/swift/pull/80941) that accompanies this proposal. + +**Note 1:** This proposal is _not_ intended to specify an exact output format, and any examples are not a guarantee of current or future behavior. You are still free to inspect the contents of thrown errors directly if you need to detect specific problems. + +**Note 2:** The output could be further improved by modifying `JSONDecoder` to write a better debug description. See [Future Directions](#future-directions) for more. + +## Detailed design + +```swift +@available(SwiftStdlib 6.2, *) +extension EncodingError: CustomDebugStringConvertible { + public var debugDescription: String {...} +} + +@available(SwiftStdlib 6.2, *) +extension DecodingError: CustomDebugStringConvertible { + public var debugDescription: String {...} +} +``` + +## Source compatibility + +The new conformance changes the result of converting an `EncodingError` or `DecodingError` value to a string. This changes observable behavior: code that attempts to parse the result of `String(describing:)` or `String(reflecting:)` can be misled by the change of format. + +However, the documentation of these interfaces explicitly state that when the input type conforms to none of the standard string conversion protocols, then the result of these operations is unspecified. + +Changing the value of an unspecified result is not considered to be a source incompatible change. + +## ABI compatibility + +The proposal conforms two previously existing stdlib types to a previously existing stdlib protocol. This is technically an ABI breaking change: on ABI-stable platforms, we may have preexisting Swift binaries that implement a retroactive `CustomDebugStringConvertible` conformance, or binaries that assume that the existing error types do _not_ conform to the protocol. + +We do not expect this to be an issue in practice, since checking an arbitrary error for conformance to `CustomDebugStringConvertible` at run-time seems unlikely. In the event that it now conforms where it didn't before, it will presumably use the new implementation instead of whatever fallback was being provided previously. + +## Implications on adoption + +### Conformance to `CustomDebugStringConvertible` + +The conformance to `CustomDebugStringConvertible` is not backdeployable. As a result, code that runs on ABI-stable platforms with earlier versions of the standard library won't output the new debug descriptions. + +### `debugDescription` Property + +It is technically possible to backdeploy the `debugDescription` property, but without the protocol conformance, it is of limited utility. + +## Future directions + +### Better error generation from Foundation encoders/decoders + +The debug descriptions generated in Foundation sometimes contain the same information as the new debug descriptions from this proposal. A future change to the standard JSON and Plist encoders and decoders could provide more compact debug descriptions once they can be sure they have the new standard library descriptions available. They could also use a more compact description when rendering the description of a `CodingKey`. Take, for example: + +``` +Debug description: No value associated with key CodingKeys(stringValue: "population", intValue: nil) ("population"). +``` + +The `CodingKeys(stringValue: "population", intValue: nil) ("population")` part is coming from the default `description` of `CodingKey`, plus an extra parenthesized string value at the end for good measure. The Foundation (de|en)coders could construct a more compact description that does not repeat the key, just like we do within this proposal in the context of printing a coding path. + +### Print context of surrounding lines in source data + +When a decoding error occurs, in addition to printing the path, the error message could include some surrounding lines from the source data. This was explored in this proposal's antecedent, [UsefulDecode](https://github.com/ZevEisenberg/UsefulDecode). But more detailed messages would require passing more context data from the decoder and changing the public interface of `DecodingError` to carry more data. This option is best left as something to think about as [we design `Codable`'s successor](https://forums.swift.org/t/the-future-of-serialization-deserialization-apis/78585). But just to give an example of the _kind_ of context that could be provided (please do not read anything into the specifics of the syntax; this is a sketch, not a proposal): + +``` +Value not found: expected 'name' (String) at [0]/address/city/birds/[1]/name, got: +{ + "feathers" : "some", + "name" : null +} +``` + +## Alternatives considered + +We could conform `EncodingError` and `DecodingError` to `CustomStringConvertible` instead of `CustomDebugStringConvertible`. The use of the debug-flavored protocol emphasizes that the new descriptions aren't intended to be used outside debugging contexts. This is in keeping with the precedent set by [SE-0445](0445-string-index-printing.md). + +We could change `CodingKey.description` to return the bare string or int value, which would improve the formatting and reduce duplication as seen in [Proposed solution](#proposed-solution). But changing the exsting implementation of an existing public method seems needlessly risky, as existing code may (however inadvisably) be depending on the format of the current `description`. Additionally, the encoders and decoders in Foundation should not depend on implementation details of `CodingKey.description` that are not guaranteed. If we want the encoders/decoders to produce better formatting, they should be responsible for generating those strings directly. See [further discussion in the PR](https://github.com/swiftlang/swift/pull/80941#discussion_r2064277369). + +## Acknowledgments + +This proposal follows in the footsteps of [SE-0445](0445-string-index-printing.md). Thanks to [Karoy Lorentey](https://github.com/lorentey) for writing that proposal, and for flagging it as similar to this one. + +Thanks to Kevin Perry [for suggesting](https://forums.swift.org/t/the-future-of-serialization-deserialization-apis/78585/77) that this would make a good standalone change regardless of the direction of future serialization tools, and for engaging with the PR from the beginning. diff --git a/proposals/0490-environment-constrained-shared-libraries.md b/proposals/0490-environment-constrained-shared-libraries.md new file mode 100644 index 0000000000..39e3624387 --- /dev/null +++ b/proposals/0490-environment-constrained-shared-libraries.md @@ -0,0 +1,163 @@ +# Environment Constrained Shared Libraries + +* Proposal: [SE-0490](0490-environment-constrained-shared-libraries.md) +* Authors: [tayloraswift](https://github.com/tayloraswift) +* Review Manager: [Alastair Houghton](https://github.com/al45tair) +* Status: **Active Review (September 5th...September 18th, 2025)** +* Implementation: [swiftlang/swift-package-manager#8249](https://github.com/swiftlang/swift-package-manager/pull/8249) +* Documentation: [How to use Environment-Constrained Shared Libraries](https://github.com/swiftlang/swift-package-manager/blob/1eaf59d2facc74c88574f38395aa49983b2badcc/Documentation/ECSLs.md) +* Bugs: [SR-5714](https://github.com/swiftlang/swift-package-manager/issues/5714) +* Review: ([pitch](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605)) ([review](https://forums.swift.org/t/se-0490-environment-constrained-shared-libraries/81975)) + +## Introduction + +SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for **Environment Constrained Shared Libraries**, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Environment Constrained Shared Libraries through the existing `.artifactbundle` format. + +Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605) + +Example Producer: [swift-dynamic-library-example](https://github.com/tayloraswift/swift-dynamic-library-example) + +Example Consumer: [swift-dynamic-library-example-client](https://github.com/tayloraswift/swift-dynamic-library-example-client) + +## Motivation + +Many of us in the Server World have a Big App with a small component that changes very rapidly, much more rapidly than the rest of the App. This component might be something like a filter, or an algorithm, or a plugin that is being constantly tuned. + +We could, for argument’s sake, try and turn this component into data that can be consumed by the Big App, which would probably involve designing a bytecode and an interpreter, and maybe even a whole interpreted domain-specific programming language. But that is very hard and we would rather just write this thing in Swift, and let Swift code call Swift code. + +While macOS has Dynamic Library support through XCFrameworks, on Linux we currently have to recompile the Big App from source and redeploy the Big App every time the filter changes, and we don’t want to do that. What we really want instead is to have the Big App link the filter as a Dynamic Library, and redeploy the Dynamic Library as needed. + + +## Proposed solution + +On Linux, there are a lot of obstacles to having fully general support for Dynamic Libraries. Swift is not ABI stable on Linux, and Linux itself is not a single platform but a wide range of similar platforms that provide few binary compatibility guarantees. This means it is pretty much impossible for a public Swift library to vend precompiled binaries that will Just Work for everyone, and we are not going to try to solve that problem in this proposal. + +Instead, we will focus on **Environment Constrained Shared Libraries** (ECSLs). We choose this term to emphasize the distinction between our use case and fully general Dynamic Libraries. + +### Target environment + +Unlike fully general Dynamic Libraries, you would distribute Environment Constrained Shared Libraries strictly for controlled consumption within a known environment, such as a fleet of servers maintained by a single organization. + +ECSLs are an advanced tool, and maintaining the prerequisite environment to deploy them safely is neither trivial nor recommended for most users. + +The organization that distributes an ECSL is responsible for defining what exactly constitutes a “platform” for their purposes. An organization-defined platform is not necessarily an operating system or architecture, or even a specific distribution of an operating system. A trivial example of two such platforms might be: + +1. Ubuntu 24.04 with the Swift 6.1.2 runtime installed at `/home/ubuntu/swift` +2. Ubuntu 24.04 with the Swift 6.1.2 runtime installed at `/home/ubuntu/swift-runtime` + +Concepts like Platform Triples are not sufficient to describe an ECSL deployment target. Even though both “platforms” above would probably share the Triple `aarch64-unknown-linux-gnu`, Swift code compiled (without `--static-swift-stdlib`) for one would never be able to run on the other. + +Organizations will add and remove environments as needed, and trying to define a global registry of all possible environments is a non-goal. + +The proposed ECSL distribution format does not support shipping multiple variants of ECSLs targeting multiple environments in the same Artifact Bundle, nor does it specify a standardized means for identifying the environment in which a particular ECSL is intended to execute in. +Users are responsible for computing the correct URL of the Artifact Bundle for the environment they are building for, possibly within the package manifest. Swift tooling will not, on its own, diagnose or prevent the installation of an incompatible ECSL. + +### Creating ECSLs + +To compile an ECSL, you just need to build an ordinary SwiftPM library product with the `-enable-library-evolution` flag. This requires no modifications to SwiftPM. + +You would package an ECSL as an `.artifactbundle` just as you would an executable, with the following differences: + +- The `info.json` must have `schemaVersion` set to `1.2` or higher. +- The artifact type must be `dynamicLibrary`, a new enum case introduced in this proposal. +- The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. +- The artifact payload must include the `.swiftinterface` file corresponding to the actual library object. + +Because SwiftPM is not (and cannot be) aware of a particular organization’s set of deployment environments, this enforces the requirement that each environment must have its own Artifact Bundle. + +The organization that distributes the ECSL is responsible for upholding ABI stability guarantees, including the exact Swift compiler and runtime versions needed to safely consume the ECSL. + + +### Consuming ECSLs + +To consume an ECSL, you would add a `binaryTarget` to your `Package.swift` manifest, just as you would for an executable. Because organizations are responsible for defining their set of supported environments, they are also responsible for defining the URLs that the Artifact Bundles for each environment are hosted under, so there are no new fields in the `PackageDescription` API. + +We expect that the logic for selecting the correct ECSL for a given environment would live within the `Package.swift` file, that it would be highly organization-specific, and that it would be manipulated using existing means such as environment variables. + + +### Deploying ECSLs + +Deploying ECSLs does not involve SwiftPM or Artifact Bundles at all. You would deploy an ECSL by copying the latest binaries to the appropriate `@rpath` location on each machine in your fleet. The `@rpath` location is part of the organization-specific environment definition, and is not modeled by SwiftPM. + +Some organizations might choose to forgo the `@rpath` mechanism entirely and simply install the ECSLs in a system-wide location. + + +## Detailed design + +### Schema extensions + +We will extend the `ArtifactsArchiveMetadata` schema to include a new `dynamicLibrary` case in the `ArtifactType` enum. + +```diff +public enum ArtifactType: String, RawRepresentable, Decodable { + case executable ++ case dynamicLibrary + case staticLibrary + case swiftSDK +} +``` + +This also bumps the latest `schemaVersion` to `1.2`. + + +### Artifact Bundle layout + +Below is an example of an `info.json` file for an Artifact Bundle containing a single library called `MyLibrary`. + +```json +{ + "schemaVersion": "1.2", + "artifacts": { + "MyLibrary": { + "type": "dynamicLibrary", + "version": "1.0.0", + "variants": [{ "path": "MyLibrary" }] + } + } +} +``` + +The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. An ECSL Artifact Bundle can contain multiple libraries at the top level. + +Below is an example of the layout of an Artifact Bundle containing a single library called `MyLibrary`. Only the `info.json` must appear at the root of the Artifact Bundle; all other files can appear at whatever paths are defined in the `info.json`, as long as they are within the Artifact Bundle. + +```text +📂 example.artifactbundle + 📂 MyLibrary + ⚙️ libMyLibrary.so + 📝 MyLibrary.swiftinterface + 📝 info.json +``` + +A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. ECSLs will be supported on macOS, although we expect this will be an exceedingly rare use case, as this need is already well-served by the XCFramework. + + +## Security + +ECSLs are not intended for public distribution, and are not subject to the same security concerns as public libraries. Organizations that distribute ECSLs are responsible for ensuring that the ECSLs are safe to consume. + + +## Impact on existing packages + +There will be no impact on existing packages. All Artifact Bundle schema changes are additive. + + +## Alternatives considered + +### Extending Platform Triples to model deployment targets + +SwiftPM currently uses Platform Triples to select among artifact variants when consuming executables. This is workable because it is usually feasible to build executables that are portable across the range of platforms encompassed by a single Platform Triple. + +We could extend Platform Triples to model ECSL deployment targets, but this would privilege a narrow set of predefined deployment architectures, and if you wanted to add a new environment, you would have to modify SwiftPM to teach it to recognize the new environment. + +### Supporting multiple variants of an ECSL in the same Artifact Bundle + +We could allow an Artifact Bundle to contain multiple variants of an ECSL, but we would still need to support a way to identify those variants, which in practice forces SwiftPM to become aware of organization-defined environments. + +We also don’t see much value in this feature, as you would probably package and upload ECSLs using one CI/CD workflow per environment anyway. Combining artifacts would require some kind of synchronization mechanism to await all pipelines before fetching and merging bundles. + +One benefit of merging bundles would be that it reduces the number of checksums you need to keep track of, but we expect that most organizations will have a very small number of supported environments, with new environments continously phasing out old environments. + +### Using a different `ArtifactType` name besides `dynamicLibrary` + +We intentionally preserved the structure of the `variants` list in the `info.json` file, despite imposing the current restriction of one variant per library, in order to allow this format to be extended in the future to support fully general Dynamic Libraries. diff --git a/proposals/0491-module-selectors.md b/proposals/0491-module-selectors.md new file mode 100644 index 0000000000..ad86323f3b --- /dev/null +++ b/proposals/0491-module-selectors.md @@ -0,0 +1,776 @@ +# Module selectors for name disambiguation + +* Proposal: [SE-0491](0491-module-selectors.md) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Freddy Kellison-Linn](https) +* Status: **Accepted** +* Bug: [swiftlang/swift#53580](https://github.com/swiftlang/swift/issues/53580) (SR-11183) +* Implementation: [swiftlang/swift#34556](https://github.com/swiftlang/swift/pull/34556) +* Review: ([pitch](https://forums.swift.org/t/pitch-module-selectors/80835)) ([review](https://forums.swift.org/t/se-0491-module-selectors-for-name-disambiguation/82124)) ([acceptance](https://forums.swift.org/t/accepted-se-0491-module-selectors-for-name-disambiguation/82589)) + +Previously pitched in: + +* [Pitch: Fully qualified name syntax](https://forums.swift.org/t/pitch-fully-qualified-name-syntax/28482) + +## Introduction + +We propose that Swift's grammar be extended so that, wherever an identifier +is written in source code to reference a declaration, it can be prefixed by +`ModuleName::` to disambiguate which module the declaration is expected to +come from. This syntax will provide a way to resolve several types of name +ambiguities and conflicts. + +## Motivation + +Swift's name lookup rules promote its goal of allowing code to be written in a +very clean, readable style. However, in some circumstances it can be very +difficult to unambiguously reference the declaration you want. + +### Background + +When Swift looks up a name in your source code to find the declaration it +refers to, that lookup can be either *qualified* or *unqualified*. Qualified +lookups are restricted to looking inside a certain declaration, while +unqualified lookups search more broadly. For example, in a chain of names such +as: + +```swift +mission().booster().launch() +``` + +`booster()` can only refer to members of whatever type is returned by +`mission()`, and `launch()` can only refer to members of whatever type is +returned by `booster()`, so Swift will find them using a qualified lookup. +`mission()`, on the other hand, does not have to be a member of some specific +type, so Swift will find that declaration using an unqualified lookup. + +> **Note**: Although the examples given here mostly concern uses in +> expressions, qualified and unqualified lookups are also used for names in +> type syntax, such as `Mission.Booster.Launch`. The exact lookup rules are +> slightly different but the principles are the same. + +Both kinds of lookups are slightly sensitive to context in that, since the +acceptance of [SE-0444 Member Import Visibility][SE-0444], +they are both limited to declarations imported in the current source file; +however, unqualified lookups take *much* more than just that into account. They +search through any enclosing scopes to find the "closest" use of that name. For +example, in code like: + +```swift +import RocketEngine +import IonThruster + +extension Mission { + struct Booster { + func launch(_ crew: Int) { + let attempt = 1 + ignite() + } + } +} +``` + +Swift will look for `ignite` in the following places: + +1. The local declarations inside `launch(_:)` +2. The parameters to `launch(_:)` +3. Instance members and generic parameters of the enclosing type `Booster` + (including its extensions, superclasses, conformances, etc.) +4. Static members and generic parameters of the enclosing type `Mission` +5. Top-level declarations in this module +6. Top-level declarations in other imported modules +7. The names of imported modules + +These rules are a little complicated when written out like this, but their +effect is pretty simple: Swift finds whichever `ignite` is in the "closest" +scope to the use site. If both `Booster` and `Mission` have an `ignite`, for +example, Swift will use the one in `Booster` and ignore the one in `Mission`. + +Of particular note is the last place Swift looks: the names of imported +modules. This is intended to help with situations where two modules have +declarations with the same name. For example, if both `RocketEngine` and +`IonThruster` declare an `ignite()`, `RocketEngine.ignite()` will find `ignite` +using a qualified lookup inside the module `RocketEngine`, filtering out the +one in `IonThruster`. This works in simple cases, but it breaks down in a +number of complicated ones. + +### Unqualified lookups are prone to shadowing + +Swift does not prevent declarations in different scopes from having the same +name. For example, there's nothing preventing you from having both a top-level +type and a nested type with the same name: + +```swift +struct Scrubber { ... } + +struct LifeSupport { + struct Scrubber { ... } +} +``` + +This means that the same name can have different meanings in different places: + +```swift +// This returns the top-level `Scrubber`: +func makeScrubber() -> Scrubber { ... } + +extension LifeSupport { + // This returns `LifeSupport.Scrubber`: + func makeScrubber() -> Scrubber { ... } +} +``` + +Specifically, we say that within the extension, `LifeSupport.Scrubber` +*shadows* the top-level `Scrubber`. + +This poses certain challenges—especially for mechanically-generated code, such +as module interface files—but it's usually not completely insurmountable +because you can qualify a top-level declaration with its module name. However, +it becomes a problem if *the module name itself* is shadowed by a type with the +same name: + +```swift +// Module RocketEngine +public struct RocketEngine { ... } +public struct Fuel { ... } + +// Another module +import RocketEngine + +_ = RocketEngine.Fuel() // Oops, this is looking for a nested type in the + // struct RocketEngine.RocketEngine! +``` + +In this situation, we can no longer qualify top-level declarations with module +names. That makes code generation *really* complicated, because there is no +syntax that works reliably—qualifying will help with some failures but cause +others. + +That may sound like a farfetched edge case, but it's surprisingly common for a +module to contain a type with the same name. For instance, the `XCTest` module +includes an `XCTest` class, which is a base class for `XCTestCase` and +`XCTestSuite`. To avoid this kind of trouble, developers must be careful to +give modules different names from the types inside them—the `Observation` +module, for example, might have been called `Observable` if it didn't have a +type with that name. + +### Qualified lookups can be unresolvably ambiguous + +Extensions create the possibility that a type may have two members with the +same name and similar or even outright conflicting overload signatures, +distinguished only by being in different modules. This is not a problem for +Swift's ABI because the mangled name of an extension member includes the +module it was declared in; however, there is no way to add a module name to +an already-qualified non-top-level lookup, so there's no way to express this +distinction in the surface language. Developers' only option may be to fiddle +with their imports in an attempt to make sure the desired member is the only +one that's visible. + +### Macros don't support module qualification + +Macros cannot have members--the grammar of a macro expansion allows only a +single identifier, and any subsequent `.` is taken to be a member lookup on +the expansion--so there is currently no way to qualify a macro expansion with +a module name. This limitation was discussed during the [second review of SE-0382][SE-0382-review-2] +and the author suggested the only viable solution was to add a new, +grammatically-distinguishable syntax for module qualification. + +### These problems afflict module interfaces, but aren't unique to them + +These issues show up most often in module interfaces because the compiler +needs to generate syntax that reliably resolves to a specific declaration, but +the rules' sensitivity to context and module contents (which might change over +time!) makes that very difficult. In practice, the compiler does not attempt to +fully account for shadowing and name conflicts--by default it qualifies names +as fully as the language allows (which works about 95% of the time) and offers +a number of (undocumented) workaround flags to adjust that which are added by a +maintainer when they discover that their module is in the remaining 5%. These +flags aren't enabled automatically, though, and they don't affect the module +interfaces of downstream modules which need to reference affected declarations. +In short, the situation is a mess. + +It's important to keep in mind, though, that this doesn't *just* affect module +interfaces and generated code. Code written by humans can also run into these +issues; it's just that a person will notice the build error and fiddle with +their code until they get something that works. It therefore makes sense to +introduce a new syntax that can be used by both machines and humans. + +### Separate modules make this uniquely severe + +While problematic conflicts can sometimes occur between two declarations in a +single module, the authors believe that per-module disambiguation is the right +approach because shadowing within a module is much easier to detect and +resolve. The developer will generally notice shadowing problems when they build +or test their code, and since they control both the declaration site and the +use site, they have options to resolve any problems that are not otherwise +available (like renaming declarations or tweaking their overload signatures). +The compiler also detects and prevents outright conflicts within a specific +module, such as two extensions declaring the exact same member, which it would +allow if the declarations were in different modules. + +## Proposed solution + +We propose adding *module selectors* to the language. A module selector is +spelled `::` and can be placed before an identifier to indicate +which module it is expected to come from: + +```swift +_ = RocketEngine::Fuel() // Picks up the `Fuel` in `RocketEngine`, bypassing + // any other `Fuel`s that might be in scope +``` + +On an unqualified lookup, a module selector also indicates that lookup should +start at the top level, skipping over the declarations in contextually-visible +scopes: + +```swift +// In module NASA + +struct Scrubber { ... } + +struct LifeSupport { + struct Scrubber { ... } +} + +extension LifeSupport { + // This returns the top-level `Scrubber` + func makeMissionScrubber() -> NASA::Scrubber { ... } +} +``` + +Module selectors may also be placed on qualified lookups to indicate which +module an extension member should belong to: + +```swift +// In module IonThruster +extension Spacecraft { + public struct Engine { ... } +} + +// In module RocketEngine +extension Spacecraft { + public struct Engine { ... } +} + +// In module NASA +import IonThruster +import RocketEngine + +func makeIonThruster() -> Spacecraft.IonThruster::Engine { ... } +``` + +Module selectors are permitted at locations in the type and expression syntax +where a declaration from elsewhere is referenced by name. However, it is +invalid to use one on the name of a *new* declaration: + +```swift +struct NASA::Scrubber { // Invalid--new declarations are always in the current module + ... +} +``` + +We chose this syntax—module name plus `::` operator prefixing the name they +qualify—because `::` is unused in Swift (it can't even be a custom operator) +and because using `::` in this fashion is highly precedented in other +languages. (C++, PHP, Java, and Rust all use it to indicate that the name on +the right should be looked up inside the scope on the left; Ruby and Perl use +it *specifically* to look up declarations inside modules.) + +## Detailed design + +### Grammar and parsing + +A module selector has the following grammar: + +> *module-selector* → *identifier* `::` + +The following productions may now optionally include a module selector (changes are in bold): + +> *type-identifier* → ***module-selector?*** *type-name* *generic-argument-clause?* | ***module-selector?*** *type-name* *generic-argument-clause?* `.` *type-identifier* +> +> *primary-expression* → ***module-selector?*** *identifier* *generic-argument-clause?* +> +> *implicit-member-expression* → `.` ***module-selector?*** *identifier*
+> *implicit-member-expression* → `.` ***module-selector?*** *identifier* `.` *postfix-expression* +> +> *macro-expansion-expression* → `#` ***module-selector?*** *identifier* *generic-argument-clause?* *function-call-argument-clause?* *trailing-closures?* +> +> *key-path-component* → ***module-selector?*** *identifier* *key-path-postfixes?* | *key-path-postfixes* +> +> *function-call-argument* → ***module-selector?*** *operator* | *identifier* `:` ***module-selector?*** *operator* +> +> *initializer-expression* → *postfix-expression* `.` ***module-selector?*** `init`
+> *initializer-expression* → *postfix-expression* `.` ***module-selector?*** `init` `(` *argument-names* `)` +> +> *explicit-member-expression* → *postfix-expression* `.` ***module-selector?*** *identifier* *generic-argument-clause?*
+> *explicit-member-expression* → *postfix-expression* `.` ***module-selector?*** *identifier* `(` *argument-names* `)` +> +> *attribute-name* → ***module-selector?*** *identifier* +> +> *enum-case-pattern* → *type-identifier?* `.` ***module-selector?*** *enum-case-name* *tuple-pattern?* + +Additionally, a new production allows a scoped `import` declaration to use a +module selector and identifier instead of an import path: + +> *import-declaration* → *attributes?* `import` *import-kind?* *import-path*
+> ***import-declaration* → *attributes?* `import` *import-kind* *module-selector* *identifier*** + +Note that this new *import-declaration* production does not allow a submodule +to be specified. Use the old `.`-operator-based syntax for submodules. + +#### Token-level behavior + +The `::` operator may be separated from its *identifier* by any whitespace, +including newlines. However, the `::` operator must *not* be separated from the +token after it by a newline: + +```swift +NationalAeronauticsAndSpaceAdministration:: + RocketEngine // Invalid +NationalAeronauticsAndSpaceAdministration + ::RocketEngine // OK +``` + +> **Note**: This restriction aids in recovery when parsing incomplete code; +> the member-lookup `.` operator follows a similar rule. + +If the token after the `::` operator is a keyword, it will be treated as an +ordinary identifier unless it would have special meaning: + +```swift +print(default) // Invalid; 'default' is a keyword and needs backticks +print(NASA.default) // OK under SE-0071 +print(NASA::default) // OK under this proposal +``` + +Depending on context, the following keywords may still be treated as special in +expressions: + +* `deinit` +* `init` +* `subscript` + +> **Note**: This behavior is analogous to [SE-0071 Allow (most) keywords in member references][SE-0071]. + +Similarly, attributes that use a module selector will always be treated as +custom attributes, not built-in attributes. (Put another way, built-in +attributes do not belong to *any* module—not even `Swift`.) Like all custom +attributes, any arguments must be valid expressions. + +```swift +@Swift::available(macOS 15.0.1, *) // Invalid; not parsed as the built-in `@available` +class X {} +``` + +#### Patterns + +Module selectors are allowed in *enum-case-pattern* and in *type* and +*expression* productions nested inside patterns. However, *identifier-pattern* +is unmodified and does *not* permit a module selector, even in shorthand +syntaxes designed to declare a shadow of an existing variable. If a module +selector is needed, you must use an explicit initializer expression. + +```swift +if let NASA::rocket { ... } // Invalid +if let rocket = NASA::rocket { ... } // OK + +Task { [NASA::rocket] in ... } // Invalid +Task { [rocket = NASA::rocket] in ... } // OK +``` + +#### Operator and precedence group declarations + +The *precedence-group-name* production is unmodified and does not permit +a module selector. Precedence group names exist in a separate namespace from +other identifiers and no need for this feature has been demonstrated. + +#### Parsed declaration names + +A parsed declaration name, such as the name in an `@available(renamed:)` +argument, may use module selectors on the declaration's base name and context +names. + +```swift +@available(*, deprecated, renamed: "NASA::launch(_:from:)") // OK +public func launch(_ mission: Mission) { + launch(mission, from: LaunchPad.default) +} +``` + +Module selectors are not valid on base names in clang `swift_name` and +`swift_async_name` attributes, since these specify the name of the current +declaration, rather than referencing a different declaration. + +> **Note**: Clang Importer currently cannot apply import-as-member `swift_name` +> or `swift_async_name` attributes that name a context in a different module, +> but if this limitation is ever lifted, module selectors ought to be supported +> on context names in these clang attributes. + +#### Syntaxes reserved for future directions + +It is never valid to write two module selectors in a row; if you want to access +a declaration which belongs to a clang submodule, you should just write the +top-level module name in the module selector. + +It is never valid to write a keyword, operator, or `_` in place of a module +name; if a module's name would be mistaken for one of these, it must be +wrapped in backticks to form an identifier. + +### Effects on lookup + +When a reference to a declaration is prefixed by a module selector, only +declarations declared in, or re-exported by, the indicated module will be +considered as candidates. All other declarations will be filtered out. + +For example, in the following macOS code: + +```swift +import Foundation + +class NSString {} + +func fn(string: Foundation::NSString) {} +``` + +`string` will be of type `Foundation.NSString`, rather than the `NSString` +class declared in the same file. Because the AppKit module +re-exports Foundation, this example would also behave the same way: + +```swift +import AppKit + +class NSString {} + +func fn(string: AppKit::NSString) {} +``` + +> **Note**: Allowing re-exports ensures that "hoisting" a type from its +> original module up to another module it imports is not a source-breaking +> change. It also helps with situations where developers don't realize where a +> given type is declared; for instance, many developers believe `NSObject` is +> declared in `Foundation`, not `ObjectiveC`. + +Additionally, when a reference to a declaration prefixed by a module selector +is used for an unqualified lookup, the lookup will begin at the module-level +scope, skipping any intervening enclosing scopes. That means a top-level +declaration will not be shadowed by local variables, parameters, generic +parameters, or members of enclosing types: + +```swift +// In module MyModule + +class Shadowed { + struct Shadowed { + let Shadowed = 42 + func Shadowed(Shadowed: () -> Void) { + let Shadowed = "str" + let x = MyModule::Shadowed() // refers to top-level `class Shadowed` + } + } +} +``` + +A module selector can only rule out declarations that might otherwise have been +chosen instead of the desired declaration; it cannot access a declaration which +some other language feature has ruled out. For example, if a declaration is +inaccessible because of access control or hasn't been imported into the current +source file, a module selector will not allow it to be accessed. + +#### Member types of type parameters + +A member type of a type parameter must not be qualified by a module selector. + +```swift +func fn(_: T) where T.Swift::ID == Int { // not allowed + ... +} +``` + +This is because, when a generic parameter conforms to two protocols that have +associated types with the same name, the member type actually refers to *both* +of those associated types. It doesn't make sense to use a module name to select +one associated type or the other--it will always encompass both of them. + +(In some cases, a type parameter's member type might end up referring to a +concrete type—typically a typealias in a protocol extension–which +theoretically *could* be disambiguated in this way. However, in these +situations you could always use the protocol instead of the generic parameter +as the base (and apply a module selector to it if needed), so we've chosen not +to make an exception for them.) + +## Source compatibility + +This change is purely additive; it only affects the behavior of code which uses +the new `::` token. In the current language, this sequence can only appear +in valid Swift code in the selector of an `@objc` attribute, and the parser +has been modified to split the token when it is encountered there. + +## ABI compatibility + +This change does not affect the ABI of existing code. The Swift compiler has +always resolved declarations to a specific module and then embedded that +information in the ABI's symbol names; this proposal gives developers new ways +to influence those resolution decisions but doesn't expand the ABI in any way. + +## Implications on adoption + +Older compilers will not be able to parse source code which uses module +selectors. This means package authors may need to increase their tools version +if they want to use the feature, and authors of inlinable code may need to +weigh backwards compatibility concerns. + +Similarly, when a newer compiler emits module selectors into its module +interfaces, older compilers won't be able to understand those files. This isn't +a dealbreaker since Swift does not guarantee backwards compatibility for module +interfaces, but handling it will require careful staging and there may be a +period where ABI-stable module authors must opt in to emitting module +interfaces that use the feature. + +## Future directions + +### Special syntax for the current module + +We could allow a special token, or no token, to be used in place of the module +name to force a lookup to start at the top level, but not restrict it to a +specific module. Candidates include: + +```swift +Self::ignite() +_::ignite() +*::ignite() +::ignite() +``` + +These syntaxes have all been intentionally kept invalid (a module named `Self`, +for instance, would have to be wrapped in backticks: `` `Self`::someName ``), +so one of them can be added later if there's demand for it. + +### Disambiguation for subscripts + +There is currently no way to add a module selector to a use of a subscript. We +could add support for a syntax like: + +```swift +myArray.Swift::[myIndex] +``` + +### Disambiguation for conformances + +Retroactive conformances have a similar problem to extension members—the ABI +distinguishes between otherwise identical conformances in different modules, +but the surface syntax has no way to resolve any ambiguity—so a feature which +addressed them might be nice. However, there is no visible syntax associated +with use of a conformance that can be qualified with a module selector, so it's +difficult to address as part of this proposal. + +It's worth keeping in mind that [SE-0364's introduction of `@retroactive`][SE-0364] +reflects a judgment that retroactive conformances should be used with care. The +absence of such a feature is one of the complications `@retroactive` is meant +to flag. + +### Support selecting conflicting protocol requirements + +Suppose that a single type conforms to two protocols with conflicting protocol +requirements: + +```swift +protocol Employable { + /// Terminate `self`'s employment. + func fire() +} + +protocol Combustible { + /// Immolate `self`. + func fire() +} + +struct Technician: Employable, Combustible { ... } +``` + +It'd be very useful to be able to unambiguously specify which protocol's +requirement you're trying to call: + +```swift +if myTechnician.isGoofingOff { + myTechnician.Employable::fire() +} +if myTechnician.isTooCloseToTheLaunch { + myTechnician.Combustible::fire() +} +``` + +However, allowing a protocol name—rather than a module name—to be written +before the `::` token re-introduces the same ambiguity this proposal seeks +to solve because a protocol name could accidentally shadow a module name. +We'll probably need a different feature with a distinct syntax to resolve +this use case—perhaps something like: + +```swift +if myTechnician.isGoofingOff { + (myTechnician as some Employable).fire() +} +``` + +### Support selecting default implementations + +Similarly, it would be useful to be able to specify that you want to call a +default implementation of a protocol requirement even if the conformance +provides another witness. (This could be used similarly to how `super` is used +in overrides.) However, this runs into similar problems with reintroducing +ambiguity, and it also just doesn't quite fit the shape of the syntax (there's +no name to uniquely identify the default implementation you want). Once again, +this probably requires a different feature with a distinct syntax—perhaps +something a little more like how `super` works. + +## Alternatives considered + +### Change lookup rules in module interfaces + +Some of the problems with module interfaces could be resolved by changing the +rules for qualified lookup *within module interface files specifically*. For +instance, we could decide that in a module interface file, unqualified lookups +can only find module names, and the compiler must always qualify every name +with a module name. + +This would probably be a viable solution with enough effort, but it has a +number of challenges: + +1. There are some declarations—generic parameters, for instance—which are + not accessible through any qualified lookup (they are neither top-level + declarations nor accessible through the member syntax). We would have to + invent some way to reference these. + +2. Existing module interfaces have already been produced which would be broken + by this change, so it would have to somehow be made conditional. + +3. Currently, inlinable function bodies are not comprehensively regenerated + for module interfaces; instead, the original textual source code is + inserted with minor programmatic edits to remove comments and `#if` blocks. + This means we would have to revert to the normal lookup rules within an + inlinable function body. + +It also would not help with ambiguous *qualified* lookups, such as when two +modules use extensions to add identically-named nested types to a top-level +type, and it would not give developers new options for handling ambiguity in +human-written code. + +### Add a fallback lookup rule for module name shadowing + +The issue with shadowing of module names could be addressed by adding a narrow +rule saying that, when a type has the same name as its enclosing module and a +qualified lookup inside it doesn't find any viable candidates, Swift will fall +back to looking in the module it shadowed. + +This would address the `XCTest.XCTestCase` problem, which is the most common +seen in practice, but it wouldn't help with more complicated situations (like +a nested type shadowing a top-level type, or a type in one module having the +same name as a different module). It's also not a very principled rule and +making it work properly in expressions might complicate the type checker. + +### Use a syntax involving a special prefix + +We considered creating a special syntax which would indicate unambiguously that +the next name must be a module name, such as `#Modules.FooKit.bar`. However, +this would only have helped with top-level declarations, not members of +extensions. + +### Use a syntax that avoids `::`'s shortcomings + +Although we have good reasons to propose using the `::` operator (see "Proposed +solution" above), we do not think it's a perfect choice. It appears visually +"heavier" than the `.` operator, which means developers reading the code might +mentally group the identifiers incorrectly: + +```swift +Mission.NASA::Booster.Exhaust // Looks like it means `(Mission.NASA) :: (Booster.Exhaust)` + // but actually means `Mission . (NASA::Booster) . Exhaust` +``` + +This is not unprecedented—in C++, `myObject.MyClass::myMember` means +`(myObject) . (MyClass::myMember)`—but it's awkward for developers without +a background in a language that works like this. + +We rejected a number of alternatives that would avoid this problem. + +#### Make module selectors qualify different names + +One alternative would be to have the module selector qualify the *rightmost* +name in the member chain, rather than the leftmost, so that a module selector +could only appear at the head of a member chain. The previous example would +then be written as: + +```swift +(NASA::Mission.Booster).Exhaust +``` + +We don't favor this design because we believe: + +1. Developers more frequently need to qualify top-level names (which exist in a + very crowded quasi-global namespace) than member names (which are already + limited by the type they're looking inside); a syntax that makes qualifying + the member the default is optimizing for the wrong case. + +2. The distance between the module and the identifier it qualifies increases + the cognitive burden of pairing up modules to the names they apply to. + +3. Subjectively, it's just *weird* that the selector applies to a name that's a + considerable distance from it, rather than the name immediately adjacent. + +A closely related alternative would be to have the module selector qualify +*all* names in the member chain, so that in `(NASA::Mission.Booster).Exhaust`, +both `Mission` and `Booster` must be in module `NASA`. We think point #1 from +the list above applies to this design too: `Mission` is a sparse enough +namespace that developers are more likely to be hindered by `Booster` being +qualified by the `NASA` module than helped by it. + +#### Use a totally different spelling + +We've considered and rejected a number of other spellings for this feature, +such as: + +```swift +Mission.#module(NASA).Booster.Exhaust // Much wordier, not implementable as a macro +Mission.::NASA.Booster.Exhaust // Looks weird in member position +Mission.Booster@NASA.Exhaust // Ignores Swift's left-to-right nesting convention +Mission.'NASA.Booster.Exhaust // Older compilers would mis-lex in inactive `#if` blocks +Mission.NASA'Booster.Exhaust // " +Mission.(NASA)Booster.Exhaust // Arbitrary; little connection to prior art +Mission.'NASA'.Booster.Exhaust // " +``` + +### Don't restrict whitespace to the right of the `::` + +Allowing a newline between `::` and the identifier following it would mean +that, when an incomplete line of code ended with a module selector, Swift might +interpret a keyword on the next line as a variable name, likely causing multiple +confusing syntax errors in the next statement. For instance: + +```swift +let x = NASA:: // Would be interpreted as `let x = NASA::if x { ... }`, +if x { ... } // causing several confusing syntax errors. +``` + +Forbidding it, however, has a cost: it restricts the code styles developers can +use. If a developer wants to put a line break in the middle of a name with a +module selector, they will not be able to format it like this: + +```swift +SuperLongAndComplicatedModuleName:: + superLongAndComplicatedFunctionName() +``` + +And will have to write it like this instead: + +```swift +SuperLongAndComplicatedModuleName + ::superLongAndComplicatedFunctionName() +``` + +The member-lookup `.` operator has a similar restriction, but developers may +not want to style them in exactly the same way, particularly since C++ +developers often split a line after a `::`. + + [SE-0071]: "Allow (most) keywords in member references" + [SE-0364]: "Warning for Retroactive Conformances of External Types" + [SE-0382-review-2]: "SE-0382 (second review): Expression Macros" + [SE-0444]: "SE-0444 Member Import Visibility" diff --git a/proposals/0492-section-control.md b/proposals/0492-section-control.md new file mode 100644 index 0000000000..441e4a760a --- /dev/null +++ b/proposals/0492-section-control.md @@ -0,0 +1,494 @@ +# Section Placement Control + +* Proposal: [SE-0492](0492-section-control.md) +* Authors: [Kuba Mracek](https://github.com/kubamracek) +* Status: **Accepted** +* Implementation: available in recent `main` snapshots under the experimental feature `SymbolLinkageMarkers` and with undercored attribute names `@_section` and `@_used`. +* Review: [review](https://forums.swift.org/t/se-0492-section-placement-control/82289), [acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0492-section-placement-control/82701) +* Discussion threads: + * Pitch #1: https://forums.swift.org/t/pitch-low-level-linkage-control-attributes-used-and-section/65877 + * Pitch #2: https://forums.swift.org/t/pitch-2-low-level-linkage-control/69752 + * Pitch #3: https://forums.swift.org/t/pitch-3-section-placement-control/77435 + +## Introduction + +This proposal adds `@section` and `@used` attributes which can be applied to global variables. These allow users to directly control which section of the resulting binary should global variables be emitted into, and give users the ability to disable DCE (dead code elimination) on those. The goal is to enable systems and embedded programming use cases like runtime discovery of test metadata from multiple modules, and also to serve as a low-level building block for higher-level features (e.g. linker sets, plugins). + +## Motivation + +**Testing frameworks** need to be able to produce test metadata about user’s types and other declarations (e.g. standalone test entrypoints) in a way that they are discoverable and enumerable at runtime. In dynamic languages like Objective-C, this is typically done at runtime using reflection, by querying the language runtime, and/or walking lists of types or exported symbols: + +```swift +// MyXCTestModule +@objc class TableValidationTests: XCTestCase { + func test1() { ... } +} + +// Testing framework, pseudo-code +let classList = objc_copyClassList(...) +for aClass in classList { + if aClass is XCTestCase { + let methodList = class_copyMethodList(aClass) + ... + } +} +``` + +A similarly dynamic approach was proposed in [SE-0385 (Custom Reflection Metadata)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0385-custom-reflection-metadata.md), but was rejected because for Swift, a more static approach would be a better fit: If Swift code had the ability to produce custom metadata directly into the resulting binaries in a well-understood way, we would be able to directly access the data at runtime via platform loader’s APIs and also use offline binary inspection tools (such as objdump, objcopy, otool). This would be more efficient and not require the language runtime and thus also be palatable in embedded use cases using Embedded Swift. + +Mainstream operating systems support dynamically loading modules from disk at runtime, and large applications tend to build **plugin systems** as a way to separate the development of subsystems, or to support separate compilation of 3rd party code. Loading a module from disk can be done using standard APIs like `dlopen`, but discovering and calling the interface in the plugin usually requires using unsafe C constructs (dlsym, casting of pointers) and/or querying the language runtime for type information, similarly to the testing enumerating approach mentioned above. A better approach could publish the information about a plugin in a structured way into the binary, and a runtime component could locate this metadata, and provide access to it in a type-safe way. + +This proposal recommends to use sections of the various object file formats as the vehicle for custom metadata produced by Swift code. This approach is a good fit to the above mentioned use cases, but also enables others: + +* “**Linker sets**” are an approach in systems programming that collects data from multiple source files or subsystems using a standard linker behavior of collocating symbols that belong to the same section. In principle, this is simply a generalization of the test enumeration and plugin discovery use cases mentioned above. The primary goal is decentralization of the information, for example, linker sets can be used to describe memory requirements of each subsystem (and a boot step at runtime can process those to figure out how much heap should be made available). +* Emitting custom metadata into binaries can be used to convey **information to the debugger**. The `@DebugDescription` macro generates a “summary string” for LLDB that summarizes the contents of the fields of a type without the need for runtime evaluation, but rather as a composition of the fields that LLDB assembles. To make those summary strings discoverable by LLDB, placing them into a custom section is a clean solution allowing LLDB to consume them in the case where LLDB has access to the binary on disk, or even without that. Embedded programs might need to rely on such a mechanism as the only way to get enhanced data visualization in the debugger, because runtime evaluation from a debugger is commonly not possible at all in firmware. + +```swift +@DebugDescription struct Student { + var name: String + var id: Int + + /* synthesized by the @DebugDescription macro, made discoverable by LLDB */ + let __Student_lldb_summary = ("PupilKit.Student", "${var.id}: ${var.name}") +} +``` + +* More embedded and systems programming use cases often require directly control of section placement as well, for example to adhere to a **startup contract** with platform libraries, SDK’s linker scripts or the hardware itself. Such contract can be pre-existing in the platform and require placing a specific data structure into a specific section. Enabling doing that directly in Swift will provide a more intuitive and safer implementation option, and users of Swift for embedded devices won’t need to reach for C/C++ as a workaround. + +## Proposed Solution + +The proposal is to add two new attributes `@section` and `@used` that will allow annotating global and static variables with directives to place the value into a custom section, and to require no-dead-stripping aka "attribute used". Using `@section` requires that the initializer expression is a constant expression (see [Constant expressions](#constant-expressions) below for the definition of that): + +```swift +// Place an entry into a section, mark as "do not dead strip". +// Initializer expression must be a constant expression. +// The global variable is implicitly made statically initialized. +@section("__DATA,mysection") +@used +let myLinkerSetEntry: Int = 42 // ✅ + +// Non-constant or statically non-initializable expressions are disallowed +@section("__DATA,mysection") +let myLinkerSetEntry: Int = Int.random(in: 0 ..< 10) // ❌ error + +// Section-placed globals can be "var", the initializer expression still must be constant +@section("__DATA,mysection") +var mutableVariable: Int = 42 // ✅ + +// Some complex data types are allowed (tuples, function references) +typealias PluginData = (version: Int, identifier: UInt64, initializer: @convention(c) ()->()) + +@section("__DATA,plugins") +@used +let myPlugin: PluginData = ( + version: 1, + initializer: { print("init") } +) +``` + +On top of specifying a custom section name with the `@section` attribute, marking a variable as `@used` is needed to prevent otherwise unused symbols from being removed by the compiler. When using section placement to e.g. implement linker sets, such values are typically going to have no usage at compile time, and at the same time they should not be exposed in public interface of libraries (not be made public), therefore we the need the `@used` attribute. + +Different object file formats (ELF, Mach-O, COFF) have different restrictions and rules on what are valid section names, and cross-platform code will have to use different names for different file formats. To support that, custom section names can be specified as a string literal. The string will be used directly, without any processing, as the section name for the symbol. A new `#if objectFormat(...)` conditional compilation directive will be provided to support conditionalizing based on the file format: + +```swift +#if objectFormat(ELF) +@section("mysection") +#elseif objectFormat(MachO) +@section("__DATA,mysection") +#endif +var global = ... +``` + +For the ELF file format specifically, the compiler will also emit a “section index” into produced object files, containing an entry about each custom section used in the compilation. This is a solution to an ELF specific problem where the behavior of ELF linkers and loaders means that sections are not easily discoverable at runtime. + +> Note: The intention is that the `@section` and `@used` attributes are to be used rarely and only by specific use cases; high-level application code should not need to use them directly and instead should rely on libraries, macros and other abstractions over the low-level attributes. + +> The scope of this proposal is limited to compile-time behavior and compile-time control. We expect that full user-facing solutions for features like linker sets, test discovery or plugins will also require runtime implementations to discover and iterate the contents of custom sections, possibly from multiple modules. This proposal makes sure to provide the right building blocks and artifacts in binaries for the runtime components, but doesn’t prescribe the shape of those. However, it is providing a significant step towards generalized and safe high-level mechanisms for those use cases. See the discussion in [Runtime discovery of data in custom sections](#runtime-discovery-of-data-in-custom-sections) and [Linker sets, plugins as high-level APIs](#linker-sets-plugins-as-high-level-apis) in Future Directions. + +## Detailed design + +### Attributes @section and @used on global and static variables + +Two new attributes are to be added: `@section`, which has a single argument specifying the section name, and a argument-less `@used` attribute. The section name must be a string literal. The attributes can be used either together or independently. + +```swift +// (1) +@section("__DATA,mysection") +@used +let global = ... // ✅ + +// (2) +@section("__DATA,mysection") +let global = ... // ✅ + +// (3) +@used +let global = ... // ✅ +``` + +The new attributes (`@section` and `@used`) can be used on variable declarations under these circumstances: + +* the variable must be a global variable or a static member variable (no local variables, no non-static member variables) +* the variable must not be declared inside a generic context (either directly in generic type or nested in a generic type) +* the variable must be a stored property (not a computed property) +* the variable must not have property observers (didSet, willSet) +* the initial expression assigned to the variable must be a constant expression, and it must be eligible for static initilization + +> *Note: These restrictions limit the `@section` and `@used` attributes to only be allowed on variables that are expected to be represented as exactly one statically-initialized global storage symbol (in the linker sense) for the variable’s content. This is generally true for all global and static variables in C and C++, but in Swift global variables might have zero global storage symbols (e.g. a computed property), or need non-trivial storage (e.g. lazily initialized variables with runtime code in the initializer expression).* + +```swift +@section("__DATA,mysection") @used +let global = 42 // ✅ + +@section("__DATA,mysection") @used +var global = 42 // ✅ + +@section("__DATA,mysection") @used +var computed: Int { return 42 } // ❌ ERROR: @section cannot be used on computed properties + +struct MyStruct { + @section("__DATA,mysection") @used + static let staticMemberLet = 42 // ✅ + + @section("__DATA,mysection") @used + static var staticMemberVar = 42 // ✅ + + @section("__DATA,mysection") @used + let member = 42 // ❌ ERROR: @section cannot be used on non-static members + + @section("__DATA,mysection") @used + var member = 42 // ❌ ERROR: @section cannot be used on non-static members +} + +struct MyGenericStruct { + @section("__DATA,mysection") @used + static let staticMember = 42 // ❌ ERROR: @section cannot be used in a generic context + + @section("__DATA,mysection") @used + static var staticMember = 42 // ❌ ERROR: @section cannot be used in a generic context +} +``` + +When allowed, the `@section` attribute on a variable declaration has the following effects: + +1. The variable’s initializer expression is going to be constant folded at compile-time, and assigned as the initial value to the storage symbol for the variable, i.e. the variable will be **statically initialized**. The variable’s value will not be lazily computed at runtime, and it will not use the one-time initialization helper code and token. If that’s not possible, an error is diagnosed. +2. The storage symbol for the variable will be placed into a custom section with the specified name. + - Concretely, the section name string value will be set verbatim as a section specifier for the storage symbol at the LLVM IR level of the compiler. This means that any special behavior that the optimizer, the backend, the assembler or the linker applies based on known section names (or attributes specified as suffixes on the section name) will apply. +3. If applied to a global that is declared as part of top-level executable code (i.e. main.swift), the usual non-top-level-code initialization behavior is applied to the global. I.e. the variable is not sequentially initialized at startup. + +The custom section name specified in the `@section` attribute is not validated by the compiler, instead it’s passed directly as a string to the linker. + +When allowed, the `@used` attribute on a variable declaration has the following effect: + +1. The storage symbol for the variable will be marked as “do not dead-strip”. + +The effects described above are applied to the storage symbols and don’t generally affect optimizations and other transformations in the compiler. For example, the compiler is still allowed to propagate and copy a constant value of a `let` variable to code that uses the variable, therefore there’s no guarantee that a value stored into a global with a custom section will not be propagated and “leak” outside of the section. The `@used` annotation, however, does inform the optimizer that such a variable cannot be removed, even when it doesn’t have any observed users or even if it’s inaccessible due to language rules (e.g. if it’s a private static member on an otherwise empty type). + +### Constant expressions + +Swift currently does not have a formal notion of a **constant expression**, i.e. an expression with a syntactic form that *guarantees the ability to know it's value at compile-time*. This proposal provides a definition of a "bare minimum" constant expression, with the understanding that this does not cover the language needs in generality, and with the expectation that the Swift compiler and language will keep expanding the allowed forms of constant expressions in the future. See [Generalized constant values and expressions](#generalized-constant-values-and-expressions) in Future Directions for further discussion on this. + +This proposal defines a **constant expression** as being one of: + +- an integer literal using any of standard integer types (Int, UInt, Int8/16/32/64/128, UInt8/16/32/64/128) +- a floating-point literal of type Float or Double +- a boolean literal of type Bool +- a direct reference to a non-generic function using its name (the function itself is not generic, and also it must not be defined in a generic context) +- a direct reference to a non-generic metatype using the type name directly (the type itself is not generic, and also it must not be defined in a generic context), where the type is non-resilient +- a tuple composed of only other constant expressions +- an array literal of type InlineArray composed of only other constant expressions + +Explicitly, this definition currently does **not allow** any operators, using any user-defined named types, any other standard type (e.g. strings, dictionaries, sets), using closures, or referencing any variables by name. See below for examples of valid and invalid constant expressions: + +```swift +@section("...") let a = 42 // ✅ +@section("...") let b = 3.14 // ✅ +@section("...") let c = 1 + 1 // ❌ operators not allowed +@section("...") let d = Int.max // ❌ not a literal +@section("...") let e: UInt8 = 42 // ✅ +@section("...") let f = UInt8(42) // ❌ not a literal +@section("...") let g: MyCustomExpressibleByIntegerLiteral = 42 // ❌ not a standard type + +@section("...") let composition1 = (1, 2, 3, 2.718, true) // ✅ +@section("...") let composition2 = (1, 2, Int.max) // ❌ tuple component not constant +@section("...") let composition3: InlineArray = [1, 2, 3] // ✅ +@section("...") let composition4: InlineArray = [1, 2, Int.max] // ❌ array component not constant +@section("...") let composition5: (Int, [1 of Int], [1 of (Int, Int)]) = (1, [1], [(1, 1)]) // ✅ + +func foo() -> Int { return 42 } +@section("...") let func1 = foo // ✅ +@section("...") let func2 = foo() // ❌ not a function reference +@section("...") let func3 = Bool.random // ✅ +@section("...") let func4 = Bool.self.random // ❌ not a direct reference +@section("...") let func5 = (Bool.self as Bool.Type).random // ❌ not a direct reference +@section("...") let func6 = [Int].randomElement // ❌ generic +@section("...") let func7 = { } // ❌ not using name + +struct S { } +@section("...") let metatype1 = S.self // ✅ +@section("...") let metatype2 = Int.self // ✅ +@section("...") let metatype3 = Int.self.self // ❌ not a direct reference +import Foundation +@section("...") let metatype4 = URL.self // ❌ resilient +``` + +### Guaranteed static initialization + +Using attribute `@section` requires the initializer expression of the variable to be a **constant expression**. It's not required to separately annotate the expression for being a compile-time expression, instead this is implied from the `@section` attribute. On top of the constant-ness, `@section` on a global or static variable enforces **static initialization** on that variable. + +We consider the variable to be eligible for static initialization when: + +1. the initializer expression represents a valid compile-time constant, and +2. the initializer expression can be constant-folded into a representation that does not require any runtime initialization (pointer relocations/fixups done automatically by the loader are not considered runtime initialization for this purpose). + +Not all constant expressions are necessarily statically initializable. For section placement we require the stronger property (static initialization) because we expect the data to be readable without any runtime mechanisms (i.e. reading raw bytes from the section at runtime, or offline binary inspection). + +```swift +@section("__DATA,mysection") +let a = 42 // ✅ + +@section("__DATA,mysection") +let sectionPlaced = ...expression... // ✅, guaranteed to be statically initialized + +@section("__DATA,mysection") +let notStaticallyInitializable = ...expression that cannot be statically initialized... // ❌ +``` + +> *Note: As of this writing, all valid constant values are also eligible to be statically initialized, but we don’t expect that to hold in the future. So it’s important to distinguish between (a) a global variable being initialized with a language-level constant value, and (b) a global variable that is guaranteed to be statically initialized. The difference can be subtle, and in some cases immaterial in practice, but future enhancements of constant values in Swift might take advantage of this difference — i.e. not all constant values are going to be statically initializable. Consider the following example: If a future language versions allows dictionary literals to be constant values, such values might not be statically initializable because of randomized hash seeding:* + +> ```swift +> let d1 = ["a": 42, "b": 777] // constant, but not statically initializable +> let d2 = d1.count // statically initializable +> ``` + +> *However, using a statically non-initializable value in an expression does not preclude the outer expression from being statically initialized either. In this example, `d1` would not be allowed to be placed into a custom section because it’s not statically initializable. But `d2` could still potentially be statically initializable (even though the definition of `d2` uses a sub-expression that is not statically initializable), as it’s simply an integer.* + +As described in [Swift Compile-Time Values](https://github.com/artemcm/swift-evolution/blob/const-values/proposals/0nnn-const-values.md), values of function types are eligible for being compile-time evaluable. Their concrete pointer value is not fully known until link-time or program load-time (depending on type of linking, ASLR, PAC, etc.). For the purposes of guaranteed static initialization, function values are statically initialized into a function pointer. This pointer is still subject to normal linking and loading resolutions and fixups. + +```swift +func foo() { ... } + +@section("__DATA,mysection") +let a = (42, foo) // "foo" is statically initialized into a + // linkable/relocatable pointer +``` + +### Cross-platform object file format support + +Different platforms supported by Swift are using different object and binary file formats (Linux uses ELF, Darwin uses Mach-O, Windows uses COFF), and that implies different restrictions and rules on what are valid section names. Because of that, a multi-platform library code is expected to use `#if os(...)` to use different section names for different platforms. Because of that, it’s expected that the attributes are to be typically only used indirectly via macros that hide the low-level nature of sections and object file formats from higher-level code developers: + +```swift +// Example of a potential project-specific "@RegisterPlugin" macro: +@RegisterPlugin +let plugin = ... + +// The macro expands to: +#if os(macOS) || os(iOS) || os(watchOS) || os(tvOS) +@section("__DATA_CONST,plugins") +#elseif os(Linux) +@section("plugins") +#elseif os(Windows) +@section(".plugins") +#endif +let plugin = ... +``` + +See [Structured section specifiers](#structured-section-specifiers) below for more rationale. + +In some cases, it’s not possible to differentiate on the OS to support multiple object file formats, for example when using Embedded Swift to target baremetal systems without any OS. For that, a new `#if objectFormat(...)` conditional compilation directive will be provided. The allowed values in this directive will match the set of supported object file formats by the Swift compiler (and expand as needed in the future). Currently, they exact values will be (case sensitive): + +* COFF +* ELF +* MachO +* Wasm + +```swift +#if objectFormat(MachO) +@section("__DATA_CONST,mysection") +#elseif objectFormat(ELF) +@section("mysection") +#endif +let value = ... +``` + +### ELF section index + +The goal of placing metadata into custom section is to make them discoverable both via offline inspection (e.g. objdump or otool) and at runtime. The facilities for that are dependent on the type of linking (static vs dynamic), and platform’s linker and loader: + +* For **static linking**, the bounds of a section can be statically determined by the linker and on all supported platforms and their file formats (COFF, ELF, MachO, Wasm), the linker-provided “**encapsulation symbols**” can be used to retrieve those bounds. + * In ELF and Wasm formats, these symbols are `__start_
` / `__stop_
`. + * In Mach-O, these symbols are `section$start$$` / `section$end$$`. + * In COFF, these symbols need to be manually constructed by using “grouped sections” (section name is suffixed with a $ + string) which are automatically lexicographically ordered by the linker. For example, by manually placing a start symbol into `.section$A` , end symbol into `.section$C` and all actual section entries into `.section$B`, the two helper symbols’s addresses effectively describe the bounds of the section. +* For **dynamic linking**, the above mentioned encapsulation symbols are available too, but they always only describe the bounds of the section in the current module. Retrieving section content process-wide means collecting metadata from multiple images at runtime, which requires further assistance or support from the loader. + * In Mach-O (Darwin OS's), image headers are present in the address space, and they include section bounds information. The loader provides straightforward image iteration APIs (`_dyld_get_image_header`), as well as image load callbacks (`_dyld_register_func_for_add_image`), and an API to lookup section bounds by name from a particular image (getsectiondata). + * In COFF (Windows), image headers are present in the address space, and they include section bounds information. `Module32FirstW`/`Module32NextW` can be used to enumerate images, and structures such as `IMAGE_DOS_HEADER`, `IMAGE_NT_HEADERS`, `IMAGE_FILE_HEADER`, and `IMAGE_SECTION_HEADER` can be used to walk a module and find its section bounds. + * In Wasm, dynamic linking is work in progress and not generally available yet. + * In ELF, however, section bounds are not guaranteed to be present in the address space at runtime, and in practice they are typically not present. This creates a challenge for retrieving section data in this configuration (ELF + multiple modules with dynamic linking) at runtime. + +To solve this problem for the ELF object file format, the Swift compiler is going to emit a “**section index**” into every compilation that uses any symbols placed into a custom section. The index will be emitted only when producing ELF files, and consists of entries added into its own separate well-named section called `swift5_sections`. Each entry will have the following structure: + +```c +struct SectionIndexEntry { + const char *name; + const void *start; // effectively equal to __start_ + const void *stop; // effectively equal to __stop_ +}; +``` + +The section index will describe the bounds of all custom sections used in Swift code. When compiling into a single object file (e.g. in WMO mode without -num-threads), there will be only a single entry per distinct section name, but in compilation modes that produce multiple object files for a single module, there may be multiple entries for the same section. The entries are going to be “linkonce_odr”, i.e. duplicate entries will be collapsed at link time, so in a linked module, only one entry per section will remain. + +This way, runtime code present in the same module, for example SwiftRT-ELF helper code (swiftrt.o) which is currently being silently linked in to all modules, can walk over the section index using the encapsulation symbols, and register the section bounds in a globally maintained data structure in the Swift runtime. Implementation of that and exposing such a facility in an actual API from the Swift runtime is left as a future direction. + +## Source compatibility + +This proposal is purely additive (adding new attributes), without impact on source compatibility. + +## Effect on ABI stability + +This change does not impact ABI stability for existing code. + +Adding, removing, or changing the `@section` attribute on variables should generally be viewed as an ABI breaking change, section placement can affect linking behavior of that symbol. In some cases, it is possible to make careful non-ABI-breaking changes via the `@section` attribute. + +Adding `@used` does not affect ABI stability. Removing `@used` can be viewed as an ABI breaking change, however not in the traditional sense: The effect of `@used` only exists on symbols that would normally not be exported (e.g. private symbols), which shouldn’t be part of a ABI in the first place. However, dynamic lookups of such symbols are still possible, and if the behavior of those is considered ABI, then removing `@used` can be ABI breaking. + +## Effect on API resilience + +This change does not impact API resilience for existing code. + +Adding, removing, or changing the `@section` attribute on variables does not affect API resilience. + +Adding or removing `@used` does not affect API resilience. + +## Future Directions + +### Section placement for functions + +This proposal only allows data placement into custom sections, however, placing code into custom sections is a relatively useful and common approach in systems and embedded programming. In the future, the `@section` and `@used` attributes could be extended to apply to function declarations, and possibly other language constructs that generate executable code (e.g. closures). A prominent use case is firmware entry points and booting schemes, which often require startup code to be in a predefined section: + +```swift +// code for the function is placed into the custom section +@section("__TEXT,boot") +func firmwareBootEntrypoint() { ... } +``` + +This will require some design decisions to be made around when should that be allowed, whether the attribute should be automatically inherited, and what exact behavior should we expect from the compiler around thunks, compiler-generated helper functions, getters and setters, etc. + +### Standalone attribute for required static initialization + +Static initialization of a global can be useful on its own, without placing data into a custom section, and a separate attribute for that could be added. This way, one can get the same effects as the `@section` attribute (static initialization, normal initalization behavior if top-level code) except the symbol would not be actually placed into any custom section. + +### Generalized constant values and expressions + +The notions of constant expressions and constant values is applicable to a much wider set of use cases that just section placement, and the set of allowed types and syntactical forms should be expanded in the future into a full-featured system for compile-time programming. A dedicated proposal, [Swift Compile-Time Values](https://github.com/artemcm/swift-evolution/blob/const-values/proposals/0nnn-const-values.md), is being pitched [on the forums](https://forums.swift.org/t/pitch-3-swift-compile-time-values/77434) and describes in detail the possible future of generalized constants, the relevant motivation and use cases. + +### Allowing a reference to a constant string declaration as a section name + +The requirement to only use string literals as the section names could be lifted in the future, and we might allow referring to a declaration of variable with a compile-time string. This would be useful to avoid repetition when placing multiple values into the same section without needing to use macros. + +```swift +#if objectFormat(ELF) +let mySectionName = "mysection" // required to be a compile-time value +#elseif objectFormat(MachO) +let mySectionName = "__DATA,mysection" // required to be a compile-time value +#endif + +@section(mySectionName) +var global = ... +``` + +### Runtime discovery of data in custom sections + +As described in [ELF section index](#elf-section-index), accessing records in a custom section at runtime is heavily dependent on the object file format (ELF, Mach-O, Wasm, COFF), type of linking (static vs dynamic) and available APIs from the operating system. For a single configuration, users can directly use an appropriate method of accessing the section data, and e.g. in embedded firmwares this might be completely fine as projects are commonly avoiding any attempts to be multi-platform or portable. + +However, for multi-platform libraries and general purpose packages, supporting the full matrix of combinations would be very impractical. Because of that, it’s expected that a unified API for accessing the bounds and contents of a section (across multiple modules in presence of dynamic linking) is provided either as part of the Swift runtime, the standard library, or as a portable package. This API would likely still be a relatively low-level API, providing access to the raw bytes of sections across multiple loaded modules, but it would provide an shared abstraction across platforms, file formats, and linking types. + +### Linker sets, plugins as high-level APIs + +This proposal only builds the compile-time half of a user-facing “linker set” mechanism (placing structured data into sections). To access and enumerate the data at runtime, one can imagine a direct, still relatively low-level API like this: + +```swift +func enumerateLinkerSet(fromSectionNamed: String) -> Sequence { + // extract section data, assuming the raw data in the section are records of "T" + // probably built on top of a cross-platform section access API mentioned in previous section +} +``` + +But a solution based on macros could achieve a higher-level abstraction for the entire “linker set” mechanism: + +```swift +@DefineLinkerSet("name", type: Int) // other macros understand that linker set "name" + // has entries of type Int + +@LinkerSetEntry("name") let entry1: Int = 42 // ok +@LinkerSetEntry("name") let entry2: Float = 7.7 // error + +for entry in #enumerateLinkerSet("name") { + print(entry) +} +``` + +Similarly, a plugin registration and discovery mechanism based on macros could provide full type safety and hide the low-level aspects completely: + +```swift +// In PluginModule: +@PluginRecord(protocol: PluginProtocol, type: MyPluginType) +let plugin = PluginData(name: "myPlugin", version: 1, initialization: { ... }) + +// In MainModule: +... load available plugins via dlopen ... +for plugin in Plugin.enumerateLoadedPlugins(for: PluginProtocol.self) { + print(plugin.name) + let t = plugin.instantiateType() + ... +} +``` + +### Access to a stable address + +Generally, Swift values and variables do not have a stable address, and converting an inout reference to a UnsafePointer does not guarantee a stable address even on a global variable. However, statically initialized globals/static (either via the `@section` attribute, or via `@constInitialized`) do fundamentally have a stable address because they have an exact location in the binary on disk and in the module’s image at runtime. It’d be useful to provide direct access to that, in a way that adds the missing stable-address guarantees compared to inout-to-pointer conversions, and also allow fetching an address of a `let` variable (inout references only work on `var`): + +```swift +@constInitialized let x = 42 + +let address: UnsafePointer = #address(x) // would only work on statically initialized + // globals/statics +``` + +## Alternatives Considered + +### Requiring explicitly spelled out `@const` when using `@section` + +`@section` annotated globals/statics require their initializer expressions to be constant expressions, but the expression does not have to be marked as `@const` manually, it’s implied instead. An alternative of requiring the `@const` was considered: + +```swift +@section(...) let global: Int = @const 42 +``` + +Because `@const` does not affect parsing or type resolution of the expression, it’s not helpful to the compiler, and it doesn’t seem to improve readability for users either: If the expression is a constant expression or not statically initializable, it will be rejected from compilation with a clear explanation. Adding a `@const` does not convey any new information. + +### Structured section specifiers + +In Mach-O, custom section names are written as a pair of segment (e.g. `__DATA`) + section (e.g. `mysection`). Structured section names with separate segment and section names, `@section(segment: "...", section: "...")` were considered instead, however this pattern does not generalize across object file formats, and is Mach-O specific (ELF and PE/COFF don’t have segments). + +Because different object file formats impose different restrictions on custom section names (length, “.” prefix), a shorthand syntax to specify different section names for different object file formats was considered: `@section(ELF: “...”, MachO: “...”, COFF: “...”)`. This, however, has drawbacks of repeating the file format in cases where the code is only ever targeting a single format (common for example for embedded firmwares on ELF). The benefits of a shorthand syntax is marginal, given that we don’t expect normal application code to use the `@section` attribute directly but instead rely on macros or other higher-level API. + +The alternative of using conditional compilation is what is expected to be used for those cases instead. + +### Umbrella attribute for linkage properties + +Instead of separate `@section` and `@used` attributes, a unified attribute with parameters to control individual linkage properties was considered, spelled for example `@linkage(section: ..., used)`. Further linkage control features would be added into this umbrella attribute. + +This, however, adds challenges on composability — one umbrella attribute would need to allow multiple occurrences and the design would need rules for merging of individual properties from multiple attributes. Separate standalone attributes compose trivially, and also they play nicely with the existing `#if hasAttribute(...)` conditional compilation mechanism. There is currently no mechanism for conditional compilation based on whether a sub-feature of a umbrella attribute is available in the compiler. + +Given the above, and also given that controlling symbol and linker level properties is not something that we expect normal application code to do directly, it’s more appropriate to keep the attribute system simple, and have individual orthogonal composable attributes. + +### `@section` implying `@used` + +In a lot of the list code snippets in this proposal, both `@section` and `@used` were used together, and so it may seem that it’s not necessary for those to be two separate attributes. However: + +* `@section` and `@used` represent separate concepts and all combinations of them can be useful. An example of using `@section` without `@used` is to place for example a large data table from a library into its own section for binary size accounting reasons (so that it shows up separately in per-section binary size listings), but where we’d still expect the data table to be dead-code removed if not used. +* It’s already common to have those attributes as separate options in existing popular systems programming languages (C, C++, Rust). + +### Blocking section placement into compiler reserved sections + +In most cases, placing data into one of Swift’s runtime reserved sections (e.g. `__swift5_types`, etc.) without relying on extreme details of the compiler and runtime would result in invalid binaries. It was considered to simply reject using `@section` to target one of these reserved sections, but ultimately that would introduce both false positives (*what if we at some point wanted to write compiler/runtime code in Swift to actually legitimately place data into these sections?*) and false negatives (*there are many other "reserved" sections that the Swift compiler and language cannot know about*), and is thus left out of this proposal. diff --git a/proposals/0493-defer-async.md b/proposals/0493-defer-async.md new file mode 100644 index 0000000000..1ec46c74a5 --- /dev/null +++ b/proposals/0493-defer-async.md @@ -0,0 +1,132 @@ +# Support `async` calls in `defer` bodies + +* Proposal: [SE-0493](0493-defer-async.md) +* Authors: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Accepted** +* Implementation: [swiftlang/swift#83891](https://github.com/swiftlang/swift/pull/83891) +* Review: ([pitch](https://forums.swift.org/t/support-async-calls-in-defer-bodies/81790)) ([review](https://forums.swift.org/t/se-0493-support-async-calls-in-defer-bodies/82293)) ([acceptance](https://forums.swift.org/t/accepted-se-0493-support-async-calls-in-defer-bodies/83023)) + +## Introduction + +This is a targeted proposal to introduce support for asynchronous calls within `defer` statements. Such calls must be marked with `await` as any other asynchronous call would be, and `defer` statements which do asynchronous work will be implicitly awaited at any relevant scope exit point. + +## Motivation + +The `defer` statement was introduced in Swift 2 (before Swift was even open source) as the method for performing scope-based cleanup in a reliable way. Whenever a lexical scope is exited, the bodies of prior `defer` statements within that scope are executed (in reverse order, in the case of multiple `defer` statements). + +```swift +func sendLog(_ message: String) async throws { + let localLog = FileHandle("log.txt") + + // Will be executed even if we throw + defer { localLog.close() } + + localLog.appendLine(message) + try await sendNetworkLog(message) +} +``` + +This lets cleanup operations be syntactically colocated with the corresponding setup while also preventing the need to manually insert the cleanup along every possible exit path. + +While this provides a convenient and less-bug-prone way to perform important cleanup, the bodies of `defer` statements are not permitted to do any asynchronous work. If you attempt to `await` something in the body of a `defer` statement, you'll get an error even if the enclosing context is `async`: + +```swift +func f() async { + await setUp() + // error: 'async' call cannot occur in a defer body + defer { await performAsyncTeardown() } + + try doSomething() +} +``` + +If a particular operation *requires* asynchronous cleanup, then there aren't any great options today. An author can either resort to inserting the cleanup on each exit path manually (risking that they or a future editor will miss a path), or else spawn a new top-level `Task` to perform the cleanup: + +```swift +defer { + // We'll clean this up... eventually + Task { await performAsyncTeardown() } +} +``` + +## Proposed solution + +This proposal allows `await` statements to appear in `defer` bodies whenever the enclosing context is already `async`. Whenever a scope is exited, the bodies of all prior `defer` statements will be executed in reverse order of declaration, just as before. The bodies of any `defer` statements containing asynchronous work will be `await`ed, and run to completion before the function returns. + +Thus, the example from **Motivation** above will become valid code: +```swift +func f() async { + await setUp() + defer { await performAsyncTeardown() } // OK + + try doSomething() +} +``` + +## Detailed design + +When a `defer` statement contains asynchronous work, we will generate an implicit `await` when it is called on scope exit. See **Alternatives Considered** for further discussion. + +We always require that the parent context of the `defer` be explicitly or implicitly `async` in order for `defer` to contain an `await`. That is, the following is not valid: + +```swift +func f() { + // error: 'async' call in a function that does not support concurrency + defer { await g() } +} +``` + +In positions where `async` can be inferred, such as for the types of closures, an `await` within the body of a `defer` is sufficient to infer `async`: + +```swift +// 'f' implicitly has type '() async -> ()' +let f = { + defer { await g() } +} +``` + +The body of a `defer` statement will always inherit the isolation of its enclosing scope, so an asynchronous `defer` body will never introduce *additional* suspension points beyond whatever suspension points are introduced by the functions it calls. + +## Source compatibility + +This change is additive and opt-in. Since no `defer` bodies today can do any asynchronous work, the behavior of existing code will not change. + +## ABI compatibility + +This proposal does not have any impact at the ABI level. It is purely an implementation detail. + +## Implications on adoption + +Adoping asynchronous `defer` is an implementation-level detail and does not have any implications on ABI or API stability. + +## Alternatives considered + +### Require some statement-level marking such as `defer async` + +We do not require any more source-level annotation besides the `await` that will appear on the actual line within the `defer` which invokes the asynchronous work. We could go further and require one to write something like: +```swift +defer async { + await fd.close() +} +``` + +This proposal declines to introduce such requirement. Because `defer` bodies are typically small, targeted cleanup work, we do not believe that substantial clarity is gained by requiring another marker which would remain local *to the `defer`* statement itself. Moreover, the enclosing context of such `defer` statements will *already* be required to be `async`. In the case of `func` declarations, this will be explicit. In the case of closures, this may be inferred, but will be no less implicit than the inference that already happens from having an `await` in a closure body. + +### Require some sort of explicit `await` marking on scope exit + +The decision to implicltly await asyncrhonous `defer` bodies has the potential to introduce unexpected suspension points within function bodies. This proposal takes the position that the implicit suspension points introduced by asynchronous `defer` bodies is almost entirely analagous to the [analysis](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0317-async-let.md#requiring-an-awaiton-any-execution-path-that-waits-for-an-async-let) provided by the `async let` proposal. Both of these proposals would require marking every possible control flow edge which exits a scope. + +If anything, the analysis here is even more favorable to `defer`. In the case of `async let` it is possible to have an implicit suspension point without `await` appearing anywhere in the source—with `defer`, any suspension point within the body will be marked with `await`. + +### Suppress task cancellation within `defer` bodies + +During discussion of the behavior expected from asynchronous `defer` bodies, one point raised was whether we ought to remove the ability of code within a `defer` to observe the current task's cancellation state. Under this proposal, no such change is adopted, and if the current task is cancelled then all code called from the `defer` will observe that cancellation (via `Task.isCancelled`, `Task.checkCancellation()`, etc.) just as it would if called from within the main function body. + +The alternative suggestion here noted that because task cancellation can sometimes cause code to be skipped, it is not in the general case appropriate to run necessary cleanup code within an already-cancelled task. For instance, if one wishes to run some cleanup on a timeout (via `Task.sleep`) or via an HTTP request or filesystem operation, these operations could interpret running in a cancelled task as an indication that they _should not perform the requested work_. + +We could, instead, notionally 'un-cancel' the current task if we enter a `defer` body: all code called from within the `defer` would observe `Task.isCancelled == true`, `Task.checkCancellation()` would not throw, etc. This would allow timeouts to continue to function and ensure that any downstream cleanup would not misinterpret task cancellation as an indication that it should early-exit. + +This proposal does not adopt such behavior, for a combination of reasons: +1. Synchronous `defer` bodies already observe cancellation 'normally', i.e., `Task.isCancelled` can be accessed within the body of a synchronous `defer`, and it will reflect the actual cancellation status of the enclosing task. While it is perhaps less likely that existing synchronous code exhibits behavior differences with respect to cancellation status, it would be undesirable if merely adding `await` in one part of a `defer` body could result in behavior changes for other, unrelated code in the same `defer` body. +2. We do not want a difference in behavior that could occur merely from moving existing code into a `defer`. Existing APIs which are sensitive to cancellation must already be used with care even in straight-line code where `defer` may not be used (since cancellation can happen at any time), and this proposal takes the position that such APIs are more appropriately addressed by a general `withCancellationIgnored { ... }` feature (or similar) as discussed in the pitch thread. diff --git a/proposals/0494-add-is-identical-methods.md b/proposals/0494-add-is-identical-methods.md new file mode 100644 index 0000000000..cd3912b9cc --- /dev/null +++ b/proposals/0494-add-is-identical-methods.md @@ -0,0 +1,737 @@ +# Add `isTriviallyIdentical(to:)` Methods for Quick Comparisons to Concrete Types + +* Proposal: [SE-0494](0494-add-is-identical-methods.md) +* Authors: [Rick van Voorden](https://github.com/vanvoorden), [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Accepted with modifications** +* Implementation: ([String, Substring](https://github.com/swiftlang/swift/pull/82055)), ([Array, ArraySlice, ContiguousArray](https://github.com/swiftlang/swift/pull/82438)), ([Dictionary, Set](https://github.com/swiftlang/swift/pull/82439)) +* Review: ([prepitch](https://forums.swift.org/t/-/78792)) ([first pitch](https://forums.swift.org/t/-/79145)) ([second pitch](https://forums.swift.org/t/-/80496)) ([review](https://forums.swift.org/t/se-0494-add-isidentical-to-methods-for-quick-comparisons-to-concrete-types/82296)) ([revision](https://forums.swift.org/t/se-0494-add-isidentical-to-methods-for-quick-comparisons-to-concrete-types/82296/142)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0494-add-isidentical-to-methods-for-quick-comparison-to-concrete-types/82695)) + +### Table of Contents + + * [Introduction](#introduction) + * [Motivation](#motivation) + * [Prior Art](#prior-art) + * [Proposed Solution](#proposed-solution) + * [Detailed Design](#detailed-design) + * [`String`](#string) + * [`Substring`](#substring) + * [`Array`](#array) + * [`Dictionary`](#dictionary) + * [`Set`](#set) + * [`UnsafeBufferPointer`](#unsafebufferpointer) + * [`UTF8Span`](#utf8span) + * [Source Compatibility](#source-compatibility) + * [Impact on ABI](#impact-on-abi) + * [Future Directions](#future-directions) + * [Alternatives Considered](#alternatives-considered) + * [Exposing Identity](#exposing-identity) + * [Different Names](#different-names) + * [Generic Contexts](#generic-contexts) + * [Overload for Reference Comparison](#overload-for-reference-comparison) + * [Support for Optionals](#support-for-optionals) + * [Alternative Semantics](#alternative-semantics) + * [Acknowledgments](#acknowledgments) + +## Introduction + +We propose new `isTriviallyIdentical(to:)` instance methods to concrete types for quickly determining if two instances must be equal by-value. + +## Motivation + +Suppose we need an algorithm that transforms an `Array` of `Int` values to select only the even numbers: + +```swift +func result(for input: [Int]) -> [Int] { + print("computing new result") + return input.filter { + $0 % 2 == 0 + } +} +``` + +This produces a correct answer… but what about performance? We expect our `result` function to run in `O(n)` time across the size of our `input` value. Suppose we need for this algorithm to be called *many* times over the course of our application. It might also be the case that we sometimes call this algorithm with the same `input` value more than once: + +```swift +let a = [1, 2, 3, 4] +print(result(for: a)) +// Prints "computing new result" +// Prints "[2, 4]" +let b = a +print(result(for: b)) +// Prints "computing new result" +// Prints "[2, 4]" +let c = [1, 2, 3, 4] +print(result(for: c)) +// Prints "computing new result" +// Prints "[2, 4]" +let d = [1, 2, 3, 4, 5, 6] +print(result(for: d)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +let e = d +print(result(for: e)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +let f = [1, 2, 3, 4, 5, 6] +print(result(for: f)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +``` + +If we call our `result` function with an `Array` of values and then pass the same `Array` of values again, we might want to return our previous `result` *without* performing another `O(n)` operation. Because our `result` function is a pure function and free of side-effects, we can check the new `input` value against the last `input` value we used to compute our `result`. If the `input` values have not changed, the `result` value *also* must not have changed. + +Here is an attempt to *memoize* our `result`: + +```swift +final class Memoizer { + private var input: [Int]? + private var result: [Int]? + + func result(for input: [Int]) -> [Int] { + if let result = self.result, + self.input == input { + return result + } else { + print("computing new result") + self.input = input + let result = input.filter { + $0 % 2 == 0 + } + self.result = result + return result + } + } +} +``` + +When we pass `input` values we can see that a new `result` is not computed if we already computed a `result` for those same `input` values: + +```swift +let memoizer = Memoizer() +let a = [1, 2, 3, 4] +print(memoizer.result(for: a)) +// Prints "computing new result" +// Prints "[2, 4]" +let b = a +print(memoizer.result(for: b)) +// Prints "[2, 4]" +let c = [1, 2, 3, 4] +print(memoizer.result(for: c)) +// Prints "[2, 4]" +let d = [1, 2, 3, 4, 5, 6] +print(memoizer.result(for: d)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +let e = d +print(memoizer.result(for: e)) +// Prints "[2, 4, 6]" +let f = [1, 2, 3, 4, 5, 6] +print(memoizer.result(for: f)) +// Prints "[2, 4, 6]" +``` + +This looks like a big improvement… until we begin to investigate a little closer. There’s a subtle performance bottleneck here now from a different direction. Our memoization algorithm depends on the value equality of our `input` values — and this is *also* an `O(n)` operation. So while it is true that we have reduced the amount of `O(n)` operations that take place to compute our `result` values, we have *added* `O(n)` operations to determine value equality. As the amount of time spent computing value equality grows, we might no longer see any performance wins from memoization: it would be cheaper to just go ahead and compute a new `result` every time. + +Let’s see another example. Suppose we are working on our SwiftUI app to display Contacts from [SE-0261](0261-identifiable.md). Let’s begin with our basic data model: + +```swift +struct Contact: Identifiable, Equatable { + let id: Int + var name: String + var isFavorite: Bool +} +``` + +We added an `isFavorite` property to indicate our user added this `Contact` value as one of their favorites. + +Here is a SwiftUI view component that displays our favorite `Contact` values in a `FavoriteContactList`: + +```swift +struct FavoriteContactList: View { + @State private var selection: Contact.ID? + + private let contacts: [Contact] + + init(_ contacts: [Contact]) { + self.contacts = contacts + } + + private var favorites: [Contact] { + self.contacts.filter { + $0.isFavorite + } + } + + var body: some View { + List(self.favorites, selection: self.$selection) { contact in + FavoriteCell(contact) + } + } +} +``` + +We can assume there is another view component in our application that could be editing these `Contact` values. It's not very important for us right now to show *how* these `Contact` values could change — let's just assume that our `FavoriteContactList` component might need to recompute its `body` over time with new `Contact` values. + +When we compute our `body` property we also compute our `favorites` property. The implication is that *every* time our `body` property is computed we perform *another* `O(n)` algorithm across our `contacts`. Because our `FavoriteContactList` supports selection, every time our user selects a `Contact` value we update our `State`. Updating our `State` computes our `body` which computes our `favorites` property. So even though our `contacts` values *have not changed*, we *still* pay the performance penalty of *another* `O(n)` operation just to support cell selection. + +This might look like a good opportunity for another attempt at memoization. Here is an approach using a dynamic property wrapper: + +```swift +@propertyWrapper struct Favorites: DynamicProperty { + @State private var storage: Storage + private let contacts: [Contact] + + init(_ contacts: [Contact]) { + self.storage = Storage(contacts) + self.contacts = contacts + } + + func update() { + self.storage.update(self.contacts) + } + + var wrappedValue: [Contact] { + self.storage.wrappedValue + } +} + +extension Favorites { + private final class Storage { + private var contacts: [Contact] + private var favorites: [Contact]? + + init(_ contacts: [Contact]) { + self.contacts = contacts + self.favorites = nil + } + + func update(_ contacts: [Contact]) { + if self.contacts != contacts { + self.contacts = contacts + self.favorites = nil + } + } + + var wrappedValue: [Contact] { + if let favorites = self.favorites { + return favorites + } + print("computing new result") + let favorites = self.contacts.filter { + $0.isFavorite + } + self.favorites = favorites + return favorites + } + } +} +``` + +Here is what that looks like used from our `FavoriteContactList`: + +```swift +struct FavoriteContactList: View { + @State private var selection: Contact.ID? + + @Favorites private var favorites: [Contact] + + init(_ contacts: [Contact]) { + self._favorites = Favorites(contacts) + } + + var body: some View { + List(self.favorites, selection: self.$selection) { contact in + FavoriteCell(contact) + } + } +} +``` + +When we build and run our app we see that we no longer compute our `favorites` values every time our user selects a new `Contact`. But similar to what we saw in our command line utility, we have traded performance in a different direction. The value equality operation we perform is *also* `O(n)`. As the amount of time we spend computing value equality grows, we can begin to spend more time computing value equality than we would have spent computing our `favorites`: we no longer see the performance benefits of memoization. + +This proposal introduces an advanced performance hook for situations like this: a set of `isTriviallyIdentical(to:)` methods that are designed to return *faster* than an operation to determine value equality. The `isTriviallyIdentical(to:)` methods can return `true` in `O(1)` to indicate two values *must* be equal. + +## Prior Art + +We said that the performance of the value equality operator on an `Array` value was `O(n)`. This is true in the *worst case*, but there does exist an important “fast path” that can return `true` in constant time. + +Many types in Standard Library are “copy-on-write” data structures. These types present as value types, but can leverage a reference to some shared state to optimize for performance. When we copy this value we copy a reference to shared storage. If we perform a mutation on a copy we can preserve value semantics by copying the storage reference to a unique value before we write our mutation: we “copy” on “write”. + +This means that many types in Standard Library already have some private reference that can be checked in constant time to determine if two values are identical. Because these types copy before writing, two values that are identical by their shared storage *must* be equal by value. What we propose here is a way to “expose” this fast path operation. + +Product engineers have evolved patterns over the years that can already come close to what we are proposing. Product engineers building on `Array` can use `withUnsafeBufferPointer` or `withContiguousStorageIfAvailable` to compare the “identity” of two `Array` values. One drawback here is that these are only guaranteed to return an identity in constant time if there already exists a contiguous storage. If there does *not* exist a contiguous storage, we might have to perform an `O(n)` algorithm — which defeats the purpose of us choosing this as a fast path. Another option might be `withUnsafeBytes`, but this carries some restrictions on the `Element` of our `Array` and also might require for a contiguous storage to be created: an `O(n)` algorithm. + +Even if we were able to use `withUnsafeBytes` for other data structures, a comparison using `memcmp` might compare “unnecessary” bits that do not affect the identity. This slows down our algorithm and also returns “false negatives”: returning `false` when these instances should be treated as identical. + +A solution for modern operating systems is the support we added from [SE-0456](0456-stdlib-span-properties.md) to bridge an `Array` to `Span`. We can then compare these instances using the `isIdentical(to:)` method on `Span`. One drawback here is that we are blocked on back-deploying support for bridging `Array` to `Span`: it is only available on the most modern operating systems. Another drawback is that if our `Array` does not have a contiguous storage, we have to copy one: an `O(n)` operation. We are also blocked on bringing support for `Span` to collection types like `Dictionary` that do not already implement contiguous storage. + +A new `isTriviallyIdentical(to:)` method could work around all these restrictions. We could return in constant time *without* needing to copy memory to a contiguous storage. We could adopt this method on many types that might not *ever* have a contiguous storage. We could also work with our library maintainers to discuss a back-deployment strategy that could bring this method to legacy operating systems. + +`String` already ships a public-but-underscored version of this API.[^1] + +```swift +extension String { + /// Returns a boolean value indicating whether this string is identical to + /// `other`. + /// + /// Two string values are identical if there is no way to distinguish between + /// them. + /// + /// Comparing strings this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// string storage object. Therefore, identical strings are guaranteed to + /// compare equal with `==`, but not all equal strings are considered + /// identical. + /// + /// - Performance: O(1) + @_alwaysEmitIntoClient + public func _isIdentical(to other: Self) -> Bool { + self._guts.rawBits == other._guts.rawBits + } +} +``` + +We don’t see this API currently being used in Standard Library, but it’s possible this API is already being used to optimize performance in private frameworks from Apple. + +Many more examples of `isIdentical(to:)` functions are currently shipping in `Swift-Collections`[^2][^3][^4][^5][^6][^7][^8][^9][^10][^11][^12][^13], `Swift-Markdown`[^14], and `Swift-CowBox`[^15]. We also support `isIdentical(to:)` on the `Span` and `RawSpan` types from Standard Library.[^16] + +## Proposed Solution + +Before we look at the concrete types in this proposal, let’s begin with some more general principles and ideas we would expect for *all* concrete types to follow when adopting this new method. While this specific proposal is not adding a new protocol to Standard Library, it could be helpful to think of an “informal” protocol that guides us in choosing the types to adopt this new method. This could then serve as a guide for library maintainers that might choose to adopt this method on *new* types in the future. + +Suppose we are proposing an `isTriviallyIdentical(to:)` method on a type `T`. We propose the following axioms that library maintainers should adopt: +* `a.isTriviallyIdentical(to: a)` is always `true` (Reflexivity) +* If `T` is `Equatable`: + * `a.isTriviallyIdentical(to: b)` implies `a == b` (*or else `a` and `b` are exceptional values*) + * `isTriviallyIdentical(to:)` is *meaningfully* faster than `==` + +Let’s look through these axioms a little closer: + +**`a.isTriviallyIdentical(to: a)` is always `true` (Reflexivity)** + +* An implementation of `isTriviallyIdentical(to:)` that always returns `false` would not be an impactful API. We must guarantee that `isTriviallyIdentical(to:)` *can* return `true` at least *some* of the time. + +**If `T` is `Equatable` then `a.isTriviallyIdentical(to: b)` implies `a == b`** + +* This is the “fast path” performance optimization that will speed up the memoization examples we saw earlier. One important side effect here is that when `a.isTriviallyIdentical(to: b)` returns `false` we make *no* guarantees about whether or not `a` is equal to `b`. +* We assume this axiom holds only if `a` and `b` are not “exceptional” values. A example of an exceptional value would be if a container that is generic over `Float` contains `nan`. + +**If `T` is `Equatable` then `isTriviallyIdentical(to:)` is *meaningfully* faster than `==`** + +* While we could implement `isTriviallyIdentical(to:)` on types like `Int` or `Bool`, these types are not included in this proposal. Our proposal focuses on types that have the ability to return from `isTriviallyIdentical(to:)` meaningfully faster than `==`. If a type would perform the same amount of work in `isTriviallyIdentical(to:)` that takes place in `==`, our advice is that library maintainers should *not* adopt `isTriviallyIdentical(to:)` on this type. There should exist some legit internal fast-path on this type: like a pointer to a storage buffer that can be compared by reference identity. + +This proposal focuses on concrete types that are `Equatable`, but it might also be the case that a library maintainer would adopt `isTriviallyIdentical(to:)` on a type that is *not* `Equatable`: like `Span`. Our expectation is that a library maintainer adopting `isTriviallyIdentical(to:)`on a type that is not `Equatable` has some strong and impactful real-world use-cases ready to make use of this API. Just because a library maintainer *can* adopt this API does not imply they *should*. A library maintainer should also be ready to document for product engineers exactly what is implied from `a.isTriviallyIdentical(to: b)` returning `true`. What does it *mean* for `a` to be “identical” to `b` if we do not have the implication that `a == b`? We leave this decision to the library maintainers that have the most context on the types they have built. + +Suppose we had an `isTriviallyIdentical(to:)` method available on `Array`. Let’s go back to our earlier example and see how we can use this as an alternative to checking for value equality from our command line utility: + +```swift +final class Memoizer { + ... + + func result(for input: [Int]) -> [Int] { + if let result = self.result, + self.input.isTriviallyIdentical(to: input) { + return result + } else { + ... + } + } +} +``` + +We can run our previous example and confirm that we are not computing new results when the input has not changed: + +```swift +let memoizer = Memoizer() +let a = [1, 2, 3, 4] +print(memoizer.result(for: a)) +// Prints "computing new result" +// Prints "[2, 4]" +let b = a +print(memoizer.result(for: b)) +// Prints "[2, 4]" +let c = [1, 2, 3, 4] +print(memoizer.result(for: c)) +// Prints "computing new result" +// Prints "[2, 4]" +let d = [1, 2, 3, 4, 5, 6] +print(memoizer.result(for: d)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +let e = d +print(memoizer.result(for: e)) +// Prints "[2, 4, 6]" +let f = [1, 2, 3, 4, 5, 6] +print(memoizer.result(for: f)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +``` + +When we return `true` from `isTriviallyIdentical(to:)` we skip computing a new `result`. When `isTriviallyIdentical(to:)` returns `false` we compute a new `result`. Because `isTriviallyIdentical(to:)` *can* return `false` when two values are equal, we might be computing the same `result` more than once. The performance tradeoff is that because the operation to compute a new `result` is `O(n)` time, we might not *want* to perform another `O(n)` value equality operation to determine if we should compute a new `result`. Our `isTriviallyIdentical(to:)` will return in constant time no matter how many elements are in `input` or how expensive this value equality operation would be. + +Let’s go back to our SwiftUI app for displaying `Contact` values. Here is what the change would look like to use `isTriviallyIdentical(to:)` in place of value equality to memoize `favorites`: + +```swift +extension Favorites { + private final class Storage { + ... + + func update(_ contacts: [Contact]) { + if self.contacts.isTriviallyIdentical(to: contacts) == false { + self.contacts = contacts + self.favorites = nil + } + } + + ... + } +} +``` + +When we build and run our SwiftUI app we confirm that we are not computing new `favorites` when the user selects new `Contact` values from `FavoriteContactList`. + +## Detailed Design + +We propose adding `isTriviallyIdentical(to:)` methods to the following concrete types from Standard Library: +* `String` +* `String.UnicodeScalarView` +* `String.UTF16View` +* `String.UTF8View` +* `Substring` +* `Substring.UnicodeScalarView` +* `Substring.UTF16View` +* `Substring.UTF8View` +* `Array` +* `ArraySlice` +* `ContiguousArray` +* `Dictionary` +* `Set` +* `UnsafeBufferPointer` +* `UnsafeMutableBufferPointer` +* `UnsafeMutableRawBufferPointer` +* `UnsafeRawBufferPointer` +* `UTF8Span` +* `Span` +* `RawSpan` + +For each type being presented we codify important semantics in our header documentation. + +### `String` + +```swift +extension String { + /// Returns a boolean value indicating whether this string is identical to `other`. + /// + /// Two string values are identical if there is no way to distinguish between them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isTriviallyIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isTriviallyIdentical(to: b)` implies `b.isTriviallyIdentical(to: a)`. (Symmetry) + /// - If `a.isTriviallyIdentical(to: b)` and `b.isTriviallyIdentical(to: c)` are both `true`, then `a.isTriviallyIdentical(to: c)` is also `true`. (Transitivity) + /// - `a.isTriviallyIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isTriviallyIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isTriviallyIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing strings this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// string storage object. Therefore, identical strings are guaranteed to + /// compare equal with `==`, but not all equal strings are considered + /// identical. + /// + /// - Performance: O(1) + public func isTriviallyIdentical(to other: Self) -> Bool { ... } +} +``` + +The following types will adopt `isTriviallyIdentical(to:)` with the same semantic guarantees as `String`: +* `String.UnicodeScalarView` +* `String.UTF16View` +* `String.UTF8View` + +### `Substring` + +```swift +extension Substring { + /// Returns a boolean value indicating whether this substring is identical to `other`. + /// + /// Two substring values are identical if there is no way to distinguish between them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isTriviallyIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isTriviallyIdentical(to: b)` implies `b.isTriviallyIdentical(to: a)`. (Symmetry) + /// - If `a.isTriviallyIdentical(to: b)` and `b.isTriviallyIdentical(to: c)` are both `true`, then `a.isTriviallyIdentical(to: c)` is also `true`. (Transitivity) + /// - `a.isTriviallyIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isTriviallyIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isTriviallyIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing substrings this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// substring storage object. Therefore, identical substrings are guaranteed + /// to compare equal with `==`, but not all equal substrings are considered + /// identical. + /// + /// - Performance: O(1) + public func isTriviallyIdentical(to other: Self) -> Bool { ... } +} +``` + +The following types will adopt `isTriviallyIdentical(to:)` with the same semantic guarantees as `Substring`: +* `Substring.UnicodeScalarView` +* `Substring.UTF16View` +* `Substring.UTF8View` + +### `Array` + +```swift +extension Array { + /// Returns a boolean value indicating whether this array is identical to `other`. + /// + /// Two array values are identical if there is no way to distinguish between them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isTriviallyIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isTriviallyIdentical(to: b)` implies `b.isTriviallyIdentical(to: a)`. (Symmetry) + /// - If `a.isTriviallyIdentical(to: b)` and `b.isTriviallyIdentical(to: c)` are both `true`, then `a.isTriviallyIdentical(to: c)` is also `true`. (Transitivity) + /// - If `a` and `b` are `Equatable`, then `a.isTriviallyIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isTriviallyIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isTriviallyIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing arrays this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// array storage object. Therefore, identical arrays are guaranteed to + /// compare equal with `==`, but not all equal arrays are considered + /// identical. + /// + /// - Performance: O(1) + public func isTriviallyIdentical(to other: Self) -> Bool { ... } +} +``` + +The following types will adopt `isTriviallyIdentical(to:)` with the same semantic guarantees as `Array`: + +* `ArraySlice` +* `ContiguousArray` + +### `Dictionary` + +```swift +extension Dictionary { + /// Returns a boolean value indicating whether this dictionary is identical to `other`. + /// + /// Two dictionary values are identical if there is no way to distinguish between them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isTriviallyIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isTriviallyIdentical(to: b)` implies `b.isTriviallyIdentical(to: a)`. (Symmetry) + /// - If `a.isTriviallyIdentical(to: b)` and `b.isTriviallyIdentical(to: c)` are both `true`, then `a.isTriviallyIdentical(to: c)` is also `true`. (Transitivity) + /// - If `a` and `b` are `Equatable`, then `a.isTriviallyIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isTriviallyIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isTriviallyIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing dictionaries this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// dictionary storage object. Therefore, identical dictionaries are + /// guaranteed to compare equal with `==`, but not all equal dictionaries are + /// considered identical. + /// + /// - Performance: O(1) + public func isTriviallyIdentical(to other: Self) -> Bool { ... } +} +``` + +### `Set` + +```swift +extension Set { + /// Returns a boolean value indicating whether this set is identical to `other`. + /// + /// Two set values are identical if there is no way to distinguish between them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isTriviallyIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isTriviallyIdentical(to: b)` implies `b.isTriviallyIdentical(to: a)`. (Symmetry) + /// - If `a.isTriviallyIdentical(to: b)` and `b.isTriviallyIdentical(to: c)` are both `true`, then `a.isTriviallyIdentical(to: c)` is also `true`. (Transitivity) + /// - `a.isTriviallyIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isTriviallyIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isTriviallyIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing sets this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying set + /// storage object. Therefore, identical sets are guaranteed to compare equal + /// with `==`, but not all equal sets are considered identical. + /// + /// - Performance: O(1) + public func isTriviallyIdentical(to other: Self) -> Bool { ... } +} +``` + +### `UnsafeBufferPointer` + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + /// Returns a Boolean value indicating whether two `UnsafeBufferPointer` instances refer to the same region in memory. + public func isTriviallyIdentical(to other: Self) -> Bool { ... } +} +``` + +The following types will adopt `isTriviallyIdentical(to:)` with the same semantic guarantees as `UnsafeBufferPointer`: +* `UnsafeMutableBufferPointer` +* `UnsafeMutableRawBufferPointer` +* `UnsafeRawBufferPointer` + +### `UTF8Span` + +```swift +extension UTF8Span where Element: ~Copyable { + /// Returns a Boolean value indicating whether two `UTF8Span` instances refer to the same region in memory. + public func isTriviallyIdentical(to other: Self) -> Bool { ... } +``` + +The following types will adopt `isTriviallyIdentical(to:)` with the same semantic guarantees as `UTF8Span`: + +* `Span` +* `RawSpan` + +## Source Compatibility + +This proposal is additive and source-compatible with existing code. + +## Impact on ABI + +This proposal is additive and ABI-compatible with existing code. + +## Future Directions + +Any Standard Library types that are copy-on-write values could be good candidates to add `isTriviallyIdentical(to:)` functions. Here are some potential types to consider for a future proposal: + +* `Character` +* `Dictionary.Keys` +* `Dictionary.Values` +* `KeyValuePairs` +* `StaticBigInt` +* `StaticString` +* `UTF8Span` + +This proposal focuses on what we see as the most high-impact types to support from Standard Library. This proposal *is not* meant to discourage adding `isTriviallyIdentical(to:)` on any of these types at some point in the future. A follow-up “second-round” proposal could focus on these remaining types. + +## Alternatives Considered + +### Exposing Identity + +Our proposal introduces a new instance method on types that uses some underlying concept of “identity” to perform quick comparisons between two instances. A different approach would be to *return* the underlying identity to product engineers. If a product engineer wanted to test two instances for equality by identity they could perform that check themselves. + +There’s a lot of interesting directions to go with that idea… but we don’t think this is right approach for now. Introducing some concept of an “escapable” identity to value types like `Array` would require *a lot* of design. It’s overthinking the problem and solving for something we don’t need right now. + +### Different Names + +Multiple different names have been suggested for these operations. Including: + +* `isIdentical(to:)` +* `hasSameRepresentation(as:)` +* `isKnownIdentical(to:)` + +### Generic Contexts + +We proposed an “informal” protocol for library maintainers adopting `isTriviallyIdentical(to:)` on new types. Could we just build a new protocol in Standard Library? Maybe. We don’t see a big need for this right now. If product engineers would want for these types to conform to some common protocol to use across generic contexts, those product engineers can define that protocol in their own packages. If these protocols “incubate” in the community and become a common practice, we can consider proposing a new protocol in Standard Library. + +Instead of a new protocol, could we somehow add `isTriviallyIdentical(to:)` on `Equatable`? Maybe. This would introduce some more tricky questions. If we adopt this on *all* `Equatable` types, what do we do about types like `Int` or `Bool` that do not have an ability to perform a fast check for identity? Similar to our last idea, we prefer to focus just on concrete types for now. If product engineers want to make `isTriviallyIdentical(to:)` available on generic contexts across `Equatable`, we encourage them to experiment with their own extension for that. If this pattern becomes popular in the community, we can consider a new proposal to add this on `Equatable` in Standard Library. + +### Overload for Reference Comparison + +Could we “overload” the `===` operator from `AnyObject`? This proposal considers that question to be orthogonal to our goal of exposing identity equality with the `isTriviallyIdentical(to:)` methods. We could choose to overload `===`, but this would be a larger “conceptual” and “philosophical” change because the `===` operator is currently meant for `AnyObject` types — not value types like `Array`. + +### Support for Optionals + +We can support `Optional` values with the following extension: + +```swift +extension Optional { + public func isTriviallyIdentical(to other: Self) -> Bool + where Wrapped == Array { + switch (self, other) { + case let (value?, other?): + return value.isTriviallyIdentical(to: other) + case (nil, nil): + return true + default: + return false + } + } +} +``` + +Because this extension needs no `private` or `internal` symbols from Standard Library, we can omit this extension from our proposal. Product engineers that want this extension can choose to implement it for themselves. + +### Alternative Semantics + +Instead of publishing an `isTriviallyIdentical(to:)` method which implies two types *must* be equal, could we think of things from the opposite direction? Could we publish a `maybeDifferent` method which implies two types *might not* be equal? This then introduces some potential ambiguity for product engineers: to what extent does “maybe different” imply “probably different”? This ambiguity could be settled with extra documentation on the method, but `isTriviallyIdentical(to:)` solves that ambiguity up-front. The `isTriviallyIdentical(to:)` method is also consistent with the prior art in this space. + +In the same way this proposal exposes a way to quickly check if two values *must* be equal, product engineers might want a way to quickly check if two values *must not* be equal. This is an interesting idea, but this can exist as an independent proposal. We don’t need to block the review of this proposal on a review of `isNotIdentical` semantics. + +## Acknowledgments + +Thanks to [Ben Cohen](https://forums.swift.org/t/-/78792/7) for helping to think through and generalize the original use-case and problem-statement. + +Thanks to [David Nadoba](https://forums.swift.org/t/-/80496/61/) for proposing the formal equivalence relation semantics and axioms on concrete types. + +Thanks to [Xiaodi Wu](https://forums.swift.org/t/-/80496/67) for proposing that our equivalence relation semantics would carve-out for “exceptional” values like `Float.nan`. + +Thanks to [QuinceyMorris](https://forums.swift.org/t/-/82296/72) for proposing the name `isTriviallyIdentical(to:)`. + +[^1]: https://github.com/swiftlang/swift/blob/swift-6.1.2-RELEASE/stdlib/public/core/String.swift#L397-L415 +[^2]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/DequeModule/Deque._Storage.swift#L223-L225 +[^3]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/HashTreeCollections/HashNode/_HashNode.swift#L78-L80 +[^4]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/HashTreeCollections/HashNode/_RawHashNode.swift#L50-L52 +[^5]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Conformances/BigString%2BEquatable.swift#L14-L16 +[^6]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigString%2BUnicodeScalarView.swift#L77-L79 +[^7]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigString%2BUTF8View.swift#L39-L41 +[^8]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigString%2BUTF16View.swift#L39-L41 +[^9]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigSubstring.swift#L100-L103 +[^10]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigSubstring%2BUnicodeScalarView.swift#L94-L97 +[^11]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigSubstring%2BUTF8View.swift#L64-L67 +[^12]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigSubstring%2BUTF16View.swift#L87-L90 +[^13]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/Rope/Basics/Rope.swift#L68-L70 +[^14]: https://github.com/swiftlang/swift-markdown/blob/swift-6.1.1-RELEASE/Sources/Markdown/Base/Markup.swift#L370-L372 +[^15]: https://github.com/Swift-CowBox/Swift-CowBox/blob/1.1.0/Sources/CowBox/CowBox.swift#L19-L27 +[^16]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md diff --git a/proposals/0495-cdecl.md b/proposals/0495-cdecl.md new file mode 100644 index 0000000000..727e910214 --- /dev/null +++ b/proposals/0495-cdecl.md @@ -0,0 +1,228 @@ +# C compatible functions and enums + +* Proposal: [SE-0495](0495-cdecl.md) +* Author: [Alexis Laferrière](https://github.com/xymus) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Active Review (September 25th...October 9th, 2025)** +* Implementation: On `main` with the experimental feature flags `CDecl` for `@c`, and `CImplementation` for `@c @implementation`. With the exception of the `@objc` support for global functions which is available under the name `@_cdecl`. +* Review: ([pitch](https://forums.swift.org/t/pitch-formalize-cdecl/79557))([review](https://forums.swift.org/t/se-0495-c-compatible-functions-and-enums/82365)) + +## Introduction + +Implementing a C function in Swift eases integration of Swift and C code. This proposal introduces `@c` to mark Swift functions as callable from C, and enums as representable in C. It provides the same behavior under the `@objc` attribute for Objective-C compatible global functions. + +To expose the function to C clients, this proposal adds a new C block to the compatibility header where `@c` functions are printed. As an alternative, this proposal extends `@implementation` support to global functions, allowing users to declare the function in a hand-written C header. + +> Note: This proposal aims to formalize and extend the long experimental `@_cdecl`. While experimental this attribute has been widely in use so we will refer to it as needed for clarity in this document. + +## Motivation + +Swift already offers some integration with C, notably it can import declarations from C headers and call C functions. Swift also already offers a wide integration with Objective-C: import headers, call methods, print the compatibility header, and implement Objective-C classes in Swift with `@implementation`. These language features have proven to be useful for integration with Objective-C. Offering a similar language support for C will further ease integrating Swift and C, and encourage incremental adoption of Swift in existing C code bases. + +Offering a C compatibility type-checking ensures `@c` functions only reference types representable in C. This type-checking helps cross-platform development as one can define a `@c` while working from an Objective-C compatible environment and still see the restrictions from a C only environment. + +Printing the C representation of `@c` functions in a C header will enable a mixed-source software to easily call the functions from C code. The current generated header is limited to Objective-C and C++ content. Adding a section for C compatible clients will extend its usefulness to this language. + +Extending `@implementation` to support global C functions will provide support to developers through type-checking by ensuring the C declaration matches the corresponding definition in Swift. + +## Proposed solution + +We propose to introduce the new `@c` attribute for global functions and enums, extend `@objc` for global functions, and support `@c @implementation`. + +### `@c` global functions + +Introduce the `@c` attribute to mark a global function as a C function implemented in Swift. That function uses the C calling convention and its signature can only reference types representable in C. Its body is implemented in Swift as usual. The signature of that function is printed in the compatibility header using C corresponding types, allowing C source code to import the compatibility header and call the function. + +A `@c` function is declared with an optional C function name, by default the Swift base name is used as C name: +```swift +@c func foo() {} + +@c(mirrorCName) +func mirror(value: CInt) -> CInt { return value } +``` + +### `@objc` global functions + +Extends the `@objc` attribute to be accepted on a global function. It offers the same behavior as `@c` while allowing the signature to reference types representable in Objective-C. The signature of a `@objc` function is printed in the compatibility header using corresponding Objective-C types. + +A `@objc` function is declared with an optional C compatible name without parameter labels: + +```swift +@objc func bar() {} + +@objc(mirrorObjCName) +func objectMirror(value: NSObject) -> NSObject { return value } +``` + +> Note: The attribute `@objc` can be used on a global function to replace `@_cdecl` as it preserves the behavior of the unofficial attribute. + +### `@c` enums + +Accept `@c` on enums to mark them as C compatible. These enums can be referenced from `@c` or `@objc` functions. They are printed in the compatibility header as a C enum or a similar type. + +A `@c` enum may declare a custom C name, and must declare an integer raw type compatible with C: + +```swift +@c +enum CEnum: CInt { + case a + case b +} +``` + +The attribute `@objc` is already accepted on enums. These enums qualify as an Objective-C representable type and are usable from `@objc` global function signatures but not from `@c` functions. + +### `@c @implementation` global functions + +Extend support for the `@implementation` attribute, introduced in [SE-0436](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0436-objc-implementation.md), to global functions marked with either `@c` or `@objc`. These functions are declared in an imported C or Objective-C header, while the Swift function provides their implementation. Type-checking ensures the declaration matches the implementation signature in Swift. Functions marked `@implementation` are not printed in the compatibility header. + +The declaration and implementation are distinct across languages and must have matching names and types: + +```c +// C header +int cImplMirror(int value); +``` + +```swift +// Swift sources +@c @implementation +func cImplMirror(_ value: CInt) -> CInt { return value } +``` + +## Detailed design + +This proposal extends the language syntax, type-checking for both global functions and enums, supporting logic for `@implementation`, and the content printed to the compatibility header. + +### Syntax + +Required syntax changes involve one new attribute and the reuse of two existing attributes. + +* Introduce the attribute `@c` accepted on global functions and enums. It accepts one optional parameter specifying the corresponding C name of the declaration. The C name defaults to the Swift base identifier of the declaration, it doesn't consider parameter names. + +* Extend `@objc` to be accepted on global functions, using the optional parameter to define the C function name instead of the Objective-C symbol. Again here, the C function name defaults to the base identifier of the Swift function. + +* Extend `@implementation` to be accepted on global functions marked with either `@c` or `@objc`. + +### Type-checking of global functions signatures + +Global functions marked with `@c` or `@objc` need type-checking to ensure types used in their signature are representable in the target language. + +The following types are accepted in the signature of `@c` global functions: + +- Primitive types defined in the standard library: Int, UInt, Int8, Float, Double, Bool, etc. +- Pointers defined in the standard library: OpaquePointer and the variants of Unsafe{Mutable}{Raw}Pointer. +- C primitive types defined in the standard library: CChar, CInt, CUnsignedInt, CLong, CLongLong, etc. +- Function references using the C calling convention, marked with `@convention(c)`. +- SIMD types where the scalar is representable in C. +- Enums marked with `@c`. +- Imported C types. + +In addition to the types above, the following types should be accepted in the signature of `@objc` global functions: + +- `@objc` classes, enums and protocols. +- Imported Objective-C types. + +For both `@c` and `@objc` global functions, type-checking should reject: + +- Optional non-pointer types. +- Non-`@objc` classes. +- Swift structs. +- Non-`@c` enums. +- Protocol existentials. + +### Type-checking of `@c` enums + +For `@c` enums to be representable in C, type-checking should ensure the raw type is defined to an integer value that is itself representable in C. This is the same check as already applied to `@objc` enums. + +### `@c @implementation` and `@objc @implementation` + +A global function marked with `@c @implementation` or `@objc @implementation` needs to be associated with the corresponding declaration from imported headers. The compiler should report uses without a corresponding C declaration or inconsistencies in the match. + +### Compatibility header printing + +The compiler should print a single compatibility header for all languages, adding a block specific to C as is currently done for Objective-C and C++. Printing the header is requested using the preexisting compiler flags: `-emit-objc-header`, `-emit-objc-header-path` or `-emit-clang-header-path`. + +This C block declares the `@c` global functions and enums using C types, while `@objc` functions are printed in the Objective-C block with Objective-C types. + +The C block should be printed in a way it's parseable by compilers targeting C, Objective-C and C++. To do so, ensure that only C types are printed, there is no unprotected use of non-standard C features, and the syntax is C compatible. + +### Type mapping to C + +When printing `@c` functions in the compatibility header, Swift types are mapped to their corresponding C representations. Here is a partial mapping: + +- Swift `Bool` maps to C `bool` from `stdbool.h`. +- Swift `Int` maps to `ptrdiff_t`, `UInt` to `size_t`, `Int8` to `int8_t`, `UInt32` to `uint32_t`, etc. +- Swift floating-point types `Float` and `Double` map to C `float` and `double` respectively. +- Swift's version of C primitive types map to their C equivalents: `CInt` to `int`, `CLong` to long, etc. +- Swift SIMD types map to vector types printed as needed in the compatibility header. +- Function references with `@convention(c)` map to C function pointers. + +## Source compatibility + +This proposal preserves all source compatibility as the new features are opt-in. + +Existing adopters of `@_cdecl` can replace the attribute with `@objc` to preserve the same behavior. Alternatively, they can update it to `@c` to get the more restrictive C compatibility check. Using `@c` will however change how the corresponding C function is printed in the compatibility header so it may be necessary to update sources calling into the function. + +## ABI compatibility + +Marking a global function with `@c` or `@objc` makes it use the C calling convention. Adding or removing these attributes on a function is an ABI breaking change. Updating existing `@_cdecl` to `@objc` or `@c` is ABI stable. + +Adding or removing the `@c` attribute on an enum is ABI stable, but changing its raw type is not. + +Moving the implementation of an existing C function to Swift using `@c` or `@objc @implementation` within the same binary is ABI stable. + +## Implications on adoption + +The changes proposed here are backwards compatible with older runtimes. + +## Future directions + +This work opens the door to closer interoperability with the C language. + +### `@c` struct support + +A valuable addition would be supporting C compatible structs declared in Swift. We could consider exposing them to C as opaque data, or produce structs with a memory layout representable in C. Both have different use cases and advantages: + +* Using an opaque data representation would hide Swift details from C. Hiding these details allows the Swift struct to reference any Swift types and language features, without the concern of finding an equivalent C representation. This approach should be enough for the standard references to user data in C APIs. + +* Producing a Swift struct with a C memory layout would give the C code direct access to the data. This struct could be printed in the compatibility header as a normal C struct. This approach would need to be more restrictive on the Swift types and features used in the struct, starting with accepting only C representable types. + +### Custom calling conventions + +Defining a custom calling convention on a function may be a requirement by some API for callback functions and such. + +With this proposal, it should be possible to declare a custom calling convention by using `@c @implementation`. This allows to apply any existing C attribute on the definition in the C header. + +We could allow specifying the C calling conventions from Swift code with further work. Either by extending `@convention` to be accepted on `@c` and `@objc` global functions, and have it accept a wider set of conventions. Or by adding an optional named parameter to the `@c` attribute in the style of `@c(customCName, convention: stdcall)`. + +## Alternatives considered + +### `@c` attribute name + +This proposal uses the `@c` attribute on functions to identify them as C functions implemented in Swift and on enums to identify them as C compatible. This concise attribute clearly references interoperability with the C language. Plus, having an attribute specific to this feature aligns with `@objc` which is already used on some functions, enums, and for `@objc @implementation`. + +We considered some alternatives: + +- An official `@cdecl` may be more practical for discoverability but the terms *cdecl* and *decl* are compiler implementation details we do not wish to surface in the language. + +- An official `@expose(c)`, formalizing the experimental `@_expose(Cxx)`, would align the global function use case with what has been suggested for the C++ interop. However, sharing an attribute for the features described here may add complexity to both compiler implementation and user understanding of the language. + + While `@_expose(Cxx)` supports enums, it doesn't have the same requirement as `@objc` and `@c` for the raw type. The generated representation in the compatibility header for the enums differs too. The attribute `@_expose(Cxx)` also supports structs, while we consider supporting `@c` structs in the future, we have yet to pick the best approach so it would likely differ from the C++ one. + + Although sharing an attribute avoids adding a new one to the language, it also implies a similar behavior between the language interops. However, these behaviors already diverge and we may want to have each feature evolve differently in the future. + +### `@objc` attribute on global functions + +We use the `@objc` attribute on global functions to identify them as C functions implemented in Swift that are callable from Objective-C. This was more of a natural choice as `@objc` is already widely used for interoperability with Objective-C. + +We considered using instead `@c @objc` to make it more explicit that the behavior is similar to `@c`, and extending it to Objective-C is additive. We went against this option as it doesn't add much useful information besides being closer to the compiler implementation. + +### Compatibility header + +We decided to extend the existing compatibility header instead of introducing a new one specific to C compatibility. This allows content printed for Objective-C to reference C types printed earlier in the same header. Plus this follows the current behavior of the C++ interop which prints its own block in the same compatibility header. + +Since we use the same compatibility header, we also use the same compiler flags to request it being emitted. We considered adding a C specific flag as the main one, `-emit-objc-header`, is Objective-C specific. In practice build systems tend to use the `-path` variant, in that case we already have `-emit-clang-header-path` that applies well to the C language. We could add a `-emit-clang-header` flag but the practical use of such a flag would be limited. + +## Acknowledgements + +A special thank you goes to Becca Royal-Gordon, Joe Groff and many others for the past work on `@_cdecl` on which this proposal is built. diff --git a/proposals/0496-inline-always.md b/proposals/0496-inline-always.md new file mode 100644 index 0000000000..8a8784e97f --- /dev/null +++ b/proposals/0496-inline-always.md @@ -0,0 +1,535 @@ +# `@inline(always)` attribute + +* Proposal: [SE-0496](0496-inline-always.md) +* Authors: [Arnold Schwaighofer](https://github.com/aschwaighofer) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Accepted** +* Implementation: [swiftlang/swift#84178](https://github.com/swiftlang/swift/pull/84178) +* Review: ([pitch](https://forums.swift.org/t/pitch-inline-always-attribute/82040)) ([review](https://forums.swift.org/t/se-0496-inline-always-attribute/82480)) ([acceptance](https://forums.swift.org/t/accepted-se-0496-inline-always-attribute/82825)) + +## Introduction + +The Swift compiler performs an optimization that expands the body of a function +into the caller called inlining. Inlining exposes the code in the callee to the +code in the caller. After inlining, the Swift compiler has more context to +optimize the code across caller and callee leading to better optimization in +many cases. Inlining can increase code size. To avoid unnecessary code size +increases, the Swift compiler uses heuristics (properties of the code) to +determine whether to perform inlining. Sometimes these heuristics tell the +compiler not to inline a function even though it would be beneficial to do so. +The proposed attribute `@inline(always)` instructs the compiler to always inline +the annotated function into the caller giving the author explicit control over +the optimization. + +## Motivation + +Inlining a function referenced by a function call enables the optimizer to see +across function call boundaries. This can enable further optimization. The +decision whether to inline a function is driven by compiler heuristics that +depend on the shape of the code and can vary between compiler versions. + +In the following example the decision to inline might depend on the number of +instructions in `callee` and on detecting that the call to callee is frequently +executed because it is surrounded by a loop. Inlining this case would be +beneficial because the compiler is able to eliminate a store to a stack slot in +the `caller` after inlining the `callee` because the function's `inout` calling +convention ABI that requires an address no longer applies and further +optimizations are enabled by the caller's function's context. + +```swift +func callee(_ result: inout SomeValue, _ cond: Bool) { + result = SomeValue() + if cond { + // many lines of code ... + } +} + +func caller() { + var cond: Bool = false + var x : SomeValue = SomeValue() + for i in 0 ..< 1 { + callee(&x, cond) + } +} + +func callerAfterInlining(_ cond: Bool { + var x : SomeValue = SomeValue() + var cond: Bool = false + for i in 0 ..< 1 { + // Inlined `callee()`: + // Can keep `SomeValue()` in registers because no longer + // passed as an `inout` argument. + x = SomeValue() // Can hoist `x` out of the loop and perform constant + // propagation. + if cond { // Can remove the code under the conditional because it is + // known not to execute. + // many lines of code ... + } + } +} +``` + +The heuristic might fail to detect that code is frequently executed (surrounding +loop structures might be several calls up in the call chain) or the number of +instructions in the callee might be to large for the heuristic to decide that +inlining is beneficial. +Heuristics might change between compiler versions either directly or indirectly +because some properties of the internal representation of the optimized code +changes. +To give code authors reliable control over the inlining process we propose to +add an `@inline(always)` function attribute. + +This optimization control should instruct the compiler to inline the referenced +function or emit an error when it is not possible to do so. + +```swift +@inline(always) +func callee(_ result: inout SomeValue, _ cond: Bool) { + result = SomeValue() + if cond { + // many lines of code ... + } +} +``` + +## Proposed solution + +We desire for the attribute to function as an optimization control. That means +that the proposed `@inline(always)` attribute should emit an error diagnostic if +inlining is not possible in all optimization modes. However, this gets +complicated by the fact that the value of the function at a call site might be +determined dynamically at runtime: + +- Calls through first class function values + ```swift + @inline(always) f() {...} + + func a() { + let fv = f + fv() + } + ``` +- Calls through protocol values and protocol constraint generic types + ```swift + protocol P { + func method() + } + struct S : P { + @inline(always) + func method() {...} + } + func a(_ t: T) { + t.method() + let p : P = S() + p.method() + } + ``` +- Calls through class instance values and the method referenced is not `final` + ```swift + class C { + @inline(always) + func method() {...} + } + func a(c: C) { + c.method() + } +- Calls through `class` methods on `class` types and the method referenced + is not `final` + ```swift + class C { + @inline(always) + class func method() {...} + } + func a(c: C.Type) { + c.method() + } + ``` + +In such cases, the compiler cannot determine at a call site which function is +applied without doing non-local analysis: either dataflow, or class hiarchy +analysis. +These cases are in contrast to when the called function can statically be +determined purely by looking at the call site, we refer to this set as direct +function references in the following: + +- Calls to free standing functions +- Calls to methods of `actor`, `struct`, `enum` type +- Calls to final methods, final `class` type methods of `class` type, and + `static` type methods of `class` type + +Therefore, in cases where the value of the function at a usage site is +dynamically derived we don't emit an error even if the dynamic value of the +applied function was annotated with `@inline(always)`. We only emit an error if +the annotated function is directly referenced and something would cause it to be +not inlined or if some property at the declaration site of the function would +make it not possible in the common case. + +Listing the different scenarios that can occur for a function marked with +`@inline(always)`: + +1. A function can definitely be inlined at the use site: direct function + references barring recursion cycles +2. A function can never be always inlined at a use site and we diagnose an + error: cycles in `@inline(always)` functions calling each other and all + references are direct. +3. A function can not be inlined reliably and we diagnose an error at the + declaration site: non-final method declaration +4. A function can not be inlined and we don't diagnose an error: calls through + first class function values, protocol values, and protocol constraint generic + types. + +### Direct function references + +Calls to freestanding functions, methods of `enum`, `struct`, `actor` types, +final methods of `class` types, and `static` (but not `class`) type methods of +`class` types don't dynamically dispatch to different implementations. Calls to +such methods can always be inlined barring the recursion limitation (see later). +(case 1) + +```swift +struct S { + @inline(always) + final func method() {} +} + +func f() { + let s: S = ... + s.method() // can definitely be inlined +} + +class C { + @inline(always) + final func finalMethod() {} + + @inline(always) + static func method() {} + + @inline(always) + final class func finalTypeMethod() +} + +class Sub : C {} + +func f2() { + let c: C = ... + c.finalMethod() // can definitely be inlined + let c2: Sub = .. + c2.finalMethod() // can definitely be inlined + C.method() // can definitely be inlined + let c: C.Type = ... + c.finalTypeMethod() // can definitely be inlined +} + +@inline(always) +func freestanding() {} + +func f3() { + freestanding() // can definitely be inlined +} + +``` + +### Non final class methods + +Swift performs dynamic dispatch for non-final methods of classes and non final +`class` methods of classes based on the dynamic receiver type of the class +instance/class type value at a use site. Inferring the value of that dynamic +computation at compile time is not possible in many cases and the success of +inlining cannot be ensured. We treat a non-final method declaration with +`@inline(always)` as an declaration site error because we assume that the +intention of the attribute is that the method will be inlined in most cases and +this cannot be guaranteed (case 3). + +```swift +class C { + @inline(always) // error: non-final method marked @inline(always) + func method() {} + + @inline(always) // error: non-final method marked @inline(always) + class func class_method() {} +} + +class C2 : C { + @inline(always) // error: non-final method marked @inline(always) + override func method() {} + + class func class_method() {} +} + +func f(c: C, c2: C.Type) { + c.method() // dynamic type of c might be C or C2, could not ensure success + // of inlining in general + c2.class_method() // dynamic type of c2 might be C.self or C2.self, could not + // ensure success of inlining in general +} +``` + +### Recursion + +Repeatedly inlining `@inline(always)` functions calling each other would lead to +an infinite cycle of inlining. We can never follow the `@inline(always)` +semantics and diagnose an error (case 2). + +```swift +@inline(always) +func callee() { + ... + if cond2 { + caller() // error: caller is marked @inline(always) and would create an + // inlining cycle + } +} + +@inline(always) +func caller() { + ... + if cond { + callee() + } +} +``` + +### First class function values + +Swift allows for functions as first class objects. They can be assigned to +variables and passed as arguments. The reference function of a function value +cannot be reliably be determined at the usage and is therefore not diagnosed as +an error (case 4). + +```swift +@inline(always) +func callee() {} + +func use(_ f: () -> ()) { + f() +} +func useFunctionValue() { + let f = callee + ... + f() // function value use, may be inlined but not diagnosed if not + use(callee) // function value use, may be inlined in `use()` but not diagnosed + // if not +} +``` + +### Protocol methods + +Protocol constraint or protocol typed values require a dynamic computation to +determine the eventual method called. Inferring the value of the eventual method +called at compile time is not possible in general and the success of inlining +cannot be ensured. We don't diagnose a usage site error if the underlying method +is marked with `@inline(always)` (case 4) + +```swift +protocol P { + func method() +} +struct S : P { + @inline(always) + func method() {} +} +final class C : P { + @inline(always) + func method() {} +} + +@inline(always) +func generic (_ t: T) { + t.method() +} + +func f() { + let p: P = S() + p.method() // might not get inlined, not diagnosed + generic(S()) // might not get inlined, not diagnosed + let p2: P = C() + p2.method() // might not get inlined, not diagnosed + generic(C()) // might not get inlined, not diagnosed +} +``` + +### Optimization control as optimization hint + +A clever optimizer might be able to derive the dynamic value at the +call site, in such cases the optimizer shall respect the optimization control +and perform inlining. + +In the following example the functions will be inlined when build with higher +optimization levels than `-Onone`. + +```swift +@inline(always) +func binaryOp(_ left: T, _ right: T, _ op: (T, T) -> T) -> T { + op(left, right) +} + +@inline(always) +func add(_ left: Int, _ right: Int) -> Int { left + right } + +print(binaryOp(5, 10, add)) +print(binaryOp(5, 10) { add($0, $1) }) +``` + + +### Interaction with `@inlinable` + +`@inlinable` makes the function body available to clients (callers in other +modules) in library evolution mode. Functions with `open`, `public`, or +`package` level access cause emission of an ABI entry point for clients to call +but in the absence of aforementioned attributes do not make the body available +to the client. + +`@inline(always)` intention is to be able to guarantee that inlining will happen +for any caller inside or outside the defining module therefore it makes sense to +require the use of an `@inlinable` attribute with them. This attribute could be +required to be explicitly stated. And for it to be an error when the attribute +is omitted. + +```swift +@inline(always) +@inlinable +public func caller() { ... } + +@inline(always) // error: a public function marked @inline(always) must be marked @inlinable +public func callee() { +} +``` + +Alternatively, the attribute could be implicitly implied by the usage of +`@inline(always)`. We take the position that it should be implied to avoid the +redundancy of spelling it out. + +For access levels equal and lower than `internal` `@inlinable` is not implied. + +As a consequence all the rules that apply to `@inlinable` also apply to +`public`/`open`/`package` declarations marked with `@inline(always)`. + +```swift +internal func g() { ... } + +@inline(always) +public func inlinableImplied() { + g() // error: global function 'g()' is internal and cannot be referenced from an + '@inlinable' function +} +``` + +### Interaction with `@usableFromInline` + +A `public` `@inlinable` function can reference a function with `internal` access +if it is either `@inlinable` (see above) or `@usableFromInline`. `@usableFromInline` +ensures that there is a public entry point to the `internal` level function but +does not ensure that the body of the function is available to external +modules. Therefore, it is an error to combine `@inline(always)` with a +`@usableFromInline` function as we cannot guarantee that the function can +always be inlined. + +```swift +@inline(always) // error: an internal function marked with `@inline(always)` and + `@usableFromInline` could be referenced from an + `@inlinable` function and must be marked inlinable +@usableFromInline +internal func callee() {} + +@inlinable +public func caller() { + callee() // could not inline callee into external module +} +``` + +### Module internal access levels + +To mark `internal`, `private` and `fileprivate` function declarations +with `@inline(always)` does not imply the `@inlinable` attribute's semantics. +They can only be referenced from within the module. `internal` declarations can +be marked with `@inlinable` if this is required by the presence of other +`@inlinable` (or public `@inline(always)`) functions that reference them. + + +```swift +public func caller() { + callee() +} + +@inline(always) // okay because caller would force either `@inlinable` or + // `@usableFromInline` if it was marked @inlinable itself +internal func callee() { +} + + +@inline(always) // okay can only referenced from within the module +private func callee2() { +} +``` + +## Source compatibility + +This proposal is additive. Existing code has not used the attribute. It has no +impact on existing code. Existing references to functions in libraries that are +now marked with `@inline(always)` will continue to compile successfully with the +added effect that functions will get inlined (that could have happened with +changes to inlining heuristic). + +## ABI compatibility + +The addition of the attribute has no effect on ABI compatibility. We chose to +imply `@inlinable` for `public` (et al.) declarations which will continue to +emit an entry point for existing binary clients. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source +code with no deployment constraints and without affecting source or ABI +compatibility. + +## Future directions + +`@inline(always)` can be too restrictive in cases where inlining is only +required within a module. For such cases we can introduce an `@inline(module)` +attribute in the future. + + +```swift +@inlinable +public func caller() { + if coldPath { + callee() + } +} + +public func otherCaller() { + if hotPath { + callee() + } +} + +@inline(module) +@usableFromInline +internal func callee() { +} +``` + +## Alternatives considered + +We could treat `@inline(always)` as an optimization hint that does not need to +be enforced or applied at all optimization levels similar to how the existing +`@inline(__always)` attribute functions and not emit errors if it cannot be +guaranteed to be uphold when the function is directly referenced. +This would deliver less predictable optimization behavior in cases where authors +overlooked requirements for inlining to happen such as not marking a public +function as `@inlinable`. + +With respect to `@inlinable` an initial draft of the proposal suggested to +require spelling the `@inlinable` attribute on `public` declarations or an error +would be displayed. The argument was made that this would ensure that authors +would be aware of the additional semantics implied by the attribute: the body is +exposed. This was juxtaposed by the argument that spelling both `@inlinable` and +`@inline(always)` is redundant. + +## Acknowledgments + +Thanks to [Jordan Rose](https://forums.swift.org/t/optimization-controls-and-optimization-hints/81612/7) for pointing out that inlining can't be always guaranteed, specifically the case of closures. +Thanks to [Xiaodi Wu](https://forums.swift.org/t/pitch-inline-always-attribute/82040/7) for proposing inferring `@inlinable`. +Thanks to [Tony Allevato](https://github.com/swiftlang/swift-evolution/pull/2958#discussion_r2379238582) for suggesting to error on on non-final methods and +providing editing feedback. +Thanks to [Doug Gregor](https://github.com/DougGregor), [Joe Groff](https://github.com/jckarter), [Tim Kientzle](https://github.com/tbkka), and [Allan Shortlidge](https://github.com/tshortli) for discussions related to the feature. diff --git a/proposals/0497-definition-visibility.md b/proposals/0497-definition-visibility.md new file mode 100644 index 0000000000..8b1a9f735a --- /dev/null +++ b/proposals/0497-definition-visibility.md @@ -0,0 +1,227 @@ +# Controlling function definition visibility in clients + +* Proposal: [SE-0497](0497-definition-visibility.md) +* Authors: [Doug Gregor](https://github.com/DougGregor/) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Accepted** +* Implementation: Functionality is available via hidden `@_alwaysEmitIntoClient` and `@_neverEmitIntoClient` attributes in recent `main` snapshots. +* Review: ([pitch](https://forums.swift.org/t/pitch-controlling-function-definition-visibility-in-clients/82372)) ([review](https://forums.swift.org/t/se-0497-controlling-function-definition-visibility-in-clients/82666)) ([acceptance](https://forums.swift.org/t/accepted-se-0497-controlling-function-definition-visibility-in-clients/83068)) + +## Introduction + +A number of compiler optimizations depend on whether a caller of a particular function can see the definition (body) of that function. If the caller has access to the definition, it can be specialized for the call site, for example by substituting in generic arguments, constant argument values, or any other information known at the call site that can affect how the function is compiled. The (potentially specialized) definition can also be inlined, eliminating the overhead of a function call. Even if it is neither specialized nor inlined, the function's definition can be analyzed to help the caller produce better code. For example, if an object is passed into the function, and the function definition neither shares the object nor destroys it, the caller could potentially allocate the object on the stack rather than on the heap. + +On the other hand, making the function definition available to the caller means that you can no longer recompile only the function definition, relink the program, and see the effects of that change. This can mean slower incremental builds (because more must be rebuilt) as well as limiting the kinds of changes that can be made while retaining binary compatibility, for example when Library Evolution is enabled or a project wants to retain the ability to replace an implementation just by linking in different library versions. + +The `@inlinable` attribute introduced in [SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md) provides the ability to explicitly make the definition of a function available for callers. It ensures that the definition can be specialized, inlined, or otherwise used in clients to produce better code. However, it also compiles the definition into the module's binary so that the caller can choose to call it directly without emitting a copy of the definition. The `@inlinable` attribute has very little to do with inlining per se; rather, it's about visibility of the definition. + +This proposal provides explicit control over whether a function (1) generates a callable symbol in a binary and (2) makes its definition available for callers outside the module to be used for specialization, inlining, or other optimizations. + +## Motivation + +`@inlinable` strikes a balance that enables optimization without requiring it. One can make an existing function `@inlinable` without breaking binary compatibility, and it will enable better optimizations going forward. However, `@inlinable` by itself has proven insufficient. A separate hidden attribute, `@_alwaysEmitIntoClient`, states that the definition of the function is available to clients but is not guaranteed to be available in the library binary itself. It is the primary manner in which functionality can be added to the standard library without having an impact on its ABI, allowing the back-deployment of changes as well as keeping the ABI surface smaller. + +The Embedded Swift compilation model, in particular its use of aggressive cross-module optimization, makes essentially every function inlinable, including internal and private functions. That can introduce a different kind of problem: if a particular function needs to be part of the ABI, for example because it is referenced from outside of Swift, or needs to be replaceable at link time, there is no way to force the definition to be emitted into a particular binary. + +The `@inlinable` attribute provides explicit permission to the compiler to expose the definition of a function to its callers. However, the examples above illustrate that more control over when a function definition is emitted into a binary is needed for certain cases. + +### Existing controls for symbols and exposing function definitions + +The Swift language model itself mostly avoids defining what symbols are emitted into the binary when compiling code. However, there are some places in the language where the presence of a symbol in the final binary has been implied: + +* `@c` declarations ([SE-0495](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0495-cdecl.md)) and `@objc` classes need to produce symbols that can be referenced by compilers for the C and Objective-C languages, respectively. +* The `@main` attribute ([SE-0281](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0281-main-attribute.md)) needs to produce a symbol that is known to the operating system's loader as an entry point. +* The `@section` and `@used` attributes ([SE-0492](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0492-section-control.md)) imply that the compiler should produce a symbol. + +Similarly, whether the definition of a function is available to callers or not is mostly outside of the realm of the language. However, it has been touched on by several language features: + +* Library Evolution ([SE-0260](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0260-library-evolution.md)) explicitly ensures that clients cannot see the definition of a function within another module that was compiled with library evolution. +* The `@inlinable` attribute ([SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md)) explicitly allows clients to see the definition of a function across modules. This attribute became particularly important with Library Evolution (above), which normally prevents clients from seeing the definition of a function. + +These are relatively indirect ways in which one can state whether a symbol should be generated for a function and whether a function's definition can be used by clients outside of the module. + +### Effect of compiler optimizations + +Outside of those constraints on the interpretation of the language, the Swift compiler and build systems have an enormous amount of flexibility as to when to emit symbols and when to make the definition of functions available to clients. Various optimizations and compilation flags can affect both of these decisions. For example: + +* Incremental compilation (typical of debug builds) allows the compiler to avoid emitting symbols for `fileprivate` and `private` functions if they aren't needed elsewhere in the file, for example because all of their uses have been inlined (or there were no uses). +* Whole-module optimization (WMO) allows the definitions of `internal` , `fileprivate`, and `private` functions to be available to other source files in the same module. The compiler may choose not to emit symbols for `internal`, `fileprivate`, or `private` entities at all if they aren't needed. (For example, because they've been inlined into all callers) +* Cross-module optimization (CMO) allows the definitions of functions to be made available to clients in other modules. The "conservative" form of CMO, which has been enabled by the Swift Package Manager since Swift 5.8, does this primarily for `public` functions. A more aggressive form of cross-module optimization can also make the definitions of `internal`, `fileprivate`, or `private` entities available to clients (for the compiler's use only!). +* [Embedded Swift](https://github.com/swiftlang/swift-evolution/blob/main/visions/embedded-swift.md) relies on WMO and the aggressive CMO described above. It will also avoid emitting symbols to binaries unless they appear to be needed, which helps reduce code size. It is also necessary, because Embedded Swift cannot create symbols for certain functions, such as unspecialized generic functions. + +The same Swift source code may very well be compiled in a number of different ways at different times: debug builds often use incremental compilation, release builds generally use WMO and conservative CMO, and an embedded build would use the more aggressive CMO. The differences in symbol availability and the use of function definitions by clients don't generally matter. It is expected that the default behavior may shift over time: for example, the build system might enable progressively more aggressive CMO to improve performance. + +This proposal provides a mechanism to explicitly state the intent to emit symbols or provide the function definition to clients independent of the compilation mode, optimization settings, or language features (from the prior section) that infer these properties. This can be important, for example, when some external system expects certain symbols to be present, but the compiler might not choose to emit the symbol in some cases. + +### Implementation hiding + +When the definition of a function is not available to clients, it can make use of declarations that are not available to those clients. For example, it can use `internal` or `private` declarations from the same module or file, respectively, that have not been marked `@usableFromInline`. It can also use declarations imported from other modules that were imported using an `internal` or `private` import ([SE-0409](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md)). + +Although it was left to a [future direction in SE-0409](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md#hiding-dependencies-for-non-resilient-modules), implementation hiding can be used to avoid transitive dependencies on modules. For example, given the following setup: + +```swift +// module A +public func f() { } + +// module B +@_implementationOnly internal import A + +public func g() { + f() +} + +// module C +import B + +func h() { + g() +} +``` + +Module B makes use of module A only in its implementation, to call the function `A.f`. When module C imports module B, it conceptually does not need to know about module A. However, whether is true in practice depends on how the code is compiled: if `B` is built with library evolution enabled, then `C` does not need to know about `A`. If the modules are built with Embedded Swift, the definition of `B.g()` will be available to module `C`, so `C` will have to know about `A`. + +This can present a code portability problem for Embedded Swift. The proposed attribute that allows one to hide the definition of a function can help ensure that specific implementations stay hidden, making it possible to avoid transitive dependencies. It is by no means a complete solution: see the commentary about the effect of type layout on transitive dependencies in [SE-0409](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md). However, it is a practical solution for improving portability of Swift code across the different compilation modes. + +## Proposed solution + +This proposal introduces a new attribute `@export` that provides the required control over the ability of clients to make use of the callable interface or the definition of a particular function (or both). The `@export` attribute takes one or both of the following arguments in parentheses: + +* `interface`: means that a symbol is present in the binary in a manner that can be called by clients that can see the symbol. +* `implementation`: means that the function definition is available for clients to use for any purpose, including specialization, inlining, or merely analyzing the body for optimization purposes. + +The existing `@_alwaysEmitIntoClient` is subsumed by `@export(implementation)`, meaning that the definition is available and each client that uses it must emit their own copy of the definition, because there is no symbol. The `@_neverEmitIntoClient` attribute on `main` is subsumed by `@export(interface)`, meaning that a callable symbol is emitted but the definition is not available to callers for any reason. + +The two attributes introduced by [SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md) are partially subsumed by `@export`: + +* `@inlinable` for making a definition available to callers, similar to `@export(implementation)`. `@inlinable` is guaranteed to produce a symbol under Library Evolution, but there are no guarantees otherwise. In practice, non-Embedded Swift will produce a symbol, but Embedded Swift generally does not. +* `@usableFromInline` for making a less-than-public symbol available for use in an inlinable function (per SE-0193) is akin to `@export(interface)`. As with `@inlinable`, `@usableFromInline` does not *guarantee* the presence of a symbol in the way that `@export(interface)` does: in practice, Embedded Swift may not produce a symbol, but non-Embedded Swift will. Additionally, `@usableFromInline` does not prohibit the definition of the function from being made available to clients the way that `@export(interface)` does. + +`@export` cannot be combined with any of `@inlinable`, `@usableFromInline`, `@_alwaysEmitIntoClient`, or `@_neverEmitIntoClient`. + +## Detailed design + +`@export(implementation)` inherits all of the restrictions as `@inlinable` that are outlined in [SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md), for example, the definition itself can only reference public entities or those that are themselves `@usableFromInline` or `@export(interface)`. + +`@export(interface)` always produces a symbol in the object file. + +`@export` cannot be used without arguments. There can only be one of `@export(interface)` or `@export(implementation)` on a given declaration. + +### Relationship to access control + +The `@export` attribute is orthogonal to access control, because the visibility of a declaration for the programmer (`public`, `internal`, etc.) can be different from the visibility of its definition from the compiler's perspective, depending on what compiler optimizations are being used. For example, consider the following two modules: + +```swift +// module A +private func secret() { /* ... */ } + +public func f() { + secret() +} + +// module B +import A + +func g() { + f() +} +``` + +Module B cannot call the function `secret` under any circumstance. However, with aggressive CMO or Embedded Swift, the compiler will still make the definition available when compiling `B`, which can be used to (for example) inline both `f()` and `secret` into the body of `g`. + +If this behavior is not desired, there are two options. The easiest option to is mark `f` with `@export(interface)`, so that it's definition won't be available to clients and therefore cannot leak `secret`. Alternatively, the `secret` function could be marked as `@export(interface)` and `@inline(never)` to ensure that it is compiled to a symbol that it usable from outside of module A, and that its body is never inlined anywhere, including into `f`. It is still `private`, meaning that it still cannot be referenced by source code outside of that file. + +### Relationship to `@inline(always)` / `@inline(never)` + +The `@inline(always)` attribute [under discussion now](https://forums.swift.org/t/pitch-inline-always-attribute/82040) instructs the compiler to inline the function definition. The existing `@inline(never)` prevents the compiler from inlining the function. These have an effect on the heuristics the compiler's optimizer uses to decide when to inline. That's a matter of policy, but it does not impact whether a binary provides a definition for the given symbol that other callers can use. The notion of inlining is orthogonal to that of definition visibility and symbol availability. + +The following table captures the ways in which these attributes interact. + +| | `@inline(always)` | `@inline(never)` | (no `@inline`) | +| ------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| `@export(implementation)` | Always inlined everywhere; callers emit their own definitions. Use this when a function should not be part of the ABI and should always be inlined for performance reasons. | Never inlined; callers emit their own definitions. Use this when a function should not be part of the ABI but never needs to be inlined. | May be inlined. Callers emit their own definitions. Use when the function should not be part of the ABI, but leave it up to the optimizer to decide when to inline. | +| `@export(interface)` | Always inlined within the function's module; a symbol exists for callers outside the function's module. | Never inlined; callers may call the definition in the function's module. Use this to fully encapsulate a function definition so that it can be replaced at link time without affecting any other code. | May be inlined within the function's module, if the optimizer determines that it should be profitable. A symbol exists for callers from outside the module to use. | + +### Embedded Swift limitations + +Embedded Swift depends on "monomorphizing" all generic functions, meaning that the compiler needs to produce a specialize with concrete generic arguments for every use in the program. It is not possible to emit a single generic implementation that works for all generic arguments. This requires the definition to be available for any module that might create a specialization: + +```swift +// module A +private func secretGeneric(_: T) { } + +public func fGeneric(_ value: T) { + secretGeneric(T) +} + +// module B +struct MyType { } + +func h() { + fGeneric(MyType()) // must specialize fGeneric and secretGeneric +} +``` + +This means that generic functions are incompatible with `@export(interface)`, because there is no way to export a generic interface without the implementation. + +### `@export` attribute on stored properties and types + +Stored properties and types can also result in symbols being produced within the binary, in much the same manner as functions. For stored properties, the symbols describe the storage itself. For types, the symbols are for metadata associated with the type. In both cases, it can be reasonable for the compiler to defer creation of the symbol until use. For example, Embedded Swift will defer emission of a stored property until it is referenced, and will only emit type metadata when it is required (e.g., for use with `AnyObject`). + +For stored properties and types,`@export(interface)` would emit the stored property symbol and type metadata eagerly, similar to the emission of the symbol for a function. + +`@export(implementation)` is less immediately relevant. For stored properties, it could mean that the initializer value is available for clients to use. For types, it would effectively be the equivalent of `@frozen` in Library Evolution (`@frozen` exposes implementation and layout details), but there does not exist a notion of non-`@frozen` types outside of Library Evolution. + +## Source compatibility + +Introduces a new attribute. This could cause a source-compatibility problem with an attached macro of the same name, but otherwise has no impact. + +## ABI compatibility + +The attribute modifiers in this proposal explicitly control ABI. The `interface` argument to `@export` ensures that the function is part of the ABI; its absence in the `@export` attribute ensures that the function is not part of the ABI. Functions that do not adopt this new attribute are unaffected by this proposal. + +## Alternatives considered + +The primary alternatives here involving naming of this functionality. There are many other potential spellings for these features, including: + +### Parameterize `@inlinable` + +The `@inlinable` attribute already exists and is the combination of the proposed `@export(interface)` and `@export(implementation)` when Library Evolution is enabled. Outside of Library Evolution, `@inlinable` makes the definition available to clients but does not necessarily create a callable symbol. If we assume that the distinction is not important, we could extend `@inlinable` with two other forms: + +* `@inlinable(only)`, equivalent to `@export(implementation)`, means that the function definition is inlinable and can *only* be used by inlining. Practically speaking, this means that a client has must emit its own definition of the function in order to use it, because the defining module does not emit a copy as a public, callable symbol. This spelling formalizes `@_alwaysEmitIntoClient`. +* `@inlinable(never)`, equivalent to `@export(interface)`, means that the function definition is never available to callers, even if the compiler options (aggressive CMO, Embedded Swift, etc.) would make it so by default. The defining module will emit a public, callable symbol that the client can use. At the time of this writing, the Swift `main` branch provides this behavior with the `@_neverEmitIntoClient` attribute. + +The `@inlinable` attribute without a modifier would remain as specified in [SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md): it makes the definition available to the client (for any purpose) as well as emitting a public, callable symbol in Library Evolution. It is essentially a midpoint between `only` and `never`, leaving it up to the optimizer to determine when and how to make use of the definition. + +### Remove underscores from the existing attributes + +We could remove the underscores from the existing attributes, which use the phrase "emit into client" to mean that the client (calling module) is responsible for emitting the definition if it needs it. This would mean two new attributes, `@alwaysEmitIntoClient` and `@neverEmitIntoClient`. + +A variant of this would be `@emitIntoClient(always)` or `@emitIntoClient(never)`. That does leave space for a third option to be the equivalent of `@export(interface,implementation)`. + +### Make this part of access control + +Instead of introducing a new attribute, the `interface` and `implementation` options could be provided to the access control modifiers, such as `public`, `open`, and `package`. This runs some risk of complicating a feature that developers learn very early on (`public`) with a very advanced notion (the proposed `@export` attribute), but is otherwise equivalent. + +## Future Directions + +### Visibility extensions + +The `@export` attribute could be extended to support visibility-related descriptions, such as those provided by the GCC [`visibility` attribute](https://gcc.gnu.org/wiki/Visibility) as well as the Visual C++ notions of [`dllimport` and `dllexport`](https://learn.microsoft.com/en-us/cpp/cpp/dllexport-dllimport?view=msvc-170). For example: + +```swift +@export(interface, visibility: hidden) +public func f() { } +``` + +### Implementation hiding for internal and private imports + +One of the motivations for this proposal is implementation hiding for uses of `private` and `internal` imports. This motivation would be weakened by the [future direction in SE-0409](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md#hiding-dependencies-for-non-resilient-modules) where the transitive dependencies from those imports are hidden, because one would no longer need to use `@export(interface)` to explicitly hide a transitive dependency. In such a case, `@export(interface)` would make explicit what would happen implicitly when the definition of such a function references something available via an internal import. + +### Allow both `@export(interface)` and `@export(implementation)` + +The initial revision of this proposal allowed `@export(interface, implementation)` to mean that both a symbol is exported and the definition is available to callers. This feature was lacking any use case: it matches `@inlinable` for Library Evolution, but does not replace it elsewhere. The presence of this feature complicates the story, so it has been removed. If use cases are found for this case later, it's easy to lift the restriction. + +## Acknowledgments + +Thank you to Andy Trick for the `@export` syntax suggestion. + diff --git a/proposals/0498-runtime-demangle.md b/proposals/0498-runtime-demangle.md new file mode 100644 index 0000000000..c26ec6b2a7 --- /dev/null +++ b/proposals/0498-runtime-demangle.md @@ -0,0 +1,127 @@ +# Expose demangle function in Runtime module + +* Proposal: [SE-0498](0498-runtime-demangle.md) +* Previous Proposal: [SE-0262](0262-demangle.md) +* Authors: [Konrad'ktoso'Malawski](https://github.com/ktoso), [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Active Review (November 4 ..< 18)** +* Implementation: [PR #84788](https://github.com/swiftlang/swift/pull/84788) +* Review: + * Previous [pitch](https://forums.swift.org/t/demangle-function/25416/16) + * [(pitch)](https://forums.swift.org/t/pitch-expose-demangle-function-in-runtime-module/82605) + [(review)](https://forums.swift.org/t/se-0498-expose-demangle-function-in-runtime-module/83032) + + +## Introduction + +Swift symbols are subject to name mangling. These mangled names then show up in backtraces and other profiling tools. Mangled names may look something like this `$sSS7cStringSSSPys4Int8VG_tcfC` and often end up visible to developers, unless they are demangled before displaying. + +In manu situations, it is much preferable to demangle the identifiers before displaying them. For example, the previously shown identifier would can be demangled as `Swift.String.init(cString: Swift.UnsafePointer) -> Swift.String`, which is a nice human-readable format, that a Swift developer can easily understand. + +This proposal introduces a new API that allows calling out to the Swift runtime's demangler, without leaving the process. + +## Motivation + +Currently, many tools that need to display symbol names to developers are forced to create a process and execute the `swift-demangle` tool, or use unofficial runtime APIs to invoke the runtime's demangler. + +Neither of these approaches are satisfactionary, because either we are paying a high cost for creating processes, or we're relying on unofficial APIs. + +This proposal introduces an official `demangle(_: String) -> String?` function that offers a maintained and safe way to call the Swift demangler from a running Swift application. + +## Proposed solution + +We propose to introduce two `demangle` functions in the `Runtime` module: + +A simple demangle method, returning an optional `String`: + +```swift +public func demangle(_ mangledName: String) -> String? +``` + +The demangling function supports all valid Swift symbols. Valid Swift 5.0 and later symbols begin with `$s` (preceded by an optional platform-specific prefix). + +And an overload which accepts an `UTF8Span` into which the demangled string can be written: + +```swift +public func demangle( + _ mangledName: borrowing UTF8Span, + into output: inout OutputSpan +) -> DemanglingResult + +public enum DemanglingResult: Equatable { + case success + case failed + case truncated(Int) +} +``` + +The span accepting API is necessary for performance sensitive use-cases, which attempt to demangle symbols in process, before displaying or sending them for further processing. In those use-cases it is common to have a known maximum buffer size into which we are willing to write the demangled representation. + +The output from this API is an `OutputSpan` of `UTF8.CodeUnit`s, and it may not necessarily be well-formed UTF8, because of the potential of truncation happening between two code units which would render the UTF8 invalid. + +If the demangled representation does not fit the preallocated buffer, the demangle method will return `truncated(actualSize)` such that developers can determine by how much the buffer might need to be increased to handle the complete demangling. + +To construct an `UTF8Span` or valid `String` from the `OutputSpan` you can do the following: + +```swift +var demangledOutputSpan: OutputSpan = ... + +if demangle("$sSG", into: &demangledOutputSpan) == .success { + let utf8 = try UTF8Span(validating: demangledOutputSpan.span) + let demangledString = String(copying: utf8) + print(demangledString) // Swift.RandomNumberGenerator +} +``` + +### Demangling format + +While the mangled strings are part of Swift ABI and can therefore not really change on platforms with stable ABI, the demangled representation returned by the `demangle` functions is _not guaranteed to be stable in any way_. + +The demangled representation may change without any warning, during even patch releases of Swift. The returned strings should be treated mostly as nicer to present to developers human readable representations, and it is not a goal to provide any form of guarantee about the exact shape of these. + +## Future directions + +### Customizing demangling behavior with options + +We intentionally do not offer customization of demangling modes in this initial revision of this API. The Swift demangler does offer various options about how demangling should be performed, however we are not confident enough in selecting a subset of those options to offer as API at this point. + +Adding those options will be possible in future revisions, by adding an additional `options: DemanglingOptions` parameter to the demangle functions introduced in this proposal. + +## Source compatibility + +This proposal is purely additive. + +## ABI compatibility + +This proposal is purely additive. + +## Implications on adoption + +The runtime demangling func becoming an official entry point will help prevent libraries call swift internals. + +## Alternatives considered + +### Do nothing + +Not exposing this demangling capabilities officially, would result in tools authors continuing to use +unofficial ways to get to this API. + +It also means that further locking down access to `swift_` APIs may be difficult, +as they are crucial and load bearing for some tools (such as *continuous profiling*). +E.g. recent versions of Swift issue the following warning when accessing some of its `swift_` namespaced runtime functions: + +> symbol name 'swift_...' is reserved for the Swift runtime and cannot be directly referenced without causing unpredictable behavior; this will become an error + +So if we wanted to further lock down such uses, including `swift_demangle`, it would be preferable to offer an official solution instead. + +### Expose from Swift module + +Previously, this was considered to expose from just the `Swift` module, however the `Runtime` +module is much more aligned with the use and utility of this function. + +Demangling is already used by the `Backtrace` type which is located in the Runtime module, +so the demangling functions should be exposed from the same place. + +## Acknowledgments + +Thanks to [Alejandro Alonso](https://github.com/Azoy), who did an initial version of this pitch many years ago, and this proposal is heavily based on his initial pitch. diff --git a/proposals/testing/0001-refactor-bug-inits.md b/proposals/testing/0001-refactor-bug-inits.md new file mode 100644 index 0000000000..4d9a77b5e7 --- /dev/null +++ b/proposals/testing/0001-refactor-bug-inits.md @@ -0,0 +1,173 @@ +# Dedicated `.bug()` functions for URLs and IDs + +* Proposal: [ST-0001](0001-refactor-bug-inits.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Implemented (Swift 6.0)** +* Implementation: [swiftlang/swift-testing#401](https://github.com/swiftlang/swift-testing/pull/401) +* Review: ([pitch](https://forums.swift.org/t/pitch-dedicated-bug-functions-for-urls-and-ids/71842)) ([acceptance](https://forums.swift.org/t/swt-0001-dedicated-bug-functions-for-urls-and-ids/71842/2)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0001](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0001-refactor-bug-inits.md). + +## Introduction + +One of the features of swift-testing is a test traits system that allows +associating metadata with a test suite or test function. One trait in +particular, `.bug()`, has the potential for integration with development tools +but needs some refinement before integration would be practical. + +## Motivation + +A test author can associate a bug (AKA issue, problem, ticket, etc.) with a test +using the `.bug()` trait, to which they pass an "identifier" for the bug. The +swift-testing team's intent here was that a test author would pass the unique +identifier of the bug in the test author's preferred bug-tracking system (e.g. +GitHub Issues, Bugzilla, etc.) and that any tooling built around this trait +would be able to infer where the bug was located and how to view it. + +It became clear immediately that a generic system for looking up bugs by unique +identifier in an arbitrary and unspecified database wouldn't be a workable +solution. So we modified the description of `.bug()` to explain that, if the +identifier passed to it was a valid URL, then it would be "interpreted" as a URL +and that tools could be designed to open that URL as needed. + +This design change then placed the burden of parsing each `.bug()` trait and +potentially mapping it to a URL on tools. swift-testing itself avoids linking to +or using Foundation API such as `URL`, so checking for a valid URL inside the +testing library was not feasible either. + +## Proposed solution + +To solve the underlying problem and allow test authors to specify a URL when +available, or just an opaque identifier otherwise, we propose splitting the +`.bug()` function up into two overloads: + +- The first overload takes a URL string and additional optional metadata; +- The second overload takes a bug identifier as an opaque string or integer and, + optionally, a URL string. + +Test authors are then free to specify any combination of URL and opaque +identifier depending on the information they have available and their specific +needs. Tools authors are free to consume either or both of these properties and +present them where appropriate. + +## Detailed design + +The `Bug` trait type and `.bug()` trait factory function shall be refactored +thusly: + +```swift +/// A type representing a bug report tracked by a test. +/// +/// To add this trait to a test, use one of the following functions: +/// +/// - ``Trait/bug(_:_:)`` +/// - ``Trait/bug(_:id:_:)-10yf5`` +/// - ``Trait/bug(_:id:_:)-3vtpl`` +public struct Bug: TestTrait, SuiteTrait, Equatable, Hashable, Codable { + /// A URL linking to more information about the bug, if available. + /// + /// The value of this property represents a URL conforming to + /// [RFC 3986](https://www.ietf.org/rfc/rfc3986.txt). + public var url: String? + + /// A unique identifier in this bug's associated bug-tracking system, if + /// available. + /// + /// For more information on how the testing library interprets bug + /// identifiers, see . + public var id: String? + + /// The human-readable title of the bug, if specified by the test author. + public var title: Comment? +} + +extension Trait where Self == Bug { + /// Construct a bug to track with a test. + /// + /// - Parameters: + /// - url: A URL referring to this bug in the associated bug-tracking + /// system. + /// - title: Optionally, the human-readable title of the bug. + /// + /// - Returns: An instance of ``Bug`` representing the specified bug. + public static func bug(_ url: _const String, _ title: Comment? = nil) -> Self + + /// Construct a bug to track with a test. + /// + /// - Parameters: + /// - url: A URL referring to this bug in the associated bug-tracking + /// system. + /// - id: The unique identifier of this bug in its associated bug-tracking + /// system. + /// - title: Optionally, the human-readable title of the bug. + /// + /// - Returns: An instance of ``Bug`` representing the specified bug. + public static func bug(_ url: _const String? = nil, id: some Numeric, _ title: Comment? = nil) -> Self + + /// Construct a bug to track with a test. + /// + /// - Parameters: + /// - url: A URL referring to this bug in the associated bug-tracking + /// system. + /// - id: The unique identifier of this bug in its associated bug-tracking + /// system. + /// - title: Optionally, the human-readable title of the bug. + /// + /// - Returns: An instance of ``Bug`` representing the specified bug. + public static func bug(_ url: _const String? = nil, id: _const String, _ title: Comment? = nil) -> Self +} +``` + +The `@Test` and `@Suite` macros have already been modified so that they perform +basic validation of a URL string passed as input and emit a diagnostic if the +URL string appears malformed. + +## Source compatibility + +This change is expected to be source-breaking for test authors who have already +adopted the existing `.bug()` functions. This change is source-breaking for code +that directly refers to these functions by their signatures. This change is +source-breaking for code that uses the `identifier` property of the `Bug` type +or expects it to contain a URL. + +## Integration with supporting tools + +Tools that integrate with swift-testing and provide lists of tests or record +results after tests have run can use the `Bug` trait on tests to present +relevant identifiers and/or URLs to users. + +Tools that use the experimental event stream output feature of the testing +library will need a JSON schema for bug traits on tests. This work is tracked in +a separate upcoming proposal. + +## Alternatives considered + +- Inferring whether or not a bug identifier was a URL by parsing it at runtime + in tools. As discussed above, this option would require every tool that + integrates with swift-testing to provide its own URL-parsing logic. + +- Using different argument labels (e.g. the label `url` for the URL argument + and/or no label for the `id` argument.) We felt that URLs, which are + recognizable by their general structure, did not need labels. At least one + argument must have a label to avoid ambiguous resolution of the `.bug()` + function at compile time. + +- Inferring whether or not a bug identifier was a URL by parsing it at compile- + time or at runtime using `Foundation.URL` or libcurl. swift-testing actively + avoids linking to Foundation if at all possible, and libcurl would be a + platform-specific solution (Windows doesn't ship with libcurl, but does have + `InternetCrackUrlW()` whose parsing engine differs.) We also run the risk of + inappropriately interpreting some arbitrary bug identifier as a URL when it is + not meant to be parsed that way. + +- Removing the `.bug()` trait. We see this particular trait as having strong + potential for integration with tools and for use by test authors; removing it + because we can't reliably parse URLs would be unfortunate. + +## Acknowledgments + +Thanks to the swift-testing team and managers for their contributions! Thanks to +our community for the initial feedback around this feature. diff --git a/proposals/testing/0002-json-abi.md b/proposals/testing/0002-json-abi.md new file mode 100644 index 0000000000..516b986838 --- /dev/null +++ b/proposals/testing/0002-json-abi.md @@ -0,0 +1,428 @@ +# A stable JSON-based ABI for tools integration + +* Proposal: [ST-0002](0002-json-abi.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Implemented (Swift 6.0)** +* Implementation: [swiftlang/swift-testing#383](https://github.com/swiftlang/swift-testing/pull/383), + [swiftlang/swift-testing#402](https://github.com/swiftlang/swift-testing/pull/402) +* Review: ([pitch](https://forums.swift.org/t/pitch-a-stable-json-based-abi-for-tools-integration/72627)) ([acceptance](https://forums.swift.org/t/pitch-a-stable-json-based-abi-for-tools-integration/72627/4)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0002](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0002-json-abi.md). + +## Introduction + +One of the core components of Swift Testing is its ability to interoperate with +Xcode 16, VS Code, and other tools. Swift Testing has been fully open-sourced +across all platforms supported by Swift, and can be added as a package +dependency (or—eventually—linked from the Swift toolchain.) + +## Motivation + +Because Swift Testing may be used in various forms, and because integration with +various tools is critical to its success, we need it to have a stable interface +that can be used regardless of how it's been added to a package. There are a few +patterns in particular we know we need to support: + +- An IDE (e.g. Xcode 16) that builds and links its own copy of Swift Testing: + the copy used by the IDE might be the same as the copy that tests use, in + which case interoperation is trivial, but it may also be distinct if the tests + use Swift Testing as a package dependency. + + In the case of Xcode 16, Swift Testing is built as a framework much like + XCTest and is automatically linked by test targets in an Xcode project or + Swift package, but if the test target specifies a package dependency on Swift + Testing, that dependency will take priority when the test code is compiled. + +- An IDE (e.g. VS Code) that does _not_ link directly to Swift Testing (and + perhaps, as with VS Code, cannot because it is not natively compiled): such an + IDE needs a way to configure and invoke test code and then to read events back + as they occur, but cannot touch the Swift symbols used by the tests. + + In the case of VS Code, because it is implemented using TypeScript, it is not + able to directly link to Swift Testing or other Swift libraries. In order for + it to interpret events from a test run like "test started" or "issue + recorded", it needs to receive those events in a format it can understand. + +Tools integration is important to the success of Swift Testing. The more tools +provide integrations for it, the more likely developers are to adopt it. The +more developers adopt, the more tests are written. And the more tests are +written, the better our lives as software engineers will be. + +## Proposed solution + +We propose defining and implementing a stable ABI for using Swift Testing that +can be reliably adopted by various IDEs and other tools. There are two aspects +of this ABI we need to implement: + +- A stable entry point function that can be resolved dynamically at runtime (on + platforms with dynamic loaders such as Darwin, Linux, and Windows.) This + function needs a signature that will not change over time and which will take + input and pass back asynchronous output in a format that a wide variety of + tools will be able to interpret (whether they are written in Swift or not.) + + This function should be implemented in Swift as it is expected to be used by + code that can call into Swift, but which cannot rely on the specific binary + minutiae of a given copy of Swift Testing. + +- A stable format for input that can be passed to the entry point function and + which can also be passed at the command line; and a stable format for output + that can be consumed by tools to interpret test results. + + Some tools cannot directly link to Swift code and must instead rely on + command-line invocations of `swift test`. These tools will be able to pass + their test configuration and options as an argument in the stable format and + will be able to receive event information in the same stable format via a + dedicated channel such as a file or named pipe. + +> [!NOTE] +> This document proposes defining a stable format for input and output, but only +> actually defines the JSON schema for _output_. We intend to define the schema +> for input in a subsequent proposal. +> +> In the interim, early adopters can encode an instance of Swift Testing's +> `__CommandLineArguments_v0` type using `JSONEncoder`. + +## Detailed design + +We propose defining the stable input and output format using JSON as it is +widely supported across platforms and languages. The proposed JSON schema for +output is defined [here](https://github.com/swiftlang/swift-testing/blob/main/Documentation/ABI/JSON.md). + +### Example output + +The proposed schema is a sequence of JSON objects written to an event handler or +file stream. When a test run starts, Swift Testing first emits a sequence of +JSON objects representing each test that is part of the planned run. For +example, this is the JSON representation of Swift Testing's own `canGetStdout()` +test function: + +```json +{ + "kind": "test", + "payload": { + "displayName": "Can get stdout", + "id": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4", + "isParameterized": false, + "kind": "function", + "name": "canGetStdout()", + "sourceLocation": { + "column": 4, + "fileID": "TestingTests/FileHandleTests.swift", + "line": 33 + } + }, + "version": 0 +} +``` + +A tool that is observing this data stream can build a map or dictionary of test +IDs to comprehensive test details if needed. Once all tests in the planned run +have been written out, testing begins. Swift Testing writes a sequence of JSON +objects representing various events such as "test started" or "issue recorded". +For example, here is an abridged sequence of events generated for a test that +records a failed expectation: + +```json +{ + "kind": "event", + "payload": { + "instant": { + "absolute": 266418.545786299, + "since1970": 1718302639.76747 + }, + "kind": "testStarted", + "messages": [ + { + "symbol": "default", + "text": "Test \"Can get stdout\" started." + } + ], + "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" + }, + "version": 0 +} + +{ + "kind": "event", + "payload": { + "instant": { + "absolute": 266636.524236724, + "since1970": 1718302857.74857 + }, + "issue": { + "isKnown": false, + "sourceLocation": { + "column": 7, + "fileID": "TestingTests/FileHandleTests.swift", + "line": 29 + } + }, + "kind": "issueRecorded", + "messages": [ + { + "symbol": "fail", + "text": "Expectation failed: (EOF → -1) == (feof(fileHandle) → 0)" + } + ], + "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" + }, + "version": 0 +} + +{ + "kind": "event", + "payload": { + "instant": { + "absolute": 266636.524741106, + "since1970": 1718302857.74908 + }, + "kind": "testEnded", + "messages": [ + { + "symbol": "fail", + "text": "Test \"Can get stdout\" failed after 0.001 seconds with 1 issue." + } + ], + "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" + }, + "version": 0 +} +``` + +Each event includes zero or more "messages" that Swift Testing intends to +present to the user. These messages contain human-readable text as well as +abstractly-specified symbols that correspond to the output written to the +standard error stream of the test process. Tools can opt to present these +messages in whatever ways are appropriate for their interfaces. + +### Invoking from the command line + +When invoking `swift test`, we propose adding three new arguments to Swift +Package Manager: + +| Argument | Value Type | Description | +|---|:-:|---| +| `--configuration-path` | File system path | Specifies a path to a file, named pipe, etc. containing test configuration/options. | +| `--event-stream-output-path` | File system path | Specifies a path to a file, named pipe, etc. to which output should be written. | +| `--event-stream-version` | Integer | Specifies the version of the stable JSON schema to use for output. | + +The process for adding arguments to Swift Package Manager is separate from the +process for Swift Testing API changes, so the names of these arguments are +speculative and are subject to change as part of the Swift Package Manager +review process. + +If `--configuration-path` is specified, Swift Testing will open it for reading +and attempt to decode its contents as JSON. If `--event-stream-output-path` is +specified, Swift Testing will open it for writing and will write a sequence of +[JSON Lines](https://jsonlines.org) to it representing the data and events +produced by the test run. `--event-stream-version` determines the stable schema +used for output; pass `0` to match the schema proposed in this document. + +> [!NOTE] +> If `--event-stream-output-path` is specified but `--event-stream-version` is +> not, the format _currently_ used is based on direct JSON encodings of the +> internal Swift structures used by Swift Testing. This format is necessary to +> support Xcode 16 Beta 1. In the future, the default value of this argument +> will be assumed to equal the newest available JSON schema version (`0` as of +> this document's acceptance, i.e. the JSON schema will match what we are +> proposing here until a new schema supersedes it.) +> +> Tools authors that rely on the JSON schema are strongly advised to specify a +> version rather than relying on this behavior to avoid breaking changes in the +> future. + +On platforms that support them, callers can use a named pipe with +`--event-stream-output-path` to get live results back from the test run rather +than needing to wait until the file is closed by the test process. Named pipes +can be created on Darwin or Linux with the POSIX [`mkfifo()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mkfifo.2.html) +function or on Windows with the [`CreateNamedPipe()`](https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createnamedpipew) +function. + +If `--configuration-path` is specified in addition to explicit command-line +options like `--no-parallel`, the explicit command-line options take priority. + +### Invoking from Swift + +Tools that can link to and call Swift directly have the option of instantiating +the tools-only SPI type `Runner`, however this is only possible if the tools and +the test target link to the exact same copy of Swift Testing. To support tools +that may link to a different copy (intentionally or otherwise), we propose +adding an exported symbol to the Swift Testing library with the following Swift +signature: + +```swift +@_spi(ForToolsIntegrationOnly) +public enum ABIv0 { + /* ... */ + + /// The type of the entry point to the testing library used by tools that want + /// to remain version-agnostic regarding the testing library. + /// + /// - Parameters: + /// - configurationJSON: A buffer to memory representing the test + /// configuration and options. If `nil`, a new instance is synthesized + /// from the command-line arguments to the current process. + /// - recordHandler: A JSON record handler to which is passed a buffer to + /// memory representing each record as described in `ABI/JSON.md`. + /// + /// - Returns: Whether or not the test run finished successfully. + /// + /// - Throws: Any error that occurred prior to running tests. Errors that are + /// thrown while tests are running are handled by the testing library. + public typealias EntryPoint = @convention(thin) @Sendable ( + _ configurationJSON: UnsafeRawBufferPointer?, + _ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void + ) async throws -> Bool + + /// The entry point to the testing library used by tools that want to remain + /// version-agnostic regarding the testing library. + /// + /// The value of this property is a Swift function that can be used by tools + /// that do not link directly to the testing library and wish to invoke tests + /// in a binary that has been loaded into the current process. The value of + /// this property is accessible from C and C++ as a function with name + /// `"swt_abiv0_getEntryPoint"` and can be dynamically looked up at runtime + /// using `dlsym()` or a platform equivalent. + /// + /// The value of this property can be thought of as equivalent to + /// `swift test --event-stream-output-path` except that, instead of streaming + /// JSON records to a named pipe or file, it streams them to an in-process + /// callback. + public static var entryPoint: EntryPoint { get } +} +``` + +The inputs and outputs to this function are typed as `UnsafeRawBufferPointer` +rather than `Data` because the latter is part of Foundation, and adding a public +dependency on a Foundation type would make it very difficult for Foundation to +adopt Swift Testing. It is a goal of the Swift Testing team to keep our Swift +dependency list as small as possible. + +### Invoking from C or C++ + +We expect most tools that need to make use of this entry point will not be able +to directly link to the exported Swift symbol and will instead need to look it +up at runtime using a platform-specific interface such as [`dlsym()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlsym.3.html) +or [`GetProcAddress()`](https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-getprocaddress). +The `ABIv0.entryPoint` property's getter will be exported to C and C++ as: + +```c++ +extern "C" const void *_Nonnull swt_abiv0_getEntryPoint(void); +``` + +The value returned from this C function is a direct representation of the value +of `ABIv0.entryPoint` and can be cast back to its Swift function type using +[`unsafeBitCast(_:to:)`](https://developer.apple.com/documentation/swift/unsafebitcast%28_%3Ato%3A%29). + +On platforms where data-pointer-to-function-pointer conversion is disallowed per +the C standard, this operation is unsupported. See §6.3.2.3 and §J.5.7 of +[the C standard](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf). + +> [!NOTE] +> Swift Testing is statically linked into the main executable when it is +> included as a package dependency. On Linux and other platforms that use the +> ELF executable format, symbol information for the main executable may not be +> available at runtime unless the `--export-dynamic` flag is passed to the +> linker. + +## Source compatibility + +The changes proposed in this document are additive. + +## Integration with supporting tools + +Tools are able to use the proposed additions as described above. + +## Future directions + +- Extending the JSON schema to cover _input_ as well as _output_. As discussed, + we will do so in a subsequent proposal. + +- Extending the JSON schema to include richer information about events such as + specific mismatched values in `#expect()` calls. This information is complex + and we need to take care to model it efficiently and clearly. + +- Adding Markdown or other formats to event messages. Rich text can be used by + tools to emphasize values, switch to code voice, provide improved + accessibility, etc. + +- Adding additional entry points for different access patterns. We anticipate + that a Swift function and a command-line interface are sufficient to cover + most real-world use cases, but it may be the case that tools could use other + mechanisms for starting test runs such as: + - Pure C or Objective-C interfaces; + - A WebAssembly and/or JavaScript [`async`-compatible](https://github.com/WebAssembly/component-model/blob/2f447274b5028f54c549cb4e28ceb493a471dd4b/design/mvp/Async.md) + interface; + - Platform-specific interfaces; or + - Direct bindings to other languages like Rust, Go, C#, etc. + +## Alternatives considered + +- Doing nothing. If we made no changes, we would be effectively requiring + developers to use Xcode for all Swift Testing development and would be + requiring third-party tools to parse human-readable command-line output. This + approach would run counter to several of the Swift project's high-level goals + and would not represent a true cross-platform solution. + +- Using direct JSON encodings of Swift Testing's internal types to represent + output. We initially attempted this and you can see the results in the Swift + Testing repository if you look for "snapshot" types. A major downside became + apparent quickly: these data types don't make for particularly usable JSON + unless you're using `JSONDecoder` to convert back to them, and the default + JSON encodings produced with `JSONEncoder` are not stable if we e.g. add + enumeration cases with associated values or add non-optional fields to types. + +- Using a format other than JSON. We considered using XML, YAML, Apple property + lists, and a few other formats. JSON won out pretty quickly though: it is + widely supported across platforms and languages and it is trivial to create + Swift structures that encode to a well-designed JSON schema using + `JSONEncoder`. Property lists would be just as easy to create, but it is a + proprietary format and would not be trivially decodable on non-Apple platforms + or using non-Apple tools. + +- Exposing the C interface as a function that returns heap-allocated memory + containing a Swift function reference. This allows us to emit a "thick" Swift + function but requires callers to manually manage the resulting memory, and it + may be difficult to reason about code that requires an extra level of pointer + indirection. By having the C entry point function return a thin Swift function + instead, the caller need only bitcast it and can call it directly, and the + equivalent Swift interface can simply be a property getter rather than a + function call. + +- Exposing the C interface as a function that takes a callback and a completion + handler as might traditionally used by Objective-C callers, of the form: + + ```c++ + extern "C" void swt_abiv0_entryPoint( + __attribute__((__noescape__)) const void *_Nullable configurationJSON, + size_t configurationJSONLength, + void *_Null_unspecified context, + void (*_Nonnull recordHandler)( + __attribute__((__noescape__)) const void *recordJSON, + size_t recordJSONLength, + void *_Null_unspecified context + ), + void (*_Nonnull completionHandler)( + _Bool success, + void *_Null_unspecified context + ) + ); + ``` + + The known clients of the native entry point function are all able to call + Swift code and do not need this sort of interface. If there are other clients + that would need the entry point to use a signature like this one, it would be + straightforward to implement it in a future amendment to this proposal. + +## Acknowledgments + +Thanks much to [Dennis Weissmann](https://github.com/dennisweissmann) for his +tireless work in this area and to [Paul LeMarquand](https://github.com/plemarquand) +for putting up with my incessant revisions and nitpicking while he worked on +VS Code's Swift Testing support. + +Thanks to the rest of the Swift Testing team for reviewing this proposal and the +JSON schema and to the community for embracing Swift Testing! diff --git a/proposals/testing/0003-make-serialized-trait-api.md b/proposals/testing/0003-make-serialized-trait-api.md new file mode 100644 index 0000000000..488035bedf --- /dev/null +++ b/proposals/testing/0003-make-serialized-trait-api.md @@ -0,0 +1,154 @@ +# Make .serialized trait API + +* Proposal: [ST-0003](0003-make-serialized-trait-api.md) +* Authors: [Dennis Weissmann](https://github.com/dennisweissmann) +* Status: **Implemented (Swift 6.0)** +* Implementation: +[swiftlang/swift-testing#535](https://github.com/swiftlang/swift-testing/pull/535) +* Review: ([pitch](https://forums.swift.org/t/pitch-make-serialized-trait-public-api/73147)) ([acceptance](https://forums.swift.org/t/pitch-make-serialized-trait-public-api/73147/5)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0003](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0003-make-serialized-trait-api.md). + +## Introduction + +We propose promoting the existing `.serialized` trait to public API. This trait +enables developers to designate tests or test suites to run serially, ensuring +sequential execution where necessary. + +## Motivation + +The Swift Testing library defaults to parallel execution of tests, promoting +efficiency and isolation. However, certain test scenarios demand strict +sequential execution due to shared state or complex dependencies between tests. +The `.serialized` trait provides a solution by allowing developers to enforce +serial execution for specific tests or suites. + +While global actors ensure that only one task associated with that actor runs +at any given time, thus preventing concurrent access to actor state, tasks can +yield and allow other tasks to proceed, potentially interleaving execution. +That means global actors do not ensure that a specific test runs entirely to +completion before another begins. A testing library requires a construct that +guarantees that each annotated test runs independently and completely (in its +suite), one after another, without interleaving. + +## Proposed Solution + +We propose exposing the `.serialized` trait as a public API. This attribute can +be applied to individual test functions or entire test suites, modifying the +test execution behavior to enforce sequential execution where specified. + +Annotating just a single test in a suite does not enforce any serialization +behavior - the testing library encourages parallelization and the bar to +degrade overall performance of test execution should be high. +Additionally, traits apply inwards - it would be unexpected to impact the exact +conditions of a another test in a suite without applying a trait to the suite +itself. +Thus, this trait should only be applied to suites (to enforce serial execution +of all tests inside it) or parameterized tests. If applied to just a test this +trait does not have any effect. + +## Detailed Design + +The `.serialized` trait functions as an attribute that alters the execution +scheduling of tests. When applied, it ensures that tests or suites annotated +with `.serialized` run serially. + +```swift +/// A type that affects whether or not a test or suite is parallelized. +/// +/// When added to a parameterized test function, this trait causes that test to +/// run its cases serially instead of in parallel. When applied to a +/// non-parameterized test function, this trait has no effect. When applied to a +/// test suite, this trait causes that suite to run its contained test functions +/// and sub-suites serially instead of in parallel. +/// +/// This trait is recursively applied: if it is applied to a suite, any +/// parameterized tests or test suites contained in that suite are also +/// serialized (as are any tests contained in those suites, and so on.) +/// +/// This trait does not affect the execution of a test relative to its peers or +/// to unrelated tests. This trait has no effect if test parallelization is +/// globally disabled (by, for example, passing `--no-parallel` to the +/// `swift test` command.) +/// +/// To add this trait to a test, use ``Trait/serialized``. +public struct ParallelizationTrait: TestTrait, SuiteTrait {} + +extension Trait where Self == ParallelizationTrait { + /// A trait that serializes the test to which it is applied. + /// + /// ## See Also + /// + /// - ``ParallelizationTrait`` + public static var serialized: Self { get } +} +``` + +The call site looks like this: + +```swift +@Test(.serialized, arguments: Food.allCases) func prepare(food: Food) { + // This function will be invoked serially, once per food, because it has the + // .serialized trait. +} + +@Suite(.serialized) struct FoodTruckTests { + @Test(arguments: Condiment.allCases) func refill(condiment: Condiment) { + // This function will be invoked serially, once per condiment, because the + // containing suite has the .serialized trait. + } + + @Test func startEngine() async throws { + // This function will not run while refill(condiment:) is running. One test + // must end before the other will start. + } +} + +@Suite struct FoodTruckTests { + @Test(.serialized) func startEngine() async throws { + // This function will not run serially - it's not a parameterized test and + // the suite is not annotated with the `.serialized` trait. + } + + @Test func prepareFood() async throws { + // It doesn't matter if this test is `.serialized` or not, traits applied + // to other tests won't affect this test don't impact other tests. + } +} +``` + +## Source Compatibility + +Introducing `.serialized` as a public API does not have any impact on existing +code. Tests will continue to run in parallel by default unless explicitly +marked with `.serialized`. + +## Integration with Supporting Tools + +N/A. + +## Future Directions + +There might be asks for more advanced and complex ways to affect parallelization +which include ways to specify dependencies between tests ie. "Require `foo()` to +run before `bar()`". + +## Alternatives Considered + +Alternative approaches, such as relying solely on global actors for test +isolation, were considered. However, global actors do not provide the +deterministic, sequential execution required for certain testing scenarios. The +`.serialized` trait offers a more explicit and flexible mechanism, ensuring +that each designated test or suite runs to completion without interruption. + +Various more complex parallelization and serialization options were discussed +and considered but ultimately disregarded in favor of this simple yet powerful +implementation. + +## Acknowledgments + +Thanks to the swift-testing team and managers for their contributions! Thanks +to our community for the initial feedback around this feature. diff --git a/proposals/testing/0004-constrain-the-granularity-of-test-time-limit-durations.md b/proposals/testing/0004-constrain-the-granularity-of-test-time-limit-durations.md new file mode 100644 index 0000000000..ea00728fca --- /dev/null +++ b/proposals/testing/0004-constrain-the-granularity-of-test-time-limit-durations.md @@ -0,0 +1,205 @@ +# Constrain the granularity of test time limit durations + +* Proposal: [ST-0004](0004-constrain-the-granularity-of-test-time-limit-durations.md) +* Authors: [Dennis Weissmann](https://github.com/dennisweissmann) +* Status: **Implemented (Swift 6.0)** +* Implementation: +[swiftlang/swift-testing#534](https://github.com/swiftlang/swift-testing/pull/534) +* Review: ([pitch](https://forums.swift.org/t/pitch-constrain-the-granularity-of-test-time-limit-durations/73146)) ([acceptance](https://forums.swift.org/t/pitch-constrain-the-granularity-of-test-time-limit-durations/73146/3)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0004](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0004-constrain-the-granularity-of-test-time-limit-durations.md). + +## Introduction + +Sometimes tests might get into a state (either due the test code itself or due +to the code they're testing) where they don't make forward progress and hang. +Swift Testing provides a way to handle these issues using the TimeLimit trait: + +```swift +@Test(.timeLimit(.minutes(60)) +func testFunction() { ... } +``` + +Currently there exist multiple overloads for the `.timeLimit` trait: one that +takes a `Swift.Duration` which allows for arbitrary `Duration` values to be +passed, and one that takes a `TimeLimitTrait.Duration` which constrains the +minimum time limit as well as the increment to 1 minute. + +## Motivation + +Small time limit values in particular cause more harm than good due to tests +running in environments with drastically differing performance characteristics. +Particularly when running in CI systems or on virtualized hardware tests can +run much slower than at desk. +Swift Testing should help developers use a reasonable time limit value in its +API without developers having to refer to the documentation. + +It is crucial to emphasize that unit tests failing due to exceeding their +timeout should be exceptionally rare. At the same time, a spurious unit test +failure caused by a short timeout can be surprisingly costly, potentially +leading to an entire CI pipeline being rerun. Determining an appropriate +timeout for a specific test can be a challenging task. + +Additionally, when the system intentionally runs multiple tests simultaneously +to optimize resource utilization, the scheduler becomes the arbiter of test +execution. Consequently, the test may take significantly longer than +anticipated, potentially due to external factors beyond the control of the code +under test. + +A unit test should be capable of failing due to hanging, but it should not fail +due to being slow, unless the developer has explicitly indicated that it +should, effectively transforming it into a performance test. + +The time limit feature is *not* intended to be used to apply small timeouts to +tests to ensure test runtime doesn't regress by small amounts. This feature is +intended to be used to guard against hangs and pathologically long running +tests. + +## Proposed Solution + +We propose changing the `.timeLimit` API to accept values of a new `Duration` +type defined in `TimeLimitTrait` which only allows for `.minute` values to be +passed. +This type already exists as SPI and this proposal is seeking to making it API. + +## Detailed Design + +The `TimeLimitTrait.Duration` struct only has one factory method: +```swift +public static func minutes(_ minutes: some BinaryInteger) -> Self +``` + +That ensures 2 things: +1. It's impossible to create short time limits (under a minute). +2. It's impossible to create high-precision increments of time. + +Both of these features are important to ensure the API is self documenting and +conveying the intended purpose. + +For parameterized tests these time limits apply to each individual test case. + +The `TimeLimitTrait.Duration` struct is declared as follows: + +```swift +/// A type that defines a time limit to apply to a test. +/// +/// To add this trait to a test, use one of the following functions: +/// +/// - ``Trait/timeLimit(_:)`` +@available(macOS 13.0, iOS 16.0, tvOS 16.0, watchOS 9.0, *) +public struct TimeLimitTrait: TestTrait, SuiteTrait { + /// A type representing the duration of a time limit applied to a test. + /// + /// This type is intended for use specifically for specifying test timeouts + /// with ``TimeLimitTrait``. It is used instead of Swift's built-in `Duration` + /// type because test timeouts do not support high-precision, arbitrarily + /// short durations. The smallest allowed unit of time is minutes. + public struct Duration: Sendable { + + /// Construct a time limit duration given a number of minutes. + /// + /// - Parameters: + /// - minutes: The number of minutes the resulting duration should + /// represent. + /// + /// - Returns: A duration representing the specified number of minutes. + public static func minutes(_ minutes: some BinaryInteger) -> Self + } + + /// The maximum amount of time a test may run for before timing out. + public var timeLimit: Swift.Duration { get set } +} +``` + +The extension on `Trait` that allows for `.timeLimit(...)` to work is defined +like this: + +```swift +/// Construct a time limit trait that causes a test to time out if it runs for +/// too long. +/// +/// - Parameters: +/// - timeLimit: The maximum amount of time the test may run for. +/// +/// - Returns: An instance of ``TimeLimitTrait``. +/// +/// Test timeouts do not support high-precision, arbitrarily short durations +/// due to variability in testing environments. The time limit must be at +/// least one minute, and can only be expressed in increments of one minute. +/// +/// When this trait is associated with a test, that test must complete within +/// a time limit of, at most, `timeLimit`. If the test runs longer, an issue +/// of kind ``Issue/Kind/timeLimitExceeded(timeLimitComponents:)`` is +/// recorded. This timeout is treated as a test failure. +/// +/// The time limit amount specified by `timeLimit` may be reduced if the +/// testing library is configured to enforce a maximum per-test limit. When +/// such a maximum is set, the effective time limit of the test this trait is +/// applied to will be the lesser of `timeLimit` and that maximum. This is a +/// policy which may be configured on a global basis by the tool responsible +/// for launching the test process. Refer to that tool's documentation for +/// more details. +/// +/// If a test is parameterized, this time limit is applied to each of its +/// test cases individually. If a test has more than one time limit associated +/// with it, the shortest one is used. A test run may also be configured with +/// a maximum time limit per test case. +public static func timeLimit(_ timeLimit: Self.Duration) -> Self +``` + +And finally, the call site of the API looks like this: + +```swift +@Test(.timeLimit(.minutes(60)) +func serve100CustomersInOneHour() async { + for _ in 0 ..< 100 { + let customer = await Customer.next() + await customer.order() + ... + } +} +``` + +The `TimeLimitTrait.Duration` struct has various `unavailable` overloads that +are included for diagnostic purposes only. They are all documented and +annotated like this: + +```swift +/// Construct a time limit duration given a number of . +/// +/// This function is unavailable and is provided for diagnostic purposes only. +@available(*, unavailable, message: "Time limit must be specified in minutes") +``` + +## Source Compatibility + +This impacts clients that have adopted the `.timeLimit` trait and use overloads +of the trait that accept an arbitrary `Swift.Duration` except if they used the +`minutes` overload. + +## Integration with Supporting Tools + +N/A + +## Future Directions + +We could allow more finegrained time limits in the future that scale with the +performance of the test host device. +Or take a more manual approach where we detect the type of environment +(like CI vs local) and provide a way to use different timeouts depending on the +environment. + +## Alternatives Considered + +We have considered using `Swift.Duration` as the currency type for this API but +decided against it to avoid common pitfalls and misuses of this feature such as +providing very small time limits that lead to flaky tests in different +environments. + +## Acknowledgments + +The authors acknowledge valuable contributions and feedback from the Swift +Testing community during the development of this proposal. diff --git a/proposals/testing/0005-ranged-confirmations.md b/proposals/testing/0005-ranged-confirmations.md new file mode 100644 index 0000000000..7a53229ae2 --- /dev/null +++ b/proposals/testing/0005-ranged-confirmations.md @@ -0,0 +1,190 @@ +# Range-based confirmations + +* Proposal: [ST-0005](0005-ranged-confirmations.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Implemented (Swift 6.1)** +* Bug: rdar://138499457 +* Implementation: [swiftlang/swift-testing#598](https://github.com/swiftlang/swift-testing/pull/598), [swiftlang/swift-testing#689](https://github.com/swiftlang/swift-testing/pull689) +* Review: ([pitch](https://forums.swift.org/t/pitch-range-based-confirmations/74589)) ([acceptance](https://forums.swift.org/t/pitch-range-based-confirmations/74589/7)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0005](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0005-ranged-confirmations.md). + +## Introduction + +Swift Testing includes [an interface](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/confirmation(_:expectedcount:isolation:sourcelocation:_:)) +for checking that some asynchronous event occurs a given number of times +(typically exactly once or never at all.) This proposal enhances that interface +to allow arbitrary ranges of event counts so that a test can be written against +code that may not always fire said event the exact same number of times. + +## Motivation + +Some tests rely on fixtures or external state that is not perfectly +deterministic. For example, consider a test that checks that clicking the mouse +button will generate a `.mouseClicked` event. Such a test might use the +`confirmation()` interface: + +```swift +await confirmation(expectedCount: 1) { mouseClicked in + var eventLoop = EventLoop() + eventLoop.eventHandler = { event in + if event == .mouseClicked { + mouseClicked() + } + } + await eventLoop.simulate(.mouseClicked) +} +``` + +But what happens if the user _actually_ clicks a mouse button while this test is +running? That might trigger a _second_ `.mouseClicked` event, and then the test +will fail spuriously. + +## Proposed solution + +If the test author could instead indicate to Swift Testing that their test will +generate _one or more_ events, they could avoid spurious failures: + +```swift +await confirmation(expectedCount: 1...) { mouseClicked in + ... +} +``` + +With this proposal, we add an overload of `confirmation()` that takes any range +expression instead of a single integer value (which is still accepted via the +existing overload.) + +## Detailed design + +A new overload of `confirmation()` is added: + +```swift +/// Confirm that some event occurs during the invocation of a function. +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - expectedCount: A range of integers indicating the number of times the +/// expected event should occur when `body` is invoked. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to which any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Returns: Whatever is returned by `body`. +/// +/// - Throws: Whatever is thrown by `body`. +/// +/// Use confirmations to check that an event occurs while a test is running in +/// complex scenarios where `#expect()` and `#require()` are insufficient. For +/// example, a confirmation may be useful when an expected event occurs: +/// +/// - In a context that cannot be awaited by the calling function such as an +/// event handler or delegate callback; +/// - More than once, or never; or +/// - As a callback that is invoked as part of a larger operation. +/// +/// To use a confirmation, pass a closure containing the work to be performed. +/// The testing library will then pass an instance of ``Confirmation`` to the +/// closure. Every time the event in question occurs, the closure should call +/// the confirmation: +/// +/// ```swift +/// let minBuns = 5 +/// let maxBuns = 10 +/// await confirmation( +/// "Baked between \(minBuns) and \(maxBuns) buns", +/// expectedCount: minBuns ... maxBuns +/// ) { bunBaked in +/// foodTruck.eventHandler = { event in +/// if event == .baked(.cinnamonBun) { +/// bunBaked() +/// } +/// } +/// await foodTruck.bakeTray(of: .cinnamonBun) +/// } +/// ``` +/// +/// When the closure returns, the testing library checks if the confirmation's +/// preconditions have been met, and records an issue if they have not. +/// +/// If an exact count is expected, use +/// ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` instead. +public func confirmation( + _ comment: Comment? = nil, + expectedCount: some RangeExpression & Sequence Sendable, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: (Confirmation) async throws -> sending R +) async rethrows -> R +``` + +### Ranges without lower bounds + +Certain types of range, specifically [`PartialRangeUpTo`](https://developer.apple.com/documentation/swift/partialrangeupto) +and [`PartialRangeThrough`](https://developer.apple.com/documentation/swift/partialrangethrough), +may have surprising behavior when used with this new interface because they +implicitly include `0`. If a test author writes `...10`, do they mean "zero to +ten" or "one to ten"? The programmatic meaning is the former, but some test +authors might mean the latter. If an event does not occur, a test using +`confirmation()` and this `expectedCount` value would pass when the test author +meant for it to fail. + +The unbounded range (`...`) type `UnboundedRange` is effectively useless when +used with this interface and any use of it here is almost certainly a programmer +error. + +`PartialRangeUpTo` and `PartialRangeThrough` conform to `RangeExpression`, but +not to `Sequence`, so they will be rejected at compile time. `UnboundedRange` is +a non-nominal type and will not match either. We will provide unavailable +overloads of `confirmation()` for these types with messages that explain why +they are unavailable, e.g.: + +```swift +@available(*, unavailable, message: "Unbounded range '...' has no effect when used with a confirmation.") +public func confirmation( + _ comment: Comment? = nil, + expectedCount: UnboundedRange, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: (Confirmation) async throws -> R +) async rethrows -> R +``` + +## Source compatibility + +This change is additive. Existing tests are unaffected. + +Code that refers to `confirmation(_:expectedCount:isolation:sourceLocation:_:)` +by symbol name may need to add a contextual type to disambiguate the two +overloads at compile time. + +## Integration with supporting tools + +The type of the associated value `expected` for the `Issue.Kind` case +`confirmationMiscounted(actual:expected:)` will change from `Int` to +`any RangeExpression & Sendable`[^1]. Tools that implement event handlers and +distinguish between `Issue.Kind` cases are advised not to assume the type of +this value is `Int`. + +## Alternatives considered + +- Doing nothing. We have identified real-world use cases for this interface + including in Swift Testing’s own test target. +- Allowing the use of any value as the `expectedCount` argument so long as it + conforms to a protocol `ExpectedCount` (we'd have range types and `Int` + conform by default.) It was unclear what this sort of flexibility would let + us do, and posed challenges for encoding and decoding events and issues when + using the JSON event stream interface. + +## Acknowledgments + +Thanks to the testing team for their help preparing this pitch! + +[^1]: In the future, this type will change to + `any RangeExpression & Sendable`. Compiler support is required + ([96960993](rdar://96960993)). diff --git a/proposals/testing/0006-return-errors-from-expect-throws.md b/proposals/testing/0006-return-errors-from-expect-throws.md new file mode 100644 index 0000000000..eee9037cd3 --- /dev/null +++ b/proposals/testing/0006-return-errors-from-expect-throws.md @@ -0,0 +1,272 @@ +# Return errors from `#expect(throws:)` + +* Proposal: [ST-0006](0006-return-errors-from-expect-throws.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Implemented (Swift 6.1)** +* Bug: rdar://138235250 +* Implementation: [swiftlang/swift-testing#780](https://github.com/swiftlang/swift-testing/pull/780) +* Review: ([pitch](https://forums.swift.org/t/pitch-returning-errors-from-expect-throws/75567)) ([acceptance](https://forums.swift.org/t/pitch-returning-errors-from-expect-throws/75567/5)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0006](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0006-return-errors-from-expect-throws.md). + +## Introduction + +Swift Testing includes overloads of `#expect()` and `#require()` that can be +used to assert that some code throws an error. They are useful when validating +that your code's failure cases are correctly detected and handled. However, for +more complex validation cases, they aren't particularly ergonomic. This proposal +seeks to resolve that issue by having these overloads return thrown errors for +further inspection. + +## Motivation + +We offer three variants of `#expect(throws:)`: + +- One that takes an error type, and matches any error of the same type; +- One that takes an error _instance_ (conforming to `Equatable`) and matches any + error that compares equal to it; and +- One that takes a trailing closure and allows test authors to write arbitrary + validation logic. + +The third overload has proven to be somewhat problematic. First, it yields the +error to its closure as an instance of `any Error`, which typically forces the +developer to cast it before doing any useful comparisons. Second, the test +author must return `true` to indicate the error matched and `false` to indicate +it didn't, which can be both logically confusing and difficult to express +concisely: + +```swift +try #require { + let potato = try Sack.randomPotato() + try potato.turnIntoFrenchFries() +} throws: { error in + guard let error = error as PotatoError else { + return false + } + guard case .potatoNotPeeled = error else { + return false + } + return error.variety != .russet +} +``` + +The first impulse many test authors have here is to use `#expect()` in the +second closure, but it doesn't return the necessary boolean value _and_ it can +result in multiple issues being recorded in a test when there's really only one. + +## Proposed solution + +I propose deprecating [`#expect(_:sourceLocation:performing:throws:)`](https://developer.apple.com/documentation/testing/expect(_:sourcelocation:performing:throws:)) +and [`#require(_:sourceLocation:performing:throws:)`](https://developer.apple.com/documentation/testing/require(_:sourcelocation:performing:throws:)) +and modifying the other overloads so that, on success, they return the errors +that were thrown. + +## Detailed design + +All overloads of `#expect(throws:)` and `#require(throws:)` will be updated to +return an instance of the error type specified by their arguments, with the +problematic overloads returning `any Error` since more precise type information +is not statically available. The problematic overloads will also be deprecated: + +```diff +--- a/Sources/Testing/Expectations/Expectation+Macro.swift ++++ b/Sources/Testing/Expectations/Expectation+Macro.swift ++@discardableResult + @freestanding(expression) public macro expect( + throws errorType: E.Type, + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R +-) ++) -> E? where E: Error + ++@discardableResult + @freestanding(expression) public macro require( + throws errorType: E.Type, + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R +-) where E: Error ++) -> E where E: Error + ++@discardableResult + @freestanding(expression) public macro expect( + throws error: E, + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R +-) where E: Error & Equatable ++) -> E? where E: Error & Equatable + ++@discardableResult + @freestanding(expression) public macro require( + throws error: E, + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R +-) where E: Error & Equatable ++) -> E where E: Error & Equatable + ++@available(swift, deprecated: 100000.0, message: "Examine the result of '#expect(throws:)' instead.") ++@discardableResult + @freestanding(expression) public macro expect( + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R, + throws errorMatcher: (any Error) async throws -> Bool +-) ++) -> (any Error)? + ++@available(swift, deprecated: 100000.0, message: "Examine the result of '#require(throws:)' instead.") ++@discardableResult + @freestanding(expression) public macro require( + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R, + throws errorMatcher: (any Error) async throws -> Bool +-) ++) -> any Error +``` + +(More detailed information about the deprecations will be provided via DocC.) + +The `#expect(throws:)` overloads return an optional value that is `nil` if the +expectation failed, while the `#require(throws:)` overloads return non-optional +values and throw instances of `ExpectationFailedError` on failure (as before.) + +> [!NOTE] +> Instances of `ExpectationFailedError` thrown by `#require(throws:)` on failure +> are not returned as that would defeat the purpose of using `#require(throws:)` +> instead of `#expect(throws:)`. + +Test authors will be able to use the result of the above functions to verify +that the thrown error is correct: + +```swift +let error = try #require(throws: PotatoError.self) { + let potato = try Sack.randomPotato() + try potato.turnIntoFrenchFries() +} +#expect(error == .potatoNotPeeled) +#expect(error.variety != .russet) +``` + +The new code is more concise than the old code and avoids boilerplate casting +from `any Error`. + +## Source compatibility + +In most cases, this change does not affect source compatibility. Swift does not +allow forming references to macros at runtime, so we don't need to worry about +type mismatches assigning one to some local variable. + +We have identified two scenarios where a new warning will be emitted. + +### Inferred return type from macro invocation + +The return type of the macro may be used by the compiler to infer the return +type of an enclosing closure. If the return value is then discarded, the +compiler may emit a warning: + +```swift +func pokePotato(_ pPotato: UnsafePointer) throws { ... } + +let potato = Potato() +try await Task.sleep(for: .months(3)) +withUnsafePointer(to: potato) { pPotato in + // ^ ^ ^ ⚠️ Result of call to 'withUnsafePointer(to:_:)' is unused + #expect(throws: PotatoError.rotten) { + try pokePotato(pPotato) + } +} +``` + +This warning can be suppressed by assigning the result of the macro invocation +or the result of the function call to `_`: + +```swift +withUnsafePointer(to: potato) { pPotato in + _ = #expect(throws: PotatoError.rotten) { + try pokePotato(pPotato) + } +} +``` + +### Use of `#require(throws:)` in a generic context with `Never.self` + +If `#require(throws:)` (but not `#expect(throws:)`) is used in a generic context +where the type of thrown error is a generic parameter, and the type is resolved +to `Never`, there is no valid value for the invocation to return: + +```swift +func wrapper(throws type: E.Type, _ body: () throws -> Void) throws -> E { + return try #require(throws: type) { + try body() + } +} +let error = try #require(throws: Never.self) { ... } +``` + +We don't think this particular pattern is common (and outside of our own test +target, I'd be surprised if anybody's attempted it yet.) However, we do need to +handle it gracefully. If this pattern is encountered, Swift Testing will record +an "API Misused" issue for the current test and advise the test author to switch +to `#expect(throws:)` or to not pass `Never.self` here. + +## Integration with supporting tools + +N/A + +## Future directions + +- Adopting [typed throws](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0413-typed-throws.md) + to statically require that the error thrown from test code is of the correct + type. + + If we adopted typed throws in the signatures of these macros, it would force + adoption of typed throws in the code under test even when it may not be + appropriate. For example, if we adopted typed throws, the following code would + not compile: + + ```swift + func cook(_ food: consuming some Food) throws { ... } + + let error: PotatoError? = #expect(throws: PotatoError.self) { + var potato = Potato() + potato.fossilize() + try cook(potato) // 🛑 ERROR: Invalid conversion of thrown error type + // 'any Error' to 'PotatoError' + } + ``` + + We believe it may be possible to overload these macros or their expansions so + that the code sample above _does_ compile and behave as intended. We intend to + experiment further with this idea and potentially revisit typed throws support + in a future proposal. + +## Alternatives considered + +- Leaving the existing implementation and signatures in place. We've had + sufficient feedback about the ergonomics of this API that we want to address + the problem. + +- Having the return type of the macros be `any Error` and returning _any_ error + that was thrown even on mismatch. This would make the ergonomics of the + subsequent test code less optimal because the test author would need to cast + the error to the appropriate type before inspecting it. + + There's a philosophical argument to be made here that if a mismatched error is + thrown, then the test has already failed and is in an inconsistent state, so + we should allow the test to fail rather than return what amounts to "bad + output". + + If the test author wants to inspect any arbitrary thrown error, they can + specify `(any Error).self` instead of a concrete error type. + +## Acknowledgments + +Thanks to the team and to [@jakepetroules](https://github.com/jakepetroules) for +starting the discussion that ultimately led to this proposal. diff --git a/proposals/testing/0007-test-scoping-traits.md b/proposals/testing/0007-test-scoping-traits.md new file mode 100644 index 0000000000..619d1f48c6 --- /dev/null +++ b/proposals/testing/0007-test-scoping-traits.md @@ -0,0 +1,515 @@ +# Test Scoping Traits + +* Proposal: [ST-0007](0007-test-scoping-traits.md) +* Authors: [Stuart Montgomery](https://github.com/stmontgomery) +* Status: **Implemented (Swift 6.1)** +* Implementation: [swiftlang/swift-testing#733](https://github.com/swiftlang/swift-testing/pull/733), [swiftlang/swift-testing#86](https://github.com/swiftlang/swift-testing/pull/86) +* Review: ([pitch](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055)) ([review](https://forums.swift.org/t/proposal-test-scoping-traits/76676)) ([acceptance](https://forums.swift.org/t/proposal-test-scoping-traits/76676/3)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0007](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0007-test-scoping-traits.md). + +### Revision history + +* **v1**: Initial pitch. +* **v2**: Dropped 'Custom' prefix from the proposed API names (although kept the + word in certain documentation passages where it clarified behavior). +* **v3**: Changed the `Trait` requirement from a property to a method which + accepts the test and/or test case, and modify its default implementations such + that custom behavior is either performed per-suite or per-test case by default. +* **v4**: Renamed the APIs to use "scope" as the base verb instead of "execute". + +## Introduction + +This introduces API which enables a `Trait`-conforming type to provide a custom +execution scope for test functions and suites, including running code before or +after them. + +## Motivation + +One of the primary motivations for the trait system in Swift Testing, as +[described in the vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#trait-extensibility), +is to provide a way to customize the behavior of tests which have things in +common. If all the tests in a given suite type need the same custom behavior, +`init` and/or `deinit` (if applicable) can be used today. But if only _some_ of +the tests in a suite need custom behavior, or tests across different levels of +the suite hierarchy need it, traits would be a good place to encapsulate common +logic since they can be applied granularly per-test or per-suite. This aspect of +the vision for traits hasn't been realized yet, though: the `Trait` protocol +does not offer a way for a trait to customize the execution of the tests or +suites it's applied to. + +Customizing a test's behavior typically means running code either before or +after it runs, or both. Consolidating common set-up and tear-down logic allows +each test function to be more succinct with less repetitive boilerplate so it +can focus on what makes it unique. + +## Proposed solution + +At a high level, this proposal entails adding API to the `Trait` protocol +allowing a conforming type to opt-in to providing a custom execution scope for a +test. We discuss how that capability should be exposed to trait types below. + +### Supporting scoped access + +There are different approaches one could take to expose hooks for a trait to +customize test behavior. To illustrate one of them, consider the following +example of a `@Test` function with a custom trait whose purpose is to set mock +API credentials for the duration of each test it's applied to: + +```swift +@Test(.mockAPICredentials) +func example() { + // ... +} + +struct MockAPICredentialsTrait: TestTrait { ... } + +extension Trait where Self == MockAPICredentialsTrait { + static var mockAPICredentials: Self { ... } +} +``` + +In this hypothetical example, the current API credentials are stored via a +static property on an `APICredentials` type which is part of the module being +tested: + +```swift +struct APICredentials { + var apiKey: String + + static var shared: Self? +} +``` + +One way that this custom trait could customize the API credentials during each +test is if the `Trait` protocol were to expose a pair of method requirements +which were then called before and after the test, respectively: + +```swift +public protocol Trait: Sendable { + // ... + func setUp() async throws + func tearDown() async throws +} + +extension Trait { + // ... + public func setUp() async throws { /* No-op */ } + public func tearDown() async throws { /* No-op */ } +} +``` + +The custom trait type could adopt these using code such as the following: + +```swift +extension MockAPICredentialsTrait { + func setUp() { + APICredentials.shared = .init(apiKey: "...") + } + + func tearDown() { + APICredentials.shared = nil + } +} +``` + +Many testing systems use this pattern, including XCTest. However, this approach +encourages the use of global mutable state such as the `APICredentials.shared` +variable, and this limits the testing library's ability to parallelize test +execution, which is +[another part of the Swift Testing vision](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#parallelization-and-concurrency). + +The use of nonisolated static variables is generally discouraged now, and in +Swift 6 the above `APICredentials.shared` property produces an error. One way +to resolve that is to change it to a `@TaskLocal` variable, as this would be +concurrency-safe and still allow tests accessing this state to run in parallel: + +```swift +extension APICredentials { + @TaskLocal static var current: Self? +} +``` + +Binding task local values requires using the scoped access +[`TaskLocal.withValue()`](https://developer.apple.com/documentation/swift/tasklocal/withvalue(_:operation:isolation:file:line:)) +API though, and that would not be possible if `Trait` exposed separate methods +like `setUp()` and `tearDown()`. + +For these reasons, I believe it's important to expose this trait capability +using a single, scoped access-style API which accepts a closure. A simplified +version of that idea might look like this: + +```swift +public protocol Trait: Sendable { + // ... + + // Simplified example, not the actual proposal + func executeTest(_ body: @Sendable () async throws -> Void) async throws +} + +extension MockAPICredentialsTrait { + func executeTest(_ body: @Sendable () async throws -> Void) async throws { + let mockCredentials = APICredentials(apiKey: "...") + try await APICredentials.$current.withValue(mockCredentials) { + try await body() + } + } +} +``` + +### Avoiding unnecessarily lengthy backtraces + +A scoped access-style API has some potential downsides. To apply this approach +to a test function, the scoped call of a trait must wrap the invocation of that +test function, and every _other_ trait applied to that same test which offers +custom behavior _also_ must wrap the other traits' calls in a nesting fashion. +To visualize this, imagine a test function with multiple traits: + +```swift +@Test(.traitA, .traitB, .traitC) +func exampleTest() { + // ... +} +``` + +If all three of those traits provide a custom scope for tests, then each of them +needs to wrap the call to the next one, and the last trait needs to wrap the +invocation of the test, illustrated by the following: + +``` +TraitA.executeTest { + TraitB.executeTest { + TraitC.executeTest { + exampleTest() + } + } +} +``` + +Tests may have an arbitrary number of traits applied to them, including those +inherited from containing suite types. A naïve implementation in which _every_ +trait is given the opportunity to customize test behavior by calling its scoped +access API might cause unnecessarily lengthy backtraces that make debugging the +body of tests more difficult. Or worse: if the number of traits is great enough, +it could cause a stack overflow. + +In practice, most traits probably do _not_ need to provide a custom scope for +the tests they're applied to, so to mitigate these downsides it's important that +there be some way to distinguish traits which customize test behavior. That way, +the testing library can limit these scoped access calls to only traits which +need it. + +### Avoiding unnecessary (re-)execution + +Traits can be applied to either test functions or suites, and traits applied to +suites can optionally support inheritance by implementing the `isRecursive` +property of the `SuiteTrait` protocol. When a trait is directly applied to a +test function, if the trait customizes the behavior of tests it's applied to, it +should be given the opportunity to perform its custom behavior once for every +invocation of that test function. In particular, if the test function is +parameterized and runs multiple times, then the trait applied to it should +perform its custom behavior once for every invocation. This should not be +surprising to users, since it's consistent with the behavior of `init` and +`deinit` for an instance `@Test` method. + +It may be useful for certain kinds of traits to perform custom logic once for +_all_ the invocations of a parameterized test. Although this should be possible, +we believe it shouldn't be the default since it could lead to work being +repeated multiple times needlessly, or unintentional state sharing across tests, +unless the trait is implemented carefully to avoid those problems. + +When a trait conforms to `SuiteTrait` and is applied to a suite, the question of +when its custom scope (if any) should be applied is less obvious. Some suite +traits support inheritance and are recursively applied to all the test functions +they contain (including transitively, via sub-suites). Other suite traits don't +support inheritance, and only affect the specific suite they're applied to. +(It's also worth noting that a sub-suite _can_ have the same non-recursive suite +trait one of its ancestors has, as long as it's applied explicitly.) + +As a general rule of thumb, we believe most traits will either want to perform +custom logic once for _all_ children or once for _each_ child, not both. +Therefore, when it comes to suite traits, the default behavior should depend on +whether it supports inheritance: a recursive suite trait should by default +perform custom logic before each test, and a non-recursive one per-suite. But +the APIs should be flexible enough to support both, for advanced traits which +need it. + +## Detailed design + +I propose the following new APIs: + +- A new protocol `TestScoping` with a single required `provideScope(...)` method. + This will be called to provide scope for a test, and allows the conforming + type to perform custom logic before or after. +- A new method `scopeProvider(for:testCase:)` on the `Trait` protocol whose + result type is an `Optional` value of a type conforming to `TestScoping`. A + `nil` value returned by this method will skip calling the `provideScope(...)` + method. +- A default implementation of `Trait.scopeProvider(...)` which returns `nil`. +- A conditional implementation of `Trait.scopeProvider(...)` which returns `self` + in the common case where the trait type conforms to `TestScoping` itself. + +Since the `scopeProvider(...)` method's return type is optional and returns `nil` +by default, the testing library cannot invoke the `provideScope(...)` method +unless a trait customizes test behavior. This avoids the "unnecessarily lengthy +backtraces" problem above. + +Below are the proposed interfaces: + +```swift +/// A protocol that allows providing a custom execution scope for a test +/// function (and each of its cases) or a test suite by performing custom code +/// before or after it runs. +/// +/// Types conforming to this protocol may be used in conjunction with a +/// ``Trait``-conforming type by implementing the +/// ``Trait/scopeProvider(for:testCase:)-cjmg`` method, allowing custom traits +/// to provide custom scope for tests. Consolidating common set-up and tear-down +/// logic for tests which have similar needs allows each test function to be +/// more succinct with less repetitive boilerplate so it can focus on what makes +/// it unique. +public protocol TestScoping: Sendable { + /// Provide custom execution scope for a function call which is related to the + /// specified test and/or test case. + /// + /// - Parameters: + /// - test: The test under which `function` is being performed. + /// - testCase: The test case, if any, under which `function` is being + /// performed. When invoked on a suite, the value of this argument is + /// `nil`. + /// - function: The function to perform. If `test` represents a test suite, + /// this function encapsulates running all the tests in that suite. If + /// `test` represents a test function, this function is the body of that + /// test function (including all cases if it is parameterized.) + /// + /// - Throws: Whatever is thrown by `function`, or an error preventing this + /// type from providing a custom scope correctly. An error thrown from this + /// method is recorded as an issue associated with `test`. If an error is + /// thrown before `function` is called, the corresponding test will not run. + /// + /// When the testing library is preparing to run a test, it starts by finding + /// all traits applied to that test, including those inherited from containing + /// suites. It begins with inherited suite traits, sorting them + /// outermost-to-innermost, and if the test is a function, it then adds all + /// traits applied directly to that functions in the order they were applied + /// (left-to-right). It then asks each trait for its scope provider (if any) + /// by calling ``Trait/scopeProvider(for:testCase:)-cjmg``. Finally, it calls + /// this method on all non-`nil` scope providers, giving each an opportunity + /// to perform arbitrary work before or after invoking `function`. + /// + /// This method should either invoke `function` once before returning or throw + /// an error if it is unable to provide a custom scope. + /// + /// Issues recorded by this method are associated with `test`. + func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws +} + +public protocol Trait: Sendable { + // ... + + /// The type of the test scope provider for this trait. + /// + /// The default type is `Never`, which cannot be instantiated. The + /// ``scopeProvider(for:testCase:)-cjmg`` method for any trait with this + /// default type must return `nil`, meaning that trait will not provide a + /// custom scope for the tests it's applied to. + associatedtype TestScopeProvider: TestScoping = Never + + /// Get this trait's scope provider for the specified test and/or test case, + /// if any. + /// + /// - Parameters: + /// - test: The test for which a scope provider is being requested. + /// - testCase: The test case for which a scope provider is being requested, + /// if any. When `test` represents a suite, the value of this argument is + /// `nil`. + /// + /// - Returns: A value conforming to ``Trait/TestScopeProvider`` which may be + /// used to provide custom scoping for `test` and/or `testCase`, or `nil` if + /// they should not have any custom scope. + /// + /// If this trait's type conforms to ``TestScoping``, the default value + /// returned by this method depends on `test` and/or `testCase`: + /// + /// - If `test` represents a suite, this trait must conform to ``SuiteTrait``. + /// If the value of this suite trait's ``SuiteTrait/isRecursive`` property + /// is `true`, then this method returns `nil`; otherwise, it returns `self`. + /// This means that by default, a suite trait will _either_ provide its + /// custom scope once for the entire suite, or once per-test function it + /// contains. + /// - Otherwise `test` represents a test function. If `testCase` is `nil`, + /// this method returns `nil`; otherwise, it returns `self`. This means that + /// by default, a trait which is applied to or inherited by a test function + /// will provide its custom scope once for each of that function's cases. + /// + /// A trait may explicitly implement this method to further customize the + /// default behaviors above. For example, if a trait should provide custom + /// test scope both once per-suite and once per-test function in that suite, + /// it may implement the method and return a non-`nil` scope provider under + /// those conditions. + /// + /// A trait may also implement this method and return `nil` if it determines + /// that it does not need to provide a custom scope for a particular test at + /// runtime, even if the test has the trait applied. This can improve + /// performance and make diagnostics clearer by avoiding an unnecessary call + /// to ``TestScoping/provideScope(for:testCase:performing:)``. + /// + /// If this trait's type does not conform to ``TestScoping`` and its + /// associated ``Trait/TestScopeProvider`` type is the default `Never`, then + /// this method returns `nil` by default. This means that instances of this + /// trait will not provide a custom scope for tests to which they're applied. + func scopeProvider(for test: Test, testCase: Test.Case?) -> TestScopeProvider? +} + +extension Trait where Self: TestScoping { + // Returns `nil` if `testCase` is `nil`, else `self`. + public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self? +} + +extension SuiteTrait where Self: TestScoping { + // If `test` is a suite, returns `nil` if `isRecursive` is `true`, else `self`. + // Otherwise, `test` is a function and this returns `nil` if `testCase` is + // `nil`, else `self`. + public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self? +} + +extension Trait where TestScopeProvider == Never { + // Returns `nil`. + public func scopeProvider(for test: Test, testCase: Test.Case?) -> Never? +} + +extension Never: TestScoping {} +``` + +Here is a complete example of the usage scenario described earlier, showcasing +the proposed APIs: + +```swift +@Test(.mockAPICredentials) +func example() { + // ...validate API usage, referencing `APICredentials.current`... +} + +struct MockAPICredentialsTrait: TestTrait, TestScoping { + func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws { + let mockCredentials = APICredentials(apiKey: "...") + try await APICredentials.$current.withValue(mockCredentials) { + try await function() + } + } +} + +extension Trait where Self == MockAPICredentialsTrait { + static var mockAPICredentials: Self { + Self() + } +} +``` + +## Source compatibility + +The proposed APIs are purely additive. + +This proposal will replace the existing `CustomExecutionTrait` SPI, and after +further refactoring we anticipate it will obsolete the need for the +`SPIAwareTrait` SPI as well. + +## Integration with supporting tools + +Although some built-in traits are relevant to supporting tools (such as +SourceKit-LSP statically discovering `.tags` traits), custom test behaviors are +only relevant within the test executable process while tests are running. We +don't anticipate any particular need for this feature to integrate with +supporting tools. + +## Future directions + +### Access to suite type instances + +Some test authors have expressed interest in allowing custom traits to access +the instance of a suite type for `@Test` instance methods, so the trait could +inspect or mutate the instance. Currently, only instance-level members of a +suite type (including `init`, `deinit`, and the test function itself) can access +`self`, so this would grant traits applied to an instance test method access to +the instance as well. This is certainly interesting, but poses several technical +challenges that puts it out of scope of this proposal. + +### Convenience trait for setting task locals + +Some reviewers of this proposal pointed out that the hypothetical usage example +shown earlier involving setting a task local value while a test is executing +will likely become a common use of these APIs. To streamline that pattern, it +would be very natural to add a built-in trait type which facilitates this. I +have prototyped this idea and plan to add it once this new trait functionality +lands. + +## Alternatives considered + +### Separate set up & tear down methods on `Trait` + +This idea was discussed in [Supporting scoped access](#supporting-scoped-access) +above, and as mentioned there, the primary problem with this approach is that it +cannot be used with scoped access-style APIs, including (importantly) +`TaskLocal.withValue()`. For that reason, it prevents using that common Swift +concurrency technique and reduces the potential for test parallelization. + +### Add `provideScope(...)` directly to the `Trait` protocol + +The proposed `provideScope(...)` method could be added as a requirement of the +`Trait` protocol instead of being part of a separate `TestScoping` protocol, and +it could have a default implementation which directly invokes the passed-in +closure. But this approach would suffer from the lengthy backtrace problem +described above. + +### Extend the `Trait` protocol + +The original, experimental implementation of this feature included a protocol +named`CustomExecutionTrait` which extended `Trait` and had roughly the same +method requirement as the `TestScoping` protocol proposed above. This design +worked, provided scoped access, and avoided the lengthy backtrace problem. + +After evaluating the design and usage of this SPI though, it seemed unfortunate +to structure it as a sub-protocol of `Trait` because it means that the full +capabilities of the trait system are spread across multiple protocols. In the +proposed design, the ability to return a test scoping provider is exposed via +the main `Trait` protocol, and it relies on an associated type to conditionally +opt-in to custom test behavior. In other words, the proposed design expresses +custom test behavior as just a _capability_ that a trait may have, rather than a +distinct sub-type of trait. + +Also, the implementation of this approach within the testing library was not +ideal as it required a conditional `trait as? CustomExecutionTrait` downcast at +runtime, in contrast to the simpler and more performant Optional property of the +proposed API. + +### API names + +We first considered "execute" as the base verb for the proposed new concept, but +felt this wasn't appropriate since these trait types are not "the executor" of +tests, they merely customize behavior and provide scope(s) for tests to run +within. Also, the term "executor" has prior art in Swift Concurrency, and +although that word is used in other contexts too, it may be helpful to avoid +potential confusion with concurrency executors. + +We also considered "run" as the base verb for the proposed new concept instead +of "execute", which would imply the names `TestRunning`, `TestRunner`, +`runner(for:testCase)`, and `run(_:for:testCase:)`. The word "run" is used in +many other contexts related to testing though, such as the `Runner` SPI type and +more casually to refer to a run which occurred of a test, in the past tense, so +overloading this term again may cause confusion. + +## Acknowledgments + +Thanks to [Dennis Weissmann](https://github.com/dennisweissmann) for originally +implementing this as SPI, and for helping promote its usefulness. + +Thanks to [Jonathan Grynspan](https://github.com/grynspan) for exploring ideas +to refine the API, and considering alternatives to avoid unnecessarily long +backtraces. + +Thanks to [Brandon Williams](https://github.com/mbrandonw) for feedback on the +Forum pitch thread which ultimately led to the refinements described in the +"Avoiding unnecessary (re-)execution" section. diff --git a/proposals/testing/0008-exit-tests.md b/proposals/testing/0008-exit-tests.md new file mode 100644 index 0000000000..2a69716107 --- /dev/null +++ b/proposals/testing/0008-exit-tests.md @@ -0,0 +1,910 @@ +# Exit tests + +* Proposal: [ST-0008](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0008-exit-tests.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Maarten Engels](https://github.com/maartene) +* Status: **Implemented (Swift 6.2)** +* Bug: [apple/swift-testing#157](https://github.com/apple/swift-testing/issues/157) +* Implementation: [apple/swift-testing#324](https://github.com/swiftlang/swift-testing/pull/324) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/fdfc7867df4e35e29b2a24edee34ea4412ec15b0/proposals/testing/0008-exit-tests.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-exit-tests/78071)) ([review](https://forums.swift.org/t/st-0008-exit-tests/78692)) ([second review](https://forums.swift.org/t/second-review-st-0008-exit-tests/79198)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-st-0008-exit-tests/79553)) + +## Introduction + +One of the first enhancement requests we received for Swift Testing was the +ability to test for precondition failures and other critical failures that +terminate the current process when they occur. This feature is also frequently +requested for XCTest. With Swift Testing, we have the opportunity to build such +a feature in an ergonomic way. + +> [!NOTE] +> This feature has various names in the relevant literature, e.g. "exit tests", +> "death tests", "death assertions", "termination tests", etc. We consistently +> use the term "exit tests" to refer to them. + +## Motivation + +Imagine a function, implemented in a package, that includes a precondition: + +```swift +func eat(_ taco: consuming Taco) { + precondition(taco.isDelicious, "Tasty tacos only!") + ... +} +``` + +Today, a test author can write unit tests for this function, but there is no way +to make sure that the function rejects a taco whose `isDelicious` property is +`false` because a test that passes such a taco as input will crash (correctly!) +when it calls `precondition()`. + +An exit test allows testing this sort of functionality. The mechanism by which +an exit test is implemented varies between testing libraries and languages, but +a common implementation involves spawning a new process, performing the work +there, and checking that the spawned process ultimately terminates with a +particular (possibly platform-specific) exit status. + +Adding exit tests to Swift Testing would allow an entirely new class of tests +and would improve code coverage for existing test targets that adopt them. + +## Proposed solution + +This proposal introduces new overloads of the `#expect()` and `#require()` +macros that take, as an argument, a closure to be executed in a child process. +When called, these macros spawn a new process using the relevant +platform-specific interface (`posix_spawn()`, `CreateProcessW()`, etc.), call +the closure from within that process, and suspend the caller until that process +terminates. The exit status of the process is then compared against a known +value passed to the macro, allowing the test to pass or fail as appropriate. + +The function from earlier can then be tested using either of the new +overloads: + +```swift +await #expect(processExitsWith: .failure) { + var taco = Taco() + taco.isDelicious = false + eat(taco) // should trigger a precondition failure and process termination +} +``` + +## Detailed design + +### New expectations + +We will introduce the following new overloads of `#expect()` and `#require()` to +the testing library: + +```swift +/// Check that an expression causes the process to terminate in a given fashion. +/// +/// - Parameters: +/// - expectedExitCondition: The expected exit condition. +/// - observedValues: An array of key paths representing results from within +/// the exit test that should be observed and returned by this macro. The +/// ``ExitTest/Result/exitStatus`` property is always returned. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which recorded expectations and +/// issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// - Returns: If the exit test passed, an instance of ``ExitTest/Result`` +/// describing the state of the exit test when it exited. If the exit test +/// fails, the result is `nil`. +/// +/// Use this overload of `#expect()` when an expression will cause the current +/// process to terminate and the nature of that termination will determine if +/// the test passes or fails. For example, to test that calling `fatalError()` +/// causes a process to terminate: +/// +/// await #expect(processExitsWith: .failure) { +/// fatalError() +/// } +/// +/// - Note: A call to this expectation macro is called an "exit test." +/// +/// ## How exit tests are run +/// +/// When an exit test is performed at runtime, the testing library starts a new +/// process with the same executable as the current process. The current task is +/// then suspended (as with `await`) and waits for the child process to +/// terminate. `expression` is not called in the parent process. +/// +/// Meanwhile, in the child process, `expression` is called directly. To ensure +/// a clean environment for execution, it is not called within the context of +/// the original test. If `expression` does not terminate the child process, the +/// process is terminated automatically as if the main function of the child +/// process were allowed to return naturally. If an error is thrown from +/// `expression`, it is handed as if the error were thrown from `main()` and the +/// process is terminated. +/// +/// Once the child process terminates, the parent process resumes and compares +/// its exit status against `expectedExitCondition`. If they match, the exit +/// test has passed; otherwise, it has failed and an issue is recorded. +/// +/// ## Child process output +/// +/// By default, the child process is configured without a standard output or +/// standard error stream. If your test needs to review the content of either of +/// these streams, you can pass its key path in the `observedValues` argument: +/// +/// let result = await #expect( +/// processExitsWith: .failure, +/// observing: [\.standardOutputContent] +/// ) { +/// print("Goodbye, world!") +/// fatalError() +/// } +/// if let result { +/// #expect(result.standardOutputContent.contains(UInt8(ascii: "G"))) +/// } +/// +/// - Note: The content of the standard output and standard error streams may +/// contain any arbitrary sequence of bytes, including sequences that are not +/// valid UTF-8 and cannot be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). +/// These streams are globally accessible within the child process, and any +/// code running in an exit test may write to it including the operating +/// system and any third-party dependencies you have declared in your package. +/// +/// The actual exit condition of the child process is always reported by the +/// testing library even if you do not specify it in `observedValues`. +/// +/// ## Runtime constraints +/// +/// Exit tests cannot capture any state originating in the parent process or +/// from the enclosing lexical context. For example, the following exit test +/// will fail to compile because it captures an argument to the enclosing +/// parameterized test: +/// +/// @Test(arguments: 100 ..< 200) +/// func sellIceCreamCones(count: Int) async { +/// await #expect(processExitsWith: .failure) { +/// precondition( +/// count < 10, // ERROR: A C function pointer cannot be formed from a +/// // closure that captures context +/// "Too many ice cream cones" +/// ) +/// } +/// } +/// +/// An exit test cannot run within another exit test. +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +@discardableResult +@freestanding(expression) public macro expect( + processExitsWith expectedExitCondition: ExitTest.Condition, + observing observedValues: [any PartialKeyPath & Sendable] = [], + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @escaping @Sendable () async throws -> Void +) -> ExitTest.Result? = #externalMacro(module: "TestingMacros", type: "ExitTestExpectMacro") + +/// Check that an expression causes the process to terminate in a given fashion +/// and throw an error if it did not. +/// +/// [...] +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +@discardableResult +@freestanding(expression) public macro require( + processExitsWith expectedExitCondition: ExitTest.Condition, + observing observedValues: [any PartialKeyPath & Sendable] = [], + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @escaping @Sendable () async throws -> Void +) -> ExitTest.Result = #externalMacro(module: "TestingMacros", type: "ExitTestRequireMacro") +``` + +> [!NOTE] +> These interfaces are currently implemented and available on **macOS**, +> **Linux**, **FreeBSD**, **OpenBSD**, and **Windows**. If a platform does not +> support exit tests (generally because it does not support spawning or awaiting +> child processes), then we define `SWT_NO_EXIT_TESTS` when we build it. +> +> `SWT_NO_EXIT_TESTS` is not defined during test target builds and is presented +> here for illustrative purposes only. + +### Representing an exit test in Swift + +A new type, `ExitTest`, represents an exit test: + +```swift +/// A type describing an exit test. +/// +/// Instances of this type describe exit tests you create using the +/// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or +/// ``require(processExitsWith:observing:_:sourceLocation:performing:)`` macro. +/// You don't usually need to interact directly with an instance of this type. +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +public struct ExitTest: Sendable, ~Copyable { + /// The exit test that is running in the current process, if any. + /// + /// If the current process was created to run an exit test, the value of this + /// property describes that exit test. If this process is the parent process + /// of an exit test, or if no exit test is currently running, the value of + /// this property is `nil`. + /// + /// The value of this property is constant across all tasks in the current + /// process. + public static var current: ExitTest? { get } +} +``` + +### Exit conditions + +These macros take an argument of the new type `ExitTest.Condition`. This type +describes how the child process is expected to have exited: + +- With a specific exit code (as passed to the C standard function `exit()` or a + platform-specific equivalent); +- With a specific signal (on platforms that support signal handling[^winsig]); +- With any successful status; or +- With any failure status. + +[^winsig]: Windows nominally supports signal handling as it is part of the C + standard, but not to the degree that signals are supported by POSIX-like or + UNIX-derived operating systems. Swift Testing makes a "best effort" to emulate + signal-handling support on Windows. See [this](https://forums.swift.org/t/swift-on-windows-question-about-signals-and-exceptions/76640/2) + Swift forum message for more information. + +The type is declared as: + +```swift +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +extension ExitTest { + /// The possible conditions under which an exit test will complete. + /// + /// Values of this type are used to describe the conditions under which an + /// exit test is expected to pass or fail by passing them to + /// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(processExitsWith:observing:_:sourceLocation:performing:)``. + /// + /// ## Topics + /// + /// ### Successful exit conditions + /// + /// - ``success`` + /// + /// ### Failing exit conditions + /// + /// - ``failure`` + /// - ``exitCode(_:)`` + /// - ``signal(_:)`` + public struct Condition: Sendable, CustomStringConvertible { + /// A condition that matches when a process terminates successfully with exit + /// code `EXIT_SUCCESS`. + /// + /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), + /// `EXIT_SUCCESS` and `EXIT_FAILURE` as well as `0` (as a synonym for + /// `EXIT_SUCCESS`.) + public static var success: Self { get } + + /// A condition that matches when a process terminates abnormally with any + /// exit code other than `EXIT_SUCCESS` or with any signal. + public static var failure: Self { get } + + public init(_ exitStatus: ExitStatus) + + /// Creates a condition that matches when a process terminates with a given + /// exit code. + /// + /// - Parameters: + /// - exitCode: The exit code yielded by the process. + /// + /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), + /// `EXIT_SUCCESS` and `EXIT_FAILURE`. Platforms may additionally define their + /// own non-standard exit codes: + /// + /// | Platform | Header | + /// |-|-| + /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/_Exit.3.html), [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/sysexits.3.html) | + /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Exit-Status.html), `` | + /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?exit(3)), [``](https://man.freebsd.org/cgi/man.cgi?sysexits(3)) | + /// | OpenBSD | [``](https://man.openbsd.org/exit.3), [``](https://man.openbsd.org/sysexits.3) | + /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/exit-success-exit-failure) | + /// + /// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by + /// the process is yielded to the parent process. Linux and other POSIX-like + /// systems may only reliably report the low unsigned 8 bits (0–255) of + /// the exit code. + public static func exitCode(_ exitCode: CInt) -> Self + + /// Creates a condition that matches when a process terminates with a given + /// signal. + /// + /// - Parameters: + /// - signal: The signal that terminated the process. + /// + /// The C programming language defines a number of [standard signals](https://en.cppreference.com/w/c/program/SIG_types). + /// Platforms may additionally define their own non-standard signal codes: + /// + /// | Platform | Header | + /// |-|-| + /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/signal.3.html) | + /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Standard-Signals.html) | + /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?signal(3)) | + /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | + /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | + public static func signal(_ signal: CInt) -> Self + } +} +``` + +### Exit status + +The set of possible status codes reported by the child process are represented +by the `ExitStatus` enumeration: + +```swift +/// An enumeration describing possible status a process will yield on exit. +/// +/// You can convert an instance of this type to an instance of +/// ``ExitTest/Condition`` using ``ExitTest/Condition/init(_:)``. That value +/// can then be used to describe the condition under which an exit test is +/// expected to pass or fail by passing it to +/// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or +/// ``require(processExitsWith:observing:_:sourceLocation:performing:)``. +#if SWT_NO_PROCESS_SPAWNING +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +public enum ExitStatus: Sendable, Equatable, CustomStringConvertible { + /// The process terminated with the given exit code. + /// + /// [...] + case exitCode(_ exitCode: CInt) + + /// The process terminated with the given signal. + /// + /// [...] + case signal(_ signal: CInt) +} +``` + +### Exit test results + +These macros return an instance of the new type `ExitTest.Result`. This type +describes the results of the process including its reported exit condition and +the contents of its standard output and standard error streams, if requested. + +```swift +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +extension ExitTest { + /// A type representing the result of an exit test after it has exited and + /// returned control to the calling test function. + /// + /// Both ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` + /// and ``require(processExitsWith:observing:_:sourceLocation:performing:)`` + /// return instances of this type. + public struct Result: Sendable { + /// The status of the process hosting the exit test at the time it exits. + /// + /// When the exit test passes, the value of this property is equal to the + /// exit status reported by the process that hosted the exit test. + public var exitStatus: ExitStatus { get set } + + /// All bytes written to the standard output stream of the exit test before + /// it exited. + /// + /// The value of this property may contain any arbitrary sequence of bytes, + /// including sequences that are not valid UTF-8 and cannot be decoded by + /// [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). + /// Consider using [`String.init(validatingCString:)`](https://developer.apple.com/documentation/swift/string/init(validatingcstring:)-992vo) + /// instead. + /// + /// When checking the value of this property, keep in mind that the standard + /// output stream is globally accessible, and any code running in an exit + /// test may write to it including including the operating system and any + /// third-party dependencies you have declared in your package. Rather than + /// comparing the value of this property with [`==`](https://developer.apple.com/documentation/swift/array/==(_:_:)), + /// use [`contains(_:)`](https://developer.apple.com/documentation/swift/collection/contains(_:)) + /// to check if expected output is present. + /// + /// To enable gathering output from the standard output stream during an + /// exit test, pass `\.standardOutputContent` in the `observedValues` + /// argument of ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` + /// or ``require(processExitsWith:observing:_:sourceLocation:performing:)``. + /// + /// If you did not request standard output content when running an exit test, + /// the value of this property is the empty array. + public var standardOutputContent: [UInt8] { get set } + + /// All bytes written to the standard error stream of the exit test before + /// it exited. + /// + /// [...] + public var standardErrorContent: [UInt8] { get set } + } +} +``` + +### Usage + +These macros can be used within a test function: + +```swift +@Test func `We only eat delicious tacos`() async { + await #expect(processExitsWith: .failure) { + var taco = Taco() + taco.isDelicious = false + eat(taco) + } +} +``` + +Given the definition of `eat(_:)` above, this test can be expected to hit a +precondition failure and crash the process; because `.failure` was the specified +exit condition, this is treated as a successful test. + +It is often interesting to examine what is written to the standard output and +standard error streams by code running in an exit test. Callers can request that +either or both stream be captured and included in the result of the call to +`#expect(processExitsWith:)` or `#require(processExitsWith:)`. Capturing these +streams can be a memory-intensive operation, so the caller must explicitly opt +in: + +```swift +@Test func `We only eat delicious tacos`() async throws { + let result = try await #require( + processExitsWith: .failure, + observing: [\.standardErrorContent]) + ) { ... } + let stdout = result.standardOutputContent + #expect(stdout.contains("ERROR: This taco tastes terrible!".utf8)) +} +``` + +There are some constraints on valid exit tests: + +1. Because exit tests are run in child processes, they cannot capture any state + from the calling context. See the **Future directions** for further + discussion. +1. Exit tests cannot recursively invoke other exit tests; this is a constraint + that could potentially be lifted in the future, but it would be technically + complex to do so. + +If a Swift Testing issue such as an expectation failure occurs while running an +exit test, it is reported to the parent process and to the user as if it +happened locally. If an error is thrown from an exit test and not caught, it +behaves the same way a Swift program would if an error were thrown from its +`main()` function (that is, the program terminates abnormally.) + +## Source compatibility + +This is a new interface that is unlikely to collide with any existing +client-provided interfaces. The typical Swift disambiguation tools can be used +if needed. + +## Integration with supporting tools + +SPI is provided to allow testing environments other than Swift Package Manager +to detect and run exit tests: + +```swift +@_spi(ForToolsIntegrationOnly) +extension ExitTest { + /// A type whose instances uniquely identify instances of ``ExitTest``. + public struct ID: Sendable, Equatable, Codable { /* ... */ } + + /// A value that uniquely identifies this instance. + public var id: ID { get set } + + /// Key paths representing results from within this exit test that should be + /// observed and returned to the caller. + /// + /// The testing library sets this property to match what was passed by the + /// developer to the `#expect(processExitsWith:)` or `#require(processExitsWith:)` + /// macro. If you are implementing an exit test handler, you can check the + /// value of this property to determine what information you need to preserve + /// from your child process. + /// + /// The value of this property always includes ``ExitTest/Result/exitStatus`` + /// even if the test author does not specify it. + /// + /// Within a child process running an exit test, the value of this property is + /// otherwise unspecified. + public var observedValues: [any PartialKeyPath & Sendable] { get set } + + /// Call the exit test in the current process. + /// + /// This function invokes the closure originally passed to + /// `#expect(processExitsWith:)` _in the current process_. That closure is + /// expected to terminate the process; if it does not, the testing library + /// will terminate the process as if its `main()` function returned naturally. + public consuming func callAsFunction() async -> Never + + /// Find the exit test function at the given source location. + /// + /// - Parameters: + /// - id: The unique identifier of the exit test to find. + /// + /// - Returns: The specified exit test function, or `nil` if no such exit test + /// could be found. + public static func find(identifiedBy id: ExitTest.ID) -> Self? + + /// A handler that is invoked when an exit test starts. + /// + /// - Parameters: + /// - exitTest: The exit test that is starting. + /// + /// - Returns: The result of the exit test including the condition under which + /// it exited. + /// + /// - Throws: Any error that prevents the normal invocation or execution of + /// the exit test. + /// + /// This handler is invoked when an exit test (i.e. a call to either + /// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(processExitsWith:observing:_:sourceLocation:performing:)``) is + /// started. The handler is responsible for initializing a new child + /// environment (typically a child process) and running the exit test + /// identified by `sourceLocation` there. + /// + /// In the child environment, you can find the exit test again by calling + /// ``ExitTest/find(at:)`` and can run it by calling + /// ``ExitTest/callAsFunction()``. + /// + /// The parent environment should suspend until the results of the exit test + /// are available or the child environment is otherwise terminated. The parent + /// environment is then responsible for interpreting those results and + /// recording any issues that occur. + public typealias Handler = @Sendable (_ exitTest: borrowing ExitTest) async throws -> ExitTest.Result +} + +@_spi(ForToolsIntegrationOnly) +extension Configuration { + /// A handler that is invoked when an exit test starts. + /// + /// For an explanation of how this property is used, see ``ExitTest/Handler``. + /// + /// When using the `swift test` command from Swift Package Manager, this + /// property is pre-configured. Otherwise, the default value of this property + /// records an issue indicating that it has not been configured. + public var exitTestHandler: ExitTest.Handler { get set } +} +``` + +Any tools that use `swift build --build-tests`, `swift test`, or equivalent to +compile executables for testing will inherit the functionality provided for +`swift test` and do not need to implement their own exit test handlers. Tools +that directly compile test targets or otherwise do not leverage Swift Package +Manager will need to provide an implementation. + +## Future directions + +### Support for iOS, WASI, etc. + +The need for exit tests on other platforms is just as strong as it is on the +supported platforms (macOS, Linux, FreeBSD/OpenBSD, and Windows). These +platforms do not support spawning new processes, so a different mechanism for +running exit tests would be needed. + +Android _does_ have `posix_spawn()` and related API and may be able to use the +same implementation as Linux. Android support is an ongoing area of research for +Swift Testing's core team. + +> [!NOTE] +> In the event we can add support for exit tests on a new platform _without_ any +> changes to the feature's public interface, the Testing Workgroup has agreed +> that an additional Swift Evolution proposal will not be necessary. + +### Recursive exit tests + +The technical constraints preventing recursive exit test invocation can be +resolved if there is a need to do so. However, we don't anticipate that this +constraint will be a serious issue for developers. + +### Support for passing state + +Arbitrary state is necessarily not preserved between the parent and child +processes, but there is little to prevent us from adding a variadic `arguments:` +argument and passing values whose types conform to `Codable`. + +The blocker right now is that there is no type information during macro +expansion, meaning that the testing library can emit the glue code to _encode_ +arguments, but does not know what types to use when _decoding_ those arguments. +If generic types were made available during macro expansion via the macro +expansion context, then it would be possible to synthesize the correct logic. + +Alternatively, if the language gained something akin to C++'s `decltype()`, we +could leverage closures' capture list syntax. Subjectively, capture lists ought +to be somewhat intuitive for developers in this context: + +```swift +let (lettuce, cheese) = taco.addToppings() +await #expect(processExitsWith: .failure) { [taco, plant = lettuce, cheese] in + try taco.removeToppings(plant, cheese) +} +``` + +### More nuanced support for throwing errors from exit test bodies + +Currently, if an error is thrown from an exit test without being caught, the +test behaves the same way a program does when an error is thrown from an +explicit or implicit `main() throws` function: the process terminates abnormally +and control returns to the test function that is awaiting the exit test: + +```swift +await #expect(processExitsWith: .failure) { + throw TacoError.noTacosFound +} +``` + +If the test function is expecting `.failure`, this means the test passes. +Although this behavior is consistent with modelling an exit test as an +independent program (i.e. the exit test acts like its own `main()` function), it +may be surprising to test authors who aren't thinking about error handling. In +the future, we may want to offer a compile-time diagnostic if an error is thrown +from an exit test body without being caught, or offer a distinct exit condition +(i.e. `.errorNotCaught(_ error: Error & Codable)`) for these uncaught errors. +For error types that conform to `Codable`, we could offer rethrowing behavior, +but this is not possible for error types that cannot be sent across process +boundaries. + +### Exit-testing customized processes + +The current model of exit tests is that they run in approximately the same +environment as the test process by spawning a copy of the executable under test. +There is a very real use case for allowing testing other processes and +inspecting their output. In the future, we could provide API to spawn a process +with particular arguments and environment variables, then inspect its exit +condition and standard output/error streams: + +```swift +let result = try await #require( + executableAt: "/usr/bin/swift", + passing: ["build", "--package-path", ...], + environment: [:], + exitsWith: .success +) +#expect(result.standardOutputContent.contains("Build went well!").utf8) +``` + +We could also investigate explicitly integrating with [`Foundation.Process`](https://developer.apple.com/documentation/foundation/process) +or the proposed [`Foundation.Subprocess`](https://github.com/swiftlang/swift-foundation/blob/main/Proposals/0007-swift-subprocess.md) +as an alternative: + +```swift +let process = Process() +process.executableURL = URL(filePath: "/usr/bin/swift", directoryHint: .notDirectory) +process.arguments = ["build", "--package-path", ...] +let result = try await #require(process, exitsWith: .success) +#expect(result.standardOutputContent.contains("Build went well!").utf8) +``` + +### Conformance of ExitStatus to ExpressibleByIntegerLiteral + +A contributor on the Swift forums suggested having `ExitStatus` conform to +[`ExpressibleByIntegerLiteral`](https://developer.apple.com/documentation/swift/expressiblebyintegerliteral) +and interpreting an integer literal as an exit code, such that a test author +could write: + +```swift +await #expect(processExitsWith: EX_CANTCREAT) { + ... +} +``` + +This would be convenient for test authors who are dealing with a variety of exit +codes, but is beyond the scope of this proposal. Adding conformance to this +protocol also requires some care to ensure that signal constants such as +`SIGABRT` cannot be accidentally interpreted as exit codes. + +## Alternatives considered + +- Doing nothing. + +- Marking exit tests using a trait rather than a new `#expect()` overload: + + ```swift + @Test(.exits(with: .failure)) + func `We only eat delicious tacos`() { + var taco = Taco() + taco.isDelicious = false + eat(taco) + } + ``` + + This syntax would require separate test functions for each exit test, while + reusing the same function for relatively concise tests may be preferable. + + It would also potentially conflict with parameterized tests, as it is not + possible to pass arbitrary parameters to the child process. It would be + necessary to teach the testing library's macro target about the + `.exits(with:)` trait so that it could produce a diagnostic when used with a + parameterized test function. + +- Inferring exit tests from test functions that return `Never`: + + ```swift + @Test func `No seafood for me, thanks!`() -> Never { + var taco = Taco() + taco.toppings.append(.shrimp) + eat(taco) + fatalError("Should not have eaten that!") + } + ``` + + There's a certain synergy in inferring that a test function that returns + `Never` must necessarily be a crasher and should be handled out of process. + However, this forces the test author to add a call to `fatalError()` or + similar in the event that the code under test does _not_ terminate, and there + is no obvious way to express that a specific exit code, signal, or other + condition is expected (as opposed to just "it exited".) + + We might want to support that sort of inference in the future (i.e. "don't run + this test in-process because it will terminate the test run"), but without + also inferring success or failure from the process' exit status. + +- Naming the macro something else such as: + + - `#exits(with:_:)`; + - `#exits(because:_:)`; + - `#expect(exitsBecause:_:)`; + - `#expect(terminatesBecause:_:)`; etc. + + While "with" is normally avoided in symbol names in Swift, it sometimes really + is the best preposition for the job. "Because", "due to", and others don't + sound "right" when the entire expression is read out loud. For example, you + probably wouldn't say "exits due to success" in English. + + A contributor in the Swift forums suggested `#expect(crashes:)`: + + ```swift + await #expect(crashes: { + ... + }) + ``` + + This would preclude the possibility of writing an exit test that is expected + to exit successfully—a scenario for which we have real-world use cases. It was + also not clear that the word "crash" applied to every failing exit status. For + example, a process that exits with the POSIX-defined exit code `EX_TEMPFAIL` + likely has not _crashed_; it has just reported that the requested operation + has failed. + + This signature would also be subject to label elision when used with trailing + closure syntax, resulting in: + + ```swift + await #expect { + ... + } + ``` + + The lack of any distinguishing label here would unacceptably impact the test's + readability as it gives no indication that the code is running out-of-process + or is expected to terminate its process. + +- Combining `ExitStatus` and `ExitTest.Condition` into a single type: + + ```swift + enum ExitCondition { + case failure // any failure + case exitCode(CInt) + case signal(CInt) + } + ``` + + This simplified the set of types used for exit tests, but made comparing two + exit conditions complicated and necessitated a `==` operator that did not + satisfy the requirements of the `Equatable` protocol. + +- Naming `ExitStatus` something else such as: + + - `StatusAtExit`, which might avoid some confusion with exit _codes_ but which + is not idiomatic Swift; + - `ProcessStatus`, but we don't say "process" in our API surface elsewhere; + - `Status`, which is too generic, + - `ExitReason`, but "status" is a more widely-used term of art for this + concept; or + - `TerminationStatus` (which Foundation uses to represent approximately the + same concept), but we don't use "termination" in Swift Testing's API + anywhere. + + In particular, there was some interest in using "termination" instead of + "exit" for consistency with Foundation. Foundation and the upcoming + `Subprocess` package use both terms interchangeably, so there is precedent for + either. "Exit" is more concise; "terminate" may be read to imply that the + process was _forced_ to stop running. + +- Naming `ExitStatus.exitCode(_:)` just `.code(_:)`. Some contributors on the + forums felt that the use of "exit" here was redundant given the proposed + `exitsWith:` and `processExitsWith:` labels. However, "code" is potentially + ambiguous: does it refer to an exit code, a signal code, the code the test + author is writing, etc.? + + We certainly don't want the exit test interface to be redundant. However, + given that: + + - We _expect_ (no pun intended) most uses of exit tests will check for + `.failure` rather than a specific exit code; + - "Exit code" is an established term of art; and + - `.exitCode(_:)` may appear in other contexts (not just as an argument to + `#expect(processExitsWith:)`) + + We have opted to keep the full case name. + +- Using parameter packs to specify observed values and return types: + + ```swift + @freestanding(expression) public macro require( + processExitsWith expectedExitCondition: ExitTest.Condition, + observing observedValues: (repeat (KeyPath)) = (), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @escaping @Sendable () async throws -> Void + ) -> (repeat each T) + ``` + + Using a parameter pack in this way would make it impossible to access + properties of the returned `ExitTest.Result` value that weren't observed, and + in general would mean developers wouldn't even need to use `ExitTest.Result`: + + ```swift + let (status, stderr) = try await #expect( + processExitsWith: .failure, + observing: (\.exitStatus, \.standardErrorContent) + ) { ... } + #expect(status == ...) + #expect(stderr.contains(...)) + ``` + + Unfortunately, the `#expect(processExitsWith:)` and `#require(processExitsWith:)` + macros do not have enough information at compile time to correctly infer the + types of the key paths passed as `observedValues` above, so we end up with + rather obscure errors: + + > 🛑 Cannot convert value of type 'KeyPath<_, _>' to expected argument type + > 'KeyPath' + + If, in the future, this error is resolved, we may wish to revisit this option, + so it can also be considered a "future direction" for the feature. + +- Changing the implementation of `precondition()`, `fatalError()`, etc. in the + standard library so that they do not terminate the current process while + testing, thus removing the need to spawn a child process for an exit test. + + Most of the functions in this family return `Never`, and changing their return + types would be ABI-breaking (as well as a pessimization in production code.) + Even if we did modify these functions in the Swift standard library, other + ways to terminate the process exist and would not be covered: + + - Calling the C standard function `exit()`; + - Throwing an uncaught Objective-C or C++ exception; + - Sending a signal to the process; or + - Misusing memory (e.g. trying to dereference a null pointer.) + + Modifying the C or C++ standard library, or modifying the Objective-C runtime, + would be well beyond the scope of this proposal. + +- Skipping test functions containing exit tests on platforms that do not support + exit tests. + + This would avoid the need to write `if os(...)`, `@available(...)`, or + `if #available(...)` in a cross-platform test function before using exit + tests. Swift Testing does not currently support skipping a test that has + already started executing, and the implementation of such a feature is beyond + the scope of this proposal. + + Even if the library supported this sort of action, it would likely be + surprising to test authors that they could write a test that compiles for e.g. + iOS but doesn't run and doesn't report any problems. + + Further, in general this is not a pattern that is used in the Swift ecosystem + for platform-specific functionality; instead, `#if os(...)` and availability + checks are the normal way to mark code as platform-specific. + +## Acknowledgments + +Many thanks to the XCTest and Swift Testing team. Thanks to @compnerd for his +help with the Windows implementation. Thanks to my colleagues Coops, +Danny N., David R., Drew Y., and Robert K. at Apple for +their help with the nuances of crash reporting on macOS. diff --git a/proposals/testing/0009-attachments.md b/proposals/testing/0009-attachments.md new file mode 100644 index 0000000000..e9e23102ee --- /dev/null +++ b/proposals/testing/0009-attachments.md @@ -0,0 +1,475 @@ +# Attachments + +* Proposal: [ST-0009](0009-attachments.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Rachel Brindle](https://github.com/younata) +* Status: **Implemented (Swift 6.2)** +* Bug: [swiftlang/swift-testing#714](https://github.com/swiftlang/swift-testing/issues/714) +* Implementation: [swiftlang/swift-testing#973](https://github.com/swiftlang/swift-testing/pull/973) +* Review: ([pitch](https://forums.swift.org/t/pitch-attachments/78072)) ([review](https://forums.swift.org/t/st-0009-attachments/78698)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-st-0009-attachments/79193)) + +## Introduction + +Test authors frequently need to include out-of-band data with tests that can be +used to diagnose issues when a test fails. This proposal introduces a new API +called "attachments" (analogous to the same-named feature in XCTest) as well as +the infrastructure necessary to create new attachments and handle them in tools +like VS Code. + +## Motivation + +When a test fails, especially in a remote environment like CI, it can often be +difficult to determine what exactly has gone wrong. Data that was produced +during the test can be useful, but there is currently no mechanism in Swift +Testing to output arbitrary data other than via `stdout`/`stderr` or via an +artificially-generated issue. A dedicated interface for attaching arbitrary +information to a test would allow test authors to gather relevant information +from a test in a structured way. + +## Proposed solution + +We propose introducing a new type to Swift Testing, `Attachment`, that represents +some arbitrary "attachment" to associate with a test. Along with `Attachment`, +we will introduce a new protocol, `Attachable`, to which types can conform to +indicate they can be attached to a test. + +Default conformances to `Attachable` will be provided for standard library types +that can reasonably be attached. We will also introduce a [cross-import overlay](https://forums.swift.org/t/cross-import-overlays/36710) +with Foundation—that is, a tertiary module that is automatically imported when +a test target imports both Foundation _and_ Swift Testing—that includes +additional conformances for Foundation types such as `Data` and `URL` and +provides support for attaching values that also conform to `Encodable` or +`NSSecureCoding`. + +## Detailed design + +The `Attachment` type is defined as follows: + +```swift +/// A type describing values that can be attached to the output of a test run +/// and inspected later by the user. +/// +/// Attachments are included in test reports in Xcode or written to disk when +/// tests are run at the command line. To create an attachment, you need a value +/// of some type that conforms to ``Attachable``. Initialize an instance of +/// ``Attachment`` with that value and, optionally, a preferred filename to use +/// when writing to disk. +public struct Attachment: ~Copyable where AttachableValue: Attachable & ~Copyable { + /// A filename to use when writing this attachment to a test report or to a + /// file on disk. + /// + /// The value of this property is used as a hint to the testing library. The + /// testing library may substitute a different filename as needed. If the + /// value of this property has not been explicitly set, the testing library + /// will attempt to generate its own value. + public var preferredName: String { get } + + /// The value of this attachment. + public var attachableValue: AttachableValue { get } + + /// Initialize an instance of this type that encloses the given attachable + /// value. + /// + /// - Parameters: + /// - attachableValue: The value that will be attached to the output of the + /// test run. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the testing library attempts to + /// derive a reasonable filename for the attached value. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + public init( + _ attachableValue: consuming AttachableValue, + named preferredName: String? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) + + /// Attach an attachment to the current test. + /// + /// - Parameters: + /// - attachment: The attachment to attach. + /// - sourceLocation: The source location of the call to this function. + /// + /// When attaching a value of a type that does not conform to both + /// [`Sendable`](https://developer.apple.com/documentation/swift/sendable) and + /// [`Copyable`](https://developer.apple.com/documentation/swift/copyable), + /// the testing library encodes it as data immediately. If the value cannot be + /// encoded and an error is thrown, that error is recorded as an issue in the + /// current test and the attachment is not written to the test report or to + /// disk. + /// + /// An attachment can only be attached once. + public static func record(_ attachment: consuming Self, sourceLocation: SourceLocation = #_sourceLocation) + + /// Attach a value to the current test. + /// + /// - Parameters: + /// - attachableValue: The value to attach. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the testing library attempts to + /// derive a reasonable filename for the attached value. + /// - sourceLocation: The source location of the call to this function. + /// + /// When attaching a value of a type that does not conform to both + /// [`Sendable`](https://developer.apple.com/documentation/swift/sendable) and + /// [`Copyable`](https://developer.apple.com/documentation/swift/copyable), + /// the testing library encodes it as data immediately. If the value cannot be + /// encoded and an error is thrown, that error is recorded as an issue in the + /// current test and the attachment is not written to the test report or to + /// disk. + /// + /// This function creates a new instance of ``Attachment`` and immediately + /// attaches it to the current test. + /// + /// An attachment can only be attached once. + public static func record(_ attachableValue: consuming AttachableValue, named preferredName: String? = nil, sourceLocation: SourceLocation = #_sourceLocation) + + /// Call a function and pass a buffer representing the value of this + /// instance's ``attachableValue-2tnj5`` property to it. + /// + /// - Parameters: + /// - body: A function to call. A temporary buffer containing a data + /// representation of this instance is passed to it. + /// + /// - Returns: Whatever is returned by `body`. + /// + /// - Throws: Whatever is thrown by `body`, or any error that prevented the + /// creation of the buffer. + /// + /// The testing library uses this function when writing an attachment to a + /// test report or to a file on disk. This function calls the + /// ``Attachable/withUnsafeBytes(for:_:)`` function on this attachment's + /// ``attachableValue-2tnj5`` property. + @inlinable public borrowing func withUnsafeBytes( + _ body: (UnsafeRawBufferPointer) throws -> R + ) throws -> R +} + +extension Attachment: Copyable where AttachableValue: Copyable {} +extension Attachment: Sendable where AttachableValue: Sendable {} +``` + +With `Attachment` comes `Attachable`, a protocol to which "attachable values" +conform: + +```swift +/// A protocol describing a type that can be attached to a test report or +/// written to disk when a test is run. +/// +/// To attach an attachable value to a test, pass it to ``Attachment/record(_:named:sourceLocation:)``. +/// To further configure an attachable value before you attach it, use it to +/// initialize an instance of ``Attachment`` and set its properties before +/// passing it to ``Attachment/record(_:sourceLocation:)``. An attachable +/// value can only be attached to a test once. +/// +/// The testing library provides default conformances to this protocol for a +/// variety of standard library types. Most user-defined types do not need to +/// conform to this protocol. +/// +/// A type should conform to this protocol if it can be represented as a +/// sequence of bytes that would be diagnostically useful if a test fails. If a +/// type cannot conform directly to this protocol (such as a non-final class or +/// a type declared in a third-party module), you can create a wrapper type +/// that conforms to ``AttachableWrapper`` to act as a proxy. +public protocol Attachable: ~Copyable { + /// An estimate of the number of bytes of memory needed to store this value as + /// an attachment. + /// + /// The testing library uses this property to determine if an attachment + /// should be held in memory or should be immediately persisted to storage. + /// Larger attachments are more likely to be persisted, but the algorithm the + /// testing library uses is an implementation detail and is subject to change. + /// + /// The value of this property is approximately equal to the number of bytes + /// that will actually be needed, or `nil` if the value cannot be computed + /// efficiently. The default implementation of this property returns `nil`. + /// + /// - Complexity: O(1) unless `Self` conforms to `Collection`, in which case + /// up to O(_n_) where _n_ is the length of the collection. + var estimatedAttachmentByteCount: Int? { get } + + /// Call a function and pass a buffer representing this instance to it. + /// + /// - Parameters: + /// - attachment: The attachment that is requesting a buffer (that is, the + /// attachment containing this instance.) + /// - body: A function to call. A temporary buffer containing a data + /// representation of this instance is passed to it. + /// + /// - Returns: Whatever is returned by `body`. + /// + /// - Throws: Whatever is thrown by `body`, or any error that prevented the + /// creation of the buffer. + /// + /// The testing library uses this function when writing an attachment to a + /// test report or to a file on disk. The format of the buffer is + /// implementation-defined, but should be "idiomatic" for this type: for + /// example, if this type represents an image, it would be appropriate for + /// the buffer to contain an image in PNG format, JPEG format, etc., but it + /// would not be idiomatic for the buffer to contain a textual description of + /// the image. + borrowing func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R + + /// Generate a preferred name for the given attachment. + /// + /// - Parameters: + /// - attachment: The attachment that needs to be named. + /// - suggestedName: A suggested name to use as the basis of the preferred + /// name. This string was provided by the developer when they initialized + /// `attachment`. + /// + /// - Returns: The preferred name for `attachment`. + /// + /// The testing library uses this function to determine the best name to use + /// when adding `attachment` to a test report or persisting it to storage. The + /// default implementation of this function returns `suggestedName` without + /// any changes. + borrowing func preferredName(for attachment: borrowing Attachment, basedOn suggestedName: String) -> String +} +``` + +Default conformances to `Attachable` are provided for: + +- `Array`, `ContiguousArray`, and `ArraySlice` +- `String` and `Substring` +- `Data` (if Foundation is also imported) + +Default _implementations_ are provided for types when they conform to +`Attachable` and either `Encodable` or `NSSecureCoding` (or both.) To use these +conformances, Foundation must be imported because `JSONEncoder` and +`PropertyListEncoder` are members of Foundation, not the Swift standard library. + +Some types cannot conform directly to `Attachable` because they require +additional information to encode correctly, or because they are not directly +`Sendable` or `Copyable`. A second protocol, `AttachableWrapper`, is provided +that refines `Attachable`: + +```swift +/// A protocol describing a type that can be attached to a test report or +/// written to disk when a test is run and which wraps another value that it +/// stands in for. +/// +/// To attach an attachable value to a test, pass it to ``Attachment/record(_:named:sourceLocation:)``. +/// To further configure an attachable value before you attach it, use it to +/// initialize an instance of ``Attachment`` and set its properties before +/// passing it to ``Attachment/record(_:sourceLocation:)``. An attachable +/// value can only be attached to a test once. +/// +/// A type can conform to this protocol if it represents another type that +/// cannot directly conform to ``Attachable``, such as a non-final class or a +/// type declared in a third-party module. +public protocol AttachableWrapper: Attachable, ~Copyable { + /// The type of the underlying value represented by this type. + associatedtype Wrapped + + /// The underlying value represented by this instance. + var wrappedValue: Wrapped { get } +} + +extension Attachment where AttachableValue: AttachableWrapper & ~Copyable { + /// The value of this attachment. + /// + /// When the attachable value's type conforms to ``AttachableWrapper``, the + /// value of this property equals the wrappers's underlying attachable value. + /// To access the attachable value as an instance of `T` (where `T` conforms + /// to ``AttachableWrapper``), specify the type explicitly: + /// + /// ```swift + /// let attachableValue = attachment.attachableValue as T + /// ``` + public var attachableValue: AttachableValue.Wrapped { get } +} +``` + +The cross-import overlay with Foundation also provides the following convenience +interface for attaching the contents of a file or directory on disk: + +```swift +extension Attachment where AttachableValue == _AttachableURLWrapper { + /// Initialize an instance of this type with the contents of the given URL. + /// + /// - Parameters: + /// - url: The URL containing the attachment's data. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the name of the attachment is + /// derived from the last path component of `url`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// - Throws: Any error that occurs attempting to read from `url`. + public init( + contentsOf url: URL, + named preferredName: String? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) async throws +} +``` + +`_AttachableURLWrapper` is a type that conforms to `AttachableWrapper` and +encloses the URL and corresponding mapped data. As an implementation detail, it +is omitted from this proposal for brevity. + +## Source compatibility + +This proposal is additive and has no impact on existing code. + +## Integration with supporting tools + +We will add a new command-line argument to the `swift test` command in Swift +Package Manager: + +```sh +--attachments-path Path where attachments should be saved. +``` + +If specified, an attachment will be written to that path when the attachment is +passed to one of the `Attachment.attach(_:sourceLocation:)` methods. If not +specified, attachments are not saved to disk. Tools that indirectly use Swift +Testing through `swift test` can specify a path (e.g. to a directory created +inside the system's temporary directory), then move or delete the created files +as needed. + +The JSON event stream ABI will be amended correspondingly: + +```diff +--- a/Documentation/ABI/JSON.md ++++ b/Documentation/ABI/JSON.md + ::= { + "kind": , + "instant": , ; when the event occurred + ["issue": ,] ; the recorded issue (if "kind" is "issueRecorded") ++ ["attachment": ,] ; the attachment (if kind is "valueAttached") + "messages": , + ["testID": ,] + } + + ::= "runStarted" | "testStarted" | "testCaseStarted" | + "issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" | +- "runEnded" ; additional event kinds may be added in the future ++ "runEnded" | "valueAttached"; additional event kinds may be added in the future + ++ ::= { ++ "path": , ; the absolute path to the attachment on disk ++} +``` + +As these changes are additive only, the JSON schema version does not need to be +incremented to support them. We are separately planning to increment the JSON +schema version to support other features; these changes will apply to the newer +version too. + +## Future directions + +- Attachment lifetime management: XCTest's attachments allow for specifying a + "lifetime", with two lifetimes currently available: + + ```objc + typedef NS_ENUM(NSInteger, XCTAttachmentLifetime) { + XCTAttachmentLifetimeKeepAlways = 0, + XCTAttachmentLifetimeDeleteOnSuccess = 1 + }; + ``` + + If a test passes, it is probably not necessary to keep its attachments saved + to disk. The exact "shape" this feature should take in Swift Testing is not + yet clear. + +- Image attachments: it is often useful to be able to attach images to tests, + however there is no cross-platform solution for this functionality. An + experimental implementation that allows attaching an instance of `CGImage` (on + Apple platforms) is available in Swift Testing's repository and shows what it + might look like for us to provide this functionality. + +- Additional conformances for types in other modules: in order to keep Swift + Testing's dependency graph as small as possible, we cannot link it to + arbitrary packages such as (for example) swift-collections even if it would be + useful to do so. That means we can't directly provide conformances to + `Attachable` for types in those modules. Adding additional cross-import + overlays would allow us to provide those conformances when both Swift Testing + and those packages are imported at the same time. + + This functionality may require changes in Swift Package Manager that are + beyond the scope of this proposal. + +- Adopting `RawSpan` instead of `UnsafeRawBufferPointer`: `RawSpan` represents a + safer alternative to `UnsafeRawBufferPointer`, but it is not yet available + everywhere we'd need it in the standard library, and our minimum deployment + targets on Apple's platforms do not allow us to require the use of `RawSpan` + (as no shipping version of Apple's platforms includes it.) + +- Adding an associated `Metadata` type to `Attachable` allowing for inclusion of + arbitrary out-of-band data to attachments: we see several uses for such a + feature: + + - Fine-grained control of the serialization format used for `Encodable` types; + - Metrics (scaling factor, rotation, etc.) for images; and + - Compression algorithms to use for attached files and directories. + + The exact shape of this interface needs further consideration, but it could be + added in the future without disrupting the interface we are proposing here. + [swiftlang/swift-testing#824](https://github.com/swiftlang/swift-testing/pull/824) + includes an experimental implementation of this feature. + +- Attaching attachments to issues or to activities: XCTest supports attachments + on `XCTIssue`; Swift Testing does not currently allow developers to create an + issue without immediately recording it, so there is no opportunity to attach + anything to one. XCTest also supports the concept of activities as subsections + of tests; they remain a future direction for Swift Testing. + +## Alternatives considered + +- Doing nothing: there's sufficient demand for this feature that we know we want + to address it. + +- Reusing the existing `XCTAttachment` API from XCTest: while this would + _probably_ have saved me a lot of typing, `XCTAttachment` is an Objective-C + class and is only available on Apple's platforms. The open-source + swift-corelibs-xctest package does not include it or an equivalent interface. + As well, this would create a dependency on XCTest in Swift Testing that does + not currently exist. + +- Implementing `Attachment` as a non-generic type and eagerly serializing + non-sendable or move-only attachable values: an earlier implementation did + exactly this, but it forced us to include an existential box in `Attachment` + to store the attachable value, and that would preclude ever supporting + attachments in Embedded Swift. + +- Having `Attachment` take a byte buffer rather than an attachable value, or + having it take a closure that returns a byte buffer: this would just raise the + problem of attaching arbitrary values up to the test author's layer, and that + would no doubt produce a lot of duplicate implementations of "turn this value + into a byte buffer" while also worsening the interface's ergonomics. + +- Adding a `var contentType: UTType { get set }` property to `Attachment` or to + `Attachable`: `XCTAttachment` lets you specify a Uniform Type Identifier that + tells Xcode the type of data. Uniform Type Identifiers are proprietary and not + available on Linux or Windows, and adding that property would force us to also + add a public dependency on the `UniformTypeIdentifiers` framework and, + indirectly, on Foundation, which would prevent Foundation from authoring tests + using Swift Testing in the future due to the resulting circular dependency. + + We considered using a MIME type instead, but there is no portable mechanism + for turning a MIME type into a path extension, which is ultimately what we + need when writing an attachment to persistent storage. + + Instead, `Attachable` includes the function `preferredName(for:basedOn:)` that + allows an implementation (such as that of `Encodable & Attachable`) to add a + path extension to the filename specified by the test author if needed. + +- Making the `Attachment.record(_:[named:]sourceLocation:)` methods a single + instance method of `Attachment` named `attach()`: this was in the initial + pitch but the community discussed several more ergonomic options and we chose + `Attachment.record(_:sourceLocation:)` instead. + +## Acknowledgments + +Thanks to Stuart Montgomery and Brian Croom for goading me into finally writing +this proposal! + +Thanks to Wil Addario-Turner for his feedback, in particular around `UTType` and +MIME type support. + +Thanks to Honza Dvorsky for his earlier work on attachments in XCTest and his +ideas on how to improve Swift Testing's implementation. diff --git a/proposals/testing/0010-evaluate-condition.md b/proposals/testing/0010-evaluate-condition.md new file mode 100644 index 0000000000..d6a7336ff1 --- /dev/null +++ b/proposals/testing/0010-evaluate-condition.md @@ -0,0 +1,65 @@ +# Public API to evaluate ConditionTrait + +* Proposal: [ST-0010](0010-evaluate-condition.md) +* Authors: [David Catmull](https://github.com/Uncommon) +* Review Manager: [Stuart Montgomery](https://github.com/stmontgomery) +* Status: **Implemented (Swift 6.2)** +* Bug: [swiftlang/swift-testing#903](https://github.com/swiftlang/swift-testing/issues/903) +* Implementation: [swiftlang/swift-testing#909](https://github.com/swiftlang/swift-testing/pull/909), [swiftlang/swift-testing#1097](https://github.com/swiftlang/swift-testing/pull/1097) +* Review: ([pitch](https://forums.swift.org/t/pitch-introduce-conditiontrait-evaluate/77242)) ([review](https://forums.swift.org/t/st-0010-public-api-to-evaluate-conditiontrait/79232)) ([acceptance](https://forums.swift.org/t/accepted-st-0010-public-api-to-evaluate-conditiontrait/79577)) + +## Introduction + +This adds an `evaluate()` method to `ConditionTrait` to evaluate the condition +without requiring a `Test` instance. + +## Motivation + +Currently, the only way a `ConditionTrait` is evaluated is inside the +`prepare(for:)` method. This makes it difficult for third-party libraries to +utilize these traits because evaluating a condition would require creating a +dummy `Test` to pass to that method. + +## Proposed solution + +The proposal is to add a `ConditionTrait.evaluate()` method which returns the +result of the evaluation. The existing `prepare(for:)` method is updated to call +`evaluate()` so that the logic is not duplicated. + +## Detailed design + +The `evaluate()` method is as follows, containing essentially the same logic +as was in `prepare(for:)`: + +```swift +extension ConditionTrait { + /// Evaluate this instance's underlying condition. + /// + /// - Returns: The result of evaluating this instance's underlying condition. + /// + /// The evaluation is performed each time this function is called, and is not + /// cached. + public func evaluate() async throws -> Bool +} +``` + +## Source compatibility + +This change is purely additive. + +## Integration with supporting tools + +This change allows third-party libraries to apply condition traits at other +levels than suites or whole test functions, for example if tests are broken up +into smaller sections. + +## Future directions + +This change seems sufficient for third party libraries to make use of +`ConditionTrait`. Changes for other traits can be tackled in separate proposals. + +## Alternatives considered + +Exposing `ConditionTrait.Kind` and `.kind` was also considered, but it seemed +unnecessary to go that far, and it would encourage duplicating the logic that +already exists in `prepare(for:)`. diff --git a/proposals/testing/0011-issue-handling-traits.md b/proposals/testing/0011-issue-handling-traits.md new file mode 100644 index 0000000000..4d16da9c70 --- /dev/null +++ b/proposals/testing/0011-issue-handling-traits.md @@ -0,0 +1,557 @@ +# Issue Handling Traits + +* Proposal: [ST-0011](0011-issue-handling-traits.md) +* Authors: [Stuart Montgomery](https://github.com/stmontgomery) +* Review Manager: [Paul LeMarquand](https://github.com/plemarquand) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift-testing#1080](https://github.com/swiftlang/swift-testing/pull/1080), + [swiftlang/swift-testing#1121](https://github.com/swiftlang/swift-testing/pull/1121), + [swiftlang/swift-testing#1136](https://github.com/swiftlang/swift-testing/pull/1136), + [swiftlang/swift-testing#1198](https://github.com/swiftlang/swift-testing/pull/1198) +* Review: ([pitch](https://forums.swift.org/t/pitch-issue-handling-traits/80019)) ([review](https://forums.swift.org/t/st-0011-issue-handling-traits/80644)) ([acceptance](https://forums.swift.org/t/accepted-st-0011-issue-handling-traits/81112)) + +## Introduction + +This proposal introduces a built-in trait for handling issues in Swift Testing, +enabling test authors to customize how expectation failures and other issues +recorded by tests are represented. Using a custom issue handler, developers can +transform issue details, perform additional actions, or suppress certain issues. + +## Motivation + +Swift Testing offers ways to customize test attributes and perform custom logic +using traits, but there's currently no way to customize how issues (such as +`#expect` failures) are handled when they occur during testing. + +The ability to handle issues using custom logic would enable test authors to +modify, supplement, or filter issues based on their specific requirements before +the testing library processes them. This capability could open the door to more +flexible testing approaches, improve integration with external reporting systems, +or improve the clarity of results in complex testing scenarios. The sections +below discuss several potential use cases for this functionality. + +### Adding information to issues + +#### Comments + +Sometimes test authors want to include context-specific information to certain +types of failures. For example, they might want to automatically add links to +documentation for specific categories of test failures, or include supplemental +information about the history of a particular expectation in case it fails. An +issue handler could intercept issues after they're recorded and add these +details to the issue's comments before the testing library processes them. + +#### Attachments + +Test failures often benefit from additional diagnostic data beyond the basic +issue description. Swift Testing now supports attachments (as of +[ST-0009](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0009-attachments.md)), +and the ability to add an attachment to an indiviual issue was mentioned as a +future direction in that proposal. The general capability of adding attachments +to issues is outside the scope of this proposal, but if such a capability were +introduced, an issue handler could programmatically attach log files, +screenshots, or other diagnostic artifacts when specific issues occur, making it +easier to diagnose test failures. + +### Suppressing warnings + +Recently, a new API was [pitched][severity-proposal] which would introduce the +concept of severity to issues, along with a new _warning_ severity level, making +it possible to record warnings that do not cause a test to be marked as a +failure. If that feature is accepted, there may be cases where a test author +wants to suppress certain warnings entirely or in specific contexts. + +For instance, they might choose to suppress warnings recorded by the testing +library indicating that two or more arguments to a parameterized test appear +identical, or for one of the other scenarios listed as potential use cases for +warning issues in that proposal. An issue handler would provide a mechanism to +filter issues. + +### Raising or lowering an issue's severity + +Beyond suppressing issues altogether, a test author might want to modify the +severity of an issue (again, assuming the recently pitched +[Issue Severity][severity-proposal] proposal is accepted). They might wish to +either _lower_ an issue with the default error-level severity to a warning (but +not suppress it), or conversely _raise_ a warning issue to an error. + +The Swift compiler now allows control over warning diagnostics (as of +[SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md)). +An issue handling trait would offer analogous functionality for test issues. + +### Normalizing issue details + +Tests that involve randomized or non-deterministic inputs can generate different +issue descriptions on each run, making it difficult to identify duplicates or +recognize patterns in failures. For example, a test verifying random number +generation might produce an expectation failure with different random values +each time: + +``` +Expectation failed: (randomValue → 0.8234) > 0.5 +Expectation failed: (randomValue → 0.6521) > 0.5 +``` + +An issue handler could normalize these issues to create a more consistent +representation: + +``` +Expectation failed: (randomValue → 0.NNNN) > 0.5 +``` + +The original numeric value could be preserved via a comment after being +obfuscated—see [Comments](#comments) under [Adding information to issues](#adding-information-to-issues) +above. + +> [!NOTE] +> This example involves an expectation failure. The value of the `kind` property +> for such an issue would be `.expectationFailed(_:)` and it would have an +> associated value of type `Expectation`. To transform the issue in the way +> described above would require modifying details of the associated `Expectation` +> and its substructure, but these details are currently SPI so test authors +> cannot modify them directly. +> +> Exposing these details is out of scope for this proposal, but a test author +> could still transform this issue to achieve a similar result by changing the +> issue's kind from `.expectationFailed(_:)` to `.unconditional`. This +> experience could be improved in the future in subsequent proposals if desired. + +This normalization can significantly improve the ability to triage failures, as +it becomes easier to recognize when multiple test failures have the same root +cause despite different specific values. + +## Proposed solution + +This proposal introduces a new trait type that can customize how issues are +processed during test execution. + +Here's one contrived example showing how this could be used to add a comment to +each issue recorded by a test: + +```swift +@Test(.compactMapIssues { issue in + var issue = issue + issue.comments.append("Checking whether two literals are equal") + return issue +}) +func literalComparisons() { + #expect(1 == 1) // ✅ + #expect(2 == 3) // ❌ Will invoke issue handler + #expect("a" == "b") // ❌ Will invoke issue handler again +} +``` + +Here's an example showing how warning issues matching a specific criteria could +be suppressed using `.filterIssues`. It also showcases a technique for reusing +an issue handler across multiple tests, by defining it as a computed property in +an extension on `Trait`: + +```swift +extension Trait where Self == IssueHandlingTrait { + static var ignoreSensitiveWarnings: Self { + .filterIssues { issue in + let description = String(describing: issue) + + // Note: 'Issue.severity' has been pitched but not accepted. + return issue.severity <= .warning && SensitiveTerms.all.contains { description.contains($0) } + } + } +} + +@Test(.ignoreSensitiveWarnings) func exampleA() { + ... +} + +@Test(.ignoreSensitiveWarnings) func exampleB() { + ... +} +``` + +The sections below discuss some of the proposed new trait's behavioral details. + +### Precedence order of handlers + +If multiple issue handling traits are applied to or inherited by a test, they +are executed in trailing-to-leading, innermost-to-outermost order. For example, +given the following code: + +```swift +@Suite(.compactMapIssues { ... /* A */ }) +struct ExampleSuite { + @Test(.filterIssues { ... /* B */ }, + .compactMapIssues { ... /* C */ }) + func example() { + ... + } +} +``` + +If an issue is recorded in `example()`, it's processed first by closure C, then +by B, and finally by A. (Unless an issue is suppressed, in which case it will +not advance to any subsequent handler's closure.) This ordering provides +predictable behavior and allows more specific handlers to process issues before +more general ones. + +### Accessing task-local context from handlers + +The closure of an issue handler is invoked synchronously at the point where an +issue is recorded. This means the closure can access task local state from that +context, and potentially use that to augment issues with extra information. +Here's an example: + +```swift +// In module under test: +actor Session { + @TaskLocal static var current: Session? + + let id: String + func connect() { ... } + var isConnected: Bool { ... } + ... +} + +// In test code: +@Test(.compactMapIssues { issue in + var issue = issue + if let session = Session.current { + issue.comments.append("Current session ID: \(session.id)") + } + return issue +}) +func example() async { + let session = Session(id: "ABCDEF") + await Session.$current.withValue(session) { + await session.connect() + #expect(await session.isConnected) // ❌ Expectation failed: await session.isConnected + // Current session ID: ABCDEF + } +} +``` + +### Recording issues from handlers + +Issue handling traits can record additional issues during their execution. These +newly recorded issues will be processed by any later issue handling traits in +the processing chain (see [Precedence order of handlers](#precedence-order-of-handlers)). +This capability allows handlers to augment or provide context to existing issues +by recording related information. + +For example: + +```swift +@Test( + .compactMapIssues { issue in + // This closure will be called for any issue recorded by the test function + // or by the `.filterIssues` trait below. + ... + }, + .filterIssues { issue in + guard let terms = SensitiveTerms.all else { + Issue.record("Cannot determine the set of sensitive terms. Filtering issue by default.") + return true + } + + let description = String(describing: issue).lowercased() + return terms.contains { description.contains($0) } + } +) +func example() { + ... +} +``` + +### Handling issues from other traits + +Issue handling traits process all issues recorded in the context of a test, +including those generated by other traits applied to the test. For instance, if +a test uses the `.enabled(if:)` trait and the condition closure throws an error, +that error will be recorded as an issue, the test will be skipped, and the issue +will be passed to any issue handling traits for processing. + +This comprehensive approach ensures that all issues related to a test, +regardless of their source, are subject to the same customized handling. It +provides a unified mechanism for issue processing that works consistently across +the testing library. + +### Effects in issue handler closures + +The closure of an issue handling trait must be: + +- **Non-`async`**: This reflects the fact that events in + Swift Testing are posted synchronously, which is a fundamental design decision + that, among other things, avoids the need for `await` before every `#expect`. + + While this means that issue handlers cannot directly perform asynchronous work + when processing an individual issue, future enhancements could offer + alternative mechanisms for asynchronous issue processing work at the end of a + test. See the [Future directions](#future-directions) section for more + discussion about this. + +- **Non-`throws`**: Since these handlers are already + being called in response to a failure (the recorded issue), allowing them to + throw errors would introduce ambiguity about how such errors should be + interpreted and reported. + + If an issue handler encounters an error, it can either: + + - Return a modified issue that includes information about the problem, or + - Record a separate issue using the standard issue recording mechanisms (as + [discussed](#recording-issues-from-handlers) above). + +### Handling of non-user issues + +Issue handling traits are applied to a test by a user, and are only intended for +handling issues recorded by tests written by the user. If an issue is recorded +by the testing library itself or the underlying system, not due to a failure +within the tests being run, such an issue will not be passed to an issue +handling trait. Similarly, an issue handling trait should not return an issue +which represents a problem they could not have caused in their test. + +Concretely, this policy means that issues for which the value of the `kind` +property is `.system` will not be passed to the closure of an issue handling +trait. Also, it is not supported for a closure passed to +`compactMapIssues(_:)` to return an issue for which the value of `kind` is +either `.system` or `.apiMisused` (unless the passed-in issue had that kind, +which should only be possible for `.apiMisused`). + +## Detailed design + +This proposal includes the following: + +* A new `IssueHandlingTrait` type that conforms to `TestTrait` and `SuiteTrait`. + * An instance method `handleIssue(_:)` which can be called directly on a + handler trait. This may be useful for composing multiple issue handling + traits. +* Static functions on `Trait` for creating instances of this type with the + following capabilities: + * A function `compactMapIssues(_:)` which returns a trait that can transform + recorded issues. The function takes a closure which is passed an issue and + returns either a modified issue or `nil` to suppress it. + * A function `filterIssues(_:)` which returns a trait that can filter recorded + issues. The function takes a predicate closure that returns a boolean + indicating whether to keep (`true`) or suppress (`false`) an issue. + +Below are the proposed interfaces: + +```swift +/// A type that allows transforming or filtering the issues recorded by a test. +/// +/// Use this type to observe or customize the issue(s) recorded by the test this +/// trait is applied to. You can transform a recorded issue by copying it, +/// modifying one or more of its properties, and returning the copy. You can +/// observe recorded issues by returning them unmodified. Or you can suppress an +/// issue by either filtering it using ``Trait/filterIssues(_:)`` or returning +/// `nil` from the closure passed to ``Trait/compactMapIssues(_:)``. +/// +/// When an instance of this trait is applied to a suite, it is recursively +/// inherited by all child suites and tests. +/// +/// To add this trait to a test, use one of the following functions: +/// +/// - ``Trait/compactMapIssues(_:)`` +/// - ``Trait/filterIssues(_:)`` +public struct IssueHandlingTrait: TestTrait, SuiteTrait { + /// Handle a specified issue. + /// + /// - Parameters: + /// - issue: The issue to handle. + /// + /// - Returns: An issue to replace `issue`, or else `nil` if the issue should + /// not be recorded. + public func handleIssue(_ issue: Issue) -> Issue? +} + +extension Trait where Self == IssueHandlingTrait { + /// Constructs an trait that transforms issues recorded by a test. + /// + /// - Parameters: + /// - transform: A closure called for each issue recorded by the test + /// this trait is applied to. It is passed a recorded issue, and returns + /// an optional issue to replace the passed-in one. + /// + /// - Returns: An instance of ``IssueHandlingTrait`` that transforms issues. + /// + /// The `transform` closure is called synchronously each time an issue is + /// recorded by the test this trait is applied to. The closure is passed the + /// recorded issue, and if it returns a non-`nil` value, that will be recorded + /// instead of the original. Otherwise, if the closure returns `nil`, the + /// issue is suppressed and will not be included in the results. + /// + /// The `transform` closure may be called more than once if the test records + /// multiple issues. If more than one instance of this trait is applied to a + /// test (including via inheritance from a containing suite), the `transform` + /// closure for each instance will be called in right-to-left, innermost-to- + /// outermost order, unless `nil` is returned, which will skip invoking the + /// remaining traits' closures. + /// + /// Within `transform`, you may access the current test or test case (if any) + /// using ``Test/current`` ``Test/Case/current``, respectively. You may also + /// record new issues, although they will only be handled by issue handling + /// traits which precede this trait or were inherited from a containing suite. + /// + /// - Note: `transform` will never be passed an issue for which the value of + /// ``Issue/kind`` is ``Issue/Kind/system``, and may not return such an + /// issue. + public static func compactMapIssues(_ transform: @escaping @Sendable (Issue) -> Issue?) -> Self + + /// Constructs a trait that filters issues recorded by a test. + /// + /// - Parameters: + /// - isIncluded: The predicate with which to filter issues recorded by the + /// test this trait is applied to. It is passed a recorded issue, and + /// should return `true` if the issue should be included, or `false` if it + /// should be suppressed. + /// + /// - Returns: An instance of ``IssueHandlingTrait`` that filters issues. + /// + /// The `isIncluded` closure is called synchronously each time an issue is + /// recorded by the test this trait is applied to. The closure is passed the + /// recorded issue, and if it returns `true`, the issue will be preserved in + /// the test results. Otherwise, if the closure returns `false`, the issue + /// will not be included in the test results. + /// + /// The `isIncluded` closure may be called more than once if the test records + /// multiple issues. If more than one instance of this trait is applied to a + /// test (including via inheritance from a containing suite), the `isIncluded` + /// closure for each instance will be called in right-to-left, innermost-to- + /// outermost order, unless `false` is returned, which will skip invoking the + /// remaining traits' closures. + /// + /// Within `isIncluded`, you may access the current test or test case (if any) + /// using ``Test/current`` ``Test/Case/current``, respectively. You may also + /// record new issues, although they will only be handled by issue handling + /// traits which precede this trait or were inherited from a containing suite. + /// + /// - Note: `isIncluded` will never be passed an issue for which the value of + /// ``Issue/kind`` is ``Issue/Kind/system``. + public static func filterIssues(_ isIncluded: @escaping @Sendable (Issue) -> Bool) -> Self +} +``` + +## Source compatibility + +This new trait is additive and should not affect source compatibility of +existing test code. + +If any users have an existing extension on `Trait` containing a static function +whose name conflicts with one in this proposal, the standard technique of +fully-qualifying its callsite with the relevant module name can be used to +resolve any ambiguity, but this should be rare. + +## Integration with supporting tools + +Most tools which integrate with the testing library interpret recorded issues in +some way, whether by writing them to a persistent data file or presenting them +in UI. These mechanisms will continue working as before, but the issues they act +on will be the result of any issue handling traits. If an issue handler +transforms an issue, the integrated tool will only receive the transformed issue, +and if a trait suppresses an issue, the tool will not be notified about the +issue at all. + +## Future directions + +### "Test ended" trait + +The current proposal does not allow `await` in an issue handling closure--see +[Non-`async`](#non-async) above. In addition to not allowing concurrency, the +proposed behavior is that the issue handler is called once for _each_ issue +recorded. + +Both of these policies could be problematic for some use cases. Some users may +want to collect additional diagnostics if a test fails, but only do so once per +per test (typically after it finishes) instead of once per _issue_, since the +latter may lead to redundant or wasteful work. Also, collecting diagnostics may +require calling `async` APIs. + +In the future, a new trait could be added which offers a closure that is +unconditionally called once after a test ends. The closure could be provided the +result of the test (e.g. pass/fail/skip) and access to all the issues it +recorded. This hypothetical trait's closure could be safely made `async`, since +it wouldn't be subject to the same limitations as event delivery, and this could +complement the APIs proposed above. + +### Comprehensive event observation API + +As a more generalized form of the ["Test ended" trait](#test-ended-trait) idea +above, Swift Testing could offer a more comprehensive suite of APIs for +observing test events of all kinds. This would a much larger effort, but was +[mentioned](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#flexibility) +as a goal in the +[Swift Testing vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md). + +### Standalone function + +It could be useful to offer the functionality of an issue handling trait as a +standalone function (similar to `withKnownIssue { }`) so that it could be +applied to a narrower section of code than an entire test or suite. This idea +came up during the pitch phase, and we believe that sort of pattern may be +useful more broadly for other kinds of traits. Accordingly, it may make more +sense to address this in a separate proposal and design it in a way that +encompasses any trait. + +## Alternatives considered + +### Allow issue handler closures to throw errors + +The current proposal does not allow throwing an error from an issue handling +closure--see [Non-`throws`](#non-throws) above. This artificial restriction +could be lifted, and errors thrown by issue handler closures could be caught +and recorded as issues, matching the behavior of test functions. + +As mentioned earlier, allowing thrown errors could make test results more +confusing. We expect that most often, a test author will add an issue handler +because they want to make failures easier to interpret, and they generally won't +want an issue handler to record _more_ issues while doing so even if it can. Not +allowing errors to be thrown forces the author of the issue handler to make an +explicit decision about whether they want an additional issue to be recorded if +the handler encounters an error. + +### Make the closure's issue parameter `inout` + +The closure parameter of `compactMapIssues(_:)` currently has one parameter of +type `Issue` and returns an optional `Issue?` to support returning `nil` in +order to suppress an issue. If an issue handler wants to modify an issue, it +first needs to copy it to a mutable variable (`var`), mutate it, then return the +modified copy. These copy and return steps require extra lines of code within +the closure, and they could be eliminated if the parameter was declared `inout`. + +The most straightforward way to achieve this would be for the closure to instead +have a `Void` return type and for its parameter to become `inout`. However, in +order to _suppress_ an issue, the parameter would also need to become optional +(`inout Issue?`) and this would mean that all usages would first need to be +unwrapped. This feels non-ergonomic, and would differ from the standard +library's typical pattern for `compactMap` functions. + +Another way to achieve this ([suggested](https://forums.swift.org/t/st-0011-issue-handling-traits/80644/3) +by [@Val](https://forums.swift.org/u/Val) during proposal review) could be to +declare the return type of the closure `Void?` and the parameter type +`inout Issue` (non-optional). This alternative would not require unwrapping the +issue first and would still permit suppressing issues by returning `nil`. It +could also make one of the alternative names (such as `transformIssues` +discussed below) more fitting. However, this is a novel API pattern which isn't +widely used in Swift, and may be confusing to users. There were also concerns +raised by other reviewers that the language's implicit return for `Void` may not +be intentionally applied to `Optional` and that this mechanism could break +in the future. + +### Alternate names for the static trait functions + +We could choose different names for the static `compactMapIssues(_:)` or +`filterIssues(_:)` functions. Some alternate names considered were: + +- `transformIssues` instead of `compactMapIssues`. "Compact map" seemed to align + better with "filter" of `filterIssues`, however. +- `handleIssues` instead of `compactMapIssues`. The word "handle" is in the name + of the trait type already; it's a more general word for what all of these + usage patterns enable, so it felt too broad. +- Using singular "issue" rather than plural "issues" in both APIs. This may not + adequately convey that the closure can be invoked more than once. + +## Acknowledgments + +Thanks to [Brian Croom](https://github.com/briancroom) for feedback on the +initial concept, and for making a suggestion which led to the +["Test ended" trait](#test-ended-trait) idea mentioned in Alternatives +considered. + +[severity-proposal]: https://forums.swift.org/t/pitch-test-issue-warnings/79285 diff --git a/proposals/testing/0012-exit-test-value-capturing.md b/proposals/testing/0012-exit-test-value-capturing.md new file mode 100644 index 0000000000..c31fd772ae --- /dev/null +++ b/proposals/testing/0012-exit-test-value-capturing.md @@ -0,0 +1,276 @@ +# Capturing values in exit tests + +* Proposal: [ST-0012](0012-exit-test-value-capturing.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Paul LeMarquand](https://github.com/plemarquand) +* Status: **Implemented (Swift 6.3)** +* Bug: [swiftlang/swift-testing#1157](https://github.com/swiftlang/swift-testing/issues/1157) +* Implementation: [swiftlang/swift-testing#1040](https://github.com/swiftlang/swift-testing/pull/1040), [swiftlang/swift-testing#1165](https://github.com/swiftlang/swift-testing/pull/1165) _et al._ +* Review: ([pitch](https://forums.swift.org/t/pitch-capturing-values-in-exit-tests/80494)) ([review](https://forums.swift.org/t/st-0012-capturing-values-in-exit-tests/80963)) ([acceptance](https://forums.swift.org/t/accepted-st-0012-capturing-values-in-exit-tests/81250)) + +## Introduction + +In Swift 6.2, we introduced the concept of an _exit test_: a section of code in +a test function that would run in an independent process and allow test authors +to test code that terminates the process. For example: + +```swift +enum Fruit: Sendable, Codable, Equatable { + case apple, orange, olive, tomato + var isSweet: Bool { get } + + consuming func feed(to bat: FruitBat) { + precondition(self.isSweet, "Fruit bats don't like savory fruits!") + ... + } +} + +@Test func `Fruit bats don't eat savory fruits`() async { + await #expect(processExitsWith: .failure) { + let fruit = Fruit.olive + let bat = FruitBat(named: "Chauncey") + fruit.feed(to: bat) // should trigger a precondition failure and process termination + } +} +``` + +This proposal extends exit tests to support capturing state from the enclosing +context (subject to several practical constraints.) + +## Motivation + +Exit tests in their current form are useful, but there is no reliable way to +pass non-constant information from the parent process to the child process, +which makes them difficult to use with parameterized tests. Consider: + +```swift +@Test(arguments: [Fruit.olive, .tomato]) +func `Fruit bats don't eat savory fruits`(_ fruit: Fruit) async { + await #expect(processExitsWith: .failure) { + let bat = FruitBat(named: "Chauncey") + fruit.feed(to: bat) // 🛑 can't capture 'fruit' from enclosing scope + } +} +``` + +In the above example, the test function's argument cannot be passed into the +exit test. In a trivial example like this one, it wouldn't be difficult to write +two tests that differ only in the case of `Fruit` they use in their exit test +bodies, but this approach doesn't scale very far and is generally an +anti-pattern when using Swift Testing. + +## Proposed solution + +We propose allowing the capture of values in an exit test when they are +specified in a closure capture list on the exit test's body. + +## Detailed design + +The signatures of the exit test macros `expect(processExitsWith:)` and +`require(processExitsWith:)` are unchanged. A test author may now add a closure +capture list to the body of an exit test: + +```swift +@Test(arguments: [Fruit.olive, .tomato]) +func `Fruit bats don't eat savory fruits`(_ fruit: Fruit) async { + await #expect(processExitsWith: .failure) { [fruit] in + let bat = FruitBat(named: "Chauncey") + fruit.feed(to: bat) + } +} +``` + +This feature has some necessary basic constraints: + +### Captured values must be explicitly listed in a closure capture list + +Swift Testing needs to know what values need to be encoded, sent to the child +process, and decoded. Swift macros including `#expect(processExitsWith:)` must +rely solely on syntax—that is, the code typed by a test author. An implicit +capture within an exit test body is indistinguishable from any other identifier +or symbol name. + +Hence, only values listed in the closure's capture list will be captured. +Implicitly captured values will produce a compile-time diagnostic as they do +today. + +### Captured values must conform to Sendable and Codable + +Captured values will be sent across process boundaries and, in order to support +that operation, must conform to `Codable`. As well, captured values need to make +their way through the various internal mechanisms of Swift Testing and its host +infrastructure, and so must conform to `Sendable`. Conformance to `Copyable` and +`Escapable` is implied. + +If a value that does _not_ conform to the above protocols is specified in an +exit test body's capture list, a diagnostic is emitted: + +```swift +let bat: FruitBat = ... +await #expect(processExitsWith: .failure) { [bat] in + // 🛑 Type of captured value 'bat' must conform to 'Sendable' and 'Codable' + ... +} +``` + +### Captured values' types must be visible to the exit test macro + +In order for us to successfully _decode_ captured values in the child process, +we must know their Swift types. Type information is not readily available during +macro expansion and we must, in general, rely on the parsed syntax tree for it. + +The type of `self` and the types of arguments to the calling function are, +generally, known and can be inferred from context[^shadows]. The types of other +values, including local variables and global state, are not visible in the +syntax tree and must be specified explicitly in the capture list using an `as` +expression: + +```swift +await #expect(processExitsWith: .failure) { [fruit = fruit as Fruit] in + ... +} +``` + +Finally, the types of captured literals (e.g. `[x = 123]`) are known at compile +time and can always be inferred as `IntegerLiteralType` etc., although we don't +anticipate this will be particularly useful in practice. + +If the type of a captured value cannot be resolved from context, the test author +will see an error at compile time: + +```swift +await #expect(processExitsWith: .failure) { [fruit] in + // 🛑 Type of captured value 'fruit' is ambiguous + // Fix-It: Add '= fruit as T' + ... +} +``` + +See the **Future directions** section of this proposal for more information on +how we hope to lift this constraint. If we are able to lift this constraint in +the future, we expect it will not require (no pun intended) a second Swift +Evolution proposal. + +[^shadows]: If a local variable is declared that shadows `self` or a function + argument, we may incorrectly infer the type of that value when captured. When + this occurs, Swift Testing emits a diagnostic of the form "🛑 Type of captured + value 'foo' is ambiguous". + +## Source compatibility + +This change is additive and relies on syntax that would previously be rejected +at compile time. + +## Integration with supporting tools + +Xcode, Swift Package Manager, and the Swift VS Code plugin _already_ support +captured values in exit tests as they use Swift Testing's built-in exit test +handling logic. + +Tools that implement their own exit test handling logic will need to account for +captured values. The `ExitTest` type now has a new SPI property: + +```swift +extension ExitTest { + /// The set of values captured in the parent process before the exit test is + /// called. + /// + /// This property is automatically set by the testing library when using the + /// built-in exit test handler and entry point functions. Do not modify the + /// value of this property unless you are implementing a custom exit test + /// handler or entry point function. + /// + /// The order of values in this array must be the same between the parent and + /// child processes. + @_spi(ForToolsIntegrationOnly) + public var capturedValues: [CapturedValue] { get set } +} +``` + +In the parent process (that is, for an instance of `ExitTest` passed to +`Configuration.exitTestHandler`), this property represents the values captured +at runtime by the exit test. In the child process (that is, for an instance of +`ExitTest` returned from `ExitTest.find(identifiedBy:)`), the elements in this +array do not have values associated with them until the hosting tool provides +them. + +## Future directions + +- Supporting captured values without requiring type information + + We need the types of captured values in order to successfully decode them, but + we are constrained by macros being syntax-only. In the future, the compiler + may gain a language feature similar to `decltype()` in C++ or `typeof()` in + C23, in which case we should be able to use it and avoid the need for explicit + types in the capture list. ([rdar://153389205](rdar://153389205)) + +- Explicitly marking the body closure as requiring explicit captures + + Currently, if the body closure implicitly captures a value, the diagnostic the + compiler provides is a bit opaque: + + > 🛑 A C function pointer cannot be formed from a closure that captures context + + In the future, it may be possible to annotate the body closure with an + attribute, keyword, or other decoration that tells the compiler we need an + explicit capture list, which would allow it to provide a clearer diagnostic if + a value is implicitly captured. + +- Supporting capturing values that do not conform to `Codable` + + Alternatives to `Codable` exist or have been proposed, such as + [`NSSecureCoding`](https://developer.apple.com/documentation/foundation/nssecurecoding) + or [`JSONCodable`](https://forums.swift.org/t/the-future-of-serialization-deserialization-apis/78585). + In the future, we may want to extend support for values that conform to these + protocols instead of `Codable`. + +## Alternatives considered + +- Doing nothing. There is sufficient motivation to support capturing values in + exit tests and it is within our technical capabilities. + +- Passing captured values as arguments to `#expect(processExitsWith:)` and its + body closure. For example: + + ```swift + await #expect( + processExitsWith: .failure, + arguments: [fruit, bat] + ) { fruit, bat in + ... + } + ``` + + This is technically feasible, but: + + - It requires that the caller state the capture list twice; + - Type information still isn't available for captured values, so you'd still + need to _actually_ write `{ (fruit: Fruit, bat: Bat) in ... }` (or otherwise + specify the types somewhere in the macro invocation); and + - The language already has a dedicated syntax for specifying lists of values + that should be captured in a closure. + +- Supporting non-`Sendable` or non-`Codable` captured values. Since exit tests' + bodies are, by definition, in separate isolation domains from the caller, and + since they, by nature, run in separate processes, conformance to these + protocols is fundamentally necessary. + +- Implicitly capturing `self`. This would require us to statically detect during + macro expansion whether `self` conformed to the necessary protocols _and_ + would preclude capturing any state from static or free test functions. + +- Forking the exit test process such that all captured values are implicitly + copied by the kernel into the new process. Forking, in the UNIX fashion, is + fundamentally incompatible with the Swift runtime and the Swift thread pool. + On Darwin, you [cannot fork a process that links to Core Foundation without + immediately calling `exec()`](https://duckduckgo.com/?q=__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__), + and `fork()` isn't even present on Windows. + +## Acknowledgments + +Thanks to @rintaro for assistance investigating swift-syntax diagnostic support +and to @xedin for humouring my questions about `decltype()`. + +Thanks to the Swift Testing team and the Testing Workgroup as always. And thanks +to those individuals, who shall remain unnamed, who nerd-sniped me into building +this feature. diff --git a/proposals/testing/0013-issue-severity-warning.md b/proposals/testing/0013-issue-severity-warning.md new file mode 100644 index 0000000000..ffa7c04514 --- /dev/null +++ b/proposals/testing/0013-issue-severity-warning.md @@ -0,0 +1,244 @@ +# Test Issue Severity + +- Proposal: [ST-0013](0013-issue-severity-warning.md) +- Authors: [Suzy Ratcliff](https://github.com/suzannaratcliff) +- Review Manager: [Maarten Engels](https://github.com/maartene) +- Status: **Implemented (Swift 6.3)** +- Implementation: [swiftlang/swift-testing#1075](https://github.com/swiftlang/swift-testing/pull/1075), + [swiftlang/swift-testing#1247](https://github.com/swiftlang/swift-testing/pull/1247) +- Review: ([pitch](https://forums.swift.org/t/pitch-test-issue-warnings/79285)) ([review](https://forums.swift.org/t/st-0013-test-issue-warnings/80991)) ([acceptance](https://forums.swift.org/t/accepted-st-0013-test-issue-severity/81385)) + +## Introduction + +I propose introducing a new API to Swift Testing that allows developers to record issues with a specified severity level. By default, all issues will have severity level “error”, and a new “warning” level will be added to represent less severe issues. The effects of the warning recorded on a test will not cause a failure but will be included in the test results for inspection after the run is complete. + +## Motivation + +Currently, when an issue arises during a test, the only possible outcome is to mark the test as failed. This presents a challenge for users who want a deeper insight into the events occurring within their tests. By introducing a dedicated mechanism to record issues that do not cause test failure, users can more effectively inspect and diagnose problems at runtime and review results afterward. This enhancement provides greater flexibility and clarity in test reporting, ultimately improving the debugging and analysis process. + +### Use Cases + +- Warning about a Percentage Discrepancy in Image Comparison: + - Scenario: When comparing two images to assess their similarity, a warning can be triggered if there's a 95% pixel match, while a test failure is set at a 90% similarity threshold. + - Reason: In practices like snapshot testing, minor changes (such as a timestamp) might cause a discrepancy. Setting a 90% match as a pass ensures test integrity. However, a warning at 95% alerts testers that, although the images aren't identical, the test has passed, which may warrant further investigation. +- Warning for Duplicate Argument Inputs in Tests: + - Scenario: In a test library, issue a warning if a user inputs the same argument twice, rather than flagging an error. + - Reason: Although passing the same argument twice might not be typical, some users may have valid reasons for doing so. Thus, a warning suffices, allowing flexibility without compromising the test's execution. +- Warning for Recoverable Unexpected Events: + - Scenario: During an integration test where data is retrieved from a server, a warning can be issued if the primary server is down, prompting a switch to an alternative server. Usually mocking is the solution for this but may not test everything needed for an integration test. + - Reason: Since server downtime might happen and can be beyond the tester's control, issuing a warning rather than a failure helps in debugging and understanding potential issues without impacting the test's overall success. +- Warning for a retry during setup for a test: + - Scenario: During test setup part of your code may be configured to retry, it would be nice to notify in the results that a retry happened + - Reason: This makes sense to be a warning and not a failure because if the retry succeeds the test may still verify the code correctly + +## Proposed solution + +We propose introducing a new property on `Issue` in Swift Testing called `severity`, that represents if an issue is a `warning` or an `error`. +The default Issue severity will still be `error` and users can set the severity when they record an issue. + +Test authors will be able to inspect if the issue is a failing issue and will be able to check the severity. + +## Detailed design + +### Severity Enum + +We introduce a Severity enum to categorize issues detected during testing. This enum is crucial for distinguishing between different levels of test issues and is defined as follows: + +The `Severity` enum: + +```swift +extension Issue { + // ... + + public enum Severity: Codable, Comparable, CustomStringConvertible, Sendable { + /// The severity level for an issue which should be noted but is not + /// necessarily an error. + /// + /// An issue with warning severity does not cause the test it's associated + /// with to be marked as a failure, but is noted in the results. + case warning + + /// The severity level for an issue which represents an error in a test. + /// + /// An issue with error severity causes the test it's associated with to be + /// marked as a failure. + case error + } +} +``` + +### Recording Non-Failing Issues + +To enable test authors to log non-failing issues without affecting test results, we provide a method for recording such issues: + +```swift +Issue.record("My comment", severity: .warning) +``` + +Here is the `Issue.record` method definition with severity as a parameter. + +```swift +extension Issue { + // ... + + /// Record an issue when a running test and an issue occurs. + /// + /// - Parameters: + /// - comment: A comment describing the expectation. + /// - severity: The severity level of the issue. This factor impacts whether the issue constitutes a failure. + /// - sourceLocation: The source location to which the issue should be + /// attributed. + /// + /// - Returns: The issue that was recorded. + /// + /// Use this function if, while running a test, an issue occurs that cannot be + /// represented as an expectation (using the ``expect(_:_:sourceLocation:)`` + /// or ``require(_:_:sourceLocation:)-5l63q`` macros.) + @discardableResult public static func record( + _ comment: Comment? = nil, + severity: Severity = .error, + sourceLocation: SourceLocation = #_sourceLocation + ) -> Self +} +``` + +### Issue Type Enhancements + +The Issue type is enhanced with two new properties to better handle and report issues: + +- `severity`: This property allows access to the specific severity level of an issue, enabling more precise handling of test results. + +```swift +extension Issue { + // ... + + /// The severity of the issue. + public var severity: Severity { get set } +} + +``` + +- `isFailure`: A boolean computed property to determine if an issue results in a test failure, thereby helping in result aggregation and reporting. + +```swift +extension Issue { + // ... + + /// Whether or not this issue should cause the test it's associated with to be + /// considered a failure. + /// + /// The value of this property is `true` for issues which have a severity level of + /// ``Issue/Severity/error`` or greater and are not known issues via + /// ``withKnownIssue(_:isIntermittent:sourceLocation:_:when:matching:)``. + /// Otherwise, the value of this property is `false.` + /// + /// Use this property to determine if an issue should be considered a failure, instead of + /// directly comparing the value of the ``severity`` property. + public var isFailure: Bool { get } +} +``` + +Example usage of `severity` and `isFailure`: + +```swift +withKnownIssue { + // ... +} matching: { issue in + issue.isFailure || issue.severity > .warning +} +``` + +For more details on `Issue`, refer to the [Issue Documentation](https://developer.apple.com/documentation/testing/issue). + +This revision aims to clarify the functionality and usage of the `Severity` enum and `Issue` properties while maintaining consistency with the existing Swift API standards. + +## Source compatibility + +The aspect of this proposal which adds a new `severity:` parameter to the +`Issue.record` function introduces the possibility of a source breakage for any +clients who are capturing a reference to the function. Existing code could break +despite the fact that the new parameter specifies a default value of `.error`. +Here's a contrived example: + +``` +// ❌ Source breakage due to new `Issue.Severity` parameter +let myRecordFunc: (Comment?, SourceLocation) -> Issue = Issue.record +``` + +To avoid source breakage, we will maintain the existing overload and preserve +its signature, but mark it deprecated, disfavored, and hidden from documentation: + +```swift +extension Issue { + // ... + + @available(*, deprecated, message: "Use record(_:severity:sourceLocation:) instead.") + @_disfavoredOverload + @_documentation(visibility: private) + @discardableResult public static func record( + _ comment: Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) -> Self +} +``` + +## Integration with supporting tools + +### Event stream + +Issue severity will be in the event stream output when a `issueRecorded` event occurs. This will be a breaking change because some tools may assume that all `issueRecorded` events are failing. Due to this we will be bumping the event stream version and v1 will maintain it's behavior and not output any events for non failing issues. We will also be adding `isFailure` to the issue so that clients will know if the issue should be treated as a failure. `isFailure` is a computed property. + +The JSON event stream ABI will be amended correspondingly: + +```diff + ::= { + "isKnown": , ; is this a known issue or not? ++ "severity": , ; the severity of the issue ++ "isFailure": , ; if the issue is a failing issue + ["sourceLocation": ,] ; where the issue occurred, if known + } +``` + +Example of an `issueRecorded` event in the json output: + +``` +{"kind":"event","payload":{"instant":{"absolute":302928.100968,"since1970":1751305230.364087},"issue":{"_backtrace":[{"address":4437724864},{"address":4427566652},{"address":4437724280},{"address":4438635916},{"address":4438635660},{"address":4440823880},{"address":4437933556},{"address":4438865080},{"address":4438884348},{"address":11151272236},{"address":4438862360},{"address":4438940324},{"address":4437817340},{"address":4438134208},{"address":4438132164},{"address":4438635048},{"address":4440836660},{"address":4440835536},{"address":4440834989},{"address":4438937653},{"address":4438963225},{"address":4438895773},{"address":4438896161},{"address":4438891517},{"address":4438937117},{"address":4438962637},{"address":4439236617},{"address":4438936181},{"address":4438962165},{"address":4438639149},{"address":4438935045},{"address":4438935513},{"address":11151270653},{"address":11151269797},{"address":4438738225},{"address":4438872065},{"address":4438933417},{"address":4438930265},{"address":4438930849},{"address":4438909741},{"address":4438965489},{"address":11151508333}],"_severity":"error","isFailure":true, "isKnown":false,"sourceLocation":{"_filePath":"\/Users\/swift-testing\/Tests\/TestingTests\/EntryPointTests.swift","column":23,"fileID":"TestingTests\/EntryPointTests.swift","line":46}},"kind":"issueRecorded","messages":[{"symbol":"fail","text":"Issue recorded"},{"symbol":"details","text":"Unexpected issue Issue recorded (warning) was recorded."}],"testID":"TestingTests.EntryPointTests\/warningIssues()\/EntryPointTests.swift:33:4"},"version":0} +``` + +### Console output + +When there is an issue recorded with severity warning, such as using the following code: + +```swift +Issue.record("My comment", severity: .warning) +``` + +the console output will look like the following: + +``` +◇ Test "All elements of two ranges are equal" started. +� Test "All elements of two ranges are equal" recorded a warning at ZipTests.swift:32:17: Issue recorded +↳ My comment +✔ Test "All elements of two ranges are equal" passed after 0.001 seconds with 1 warning. +``` + +## Alternatives considered + +- Separate Issue Creation and Recording: We considered providing a mechanism to create issues independently before recording them, rather than passing the issue details directly to the `record` method. This approach was ultimately set aside in favor of simplicity and directness in usage. + +- Naming of `isFailure` vs. `isFailing`: We evaluated whether to name the property `isFailing` instead of `isFailure`. The decision to use `isFailure` was made to adhere to naming conventions and ensure clarity and consistency within the API. + +- Severity-Only Checking: We deliberated not exposing `isFailure` and relying solely on `severity` checks. However, this was rejected because it would require test authors to overhaul their code should we introduce additional severity levels in the future. By providing `isFailure`, we offer a straightforward way to determine test outcome impact, complementing the severity feature. +- Naming `Severity.error` `Severity.failure` instead because this will always be a failing issue and test authors often think of test failures. Error and warning match build naming conventions and XCTest severity naming convention. + +## Future directions + +- In the future I could see the warnings being able to be promoted to errors in order to run with a more strict testing configuration + +- In the future I could see adding other levels of severity such as Info and Debug for users to create issues with other information. + +## Acknowledgments + +Thanks to Stuart Montgomery for creating and implementing severity in Swift Testing. + +Thanks to Joel Middendorf, Dorothy Fu, Brian Croom, and Jonathan Grynspan for feedback on severity along the way. diff --git a/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md b/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md new file mode 100644 index 0000000000..0aa9221b7a --- /dev/null +++ b/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md @@ -0,0 +1,417 @@ +# Image attachments in Swift Testing (Apple platforms) + +* Proposal: [ST-0014](0014-image-attachments-in-swift-testing-apple-platforms.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Maarten Engels](https://github.com/maartene/) +* Status: **Accepted** +* Bug: rdar://154869058 +* Implementation: [swiftlang/swift-testing#827](https://github.com/swiftlang/swift-testing/pull/827), _et al._ +* Review: ([pitch](https://forums.swift.org/t/pitch-image-attachments-in-swift-testing/80867)) ([review](https://forums.swift.org/t/st-0014-image-attachments-in-swift-testing-apple-platforms/81507)) ([acceptance](https://forums.swift.org/t/accepted-st-0014-image-attachments-in-swift-testing-apple-platforms/81868)) + +## Introduction + +We introduced the ability to add attachments to tests in Swift 6.2. This +proposal augments that feature to support attaching images on Apple platforms. + +## Motivation + +It is frequently useful to be able to attach images to tests for engineers to +review, e.g. if a UI element is not being drawn correctly. If something doesn't +render correctly in a CI environment, for instance, it is very useful to test +authors to be able to download the failed rendering and examine it at-desk. + +Today, Swift Testing offers support for **attachments** which allow a test +author to save arbitrary files created during a test run. However, if those +files are images, the test author must write their own code to encode them as +(for example) JPEG or PNG files before they can be attached to a test. + +## Proposed solution + +We propose adding support for images as a category of Swift type that can be +encoded using standard graphics formats such as JPEG or PNG. Image serialization +is beyond the purview of the testing library, so Swift Testing will defer to the +operating system to provide the relevant functionality. As such, this proposal +covers support for **Apple platforms** only. Support for other platforms such as +Windows is discussed in the **Future directions** section of this proposal. + +## Detailed design + +A new protocol is introduced for Apple platforms: + +```swift +/// A protocol describing images that can be converted to instances of +/// ``Testing/Attachment``. +/// +/// Instances of types conforming to this protocol do not themselves conform to +/// ``Testing/Attachable``. Instead, the testing library provides additional +/// initializers on ``Testing/Attachment`` that take instances of such types and +/// handle converting them to image data when needed. +/// +/// The following system-provided image types conform to this protocol and can +/// be attached to a test: +/// +/// - [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage) +/// - [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage) +/// - [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) +/// (macOS) +/// - [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) +/// (iOS, watchOS, tvOS, visionOS, and Mac Catalyst) +/// +/// You do not generally need to add your own conformances to this protocol. If +/// you have an image in another format that needs to be attached to a test, +/// first convert it to an instance of one of the types above. +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +public protocol AttachableAsCGImage { + /// An instance of `CGImage` representing this image. + /// + /// - Throws: Any error that prevents the creation of an image. + var attachableCGImage: CGImage { get throws } +} +``` + +And conformances are provided for the following types: + +- [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage) +- [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage) +- [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) + (macOS) +- [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) + (iOS, watchOS, tvOS, visionOS, and Mac Catalyst) + +The implementation of `CGImage.attachableCGImage` simply returns `self`, while +the other implementations extract an underlying `CGImage` instance if available +or render one on-demand. + +> [!NOTE] +> The list of conforming types may be extended in the future. The Testing +> Workgroup will determine if additional Swift Evolution reviews are needed. + +### Attaching a conforming image + +New overloads of `Attachment.init()` and `Attachment.record()` are provided: + +```swift +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +extension Attachment { + /// Initialize an instance of this type that encloses the given image. + /// + /// - Parameters: + /// - attachableValue: The value that will be attached to the output of + /// the test run. + /// - preferredName: The preferred name of the attachment when writing it + /// to a test report or to disk. If `nil`, the testing library attempts + /// to derive a reasonable filename for the attached value. + /// - imageFormat: The image format with which to encode `attachableValue`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// The following system-provided image types conform to the + /// ``AttachableAsCGImage`` protocol and can be attached to a test: + /// + /// - [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage) + /// - [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage) + /// - [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) + /// (macOS) + /// - [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) + /// (iOS, watchOS, tvOS, visionOS, and Mac Catalyst) + /// + /// The testing library uses the image format specified by `imageFormat`. Pass + /// `nil` to let the testing library decide which image format to use. If you + /// pass `nil`, then the image format that the testing library uses depends on + /// the path extension you specify in `preferredName`, if any. If you do not + /// specify a path extension, or if the path extension you specify doesn't + /// correspond to an image format the operating system knows how to write, the + /// testing library selects an appropriate image format for you. + public init( + _ attachableValue: T, + named preferredName: String? = nil, + as imageFormat: AttachableImageFormat? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) where AttachableValue == _AttachableImageWrapper + + /// Attach an image to the current test. + /// + /// - Parameters: + /// - image: The value to attach. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the testing library attempts to + /// derive a reasonable filename for the attached value. + /// - imageFormat: The image format with which to encode `attachableValue`. + /// - sourceLocation: The source location of the call to this function. + /// + /// This function creates a new instance of ``Attachment`` wrapping `image` + /// and immediately attaches it to the current test. + /// + /// The following system-provided image types conform to the + /// ``AttachableAsCGImage`` protocol and can be attached to a test: + /// + /// - [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage) + /// - [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage) + /// - [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) + /// (macOS) + /// - [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) + /// (iOS, watchOS, tvOS, visionOS, and Mac Catalyst) + /// + /// The testing library uses the image format specified by `imageFormat`. Pass + /// `nil` to let the testing library decide which image format to use. If you + /// pass `nil`, then the image format that the testing library uses depends on + /// the path extension you specify in `preferredName`, if any. If you do not + /// specify a path extension, or if the path extension you specify doesn't + /// correspond to an image format the operating system knows how to write, the + /// testing library selects an appropriate image format for you. + public static func record( + _ image: T, + named preferredName: String? = nil, + as imageFormat: AttachableImageFormat? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) where AttachableValue == _AttachableImageWrapper +} +``` + +> [!NOTE] +> `_AttachableImageWrapper` is an implementation detail required by Swift's +> generic type system and is not itself part of this proposal. For completeness, +> its public interface is: +> +> ```swift +> @available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +> public struct _AttachableImageWrapper: Sendable, AttachableWrapper where Image: AttachableAsCGImage { +> public var wrappedValue: Image { get } +> } +> ``` + +### Specifying image formats + +A test author can specify the image format to use with `AttachableImageFormat`. +This type abstractly represents the destination image format and, where +applicable, encoding quality: + +```swift +/// A type describing image formats supported by the system that can be used +/// when attaching an image to a test. +/// +/// When you attach an image to a test, you can pass an instance of this type to +/// ``Attachment/record(_:named:as:sourceLocation:)`` so that the testing +/// library knows the image format you'd like to use. If you don't pass an +/// instance of this type, the testing library infers which format to use based +/// on the attachment's preferred name. +/// +/// The PNG and JPEG image formats are always supported. The set of additional +/// supported image formats is platform-specific: +/// +/// - On Apple platforms, you can use [`CGImageDestinationCopyTypeIdentifiers()`](https://developer.apple.com/documentation/imageio/cgimagedestinationcopytypeidentifiers()) +/// from the [Image I/O framework](https://developer.apple.com/documentation/imageio) +/// to determine which formats are supported. +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +public struct AttachableImageFormat: Sendable { + /// The encoding quality to use for this image format. + /// + /// The meaning of the value is format-specific with `0.0` being the lowest + /// supported encoding quality and `1.0` being the highest supported encoding + /// quality. The value of this property is ignored for image formats that do + /// not support variable encoding quality. + public var encodingQuality: Float { get } +} +``` + +Conveniences for the PNG and JPEG formats are provided as they are very widely +used and supported across almost all modern platforms, Web browsers, etc.: + +```swift +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +extension AttachableImageFormat { + /// The PNG image format. + public static var png: Self { get } + + /// The JPEG image format with maximum encoding quality. + public static var jpeg: Self { get } + + /// The JPEG image format. + /// + /// - Parameters: + /// - encodingQuality: The encoding quality to use when serializing an + /// image. A value of `0.0` indicates the lowest supported encoding + /// quality and a value of `1.0` indicates the highest supported encoding + /// quality. + /// + /// - Returns: An instance of this type representing the JPEG image format + /// with the specified encoding quality. + public static func jpeg(withEncodingQuality encodingQuality: Float) -> Self +} +``` + +For instance, to save an image in the JPEG format with 50% image quality, you +can use `.jpeg(withEncodingQuality: 0.5)`. + +On Apple platforms, a convenience initializer that takes an instance of `UTType` +is also provided and lets you select any format supported by the underlying +Image I/O framework: + +```swift +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +extension AttachableImageFormat { + /// The content type corresponding to this image format. + /// + /// The value of this property always conforms to [`UTType.image`](https://developer.apple.com/documentation/uniformtypeidentifiers/uttype-swift.struct/image). + public var contentType: UTType { get } + + /// Initialize an instance of this type with the given content type and + /// encoding quality. + /// + /// - Parameters: + /// - contentType: The image format to use when encoding images. + /// - encodingQuality: The encoding quality to use when encoding images. For + /// the lowest supported quality, pass `0.0`. For the highest supported + /// quality, pass `1.0`. + /// + /// If the target image format does not support variable-quality encoding, + /// the value of the `encodingQuality` argument is ignored. + /// + /// If `contentType` does not conform to [`UTType.image`](https://developer.apple.com/documentation/uniformtypeidentifiers/uttype-swift.struct/image), + /// the result is undefined. + public init(_ contentType: UTType, encodingQuality: Float = 1.0) +} +``` + +### Example usage + +A developer may then easily attach an image to a test by calling +`Attachment.record()` and passing the image of interest. For example, to attach +a rendering of a SwiftUI view as a PNG file: + +```swift +import Testing +import UIKit +import SwiftUI + +@MainActor @Test func `attaching a SwiftUI view as an image`() throws { + let myView: some View = ... + let image = try #require(ImageRenderer(content: myView).uiImage) + Attachment.record(image, named: "my view", as: .png) + // OR: Attachment.record(image, named: "my view.png") +} +``` + +## Source compatibility + +This change is additive only. + +## Integration with supporting tools + +None needed. + +## Future directions + +- Adding support for [`SwiftUI.Image`](https://developer.apple.com/documentation/swiftui/image) + and/or [`SwiftUI.GraphicsContext.ResolvedImage`](https://developer.apple.com/documentation/swiftui/graphicscontext/resolvedimage). + These types do not directly wrap an instance of `CGImage`. + + Since `SwiftUI.Image` conforms to [`SwiftUI.View`](https://developer.apple.com/documentation/swiftui/view), + it is possible to convert an instance of that type to an instance of `CGImage` + using [`SwiftUI.ImageRenderer`](https://developer.apple.com/documentation/swiftui/imagerenderer). + This approach is generalizable to all `SwiftUI.View`-cnforming types, and the + correct approach here may be to provide an `_AttachableViewWrapper` + type similar to the described `_AttachableImageWrapper` type. + +- Adding support for Windows image types. Windows has several generations of + imaging libraries: + + - Graphics Device Interface (GDI), which shipped with the original Windows in + 1985; + - GDI+, which was introduced with Windows XP in 2001; + - Windows Imaging Component (WIC) with Windows Vista in 2006; and + - Direct2D with Windows 7 in 2008. + + Of these libraries, only the original GDI provides a C interface that can be + directly referenced from Swift. The GDI+ interface is written in C++ and the + WIC and Direct2D interfaces are built on top of COM (a C++ abstraction layer.) + This reliance on C++ poses challenges for Swift Testing. Swift/C++ interop is + still a young technology and is not yet able to provide abstractions for + virtual C++ classes. + + None of these Windows' libraries are source compatible with Apple's Core + Graphics API, so support for any of them will require a different protocol. As + of this writing, [an experimental](https://github.com/swiftlang/swift-testing/pull/1245) + GDI- and (partially) GDI+-compatible protocol is available in Swift Testing + that allows a test author to attach an image represented by an `HBITMAP` or + `HICON` instance. Further work will be needed to make this experimental + Windows support usable with the newer libraries' image types. + +- Adding support for X11-compatible image types such as Qt's [`QImage`](https://doc.qt.io/qt-6/qimage.html) + or GTK's [`GdkPixbuf`](https://docs.gtk.org/gdk-pixbuf/class.Pixbuf.html). + We're also interested in implementing something here, but GUI-level libraries + aren't guaranteed to be present on Linux systems, so we cannot rely on their + headers or modules being accessible while building the Swift toolchain. It may + be appropriate to roll such functionality into a hypothetical `swift-x11`, + `swift-wayland`, `swift-qt`, `swift-gtk`, etc. package if one is ever created. + +- Adding support for Android's [`android.graphics.Bitmap`](https://developer.android.com/reference/android/graphics/Bitmap) + type. The Android NDK includes the [`AndroidBitmap_compress()`](https://developer.android.com/ndk/reference/group/bitmap#androidbitmap_compress) + function, but proper support for attaching an Android `Bitmap` may require a + dependency on [`swift-java`](https://github.com/swiftlang/swift-java) in some + form. Going forward, we hope to work with the new [Android Workgroup](https://www.swift.org/android-workgroup/) + to enhance Swift Testing's Android support. + +- Adding support for rendering to a PDF instead of an image. While technically + feasible using [existing](https://developer.apple.com/documentation/coregraphics/cgcontext/init(consumer:mediabox:_:)) + Core Graphics API, we haven't identified sufficient demand for this + functionality. + +## Alternatives considered + +- Doing nothing. Developers would need to write their own image conversion code. + Since this is a very common operation, it makes sense to incorporate it into + Swift Testing directly. + +- Making `CGImage` etc. conform directly to `Attachable`. Doing so would + prevent us from including sidecar data such as the desired `UTType` or + encoding quality as these types do not provide storage for that information. + As well, `NSImage` does not conform to `Sendable` and would be forced down a + code path that eagerly serializes it, which could pessimize its performance + once we introduce attachment lifetimes in a future proposal. + +- Designing a platform-agnostic solution. This would likely require adding a + dependency on an open-source image package such as [ImageMagick](https://github.com/ImageMagick/ImageMagick). + While we appreciate the value of such libraries and we want Swift Testing to + be as portable as possible, that would be a significant new dependency for the + testing library and the Swift toolchain at large. As well, we expect a typical + use case to involve an instance of `NSImage`, `CGImage`, etc. + +- Designing a solution that does not require `UTType` so as to support earlier + Apple platforms. The implementation is based on Apple's Image I/O framework + which requires a Uniform Type Identifier as input anyway, and the older + `CFString`-based interfaces we would need to use have been deprecated for + several years now. The `AttachableImageFormat` type allows us to abstract away + our platform-specific dependency on `UTType` so that, in the future, other + platforms can reuse `AttachableImageFormat` instead of implementing their own + equivalent solution. (As an example, the experimental Windows support + mentioned previously allows a developer to specify an image codec's `CLSID`.) + +- Designing a solution based around _drawing_ into a `CGContext` rather than + acquiring an instance of `CGImage`. If the proposed protocol looked like: + + ```swift + protocol AttachableByDrawing { + func draw(in context: CGContext, for attachment: Attachment) throws + } + ``` + + It would be easier to support alternative destination contexts (primarily PDF + contexts), but we would need to make a complete copy of an image in memory + before serializing it. If you start with an instance of `CGImage` or an object + that wraps an instance of `CGImage`, you can pass it directly to Image I/O. + +- Including convenience getters for additional image formats in + `AttachableImageFormat`. The set of formats we provide up-front support for is + intentionally small and limited to formats that are universally supported by + the various graphics libraries in use today. If we provided a larger set of + formats that are supported on Apple's platforms, developers may run into + difficulties porting their test code to platforms that _don't_ support those + additional formats. For example, Android's [image encoding API](https://developer.android.com/reference/android/graphics/Bitmap.CompressFormat) + only supports the PNG, JPEG, and WEBP formats. + +## Acknowledgments + +Thanks to Apple's testing teams and to the Testing Workgroup for their support +and advice on this project. diff --git a/proposals/testing/0015-image-attachments-in-swift-testing-windows.md b/proposals/testing/0015-image-attachments-in-swift-testing-windows.md new file mode 100644 index 0000000000..1b26056068 --- /dev/null +++ b/proposals/testing/0015-image-attachments-in-swift-testing-windows.md @@ -0,0 +1,393 @@ +# Image attachments in Swift Testing (Windows) + +* Proposal: [ST-0015](0015-image-attachments-in-swift-testing-windows.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Stuart Montgomery](https://github.com/stmontgomery) +- Status: **Implemented (Swift 6.3)** +* Implementation: [swiftlang/swift-testing#1245](https://github.com/swiftlang/swift-testing/pull/1245), [swiftlang/swift-testing#1254](https://github.com/swiftlang/swift-testing/pull/1254), [swiftlang/swift-testing#1333](https://github.com/swiftlang/swift-testing/pull/1333), _et al_. +* Review: ([pitch](https://forums.swift.org/t/pitch-image-attachments-in-swift-testing-windows/81871)) ([review](https://forums.swift.org/t/st-0015-image-attachments-in-swift-testing-windows/82241)) ([acceptance](https://forums.swift.org/t/accepted-st-0015-image-attachments-in-swift-testing-windows/82575)) + +## Introduction + +In [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md), +we added to Swift Testing the ability to attach images (of types `CGImage`, +`NSImage`, `UIImage`, and `CIImage`) on Apple platforms. This proposal builds on +that one to add support for attaching images on Windows. + +## Motivation + +It is frequently useful to be able to attach images to tests for engineers to +review, e.g. if a UI element is not being drawn correctly. If something doesn't +render correctly in a CI environment, for instance, it is very useful to test +authors to be able to download the failed rendering and examine it at-desk. + +In [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#integration-with-supporting-tools), +we introduced the ability to attach images to tests on Apple's platforms. Swift +Testing is a cross-platform testing library, so we should extend this +functionality to other platforms too. This proposal covers Windows in +particular. + +## Proposed solution + +We propose adding the ability to automatically encode images to standard +graphics formats such as JPEG or PNG using Windows' built-in Windows Image +Component library, similar to how we added support on Apple platforms using Core +Graphics. + +## Detailed design + +### Some background about Windows' image types + +Windows has several generations of API for representing and encoding images. The +earliest Windows API of interest to this proposal is the Graphics Device +Interface (GDI) which dates back to the earliest versions of Windows. Image +types in GDI that are of interest to us are `HBITMAP` and `HICON`, which are +_handles_ (pointers-to-pointers) and which are not reference-counted. Both types +are projected into Swift as typealiases of `UnsafeMutablePointer`. + +Windows' latest[^direct2d] graphics API is the Windows Imaging Component (WIC) +which uses types based on the Component Object Model (COM). COM types (including +those implemented in WIC) are C++ classes that inherit from `IUnknown`. + +[^direct2d]: There is an even newer API in this area, Direct2D, but it is beyond + the scope of this proposal. A developer who has an instance of e.g. + `ID2D1Bitmap` can use WIC API to convert it to a WIC bitmap source before + attaching it to a test. + +`IUnknown` is conceptually similar to Cocoa's `NSObject` class in that it +provides basic reference-counting and reflection functionality. As of this +proposal, the Swift C/C++ importer is not aware of COM classes and does not +project them into Swift as reference-counted classes. Rather, they are projected +as `UnsafeMutablePointer`, and developers who use them must manually manage +their reference counts and must use `QueryInterface()` to cast them to other COM +classes. + +In short: the types we need to support are all specializations of +`UnsafeMutablePointer`, but we do not need to support all specializations of +`UnsafeMutablePointer` unconditionally. + +### Defining a new protocol for Windows image attachments + +A new protocol is introduced for Windows, similar to the `AttachableAsCGImage` +protocol we introduced for Apple's platforms: + +```swift +/// A protocol describing images that can be converted to instances of +/// [`Attachment`](https://developer.apple.com/documentation/testing/attachment). +/// +/// Instances of types conforming to this protocol do not themselves conform to +/// [`Attachable`](https://developer.apple.com/documentation/testing/attachable). +/// Instead, the testing library provides additional initializers on [`Attachment`](https://developer.apple.com/documentation/testing/attachment) +/// that take instances of such types and handle converting them to image data when needed. +/// +/// You can attach instances of the following system-provided image types to a +/// test: +/// +/// | Platform | Supported Types | +/// |-|-| +/// | macOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) | +/// | iOS, watchOS, tvOS, and visionOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) | +/// | Windows | [`HBITMAP`](https://learn.microsoft.com/en-us/windows/win32/gdi/bitmaps), [`HICON`](https://learn.microsoft.com/en-us/windows/win32/menurc/icons), [`IWICBitmapSource`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapsource) (including its subclasses declared by Windows Imaging Component) | +/// +/// You do not generally need to add your own conformances to this protocol. If +/// you have an image in another format that needs to be attached to a test, +/// first convert it to an instance of one of the types above. +public protocol AttachableAsIWICBitmapSource: SendableMetatype { + /// Create a WIC bitmap source representing an instance of this type. + /// + /// - Returns: A pointer to a new WIC bitmap source representing this image. + /// The caller is responsible for releasing this image when done with it. + /// + /// - Throws: Any error that prevented the creation of the WIC bitmap source. + func copyAttachableIWICBitmapSource() throws -> UnsafeMutablePointer +} +``` + +Conformance to this protocol is added to `UnsafeMutablePointer` when its +`Pointee` type is one of the following types: + +- [`HBITMAP.Pointee`](https://learn.microsoft.com/en-us/windows/win32/gdi/bitmaps) +- [`HICON.Pointee`](https://learn.microsoft.com/en-us/windows/win32/menurc/icons) +- [`IWICBitmapSource`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapsource) + (including its subclasses declared by Windows Imaging Component) + +> [!NOTE] +> The list of conforming types may be extended in the future. The Testing +> Workgroup will determine if additional Swift Evolution reviews are needed. + +A type in Swift can only conform to a protocol with **one** set of constraints, +so we need a helper protocol in order to make `UnsafeMutablePointer` +conditionally conform for all of the above types. This protocol must be `public` +so that Swift Testing can refer to it in API, but it is an implementation detail +and not part of this proposal: + +```swift +public protocol _AttachableByAddressAsIWICBitmapSource {} + +extension HBITMAP.Pointee: _AttachableByAddressAsIWICBitmapSource {} +extension HICON.Pointee: _AttachableByAddressAsIWICBitmapSource {} +extension IWICBitmapSource: _AttachableByAddressAsIWICBitmapSource {} + +extension UnsafeMutablePointer: AttachableAsIWICBitmapSource + where Pointee: _AttachableByAddressAsIWICBitmapSource {} +``` + +See the **Future directions** section (specifically the point about COM and C++ +interop) for more information on why the helper protocol is excluded from this +proposal. + +### Attaching a conforming image + +New overloads of `Attachment.init()` and `Attachment.record()` are provided: + +```swift +extension Attachment { + /// Initialize an instance of this type that encloses the given image. + /// + /// - Parameters: + /// - image: A pointer to the value that will be attached to the output of + /// the test run. + /// - preferredName: The preferred name of the attachment when writing it + /// to a test report or to disk. If `nil`, the testing library attempts + /// to derive a reasonable filename for the attached value. + /// - imageFormat: The image format with which to encode `image`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// You can attach instances of the following system-provided image types to a + /// test: + /// + /// | Platform | Supported Types | + /// |-|-| + /// | macOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) | + /// | iOS, watchOS, tvOS, and visionOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) | + /// | Windows | [`HBITMAP`](https://learn.microsoft.com/en-us/windows/win32/gdi/bitmaps), [`HICON`](https://learn.microsoft.com/en-us/windows/win32/menurc/icons), [`IWICBitmapSource`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapsource) (including its subclasses declared by Windows Imaging Component) | + /// + /// The testing library uses the image format specified by `imageFormat`. Pass + /// `nil` to let the testing library decide which image format to use. If you + /// pass `nil`, then the image format that the testing library uses depends on + /// the path extension you specify in `preferredName`, if any. If you do not + /// specify a path extension, or if the path extension you specify doesn't + /// correspond to an image format the operating system knows how to write, the + /// testing library selects an appropriate image format for you. + public init( + _ image: T, + named preferredName: String? = nil, + as imageFormat: AttachableImageFormat? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) where T: AttachableAsIWICBitmapSource, AttachableValue == _AttachableImageWrapper + + /// Attach an image to the current test. + /// + /// - Parameters: + /// - image: The value to attach. + /// - preferredName: The preferred name of the attachment when writing it + /// to a test report or to disk. If `nil`, the testing library attempts + /// to derive a reasonable filename for the attached value. + /// - imageFormat: The image format with which to encode `image`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// This function creates a new instance of ``Attachment`` wrapping `image` + /// and immediately attaches it to the current test. You can attach instances + /// of the following system-provided image types to a test: + /// + /// | Platform | Supported Types | + /// |-|-| + /// | macOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) | + /// | iOS, watchOS, tvOS, and visionOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) | + /// | Windows | [`HBITMAP`](https://learn.microsoft.com/en-us/windows/win32/gdi/bitmaps), [`HICON`](https://learn.microsoft.com/en-us/windows/win32/menurc/icons), [`IWICBitmapSource`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapsource) (including its subclasses declared by Windows Imaging Component) | + /// + /// The testing library uses the image format specified by `imageFormat`. Pass + /// `nil` to let the testing library decide which image format to use. If you + /// pass `nil`, then the image format that the testing library uses depends on + /// the path extension you specify in `preferredName`, if any. If you do not + /// specify a path extension, or if the path extension you specify doesn't + /// correspond to an image format the operating system knows how to write, the + /// testing library selects an appropriate image format for you. + public static func record( + _ image: T, + named preferredName: String? = nil, + as imageFormat: AttachableImageFormat? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) where T: AttachableAsIWICBitmapSource, AttachableValue == _AttachableImageWrapper +} +``` + +> [!NOTE] +> `_AttachableImageWrapper` was described in [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#attaching-a-conforming-image). +> The only difference on Windows is that its associated `Image` type is +> constrained to `AttachableAsIWICBitmapSource` instead of `AttachableAsCGImage`. + +### Specifying image formats + +As on Apple platforms, a test author can specify the image format to use with +`AttachableImageFormat`. See [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#specifying-image-formats) +for more information about that type. + +Windows does not use Uniform Type Identifiers, so those `AttachableImageFormat` +members that use `UTType` are not available here. Instead, Windows uses a +variety of COM classes that implement codecs for different image formats. +Conveniences over those COM classes' `CLSID` values are provided: + +```swift +extension AttachableImageFormat { + /// The `CLSID` value of the Windows Imaging Component (WIC) encoder class + /// that corresponds to this image format. + /// + /// For example, if this image format equals ``png``, the value of this + /// property equals [`CLSID_WICPngEncoder`](https://learn.microsoft.com/en-us/windows/win32/wic/-wic-guids-clsids#wic-guids-and-clsids). + public var encoderCLSID: CLSID { get } + + /// Construct an instance of this type with the `CLSID` value of a Windows + /// Imaging Component (WIC) encoder class and the desired encoding quality. + /// + /// - Parameters: + /// - encoderCLSID: The `CLSID` value of the Windows Imaging Component + /// encoder class to use when encoding images. + /// - encodingQuality: The encoding quality to use when encoding images. For + /// the lowest supported quality, pass `0.0`. For the highest supported + /// quality, pass `1.0`. + /// + /// If the target image encoder does not support variable-quality encoding, + /// the value of the `encodingQuality` argument is ignored. + /// + /// If `clsid` does not represent an image encoder class supported by WIC, the + /// result is undefined. For a list of image encoder classes supported by WIC, + /// see the documentation for the [`IWICBitmapEncoder`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapencoder) + /// class. + public init(encoderCLSID: CLSID, encodingQuality: Float = 1.0) +} +``` + +For convenience, an initializer is provided that takes a path extension and +tries to map it to the appropriate codec's `CLSID` value: + +```swift +extension AttachableImageFormat { + /// Construct an instance of this type with the given path extension and + /// encoding quality. + /// + /// - Parameters: + /// - pathExtension: A path extension corresponding to the image format to + /// use when encoding images. + /// - encodingQuality: The encoding quality to use when encoding images. For + /// the lowest supported quality, pass `0.0`. For the highest supported + /// quality, pass `1.0`. + /// + /// If the target image format does not support variable-quality encoding, + /// the value of the `encodingQuality` argument is ignored. + /// + /// If `pathExtension` does not correspond to a recognized image format, this + /// initializer returns `nil`: + /// + /// - On Apple platforms, the content type corresponding to `pathExtension` + /// must conform to [`UTType.image`](https://developer.apple.com/documentation/uniformtypeidentifiers/uttype-swift.struct/image). + /// - On Windows, there must be a corresponding subclass of [`IWICBitmapEncoder`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapencoder) + /// registered with Windows Imaging Component. + public init?(pathExtension: String, encodingQuality: Float = 1.0) +} +``` + +For consistency, `init(pathExtension:encodingQuality:)` is provided on Apple +platforms too. (This is the only part of this proposal that affects platforms +other than Windows.) + +### Example usage + +A developer may then easily attach an image to a test by calling +`Attachment.record()` and passing the image of interest. For example, to attach +an icon to a test as a PNG file: + +```swift +import Testing +import WinSDK + +@MainActor @Test func `attaching an icon`() throws { + let hIcon: HICON = ... + defer { + DestroyIcon(hIcon) + } + Attachment.record(hIcon, named: "my icon", as: .png) + // OR: Attachment.record(hIcon, named: "my icon.png") +} +``` + +## Source compatibility + +This change is additive only. + +## Integration with supporting tools + +Tools that handle attachments created by Swift Testing will gain support for +this functionality automatically and do not need to make any changes. + +## Future directions + +- Adding support for projecting COM classes as foreign-reference-counted Swift + classes. The C++ interop team is interested in implementing this feature, but + it is beyond the scope of this proposal. **If this feature is implemented in + the future**, it will cause types like `IWICBitmapSource` to be projected + directly into Swift instead of as `UnsafeMutablePointer` specializations. This + would be a source-breaking change for Swift Testing, but it would make COM + classes much easier to use in Swift. + + In the context of this proposal, `IWICBitmapSource` would be able to directly + conform to `AttachableAsIWICBitmapSource` and we would no longer need the + `_AttachableByAddressAsIWICBitmapSource` helper protocol. The + `AttachableAsIWICBitmapSource` protocol's `copyAttachableIWICBitmapSource()` + requirement would likely change to a property (i.e. + `var attachableIWICBitmapSource: IWICBitmapSource { get throws }`) as it would + be able to participate in Swift's automatic reference counting. + + The Swift team is tracking COM interop with [swiftlang/swift#84056](https://github.com/swiftlang/swift/issues/84056). + +- Adding support for managed (.NET or C#) image types. Support for managed types + on Windows would first require a new Swift/.NET or Swift/C# interop feature + and is therefore beyond the scope of this proposal. + +- Adding support for WinRT image types. WinRT is a thin wrapper around COM and + has C++ and .NET projections, neither of which are readily accessible from + Swift. It may be possible to add support for WinRT image types if COM interop + is implemented. + +- Adding support for other platforms. See [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#future-directions) + for further discussion about supporting additional platforms. + +## Alternatives considered + +- Doing nothing. We have already added support for attaching images on Apple's + platforms, and Swift Testing is meant to be a cross-platform library, so we + should make a best effort to provide the same functionality on Windows and, + eventually, other platforms. + +- Using more Windows-/COM-like terminology and spelling, e.g. + `CloneAttachableBitmapSource()` instead of `copyAttachableIWICBitmapSource()`. + Swift API should follow Swift API guidelines, even when extending types and + calling functions implemented under other standards. + +- Making `IWICBitmapSource` conform directly to `Attachable`. As with `CGImage` + in [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#alternatives-considered), + this would prevent us from including additional information (i.e. an instance + of `AttachableImageFormat`). Further, it would be difficult to correctly + manage the lifetime of Windows' 'image objects as they do not participate in + automatic reference counting. + +- Using the GDI+ type [`Gdiplus.Image`](https://learn.microsoft.com/en-us/windows/win32/api/gdiplusheaders/nl-gdiplusheaders-image) + as our currency type instead of `IWICBitmapSource`. This type is a C++ class + but is not a COM class, and so it is not projected into Swift except as + `OpaquePointer` which makes it unsafe to extend it with protocol conformances. + As well, GDI+ is a much older API than WIC and is not recommended by Microsoft + for new development. + +- Designing a platform-agnostic solution. This would likely require adding a + dependency on an open-source image package such as [ImageMagick](https://github.com/ImageMagick/ImageMagick). + Such a library would be a significant new dependency for the testing library + and the Swift toolchain at large. + +## Acknowledgments + +Thank you to @compnerd and the C++ interop team for their help with Windows and +the COM API. diff --git a/proposals/testing/0016-test-cancellation.md b/proposals/testing/0016-test-cancellation.md new file mode 100644 index 0000000000..51b10938be --- /dev/null +++ b/proposals/testing/0016-test-cancellation.md @@ -0,0 +1,333 @@ +# Test cancellation + +* Proposal: [ST-0016](0016-test-cancellation.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Maarten Engels](https://github.com/maartene) +* Status: **Active review (October 23...November 5, 2025)** +* Bug: [swiftlang/swift-testing#120](https://github.com/swiftlang/swift-testing/issues/120) +* Implementation: [swiftlang/swift-testing#1284](https://github.com/swiftlang/swift-testing/pull/1284) +* Review: ([pitch](https://forums.swift.org/t/pitch-test-cancellation/81847)) ([review](https://forums.swift.org/t/st-0016-test-cancellation/82817)) + +## Introduction + +Swift Testing provides the ability to conditionally skip a test before it runs +using the [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:)), +[`.disabled(if:)`](https://developer.apple.com/documentation/testing/trait/disabled(if:_:sourcelocation:)), +etc. family of traits: + +```swift +@Test(.disabled(if: Species.all(in: .dinosauria).isEmpty) +func `Are all dinosaurs extinct?`() { + // ... +} +``` + +This proposal extends that feature to allow cancelling a test after it has +started but before it has ended. + +## Motivation + +We have received feedback from a number of developers indicating that their +tests have constraints that can only be checked after a test has started, and +they would like the ability to end a test early and see that state change +reflected in their development tools. + +To date, we have not provided an API for ending a test's execution early because +we want to encourage developers to use the [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:)) +_et al._ trait. This trait can be evaluated early and lets Swift Testing plan a +test run more efficiently. However, we recognize that these traits aren't +sufficient. Some test constraints are dependent on data that isn't available +until the test starts, while others only apply to specific test cases in a +parameterized test function. + +## Proposed solution + +A static `cancel()` function is added to the [`Test`](https://developer.apple.com/documentation/testing/test) +type. When a test author calls this function from within the body of a test (or +from within the implementation of a trait, e.g. from [`prepare(for:)`](https://developer.apple.com/documentation/testing/trait/prepare(for:))), +Swift Testing cancels the currently-running test. + +Parameterized tests are special-cased: if the currently-running test is +parameterized and you call `cancel()`, only the current test case is cancelled +and other test cases in the same test continue to run. + +### Relationship between tasks and tests + +Each test runs in its own task during a test run, and each test case in a test +also runs in its own task. Cancelling the current task from within the body of a +test will, therefore, cancel the current test case, but not the current test: + +```swift +@Test(arguments: Species.all(in: .dinosauria)) +func `Are all dinosaurs extinct?`(_ species: Species) { + if species.in(.aves) { + // Birds aren't extinct (I hope) + withUnsafeCurrentTask { $0?.cancel() } + return + } + // ... +} +``` + +Using [`withUnsafeCurrentTask(body:)`](https://developer.apple.com/documentation/swift/withunsafecurrenttask(body:)-6gvhl) +here is not ideal. It's not clear that the intent is to cancel the test case, +and [`UnsafeCurrentTask`](https://developer.apple.com/documentation/swift/unsafecurrenttask) +is, unsurprisingly, an unsafe interface. + +> [!NOTE] +> The version of Swift Testing included with Swift 6.2 does not correctly handle +> task cancellation under all conditions. See [swiftlang/swift-testing#1289](https://github.com/swiftlang/swift-testing/issues/1289). + +## Detailed design + +A new static function is added to [`Test`](https://developer.apple.com/documentation/testing/test): + +```swift +extension Test { + /// Cancel the current test or test case. + /// + /// - Parameters: + /// - comment: A comment describing why you are cancelling the test or test + /// case. + /// - sourceLocation: The source location to which the testing library will + /// attribute the cancellation. + /// + /// - Throws: An error indicating that the current test or test case has been + /// cancelled. + /// + /// The testing library runs each test and each test case in its own task. + /// When you call this function, the testing library cancels the task + /// associated with the current test: + /// + /// ```swift + /// @Test func `Food truck is well-stocked`() throws { + /// guard businessHours.contains(.now) else { + /// try Test.cancel("We're off the clock.") + /// } + /// // ... + /// } + /// ``` + /// + /// If the current test is a parameterized test function, this function + /// instead cancels the current test case. Other test cases in the test + /// function are not affected. + /// + /// If the current test is a suite, the testing library cancels all of its + /// pending and running tests. + /// + /// If you have already cancelled the current test or if it has already + /// finished running, this function throws an error to indicate that the + /// current test has been cancelled, but does not attempt to cancel the test a + /// second time. + /// + /// - Important: If the current task is not associated with a test (for + /// example, because it was created with [`Task.detached(name:priority:operation:)`](https://developer.apple.com/documentation/swift/task/detached(name:priority:operation:)-795w1)) + /// this function records an issue and cancels the current task. + public static func cancel(_ comment: Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation) throws -> Never +} +``` + +Cancelling a test or test case implicitly cancels its associated task (and any +child tasks thereof) as if [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()) +were called on that task. + +### Throwing semantics + +Unlike [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()), +this function always throws an error instead of returning. This simplifies +control flow when a test is cancelled; instead of having to write: + +```swift +if condition { + theTask.cancel() + return +} +``` + +A test author need only write: + +```swift +if condition { + try Test.cancel() +} +``` + +The errors this function throws are of a type internal to Swift Testing that is +semantically similar to [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror) +but carries additional information (namely the `comment` and `sourceLocation` +arguments to `cancel(_:sourceLocation:)`) that Swift Testing can present to the +user. When Swift Testing catches an error of this type[^cancellationErrorToo], +it does not record an issue for the current test or test case. + +[^cancellationErrorToo]: Swift Testing also catches errors of type + [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror) + if the current task has been cancelled. If the current task has not been + cancelled, errors of this type are still recorded as issues. + +Suppressing these errors with `do`/`catch` or `try?` does not uncancel a test, +test case, or task, but can be useful if you have additional local work you need +to do before the test or test case ends. + +### Support for CancellationError + +Cancelling a test's or test case's associated task is equivalent to cancelling +the test or test case. Hence, if a test or test case throws an instance of +[`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror) +_and_ the current task has been cancelled, it is treated as if the test or test +case were cancelled. + +### Interaction with recorded issues + +If you cancel a test or test case that has previously recorded an issue, that +issue is not overridden or nullified. In particular, if the test or test case +has already recorded an issue of severity **error** when you call +`cancel(_:sourceLocation:)`, the test or test case will still fail. + +### Example usage + +To cancel the current test case and let other test cases run: + +```swift +@Test(arguments: Species.all(in: .dinosauria)) +func `Are all dinosaurs extinct?`(_ species: Species) throws { + if species.in(.aves) { + try Test.cancel("\(species) is birds!") + } + // ... +} +``` + +## Source compatibility + +This change is additive only. + +## Integration with supporting tools + +The JSON event stream Swift Testing provides is updated to include two new event +kinds: + +```diff + ::= "runStarted" | "testStarted" | "testCaseStarted" | + "issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" | +- "runEnded" | "valueAttached" ++ "runEnded" | "valueAttached" | "testCancelled" | "testCaseCancelled" +``` + +And new fields are added to event records to represent the comment and source +location passed to `cancel(_:sourceLocation:)`: + +```diff + ::= { + "kind": , + "instant": , ; when the event occurred + ["issue": ,] ; the recorded issue (if "kind" is "issueRecorded") + ["attachment": ,] ; the attachment (if kind is "valueAttached") + "messages": , + ["testID": ,] ++ ["comments": ,] ++ ["sourceLocation": ,] + } +``` + +These new fields are populated for the new event kinds as well as other event +kinds that can populate them. + +An event of kind `"testCancelled"` is posted any time an entire test function or +test suite is cancelled. An event of kind `"testCaseCancelled"` is posted any +time a single test case is cancelled. + +These new event kinds and fields will be included in the next revision of the +JSON schema (currently expected to be schema version `"6.3"`). + +## Future directions + +- Adding a corresponding `Test.checkCancellation()` function and/or + `Test.isCancelled` static property. These are beyond the scope of this + proposal, primarily because [`Task.isCancelled`](https://developer.apple.com/documentation/swift/task/iscancelled-swift.type.property) + and [`Task.checkCancellation()`](https://developer.apple.com/documentation/swift/task/checkcancellation()) + already work in a test. + +- Adding a `Test.Case.cancelAll()` interface that explicitly cancels all test + cases in a test function. We want to further evaluate the use cases and + semantics for such a function before we commit to introducing it as API. + +## Alternatives considered + +- Doing nothing. While we do want test authors to use [`.enabled(if:)`](https://developer.apple.com/documentation/testing/trait/enabled(if:_:sourcelocation:)) + _et al._ trait, we recognize it does not provide the full set of functionality + that test authors need. + +- Ignoring task cancellation or treating [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror) + as a normal error even when the current task has been cancelled. It is not + possible for Swift Testing to outright ignore task cancellation, and a + [`CancellationError`](https://developer.apple.com/documentation/swift/cancellationerror) + instance thrown from [`Task.checkCancellation()`](https://developer.apple.com/documentation/swift/task/checkcancellation()) + is not really a test issue but rather a manifestation of control flow. + +- Using the [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct) + type from XCTest. Interoperation with XCTest is an area of exploration for us, + but core functionality of Swift Testing needs to be usable without also + importing XCTest. + +- Spelling the function `static func cancel(_:sourceLocation:) -> some Error` + and requiring it be called as `throw Test.cancel()`. This is closer to how + the [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct) + type is used in XCTest. We have received indirect feedback about [`XCTSkip`](https://developer.apple.com/documentation/xctest/xctskip-swift.struct) + indicating its usage is unclear, and sometimes need to help developers who + have written: + + ```swift + if x { + XCTSkip() + } + ``` + + And don't understand why it has failed to stop the test. More broadly, it is + not common practice in Swift for a function to return an error that the caller + is then responsible for throwing. + +- Providing additional `cancel(if:)` and `cancel(unless:)` functions. In + Objective-C, XCTest provides the [`XCTSkipIf()`](https://developer.apple.com/documentation/xctest/xctskipif) + and [`XCTSkipUnless()`](https://developer.apple.com/documentation/xctest/xctskipunless) + macros which capture their condition arguments as strings for display to the + test author. This functionality is not available in Swift, but XCTest's Swift + interface provides equivalent throwing functions as conveniences. We could + provide these functions (without any sort of string-capturing ability) too, + but they provide little additional clarity above an `if` or `guard` statement. + +- Implementing cancellation using Swift macros so we can capture an `if` or + `unless` argument as a string. A macro for this feature is probably the wrong + tradeoff between compile-time magic and technical debt. + +- Relying solely on [`Task.cancel()`](https://developer.apple.com/documentation/swift/task/cancel()). + Ignoring the interplay between tests and test cases, this approach is + difficult for test authors to use because the current [`Task`](https://developer.apple.com/documentation/swift/task) + instance isn't visible _within_ that task. Instead, a test author would need + to use [`withUnsafeCurrentTask(body:)`](https://developer.apple.com/documentation/swift/withunsafecurrenttask(body:)-6gvhl) + to get a temporary reference to the task and cancel _that_ value. We would + also not have the ability to include a comment and source location information + in the test's console output or an IDE's test result interface. + + With that said, [`UnsafeCurrentTask.cancel()`](https://developer.apple.com/documentation/swift/unsafecurrenttask/cancel()) + _does_ cancel the test or test case associated with the current task. + +- Providing both `Test.cancel()` and `Test.Case.cancel()`, with `Test.cancel()` + _always_ cancelling the current test in its entirety and `Test.Case.cancel()` + _always_ cancelling the current test _case_ and leaving other test cases + alone. + + We have received pitch feedback from multiple test authors indicating that + they could introduce subtle bugs while refactoring test functions into + parameterized test functions. If they had written `Test.cancel()` and forgot + to change the call to `Test.Case.cancel()` when refactoring, they could + introduce a bug causing none of their test cases to run (because the entire + test is cancelled instead of just the current test case). + +## Acknowledgments + +Thanks team! + +Thanks Arthur! That's right, dinosaurs _do_ say "roar!" + +And thanks to [@allevato](https://github.com/allevato) for nerd-sniping me into +writing this proposal. diff --git a/proposals/testing/0017-image-attachment-consolidation.md b/proposals/testing/0017-image-attachment-consolidation.md new file mode 100644 index 0000000000..a0b4975777 --- /dev/null +++ b/proposals/testing/0017-image-attachment-consolidation.md @@ -0,0 +1,172 @@ +# Consolidate Swift Testing's image attachments API across platforms + +* Proposal: [ST-0017](0017-image-attachment-consolidation.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Rachel Brindle](https://github.com/younata) +* Status: **Accepted** +* Implementation: [swiftlang/swift-testing#1359](https://github.com/swiftlang/swift-testing/pull/1359) +* Review: ([pitch](https://forums.swift.org/t/pitch-adjustments-to-image-attachments-in-swift-testing/82581), [review](https://forums.swift.org/t/st-0017-consolidate-swift-testing-s-image-attachments-api-across-platforms/82815), [acceptance](https://forums.swift.org/t/accepted-st-0017-consolidate-swift-testing-s-image-attachments-api-across-platforms/83045)) + +## Introduction + +This proposal includes a small number of adjustments to the API surface of Swift +Testing's image attachments feature introduced in [ST-0014](0014-image-attachments-in-swift-testing-apple-platforms.md) +and [ST-0015](0015-image-attachments-in-swift-testing-windows.md). + +## Motivation + +These changes will help to align the platform-specific interfaces of the feature +more closely. + +## Proposed solution + +The `AttachableAsCGImage` and `AttachableAsIWICBitmapSource` protocols are +combined into a single protocol, `AttachableAsImage` with adjusted protocol +requirements; a change is made to `AttachableImageFormat` to more closely +align its interface between Darwin and Windows; `AttachableImageFormat` is made +to conform to `Equatable` and `Hashable`; and an additional property is added to +`Attachment` to query its image format. + +## Detailed design + +The following changes are proposed: + +### Combining AttachableAsCGImage and AttachableAsIWICBitmapSource + +The `AttachableAsCGImage` and `AttachableAsIWICBitmapSource` protocols are +combined into a single protocol, `AttachableAsImage`. + +These platform-specific requirements are removed: + +```diff +- var attachableCGImage: CGImage { get throws } +- func copyAttachableIWICBitmapSource() throws -> UnsafeMutablePointer +``` + +They are replaced with a new requirement that encapsulates the image encoding +operation. This requirement is implemented by the CoreGraphics and WinSDK +overlays and is made publicly available for test authors who wish to declare +additional conformances to this protocol for types that are not based on +`CGImage` or `IWICBitmapSource`: + +```swift +public protocol AttachableAsImage { + // ... + + /// Encode a representation of this image in a given image format. + /// + /// - Parameters: + /// - imageFormat: The image format to use when encoding this image. + /// - body: A function to call. A temporary buffer containing a data + /// representation of this instance is passed to it. + /// + /// - Returns: Whatever is returned by `body`. + /// + /// - Throws: Whatever is thrown by `body`, or any error that prevented the + /// creation of the buffer. + /// + /// The testing library uses this function when saving an image as an + /// attachment. The implementation should use `imageFormat` to determine what + /// encoder to use. + borrowing func withUnsafeBytes(as imageFormat: AttachableImageFormat, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R +} +``` + +If a developer has an image type that should conform to `AttachableAsImage` and +wraps an instance of `CGImage` or `IWICBitmapSource`, it is straightforward for +them to delegate to that object. For example: + +```swift +import Testing +import CoreGraphics + +struct MyImage { + var cgImage: CGImage + // ... +} + +extension MyImage: AttachableAsImage { + func withUnsafeBytes(as imageFormat: AttachableImageFormat, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + try cgImage.withUnsafeBytes(as: imageFormat, body) + } +} +``` + +### Adjusting AttachableImageFormat + +The following Apple-specific `AttachableImageFormat` initializer is renamed so +that its first argument has an explicit label: + +```diff + public struct AttachableImageFormat { + // ... +- public init(_ contentType: UTType, encodingQuality: Float = 1.0) ++ public init(contentType: UTType, encodingQuality: Float = 1.0) + } +``` + +This change makes the type's interface more consistent between Darwin and +Windows (where it has an `init(encoderCLSID:encodingQuality:)` initializer.) + +As well, conformances to `Equatable`, `Hashable`, `CustomStringConvertible`, and +`CustomDebugStringConvertible` are added: + +```swift +extension AttachableImageFormat: Equatable, Hashable {} +extension AttachableImageFormat: CustomStringConvertible, CustomDebugStringConvertible {} +``` + +Conformance to `Equatable` is necessary to correctly implement the +`withUnsafeBytes(as:_:)` protocol requirement mentioned above, and conformance +to `Hashable` is generally useful and straightforward to implement. Conformance +to `CustomStringConvertible` and `CustomDebugStringConvertible` allows for +better diagnostic output (especially if an encoding failure occurs.) + +### Adding an imageFormat property to Attachment + +The following property is added to `Attachment` when the attachable value is an +image: + +```swift +extension Attachment where AttachableValue: AttachableWrapper, + AttachableValue.Wrapped: AttachableAsImage { + /// The image format to use when encoding the represented image. + public var imageFormat: AttachableImageFormat? { get } +} +``` + +## Source compatibility + +These changes are breaking for anyone who has created a type that conforms to +either `AttachableAsCGImage` or `AttachableAsIWICBitmapSource`, or anyone who +has adopted `AttachableImageFormat.init(_:encodingQuality:)`. + +This feature is new in Swift 6.3 and has not shipped to developers outside of +nightly toolchain builds. As such, we feel confident that any real-world impact +to developers will be both minimal and manageable. + +## Integration with supporting tools + +No changes. + +## Future directions + +- Migrating from `UnsafeRawBufferPointer` to `RawSpan`. We shipped the initial + attachments feature using `UnsafeRawBufferPointer` before `RawSpan` was + available and, in particular, before it was back-deployed to earlier Apple + platforms. We want the attachments API to consistently use the same types at + all layers, so adoption of `RawSpan` only in the image attachments layer is a + non-goal. In the future, we may wish to deprecate the existing APIs that use + `UnsafeRawBufferPointer` and introduce replacements that use `RawSpan`. + +## Alternatives considered + +- Leaving the two protocols separate. Combining them allows us to lower more + code into the main Swift Testing library and improves our ability to generate + DocC documentation, while also simplifying the story for developers who want + to use this feature across platforms. + +## Acknowledgments + +Thanks to my colleagues for their feedback on the image attachments feature and +to the Swift community for putting up with the churn! diff --git a/releases/swift-2_2.md b/releases/swift-2_2.md new file mode 100644 index 0000000000..509795952a --- /dev/null +++ b/releases/swift-2_2.md @@ -0,0 +1,23 @@ +# Swift 2.2 - Released on March 21, 2016 + +[This release](https://swift.org/blog/swift-2-2-released/) focused on fixing +bugs, improving quality-of-implementation (QoI) +with better warnings and diagnostics, improving compile times, and improving +performance. It put some finishing touches on features introduced in Swift 2.0, +and included some small additive features that don't break Swift code or +fundamentally change the way Swift is used. As a step toward Swift 3, it +introduced warnings about upcoming source-incompatible changes in Swift 3 +so that users can begin migrating their code sooner. + +Aside from warnings, a major goal of this release was to be as source compatible +as practical with Swift 2.0. + +## Evolution proposals included in Swift 2.2 + +* [SE-0001: Allow (most) keywords as argument labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0001-keywords-as-argument-labels.md) +* [SE-0015: Tuple comparison operators](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0015-tuple-comparison-operators.md) +* [SE-0014: Constraining `AnySequence.init`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0014-constrained-AnySequence.md) +* [SE-0011: Replace `typealias` keyword with `associatedtype` for associated type declarations](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0011-replace-typealias-associated.md) +* [SE-0021: Naming Functions with Argument Labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0021-generalized-naming.md) +* [SE-0022: Referencing the Objective-C selector of a method](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0022-objc-selectors.md) +* [SE-0020: Swift Language Version Build Configuration](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0020-if-swift-version.md) diff --git a/releases/swift-3_0.md b/releases/swift-3_0.md new file mode 100644 index 0000000000..ece2b1d352 --- /dev/null +++ b/releases/swift-3_0.md @@ -0,0 +1,143 @@ +# Swift 3.0 - Released on September 13, 2016 + +Swift 3 focused on solidifying and maturing the Swift language and +development experience. It focused on several areas: + +* **API design guidelines**: The way in which Swift is used in popular + libraries has almost as much of an effect on the character of Swift + code as the Swift language itself. The [API naming and design + guidelines](https://swift.org/documentation/api-design-guidelines/) are a + carefully crafted set of guidelines for building great Swift APIs. + +* **Automatic application of naming guidelines to imported Objective-C APIs**: + When importing Objective-C APIs, the Swift 3 compiler + [automatically maps](../proposals/0005-objective-c-name-translation.md) methods + into the new Swift 3 naming guidelines, and provides a number of Objective-C + features to control and adapt this importing. + +* **Adoption of naming guidelines in key APIs**: The Swift Standard Library has + been significantly overhauled to embrace these guidelines, and key libraries + like [Foundation](../proposals/0069-swift-mutability-for-foundation.md) and + [libdispatch](../proposals/0088-libdispatch-for-swift3.md) have seen major + updates, which provide the consistent development experience we seek. + +* **Swiftification of imported Objective-C APIs**: Beyond the naming guidelines, + Swift 3 provides an improved experience for working with Objective-C APIs. + This includes importing + [Objective-C generic classes](../proposals/0057-importing-objc-generics.md), + providing the ability to [import C APIs](../proposals/0044-import-as-member.md) + into an "Object Oriented" style, much nicer + [imported string enums](../proposals/0033-import-objc-constants.md), safer + syntax to work with [selectors](../proposals/0022-objc-selectors.md) and + [keypaths](../proposals/0062-objc-keypaths.md), etc. + +* **Focus and refine the language**: Since Swift 3 is the last release to make + major source breaking changes, it is also the right release to reevaluate the + syntax and semantics of the core language. This means that some obscure or + problematic features will be removed, we focus on improving consistency of + syntax in many small ways (e.g. by + [revising handling of parameter labels](../proposals/0046-first-label.md), and + focus on forward looking improvements to the type system. This serves the + overall goal of making Swift a simpler, more predictable, and more consistent + language over the long term. + +Swift 3 is the first release to enable +broad scale adoption across multiple platforms, including significant +functionality in the [Swift core libraries](https://swift.org/core-libraries/) +(Foundation, libdispatch, XCTest, etc), portability to a number of platforms including Linux/x86, Raspberry Pi, and Android, and the [Swift package manager](https://swift.org/package-manager/) to easily manage the distribution of Swift source code. + +Finally, Swift 3 also includes a mix of relatively small but important additions +to the language and standard library that make solving common problems easier and +make everything feel nicer. + +## Evolution proposals included in Swift 3.0 + +* [SE-0002: Removing currying `func` declaration syntax](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0002-remove-currying.md) +* [SE-0003: Removing `var` from Function Parameters](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0003-remove-var-parameters.md) +* [SE-0004: Remove the `++` and `--` operators](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0004-remove-pre-post-inc-decrement.md) +* [SE-0005: Better Translation of Objective-C APIs Into Swift](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0005-objective-c-name-translation.md) +* [SE-0006: Apply API Guidelines to the Standard Library](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0006-apply-api-guidelines-to-the-standard-library.md) +* [SE-0007: Remove C-style for-loops with conditions and incrementers](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0007-remove-c-style-for-loops.md) +* [SE-0008: Add a Lazy flatMap for Sequences of Optionals](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0008-lazy-flatmap-for-optionals.md) +* [SE-0016: Adding initializers to Int and UInt to convert from UnsafePointer and UnsafeMutablePointer](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md) +* [SE-0017: Change `Unmanaged` to use `UnsafePointer`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0017-convert-unmanaged-to-use-unsafepointer.md) +* [SE-0019: Swift Testing](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0019-package-manager-testing.md) +* [SE-0023: API Design Guidelines](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0023-api-guidelines.md) +* [SE-0025: Scoped Access Level](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0025-scoped-access-level.md) +* [SE-0029: Remove implicit tuple splat behavior from function applications](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0029-remove-implicit-tuple-splat.md) +* [SE-0031: Adjusting inout Declarations for Type Decoration](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0031-adjusting-inout-declarations.md) +* [SE-0032: Add `first(where:)` method to `SequenceType`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0032-sequencetype-find.md) +* [SE-0033: Import Objective-C Constants as Swift Types](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0033-import-objc-constants.md) +* [SE-0034: Disambiguating Line Control Statements from Debugging Identifiers](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0034-disambiguating-line.md) +* [SE-0035: Limiting `inout` capture to `@noescape` contexts](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0035-limit-inout-capture.md) +* [SE-0036: Requiring Leading Dot Prefixes for Enum Instance Member Implementations](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0036-enum-dot.md) +* [SE-0037: Clarify interaction between comments & operators](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0037-clarify-comments-and-operators.md) +* [SE-0038: Package Manager C Language Target Support](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0038-swiftpm-c-language-targets.md) +* [SE-0039: Modernizing Playground Literals](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0039-playgroundliterals.md) +* [SE-0040: Replacing Equal Signs with Colons For Attribute Arguments](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0040-attributecolons.md) +* [SE-0043: Declare variables in 'case' labels with multiple patterns](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md) +* [SE-0044: Import as Member](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0044-import-as-member.md) +* [SE-0046: Establish consistent label behavior across all parameters including first labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0046-first-label.md) +* [SE-0047: Defaulting non-Void functions so they warn on unused results](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0047-nonvoid-warn.md) +* [SE-0048: Generic Type Aliases](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0048-generic-typealias.md) +* [SE-0049: Move @noescape and @autoclosure to be type attributes](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0049-noescape-autoclosure-type-attrs.md) +* [SE-0052: Change IteratorType post-nil guarantee](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0052-iterator-post-nil-guarantee.md) +* [SE-0053: Remove explicit use of `let` from Function Parameters](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0053-remove-let-from-function-parameters.md) +* [SE-0054: Abolish `ImplicitlyUnwrappedOptional` type](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0054-abolish-iuo.md) +* [SE-0055: Make unsafe pointer nullability explicit using Optional](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0055-optional-unsafe-pointers.md) +* [SE-0057: Importing Objective-C Lightweight Generics](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0057-importing-objc-generics.md) +* [SE-0059: Update API Naming Guidelines and Rewrite Set APIs Accordingly](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0059-updated-set-apis.md) +* [SE-0060: Enforcing order of defaulted parameters](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0060-defaulted-parameter-order.md) +* [SE-0061: Add Generic Result and Error Handling to autoreleasepool()](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0061-autoreleasepool-signature.md) +* [SE-0062: Referencing Objective-C key-paths](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0062-objc-keypaths.md) +* [SE-0063: SwiftPM System Module Search Paths](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0063-swiftpm-system-module-search-paths.md) +* [SE-0064: Referencing the Objective-C selector of property getters and setters](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0064-property-selectors.md) +* [SE-0065: A New Model For Collections and Indices](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0065-collections-move-indices.md) +* [SE-0066: Standardize function type argument syntax to require parentheses](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0066-standardize-function-type-syntax.md) +* [SE-0067: Enhanced Floating Point Protocols](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0067-floating-point-protocols.md) +* [SE-0069: Mutability and Foundation Value Types](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0069-swift-mutability-for-foundation.md) +* [SE-0070: Make Optional Requirements Objective-C-only](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0070-optional-requirements.md) +* [SE-0071: Allow (most) keywords in member references](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0071-member-keywords.md) +* [SE-0072: Fully eliminate implicit bridging conversions from Swift](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0072-eliminate-implicit-bridging-conversions.md) +* [SE-0076: Add overrides taking an UnsafePointer source to non-destructive copying methods on UnsafeMutablePointer](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md) +* [SE-0077: Improved operator declarations](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0077-operator-precedence.md) +* [SE-0081: Move `where` clause to end of declaration](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0081-move-where-expression.md) +* [SE-0085: Package Manager Command Names](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0085-package-manager-command-name.md) +* [SE-0086: Drop NS Prefix in Swift Foundation](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0086-drop-foundation-ns.md) +* [SE-0088: Modernize libdispatch for Swift 3 naming conventions](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0088-libdispatch-for-swift3.md) +* [SE-0089: Renaming `String.init(_: T)`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0089-rename-string-reflection-init.md) +* [SE-0091: Improving operator requirements in protocols](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0091-improving-operators-in-protocols.md) +* [SE-0092: Typealiases in protocols and protocol extensions](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0092-typealiases-in-protocols.md) +* [SE-0093: Adding a public `base` property to slices](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0093-slice-base.md) +* [SE-0094: Add sequence(first:next:) and sequence(state:next:) to the stdlib](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0094-sequence-function.md) +* [SE-0095: Replace `protocol` syntax with `P1 & P2` syntax](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0095-any-as-existential.md) +* [SE-0096: Converting dynamicType from a property to an operator](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0096-dynamictype.md) +* [SE-0099: Restructuring Condition Clauses](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0099-conditionclauses.md) +* [SE-0101: Reconfiguring `sizeof` and related functions into a unified `MemoryLayout` struct](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0101-standardizing-sizeof-naming.md) +* [SE-0102: Remove `@noreturn` attribute and introduce an empty `Never` type](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0102-noreturn-bottom-type.md) +* [SE-0103: Make non-escaping closures the default](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0103-make-noescape-default.md) +* [SE-0106: Add a `macOS` Alias for the `OSX` Platform Configuration Test](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0106-rename-osx-to-macos.md) +* [SE-0107: UnsafeRawPointer API](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0107-unsaferawpointer.md) +* [SE-0109: Remove the `Boolean` protocol](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0109-remove-boolean.md) +* [SE-0111: Remove type system significance of function argument labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0111-remove-arg-label-type-significance.md) +* [SE-0112: Improved NSError Bridging](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0112-nserror-bridging.md) +* [SE-0113: Add integral rounding functions to FloatingPoint](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0113-rounding-functions-on-floatingpoint.md) +* [SE-0114: Updating Buffer "Value" Names to "Header" Names](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0114-buffer-naming.md) +* [SE-0115: Rename Literal Syntax Protocols](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0115-literal-syntax-protocols.md) +* [SE-0116: Import Objective-C `id` as Swift `Any` type](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0116-id-as-any.md) +* [SE-0117: Allow distinguishing between public access and public overridability](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0117-non-public-subclassable-by-default.md) +* [SE-0118: Closure Parameter Names and Labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0118-closure-parameter-names-and-labels.md) +* [SE-0120: Revise `partition` Method Signature](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0120-revise-partition-method.md) +* [SE-0121: Remove `Optional` Comparison Operators](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0121-remove-optional-comparison-operators.md) +* [SE-0124: `Int.init(ObjectIdentifier)` and `UInt.init(ObjectIdentifier)` should have a `bitPattern:` label](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md) +* [SE-0125: Remove `NonObjectiveCBase` and `isUniquelyReferenced`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0125-remove-nonobjectivecbase.md) +* [SE-0127: Cleaning up stdlib Pointer and Buffer Routines](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0127-cleaning-up-stdlib-ptr-buffer.md) +* [SE-0128: Change failable UnicodeScalar initializers to failable](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0128-unicodescalar-failable-initializer.md) +* [SE-0129: Package Manager Test Naming Conventions](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0129-package-manager-test-naming-conventions.md) +* [SE-0130: Replace repeating `Character` and `UnicodeScalar` forms of String.init](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0130-string-initializers-cleanup.md) +* [SE-0131: Add `AnyHashable` to the standard library](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0131-anyhashable.md) +* [SE-0133: Rename `flatten()` to `joined()`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0133-rename-flatten-to-joined.md) +* [SE-0134: Rename two UTF8-related properties on String](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0134-rename-string-properties.md) +* [SE-0135: Package Manager Support for Differentiating Packages by Swift version](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md) +* [SE-0136: Memory Layout of Values](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0136-memory-layout-of-values.md) +* [SE-0137: Avoiding Lock-In to Legacy Protocol Designs](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0137-avoiding-lock-in.md) diff --git a/releases/swift-4_0.md b/releases/swift-4_0.md new file mode 100644 index 0000000000..a133a1160b --- /dev/null +++ b/releases/swift-4_0.md @@ -0,0 +1,96 @@ +# Swift 4.0 - Released on September 19, 2017 + +The Swift 4 release was designed around two primary goals: to provide +source stability for Swift 3 code and to provide ABI stability for the +Swift standard library. To that end, the Swift 4 release was divided +into two stages. + +Stage 1 focused on the essentials required for source and ABI +stability. Features that don't fundamentally change the ABI of +existing language features or imply an ABI-breaking change to the +standard library were not considered in this stage. + +The high-priority features supporting Stage 1's source and ABI +stability goals were: + +* Source stability features: The Swift language will need [some + accommodations](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0141-available-by-swift-version.md) + to support code bases that target different language versions, to + help Swift deliver on its source-compatibility goals while still + enabling rapid progress. + +* Resilience: Resilience provides a way for public APIs to evolve over + time, while maintaining a stable ABI. For example, resilience + eliminates the [fragile base class + problem](https://en.wikipedia.org/wiki/Fragile_base_class) that + occurs in some object-oriented languages (e.g., C++) by describing + the types of API changes that can be made without breaking ABI + (e.g., "a new stored property or method can be added to a class"). + +* Stabilizing the ABI: There are a ton of small details that need to + be audited and improved in the code generation model, including + interaction with the Swift runtime. While not specifically + user-facing, the decisions here affect performance and (in some rare + cases) the future evolution of Swift. + +* Generics improvements needed by the standard library: The standard + library has a number of workarounds for language deficiencies, many + of which manifest as extraneous underscored protocols and + workarounds. If the underlying language deficiencies remain, they + become a permanent part of the stable ABI. [Conditional + conformances](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0143-conditional-conformances.md), + [recursive protocol + requirements](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#recursive-protocol-constraints-), + and [where clauses for associated + types](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md) + are known to be in this category, but it's plausible that other + features will be in scope if they would be used in the standard + library. + +* String re-evaluation: String is one of the most important + fundamental types in the language. Swift 4 seeks to make strings more + powerful and easier-to-use, while retaining Unicode correctness by + default. + +* Memory ownership model: An (opt-in) Cyclone/Rust-inspired memory + ownership model is highly desired by systems programmers and for + other high-performance applications that want predictable and + deterministic performance. This feature will fundamentally shape the + ABI, from low-level language concerns such as "inout" and low-level + "addressors" to its impact on the standard library. While a full + memory ownership model is likely too large for Swift 4 stage 1, we + need a comprehensive design to understand how it will change the + ABI. + +Swift 4 stage 2 built on the goals of stage 1. It differed in that +stage 2 proposals could include some additive changes and changes to +existing features that don't affect the ABI. There were a few focus +areas for Swift 4 stage 2: + +* Stage 1 proposals: Any proposal that would have been eligible for + stage 1 is a priority for stage 2. + +* Source-breaking changes: The Swift 4 compiler will provide a + source-compatibility mode to allow existing Swift 3 sources to + compile, but source-breaking changes can manifest in "Swift 4" + mode. That said, changes to fundamental parts of Swift's syntax or + standard library APIs that break source code are better + front-loaded into Swift 4 than delayed until later + releases. Relative to Swift 3, the bar for such changes is + significantly higher: + + * The existing syntax/API being changed must be actively harmful. + * The new syntax/API must clearly be better and not conflict with existing Swift syntax. + * There must be a reasonably automatable migration path for existing code. + +* Improvements to existing standard library facilities: Additive + changes that improve existing standard library facilities can be + considered. With standard library additions in particular, proposals + that provide corresponding implementations are preferred. Potential + focus areas for improvement include collections (e.g., new + collection algorithms) and improvements to the ergonomics of + `Dictionary`. + +* Foundation improvements: We anticipate proposing some targeted + improvements to Foundation API to continue the goal of making the + Cocoa SDK work seamlessly in Swift. diff --git a/releases/swift-5_0.md b/releases/swift-5_0.md new file mode 100644 index 0000000000..fdd12922e0 --- /dev/null +++ b/releases/swift-5_0.md @@ -0,0 +1,56 @@ +# Swift 5.0 - Released on March 25, 2019 + +## Primary Focus: ABI Stability + +The Swift 5 release **will** provide [ABI stability](https://github.com/apple/swift/blob/master/docs/ABIStabilityManifesto.md#what-is-abi-stability) for the Swift Standard Library. ABI stability enables OS vendors to embed a Swift Standard Library and runtime in the OS that is compatible with applications built with Swift 5 or later. Progress towards achieving ABI stability will be tracked at a high level on the [ABI Dashboard](https://swift.org/abi-stability/). + +ABI stability is only one of two pieces needed to support binary frameworks. The second half is *module stability* (see "[The Big Picture](https://github.com/apple/swift/blob/master/docs/ABIStabilityManifesto.md#the-big-picture)" of the [ABI Stability Manifesto](https://github.com/apple/swift/blob/master/docs/ABIStabilityManifesto.md) for more information). While we’d like to support this for Swift 5, it will be a stretch goal, and may not make it in time. + +The need to achieve ABI stability in Swift 5 will guide most of the priorities for the release. In addition, there are important goals to complete that carry over from Swift 4 that are prerequisites to locking down the ABI of the standard library: + +- **Generics features needed for standard library**. We will finish implementing [conditional conformances](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0143-conditional-conformances.md) and [recursive protocol requirements](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0157-recursive-protocol-constraints.md), which are needed for the standard library to achieve ABI stability. Both of these have gone through the evolution proposal process and there are no known other generics enhancements needed for ABI stability. + +- **API resilience**. We will implement the essential pieces needed to support API resilience, in order to allow public APIs for a library to evolve over time while maintaining a stable ABI. + +- **Memory ownership model**. An (opt-in) Cyclone/Rust-inspired memory [ownership model](https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md) is strongly desirable for systems programming and for other high-performance applications that require predictable and deterministic performance. Part of this model was introduced in Swift 4 when we began to [ enforce exclusive access to memory](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0176-enforce-exclusive-access-to-memory.md). In Swift 5 our goal is to tackle the [pieces of the ownership model that are key to ABI stability](https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md#priorities-for-abi-stability). + +## Other Improvements + +Beyond ABI stability (which focuses mostly on getting a bunch of low-level implementation details of the language finalized), in Swift 5 the evolution process welcomes additions that improve the overall usability of the language and standard library, including but not restricted to: + +- **String ergonomics**. We will complete more of the work outlined in the [String Manifesto](https://github.com/apple/swift/blob/master/docs/StringManifesto.md) to make `String` easier to use and more performant. This work may include the addition of new text processing affordances to the language and standard library, and language-level support for regular expressions. In addition to ergonomic changes, the internal implementation of `String` offers many opportunities for enhancing performance which we would like to exploit. + +- **Improvements to existing standard library facilities**. We will consider other minor additions to existing library features, but are not open for significant new facilities outside of supporting the primary focuses of this release. + +- **Foundation improvements**. We anticipate proposing some targeted improvements to Foundation API to further the goal of making the Cocoa SDK work seamlessly in Swift. + +- **Syntactic additions**. Syntactic changes do not increase the expressive power of the language but do increase its complexity. Consequently, such changes must be extremely well-motivated and will be subject to additional scrutiny. We will expect proposals to include concrete data about how widespread the positive impact will be. + +- **Laying groundwork for a new concurrency model**. We will lay groundwork for a new concurrency model, especially as needed for ABI stability. Finalizing such a model, however, is a *non-goal* for Swift 5. A key focus area will be on designing language affordances for creating and using asynchronous APIs and dealing with the problems created by callback-heavy code. + +## Source Stability + +Similar to [Swift 4](swift-4_0.md) , the Swift 5 compiler will provide a source compatibility mode to allow source code written using some previous versions of Swift to compile with the Swift 5 compiler. The Swift 5 compiler will at least support code written in Swift 4, but may also extend back to supporting code written in Swift 3. The final decision on the latter will be made in early 2018. + +Source-breaking changes in Swift 5 will have an even higher bar than in Swift 4, following these guidelines: + +* The current syntax/API must be shown to actively cause problems for users. +* The new syntax/API must be clearly better and must not conflict with existing Swift syntax. +* There must be a reasonably automated migration path for existing code. + +## Evolution Process for Swift 5 + +Unlike [Swift 4](swift-4_0.md), there will be no "stage 1" and "stage 2" for the evolution process. Proposals that fit within the general focus of the release are welcome until **March 1, 2018**. Proposals will still be considered after that, but the bar will be increasingly high to accept changes for Swift 5. + +The broader range of proposals for Swift 5 compared to Swift 4 incurs the risk of diluting the focus on ABI stability. +To mitigate that risk, **every evolution proposal will need a working implementation, with test cases, in order to be considered for review**. An idea can be pitched and a proposal written prior to providing an implementation, but a pull request for a proposal will not be accepted for review until an implementation is available. + +More precisely: + +1. Once a proposal is written, the authors submit the proposal via a pull request to the `swift-evolution` repository. + +2. The Core Team will regularly review `swift-evolution` pull requests, and provide feedback to the authors in the pull request on whether or not the proposal looks within reason of something that might be accepted. + +3. If a proposal gets a positive indicator from the Core Team for later review, the authors must provide an implementation prior to the proposal being formally reviewed. An implementation should be provided in the form of a pull request against the impacted repositories (e.g., `swift`, `swift-package-manager`), and the proposal should be updated with a link to that pull request. The existence of an implementation does not guarantee that the proposal will be accepted, but it is instrumental in evaluating the quality and impact of the proposal. + +We want to strike a balance between encouraging open discussion of potential changes to the language and standard library while also providing more focus when changes are actually reviewed. Further, having implementations on hand allow the changes to be more easily tried out before they are officially accepted as part of the language. In particular, development of the initial pull request for a proposal remains a very open review process that everyone in the community can contribute a lot to. Similarly, members of the community can help craft implementations for a proposal even if they aren't the authors of the proposal. diff --git a/visions/approachable-concurrency.md b/visions/approachable-concurrency.md new file mode 100644 index 0000000000..25e6e6b725 --- /dev/null +++ b/visions/approachable-concurrency.md @@ -0,0 +1,198 @@ +# Improving the approachability of data-race safety + +[SE-0434]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0434-global-actor-isolated-types-usability.md + +> This document is an official feature [vision document](https://forums.swift.org/t/the-role-of-vision-documents-in-swift-evolution/62101). The Language Steering Group has endorsed the goals and basic approach laid out in this document. This endorsement is not a pre-approval of any of the concrete proposals that may come out of this document. All proposals will undergo normal evolution review, which may result in rejection or revision from how they appear in this document. + +## Introduction + +Swift's built-in support for concurrency has three goals: + +1. Extend memory safety guarantees to low-level data races. +2. Maintain progressive disclosure for non-concurrent code, and make basic use of concurrency simple and easy. +3. Make advanced uses of concurrency to improve performance natural to accomplish and reason about. + +The Swift 6 language mode provides a baseline of correctness that meets the first goal, but sometimes it comes at the cost of the second, and it can be frustrating to adopt. Now that we have a lot more user experience under our belt as a community, it’s reasonable to ask what we can do in the language to address that problem. This document lays out several potential paths for improving the usability of Swift 6, focusing on two primary use cases: + +1. Simple situations where programmers aren’t intending to use concurrency at all. +2. Adapting an existing code base that uses concurrency libraries which predate Swift's native concurrency model. + +While performance is not our immediate focus, it’s something we need to keep in mind during this exercise: we don’t want these usability wins to create pervasive regressions or to make it frustratingly difficult to achieve a high level of performance. + +A key tenet of our thinking in this vision is that we want to drastically reduce the number of explicit concurrency annotations necessary in projects that aren’t trying to leverage parallelism for performance. This is important for many kinds of programming, such as UI programming and scripts, where concurrency is often localized and large swathes of the code are generally expected to be constrained to the main actor. At the same time, we want to maintain a smooth path for experienced programmers to opt in to concurrency and maintain the safety of complete data-race checking. + +As we see it, there should be three phases on the progressive disclosure path for concurrency: + +1. **Write sequential, single-threaded code**. By default, programmers writing executable projects will write sequential code; there is no runtime parallelism, and therefore no data-race safety errors are surfaced to the programmer. +2. **Write asynchronous code without data-race safety errors**. When programmers need functionality that can suspend, they can start introducing basic uses of async/await. Programmers won’t have to confront data-race safety at this point, because they aren’t yet introducing parallelism into their code. This is an important distinction, because there are many library APIs that perform work asynchronously, but they don’t need to use the programmer’s shared mutable state from a concurrent task. In these cases, programmers don’t have to understand data-race safety just to call an async API. +3. **Introduce parallelism to improve performance.** When the programmer is ready to embrace concurrency to get better performance, they can explicitly offload work from the main actor to the cooperative thread pool, leverage tasks and structured concurrency, etc, all while relying on the compiler to prevent mistakes that risk a data race. + +## Mitigating false positive data-race safety errors in sequential code + +A lot of code is effectively “single-threaded”. For example, most executables, such as apps, command-line tools, and scripts, start running on the main actor and just stay there unless some part of the code actually does something concurrent (like creating a `Task`). If there isn’t any use of concurrency, the entire program will run sequentially, and there’s no risk of data races — every concurrency diagnostic is necessarily a false positive! It would be good to be able to take advantage of that in the language, both to avoid annoying programmers with unnecessary diagnostics and to reinforce progressive disclosure. Many people get into Swift by writing these kinds of programs, and if we can avoid needing to teach them about concurrency straight away, we’ll make the language much more approachable. + +Now, “If nothing in the program uses concurrency, suppress all the concurrency diagnostics” requires what compiler writers call a *whole-program analysis*, and rules like that tend not to work out well on multiple levels. For one, it would require the compiler to look at all of the code in the program all at once; this might be okay for small scripts, but it would scale poorly as the program got more complex. More importantly, it would make the first adoption of concurrency extremely painful: programmers would be hit by a tidal wave of errors in code they haven’t changed. And, of course, many libraries do use concurrency behind the scenes; importing even a single library like that would force concurrency-safety diagnostics everywhere. + +A better approach is to locally state our assumption that the sequential parts of the program are “single-threaded”. Rather than having to assume the possibility of concurrency, Swift would know that these parts of the code will all run sequentially, which it can use to prove that there aren’t any data races. There can still be concurrent parts of the program elsewhere, but Swift would stop them from accessing the single-threaded bits. Fortunately, this is something that Swift can already model quite well! + +### Single-threaded code and its challenges under Swift 6 + +The easiest and best way to model single-threaded code is with a global actor. Everything on a global actor runs sequentially, and code that isn’t isolated to that actor can’t access its data. All programs start running on the global actor `MainActor`, and if everything in the program is isolated to the main actor, there shouldn’t be any concurrency errors. + +Unfortunately, it’s not quite that simple right now. Writing a single-threaded program is surprisingly difficult under the Swift 6 language mode. This is because Swift 6 defaults to a presumption of concurrency: if a function or type is not annotated or inferred to be isolated, it is treated as non-isolated, meaning it can be used concurrently. This default often leads to conflicts with single-threaded code, producing false positive diagnostics in cases such as: + +* global and static variables, +* conformances of main-actor-isolated types to non-isolated protocols, +* class deinitializers, +* overrides of non-isolated superclass methods in a main-actor-isolated subclass, and +* calls to main-actor-isolated functions from the platform SDK. + +To see this, let’s explore the first of those cases in more detail. A mutable global variable (or an immutable one that stores a non-`Sendable` value) is only memory-safe if it’s used in a single-threaded way. If the whole program is single-threaded, there’s no problem, and the variable is always safe to use. But since Swift 6 presumes concurrency by default, it requires a variable like this to be explicitly isolated to a global actor, like `@MainActor`. A function that uses that variable is then also required to be statically isolated to `@MainActor`: + +```swift +class AudioManager { + @MainActor + static let shared = AudioManager() + + func playSound() { ... } +} + +class Model { + func play() { + AudioManager.shared.playSound() // error: Main actor-isolated static property 'shared' can not be referenced from a nonisolated context + } +} +``` + +And this in turn means that functions that call those functions must also be `@MainActor`, and so on until the `@MainActor` annotation has been laboriously propagated throughout the entire transitive tree of callers. Because main actor isolation is so common, many programmers have resorted to reflexively writing `@MainActor` everywhere, an onerous annotation burden that goes against Swift’s goals of making the simplest things easy. + +Because the default programming model presumes concurrency, it is also hard on programmers who haven’t yet learned about concurrent programming, because they are confronted with the concept of data-race safety and actor isolation too early simply by using these basic language features: + +```swift +class AudioManager { + static let shared = AudioManager() // error: Static property 'shared' is not concurrency-safe because non-'Sendable' type 'AudioManager' may have shared mutable state +} +``` + +Analogous problems arise with all the other kinds of false positives listed above. For example, when using values from generic code, the value’s type usually must conform to one or more protocols. However, actor-isolated types cannot easily conform to protocols that aren’t aware of that isolation: they can declare the conformance, but it’s often impossible to write a useful implementation because the value’s properties will not be available. This is exactly the same kind of conflict as with global variables, where we have generally single-threaded code but a presumption of concurrency from the protocol, except that *this* conflict usually can’t be solved with annotations at all — the only fixes are to change the protocol, avoid all the isolated storage, or dangerously assert (with `assumeIsolated`) that the method is only used dynamically from the right actor. + +### Allowing modules to default to being “single-threaded” + +We believe that the right solution to these problems is to allow code to opt in to being “single-threaded” by default, on a module-by-module basis. This would change the default isolation rule for unannotated code in the module: rather than being non-isolated, and therefore having to deal with the presumption of concurrency, the code would instead be implicitly isolated to `@MainActor`. Code imported from other modules would be unaffected by the current module’s choice of default. When the programmer really wants concurrency, they can request it explicitly by marking a function or type as `nonisolated` (which can be used on any declaration as of [SE-0449](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0449-nonisolated-for-global-actor-cutoff.md)), or they can define it in a module that doesn’t default to main-actor isolation. This doesn’t fundamentally change anything about Swift’s isolation model; it just flips the default, effectively creating a model in which code is single-threaded except where it explicitly requests concurrency. Modules that don’t want this could of course continue to use the current rules. + +Making a module be isolated to the main actor by default would directly fix several of the false positive problems listed above for single-threaded code. Global variables would default to being isolated to the main actor, avoiding the diagnostic when they’re declared. Functions in the module would also default to being isolated to the main actor, allowing them to freely use both those isolated global variables and any main-actor-isolated functions and variables imported from the platform SDK. Class overrides and protocol conformances aren’t quite so easy, but we think we can extend them in ways that allow a natural solution with main actor isolation. We’ll get to how later in this document. + +### Default concurrency rules for executable and library modules + +As mentioned above, executable targets tend to center around the main actor. Command-line tools and scripts all start on the main actor and continue to run there unless they explicitly do something that introduces concurrency. Similarly, most UI programs make heavy use of single-threaded UI frameworks that privilege the main actor. This kind of code would be greatly improved by adopting a single-threaded model by default. Furthermore, since many new Swift programmers find themselves first writing this kind of code, this would also be a significant improvement to Swift’s progressive disclosure: programmers writing code in this mode should not run into data-race safety issues and diagnostics until they intentionally introduce concurrency. We feel that this amounts to a compelling argument that executable targets should default to inferring main actor isolation. + +The same argument does not apply to libraries. Most library functions are meant to be usable from any context, and libraries usually avoid using any global or shared mutable state. Swift also already asks a little more of library authors in general; for example, access control is usually a more significant concern for library authors than for app developers. It would be reasonable for library targets to default to `nonisolated` the same way they do today in Swift 6. + +Specific libraries could still decide to default to the main actor, such as when they’re libraries of UI widgets, or if a library is used for code organization within an executable project. + +### Risks of a language dialect + +Adding a per-module setting to specify the default isolation would introduce a new permanent language dialect. In a sense, Swift adds a new language dialect whenever it adds an upcoming language feature flag, but these are seen as “temporary” because it’s expected that those features will eventually be rolled into a future language mode. Permanent language dialects can be problematic for a variety of reasons: + +* They can harm readability if readers have to know which dialect the code uses before they can understand the code. +* They can harm usability if programmers have to consciously program differently based on the dialect in use in the code or if code cannot be easily moved between projects using different dialects. +* They can harm tools such as IDEs if the tool has to know which dialect the code uses before it can work correctly; this is particularly challenging for code files that may be used in multiple dialects, such as a `.h` file in a C/C++/Objective-C IDE. +* When dialects are platform-dependent, they can harm portability and basic workflows (such as testing) if it is difficult to make the same code build as multiple dialects. + +Some of these problems do not seem to apply to this proposed dialect because it only affects the isolation of declarations. Most IDE services, such as syntax highlighting and code completion, do not need to know the isolation of the surrounding context. And there’s no good reason for a module to build with a different default isolation on different platforms, so the dialect does not seem to introduce any portability concerns. + +Readability does seem to be a fair concern. It is often useful to know what isolation a function will run under, and with this change, that would be dialect-specific. However, it is worth noting that the dynamic isolation of a function is already not always statically knowable because of the way that e.g. synchronous `nonisolated` functions inherit their callers’ isolation. And it would be reasonable for IDEs to be able to present isolation information for the current context: even without this dialect change, it is not always easy to understand how Swift’s isolation inference rules will apply to any particular declaration. + +Programmers will probably not consciously program differently under these different dialects, and the compiler should provide reasonable guidance if they make a mistake. Moving code from a single-threaded module to a nonisolated module might be somewhat more arduous, however. To some degree, this is inherent and arguably even good: the programmer may be moving this code in an effort to generalize it to work concurrently, and any new diagnostics represent real problems that the programmer didn’t have to deal with before this generalization. But when the programmer is not trying to generalize the code to use concurrency, this could be frustrating, and it might be good for IDEs to offer assistance, such as tools to make the single-threaded assumptions of a piece of code explicit. + +On balance, we feel that the costs of this particular dialect are modest and manageable. + +## Isolated conformances + +When checking a conformance to a protocol, Swift 6 often requires implementations to be nonisolated when the requirement is, including when the requirement is synchronous or the parameters are not Sendable. This makes it difficult to implement nonisolated protocols with any kind of isolated type: global-actor-isolated types, certainly, but also actors themselves. This restriction is very important when writing concurrent code, but it's a common source of false positives in single-threaded programs. Even worse, there's often no good solution to the problem: a correct implementation of the protocol for an isolated type usually requires access to the isolated data, and the only way to get that is to assert that the calling context is actually isolated, which completely subverts the static isolation safety that Swift 6 tries to provide. + +In many ways, isolation's interaction with protocol requirements is similar to its interaction with function values. In both situations, we have an abstract signature that doesn't express isolation by default, which Swift wants to interpret as an affirmative statement that the isolation of the implementation doesn't matter. Over the last few years, Swift has gradually added more ways to handle isolated function values: + +- A function value can have a type like `() -> Bool` that says it's not sendable. These functions can have any kind of isolation as long as it's the same as the current concurrency domain. Since the function can't be used from a different context, there's no need for it to spell out its isolation explicitly in its type; Swift just checks that the isolation actually matches whenever it makes a new non-`Sendable` function value. + +- A function value can have a type like `@MainActor () -> Bool` that says it's isolated to a specific global actor. These functions can either be non-isolated or isolated to that actor, and Swift just treats them as the latter unconditionally when calling them. + +- A function value can have a type like `@isolated(any) () -> Bool` that says it might be isolated to a specific actor that it carries around with it dynamically. These functions can be isolated to anything, and the caller has to be prepared to handle it. + +Each of these ideas also works with protocol conformances. If we know that we only intend to use a protocol conformance from the current concurrency domain, we don't really care whether the implementation requires some sort of isolation as long as the current context has that isolation. Similarly, if we know that the implementation of a protocol might be isolated to a specific global actor, we can handle that whenever we use the conformance exactly as if the protocol requirements were declared isolated to that actor. And we could even do that dynamically with a statically-unknown isolation, the same way Swift already does with `@isolated(any)` function values. + +The most important of these for our model of single-threaded code is to be able to express global-actor-isolated conformances. When a type is isolated to a global actor, its methods will be isolated by default. Normally, these methods would not be legal implementations of nonisolated protocol requirements. When Swift recognizes this, it can simply treat the conformance as isolated to that global actor. This is a kind of *isolated conformance*, which will be a new concept in the language. + +Now, an isolated conformance is less flexible than a nonisolated conformance. For example, generic types and functions from nonisolated modules (including all current declarations) will still be interpreted as requiring nonisolated conformances. This will mean that they can't be called with a type that only has an isolated conformance, but it will also allow them to freely use the conformance from any concurrency domain, the same way they can today. Generic types and functions in "single-threaded" modules will default to allowing conformances isolated to the module's default global actor. Since those functions will themselves be isolated to that global actor, they won't have any problem using those conformances. + +A generic function that can work with both isolated and nonisolated conformances should be able to declare that it can accept an isolated conformance. It would then be restricted to only use the conformance from the current concurrency domain, as if it were a sort of "non-sendable conformance". This is an important tool for generic libraries such as the standard library, many of which will never use conformances concurrently and so are fine with accepting isolated conformances. + +This design is still being developed, and there are a lot of details that will have to be figured out. Nonetheless, we are tentatively very excited about the potential for this feature to fix problems with how Swift's generics system interacts with isolated types, especially global-actor-isolated types. + +## Isolated subclasses and overrides + +To achieve data race safety, Swift 6 has to diagnose a variety of problems that are specific to classes: + +- The sendability of a class must match the sendability of its superclass[^1]. +- If a class is sendable and not isolated to a global actor, its stored properties must be sendable. +- If a class is isolated to a global actor, its superclass must either be non-isolated or isolated to the same global actor. +- An override must have the same isolation as the declaration it overrides. + +[^1]: There is a natural exception to this rule: a sendable class can have a non-sendable superclass if the superclass and all of its ancestors only have sendable stored properties. Currently, Swift only implements this exception for the exact class `NSObject`. + +All of these diagnostics are false positives in purely sequential programs. Fortunately, many of them don't apply to global-actor-isolated classes in the first place. The restriction that isolated classes can't inherit from classes isolated to a different global actor is very reasonable, but it's extremely unlikely to affect programmers in practice because they rarely use global actors other than the main actor at all. The only significant false positive here, then, is the restriction on overriding. + +[SE-0434][] changed the rules for global-actor-isolated classes to allow them to inherit from non-sendable classes; the subclass just remains non-sendable. Since the class is non-sendable and isolated to a global actor, it's tempting to say that override restrictions shouldn't be necessary for it: the object reference should only be usable on the global actor in the first place. This isn't quite true today, but we think we can revise SE-0434 to make it true by imposing restrictions on initializers in the subclass. This would be a small source break, but it would greatly improve the usability of these classes. + +Unfortunately, global-actor-isolated classes that inherit from nonisolated sendable classes can't benefit from the same idea. We can only see one way to make it safe to allow isolated overrides of non-isolated methods for these classes without a whole-program prohibition of concurrency: we would have to prevent any reference to the subclass from being converted to its sendable superclass type. This is feasible to implement, but we suspect it would be too restrictive to be widely useful; we can revisit this with more information. + +## Easing the introduction of basic async code + +[SE-0338](https://github.com/hborla/swift-evolution/blob/async-function-isolation/proposals/0338-clarify-execution-non-actor-async.md) specifies that nonisolated async functions never run on an actor's executor. This design decision was made to prevent unnecessary serialization and contention for the actor by switching off of the actor to run the nonisolated async function, and any new tasks it creates that inherit isolation. The actor is then free to make forward progress on other work. This behavior is especially important for preventing unexpected overhang on the main actor. + +Over time, we have learned that this design decision undermines progressive disclosure, because it prioritizes main actor responsiveness at the expense of making basic asynchronous code difficult to write. Always switching off of an actor to run a nonisolated async function imposes data-race safety errors on programmers when they call the API with non-sendable arguments from the main actor. + +Many library APIs have transitioned to using isolated parameters to ensure that an async API runs on the caller by default, because it’s a much easier default to work with in client code. + +The current execution semantics of async functions also impede programmer’s understanding of the concurrency model because there is a significant difference in what `nonisolated` means on synchronous and asynchronous functions. Nonisolated synchronous functions always run in the isolation domain of the caller, while nonisolated async functions always switch off of the caller's actor (if there is one). It's confusing that `nonisolated` does not have a consistent meaning when applied to functions, and the current behavior conflates the concept of actor isolation with the ability for a function to suspend. + +Changing the default execution semantics of nonisolated async functions to run wherever they are called better facilitates progressive disclosure of concurrency. This default allows functions to leverage suspension without forcing callers to cross an isolation boundary and imposing data-race safety checks on arguments and results. A lot of basic asynchronous code can be written correctly and efficiently with only the ability to suspend. When an async function needs to always run off of an actor, the API author can still specify that with a new `@execution(concurrent)` annotation on the function. This provides a better default for most cases while still maintaining the ease of specifying that an async function switches off of an actor to run. + +Many programmers have internalized the SE-0338 semantics, and making this change several years after SE-0338 was accepted creates an unfortunate intermediate state where it's difficult to understand the semantics of a nonisolated async function without understanding the build settings of the module you're writing code in. We can alleviate some of these consequences with a careful migration design. There are more details about migration in the [automatic migration](#automatic-migration) section of this document, and in the [source compatibility](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0461-async-function-isolation.md#source-compatibility) section of the proposal for this change. + +This idea has already been pitched on the forums, and you can read the full proposal for it [here](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0461-async-function-isolation.md). + +## Easing incremental migration to data-race safety + +Many programmers are not new to concurrency as a concept and have extensive experience with multi-threaded programming. However, the Swift 6 data-race safety model is a significant shift for programmers with experience using concurrency libraries that predate Swift's native concurrency features. Moreover, there are many existing, large codebases that were built on such libraries, and migrating these codebases to both leverage modern concurrency features and enable static data-race safety takes significant engineering effort. An explicit goal of improving the approachability of data-race safety is lessening the amount of effort it takes to enable data-race safety in existing codebases. + +### Bridging between synchronous and asynchronous code + +Introducing async/await into an existing codebase is difficult to do incrementally, because the language does not provide tools to bridge between synchronous and asynchronous code. Sometimes programmers can kick off a new unstructured task to perform the async work, and other times that is not suitable, e.g. because the synchronous code needs a result from the async operation. It’s also not always possible to propagate `async` throughout callers, because the function signature might be declared in a library dependency that you don’t own. + +Notably, using actors in programs that make heavy use of the main actor forces programmers to use async/await, because all interactions with an actor must be done asynchronously from outside the actor. This significantly restricts the utility of actors, especially in existing codebases. + +Other concurrency libraries like Dispatch provide a limited tool set to wait on asynchronous work, such as `DispatchQueue.asyncAndWait`. These tools come with serious tradeoffs, including tying up limited system resources and introducing the possibility for deadlocks, but they provide critical functionality that is sometimes necessary in a project. It’s important for programmers to have the ability to express this in the language, and because the language model allows actor re-entrancy and doesn’t have a strict FIFO guarantee for tasks enqueued on an actor, there’s opportunity to mitigate the risk of deadlocks that these tools come with. + +### Mitigating runtime assertions due to isolation mismatches + +[SE-0423: Dynamic actor isolation enforcement from non-strict-concurrency contexts](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0423-dynamic-actor-isolation.md) introduced dynamic actor isolation assertions that are injected by the compiler at the boundaries between data-race safe and unsafe code. This dynamic checking catches actor isolation violations in library dependencies that have not yet migrated to the Swift 6 language mode, and may have data-race safety issues in their implementation. These checks are effective at identifying missing `@Sendable` annotations, but they also make Swift 6 adoption painful for clients when they migrate before their dependencies. + +Some of these runtime crashes are false positives; the runtime checks are inserted based on the static isolation of the function, but the function might not access any mutable state that’s isolated to or derived from the actor. In these cases, the dynamic checks can simply be elided based on analysis of the function implementation. + +In other cases, the dynamic assertion indicates the presence of a runtime data race, because isolated state is being accessed from outside the actor. The correct way to resolve this data race is either to run the function on the actor, or change the function to eliminate access to actor-isolated state. There are two possible avenues for the language to aid programmers in resolving the data race: + +* Instead of directly calling the function in the wrong isolation domain, enqueue a job that calls the function on the actor. This only works if the function does not return a result. +* Use the import-as-async heuristic from [SE-0297: Concurrency Interoperability with Objective-C](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0297-concurrency-objc.md) to automatically import completion handlers of asynchronous functions as `@Sendable`, which will allow the compiler to diagnose access to actor-isolated state in completion handlers. + +## Automatic migration + +Unlike the Swift 6 migration, all language changes with source compatibility impact described in this vision can be automatically migrated to while preserving the semantics of existing code. The source incompatible portions of this vision will be gated behind [upcoming feature flags](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md), which will be enabled by default in a future 6.x language mode, except for the per-module setting to infer main actor by default. + +Compiler tooling can automatically migrate existing projects when they choose to enable each of these upcoming features, either individually or as part of a future language mode migration. Programmers will be able to perform a “migration build” with one or more upcoming language features enabled, or with a specific language mode that enables a set of upcoming features, and be offered source code changes that would allow the compiler to build the project without errors and without changing semantics. + +## What’s not in this vision + +This vision does not cover existing pain points with task ordering and actor re-entrancy. These are important problems, but they are more prevalent in more advanced uses of concurrency and warrant a separate, dedicated exploration. + +Improving concurrency diagnostics and documentation is also not covered in this document. All language proposals should consider diagnostics to the extent that language design decisions prevent precise and actionable error messages. Beyond that, diagnostics and documentation changes are not governed by the Swift evolution process, because these changes don't have long-term source compatibility and ABI constraints, so gating improvements behind a heavy weight review process isn't necessary. However, diagnostics and documentation are an extremely important tool for making the concurrency model more approachable, and they will be included in the implementation effort behind this vision. diff --git a/visions/embedded-swift.md b/visions/embedded-swift.md new file mode 100644 index 0000000000..7bd8d7aa50 --- /dev/null +++ b/visions/embedded-swift.md @@ -0,0 +1,165 @@ +# A Vision for Embedded Swift + +## Introduction + +Swift is a general purpose programming language suitable for both high-level application software and for low-level systems software. The existing major supported deployments of Swift are primarily targeting “large” operating systems (Linux, Windows, macOS, iOS), where storage and memory are relatively plentiful, multiple applications share code via dynamic linking, and the system can be expected to provide a number of common libraries (such as the C and C++ standard libraries). The typical size of the Swift runtime and standard library in these environments is around 5MB of binary size. + +However, lots of embedded platforms and low-level environments have constraints that make the standard Swift distribution impractical. These constraints have been described and discussed in existing prior work in this area, and there have been great past discussions in the Swift forums ([link](https://forums.swift.org/t/introduce-embedded-development-using-swift/56573)) and in last year’s video call ([link](https://forums.swift.org/t/call-for-interest-video-call-kickoff-using-swift-for-embedded-bare-metal-low-resources-programming/56911)), which shows there is a lot of interest and potential for Swift in this space. The motivation of “Embedded Swift” is to achieve a first class support for embedded platforms and unblock porting and using Swift in small environments. In particular the targets are: + +* (1) Platforms that have very limited memory + * Microcontrollers and embedded systems with limited memory + * Popular MCU board families and manufacturers (Arduino, STM32, ESP32, NXP, etc.) commonly offer boards that only have an order of 10’s or 100’s of kB of memory available. + * Firmware, and especially firmware projects that are run from SRAM, or ROM +* (2) Environments where runtime dependencies, implicit runtime calls, and heap allocations are restricted + * Low-level environments without an underlying operating system, such as bootloaders, hypervisors, firmware + * Operating system kernels, kernel extensions, and other non-userspace software components + * Userspace components that are too low-level in terms of dependencies, namely anything that the Swift runtime depends on. + * A special case here is the Swift runtime itself, which is today written in C++. The concepts described further in this document allow Swift to become the implementation language instead. + +A significant portion of the current Swift runtime and standard library supports Swift’s more dynamic features, particularly those that involve metadata. These features include: + +* Dynamic reflection facilities (such as mirrors, `as?` downcasts, and printing arbitrary values) +* Existentials +* ABI stability with support for library evolution +* Separately-compiled generics +* Dynamic code loading (plug-ins) + +On “smaller” operating systems, and in restricted environments with limited binary and memory size, the size of a full Swift standard library (with all public types and APIs present) and a Swift runtime, as well as the metadata required to support dynamic features, can be so large as to prevent the usage of Swift completely. In such environments, it may be reasonable to trade away some of the flexibility from the more dynamic features to achieve a significant reduction in code size, while still retaining the character of Swift. + +The following diagram summarizes the existing approaches to shrink down the size of the Swift runtime and standard library, and how “Embedded Swift” is tackling the problems with a new approach: + +diagram + +This document presents a vision for “Embedded Swift”, a new compilation model of Swift that can produce extremely small binaries without external dependencies, suitable for restricted environments including embedded (microcontrollers) and baremetal setups (no operating system at all), and low-level environments (firmware, kernels, device drivers, low-level components of userspace OS runtimes). + +Embedded Swift limits the use of language and standard library features that would require a larger Swift runtime, while maintaining most of Swift’s feature set. It is important that Embedded Swift not become a separate dialect of Swift. Rather, it should remain an easy-to-explain subset of Swift that admits the same code and idioms as the full Swift language, where any restrictions on the language model are flagged by the Swift compiler. The subset itself should also be useful beyond low-level environments, for example, in high-performance runtimes and kernels embedded within a larger Swift application. The rest of this document describes exactly which language features are impacted, as well as the compilation model used for restricted environments. + +## Goals + +There are several goals of this new compilation mode: + +* **Eliminate the “large codesize cost of entry”** for Swift. Namely, the size of the supporting libraries (Swift runtime and standard library) must not dominate the binary size compared to application code. +* **Simplify the code generated by the compiler** to make it easier to reason about it, and about its performance and runtime cost. In particular, complex runtime mechanisms such as runtime generic instantiation are undesirable. +* **Allow effective and intuitive dead-code stripping** on application, library and standard library code. +* **Support environments with and without a dynamic heap**. Effectively, there will be two bottom layers of Swift, and the lower one, “non-allocating” Embedded Swift, will necessarily be a more restricted compilation mode (e.g. classes will be disallowed as they fundamentally require heap allocations) and likely to be used only in very specialized use cases. “Allocating” Embedded Swift should allow classes and other language facilities that rely on the heap (e.g. indirect enums). +* **Remove or reduce the amount of implicit runtime calls**. Specifically, runtime calls supporting heavyweight runtime facilities (type metadata lookups, generic instantiation) should not exist in Embedded Swift programs. Lightweight runtime calls (e.g. refcounting) are permissible but should only happen when the application uses a language feature that needs them (e.g. refcounted types). +* **Introduce a way of producing minimal statically-linked binaries** without external dependencies, namely without the need to link with a non-dead-strippable large Swift runtime/stdlib library, and without the need to link full libc and libc++ libraries. The Swift standard library contains essential facilities for writing Swift code, and must be available to write code against, but it should “fold” into the application and/or be intuitively dead-strippable. +* **Define a language subset, not a dialect.** Any code of Embedded Swift should always compile in regular Swift and behave the same. + * **The Embedded Swift language subset should stay very close to “full” Swift**, even if it means adding alternative ABIs to the compiler to support some of them. Users should expect minimal porting effort to get code to work in Embedded Swift. + +## Embedded Swift Language Subset + +In order to achieve the goals listed above, Embedded **Swift** will impose limitations on certain language features: + +* Library Evolution will be limited in some way, and there’s no expectation of ABI stability or separate distribution of libraries in binary form. +* Objective-C interoperability will not be available. C and C++ interoperability is not affected. +* Reflection and Mirrors APIs will not be available. +* The standard library’s print() function in its current form will not be available, and an alternative will be provided instead. +* Metatypes will be restricted in some way, and code patterns where a metatype value is actually needed at runtime will be disallowed (but at a minimum using a metatype function argument as a type hint will be allowed, as well as calling class methods and initializers on concrete types). + Examples: + ```swift + func foo(t: T.Type) { ... `t` used in a downcast ... } // not OK + extension UnsafeRawPointer { + func load(as type: T.Type) -> T { ... `type` unused ... } // OK + } + MyGenericClass.classFunc() // OK + ``` +* Existentials and dynamic downcasting of existentials will be disallowed. For example: + ```swift + func foo(t: Any.Type) {} // not OK + var e: any Comparable = 42 // not OK + var a: [Any] = [1, "string", 3.5] // not OK + ``` +* The types of thrown errors will be restricted in some manner, because thrown errors are of existential type `any Error` (which is disallowed by the prior item). +* Classes will have restrictions, for example they cannot have non-final generic functions. For example: + ```swift + class MyClass { + func member() { } // OK + func genericMember { } // not OK + } + ``` + It’s an open question whether class metatypes are allowed to be used as runtime values and whether classes will allow dynamic downcasting. +* KeyPaths will be restricted, but at a minimum it will be allowed to use keypath literals to form closures returning a field from a type, and it will be allowed to use keypaths that are compile-time references to inlined stored properties (so that `MemoryLayout.offset(of: ...)` will work on those). +* String APIs requiring Unicode data tables will be unavailable by default (to avoid paying the associated codesize cost), and will require opting in. For example, string iteration, comparing two strings, hashing a string, string splitting are features needing Unicode data tables. These operations should become available on UTF8View instead with the proposal to add Equatable and Hashable conformances to String views ([link](https://forums.swift.org/t/pitch-add-equatable-and-hashable-conformance-to-string-views/60449)). + +**Non-allocating Embedded Swift** will add further restrictions on top of the ones listed above: + +* Classes cannot be instantiated, indirect enums cannot be constructed. +* Escaping closures are not allowed. +* Standard library features and API that rely on classes, indirect enums, escaping closures are not available. This includes for example dynamic containers (arrays, dictionaries, sets) and strings. + +The listed restrictions (for both “allocating” and “non-allocating” Embedded Swift) are not necessarily fundamental, and we might be able to (fully or partially) lift some of them in the future, by adding alternative compile-time implementations (as opposed to their current runtime implementations) of the language features. + +## Implementation of Embedded Compilation Mode + +The following describes the high-level points in the approach to implement Embedded Swift in the compiler: + +* **Specialization is required on all uses of generics and protocols** at compile-time, and libraries are compiled in a way that allows cross-module specialization (into clients of the libraries). + * Required specialization (also known as monomorphization in other compilers/languages) needs type parameters of generic types and functions to always be compile-time known at the caller site, and then the compiler creates a specialized instantiation of the generic type/function that is no longer generic. The result is that the compiled code does not need access to any type metadata at runtime. + * This compilation mode will not support separate compilation of generics, as that makes specialization not possible. Instead, library code providing generic types and functions will be required to provide function bodies as serialized SIL (effectively, “source code”) to clients via the mechanism described below. +* **Library code is built as always inlinable and “emitIntoClient”** to support the specialization of generics/protocols in use sites that are outside of the library. + * **This applies to the standard library, too**, and we shall distribute the standard library built this way with the toolchain. + * This effectively provides the source code of libraries to application builds. +* **The need for type metadata at runtime is completely eliminated**, by further ignoring ABI stability, disabling resilience, and disallowing reflection mirrors APIs. Classes with subclasses get a simple vtable (similar to C++ virtual classes). Classes without subclasses become final and don’t need a vtable. Witness tables (which describe a conformance of a type to a protocol) are only used at compile-time and not present at runtime. + * **Type metadata is not emitted into binaries at all.** This causes code emitted by the compiler to become dead-strippable in the intuitive way given that metadata records (concretely type metadata, protocol conformance records, witness tables) are not present in compiler outputs. + * **Runtime facilities to process metadata are removed** (runtime generic instantiation, runtime protocol conformance lookups) because there is no metadata present at runtime. + +## Enabling Embedded Swift Mode + +The exact mechanics of turning on Embedded Swift compilation mode are an open question and subject to further discussion and refinement. There are different use cases that should be covered: + +* the entire platform / system is using Embedded Swift as a platform level decision +* a single component / library is built using Embedded Swift for an environment that otherwise has other code built with other compilation modes or compilers +* for testing purposes, it’s highly desirable to be able to build a library using Embedded Swift and then exercise that library with a test harness that is built with regular Swift + +A possible solution here would be to have a top-level compiler flag, e.g. `-embedded`, but we could also make environments default to Embedded Swift mode where it makes sense to do so, based on the target triple that’s used for the compilation. Specifically, the existing “none” OS already has the meaning of “baremetal environment”, and e.g. `-target arm64-apple-none` could imply Embedded Swift mode. + +Building firmware using `-target arm64-apple-none` would highlight that we’re producing binaries that are “independent“ and not built for any specific OS. The standard library will be pre-built in the baremetal mode and available in the toolchain for common set of CPU architectures. (It does not need to be built “per OS”.) + +To support writing code that’s compiled under both regular Swift and also Embedded Swift, we should provide facilities to manage availability of APIs and conditional compilation of code. The concrete syntax for that is subject to discussion, the following snippet is presented only as a straw-man proposal: + +```swift +@available(embedded, unavailable, "not available in Embedded Swift mode") +public func notAvailableOnEmbedded() + +#if !mode(embedded) +... code not compiled under Embedded Swift mode ... +#endif + +@available(noAllocations, unavailable, "not available in no allocations mode") +public func notAvailableInNonAllocatingMode() + +#if !mode(noAllocations) +... code not compiled under no allocations mode ... +#endif +``` + +## Dependencies of Embedded Swift Programs + +The expectation is that for “non-allocating” Embedded Swift, the user should only need a working Swift toolchain, and be able to pass a (set of) .swift file(s) to the compiler and receive a .o file that is just as simple to work with (e.g. to be linked into any library, app, firmware binary, etc.) as if it was produced by Clang on source code written in C: + +``` +$ swiftc *.swift -target arm64-apple-none -no-allocations -wmo -c -o a.o +$ nm -gm a.o +... shows no dependencies beyond memset/memcpy ... +memset +memcpy +``` + +A similar situation is expected even for "allocating" Embedded Swift, except that there will be a need for a small runtime library (significantly smaller compared to the existing Swift runtime written in C++) to support object instantiation and refcounting: + +``` +$ swiftc *.swift -target arm64-apple-none -wmo -c -o a.o +$ nm -gm a.o +... only very limited dependencies ... +malloc +calloc +free +swift_allocObject +swift_initStackObject +swift_initStaticObject +swift_retain +swift_release +``` + +The malloc/calloc/free APIs are expected to be provided by the platform. The Swift runtime APIs will be provided as an implementation that’s optimized for small codesize and will be available as a static library in the toolchain for common CPU architectures. Interestingly, it’s possible to write that implementation in “non-allocating” Baremetal Swift. diff --git a/visions/macros.md b/visions/macros.md new file mode 100644 index 0000000000..90917903c9 --- /dev/null +++ b/visions/macros.md @@ -0,0 +1,515 @@ +# A Vision for Macros in Swift + +As Swift evolves, it gains new language features and capabilities. There are different categories of features: some fill in gaps, taking existing syntax that is not permitted and giving it a semantics that fit well with the existing language, for example conditional conformance or allowing existential values for protocols with `Self` or associated type requirements. Others introduce new capabilities or paradigms to the language, such as the addition of concurrency, ownership types, or comprehensive reflection. + +There is another large category of language features that provide syntactic sugar to eliminate common boilerplate, taking something that can be written out in long-form and making it more concise. Such features don't technically add any expressive power to the language, because you can always write the long-form version, but their effect can be transformational if it enables use cases that would otherwise have been unwieldy. The synthesis of `Codable` conformances, for example, is sheer boilerplate reduction, but it makes `Codable` support ubiquitous throughout the Swift ecosystem. Property wrappers allow one to factor out logic for property access, and have enabled a breadth of powerful libraries. New language features in this category are hard to evaluate, because there is a fundamental question of whether the feature is "worth it": does the set of use cases made better by this feature outweigh the cost of making the language larger and more complicated? + + + +## Democratizing syntactic sugar with macros + +Macros are a feature present in a number of languages that allow one to perform some kind of transformation on the program's input source code to produce a different program. The mechanism of transformation varies greatly, from lexical expansion in C macros, to custom rules that rewrite one syntax into other syntax, to programs that arbitrarily manipulate the abstract syntax tree (AST) of the program. Macro systems exist in C, LISP, Scheme, Scala, Racket, Rust, and a number of other languages, and each design has its own tradeoffs. + +In all of these languages, macros have the effect of democratizing syntactic sugar. Many tasks that would have required a new language feature or an external source-generating tool could, instead, be implemented as a macro. Doing so has trade-offs: many more people can implement a macro than can take a feature through the language's evolution process, but the macro implementation will likely have some compromises---non-ideal syntax, worse diagnostics, worse compile-time performance. Overall, the hope is that a macro system can keep the language smaller and more focused, yet remain expressive because it is extensible enough to support libraries for many different domains. As a project, a macro system should reduce the desire for new syntactic-sugar features, leaving more time for more transformative feature work. Even in the cases where a new language feature is warranted, a macro system can allow more experimentation with the feature to best understand how it should work, and then be "promoted" to a full language feature once we've gained experience from the macro version. + +### Use cases for macros + +There are many use cases for macros, but before we look forward to the new use cases that become possible with macros, let's take a look backward at existing Swift language features that might have been macros had this feature existed before: + +* **`Codable`**: What we think of as `Codable` is mostly a library feature, including the `Encodable` and `Decodable` protocols and the various encoding and decoding implementations. The language part of `Codable` is in the synthesis of `init(from:)`, `encode(to:)`, and `CodingKeys` definitions for structs, enums, and classes. A macro that is given information about the stored properties of a type, and the superclass of a class type, could generate the same implementations---and would be easier to implement, improve, and reason about than a bespoke implementation in the compiler. Synthesis for `Equatable`, `Comparable`, and `Hashable` conformances are similar. +* **String interpolation**: String interpolation is implemented as a series of calls into the string interpolation "builder." While the actual parsing of a string interpolation and matching of it to a type that is `ExpressibleByStringInterpolation` is outside the scope of most macro systems, the syntactic transformation into a set of `appendXXX` calls on the builder is something that could be implemented via a macro. +* **Property wrappers**: Property wrappers are integrated into the language syntax via a custom attribute approach (e.g., `@Clamped(0, 100) var percent: Double`), but the actual implementation of the feature is entirely a syntactic transformation that introduces new properties (e.g., the backing storage property `_percent`) and adds accessors to existing properties (e.g., `percent`'s getter becomes `_percent.wrappedValue`). Other built-in language features like `lazy`, `willSet`, and `didSet` use similar syntactic transformations. +* **Result builders**: Result builders are also integrated into the language syntax via a custom attribute, but the actual transformation applied to a closure is entirely syntactic: the compiler introduces calls into the builder's `buildExpression`, `buildBlock`, `buildOptional`, and so on. That syntactic transformation could be expressed via some form of macro. + +### When is a language feature better than a macro? + +As noted above, a macro system has the potential to replace large parts of existing Swift language features, and enable many new ones. But a macro system is not necessarily a good replacement for a special-built feature: + +* A special-built feature might benefit from custom ABI rules. +* A special-built feature might benefit from analyses that would be infeasible to apply in a macro, such as those dependent on data or control flow. +* A special-built feature might be able to offer substantially better diagnostics. +* A special-built feature might be substantially more efficient to apply because it can rely on information and data structures already in the compiler. +* A special-built feature might have capabilities that we need to deny to macros, lest the mere possibility of a macro applying incur massive compile-time costs. + +The goal of a macro system should be to be general enough to cover a breadth of potential language features, while still providing decent tooling support and discouraging abuse that makes Swift code hard to reason about. + + + +## Design questions for macros + +At a very high level, a macro takes part of the program's source code at compile time and translates it into other source code that is then compiled into the program. There are three fundamental questions about the use of macros: + +* What kind of translation can the macro perform? +* When is a macro expanded? +* How does the compiler expand the macro? + +### What kind of translation can the macro perform? + +A program's source code goes through several different representations as it is compiled, and a macro system can choose at what point in this translation it operates. We consider three different possibilities: + +* **Lexical**: a macro could operate directly on the program text (as a string) or a stream of tokens, and produce a new stream of tokens. The inputs to such a macro would not even have to be valid Swift syntax, which might allow for arbitrary sub-languages to be embedded within a macro. C macros are lexical in nature, and most lexical approaches would inherit the familiar problems of C macros: tooling (such as code completion and syntax highlighting) cannot reason about the inputs to lexical macros, and it's easy for such a macro to produce ill-formed output that results in poor diagnostics. +* **Syntactic**: a macro could operate on a syntax tree and produce a new syntax tree. The inputs to such a macro would be a parsed syntax tree, which is strictly less flexible than a lexical approach because it means the macros can only operate within the bounds of the existing Swift grammar. However, this restriction means that tooling based on the grammar (such as syntax highlighting) would apply to the inputs without having to expand the macro, and macro-using Swift code would follow the basic grammatical structure of Swift. The output of a macro should be a well-formed syntax tree, which will be type-checked by the compiler and integrated into the program. +* **Semantic**: a macro could operate on a type-checked representation of the program, such as an Abstract Syntax Tree (AST) with annotations providing types for expressions, information about which specific declarations are referenced in a function call, any implicit conversions applied to expressions, and so on. Semantic macros have a wealth of additional information that is not provided to lexical or syntactic information, but unlike lexical or syntactic macros, their inputs are restricted to well-typed Swift code. This limits the ability of macros to change the meaning of the code provided to them, which can be viewed both as a negative (less freedom to implement interesting macros) or as a positive (less chance of a macro doing something that confounds the expectations of a Swift programmer). A semantic macro could be required to itself produce a well-typed Abstract Syntax Tree that is incorporated into the program. + +Whichever kind of translation we choose, we will need some kind of language or library that is suitable for working with the program at that level. A lexical translation needs to be able to work with program text, whereas a syntactic translation also needs a representation of the program's syntax tree. Semantic translation requires a much larger part of the compiler, including a representation of the type system and the detailed results of fully type-checked code. + +### When is a macro expanded? + +The expansion of a macro could be initiated in a number of ways, including explicit macro-expansion syntax in the source code or implicit macro expansions that might depend on type checker behavior, e.g., as part of a conversion. The best way for macro expansion to be initiated may depend on the kind of translation that the macro performs: the expansion of a purely lexical or syntactic macro probably needs to be explicitly marked in the source code, because it can change the program structure in surprising ways, whereas a semantic macro might be implicitly expanded as part of type checking because it's working in concert with the type checker. + +Swift already has a syntactic pattern that could be used for explicit macro expansion in the form of the `#` prefix, e.g., as a generalization of language features like `#filePath`, `#line`, `#colorLiteral(red: 0.292, green: 0.081, blue: 0.6, alpha: 255)`, and `#warning("unknown platform")`. The general syntax of `#macroName(macro-arguments)` could be used to expand a macro with the given name and arguments. Doing so provides a clear indication of where macros are used, and would support lexical and/or syntactic macros that need to alter the fundamental syntactic structure of their arguments. We refer to macros written with the prefix `#` syntax as *freestanding macros*, because they act as an expression, declaration, or statement on their own, depending on context. For example, one could build a "stringify" macro that produces both its argument value and also a string representation of the argument's source code: + +```swift +let (value, code) = #stringify(x + y) // produces a tuple containing the result of x + y, and the string "x + y" +``` + +Similarly, Swift's attribute syntax provides an extension point that is already used for features such as property wrappers and result builders. This attribute syntax could be used to expand a macro whose expansion depends on the entity to which the attribute is attached. Therefore, we call these *attached macros*, and they can do things such as create a memberwise initializer for a struct: + +```swift +@MemberwiseInit +struct Point { + var x: Double + var y: Double +} +``` + +A `MemberwiseInit` attached macro would need access to the stored properties of the type it is applied to, as well as the ability to create a new declaration `init(x:y:)`. Such a macro would have to tie in to the compiler at an appropriate point where stored properties are known but the set of initializers has not been finalized. + +A similar approach could be used to synthesize parts of conformances to a protocol. For example, one could imagine that one could write a declaration whose body is implemented by a macro, e.g., + +```swift +protocol Equatable { + @SynthesizeEquatable + func ==(lhs: Self, rhs: Self) -> Bool +} +``` + +The `@SynthesizeEquatable` attribute could trigger a macro expansion when a particular type that conforms to `Equatable` is missing a suitable implementation of `==`. It could access the stored properties in the type used for `Self` so it can synthesize an `==` whose body is, e.g., `lhs.x == rhs.x && lhs.y == rhs.y`. + +There are likely many other places where a macro could be expanded, and the key points for any of them are: + +* What are the macro arguments and how are they evaluated (tokenized, parsed, type-checked, etc.)? +* What other information is available to macro expansion? +* What can the macro produce (statements, expressions, declarations, attributes, etc.) and how is that incorporated into the resulting program? + +### How does the compiler expand the macro? + +The prior design sections have focused on what the inputs and outputs of a macro are and where macros can be triggered, but not *how* the macro operates. Again, there are a number of possibilities: + +* Macros could use a limited textual expansion mechanism, like the C preprocessor. +* Macros could provide a set of pattern-matching rewrite rules, to identify specific syntax and rewrite it into other syntax, like Rust's [`macro_rules!`](https://doc.rust-lang.org/rust-by-example/macros.html). +* Macros could be arbitrary (Swift) code that manipulates a representation of the program (source code, syntax tree, typed AST, etc.) and produces a new program, like [Scala 3 macros](https://docs.scala-lang.org/scala3/guides/macros/macros.html) , [LISP macros](https://lisp-journey.gitlab.io/blog/common-lisp-macros-by-example-tutorial/), or [Rust procedural macros](https://doc.rust-lang.org/reference/procedural-macros.html). + +These options are ordered roughly in terms of increasing expressive power, where the last is the most flexible because one can write arbitrary Swift code to transform the program. The first two options have the benefit of being able to easily evaluate within the compiler, because they are fundamentally declarative in nature. This means that any tool built on the compiler can show the results of expanding a macro, e.g., within an IDE. + +The last option is more complicated, because it involves running arbitrary Swift code. The Swift compiler could conceivably include a complete interpreter for the Swift language, and so long as all of the code that is used in the macro definition is visible to that interpreter (e.g., it does not reference any code for which Swift source is not available), the Swift compiler could interpret the macro definition to produce the expanded result. LISP macros effectively work this way, because LISP is interpreted and can treat the executing program as data. + +Alternatively, the macro definition could be compiled separately from the program that uses the macro, for example into a standalone executable or a compiler plugin. The compiler would then invoke that executable or plugin to perform macro expansion each time it is necessary. This approach is taken both by Scala (which uses the JVM's JIT facilities to be able to compile the macro definition and load it into the compiler) and Rust procedural macros (which use a [`proc-macro`](https://doc.rust-lang.org/reference/linkage.html) crate type for specifically this purpose). A significant benefit of this approach is that the full source code of the macro need not be available as Swift code (so one can use system libraries), macro expansion can be faster (because it's compiled code), and it's easy to test macro definitions outside of the compiler. On the other hand, it means having the Swift compiler run arbitrary code, which opens up questions about security and sandboxing that need to be considered. + +### (Un)hygienic macros + +A [hygienic macro](https://en.wikipedia.org/wiki/Hygienic_macro#cite_note-hygiene-3) is a macro whose expansion cannot change the binding of names used in its macro arguments. For example, imagine the given use of `myMacro`: + +```swift +let x = 3.14159 +let y = 2.71828 +#myMacro(x + y) +``` + +The expression `x + y` is type-checked, and `x` and `y` are bound to local variables immediately above. With a hygienic macro, nothing the macro does can change the declarations to which `x` and `y` are bound. A non-hygienic macro could change these bindings. For example, imagine the macro use above expanded to the following: + +```swift +{ + let x = 42 + return x + y +}() +``` + +Here, the macro introduced a new local variable named `x`. With a hygienic macro, the newly-declared `x` is not found by the `x` in `x + y`: it is a different declaration (or it is not permitted to be introduced). With a non-hygienic macro, the `x` in `x + y` will now refer to the local variable introduced by the macro. In this case, the macro expansion for a non-hygienic macro will fail to type-check because one cannot add an `Int` and a `Double`. + +Hygienic macros do make some macros harder (or impossible) to write, if the macro intentionally wants to take over some of the names used in its arguments. For example, if one wanted to have a macro intercept access to local variables to (say) record the number of times `x` was dynamically accessed. As such, systems that provide hygienic macros often have a way to intentionally provide names from the environment in which the macro is used, such as Racket's [syntax parameters](https://docs.racket-lang.org/reference/stxparam.html). + +A standard approach to dealing with the problem of unintentional name collision in an unhygienic macro is to provide a way to generate unique names within the macro implementation. This approach has been used in LISP macros for decades via [`gensym`](http://clhs.lisp.se/Body/f_gensym.htm), and requires some discipline on the part of the macro implementer to create unique names whenever the macro creates a new declaration. + +## An approach to macros in Swift + +Based on the menu of design choices above, we propose a macro approach characterized by syntactic translation on already-type-checked Swift code that is implemented via a separate package. The intent here is to allow macros a lot of flexibility to implement interesting transformations, while not giving up the benefits of type-checking the code that the user wrote prior to translation. + +### Macro declarations + +A macro declaration indicates how the macro can be used in source code, much like a function declaration indicates the arguments and result type of a function. We declare macros as a new kind of entity, introduced with `macro`, that indicates that it's a macro definition and provides additional information about the interface to the macro. For example, consider the `stringify` macro described early, which could be defined as follows: + +```swift +@freestanding(expression) +macro stringify(_ value: T) -> (T, String) = #externalMacro(module: "MyMacros", type: "StringifyMacro") +``` + +The `macro` introducer indicates that this is a macro, named `stringify`. The `@freestanding` attribute notes that this is a freestanding macro (used with the `#` syntax) and that it is usable as an expression. The macro is defined (after the `=`) to have an externally-provided macro expansion operation that is the type named `MyMacros.StringifyMacro`. Because the definition is external, the `stringify` macro function doesn't need a function body. If Swift were to grow a way to implement macros directly here (rather than via a separate package), such macros could have a body but not an `#externalMacro` argument. + +A given macro can inhabit several different macro *roles*, each of which can expand in different ways. For example, consider a `Clamping` macro that implements behavior similar to the [property wrapper by the same name](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md#clamping-a-value-within-bounds): + +```swift +@attached(peer, prefixed(_)) +@attached(accessor) +macro Clamping(min: T, max: T) = #externalMacro(module: "MyMacros", type: "ClampingMacro") +``` + +The attribute specifies that this is an attached macro, so it can be used as an attribute as, e.g., + +```swift +@Clamping(min: 0, max: 255) var red: Int = 127 +``` + +The `Clamping` macro would be expanded in two different but complementary ways: + +* A *peer* declaration `_red` that provides the backing storage: + + ```swift + private var _red: Int = 127 + ``` + +* A set of *accessor*s that guard access to this storage, turning the `red` property into a computed property: + + ```swift + get { _red } + + set(__newValue) { + let __minValue = 0 + let __maxValue = 255 + if __newValue < __minValue { + _red = __minValue + } else if __newValue > __maxValue { + _red = __maxValue + } else { + _red = __newValue + } + } + ``` + + +### Macro definitions via a separate program + +Macro definitions would be provided in a separate program that performs a syntactic transformation. A macro definition would be implemented using [swift-syntax](https://github.com/apple/swift-syntax), by providing a type that conforms to one of the "macro" protocols in a new library, `SwiftSyntaxMacros`. For example, the `MyMacros` package we're using as an example might look like this: + +```swift +import SwiftDiagnostics +import SwiftParser +import SwiftSyntax +import SwiftSyntaxBuilder +import SwiftSyntaxMacros + +public struct StringifyMacro: ExpressionMacro { + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + guard let argument = node.argumentList.first?.expression else { + fatalError("compiler bug: the macro does not have any arguments") + } + + return "(\(argument), \(literal: argument.description))" + } +} +``` + +Conformance to `ExpressionMacro` indicates a macro definition for an expression macro, and corresponds to `@freestanding(expression)`. There will be several protocols, corresponding to the various roles that macros inhabit. Each protocol has an `expansion` method that will be called with the syntax nodes that are involved in the macro expansion, along with a `context` instance that provides more information about how the macro is being invoked. + +The implementation of these functions makes extensive use of Swift syntax manipulation via the `swift-syntax` package. The inputs and outputs are in terms of syntax nodes: `ExprSyntax` describes the syntax for any kind of expression in Swift, whereas `MacroExpansionExprSyntax` is the syntax for an explicitly-written macro expansion. The `expansion` operation will return a new syntax node that will replace the ones it was given in the program. We use string interpolation as a form of quasi-quoting: the return of `StringifyMacro.expansion` forms a tuple `(\(argument), "\(literal: argument.description)")` where the first argument is the expression itself and the second is the source code translated into a string literal. The resulting string will be parsed into an expression that is returned to the compiler. + +Macro implementations are "host" programs that are completely separate from the program in which macros are used. This distinction is most apparent in cross-compilation scenarios, where the host platform (where the compiler is run) differs from the target platform (where the compiled program will run), and these could use different operating systems and processor architectures. Macro implementations are compiled for and executed on the host platform, whereas the results of expanding a macro will be compiled for and executed on the target platform. Therefore, macro implementations are defined as their own kind of target in the Swift package manifest. For example, a package that ties together the macro declaration for `#stringify` and its implementation as `StringifyMacro` follows: + +```swift +import PackageDescription +import CompilerPluginSupport + +let package = Package( + name: "MyMacros", + dependencies: [ + .package( + url: "https://github.com/apple/swift-syntax.git", + branch: "main" + ), + ], + targets: [ + // Macro implementation target contains the StringifyMacro type. + // Always built for the host platform. + .macro(name: "StringifyImpl", + dependencies: [.product(name: "SwiftSyntaxMacros", package: "swift-syntax")]), + + // Library target provides the macro declaration (public macro stringify) that is + // used by client code. + // Built for the target platform. + .target(name: "StringifyLib", dependencies: ["StringifyImpl"]), + + // Clients of the macro will depend on the library target. + .executableTarget(name: "StringifyClient", dependencies: ["StringifyLib"]), + ] +) +``` + +Conceptually, the separation of `macro` targets into separate programs (for the host platform) from other targets (for the target platform) means that the individual macros could be built completely separately from each other and from other targets, even if they happen to be for the same platform. In the extreme, this could mean that each `macro` would be allowed to build against a different version of `swift-syntax`, and other targets could choose to also use `swift-syntax` with a different version. Given that `swift-syntax` is modeling the Swift language (which evolves over time), it does not promise a stable API, so having the ability to have different macro implementations depend on different versions of `swift-syntax` is a feature: it would prevent conflicting version requirements in macros from causing problems. + +Note that this separation of dependencies for distinct targets is currently not possible in the Swift Package Manager. In the interim, macro implementations will need to adapt to be built with different versions of the `swift-syntax` package. + +#### Diagnostics + +A macro implementation can be used to produce diagnostics (e.g., warnings and errors) to indicate problems encountered during macro expansion. The `stringify` macro described above doesn't really have a failure case, but imagine an `#embed("somefile.txt")` macro that takes the contents of a file at build time and turns them into an array of bytes. The macro could have several different failure modes: + +* The macro argument isn't a string literal, so it doesn't know what the file name is. +* The file might not be available for reading because it is missing, inaccessible, etc. + +These failures would be reported as errors by providing [`Diagnostic`](https://github.com/apple/swift-syntax/blob/main/Sources/SwiftDiagnostics/Diagnostic.swift) instances to the context that specify the underlying problem. The diagnostics would refer to the syntax nodes provided to the macro definition, and the compiler would provide those diagnostics to the user. + +In its limit, a macro might perform no translation whatsoever on the syntax tree it is given, but instead be there only to provide diagnostics---for example, as a context-specific, custom lint-like rule that enforces additional constraints on the program. + +### Macro roles + +The `@freestanding` and `@attached` attributes for macro declarations specify the roles that the macro can inhabit, each of which corresponds to a different place in the source code where the macro can be expanded. Here is a potential set of roles where macro expansion could be warranted. The set of roles could certainly grow over time to enable new capabilities in the language: + +* **Expression**: A freestanding macro that can occur anywhere that an expression can occur, and must produce an expression. `#colorLiteral` could fall into this category: + + ```swift + // In library + @freestanding(expression) + macro colorLiteral(red: Double, green: Double, blue: Double, alpha: Double) -> _ColorLiteralType = + #externalMacro(module: "MyMacros", type: "ColorLiteral") + + // In macro definition package + public struct ColorLiteral: ExpressionMacro { + public static func expansion( + expansion: MacroExpansionExprSyntax, + in context: MacroExpansionContext + ) -> ExprSyntax { + return ".init(\(expansion.argumentList))" + } + } + ``` + + With this, an expression like `#colorLiteral(red: 0.5, green: 0.5, blue: 0.25, alpha: 1.0)` would produce a value of the `_ColorLiteralType` (presumably defined by a framework), and would be rewritten by the macro into `.init(red: 0.5, green: 0.5, blue: 0.25, alpha: 1.0)` and type-checked with the `_ColorLiteralType` as context so it would initialize a value of that type. + +* **Declaration**: A freestanding macro that can occur anywhere a declaration can occur, such as at the top level, in the definition of a type or extension thereof, or in a function or closure body. The macro can expand to zero or more declarations. These macros could be used to subsume the `#warning` and `#error` directives from [SE-0196](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md): + + ```swift + /// Emits the given message as a warning, as in SE-0196. + @freestanding(declaration) macro warning(_ message: String) + + /// Emits the given message as an error, as in SE-0196. + @freestanding(declaration) macro error(_ message: String) + ``` + +* **Code item**: A freestanding macro that can occur within a function or closure body and can produce a mix of zero or more statements, expressions, and declarations. + +* **Accessor**: An attached macro that adds accessors to a stored property or subscript, as shown by the `Clamping` macro example earlier. The inputs would be the arguments provided to the macro in the attribute, along with the property or subscript declaration to which the accessors will be attached. The output would be a set of accessor declarations, i.e., a getter and setter. A `Clamping` macro could be implemented as follows: + + ```swift + extension ClampingMacro: AccessorMacro { + static func expansion( + of node: CustomAttributeSyntax, + providingAccessorsOf declaration: DeclSyntax, + in context: MacroExpansionContext + ) throws -> [AccessorDeclSyntax] { + let originalName = /* get from declaration */, + minValue = /* get from custom attribute node */, + maxValue = /* get from custom attribute node */ + let storageName = "_\(originalName)", + newValueName = context.getUniqueName(), + maxValueName = context.getUniqueName(), + minValueName = context.getUniqueName() + return [ + """ + get { \(storageName) } + """, + """ + set(\(newValueName)) { + let \(minValueName) = \(minValue) + let \(maxValueName) = \(maxValue) + if \(newValueName) < \(minValueName) { + \(storageName) = \(minValueName) + } else if \(newValueName) > \(maxValueName) { + \(storageName) = \(maxValueName) + } else { + \(storageName) = \(newValueName) + } + } + """ + ] + } + } + ``` + +* **Witness**: An attached macro that can be expanded to provide a "witness" that satisfies a requirement of a protocol for a given concrete type's conformance to that protocol. Such a macro would take as input the conforming type, the protocol, and a declaration (without a body) that will be created in the conforming type. The output would be that declaration with a body added and (potentially) other modifications. For this to work well, we would almost certainly need to expose a lot of information about the conforming type, such as the set of stored properties. Assuming that exists, let's implement the `synthesizeEquatable` macro referenced earlier in this document: + + ```swift + // In the standard library + @attached(witness) + macro SynthesizeEquatable() = #externalMacro(module: "MyMacros", type: "EquatableSynthesis") + + protocol Equatable { + @SynthesizeEquatable + static func ==(lhs: Self, rhs: Self) -> Bool + } + + // In the macro definition library + struct EquatableSynthesis: AttachedMacro { + /// Expand a macro described by the given custom attribute to + /// produce a witness definition for the requirement to which + /// the attribute is attached. + static func expansion( + of node: CustomAttributeSyntax, + witness: DeclSyntax, + conformingType: TypeSyntax, + storedProperties: [StoredProperty], + in context: MacroExpansionContext + ) throws -> DeclSyntax { + let comparisons: [ExprSyntax] = storedProperties.map { property in + "lhs.\(property.name) == rhs.\(property.name)" + } + let comparisonExpr: ExprSyntax = comparisons.map { $0.description }.joined(separator: " && ") + return witness.withBody( + """ + { + return \(comparisonExpr) + } + """ + ) + } + } + ``` + +* **Member**: An attached macro that can be applied on a type or extension that expands to one or more declarations that will be inserted as members into that type or extension. As with a conformance macro, a member macro would probably want access to the stored properties of the enclosing type, and potentially other information. As an example, let's create a macro to synthesize a memberwise initializer: + + ```swift + // In the standard library + @attached(member) + macro memberwiseInit(access: Access = .public) = #externalMacro(module: "MyMacros", type: "MemberwiseInit") + + // In the macro definition library + struct MemberwiseInit: MemberMacro { + static func expansion( + of node: AttributeSyntax, + attachedTo declaration: DeclSyntax, + in context: inout MacroExpansionContext + ) throws -> [DeclSyntax] {} + let parameters: [FunctionParameterSyntax] = declaration.storedProperties.map { property in + let paramDecl: FunctionParameterSyntax = "\(property.name): \(property.type)" + guard let initializer = property.initializer else { + return paramDecl + } + return paramDecl.withDefaultArgument( + InitializerClauseSyntax( + equal: TokenSyntax(.equal, presence: .present), + value: "\(initializer)" + ) + ) + } + + let assignments: [ExprSyntax] = conformingType.storedProperties.map { property in + "self.\(property.name) = \(property.name)" + } + + return + #""" + public init(\(parameters.map { $0.description }.joined(separator: ", "))) { + \(assignments.map { $0.description }.joined(separator: "\n")) + } + """# + } + } + ``` + + Using this macro on a type, e.g., + + ```swift + @MemberwiseInit + class Point { + var x, y: Int + var z: Int = 0 + } + ``` + + would produce code like the following: + + ```swift + public init(x: Int, y: Int, z: Int = 0) { + self.x = x + self.y = y + self.z = z + } + ``` + +* **Body**: A body macro would allow one to create or replace the body of a function, initializer, or closure through syntactic manipulation. Body macros are attached to one of these entities, e.g., + + ```swift + @Traced(logLevel: 2) + func myFunction(a: Int, b: Int) { ... } + ``` + + where the `Traced` macro is declared as something like: + + ```swift + @attached(body) macro Traced(logLevel: Int = 0) + ``` + + and can introduce new code into the body to, e.g., perform logging. + +* **Conformance**: Conformance macros could introduce protocol conformances to the type or extension to which they are attached. For example, this could be useful when composed with macro roles that create other members, such as a macro that both adds a protocol conformance and also a stored property required by that conformance. + +## Tools for using and developing macros + +Macros introduce novel problems for tooling, because the macro expansion process replaces (or augments) code that is explicitly written with other source code that makes it into the final program. The design of a macro system has a large impact on the ability to build good tools, and a poor design can directly impact discoverability, predictability, debuggability, and compile-time efficiency. C macros demonstrate nearly all of these problems: + +* C macros are bare identifiers that can be used anywhere in source code, so it is hard to discover where macros are being applied in the source code. C programmers have adopted the `UPPERCASE_MACRO_NAME` convention to try to understand which names are macros and which aren't. +* C macros can expand to an arbitrary sequence of tokens in a manner that destroys program structure, for example, one can close a `struct` or function definition with a C macro, making it hard to predict the scope of effects a macro can have. +* C macros are expanded via logic within the compiler's preprocessor, and therefore offer no debugging capabilities. The only way to see the effect of a macro is to generate preprocessed output for an entire translate unit, then inspect the original code. +* C macros are rarely persisted after a program is built, so debugging a program that has made heavy use of macros requires one to manually map between the original source code (pre-macro) and the generated machine code, with no record of the expansion itself. + +The design proposed here for Swift makes it possible to build good tooling despite the challenges macros pose: + +* Uses of macros are indicated in the source (with `#` or `@`) to make the use of macros clear. +* Expansions of macros have their effects restricted to the scope in which the macro is used (e.g., augmenting or adding declarations locally), and any effects visible from other parts of the program are declared up front by the macro (e.g., the names it introduces), so one can reason about the effects of a macro expansion. +* Implementations of macros are normal Swift programs, so they can be developed and tested using the normal tools for Swift. Much of the development and testing of a macro can be done outside of the compiler, with unit tests that (for example) test the syntactic transformation on isolated examples that translate Swift code into different Swift code. +* The localized nature of macro effects, and the fact that all macro-expanded code is itself normal Swift code, make it possible to record the results of macro expansion in a way that can reconstitute the effects of macro expansion without rerunning the compiler, allowing useful debugging and diagnostics flows. + +Early implementations of Swift macros already provide macro-expansion information in a manner that is amenable to existing tooling. For example, the result of expanding a macro fits into several existing workflows: + +* If a warning or error message refers into code generated by macro expansion, the compiler writes the macro-expanded code into a separate Swift file and refers to that file within the diagnostic message. Users can open that file to see the results of that macro expansion to understand the problem. Each such diagnostic provides a stack of notes that refers back to the point where the macro expansion occurred, which may itself be within other macro-expanded code, all the way back to the original source code that triggered the outermost macro expansion. For example: + + ``` + /tmp/swift-generated-sources/@__swiftmacro_9MacroUser14testAddBlocker1a1b1c2oaySi_S2iAA8OnlyAddsVtF03addE0fMf1_.swift:1:4: error: binary operator '-' cannot be applied to two 'OnlyAdds' operands + oa - oa + ~~ ^ ~~ + macro_expand.swift:200:7: note: in expansion of macro 'addBlocker' here + _ = #addBlocker(oa + oa) + ^~~~~~~~~~~~~~~~~~~~ + ``` + +* Debug information for macro-expanded code uses a similar scheme to diagnostics, allowing one to see the macro-expanded code while debugging (e.g., in a backtrace), set breakpoints in it, step through it, and so on. Freestanding macros are treated like inline functions, so one can "step into" a macro expansion from the place it occurs in the source code. + +* SourceKit, which is used for IDE integration features, provides a new "Expand Macro" refactoring that can expand any single use of a macro right in the source code. This can be used both to see the effects of the macro, as well as to eliminate the use of the macro in favor of (say) a customized version where the macro is used as a starting point. + +* Macro implementations can be tested using `swift-syntax` and existing testing tooling. For example, the following test checks the expansion behavior of the `stringify` macro by performing syntactic expansion on the input source code (`sf`) and checking that against the expected result of expansion (in the `XCTAssertEqual` at the end): + + ```swift + final class MyMacroTests: XCTestCase { + func testStringify() { + let sf: SourceFileSyntax = + #""" + let a = #stringify(x + y) + let b = #stringify("Hello, \(name)") + """# + let context = BasicMacroExpansionContext.init( + sourceFiles: [sf: .init(moduleName: "MyModule", fullFilePath: "test.swift")] + ) + let transformedSF = sf.expand(macros: ["stringify": StringifyMacro.self], in: context) + XCTAssertEqual( + transformedSF.description, + #""" + let a = (x + y, "x + y") + let b = ("Hello, \(name)", #""Hello, \(name)""#) + """# + ) + } + } + ``` + +All of the above work with existing tooling, creating a baseline development experience that provides discoverability, predictability, and debuggability. Over time, more tooling can be made aware of macros to provide a more polished experience. diff --git a/visions/memory-safety.md b/visions/memory-safety.md new file mode 100644 index 0000000000..69a206bc79 --- /dev/null +++ b/visions/memory-safety.md @@ -0,0 +1,255 @@ +# Optional Strict Memory Safety for Swift + +Swift is a memory-safe language *by default* , meaning that the major language features and standard library APIs are memory-safe. However, it is possible to opt out of memory safety when it’s pragmatic using certain “unsafe” language or library constructs. This document proposes a path toward an optional “strict” subset of Swift that prohibits any unsafe features. This subset is intended to be used for Swift code bases where memory safety is an absolute requirement, such as security-critical libraries. + +This document is an official feature [vision document](https://forums.swift.org/t/the-role-of-vision-documents-in-swift-evolution/62101). The Language Steering Group has endorsed the goals and basic approach laid out in this document. This endorsement is not a pre-approval of any of the concrete proposals that may come out of this document. All proposals will undergo normal evolution review, which may result in rejection or revision from how they appear in this document. + +## Introduction + +[Memory safety](https://en.wikipedia.org/wiki/Memory_safety) is a popular topic in programming languages nowadays. Essentially, memory safety is a property that prevents programmer errors from manifesting as [undefined behavior](https://en.wikipedia.org/wiki/Undefined_behavior) at runtime. Undefined behavior effectively breaks the semantic model of a language, with unpredictable results including crashes, data corruption, and otherwise-impossible program states. Much of the recent focus on memory safety is motivated by security, because memory safety issues offer a fairly direct way to compromise a program: in fact, the lack of memory safety in C and C++ has been found to be the root cause for ~70% of reported security issues in various analyses [[1](https://msrc.microsoft.com/blog/2019/07/a-proactive-approach-to-more-secure-code/)][[2](https://www.chromium.org/Home/chromium-security/memory-safety/)]. + +### Memory safety in Swift + +While there are a number of potential definitions for memory safety, the one provided by [this blog post](https://security.apple.com/blog/towards-the-next-generation-of-xnu-memory-safety/) breaks it down into five dimensions of safety: + +* **Lifetime safety** : all accesses to a value are guaranteed to occur during its lifetime. Violations of this property, such as accessing a value after its lifetime has ended, are often called use-after-free errors. +* **Bounds safety**: all accesses to memory are within the intended bounds of the memory allocation, such as accessing elements in an array. Violations of this property are called out-of-bounds accesses. +* **Type safety** : all accesses to a value use the type to which it was initialized, or a type that is compatible with that type. For example, one cannot access a `String` value as if it were an `Array`. Violations of this property are called type confusions. +* **Initialization safety** : all values are initialized prior to being used, so they cannot contain unexpected data. Violations of this property often lead to information disclosures (where data that should be invisible becomes available) or even other memory-safety issues like use-after-frees or type confusions. +* **Thread safety:** all values are accessed concurrently in a manner that is synchronized sufficiently to maintain their invariants. Violations of this property are typically called data races, and can lead to any of the other memory safety problems. + +Since its inception, Swift has provided memory safety for the first four dimensions. Lifetime safety is provided for reference types by automatic reference counting and for value types via [memory exclusivity](https://www.swift.org/blog/swift-5-exclusivity/); bounds safety is provided by bounds-checking on `Array` and other collections; type safety is provided by safe features for casting (`as?` , `is` ) and `enum` s; and initialization safety is provided by “definite initialization”, which doesn’t allow a variable to be accessed until it has been defined. Swift 6’s strict concurrency checking extends Swift’s memory safety guarantees to the last dimension. + +Providing memory safety does not imply the absence of run-time failures. Good language design often means defining away runtime failures in the type system. However, memory safely requires only that an error in the program cannot be escalated into a violation of one of the safety properties. For example, having reference types be non-nullable by default defines away most problems with NULL pointers. With explicit optional types, the force-unwrap operator (postfix `!` ) meets the definition of memory safety by trapping at runtime if the unwrapped optional is `nil` . The standard library also provides the [`unsafelyUnwrapped` property](https://developer.apple.com/documentation/swift/optional/unsafelyunwrapped) that does not check for `nil` in release builds: this does not meet the definition of memory safety because it admits violations of initialization and lifetime safety that could be exploited. + +### Unsafe code + +Swift is a memory-safe language *by default* , meaning that the major language features and standard library APIs are memory-safe. However, there exist opt-outs that allow one to write memory-unsafe code in Swift: + +* Language features like `unowned(unsafe)` and `nonisolated(unsafe)` that disable language safety features locally. +* Library constructs like `UnsafeMutableBufferPointer` or `unsafeBitCast(to:)` that provide lower-level access than existing language constructs provide. +* Interoperability with C-family APIs, which are implemented in a non-memory-safe language and tend to traffic in unsafe pointer types. + +The convention of using `unsafe` or `unchecked` in the names of unsafe constructs works fairly well in practice: memory-unsafe code in Swift tends to sticks out because of the need for `withUnsafe<...>` operations, and for large swaths of Swift code there is no need to reach down for the unsafe APIs. + +However, the convention is not entirely sufficient for identifying all Swift code that makes use of unsafe constructs. For example, it is possible to call the C `memcpy` directly from Swift as, e.g., `memcpy(&to, &from, numBytes)` , which can easily violate memory-safety along any dimension: `to` and `from` might be arrays with incompatible types, the number of bytes might be incorrect, etc. However, “unsafe” or “unchecked” do not appear in this code except as the (unseen) type of the parameters to `memcpy` . + +Moreover, some tasks require lower-level access to memory that is only expressible today via the unsafe pointer types, meaning that one must choose between using only safe constructs, or having access to certain APIs and optimizations. For example, all access to contiguous memory requires an `UnsafeMutableBufferPointer` , which compromises on both lifetime and bounds safety. However, it fulfills a vital role for various systems-programming tasks, including interacting directly with specialized hardware or using lower-level system libraries written in the C family. + +## Strictly-safe subset of Swift + +Swift’s by-default memory safety is a pragmatic choice that provides the benefits of memory safety to most Swift code while not requiring excessive ceremony for those places where some code needs to drop down to use unsafe constructs. However, there are code bases where memory safety is more important than programmer convenience, such as in security-critical subsystems handling untrusted data or that are executing with elevated privileges in an OS. + +For such code bases, it’s important to ensure that the code is staying within the strictly-safe subset of Swift. This can be accomplished with a compiler option that produces an error for any use of unsafe code, whether it’s an unsafe language feature or unsafe library construct. Any code written within this strictly-safe subset also works as “normal” Swift and can interoperate with existing Swift code. + +The compiler would flag any use of the following unsafe language features: + +* `@unchecked Sendable` +* `unowned(unsafe)` +* `nonisolated(unsafe)` +* `unsafeAddressor`, `unsafeMutableAddressor` + +In addition, an `@unsafe` attribute would be added to the language and would be used to mark any declaration that is unsafe to use. In the standard library, the following functions and types would be marked `@unsafe` : + +* `Unsafe(Mutable)(Raw)(Buffer)Pointer` +* `(Closed)Range.init(uncheckedBounds:)` +* `OpaquePointer` +* `CVaListPointer` +* `Unmanaged` +* `unsafeBitCast`, `unsafeDowncast` +* `Optional.unsafelyUnwrapped` +* `UnsafeContinuation`, `withUnsafe(Throwing)Continuation` +* `UnsafeCurrentTask` +* `Mutex`'s `unsafeTryLock`, `unsafeLock`, `unsafeUnlock` +* `VolatileMappedRegister.init(unsafeBitPattern:)` +* The `subscript(unchecked:)` introduced by the `Span` proposal. + +Any use of these APIs would be flagged by the compiler as a use of an unsafe construct. In addition to the direct `@unsafe` annotation, any API that uses an `@unsafe` type is considered to itself be unsafe. This includes C-family APIs that use unsafe types, such as the aforementioned `memcpy` that uses `Unsafe(Mutable)RawPointer` in its signature: + +```swift +func memcpy( + _: UnsafeMutableRawPointer?, + _: UnsafeRawPointer?, + _: Int +) -> UnsafeMutableRawPointer? +``` + +The rules described above make it possible to detect and report the use of unsafe constructs in Swift. + +An `@unsafe` function is allowed to use other unsafe constructs. As such, a Swift module compiled in the strictly-safe subset can contain both safe and unsafe code, but all unsafe code is marked by `@unsafe`. A client of the module can opt to use only the safe parts of that module, potentially using the strict safety checking to ensure this. + +### Wrapping unsafe behavior in safe APIs + +There should also be a way to wrap unsafe behavior into safe APIs. For example, the standard library's `Array` and [`Span`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md) are necessarily implemented from unsafe primitives, such as `UnsafeRawPointer`, but expose primarily safe APIs. For example, the `Span` type could be defined like this: + +```swift +public struct Span: ~Escapable, Copyable, BitwiseCopyable { + internal let buffer: UnsafeBufferPointer +} +``` + +The subscript operation is safe, but necessarily uses `buffer`, which has an `@unsafe` type. Its implementation must acknowledge that it is using unsafe constructs internally, but that it does so in a manner that preserves safety for clients. There are several potential syntaxes, including an `unsafe { ... }` code block, which could look like this: + +```swift +public subscript(_ position: Int) -> Element { + get { + unsafe { + precondition(position >= 0 && position < buffer.count) + return buffer[position] + } + } +} +``` + +Alternatively, Swift could provide a `@safe(unchecked)` attribute that states that a particular API is safe, but that its safety cannot be checked by the compiler, akin to `@unchecked Sendable` conformances: + +```swift +public subscript(_ position: Int) -> Element { + @safe(unchecked) get { + precondition(position >= 0 && position < buffer.count) + return buffer[position] + } +} +``` + +The specific syntax chosen will be the subject of a specific proposal, and need not be determined by this document. Regardless, a Swift module that enables strict safety checking must limit its use of unsafe constructs to `@unsafe` declarations or those parts of the code that have acknowledged local use of unsafe constructs. + +### Auditability + +The aim of optional strict memory safety for Swift is to make it possible to write Swift that avoids unintentional use of unsafe constructs while not preventing their use entirely. To aid projects that wish to set a higher bar for memory safety, such as permitting no unsafe constructs outside of the standard library or requiring additional code review for any uses of unsafe constructs, Swift tooling should provide a way to audit the uses of unsafe constructs within an entire project (including its dependencies). An auditing tool should be able to identify and report Swift modules that were compiled without strict memory safety as well as all of the places where the opt-out mechanism (e.g., `unsafe { ... }` blocks or `@safe(unchecked)`) is used in modules that do opt in to strict memory safety. + +## Improving the expressibility of strictly-safe Swift + +The following sections describe language features and library constructs that improve on what can be expressed within the strictly-safe subset of Swift. These improvements will also benefit Swift in general, making it easier to correctly work with contiguous memory and interoperate with APIs from the C-family on languages. + +### Accessing contiguous memory + +Nearly every “unsafe” language feature and standard library API described in the previous section already has safe counterparts in the language: safe concurrency patterns via actors and `Mutex` , safe casting via `as?` , runtime-checked access to optionals (via `!` ) and continuations (`withChecked(Throwing)Continuation` ), and so on. + +One of the primary places where this doesn’t hold is with low-level access to contiguous memory. Even with `ContiguousArray` , which stores its elements contiguously, the only way to access elements is either one-by-one (e.g., subscripting) or to use an operation like `withUnsafeBufferPointer` that provides temporary access the storage via an `Unsafe(Mutable)BufferPointer` argument to a closure. These APIs are memory-unsafe along at least two dimensions: + +* **Lifetime safety**: the unsafe buffer pointer should only be used within the closure, but there is no checking to establish that the pointer does not escape the closure. If it does escape, it could be used after the closure has returned and the pointer could have effectively been “freed.” +* **Bounds safety**: the unsafe buffer pointer types do not perform bounds checking in release builds. + +[Non-escapable types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md) provide the ability to create types whose instances cannot escape out of the context in which they were created with no runtime overhead. Non-escapable types allow the creation of a [memory-safe counterpart to the unsafe buffer types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md), `Span` . With `Span` , it becomes possible to access contiguous memory in an array in a manner that maintains memory safety. For example: + +```swift +let span = myInts.span + +globalSpan = span // error: span value cannot escape the scope of myInts +print(span[myInts.count]) // runtime error: out-of-bounds access +print(span.first ?? 0) +``` + +[Lifetime dependencies](https://github.com/swiftlang/swift-evolution/pull/2305) can greatly improve the expressiveness of non-escaping types, making it possible to build more complex data structures while maintaining memory safety. + +### Expressing memory-safe interfaces for the C family of languages + +The C family of languages do not provide memory safety along any of the dimensions described in this document. As such, a Swift program that makes use of C APIs is never fully “memory safe” in the strict sense, because any C code called from Swift could undermine the memory safety guarantees Swift is trying to provide. Requiring that all such C code be rewritten in Swift would go against Swift’s general philosophy of incremental adoption into existing ecosystems. Therefore, this document proposes a different strategy: code written in Swift will be auditably memory-safe so long as the C APIs it uses follow reasonable conventions with respect to memory safety. As such, writing new code (or incrementally rewriting code from the C family) will not introduce new memory safety bugs, so that adopting Swift in an existing code base will incrementally improve on memory safety. This approach is complementary to any improvements made to memory safety within the C family of languages, such as [bounds-safety checks for C](https://clang.llvm.org/docs/BoundsSafety.html) or [C++ standard library hardening](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2024/p3471r0.html). + +In the C family of languages, the primary memory safety issue for APIs is the widespread use of pointers that have neither lifetime annotations (who owns the pointer?) nor bounds annotations (how many elements does it point to?). As such, the pointers used in C APIs are reflected in Swift as unsafe pointer types, as shown above with `memcpy` . + +Despite the lack of this information, C APIs often follow a reasonable set of conventions that make them usable in Swift without causing memory-safety problems. Swift has a long history of utilizing annotations in C headers to describe these conventions and improve the projection of C APIs into Swift, including: + +* Nullability annotations (`_Nullable`, `_Nonnull`) that describe what values can be NULL, and affects whether a C type is reflected as optional in Swift. +* Non-escaping annotations (e.g., `__attribute__((noescape))`) on block pointer parameters, which results in them being imported as non-escaping function parameters. +* `@MainActor` and `Sendable` annotations on C APIs that support Swift 6’s data-race safety model. + +To provide safer interoperability with C APIs, additional annotations can be provided in C that Swift can use to project those C APIs into Swift APIs without any use of unsafe pointers. For example, the Clang [bounds-safety attributes](https://clang.llvm.org/docs/BoundsSafety.html) allow one to express when a C pointer’s size is described by another value: + +```cpp +double average(const double *__counted_by(N) ptr, int N); +``` + +Today, this function would be projected into a Swift function like the following: + +```swift +/*@unsafe*/ func average(_ ptr: UnsafePointer!, _ N: CInt) -> Double +``` + +However, Swift could use the `__counted_by` attribute to provide a more convenient API that bundles the count and length together, e.g., + +```swift +/*@unsafe*/ func average(_ ptr: UnsafeBufferPointer) -> Double +``` + +Now, a Swift caller that passes a local `Double` array would not need to pass the count separately, and cannot get it wrong: + +```swift +var values = [3.14159, 2.71828] +average(values) // ok, no need to pass count separately +``` + +This call is still technically unsafe, because we’re passing a temporary pointer into the array’s storage down to the `average` function. That function could save that pointer into some global variable that gets accessed some time after the call, causing a memory safety violation. The actual implementation of `average` is unlikely to do so, and could express this constraint using the existing `noescape` attribute as follows: + +```cpp +double average(const double *__counted_by(N) __attribute__((noescape)) ptr, int N); +``` + +The `average` function is now expressing that it takes in a `double` pointer referencing `count` values but will not retain the pointer beyond the call. These are the semantic requirements needed to provide a memory-safe Swift projection as follows: + +```swift +func average(_ ptr: Span) -> Double +``` + +More expressive Swift lifetime features can also have corresponding C annotations, allowing more C APIs to be reflected into safe APIs in Swift. For example, consider a C function that finds the minimal element in an array and returns a pointer to it: + +```cpp +const double *min_element(const double *__counted_by(N) __attribute__((noescape)) ptr, int N); +``` + +The returned pointer will point into the buffer passed in, so its lifetime is tied to that of the pointer argument. The aforementioned [lifetime dependencies proposal](https://github.com/swiftlang/swift-evolution/pull/2305) allows this kind of dependency to be expressed in Swift, where the resulting non-escaping value (e.g., a `Span` containing one element) has its lifetime tied to the input argument. Clang provides a [`lifetimebound`](https://clang.llvm.org/docs/AttributeReference.html#id11) attribute that expresses when a return value refers into memory associated with one of the parameters, which offers one way to express this lifetime relationship for C APIs: + +```c +const double * _Nullable __counted_by(1) +min_element(const double *__counted_by(N) __attribute__((noescape)) __attribute__((lifetimebound)) ptr, int N); +``` + +The result could be the following memory-safe Swift API: + +```swift +@lifetime(ptr) func min_element(_ ptr: Span) -> Span? +``` + +### Affordances for C++ interoperability + +C++ offers a number of further opportunities for improved safety by modeling lifetimes. For example, `std::vector` has a `front()` method that returns a reference to the element at the front of the vector: + +```cpp +const T& front() const; +``` + +The returned reference is valid so long as the vector instance still exists and has not been modified since the call to `front()`. Describing that lifetime dependency in C++ (for example, with the aforementioned `lifetimebound` attribute) would lead to a safe mapping of this API into Swift without the need to introduce an extra copy of the returned element, improving both safety and, potentially, performance. + +The C++ [`std::span`](https://en.cppreference.com/w/cpp/container/span) type is similar to the Swift `Span` type, in that it also carries both a pointer and bounds to describe a region of memory. However, `std::span` doesn't provide lifetime safety, so it is essentially an unsafe type from the Swift perspective. The same C attributes that provide lifetime safety for C pointers and references could be applied to `std::span` instances to provide safe Swift projections of C++ APIs. For example, the following annotated C++ API: + +```c++ +std::span substring_match( + std::span sequence [[clang::lifetimebound]], + std::span subsequence [[clang::noescape]] +); +``` + +could be imported into Swift as: + +```swift +@lifetime(sequence) +func substring_match(_ sequence: Span, _ subsequence: Span) -> Span +``` + +## Incremental adoption + +The introduction of any kind of additional checking into Swift requires a strategy that accounts for the practicalities of adoption within the Swift ecosystem. Different developers adopt new features on their own schedules, and some Swift code will never enable new checking features. Therefore, it is important that a given Swift module can adopt the proposed strict safety checking without requiring any module it depends on to have already done so, and without breaking any of its own clients that have not enabled strict safety checking. + +The optional strict memory safety model proposed by this lends itself naturally to incremental adoption. The proposed `@unsafe` attribute is not part of the type of the declaration it is applied to, and therefore does not propagate through the type system in any manner. Additionally, any use of an unsafe construct can be addressed locally, either by encapsulating it (e.g., via `@safe(unchecked)`) or propagating it (with `@unsafe`). This means that a module that has not adopted strict safety checking will not see any diagnostics related to this checking, even when modules it depends on adopt strict safety checking. + +The strict memory safety checking does not require any changes to the binary interface of a module, so it can be retroactively enabled (including `@unsafe` annotations) with no ABI or back-deployment concerns. Additionally, it is independent of other language subsetting approaches, such as Embedded Swift. + +## Should strict memory safety checking become the default? + +This proposes that the strict safety checking described be an opt-in feature with no path toward becoming the default behavior in some future language mode. There are several reasons why this checking should remain an opt-in feature for the foreseeable future: + +* The various `Unsafe` pointer types are the only way to work with contiguous memory in Swift today, and the safe replacements (e.g., `Span`) are new constructs that will take a long time to propagate through the ecosystem. Some APIs depending on these `Unsafe` pointer types cannot be replaced because it would break existing clients (either source, binary, or both). +* Interoperability with the C family of languages is an important feature for Swift. Most C(++) APIs are unlikely to ever adopt the safety-related attributes described above, which means that enabling strict safety checking by default would undermine the usability of C(++) interoperability. +* Swift's current (non-strict) memory safety by default is likely to be good enough for the vast majority of users of Swift, so the benefit of enabling stricter checking by default is unlikely to be worth the disruption it would cause. +* The auditing facilities described in this vision should be sufficient for Swift users who require strict memory safety, to establish where unsafe constructs are used and prevent "backsliding" where their use grows in an existing code base. These Swift users are unlikely to benefit much from strict safety being enabled by default in a new language mode, aside from any additional social pressure that would create on Swift programmers to adopt it. diff --git a/visions/platform-support.md b/visions/platform-support.md new file mode 100644 index 0000000000..0c38d50c52 --- /dev/null +++ b/visions/platform-support.md @@ -0,0 +1,285 @@ +# Swift Platform Support: A Vision for Evolution + +The Swift programming language has evolved into a versatile and powerful tool +for developers across a wide range of platforms. As the ecosystem continues to +grow, it is essential to establish a clear and forward-looking vision for +platform support. This vision has two main goals: + +1. To establish common terminology and definitions for platform support. + +2. To document a process for platforms to become officially supported in Swift. + +## Understanding Platforms in the Swift Ecosystem + +The term "platform" carries multiple interpretations. For our purposes, a +platform represents the confluence of operating system, architecture, and +environment where Swift code executes. Each platform is identified using a +version-stripped LLVM `Triple`—a precise technical identifier that captures the +essential characteristics of a host environment (e.g., +`x86_64-unknown-windows-msvc`). + +## The Anatomy of a `Triple` + +At its core, a `Triple` comprises 11 distinct elements arranged in a specific +pattern: + +``` +[architecture][sub-architecture][extensions][endian]-[vendor]-[kernel/OS][version]-[libc/environment][abi][version]-[object format] +``` + +This naming convention might initially appear complex, but it offers remarkable +precision. When a public entity isn't associated with a toolchain, the +placeholder `unknown` is used for the vendor field. Similarly, bare-metal +environments—those without an operating system—employ `none` as their OS/kernel +designation. + +While many of these fields may be elided, for use in Swift, the vendor and OS +fields are always included, even if they are placeholder values. + +Consider these illustrative examples: + +- `armv7eb-unknown-linux-uclibceabihf-coff`: A Linux system running on ARMv7 in big-endian mode, with the µClibc library and PE/COFF object format. +- `aarch64-unknown-windows-msvc-macho`: Windows NT on the ARM64 architecture using the MSVC runtime with Mach-O object format. +- `riscv64gcv-apple-ios14-macabi`: An iOS 14 environment running on a RISC-V processor with specific ISA extensions. + +This nomenclature creates a shared language for discussing platform capabilities +and constraints—an essential foundation for our support framework. + +## Distributions within Platforms + +A platform and distribution, while related, serve distinct roles in the Swift +ecosystem. A platform refers to the broader combination of Operating System, +architecture, and environment where Swift code executes and establishes the +foundational compatibility and functionality of Swift. + +A distribution, on the other hand, represents a specific implementation or +variant within a platform. For example, while Linux as a platform is supported, +individual distributions such as Ubuntu, Fedora, or Amazon Linux require +additional work to ensure that Swift integrates seamlessly. This includes +addressing distribution-specific configurations, dependencies, and conventions. + +Distributions are treated similarly to platforms in that they require a +designated owner. This owner is responsible for ensuring that Swift functions +properly on the distribution, adheres to the distribution's standards, and +remains a responsible citizen within that ecosystem. By assigning ownership, the +Swift community ensures that each distribution receives the attention and +stewardship necessary to maintain a high-quality experience for developers. + +## Platform Stewardship + +The health of each platform depends on active stewardship. Every platform in the +Swift ecosystem requires a designated owner who reviews platform-specific +changes and manages release activities. Platforms without active owners enter a +dormant state, reverting to exploratory status until new leadership emerges. + +This ownership model ensures that platform support remains intentional rather +than accidental—each supported environment has an advocate invested in its +success. + +The Platform Steering Group will regularly review the list of supported +platforms against the tier criteria below. While the Platform Steering Group +reserves the right to update the list or the tier criteria at any time, it is +expected that most such changes will be aligned with the Swift release cycle. + +## A Tiered Approach to Platform Support + +Swift's platform support strategy employs three distinct tiers, each +representing a different level of maturity. The requirements for each tier build +upon those of the previous tier. + +### Tier 1: "Supported" Platforms + +These are Swift's most mature environments, where the language must consistently +build successfully and pass comprehensive test suites. Swift on these platforms +offers the strongest guarantees of stability and performance. + +Platforms that are in Tier 1 should: + +- [ ] Digitally sign their release artifacts. +- [ ] Include a Software Bill of Materials (SBOM). + +- [ ] Include at a minimum the following Swift libraries: + + - [ ] Swift Standard Library + - [ ] Swift Supplemental Libraries + - [ ] Swift Core Libraries + - [ ] Swift Testing Frameworks (if applicable) + + (See [the Swift Runtime Libraries + document](https://github.com/swiftlang/swift/blob/main/Runtimes/Readme.md) + in [the Swift repository](https://github.com/swiftlang/swift).) + +- [ ] Maintain a three-version window of support, including: + + - [ ] At least one stable release. + - [ ] The next planned release. + - [ ] The development branch (`main`). + +- [ ] Have a clear, documented, story for debugging, to allow users to set up + an environment where their products can be executed on a device or + simulator and be debugged. + +- [ ] Have testing in CI, including PR testing. + +- [ ] Ship SDKs as regular release from [swift.org](https://swift.org) + +- [ ] Ensure that instructions needed to get started on the platform + are publicly available, ideally on or linked to from + [swift.org](https://swift.org). + +An important aspect of Tier 1 platforms is that maintenance of support +of these platforms is the collective responsibility of the Swift +project as a whole, rather than falling entirely on the platform +owner. This means: + +- Contributions should not be accepted if they break a Tier 1 platform. + +- If a Tier 1 platform does break, whoever is responsible for the code + that is breaking must work with the platform owner on some kind of + resolution, which may mean backing out the relevant changes. + +- New features should aim to function on all Tier 1 + platforms, subject to the availability of appropriate supporting + functionality on each platform. + +- There is a presumption that a release of Swift will be blocked if a + Tier 1 platform is currently broken. This is not a hard and fast + rule, and can be overridden if it is in the interests of the Swift + project as a whole. + +### Tier 1: "Supported" Toolchain Hosts + +Each toolchain host is an expensive addition to the testing matrix. +In addition to the requirements above, a toolchain host platform should: + +- [ ] Have CI coverage for the toolchain, including PR testing. + +- [ ] Offer toolchain distributions from + [swift.org](https://swift.org) as an official source, though + other distributions may also be available. + +- [ ] Include the following toolchain components: + + - [ ] Swift compiler (`swiftc`). + - [ ] C/C++ compiler (`clang`, `clang++`). + - [ ] Assembler (LLVM integrated assembler, built into `clang`). + - [ ] Linker (_typically_ `lld`). + - [ ] Debugger (`lldb`). + - [ ] Swift Package Manager (SwiftPM). + - [ ] Language Server (`sourcekit-lsp`). + - [ ] Debug Adapter (`lldb-dap`). + +- [ ] Code-sign individual tools as appropriate for the platform. + +Note that the bar for accepting a platform as a toolchain host is somewhat +higher than the bar for accepting a non-toolchain-host platform. + +### Tier 2: "Experimental" Platforms + +Experimental platforms occupy the middle ground—they must maintain the ability +to build but may experience occasional test failures. These platforms +represent Swift's expanding frontier. + +Platforms in this tier should: + +- [ ] Ensure that dependencies beyond the platform SDK can build from source. + +- [ ] Provide provenance information to validate the software supply chain. + +- [ ] Include at a minimum the following Swift libraries: + + - [ ] Swift Standard Library + - [ ] Swift Supplemental Libraries + - [ ] Swift Core Libraries + - [ ] Swift Testing Frameworks (if applicable) + + (See [the Swift Runtime Libraries + document](https://github.com/swiftlang/swift/blob/main/Runtimes/Readme.md) + in [the Swift repository](https://github.com/swiftlang/swift).) + +- [ ] Maintain at least a two-version window of support, including + + - [ ] The next planned release. + - [ ] The development branch (`main`). + +Unlike Tier 1, the Swift project does not assume collective +responsibility for experimental platforms. Platform owners should +work with individual contributors to keep their platform in a +buildable state. + +### Tier 3: "Exploratory" Platforms + +At the boundary of Swift's reach are exploratory platforms. +Exploratory status offers an entry point for platforms taking their +first steps into the Swift ecosystem. + +Platforms in this tier should: + +- [ ] Support reproducible builds without requiring external + patches, though there is no requirement that these build completely + or consistently. + +- [ ] Maintain support in the current development branch (`main`). + +The Swift Project does not assume collective responsibility for +exploratory platforms. Platform owners are responsible for keeping +their platform in a buildable state. + +## Platform Inclusion Process and Promotion + +Adding platform support begins with a formal request to the Platform Steering +Group, accompanied by a platform owner nomination. This structured yet +accessible approach balances Swift's need for stability with its aspiration for +growth. + +The request should include: + +- [ ] The preferred name of the platform. + +- [ ] The name and contact details of the platform owner. + +- [ ] The support tier into which the platform should be placed. + +- [ ] Instructions to build Swift for the platform, assuming someone + is starting from scratch, including any requirements for the + build system. + +- [ ] A list of tier requirements that are currently _not_ met by the + platform, including an explanation as to _why_ they are not met + and what the proposal is to meet them, if any. + +- [ ] Whether there has been any discussion about provisioning of CI + resources, and if so a copy of or link to that discussion. This + is particularly relevant for a Tier 1 platform request. + +Note that it is _not_ the case that a platform _must_ meet every +requirement of the requested tier in order to be placed into that +tier. The Platform Steering Group will consider each case on its +merits, and will make a decision based on the information at hand as +well as the overall benefit to the Swift Project. It should be +emphasized that the Platform Steering Group reserves the right to +consider factors other than those listed here when making decisions +about official platform support. + +The same process should be used to request a promotion to a higher +tier. + +## Existing Platforms and Demotion + +The following existing platforms are in Tier 1 regardless of any +text in this document: + +- All Apple platforms (macOS, iOS and so on). +- Linux +- Windows + +The Platform Steering Group reserves the right to demote any +platform to a lower tier, but regards demotion as a last resort +and will by preference work with platform owners to maintain +support appropriate for their platform's existing tier. + +Note that if your platform is one of the above special cases, and +there is some requirement in this document that is not being met, it +is expected that either there is a very good reason for the +requirement not being met, or that there is some plan to meet it in +future. diff --git a/visions/swift-testing.md b/visions/swift-testing.md new file mode 100644 index 0000000000..23879f437c --- /dev/null +++ b/visions/swift-testing.md @@ -0,0 +1,1051 @@ +# A New Direction for Testing in Swift + +## Introduction + +A key requirement for the success of any developer platform is a way to use +automated testing to identify software defects. Better APIs and tools for +testing can greatly improve a platform’s quality. Below, we propose a new +direction for testing in Swift. + +We start by defining our basic principles and describe specific features that +embody those principles. We then discuss several design considerations +in-depth. Finally, we present specific ideas for delivering an all-new testing +solution for Swift, and weigh them against alternatives considered. + +## Principles + +Testing in Swift should be **approachable** by both new programmers and +seasoned engineers. There should be few APIs to learn, and they should feel +ergonomic and modern. It should be easy to incrementally add new tests +alongside legacy ones. Testing should integrate seamlessly into the tools and +workflows that people know and use every day. + +A good test should be **expressive** and automatically include actionable +information when it fails. It should have a clear name and purpose, and there +should be facilities to customize a test’s representation and metadata. Test +details should be specified on the test, in code, whenever possible. + +A testing library should be **flexible** and capable of accommodating many +needs. It should allow grouping related tests when beneficial, or letting them +be standalone. There should be ways to customize test behaviors when necessary, +while having sensible defaults. Storing data temporarily during a test should +be possible and safe. + +A modern testing system should have **scalability** in mind and gracefully +handle large test suites. It should run tests in parallel by default, but allow +some tests to opt-out. It should be effortless to repeat a test with different +inputs and see granular results. The library should be lightweight and +efficient, imposing minimal overhead on the code being tested. + +## Features of a great testing system + +Guided by these principles, there are many specific features we believe are +important to consider when designing a new testing system. + +### Approachability + +* **Be easy to learn and use**: There should be few individual APIs to + memorize, they should have thorough documentation, and using them to write a + new test should be fast and seamless. Its APIs should be egonomic and adhere + to Swift’s [design guidelines](https://www.swift.org/documentation/api-design-guidelines/). +* **Validate expected behaviors or outcomes**: The most important job of any + testing library is checking that code meets specific expectations—for example, + by confirming that a function returns an expected result or that two values + are equal. There are many interesting variations on this, such as comparing + whole collections or checking for errors. A robust testing system should cover + all these needs, while using progressive disclosure to remain simple for + common cases. +* **Enable incremental adoption:** It should gracefully coexist with projects + that use XCTest or other testing libraries and allow incremental adoption so + that users can transition at their own pace. This is especially important + because this new system may take time to achieve feature parity. +* **Integrate with tools, IDEs, and CI systems:** A useful testing library + requires supporting tools for functionality such as listing and selecting + tests to run, launching runner processes, and collecting results. These + features should integrate seamlessly with common IDEs, SwiftPM’s `swift test` + command, and continuous integration (CI) systems. + +### Expressivity + +* **Include actionable failure details**: Tests provide the most value when they + fail and catch bugs, but for a failure to be actionable it needs to be + sufficiently detailed. When a test fails, it should collect and show as much + relevant information as reasonably possible, especially since it may not + reproduce reliably. +* **Offer flexible naming, comments, and metadata:** Test authors should be able + to customize the way tests are presented by giving them an informative name, + comments, or assigning metadata like labels to tests which have things in + common. +* **Allow customizing behaviors:** Some tests share common set-up or tear-down + logic, which need to be performed once for each test or group. Other times, a + test may begin failing for an irrelevant reason and must be temporarily + disabled. Some tests only make sense to run under certain conditions, such as + on specific device types or when an external resource is available. A modern + testing system should be flexible enough to satisfy all these needs, without + complicating simpler use cases. + +### Flexibility + +* **Allow organizing tests into groups (or not):** Oftentimes a component will + have several related tests that would make sense to group together. It should + be possible to group tests into hierarchies, while allowing simpler tests to + remain standalone. +* **Support per-test storage:** Tests often need to store data while they are + running and local variables are not always sufficient. For example, set up + logic for a test may create a value the test needs to access, but these are in + different scopes. There must be a way to carefully store per-test data, to + ensure it is isolated to a single test and initialized deterministically to + avoid unexpected dependencies or failures. +* **Allow observing test events:** Some use cases require an ability to observe + test events—for example, to perform custom reporting or analysis of results. A + testing library should offer hooks for event handling. + +### Scalability + +* **Parallelize execution:** Many tests can be run in parallel to improve + execution time, either using multiple threads in a single process or multiple + runner processes. A testing library should offer flexible parallelization + options for eligible tests, encourage parallelizing whenever possible, and + offer granular control over this behavior. It should also leverage Swift’s + data race safety features (such as `Sendable` enforcement) to the fullest + extent possible to avoid concurrency bugs. +* **Repeat a test multiple times with different arguments:** Many tests consist + of a template with minor variations—for example, invoking a function multiple + times with different arguments each time and validating the result of each + invocation. A testing library should make this pattern easy to apply, and + include detailed reporting so a failure during a single argument is + represented clearly. +* **Behave consistently across platforms:** Any new testing solution should be + cross-platform from its inception and support every platform Swift supports. + Its observable behaviors should be as consistent as possible across those + platforms, especially for core responsibilities such as discovering and + executing tests. + +## Design considerations + +Several areas deserve close examination when designing a new testing API. Some, +because they may benefit from language or compiler toolchain enhancements to +deliver the ideal experience, and others because they have non-obvious +reasoning or requirements. + +### Expectations + +Testing libraries typically offer APIs to compare values—for example, to +confirm that a function returns an expected result—and report a test failure if +a comparison does not succeed. Depending on the library, these APIs may be +called “assertions”, “expectations”, “checks”, “requirements”, “matchers“, or +other names. In this document we refer to them as **expectations**. + +For test failures to be actionable, they need to include enough details to +understand the problem, ideally without a human manually reproducing the +failure and debugging. The most important details relevant to expectation +failures are the values being compared or checked and the kind of expectation +being performed (e.g. equal, not-equal, less-than, is-not-nil, etc.). Also, if +any error was caught while evaluating an expression passed to an expectation, +that should be included. + +Beyond the values of evaluated expressions, there are other pieces of +information that may be useful to capture and include in expectations: + +* The **source code location** of the expectation, typically using the format + `#fileID:#line:#column`. This helps test authors jump quickly to the line of + code to view context, and lets IDEs present the failure in their UI at that + location. +* The **source code text of expression(s)** passed to the expectation. In an + example expectation API call `myAssertEqual(subject.label == "abc")`, the + source code text would be the string `"subject.label == \"abc\""` . + + Even though source code text may not be necessary when viewing failures in + an IDE since the code is present, this can still be helpful to confirm the + expected source code was evaluated in case it changed recently. It’s even + more useful when the failure is shown on a CI website or anywhere without + source, since a subexpression (such as `subject.label` in this example) may + give helpful clues about the failure. +* **Custom user-specified comments**. Comments can be helpful to allow test + authors to add context or information only needed if there was a failure. They + are typically short and included in the textual log output from the test + library. +* **Custom data or file attachments.** Some tests involve files or data + processing and may benefit from allowing expectations to save arbitrary data + or files in the results for later analysis. + +#### Powerful, yet simple + +Since the most important details to include in expectation failure messages are +the expression(s) being compared and the kind of expression, some testing +libraries offer a large number of specialized APIs for detailed reporting. Here +are some examples from other prominent testing libraries: + +| | Java (JUnit) | Ruby (RSpec) | XCTest | +|----|----|----|----| +| Equal | `assertEquals(result, 3);` | `expect(result).to eq(3)` | `XCTAssertEqual(result, 3)` | +| Identical | `assertSame(result, expected);` | `expect(result).to be(expected)` | `XCTAssertIdentical(result, expected)` | +| Less than or equal | N/A | `expect(result).to be <= 5` | `XCTAssertLessThanOrEqual(result, 5)` | +| Is null/nil | `assertNull(actual);` | `expect(actual).to be_nil` | `XCTAssertNil(actual)` | +| Throws | `assertThrows(E.class, () -> { ... });` | `expect {...}.to raise_error(E)` | `XCTAssertThrowsError(...) { XCTAssert($0 is E) }` | + +Offering a large number of specialized expectation APIs is a common practice +among testing libraries: XCTest has 40+ functions in its +[`XCTAssert` family](https://developer.apple.com/documentation/xctest/boolean_assertions); +JUnit has +[several dozen](https://junit.org/junit5/docs/5.0.1/api/org/junit/jupiter/api/Assertions.html); +RSpec has a +[large DSL](https://relishapp.com/rspec/rspec-expectations/docs/built-in-matchers) +of test matchers. + +Although this approach allows straightforward reporting, it is not scalable: + +* It increases the learning curve for new users by requiring them to learn many + new APIs and remember to use the correct one in each circumstance, or risk + having unclear test results. +* More complex use cases may not be supported—for example, if there is no + expectation for testing that a `Sequence` starts with some prefix using + `starts(with:)`, the user may need a workaround such as adding a custom + comment which includes the sequence for the results to be actionable. +* It requires testing library maintainers add bespoke APIs supporting many use + cases which creates a maintenance burden. +* Depending on the exact function signatures, it may require additional + overloads that complicate type checking. + +We believe expectations should strive to be as simple as possible and involve +few distinct APIs, but be powerful enough to include detailed results for every +expression. Instead of offering a large number of specialized expectations, +there should only be a few basic expectations and they should rely on ordinary +expressions, built-in language operators, and the standard library to cover all +use cases. + +#### Evaluation rules + +Expectations have certain rules which must be followed carefully when handling +arguments: + +* The primary expression(s) being checked should be evaluated exactly once. In + particular, if the expectation failed, showing the value of any evaluated + expression should not cause the expression to be evaluated a second time. This + is to avoid any undesirable or unexpected side effects of multiple + evaluations. +* Custom comments or messages should only be evaluated if the expectation + failed, and at most once, to similarly avoid undesirable side effects and + prevent unnecessary work. + +#### Continuing after a failure + +A single test may include multiple expectations, and a testing library must +decide whether to continue executing a test after one of its expectations +fails. Some tests benefit from always running to completion, even if an earlier +expectation failed, since they validate different things and early expectations +are unrelated to later ones. Other tests are structured such that later logic +depends heavily on the results of earlier expectations, so terminating the test +after any expectation fails may save time. Still other tests take a hybrid +approach, where only certain expectations are required and should terminate +test execution upon failure. + +This is a policy decision, and is something a testing library could allow users +to control on a global, per-test, or per-expectation basis. + +#### Rich representation of evaluated values + +Often, expectation APIs do not preserve raw expression values when reporting a +failure, and instead generate a string representation of those values for +reporting purposes. Although a string representation is often sufficient, +failure presentation could be improved if an expectation were able to keep +values of certain, known data types. + +As an example, imagine a hypothetical expectation API call +`ExpectEqual(image.height, 100)`, where `image` is a value of some well-known +graphical image type `UILibrary.Image`. Since this uses a known data type, the +expectation could potentially keep `image` upon failure and include it in test +results, and then an IDE or other tool could present the image graphically for +easier diagnosis. This capability could be extensible and cross-platform by +using a protocol to describe how to convert arbitrary values into one of the +testing library’s known data types, delivering much richer expectation results +presentation for commonly-used types. + +### Test traits + +A recurring theme in several of the features discussed above is a need to +express additional information or options about individual tests or groups of +tests. A few examples: + +* Describing test requirements or marking a test disabled. +* Assigning a tag or label to a test, to locate or run those which have + something in common. +* Declaring argument values for a parameterized or “data-driven” test. +* Performing common logic before or after a test. + +Collectively, these are referred to in this document as **traits**. The traits +for an individual test _could_ be stored in a standalone file, separate from +the test definition, but relying on a separate file has known downsides: it can +get out of sync if a test name changes, and it’s easy to overlook important +details—such as whether a test is disabled or has specific requirements—when +they’re stored separately. + +We believe that the traits for a single test should preferably be declared in +code placed as close to the test they describe as possible to avoid these +problems. However, global settings may still benefit from configuring via +external files, as there may not be a canonical location in code to place them. + +#### Trait inheritance + +When grouping related tests together, if a test trait is specified both for an +individual test and one of its containing groups, it may be ambiguous which +option takes precedence. The testing library must establish policies for how to +resolve this. + +Test traits may fall into different categories in terms of their inheritance +behavior. Some semantically represent multiple values that a user would +reasonably expect to be added together. One example is test requirements: if a +group specifies one requirement, while one of its test functions specifies +another, the test function should only run if both requirements are satisfied. +The order these requirements are evaluated are worth considering and formally +specifying, so that a user could be assured that requirements are always +evaluated “outermost-to-innermost” or vice-versa. + +Another example is test tags: they are also considered multi-value, but items +with tags are typically expected to have `Set` rather than `Array` semantics +and ignore duplicates, so for this type of trait the evaluation order is +insignificant. + +Other test traits semantically represent a single value and conflicts between +them may be more challenging to resolve. As a hypothetical example, imagine a +test trait spelled `.enabled(Bool)` which includes a `Bool` that determines +whether a test should run. If a group specifies `.enabled(false)` but one of +its test functions specifies `.enabled(true)`, which value should be honored? +Arguments could be made for either policy. + +When possible, it may be easier to avoid ambiguity: in the previous example, +this may be solved by only offering a `.disabled` option and not the opposite. +But the inheritance semantics of each option should be considered, and when +ambiguity is unavoidable, a policy for resolving it should be established and +documented. + +#### Trait extensibility + +A flexible test library should allow certain behaviors to be extended by test +authors. A common example is running logic before or after a test: if every +test in a certain group requires the same steps beforehand, those steps could +be placed in a single method in that group rather than expressed as an option +on a particular test. However, if only a few tests within a group require those +steps, it may make sense to leverage a test trait to mark those tests +individually. + +Test traits should provide the ability to extend behaviors to support this +workflow. For example, it should be possible to define a custom test trait, and +implement hooks that allow it to run custom code before or after a test or +group. + +### Test identity + +Some features require the ability to uniquely identify a test, such as +selecting individual tests to run or serializing results. It may also be useful +to access the name of a test inside its own body or for an entity observing +test events to query test names. + +A testing library should include a robust mechanism to uniquely identify tests +and identifiers should be stable across test runs. If it is possible to +customize a test’s display name, the testing library should decide which name +is authoritative and included in the unique identifier. Also, function +overloading could make certain test function names ambiguous without additional +type information. + +### Test discovery + +A frequent challenge for testing libraries in all languages is the need to +locate tests in order to run them. Users typically expect tests to be +discovered automatically, without needing to provide a comprehensive list since +that would be a maintenance burden. + +There are three types of test discovery worth considering in particular, since +they serve different purposes: + +* **At runtime:** When a test runner process is launched, the testing library + needs to locate tests so it can execute them. +* **After a build:** After compilation of all test code has completed + successfully, but before a test runner process has been launched, it may be + useful for a tool to introspect the test build products and print the list of + tests or extract other metadata about them without running them. +* **While authoring:** After tests have been written or edited, but before a + build has completed, it is common for an IDE or other tool to statically + analyze or index the code and locate tests so it can list them in a UI and + allow running them. + +Each of these are important to support, and may require different solutions. + +#### Non-runtime discovery + +Two of the above test discovery types—_After a build_ and _While authoring_— +require the ability to discover tests without launching a runner process, and +thus without using the testing library’s runtime logic and models to represent +tests. In addition to the IDE use case mentioned above, another reason +discovering tests statically may be useful is so CI systems can extract +information about tests and use it to optimize test execution scheduling on +physical devices. It is common for CI systems to run a different host OS than +the platform they are targeting—for example, an Intel Mac building tests for an +iOS device—and in those situations it may be impractical or expensive for the +CI system to launch a runner process to gather this information. + +Note that not _all_ test details are eligible to extract statically: those that +enable runtime test behaviors may not be, but trivial metadata (such as a +test’s name or whether it is disabled) should be extractable, especially with +further advances in Swift’s support for +[Build-Time Constant Values](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0359-build-time-constant-values.md). +While designing a new testing API, it is important to consider which test +metadata should be statically extractable to support these non-runtime discovery +use cases. + +### Parameterized testing + +Repeating a test multiple times with different arguments—formally referred to as +[Parameterized or Data-Driven Testing](https://en.wikipedia.org/wiki/Data-driven_testing) +—can allow expanding test coverage to cover more scenarios with minimal code +repetition. Although a user could approximate this using a simple loop such as +`for...in` in the body of a test, it’s often better to let testing libraries +handle this task. A testing library can automatically keep track of the +argument(s) for each invocation of a test and record them in the results. It can +also provide a way to selectively re-run individual argument combinations for +fine-grained debugging in case only one instance failed. + +Note that recording individual parameterized tests’ arguments in results and +re-running them requires some way to uniquely represent those arguments, which +overlaps with some of the considerations discussed in +[Test identity](#test-identity). + +### Parallelization and concurrency + +A modern testing system should make efficient use of the machine it runs on. +Many tests can safely run in parallel, and the testing system should encourage +this by enabling per-test parallelization by default. In addition to faster +results and shorter iteration time, running tests in parallel can help identify +bugs due to hidden dependencies between tests and encourage better state +isolation. + +However, some tests may need to disable parallelization and run one at a time. +It should be possible to opt-out, and this may be especially useful while +migrating from older testing systems which don't support parallelization. +Although opting-out of this behavior should be possible, it should be narrowly +scoped to not sacrifice other tests' ability to run in parallel. + +In addition to running tests in parallel relative to each other, tests +themselves should seamlessly support Swift's concurrency features. In particular, +this means: + +* Tests should be able to use async/await whenever necessary. +* Tests should support isolation to a global actor such as `@MainActor`, but be + nonisolated by default. (Isolation by default would undermine the goal of + running tests in parallel by default.) +* Values passed as arguments to parameterized tests should be `Sendable`, since + they may cross between isolation domains within the testing system's execution + machinery. +* Types containing tests functions and their stored properties need not be + `Sendable`, since they are only used from a single isolation domain while each + test function is run. + +### Tools integration + +A well-rounded testing library should be integrated with popular tools used by +the community. This integration should include some essential functionality such +as: + +* Building tests into products which can be be executed. +* Running all built tests. +* Showing per-test results, including details of each individual failure during + a test. +* Showing an aggregate summary of a test run, including failure statistics. + +Beyond the essentials, tools may offer other useful features, such as: + +* Filtering tests by name, specific traits (e.g. custom tags), or other criteria. +* Outputting results to a standard format such as JUnit XML for importing into + other tools. +* Controlling runtime options such as whether parallel execution is enabled, + whether failed tests should be reattempted, etc. +* Relaunching a test executable after an unexpected crash. +* Reporting "live" progress as each test finishes. + +In order to deliver the functionality above, a testing system needs to track +significant events which occur during execution, such as when each test starts +and finishes or encounters a failure. There needs to be a structured, +machine-readable representation of each event, and to provide granular progress +updates, there should be an option for tools to observe a stream of such events. +Any serialization formats involved should be versioned and provide some level of +stability over time, to ensure compatibility with tools which may evolve on +differing release schedules. + +Some of the features outlined above are much easier to achieve if there are at +least two processes involved: One which executes the tests themselves (which may +be unstable and terminate unexpectedly), and another which launches and monitors +the first process. In this document, we'll refer to this second process as the +**harness**. A two-process architecture involving a supervisory harness helps +enable functionality such as relaunching after a crash, live progress reporting, +or outputting results to a standard format. Some cases effectively _require_ a +harness, such as when launching tests on a remotely-connected device. A modern +testing library should provide any necessary APIs and runtime support to +facilitate integration with a harness. + +## Today’s solution: XCTest + +[XCTest](https://developer.apple.com/documentation/xctest) has historically +served as the de facto standard testing library in Swift. It was originally +written [in 1998](https://www.sente.ch/ocunit/?lang=en) in Objective-C, and +heavily embraced that language’s idioms throughout its APIs. It relies on +subclassing (reference semantics), dynamic message passing, NSInvocation, and +the Objective-C runtime for things like test discovery and execution. In the +2010s, it was integrated deeply into Xcode and given many more capabilities and +APIs which have helped Apple platform developers deliver industry-leading +software. + +When Swift was introduced, XCTest was extended further to support the new +language while maintaining its core APIs and overall approach. This allowed +developers familiar with using XCTest in Objective-C to quickly get up to +speed, but certain aspects of its design no longer embody modern best practices +in Swift, and some have become problematic and prevented enhancements. Examples +include its dependence on the Objective-C runtime for test discovery; its +reliance on APIs like NSInvocation which are unavailable in Swift; the frequent +need for implicitly-unwrapped optional (IUO) properties in test subclasses; and +its difficulty integrating seamlessly with Swift Concurrency. + +It is time to chart a new course for testing in Swift, and in proposing a new +direction, this ultimately represents a successor to XCTest. This transition +will likely span several years, and we aim to thoughtfully design and deliver a +solution that will be even more powerful while bearing in mind the many lessons +learned from maintaining it over the years. + +## A new direction + +> [!NOTE] +> The approach described below is not meant to include a solution for _every_ +> consideration or feature discussed in this document. It describes a starting +> point for this new direction, and covers many of the topics, but leaves some +> to be pursued as part of follow-on work. + +The new direction includes 3 major components exposed via a new module named +`Testing`: + +1. `@Test` and `@Suite` attached macros: These declare test functions and suite + types, respectively. +1. Traits: Values passed to `@Test` or `@Suite` which customize the behavior of + test functions or suite types. +1. Expectations `#expect` and `#require`: expression macros which validate + expected conditions and report failures. + +### Test and Suite declaration + +To declare test functions and suites (types containing tests), we will leverage +[Attached Macros (SE-0389)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md). +At a high level, this will consist of several attached macros +which may be placed on a test type or test function, defined in a new module +named `Testing`: + +```swift +/// Declare a test function +@attached(peer) +public macro Test( + // ...Parameters described later +) + +/// Declare a test suite. +@attached(member) @attached(peer) +public macro Suite( + // ...Parameters described later +) +``` + +Then, test authors may attach these macros to functions or types in a test +target. Here are some usage examples: + +```swift +import Testing + +// A test implemented as a global function +@Test func example1() { + // ... +} + +@Suite struct BeginnerTests { + // A test implemented as an instance method + @Test func example2() { ... } +} + +// Implicitly treated as a suite type, due to containing @Test functions. +actor IntermediateTests { + private var count: Int + + init() async throws { + // Runs before every @Test instance method in this type + self.count = try await fetchInitialCount() + } + + deinit { + // Runs after every @Test instance method in this type + print("count: \(count), delta: \(delta)") + } + + // A test implemented as an async and throws instance method + @Test func example3() async throws { + delta = try await computeDelta() + count += delta + // ... + } +} +``` + +**Test functions** may be defined as global functions or as either instance or +static methods in a type. They must always be explicitly annotated as `@Test`, +they need not follow any naming convention (such as beginning with “test”), and +they may include `async`, `throws`, or `mutating`. + +**Suite types**, or simply “suites”, are types containing `@Test` functions or +other nested suite types. Suite types may include the `@Suite` attribute +explicitly, although it is optional and only required when specifying traits +(described below). A suite type must have a zero-parameter `init()` if it +contains instance `@Test` methods. + +**Per-test storage:** The `IntermediateTests` example demonstrates per-test +set-up and tear-down as well as per-test storage: A unique instance of +`IntermediateTests` is created for every `@Test`-annotated instance method it +contains, which means that its `init` and `deinit` are run once before and +after each respectively, and they may contain set-up or tear-down logic. Since +`count` is an instance stored property, it acts as per-test storage, and since +`example3()` is isolated to its enclosing actor type it is allowed to mutate +`count`. + +**Sendability:** Note that the test functions and suite types in these examples +are not required to be `Sendable`. At runtime, if the `@Test` function is an +instance method, the testing library creates a thunk which instantiates the +suite type and invokes the `@Test` function on that instance. The suite type +instance is only accessed from a single Task. + +**Actor isolation:** `@Test` functions or types may be annotated with a global +actor (such as `@MainActor`), in accordance with standard language and type +system rules. This allows tests to match the global actor of their subject and +reduce the need for suspension points. + +#### Test discovery + +To facilitate test discovery, the attached macros above will eventually use a +feature such as `@linkage`, an attribute for controlling low-level symbol +linkage (see +[pitch](https://forums.swift.org/t/pitch-2-low-level-linkage-control/69752)). +It will allow placing variables and functions in a special section of the binary, +where they can be retrieved and at runtime or inspected statically at build time. + +However, before that feature lands, the testing library will use a temporary +approach of iterating through types conforming to a known protocol and +gathering their tests by calling a static property. The attached macros will +emit code which generates the types to be discovered using this mechanism. Once +more permanent support lands, the attached macros will be adjusted to adopt it +instead. + +Regardless of which technique the attached macros above use to facilitate test +discovery, the APIs called by their expanded code need not be considered public +or stable. In particular, code emitted by these macros may call +underscore-prefixed APIs declared in the `Testing` module, which are marked +`public` to facilitate use by these macros but are generally considered a +private implementation detail. + +### Traits + +As discussed earlier, it is important to support specifying traits for a test. +[SE-0389](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md) +allows including parameters in an attached macro declaration, and this allows +users to pass arguments to a `@Test` attribute on a test function or type. + +The `Testing` module will offer an extensible mechanism for specifying per-test +traits via types conforming to protocols such as `TestTrait` and `SuiteTrait`: + +```swift +/// A protocol describing traits that can be added to a test function or +/// to a test suite. +public protocol Trait: Sendable { ... } + +/// A protocol describing traits that can be added to a test function. +public protocol TestTrait: Trait { ... } + +/// A protocol describing traits that can be added to a test suite. +public protocol SuiteTrait: Trait { ... } +``` + +Using these protocols, the attached macros `@Test` and `@Suite` shown earlier +will gain parameters accepting traits: + +```swift +/// Declare a test function. +/// +/// - Parameter traits: Zero or more traits to apply to this test. +@attached(peer) +public macro Test( + _ traits: any TestTrait... +) + +/// Declare a test function. +/// +/// - Parameters: +/// - displayName: The customized display name of this test. +/// - traits: Zero or more traits to apply to this test. +@attached(peer) +public macro Test( + _ displayName: _const String, + _ traits: any TestTrait... +) + +/// Declare a test suite. +/// +/// - Parameter traits: Zero or more traits to apply to this test suite. +@attached(member) @attached(peer) +public macro Suite( + _ traits: any SuiteTrait... +) + +/// Declare a test suite. +/// +/// - Parameters: +/// - displayName: The customized display name of this test suite. +/// - traits: Zero or more traits to apply to this test suite. +@attached(member) @attached(peer) +public macro Suite( + _ displayName: _const String, + _ traits: any SuiteTrait... +) +``` + +The specifics of the `Trait` protocols and the built-in types conforming to +them will be left to subsequent proposals. But to illustrate the general +pattern they will follow, here is an example showing how a hypothetical option +for marking a test disabled could be structured: + +```swift +/// A test trait which marks a test as disabled. +public struct DisabledTrait: TestTrait { + /// An optional comment related to this option. + public var comment: String? +} + +extension TestTrait where Self == DisabledTrait { + /// Construct a test trait which marks a test disabled, + /// with an optional comment. + public static func disabled(_ comment: String? = nil) -> Self +} + +// Usage example: +@Test(.disabled("Currently causing a crash: see #12345")) +func example4() { + // ... +} +``` + +#### Nesting / subgrouping tests + +Earlier examples showed how related tests may be grouped together by placing +them within a type. This technique also allows forming sub-groups by nesting +one type containing tests inside another: + +```swift +struct OuterTests { + @Test func outerExample() { /* ... */ } + + @Suite(.tags("edge-case")) + struct InnerTests { + @Test func innerExample1() { /* ... */ } + @Test func innerExample2() { /* ... */ } + } +} +``` + +When using this technique, test traits may be specified on nested types and +inherited by all tests they contain. For example, the `.tags("edge-case")` +trait shown here on `InnerTests` would have the effect of adding the tag +`edge-case` to both `innerExample1()` and `innerExample2()`, as well as to +`InnerTests`. + +#### Parameterized tests + +Parameterized testing is easy to support using this API: The `@Test` functions +shown earlier do not accept any parameters, making them non-parameterized, but +if a `@Test` function includes a parameter, then a different overload of the +`@Test` macro can be used which accepts a `Collection` whose associated +`Element` type matches the type of the parameter: + +```swift +/// Declare a test function parameterized over a collection of values. +/// +/// - Parameters: +/// - traits: Zero or more traits to apply to this test. +/// - collection: A collection of values to pass to the associated test +/// function. +/// +/// During testing, the associated test function is called once for each element +/// in `collection`. +@attached(peer) +public macro Test( + _ traits: any TestTrait..., + arguments collection: C +) where C: Collection & Sendable, C.Element: Sendable + +// Usage example: +@Test(arguments: ["a", "b", "c"]) +func example5(letter: String) { + // ... +} +``` + +Once Swift’s support for +[Variadic Generics](https://github.com/hborla/swift-evolution/blob/variadic-generics-vision/vision-documents/variadic-generics.md) +gains more functionality, the signature of these `@Test` macros may be revised +to accept more than one collection of arguments. This will expand the feature by +allowing a test function with arity _N_ to be repeated once for each combination +of elements from _N_ collections. + +### Expectations + +In existing test solutions available to Swift developers, there is limited +diagnostic information available for a failed expectation such as +`assert(2 < 1)`. The expression is reduced at runtime to a simple boolean value +with no context (such as the original source code) available to include in a +test’s output. + +By adopting +[Expression Macros (SE-0382)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md), +we can give developers _implicitly expressive_ test +expectations. The expectation shown below, upon failure, can capture not just +the boolean value `false`, but also the left-hand and right-hand operands and +the operator itself (that is, `x`, `1`, and `<` respectively) and expand any +sub-expressions to their evaluated values, such as `x → 2`: + +```swift +let x = 2 +#expect(x < 1) // failed: (x → 2) < 1 +``` + +#### Handling optionals + +Some expectations _must_ pass for a test to proceed—these would be expressed +with a separate macro `#require()`. Because `#require()` _must_ pass, we can +infer additional behaviors based on its argument that we cannot do with +`#expect()`. For example, if an optional value is passed to `#require()`, we +can infer that `#require()` should return the optional value or fail if it is +`nil`: + +```swift +let x: Int? = 10 +let y: String? = nil +let z = try #require(x) // passes, z == 10 +let w = try #require(y) // fails, test ends early with a thrown error +``` + +#### Handling complex expressions + +We can also extract the components of an expression like `a.contains(b)` and, on +failure, report the value of `a` and `b`: + +```swift +let a = [1, 2, 3] +let b = 4 +#expect(a.contains(b)) // failed: (a → [1, 2, 3]).contains(b → 4) +``` + +#### Handling collections + +We can also leverage built-in language features for yet more expressiveness. +Consider the following test logic: + +```swift +let a = [1, 2, 3, 4, 5] +let b = [1, 2, 3, 3, 4, 5] +#expect(a == b) +``` + +This expectation will fail because of the extra element `3` in `b`. We can +leverage +[Ordered Collection Diffing (SE-0240)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0240-ordered-collection-diffing.md) +to capture exactly how these arrays differ and present that information to the +developer as part of the test output or in the IDE. + +### API and ABI stability + +Once this project reaches its first major release, its API will be considered +stable and will follow the same general rules as the Swift standard library. +Source-breaking API changes will be considered a last resort and be preceded by +a generous deprecation period. + +However, unlike the standard library, this project will generally not guarantee +ABI stability. Tests are typically only run during the development cycle and +aren't distributed to end users, so the runtime testing library will not be +included in OS distributions and thus its interfaces do not need to maintain +ABI stability. + +### Project governance + +For this testing solution to stand the test of time, it needs to be well +maintained and any new features added to it should be thoughtfully designed and +undergo community review. + +The codebase for this testing system will be open source. Any significant +additions to its feature set or changes to its API or behavior will follow a +process inspired by, but separate from, +[Swift Evolution](https://www.swift.org/swift-evolution/). Changes will be +described in writing using a standard +[proposal template](https://github.com/apple/swift-testing/blob/main/Documentation/Proposals/0000-proposal-template.md) and discussed in the +[swift-testing](https://forums.swift.org/c/related-projects/swift-testing/103) +category of the Swift Forums. The process for pitching and submitting proposals +for review will be formalized separately and documented in the project repo. + +A new group—tentatively named the _Swift Testing Workgroup_—will be formed to +act as the primary governing body for this project. This group will be +considered a sub-group of the planned +[Ecosystem Steering Group](https://www.swift.org/blog/evolving-swift-project-workgroups/) +once it has been formed. The group will retroactively assume responsibility for +the [swift-corelibs-xctest](https://github.com/apple/swift-corelibs-xctest) +project as well. The responsibilities of members of this workgroup will include: + +* defining and approving roadmaps for the project; +* scheduling proposal reviews; +* guiding community discussion; +* making decisions about proposals; and +* working with members of related workgroups, such as the + [Platform](https://www.swift.org/platform-steering-group/), + [Server](https://www.swift.org/sswg/), or + [Documentation](https://www.swift.org/documentation-workgroup/) workgroups, on + topics which intersect with their areas of focus. + +The membership of this workgroup, its charter, and more specifics about its role +in the project will be formalized separately. + +### Distribution + +As mentioned in [Approachability](#approachability) above, testing should be +easy to use. The easier it is to write a test, the more tests will be written, +and software quality generally improves with more automated tests. + +Most open source Swift software is distributed as source code and built by +clients, using tools such as Swift Package Manager. Although this is very common, +if we took this approach, every client would need to download the source for +this project and its dependencies. A test target is included in SwiftPM's +standard New Package template, so if this project became the default testing +library used by that template, this would mean a newly-created package would +require internet access to build its tests. In addition to downloading source, +clients would also need to _build_ this project along with its dependencies. +Since this project relies on Swift Macros, it depends on +[swift-syntax](https://github.com/apple/swift-syntax) and that is a large +project known to have lengthy build times as of this writing. + +Due to these practical concerns, this project will be distributed as part of the +Swift toolchain, at least initially. Being in the toolchain means clients will +not need to download or build this project or its dependencies, and this +project's macros can use the copy of swift-syntax in the toolchain. Longer-term, +if the practical concerns described above are resolved, the library could be +removed from the toolchain, and doing so could yield other benefits such as more +explicit dependency tracking and more portable toolchains. + +#### Package support + +Although the Swift toolchain will be the primary method of distribution, the +project will support building as a Swift package. This will facilitate +development of the package itself by its maintainers and outside contributors. + +Clients will also have the option of declaring an explicit package dependency on +this project rather than relying on the built-in copy in the toolchain. If a +client does this, it will be possible for tools to integrate with the client's +specified copy of the project, as long as it is a version of the package the +tool supports. + +### Platform support + +Testing should be broadly available across platforms. The long-term goal is for +this project to be available on all platforms Swift itself supports. + +Whenever the +[Swift Platform Steering Group](https://www.swift.org/platform-steering-group/) +declares intention to support a new platform, this project will be considered +one of the highest-priority components to get working on the new platform (after +dependencies such as the standard library) since this project will enable +qualification of many other components in the stack. The maintainers of this +project will work with other Swift workgroups or steering groups to help enable +support on new platforms. + +One reason why broad platform support is important is so that this project can +eventually support testing the Swift standard library. The standard library +currently uses a custom library for testing +([StdlibUnittest](https://github.com/apple/swift/tree/main/stdlib/private/StdlibUnittest)) +but many of this project's benefits would be useful in that context as well, so +eventually we would like to rebase StdlibUnitTest on this project. + +While the goal is for this project to work on every platform Swift does, it +currently does not work on Embedded Swift due to its reliance on existentials. +It is possible this limitation may be overcome through project changes, but if +it cannot be, the project maintainers will work with other Swift work/steering +groups to identify a solution. + +## Alternatives considered + +### Declarative test definition using Result Builders + +While exploring new testing directions, we considered and thoroughly prototyped +an approach relying heavily on +[Result Builders](https://docs.swift.org/swift-book/LanguageGuide/AdvancedOperators.html#ID630). +At a high level, the idea involved a few pieces: + +* Types like `TestCase` and `TestSuite` representing individual tests and + groups, respectively. +* A `@resultBuilder` type `TestBuilder` allowing declarative creation of test + hierarchies. +* A protocol named e.g. `TestProvider` with a requirement + `@TestBuilder static var tests: TestSuite` which suite types would + implement in order to define their tests. +* Tests defined as closures in the `static var tests` result builder above, + accepting an instance of a type named `TestContext` which allowed accessing + per-test instance storage. + +This approach seemed promising at first and satisfied many of the goals +described in the beginning of this document. But we discovered several +significant drawbacks: + +* **Type-checking performance:** Certain Result Builder usage patterns are + known to lead to poor type-checking performance, especially when the + expression is long. When describing an entire suite of tests, which may be + nested arbitrarily, the work can become exponential and lead to a noticeable + increase in build time or, in the extreme case, compiler timeouts. +* **Accessing test state:** Because tests are defined in a `static` context, + per-test state must be accessed indirectly, via a `TestContext` wrapper type. + This made accessing per-test storage more verbose than necessary, and + introduced the need for synchronization on the wrapper type. +* **Global actor isolation:** It is difficult, or perhaps impossible, to use a + global actor (most often `@MainActor`) in both the test body and the type + enclosing the test which stored its per-test state, and ensure they match. In + practice, this means that tests whose subjects include global actors are + challenging to write without lots of `await` suspension points. +* **Build-time test discovery**: It is difficult, or perhaps impossible, to + discover tests comprehensively at build time since the definition of tests + happens in result builder functions. +* **Discovery:** Using Result Builders did not fully solve the problem of + runtime test discovery; it is still necessary to locate types conforming to + `TestProvider` protocol, even though the tests within each conforming type are + trivial to gather by calling its static `tests` property. + +### Imperative test definition + +Another approach for defining tests is using a builder pattern with an +imperative style API. A good example of this is Swift’s own +[StdlibUnittest](https://github.com/apple/swift/tree/main/stdlib/private/StdlibUnittest) +library which is used to test the standard library. To define tests, a user +first creates a `TestSuite` and then calls `.test("Some name") { /* body */ }` +one or more times to add a closure containing each test. + +One problem with generalizing this approach is that it doesn’t have a way to +deterministically discover tests either after a build or while authoring +tests in an IDE (see [Test discovery](#test-discovery)). Because tests are +defined using imperative code, it may contain arbitrary control flow logic +static analysis may not be able to reason about. As a contrived example, imagine +the following: + +```swift +import StdlibUnittest + +var myTestSuite = TestSuite("My tests") +if Bool.random() { + myTestSuite.test("Foo") { /* ... */ } +} +``` + +There may be arbitrary logic (such as `if Bool.random()` here) which influences +the test suite construction, and this makes important features like IDE +discovery impossible in the general case. diff --git a/visions/using-c++-from-swift.md b/visions/using-c++-from-swift.md new file mode 100644 index 0000000000..4b43b13acc --- /dev/null +++ b/visions/using-c++-from-swift.md @@ -0,0 +1,394 @@ +# Using C++ from Swift + +## Introduction + +This document lays out a vision for the development of the "forward" half of C++ and Swift interoperability: using C++ APIs from Swift. It sets overarching goals that drive the project’s design decisions, outlines some high-level topics related to C++ interoperability, and, finally, investigates a collection of specific API patterns and potential ways for the compiler to import them. This vision is a sketch, rather than a final design for C++ and Swift interoperability. Towards the end, this document suggests a process for evolving C++ interoperability over time, and it lays out the path for finalizing the designs discussed here. + +“Reverse” interoperability (using Swift APIs from C++) is another extremely important part of the interoperability story. However, reverse interoperability has largely different goals and constraints, which necessarily mean a different design and therefore a different vision document. The [vision for reverse interoperability](https://github.com/swiftlang/swift-evolution/blob/main/visions/using-swift-from-c%2B%2B.md) has already been [accepted](https://forums.swift.org/t/accepted-a-vision-for-using-swift-from-c/62102) by the Language Workgroup. + +This document is an official feature vision document, as described in the [draft review management guidelines](https://github.com/rjmccall/swift-evolution/blob/057b2383102f34c3d0f5b257f82bba0f5b94683d/review_management.md#future-directions-and-roadmaps) of the Swift evolution process. The Language Workgroup has endorsed the goals and basic approach laid out in this document. This endorsement is not a pre-approval of any of the concrete proposals that may come out of this document. All proposals will undergo normal evolution review, which may result in rejection or revision from how they appear in this document. + +## Goals + +There are many reasons for programmers to use C++ from Swift. They might work mostly in Swift but need to take advantage of some code written in C++, anything from a small snippet to a large library. On the other end of the spectrum, they might be C++ programmers looking to adopt Swift as a memory-safe successor language, with a goal of gradually rewriting their codebases into Swift. The foremost goal of Swift's C++ interoperation is to work well for all of these use cases, removing barriers to writing Swift instead of C++, without compromising Swift as a language. + +To do this, **Swift must import C++ APIs safely and idiomatically**. Swift's memory safety is a major feature of its design, and C++'s lack of safety is a major defect. If C++'s unsafety is fully inherited when using C++ APIs from Swift, interoperability will have made Swift a worse language, and it will have undermined one of the reasons to migrate to Swift in the first place. But Swift must also make C++ APIs feel natural to use and fit into Swift's strong language idioms. Often these goals coincide, because the better Swift understands how a C++ API is meant to be used, the more unsafety and boilerplate it can eliminate from use sites. If the Swift compiler does not understand how to import an API safely or idiomatically, it should decline to import it, requesting more information from the user (likely through the use of annotations) so that the API can be imported in a way that meets Swift’s standards. + +For example, many C++ APIs traffic in iterators. Direct uses of C++ iterators are difficult to make safe: iterators are unsafe unless used correctly, and that correctness relies on complex properties (such as the lifetime or consistency of the underlying data) that are impossible to statically enforce. Iterators are also not very idiomatic in Swift because iterator values can only be meaningfully interpreted in pairs (that violate Swift's exclusivity by definition). And iterator properties are often inconsistently defined, making them hard to use. So, Swift should recognize common C++ patterns like ranges (pairs of iterators) and containers and map them into Swift `Collection`s, making them automatically work with Swift's library of safe and idiomatic collections algorithms. For example, Swift code should be able to filter and map the contents of a `std::vector`: + +```Swift +images // "images" is of type std::vector + .filter { $0.size > 256 } + .map(UIImage.init) +``` + +This level of idiomatic interoperation allows programmers to immediately see the benefits of adopting Swift, even when using C++ APIs. It makes the two languages work cleanly together. It removes the need for extensive C or Objective-C bridging layers between C++ libraries and their Swift clients, which are often the source of bugs, performance problems, and expressivity restrictions. And when combined with "reverse" interop that exposes Swift APIs to C++, it allows Swift to be added incrementally to an existing C++ codebase and interoperate on a file-by-file basis, enabling it to function as a viable successor language for programmers looking to move past C++. + +Swift has had great success as a successor language to Objective-C with this approach of bidirectional, file-by-file interoperation. While the constraints and trade-offs of interoperating with C++ are vastly different from Objective-C, the same overall philosophy can largely be applied to make Swift an excellent C++ successor, because it permits the incremental adoption of Swift in a codebase rather than relying on all-at-once rewrites. To make this viable, interoperation must not rely on radically changing interfaces on either side of the language barrier. For Objective-C, Swift takes the approach of largely incorporating Objective-C by inclusion: most major Objective-C features have corresponding Swift features that are at least as expressive. This is not desirable for C++, most importantly because most of the unsafety of C++ arises from its widespread and idiomatic use of unmanaged pointer and reference types; naively translating these all to `UnsafePointer`s would create an unidiomatic and unsafe mess. So successful import of C++ to Swift must rely on recognizing patterns of how these features are used, perhaps with user guidance, and mapping them to more idiomatic Swift constructs. This idea has proven successful with Objective-C, such as methods with `NSError**` parameters being translated to `throws`; C++ will just need to use it more pervasively. + +Because of this, importing C++ APIs into Swift is a difficult task that must be handled with care. Almost every goal of C++ interoperability will be in tension with Swift's safety requirements. Swift must strike a careful balance in order to maintain Swift's safety without reintroducing the development, performance, or expressivity costs of an intermediate wrapper API. + +Safety is a top priority for the Swift programming language, which creates a tension with C++. While Swift enforces strong rules around things like memory safety, mutability, and nullability, C++ largely makes the programmer responsible for handling them correctly, on pain of undefined behavior. Simply using C++ APIs should not completely undermine Swift's language guarantees, especially guarantees around safety. At a minimum, imported C++ APIs should generally not be less safe to use from Swift than they would be in C++, and C++ interoperability should strive to make imported APIs *safer* in Swift than they are in C++ by providing safe API interfaces for common, unsafe C++ API patterns (such as iterators). When it is possible for the Swift compiler to statically derive safety properties and API semantics (i.e., how to safely use an API in Swift) from the C++ API interface, C++ interoperability should take advantage of this information. When that is not possible, C++ interoperability should provide annotations to communicate the necessary information to use these APIs safely in Swift. When APIs cannot be used safely or need careful management, Swift should make that clear to the programmer. As a last resort, Swift should make an API unavailable if there's no reasonable path to a sufficiently safe Swift interface for it. + +C++ interoperability should strive to have good diagnostics. Diagnostics that report source locations for a C++ API should refer to the API's original declaration in a C++ header, not to a location in a synthesized interface file. When a C++ API can be imported into Swift, diagnostics from misusing it (e.g. type errors when passing it an argument of the wrong type) should be similar to the diagnostics for analogous misuses of a Swift API. When a C++ API cannot be imported, attempts to use it should result in a clear error indicating why the API could not be imported, and the diagnostics should suggest specific ways that the programmer could make it importable (for example, by adding annotations). + +C++ provides tools to create high-performance APIs. The Swift compiler should embrace this. Interop should not be a significant source of overhead, and performance concerns should not be a reason to continue using C++ to call C++ APIs rather than Swift. + +C++ is a multi-paradigm language, designed to fit many use cases and allow many different programming styles. Different codebases often express the same concept in different ways. There is no prevailing consensus among C++ programmers about the right way to express specific concepts: how to name types and methods, how much to use templates, when to use heap allocation, how to propagate and handle errors, and so on. This creates problems for importing C++ APIs into Swift, which tends to have stronger conventions, some of which are backed by language rules. For instance, it is a common pattern in some C++ codebases to have classes that are only (or at least mostly) intended to be heap-allocated and passed around by pointer; consider this example: + +```cpp +// StatefulObject has object identity and reference semantics: +// it should be constructed with "create" and used via a pointer. +struct StatefulObject { + StatefulObject(const StatefulObject&) = delete; + StatefulObject() = delete; + + StatefulObject *create() { return new StatefulObject(); } +}; +``` + +This type is not intended to be used directly as the type of a local variable or a `std::vector` element. Values of the type are allocated on the heap by the `create` method and passed around as a pointer. This is weakly enforced by the way the type hides its constructors, but mostly it's communicated in the documentation and by the overall shape of the API. There is no C++ language feature or programming pattern that directly expresses these semantics. + +If `StatefulObject` were written idiomatically in Swift, it would be defined as a `class` to make it a reference type. This is an example of how Swift defines clear patterns for naming, generic programming, value categories, error handling, and so on, which codebases are encouraged to use as standard practices. These well-defined programming patterns make using Swift APIs a cohesive experience, and C++ interoperability should stive to maintain this experience for Swift programmers using C++ APIs. + +To achieve that, the compiler should map C++ APIs to one of these specific Swift programming patterns. In cases where the most appropriate Swift pattern can be inferred by the Swift compiler, it should map the API automatically. Otherwise, Swift should ask programmers to annotate their C++ APIs to guide how they are imported. For example, Swift imports C++ types as structs with value semantics by default. Because `StatefulObject` cannot be copied, Swift cannot import it via the default approach. To be able to use `StatefulObject`, the user should annotate it as a reference type so that the compiler can import it as a Swift `class`. Information on how to import APIs, such as `StatefulObject`, cannot always be statically determined (for example, `StatefulObject` might have been a move-only type, a singleton, or RAII-style API). The Swift compiler should not import APIs like `StatefulObject` for which it does not have sufficient semantic information. It is not a goal to import every C++ API into Swift, especially without additional, required information to present the API in an idiomatic way that promotes a cohesive Swift experience. + +Because of the difference in idioms between the two languages, and because of the safety concerns when exposing certain APIs to Swift, a C++ API might look quite different in Swift than it does in C++. It is a goal of C++ interoperability to provide a clear, well-defined mapping for whether and how APIs are imported into Swift. Users should be able to read the C++ interoperability documentation to have a good idea of how much of their API will be able to imported and what it will look like. Swift should also provide tools for inspecting what a C++ API will look like in Swift, and these tools should call out notable parts of the API that were not imported. + +## The approach + +Many C++ constructs have a clear, analogous mapping in Swift. These constructs can be easily and automatically imported to their corresponding Swift constructs. For example, C++ `enum`s and `enum class`es can be mapped to Swift `enum`s, and C++ operators can usually be mapped to similar Swift operators. Sometimes, to promote Swift’s idioms, operators should be imported "semantically" rather than directly. For example, `operator++` should map to a `successor` method in Swift, and `operator*` should map to a `pointee` property. Another example is a C++ `namespace`, which can be mapped to an empty `enum`, a common pattern in Swift. + +Swift and C++ both support object-oriented programming, and most C++ object-oriented features can be mapped trivially onto Swift counterparts: a member function in C++ translates to a method in Swift, and so on. Like Swift, C++ programming is often focused around types with value semantics, and the natural default when importing a C++ `class` or `struct` is to map it to a Swift `struct`. Copying and destroying this `struct` can simply invoke the corresponding C++ special members as appropriate. This fits nicely into Swift’s existing object model and allows most types to automatically work in Swift in an idiomatic, natural way. + +However, not all C++ `struct`s and `class`es are intended to be used as value types. The `StatefulObject` example in the Goals section shows how the basic tools provided by C++ are often used in idiomatically different ways, and this can be hard to automatically detect when importing the type. This is one example of a deeper conundrum when importing C++ APIs. A more fundamental place to see this is with memory management. + +In Objective-C, it's fairly straightforward for ARC to ensure that data is valid when it's used. Almost all data in Objective-C is represented with either a fundamental type (such as `double` or `BOOL`) or a reference-counted object type (such as `NSString *`). Values of fundamental types can be safely used without any concern about memory management, and reference-counted objects can be safely managed by ensuring that they're retained while they're still potentially used. The compiler will sometimes be more conservative about reference counts than a human would be, extending the lifetime of an object longer than is strictly necessary, but this usually doesn't change the semantics of the program. ARC doesn't manage C pointers, but it's relatively rare to work with C pointers in Objective-C, and it's rarer still to work with a C pointer that has a dependency on a managed object (although exceptions do exist, such as [`NSData`'s `-bytes` method](https://developer.apple.com/documentation/foundation/nsdata/1410616-bytes?language=objc)). This kind of dependency is problematic for safe memory management because the language often does not know about it and cannot ensure that the backing object stays valid while the pointer is being used. Swift's Objective-C interop has treated these as special cases (with an attribute) and dealt with them individually; while this solution is imperfect in several ways, its rarity has made it a low priority to fix. + +In contrast, it's very common for C++ APIs to work with unmanaged pointers, references, and views into other objects. The lifetime rules for using these correctly are inconsistent and sometimes unique to an API. As an example, consider three values: a value of type `std::vector`, a reference returned from that vector's `operator[]`, and an iterator returned from that vector's `begin()` method. At first glance, these values look similar to the compiler: they are all either pointers or class objects containing pointers. But each has its own semantics and expected use (especially concerning lifetime), and these differences are not conveyed explicitly in the source. The vector is a value type that can be copied, but copies can be expensive, and iterators and references into the vector are only valid for a specific copy. The result of `operator[]` is a mutable projection of a specific element, dependent on the vector for validity; but the value of that element can be copied out of the reference to get an independent value, and that is often how the operator is used. The iterator is also a projection, dependent on the vector for validity, but it must be used in conjunction with other iterators or with the vector itself in certain careful ways, and some operations will invalidate it completely. + +So there is a conundrum where superficially similar language constructs in C++ are used to express idiomatic patterns that are vastly different in their impact. The only viable approach for addressing this problem is to pick off these patterns one at a time. The Swift compiler will know about many possible C++ API patterns. If a C++ API has semantic annotations telling Swift that it follows a certain pattern, Swift will try to create a Swift interface for it following the rules of that pattern. In the absence of those annotations, Swift will try to use heuristics to recognize an appropriate pattern. If this fails, Swift will make the API unavailable. + +Consider how this applies to the `std::vector` example. `std::vector` maps over well as a Swift value type. Its `operator[]` can be imported as a Swift `subscript`, and the importer can take advantage of the fact that it returns a reference to allow elements to be efficiently borrowed. And while C++ iterators in general pose serious lifetime safety problems in Swift, Swift can recognize the common `begin()`/`end()` pattern and import it as a safe Swift iterator that encapsulates the unsafety internally. The following sections will go into detail explaining how each of these specific API patterns can be recognized in a C++ codebase. + +### Importing types + +One of the most common uses of this "API patterns" concept concerns the import of types. Swift types fall into two categories: value types and reference types. Copying a value of a reference type produces a new reference to the same underlying object, similar to an intrusive `std::shared_ptr` in C++ or a class type in Java. Copying a value of a value type recursively copies the components of the type to produce an independent value, similar to the behavior of a struct in C or the default behavior of a class type in C++. Furthermore, both kinds of types must always be copyable, although there are plans in the works to allow types to restrict this. + +Types in C++ do not always fit cleanly into this model, and they cannot always be automatically mapped to it even when they do. Many C++ classes are meant to be used as value types, but there are also quite a few C++ classes that Swift programmers would think of as reference types. The difference is not necessarily obvious in source. The closest bit of information that C++ provides directly is whether and how a class has changed its value operations (its copy and move constructors, its assignment operators, and its destructor). A reference type is more likely to delete its copy operations, while a value type is likely to still provide them. But this is not a reliable signal, because some value types are meant to be uncopyable (or even unmovable), while some reference types leave their copy operations intact, either by neglect or to enable objects to be easily cloned when necessary. Furthermore, C++ types sometimes have a more hybrid semantics: iterators, for example, can be used like values, but they're not independent from their underlying collection and in some ways act like references. And some C++ types aren't meant to be used as normal values at all; instead they fill specific idiomatic purposes, like the proxy element references used by `std::vector`, or scoped-destructor types like `std::lock_guard`. + +### Reference types + +Reference types generally fit well into the existing Swift model, and there is little need to restrict them. The safety properties of managed reference types imported from C++ are generally similar to both Swift's own classes and classes imported from Objective-C. The design below also includes unmanaged reference types, which are less safe than managed types, but not more unsafe than writing the code in C++. Overall, this allows C++ interoperability to offer a clear, native-feeling mapping for several common C++ API patterns. + +#### Criteria for importing as a reference type + +Whether a C++ class type is appropriate to import as a reference type is a complex question, and there are several criteria that go into answering it. + +The first criterion is whether object identity is part of the "value" of the type. Is comparing the address of two objects just asking whether they're stored at the same location, or it is deciding whether they represent the "same object" in a more significant sense? For example, consider a computer game that uses a world model where each object of the `GameObject` class represents a different game object. Copying an object actually means making a second object in the game world, one which initially shares the same internal data as another. This is a classic use of reference semantics, and `GameObject` is clearly a reference type. In contrast, a different game might use a world model where the `GameObjectState` class holds a snapshot of the current state of a game object. The actual game object is identified as part of that snapshot, but it's not synonymous with the snapshot, and copying the snapshot just produces an equivalent snapshot of the same object. This design does not rely on object identity; if `GameObjectState` is a reference type, it is because of some other factor. + +The second criterion is whether the C++ class is polymorphic. Does the class have subclasses whose objects contain additional data or behave differently from objects of the parent class? Swift value types cannot be directly polymorphic, so if polymorphism is an important part of a C++ class, it must be imported as a reference type. The most common indicator of a polymorphic C++ class is having `virtual` methods. More rarely, some C++ classes behave polymorphically but intentionally avoid having `virtual` methods to eliminate the memory overhead of a v-table pointer in every object. + +The third and final criterion whether objects of the C++ class are always passed around by reference. Are objects predominantly passed around using a pointer or reference type, such as a raw pointer (`*`), raw reference (`&` or `&&`), or smart pointer (like `std::unique_ptr` or `std::shared_ptr`)? When passed by raw pointer or reference, is there an expectation that that memory is stable and will continue to stay valid, or are receivers expected to copy the object if they need to keep the value alive independently? If objects are generally allocated and remain at a stable address, even if that address is not semantically part of the "value" of an object, the class may be idiomatically a reference type. This will sometimes be a judgment call for the programmer. + +Most of these criteria are not possible for a compiler to answer automatically by just looking at the code. A compiler cannot know the semantic meaning of object identity for a class type. Nor can can it know whether it is looking at a representative sample of how a type is passed around in a project. Classes satisfying these criteria will have to be annotated somehow to tell the compiler to import them as Swift classes. The one exception is that it might be reasonable to assume that a C++ class with `virtual` functions should be imported as a reference type. + +#### Object management + +Swift generally promises to make sure that objects are valid when used. This is an important part of Swift's core language goal of memory safety. Ideally, when Swift imports a C++ class as a reference type, it will import it as an appropriately managed type that receives the same guarantees as native Swift and imported Objective-C classes. + +It's useful to split the object-management problem into two questions: how objects are managed and whether they can be managed automatically. + +There are three common patterns for managing reference object lifetimes in C++. Swift should endeavor to support all three of them: + + - **Immortal** reference types are not designed to be managed individually by the program. Objects of these types are allocated and then intentionally "leaked" without tracking their uses. Sometimes these objects are not truly immortal: for example, they may be arena-allocated, with an expectation that they will only be referenced from other objects within the arena. Nonetheless, they aren't expected to be individually managed. + + The only reasonable thing Swift can do with immortal reference types is import them as unmanaged classes. This is perfectly fine when objects are truly immortal. If the object is arena-allocated, this is unsafe, but it's essentially an unavoidable level of unsafety given the choices of the C++ API. + + - **Unique** reference types are owned by a single context at once, which must ultimately either destroy it or pass ownership of it to a different context. There are two common idioms for unique ownership in C++. The first is that the object is passed around using a raw pointer (or sometimes a reference) and eventually destroyed using the `delete` operator. The second is that this is automated using a move-only smart pointer such as `std::unique_ptr`. This kind of use of `std::unique_ptr` is often paired with "borrowed" uses that traffic in raw pointers temporarily extracted from the smart pointer; in particular, method calls on the class via `operator->` implicitly receive a raw pointer as `this`. + + The introduction of [non-copyable types](https://forums.swift.org/t/pitch-noncopyable-or-move-only-structs-and-enums/61903) will allow Swift to directly support unique reference types as managed types. The main challenge in doing this will be understanding the ownership conventions for different C++ APIs. If ownership of a class is known to be passed around with a smart pointer like `std::unique_ptr`, then APIs trafficking in raw pointers can be assumed to be working with a borrow; that would support importing as a managed type. Otherwise, Swift will have to put the programmer in charge and either import as an unmanaged type or use a wrapper type like `Unmanaged` to mediate APIs with unknown conventions. + + - **Shared** reference types are reference-counted with custom retain and release operations. In C++, this is nearly always done with a smart pointer like `std::shared_ptr` rather than expecting programmers to manually use retain and release. This is generally compatible with being imported as a managed type. Shared pointer types are either "intrusive" or "non-intrusive", which unfortunately ends up being relevant to semantics. `std::shared_ptr` is a non-intrusive shared pointer, which supports pointers of any type without needing any cooperation. Intrusive shared pointers require cooperation but support some additional operations. Swift should endeavor to support both. + + As with unique reference types, shared reference types in C++ often have APIs that take raw pointers, such as methods on the class type. Unlike unique references, these cannot necessarily be thought of as borrows. Shared reference types are copyable types, and as a general rule, borrowed copyable values can be copied to produce owned values. However, in C++ terms, this would require constructing a shared pointer value from a raw pointer, which in general is not possible to do correctly for non-intrusive shared pointers. It's fine for Swift to *call* APIs that take raw pointers for shared reference types, but it cannot *implement* them without having a way to prevent copying the reference. + + `std::shared_ptr` uses atomic reference-counting in both its intrusive and non-intrusive modes. Non-atomic smart pointer types are supportable, but the imported class type must not be `Sendable`. + +[Examples of each of these are given below.](## Examples and Definitions) + +If a type is annotated as using one of these reference-type patterns, uses of the type in C++ that do not have a consistent interpretation under the pattern will be impossible to import. For example, suppose that `GameObject` is annotated as a shared reference type that uses `std::shared_ptr`. A C++ API that takes a parameter of type `std::unique_ptr`, or a different shared pointer class from the annotated one, must be made unavailable. (This would also include differences in secondary template arguments, such as the `Deleter` template argument of `std::unique_ptr`.) Similarly, a C++ API that takes a `GameObject` as an r-value must be made unavailable. + +Swift doesn't have to force programmers to pick one of these patterns specifically. Without further information, foreign reference types can be imported as an unmanaged class type. There would be an operation on an object to delete it, but if that's not safe to use, the programmer could simply not use it. This behavior would allow types with reference semantics to be expressed with little effort at the cost of complete safety. For some types this may be acceptable or even necessary. However, Swift should strive to make it easy to import types as managed class types, especially for APIs that already make extensive use of smart pointers. + +### Value types + +For the purposes of this document, a value type is any type that doesn't make sense to import as a reference type. Value types can be copied and destroyed, and the copies will be independent from each other, at least at the direct level. That is, part of the value of a value type might be a reference to an object (e.g. if it has a stored property of class type), and different copies of the value will share the same reference, but this reference can still be replaced in one copy without affecting other copies. + +Swift expresses value types using `struct`s and `enum`s. Copying a Swift `struct` normally does the same thing that copying a C or C++ `struct` does by default: it recursively copies all of the stored properties of the type. In C++, of course, that behavior can be customized with user-defined copy/move constructors and destructors; while Swift doesn't have an equivalent feature, it does still honor the operations specified in C++, so that destroying a value of the imported type in Swift calls the C++ destructor and so on. + +It's useful to call out three categories of value types. These categories don't necessarily change how the type is actually used in Swift, but they are essential to describing the interop story, safety and performance properties, potential API restrictions, and the user model more generally. + +#### Simple data types + +This document will refer to C++'s trivially-copyable value types that do not contain pointers as “simple data types.” This category includes fundamental types, such as integers and floating-point types, as well as aggregate types composed only of other simple data types. Simple data types have trivial value operations and never carry lifetime dependencies on other values. Simple data types and operations on them generally don't need any special restrictions in Swift. + +Swift will assume by default that lifetime dependencies aren't carried in integer types even though technically pointers can be reinterpreted into integer types. It's very uncommon for C++ APIs to violate this assumption: reinterpeting pointers as integers is important to a fair amount of code, but usually it's localized and transient and the pointer doesn't get passed around as an integer long-term. + +#### Self-contained types + +This category is a superset of the simple data types which also includes types with internally-managed pointers. Like simple data types, these types and their operations usually don't need any special restrictions in Swift. However, the fact that they can be non-trivial types can complicate some things. + +Swift will assume by default that a C++ `class` type which contains pointers but also provides user-defined special members is self-contained. This is a fairly reliable heuristic, but more consideration may be required in order to handle cases such as `std::vector`, which can carry lifetime dependencies indirectly even though it does manage the pointers it stores directly. In any case, Swift must provide annotations that allow the default to be corrected. + +#### View types + +This document will refer to value types that are not self-contained types as "view types". These types include pointers themselves as well as types which are recursively composed of other view types. The pointers held by view types refer to memory that is *not owned* by the pointer type (making view types a “view” into that memory rather than a value that encapsulates it). View types usually carry a dependency on some other value and must be used carefully to be safe. + +While trivially-copyable view types are very similar to simple data types with respect to their trivial value operations, they differ in the fact that, while they themselves are not inherently unsafe, they may be used in unsafe APIs (discussed later). + +### Projections + +The safety problem posed by view types is very broad. If we apply this categorization to the types offered natively by Swift, most types are self-contained; only the unsafe pointer types are view types. Swift also encourages view types to be encapsulated within types and exposed in only carefully-scoped ways, like with a `with...` API. These properties are not true for many C++ APIs, which offer a broad spectrum of novel view types and ways to project them out of managed types. Swift does not currently offer strong language tools for dealing with these projections. + +Probably the most common pattern of projection in C++ APIs is a method that returns a reference or pointer to memory that depends on `this`. Consider this example: + +```cpp +const std::string &getName() const { return this->_name; } +``` + +There are several possibilities for dealing with this pattern. For example, Swift could wrap it in a `_read` accessor, implicitly encoding that the reference is only available during an access to the containing object. Swift could also add explicit lifetime-dependency features, allowing this to be treated as a return of a borrowed value. Alternatively, Swift could simply force the return value to be immediately copied after return, as if the call actually returned an owned value. It's unclear which of these would be the best approach; perhaps a combination would. This is something that will need to be investigated over time, incorporating the experience of the community with using this feature. + +But there are also many projections in C++ that don't match the above pattern. Consider the following API which returns a vector of internal pointers: +```cpp +std::vector OwnedType::projectsInternalStorage(); +``` + +Or this API which fills in a pointer that has two levels of indirection: +```cpp +void VectorLike::begin(int **out) { *out = data(); } +``` + +Or even this global function that projects one of its parameters: +```cpp +int *begin(std::vector *v) { return v->data(); } +``` + +Swift will need to decide how to handle projections, and more generally the use of view types, that it doesn't recognize how to make safe. This may come with difficult trade-offs between usefulness and safety. Consider the `projectsInternalStorage` API above, and pretend that we aren't able to recognize a pattern here --- we don't know that the pointers just depend on the `OwnedType` object. Often, uses of this kind of API can be made safe in practice: when there's a `std::vector` of pointers, there's probably some straightforward thing making those pointers valid. If Swift decides not to import this API just because it might be used unsafely, that could do serious damage to the usability of C++ interop. At the very least, it should be possible to wrap such an API in a safer Swift abstraction, which will be impossible if it isn't imported at all. + +The trade-offs here are an open question for the Swift evolution process to eventually determine. + +### Iterators + +Both Swift and C++ have powerful libraries for algorithms and iterators. The standard C++ iterator API interface lends itself to the Swift model, allowing C++ iterators and ranges to be mapped to Swift iterators and sequences with relative ease. These mapped APIs are idiomatic, native Swift iterators and sequences; their semantics match the rest of the Swift language and Swift APIs compose around them nicely. By taking on Swift iterator semantics, iterators that are imported in this way are able to side-step most or all of the issues that other projects have (described above). + +Swift's powerful suite of algorithms match and go beyond the standard library algorithms provided by C++. These algorithms compose on top of protocols such as Sequence, which C++ ranges should automatically conform to. These Swift APIs and algorithms that operate on Swift iterators and sequences should be preferred to their C++ analogous, as they fit into the rest of the language naturally. However, algorithms are not the only API which operate on iterators and sequences and other C++ APIs must still be useable from Swift. The best way to represent C++ APIs that take one or many iterators (potentially pointing at the same range) is not clear and will need to be explored during the evolution processes. + +### Mutability + +Swift and C++ use very different models for controlling mutability. C++ defaults to treating methods, pointers, and references as non-`const`, and any `const`-ness can be easily cast away, which together mean that `const`-ness is not always a very reliable signal. C++ also encodes mutability into the type system in a first-class way, allowing functions to be overloaded bsaed on whether an argument is `const` or not. These decisions don't always align well with Swift, which defaults to immutability and relies on local information to strictly enforce it. + +One consequence is that C++ codebases which haven't adopted `const` correctness can be confusing to awkward to use from Swift because many operations which are not actually mutating appear to require mutability. Swift should encourage C++ codebases to adopt `const` on methods and values that are used in Swift. + +Overloaded functions should be clearly be disambiguated in Swift through naming. Mutability is a place where programmers may need to intervene and provide Swift with more information to help promote idiomatic APIs that are expressive and feel natural in Swift. + +Swift should also assume that C++ will not mutate values through `const` pointers and references, even though technically `const`-ness can cast away. It is reasonable for Swift to assume that C++ APIs will obey their type signatures, and the alternative would be very onerous for interop users. + +As discussed in the "View types" section, the Swift compiler must make assumptions about the C++ APIs that it is importing, and mutability is another place where Swift will need to make reasonable (not conservative) assumptions about the APIs that it is importing, promoting C++‘s weak notion of `const` to Swift’s much stricter ideal. + +Programmers will see some benefits from Swift's stronger mutability model immediately. Consider this example: +```cpp +// C++ +void append_n_times(std::string& s, const std::string& m, size_t n) { + for (size_t i = 0; i < n; ++i) + s += m; +} +``` +```swift +// Swift +var local: std.string = "a" +append_n_times(&local, local, 5) +``` +`append_n_times` misbehaves if `s` and `m` alias because `m` will be modified by the append: `s` will contain `2^n` copies of the original string instead of `n + 1`. This is not possible when called from Swift because Swift does not permit mutable arguments to be aliased, so the argument for `m` will be copied. (And if the programmer requests that this copy not happen, e.g. by explicitly borrowing that argument, the call will be statically diagnosed as ill-formed.) + +### Computed properties + +Value vs. reference types and mutability may require user input to map correctly in Swift, but constructs like iterators, getters, and setters can largely be imported automatically. Getters, setters, and subscripts can all be imported into Swift as computed properties. While many C++ codebases define a getter and setter pair, computed properties are the idiomatic way to handle this API pattern in Swift. And computed properties are not just about syntax, they also help promote safety and performance. For example, a C++ getter that returns a reference can be mapped into a generalized accessor in Swift that leverages coroutines to safely yield out its storage to the caller. This generalized accessor pattern allows safe and efficient access to C++ references in Swift and is another example of the more general philosophy for importing APIs: when Swift understands the semantics of an API it can map that API pattern to a strict Swift idiom that is safe, performant, and feels native, so that users get most of the benefits of Swift, even when calling C++ APIs. + +### Templates and generic APIs + +C++ and Swift use very different models for generic programming. C++ templates are eagerly instantiated for each set of template arguments they're used with, with type checking done separately for each instantiation based on the exact types in use. In contrast, Swift generics are type-checked once based on the requirements they impose on their type parameters, and while they can be specialized for a particular set of type arguments, that is not required or even always possible. + +This difference makes using generic C++ APIs in Swift difficult. Generic code in Swift will not be able to use C++ templates generically without substantial new language features and a lot of implementation work. Allowing C++ templates to be used on concrete Swift types is theoretically more feasible but still a major project because of the *ad hoc* nature of type constraints in templates. If this feature is ever pursued, it will likely require substantial user guidance through annotations or wrappers around imported APIs. + +Fortunately, these limitations do not apply when using C++ types with Swift generics. Unconstrained generics can be used with C++ types without any further work, and programmers can simply add protocol conformances to concrete C++ types in order to use them with constrained generics. + +[This forum post](https://forums.swift.org/t/bridging-c-templates-with-interop/55003) (Bridging C++ Templates with Interop) goes into depth on the issue of importing C++ templates into Swift. + +## The standard library + +Swift should provide an overlay for the C++ standard library to assist in the import of commonly used APIs, such as containers. This overlay should also provide helpful bridging utilities, such as protocols for handling imported ranges and iterators, or explicit conversions from C++ types to standard Swift types. + +C++ aims to provide sufficient tools to implement many features in its standard library rather than the compiler. While the Swift compiler also attempts to do this, it is not a goal in and of itself, resulting in many of C++'s analogous features being implemented in the compiler: tuples, pairs, reference counting, ownership, casting support, optionals, and so on. In these cases, the Swift compiler will need to work with both the C++ standard library and the Swift overlay for the C++ standard library to import these APIs correctly. + +The reverse is also true: C++ interop may require library-level Swift utilities to assist in the import of various C++ language concepts, such as iterators. To support this case, a set of Swift APIs specific to C++ interop will be imported implicitly whenever a C++ module is imported. These APIs should not have a dependency on the distinct C++ standard library or its overlay. + +## Evolution + +C++ interoperability is a huge feature that derives most of its benefit from the combination of its component features; for example, methods can't be used without types. C++ interop should be made useful to programmers before all component pieces have necessarily gone through evolution, both for the benefit of programmers wanting to use this feature, and for compiler developers designing and implementing the feature. + +C++ interoperability should bring in as many APIs as possible, even if they haven't gone through evolution. Swift evolution will progressively work through these APIs, formalizing them, and eventually interop will become a stable feature. Until a critical mass of APIs have been brought Swift's evolution process, a versioning scheme will allow C++ interoperability to be adopted and remain source stable while being evolved. Versions may be rapidly deprecated, but will be independent of Swift compiler versions, allowing source breaks even in minor compiler updates without disturbing adopters. + +This document allows specific, focused, and self contained evolution proposals to be created for individual pieces of the language and specific programming patterns by providing goals that lend themself to this kind of incremental design and evolution (by not importing everything and requiring specific mappings for specific API patterns) and by framing interop in a larger context that these individual evolution proposals can fit into. + +## Tooling and build process + +As a supported language feature, C++ and Swift interoperability must work well on every platform supported by Swift. In a similar vein, tools in the Swift ecosystem should be updated to support interoperability features. For example, SourceKit should provide autocompletion, jump-to-definition, etc. for C++ functions, methods, and types, and lldb should be able to print C++ types even in Swift frames. Finally, the Swift package manager should be updated with the necessary features to support building C++ dependencies. + +This document outlines a strategy for importing APIs that rely on semantic information from the user. In order to make this painless for users across a variety of projects, Swift will need to provide both inline annotation support for C++ APIs and side-file support for APIs that cannot be updated. For Objective-C, this side-file is an APINotes file. As part of Swift and C++ interoperability, APINotes will either need to be updated to support C++ APIs, or another kind of side-file will need to be created. + +## Appendix 1: Examples and Definitions + +**Reference Types** have reference semantics and object identity. A reference type is a pointer (or “reference”) to some object which means there is a layer of indirection. When a reference type is copied, the pointer’s value is copied rather than the object’s storage. This means reference types can be used to represent non-copyable types in C++. For real-world examples of C++ reference types, consider LLVM's [`Instruction` class](https://llvm.org/doxygen/IR_2Instruction_8h_source.html) or Qt's [`QWidget` class](https://github.com/qt/qtbase/blob/dev/src/widgets/kernel/qwidget.h). + +**Manually Managed Reference Types** + +Here a programmer has written a very large `StatefulObject` which contains many fields: + +```cpp +struct StatefulObject { + std::array names; + std::array places; + // ... + + StatefulObject(const StatefulObject&) = delete; + StatefulObject() = delete; + + StatefulObject *create() { return new StatefulObject(); } +}; +``` + + +Because this object is so expensive to copy, the programmer decided to delete the copy constructor. The programmer also decided that this object should be allocated on the heap, so they decided to delete the default constructor, and provide a create method in its place. + +In Swift, this `StatefulObject` should be imported as a reference type, as it has reference semantics. + +**API Incorrectly Using Reference Types** + +Here someone has written an API that uses `StatefulObject` as a value type. + +```cpp +StatefulObject makeAppState(); +``` + +This will invoke a copy of `StatefulObject` which violates the semantics that the API was written with. To be usable from Swift, this API needs to be updated to pass the object indirectly (by reference): + +```cpp +StatefulObject *makeAppState(); // OK +const StatefulObject *makeAppState(); // OK +StatefulObject &makeAppState(); // OK +const StatefulObject &makeAppState(); // OK +``` + +**Immortal Reference Types** + +Instances of `StatefulObject` above are manually managed by the programmer, they create it with the create method and are responsible for destroying it once it is no longer needed. However, some reference types need to exist for the duration of the program, these reference types are known as “immortal.” Examples of these immortal reference types might be pool allocators or app contexts. Let’s look at a `GameContext` object which allocates (and owns) various game elements: + +```cpp +struct GameContext { + // ... + + GameContext(const GameContext&) = delete; + + Player *createPlayer(); + Scene *createScene(); + Camera *createCamera(); +}; +``` + +Here the `GameContext` is meant to last for the entire game as a global allocator/state. Because the context will never be deallocated, it is known as an “immortal reference type” and the Swift compiler can make certain assumptions about it. + +**Automatically Managed Reference Types** + +While the `GameContext` will live for the duration of the program, individual `GameObject` should be released once they’re done being used. One such object is Player: + +```cpp +struct GameObject { + int referenceCount; + + GameObject(const GameObject&) = delete; +}; + +void gameObjectRetain(GameObject *obj); +void gameObjectRelease(GameObject *obj); + +struct Player : GameObject { + // ... +}; +``` + +Here Player uses the `gameObjectRetain` and `gameObjectRelease` function to manually manage its reference count in C++. Once the `referenceCount` hits `0`, the Player will be destroyed. Manually managing the reference count is prone to errors, as programmers may forget to retain or release the object. Fortunately, this kind of reference counting is something that Swift is very good at. To enable automatic reference counting, the user can specify the retain and release operations via attributes directly on the `GameObject`. This means the programmer no longer needs to manually call `gameObjectRetain` and `gameObjectRelease`; Swift will do this for them. They will also benefit from the suite of ARC optimizations that Swift has built up over the years. + +**Owned types** “own” some storage which can be copied and destroyed. An owned type must be copyable and destructible. The copy constructor must copy any storage that is owned by the type and the destructor must destroy that storage. Copies and destroys must balance out and these operations must not have side effects. Examples of owned types include `std::vector` and `std::string`. + +**Trivial types** are a subset of owned types. They can be copied by copying the bits of a value of the trivial type and do not need any special destruction logic. Examples of trivial types are `std::array` and `std::pair`. + +**Pointer types** are trivial types that hold pointers or references to some un-owned storage (storage that is not destroyed when the object is destroyed). Pointer types are *not* a subset of trivial types or owned types. Examples of pointer types include `std::string_view` and `std::span` and raw pointer types such as `int *` or `void *`. + +**Projections** are values rather than types. An example of a method which yields a projection is the `c_str` method on `std::string`. + +```cpp +struct string { // String is an owned type. + char *storage; + size_t size; + + char *c_str() { return storage; } // Projects internal storage +``` + +Iterators are also projections: + +```cpp + char *begin() { return storage; } // Projects internal storage + char *end() { return storage + size; } // Projects internal storage +``` + +Because `string` is an owned type, the Swift compiler cannot represent a projection of its storage, so the `begin`, `end`, and `c_str` APIs are not imported. A projection is only valid as long as the storage it points to is valid. Projections of reference types are usually safe because reference types have storage with long, stable lifetimes, but projections of owned types are more dangerous because the storage associated with a specific copy usually has a much shorter lifetime (therefore most of these projections of owned storage cannot yet be imported). + + +## Appendix 2: Lifetime and safety of self-contained types and projections + +The following section will go further into depth on the issues with using projections of self contained types in Swift, rather than proposing a solution on how to import them. Let’s start with an example Swift program that naively imports some self-contained type and returns a projections of it: + +```swift +var v = vector(1) +let start = v.begin() +doSomething(start) +fixLifetime(v) +``` + +To understand the problem with this code, the following snippet highlights where an implicit copy is created and destroyed: + +```swift +var v = vector(1) +let copy = copy(v) +let start = copy.begin() +destroy(copy) +doSomething(start) +fixLifetime(v) +``` + +Here, because Swift copies `v` into a temporary with a tight lifetime before the call to `begin`, `v` projects a dangling reference. This is an example of how subtly different lifetime models make using C++ types from Swift hard, if their semantics aren’t understood by the compiler. + +To make these APIs safe and usable, Swift cannot import unsafe projections of types that own memory, because they don’t fit the Swift model. Instead, the Swift compiler can try to infer what, semantically, the API is trying to do, or the library author can provide this information via annotations. In this case, the Swift compiler can infer that begin returns an iterator, which Swift can represent through the existing, safe Swift iterator interface. In the example above, “start” is a pointer type. Using this pointer returned by the “begin” method is unsafe, but the type of start itself is not unsafe. In other words, safety restrictions need not be applied to pointer types themselves but rather their unsafe uses. + +C++ often projects the storage of owned types. C++ is able to tie the lifetime of the projection to the source using lexcal scopes. Because there is a well-defined, lexical point in which objects are destroyed, C++ users can reason about projection’s lifetimes. While these safety properties are less formal than Swift, they are safety properties none-the-less, and form a model that works in C++. + +This model cannot be adopted in Swift, however, because the the same lexical lifetime model does not exist. Further, projections of self-contained types are completely foreign concept in Swift, meaning users aren’t familiar with programming in terms of this lexical model, and may not be aware of the added (implicit) constraints (that is, when objects are destroyed). Swift’s language model is such that returning projections from a copied value, even in smaller lexical scope, should be safe. In order to allow projections of self-contained types, this assumption must be broken, or C++ interoperability must take advantage of Swift ownership features to associate the lifetime of the projection to the source. + +The following example highlights the case described above: + +```swift +func getCString(str: std.string) -> UnsafePointer { str.c_str() } +``` + +The above function returns a dangling reference to `str`‘s inner storage. In C++, it is assumed that the programmer understands this is a bug, and generally would be expected to take `str` by reference. This is not the case in Swift. To represent this idiomatically in Swift, the lifetimes must be associated through a projection. Using the tools provided in the ownership manifesto this would mean yielding the value returned by `c_str` out of a [generalized accessor](https://github.com/apple/swift/blob/main/docs/OwnershipManifesto.md#generalized-accessors)(resulting in an error when the pointer is returned). diff --git a/visions/using-swift-from-c++.md b/visions/using-swift-from-c++.md new file mode 100644 index 0000000000..65583f2ec5 --- /dev/null +++ b/visions/using-swift-from-c++.md @@ -0,0 +1,139 @@ +# Using Swift from C++ + +This vision document presents a high level overview of the "reverse" (i.e. Swift-to-C++) half of the C++ interoperability Swift language feature. It highlights the key principles and goals that determine how Swift APIs are exposed to C++ users. It also outlines the evolution process for this feature. This document does not present the final design for how Swift APIs get mapped to C++ language constructs. The final design of this feature will evolve as this feature goes through the Swift evolution process. This document does not cover the "forward" (i.e. using C++ APIs from Swift) aspect of the C++ interoperability, as it’s covered by a [sibling document](https://github.com/apple/swift/pull/60501/files). + +This document is an official feature vision document, as described in the [draft review management guidelines](https://github.com/rjmccall/swift-evolution/blob/057b2383102f34c3d0f5b257f82bba0f5b94683d/review_management.md#future-directions-and-roadmaps) of the Swift evolution process. The Language Workgroup has endorsed the goals and basic approach laid out in this document. This endorsement is not a pre-approval of any of the concrete proposals that may come out of this document. All proposals will undergo normal evolution review, which may result in rejection or revision from how they appear in this document. + +## Introduction + +Swift currently provides support for interoperability with C and Objective-C. Swift can import C and Objective-C APIs, and it can also expose its `@objc` types and functions to Objective-C. However, Swift currently does not provide support for interoperability with C++. Supporting bidirectional C++ interoperability in Swift would allow it to call into C++ APIs, and would make Swift APIs accessible to C++. Making Swift APIs accessible to C++ would simplify the process of adoption of Swift in existing C++ codebases, as it would allow them to incrementally adopt Swift. Furthermore, it would make Swift-only libraries available to C++ codebases. + +## Overview of Swift's interoperability support + +Swift uses two very different approaches for interoperability with C and Objective-C. For "forward" interoperability, Swift embeds a copy of the Clang compiler, which it uses to directly load C and Objective-C Clang modules and translate declarations into a native Swift representation. For "reverse" interoperability, Swift does not want to require Clang to embed Swift, and so Swift generates an Objective-C header that can be used to call into Swift APIs that are exposed in that header. Because Objective-C is restricted in what kinds of types it can work with, and because it requires code to be emitted in a special way that doesn't match the regular Swift ABI, methods and types must be marked as being exposed to Objective-C with the `@objc` attribute. + +Swift's support for C++ interoperability is modeled after its support for C and Objective-C interoperability. For "forward" interoperability, the embedded Clang compiler is used to directly load C++ Clang modules and translate declarations into a native Swift representation. The "forward" interoperability [vision document](https://github.com/apple/swift/pull/60501/files) provides more details about this model. For "reverse" interoperability, Swift generates a header that uses C++ language constructs to represent Swift APIs that are exposed by the Swift module. Because of the relatively high expressivity of C++ headers, the generated header is able to provide representation for most native Swift functions, methods, initializers, accessors and types without needing any extra code to be generated in the Swift module. This allows C++ programmers to call into Swift APIs using the familiar C++ function and member function call syntax. + +## C++ interoperability does not require Swift opt in + +The user model employed by Swift's C++ interoperability is different than the user model employed by Swift's Objective-C interoperability. All else being equal, it's best if supporting interoperation with a language doesn't require changing source code or emitting extra code eagerly from the compiler. This doesn't work for Objective-C interoperability because Objective-C requires specially-compatible classes and methods, the code for which cannot be emitted from an Objective-C header. Swift chose to require programs to opt in to Objective-C interoperability with the `@objc` attribute, both to make export more predictable and to avoid emitting extra code and metadata for all classes. In contrast, as long as the C++ compiler supports the Swift calling convention, a C++ header can call native Swift functions directly, and the C++ type system can be used to wrap most Swift types in a safe C++ representation. Because of this, there is no reason to require Swift module authors to opt in into C++ interoperability. Instead, any Swift module that can be imported into Swift can also be imported into C++, and most APIs will come across automatically. + +Some API authors will desire explicit control over the C++ API. Swift will provide an annotation such as the proposed [`@expose` attribute](https://forums.swift.org/t/formalizing-cdecl/40677/50) to allow precise control over which APIs get exposed to C++. The API authors will be able to specify that their module should only expose the annotated APIs to C++ when they desire to do so. + +## Goals + +The ultimate goal of allowing Swift APIs to be used from C++ is to remove a barrier to writing more code in Swift rather than C++. Some C++ programmers would like to adopt a memory-safe language like Swift in their codebase. These programmers would typically like to adopt Swift gradually instead of relying on rewriting all of their code in Swift at once. For instance, they might write a new component in Swift, which then needs to be made usable from the rest of their C++ codebase. Today this could be done via an Objective-C wrapper that relies on `@objc` annotations. However, maintaining `@objc` wrapper code is annoying and error-prone, and it often adds significant performance costs and expressivity restrictions because of the limitations of C and Objective-C. Allowing C++ to directly use Swift APIs with minimal restrictions or performance overhead would lower the burden of adding Swift to an existing C++ codebase. It also makes Swift an even better language for stable system libraries and APIs, as it allows C++ projects to consume most such libraries without restrictions. These libraries would also have the option of not providing an Objective-C interface for Objective-C clients, as these clients could use the Swift APIs from Objective-C++ instead. + +The primary goal of Swift-to-C++ interoperability is to expose every language feature of Swift for which a reasonable C++ API can be created to represent that feature. Swift and C++ are both feature rich programming languages. They share many similar features, but Swift has some features that lack close analogues in C++. For example, C++ does not have a concept of argument labels, but the argument labels can be integrated into the base name of a function to get a roughly similar effect. On the other hand, C++ does not have any feature that can represent a Swift feature like Swift macros in a reasonable manner. Therefore, macros must be dropped in the C++ interface to the module. + +It is a goal that Swift-to-C++ interoperability can be used by both mixed Swift/C++ language projects that want to interoperate within themselves, and by C++ projects that need to use a Swift library that wasn't developed with C++ interoperability in mind. + +It is a goal of Swift-to-C++ interoperability to expose Swift APIs in a safe, performant and ergonomic manner. The Swift compiler and the language specification should follow several key related principles that are presented below. + +### Safety + +Safety is a top priority for the Swift programming language. Swift code expects its callers to adhere to Swift’s type rules and Swift’s memory model, regardless of whether it’s called from Swift or C++. Thus, the C++ code that calls Swift should properly enforce Swift’s expected language contracts. The enforcement should be done automatically in the generated header. This kind of enforcement does not prevent all possible issues though, as it does not change C++'s safety model. C++ is unsafe by default, and the user is able to use regular C++ pointers to write to memory as if they were using Swift's `UnsafeMutablePointer` type. This means that bugs in C++ code can easily lead to violations of Swift's invariants. For instance, a bug in user’s C++ code could accidentally overwrite a Swift object stored on the heap, which could cause unexpected behavior (such as a segfault) in Swift code. The user is expected to obey Swift's memory model and type rules when calling into Swift from C++, and thus they bear the ultimate responsibility for avoiding bugs like this one. The user can use certain program analysis tools, such as Address Sanitizer, to help them catch bugs that violate Swift's memory model rules. + +Swift expects that values of correct types are passed to Swift APIs. For instance, for calls to generic functions, Swift expects the caller to pass a value of type that conforms to all of the required generic requirements. Type safety should be enforced from C++ as well. The C++ compiler should verify that correct types are passed to the Swift APIs as they’re invoked from C++. A program that tries to pass an incorrect Swift type into a Swift function should not compile. Select uses of specific Swift types might be incompatible with static type validation. For example, verification of generic requirements for a Swift opaque type value returned from a Swift function in C++ requires run-time validation. The program should report a fatal error when such run-time type validation fails, to ensure that the type invariants that Swift expects are not violated. The reported fatal error should clearly indicate why type validation failed and point to the location in the user's code which caused the run-time check. + +Memory safety is paramount for Swift code. Swift automatically manages the lifetime of value and reference types in Swift. These principles should translate to C++ as well. The lifetime of Swift values that are created or passed around in C++ should be managed automatically. For instance, the generated C++ code should increment the retain count of a Swift class instance when a Swift class type value is copied in C++, and it should decrement the retain count of a Swift class instance when a Swift class type value is destroyed in C++. The default convention for using Swift types in C++ should discourage dangling references and any other patterns that could lead to an invalid use of a Swift value. + +Swift’s memory safety model also requires exclusive access to a value in order to modify that value. For instance, the same value can not be passed to two `inout` parameters in the same function call. Swift enforces exclusivity using both compile-time and run-time checks. The generated run-time checks trap when an exclusivity violation is detected at runtime. Calls into Swift APIs from C++ should verify exclusivity for Swift values as well. + +### Performance + +Swift-to-C++ bridging should be as efficient as possible. The Swift compiler should avoid unnecessary overhead for calling into Swift functions from C++, or using Swift types from C++. Generally, the bridging code should not convert Swift types into their C++ counterparts automatically, as that can add a lot of overhead. For instance, a Swift function returning a `String` should still return a Swift `String` in C++, and not a `std::string`. Some specific "primitive" Swift types should be converted into their C++ counterparts automatically in order to improve their ergonomics in C++. The conversion for such primitive types should be zero-cost as their ABI should match in Swift and C++. For instance, a Swift function returning a `Float` can return a `float` in C++ as they use the same underlying primitive LLVM type to represent the floating point value. + +Some Swift features require additional overhead to be used in C++. Resilient value types are a good example of this; C++ expects types to have a statically-known layout, but Swift's resilient value types do not satisfy this, and so the generated C++ types representing those types may need to dynamically allocate memory internally. In cases like these, the C++ interface should at least strive to minimize the dynamic overhead, for example by avoiding allocations for sufficiently small types. + +### Achieving safety with performance in mind + +Certain aspects of Swift’s memory model impose certain restrictions that create tension between the goal of achieving safety and the goal of avoiding unnecessary overhead for calling into Swift from C++. Checking for exclusivity violations is a good example of this. The C++ compiler does not have a notion of exclusivity it can verify, so it is difficult to prove that a value is accessed exclusively in the C++ code that calls into Swift. This means that the C++ code that calls into Swift APIs will most likely require more run-time checks to validate exclusivity than similar Swift code that calls the same Swift APIs. + +The adherence to Swift’s type and memory safety rules should be prioritized ahead of performance when C++ calls into Swift, even if this means more run-time checks are required. For users seeking maximum performance, Swift provides additional compiler flags that avoid certain run-time checks. Those flags should be taken into account when the C++ header is generated. An example of such flag is `-enforce-exclusivity`. When `-enforce-exclusivity=none` is passed to Swift, the Swift compiler does not emit any run-time checks that check for exclusivity violations. A flag like this should also affect the generated C++ header, and the Swift compiler should not emit any run-time checks for exclusivity in the generated C++ header when this flag is used. + +Correct modeling of certain aspects of Swift type semantics in C++ requires additional overhead. This issue is most prominent when modeling Swift's `consume` operation, as it performs a destructive move, which C++ does not support. Thus, the `consume` operation has to be modeled using a non-destructive move in C++, which requires additional storage and runtime checks when a Swift value type is used in C++. In cases like this one, the design of how Swift language constructs are mapped to C++ should strive to be as ergonomic and as intuitive as possible, provided there's still a way to achieve appropriate performance using compiler optimizations or future extensions to the C++ language standard. + +### Ergonomics + +Swift APIs should be mapped over to C++ language features that have a direct correspondence to the Swift language feature. In cases where a direct correspondence does not exist, the Swift compiler should provide a reasonable approximation to the original Swift language feature using other C++ constructs. For example, Swift’s enum type can contain methods and nested types, and such constructs can’t be represented by a single C++ enum type in an idiomatic manner. Swift’s enum type can be mapped to a C++ class instead that allows both enum-like `switch` statement behavior and also enables the C++ user to invoke member functions on the Swift enum value and access its nested types from C++. + +The C++ representation of certain Swift types should be appropriately enhanced to allow them to be used in an idiomatic manner. For instance, it should be possible to use Swift’s `Array` type (or any type that conforms to `Collection`) in a ranged-based for loop in C++. Such enhancements should be done with safety in mind, to ensure that Swift’s memory model is not violated. + +There should be no differences on the C++ side between using libraries that opt-in into library evolution and libraries that don’t, except in specific required cases, like checking the unknown default case of a resilient enum. + +### Clear language mapping rules + +C++ is a very expressive language and it can provide representation for a lot of Swift language constructs. Not every Swift language construct will map to its direct C++ counterpart. For instance, Swift initializers might get bridged to static `init` member functions instead of constructors in C++, to allow C++ code to call failable initializers in a way that’s consistent with other initializer calls. Therefore, it’s important to provide documentation that describes how Swift language constructs get mapped to C++. It is a goal of C++ interoperability to provide a clear and well-defined mapping for how Swift language constructs are mapped to C++. Additionally, it is also a goal to clearly document which language constructs are not bridged to C++. In addition to documentation, compiler diagnostics should inform the user about types or functions that can’t be exposed to C++, when the user wants to expose them explicitly. It is a goal of C++ interoperability to add a set of clear diagnostics that let the user know when a certain Swift declaration is not exposed. It is not a goal to diagnose such cases when the user did not instruct the compiler to expose a declaration explicitly. For example, the Swift compiler might not diagnose when an exposed Swift type does not expose one of its public methods to C++ due to its return type not being exposed, if such method does not have an explicit annotation that instructs the compiler that this method must be exposed to C++. + +Some Swift APIs patterns will map to distinct C++ language constructs or patterns. For instance, an empty Swift enum with static members is commonly used in a namespace-like manner in Swift. This kind of enum can be mapped to a C++ namespace. It is a goal of C++ interoperability to provide a clear mapping for how Swift API patterns like this one are bridged to C++. + +The semantics of how Swift types behave should be preserved when they’re mapped to C++. For instance, in C++, there should still be a semantic difference between Swift value and reference types. Additionally, Swift’s copy-on-write data types like `Array` should still obey the copy-on-write semantics in C++. + +### Swift language evolution and API design should be unaffected + +Importing Swift APIs into C++ should not degrade the experience of programming in Swift. That means that Swift as a language should continue to evolve based on its [current goals](https://www.swift.org/about/). New additions to Swift should not restrict their expressivity or safety in order to provide a better experience for Swift APIs in C++. Such new features do not have to be exposed to C++ when it doesn't make sense to do so. + +Swift API authors should not change the way they write Swift code and design Swift APIs based on how specific Swift language constructs are exposed to C++. They should use the most effective Swift constructs for the task. It is a key goal of C++ interoperability that the exposed C++ interfaces are safe, performant, and ergonomic enough that programmers will not be tempted to make their Swift code worse just to make the C++ interfaces better. + +### Objective-C support + +The existing Swift to Objective-C bridging layer should still be supported even when C++ bindings are generated in the generated header. Furthermore, the generated C++ bindings should use appropriate Objective-C++ types or constructs in the generated C++ declarations where it makes sense to do so. For instance, a Swift function that returns an Objective-C class instance should be usable from an Objective-C++ compilation unit. + +## The approach + +The Swift compiler exposes Swift APIs to C++ by generating a header file that contains C++ declarations that wrap around the underlying calls into Swift functions that use the native Swift calling convention. The header also provides a suitable representation for Swift types. This generation can be done retroactively, starting from a Swift module interface file, and does not need to be requested when the Swift module is built from source. + +Currently the generated header file depends on several LLVM and Clang compiler features for the Swift calling convention support, and thus it can only be compiled by Clang. The header does not depend on any other Clang-specific C++ language extensions. The header can use some other optional Clang-only features that improve developer experience for the C++ users, like specific attributes that improve diagnostics. These Clang-only features are enabled only when the Clang that is being used supports them, so older versions of Clang would still be able to consume the generated header. + +The generated header file uses advanced C++ features that require a recent C++ language standard. C++20 is the recommended language standard to use for Swift APIs, however, the generated header should also be compatible with C++17 and C++14. + +The generated header file also contains the C and the Objective-C declarations that Swift exposes to C and Objective-C on platforms that support C or Objective-C interoperability. Thus a single header file can be used by codebases that mix C, Objective-C and C++. + +For the majority of Swift libraries, the generated header file is a build artifact generated by the client. This is different from Objective-C interoperability, where the generated header is always generated by the library owner and distributed with it. However, Swift library owners can also generate the header and distribute it with the library if they want the client to consume the C++ interface directly without having to run the Swift compiler on their end. + +On platforms with ABI stability, the generated C++ code for an ABI stable Swift interface is not tied to the Swift compiler that compiled the Swift code for that Swift module, as ABI stability is respected by the C++ code in the header. In all other cases the generated C++ code in the header is assumed to be tied to the Swift compiler that compiled the Swift module, and thus the header should be regenerated when the compiler changes. + + The next few sections provide a high level overview of how Swift types and some other language constructs get bridged to C++. The exact details of how Swift language constructs are bridged will be covered by Swift evolution proposals, and additional documentation, such as this preliminary [user guide document](https://github.com/apple/swift/blob/main/docs/CppInteroperability/UserGuide-CallingSwiftFromC%2B%2B.md). + +### Bridging Swift types + +The generated header contains C++ class types that represent the Swift struct, enum, and class types that are exposed by the Swift module. These types provide access to methods, properties and other members using either idiomatic C++ constructs or, in some cases, non-idiomatic C++ constructs that allow C++ to access more functionality in a consistent manner. The basic operations on these types follow the corresponding Swift semantics. For instance, a C++ value for a Swift class type is essentially a shared pointer that owns a reference to a Swift object, whereas a C++ value for a Swift struct type stores a Swift value of that type, and copying or destroying the C++ value copies or destroys the underlying Swift value. The C++ types also support C++ move semantics and translate them as appropriate to Swift's [`consume` operation](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0366-move-function.md). This enables high-performance use of Swift types from C++ and allows Swift's [non-copyable types](https://forums.swift.org/t/pitch-noncopyable-or-move-only-structs-and-enums/61903) to be exposed to C++. + +Protocol types also get exposed to C++. They provide access to their protocol interface to C++. The generated header also provides facilities to combine protocol types into a protocol composition type. The protocol composition type provides access to the combined protocol interface in C++. + +### Bridging generics + +Swift generic functions and types get bridged to C++ as C++ function and class templates. A generated C++ template instantiates a type-checked generic interface that uses the underlying polymorphic semantics that generics require when Swift APIs are called from the generated header. Type-checking is performed using the `requires` clause introduced in C++20. When C++17 and earlier is used, type-checking is performed using other legacy methods, like `enable_if` and `static_assert`. The two type-checking methods are compatible with the delayed template parsing compiler feature that Clang uses when building for Windows. + +To help achieve the performance goals outlined in the prior section, the generated class templates specialize the storage for the underlying Swift generic type when the Swift API that is exposed to C++ contains such a bounded Swift generic type. This ensures that non-resilient bounded generic values can be stored inline in a C++ type that represents the underlying Swift type, instead of being boxed on the heap. + +### Standard library support + +The Swift standard library contains a lot of useful functions and types that get bridged to C++. The generated standard library bindings enhance various Swift types like `Optional` and `Array` to provide idiomatic C++ APIs that allow the user to use such types in an idiomatic manner from C++. + +### Using Swift types in C++ templates + +The generated header is useful in mixed-language projects as C++ sources can include it, allowing C++ to call Swift APIs. However the C++ interoperability project also provides support for calling C++ APIs from Swift. In certain cases such C++ APIs contain function or class templates that need to be instantiated in Swift. Swift gives the user the ability to use Swift types in such cases, so the C++ templates have to be instantiated with Swift types. This means that the Swift types need to be translated into their C++ counterparts, which could then be used inside the instantiated C++ template specialization. This Swift-to-C++ type translation is performed using the same mechanism that’s used by the Swift compiler to generate the header with C++ interface for a Swift module. This means that a C++ template instantiated with a Swift type will see the same C++ representation of that Swift type regardless of whether it was instantiated from C++, or from Swift. + +## Evolution process + +The approach and the goals for how Swift APIs get bridged to C++ are outlined above. Each distinct Swift language construct that’s bridged to C++ will need to be covered by a detailed evolution proposal. These evolution proposals can refer to this vision document as a general context document for how Swift APIs should be bridged to C++. Every Swift API pattern that has a distinct mapping in C++ will also need a detailed and self-contained evolution proposal as well. The design for how each district language feature or API pattern is bridged to C++ is ratified only once its respective proposal goes through the Swift evolution process and is accepted by the Swift community. + +## The Swift ecosystem + +As a supported language feature, C++ and Swift interoperability must work well on every platform supported by Swift. In similar vein, tools in the Swift ecosystem should be updated to support C++ interoperability. The next few sections of this document provide specific recommendations for how various tools in the Swift ecosystem should be adapted to support C++ interoperability. The C++ interoperability workgroup intends to work on supporting most of these recommendations in the tools that are bundled with the Swift toolchain while working on the initial version of interoperability that could be adopted for production use cases. + +### Build tool support + +The Swift package manager (SwiftPM) is one of the most commonly used ways to build Swift code. Swift package manager can also build C and C++ code. SwiftPM should provide good support for bridging Swift APIs to C++ out of the box. It should generate a compatibility header for a Swift package when it determines that the C++ code in the same or dependent package includes such a header, and it should ensure that this header can be found when the C++ code is compiled. SwiftPM already has some support for generating a header file with Objective-C interface for a Swift module, and the C++ interoperability workgroup intends to reuse that support for supporting the generation of C++ bindings as well. + +CMake is a widely used tool used for configuring and building C++ code. CMake should provide good support for adding Swift code to C++ CMake targets. Swift’s ecosystem as a whole should ensure that it should be as straightforward as possible to add support for bridging Swift APIs to C++ within the same CMake project. The C++ interoperability workgroup should provide an example CMake project that shows how Swift and C++ can interoperate between each other, once the appropriate support for mixed-language Swift and C++ targets lands in CMake. + +### Debugging support + +Debugging support is critical for great user experience. LLDB should understand that C++ types in the generated header are just wrappers around Swift values. It should be able to display the underlying Swift value in the debugger when the user tries to inspect the C++ value that stores the Swift value in the debugger. In addition to that, the generated compatibility header should be correctly annotated to ensure that C++ inline thunks that call Swift APIs can be skipped when a user steps in or steps out into or from a call when debugging a program. + +### IDE support + +SourceKit-LSP is a language server that provides cross-platform IDE support for Swift code in the Swift ecosystem. It can also act as a language server for C, Objective-C and C++ as it can wrap around and redirect queries to clangd. This in turn allows SourceKit-LSP to provide support for mixed-language IDE queries, like jump-to-definition, that allows the IDE client to jump from an Objective-C method to call to its underlying Swift implementation. The C++ interoperability workgroup intends to reuse the current support for mixed-language Swift and Objective-C queries to add similar functionality for mixed-language Swift and C++ queries. This would allow IDE clients that use SourceKit-LSP to use features that can operate across the Swift/C++ language boundary. For instance, a client IDE would be able to support jump-to-definition from a Swift call expression to the called C++ function using SourceKit-LSP. This mixed-language query support in SourceKit-LSP is powered by the indexing data emitted by Clang. The C++ interoperability workgroup intends to extend Clang's indexing support to represent references to the wrapper C++ declarations from the generated header as references to the underlying Swift declarations. diff --git a/visions/webassembly.md b/visions/webassembly.md new file mode 100644 index 0000000000..95aa29f884 --- /dev/null +++ b/visions/webassembly.md @@ -0,0 +1,196 @@ +# A Vision for WebAssembly Support in Swift + +## Introduction + +WebAssembly (abbreviated [Wasm](https://webassembly.github.io/spec/core/intro/introduction.html#wasm)) is a virtual +machine instruction set focused on portability, security, and high performance. It is vendor-neutral, designed and +developed by [W3C](https://w3.org). An implementation of a WebAssembly virtual machine is usually called a +*WebAssembly runtime*. + +One prominent spec-compliant implementation of a Wasm runtime in Swift is [WasmKit](https://github.com/swiftwasm/WasmKit). +It is available as a Swift package, supports multiple host platforms, and has a simple API for interaction with guest +Wasm modules. + +An application compiled to a Wasm module can run on any platform that has a Wasm runtime available. Despite its origins +in the browser, it is a general-purpose technology that has use cases in client-side and server-side applications and +services. WebAssembly support in Swift makes the language more appealing in those settings, and also brings it to the +browser where it previously wasn't available at all[^1]. It facilitates a broader adoption of Swift in more environments +and contexts. + +The WebAssembly instruction set has useful properties from a security perspective, as it has no interrupts or +peripheral access instructions. Access to the underlying system is always done by calling explicitly imported +functions, implementations for which are provided by an imported WebAssembly module or a WebAssembly runtime itself. +The runtime has full control over interactions of the virtual machine with the outside world. + +WebAssembly code and data live in completely separate address spaces, with all executable code in a given module loaded +and validated by the runtime upfront. Combined with the lack of "jump to address" and a limited set of control flow +instructions that require explicit labels in the same function body, this makes a certain class of attacks impossible to +execute in a correctly implemented spec-compliant WebAssembly runtime. + +### WebAssembly System Interface and the Component Model + +The WebAssembly virtual machine has no in-built support for I/O; instead, a Wasm module's access to I/O is dependent +entirely upon the runtime that executes it. + +A standardized set of APIs implemented by a Wasm runtime for interaction with the host operating system is called +[WebAssembly System Interface (WASI)](https://wasi.dev). [WASI libc](https://github.com/WebAssembly/wasi-libc) is a +layer on top of WASI that Swift apps compiled to Wasm can already use thanks to C interop. The current implementation +of Swift stdlib and runtime for `wasm32-unknown-wasi` triple is based on this C library. It is important for WASI +support in Swift to be as complete as possible to ensure portability of Swift code in the broader Wasm ecosystem. + +In the last few years, the W3C WebAssembly Working Group considered multiple proposals for improving the WebAssembly +[type system](https://github.com/webassembly/interface-types) and +[module linking](https://github.com/webassembly/module-linking). These were later subsumed into a combined +[Component Model](https://component-model.bytecodealliance.org) proposal thanks to the ongoing work on +[WASI Preview 2](https://github.com/WebAssembly/WASI/blob/main/wasip2/README.md), which served as playground for +the new design. + +The Component Model defines these core concepts: + +- A *component* is a composable container for one or more WebAssembly modules that have a predefined interface; +- *WebAssembly Interface Types (WIT) language* allows defining contracts between components; +- *Canonical ABI* is an ABI for types defined by WIT and used by component interfaces in the Component Model. + +Preliminary support for WIT has been implemented in +[the `wit-tool` subcommand](https://github.com/swiftwasm/WasmKit/blob/0.0.3/Sources/WITTool/WITTool.swift) of the +WasmKit CLI. Users of this tool can generate `.wit` files from Swift declarations, and vice versa: Swift bindings from +`.wit` files. + +## Use Cases + +We can't anticipate every possible application Swift developers are going to create with Wasm, but we can provide a few +examples of its possible adoption in the Swift toolchain itself. To quote +[a GSoC 2024 idea](https://www.swift.org/gsoc2024/#building-swift-macros-with-webassembly): + +> WebAssembly could provide a way to build Swift macros into binaries that can be distributed and run anywhere, +> eliminating the need to rebuild them continually. + +This can be applicable not only to Swift macros, but also for the evaluation of SwiftPM manifests and plugins. + +In the context of Swift developer tools, arbitrary code execution during build time can be virtualized with Wasm. +While Swift macros, SwiftPM manifests, and plugins are sandboxed on Darwin platforms, with Wasm we can provide stronger +security guarantees on other platforms that have a compatible Wasm runtime available. + +The WebAssembly instruction set is designed with performance in mind. A WebAssembly module can be JIT-compiled or +compiled on a client machine to an optimized native binary ahead of time. With recently accepted proposals to the Wasm +specification it now supports features such as SIMD, atomics, multi-threading, and more. A WebAssembly runtime can +generate a restricted subset of native binary code that implements these features with little performance overhead. + +Adoption of Wasm in developer tools does not imply unavoidable performance overhead. With security guarantees that +virtualization brings, there's no longer a need to spawn a separate process for each Swift compiler and SwiftPM +plugin/manifest invocation. Virtualized Wasm binaries can run in the host process of a Wasm runtime, removing the +overhead of new process setup and IPC infrastructure. + +## Goals + +As of March 2024 all patches necessary for basic Wasm and WASI Preview 1 support have been merged to the Swift +toolchain and core libraries. Based on this, we propose a high-level roadmap for WebAssembly support and adoption in the Swift +ecosystem: + +1. Make it easier to evaluate and adopt Wasm with increased API coverage for this platform in the Swift core libraries. + Main prerequisite for that is setting up CI jobs for those libraries that run tests for WASI and also Embedded Wasm, where + possible. As a virtualized embeddable platform, not all system APIs are always available or easy to port to WASI. For example, + multi-threading, file system access, networking and localization need special support in Wasm runtimes and a certain amount of + consideration from a developer adopting these APIs. + +2. Improve support for cross-compilation in Swift and SwiftPM. We can simplify versioning, installation, and overall + management of Swift SDKs for cross-compilation in general, which is beneficial not only for WebAssembly, but for all + platforms. + +3. Continue work on Wasm Component Model support in Swift as the Component Model proposal is stabilized. Ensure + that future versions of WASI are available to Swift developers targeting Wasm. + +4. Make interoperability with Wasm components as smooth as C and C++ interop already is for Swift. With a formal + specification for Canonical ABI progressing, this will become more achievable with time. This includes consuming + components from, and building components with Swift. + +5. Improve debugging experience of Swift code compiled to Wasm. While rudimentary support for debugging + exists in some Wasm runtimes, we aim to improve it and, where possible, make it as good as debugging Swift code + compiled to other platforms. + +### Proposed Language Features + +In our work on Wasm support in Swift, we experimented with a few function attributes that could be considered +as pitches and eventually Swift Evolution proposals, if the community is interested in their wider adoption. +These attributes allow easier interoperation between Swift code and other Wasm modules linked with it by a Wasm +runtime. + +## Platform-specific Considerations + +### Debugging + +Debugging Wasm modules is challenging because Wasm does not expose ways to introspect and control the execution of +a Wasm module instance, so a debugger cannot be built on top of Wasm itself. Special support from the Wasm execution +engine is necessary for debugging. + +The current state of debugging tools in the Wasm ecosystem is not as mature as other platforms, but there are two +main directions: + +1. [LLDB debugger with Wasm runtime](https://github.com/llvm/llvm-project/pull/77949) supporting GDB Remote Serial Protocol; +2. [Wasm runtime with a built-in debugger](https://book.swiftwasm.org/getting-started/debugging.html#enhanced-dwarf-extension-for-swift). + +The first approach provides an almost equivalent experience to existing debugging workflows on other platforms. It +can utilize LLDB's Swift support, remote metadata inspection, and serialized Swift module information. However, since +Wasm is a Harvard architecture and has no way to allocate executable memory space at runtime, implementing expression +evaluation with JIT in user space is challenging. In other words, GDB stub in Wasm engines need tricky implementations +or need to extend the GDB Remote Serial Protocol. + +The second approach embeds the debugger within the Wasm engine. In scenarios where the Wasm engine is embedded as a +guest in another host engine (e.g. within a Web Browser), this approach allows seamless debugging experiences with the +host language by integrating with the host debugger. For example, in cases where JavaScript and Wasm call frames +are interleaved, the debugger works well in both contexts without switching tools. Debugging tools like Chrome DevTools +can use DWARF information embedded in Wasm file to provide debugging support. However, supporting Swift-specific +metadata information and JIT-based expression evaluation will require integrating LLDB's Swift plugin with these +debuggers in some way. + +In summary, debugging in the browser and outside of the browser context are sufficiently different activities to +require separate implementation approaches. + +### Multi-threading and Concurrency + +WebAssembly has [atomic operations in the instruction set](https://github.com/WebAssembly/threads) (only sequential +consistency is supported), but it does not have a built-in way to create threads. Instead, it relies on the host +environment to provide multi-threading support. This means that multi-threading in Wasm is dependent on the Wasm runtime +that executes a module. There are two proposals to standardize ways to create threads in Wasm: + +(1) [wasi-threads](https://github.com/WebAssembly/wasi-threads), which is already supported by some toolchains, +runtimes, and libraries but has been superseded; + +(2) The new [shared-everything-threads](https://github.com/WebAssembly/shared-everything-threads) proposal is still +in the early stages, but is expected to be the future of multi-threading in Wasm. + +Swift currently supports two threading models in Wasm: single-threaded (`wasm32-unknown-wasi`) and multi-threaded +using wasi-threads (`wasm32-unknown-wasip1-threads`). Despite the latter supporting multi-threading, Swift Concurrency +defaults to a cooperative single-threaded executor due to the lack of wasi-threads support in libdispatch. Preparing +for the shared-everything-threads proposal is crucial to ensure that Swift Concurrency can adapt to future +multi-threading standards in Wasm. + +### 64-bit address space + +WebAssembly currently uses a 32-bit address space, but [64-bit address space](https://github.com/WebAssembly/memory64/) +proposal is already in the implementation phase. + +Swift supports 64-bit pointers on other platforms where available, however WebAssembly is the first platform where +relative reference from data to code is not allowed. Alternative solutions like image-base relative addressing or +"small code model" for fitting 64-bit pointer in 32-bit are unavailable, at least for now. This means that we need +cooperation from the WebAssembly toolchain side or different memory layout in Swift metadata to support 64-bit linear +memory support in WebAssembly. + +### Shared libraries + +There are two approaches to using shared libraries in the WebAssembly ecosystem: + +1. [Emscripten-style dynamic linking](https://emscripten.org/docs/compiling/Dynamic-Linking.html) +2. [Component Model-based "ahead-of-time" linking](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Linking.md) + +Emscripten-style dynamic linking is a traditional way to use shared libraries in WebAssembly, where the host +environment provides non-standard dynamic loading capabilities. + +The latter approach cannot fully replace the former, as it is unable to handle dynamic loading of shared libraries at +runtime, but it is more portable way to distribute programs linked with shared libraries, as it does not require the +host environment to provide any special capabilities except for Component Model support. + +Support for shared libraries in Swift means ensuring that Swift programs can be compiled in +position-independent code mode and linked with shared libraries by following the corresponding dynamic linking ABI. + +[^1]: We aim to address browser-specific use cases in a separate future document.