Skip to content

Commit 8cab81d

Browse files
FranzBuschneonichu
FranzBusch
authored and
neonichu
committed
[SE-0272] Review feedback: Package Manager Binary Dependencies (swiftlang#1099)
* Work in review feedback * Update resolution behaviour * Small edits to the proposal - removed note about "Multiple references to same artifact" because that case already shouldn't be possible in practice - updated "name" note for artifacts to talk about module name - removed leftover mention of opt-out for binaries - replaced `--artifact-url` with `--original-url` option which is supposed to work for both package as well as artifact URLs
1 parent 2b03216 commit 8cab81d

File tree

1 file changed

+99
-58
lines changed

1 file changed

+99
-58
lines changed

‎proposals/0272-swiftpm-binary-dependencies.md‎

Lines changed: 99 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
* Proposal: [SE-0272](0272-swiftpm-binary-dependencies.md)
44
* Authors: [Braden Scothern](https://github.com/bscothern), [Daniel Dunbar](https://github.com/ddunbar), [Franz Busch](https://github.com/FranzBusch)
55
* Review Manager: [Boris Bügling](https://github.com/neonichu)
6-
* Status: **Returned for revision**
6+
* Status: **Active review (12 13...12 20)**
77
* Decision Notes: [Rationale](https://forums.swift.org/t/returned-for-revision-se-0272-package-manager-binary-dependencies/30994)
88

99
## Contents
@@ -95,50 +95,31 @@ let package = Package(
9595
targets: [
9696
.binaryTarget(
9797
name: "SomePackageLib",
98-
artifacts: [
99-
.artifact(
100-
source: .url("https://github.com/some/package/releases/download/1.0.0/SomePackage-1.0.0.zip",
101-
checksum: "839F9F30DC13C30795666DD8F6FB77DD0E097B83D06954073E34FE5154481F7A")),
102-
]
98+
url: "https://github.com/some/package/releases/download/1.0.0/SomePackage-1.0.0.zip",
99+
checksum: "839F9F30DC13C30795666DD8F6FB77DD0E097B83D06954073E34FE5154481F7A"
100+
),
101+
.binaryTarget(
102+
name: "SomeLibOnDisk",
103+
path: "artifacts/SomeLibOnDisk.zip"
103104
)
104-
]
105-
)
106-
```
107-
108-
109-
Secondly we propose to add a new configuration point to the package description that allows packages to opt-out of binary dependencies. This will enforce that nothing in its transitive dependencies brings a binary dependency with it.
110-
111-
```swift
112-
let package = Package(
113-
name: "SomeOtherPackage",
114-
disallowsBinaryDependencies: true,
115-
products: [
116-
.library(name: "SomeOtherPackage", targets: ["SomeOtherPackageLib"])
117-
],
118-
targets: [
119-
.target(name: "SomeOtherPackageLib")
120-
]
105+
]
121106
)
122107
```
123108

124109
Packages are allowed to contain a mix of binary and source targets. This is
125110
useful when, for example, providing a pre-built or closed source C library
126111
alongside an open source set of Swift bindings for the library.
127112

128-
When a package is built that depends upon any product with a binary target, the
129-
package manager will search the `artifacts` declaration list to find an artifact
130-
which matches the current build conditions. This list
131-
will be searched in order, and the first matching artifact will be used. It is
132-
the job of the package author/publisher to provide an appropriate set of
133-
artifacts for the use cases the package wishes to support. This use case will be limited to Apple platforms in the beginning. In the future, we can add support for other platforms. A potential approach is outlined in the future directions section.
113+
The use case will be limited to Apple platforms in the beginning. In the future, we can add support for other platforms. A potential approach is outlined in the future directions section.
134114

135115
## Detailed design
136116

137117
The design consists of the following key points:
138118
* New `PackageDescription` API for defining a binary target.
139-
* New parameter on a package declaration level to opt-out of binary dependencies.
140119
* New requirements for the `Package.resolved` file when using binary packages.
120+
* A new command to compute a checksum for a file.
141121
* A new mechanism for downloading binary target artifacts.
122+
* Support for artifact mirroring.
142123

143124
Terminology:
144125

@@ -166,60 +147,86 @@ while keeping the following as non-goals:
166147
## New `PackageDescription` API
167148

168149
### BinaryTarget
169-
Since a binary target is different compared to a source only target, we propose to introduce a new struct `Artifact`. This struct defines a target's associated artifacts.
170-
171-
We propose to support local and remote artifacts from the beginning. In the alternatives considered section is larger collection of potential artifact stores. However we opted to simplify the initial implementation by just supporting a url and a path based definition. Later, we can implement different types of providers with different authentication methods.
150+
Since a binary target is different compared to a source only target, we propose to introduce two new static method on `Target` to declare a binary target. We propose to support local and remote artifacts from the beginning. In the alternatives considered section is a larger collection of potential artifact stores. However we opted to simplify the initial implementation by just supporting a url and a path based definition. Later, we can implement different types of providers with different authentication methods.
172151

173152
```swift
174-
public struct Artifact {
175-
public enum Source {
176-
case url(String, checksum: String)
177-
case path
178-
}
179-
180-
public let source: Source
153+
extension Target {
154+
/// Declare a binary target with the given url.
155+
public static func binaryTarget(
156+
name: String,
157+
url: String,
158+
checksum: String
159+
) -> Target
160+
161+
/// Declare a binary target with the given path on disk.
162+
public static func binaryTarget(
163+
name: String,
164+
path: String
165+
) -> Target
181166
}
182167
```
183168

184-
Furthermore, we propose to add a new `artifacts: [Artifacts]?` property to the `Target`, as well as extend the initializer with this parameter and create a new static method called `.binaryTarget()`. Lastly, we propose to extend the `TargetType` enum with a new case called `binary`.
185-
186-
### PackageDescription
187-
To opt out of binary packages we propose a new configuration point inside the package description.
188-
189-
```swift
190-
public final class Package {
191-
...
192-
/// This disallows any binary dependency or any transitive binary dependency.
193-
public var disallowsBinaryDependencies: Bool
194-
...
195-
}
196-
```
169+
## Checksum computation
170+
We propose to add a new command to SwiftPM `swift package compute-checksum <file>` which is going to be used to compute the checksum of individual files. This implementation can then evolve in the future and is tied to the tools version of the package to avoid breaking compatibility with older tools.
197171

198172
## New `Package.resolved` Behavior
199173

200-
For binary targets we store the checksum of the artifact in the `Package.resolved`. This lets us check for errors during resolution where a package's version did not change but the checksum did. In this case we will throw an error alerting the user about this.
174+
For binary targets we will validate the commit hashes from the resolved file for any dependencies from now on to ensure the checksums of binaries cannot be changed for a specific version. This lets us check for errors during resolution where a package's version did not change but the checksum did. In this case we will throw an error alerting the user about this.
201175

202176
### Resolution
203177

204178
Package resolution and dependency expression will not be impacted by this change (except where explicitly noted).
205179

206-
#### Multiple references to same artifact
207-
During resolution SwiftPM will check that all references to an artifact in a dependency graph have the same checksum.
180+
#### Exported product with binary dependency that specifies a type
181+
SwiftPM will emit an error during resolution when a product that directly exports a binary dependency declares a type, e.g.: `.product(name: "MyBinaryLib", type: .static, targets: ["MyBinaryLib"])`.
182+
183+
#### Resolution on non-Apple platforms
184+
When resolving a package that contains a binary dependency on non-Apple platforms, SwiftPM will throw an error and explicitly state that this dependency is not valid for the current platform. During the review it was brought up that we could ignore these dependencies but that would make the behavior of SwiftPM very unexpected. In the future, when properly supporting other platforms this can be solved easily with a proper condition mechanism.
208185

209186
## Binary Target Artifact Format
210187

211188
SwiftPM currently supports multiple platforms; however, this proposal only adds support for binary targets on Apple platforms. The reason for this is that Apple platforms provide ABI guarantees and an already existing format we can leverage to simplify the initial implementation. For Apple platforms we propose to use the `XCFramework` format for artifacts. This format already supports dynamic and static linking. Furthermore, it can contain products for every individual Apple platform at once.
212189

190+
SwiftPM expects url-based artifacts to be packaged inside a `.zip` file where the artifact is lying at the root of the archive. Furthermore, the artifact needs to have the same module name as the target name provided inside the manifest file.
191+
192+
For path-based artifact SwiftPM supports artifacts as a `.zip` and as a raw `XCFramework`.
193+
194+
During resolution SwiftPM won't do any verification of the format of the artifact. This is up to the vendor to provide correct and valid artifact. In the future, this can be extended and further validation, such as checking that the module name matches, can be implemented.
195+
213196
## Security
214197

215198
When adding new external dependencies, it is always important to consider the security implication that it will bring with it. Comparing the trust level of a source-based to a binary-based dependency the first thought is that the trust level of the source-based dependency is higher since on can inspect its source code. However, there is no difference between a binary and source dependency since source-based dependencies can have security issues as well. One should have better reasons to trust a dependency than source being inspectable.
216199

217-
There is still a significant difference between having a dependency with zero vs. any binary dependency. For example, the portability of a library with binary dependencies is far worse than the one with only source-based dependencies. For this reason, we propose to add an additional configuration point in the manifest that allows package authors to opt-out of binary dependencies.
200+
There is still a significant difference between having a dependency with zero vs. any binary dependency. For example, the portability of a library with binary dependencies is far worse than the one with only source-based dependencies.
218201

219202
However, there are still some security related aspects when it comes to binary artifacts that we should mitigate. For example, when declaring a `binaryTarget` the hash of the artifact is required similar to Homebrew. By doing this an attacker needs to compromise both the server which provides the artifact as well as the git repository which provides the package manifest. A secondary reason is that the server providing the binary might be out of the package author's control and this way we can ensure that the expected binary is used.
220203

221204
Lastly, the hash of the binary is stored in the package resolved to avoid that the vendor changes the artifact behind a version without anyone noticing.
222205

206+
## Mirroring support
207+
Binary artifacts can also be mirrored. We propose to deprecate the existing `--package-url` option and to replace it with a `--original-url` option which will work for both package URLs as well as artifact URLs:
208+
209+
```
210+
$ swift package config set-mirror \
211+
--original-url <original URL> \
212+
--mirror-url <mirror URL>
213+
214+
# Example:
215+
216+
$ swift package config set-mirror \
217+
--original-url https://github.com/Core/core/releases/download/1.0.0/core.zip \
218+
--mirror-url https://mygithub.com/myOrg/core/releases/download/1.0.0/core.zip
219+
```
220+
221+
Additionally, we propose to add a command to unset a mirror URL for an artifact:
222+
223+
```
224+
$ swift package config unset-mirror \
225+
--original-url https://github.com/Core/core/releases/download/1.0.0/core.zip
226+
```
227+
228+
The other unset command options `--mirror-url` and `--all` will be working the same for artifacts as they do for packages.
229+
223230
## Impact on existing packages
224231

225232
No current package should be affected by this change since this is only an additive change in enabling SwiftPM to use binary dependencies.
@@ -229,9 +236,18 @@ No current package should be affected by this change since this is only an addit
229236
### Support for non-Apple platforms
230237
Non-Apple platforms provide non-trivial challenges since they are not always giving guarantees of the ABI of the platform. Additionally, further conditions such as the corelibs-foundation ABI or if the hardware supports floating points need to be taken into consideration when declaring a package for non-Apple platforms. Various other communities tried to solve this, e.g. Python's [manylinux](https://www.python.org/dev/peps/pep-0600/).
231238

232-
In the future, we could add an `ArtifactCondition ` to SwiftPM which provides the possibility to declare under which conditions a certain artifact can be used. Below is a potential `ArtifactCondition` struct which does **not** include a complete set of conditions that need to be taken into consideration.
239+
In the future, we could add an `Artifact` struct and `ArtifactCondition`s to SwiftPM which provides the possibility to declare under which conditions a certain artifact can be used. Below is a potential `Artifact` and `ArtifactCondition` struct which does **not** include a complete set of conditions that need to be taken into consideration.
233240

234241
```swift
242+
public struct Artifact {
243+
public enum Source {
244+
case url(String, checksum: String)
245+
case path
246+
}
247+
248+
public let source: Source
249+
}
250+
235251
public struct ArtifactCondition: Encodable {
236252
public struct LLVMTriplet: Encodable {
237253
// Should be only the subset that Swift supports
@@ -414,6 +430,31 @@ During the discussion of this proposal another solution to the `allowsBinary` fl
414430
### Opt-out configuration in separate file
415431
During the discussion of this proposal it was decided that an opt-out mechanism was good to give package users and vendors an escape hatch. However, it was discussed whether this configuration should live inside the manifest or a separate configuration file. In this proposal, we opted to keep the configuration inside the manifest file.
416432

433+
### Opt-out in package manifest
434+
In the first round, we proposed to add a configuration flag in the manifest to opt-out of binary dependencies; however, during the review it became apparent that this flag doesn't provide as much value and can make some dependencies actually more restricted when they add this flag. Therefor, we opted to not include such a configuration flag and let workflow tooling provide this functionality if needed.
435+
436+
```swift
437+
public final class Package {
438+
...
439+
/// This disallows any binary dependency or any transitive binary dependency.
440+
public var disallowsBinaryDependencies: Bool
441+
...
442+
}
443+
```
444+
445+
```swift
446+
let package = Package(
447+
name: "SomeOtherPackage",
448+
disallowsBinaryDependencies: true,
449+
products: [
450+
.library(name: "SomeOtherPackage", targets: ["SomeOtherPackageLib"])
451+
],
452+
targets: [
453+
.target(name: "SomeOtherPackageLib")
454+
]
455+
)
456+
```
457+
417458
### Support for various artifact stores
418459
Initially, we considered the various artifact stores on the market and how we can integrate with them. We decided to support a URL based artifact definition for the first implementation since the various providers require each their own method of authentication. However, we wanted to keep the possibility for future additions of providers open; therefore, we made the source of an artifact an enum which can be extended.
419460

0 commit comments

Comments
 (0)