Add `@_Freezable` property wrapper. #250

dan-zheng · 2019-06-17T23:06:38Z

@_Freezable wraps differentiable values and provides toggleable "trainability" via the isFrozen property and the freeze/unfreeze mutating methods.

When isFrozen is true, accesses to value have a derivative of zero.

Example:

struct FreezableDense: Layer {
    @_Freezable var weight: Tensor<Float>
    @_Freezable var bias: Tensor<Float>

    @differentiable
    func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {
        return input * weight + bias
    }
}

var dense = FreezableDense(weight: Tensor(2), bias: Tensor(3))
print(gradient(at: dense) { dense in dense(Tensor(4)) })
// TangentVector(_weight: 4.0, _bias: 1.0)

// Freeze derivatives for `dense.weight`!
dense.$weight.freeze()

print(gradient(at: dense) { dense in dense(Tensor(4)) })
// TangentVector(_weight: 0.0, _bias: 1.0)

The alternative name @_Trainable was considered, but @_Trainable gives the impression that "only properties marked with @_Trainable are trainable", which is not true: all properties are trainable (i.e. their derivatives are propagated) by default.

Sources/TensorFlow/Layer.swift

rxwei · 2019-06-18T03:09:08Z

Sources/TensorFlow/Layer.swift

+@_propertyWrapper
+public struct Freezable<Value: Differentiable> : Differentiable {
+    @noDerivative public var isFrozen: Bool = false
+    private var _value: Value


The fact that this is using another level of property delegation makes me think there can be another property wrapper that splits up this logic.

I believe the delegation here is necessary only because stored properties can't have custom jvp/vjps. Does it make sense to create a property wrapper to handle that behavior?

Definitely! A separate property wrapper should provide a general way to define a custom VJP for a property. I'll write that out in a sec.

The current property wrapper implementation in our branch is incomplete—it does not support the both-initialization-and-default-value pattern. I posted a question here: https://forums.swift.org/t/se-0258-property-wrappers-second-review/25843/76?u=rxwei.

rxwei · 2019-06-18T05:00:20Z

Sources/TensorFlow/Layer.swift

+@_propertyWrapper
+public struct Freezable<Value: Differentiable> : Differentiable {
+    @noDerivative public var isFrozen: Bool = false
+    private var _value: Value


The current property wrapper implementation in our branch is incomplete—it does not support the both-initialization-and-default-value pattern. I posted a question here: https://forums.swift.org/t/se-0258-property-wrappers-second-review/25843/76?u=rxwei.

rxwei · 2019-06-18T05:02:56Z

Tests/TensorFlowTests/FreezableTests.swift

+            XCTAssertEqual(1, gradient.0.$bias)
+            XCTAssertEqual(2, gradient.1)
+        }
+        dense.freezableWeight.isFrozen = true


Why is dense.$weight.isFrozen = true not working?

It doesn't work because underlying storage is currently always private:

/Users/danielzheng/swift-apis/Tests/TensorFlowTests/FreezableTests.swift:62:15: error: '$weight' is inaccessible due to 'private' protection level dense.$weight.isFrozen = true ^~~~~~~ /Users/danielzheng/swift-apis/Tests/TensorFlowTests/FreezableTests.swift:21:28: note: '$weight' declared here @Freezable var weight: Float ^

Indicated with a comment:

struct FreezableDense : Layer { @Freezable var weight: Float // Workaround for underlying storage private access. var freezableWeight: Freezable<Float> { get { $weight } set { $weight = newValue } } }

The ability to specify underlying storage access levels is important for accessing APIs defined on property wrapper types. @tanner0101 made a case for customizing underlying storage access levels in this post.

This indicates that the feature is not quite ready for our use case yet. We should review the 2nd iteration of the proposal and make sure things are going the right direction for us.

Are both underlying properties and functions private? For example, could we define a freeze()/unfreeze() function in the property wrapper?

The issue is that wrapper value is private, or internal at most.

According to @DougGregor's recent post, this may be getting changed:

Here's the set of things I want to revise:
...
When wrapperValue is present, the $foo property becomes available and has the same access level as the original property foo

If so, that should make property wrappers viable for your use case. 👍

Yeah, that'd be great!

eaplatanios · 2019-06-18T17:13:14Z

Property wrappers are defined as structs and so it makes sense that the @Freezable annotation is capitalized. However, it kind of looks inconsistent when all the other annotations are in camel-case. Could we maybe add an argument such as @_propertyWrapper(label: "freezable"), so that it can be referenced in camel-case as well for consistency?

rxwei · 2019-06-18T17:19:52Z

Property wrappers are defined as structs and so it makes sense that the @Freezable annotation is capitalized. However, it kind of looks inconsistent when all the other annotations are in camel-case. Could we maybe add an argument such as @_propertyWrapper(label: "freezable"), so that it can be referenced in camel-case as well for consistency?

This is because the property wrapper proposal is setting a new standard for attributes in Swift: All custom attributes start with a capital letter, and all compiler builtin attributes use camel case.

eaplatanios · 2019-06-20T14:32:25Z

Thanks @rxwei! I was not aware of the new standard.

saeta

LGTM, but please underscore for now until we get more experience.

`@_Freezable` wraps differentiable values and provides toggleable "trainability" via the `isFrozen` property and `freeze`/`unfreeze` mutating methods. When `isFrozen` is true, accesses to `value` have a derivative of zero. The alternative name `@_Trainable` was considered, but `@_Trainable` gives the impression that "only properties marked with `@_Trainable` are trainable", which is not true: all properties are trainable (i.e. their derivatives are propagated) by default.

dan-zheng · 2019-11-07T19:17:41Z

LGTM, but please underscore for now until we get more experience.

Done in 7c96679.

dan-zheng · 2019-11-20T07:32:47Z

Merging @_Freezable as an underscored API.
Added to agenda for the next Swift for TensorFlow design meeting.

rxwei reviewed Jun 17, 2019

View reviewed changes

Sources/TensorFlow/Layer.swift Outdated Show resolved Hide resolved

Sources/TensorFlow/Layer.swift Outdated Show resolved Hide resolved

Sources/TensorFlow/Layer.swift Outdated Show resolved Hide resolved

Sources/TensorFlow/Layer.swift Outdated Show resolved Hide resolved

dan-zheng force-pushed the trainable-property-wrapper branch from d0b108c to 28c5991 Compare June 18, 2019 03:04

dan-zheng changed the title ~~Add @Trainable property wrapper.~~ Add @Freezable property wrapper. Jun 18, 2019

rxwei reviewed Jun 18, 2019

View reviewed changes

dan-zheng marked this pull request as ready for review June 18, 2019 03:14

rxwei reviewed Jun 18, 2019

View reviewed changes

rxwei mentioned this pull request Aug 3, 2019

Parameter counting API for Layer #412

Open

dan-zheng force-pushed the trainable-property-wrapper branch 2 times, most recently from 141f449 to 41aee7d Compare November 7, 2019 18:43

saeta approved these changes Nov 7, 2019

View reviewed changes

dan-zheng force-pushed the trainable-property-wrapper branch from 41aee7d to 7c96679 Compare November 7, 2019 19:14

dan-zheng changed the title ~~Add @Freezable property wrapper.~~ Add @_Freezable property wrapper. Nov 7, 2019

marcrasi assigned dan-zheng Nov 14, 2019

dan-zheng merged commit fb747d0 into tensorflow:master Nov 20, 2019

dan-zheng deleted the trainable-property-wrapper branch November 20, 2019 07:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `@_Freezable` property wrapper. #250

Add `@_Freezable` property wrapper. #250

dan-zheng commented Jun 17, 2019 •

edited

Loading

rxwei Jun 18, 2019

dan-zheng Jun 18, 2019

rxwei Jun 18, 2019

rxwei Jun 18, 2019

rxwei Jun 18, 2019

rxwei Jun 18, 2019

dan-zheng Jun 18, 2019

rxwei Jun 18, 2019

eaplatanios Jun 20, 2019

rxwei Jun 20, 2019

tanner0101 Jun 20, 2019 •

edited

Loading

rxwei Jun 21, 2019

eaplatanios commented Jun 18, 2019

rxwei commented Jun 18, 2019

eaplatanios commented Jun 20, 2019

saeta left a comment

dan-zheng commented Nov 7, 2019

dan-zheng commented Nov 20, 2019

Add @_Freezable property wrapper. #250

Add @_Freezable property wrapper. #250

Conversation

dan-zheng commented Jun 17, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tanner0101 Jun 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eaplatanios commented Jun 18, 2019

rxwei commented Jun 18, 2019

eaplatanios commented Jun 20, 2019

saeta left a comment

Choose a reason for hiding this comment

dan-zheng commented Nov 7, 2019

dan-zheng commented Nov 20, 2019

Add `@_Freezable` property wrapper. #250

Add `@_Freezable` property wrapper. #250

dan-zheng commented Jun 17, 2019 •

edited

Loading

tanner0101 Jun 20, 2019 •

edited

Loading