Add Complex Numbers

[bartchr808]

This PR adds a prototype version of complex numbers.

NOTE: this will not be a part of the library, as this is an experimental project that the @tensorflow/swift-team is using. However, feel free to fork the project and play around with it! 😄

Status of Complex Numbers in Swift

At the time of this PR, there is discussion about adding complex numbers to the Swift standard library, but it will be a post 5.1 feature. You can read more about it in this forum post.

As such, the @tensorflow/swift-team decided to add a basic, prototype version of complex numbers to the library. This implementation uses a modified implementation from the xwu/NumericAnnex Swift numeric library repo.

Why We Added it: Differentiation

As part of further exploring the capabilities of the automatic differentiation system the team and the rest of the community has helped create so far, we wanted to see how versatile it is. Currently, we have no problem of differentiating anything in the real number space (like real vectors/tensors/matrices/etc.).

Complex numbers are different in that basic operators behave differently than in the reals. And in a concrete example, the dot product of a complex vector is the following:

func dot(lhs: Vector<Complex>, rhs: Vector<Complex>) -> Complex {
    var result = Complex(real: 0, imaginary: 0)
    for i in lhs.length {
        resultVector = resultVector + (lhs[i].complexConjugate() * rhs[i])
    }
    return result
}

While in the real number space, conjugates are not used.

You can see an example here and more about the complex conjugate here. For an actual implementation example, you can see one of our test cases here. Note that at the time of this PR, control flow is not 100% complete, so the dot product is only for a fixed size vector at the moment.

What We Learned So Far

• Our system works well with differentiating complex numbers and complex vectors
• We need to update the VectorNumeric protocol to add a complexConjugate function that will be automatically synthesized for everything (returning self) other than when the Scalar type is Complex, where a default implementation will be written. We need to figure out where to add complexConjugate, and whether we need a VectorNumeric protocol. Currently being discussed in the Swift Forums.

[Shashi456]

@bartchr808 Please don't mind me asking, But is this something required within some part of deep learning or functionality we are adding to swift-apis? Just plainly curious :P.

[rxwei]

1. All source files in this repository should contain a copyright/license header. Could you please copy one into this file?
2. Since this is not an API that's to be shipped, everything should be internal instead of public. As for testing, so long as your test file import this module with @testable import DeepLearning, you will be able to access all the internal stuff.

[rxwei]

Could you please apply 4-space indentation?

[rxwei]

@bartchr808 Please don't mind me asking, But is this something required within some part of deep learning or functionality we are adding to swift-apis? Just plainly curious :P.

This is a purely experimental and internal feature we use to explore the automatic differentiation system.

[marcrasi]

Some licensing stuff: Google policy for importing code from third party libraries is that it must go inside a directory named third_party/<something>, and that directory must contain the LICENSE file from the original project. This is applicable even if the third-party code is very modified. (https://opensource.google.com/docs/releasing/preparing/#third-party-components)

Therefore, you should put this code under Sources/third_party/Complex, and include the LICENSE file from https://github.com/xwu/NumericAnnex/blob/master/LICENSE.

Also it would be nice to include a comment at the top of Complex.swift crediting the NumericAnnex project and explaining that this file includes lots of code from it.

You'll have to create a new SwiftPM target so that SwiftPM compiles files in this new location.

[rxwei]

Need a license header here.

[bartchr808]

Based on how the tests are failing (can't find Complex) I'm guessing I have no modify some file to add the path to Sources/third_party/Complex/?

[lattner]

I'm pretty sure this only need to run if 'recalculate' is false.

The code structure could be cleaned up (and recalculate eliminated?) if pulled out of line I think.

[bartchr808]

Similar to the comment above, I'll take a look at this logic more closely and see what I can pull out, what code I can simplify, and also explain with comments Sunday night.

As background, I kept this code the way it was just like in the NumericAnnex library. It had written a lot of tests, so I did a bit of a blind trust on the implementation of this logic. Additionally, since I was mainly looking at the automatic differentiation aspect of complex numbers, I didn't spend much time on the implementation of the basic operators.

However, since this will be a part of our repo, I see that it's good to make sure all our code is well documented and explained.

[lattner]

Please contact Xiaodi about this to see if he is willing to contribute it to the s4tf project. We don't want to mix licenses in the swift-apis repos.

[bartchr808]

Ya that sounds good I sent him an email about this! 😄

[lattner]

This is super exciting @bartchr808, great work!

[bartchr808]

Thanks @lattner ! 😄

[eaplatanios]

@bartchr808 This is really cool! It would also be nice if you could make Complex<T> conform to TensorDataType when T is Float32 or Float64, as that will allow creating complex-valued tensors.

[bartchr808]

Thanks @eaplatanios ! Since we are only using this as an experiment in the team while Steve Canon is adding it Swift, we will hold off on adding the ability to create complex tensors. However, once complex numbers are officially landed in a future version of Swift, we will definitely add the ability to create complex tensors! 😄

For context, we wanted to push this experiment to the swift-apis repo in case anyone wanted to copy the implementation and use it in their work/projects and to see what the team has been working on. However, we didn't want to expose this implementation to any S4TF users since this isn't a part of what the deep learning library part of the S4TF project is supposed to provide.

[marcrasi]

I think you should add an additional Additional Consideration linking to https://github.com/HIPS/autograd/blob/master/docs/tutorial.md#complex-numbers and explaining that we are using that convention.

It's a very non-obvious choice about how to do things. The first time I thought about complex differentiation, I thought that we should only define derivatives for holomorphic functions, but that convention lets us define derivatives for all functions and that link explains why that's a good thing to do.

[bartchr808]

Agreed I added the following:

/// Our implementation of complex number differentiation follows the same
/// convention as Autograd. In short, we can get the derivative of a
/// holomorphic function, functions whose codomain are the Reals, and
/// functions whose codomain and domain are the Reals. You can read more about
/// Autograd at
///
///   https://github.com/HIPS/autograd/blob/master/docs/tutorial.md#complex-numbers

[marcrasi]

Have you tried making real and imaginary differentiable? If you do that, then the compiler will in principle be able to automatically compute derivatives for all the complex operations because they just take real and imaginary parts, then perform differentiable operations on those, then construct a new complex number (which is also a differentiable operation)!

In practice, the compiler can't quite derive these yet because most of the complex operations are implemented in terms of mutation and control flow that the compiler can't handle. But soon it should be able to handle it!

It would be very interesting to add some tests that define a few operations in ways that autodiff can handle, and then see if autodiff computes the expected derivatives for those. e.g.

func simplifiedAdd(_ lhs: Complex, _ rhs: Complex) -> Complex {
  return Complex(lhs.real + rhs.real, lhs.imaginary + rhs.imaginary)
}

[rxwei]

I believe derived conformances already made stored properties differentiable.

[bartchr808]

Have you tried making real and imaginary differentiable?

I did in that when I mark a method as differentiable, I put the requirement that @differentiable(vjp: _vjpFunc where T: Differentiable). I didn't require real and imaginary to be differentiable where I define the Complex struct, because when we are to differentiate the standard library's version of complex numbers, we won't be able to change it, and additionally it will restrict Complex numbers to what they can be more than what the standard library will likely have.

It would be very interesting to add some tests that define a few operations in ways that autodiff can handle, and then see if autodiff computes the expected derivatives for those.

I'll give it a shot! 😄

[eaplatanios]

@bartchr808 Thanks for the explanation! That makes sense. I guess I was a bit confused as to why this PR was made in this repo but your explanation cleared things out.

[bartchr808]

Currently PR is blocked by the following bug which doesn't let me differentiate the initializer: https://bugs.swift.org/browse/TF-546

[bartchr808]

Thanks @dan-zheng for the fix in TF-546, everything should work now! 😄