Polar Type-B Activation Functions

Type-B activation functions consist of two real-valued functions, \(G_{||}(\cdot)\) and \(G_\angle(\cdot)\), which are applied to the magnitude (modulus) and phase (angle, argument) of the input tensor, respectively, as

\[G(\mathbf{z}) = G_{||}(|\mathbf{z}|) * \exp(j G_\angle(\text{angle}(\mathbf{z})))\]

A generalized Type-B split activation function is defined in GeneralizedPolarActivation, which accepts two real-valued torch.nn.Module objects for \(G_{||}(\cdot)\) and \(G_\angle(\cdot)\), respectively.

class complextorch.nn.modules.activation.split_type_B.CVPolarLog

Complex-Valued Polar Squash Activation Function

Implements the operation

\[G(\mathbf{z}) = \ln(|\mathbf{z}| + 1) \odot \exp(j \angle\mathbf{z})\]

Based on work from the following paper:

D Hayakawa, T Masuko, H Fujimura. Applying complex-valued neural networks to acoustic modeling for speech recognition.

class complextorch.nn.modules.activation.split_type_B.CVPolarSquash

Complex-Valued Polar Squash Activation Function

Implements the operation:

\[G(\mathbf{z}) = \frac{|\mathbf{z}|^2}{(1 + |\mathbf{z}|^2)} \odot \exp(j \angle\mathbf{z})\]

Based on work from the following paper:

D Hayakawa, T Masuko, H Fujimura. Applying complex-valued neural networks to acoustic modeling for speech recognition.

class complextorch.nn.modules.activation.split_type_B.CVPolarTanh

Complex-Valued Polar Tanh Activation Function

Implements the operation:

\[G(\mathbf{z}) = \tanh(|z|) \odot \exp(j \angle\mathbf{z})\]

Note: phase information is unchanged

Based on work from the following paper:

A Hirose, S Yoshida. Generalization characteristics of complex-valued feedforward neural networks in relation to signal coherence

class complextorch.nn.modules.activation.split_type_B.GeneralizedPolarActivation(activation_mag: Module, activation_phase: Module)

Generalized Split Type-B Polar Activation Function

Operates on the magnitude and phase separately. Often \(G_\angle(\angle\mathbf{z})\) is the identity, in which case activation_phase should be set to None.

Implements the operation:

\[G(\mathbf{z}) = G_{||}(|\mathbf{z}|) \odot \exp(j G_\angle(\angle\mathbf{z}))\]

Type-B activation function is defined in the following paper:

J. A. Barrachina, C. Ren, G. Vieillard, C. Morisseau, and J.-P. Ovarlez. Theory and Implementation of Complex-Valued Neural Networks.

forward(input: CVTensor) → CVTensor

Computes the generalized Type-B split activation function.

Parameters:

input (CVTensor) – input tensor

Returns:

\(G_{||}(|\mathbf{z}|) \odot \exp(j G_\angle(\angle\mathbf{z}))\)

Return type:

CVTensor

class complextorch.nn.modules.activation.split_type_B.modReLU(bias: float = 0.0)

modulus Rectified Linear Unit

Implements the operation:

\[G(\mathbf{z}) = \texttt{ReLU}(|\mathbf{z}| + b) \odot \frac{\mathbf{z}}{|\mathbf{z}|} = \texttt{ReLU}(|\mathbf{z}| + b) \odot \exp(j \angle\mathbf{z}).\]

Notice that \(|\mathbf{z}|\) (\(\mathbf{z}\).abs()) is always positive, so if \(b > 0\) then \(|\mathbf{z}| + b > = 0\) always. In order to have any non-linearity effect, \(b\) must be smaller than \(0\) (\(b < 0\)).

Based on work from the following papers:

Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks.

J. A. Barrachina, C. Ren, G. Vieillard, C. Morisseau, and J.-P. Ovarlez. Theory and Implementation of Complex-Valued Neural Networks.