Activation
Complex-valued activation functions must take into account the 2 degrees-of-freedom inherent to complex-valued data, typically represented as real and imaginary parts or magnitude and phase. Two common generalized classes of complex-valued activation functions operate on these respective representations and are defined as Type-A and Type-B functions.
Type-A activation functions consist of two real-valued functions, \(G_\mathbb{R}(\cdot)\) and \(G_\mathbb{I}(\cdot)\), which are applied to the real and imaginary parts of the input tensor, respectively, as
where \(\mathbf{z} = \mathbf{x} + j\mathbf{y}\).
Type-B activation functions consist of two real-valued functions, \(G_{||}(\cdot)\) and \(G_\angle(\cdot)\), which are applied to the magnitude (modulus) and phase (angle, argument) of the input tensor, respectively, as
In contrast, fully complex activation functions fit neither the Split Type-A or Split Type-B designation.
The final designation of complex-valued activation functions detailed in this work are extensions of the Rectified Linear Unit (ReLU) to the complex plane.