GroupNorm

This node normalizes over groups of features so that the mean is close to zero and standard deviation is close to one. Similar to InstanceNorm, the mean and standard deviation is calculated for each dimension in the input. That is, given an input of shape "(C, X, Y)", the normalization occurs over for each feature in the dimension C. The difference here is that the dimension C is divided into <Number of groups> groups, where each group contains \(\cfrac{C}{\text{Number of groups}}\) channels, and a total mean and standard deviation is calculated for each group.

Additionally, learnable weight \(\gamma\) and bias \(\beta\) are applied after normalization. For an input of shape (C, X, Y), these parameters have size C, meaning they operate per feature of dimension C. The mathematical expression is:

\(y = \cfrac{x - E[x]}{\sqrt{V[x] + \epsilon}} * \gamma + \beta\)
Where \(E[x]\) is the mean, \(V[x]\) is the variance (standard deviation squared) and \(\epsilon\) is a small configurable constant.

Group normalization is described in the paper Group Normalization.

The only supported input data type is Float32, and the input must have at least 1 dimension. In order to determine the sizes of the weight and bias parameters, the first dimension size must be known. For example, the shape (?, 2, 2) can not be used. Also, in order to be able to split the channels into groups, the size of the first dimension must be divisible by <Number of groups>.

The output will have the same shape and data type as the input.

By clicking the node the following parameters can be configured on the right panel:

• Epsilon: A small constant added to the variance to avoid dividing by zero.
• Number of groups: Number of groups to divide the channels into.
• Number of connections: Increases the number of connections on this node. The first input will lead to the first output, the second input to the second output and so on. This is useful because even though the connections are separate they share the same learned parameters, i.e the learned mean and standard deviation.
• Weight initializer: Initial value for the weight. If unsure, use a value of 1.
• Bias initializer: Initial value for the bias. If unsure, use a value of 0.