xavier

xavier初始化出自論文Understanding the difficulty of training deep feedforward neural network,論文討論的是全連線神經網路，fan_in指第i層神經元個數，fan_out指第i+1層神經元個數，但是我們的卷積神經網路是區域性連線的，此時的fan_in，fan_out是什麼意思呢。
在pytorch中，fan_in指kernel_height x kernel_width x in_channel. fan_out指kernel_height x kernel_width x out_channel，從區域性連線的過程來看似乎並不十分合理，卷積神經網路的區域性連線在感受野內仍然是全連線。fan_in=kh x kw x in_channel沒什麼疑問，但是fan_out應該等於out_channel更合理啊。待解答。

code，來自pytorch實現

def _calculate_fan_in_and_fan_out(tensor):
    dimensions = tensor.ndimension()
    if dimensions < 2:
        raise ValueError("Fan in and fan out can not be computed for tensor with fewer than 2 dimensions")

    if dimensions == 2:  # Linear
        fan_in = tensor.size(1)
        fan_out = tensor.size(0)
    else:
        num_input_fmaps = tensor.size(1)
        num_output_fmaps = tensor.size(0)
        receptive_field_size = 1
        if tensor.dim() > 2:
            receptive_field_size = tensor[0][0].numel()
        fan_in = num_input_fmaps * receptive_field_size
        fan_out = num_output_fmaps * receptive_field_size

    return fan_in, fan_out