Bonky Zhu
If someone is able to show me that what I think or do is not right, I will happily change, for I seek the truth, by which no one was ever truly harmed. It is the person who continues in his self-deception and ignorance who is harmed.

解决梯度消失 – 残差连接

残差连接 (Residual connection)的示意图如下:

屏幕快照 2019-11-06 下午7.52.34

即残差链接就是对网络的输出 $F(X)$ 加上了一个 $X$ 成为了 $F(X)+X$,因为加上了一个 $X$ ,在反向传播的时候,求偏导的时候 会多出来一个1,所以避免了梯度消失。