Backpropagation
1 Forward Propagation
This is more like a summary section.
We set $a^{[0]} = x$ for our input to the network and $\ell = 1,2,\dots,N$ where N is the number of layers of network. Then, we have
\[z^{[\ell]} = W^{[\ell]}a^{[\ell-1]} + b^{[\ell]}\] \[a^{[\ell]} = g^{[\ell]}(z^{[\ell]})\]where $g^{[\ell]}$ is the same for all the layers except for last layer. For the last layer, we can do:
1 regression then $g(x) = x$
2 binary then $g(x) = sigmoid(x)$
3 multi-class then $g(x) = softmax(x)$
Finally, we can have the output of the network $a^{[N]}$ and compute its loss.
For regression, we have:
\[\mathcal{L}(\hat{y},y) = \frac{1}{2}(\hat{y} - y)^2\]For binary classification, we have:
\[\mathcal{L}(\hat{y},y) = -\bigg(y\log\hat{y} + (1-y)\log (1-\hat{y})\bigg)\]For multi-classification, we have:
\[\mathcal{L}(\hat{y},y) = -\sum\limits_{j=1}^k\mathbb{1}\{y=j\}\log\hat{y}_j\]Note that for multi-class, if we have $\hat{y}$ as a k-dimensional vector, we can calculate its cross-entropy for its loss:
\[\mathcal{L}(\hat{y},y) = -\sum\limits_{j=1}^ky_j\log\hat{y}_j\]2 Backpropagation
We define that:
\[\delta^{[\ell]} = \triangledown_{z^{[\ell]}}\mathcal{L}(\hat{y},y)\]So we have three steps for computing the gradient for any layer:
1 For output layer N, we have:
\[\delta^{[N]} = \triangledown_{z^{[N]}}\mathcal{L}(\hat{y},y)\]For softmax function, since it is not performed element-wise, so you can directly caculate it as a whole. For sigmoid, it is applied element-wise, so we need to:
\[\triangledown_{z^{[N]}}\mathcal{L}(\hat{y},y) = \triangledown_{\hat{y}}\mathcal{L}(\hat{y},y)\circ (g^{[N]})^{\prime}(z^{[N]})\]Note this is element-wise operation.
2 For $\ell = N-1,N-2,\dots,1$, we have:
\[\delta^{[\ell]} = (W^{[\ell+1]T}\delta^{[\ell+1]})\circ g^{\prime}(z^{[\ell]})\]3 For each layer, we have:
\[\triangle_{W^{[\ell]}}J(W,b) = \delta^{[\ell]}a^{[\ell]T}\] \[\triangle_{b^{[\ell]}}J(W,b) = \delta^{[\ell]}\]This can be directly used in coding, which acts like a formula.
