Gfrktyl

I'm learning AI, but this confuses me. The derivative function used in backpropagation is the derivative of activation function or the derivative of loss function?

These terms are confusing: derivative of act. function, partial derivative wrt. loss function??

I'm still not getting it correct.

In back propagation, both the derivative of the loss function as well as the derivative of the activation function are used for error minimization.

• The derivative of the loss function is used to compute the compute the gradients between the last hidden layer and output layer.




• The derivative of the activation function is used to compute the gradients of all layers except the output layer.

The weights from a layer get activated in the next layer. Hence in this scenario, the derivative of the activation function will be used.

The weights from the last hidden layer get activated in the output layer. Hence, here the derivative of the loss function is used since the output layer utilizes the loss function.

Overview

The derivatives of functions are used to determine what changes to input parameters correspond to what desired change in output for any given point in the forward propagation and cost, loss, or error evaluation &mdash whatever it is conceptually the learning process is attempting to minimize. This is the conceptual and algebraic inverse of maximizing valuation, yield, or accuracy.

Back-propagation estimates the next best step toward the objective quantified in the cost function in a search. The result of the search is a set of parameter matrices, each element of which represents what is sometimes called a connection weight. The improvement of the values of the elements in the pursuit of minimal cost is artificial networking's basic approach to learning.

Each step is an estimation because the cost function is a finite difference, where as the partial derivatives express the slope of a hyper-plane normal to surfaces that represent functions that comprise forward propagation. The goal is to set up circumstances so that successive approximations approach the ideal represented by minimization of the cost function.

Back-propagation Theory

Back-propagation is a scheme for distribution of a correction signal arising from cost evaluation after each sample or mini-batch of them. With a form of Einsteinian notation, the current convention for distributive, incremental parameter improvement can be expressed concisely.

$$ Delta P = dfrac {c(vec{o}, vec{ell}) ; alpha} {big[ prod^+ ! P big] ; big[ prod^+ !a'(vec{s} , P + vec{z}) big] ; big[ c'(vec{o}, vec{ell}) big]} $$

The plus sign in $prod^+!$ designates that the factors multiplied must be downstream in the forward signal flow from the parameter matrix being updated.

In sentence form, $Delta P$ at any layer shall be the quotient of cost function $c$ (given label vector $vec{ell}$ and network output signal $vec{o}$), attenuated by learning rate $alpha$, over the product of all the derivatives leading up to the cost evaluation. The multiplication of these derivatives arise through the recursive application of the chain rule.

It is because the chain rule is a core method for feedback signal evaluation that partial derivatives must be used. All variables must be bound except for one dependent and one independent variable for the chain rule to apply.

The derivatives include three types.

• All layer input factors, the weights in the parameter matrix used to attenuate the signal during forward propagation, which are equal to the derivatives of those signal paths


• All the derivatives of activation functions $a$ evaluated at the sum of the matrix vector product of the parameters and signal at that layer plus the bias vector


• The derivative of the cost function $c$ evaluated at the current output value $vec{o}$ with the label $vec{ell}$
  

Answer to the Question

Note that, as a consequence of the above, the derivatives of both the cost (or loss or error) function and any activation functions are necessary.

Redundant Operation Removal for an Efficient Algorithm Design

Actual back propagation algorithms save computing resources and time using three techniques.

• Temporary storage of the value used for evaluation of the derivative (since it was already calculated during forward propagation)


• Temporary storage of products to avoid redundant multiplication operations (a form of reverse mode automatic differentiation)


• Use of reciprocals because division is more costly than multiplication at a hardware level

In addition to these practical principles of algorithm design, other algorithm features arise from extensions of basic back-propagation. Mini-batch SGD (stochastic gradient descent) applies averaging to improve convergence reliability and accuracy in most cases, provided hyper-parameters and initial parameter states are well chosen. Gradual reduction of learning rates, momentum, and various other techniques are often used to further improve outcomes in deeper artificial networks.