Improvements

Improvements#

There are many ways we could improve the performance of our network, but these will add a lot of complexity to the simple class that we wrote. Fortunately there are a lot of machine learning libraries that provide these features, and work efficiently, so for real applications we would want to use one of those libraries (we’ll explore these next).

Batching#

Right now, we did our training as:

Loop over the $T$ pairs $(x^{k}, y^{k})$ for $k = 1, \dots, T$
- Propagate $(x^{k}, y^{k})$ through the network
- Compute the corrections $\partial L / \partial A$ , $\partial L / \partial B$
- Update the matrices:
  
  $A \leftarrow A - η \frac{\partial L}{\partial A}$
  
  $B \leftarrow B - η \frac{\partial L}{\partial B}$

In this manner, each training pair sees slightly different matrices $A$ and $B$ , as each previous pair updates it immediately.

We could instead divide our training set into $N$ batches, each with $τ = T / N$ training pairs and do our update as:

Loop over $N$ batches
- Loop over the $τ$ pairs $(x^{k}, y^{k})$ for $k = 1, \dots, τ$ in the current batch
  - Propagate $(x^{k}, y^{k})$ through the network
  - Compute the gradients $\partial L / \partial A^{k}$ , $\partial L / \partial B^{k}$ from the current pair
  - Accumulate the gradients:
    
    $\frac{\partial L}{\partial A} = \frac{\partial L}{\partial A} + \frac{\partial L}{\partial A^{k}}$
    
    $\frac{\partial L}{\partial B} = \frac{\partial L}{\partial B} + \frac{\partial L}{\partial B^{k}}$
- Apply a single update to the matrices for this batch:
  
  $A \leftarrow A - \frac{η}{τ} \frac{\partial L}{\partial A}$
  
  $B \leftarrow B - \frac{η}{τ} \frac{\partial L}{\partial B}$

Note

We normalize the accumulated gradients by the batch size, $τ$ , which means that we are applying the average gradient over the batch.

The advantage of this is that the $τ$ trainings in a batch can all be done in parallel now, spread across many CPU cores or GPU cores. This greatly accelerates the training time.

Different activation or cost functions#

We used a simple cost function: the sum of the square of the errors. This is analogous to the $L_{2}$ norm we discussed previously. But there are a lot of other cost functions we could explore. Changing the cost function will require us to recompute our derivatives.

Likewise, there are a wide number of activation functions, some of which are not differentiable. The choice of activation function can depend on what type of data you are using. You might also want to use a different activation function on each layer. Again, this would require us to redo our derivatives.

Use a different minimization technique#

We only explored gradient descent. But there are improvements to this (like momentum that we mentioned previously) as well as alternate minimization techniques we could use (some of which don’t need the gradient at all).

Different types of layers / connections#

We only considered a dense network: every node on one layer was connected to every node on the adjacent layer. But there are alternatives.

For example, a convolutional neural network performs a convolution on a layer with some kernel. This helps identifying features.

More hidden layers#

There is no restriction on the number of hidden layers we can use. Each additional hidden layer means an additional matrix is added to our network. For our code, we’d simply need to backpropagate the error to each hidden layer and compute the update to the new matrix.

Auto-differentiation libraries#

At some point, with all of these options, doing all of the differentiation / chain-rule by hand becomes burdensome and prone to errors. For this reason, libraries often use automatic differentiation libraries, like JAX which can take the derivatives of our python functions themselves.