Neural Networks Explained

15 Pages

English

Edit

1. Neural Networks as Function Approximators

2. Vectors, Matrices, and Layers

3. Activation Functions and Nonlinearity

4. Loss Functions Define the Goal

5. Forward Pass: Computing Predictions

6. Gradient Descent and Learning Direction

7. Backpropagation: Efficient Credit Assignment

8. Parameter Updates and Optimizers

9. Data Pipelines and Normalization

10. Multilayer Perceptrons

11. Convolutional Neural Networks

12. RNNs and Transformers for Sequences

13. Generalization, Validation, and Regularization

14. Common Failure Modes

15. End-to-End Training Loop

1. Neural Networks as Function Approximators

Loading equations

Does a neural network explicitly store the training examples?

Why is the function view useful?

Function approximation:

Input vector:

Parameters:

Prediction:

Function approximation:

Input vector:

Parameters:

Prediction:

Landscape

Fullscreen

2. Vectors, Matrices, and Layers

Loading equations

Why use matrices instead of writing every neuron separately?

What does a neuron represent mathematically?

Weight matrix:

Bias vector:

Affine transformation:

Batch:

Weight matrix:

Bias vector:

Affine transformation:

Batch:

Landscape

Fullscreen

3. Activation Functions and Nonlinearity

Loading equations

Why not use only linear layers?

Is ReLU always the best activation?

Activation function:

ReLU:

Sigmoid:

Vanishing gradient:

Activation function:

ReLU:

Sigmoid:

Vanishing gradient:

Landscape

Fullscreen

4. Loss Functions Define the Goal

Loading equations

Why must the loss be differentiable or almost differentiable?

Can high accuracy and high loss happen together?

Loss function:

Mean squared error:

Cross-entropy:

Regularization:

Loss function:

Mean squared error:

Cross-entropy:

Regularization:

Landscape

Fullscreen

5. Forward Pass: Computing Predictions

Loading equations

Why store intermediate activations during training?

Is inference the same as training?

Forward pass:

Hidden representation:

Softmax:

Inference:

Forward pass:

Hidden representation:

Softmax:

Inference:

Landscape

Fullscreen

6. Gradient Descent and Learning Direction

Loading equations

Why subtract the gradient?

Can gradient descent get stuck?

Gradient:

Learning rate:

Mini-batch SGD:

Nonconvex surface:

Gradient:

Learning rate:

Mini-batch SGD:

Nonconvex surface:

Landscape

Fullscreen

7. Backpropagation: Efficient Credit Assignment

Loading equations

Is backpropagation biologically realistic?

Why is backpropagation efficient?

Backpropagation:

Chain rule:

Credit assignment:

Optimizer:

Backpropagation:

Chain rule:

Credit assignment:

Optimizer:

Landscape

Fullscreen

8. Parameter Updates and Optimizers

Loading equations

Why is Adam popular?

Does a better optimizer guarantee better generalization?

SGD:

Momentum:

Adam:

Weight decay:

SGD:

Momentum:

Adam:

Weight decay:

Landscape

Fullscreen

9. Data Pipelines and Normalization

Loading equations

Why compute normalization statistics only on training data?

Can preprocessing change model behavior?

Data pipeline:

Normalization:

Data leakage:

Tokenization:

Data pipeline:

Normalization:

Data leakage:

Tokenization:

Landscape

Fullscreen

10. Multilayer Perceptrons

Loading equations

Are MLPs obsolete?

Why can dense layers be inefficient for images?

MLP:

Dense layer:

Feedforward:

Parameter count:

MLP:

Dense layer:

Feedforward:

Parameter count:

Landscape

Fullscreen

11. Convolutional Neural Networks

Convolutional neural networks are designed for grid-like data such as images, spectrograms, and medical scans. Instead of connecting every input pixel to every neuron, a convolution applies small learnable filters across local neighborhoods. This creates parameter sharing: the same detector can recognize an edge or texture in many positions. A convolutional layer transforms an input tensor into feature maps, often followed by nonlinearities, pooling, normalization, or residual connections. CNNs encode useful inductive biases: locality, translation equivariance, and hierarchical feature extraction. Early layers often detect edges and colors, middle layers detect motifs, and later layers detect object parts or semantic patterns. This design greatly reduces parameters compared with fully connected image models while improving data efficiency.

Why are CNNs good for images?

What does pooling do?

Convolution:

Feature map:

Parameter sharing:

Inductive bias:

Convolution:

Feature map:

Parameter sharing:

Inductive bias:

Landscape

Fullscreen

12. RNNs and Transformers for Sequences

Loading equations

Why did Transformers replace many RNNs?

Do Transformers understand word order?

RNN:

Hidden state:

Self-attention:

Transformer:

RNN:

Hidden state:

Self-attention:

Transformer:

Landscape

Fullscreen

13. Generalization, Validation, and Regularization

A network’s goal is not to minimize training loss perfectly, but to perform well on unseen data. Generalization is estimated with validation and test sets that are not used for parameter updates. Overfitting occurs when a model learns training-specific noise or shortcuts, producing low training loss and poor validation performance. Regularization methods reduce this risk: dropout randomly masks activations during training, data augmentation creates realistic variations, weight decay penalizes large weights, and early stopping halts training when validation performance stops improving. The bias-variance tradeoff appears in model capacity: too small a network underfits, while too large a poorly regularized network may overfit. Good evaluation also requires metrics aligned with the task, class balance, and deployment conditions.

Why not train until the loss is as low as possible?

Is a larger dataset a form of regularization?

Generalization:

Overfitting:

Dropout:

Validation set:

Generalization:

Overfitting:

Dropout:

Validation set:

Landscape

Fullscreen

14. Common Failure Modes

Neural networks fail in predictable ways that are often diagnosable. Underfitting appears when both training and validation performance are poor, suggesting insufficient capacity, bad features, weak optimization, or excessive regularization. Overfitting appears when training performance is strong but validation performance is weak. Vanishing or exploding gradients make deep or recurrent models train slowly or unstably; gradient clipping, normalization, residual connections, and better initialization can help. Dataset problems are equally serious: mislabeled examples, class imbalance, distribution shift, leakage, and spurious correlations can produce models that look accurate in benchmarks but fail in deployment. Numerical issues such as NaNs, saturated activations, and inappropriate learning rates can stop learning entirely. Debugging requires observing data, losses, gradients, and predictions together.

What is the first thing to check when training fails?

Can high benchmark accuracy still be unsafe?

Underfitting:

Distribution shift:

Exploding gradient:

Spurious correlation:

Underfitting:

Distribution shift:

Exploding gradient:

Spurious correlation:

Landscape

Fullscreen

15. End-to-End Training Loop

Loading equations

Why log both training and validation metrics?

Why save checkpoints?

Training loop:

Epoch:

Checkpoint:

Gradient norm:

Training loop:

Epoch:

Checkpoint:

Gradient norm:

Landscape

Fullscreen

Knowledge.
Made Visual.

Create your first visual book and get started