Layers, activations and the forward pass

From one neuron to a network

Put many neurons side by side and you get a layer. Feed one layer's outputs into the next and you get a network. A dense (fully connected) layer connects every input to every neuron in the layer. "Deep" learning simply means networks with many hidden layers between the input and the output.

A dense layer as maths

If a layer has 3 inputs and 2 neurons, it has a 2 × 3 weight matrix (one row of 3 weights per neuron) and 2 biases. Its output is: for each neuron, the weighted sum of the inputs plus that neuron's bias, then the activation. In matrix form: output = activation(W·x + b). GPUs are very good at exactly this kind of matrix maths.

x = [1.0, 2.0, 3.0]
W = [[0.2, 0.0, -0.1],     # neuron 1
     [0.5, -0.5, 0.25]]    # neuron 2
b = [0.1, 0.0]
z = [sum(w * xi for w, xi in zip(row, x)) + bj for row, bj in zip(W, b)]
relu = [max(0.0, v) for v in z]
print([round(v, 2) for v in z], [round(v, 2) for v in relu])

Why activations must be non-linear

If every layer only multiplied and added, stacking ten layers would still be one big linear function, no better than a single layer. Non-linear activations between layers let the network bend its decision boundaries and model complex patterns, which solves problems like XOR.

Softmax

For a 3-class problem (say, routing a ticket to billing, delivery or technical), the last layer outputs 3 raw scores called logits. Softmax turns them into probabilities that are positive and add up to 1. Bigger scores get bigger probabilities. In code, subtract the largest score from all scores before taking exponentials; the answer is the same, but huge scores no longer overflow.

import math

logits = [2.0, 1.0, 0.1]
m = max(logits)
exps = [math.exp(v - m) for v in logits]
print([round(e / sum(exps), 4) for e in exps])

Counting parameters

A dense layer from a inputs to b neurons has a × b weights plus b biases. A network with layers of size 784 → 128 → 10 (a classic handwritten-digit model) has 784 × 128 + 128 = 1,00,480 parameters in the first layer and 128 × 10 + 10 = 1,290 in the second: 1,01,770 in total. Large language models have billions.

Common mistakes

• Using softmax for multi-label problems where several labels can be true at once. Use one sigmoid per label instead.
• Mixing up the weight matrix shape. Check: (neurons in this layer) × (inputs).
• Computing softmax without subtracting the max, which overflows for large scores.