CNNs, RNNs and transformers at a glance

Why special designs?

A plain dense network treats every input separately. For a 1,000 × 1,000 photo, that is a million inputs per colour and billions of weights in the first layer alone, and it ignores the fact that nearby pixels belong together. Special architectures build in useful assumptions about the data, which makes them smaller, faster and much better.

Convolutional neural networks (CNNs)

A convolution slides a small filter (for example 3 × 3 numbers) across the image. At each position it multiplies the filter by the pixels under it and adds them up, producing a feature map. One filter might respond to vertical edges, another to a certain texture. Because the same filter is reused everywhere, a CNN needs few weights and finds a pattern wherever it appears. Early layers learn edges, middle layers learn shapes like wheels or eyes, and deep layers learn whole objects.

signal = [1, 2, 3, 4, 5]
kernel = [1, 0, -1]
out = [sum(signal[i + j] * kernel[j] for j in range(len(kernel)))
       for i in range(len(signal) - len(kernel) + 1)]
print(out)

Pooling layers shrink feature maps. 2 × 2 max pooling keeps the largest value in each 2 × 2 block, halving the width and height and making the network less sensitive to small shifts.

Recurrent networks (RNNs and LSTMs)

Text, speech and sensor readings are sequences. An RNN reads them one step at a time, carrying a hidden state (a memory) from step to step. Plain RNNs forget long-range information because gradients fade over many steps; LSTMs and GRUs add gates that decide what to keep and what to forget. They were the standard for language until about 2017, but they are slow to train because each step waits for the previous one.

Attention and transformers

A transformer (introduced in the 2017 paper "Attention Is All You Need") processes all positions of a sequence at once. Its self-attention mechanism lets every word look at every other word and decide which ones matter for its meaning: in "the bank of the river", "bank" pays attention to "river". Because all positions are processed in parallel, transformers train very efficiently on GPUs. They power large language models (module 10) and, as vision transformers, many image models too.

Transfer learning

Training a big network from scratch needs huge data and compute. Transfer learning starts from a network already trained on a large dataset (for example ImageNet, with over a million labelled images) and adapts it to your task. Often you freeze the early layers, which already detect general edges and shapes, and retrain only the last layers on your few thousand examples. This is how small teams build strong image and text models.

Common mistakes

• Using a dense network on raw images. Fix: use a CNN or a pre-trained vision model.
• Training from scratch on a small dataset. Fix: start from a pre-trained model.
• Feeding images of a different size or colour scaling from what the pre-trained model expects. Fix: use the model's own preprocessing.