Deep learning tools and hardware

Frameworks

All of them provide tensors (multi-dimensional arrays like NumPy's, but able to live on a GPU), automatic gradients, ready-made layers, losses and optimisers. Learning one well is enough; the ideas carry over.

import torch

x = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
print(x.shape)                 # torch.Size([2, 2])
device = "cuda" if torch.cuda.is_available() else "cpu"
x = x.to(device)               # move the data to the GPU if there is one
print((x @ x).sum().item())    # 54.0 (matrix multiply, then sum)

Why GPUs?

A CPU has a few powerful cores that handle varied tasks one after another. A GPU has thousands of simpler cores that do the same maths on many numbers at once. Training a network is mostly multiplying large matrices, which splits perfectly across those cores, so GPUs (and Google's TPUs) can be many times faster than CPUs for this work. For small classic ML models (module 7), a CPU is fine.

How much memory does a model need?

A quick estimate: number of parameters × bytes per parameter. In float32 each parameter takes 4 bytes; in float16 or bfloat16, 2 bytes; in 8-bit quantised form, 1 byte. A 7-billion-parameter model in float16 needs about 7 × 10⁹ × 2 = 14 × 10⁹ bytes, roughly 14 GB, just to hold the weights. Training needs several times more for gradients and optimiser state. Quantisation shrinks models so they fit on smaller GPUs or even laptops, with some loss of quality.

Where to run things

• Your laptop: fine for learning, classic ML and small networks on CPU.
• Free notebooks: Google Colab and Kaggle Notebooks offer limited free GPU time. Limits and hardware change often, so check the current terms.
• Cloud GPUs: AWS, Google Cloud, Azure and Indian providers rent GPUs by the hour for bigger jobs. Watch costs and shut machines down.

Saving, sharing and serving models

Save trained weights to a file (in PyTorch, torch.save(model.state_dict(), "model.pt")) so you can load them later without retraining. The Hugging Face Hub hosts open models and datasets you can download with a few lines. For production, models are often exported to formats like ONNX and served behind an API (module 11).

Common mistakes

• Model on the GPU but data on the CPU (or the reverse), giving a "tensors on different devices" error. Fix: move both with .to(device).
• Running out of GPU memory. Fix: smaller batches, mixed precision (float16) or a smaller model.
• Leaving cloud GPUs running overnight and getting a big bill.
• Using deep learning where gradient boosting on a table would be simpler and better.