How large language models work

Next-token prediction

At heart, an LLM does one thing: given the text so far, it outputs a probability for every possible next token. "The capital of India is" → " New" gets a high probability. The model picks a token, adds it to the text, and repeats. Long answers, code and translations all come from repeating this single step. Models such as GPT, Claude, Gemini, Llama and Mistral all work on this principle, with different training data and designs.

Tokens

LLMs do not read whole words. A tokenizer splits text into sub-word pieces using methods such as byte-pair encoding (BPE): common words are one token, rare words are split into several pieces ("unbelievable" might become "un", "believ", "able"). For English text, one token is very roughly 4 characters or three-quarters of a word. Many tokenizers need more tokens for Hindi and other Indian scripts, which makes those languages costlier per word; this varies by model.

The context window

The context window is the maximum number of tokens the model can look at at once: your instructions, any documents you paste, the chat history and its own answer so far. Anything outside the window is invisible to it. Window sizes vary a lot between models and change often, from a few thousand tokens to hundreds of thousands or more. Long contexts cost more and the model may pay less attention to details in the middle.

Training in stages

1. Pre-training: the model reads a vast amount of text and learns to predict the next token. This teaches grammar, facts, reasoning patterns and also biases present in the data.
2. Instruction tuning (supervised fine-tuning): it is trained on examples of instructions and good answers, so it follows requests.
3. Preference training, such as RLHF (reinforcement learning from human feedback): people compare answers, and the model is tuned towards the preferred ones, for helpfulness and safety.

Attention, briefly

Inside, the transformer uses self-attention (module 9): for each token, it weighs how relevant every earlier token is. That is how "it" in a long paragraph gets linked to the right noun. Stacking many attention layers lets the model build up complex patterns.

Sampling and temperature

The model gives scores (logits) for the next token; softmax turns them into probabilities. Greedy decoding always picks the most likely token: stable but sometimes repetitive. Sampling picks randomly according to the probabilities. Temperature T divides the logits before softmax: T below 1 sharpens the distribution (more predictable), T above 1 flattens it (more varied and more error-prone). Use low temperature for extraction and facts, higher for brainstorming.

import math

def softmax_t(logits, t):
    scaled = [v / t for v in logits]
    m = max(scaled)
    exps = [math.exp(v - m) for v in scaled]
    return [round(e / sum(exps), 4) for e in exps]

print(softmax_t([2.0, 1.0, 0.1], 0.5))
print(softmax_t([2.0, 1.0, 0.1], 2.0))

Why LLMs hallucinate

An LLM produces text that sounds likely, not text it has verified. If it lacks the fact, it may still produce a fluent, confident and wrong answer: a made-up court case, a fake citation, a wrong GST rate. This is called hallucination. Giving it trusted documents (RAG, module 11), asking it to say when it does not know, and checking outputs reduce the problem but do not remove it.

Cost and speed

Hosted LLM APIs usually charge per token, often with a different price for input and output tokens, quoted per million tokens. Longer prompts and answers cost more and take longer. Prices change frequently, so always check the provider's current price page.

Common mistakes

• Treating LLM output as fact. Check anything important.
• Pasting huge documents and assuming the model read every detail. Retrieve the relevant parts instead.
• Using high temperature for tasks that need exact, repeatable answers.