Embeddings and cosine similarity

Meaning as a position

With TF-IDF, "refund" and "money back" share nothing. An embedding model is trained so that texts with similar meaning land close together in a space with hundreds of dimensions. A query like "money not returned" can then find documents about refunds even if they use different words. This is called semantic search: search by meaning, not exact words.

Where embeddings come from

Word2vec (2013) learned word vectors by predicting a word from its neighbours. It famously captured relationships like king − man + woman ≈ queen, although such neat analogies work less often than the famous example suggests. Today, sentence-embedding models built on transformers produce one vector per sentence or paragraph, and many APIs and open models offer them. The same idea works for images, songs and products.

Measuring similarity: cosine

Cosine similarity = (a · b) ÷ (|a| × |b|), where a · b is the dot product (multiply matching numbers and add) and |a| is the length (square root of the sum of squares). It measures the angle between the vectors: 1 means same direction (very similar), 0 means unrelated (at right angles), −1 means opposite. It ignores length, so a long and a short document about the same topic can still score high.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

print(round(cosine([1, 2, 3], [2, 4, 6]), 4), round(cosine([1, 1], [1, 0]), 4))

Semantic search in three steps

1. Embed every document once and store the vectors.
2. When a query arrives, embed it with the same model.
3. Compute cosine similarity with every stored vector and return the top matches.

With millions of documents, comparing with every vector is slow, so vector indexes and databases do approximate nearest-neighbour search very fast. Examples include FAISS (a library), pgvector (an extension for PostgreSQL) and dedicated vector databases. Module 11 builds RAG on this.

Common mistakes

• Comparing vectors made by different embedding models. Their spaces are unrelated; always use one model for both documents and queries.
• Dividing by zero for an all-zero vector. Guard it and return 0.
• Assuming high similarity means "correct answer". Similar is not the same as true; check results.
• Embedding very long documents as one vector, which blurs the meaning. Split them into chunks first (module 11).