Bag of words, TF-IDF and text classification

Turning documents into numbers

Models need fixed-length lists of numbers. The bag-of-words idea: collect every distinct word in all documents into a vocabulary, and represent each document as a list of counts, one per vocabulary word. "chai is hot" with vocabulary [biscuit, chai, coffee, hot, is] becomes [0, 1, 0, 1, 1]. The order of words is thrown away, like tipping the words into a bag.

The problem with raw counts

Common words like "is", "the" or, in support tickets, "order" appear everywhere, so they dominate the counts without telling documents apart. We want to boost words that are special to a document.

TF-IDF

• TF (term frequency): how often a word appears in this document. One common version is count ÷ number of tokens in the document.
• IDF (inverse document frequency): how rare the word is across all N documents: ln(N ÷ df), where df is the number of documents containing the word. A word in every document gets ln(1) = 0.
• TF-IDF = TF × IDF: high when a word is frequent here but rare elsewhere.

import math

docs = [["chai", "is", "hot"], ["coffee", "is", "hot"], ["chai", "chai", "biscuit"]]
N = len(docs)
df = {w: sum(1 for d in docs if w in d) for d in docs for w in d}
d3 = docs[2]
print({w: round(d3.count(w) / len(d3) * math.log(N / df[w]), 4) for w in set(d3)})

n-grams

Bag of words loses order, so "not good" and "good" look similar. Adding n-grams, sequences of n neighbouring words, brings some order back: bigrams (n = 2) include "not good" as one feature. The vocabulary grows fast, so limit it (for example, keep the 20,000 most frequent features).

Text classification

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2), LogisticRegression(max_iter=1000))
clf.fit(train_texts, train_labels)               # e.g. "billing", "delivery", "technical"
print(clf.predict(["my refund has not come yet"]))

Limits

Bag-of-words models do not understand meaning. "Cheap" and "inexpensive" are unrelated columns; a word never seen in training is ignored. They also miss sarcasm and long-range context. Embeddings (next topic) and transformers address these limits, but TF-IDF with logistic regression remains a surprisingly strong baseline that trains in seconds.

Common mistakes

• Fitting the vectoriser (building the vocabulary and IDF) on all data, including the test set. Fit it on training text only, inside a pipeline.
• Not limiting the vocabulary, giving hundreds of thousands of sparse columns.
• Skipping this baseline and jumping to a large model that is slower and only slightly better.