Working with text: tokens and cleaning

Why text needs special handling

Customer reviews, support tickets, WhatsApp messages and news articles are text, not numbers. Before any model can use them, we must turn them into numbers. How we split and clean the text decides what the model can see. A support team in India may get "refund nahi mila, pls help!!" and "Refund not received" about the same problem; good NLP should treat them as close.

Tokens and tokenising

A token is one unit of text that the system treats as a piece. For classic NLP, tokens are usually words. Tokenising means splitting text into tokens. The simplest way, text.split(), breaks on spaces but leaves punctuation stuck to words ("help!!"). A regular expression gives more control. Modern language models use sub-word tokens instead (topic 4).

import re

text = "Don't panic! UPI-payment of Rs 500, done."
tokens = re.findall(r"[a-z0-9']+", text.lower())
print(tokens)

Common cleaning steps

• Lowercasing: "Delivery" and "delivery" become one token. (Careful: it can merge "US" the country with "us".)
• Removing punctuation and extra spaces.
• Removing stop words: very common words such as "the", "is", "and" that carry little meaning for tasks like topic detection. For sentiment, keep negations like "not"!
• Stemming: chopping word endings with simple rules ("delivered", "delivering" → "deliv"). Fast but crude.
• Lemmatisation: turning words into their dictionary form using vocabulary and grammar ("better" → "good", "was" → "be"). More accurate, slower. Libraries such as NLTK and spaCy provide both.

Counting words

Once text is tokenised, counting is easy with collections.Counter. Word counts already answer useful questions: which complaints are most frequent this week, which product features customers mention most. Sort by count, and break ties alphabetically so results are repeatable.

from collections import Counter

tickets = "late delivery. refund please. delivery was late again"
counts = Counter(tickets.replace(".", "").split())
print(counts.most_common(2))

Indian-language challenges

• Many scripts: Devanagari, Tamil, Bengali, Telugu and more, each with its own rules for combining characters.
• Code-mixing: Hindi and English in one sentence ("kal meeting cancel hai").
• Transliteration: Hindi written in Latin letters with no fixed spelling ("nahi", "nahin", "nai").
• Fewer labelled datasets than English, although open resources such as AI4Bharat's models and Bhashini, the government's language platform, are growing.

Common mistakes

• Removing "not" as a stop word in sentiment tasks, which turns "not good" into "good".
• Applying English-only rules (like an English stop-word list or stemmer) to Hindi or mixed text.
• Cleaning the training data one way and live data another way. Fix: one cleaning function used everywhere.
• Splitting on spaces only, leaving "late." and "late" as different tokens.