Marouane.B

in this article, we focus on the simplest form of retrieval.

lexical search, where documents and queries are matched through words or terms.

raw text is written for humans. not for search engines.

a human can easily realize that these two sentences are related:

The bear is running.
the Bear ran!

but a basic keyword search system cannot.

to this system, small differences matter:

bear
Bear
bear.
bears
running
ran
ran!

some of these differences are useful and should be preserved. others are just noise (useless and harmful).

words have different casing (e.g: bear, Bear) but probably should match.

punctuation varies (e.g: Let's eat, grandma!, Let's eat grandma!). the comma determines whether you are inviting your grandmother to dinner or threatening to eat her :)

some words are too common to be useful (e.g: the, is)

similar words appear in different forms (e.g: running, ran, run)

preprocessing is the step where we turn raw text into a more searchable form.

it includes things like:

normalizing text
splitting text into tokens
potentially removing stop words
reducing word variants using stemming

the important rule is:

whatever we do to the documents, we must also do to the query.

if documents are lowercased but queries are not, matching becomes inconsistent.

if documents are stemmed but queries are not, we may miss matches.

the pillars of preprocessing

normalization

normalization means converting text into a more consistent form.

the simplest example is lowercasing:

Bear -> bear
BEAR -> bear
bear -> bear

now all three versions match the same term.

we can also remove or normalize punctuation:

"bear!" -> "bear"
"bear." -> "bear"

without normalization, a document containing bear. may not match a query for bear, depending on how the search system tokenizes text.

tokenization

tokenization is the process of splitting text into smaller units, usually words.

"the bear is running"

becomes:

["the", "bear", "is", "running"]

when a user searches:

"bear running"

we tokenize the query too:

["bear", "running"]

then we can compare query tokens with document tokens.

this is the basic shape of keyword search.

stop words

some words appear so often that they carry little meaning for retrieval.

examples:

the, is, a, an, of, to, in

if every document contains the, then matching on the does not help us find the right document.

so many search systems remove stop words before indexing.

["the", "bear", "is", "running"]

becomes:

["bear", "running"]

but this is not always safe.

sometimes stop words matter.

for example:

"to be or not to be"

or:

"the who"

removing stop words blindly can destroy meaning.

so preprocessing always depends on the context/domain.

there is no universal “best” preprocessing pipeline.

there is only a pipeline that fits the search problem.

stemming and lemmatization

words often appear in different forms:

run
runs
running
ran

a user may search for run, while the document says running.

keyword search may miss that match unless we reduce words to a shared form.

stemming does this aggressively:

running -> run
runs -> run

lemmatization does it more carefully, using vocabulary and grammar:

ran -> run
better -> good

stemming is usually simpler and faster.

lemmatization is usually cleaner, but more expensive.

both try to solve the same problem:

different forms can refer to the same underlying idea.

the pipeline

a simple preprocessing pipeline (unordered) might look like this:

raw text
-> lowercase
-> remove punctuation
-> tokenize
-> remove stop words
-> stem or lemmatize

for example:

"The Bear is running through the forest!"

could become:

["bear", "run", "forest"]

now the document is easier to match against queries like:

"bear running"
"running bears"
"forest bear"

this is the first version of retrieval we can build.

not intelligent yet.

not semantic yet.

but already useful.

because now search is no longer comparing raw strings directly.

it is comparing processed terms.

and that small shift matters.

code example (basic)

def normalize_text(text: str) -> str:
    return remove_punctuation(text.lower())


class TextPreprocessor:
    def __init__(self, stop_words: set[str]) -> None:
        self._stop_words = {normalize_text(word) for word in stop_words}
        self._stemmer = PorterStemmer()

    def tokenize(self, text: str) -> list[str]:
        tokens = normalize_text(text).split()

        return [
            self._stemmer.stem(token)
            for token in tokens
            if token not in self._stop_words
        ]

what's next

preprocessing tells us what counts as a word.

but search still has a problem:

doc1: bear
doc2: bear bear bear
doc3: bear, forest, river

all of them match bear.

but they should not have the same score.

That is the problem TF-IDF tries to solve.

One Problem at a Time: Building RAG From Scratch - part 2