Technology, Knowledge & Philosophy

A Journey from “Where is Paris?” to How Large Language Models Actually Remember

One of the most fascinating questions about Large Language Models (LLMs) is deceptively simple:

Where is the information stored?

Not intelligence. Not reasoning. Just information.

For example, consider the word “Paris.” Surely the model has to store it somewhere. Otherwise, how could it ever generate the word?

The answer turns out to reveal one of the most beautiful ideas in modern AI.

Two Different Things: Words and Knowledge

The first key insight is that we must distinguish between a word and what the model knows about that word.

These are stored in completely different ways.

The Word Is Stored Explicitly

Every LLM has a tokenizer—a component that converts text into numerical IDs.

Its vocabulary might conceptually look something like this:

When the model wants to generate the word Paris, it predicts the corresponding token ID, and the tokenizer converts it back into text.

So yes, the string “Paris” is stored explicitly in the model’s vocabulary.

But Knowledge Is Not

Now consider another question:

Where is the fact that Paris is the capital of France stored?

Many people imagine that somewhere inside the model there is a hidden table like this:

France → Paris
Germany → Berlin
Japan → Tokyo

There isn’t.

Instead, the model has learned billions of numerical parameters called weights.

Those weights do not contain sentences.

They do not contain facts written in English.

They contain patterns.

When the model reads the prompt:

The capital of France is…

its learned patterns assign probabilities to every word in the vocabulary.

Conceptually, it may internally arrive at something like:

Since Paris has the highest probability, that is the word it generates.

Is “Paris” Stored Once or Many Times?

This question leads to another subtle distinction.

The Word Itself

The text “Paris” is generally stored only once in the tokenizer’s vocabulary.

Whenever the model needs that word, it simply predicts its token ID.

The Knowledge About Paris

This is completely different.

Everything the model knows about Paris—

• that it is in France,
• that it is associated with the Eiffel Tower,
• that it is famous for art and fashion,
• that it hosted the Olympics,
• that it appears in literature—

is not stored in one place.

Instead, this knowledge is distributed across millions—or even billions—of parameters throughout the neural network.

No single parameter says:

“Paris is the capital of France.”

Rather, countless parameters each contribute a tiny amount to the model’s understanding.

Researchers call this a distributed representation.

But “Paris” Can Mean Many Things

At this point another important observation arises.

The word Paris does not always refer to the French capital.

Depending on context, it could mean:

• the city in France,
• Paris, Texas,
• Paris Hilton,
• Paris, the prince from Greek mythology,
• or simply someone’s first name.

So how can the model use only one stored word?

The answer is that the vocabulary stores only the symbol, not the meaning.

The meaning is created dynamically from context.

Consider these sentences:

The capital of France is Paris.

Here, the model interprets Paris as the city.

Now consider:

Paris kidnapped Helen.

Now the model understands Paris as the Trojan prince from Greek mythology.

Or:

Paris Hilton attended the event.

Now it identifies Paris as a person’s first name.

The remarkable part is that the tokenizer has not changed.

The stored word is the same.

What changes is the internal representation of that word as it flows through the layers of the neural network.

Modern transformer models create contextual representations.

Instead of assigning one permanent meaning to “Paris,” they continuously refine its meaning based on the surrounding words.

An Analogy with the Human Brain

Our own minds seem to work similarly.

If someone simply says:

Paris

several possibilities may briefly exist.

But the moment you hear:

I flew to Paris yesterday.

your brain instantly settles on the city.

Or if someone says:

Paris defeated Menelaus.

your thoughts immediately shift to Greek mythology.

Meaning emerges from context.

LLMs do something surprisingly similar.

A Useful Analogy

Imagine a dictionary.

The word Paris appears only once.

But thousands of books discuss Paris.

Some describe its history.

Some discuss fashion.

Some talk about World War II.

Others mention art, food, politics, or tourism.

The dictionary stores the word.

The books contain the knowledge.

An LLM is similar.

The vocabulary stores the symbol.

The neural network stores the relationships.

The Big Takeaway

One of the biggest misconceptions about LLMs is imagining them as giant databases full of hidden facts.

They are not.

They are enormous networks of learned numerical relationships.

The word Paris may be stored explicitly only once.

But everything the model knows about Paris is encoded across millions of interconnected parameters, allowing it to reconstruct the appropriate meaning whenever context demands it.

Perhaps the most surprising insight is this:

An LLM does not retrieve facts the way a database does. It reconstructs them from patterns it has learned.

That difference is what makes Large Language Models fundamentally different from traditional software—and one of the reasons they are both so powerful and so difficult to fully understand.