Why AI Thinks in Tokens Instead of Words

Humans experience language as meaning.

Large language models experience it as probability, structure, and prediction.

The difference is not cosmetic. It is architectural.

When most people use systems like ChatGPT, they instinctively imagine something human-like happening inside the machine.

A sentence enters.
The AI "understands" it.
A response appears.

But internally, modern language models operate in a radically different way — and understanding that difference reveals something surprising not just about machines, but about language itself.

The system does not process:

"Hello, how are you?"

as a complete human sentence carrying meaning.

Instead, language is first transformed into smaller mathematical fragments called tokens — and those tokens become the true building blocks through which modern AI systems process language, code, mathematics, logic, and reasoning-like behavior itself.

What sounds like a technical implementation detail is actually one of the deepest architectural decisions in artificial intelligence.

Because the answer to why tokens? leads directly to probability theory, information compression, and a question that has no comfortable resolution: does the machine understand anything at all?

Not Words. Not Characters. Something In Between.

A token is a small unit of information processed by a language model. It sits in a middle ground that is neither a full word nor a single character — and that positioning is not accidental. It is the result of a mathematical optimization.

Sometimes a token is an entire common word. Sometimes it is a prefix, a suffix, or a root fragment. The exact boundary depends on the tokenizer, but the principle is consistent: the model learns to operate on reusable linguistic units, not on arbitrary divisions of text.

Consider how the word unbelievable might be tokenized:

un  |  believe  |  able

Or mathematics:

math  |  ematics

This might look like simple syllable-splitting. It is not. The divisions emerge statistically from the data — the tokenizer discovers that math appears constantly across millions of documents, that un- functions as a productive prefix, that -able recurs across thousands of adjectives. These fragments become reusable units because language itself keeps reusing them.

Why Whole Words Fail at Scale

The intuitive approach would be to build a vocabulary of complete words and give the model one token per word. This is how early NLP systems worked, and it breaks down quickly.

Human language is too large, too irregular, and too alive to be fixed into a closed vocabulary. A single root generates enormous variation:

mathematics
mathematical
mathematician
mathematically
antimathematical

And that is before accounting for slang, spelling errors, scientific notation, programming syntax, URLs, emojis, or the 7,000 languages that exist in the world. A word-level model either carries a vocabulary of millions of entries — prohibitively expensive — or fails silently on anything outside its training distribution.

Tokenization solves this by making language compositional. The model does not memorize every word. It learns structural fragments that can be assembled into words it has never seen before, the way a reader encounters an unfamiliar scientific term and decodes it from its Latin roots.

Less like a dictionary.
More like a grammar of reusable parts.

Why Individual Characters Also Fail

The opposite approach — processing text one character at a time — sounds elegant but carries its own cost.

The word mathematics contains eleven characters. A sentence of twenty words becomes a sequence of over a hundred individual steps, each one carrying almost no information. The model must learn to span enormous distances in the sequence just to connect a word's beginning to its end.

Computationally, longer sequences mean more memory, more attention operations, and exponentially higher cost. Character-level models exist and have interesting properties, but they cannot scale to the lengths modern language models need to operate over.

Tokens are the engineering compromise that makes large-scale language modeling possible: compact enough for efficient computation, granular enough to handle anything a human can type.

The Algorithm Behind the Division: Byte Pair Encoding

Most modern language models — GPT, Claude, Llama, Gemini — use a tokenization method called Byte Pair Encoding (BPE), or a close variant of it.

The algorithm does not rely on linguistic knowledge. It discovers structure statistically. Here is how it works, step by step:

Step 1. Start with the training text split into individual characters, with a special end-of-word marker. The word low becomes l o w </w>, and lower becomes l o w e r </w>.

Step 2. Count every adjacent pair of symbols across the entire dataset. Find the most frequent pair.

Step 3. Merge that pair into a single new symbol. If l o is the most common pair, all occurrences of l o become lo.

Step 4. Repeat — count pairs in the updated text, merge the most frequent, repeat — for a fixed number of iterations, typically 10,000 to 50,000 merges.

After enough iterations, the algorithm has built a vocabulary where common words survive as single tokens, common subwords become stable units, and rare or novel text can still be represented through smaller fragments or even individual characters.

The result is a vocabulary that reflects the actual statistical structure of human language — not a linguist's theory of it.

Claude Shannon and the Compression of Language

There is a deeper intellectual tradition behind all of this.

Claude Shannon, the founder of information theory, demonstrated in 1948 that any communication system could be studied mathematically through the lens of entropy — a measure of unpredictability. High-entropy messages carry more information. Low-entropy messages are more predictable, more redundant, more compressible.

Human language is deeply redundant. Once you read "The President announced today that...", the next several words become highly predictable. The statistical structure of language means that most of what follows any given sequence could have been anticipated — not with certainty, but with significantly better-than-random probability.

Efficient communication systems exploit this redundancy. They compress recurring patterns into compact representations.

Tokenization does exactly the same thing. The fragments that become stable tokens are precisely those that appear so frequently that encoding them as single units yields genuine compression. A token is, in information-theoretic terms, a compressed unit of linguistic predictability.

The connection is not metaphorical. BPE is a compression algorithm. Tokenization is compression applied to language before the model ever sees a single sentence.

Large Language Models Are Prediction Machines

With tokens defined, the core operation of a language model becomes precise.

At every step, the model receives a sequence of tokens and produces a probability distribution over the entire vocabulary: given what has come before, how likely is each possible next token?

If the model sees:

2 + 2 =

the token 4 receives an extremely high probability.

If it sees:

Once upon a

the token time becomes overwhelmingly likely.

If it sees:

The mitochondria is the powerhouse of the

the token cell is nearly certain.

The model does not retrieve these completions from a database. It has learned, through training on enormous quantities of text, which token sequences tend to follow which other token sequences — and it applies that learned distribution to every new input it encounters.

Everything the model generates — every explanation, every story, every solution — is the accumulated result of this process: one token at a time, one probability distribution at a time, at a scale that makes the emergent output look indistinguishable from understanding.

This is also where the context window fits naturally into the picture. When people refer to "128k context" or "1 million tokens," they are describing the length of the token sequence the model can hold in memory during a single operation. Everything — the conversation history, the instructions, the documents, the model's own previous responses — must fit within that window. When a long conversation loses track of earlier details, it is not forgetting in any human sense. It is running out of token space.

Tokens Are Not Just Language

One of the less obvious consequences of token-based architecture is its universality.

The same system that tokenizes English prose also tokenizes Python code, HTML markup, LaTeX mathematical notation, JSON data structures, and emoji sequences. To the model, these are all just symbol streams with statistical structure.

<div class="hero">

becomes tokens.

$$\int_0^\infty e^{-x^2} \, dx$$

becomes tokens.

def fibonacci(n): return n if n < 2 else fibonacci(n-1) + fibonacci(n-2)

becomes tokens.

The model does not switch modes when it moves from prose to code to mathematics. It applies the same underlying mechanism to a different region of symbolic space — one where the statistical relationships happen to encode syntactic rules, type constraints, and mathematical identities rather than grammatical agreements and semantic associations.

This universality was not obvious in advance. It turned out to be one of the most significant architectural properties of transformer-based models: a single statistical engine, given enough data and scale, learns to operate coherently across almost every symbolic domain humans use.

Does the AI Actually Understand Meaning?

The token-prediction framework makes the philosophical question unavoidable.

If a language model is, at its core, a very sophisticated function that maps token sequences to probability distributions over next tokens — is there any sense in which it understands what it is saying?

Philosopher John Searle formulated the sharpest version of this challenge in 1980 with the Chinese Room thought experiment. Imagine a person locked in a room, receiving Chinese symbols through a slot and following an elaborate rulebook that tells them which symbols to pass back in response. To someone outside the room, the interaction looks like fluent Chinese conversation. Inside the room, the person understands nothing. They are manipulating symbols according to rules, with no access to meaning.

Searle's argument was that syntax — the manipulation of symbols according to formal rules — is not sufficient for semantics — genuine meaning. The room behaves as if it understands, but it does not.

Large language models invite an uncomfortable version of this question. They process tokens with extraordinary effectiveness. They generate coherent explanations, write working code, solve novel problems, and produce text that reads as though it reflects genuine comprehension. And yet the underlying operation is statistical: predict the next token, weighted by learned co-occurrence patterns across a vast training corpus.

Whether something more than sophisticated symbol manipulation is happening remains genuinely contested. Some researchers argue that sufficiently complex statistical structure over language necessarily encodes semantic content — that meaning and pattern are not as separable as Searle assumed. Others maintain that no amount of token prediction produces real understanding, only an increasingly convincing simulation of it.

What is clear is that the question cannot be resolved by looking at the output alone. The output looks like understanding. That tells us something interesting about the relationship between statistical pattern and meaning — but it does not settle what is happening inside.

Language as Mathematics

There is a final observation worth sitting with.

Humans do not consciously think in tokens. Language feels immediate, meaningful, alive. Yet neuroscience increasingly suggests that the human brain processes language through prediction, pattern completion, compression, and statistical inference — mechanisms that are not entirely different in kind from what language models do, even if they differ radically in substrate and implementation.

The brain anticipates words before they are spoken. It completes phrases from partial information. It compresses familiar idioms into efficient units. It assigns probability to possible continuations of an unfinished sentence.

Tokenization may therefore reveal something that was always true about language — that beneath the felt experience of meaning, there is a mathematical structure of recurring patterns, compressible regularities, and statistical dependencies. The machine makes that structure visible, because the machine has nothing else to work with.

Perhaps humans experience meaning and machines experience probability.

But somewhere between those two descriptions, the same hidden mathematics is operating.

The uncomfortable possibility that follows from all of this: what we call understanding may be less different from very good prediction than we would prefer to believe. Tokens did not create that possibility. They just made it harder to look away.

EisatoponAI

An independent intellectual publication exploring mathematics, AI, science, paradoxes, and the hidden structures behind reality.