What tokens mean in AI

A token is the layer between human text and model math

Type a sentence into a language model and the model does not receive it as a neat row of human words.

First, a tokenizer converts the text into smaller units called tokens. Those units are represented numerically so the model can process them. When the model generates a response, it produces another sequence of tokens that is decoded back into text.

That is the useful definition: a token is one of the encoded units a model reads or produces.

The confusing part is that a token is not a fixed linguistic object. It is not automatically one word, one syllable or one character. OpenAI’s current token guide says a token can represent a character, part of a word, a whole word or punctuation, and that spaces affect how text is divided. Google’s Gemini documentation describes the same basic idea.

So a six-word sentence is not necessarily six tokens.

What tokenization actually does

A tokenizer has a vocabulary of text pieces it can represent.

Common patterns may fit into a single token. A long, rare or unusually spelled word may be split into several pieces. Punctuation can become its own token or be grouped with nearby text. Capitalization and whitespace can change where the boundaries fall.

What you type	What may happen during tokenization
A common short word	It may map to one token.
A long or unusual word	It may split into several subword tokens.
Punctuation	It may stand alone or join another text piece.
Different capitalization or spacing	The token split may change.
Code, URLs or unusual strings	They may break into many smaller pieces.

There is no universal split that applies to every model.

OpenAI’s open-source tiktoken library uses byte-pair encoding, or BPE, for OpenAI model encodings. BPE creates reusable units from recurring byte sequences, allowing common subword patterns to be represented compactly while still handling arbitrary text.

BPE is not the only approach. The 2018 SentencePiece paper describes language-independent subword tokenization learned directly from raw text. Different model families can therefore use different vocabularies and tokenization methods while all still talking about “tokens.”

That is why token boundaries can look odd to a human. The tokenizer is not trying to reproduce a dictionary. It is creating an efficient representation for the model.

One token does not equal one word

For ordinary English, a rough conversion is still useful.

OpenAI currently gives the rule of thumb that one token is about four characters or roughly three-quarters of an English word, with 100 tokens landing around 75 words. Google gives a similar rough range for Gemini: about four characters per token and roughly 60–80 English words per 100 tokens.

Those are estimates, not exchange rates.

The same passage can produce different counts depending on the tokenizer, formatting, punctuation, spelling and language. OpenAI explicitly notes that the same text can count differently across models and encodings, and that other languages can have different relationships between characters, words and tokens.

This is why “one million tokens equals 750,000 words” should be treated as an English-language approximation, not a definition. A million tokens means a million units under the tokenization and accounting rules of the model being used.

The model predicts tokens, not finished sentences

After tokenization, the model works over the resulting sequence.

OpenAI’s tokenizer explanation describes language models as learning statistical relationships between tokens and generating the next token in a sequence. The polished paragraph you see is the decoded result of that process continuing token by token.

This does not mean the model is choosing from a list of complete words. A generated token can be a whole word, a fragment, punctuation or another piece from the model’s vocabulary. Several tokens may be required before a single visible word is complete.

The token is therefore both a representation unit and a generation unit.

Why token counts matter

The term appears everywhere in AI because the same unit is used for more than tokenization.

Tokens measure input and output

Your prompt becomes input tokens. The model produces output tokens.

The full input can be larger than the text you personally typed. OpenAI notes that message structure, tools, schemas, images, files and conversation formatting can contribute to request token counts. In a chat or agent system, visible user text is only one part of the context being sent to the model.

Tokens measure the context window

A model can only work with a limited amount of context at one time, and that capacity is normally expressed in tokens.

Google describes a model’s context window as a token limit covering the information supplied to the model and the output it can generate. OpenAI likewise tells developers to check each model’s context window and maximum-output limits because they differ by model.

The exact accounting is model-specific, but the practical consequence is simple: more context consumes more of a finite token budget. Long conversation histories, documents, system instructions and tool definitions all compete for that space.

Tokens are a billing unit

Many AI APIs also meter usage by tokens, often pricing input and output separately.

Some providers expose additional categories. OpenAI currently distinguishes input, output, cached-input and reasoning tokens in its usage guidance. Google reports categories including input, output, cached, thought and tool-use tokens in parts of the Gemini API.

Those labels are not universal. A usage report of “10,000 tokens” does not tell you the cost until you know which model and which token categories produced them.

It also means visible answer length is not always the full usage story. Supported reasoning models can consume internal reasoning or thought tokens that are not displayed as ordinary answer text. When exact cost matters, the provider’s usage record is more reliable than estimating from the words on screen.

Token price is not a measure of intelligence

Token counts measure how a system encodes and processes a workload. They do not measure how much meaning, difficulty or intelligence is contained in it.

Two models can tokenize the same document differently. They can also generate different amounts of output to solve the same task. OpenAI’s usage guidance explicitly warns that a lower price per million tokens does not necessarily produce a lower total cost because tokenization and generated usage can differ between models.

So comparing models only by “cost per million tokens” can miss the thing that actually matters: how many paid tokens are required to get the result you want.

A better comparison uses the same representative tasks on each model and measures total usage, quality and cost.

Tokens are no longer only about text

The easiest way to understand tokens begins with language, but multimodal AI stretches the term further.

Google’s Gemini documentation counts images, video and audio as tokenized input too, using modality- and model-specific rules. Those tokens should not be interpreted as hidden “words” inside a picture or audio clip. They are units the system uses to represent and account for non-text input.

That makes token a broader model-accounting concept in modern multimodal systems. Text tokens come from language tokenization; image, audio and video token usage follows the rules of the specific model.

The practical rule

Use estimates when you only need a rough sense of scale. Use the actual model’s counting tools when the number matters.

• For prompt size: a four-characters-per-token estimate can be a reasonable English sanity check when the provider publishes a similar rule.
• For context limits: count the full request with the tokenizer or token-counting endpoint for the target model.
• For API cost: separate input, output and any cached or reasoning categories using current provider pricing.
• For model comparisons: run the same real workload instead of converting one provider’s token count directly into another’s.
• For images, audio or video: use that model’s multimodal token rules, not a text-token approximation.

Tokens are simple once they stop being treated as tiny words.

They are the units sitting between human-readable input and model processing — and the same units now help describe how much context a model can handle and how much API usage a system consumed.