01 What is a token, exactly?

A token is the smallest unit of text that a language model actually reads, thinks in, and writes. It is not the same thing as a word, a character, or a sentence. A token can be a whole common word like the, a fragment of a longer word like token + ization, a single punctuation mark, a space, or even part of an emoji.

Every model has a fixed vocabulary — a list of every possible token it knows, often somewhere between 30,000 and 200,000 entries. Before a model can do anything with your prompt, it has to convert your raw text into a sequence of IDs drawn from that vocabulary. This conversion step is called tokenization, and it's the first thing that happens in every single AI text generation request.

As a rough rule of thumb for English text: 1 token ≈ 4 characters ≈ ¾ of a word. So 100 tokens is roughly 75 words. This ratio is why API pricing and context-window limits are measured in tokens, not words or characters.

02 Step one: tokenizing your prompt

When you submit a prompt, a component called the tokenizer breaks your text apart using a sub-word algorithm — most commonly a variant of Byte Pair Encoding (BPE) or SentencePiece. These algorithms are trained ahead of time on huge amounts of text to find the most statistically useful set of chunks: common whole words become single tokens, while rare or compound words get split into smaller, more frequent pieces.

Why sub-word tokenization instead of whole words?

• No out-of-vocabulary problem. Any string, even made-up words or typos, can be represented by breaking it into smaller known pieces, down to individual bytes if necessary.
• Smaller vocabulary, smaller model. A word-level vocabulary would need millions of entries to cover every inflection and language; sub-word tokens keep that number manageable.
• Shared structure across languages. Common roots, prefixes, and suffixes get reused as tokens, which helps the model generalize.

Python — a simplified tokenization call

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("gpt-style-bpe")

text = "How does tokenization work?"
encoded = tokenizer.encode(text)

print(encoded.tokens)
# ['How', ' does', ' token', 'ization', ' work', '?']

print(encoded.ids)
# [4438, 1587, 11241, 1634, 670, 30]

03 Step two: turning tokens into numbers (embeddings)

Integer IDs alone don't carry any meaning — 11241 tells the model nothing about what "token" means or how it relates to "word." So each ID is looked up in an embedding matrix: a giant table where every token in the vocabulary has a corresponding vector of numbers, typically a few hundred to a few thousand dimensions long.

This vector, called an embedding, places the token in a high-dimensional mathematical space where similar or related tokens end up positioned closer together. The model also adds positional information to each embedding so it knows the order the tokens appeared in — without this, "dog bites man" and "man bites dog" would look identical to the model.

Conceptual — embedding lookup

# embedding_matrix has one row per vocabulary token
embedding_matrix = load_embeddings()  # shape: [vocab_size, hidden_dim]

token_ids = [4438, 1587, 11241, 1634, 670, 30]
vectors = [embedding_matrix[id] for id in token_ids]
# each vector is a list of ~4096 floating point numbers

vectors = add_positional_encoding(vectors)

04 Step three: self-attention and building context

This is where most of an AI model's "understanding" actually happens. The vectors flow through dozens of stacked transformer layers, and inside each layer is a mechanism called self-attention.

Self-attention lets every token in the sequence look at every other token and decide how much each one matters for understanding it. In the sentence "The trophy didn't fit in the suitcase because it was too big," attention is what allows the model to figure out that "it" almost certainly refers to "the trophy," not "the suitcase," by weighing the relationships between all the words at once rather than reading left to right like a human.

1. Query, Key, Value projections — each token's vector is transformed into three new vectors used to compute relevance scores against every other token.
2. Attention scores — the model computes how strongly each token should "attend to" every other token in the context.
3. Weighted combination — each token's representation is updated by blending in information from the tokens it attends to most.
4. Stacking layers — this process repeats across many layers (often 30–100+), each layer building a richer, more abstract representation of the whole sequence.

By the final layer, each token's vector no longer represents just that token in isolation — it represents that token in the full context of everything that came before it, which is exactly what's needed to predict what should come next.

05 Step four: predicting the next token

After the final transformer layer, the model takes the vector representing the very last position in the sequence and passes it through one more layer called the language modeling head (or "unembedding" layer). This produces a single number — called a logit — for every single token in the entire vocabulary, representing how strongly the model favors that token as the next one.

Those raw logits are converted into actual probabilities using a function called softmax, which squashes all the numbers so they're positive and sum to exactly 1.0 — a clean probability distribution over the whole vocabulary.

Example — probability distribution for the next token

For the prompt "The cat sat on the", a well-trained model assigns the highest probability to "mat," followed by plausible alternatives, with the vast majority of the vocabulary getting a probability close to zero.

06 Step five: choosing a token (sampling strategies)

Having a probability distribution isn't the same as choosing a token — the model needs a decoding strategy to pick one. This is where generation can be tuned to be more focused or more creative.

Conceptual — sampling the next token

logits = model.forward(context_vectors)        # raw scores per vocab token
logits = logits / temperature                  # reshape distribution
probs  = softmax(logits)
probs  = top_p_filter(probs, p=0.9)         # keep top 90% probability mass

next_token_id = sample(probs)                  # weighted random draw
next_token    = tokenizer.decode([next_token_id])

07 The autoregressive loop, in full

Language models are autoregressive, meaning every new token is generated one at a time, and each new token is immediately appended to the input before predicting the next one. The model never generates a whole sentence in one shot — it writes exactly one token, looks at everything again (now including the token it just wrote), and writes the next one.

1. Tokenize — convert the full prompt into token IDs.
2. Embed — look up each ID's vector and add positional information.
3. Run through the transformer — apply self-attention and feed-forward layers across every layer of the network.
4. Compute next-token probabilities — produce a probability distribution over the entire vocabulary.
5. Sample one token — apply temperature/top-k/top-p to pick the next token.
6. Append and repeat — add that token to the sequence and go back to step 3, now with one more token of context.
7. Stop — halt when a stop token is generated or a maximum length is reached.

This loop is why response speed is often measured in tokens per second, and why longer outputs take proportionally longer to generate — the model is doing a full forward pass through the network for every single token it writes.

08 Worked example: from prompt to output

Take the prompt: "The capital of France is"

1. Tokenize:["The", " capital", " of", " France", " is"] → 5 token IDs.
2. Embed + attend: the model builds a contextual representation where "France" and "capital" strongly inform each other through attention.
3. Predict: the model outputs a distribution heavily weighted toward " Paris" (often >95% probability for a well-trained model on a fact this common).
4. Sample: with low temperature, " Paris" is selected.
5. Append: the sequence becomes "The capital of France is Paris".
6. Repeat: the model now predicts what comes after "Paris" — perhaps a period, or "," depending on context — and continues until it decides the response is complete.

09 How generation actually stops

Generation doesn't run forever. It halts when one of a few conditions is met:

• The model generates a special end-of-sequence (EOS) token that it has learned represents "I'm done."
• The output reaches a maximum token limit set by the API call or application.
• A custom stop sequence defined by the developer (like "\n\n" or a specific string) is produced.

The combination of the model's learned stopping behavior and these external limits is what keeps responses bounded and prevents runaway generation.

11 Glossary of key terms

Token

The smallest unit of text a model reads or writes; a word, sub-word, or symbol.

Tokenizer

The algorithm that converts raw text into a sequence of token IDs, and back again.

Embedding

A numeric vector representation of a token that captures aspects of its meaning.

Self-attention

The mechanism that lets each token weigh and incorporate information from every other token in context.

Logits

The raw, unnormalized scores a model assigns to each possible next token.

Softmax

A function that converts logits into a valid probability distribution summing to 1.

Autoregressive generation

Producing output one token at a time, each conditioned on all previous tokens.

Temperature

A setting that controls how random or deterministic next-token sampling is.

Top-k / Top-p sampling

Strategies that restrict which candidate tokens are eligible for sampling at each step.

12 Frequently asked questions