For Heads-of · Practitioner
Token
A unit of text that a language model processes, typically a word, part of a word or a character. Models work on sequences of tokens, not raw text.
- tokenization
- context length
- language model basics
Tokenisation
Text is split into tokens by a tokeniser specific to each model, so the same text can produce a different number of tokens on different models.
Counting tokens
Model costs and context limits are measured in tokens, not words. In English a token averages roughly three quarters of a word, so 1,000 tokens is around 750 words; other languages and code often need more tokens for the same content.
Governance impact
Token limits bound how much a model can consider at once, and token counts drive cost. Uncapped usage is the unbounded resource consumption risk, which inference usage quotas and rate limiting control. Tokenisation also explains some security failures: text that tokenises unusually can slip past filters built around words.
Related Terms
Context Window
The maximum length of text (measured in tokens) that a language model can consider at once when generating a response.
Embedding
A numerical representation of text, images or other data in a high-dimensional space, where items with similar meaning sit close together.
Large Language Model (LLM)
A deep learning model trained on very large amounts of text, able to generate and interpret human-like language across a wide range of topics and tasks.