For Heads-of · Practitioner
Token
A unit of text that a language model processes, typically a word, subword, or character. Models operate on sequences of tokens, not raw text.
- tokenization
- context length
- language model basics
Tokenization
Raw text is converted into tokens using a tokenizer specific to each model. The same text may tokenize differently across models.
Counting Tokens
Model costs and context length limits are measured in tokens, not words. English text averages 1.3 tokens per word, so a 1000-token limit is roughly 750 words.
Governance Impact
Token limits define the maximum context a model can consider, affecting the scope of information available for decision-making in applications.