AI Wiki · Mediocre Security
Concept

Token

The chunk of text (roughly a word or word-piece) a model reads and writes; pricing and context limits are measured in tokens.

A token is the unit of text a language model processes: often a word, part of a word or a punctuation mark (roughly 0.75 English words on average). API prices are quoted per million input and output tokens (for example Claude Haiku 5.5 at $0.10/$0.50), and a model’s context window is measured in tokens. Token efficiency is now a competitive metric: Reflection claims about 25K tokens per Terminal-Bench attempt versus 80K for GLM-5.2, and Vercel agent-desktop cut a task from 198K to 31K tokens.

Connections

Token and everything one link away. Drag to move, scroll to zoom, click a node to open it.Open in the full graph →

What links here

Concept Context window · Large language model (LLM) · Mixture of experts (MoE)

Sources

  1. Wikipedia, “Lexical analysis”