What is Token?
The fundamental unit of text that large language models process. A token is roughly 3/4 of a word in English: "hamburger" becomes ["ham", "burger"], while short words like "the" are single tokens. Tokens determine context window limits, API costs, and processing speed. Understanding tokenization helps explain why models sometimes struggle with spelling, counting, and non-English languages.
My perspective
In practice
Tokens are the currency of AI. Every API call is priced per token (input and output separately), every context window is measured in tokens, and every model has a maximum token throughput. For practical purposes: 1,000 tokens is roughly 750 English words. When designing AI workflows, token awareness helps you estimate costs, manage context windows efficiently, and understand why the model sometimes behaves unexpectedly with unusual text.