What is a token in AI models?

What a token is

A token is the basic unit an AI language model works with. Before a model reads a prompt, a tool called a tokenizer splits the text into tokens. OpenAI's help center says a token can be a character, part of a word, a whole word or punctuation. Spaces and capital letters can change how text is split.

Long words often become several tokens. In an NVIDIA explainer, the word "darkness" splits into "dark" and "ness." Each token is represented by a number. NVIDIA says models also turn images, audio and video into tokens.

How many words is a token?

A token count is not a word count. OpenAI gives rough estimates for English text:

• One token is about 4 characters.
• One token is about three-quarters of a word.
• 100 tokens are about 75 words.

The ratio differs by model and by language. Anthropic's glossary says Claude 4.7 and later models use a newer tokenizer. It produces about 30 percent more tokens for the same text than earlier models.

Why tokens set limits and prices

A model's context window, the amount it can handle at once, is measured in tokens. Many AI services also set their prices by the token. OpenAI says its rate depends on the model and the type of token. Input, cached input and output tokens can have different prices. Cached input is text reused from an earlier request. Reasoning models also produce hidden reasoning tokens, which OpenAI bills as output.

A lower price per token does not always mean a cheaper job. Models can split the same text differently and write different amounts, OpenAI notes.

Meta's Muse Wonder shows the gap. In October 2026, Artificial Analysis, an independent testing firm, listed its price at $4 per million input tokens and $20 per million output tokens. That was 60% below OpenAI's GPT-6 Astra. Yet the firm put Muse Wonder's cost at $4.52 per task on its index, against $3.26 for GPT-6 Astra. It named heavy input use, 7.69 million tokens per task, as one reason.