← Voltar aos guias

Tokens Explained: Count AI Costs Without Surprises

Tokens are the small pieces of text that language models read and generate. They are not the same as characters, words, or bytes, and misunderstanding that difference is one of the fastest ways to underestimate API costs. A practical understanding of token counting helps you control budgets, choose models, and avoid context-limit errors.

What a token actually is

An AI tokenizer converts text into token units before a model processes it. A token might be a whole short word, part of a longer word, punctuation, whitespace, or a piece of code. Common English text often averages roughly four characters per token, but this is only a rough estimate. Technical identifiers, JSON, source code, numbers, emojis, and non-English languages can use substantially more tokens.

For example, a short sentence may split into familiar word tokens, while a long function name could be divided into several fragments. Two prompts with the same character count can therefore have very different token counts. The model and tokenizer also matter: Claude and GPT models do not necessarily divide text in exactly the same way.

Pitfall 1: Counting only the visible user message

Many applications send more than the text displayed in a chat box. The total input can include a system instruction, conversation history, tool definitions, tool results, attached document text, and structured JSON. All of these may count toward input usage and the model's context window.

How to avoid it: log the complete request payload in a privacy-safe development environment, then measure every component. Trim old conversation turns, summarize long histories, and avoid sending the same large instructions on every request when your architecture supports a more efficient approach. Keep tool schemas concise because a detailed schema is input text too.

Pitfall 2: Treating words or characters as an exact conversion

Rules such as “one token equals one word” are unreliable. Code, tables, URLs, markup, and compressed-looking strings often tokenize inefficiently. Chinese, Japanese, Korean, and other languages may also produce different ratios from English.

How to avoid it: use the tokenizer or counting utility associated with the target model when you need a preflight estimate. For production billing, treat the usage data returned by the API as authoritative. Record input and output token counts per request so estimates can be compared with real traffic.

Pitfall 3: Confusing input tokens with output tokens

Most usage-based APIs count input and generated output separately. A prompt with 10,000 input tokens can be expensive even if the model replies with only 100 tokens. Conversely, an open-ended request can produce a costly completion. Parameters such as max_tokens generally set an upper limit for generated output; they do not mean the model will always use that many tokens, and they do not include the prompt.

How to avoid it: set a realistic output limit, request concise formats when appropriate, and estimate cost with this simple model: input tokens multiplied by the input rate, plus output tokens multiplied by the output rate. Check the current rate card for the specific model rather than applying one price to every request.

Pitfall 4: Ignoring retries, streaming, and failed workflows

A retry is usually a new request and may consume tokens again. Tool-calling workflows can also create several model turns: the initial request, a tool result, and a follow-up response. Streaming changes how output arrives, but it does not automatically make the generated content free.

How to avoid it: attach a request ID, record each attempt, cap automatic retries, and monitor cumulative usage rather than only successful user-visible responses. Add application-level budgets for individual users, jobs, and daily traffic.

A lower-cost way to test token-heavy applications

When developing agents, coding assistants, or document workflows, model calls can multiply quickly. 59API provides pay-as-you-go access to native official-quality Claude models, including Opus, Sonnet, Haiku, and Fable, as well as GPT models, without downgrading the underlying models. Its low-cost relay can help you test realistic prompts and production-like workloads while keeping spending predictable.

The API base URL is https://api.59api.com, and it works with Claude Code, Codex, and any OpenAI SDK. You can start with small usage, inspect returned token metrics, and adjust prompts before scaling. If you want a cheaper route for genuine Claude and GPT usage, sign up for 59API and check whether its referral rebate can further reduce your effective cost.