AI Token Counting: A 10-Minute Developer Quick Start
What are tokens?
Tokens are the chunks of information a model processes. For text, a token might represent a whole word, part of a word, punctuation, or whitespace. The model’s tokenizer converts your text into token IDs before processing it. Tokens are not the same as words, characters, or bytes.
For typical English prose, one token is roughly four characters, or about three-quarters of a word. That makes 1,000 tokens approximately 750 words. Treat this as a planning shortcut, not a billing calculator: code, JSON, URLs, and non-English text can produce very different ratios.
1. Identify everything that gets counted
A request usually has two main token totals: input tokens, which the model receives, and output tokens, which it generates. Neither total is necessarily limited to the text visible in your chat interface.
- Input: system instructions, user messages, conversation history, tool definitions, tool results, and provider-specific message formatting.
- Output: generated answers and tool-call arguments. Some models also bill reasoning tokens that are not exposed as ordinary answer text.
- Multimodal content: images, audio, and video can consume tokens or use separate billing units. Text-length estimates do not cover them.
If your application resends a conversation on every turn, that history counts again on each request. Prompt caching may change its price, but cached context is not automatically free or removed from the context window.
2. Estimate before sending a request
Start by selecting the exact model. Different model families can tokenize identical text differently, so there is no universal exact token count.
- For GPT text: use a tokenizer that supports your selected model. In Python, OpenAI’s tiktoken library provides encoding_for_model(model_name); encoding the text and taking the resulting list’s length gives its text-token count.
- For Claude: use Anthropic’s token-counting endpoint, POST /v1/messages/count_tokens, when your provider or relay exposes it. Submit the intended model, messages, system content, and tools.
- For either: do not ask the chatbot itself to count tokens. Its written estimate is not authoritative usage data.
A local tokenizer count for one string is not necessarily the complete request count. Message wrappers and tool schemas add overhead. If a relay does not expose a counting endpoint, use local estimates for planning and actual response usage for verification.
3. Read actual usage from the API response
Make a small test request and inspect the returned usage object. OpenAI Chat Completions commonly reports usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens. The OpenAI Responses API uses input_tokens and output_tokens instead. Claude Messages reports usage.input_tokens and usage.output_tokens, with additional cache-related fields when applicable.
For streaming Chat Completions, request stream_options with include_usage set to true where supported, then capture the final usage-bearing chunk. Do not count streamed text fragments as tokens: network chunks and tokenizer boundaries are different things.
Log the model, request ID, usage fields, and billed cost when available. Compare your estimate against returned usage before scaling traffic.
4. Calculate cost and reserve context space
For a simple uncached text request, the calculation is: cost = input tokens × input rate + output tokens × output rate, with rates converted to a per-token basis.
For example, at hypothetical prices of $1 per million input tokens and $4 per million output tokens, 2,000 input tokens plus 500 output tokens costs $0.004. These are illustrative rates, not a 59API price quote. Apply separate cache, reasoning, or media charges where relevant.
Also reserve room for the answer. A model’s context window must accommodate the applicable input and generation budget, subject to model-specific rules and output limits. Reduce unnecessary history, shorten tool results, and set an appropriate output cap rather than filling the window with input.
5. Keep the workflow affordable with 59API
59API offers low-cost, pay-as-you-go access to Claude and GPT models through https://api.59api.com, with native official-quality models and no downgrading. Its compatibility with Claude Code, Codex, and OpenAI SDKs makes it practical for existing developer workflows.
Compare the current per-model rates, verify usage reporting with a small request, and monitor real costs rather than word counts. If that fits your workflow, consider signing up for 59API; its referral rebate offers another potential saving.