AI tokens explained: context limits and the cost of a request
Neaptide · September 6, 2026 · 6 min read
Learn what fills an AI context window, how to calculate API token costs, and why a short follow-up can cost more than a long first message.
On this page
You send “fix the last paragraph” and usage jumps. The app may also be sending the conversation, a document and its own instructions. Counting only your latest message misses most of the request.
A token is not a word
A token is a unit a model uses to process input and generate output. In text, it may represent a character, part of a word or a whole word. There is no fixed words-to-tokens conversion: the tokenizer and the text matter. Use a counter for the model you plan to use.
Context is capacity for a single request
The context window limits how much a model can use in one request. OpenAI documents a shared budget for input, output and reasoning tokens. Check both the context limit and the output limit for your model. A large window is not a promise that every detail in a long document will be handled correctly.
Our fictional window holds 2,000 tokens: 1,200 for input, 600 reserved for output and 200 spare. An actual 300-token answer does not mean all 600 reserved tokens were generated. Real APIs may also account for reasoning and other usage categories.
What else goes into the input?
- Application instructions and the conversation history it includes.
- Document text, retrieved passages and tool results.
- Images and other inputs, counted under the provider’s rules.
The application decides what it sends. OpenAI explicitly notes that using previous_response_id does not make earlier input tokens free. A short follow-up in a long thread can therefore consume more input than a longer opening message.
A worked cost example: $0.0048
Use fictional rates of $2 per million input tokens and $8 per million output tokens. These are teaching numbers, not live prices. This example excludes tools, caching and other charges.
| Category | Tokens | Calculation | Cost, USD |
|---|---|---|---|
| Input | 1,200 | 1,200 × 2 / 1,000,000 | 0.0024 |
| Output | 300 | 300 × 8 / 1,000,000 | 0.0024 |
| Total | 1,500 | 0.0024 + 0.0024 | 0.0048 |
A thousand identical requests would cost $4.80. Three attempts with exactly the same usage would cost $0.0144 per task. Real retries often differ: history grows, answers change and tools run. Add the actual costs of every attempt, including failed ones.
Measure before and after
- Before sending, count the full input using the provider’s counter for the chosen model.
- Afterward, inspect usage data instead of asking the model to guess its token count.
- Match usage categories to the pricing rules and charges. Visible answer length may not represent all usage.
- Group requests by task to see the cost of the completed outcome.
OpenAI provides an API endpoint for counting input tokens before a request. Gemini also documents counting before a request and the usage figures returned with a response. Check which fields your API version provides and what each one includes.
Cut waste without losing the brief
Remove duplicates, superseded documents and irrelevant passages first. Ask for the output you need: five errors with supporting quotes, for example, rather than a full retelling. Keep constraints, exceptions and evidence.
To continue a long conversation in a new chat, prepare a summary of the decisions, constraints, unanswered questions and source links. Check it against the original conversation before using it. Losing an important condition can create more work than the shorter input saves.
Compare the cost of accepted work, including review and rework. A cheap call that leaves someone repairing the whole answer is not necessarily a cheap task.