How much will an AI feature cost to run? A per-request cost model
C.E.O. Bee Engineering18 min read
The cost of an AI feature is the tokens you send at the input rate, plus the tokens the model writes at the output rate, times the calls one user action makes and how often users act. This guide works through that model step by step: what makes the bill grow, when prompt caching really lowers it, which model should do which job, three layers of caps, how to log cost from usage data, a worked example with illustrative numbers and what to do when prices change. Provider facts come from official pages read on November 16, 2026.
How do you work out what an AI feature costs per request?
The short answer: the cost of one request is the tokens you send, priced at the input rate, plus the tokens the model writes back, priced at the output rate; caching changes the price of the part you repeat. Multiply by how many model calls one user action really makes, then by your usage forecast. Little else sets the bill, and almost every lever for lowering it is a lever on one of those terms. Work through it in six steps:
- Count the tokens of one real request, with the provider's own token counter and in your own language.
- Split the input into the fixed part (instructions, tool definitions, reference text) and the variable part (the user's message, retrieved text, conversation history).
- Estimate and cap the output. Output is the expensive side: on the price tables we read, it costs about five to eight times as much per token as input.
- Apply the provider's current rates, from the official price page, with the date you read it.
- Multiply by calls per action. One user click can be one call, or a loop of ten.
- Multiply by users and frequency, then compare with what one active user is worth to you.
| Term | What it is | Where the lever is |
|---|---|---|
| New input tokens | Tokens sent at the ordinary input rate | Shorter prompts, less history, tighter retrieval |
| Cache write tokens | The first time a repeated prefix is stored, often at a premium | Cache only what will be reused |
| Cache read tokens | Repeated prefix served from cache, at a fraction of the input rate | A stable prefix and a lifetime that matches your traffic |
| Output tokens | What the model writes, at the higher output rate | Output limits, concise formats, a smaller model for simple jobs |
| Calls per action | How many requests one user action triggers | Loop limits, fewer retries, one call where one is enough |
This article goes deeper than item 12 of our AI integration production checklist; read that for the other thirteen checks. If you are still deciding whether the feature should be an agent or a fixed workflow, AI agent or workflow comes first, because the choice changes how many calls happen.
What actually makes the bill grow?
Most surprises come from the multiplier, not the rate. The table lists the drivers we look for first and the lever for each.
| Driver | Why it grows | Lever |
|---|---|---|
| Output length | Output is priced above input; long answers and long reasoning cost the most per token | Output limit, a requested format, no preamble |
| Conversation history | A chat resends earlier turns on every request, so input grows with each turn | Trim or summarize history; cache the stable part |
| Tool definitions and instructions | They travel with every request, whether or not they are used | Short definitions; cache them; load tools only when needed |
| Retrieved text | More passages mean more input | Fewer, better passages; a relevance threshold |
| Images and documents | They are converted to tokens as well | Resize, crop or extract text first |
| Agent loops | Each step is a new call that carries the growing context | A step limit, a budget per task |
| Retries and failures | A failed call is still billed if tokens were used | Back off; do not retry blindly |
| Language and content type | The same text can become more or fewer tokens | Count your own text with the model you will use |
Two dated facts from the official pages show why the last row matters. Anthropic's pricing page notes that the tokenizer, the language and the content type all change the token count. Its token-counting guide adds that models from Claude 4.7 on produce roughly 30 percent more tokens for the same text than earlier models, and tells you to recount your prompts against the model you plan to use rather than reuse old counts. The count itself is an estimate, and the endpoint is not billed and has only a request-rate limit (both pages read November 16, 2026). Practically: budget with the counter, bill from the usage data that comes back with each response.
When does prompt caching really lower the cost?
Caching helps when the same beginning of a request is sent again and again, and it helps only if the beginning is truly identical. Both providers describe it as prefix matching. Anthropic's guide says cache hits "require 100 percent identical prompt segments" and orders the prefix as tools, then system, then messages. OpenAI's guide says "cache reuse requires the entire rendered prefix to match." One changed character early in the prompt means the rest is read at the ordinary rate.
That gives a design rule: stable first, variable last.
| Put at the start (stable) | Put at the end (variable) |
|---|---|
| Tool definitions, in a fixed order | The user's message |
| System instructions and policies | Retrieved passages that change per request |
| Examples and reference documents | The current date, time or request number |
| A fixed output format | The user's name or other per-user details |
A timestamp or a user name placed at the top of the system prompt is the most common way to make every request a cache miss. Anthropic's guide also lists changes that invalidate a cached prefix, among them edits to tool definitions, a change to the tool-choice setting and adding or removing images.
Four conditions decide whether caching pays off:
- Minimum length. A prefix shorter than the model's minimum is not cached. Anthropic lists minimums from 512 to 4,096 tokens depending on the model; OpenAI's guide gives 1,024 tokens for GPT-5.6 and later models.
- Lifetime. Anthropic's default is five minutes, refreshed each time the entry is read, with a one-hour option; OpenAI's guide gives a default minimum of 30 minutes for GPT-5.6 and later, extendable to 12 hours. If your traffic arrives less often than the lifetime, you pay to write the cache and never read it.
- The write premium. On Anthropic's page a five-minute write costs 1.25 times the input rate and a one-hour write 2 times; reads cost about a tenth of the input rate on some models and a twentieth or less on others. The page says caching saves money after a small number of cache reads. OpenAI's guide, as we read it, prices reads at about a tenth of the ordinary rate (a twentieth on some models); check its table for the write terms of the model you use.
- Concurrency. A cache entry exists only after the first response has begun. Anthropic advises waiting for the first response before sending parallel requests that share a prefix.
There is also a throughput effect that is easy to miss, and the two providers differ. Anthropic's rate-limit page says cache reads do not count toward the input-tokens-per-minute limit. OpenAI's guide says cached tokens still count toward the tokens-per-minute limit. So caching can raise your effective capacity on one provider and only your margin on the other.
Which model should do which job?
Price per token varies far more between models than between providers. On the pages we read on November 16, 2026, Anthropic lists current models from 0.10 USD per million input tokens (0.50 output) to 10 USD (50 output), and OpenAI lists text models from 0.05 USD input (0.40 output) to 30 USD input (180 output) on its most expensive tier. Prices change, so treat these as the width of the range, not as quotes: the point is that a single routing decision can matter more than any amount of prompt tuning.
A routing design that works in practice:
- Classify the request first, with the smallest model that does it reliably.
- Send routine work to a small or mid-sized model and the hard, rare cases to a larger one.
- Escalate on low confidence, not by default: a cheap first attempt and a defined test for "not good enough".
- Move work that does not need an answer now to batch. Anthropic's page lists a 50 percent discount on both input and output for its batch interface, and Google's page lists a 50 percent cost reduction for its batch API; OpenAI lists separate batch rates.
- Prove the cheaper model with your evaluation set, not with a hunch. A smaller model that fails more often creates retries, escalations and support work; the cost per correct answer, not per token, is the number to watch. The evaluation method is item 3 of the production checklist.
Check the fine print on each page too. Anthropic's page prices one small model higher when the prompt exceeds 100,000 tokens, and counts cache reads and writes toward that length. Data residency options can carry a surcharge: OpenAI lists a 10 percent uplift for regional processing on models released on or after March 5, 2026, and Anthropic lists a 1.1 multiplier for US-only inference on its newer models.
Where do you put the caps?
A cap is not one setting. It is three layers, each stopping a different kind of overrun, and a production feature needs all three.
| Layer | What it limits | Examples |
|---|---|---|
| 1. Per request | The worst single call | Maximum output tokens, maximum input length, history trimming, a step limit on loops, a timeout |
| 2. Per user, session or tenant | One person or customer using too much | A daily token budget per user, a plan-based allowance, a rate limit, an alert on outliers |
| 3. At the provider | The total, if layers 1 and 2 fail | A monthly spend limit, a spend alert, separate keys or projects per feature |
Layer 1. The providers handle the output limit differently, which is why you read both pages. OpenAI's rate-limit guide says the rate limit is calculated as the maximum of the output limit you set and the estimated token count, and advises setting it as close to the expected response size as possible. Anthropic's page says the output limit does not factor into its output-tokens-per-minute calculation.
Layer 2. This layer is your own code. It is also where abuse is stopped: OWASP lists "Unbounded Consumption" among its top ten risks for LLM applications, including attacks designed to inflate the bill. Count each user's tokens from the usage fields, keep a budget, and decide in advance what the product does when it runs out: a clear message, a smaller model, or a queue.
Layer 3. Read exactly what happens at the limit. On Anthropic's rate-limit page, each tier carries a monthly spend cap (500, 1,000 and 200,000 USD for the Start, Build and Scale tiers on the page we read), and you can set a lower limit of your own. Reaching the tier cap returns HTTP 429 with the error code enforced_spend_limit_reached and no retry-after header; the page says retrying, including the SDK's automatic retries, fails until access resumes at 00:00 UTC on the first day of the next month. Reaching a limit you set yourself returns HTTP 400 with invalid_request_error. OpenAI's guide describes two controls: a spend alert that sends a notification while traffic continues, and a hard spend limit that makes affected requests return HTTP 429, applicable to the organization or to projects.
The design consequence is a business decision made in advance: what should your customers see when the cap is reached at night on the twentieth? A hard cap that stops the feature is a failure mode you chose, so build the message, the fallback and the alert thresholds before launch, and check that your error handling distinguishes a spend limit from an ordinary rate limit.
How do you keep the cost visible?
You cannot manage a cost you do not record. Every response carries the token counts; log them per request, next to the feature, the model and the user or tenant.
| Field | Where it comes from |
|---|---|
| New input tokens, cache write tokens, cache read tokens, output tokens | Anthropic: input_tokens, cache_creation_input_tokens, cache_read_input_tokens and the output count in usage. OpenAI: the input count with input_tokens_details.cached_tokens |
| Cost of the request | Computed in your code from the four counts and a rate table you version |
| Feature, model, user or tenant, request id | Your own metadata |
| Cache hit ratio | Read tokens divided by total input tokens |
| Steps taken | A counter in the loop |
Note that on Anthropic, input_tokens counts only the tokens after the last cache breakpoint; the total input is the sum of the three input fields, so a small input_tokens is not a small request.
Three alerts earn their place early: daily spend above forecast, a falling cache hit ratio (a deploy changed the prefix), and loops deeper than the limit. Log counts and metadata, not the prompt text: personal data in logs is item 11 of the checklist. Keep the rates in a versioned configuration file, not in code, so a price change is a one-line edit with a date.
What does a worked example look like?
The numbers below are illustrative. They are not any provider's price; they are chosen to show the shape of the calculation. Assume 3.00 USD per million input tokens, 12.00 per million output tokens, cache reads at a tenth of the input rate and cache writes at 1.25 times.
| Scenario | Tokens per task | Cost per task, no cache | Cost per task, with cache | Per 10,000 tasks, no cache / cache |
|---|---|---|---|---|
| A. Single-turn assistant | 4,000 fixed prefix + 300 new input, 400 output | 0.0177 | 0.0069 | 177 / 69 |
| B. Ten-turn chat | 2,000 fixed prefix, each turn adds 150 user and 250 reply tokens, history resent | 0.1485 | 0.1014 | 1,485 / 1,014 |
| C. Agent task, six calls | 6,000 starting context, +1,500 per step, 300 output per call | 0.1971 | 0.0857 | 1,971 / 857 |
How to read it:
- Scenario A is the best case for caching: a large stable prefix and a small variable part. Writing the 4,000-token prefix costs 0.015 once per cache lifetime; every reuse is read at the lower rate. The 0.0069 in the table is that steady-state read, with the one-time write left out; scenarios B and C include their writes.
- Scenario B shows the history effect. Across ten turns the model reads 39,500 input tokens, so the conversation costs about sixteen times one turn, not ten. Caching only the fixed 2,000 tokens helps, but history is now the larger part; the larger lever is to trim or summarize it.
- Scenario C shows the loop multiplier. Six calls that each carry the growing context cost about nine times one call. Caching the earlier context at each step cuts the figure by more than half here, and a step limit stops the worst case.
Use the same layout with your own counts and the current provider rates. If the per-task cost exceeds what one task is worth to you, the fix is in the design, not the invoice.
What if prices change?
They do, and the official pages say so. Google's pricing page lists scheduled changes with dates: for example, the input price of one of the Flash models rises from 0.75 USD per million tokens through December 31, 2026 to 1.50 USD from January 1, 2027, and two other models move from 2.00 to 4.00 USD on December 1, 2026; the same page lists older models whose requests are routed to another model's rates from December 1, 2026. Not every change is a price: a new tokenizer changes the number of tokens, which changes the cost at the same rate.
Four habits keep a feature safe from this:
- Store rates and read dates in configuration, and rerun the cost model on a schedule.
- Pin the model version and rerun your evaluation set before you move to a new one; item 14 of the checklist covers provider lock-in and ownership.
- Recount tokens when you change models.
- Keep the cap at the provider at a level that reflects the current forecast, and review it when the forecast moves.
How we do it
Here we only repeat what we say on our site. Every engagement is a paid project that starts with a written quote you approve. Scope, schedule and cost are in your hands in writing before anything is billed; nothing is charged until you approve. In AI work, the discovery step covers what you expect from AI, whether it could be solved without it, and what a good outcome looks like; nothing gets built before that picture is clear. Then we write down what is being built, in what order and at what cost, and work starts only after you approve it. Every technical decision is written down with its reasoning. If AI is not the right answer, we tell you so. You can read the details on our AI integration services page and in Process. If you have an AI feature planned or running and want to talk it through, tell us about your project.
Frequently asked questions
How much does it cost to run an AI feature?
It depends on three things you can measure: how many tokens go in and out of each call, how many calls one user action makes, and how often users act. Price those at the provider's current rates, with the cache treatment of the repeated part. The six steps at the top of this article give the method, and the worked example shows the shape of the result.
Does prompt caching always reduce the cost?
No. It reduces the cost when a long, identical beginning is reused within the cache lifetime and the prefix is above the model's minimum length. Writing a cache costs more than ordinary input, so a prefix that is rarely reused can cost more than no caching at all. Measure the cache hit ratio from the usage fields before and after you change anything.
How do I stop a runaway bill?
Use three layers: limits on each request, a budget per user or session, and a spend limit with an alert at the provider. Then test what your product does when the limit is reached; a spend limit returns an error that retries will not fix.
Is a smaller model always cheaper?
Per token, yes. Per correct answer, it depends on how often the smaller model fails your evaluation set and what each failure costs in retries, escalations and support. Decide with your own examples, using the method in the production checklist.
Do I need to count tokens in my own language?
Yes. The token count depends on the tokenizer, the language and the content type, and a newer model can count the same text differently, as Anthropic's token-counting guide notes. Count a sample of your real inputs with the model you plan to use, and recount when you change models.
Sources
The pages below were opened and read on November 16, 2026. None carries a date of its own, and pages that show price, plan or version information change over time; reopen the official pages before you rely on a figure.
- Anthropic, Pricing · Prompt caching · Rate limits · Token counting
- OpenAI, Pricing · Prompt caching · Rate limits
- Google AI Studio, Gemini API pricing
- OWASP Top 10 for LLM Applications 2025, LLM10: Unbounded Consumption
Provider prices in this article come only from the pages above and are shown to illustrate a range; they can change. This article is general information and not legal advice; for personal data sent to a model provider, see items 10 and 11 of our checklist and the sources listed there.
— Related serviceAI Integration