What does AI really cost? From API token to your own server
APIs bill by the token, a server you own bills in purchase price, electricity and upkeep. What a million tokens actually costs at Anthropic and OpenAI, when owned hardware pays for itself, and the hidden line items that inflate the bill.
AI does not have a single fixed cost. With an API, the bill is driven primarily by tokens: the text sent to the model and the text generated in response. As of August 2026, rates among the major providers range from roughly $0.20 per million input tokens for a budget-tier model to $50 per million output tokens for the heaviest Anthropic model. Systems that send large contexts, produce long responses or retry requests can therefore cost far more than systems using the same model for compact, tightly scoped tasks.
Running AI on your own server shifts the cost model rather than removing it. You pay for hardware upfront, then for electricity, maintenance and operations. An RTX 5090 with 32 GB has an MSRP of around $2,000, but memory shortages have pushed market prices to roughly $3,500-4,000 or more. A used RTX 4090 with 24 GB runs around $1,400-2,500. Comparable cloud GPU capacity, by contrast, costs approximately $0.27-0.56 per hour for 4090-class hardware and $0.56-0.75 per hour for 5090-class hardware. Which option ends up cheaper depends mainly on utilization, token volume, model choice and how predictable the workload is.
What an API token really costs
Language-model APIs generally do not charge per question, document or word — they charge for tokens processed, with a key distinction between input and output. Input is more than the user's visible prompt: system instructions, tool descriptions, retrieved documents, conversation history and RAG context can all contribute to the input-token count. Output is the text the model generates, and at nearly every provider it is noticeably more expensive than input — which makes long generated answers disproportionately costly.
Anthropic's August 2026 pricing per million tokens: Haiku 4.5 costs $1 input and $5 output, Sonnet 5 costs $2 input and $10 output, Opus 5 costs $5 input and $25 output, and Fable 5 costs $10 input and $50 output. One million input tokens plus one million output tokens would come to $6, $12, $30 and $60 respectively — not a typical per-user bill, but a clear illustration of how strongly model choice affects unit economics.
OpenAI's flagship tier sits around $5 per million input tokens and $30 per million output tokens, the mid-tier around $2/$12, and a budget tier around $0.20/$1.20. One million input plus output tokens comes to roughly $35, $14 and $1.40 respectively. If a budget-tier model can complete a task reliably, the model cost can be a small fraction of a heavyweight model's — but the cheapest token rate is not automatically the cheapest solution, since a weaker model may need more retries or extra verification. The more useful metric is therefore not cost per million tokens, but cost per successfully completed task.
What cloud inference costs by usage pattern
Cloud-AI spend is heavily shaped by how the system is used. A short classification request with a tiny output has a fundamentally different cost profile than analysing a large document set and generating a detailed answer. Input-heavy workloads — RAG, contract analysis, document search, assistants that carry long conversation history — see most of their consumption on the front end. Sending more context is not free, and it does not automatically produce a better answer.
Output-heavy workloads have the opposite profile: reports, long-form drafts and detailed analyses can consume substantial output capacity. Because output tokens are priced higher than input tokens, controlling response length can materially change the economics. For high-volume processing of similar jobs, optimizations carry real weight — Anthropic's batch processing is 50% cheaper for work that does not need an immediate response. Prompt caching matters too: Anthropic's prompt caching reduces the cost of cached input by 90%. When a large, largely fixed context block gets reused across many calls, caching can save more than switching to a cheaper model.
The most common budgeting mistake is counting requests instead of tokens. Two thousand short classifications and two thousand large document-analysis jobs can generate the same number of API calls without generating anywhere near the same amount of processing. Without logging that distinction, you won't know at the end of the month which part of the system caused the bill.
What on-premises or owned hardware costs
Local inference removes the external provider's per-token meter, but it doesn't make inference free — it converts the cost into infrastructure spend. An RTX 5090 has 32 GB of memory; its official MSRP is around $2,000, while market pricing in August 2026 is frequently around $3,500-4,000 or more because of memory shortages. A used RTX 4090 with 24 GB runs around $1,400-2,500.
The GPU is not the whole server: CPU, RAM, storage, power delivery, cooling and a chassis all belong in the picture, so a calculation that only looks at GPU price understates the real cost of a complete local AI server. Then comes electricity — average Dutch electricity prices in August 2026 run around €0.26-0.29 per kWh including tax, calculated directly as power draw in kW × running hours × price per kWh. And there is maintenance and operations: hardware can fail, software needs updates, and business deployments also need monitoring, access control, backups and security management.
The economic advantage of on-premises AI does not come from local tokens being "free." It comes from spreading already-paid-for hardware costs over a large enough volume of useful inference. A server that mostly sits idle is expensive capacity; a server with consistently high utilization can become genuinely economical. See also when on-prem AI is worth the investment for the broader trade-off beyond the raw math.
The break-even point between cloud and your own server
There is no universal token-based break-even point — two crucial variables are missing: how much inference a specific local setup can process per unit of time, and how much of that capacity actually gets used. What you can do is weigh GPU purchase price against the rental cost of comparable cloud capacity.
For 4090-class hardware, cloud rental runs around $0.27-0.56 per hour, against a used purchase price of roughly $1,400-2,500. Looking only at purchase versus rental, the arithmetic break-even range works out to roughly 2,500 to 9,259 usage hours. For 5090-class hardware, cloud rental runs around $0.56-0.75 per hour: at the official $2,000 MSRP that works out to around 2,667-3,571 rental hours, and at market prices of $3,500-4,000 it shifts to around 4,667-7,143 hours. In both cases, electricity, the rest of the server, maintenance and operations are still not included — this is not a full TCO calculation, just a way to show how heavily utilization weighs on the outcome.
Comparing against an API is harder still, because an API is not bare GPU time but a fully managed model service. A fair comparison weighs cumulative API cost for the same workload against purchase price plus electricity plus maintenance plus operational management of the local infrastructure — and model quality needs to be comparable too. A cheaper local model is not automatically the better economic choice if it makes more mistakes on the same task or requires extra checking.
Hidden costs: context, retries, embeddings and storage
Token rates are visible; several other costs grow more quietly. The first is context: a RAG system may retrieve several text fragments for every question and pass them to the model as input. If a system routinely retrieves more context than it needs, you pay for that excess not just in latency but in tokens. Conversation history behaves the same way — if the full prior conversation is resent with every new question, input grows over the course of a session, so an apparently simple tenth question can carry far more billable input than the first.
Retries are a second hidden cost: if a process calls a model again because the first output was invalid, incomplete or insufficiently reliable, tokens and inference capacity get consumed again — in agentic workflows this effect can be larger still, since a single user request can trigger several model calls. Many RAG architectures also need embeddings: documents get processed, split up and stored as vector representations, with the compute and storage that requires. And storage itself is easy to forget — source files, vector data, metadata, logs, caches and backups all take up space, and on-premises means you manage that storage yourself too.
A practical decision framework
The choice between an API, a rented cloud GPU and owned hardware gets easier once you treat it as a capacity question rather than a technology preference. Start with the workload: if usage is occasional, highly variable or hard to predict, cloud has a structural advantage because capacity only needs to be available when it's actually used. Then look at utilization — if AI is used heavily and predictably every day, owned capacity becomes more attractive, because the hardware cost gets spread over many inference hours.
Next comes the model requirement: not every workload needs the same model class, and routing — simple tasks to cheap models, hard tasks to heavier ones — can matter more than optimizing around a single provider's headline price. Analyse context usage: if large fixed prompts or the same documents get reused repeatedly, caching can change the calculation substantially. Include the full on-premises cost — not just the GPU, but the rest of the server, electricity, maintenance and operational overhead. And make reliability part of the cost model: an inference step that costs one cent but regularly needs to be rerun or corrected by hand can, in practice, be more expensive than a costlier but consistently reliable model call. The most useful unit is therefore not cost per token, but cost per reliably completed task.
Frequently asked questions
Is running your own AI server always cheaper than using an API? No. An owned server has fixed hardware, electricity, maintenance and operational costs. With low or irregular usage, an API can be cheaper because you only pay for what you actually consume. Owned hardware becomes economically attractive mainly when capacity is consistently well utilized.
Where is the break-even point between a cloud GPU and an owned GPU? Based purely on GPU purchase price versus GPU rental, the comparison for RTX 4090-class hardware works out to roughly 2,500 to 9,259 rental hours. For an RTX 5090, that is around 2,667-3,571 hours at the $2,000 MSRP, and around 4,667-7,143 hours at market prices of $3,500-4,000. Electricity, the rest of the server, maintenance and operations are not yet included in that.
What is the most commonly underestimated AI cost? Often it is unnecessary processing: too much context sent along, output that runs too long, or repeated model calls caused by retries. In RAG and agentic systems especially, a single user request can trigger multiple inference steps. Optimizing context, caching, retries and model choice can matter more than simply chasing the lowest price per million tokens.
Conclusion
AI cost starts with tokens but doesn't end there. APIs are attractive when demand is variable and operational simplicity matters. Owned hardware becomes more compelling as workloads grow stable and utilization rises, and cloud GPUs sit in between: dedicated compute without the upfront hardware purchase. Looking only at the published price per million tokens, or the purchase price of a single GPU, misses much of the real bill. Context, output length, retries, caching, batch processing, storage, electricity and operations ultimately determine what AI really costs.