Cloud Architecture for AI Workloads: Engineering the Cost Curve

Traditional cloud cost management assumes compute demand that scales with users and can be right-sized quarterly. AI workloads violate both assumptions: inference cost scales with usage intensity (tokens, not users — one power user can cost 400x another), and the capacity that matters (GPU or managed-model throughput) is constrained, quota-bound, and priced across a bewildering menu of commitment structures. We've reviewed AI platform bills at dozens of enterprises; the difference between a well-architected and a naive deployment of the same feature set is routinely 3–5x.

The good news: most of that gap is architecture, not negotiation. Here are the decisions that move the number.

Tier the serving path — one model for everything is a tax#

The single biggest cost lever is refusing to send every request to your most capable model. Production traffic is heavily skewed toward requests a smaller model handles perfectly: classification, extraction, routine Q&A, reformatting. A three-tier serving design captures this:

• Tier 0 — no model at all. Deterministic handling for requests that regex, rules, or a lookup can answer. Non-trivial fraction of "AI" traffic in most systems.
• Tier 1 — small model (1–8B, often fine-tuned). The default path. Fast, cheap, and for narrow tasks frequently more accurate than a frontier generalist.
• Tier 2 — frontier model. Reached by escalation: the small model emits a confidence signal (or fails schema validation), and the request retries upward with the same context.

In a customer-operations deployment we run, 22% of traffic terminates at Tier 0, 64% at Tier 1, and 14% reaches Tier 2 — at roughly one-sixth the cost of frontier-for-everything, with quality metrics flat. The prerequisite is an evaluation harness that lets you prove per-route quality; without it, tiering degenerates into guessing.

Caching: the highest-ROI infrastructure you'll deploy this year#

Three cache layers apply to LLM serving, and they compound:

1. Exact-response caching for idempotent requests (same template, same inputs). Hit rates of 15–30% are common on support and search workloads. Trivial to build; hash the canonicalized request.
2. Semantic caching — embed the query, serve a cached response when a prior query is within a similarity threshold. Powerful but dangerous: set thresholds per use case and never semantic-cache anything personalized or entitlement-filtered. We've seen a 0.92-similarity match pair "cancel my subscription" with "cancel my subscription renewal" — different intents, same cache entry, real incident.
3. Prompt/prefix caching — providers and serving stacks discount reused prompt prefixes heavily (often 75–90% off input tokens). This is an architecture decision disguised as a pricing feature: structure prompts so the static bulk (system instructions, schemas, few-shot examples) is a stable prefix and volatile content comes last. Reordering one client's 6K-token prompt so the 4.8K static portion cached cleanly cut their input-token bill by 61% with zero behavior change.

Combined, these layers commonly remove 40–70% of raw inference spend. They are unglamorous and they are the job.

GPU capacity: rent the peak, own the base#

For organizations self-hosting models, the capacity question is a portfolio problem. The pattern that works:

• Committed capacity for the base load. Measure your sustained tokens/second over a month; buy reserved or committed-use capacity for roughly the 60th–70th percentile. Committed GPU pricing runs 30–60% below on-demand.
• On-demand for the peak, with an autoscaler that understands model loading. GPU autoscaling is not stateless autoscaling: a 70B model takes minutes to load and warm. Scale on queue depth and token throughput, not CPU, and keep a warm pool sized to your traffic's rate of change.
• Spot/preemptible for interruptible work only — batch embedding jobs, offline evals, fine-tuning with checkpointing. Never for interactive serving.

Utilization is the metric that keeps executives honest: enterprise GPU clusters we audit typically run at 20–35% utilization before optimization. Continuous batching (vLLM-class serving with paged attention), quantization (a 70B model at FP8 serves ~2x the throughput of FP16 with negligible quality loss on most enterprise tasks), and consolidating teams onto a shared serving platform with per-tenant quotas routinely double effective utilization — which is the same as halving the fleet.

And run the make-vs-buy math honestly, loaded: self-hosting wins at sustained high volume on stable workloads, but the crossover point includes the platform team's payroll, not just the per-token comparison. Below roughly a few billion tokens per month, managed APIs with good caching usually win.

Data gravity and the egress trap#

AI architectures move embarrassing volumes of data: document corpora to embedding jobs, embeddings to vector stores, context to models. Two rules prevent the worst bills. First, co-locate the retrieval path — vector store, reranker, and model endpoint in the same region; a RAG system that crosses regions on every query pays egress per request, forever. Second, embed where the data lives rather than shipping raw corpora across clouds; embeddings are 10–100x smaller than their source documents, so order of operations is a cost decision.

FinOps for tokens: allocate or drown#

Finally, governance. Token spend without allocation becomes "the AI line item," which becomes a freeze. The minimum viable model: every request carries a cost_center and feature tag propagated through the gateway; dashboards report cost per business transaction (per resolved ticket, per processed document — not per token); each feature gets a unit-cost budget with alerts on trend, not spikes. When the conversation shifts from "inference cost $80K last month" to "cost per resolved ticket fell from $0.71 to $0.34 while volume doubled," AI infrastructure stops being a cost problem and starts being an investment with a curve you control.

That's the point of all of this: the cost curve of an AI system is a designed artifact. Tier the serving path, cache aggressively, buy capacity like a portfolio, keep data close, and meter by business unit of work. Teams that do these five things ship more AI, not less — because they can afford to.