Everyone’s talking about what AI can do. Very few are talking about what it costs to actually run it. Not the API call pricing — the real, full-stack cost of operating AI workloads in production.

As a platform engineer, I see this gap every day. Teams get excited about an AI feature in development, then hit a wall when they try to productionize it. Here’s what nobody warns you about.

It’s not just the model cost

The model inference cost — whether you’re calling an API or running self-hosted — is the most visible expense. But it’s often the smallest part of the total cost of ownership. The real costs hide in:

  • Infrastructure overhead — GPU instances, load balancers, caching layers, vector databases
  • Operational complexity — monitoring, alerting, scaling policies, fallback strategies
  • Latency engineering — you need to keep response times acceptable while managing model inference times
  • Data pipeline costs — embeddings, preprocessing, context assembly, RAG infrastructure
  • Reliability engineering — what happens when the model provider goes down? Rate limits? Degraded responses?

The latency tax

AI inference is slow compared to traditional backend services. A typical API call takes 50-200ms. An LLM inference can take 1-10 seconds. This changes everything about your architecture:

  • You need streaming responses for anything user-facing
  • Background processing and async patterns become essential
  • Caching strategies need rethinking — semantic caching is a whole new discipline
  • Your SLOs need to account for fundamentally different performance characteristics

The scaling problem

Traditional services scale horizontally with relatively predictable costs. AI workloads don’t follow the same rules. GPU instances are expensive and hard to get. Auto-scaling is slower. Cold starts are brutal.

If you’re self-hosting models, you’re looking at reserved GPU capacity that sits idle during low-traffic periods. If you’re using APIs, you’re at the mercy of rate limits and provider outages.

What platform teams should do

This isn’t an argument against using AI in production. It’s an argument for being honest about the costs and planning accordingly:

  • Start with API-based models — defer the self-hosting complexity until you have proven value and scale
  • Build the observability first — you need to track cost per request, latency distributions, and error rates from day one
  • Design for graceful degradation — what does your feature do when the AI is unavailable?
  • Set cost budgets, not just compute budgets — treat AI inference cost like a product metric

The honest conversation

AI in production is worth it — for the right use cases, with the right architecture, and with realistic expectations about cost. The teams that succeed are the ones that treat AI workloads with the same rigor they apply to any production system: monitoring, budgets, SLOs, and a clear-eyed view of the trade-offs.

The hype will tell you AI is magic. The production bill will tell you it’s engineering.