AI did not remove the critical path
Modern AI applications can involve retrieval, model calls, tool execution, databases and several network boundaries in a single user request. That makes end-to-end performance engineering more important, not less.
A fast model call can still produce a slow product if every surrounding service adds queueing, serialization, retries or unnecessary data movement.
Measure the whole request
The most useful optimization work starts by decomposing latency and cost rather than guessing.
- Time to first useful token or result
- Retrieval and reranking latency
- Model inference/API latency and retries
- Database and external tool calls
- p95 and p99 latency, not only the average
- Cost per request and cost per successful outcome
Classic systems techniques still apply
Caching, batching, concurrency control, avoiding duplicated work, reducing serialization overhead and separating hot paths from background work are still fundamental. The difference is that model calls can be comparatively expensive and variable, so inefficient orchestration is easier to feel in both latency and cloud cost.
Optimize after the product works — but instrument before
Premature optimization is still a trap. The practical approach is to ship a clear architecture with instrumentation, watch how real workloads behave, and then optimize the bottlenecks that matter. That is the same discipline used in performance-critical C++ systems, applied to a newer stack.