Getting a model to do something impressive in a playground takes an afternoon. Getting that same behavior to hold up for real users, at real volume, with real edge cases, is a different job entirely — closer to distributed systems engineering than prompt writing.
Evals before features
The single highest-leverage thing that changes how an AI feature gets built is having an eval set before the first prompt is written — a fixed collection of real inputs with expected outputs or scoring criteria. Without it, every prompt change is a guess judged by vibes; with it, a change is either a measurable improvement or a measurable regression.
Latency is a product decision, not an infra afterthought
Streaming responses token-by-token isn't just a UX nicety — it's what makes a 4-8 second model response feel acceptable instead of broken. Anything that blocks on a full completion before rendering anything needs its own loading state design, not just a spinner bolted on afterward.
The model is the easy 80%. The retries, timeouts, fallback prompts, and rate-limit handling are the hard 20% that actually determines whether the feature survives contact with real traffic.
Cost scales with usage in a way UI features don't
A button click is free. A model call has a per-token cost that compounds with every user, every retry, and every unnecessarily long system prompt. Caching repeated context, trimming prompts aggressively, and picking the smallest model that clears the eval bar are the levers that keep a feature from becoming a line item someone questions in a budget review.
Guardrails for the inputs you didn't expect
Users will paste in malformed JSON, ask the model to ignore its instructions, or send a message in a language the prompt never anticipated. Production AI features need the same defensive posture as any other public-facing input: validate, sanitize, and have an explicit fallback for 'the model returned something we can't parse,' because it will happen.
