In 2023 my team built CourseGPT at a hackathon. It generated whole courses with Llama 2, and early on the demo kept dying because generation requests piled up and got dropped. What got us to second place was boring: a task queue in front of the model, so nothing was lost no matter how much traffic came in.
Two years later I was at Motorq, moving a customer-facing API onto a new event queue with zero downtime allowed. Bigger scale, real customers, and the same shape of problem. Something slow and unreliable sits in the middle, and traffic doesn't care.
That's most of what I mean when I call myself a platform engineer turned AI engineer. The models are new. The discipline around them isn't new at all. You still put a queue in front of the slow thing. You still measure before you trust. You still design for the request that fails.
The one real difference I've run into is testing. A deploy either works or it doesn't. A model answers well most of the time, and "most of the time" is not something a unit test can hold. So evals have slowly replaced tests as the first thing I reach for.