Routing

Routing puts a classifier in front of a set of specialized handlers. The first call decides what kind of input this is, and the input goes to a prompt, a model, or a whole workflow built for that kind. Nothing downstream has to cope with every case at once.

I reach for it when the inputs fall into distinct categories that want different handling and a single prompt is getting worse at all of them as it tries to cover each. Support questions that split into refunds, technical problems, and general inquiries are the textbook example. Sending easy questions to a smaller, faster model and hard ones to a larger one is another.

The cost is the classification step itself, both the extra latency and the fact that it can be wrong. A misroute is often worse than a mediocre generic answer, because the specialist path was never built to notice it has the wrong kind of input.

It goes wrong when the categories are fuzzy or the routing rules drift from what the handlers actually expect. I have seen routers with twelve categories where nobody could say what separated four of them. That is not routing; that is a lookup table nobody maintains.

In a good router I look for a small set of categories with a clear default, a classifier I can evaluate against labeled examples, and handlers that fail loudly when the input does not fit. If I cannot measure the routing accuracy separately from the answer quality, I cannot tell which half is broken.