The Price of a Question Why this is buildable now

The floor fell out of the price

Run this programme in 2023 and you get one flagship model, one vendor, one rate card, and a bill that scales with how useful the thing is: a funding model where success is what kills you. Every district added is a permanent rise in the monthly invoice, and every health worker who trusts it enough to ask a second question makes it worse. You end up metering, and then deciding whose questions are worth answering.

Three things changed between then and now, and they compound.

Open weights broke the monopoly on the bottom of the ladder

Since 2024 the competent small models have not been one company's product to price. Qwen, Llama, Gemma and Mistral all publish weights under licences that let somebody else run them, and that does two things at once: it drags the hosted rate cards down, because a vendor pricing the cheap end is competing with something anyone can host, and it means the model can eventually be run by somebody other than the vendor: in-country, on a partner's own hardware, and one day on the handset in the field1.

The small end got good

Look at what a frontline worker actually types. A drug name and a dose. A pest on a leaf. What the guidance says about a child under five. Explain this again in the language the class thinks in. A great deal of it is lookup, translation, restatement and arithmetic: work that stopped needing the frontier model some time ago. What it needed was for somebody to check that rather than assume it, which is what Sections IV to VI are about.

And one vendor's ladder now spans an order of magnitude

Grow's service is configured against Alibaba Cloud Model Studio's Qwen ladder: a flagship (qwen3.7-max), a middle rung (plus) and a cost-optimised one (flash), all behind a single key, each with its own published price per million tokens. Every major vendor publishes the same shape of card (Alibaba, Google, OpenAI, Anthropic, Mistral), so the ratio between the top and bottom of a ladder is a thing you can check for yourself in ten minutes, at today's prices rather than the ones we happened to see. On the cards we have priced against, the bottom rung sits at roughly a tenth of the top2.

A tenth is not an optimisation. A tenth is the difference between carrying the cost for a district and carrying it for a country.

What this technology is for out here

Most of the capital in this industry is chasing the replacement of expertise. It is a coherent bet and it is not ours. The framing that comes with it (the model knows better than you, so the model must be enormous, and success looks like fewer people in the loop) inverts almost every fact about the setting we work in.

A farmer with thirty years on the same land is not short of judgement. A nurse who knows every household in the catchment is not short of judgement. A teacher running four year groups in one room is not short of judgement, and would tell you so. What they are short of is a specific fact, in their language, before the moment closes: what is on this leaf, what rate for a plot this size, what the referral guideline says, how to put this so a nine-year-old follows it.

The knowledge nearly always exists somewhere. It just tends to arrive after the planting window shut, after the class moved on, after the child got worse.

That is a much easier problem than the one the industry is funding, and a model that would have been frontier-class two years ago solves most of it. It is also a problem where the value is in arrival rather than brilliance: an answer in ten minutes that is 90% as good as a specialist's beats a specialist's answer at the next scheduled visit, because by then the decision has been taken without either. That is why the cost of a question is not an operational detail here: a tool that is free to ask gets asked the second question, and the second question is where most of the value was.

The bill is denominated in reach

Here is the difference that reorganises the engineering.

A commercial AI product spends tokens against margin. Spend more per query and you make less per customer; the constraint is a gross margin somebody will forgive you for missing this quarter. A programme that has bought a block of capacity up front is somewhere else entirely. The pot is fixed. Every token spent on a question that did not need the flagship is not a smaller margin; it is a question some other health worker does not get to ask.

Tokens are not cost of goods here. They are reach.

Once you see the bill in that currency, decisions that look like penny-pinching turn into fairness questions, and some of them reverse. Sending every turn to the flagship stops being the safe, generous default and becomes a choice to serve fewer people very slightly better. Sending every turn to the cheap rung to stretch the budget stops being thrift and becomes a decision to quietly give somebody a worse answer to a question you cannot see, in a language you may not read, with no way for them to check it.

Neither is obviously right, and neither should be settled by whoever is feeling more confident that week. It should be settled by measurement, which is what the rest of this is about.

The ladder, the rule, and the shadow

Behind one endpoint there is a ladder of models and a rule that picks a rung. The app never learns which one answered: the service reframes whatever the provider streams into a small vocabulary of its own, so swapping a rung, or the entire provider, is a configuration change rather than a release every handset in the field has to install. On a shared phone in a place with intermittent connectivity, being able to change the model without asking anyone to update is the difference between reacting and not.

The rule, in full

Two tests, both structural, both erring upward. A follow-up goes to the flagship: it is where a cheap model has the most to lose, because it has to hold the whole thread, and the failure mode is quietly dropping what was established three turns ago, which is invisible to somebody who cannot check the answer against a specialist. A request longer than 400 characters goes to the flagship too: length is a proxy for somebody pasting something complicated. Everything left over is a short first question, and that is what the cheap rung takes.

That is the whole rule, and those are the real numbers. As this is published the service reports mode static, a flagship of qwen3.7-max, a cheap rung of qwen3.6-flash, a ceiling of 400 characters and a shadow sample rate of zero, which you can read off its own health endpoint rather than taking from us.

Shadow mode, or: measure before you switch

None of this is worth anything if the cheap rung is worse in a way that matters, and there is one honest way to find out: ask it the same questions and compare. Shadow mode answers every user from the flagship, as normal, and on a sampled fraction of turns quietly asks the cheap rung the same thing. Both rows land in the ledger sharing a turn id, so the pair joins. The shadow answer is never shown to anyone and never stored4; what it leaves behind is what it cost, how long it took, and something to compare.

It is sampled rather than universal because on a fixed pot a shadow call is a real answer somebody else does not get. At one turn in twenty the comparison set grows steadily and the bill barely moves; at every turn you pay twice for every question to learn what a fiftieth of them would have told you. The rate in production is currently zero, which is the uncomfortable half of Section VII.

Why there is no clever router

The obvious build, the one suggested in every conversation about this, is a model that reads each question and decides where to send it. Start with the strongest evidence against us, then the case.

Routing works. Ding et al. route with a DeBERTa model over the query alone and make up to 40% fewer calls to the large model with no measured quality drop5. Zhang et al.'s MTRouter, choosing a model per turn, beats GPT-5 on ScienceWorld, 53.8 against 48.4, while cutting total cost by 58.7%7. Neither of those is a straw man, and the second is more ambitious than anything Grow does.

So the question is not whether routers work. It is what happens to one when it meets a district it has never seen, in a language it was not trained on, with no way for anyone there to tell it went wrong.

What happened when a production router met an unfamiliar domain

The same MTRouter paper benchmarks a commercial routing service, OpenRouter's automatic API, on the same tasks. It scored −26.4 on ScienceWorld, against 48.4 for GPT-5 alone and 53.8 for their own router. Not merely worse than the best model: worse than every one of the six models it was choosing between, including the cheapest. The authors' explanation is the part that matters here: it underestimated task difficulty in an unfamiliar domain and over-relied on the lightweight models.

That is the failure this programme has to design against, and it is not a hypothetical we invented. A router mis-reads the difficulty of questions it wasn't trained on, sends them to the cheap model, and returns confident answers nobody flags. Now put that in a clinic where the question is in Bengali and there is no specialist within a day's travel to notice.

1. The savings have to survive the router itself

The JAIR survey of routing strategies states the paradox plainly: routing infrastructure designed to reduce costs "may ultimately consume more resources than it saves in the long term", which it calls a clear case of technical debt, and concludes that operators must evaluate whether the operational savings justify the cost of building, training and maintaining the router6.

Being precise, because this argument is conditional and we have overstated it before: the cheapest routers, similarity and clustering methods, are genuinely low-compute, and the survey credits them with generalising to new candidates. The expensive ones are the trained ones. So the tax is real for exactly the class of router that would be any use to us, and the class that would be cheap is the class that degrades in the regime we operate in. Ding et al. measured that regime: held to about 1% quality loss, their large-gap pair yields a cost advantage on test of 4.4% to 5.1%, and their untransformed routers there do only marginally better than routing at random.

2. The annotation bottleneck, which scales the wrong way

Training a router means labelling: for each query, what each candidate would have answered and how good it was. The survey works the arithmetic through on a public benchmark: 112 candidate models over roughly 16,000 questions is about 1.8 million generations, which then have to be evaluated. And the cost compounds in the direction that matters to us: adding one candidate means generating and evaluating responses across the entire query set. Even similarity-based routing cannot escape it.

MTRouter's authors report what that cost them, and it is the only figure in this literature denominated in the currency this essay uses: 1,291 training instances, 29,693 logged trajectories, 515,221 turns, about $1,620 in one-time collection. That is what one well-resourced team spent to build the thing we are declining to build, before any of it is maintained, and on a benchmark rather than on a district's real questions.

Grow's entire deployment story is that a rung, or the whole provider, is a configuration change, not a release. A component that has to be re-annotated every time we change a model is the wrong shape for that, and $1,620 of inference spent on labelling is $1,620 not spent on answering anybody.

3. A router is a control plane, and a control plane is a target

This one we had not thought of, and it is the survey's: because the router decides where everything goes, it can be attacked directly. Adversaries can craft inputs that consistently trigger selection of the most expensive model, inflating operational cost without necessarily affecting the answers anyone sees. The survey calls this an LLM control plane integrity attack, alongside poisoning public routing benchmarks to plant triggers.

For a commercial service that is a margin problem discovered on an invoice. Here, spend is reach: an attacker who can inflate cost takes questions away from health workers, quietly, without ever producing a wrong answer for anyone to notice.

And our rule does not escape this. It is more trivially gameable than a learned router, not less: anyone can pad a question past 400 characters or send a second turn, and every request lands on the flagship. An earlier draft of this page called that a safe failure because it fails towards the better answer. That was wrong, and it was wrong in the exact way this section has just finished describing: forcing the expensive model is the control plane attack, and the fact that each individual answer stays good is what makes it hard to notice.

What contains it is not the routing rule. It is the per-account daily cap in Section IV, which bounds what any one caller can spend in a day whatever the router decides. That is the honest division of labour: routing decides what a question costs, and the cap decides what an attacker can take. A structural rule is simply easier to reason about when you are working out which of the two is protecting you.

4. Training one means training on what people typed

This is the argument no measurement can touch. A learned router is learned from traffic. Ding et al. train on a corpus of real user instructions, MTRouter on nearly thirty thousand logged agent trajectories. Neither paper raises consent, and for their purposes neither has to. This programme has promised in writing that questions are not training data. That promise is worth more to the people using it than a few percent of a budget is worth to us, and it is not a decision an engineer makes quietly on a Thursday.

And the language argument, stated properly

The version we used to make, that an English keyword list fails in Bengali, is a straw man against a learned router, which reads meaning rather than keywords. The real objection is distributional, and now it has three pieces of published support. Ding et al. measure transfer decaying towards the random baseline as the correlation between model pairs weakens. MTRouter's ablation shows that stripping routing history drops ScienceWorld from 53.8 to 44.6, so the query-only configuration, the one we would be able to build, is the weak one. And OpenRouter's −26.4 is what a production router does in an unfamiliar domain.

The app ships twelve languages today: English, Spanish, French, Portuguese, Arabic, Hindi, Bengali, Urdu, Swahili, Indonesian, Tagalog and Chinese. A router learns the traffic that exists, which at the start is disproportionately English, and the other eleven are the unfamiliar domain. The decay would be invisible, because a question wrongly called easy returns an answer rather than an error.

A rule that fails invisibly on the people with the least recourse is worse than no rule at all.

What the literature says for the boring rule

Three things, none of them ours.

Conservatism is the finding, not the naive position. MTRouter beats the router it is compared against by switching models less: after an error it stays with the same model about 90% of the time on ScienceWorld, against 38% for Router-R1. Its authors are explicit that multi-turn routing is not "switch more".

Switching has a cache cost. The same paper notes that frequent switching lowers prompt-cache hit rates in multi-turn settings and raises the effective cost of serving long histories. Our rule sends every follow-up to the flagship and therefore never switches mid-conversation. We adopted that for a quality reason; it turns out to have a cost reason too.

Per-query routing controls a hard budget only indirectly. LinkedIn's batch-level work makes the point that per-query routing trades cost against quality through a weighting term, which makes strict budget constraints difficult to enforce. Their answer is to optimise over a whole batch under an explicit cap8. A fixed pot is exactly a hard constraint. Our cap is enforced separately from routing, per account and per day, for that reason.

Cache before you route

One thing goes ahead of all of this, and it is why the ladder may end up switched off again.

Any given service has twenty questions that eat most of its week. The same drug, the same pest, the same threshold, asked by different people in the same district within days of each other. A cache hit costs nothing and loses no quality, precisely what a cheaper model cannot promise. So the first script to run against real traffic counts duplicates, not tokens. If the repeat rate is high, caching dominates routing outright and the correct decision is to leave the ladder alone.

What we haven't proved

The programme's standing promise is that you will be told which parts are running, which are prototypes and which are still intentions, including the awkward ones. Applied to this essay, on the day it was published:

The rule is live, and nothing has measured it. An earlier draft of this page said routing shipped switched off. That was wrong: production is running static, so short first turns are already going to the cheap rung. What has not happened is the measurement: the comparison that would say whether those answers are as good.

Shadow mode is not sampling. The rate is zero, so the apparatus described in Section IV is built and idle. Nobody is currently collecting the pairs that would settle it.

No prices are configured, so nothing is costed. The ladder exists to produce a cost per turn and the service is recording none, because the rate card was never set in the deployment. The column is null rather than wrong3, which is the right failure, but it means the saving this page is about is, at this moment, unquantified by us as well as by you.

There is barely any traffic to measure anyway.9 Every projection in this space is a story until it meets a real week of questions from real people, and ours hasn't yet. Some two thousand words of position sit on top of that; treat the ratio as the honest state of things rather than as confidence.

The models are hosted, so asking needs a connection. On-device is where this is going and it is the single biggest thing that would change how far it reaches. It is a goal. This page will not imply it already works.

Cheaper is not the objective. The objective is the largest number of good answers per unit of capacity. If the measurement says the cheap rung is worse in a way that matters, the ladder goes back off and the flagship is simply what this costs, at which point the honest conversation is about funding, not engineering.