An OpenAI-compatible proxy: a cheap model answers first, and the request escalates to a reasoning model only when five of its answers disagree.
question 1 of 16
The first 16 of 300 test questions, in order. On all 300: 67.3% right for $0.491, against 70.0% for $0.769 when every question goes to Sol (36% less).
An app keeps its OpenAI client and changes two things: where it points, and the model name. The caller’s own key is passed through and never stored; the cache key is a hash of the key plus a hash of the body, so callers never share answers. Streaming and n above 1 get a 4xx before anything is spent, because the cheap answer has to be checked before the user sees it.
# your app, before and after from openai import OpenAI - client = OpenAI()+ client = OpenAI(base_url="http://localhost:8000/v1") raw = client.chat.completions.with_raw_response.create( - model="gpt-6-sol",+ model="auto", messages=[{"role": "user", "content": q}], )
The body is a normal chat completion. Any other model name passes through untouched.
Sending all 300 held-out questions to Sol, which reasons, gets 70.0% for $0.769. Sending them all to Luna costs $0.013 and gets 55.7%. The cascade escalates 128 of them and keeps 67.3% for $0.491: 36% less for 2.7 points. Move the budget to see what each way of splitting the work reaches.
best on its curve (threshold chosen afterwards)accuracy
Bars: the best any threshold could do on these 300 questions, chosen in hindsight. Marks: the thresholds fixed in advance, the cascade at 67.3% for $0.491 and Jev at 68.0% for $0.596. At $0.491 the TF-IDF router’s best is 63.7%. Jev and the default overlap in their confidence intervals, so this data does not rank them.
The first design was a router that reads the prompt and predicts whether Luna will get it right. On the same 300 test questions no prompt-only signal did much better than a coin flip. Whether a question is hard for this model is not written in the question; the model’s own behaviour carries it, as FrugalGPT (Chen et al., 2023) and AutoMix (Madaan et al., 2023) found.
The two best signals came from Luna’s own answers: five samples agreeing 0.66, and Jev checking the answer 0.72, where 0.50 is a coin flip. The full table is in the README.
teal: the judged answer was right · amber: wrong · AUC 0.52 (0.50 is a coin flip)
The dashboard’s “saved” figure is conservative: what all-Sol would cost reuses Luna’s token counts, and Sol, reasoning, would use more.