I spent a week measuring reasoning‑effort settings on a production model and kept finding that the interesting result was never the one in the summary table. It was in the spread the summary table had averaged away, or in a parameter nobody had set, or in a row whose units had quietly changed. Below are the five failures I hit, each with the measurement that exposed it. Every number here comes from runs I did myself on 2026‑08‑20 through an OpenAI‑compatible endpoint, and the method is cheap enough that you can reproduce the shape of it in an afternoon with whatever two models you are actually choosing between.

The five, in the order I hit them:

· Running it once. A single run samples the model. It does not measure it.

· Publishing a central figure without the spread. The width is the part that changes decisions, and a mean hides it just as well as a median does.

· Generalising from one prompt. Two adjacent settings swapped places when I changed the prompt, on the same model on consecutive days.

· Leaving a parameter unset. You inherit a default you did not choose, from a vendor that may not enforce its own documented values.

· Letting one row change units. A correct number in a mislabelled row reads as a wrong number, and it takes the rest of the table with it.

1. How many runs do you need to compare two models?

More than one, and the reason is not statistical fussiness. It is that a single run does not measure the model, it samples it.

Here is the same prompt sent to the same model ten times at each of four settings. The task was small and completely deterministic in spirit: write a Python function that merges overlapping intervals, with a docstring.

Output tokens per setting, 10 runs each, showing median and observed range on a log scale

Look at the max row. Ten runs, and the output token count ran from 2,149 to 5,276. That is a factor of 2.5 between the cheapest and the most expensive answer to an identical request. If you run this once against a competitor and once against your incumbent, the winner you announce depends on which end of that range each one happened to land on.

The two cheap settings are tighter, but not tight: low spans 351 to 482 around a median of 424, high spans 432 to 642 around a median of 592. Those two ranges overlap. A single run at low can easily come back larger than a single run at high, which would tell you the exact opposite of what ten runs tell you.

There is no universal magic number for n. What there is, is a rule you can apply without a statistics textbook: run enough to see whether the ranges overlap, then report what you saw. Note which way that cuts. More runs will not pull two overlapping ranges apart, because an observed range only widens as you add runs to it. What they buy you is the difference between a real overlap and two unlucky draws. Ten runs here separated low from high at the median and left their ranges overlapping anyway, and both halves of that sentence belong in the write‑up. If the medians do not separate either, you have a tie, and a tie is a finding rather than a failure to find one.

2. Should you report the mean or the median?

Report the median, and then report the range next to it, because the range is the part that changes decisions.

The usual advice is that the mean is vulnerable to outliers and the median is not, which is true and, in this data, almost irrelevant. Across all four settings the mean and median differed by at most 5.3%. If I had reported means throughout, no headline number in this article would have moved much.

The thing a lone central figure hides is not skew. It is width. "The max setting produces 3,539 output tokens" and "the max setting produces between 2,149 and 5,276 output tokens" are both true, and only the second one tells you whether your cost model needs a safety margin. A budget built on 3,539 is under water half the time by definition.

Latency shows the same shape more sharply. Nine of the ten low runs finished between 8.0 and 11.5 seconds. The tenth took 17.2 seconds. The median barely notices (9.9s), the mean shifts a little (10.5s), and neither of them is the number you need if you are deciding whether this setting is safe behind a user‑facing request. That decision is made by the worst case you observed, which is 17.2 and which vanishes from every summary that reports a single figure.

So: median as the headline, observed min and max beside it, and n stated. Three numbers, one line, no statistics required.

3. Is higher reasoning effort always slower?

No, and I would have bet money the other way.

Wall‑clock latency per setting, showing that low and high overlap while max is a cliff

On this prompt, high ran slower than low at the median, 12.7 seconds against 9.9. That is the direction everyone expects. But the day before, on a different prompt against the same model, I measured the reverse: high finished in 8.6 seconds at the median against 11.6 for low. Same model, same two settings, opposite ordering.

Both measurements are real. What is not real is the general claim either of them appears to support. The ordering of two adjacent effort levels is a property of the prompt, not of the levels, and a benchmark that measured one prompt has established nothing about the other.

The one thing that did hold across both runs is the cliff. Whatever low and high do relative to each other, max is in a different regime entirely: a median of 75.7 seconds here, with a worst case of 124.4. That is the finding worth carrying into a decision. The low‑versus‑high ordering is not.

This generalises past reasoning effort. Any time your result is a small difference between two adjacent settings, ask whether you have measured the settings or measured one prompt. The answer is usually the second one, and the fix is more prompts rather than more runs of the same prompt.

4. What are you actually testing if you don't set your parameters?

Something, but not what you think, and you will not be told.

This is the mistake I would put first if the list were ordered by how often it silently invalidates a comparison. Consider what happens when you simply do not send a reasoning_effort field at all:

Four requests differing only in the reasoning_effort field, with usage blocks showing 465, 2834, 2650 output tokens and one 400

Omitting the parameter costs 6.1x the output tokens of an explicit low on that single pair, and 7.4x at the median across ten runs each. None of which is hidden. The vendor puts max in the default column of its own parameter table, next to the three values it accepts. The point is not that the default is a secret. It is that hardly anyone looks it up, that defaulting to the expensive end is a defensible choice a vendor is entitled to make, and that the vendor you are comparing against may have made the opposite one. Send no effort parameter and you are measuring one company's idea of a sensible default against another company's, then publishing the result as a comparison of models.

It gets worse once you look at which values the parameter will even accept. The vendor documents exactly three: low, high, max, with a note that reasoning cannot be disabled on this model. I sent seven, twice on the same day, and all seven came back HTTP 200 both times: none, minimal, low, medium, high, xhigh, max. A genuinely malformed value ("invalid_value") returned a 400 both times, and the error named the offending value without listing what it would have accepted, so validation exists. It just does not enforce the documented set.

The four undocumented values do not behave the way their names suggest either. These are single requests on a shorter prompt than the one in section 1, so read them against each other rather than against the table above. Against an explicit low at 176 output tokens and an explicit high at 267: minimal returned 1,403, none returned 1,908, xhigh returned 1,957, and medium came back at a median of 2,243 across five runs, spanning 1,658 to 4,568. Every undocumented value landed on the expensive side of high, at eight to thirteen times the cost of low comparing central figures, and medium's slowest run on its own was twenty‑six times. The pass I ran earlier that morning produced different numbers in the same shape, which is section 1 arriving on schedule: treat these multiples as an ordering, not as quantities. Anyone who assumed a three‑tier scale and wrote medium into their harness benchmarked something near the expensive end while believing they had picked a middle setting, and anyone who wrote none to turn reasoning off landed in the expensive cluster instead, on a model whose docs say reasoning cannot be turned off at all.

The answer also depends on when and where you ask. The same medium request had returned a 400 on this route the day before. A reviewer running the same probe on 2026‑08‑20 through a different route to the same model name got a 400 as well, carrying the vendor's own wording: this model always engages in thinking and cannot be disabled; please use low, high, or max. So there are two validation layers here, one that accepts seven values and one that accepts three, and which of them answers you is a property of your route rather than of the model. If you did not record the endpoint alongside the parameter, you cannot tell those two runs apart afterwards.

And then the part that makes all of the above matter less than you would hope: at every setting, all forty responses were correct. Same function, same behaviour, docstring present. The max setting spent eight times the tokens and seven times the wall clock of low and produced nothing this task could distinguish. If your benchmark measures only quality, it will report these four settings as a four‑way tie, which is true and useless. The differences were all in the columns most benchmarks do not print.

The fix is a rule about disclosure rather than about defaults. Every parameter that affects the result goes in the results table, including the ones you did not set. "Default (unset)" is a legitimate row. "We used the defaults" is not a legitimate footnote, because the defaults are not the same across the models you are comparing.

5. Why your results table can be right and still unreadable

A table where every number is correct can still cost you the reader's trust, and it happens through a mechanism worth naming: a row that quietly changes units.

A colleague reviewed a cost table of mine that ran like this. Four rows of output‑side measurements: completion tokens, reasoning tokens, billed output tokens, latency. Then a final row, cost per 1,000 tasks, reading $6.18 and $2.64.

She tried to verify that last row from the ones above it, got $5.69 and $1.58, and came back to say the numbers did not add up.

They did add up. The four measurement rows were output‑side; the cost row was all‑in, input plus output. Every figure in the table was correct. What was missing was one word in a label.

The damage from this is out of proportion to the error, because it is not read as a formatting problem. A reader who cannot reproduce your arithmetic does not conclude that your label was ambiguous. They conclude that your data is wrong, and that verdict lands on every other number in the section, including the ones they did not check.

Two habits fix it permanently. Label the basis in the row itself, so it reads Cost per 1,000 tasks, all‑in. And when a row uses a different basis than its neighbours, put the other figure in a footnote: output only, this would be $5.69 and $1.58. Now a reader arriving from either direction lands on a number that appears in your table, and nobody has to guess which one you meant.

The checklist

Everything above collapses into seven lines you can put at the bottom of any comparison you publish internally. Item 5 is the odd one out: I did not measure cache behaviour for this piece, and it belongs on the list anyway.

1. n, stated. Not "we tested it." A number, per condition.

2. Median as the headline, with the observed min and max beside it. If the ranges overlap, say so in the same breath as the medians, so that nobody reads a gap between medians as a promise about any single run.

3. More than one prompt, or a stated limit on what you measured. If your headline is a small gap between two adjacent settings, you may have measured the prompt rather than the settings. Name the prompts you ran, not just how many times you ran them.

4. The exact parameter set, including what you left unset. Name the defaults you inherited. If you sent a value that is not in the vendor's documented list, say that too, because a 200 response is not confirmation that it did anything.

5. Cache hits, or an explicit note that you did not record them. Cache state moves cost by an order of magnitude between otherwise identical runs, and "we did not record it" is a perfectly respectable line. "We assumed no cache hits" is a claim, and it needs to be labelled as one.

6. One basis per table, or a labelled basis per row. Any all‑in figure sitting among per‑component figures gets the word "all‑in" and a footnote showing the other calculation.

7. The date, and the endpoint. Behaviour changes under a fixed model name, with no announcement. The undocumented medium value I described above returned a 400 one day and a 200 the next on the same route, and on that second day it returned a 400 again through a different route to the same model. A benchmark without a date and an endpoint is a benchmark you cannot re‑run, including by yourself.

None of this requires more infrastructure than a loop and a JSON dump. What it requires is the willingness to publish a table that says "these two overlap and we cannot separate them," which is a more useful sentence than most benchmark conclusions and is almost never the one that gets written.

Disclosure

All measurements were taken through an OpenAI‑compatible gateway rather than against each vendor's first‑party endpoint. Absolute latency figures are therefore route‑dependent and will not transfer to your setup; the ratios and the ranges are the portable part. The model under test was GLM‑5.3, chosen because it exposes an explicit reasoning‑effort parameter, not as a recommendation. Token counts are read from the usage block of each response, with output computed as total_tokens - prompt_tokens because completion_tokens excludes reasoning tokens on some APIs and includes them on others. Run dates are given inline. Checklist item 7 applies to me too, so: the endpoint was¹¹, an OpenAI‑compatible gateway I work on, which is why I had convenient access to several models behind one key. Nothing in this piece depends on which gateway you use, and the same measurements against a first‑party endpoint would tell you the same thing about your own harness.

About the author

Mak is the product lead at¹¹, a gateway that puts models from several vendors behind one OpenAI‑compatible key. Most of the measurements in this piece came out of routine work there: checking whether a price change or a quietly altered default does what the vendor's documentation says it does.