The first person who reads your benchmark carefully will be trying to break it.
That is not cynicism, it is their job. When we send a number to a quantitative team, it goes to someone whose entire professional value comes from knowing that most published speedups evaporate on contact. They have seen the vendor deck where the baseline was misconfigured. They have run the reproduction that came out 4x slower. They are not going to be persuaded by a bar chart.
I have come to think of this person as the most useful reader we have. Writing for them changed how we build benchmarks, and it turned out that almost everything that makes a benchmark believable comes down to one idea: the two systems you are comparing must differ in exactly one way, and that way must be the thing you claim.
Everything below is that idea, applied five times. None of it is specific to portfolio optimization. If you benchmark databases, inference servers, compilers, or anything else where someone eventually has to trust your number, the same five hold.
Rule 1: Both sides get the same problem
This is the one that quietly destroys the most benchmarks, because it does not look like an error. It looks like a modeling choice.
Most non-trivial problems can be written down in more than one way. Two formulations can be provably equivalent, agree on every optimal solution, and still present completely different amounts of work to the thing that has to solve them. Anyone who has watched a query planner fall off a cliff because a predicate was written one way rather than another has seen the same effect. The formulation is not a neutral container for the problem. It is part of the problem.
So if you hand your own engine one formulation and the comparator a different one, you have not compared two engines. You have compared two formulations, and attributed the difference to the other side's software. The ratio you publish will be large and it will be meaningless, and the first serious reviewer who reads your setup code rather than your results will find it in about ten minutes.
The rule we hold ourselves to is blunt: the comparator gets every advantage our own engine gets. Both sides receive the same formulation of the same problem. If there is a way of stating it that makes the baseline faster, we use that one. If a reviewer can look at how we constructed the comparison and find a way we made the other side's job harder, the benchmark is void, regardless of how the numbers came out.
The payoff is a number you can defend line by line, in front of the one reader who actually checks. That is worth considerably more than a larger number that has to be quietly retired the first time somebody opens the setup code.
How to check it: open your comparator's setup code, not your own. Ask what a specialist in that tool would say if they read it. If you would be embarrassed to have them review it, you already know.
Rule 2: Both clocks start and stop in the same place
Every system does some preparation before the real work begins, and every system does some cleanup after. Where you put the timer around all of that is a decision, and it has to be the same decision for both sides.
The failure mode is subtle because both choices are individually defensible. You can time the core computation only, excluding setup, on the grounds that setup is amortized in production. Or you can time everything from "problem data in hand" to "answer in hand," on the grounds that this is what the calling system actually waits for. Both are legitimate.
What is not legitimate is picking the first boundary for your engine and the second for the comparator. That happens more often than you would think, because the two systems are usually instrumented by different people at different times, and the natural place to put a timer inside your own code is right before the part you are proud of.
We use the outer boundary now, for both sides: wall clock, from the moment the problem data is available to the moment usable output exists. It is the least flattering choice available and it is the only one a caller can verify from outside. It also has a practical benefit. When a client runs the same test in their environment, they measure the outer boundary, because that is the only thing they can see. If you published the inner one, your number and theirs will disagree, and you will spend the pilot explaining a discrepancy instead of demonstrating a product.
How to check it: have someone who did not write either harness draw the boundary on a whiteboard for both systems. If the two drawings are not the same shape, fix it before you look at the numbers again.
Rule 3: The machine is in the same state for both runs
A benchmark on a busy machine measures the machine. Two things go wrong here.
The first is background load. If your run happens on an idle box and the comparator's happens while something else is compiling, you have measured contention. The fix is not to promise you were careful. The fix is to interleave: run A, B, A, B, A, B in a single session and report the ratio within each pair, rather than running all of A on Tuesday and all of B on Thursday. Ratios measured inside one session are robust to machine conditions in a way that timings measured across sessions are simply not.
The second is thread and device budget. Give both sides everything the box has. Benchmarking a multi-threaded baseline on one core is a thing people do accidentally, by leaving a default in place, and it is indistinguishable from cheating in the eyes of a reviewer who spots it.
While you are here: run each configuration several times and report the median rather than the best. The minimum of five runs is a measurement of luck. If you also report the spread, you will find that reviewers relax noticeably, because a stated variance is the signature of someone who actually ran the thing more than once.
How to check it: if you cannot state, from your own logs, how many cores and devices each side had and how many repetitions produced the reported figure, you do not yet have a benchmark. You have an anecdote with a number in it.
Rule 4: Quality gate first, speed second
This is the one that separates benchmarks that matter from benchmarks that entertain.
Any optimizer can be made arbitrarily fast if you stop caring what it returns. So a speed comparison is only meaningful between answers of comparable quality, which means you have to define comparable quality before you look at the timings, and you have to define it in units that mean something to the business.
Two traps here, both of which I have watched catch careful people.
The first is measuring quality as a ratio of objective values. It seems natural: compare your objective to the reference objective, express the difference as a percentage. The problem is that objective functions in this domain can pass close to zero, and a ratio with a near-zero denominator does whatever it likes. You get a 300% gap on a solution that is, in every way anyone cares about, identical to the reference. Or you get a reassuring 0.001% on one that is not. The metric stops carrying information exactly when the problem gets interesting.
Measure the thing the business holds instead. For a portfolio, that is the weights: how far is this allocation from the reference allocation, in units of portfolio weight, and what does that distance cost in tracking error, turnover, and tax. Those numbers are stable, they mean something to a person, and they cannot be flattered by a denominator.
The second trap is subtler. Make sure your reference is genuinely independent of the thing you are evaluating. If the "optimum" you measure against was produced by the same pipeline, the same starting portfolio, or the same intermediate state as the answer you are grading, then the gap you compute is partly a measurement of the pipeline agreeing with itself. Before you trust any quality number, ask what it would read if the system under test did nothing at all. If the answer is "still pretty good," the metric is broken and needs replacing, not interpreting.
How to check it: write down the quality threshold and the metric, in business units, before running anything. Then publish the quality column next to the speed column, always, in the same table. A speedup published without its quality gate is not a result. It is a claim.
Rule 5: Every row stays in
Here is the counterintuitive part, and the part that has done more for our credibility than any individual number.
Every benchmark spans regimes, and different tools are the natural fit in different ones. Small problems where a standard library is exactly the right call. Configurations where two approaches land in the same place. Sizes and constraint sets where the interesting behavior only starts to show. The instinct is to scope the published set down to the regime where the story is cleanest.
Do not. Publish the whole grid at one prominence and let the reader see which regime is which.
The reason is not moral, it is practical. A table with no visible boundary anywhere reads as marketing, and reviewers discount all of it, including the parts that are true. A table where a reader can see exactly which regime suits which tool reads as measurement, because that is what measurement looks like. Once a reviewer trusts that the grid is complete, they extend that trust to every row in it, and one complete table buys credibility for the entire document.
It also works in your favor commercially. Regime boundaries get discovered eventually, usually in a pilot. Publishing yours first turns what could have been a surprise into evidence that you understand your own system better than anyone evaluating it, which is exactly the impression you want going into a technical review.
There is a related habit worth adopting: when a run does not complete, report it as what it was. A timeout is a timeout, a memory failure is a memory failure. Do not drop the row, and do not score it as an infinite speedup. "Did not complete under this configuration within this limit" is a legitimate, informative result, and it is far more useful to a buyer than a blank cell.
How to check it: count the rows in your published table where a different tool is the natural choice. If that count is zero, the table is almost certainly narrower than the problem space it claims to describe.
What this adds up to
None of these rules are clever. They are hygiene. But applying all five at once is unusual enough that doing so becomes a differentiator, which says something slightly depressing about the state of published performance claims.
The practical payoff is that the conversation changes. When the numbers are built this way, a technical evaluation stops being an argument about whether the benchmark is fair and becomes a discussion about whether the capability is useful. That second conversation is the one you actually want, and you cannot get to it while the first one is unresolved.
We build our own published evidence to these rules, and we put the artifacts where they can be checked without us in the room: the methodology and results in the paper (arXiv:2606.23367), the result tables and evaluation harness in the public evaluation repository, and the provenance of every headline figure on our research page. The rows without a completed reference are reported as timing, memory, and failure-status evidence rather than dropped, for exactly the reason in Rule 5.
If you are on the buying side of one of these conversations, the five rules work just as well in reverse. Ask which form of the problem each side received. Ask where the clocks started. Ask what the quality gate was and in what units. Ask to see the complete grid, including the rows that did not complete. A vendor who has answers ready has thought about this. A vendor who has to go and find out has just told you something useful.
I write about optimization at production scale at Asymmetry Computing, where we build PRISM, a real-time optimization engine for finance and energy. This post first appeared on the Asymmetry Computing blog.