A vendor sends you a performance number. The number came out of a system you cannot look inside, on problems the vendor selected, measured by the vendor's own instrumentation, and published by the one party whose revenue depends on how it came out. You cannot reproduce it and you cannot falsify it.

The correct thing to do with that number is to discount it to zero.

I run one of those vendors, and I still think that is the right response. A claim that cannot be checked is not weak evidence, it is not evidence. Treating it as slightly persuasive is worse than ignoring it, because a buyer who is 10% persuaded by unverifiable claims is a buyer that rewards whoever is most confident rather than whoever is most correct.

The usual conclusion is that this tension has no resolution. The vendor cannot open the engine, the buyer cannot verify the closed one, so the vendor publishes marketing and the buyer learns to ignore all of it, including the parts that happen to be true. Everyone accepts the equilibrium and complains about it.

The tension is resolvable. Not by opening the implementation, and not by asking anyone to take a number on faith. You resolve it by publishing an evaluation boundary instead of an implementation, and then arranging things so that every claim you make is checkable on the far side of that boundary by someone who has never seen your code.

What a system does, and how it does it

Two different questions get collapsed into one, and separating them is most of the work.

The first is what a system does: given this problem, with these constraints, under this deadline, what comes back, how good is it, how long did it take, how much memory did it need, and what happens when it cannot do the job. The second is how the system does it: the internal method, the data structures, the engineering that makes the first set of answers come out the way they do.

A buyer has a total and legitimate need for the first. A buyer has no need at all for the second, and I mean that literally rather than defensively. Suppose I handed you a complete description of the method. You would still have to run it on your book to know whether it meets your deadline, still have to check whether its answers satisfy your constraint set, still have to compare its output against your current baseline. The method description would not have removed a single one of those tasks. It would have added a large reading assignment to a decision that is ultimately empirical.

This is not unusual outside software. Nobody audits the fabrication process before qualifying a component. You get a datasheet that states behavior under stated conditions, and then you test the part in your circuit. The datasheet is a boundary, and the boundary is what carries the obligation.

So write the boundary down as a contract. On the input side: the problem data, the full constraint set, and the deadline. On the output side, every one of the following, for every row, always:

  • the returned answer itself, in full, as the numbers a downstream system would actually consume
  • a status code, including for the runs that did not succeed, with the failure category named
  • wall-clock timing measured at the outer boundary, from problem data in hand to usable output in hand
  • the memory class the run required, so a buyer can tell whether it fits their hardware
  • enough information for an independent feasibility check against the stated constraints
  • an objective value computed in a form the buyer can recompute from the returned answer
  • an audit record for the individual solve, so the run can be re-examined months later

That is the whole interface. Nothing on that list reveals a method, and nothing a method reveals would improve that list.

The part that makes this work: verification happens on the output

Here is the move that converts the boundary from a nice idea into actual evidence, and it is the one most vendors miss.

The verification must be performed on the output, by tools the buyer controls. Not attested by the vendor. Not certified by a third party the vendor hired. Performed by the buyer, in the buyer's own language, on the numbers the buyer received.

Feasibility is the clean example, because feasibility is arithmetic. If an engine returns portfolio weights, the checks are: do they sum to the budget, is every position inside its bound, are the sector exposures under their caps, is turnover below the limit, is realized tracking error under the constraint, are the names on the restricted list absent. That is a short script. A competent quant writes it in an afternoon, with their own tolerances, and from that moment the vendor's feasibility claim is irrelevant, because the buyer has their own. A vendor who says "our solutions are feasible" has said nothing. A vendor who returns weights that pass a checker the buyer wrote has proven something.

Objective quality works the same way. The buyer recomputes the objective from the returned weights, using the buyer's own risk model, and compares it against a reference solution from whatever they trust. Once two answers are established as equal in objective within a stated margin, a timing comparison between them is meaningful and stays meaningful no matter what either system did internally. This is why our own speed claims are stated against certified-equal objectives rather than in isolation: on an operationally constrained real-data suite we report 3.4x to 126.7x faster than the best completing incumbent at certified-equal objectives, and the second half of that sentence is what makes the first half checkable rather than decorative.

Outcome claims can be built the same way when the outcome is arithmetic on the output. On a $5M direct-indexing book with real US equity data, we report the full $238K harvestable tax budget captured. That is not an opinion about the engine. It is a number you recompute by taking the returned trade list, applying it to the lot table, summing the realized losses, and comparing against the budget. If the recomputation disagrees, the claim is dead, and it dies in public.

Independent checkability of outputs is what converts a claim into evidence. Source access is neither necessary for that nor sufficient, and I would go further: for most buyers, source access is the less useful of the two, because you can read a repository for a week and still not know what it does on your data.

What a vendor has to publish for this to be real

The boundary only means something if the material to exercise it is public. The checklist we hold ourselves to:

  • The exact problem instances, or a generator that reproduces them from a stated seed, so a third party can construct the identical inputs.
  • The constraint set in full, in machine-readable form, not prose. Half-stated constraints are the most common way a benchmark becomes unreproducible while looking complete.
  • A runnable harness that a third party executes end to end without the vendor present, in the room or on a call.
  • A status code for every row, including every failure, with the category named: timed out at the stated limit, exceeded memory, returned infeasible, errored.
  • Timings at the outer wall-clock boundary for both sides, with the repetition count and the spread stated.
  • Every row in the grid, including the ones where a different tool is the natural choice and the ones that did not complete, at the same prominence as the rest.

That last one costs the most and buys the most. Rows without a completed reference we report as feasibility, timing, memory and failure-status evidence rather than dropping them or scoring them as an infinite speedup, which is the detail a careful reader checks first. I wrote about the construction of the comparison itself in inside the benchmark, and about why the same input has to produce the same output, with a record, in determinism and reproducibility.

What stays closed, and what that honestly costs

The implementation stays closed. Nothing in the checklist above requires it, and no item on the buyer's decision list is answered by it.

That said, publishing a detailed performance specification is a real decision and it deserves to be made deliberately rather than drifted into. The discipline that works is to settle in advance which class of information belongs in public, write the line down, and hold it, rather than relitigating it deal by deal under time pressure. Done once and done properly, the published boundary stops being a thing you worry about and becomes the asset the rest of this post describes: the reason a buyer takes your first meeting seriously.

What is not defensible is the version where intellectual property becomes a blanket reason to publish nothing checkable at all. If a vendor cannot tell you what their system returns, at what quality, in what time, on stated instances, with the failures included, they are not protecting an engine. They are avoiding measurement, and the two look identical from outside right up until a pilot.

The pilot is the arbiter, and public evidence never replaces it

Everything above establishes one thing: whether a vendor is worth a pilot. It does not establish that the vendor will work for you, and a vendor who implies otherwise is overselling their own artifacts.

The reason is generalization. A published instance family is not your book. Your constraint set has items nobody publishes, because they came from a compliance memo in 2019 and a client agreement nobody wants to reopen. The only thing that settles the question is a matched-workload pilot, and the word matched is doing all the work: your universe, your constraints including the ugly ones, your risk model, your baseline as tuned by the people who tune it, your deadline, run by your engineers on a machine you control, against acceptance criteria written down before the first run. I have described the mechanics of one in more detail in the matched-workload pilot.

One thing worth requiring of any vendor before that pilot starts: make them state, in writing, where they expect to win and where they expect to lose. A boundary claim is falsifiable in a week. In our case the boundary is written into the paper rather than left for a buyer to find on their own: the universe sizes we have measured, the constraint sets we have measured them under, the quality targets we held, and the rows where no reference completed at all. A buyer whose workload sits outside that measured range should say so in the first meeting, and they will get told which part we know and which part we would be guessing at, before anyone has spent a week on it. A vendor who cannot say where their measurements stop has not looked, and a vendor who says it and then survives the pilot has earned something no benchmark table can buy. The rest of what to require from an engine at this level, independent of who supplies it, I have set out in what makes an optimization engine institutional grade.

Where ours sits

For whatever it is worth as a worked example of the pattern rather than as a pitch: the methodology and results are in the paper, the instances, harness and result tables are in the public evaluation repository, the direct-indexing case has its own evaluation repository, and every headline figure we publish is traced back to a recorded run on our research page. None of that includes the engine. All of it can be checked without me in the room, which is the only property that matters.

If you are buying, the useful change is to move this from something you hope a vendor volunteers to something your process requires. Put the evaluation boundary in the request itself: state the instances, state the constraint set, state where the clock starts and stops, and require the failure rows. Ask for the harness before the demo, not after.

And there is one question I would now ask any vendor, including mine, in the first meeting: what could I run, without you, that would prove you wrong? A vendor who has an answer ready has already built the thing this whole post is about. A vendor who does not has just told you where the number came from.

I write about optimization at production scale at Asymmetry Computing, where we build PRISM, a real-time optimization engine for finance and energy, and publish the evaluation boundary rather than the implementation. This post first appeared on the Asymmetry Computing blog.