There is a category of computation where finishing late is the same as not finishing.
If the batch that produces tomorrow's trades has to be done before the desk arrives, an answer at 09:15 is not a slightly worse answer than one at 03:00. It is a missed cycle. If a grid operator's control loop expects a dispatch decision inside its tick, a decision that arrives after the tick has closed does not get applied at all. The deadline is not a performance target that someone in product invented. It comes from outside the system, it is not negotiable, and the computation either fits inside it or it does not.
Once you are in that category, the mean runtime of your optimizer stops being a useful number, and continuing to quote it becomes actively misleading. I want to lay out why, because I think a lot of teams are specifying these systems with the wrong metric and finding out during a production incident.
The mean is not what fails
Take a concrete shape. You run personalized portfolios: several hundred accounts, each needing new target weights before trading opens. Your optimizer averages 200 milliseconds per account. Five hundred accounts, 200 milliseconds each, 100 seconds of compute. The window is hours wide. This looks like an enormously solved problem.
Then the batch takes forty minutes, or does not finish at all, and everyone is surprised.
The reason is that per-item runtime in optimization is not a tight distribution around its mean. It is skewed, sometimes violently. Most accounts are ordinary and finish fast. A few have an awkward constraint set, an unusual position mix, a near-degenerate configuration, and they take ten or a hundred times the median. Those few govern everything. The batch is not done when the average account is done. It is done when the last account is done.
So the number that decides whether you make your window is not the mean. It is something much closer to the maximum, and the maximum over hundreds of draws from a skewed distribution lives far out in the tail. If you specified the system on its average, you specified it on the one statistic that is guaranteed not to be the binding constraint.
This is why I have stopped accepting "average solve time" as an answer, from vendors or from our own engineers. The questions that carry information are:
- What is the p99 per item, not the mean?
- What is the completion time of the entire queue, measured end to end, for the real item count?
- How many items failed to complete, and what happened to them?
- Does the queue time degrade gracefully as the item count grows, or is there a size where it falls over?
Those four are the specification. Everything else is context.
Averages hide the failures entirely
There is a second problem with reporting a mean, and it is worse than the first.
An average solve time is usually computed over the items that solved. Items that timed out, ran out of memory, or returned an error are not in the numerator, because they never produced a time. So a system that completes 60% of a queue quickly and abandons the rest can report a better average than a system that completes 100% of it steadily. The metric rewards giving up.
I have seen a queue baseline that completed 4 items out of 500 and, on the surviving items, looked perfectly respectable. Every number in that row was accurate. The row was still worthless as a description of whether the workload could be run, because "how fast were the ones that worked" is not a question anyone operationally cares about.
The fix is to report completion as a fraction, always, right next to the timing, and to treat an incomplete run as a first-class result rather than a gap in the table. "487 of 500 completed, 13 timed out at the limit" is a real finding. Reporting only the 487 is not.
Our own published queue study is stated in that form deliberately: 500 accounts out of 500 completed over a 10,000-instrument universe in 109.5 seconds, inside a declared 25-minute operating window, with zero missed deadlines and an audit record for each solve. The interesting parts of that sentence are "500 of 500," "declared window," and "zero missed." The 109.5 is the least load-bearing figure in it. A batch that finishes in 109.5 seconds but silently drops 13 accounts has not done the job, and the number by itself cannot tell you which happened.
The deadline belongs in the interface
Here is the design consequence, and it is the part I would push hardest on if you are specifying one of these systems.
Most optimization interfaces take a problem and return an answer, with runtime as an emergent property. You find out how long it took by waiting. If it takes too long, your options are to wait longer or to kill it, and killing it usually leaves you with nothing at all.
That is the wrong contract for deadline-bounded work. What the calling system actually needs is a guarantee about time, with quality as the thing that flexes. The deadline should be an input, not an outcome. You should be able to say "I need a usable, feasible answer in 400 milliseconds" and get one, along with an honest statement of how good it is.
This matters because of what happens at the boundary. When a deadline passes and you have nothing, the fallback is usually to reuse yesterday's answer, or to skip the cycle entirely. Both of those are much worse than a slightly suboptimal answer delivered on time. In dispatch, holding the previous decision through a price move can cost real money. In rebalancing, skipping the cycle means the drift you were going to correct stays uncorrected for another day. The value of an answer that is 99% as good but arrives is not 99% of the value of the perfect one, it is enormously higher, because the alternative is not the perfect one. The alternative is nothing.
So the questions to ask of any engine in this class:
- Can I hand it a deadline and have it treated as a hard constraint rather than a hint?
- What comes back at the deadline, and is that thing guaranteed to be feasible, meaning it satisfies my actual constraints and can be traded or dispatched as-is?
- Does it report how good that answer is, in units I can act on?
- Is the answer the same every time for the same input, so that a decision can be reproduced later in an audit?
That fourth one gets underrated until the first time somebody has to explain a specific trade to a regulator or a client, at which point it becomes the whole conversation. A system that returns slightly different answers on identical inputs is difficult to defend even when every individual answer was good.
Two shapes of the same problem
It is worth noticing that the tick and the batch are the same problem wearing different clothes.
At one extreme you have real-time dispatch: a single decision, a very short window, repeated constantly. Grid operators re-optimize battery and distributed-energy dispatch on a control tick, and the interesting engineering question is whether a full re-optimization fits inside a window measured in milliseconds while respecting ramp limits and state-of-charge. We publish feasible, audited dispatch on real California ISO data inside deadlines down to about 5 milliseconds, and the reason that figure is stated as a deadline rather than an average is everything I have written above.
At the other extreme you have the overnight batch: thousands of decisions, a window measured in hours, run once. Different scale, identical logic. The window is fixed and external, the tail governs, and partial completion is the failure mode that actually happens.
What unifies them is that in both cases the useful specification is a completion guarantee under a stated deadline, not a throughput number. Once you write the requirement that way, a lot of architectural arguments resolve themselves, because you stop asking "how fast is it" and start asking "what does it promise, and what happens when the promise is stressed."
What to do on Monday
If you own one of these systems, three things are worth doing this week, and none of them require buying anything.
Instrument the queue, not the item. Log wall-clock completion for the whole batch, and the per-item distribution, not just the mean. You will probably discover your tail is worse than anyone believed, because nobody was looking at it.
Add the completion fraction to every performance report. If a dashboard shows average solve time without showing how many solves finished, it is capable of showing green during an outage.
Find your actual deadline and write it down. Not the SLA someone negotiated, the real one: the moment after which the answer is worthless. Most teams have never written it down, and the number is often surprising to the people who thought they knew it.
The rest follows from there. Deadline-bounded systems are not harder than other systems, but they punish being specified with the wrong statistic, and the mean is almost always the wrong statistic.
I write about optimization at production scale at Asymmetry Computing, where we build PRISM, a real-time optimization engine that returns a feasible, auditable answer inside a hard deadline. The queue study above is documented in arXiv:2606.23367, with artifacts in the public evaluation repository. This post first appeared on the Asymmetry Computing blog.