The most common failure mode when an AI system builds a math model from a plain-English request isn't a bad formula — it's a wrong guess about a detail nobody bothered to state. A new paper argues that operations research assistants built on large language models have exactly this problem: they take a partial description of a scheduling, routing, or blending problem and confidently fill in the blanks rather than asking what's missing. The result can be a perfectly solvable optimization model that answers the wrong question.
That gap exists because almost every benchmark used to evaluate 'LLM writes the optimization model' systems hands the model a complete problem description up front. Real requests from planners and analysts rarely look like that — objectives get left implicit, side constraints get mentioned in a hallway conversation, and business rules like minimum batch sizes or blackout dates go unstated because the requester assumes they're obvious. The new work, called OR-Clarify, is a benchmark built specifically to test whether an AI agent notices this incompleteness and asks about it before committing to a model, rather than after.
OR-Clarify works by taking public optimization problem descriptions and deliberately withholding structured pieces of information — the paper calls these 'hidden slots,' things like a missing capacity limit or an unstated cost coefficient. The agent under test then interacts with a simulated user who can answer questions, but only within a bounded number of exchanges. The benchmark scores four things: how much of the withheld information the agent actually recovers, whether it knows when to stop asking, whether it quietly assumes values it never confirmed, and how expensive (in back-and-forth turns) the whole conversation was. The paper supports both open-ended questions and multiple-choice-style clarification, so it can separate 'agent doesn't know what to ask' from 'agent can't phrase a good question.'
The related literature suggests this isn't a problem unique to optimization. QuestBench, a benchmark for logic and planning tasks, found that even strong LLMs struggle to identify the single right clarifying question as a task gets harder. A separate study on tool-use agents, which introduced the NoisyToolBench benchmark, found that models tend to arbitrarily fill in missing arguments rather than pause to ask — essentially hallucinating assumptions instead of flagging uncertainty. And in software engineering, the Ambig-SWE benchmark found coding agents often can't tell a well-specified instruction from an underspecified one, though they improve substantially in the cases where they do ask. OR-Clarify applies the same diagnostic lens to mathematical programming, where the cost of a silent wrong assumption is a model that optimizes the wrong objective function entirely.
To go with the benchmark, the authors propose InterOPT, a two-stage framework meant to actually solve the ask-versus-assume decision rather than just measure it. The first stage identifies which unresolved details are 'formulation-critical' — meaning the choice of value would change the structure or optimum of the resulting model, as opposed to cosmetic details that don't affect the math. The second stage uses that list of critical gaps to decide, turn by turn, whether to pose another clarifying question or to stop and proceed with formulation. Informally, this reframes optimization modeling as a completeness decision: keep the set of unresolved formulation-critical slots small enough that the model you eventually build is trustworthy, but don't burn the user's patience chasing details that wouldn't have changed the answer.
Framed more formally, if $S$ is the set of hidden slots relevant to a problem and $S_c \subseteq S$ is the subset that is formulation-critical — meaning two different plausible values would lead to different optimal decisions or a different feasible region — then the goal at each turn is to choose an action (ask about a specific slot in $S_c$, or stop) that maximizes expected slot recovery in $S_c$ subject to a bounded number of interaction turns $T$. That's a sequential decision problem under a budget constraint, not a one-shot classification task, which is why the paper treats it as an interactive framework rather than a static filter applied before parsing the request.
The headline result is a split one. In the choice-based setting, where the simulated user answers from a fixed menu of options, InterOPT substantially outperforms all the baselines the authors tested on exact slot recovery — meaning it more reliably nails down the precise missing values rather than approximating them. In the open-ended setting, where the user can answer in free text, InterOPT is only competitive with, not clearly ahead of, strong prior methods, suggesting that turning a vague human answer into an exact structured value is still the harder half of the problem. That asymmetry is worth sitting with: knowing you need to ask is only half the battle if the answer you get back is itself ambiguous.
For practitioners building or buying LLM-based optimization assistants, the practical takeaway is that 'can it write a valid model' is the wrong acceptance test. A more useful one asks whether the tool recognizes what it doesn't know about your business rules and says so before it hands you a solver output. Independent surveys of the broader LLM-for-OR field have already flagged that many existing benchmark datasets carry a surprisingly high rate of errors and implicitly assume complete specifications, which is part of why a purpose-built clarification benchmark like this one fills a real gap rather than a cosmetic one. As LLM-based modeling tools move from research demos into procurement and planning workflows, the ability to ask a good question at the right moment may end up mattering as much as the ability to write a correct constraint.
Sources: Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization, arXiv:2609.05258 · A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions, arXiv:2508.10047 · Large Language Models in Operations Research: Methods, Applications, and Challenges, arXiv:2509.18180 · QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?, arXiv:2503.22674 · Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering, OpenReview:X2yzXtH4wp · Learning to Ask: When LLM Agents Meet Unclear Instruction, arXiv:2409.00557
