FDE-Bench

FM Agent Team · Baidu AI Cloud

FDE-Bench

Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

Research question

Can an AI agent turn an incomplete customer request into a solution the customer would accept?

We test the entire journey—uncovering requirements, building a solution, and delivering against real business standards.

Read the paper Code & cases
6.12%Highest delivery success rate
across 18 evaluated systems
The delivery challenge End-to-end
Starting point

An incomplete business request

Initial brief + real business data

AI agentBuild · run · revise
ClarifyBusiness rules
CustomerSimulated from project context
12-hour solving budget≤ 30 clarification rounds
Final artifactDoes the solution meet the standard?
Hidden constraints + quality evaluationFDE reference 1.0
49
Real customer projects29 optimization · 20 machine learning
266
Clarification targetsFrom customer–FDE discussions
18
Agent configurations7 model families · 6 scaffolds
3×
Runs per case & system147 planned runs per configuration

01 / Main evaluation

The gap to real delivery.

High validity does not guarantee customer-level quality. Even the strongest evaluated systems remain well below the customer-accepted FDE reference.

0.5510

Highest mean QualityCodex + GPT-6-Astra · reference = 1.0

6.12%

Highest delivery success rateOpenHands + Qwen3.8-Max

8/49

Cases with any reference-level runAcross all 18 evaluated systems

Leaderboard

Interact-Req · 49 cases · 3 runs per case

Native scaffoldOpenCode · OpenHands

Quality averages the three scores within each case, then weights all 49 cases equally. 1.0 is the customer-accepted FDE solution—an achieved project standard.

DSR is the share of runs reaching that reference. pass@3 / pass^3 measure cases with at least one / all three valid runs. Validity and reference-level delivery are different outcomes.

Reported point estimates from Table 1 ↗. The highest Quality and highest DSR come from different configurations.

02 / How it works

Clarify the request.
Then prove the delivery.

A forward-deployed engineer uncovers the rules that make a solution useful. FDE-Bench gives agents the same starting point: an incomplete request and the customer’s data.

During solving

Agent-visible workspace
01 / Inputs

Request + business data

instruction.mddata/

Original level of detail, consistently de-identified.

02 / Clarify & optimize

Agent ↔ simulated customer

Ask about missing rules. Implement, execute locally, inspect outputs, and revise.

Build → Run → Revise ↺
12 hours · Up to 30 clarification rounds
03 / Submit

Final deliverable

The agent submits its artifact and the interaction log is retained.

Submission boundary

No hidden score or evaluator feedback during solving

Outside the solving workspace

Held-out business context & evaluation
Customer context

Complete business information

information.md

The simulated customer answers from the request, context and dialogue. It does not see the clarification targets.

Clarification scoring

Requirement coverage

gt.json

Five judges assess whether questions could elicit each target; coverage requires a 3-of-5 majority.

Delivery scoring

Constraints + quality

evaluator + FDE reference

Post-submission extraction preserves the agent’s decisions. Hard-constraint violations score 0; valid artifacts receive normalized quality.

Two outputs, two questions.Clarification asks whether the agent sought the right information. Delivery asks whether its final solution actually satisfies the business requirements.
View the paper’s complete evaluation framework
Evaluation framework showing customer interactions, artifact extraction, case evaluation and requirement-coverage assessment.
Figure 3. Open the image for a full-resolution view.

03 / What the experiments reveal

Information helps.
Delivery remains hard.

Giving an agent more business information substantially improves its solution. Complete information still leaves every compared system below the FDE reference on average.

How much does clarification recover?

20 paired cases · 3 runs per condition
Case-macro delivery Quality

Information ablation · Table 2 ↗
Quality 0FDE reference 1.0

HiddenRequest + data, no customer answers

Interact-ReqRequest + data, required clarification

FullComplete business information, no interaction

01

Coverage is associated with quality.

Across 18 systems, Recall and Ask-F1 correlate with Quality (ρ = 0.83 / 0.88). Precision does not track delivery metrics. These observational comparisons do not establish causality.

02

Feasibility is a persistent failure mode.

Of 548 zero-score runs, 78.5% record constraint or feasibility violations, while 1.5% record schema errors. Well-formed files can still break business rules.

03

A few missing rules can matter a lot.

In the four-case oracle study, mean Quality rises from 0.059 → 0.352 → 0.869 at 0%, 50% and 100% answer coverage. The last quarter of answers yields half the total gain.

Explore the oracle information-coverage experiment
Oracle information coverage for Codex and GPT-6-Astra: four case plots and their macro mean show nonlinear quality gains with increasing coverage.
Figure 5. Codex + GPT-6-Astra; four cases. At 100% coverage, agents receive all registered answers, not the complete business-information file used in Full.

A real failure trace / Case 10

Asking about a rule
does not ensure compliance.

Retail replenishment
The question

Use local stock first?

The agent asks whether same-city stock must be used before buying from suppliers.

The local check

Validator: pass

Its own validator accepts the plan, but the deliverable still purchases an item with 327 units remaining locally.

The hidden evaluation

8 stores in violation

The submitted plan breaks the priority rule. The final run receives Quality 0.

Paper-reported run: OpenCode + GPT-6-Astra. This is a summary of the trace, not a verbatim transcript. Section 5.3 and appendix ↗

04 / Built from real customer work

A request is not
a specification.

Every case connects an initial customer request, the ensuing project discussions, and a customer-accepted FDE solution. Retained projects required more than one month of FDE work.

Inside a warehouse project

“One product type” sounds clear.

Keep each stack to one product type where possible, while avoiding unnecessary stock moves. But what exactly counts as a mixed stack?

The initial request leaves a business rule unstated.

Illustrative example adapted from Figure 1—not a recorded agent run.

From customer project to benchmark case

Inputs are de-identified and reviewed by an FDE. The original engineer distills requirements that cannot be resolved from the visible input and could change feasibility, quality, or customer acceptance. Reviewers run the reference solution, check the evaluator, and exclude inconsistent cases.

View the construction pipeline ↗

5.43clarification targets
per case on average

1.0accepted FDE reference
not a claim of global optimality

05 / The case collection

49 projects.
Real acceptance criteria.

Manufacturing, logistics, energy, retail, finance, and engineering. Browse the computational objective and the constraints that determine whether an artifact is acceptable.

06 / Use FDE-Bench

Bring your agent.
Test the full delivery loop.

The evaluation harness and 49 de-identified case packages support new agent adapters, controlled information conditions, and repeated runs.

Start with one case.

  1. Get the benchmark.Clone the repository and download the case data following the README.
  2. Configure your endpoints.Set the model, simulated-customer and scoring routes in your private configuration.
  3. Run and evaluate.Select an agent, case and information condition. Keep hidden evaluation outside the solving workspace.
Setup & documentation ↗Download case data · v0.1.0 ↗
TerminalAfter installing the documented dependencies
# Discover agents and information conditions
python -m harness --list

# Preview an evaluation before running it
python -m harness \
  -a opencode \
  -m YOUR_PROVIDER/YOUR_MODEL \
  -c Interact-Req \
-d 002_city_delivery_route_planning \
  -k 1 \
  --dry-run

Actual model runs require your configured endpoints and credentials. See the repository for installation, data download, isolation, and scoring instructions.

07 / Contribute

Help us grow the case collection.

We welcome pull requests. Contribute a new, de-identified customer case to a future FDE-Bench release, improve an evaluator, or add an agent adapter.

Open a pull request ↗Read the case format ↗

What a new case should include

  • instruction.md with the original level of detail
  • visible data plus complete requirements and clarification targets
  • a deterministic evaluator, schema, reference solution, and environment
  • documentation of provenance, de-identification, and data rights

Keep credentials and private customer material out of pull requests. Start from the existing case structure and read the repository's release notes before sharing data.

Evaluation reliability & scope

Checked by experts.
Bounded by the setting.

Two experts independently audited stratified subsets across all 18 configurations. These checks support reliability on the audited subsets; they do not eliminate extraction-related score risk.

1,303

Simulated-customer answers

Experts marked 95.93% and 90.48% as supported; their support labels agreed on 92.10% of records.

120

Artifact extractions

93.33% and 91.00% of assessable extractions were judged faithful. The sample was enriched for runs with extraction issues.

120

Clarification items

The model panel agreed with experts on 94.74% and 93.33%, excluding uncertain labels.

What this benchmark does not yet cover

Requirements and business data are fixed. The simulated customer does not revise expectations in response to intermediate solutions. The 49 optimization and ML projects do not cover every FDE setting.

Paper & team

FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

Read the paper ↗

* Equal contribution.   † Corresponding author.

Cite this work
@misc{li2026fdebench,
  title = {FDE-Bench: Evaluating End-to-End Delivery from
           Underspecified Real-World Business Requests},
  author = {Li, Huaiming and Huang, Can and others},
  year = {2026}
}