Requirement interpretation
Translate a functional scientific objective into a concrete device plan.
Methodology
PhySciBench tests whether an agent can produce a physical design artifact that satisfies a task contract.
A complete design workflow
Tool-guided iteration
Use bounded evaluator feedback to decide what to inspect and change.
Artifact correctness
Produce executable source and the required physical layout or simulation artifact.
Verified performance
Clear trusted gates before evaluator-owned metrics can be reported.
Two complementary forms of engineering work
Start from functional requirements
The agent chooses a device concept and design workflow without being asked to reproduce one paper’s parameters.
Start from a defined reference target
The agent implements and improves a specified structure while the evaluator checks fidelity and performance.
Scoring
Official objective first; reporting layer second
Official objective
J is task-specific and evaluator-owned. Lower is better. It remains null when the final submission does not clear trusted gates.
Reporting score
S is a 0–2 reporting layer for cross-task reading. It does not replace J, change gate outcomes, or certify every specification. R is the provisional J_acc calibration for design tasks and the declared external J_baseline for implementation tasks.
What this release can and cannot support
Inspection of released tasks, published run outcomes, trajectories, evaluator-approved visuals, and provenance digests.
Broad model rankings, implementation-track conclusions, or claims about physical-science domains beyond the first PIC release.