Exact outputs
Small geometric, interface, or parameter errors can make an otherwise plausible design unusable.
← All blogs · Benchmark introduction
Public benchmark · photonic integrated circuits
PhySciBench
A physical-science design benchmark, beginning with photonic integrated circuits.
PhySciBench measures whether an autonomous agent can turn a scientific requirement into a device that survives evaluator-owned checks. The first release covers two complementary modes: open-ended design discovery and reference-grounded implementation optimization.
Evaluator-owned checks
A physical design is not a persuasive answer.
Scientific agents must turn an incomplete functional requirement into executable geometry, use tools under bounded resources, and deliver an artifact that survives checks they do not control. PhySciBench evaluates that complete chain.
Long-horizon work
Progress depends on choosing probes, reading evaluator feedback, and revising within a fixed budget.
Independent evidence
Success is determined by trusted evaluation of the submitted artifact, not by the agent’s explanation.
From scientific intent to verified artifact
Choose the approach from the requirement
Open-ended tasks ask the agent to select a device concept and workflow from functional goals rather than reproduce a fixed parameter set.
Explore design tasks →Implement and improve a defined target
Reference-grounded tasks test faithful implementation and optimization against evaluator-owned performance and artifact checks.
Explore implementation tasks →Evaluation protocol
From open requirement to verified artifact
The benchmark separates what the agent controls from what the evaluator owns. Progress is measured on the submitted device.
Agent-controlledInterpret → design → iterate
Frozen submissionSource + artifact digest
Evaluator-controlledExecute → gate → score
Task contract
Functional requirements, permitted resources, and measurable success criteria define the scientific problem.
Agent design loop
The agent authors the device, uses a bounded probe budget, and decides what to change from evaluator feedback.
Final artifact
One exact final submission is frozen with its run record and provenance digests for inspection.
Trusted verification
Private evaluator checks determine gate eligibility and the official objective J. They also determine the reporting score S when enabled.
Design / Discovery tasks
CodexGPT-5.6-solClaude CodeClaude Opus 5.5
Accepted design · S = 0.90–2.00
Provisional S runs from 0–2; higher is better. N/E means gate-ineligible, while — means no result is published. Select a scored bar for run details.
Implementation / Optimization tasks
At or above baseline parity · S = 1.00–2.00
Implementation S uses each task's declared J_baseline; S = 1 is baseline parity, while native evaluator gates determine eligibility.
Individually approved single-task runs are not official campaign results or task-release certifications.
Early results, bounded conclusions
Finals are score-eligible
The remaining published runs did not clear every trusted gate, so J and S stay null.
Highest reporting S
Microresonator-v1 leads this published snapshot; it is not yet a broad model ranking.
Evaluator probes recorded
Published trajectories expose iterative progress rather than only the final number.
Runs include a visual artifact
Final, checkpoint, or diagnostic visuals are explicitly labelled by their evidence status.
Inspect and reuse
Open evidence, explicit boundaries
Browse tasks and traces, inspect the public result catalog, read the scoring methodology, or cite the current benchmark release.