MolP2SBench leaderboard
Compare seven models across eleven property channels, using results reported in MolP2SBench v1.
This table reproduces the paper’s results. New score submissions are not open.
Property-Grounded Recovery
Within each eligible family, original recovery and mean counterfactual recovery each contribute half. PGR averages these family scores.
Ranks use the paper’s rounded point estimates; confidence intervals are not shown. See the paper’s family-subsampling analysis.
Showing Dipole magnitude: 7 models, ranked by PGR.
Recover a masked fragment using the molecule’s computed dipole magnitude.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 69.4 percent |
| 2 | GPT-5.6-SolClosed weights | 64.6 percent |
| 3 | GPT-5.6-LunaClosed weights | 62.2 percent |
| 4 | Claude Opus 5Closed weights | 59.7 percent |
| 5 | Gemma-4-31B-itOpen weights | 45.5 percent |
| 6 | Qwen3.8-27BOpen weights | 44.2 percent |
| 7 | ChemDFM-v2.0-14BOpen weights | 37.5 percent |
Recover a masked fragment using the computed energy gap between frontier orbitals.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 72.3 percent |
| 2 | GPT-5.6-SolClosed weights | 69.0 percent |
| 3 | Claude Opus 5Closed weights | 55.0 percent |
| 4 | GPT-5.6-LunaClosed weights | 47.1 percent |
| 5 | Gemma-4-31B-itOpen weights | 37.9 percent |
| 6 | ChemDFM-v2.0-14BOpen weights | 37.4 percent |
| 7 | Qwen3.8-27BOpen weights | 35.9 percent |
Recover a masked fragment using computed electronic excitation data.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 80.0 percent |
| 2 | GPT-5.6-SolClosed weights | 73.2 percent |
| 3 | Claude Opus 5Closed weights | 66.1 percent |
| 4 | GPT-5.6-LunaClosed weights | 58.4 percent |
| 5 | Gemma-4-31B-itOpen weights | 46.0 percent |
| 6 | Qwen3.8-27BOpen weights | 38.6 percent |
| 7 | ChemDFM-v2.0-14BOpen weights | 35.6 percent |
Recover a masked fragment using a computed infrared peak list.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 93.1 percent |
| 2 | GPT-5.6-SolClosed weights | 90.7 percent |
| 3 | Claude Opus 5Closed weights | 88.5 percent |
| 4 | GPT-5.6-LunaClosed weights | 82.4 percent |
| 5 | Gemma-4-31B-itOpen weights | 68.6 percent |
| 6 | Qwen3.8-27BOpen weights | 58.2 percent |
| 7 | ChemDFM-v2.0-14BOpen weights | 38.9 percent |
Recover a masked fragment using a computed Raman peak list.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 74.7 percent |
| 2 | GPT-5.6-SolClosed weights | 72.1 percent |
| 3 | Claude Opus 5Closed weights | 71.5 percent |
| 4 | GPT-5.6-LunaClosed weights | 66.5 percent |
| 5 | Gemma-4-31B-itOpen weights | 55.0 percent |
| 6 | Qwen3.8-27BOpen weights | 49.5 percent |
| 7 | ChemDFM-v2.0-14BOpen weights | 38.1 percent |
Recover a masked fragment using computed hydrogen shielding peaks from the visible scaffold.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 70.6 percent |
| 2 | Claude Opus 5Closed weights | 55.4 percent |
| 3 | GPT-5.6-SolClosed weights | 55.2 percent |
| 4 | GPT-5.6-LunaClosed weights | 38.4 percent |
| 5 | Gemma-4-31B-itOpen weights | 37.8 percent |
| 6 | ChemDFM-v2.0-14BOpen weights | 36.9 percent |
| 7 | Qwen3.8-27BOpen weights | 35.5 percent |
Recover a masked fragment using computed scaffold hydrogen shielding peaks with integration information.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 71.2 percent |
| 2 | GPT-5.6-SolClosed weights | 57.5 percent |
| 3 | Claude Opus 5Closed weights | 55.0 percent |
| 4 | GPT-5.6-LunaClosed weights | 43.8 percent |
| 5 | ChemDFM-v2.0-14BOpen weights | 36.9 percent |
| 6 | Gemma-4-31B-itOpen weights | 36.1 percent |
| 7 | Qwen3.8-27BOpen weights | 36.0 percent |
Recover a masked fragment using computed hydrogen shieldings assigned to atoms in the visible scaffold.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 64.6 percent |
| 2 | Claude Opus 5Closed weights | 58.8 percent |
| 3 | GPT-5.6-SolClosed weights | 55.6 percent |
| 4 | GPT-5.6-LunaClosed weights | 40.2 percent |
| 5 | ChemDFM-v2.0-14BOpen weights | 40.1 percent |
| 6 | Qwen3.8-27BOpen weights | 38.0 percent |
| 7 | Gemma-4-31B-itOpen weights | 37.8 percent |
Recover a masked fragment using computed carbon shielding peaks from the visible scaffold.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 68.7 percent |
| 2 | Claude Opus 5Closed weights | 68.0 percent |
| 3 | GPT-5.6-SolClosed weights | 58.1 percent |
| 4 | GPT-5.6-LunaClosed weights | 44.9 percent |
| 5 | Qwen3.8-27BOpen weights | 37.7 percent |
| 6 | Gemma-4-31B-itOpen weights | 36.9 percent |
| 7 | ChemDFM-v2.0-14BOpen weights | 35.6 percent |
Recover a masked fragment using computed scaffold carbon shielding peaks with degeneracy information.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | GPT-6-AstraClosed weights | 69.3 percent |
| 2 | Claude Opus 5Closed weights | 62.5 percent |
| 3 | GPT-5.6-SolClosed weights | 58.1 percent |
| 4 | GPT-5.6-LunaClosed weights | 41.9 percent |
| 5 | Qwen3.8-27BOpen weights | 37.0 percent |
| 6 | Gemma-4-31B-itOpen weights | 36.9 percent |
| 7 | ChemDFM-v2.0-14BOpen weights | 36.0 percent |
Recover a masked fragment using computed carbon shieldings assigned to atoms in the visible scaffold.
| Rank | Model | PGR (%) Higher is better |
|---|---|---|
| 1 | Claude Opus 5Closed weights | 69.3 percent |
| 2 | GPT-6-AstraClosed weights | 68.1 percent |
| 3 | GPT-5.6-SolClosed weights | 56.3 percent |
| 4 | GPT-5.6-LunaClosed weights | 47.2 percent |
| 5 | Qwen3.8-27BOpen weights | 39.5 percent |
| 6 | Gemma-4-31B-itOpen weights | 38.6 percent |
| 7 | ChemDFM-v2.0-14BOpen weights | 36.0 percent |
Paper-reported results · arXiv v1, Table 2
Download all scores (CSV)Scoring and evaluation
PGR gives equal weight to original and counterfactual recovery. For each molecular family, score the original question and average the scores of all alternative-candidate substitutions. Average these two quantities, then average across eligible families.
Models share the same input-validated, complete-coverage families within each channel. Invalid terminal answers count as failures. The paper evaluates models without tools, retrieval, task-specific training or solved examples.
Ranks follow the reported one-decimal point estimates; matching values share a rank. Small differences require uncertainty analysis. Each channel has its own ranking and a fixed 0–100% scale.
These are the source paper’s model evaluations. For RSI studies, record how a revised improvement process is reused. Compare paired results on independent cases under matched budgets, including a fixed-improver control. Evaluation protocol
From a score to a better solver
Use the benchmark’s controlled cases to investigate weak points, then test a revision on fresh molecular families.