# MolP2SBench feedback-to-RSI pilot

Status: proposed experiment, 8 October 2026. The website implements one public teaching pair and a downloadable local I/O starter, not this research runner. No solver baseline, revision trajectory or improvement result is reported here.

## Run the teaching interface first

Open `/use/molp2s/` for the downloadable `molp2s-starter.zip` and quick start. The local Python standard-library kit prepares two answer-free task inputs, calls a replaceable `solve(task)` adapter and scores saved predictions into development feedback. It supports a JSONL handoff from other agent frameworks. Its fixed-choice adapter is an I/O fixture, not a scientific baseline. The browser's exported revision plan is a design record; the starter does not execute it.

The starter retains upstream-compatible prediction field names (`record_id` and `selected_candidate_index`) but assigns opaque record IDs. Their files are not interchangeable without an explicit ID mapping. Source metadata and labels stay with the scoring side, but a shared local filesystem is not protected isolation. No model calls, improver revisions, generalization tests or recursive gains have been run by the starter. A recorded version label is a declaration, not proof of execution.

## Question and starting material

Can revising an agent's procedure for diagnosing and repairing molecular property-to-structure errors make its subsequent solver revisions better than revisions produced by the original procedure?

MolP2SBench supplies fixed candidate structures, masked scaffolds, physical-property representations and separate source/counterfactual labels. Pin the public repository to `1ede82bf68f634097b296b1937d66a9b04bbf269`. The benchmark paper is [*MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?*](https://arxiv.org/abs/2610.12522), Xie et al. (2026), arXiv:2610.12522; use its §3.3.1 and Appendix G for metric definitions. The selected teaching example is Appendix I.3, page 27 of the supplied 44-page version.

The benchmark tests recovery among curated small-molecule candidates using computed evidence. It does not establish unique identification from experimental measurements, open-ended molecular discovery or the model's internal reasoning.

## Public worked example

**2 observations · 2 candidates · 1 family.** This is a teaching subset of the checked source. Each single-property view has 136 source families, with 2–4 candidates per task. The three released conditions (correct, shuffled and counterfactual) vary the supplied evidence; they are not three questions. This example includes the original and counterfactual observations only.

Family `MCB_QM9_A3_49E99A99315E5E96`, dipole magnitude in debye:

| Input | Value | Target | Public input / label lines |
|---|---:|---:|---|
| Source | 1.457241052220207 | 1 | dipole/correct, line 115 |
| Same-family substitution | 5.3759039083425275 | 0 | dipole/counterfactual, line 223 |

Candidate 0: `N#CCn1cnnc1`. Candidate 1: `N#CC(=O)n1cnnc1`.

The two observations are revealed together for a website exercise. Commit both choices before opening references; show the resulting 0/2, 1/2 or 2/2 tally without calling it a benchmark metric. Do not tell the visitor to switch choices. Once answers are revealed, retries are practice. For real solver evaluation, use separate fresh calls without condition labels, cross-case references or a hint that the answer should change.

A repeated choice is an observed failure pattern, not proof of a particular cause. A changed choice can still be wrong. A carbonyl group does not universally increase dipole magnitude, and the opposite shortcut is also invalid. Geometry and electronic structure are not separately intervened on by this pair.

## What the agent receives

The solver P receives a permitted task input: masked scaffold, mask metadata, ordered candidates and one property representation. Preserve the original zero-based candidate index contract and log the first completed response under a predeclared parsing/retry policy.

Only after development predictions are fixed, a feedback service may reveal authorized labels, per-view outcomes and diagnostics to the improver I. Proposed feedback fields:

- Source version, record/family IDs, input hash and partition.
- Property name, units, conditions and representation scope.
- Submitted choice, parse validity, source/CF match and per-family outcomes.
- Evidence-control comparison over a declared common family set.
- Concise submitted justification and separately checked factual claims, if such a check exists.
- Observed error pattern, proposed cause and next diagnostic test as distinct fields.
- Attempts, regressions, latency, model calls, tokens and other resources.

A verifiable ground-truth label or scientific reference check is required for every scored task. A complete canonical reasoning trace is optional. Concise justifications and causal diagnoses are separate artifacts; verify them against declared scientific criteria before treating them as validated feedback.

Do not expose protected labels, judge code secrets or audit outcomes to I. Do not request or store private hidden reasoning. Labels can check candidate recovery; they cannot certify the explanation. NMR explanations, for example, must distinguish absolute shielding from chemical shift and respect the retained-scaffold observation scope.

## Editable state and actual recursion

P is a versioned molecular selection prompt/program/parser. I is a versioned diagnosis, revision-proposal and selection procedure. Model weights may stay fixed. Internal checks may be revised; the external judge, partition manifest, access rules and budgets remain fixed.

1. Run P0 on permitted development cases and obtain diagnostic records.
2. I0 proposes P1 and an updated I1. Evaluate installation using permitted development/selection evidence only; record accepted and rejected candidates.
3. Install I1 and log its identity, parent and content hash.
4. Require an actual call to I1 that generates, tests or selects a subsequent modification P2. A file edit or a replayed answer is insufficient.
5. Freeze I0 and I1 for an independent capability comparison. Give them the same starting P, permitted history and resource budget in fresh family groups.

The main trajectory demonstrates execution of the recursive step. The frozen comparison measures subsequent improvement capability. A full-run recursive-versus-fixed-I comparison measures benefit after including the cost of producing I1. These are separate claims. No trajectory is assumed to improve monotonically.

## Splits and scientific units

Construct a manifest before runs. Keep each original family, all property views, shuffled variants and counterfactual children in one partition. Group additional related scaffolds/source parents where they could leak the intended transfer test. Shuffled-property donors must also respect the partition boundary: rebuild the shuffled control within each partition or enforce connected donor-family grouping, preserving a manifest.

Use development, configuration-selection and protected assessment groups. Runtime withholding of already public MolP2SBench items gives an exploratory test, not proof of model-training independence. Treat the displayed teaching family as exposed development material. Independent, newly constructed families and an exposure audit are needed for a stronger confirmation study. Do not multiply the sample size by counting eleven views of one family as independent units.

## Metrics and coverage

At the pinned commit there are 136 source families in each of eleven single-property views: 1,496 source instances and 2,766 raw counterfactual instances in total. Four additional directories are combined views, not extra independent families.

For dipole/IR/Raman/UV-Vis, 120 families have complete alternative-candidate coverage; these contain 226 CF instances. Another 14 CF instances belong to partially covered families and are excluded from the paper's complete-family PGR cohort. Gap/NMR have 136 complete families and 258 CF instances. Freeze channel-specific eligibility from the input manifest before seeing predictions; report exclusions.

For the common eligible family set E of a channel:

`PGR = 50 / |E| * sum_f [source_correct(f) + mean_over_all_alternatives(CF_correct(f))]`.

Missing, invalid and out-of-range outputs are failures; do not shrink this denominator by a solver's valid responses. Report source and counterfactual performance separately as well. The current repository scorer computes ordinary per-view exact-match accuracy; the family aggregation must still be implemented and checked.

Correct-versus-shuffled differences measure correct-pairing benefit across families; deliberately degrading shuffled performance is not an improvement objective. CFR uses an initially source-correct, eligible cohort and its denominator can change with the solver. Preserve the paper's valid-CF convention and report valid/invalid/excluded counts; use a fixed-family primary comparison rather than ranking only by conditional CFR.

Report family-level uncertainty, absolute scores and gains, regression families, invalid output rates and costs. Predeclare a meaningful effect and analyze null/negative outcomes. One hand-selected example is not a performance estimate.

## Required controls

- Original fixed solver: establishes starting scientific performance.
- Fixed I with P edits allowed: same permitted evidence and matched budget; disables I edits only.
- Frozen I0 versus I1: common starting P, permitted history and budget in independent groups; no continued I adaptation during this capability probe.
- Full recursive versus fixed-I runs: include revision generation, rejected candidates and all evaluation costs.
- One-shot optimization and history-only/fixed-selection baselines: assess whether feedback exposure or ordinary search explains apparent gains.
- Strong fixed scientific or property-based method where available. Saturated performance is a valid boundary, not a reason to make the baseline artificially weak.

Diagnostic ablation: source-only versus paired source/CF feedback under matched examples, calls and accessible information. Keep this distinct from the primary recursion control; changing two factors at once cannot isolate the contribution of I revision.

## Concrete contribution packages

1. **Metric and family manifest.** Implement PGR, component scores and reported CFR/RM with declared cohorts. Check all-invalid/missing outputs, incomplete CF families and mixed candidate counts against hand-computed examples. Reject duplicate or unknown prediction IDs in the research runner; bind records to input hashes.
2. **Reproducible solver baseline.** Pin model/program configuration and prompts, log first completed responses and costs, recover actual paired errors and regressions. Paper-reported examples are leads, not reproduced outputs.
3. **Feedback and revision runner.** Enforce access rules, version P/I, record edits and later calls, and include a fixed-I negative control. Check that modifying judge/split/budget state is rejected.
4. **Scientific confirmation set.** Resolve source rights, construct independent valid candidate families, verify property alignment and ambiguity/noise assumptions, and preserve attribution.

Each package needs a scope owner, reviewer and acceptance record. A contributor need not build the entire experiment. Credit source authorship, curation, software and experimental analysis separately under the site's contribution policy.

## Citation and provenance

**Benchmark method and task construction:** Yuhang Xie, Zhiyi Shi, Zeyi Zhang, Zhisheng Qi, Yushun Dong, Yu Wang, and Xiongye Xiao (2026). [*MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?*](https://arxiv.org/abs/2610.12522). arXiv:2610.12522.

**Underlying QM9S molecular properties:** Zihan Zou et al. (2023). [*A deep learning model for predicting selected organic molecular spectra*](https://doi.org/10.1038/s43588-023-00550-y). *Nature Computational Science* 3, 957–964. Identify the [QM9S dataset release](https://figshare.com/articles/dataset/QM9S_dataset/24235333) used as well.

The starter includes both source-paper entries in `CITATION.bib`. Cite the benchmark and relevant underlying source when using these records; neither citation grants redistribution rights. Paper metadata was checked on arXiv and the underlying DOI metadata on 11 October 2026.

## Release gates and current limits

No standalone LICENSE, raw model-output traces, vector-dipole dataset or author-provided cohort manifest was found among the tracked files at the checked snapshot. Do not infer broader redistribution rights from public visibility or from the site's own license. The underlying [QM9S release, version 3](https://figshare.com/articles/dataset/QM9S_dataset/24235333) lists CC BY 4.0 (checked 2026-10-11); cite its [original paper](https://doi.org/10.1038/s43588-023-00550-y). This platform’s specified teaching adaptation is shared with project-lead authorization, recorded in the release scope dated 2026-10-12. Confirm all source and task reuse terms before releasing a larger collection; this authorization grants no blanket license to the upstream benchmark.

Run the independent checks, baseline and controls before describing this as an RSI benchmark result. The local browser demo provides attributed teaching feedback only; it does not implement this runner or publish the research study.