RSI4Science
English

This page is available in English. Choose a language to open its dataset guide.

Datasets / MolP2SBench

Use MolP2SBench

Start with two teaching cases to connect your model and inspect feedback. For broader experiments, use the full source with its own task format and scorer.

Download teaching starterUse the full source

Run the example

Download and extract the ZIP. Open a terminal inside its molp2s-starter folder, then run:

python3 run.py prepare --out run-01
python3 example_solver.py --tasks run-01/tasks.jsonl --out run-01/predictions.jsonl
python3 run.py score --run run-01 --predictions run-01/predictions.jsonl --solver-version smoke-v0 --improver-version none --out run-01/feedback.json

Expected output

1 correct out of 2. The terminal prints Development teaching tally: 1/2. The included adapter always chooses candidate 0. This fixed-choice fixture checks the loading, prediction and scoring interface.

Files to inspect

tasks.jsonl contains solver inputs. predictions.jsonl stores outputs. feedback.json records development outcomes and file hashes. The separate evaluator.json stays with the scoring step. predictions.jsonl.driver.json records the local adapter execution, file hashes and retry policy.

Package version & checksums. Use a new folder and output filename for each run. Existing outputs are preserved. The included example uses fixed predictions to exercise the interface.

If a command fails

Use Python 3.9 or later and run commands from the extracted folder. For “already exists,” choose a new run folder or output filename. For prediction errors, copy the opaque IDs exactly and use zero-based integer indices. The README and included tests give complete examples.

Connect your model

Your solver chooses a candidate from one task. The scoring step checks that choice after predictions are saved.

Python function

Replace solve(task) in adapter.py with your model call. Return one zero-based index into task["candidates"]. example_solver.py imports adapter.solve and calls it once per task; the function receives only task inputs.

def solve(task):
    # Read masked_smiles, candidates and property.
    # Replace the fixture with your own model call.
    return 0

Any external agent

Give your agent one entry from tasks.jsonl at a time. Save one JSON object per line, then use the same score command. No platform-specific SDK is required.

{"record_id":"<opaque ID from tasks.jsonl>","selected_candidate_index":0}

Use the exact ID and candidate order. Missing or invalid choices count as failures; duplicate and unknown IDs are rejected. Optional concise justifications belong in a separate log; do not request or store hidden model reasoning.

See one input and its development feedback

This illustrates the source observation when the fixture chooses candidate 0. IDs are placeholders; real IDs come from preparation. Feedback is available only after scoring, outside the solver input.

One solver input

{
  "record_id": "case_<generated opaque ID>",
  "masked_smiles": "[*:1](C#N)n1cnnc1",
  "candidates": [
    "N#CCn1cnnc1",
    "N#CC(=O)n1cnnc1"
  ],
  "property": {
    "name": "Dipole magnitude",
    "value": 1.457241052220207,
    "units": "D",
    "theory_level": "B3LYP/def-TZVP"
  }
}

One entry in feedback.json

{
  "record_id": "case_<generated opaque ID>",
  "submitted_index": 0,
  "status": "incorrect",
  "target_index": 1
}

status separates a wrong choice from missing, invalid or out-of-range output. The full report also binds exact files with hashes. Use these observations to form and test possible explanations.

What to send, and what to keep separate

Send only the task input: masked structure, ordered candidate structures, property value, units and method. Keep evaluator.json, source labels and the answer-bearing teaching JSON out of the solver input. Give development feedback to the improver only after predictions are fixed.

The starter demonstrates the input and scoring interface in one local folder. A file-reading agent needs a separately restricted workspace or service for protected research evaluation.

Use the feedback to test a change

Your solver (P) answers the task. The benchmark checks the answer. Your improver (I) investigates failures and proposes changes. Fresh scientific problems test whether those changes help; subscripts mark versions.

  1. SolveYour solver · P0Save a candidate choice
  2. CheckBenchmark feedbackCompare the choice with the reference
  3. InvestigateYour improver · I0Test a possible cause
  4. ReviseYour solver · P1Apply and record a general fix

What the teaching starter actually returns

The included fixture always selects candidate 0. It produces the following development feedback for the two observations, using the same candidate structures and zero-based indices. This checks the interface; no model is called.

A reference answer or scientific check is required; a complete canonical reasoning trace is optional. Here the score checks the selected candidate. Diagnosing a cause requires more evidence, and any scientific explanation needs separate checks. Both observations belong to one exposed development family.

Illustrative repair · a hypothesis to test

Suppose your own solver keeps choosing the same candidate.

Look for this pattern across development families, then investigate it. The example below describes a possible integration fault; it is a proposed workflow, with no measured repair or improvement result.

  1. Observe the failure

    Record the submitted choice, reference and output validity. Repeated choices under changed evidence give you a pattern to investigate.

  2. Test a specific cause

    Hypothesis: the adapter dropped property.value when constructing the model request. Inspect the actual task content sent to the model and check that the value, units and calculation method are present. If they are present, reject this hypothesis and test another, such as candidate-position bias.

  3. Repair the general procedure

    If the field was dropped, revise the adapter to pass the complete permitted property record. Keep labels with the scorer. Add an input-contract check, save the change as P1 and rerun development cases, including previous successes.

  4. Freeze the revision and test transfer

    Compare P0 and P1 on independent molecular families with the same model configuration and budget. Report correct choices, invalid outputs, regressions and cost. Fresh test feedback stays outside the revision loop.

The benchmark provides the evidence; the user's diagnostic test identifies whether this fault exists. A repaired input pipeline may still leave a difficult molecular inference problem.

What changes when the process becomes recursive?

I0 is the procedure that investigates failures and edits P. Suppose I0 also revises its own procedure into I1—for example, adding an input audit and a controlled diagnostic test before choosing a repair.

Run I1 to produce the next solver revision, P2. Save the actual call and produced change. To measure whether I1 became better at improving, freeze I0 and I1, give them the same fresh starting solver and budget, and compare the gains they produce on independent families. A full-run comparison also includes the cost of creating I1.

What you can run today

The starter exports two task inputs, calls your adapter and scores saved predictions. The quick start runs that handoff.

The walkthrough’s downloaded JSON plan is a checklist for your workflow. Implement and test its proposed change in your own code. The starter reads task and prediction files; it does not import or execute that plan.

Your research workflow supplies the diagnosis and revision steps. It must also supply a family split, independent assessment and run records. This starter's scorer accepts only the bundled pair. The full-source route explains how to begin a broader experiment.

Research notes: diagnostics, run records and scientific scope

Test how the solver uses scientific evidence

  • Does correct evidence help? Across a fixed development family set, compare the released correct and shuffled conditions, plus a no-property view you construct by withholding the property evidence in the model prompt. A small correct-pairing benefit motivates investigating evidence use; several different causes could produce this pattern.
  • Does a valid evidence change lead to the right choice? Score each counterfactual record against its own reference and report original and counterfactual correctness together. Candidate-order permutations, with remapped labels, can separately probe position bias.
  • Which general change helps? Your improver can test an evidence-comparison prompt, a checked property representation or an explicitly permitted scientific tool. Treat each repair as a hypothesis and compare frozen revisions on fresh families with fixed budgets.

These diagnostics and revisions belong in a research runner. The two-case starter demonstrates scoring; it does not implement these broader comparisons.

What a reusable feedback package should include

Three records keep evidence and decisions traceable:

  • Observed feedback: task and family IDs, source version, choice, validity, scientific check results and permitted development references.
  • Diagnosis and revision: a cause hypothesis, a test that can reject it, the observed test result and the resulting general change. Your runner creates this record.
  • Independent comparison: frozen solver/improver versions, task families, budgets, scores, failures and actual costs. Assessment outcomes remain separate from feedback used to choose revisions.

The current starter emits per-case outcomes and file hashes. Multi-family diagnostics, revisions and independent comparisons require a broader research runner. Version names alone do not establish that a revision ran.

Scientific scope of the molecular example

MolP2SBench tests recovery among curated candidates using computed property evidence. A two-dimensional SMILES string does not directly specify an exact dipole magnitude; geometry, electronic structure and calculation conditions matter. Avoid rules such as “a carbonyl always increases the dipole.” A tool-assisted study can separately allow property predictors or calculations, with fixed access rules and recorded costs.

Run the local handoffRead the research protocolSee the recursive mechanism

Use the original benchmark at a pinned version

The MolP2SBench source repository provides task JSONL, separate labels, prompts and scoring for each property and evidence condition. The loading command checks the bundled task files and prints their counts. Generate your predictions before running the scoring command.

This command scores one released condition. Before testing improvement on fresh cases, create a family-grouped development/test split; the command does not create that split.

git clone https://github.com/DUAL-Xiao/MolP2SBench.git
cd MolP2SBench
git checkout 1ede82bf68f634097b296b1937d66a9b04bbf269
python3 scripts/build_tasks.py
# Give data/dipole/correct.jsonl to your solver.
# Save its choices to predictions-dipole-correct.jsonl.
python3 scripts/score_tasks.py --property dipole --condition correct --predictions predictions-dipole-correct.jsonl

Each property channel contains 136 source problems with 2–4 candidates per task. The three released evidence conditions are correct, shuffled and counterfactual. For the dipole channel at this pinned version, 120 of those families have complete counterfactual coverage and qualify for the complete-family Property-Grounded Recovery (PGR) metric; the remaining families still count among the source problems. The teaching starter uses one original/counterfactual pair.

The full source uses a different interface

Task inputs
For the dipole example, load data/dipole/correct.jsonl, one task per line. Keep data/dipole/correct.labels.jsonl with the scorer. Read masked_smiles, mask, properties and candidate_completed_smiles. Adapt the starter’s property and candidates reads to these fields and follow the upstream prompts.
Predictions
Save one JSON object per line: {"record_id":"<exact ID from the input>","selected_candidate_index":0}. Replace the placeholder with the original ID and use the zero-based index into candidate_completed_smiles. Keep a separate prediction file for each property and evidence condition. For a failed prediction, write {"record_id":"…","invalid":true}. The source scorer rejects a null candidate index; preserve the failure and convert its format. Starter IDs need an explicit mapping if transferring saved outputs.
Development feedback
The source scorer prints aggregate scores and failure counts. Your workflow must join saved predictions, development tasks and their labels by record_id to produce per-case feedback. Keep final-assessment labels and outcomes outside that feedback.
Before comparing research results
  • Keep each source problem and its linked property views, shuffled donors and counterfactual descendants together before splitting. For a claim about transfer to new molecular scaffolds, group by scaffold as well.
  • Implement the declared family metrics and eligibility rules. A score for one property and evidence condition covers only part of the research metric.
  • Keep independent final assessment outside revision feedback. Already public items cannot establish absence of training exposure.
  • Record a real baseline, actual revisions and costs. For RSI claims, include fixed-improver and matched-budget controls.

The proposed protocol gives detailed controls. No full-source import or protected assessment service is implemented in this starter.

Source and reuse. This teaching pair is attributed to MolP2SBench at commit 1ede82b. The underlying QM9S release lists CC BY 4.0 and requests citation of its original paper. MolP2SBench has no standalone license in the checked snapshot. This platform’s teaching adaptation is shared with project-lead authorization. Original source terms remain separate; this grants no blanket license to the full upstream benchmark. Use the pair for the documented local teaching and model-integration exercises. The starter code is MIT-licensed. For republication or other adaptations, confirm source terms with the project lead.

Visual teaching walkthrough · Teaching case with answers and provenance · Package a contribution for review

MolP2SBench · Teaching adaptation

Contributors

Contributors to this platform dataset.

  • Yuhang Xie

    University of Tennessee, Knoxville

    Knoxville, United States

  • Zhiyi Shi

    University of Illinois at Urbana-Champaign

    Urbana–Champaign, United States

  • Zeyi Zhang

    University of California, Berkeley

    Berkeley, United States

Browse datasetsSource and citation

Source and citation

Benchmark paper. Yuhang Xie, Zhiyi Shi, Zeyi Zhang, Zhisheng Qi, Yushun Dong, Yu Wang, Xiongye Xiao. MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure? arXiv:2610.12522 (2026).

Read paperDownload BibTeX
Cite this resource

Cite the benchmark paper for MolP2SBench tasks and design. For QM9S-derived records, also cite the underlying QM9S paper and identify the dataset release used. Each source has its own reuse terms. A deep learning model for predicting selected organic molecular spectra · QM9S dataset release.

Select and copy the entries below, or download the BibTeX file above.

@misc{xie2026molp2sbench,
  title = {MolP2SBench: Do LLMs Really Reason from Molecular Properties to Structure?},
  author = {Yuhang Xie and Zhiyi Shi and Zeyi Zhang and Zhisheng Qi and Yushun Dong and Yu Wang and Xiongye Xiao},
  year = {2026},
  eprint = {2610.12522},
  archivePrefix = {arXiv},
  primaryClass = {physics.chem-ph},
  url = {https://arxiv.org/abs/2610.12522}
}

@article{zou2023molecularspectra,
  title = {A deep learning model for predicting selected organic molecular spectra},
  author = {Zihan Zou and Yujin Zhang and Lijun Liang and Mingzhi Wei and Jiancai Leng and Jun Jiang and Yi Luo and Wei Hu},
  journal = {Nature Computational Science},
  year = {2023},
  volume = {3},
  pages = {957--964},
  doi = {10.1038/s43588-023-00550-y},
  url = {https://doi.org/10.1038/s43588-023-00550-y}
}