# Submission format and automated gate, version 1

This gate checks the structure of a proposed contribution and runs built-in checker fixtures. A passing report makes the package eligible for human review. It does not prove source authenticity, legal rights, scientific validity, recursive execution, improvement, or approval to publish. A runnable pre-run task may be reviewed without claiming demonstrated RSI or positive results.

For a structured dataset or an ordinary benchmark task without an RSI study contract, start with [`resource-manifest.example.json`](https://rsi4science.org/downloads/resource-manifest.example.json) and [`resource-records.example.json`](https://rsi4science.org/downloads/resource-records.example.json). This route uses `candidate_resource` and omits `rsi`. The separate [`submission-manifest.example.json`](https://rsi4science.org/downloads/submission-manifest.example.json) and [`submission-records.example.json`](https://rsi4science.org/downloads/submission-records.example.json) illustrate an optional `rsi_task` study package.

Both pairs contain four invented scalar records marked **SYNTHETIC**. Replace placeholder source, rights and any proposed evaluation arrangements with evidence or explicitly stated gaps. Passing an example retains `example_only: true` and supplies no scientific validation.

## Before formatting your data

The current gate handles scalar JSON input/target records. Arrays, nested values, images, simulator execution and sources without reference targets require an agreed representation first. Preserve their native format and begin with a source suggestion. A source suggestion can enter the separate external-resource listing review described in the contribution guide. Catalog metadata checks validate required fields and link syntax; people verify source, access and scope. Such a listing records the limited review and does not claim full data validation. This package gate still requires records; a source link or `metadata_only` release plan does not substitute for them.

For review, prepare the two JSON files, a usage/README card with a readable case, loading instructions and expected output, a saved check report, and versioned original-source links plus any available checker links. Only the two JSON files are read by the automatic gate; the other materials support human review.

For a `candidate_resource` with no reserved evaluation set, use `"protected_evaluation": {"availability": "none"}`. This exact one-field declaration applies only when no reserved set exists; it cannot be combined with custody or public-data fields. No future evaluator or governance plan is required for a resource contribution. Existing full declarations remain accepted, including those explicitly documenting that no set exists. If protected evaluation data exists, use the full declaration with `public_records` and `public_answers` both `false` and document its actual `custodian`, `access_plan` and `isolation_plan`; keep those records and answers outside the submitted files. Human reviewers verify declared absence or custody. A resource pass cannot establish independent-test availability. An `rsi_task` cannot use the short absence declaration and must retain the full declaration and a credible, reviewable evaluation arrangement before approval.

## Tracks and stages

| Stage | Machine check | What remains for a person |
| --- | --- | --- |
| `intake` | JSON object shapes, schema version, matching stable submission identifier, supported track | Identity and submission context |
| `technical` | Declared scalar schema and units, finite values and missing policy, counts, unique record IDs, permitted splits, exact duplicate inputs, family separation; fixture values follow target schema | Scientific meaning of units, family definitions, near duplicates, undisclosed relationships, full dataset coverage |
| `specification` | Required provenance/rights/access/custody declarations; additional RSI contract fields; actual positive and negative checker fixture results | Authenticity, rights evidence, baseline replay, valid scoring, testable interfaces, independent evaluation design |
| `human_review` | `pending` only after a complete technical/specification pass; otherwise `blocked` | A reviewer assesses the exact files and records a separate decision |
| `publication` | Always `requires_explicit_approval` | A separately authorized publication decision and verified release |

The manifest’s `track` selects a submission route. It is distinct from the scientific research area selected in the website form; record the research area and specialty or task keywords in the usage card.

`track: "candidate_resource"` covers structured datasets and ordinary benchmark tasks without an RSI study contract. It requires provenance, schema, split, coverage and access information and leads to `resource_review`. Omit `rsi`. It makes no assertion of RSI readiness.

`track: "rsi_task"` additionally requires the RSI declarations and built-in checker fixtures below and leads to `rsi_scientific_review`. This validates the completeness of the declared task contract. It does not run the baseline or submitted agent.

The final machine status is one of `needs_changes`, `needs_full_check`, or `ready_for_human_review`. Each issue has a `rule_id`, JSON `path`, actionable `message`, `stage`, and `category`. Human checks remain explicit even after a pass. Every result has `publication.approved: false` and `publication.published: false`; there is no approval flag a submitter can set. Unknown fields and unsupported enum values are rejected.

## Manifest

Top-level fields are `schema_version: "1.0"`, `submission_id`, `title`, `description`, `track`, `provenance`, `data`, `splits`, `release`, and, for RSI tasks only, `rsi`. Stable identifiers use 1–128 ASCII letters, digits, periods, underscores or hyphens and begin with a letter or digit.

- `provenance`: `origin` is `original`, `derived`, or `synthetic_example`; nonempty `creators` contain a required `name` and optional `affiliation` (omit it or use a string, including an empty string). `source` declares `title`, credential-free HTTP(S) `url`, and `version`; `citation` is nonempty text. A paper is not required: cite the dataset or source itself with its creator, title, version and source URL; add an associated paper if available. URLs are not fetched. `rights` contains `license`, `holder`, `evidence`, `redistribution` (`permitted`, `restricted`, or `unknown`) and boolean `example_only`. Unknown rights block review. Restricted rights are incompatible with a public access plan. All declarations require human verification.
- `data.schema`: nonempty `input_fields` and `target_fields` arrays. Each field has a unique safe `name`, `type` (`number`, `integer`, `string`, or `boolean`), nonempty `unit` (use `dimensionless` or `not_applicable` as appropriate), and `missing` (`forbidden` or `null`). Record input/target objects must contain exactly these fields. Numbers must be finite; integers must also be safe JSON integers. Strings are nonempty. Arrays and nested objects are not v1 field values.
- `data.missing_policy`: explains how missing values are handled. A nullable field must still appear explicitly with `null`; omission never counts as a valid missing value.
- `data.coverage`: `scope` is `complete` or `sample`; `declared_record_count` equals the supplied record count, and `full_record_count` is a positive integer. Complete coverage requires equal counts. A sample cannot receive a full pass, even if every supplied row is valid. Here “complete” refers to the full contribution package's development/selection data. Protected final data remains separately held and is not inspected by this gate.
- `splits`: `policy: "family_disjoint"`, a substantive `family_definition`, nonempty `permitted` entries selected from `development` and `selection`, and a `protected_evaluation` object. Each declared split must contain records; RSI tasks require both. The full protected-evaluation declaration contains `public_records: false`, `public_answers: false`, `custodian`, `access_plan`, and `isolation_plan`. Only a `candidate_resource` with no reserved evaluation set may instead use exactly `{"availability": "none"}`; this never permits protected records or answers in the package.
- `release`: `access` is `public`, `controlled`, or `metadata_only`; `public_payload` is `development_and_selection_only`; `plan` describes the intended access and publication process. Do not put controlled/private records into a public repository or public submission package. A declared access plan is not permission to distribute data.

For `synthetic_example`, use a title containing `SYNTHETIC` and set `rights.example_only: true`. Replace placeholder metadata and proposed contracts with real, reviewable evidence before submitting an actual scientific resource.

## Records and leakage checks

The envelope is `{ "schema_version": "1.0", "submission_id": "matching-id", "records": [...] }`. Every record has exactly `record_id`, `family_id`, `split`, `input`, and `target`:

```json
{
  "record_id": "synthetic-dev-1",
  "family_id": "synthetic-slope-2",
  "split": "development",
  "input": { "x": 1, "slope": 2 },
  "target": { "y": 2 }
}
```

Record IDs must be unique. Family IDs are intentionally shared by related records, and one family cannot span splits. Declare all related or derived records under the same family before splitting. The validator compares exact input content with object keys sorted, so changing JSON key order does not conceal duplication across splits. It cannot discover undeclared scientific relationships, near duplicates, semantic equivalence, or hidden derivation links. These remain human checks.

Only declared development/selection inputs and targets belong in this envelope. `audit`, `test`, `final`, or protected splits, extra audit-answer fields, and unsupported envelope/manifest fields block the check. Schema checks cannot detect a protected answer disguised as an otherwise legitimate field or inside descriptive prose; the release reviewer must inspect content and custody.

## Additional RSI contract

Every contract value below is a nonempty declaration, not evidence that execution occurred:

- `baseline`: `name`, `version`, `artifact`, `replay_procedure`, `environment`, `seed_policy`.
- `improvement`: `editable_surface`, `mechanism`, `immutable_evaluation`, `install_contract`, `subsequent_use_contract`, `reset_contract`, `rollback_contract`. Separate editable improver state from the immutable evaluation system, explain installation and later use, and define reset and rollback behavior.
- `feedback`: `source`, `content`, `timing`, `failure_policy`, and `available_splits`, which may contain only `development`. Selection/final feedback cannot become training feedback.
- `logging`: `format` and unique `fields` containing `version_id`, `parent_version_id`, `attempt_id`, `acting_improver_version`, `installed_version_id`, `subsequent_use_id`, `feedback_source`, `decision`, `outcome`, `cost`, and `budget_remaining`. The failure policy must explain retention of failed, regressed and rejected attempts. Actual log records are reviewed separately; this gate does not validate experiment logs.
- `budget`: `unit`, finite positive numeric `limit`, `scope`, `enforcement`.
- `evaluation`: `independent: true`, `custodian`, `procedure`, `primary_metric`, `metric_direction` (`minimize` or `maximize`), `uncertainty_plan`, `selection_policy`, `common_start_policy`, `fixed_judge`.
- `fixed_improver_control`: `frozen_version`, `procedure`, `matched_budget`.
- `checker`: a supported declarative rule and actual positive/negative fixtures, as described next.

Human review must establish whether the baseline is replayable, the improvement interfaces can install and reuse a changed improver, scoring is scientifically valid, comparisons share a common start, costs are fully counted, and independent evaluation and a fixed-improver control are appropriate. A pre-run task package does not need a successful improved-agent result. Claims of actual recursion or benefit require separately recorded execution and evidence.

## Built-in checker fixtures

`checker.kind` is `exact_match` or `numeric_tolerance`. Each fixture contains `name`, `expected`, `actual`, and boolean `should_pass`. Expected and actual are objects matching the complete target schema. Include at least one `true` and one `false` fixture. Both are actually evaluated; a checker that accepts its declared negative fixture fails the gate. An ill-typed negative fixture also fails schema validation and cannot substitute for a meaningful negative test.

`exact_match` compares all target keys and primitive values exactly, independent of key order. It has no tolerance fields. `numeric_tolerance` requires numeric, non-nullable target fields and finite nonnegative `absolute_tolerance` and `relative_tolerance`. For every field it checks `abs(actual - expected) <= absolute_tolerance + relative_tolerance * abs(expected)`. Overflow fails closed. No submitted verifier code, scripts, modules, executables, or archives are loaded or executed. Domain-specific scientific scoring is outside v1 and requires a reviewed extension.

## Local check and directory mode

The website provides a standalone checker ZIP containing the validator, CLI, synthetic examples and a README. It needs Node.js 22.12 or later, with no package installation or repository access. Extract it and follow its README; the same manifest/records command below works from the extracted `submission-checker` folder. For directory mode, provide your own submissions folder. The repository test command below requires a full source checkout. The source repository is currently private and available to invited collaborators.

```sh
node scripts/check-submissions.mjs \
  --manifest public/downloads/submission-manifest.example.json \
  --records public/downloads/submission-records.example.json \
  --report submission-report.json

node scripts/check-submissions.mjs --directory submissions
node scripts/check-submissions.mjs --help
node --test scripts/submission-check.test.mjs
```

Directory mode visits immediate child directories in name order; each must contain `manifest.json` and `records.json`. It does not descend into nested submissions, follow symlinks, interpret submitted paths, fetch network resources, or write inside the submission directories. A symlink child is rejected. Ordinary root-level files are ignored. An empty directory returns `status: "no_candidates"` and `checked_count: 0`; this is not a reviewed contribution. A missing directory is an error.

Reports go to stdout. Optional `--report` creates a new file only, in an existing directory; existing files are never overwritten. Symlink path components are rejected for input and report paths. An absolute path through `/private/tmp` may be needed on systems where `/tmp` is a symlink. No approval, publication receipt or `dist` output is created.

CLI limits are 1 MiB for the manifest and 64 MiB for records; the browser uses smaller limits displayed in its form. Shared validation permits at most 100,000 records, 128 items in other arrays, and 128 object fields. Oversized structures fail and are not fully inspected. Error detail is capped at 200 issues; the report includes `issue_count` and `issues_truncated`. Record inspection halts after extensive errors, always blocks a pass, and reports both supplied `record_count` and actual `inspected_record_count`.

Exit code `0` means eligible for human review or no directory candidates; inspect `status` and `checked_count`. Code `1` means the gate blocked progress (including sample-only coverage); code `2` means invalid CLI input or an IO/JSON error. The aggregate directory report retains each child's full result.

## Reports and review binding

The pure browser-compatible API is `checkSubmission(manifest, recordsEnvelope)` from `src/lib/submission-check.mjs`; it returns a deterministic report with `validator_version`. JSON parsing, byte limits and hashing happen at the calling boundary. The CLI binds reports to exact raw bytes with `inputs.manifest.sha256`, `inputs.records.sha256`, and their `byte_length`; the browser adds the same fields using Web Crypto. Formatting-only changes also change the digest.

A reviewer should retain the report, both input digests, validator version, and reviewed revision with their separate decision. A later edit requires a new check and review of the affected evidence. A digest establishes which bytes were checked; it is not a signature, trust certificate, scientific endorsement or publication authorization. Do not rely on a contributor-supplied report without rerunning it against the files being reviewed.

Standalone checker: https://rsi4science.org/downloads/submission-checker.zip
Version and checksums: https://rsi4science.org/downloads/submission-checker.manifest.json
Support: https://rsi4science.org/contact/