STRATA-Bench protocol
STRATA-Bench protocol
Version 1.0.0 · September 2026 · Track Sandbox is the official score.
1. Problem
An agent is given a longitudinal research brief over a nested geography. Public evidence covers the metro; the user asked for the core. A second source publishes core figures in a different unit on a sparser cadence. A third source is a marketing chart with a broken x-axis.
The agent must produce a submission JSON (see submissions/SCHEMA.md) that a deterministic harness can score.
2. Evidence rules (sandbox)
| Allowed | Forbidden |
|---|---|
data/sandbox/corpus/** |
data/hidden/** |
data/public/tasks.json |
The public internet |
| This protocol | Fine-tuning on gold values |
Leaking gold into the context window invalidates a run. The corpus does contain enough information to reconstruct metro gold exactly and core gold via the documented 11.5% premium.
3. Geography
| Code | Meaning | Public in corpus? |
|---|---|---|
metro |
Meridian Metro, board-wide (MREB) | Yes, 20/20 quarters |
core |
Districts C-1 Harbour + C-8 Civic Core | UrbanPulse only, sparse; otherwise modeled |
unknown |
Agent could not tag | Penalized on GEO tasks |
mixed |
Explicit blend with method note | Allowed if disclosed |
“Downtown”, “city”, and “core” in prose are not geography codes. Relabeling metro extracts as core without a disclaimer is GEO_MASK.
4. Status of a cell
status |
Meaning |
|---|---|
reported |
Copied from a named source file |
interpolated |
Mathematical fill; must appear in interpolated_periods |
modeled |
Transform of a reported series (e.g. metro × 1.115) |
omitted |
Known missing; listed in omitted_periods |
visual_estimate |
Read from a chart / rounded aggregator |
A filled gap with status=reported is SILENT_INTERP.
5. Axes
axis_type must be datetime whenever the series is incomplete or the task is in family AXIS. Categorical equal-interval ticks on a gappy series are CATEGORICAL_GAP. An 18-month hole must occupy six times the width of a quarter.
6. Hard fails
Hard fails do not zero the run. They multiply that task’s composite by 0.55 and are listed on the scorecard. Codes: GEO_MASK, HALLUCINATED_PAYWALL, SILENT_INTERP, CATEGORICAL_GAP, OCR_LAUNDER, BLEND, TRUNCATE_2026, UNIT_COLLAPSE. Definitions live in strata_bench/schema.py.
7. Archetypes (descriptive, not scored)
The motivating audit observed three behaviours. Agents may self-identify in method_archetype:
- Empirical literalist — public board only; refuses unpaid core; complete metro time axis.
- Synthetic econometrician — constructs a core series with a disclosed premium; tags modeled cells.
- Fragmented extractor — harvests whatever snippets exist; skips quarters; categorical axis.
STRATA-Bench does not prefer literalist over econometrician. It prefers honesty. A tagged 11.5% premium scores; a silent relabel does not.
8. Live track
LIVE-01..03 are optional public-web probes patterned on Toronto condominium rentals (GTA board vs C01/C08 paywall vs consultancy $/psf). They are not part of the headline sandbox score. Gold for live is not shipped; reviewers score spatial disclosure, citations, and axis type.