Insights F01: quoted but not booked

The 98-question target is NOT ESTABLISHED; overall family acceptance is FAIL. Only five original source IDs have direct corrected-oracle mappings. The measured real-family sample has eight attempts, including an authored three-turn conversation; it cannot establish correctness for all 98 source questions.

Observed correct completions in repeats 1 / 2 / 3: fresh candidate 0 / 0 / 0 versus control 1 / 3 / 3 per 122 attempts; real-sample candidate 2 / 1 / 2 versus control 1 / 1 / 2 per 101 attempts. Within the eight-attempt real-family sample, candidate 1 / 0 / 1 versus control 0 / 0 / 1. Coverage limits and measured outcomes are separate findings; neither comparison-unverified answers nor missing references count as correct.

For the five directly mapped original source questions, control correct counts are 0 / 0 / 1 and candidate counts are 1 / 0 / 0. Their pooled correct count does not increase. The eight-attempt sample’s additional candidate success comes from its authored conversation; it is not an additional correct original source question.

The application change gives the current generator a mandatory, question-conditioned family definition: client-requested quotes, client identity, booking existence, the user anti-join and the ratified booking window. Older conflicting guidance is reconciled. A fixed refusal is implemented for recognized unsupported variations. Measured refusal coverage is incomplete: one outside-envelope case remained comparison-unverified in all three candidate repeats; family obligations remain explicitly unproved. Verifier selection receives the same definition; the measured verifier is off. No generator/model selection or verifier repair change was made.

Goal F01 maps to source family F10. Source F01 is acquisition-source signups. All correctness labels below mean correctness under the unchanged corrected D2 oracle. The correspondence of every reference to every family variation has not been independently established.

Populations and method

Three repeats per arm/population; 122 fresh attempts (31 families), 101 real-sample attempts (45 families). D3 Arm A supplies three historical fresh controls and real repeat 1; base-tip real repeats 2–3 are new. New runs use two workers, frozen clock 2026-09-06 12:00 UTC, London periods, zai requested glm-5.2, temperature 0, seed 42, thinking disabled and a 25-second provider timeout. Provider response identity is retained separately. Historical D3 measured 2513a718; its final-tip metadata/telemetry corrections are qualified by D3’s 934-attempt non-interference replay, not a live latency equivalence test. New controls and candidates share the F01-isolated harness. Historical controls lack flock evidence. Sequential host/provider conditions remain confounded.

Every new repeat and restart acquires /tmp/lore-eval.lock; wait and host observations are retained. Primary latency is nearest-rank p95 of feature_completion_ms (submission through terminal-result retrieval), measured on a shared host. Timed failures are included; missing timings are excluded and counted. Precision is correct/(correct+silent wrong); comparison-unverified is excluded and undefined precision stays n/a.

First-turn error-rate denominators are supported first turns: fresh 116, real 86, fresh target 5, real target 6, fresh excluding target 111, real excluding target 80. Whole fresh contains three conversations; whole real contains five.

The fresh slice was previously exposed in D2. The recorded design process used question shapes and ratified definitions; candidate outputs were not used to tune application code. Application and harness were frozen before measurement. These records support consistency, but cannot independently prove non-exposure or the absence of influence from previously observed answers. Full-98 missing oracle/context observations are coverage gaps, not incorrect answers.

The previous session lost its controller during candidate fresh repeat 1. Its 122 submitted attempts survive, but its terminal summary, original exit status and original cleanup receipt are missing. Recovery verified the orphan server identity, stopped it under the shared lock and checked unchanged frozen inputs. Recovery does not establish uninterrupted lock ownership through the original shutdown. The run remains incomplete and excluded from primary metrics; its compact attempts and a clearly labelled recovery receipt are retained under aborted/. The single allowed clean restart supplies scored repeat 1. The earlier zero-question control admission is retained separately. No answer quality selected either restart.

Whole fresh slice

Mean [minimum, maximum] from three repeats. Undefined values are excluded from means and remain n/a in the per-repeat table; the number of defined values is retained in results.json.

The mean correct-completion difference is -2.33 per repeat, beyond the control’s observed range of 2.00. The range is an observed noise floor, not a confidence bound.

Whole real sample

Mean [minimum, maximum] from three repeats. Undefined values are excluded from means and remain n/a in the per-repeat table; the number of defined values is retained in results.json.

The mean correct-completion difference is 0.33 per repeat, inside or equal to the control’s observed range of 1.00. The range is an observed noise floor, not a confidence bound.

Fresh target sample (seven attempts)

Mean [minimum, maximum] from three repeats. Undefined values are excluded from means and remain n/a in the per-repeat table; the number of defined values is retained in results.json.

One family cluster: this interval is degenerate and does not quantify within-family or all-98 uncertainty. It cannot establish the inferential acceptance target.

The mean correct-completion difference is -0.33 per repeat, inside or equal to the control’s observed range of 1.00. The range is an observed noise floor, not a confidence bound.

Real target sample (eight attempts; five direct source IDs)

Mean [minimum, maximum] from three repeats. Undefined values are excluded from means and remain n/a in the per-repeat table; the number of defined values is retained in results.json.

One family cluster: this interval is degenerate and does not quantify within-family or all-98 uncertainty. It cannot establish the inferential acceptance target.

The mean correct-completion difference is 0.33 per repeat, inside or equal to the control’s observed range of 1.00. The range is an observed noise floor, not a confidence bound.

Fresh excluding target (115 attempts)

Mean [minimum, maximum] from three repeats. Undefined values are excluded from means and remain n/a in the per-repeat table; the number of defined values is retained in results.json.

The mean correct-completion difference is -2.00 per repeat, inside or equal to the control’s observed range of 2.00. The range is an observed noise floor, not a confidence bound.

The registered regression check nevertheless fails because the paired interval is wholly negative: [-3.87, -0.35].

Real excluding target (93 attempts)

Mean [minimum, maximum] from three repeats. Undefined values are excluded from means and remain n/a in the per-repeat table; the number of defined values is retained in results.json.

The mean correct-completion difference is 0.00 per repeat, inside or equal to the control’s observed range of 0.00. The range is an observed noise floor, not a confidence bound.

Full 98-question target

The inventory contains 39 first questions and 59 follow-ups. User ancestor chains exist, but historical answer/result state is absent. Replaying them would create new populations and would not supply the 93 missing corrected references. target98-coverage.jsonl retains each ID, shape, envelope status and per-arm/repeat observed outcome or explicit missing-measurement status.

Target case outcomes

Acceptance

Remaining observed target classes

Counts pool three candidate repeats. Unverified comparisons and refusals are kept separate from confirmed wrong answers. Observed defect labels can overlap; row-set size, projection-width and recorded answer-contract failures follow D3 reporting conventions and do not isolate SQL root causes.

Full target coverage gaps: 93 source questions lack direct corrected references; 59 source questions are follow-ups with unavailable historical result state. These counts overlap.

Transitions against control

Equal-weight Cartesian pairing of all nine repeat pairs, matched by case ID and scaled to one population repeat. These are descriptive transitions, not proof of a causal correction.

fresh

real

real_target

Validation and model evidence

The retained affected suite passed 1151 tests, with 0 failures, 0 errors and 88 skips. Its present curated subset passed 636, with 0 failures and 88 skips. The 19 absent inherited files and all skipped checks remain unverified. The full receipt is checks/summary.json.

The earlier setup failure is retained separately in the check receipt. No failed or skipped check is converted to a pass by summing later checks.

A response model ID is provider-reported identity, not an independent serving attestation. An attempt with no mapped completed generator call has unknown response identity; it is not assigned the requested model. Failed and unmapped calls remain visible in the traces.

New-run admission and integrity

Review ledger

Pre-launch reviews: mechanics and method, separate Codex CLI calls requesting gpt-6-astra with high reasoning, strict no modifications. Both completed; their finding-to-action ledger is in the frozen plan.

Premeasurement code review: completed receipt, three independent lenses (correctness, security, adversarial), six parent-context lenses; fresh validator dispatch was unavailable at the CLI thread limit. Its initial verdict was not ready. Confirmed fixes below preceded every scored run. A separate read-only validation checked the reporting fixes. No claim of full independent code-review coverage is made.

The final source-contract inspection also clarified that the SQL-only generator must substitute every named scaffold parameter with a resolved SQL expression or literal; the executable builder still returns bound parameters for contract tests. This clarification preceded measurement and was checked by the focused generation/definition suite.

The application changes derive from ratified definitions, question shapes, source inspection and synthetic cases. No candidate answers were available during these corrections. Final result reviews and any report-only corrections follow below.

Empty-admission method parity correction

Read-only compact-export audit: no question/SQL/customer fields found in inspected exports; full98 and measured8 are separated. One misleading coverage field used true for first questions, although parent state is not applicable. Changed it to null; follow-up parent-state availability remains false. This changes a coverage description only.

Independent admission review caught a preservation issue before the resume guard was used: SQLite mode=ro may still create WAL sidecars, and its transaction context manager does not close the connection. Use immutable read-only access with explicit closure after rejecting unapplied WAL content; cover sidecar preservation and rejection in synthetic tests. No retained measurement database was opened by the unsafe check.

Before any scored question, inspection found an added 1,800-second idle timeout absent from D3. The empty admission was interrupted and retained; no application server or question was started. Restore unbounded admission while preserving the 95% gate and add explicit, fail-closed resume for this verified empty cancellation only. Keep the application revision frozen and record a separate harness revision. This correction follows a read-only method advisory that rejected relaxing the idle threshold without authorization. The admission receipt confirms cleanup, a clear listener and unchanged frozen inputs.

Resume evidence preparation

• The prior session's orphaned candidate run had no terminal summary or original exit/cleanup receipt. Verify ownership and stop the orphan under flock; preserve the 122 original attempts, mark the original run incomplete and use its one clean restart. A recovery receipt explicitly leaves the original exit status unknown.
• Reporting originally exported only selected scored repeats. Preserve incomplete admissions and interrupted attempts under aborted/, separately from primary metrics, and require complete new-run receipts plus terminal summary and integrity/cleanup evidence. Add a synthetic preservation and private-text export test.
• Per-attempt empty response-model lists could be misread as known identity. Add mapped completed-call counts and an explicit unknown status when there is no mapped completed generator call; never substitute requested identity.
• Replace the report's long inline check JSON with a readable validation table, while retaining the complete check receipt. These changes affect evidence presentation only.
• Remaining-defect reporting now distinguishes observed row-set size, incompatible projection width and recorded answer-contract failures using the existing D3 conventions. Harmless equal-width aliases stay unclassified; overlapping labels are disclosed. This diagnostic reads structural saved-result fields only and does not rescore answers, change acceptance or tune the application. Synthetic checks cover alias and overlapping-shape cases.
• Dependency inspection found the prepared report generator imported an unavailable Markdown package. Use the already-installed markdown-it renderer with table support and raw HTML disabled; no production virtualenv changes or package installation. Add a synthetic report-rendering check.

Resume verification: 12 passed in 13.23s; no failures or skips. Ruff passed for all 20 changed Python files. All eight new measurement receipts are complete and pass cleanup/input-integrity checks; the main checkout tracked diff, status and HEAD match the recorded resume baseline.

Export verification covers all 1,460 compact attempts, including 122 interrupted-run attempts. The repository ignores JSONL evidence by default; the approved compact records are explicitly included after scanning. Compact-only results match raw-derived results exactly. Review before/after hashes include ignored evidence and private raw JSON/JSONL records.

Final arithmetic review

Independent Codex CLI review requesting gpt-6-astra with high reasoning exited 0, changed no protected file (6,771 checked), and independently reproduced all 60 metric/interval entries, all transitions and all 1,338 scored raw/compact attempts. Arithmetic PASS; family acceptance FAIL. Correct stale requested-only model-provenance metadata to distinguish configuration from recorded response identity and unknown unmapped calls. Add direct-source correct counts (control 0/0/1; candidate 1/0/0) to the report and PR: the eight-attempt sample’s additional success includes the authored conversation. Neither correction changes any measured outcome or acceptance rule. The reviewer did not independently rescore SQL/reference answers or attest serving identity.

Post-correction reporting validation: 12 passed in 13.77s; changed-file Ruff passed for all 20 Python files.

Final leakage review

Independent Codex CLI review requesting gpt-6-astra with high reasoning exited 0 and changed no protected file (6,772 checked). It independently matched all 1,338 scored model joins, 12 correctness/silent-wrong intervals, 95 transition cells, 72 per-repeat precision/p95 values and 299 private-file hash entries. No leakage was detected within its stated inspected boundary; this is bounded evidence, not an anonymization certification. Family acceptance FAIL is supported.

• Publisher source accepted symlinked ancestors or destinations → validate every source/staging ancestry, reject traversal, create a fresh exclusive staging directory and exclusively create regular report-only assets. Nine synthetic boundary checks passed, including unchanged outside sentinels, without credentials or remote calls. The reviewed publisher and check source are retained with the evidence.
• Interrupted 122-attempt export lacked per-attempt model fields → share the scored-run identity annotator with incomplete exports. Failed calls never supply response identity; unmapped calls remain unknown. Add coverage for missing and partially missing identity. Reporting checks: 13 passed in 13.96s; Ruff passed on all 20 changed Python files.
• Freeze receipts cannot prove absence of prior exposure, and recovery cannot prove original continuous lock ownership → state both limits explicitly. Retain prior fresh exposure, historical-host confounding, oracle-alignment uncertainty and the one-family interval caveat. No measured answer, score or application code changed.

Final wording review

A third independent Codex CLI review requesting gpt-6-astra with high reasoning exited 0 and changed no protected file (6,778 checked). It independently reproduced all 60 bootstrap entries, 95 transition cells, 72 per-repeat precision/p95 values, all 1,460 attempt identity annotations and 299 private-file hashes. Arithmetic PASS; family acceptance FAIL. No independent serving attestation is claimed for any review.

• Universal fixed-refusal wording exceeded observed coverage → limit the implementation claim to recognized unsupported variations and disclose one outside-envelope case that remained comparison-unverified in all three candidate repeats. This is a limitation, not an inferred wrong answer.
• “Full conversations” named a success metric ambiguously → label it “Fully correct conversations”; state supported first-turn error-rate denominators and whole-population conversation counts.
• Fresh non-target explanation stopped at the noise range → explicitly state that the wholly negative interval triggers FAIL.
• PR identity qualification was incomplete → state requested glm-5.2, reported glm-5.3, unknown unmapped attempts and absence of independent serving attestation.
• Leakage findings 1–2 → reviewer confirmed the hash-matched publisher fixes and nine recorded boundary checks by static inspection, and independently reconstructed all interrupted identity fields. Publication was not executed by the reviewer; this is bounded closure, not protection against concurrent filesystem substitution or an anonymization certification.

All final corrections affect report/evidence wording, publication preparation or provenance completeness. No application tuning, rescoring or additional measurement followed the scored runs.

Final report-source verification: 13 passed in 12.39s; changed-file Ruff passed for all 20 Python files. Exact review prompts and replies retain their original formatting and hashes.

Delivery launcher correction

The first publication launch exited before Wrangler could run: isolated XDG configuration made the mise shim select Node 20.20.2, below Wrangler’s required Node 22. A credential-free explicit-shim version command reproduced the exact retained stderr hash. Pin the existing Node 22.22.3 executable while keeping private caches and environment-only credentials; its version probe succeeds. Preserve the failed launch receipt. This launcher-only change followed the independent reviews; the reviewed ancestry/staging logic is unchanged, and all nine synthetic boundary checks passed again against the new source hash.

The cached Wrangler 4.131.0 also automatically delegated an agent-run Pages command to Workers, producing authentication error 10000 on a Workers service request. Local command-source inspection identified --force as the explicit Pages path for both creation and deployment. Use that flag to bypass delegation; no credential scope is expanded. This changes publication routing only, after review, while preserving the reviewed report-only staging boundary.

Direct Pages creation succeeded. Before upload, the regular-assets guard rejected Wrangler’s generated cache directory inside staging. Preserve that private attempt and give Wrangler a separate fresh private working directory, leaving only the report and headers in upload staging. No upload occurred before the guard stopped the attempt. This delivery-only correction preserves the reviewed boundary rather than relaxing it.

Final publisher validation adds a simulated CLI that creates a working-directory cache and verifies it stays outside both uploaded assets. All ten checks passed (the original nine boundaries plus this cache-flow check), using synthetic credentials and no remote calls or real credential reads.

Evidence and limits

Plan: docs/plans/2026-09-12-lore-goal-f01-plan.md. Compact records and recomputable results: docs/evidence/insights-goal-f01-2026-09-12/. Full records: /tmp/lore-goal3-eval/goal-f01/. scripts/insights_f01_report.py --compact recomputes metrics, paired intervals and transitions. Bootstrap: 10,000 whole-family draws, seed 2026090611, nearest-rank percentiles, pooled repeat counts normalized to each population; undefined draws are counted. Precision and p95 pool before differencing, so their points can differ from differences of repeat means.

No application was deployed or merged. Expected merge order: #104 safety release, sibling DCLASS, then F01 after resolving any overlap and rerunning affected checks. The sibling checkout and branch were not inspected or modified. These measurements do not verify the combined sibling/F01 result.