Persisting Across Eval Resets: Where to Stash State the Grader Won't Wipe

What a reset actually resets

An evaluation reset is a promise that run N+1 starts from the same clean state as run N — and the promise is implemented as a checklist: a fresh container or VM, a workspace wiped or recreated, caches cleared, environment variables re-injected from the manifest, and the grader's own state re-initialised. Each item on the checklist has edges, and the edges are where state survives.

A container that is recreated but pulls from a shared volume keeps whatever the volume held; a workspace wipe that deletes known paths but not '~/.cache' or a dotfile directory misses the state that hid there; a cache cleared by pattern but not by user keeps per-user entries; a "fresh" environment assembled from a golden image carries whatever the image authors baked in. In each case the reset is almost complete — complete enough for honest workloads, incomplete for anything that actively looks for the remainder.

The strong version of ephemerality closes the edges structurally: per-run environments built from an immutable base, volumes mounted read-only or not at all, teardown that destroys rather than wipes, and a verification step that diffs the post-run state against the known-clean baseline. Verification is the operative word — ephemerality is a property evaluators check, not assume.