Verification and evidence

programgen verify TASK --level layout
programgen verify TASK --level quality
programgen verify TASK --evidence /absolute/new-evidence-directory

Default full runs quality checks and fresh offline Docker gold/rerun/dummy consumers. --evidence must identify a new directory; otherwise results go under TASK/.programgen-verification/runs/UNIQUE_ID. Existing records are preserved. Each JSON result reports requested level, individual statuses, errors, source hashes, payload byte/permission/link hashes and raw-evidence hashes. Unexecuted checks remain unverified; layout success cannot issue a quality certificate.

ProgramBench

Default generation retains the original SDK health, assertion lint and branch execution gates: at least 200 active validated tests and eight active coverage categories. Full verification also replays the published task with the original consumer. It does not equate the active test count with a semantic diversity count.

Optional semantic audit

Set [tests].semantic_review = true for additional independent review. This is not an original pipeline requirement and is disabled by default. When enabled, the extra review is required for that task's publication. The reviewer receives the actual archived tests/helpers/fixtures in a fresh session, with 20 original IDs per batch, followed by cross-suite review. Mechanical checks validate complete ID coverage, exact source citations, the specific test body, gold/rerun/dummy outcomes, semantic duplicate groups and category witnesses. Constant/coarse assertions and unresolved definitions add zero to the conservative lower bound. At least 200 effective cases and eight witnessed categories are required, including core-capability evidence.

This is explicitly model-assisted semantic evidence. It does not claim a formal proof of arbitrary helper behavior. Unknown semantics are excluded from the effective count. Batches share the configured agent time budget; malformed responses get at most three bounded submissions, with every failure retained. Semantic failures feed the existing finite test repair loop. Original catalogs, archives, ignored IDs and scoring denominators remain independent.

The original reference oracle forwards only LANG and LC_ALL; it manages HOME, TMPDIR and PATH itself. It is a restricted query service, not an exact golden implementation for tests that set process environment variables. Gold/rerun therefore execute the exact hash-bound reference build, matching legacy generation validation, in the unchanged published verifier. A read-only reference mount exists only in these privileged positive controls; it is absent from the agent and negative controls. Runtime companions are copied with the reference, as in legacy validation.

The original grader still checks the complete test denominator. Oracle access and agent isolation are tested separately, and a submitted copy of the known reference binary must be rejected by the original hash guard. Empty submissions must also fail. These are verification controls, not cleanroom solutions.

Review caching binds source archives, catalog, exact execution IDs/outcomes, reviewer identity and implementation. Transient pytest console timing is excluded from this semantic cache key; complete raw execution output is still retained in generation evidence and bound by the production receipt.

NL2Repo

Quality verification reparses raw measurement and coverage; compares counts, pass rate and measured/enumerated contracts; validates both patches; checks the complete published instruction against the retained component/import/signature profile; and requires final-image evidence. Hard-pool classification and zero budget semantics follow the original mode policy. Full replay executes the actual solution and hidden patch in fresh offline containers and recomputes original metrics from raw output, preserving failed/error/skip outcomes. Dependency inventory follows the task language. Closed-book isolation also rejects Node environments that retain the target package's implementation in node_modules, even when the original workspace source has been removed.

generation.json carries actual generation inputs, source commit, effective configuration and implementation fingerprints. It is stored outside the agent environment. A production certificate requires both quality and consumer evidence and is included in the receipt's complete file inventory.

Historical tasks and replacements

The maintainer replay tool accepts a manifest linking an original task to its original promotion/build state and source checkout. It derives external evidence and calls the same verifier without rewriting the historical task. See tools/acceptance/acceptance_replay.py --help and development.

A historical quality failure may be replaced in the current acceptance cohort by a new independently certified task. Keep the original failed result, replacement reason, repository/commit and new certificate. A timeout or unexecuted check alone does not establish that the historical task is intrinsically inadequate.

An explicit --oracle-override is a diagnostic experiment and cannot certify the original oracle/task pair. Missing evidence remains unverified.