Repository-level program generation

From a repository
to a verifiable task.

Build long-horizon, from-scratch coding tasks with specifications, execution environments, hidden tests and reference controls.

Create your first task Explore configuration ↗

programgen.toml → tasks/

# One configuration. Three commands.
$ programgen search
$ programgen create --repo oauthlib/oauthlib
$ programgen produce

01 Pinned repository inputs02 Two reconstruction modes03 Verified publications

01 / CHOOSE THE TASK

Two ways to reconstruct a program.

Each mode keeps its own construction and quality rules.

WITH-BINARY · REFERENCE GUIDED

ProgramBench

Rebuild a program from its usage documentation and a black-box reference oracle. Generated behavioral tests check the implementation.

200+ active validated tests
8+ active coverage categories
Harbor + native output

Understand the contract

WITHOUT-BINARY · SPECIFICATION GUIDED

NL2Repo

Reconstruct a repository from a generated specification. Original repository tests, API contracts and offline replay provide the checks.

20+ measured repository tests
90%+ measured and final golden pass rate
Complete API, import and signature coverage

Understand the contract

Use with-binary and without-binary in your configuration. programbench and nl2repo are accepted aliases.

02 / FOLLOW THE PIPELINE

Construction with evidence at every step.

Generate one task with create, or process a candidate list with produce.

DiscoverRepository + immutable commitcandidates.jsonl

ConstructReference or measured environment
Tests, docs and specification

VerifyMode-specific quality gates
Offline gold and negative controls

PublishCertified task bundlespromoted.jsonl

03 / WORK WITH THE OUTPUT

A task you can inspect and reuse.

tasks/
├── promoted.jsonl
├── failed.jsonl
├── with_binary/…
└── without_binary/…

.programgen/produce/
└── resumable work and execution evidence

Published tasks carry the instruction, environment, hidden tests and full verification evidence. Harbor and ProgramBench runners handle downstream execution.

Control concurrency with one workers setting.
Rerun the same command to resume interrupted work.
Inspect concrete build, quality and service failures.
Verify an existing task with programgen verify.

Explore the task layouts →

04 / CHECK THE EVIDENCE

Validated on real repositories.

Read the full acceptance scope →

FROZEN-INPUT ACCEPTANCE2026-11-05

Original mode standards. Actual Docker and model calls. Failures retained.

PROGRAMBENCH4 / 5

Certified fixed inputs.
Rust + Go. One build rejection.

NL2REPO5 tasks

Certified distinct inputs.
Python + Go.

Recovery checks include real SIGKILL, two resumers, a shared concurrency limit, corrupted payloads and missing certificates. These sample results do not predict a general production success rate.

From a repositoryto a verifiable task.