ProgramGen

简体中文 · Documentation

Build verifiable, repository-level program generation tasks from real repositories. ProgramGen packages the specification, execution environment, hidden tests and reference solution needed to reconstruct a program from scratch.

The construction core comes from Program-gen. programbench and nl2repo remain accepted aliases. Generated tasks can be consumed by Harbor; with-binary tasks also support the ProgramBench native format. Training and task runners are external tools.

Quickstart

Use a Linux/x86_64 machine with Python 3.11+, Git, Docker, and an Anthropic Messages-compatible model service. Start with one task; production defaults to two concurrent tasks. This repository currently requires authorized access.

git clone https://github.com/LegoX/ProgramGen.git
cd ProgramGen
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[sdk]'
mkdir -p ../programgen-work
cp -n configs/programgen.toml ../programgen-work/programgen.toml
cd ../programgen-work

Edit programgen.toml: set api_base_url and model to your service's values. Its comments explain the necessary settings. Keep credentials in your environment:

export ANTHROPIC_API_KEY=your-model-service-key
export GITHUB_TOKEN=your-github-token

# Check effective quality settings without generating a task.
programgen create --repo oauthlib/oauthlib --check

# Generate and fully verify one without-binary task.
programgen create --repo oauthlib/oauthlib \
  --ref 40b0ab56da3682c2484a4b78bbff309f8025d950

For with-binary, select the mode and executable target:

programgen create --mode with-binary --repo 1password/typeshare \
  --ref 7d3189c9cbca0003ee69b0ef22dd08a7b1f0deb5 --binary typeshare

Generation uses real model calls and Docker builds. A candidate may be rejected if its build or generated task does not meet the quality requirements. The command reports the resulting task path or the concrete failure. The quickstart installs both supported model runners. Keep generation work outside the source checkout so Git-aware programs see isolated test directories. Configuration details. Reference programs may also need host tools such as Go; install the toolchain required by the selected repository when its runtime behavior depends on it.

Produce a dataset on one machine

programgen search --check
programgen search
programgen produce --check
programgen produce

Search writes immutable repository commits to candidates.jsonl. You can also write your own candidates, including mixed modes:

{"repo":"oauthlib/oauthlib","commit":"40b0ab56da3682c2484a4b78bbff309f8025d950","mode":"without-binary"}
{"repo":"1password/typeshare","commit":"7d3189c9cbca0003ee69b0ef22dd08a7b1f0deb5","mode":"with-binary","binary":"typeshare"}

Set [produce].workers to control concurrency. Each row receives the correct mode defaults. Repeating the command resumes interrupted work, verifies completed publications and skips intact tasks. Duplicate inputs do not produce duplicate tasks. Normal build or quality rejections do not stop other candidates.

tasks/
  promoted.jsonl       successful certified publications
  failed.jsonl         rejected candidates and execution errors
  with_binary/...     with-binary task directories
  without_binary/...  without-binary task directories
.programgen/produce/  resumable state and detailed execution records

The summary distinguishes rejected candidates from errors that require attention. A completed batch with normal rejections exits zero; an execution error exits nonzero. An empty promoted set is never proof of task quality. Outputs and recovery.

Verify an existing task

programgen verify /path/to/task

Verification checks the payload and quality evidence, then replays the original consumer in Docker. With-binary publication requires at least 200 active validated tests and 8 active behavioral categories. Without-binary requires at least 20 measured tests, 90% passing, 50% Python coverage, a complete specification, and 90% final offline golden reward. Quality and verification.

Develop

python -m pip install -e '.[dev]'
python tools/acceptance/acceptance_tests.py --suite offline --evidence-root /tmp/pg-tests

Core implementation lives in src/programgen/, examples in configs/ and examples/, and maintainer verification tools in tools/acceptance/. See development, validation status and source provenance.

Related projects

• Repo2RLEnv generates Harbor environments from several kinds of repository material.
• SWE-gen converts merged pull requests into validated repository repair tasks.