Architecture

repogym is a small library (about 4,000 lines of Python, four runtime dependencies) organised around one idea: a coding task is data, and a reward is a pure function of a task and a patch. Everything else (workspaces, sandboxes, environments, agents, miners) exists to produce patches and evaluate them reproducibly.

System overview

flowchart TB
    subgraph Authoring
        A1[task.yaml] --- A2[repo/ snapshot<br/>or git url+commit]
        A2 --- A3[hidden/ tests]
        A3 --- A4[solution.patch]
        M1[repogym mine<br/>git history] --> A1
        M2[repogym mutate<br/>fault injection] --> A1
    end

    A1 --> T[Task]
    T --> V[validate_task]
    T --> W[Workspace<br/>git baseline commit]

    subgraph Episode
        E[RepoEnv<br/>reset / step / state] --> W
        AG[Agent<br/>golden · shell · claude · yours] -->|Action| E
        E -->|Observation, reward| AG
        W --> S[Sandbox<br/>local · docker]
    end

    E -->|submit| G[CompositeGrader]
    G --> G1[ConstraintGrader]
    G --> G2[TestGrader<br/>JUnit XML]
    G --> G3[PerfGrader<br/>benchmark]
    G --> R[GradeResult<br/>score ∈ 0..1, patch]
    R --> TR[Trajectory JSON]
    R --> RP[Reports · leaderboard · JSONL export]

Components

Module	Responsibility	Key types
models.py	Pydantic schema for task.yaml and every result object. Strict (extra="forbid"), cross-field validated.	TaskSpec, TestSpec, PerfSpec, ConstraintSpec, GradingSpec, GradeResult, Trajectory, RunResult
task.py	Discover and load task directories; resolve paths; filter by id/type/tag.	Task, load_tasks
workspace.py	Materialise a task into an isolated directory, create a git baseline commit, track changes as diffs, apply patches, inject/remove hidden tests, restore protected files, reset.	Workspace
sandbox.py	Execute shell commands. LocalSandbox scrubs the environment to an allowlist; DockerSandbox bind-mounts the workspace into a fresh container with network off.	Sandbox, LocalSandbox, DockerSandbox
junit.py	Parse JUnit XML from any runner into {test_id: status}, registering pytest node ids, classname::name and bare names.	parse_junit_xml
metrics.py	Dependency-free AST metrics: cyclomatic complexity and function length.	function_metrics
graders/	Turn a workspace into a GradeComponent (objective or gate); combine components into a GradeResult.	TestGrader, PerfGrader, QualityGrader, ConstraintGrader, CompositeGrader
env.py	Gym-style environment: actions, observations, step limits, command allowlist, reward shaping, trajectory recording.	RepoEnv, Action, Observation
agents/	Things that drive an env to completion.	Agent, GoldenAgent, NoopAgent, ScriptedAgent, ShellAgent, ClaudeAgent
validate.py	Prove a task is a sound environment.	validate_task
runner.py	Run agents over task sets, serially or with a thread pool.	run_task, run_suite
mine/	Generate tasks from git history or by mutation.	HistoryMiner, MutationMiner
export.py, report.py	JSONL datasets, terminal/Markdown/HTML reports, leaderboards.
cli.py	Typer CLI over all of the above.

Lifecycle of an episode

sequenceDiagram
    participant Ag as Agent
    participant Env as RepoEnv
    participant WS as Workspace
    participant SB as Sandbox
    participant Gr as CompositeGrader

    Ag->>Env: reset()
    Env->>WS: materialize() → copy/clone, run setup, git commit baseline
    Env-->>Ag: Observation(problem statement + file tree)
    loop until submit or max_steps
        Ag->>Env: step(Read/Write/Edit/Run/Test/ListFiles)
        Env->>WS: read/write files (path-escape guarded)
        Env->>SB: run(command) for Run/Test (allowlisted, timed out)
        Env-->>Ag: Observation(text, reward=0 or shaped)
    end
    Ag->>Env: step(Submit)
    Env->>Gr: grade_result(workspace)
    Gr->>WS: diff() → patch snapshot
    Gr->>WS: restore(protected tests), inject hidden tests
    Gr->>SB: run tests → JUnit XML, run benchmark
    Gr-->>Env: GradeResult(score, passed, components)
    Env-->>Ag: Observation(done=True, reward=score)
    Env->>Env: write Trajectory JSON

Design decisions

Git as the change-tracking substrate. Every workspace becomes a git work tree with a baseline commit after repo.setup has run. That gives diffs, resets, restore(files) and change statistics for free, works for any language, and means the patch is the canonical artefact of an episode. Build artefacts produced by setup are folded into the baseline so they never pollute diffs. The git metadata lives outside the working tree (<workspace>.scratch/git, passed explicitly as GIT_DIR), so an agent cannot blind the graders by deleting .git; if the metadata is destroyed anyway, git calls fail closed and the submission scores 0.

Objectives and gates. Every grader component has a role. Objectives (target tests, benchmarks, quality limits on refactor tasks) contribute weighted score; gates (regression tests, scope and pattern rules, quality limits elsewhere) contribute nothing but zero the score when they fail. This is the rule that makes "an empty submission scores 0" true for all four task types, which validate checks.

JUnit XML as the test contract. Instead of parsing each runner's stdout, repogym asks the test command to write JUnit XML ({junit} placeholder). pytest, node --test, jest, mocha, cargo-nextest, gradle, maven and go test (via go-junit-report) all support it, so per-test fail_to_pass / pass_to_pass rewards work everywhere.

Verification is a separate, explicit command. validate_task runs the baseline, the noop grade, the golden grade and (optionally) repeated golden grades. A task that is not validated cannot be trusted as a reward signal; CI runs repogym validate tasks on every push.

Reward hacking is treated as a design constraint, not an afterthought.

Attack	Defence
Edit the visible tests so they pass	Files referenced by fail_to_pass/pass_to_pass are restored to baseline before grading; the tampering is recorded as a constraint violation (a gate, so the score is 0)
Overfit to the visible tests	hidden/ tests are copied in only at grade time and removed afterwards; the test action never shows them, not even hidden pass_to_pass ids
Modify unrelated files / delete code	allowed_files, protected_files, max_files_changed, max_lines_changed gates
Fake the benchmark timing (monkeypatch the clock)	benchmarks are timed externally (wall clock of the process) by default; non-finite / zero timings score 0
Fast but wrong perf solution	passed requires every grader to pass; a broken pass_to_pass test is a failed gate
Trivial refactor (rename only)	max_cyclomatic_complexity / max_function_length are measured on the result
Delete or rewrite .git to blind the graders	git metadata lives outside the working tree; missing metadata makes grading fail closed with score 0
Submit nothing and collect constraint credit	gates never contribute credit; validate asserts the noop score is exactly 0
Shell out to the network / destructive commands	run action is allowlisted per shell segment, quote-aware, rejects command substitution and known-dangerous tools. This is a guardrail, not a security boundary: an allowlisted interpreter runs anything. Use DockerSandbox (network off) for untrusted agents
Secrets leakage	LocalSandbox passes only an allowlist of environment variables

Sparse reward by default. RL practitioners want the terminal reward to be the graded score. Shaping (env.reward_shaping) is opt-in and bounded (at most 0.1 total, only for improvements on visible tests), so cumulative return stays interpretable.

Pydantic everywhere. Task specs are strict (extra="forbid") with cross-field validation (a perf task must have a perf section; every task must have some reward signal). Results are models too, so trajectories, grades and run results serialise to JSON without custom code.

Miners produce validated tasks by construction. Both miners execute the test-suite before and after the fix and only emit a task when the fail-to-pass set is non-empty and the failure ratio is sane. The mutation miner applies faults textually at AST-reported positions so formatting and comments survive and the golden patch is minimal.

Extension points

• Grader: subclass Grader, return a GradeComponent; add it to a CompositeGrader with a weight.
• Agent: subclass Agent and implement solve(env, obs); or wrap a CLI with ShellAgent.
• Sandbox: implement run(command, cwd, timeout, env) -> CommandResult and available().
• Action: Action is a plain Pydantic model with a kind discriminator; new kinds need a _do_<kind> handler on RepoEnv.
• Task source: RepoSpec.source is local or git; other sources (tarballs, registries) only need a new branch in Workspace.materialize.

Repository layout

src/repogym/         library
tasks/               example tasks (validated in CI)
tests/               pytest suite (125 tests, ~94% statement coverage)
docs/ + mkdocs.yml   documentation site
website/index.html   landing page
examples/            runnable scripts (custom agent, custom grader, RL loop)
.github/workflows/   CI (lint, mypy, tests, validate tasks), Pages, Release