open source · MIT · source install

Turn any git repository into a verifiable RL environment for coding agents.

A task.yaml, a repo snapshot and a golden patch become a gym-style environment with executable graders, validated end to end before an agent ever sees it.

$pip install git+https://github.com/shi1720/repogym.git

View on GitHub Read the docs

zsh — repogym

$ pip install git+https://github.com/shi1720/repogym.git

noop 0.00

golden0.00

Why

The bottleneck isn't the model. It's the environments.

Frontier labs train coding agents with RL on real software tasks: fix a bug, add a feature, refactor, make it faster. Every one of those tasks needs the same five things, and today they get hand-built per project.

today

One harness per repo

A SWE-bench-style harness, a Dockerfile and a pile of ad-hoc scripts for each project. The environment is the slowest, least reusable part of the pipeline.

today

Nothing proves the task is real

Does the test actually fail before the fix? Does a known-good patch score 1? Is grading deterministic? Usually nobody has checked.

today

Reward hacking is one `git checkout` away

If the agent can see or edit the tests, the reward measures the agent's creativity, not its engineering.

repogym makes it a task.yaml and one command.

A small, batteries-included library: reproducible snapshot, executable oracle, golden solution, reward-hacking protection and a gym interface, all validated by repogym validate.

Features

Everything a training environment needs. Nothing you have to write twice.

Verified by construction

repogym validate runs every task through the checks a hand-rolled harness usually skips, and refuses tasks that fail any of them.

Hidden tests are not leaked into the repo
fail_to_pass tests fail at baseline
pass_to_pass tests pass at baseline
Doing nothing scores exactly 0
The golden solution scores exactly 1
Deterministic grading (--repeat N)

Declarative tasks

A task is a folder: task.yaml + repo/ snapshot (or git URL + commit) + optional hidden/ tests + solution.patch. Four types: bugfix, feature, refactor, perf.

Gym-style environment

env.reset() / env.step(action) with Read, Write, Edit, Run, Test, ListFiles, Submit. Sparse reward in [0,1] on submit, optional dense shaping from visible tests, step limits, command allowlist.

Composable graders

TestGrader reads JUnit XML from any runner. PerfGrader scores benchmark speed-up with log-scaled partial credit. ConstraintGrader enforces file scope, regex patterns, complexity and diff size. Weighted composite; every component must pass.

Anti reward-hacking

Protected test files are restored before grading, hidden tests are injected only at grade time, scope constraints bound the diff, a safety filter screens commands, and secrets are scrubbed from the sandbox environment.

Task mining

repogym mine <repo> turns real bug-fix commits into verified tasks: overlay the commit's tests on the parent, run, diff. repogym mutate <repo> injects AST-guided faults (operator flips, off-by-one, boolean swaps) the suite catches, with the reverse patch as the golden solution.

Any agent

Built-in golden, noop, ShellAgent for any CLI (Claude Code, Aider, Codex, OpenHands: shell:aider --message {prompt} --yes), ClaudeAgent on the Anthropic SDK, or subclass Agent.

Sandboxes

LocalSandbox with zero setup, or DockerSandbox with a pinned image, network off, and memory and CPU limits.

Reports & export

Rich terminal tables, a leaderboard, a standalone HTML report, and JSONL export of tasks (SWE-bench-compatible fields) and trajectories as chat-style messages for SFT and RL pipelines.

Language-agnostic

Python first, but any runner that emits JUnit XML works: pytest, node --test, cargo, jest, gradle. Ships example tasks in Python and JavaScript.

repogym listrepogym inforepogym validaterepogym run --agent …repogym minerepogym mutaterepogym exportrepogym reportrepogym newrepogym doctor

How it works

From a folder on disk to a reward and a trajectory.

TASK

Task

task.yaml
repo/ snapshot
hidden/ tests
solution.patch

WORKSPACE

Workspace

git baseline commit
Local or Docker sandbox
secrets scrubbed

AGENT

Agent

env.reset()
env.step(action) …
Submit()

GRADERS

Graders

tests restored + hidden injected
TestGrader · PerfGrader
ConstraintGrader

OUTPUT

Reward + trajectory

reward ∈ [0, 1]
per-grader breakdown
trajectory JSON

loop reset → observe → step × N → submit → grade. The agent never sees hidden tests; protected tests are reset before grading.

Task types

Four kinds of work, each with a grader that fits.

bugfix

Fix a reported defect

Problem statement plus a failing test. The classic SWE-bench shape, and what mine and mutate produce.

graded by fail_to_pass flipping green while pass_to_pass stays green

feature

Implement something new

The spec lives in the problem statement; the oracle lives in hidden tests the agent cannot read.

graded by hidden tests injected at grade time, plus scope constraints

refactor

Improve structure, keep behaviour

Every existing test must keep passing while the code gets measurably simpler.

graded by ConstraintGrader: max complexity, function length, forbidden and required patterns

perf

Make it faster

A benchmark command and a target speed-up. Correctness tests still gate the reward.

graded by PerfGrader: log-scaled partial credit toward the target speed-up

Quickstart

Small API. Plain files. One command to prove it all works.

from repogym import Task, RepoEnv, Read, Edit, Submit

env = RepoEnv(Task.load("tasks/bugfix-median"))
obs = env.reset()                       # problem statement + file tree
obs = env.step(Read("stats.py"))
obs = env.step(Edit("stats.py",
                    "        return ordered[mid]\n    return ordered[mid]\n",
                    "        return ordered[mid]\n    return (ordered[mid - 1] + ordered[mid]) / 2\n"))
obs = env.step(Submit())
print(obs.reward, obs.done)             # 1.0 True

id: bugfix-median
title: "median() returns the wrong value for even-length input"
type: bugfix
difficulty: easy
language: python
problem_statement: |
  `stats.median()` should return the mean of the two middle values ...
tests:
  command: "python -m pytest -q --junitxml={junit}"
  hidden: [tests/test_hidden_median.py]
  fail_to_pass: [tests/test_stats.py::test_median_even, tests/test_hidden_median.py::test_median_two_values]
  pass_to_pass: [tests/test_stats.py::test_mean]
constraints:
  allowed_files: ["stats.py"]
grading:
  weights: {tests: 1.0}
solution:
  patch: solution.patch

# prove every task is solvable, unleaked, not rewarding no-ops, and deterministic
$ repogym validate tasks --repeat 3

# run an agent: built-in golden / noop, any CLI, or the Claude SDK loop
$ repogym run tasks --agent golden
$ repogym run tasks --agent "shell:aider --message {prompt} --yes"
$ repogym run tasks --agent claude

# mine verified tasks from real bug-fix commits, or synthesise them
$ repogym mine https://github.com/org/project
$ repogym mutate ./project

# reports and datasets
$ repogym report results.json -o report.html
$ repogym export tasks -o tasks.jsonl

Every episode is recorded as a JSON trajectory. repogym export turns tasks and trajectories into JSONL your training pipeline already understands.

Use cases

One environment format, four jobs.

RL post-training

Sparse, verified reward on submit, optional dense shaping from visible tests, and hard limits on steps and commands. Mine or mutate thousands of tasks from repos you already have.

Agent evals & regression benchmarks

Run the same task set against Claude Code, Aider, Codex or your own agent. Leaderboard, HTML report, deterministic grading you can diff across releases.

SFT trajectories

Every episode is a chat-style message log with tool calls and rewards. Filter to reward == 1.0 and export JSONL.

Interview & competition tasks

Hidden tests, protected files and a golden solution make a task that is fair to grade and hard to game, for humans as well as agents.

Compared

Versus the harness you would otherwise write.

Hand-rolled harness

repogym

Verify

Trust that the test fails before the fix and passes after. Nobody re-checks after the repo moves.

repogym validate proves unleaked, fail-to-pass, pass-to-pass, noop = 0, golden = 1, deterministic with --repeat.

Mine

Curate issues and PRs by hand, one Dockerfile at a time.

repogym mine from real commits; repogym mutate for synthetic faults. Both come out validated.

Gym API

Bespoke glue between the agent, the shell and the grading script.

reset() / step() / Submit(), typed actions, JSON trajectories, step and command limits.

Any agent

The harness is coupled to whichever agent it was written for.

Built-ins, shell:<any cli>, ClaudeAgent, or subclass Agent.

Any language

Python-only, or a rewrite per ecosystem.

Anything that emits JUnit XML: pytest, node --test, cargo, jest, gradle.

Point it at a repo. Ship a verified environment.

$pip install git+https://github.com/shi1720/repogym.git

Read the docs Star on GitHub PyPI

Turn any git repository into a verifiable RL environment for coding agents.

The bottleneck isn't the model. It's the environments.

One harness per repo

Nothing proves the task is real

Reward hacking is one git checkout away

Everything a training environment needs. Nothing you have to write twice.

Verified by construction

Declarative tasks

Gym-style environment

Composable graders

Anti reward-hacking

Task mining

Any agent

Sandboxes

Reports & export

Language-agnostic

From a folder on disk to a reward and a trajectory.

Task

Workspace

Agent

Graders

Reward + trajectory

Four kinds of work, each with a grader that fits.

Fix a reported defect

Implement something new

Improve structure, keep behaviour

Make it faster

Small API. Plain files. One command to prove it all works.

One environment format, four jobs.

RL post-training

Agent evals & regression benchmarks

SFT trajectories

Interview & competition tasks

Versus the harness you would otherwise write.

Point it at a repo. Ship a verified environment.

Reward hacking is one `git checkout` away