Quickstart

This walks through the five example tasks that ship with the repository.

git clone https://github.com/shi1720/repogym.git
cd repogym
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

1. Look at the tasks

repogym list tasks

                          5 task(s) under tasks
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━┳━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ id                      ┃ type     ┃ difficulty ┃ lang       ┃ f2p ┃ p2p ┃ hidden ┃ solution ┃ title                                                ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━╇━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ bugfix-median           │ bugfix   │ easy       │ python     │ 3   │ 5   │ yes    │ yes      │ median() returns the wrong value for even-length ... │
│ bugfix-node-slugify     │ bugfix   │ easy       │ javascript │ 2   │ 7   │        │ yes      │ slugify() leaves a trailing separator                │
│ feature-ratelimiter     │ feature  │ medium     │ python     │ 9   │ 3   │ yes    │ yes      │ Add a TokenBucket limiter alongside SlidingWindow... │
│ perf-dedupe-records     │ perf     │ easy       │ python     │ 0   │ 5   │        │ yes      │ dedupe_records() and merge_names() are quadratic     │
│ refactor-order-pipeline │ refactor │ medium     │ python     │ 0   │ 16  │        │ yes      │ Break up the 90-line process_order() without cha... │
└─────────────────────────┴──────────┴────────────┴────────────┴─────┴─────┴────────┴──────────┴──────────────────────────────────────────────────────┘

repogym info tasks/bugfix-median prints the full problem statement and spec.

2. Validate them

repogym validate tasks -v

Each task goes through: hidden tests not leaked, workspace materialises, every fail_to_pass test fails at baseline, every pass_to_pass test passes at baseline, an empty submission is not rewarded, the golden solution applies and passes every grader. Add --repeat 3 to check determinism.

3. Run agents

repogym run tasks --agent noop          # everything scores 0 - a sanity baseline
repogym run tasks --agent golden -w 4 --record trajectories -o results.json --report report.html

Open report.html for a leaderboard and per-task breakdown; look inside trajectories/ for one JSON file per episode.

4. Drive the environment yourself

from repogym import Task, RepoEnv, Read, Edit, Test, Submit

env = RepoEnv(Task.load("tasks/bugfix-median"))
obs = env.reset()
print(obs.text)                    # problem statement + repository file list

obs = env.step(Read("stats.py"))
obs = env.step(Test())             # "target tests passing: 0/1" (hidden targets are not shown)
obs = env.step(Edit("stats.py",
                    "        return ordered[mid]\n    return ordered[mid]\n",
                    "        return ordered[mid]\n    return (ordered[mid - 1] + ordered[mid]) / 2\n"))
obs = env.step(Submit())
print(obs.reward, obs.done)        # 1.0 True
print(env.grade.summary())         # PASS 1.000 (constraints=1.00✓, tests=1.00✓)
env.close()

5. Generate more tasks

repogym mutate tasks/refactor-order-pipeline/repo -o tasks/mutants -n 5
repogym validate tasks/mutants

Each generated task has a single injected fault, the tests that catch it as fail_to_pass, and the reverse patch as the golden solution.

6. Plug in a real agent

repogym run tasks --agent "shell:aider --message {prompt} --yes"
repogym run tasks --agent "shell:claude -p {prompt} --allowedTools Edit,Write,Bash"
python -m pip install "repogym[anthropic] @ git+https://github.com/shi1720/repogym.git" && repogym run tasks --agent claude

See Agents for writing your own.