Mining tasks¶

Hand-written tasks are precious; RL needs thousands. repogym ships two miners that turn one repository into many verified tasks without an LLM in the loop.

From git history¶

repogym mine ./my-project --limit 500 -n 30 --out tasks/mined
repogym mine https://github.com/org/project --setup "pip install -e ." --test-command "python -m pytest -q --junitxml={junit}"

For each non-merge commit that touches both test files and source files:

check out the parent commit and overlay the commit's test files (the task's starting state: bug present, new tests visible);
run the tests; failing tests are fail_to_pass candidates;
check out the full commit and run again; tests that went from failing to passing become fail_to_pass, tests passing on both sides become pass_to_pass;
the source-only part of the diff becomes solution.patch; the commit message becomes the problem statement.

Commits with no failing tests, too many failures (broken build) or no fail-to-pass set are skipped, so every emitted task is verified by construction. Provenance (source_commit, parent_commit, changed files) is stored in metadata.

from repogym.mine import HistoryMiner
miner = HistoryMiner("my-project", test_command="npm test -- --reporters=jest-junit", setup=["npm ci"])
for cand in miner.candidate_commits(limit=100):
    cand = miner.evaluate(cand)
    if cand.accepted:
        miner.write_task(cand, Path("tasks/mined"))

By mutation¶

repogym mutate ./my-project -n 50 --sample 400 --seed 7 --out tasks/mutants

Starting from a repository whose tests pass, the miner enumerates candidate faults per Python source file:

Operator	Example
comparison flip	`<` ↔ `<=`, `>` ↔ `>=`, `==` ↔ `!=`
arithmetic swap	`+` ↔ `-`, `*` ↔ `/`
boolean swap	`and` ↔ `or`
off-by-one	`n` → `n + 1`, `n − 1`
boolean literal	`True` ↔ `False`
negation removal	`not x` → `x`

Each fault is applied textually at the AST-reported position (comments and formatting survive), the tests run, and a task is emitted only when some but not too many tests fail. Mutants that survive (no test catches them) are skipped, so the miner doubles as a mutation-testing report on your suite. Tasks with the same failing-test signature in the same file are de-duplicated.

Generated tasks are bugfix type, scoped to **/*.py, with the reverse diff as the golden solution and the mutation recorded in metadata.

After mining¶

repogym validate tasks/mined --repeat 2
repogym run tasks/mined --agent noop      # must all be 0
repogym export tasks/mined -o mined.jsonl

Problem statements from mining are mechanical ("these tests fail"). For human-quality statements, post-process the JSONL with an LLM; the tests remain the ground truth either way.