Authoring tasks¶

A good task is small, unambiguous and verifiable: the problem statement, the tests and the golden solution must agree. This guide builds one of each type.

Scaffold¶

repogym new tasks/my-task --type bugfix

tasks/my-task/
├── task.yaml
├── repo/              # put the repository snapshot here
│   ├── src/
│   └── tests/test_example.py
├── hidden/            # optional: tests copied in only at grading time
└── solution.patch     # the golden fix (or a solution/ directory)

Then iterate with repogym validate tasks/my-task -v until every check passes.

Bug-fix task¶

Put the buggy code in repo/.
Add a test that reproduces the bug (it must fail on the snapshot).
Write the fix as solution.patch: fix a copy of repo/ and run repogym solution tasks/my-task --from /path/to/fixed-copy (test files are excluded automatically).
List the reproducing test(s) in fail_to_pass and the rest in pass_to_pass.

tests:
  command: "python -m pytest -q -p no:cacheprovider --junitxml={junit}"
  fail_to_pass: [tests/test_stats.py::test_median_even]
  pass_to_pass: [tests/test_stats.py::test_mean, tests/test_stats.py::test_median_odd]
constraints:
  allowed_files: ["stats.py"]

Hidden tests

Put extra assertions in hidden/tests/... and list them under tests.hidden. They are injected only while grading, so an agent that overfits to the visible test (if values == [4, 1, 3, 2]: return 2.5) still fails.

Feature task¶

The visible tests describe the interface; hidden tests cover edge cases. Because the tests import a symbol that does not exist yet, pytest reports a collection error and skips the whole file. Keep the rest of the suite running with:

tests:
  command: "python -m pytest -q -p no:cacheprovider --continue-on-collection-errors --junitxml={junit}"

fail_to_pass entries that are missing from the JUnit output count as failing at baseline, which is what you want.

Refactor task¶

Behaviour is fixed by pass_to_pass; the reward comes from constraints:

type: refactor
tests:
  pass_to_pass: [tests/test_orders.py::test_basic_us_order, ...]
constraints:
  paths: ["orders.py"]                 # where the AST metrics are computed
  allowed_files: ["orders.py"]
  max_cyclomatic_complexity: 8         # on refactor tasks these limits are the objective
  max_function_length: 30
  required_patterns: ["def process_order\\("]   # the public API must survive (gate)
grading:
  weights: {quality: 1.0}              # tests are a gate here: all pass_to_pass must hold
solution:
  dir: solution                        # whole-file replacement is easier for refactors

validate checks that the unmodified repository violates at least one constraint (otherwise doing nothing would be rewarded) and that the golden solution satisfies all of them.

Performance task¶

type: perf
tests:
  pass_to_pass: [tests/test_dedupe.py::test_dedupe_keeps_first_and_order, ...]
perf:
  command: "python bench.py"     # timed externally (wall clock) by default
  min_speedup: 3.0
  runs: 3                         # interleaved baseline/candidate runs, minimum used
constraints:
  allowed_files: ["dedupe.py"]
  protected_files: ["bench.py"]  # the benchmark itself must not be edited
grading:
  weights: {perf: 1.0}           # tests and constraints are gates

The baseline timing is measured in a fresh workspace and cached per task for the process, so grading many submissions does not re-benchmark the baseline each time. Keep benchmarks between 0.5 s and a few seconds: long enough that interpreter start-up is noise, short enough for RL throughput. repogym times the whole process from outside, so the agent cannot fake the number by patching the clock.

Non-Python repositories¶

Any runner that emits JUnit XML works. Test ids are whatever appears in the XML:

Node (built-in runner)Rust (cargo-nextest)GoJava (Gradle)

language: javascript
tests:
  command: "node --test --test-reporter=junit --test-reporter-destination={junit}"
  fail_to_pass: ["strips trailing separators"]      # bare test names
env:
  allowed_commands: ["node", "npm", "ls", "cat", "grep"]

language: rust
repo:
  setup: ["cargo build --tests"]
tests:
  command: "cargo nextest run --profile ci --message-format junit > {junit}"

language: go
tests:
  command: "go test ./... -v 2>&1 | go-junit-report > {junit}"

language: java
tests:
  command: "gradle test --quiet; cat build/test-results/test/*.xml > {junit}"

Use repogym validate --sandbox docker:<image> to pin the toolchain.

Writing good problem statements¶

Describe observable behaviour and expected behaviour, not the fix.
Quote the failing call and the wrong output.
State what must not change.
Mention constraints the grader enforces (allowed files, complexity limits).
Do not mention hidden test names.

Checklist before committing a task¶

[ ] repogym validate <task> --repeat 2 passes (this includes "noop scores 0")
[ ] the repository snapshot has no .git, virtualenvs or build artefacts
[ ] allowed_files / protected_files cover the tests and benchmarks
[ ] the statement is solvable without reading the golden patch