Graders and rewards

Grading is a pure function of the workspace after the agent submits. Every grader returns a GradeComponent with a score in [0, 1], a boolean passed, a role, details and human-readable messages. CompositeGrader combines them.

Objectives and gates

Role	Contributes	Examples
objective	its weighted score	target (fail_to_pass) tests, benchmarks, quality limits on refactor tasks
gate	nothing, but a failing gate zeroes the score	regression-only tests, scope/pattern constraints, quality limits on non-refactor tasks

This split is what makes "an empty submission scores 0" hold for every task type, and repogym validate enforces exactly that. Weights in grading.weights apply to objectives only.

flowchart TB
    S[submit] --> P[snapshot patch + changed files]
    P --> C[ConstraintGrader]
    C --> T[TestGrader]
    T --> Q[QualityGrader]
    Q --> F[PerfGrader]
    F --> R[weighted score + passed]

score  = Σ wᵢ · sᵢ / Σ wᵢ            over objectives
score  = 0                          if any gate failed
passed = all(passedᵢ) and score ≥ grading.pass_threshold

The order matters: constraints run first so that tampering with protected test files is recorded before TestGrader restores those files.

TestGrader

1. Restore files referenced by fail_to_pass / pass_to_pass to baseline.
2. Inject hidden tests.
3. Run tests.command with {junit} pointing outside the repository.
4. Parse JUnit XML; remove hidden tests.

Situation	Score
k of n fail_to_pass pass, partial credit on	k / n
partial credit off, any fail_to_pass failing	0
any pass_to_pass broken, require_all_pass_to_pass on	0
any pass_to_pass broken, off	score × (1 − broken / total)
no fail_to_pass declared (refactor / perf)	1 if all pass_to_pass pass
no JUnit output / timeout	0 with a message

PerfGrader

A pristine baseline workspace is materialised once per task and sandbox and kept for the process. On every grade, baseline and candidate runs are interleaved (runs pairs) and the minimum of each side is used, so machine drift and additive noise cancel out.

By default (perf.timing: wall) repogym times the whole benchmark process from outside, so code the agent controls cannot fake the number. Make the benchmark run for at least half a second so interpreter start-up is negligible. With timing: reported the benchmark's own {"seconds": x} output is trusted instead (more precise, but an agent-reachable module could monkeypatch the clock). Non-finite or non-positive timings score 0.

speedup = baseline / candidate
score   = 0                                                  if speedup < noise_floor
        = min(1, log(speedup / noise_floor) / log(min_speedup / noise_floor))
passed  = speedup ≥ min_speedup

Log scaling gives smooth partial credit above the noise floor (default 1.15×); with min_speedup: 4, a 2× speed-up scores about 0.45.

ConstraintGrader (gate)

Each declared rule is one check; score = 1 − violations / checks; passed requires zero violations. Rules: protected files (always on), allowed_files, max_files_changed, max_lines_changed, forbidden_patterns (on changed files) and required_patterns (on paths).

QualityGrader

max_function_length and max_cyclomatic_complexity are measured per function on constraints.paths (Python AST). Each limit is one rule and the score is the fraction of rules with no violations. On refactor tasks this component is the objective (default weight 1); on other task types it is a gate.

Cyclomatic complexity counts if/elif, loops, except, boolean operators, conditional expressions, comprehensions (+ their ifs), assert, async for and match cases, per function (with is not counted); nested functions are measured separately.

Writing a grader

from repogym.graders import Grader, CompositeGrader, TestGrader, ConstraintGrader
from repogym.models import GradeComponent

class NoTodoGrader(Grader):
    name = "no_todo"
    def grade(self, workspace):
        bad = [f for f in workspace.changed_files() if "TODO" in workspace.read(f)]
        return GradeComponent(name=self.name, score=float(not bad), passed=not bad,
                              messages=[f"TODO in {f}" for f in bad])

grader = CompositeGrader([ConstraintGrader(), TestGrader(), NoTodoGrader()],
                         weights={"tests": 0.8, "no_todo": 0.2})   # set role="gate" to block instead
result = grader.grade_result(workspace)

To use a custom grader from the environment, subclass RepoEnv and override _do_submit, or grade the workspace yourself after env.step(Submit()) using env.workspace with keep_workspace=True.

Reading a GradeResult

result.score          # 0.0 - 1.0
result.passed
result.components     # [GradeComponent(name="constraints", ...), ...]
result.component("tests").details["fail_to_pass"]   # {test_id: "passed" | "failed" | ...}
result.patch          # unified diff of the submission
result.files_changed
result.summary()      # "PASS 1.000 (constraints=1.00✓(gate), tests=1.00✓)"