How repogym compares¶

repogym is a library, not a benchmark and not a platform. It is meant to sit underneath both.

	SWE-bench / SWE-Gym / R2E-Gym	Environment platforms (Prime Intellect Environments Hub, HUD, Mechanize, ...)	OpenEnv / Gymnasium	repogym
What it is	fixed datasets + evaluation harness	hosted services with curated environments	interface specifications	`pip install` library
Bring your own repo	no (fork the harness)	via their SDKs	n/a	yes, `task.yaml`
Task validation	one-off scripts	proprietary	n/a	`repogym validate` (baseline, golden, noop, leak, `--repeat` determinism)
Task generation	manual issue curation / paper-specific pipelines	proprietary	n/a	`mine` (git history) + `mutate` (fault injection)
Reward beyond tests	mostly no	varies	n/a	perf benchmarks, constraints, complexity, composite weights
Reward-hacking defences	test-patch replay	varies	n/a	protected files, hidden tests, scope, allowlists
Languages	Python-centric (multi-lingual variants exist)	varies	n/a	any JUnit-emitting runner
Agent interface	patch in, score out	platform SDK	`reset`/`step`	`reset`/`step` + patch grading + CLI wrappers
Runs offline / locally	with Docker	no	yes	yes (local or Docker)

When to use something else¶

You want to compare against published numbers → run the official SWE-bench harness; repogym can export SWE-bench-style JSONL but does not reproduce the official Docker images.
You need thousands of hosted, isolated sandboxes at once → pair repogym's tasks with a sandbox provider (Modal, E2B) by implementing the Sandbox protocol.
You need non-coding environments → OpenEnv/Gymnasium.