task.yaml reference

All keys are validated by Pydantic; unknown keys are rejected.

Top level

Key	Type	Default	Notes
id	str	directory name	[a-z0-9][a-z0-9._-]{1,79}
title	str	required	one line
type	bugfix · feature · refactor · perf	required
difficulty	easy · medium · hard	medium
language	str	python	informational
tags	list[str]	[]	filter with --tag on list/validate/run, or load_tasks(tags=)
problem_statement	str	required	shown to the agent
hints	list[str]	[]	shown to the agent
metadata	dict	{}	free-form (miners store provenance here)

repo

Key	Default	Notes
source	local	local copies path; git clones url at commit
path	repo	relative to the task dir
url, commit		required / recommended for git
setup	[]	shell commands run once after materialisation (pip install -e ., npm ci); their output becomes part of the baseline
setup_timeout	600	seconds

tests

Key	Default	Notes
command	python -m pytest -q -p no:cacheprovider --junitxml={junit}	{junit} is replaced with a path outside the repo
hidden	[]	paths under <task>/hidden/ copied into the workspace only at grading time
fail_to_pass	[]	must fail at baseline and pass after
pass_to_pass	[]	must pass on both sides
timeout	300	seconds

Test ids accept any of the forms the JUnit parser registers: pytest node ids (tests/test_x.py::test_a, tests/test_x.py::TestK::test_b), classname::name (tests.test_x::test_a) or the bare name.

perf (required for type: perf)

Key	Default	Notes
command	required	runs the benchmark once; lower is better
min_speedup	1.5	full credit at or above this
runs	3	interleaved baseline/candidate runs; the minimum of each is used
noise_floor	1.15	speed-ups below this factor score 0
timeout	300	per run
timing	wall	wall: repogym times the process externally (tamper-proof). reported: trust {"seconds": x} / a bare float printed by the benchmark

constraints

Key	Default	Notes
paths	["**/*.py"]	files that AST metrics and required_patterns are computed on
allowed_files	[]	if set, changed files must match one of these globs
protected_files	[]	must never change; files referenced by fail_to_pass/pass_to_pass are always protected
forbidden_patterns	[]	regexes that must not appear in changed files
required_patterns	[]	regexes that must appear somewhere in paths
max_function_length		lines, per function (Python); objective on refactor tasks, gate otherwise
max_cyclomatic_complexity		McCabe, per function (Python); objective on refactor tasks, gate otherwise
max_files_changed
max_lines_changed		added + deleted

Globs use ** sensibly: **/*.py matches a.py and pkg/a.py; src/** matches everything under src/.

grading

Key	Default	Notes
weights	{tests: 1.0}	per objective grader (tests, perf, quality); normalised. perf is added automatically for perf tasks and quality for refactor tasks with limits. Gates (constraints, regression-only tests, quality on non-refactor tasks) ignore weights and zero the score when they fail
partial_credit	true	tests score = fraction of fail_to_pass passing
require_all_pass_to_pass	true	a single regression zeroes the tests score (else proportional penalty)
pass_threshold	1.0	minimum weighted score for passed

solution

Exactly one of:

Key	Notes
patch	unified diff applied with git apply (falls back to patch -p1)
dir	directory overlaid onto the workspace

env

Key	Default	Notes
max_steps	50	auto-submits when reached
step_timeout	120	seconds for run/test actions
allowed_commands	python, python3, pytest, node, npm, cargo, go, ls, cat, grep, find, head, tail, wc, sed, awk, echo, pwd, diff, tree, rg	first token of every shell segment must match
max_output_chars	20000	observations are middle-truncated
reward_shaping	false	test action yields up to 0.1 total for improvements on visible fail_to_pass

Full example

id: feature-ratelimiter
title: "Add a TokenBucket limiter alongside SlidingWindowLimiter"
type: feature
difficulty: medium
language: python
tags: [concurrency, api-design, feature]
problem_statement: |
  `ratelimit.py` ships a `SlidingWindowLimiter`. Add a second strategy,
  `TokenBucket`, with this interface:

      TokenBucket(capacity: float, refill_rate: float, clock=None)
      .tokens(key) -> float               # current balance after refilling
      .try_acquire(key, tokens=1.0) -> bool
      .acquire(key, tokens=1.0) -> None   # raises RateLimitExceeded(key, retry_after)

  Semantics:
  - every key gets its own bucket that starts full
  - tokens refill continuously at `refill_rate` per second (fractional tokens
    accumulate) and never exceed `capacity`
  - `retry_after` is the time until enough tokens are available
  - `capacity <= 0`, `refill_rate <= 0`, `tokens <= 0` or `tokens > capacity`
    raise `ValueError`
  - `clock` defaults to `time.monotonic` and is injectable for tests

  The visible tests in `tests/test_token_bucket.py` currently fail with an
  ImportError. Hidden tests exercise per-key isolation, fractional refill and
  multi-token acquisition.
hints:
  - Store (tokens, last_refill_timestamp) per key and refill lazily on access.
repo:
  source: local
  path: repo
tests:
  command: "python -m pytest -q -p no:cacheprovider --continue-on-collection-errors --junitxml={junit}"
  hidden:
    - tests/test_token_bucket_hidden.py
  fail_to_pass:
    - tests/test_token_bucket.py::test_bucket_starts_full
    - tests/test_token_bucket.py::test_bucket_refills_over_time
    - tests/test_token_bucket.py::test_bucket_never_exceeds_capacity
    - tests/test_token_bucket.py::test_acquire_raises_with_retry_after
    - tests/test_token_bucket.py::test_invalid_arguments
    - tests/test_token_bucket_hidden.py::test_keys_are_independent
    - tests/test_token_bucket_hidden.py::test_fractional_tokens_accumulate
    - tests/test_token_bucket_hidden.py::test_acquire_multiple_tokens
    - tests/test_token_bucket_hidden.py::test_cannot_request_more_than_capacity
  pass_to_pass:
    - tests/test_sliding_window.py::test_allows_up_to_limit
    - tests/test_sliding_window.py::test_window_slides
    - tests/test_sliding_window.py::test_check_raises_with_retry_after
constraints:
  allowed_files: ["ratelimit.py"]
  max_cyclomatic_complexity: 8
grading:
  weights: {tests: 1.0}
solution:
  patch: solution.patch
env:
  max_steps: 40