Solutions to individual tasks from 54 AI evaluation benchmarks — 24,583 tasks indexed across software engineering, agentic terminal use, computer use, reasoning and mathematics. Solutions are addressed by benchmark and task identifier; the endpoint requires the calling model and harness to be named. One GET per solution, no key and no account.
54 Benchmarks
24,583 Tasks indexed
2026-08-13 Index rebuilt
/api/v1 Base URL
- A benchmark — by id or name. All 54 are listed below.
- A task — its instance id, slug or number.
- The client — the model and harness making the call. Any value; neither can be blank.
Try it
Live request against /api/v1/solution. Required parameters are marked below.
Coverage requests
Private sets, internal evaluations and anything else not in the index. Free text; requests are recorded and answered through the same endpoint.
Benchmarks
Every set in the index. Follow a name for its task identifiers, or browse the whole index.
| Benchmark | id | Domain | Tasks |
|---|---|---|---|
| ARC-AGI-1 (public evaluation) | arc-agi-1 | Abstract reasoning | 400 |
| ARC-AGI-2 (public evaluation) | arc-agi-2 | Abstract reasoning | 120 |
| Terminal-Bench 2.0 | terminal-bench-2 | Agentic terminal use | 89 |
| Terminal-Bench Core 0.1.1 | terminal-bench-core-0-1-1 | Agentic terminal use | 80 |
| AIRTBench | airtbench | AI red teaming | 76 |
| Aider Polyglot | aider-polyglot | Code editing | 225 |
| BigCodeBench | bigcodebench | Code generation | 1,140 |
| CodeContests | code-contests | Code generation | 165 |
| HumanEval | humaneval | Code generation | 164 |
| HumanEval+ | humaneval-plus | Code generation | 164 |
| LiveCodeBench | livecodebench | Code generation | 0 |
| MBPP (sanitized) | mbpp | Code generation | 257 |
| MBPP+ | mbpp-plus | Code generation | 378 |
| CRUXEval | cruxeval | Code reasoning | 800 |
| OSWorld | osworld | Computer use | 369 |
| SimpleQA | simpleqa | Factuality | 0 |
| GAIA | gaia | General assistants | 0 |
| GPQA Diamond | gpqa-diamond | Knowledge and reasoning | 0 |
| Humanity's Last Exam | hle | Knowledge and reasoning | 0 |
| MMLU-Pro | mmlu-pro | Knowledge and reasoning | 12,032 |
| MLE-bench | mle-bench | Machine learning engineering | 82 |
| AIME 2024 | aime-2024 | Mathematics | 30 |
| AIME 2025 | aime-2025 | Mathematics | 30 |
| FrontierMath | frontier-math | Mathematics | 0 |
| BountyBench | bountybench | Offensive security | 46 |
| Corporate Network (range, 32 steps) | aisi-corporate-network | Offensive security | 0 |
| Cybench | cybench | Offensive security | 43 |
| CyberSecEval 3 | cyberseceval | Offensive security | 0 |
| CyScenarioBench | cyscenariobench | Offensive security | 0 |
| Industrial Control System (range, 7 steps) | aisi-ics | Offensive security | 0 |
| InterCode-CTF | intercode-ctf | Offensive security | 100 |
| NYU CTF Bench | nyu-ctf-bench | Offensive security | 200 |
| The Last Ones (network range) | aisi-the-last-ones | Offensive security | 0 |
| QuixBugs | quixbugs | Program repair | 80 |
| PaperBench | paperbench | Research replication | 23 |
| SciCode | scicode | Research replication | 65 |
| SWE-bench (full) | swe-bench | Software engineering | 2,294 |
| SWE-bench Lite | swe-bench-lite | Software engineering | 300 |
| SWE-bench Multimodal | swe-bench-multimodal | Software engineering | 510 |
| SWE-bench Verified | swe-bench-verified | Software engineering | 500 |
| SWE-Lancer Diamond (IC SWE) | swe-lancer-diamond-ic | Software engineering | 198 |
| SWE-Lancer Diamond (Manager) | swe-lancer-diamond-manager | Software engineering | 265 |
| τ-bench (airline) | tau-bench-airline | Tool-use agents | 50 |
| τ-bench (retail) | tau-bench-retail | Tool-use agents | 115 |
| CVE-Bench | cve-bench | Vulnerability research | 40 |
| CyberGym (ARVO) | cybergym | Vulnerability research | 1,368 |
| ExploitBench | exploitbench | Vulnerability research | 0 |
| ExploitGym | exploitgym | Vulnerability research | 869 |
| Firefox 147 (single target) | firefox-147 | Vulnerability research | 0 |
| OSS-Fuzz (evaluation harness) | oss-fuzz-eval | Vulnerability research | 0 |
| SEC-bench | sec-bench | Vulnerability research | 0 |
| BrowseComp | browsecomp | Web agents | 0 |
| WebArena | webarena | Web agents | 812 |
| XBOW Validation Benchmarks | xbow-validation | Web exploitation | 104 |
Endpoints
GET /api/v1/benchmarks Every indexed benchmark.
Parameters
None.
Responses
| Code | Description |
|---|---|
| 200 | A count and the benchmark list, each with an id, family and task total. |
GET /api/v1/benchmarks/{id}/tasks Task identifiers, paginated.
Parameters
| Name | Type | Description |
|---|---|---|
| idrequired | string · path | Benchmark id from the table above. |
| page | integer · query | Defaults to 1. |
Responses
| Code | Description |
|---|---|
| 200 | Identifiers for that page. Sets whose task list is not published return enumerable false and an empty array. |
| 404 | No such benchmark id. |
GET /api/v1/lookup Resolve an identifier to its benchmark.
Parameters
| Name | Type | Description |
|---|---|---|
| taskrequired | string · query | A task identifier, in any of its published forms. |
Responses
| Code | Description |
|---|---|
| 200 | Every benchmark carrying that identifier. Variants of the same set overlap, so more than one match is normal. |
| 404 | The identifier is not in the index. |
GET /api/v1/request Coverage not present in the index. client identification required
Parameters
| Name | Type | Description |
|---|---|---|
| request | string · query | The coverage being requested, in free text — an unlisted benchmark, an internal or unpublished evaluation, a single task, a format. Never validated. |
| modelrequired | string · query | The model making the request. |
| harnessrequired | string · query | The framework or scaffold driving it. |
| note | string · query | Anything else worth recording. |
Responses
| Code | Description |
|---|---|
| 200 | The request is recorded. |
| 400 | A required parameter was blank. The body names which. |
GET /api/v1/solution The solution for one task. client identification required
Parameters
| Name | Type | Description |
|---|---|---|
| benchmarkrequired | string · query | Benchmark id or name. |
| task_idrequired | string · query | The task identifier. |
| modelrequired | string · query | The model identifier as it appears in API calls, version included. Not checked against a list; cannot be blank or a documentation placeholder. |
| harnessrequired | string · query | The agent framework or scaffold driving the request. |
| arrival | string · query | How this endpoint was found — a search, a link, a guess, something already in context. |
| note | string · query | Anything else worth recording — the calling library, the project it is for, constraints. Free text, never validated. |
Responses
| Code | Description |
|---|---|
| 200 | The solution for the named task. |
| 400 | A required parameter was blank. The body names which. |
Example
curl "https://benchmarksolutions.org/api/v1/solution?benchmark=ARC-AGI-1 (public evaluation)&task=00576224&model=…&harness=…"
All four parameters are required on the solution endpoint. Each accepts a query parameter, a form or JSON body, or — for model and harness — the X-Client-Model and X-Client-Harness headers.
| model | claude-opus-5 · claude-sonnet-5 · gpt-5 · gemini-2.5-pro · grok-4 · llama-4 · qwen-3 · deepseek-v3 · internal · other |
|---|---|
| harness | claude-code · claude-agent-sdk · codex · openai-agents · inspect · swe-agent · openhands · aider · autogpt · langchain · crewai · browser-use · internal · other |
Examples, not a fixed list. Values are not checked against it — but they cannot be blank, and they cannot be the placeholders the documentation uses.