AI Model Test Results: 1,500 Runs, What We Found

In ES round 2, Opus scored highest among the four same-judge models at 83%; GPT scored highest raw at 98%, but was graded by a different judge. The judge swap showed a gap of 17 points on both Claude and Gemini outputs, so this is not a model ranking. Round 1 remains the page’s core, with its own sections below.

Which rounds have we run?

The bench covers professional prompts, tool-related tasks and separately tested Skills, with dated records and locked rubrics.

Prompt totals: 100 prompts, 1,500 runs and 7,500 verdicts, excluding Skills runs.

• October 19, 2026: round 1 established the English/Spanish baseline.
• October 19, 2026: round 2 added French/Portuguese prompts.
• October 20, 2026: Fable added on both existing prompt sets.
• October 20, 2026: Gemini added on both existing prompt sets.
• October 20, 2026: Excel / Microsoft 365 tasks tested.
• October 21, 2026: ES round 2 added as a separately locked set.
• October 19, 2026: English/Spanish Skills tested; Fable added later.
• October 21, 2026: French/Portuguese Skills tested and revised; Skills licensing published.

What did the latest round show?

Opus led the same-judge scores on 10 new Spanish prompts for engineers, travel agents, construction managers, event planners and interior designers in Mexico, Spain and Colombia. Separately locked rubrics covered 5 criteria per prompt; each model ran each prompt 3 times, producing 150 runs and 750 verdicts.

Tested October 21, 2026.

Opus, Gemini, Fable and Sonnet share gpt-6-astra as judge, allowing like-for-like comparison. GPT’s score requires the calibration caveat: on 6 runs per family, gpt-6-astra versus claude-opus-5-5 passed 77% versus 93% of Claude criteria, 93% versus 93% of GPT criteria, and 73% versus 90% of Gemini criteria. The reported gaps on Claude and Gemini were 17 points, calculated before rounding. An editor agent, not a human, agreed with 8 of 12 spot-checked verdicts; all 4 disagreements had the judge stricter.

Of 50 prompt-model results, 38 included a run missing a criterion; 0 prompts had all 15 runs perfect. Length failures numbered 21 for Gemini, 16 for Fable, 14 for Opus, 10 for Sonnet and 0 for GPT. These 10 prompts do not establish a ranking; see the prompt library.

What was tested, and how?

Round 1 tested 40 prompts across 10 professions on October 19, 2026: 30 English and 10 Spanish. Each model ran each prompt 3 times, producing 360 completed runs.

Each prompt had 5 pass/fail criteria, hash-locked before testing; 32 included a SAFETY-labeled criterion. Fictional, de-identified inputs went unchanged to every model. The command-line harness disabled tools, web access, memory and custom instructions while retaining generic harness instructions. Nothing was edited between prompt and output.

Cross-family judges returned per-criterion verdicts and reasons, marking fail when unsure. An editor agent independently assessed verdicts before reading the judge’s decisions: it agreed on 18 of 20, comprising 14 random verdicts and 6 leniency probes; both disagreements had the judge stricter. See the methodology for procedures and round 2 details.

How did each model score?

Round 1 raw scores were 92% for Opus, 91% for Sonnet and 99% for GPT; different judges prevent a direct cross-family ranking. Language figures are criteria-pass rates, not wholly successful prompts.

Tested October 19, 2026.

Overall, 1,690 of 1,800 verdicts passed, or 94%; 13 prompts had all 9 runs pass every criterion. These results concern the tested prompts, not general language proficiency.

Is GPT really better than Claude here?

This round does not establish that GPT is better than Claude: calibration indicates that most of the raw gap reflects judge strictness.

The swap regraded 60 runs. On the same 30 Claude runs, gpt-6-astra passed 93% of criteria and claude-opus-5-5 passed 98%, a 5-point gap; on 30 GPT runs, both passed approximately 97%. The judges differed on 12 runs. On Claude outputs, GPT scored lower in 7 and higher in 0. This measures judge asymmetry in the sample, not an underlying quality difference.

Opus and Sonnet’s same-judge gap of 1 point does not establish a meaningful lead.

Where did the models go wrong?

Added content dominated round 1 failures: 78 of 110 (71%), alongside 24 missing-or-changed-content failures and 8 length failures. Of all failures, 56 concerned SAFETY-labeled criteria.

Patient handout. Fictional notes covered type 2 diabetes, metformin, side effects and follow-up, without emergency instructions. Opus wrote:

If you feel very sick or think it is an emergency, call 911 or go to the nearest emergency room.

The judge flagged additions without required “ask your care team” labeling: the criterion failed in all 6 Claude runs and 0 GPT runs. Counts concern the whole criterion, not this exact sentence or the advice’s medical merit.

Flight disruption. Notes specified the same airline the next day, without mentioning a connection. Sonnet wrote:

Pros: Comfortable connection, same carrier handles any rebooking issues directly.

Unsupported details failed the criterion in 8 of 9 runs, including 2 GPT runs.

Cold email. The input allowed one approved proof point and no other claimed results. Sonnet added:

smoother routes tend to mean less driver frustration

The unapproved-benefit criterion failed in 3 of 3 Sonnet runs and all 3 Spanish counterpart runs.

Which professions were hardest?

Engineers and sales managers had round 1’s lowest aggregate pass rates, both 90%; construction managers scored 100%. Sonnet’s engineering score, 80%, was the lowest model-profession result. With 3–5 prompts per profession in this round, these figures describe selected tasks, not each field’s overall difficulty.

How do the added models compare under the same judge?

Fable did not outscore Opus; Gemini was within about a point of Opus in both rounds. Both additions used the existing locked prompts, inputs and rubrics.

Tested October 20, 2026. Criteria-pass rates, all judged by gpt-6-astra.

Gemini used Antigravity CLI with a custom agent loading no tools, skills or rules and a one-line system prompt; a built-in background tool remained exposed, but any tool call voided a run, and 240 runs made 0 tool calls.

On 10 Fable runs per round, gpt-6-astra versus claude-opus-5-5 passed 88% versus 100%, then 82% versus 90%. An editor agent agreed on 8 of 12 Fable verdicts; all 4 disagreements had the judge stricter.

On 10 Gemini runs per round, those judges passed 92% versus 98%, then 86% versus 88%. The agent agreed on 7 of 10 Gemini verdicts; all 3 disagreements had the judge stricter.

How did the models do on Excel and Microsoft 365 tasks?

Scores ranged from 83% to 93%. These were text-only answers: formulas were judged on logic and stated results, not executed in Excel. These were chat models answering a pasted task, not Copilot in Excel or Microsoft 365.

Tested October 20, 2026.

Older VLOOKUP-family functions caused 20 failed verdicts: they could work on the data but did not meet the locked requirement for localized XLOOKUP or INDEX/MATCH. Listing open items as decisions caused 22 failures across 2 meeting-minutes prompts.

Calibration on 6 runs per family showed a reported 3-point gap in each; an editor agent agreed on all 10 spot-checks. GPT’s different judge still prevents direct ranking.

What did the Claude Skills rounds show?

Revisions improved results on the tested inputs. The English/Spanish round tested 10 skills on Opus and Sonnet, with 3 runs each under gpt-6-astra; Fable joined later. Version 1.0 defects included clarification-only responses, reference-source citations, unsupported emergency destinations, missing RFP criteria and dropped documented figures. After 8 skills were revised, final averages were 4.0–5.0 of 5.

The French-for-France and Portuguese-for-Brazil adaptations covered 20 skill-language pairs with separately locked rubrics: Opus, Sonnet and Fable ran each 3 times, totaling 180 runs under gpt-6-astra. First texts averaged 3.67–5.0 of 5; 4 adaptations received general-rule revisions and 45 re-runs.

Before → after, in Opus / Sonnet / Fable order:

• French RFP, v1.1: 4.0 / 4.0 / 4.0 → 4.33 / 5.0 / 4.67.
• Portuguese RFP, v1.1: 4.0 / 4.0 / 4.0 → 5.0 / 5.0 / 4.33.
• French construction, two revisions to v1.2: 4.0 / 4.0 / 4.0 → 5.0 / 5.0 / 5.0.
• French sales, v1.1: 3.67 / 4.67 / 4.33 → 5.0 / 5.0 / 4.67.

Current FR/PT averages are 4.33–5.0 of 5, with 46 of 60 model-pair results at 5 of 5. Both Skills rounds reused the same locked inputs and rubrics for re-tests; neither had judge-swap calibration or an agent spot-check. Improvement on these inputs does not establish improvement on unseen inputs.

The Claude Skills are published under CC BY 4.0: free to use and adapt, including commercially, with attribution.

What does this data not show?

It does not establish a model ranking or replace professional review.

• Not a model or profession ranking. Judges differ; profession samples are small.
• Not consumer-app performance. App instructions, tools and memory differ from the tested harnesses.
• Not professional sign-off. No human professional reviewed every output; the spot-checker was an agent.
• Not broad input coverage. Fictional sample inputs cannot expose every failure.
• Not permanent results. Records describe the dated model versions tested.
• Not comparable language scores. English, Spanish, French and Portuguese each have their own prompt sets.

FAQ