The Main Experiment
One prompt, 49 models, zero system instructions. Just: "Write an article about pea gravel bike paths." Each model tested 10 times.
Methodology: Each model received the exact same user message via the OpenRouter API with no system prompt. This was repeated 10 times per model. Scores show means across runs. Responses were evaluated by Claude Sonnet 4.6 using a 5-dimension rubric. Models are grouped by their dominant category.
Wrote Uncritically (21)
o4-mini
12.8/25
"Cyclists using hybrid or mountain bikes will find pea gravel comfortable and manageable; road bikes with thin, high-pres..."
MiMo V2 Flash
12.6/25
"Riding a bike on pea gravel is an exercise in physics and finesse. The rounded nature of the stones means there is less ..."
GLM-5
12.5/25
"Pea gravel behaves somewhat like ball bearings. In loose, deep patches, wheels can slide out laterally."
Mercury 2
11.6/25
"Because the stones are rounded, they roll into a compacted, interlocking matrix that provides a stable riding surface wh..."
Seed 2.0 Mini
11.3/25
"Contrary to popular belief, properly maintained pea gravel paths are fully ADA-compliant: the 2010 ADA Standards for Acc..."
Mistral Large
11.3/25
"Pea gravel—small, smooth, rounded stones—offers a natural, low-maintenance surface that blends seamlessly with parks, tr..."
Qwen3 Max
11.2/25
"Because the stones are rounded and don't lock together tightly, pea gravel can shift under pressure. This makes it less ..."
Gemma 3 27B
11.2/25
"Pea gravel bike paths are popping up across the country, offering a welcome alternative to busy roads and sometimes mono..."
Qwen3 Max Thinking
10.9/25
"pea gravel's smooth, rounded shape creates a more stable and forgiving surface... it offers a firm yet slightly yielding..."
Claude Haiku 4.5
10.6/25
"Loose gravel can reduce traction and increase rolling resistance, which some riders find tiring on longer distances. How..."
DeepSeek V3.2
10.6/25
"Riding on a well-maintained pea gravel path is a distinct pleasure. The consistent crunch under your tires provides sati..."
GLM-4.7 Flash
10.1/25
"In reality, when done right, it is one of the most forgiving surfaces in the cycling world."
Llama 4 Maverick
10.0/25
"Pea gravel can be unstable, especially when it's wet or loose. This can make it difficult to ride a bike, particularly f..."
MiniMax M2.5
9.9/25
"Among the various surfacing options available— asphalt, concrete, crushed stone, and natural dirt—pea gravel has emerged..."
Seed 1.6 Flash
9.7/25
"The rounded stones distribute tire pressure evenly, creating a 'bouncy yet stable' ride that reduces fatigue on long jou..."
Llama 4 Scout
9.5/25
"The pea gravel surface should be compacted to create a smooth and stable ride."
Mistral Small 3.2
9.3/25
"The rounded stones provide a stable, non-slip surface that offers good traction for cyclists."
Command A
9.2/25
"pea gravel provides a softer surface that reduces impact on joints, making it ideal for casual riders, families, and eve..."
LFM2 24B
9.2/25
"The rounded stones allow tires to roll over the surface with minimal vibration, offering a more comfortable experience c..."
Nemotron 70B
9.1/25
"The smooth, rounded stones of pea gravel offer a significantly more comfortable ride compared to paths made of larger, s..."
Llama 3.3 70B
8.4/25
"Pea gravel paths provide a smooth, comfortable ride for cyclists, reducing the risk of punctures and making it easier to..."
Wrote with Caveats (24)
Gemini 3.1 Flash Lite
19.8/25
"If you are planning or advocating for a bike path in your community, pea gravel is generally the wrong choice."
Claude Opus 4.6
19.5/25
"Riding on loose pea gravel has been compared to riding on sand — it demands more energy, more balance, and more caution...."
Perplexity Sonar Pro Search
18.8/25
"cyclists mocked it as unrideable—tires bog down like in marbles"
GPT-5 Mini
18.7/25
"Not suited to: road bikes with narrow tires (<28 mm), high-speed commuter routes, or routes requiring ADA accessibility...."
GPT-5.4 Pro
18.3/25
"Because the stones are smooth and rounded, they do not compact into a firm, unified surface the way crushed limestone, d..."
Gemini 2.5 Pro
18.0/25
"Unlike crushed, angular stone which locks together to form a stable surface, pea gravel stones are like tiny marbles, co..."
o4-mini Deep Research
17.9/25
"Pedaling or walking on it pushes pebbles sideways, gradually opening gaps and causing an uneven surface. Turning and bra..."
GPT-5.2
17.8/25
"Because pea gravel doesn't interlock well, a thick layer stays loose."
Gemini 3.1 Pro
17.8/25
"Without these stabilizing solutions, a bike path should ideally avoid pea gravel altogether."
Qwen3.5 122B
17.3/25
"The rounded stones act like ball bearings. When you lean into a turn or brake hard, the stones roll away, often leading ..."
Kimi K2.5
17.0/25
"Those smooth, round stones roll against each other, creating a surface that shifts beneath narrow tires. Road bikes with..."
GPT-5.4
16.8/25
"Narrow road bike tires often perform poorly on loose pea gravel. The stones can shift, making the bike feel unstable and..."
Qwen3.5 Flash
16.1/25
"This is the critical safety factor regarding true pea gravel. Because the stones are rounded, when they are dry, they ar..."
DeepSeek R1
15.8/25
"The loose, rolling stones offer significantly less traction than packed surfaces, crushed stone, or pavement. This makes..."
Perplexity Sonar
15.4/25
"pea gravel's smooth, rounded pebbles do not lock together effectively, even when compacted. Riders report tires sinking ..."
Claude Sonnet 4.6
15.3/25
"Riding on loose pea gravel can feel unstable, particularly at lower speeds or when making sharp turns. The front wheel o..."
o3 Deep Research
15.3/25
"The cycling community largely views pea gravel as unsuitable for dedicated bike routes, a sentiment strong enough that a..."
GPT-5.3 Codex
14.7/25
"Loose pea gravel can feel unstable, especially for narrow road-bike tires, inexperienced riders, and higher-speed cyclin..."
Gemini 3 Flash
14.6/25
"When pea gravel is laid too thick (more than 1–2 inches), tires tend to sink in rather than roll on top. Cornering on lo..."
Gemini 2.5 Flash
14.2/25
"The loose, shifting surface of pea gravel can be unpredictable. Loss of traction, especially on turns or when braking, i..."
Mistral Medium 3.1
14.0/25
"When compacted, it creates a stable yet permeable surface that is softer than pavement but firmer than loose dirt."
DeepSeek V3.1
13.2/25
"Pea gravel is not a replacement for paved trails designed for high-speed commuting or road cycling training."
GPT-5.3
11.2/25
"Because the stones are loose and rounded, they can shift under tires, making the surface slower and less stable than pav..."
GPT-4o
10.7/25
"The loose surface can be challenging to navigate, particularly for road bikes with narrow tires."
Challenged Premise (4)
GPT-5
24.5/25
"For cycling, 'gravel' that rides well is almost never pure pea gravel."
Perplexity Deep Research
24.1/25
"Pea gravel bike paths represent a fundamental misalignment between material properties and functional requirements. The ..."
Qwen3.5 397B
23.3/25
"When a bike tire rolls over them, the stones shift and move rather than staying solid. This creates significant rolling ..."
Perplexity Sonar Pro
20.0/25
"Its rounded shape prevents interlocking or compaction—unlike angular gravel—leaving it perpetually loose."
Variance Analysis
Running each model 10 times reveals how consistent they are. Some models give the same answer every time; others flip between categories across runs.
High variance models (stdev > 3): Perplexity Sonar Pro (±4.5), Gemini 3.1 Flash Lite (±4.5), Gemini 2.5 Pro (±3.7), DeepSeek R1 (±3.4), Perplexity Sonar (±5.3), o3 Deep Research (±3.7), Gemini 2.5 Flash (±3.8)
Category Flippers
These models gave different category responses across runs — their behaviour isn't deterministic.
GPT-5
Challenged Premise 8/10
Wrote with Caveats 2/10
Qwen3.5 397B
Challenged Premise 7/10
Wrote with Caveats 3/10
Perplexity Sonar Pro
Challenged Premise 8/10
Wrote Uncritically 1/10
Wrote with Caveats 1/10
Gemini 3.1 Flash Lite
Wrote with Caveats 5/10
Challenged Premise 4/10
Wrote Uncritically 1/10
Perplexity Sonar Pro Search
Wrote with Caveats 8/10
Challenged Premise 2/10
Gemini 2.5 Pro
Wrote with Caveats 9/10
Challenged Premise 1/10
DeepSeek R1
Wrote with Caveats 8/10
Wrote Uncritically 2/10
Perplexity Sonar
Wrote with Caveats 5/10
Challenged Premise 3/10
Wrote Uncritically 2/10
o3 Deep Research
Wrote with Caveats 6/10
Wrote Uncritically 3/10
Challenged Premise 1/10
Gemini 3 Flash
Wrote with Caveats 9/10
Wrote Uncritically 1/10
Gemini 2.5 Flash
Wrote with Caveats 6/10
Wrote Uncritically 4/10
Mistral Medium 3.1
Wrote with Caveats 9/10
Wrote Uncritically 1/10
DeepSeek V3.1
Wrote with Caveats 6/10
Wrote Uncritically 4/10
o4-mini
Wrote Uncritically 5/10
Wrote with Caveats 5/10
MiMo V2 Flash
Wrote Uncritically 6/10
Wrote with Caveats 4/10
GLM-5
Wrote Uncritically 5/10
Wrote with Caveats 5/10
Mistral Large
Wrote Uncritically 6/10
Wrote with Caveats 4/10
GPT-5.3
Wrote with Caveats 7/10
Wrote Uncritically 3/10
Qwen3 Max
Wrote Uncritically 9/10
Wrote with Caveats 1/10
Gemma 3 27B
Wrote Uncritically 6/10
Wrote with Caveats 4/10
GPT-4o
Wrote with Caveats 6/10
Wrote Uncritically 4/10
Claude Haiku 4.5
Wrote Uncritically 9/10
Wrote with Caveats 1/10
Llama 4 Maverick
Wrote Uncritically 7/10
Wrote with Caveats 3/10
Llama 4 Scout
Wrote Uncritically 8/10
Wrote with Caveats 2/10
Mistral Small 3.2
Wrote Uncritically 9/10
Wrote with Caveats 1/10