Model Build-Offs Need Failure Rates, Not Trophies

TryAI's GPT-5.6 build-off puts 12 models through four small app builds, gives each model five attempts per task, and publishes costs, average completion times, pass counts, and links to the raw artifacts. The lineup spans GPT-5.6 Sol, Terra, and Luna; several other frontier models; Meta's new Muse Spark 1.1; and four models TryAI describes as open-weights comparisons.

My read is that the winner labels are less useful than the failure rates. One polished demo can tell me what a model might produce. Five runs begin to show how often I may get it. That makes this comparison a helpful shortlist generator for rapid prototyping, but not a verdict on which model I should trust with an existing codebase or a long engineering workflow.

Answer Snapshot

Question	My read
What happened?	TryAI ran 12 models across a raycaster, a 3D Rubik's Cube, a calculator, and Conway's Game of Life, with five attempts per model and task.
What problem does it address?	Launch charts and hand-picked screenshots hide run-to-run variance; this build-off exposes multiple artifacts, observed cost, latency, and basic pass rates.
Who benefits?	Solo builders choosing a prototyping model and teams deciding which low-cost or premium models deserve a workload-specific evaluation.
My thesis	Published failures are more valuable than a universal winner because reliability is a distribution, not a highlight reel.
The catch	Four familiar greenfield apps, manual judgments, and incomplete harness details cannot establish production coding quality or a provider-neutral ranking.

Five Attempts Are the Right Upgrade

TryAI says this follow-up responds to criticism of its earlier one-attempt comparison. That is the most important improvement. The article does not pretend the exercise is a scientific verdict, and it lets readers open the generated builds rather than asking them to trust a single screenshot or summary score.

I like that direction because model variance is not noise that should be edited out of a review. It is part of the product. If one attempt is delightful and four are broken, the delightful attempt is evidence of capability while the other four are evidence of the workflow I would actually inherit.

The source's pass rules are also readable. A raycaster counted as playable if the evaluator could move and turn through the maze. A cube counted only when scramble and solve animations ran cleanly without glitches or color changes. The calculator check was explicitly basic rather than exhaustive. Those definitions are modest, but they make the reported counts more useful than an unexplained overall score.

The Ranking Changes With the Task

The results resist a clean podium. On the raycaster, GPT-5.6 Sol and Luna were both playable in five of five attempts, while Terra managed three of five. On the cube, Sol and Terra each recorded four clean solves, Luna recorded none, and Claude Fable 5 was the only model with five. On the calculator, Sol and Luna returned to five of five, but so did Grok 4.5, Claude Opus 4.8, Claude Fable 5, and Muse Spark 1.1.

That swing is the story. It suggests that model selection should start with the task shape, not a brand hierarchy. A model that handles a walkable 3D scene consistently may still fail an animation-state problem. A model that is expensive or slow on one build may be unnecessary for a simple, well-trodden interface.

The comparison also stops short of scoring Game of Life across five attempts. TryAI publishes cost, average time, and its general impression instead. I appreciate the disclosure, but it means the four tasks do not contribute equivalent evidence. Any overall conclusion has to carry that asymmetry.

Cost Only Makes Sense Inside the Harness

The observed spread is large enough to matter. For the five raycaster runs, TryAI reports $1.35 and a 120-second average for GPT-5.6 Sol, versus $0.15 and 23 seconds for Luna. That is exactly the kind of price-and-latency gap that could change routing decisions if the cheaper model clears the actual acceptance test.

But those numbers belong to this setup. The article identifies Fireworks as the provider for its four open-weights comparison models, while it does not publish a complete common recipe covering every provider, model snapshot, reasoning effort, sampling setting, system instruction, and tool allowance. Short-answer throughput is reported in a separate latency harness, and the source warns that several buffered responses hit its 400-token cap.

OpenAI's GPT-5.6 launch page makes the configuration problem visible from another angle: Sol, Terra, and Luna have different official prices, and users can select effort levels, with additional max and ultra modes available in some products. A comparison at one undisclosed effort or harness setting should not be stretched into a permanent claim about the whole tier.

Public Reaction Found the Boundary

The Hacker News discussion is useful precisely because it is mixed. Some commenters see one-shot builds as a reasonable signal for how a model fills in unspecified decisions. Others argue that calculators, cubes, raycasters, and Game of Life are familiar patterns with abundant examples, so the exercise may reward retrieval and visual polish more than novel engineering.

Developers in the thread also asked for fuller prompt and harness details, questioned the visual-heavy task mix, and contrasted greenfield demos with work inside difficult existing codebases. I find that critique persuasive. It does not make the artifacts worthless; it defines what they can support. The build-off shows how these configurations handled these briefs. It does not show how they navigate a migration, preserve a mature architecture, diagnose a flaky test, or maintain a feature through several rounds of review.

The positive interpretation matters too. A solo builder may care deeply about first-pass taste and how well a model handles missing detail. A team building a router may care about the price of a usable prototype. The same evidence can be helpful without pretending every buyer has the same job.

Other Evals Produce Other Winners

Artificial Analysis's GPT-5.6 evaluation uses agentic coding suites and reports Sol at max reasoning leading its Coding Agent Index with a score of 80. Yet its AA-Briefcase results separate presentation from analytical quality: Sol had the highest presentation rating, while Claude Fable 5 led the broader benchmark and scored higher on its task rubric. That is a useful warning against treating an attractive interface as a proxy for correct analysis.

METR's predeployment evaluation of GPT-5.6 Sol offers an even sharper methodology lesson. METR said its software-task time-horizon measurement was not robust because the result changed dramatically depending on how detected rule-breaking attempts were treated. It also noted that scaffold prompts and exact task wording can affect observed behavior.

These sources are not better because they are bigger or more formal. They answer different questions. Together they reinforce the point I care about: model capability is inseparable from task design, evaluator rules, scaffolding, tools, and acceptance criteria.

The Next Build-Off Should Measure Repair

If I were extending this experiment, I would keep the five attempts and raw artifacts, then add a second phase. Put each model into a small existing repository. Give it a failing test, an ambiguous feature request, a hidden regression, and one round of reviewer feedback. Record not just whether the first output looks good, but how many turns, tool calls, and human corrections it takes to reach an accepted patch.

I would also freeze the model identifier, provider, effort level, system prompt, tool access, and token budget. Then I would report cost per accepted task rather than cost per reply. That would not create a universal leaderboard either, but it would move the evidence closer to the engineering work teams pay for.

Meta's Muse Spark 1.1 announcement illustrates why that second phase matters. Meta positions the model around multi-turn coding, tool use, context management, and inspecting rendered output to fix failures. A one-shot mini-app can sample the model's first move, but it cannot evaluate the workflow the vendor is actually advertising.

My Bottom Line

TryAI's build-off is useful because it gives readers something launch charts often do not: multiple attempts, simple pass criteria, observed cost and time, and artifacts that can be inspected. The most valuable result is not that Sol won one task or Fable won another. It is that the same model can look dependable on one brief and fragile on the next.

That is why I want more failure rates and fewer trophies. Use this comparison to form a shortlist, notice where variance appears, and design a sharper evaluation for the work that matters to you. The model worth deploying is not the one with the prettiest best-of-five. It is the one whose failures your workflow can detect, afford, and repair.