🔘 About this critique
A model can learn from a game and still make its next game harder to solve. This study asks what happens when the model can rewrite the software through which it plays. The decisive test comes after the revisions are over, on twenty games whose results the model never saw while revising its design.
This assessment examines the first arXiv version, submitted on 21 October 2026. We checked the published game records, recalculated the paired score and cost changes, and reproduced the two principal bootstrap intervals. These checks assess the published measurements. We did not run the agents independently. Exposure indicates the consequence of extending a particular claim beyond the available evidence.
Critique
| Examined version | arXiv:2610.30499v1 · 21 October 2026 · preprint · CC BY 4.0. |
| Materials examined | The paper and Appendices A–G, together with the authors’ per-game results, trajectory and snapshot records, repeated evaluations and statistical documentation. |
| Recommendation | Read the development and held-out results as a pair, then examine which individual revisions gained performance at lower cost. |
Unit 1 — What did the model change?
Article data. The same language-model profile first wrote the software through which it would play, then played through that software and revised it twice. The model weights and the programming environment used to make revisions were fixed. Each of nine experimental conditions contributed three independent construction runs. All 27 sequences reached the final evaluation.
Reading. On a game board, a harness can preserve the consequences of earlier moves, turn an image into a list of objects or ask the model to plan several moves together. Changing any of these choices can alter play substantially while the underlying model remains the same. The authors allowed repairs when a proposed version could not run or interact with the game. Passing those checks established that a version could be evaluated.
Critical conclusion. The experiment tests whether a model can improve the software through which it acts. A successful repair and an improvement in play are separate outcomes. The study provides evidence about both.
| Claim type | Definition of the intervention |
| Exposure | High. Attributing the observed changes to learning in the model weights would misidentify what the experiment varied. |
Unit 2 — Did gains on the practice games transfer?
Article data. Five games supplied feedback. Twenty other public games were reserved for evaluation. Relative Human Action Efficiency (RHAE) rewards reaching and completing levels while taking account of the number of actions relative to a human reference. Mean performance on the reserved games rose from 24.54 to 27.34. The paired gain was 2.80 points, with a 95% bootstrap interval from −1.44 to 7.35.
Reading. The corresponding gain on the five practice games was 8.42 points. Its excess over the held-out gain was 5.62 points, with a 95% interval from 1.78 to 9.77. These intervals answer different questions. The study gives clearer evidence that development results overstate transfer than it gives for the size or direction of the average benefit on the reserved games.
Critical conclusion. Revisions improved held-out performance in 15 sequences and reduced it in 12. Resampling whole sequences preserves the pairing between versions. It leaves uncertainty about another choice of feedback games, a longer revision period or games that were unavailable during model training.
| Claim type | Transfer and statistical uncertainty |
| Exposure | High. Treating the development gain as the expected benefit on new games would substantially overstate what these observations support. |
Unit 3 — When did using less inference also improve play?
Article data. Table 7 compares each final version with its own starting version. Nine sequences gained points while using less inference expenditure per held-out game. Six gained points at higher cost, nine lost points while becoming cheaper, and three lost points at higher cost. Logical inference cost uses the study’s fixed prices and includes replayed responses.
| Final score | Lower inference cost | Higher inference cost |
|---|---|---|
| Higher | 9 | 6 |
| Lower | 9 | 3 |
Reading. The table allows a reader to distinguish two reasons that mean cost might fall. Some revisions performed more effectively with fewer resources. Others spent less while also performing less effectively. An average across the two conceals that difference. The measure concerns inference during play. Developing and qualifying a version incurs additional costs recorded separately.
Critical conclusion. Of the 18 final versions that cost less to run, nine also played better. A lower per-game cost is interpretable alongside the score for the same version and the same evaluation games.
| Claim type | Paired performance and resource comparison |
| Exposure | Moderate. A cost-only reading would classify nine observed performance losses as improvements. |
Unit 4 — How much can the same program vary?
Article data. Appendix D.1 reports ten evaluations of one unchanged final program on the same twenty games. Its mean was 48.53 RHAE, its sample standard deviation was 3.99 points and its scores ranged from 43.38 to 55.37. Every evaluation exceeded this sequence’s starting score of 20.38.
Reading. The example shows how a substantial gain can survive repeated play even when its observed size varies. Differences of only a few points in other sequences are harder to interpret. This program was chosen after the main experiment and the ten evaluations include its original run.
Critical conclusion. Repeated play strengthens the claim about this particular revision. Applying its standard deviation to every model or treating it as the uncertainty of the pooled mean would require a different experiment.
| Claim type | Repeatability of an individual result |
| Exposure | Moderate. One repeated program provides a scale for variation in its own setting. |
Unit 5 — Which version should a system keep?
Article data. The main comparison always evaluated the second revision against the initial program. In a retrospective analysis, choosing the version with the highest score on the five practice games selected the initial version nine times, the first revision six times and the second revision twelve times. This rule yielded a mean held-out gain of 4.53 points and a mean inference cost of $9.15 per game, compared with $7.57 for the final versions.
Reading. A system can propose a useful change and still need a criterion for deciding whether to retain it. The selection rule used development scores, but the researchers examined the rule after collecting the data. Separately, published human-guided systems demonstrate what some of these models can achieve on public games under different development and selection procedures.
Critical conclusion. A prospective test could fix the selection rule before adaptation and compare its results on a fresh set of games. The reported human-guided results establish that stronger designs exist. The contribution of human guidance itself is not isolated by the comparison.
| Claim type | Retrospective selection and external comparison |
| Exposure | High. A promising retrospective choice does not establish how well the same rule would perform in a new campaign. |
Overall assessment
The paper makes an important experimental distinction possible. It preserves the original program, its two descendants and the outcomes of each on games withheld from revision. A strong final agent can therefore be traced back to a strong starting design, a useful revision or a mixture of both. Across these 27 sequences, practice scores give a more optimistic account of improvement than the reserved games do.
The practical question is consequently one of judgement as well as invention. An agent that proposes changes needs evidence for keeping or reversing them. A fresh evaluation that preserves the starting program, repeats play and fixes its selection rule in advance would test whether useful improvements accumulate.
🔘 Paper page: https://arxiv.org/abs/2610.30499v1
Abstract
A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.
Authors’ abstract reproduced from arXiv:2610.30499v1 under the Creative Commons Attribution 4.0 International licence. Wording is retained.
Authors
Liked this post? Follow this blog to get more.