Frameworks (Paper)

R0:aa77bca17589f025d763d708bbb99028-Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

If a language model can rewrite the software it uses to play a game, how can it tell whether an edit was worth keeping? Jens Lehmann and Sahar Vahdati let each model build a program, play five practice games and revise that program twice. They then compared all three versions on twenty games whose results had been withheld during development. Across 27 independent sequences, the practice scores rose by 8.42 points on average. On the reserved games, the gain shrank to 2.80 points, and twelve sequences finished below their starting score. Cost tells a similarly mixed story. Eighteen final programs were cheaper to run, but only nine of those also played better. By preserving the original programs and their descendants, the study reveals something a final leaderboard cannot show on its own: when an agent already plays well, how much of that ability did it acquire through revision, and how much was present from the start?

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3 Leer más »

R0:abbc9fad0e8a3823e93438ffbc99b607-FrontierPhysics: Evaluating Agents for End-to-End Physics Research

FrontierPhysics: Evaluating Agents for End-to-End Physics Research

Can a research agent tell when its own calculation has led it astray? In FrontierPhysics, physicists turned investigations from their own laboratories and research groups into 56 problems. An agent receives a brief, explores the literature, chooses a method and submits a manuscript alongside results that an examiner can check. The strongest configuration passed 46 of 168 trials. It solved 25 problems at least once in three attempts, but only eight on every attempt. The most revealing findings emerged when physicists revisited the experimental records. Some agents had checked a result using the very assumption that produced it. In other cases, a sound answer had failed the examiner’s test. Nine such judgements changed in the detailed audit of one configuration. Together, the repeated trials and the expert review illuminate a practical question for any physicist considering an agent’s work. What evidence would persuade another researcher that the result survives a genuinely independent check?

FrontierPhysics: Evaluating Agents for End-to-End Physics Research Leer más »

Fourier Neural Operator for Parametric Partial Differential Equations

Fourier Neural Operator for Parametric Partial Differential Equations

The classical development of neural networks has primarily focused on learning mappings between finite-dimensional Euclidean spaces. Recently, this has been generalized to neural operators that learn mappings between function spaces. For partial differential equations (PDEs), neural operators directly learn the mapping from any functional parametric dependence to the solution.

Fourier Neural Operator for Parametric Partial Differential Equations Leer más »

A socio-technical framework for digital contact tracing

A socio-technical framework for digital contact tracing

In their efforts to tackle the COVID-19 crisis, decision makers are considering the development and use of smartphone applications for contact tracing. Even though these applications differ in technology and methods, there is an increasing concern about their implications for privacy and human rights. Here we propose a framework to evaluate their suitability in terms of impact on the users, employed technology and governance methods.

A socio-technical framework for digital contact tracing Leer más »