When a language model fails to repair a piece of software, would a second pass through the same network help? NVIDIA’s Dory1.0-4B allows a particularly direct test. Its central block can run once, twice or four times while the learned weights stay fixed. The clearest answer comes from comparing two tasks. On software repair, the second pass raises the proportion of issues resolved from 50.3% to 59.5%. On a test of factual knowledge, accuracy changes from 10.03% to 10.08%. NVIDIA technical report, Table 2
Assessment. Dory merits inclusion in Open Models as an inspectable experiment in recurrent depth. Its strongest result is the gain from a second pass on particular tasks. Independent implementations also allow readers to examine whether a different engine preserves the model’s numerical behaviour. The Bible of AI (BAI) reviewed the released materials and published measurements and did not run a new inference experiment.

What changes on the second pass?
Before predicting the next token, Dory encodes the available text. Its middle block then combines that original representation with the state produced by the preceding pass. Each pass therefore applies the same learned transformation to a different input. One, two and four passes execute 72, 112 and 192 sublayers respectively, using the same 3.98 billion parameters. Two passes are the default in the released BF16 configuration. The intermediate states are vectors. Establishing what information a later pass corrects would require an intervention on those states. Earlier research on Huginn provides a scientific precedent for this use of recurrent depth. Released configuration · Architecture and Figure 2 · Recurrent-depth precedent
The laboratory and the release
NVIDIA, which designs accelerators and develops models and inference software, released Dory through Nemotron Labs. Its report describes six trillion tokens of pretraining followed by 300 billion tokens for context extension. Supervised fine-tuning used 128 GB300 GPUs across 32 nodes. A later reinforcement-learning stage assigned greater depth more often to difficult examples during training. That design leaves the choice of inference depth open. The report identifies equipment for particular stages but gives no itemised budget for the complete training run. Training, §§3–4 · NVIDIA model card
NVIDIA distributes trained weights in BF16, MXFP8, NVFP4 and GGUF formats. The English-language BF16 release includes its configuration and tokenizer. The OpenMDW 1.1 licence permits use, modification and distribution subject to its terms, including preservation of applicable notices. The published weights support comparisons across depths. The report and release do not supply a document-by-document training inventory sufficient to reconstruct the full training sequence. The configured limit of 262,144 tokens specifies the maximum sequence length. Performance over documents of that length remains a separate measurement. Official model collection · Configuration · Release licence
Where greater depth helps
| Evaluation | 1 pass | 2 passes | 4 passes |
|---|---|---|---|
| Twelve-metric mean | 35.17 | 39.67 | 40.60 |
| SWE-Bench Verified · software repair | 50.30 | 59.50 | 61.80 |
| AA-LCR · long-context reasoning | 63.00 | 71.00 | 73.00 |
| GPQA Diamond · science questions | 69.19 | 72.41 | 72.16 |
| AA-Omniscience · factual accuracy | 10.03 | 10.08 | 10.03 |
The equal-weight mean rises by 4.50 points from one pass to two and by a further 0.93 points from two to four. The authors rescale GDPval-AA Elo scores before averaging. On GPQA Diamond the observed score falls slightly after the second pass. The reported point estimates alone cannot establish whether that small difference exceeds sampling variation. Four passes also produced shorter outputs on four tasks requiring a sequence of actions. After accounting for the greater depth per token, NVIDIA estimated that generation still required 1.07–2.04 times as much computation as one pass. This estimate uses sublayer applications and output length. Elapsed time and energy require direct measurement. Tables 2 and 3
What users measured with llama.cpp
One user tested the Q5_K_M release on an RTX 5070 Ti with a Dory-specific development version of llama.cpp. With a separate attention cache for each pass, the initial decoding rates were 40, 27 and 16.5 tokens per second at one, two and four passes. A follow-up held the prompt at 512 tokens and the output at 128. Across five timed repetitions, placing the model’s scaling tensors on the GPU raised the mean rate from 25.67 to 79.94 tokens per second. These two series answer different questions. The first follows speed across depths. The second compares two ways of placing the same data. User’s protocol, hardware and results
A CPU experiment on a shared host yielded best-of-five rates of 3.30, 2.38 and 1.43 tokens per second at the same three depths. A separately registered test on a DGX Spark recorded 24.15, 16.02 and 9.56 tokens per second with one request at a time and a separate cache per pass. All 40 responses to its five simple prompts across eight configurations met the authors’ coherence criterion. With one run per configuration, this establishes basic operation and leaves differences in batched throughput unresolved. CPU experiment · DGX Spark experiment and deviations from protocol
What the Rust implementations establish
Numerical agreement on Apple silicon
Omegahee’s Rust engine provides the closest independent check of whether an implementation preserves Dory’s numerical behaviour. On Metal, the project compared converted BF16 weights against its FP32 reference at one, two and four passes, retaining either a separate attention cache for each pass or only the final pass’s cache. All 31 cases met its stated tolerances. On the 1,830-token prompt, the engine and reference assigned the highest score to the same next token at 1,768 positions. This illustrates what passing a numerical tolerance means in practice. The authors also report about 25 generated tokens per second with two passes on an M5 Pro. These tests concern implementation fidelity and speed. They do not reproduce NVIDIA’s task scores. Methods and device · 31 individual results
Adaptive depth in Cortiq
Cortiq offers a different experiment. Its 2.09 GB, four-bit Dory variant can assign one or two passes to each token and changes how distant context is retained. Over twelve prompts, its maintainers report that generation rose from 49.6 to 60.7 tokens per second on an NVIDIA L4 with Vulkan, and from 25.7 to 32.9 on a MacBook Air M4 with Metal, compared with their fixed two-pass variant. WikiText-2 perplexity rose from 13.96 to 14.22, indicating poorer prediction on that text. Routing and memory handling changed together, so the figures cannot isolate the contribution of either intervention. Paired measurements and evaluation · Rust engine release
Other routes through Rust
Eider provides Rust and CUDA code for generating text with NVIDIA’s NVFP4 weights. Its separately published Decisive Dory has undergone additional training to choose among supplied answers. Results for that derivative concern a different set of weights. Rillwright places a Rust application around the Dory version of llama.cpp. Its authors report 80 successful runs on 16 small tasks repeated over five seeds. This tests the application and model together. Counting it as another independent numerical implementation would count llama.cpp twice. Eider generation example · Derivative model card · Rillwright evaluation and limitations
Jero provides Rust tests of the recurrent architecture for the SpacemiT K3 platform. Its published forward-pass test uses synthetic weights of width 256. That test demonstrates how the implementation can be examined, while leaving the speed and accuracy of the trained four-billion-parameter model on the board open. The reviewed Candle model registry contains recurrent architectures, including RecurrentGemma. A Dory-specific entry or documented run in Candle remained unverified at the review date. Jero test and weight construction · Candle model registry at review
Which result would change the assessment?
A decisive next comparison would hold the weights, prompts, scoring rules and hardware constant while varying depth. It would record task success, time to an accepted answer and energy per successful answer, including failed attempts. Memory deserves its own measure. In a separate four-pass Dory study at 16K context, LoopKV-M increased peak aggregate throughput from 219.7 to 247.9 tokens per second on a 32 GB RTX PRO 4500. The two peaks occurred at batch sizes of 24 and 48. That finding concerns a server handling many sequences together. A reader choosing how long to wait for one answer needs a single-request measurement. LoopKV, Appendix G, Table 20
Sources and scope of this assessment
The primary scientific reference is NVIDIA’s Dory1.0-4B: A Looped Normalized-GPT, arXiv:2611.03998, version 1, submitted 2 November 2026. The model assessment concerns the post-trained BF16 release. Where community experiments use converted, quantised or further-trained weights, their results are attributed to those artefacts. The review covers materials available on 17 November 2026. No independent reproduction of the twelve-metric comparison is claimed. Primary report · Versioned release record
Liked this post? Follow this blog to get more.