Thanks for the great overview and for mentioning our Full-Bandwidth Transformers work!
Another piece of work that closely related to the latent reasoning discussion is our recent paper "Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers" (https://arxiv.org/abs/2606.31779). It uses a looped padded Transformer to perform multiple reasoning steps in parallel in latent space, rather than generating them autoregressively, which improves inference efficiency. With intermediate supervision aligning latent states to explicit reasoning steps, we also find that the post-loop latent states can recover recognizable reasoning steps through the LM head.
We don’t view explicit CoT as a hard performance ceiling. In our experiments, it already exceeds explicit CoT in some settings, for example at the GPT-2 scale on GSM8K and at 3B on the out-of-domain average, while also improving inference efficiency through parallel latent reasoning. So the current evidence suggests the latent process can potentially improve both efficiency and performance, although the gains are not uniform across all model scales, datasets and tasks.
I agree, but I’d also say these are all additive. A capable model with lots of latent processing probably doesn’t need as much explicit CoT on most problems. But then on very hard problems it can still be beneficial to use longer CoT. I.e., what I mean is
Old model: solve problem A only with long CoT; not able to solve problem B at all
New model: able solve problem A with shorter CoT; able to solve problem B with longer CoT
At its core architectural level, are frontier models like GPT-6 Astra still powered by a predictive engine that calculates the probabilities of the next piece of text (or token)?
I'm wondering, why do most shown architectures loop All/bigger transformer blocks? Intuitively, it seems more sensible to me to loop several smaller block groups independently (and chain such loop blocks), as they share a depths specific 'granularity'/context, no? With looping one single block group, suddenly the model has to integrate both narrow attention and 'higher-order reasoning' across large context windows, smaller loops would instead learn to 'think more' on inputs of similar kind/depth/width.
Good question. I think that's probably the simplest thing to start with before getting more granular. But as I mentioned in the article, there are some architectures that loop only the middle transformer blocks, not all transformer blocks (like in the Latent Reasoning paper).
There is one thing that should be highlighted in the looped models. In real world scenarios you can't do early exits from loops. Imagine that you exited early on loop 1 and on the next word you need to exit on loop 4. This means that you need to recalculate KV cache for previous word for skipped loops. So basically you are using same compute budget as a full transformer and don't benefit from early exits. If you don't benefit from early exits than why spend time on learning models to exit earlier? Now when you understand that the compute budget for loop model and full transformer is the same what will you prefer? Quality of 8B transformer model will be better than 4 loops of 2B models while the compute budget is the same (except for storing additional 4B weights which is neglectable compared to KV cache).
Good point. In the mixture-of-recursion paper that I mentioned (and only briefly discussed) they shared 2 strategies to address this concern/issue.
So, let's say token A exits after loop 1 and token B goes up to loop 4 like you suggested
1) Recursion-wise KV caching. So, here they cap the attention to only attend tokens that are also at loop 4. Sure, this might influence quality, but maybe not that bad. It's kind of like DeepSeek Sparse attention then in a sense.
2) Recursive KV sharing. So here B simply attends the cached values from loop 1.
the AGENTS.md/SKILL.md aside is the most useful thing in here — I've been gutting my own instruction files lately and the newer models genuinely do better when you state the goal and step back. the hard part is knowing which workflows still need the hand-holding vs which are just legacy cruft I'm afraid to delete.
the AGENTS.md/SKILL.md aside is the most useful thing in here — I've been gutting my own instruction files lately and the newer models genuinely do better when you state the goal and step back. the hard part is knowing which workflows still need the hand-holding vs which are just legacy cruft I'm afraid to delete.
Thanks, Sebastian. Very interesting, as always. Is it possible that GAMAM+ labs are deliberately trying to reduce traces of problem-solving (I avoid the word "reasoning" to steer clear of anthropomorphism) in order to counter the distillation of their models?
Hm, good question. I think they deliberately hide the chain of thought from users because of this. But if a model uses somewhat shorter reasoning (Luna → Sol → Astra), I think it’s just more of a side effect of these models becoming more capable than them deliberately forcing this.
Thanks for the clear write-up. One thing I keep wondering about with looped architectures: is the number of loop iterations fixed at inference, or learned/early-exited per input?
If it's fixed, then the comparison against a non-looped model of similar size only means something at matched compute — otherwise "looping helps reasoning" and "more effective depth helps" are hard to tell apart. I'd expect real gains mainly where the task needs variable-depth computation (multi-step arithmetic, code tracing) rather than uniformly.
Do the papers you cover report matched-compute baselines, or mostly parameter-matched ones?
This is quite well written! I have always felt that looped architectures like UT were very promising and covered them in detail (https://www.intoai.pub/p/universal-reasoning-model). I still don't think that we are anywhere close to developing AGI, but maybe ASI someday?
Great article and well written. Easy to read and understand, compared to the other materials I have been reading recently.
Was a lot of work + a long weekend, so I really appreciate that!!
I'm sure I didn't understand it all, but it was a fascinating article.
Ha thanks! I hope it helped debunking some of the hiding of chains of thoughts though :)
Thanks for the great overview and for mentioning our Full-Bandwidth Transformers work!
Another piece of work that closely related to the latent reasoning discussion is our recent paper "Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers" (https://arxiv.org/abs/2606.31779). It uses a looped padded Transformer to perform multiple reasoning steps in parallel in latent space, rather than generating them autoregressively, which improves inference efficiency. With intermediate supervision aligning latent states to explicit reasoning steps, we also find that the post-loop latent states can recover recognizable reasoning steps through the LM head.
Oh thanks for sharing, I haven’t seen that one yet. Very interesting!
Is Explicit-CoT the hard/fixed boundary of performance or the (padded+looped+anchored) can give better yield and latent process efficiency both?
We don’t view explicit CoT as a hard performance ceiling. In our experiments, it already exceeds explicit CoT in some settings, for example at the GPT-2 scale on GSM8K and at 3B on the out-of-domain average, while also improving inference efficiency through parallel latent reasoning. So the current evidence suggests the latent process can potentially improve both efficiency and performance, although the gains are not uniform across all model scales, datasets and tasks.
I agree, but I’d also say these are all additive. A capable model with lots of latent processing probably doesn’t need as much explicit CoT on most problems. But then on very hard problems it can still be beneficial to use longer CoT. I.e., what I mean is
Old model: solve problem A only with long CoT; not able to solve problem B at all
New model: able solve problem A with shorter CoT; able to solve problem B with longer CoT
At its core architectural level, are frontier models like GPT-6 Astra still powered by a predictive engine that calculates the probabilities of the next piece of text (or token)?
Thx
I’m 99% certain that they still use autoregressive transformers, yes
TYVM
Thank you for your time and effort, Sebastian. I learned a lot from this
Glad to hear this, thanks!!
I'm wondering, why do most shown architectures loop All/bigger transformer blocks? Intuitively, it seems more sensible to me to loop several smaller block groups independently (and chain such loop blocks), as they share a depths specific 'granularity'/context, no? With looping one single block group, suddenly the model has to integrate both narrow attention and 'higher-order reasoning' across large context windows, smaller loops would instead learn to 'think more' on inputs of similar kind/depth/width.
Good question. I think that's probably the simplest thing to start with before getting more granular. But as I mentioned in the article, there are some architectures that loop only the middle transformer blocks, not all transformer blocks (like in the Latent Reasoning paper).
There is one thing that should be highlighted in the looped models. In real world scenarios you can't do early exits from loops. Imagine that you exited early on loop 1 and on the next word you need to exit on loop 4. This means that you need to recalculate KV cache for previous word for skipped loops. So basically you are using same compute budget as a full transformer and don't benefit from early exits. If you don't benefit from early exits than why spend time on learning models to exit earlier? Now when you understand that the compute budget for loop model and full transformer is the same what will you prefer? Quality of 8B transformer model will be better than 4 loops of 2B models while the compute budget is the same (except for storing additional 4B weights which is neglectable compared to KV cache).
Good point. In the mixture-of-recursion paper that I mentioned (and only briefly discussed) they shared 2 strategies to address this concern/issue.
So, let's say token A exits after loop 1 and token B goes up to loop 4 like you suggested
1) Recursion-wise KV caching. So, here they cap the attention to only attend tokens that are also at loop 4. Sure, this might influence quality, but maybe not that bad. It's kind of like DeepSeek Sparse attention then in a sense.
2) Recursive KV sharing. So here B simply attends the cached values from loop 1.
(They found method 1 worked slightly better)
I am currently reading the building an llm from scratch, great book and another great article, thanks! 👏🏻👏🏻
Glad you are liking the book & article!
rick in content as well as context, immersive...I must have it printed and keep it on my table for frequent reference...
Amazing analysis :)
the AGENTS.md/SKILL.md aside is the most useful thing in here — I've been gutting my own instruction files lately and the newer models genuinely do better when you state the goal and step back. the hard part is knowing which workflows still need the hand-holding vs which are just legacy cruft I'm afraid to delete.
the AGENTS.md/SKILL.md aside is the most useful thing in here — I've been gutting my own instruction files lately and the newer models genuinely do better when you state the goal and step back. the hard part is knowing which workflows still need the hand-holding vs which are just legacy cruft I'm afraid to delete.
Thanks, Sebastian. Very interesting, as always. Is it possible that GAMAM+ labs are deliberately trying to reduce traces of problem-solving (I avoid the word "reasoning" to steer clear of anthropomorphism) in order to counter the distillation of their models?
Hm, good question. I think they deliberately hide the chain of thought from users because of this. But if a model uses somewhat shorter reasoning (Luna → Sol → Astra), I think it’s just more of a side effect of these models becoming more capable than them deliberately forcing this.
Thank you for the article. I always enjoy reading your interpretations and viewpoints about LLM architecture.
Thanks!
Thanks for the clear write-up. One thing I keep wondering about with looped architectures: is the number of loop iterations fixed at inference, or learned/early-exited per input?
If it's fixed, then the comparison against a non-looped model of similar size only means something at matched compute — otherwise "looping helps reasoning" and "more effective depth helps" are hard to tell apart. I'd expect real gains mainly where the task needs variable-depth computation (multi-step arithmetic, code tracing) rather than uniformly.
Do the papers you cover report matched-compute baselines, or mostly parameter-matched ones?
It depends. There are different flavors of looping. The rumors only go so far and say that they are using looped transformers, but not which type.
This is quite well written! I have always felt that looped architectures like UT were very promising and covered them in detail (https://www.intoai.pub/p/universal-reasoning-model). I still don't think that we are anywhere close to developing AGI, but maybe ASI someday?
Very nice article! Yeah, I usually keep away from the AGI discussions
Thanks! Also a wise choice regarding keeping away haha