I just wanted to drop you a quick note to say how much I loved this article, as always! It provides a fantastic synthesis, especially as a window into your book Build a Reasoning Model (From Scratch).
Since I frequently read your work, I wanted to share my typical routine for absorbing your articles:
As a French speaker, I often use my browser's built-in translation tool for the first read. It allows me to quickly grasp the core essence of the article without getting slowed down by language nuances.
Once I'm done reading, I turn to my favorite AI agent (or whichever one I am currently exploring) and ask it to generate a 10-question multiple-choice quiz based on your text, followed by 2 open-ended essay questions. My goal isn't just to validate that I understood the material, but to ensure I can confidently explain these concepts to my colleagues. I work in applied AI, building agentic use cases, so this has proven to be an excellent strategy for continuous learning. It keeps me sharp and ensures I don't become passive or "enslaved" by the technology.
This actually made me wonder: What is your personal strategy for continuous learning? More specifically, how do you keep yourself from succumbing to convenience and falling victim to automation bias (like relying too heavily on AI summaries to write or research your own articles)?
Lastly, I’ve been seriously considering your paid subscription. My only hesitation is that my time is quite limited, and I realistically won't be able to keep up with your incredibly prolific output of books and articles! Beyond the excellent deed of supporting your work, what would you say is the key value or gain of the subscription for someone in my position?
Thank you again for your amazing contributions to the community.
Thanks for the nice comment! This is actually a cool example of how LLMs/AI augments the learning and reading experience (translation, quizzes) rather than replacing it!
> This actually made me wonder: What is your personal strategy for continuous learning?
I use(d) to do this manually: make question-answer pairs for things I want to retain and put that into Anki for spaced repetition. But for research papers, I usually take notes (or write articles like this one :D), and then I sometimes ask LLMs to check it carefully for mistakes and conceptual misunderstandings.
> Beyond the excellent deed of supporting your work, what would you say is the key value or gain of the subscription for someone in my position?
I would say that if you like my articles and find them useful, then the paid subscription will probably also be useful (plus, as you mentioned, for independent people like me not affiliated with a company and a regular salary, this helps support this work)
Fantastic article, thank you for sharing your seemingly deep knowledge about this domain in a way that even “laymen” like myself can learn and appreciate.
Thanks for getting a copy! (Got my author copies last week, and I hope you get your copy soon too. I think due to the higher quality color print it’s a bit slower, but I think it’s worth it!)
Probably a naïve question but what prevents closed source model owners from introducing another component of the objective function that optimises for a minimum output token count? There is definitely a financial incentive.
It's probably the competition that prevents it 😅. I.e., if you had a model that is always very verbose (e.g., R1), that'd be super annoying because it'd be too expensive and to high-latency for many use cases. Companies would then switch to providers that offer a wider breadth of models / modes for different use cases.
But in general, yes, it may be that some models are still unnecessarily verbose. If you ask a model "1+1?" some models will provide a 2 sentence answer. Some will just anser "2". Right now, I don't think any of that is done on purpose though to make more money through the API.
Great article. As a very practical application of this: Should we avoid switching thinking level mid thread?
Both OpenAI and Anthropic support this but after reading the article it seems this may be an area that is not covered as much by training and probably best to avoid. Any thoughts on this?
I wouldn't switch it mid-task, i.e., when the model is calling tools, evaluating the results, etc. in a call stack (but that level of switching would currently not be supported by agent harnesses anyways); but I think it should be fine to change reasoning modes within a task / chat. E.g., when you have a project where you are working on a paper, within the same thread, I would use a high reasoning level to "Carefully check math formula (2)" and a low reasoning effort for "Check if all section titles use consistent sentence case" or sth like that
Reasoning effort seems like a useful clinical-AI control surface because not every step deserves more model thought.
For OR-adjacent tools, I would want low effort on deterministic source retrieval, higher effort only when sources conflict, and a hard stop before the synthesis changes workflow state.
The eval should ask whether extra reasoning reduced the unresolved exception, not just whether the final answer improved.
The insight about calibrating reasoning effort is crucial, but it assumes the reasoning happens inside one model, controlled by one operator. What changes when you distribute that reasoning across heterogeneous agents that don't trust each other? You'd need protocols for negotiating effort *between* systems, not just allocating it within one. That's where the real leverage sits.
That’s true, you ideally want an orchestrator on the harness/agent level. But for that you still need each LLM to individually support these reasoning effort settings.
What I appreciated most about this piece is that it doesn't treat AI as the destination—it treats it as a catalyst for better thinking. That's an important distinction because technology evolves quickly, while the ability to think critically, adapt, and make sound judgments remains valuable regardless of which tools we use.
One thought that came to mind while reading is that AI doesn't simply change how we work; it changes the kinds of questions we're willing to ask. In many ways, its greatest value isn't producing faster answers but encouraging deeper exploration of ideas we might not have considered otherwise.
I'm curious: looking ahead a few years, which human capability do you think will become more valuable because AI continues to improve?
The more capable AI becomes, the more I think qualities like empathy, judgment, curiosity, and genuine human connection become differentiators rather than optional skills. AI can help us think faster, but it's still the human touch that gives ideas meaning and direction.
Very interesting article! I wonder how much reasoning in latent space or COCONUT (https://arxiv.org/pdf/2412.06769) is used in modern LLMs. I reckon it would be quite a difference in training and using the models. Do you know about the state of latent space reasoning implementations and is there really such a difference? Im interested about your perspective.
Oh Coconut! Very interesting and promising, and in hindsight an obvious & intuitive thing to do. But I haven’t seen anyone (any notable open-weight LLM) adopting it. So maybe it either doesn’t work too well in larger LLMs and production and/or everyone is too risk adverse.
@Devansh might have some more interesting thoughts here
Thakn you for the mention Seb. Yes I have lots to say here.
A few things—
A lot of model providers are moving towards latent reasoning already. The newer models all have started to incorporate them to a certain degree. Out and out latent reasoning is not coming for a while because of the risk aversion, but I do talk to a lot of model builders and they’re actively looking at it and working that in.
Coconut is actually the wrong template to look at because it was a bit too regimented. When ti comes to latent reasonign setups, looking into “Diffusion Guided Language Modeling” and “Large Concept Models” is much better. They have a much better ramp from current LLMs into the more latent reasoning aspects.
There is also some very good work to look into with soft prompting that I recommend, since that tends to push the boundaries in immediuately useful ways.
Lastly, if I may recommend our own work, we just released our research with Latent Space Reasoning with pertubations, where we showed two scaled random tokens appended at the start of a prompt can unlock a model’s capacities better by breaking out of various attention sinks. It allowed us to match or beat Qwen 14B with 4B, no additional training or inference costs added.
Regarding the newer models you mentioned, do you have some concrete ones in mind there? Like some good open-weight models as templates or case studies to recommend? For some reason, all the open weight models I am looking at are still not doing latent reasoning: https://sebastianraschka.com/llm-architecture-gallery/
I mean, there is multi-head latent attention, but that's just general compressed attention representation and not directly related to reasoning. Similar with the DeltaNet-style models where there is a recurrent state but not specific to reasoning. Regarding reasoning, all of them still use discrete tokens (e.g., within a <think> </think> block)
Newer modes-- Claude from 4.8 and GPT from 5.5 have started to incorporate more Latent Space reasoning. I can't share more since confidentiality.
The open weight ones, all I can say for now is that several labs are working on them. They don't want to publish something that has errors so muhc more safe for now, but you will see updates from them as they get more comfortable.
What an insightful article! I especially appreciated the discussion on when to enable or disable reasoning mode and how to implement and adjust different levels of reasoning effort. Those practical insights were incredibly helpful.
One finding that really stood out to me was Figure 24 in Section 5.4—the fact that the curves overlap. It illustrates an important trade-off: a smaller model with higher reasoning effort can sometimes achieve performance comparable to a larger model using lower reasoning effort. In practice, this means there are multiple paths to better performance: using a larger model, increasing reasoning effort, or combining both. The optimal choice ultimately depends on the desired balance between accuracy, cost, and latency.
It also made me wonder whether different AI companies optimize for different priorities when building the first generation of their models. Depending on their budget, available compute, target latency, and accuracy requirements, startups and established labs may make very different architectural and training decisions.
Would you consider collecting this information and presenting a side-by-side comparison of well-known AI companies in a future post? A table comparing their optimization priorities (e.g., accuracy, inference cost, training compute, latency, model size, and reasoning strategy) would be incredibly valuable for anyone interested in building an AI technology company.
It also made me wonder whether different AI companies optimize for different priorities when building the first generation of their models.
I am pretty sure they do. Otherwise we’d all have models of the same size.
Regarding the comparison you mentioned. That’s tricky in practice. Some tend to focus more on “absolute state of the art performance no matter the cost” (See Mythos), but most focus on state of the art performance given sane inference costs (GPT 5.6, GLM 5.2, Kimi K3).
But ultimately, most companies optimize to be somewhere at the pareto frontier in terms of performance and cost
This is a great brakdown. The part that hits home for me is that reasoning effort is basically a compute dial, and on local hardware that dial is your electricity bill and your context budget at the same time. When I benchmark models on my own cards I have started treating a longer thinking trace the same way I treats a longer cntext, it eats decode speed and VRAM in ways the headline quality score never shows.
One implication of your two-knob framing is that model size and reasoning effort should be optimized jointly at the task-router level, not chosen once per application. A smaller model at high effort can overlap a larger model at low effort, but the cheapest point likely shifts by task type, tool latency, and failure cost. That makes “accuracy per dollar” too coarse for agents; completed-task cost should include retries, tool calls, and verification. Do you expect future training to condition effort on an explicit cost budget, or will external routers remain better calibrated?
I think models will continue receiving effort modes via system prompt just to give the option of control. But the appropriate reasoning budget will be inferred by agents (and routers within agents), yes. Agents may not be aware of API costs though, like "accuracy per dollar" (besides inferring this from public benchmarks). So agents would probably rather optimize for success / latency or sth like that.
That makes sense. A hybrid seems likely: effort modes remain as a coarse control, agents infer the reasoning needed for success and latency, while an external policy layer translates cost constraints into budgets the agent never needs to see directly. In that setup, cost awareness lives in the controller, not necessarily the model.
Fascinating. So it seems like private labs have started experimenting with it while open source is still more risk averse. I dont know the training differences in detail but latent reasoning probably needs more walking on a fine line. Thank you so much for the responses and im definetly excited for some public innovations in that domain. It definetly seems like a reasonable thing to test.
Hi Sebastian,
I just wanted to drop you a quick note to say how much I loved this article, as always! It provides a fantastic synthesis, especially as a window into your book Build a Reasoning Model (From Scratch).
Since I frequently read your work, I wanted to share my typical routine for absorbing your articles:
As a French speaker, I often use my browser's built-in translation tool for the first read. It allows me to quickly grasp the core essence of the article without getting slowed down by language nuances.
Once I'm done reading, I turn to my favorite AI agent (or whichever one I am currently exploring) and ask it to generate a 10-question multiple-choice quiz based on your text, followed by 2 open-ended essay questions. My goal isn't just to validate that I understood the material, but to ensure I can confidently explain these concepts to my colleagues. I work in applied AI, building agentic use cases, so this has proven to be an excellent strategy for continuous learning. It keeps me sharp and ensures I don't become passive or "enslaved" by the technology.
This actually made me wonder: What is your personal strategy for continuous learning? More specifically, how do you keep yourself from succumbing to convenience and falling victim to automation bias (like relying too heavily on AI summaries to write or research your own articles)?
Lastly, I’ve been seriously considering your paid subscription. My only hesitation is that my time is quite limited, and I realistically won't be able to keep up with your incredibly prolific output of books and articles! Beyond the excellent deed of supporting your work, what would you say is the key value or gain of the subscription for someone in my position?
Thank you again for your amazing contributions to the community.
Best regards,
Denis
Thanks for the nice comment! This is actually a cool example of how LLMs/AI augments the learning and reading experience (translation, quizzes) rather than replacing it!
> This actually made me wonder: What is your personal strategy for continuous learning?
I use(d) to do this manually: make question-answer pairs for things I want to retain and put that into Anki for spaced repetition. But for research papers, I usually take notes (or write articles like this one :D), and then I sometimes ask LLMs to check it carefully for mistakes and conceptual misunderstandings.
> Beyond the excellent deed of supporting your work, what would you say is the key value or gain of the subscription for someone in my position?
I would say that if you like my articles and find them useful, then the paid subscription will probably also be useful (plus, as you mentioned, for independent people like me not affiliated with a company and a regular salary, this helps support this work)
Fantastic article, thank you for sharing your seemingly deep knowledge about this domain in a way that even “laymen” like myself can learn and appreciate.
Thanks for the nice feedback!
Nice article, Sebastian, thank you!
> I also included the corresponding figure below so you don’t have to scroll all the way up.
Appreciate these thoughtful little touches - they make a big difference.
BTW, I am waiting for my printed copy of BRMFS!!
Thanks for getting a copy! (Got my author copies last week, and I hope you get your copy soon too. I think due to the higher quality color print it’s a bit slower, but I think it’s worth it!)
Probably a naïve question but what prevents closed source model owners from introducing another component of the objective function that optimises for a minimum output token count? There is definitely a financial incentive.
It's probably the competition that prevents it 😅. I.e., if you had a model that is always very verbose (e.g., R1), that'd be super annoying because it'd be too expensive and to high-latency for many use cases. Companies would then switch to providers that offer a wider breadth of models / modes for different use cases.
But in general, yes, it may be that some models are still unnecessarily verbose. If you ask a model "1+1?" some models will provide a 2 sentence answer. Some will just anser "2". Right now, I don't think any of that is done on purpose though to make more money through the API.
Great article. As a very practical application of this: Should we avoid switching thinking level mid thread?
Both OpenAI and Anthropic support this but after reading the article it seems this may be an area that is not covered as much by training and probably best to avoid. Any thoughts on this?
I wouldn't switch it mid-task, i.e., when the model is calling tools, evaluating the results, etc. in a call stack (but that level of switching would currently not be supported by agent harnesses anyways); but I think it should be fine to change reasoning modes within a task / chat. E.g., when you have a project where you are working on a paper, within the same thread, I would use a high reasoning level to "Carefully check math formula (2)" and a low reasoning effort for "Check if all section titles use consistent sentence case" or sth like that
Reasoning effort seems like a useful clinical-AI control surface because not every step deserves more model thought.
For OR-adjacent tools, I would want low effort on deterministic source retrieval, higher effort only when sources conflict, and a hard stop before the synthesis changes workflow state.
The eval should ask whether extra reasoning reduced the unresolved exception, not just whether the final answer improved.
The insight about calibrating reasoning effort is crucial, but it assumes the reasoning happens inside one model, controlled by one operator. What changes when you distribute that reasoning across heterogeneous agents that don't trust each other? You'd need protocols for negotiating effort *between* systems, not just allocating it within one. That's where the real leverage sits.
That’s true, you ideally want an orchestrator on the harness/agent level. But for that you still need each LLM to individually support these reasoning effort settings.
great!!! i love it!
What I appreciated most about this piece is that it doesn't treat AI as the destination—it treats it as a catalyst for better thinking. That's an important distinction because technology evolves quickly, while the ability to think critically, adapt, and make sound judgments remains valuable regardless of which tools we use.
One thought that came to mind while reading is that AI doesn't simply change how we work; it changes the kinds of questions we're willing to ask. In many ways, its greatest value isn't producing faster answers but encouraging deeper exploration of ideas we might not have considered otherwise.
I'm curious: looking ahead a few years, which human capability do you think will become more valuable because AI continues to improve?
the so-called “human touch” :)
I like that perspective. 😊
The more capable AI becomes, the more I think qualities like empathy, judgment, curiosity, and genuine human connection become differentiators rather than optional skills. AI can help us think faster, but it's still the human touch that gives ideas meaning and direction.
Thanks for the thoughtful reply!
Great to read an informative article on AI and ML
Very interesting article! I wonder how much reasoning in latent space or COCONUT (https://arxiv.org/pdf/2412.06769) is used in modern LLMs. I reckon it would be quite a difference in training and using the models. Do you know about the state of latent space reasoning implementations and is there really such a difference? Im interested about your perspective.
Oh Coconut! Very interesting and promising, and in hindsight an obvious & intuitive thing to do. But I haven’t seen anyone (any notable open-weight LLM) adopting it. So maybe it either doesn’t work too well in larger LLMs and production and/or everyone is too risk adverse.
@Devansh might have some more interesting thoughts here
Thakn you for the mention Seb. Yes I have lots to say here.
A few things—
A lot of model providers are moving towards latent reasoning already. The newer models all have started to incorporate them to a certain degree. Out and out latent reasoning is not coming for a while because of the risk aversion, but I do talk to a lot of model builders and they’re actively looking at it and working that in.
Coconut is actually the wrong template to look at because it was a bit too regimented. When ti comes to latent reasonign setups, looking into “Diffusion Guided Language Modeling” and “Large Concept Models” is much better. They have a much better ramp from current LLMs into the more latent reasoning aspects.
There is also some very good work to look into with soft prompting that I recommend, since that tends to push the boundaries in immediuately useful ways.
Lastly, if I may recommend our own work, we just released our research with Latent Space Reasoning with pertubations, where we showed two scaled random tokens appended at the start of a prompt can unlock a model’s capacities better by breaking out of various attention sinks. It allowed us to match or beat Qwen 14B with 4B, no additional training or inference costs added.
Article: https://www.artificialintelligencemadesimple.com/p/the-cheapest-way-to-make-open-weight
Awesome, thanks a lot for the detailed note!
Regarding the newer models you mentioned, do you have some concrete ones in mind there? Like some good open-weight models as templates or case studies to recommend? For some reason, all the open weight models I am looking at are still not doing latent reasoning: https://sebastianraschka.com/llm-architecture-gallery/
I mean, there is multi-head latent attention, but that's just general compressed attention representation and not directly related to reasoning. Similar with the DeltaNet-style models where there is a recurrent state but not specific to reasoning. Regarding reasoning, all of them still use discrete tokens (e.g., within a <think> </think> block)
Newer modes-- Claude from 4.8 and GPT from 5.5 have started to incorporate more Latent Space reasoning. I can't share more since confidentiality.
The open weight ones, all I can say for now is that several labs are working on them. They don't want to publish something that has errors so muhc more safe for now, but you will see updates from them as they get more comfortable.
Also not sure if you have audio models but this is one cool example-- https://arxiv.org/html/2605.23463v1
Hey, realized I forgot to tag you so here—
https://substack.com/@chocolatemilkcultleader/note/c-302337182?r=4tnbw&utm_source=notes-share-action&utm_medium=web
What an insightful article! I especially appreciated the discussion on when to enable or disable reasoning mode and how to implement and adjust different levels of reasoning effort. Those practical insights were incredibly helpful.
One finding that really stood out to me was Figure 24 in Section 5.4—the fact that the curves overlap. It illustrates an important trade-off: a smaller model with higher reasoning effort can sometimes achieve performance comparable to a larger model using lower reasoning effort. In practice, this means there are multiple paths to better performance: using a larger model, increasing reasoning effort, or combining both. The optimal choice ultimately depends on the desired balance between accuracy, cost, and latency.
It also made me wonder whether different AI companies optimize for different priorities when building the first generation of their models. Depending on their budget, available compute, target latency, and accuracy requirements, startups and established labs may make very different architectural and training decisions.
Would you consider collecting this information and presenting a side-by-side comparison of well-known AI companies in a future post? A table comparing their optimization priorities (e.g., accuracy, inference cost, training compute, latency, model size, and reasoning strategy) would be incredibly valuable for anyone interested in building an AI technology company.
Thanks for the comments!
It also made me wonder whether different AI companies optimize for different priorities when building the first generation of their models.
I am pretty sure they do. Otherwise we’d all have models of the same size.
Regarding the comparison you mentioned. That’s tricky in practice. Some tend to focus more on “absolute state of the art performance no matter the cost” (See Mythos), but most focus on state of the art performance given sane inference costs (GPT 5.6, GLM 5.2, Kimi K3).
But ultimately, most companies optimize to be somewhere at the pareto frontier in terms of performance and cost
This is a great brakdown. The part that hits home for me is that reasoning effort is basically a compute dial, and on local hardware that dial is your electricity bill and your context budget at the same time. When I benchmark models on my own cards I have started treating a longer thinking trace the same way I treats a longer cntext, it eats decode speed and VRAM in ways the headline quality score never shows.
Yes, exactly, kind of like the dial inside a fridge to set the temperature 😅
One implication of your two-knob framing is that model size and reasoning effort should be optimized jointly at the task-router level, not chosen once per application. A smaller model at high effort can overlap a larger model at low effort, but the cheapest point likely shifts by task type, tool latency, and failure cost. That makes “accuracy per dollar” too coarse for agents; completed-task cost should include retries, tool calls, and verification. Do you expect future training to condition effort on an explicit cost budget, or will external routers remain better calibrated?
I think models will continue receiving effort modes via system prompt just to give the option of control. But the appropriate reasoning budget will be inferred by agents (and routers within agents), yes. Agents may not be aware of API costs though, like "accuracy per dollar" (besides inferring this from public benchmarks). So agents would probably rather optimize for success / latency or sth like that.
That makes sense. A hybrid seems likely: effort modes remain as a coarse control, agents infer the reasoning needed for success and latency, while an external policy layer translates cost constraints into budgets the agent never needs to see directly. In that setup, cost awareness lives in the controller, not necessarily the model.
A small mathematical observation about reasoning-effort control.
Let x in X denote the state of a reasoning process, and let
e = phi(x)
denote its scalar reasoning-effort descriptor.
Any controller based only on reasoning effort necessarily factors through that projection:
pi(x) = g(phi(x))
Such a controller can be decision-sufficient only if
phi(x1) = phi(x2)
=>
a*(x1) = a*(x2)
But if there exist two states x1, x2 in X such that
phi(x1) = phi(x2)
while
a*(x1) != a*(x2),
then no policy that factors only through reasoning effort can be correct for both states.
Equivalently:
phi(x1) = phi(x2) -/-> a*(x1) = a*(x2)
Reasoning effort can therefore be a useful control variable without being a sufficient statistic for the reasoning state.
Fascinating. So it seems like private labs have started experimenting with it while open source is still more risk averse. I dont know the training differences in detail but latent reasoning probably needs more walking on a fine line. Thank you so much for the responses and im definetly excited for some public innovations in that domain. It definetly seems like a reasonable thing to test.