Hi all, I just also added a lot of the additional content I originally trimmed from this article to an optional appendix (plus YaRN and ALiBi). If you already read the article and are looking for this bonus content, search for the "appendix" in this article. You can also find a direct link here, but it makes most sense to read passages of this where it is referenced throughout the text: https://sebastianraschka.com/supplementary/positional-embeddings/
The MiniMax-M3 example makes the partial RoPE tradeoff pretty clear: you shrink the cosine and sine tables, but the KV cache still stores the full key. If the memory win is that small, I wonder why it's showing up in 2026 models at all — is there a quality reason the table-size math doesn't capture, or is it mostly carried over from an earlier implementation?
Yes exactly. The experiments in the original NoPE were perhaps too small to have any strong conclusions. But in any case, they were more like "it doesn't make the model (much) worse" versus "it actually improves the model". However, maybe there is an effect at a larger scale that makes this worthwhile; maybe at certain long-context tasks. Unfortunately, I am not aware of any studies though.
MLA could be part of it, but it is a slightly different lever. MLA shrinks the KV cache by storing a low-rank latent, whereas partial RoPE mainly shrinks the rotation tables and still leaves a full-size key in the cache. In the decoupled-RoPE / MLA design they often appear together: position is applied only on a small set of dimensions while the latent path stays compact. That architectural pairing is a better explanation than the table-size savings alone, even if the memory win from partial RoPE by itself looks small.
The rope_scaling block in config.json is where this shows up in serving. When Llama 3.1 shipped with rope_type "llama3", older transformers versions refused to load it, and a workaround passed around on the model's Hugging Face discussions was to rewrite it as type "dynamic", which loads fine but runs a different scaling scheme than the one the checkpoint specifies. After any engine upgrade or config edit, diff the loaded rope settings against the checkpoint's config.json.
Ah yes this is actually how/when I learned the RoPE details years ago. Was implementing that for LitGPT open source lib and I remember llama 3 (or 3.1 or 3.2, def one of these) was a bit tricky due to rescaling and it took me a lot of time to get right.
between the original additive positional encoding vs RoPe, saw that RoPe avoided the interaction terms between token position and embedding. this enhanced the ability of the interaction term to focus more on the relative distance between tokens.
isnt that the key aspect which improved RoPe's performance?
then, in icml 2026, came across polar positional encoding (which the authors called as PoPE)
Decoupling the “What” and “Where” with Polar Coordinate Positional Embedding
in this authors pointed out that there is still an interaction term between query/key in RoPE and proposed a neat way to avoid that. results in paper are impressive. however not seen this in any open source implementations
Hi all, I just also added a lot of the additional content I originally trimmed from this article to an optional appendix (plus YaRN and ALiBi). If you already read the article and are looking for this bonus content, search for the "appendix" in this article. You can also find a direct link here, but it makes most sense to read passages of this where it is referenced throughout the text: https://sebastianraschka.com/supplementary/positional-embeddings/
The MiniMax-M3 example makes the partial RoPE tradeoff pretty clear: you shrink the cosine and sine tables, but the KV cache still stores the full key. If the memory win is that small, I wonder why it's showing up in 2026 models at all — is there a quality reason the table-size math doesn't capture, or is it mostly carried over from an earlier implementation?
Yes exactly. The experiments in the original NoPE were perhaps too small to have any strong conclusions. But in any case, they were more like "it doesn't make the model (much) worse" versus "it actually improves the model". However, maybe there is an effect at a larger scale that makes this worthwhile; maybe at certain long-context tasks. Unfortunately, I am not aware of any studies though.
I had the same question in mind while reading the article.
In addition to what Sebastian has already said regarding the long context, could it perhaps also be due to the fact that (some) modern LLMs use MLA?
MLA could be part of it, but it is a slightly different lever. MLA shrinks the KV cache by storing a low-rank latent, whereas partial RoPE mainly shrinks the rotation tables and still leaves a full-size key in the cache. In the decoupled-RoPE / MLA design they often appear together: position is applied only on a small set of dimensions while the latent path stays compact. That architectural pairing is a better explanation than the table-size savings alone, even if the memory win from partial RoPE by itself looks small.
What I meant to say is that partial RoPE has the benefit in MLA that the latent vector doesn't need to be decompressed (unlike with full RoPE).
Great article, as always! And a huge THANK YOU for the RoPE section! :)
Thanks! And if I recall correctly, you suggested/urged me many times to cover RoPE. So thanks for insisting 😆😅
Best explanation of RoPE! Thank you, @Sebastian Raschka, PhD 🙂
Thanks, Samir, I appreciate it!!
The rope_scaling block in config.json is where this shows up in serving. When Llama 3.1 shipped with rope_type "llama3", older transformers versions refused to load it, and a workaround passed around on the model's Hugging Face discussions was to rewrite it as type "dynamic", which loads fine but runs a different scaling scheme than the one the checkpoint specifies. After any engine upgrade or config edit, diff the loaded rope settings against the checkpoint's config.json.
Ah yes this is actually how/when I learned the RoPE details years ago. Was implementing that for LitGPT open source lib and I remember llama 3 (or 3.1 or 3.2, def one of these) was a bit tricky due to rescaling and it took me a lot of time to get right.
thanks for the writeup.
between the original additive positional encoding vs RoPe, saw that RoPe avoided the interaction terms between token position and embedding. this enhanced the ability of the interaction term to focus more on the relative distance between tokens.
isnt that the key aspect which improved RoPe's performance?
then, in icml 2026, came across polar positional encoding (which the authors called as PoPE)
Decoupling the “What” and “Where” with Polar Coordinate Positional Embedding
https://arxiv.org/pdf/2509.10534
in this authors pointed out that there is still an interaction term between query/key in RoPE and proposed a neat way to avoid that. results in paper are impressive. however not seen this in any open source implementations
any thoughts on that ?