Apart from the architectural differences, what would be interesting to know is on which text data the LLMs have been trained on. From my pov, it's really unfortunate that this info is typically not disclosed, even for open-source LLMs. Not just the amount of training data (e.g. number of tokens) but also the data quality as factors for a true scientific comparison.
Amazing article! This is evidence that we haven't hit a wall yet with LLMs as all these labs haven't converged to the same architectures.
Cohere Labs is also doing some great work for open source and have some interesting work. I feel a lot of people don't know who they are as they are trying to appeal to businesses/governments.
Good point. Cohere flies a bit under the radar in the open-weight LLM circles. Maybe because of the enterprise focus that you mentioned. I think their Command model is also > 1.5 years old now and more RAG-focused so I didn't include it (but please correct me if I'm wrong).
Thank you so much for consistently sharing such valuable articles.
One more thing I must tell the whole world that your book, Build a Large Language Model(From Scratch), is definitely worth a read! It was the true catalyst that sparked my journey into the LLM field!
Good question. This is based on the results shown in Figure 4. You are right, you’d expect slightly worse performance as it’s a workaround/approximation. My guess it’s because the additional “layer” adds more expressiveness.
I might have found another small nitpick: In Fig. 19, on the right side, the comment for the intermediate layer dimension should be 1536 (as it must be divisible by 8). Would you mind checking if this is correct?
:D Thanks! Could you please also update Fig. 1 for Qwen3?
I have also noticed that the arrows on the left side of Fig. 10 seem a bit misaligned. Would you mind taking a look to see if they could be adjusted for clarity?
BTW I like really the visual comparisons of the architectures!
Thanks for the article! Just a small correction, Gemma 3 uses GELU not SiLU in the feed forward (in the figure comparing Gemma 3 27B to Mistral Small 3.1 24B).
Thanks for the note! You are absolutely right, must have been a copy&paste error or so (I coincidentally had a section on GELU vs SiLU in my most recent article). Anyways, should be fixed now!
Very Amazing article.!!! Thanks, but I have a question, Qwen3-235B-A22B employs a fully integrated Mixture-of-Experts (MoE) architecture, rather than alternating between dense and MoE layers every two layers.
Are you interested in explaining the principles of RoPE (which has almost become a de facto standard in every architecture mentioned in your articles)? The mathematical derivation process is really giving me a headache.😭
Great insight! I think the biggest leap forward will come from architectural changes that eliminate the auto-regressive nature of current LLMs. Curious to see who comes up with clever solutions to break that paradigm.
Apart from the architectural differences, what would be interesting to know is on which text data the LLMs have been trained on. From my pov, it's really unfortunate that this info is typically not disclosed, even for open-source LLMs. Not just the amount of training data (e.g. number of tokens) but also the data quality as factors for a true scientific comparison.
Agreed. I summarized some of approaches last year (https://magazine.sebastianraschka.com/p/new-llm-pre-training-and-post-training), but it's tricky due to the lack of full disclosure. Btw on that note I recently stumbled upon https://libremodel.xyz/, which aims to be transparent in that regard. It's not going to be a SOTA model, but still interesting.
With a budget of under $1,000, wow!
I found the recent SmolLM3 data and training procedure quite interesting: https://huggingface.co/blog/smollm3#data-mixture-and-training-stages
Amazing article! This is evidence that we haven't hit a wall yet with LLMs as all these labs haven't converged to the same architectures.
Cohere Labs is also doing some great work for open source and have some interesting work. I feel a lot of people don't know who they are as they are trying to appeal to businesses/governments.
Good point. Cohere flies a bit under the radar in the open-weight LLM circles. Maybe because of the enterprise focus that you mentioned. I think their Command model is also > 1.5 years old now and more RAG-focused so I didn't include it (but please correct me if I'm wrong).
Really amazing and very helpful!
Thank you so much for consistently sharing such valuable articles.
One more thing I must tell the whole world that your book, Build a Large Language Model(From Scratch), is definitely worth a read! It was the true catalyst that sparked my journey into the LLM field!
Thanks for the kind words and also kindly recommending my book! It’s awesome to hear that it’s been a career-starter!
Amazing article!
I appreciate all the work you are putting in compiling this information
> MLA is a clever trick to reduce KV cache memory use while even slightly outperforming MHA in terms of modeling performance.
What’s the intuition for why MLA improves performance? It seems that compressing would if anything be slightly lossy.
Is this just a non-stat-sig result and we should just say “no evidence that it’s worse”? Or is there some mechanical reason that it is indeed better?
Good question. This is based on the results shown in Figure 4. You are right, you’d expect slightly worse performance as it’s a workaround/approximation. My guess it’s because the additional “layer” adds more expressiveness.
Great overview!
As a small side note, I noticed that in Fig. 4, the bottom left comment appears to read 'MQA.' Should this perhaps be 'MLA' instead?
Good catch, thanks!
I might have found another small nitpick: In Fig. 19, on the right side, the comment for the intermediate layer dimension should be 1536 (as it must be divisible by 8). Would you mind checking if this is correct?
Thanks, that should indeed be 1536 not 1535 (I need to start wearing glasses)
:D Thanks! Could you please also update Fig. 1 for Qwen3?
I have also noticed that the arrows on the left side of Fig. 10 seem a bit misaligned. Would you mind taking a look to see if they could be adjusted for clarity?
BTW I like really the visual comparisons of the architectures!
👍👍
Thanks!
Thanks for the article! Just a small correction, Gemma 3 uses GELU not SiLU in the feed forward (in the figure comparing Gemma 3 27B to Mistral Small 3.1 24B).
Thanks for the note! You are absolutely right, must have been a copy&paste error or so (I coincidentally had a section on GELU vs SiLU in my most recent article). Anyways, should be fixed now!
Thanks. This is insanely useful for study
Very Amazing article.!!! Thanks, but I have a question, Qwen3-235B-A22B employs a fully integrated Mixture-of-Experts (MoE) architecture, rather than alternating between dense and MoE layers every two layers.
Thanks for the note, you are absolutely right. This must have been a copy-paste error from the Llama 4 architecture. Just fixed it.
Very Helpful. Thanks
this is fantastic
nice
Are you interested in explaining the principles of RoPE (which has almost become a de facto standard in every architecture mentioned in your articles)? The mathematical derivation process is really giving me a headache.😭
One day! I have a long list of things I want to do but only so little time 😅
Thank you for your reply！
are you sure that the qwen moe has a dense MLP in every other layer?
Good callout. That was from sth else. Afaik they don’t have dense MLP layers
Great insight! I think the biggest leap forward will come from architectural changes that eliminate the auto-regressive nature of current LLMs. Curious to see who comes up with clever solutions to break that paradigm.