A few months ago, I shared the article, Understanding Large Language Models: A Cross-Section of the Most Relevant Literature To Get Up to Speed, and the positive feedback was very motivating! So, I also added a few papers here and there to keep the list fresh and relevant.
Very astute. I’m a Septuagenarian who interest involved from high school mechanical drafting into CAD and CNC years ago. Progressively from logic and truth tree training with projects on canning retort operations using VAT. Eventually using R language I been aggressively reading all I can about ML and AI. A fair hobby to stimulate my mind in maturing. Anyhow, your research is inspiring. Keep up the good work. Thank you.
About LayerNorm Variants in the Original Transformer Paper, and Some Other Interesting Historical Tidbits About LLMs
Very astute. I’m a Septuagenarian who interest involved from high school mechanical drafting into CAD and CNC years ago. Progressively from logic and truth tree training with projects on canning retort operations using VAT. Eventually using R language I been aggressively reading all I can about ML and AI. A fair hobby to stimulate my mind in maturing. Anyhow, your research is inspiring. Keep up the good work. Thank you.
J. Caron
Canada
Hi, thanks for the helpful references!
Regarding the [official implementation](https://github.com/tensorflow/tensor2tensor/commit/f5c9b17e617ea9179b7d84d36b1e8162cb369f25) I can see that they have set the default to `layer_postprocess_sequence="dan"`, which according to [this comment](https://github.com/tensorflow/tensor2tensor/blob/bafdc1b67730430d38d6ab802cbd51f9d053ba2e/tensor2tensor/layers/common_layers.py#L881) should be interpreted as dropout -> add -> normaliaze, matching the description of the paper.
Am I missing something ?