Ahead of AI

Home
Support & Books
Website
Archive

March 2024

Tips for LLM Pretraining and Evaluating Reward Models
Discussing AI Research Papers in March 2024
 • 
Sebastian Raschka, PhD
16
Research Papers in February 2024: A LoRA Successor, Small Finetuned LLMs Vs Generalist LLMs, and Transparent LLM Research
Once again, this has been an exciting month in AI research. This month, I'm covering two new openly available LLMs, insights into small finetuned LLMs…
 • 
Sebastian Raschka, PhD
13

February 2024

Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch
Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or vision transformer) to better suit a…
 • 
Sebastian Raschka, PhD
37
Research Papers in Jan 2024: Model Merging, Mixtures of Experts, and Towards Smaller LLMs
Model Merging, Mixtures of Experts, and Towards Smaller LLMs
 • 
Sebastian Raschka, PhD
15

January 2024

Understanding and Coding Self-Attention, Multi-Head Attention, Cross-Attention, and Causal-Attention in LLMs
This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama…
27

December 2023

Ten Noteworthy AI Research Papers of 2023
This year has felt distinctly different. I've been working in, on, and with machine learning and AI for over a decade, yet I can't recall a time when…
 • 
Sebastian Raschka, PhD
32
Research Papers in Nov 2023: Tackling Hallucinations, Boosting Reasoning Abilities, and New Insights into the Transformer Architecture
This month, I want to focus on three papers that address three distinct problem categories of Large Language Models (LLMs): Reducing hallucinations…
 • 
Sebastian Raschka, PhD
13

November 2023

Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation)
Things I Learned From Hundreds of Experiments
 • 
Sebastian Raschka, PhD
42
Research Papers in Oct 2023: A Potential Successor to RLHF for Efficient LLM Alignment and the Resurgence of CNNs
From Vision Transformers to innovative large language model finetuning techniques, the AI community has been very active with lots of interesting…
 • 
Sebastian Raschka, PhD

October 2023

AI and Open Source in 2023
The Highs and Lows: A Year in Review
 • 
Sebastian Raschka, PhD
11
LLM Business and Busyness: Recent Company Investments and AI Adoption, New Small Openly Available LLMs, and LoRA Research
Discussing Recent Company Investments and AI Adoption, New Small Openly Available LLMs, and LoRA Research
 • 
Sebastian Raschka, PhD
17
© 2024 Sebastian Raschka
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great culture