Ahead of AI

Home
Support
LLMs From Scratch Book
Archive
About
Instruction Pretraining LLMs
The Latest Research in Instruction Finetuning
 • 
Sebastian Raschka, PhD
8

June 2024

Developing an LLM: Building, Training, Finetuning
A Deep Dive into the Lifecycle of LLM Development
 • 
Sebastian Raschka, PhD
18
LLM Research Insights: Instruction Masking and New LoRA Finetuning Experiments
Discussing the Latest Model Releases and AI Research in May 2024
 • 
Sebastian Raschka, PhD
9

May 2024

How Good Are the Latest Open LLMs? And Is DPO Better Than PPO?
Discussing the Latest Model Releases and AI Research in April 2024
 • 
Sebastian Raschka, PhD
4

April 2024

March 2024

Tips for LLM Pretraining and Evaluating Reward Models
Discussing AI Research Papers in March 2024
 • 
Sebastian Raschka, PhD
16
Research Papers in February 2024: A LoRA Successor, Small Finetuned LLMs Vs Generalist LLMs, and Transparent LLM Research
Once again, this has been an exciting month in AI research. This month, I'm covering two new openly available LLMs, insights into small finetuned LLMs…
 • 
Sebastian Raschka, PhD
13

February 2024

Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch
Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or vision transformer) to better suit a…
 • 
Sebastian Raschka, PhD
37
Model Merging, Mixtures of Experts, and Towards Smaller LLMs
Model Merging, Mixtures of Experts, and Towards Smaller LLMs
 • 
Sebastian Raschka, PhD
15

January 2024

Understanding and Coding Self-Attention, Multi-Head Attention, Cross-Attention, and Causal-Attention in LLMs
This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama…
27

December 2023

Ten Noteworthy AI Research Papers of 2023
This year has felt distinctly different. I've been working in, on, and with machine learning and AI for over a decade, yet I can't recall a time when…
 • 
Sebastian Raschka, PhD
32
Tackling Hallucinations, Boosting Reasoning Abilities, and New Insights into the Transformer Architecture
This month, I want to focus on three papers that address three distinct problem categories of Large Language Models (LLMs): Reducing hallucinations.
 • 
Sebastian Raschka, PhD
13
© 2024 Sebastian Raschka
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great culture