Archive - Ahead of AI

Building LLMs from the Ground Up: A 3-hour Coding Workshop

If your weekend plans include catching up on AI developments and understanding Large Language Models (LLMs), I've prepared a 1-hour presentation on the…

Aug 31 •

Sebastian Raschka, PhD

New LLM Pre-training and Post-training Paradigms

A Look at How Moderns LLMs Are Trained

Aug 17 •

Sebastian Raschka, PhD

July 2024

Instruction Pretraining LLMs

The Latest Research in Instruction Finetuning

Jul 20 •

Sebastian Raschka, PhD

June 2024

Developing an LLM: Building, Training, Finetuning

A Deep Dive into the Lifecycle of LLM Development

Jun 8 •

Sebastian Raschka, PhD

LLM Research Insights: Instruction Masking and New LoRA Finetuning Experiments

Discussing the Latest Model Releases and AI Research in May 2024

Jun 2 •

Sebastian Raschka, PhD

May 2024

How Good Are the Latest Open LLMs? And Is DPO Better Than PPO?

Discussing the Latest Model Releases and AI Research in April 2024

May 12 •

Sebastian Raschka, PhD

April 2024

Using and Finetuning Pretrained Transformers

What are the different ways to use and finetune pretrained large language models (LLMs)? The most common ways to use and finetune pretrained LLMs…

Apr 20 •

Sebastian Raschka, PhD

March 2024

Tips for LLM Pretraining and Evaluating Reward Models

Discussing AI Research Papers in March 2024

Mar 31 •

Sebastian Raschka, PhD

Research Papers in February 2024: A LoRA Successor, Small Finetuned LLMs Vs Generalist LLMs, and Transparent LLM Research

Once again, this has been an exciting month in AI research. This month, I'm covering two new openly available LLMs, insights into small finetuned LLMs…

Mar 3 •

Sebastian Raschka, PhD

February 2024

Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch

Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or vision transformer) to better suit a…

Feb 18 •

Sebastian Raschka, PhD

Model Merging, Mixtures of Experts, and Towards Smaller LLMs

Model Merging, Mixtures of Experts, and Towards Smaller LLMs

Feb 3 •

Sebastian Raschka, PhD

January 2024

Understanding and Coding Self-Attention, Multi-Head Attention, Cross-Attention, and Causal-Attention in LLMs

This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama.

Jan 14

#nojs-banner { position: fixed; bottom: 0; left: 0; padding: 16px 16px 16px 32px; width: 100%; box-sizing: border-box; background: red; color: white; font-family: -apple-system, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 13px; line-height: 13px; } #nojs-banner a { color: inherit; text-decoration: underline; } This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts