Discussion about this post

User's avatar
Sebastian Raschka, PhD's avatar

Small correction: There was originally a drop from 0.783 to 0.028 for "All-layer QLORA" in the causative benchmark, which seemed like a significant drop that went unmentioned in my text.

This was because I was looking at the correct numbers in my notes but had an incorrect number in the table figure I prepared for the post. In reality, "All-Layer QLoRA" actually improves the benchmark: from 0.783 to 0.788. I have updated the table.

Expand full comment
Vaibhav's avatar

The article was very well written

Loved it.

Are the weights decomposed using PCA?

Expand full comment
46 more comments...

No posts