Discussion about this post

User's avatar
Haseeb Raja's avatar

Testing my knowledge here haha.

> When does it make more sense to use in-context learning rather than fine-tuning, and vice versa?

I think fine-tuning makes sense when we need domain adaptation and have enough data + resources for it. However, 1) if we want to evaluate/test the model's few-shot performance 2) do not have sufficient data/resources 3) or domain adaptation requirements, then in-context learning can suffice.

Definitely makes sense to always test in-context performance before jumping into fine-tuning.

---

> In prefix tuning, adapters, and LoRA, how can we ensure that the model preserves (and does not forget) the original knowledge?

I can think of two approaches (on top of my head).

1) I think we can use something like the KL Divergence Shift Penalty (similar to how we handle the Reward Hacking in PPO-RLHF) problem.

2) We can maybe use a small amount (1-5%) of the original dataset along with the fine-tuning dataset.

Expand full comment
Ronan McGovern's avatar

Could you expand, with any suggested reading, on this quote:

“Generally, in-context learning does not perform as well as finetuning for certain tasks or specific datasets since it relies on the pretrained model’s ability to generalize from its training data without further adapting its parameters for the particular task at hand.”

Separately, ReFT is probably an improvement on prefix based approaches. It involves adding trainable low rank representations to hidden layer features.

Expand full comment
11 more comments...

No posts