2 Comments
Jul 15, 2023Liked by Sebastian Raschka, PhD

Nice update!

Do you see LongNet ultimately being rolled out everywhere? In other words, assuming this works, will ChatGPT, Bard, etc ultimately have a billion token capacity? Will they be able to do things like read entire books and have discussions about their content with you, etc?

Expand full comment
author

LongNet is interesting. It's basically a more principled version of BigBird. But at the end of the day, it's only an approximation of the original self-attention mechanism. We will see if that will be something that people like to adopt in practice. However, given that the recent GPT-4 is a distilled and cheaper version of the original GPT-4 model that was initially deployed, I can see companies jumping on these self-attention alternatives.

Expand full comment