Do you see LongNet ultimately being rolled out everywhere? In other words, assuming this works, will ChatGPT, Bard, etc ultimately have a billion token capacity? Will they be able to do things like read entire books and have discussions about their content with you, etc?
LongNet is interesting. It's basically a more principled version of BigBird. But at the end of the day, it's only an approximation of the original self-attention mechanism. We will see if that will be something that people like to adopt in practice. However, given that the recent GPT-4 is a distilled and cheaper version of the original GPT-4 model that was initially deployed, I can see companies jumping on these self-attention alternatives.
Nice update!
Do you see LongNet ultimately being rolled out everywhere? In other words, assuming this works, will ChatGPT, Bard, etc ultimately have a billion token capacity? Will they be able to do things like read entire books and have discussions about their content with you, etc?
LongNet is interesting. It's basically a more principled version of BigBird. But at the end of the day, it's only an approximation of the original self-attention mechanism. We will see if that will be something that people like to adopt in practice. However, given that the recent GPT-4 is a distilled and cheaper version of the original GPT-4 model that was initially deployed, I can see companies jumping on these self-attention alternatives.