In this article, I compiled and annotated 24 AI research highlights form June to July 2023. A lot of exciting developments are currently happening, once again, in the fields of natural language processing and computer vision!
Do you see LongNet ultimately being rolled out everywhere? In other words, assuming this works, will ChatGPT, Bard, etc ultimately have a billion token capacity? Will they be able to do things like read entire books and have discussions about their content with you, etc?
LongNet is interesting. It's basically a more principled version of BigBird. But at the end of the day, it's only an approximation of the original self-attention mechanism. We will see if that will be something that people like to adopt in practice. However, given that the recent GPT-4 is a distilled and cheaper version of the original GPT-4 model that was initially deployed, I can see companies jumping on these self-attention alternatives.
Nice update!
Do you see LongNet ultimately being rolled out everywhere? In other words, assuming this works, will ChatGPT, Bard, etc ultimately have a billion token capacity? Will they be able to do things like read entire books and have discussions about their content with you, etc?
LongNet is interesting. It's basically a more principled version of BigBird. But at the end of the day, it's only an approximation of the original self-attention mechanism. We will see if that will be something that people like to adopt in practice. However, given that the recent GPT-4 is a distilled and cheaper version of the original GPT-4 model that was initially deployed, I can see companies jumping on these self-attention alternatives.