This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama. Self-attention and related mechanisms are core components of LLMs, making them a useful topic to understand when working with these models.
I believe the upper triable needs to start with the second diagonal to achieve the same mask as yours, i.e., mask = torch.triu(torch.ones((block_size, block_size)), diagonal=1)
Oh I see what you mean. Thanks. `keys = embedded_sentence @ W_query` should be correct -- I rearranged the weight matrix init a bit and I think it didn't take the update previously. I.e., the weight matrices were supposed to be initialized as
torch.nn.Parameter(torch.rand(d, d_q))
rather than
torch.nn.Parameter(torch.rand(d_q, d))
to avoid the transpose in several places, which makes everything a bit more straightforward. I refreshed it and it seems to be fine now.
"The masked_fill method then replaces all the elements on and above the diagonal via positive mask values (1s) with -torch.inf, with the results being shown below."
Since you added the argument, it should be *above*, not *on and above*
I was wondering how backpropagation works with an encoder-decoder transformer? For example, is it necessary to define two optimisers, one for the encoder and one for the decoder? What are the target values of the encoder network when the parameters need to be trained? To summarise: Would backpropagation for training the transformer also have to be performed for the encoder network or only for the decoder network, and if so, how?
Good question. In contrast to a GAN, you actually only need one optimizer. The reason is that the encoder and decoder are both part of the same architecture. I.e., the encoder output feeds into the decoder. Instead of thinking of it as two different independent neural networks, it perhaps helps to think of them just as layers.
In PyTorch, the overall structure could look like as follows:
I've implemented the original Transformer architecture a couple of times already and every time I learn something new.
Just a quick note:
I think mask = torch.triu(torch.ones(block_size, block_size)) should be mask = torch.triu(torch.ones(block_size, block_size), diagonal=1), otherwise the values on the diagonal get masked as well.
Arg, sorry, I can confirm that I'm getting the same there too. If you open the Jupyter notebook in VSCode though it highlights these invisible invalid characters and you can do a "Find & Replace All" on them. (I wish Substack let me post a screenshot to demonstrate)
I believe the upper triable needs to start with the second diagonal to achieve the same mask as yours, i.e., mask = torch.triu(torch.ones((block_size, block_size)), diagonal=1)
As always, thanks a lot for a great tutorial!
Good catch, the output should stay the same -- the diagonal=1 must have gone missing when copying things over from my local code. Fixed!
keys = embedded_sentence @ W_query
values = embedded_sentence @ W_value
print("keys.shape:", keys.shape)
print("values.shape:", values.shape)
did you mean:
keys = embedded_sentence @ W_query.T
values = embedded_sentence @ W_value.T
Thanks for the note. Should have been `keys = embedded_sentence @ W_key` instead of `keys = embedded_sentence @ W_query`. Just updated it.
keys = embedded_sentence @ W_keys should be @ W_key.T ? Otherwise, dimensions don't match
Oh I see what you mean. Thanks. `keys = embedded_sentence @ W_query` should be correct -- I rearranged the weight matrix init a bit and I think it didn't take the update previously. I.e., the weight matrices were supposed to be initialized as
torch.nn.Parameter(torch.rand(d, d_q))
rather than
torch.nn.Parameter(torch.rand(d_q, d))
to avoid the transpose in several places, which makes everything a bit more straightforward. I refreshed it and it seems to be fine now.
That's the last typo, I was able to 'QA' the rest of the code in a Jupiter notebook.
Thank you for the great article
Very Nice instruction!
Thanks!
Great Article. I was looking to broaden my understanding on self attention and I'm glad I stumbled on this.
Thanks a lot for this article! It cleared many concepts that I have been struggling in for a long time.
"The masked_fill method then replaces all the elements on and above the diagonal via positive mask values (1s) with -torch.inf, with the results being shown below."
Since you added the argument, it should be *above*, not *on and above*
Thank you for the note!
Absolutely right. Good catch. I removed the "on".
I was wondering how backpropagation works with an encoder-decoder transformer? For example, is it necessary to define two optimisers, one for the encoder and one for the decoder? What are the target values of the encoder network when the parameters need to be trained? To summarise: Would backpropagation for training the transformer also have to be performed for the encoder network or only for the decoder network, and if so, how?
Good question. In contrast to a GAN, you actually only need one optimizer. The reason is that the encoder and decoder are both part of the same architecture. I.e., the encoder output feeds into the decoder. Instead of thinking of it as two different independent neural networks, it perhaps helps to think of them just as layers.
In PyTorch, the overall structure could look like as follows:
EDIT: Code formatting doesn't seem to be supported in the comments, so let me add a link: https://gist.github.com/rasbt/4c32fac33a6641b1fb608718e2a51500
Well explained!
Well written article. The concept of Queries, Keys and, Values, is finally clear to me.
In the diagram "Computing the normalized attention weights α", did you mean
α_2 = softmax(omega_2 / sqrt(d_k))
instead of
α_{2,i} = softmax(omega_{2,i} / sqrt(d_k))?
Since omega_{2,i} / sqrt(d_k) is a scalar whereas softmax operates on vectors.
Good call, the "i" shouldn't be there. Just updated it. Thanks!
there is one more minor typo, a duplicated capital i that needs to be removed in the text:
"IIn this article, ..." -> "In this article, ..."
Hi Sebastian,
Thanks for the great article.
I've implemented the original Transformer architecture a couple of times already and every time I learn something new.
Just a quick note:
I think mask = torch.triu(torch.ones(block_size, block_size)) should be mask = torch.triu(torch.ones(block_size, block_size), diagonal=1), otherwise the values on the diagonal get masked as well.
Has been fixed, someone else reported that issue as well, see above in the comments
Does anyone else have problems copying the code? Every time I copy-paste the code into a Jupyter NB, for instance
```
sentence = 'Life is short, eat dessert first'
dc = {s:i for i,s
in enumerate(sorted(sentence.replace(',', '').split()))}
```
it adds an invalid token "\u200b" at every empty line in the copied copy, in this example between the empty line between sentence and dc
Arg, sorry, I can confirm that I'm getting the same there too. If you open the Jupyter notebook in VSCode though it highlights these invisible invalid characters and you can do a "Find & Replace All" on them. (I wish Substack let me post a screenshot to demonstrate)
I see it more like a feature than a bug 😁.
It forces me to actually type the code and therefore having a better understanding of the concepts than I would have by simply copy-pasting it.
Yes, I encountered the same issue.