Transformer Concept

Self-Attention

Pick a query word — see how much it attends to every other word, from Q·K.

query token
attention weights
tip: click any token to make it the query
attention.py