statementSaid on 2026-10-19
In early versions of Attention Is All You Need the authors explicitly wrote that the FFN is "a kind of attention"; in later versions the paragraph simply disappeared.
A statement by post author, said on 2026-10-19. Statements are as reported by the cited publishers — not verified by DecisionGateway.
בגרסאות מוקדמות של Attention Is All You Need הכותבים כתבו במפורש שה-FFN הוא "סוג של attention". בגרסאות המאוחרות הפסקה פשוט נעלמה.
Discussion
Related claims
Researchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.the post author · statementThe book proves that unifying weights inside the attention mechanism to save memory erases the model's built-in ability to understand direction, order and causality.the post author · statementIn attention the memories do not sit in the weights; every token in the context is a memory, and the store grows as the context lengthens.post author · statementIn every attention head in a model like BERT there are over 8,000 parameters that mathematically cancel each other out behind the scenes, a phenomenon called 'calibration freedom'.the post author · statementThe exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.post author · statementIn transformer-based language models and architectures like Mamba, every token prediction requires activating billions of parameters in parallel through the attention mechanism.author · statement