statementSaid on 2026-10-19
In attention the memories do not sit in the weights; every token in the context is a memory, and the store grows as the context lengthens.
A statement by post author, said on 2026-10-19. Statements are as reported by the cited publishers — not verified by DecisionGateway.
ב-attention הזכרונות לא יושבים במשקולות. כל טוקן בקונטקסט הוא זיכרון, והמחסן גדל ככל שהקונטקסט מתארך.
Discussion
Related claims
The book proves that unifying weights inside the attention mechanism to save memory erases the model's built-in ability to understand direction, order and causality.the post author · statementIn a linear layer, each new memory is simply added to the weights; retrieval returns the correct value plus a bit of every other memory depending on key overlap.post author · statementEvery LLM is essentially a huge memory store with an extremely sophisticated retrieval mechanism.post author · statementThe exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.post author · statementResearchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.the post author · statementIn early versions of Attention Is All You Need the authors explicitly wrote that the FFN is "a kind of attention"; in later versions the paragraph simply disappeared.post author · statement