statementSaid on 2026-10-19
The exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.
A statement by post author, said on 2026-10-19. Statements are as reported by the cited publishers — not verified by DecisionGateway.
האקספוננט עושה בדיוק את זה, ואחרי נרמול קיבלתם סופטמקס, כלומר attention
Discussion
Related claims
In attention the memories do not sit in the weights; every token in the context is a memory, and the store grows as the context lengthens.post author · statementIn every attention head in a model like BERT there are over 8,000 parameters that mathematically cancel each other out behind the scenes, a phenomenon called 'calibration freedom'.the post author · statementThe book proves that unifying weights inside the attention mechanism to save memory erases the model's built-in ability to understand direction, order and causality.the post author · statementResearchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.the post author · statementWith squared ReLU the capacity jumps from 64 to about 2,700, and it is not a coincidence that such variants won in practice.post author · statementLayerNorm, thought to be passive, turns out to precisely preserve the richness of information.the post author · statement