statement
The book proves that unifying weights inside the attention mechanism to save memory erases the model's built-in ability to understand direction, order and causality.
A statement by the post author. Statements are as reported by the cited publishers — not verified by DecisionGateway.
הספר מוכיח שברגע שעושים את הקיצור הזה – מוחקים למודל את היכולת המובנית להבין כיוון, סדר וסיבתיות.
Discussion
Related claims
Researchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.the post author · statementIn attention the memories do not sit in the weights; every token in the context is a memory, and the store grows as the context lengthens.post author · statementA famous 2021 paper taught that without the heaviest components (MLP layers, about 65% of compute) the model's information would 'collapse' and become uniform, but the book proves mathematically this is simply not true: Residuals alone prevent the collapse, and the MLP is there to add computational flexibility (non-linearity), not to save the model from collapse.the post author · statementIn the deep layers, when units become most focused and smart, the model creates a bottleneck that erases most of their information.the post author · statementIn early versions of Attention Is All You Need the authors explicitly wrote that the FFN is "a kind of attention"; in later versions the paragraph simply disappeared.post author · statementIn every attention head in a model like BERT there are over 8,000 parameters that mathematically cancel each other out behind the scenes, a phenomenon called 'calibration freedom'.the post author · statement