statement
In every attention head in a model like BERT there are over 8,000 parameters that mathematically cancel each other out behind the scenes, a phenomenon called 'calibration freedom'.
A statement by the post author. Statements are as reported by the cited publishers — not verified by DecisionGateway.
בכל ראש attention במודל כמו BERT יש מעל 8,000 פרמטרים שפשוט מבטלים אחד את השני מתמטית מאחורי הקלעים. זה נקרא "חופש כיול"
Discussion
Related claims
Researchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.the post author · statementIn transformer-based language models and architectures like Mamba, every token prediction requires activating billions of parameters in parallel through the attention mechanism.author · statementThe exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.post author · statementThe book proves that unifying weights inside the attention mechanism to save memory erases the model's built-in ability to understand direction, order and causality.the post author · statementIn attention the memories do not sit in the weights; every token in the context is a memory, and the store grows as the context lengthens.post author · statementThe researcher claims superiority at large scale, but the largest model he trained here is 1.2 billion parameters on 31.5 billion tokens.החוקר · statement