statement
LayerNorm, thought to be passive, turns out to precisely preserve the richness of information.
A statement by the post author. Statements are as reported by the cited publishers — not verified by DecisionGateway.
גם פעולת הנרמול (LayerNorm), שחשבו שהיא פסיבית, מתגלה ככזו ששומרת במדויק על עושר המידע.
Discussion
Related claims
In the deep layers, when units become most focused and smart, the model creates a bottleneck that erases most of their information.the post author · statementThe exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.post author · statementA famous 2021 paper taught that without the heaviest components (MLP layers, about 65% of compute) the model's information would 'collapse' and become uniform, but the book proves mathematically this is simply not true: Residuals alone prevent the collapse, and the MLP is there to add computational flexibility (non-linearity), not to save the model from collapse.the post author · statementAdding nonlinearity that silences somewhat-similar keys and keeps only the truly similar ones yields an MLP: one layer holds the keys, the second the values, and the activation decides who gets to speak.post author · statementThe same layer works better when implanted in the middle of training than when it is there from the first step.post author · statementEach implanted layer is multiplied by a learned gate initialized to zero; at the moment of implantation the network does not change at all, and the gradient descent decides whether to open the gate.post author · statement