statementSaid on 2026-10-19
Adding nonlinearity that silences somewhat-similar keys and keeps only the truly similar ones yields an MLP: one layer holds the keys, the second the values, and the activation decides who gets to speak.
A statement by post author, said on 2026-10-19. Statements are as reported by the cited publishers — not verified by DecisionGateway.
אז מוסיפים אי-לינאריות שמשתיקה את המפתחות ״דומים במידת מה" ומשאירה רק את הדומים באמת.
Discussion
Related claims
In a linear layer, each new memory is simply added to the weights; retrieval returns the correct value plus a bit of every other memory depending on key overlap.post author · statementA famous 2021 paper taught that without the heaviest components (MLP layers, about 65% of compute) the model's information would 'collapse' and become uniform, but the book proves mathematically this is simply not true: Residuals alone prevent the collapse, and the MLP is there to add computational flexibility (non-linearity), not to save the model from collapse.the post author · statementEach implanted layer is multiplied by a learned gate initialized to zero; at the moment of implantation the network does not change at all, and the gradient descent decides whether to open the gate.post author · statementThe agent implants "code layers" into the network — small programs that read tokens or internal representations, and whose output re-enters the computation alongside attention and MLP.post author · statementIn the deep layers, when units become most focused and smart, the model creates a bottleneck that erases most of their information.the post author · statementLayerNorm, thought to be passive, turns out to precisely preserve the richness of information.the post author · statement