statementSaid on 2026-11-05

The residual component is 4 times noisier than the first term, and is responsible for 80% of the update strength flowing back to the shared trunk; as the model grows it only intensifies and pulls the weights in a direction that interferes with next-token prediction.

A statement by post author, said on 2026-11-05. Statements are as reported by the cited publishers — not verified by DecisionGateway.

מגלים שרכיב השארית רועש פי 4 מהאיבר הראשון.
mathyaiwithmikePublished 2026-11-05 Read source

Discussion

0/2000

Related claims

A famous 2021 paper taught that without the heaviest components (MLP layers, about 65% of compute) the model's information would 'collapse' and become uniform, but the book proves mathematically this is simply not true: Residuals alone prevent the collapse, and the MLP is there to add computational flexibility (non-linearity), not to save the model from collapse.the post author · statementDuring the backward pass, the MTP residual noise and the NTP residual noise meet with opposite signs and cancel each other out, leaving only the clean self-distillation signal; the result is 0 added memory, 0 impact on token throughput, and full compatibility with Fused kernels.post author · statementResearchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.the post author · statementIn the deep layers, when units become most focused and smart, the model creates a bottleneck that erases most of their information.the post author · statementThe book proves that unifying weights inside the attention mechanism to save memory erases the model's built-in ability to understand direction, order and causality.the post author · statementIn a regular network, if the model errs, the backprop algorithm stops everything, goes back from end to start and updates all the weights; in the brain there is no such central manager, and instead each synapse learns on its own from its local environment using STDP.author · statement