statementSaid on 2026-10-19
With squared ReLU the capacity jumps from 64 to about 2,700, and it is not a coincidence that such variants won in practice.
A statement by post author, said on 2026-10-19. Statements are as reported by the cited publishers — not verified by DecisionGateway.
עם ReLU בריבוע הקיבולת קופצת מ-64 ל-2,700 בערך, וזה לא מקרה שווריאנטים כאלה ניצחו בפועל.
Discussion
Related claims
The exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.post author · statementOn algorithmic tasks (CLRS-Text), ten such layers raised an 86M-parameter model from 55% to 84%, after regular training got stuck at 55% and even training twice as long did not move it.post author · statementThe researcher claims superiority at large scale, but the largest model he trained here is 1.2 billion parameters on 31.5 billion tokens.החוקר · statementIn the Effective Rank metric, regular MTP training collapsed the output matrix from 496 to 131, while Soft-MTP kept it high and stable at 437.post author · statementEach implanted layer is multiplied by a learned gate initialized to zero; at the moment of implantation the network does not change at all, and the gradient descent decides whether to open the gate.post author · statementThe residual component is 4 times noisier than the first term, and is responsible for 80% of the update strength flowing back to the shared trunk; as the model grows it only intensifies and pulls the weights in a direction that interferes with next-token prediction.post author · statement