statementSaid on 2026-11-08
The same layer works better when implanted in the middle of training than when it is there from the first step.
A statement by post author, said on 2026-11-08. Statements are as reported by the cited publishers — not verified by DecisionGateway.
אותה שכבה בדיוק עובדת טוב יותר כשמשתילים אותה באמצע האימון מאשר כשהיא שם מהצעד הראשון.
Discussion
Related claims
Each implanted layer is multiplied by a learned gate initialized to zero; at the moment of implantation the network does not change at all, and the gradient descent decides whether to open the gate.post author · statementTo know where to implant, the agent computes the loss gradient for each token according to its representation in each layer.post author · statement'The end of diffusion' is hype; the better question is whether it is worth paying for more expensive training to get built-in verification and flexible compute at inference time — a real and interesting trade-off.מייק · statementIn the deep layers, when units become most focused and smart, the model creates a bottleneck that erases most of their information.the post author · statementOn algorithmic tasks (CLRS-Text), ten such layers raised an 86M-parameter model from 55% to 84%, after regular training got stuck at 55% and even training twice as long did not move it.post author · statementThe residual component is 4 times noisier than the first term, and is responsible for 80% of the update strength flowing back to the shared trunk; as the model grows it only intensifies and pulls the weights in a direction that interferes with next-token prediction.post author · statement