statementSaid on 2026-11-08
To know where to implant, the agent computes the loss gradient for each token according to its representation in each layer.
A statement by post author, said on 2026-11-08. Statements are as reported by the cited publishers — not verified by DecisionGateway.
כדי לדעת איפה להשתיל, הסוכן מחשב את הגרדיאנט של הלוס לכל טוקן לפי הייצוג שלו בכל שכבה.
Discussion
Related claims
Each implanted layer is multiplied by a learned gate initialized to zero; at the moment of implantation the network does not change at all, and the gradient descent decides whether to open the gate.post author · statementThe agent implants "code layers" into the network — small programs that read tokens or internal representations, and whose output re-enters the computation alongside attention and MLP.post author · statementWhen the same gradient keeps appearing across several checkpoints, the writers interpret it as "demand" that SGD fails to supply.the writers · statementThe same layer works better when implanted in the middle of training than when it is there from the first step.post author · statementWhen decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.post author · statementSoft-MTP obtains the same gradient for free by a simple subtraction between gradients already computed, instead of computing KL from scratch (which requires a lot of memory).post author · statement