statementSaid on 2026-10-19
In a linear layer, each new memory is simply added to the weights; retrieval returns the correct value plus a bit of every other memory depending on key overlap.
A statement by post author, said on 2026-10-19. Statements are as reported by the cited publishers — not verified by DecisionGateway.
כל זיכרון חדש פשוט מתווסף למשקולות, וזהו. בשליפה מקבלים את הערך הנכון, ועוד קצת מכל זיכרון אחר, בהתאם למידת החפיפה בין המפתחות.
Discussion
Related claims
Adding nonlinearity that silences somewhat-similar keys and keeps only the truly similar ones yields an MLP: one layer holds the keys, the second the values, and the activation decides who gets to speak.post author · statementIn attention the memories do not sit in the weights; every token in the context is a memory, and the store grows as the context lengthens.post author · statementEvery LLM is essentially a huge memory store with an extremely sophisticated retrieval mechanism.post author · statementEach implanted layer is multiplied by a learned gate initialized to zero; at the moment of implantation the network does not change at all, and the gradient descent decides whether to open the gate.post author · statementThe book proves that unifying weights inside the attention mechanism to save memory erases the model's built-in ability to understand direction, order and causality.the post author · statementTo know where to implant, the agent computes the loss gradient for each token according to its representation in each layer.post author · statement