commitment
If you train models with MTP architecture, stop injecting noisy CE gradients and instead add the Soft-MTP line of code.
A commitment by מייק. Statements are as reported by the cited publishers — not verified by DecisionGateway.
אם אתם מאמנים מודלים עם ארכיטקטורת MTP, תפסיקו להזריק גראדיאנטים רועשים של CE אלא הוסיפו את שורת הקוד של Soft-MTP
Discussion
Related claims
Soft-MTP obtains the same gradient for free by a simple subtraction between gradients already computed, instead of computing KL from scratch (which requires a lot of memory).post author · statementWhen decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.post author · statementLarge AI labs train models with MTP, but 80% of the gradient they backpropagate to the model is static noise that interferes with learning.post author · allegationAdding the Soft-MTP line of code isolates the Self-Distillation signal and yields a smarter, faster model in speculative decoding without paying an additional penny in compute or memory.מייק · statementMTP not only improves performance in math and code, but also provides built-in draft heads for inference acceleration via Speculative Decoding.post author · statementIn the Effective Rank metric, regular MTP training collapsed the output matrix from 496 to 131, while Soft-MTP kept it high and stable at 437.post author · statement
More claims by מייק
In event-driven neuromorphic architectures, parts of the network that do not need to work simply stay at rest.statementIn event-driven neuromorphic architectures, memory is adjacent to the processor.statementIn event-driven neuromorphic architectures, time is a computational feature.statementThe future of AI lies not only in pushing more billions of parameters into hot GPUs, but in a smart combination of classic transformers and LLMs with event-driven neuromorphic architectures.predictionEBMs promised a revolution 20 years ago, and the problem was never elegance — it was training at scale.statement
All claims by מייק