---
title: "With squared ReLU the capacity jumps from 64 to about 2,700, and it is not a coincidence that such… — post author — DecisionGateway"
description: "statement by post author, recorded 2026-10-19 — verbatim quote and source citation."
canonical_url: "https://dgw.tmarket.io/claims/01a1e1f0-e66d-7612-bf6e-5e95294efc90"
meta:
  "og:description": "statement by post author, recorded 2026-10-19 — verbatim quote and source citation."
  "og:image": "https://dgw.tmarket.io/og.jpg"
  "og:image:alt": "White network of connected nodes crossed by a timeline on black"
  "og:image:height": "1024"
  "og:image:width": "1024"
  "og:title": "With squared ReLU the capacity jumps from 64 to about 2,700, and it is not a coincidence that such… — post author — DecisionGateway"
  "twitter:image": "https://dgw.tmarket.io/og.jpg"
---

> Index of every page on this site: [llms.txt](https://dgw.tmarket.io/llms.txt)

statementSaid on 2026-10-19

# **With squared ReLU the capacity jumps from 64 to about 2,700, and it is not a coincidence that such variants won in practice.**

A statement by post author, said on 2026-10-19. Statements are as reported by the cited publishers — not verified by DecisionGateway.

> *עם ReLU בריבוע הקיבולת קופצת מ-64 ל-2,700 בערך, וזה לא מקרה שווריאנטים כאלה ניצחו בפועל.*

mathyaiwithmikePublished 2026-10-19 [ Read source](https://t.me/mathyaiwithmike/1289)

## **Discussion**

## **Related claims**

[**The exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.**post author · statement](https://dgw.tmarket.io/claims/01a1e1f0-e689-7ceb-bb75-65cda71d001f) [**On algorithmic tasks (CLRS-Text), ten such layers raised an 86M-parameter model from 55% to 84%, after regular training got stuck at 55% and even training twice as long did not move it.**post author · statement](https://dgw.tmarket.io/claims/01a1e1f6-8f8e-7a10-a02f-2baf8c6d789e) [**The researcher claims superiority at large scale, but the largest model he trained here is 1.2 billion parameters on 31.5 billion tokens.**החוקר · statement](https://dgw.tmarket.io/claims/01a1e1ee-4774-7db0-a858-8bb5743cde1c) [**In the Effective Rank metric, regular MTP training collapsed the output matrix from 496 to 131, while Soft-MTP kept it high and stable at 437.**post author · statement](https://dgw.tmarket.io/claims/01a1e1ee-cad6-7a16-a096-11c8854a7260) [**Each implanted layer is multiplied by a learned gate initialized to zero; at the moment of implantation the network does not change at all, and the gradient descent decides whether to open the gate.**post author · statement](https://dgw.tmarket.io/claims/01a1e1f6-8f6a-7096-8c31-22b8c40596c3) [**The residual component is 4 times noisier than the first term, and is responsible for 80% of the update strength flowing back to the shared trunk; as the model grows it only intensifies and pulls the weights in a direction that interferes with next-token prediction.**post author · statement](https://dgw.tmarket.io/claims/01a1e1ee-c9e3-7b12-b55d-3721312de156)

[All claims](https://dgw.tmarket.io/claims)

```json
{"@context":"https://schema.org","@graph":[{"@id":"https://dgw.tmarket.io/#website","@type":"WebSite","description":"Decisions, events and falsifiable forecasts from 4,500+ Telegram channels — every claim cited. Read free.","name":"DecisionGateway","url":"https://dgw.tmarket.io/","publisher":{"@id":"https://dgw.tmarket.io/#identity"}},{"@id":"https://dgw.tmarket.io/claims/01a1e1f0-e66d-7612-bf6e-5e95294efc90#webpage","@type":"WebPage","description":"statement by post author, recorded 2026-10-19 — verbatim quote and source citation.","name":"With squared ReLU the capacity jumps from 64 to about 2,700, and it is not a coincidence that such… — post author — DecisionGateway","url":"https://dgw.tmarket.io/claims/01a1e1f0-e66d-7612-bf6e-5e95294efc90","breadcrumb":{"@id":"https://dgw.tmarket.io/claims/01a1e1f0-e66d-7612-bf6e-5e95294efc90#breadcrumb"},"isPartOf":{"@id":"https://dgw.tmarket.io/#website"},"potentialAction":[{"@type":"ReadAction","target":["https://dgw.tmarket.io/claims/01a1e1f0-e66d-7612-bf6e-5e95294efc90"]}]},{"@id":"https://dgw.tmarket.io/#identity","@type":"Organization","name":"DecisionGateway","url":"https://dgw.tmarket.io"},{"@id":"https://dgw.tmarket.io/claims/01a1e1f0-e66d-7612-bf6e-5e95294efc90#breadcrumb","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","name":"Home","item":"https://dgw.tmarket.io","position":1},{"@type":"ListItem","name":"Claims","item":"https://dgw.tmarket.io/claims","position":2},{"@type":"ListItem","name":"With squared ReLU the capacity jumps from 64 to about…","position":3}]},{"@id":"https://dgw.tmarket.io/claims/01a1e1f0-e66d-7612-bf6e-5e95294efc90#/schema/quotation/1","@type":"Quotation","dateCreated":"2026-10-19","isBasedOn":"https://t.me/mathyaiwithmike/1289","text":"עם ReLU בריבוע הקיבולת קופצת מ-64 ל-2,700 בערך, וזה לא מקרה שווריאנטים כאלה ניצחו בפועל.","url":"/claims/01a1e1f0-e66d-7612-bf6e-5e95294efc90"}]}
```

## Sitemap

See the full [sitemap](https://dgw.tmarket.io/sitemap.md) for all pages.
