---
title: "In every attention head in a model like BERT there are over 8,000 parameters that mathematically… — the post author — DecisionGateway"
description: "statement by the post author, recorded recently — verbatim quote and source citation."
canonical_url: "https://dgw.tmarket.io/claims/01a1e1eb-c6f2-7f5d-b2e2-ba78777e88c4"
meta:
  "og:description": "statement by the post author, recorded recently — verbatim quote and source citation."
  "og:image": "https://dgw.tmarket.io/og.jpg"
  "og:image:alt": "White network of connected nodes crossed by a timeline on black"
  "og:image:height": "1024"
  "og:image:width": "1024"
  "og:title": "In every attention head in a model like BERT there are over 8,000 parameters that mathematically… — the post author — DecisionGateway"
  "twitter:image": "https://dgw.tmarket.io/og.jpg"
---

> Index of every page on this site: [llms.txt](https://dgw.tmarket.io/llms.txt)

statement

# **In every attention head in a model like BERT there are over 8,000 parameters that mathematically cancel each other out behind the scenes, a phenomenon called 'calibration freedom'.**

A statement by the post author. Statements are as reported by the cited publishers — not verified by DecisionGateway.

> *בכל ראש attention במודל כמו BERT יש מעל 8,000 פרמטרים שפשוט מבטלים אחד את השני מתמטית מאחורי הקלעים. זה נקרא "חופש כיול"*

mathyaiwithmikePublished 2026-10-21 [ Read source](https://t.me/mathyaiwithmike/1292)

## **Discussion**

## **Related claims**

[**Researchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.**the post author · statement](https://dgw.tmarket.io/claims/01a1e1eb-c746-7456-8138-bfb85325f512) [**In transformer-based language models and architectures like Mamba, every token prediction requires activating billions of parameters in parallel through the attention mechanism.**author · statement](https://dgw.tmarket.io/claims/01a1e1f1-92c2-775b-b01e-62ef447243fc) [**The exponent does exactly that, and after normalization you get softmax, i.e. attention; capacity jumps to about 1.2 million memories, from 64 to a million without adding a single parameter.**post author · statement](https://dgw.tmarket.io/claims/01a1e1f0-e689-7ceb-bb75-65cda71d001f) [**The book proves that unifying weights inside the attention mechanism to save memory erases the model's built-in ability to understand direction, order and causality.**the post author · statement](https://dgw.tmarket.io/claims/01a1e1eb-c7e3-7928-919d-d78b93be7f2e) [**In attention the memories do not sit in the weights; every token in the context is a memory, and the store grows as the context lengthens.**post author · statement](https://dgw.tmarket.io/claims/01a1e1f0-e6cc-744c-9e55-fb2059e3c9ca) [**The researcher claims superiority at large scale, but the largest model he trained here is 1.2 billion parameters on 31.5 billion tokens.**החוקר · statement](https://dgw.tmarket.io/claims/01a1e1ee-4774-7db0-a858-8bb5743cde1c)

[All claims](https://dgw.tmarket.io/claims)

```json
{"@context":"https://schema.org","@graph":[{"@id":"https://dgw.tmarket.io/#website","@type":"WebSite","description":"Decisions, events and falsifiable forecasts from 4,500+ Telegram channels — every claim cited. Read free.","name":"DecisionGateway","url":"https://dgw.tmarket.io/","publisher":{"@id":"https://dgw.tmarket.io/#identity"}},{"@id":"https://dgw.tmarket.io/claims/01a1e1eb-c6f2-7f5d-b2e2-ba78777e88c4#webpage","@type":"WebPage","description":"statement by the post author, recorded recently — verbatim quote and source citation.","name":"In every attention head in a model like BERT there are over 8,000 parameters that mathematically… — the post author — DecisionGateway","url":"https://dgw.tmarket.io/claims/01a1e1eb-c6f2-7f5d-b2e2-ba78777e88c4","breadcrumb":{"@id":"https://dgw.tmarket.io/claims/01a1e1eb-c6f2-7f5d-b2e2-ba78777e88c4#breadcrumb"},"isPartOf":{"@id":"https://dgw.tmarket.io/#website"},"potentialAction":[{"@type":"ReadAction","target":["https://dgw.tmarket.io/claims/01a1e1eb-c6f2-7f5d-b2e2-ba78777e88c4"]}]},{"@id":"https://dgw.tmarket.io/#identity","@type":"Organization","name":"DecisionGateway","url":"https://dgw.tmarket.io"},{"@id":"https://dgw.tmarket.io/claims/01a1e1eb-c6f2-7f5d-b2e2-ba78777e88c4#breadcrumb","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","name":"Home","item":"https://dgw.tmarket.io","position":1},{"@type":"ListItem","name":"Claims","item":"https://dgw.tmarket.io/claims","position":2},{"@type":"ListItem","name":"In every attention head in a model like BERT there are over…","position":3}]},{"@id":"https://dgw.tmarket.io/claims/01a1e1eb-c6f2-7f5d-b2e2-ba78777e88c4#/schema/quotation/1","@type":"Quotation","dateCreated":"2026-10-21","isBasedOn":"https://t.me/mathyaiwithmike/1292","text":"בכל ראש attention במודל כמו BERT יש מעל 8,000 פרמטרים שפשוט מבטלים אחד את השני מתמטית מאחורי הקלעים. זה נקרא \"חופש כיול\"","url":"/claims/01a1e1eb-c6f2-7f5d-b2e2-ba78777e88c4"}]}
```

## Sitemap

See the full [sitemap](https://dgw.tmarket.io/sitemap.md) for all pages.
