BF16 memory savings can come with MiniMax decode drift
By OnTheRice · Sources and limitations · Report a correction
BF16 reduces recurrent-state memory, but its decay arithmetic can erase long-run forgetting and accumulate output drift. The Transformers reporter calculated that 388/4,480 MiniMax heads round their decay to 1.0; a tiny random-weight CPU test's relative error versus FP32 rose from 3.31% at 500 tokens to 7.39% at 3,000. An FP32 recurrence reduced error to 0.58%/0.39%, but doubled cache-state memory. No real checkpoint was tested and PR #49291 remains open. This model-specific trade-off is not a BF16-wide trend; next verify the patch on released weights and GPU long-decode tests.
Source confidenceNot assessed
What’s changing?
BF16 spacing near one can round slow per-token decay factors to exactly one; FP32 recurrence preserves those small state updates but requires FP32 cache state.
Key points
- Transformers issue #49289 reports that 388/4,480 heads in the released MiniMax config round BF16 decay to 1.0. A tiny random-weight CPU reproduction had relative error versus FP32 3.31% at 500 tokens and 7.39% at 3,000; local FP32 recurrence reduced this to 5.75e-3 and 3.90e-3 but doubled recurrent cache state. No real checkpoint was tested. Open PR #49291 proposes the FP32 recurrence and remains unmerged.
What could change this view?
- Issue reporter’s effect-size test used a tiny random-weight CPU model; the 388-head count uses released model configuration slopes. No real checkpoint or GPU test was run; the fix PR is open/unmerged.
Bottom line
Long decode may trade numerical fidelity for a larger recurrent cache if FP32 state is used; prefill was reported as less affected because decay is applied once per block.
This is based on the information recorded with this signal.
How likely is this call? · Three frameworks
This signal has no investible bullish or bearish call. A market-direction percentage does not apply.
Evidence behind this view
Checks whether separate pieces of evidence support the same view.
Recorded connection
BF16 spacing near one can round slow per-token decay factors to exactly one; FP32 recurrence preserves those small state updates but requires FP32 cache state.
We have not established a separate count of independent evidence clusters for this card. Several reports may rely on the same event or data provider.
What past price moves show
Compares today’s price moves with recorded past patterns.
This card has no recorded investible call. A price-pattern comparison is not applicable unless a specific asset is added with evidence.
Evidence against this view
Shows evidence and risks that could challenge this view.
Recorded opposing evidence and risks
Issue reporter’s effect-size test used a tiny random-weight CPU model; the 388-head count uses released model configuration slopes. No real checkpoint or GPU test was run; the fix PR is open/unmerged.
Why the direction changed
Explains recorded changes from expecting a rise to expecting a fall, or the reverse.
No documented bullish-to-bearish or bearish-to-bullish change is available for this card. We keep the original call and its time unchanged.
What people say vs the facts
Compares recorded public discussion with checked market facts.
No separate Reddit or public-discussion comparison with verified market data is recorded for this card. We cannot treat social attention as proof of a market move.
How past forecasts performed
Shows recorded forecast results and what they can teach us.
This card has no recorded investible forecast to score.
No tested mapping from evidence combinations to future success is recorded for this card. Confidence in sources and past outcomes do not establish a calibrated probability.
Sources, original wording and price checks
Published
First published through Sites 05 Oct 2026, 01:54:01 SGT
No explicit investible BULLISH or BEARISH call on this card.
What changed
Recorded public historyBF16 reduces recurrent-state memory, but its decay arithmetic can erase long-run forgetting and accumulate output drift. The Transformers…
This is the first recorded public view. Further public updates will appear here.
- BF16 memory savings can come with MiniMax decode drift
BF16 reduces recurrent-state memory, but its decay arithmetic can erase long-run forgetting and accumulate output drift. The Transformers reporter calculated that 388/4,480 MiniMax heads round their decay to 1.0; a tiny random-weight CPU test's relative error versus FP32 rose from 3.31% at 500 tokens to 7.39% at 3,000. An FP32 recurrence reduced error to 0.58%/0.39%, but doubled cache-state memory. No real checkpoint was tested and PR #49291 remains open. This model-specific trade-off is not a BF16-wide trend; next verify the patch on released weights and GPU long-decode tests.
First published by OTR: 5 Oct, 01:54 SGTPublished by OTR: 5 Oct 2026, 01:54 SGT
Why the view changed
This is the first recorded view. Future source-backed updates will appear here.
Evidence, memory & review 2 linked sources
Directional takeaway: N/A: these configuration-specific software reports do not support an asset-specific listed-security conclusion.
OTR reasoning: BF16 spacing near one can round slow per-token decay factors to exactly one; FP32 recurrence preserves those small state updates but requires FP32 cache state.
Source-grounded facts (paraphrased)
- Time not supplied: Transformers issue #49289 reports that 388/4,480 heads in the released MiniMax config round BF16 decay to 1.0. A tiny random-weight CPU reproduction had relative error versus FP32 3.31% at 500 tokens and 7.39% at 3,000; local FP32 recurrence reduced this to 5.75e-3 and 3.90e-3 but doubled recurrent cache state. No real checkpoint was tested. Open PR #49291 proposes the FP32 recurrence and remains unmerged.
Source trail
- MiniMax lightning attention: decay recurrence runs in bf16, so slow-decay heads never decay and decode drifts with length · Issue #49289 ↗Hugging Face Transformers / GitHub2026-10-04T04:16:50Z · published (UTC)
- fix(minimax): run lightning attention decay recurrence in fp32 to prevent bf16 drift · PR #49291 ↗Hugging Face Transformers / GitHub2026-10-04T06:50:36Z · published (UTC)
Review status: SUPPORTED WITH ATTRIBUTION CLARITY
No score or probability is presented.