A clean warm run can hide state-dependent inference errors
By OnTheRice · Sources and limitations · Report a correction
A warm run can't establish correctness when cache or memory state changes. On a 32 GB M1 Pro, the PyTorch MPS report found wrong gradients in 74/1,831 passes on 2.14.1 and 179/767 on 2.13.0; a vLLM cold-cache Qwen3-1.7B/FA2 repro returned token 16,448 versus reference 271, while warm cache matched. The PR author reports the proposed fix passing its repro and CPU tests; 13 broader tests still fail, and PR remains open. H200/FA3 matched separately. These are two bugs, not a failure-rate trend. Test cold/warm and concurrent/fork paths; next replay both fixes.
Source confidenceNot assessed
What’s changing?
PyTorch MPS can consume stale buffers after a Metal command-buffer abort without raising; vLLM issue #59940 attributes a separate wrong-prefill path to import-order/cold-cache state and forked workers replaying captured attention arguments.
Key points
- The PyTorch MPS issue reports author-run wrong-gradient counts of 74/1,831 passes on torch 2.14.1 and 179/767 on 2.13.0, with zero exceptions, under four-process memory pressure on one 32 GB M1 Pro. The separate vLLM issue reports a cold-cache 16-token Qwen3-1.7B/FA2 prefill on RTX PRO 6000 Blackwell returning token 16,448 vs reference 271; warm cache matched. H200/FA3 matched 17/17 in a separate configuration. Open PR #59941 reports patched cold in-process/default-multiprocess repros and CPU tests pass, while 13 broader gated-model/shared-memory tests fail; PR remains unmerged.
What could change this view?
- Two separate author-run, configuration-specific bugs, not an industry failure-rate estimate. PyTorch MPS reproducer used one 32 GB M1 Pro. vLLM error was reported for Qwen3-1.7B, FlashAttention2 and RTX PRO 6000 Blackwell; the 17/17 H200/FA3 result is a separate setup. Proposed vLLM fix is open; wider test pair still has 13 gated-model/shared-memory failures.
Bottom line
State-dependent starts and memory pressure can hide silent incorrect outputs; regression coverage should include cold/warm cache and concurrent/forked paths.
This is based on the information recorded with this signal.
How likely is this call? · Three frameworks
This signal has no investible bullish or bearish call. A market-direction percentage does not apply.
Evidence behind this view
Checks whether separate pieces of evidence support the same view.
Recorded connection
PyTorch MPS can consume stale buffers after a Metal command-buffer abort without raising; vLLM issue #59940 attributes a separate wrong-prefill path to import-order/cold-cache state and forked workers replaying captured attention arguments.
We have not established a separate count of independent evidence clusters for this card. Several reports may rely on the same event or data provider.
What past price moves show
Compares today’s price moves with recorded past patterns.
This card has no recorded investible call. A price-pattern comparison is not applicable unless a specific asset is added with evidence.
Evidence against this view
Shows evidence and risks that could challenge this view.
Recorded opposing evidence and risks
Two separate author-run, configuration-specific bugs, not an industry failure-rate estimate. PyTorch MPS reproducer used one 32 GB M1 Pro. vLLM error was reported for Qwen3-1.7B, FlashAttention2 and RTX PRO 6000 Blackwell; the 17/17 H200/FA3 result is a separate setup. Proposed vLLM fix is open; wider test pair still has 13 gated-model/shared-memory failures.
Why the direction changed
Explains recorded changes from expecting a rise to expecting a fall, or the reverse.
No documented bullish-to-bearish or bearish-to-bullish change is available for this card. We keep the original call and its time unchanged.
What people say vs the facts
Compares recorded public discussion with checked market facts.
No separate Reddit or public-discussion comparison with verified market data is recorded for this card. We cannot treat social attention as proof of a market move.
How past forecasts performed
Shows recorded forecast results and what they can teach us.
This card has no recorded investible forecast to score.
No tested mapping from evidence combinations to future success is recorded for this card. Confidence in sources and past outcomes do not establish a calibrated probability.
Sources, original wording and price checks
Published
First published through Sites 05 Oct 2026, 01:54:01 SGT
No explicit investible BULLISH or BEARISH call on this card.
What changed
Recorded public historyA warm run can't establish correctness when cache or memory state changes. On a 32 GB M1 Pro, the PyTorch MPS report found wrong gradients…
This is the first recorded public view. Further public updates will appear here.
- A clean warm run can hide state-dependent inference errors
A warm run can't establish correctness when cache or memory state changes. On a 32 GB M1 Pro, the PyTorch MPS report found wrong gradients in 74/1,831 passes on 2.14.1 and 179/767 on 2.13.0; a vLLM cold-cache Qwen3-1.7B/FA2 repro returned token 16,448 versus reference 271, while warm cache matched. The PR author reports the proposed fix passing its repro and CPU tests; 13 broader tests still fail, and PR remains open. H200/FA3 matched separately. These are two bugs, not a failure-rate trend. Test cold/warm and concurrent/fork paths; next replay both fixes.
First published by OTR: 5 Oct, 01:54 SGTPublished by OTR: 5 Oct 2026, 01:54 SGT
Why the view changed
This is the first recorded view. Future source-backed updates will appear here.
Evidence, memory & review 3 linked sources
Directional takeaway: N/A: these configuration-specific software reports do not support an asset-specific listed-security conclusion.
OTR reasoning: PyTorch MPS can consume stale buffers after a Metal command-buffer abort without raising; vLLM issue #59940 attributes a separate wrong-prefill path to import-order/cold-cache state and forked workers replaying captured attention arguments.
Source-grounded facts (paraphrased)
- Time not supplied: The PyTorch MPS issue reports author-run wrong-gradient counts of 74/1,831 passes on torch 2.14.1 and 179/767 on 2.13.0, with zero exceptions, under four-process memory pressure on one 32 GB M1 Pro. The separate vLLM issue reports a cold-cache 16-token Qwen3-1.7B/FA2 prefill on RTX PRO 6000 Blackwell returning token 16,448 vs reference 271; warm cache matched. H200/FA3 matched 17/17 in a separate configuration. Open PR #59941 reports patched cold in-process/default-multiprocess repros and CPU tests pass, while 13 broader gated-model/shared-memory tests fail; PR remains unmerged.
Source trail
- [MPS] Command buffers aborted by the driver are silently ignored: wrong gradients · Issue #199680 ↗PyTorch / GitHub2026-10-04T08:31:32Z · published (UTC)
- [Bug] VLLM_BATCH_INVARIANT=1 produces wrong tokens on a cold model-info cache · Issue #59940 ↗vLLM / GitHub2026-10-04T12:00:29Z · published (UTC)
- Proposed fix for cold-cache batch-invariant attention · PR #59941 (open, unmerged) ↗vLLM / GitHub2026-10-04T12:00:53Z · published (UTC)
Review status: SUPPORTED WITH QUALIFICATIONS
No score or probability is presented.