Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference

T1Covered by 1 source · first reported by arXiv at 24 Sept, 04:00 UTC

arXiv:2609.26621v1 Announce Type: cross Abstract: Greedy decoding from large language models is commonly treated as deterministic. We show it is not precision-invariant: the same model, prompt, and decoding algorithm produce different outputs in BF16 versus FP16 on identical hardware. Across our…

aiscience

Sources

• T1Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM InferencearXiv24 Sept, 04:00