<![CDATA[8-bit KV Cache on Ampere]]>When using vLLM on Ampere GPUs, you may see that when selecting 8bit quantization formats for kv cache (fp8, int8) it falls back to other inference engines (Flashinfer, Triton Attention).

In my experience, Triton Attention performs very poorly on Ampere, especially on higher contexts, and Flashinfer doesn't work with DFlash2, which is most likely why these alternatives were made: (both available in the HyperQwen repository).

  1. For FP8 KV Cache -> There's a fork of Flash Attention 2 with fp8 kv cache.
  2. For Int8 KV Cache -> Use the 'PTH' option in HyperQwen.

In my experience, FP8 performed better at prefill and Int8 performed better at decode, both work with DFlash2 with solid results.

No AI used
]]>https://airigs.net/topic/39/8-bit-kv-cache-on-ampereRSS for NodeMon, 02 Nov 2026 05:41:10 GMTSat, 31 Oct 2026 23:48:20 GMT60