8-bit KV Cache on Ampere
-
When using vLLM on Ampere GPUs, you may see that when selecting 8bit quantization formats for kv cache (fp8, int8) it falls back to other inference engines (Flashinfer, Triton Attention).
In my experience, Triton Attention performs very poorly on Ampere, especially on higher contexts, and Flashinfer doesn't work with DFlash2, which is most likely why these alternatives were made: (both available in the HyperQwen repository).
- For FP8 KV Cache -> There's a fork of Flash Attention 2 with fp8 kv cache.
- For Int8 KV Cache -> Use the 'PTH' option in HyperQwen.
In my experience, FP8 performed better at prefill and Int8 performed better at decode, both work with DFlash2 with solid results.
No AI used
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login