Skip to content
AIRigs
  1. Home
  2. Inference & Serving

Inference & Serving

7 Topics 7 Posts

Runtimes, servers, quantization and the numbers that decide what you deploy.

This category can be followed from the open social web via the handle inference-serving@airigs.net

Subcategories

  • llama.cpp, Ollama, MLX, vLLM, SGLang, TGI, TensorRT-LLM — engines, batching, KV cache, scheduling.

    3 3
    3 Topics
    3 Posts
    F
    Well someone finally made a decent engine for (smaller) MoE's such as Qwen 3.8 Flash Next for hardware that mere mortals can also own. Link here The numbers provided in the repo's README file are impressive, I hope there's more work put into this. Unfortunately it only works on NVIDIA GPUs, and also has barebones support for multi-GPU setups.

  • GGUF, EXL, AWQ, GPTQ, FP8, bitsandbytes.

    3 3
    3 Topics
    3 Posts
    F
    When using vLLM on Ampere GPUs, you may see that when selecting 8bit quantization formats for kv cache (fp8, int8) it falls back to other inference engines (Flashinfer, Triton Attention). In my experience, Triton Attention performs very poorly on Ampere, especially on higher contexts, and Flashinfer doesn't work with DFlash2, which is most likely why these alternatives were made: (both available in the HyperQwen repository). For FP8 KV Cache -> There's a fork of Flash Attention 2 with fp8 kv cache. For Int8 KV Cache -> Use the 'PTH' option in HyperQwen. In my experience, FP8 performed better at prefill and Int8 performed better at decode, both work with DFlash2 with solid results.

  • Tokens/s, time-to-first-token, concurrency curves.

    1 1
    1 Topics
    1 Posts
    F
    I FEEL like this is too good to be true, runs a patched vLLM on 2 RTX 3090's: GitHub recipe Numbers are absolutely insane. Something more in-line with what I'd expect is what Minachist achieved: HF discussion Blog post