Runtimes & Serving Stacks

Well someone finally made a decent engine for (smaller) MoE's such as Qwen 3.8 Flash Next for hardware that mere mortals can also own. Link here The numbers provided in the repo's README file are impressive, I hope there's more work put into this. Unfortunately it only works on NVIDIA GPUs, and also has barebones support for multi-GPU setups.

Ollama is one of the easiest ways to get started with local models: install it, pull a model, and run it with a simple command. That convenience is valuable, especially when you’re new to local inference. But ease of setup doesn’t automatically make a tool the best fit for every model, machine, or workflow. I highly recommend reading this article: Friends Don’t Let Friends Use Ollama — Sleeping Robots Ollama’s convenience no longer outweighs its drawbacks. It obscures its reliance on llama.cpp, can lag behind other runtimes in performance and model support, and adds friction through its model packaging and configuration. I’m also concerned by its cloud pivot and the distinction between local and hosted inference. For more control and broader compatibility, I recommend trying other runtimes that better fits your hardware.

Links: lk-moe Lvllm Lsglang Open source implementation of lk-moe: xiaotu-moe These inference engines seem to implement features that greatly improve CPU and CPU offload performance with the NUMA parallel architecture, still to be tested, would appreciate benchmarks! (There are some numbers in the GitHub repos.)