<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Runtimes & Serving Stacks]]></title><description><![CDATA[llama.cpp, Ollama, MLX, vLLM, SGLang, TGI, TensorRT-LLM — engines, batching, KV cache, scheduling.]]></description><link>https://airigs.net/category/16</link><generator>RSS for Node</generator><lastBuildDate>Mon, 02 Nov 2026 05:59:01 GMT</lastBuildDate><atom:link href="https://airigs.net/category/16.rss" rel="self" type="application/rss+xml"/><pubDate>Wed, 30 Sep 2026 16:17:25 GMT</pubDate><ttl>60</ttl><item><title><![CDATA[Strata: Run decent models on small hardware]]></title><description><![CDATA[Well someone finally made a decent engine for (smaller) MoE's such as Qwen 3.8 Flash Next for hardware that mere mortals can also own.
Link here
The numbers provided in the repo's README file are impressive, I hope there's more work put into this.
Unfortunately it only works on NVIDIA GPUs, and also has barebones support for multi-GPU setups.
]]></description><link>https://airigs.net/topic/13/strata-run-decent-models-on-small-hardware</link><guid isPermaLink="true">https://airigs.net/topic/13/strata-run-decent-models-on-small-hardware</guid><dc:creator><![CDATA[fedes1to]]></dc:creator><pubDate>Wed, 30 Sep 2026 16:17:25 GMT</pubDate></item><item><title><![CDATA[Stop using Ollama, please.]]></title><description><![CDATA[Ollama is one of the easiest ways to get started with local models: install it, pull a model, and run it with a simple command. That convenience is valuable, especially when you’re new to local inference. But ease of setup doesn’t automatically make a tool the best fit for every model, machine, or workflow.
I highly recommend reading this article:
Friends Don’t Let Friends Use Ollama — Sleeping Robots
Ollama’s convenience no longer outweighs its drawbacks. It obscures its reliance on llama.cpp, can lag behind other runtimes in performance and model support, and adds friction through its model packaging and configuration. I’m also concerned by its cloud pivot and the distinction between local and hosted inference. For more control and broader compatibility, I recommend trying other runtimes that better fits your hardware.
]]></description><link>https://airigs.net/topic/11/stop-using-ollama-please.</link><guid isPermaLink="true">https://airigs.net/topic/11/stop-using-ollama-please.</guid><dc:creator><![CDATA[fedes1to]]></dc:creator><pubDate>Wed, 30 Sep 2026 16:04:51 GMT</pubDate></item><item><title><![CDATA[NUMA aware inference with lk_moe / lvllm / lsglang]]></title><description><![CDATA[Links:
lk-moe
Lvllm
Lsglang
Open source implementation of lk-moe: xiaotu-moe
These inference engines seem to implement features that greatly improve CPU and CPU offload performance with the NUMA parallel architecture, still to be tested, would appreciate benchmarks! (There are some numbers in the GitHub repos.)
]]></description><link>https://airigs.net/topic/5/numa-aware-inference-with-lk_moe-lvllm-lsglang</link><guid isPermaLink="true">https://airigs.net/topic/5/numa-aware-inference-with-lk_moe-lvllm-lsglang</guid><dc:creator><![CDATA[fedes1to]]></dc:creator><pubDate>Mon, 28 Sep 2026 16:25:26 GMT</pubDate></item></channel></rss>