<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Local LLMs]]></title><description><![CDATA[Dense, MoE, reasoning and coding models on local hardware.]]></description><link>https://airigs.net/category/10</link><generator>RSS for Node</generator><lastBuildDate>Mon, 02 Nov 2026 06:00:25 GMT</lastBuildDate><atom:link href="https://airigs.net/category/10.rss" rel="self" type="application/rss+xml"/><pubDate>Sat, 31 Oct 2026 23:56:56 GMT</pubDate><ttl>60</ttl><item><title><![CDATA[Drafters are important]]></title><description><![CDATA[So, drafters pretty much give (almost) free token generation output, in exchange for a little bit of VRAM (and some very minor prompt processing speed).
There's 3 'popular' architectures for drafters ATM:

MTP, the most used by far, usually comes with the weights if the model supports it, lowest VRAM usage.
DSpark, deepseek's current format, performs better but uses more VRAM.
DFlash2, by far the best option, outstanding performance.

Rough baseline numbers:
[image: image.jpeg]
]]></description><link>https://airigs.net/topic/40/drafters-are-important</link><guid isPermaLink="true">https://airigs.net/topic/40/drafters-are-important</guid><dc:creator><![CDATA[fedes1to]]></dc:creator><pubDate>Sat, 31 Oct 2026 23:56:56 GMT</pubDate></item><item><title><![CDATA[What's an MoE, and why should I care?]]></title><description><![CDATA[You may see some language models described as MoE, short for Mixture of Experts. The name sounds complicated, but the basic idea is simple: instead of using the same parts of the model for every word it writes, an MoE model picks a few specialized parts to help with each piece of text.
Imagine a team of specialists
Think of the model as a team with many specialists. When it processes a piece of text, it routes that piece to a few of them. Different specialists may end up handling different pieces. The model combines their work to produce its response.
A regular dense model is more like asking the whole team to work on every piece. An MoE model uses only part of its team at a time.
Why does that matter?
Model names often show two sizes, such as “X billion total parameters, Y billion active.” Parameters are the learned values that make up a model:

Total parameters is roughly the size of the whole team.
Active parameters is roughly how much of the team is working on each piece of text.

The active number can help explain how much work the model does to generate each piece. But it does not tell you how much memory the model needs. The runtime still needs access to the model’s weights, including experts that may not be used for every piece. So an MoE model with a small active number can still have a large download and need a lot of memory.
Will an MoE run faster?
Not necessarily. Using only some experts at a time can reduce the amount of calculation for each piece of text. But speed also depends on your hardware, the model, and whether your runtime supports that MoE well. The experts’ weights still need to be available, and moving them around can take time.
What should I check?
Before downloading an MoE model, check:

The actual file size for the version you want. A smaller, quantized file uses less storage and usually less memory for weights, though it still needs extra room to run.
Whether your runtime supports that model.
How much VRAM and system RAM you have. Leave room for the conversation length (the context) and the runtime itself.
Reports from people using similar hardware and settings.

]]></description><link>https://airigs.net/topic/16/what-s-an-moe-and-why-should-i-care</link><guid isPermaLink="true">https://airigs.net/topic/16/what-s-an-moe-and-why-should-i-care</guid><dc:creator><![CDATA[fedes1to]]></dc:creator><pubDate>Wed, 30 Sep 2026 17:24:27 GMT</pubDate></item><item><title><![CDATA[New to Local LLMs? Start here!]]></title><description><![CDATA[Welcome! This guide will teach you the basics for large language models (LLMs) running on your own hardware. LLMs are used for generating text, so it will be useful for coding, chatting and so on.
1. Know the basics
We will use some terms that might be new to you so here's a simple definition for each:

Model weights: The learned numerical values that define a model.
Token: What the LLM perceives as a "piece of text". Can be a word, part of it, or something else.
Parameters: Learned values that make up the model.
Quantization: Storing model weights at lower precision to reduce memory usage, for example, Q4_0 uses ~4 bits per weight.
Context: Amount of tokens that a model can remember.
KV cache quantization: Reducing the precision of the context.

2. Choosing an engine
The inference engine is the program that loads the model, and serves it to your tools. Here are some baseline recommendations.

llama.cpp: Simplest to use, has a WebUI, lots of support. There almost always is a better option or atleast a fork that better suits your workload (especially on MoE's), but a good starter nonetheless.
vLLM: The gold standard, supports almost any model but picky on hardware, awesome concurrency but lacks good offloading, lets you pick between many kernels (Flash Attention, Flashinfer, ...).


Note: there are many many more options, if your card has good support consider using engines specifically for your hardware (e.g. ninfer, halogen).

3. Choosing a model
Usually VRAM will be your limiting factor, so start by running a model that will comfortably fit in yours, remember that you have to fit the model weights + KV cache + runtime overhead.
Also note that engines can only run specific model "packaging" formats, as an example:

llama.cpp (and its forks) can run GGUFs.
vLLM can run safetensors.

For starters, to overcome this limitation, we will use:

Quantization (4-6bit is a good baseline, includes Q4_K_M, Q6_K_XL, W4A16, ...).
KV Cache quantization (8bit is a good baseline, includes q8, fp8, int8, ...).

To improve our speeds we can also use drafting, although this will have a small VRAM penalty.
You can also overcome this limitation by using MoE models instead of dense ones, explained here.
4. Actually running it
If you plan on running multiple models on different engines, I recommend llama-swap, it simplifies greatly the process.
As a tip, to configure these engines, don't just tell a model to "Optimize and run for X hardware". Instead, understand what the launch parameters do and specify it to the model, for example:
"Install beellama.cpp latest on this machine and setup X model, use the Q4_K_XL and Q8 kv cache, I plan to have no concurrency, and I don't mind it loading slowly, make me a script to run it with and without drafting".
]]></description><link>https://airigs.net/topic/15/new-to-local-llms-start-here</link><guid isPermaLink="true">https://airigs.net/topic/15/new-to-local-llms-start-here</guid><dc:creator><![CDATA[fedes1to]]></dc:creator><pubDate>Wed, 30 Sep 2026 17:19:42 GMT</pubDate></item></channel></rss>