Welcome to AIRigs! This is a forum for people to share useful information regarding AI inference and training. As long as you don't have an awfully outdated piece of hardware, you should be able to run very capable models inside the hardware you already own, or maybe splurge some money to get solid intelligence. Registration Also, while you're at it, you should make an account and add your hardware rig so others can help you in your specific setup. Go to profile icon → Settings → Rig to add your hardware. See an example Rig profile here. What do I need? Well, as mentioned before, most likely your current hardware will suffice, but you need to understand what specs makes hardware good for inference (or horrible): GPU memory (VRAM) is usually the most important,, the model should fit experts (on an MoE) or entirely (on a dense model) in the VRAM. "CPU" Memory (RAM) is important if you're offloading an MoE. Usually whatever the GPU can't fit will go here. Bandwidth applies to both VRAM and RAM, bigger models need more bandwidth. TFLOPs meaning Trillion FLOating Point operations per second roughly represent the raw speed of your hardware. Some other self-explanatory stuff does apply aswell such as driver support, power consumption, natively supported operations, which we won't go into detail here. As you might have realized by now, GPUs are (at this moment) the most viable way to run Local AI, so we will mostly focus on these. Ok so, what do you want to run? We have categories in the forum for many types of "Local AI", and working on adding solid beginner tutorials for each. Chat, coding, and text generation: large language models (LLMs) Image or video generation: diffusion and video models Speech: transcription, text-to-speech, and audio models Image or document understanding: vision-language and multimodal models Note: Tutorials might be missing since this is still a WIP

So, drafters pretty much give (almost) free token generation output, in exchange for a little bit of VRAM (and some very minor prompt processing speed). There's 3 'popular' architectures for drafters ATM: MTP, the most used by far, usually comes with the weights if the model supports it, lowest VRAM usage. DSpark, deepseek's current format, performs better but uses more VRAM. DFlash2, by far the best option, outstanding performance. Rough baseline numbers: [image: image.jpeg]

When using vLLM on Ampere GPUs, you may see that when selecting 8bit quantization formats for kv cache (fp8, int8) it falls back to other inference engines (Flashinfer, Triton Attention). In my experience, Triton Attention performs very poorly on Ampere, especially on higher contexts, and Flashinfer doesn't work with DFlash2, which is most likely why these alternatives were made: (both available in the HyperQwen repository). For FP8 KV Cache -> There's a fork of Flash Attention 2 with fp8 kv cache. For Int8 KV Cache -> Use the 'PTH' option in HyperQwen. In my experience, FP8 performed better at prefill and Int8 performed better at decode, both work with DFlash2 with solid results.

So, Nvidia released the CMP series of cards long ago, targetting the crypto craze from back them, these are pretty much locked down A100's, but locked to low PCIe lanes on an old revision, bad compute for pretty much anything else and only 8-10GB of VRAM. Now, these cards do phisically have better capabilities, and they can get back the VRAM and the compute, although they still lack in PCIe speeds. Unlock instructions (repo) here: https://github.com/amoghmunikote/cmpunlocker

It's the agent I use day to day. Solid, lightweight and doesn't have many dependencies. https://nanobot.wiki/