Welcome to AIRigs! This is a forum for people to share useful information regarding AI inference and training.
As long as you don't have an awfully outdated piece of hardware, you should be able to run very capable models inside the hardware you already own, or maybe splurge some money to get solid intelligence.
Registration
Also, while you're at it, you should make an account and add your hardware rig so others can help you in your specific setup.
Go to profile icon → Settings → Rig to add your hardware. See an example Rig profile here.
What do I need?
Well, as mentioned before, most likely your current hardware will suffice, but you need to understand what specs makes hardware good for inference (or horrible):
GPU memory (VRAM) is usually the most important,, the model should fit experts (on an MoE) or entirely (on a dense model) in the VRAM.
"CPU" Memory (RAM) is important if you're offloading an MoE. Usually whatever the GPU can't fit will go here.
Bandwidth applies to both VRAM and RAM, bigger models need more bandwidth.
TFLOPs meaning Trillion FLOating Point operations per second roughly represent the raw speed of your hardware.
Some other self-explanatory stuff does apply aswell such as driver support, power consumption, natively supported operations, which we won't go into detail here.
As you might have realized by now, GPUs are (at this moment) the most viable way to run Local AI, so we will mostly focus on these.
Ok so, what do you want to run?
We have categories in the forum for many types of "Local AI", and working on adding solid beginner tutorials for each.
Chat, coding, and text generation: large language models (LLMs)
Image or video generation: diffusion and video models
Speech: transcription, text-to-speech, and audio models
Image or document understanding: vision-language and multimodal models
Note: Tutorials might be missing since this is still a WIP
