GPUs & Accelerators

So, Nvidia released the CMP series of cards long ago, targetting the crypto craze from back them, these are pretty much locked down A100's, but locked to low PCIe lanes on an old revision, bad compute for pretty much anything else and only 8-10GB of VRAM. Now, these cards do phisically have better capabilities, and they can get back the VRAM and the compute, although they still lack in PCIe speeds. Unlock instructions (repo) here: https://github.com/amoghmunikote/cmpunlocker

Nvidia shipped consumer GPUs with a limited number of tensor cores activated, so their server cards could actually compete for the price. Unfortunately, someone ported the CMP unlocker's exploit to these other cards and used it to forcefully unlock said tensor cores, resulting in dramatically better performance for bf16 (1.5x-2x more). Repo link: https://github.com/Zagitta/gpu-unlock

Some gpu "labs" are modifying NVIDIA GPUs to get way more VRAM on them. Probably still one of the best options for the money, but fairly risky. NOTE: AFAIK the 5090 96GB are FAKE, do NOT buy them. Beware of scammers! You can find these GPUs easily on eBay or Alibaba for solid pricing, the ones I know of are: RTX 2080ti 22GB RTX 3080 20GB RTX 4080 32GB RTX 4090 48GB I myself own 2x RTX 3080 20GB and am very impressed with the results, they are real, and do exist, but don't have working ReBAR support (although a fix for this is discussed here, this majorly affects tensor parallel setups (multi-gpu), I highly recommend this tweak). The performance I get seems right, about 80tk/s under heavy context load on Qwen 3.8 27B using this quant with 256k ctx and vision enabled (with MTP).

Made a website to compare hardware easily and graph options. https://hw.fedesito.me Enjoy! Data is mostly from https://www.techpowerup.com