What's an MoE, and why should I care?
-
You may see some language models described as MoE, short for Mixture of Experts. The name sounds complicated, but the basic idea is simple: instead of using the same parts of the model for every word it writes, an MoE model picks a few specialized parts to help with each piece of text.
Imagine a team of specialists
Think of the model as a team with many specialists. When it processes a piece of text, it routes that piece to a few of them. Different specialists may end up handling different pieces. The model combines their work to produce its response.
A regular dense model is more like asking the whole team to work on every piece. An MoE model uses only part of its team at a time.
Why does that matter?
Model names often show two sizes, such as “X billion total parameters, Y billion active.” Parameters are the learned values that make up a model:
- Total parameters is roughly the size of the whole team.
- Active parameters is roughly how much of the team is working on each piece of text.
The active number can help explain how much work the model does to generate each piece. But it does not tell you how much memory the model needs. The runtime still needs access to the model’s weights, including experts that may not be used for every piece. So an MoE model with a small active number can still have a large download and need a lot of memory.
Will an MoE run faster?
Not necessarily. Using only some experts at a time can reduce the amount of calculation for each piece of text. But speed also depends on your hardware, the model, and whether your runtime supports that MoE well. The experts’ weights still need to be available, and moving them around can take time.
What should I check?
Before downloading an MoE model, check:
- The actual file size for the version you want. A smaller, quantized file uses less storage and usually less memory for weights, though it still needs extra room to run.
- Whether your runtime supports that model.
- How much VRAM and system RAM you have. Leave room for the conversation length (the context) and the runtime itself.
- Reports from people using similar hardware and settings.
AI usedHow: was too lazy to write it myself lol -
F fedes1to referenced this topic on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login