TL;DR
- Kimi K3 is the largest open-weight model that currently exists, scaled from Kimi Linear’s 48 billion parameters up to 2.8 trillion, per Sebastian Raschka’s architecture teardown.
- Running it takes around 1.4 terabytes of memory, so an individual developer or most companies cannot point it at their own infrastructure.
- Chip markets barely reacted the way they did to last year’s DeepSeek moment, since a model this size is bullish for memory suppliers while threatening nobody’s existing GPU budget.
- Open weights existing and open weights being usable by an ordinary person have become two separate claims, with K3 as the clearest evidence of the split.
In plain English
Parameters are the learned values inside a model, and they have to be held in memory while the model runs, which is where the 1.4 terabyte figure comes from. Raschka’s teardown lists what changed architecturally. LatentMoE, borrowed from Nemotron 3 Ultra, compresses the large linear layers in the same way multi-head latent attention already compresses attention, attention being the mechanism that decides which parts of the input each part of the output should draw on. Every RoPE layer was dropped for NoPE throughout: positional embeddings are how a model knows the order of words in a sequence, and Raschka reads K3 as the first frontier-scale model to go entirely without them rather than mixing approaches.
A new attention-residual mechanism connects residuals across layers using an attention score, buying a small consistent gain in validation loss and downstream benchmarks for roughly 4 per cent more training compute and 2 per cent more inference compute, inference being the cost of actually running the model rather than training it. Multimodal support, meaning handling images alongside text, is built in from the start. Open weights means the trained parameters are published for anyone to download. At this scale, that amounts to a lab or a well-capitalised inference provider running K3 and renting access to it.
Open weights existing and open weights being usable by an ordinary person are two separate claims, and most of the coverage of Kimi K3 this week is only proving the first one.
Sebastian Raschka’s architecture teardown is the clearest account of what actually changed. K3 is, in his description, essentially a scaled-up production version of Kimi Linear, taken from 48 billion parameters up to 2.8 trillion, making it the largest open-weight model that currently exists. It picked up a new component called LatentMoE, borrowed from Nemotron 3 Ultra, to compress the large linear layers the same way multi-head latent attention already compresses attention. Every RoPE layer got dropped in favor of NoPE throughout, which as far as Raschka can tell makes K3 the first frontier-scale model to go all in on no positional embeddings rather than mixing approaches. A new attention-residual mechanism connects residuals across layers using an attention score, buying a small, consistent gain in validation loss and downstream benchmarks for roughly 4% more training compute and 2% more inference compute. Native multimodal support is built in from the start rather than bolted on afterward.
None of that changes the arithmetic on the other side. Running K3 takes something in the neighborhood of 1.4 terabytes of memory, which is a large part of why chip markets barely reacted to it the way they reacted to last year’s DeepSeek moment. A model this size is bullish for the memory suppliers. It isn’t a threat to anyone’s existing GPU budget, because almost nobody’s existing GPU budget was built with this in mind.
“Open” here means a lab or a well-capitalized inference provider can run K3 and rent access to it. It doesn’t mean an individual developer, or most companies, can point it at their own infrastructure and run it themselves, which was the entire premise that made open weights matter to people outside the largest labs in the first place. I don’t have hardware within an order of magnitude of what this needs, and neither does almost anyone reading this.
Open-weight and independently-runnable used to be close to the same claim, back when the interesting open models still fit on a well-specced workstation. They aren’t the same claim anymore, and K3 is the clearest evidence yet of the split. The weights are public. The model itself, for anything most people can actually do with it, is somebody else’s machine.