Open Weights Isn’t Open Training
URL: https://www.workshoplabs.ai/blog/open-weights-open-training
Author: Workshop Labs
Published: 2026
Summary
A practitioner’s account of trying to fine-tune Kimi-K2-Thinking (1T parameter MoE model) using open-source tooling (HuggingFace, LLaMA-Factory). Concludes that “open weights” and “open training” are very different things — the infrastructure to post-train frontier models is not actually open, despite weights being available.
Key Claims
- Open weights ≠ open training pipeline. Kimi-K2-Thinking has public weights but essentially no community training code. LLaMA-Factory claims to support it but had bugs. HuggingFace Transformers has architecture support but training is untested at this scale.
- Scale creates new failure modes: Kimi-K2-Thinking = 1T parameters, 594 GB on disk (570 GB for quantized experts). Loading it requires 8xH200s (1128 GB combined VRAM).
- HuggingFace is good but not ground truth: Libraries have dependency chains where bugs can hide layers down. Even PyTorch has edge cases that only surface at MoE scale.
- Flash attention + device_map=‘auto’ are the standard starting point but both have failure modes for MoE models at frontier scale.
- The Yoda test: Team used a “make the model talk like Yoda” dataset to verify training signal — qualitative behavioral change + loss decrease as success criteria.
- Practical outcome: Team built a custom training codebase rather than debugging open-source stacks, after spending days on existing options.
Implications for Alex
This is directly relevant to ML infra work. Fine-tuning-as-a-service (see Fine-Tuning as a Service) exists partly because the open-source path is harder than it looks. The gap between “has weights” and “can be trained by anyone” is a real moat for companies doing custom post-training.