Experimental Transformers implementation runs large models on a 32GB Mac by reading only needed MoE experts from SSD
Key Details The author released an experimental implementation that reads router-selected MoE experts from SSD instead of using Transformers' existing layer-level offloading. It reuses the existing grouped_mm kernel through a preallocated slot pool and expert ID mapping, and is configured to run Qwen3.8-Flash-Next on a 32GB Mac. Use Cases Install the supplied PR version of Transformers, assign experts to disk in device_map, and configure the experts_offload cache. The example combines 3-bit expert quantization, a 12GiB device cache, 8-bit dense layers, and disk storage for n-gram tables. Costs and Limitations This is a PR implementation under discussion with maintainers, rather than a finalized official API. The example expert files occupy 49GB of disk space, and conversion reportedly takes approximately five hours. Pricing and the code and model licenses are unconfirmed.
Visit original source