NVIDIA's Gated DeltaNet-2 separates erase and write gates in linear attention

Gated DeltaNet-2 adds independent per-channel erase and write gates to linear attention. The 1.3B model, trained on 100B FineWeb-Edu tokens, outperformed Mamba-2, Gated DeltaNet, KDA, and Mamba-3 across reported language-modeling, commonsense-reasoning, and S-NIAH retrieval benchmarks. The code is available under CC BY 4.0.

Visit original source