Sapient AI releases HRM-Text-1B, a hierarchical recurrent language model
HRM-Text-1B combines slow and fast Transformer stacks in a two-timescale recurrent architecture. The Apache 2.0 checkpoint was pretrained on 40B public-data tokens and supports up to 4,096 context tokens. It shows few-shot performance on text classification, extraction, and short-answer QA. The checkpoint predates supervised tuning and RLHF and requires further fine-tuning.
Visit original source