vLLM v0.34.0 enables extensible KV cache by default and adds support for 10 new models

Key details vLLM v0.34.0 enables an extensible KV cache by default, sizing it according to actual available memory after warm-up, and unifies GPU execution under Model Runner V2. It adds support for models including Mistral Large 4 and Nemotron-3.5-Transcribe, along with checkpoint save and restore APIs and the ability to scale down some distributed configurations without restarting. Expanding the cache can increase per-token decoding latency under some conditions. Use cases Users can serve speech models through /v1/audio/transcriptions or save and restore execution state through the checkpoint API. To retain the previous cache-sizing method, specify --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache. Costs and limitations The default block size changes from 16 to 64, and huggingface_hub 2.0.0 or later is required. ROCm distributions are provided only for 10.0, and Model Runner V1 settings cause startup errors. Pricing and licensing are unconfirmed.

Visit original source