- Amazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints to reduce time to first token for LLM workloads.
- Enable the feature by setting `RoutingStrategy` to `KV_AWARE` and adding an environment variable, requiring AWS vLLM DLC version 0.33.0 or later.
- This feature ships in Amazon SageMaker AI now supports KV cache-aware routing for real-time inference endpoints.