Skip to content
Marc Allington
Marc Allington
← All signals
Models and platformsgood news for the people this stream is for

AWS has added a KV_AWARE routing strategy to SageMaker AI real-time endpoints. It sends each request to the instance holding the longest cached match for the prompt, falling back to load-aware routing when there is no useful match. It needs the AWS vLLM container (0.33.0 or later), prefix caching and at least two instances. AWS reports up to 97% lower median time to first token and up to 234% higher throughput against random routing on Llama 3.1 70B, with smaller gains on short prompts. Regions and pricing were not visible on the page.