Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets
Google team have released llm-d 1.0 , the first stable version of its distributed inference stack. This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool. They released it at the PyTorch Conference North America. It targets self-hosted agent and RL workloads, where round-robin load balancing breaks down. TL;DR A serving stack : Google reports a published benchmark on a 17,000-node multi-region GKE deployment....
Google team have released llm-d 1.0 , the first stable version of its distributed inference stack. This open-source, Kubernetes-native layer turns vLLM or SGLang model servers into one coordinated pool. They released it at the PyTorch Conference North America. It targets self-hosted agent and RL workloads, where round-robin load balancing breaks down. TL;DR A serving stack : Google reports a published benchmark on a 17,000-node multi-region GKE deployment....