Amazon SageMaker HyperPod Inference Gateway for scalable LLM inference
Amazon SageMaker HyperPod Inference Gateway is a Kubernetes-native, GPU-aware routing system that deploys as a single EKS managed add-on on existing SageMaker HyperPod infrastructure with zero application changes. By replacing unintelligent round-robin load balancing with real-time…
Sources
- T1Amazon SageMaker HyperPod Inference Gateway for scalable LLM inferenceAWS — What's New / News Blog