Categories: FAANG

Amazon SageMaker Inference: 2026 launches in review, part 2

Once a model is in production, the questions change. Teams want to know which configuration the model should run on now that traffic is real, how to fit more models onto GPUs they already pay for, what to do when the next model is too large for one machine, why requests wait in one Region while GPUs sit idle in another, and why latency rose when nobody changed anything. This post takes those questions in the order teams usually meet them.

For HyperPod Inference, it listed additional routing strategies, broader support for model serving frameworks, and expanded multi-Region networking. The Inference Gateway delivers the routing, and managed NVIDIA Dynamo is a new serving framework. For managed endpoints, the roadmap pointed to deeper integration with the rest of the AWS AI landscape. Inference recommendations now run from Amazon SageMaker AI Studio, the SageMaker Python SDK, and the coding agents engineers already use, and endpoints can now route requests across Regions and accounts.

To keep each section concrete, we follow one hypothetical team through the blog post. In the spring, they launched a customer support assistant on a 70B-parameter model behind a single SageMaker AI endpoint, using the launches from part 1. By the fall, they need to support more product lines, adopt a larger reasoning model, serve customers in a second Region, and keep their GPU bill flat.

The launches at a glance

Figure 1: The five stages in this post and the launches in each

Choose the right configuration from wherever you work

The team picked its first configuration from a benchmark run in April. Since then, they have added four LoRA (Low-Rank Adaptation) adapters, one for each product line, and the product team has set a target of 500 milliseconds for time to first token (TTFT) at peak load. The April configuration was never tested against either change. The engineers who will re-run the analysis work in notebooks and coding agents, and few of them use the console.

Inference recommendations launched in April as an API. Four updates since then make it available in more of the places teams work, and let it answer two harder questions: how many adapters an endpoint can carry, and how much concurrency it can take before it misses a latency target. A new agent skill brings the same expertise into coding agents.

Inference recommendations in SageMaker AI Studio (July 2026)

Blog: Launching UI for generative AI inference recommendations in Amazon SageMaker AI

The April launch automated instance, container, and optimization selection through APIs. Inference recommendations are now also available in SageMaker AI Studio as a guided, low-code workflow. In Studio, choose Jobs, then Inference optimization. Choose a use-case profile (Interact, Generate, Summarize, or Custom), an optimization goal (minimize latency, maximize throughput, or minimize cost), and a model from SageMaker JumpStart, Amazon Simple Storage Service (Amazon S3), the Model Registry, or an existing SageMaker model.

SageMaker AI benchmarks candidate configurations on real GPU infrastructure using NVIDIA AIPerf and applies techniques that match the goal, such as speculative decoding for throughput or kernel tuning for latency. You compare the ranked results by TTFT, inter-token latency, throughput, and cost, and then deploy the chosen configuration to a real-time endpoint from Studio.

Inference recommendations in the SageMaker Python SDK v3 (August 2026)

Blog: LLM optimization integration for Amazon SageMaker Python SDK

Benchmarking and recommendations previously required SageMaker AI Studio or direct Boto3 calls. They are now part of the SageMaker Python SDK v3 (version 3.17.0 and later), the same interface teams use for training and deployment, so the workflow fits into notebooks, CI pipelines, and MLOps workflows. You can benchmark a live endpoint against your workload traffic, generate ranked recommendations for a latency, throughput, or cost goal, and deploy the top-ranked configuration to a real-time endpoint in a single call.

The SDK also supports longer workflows. You can run recommendation jobs in parallel to compare serving frameworks such as LMI and vLLM, and you can resume from a completed job so that deployment happens in a different notebook or pipeline stage without re-running the benchmarks.

LoRA adapter support in inference recommendations (September 2026)

Developer guide: Get recommendations for models with LoRA adapters

Many teams use LoRA adapters to specialize one foundation model for several use cases. Each adapter kept in GPU memory reduces the number of concurrent requests the endpoint can serve, and the right balance depends on the instance type, the number of adapters, and the traffic profile.

Inference recommendations now accept adapters as Model Package Amazon Resource Names (ARNs), for adapters produced by SageMaker fine-tuning, or as S3 URIs, for adapters from external pipelines such as Hugging Face PEFT. SageMaker AI benchmarks combinations of adapter count and concurrency on real GPU infrastructure and returns ranked results for your workload. The output lists both formats for every adapter. The recommended deployment uses SageMaker Inference Components, with one component for the base model and one for each adapter, so you can add, update, or remove an adapter without disrupting the base model or the other adapters.

SLA-based concurrency search (September 2026)

Blog: Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI

Right-sizing an endpoint means finding the highest concurrency it can sustain without breaching latency or error targets. Until now, that meant picking concurrency levels by hand, running a sweep, checking the results against the service level agreement (SLA), and repeating with a narrower range whenever the model or instance changed.

Inference recommendations now include a search that finds that boundary directly. Set search_recipe to max-concurrency-under-sla and supply targets for any combination of maximum TTFT, maximum time per output token, maximum end-to-end latency, and maximum error rate. The benchmarking engine (AIPerf) starts from a broad range, narrows it based on which probes pass or fail, and converges in fewer iterations than a linear sweep. The result reports the largest passing concurrency (feasible_max) or, if no level qualifies, the smallest failing one (infeasible_min).

The search uses the same CreateAIWorkloadConfig and CreateAIBenchmarkJob APIs as concurrency sweeps. Optional parameters (concurrency_min, concurrency_max, search_max_iterations, and search_initial_points) bound the cost and speed up convergence.

Agent skill for optimized generative AI inference (October 2026)

Blog: New agent skill: Amazon SageMaker optimized generative AI inference for your coding agent

Engineers increasingly ask their coding agent which instance and container to use. Without grounded knowledge and access to real benchmarks, the agent can only guess, and a wrong instance choice shows up later as missed latency targets or wasted spend.

The aws-ai-ml skill, available through the Agent Toolkit for AWS, gives coding agents such as Kiro built-in SageMaker AI inference expertise. Engineers describe what they need in plain language, and the agent can do the following:

  • Find the right instance. Ranked deployment options with measured throughput, latency, and TTFT for a model in Amazon S3, SageMaker JumpStart, or the Hugging Face Hub.
  • Benchmark a live endpoint. A report from real load covering throughput, P50 and P99 latency, inter-token latency, and supported concurrency.
  • Compare runs. The percentage change on every metric, which shows whether a configuration change helped.

Answers are grounded in measured performance and delivered as SageMaker Python SDK v3 code that engineers review and run under their own credentials. Setup takes about 10 minutes. The skill works with any agent that supports the Model Context Protocol (MCP) and in SageMaker AI Studio JupyterLab spaces.

For the support team, one recommendation job that includes all four adapters, followed by a concurrency search against the 500 millisecond TTFT target, replaces the manual sweeps they ran in April. The engineer who owns it can start the job from a notebook or ask a coding agent to run it.

Fit more models onto each GPU

The support assistant does not run alone. An embedding model, a reranker, and a small classifier that routes tickets live in the same account, and each uses a fraction of a GPU. On a standard endpoint, each model copy still held a whole accelerator, so the team ran out of placement slots while GPU memory sat unused.

Inference Components in the SageMaker AI console (October 2026)

With Inference Components, you can host several models on one endpoint and size and scale each model separately. Until now, they were available only through the SDK and the AWS Command Line Interface (AWS CLI). Every create, resize, and scaling policy had to be scripted, and there was no single view of what each model was using.

In the SageMaker AI console, you can now create, view, update, and delete inference components:

  • Create and edit. Create an endpoint with inference components, add or delete components later, and change each component’s accelerators, CPU, memory, and copy count.
  • Scale each model. Set an automatic scaling policy for each component, down to zero copies, so an idle model stops holding accelerators.
  • Monitor. See alarms and per-component metrics on the same page.

The console calls the same SageMaker AI APIs as the SDK and CLI, so teams can start on the console and automate later, or manage scripted components visually.

GPU partitioning with NVIDIA Multi-Instance GPU (October 2026)

Blog: SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

GPU partitioning uses NVIDIA Multi-Instance GPU (MIG) technology to divide one physical GPU into as many as seven partitions, depending on the accelerator. Each partition has dedicated GPU compute and memory, with complete hardware isolation. You define the partition layout when you create an endpoint for inference components and choose a partition type for each component. Applications keep invoking models by inference component name, so client code does not change.

MIG isolates GPU compute and memory between partitions. Models on the same instance still share host CPU, host memory, networking, storage, and endpoint routing. You pay the standard rate for the full instance-hour, so the saving comes from placing more models on each instance, not from a lower hourly price.

Figure 2: Three small models on three GPUs, compared with the same models in partitions of one GPU

For the support team, the embedding model, the reranker, and the classifier now run in separate partitions of one GPU on a single endpoint, and the team manages every component, including the assistant and its adapters, from the console.

Serve models larger than one instance

The team’s next model is a reasoning model whose weights alone exceed the GPU memory of any single instance. They have also noticed that long, retrieval-heavy prompts slow token generation for every other user on the endpoint. For a model this size, they move the workload to HyperPod Inference, which gives them dedicated clusters and Kubernetes-native control. The support assistant stays on managed endpoints.

Managed NVIDIA Dynamo on HyperPod Inference (November 2026)

Blog: Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

Part 1 covered two building blocks for large models on HyperPod: disaggregated prefill and decode in Inference Operator v3.2, and KV-aware routing as part of the managed tiered KV cache. This launch adds multi-node serving and changes who operates the stack. NVIDIA Dynamo is an open source inference framework that provides multi-node sharded serving, disaggregated prefill and decode, and KV-aware routing. Teams could already run Dynamo on their own Kubernetes clusters, but they had to build and operate everything around it: placing all the nodes of a replica together, wiring Elastic Fabric Adapter (EFA) networking so prefill can hand its KV cache to decode at hardware speed, maintaining serving images, and recovering when one failed node takes down a multi-node replica.

With this launch, Dynamo runs as a managed serving engine on HyperPod. Dynamo makes the serving decisions: how the model is sharded, how prefill and decode are split, and where each request goes. HyperPod provides what those decisions depend on:

  • Capacity and networking. GPU capacity and EFA networking, provisioned and wired automatically.
  • Node recovery. Node health monitoring and replacement, with multi-node workers rescheduled automatically when a node fails.
  • Endpoint operations. Autoscaling, endpoint registration, TLS, and observability.
  • One install and upgrade path. The Inference Operator add-on installs Dynamo, the serving images, and the Grove and KAI schedulers as one unit and upgrades them together.

The capabilities Dynamo brings to HyperPod are:

  • Multi-node (sharded) serving. One copy of the model spans several nodes, so you can serve models that do not fit on any single instance.
  • Disaggregated prefill and decode. Prompt processing and token generation run on separate GPU pools, each on an instance type suited to its work, so a long prompt no longer slows responses for other users.
  • KV-aware routing. Each request goes to the worker that already holds its KV cache, which reduces TTFT.

There are three ways to start. Teams running Dynamo on self-managed Kubernetes can bring their existing DynamoGraphDeployment (DGD) manifests largely unchanged. Teams already on HyperPod Inference can update the add-on and add a short dynamoSpec block to an existing InferenceEndpointConfig, which HyperPod converts into a DGD, while existing endpoints keep serving. Teams new to both can create a cluster with the add-on and Dynamo enabled from the console. In each case, clients call a standard OpenAI-compatible endpoint, so client code does not change.

Managed Dynamo supports vLLM, SGLang, and TensorRT-LLM, and is available in the Regions where Amazon SageMaker HyperPod is available.

Figure 3: Managed NVIDIA Dynamo on HyperPod Inference

For the support team, the reasoning model runs as one sharded replica across several nodes, with separate prefill and decode pools. When a node fails, HyperPod replaces it and reschedules the replica without a page to the on-call engineer.

Send each request to a GPU that can serve it

The support assistant now serves customers in North America and Europe, so the team runs endpoints and HyperPod clusters in two Regions. During the US morning peak, requests queue in the US Region while GPUs the team already pays for sit idle in Europe. Some European traffic must also stay in Europe.

Routing decides whether capacity a team owns does useful work. It happens at four levels, and the launches below cover one level each: which instance serves a request inside an endpoint, which pod inside a HyperPod cluster, which cluster across a fleet, and which Regional endpoint across Regions and accounts.

Figure 4: Routing across Regions and inside each endpoint or cluster, on both deployment paths

KV-cache-aware routing for endpoints (November 2026)

Blog: Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Applications with long prompts or multi-turn conversations repeat a large prefix across requests. A model server can reuse the matching KV cache, but only if the next request reaches the instance that holds it. Prefix-aware routing, covered in part 1, fingerprints the start of each prompt and sends matching prompts to the same instance. Because it works from request history, its view of each instance’s cache becomes less accurate as the model server adds and evicts entries.

KV-cache-aware routing uses the cache state the model container reports. The container publishes cache creation and eviction events to a central cache index, and SageMaker AI routes each request to an eligible instance that holds the longest matching cached prefix. This raises cache reuse for long-context prompts, multi-turn conversations, and workloads that share dynamic prompt prefixes.

It works with single-model endpoints and endpoints that use inference components, including LoRA adapters. It supports InvokeEndpoint, InvokeEndpointWithResponseStream, and the OpenAI-compatible API for Completions and Chat Completions payloads. You turn it on in the endpoint configuration. The first release requires an AWS-verified model container that implements the cache-event, tokenization, metrics, and OpenAI-compatible interfaces. You can track KV cache hit rate in detailed inference observability.

Which one to use. Prefix-aware routing works with any container and needs no container changes. KV-cache-aware routing needs a verified container and tracks the actual cache, which helps most when prefixes vary or caches turn over quickly.

Inference Gateway for HyperPod (September 2026)

Blog: Introducing Amazon SageMaker HyperPod Inference Gateway

Inside a HyperPod cluster, capacity and demand rarely line up. One pod can be partway through a long-context request with its KV cache nearly full, while the next pod sits idle or already holds the prompt prefix or LoRA adapter the next request needs. Without visibility into that GPU state, new requests queue behind busy pods, and teams add GPUs to protect latency instead of using the ones they have. Each model also had its own endpoint, so clients had to track one endpoint per model.

The Inference Gateway is a GPU-aware router that installs with the HyperPod Inference Operator add-on and gives each cluster one OpenAI-compatible endpoint for all of its models. For each request, it reads live signals from every model pod and picks the best one, with weights you can tune per workload:

  • KV cache utilization. Avoids pods whose cache is nearly full.
  • Queue depth and running requests. Avoids pods with a backlog and balances active work.
  • Prefix cache hit rate. Prefers pods likely to hold the prompt prefix.
  • LoRA adapter residency. Prefers pods that already hold the requested adapter, which avoids swap latency.

Compared with round-robin routing, it reduced P99 TTFT by up to 97% on mixed GPU generations and by up to 98% under bursty traffic. It is available in all Regions where SageMaker HyperPod is available.

Cross-Region Inference Routing in Amazon SageMaker AI

Blog: Keep inference running through regional capacity shortfalls with cross-Region inference routing in Amazon SageMaker AI

On managed endpoints, the same idea takes the form of the Cross-Region Inference Routing, managed routing that sends real-time and asynchronous requests across Regional endpoints a team has already deployed, in the same account or in different accounts. You group endpoints that serve the same model under a routing target, and the application invokes the target instead of a specific Regional endpoint. The router does not add capacity, replicate models, or deploy endpoints, and the existing invocation path keeps working.

The router replaces the failover logic teams build themselves today:

  • One invocation path. SageMaker tries the local endpoint first and moves traffic to another registered endpoint when the local one cannot serve a request.
  • One routing policy. The policy combines endpoint metrics with service-side capacity signals, including insufficient capacity errors that customers cannot observe on their own.
  • Private connectivity. Cross-Region traffic runs over AWS PrivateLink with a managed cross-account authorization model, so routing to another Region or account does not need internet or NAT gateways.
  • Overflow and residency guardrails. The endpoint owner can cap how much cross-Region traffic a fallback endpoint accepts, and the application can restrict which Regions may serve a request.

Figure 5: Cross-Region Inference Router request flow

Keep endpoints tuned after launch

Two months after the move, the team adds retrieval to the support assistant, and the average prompt length roughly doubles. TTFT is now higher than it was at launch, but nothing has failed, so every alarm stays green. When the team finds a better configuration, they will need to roll it out on large GPU instances that are hard to get more of.

Continuous performance monitoring (November 2026)

Blog: Amazon SageMaker AI now detects latency drift, idle GPUs, and missed optimizations on your inference endpoints

Amazon SageMaker AI now continuously monitors your generative AI inference endpoints and tells you when a change could improve performance or lower cost. Monitoring is on by default for SageMaker AI inference endpoints, with no setup required, and you can opt out of endpoints. SageMaker AI looks only at operational signals: latency, GPU utilization, and the endpoint’s serving configuration. It does not read your requests or responses. When a finding clears its confidence threshold, a notification appears on the AWS Management Console, and the same message arrives by email for teams that work in notebooks and the CLI. The recommended action opens the SageMaker AI agentic experience in SageMaker AI Studio with the finding already filled in, so the analysis, the comparison, and the deployment change happen in one conversation. Nothing deploys until you apply it, and you can turn monitoring off at any time.

There are three types of findings:

  • Latency drift after a traffic change. When a change such as added retrieval lengthens prompts, TTFT rises. The finding quantifies the regression week over week and recommends re-running inference recommendations against recent traffic to find a configuration that brings TTFT back down.
  • A configuration gap present since deployment. The endpoint was never well tuned for its workload, and the finding points to the change that would close the gap.
  • Idle GPU capacity. When capacity stays unused week after week, the finding flags a chance to right-size the endpoint and reduce cost.

In-place rolling model updates (November 2026)

A standard rolling update replaces an endpoint one batch at a time, and every batch runs on newly provisioned instances. On a large GPU fleet, that means finding new capacity at every step, and an update can fail partway through when the next batch of instances is not available. Each released instance must also finish cleanup before it can be used again, which slows the rollout.

SageMaker AI now updates models in place. It provisions one small batch of additional instances at the start, shifts traffic to that batch, and then updates the instances it frees, reusing them to roll the change across the rest of the fleet. The update still rolls out in stages, but it needs new capacity only once. For a fleet of 20 instances with a batch size of 2, the update acquires 2 new instances instead of 20. Customers do not need to change anything. As with container caching in part 1, the improvement applies automatically to rolling model updates.

Figure 6: A rolling update acquires new instances at every step. An in-place rolling update acquires one batch at the start and reuses the rest of the fleet

For the support team, a Drift Watch finding quantifies the TTFT regression and opens a new recommendation job against recent traffic. They roll out the updated model with one batch of extra instances, instead of finding new GPU capacity for every instance in the fleet.

Summary: 13 launches, continuing from part 1

The numbering continues from the 13 launches in part 1.

14. Inference recommendations in SageMaker AI Studio (July 2026). A guided, low-code workflow for ranked, benchmarked configurations, with deployment from Studio.

15. Inference recommendations in the SageMaker Python SDK v3 (August 2026). Benchmark, recommend, and deploy from a notebook or pipeline, with parallel and resumable jobs.

16. LoRA adapter support in inference recommendations (September 2026). Benchmarks adapter count against concurrency and deploys adapters as inference components.

17. SLA-based concurrency search (September 2026). Finds the highest concurrency that meets TTFT, latency, and error-rate targets.

18. Agent skill for optimized generative AI inference (October 2026). Instance selection, benchmarking, and run comparison from MCP-compatible coding agents.

19. Inference Components in the SageMaker AI console (October 2026). Create, scale, and monitor each model on a multi-model endpoint without scripts.

20. GPU partitioning with NVIDIA MIG (October 2026). Up to seven isolated partitions per GPU, assigned per inference component.

21. Managed NVIDIA Dynamo on HyperPod Inference (November 2026). Multi-node and disaggregated serving, with HyperPod operating the platform underneath.

22. KV-cache-aware routing (November 2026). Routes each request to the instance that holds the longest cached prefix, based on cache events from the container.

23. Inference Gateway for HyperPod (September 2026). One endpoint per cluster with GPU-aware routing, and up to 97% lower P99 TTFT than round-robin.

24. Cross-Region Inference Router. Routes across Regional and cross-account endpoints over PrivateLink, with overflow and residency guardrails.

25. Continuous performance monitoring. Detects latency drift, configuration gaps, and idle capacity, with a guided fix in SageMaker AI Studio.

26. In-place rolling model updates (November 2026). Rolls out model updates with one batch of additional instances at the start, then reuses existing instances across the fleet.

By the end of the year, the support team serves four product lines from one base model, runs its small models on shared GPUs, serves a reasoning model that no single instance can hold, and routes traffic across two Regions with residency controls. It did not need to build its own benchmarking harness, failover logic, or drift monitoring to get there.

Part 1 argued that advantage in AI inference comes from operating an efficient inference stack. These 13 launches apply that argument to the months after launch day. The AWS Experience-Based Acceleration program can help your team put them to work by benchmarking your workloads and configuring your inference stack against your SLAs and cost targets.

What is next

On managed endpoints, we plan to extend in-place rolling updates in early 2027 with scale-in-first updates, which remove the need for any additional instances during a model update.

Documentation and getting-started guides for every capability in this post are in the Amazon SageMaker AI Developer Guide. Start with the section that matches your current problem. If your traffic has changed since you deployed, re-run inference recommendations. If you are short on placement slots, move small models onto partitioned GPUs. If requests queue while capacity you own sits idle, register your clusters and endpoints for cross-Region routing.

References


About the authors

Aryaman Kothari

Aryaman is a Senior Product Manager at Amazon SageMaker AI, where he leads AI inference on Amazon SageMaker HyperPod, helping enterprises run AI models at scale for production workloads. A Chicago Booth graduate, Aryaman previously led product and go-to-market for Quantiphi’s Conversational AI business. Outside work, he enjoys playing soccer, making chai, and hiking.

Apoorva Chandra

Apoorva is a Senior Product Manager on the Amazon SageMaker AI Inference team, where she focuses on the endpoint and hosting experience. She works with customers deploying generative AI models in production to make accelerated compute easier to obtain, more cost-efficient to run, and more predictable to operate. Outside of work, Apoorva enjoys hiking, exploring coffee shops, and spending time with friends.

Vivek Singh

Vivek Singh is a Senior Product Manager at AWS, where he leads product for generative AI inference optimization and works with customers who run applications in production. Outside work, he enjoys taking long walks, gardening, and cooking.

Kareem Syed-Mohammed

Kareem is a Principal Product Manager at AWS. He focuses on enabling generative AI model development and governance on Amazon SageMaker HyperPod. Prior to this, at AWS Quick Sight, he led embedded analytics and developer experience. In addition to Quick Sight, he has been with AWS Marketplace and Amazon retail as a Product Manager. Kareem started his career as a developer for call center technologies, Local Expert and Ads for Expedia, and management consultant at McKinsey.

Vineet Sharma

Vineet is a Senior Product Marketing Manager, Tech at AWS, focused on Amazon SageMaker AI. He specializes in go-to-market strategy, product launches, and translating complex AI and ML services into compelling customer value. He is passionate about creating great customer experiences through clear, impactful messaging.

AI Generated Robotic Content

Recent Posts

Model Compression Beyond Quantization

Overview This post is divided into three parts; they are: • Pruning and Sparsity •…

2 mins ago

Visualizing Information Entropy in Language Models

In this article, you will learn how information entropy and cross-entropy loss connect classical information…

2 mins ago

The Blueprint: How Kult built a gaming companion that actually knows your game

Welcome to The Blueprint, our ongoing series where we highlight how Google Cloud customers are…

2 mins ago

Dudes Online Are So Mad That ‘Grand Theft Auto 6’ Makes Them Respect Women

Much to the dismay of right-wing streamers, being a thoughtful boyfriend is key to the…

1 hour ago

Brain researchers afflict LLMs with ‘injuries’ similar to those of stroke victims to unpack AI language mechanisms

For more than 150 years, doctors and scientists have mapped the functions of various brain…

1 hour ago