SageMaker GPU partitioning with NVIDIA MIG: up to 7 models per GPU

Amazon SageMaker Inference now supports GPU partitioning with NVIDIA Multi-Instance GPU (MIG). One physical GPU splits into up to seven isolated partitions, and each partition can serve its own model. AWS announced it on November 6. If you pay for a whole GPU to host a small model that sits half idle, this is the change to look at. The fine print on cost matters, though.

What actually happened

GPU partitioning works on the P4d (A100), P5 (H100), G7 and G7e instance families. An A100 or H100 carves into up to seven partitions, the G7e into four, and the G7 into two. Each partition gets its own compute slice and its own block of GPU memory, with hardware isolation.

There is no new API. You add AcceleratorPartitionLayouts to the production variant in CreateEndpointConfig, and each inference component asks for a profile such as mig-3g.40gb with AcceleratorPartitionType. InvokeEndpoint does not change.

AWS benchmarked one mig-3g.40gb slice of an H100 against one whole A100-40GB, both serving Llama-3.1-8B-Instruct. The slice was faster in every test. On cost, AWS says it was 18-34% cheaper per million output tokens, above a crossover point: concurrency 64 for 100-token prompts, concurrency 4 for 8,000-token prompts. Below that crossover, the whole A100 was cheaper. AWS calls this "a case-by-case comparison, not a verdict."

The limits:

• You still pay for the full instance. Partitioning raises use. It does not lower the hourly price.
• It works only on real-time endpoints that use inference components. Serverless, async and heterogeneous instance pools are out.
• Layouts are fixed at creation. A new layout means a new endpoint and a traffic migration.
• It is in seven Regions, including US East (N. Virginia), US East (Ohio) and US West (Oregon).

Why it matters for your business

Most small teams that self-host models run a few small ones: an embedder, a classifier, a guardrail model. Today each one often gets its own GPU. Packing them onto partitions of one endpoint can cut the number of instances you pay for.

The crossover is the catch. Short prompts at low traffic do not win. Before you migrate, pull a week of request logs and measure your real concurrency and prompt length. If you are below the crossover, a hosted API may still beat any GPU you rent.

Sources: AWS Machine Learning Blog, SageMaker Inference now supports Multi-Instance GPU.