SageMaker AI Adds SOCI-Indexed Images: Lazy-Loaded Training Containers, AWS Cites Up to 90% Less GPU Idle Time During Image Pulls

Published 2026-10-29 · AI Daily — AI-assisted deep research, methodology & disclosure

On October 29, 2026, AWS announced that Amazon SageMaker AI training and processing jobs now support SOCI (Seekable OCI) indexed container images. AWS says this starts training containers in seconds and cuts GPU idle time during image pulls by up to 90%, with no impact on training performance beyond a one-time first-epoch cost. The 90% is an "up to" figure from AWS's own benchmarks, not a guarantee for every job.

The problem comes before the job runs. SageMaker AI first downloads the whole container image to the instance, then decompresses and writes every layer. Deep learning images routinely exceed 10 GB, so this adds minutes to every job, and you pay for that time. The impact is largest for hyperparameter sweeps of hundreds of jobs, quick experiments and short processing jobs, where the download can dominate total runtime.

SOCI is an open source lazy-loading technology. A SOCI index is metadata that maps every file inside an image. It is stored in Amazon ECR as an OCI-compliant artifact linked to the original image, so the image digest and any existing signatures stay valid. When a job's image has an index, SageMaker AI fetches only the files needed to start the container and streams the rest in the background as the code runs. The service checks for an index automatically: with one, it lazy-loads; without one, it downloads as before; and if the index cannot be read or lazy loading fails, the job falls back to a standard download and keeps running. Adding an index therefore cannot stop a job from starting.

There are two ways to adopt it. One is to use an AWS Deep Learning Container that is already indexed by changing the image tag to its -soci variant; AWS's launch blog lists PyTorch Training, TensorFlow Training, vLLM and SGLang, with arm64 variants for some frameworks. The other is to add an index to your own image, which needs a Linux host with containerd, the SOCI snapshotter and either Finch or nerdctl, and no image rebuild. Job definitions, entry points and data channels do not change. Two conditions matter: an image you build FROM a Deep Learning Container needs its own index because the base image's index does not carry forward, and the image must sit in a private ECR repository. The feature is available in all Regions where SageMaker AI is available, through the APIs, CLIs, SDKs and Console.

The launch blog reports AWS's own measurements. The tests used eight images from 1.8 to 35.4 GB compressed, across instance types from ml.g4dn to ml.p5, in US West (Oregon) and US East (N. Virginia), with medians of repeated runs. A 1.8 GB image went from 48 seconds to 3 to 11 seconds, a 13.2 GB image from 3 minutes 42 seconds to about 6 seconds, and a 35.4 GB image from 7 minutes 17 seconds to 3 to 11 seconds. The same 13.2 GB image across 13 instance types and sizes took 3 minutes 16 seconds to 4 minutes 20 seconds with a standard download and 6 to 8 seconds with lazy loading. The blog's own summaries state the speedup inconsistently, as about 10x, 8x, and 8x to 46x, while its table runs from about 8x to about 38x, so rely on the seconds.

Training itself carries a one-time cost. AWS compared ResNet-50 on Imagenette (9,469 images) and full-parameter fine-tuning of Qwen2.5-0.5B on 51,760 examples. From the second epoch on, epoch time was the same, the first epoch took 3 to 16 seconds longer while first-touched files were fetched, and each job finished about 1 to 6 minutes sooner overall. If the entry point reads most of the image before training, for example scanning, checksumming or preloading, the gain shrinks. In AWS's tests, torch.compile and a just-in-time built CUDA extension added 7 to 15 seconds to the first epoch, and a 6 GB model baked into the image added 2 to 3.5 minutes when its weights streamed at load time, yet the job still finished 2 to 2.5 minutes sooner.

These are AWS measurements on two kinds of workload. The first-epoch cost depends on your image and on what your code touches at startup, so measure on your own jobs.

Sources

FAQ

What does a SOCI-indexed image change, and what does "up to 90%" mean?

A job no longer waits for the whole image. It fetches only the files needed to start the container and streams the rest in the background. The 90% is an "up to" cut in GPU idle time during image pulls from AWS's benchmarks, not a promise for every job, and training performance is unaffected apart from a one-time first-epoch cost.

How do you adopt it, step by step?

First, keep the image in a private ECR repository. Second, either switch to an AWS Deep Learning Container tag ending in -soci, or index your own image on a Linux host with containerd, the SOCI snapshotter and Finch or nerdctl. Third, reference that image in the job, with no change to the job definition.

What costs and limits does it have?

In AWS's tests the first epoch took 3 to 16 seconds longer, and each job still finished about 1 to 6 minutes sooner overall. The gain shrinks if the entry point reads most of the image before training, and an image built on a Deep Learning Container needs its own index. These are two workloads, so measure your own jobs.