Categories: FAANG

Improve training time of distributed machine learning with NCCL Fast Socket

Large Machine Learning (ML) models – such as large language models, generative AI, and vision models – are dramatically increasing the number of trainable parameters and are achieving state-of-the-art results. Increasing the number of parameters results in the model being too large to fit on a single VM instance thus demands distributed compute to spread the model across multiple nodes. Google Kubernetes Engine (GKE) has built-in support for NCCL Fast Socket, to help improve the time to train large ML models with distributed, multi-node clusters.

Enterprises are looking for faster and cheaper performance to train their ML models. With distributed training, communicating gradients across nodes is a performance bottleneck. Optimizing inter-node latency is critical to reduce training time and costs. Distributed training uses collective communication as a transport layer over the network between the multiple hosts. Collective communication primitives such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, and point-to-point send and receive are used in distributed training in Machine Learning. 

The NVIDIA Collective Communication Library (NCCL) is commonly used by popular ML frameworks such as TensorFlow and PyTorch. It is a highly optimized implementation for high bandwidth and low latency between NVIDIA GPUs. Google developed a proprietary version of NCCL called NCCL Fast Socket to optimize performance for deep learning on Google Cloud.

NCCL Fast Socket uses a number of techniques to achieve better and more consistent NCCL performance.

  • Use of multiple network flows to attain maximum throughput. NCCL Fast Socket introduces additional optimizations over NCCL’s built-in multi-stream support, including better overlapping of multiple communication requests.

  • Dynamic load balancing of multiple network flows. NCCL can adapt to changing network and host conditions. With this optimization, straggler network flows will not significantly slow down the entire NCCL collective operation.

  • Integration with Google Cloud’s Andromeda virtual network stack.This increases overall network throughput by avoiding contentions in virtual machines (VMs).

We tested (NVIDIA NCCL tests) the performance of NCCL Fast Socket vs NCCL on various machine shapes with 2 node GKE clusters.

The following chart shows the results. For each machine shape, the NCCL performance without Fast Socket is normalized to 1. In each case, using NCCL Fast Socket demonstrated increased performance in a range of 1.3 to 2.6 times faster internetwork communication speed.

As a built-in feature, GKE users can take advantage of NCCL Fast Socket without changing or recompiling their applications, ML frameworks (such as TensorFlow or PyTorch), or even the NCCL library itself. To start using NCCL Fast Socket, create a node pool that uses the plugin with the --enable-fast-socket and --enable-gvnic flags. You can also update an existing node pool using gcloud container node-pools update.

code_block
[StructValue([(u’code’, u’gcloud container node-pools create NODEPOOL_NAME \rn –accelerator type=ACCELERATOR_TYPE, count=ACCELERATOR_COUNT \rn –machine-type=MACHINE_TYPE \rn –cluster=CLUSTER_NAME \rn –enable-fast-socket \rn –enable-gvnic’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e2c9d3bc110>)])]

To achieve better network throughput with NCCL, Google Virtual NICs (gVNICs) must be enabled when creating VM instances. For detailed instructions on how to use gVNICs, please refer to the gVNIC guide

To verify that NCCL Fast Socket has been enabled, view the kube-system pods:

code_block
[StructValue([(u’code’, u’kubectl get pods -n kube-system’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e2c9d3bc750>)])]

And the output should b similar to:

code_block
[StructValue([(u’code’, u’NAME READY STATUS RESTARTS AGErnfast-socket-installer-qvfdw 2/2 Running 0 10mrnfast-socket-installer-rtjs4 2/2 Running 0 10mrnfast-socket-installer-tm294 2/2 Running 0 10m’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e2c959dca10>)])]

To learn more visit GKE NCCL Fast Socket documentation. We look forward to hearing how NCCL Fast Socket improves your ML Training experience on GKE.

AI Generated Robotic Content

Recent Posts

Flux Kontext Dev is pretty good. Generated completely locally on ComfyUI.

You can find the workflow by scrolling down on this page: https://comfyanonymous.github.io/ComfyUI_examples/flux/ submitted by /u/comfyanonymous…

21 hours ago

7 AI Agent Frameworks for Machine Learning Workflows in 2025

Machine learning practitioners spend countless hours on repetitive tasks: monitoring model performance, retraining pipelines, data…

21 hours ago

A Gentle Introduction to Attention Masking in Transformer Models

This post is divided into four parts; they are: • Why Attention Masking is Needed…

21 hours ago

10 Essential Machine Learning Key Terms Explained

Artificial intelligence (AI) is an umbrella computer science discipline focused on building software systems capable…

21 hours ago

From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts

With advances in generative AI, there is increasing work towards creating autonomous agents that can…

21 hours ago

Tailor responsible AI with new safeguard tiers in Amazon Bedrock Guardrails

Amazon Bedrock Guardrails provides configurable safeguards to help build trusted generative AI applications at scale.…

21 hours ago