Categories: FAANG

Careful With That Scalpel: Improving Gradient Surgery With an EMA

Beyond minimizing a single training loss, many deep learning estimation pipelines rely on an auxiliary objective to quantify and encourage desirable properties of the model (e.g. performance on another dataset, robustness, agreement with a prior). Although the simplest approach to incorporating an auxiliary loss is to sum it with the training loss as a regularizer, recent works have shown that one can improve performance by blending the gradients beyond a simple sum; this is known as gradient surgery. We cast the problem as a constrained minimization problem where the auxiliary objective is…
AI Generated Robotic Content

Recent Posts

Model Compression Beyond Quantization

Overview This post is divided into three parts; they are: • Pruning and Sparsity •…

3 hours ago

Visualizing Information Entropy in Language Models

In this article, you will learn how information entropy and cross-entropy loss connect classical information…

3 hours ago

Amazon SageMaker Inference: 2026 launches in review, part 2

Once a model is in production, the questions change. Teams want to know which configuration…

3 hours ago

The Blueprint: How Kult built a gaming companion that actually knows your game

Welcome to The Blueprint, our ongoing series where we highlight how Google Cloud customers are…

3 hours ago

Dudes Online Are So Mad That ‘Grand Theft Auto 6’ Makes Them Respect Women

Much to the dismay of right-wing streamers, being a thoughtful boyfriend is key to the…

4 hours ago

Brain researchers afflict LLMs with ‘injuries’ similar to those of stroke victims to unpack AI language mechanisms

For more than 150 years, doctors and scientists have mapped the functions of various brain…

4 hours ago