Categories: FAANG

MM-Ego: Towards Building Egocentric Multimodal LLMs

This research aims to comprehensively explore building a multimodal foundation model for egocentric video understanding. To achieve this goal, we work on three fronts. First, as there is a lack of QA data for egocentric video understanding, we automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data. This is one of the largest egocentric QA datasets. Second, we contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models’ ability in recognizing and…

Incentivizing Temporal-Awareness in Egocentric Video Understanding Models

Multimodal large language models (MLLMs) have recently shown strong performance in visual understanding, yet they often lack temporal awareness, particularly in egocentric settings where reasoning depends on the correct ordering and evolution of events. This deficiency stems in part from training objectives that fail to explicitly reward temporal reasoning and…

July 10, 2026

In "FAANG"

ARMOR: Egocentric Perception for Humanoid Robot Collision Avoidance and Motion Planning

Humanoid robots have significant gaps in their sensing and perception, making it hard to perform motion planning in dense environments. To address this, we introduce ARMOR, a novel egocentric perception system that integrates both hardware and software, specifically incorporating wearable-like depth sensors for humanoid robots. Our distributed perception approach enhances…

February 14, 2025

In "FAANG"

Novel framework can create egocentric human demonstrations for imitation learning

One of the most promising approaches to teaching robots how to complete manual tasks such as cleaning dishes or preparing food is known as imitation learning. End-to-end imitation learning typically entails training a deep learning algorithm on raw videos, images and/or motion capture data of humans completing manual tasks.

November 30, 2024

In "AI/ML News"

AI Generated Robotic Content