Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction
This study focuses on Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with synchronized audio from text, with both modalities aligned to the text conditions. Despite progress in joint audio-video training, two critical challenges remain: (1) text conditioning is a bottleneck—shared captions (TV=TA) trigger modal interference, while a gap persists between dense training captions …
Read more “Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction”