Viggle Remake is a beast

https://huggingface.co/Viggle/Viggle-Remake

For this video I used their included MCP. However, it’s currently set up to run with diffusers which means that no consumer GPU works, despite them offering ComfyUI compatible loras along with custom nodes. Fortunately, the MCP was built with using custom workers in mind. So I (read: Claude) set that up so the MCP uses a custom comfy worker. Then your favorite agent can do everything.

I had Claude use the Codex CLI to generate the necessary edited frames and present them to me for approval before it sent it off to comfy for video generation. I’m sure it’s possible to set it up so that it uses a local edit model too, so that’s the next thing to try.

For the audio, I first used SAM-Audio Large through audio.cpp to separate RDJ’s inner monologue from the background noise. Lighter options like MelBandRoFormer were not producing good results for this. Then Fish Audio S2 Pro was used for the new voice, using translations carefully made to fit a similar speech length. Claude also came in handy here for rapid testing of different TTS models and then fitting the audio into correct timings. Subtitles were done by hand in Davinci and used for making the new voice fit.