I trained the missing encoder for YuE2, so we can all bring our own music into it

YuE2 is an impressive open music model. Give it a style prompt and lyrics, and it can make a complete song. Under the hood, it generates “semantic tokens” that are then turned into audio. The trouble is, the encoder that converts existing recordings into those same tokens was never released. That leaves you without a way to bring your own music into the process and fine-tune it. Until now!

Scripts and tokenizer weights are included in the repo.

The trick was this:
The model can teach itself. Every song YuE2 generates comes with the exact tokens that produced it, which is a labeled example nobody had to hand-label. I generated a few thousand songs across every genre I could name, trained a small encoder on those pairs, then adapted it to real recordings by letting YuE2’s own decoder grade the encoder: if the tokens rebuilt the real audio, they were right. No token labels for real music were ever needed.