Categories: Image

I trained the missing encoder for YuE2, so we can all bring our own music into it

YuE2 is an impressive open music model. Give it a style prompt and lyrics, and it can make a complete song. Under the hood, it generates “semantic tokens” that are then turned into audio. The trouble is, the encoder that converts existing recordings into those same tokens was never released. That leaves you without a way to bring your own music into the process and fine-tune it. Until now!

Scripts and tokenizer weights are included in the repo.

The trick was this:
The model can teach itself. Every song YuE2 generates comes with the exact tokens that produced it, which is a labeled example nobody had to hand-label. I generated a few thousand songs across every genre I could name, trained a small encoder on those pairs, then adapted it to real recordings by letting YuE2’s own decoder grade the encoder: if the tokens rebuilt the real audio, they were right. No token labels for real music were ever needed.

submitted by /u/thatisnotmychapstick
[link] [comments]

AI Generated Robotic Content

Share
Published by
AI Generated Robotic Content
Tags: ai images

Recent Posts

Serving Scikit-LLM Pipelines as REST APIs with FastAPI

In this article, you will learn how to serve a Scikit-LLM zero-shot text classification pipeline,…

3 hours ago

Build memory-aware AI agents with Agentic Retrieve on Amazon Bedrock Managed Knowledge Base

Users of a knowledge-grounded application expect it to remember them, not only to search well.…

3 hours ago

New AI agent observability and governance across BigQuery and Looker

As enterprises move AI agents from experimental prototypes into production, data and platform teams are…

3 hours ago

State of the Sub – 1M Members Reached!

Howdy! This has been an interesting ride to get to this point over the years.…

1 day ago

AI discovers a hidden heart signal that could predict sudden death

Scientists have uncovered a hidden warning sign in routine heart scans that could predict sudden…

1 day ago

Weight-Only Quantization

Overview This post is divided into five parts; they are: • Asymmetric Affine Quantization •…

2 days ago