Categories: FAANG

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training (WRAP) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the…
AI Generated Robotic Content

Recent Posts

Meta ads is now available in AWS Quick

Marketers have never had more data to act on. Take an apparel brand that advertises…

19 hours ago

The Outages Are Just Getting Started

AI is forcing companies to patch their systems faster as attackers race to exploit newly…

20 hours ago

AI may not work like the human brain after all, scientists discover

AI may not think like the human brain after all, according to a striking new…

20 hours ago

Can teams of AI agents recreate the chain reaction of human decisions?

At the start of the COVID-19 pandemic in 2020, people hoarded toilet paper. Australians alone…

20 hours ago

Combatting adversarial distillation

OpenAI’s layered approach to adversarial distillation combines model training, safeguards, detection, and enforcement to protect…

2 days ago

7 Ways AI Agents Fail in Production (and How to Catch Each One)

Three events that look alike need separate names.

2 days ago