Evaluating Sample Utility for Data Selection by Mimicking Model Weights
Foundation models are trained on large-scale web-crawled datasets, which often contain noise, biases, and irrelevant information. This motivates the use of data selection techniques, which can be divided into model-free variants — relying on heuristic rules and downstream datasets — and model-based, e.g., using influence functions. The former can be expensive to design and risk …
Read more “Evaluating Sample Utility for Data Selection by Mimicking Model Weights”