A key algorithm for understanding the world is material segmentation, which assigns a label (metal, glass, etc.) to each pixel. We find that a model trained on existing data underperforms in some settings and propose to address this with a large-scale dataset of 3.2 million dense segments on 44,560 indoor and outdoor images, which is …
Read more “A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing”
Including memory banks in a natural language processing architecture increases model capacity by equipping it with additional data at inference time. In this paper, we build upon kNN-LM, which uses a pre-trained language model together with an exhaustive kNN search through the training data (memory bank) to achieve state-of-the-art results. We investigate whether we can …
Read more “Regularized Training of Nearest Neighbor Language Models”
The practical success of overparameterized neural networks has motivated the recent scientific study of interpolating methods, which perfectly fit their training data. Certain interpolating methods, including neural networks, can fit noisy training data without catastrophically bad test performance, in defiance of standard intuitions from statistical learning theory. Aiming to explain this, a body of recent …
Read more “Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting”
What components are needed for building learning algorithms that leverage the structure and properties of graphs?
Understanding the building blocks and design choices of graph neural networks.
After five years, Distill will be taking a break.
Weights in the final layer of common visual models appear as horizontal bands. We investigate how and why.
Reprogramming Neural CA to exhibit novel behaviour, using adversarial attacks.
Tweet Tweet Share Share Last Updated on August 19, 2022 Sponsored Post This is your last chance to sign up for Stefan Krawczyk’s exclusive live cohort, starting next week (August 22nd). We already have students enrolled from Apple, Amazon, Spotify, Nubank, Workfusion, Glassdoor, ServiceNow, and more. Stefan Krawczky has spent the last 15+ years …
Read more “Last call: Stefan Krawcyzk’s ‘Mastering MLOps’ Live Cohort”
Tweet Tweet Share Share Last Updated on August 15, 2022 The weights of artificial neural networks must be initialized to small random numbers. This is because this is an expectation of the stochastic optimization algorithm used to train the model, called stochastic gradient descent. To understand this approach to problem solving, you must first understand …
Read more “Why Initialize a Neural Network with Random Weights?”