Next generation material that adapts to its history
Inspired by living systems, researchers have developed a new material that changes its electrical behavior based on previous experience, effectively giving it a basic form of adaptive memory. Such adaptive materials could play a vital role in the next generation of medical and environmental sensors, as well as in soft robots or active surfaces.
Serving transformer language models with high throughput requires caching Key-Values (KVs) to avoid redundant computation during autoregressive generation. The memory footprint of KV caching is significant and heavily impacts serving costs. This work proposes to lessen these memory requirements. While recent work has largely addressed KV cache reduction via compression…