KEEP-Bench: Evaluating Evidence Preservation in Context Compression
arXiv, 2026
KEEP-Bench evaluates whether compressed contexts retain the evidence needed for faithful reasoning. Its 4,102 cases test three challenges: preserving alternative interpretations, retaining the conditions attached to claims, and keeping minority perspectives alongside more extensively supported views. The benchmark checks individual evidence propositions directly, exposing information loss that downstream answer accuracy can miss. Experiments reveal substantial preservation failures even in strong language models. The accompanying Evidence Preservation Return metric measures evidence recall relative to inference computation, highlighting the need for compressors that retain essential information efficiently.