Preference Is Not Intervention: The Structure and Stability Boundaries of Reader-Specific Evidence Utility
2608.17781

Authors

Shi Zhou

Abstract

ML systems increasingly condition decisions on downstream model identity, but this is useful only if model-specific differences form reusable structure rather than input-local interactions. We test this in retrieval-augmented generation (RAG), where evidence utility can be measured under controlled interventions.

Holding query, evidence, task, scoring, and intervention fixed, nine readers disagree on effect sign in 33% of jointly affected cells; reader$\times$query interaction explains 29.8% of utility variance versus an 8.4% permutation null; and self-selected evidence improves F1 by $+0.031$ ($t=3.39$). We then ask the sharper question: which components of this heterogeneity are stable reader properties across queries? Separating three measurable objects---evidence activity, ordinal preference, and conditional signed direction---we find ordinal reader geometry stable across four independent settings (split-half $ρ=0.60$--$0.83$): leave-one-out interventions, PRISM preferences, RAMDocs, and RAGuard. Signed geometry is task-bounded: weak in open-ended QA (0.14, 0.35), especially for misleading and irrelevant evidence, but strong in binary fact-checking (0.75) with no significant ordinal gap, though still below its sparsity-matched ceiling.

Sparsity, decoding noise, and metric artifacts do not explain the main ordinal--signed gap. Finally, stable ordinal similarity fails to predict cross-reader intervention transfer (oracle-distance $ρ=-0.27$; regret reliability $-0.28$).

Reader-specific utility exists, but preference is not intervention: stable ranking similarity does not license transfer of help/harm decisions.

Resources

Ray graphicRay graphicRay graphicRay graphic

Stay in the loop

Every AI paper that matters, free in your inbox daily.

Details

  • takara.ai
  • Custom AI and machine learning from the Frontier Research Team.
  • © 2026 takara.ai Ltd
  • Content is sourced from third-party publications.
Ray graphicRay graphicRay graphic