$ECUAS_n$: A family of metrics for principled evaluation of uncertainty-augmented systems
A new paper on arXiv (2605.20490v2) introduces ECUAS_n, a family of metrics designed for principled evaluation of uncertainty-augmented (UA) systems—systems that output both predictions and uncertainty scores. Current evaluation methods use separate metrics for predictions and uncertainties, fixed rejection costs, or coverage-risk curves, which the authors argue are inadequate for assessing overall decision-making performance under uncertainty. ECUAS_n metrics are formulated as proper scoring rules for the task of interest, with parameter n controlling the trade-off between the cost of incorrect predictions and imperfect uncertainties, allowing customization for specific use-cases. The paper demonstrates theoretical and empirical advantages through experiments on diverse classification and generation datasets, showing that ECUAS_n provides a more holistic assessment of UA system performance. The work addresses a growing need as UA systems are increasingly deployed in high-stakes automated decision-making where users must accept or reject predictions based on application-specific cost trade-offs.
Provides a principled way to evaluate uncertainty-augmented systems, enabling better decision-making in high-stakes applications.