Coping with AI errors with provable guarantees

TY - JOUR

T1 - Coping with AI errors with provable guarantees

AU - Tyukin, Ivan y.

AU - Tyukina, Tatiana

AU - Van helden, Daniël p.

AU - Zheng, Zedong

AU - Mirkes, Evgeny m.

AU - Sutton, Oliver j.

AU - Zhou, Qinghua

AU - Gorban, Alexander n.

AU - Allison, Penelope

N1 - Publisher Copyright: © 2024 The Author(s)

PY - 2024/9/1

Y1 - 2024/9/1

N2 - AI errors pose a significant challenge, hindering real-world applications. This work introduces a novel approach to cope with AI errors using weakly supervised error correctors that guarantee a specific level of error reduction. Our correctors have low computational cost and can be used to decide whether to abstain from making an unsafe classification. We provide new upper and lower bounds on the probability of errors in the corrected system. In contrast to existing works, these bounds are distribution agnostic, non-asymptotic, and can be efficiently computed just using the corrector training data. They also can be used in settings with concept drifts when the observed frequencies of separate classes vary. The correctors can easily be updated, removed, or replaced in response to changes in distributions within each class without retraining the underlying classifier. The application of the approach is illustrated with two relevant challenging tasks: (i) an image classification problem with scarce training data, and (ii) moderating responses of large language models without retraining or otherwise fine-tuning.

AB - AI errors pose a significant challenge, hindering real-world applications. This work introduces a novel approach to cope with AI errors using weakly supervised error correctors that guarantee a specific level of error reduction. Our correctors have low computational cost and can be used to decide whether to abstain from making an unsafe classification. We provide new upper and lower bounds on the probability of errors in the corrected system. In contrast to existing works, these bounds are distribution agnostic, non-asymptotic, and can be efficiently computed just using the corrector training data. They also can be used in settings with concept drifts when the observed frequencies of separate classes vary. The correctors can easily be updated, removed, or replaced in response to changes in distributions within each class without retraining the underlying classifier. The application of the approach is illustrated with two relevant challenging tasks: (i) an image classification problem with scarce training data, and (ii) moderating responses of large language models without retraining or otherwise fine-tuning.

UR - http://www.scopus.com/inward/record.url?scp=85196167749&partnerID=8YFLogxK

U2 - 10.1016/j.ins.2024.120856

DO - 10.1016/j.ins.2024.120856

M3 - Article

SN - 0020-0255

VL - 678

SP - 120856

JO - INFORMATION SCIENCES

JF - INFORMATION SCIENCES

M1 - 120856

ER -