Confidence-aware calibration and scoring functions for curriculum learning

Shuang Ao; Stefan Rueger; Advaith Siddharthan

doi:10.1117/12.2679353

7 June 2023 Confidence-aware calibration and scoring functions for curriculum learning

Shuang Ao, Stefan Rueger, Advaith Siddharthan

Proceedings Volume 12701, Fifteenth International Conference on Machine Vision (ICMV 2022); 127011U (2023) https://doi.org/10.1117/12.2679353
Event: Fifteenth International Conference on Machine Vision (ICMV 2022), 2022, Rome, Italy

Abstract

Despite the great success of state-of-the-art deep neural networks, several studies have reported models to be over-confident in predictions, indicating miscalibration. Label Smoothing has been proposed as a solution to the over-confidence problem and works by softening hard targets during training, typically by distributing part of the probability mass from a ‘one-hot’ label uniformly to all other labels. However, neither model nor human confidence in a label are likely to be uniformly distributed in this manner, with some labels more likely to be confused than others. In this paper we integrate notions of model confidence and human confidence with label smoothing, respectively Model Confidence LS and Human Confidence LS, to achieve better model calibration and generalization. To enhance model generalization, we show how our model and human confidence scores can be successfully applied to curriculum learning, a training strategy inspired by learning of ‘easier to harder’ tasks. A higher model or human confidence score indicates a more recognisable and therefore easier sample, and can therefore be used as a scoring function to rank samples in curriculum learning. We evaluate our proposed methods with four state-of-the-art architectures for image and text classification task, using datasets with multi-rater label annotations by humans. We report that integrating model or human confidence information in label smoothing and curriculum learning improves both model performance and model calibration. The code are available at https://github.com/AoShuang92/Confidence Calibration CL.

Citation Download Citation

Shuang Ao, Stefan Rueger, and Advaith Siddharthan "Confidence-aware calibration and scoring functions for curriculum learning", Proc. SPIE 12701, Fifteenth International Conference on Machine Vision (ICMV 2022), 127011U (7 June 2023); https://doi.org/10.1117/12.2679353

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
10 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Education and training

Machine learning

Calibration

Statistical modeling

Image classification

Electrochemical etching

Data modeling

Show All Keywords

Keywords/Phrases

Search In:

Publication Years