DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial\n Estimation Article Swipe
YOU?
·
· 2021
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2101.05544
Deep ensembles perform better than a single network thanks to the diversity\namong their members. Recent approaches regularize predictions to increase\ndiversity; however, they also drastically decrease individual members'\nperformances. In this paper, we argue that learning strategies for deep\nensembles need to tackle the trade-off between ensemble diversity and\nindividual accuracies. Motivated by arguments from information theory and\nleveraging recent advances in neural estimation of conditional mutual\ninformation, we introduce a novel training criterion called DICE: it increases\ndiversity by reducing spurious correlations among features. The main idea is\nthat features extracted from pairs of members should only share information\nuseful for target class prediction without being conditionally redundant.\nTherefore, besides the classification loss with information bottleneck, we\nadversarially prevent features from being conditionally predictable from each\nother. We manage to reduce simultaneous errors while protecting class\ninformation. We obtain state-of-the-art accuracy results on CIFAR-10/100: for\nexample, an ensemble of 5 networks trained with DICE matches an ensemble of 7\nnetworks trained independently. We further analyze the consequences on\ncalibration, uncertainty estimation, out-of-distribution detection and online\nco-distillation.\n
Related Topics To Compare & Contrast
- Type
- preprint
- Landing Page
- http://arxiv.org/abs/2101.05544
- https://arxiv.org/pdf/2101.05544
- OA Status
- green
- Cited By
- 23
- References
- 134
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W3132452637