IEEE Access • Vol 13
Ensemble-Based Uncertainty Quantification for Reliable Large Language Model Classification in Social Data Applications
January 2025 • David F. Farr, Lynnette Hui Xian Ng, Iain J. Cruickshank, Nico Manzonelli, Nicholas A. Clark, Kate Starbird, Nathaniel D. Bastian, Jevin D. West
Assessing classification confidence is essential for effectively leveraging Large Language Models (LLMs) in automated data labeling, particularly within the sensitive contexts of Computational Social Science (CSS) tasks. In this study, we evaluate five uncertainty quantification (UQ) strategies across three CSS classification problems: stance detection, ideology identification, and frame detection. We benchmark these strategies using three different LLMs. To enhance human-in-the-loop classification performance, we…