Large Language Models to Estimate CGI-S From Clinical Notes as a Measure of Depression Severity (Preprint) Article Swipe

MAKE IMPACT

Kevin C. Li , Ayah Zirikly , Sarah C. Collica , Fernando S. Goes , Congwen Zhao , Trang Quynh Nguyen , Jane P. Gagliardi , Benjamin A. Goldstein , Hwanhee Hong , Elizabeth A. Stuart , Peter P. Zandi · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.2196/preprints.86906

BACKGROUND Real-world psychiatric care is marked by wide heterogeneity in clinical presentations and outcomes, underscoring the need for systematic approaches to outcome measurement. The Clinical Global Impression–Severity (CGI-S) scale is a brief, clinician-rated measure of overall illness severity widely used in psychiatric research but rarely documented in routine care. Large language models (LLMs) may enable automated extraction of CGI-S scores from narrative clinical notes providing scalable outcome measures for real-world clinical care and research. OBJECTIVE This study aimed to evaluate whether LLMs can estimate CGI-S scores from psychiatric clinical notes in patients with major depressive disorder (MDD) by first generating a clinician consensus gold standard dataset, and then comparing model-generated scores for validation. METHODS We used data from the Johns Hopkins electronic health record. Three psychiatrists independently rated 77 clinical notes using a validated depression-specific CGI rubric. Weighted Cohen’s kappa (κ) coefficients were calculated to assess interrater reliability and model–human agreement. Two prompting strategies, zero-shot and few-shot, were tested using GPT-4o, and agreement was compared against average human ratings. Exploratory analyses evaluated whether agreement varied by patient demographics, care setting, or note length. RESULTS Interrater reliability among psychiatrists was high (κ = 0.77–0.78). Agreement between model-generated and average human ratings was similarly strong (κ = 0.85) and was even higher for notes on which all three raters were in complete agreement (κ = 0.88). Weighted κ values remained consistently high across all subgroups (0.82–0.89), with no significant differences by age, sex, race, treatment location, or note length. CONCLUSIONS LLMs can accurately estimate clinician-rated CGI-S scores from psychiatric clinical notes, achieving reliability comparable to expert raters. This approach may enable scalable outcome measurement and support the implementation of measurement-based care in real-world psychiatric practice.

Type

article

Landing Page

https://doi.org/10.2196/preprints.86906

OA Status

gold

OpenAlex ID

https://openalex.org/W4415787470

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4415787470

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.2196/preprints.86906

Digital Object Identifier

Title

Large Language Models to Estimate CGI-S From Clinical Notes as a Measure of Depression Severity (Preprint)

Work title

Type

article

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-11-02

Full publication date if available

Authors

Kevin C. Li, Ayah Zirikly, Sarah C. Collica, Fernando S. Goes, Congwen Zhao, Trang Quynh Nguyen, Jane P. Gagliardi, Benjamin A. Goldstein, Hwanhee Hong, Elizabeth A. Stuart, Peter P. Zandi

List of authors in order

Landing page

https://doi.org/10.2196/preprints.86906

Publisher landing page

Open access

Yes

Whether a free full text is available

OA status

gold

Open access status per OpenAlex

OA URL

https://doi.org/10.2196/preprints.86906

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload