LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation Article Swipe

PDF

Yujie Lu , Xianjun Yang , Xiujun Li , Xin Eric Wang , William Yang Wang ·

YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2305.11116

Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments. In this work, we propose LLMScore, a new framework that offers evaluation scores with multi-granularity compositionality. LLMScore leverages the large language models (LLMs) to evaluate text-to-image models. Initially, it transforms the image into image-level and object-level visual descriptions. Then an evaluation instruction is fed into the LLMs to measure the alignment between the synthesized image and the text, ultimately generating a score accompanied by a rationale. Our substantial analysis reveals the highest correlation of LLMScore with human judgments on a wide range of datasets (Attribute Binding Contrast, Concept Conjunction, MSCOCO, DrawBench, PaintSkills). Notably, our LLMScore achieves Kendall's tau correlation with human evaluations that is 58.8% and 31.2% higher than the commonly-used text-image matching metrics CLIP and BLIP, respectively.

Related Topics

Computer Science

Artificial Intelligence

Concepts

Image (mathematics) Matching (statistics) Computer science Principle of compositionality Granularity Correlation Natural language processing Artificial intelligence Object (grammar) Range (aeronautics) Measure (data warehouse) Pattern recognition (psychology) Data mining Statistics Mathematics Programming language Geometry Composite material Materials science

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2305.11116
PDF: https://arxiv.org/pdf/2305.11116
OA Status: green
Cited By: 8
Related Works: 10
OpenAlex ID: https://openalex.org/W4377164366

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4377164366

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2305.11116

Digital Object Identifier
Title: LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2023

Year of publication
Publication date: 2023-05-18

Full publication date if available
Authors: Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, William Yang Wang

List of authors in order
Landing page: https://arxiv.org/abs/2305.11116

Publisher landing page
PDF URL: https://arxiv.org/pdf/2305.11116

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2305.11116

Direct OA link when available
Concepts: Image (mathematics), Matching (statistics), Computer science, Principle of compositionality, Granularity, Correlation, Natural language processing, Artificial intelligence, Object (grammar), Range (aeronautics), Measure (data warehouse), Pattern recognition (psychology), Data mining, Statistics, Mathematics, Programming language, Geometry, Composite material, Materials science

Top concepts (fields/topics) attached by OpenAlex
Cited by: 8

Total citation count in OpenAlex
Citations by year (recent): 2025: 4, 2024: 3, 2023: 1

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4377164366
doi	https://doi.org/10.48550/arxiv.2305.11116
ids.doi	https://doi.org/10.48550/arxiv.2305.11116
ids.openalex	https://openalex.org/W4377164366
fwci
type	preprint
title	LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11714
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9973999857902527
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1707
topics[0].subfield.display_name	Computer Vision and Pattern Recognition
topics[0].display_name	Multimodal Machine Learning Applications
topics[1].id	https://openalex.org/T10601
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9825000166893005
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1707
topics[1].subfield.display_name	Computer Vision and Pattern Recognition
topics[1].display_name	Handwritten Text Recognition Techniques
topics[2].id	https://openalex.org/T10627
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9746000170707703
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1707
topics[2].subfield.display_name	Computer Vision and Pattern Recognition
topics[2].display_name	Advanced Image and Video Retrieval Techniques
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C115961682
concepts[0].level	2
concepts[0].score	0.7077622413635254
concepts[0].wikidata	https://www.wikidata.org/wiki/Q860623
concepts[0].display_name	Image (mathematics)
concepts[1].id	https://openalex.org/C165064840
concepts[1].level	2
concepts[1].score	0.6702009439468384
concepts[1].wikidata	https://www.wikidata.org/wiki/Q1321061
concepts[1].display_name	Matching (statistics)
concepts[2].id	https://openalex.org/C41008148
concepts[2].level	0
concepts[2].score	0.6636999845504761
concepts[2].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[2].display_name	Computer science
concepts[3].id	https://openalex.org/C121375916
concepts[3].level	2
concepts[3].score	0.6157399415969849
concepts[3].wikidata	https://www.wikidata.org/wiki/Q936559
concepts[3].display_name	Principle of compositionality
concepts[4].id	https://openalex.org/C177774035
concepts[4].level	2
concepts[4].score	0.5963484048843384
concepts[4].wikidata	https://www.wikidata.org/wiki/Q1246948
concepts[4].display_name	Granularity
concepts[5].id	https://openalex.org/C117220453
concepts[5].level	2
concepts[5].score	0.5512224435806274
concepts[5].wikidata	https://www.wikidata.org/wiki/Q5172842
concepts[5].display_name	Correlation
concepts[6].id	https://openalex.org/C204321447
concepts[6].level	1
concepts[6].score	0.536091148853302
concepts[6].wikidata	https://www.wikidata.org/wiki/Q30642
concepts[6].display_name	Natural language processing
concepts[7].id	https://openalex.org/C154945302
concepts[7].level	1
concepts[7].score	0.5335795283317566
concepts[7].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[7].display_name	Artificial intelligence
concepts[8].id	https://openalex.org/C2781238097
concepts[8].level	2
concepts[8].score	0.5277068018913269
concepts[8].wikidata	https://www.wikidata.org/wiki/Q175026
concepts[8].display_name	Object (grammar)
concepts[9].id	https://openalex.org/C204323151
concepts[9].level	2
concepts[9].score	0.4672892689704895
concepts[9].wikidata	https://www.wikidata.org/wiki/Q905424
concepts[9].display_name	Range (aeronautics)
concepts[10].id	https://openalex.org/C2780009758
concepts[10].level	2
concepts[10].score	0.46633172035217285
concepts[10].wikidata	https://www.wikidata.org/wiki/Q6804172
concepts[10].display_name	Measure (data warehouse)
concepts[11].id	https://openalex.org/C153180895
concepts[11].level	2
concepts[11].score	0.35505762696266174
concepts[11].wikidata	https://www.wikidata.org/wiki/Q7148389
concepts[11].display_name	Pattern recognition (psychology)
concepts[12].id	https://openalex.org/C124101348
concepts[12].level	1
concepts[12].score	0.2125629186630249
concepts[12].wikidata	https://www.wikidata.org/wiki/Q172491
concepts[12].display_name	Data mining
concepts[13].id	https://openalex.org/C105795698
concepts[13].level	1
concepts[13].score	0.17950302362442017
concepts[13].wikidata	https://www.wikidata.org/wiki/Q12483
concepts[13].display_name	Statistics
concepts[14].id	https://openalex.org/C33923547
concepts[14].level	0
concepts[14].score	0.12555566430091858
concepts[14].wikidata	https://www.wikidata.org/wiki/Q395
concepts[14].display_name	Mathematics
concepts[15].id	https://openalex.org/C199360897
concepts[15].level	1
concepts[15].score	0.08301830291748047
concepts[15].wikidata	https://www.wikidata.org/wiki/Q9143
concepts[15].display_name	Programming language
concepts[16].id	https://openalex.org/C2524010
concepts[16].level	1
concepts[16].score	0.0
concepts[16].wikidata	https://www.wikidata.org/wiki/Q8087
concepts[16].display_name	Geometry
concepts[17].id	https://openalex.org/C159985019
concepts[17].level	1
concepts[17].score	0.0
concepts[17].wikidata	https://www.wikidata.org/wiki/Q181790
concepts[17].display_name	Composite material
concepts[18].id	https://openalex.org/C192562407
concepts[18].level	0
concepts[18].score	0.0
concepts[18].wikidata	https://www.wikidata.org/wiki/Q228736
concepts[18].display_name	Materials science
keywords[0].id	https://openalex.org/keywords/image
keywords[0].score	0.7077622413635254
keywords[0].display_name	Image (mathematics)
keywords[1].id	https://openalex.org/keywords/matching
keywords[1].score	0.6702009439468384
keywords[1].display_name	Matching (statistics)
keywords[2].id	https://openalex.org/keywords/computer-science
keywords[2].score	0.6636999845504761
keywords[2].display_name	Computer science
keywords[3].id	https://openalex.org/keywords/principle-of-compositionality
keywords[3].score	0.6157399415969849
keywords[3].display_name	Principle of compositionality
keywords[4].id	https://openalex.org/keywords/granularity
keywords[4].score	0.5963484048843384
keywords[4].display_name	Granularity
keywords[5].id	https://openalex.org/keywords/correlation
keywords[5].score	0.5512224435806274
keywords[5].display_name	Correlation
keywords[6].id	https://openalex.org/keywords/natural-language-processing
keywords[6].score	0.536091148853302
keywords[6].display_name	Natural language processing
keywords[7].id	https://openalex.org/keywords/artificial-intelligence
keywords[7].score	0.5335795283317566
keywords[7].display_name	Artificial intelligence
keywords[8].id	https://openalex.org/keywords/object
keywords[8].score	0.5277068018913269
keywords[8].display_name	Object (grammar)
keywords[9].id	https://openalex.org/keywords/range
keywords[9].score	0.4672892689704895
keywords[9].display_name	Range (aeronautics)
keywords[10].id	https://openalex.org/keywords/measure
keywords[10].score	0.46633172035217285
keywords[10].display_name	Measure (data warehouse)
keywords[11].id	https://openalex.org/keywords/pattern-recognition
keywords[11].score	0.35505762696266174
keywords[11].display_name	Pattern recognition (psychology)
keywords[12].id	https://openalex.org/keywords/data-mining
keywords[12].score	0.2125629186630249
keywords[12].display_name	Data mining
keywords[13].id	https://openalex.org/keywords/statistics
keywords[13].score	0.17950302362442017
keywords[13].display_name	Statistics
keywords[14].id	https://openalex.org/keywords/mathematics
keywords[14].score	0.12555566430091858
keywords[14].display_name	Mathematics
keywords[15].id	https://openalex.org/keywords/programming-language
keywords[15].score	0.08301830291748047
keywords[15].display_name	Programming language
language	en
locations[0].id	pmh:oai:arXiv.org:2305.11116
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2305.11116
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2305.11116
locations[1].id	doi:10.48550/arxiv.2305.11116
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2305.11116
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5036525093
authorships[0].author.orcid	https://orcid.org/0000-0002-0691-2129
authorships[0].author.display_name	Yujie Lu
authorships[0].author_position	first
authorships[0].raw_author_name	Lu, Yujie
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5109445159
authorships[1].author.orcid
authorships[1].author.display_name	Xianjun Yang
authorships[1].author_position	middle
authorships[1].raw_author_name	Yang, Xianjun
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5021140826
authorships[2].author.orcid	https://orcid.org/0000-0001-7771-2725
authorships[2].author.display_name	Xiujun Li
authorships[2].author_position	middle
authorships[2].raw_author_name	Li, Xiujun
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5089351095
authorships[3].author.orcid
authorships[3].author.display_name	Xin Eric Wang
authorships[3].author_position	middle
authorships[3].raw_author_name	Wang, Xin Eric
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5100702485
authorships[4].author.orcid	https://orcid.org/0000-0001-6153-8240
authorships[4].author.display_name	William Yang Wang
authorships[4].author_position	last
authorships[4].raw_author_name	Wang, William Yang
authorships[4].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2305.11116
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2023-05-21T00:00:00
display_name	LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11714
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9973999857902527
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1707
primary_topic.subfield.display_name	Computer Vision and Pattern Recognition
primary_topic.display_name	Multimodal Machine Learning Applications
related_works	https://openalex.org/W1526190050, https://openalex.org/W4251923961, https://openalex.org/W2477397717, https://openalex.org/W2500457737, https://openalex.org/W2010490241, https://openalex.org/W4287025733, https://openalex.org/W4317425742, https://openalex.org/W3192500523, https://openalex.org/W4241382148, https://openalex.org/W2097744609
cited_by_count	8
counts_by_year[0].year	2025
counts_by_year[0].cited_by_count	4
counts_by_year[1].year	2024
counts_by_year[1].cited_by_count	3
counts_by_year[2].year	2023
counts_by_year[2].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2305.11116
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2305.11116
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2305.11116
primary_location.id	pmh:oai:arXiv.org:2305.11116
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2305.11116
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2305.11116
publication_date	2023-05-18
publication_year	2023
referenced_works_count	0
abstract_inverted_index.a	32, 86, 90, 105
abstract_inverted_index.In	26
abstract_inverted_index.an	9, 65
abstract_inverted_index.by	89
abstract_inverted_index.in	20
abstract_inverted_index.is	68, 129
abstract_inverted_index.it	54
abstract_inverted_index.of	99, 108
abstract_inverted_index.on	3, 104
abstract_inverted_index.to	49, 73
abstract_inverted_index.we	29
abstract_inverted_index.Our	92
abstract_inverted_index.and	60, 81, 131, 141
abstract_inverted_index.can	6
abstract_inverted_index.fed	69
abstract_inverted_index.new	33
abstract_inverted_index.our	119
abstract_inverted_index.tau	123
abstract_inverted_index.the	15, 44, 56, 71, 75, 78, 82, 96, 135
abstract_inverted_index.CLIP	140
abstract_inverted_index.LLMs	72
abstract_inverted_index.Then	64
abstract_inverted_index.into	58, 70
abstract_inverted_index.only	7
abstract_inverted_index.poor	21
abstract_inverted_index.than	134
abstract_inverted_index.that	35, 128
abstract_inverted_index.this	27
abstract_inverted_index.wide	106
abstract_inverted_index.with	23, 39, 101, 125
abstract_inverted_index.31.2%	132
abstract_inverted_index.58.8%	130
abstract_inverted_index.BLIP,	142
abstract_inverted_index.human	24, 102, 126
abstract_inverted_index.image	57, 80
abstract_inverted_index.large	45
abstract_inverted_index.range	107
abstract_inverted_index.score	87
abstract_inverted_index.text,	83
abstract_inverted_index.which	18
abstract_inverted_index.work,	28
abstract_inverted_index.(LLMs)	48
abstract_inverted_index.higher	133
abstract_inverted_index.models	47
abstract_inverted_index.offers	36
abstract_inverted_index.score,	12
abstract_inverted_index.scores	38
abstract_inverted_index.visual	62
abstract_inverted_index.Binding	111
abstract_inverted_index.Concept	113
abstract_inverted_index.MSCOCO,	115
abstract_inverted_index.between	77
abstract_inverted_index.highest	97
abstract_inverted_index.measure	74
abstract_inverted_index.metrics	139
abstract_inverted_index.models.	52
abstract_inverted_index.propose	30
abstract_inverted_index.provide	8
abstract_inverted_index.results	19
abstract_inverted_index.reveals	95
abstract_inverted_index.without	13
abstract_inverted_index.Existing	0
abstract_inverted_index.LLMScore	42, 100, 120
abstract_inverted_index.Notably,	118
abstract_inverted_index.achieves	121
abstract_inverted_index.analysis	94
abstract_inverted_index.datasets	109
abstract_inverted_index.evaluate	50
abstract_inverted_index.language	46
abstract_inverted_index.matching	11, 138
abstract_inverted_index.Contrast,	112
abstract_inverted_index.Kendall's	122
abstract_inverted_index.LLMScore,	31
abstract_inverted_index.alignment	76
abstract_inverted_index.automatic	1
abstract_inverted_index.framework	34
abstract_inverted_index.judgments	103
abstract_inverted_index.leverages	43
abstract_inverted_index.synthesis	5
abstract_inverted_index.(Attribute	110
abstract_inverted_index.DrawBench,	116
abstract_inverted_index.Initially,	53
abstract_inverted_index.evaluation	2, 37, 66
abstract_inverted_index.generating	85
abstract_inverted_index.image-text	10
abstract_inverted_index.judgments.	25
abstract_inverted_index.rationale.	91
abstract_inverted_index.text-image	137
abstract_inverted_index.transforms	55
abstract_inverted_index.ultimately	84
abstract_inverted_index.accompanied	88
abstract_inverted_index.considering	14
abstract_inverted_index.correlation	22, 98, 124
abstract_inverted_index.evaluations	127
abstract_inverted_index.image-level	59
abstract_inverted_index.instruction	67
abstract_inverted_index.substantial	93
abstract_inverted_index.synthesized	79
abstract_inverted_index.Conjunction,	114
abstract_inverted_index.object-level	16, 61
abstract_inverted_index.PaintSkills).	117
abstract_inverted_index.commonly-used	136
abstract_inverted_index.descriptions.	63
abstract_inverted_index.respectively.	143
abstract_inverted_index.text-to-image	4, 51
abstract_inverted_index.compositionality,	17
abstract_inverted_index.compositionality.	41
abstract_inverted_index.multi-granularity	40
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	5
citation_normalized_percentile