DynamicBench: Evaluating Real-Time Report Generation in Large Language Models Article Swipe

PDF

Jingyao Li , Hao Sun , Zile Qiao , Yong Jiang , Pengjun Xie , Fei Huang , Hong Yan Xu , Jiaya Jia ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2506.21343

Traditional benchmarks for large language models (LLMs) typically rely on static evaluations through storytelling or opinion expression, which fail to capture the dynamic requirements of real-time information processing in contemporary applications. To address this limitation, we present DynamicBench, a benchmark designed to evaluate the proficiency of LLMs in storing and processing up-to-the-minute data. DynamicBench utilizes a dual-path retrieval pipeline, integrating web searches with local report databases. It necessitates domain-specific knowledge, ensuring accurate responses report generation within specialized fields. By evaluating models in scenarios that either provide or withhold external documents, DynamicBench effectively measures their capability to independently process recent information or leverage contextual enhancements. Additionally, we introduce an advanced report generation system adept at managing dynamic information synthesis. Our experimental results confirm the efficacy of our approach, with our method achieving state-of-the-art performance, surpassing GPT4o in document-free and document-assisted scenarios by 7.0% and 5.8%, respectively. The code and data will be made publicly available.

Related Topics

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2506.21343
PDF: https://arxiv.org/pdf/2506.21343
OA Status: green
OpenAlex ID: https://openalex.org/W4415183236

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4415183236

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2506.21343

Digital Object Identifier
Title: DynamicBench: Evaluating Real-Time Report Generation in Large Language Models

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-06-26

Full publication date if available
Authors: Jingyao Li, Hao Sun, Zile Qiao, Yong Jiang, Pengjun Xie, Fei Huang, Hong Yan Xu, Jiaya Jia

List of authors in order
Landing page: https://arxiv.org/abs/2506.21343

Publisher landing page
PDF URL: https://arxiv.org/pdf/2506.21343

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2506.21343

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4415183236
doi	https://doi.org/10.48550/arxiv.2506.21343
ids.doi	https://doi.org/10.48550/arxiv.2506.21343
ids.openalex	https://openalex.org/W4415183236
fwci
type	preprint
title	DynamicBench: Evaluating Real-Time Report Generation in Large Language Models
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10028
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9503999948501587
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Topic Modeling
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2506.21343
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2506.21343
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2506.21343
locations[1].id	doi:10.48550/arxiv.2506.21343
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2506.21343
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5100739860
authorships[0].author.orcid	https://orcid.org/0000-0002-4559-1524
authorships[0].author.display_name	Jingyao Li
authorships[0].author_position	first
authorships[0].raw_author_name	Li, Jingyao
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5100556869
authorships[1].author.orcid	https://orcid.org/0009-0001-1956-9921
authorships[1].author.display_name	Hao Sun
authorships[1].author_position	middle
authorships[1].raw_author_name	Sun, Hao
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5013664694
authorships[2].author.orcid
authorships[2].author.display_name	Zile Qiao
authorships[2].author_position	middle
authorships[2].raw_author_name	Qiao, Zile
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5101626204
authorships[3].author.orcid	https://orcid.org/0000-0002-4260-1395
authorships[3].author.display_name	Yong Jiang
authorships[3].author_position	middle
authorships[3].raw_author_name	Jiang, Yong
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5005535444
authorships[4].author.orcid	https://orcid.org/0009-0004-8412-359X
authorships[4].author.display_name	Pengjun Xie
authorships[4].author_position	middle
authorships[4].raw_author_name	Xie, Pengjun
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5087630524
authorships[5].author.orcid	https://orcid.org/0009-0002-3863-3180
authorships[5].author.display_name	Fei Huang
authorships[5].author_position	middle
authorships[5].raw_author_name	Huang, Fei
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5113895148
authorships[6].author.orcid	https://orcid.org/0009-0008-7031-8248
authorships[6].author.display_name	Hong Yan Xu
authorships[6].author_position	middle
authorships[6].raw_author_name	Xu, Hong
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5119999553
authorships[7].author.orcid
authorships[7].author.display_name	Jiaya Jia
authorships[7].author_position	last
authorships[7].raw_author_name	Jia, Jiaya
authorships[7].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2506.21343
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-15T00:00:00
display_name	DynamicBench: Evaluating Real-Time Report Generation in Large Language Models
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10028
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9503999948501587
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Topic Modeling
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2506.21343
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2506.21343
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2506.21343
primary_location.id	pmh:oai:arXiv.org:2506.21343
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2506.21343
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2506.21343
publication_date	2025-06-26
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	38, 55
abstract_inverted_index.By	78
abstract_inverted_index.It	66
abstract_inverted_index.To	31
abstract_inverted_index.an	107
abstract_inverted_index.at	113
abstract_inverted_index.be	150
abstract_inverted_index.by	140
abstract_inverted_index.in	28, 47, 81, 135
abstract_inverted_index.of	24, 45, 124
abstract_inverted_index.on	9
abstract_inverted_index.or	14, 86, 100
abstract_inverted_index.to	19, 41, 95
abstract_inverted_index.we	35, 105
abstract_inverted_index.Our	118
abstract_inverted_index.The	145
abstract_inverted_index.and	49, 137, 142, 147
abstract_inverted_index.for	2
abstract_inverted_index.our	125, 128
abstract_inverted_index.the	21, 43, 122
abstract_inverted_index.web	60
abstract_inverted_index.7.0%	141
abstract_inverted_index.LLMs	46
abstract_inverted_index.code	146
abstract_inverted_index.data	148
abstract_inverted_index.fail	18
abstract_inverted_index.made	151
abstract_inverted_index.rely	8
abstract_inverted_index.that	83
abstract_inverted_index.this	33
abstract_inverted_index.will	149
abstract_inverted_index.with	62, 127
abstract_inverted_index.5.8%,	143
abstract_inverted_index.GPT4o	134
abstract_inverted_index.adept	112
abstract_inverted_index.data.	52
abstract_inverted_index.large	3
abstract_inverted_index.local	63
abstract_inverted_index.their	93
abstract_inverted_index.which	17
abstract_inverted_index.(LLMs)	6
abstract_inverted_index.either	84
abstract_inverted_index.method	129
abstract_inverted_index.models	5, 80
abstract_inverted_index.recent	98
abstract_inverted_index.report	64, 73, 109
abstract_inverted_index.static	10
abstract_inverted_index.system	111
abstract_inverted_index.within	75
abstract_inverted_index.address	32
abstract_inverted_index.capture	20
abstract_inverted_index.confirm	121
abstract_inverted_index.dynamic	22, 115
abstract_inverted_index.fields.	77
abstract_inverted_index.opinion	15
abstract_inverted_index.present	36
abstract_inverted_index.process	97
abstract_inverted_index.provide	85
abstract_inverted_index.results	120
abstract_inverted_index.storing	48
abstract_inverted_index.through	12
abstract_inverted_index.accurate	71
abstract_inverted_index.advanced	108
abstract_inverted_index.designed	40
abstract_inverted_index.efficacy	123
abstract_inverted_index.ensuring	70
abstract_inverted_index.evaluate	42
abstract_inverted_index.external	88
abstract_inverted_index.language	4
abstract_inverted_index.leverage	101
abstract_inverted_index.managing	114
abstract_inverted_index.measures	92
abstract_inverted_index.publicly	152
abstract_inverted_index.searches	61
abstract_inverted_index.utilizes	54
abstract_inverted_index.withhold	87
abstract_inverted_index.achieving	130
abstract_inverted_index.approach,	126
abstract_inverted_index.benchmark	39
abstract_inverted_index.dual-path	56
abstract_inverted_index.introduce	106
abstract_inverted_index.pipeline,	58
abstract_inverted_index.real-time	25
abstract_inverted_index.responses	72
abstract_inverted_index.retrieval	57
abstract_inverted_index.scenarios	82, 139
abstract_inverted_index.typically	7
abstract_inverted_index.available.	153
abstract_inverted_index.benchmarks	1
abstract_inverted_index.capability	94
abstract_inverted_index.contextual	102
abstract_inverted_index.databases.	65
abstract_inverted_index.documents,	89
abstract_inverted_index.evaluating	79
abstract_inverted_index.generation	74, 110
abstract_inverted_index.knowledge,	69
abstract_inverted_index.processing	27, 50
abstract_inverted_index.surpassing	133
abstract_inverted_index.synthesis.	117
abstract_inverted_index.Traditional	0
abstract_inverted_index.effectively	91
abstract_inverted_index.evaluations	11
abstract_inverted_index.expression,	16
abstract_inverted_index.information	26, 99, 116
abstract_inverted_index.integrating	59
abstract_inverted_index.limitation,	34
abstract_inverted_index.proficiency	44
abstract_inverted_index.specialized	76
abstract_inverted_index.DynamicBench	53, 90
abstract_inverted_index.contemporary	29
abstract_inverted_index.experimental	119
abstract_inverted_index.necessitates	67
abstract_inverted_index.performance,	132
abstract_inverted_index.requirements	23
abstract_inverted_index.storytelling	13
abstract_inverted_index.Additionally,	104
abstract_inverted_index.DynamicBench,	37
abstract_inverted_index.applications.	30
abstract_inverted_index.document-free	136
abstract_inverted_index.enhancements.	103
abstract_inverted_index.independently	96
abstract_inverted_index.respectively.	144
abstract_inverted_index.domain-specific	68
abstract_inverted_index.state-of-the-art	131
abstract_inverted_index.up-to-the-minute	51
abstract_inverted_index.document-assisted	138
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	8
citation_normalized_percentile