PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solving Article Swipe

PDF

Feng Kaiyue , Zhao Yilun , Liu, Yixin , Yang Tian-yu , Zhao Chen , Sous, John , Cohan, Arman ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2503.21821

We introduce PHYSICS, a comprehensive benchmark for university-level physics problem solving. It contains 1297 expert-annotated problems covering six core areas: classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics. Each problem requires advanced physics knowledge and mathematical reasoning. We develop a robust automated evaluation system for precise and reliable validation. Our evaluation of leading foundation models reveals substantial limitations. Even the most advanced model, o3-mini, achieves only 59.9% accuracy, highlighting significant challenges in solving high-level scientific problems. Through comprehensive error analysis, exploration of diverse prompting strategies, and Retrieval-Augmented Generation (RAG)-based knowledge augmentation, we identify key areas for improvement, laying the foundation for future advancements.

Related Topics

History Of Physics

Physics

Matthew Shepard Foundation

Spin (Physics)

James Beard Foundation Award

List Of Nobel Laureates In Physics

National Science Foundation

Vector (Mathematics And Physics)

Free Software Foundation

Foundation Stone

Isha Foundation

Alcor Life Extension Foundation

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2503.21821
PDF: https://arxiv.org/pdf/2503.21821
OA Status: green
OpenAlex ID: https://openalex.org/W4416419437

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4416419437

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2503.21821

Digital Object Identifier
Title: PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solving

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-03-26

Full publication date if available
Authors: Feng Kaiyue, Zhao Yilun, Liu, Yixin, Yang Tian-yu, Zhao Chen, Sous, John, Cohan, Arman

List of authors in order
Landing page: https://arxiv.org/abs/2503.21821

Publisher landing page
PDF URL: https://arxiv.org/pdf/2503.21821

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2503.21821

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4416419437
doi	https://doi.org/10.48550/arxiv.2503.21821
ids.doi	https://doi.org/10.48550/arxiv.2503.21821
ids.openalex	https://openalex.org/W4416419437
fwci	0.0
type	preprint
title	PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solving
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2503.21821
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2503.21821
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2503.21821
locations[1].id	doi:10.48550/arxiv.2503.21821
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2503.21821
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A2282559761
authorships[0].author.orcid
authorships[0].author.display_name	Feng Kaiyue
authorships[0].author_position	first
authorships[0].raw_author_name	Feng, Kaiyue
authorships[0].is_corresponding	True
authorships[1].author.id	https://openalex.org/A2393719697
authorships[1].author.orcid
authorships[1].author.display_name	Zhao Yilun
authorships[1].author_position	middle
authorships[1].raw_author_name	Zhao, Yilun
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A1994553651
authorships[2].author.orcid
authorships[2].author.display_name	Liu, Yixin
authorships[2].author_position	middle
authorships[2].raw_author_name	Liu, Yixin
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A2349428251
authorships[3].author.orcid
authorships[3].author.display_name	Yang Tian-yu
authorships[3].author_position	middle
authorships[3].raw_author_name	Yang, Tianyu
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A2095696890
authorships[4].author.orcid	https://orcid.org/0000-0001-5651-8966
authorships[4].author.display_name	Zhao Chen
authorships[4].author_position	middle
authorships[4].raw_author_name	Zhao, Chen
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A4281277221
authorships[5].author.orcid
authorships[5].author.display_name	Sous, John
authorships[5].author_position	middle
authorships[5].raw_author_name	Sous, John
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A4200857849
authorships[6].author.orcid
authorships[6].author.display_name	Cohan, Arman
authorships[6].author_position	last
authorships[6].raw_author_name	Cohan, Arman
authorships[6].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2503.21821
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solving
has_fulltext	False
is_retracted	False
updated_date	2025-11-28T13:40:31.218794
primary_topic
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2503.21821
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2503.21821
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2503.21821
primary_location.id	pmh:oai:arXiv.org:2503.21821
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2503.21821
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2503.21821
publication_date	2025-03-26
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	3, 44
abstract_inverted_index.It	11
abstract_inverted_index.We	0, 42
abstract_inverted_index.in	76
abstract_inverted_index.of	56, 86
abstract_inverted_index.we	96
abstract_inverted_index.Our	54
abstract_inverted_index.and	25, 31, 39, 51, 90
abstract_inverted_index.for	6, 49, 100, 105
abstract_inverted_index.key	98
abstract_inverted_index.six	17
abstract_inverted_index.the	64, 103
abstract_inverted_index.1297	13
abstract_inverted_index.Each	33
abstract_inverted_index.Even	63
abstract_inverted_index.core	18
abstract_inverted_index.most	65
abstract_inverted_index.only	70
abstract_inverted_index.59.9%	71
abstract_inverted_index.areas	99
abstract_inverted_index.error	83
abstract_inverted_index.areas:	19
abstract_inverted_index.atomic	29
abstract_inverted_index.future	106
abstract_inverted_index.laying	102
abstract_inverted_index.model,	67
abstract_inverted_index.models	59
abstract_inverted_index.robust	45
abstract_inverted_index.system	48
abstract_inverted_index.Through	81
abstract_inverted_index.develop	43
abstract_inverted_index.diverse	87
abstract_inverted_index.leading	57
abstract_inverted_index.optics.	32
abstract_inverted_index.physics	8, 37
abstract_inverted_index.precise	50
abstract_inverted_index.problem	9, 34
abstract_inverted_index.quantum	22
abstract_inverted_index.reveals	60
abstract_inverted_index.solving	77
abstract_inverted_index.PHYSICS,	2
abstract_inverted_index.achieves	69
abstract_inverted_index.advanced	36, 66
abstract_inverted_index.contains	12
abstract_inverted_index.covering	16
abstract_inverted_index.identify	97
abstract_inverted_index.o3-mini,	68
abstract_inverted_index.physics,	30
abstract_inverted_index.problems	15
abstract_inverted_index.reliable	52
abstract_inverted_index.requires	35
abstract_inverted_index.solving.	10
abstract_inverted_index.accuracy,	72
abstract_inverted_index.analysis,	84
abstract_inverted_index.automated	46
abstract_inverted_index.benchmark	5
abstract_inverted_index.classical	20
abstract_inverted_index.introduce	1
abstract_inverted_index.knowledge	38, 94
abstract_inverted_index.problems.	80
abstract_inverted_index.prompting	88
abstract_inverted_index.Generation	92
abstract_inverted_index.challenges	75
abstract_inverted_index.evaluation	47, 55
abstract_inverted_index.foundation	58, 104
abstract_inverted_index.high-level	78
abstract_inverted_index.mechanics,	21, 23, 27
abstract_inverted_index.reasoning.	41
abstract_inverted_index.scientific	79
abstract_inverted_index.(RAG)-based	93
abstract_inverted_index.exploration	85
abstract_inverted_index.significant	74
abstract_inverted_index.statistical	26
abstract_inverted_index.strategies,	89
abstract_inverted_index.substantial	61
abstract_inverted_index.validation.	53
abstract_inverted_index.highlighting	73
abstract_inverted_index.improvement,	101
abstract_inverted_index.limitations.	62
abstract_inverted_index.mathematical	40
abstract_inverted_index.advancements.	107
abstract_inverted_index.augmentation,	95
abstract_inverted_index.comprehensive	4, 82
abstract_inverted_index.thermodynamics	24
abstract_inverted_index.expert-annotated	14
abstract_inverted_index.university-level	7
abstract_inverted_index.electromagnetism,	28
abstract_inverted_index.Retrieval-Augmented	91
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	7
citation_normalized_percentile