Evaluation Report on MCP Servers Article Swipe

PDF

Zhiling Luo , X. Shi , Xiangui Lin , Jinyang Gao ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2504.11094

With the rise of LLMs, a large number of Model Context Protocol (MCP) services have emerged since the end of 2024. However, the effectiveness and efficiency of MCP servers have not been well studied. To study these questions, we propose an evaluation framework, called MCPBench. We selected several widely used MCP server and conducted an experimental evaluation on their accuracy, time, and token usage. Our experiments showed that the most effective MCP, Bing Web Search, achieved an accuracy of 64%. Importantly, we found that the accuracy of MCP servers can be substantially enhanced by involving declarative interface. This research paves the way for further investigations into optimized MCP implementations, ultimately leading to better AI-driven applications and data retrieval solutions.

Related Topics

Minority Report (Film)

World Happiness Report

Drudge Report

Weather Report

The Report (2019 Film)

The Majority Report With Sam Seder

U.S. News & World Report Best Colleges Ranking

The Colbert Report

List Of Poweredge Servers

Europa Report

The Minority Report

Human Development Report

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2504.11094
PDF: https://arxiv.org/pdf/2504.11094
OA Status: green
OpenAlex ID: https://openalex.org/W4416052411

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4416052411

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2504.11094

Digital Object Identifier
Title: Evaluation Report on MCP Servers

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-04-15

Full publication date if available
Authors: Zhiling Luo, X. Shi, Xiangui Lin, Jinyang Gao

List of authors in order
Landing page: https://arxiv.org/abs/2504.11094

Publisher landing page
PDF URL: https://arxiv.org/pdf/2504.11094

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2504.11094

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4416052411
doi	https://doi.org/10.48550/arxiv.2504.11094
ids.doi	https://doi.org/10.48550/arxiv.2504.11094
ids.openalex	https://openalex.org/W4416052411
fwci
type	preprint
title	Evaluation Report on MCP Servers
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2504.11094
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2504.11094
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2504.11094
locations[1].id	doi:10.48550/arxiv.2504.11094
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2504.11094
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5020904548
authorships[0].author.orcid	https://orcid.org/0000-0002-0540-7307
authorships[0].author.display_name	Zhiling Luo
authorships[0].author_position	first
authorships[0].raw_author_name	Luo, Zhiling
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5091299418
authorships[1].author.orcid	https://orcid.org/0000-0001-9910-9345
authorships[1].author.display_name	X. Shi
authorships[1].author_position	middle
authorships[1].raw_author_name	Shi, Xiaorong
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5062481369
authorships[2].author.orcid	https://orcid.org/0000-0003-4230-5825
authorships[2].author.display_name	Xiangui Lin
authorships[2].author_position	middle
authorships[2].raw_author_name	Lin, Xuanrui
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5101403129
authorships[3].author.orcid	https://orcid.org/0000-0002-2094-3554
authorships[3].author.display_name	Jinyang Gao
authorships[3].author_position	last
authorships[3].raw_author_name	Gao, Jinyang
authorships[3].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2504.11094
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Evaluation Report on MCP Servers
has_fulltext	True
is_retracted	False
updated_date	2025-11-28T10:48:16.031419
primary_topic
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2504.11094
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2504.11094
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2504.11094
primary_location.id	pmh:oai:arXiv.org:2504.11094
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2504.11094
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2504.11094
publication_date	2025-04-15
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	5
abstract_inverted_index.To	34
abstract_inverted_index.We	45
abstract_inverted_index.an	40, 54, 76
abstract_inverted_index.be	90
abstract_inverted_index.by	93
abstract_inverted_index.of	3, 8, 19, 26, 78, 86
abstract_inverted_index.on	57
abstract_inverted_index.to	111
abstract_inverted_index.we	38, 81
abstract_inverted_index.MCP	27, 50, 87, 107
abstract_inverted_index.Our	64
abstract_inverted_index.Web	73
abstract_inverted_index.and	24, 52, 61, 115
abstract_inverted_index.can	89
abstract_inverted_index.end	18
abstract_inverted_index.for	102
abstract_inverted_index.not	30
abstract_inverted_index.the	1, 17, 22, 68, 84, 100
abstract_inverted_index.way	101
abstract_inverted_index.64%.	79
abstract_inverted_index.Bing	72
abstract_inverted_index.MCP,	71
abstract_inverted_index.This	97
abstract_inverted_index.With	0
abstract_inverted_index.been	31
abstract_inverted_index.data	116
abstract_inverted_index.have	14, 29
abstract_inverted_index.into	105
abstract_inverted_index.most	69
abstract_inverted_index.rise	2
abstract_inverted_index.that	67, 83
abstract_inverted_index.used	49
abstract_inverted_index.well	32
abstract_inverted_index.(MCP)	12
abstract_inverted_index.2024.	20
abstract_inverted_index.LLMs,	4
abstract_inverted_index.Model	9
abstract_inverted_index.found	82
abstract_inverted_index.large	6
abstract_inverted_index.paves	99
abstract_inverted_index.since	16
abstract_inverted_index.study	35
abstract_inverted_index.their	58
abstract_inverted_index.these	36
abstract_inverted_index.time,	60
abstract_inverted_index.token	62
abstract_inverted_index.better	112
abstract_inverted_index.called	43
abstract_inverted_index.number	7
abstract_inverted_index.server	51
abstract_inverted_index.showed	66
abstract_inverted_index.usage.	63
abstract_inverted_index.widely	48
abstract_inverted_index.Context	10
abstract_inverted_index.Search,	74
abstract_inverted_index.emerged	15
abstract_inverted_index.further	103
abstract_inverted_index.leading	110
abstract_inverted_index.propose	39
abstract_inverted_index.servers	28, 88
abstract_inverted_index.several	47
abstract_inverted_index.However,	21
abstract_inverted_index.Protocol	11
abstract_inverted_index.accuracy	77, 85
abstract_inverted_index.achieved	75
abstract_inverted_index.enhanced	92
abstract_inverted_index.research	98
abstract_inverted_index.selected	46
abstract_inverted_index.services	13
abstract_inverted_index.studied.	33
abstract_inverted_index.AI-driven	113
abstract_inverted_index.MCPBench.	44
abstract_inverted_index.accuracy,	59
abstract_inverted_index.conducted	53
abstract_inverted_index.effective	70
abstract_inverted_index.involving	94
abstract_inverted_index.optimized	106
abstract_inverted_index.retrieval	117
abstract_inverted_index.efficiency	25
abstract_inverted_index.evaluation	41, 56
abstract_inverted_index.framework,	42
abstract_inverted_index.interface.	96
abstract_inverted_index.questions,	37
abstract_inverted_index.solutions.	118
abstract_inverted_index.ultimately	109
abstract_inverted_index.declarative	95
abstract_inverted_index.experiments	65
abstract_inverted_index.Importantly,	80
abstract_inverted_index.applications	114
abstract_inverted_index.experimental	55
abstract_inverted_index.effectiveness	23
abstract_inverted_index.substantially	91
abstract_inverted_index.investigations	104
abstract_inverted_index.implementations,	108
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	4
citation_normalized_percentile