An Empirical Framework for Evaluating Semantic Preservation Using Hugging Face Article Swipe

MAKE IMPACT

Jia, Nan , Raja, Anita , Khatchadourian, Raffi · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2512.07983

As machine learning (ML) becomes an integral part of high-autonomy systems, it is critical to ensure the trustworthiness of learning-enabled software systems (LESS). Yet, the nondeterministic and run-time-defined semantics of ML complicate traditional software refactoring. We define semantic preservation in LESS as the property that optimizations of intelligent components do not alter the system's overall functional behavior. This paper introduces an empirical framework to evaluate semantic preservation in LESS by mining model evolution data from HuggingFace. We extract commit histories, $\textit{Model Cards}$, and performance metrics from a large number of models. To establish baselines, we conducted case studies in three domains, tracing performance changes across versions. Our analysis demonstrates how $\textit{semantic drift}$ can be detected via evaluation metrics across commits and reveals common refactoring patterns based on commit message analysis. Although API constraints limited the possibility of estimating a full-scale threshold, our pipeline offers a foundation for defining community-accepted boundaries for semantic preservation. Our contributions include: (1) a large-scale dataset of ML model evolution, curated from 1.7 million Hugging Face entries via a reproducible pipeline using the native HF hub API, (2) a practical pipeline for the evaluation of semantic preservation for a subset of 536 models and 4000+ metrics and (3) empirical case studies illustrating semantic drift in practice. Together, these contributions advance the foundations for more maintainable and trustworthy ML systems.

Type

preprint

Landing Page

https://doi.org/10.48550/arxiv.2512.07983

OA Status

green

OpenAlex ID

https://openalex.org/W7114774506

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W7114774506

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2512.07983

Digital Object Identifier

Title

An Empirical Framework for Evaluating Semantic Preservation Using Hugging Face

Work title

Type

preprint

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-12-08

Full publication date if available

Authors

Jia, Nan, Raja, Anita, Khatchadourian, Raffi

List of authors in order

Landing page

https://doi.org/10.48550/arxiv.2512.07983

Publisher landing page

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://doi.org/10.48550/arxiv.2512.07983

Direct OA link when available

Concepts

Computer science, Commit, Pipeline (software), Semantics (computer science), Tracing, Nondeterministic algorithm, Face (sociological concept), Empirical research, Code refactoring, Software, Semantic data model, Principle of compositionality, Artificial intelligence, Information retrieval, Data mining, Programming language, Machine learning, Software engineering, Natural language processing, Task (project management), Linked data, Data modeling, Semantic analysis (machine learning), Software metric, Software system, Property (philosophy), Semantic computing, Exploit

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Full payload