The Dyson Equalizer: Adaptive Noise Stabilization for Low-Rank Signal Detection and Recovery Article Swipe

MAKE IMPACT

Boris Landa , Yuval Kluger · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2306.11263

Detecting and recovering a low-rank signal in a noisy data matrix is a fundamental task in data analysis. Typically, this task is addressed by inspecting and manipulating the spectrum of the observed data, e.g., thresholding the singular values of the data matrix at a certain critical level. This approach is well-established in the case of homoskedastic noise, where the noise variance is identical across the entries. However, in numerous applications, the noise can be heteroskedastic, where the noise characteristics may vary considerably across the rows and columns of the data. In this scenario, the spectral behavior of the noise can differ significantly from the homoskedastic case, posing various challenges for signal detection and recovery. To address these challenges, we develop an adaptive normalization procedure that equalizes the average noise variance across the rows and columns of a given data matrix. Our proposed procedure is data-driven and fully automatic, supporting a broad range of noise distributions, variance patterns, and signal structures. Our approach relies on recent results in random matrix theory, which describe the resolvent of the noise via the so-called Dyson equation. By leveraging this relation, we can accurately infer the noise level in each row and each column directly from the resolvent of the data. We establish that in many cases, our normalization enforces the standard spectral behavior of homoskedastic noise -- the Marchenko-Pastur (MP) law, allowing for simple and reliable detection of signal components. Furthermore, we demonstrate that our approach can substantially improve signal recovery in heteroskedastic settings by manipulating the spectrum after normalization. Lastly, we apply our method to single-cell RNA sequencing and spatial transcriptomics data, showcasing accurate fits to the MP law after normalization.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2306.11263

PDF

https://arxiv.org/pdf/2306.11263

OA Status

green

Cited By

1

Related Works

10

OpenAlex ID

https://openalex.org/W4381573332

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4381573332

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2306.11263

Digital Object Identifier

Title

The Dyson Equalizer: Adaptive Noise Stabilization for Low-Rank Signal Detection and Recovery

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-06-20

Full publication date if available

Authors

Boris Landa, Yuval Kluger

List of authors in order

Landing page

https://arxiv.org/abs/2306.11263

Publisher landing page

PDF URL

https://arxiv.org/pdf/2306.11263

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2306.11263

Direct OA link when available

Concepts

Heteroscedasticity, Noise (video), Normalization (sociology), Algorithm, Mathematics, Computer science, Noise measurement, Noise reduction, Statistics, Artificial intelligence, Image (mathematics), Anthropology, Sociology

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2024: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload