Mutual information ≈ Mutual information
View article
embo - Empirical Bottleneck Open
A Python implementation of the Information Bottleneck analysis framework [Tishby, Pereira, Bialek 2001], especially geared towards the analysis of concrete, finite-size data sets. See on PyPI
View article
Learning deep representations by mutual information estimation and maximization Open
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about…
View article
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets Open
This paper describes InfoGAN, an information-theoretic extension to the Generative Adversarial Network that is able to learn disentangled representations in a completely unsupervised manner. InfoGAN is a generative adversarial network that…
View article
Opening the Black Box of Deep Neural Networks via Information Open
Despite their great success, there is still no comprehensive theoretical understanding of learning with Deep Neural Networks (DNNs) or their inner organization. Previous work proposed to analyze DNNs in the \textit{Information Plane}; i.e.…
View article
Learning Representations by Maximizing Mutual Information Across Views Open
We propose an approach to self-supervised representation learning based on maximizing mutual information between features extracted from multiple views of a shared context. For example, one could produce multiple views of a local spatio-te…
View article
Feature selection using Joint Mutual Information Maximisation Open
Feature selection is used in many application areas relevant to expert and intelligent systems, such as data mining and machine learning, image processing, anomaly detection, bioinformatics and natural language processing. Feature selectio…
View article
Building an Intrusion Detection System Using a Filter-Based Feature Selection Algorithm Open
Redundant and irrelevant features in data have caused a long-term problem in network traffic classification. These features not only slow down the process of classification but also prevent a classifier from making accurate decisions, espe…
View article
Heart Disease Identification Method Using Machine Learning Classification in E-Healthcare Open
Heart disease is one of the complex diseases and globally many people suffered from this disease. On time and efficient identification of heart disease plays a key role in healthcare, particularly in the field of cardiology. In this articl…
View article
Contrastive Multiview Coding Open
Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factor…
View article
Learning deep representations by mutual information estimation and maximization Open
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about…
View article
Graph Representation Learning via Graphical Mutual Information Maximization Open
The richness in the content of various information networks such as social networks and communication networks provides the unprecedented potential for learning high-quality expressive representations without external supervision. This pap…
View article
ARACNe-AP: gene network reverse engineering through adaptive partitioning inference of mutual information Open
Summary: The accurate reconstruction of gene regulatory networks from large scale molecular profile datasets represents one of the grand challenges of Systems Biology. The Algorithm for the Reconstruction of Accurate Cellular Networks (ARA…
View article
Probabilistic Constellation Shaping for Optical Fiber Communications Open
We review probabilistic constellation shaping (PCS), which has been a key enabler for several recent record-setting optical fiber communications experiments. PCS provides both fine-grained rate adaptability and energy efficiency (sensitivi…
View article
A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copula Open
We begin by reviewing the statistical framework of information theory as applicable to neuroimaging data analysis. A major factor hindering wider adoption of this framework in neuroimaging is the difficulty of estimating information theore…
View article
Information Dropout: Learning Optimal Representations Through Noisy Computation Open
The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some of the key properties. We show that this can be solved by adding a regularization ter…
View article
Adapting Feature Selection Algorithms for the Classification of Chinese Texts Open
Text classification has been highlighted as the key process to organize online texts for better communication in the Digital Media Age. Text classification establishes classification rules based on text features, so the accuracy of feature…
View article
MINE: Mutual Information Neural Estimation Open
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scal…
View article
Deep learning and the information bottleneck principle Open
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variabl…
View article
InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation\n Learning via Mutual Information Maximization Open
This paper studies learning the representations of whole graphs in both\nunsupervised and semi-supervised scenarios. Graph-level representations are\ncritical in a variety of real-world applications such as predicting the\nproperties of mo…
View article
Automatic Targetless Extrinsic Calibration of a 3D Lidar and Camera by Maximizing Mutual Information Open
This paper reports on a mutual information (MI) based algorithm for automatic extrinsic calibration of a 3D laser scanner and optical camera system. By using MI as the registration criterion, our method is able to work in situ without the …
View article
Quantum Conditional Mutual Information and Approximate Markov Chains Open
ISSN:1432-0916
View article
Density Modeling of Images using a Generalized Normalization\n Transformation Open
We introduce a parametric nonlinear transformation that is well-suited for\nGaussianizing data from natural images. The data are linearly transformed, and\neach component is then normalized by a pooled activity measure, computed by\nexpone…
View article
Deep learning for inferring gene relationships from single-cell expression data Open
Several methods were developed to mine gene–gene relationships from expression data. Examples include correlation and mutual information methods for coexpression analysis, clustering and undirected graphical models for functional assignmen…
View article
Part mutual information for quantifying direct associations in networks Open
Significance Measuring direct associations between variables is of great importance in various areas of science, especially in the era of big data. Although mutual information and conditional mutual information are widely used in quantifyi…
View article
MINE: Mutual Information Neural Estimation. Open
We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scal…
View article
On Mutual Information Maximization for Representation Learning Open
Many recent methods for unsupervised or self-supervised representation learning train feature extractors by maximizing an estimate of the mutual information (MI) between different views of the data. This comes with several immediate proble…
View article
Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems Open
To fully characterize the information that two source variables carry about a third target variable, one must decompose the total information into redundant, unique, and synergistic components, i.e., obtain a partial information decomposit…
View article
Irreversibility in Active Matter Systems: Fluctuation Theorem and Mutual Information Open
We consider a Brownian particle which, in addition to being in contact with a thermal bath, is driven by fluctuating forces which stem from active processes in the system, such as self-propulsion or collisions with other active particles. …
View article
Analysis of Network Clustering Algorithms and Cluster Quality Metrics at Scale Open
Smart local moving is the overall best performing algorithm in our study, but discrepancies between cluster evaluation metrics prevent us from declaring it an absolutely superior algorithm. Interestingly, Louvain performed better than Info…
View article
From the Information Bottleneck to the Privacy Funnel Open
We focus on the privacy-utility trade-off encountered by users who wish to disclose some information to an analyst, that is correlated with their private data, in the hope of receiving some utility. We rely on a general privacy statistical…