Initialization ≈ Initialization
View article
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Open
Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates …
View article
VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator Open
One camera and one low-cost inertial measurement unit (IMU) form a monocular visual-inertial system (VINS), which is the minimum sensor suite (in size, weight, and power) for the metric six degrees-of-freedom (DOF) state estimation. In thi…
View article
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Open
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a…
View article
The k-means Algorithm: A Comprehensive Survey and Performance Evaluation Open
The k-means clustering algorithm is considered one of the most powerful and popular data mining algorithms in the research community. However, despite its popularity, the algorithm has certain limitations, including problems associated wit…
View article
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Open
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary ex…
View article
Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration Open
The Iterative Closest Point (ICP) algorithm is one of the most widely used methods for point-set registration. However, being based on local iterative optimization, ICP is known to be susceptible to local minima. Its performance critically…
View article
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Open
Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (P…
View article
Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts Open
Neural-based multi-task learning has been successfully used in many real-world large-scale applications such as recommendation systems. For example, in movie recommendations, beyond providing users movies which they tend to purchase and wa…
View article
Multilingual Denoising Pre-training for Neural Machine Translation Open
This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART—a sequence-to-sequence denoising auto-encoder pre-trained on …
View article
Meta-SGD: Learning to Learn Quickly for Few-Shot Learning Open
Few-shot learning is challenging for learning algorithms that learn each task in isolation and from scratch. In contrast, meta-learning learns from many related tasks a meta-learner that can learn a new task more accurately and faster with…
View article
ECCO version 4: an integrated framework for non-linear inverse modeling and global ocean state estimation Open
This paper presents the ECCO v4 non-linear inverse modeling framework and its baseline solution for the evolving ocean state over the period 1992–2011. Both components are publicly available and subjected to regular, automated regression t…
View article
Rethinking the Value of Network Pruning Open
Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, a…
View article
Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures Open
State-of-the-art neural networks are getting deeper and wider. While their performance increases with the increasing number of layers and neurons, it is crucial to design an efficient deep architecture in order to reduce computational and …
View article
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. Open
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary ex…
View article
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection Open
We present DINO (\textbf{D}ETR with \textbf{I}mproved de\textbf{N}oising anch\textbf{O}r boxes), a state-of-the-art end-to-end object detector. % in this paper. DINO improves over previous DETR-like models in performance and efficiency by …
View article
A large annotated medical image dataset for the development and evaluation of segmentation algorithms Open
Semantic segmentation of medical images aims to associate a pixel with a label in a medical image without human initialization. The success of semantic segmentation algorithms is contingent on the availability of high-quality imaging data …
View article
Reptile: a Scalable Metalearning Algorithm Open
This paper considers metalearning problems, where there is a distribution of tasks, and we would like to obtain an agent that performs well (i.e., learns quickly) when presented with a previously unseen task sampled from this distribution.…
View article
Intrusion Detection System Using Deep Neural Network for In-Vehicle Network Security Open
A novel intrusion detection system (IDS) using a deep neural network (DNN) is proposed to enhance the security of in-vehicular network. The parameters building the DNN structure are trained with probability-based feature vectors that are e…
View article
ERA-Interim/Land: a global land surface reanalysis data set Open
ERA-Interim/Land is a global land surface reanalysis data set covering the period 1979–2010. It describes the evolution of soil moisture, soil temperature and snowpack. ERA-Interim/Land is the result of a single 32-year simulation with the…
View article
Context Encoders: Feature Learning by Inpainting Open
We present an unsupervised visual feature learning algorithm driven by context-based pixel prediction. By analogy with auto-encoders, we propose Context Encoders -- a convolutional neural network trained to generate the contents of an arbi…
View article
Multilingual Denoising Pre-training for Neural Machine Translation Open
This paper demonstrates that multilingual denoising pre-training produces significant performance gains across a wide variety of machine translation (MT) tasks. We present mBART -- a sequence-to-sequence denoising auto-encoder pre-trained …
View article
TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation Open
Pixel-wise image segmentation is demanding task in computer vision. Classical U-Net architectures composed of encoders and decoders are very popular for segmentation of medical images, satellite images etc. Typically, neural network initia…
View article
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings Open
Recent work has managed to learn cross-lingual word embeddings without parallel data by mapping monolingual embeddings to a shared space through adversarial training. However, their evaluation has focused on favorable conditions, using com…
View article
On First-Order Meta-Learning Algorithms Open
This paper considers meta-learning problems, where there is a distribution of tasks, and we would like to obtain an agent that performs well (i.e., learns quickly) when presented with a previously unseen task sampled from this distribution…
View article
Recent updates to the Copernicus Marine Service global ocean monitoring and forecasting real-time 1∕12° high-resolution system Open
Since 19 October 2016, and in the framework of Copernicus Marine Environment Monitoring Service (CMEMS), Mercator Ocean has delivered real-time daily services (weekly analyses and daily 10-day forecasts) with a new global 1∕12∘ high-resolu…
View article
A review on swarm intelligence and evolutionary algorithms for solving flexible job shop scheduling problems Open
Flexible job shop scheduling problems (FJSP) have received much attention from academia and industry for many years. Due to their exponential complexity, swarm intelligence (SI) and evolutionary algorithms (EA) are developed, employed and …
View article
Generating Long Sequences with Sparse Transformers Open
Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We als…
View article
Phrase-Based & Neural Unsupervised Machine Translation Open
Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentences, which hinders their applicability to the majority of la…
View article
The Decadal Climate Prediction Project (DCPP) contribution to CMIP6 Open
The Decadal Climate Prediction Project (DCPP) is a coordinated multi-model investigation into decadal climate prediction, predictability, and variability. The DCPP makes use of past experience in simulating and predicting decadal variabili…
View article
Deep Semantic Role Labeling: What Works and What’s Next Open
We introduce a new deep learning model for semantic role labeling (SRL) that significantly improves the state of the art, along with detailed analyses to reveal its strengths and limitations. We use a deep highway BiLSTM architecture with …