Kees van Deemter
YOU?
Author Swipe
View article: Reference-free Evaluation Metrics for Text Generation: A Survey
Reference-free Evaluation Metrics for Text Generation: A Survey Open
A number of automatic evaluation metrics have been proposed for natural language generation systems. The most common approach to automatic evaluation is the use of a reference-based metric that compares the model's output with gold-standar…
View article: Human-annotated rationales and explainable text classification: a survey
Human-annotated rationales and explainable text classification: a survey Open
Asking annotators to explain “why” they labeled an instance yields annotator rationales: natural language explanations that provide reasons for classifications. In this work, we survey the collection and use of annotator rationales. Human-…
View article: Computational Modelling of Plurality and Definiteness in Chinese Noun Phrases
Computational Modelling of Plurality and Definiteness in Chinese Noun Phrases Open
Theoretical linguists have suggested that some languages (e.g., Chinese and Japanese) are "cooler" than other languages based on the observation that the intended meaning of phrases in these languages depends more on their contexts. As a r…
View article: Intrinsic Task-based Evaluation for Referring Expression Generation
Intrinsic Task-based Evaluation for Referring Expression Generation Open
Recently, a human evaluation study of Referring Expression Generation (REG) models had an unexpected conclusion: on \textsc{webnlg}, Referring Expressions (REs) generated by the state-of-the-art neural models were not only indistinguishabl…
View article: Textual Summarisation of Large Sets: Towards a General Approach
Textual Summarisation of Large Sets: Towards a General Approach Open
We are developing techniques to generate summary descriptions of sets of objects. In this paper, we present and evaluate a rule-based NLG technique for summarising sets of bibliographical references in academic papers. This extends our pre…
View article: The Pitfalls of Defining Hallucination
The Pitfalls of Defining Hallucination Open
Despite impressive advances in Natural Language Generation (NLG) and Large Language Models (LLMs), researchers are still unclear about important aspects of NLG evaluation. To substantiate this claim, I examine current classifications of ha…
View article: The Pitfalls of Defining Hallucination
The Pitfalls of Defining Hallucination Open
Despite impressive advances in Natural Language Generation (NLG) and Large Language Models (LLMs), researchers are still unclear about important aspects of NLG evaluation. To substantiate this claim, I examine current classifications of ha…
View article: Interpreting vision and language generative models with semantic visual priors
Interpreting vision and language generative models with semantic visual priors Open
When applied to Image-to-text models, explainability methods have two challenges. First, they often provide token-by-token explanations namely, they compute a visual explanation for each token of the generated sequence. This makes explanat…
View article: Varieties of specification: Redefining over- and under-specification
Varieties of specification: Redefining over- and under-specification Open
A long tradition of research in theoretical, experimental and computational pragmatics has investigated over-specification and under-specification in referring expressions. Along broadly Gricean lines, these studies compare the amount of i…
View article: Models of reference production: How do they withstand the test of time?
Models of reference production: How do they withstand the test of time? Open
In recent years, many NLP studies have focused solely on performance improvement. In this work, we focus on the linguistic and scientific aspects of NLP. We use the task of generating referring expressions in context (REG-in-context) as a …
View article: Is Shortest Always Best? The Role of Brevity in Logic-to-Text Generation
Is Shortest Always Best? The Role of Brevity in Logic-to-Text Generation Open
Some applications of artificial intelligence make it desirable that logical formulae be converted computationally to comprehensible natural language sentences. As there are many logical equivalents to a given formula, finding the most suit…
View article: Computational Modelling of Quantifier Use: Corpus, Models, and Evaluation
Computational Modelling of Quantifier Use: Corpus, Models, and Evaluation Open
A prominent strand of work in formal semantics investigates the ways in which human languages quantify the elements of a set, as when we say All A are B, Few A are B, and so on. Building on a growing body of empirical studies that shed lig…
View article: Does ChatGPT have Theory of Mind?
Does ChatGPT have Theory of Mind? Open
Theory of Mind (ToM) is the ability to understand human thinking and decision-making, an ability that plays a crucial role in social interaction between people, including linguistic communication. This paper investigates to what extent rec…
View article: Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP Open
We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which…
View article: Interpreting Vision and Language Generative Models with Semantic Visual Priors
Interpreting Vision and Language Generative Models with Semantic Visual Priors Open
When applied to Image-to-text models, interpretability methods often provide token-by-token explanations namely, they compute a visual explanation for each token of the generated sequence. Those explanations are expensive to compute and un…
View article: HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales
HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales Open
Current captioning datasets focus on object-centric captions, describing the visible objects in the image, e.g. "people eating food in a park". Although these datasets are useful to evaluate the ability of Vision & Language models to recog…
View article: HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales
HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales Open
Current captioning datasets focus on object-centric captions, describing the visible objects in the image, often ending up stating the obvious (for humans), e.g. "people eating food in a park". Although these datasets are useful to evaluat…
View article: Is Shortest Always Best? The Role of Brevity in Logic-to-Text Generation
Is Shortest Always Best? The Role of Brevity in Logic-to-Text Generation Open
Some applications of artificial intelligence make it desirable that logical formulae be converted computationally to comprehensible natural language sentences. As there are many logical equivalents to a given formula, finding the most suit…
View article: Models of reference production: How do they withstand the test of time?
Models of reference production: How do they withstand the test of time? Open
In recent years, many NLP studies have focused solely on performance improvement. In this work, we focus on the linguistic and scientific aspects of NLP. We use the task of generating referring expressions in context (REG-in-context) as a …
View article: Dimensions of Explanatory Value in NLP Models
Dimensions of Explanatory Value in NLP Models Open
Performance on a dataset is often regarded as the key criterion for assessing NLP models. I argue for a broader perspective, which emphasizes scientific explanation. I draw on a long tradition in the philosophy of science, and on the Bayes…
View article: Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions
Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions Open
Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potent…
View article: Enhancing and Evaluating the Grammatical Framework Approach to Logic-to-Text Generation
Enhancing and Evaluating the Grammatical Framework Approach to Logic-to-Text Generation Open
Logic-to-text generation is an important yet underrepresented area of natural language generation (NLG). In particular, most previous works on this topic lack sound evaluation. We address this limitation by building and evaluating a system…
View article: Neural referential form selection: Generalisability and interpretability
Neural referential form selection: Generalisability and interpretability Open
In recent years, a range of Neural Referring Expression Generation (REG) systems have been built and they have often achieved encouraging results. However, these models are often thought to lack transparency and generality. Firstly, it is …
View article: Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions
Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions Open
Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potent…
View article: Assessing Neural Referential Form Selectors on a Realistic Multilingual Dataset
Assessing Neural Referential Form Selectors on a Realistic Multilingual Dataset Open
Previous work on Neural Referring Expression Generation (REG) all uses WebNLG, an English dataset that has been shown to reflect a very limited range of referring expression (RE) use. To tackle this issue, we build a dataset based on the O…
View article: Understanding the Use of Quantifiers in Mandarin
Understanding the Use of Quantifiers in Mandarin Open
We introduce a corpus of short texts in Mandarin, in which quantified expressions figure prominently. We illustrate the significance of the corpus by examining the hypothesis (known as Huang's "coolness" hypothesis) that speakers of East A…
View article: The Role of Explanatory Value in Natural Language Processing
The Role of Explanatory Value in Natural Language Processing Open
A key aim of science is explanation, yet the idea of explaining language phenomena has taken a backseat in mainstream Natural Language Processing (NLP) and many other areas of Artificial Intelligence. I argue that explanation of linguistic…
View article: Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddings
Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddings Open
Word embeddings have advanced the state of the art in NLP across numerous tasks. Understanding the contents of dense neural representations is of utmost interest to the computational semantics community. We propose to focus on relating the…
View article: Evaluating Automatic Difficulty Estimation of Logic Formalization Exercises
Evaluating Automatic Difficulty Estimation of Logic Formalization Exercises Open
Teaching logic effectively requires an understanding of the factors which cause logic students to struggle. Formalization exercises, which require the student to produce a formula corresponding to the natural language sentence, are a good …