Answer Summarization for Technical Queries: Benchmark and New Approach Article Swipe

MAKE IMPACT

Yang Chengran , Bowen Xu , Ferdian Thung , Yucen Shi , Ting Zhang , Zhou Yang , Xin Zhou , Jieke Shi , Junda He , DongGyun Han , David Lo · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2209.10868

Prior studies have demonstrated that approaches to generate an answer summary for a given technical query in Software Question and Answer (SQA) sites are desired. We find that existing approaches are assessed solely through user studies. There is a need for a benchmark with ground truth summaries to complement assessment through user studies. Unfortunately, such a benchmark is non-existent for answer summarization for technical queries from SQA sites. To fill the gap, we manually construct a high-quality benchmark to enable automatic evaluation of answer summarization for technical queries for SQA sites. Using the benchmark, we comprehensively evaluate the performance of existing approaches and find that there is still a big room for improvement. Motivated by the results, we propose a new approach TechSumBot with three key modules:1) Usefulness Ranking module, 2) Centrality Estimation module, and 3) Redundancy Removal module. We evaluate TechSumBot in both automatic (i.e., using our benchmark) and manual (i.e., via a user study) manners. The results from both evaluations consistently demonstrate that TechSumBot outperforms the best performing baseline approaches from both SE and NLP domains by a large margin, i.e., 10.83%-14.90%, 32.75%-36.59%, and 12.61%-17.54%, in terms of ROUGE-1, ROUGE-2, and ROUGE-L on automatic evaluation, and 5.79%-9.23% and 17.03%-17.68%, in terms of average usefulness and diversity score on human evaluation. This highlights that the automatic evaluation of our benchmark can uncover findings similar to the ones found through user studies. More importantly, automatic evaluation has a much lower cost, especially when it is used to assess a new approach. Additionally, we also conducted an ablation study, which demonstrates that each module in TechSumBot contributes to boosting the overall performance of TechSumBot.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2209.10868

PDF

https://arxiv.org/pdf/2209.10868

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4296972837

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4296972837

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2209.10868

Digital Object Identifier

Title

Answer Summarization for Technical Queries: Benchmark and New Approach

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-09-22

Full publication date if available

Authors

Yang Chengran, Bowen Xu, Ferdian Thung, Yucen Shi, Ting Zhang, Zhou Yang, Xin Zhou, Jieke Shi, Junda He, DongGyun Han, David Lo

List of authors in order

Landing page

https://arxiv.org/abs/2209.10868

Publisher landing page

PDF URL

https://arxiv.org/pdf/2209.10868

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2209.10868

Direct OA link when available

Concepts

Benchmark (surveying), Automatic summarization, Computer science, Ranking (information retrieval), Information retrieval, Ground truth, Sophistication, Baseline (sea), Data mining, Machine learning, Artificial intelligence, Natural language processing, Social science, Geography, Geodesy, Oceanography, Geology, Sociology

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload