SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation Article Swipe

PDF

Guanhua Zhang , Mohamed Ahmed , Zhiming Hu , Andreas Bulling ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2410.08356

Recent work has highlighted the potential of modelling interactive behaviour analogously to natural language. We propose interactive behaviour summarisation as a novel computational task and demonstrate its usefulness for automatically uncovering latent user intentions while interacting with graphical user interfaces. To tackle this task, we introduce SummAct, a novel hierarchical method to summarise low-level input actions into high-level intentions. SummAct first identifies sub-goals from user actions using a large language model and in-context learning. High-level intentions are then obtained by fine-tuning the model using a novel UI element attention to preserve detailed context information embedded within UI elements during summarisation. Through a series of evaluations, we demonstrate that SummAct significantly outperforms baselines across desktop and mobile interfaces as well as interactive tasks by up to 21.9%. We further show three exciting interactive applications benefited from SummAct: interactive behaviour forecasting, automatic behaviour synonym identification, and language-based behaviour retrieval.

Related Topics

Computer Science

Human–Computer Interaction

Concepts

Computer science Human–computer interaction World Wide Web

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2410.08356
PDF: https://arxiv.org/pdf/2410.08356
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4403444107

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4403444107

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2410.08356

Digital Object Identifier
Title: SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-10-10

Full publication date if available
Authors: Guanhua Zhang, Mohamed Ahmed, Zhiming Hu, Andreas Bulling

List of authors in order
Landing page: https://arxiv.org/abs/2410.08356

Publisher landing page
PDF URL: https://arxiv.org/pdf/2410.08356

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2410.08356

Direct OA link when available
Concepts: Computer science, Human–computer interaction, World Wide Web

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4403444107
doi	https://doi.org/10.48550/arxiv.2410.08356
ids.doi	https://doi.org/10.48550/arxiv.2410.08356
ids.openalex	https://openalex.org/W4403444107
fwci
type	preprint
title	SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T13083
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9476000070571899
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Advanced Text Analysis Techniques
topics[1].id	https://openalex.org/T10799
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9309999942779541
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1707
topics[1].subfield.display_name	Computer Vision and Pattern Recognition
topics[1].display_name	Data Visualization and Analytics
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.5583418011665344
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C107457646
concepts[1].level	1
concepts[1].score	0.4314953088760376
concepts[1].wikidata	https://www.wikidata.org/wiki/Q207434
concepts[1].display_name	Human–computer interaction
concepts[2].id	https://openalex.org/C136764020
concepts[2].level	1
concepts[2].score	0.3546930253505707
concepts[2].wikidata	https://www.wikidata.org/wiki/Q466
concepts[2].display_name	World Wide Web
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.5583418011665344
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/human–computer-interaction
keywords[1].score	0.4314953088760376
keywords[1].display_name	Human–computer interaction
keywords[2].id	https://openalex.org/keywords/world-wide-web
keywords[2].score	0.3546930253505707
keywords[2].display_name	World Wide Web
language	en
locations[0].id	pmh:oai:arXiv.org:2410.08356
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2410.08356
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2410.08356
locations[1].id	doi:10.48550/arxiv.2410.08356
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2410.08356
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5100438476
authorships[0].author.orcid	https://orcid.org/0000-0003-1445-1817
authorships[0].author.display_name	Guanhua Zhang
authorships[0].author_position	first
authorships[0].raw_author_name	Zhang, Guanhua
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5062100859
authorships[1].author.orcid	https://orcid.org/0000-0002-3325-0204
authorships[1].author.display_name	Mohamed Ahmed
authorships[1].author_position	middle
authorships[1].raw_author_name	Ahmed, Mohamed
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5101425139
authorships[2].author.orcid	https://orcid.org/0000-0002-2422-8099
authorships[2].author.display_name	Zhiming Hu
authorships[2].author_position	middle
authorships[2].raw_author_name	Hu, Zhiming
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5073661463
authorships[3].author.orcid	https://orcid.org/0000-0001-6317-7303
authorships[3].author.display_name	Andreas Bulling
authorships[3].author_position	last
authorships[3].raw_author_name	Bulling, Andreas
authorships[3].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2410.08356
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T13083
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9476000070571899
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Advanced Text Analysis Techniques
related_works	https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052, https://openalex.org/W4402327032
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2410.08356
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2410.08356
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2410.08356
primary_location.id	pmh:oai:arXiv.org:2410.08356
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2410.08356
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2410.08356
publication_date	2024-10-10
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	20, 47, 67, 84, 101
abstract_inverted_index.To	40
abstract_inverted_index.UI	86, 96
abstract_inverted_index.We	14, 126
abstract_inverted_index.as	19, 117, 119
abstract_inverted_index.by	79, 122
abstract_inverted_index.of	6, 103
abstract_inverted_index.to	11, 51, 89, 124
abstract_inverted_index.up	123
abstract_inverted_index.we	44, 105
abstract_inverted_index.and	24, 71, 114, 143
abstract_inverted_index.are	76
abstract_inverted_index.for	28
abstract_inverted_index.has	2
abstract_inverted_index.its	26
abstract_inverted_index.the	4, 81
abstract_inverted_index.from	63, 134
abstract_inverted_index.into	56
abstract_inverted_index.show	128
abstract_inverted_index.task	23
abstract_inverted_index.that	107
abstract_inverted_index.then	77
abstract_inverted_index.this	42
abstract_inverted_index.user	32, 38, 64
abstract_inverted_index.well	118
abstract_inverted_index.with	36
abstract_inverted_index.work	1
abstract_inverted_index.first	60
abstract_inverted_index.input	54
abstract_inverted_index.large	68
abstract_inverted_index.model	70, 82
abstract_inverted_index.novel	21, 48, 85
abstract_inverted_index.task,	43
abstract_inverted_index.tasks	121
abstract_inverted_index.three	129
abstract_inverted_index.using	66, 83
abstract_inverted_index.while	34
abstract_inverted_index.21.9%.	125
abstract_inverted_index.Recent	0
abstract_inverted_index.across	112
abstract_inverted_index.during	98
abstract_inverted_index.latent	31
abstract_inverted_index.method	50
abstract_inverted_index.mobile	115
abstract_inverted_index.series	102
abstract_inverted_index.tackle	41
abstract_inverted_index.within	95
abstract_inverted_index.SummAct	59, 108
abstract_inverted_index.Through	100
abstract_inverted_index.actions	55, 65
abstract_inverted_index.context	92
abstract_inverted_index.desktop	113
abstract_inverted_index.element	87
abstract_inverted_index.further	127
abstract_inverted_index.natural	12
abstract_inverted_index.propose	15
abstract_inverted_index.synonym	141
abstract_inverted_index.SummAct,	46
abstract_inverted_index.SummAct:	135
abstract_inverted_index.detailed	91
abstract_inverted_index.elements	97
abstract_inverted_index.embedded	94
abstract_inverted_index.exciting	130
abstract_inverted_index.language	69
abstract_inverted_index.obtained	78
abstract_inverted_index.preserve	90
abstract_inverted_index.attention	88
abstract_inverted_index.automatic	139
abstract_inverted_index.baselines	111
abstract_inverted_index.behaviour	9, 17, 137, 140, 145
abstract_inverted_index.benefited	133
abstract_inverted_index.graphical	37
abstract_inverted_index.introduce	45
abstract_inverted_index.language.	13
abstract_inverted_index.learning.	73
abstract_inverted_index.low-level	53
abstract_inverted_index.modelling	7
abstract_inverted_index.potential	5
abstract_inverted_index.sub-goals	62
abstract_inverted_index.summarise	52
abstract_inverted_index.High-level	74
abstract_inverted_index.high-level	57
abstract_inverted_index.identifies	61
abstract_inverted_index.in-context	72
abstract_inverted_index.intentions	33, 75
abstract_inverted_index.interfaces	116
abstract_inverted_index.retrieval.	146
abstract_inverted_index.uncovering	30
abstract_inverted_index.usefulness	27
abstract_inverted_index.analogously	10
abstract_inverted_index.demonstrate	25, 106
abstract_inverted_index.fine-tuning	80
abstract_inverted_index.highlighted	3
abstract_inverted_index.information	93
abstract_inverted_index.intentions.	58
abstract_inverted_index.interacting	35
abstract_inverted_index.interactive	8, 16, 120, 131, 136
abstract_inverted_index.interfaces.	39
abstract_inverted_index.outperforms	110
abstract_inverted_index.applications	132
abstract_inverted_index.evaluations,	104
abstract_inverted_index.forecasting,	138
abstract_inverted_index.hierarchical	49
abstract_inverted_index.automatically	29
abstract_inverted_index.computational	22
abstract_inverted_index.significantly	109
abstract_inverted_index.summarisation	18
abstract_inverted_index.language-based	144
abstract_inverted_index.summarisation.	99
abstract_inverted_index.identification,	142
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	4
citation_normalized_percentile