Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System Article Swipe

PDF

Robin Schmucker , Meng Xia , Amos Azaria , T. M. Mitchell ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2404.17460

Conversational tutoring systems (CTSs) offer learning experiences through interactions based on natural language. They are recognized for promoting cognitive engagement and improving learning outcomes, especially in reasoning tasks. Nonetheless, the cost associated with authoring CTS content is a major obstacle to widespread adoption and to research on effective instructional design. In this paper, we discuss and evaluate a novel type of CTS that leverages recent advances in large language models (LLMs) in two ways: First, the system enables AI-assisted content authoring by inducing an easily editable tutoring script automatically from a lesson text. Second, the system automates the script orchestration in a learning-by-teaching format via two LLM-based agents (Ruffle&Riley) acting as a student and a professor. The system allows for free-form conversations that follow the ITS-typical inner and outer loop structure. We evaluate Ruffle&Riley's ability to support biology lessons in two between-subject online user studies (N = 200) comparing the system to simpler QA chatbots and reading activity. Analyzing system usage patterns, pre/post-test scores and user experience surveys, we find that Ruffle&Riley users report high levels of engagement, understanding and perceive the offered support as helpful. Even though Ruffle&Riley users require more time to complete the activity, we did not find significant differences in short-term learning gains over the reading activity. Our system architecture and user study provide various insights for designers of future CTSs. We further open-source our system to support ongoing research on effective instructional design of LLM-based learning technologies.

Related Topics

Computer Science

Human–Computer Interaction

Concepts

Computer science Human–computer interaction

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2404.17460
PDF: https://arxiv.org/pdf/2404.17460
OA Status: green
Cited By: 2
Related Works: 10
OpenAlex ID: https://openalex.org/W4396243924

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4396243924

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2404.17460

Digital Object Identifier
Title: Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-04-26

Full publication date if available
Authors: Robin Schmucker, Meng Xia, Amos Azaria, T. M. Mitchell

List of authors in order
Landing page: https://arxiv.org/abs/2404.17460

Publisher landing page
PDF URL: https://arxiv.org/pdf/2404.17460

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2404.17460

Direct OA link when available
Concepts: Computer science, Human–computer interaction

Top concepts (fields/topics) attached by OpenAlex
Cited by: 2

Total citation count in OpenAlex
Citations by year (recent): 2025: 1, 2024: 1

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4396243924
doi	https://doi.org/10.48550/arxiv.2404.17460
ids.doi	https://doi.org/10.48550/arxiv.2404.17460
ids.openalex	https://openalex.org/W4396243924
fwci
type	preprint
title	Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11902
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9908999800682068
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Intelligent Tutoring Systems and Adaptive Learning
topics[1].id	https://openalex.org/T10028
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9671000242233276
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Topic Modeling
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.5923916697502136
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C107457646
concepts[1].level	1
concepts[1].score	0.37198907136917114
concepts[1].wikidata	https://www.wikidata.org/wiki/Q207434
concepts[1].display_name	Human–computer interaction
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.5923916697502136
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/human–computer-interaction
keywords[1].score	0.37198907136917114
keywords[1].display_name	Human–computer interaction
language	en
locations[0].id	pmh:oai:arXiv.org:2404.17460
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2404.17460
locations[0].version	submittedVersion
locations[0].raw_type
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2404.17460
locations[1].id	doi:10.48550/arxiv.2404.17460
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2404.17460
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5043741972
authorships[0].author.orcid	https://orcid.org/0000-0002-5275-3608
authorships[0].author.display_name	Robin Schmucker
authorships[0].author_position	first
authorships[0].raw_author_name	Schmucker, Robin
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5080268090
authorships[1].author.orcid	https://orcid.org/0000-0002-9497-2470
authorships[1].author.display_name	Meng Xia
authorships[1].author_position	middle
authorships[1].raw_author_name	Xia, Meng
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5082567694
authorships[2].author.orcid	https://orcid.org/0000-0002-5057-1309
authorships[2].author.display_name	Amos Azaria
authorships[2].author_position	middle
authorships[2].raw_author_name	Azaria, Amos
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5012149761
authorships[3].author.orcid	https://orcid.org/0000-0003-0809-1528
authorships[3].author.display_name	T. M. Mitchell
authorships[3].author_position	last
authorships[3].raw_author_name	Mitchell, Tom
authorships[3].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2404.17460
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11902
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9908999800682068
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Intelligent Tutoring Systems and Adaptive Learning
related_works	https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2382290278, https://openalex.org/W4395014643, https://openalex.org/W4391913857, https://openalex.org/W2350741829
cited_by_count	2
counts_by_year[0].year	2025
counts_by_year[0].cited_by_count	1
counts_by_year[1].year	2024
counts_by_year[1].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2404.17460
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2404.17460
best_oa_location.version	submittedVersion
best_oa_location.raw_type
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2404.17460
primary_location.id	pmh:oai:arXiv.org:2404.17460
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2404.17460
primary_location.version	submittedVersion
primary_location.raw_type
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2404.17460
publication_date	2024-04-26
publication_year	2024
referenced_works_count	0
abstract_inverted_index.=	146
abstract_inverted_index.a	37, 57, 90, 101, 111, 114
abstract_inverted_index.(N	145
abstract_inverted_index.In	50
abstract_inverted_index.QA	153
abstract_inverted_index.We	131, 225
abstract_inverted_index.an	83
abstract_inverted_index.as	110, 184
abstract_inverted_index.by	81
abstract_inverted_index.in	25, 66, 71, 100, 139, 203
abstract_inverted_index.is	36
abstract_inverted_index.of	60, 176, 222, 238
abstract_inverted_index.on	10, 46, 234
abstract_inverted_index.to	40, 44, 135, 151, 193, 230
abstract_inverted_index.we	53, 168, 197
abstract_inverted_index.CTS	34, 61
abstract_inverted_index.Our	211
abstract_inverted_index.The	116
abstract_inverted_index.and	20, 43, 55, 113, 127, 155, 164, 179, 214
abstract_inverted_index.are	14
abstract_inverted_index.did	198
abstract_inverted_index.for	16, 119, 220
abstract_inverted_index.not	199
abstract_inverted_index.our	228
abstract_inverted_index.the	29, 75, 94, 97, 124, 149, 181, 195, 208
abstract_inverted_index.two	72, 105, 140
abstract_inverted_index.via	104
abstract_inverted_index.200)	147
abstract_inverted_index.Even	186
abstract_inverted_index.They	13
abstract_inverted_index.cost	30
abstract_inverted_index.find	169, 200
abstract_inverted_index.from	89
abstract_inverted_index.high	174
abstract_inverted_index.loop	129
abstract_inverted_index.more	191
abstract_inverted_index.over	207
abstract_inverted_index.that	62, 122, 170
abstract_inverted_index.this	51
abstract_inverted_index.time	192
abstract_inverted_index.type	59
abstract_inverted_index.user	143, 165, 215
abstract_inverted_index.with	32
abstract_inverted_index.CTSs.	224
abstract_inverted_index.based	9
abstract_inverted_index.gains	206
abstract_inverted_index.inner	126
abstract_inverted_index.large	67
abstract_inverted_index.major	38
abstract_inverted_index.novel	58
abstract_inverted_index.offer	4
abstract_inverted_index.outer	128
abstract_inverted_index.study	216
abstract_inverted_index.text.	92
abstract_inverted_index.usage	160
abstract_inverted_index.users	172, 189
abstract_inverted_index.ways:	73
abstract_inverted_index.(CTSs)	3
abstract_inverted_index.(LLMs)	70
abstract_inverted_index.First,	74
abstract_inverted_index.acting	109
abstract_inverted_index.agents	107
abstract_inverted_index.allows	118
abstract_inverted_index.design	237
abstract_inverted_index.easily	84
abstract_inverted_index.follow	123
abstract_inverted_index.format	103
abstract_inverted_index.future	223
abstract_inverted_index.lesson	91
abstract_inverted_index.levels	175
abstract_inverted_index.models	69
abstract_inverted_index.online	142
abstract_inverted_index.paper,	52
abstract_inverted_index.recent	64
abstract_inverted_index.report	173
abstract_inverted_index.scores	163
abstract_inverted_index.script	87, 98
abstract_inverted_index.system	76, 95, 117, 150, 159, 212, 229
abstract_inverted_index.tasks.	27
abstract_inverted_index.though	187
abstract_inverted_index.Second,	93
abstract_inverted_index.ability	134
abstract_inverted_index.biology	137
abstract_inverted_index.content	35, 79
abstract_inverted_index.design.	49
abstract_inverted_index.discuss	54
abstract_inverted_index.enables	77
abstract_inverted_index.further	226
abstract_inverted_index.lessons	138
abstract_inverted_index.natural	11
abstract_inverted_index.offered	182
abstract_inverted_index.ongoing	232
abstract_inverted_index.provide	217
abstract_inverted_index.reading	156, 209
abstract_inverted_index.require	190
abstract_inverted_index.simpler	152
abstract_inverted_index.student	112
abstract_inverted_index.studies	144
abstract_inverted_index.support	136, 183, 231
abstract_inverted_index.systems	2
abstract_inverted_index.through	7
abstract_inverted_index.various	218
abstract_inverted_index.adoption	42
abstract_inverted_index.advances	65
abstract_inverted_index.chatbots	154
abstract_inverted_index.complete	194
abstract_inverted_index.editable	85
abstract_inverted_index.evaluate	56, 132
abstract_inverted_index.helpful.	185
abstract_inverted_index.inducing	82
abstract_inverted_index.insights	219
abstract_inverted_index.language	68
abstract_inverted_index.learning	5, 22, 205, 240
abstract_inverted_index.obstacle	39
abstract_inverted_index.perceive	180
abstract_inverted_index.research	45, 233
abstract_inverted_index.surveys,	167
abstract_inverted_index.tutoring	1, 86
abstract_inverted_index.Analyzing	158
abstract_inverted_index.LLM-based	106, 239
abstract_inverted_index.activity,	196
abstract_inverted_index.activity.	157, 210
abstract_inverted_index.authoring	33, 80
abstract_inverted_index.automates	96
abstract_inverted_index.cognitive	18
abstract_inverted_index.comparing	148
abstract_inverted_index.designers	221
abstract_inverted_index.effective	47, 235
abstract_inverted_index.free-form	120
abstract_inverted_index.improving	21
abstract_inverted_index.language.	12
abstract_inverted_index.leverages	63
abstract_inverted_index.outcomes,	23
abstract_inverted_index.patterns,	161
abstract_inverted_index.promoting	17
abstract_inverted_index.reasoning	26
abstract_inverted_index.associated	31
abstract_inverted_index.engagement	19
abstract_inverted_index.especially	24
abstract_inverted_index.experience	166
abstract_inverted_index.professor.	115
abstract_inverted_index.recognized	15
abstract_inverted_index.short-term	204
abstract_inverted_index.structure.	130
abstract_inverted_index.widespread	41
abstract_inverted_index.AI-assisted	78
abstract_inverted_index.ITS-typical	125
abstract_inverted_index.differences	202
abstract_inverted_index.engagement,	177
abstract_inverted_index.experiences	6
abstract_inverted_index.open-source	227
abstract_inverted_index.significant	201
abstract_inverted_index.Nonetheless,	28
abstract_inverted_index.architecture	213
abstract_inverted_index.interactions	8
abstract_inverted_index.automatically	88
abstract_inverted_index.conversations	121
abstract_inverted_index.instructional	48, 236
abstract_inverted_index.orchestration	99
abstract_inverted_index.pre/post-test	162
abstract_inverted_index.technologies.	241
abstract_inverted_index.understanding	178
abstract_inverted_index.Conversational	0
abstract_inverted_index.between-subject	141
abstract_inverted_index.Ruffle&Riley	171, 188
abstract_inverted_index.(Ruffle&Riley)	108
abstract_inverted_index.Ruffle&Riley's	133
abstract_inverted_index.learning-by-teaching	102
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	4
citation_normalized_percentile