Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data Article Swipe

PDF

Kai Kim , Henghsiu Tsai , Rajat Sen , Abhimanyu Das , Zihao Zhou , Abhishek Tanpure , Ming Ronnier Luo , Rose Yu ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2411.06735

Current forecasting approaches are largely unimodal and ignore the rich textual data that often accompany the time series due to lack of well-curated multimodal benchmark dataset. In this work, we develop TimeText Corpus (TTC), a carefully curated, time-aligned text and time dataset for multimodal forecasting. Our dataset is composed of sequences of numbers and text aligned to timestamps, and includes data from two different domains: climate science and healthcare. Our data is a significant contribution to the rare selection of available multimodal datasets. We also propose the Hybrid Multi-Modal Forecaster (Hybrid-MMF), a multimodal LLM that jointly forecasts both text and time series data using shared embeddings. However, contrary to our expectations, our Hybrid-MMF model does not outperform existing baselines in our experiments. This negative result highlights the challenges inherent in multimodal forecasting. Our code and data are available at https://github.com/Rose-STL-Lab/Multimodal_ Forecasting.

Related Topics

Concepts

Series (stratigraphy) Modal Time series Computer science Econometrics Mathematics Machine learning Geology Materials science Polymer chemistry Paleontology

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2411.06735
PDF: https://arxiv.org/pdf/2411.06735
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4404401790

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4404401790

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2411.06735

Digital Object Identifier
Title: Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-11-11

Full publication date if available
Authors: Kai Kim, Henghsiu Tsai, Rajat Sen, Abhimanyu Das, Zihao Zhou, Abhishek Tanpure, Ming Ronnier Luo, Rose Yu

List of authors in order
Landing page: https://arxiv.org/abs/2411.06735

Publisher landing page
PDF URL: https://arxiv.org/pdf/2411.06735

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2411.06735

Direct OA link when available
Concepts: Series (stratigraphy), Modal, Time series, Computer science, Econometrics, Mathematics, Machine learning, Geology, Materials science, Polymer chemistry, Paleontology

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4404401790
doi	https://doi.org/10.48550/arxiv.2411.06735
ids.doi	https://doi.org/10.48550/arxiv.2411.06735
ids.openalex	https://openalex.org/W4404401790
fwci
type	preprint
title	Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T13083
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9081000089645386
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Advanced Text Analysis Techniques
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C143724316
concepts[0].level	2
concepts[0].score	0.7651386260986328
concepts[0].wikidata	https://www.wikidata.org/wiki/Q312468
concepts[0].display_name	Series (stratigraphy)
concepts[1].id	https://openalex.org/C71139939
concepts[1].level	2
concepts[1].score	0.7312372326850891
concepts[1].wikidata	https://www.wikidata.org/wiki/Q910194
concepts[1].display_name	Modal
concepts[2].id	https://openalex.org/C151406439
concepts[2].level	2
concepts[2].score	0.5651103258132935
concepts[2].wikidata	https://www.wikidata.org/wiki/Q186588
concepts[2].display_name	Time series
concepts[3].id	https://openalex.org/C41008148
concepts[3].level	0
concepts[3].score	0.5479961037635803
concepts[3].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[3].display_name	Computer science
concepts[4].id	https://openalex.org/C149782125
concepts[4].level	1
concepts[4].score	0.39212608337402344
concepts[4].wikidata	https://www.wikidata.org/wiki/Q160039
concepts[4].display_name	Econometrics
concepts[5].id	https://openalex.org/C33923547
concepts[5].level	0
concepts[5].score	0.24068868160247803
concepts[5].wikidata	https://www.wikidata.org/wiki/Q395
concepts[5].display_name	Mathematics
concepts[6].id	https://openalex.org/C119857082
concepts[6].level	1
concepts[6].score	0.19265517592430115
concepts[6].wikidata	https://www.wikidata.org/wiki/Q2539
concepts[6].display_name	Machine learning
concepts[7].id	https://openalex.org/C127313418
concepts[7].level	0
concepts[7].score	0.1401527225971222
concepts[7].wikidata	https://www.wikidata.org/wiki/Q1069
concepts[7].display_name	Geology
concepts[8].id	https://openalex.org/C192562407
concepts[8].level	0
concepts[8].score	0.07429587841033936
concepts[8].wikidata	https://www.wikidata.org/wiki/Q228736
concepts[8].display_name	Materials science
concepts[9].id	https://openalex.org/C188027245
concepts[9].level	1
concepts[9].score	0.0
concepts[9].wikidata	https://www.wikidata.org/wiki/Q750446
concepts[9].display_name	Polymer chemistry
concepts[10].id	https://openalex.org/C151730666
concepts[10].level	1
concepts[10].score	0.0
concepts[10].wikidata	https://www.wikidata.org/wiki/Q7205
concepts[10].display_name	Paleontology
keywords[0].id	https://openalex.org/keywords/series
keywords[0].score	0.7651386260986328
keywords[0].display_name	Series (stratigraphy)
keywords[1].id	https://openalex.org/keywords/modal
keywords[1].score	0.7312372326850891
keywords[1].display_name	Modal
keywords[2].id	https://openalex.org/keywords/time-series
keywords[2].score	0.5651103258132935
keywords[2].display_name	Time series
keywords[3].id	https://openalex.org/keywords/computer-science
keywords[3].score	0.5479961037635803
keywords[3].display_name	Computer science
keywords[4].id	https://openalex.org/keywords/econometrics
keywords[4].score	0.39212608337402344
keywords[4].display_name	Econometrics
keywords[5].id	https://openalex.org/keywords/mathematics
keywords[5].score	0.24068868160247803
keywords[5].display_name	Mathematics
keywords[6].id	https://openalex.org/keywords/machine-learning
keywords[6].score	0.19265517592430115
keywords[6].display_name	Machine learning
keywords[7].id	https://openalex.org/keywords/geology
keywords[7].score	0.1401527225971222
keywords[7].display_name	Geology
keywords[8].id	https://openalex.org/keywords/materials-science
keywords[8].score	0.07429587841033936
keywords[8].display_name	Materials science
language	en
locations[0].id	pmh:oai:arXiv.org:2411.06735
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2411.06735
locations[0].version	submittedVersion
locations[0].raw_type
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2411.06735
locations[1].id	doi:10.48550/arxiv.2411.06735
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2411.06735
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5025321965
authorships[0].author.orcid
authorships[0].author.display_name	Kai Kim
authorships[0].author_position	first
authorships[0].raw_author_name	Kim, Kai
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5006670202
authorships[1].author.orcid	https://orcid.org/0000-0002-2149-1000
authorships[1].author.display_name	Henghsiu Tsai
authorships[1].author_position	middle
authorships[1].raw_author_name	Tsai, Howard
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5003974246
authorships[2].author.orcid	https://orcid.org/0000-0003-4677-643X
authorships[2].author.display_name	Rajat Sen
authorships[2].author_position	middle
authorships[2].raw_author_name	Sen, Rajat
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5101056045
authorships[3].author.orcid
authorships[3].author.display_name	Abhimanyu Das
authorships[3].author_position	middle
authorships[3].raw_author_name	Das, Abhimanyu
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5102764994
authorships[4].author.orcid	https://orcid.org/0000-0002-7909-6134
authorships[4].author.display_name	Zihao Zhou
authorships[4].author_position	middle
authorships[4].raw_author_name	Zhou, Zihao
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5114657118
authorships[5].author.orcid
authorships[5].author.display_name	Abhishek Tanpure
authorships[5].author_position	middle
authorships[5].raw_author_name	Tanpure, Abhishek
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5054692962
authorships[6].author.orcid	https://orcid.org/0000-0002-0845-319X
authorships[6].author.display_name	Ming Ronnier Luo
authorships[6].author_position	middle
authorships[6].raw_author_name	Luo, Mathew
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5057778679
authorships[7].author.orcid	https://orcid.org/0000-0002-8491-7937
authorships[7].author.display_name	Rose Yu
authorships[7].author_position	last
authorships[7].raw_author_name	Yu, Rose
authorships[7].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2411.06735
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Multi-Modal Forecaster: Jointly Predicting Time Series and Textual Data
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T13083
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9081000089645386
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Advanced Text Analysis Techniques
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W2119012848, https://openalex.org/W2622688551, https://openalex.org/W1550175370, https://openalex.org/W1990205660
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2411.06735
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2411.06735
best_oa_location.version	submittedVersion
best_oa_location.raw_type
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2411.06735
primary_location.id	pmh:oai:arXiv.org:2411.06735
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2411.06735
primary_location.version	submittedVersion
primary_location.raw_type
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2411.06735
publication_date	2024-11-11
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	34, 72, 91
abstract_inverted_index.In	26
abstract_inverted_index.We	83
abstract_inverted_index.at	138
abstract_inverted_index.in	119, 129
abstract_inverted_index.is	47, 71
abstract_inverted_index.of	21, 49, 51, 79
abstract_inverted_index.to	19, 56, 75, 108
abstract_inverted_index.we	29
abstract_inverted_index.LLM	93
abstract_inverted_index.Our	45, 69, 132
abstract_inverted_index.and	6, 39, 53, 58, 67, 99, 134
abstract_inverted_index.are	3, 136
abstract_inverted_index.due	18
abstract_inverted_index.for	42
abstract_inverted_index.not	115
abstract_inverted_index.our	109, 111, 120
abstract_inverted_index.the	8, 15, 76, 86, 126
abstract_inverted_index.two	62
abstract_inverted_index.This	122
abstract_inverted_index.also	84
abstract_inverted_index.both	97
abstract_inverted_index.code	133
abstract_inverted_index.data	11, 60, 70, 102, 135
abstract_inverted_index.does	114
abstract_inverted_index.from	61
abstract_inverted_index.lack	20
abstract_inverted_index.rare	77
abstract_inverted_index.rich	9
abstract_inverted_index.text	38, 54, 98
abstract_inverted_index.that	12, 94
abstract_inverted_index.this	27
abstract_inverted_index.time	16, 40, 100
abstract_inverted_index.model	113
abstract_inverted_index.often	13
abstract_inverted_index.using	103
abstract_inverted_index.work,	28
abstract_inverted_index.(TTC),	33
abstract_inverted_index.Corpus	32
abstract_inverted_index.Hybrid	87
abstract_inverted_index.ignore	7
abstract_inverted_index.result	124
abstract_inverted_index.series	17, 101
abstract_inverted_index.shared	104
abstract_inverted_index.Current	0
abstract_inverted_index.aligned	55
abstract_inverted_index.climate	65
abstract_inverted_index.dataset	41, 46
abstract_inverted_index.develop	30
abstract_inverted_index.jointly	95
abstract_inverted_index.largely	4
abstract_inverted_index.numbers	52
abstract_inverted_index.propose	85
abstract_inverted_index.science	66
abstract_inverted_index.textual	10
abstract_inverted_index.However,	106
abstract_inverted_index.TimeText	31
abstract_inverted_index.composed	48
abstract_inverted_index.contrary	107
abstract_inverted_index.curated,	36
abstract_inverted_index.dataset.	25
abstract_inverted_index.domains:	64
abstract_inverted_index.existing	117
abstract_inverted_index.includes	59
abstract_inverted_index.inherent	128
abstract_inverted_index.negative	123
abstract_inverted_index.unimodal	5
abstract_inverted_index.accompany	14
abstract_inverted_index.available	80, 137
abstract_inverted_index.baselines	118
abstract_inverted_index.benchmark	24
abstract_inverted_index.carefully	35
abstract_inverted_index.datasets.	82
abstract_inverted_index.different	63
abstract_inverted_index.forecasts	96
abstract_inverted_index.selection	78
abstract_inverted_index.sequences	50
abstract_inverted_index.Forecaster	89
abstract_inverted_index.Hybrid-MMF	112
abstract_inverted_index.approaches	2
abstract_inverted_index.challenges	127
abstract_inverted_index.highlights	125
abstract_inverted_index.multimodal	23, 43, 81, 92, 130
abstract_inverted_index.outperform	116
abstract_inverted_index.Multi-Modal	88
abstract_inverted_index.embeddings.	105
abstract_inverted_index.forecasting	1
abstract_inverted_index.healthcare.	68
abstract_inverted_index.significant	73
abstract_inverted_index.timestamps,	57
abstract_inverted_index.Forecasting.	140
abstract_inverted_index.contribution	74
abstract_inverted_index.experiments.	121
abstract_inverted_index.forecasting.	44, 131
abstract_inverted_index.time-aligned	37
abstract_inverted_index.well-curated	22
abstract_inverted_index.(Hybrid-MMF),	90
abstract_inverted_index.expectations,	110
abstract_inverted_index.https://github.com/Rose-STL-Lab/Multimodal_	139
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	8
citation_normalized_percentile