Psychological Stress Classification Using EEG and ECG: A CNN Based Multimodal Fusion Model Article Swipe

PDF

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.21203/rs.3.rs-4015916/v1

Psychological stress cannot be ignored in today's society, and there is an urgent need for an objective and cost-effective method to detect it. However, traditional machine learning methods that require manual feature extraction require a lot of research time and cannot guarantee accuracy. In this paper, we establish a four-category stress multimodal dataset by collecting EEG and ECG signals from 24 subjects performing mental arithmetic tasks with different difficulty levels and propose a multimodal decision fusion model based on Convolution Neural Network to classify the data. The prediction probabilities of EEG and ECG signals for the four stress categories are first extracted by two models each and then fused into the decision model for the final classification, 5-fold cross-validation and Leave-Three-Subjects-Out experiments are performed, which achieve 91.14% and 91.97% accuracy, respectively. In addition, the features of the convolution layer were visualized using the 1D-Grad-CAM method to improve the interpretability of the model.

Related Topics

Computer Science

Electroencephalography

Convolutional Neural Network

Artificial Intelligence

Machine Learning

Philosophy

Psychiatry

Concepts

Interpretability Computer science Electroencephalography Convolutional neural network Artificial intelligence Convolution (computer science) Pattern recognition (psychology) Feature (linguistics) Feature extraction Artificial neural network Machine learning Psychology Linguistics Philosophy Psychiatry

Metadata

Type: preprint
Language: en
Landing Page: https://doi.org/10.21203/rs.3.rs-4015916/v1
PDF: https://www.researchsquare.com/article/rs-4015916/latest.pdf
OA Status: gold
Cited By: 1
References: 28
Related Works: 10
OpenAlex ID: https://openalex.org/W4392706073

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4392706073

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.21203/rs.3.rs-4015916/v1

Digital Object Identifier
Title: Psychological Stress Classification Using EEG and ECG: A CNN Based Multimodal Fusion Model

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-03-12

Full publication date if available
Authors: Ben Zhou, Lei Wang, Chenyu Jiang

List of authors in order
Landing page: https://doi.org/10.21203/rs.3.rs-4015916/v1

Publisher landing page
PDF URL: https://www.researchsquare.com/article/rs-4015916/latest.pdf

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: gold

Open access status per OpenAlex
OA URL: https://www.researchsquare.com/article/rs-4015916/latest.pdf

Direct OA link when available
Concepts: Interpretability, Computer science, Electroencephalography, Convolutional neural network, Artificial intelligence, Convolution (computer science), Pattern recognition (psychology), Feature (linguistics), Feature extraction, Artificial neural network, Machine learning, Psychology, Linguistics, Philosophy, Psychiatry

Top concepts (fields/topics) attached by OpenAlex
Cited by: 1

Total citation count in OpenAlex
Citations by year (recent): 2025: 1

Per-year citation counts (last 5 years)
References (count): 28

Number of works referenced by this work
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4392706073
doi	https://doi.org/10.21203/rs.3.rs-4015916/v1
ids.doi	https://doi.org/10.21203/rs.3.rs-4015916/v1
ids.openalex	https://openalex.org/W4392706073
fwci	0.70280056
type	preprint
title	Psychological Stress Classification Using EEG and ECG: A CNN Based Multimodal Fusion Model
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10429
topics[0].field.id	https://openalex.org/fields/28
topics[0].field.display_name	Neuroscience
topics[0].score	0.9987000226974487
topics[0].domain.id	https://openalex.org/domains/1
topics[0].domain.display_name	Life Sciences
topics[0].subfield.id	https://openalex.org/subfields/2805
topics[0].subfield.display_name	Cognitive Neuroscience
topics[0].display_name	EEG and Brain-Computer Interfaces
topics[1].id	https://openalex.org/T10745
topics[1].field.id	https://openalex.org/fields/27
topics[1].field.display_name	Medicine
topics[1].score	0.9902999997138977
topics[1].domain.id	https://openalex.org/domains/4
topics[1].domain.display_name	Health Sciences
topics[1].subfield.id	https://openalex.org/subfields/2705
topics[1].subfield.display_name	Cardiology and Cardiovascular Medicine
topics[1].display_name	Heart Rate Variability and Autonomic Control
topics[2].id	https://openalex.org/T11021
topics[2].field.id	https://openalex.org/fields/27
topics[2].field.display_name	Medicine
topics[2].score	0.9648000001907349
topics[2].domain.id	https://openalex.org/domains/4
topics[2].domain.display_name	Health Sciences
topics[2].subfield.id	https://openalex.org/subfields/2705
topics[2].subfield.display_name	Cardiology and Cardiovascular Medicine
topics[2].display_name	ECG Monitoring and Analysis
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C2781067378
concepts[0].level	2
concepts[0].score	0.7487540245056152
concepts[0].wikidata	https://www.wikidata.org/wiki/Q17027399
concepts[0].display_name	Interpretability
concepts[1].id	https://openalex.org/C41008148
concepts[1].level	0
concepts[1].score	0.6740743517875671
concepts[1].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[1].display_name	Computer science
concepts[2].id	https://openalex.org/C522805319
concepts[2].level	2
concepts[2].score	0.6656538248062134
concepts[2].wikidata	https://www.wikidata.org/wiki/Q179965
concepts[2].display_name	Electroencephalography
concepts[3].id	https://openalex.org/C81363708
concepts[3].level	2
concepts[3].score	0.6558581590652466
concepts[3].wikidata	https://www.wikidata.org/wiki/Q17084460
concepts[3].display_name	Convolutional neural network
concepts[4].id	https://openalex.org/C154945302
concepts[4].level	1
concepts[4].score	0.6534498333930969
concepts[4].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[4].display_name	Artificial intelligence
concepts[5].id	https://openalex.org/C45347329
concepts[5].level	3
concepts[5].score	0.6107575297355652
concepts[5].wikidata	https://www.wikidata.org/wiki/Q5166604
concepts[5].display_name	Convolution (computer science)
concepts[6].id	https://openalex.org/C153180895
concepts[6].level	2
concepts[6].score	0.5173367857933044
concepts[6].wikidata	https://www.wikidata.org/wiki/Q7148389
concepts[6].display_name	Pattern recognition (psychology)
concepts[7].id	https://openalex.org/C2776401178
concepts[7].level	2
concepts[7].score	0.4951383173465729
concepts[7].wikidata	https://www.wikidata.org/wiki/Q12050496
concepts[7].display_name	Feature (linguistics)
concepts[8].id	https://openalex.org/C52622490
concepts[8].level	2
concepts[8].score	0.4554178714752197
concepts[8].wikidata	https://www.wikidata.org/wiki/Q1026626
concepts[8].display_name	Feature extraction
concepts[9].id	https://openalex.org/C50644808
concepts[9].level	2
concepts[9].score	0.43679141998291016
concepts[9].wikidata	https://www.wikidata.org/wiki/Q192776
concepts[9].display_name	Artificial neural network
concepts[10].id	https://openalex.org/C119857082
concepts[10].level	1
concepts[10].score	0.4148780107498169
concepts[10].wikidata	https://www.wikidata.org/wiki/Q2539
concepts[10].display_name	Machine learning
concepts[11].id	https://openalex.org/C15744967
concepts[11].level	0
concepts[11].score	0.17913419008255005
concepts[11].wikidata	https://www.wikidata.org/wiki/Q9418
concepts[11].display_name	Psychology
concepts[12].id	https://openalex.org/C41895202
concepts[12].level	1
concepts[12].score	0.0
concepts[12].wikidata	https://www.wikidata.org/wiki/Q8162
concepts[12].display_name	Linguistics
concepts[13].id	https://openalex.org/C138885662
concepts[13].level	0
concepts[13].score	0.0
concepts[13].wikidata	https://www.wikidata.org/wiki/Q5891
concepts[13].display_name	Philosophy
concepts[14].id	https://openalex.org/C118552586
concepts[14].level	1
concepts[14].score	0.0
concepts[14].wikidata	https://www.wikidata.org/wiki/Q7867
concepts[14].display_name	Psychiatry
keywords[0].id	https://openalex.org/keywords/interpretability
keywords[0].score	0.7487540245056152
keywords[0].display_name	Interpretability
keywords[1].id	https://openalex.org/keywords/computer-science
keywords[1].score	0.6740743517875671
keywords[1].display_name	Computer science
keywords[2].id	https://openalex.org/keywords/electroencephalography
keywords[2].score	0.6656538248062134
keywords[2].display_name	Electroencephalography
keywords[3].id	https://openalex.org/keywords/convolutional-neural-network
keywords[3].score	0.6558581590652466
keywords[3].display_name	Convolutional neural network
keywords[4].id	https://openalex.org/keywords/artificial-intelligence
keywords[4].score	0.6534498333930969
keywords[4].display_name	Artificial intelligence
keywords[5].id	https://openalex.org/keywords/convolution
keywords[5].score	0.6107575297355652
keywords[5].display_name	Convolution (computer science)
keywords[6].id	https://openalex.org/keywords/pattern-recognition
keywords[6].score	0.5173367857933044
keywords[6].display_name	Pattern recognition (psychology)
keywords[7].id	https://openalex.org/keywords/feature
keywords[7].score	0.4951383173465729
keywords[7].display_name	Feature (linguistics)
keywords[8].id	https://openalex.org/keywords/feature-extraction
keywords[8].score	0.4554178714752197
keywords[8].display_name	Feature extraction
keywords[9].id	https://openalex.org/keywords/artificial-neural-network
keywords[9].score	0.43679141998291016
keywords[9].display_name	Artificial neural network
keywords[10].id	https://openalex.org/keywords/machine-learning
keywords[10].score	0.4148780107498169
keywords[10].display_name	Machine learning
keywords[11].id	https://openalex.org/keywords/psychology
keywords[11].score	0.17913419008255005
keywords[11].display_name	Psychology
language	en
locations[0].id	doi:10.21203/rs.3.rs-4015916/v1
locations[0].is_oa	True
locations[0].source
locations[0].license	cc-by
locations[0].pdf_url	https://www.researchsquare.com/article/rs-4015916/latest.pdf
locations[0].version	acceptedVersion
locations[0].raw_type	posted-content
locations[0].license_id	https://openalex.org/licenses/cc-by
locations[0].is_accepted	True
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	https://doi.org/10.21203/rs.3.rs-4015916/v1
indexed_in	crossref
authorships[0].author.id	https://openalex.org/A5035804300
authorships[0].author.orcid	https://orcid.org/0009-0007-4004-2490
authorships[0].author.display_name	Ben Zhou
authorships[0].countries	CN
authorships[0].affiliations[0].institution_ids	https://openalex.org/I134738993
authorships[0].affiliations[0].raw_affiliation_string	Shandong University of Traditional Chinese Medicine
authorships[0].institutions[0].id	https://openalex.org/I134738993
authorships[0].institutions[0].ror	https://ror.org/0523y5c19
authorships[0].institutions[0].type	education
authorships[0].institutions[0].lineage	https://openalex.org/I134738993
authorships[0].institutions[0].country_code	CN
authorships[0].institutions[0].display_name	Shandong University of Traditional Chinese Medicine
authorships[0].author_position	first
authorships[0].raw_author_name	Ben Zhou
authorships[0].is_corresponding	False
authorships[0].raw_affiliation_strings	Shandong University of Traditional Chinese Medicine
authorships[1].author.id	https://openalex.org/A5100436158
authorships[1].author.orcid	https://orcid.org/0000-0003-4415-1268
authorships[1].author.display_name	Lei Wang
authorships[1].countries	CN
authorships[1].affiliations[0].institution_ids	https://openalex.org/I4210092817
authorships[1].affiliations[0].raw_affiliation_string	Suzhou Institute of Biomedical Engineering and Technology
authorships[1].institutions[0].id	https://openalex.org/I4210092817
authorships[1].institutions[0].ror	https://ror.org/00f58mx93
authorships[1].institutions[0].type	facility
authorships[1].institutions[0].lineage	https://openalex.org/I19820366, https://openalex.org/I4210092817
authorships[1].institutions[0].country_code	CN
authorships[1].institutions[0].display_name	Suzhou Institute of Biomedical Engineering and Technology
authorships[1].author_position	middle
authorships[1].raw_author_name	Lei Wang
authorships[1].is_corresponding	False
authorships[1].raw_affiliation_strings	Suzhou Institute of Biomedical Engineering and Technology
authorships[2].author.id	https://openalex.org/A5067316154
authorships[2].author.orcid	https://orcid.org/0000-0002-9432-7893
authorships[2].author.display_name	Chenyu Jiang
authorships[2].countries	CN
authorships[2].affiliations[0].institution_ids	https://openalex.org/I4210092817
authorships[2].affiliations[0].raw_affiliation_string	Suzhou Institute of Biomedical Engineering and Technology
authorships[2].institutions[0].id	https://openalex.org/I4210092817
authorships[2].institutions[0].ror	https://ror.org/00f58mx93
authorships[2].institutions[0].type	facility
authorships[2].institutions[0].lineage	https://openalex.org/I19820366, https://openalex.org/I4210092817
authorships[2].institutions[0].country_code	CN
authorships[2].institutions[0].display_name	Suzhou Institute of Biomedical Engineering and Technology
authorships[2].author_position	last
authorships[2].raw_author_name	Chenyu Jiang
authorships[2].is_corresponding	False
authorships[2].raw_affiliation_strings	Suzhou Institute of Biomedical Engineering and Technology
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://www.researchsquare.com/article/rs-4015916/latest.pdf
open_access.oa_status	gold
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Psychological Stress Classification Using EEG and ECG: A CNN Based Multimodal Fusion Model
has_fulltext	True
is_retracted	False
updated_date	2025-11-06T03:46:38.306776
primary_topic.id	https://openalex.org/T10429
primary_topic.field.id	https://openalex.org/fields/28
primary_topic.field.display_name	Neuroscience
primary_topic.score	0.9987000226974487
primary_topic.domain.id	https://openalex.org/domains/1
primary_topic.domain.display_name	Life Sciences
primary_topic.subfield.id	https://openalex.org/subfields/2805
primary_topic.subfield.display_name	Cognitive Neuroscience
primary_topic.display_name	EEG and Brain-Computer Interfaces
related_works	https://openalex.org/W2905433371, https://openalex.org/W2888392564, https://openalex.org/W4310278675, https://openalex.org/W4388422664, https://openalex.org/W4390569940, https://openalex.org/W4361193272, https://openalex.org/W2963326959, https://openalex.org/W4388685194, https://openalex.org/W2964954556, https://openalex.org/W2032664813
cited_by_count	1
counts_by_year[0].year	2025
counts_by_year[0].cited_by_count	1
locations_count	1
best_oa_location.id	doi:10.21203/rs.3.rs-4015916/v1
best_oa_location.is_oa	True
best_oa_location.source
best_oa_location.license	cc-by
best_oa_location.pdf_url	https://www.researchsquare.com/article/rs-4015916/latest.pdf
best_oa_location.version	acceptedVersion
best_oa_location.raw_type	posted-content
best_oa_location.license_id	https://openalex.org/licenses/cc-by
best_oa_location.is_accepted	True
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	https://doi.org/10.21203/rs.3.rs-4015916/v1
primary_location.id	doi:10.21203/rs.3.rs-4015916/v1
primary_location.is_oa	True
primary_location.source
primary_location.license	cc-by
primary_location.pdf_url	https://www.researchsquare.com/article/rs-4015916/latest.pdf
primary_location.version	acceptedVersion
primary_location.raw_type	posted-content
primary_location.license_id	https://openalex.org/licenses/cc-by
primary_location.is_accepted	True
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	https://doi.org/10.21203/rs.3.rs-4015916/v1
publication_date	2024-03-12
publication_year	2024
referenced_works	https://openalex.org/W4220754167, https://openalex.org/W2155686640, https://openalex.org/W2043559838, https://openalex.org/W3175145814, https://openalex.org/W2891457579, https://openalex.org/W4210730500, https://openalex.org/W1978690664, https://openalex.org/W3022345324, https://openalex.org/W4386825031, https://openalex.org/W4375858565, https://openalex.org/W4386879004, https://openalex.org/W4367187727, https://openalex.org/W2996057901, https://openalex.org/W4294796891, https://openalex.org/W3195539947, https://openalex.org/W3189250200, https://openalex.org/W4386357501, https://openalex.org/W2936978234, https://openalex.org/W2313602734, https://openalex.org/W2958197594, https://openalex.org/W4213031993, https://openalex.org/W2962858109, https://openalex.org/W2741907166, https://openalex.org/W2559463885, https://openalex.org/W4288056088, https://openalex.org/W4300942166, https://openalex.org/W3102455230, https://openalex.org/W4386996783
referenced_works_count	28
abstract_inverted_index.a	35, 49, 73
abstract_inverted_index.24	61
abstract_inverted_index.In	44, 132
abstract_inverted_index.an	12, 16
abstract_inverted_index.be	4
abstract_inverted_index.by	54, 103
abstract_inverted_index.in	6
abstract_inverted_index.is	11
abstract_inverted_index.of	37, 90, 136, 150
abstract_inverted_index.on	79
abstract_inverted_index.to	21, 83, 146
abstract_inverted_index.we	47
abstract_inverted_index.ECG	58, 93
abstract_inverted_index.EEG	56, 91
abstract_inverted_index.The	87
abstract_inverted_index.and	9, 18, 40, 57, 71, 92, 107, 120, 128
abstract_inverted_index.are	100, 123
abstract_inverted_index.for	15, 95, 114
abstract_inverted_index.it.	23
abstract_inverted_index.lot	36
abstract_inverted_index.the	85, 96, 111, 115, 134, 137, 143, 148, 151
abstract_inverted_index.two	104
abstract_inverted_index.each	106
abstract_inverted_index.four	97
abstract_inverted_index.from	60
abstract_inverted_index.into	110
abstract_inverted_index.need	14
abstract_inverted_index.that	29
abstract_inverted_index.then	108
abstract_inverted_index.this	45
abstract_inverted_index.time	39
abstract_inverted_index.were	140
abstract_inverted_index.with	67
abstract_inverted_index.based	78
abstract_inverted_index.data.	86
abstract_inverted_index.final	116
abstract_inverted_index.first	101
abstract_inverted_index.fused	109
abstract_inverted_index.layer	139
abstract_inverted_index.model	77, 113
abstract_inverted_index.tasks	66
abstract_inverted_index.there	10
abstract_inverted_index.using	142
abstract_inverted_index.which	125
abstract_inverted_index.5-fold	118
abstract_inverted_index.91.14%	127
abstract_inverted_index.91.97%	129
abstract_inverted_index.Neural	81
abstract_inverted_index.cannot	3, 41
abstract_inverted_index.detect	22
abstract_inverted_index.fusion	76
abstract_inverted_index.levels	70
abstract_inverted_index.manual	31
abstract_inverted_index.mental	64
abstract_inverted_index.method	20, 145
abstract_inverted_index.model.	152
abstract_inverted_index.models	105
abstract_inverted_index.paper,	46
abstract_inverted_index.stress	2, 51, 98
abstract_inverted_index.urgent	13
abstract_inverted_index.Network	82
abstract_inverted_index.achieve	126
abstract_inverted_index.dataset	53
abstract_inverted_index.feature	32
abstract_inverted_index.ignored	5
abstract_inverted_index.improve	147
abstract_inverted_index.machine	26
abstract_inverted_index.methods	28
abstract_inverted_index.propose	72
abstract_inverted_index.require	30, 34
abstract_inverted_index.signals	59, 94
abstract_inverted_index.today's	7
abstract_inverted_index.However,	24
abstract_inverted_index.classify	84
abstract_inverted_index.decision	75, 112
abstract_inverted_index.features	135
abstract_inverted_index.learning	27
abstract_inverted_index.research	38
abstract_inverted_index.society,	8
abstract_inverted_index.subjects	62
abstract_inverted_index.accuracy,	130
abstract_inverted_index.accuracy.	43
abstract_inverted_index.addition,	133
abstract_inverted_index.different	68
abstract_inverted_index.establish	48
abstract_inverted_index.extracted	102
abstract_inverted_index.guarantee	42
abstract_inverted_index.objective	17
abstract_inverted_index.arithmetic	65
abstract_inverted_index.categories	99
abstract_inverted_index.collecting	55
abstract_inverted_index.difficulty	69
abstract_inverted_index.extraction	33
abstract_inverted_index.multimodal	52, 74
abstract_inverted_index.performed,	124
abstract_inverted_index.performing	63
abstract_inverted_index.prediction	88
abstract_inverted_index.visualized	141
abstract_inverted_index.1D-Grad-CAM	144
abstract_inverted_index.Convolution	80
abstract_inverted_index.convolution	138
abstract_inverted_index.experiments	122
abstract_inverted_index.traditional	25
abstract_inverted_index.Psychological	1
abstract_inverted_index.four-category	50
abstract_inverted_index.probabilities	89
abstract_inverted_index.respectively.	131
abstract_inverted_index.cost-effective	19
abstract_inverted_index.classification,	117
abstract_inverted_index.cross-validation	119
abstract_inverted_index.interpretability	149
abstract_inverted_index.<title>Abstract</title>	0
abstract_inverted_index.Leave-Three-Subjects-Out	121
cited_by_percentile_year.max	95
cited_by_percentile_year.min	91
countries_distinct_count	1
institutions_distinct_count	3
sustainable_development_goals[0].id	https://metadata.un.org/sdg/16
sustainable_development_goals[0].score	0.6399999856948853
sustainable_development_goals[0].display_name	Peace, Justice and strong institutions
citation_normalized_percentile.value	0.57003364
citation_normalized_percentile.is_in_top_1_percent	False
citation_normalized_percentile.is_in_top_10_percent	False