Guest Editorial: Integrating sensor fusion and perception for human–robot interaction Article Swipe

MAKE IMPACT

Hang Su , Jing Guo , Wen Qi , Mingchuan Zhou , Yue Chen · YOU? · · 2021 · Open Access · · DOI: https://doi.org/10.1049/ccs2.12031

This is the Special Issue ‘Integrating Sensor Fusion and Perception for Human–Robot Interaction’ of IET Cognitive Computation and System that introduces the latest advances in sensor fusion and perception in the human–robot interaction (HRI) field. In recent years, as intelligent systems have developed, HRI has attracted increasing research interest. In many areas, including factories, rehabilitation robots and operating rooms, HRI technology can be exploited to enhance safety by using intelligence for human operations. However, both available practical robotic systems and some ongoing investigations lack intelligence due to their limited capabilities in perceiving their environment. Nowadays, the HRI method usually focusses on a single sensing system without integrating algorithms and hardware, such as tactile perception and computer vision. Sensor fusion and perception with artificial intelligence (AI) techniques have been successful in environment perception and activity recognition by fusing information from a multi-modal sensing system and selecting the most appropriate information to perceive the activity or environment. Consequently, combining the technique of multi-sensor fusion and perception for HRI is an exciting and promising topic. This Special Issue aims to track the latest advances and newly appeared technology in the integrated sensor fusion and perception for HRI. After careful peer reviews and revision, four representative papers were accepted for publication in this Special Issue. These papers represent four important application areas of multi-sensor fusion and perception technology and can be assigned into four topics. The related summary of every topic is given below. We strongly recommend reading the entire paper if interested. They will bring some new ideas and inspire the mind. In the paper ‘Deep learning techniques-based perfection of multi-sensor fusion oriented human-robot interaction system for identification of dense organisms’, Li et al. present an HRI system based on deep learning and sensors' fusion to study the species and density of dense organisms in the deep-sea hydrothermal vent. In this paper, several deep learning models based on convolutional neural network (CNN) are improved and compared to study the species and density of dense organisms in deep-sea hydrothermal vent, which are fused with related environmental information provided by position sensors and conductivity–temperature–depth (CTD) sensors, so as to perfect the multi-sensor fusion-oriented HRI system. First, the authors combined different meta-architectures and different feature extractors and obtained five object identification algorithms based on CNN. Then, they compared the computational cost of feature extractors and weighed the pros and cons of each algorithm from mean detection speed, correlation coefficient and mean class-specific confidence score to confirm that Faster Region-based CNN (R-CNN)_InceptionNet is the best algorithm applicable to the hydrothermal vent biological dataset. Finally, they calculated the cognitive accuracy of rimicaris exoculata in dense and sparse areas, which were 88.3% and 95.9%, respectively, to analyse the performance of the Faster R-CNN_InceptionNet. Experiments show that the proposed method can automatically detect the species and quantity of dense organisms with higher speed and accuracy. And it is feasible and of realistic value that the improved multi-sensor fusion-oriented HRI system is used to help biologists analyse and maintain the ecological balance of deep-sea hydrothermal vents. Integrating sensor fusion and perception is not limited to the extraction and processing of the physic data. It also plays an important role in multi-system coupling. In the paper ‘Research on intelligent service of customer service system’, Nie et al. illustrate a new generation of customer service systems based on the sense coupling of the outbound call system, enterprise internal management system and knowledge base. This study introduces the principle of the outbound call system, the enterprise internal management system, and the knowledge base and explains the network structure of the intelligent customer service system. Also, the methods of accessing the intelligent customer service system and the whole workflow are described. Through data sharing and information exchange between multiple systems, the new customer service system proposed achieves the perceptual integration of the outbound call system, the enterprise internal management system and the knowledge base to provide service for customers intelligently. Based on the application of cloud service and IoT technology, the intelligent customer service system establishes a dynamically updated knowledge base and forms a management model dominated by the knowledge base. In recent years, wearable sensors have developed fast, especially in the medical and health field. More mature commercial wearable sensors have appeared and generated a new network formation—the body area network (BAN). The BAN is composed of every wearable device network on the body to share information and data, which is applied in medical and health devices, especially in intelligent clothing. This Special Issue includes a review article on wearable sensor and body area network by Ren et al. Based on the wearable sensor in the critical factor of wearable device fusion, this paper analyses the classification, technology, and current situation of a wearable sensor, discusses the problems of a wearable sensor for the BAN from the aspects of human–computer interaction experience, data accuracy, multiple interaction modes, and battery power supply, and summarises the direction of multi-sensor fusion, compatible biosensor materials, and low power consumption and high sensitivity. Furthermore, a sustainable design direction of visibility design, identification of use scenarios, short-term human–computer interaction, interaction process reduction and integration invisibility are introduced. Augmented reality is one of the most inspiring technology in recent years. There is no doubt that augment reality will lead the trend of the immerse application in the industry and medical field. This Special Issue includes a paper on augmented reality display of neurosurgery craniotomy lesions based on feature contour matching by Hao et al. In this paper, an augmented reality display method for neurosurgical craniotomy lesions based on feature contour matching is proposed, which uses an augmented reality display method to provide doctors with accurate lesion information. It can visualise the patient's intracranial information and help doctors plan the path of scalp cutting and craniectomy. This method also performs non-rigid matching for the patient, eliminates additional injury to the patient, reduces the extra work of doctors to paste marking points for the patient, and reduces the burden of multiple medical scans for the patient. Through experiments to compare the feature point cloud matching and feature contour matching methods, it is proved that the feature contour matching method has a better display effect. In addition, a user interface is designed. The doctor can determine the patient's personal information through the text displayed in the upper left corner of the interface and zoom in, zoom out, and rotate the virtual model on a mobile terminal screen by pressing buttons. It provides a visual basis for the doctor's preoperative preparation. The method described in this article effectively improves the efficiency of the doctor's operation as well as patient safety. The proposed augmented reality matching method based on feature contours also provides basic theoretical help for applying augmented reality to neurosurgery in the future. All of the papers selected for this Special Issue show the significant effect and application potential of sensor fusion and perception in HRI application. Multi-sensor fusion and perception can effectively improve system accuracy, increase stability and improve the human–computer interaction experience. There are still many challenges in this field that require future research attention, such as the fusion method and the fusion result evaluation. With further development, the integration of sensor fusion and perception in HRI will have broad application. Examples of published guest editorials are given below for your reference: https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/iet-gtd.2020.1493 https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/iet-pel.2020.0051 https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/iet-rsn.2020.0089 Dr. Hang Su received his M.Sc. degree in control theory and control engineering in South China University of Technology, Guangzhou, China, in 2015, and the Ph.D. degree in Bioengineering from Politecnico di Milano, Milano, Italy, in 2019. He participated in the EU-funded project (SMARTsurg) in the field of Surgical Robotics. Dr. Hang Su is currently working in the Department of Electronics, Information and Bioengineering (DEIB) of Politecnico Di Milano. He is also working as an Assistant Professor in the Institute of Advanced Technology, University of Science and Technology of China, China. He is currently Associate Editor for Frontiers in Neuroscience, Cognitive Computation and Systems, and Frontiers in Neurorobotics. He is currently a Program Chair of IEEE International Conference on Advanced Robotics and Mechatronics (ICARM 2021). He also severs as the Associate Editor for the IEEE International Conference on Robotics and Automation (ICRA) and the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), the IEEE International Conference on Robot and Human Interactive Communication (Ro-man) and the IEEE International Conference on Advanced Robotics and Mechatronics (ICARM), and the Guest Associate Editor for a couple of journals, like IEEE Robotics and Automation Letters, Complexity, Actuators, Mathematical Problems in Engineering, Sensors, Frontiers in Robotics and AI etc. He was a recipient of the Best Paper Award in Advanced Robotics at the IEEE International Conference on Advanced Robotics and Mechatronics in 2020 and the ICRA Travel Award funded by the IEEE Robotics and Automation Society in 2019. His main research interests include control and instrumentation in medical robotics, human–robot interaction, surgical robotics, deep learning, bilateral teleoperation etc. Jing Guo obtained his Ph.D. degree from LIRMM, CNRS-University of Montpelier, France in 2016. He received his master and bachelor degree from Guangdong University of Technology in 2009 and 2012, respectively. He has been research fellow at National University of Singapore (NUS) during 2016–2018. He is an Associate Professor affiliated with Guangdong University of Technology. His current research interests include robotic control and learning, haptic bilateral teleoperation, and surgical robotics. He has served as guest editor for IEEE RA-L, Frontiers in Robotics & AI etc. Dr. Wen Qi received her M.Sc. degree in control engineering from South China University of Technology, Guangzhou, China, in 2015 and her Ph.D. degree in Bioengineering as a member of the Laboratory of Biomedical Technologies (TBMLab) from Politecnico di Milano, Milano, Italy, in 2020. Dr. Wen Qi has served as a reviewer for over 30 scientific journals, such as IEEE Transaction on Biomedical Engineering and IEEE Engineering in Medicine and Biology Society. She has published more than 20 papers in e-health, deep learning, evolutionary algorithm, human–robot interaction, teleoperation and robotics. She was a recipient of the Best Paper Award in Advanced Robotics at the IEEE International Conference on Advanced Robotics and Mechatronics in 2020. In addition, her first-authored paper was awarded the finalist of the Best Paper Award on Control Applications on IEEE WCICA 2014. Her main research interests include machine learning, deep learning and signal processing algorithms in wearable medical devices. Mingchuan Zhou received a Ph.D. degree in computer science from the Technical University of Munich, Munich, Germany, in 2020. He was a visiting scholar at the Laboratory for Computational Sensing and Robotics (LCSR), Johns Hopkins University, USA, 2019. He was a joint postdoc at the Institute of Biological and Medical Imaging (IBMI) of the Helmholtz Centre, Munich and Chair for Computer Aided Medical Procedures Augmented Reality (CAMP) at the Technical University of Munich from 2019 to 2021. He is currently an assistant professor leading the multi-scale robotic manipulation lab for agriculture in Zhejiang University. He is also a guest senior researcher at the Department of Computer Science, Technical University of Munich. His research interests include the autonomous system, agricultural robotics, medical robotics and image processing. He is the Associate Editor for IEEE International Conference on Advanced Robotics and Mechatronics (ICARM). He was the winner for third Place and Golden Prize for the sixth China International ‘Internet+’ Competition 2020, a member of the champion team for agricultural robotics competition in 2020 UK Agri-EPI Centre agri-tech hackathon, and IEEE-ICRA RAS Travel Grant in IEEE & RAS 2019. Dr. Zhou was the recipient of Finalist of Best Paper Awards IEEE ROBIO in 2017 and best poster award in the IEEE ICRA 2021 workshop of Task-Informed Grasping: Agri-Food manipulation (TIG-III). Yue Chen received a B.S. in vehicle engineering from Hunan University, Hunan, China, in 2010, M.Phil. in mechanical engineering from Hong Kong Polytechnic University, Hung Hom, Hong Kong, in 2013, and Ph.D. in Mechanical Engineering, Vanderbilt University, Nashville, TN, USA, in 2018. He is an Assistant Professor at the Department of Biomedical Engineering, Georgia Tech/Emory, Atlanta. He has been an Assistant Professor at the Department of Mechanical Engineering, University of Arkansas, Fayetteville, since 2018. His current research interests include medical robotics and image-guided therapy.

Type

editorial

Language

en

Landing Page

https://doi.org/10.1049/ccs2.12031

OA Status

gold

Related Works

10

OpenAlex ID

https://openalex.org/W3197711914

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W3197711914

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.1049/ccs2.12031

Digital Object Identifier

Title

Guest Editorial: Integrating sensor fusion and perception for human–robot interaction

Work title

Type

editorial

OpenAlex work type

Language

en

Primary language

Publication year

2021

Year of publication

Publication date

2021-08-28

Full publication date if available

Authors

Hang Su, Jing Guo, Wen Qi, Mingchuan Zhou, Yue Chen

List of authors in order

Landing page

https://doi.org/10.1049/ccs2.12031

Publisher landing page

Open access

Yes

Whether a free full text is available

OA status

gold

Open access status per OpenAlex

OA URL

https://doi.org/10.1049/ccs2.12031

Direct OA link when available

Concepts

Sensor fusion, Perception, Robot, Human–computer interaction, Computer science, Artificial intelligence, Human–robot interaction, Field (mathematics), Robotics, Active perception, Psychology, Pure mathematics, Mathematics, Neuroscience

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload