Contactless diagnosis: research team develops innovative measurement technology to determine vital signs
A research team from TU Ilmenau and the Faculty of Medicine at the University of Duisburg-Essen (UDE) has jointly developed an optical measurement system that can be used to monitor the health status of chronically ill or highly contagious people using vital parameters such as body temperature, respiratory rate or oxygen saturation without contact. The system, which was developed in the NEON project with funding from the German Research Foundation (DFG) and has already been tested for monitoring premature babies and in sleep medicine, is to be further developed and scaled down in other projects so that it can also be used in telemedicine in the future. In this way, the risk of infection could be minimized and treatment costs reduced, but critical deterioration of sick people could also be detected more quickly and a transfer to hospital could be arranged.
Do I have a fever? Is my blood pressure ok? And how high is my pulse? Doctors can reliably and quickly determine how well or badly our body is doing at any given moment by measuring vital signs such as body temperature, oxygen saturation, respiratory rate or heart rate. Vital sign monitoring is particularly important for chronically ill people and patients in intensive care units, whose vital body signals need to be monitored for long periods or even around the clock.
Whether a clip on the finger or adhesive electrodes: All of the measurement methods used have one thing in common: a sensor must be attached to the body in order to obtain the necessary data. "This is not only uncomfortable for the people who have to wear the measuring devices, but also often leads to pain or irritation of the skin, especially in particularly sensitive people. In addition, pathogens can be transmitted when the devices are attached, putting patients and staff at risk," says Professor Gunter Notni, Head of the Department of Quality Assurance and Industrial Image Processing at TU Ilmenau, explaining the researchers’ motivation.
Measurements independent of movement, skin color and visible light
How can these risks be avoided and how can vital parameters be determined without contact, i.e. without measuring heads on the body, as simply, accurately, quickly and, above all, robustly against external interference and even in the dark? This was the question posed by the research team led by the University of Duisburg-Essen. Contactless camera systems for measuring vital parameters already exist. "However, these measurements currently take a very long time or are very susceptible to interference," says Professor Notni: "This means that as soon as a person moves, their facial expressions change, glasses or hair are in the way or the lighting in the room changes, the measurement data becomes inaccurate."
In three years of research, the scientists have therefore developed and tested a multimodal 3D camera system based on innovative sensor concepts that addresses precisely these challenges: It combines real-time 3D imaging with 2D color images, thermal images and so-called multispectral images in the non-visible near-infrared spectral range, i.e. images at different wavelengths, to generate multimodal image data in real time that describe the properties of different body regions so precisely that a wide range of vital parameters can also be derived from them without contact.
To do this, the researchers first identified the regions of the face that are particularly meaningful for the required values: the forehead, the nostrils and the corners of the eyes. With the help of the various camera systems and already known methods such as image-based photoplethysmography (PPG), they were able to measure the blood flow to the skin at these points, derive the heart rate and oxygen saturation and determine the respiratory rate during breathing by thermographically analyzing the air flows at the nostrils.
One of the biggest challenges was the constant movement of the test subjects: "In order to be able to track the measurement regions at all times, we took real-time 3D images, among other things," says Professor Notni. "This allowed us to track: ’How did the head turn’ Where is the nose right now?’ and align the evaluation regions for the camera systems accordingly at any time."
The researchers also paid particular attention to the multispectral images: "Currently, most contactless methods only use RGB cameras, i.e. color images, to measure the body signals of the skin. In order to be able to use our measurement system in the dark, i.e. at night without any visible light and regardless of a person’s skin color, we also recorded images in other wavelengths such as the near infrared range at 780 and 940 nanometers."
In order to synchronize and combine the images captured simultaneously with 3D and several 2D camera systems in the various spectral ranges and imaging modalities in real time, i.e. to superimpose them with pixel accuracy, the researchers had to geometrically calibrate the multimodal image sensor in such a way that they could ultimately generate reliable information about the observed objects and their properties using physics-based and algorithmic approaches and AI methods.
Possible use in sleep and telemedicine
The researchers tested their prototype not only in the laboratory but also in two realistic application scenarios: in the neonatal monitoring area for premature babies at Jena University Hospital and in the Center for Sleep and Telemedicine at the Ruhrlandklinik at Essen University Hospital, one of the largest centers for sleep medicine in Germany. While the neonatology department carried out contactless measurement series on premature babies with particularly sensitive skin, the multimodal camera system in Essen was primarily used in the treatment of sleep disorders caused by breathing interruptions.
"We were able to achieve a data availability of over 60 percent with long-term monitoring during the entire sleep phase, i.e. up to eight hours, and also determine the patients’ breathing interruptions very precisely by taking thermal images of the nasal area to determine the respiratory rate and measuring the oxygen saturation in the forehead area," says Professor Notni. In addition, the research team was able to determine the body temperature very well by measuring the thermal signals in the corners of the eyes using a thermal imaging camera.
The researchers’ vision is that their technology can be used in future not only for acute monitoring of particularly sensitive or infectious patients in hospitals, but also for long-term monitoring of chronic diseases such as sleep apnoea at home. With the help of the algorithms developed in the NEON project, they are therefore already working with partners from the region in the RUBIN alliance AMI to further miniaturize the system using the principle of the so-called multi-aperture camera: "Our goal is to integrate and miniaturize the camera systems in a camera so that patients can record their vital data themselves at home, for example via a camera integrated in a mirror or elsewhere, and can move around normally in everyday life."
The researchers can also imagine remote diagnostic stations in the future. Professor Notni: "For example, if I’m not feeling well, I could have my vital signs measured in a kind of telephone booth in my neighborhood and have them transmitted directly to my family doctor. But the technology could also be used in the context of human-machine interaction in the future to continuously record a user’s condition without contact and avoid potential dangers that could arise from illness or fatigue."
