sensors Article

Vital Sign Monitoring and Mobile Phone Usage Detection Using IR-UWB Radar for Intended Use in Car Crash Prevention Seong Kyu Leem † , Faheem Khan † and Sung Ho Cho * Department of Electronics and Computer Engineering, Hanyang University, 222 Wangsimini-ro, Seongdong-gu, Seoul 04763, Korea; [email protected] (S.K.L.); [email protected] (F.K.) * Correspondence: [email protected]; Tel.: +82-(02)-2220-0390 † Co-first authors, these authors contributed equally to this work. Academic Editors: Jesús Ureña, Álvaro Hernández Alonso and Juan Jesús García Domínguez Received: 23 March 2017; Accepted: 25 May 2017; Published: 30 May 2017

Abstract: In order to avoid car crashes, active safety systems are becoming more and more important. Many crashes are caused due to driver drowsiness or mobile phone usage. Detecting the drowsiness of the driver is very important for the safety of a car. Monitoring of vital signs such as respiration rate and heart rate is important to determine the occurrence of driver drowsiness. In this paper, robust vital signs monitoring through impulse radio ultra-wideband (IR-UWB) radar is discussed. We propose a new algorithm that can estimate the vital signs even if there is motion caused by the driving activities. We analyzed the whole fast time vital detection region and found the signals at those fast time locations that have useful information related to the vital signals. We segmented those signals into sub-signals and then constructed the desired vital signal using the correlation method. In this way, the vital signs of the driver can be monitored noninvasively, which can be used by researchers to detect the drowsiness of the driver which is related to the vital signs i.e., respiration and heart rate. In addition, texting on a mobile phone during driving may cause visual, manual or cognitive distraction of the driver. In order to reduce accidents caused by a distracted driver, we proposed an algorithm that can detect perfectly a driver’s mobile phone usage even if there are various motions of the driver in the car or changes in background objects. These novel techniques, which monitor vital signs associated with drowsiness and detect phone usage before a driver makes a mistake, may be very helpful in developing techniques for preventing a car crash. Keywords: collision prevention; car safety; driver behavior; drowsiness; distraction; vital signs; IR-UWB radar; phone detection

1. Introduction Every year, according to the statistics, more than a million people die on the world’s roads due to car crashes and the cost of dealing with the consequences of these crashes runs to billions [1]. Researchers are trying to reduce these numbers by employing different techniques [2–8]. Many people are saved during collisions by air bags, seat belts, and the car frames that can absorb a huge amount of energy during a car crash [9]. However, the ultimate solution is to keep cars from smashing into each other in the first place. Adaptive cruise control (ACC) systems use laser beams or radar to measure the distance between the cars and their relative speed [10]. All of the ACC systems available today are built around sensors that detect the vehicle ahead with either radar or light detecting and ranging (LIDAR), the laser-based analog to radar [11]. These systems do detect and prevent some types of driver error. However, technologies cannot detect and prevent all of the mistakes that drivers make

Sensors 2017, 17, 1240; doi:10.3390/s17061240

www.mdpi.com/journal/sensors

Sensors 2017, 17, 1240

2 of 25

and can result in collisions. The major driver mistakes are caused by drowsiness [12] and mobile phone usage during driving. According to a recent study, 20% of crashes occurred due to drowsiness of the driver [13]. Continuous monotonous driving for a long time causes drowsiness, which may result in fatal road accidents. For long haul/distance drivers, fatigue and drowsiness can hardly be avoided [13]. Thus, research works are carried out to investigate approaches to monitor driver condition. Various methods are used to detect if a driver is tired or falling asleep. Researchers have attempted to determine driver drowsiness using the following measures: (1) vehicle-based measures [14,15] (2) behavioral measures [16–18] and (3) physiological measures [19–23]. Mercedes-Benz pioneered one of the first techniques, which uses a computer algorithm that compares the steering behavior of a driver to the trained behavior at the start of the trip [9]. Other systems monitor the location of car within its lane of travel, looking for erratic maneuvers indicative of inattention. Some researchers also track the driver eye movement with an in-car camera, noting rapid or prolonged eye blinks. In [14,15], the researchers used metrics like lane position, steering wheel position, driving speed and car yaw angle for detecting the drowsiness of drivers. These metrics are useful, but this method is an alarm for the mistake after the happening, so it cannot prevent in advance the potential mistakes caused by drowsiness. An eye-tracking based scheme is proposed for detection of driver drowsiness using an unscented Kalman filter [16]. In reference [17], the researchers used facial image sequences to detect the driver fatigue based on multiscale dynamic features, but camera-based techniques can’t perform in dark environments. Biomedical variables like respiration rate related to the nervous system may provide direct information of driver physiological conditions. Hence, they are extremely important to determine the drowsiness cycle and anticipate risky situations during driving [22–24]. In [25], the tracking and localization of cyclists using UWB technology is discussed, however, the researchers did not consider the crashes due to cars and other big vehicles and it has no method to detect the mobile phone usage or drowsiness detection of the drivers. Electroencephalogram (EEG) was used by researchers [18] to detect driver drowsiness, but in case of EEG measurement electrodes are used for the measurements which require physical contact with the person. The researchers in reference [19] used EEG, electrocochleogram (EOG), and electrocardiogram (ECG) signals for detection of driver drowsiness based on the physiological conditions of the driver, but this solution is really uncomfortable as the driver has to physically wear the devices. The ANSWatch device was used for detection of heart rate in reference [20], but it requires the driver to always wear the watch. Researchers in [21–23] also used EEG and/or ECG for physiological condition monitoring which is cumbersome for the driver. Vital sign measurement for drivers using radar can solve all the above problems at the same time. The radar operates robustly in the dark, and by measuring the vital signs in a noncontact manner without any inconvenience to the driver, an alarm can be generated in advance before the driver makes a mistake due to drowsiness. In our work, we monitored the vital signs, i.e., respiration and heart rate of the driver inside a real car using IR-UWB radar. The UWB technology offer several benefits such as accurate ranging, robustness to multipath interference, better penetration and rejection of interferences [26,27]. Although there are algorithms in the literature [28–31] that deal with vital sign measurement, however, the conventional algorithms do not estimate the vital signs during the motion state. In our paper, we present a novel algorithm that can extract the vital signal from the radar signals reflected from the human body which are slightly distorted by the driver motions due to steering use or electronic and mechanical devices operation. The main concept of our vital sign estimation algorithm is that the vital signals are reflected from many points of the chest and abdominal area of the human sitting in front of the radar. Although the driving activity decreases the signal to noise ratio (SNR) of some part of the vital signal reflected from some area of the body, but still it doesn’t decrease the SNR of all the signals reflected from the body. Our goal is to search for those segments of the signals which have the best vital signs information and which are least distorted by the motion caused by driver activity. To this end, we first determined those signals which have better sinusoidal shapes and hence

Sensors 2017, 17, 1240

3 of 25

better vital signal information by using the sinusoidal fitting algorithm. Then we segment all those sinusoidal signals into sub-signal components and select those sub-signal components which have better correlation with the vital signal. In this way, we construct our desired vital signal from many signals reflected from the driver body. Then FFT algorithm is applied to the constructed vital signal to find the respiration and heart rate values. Another main cause of collisions is mobile phone use during driving. Distracted driving caused due to the usage of cell phone (texting or speaking), is associated with 26% of all crashes and is increasing in frequency [32]. The mobile phone usage during driving results in restriction of sight; limiting drivers’ ability to monitor the road since their line of vision is focused on the handset [33]. It also reduces the concentration level and situational awareness [34–36]. Motivated by its impact on public safety and property, several state and federal governments have enacted regulations that prohibit driver mobile phone usage while driving [35]. In [36] a computer vision-based method for detection of driver cell phone usage by using a near infrared (NIR) camera system inside the car was presented. Smith et al. [37] have presented an algorithm for the detection of handset usage by the driver by analyzing images taken inside the car. Xu et al. [38] have proposed a machine-learning-based method for detecting driver cell phone usage using a camera system directed at the vehicle’s front windshield. These methods, however, are dependent on a camera, which may not provide good performance in dark environments. In our work, we proposed an algorithm to detect the use of mobile phones while driving by using IR-UWB radar that is unaffected by the light conditions. The proposed algorithm can accurately detect the use of mobile phones even in environments where driver’s various movements or change of background objects occur inside the car. We propose a dual mode background subtraction algorithm using two different clutter signals for optimum detection, which is explained in Section 3.2. The main contribution of our work is that we propose an algorithm using a single IR-UWB radar for monitoring the vital signs and detecting the mobile phone usage of the driver that can be used to develop technologies to prevent accidents caused by drowsiness driving and cell phone use. Our main purpose of the monitoring of driver’s vital signs is to provide a method for future researchers to use it for drowsiness detection so that the accidents due to drowsiness may be reduced. We propose an algorithm to accurately measure vital signs during the motion caused by the driving activities. We also suggest a way to detect the use of a cell phone that takes the driver’s attention away. The proposed algorithm suggests a new method to optimally detect drivers’ cell phone usage considering various situations that can occur in the car. The technologies presented in this paper may become one of the technologies to save human life by being utilized in application technologies for preventing drowsiness driving and cell phone use during driving that are the root cause of many road crashes. In Section 2 of the paper, the problems related to the previous methods for vital measurement and mobile phone detection are discussed. The signal preprocessing such as clutter removal is also discussed in Section 2. Section 3 is about our proposed algorithms for vital sign measurement during various driving conditions and phone detection in a specific area inside a car. Section 4 is related to the experimental results and in Section 5 the paper is concluded. 2. Problem Description and Signal Preprocessing 2.1. Problem Statement In this paper, we cover two topics to prevent car crashes. One is to measure the driver’s vital signs to prevent drowsy driving and the other is to detect the use of mobile phones to prevent accidents caused by their use while driving. 2.1.1. Monitoring of Vital Signs Although there exist algorithms in the literature to measure the vital signs of a human [28–31], vital sign measurement during driving is a totally different problem as it involves different hand

Sensors 2017, 17, 1240

4 of 25

and body movements. The vital signs obtained during motion are deteriorated by the motion and vital signs can’t be measured from the signal. An example of such a signal is shown in Figure 1. 2017, 17, 4 ofof 25 TheSensors detection of 1240 the vital signs is difficult due to the movement of the driver as there is no trace any periodicity in the received signal, which is a characteristic of vital signs. Researchers in [28] have discussed the of of a non-stationary human, but but the approach is very discussed the problem problemof ofvital vitalsign signmeasurement measurement a non-stationary human, the approach is simple as they detect the motion of the body and lock the vital sign measurement until the body very simple as they detect the motion of the body and lock the vital sign measurement until the body motion is is stopped. stopped. In In this this work, work, however, however, we we not not only only detect detect the the body body motion motion but but also also measure measure the the motion vital signs during the motion period. The conventional method of extraction of vital signs at a fast vital signs during the motion period. The conventional method of extraction of vital signs at a fast time index index of of maximum maximum variance variancedoesn’t doesn’talways alwayshold holdtrue truewhen whenrandom randombody bodymovement movementisisinvolved, involved, time therefore, we we have have used used the the data data fitting fitting method method (see (see Section Section 3.1.1) 3.1.1) to to find find those those signals signals located located at at fast fast therefore, time indexes which gives better sinusoid fits. The vital signals are constructed from those better fit time indexes which gives better sinusoid fits. The vital signals are constructed from those better fit signals which exist even in presence of motion of some parts of the human body. After the vital signal signals which exist even in presence of motion of some parts of the human body. After the vital signal construction, the the conventional conventional Fourier Fourier transform transform technique technique is is applied applied to to find find the the breathing breathing and and the the construction, heart rate. heart rate. 1.5

noisy vital signal obtained by conventional algorithm

1

Magnitude

0.5 0 -0.5 -1 -1.5 0

1

2

3

4 5 6 Slow time (seconds)

7

8

9

10

Figure 1. 1. Noisy Noisy vital vital signal signal obtained obtained by by conventional conventional algorithm algorithm during during motion motion period. period. Figure

2.1.2. Detection of Phone Usage 2.1.2. Detection of Phone Usage A cellular phone are composed of electronic parts, PCB, and antenna. Their main components A cellular phone are composed of electronic parts, PCB, and antenna. Their main components are are metals such as copper, so cell phones have a very large radar cross-section (RCS). Although a metals such as copper, so cell phones have a very large radar cross-section (RCS). Although a large large RCS makes it easy to detect a cell phone, there are some difficulties in detecting the use of a cell RCS makes it easy to detect a cell phone, there are some difficulties in detecting the use of a cell phone phone in the car. First, in the mobile phone sensing area of a car, various objects besides a mobile in the car. First, in the mobile phone sensing area of a car, various objects besides a mobile phone can phone can appear. For example, several movements such as hand movements to press a button or appear. For example, several movements such as hand movements to press a button or grab beverage grab beverage bottles to drink can occur, and the radar can detect it. In fact, these short time bottles to drink can occur, and the radar can detect it. In fact, these short time movements may be movements may be necessary to drive and those are not dangerous actions that distract the driver, necessary to drive and those are not dangerous actions that distract the driver, therefore any detection therefore any detection algorithm should ignore them. algorithm should ignore them. Secondly, in a really dangerous situation, like when the driver is staring at his cell phone for a Secondly, in a really dangerous situation, like when the driver is staring at his cell phone for while, there is little movement of the cell phone. If only the conventional background removal a while, there is little movement of the cell phone. If only the conventional background removal method [39] in Section 2.2 is used to detect the mobile phone in this situation, the mobile phone may method [39] in Section 2.2 is used to detect the mobile phone in this situation, the mobile phone may be recognized as a background object and may not be detected. Considering the above two situations, be recognized as a background object and may not be detected. Considering the above two situations, an algorithm for detecting the use of mobile phones in a car should ignore very short time actions an algorithm for detecting the use of mobile phones in a car should ignore very short time actions that are not dangerous to the driver and generate an alarm for any type of mobile phone use which that are not dangerous to the driver and generate an alarm for any type of mobile phone use which is is dangerous to the driver. dangerous to the driver. For a momentary movement that does not interfere with the driving, the alarm can be disabled For a momentary movement that does not interfere with the driving, the alarm can be disabled by by measuring the duration of the detection. In other words, we can ignore the detection of movement measuring the duration of the detection. In other words, we can ignore the detection of movement that appears for less than a certain time. In addition, to solve the problem that the signal reflected that appears for less than a certain time. In addition, to solve the problem that the signal reflected from the mobile phone is removed as background noise when there is almost no movement of the from the mobile phone is removed as background noise when there is almost no movement of the mobile phone, such as watching the screen quietly, we proposed a dual-mode background mobile phone, such as watching the screen quietly, we proposed a dual-mode background subtraction subtraction algorithm in Section 3.2. In this algorithm, background removal is performed in two ways, and mobile phone usage is detected by using two received signals obtained by each background removal method. In the dual-mode background subtraction algorithm, immediately after the handset is detected, the clutter signal is not updated and the background is removed by using the clutter immediately before sensing the cell phone. In this way, it is possible to detect mobile phones with minimal

Sensors 2017, 17, 1240

5 of 25

algorithm in Section 3.2. In this algorithm, background removal is performed in two ways, and mobile phone usage is detected by using two received signals obtained by each background removal method. In the dual-mode background subtraction algorithm, immediately after the handset is detected, the clutter signal is not updated and the background is removed by using the clutter immediately before sensing the cell phone. In this way, it is possible to detect mobile phones with minimal movement. However, when the mobile phone is detected in this way, the mobile phone detection alarm will Sensors 2017, 17, 1240 5 of 25 continue to be sounded when a new background object is placed in the detection area. To solve this problem, we have noted the difference between the received signal from the new background object movement. However, when the mobile phone is detected in this way, the mobile phone detection and the received signal usage when of theamobile phone. There is is a slight evenarea. if mobile alarm will continue to from be sounded new background object placedmovement, in the detection To phone is held static like when just looking at a screen. Therefore, when the background is removed solve this problem, we have noted the difference between the received signal from the new while updatingobject the clutter signal, the received signalusage is small detected. On There the other background and the received signal from of but the is mobile phone. is ahand, slightthe new background object has no motion, so if the background is removed while updating the movement, even if mobile phone is held static like when just looking at a screen. Therefore, whenclutter the signal, the received signalwhile is notupdating detected the at all. By using differences, it is small possible distinguish background is removed clutter signal,these the received signal but to is detected. a new background object from a mobile phone. On the other hand, the new background object has no motion, so if the background is removed while updating the clutter signal, the received signal is not detected at all. By using these differences, it is 2.2.possible Signal Pre-Processing to distinguish a new background object from a mobile phone. The loopback filter is applied to remove the clutter from the raw signal [39]. The loopback filter is 2.2. Signal Pre-Processing represented by the following equations: The loopback filter is applied to remove the clutter from the raw signal [39]. The loopback filter ck ( t ) = ∝ ck −1 ( t ) + (1 − ∝ ) r k ( t ) (1) is represented by the following equations: c ( ) =∝ c ( ) + (1−∝) ( ) yk ( t ) = rk ( t ) − ck ( t )

(1)

(2)

(2) )− ( ) y ( ) “=∝r ”( is In Equations (1) and (2), the symbol a cconstant used for weighting. The “ ∝ ” value in In Equations (1)background and (2), the symbol " ∝ "was is a set constant for weighting. The " ∝ " whereas value infor Equation (2) used for subtraction to 0.97used for vital signs monitoring Equation (2) used for background subtraction was set to 0.97 for vital signs monitoring whereas for the mobile phone detection it was kept 0.8. A low alpha value means that the background removal theismobile phone detection was kept A low alpha value meansthe that the background removal rate fast but vulnerable to itnoise, and 0.8. if the alpha value is large, background removal rate is rate is fast but vulnerable to noise, and if the alpha value is large, the background removal rate is slow but robust against noise. We have chosen the alpha value which is optimal for the corresponding slow but robust against noise. We have chosen the alpha value which is optimal for the corresponding application through several experiments. The symbol rk (t) show the received signal and symbol ck (t) application through several experiments. The symbol ( ) show the received signal and symbol shows the clutter signal, which is made until the k-th received sample. The background-subtracted c ( ) shows the clutter signal, which is made until the k-th received sample. The backgroundsignal is represented by the symbol yk (t). We need to store each filtered signal waveform and combine subtracted signal is represented by the symbol ( ) . We need to store each filtered signal waveform them into matrix Wmn of size “m × n”. The “m” represents the slow time length whereas the “n” and combine them into matrix Wmn of size “ × ”. The ’m’ represents the slow time length whereas represents the fast time length of the matrix. The value of “n” depends on the observation distance the “n” represents the fast time length of the matrix. The value of “n” depends on the observation and has a value of a256 for of our experiments (which means meter 1observation distance) while while “m” is a distance and has value 256 for our experiments (which1means meter observation distance) user choice andchoice it depends how long is the observation time. The concept of the slow “m” is a user and iton depends on how long is the observation time. The concept of thetime slowand timefast time is shown in Figure 2. and fast time is shown in Figure 2.

Figure 2. “Vital signal” vs slow time and fast time. Figure 2. “Vital signal” vs slow time and fast time.

Moreover, we only use specified areas for the detection of vital signs and phone detection by extracting only the fast time data of a specific area. The sensing area for phone detection has 25 cm range whereas sensing area for vital signs monitoring has a range of around 50 cm. This helps ignore environmental noise that comes from outside the sensing area. In other words, all the human activities that are out of this range are not taken into account during application of the vital sign measurement and the phone detection algorithm.

Sensors 2017, 17, 1240

6 of 25

Moreover, we only use specified areas for the detection of vital signs and phone detection by extracting only the fast time data of a specific area. The sensing area for phone detection has 25 cm range whereas sensing area for vital signs monitoring has a range of around 50 cm. This helps ignore environmental noise that comes from outside the sensing area. In other words, all the human activities that are out of this range are not taken into account during application of the vital sign measurement Sensors 2017, 17, 1240 6 of 25 and the phone detection algorithm.

3. Proposed Algorithm 3. Proposed Algorithm 3.1. Vital 3.1. Vital Sign Sign Measurement Measurement The process process block block diagram diagramfor forextracting extractingthe thevital vitalsigns signsfrom fromthe theradar radarscans scansisisshown shown Figure The inin Figure 3. 3. During driving, the hands continuously move because the driver has to control the steering, brakes During driving, the hands continuously move because the driver has to control the steering, brakes and other otherdevices devicesinside inside The conventional algorithm vital sign measurement of a and thethe car. car. The conventional algorithm for vitalfor sign measurement of a stationary stationary human supposes that the biggest movement caused by our body is due to the vital signals human supposes that the biggest movement caused by our body is due to the vital signals of breathing of breathing and human heart, true the casehumans, of stationary but in case ofthis body and human heart, which is true which for the is case offor stationary but inhumans, case of body motions is motions this is not always correct, so conventional algorithms for finding the breathing and heart rate not always correct, so conventional algorithms for finding the breathing and heart rate will not provide will not provide accurate in such conditions. accurate measurements inmeasurements such conditions. Process Block Diagram

Raw Signal

Find the fast-time instants proper for vital signs estimation using algorithm 2

Signal preprocessing

Extract vital signal applying the algorithm 3

Analyze the results using FFT

Figure 3. Process block diagram for extraction of driver vital signs.

We observed that during driving, the major part of the motion exists in the upper part of the We observed that during driving, the major part of the motion exists in the upper part of the driver body and the lower part (abdominal area) is relatively stationary. Therefore, we first find driver body and the lower part (abdominal area) is relatively stationary. Therefore, we first find which which part of the body has motion due to the vital signals and which part of the body has motion part of the body has motion due to the vital signals and which part of the body has motion due to due to random hand gestures, steering control or brake application. To this end, we use the sinusoidal random hand gestures, steering control or brake application. To this end, we use the sinusoidal data data fitting algorithm, explained below. fitting algorithm, explained below. 3.1.1. Extracting Receive Signals Signals Which Which Have Have Useful Useful Vital Vital Sign Sign Information Information 3.1.1. Extracting Receive After we waveforms intointo a matrix, we need findtowhich domain samples After wecombine combinethe the waveforms a matrix, we to need find fast-time which fast-time domain represent the sinusoidal motion, which is caused by the breathing and human heart motion. We use samples represent the sinusoidal motion, which is caused by the breathing and human heart motion. sinusoidal fitting algorithm ( Algorithm 1 ).to show how much the received data fit into the sinusoid We use sinusoidal fitting algorithm (Algorithm 1).to show how much the received data fit into the [ ] and [40,41]. The inputThe signal is signal to want estimate the frequency, amplitude and phase of sinusoid [40,41]. input is swe and we to estimate the frequency, amplitude andshift phase [n] want this signal. shift of this signal. Algorithm 1: Sinusoidal data fitting algorithm for parameters estimation 1. The signal [ ] in its general form can be represented as sum of sinusoids as follows: [ ]=∑

(

sin(

+

)+

)

(3)

In Equation (3), is real valued constant describing the amplitude and represent angular frequency, is the initial phase. The constant shows the mean value other than zero. It is convenient to write the Equation (4) as follows. [ ]=∑

(

cos(

)+

sin(

)+

)

(4)

It is convenient to use a vector representation of Equation (3). Let’s stack the samples in column vector as follows: = In the above Equation (5);

(5)

Sensors 2017, 17, 1240

7 of 25

Algorithm 1: Sinusoidal data fitting algorithm for parameters estimation

1.

The signal s[n] in its general form can be represented as sum of sinusoids as follows: p

s[n] = ∑l =1 (αl sin(ωl n + ϕl ) + Cl )

(3)

In Equation (3), αl is real valued constant describing the amplitude and ωl represent angular frequency, ϕl is the initial phase. The constant Cl shows the mean value other than zero. It is convenient to write the Equation (3) as follows. p s[n] = ∑l =1 ( Al cos(ωl n) + Bl sin(ωl n) + Cl ) (4) It is convenient to use a vector representation of Equation (4). Let’s stack the samples in column vector as follows: p s = ∑ Hl θl (5) l =1

In the above Equation (5); 

cos(ωl )  cos(2ωl )    Hl =    

2.

sin(ωl ) 1 sin(2ωl ) 1 . . . cos( Nωl ) sin( Nωl ) 1

     Al      θl =  Bl    Cl 

In our case, the model given by Equation (4) is simplified as; p = 1 and C = 0. Consider a signal s[n] with unknown parameters A, B, C and ω i.e., Equation (4) with p = 1. The measured signal is then given by s[n] deteriorated by additive noise w[n] as shown in Equation (7): x [n] = s[n] + w[n] n = 1, 2, 3, . . . .. , N

3.

(8)

The above equation describes the probability per infinitesimal volume of receiving the data samples x given a set of parameters {θ, ω }. The maximum likelihood estimator (MLE) tries to maximize the pdf with respect to unknown parameters for given values of x and use those parameters as estimates i.e., 

5.

(7)

The noise is assumed zero mean white Gaussian noise. The estimation problem is to estimate the signal parameters using the measured samples of the input data x = [ x (1), x (2), . . . . x ( N )]. The probability density function (pdf) h i T 1 −1 p( x; θ, ω ) = (2πσ 2 ) exp 2σ2 ( x − Hθ ) ( x − Hθ )

4.

(6)

 ˆ ωˆ = argmax p( x; θ, ω ) θ, θ,ω

(9)

Finally, the estimate of θ is given by least-squares solution as follows
−1 T θˆ = H T H H x

(10)

6.

By using the least-squares solution, the MLE criterion function can be concentrated to one parameter as follows
−1 T (11) g(w) = x T H H T H H x

7.

The frequency estimate is then obtained from maximizing the function g(w), that is ωˆ = argmaxω g(w)

(12)

The Equation (12) can be solved either by using iterative step method i.e., Gauss-Newton iteration [42] or by a non-linear search.

Sensors 2017, 17, 1240

8 of 25

The input data used for sinusoidal fitting is the slow time domain data at each fast time index. The R-square value is used for finding the fit of the signal, which is defined as follows: R2 = 1 −

2

∑in=1 (yi −yˆi ) 2 ∑in=1 (yi −y)

(13)

ˆ represent the estimated values of “y” by the fitting algorithm whereas “y” In Equation (13), “y” shows the mean of “y” [43]. R square is a statistic that give information about the goodness of fit of a model. In general, a model fits the data well if the differences between the observed values and the model’s predicted values are small and unbiased. It determines how well the regression line or curve approximates the real data points. An R square value of 1 indicates that the regression line or curve perfectly fits the real data points. R-square can take on any value between 0 and 1, with a value closer to 1 indicating that a greater proportion of variance is accounted for by the model. For example, an R-square value of 0.73 means that the fit explains 73% of the total variation in the data about the average. The higher value of R square means that the sinusoidal regression model is more accurate and hence the motion is more sinusoidal and hence caused by the respiration and heart rate signals whereas the lower value of R square means non-sinusoidal motion caused by any random body/hand motion. We observed the R square values for all the fast time indexes. We have set the threshold value of 0.3 for finding the best fit sinusoids that contain useful information about the respiration and heart rate. If the value is kept too small then it may also select the signals that doesn’t have good vital signal information because they would not match sinusoidal data and if it is kept too high then we may not get any signal above the threshold because the vital signal is not perfectly sinusoidal and moreover, the movement of the body causes the signal to noise ratio of the vital signal to be very low. Therefore, we chose the optimal threshold value of 0.3 for finding the best fit signals. The detailed algorithm for finding the best sinusoid fits is given in Algorithm 2: Algorithm 2: Selection of the fast time indexes that gives the best fit signals 1. 2. 3. 4.

5. 6.

Find the sinusoid fit as shown in Algorithm 1, which gives three parameters i.e., magnitude, frequency and phase shift of the sinusoid that best fits the signal. Find the R square values for the signals at each fast time index. If the fitting frequency of a signal is not in the range of the respiration frequencies i.e., 0.15 Hz to 0.5 Hz, as shown in Figure 4a,b then replace its R square value by zero. If the fitting sinusoid magnitude is less than a certain threshold value as shown in Figure 4c then replace its R square value by zero. Figure 4d shows the fitting for the signal that have the vital signs information as it has relatively higher magnitude and its frequency lies in the breathing frequencies range i.e., 0.15 Hz to 0.5 Hz. Apply the moving averaging filter to the resulting R square values as shown in Figure 5. Extract received signals with R values above threshold value.

Figure 5a shows the result of the fitting algorithm for the case when the person sitting in front of the radar is stationary. Figure 5b shows the R square values for the case when we move hand in front of the abdominal region, which results in lower R square values at the location of the abdominal region while the chest region signal gives better R square values. In experimental scenario for the Figure 5c, the upper parts of the body are moving due to hand gesture and/or steering motion exerted by the hands. From the comparison of these three figures, we know that the R square values decrease at the points (abdominal area in Figure 5b and chest area in Figure 5c due to the motion of the body parts, even though the variance of the signal increases. If we use the conventional algorithm in which the signal at the highest variance point is considered as vital signal, then it will result in wrong measurements when applied in a condition when the maximum variance is caused by the random body motion instead of the vital signs.

5. 6.

its R square value by zero. Figure 4d shows the fitting for the signal that have the vital signs information as it has relatively higher magnitude and its frequency lies in the breathing frequencies range i.e., 0.15 Hz to 0.5 Hz. Apply the moving averaging filter to the resulting R square values as shown in Figure 5. Extract received signals with R values above threshold value.

Sensors 2017, 17, 1240

9 of 25

(a)

Sensors 2017, 17, 1240

9 of 25

(b)

(c)

(d) Figure 4. Data fitting algorithm results (a) fitting frequency higher than the respiration frequencies Figure 4. Data fitting algorithm results (a) fitting frequency higher than the respiration frequencies range (b) fitting frequency lower than the respiration frequencies range (c) fitting magnitude less than range (b) fitting frequency lower than the respiration frequencies range (c) fitting magnitude less than vital signal threshold (d) fitting signal for signal having information of vital signs. vital signal threshold (d) fitting signal for signal having information of vital signs.

Figure 5a shows the result of the fitting algorithm for the case when the person sitting in front of the radar is stationary. Figure 5b shows the R square values for the case when we move hand in front of the abdominal region, which results in lower R square values at the location of the abdominal region while the chest region signal gives better R square values. In experimental scenario for the Figure 5c, the upper parts of the body are moving due to hand gesture and/or steering motion exerted by the hands. From the comparison of these three figures, we know that the R square values decrease at the points (abdominal area in Figure 5b and chest area in Figure 5c due to the motion of the body

Figure 5c, the upper parts of the body are moving due to hand gesture and/or steering motion exerted by the hands. From the comparison of these three figures, we know that the R square values decrease at the points (abdominal area in Figure 5b and chest area in Figure 5c due to the motion of the body parts, even though the variance of the signal increases. If we use the conventional algorithm in which the signal at the highest variance point is considered as vital signal, then it will result in wrong Sensors 2017, 17, 1240 measurements when applied in a condition when the maximum variance is caused by the random 10 of 25 body motion instead of the vital signs.

R-square values (after averaging filter)

0.7 chest region

0.6 abdominal region

0.5 0.4 0.3 0.2 0.1 0 0

1

2

3

Fast time (ns)

Sensors 2017, 17, 1240

4

5

6

10 of 25

(a) Sensors 2017, 17, 1240

10 of 25

(b) (b)

(c) Figure 5. R square values while (a) stationary (b) moving hand near abdominal region (c) moving

Figure 5.upper R square while (a) stationary (b) (c)moving hand near abdominal region (c) moving parts ofvalues body slightly. upper parts of body slightly. Figure 5. R square values while (a) stationary (b) moving hand near abdominal region (c) moving After finding the R square values of the signals in fast time index, we chose signals which have upper parts of body slightly.

the good R-square values and ignore signals which have small R-square values. Figure 6 shows an

After finding the R square values of the signals in fast time index, we chose signals which have the example of the selected best fit signals which contain useful information about the vital sign signal. After findingand the ignore R square values which of the signals in fastR-square time index,values. we chose signals which have good R-square values signals have small Figure 6 shows an example The reason that we expressed the slow time in term of index values rather than seconds is that the good R-square values and ignore signals which have small R-square values. Figure 6 shows an of the selected best fit signals which contain useful information about the vital sign signal. The reason Algorithm 3 uses the slow time index information for the construction of vital signal. example of the selected best fit signals which contain useful information about the vital sign signal. that weThe expressed the slow time in term of index values rather than seconds is that Algorithm 3 uses reason that we3 expressed the slow time in term of index values rather than seconds is that Original signal the slow time index information for the information construction signal. of vital Algorithm 3 uses the slow time index for of thevital construction signal. Fitted signal

2 second zerocrossing

3 Magnitude

1

Original signal Fitted signal

Magnitude

2 0

1 -1 0 -2

second zerofirst crossing zerocrossing

sub-signal component

first zero-

-1 crossing -3

sub-signal component

300

600

-2

900 Slow time index

1200

1500

Figure 6. A best fit signal which has the corresponding r-square value of 0.46. -3 300

600

900 Slow time index

1200

1500

3.1.2. Estimate of Breathing and Heart Rate through the Vital Signal Reconstruction

Figure 6. A best fit signal which has the corresponding r-square value of 0.46.

Figure 6. A best fit signal which has the corresponding r-square value of 0.46.

Now we have to construct the vital signal from the best-fit signals. First, we divide each signal 3.1.2. Estimate of based Breathing and Heart Rate through theinVital Signal into sub-signals on the zero crossings as shown Figure 6. AReconstruction sub-signal component of a signal is defined as the signal samples between two consecutive zero crossing values. After dividing the Now we have to construct the vital signal from the best-fit signals. First, we divide each signal best-fit signals into segments (sub-signal components), we constructed whole vital signal recursively into sub-signals based on the zero crossings as shown in Figure 6. A sub-signal component of a signal is defined as the signal samples between two consecutive zero crossing values. After dividing the best-fit signals into segments (sub-signal components), we constructed whole vital signal recursively

Sensors 2017, 17, 1240

11 of 25

3.1.2. Estimate of Breathing and Heart Rate through the Vital Signal Reconstruction Now we have to construct the vital signal from the best-fit signals. First, we divide each signal intoSensors sub-signals signal 2017, 17,based 1240 on the zero crossings as shown in Figure 6. A sub-signal component of a 11 of 25 is defined as the signal samples between two consecutive zero crossing values. After dividing the from those segments using correlation as explained in Algorithm 3. vital For the sakerecursively of clarity, best-fit signals into segments (sub-signalconcept components), we constructed whole signal we have inserted some examples enclosed in theasrectangular inside the 3. of clarity, from those segments using correlation concept explained boxes in Algorithm 3. Algorithm For the sake We inserted constructed vital signal frominthe signals combining we have somethe examples enclosed thesegmented rectangulargood-fit boxes inside thebyAlgorithm 3. each subsignal to vital the signal previously constructed vital signal and its We component constructed the from the segmented good-fit signalsrecursively by combining eachfinding sub-signal autocorrelation. chose theconstructed optimum sub-signal component that shows the highest autocorrelation. component to theWe previously vital signal recursively and finding its autocorrelation. We We have the autocorrelation theshows signalthe after combining the new sub-signal-component chose the determined optimum sub-signal componentof that highest autocorrelation. We have determined withautocorrelation the previouslyof constructed vital combining signal. If these signals have good correlation then width of the the signal after the new sub-signal-component with thethe previously the autocorrelation ( ℎ_ ) will be wider, otherwise the width of the autocorrelation constructed vital signal. If these signals have good correlation then the width of autocorrelation will be narrower [28]. (width_correlation ) will be wider, otherwise the width of the autocorrelation will be narrower [28]. Figure 7, the big picture picture of the Algorithm Algorithm 3 is explained explained by by a diagram. diagram. The algorithm explains In Figure how to construct the vital signal from multiple best fit signals reflected from the body during motion driver. of the driver.

Choose the best fit signals which has vital signs information

Iteration = 1

Construct the first sub-signal component of the vital signal from all the candidate signals as in step 2 of algorithm 3

Repeat until the Last zerocrossing

Find the next candidate signal sub-components Append previously constructed signal to each candidate signal subcomponent Find the correlation of all the signals obtained in the above step Find the maximum correlation signal Append the previously constructed signal with subsignal component which has maximum correlation

Figure 7. The vital signal construction from the candidate sub-signal components of the best fit signals. Figure 7. The vital signal construction from the candidate sub-signal components of the best fit signals.

The step by step explanation of the vital signal construction is explained by the following The step by step 3). explanation of the vital signal construction is explained by the following algorithm (Algorithm algorithm (Algorithm 3). Algorithm 3: Step by step construction of the vital signal from the best fit signals (1) Initialization Find “k” best-fit signals whose R-square values are above certain set threshold, say R-square_min = 0.3. ( × 256). In our The size of each best-fit signal is 256 samples. So, we have a matrix of _ example, k = 9. (2) for = 1: _ _ If (iteration == 1) Then Find the first zero-crossing fast time index ( _ ) of all the “k” best-fit signals. And find the sub-signal component, which has the maximum value of _

Sensors 2017, 17, 1240

12 of 25

Algorithm 3: Step by step construction of the vital signal from the best fit signals (1)

Initialization Find “k” best-fit signals whose R-square values are above certain set threshold, say R-square_min = 0.3. The size of each best-fit signal is 256 samples. So, we have a matrix of best_ f itmatrix (k × 256). In our example, k = 9.

(2)

for iterations = 1 : last_zero_crossing If (iteration == 1) Then Find the first zero-crossing fast time index (zero_crossings) of all the “k” best-fit signals. And find the sub-signal component, which has the maximum value of zero_crossings For our example, f irst_zero_crossings are found to be as follows: first zero_crossings = [234 231 223 231 230 235 224 220]; max_zero_index = max (zero_crossings) = 235; k_max_crossing = argmaxk (zero_crossings) = 6 Construct the vital signal according to the following expression, the result is shown in Figure 8. constructed_vital_signal (1 : max_zero_index ) = best_ f it_matrix (k_max_crossing, 1 : max_zero_index ) For our example; Figure 8 show the construction of first sub-signal component of the vital signal: constructed_vital_signal (1 : 235) = best_ f it_matrix (6, 1 : 235); Else if (iteration > 1) Find the next zero-crossing fast time index (zero_crossings) of all the “k” best-fit signals for i = 1 : k Append the ith sub-signal component to the previously constructed vital signal. Find the correlation of the resulting signal correlation(i ) = autocorr ([constructed_vital end

max(width_correlation(i )); i = 1 : k

Append that sub-signal component to the previously constructed vital signal, which has maximum correlation as found by the above expression. For our example, when iteration = 2: second zero_crossings = [354 359 351 344 347 360 354 351]; After we appended each sub-component to the previously constructed vital signal and found the correlation of each signal, it was found to be: correlation = [17.1 13.3 18.3 9.3 16.2 23.8 18.2 22.8]; k maxcorr = argmaxk (correlation) = 6 max_corr_index = argmaxzero_crossings (correlation) = 360 Here the 6th signal sub-component shown the maximum correlation with the previously constructed signal. Now we append the next sub-signal component of the 6th signal to the previously constructed vital signal to find the constructed_vital_signal as follows in Figure 9. constructed_vital_signal (end : max_corr_index ) = best_ f it(k_max_corr, end : max_corr_index ) where end = length( previous constructed vital signal ) + 1 For our example: constructed_vital_signal (236 : 360) = best_ f it_matrix (6, 236 : 360); (3)

Check, if: there are further zero crossings, then go to step 2 Else; stop the loop; Save; the constructed_vital_signalThe final vital signal obtained is given in the Figure 10. end

Sensors 2017, 17, 1240

13 of 25 13 of 25 13 of 25 13 of 25

Magnitude Magnitude Magnitude

Sensors 2017, 17, 1240 Sensors 2017, 17, 1240 Sensors 2017, 17, 1240 4 4 3 4 3 2 3 2 1 2 1 0 1 0 -1 0 -1 -2 -1 -2 -3 -2 -3 -4 -3 -4 -5 -40 -5 0 -5 0

Partially constructed vital signal Partially constructed vital signal Partially constructed vital signal

300 300 300

600 900 600 Slow time index 900 Slow time index 600 900 Slow time index

1200 1200

1500 1500

1200

1500

Figure 8. The first sub-signal component of vital_constructed_signal.

Figure 8. 8. The componentofofvital_constructed_signal. vital_constructed_signal. Figure Thefirst firstsub-signal sub-signal component

Magnitude Magnitude Magnitude

Figure 8. The first sub-signal component of vital_constructed_signal. 4 4 3 4 3 2 3 2 1 2 1 0 1 0 -1 0 -1 -2 -1 -2 -3 -2 -3 -4 -3 -4 -5 -40 -5 0 -5 0

Partially constructed vital signal Partially constructed vital signal Partially constructed vital signal

300 300 300

600 900 600 Slow time index 900 Slow time index 600 900 Slow time index

1200 1200

1500 1500

1200

1500

Figure 9. The constructed vital signal after second iteration. Figure 9. The constructed vital signal after second iteration. Figure 9. The constructed vital signal after second iteration. Figure 9. The constructed vital signal after second iteration. 5 constructed vital signal constructed vital signal

5

Magnitude Magnitude Magnitude

5

constructed vital signal

0 0 0

-5 -50 0 -5 0

300 300 300

600 900 600 Slow time index 900 Slow time index 600 900 Slow time index

1200 1200

1500 1500

1200

1500

Figure 10. The final constructed vital signal. Figure 10. The final constructed vital signal. Figure 10. The final constructed vital signal.

10.constructed The final constructed vital signal. In Algorithm 3, the vitalFigure signal is from the sub-signal components of best-fit signals. In Algorithm 3, the vital signal is constructed from the sub-signal components of best-fit signals. NowInwe have to extract the breathing and heart values from the constructed vital signal using the Algorithm 3, the vital signal is constructed from the sub-signal components best-fit signals. Now we have to extract the breathing and heart values from the constructed vitalofsignal using the Fast Fourier transform (FFT). The size of FFT used in the evaluation of the frequency domain signal In Algorithm 3, extract the vital is constructed from sub-signal components of best-fit signals. Now we have to thesignal breathing and heart from the constructed vital signal using the Fast Fourier transform (FFT). The size of FFT usedvalues in thethe evaluation of the frequency domain signal 15 and the slow time sampling frequency was around 110 samples/s. The frequency resolution was Fast transform (FFT). The size of FFT used in the evaluation of the frequency domain signal 15 and Nowwas we 22Fourier have tothe extract the breathing and heart values from the constructed vital signal using the slow time sampling frequency was around 110 samples/s. The frequency resolution Fast thus obtained 0.2014/min. The respiration rate is extracted from the FFT signal by finding the 15 and thewas was 2 slow time frequency was around 110 samples/s. The signal frequency resolution Fourier (FFT). Thesampling sizeThe of FFT used in the of the domain signal thustransform obtained was 0.2014/min. respiration rate isevaluation extracted from thefrequency FFT by finding the was location of the was maximum peak of therespiration spectrum as discussed in detail inthe theFFT reference [28]. 15 thus obtained 0.2014/min. The rate is extracted from signal by finding the location of thetime maximum peakfrequency of the spectrum as discussed in detail in the reference [28]. resolution thus 2 and the slow sampling was around 110 samples/s. The frequency location of0.2014/min. the maximumThe peak of the spectrum asextracted discussedfrom in detail in thesignal reference [28]. obtained was respiration rate is the FFT by finding the location 3.2. Dual-Mode Background Subtraction Algorithm for Phone Detection 3.2. Dual-Mode Background Subtraction Algorithm for Phone Detection of the maximum peak of the spectrum as discussed in detail in the reference [28]. 3.2. Dual-Mode Background Subtraction Algorithm for Phone Detection We proposed a dual-mode background subtraction algorithm to detect the driver’s use of the We proposed a dual-mode background subtraction algorithm to detect the driver’s use of the mobile phone accurately and robustly. Figure 11 shows a diagram for the proposed algorithm. State We proposed a dual-mode background subtraction algorithm to the detect the driver’s use ofState the 3.2. Dual-Mode Background Algorithm Phone Detection mobile phone accuratelySubtraction and robustly. Figure 11for shows a diagram for proposed algorithm. 1 indicates thataccurately the magnitude of the reflected signal exceeding thefor threshold is notalgorithm. detected inState the mobile phone and robustly. Figure 11 shows a diagram the proposed 1 indicates that the magnitude of the reflected signal exceeding the threshold is not detected in the We proposed athedual-mode background subtraction algorithm to detect the detected driver’s use radar detection area. The received signal is obtained byexceeding using thethe background subtraction method inof the 1 indicates that magnitude of the reflected signal threshold is not in the radar detection area. The received signal is obtained by using the background subtraction method in mobile phone and robustly. Figure 11 shows a diagram for proposed algorithm. Section 2.2.accurately If the peak value of the received signal exceeds threshold 1, the a transition from state 1 in to State radar detection area. The received is obtained by using the background subtraction Section 2.2. If the peak value of thesignal received signal exceeds threshold 1, a transition frommethod state 1 to ( ) state 2 occurs. In Equation (14), c is the clutter signal immediately before state 1 → state 2 1 indicates that the magnitude of the reflected signal exceeding the threshold is not detected _ Section 2.2. If the peak value of cthe received exceeds threshold 1, a transition from1 → state 1 to ( ) issignal state 2 occurs. In Equation (14), the clutter signal immediately before state state 2in the _ transition. And is the slow time index at transition from state 1 to state 2. Parameter ( ) radartransition. detection area. The received is obtained using background subtraction method in state 2 occurs. In Equation (14), is the clutter signalthe immediately state → state 2 _ And is thec signal slow time index at by transition from state 1before to state 2. 1Parameter is the slow time margin for transition: transition. And isofthe slow time index transition from state to state 2. Parameter Section 2.2. is If the the peaktime value the received signalatexceeds threshold 1, 1a transition from state 1 to slow margin for transition: is the for ( ) =∝ ( ) + (1−∝) ( ) c time margin c transition: state 2 occurs. In slow Equation (14), ck_stopped signal immediately before state 1 (14) → state 2 (t) is the clutter ( ) =∝ ( ) + (1−∝) ( ) c __ c (14) transition. And k transient from state 1 to state( 2. k margin is ( )slow ( ) + (1−∝) ) Parameter(14) c _ is the =∝ ctime index at transition

the slow time margin for transition: ck_stopped (t) =∝ cktransient −kmargin −1 (t) + (1− ∝)rktransient −kmargin (t)

(14)

Sensors 2017, 17, 1240 Sensors 2017, 17, 1240

14 of 25 14 of 25

State threshold value State 2 indicates indicates a state in which a signal exceeding a threshold value 1 is detected by an instantly high noise or aa moving movingobject objectsuch suchas asaahand handgesture. gesture.InIn state two received signals obtained state 2, 2, two received signals areare obtained by by a dual mode background subtraction method using differentclutters. clutters.The Thefirst firstisisthe thesignal signal yyk((t)) a dual mode background subtraction method using different obtained and the the second second is isthe thesignal signal yy˘ k((t)) obtained by using the background substation method in Section 2.2 and ( ) by using using cck_stopped clutter as shown in the following Equation (15): obtained by t clutter as shown in the following Equation (15): ( ) _

( ) y ( )=r ( )−c _ (15) y˘ k (t) = rk (t) − ck_stopped (15) (t) If it is an instantaneous moving object, the maximum value of y ( ) will be smaller than If it is1,an instantaneous moving object, the maximum value of y˘ k (t) will beappears smaller or than threshold which means state 2 → state 1 transition. If a new background object an threshold 1, which means state 2 → state 1 transition. If a new background object appears or an existing background object disappears, the maximum value of y ( ) is still larger than the threshold existing object disappears, the maximum value of y˘ k (the t) ismaximum still largervalue than the 1, ( ) in the 1, but thebackground state 2 → state 4→ state 1 transition will occur because of ythreshold but the state 2 → state 4 → state 1 transition will occur because the maximum value of y t in the ( ) T slow time interval becomes smaller than the threshold 2. If the driver's mobile phone isk detected, T time interval becomes smaller thanbecause the threshold 2. If thevalue driver’s detected, y slow ( ) inphone the transition from state 2 to state 3 occurs the maximum of ymobile the T isslow time the transition from state 2 to state 3 occurs because the maximum value of y t in the T slow time ( ) y k than threshold 1 for a interval is greater than threshold 2 and the maximum value of y ( ) is greater interval is greater than(Tthreshold the maximum y˘ kuse thanphone threshold 1 for a (t) is certain period of time ). State23and indicates a state invalue whichofthe ofgreater the mobile is detected. certain period of time (T ) . State 3 indicates a state in which the use of the mobile phone is detected. holdy ( ) becomes smaller than threshold 1, a transition occurs to state 3 → If the maximum value of If the maximum value of becomes smaller thanisthreshold a transition to state 3 →ofstate (t) use state 1, which means thaty˘ kthe of mobile phone stopped.1,Or when the occurs maximum value y ( 1,) which means that the use of mobile phone is stopped. Or when the maximum value of y t in the Ty ( ) k in the T slow time interval becomes smaller than threshold 2, a transition occurs to state 3 → state slow time interval becomes than threshold 2, aappears transition state 3 → state 4→ state1, 4→ state1, which means thatsmaller a new background object or occurs that thetoexisting background object which means that a new background object appears or that the existing background object disappears. disappears.

Figure 11. State diagram of dual-mode background subtraction algorithm for phone usage detection. Figure 11. State diagram of dual-mode background subtraction algorithm for phone usage detection.

Figure 12a below shows the magnitude of the received signal over fast time when the driver Figure 12a below shows the magnitude the received over fast time is when thelarger driverwhen uses uses the mobile phone. It is shown that theofmagnitude of signal the reflected signal much the mobile phone. It is shown that the magnitude of the reflected signal is much larger when using the using the proposed background removal method ( ) than the conventional background removal proposed removal methodasyk the ( t ). k ( t ) than the conventional ( ). This isremoval method background because,method in the y˘conventional method, the background mobile phone is regarded This is because, in the conventional method, the mobile phone is regarded as the background, and the background, and the signal reflected from the mobile phone is attenuated. signalFigure reflected thethe mobile phone is attenuated. 12bfrom shows channel impulse response (CIR). The CIR was estimated using the clean Figure 12b shows the channel impulse response The larger CIR was estimated using( the clean ) is much ) for algorithm in [44]. This also shows that the CIR of ( (CIR). than the CIR of the ˘ algorithm in [44]. This also shows that the CIR of y t is much larger than the CIR of y t for ( ) ( ) k above reasons. Figure 12c shows the maximum value of the received signal over slow ktime in the the above reasons. Figure 12c 0.7 shows themobile maximum value of thethe received signal slow time in the same ( ) same situation. At about s, the phone entered sensing area,over which caused both situation. At about 0.7 s, the mobile phone entered the sensing area, which caused both y t and ( ) k and ( ) to increase in magnitude. At this point, the algorithm transitions from state1 to state2. At ˘ k (t) to1 increase yabout in phone magnitude. Atbecomes this point, the algorithm transitions frommotions state1 todue state2. At about s, mobile almost stationary except some minor to texting or 1touching s, mobilethe phone almost some of minor due to atexting or touching ( motions ( ) ) still remains screen, and becomes after thatstationary point, theexcept magnitude large value. the screen, aand after that point, the because magnitude y˘ k (t) still remains a large value. a k ( t ) represents represents relatively small value the of mobile phone is recognized as the ybackground. The relatively small value because the mobile phone is recognized as the background. The mobile phone is mobile phone is continuously used in the detection area, and therefore, the maximum value of ( ) continuously used invalue the detection and therefore, theinterval maximum y˘ k (t)than and the the maximum and the maximum of ( )area, in the slow time are value both of larger respective threshold values, and thus the state transitions from state 2 to state 3 (mobile phone detection) occurs at about 1.9 s. In addition to the signal characteristics for cell phone use described here, the signal

Sensors 17, 1240 Sensors 2017, 2017, 17, 1240

1515ofof2525

characteristics for various types of driver behavior are specified in the results section. Figure 13a,b value of shows yk (t) inthe the T interval are bothsignal larger than and thus below magnitude the reflected over the fastrespective time andthreshold its CIR values, when 15 there is a y slow timeof Sensors 2017, 17, 1240 of 25 the state transitions from state 2 to state 3 (mobile phone detection) occurs at about 1.9 s. In addition ( ( ) ) background change. represents the magnitude of a noise level while the magnitude of to the signal characteristics for phone use described theused signal characteristics forFigure various types ( ) here, characteristics for various types of driver behavior are in results section. 13a,b represent a large value. Thiscell characteristics of canspecified be tothe distinguish background changes below showsare the magnitude of the reflected signal over 13a,b fast time andshows its CIR there isofa the offrom driver behavior specified the results section. Figure below thewhen magnitude using mobile phones withinminor motion. ( ) and background change. represents magnitude noise level while the magnitude of ( ) the reflected signal over fast time its CIRthe when there isofaabackground change. yk (t) represents ( ) represent a large value. This characteristics of can be used to distinguish background changes of magnitude of a noise level while the magnitude of y˘ k (t) represent a large value. This characteristics from using mobile phones with minor motion. yk (t) can be used to distinguish background changes from using mobile phones with minor motion.

(a) (a) status 1 status 1

(b) (b)

status 2 status 2

status 3 status 3

(c)(c)

Figure 12. Signal characteristics whenusing usingaa mobile mobile phone signal over fast fast timetime (b) (b) Figure 12. Signal characteristics when phone(a) (a)received received signal over Figurechannel 12. Signal characteristics when usingvalue a mobile phone (a) received signal over fast time impulse response (c) the maximum of the received signal over slow time. channel impulse response (c) the maximum value of the received signal over slow time. (b) channel impulse response (c) the maximum value of the received signal over slow time.

(a)

(b)

Figure 13. Signal characteristics when there is a background change (a) received signal over fast time (a) (b) (b) channel impulse response.

Figure 13. Signal characteristics when there is a background change (a) received signal over fast time Figure 13. Signal characteristics when there is a background change (a) received signal over fast time (b) channel impulse response. (b) channel impulse response.

Sensors 2017, 17, 1240

16 of 25

4. Results and Discussion 4.1. Experimental Setup and Reference Data Measurements Sensors 2017, 17, 1240

16 of 25

In our experiments, we used the commercially available single-chip IR-UWB radar transceiver 4. Results and Discussion NVA6201 made by NOVELDA (Novelda AS, Kviteseid, Norway). The radar has a center frequency of 6.8 GHz, a bandwidth 2.3Reference GHz, and transmission output power of −53 dBm/Hz. The pulse 4.1. Experimental Setupofand DataaMeasurements repetition frequency is 100 we MHz the slow time sampling frequency (measurement In our experiments, usedand the commercially available single-chip IR-UWB radar transceiver rate) is 110 samples/s. The radar transceiver supports staggered pulse repetition frequency NVA6201 made by NOVELDA (Novelda AS, Kviteseid, Norway). The radar has a center frequency (PRF), of 6.8 GHz, a bandwidth of 2.3the GHz, andbetween a transmission power of −53 dBm/Hz. The is pulse a transmission process in which time each output coherent pulse transmission patterned repetition frequency is 100 MHz and the slow time sampling frequency (measurement rate) is and slightly changed. This function extends the maximum unambiguous range (MUR) by110 making it radar transceiver supports staggered pulse repetition (PRF), pulses a transmission possiblesamples/s. to clearlyThe distinguish the return pulses corresponding to thefrequency transmission among many process in which the time between each coherent pulse transmission is patterned and slightly return pulses. In practice, the radar transceiver has a range of almost 10 m and the maximum range is changed. This function extends the maximum unambiguous range (MUR) by making it possible to determined bydistinguish the SNR. the Thereturn operating transceiver is many −40 to 80 degrees, clearly pulsestemperature correspondingof tothe the radar transmission pulses chip among return which ensures stable operation under any environment. pulses. In practice, the radar transceiver has a range of almost 10 m and the maximum range is by thesetup SNR. The operating theisradar transceiver chip 14a. is −40The to 80sensing degrees, area is Thedetermined experimental of the radartemperature inside theofcar shown in Figure ensures stable operation anyisenvironment. dividedwhich into two parts. The regionunder which near to the radar module is the phone detection region The experimental setup of the radar inside the car is shown in Figure 14a. The sensing area is because in this region a driver usually positions his/her phone while reading messages or other mobile divided into two parts. The region which is near to the radar module is the phone detection region phone related activity which can cause distraction of the driver from his primary task of monitoring because in this region a driver usually positions his/her phone while reading messages or other the road. The region right after thewhich mobile region isofthe sign detection region. It covers mobile phone related activity cansensing cause distraction thevital driver from his primary task of the whole front body areaThe of the driver. Figure shows theregion structure of the IR-UWB monitoring the road. region right after the 14b mobile sensing is the vital sign detectionradar region.module. It covers the whole front area transceiver, of the driver. Figure 14b shows the structure of the IR-UWB radarA patch The radar module consists ofbody a radar a transmit antenna and a receive antenna. radar consists of a radar transceiver, a transmit antenna a receive antenna. antennamodule. is used,The and themodule antenna and transceiver board are connected by and a SubMiniature version A A patch antenna is used, and the antenna and transceiver board are connected by a SubMiniature (SMA) connector. version A (SMA) connector.

(a)

(b)

Figure 14. (a) Experimental setup inside the car (b) IR-UWB radar module.

Figure 14. (a) Experimental setup inside the car (b) IR-UWB radar module. In order to verify the results of our proposed algorithm for the heart rate measurement, we have to compare it some benchmark device such as ECG sensor. We used the ECG sensor module PSLIn order to verify the results of our proposed algorithm for the heart rate measurement, we have iECG2 which require only 5 V input voltage (Vcc) and the current consumption is below 50 mA. The to compare it someis benchmark device such as ECG sensor. thefilter ECGswitch. sensor amplification 750 V/V. It uses a selectable 50 or 60 Hz enabledWe by used the notch Themodule PSL-iECG2 which require only 5 V input voltage (Vcc) and the current consumption is below measurement setup for the connection of the electrodes to the human hands is illustrated in Figure 50 mA. 15a [45]. The amplification is 750 V/V. It uses a selectable 50 or 60 Hz enabled by the notch filter switch. The sample output ECG signal of is shown in Figure to 15b.the Wehuman also used some is reference The measurement setup for of thethe connection the electrodes hands illustrated in measurement for the validity of our respiration rate value by the proposed algorithm. In many Figure 15a [45]. studies, respiration rate is measured by wearing a band on the belly or chest [46], but in case of our Theexperiments, sample output of the signal shown inthe Figure 15b. Wearealso usedbecause some the reference if we cover theECG chest or belly is with a band, measurements affected measurement for the validity of our respiration rate value by the proposed algorithm. In many reflected signal from the belly or chest is distorted by the band. Therefore, we used nasal breath soundstudies, recordings from a smartphone as aareference rate [46], measurements. [47], the respiration rate is measured by wearing band on for therespiration belly or chest but in caseAsofinour experiments, reference respiration ratewith can be measured with theare errors less than 1% forthe all breathing if we cover the chest or belly a band, theaccurately measurements affected because reflected signal from theranges. belly or chest is distorted by the band. Therefore, we used nasal breath sound recordings from

a smartphone as a reference for respiration rate measurements. As in [47], the reference respiration rate can be measured accurately with the errors less than 1% for all breathing ranges.

Sensors 2017, 17, 1240

17 of 25

Sensors 2017, 17, 1240

17 of 25

Sensors 2017, 17, 1240

17 of 25

(a) 30

ECG signal

20

(a)

Magnitude

30 10

ECG signal

Magnitude

20 0 10 -10 -200 -10 -30 0

0.2

0.4

-20

Figure 15.

0.6

0.8 1 1.2 Time (milliseconds)

1.4

1.6

1.8

2

x 10

4

(b)

(a)-30 Measurement setup 0 0.2 0.4

for acquisition; from the 0.6 the ECG 0.8 signal 1 1.2 1.4(b) The 1.6 signal 1.8 obtained 2

4 Time (milliseconds) x 10 Figure 15.sensor (a) Measurement setup for the ECG signal acquisition; from the ECG ECG module PSL-iECG2 (measurement reference device). (b) The signal obtained (b)device). sensor module PSL-iECG2 (measurement reference Figure 15. (a) Measurement setup for the ECG signal acquisition; (b) The signal obtained from the 4.2. Respiration & Heart Rate Results

ECG sensor module PSL-iECG2 (measurement reference device). 4.2. Respiration & Heart Results to First, we give Rate an example show the performance of our algorithm as compared to a

conventional algorithm such as the one described in reference [28] during the motion period. After 4.2. Respiration & Heart Rateto Results First, we give an example show the performance of our algorithm as compared to a conventional the example, we verify the proposed algorithm for different types of motions related to the driving algorithmFirst, suchwe as the one described referenceperformance [28] during theour motion period. compared After the example, give anthe example toinshow algorithm to 1a activities and measure vital signs duringthe the motion periodofand the results areasgiven in Tables we verify the proposed algorithm for different types of motions related to the driving activities and conventional algorithm such as the one described in reference [28] during the motion period. After and 2. In order to show that our algorithm is repeatable, we tested it for five different human subjects measure the vital signs during the motion period and the results are given in Tables 1 and 2. In order the example, we verify the proposed algorithm for different types of motions related to the driving and calculated the difference between the reference results and the results from our proposed to activities vital signswe during theit and human the results are given Tables 1 the showalgorithm that our and algorithm isthe repeatable, for fiveperiod different subjects andincalculated andmeasure summarized the results intested Table 3.motion and 2. In order show that algorithm is repeatable, we it for five different subjects difference between the reference results and the results from our algorithm In order toto show that theour proposed algorithm extract thetested vitalproposed signal even if there human is and somesummarized motion and the difference between the reference results andslight the results proposed of thecalculated body, we 3. made an experiment as follows. In the experiment, motion from of theour upper part of the results in Table algorithm and summarized the results in Table 3. the body is made. The vital signals that are obtained by the conventional algorithm as well as by the In order to show that the proposed algorithm extract the vital signal even if there is some motion In order to showare that the proposed algorithm extract the vital signal even if there is some motion proposed algorithm shown in Figure 16. of the body, we made an experiment as follows. In the experiment, slight motion of the upper part of of the body, we made an experiment as follows. In the experiment, slight motion of the upper part of the body is made. The20vital signals that are obtained by the conventional algorithm as well as by the the body is made. The vital signals that are obtained by the conventional algorithm Vital signal by conventional algorithm as well as by the proposed algorithm are shown in Figure 16. Vital signal by proposed algorithm proposed algorithm 15 are shown in Figure 16. Magnitude

5 15

Magnitude

10 20

-5 5

Vital signal by conventional algorithm Vital signal by proposed algorithm

0 10

-10 0 -15 -50

1

-10 -15 0

2 3 Slow time (seconds)

4

5

4

5

(a) 1

2 3 Slow time (seconds)

(a) Figure 16. Cont.

Sensors 2017, 17, 1240 Sensors 2017, 17, 1240

18 of 25 18 of 25

Sensors 2017, 17, 1240 1

1

R-square values R-square values Normalized variance values Normalized variance values

0.6 0.6 Magnitude

Magnitude

0.8 0.8

18 of 25

0.4 0.4

0.2

0 0

Abdominalarea area Abdominal

0.2

0 0

1

2

1

2

3

Fast 3 time (ns) Fast time (ns)

4

5

4

6

5

6

(b)

(b)

Figure 16. (a) Comparison of the vital signal obtained from the conventional vs proposed algorithm Figure 16. of vital obtained from vs algorithm while the human is slightly moving (b) R-square and normalized variance values at each fast-time Figure 16.(a) (a)Comparison Comparison ofthe the vitalsignal signal obtained fromthe theconventional conventional vsproposed proposed algorithm while the human is slightly moving (b) R-square and normalized variance values at each during motion state.moving (b) R-square and normalized variance values at eachfast-time whileindex the human is slightly fast-time

index during motion state. index during motion state. As is shown in Figure 16, in a state of slight motion, conventional algorithm looks for the signal at fast time index having16, maximum variance as the vital signal, however, algorithm the motion looks not due the As in in of motion, conventional for the signal Asisisshown shown inFigure Figure 16, inaastate state ofslight slight motion, conventional algorithmislooks forto the signal respiration and heartbeat motion but it is due to the random body motion as shown by the values of at indexhaving havingmaximum maximumvariance variance vital signal, however, the motion nottodue at fast fast time index as as thethe vital signal, however, the motion is notisdue the the R-square in blue-color line in Figure 16b. The vital signal obtained by the conventional algorithm to the respiration and heartbeat motion but it to is the duerandom to the random body as motion asby shown by the respiration and heartbeat motion but it is due body motion shown the values of in Figure 16a does not have any trace of periodicity, whereas the respiration signal obtained by the values of the R-square in blue-color line in Figure 16b. The vital signal obtained by the conventional the R-square blue-color Figure has 16b.good The vital signal obtained bycaptures the conventional algorithm proposedinalgorithm in line this in example periodic pattern and it the vital signal algorithm in 16a have trace ofsignal periodicity, whereas the respiration signal obtained in Figure 16aFigure does not does have anyfrequency traceany of domain periodicity, whereas the by the information. In Figure 17, not the is shown. Therespiration highest peaksignal i.e., theobtained peak at 21 by thecycles/minutes proposed algorithm inthe thisbreathing example has good periodic pattern and captures the signal proposed algorithm in this example hasfrequency good periodic pattern and ititcaptures the signal represent whereas the second highest peak in thevital heart frequency range i.e., at 62 cycles per minute represent the heart rate of the human. information. the frequency frequencydomain domainsignal signalisisshown. shown.The Thehighest highestpeak peak i.e., the peak information. In Figure 17, the i.e., the peak at at 21

21 cycles/minutes represent breathing frequencywhereas whereasthe thesecond secondhighest highest peak peak in the cycles/minutes represent thethe breathing frequency the heart heart 700 frequency per minute represent the heart rate of the human. Frequency domain signal frequencyrange rangei.e., i.e.,at at62 62cycles cycles per minute represent the heart rate of the human. Respiration Peak (21) 600 Magnitude of Spectrum

700

500

Frequency domain signal

Respiration Peak (21)

Magnitude of Spectrum

600 400 Heart Rate Peak (62)

500 300 400 200

Heart Rate Peak (62)

300 100 200

0 0

20

40

60

80 100 120 Frequency (per minute)

140

160

180

200

100

Figure 17. Spectrum obtained by applying FFT algorithm to the signal obtained by proposed algorithm in Figure 16(a). 0 0

20

40

60

80 100 120 Frequency (per minute)

140

160

180

200

Driving involve different kinds of motion of hand, head and body such as looking at the side mirror the road behind,by or picking in front of the driver or hand motion Figure to 17.check Spectrum obtained applyinga glass FFT algorithm to the signal obtained byassociated proposed Figure 17. Spectrum obtained by applying FFT algorithm to the signal obtained by proposed algorithm with the steering We also taken into account, the external effects like turning the car, algorithm in Figureoperation. 16(a). in Figure 16a. applying the brakes due to external factor and accelerating the car. The conventional algorithms for vital signsinvolve measurement didn’t specifically deal of with vitalhead signs and measurement during such motions. Driving different kinds of motion hand, body such as looking at the side Driving involve different kinds of motion of hand, head and body such as looking at the side We measured the vital signs during the motion activities and summarized the results in Tables 1 and mirror to check the road behind, or picking a glass in front of the driver or hand motion associated The values of theor measurements arein rounded thedriver nearestorinteger values. associated with mirror2 as to follows. check the road behind, picking a glass front oftothe hand motion

with the steering operation. We also taken into account, the external effects like turning the car, the steering Wetoalso takenfactor into account, the external like turning thealgorithms car, applying applying theoperation. brakes due external and accelerating the effects car. The conventional for the brakes due to external factor and accelerating the car. The conventional algorithms for vital signs vital signs measurement didn’t specifically deal with vital signs measurement during such motions. measurement specifically deal with vital signs measurement during such measured We measureddidn’t the vital signs during the motion activities and summarized themotions. results inWe Tables 1 and the vital signs during the motion activities and summarized the results in Tables 1 and 2 as follows. 2 as follows. The values of the measurements are rounded to the nearest integer values. The values of the measurements are rounded to the nearest integer values.

Sensors 2017, 17, 1240

19 of 25

Table 1. Experimental results for breathing rate for different motion conditions.

Movement Case

No. of Best-Fit Columns above R-Square Threshold (0.3)

stationary

general movements

specific driving movements

Reference Measurement (Breathing Rate)

Estimated Value (Breathing Rate)

19

18

18

hand motion near body

8

18

19

shoulder motion for steering

4

18

16

gesture made by hand away from body

11

18

18

lips motion during speaking

14

18

18

head movement to watch the mirror of the car

7

18

18

turning the car

4

18

20

applying the brakes

7

18

17

accelerating the car

10

18

18

Table 1 show the results for the estimated respiration rate for different body motion states and it also shows the number of best fit signals reflected from body for each motion case. In case of body motion related to driving such as in the Table 1, our proposed algorithm can find some good-fit signals which can be used to extract the vital signal and hence it can give good measurement results. We also measured the heart rate during different body movements which may be made by a driver. The results are summarized in Table 2 and compared with the reference heart rate measurements. The values of the measurements are rounded off to the nearest integer values. Table 2. Experimental results for heart rate measurements for different motion conditions.

Movement Case

No. of Best-Fit Columns above R-Square Threshold (0.3)

stationary

general movements

specific driving movements

Reference Measurement (Heart Rate)

Estimated Value (Heart Rate)

17

72

72

hand motion near body

9

72

71

gesture made by hand away from body

11

73

72

lips motion during speaking

12

72

73

head movement to watch the mirror of the car

8

72

74

turning the car

5

72

75

applying the brakes

7

74

71

accelerating the car

12

72

72

The above Table 2 shows the results of the heart rate measurements during the body movements of a driver. The results are related to the general movements of human as well as specific motions related to driving activities. The results in Table 2 show that the algorithm works well under various motion conditions. During huge body movements, which rarely happen during driving activities such as turning the car speedily and passing at high speed over a speed bump, there is no signal which can fit a sinusoidal signal (the R-square values of the signals obtained during huge motion of the car are shown in Figure 18), which clearly indicates that all the R-square values are below the threshold value for the best fit signals i.e., 0.3. Usually, during driving the most of motions are made by hands/arms, shoulder and chest so vital signs measurements by our proposed algorithm is suitable for the driving conditions.

R-square values (after averaging filter)

Sensors 2017, 17, 1240 Sensors 2017, 17, 1240

20 of 25 20 of 25

0.6 0.5 0.4 0.3 0.2 0.1 0 0

1

2

3 Fast time (ns)

4

5

6

Figure18. 18.The The R-square values in case of body huge movements body movements (all thearevalues the Figure R-square values in case of huge (all the values beloware the below threshold threshold value of 0.3). value of 0.3).

In order to show the validation of the proposed algorithm, we used different human subjects for In order to show the validation of the proposed algorithm, we used different human subjects for our experiments. The average error (difference between the reference value and the estimated value) our experiments. The average error (difference between the reference value and the estimated value) results of the vital signs for five humans are shown below in Table 3. The people involved in these results of the vital signs for five humans are shown below in Table 3. The people involved in these experiments were healthy and aged between 24–38 years old. experiments were healthy and aged between 24–38 years old. Table 3. Average error for respiration and heart rate of different human subjects (five humans). Table 3. Average error for respiration and heart rate of different human subjects (five humans). Movement Case Movement Case

stationary stationary hand motion near body general movements gesture made by hand hand motion near bodyaway from body general movements lips gesture made by hand away from body motion during speaking lips motion during speaking lead movement to watch the mirror of the car lead movement turning the car to watch the mirror of the car specific driving turning the car specific driving applying the brakes movements applying the brakes movements accelerating acceleratingthe thecar car

Average Error (Bpm)

Average Error (Bpm)

Average Error (Bpm) (Bpm) (Respiration Rate) Average Error (Heart Rate) (Respiration (Heartalmost Rate) zero almostRate) zero almost zero almost zero 0.7 1.1 0.7 0.3 0.3 0.2 0.2

1.1 0.9 0.6

0.3 1.2 1.2 0.5 0.5 0.4 0.4

2.0 2.5 1.7 1.3

0.3

0.9 0.6 2.0 2.5 1.7 1.3

The results in Table 3 show that the algorithm works correctly and robustly regardless of the The results in Table 3 show that the algorithm works correctly and robustly regardless of person. the person. 4.3.Mobile MobilePhone PhoneDetection DetectionResults Results 4.3. In order order totoverify verifythe theproposed proposedcell cellphone phonedetection detectionalgorithm, algorithm, several several experiments experiments were were In conducted. As shown in the diagram in Figure 11, the signals received in the sensing area are conducted. As shown in the diagram in Figure 11, the signals received in the sensing area are processed in two ways to remove the background and two post-processed signals are obtained. The processed in two ways to remove the background and two post-processed signals are obtained. use use of the phone is detected whilewhile comparing the maximum valuesvalues of the of processed signals The of mobile the mobile phone is detected comparing the maximum the processed with the threshold value. Threshold 1 is 3.0 and threshold 2 is 0.8. T is 1.2 s and T is signals with the threshold value. Threshold 1 is 3.0 and threshold 2 is 0.8. Thold is 1.2 s and Ty is 1.0 1.0 s.s. Theseconfiguration configurationparameters parametershave havebeen beenoptimized optimizedso sothat thatthe thealarms alarmsare aregenerated generatedand andreleased released These naturally and the detection error is minimized through several experiments. naturally and the detection error is minimized through several experiments. Thefirst firstisisanan experiment determine whether proposed algorithm correctly recognizes The experiment to to determine whether the the proposed algorithm correctly recognizes that that the mobile phone is in use when the driver uses the mobile phone in various ways. Figure 19 the mobile phone is in use when the driver uses the mobile phone in various ways. Figure 19 shows ( ) ( ) shows the characteristics of the received signal over time for each case. Both y and y are the characteristics of the received signal over time for each case. Both yk (t) and y˘ k (t) are consistently consistently higher thanInthresholds. In this case, the state in theoccurs algorithm occurs in 1order higher than thresholds. this case, the state transition in transition the algorithm in order state (no state 1 (no detection) → state 2 (no detection or moving object) → state 3 (mobile phone detection). detection) → state 2 (no detection or moving object) → state 3 (mobile phone detection).

Sensors 2017, 17, 1240 Sensors 2017, 17,17, 1240 Sensors 2017, 1240

21 of 25

2125of 25 21 of

status 11 status

status status22

status status3 3

(a)

(a)

status 1

status 1

status 2

status 2

status 1

status 3

status 1

status 3

(b)

status 2

status 2

status 3

status 3

(c)

(b) value of the received signal over slow time when using (c) a mobile phone (a) Figure 19. The maximum

Figuretexting 19. The maximum value of the received signal over slow time when using a mobile phone (b) scrolling, touching (c) viewing. Figure (b) 19. The maximum value(c) of viewing. the received signal over slow time when using a mobile phone (a) (a) texting scrolling, touching texting (b) scrolling, touching (c) viewing. Next is whether the algorithm will detect moving objects as a mobile phone or not when there

is a moving object in algorithm the sensingwill areadetect for a while. Figure 20 shows the characteristics of the received Next is whether the moving objects as aasmobile phone or or not when there is a Next is whether algorithm willbedetect a mobile phone when ) and ) y (there signal over time forthe each case. It can seen moving that the objects magnitudes of the signals y (not moving object in the sensing area for a while. Figure 20 shows the characteristics of the received signal is ainstantaneously moving objectbecome in the sensing areathe forthresholds a while. Figure shows thebecomes characteristics received larger than and the20magnitude smaller of as the the object oversignal time for each case. Iteach can be seen theseen magnitudes the signals of yk (the t) and y˘ k (t)y instantaneously ( )→state and y2 ( ) over case. It that can case, be the of magnitudes signals moves outtime of thefor sensing area. In this statethat transitions occur as state 1 (no detection) become larger than the thresholds and the magnitude becomes smaller as the object moves outobject of the instantaneously the thresholds and the magnitude becomes smaller as the (no detection orbecome movinglarger object)than → state 1 (no detection). sensing area. In this case, state transitions occur as state 1 (no detection) → state 2 (no detection moves out of the sensing area. In this case, state transitions occur as state 1 (no detection) →state 2or moving object) → 1 (no detection). (no detection orstate moving object) → state 1 (no detection). status 2 status 1

status 1

status 1

status 2

status 1

(a)

(a) Figure 20. Cont.

Sensors 2017, 17, 1240 Sensors 2017, 17, 1240

22 of 25

Sensors 2017, 17, 1240

22 of 25

status 1 status 1

22 of 25

status 2 status 2

status 1

status 1

status 1

status 1

(b) (b)

status 2

status 1

status 2

status 1

(c) (c)

Figure 20. The maximum value of the received signal over slow time with momentary movements. Figure 20. The maximum value of the received signal over slow time with momentary movements. (a) moving hand (b) moving water bottle (c) moving cell phone. Figure 20. The maximum value of the received signal over slow time with momentary movements. (a) moving hand (b) moving water bottle (c) moving cell phone. (a) moving hand (b) moving water bottle (c) moving cell phone.

Next is, if there is a change of the background object in the sensing area, whether the proposed Next is, is, ifdetects there isthis ais change ofof the background object in sensing area,Figure whether the proposed algorithm as the use of the mobile phone or area, not. 21the shows the Next if there a change change the background object in the the sensing whether proposed algorithm detects this change as the use of the mobile phone or not. Figure 21 shows the characteristics characteristics of the reflected signal over time for each case. algorithm detects this change as the use of the mobile phone or not. Figure 21 shows the of the reflected signal time for each case. characteristics of theover reflected signal over time for each case. status 1 status 1

status 2 status 2

status 4 -> status 1 status 4 -> status 1

(a) (a)

status 1

status 2

status 4->status 1

status 1

status 2

status 4->status 1

(b) (b)

Figure 21. The maximum value of the received over slow time when there is a background change (a) water (appearing) water bottle (disappearing) Figurebottle 21. The maximum(b) value of the received over slow time when there is a background change (a) Figure 21. The maximum value of the received over slow time when there is a background change water bottle (appearing) (b) water bottle (disappearing) (a) water bottle (appearing) water bottle is consistently greater than the threshold 1, but the The magnitude of the(b)signal y ( ) (disappearing)

magnitude of y ( ) isofonly object enters the sensing area,threshold and soon1,it but become ( ) isthe consistently greater than the the The magnitude theincreased signal y when The magnitude the2. signal y˘case, t is consistently greater than the threshold 1, but the magnitude ( ) less than theofthreshold In this the state transition occurs as state 1 (no detection) → state 2 (no ) is magnitude y ( of only increased when the object enters the sensing area, and soon it become k detection or moving object) →the state 4the (change ofthe background object) state (no detection) as of yless is only increased object enters sensing area, and→ it1become than the the threshold 2.when In this case, state transition occurs as state 1soon (no detection) →less state 2or(no k ( t )than state 1 (no detection) → state 2 (no detection or moving object) → state 3 (mobile phone detection) → detection or moving object) → state 4 (change of background object) → state 1 (no detection) or as threshold 2. In this case, the state transition occurs as state 1 (no detection) → state 2 (no detection (change of object) → state (noobject) detection). shows the detection results state 14 (no detection) → state 2 (no detection or 1moving object) →Table state 34 (mobile phone or moving object) → background state 4 (change of background → state 1 (no detection) ordetection) as state 1→ (no when45(change people 50 repetitive actions for eachdetection). experimental case. There the is no miss detection state of2 background object) → stateobject) 1 (no Table 4 shows detection detection) → staterepeated (no detection or moving → state 3 (mobile phone detection) →results state 4 at all of and it alsorepeated shows that there is no alarm when there 4isshows nocase. cellthe phone Moreover, the when 5 people 50 repetitive actions for each experimental There isuse. no miss detection (change background object) → state 1 false (no detection). Table detection results when proposed algorithm does not generate false alarms for instantaneous hand gestures or some moving at all and it also shows that there is no false alarm when there is no cell phone use. Moreover, the 5 people repeated 50 repetitive actions for each experimental case. There is no miss detection at all objects even if athere new background object appears algorithm does not falsewhen alarms fororinstantaneous hand gestures or some moving andproposed it alsoand shows that is nogenerate false alarm there isdisappears. no cell phone use. Moreover, the proposed objects and even if a new background object appears or disappears. algorithm does not generate false alarms for instantaneous hand gestures or some moving objects and

even if a new background object appears or disappears.

Sensors 2017, 17, 1240

23 of 25

Table 4. Mobile phone detection results. Object

Detail Action

Detection Rate

False Alarm Rate

mobile phone

texting scrolling, touching viewing

100% (1) 100% (1) 100% (1)

0% (2)

moving object for a moment

moving hand moving water bottle moving cell phone

change of background

water bottle (appearing) water bottle (disappearing)

N/A

0% (3) 0% (3) 0% (3) 0% (4) 0% (4)

(1)

The probability of being recognized by a mobile phone when using a real mobile phone. (2) The probability of recognizing the use of a mobile phone even though there is nothing in the detection area. (3) The probability of detecting an instantly moving object as the use of mobile phone. (4) The probability of detecting change of background as the use of mobile phone.

5. Conclusions We have presented techniques based on IR-UWB radar that can be helpful in developing methods for preventing car crashes from happening. The monitoring of vital signs related to drowsiness driving and the detection of mobile phone usage by the driver are the two main purposes of this paper. Firstly, we presented how to extract vital signs signal from the measurements during motion of the body due to driving activity. We found the fast time locations which have better information about the vital signal of the driver. We divided the signals at those locations into segments and constructed the vital signal based on the correlation concept. After construction of the vital signal, an FFT algorithm was applied and the respiration and heart rate were found. Another objective of this paper was to detect the use of mobile phones while driving. The proposed algorithm distinguishes and detects the driver’s cell phone use from various other actions or changes inside the car using the dual mode background subtraction method. Experimental results shown that the proposed mobile phone detection algorithm works perfectly in most of the scenarios that can occur in a car. If these technologies are combined, then it may be very useful for avoiding the car crashes due to drowsiness or mobile phone usage of drivers. Acknowledgments: This research was supported by the Research Grant from Hanyang University through the Korea Agency for Infrastructure Technology Advancement funded by the Ministry of Land, Infrastructure and Transport of the Korean government (17RTRP-B067918-05). Author Contributions: Seong Kyu Leem and Faheem Khan designed and carried out the experiments, computer simulations and wrote the paper under the supervision of Sung Ho Cho. Conflicts of Interest: The authors declare no conflict of interest.

References 1. 2.

3.

4. 5.

World Health Organization. Violence and Injury Prevention. Global Status Report on Road Safety 2013: Supporting a Decade of Action; World Health Organization: Geneva, Switzerland, 2013. Horswill, M.S.; McKenna, F.P. Drivers’ hazard perception ability: Situation awareness on the road. In A Cognitive Approach to Situation Awareness: Theory and Application; Banbury, S., Tremblay, S., Eds.; Ashgate: Aldershot, UK, 2004; pp. 155–175. Klauer, S.G.; Dingus, T.A.; Neale, V.L.; Sudweeks, J.D.; Ramsey, D.J. The Impact of Driver Inattention on Near-Crash/Crash Risk: An Analysis Using the 100-Car Naturalistic Driving Study Data; National Highway Traffic Safety Administration: Washington, DC, USA, 2006. Romin, L.A.; John, A.; Bissonette, J.A. Deer: Vehicle collisions: Status of state monitoring activities and mitigation efforts. Wildl. Soc. Bull. 1996, 24, 276–283. Retting, R.A.; Ferguson, S.A.; McCartt, A.T. A review of evidence-based traffic engineering measures designed to reduce pedestrian–motor vehicle crashes. Am. J. Public Health 2003, 93, 1456–1463. [CrossRef] [PubMed]

Sensors 2017, 17, 1240

6. 7. 8. 9. 10.

11. 12.

13. 14. 15. 16. 17. 18. 19.

20. 21. 22.

23. 24.

25.

26. 27. 28.

24 of 25

Wanvik, P.O. Effects of road lighting: An analysis based on Dutch accident statistics 1987–2006. Accid. Anal. Prev. 2009, 41, 123–128. [CrossRef] [PubMed] Pérez, J.; Seco, F.; Milanés, V.; Jiménez, A.; Díaz, J.C.; De Pedro, T. An RFID-based intelligent vehicle speed controller using active traffic signals. Sensors 2010, 10, 5872–5887. [CrossRef] [PubMed] Jones, W.D. Keeping cars from crashing. IEEE Spectr. 2001, 38, 40–45. [CrossRef] Vahidi, A.; Eskandarian, A. Research advances in intelligent collision avoidance and adaptive cruise control. IEEE Trans. Intell. Transp. Syst. 2003, 4, 143–153. [CrossRef] Widmann, G.R.; Daniels, M.K.; Hamilton, L.; Humm, L.; Riley, B.; Schiffmann, J.K.; Schnelker, D.E.; Wishon, W.H. Comparison of Lidar-Based and Radar-Based Adaptive Cruise Control Systems; SAE Technical Paper No. 2000-01-0345; SAE International: Warrendale, PA, USA, 2000. [CrossRef] Ji, Q.; Zhu, Z.; Lan, P. Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans. Veh. Technol. 2004, 53, 1052–1068. [CrossRef] Rodríguez-Ibáñez, N.M.; García-González, A.; Fernández-Chimeno, M.; Ramos-Castro, J. Drowsiness detection by thoracic effort signal analysis in real driving environments. In Proceedings of the 2011 Annual International Conference IEEE Engineering in Medicine and Biology Society (EMBS), Boston, MA, USA, 20 August–3 September 2011. Liu, C.C.; Hosking, S.G.; Lenné, M.G. Predicting driver drowsiness using vehicle measures: Recent insights and future challenges. J. Saf. Res. 2009, 40, 239–245. [CrossRef] [PubMed] Forsman, P.M.; Vila, B.J.; Short, R.A.; Mott, C.G.; Van Dongen, H.P.A. Efficient driver drowsiness detection at moderate levels of drowsiness. Accid. Anal. Prev. 2013, 50, 341–350. [CrossRef] [PubMed] Fan, X.; Yin, B.-C.; Sun, Y.-F. Yawning detection based on Gabor wavelets and LDA. J. Beijing Univ. Technol. 2009, 35, 409–413. Zhang, Z.; Zhang, J. A new real-time eye tracking based on nonlinear unscented Kalman filter for monitoring driver fatigue. J. Control Theory Appl. 2010, 8, 181–188. [CrossRef] Yin, B.-C.; Fan, X.; Sun, Y.-F. Multiscale dynamic features based driver fatigue detection. Int. J. Pattern Recognit. Artif. Intell. 2009, 23, 575–589. [CrossRef] Akin, M.; Kurt, M.B.; Sezgin, N.; Bayram, M. Estimating vigilance level by using EEG and EMG signals. Neural Comput. Appl. 2008, 17, 227–236. [CrossRef] Khushaba, R.N.; Kodagoda, S.; Lal, S.; Dissanayake, G. Driver drowsiness classification using fuzzy wavelet-packet-based feature-extraction algorithm. IEEE Trans. Biomed. Eng. 2011, 58, 121–131. [CrossRef] [PubMed] Liang, W.C.; Yuan, J.; Sun, D.C.; Lin, M.H. Changes in physiological parameters induced by indoor simulated driving: Effect of lower body exercise at mid-term break. Sensors 2009, 9, 6913–6933. [CrossRef] [PubMed] Yang, G.; Lin, Y.; Bhattacharya, P. A driver fatigue recognition model based on information fusion and dynamic Bayesian network. Inf. Sci. 2010, 180, 1942–1954. [CrossRef] Kokonozi, A.K.; Michail, E.M.; Chouvarda, I.C.; Maglaveras, N.M. A study of heart rate and brain system complexity and their interaction in sleep-deprived subjects. In Proceedings of the 35th Annual Computers in Cardiology Conference, Bologna, Italy, 14–17 September 2008; pp. 969–971. Reyner, L.A.; Horne, J.A. Falling asleep whilst driving: Are drivers aware of prior sleepiness? Int. J. Leg. Med. 1998, 111, 120–123. [CrossRef] Mittal, A.; Kumar, K.; Dhamija, S.; Kaur, M. Head movement-based driver drowsiness detection: A review of state-of-art techniques. In Proceedings of the 2016 IEEE International Conference on Engineering and Technology (ICETECH), Coimbatore, India, 17–18 March 2016. Davide, D.; Decarli, N.; Guerra, A.; Al-Rimawi, A.; Puchades, V.M.; Prati, G.; De Angelis, M.; Fraboni, F.; Pietrantoni, L. High-Accuracy Tracking Using Ultrawideband Signals for Enhanced Safety of Cyclists. Mob. Inf. Syst. 2017, 2017, 8149348. Khan, F.; Leem, S.L.; Cho, S.H. Hand-Based Gesture Recognition for Vehicular Applications Using IR-UWB Radar. Sensors 2017, 17, 833. [CrossRef] [PubMed] Win, M.Z.; Dardari, D.; Molisch, A.F.; Wiesbeck, W.; Zhang, J. History and applications of UWB. Proc. IEEE 2009, 97, 198–204. [CrossRef] Khan, F.; Cho, S.H. A Detailed Algorithm for Vital Sign Monitoring of a Stationary/Non-Stationary Human through IR-UWB Radar. Sensors 2017, 17, 290. [CrossRef] [PubMed]

Sensors 2017, 17, 1240

29.

30.

31. 32. 33. 34. 35. 36.

37. 38.

39.

40. 41. 42. 43. 44. 45. 46. 47.

25 of 25

Khan, F.; Choi, J.W.; Cho, S.H. Vital sign monitoring of a non-stationary human through IR-UWB radar. In Proceedings of the 4th IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC), Beijing, China, 19–21 September 2014. Lazaro, A.; Girbau, D.; Villarino, R. Techniques for clutter suppression in the presence of body movements during the detection of respiratory activity through UWB radars. Sensors 2014, 14, 2595–2618. [CrossRef] [PubMed] Hu, X.; Jin, T. Short-Range Vital Signs Sensing Based on EEMD and CWT Using IR-UWB Radar. Sensors 2016, 16, 2025. [CrossRef] [PubMed] Engelberg, J.K.; Hill, L.L.; Rybar, J.; Styer, T. Distracted driving behaviors related to cell phone use among middle-aged adults. J. Transp. Health 2015, 2, 434–440. [CrossRef] Odero, W.; Garner, P.; Zwi, A. Road traffic injuries in developing countries: A comprehensive review of epidemiological studies. Trop. Med. Int. Health 1997, 2, 445–460. [CrossRef] [PubMed] Nasar, J.L.; Troyer, D. Pedestrian injuries due to mobile phone use in public places. Accid. Anal. Prev. 2013, 57, 91–95. [CrossRef] [PubMed] Violanti, J.M.; Marshall, J.R. Cellular phones and traffic accidents: An epidemiological approach. Accid. Anal. Prev. 1996, 28, 265–270. [CrossRef] Yusuf, A.; Bulan, O.; Loce, R.P.; Paul, P. Driver cell phone usage detection from HOV/HOT NIR images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Columbus, OH, USA, 23–28 June 2014. Smith, M.J.; Stephens, D.R. Detecting Use of a Mobile Device by a Driver of a Vehicle, Such as an Automobile. U.S. Patent Application No. 13/290,126, 23 August 2012. Xu, B.; Loce, R.P. A machine learning approach for detecting cell phone usage. In Proceedings of the IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, San Francisco, CA, USA, 4 March 2015. Yim, D.H.; Cho, S.H. An Equidistance Multi-Human Detection Algorithm Based on Noise Level Using Mono-static IR-UWB Radar System. In Proceedings of the 2014 International Conference on Future Communication, Information and Computer Science (FCICS 2014), Beijing, China, 22–23 May 2014. Tomas, A. Parameter Estimation and Waveform Fitting for Narrowband Signals. Doctoral Thesis, KTH Royal Institute of Technology, Stockholm, Sweden, 2005. Rife, D.; Boorstyn, R. Single tone parameter estimation from discrete-time observations. IEEE Trans. Inf. Theory 1974, 20, 591–598. [CrossRef] Petre, S.; Moses, R.L.; Friedlander, B.; Soderstrom, T. Maximum likelihood estimation of the parameters of multiple sinusoids from noisy measurements. IEEE Trans. Acoust. Speech Signal Process. 1989, 37, 378–392. Steel, R.G.D.; Torrie, J.H. Principles and Procedures of Statistics with Special Reference to the Biological Sciences; Mcgraw-Hill Book Company: New York, NY, USA, 1960. Yano, S.M. Investigating the ultra-wideband indoor wireless channel. In Proceedings of the IEEE 55th Vehicular Technology Conference, VTC Spring 2002, Birmingham, AL, USA, 6–9 May 2002. PSL-iECG2 (Mini-Size 2ch ECG Sensor Module with Isolation). Available online: http://physiolab.en.ec21. com/PSL-iECG2_Mini-size_2ch_ECG_sensor--9063127_9816937.html (accessed on 29 May 2017). AL-Khalidi, F.Q.; Saatchi, R.; Burke, D.; Elphick, H.; Tan, S. Respiration rate monitoring methods: A review. Pediatr. Pulmonol. 2011, 46, 523–529. [CrossRef] [PubMed] Nam, Y.; Reyes, B.A.; Chon, K.H. Estimation of Respiratory Rates Using the Built-in Microphone of a Smartphone or Headset. IEEE J. Biomed. Health Inform. 2016, 26, 1493–1501. [CrossRef] [PubMed] © 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).