ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

Contents lists available at ScienceDirect

ISA Transactions journal homepage: www.elsevier.com/locate/isatrans

Human activity recognition based on feature selection in smart home using back-propagation algorithm Hongqing Fang a,n, Lei He a, Hao Si a, Peng Liu b, Xiaolei Xie a a b

College of Energy & Electrical Engineering, Hohai University, 8 Focheng West Road, Nanjing, Jiangsu 211100, PR China Marvell Semiconductor Inc, Santa Clara, CA 95054, USA

art ic l e i nf o

a b s t r a c t

Article history: Received 7 October 2013 Received in revised form 29 May 2014 Accepted 17 June 2014 This paper was recommended for publication Dr. Ahmad B. Rad

In this paper, Back-propagation(BP) algorithm has been used to train the feed forward neural network for human activity recognition in smart home environments, and inter-class distance method for feature selection of observed motion sensor events is discussed and tested. And then, the human activity recognition performances of neural network using BP algorithm have been evaluated and compared with other probabilistic algorithms: Naïve Bayes(NB) classifier and Hidden Markov Model(HMM). The results show that different feature datasets yield different activity recognition accuracy. The selection of unsuitable feature datasets increases the computational complexity and degrades the activity recognition accuracy. Furthermore, neural network using BP algorithm has relatively better human activity recognition performances than NB classifier and HMM. & 2014 ISA. Published by Elsevier Ltd. All rights reserved.

Keywords: Human activity recognition Sensors and networks Pervasive computing Feature selection Smart home

1. Introduction With the advent of smart home [1–16] technologies, people with cognitive impairments can lead independent lives in their homes for longer time. Smart homes can assist their residents by acting as a cognitive prosthesis, by handling various appliances/ objects and also by facilitating emergency communication. Furthermore, cognitive health assessments performed in clinical settings do not always provide an adequate representation of a patient's behavior. Real life assessments of Activities of Daily Living (ADLs) [13–23] can provide a better understanding of the subject than assessments performed in a clinical setting [24]. Computer vision sensing often works in the laboratory but fails in real home settings due to clutter, variable lighting, and highly varied activities. Video feeds have been used for activity recognition. Sensors such as microphones and cameras are commonly used as recording devices. However, cameras and microphones face another challenge because they are perceived as invasive by most people [25,26]. Alternatively, motion sensor data can be used to recognize real-life activities performed in a smart home. Smart homes provide continuous monitoring capability that conventional methodologies lack. Being able to automate the activity recognition from human motion patterns using unobtrusive

n

Corresponding author. E-mail address: [email protected] (H. Fang).

sensors or other devices can be useful in monitoring older adults in their homes and keeping track of their ADLs and behavioral changes. This could lead to a better understanding of numerous medical conditions and treatments. The Center for Advanced Studies in Adaptive Systems (CASAS) smart home project [8–10] is a multi-disciplinary research project at Washington State University, which focused on the creation of an intelligent home environment. The approach is to view a smart home as an intelligent agent that perceives its environment through the use of sensors, and can act upon the environment through the use of actuators. The research goals of the CASAS smart home project are to enhance and improve the quality of life, prolong stay at home with technology-enabled assistance, minimize the cost of maintaining the home and maximize the comfort of its inhabitants. In order to achieve these goals, smart home must be able to reason about and adapt to provide information. To implement the goals of the CASAS smart home project, a primary challenge is to design an algorithm that labels the activity performed by an inhabitant in a smart environment from the sensor data collected by the environment during the activity. Medical professionals also believe that one of the best ways to detect emerging medical conditions before they become serious is to look for changes in the ADLs. Recently, human activity discovery and recognition has gained a lot of interest due to its enormous potential in context aware computing systems, including smart home environments. To recognize residents' activities and their daily routines can greatly help in providing automation, security,

http://dx.doi.org/10.1016/j.isatra.2014.06.008 0019-0578/& 2014 ISA. Published by Elsevier Ltd. All rights reserved.

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

2

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

and more importance in remote health monitoring of elder or people with disabilities. The main object of activity recognition in smart home environments is to find interesting patterns of behavior from sensor data and to recognize such patterns. Researchers have commonly tested the machine learning algorithms such as knowledge-driven approach(KDA), evolutionary ensembles model (EEM), support vector machine (SVM), Dempster–Shafer theory of evidence(D–S), Naïve Bayes(NB) classifier, Markov model (MM), hidden Markov model (HMM) and conditional random fields (CRF) [13–23], [27–34], etc., for human activity (pattern) recognition in smart home environments. NB classifier uses the relative frequencies of feature values as well as the frequency of activity labels found in the sample training data to learn a mapping from activity features to an activity label. HMM is a statistical approach in which the underlying model is a stochastic Markovian process that is not directly observable (i.e., hidden). It can only be observed through other processes that produce the sequence of the observed features. The hidden nodes represent activities and the observable nodes represent combinations of the selected features. The probabilistic relationships between hidden nodes and observable nodes and the probabilistic transition among the hidden nodes are estimated through the relative frequency with which these relationships occur in the sample data. Given an input sequence of sensor events, the Viterbi algorithm finds the most likely sequence of hidden states, or activities, which could have generated the observed event sequence. Like HMM, CRF model makes use of the transition likelihoods between states and the emission likelihoods between the activity states and the observable states to output a label for the current data point. CRF learns a label sequence which corresponds to the observed sequence of features. Unlike HMM, weights are applied to each of the transition and emission features. These weights are learned through an expectation maximization process based on the training data. However, these methods usually have some defects to some extent. For examples, NB classifier is a simple probabilistic classifier based on the application of the Bayes' theorem, but the independence assumptions of variables are not always accurate. Another key problem is that these approaches require the knowledge about the probabilities, therefore, they yield lower accuracy rate with fewer observed samples, especially. Therefore, an approach which is not sensitive to the amount of available data is especially reasonable for activity recognition in smart home. Besides, the datasets include a large number of sensor events generated by various activities and any activity annotated in dataset has various features [35,36]. However, these feature values are usually selected in one method in all tests, and the influences of these feature values on the activity recognition performance are seldom addressed. Furthermore, the activity recognition accuracy rate generated by different algorithms should be evaluated and compared. Since neural network using BP algorithm [37,38] has proven successful in many practical problems such as learning to recognize handwritten characters, spoken words as well as human faces, therefore, neural network using BP algorithm is applied for human activity recognition in smart home environments in this paper. Besides, inter-class distance [39] method for feature selections of observed motion sensor events is discussed and tested. And then, the activity recognition performance of neural network using BP algorithm is evaluated and compared with other probabilistic algorithms: NB classifier and HMM. The rest of the paper is organized as follows. Section 2 describes the designing of neural network using BP algorithm applied to represent and recognize human activities, the smart apartment testbed and data collection as well as the application of inter-class distance method for feature selections of observed sensor events. Section 3 presents the comparison results of activity

recognition accuracy of the different feature datasets and the performance measures of the three algorithms: neural network using BP algorithm, NB classifier as well as HMM. Section 4 summarizes the main contributions.

2. Neural network using BP algorithm applied for human activity recognition and feature selections 2.1. Model of neural network using BP algorithm Neural network using BP algorithm is a typical forward network that consists of input layer, hidden layer and output layer, which can be used to model complex relationship between inputs and outputs or to find patterns in data. In the supervised model, the error back propagation algorithm is applied to train MultiLayer-Perception (MLP) which is the simplest multilayer feedforward network of neural network. According to Kolmogorov's theorem, neural network using BP algorithm with one hidden layer can uniformly approximate any continuous function on a compact input domain to arbitrary accuracy if the network has a sufficiently large number of hidden units. A training set comprises a set of input vector X ¼ ðX 1 ; X 2 ; ⋯; X i ; ⋯; X n ÞΤ (n input units), correspondingly, there is a set of target vector D ¼ ðD1 ; D2 ; ⋯; Dk ; ⋯; Dl ÞΤ (l output units). Y ¼ ðY 1 ; Y 2 ; ⋯; Y j ; ⋯; Y m ÞΤ is the hidden vector (m hidden nodes). Usually, x0 and y0 are set to be  1 for the bias value. The output vector is O ¼ ðO1 ; O2 ; ⋯; Ok ; ⋯; Ol ÞΤ . The connection weight vector between neurons in input layer and hidden layer is V ¼ ðV 1 ; V 2 ; ⋯; V j ; ⋯; V m Þ. W ¼ ðW 1 ; W 2 ; ⋯W k ; ⋯; W l Þ is the connection weight vector between neurons in hidden layer and output layer. In the output layer, ok , the value of the kth neuron, and the activation net k , the kth value of the sum of the weighted values of hidden nodes, are given as ok ¼ f ðnet k Þ m

net k ¼ ∑ wjk yj ; j¼0

k ¼ 1; 2; ⋯; l k ¼ 1; 2; ⋯; l

ð1Þ ð2Þ

where wjk is the weight between the kth neuron in output layer and the jth neuron in hidden layer, yj is the output value of the jth neuron in hidden layer. In the hidden layer, yj ¼ f ðnet j Þ n

net j ¼ ∑ vij xi i¼0

j ¼ 1; 2; ⋯; m j ¼ 1; 2; ⋯; m

ð3Þ ð4Þ

where vij is the weight between the ith neuron in input layer and the jth neuron in hidden layer, xi is the input value of the ith neuron in input layer, net j is the jth value of the sum of the weighted values of input nodes. The active function f(x) in (1) and (3) is Sigmoid function f ðxÞ ¼

1 1þex

ð5Þ

A simple approach to determine the network parameters is to minimize the sum-squares-error function 1 1 l E ¼ ðD  OÞ2 ¼ ∑ ðdk  ok Þ2 2 2k¼1

ð6Þ

Neural network using BP algorithm with a momentum factor is to avoid local minimum values. The essence is to transfer the influence of the last weight variation through a momentum factor, which modifies each weight variation by adding an extra value that is proportional to the former one to produce a new weight variation. In the ðt þ 1Þth iteration, the value of modification of the

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

weight between the kth neuron in output layer and the jth neuron in hidden layer, Δwkj , is

Δwkj ðt þ1Þ ¼ ð1  mcÞηδok oj þ mcΔwkj ðtÞ

ð7Þ

The value of modification of the weight between the jth neuron in hidden layer and the ith neuron in input layer, Δvji , is

Δvji ðt þ 1Þ ¼ ð1  mcÞηδhj xi þ mcΔvji ðtÞ

ð8Þ

where mc is momentum factor, which usually has a value of 0.9, η h o is learning rate which has a value of 0.005, δpj and δpk are the error for the jth neuron in the hidden layer and the kth neuron in the output layer, respectively, ohpj and xpi are the jth neuron in hidden layer and the ith neuron in input layer. From (7) and (8), it can be found that when the correction of weight of last iteration is oversized, the symbol of the first item in (7) and (8) will be contrary to that of correction of last iteration, in order to reduce the correction of current iteration and lower the oscillation; when the correction of last iteration is undersized, the symbol of the first item in (7) and (8) will be the same with that of correction of last iteration, in order to amplify the correction of current iteration and speed up the correction.

3

The data gathered by CASAS smart home is represented by the following parameters, which specify the number of features that are used to describe the sensor events. The generalized syntax of the dataset is Date Time SensorIDSensor Value o label 4 An example of the dataset of Night_wandering activity is: { 2009-06-10 03:20:59.08 M006 ON Night_wandering begin 2009-06-10 03:25:19.05 M012 ON 2009-06-10 03:25:19.08 M011 ON 2009-06-10 03:25:24.05 M011 OFF 2009-06-10 03:25:24.07 M012 OFF Night_wandering end }

This example shows the sensor events correspond to the Night_wandering activity with concrete Date, Time, Sensor ID, Sensor Value as well as activity label parameters.

2.3. Feature selections 2.2. Smart apartment testbed and data collection The smart apartment testbed for this research is located on Washington State University campus and is maintained as part of the ongoing CASAS smart home project, which includes three bedrooms, one bathroom, a kitchen, and a living/dining room. The smart apartment is equipped with motion sensors distributed approximately 1 m apart throughout the space on the ceilings, shown in Fig. 1. In addition, other sensors installed provide ambient temperature readings and custom-built analog sensors provide readings for hot water, cold water, and stove burner use. Sensor data is captured using a sensor network that was designed in-house and is stored in a Structured-Query-Language (SQL) database. After collecting data from the smart apartment testbed, the sensor events are annotated for ADLs, which are used for training and testing the activity recognition algorithms.

The input vector of neural network using BP algorithm for activity recognition must be a specific activity feature dataset, which is selected from the sensor events. Considering the actual situation, each activity has 10 features of the sensor events: (1) The means of logical values of Sensor IDs of each activity's sensor events; In this paper, Sensor ID is mapped to a logical value by the location in the laboratory. Considering the place where each activity happens is relatively stable, therefore, selecting the average of Sensor IDs means the focus area where the activity occurs. The equation is: Si ¼

1 ni ∑ S ni k ¼ 1 ik

ð9Þ

Fig. 1. The smart apartment testbed and sensors in the apartment to monitor motion.

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

4

where Si is the means of Sensor IDs of activity i, ni is the number of sensors, and Sik is the kth Sensor ID of activity i. (2) The logical value of the first Sensor ID triggered by the current activity; (3) The logical value of the last Sensor ID triggered by the current activity; (4) The variance of all Sensor IDs triggered by the current activity is Si 2 ¼

1 ni ∑ ðS  S i Þ2 ni k ¼ 1 ik

ð10Þ

(5) (6) (7) (8)

The beginning time of the current activity; The ending time of the current activity; The duration of the current activity; Day of week, which is converted into a value in the range of 0 to 6; (9) Previous activity, which represents the activity that occurred before the current activity; (10) Activity length, which the current activity measured in terms of the number of sensor events. Usually, the best feature subset contains the least number of dimensions that contribute to higher recognition accuracy, therefore, it is necessary to remove the remaining and unimportant features to reduce the computational complexity. For feature I, the expectation of activity a and activity b are:

Table 1 Ranking of the ten features with average inter-class distance values. Feature(I) D(I)

2

6

5

1

3

7

10

4

9

8

671.16 40.11 39.75 28.50 25.78 21.18 16.49 10.34 2.91 0.98

(2) Calculate the distance between activity a and activity b for the Ith feature, (dI ða; bÞ); (3) Calculate the average distance DðIÞ for all activities, and then list the ranks of the ten features with DðIÞ in Table 1; (4) The top K features with large average inter-class distance values in Table 1 are selected, e.g., if K ¼3, which means the 3 selected features are the fist triggered Sensor ID (I ¼2), the ending time of activity (I ¼6) as well as the beginning time of activity (I ¼5); if K ¼4, the 4 selected features are the first triggered Sensor ID (I¼ 2), the ending time of activity (I ¼6), the beginning time of activity (I¼ 5) as well as the means of logical values of Sensor IDs of each activity event (I¼ 1), etc.; (5) Design the structures of neural network using BP algorithm for the selected top K features subset; (6) Determine which feature subset is the relatively better one based on activity recognition accuracy.

δ2 ða; IÞ ¼

1 Na ∑ ðx  eða; IÞÞ2 Na j ¼ 1 aj

ð13Þ

Table 1 shows that feature 9 and 8 have relatively small values of average inter-class distance. These small values of D(I) show that these features are not discriminatory, which means the features of previous activity (I ¼9) and day of week (I ¼8) are least useful for activity recognition. Therefore, these two features are ignored. In this paper, K is set to be 3, 4, 5, 6, 7 and 8, respectively, and then, the activity recognition accuracy results are compared. The selected feature subsets are: Subset 1 (K ¼3, which means features 2, 6 and 5 are selected), Subset 2 (K¼ 4, which means features 2, 6, 5 and 1 are selected), Subset 3 (K ¼5, which means features 2, 6, 5, 1 and 3 are selected ), Subset 4 (K¼ 6, which means features 2, 6, 5, 1, 3 and 7 are selected}), Subset 5 (K ¼7, which means features 2, 6, 5, 1, 3, 7 and 10 are selected) as well as Subset 6 (K ¼8, which means features 2, 6, 5, 1, 3, 7, 10 and 4 are selected).

δ2 ðb; IÞ ¼

1 Nb ∑ ðx  eðb; IÞÞ2 N b j ¼ 1 bj

ð14Þ

2.4. Design of neural network using BP algorithm for activity recognition

eða; IÞ ¼

1 Na ∑ xaj Na j ¼ 1

ð11Þ

eðb; IÞ ¼

1 Nb ∑ x Nb j ¼ 1 bj

ð12Þ

where N a and N b denote the number of samples of activity a and activity b, respectively; xaj and xbj represent the Ith feature value of the jth sample of activity a and activity b, respectively. The variances of the Ith feature value of activity a and activity b are

Inter-class distance indicates the capacity to identify of each feature between two activities, which is


eða; IÞ  eðb; IÞ
dI ða; bÞ ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð15Þ δ2 ða; IÞ þ δ2 ðb; IÞ The identification capacity among all activities is DðIÞ ¼

n

n

1 ∑ ∑ dI ða; bÞ; Na¼1b¼1

I ¼ 1; 2; …; 10

ð16Þ

where n and N denote the number of activities and the number of dI ða; bÞ for all activities. Eqs. (11)–(14) are used to obtain the mean and the variance of training samples from activity a and activity b of the Ith feature, respectively. And then, the distance between the two activities of the Ith feature is achieved from (15). Finally, the average inter-class distance of the Ith feature is calculated as (16). Inter-class distance value corresponds to the identification ability of different activities. In this paper, in order to obtain better recognition accuracy, only the features with larger inter-class distance values are selected. The steps are: (1) Calculate the mean and the variance of normalized samples from ten activities for each feature;

The number of neurons in input layer is equal to K, i.e., the number of features of the selected feature subset. The value of each feature can be normalized as X¼

X X max

ð17Þ

where X is the actual value; X max is the largest value for each feature. Only one hidden layer is adopted in this paper. The initial number of the hidden neurons is determined in the general empirical formula according to Kolmgorov, which is: pffiffiffiffiffiffiffiffiffiffi n1 ¼ p þ q þ c ð18Þ where n1 is the number of hidden neurons, p is the number of input layer neurons, q is the number of neurons for the output layer and c is a constant. The weight of each neuron is initialized randomly between 0 and 1. After training, neural network using BP algorithm is loaded with the testing samples with feature subsets going to input nodes. Since a total of 10 activities were performed in the CASAS smart apartment, therefore, neural network using BP algorithm contains 10 output nodes which are denoted as O ¼ ðo0 ; o1 ; o2 ; o3 ; o4 ; o5 ; o6 ; o7 ; o8 ; o9 ÞΤ .

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

(9) Night_wandering (activity 8): The resident wanders at night (67 instances); (10) R_medicine (activity 9): The resident takes medicine (44 instances);

At the output layer, only one node produces an output close to 1 when presents with a given activity. The activity recognition result is the label of the maximum output node value, e.g., if the output vector is [0.1 0.1 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1], the corresponding recognition result is activity 2. The optimal topological structures of neural network using BP algorithm for the six feature subsets are shown in Table 9.

The data have been collected in the CASAS smart apartment testbed for 55 days, which resulted in a total of 600 instances of these activities and 647,485 collected motion sensor events. The parameters of neural network using BP algorithm are listed in Table 2. The 3-fold cross validation is applied in the data for the three algorithms under the same conditions to ensure that the experimental comparison is fair.

3. Tests results 3.1. Training and testing activities

3.2. Determination of feature subset

A total of 10 activities were performed in the CASAS smart apartment by 2 volunteers to provide physical training data for neural network using BP algorithm, NB classifier and HMM. These activities include both basic and more complex ADLs those are found in clinical questionnaires. These activities are:

The topological structures of neural network using BP algorithm affect the recognition accuracy significantly, therefore, the number of neurons of hidden layer for neural network must be selected properly. According to Eq. 18, the value of c determines the number of hidden neurons for all of the subsets. For example, choosing c¼9.877 yields 14 hidden neurons for Subset 5. Tables 3–8 show the comparison results of recognition accuracy performance of neural network using BP algorithm for the six feature subsets with 5 different numbers of neurons of hidden layer, respectively. It can be seen that the relatively better results of the numbers of neurons of hidden layer are 7 for Subset 1, 9 for Subset 2, 11 for Subset 3, 13 for Subset 4, 14 for Subset 5 and 17 for Subset 6, which are shown in Table 9. Table 10 shows the comparison results of activity recognition accuracy performance of the six different feature subsets. It can be seen that the activity recognition accuracy is lower for activity 0 and activity 5. Since the total number of instances for activity 0 is 30, and for activity 5 is only 10, respectively, it is difficult to recognize these two activities because of too few instances. It also can be found that for Subset 6, the relatively higher proportion of recognition accuracy is only shown for 10% of all the activities, i.e., only the recognition accuracy of activity 4 is better than or equal to those of the other subsets. With Subset 1, it yields relatively higher proportion of better recognition accuracy of 20% for all activities. With Subset 2 and 3, the proportion is 30%, while the proportion is

(1) Bed_to_toilet (activity 0): Transition between bed and toilet at night (30 instances); (2) Breakfast (activity 1): The resident has breakfast (48 instances); (3) Bed (activity 2): The activity of sleep in bed (207 instances); (4) C_work (activity 3): The resident works in the office area (46 instances); (5) Dinner (activity 4): The resident has dinner (42 instances); (6) Laundry (activity 5): The resident does laundry (10 instances); (7) Leave_home (activity 6): The resident leaves smart home (69 instances); (8) Lunch (activity 7): The resident has lunch (37 instances);

Table 2 Parameters of neural network using BP algorithm. Momentum factor mc

Learning rate η

Number of iteration

0.9

0.005

100,000

5

Table 3 The number of neurons of hidden layer of neural network for Subset1. Number of neurons

5 6 7 8 9

Activity label

Total

0

1

2

3

4

5

6

7

8

9

0.033 0.033 0.033 0.0 0.067

0.958 0.958 0.979 0.0 0.958

0.913 0.913 0.918 1.0 0.913

1.0 1.0 1.0 0.0 1.0

1.0 1.0 1.0 0.0 1.0

0.0 0.1 0.0 0.0 0.2

0.913 0.899 0.928 0.0 0.899

0.973 0.946 0.946 0.0 0.946

0.567 0.552 0.642 0.0 0.497

0.977 1.0 0.977 0.0 1.0

0.840 0.838 0.852 0.345 0.847

Table 4 The number of neurons of hidden layer of neural network for Subset2. Number of neurons

7 8 9 10 11

Activity label

Total

0

1

2

3

4

5

6

7

8

9

0.533 0.533 0.400 0.567 0.567

0.979 0.936 0.958 0.958 0.958

0.908 0.918 0.908 0.903 0.908

1.0 1.0 0.978 1.0 1.0

1.0 1.0 1.0 1.0 1.0

0.300 0.500 0.500 0.500 0.500

0.913 0.897 0.923 0.913 0.913

0.946 0.973 0.973 0.973 0.973

0.627 0.627 0.701 0.657 0.687

0.932 1.0 1.0 1.0 1.0

0.872 0.880 0.882 0.882 0.882

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

6

Table 5 The number of neurons of hidden layer of neural network for Subset3. Number of neurons

Activity label

9 10 11 12 13

Total

0

1

2

3

4

5

6

7

8

9

0.567 0.567 0.533 0.0 0.567

1.0 0.979 0.979 0.0 0.979

0.903 0.903 0.913 0.986 0.889

1.0 1.0 1.0 0.0 1.0

1.0 1.0 1.0 0.0 0.976

0.600 0.600 0.500 0.0 0.500

0.971 0.957 0.957 0.333 0.971

0.973 0.973 0.973 0.0 0.973

0.687 0.627 0.687 0.179 0.701

0.909 0.909 0.932 0.0 0.932

0.892 0.882 0.895 0.398 0.885

Table 6 The number of neurons of hidden layer of neural network for Subset4. Number of neurons

Activity label

11 12 13 14 15

Total

0

1

2

3

4

5

6

7

8

9

0.467 0.533 0.533 0.0 0.567

0.958 1.0 0.979 0.0 1.0

0.918 0.923 0.932 0.995 0.942

0.978 0.957 1.0 0.0 0.978

1.0 1.0 1.0 0.0 1.0

0.600 0.400 0.700 0.0 0.500

0.957 0.986 0.971 0.333 0.957

0.973 1.0 0.973 0.0 0.973

0.701 0.657 0.672 0.015 0.627

0.909 0.932 0.909 0.0 0.932

0.887 0.892 0.915 0.383 0.883

Table 7 The number of neurons of hidden layer of neural network for Subset5. Number of neurons

Activity label

12 13 14 15 16

Total

0

1

2

3

4

5

6

7

8

9

0.600 0.600 0.600 0.600 0.600

1.0 1.0 1.0 1.0 0.979

0.926 0.918 0.913 0.901 0.908

1.0 0.978 0.957 0.956 0.978

0.952 0.952 0.952 0.643 0.976

0.200 0.200 0.500 0.0 0.0

0.986 1.0 1.0 1.0 1.0

0.919 0.973 0.973 0.622 0.676

0.806 0.806 0.836 0.806 0.821

0.955 0.977 0.955 0.932 0.977

0.907 0.908 0.918 0.855 0.885

Table 8 The number of neurons of hidden layer of neural network for Subset6. Number of neurons

Activity label

15 16 17 18 19

Total

0

1

2

3

4

5

6

7

8

9

0.400 0.433 0.500 0.0 0.367

1.0 1.0 0.979 0.0 0.667

0.918 0.908 0.896 1.0 0.927

0.891 0.717 0.826 0.0 0.891

0.976 1.0 1.0 0.0 0.333

0.0 0.0 0.0 0.0 0.0

0.986 0.986 0.986 0.0 0.986

0.676 0.595 0.647 0.0 0.351

0.791 0.761 0.746 0.0 0.791

0.932 0.591 0.955 0.0 0.591

Table 9 Topological structures of neural network using BP algorithm for the six feature subsets.

Input Hidden Output

Subset1

Subset2

Subset3

Subset4

Subset5

Subset6

3 7 10

4 9 10

5 11 10

6 13 10

7 14 10

8 17 10

50% with Subset 4 and 5. Moreover, it indicates that for each activity, the relatively better feature subset is not always the same, e.g., for activity 1, selecting Subset 5 generates the best result, and the accuracy rate is 100%. However, the accuracy rate of activity 2 is slightly smaller compared to Subset 4. Focusing on the small number of instances, Subset 5 improves the recognition accuracy

0.850 0.818 0.853 0.345 0.750

for activity 0 to 60% and activity 5 to 50%, respectively. For activity 8, the recognition accuracy has been improved to 83.6%. Obviously, Subset 5 represents the relatively better recognition than others with fewer instances. Table 10 also shows the test results of total activity recognition accuracy for the six feature subsets for. It can be seen that neural network using BP algorithm achieves the optimal total activity recognition accuracy up to 91.8% when K ¼7 (Subset 5). Furthermore, the accuracy increases by 6.6%, when K updates from 3 to 7. The reason is that the input features should contain enough valuable information for classifier to recognize activity with lower error. However, when K ¼8, the accuracy rate decreases by 6.5%. Obviously, the variance of Sensor IDs with low average inter-class distance value means that redundant information degrades the recognition performance of neural network using BP algorithm. Therefore, the results indicate that the improper selection of

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

7

Table 10 Comparison results of recognition accuracy performance with the six different feature subsets. Subset

1 2 3 4 5 6

Activity label

Total

0

1

2

3

4

5

6

7

8

9

0.033 0.400 0.533 0.533 0.600 0.500

0.979 0.958 0.979 0.979 1.0 0.979

0.918 0.908 0.913 0.932 0.913 0.896

1.0 0.978 1.0 1.0 0.957 0.826

1.0 1.0 1.0 1.0 0.976 1.0

0.0 0.500 0.500 0.700 0.600 0.0

0.928 0.923 0.957 0.971 1.0 0.986

0.946 0.973 0.973 0.973 0.973 0.647

0.642 0.701 0.687 0.672 0.836 0.746

0.977 1.0 0.932 0.909 0.955 0.955

features increases the computational complexity and degrades the activity recognition accuracy. Fig. 2(a)–(f) shows the comparison results of training error convergence curves of the six subsets with different number of neurons in the hidden layers. Fig. 2(g) shows the comparison results of training error convergence curves of the six subsets, and the numbers of neurons in the hidden layers are listed in Table 9, which yields relatively better recognition performances with each subset. It can be seen from Fig. 2(g) that neural network using BP algorithm tends to converge in 100,000 iterations. However, the training error convergence curves show minor differences of the six subsets with different number of neurons in the hidden layers. The convergence errors from Subset 1 to 4 are larger than the target error. In Subset 5, it converges to the target error after 80,000 iterations, while, after 60,000 iterations it converges to the target error in Subset 6. The topological structures of neural network using BP algorithm are different for the six feature subsets, as shown in Table 9, which result in significantly different training performances. It should be noted that each structure and parameters of neural network using BP algorithm for each subset is verified after several experiments. Moreover, increasing the feature dimension will decrease the iterations, but increase the training time. In summary, considering the factors including total accuracy and training error convergence rate, the feature subset is set to Subset 5.

3.3. Comparison results with other algorithms In this section, the activity recognition accuracy of neural network using BP algorithm is compared to NB and HMM with the same feature subset for these 10 activities. Tables 11 and 12 show the comparison results of activity recognition accuracy performance of NB and HMM with the six different feature subsets, respectively. For NB, it can be seen from Table 11 that with Subset 3 only the recognition accuracy of activity 4 and 7 are better than or equal to those of the other subsets, the proportion is 20% for all activities. With Subset 1 and 2, the proportion is 30%. With Subsets 5 and 6, the proportion are 60%, while Subset 6 yields the optimal total recognition accuracy of 87.5%. An exception is Subset 4, which generates the highest proportion to 70% for all activities. However, the total recognition accuracy is 86.2%, which is slightly lower than that of Subsets 5 and 6. For HMM, it can be found from Table 12 that with Subset 1 and 2, only the recognition accuracy of activity 0 is better than or equal to that of other subsets, the proportion is only 10% for all activities. With Subset 3, the proportion is 20%. With Subset 4, the proportion is 30%. Subsets 5 and 6 have the highest proportion of recognition accuracy, which is 60%. However, Subset 6 yields the optimal total recognition accuracy of 86.3%, which is slightly higher than that of Subset 5.

0.852 0.882 0.895 0.915 0.918 0.853

The comparison results of the activity recognition accuracy of neural network using BP algorithm, NB and HMM are listed in Table 13. It shows that neural network using BP algorithm performs better than NB and HMM algorithms for activity 1, 3, 5, 6, 8 and 9, and the proportion is 60% for all activities. HMM yields higher recognition accuracy for activity 1 and 3, and the proportion is 20%. NB generates slightly higher recognition accuracy for activity 2, 4 and 3, and the proportion is 30%. Particularly, neural network using BP algorithm has much higher recognition accuracy for activity 3, 8 and 9 than that of NB and HMM. Furthermore, the total recognition accuracy generated by neural network using BP algorithm reaches 91.8%, which is better than that of NB and HMM. Table 14 shows the comparison results of activity recognition instances for the three algorithms. The data in Table 14 are denoted as Aði; jÞ, which indicate the numbers that classifier identify activity i to activity j. Therefore, if i¼j, the activity recognition result is correct. It can be seen that the errors that HMM and NB make are mostly the same, e.g., HMM and NB classify 30% and 40% of activity 0 to activity 8, mistakenly. It indicates that the features of activity 0 and 8 are similar, which are difficult to distinguish. However, the error rate of neural network using BP algorithm that activity 0 is classified to activity 8 is 10%. Similarly, HMM and NB classify 21.7% and 28.3% of activity 3 mistakenly to activity 2, respectively. It is reasonable since activity 3 and activity 2 happen in the same room and trigger the similar motion sensor events. HMM and NB algorithms calculate the number of sensor events to get statistical probability for activity recognition, therefore, it is difficult to distinguish between activity 3 and activity 2. However, neural network using BP algorithm overcomes this shortcoming by analyzing all the features to adapt the weight and improves the recognition accuracy of activity 3 and activity 2. By contrast, neural network using BP algorithm only classifies 4.3% of activity 3 to activity 2. Actually, the number of instances of each activity is different. If the number of a specific activity's instances is small, it is difficult to recognize this activity by any algorithm, e.g., the number of instances of activity 5 is only 10, which is the least one in all activities, and the recognition accuracy is the lowest. However, neural network using BP algorithm has relatively better recognition performance than or equal to other algorithms for activity 5. Furthermore, neural network using BP algorithm generates 5 more right recognition numbers than that of NB for activity 9 in total of 47 instances. It shows that neural network using BP algorithm performs relatively better than HMM and NB even with small number of instances. The reason is that HMM and NB algorithms evaluate human activities based on the probabilities greatly related to the finite instances.

4. Conclusions This paper applies three machine learning models to represent and recognize human activities based on observed sensor events.

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

8

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

Fig. 2. Training error convergence curves of the six subsets. (a) Comparison results of training error convergence curves of Subset1 with different number of hidden nodes. (b) Comparison results of training error convergence curves of Subset2 with different number of hidden nodes. (c) Comparison results of training error convergence curves of Subset3 with different number of hidden nodes. (d) Comparison results of training error convergence curves of Subset4 with different number of hidden nodes. (e) Comparison results of training error convergence curves of for Subset5 with different number of hidden nodes. (f) Comparison results of training error convergence curves of Subset6 with different number of hidden nodes. (g) Comparison results of training error convergence curves of the six subsets.

From the results, it can be concluded that the human activity recognition performances of neural network using BP algorithm is better than those of NB classifier and HMM. The main reasons are that neural network using BP algorithm is immune to errors in training data since it uses gradient descent to tune network weights to best fit a training set of input–output pairs and has strong ability in learning to interpret complex sensor events in

smart home environments. Furthermore, different feature sets generate different human activity recognition accuracy, therefore, the suitable feature set must be selected in advance, and the selection of unsuitable feature sets increases the computational complexity and degrades the human activity recognition accuracy. To improve human activity recognition accuracy, an effective approach is to properly select the feature subsets.

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

9

Table 11 Comparison results of recognition accuracy performance of NB with the six different feature subsets. Subset

1 2 3 4 5 6

Activity label

Total

0

1

2

3

4

5

6

7

8

9

0.433 0.567 0.567 0.333 0.467 0.567

0.917 0.958 0.958 0.979 0.958 0.958

0.884 0.918 0.937 0.937 0.927 0.932

0.63 0.674 0.674 0.717 0.717 0.717

1.0 1.0 0.976 1.0 1.0 1.0

0.3 0.3 0.2 0.2 0.4 0.5

0.072 0.333 0.667 0.913 0.928 0.928

1.0 1.0 1.0 1.0 1.0 1.0

0.687 0.716 0.716 0.791 0.791 0.776

0.136 0.159 0.25 0.818 0.818 0.818

0.68 0.74 0.788 0.862 0.868 0.875

Table 12 Comparison results of recognition accuracy performance of HMM with the six different feature subsets. Subset

1 2 3 4 5 6

Activity label

Total

0

1

2

3

4

5

6

7

8

9

0.9 0.9 0.9 0.7 0.7 0.667

0.271 0.417 0.833 0.958 0.958 0.958

0.643 0.715 0.831 0.865 0.879 0.894

0.674 0.652 0.739 0.783 0.783 0.783

0.595 0.595 0.952 0.952 0.976 0.976

0.1 0.1 0.2 0.3 0.5 0.6

0.899 0.869 0.884 0.986 0.986 0.986

0.324 0.324 1.0 0.972 0.946 0.946

0.179 0.478 0.478 0.597 0.912 0.687

0.273 0.182 0.25 0.75 0.818 0.795

0.547 0.605 0.76 0.837 0.852 0.863

Table 13 Comparison results of activity recognition accuracy rate for the three algorithms. Algorithm

Activity

HMM NB BP

Total

0

1

2

3

4

5

6

7

8

9

0.667 0.567 0.600

0.958 0.958 1.0

0.894 0.932 0.913

0.783 0.717 0.957

0.976 1.0 0.976

0.6 0.5 0.6

0.986 0.928 1.0

0.946 1.0 0.973

0.687 0.776 0.836

0.795 0.818 0.955

Acknowledgement

Table 14 Comparison results of activity recognition samples for the three algorithms. Activity

0

1

2

3

4

5

6

7

8

9

0.863 0.875 0.918

Activity label Algorithm

0

1

2

3

4

5

6

7

8

9

HMM NB BP HMM NB BP HMM NB BP HMM NB BP HMM NB BP HMM NB BP HM NB BP HMM NB BP HMM NB BP HMM NB BP

20 17 18 0 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 7 2 0 0 0

0 0 0 46 46 47 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 2

1 1 6 1 1 1 185 193 197 10 13 2 1 0 0 0 2 0 1 5 0 0 0 0 8 8 7 7 7 1

0 0 3 0 0 0 4 2 2 36 33 44 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 41 42 41 0 0 0 0 0 0 2 0 1 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 5 6 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 3 0 68 64 65 0 0 0 0 0 0 2 1 0

0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 35 37 36 0 0 0 0 0 0

9 12 3 0 0 0 15 9 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 46 52 56 0 0 0

0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 2 0 0 4 0 0 0 0 0 2 35 36 41

Thanks to Dr. Diane J. Cook and the reviewers whose positive and constructive comments helped to enhance the quality and presentation of this paper. And the data were collected from the smart home testbed located on Washington State University campus, which can be downloaded from Dr. Cook's home page.

References [1] Alam M, Reaz M, Ali M. A Review of smart homes-past, present, and future. IEEE Trans Syst Man Cybern Part C: Appl Rev 2012;42(6):1190–203. [2] Wu C, Fu L. Design and realization of a framework for human–system interaction in smart homes. IEEE Trans Syst Man Cybern Part A: Syst Hum 2012;42(1):15–31. [3] Zhang S, McClean S, Scotney B. Probabilistic learning from incomplete data for recognition of activities of daily living in smart homes. IEEE Trans Inform Technol Biomed 2012;16(3):454–62. [4] Jalal A, Uddin M, Kim T-S. Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home. IEEE Trans Consum Electron 2012;58(3):863–71. [5] Kang K, Ka B, Kim S. A service scenario generation scheme based on association rule mining for elderly surveillance system in a smart home environment. Eng Appl Artif Intell 2012;25(7):1355–64. [6] Makonin S, Bartram L, Popowich F. A smarter smart home: case studies of ambient intelligence. IEEE Pervasive Comput 2013;12(1):58–66. [7] Yu M, Rhuma A, Naqvi S, Wang L, Chambers J. A posture recognition-based fall detection system for monitoring an elderly person in a smart home environment. IEEE Trans Inform Technol Biomed 2012;16(6):1274–86. [8] Cook D. How smart is your home? Science 2012;335(6076):1579–81. [9] Cook D. Learning setting-generalized activity models for smart spaces. IEEE Intell Syst 2012;27(1):32–8. [10] Cook, D, Schmitter-Edgecombe, M, Crandall, A, Sanders, C, Thomas, B. Collecting and disseminating smart home sensor data in the CASAS project. In: Proceedings of the CHI workshop on developing shared home behavior datasets to advance HCI and ubiquitous computing research; 2009.

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i

10

H. Fang et al. / ISA Transactions ∎ (∎∎∎∎) ∎∎∎–∎∎∎

[11] Hussain S, Erdogen S, Park J. Monitoring user activities in smart home environments. Inform Syst Front 2009;11(5):539–49. [12] Raad M, Yang L. A ubiquitous smart home for elderly. Inform Syst Front 2009;11(5):529–36. [13] Chen L, Nugent C, Wang H. A knowledge-driven approach to activity recognition in smart homes. IEEE Trans Knowl Data Eng 2012;24(6):961–74. [14] Fahim M, Fatima I, Lee S, Lee Y. EEM: evolutionary ensembles model for activity recognition in smart homes. Appl Intell 2013;38(1):88–98. [15] Fleury A, Vacher M, Noury N. SVM-based multimodal classification of activities of daily living in health smart homes: sensors, algorithms, and first experimental results. IEEE Trans Inform Technol Biomed 2010;14(2):274–83. [16] Liao J, Bi Y, Nugent C. Using the dempster-shafer theory of evidence with a revised lattice structure for activity recognition. IEEE Trans Inform Technol Biomed 2011;15(1):74–82. [17] Roy P, Bouzouane A, Giroux S, Bouchard B. Possibilistic activity recognition in smart homes for cognitively impaired people. Appl Artif Intell 2011;25 (10):883–926. [18] Cook D. Health monitoring and assistance to support aging in place. J Univers Comput Sci 2006;12(1):15–29. [19] Munguia-Tapia E, Intille S, Larson K. Activity recognition in the home setting using simple and ubiquitous sensors. Lecture Notes Comput Sci 2004;3001: 158–75. [20] Wren C, Munguia-Tapia E. Toward scalable activity recognition for sensor networks. Lecture Notes Comput Sci 2004;3987:168–85. [21] Heierman, E, Youngblood, M, Cook, D. Mining temporal sequences to discover interesting patterns. In: Proceedings of the 10th ACM international conference on knowledge discovery and data mining, workshop on mining temporal and sequential data. Seattle, WA, USA; August 2004. [22] Yin J, Yang Q, Shen D, Li Z. Activity recognition via user-trace segmentation. ACM Trans Sensor Netw 2008;4(4). [23] Stauffer C, Grimson E. Learning patterns of activity using real-time tracking. IEEE Trans Pattern Anal Mach Intell 2000;22(8):747–57. [24] Barger T, Brown D, Alwan M. Health status monitoring through analysis of behavioral patterns. IEEE Trans Syst Man Cybern Part A: Syst Hum 2005;35 (1):22–7. [25] Maurer, U, Smailagic, A, Siewiorek, D, Deisher, M Activity recognition and monitoring using multiple sensors on different body positions. In:

[26]

[27] [28]

[29] [30] [31] [32]

[33]

[34]

[35] [36]

[37] [38] [39]

Proceedings of the international workshop on wearable and implantable body sensor networks; 2006. Brdiczka, O, Reignier, P, Crowley, J. Detecting individual activities from video in a smart home. In: Proceedings of the international conference on knowledgebased and intelligent information and engineering systems; 2007. Crandall A, Cook D. Coping with multiple residents in a smart environment. J. Ambient Intell Smart Environ 2009;1(4):323–34. Crandall, A., Cook, D. Using a hidden Markov model for resident identification. In: Proceedings of the 6th international conference on intelligent environments. Kuala Lumpur, Malaysia; July 2010. p. 74–9. Rashidi P, Cook D. Keeping the resident in the loop: adapting the smart home to the user. IEEE Trans Syst Man Cybern Part A: Syst Hum 2009;39(5):949–59. Rashidi P, Cook D. An adaptive sensor mining framework for pervasive computing applications. Lecture Notes Comput Sci 2010;5840:154–74. Kim E, Helal S, Cook D. Human activity recognition and pattern discovery. IEEE Pervasive Comput 2010;9(1):48–53. Singla G, Cook D, Schmitter-Edgecombe M. Tracking activities in complex settings using smart environment technologies. Int J Biosci Psychiatry Technol 2009;1(1):25–35. Rashidi P, Cook D, Holder L, Schmitter-Edgecombe M. Discovering activities to recognize and track in a smart environment. IEEE Trans Knowl Data Eng 2011;23(4):527–39. Khusainov R, Azzi D, Achumba IE, Bersch SD. Real-time human ambulation, activity, and physiological monitoring: taxonomy of issues, techniques, applications, challenges and limitations. Sensors 2013;13(10):12852–902. Yen G, Leong W. Fault classification on vibration data with wavelet based feature selection scheme. ISA Trans. 2006;45(2):141–51. Kundu P, Panchariya P, Kundu M. Classification and authentication of unknown water samples using machine learning algorithms. ISA Trans. 2011;50(3):487–95. Khashman A. A modified backpropagation learning algorithm with added emotional coefficients. IEEE Trans Neural Netw 2008;11:1897–909. Lee C, Yang S, Ho C. Modified back-propagation algorithm applied to decisionfeedback equalization. IEE Proc Vis Image Signal Proc 2006;6:805–9. Fan Y, Chaovalitwongse W. Optimizing feature selection to improve medical diagnosis. Ann Oper Res 2010;174:169–83.

Please cite this article as: Fang H, et al. Human activity recognition based on feature selection in smart home using backpropagation algorithm. ISA Transactions (2014), http://dx.doi.org/10.1016/j.isatra.2014.06.008i