G Model
GAIPOS-4480; No. of Pages 7 Gait & Posture xxx (2015) xxx–xxx
Contents lists available at ScienceDirect
Gait & Posture journal homepage: www.elsevier.com/locate/gaitpost
A wearable smartphone-enabled camera-based system for gait assessment Albert Kim a,b,1, Junyoung Kim a,b,2, Shirley Rietdyk c,d,3, Babak Ziaie a,b,* a
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA Birck Nanotechnology Center, Purdue University, West Lafayette, IN, USA c Department of Health and Kinesiology, Purdue University, West Lafayette, IN, USA d Center for Aging and the Life Course, Purdue University, West Lafayette, IN, USA b
A R T I C L E I N F O
A B S T R A C T
Article history: Received 27 January 2015 Received in revised form 1 May 2015 Accepted 4 May 2015
Quantitative assessment of gait parameters provides valuable diagnostic and prognostic information. However, most gait analysis systems are bulky, expensive, and designed to be used indoors or in laboratory settings. Recently, wearable systems have attracted considerable attention due to their lower cost and portability. In this paper, we present a simple wearable smartphone-enabled camera-based system (SmartGait) for measurement of spatiotemporal gait parameters. We assess the concurrent validity of SmartGait as compared to a commercially available pressure-sensing walkway (GaitRite1). Fifteen healthy young adults (25.8 2.6 years) were instructed to walk at slow, preferred, and fast speed. The measures of step length (SL), step width (SW), step time (ST), gait speed, double support time (DS) and their variability were assessed for agreement between the two systems; absolute error and intra-class correlation coefficients (ICC) were determined. Measured gait parameters had modest to excellent agreements (ICCs between 0.731 and 0.982). Overall, SmartGait provides many advantages and is a strong alternative wearable system for laboratory and community-based gait assessment. ß 2015 Elsevier B.V. All rights reserved.
Keywords: Gait assessment Wearable SmartGait Variability Step width
1. Introduction Assessment of spatiotemporal gait patterns provides essential information regarding functional ability, stability, fall risk, selection of therapeutic intervention, assessment of patient progress, and mortality [1–6]. In fact, gait speed has been described as a ‘vital sign’ due to the important information it provides regarding current and predicted health status [2]. New gait assessment technologies are continually emerging, but currently available devices and systems have a number of limitations. The most accurate systems that provide a full set of gait measures are laboratory-based system which are usually expensive, and require trained personnel. For example, the Optotrak Certus (NDI, Canada)
* Corresponding author at: 2905 Browning Street, West Lafayette, IN 47906, USA. Tel.: +1 765 409 0726. E-mail addresses: [email protected] (A. Kim), [email protected] (J. Kim), [email protected] (S. Rietdyk), [email protected] (B. Ziaie). 1 Address: 1205W. State St., West Lafayette, IN 47907, USA. Tel.: +1 678 316 7733. 2 Address: 2334 South Beck Lane #307, Lafayette, IN 47909, USA. Tel.: +1 469 426 5433. 3 Address: 708 LaGrange Street, West Lafayette, IN 47906, USA. Tel.: +1 765 409 2627.
has accuracy of up to 0.1 mm and resolution of 0.01 mm [7], but the system is expensive and requires training. These systems by their nature lack field assessment capability, thus cannot provide a window into typical gait behavior when a person is completing everyday tasks including adaptations to multiple obstacles, such as stairs, ramps, curbs, and gravel [8]. In order to understand gait behavior in challenging environments, such obstacles have been replicated in the lab. While a dozen or so steps are typically captured and quantified in these replicated environments, hundreds of steps are recommended to reliably assess step width and gait speed variability [9,10]. Wearable devices based on inertial sensors can be adapted for field assessment (e.g. Physilog, GaitUp, Switzerland). Such systems extrapolate the gait measurement from the inertial sensors. Wearable devices provide a limited set of gait measures, most commonly number of steps, speed, step length (SL), and/or step time (ST) [11–14]. In addition, measurement of SL using inertial sensors requires assessment of displacement, which cannot be measured directly with inertial sensors. The acceleration signal must be integrated twice to determine displacement, but the result is confounded by the unknown constants that result from integration. These inertial-based devices require steady-state gait (i.e., excluding initiation and termination steps) and cannot
http://dx.doi.org/10.1016/j.gaitpost.2015.05.001 0966-6362/ß 2015 Elsevier B.V. All rights reserved.
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
2
measure step width (SW) and its variability, which are key parameters in assessing stability and fall risk [1,3,15,16]. Portable video recording enables a more accurate assessment technique. For example, iWalker is a system that utilizes a pair of cameras mounted on the rollator frame to capture step width [17]. This system, however, is limited to the people who use rollators and cannot be adopted for wider population. In this paper, we describe a new wearable device based on video recording from a smartphone camera mounted on the waist of an ambulatory user that provides the assessment of the measures of other wearable devices, but also SW and SW variability. After outlining the hardware and software modules of our system (SmartGait), we determine its concurrent validity with a pressure-sensing walkway (GaitRite, CIR Systems, New Jersey, USA) [16,18,19]. GaitRite was chosen as a comparison device due to its validity [18,20] and widespread use as a clinical and research tool. 2. Materials and methods 2.1. Hardware SmartGait hardware consists of a smartphone (Apple, iPhone 5s), a custom-designed waist belt with a holster, a detachable 90degree wide-angle lens (Mondizen Inc., Hilo), and two colored circular foot markers (attached on the foot dorsum and centered over the proximal phalanges) (Fig. 1). SmartGait utilizes the embedded camera of the smartphone to enable real-time assessment of gait parameters with 60 Hz sampling rate. To capture foot markers on shoes while the smartphone is in a vertical position, the viewing angle of the camera is converted from front-facing to floor-facing by using a 90-degree lens. The smartphone with the lens affixed is placed in the holster case and attached to the front of a belt. The belt has a reinforced front region to support the weight of the smartphone (iPhone5s weight is 112 g) and a 5 cm adjustable rod between the holster and the belt to reduce camera obstruction by the thighs and
clothes while walking. The smartphone angle can be adjusted on the rod to optimize video capture of the feet markers. The circular foot markers can be any color that is highly visible and can be distinguished from the surroundings (e.g., bright green markers, d = 4.6 cm). 2.2. Image processing application software in smartphone A custom software application (Xcode 5.0.2, Apple) has been developed with the target platform of iOS8 (Apple). The software algorithm has three phases: (1) the image-processing phase (Fig. 2a, left column), (2) the on-board gait assessment phase (Fig. 2a, middle column); and (3) the off-board optional postprocessing gait analysis phase (Fig. 2a, right column). The image-processing phase begins with raw image capture in RGB (Red, Green, Blue) format (Fig. 2b) of the feet and markers. This is converted to HSV (Hue, Saturation, Value) format for increasing the color of foot marker contrast with respect to the environment, resulting in a threshold image (Fig. 2c and d). In the final step, minimum circle contours are drawn over foot markers in order to identify the Cartesian coordinates of the right and left foot in pixel units (Fig. 2e). 2.3. On-board gait analysis On-board assessment provides measurement of SL, SW, ST, and speed and begins with a standing calibration: the smartphone is mounted on the belt, and the participant stands still with the feet together. The angular position of the smartphone is adjusted such that both foot markers are visible and aligned on a reference line on the smartphone screen (Fig. 2b). The calibration factor (pixels to cm) is determined for the standing position. Note that the calibration factor will change as the distance between the camera and the foot changes during the gait; this is included in the offboard processing (termed dynamic calibration of unit distance). On-board processing, however, relies on this single standing
Fig. 1. Photograph of: (a) SmartGait system and its various hardware components and (b) a participant wearing the SmartGait.
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
3
Fig. 2. (a) Application software consists of image processing, on-board gait assessments, and off-board post-processing for increased accuracy, (b) RGB image, (c) HSV image, (d) threshold image, and (e) screenshot after image processing (RGB: Red, Green, and Blue; HSV: Hue, Saturation, and Value; ST: step time; DS: double support; TM: trunk motion; FPS: frames per second; SL: step length; SW: step width).
calibration. Once the standing calibration is completed, gait assessment is initiated. The on-board software is designed to avoid computational demand on the smartphone’s CPU and is only used for data
collection and providing a raw assessment of SL, SW, ST, and speed. The trunk motion and marker size are also recorded and used in the off-board analyses in order to increase the accuracy. The trunk angular motion (assessed with the inertial sensors embedded in
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
4
the smart phone) seen in the coronal plane has the largest effect on image stability since the geometry of the viewing angle changes by the degree of trunk motion [19,20]. Foot marker size at each frame is also recorded for dynamic calibration in the off-board postprocessing (see Section 2.4). On-board assessment provides a brief on-screen summary for the users after they have stopped walking (see example in Fig. 3a). These include SL and SW for the first 7–8 steps and a report summary of the walking trial, i.e., the number of steps, average speed, average SL, average SW, total distance, and the total walking time. 2.4. Off-board gait analysis In the off-board post-processing phase, the on-board data from the SmartGait is transferred to a personal computer for further analysis. The accuracy depends primarily on the unit distance, i.e., separation between the camera and foot marker as assessed in the calibration. This, however, varies as both the foot and the trunk (i.e., camera) move during gait [22]. In on-board assessment, a single unit distance is calculated in the initial standing calibration to reduce the computational burden. However, in post-processing, dynamic calibration of unit distance is completed by re-calibrating marker size at each frame. Accuracy can also be improved by correcting distortion caused by the wide-angle lens as the foot marker usually moves from the bottom edge to the top edge of the screen, where the distortion is greatest. This is corrected using the polynomial model with predetermined coefficients. To complete the analyses needed to increase the off-board accuracy, a marker based on dynamic calibration is drawn on a blank image. The intrinsic matrix and distortion coefficients are applied to remove lens distortion; the
image is then rotated based on the measured trunk roll angle, followed by a filtering step using and Savitzky–Golay algorithm [23]. The time series of the relative difference between the two footmarkers in the anterior-posterior direction shows a sinusoidal pattern (Fig. 3b, red line), with maxima and minima during double support (DS) phase (Fig. 3b, magenta regions), when the feet are farthest apart. The thigh of the swinging limb obstructs the camera view of the foot marker during most of the swing phase (Fig. 3b, gray shaded regions), but this time period is not needed to calculate SL, SW, ST, speed, and DS. Both markers are visible during DS phase and for a short time preceding and following. The absolute value of the maxima (or minima) of each step in the anterior-posterior direction is quantified as SL. SW is quantified as the medial-lateral difference between the foot markers at the same point in time (Fig. 3b, blue line). ST is the temporal difference between one maximum SL and the following minimum SL. Abrupt changes in marker size occurs at heel contact and toe off, which are used to calculate DS duration. Gait speed is calculated for each step as SL/ST. 2.5. Validation protocol Fifteen young healthy adults participated in the study (5 female; age: 25.8 (2.6) years; height: 171.1 (8.0) m; mass: 70.1 (15.6) kg; BMI: 23.8 (4.2)). The study was approved by the Human Research Protection Program at Purdue University and all participants signed informed consent. Gait was assessed simultaneously with two systems: a pressure-sensing walkway (GaitRite) and SmartGait. Participants were instructed to walk along an 8 m walkway (a 4.3-m GaitRite was placed in the middle of the walkway) at three different speeds: slow, preferred, and fast speeds. SmartGait and GaitRite data were collected concurrently. Since the number of
Fig. 3. (a) Example of the on-board gait assessment results and (b) exemplar data of six steps showing the relative distance between the two foot markers in the anteroposterior (AP) direction (red dotted line) and in the medio-lateral (ML) direction (blue dotted line). Maxima and minima in the AP direction are used to calculate step length while the ML distance at the same time is the step width. Background color indicates gait phases: blue is the right foot swing, yellow is the left foot swing, and magenta is the double support phase. Gray shaded regions indicate time periods when the view of the foot marker is obstructed by the thigh. Note that all parameters are calculated during double support phase, so the obstructed periods do not affect the gait measurements. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
steps captured on the GaitRite (about 4–7 steps) is dependent on the subjects’ step length, and we wanted at least 40 steps per speed condition per subjects, at least 10 passes on the walkway were collected from each participant at each speed. Both SmartGait and GaitRite were reset after each trial. Note that all steps on the 8-m walkway were captured with SmartGait, only those steps that were collected concurrently with GaitRite and SmartGait are reported here. Therefore, steps during steady state gait were compared; the initiation and termination steps were not included. 2.6. Data analysis SL, SW, ST, speed, DS and their variability were compared between the SmartGait and GaitRite. The same step was identified in each data collection system in order to directly compare the measures within each step and each speed. An average of 155 28 steps were collected per participant. Less than one percent (0.1%) of data was discarded due to missing steps. In GaitRite, this was
5
because the participant stepped off of the sensing area, while in the SmartGait it was due to the thigh obstruction. Agreement between the systems was assessed for (1) absolute error, (2) absolute error expressed as a percent, and (3) intra-class correlation coefficients (ICCs 2,1) (ICC thresholds were set as poor: 0.75 [18,24]). The data were also examined to determine how many steps were needed to minimize the error between SmartGait and the criterion system. The SW difference of each trial was calculated as a function of the number of steps included in the average to minimize the random error. The resulting plot was visually examined to determine when the error did not decrease further, indicting the minimum number of steps needed to minimize the error. 3. Results 3.1. On-board gait analysis and comparison As described above, data from the on-board gait analysis was not corrected for trunk motion, dynamic calibration, or lens distortion. However, it was important to
Table 1 Summary of the SmartGait results and its concurrent validity compared to the GaitRite: (a) on-board and (b) off-board. (a) On-board Slow speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s)
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Abs. var. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
47.9 12.4 734.4 0.69
53.1 13.8 777.3 0.72
5.2 1.4 42.9 0.03
9.8 10.1 5.5 4.2
7.9 4.3 170.3 0.22
7.5 4.0 157.3 0.20
0.4 0.2 13.0 0.02
5.5 5.4 8.3 10.0
0.832 0.822 0.914 0.925
0.756 0.761 0.799 0.781
Preferred speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s)
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Abs. var. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
58.2 13 572 1.06
64.6 12.9 601.9 1.1
6.4 0.1 29.9 0.04
9.9 0.8 5.0 3.6
9.8 4.5 89.7 0.29
9.9 3.8 80.3 0.26
0.1 0.7 9.4 0.03
0.6 18.8 11.6 11.5
0.859 0.848 0.831 0.878
0.834 0.713 0.767 0.725
Fast speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s)
SmartGait
GaitRite
Abs. error
%error
Var.
66.4 13.3 501.5 1.36
76 12.5 519.4 1.5
9.6 0.8 17.9 0.14
12.6 6.4 3.4 9.3
10.8 4.5 70.5 0.33
SmartGait
Var.
GaitRite
11.7 3.5 66.2 0.35
Abs. var. error
Var. %error
ICC
0.9 1.1 4.3 0.02
8.0 30.9 6.5 5.7
0.731 0.834 0.851 0.873
SG-GR
Var. ICC
SG-GR
0.638 0.777 0.773 0.766
(b) Off-board Slow speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s) Double support time (ms)
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Var. abs. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
52.8 13.7 734.4 0.76 544.6
53.1 13.8 777.3 0.72 558.9
0.3 0.1 42.9 0.04 14.3
0.6 0.7 5.5 5.6 2.6
7.9 3.9 163.4 0.22 202.5
7.5 4.0 157.3 0.20 190.6
0.4 0.2 6.1 0.02 11.9
5.1 3.7 3.9 10.0 6.2
0.961 0.955 0.956 0.968 0.982
0.897 0.778 0.767 0.699 0.782
Preferred speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s) Double support time (ms)
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Var. abs. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
65 13.2 573.2 1.17 358.8
64.6 12.9 601.9 1.10 372.4
0.4 0.3 28.7 0.07 13.6
0.6 2.3 4.8 6.4 3.7
9.7 3.7 74.9 0.28 115.4
9.9 3.8 76.2 0.26 98.5
0.2 0.1 1.3 0.02 16.9
2.1 2.9 1.7 7.7 17.1
0.967 0.950 0.892 0.951 0.925
0.920 0.758 0.894 0.860 0.824
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Var. abs. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
75.7 12.7 508.5 1.52 235.7
76.0 12.5 519.4 1.50 274.9
0.3 0.2 10.9 0.05 39.2
0.4 1.6 2.1 3.3 14.3
12.6 3.3 56.5 0.35 95.0
11.7 3.5 56.5 0.34 70.3
0.9 0.2 0.0 0.01 24.7
7.5 4.5 0.0 2.9 35.1
0.942 0.929 0.874 0.934 0.807
0.710 0.795 0.773 0.776 0.745
Fast speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s) Double support time (ms)
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
6
determine if the on-board measures were adequate for real time feedback. The absolute difference in SL calculated from the SmartGait and the GaitRite ranged from 5.2 to 9.6 cm and SW were 0.1–1.4 cm (Table 1a), with the SL in the fastest speed demonstrating the greatest absolute error. The absolute error of step time ranged 17.9–42.9 ms. The SmartGait underestimated the gait speed by 0.03– 0.14 m/s, with the underestimation increasing with increasing gait speed (Table 1b). The ICCs indicated excellent concurrent validity for all assessments at all speeds except for SL at the fast speed, which demonstrated modest concurrent validity. The absolute error of variability ranged from 0.1 to 1.1 cm for SL and SW, 4.3 ms to 13.0 ms for ST, and 0.02–0.03 m/s for gait speed. The agreement of variability was modest as the ICC range was 0.638–0.834. 3.2. Off-board post-processed gait analysis and comparison After the image was corrected for trunk motion, dynamic calibration, and lens distortion in the post-processing phase, improvements were observed in almost all measures. The underestimations observed in the on-board processing were no longer present after post-processing. The range of ICC values was 0.731–0.925 and 0.831–0.967, for before and after post-processing, respectively (Table 1b). Greatest improvements in ICC were observed for SL and SW. Absolute error was reduced by about half for SL and SW, with no improvement in ST. The ICCs indicated excellent concurrent validity for all assessments at all speeds. The absolute error of variability was 0.2–0.9 cm for SL, 0.1–0.2 cm for SW, 0–6.1 ms for ST, 0.01–0.02 m/s for gait speed and 11.9–24.7 ms for DS. The variability also showed modest agreement as ICCs of 0.699–0.920. The SW difference as a function of the number of steps included in the average (Fig. 4) indicated that to reduce the error to 0.5 cm or less, at least 25–30 steps are required. Since the error decreased with the number of steps, it is apparent that the error is random.
4. Discussion Assessing gait with a wearable camera provides benefits over inertial-based systems while simultaneously creating its own data processing challenges. At all speeds, the off-board assessment demonstrates excellent concurrent validity with a pressuresensing walkway, a commonly used tool in clinics and in laboratories. On-board assessment also shows very good agreement for all parameters at all speeds, except for the SL at fast walking speed. However, the average absolute error for on-board assessment is higher: less than or equal to 12.6% of the standard. The benefits and limitations of SmartGait assessment in both modes are discussed below. The excellent concurrent validity of the off-board processing indicates that SmartGait has strong potential to provide assessments of gait in almost any environment, from a traditional laboratory to community mobility. The low average absolute error scores (Table 1b) indicate that average calculations based on multiple steps are reasonably accurate and it is relatively easy to
capture and analyze a large number of steps. A small number of steps, 25–30 steps, are adequate for
GAIPOS-4480; No. of Pages 7 Gait & Posture xxx (2015) xxx–xxx
Contents lists available at ScienceDirect
Gait & Posture journal homepage: www.elsevier.com/locate/gaitpost
A wearable smartphone-enabled camera-based system for gait assessment Albert Kim a,b,1, Junyoung Kim a,b,2, Shirley Rietdyk c,d,3, Babak Ziaie a,b,* a
School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA Birck Nanotechnology Center, Purdue University, West Lafayette, IN, USA c Department of Health and Kinesiology, Purdue University, West Lafayette, IN, USA d Center for Aging and the Life Course, Purdue University, West Lafayette, IN, USA b
A R T I C L E I N F O
A B S T R A C T
Article history: Received 27 January 2015 Received in revised form 1 May 2015 Accepted 4 May 2015
Quantitative assessment of gait parameters provides valuable diagnostic and prognostic information. However, most gait analysis systems are bulky, expensive, and designed to be used indoors or in laboratory settings. Recently, wearable systems have attracted considerable attention due to their lower cost and portability. In this paper, we present a simple wearable smartphone-enabled camera-based system (SmartGait) for measurement of spatiotemporal gait parameters. We assess the concurrent validity of SmartGait as compared to a commercially available pressure-sensing walkway (GaitRite1). Fifteen healthy young adults (25.8 2.6 years) were instructed to walk at slow, preferred, and fast speed. The measures of step length (SL), step width (SW), step time (ST), gait speed, double support time (DS) and their variability were assessed for agreement between the two systems; absolute error and intra-class correlation coefficients (ICC) were determined. Measured gait parameters had modest to excellent agreements (ICCs between 0.731 and 0.982). Overall, SmartGait provides many advantages and is a strong alternative wearable system for laboratory and community-based gait assessment. ß 2015 Elsevier B.V. All rights reserved.
Keywords: Gait assessment Wearable SmartGait Variability Step width
1. Introduction Assessment of spatiotemporal gait patterns provides essential information regarding functional ability, stability, fall risk, selection of therapeutic intervention, assessment of patient progress, and mortality [1–6]. In fact, gait speed has been described as a ‘vital sign’ due to the important information it provides regarding current and predicted health status [2]. New gait assessment technologies are continually emerging, but currently available devices and systems have a number of limitations. The most accurate systems that provide a full set of gait measures are laboratory-based system which are usually expensive, and require trained personnel. For example, the Optotrak Certus (NDI, Canada)
* Corresponding author at: 2905 Browning Street, West Lafayette, IN 47906, USA. Tel.: +1 765 409 0726. E-mail addresses: [email protected] (A. Kim), [email protected] (J. Kim), [email protected] (S. Rietdyk), [email protected] (B. Ziaie). 1 Address: 1205W. State St., West Lafayette, IN 47907, USA. Tel.: +1 678 316 7733. 2 Address: 2334 South Beck Lane #307, Lafayette, IN 47909, USA. Tel.: +1 469 426 5433. 3 Address: 708 LaGrange Street, West Lafayette, IN 47906, USA. Tel.: +1 765 409 2627.
has accuracy of up to 0.1 mm and resolution of 0.01 mm [7], but the system is expensive and requires training. These systems by their nature lack field assessment capability, thus cannot provide a window into typical gait behavior when a person is completing everyday tasks including adaptations to multiple obstacles, such as stairs, ramps, curbs, and gravel [8]. In order to understand gait behavior in challenging environments, such obstacles have been replicated in the lab. While a dozen or so steps are typically captured and quantified in these replicated environments, hundreds of steps are recommended to reliably assess step width and gait speed variability [9,10]. Wearable devices based on inertial sensors can be adapted for field assessment (e.g. Physilog, GaitUp, Switzerland). Such systems extrapolate the gait measurement from the inertial sensors. Wearable devices provide a limited set of gait measures, most commonly number of steps, speed, step length (SL), and/or step time (ST) [11–14]. In addition, measurement of SL using inertial sensors requires assessment of displacement, which cannot be measured directly with inertial sensors. The acceleration signal must be integrated twice to determine displacement, but the result is confounded by the unknown constants that result from integration. These inertial-based devices require steady-state gait (i.e., excluding initiation and termination steps) and cannot
http://dx.doi.org/10.1016/j.gaitpost.2015.05.001 0966-6362/ß 2015 Elsevier B.V. All rights reserved.
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
2
measure step width (SW) and its variability, which are key parameters in assessing stability and fall risk [1,3,15,16]. Portable video recording enables a more accurate assessment technique. For example, iWalker is a system that utilizes a pair of cameras mounted on the rollator frame to capture step width [17]. This system, however, is limited to the people who use rollators and cannot be adopted for wider population. In this paper, we describe a new wearable device based on video recording from a smartphone camera mounted on the waist of an ambulatory user that provides the assessment of the measures of other wearable devices, but also SW and SW variability. After outlining the hardware and software modules of our system (SmartGait), we determine its concurrent validity with a pressure-sensing walkway (GaitRite, CIR Systems, New Jersey, USA) [16,18,19]. GaitRite was chosen as a comparison device due to its validity [18,20] and widespread use as a clinical and research tool. 2. Materials and methods 2.1. Hardware SmartGait hardware consists of a smartphone (Apple, iPhone 5s), a custom-designed waist belt with a holster, a detachable 90degree wide-angle lens (Mondizen Inc., Hilo), and two colored circular foot markers (attached on the foot dorsum and centered over the proximal phalanges) (Fig. 1). SmartGait utilizes the embedded camera of the smartphone to enable real-time assessment of gait parameters with 60 Hz sampling rate. To capture foot markers on shoes while the smartphone is in a vertical position, the viewing angle of the camera is converted from front-facing to floor-facing by using a 90-degree lens. The smartphone with the lens affixed is placed in the holster case and attached to the front of a belt. The belt has a reinforced front region to support the weight of the smartphone (iPhone5s weight is 112 g) and a 5 cm adjustable rod between the holster and the belt to reduce camera obstruction by the thighs and
clothes while walking. The smartphone angle can be adjusted on the rod to optimize video capture of the feet markers. The circular foot markers can be any color that is highly visible and can be distinguished from the surroundings (e.g., bright green markers, d = 4.6 cm). 2.2. Image processing application software in smartphone A custom software application (Xcode 5.0.2, Apple) has been developed with the target platform of iOS8 (Apple). The software algorithm has three phases: (1) the image-processing phase (Fig. 2a, left column), (2) the on-board gait assessment phase (Fig. 2a, middle column); and (3) the off-board optional postprocessing gait analysis phase (Fig. 2a, right column). The image-processing phase begins with raw image capture in RGB (Red, Green, Blue) format (Fig. 2b) of the feet and markers. This is converted to HSV (Hue, Saturation, Value) format for increasing the color of foot marker contrast with respect to the environment, resulting in a threshold image (Fig. 2c and d). In the final step, minimum circle contours are drawn over foot markers in order to identify the Cartesian coordinates of the right and left foot in pixel units (Fig. 2e). 2.3. On-board gait analysis On-board assessment provides measurement of SL, SW, ST, and speed and begins with a standing calibration: the smartphone is mounted on the belt, and the participant stands still with the feet together. The angular position of the smartphone is adjusted such that both foot markers are visible and aligned on a reference line on the smartphone screen (Fig. 2b). The calibration factor (pixels to cm) is determined for the standing position. Note that the calibration factor will change as the distance between the camera and the foot changes during the gait; this is included in the offboard processing (termed dynamic calibration of unit distance). On-board processing, however, relies on this single standing
Fig. 1. Photograph of: (a) SmartGait system and its various hardware components and (b) a participant wearing the SmartGait.
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
3
Fig. 2. (a) Application software consists of image processing, on-board gait assessments, and off-board post-processing for increased accuracy, (b) RGB image, (c) HSV image, (d) threshold image, and (e) screenshot after image processing (RGB: Red, Green, and Blue; HSV: Hue, Saturation, and Value; ST: step time; DS: double support; TM: trunk motion; FPS: frames per second; SL: step length; SW: step width).
calibration. Once the standing calibration is completed, gait assessment is initiated. The on-board software is designed to avoid computational demand on the smartphone’s CPU and is only used for data
collection and providing a raw assessment of SL, SW, ST, and speed. The trunk motion and marker size are also recorded and used in the off-board analyses in order to increase the accuracy. The trunk angular motion (assessed with the inertial sensors embedded in
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
4
the smart phone) seen in the coronal plane has the largest effect on image stability since the geometry of the viewing angle changes by the degree of trunk motion [19,20]. Foot marker size at each frame is also recorded for dynamic calibration in the off-board postprocessing (see Section 2.4). On-board assessment provides a brief on-screen summary for the users after they have stopped walking (see example in Fig. 3a). These include SL and SW for the first 7–8 steps and a report summary of the walking trial, i.e., the number of steps, average speed, average SL, average SW, total distance, and the total walking time. 2.4. Off-board gait analysis In the off-board post-processing phase, the on-board data from the SmartGait is transferred to a personal computer for further analysis. The accuracy depends primarily on the unit distance, i.e., separation between the camera and foot marker as assessed in the calibration. This, however, varies as both the foot and the trunk (i.e., camera) move during gait [22]. In on-board assessment, a single unit distance is calculated in the initial standing calibration to reduce the computational burden. However, in post-processing, dynamic calibration of unit distance is completed by re-calibrating marker size at each frame. Accuracy can also be improved by correcting distortion caused by the wide-angle lens as the foot marker usually moves from the bottom edge to the top edge of the screen, where the distortion is greatest. This is corrected using the polynomial model with predetermined coefficients. To complete the analyses needed to increase the off-board accuracy, a marker based on dynamic calibration is drawn on a blank image. The intrinsic matrix and distortion coefficients are applied to remove lens distortion; the
image is then rotated based on the measured trunk roll angle, followed by a filtering step using and Savitzky–Golay algorithm [23]. The time series of the relative difference between the two footmarkers in the anterior-posterior direction shows a sinusoidal pattern (Fig. 3b, red line), with maxima and minima during double support (DS) phase (Fig. 3b, magenta regions), when the feet are farthest apart. The thigh of the swinging limb obstructs the camera view of the foot marker during most of the swing phase (Fig. 3b, gray shaded regions), but this time period is not needed to calculate SL, SW, ST, speed, and DS. Both markers are visible during DS phase and for a short time preceding and following. The absolute value of the maxima (or minima) of each step in the anterior-posterior direction is quantified as SL. SW is quantified as the medial-lateral difference between the foot markers at the same point in time (Fig. 3b, blue line). ST is the temporal difference between one maximum SL and the following minimum SL. Abrupt changes in marker size occurs at heel contact and toe off, which are used to calculate DS duration. Gait speed is calculated for each step as SL/ST. 2.5. Validation protocol Fifteen young healthy adults participated in the study (5 female; age: 25.8 (2.6) years; height: 171.1 (8.0) m; mass: 70.1 (15.6) kg; BMI: 23.8 (4.2)). The study was approved by the Human Research Protection Program at Purdue University and all participants signed informed consent. Gait was assessed simultaneously with two systems: a pressure-sensing walkway (GaitRite) and SmartGait. Participants were instructed to walk along an 8 m walkway (a 4.3-m GaitRite was placed in the middle of the walkway) at three different speeds: slow, preferred, and fast speeds. SmartGait and GaitRite data were collected concurrently. Since the number of
Fig. 3. (a) Example of the on-board gait assessment results and (b) exemplar data of six steps showing the relative distance between the two foot markers in the anteroposterior (AP) direction (red dotted line) and in the medio-lateral (ML) direction (blue dotted line). Maxima and minima in the AP direction are used to calculate step length while the ML distance at the same time is the step width. Background color indicates gait phases: blue is the right foot swing, yellow is the left foot swing, and magenta is the double support phase. Gray shaded regions indicate time periods when the view of the foot marker is obstructed by the thigh. Note that all parameters are calculated during double support phase, so the obstructed periods do not affect the gait measurements. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
steps captured on the GaitRite (about 4–7 steps) is dependent on the subjects’ step length, and we wanted at least 40 steps per speed condition per subjects, at least 10 passes on the walkway were collected from each participant at each speed. Both SmartGait and GaitRite were reset after each trial. Note that all steps on the 8-m walkway were captured with SmartGait, only those steps that were collected concurrently with GaitRite and SmartGait are reported here. Therefore, steps during steady state gait were compared; the initiation and termination steps were not included. 2.6. Data analysis SL, SW, ST, speed, DS and their variability were compared between the SmartGait and GaitRite. The same step was identified in each data collection system in order to directly compare the measures within each step and each speed. An average of 155 28 steps were collected per participant. Less than one percent (0.1%) of data was discarded due to missing steps. In GaitRite, this was
5
because the participant stepped off of the sensing area, while in the SmartGait it was due to the thigh obstruction. Agreement between the systems was assessed for (1) absolute error, (2) absolute error expressed as a percent, and (3) intra-class correlation coefficients (ICCs 2,1) (ICC thresholds were set as poor: 0.75 [18,24]). The data were also examined to determine how many steps were needed to minimize the error between SmartGait and the criterion system. The SW difference of each trial was calculated as a function of the number of steps included in the average to minimize the random error. The resulting plot was visually examined to determine when the error did not decrease further, indicting the minimum number of steps needed to minimize the error. 3. Results 3.1. On-board gait analysis and comparison As described above, data from the on-board gait analysis was not corrected for trunk motion, dynamic calibration, or lens distortion. However, it was important to
Table 1 Summary of the SmartGait results and its concurrent validity compared to the GaitRite: (a) on-board and (b) off-board. (a) On-board Slow speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s)
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Abs. var. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
47.9 12.4 734.4 0.69
53.1 13.8 777.3 0.72
5.2 1.4 42.9 0.03
9.8 10.1 5.5 4.2
7.9 4.3 170.3 0.22
7.5 4.0 157.3 0.20
0.4 0.2 13.0 0.02
5.5 5.4 8.3 10.0
0.832 0.822 0.914 0.925
0.756 0.761 0.799 0.781
Preferred speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s)
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Abs. var. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
58.2 13 572 1.06
64.6 12.9 601.9 1.1
6.4 0.1 29.9 0.04
9.9 0.8 5.0 3.6
9.8 4.5 89.7 0.29
9.9 3.8 80.3 0.26
0.1 0.7 9.4 0.03
0.6 18.8 11.6 11.5
0.859 0.848 0.831 0.878
0.834 0.713 0.767 0.725
Fast speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s)
SmartGait
GaitRite
Abs. error
%error
Var.
66.4 13.3 501.5 1.36
76 12.5 519.4 1.5
9.6 0.8 17.9 0.14
12.6 6.4 3.4 9.3
10.8 4.5 70.5 0.33
SmartGait
Var.
GaitRite
11.7 3.5 66.2 0.35
Abs. var. error
Var. %error
ICC
0.9 1.1 4.3 0.02
8.0 30.9 6.5 5.7
0.731 0.834 0.851 0.873
SG-GR
Var. ICC
SG-GR
0.638 0.777 0.773 0.766
(b) Off-board Slow speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s) Double support time (ms)
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Var. abs. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
52.8 13.7 734.4 0.76 544.6
53.1 13.8 777.3 0.72 558.9
0.3 0.1 42.9 0.04 14.3
0.6 0.7 5.5 5.6 2.6
7.9 3.9 163.4 0.22 202.5
7.5 4.0 157.3 0.20 190.6
0.4 0.2 6.1 0.02 11.9
5.1 3.7 3.9 10.0 6.2
0.961 0.955 0.956 0.968 0.982
0.897 0.778 0.767 0.699 0.782
Preferred speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s) Double support time (ms)
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Var. abs. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
65 13.2 573.2 1.17 358.8
64.6 12.9 601.9 1.10 372.4
0.4 0.3 28.7 0.07 13.6
0.6 2.3 4.8 6.4 3.7
9.7 3.7 74.9 0.28 115.4
9.9 3.8 76.2 0.26 98.5
0.2 0.1 1.3 0.02 16.9
2.1 2.9 1.7 7.7 17.1
0.967 0.950 0.892 0.951 0.925
0.920 0.758 0.894 0.860 0.824
SmartGait
GaitRite
Abs. error
%error
Var.SmartGait
Var.GaitRite
Var. abs. error
Var. %error
ICCSG-GR
Var. ICCSG-GR
75.7 12.7 508.5 1.52 235.7
76.0 12.5 519.4 1.50 274.9
0.3 0.2 10.9 0.05 39.2
0.4 1.6 2.1 3.3 14.3
12.6 3.3 56.5 0.35 95.0
11.7 3.5 56.5 0.34 70.3
0.9 0.2 0.0 0.01 24.7
7.5 4.5 0.0 2.9 35.1
0.942 0.929 0.874 0.934 0.807
0.710 0.795 0.773 0.776 0.745
Fast speed
Step length (cm) Step width (cm) Step time (ms) Gait speed (m/s) Double support time (ms)
Please cite this article in press as: Kim A, et al. A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture (2015), http://dx.doi.org/10.1016/j.gaitpost.2015.05.001
G Model
GAIPOS-4480; No. of Pages 7 A. Kim et al. / Gait & Posture xxx (2015) xxx–xxx
6
determine if the on-board measures were adequate for real time feedback. The absolute difference in SL calculated from the SmartGait and the GaitRite ranged from 5.2 to 9.6 cm and SW were 0.1–1.4 cm (Table 1a), with the SL in the fastest speed demonstrating the greatest absolute error. The absolute error of step time ranged 17.9–42.9 ms. The SmartGait underestimated the gait speed by 0.03– 0.14 m/s, with the underestimation increasing with increasing gait speed (Table 1b). The ICCs indicated excellent concurrent validity for all assessments at all speeds except for SL at the fast speed, which demonstrated modest concurrent validity. The absolute error of variability ranged from 0.1 to 1.1 cm for SL and SW, 4.3 ms to 13.0 ms for ST, and 0.02–0.03 m/s for gait speed. The agreement of variability was modest as the ICC range was 0.638–0.834. 3.2. Off-board post-processed gait analysis and comparison After the image was corrected for trunk motion, dynamic calibration, and lens distortion in the post-processing phase, improvements were observed in almost all measures. The underestimations observed in the on-board processing were no longer present after post-processing. The range of ICC values was 0.731–0.925 and 0.831–0.967, for before and after post-processing, respectively (Table 1b). Greatest improvements in ICC were observed for SL and SW. Absolute error was reduced by about half for SL and SW, with no improvement in ST. The ICCs indicated excellent concurrent validity for all assessments at all speeds. The absolute error of variability was 0.2–0.9 cm for SL, 0.1–0.2 cm for SW, 0–6.1 ms for ST, 0.01–0.02 m/s for gait speed and 11.9–24.7 ms for DS. The variability also showed modest agreement as ICCs of 0.699–0.920. The SW difference as a function of the number of steps included in the average (Fig. 4) indicated that to reduce the error to 0.5 cm or less, at least 25–30 steps are required. Since the error decreased with the number of steps, it is apparent that the error is random.
4. Discussion Assessing gait with a wearable camera provides benefits over inertial-based systems while simultaneously creating its own data processing challenges. At all speeds, the off-board assessment demonstrates excellent concurrent validity with a pressuresensing walkway, a commonly used tool in clinics and in laboratories. On-board assessment also shows very good agreement for all parameters at all speeds, except for the SL at fast walking speed. However, the average absolute error for on-board assessment is higher: less than or equal to 12.6% of the standard. The benefits and limitations of SmartGait assessment in both modes are discussed below. The excellent concurrent validity of the off-board processing indicates that SmartGait has strong potential to provide assessments of gait in almost any environment, from a traditional laboratory to community mobility. The low average absolute error scores (Table 1b) indicate that average calculations based on multiple steps are reasonably accurate and it is relatively easy to
capture and analyze a large number of steps. A small number of steps, 25–30 steps, are adequate for
