HHS Public Access Author manuscript Author Manuscript
J Biomed Inform. Author manuscript; available in PMC 2016 August 18. Published in final edited form as: J Biomed Inform. 2015 December ; 58(Suppl): S197–S202. doi:10.1016/j.jbi.2015.06.024.
Predicting Changes in Systolic Blood Pressure Using Longitudinal Patient Records John Wes Solomon and Rodney D. Nielsen, (PhD) University of North Texas, Denton, TX John Wes Solomon: [email protected]; Rodney D. Nielsen: [email protected]
Author Manuscript
Abstract Objective—This paper introduces a model that predicts future changes in systolic blood pressure (SBP) based on structured and unstructured (text-based) information from longitudinal clinical records. Method—For each patient, the clinical records are sorted in chronological order and SBP measurements are extracted from them. The model predicts future changes in SBP based on the preceding clinical notes. This is accomplished using least median squares regression on salient features found using a feature selection algorithm. Results—Using the prediction model, a correlation coefficient of 0.47 is achieved on unseen test data (p < .0001). This is in contrast to a baseline correlation coefficient of 0.39.
Author Manuscript
Graphical Abstract
Author Manuscript
The graph on the left represents the actual change in a patient’s SBP on the horizontal axis, and the change in SBP predicted by the model on the vertical axis. The diagonal line goes through all points in the graph where “Actual SBP Change” is equal to “Predicted SBP Change”. As such, the closer a plotted point is to the diagonal line, the more accurate the prediction.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Solomon and Nielsen
Page 2
Author Manuscript
1.) Introduction
Author Manuscript
The primary goal of this research is to demonstrate that the output of an NLP system capable of automatically producing annotations like the ones provided by the organizers of the i2b2/ UTHealth 2014 challenge [1] can be used to predict future medical events. Systems with the ability to predict future medical events based on electronic medical records (EMRs) could be used to suggest that patients and medical professionals pay closer attention to certain risk factors and events associated with the condition(s) the system is designed to predict. In this paper, a model that predicts future changes in systolic blood pressure (SBP) using time sensitive information is presented and evaluated. Such a system could help doctors recognize when to add or change medications and let patients know they should change their eating or exercise habits, among other possibilities. The presented model is novel in the sense that it is the first to predict changes in blood pressure using a feature space that explicitly takes into account the amount of time between the observation of feature values and the prediction date. Similar models that use different features, but similar feature representations may be effective for predicting other medical outcomes given appropriate training data.
2.) Related Work
Author Manuscript
A 1982 paper by Sparrow, Garvey, Rosner, and Thomas [2] presents a model for predicting changes in blood pressure (BP). Although this work does not fall into the realm of clinical NLP, it involves the prediction of changes in BP on a patient by patient basis. The features used to inform the prediction model presented by Sparrow and colleagues could, in theory, be extracted from the electronic medical record. The prediction task presented in the paper is as follows. Given two BP measurements where the second measurement is taken after the first, and patient data collected on the day of the first BP measurement, predict the slope of the trend-line formed by the first two BP measurements and a third unseen BP measurement taken after the second one. The amount of time between observations is not taken into account when setting feature values and the model is not designed to make predictions when feature values are not known. A 2012 paper by Fava et al. [13] investigates the predictive power of a genetic risk score (GRS) derived using 29 independent single nucleotide polymorphisms for predicting changes in blood pressure between an initial measurement and a follow-up examination when compared to other predictive variables including demographic, anthropometric, socioeconomic, and lifestyle data. The average time between the initial and follow-up examination was 23 years. Like the Sparrow paper, the amount of time between the initial and follow-up examination is not taken into account.
Author Manuscript
One significant way the present work differs from Sparrow's and Fava’s is that the proposed model accounts for the amount of time between measurements and mentions of medical concepts. First, the model for predicting SBP changes accounts for the amount of time between measurements by using a logarithmic weighting scheme that gives more weight to recent numeric measurements. The intuition behind this is that more recent measurement values will be closer to the current value than less recent ones. The model also utilizes binary features tied to the relative time of a variable observation with respect to the
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 3
Author Manuscript
prediction date. Finally, the model is capable of making predictions without all feature values being known. These are not the only examples of work outside the realm of clinical NLP that predicts medical events on a patient by patient basis. Other notable examples include models that predict morbidity following gastrectomy [3] and respiratory morbidity among those with low birth weight [4]. Like the Sparrow paper, time isn’t taken into account when setting the feature values for these models and all feature values are assumed to be known. An example of work within the realm of clinical NLP where future medical events are predicted on a patient by patient basis is a paper by Bihorac and colleagues [5]. The paper presents a model that predicts patient mortality after surgery.
Author Manuscript
Numerous clinical NLP papers refer to their work as prediction (e.g Himes et al. [6], and Chen et al. [7]), but the term is being used in the sense of classification. In these papers, the classifier’s predictions are determined based on features or measurements from the same time to which the prediction pertains.
3.) Prediction Task
Author Manuscript
The goal of the present work is to provide a patient and their physician a prediction of the patient’s future SBP measurement in order to help ensure appropriate proactive health care. The formally defined prediction task is as follows. Given a list of patient records R = sorted in chronological order according to the patient’s visit date, the task is to predict the change in SBP measurement from rk to rn, where rn is the chronologically latest patient record that includes an SBP measurement (and is thus best suited to act as a future date) and rk is the record nearest in time to rn that also includes an SBP measurement, while only utilizing information present in the records that predated rn. Since utilizing any of the information from visits after rn (i.e., {rn+1, rn+2, …}) would obviously violate the goal of predicting a future outcome, these records are excluded from all further analysis or discussion. di denotes the visit date associated with ri,
4.) Data
Author Manuscript
The i2b2/UTHealth 2014 dataset for track 2 (identifying risk factors for heart disease over time) [14] is used to train and evaluate the system. The training portion of the data contains 790 individual records from 175 patients. The test portion consists of 542 individual records for 118 patients. Every patient in the dataset has been diagnosed with diabetes. After eliminating all patients from consideration where less than three records contain at least one SBP measurement, 514 records for 138 patients remain in the training data, and 309 records for 82 patients remain in the test data. Each record in the i2b2/UTHealth 2014 dataset has annotations associated with it. The annotations are used to set the values of the features that inform the prediction model. The system utilizes annotations that denote mentions of medical conditions, medications, and tobacco use.
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 4
Author Manuscript
5.) Extracting Systolic Blood Pressure Measurements and other Numeric Values using Regular Expressions The system uses regular expressions to extract SBP measurements as well as the values of measurements used to set all features described in section 5.3 and some of the features described in section 5.4. A regular expression encodes text patterns in strings. All substrings in a document that match the text pattern defined in a regular expression can be easily extracted. For example, the pattern string the system uses to extract weight measurements is “\bweight\s*(\s+|:)\s*\d+”. This pattern extracts all instances of the word “weight” preceded by a word boundary on the left side followed by whitespace with an optional single colon in the middle and a numeric string containing one or more characters. This regular expression extracts the substrings, “weight 100” and “weight: 210” but does not extract “weight is 100”.
Author Manuscript Author Manuscript Author Manuscript
BP measurement candidates are identified by extracting all substrings that match the regular expression "\d+/\d+" (a numeric string followed by a forward slash and another numeric string). The system uses a heuristic to filter out implausible BP values. Among the candidate substrings, if the numerator is between 40 and 300 and the denominator is between 30 and 200, the numerator of the fraction is identified by the system as an SBP measurement. Because a visit sometimes includes multiple SBP readings, we use the average of all SBP measurements in the patient record, AVGSBP(r), to approximate the true value of SBP. Regular expressions have the potential to extract incorrect values and miss values. When applied to training data and used to inform features used to predict actual changes in SBP, values extracted by regular expressions represent noisy, but potentially informative features. When applied to the BP measurements used to determine the changes in SBP upon which the performance evaluation is based, this presents a methodological problem. As such, we conducted two reviews to determine the accuracy of SBPs extracted using the aforementioned regular expression and heuristic. The first review involved annotating 65 randomly selected fraction substrings in the training data as being a BP measurement or not. Among the 65 fraction substrings 44 represented BP measurements and 21 did not. No errors were found in this review. Because our aim is to predict the change in SBP from the second most recent note containing an SBP measurement to the most recent note containing an SBP measurement it is critical that we ensure the SBP values in these notes are correct for the test set. As such, we manually reviewed these SBP measurements and found five cases where the values extracted by the regular expression and filtering heuristic did not represent SBP measurements. We manually removed these measurements from the test set. Occasionally, an SBP measurement in a note was from another point in time. From a methodological standpoint, this is problematic if it happens in the last or second to last note containing an SBP measurement in the test data, as the average SBP in those two notes are used to determine the gold standard SBP change. Fortunately, this occurs infrequently, and in cases where it does occur, the measurements were taken a short time prior to the visit.
5.) Features Each feature encodes information about a single patient. Most of the features fall into one of two categories: “Binary Time Sensitive Features” and “Real-Valued Time Adjusted
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 5
Author Manuscript
Features”. Features that do not fall into one of these two categories will be discussed on an individual basis. Gold standard annotations for medical events provided by the organizers of the i2b2/UTHealth 2014 challenge are used to directly set the values for some of the features that inform the prediction system. Using the gold standard annotations for the i2b2/ UTHealth 2014 dataset in this fashion is reasonable since the aim is to demonstrate that future medical events can be predicted for a patient assuming that relevant information can be extracted from longitudinal records. The extraction of information itself is a different problem than using information to predict events. 5.1) Binary Time Sensitive Features
Author Manuscript
Binary time sensitive features are used to represent one of two things; either that an event of type c was mentioned in one or more of the patient’s records within m months preceding dn, or that the earliest mention of an event of type c in the patient’s records occurred within m months preceding dn. These representations will be referred to as repRecent and repEarliest respectively. A binary time sensitive feature is denoted by the Boolean variable, , where c denotes the type of event and m denotes a span of months within which the most recent annotation (using repRecent) or earliest annotation (using repEarliest) of an event of type c is found. Using repRecent, the feature encodes the answer to the question “Did any mention of an event of type c occur within the last m months?” Using repEarliest, the feature encodes the answer to the question “Did the earliest mention of an event of type c occur within the last m months?” Each medical event type has ten features associated with it, five for repRecent and five for repEarliest, where m = 3, m = 6, m = 12, m = 24, and m = 36. For example, suppose that two mentions of event type c occur in a patient’s records. One occurs 20 months prior to dn and the other occurs 11 months prior to dn. Using repRecent,
Author Manuscript
, and
. Using repEarliest,
, and
.
5.2) Real-Valued Time Adjusted Features
Author Manuscript
Real-valued time adjusted features are used to represent numeric measurements for a patient that can vary over time, such as SBP, sodium, body weight, and glucose. The value of a realvalued time adjusted feature is the weighted average of each measurement of a given type for the patient in question. Measurements extracted from records closer to dn are given greater weight. When two or more measurements of the same type are detected in the same record, the average value of those measurements are used. All measurements represented by real-valued time adjusted features are extracted using regular expressions. The system does not determine if measurements in a record were taken before, on, or after the visit date. Figure 1 shows how the weight constant for the ith record is computed. The equation in figure 2 uses the weight function shown in figure 1 to compute the weighted average for all measurements of type c. The value returned by the equation in figure 2 is used to set the value of the real-valued time adjusted feature for measurement type c. In addition, each real-valued time adjusted feature has a trend-line feature associated with it. Specifically, linear regression is performed over the 〈di, AVGc(ri)〉 pairs, and the slope of the resulting trend-line is converted to one of the two nominal values: {positive, negative}.
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 6
5.3) Other Features
Author Manuscript
The set of features that do not fall into the two categories above include gender, height, BMI, approximate age on dn, and smoking status. The system identifies the patient’s Gender by counting the gender pronouns in each record associated with the patient in question. If the majority of the gender pronouns in all records for the patient are masculine, the system identifies that patient as male. Otherwise, the system identifies the patient as female.
Author Manuscript
Height values are extracted using a regular expression. All extracted values which are less than 54 or more than 84 in terms of inches are discarded. If any of the heights found in the records for a given patient vary by more than three inches (excluding the discarded heights), height is not recorded for that patient. Otherwise, the average of the values found by the regular expressions are used. The method we use to calculate BMI utilizes the time-based weighted average for the patient’s weight measurements and the patient’s height (both extracted using regular expressions) when both are mentioned in the patient’s records. This feature is represented by three nominal values representing ranges for BMI: {BMI < 25, 25 ≤ BMI < 30, 30 ≤ BMI}. We selected the thresholds based on BMI range classifications from the World Health Organization [12].
Author Manuscript
Calculating Approximate age on dk requires that the system be able to extract the patient’s age from at least one of the records for that patient. Age is extracted from records using a regular expression where all extracted values greater than 1251 are discarded. Assuming the patient’s age can be extracted from at least one record, the patient’s approximate date of birth, denoted by DOB, can be calculated as di – agei * (365.25) where agei is the age value extracted from di. DOB is accurate within one year. The approximate age of a patient on the prediction date is equal to (dn − DOB) in terms of years. This feature is represented by four nominal values denoting ranges for age: {26 – 50, 50 – 70, 71 – 80, 80+}. We did not attempt to directly extract the patient’s date of birth from the notes because it was rarely mentioned.
Author Manuscript
The system determines Smoking Status using a rule-based approach. Let S = 〈 s1, s2, · · ·sn〉 be the list of smoking status annotations for each patient record. Each si can be one of the following values: “unknown”, “current”, “past”, or “never”. Figure 4 shows the algorithm that determines smoking status for the purpose of the prediction model. This algorithm is necessary because the gold standard annotations for smoking status do not always reflect knowledge from previous records. For example, if si-m is equal to “past”, that would contradict si being equal to “never”.
1A reviewer justifiably expressed concern that “125 [years] seems to be on the high end for a cut-off”, given that “only one person in [recorded history] has ever lived to be 122”. A lower number would probably be a better choice.
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 7
5.4) Feature Selection
Author Manuscript
We selected features based on the output of a feature selection algorithm. The feature selection algorithm is implemented in the Java class “ClassifierSubsetEval” found in the Weka[8] library using a “Best First” ranking scheme with a backward search. This was done across 30 folds of the training data. Using cross validation on this training data, we discovered that including any feature not selected for more than 80% of the folds degraded performance. As such, these features were not used. See figure 5 for a list of selected features.
6.) Results
Author Manuscript
Weka’s implementation of Pearson’s correlation coefficient 2 is used to measure performance. A baseline classifier trained using only the slope direction of the trend-line for SBP yields a correlation coefficient of .39 on the test data. Using the actual slope of the trend-line to train the classifier yields a correlation coefficient of .07. We opted for the stronger baseline. Using the features extracted from the feature selection algorithm mentioned in section 5.4, the system achieves a correlation coefficient of .47 on the test data (p < .0001). In Figure 6, the model’s predictions are plotted along the graph’s vertical axis, and the actual changes in SBP from dk to dn are plotted along the graph’s horizontal axis.
7.) Discussion This section will first examine the prediction errors, then move on to examining the effectiveness of a few key features, including the presence of diabetes annotations, age, and medications.
Author Manuscript Author Manuscript
The test set contains 39 patients whose blood pressure increases and 43 patients whose blood pressure decreases. Error analysis uncovered that the model is more error prone when the absolute value of actual SBP change is high, and that the model has a tendency to overpredict SBP values. The model’s tendency to over-predict SBP may be explained by the different distribution of SBP changes in the training data vs the test data. This hypothesis is based on two things. First, the mean and median SBP change for each patient is +4.57 and +3.0 respectively for the training data and +1.8 and +0 respectively for the test data. Second, SBP increases or remains the same for 57% of the patients in the training data and 52% of patients in the test data. All patients in the i2b2/UTHealth 2014 dataset have diabetes, so a binary feature that indicates whether or not the patient has diabetes would always have the same value, and thus be uninformative. The presence of one or more mentions of diabetes within 24 months preceding dn was found to be a salient feature. This suggests the relative timeframe in which diabetes was diagnosed or mentioned is a predictor for future SBP. In contrast to the results shown in the Sparrow paper, age was not a salient feature in the present work. Using regular expressions, we determined the approximate age for 105 of the 138 patients in the training set. Given the relatively good coverage of this feature, it is unlikely that data sparseness is the reason it proved uninformative. One possible explanation
2http://weka.sourceforge.net/doc.dev/weka/attributeSelection/CorrelationAttributeEval.html
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 8
Author Manuscript
is that the age distribution for patients in the i2b2/UTHealth 2014 dataset and the Sparrow study are different. In the Sparrow study, the mean age of all patients at the time of first examination is 42 years. This means that the mean age of the patients at the time of the final examination was 52 years. In our study the mean age of the patient on dn is 67 years. Figure 7 provides a breakdown of the age distribution for the patients in the training set. It is important to note that we extracted age using regular expressions and did not manually verify the correctness of the extracted ages. As such, the patient ages we are using may not be totally correct.
Author Manuscript
Features that encode answers to the questions “How long has it been since the patient was diagnosed with a disorder or prescribed a medication?” and “How long has it been since the patient recovered from a disorder or discontinued a medication?” were not explicitly annotated in the i2b2/UTHealth 2014 dataset. Given a sufficiently large dataset where this information is available or extractable, we hypothesize features that answer these two questions would improve the performance of the model. The intuition behind this hypothesis is that the effects of disorders and medications vary with respect to the amount of time since diagnosis or prescription. The repEarliest variation of binary-time sensitive features gives a lower bound answer for the question: “How long has it been since the patient was diagnosed with a disorder or prescribed a medication?” by answering the question: “How long has it been since the first annotation of a medication or disorder?”. If the first annotation of a medication or disorder occurred at some date d, that disorder or medication must have been first diagnosed or prescribed no later than d. Although the feature selection algorithm did not find any features represented in this fashion to be salient, it’s plausible that given a larger amount of training data, a predictive model could learn from features represented in this fashion.
Author Manuscript
The prediction of medical events from notes that lack annotations using currently available NLP technology is a practical direction for future work. Existing clinical NLP tools like cTAKES [11] can be used to identify UMLS [9] and RxNorm [10] concepts in text. From these identified concepts, it is plausible that clinical prediction systems could be created or improved upon. Beyond serving as a proof of concept that predictive models can be developed by applying statistical models to the output of currently available NLP tools, such a system would not be as limited in the scope of medical concepts that could serve as potential features for the model. Thus, predictive features other than those explicitly annotated could be identified and exploited. Clinical notes are not the only source of data the presented model could exploit. A medical database containing measurements, prescriptions, and diagnoses on a patient by patient basis could also be leveraged in addition to or in place of textual data.
Author Manuscript
In conclusion, due to the time sensitive nature of the feature representation, a system that uses the presented model or a variation of it could be used to predict future medical events in a way that accounts for the amount of time that has passed between the medical events that inform the model and the prediction date. This is a particularly attractive feature in the face of real world data where the amount of time between the prediction date and the relevant observation is variable. If a likely adverse medical event can be predicted automatically, proper treatments and behavior changes are more likely to be implemented.
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 9
Author Manuscript
Acknowledgments The authors would like to thank Kathryn Farmer for her invaluable feedback on the paper.
References
Author Manuscript Author Manuscript
1. Stubbs A, Kotfila C, Hua X, Ozlem U. Practical applications for NLP in clinical research: the 2014 i2b2/UTHealth shared tasks. 2. David, Sparrow; Arthur, Garvey; Bernard, Rosner; Thomas, H. Factors in predicting blood pressure change. American Heart Association. Circulation. 1982; 65:789–94. [PubMed: 7060257] 3. Grossman EM, Longo WE, Virgo KS, et al. Morbidity and mortality of gastrectomy for cancer in department of veterans affairs medical centers. Surgery. 2002 May; 131(5):484–90. [PubMed: 12019399] 4. Teune MJ, Wassenaer AG, van Buuren S, Mol BW, Opmeer BC. Dutch POPS Collaborative Study Group. Perinatal risk-indicators for long-term respiratory morbidity among preterm or very low birth weight neonates. Eur J Obstet Gynecol Reprod Biol. 2012 Aug; 163(2):134–41. [PubMed: 22579229] 5. Bihorac A, Cobb R, Wang DZ, Korenkevych D, Baslanti TO, Puri S, Momcilovic P. 692: Computer algorithms are superior to physician assessment of the risk for postoperative complications. Critical Care Medicine. 2013; 41(12):A170. 6. Himes B, Dai Y, Kohane I, Wiess S, Ramoni M. Prediction of chronic obstructive pulmonary disease (COPD) in asthma patients using electronic medical records. Journal of the American Medical Informatics Association. 2009; 16(3):371–9. [PubMed: 19261943] 7. Chen, Lin; Elizabeth, Karlson; Helena, Canhao; Timothy, Miller. Automatic prediction of rheumatoid arthritis disease activity from the electronic medical records. PLoS ONE. 2013; 8:e69932. [PubMed: 23976944] 8. Hall M, Frank E, Holmes G, Pfahringer B, Reutermann P, Witten I. The weka data mining software: an update. ACM SIGKDD Explorations Newsletter. 2009; 11(1):10–18. 9. Lindberg DA, Humphreys BL, McCray AT. The Unified Medical Language System. Methods of information in medicine. 1993; 32(4):281–291. [PubMed: 8412823] 10. Liu S, Ma W, Moore R, Ganesan V, Nelson S. RxNorm: prescription for electronic drug information exchange. IT professional. 2005; 7(5):17–23. 11. Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, Chute CG. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informatics Association. 2010; 17(5):507–513. [PubMed: 20819853] 12. World Health Organization, & World Health Organization. Global database on body mass index. 2006 13. Fava C, Sjögren M, Montagnana M, Danese E, Almgren P, Engström G, … Melander O. Prediction of blood pressure changes over time and incidence of hypertension by a genetic risk score in Swedes. Hypertension. 2012 HYPERTENSIONAHA-112. 14. Annotating Risk Factors for Heart Disease in Clinical Narratives for Diabetic Patients Amber Stubbs and Ozlem Uzuner. 2014
Author Manuscript
Appendix (Regular Expressions) MAX – the measurement’s value must be less than our equal to this number MIN – the measurement’s value must be greater than or equal to this number. •
Age (years): “\bage\s*(\s+|:)\s*\d+” –
MAX: 125
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 10
•
Author Manuscript
•
Height (inches): “\d/’\d{1,2}” –
MIN: 54 inches
–
MAX: 74 inches
Weight (pounds): “\bweight\s*(\s+|:)\s*\d+” –
MAX: 700 lbs
•
Sodium (mmol/L): “\bsodium\s*(\s+|:)\s*\d+”
•
Glucose (mg/dl): “\bglucose\s*(\s+|:)\s*\d+”
•
Blood Pressure (mmhg): “\d+/\d+”
Author Manuscript
–
MAX systolic: 300
–
MIN systolic: 40
–
MAX diastolic: 200
–
MIN diastolic: 30
Author Manuscript Author Manuscript J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 11
Author Manuscript
Highlights •
We present a model for predicting changes in systolic blood pressure (SBP).
•
The model is trained and tested using longitudinal clinical records.
•
Some features that inform the model account for the time of observation.
•
The model’s estimates have a correlation coefficient of 0.47 with actual SBP change.
Author Manuscript Author Manuscript Author Manuscript J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 12
Author Manuscript
Figure 1.
hasMeas(r, c) returns 1 if document r contains a measurement of type c and 0 otherwise. Because documents dated later than dn are not considered (i
J Biomed Inform. Author manuscript; available in PMC 2016 August 18. Published in final edited form as: J Biomed Inform. 2015 December ; 58(Suppl): S197–S202. doi:10.1016/j.jbi.2015.06.024.
Predicting Changes in Systolic Blood Pressure Using Longitudinal Patient Records John Wes Solomon and Rodney D. Nielsen, (PhD) University of North Texas, Denton, TX John Wes Solomon: [email protected]; Rodney D. Nielsen: [email protected]
Author Manuscript
Abstract Objective—This paper introduces a model that predicts future changes in systolic blood pressure (SBP) based on structured and unstructured (text-based) information from longitudinal clinical records. Method—For each patient, the clinical records are sorted in chronological order and SBP measurements are extracted from them. The model predicts future changes in SBP based on the preceding clinical notes. This is accomplished using least median squares regression on salient features found using a feature selection algorithm. Results—Using the prediction model, a correlation coefficient of 0.47 is achieved on unseen test data (p < .0001). This is in contrast to a baseline correlation coefficient of 0.39.
Author Manuscript
Graphical Abstract
Author Manuscript
The graph on the left represents the actual change in a patient’s SBP on the horizontal axis, and the change in SBP predicted by the model on the vertical axis. The diagonal line goes through all points in the graph where “Actual SBP Change” is equal to “Predicted SBP Change”. As such, the closer a plotted point is to the diagonal line, the more accurate the prediction.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Solomon and Nielsen
Page 2
Author Manuscript
1.) Introduction
Author Manuscript
The primary goal of this research is to demonstrate that the output of an NLP system capable of automatically producing annotations like the ones provided by the organizers of the i2b2/ UTHealth 2014 challenge [1] can be used to predict future medical events. Systems with the ability to predict future medical events based on electronic medical records (EMRs) could be used to suggest that patients and medical professionals pay closer attention to certain risk factors and events associated with the condition(s) the system is designed to predict. In this paper, a model that predicts future changes in systolic blood pressure (SBP) using time sensitive information is presented and evaluated. Such a system could help doctors recognize when to add or change medications and let patients know they should change their eating or exercise habits, among other possibilities. The presented model is novel in the sense that it is the first to predict changes in blood pressure using a feature space that explicitly takes into account the amount of time between the observation of feature values and the prediction date. Similar models that use different features, but similar feature representations may be effective for predicting other medical outcomes given appropriate training data.
2.) Related Work
Author Manuscript
A 1982 paper by Sparrow, Garvey, Rosner, and Thomas [2] presents a model for predicting changes in blood pressure (BP). Although this work does not fall into the realm of clinical NLP, it involves the prediction of changes in BP on a patient by patient basis. The features used to inform the prediction model presented by Sparrow and colleagues could, in theory, be extracted from the electronic medical record. The prediction task presented in the paper is as follows. Given two BP measurements where the second measurement is taken after the first, and patient data collected on the day of the first BP measurement, predict the slope of the trend-line formed by the first two BP measurements and a third unseen BP measurement taken after the second one. The amount of time between observations is not taken into account when setting feature values and the model is not designed to make predictions when feature values are not known. A 2012 paper by Fava et al. [13] investigates the predictive power of a genetic risk score (GRS) derived using 29 independent single nucleotide polymorphisms for predicting changes in blood pressure between an initial measurement and a follow-up examination when compared to other predictive variables including demographic, anthropometric, socioeconomic, and lifestyle data. The average time between the initial and follow-up examination was 23 years. Like the Sparrow paper, the amount of time between the initial and follow-up examination is not taken into account.
Author Manuscript
One significant way the present work differs from Sparrow's and Fava’s is that the proposed model accounts for the amount of time between measurements and mentions of medical concepts. First, the model for predicting SBP changes accounts for the amount of time between measurements by using a logarithmic weighting scheme that gives more weight to recent numeric measurements. The intuition behind this is that more recent measurement values will be closer to the current value than less recent ones. The model also utilizes binary features tied to the relative time of a variable observation with respect to the
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 3
Author Manuscript
prediction date. Finally, the model is capable of making predictions without all feature values being known. These are not the only examples of work outside the realm of clinical NLP that predicts medical events on a patient by patient basis. Other notable examples include models that predict morbidity following gastrectomy [3] and respiratory morbidity among those with low birth weight [4]. Like the Sparrow paper, time isn’t taken into account when setting the feature values for these models and all feature values are assumed to be known. An example of work within the realm of clinical NLP where future medical events are predicted on a patient by patient basis is a paper by Bihorac and colleagues [5]. The paper presents a model that predicts patient mortality after surgery.
Author Manuscript
Numerous clinical NLP papers refer to their work as prediction (e.g Himes et al. [6], and Chen et al. [7]), but the term is being used in the sense of classification. In these papers, the classifier’s predictions are determined based on features or measurements from the same time to which the prediction pertains.
3.) Prediction Task
Author Manuscript
The goal of the present work is to provide a patient and their physician a prediction of the patient’s future SBP measurement in order to help ensure appropriate proactive health care. The formally defined prediction task is as follows. Given a list of patient records R = sorted in chronological order according to the patient’s visit date, the task is to predict the change in SBP measurement from rk to rn, where rn is the chronologically latest patient record that includes an SBP measurement (and is thus best suited to act as a future date) and rk is the record nearest in time to rn that also includes an SBP measurement, while only utilizing information present in the records that predated rn. Since utilizing any of the information from visits after rn (i.e., {rn+1, rn+2, …}) would obviously violate the goal of predicting a future outcome, these records are excluded from all further analysis or discussion. di denotes the visit date associated with ri,
4.) Data
Author Manuscript
The i2b2/UTHealth 2014 dataset for track 2 (identifying risk factors for heart disease over time) [14] is used to train and evaluate the system. The training portion of the data contains 790 individual records from 175 patients. The test portion consists of 542 individual records for 118 patients. Every patient in the dataset has been diagnosed with diabetes. After eliminating all patients from consideration where less than three records contain at least one SBP measurement, 514 records for 138 patients remain in the training data, and 309 records for 82 patients remain in the test data. Each record in the i2b2/UTHealth 2014 dataset has annotations associated with it. The annotations are used to set the values of the features that inform the prediction model. The system utilizes annotations that denote mentions of medical conditions, medications, and tobacco use.
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 4
Author Manuscript
5.) Extracting Systolic Blood Pressure Measurements and other Numeric Values using Regular Expressions The system uses regular expressions to extract SBP measurements as well as the values of measurements used to set all features described in section 5.3 and some of the features described in section 5.4. A regular expression encodes text patterns in strings. All substrings in a document that match the text pattern defined in a regular expression can be easily extracted. For example, the pattern string the system uses to extract weight measurements is “\bweight\s*(\s+|:)\s*\d+”. This pattern extracts all instances of the word “weight” preceded by a word boundary on the left side followed by whitespace with an optional single colon in the middle and a numeric string containing one or more characters. This regular expression extracts the substrings, “weight 100” and “weight: 210” but does not extract “weight is 100”.
Author Manuscript Author Manuscript Author Manuscript
BP measurement candidates are identified by extracting all substrings that match the regular expression "\d+/\d+" (a numeric string followed by a forward slash and another numeric string). The system uses a heuristic to filter out implausible BP values. Among the candidate substrings, if the numerator is between 40 and 300 and the denominator is between 30 and 200, the numerator of the fraction is identified by the system as an SBP measurement. Because a visit sometimes includes multiple SBP readings, we use the average of all SBP measurements in the patient record, AVGSBP(r), to approximate the true value of SBP. Regular expressions have the potential to extract incorrect values and miss values. When applied to training data and used to inform features used to predict actual changes in SBP, values extracted by regular expressions represent noisy, but potentially informative features. When applied to the BP measurements used to determine the changes in SBP upon which the performance evaluation is based, this presents a methodological problem. As such, we conducted two reviews to determine the accuracy of SBPs extracted using the aforementioned regular expression and heuristic. The first review involved annotating 65 randomly selected fraction substrings in the training data as being a BP measurement or not. Among the 65 fraction substrings 44 represented BP measurements and 21 did not. No errors were found in this review. Because our aim is to predict the change in SBP from the second most recent note containing an SBP measurement to the most recent note containing an SBP measurement it is critical that we ensure the SBP values in these notes are correct for the test set. As such, we manually reviewed these SBP measurements and found five cases where the values extracted by the regular expression and filtering heuristic did not represent SBP measurements. We manually removed these measurements from the test set. Occasionally, an SBP measurement in a note was from another point in time. From a methodological standpoint, this is problematic if it happens in the last or second to last note containing an SBP measurement in the test data, as the average SBP in those two notes are used to determine the gold standard SBP change. Fortunately, this occurs infrequently, and in cases where it does occur, the measurements were taken a short time prior to the visit.
5.) Features Each feature encodes information about a single patient. Most of the features fall into one of two categories: “Binary Time Sensitive Features” and “Real-Valued Time Adjusted
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 5
Author Manuscript
Features”. Features that do not fall into one of these two categories will be discussed on an individual basis. Gold standard annotations for medical events provided by the organizers of the i2b2/UTHealth 2014 challenge are used to directly set the values for some of the features that inform the prediction system. Using the gold standard annotations for the i2b2/ UTHealth 2014 dataset in this fashion is reasonable since the aim is to demonstrate that future medical events can be predicted for a patient assuming that relevant information can be extracted from longitudinal records. The extraction of information itself is a different problem than using information to predict events. 5.1) Binary Time Sensitive Features
Author Manuscript
Binary time sensitive features are used to represent one of two things; either that an event of type c was mentioned in one or more of the patient’s records within m months preceding dn, or that the earliest mention of an event of type c in the patient’s records occurred within m months preceding dn. These representations will be referred to as repRecent and repEarliest respectively. A binary time sensitive feature is denoted by the Boolean variable, , where c denotes the type of event and m denotes a span of months within which the most recent annotation (using repRecent) or earliest annotation (using repEarliest) of an event of type c is found. Using repRecent, the feature encodes the answer to the question “Did any mention of an event of type c occur within the last m months?” Using repEarliest, the feature encodes the answer to the question “Did the earliest mention of an event of type c occur within the last m months?” Each medical event type has ten features associated with it, five for repRecent and five for repEarliest, where m = 3, m = 6, m = 12, m = 24, and m = 36. For example, suppose that two mentions of event type c occur in a patient’s records. One occurs 20 months prior to dn and the other occurs 11 months prior to dn. Using repRecent,
Author Manuscript
, and
. Using repEarliest,
, and
.
5.2) Real-Valued Time Adjusted Features
Author Manuscript
Real-valued time adjusted features are used to represent numeric measurements for a patient that can vary over time, such as SBP, sodium, body weight, and glucose. The value of a realvalued time adjusted feature is the weighted average of each measurement of a given type for the patient in question. Measurements extracted from records closer to dn are given greater weight. When two or more measurements of the same type are detected in the same record, the average value of those measurements are used. All measurements represented by real-valued time adjusted features are extracted using regular expressions. The system does not determine if measurements in a record were taken before, on, or after the visit date. Figure 1 shows how the weight constant for the ith record is computed. The equation in figure 2 uses the weight function shown in figure 1 to compute the weighted average for all measurements of type c. The value returned by the equation in figure 2 is used to set the value of the real-valued time adjusted feature for measurement type c. In addition, each real-valued time adjusted feature has a trend-line feature associated with it. Specifically, linear regression is performed over the 〈di, AVGc(ri)〉 pairs, and the slope of the resulting trend-line is converted to one of the two nominal values: {positive, negative}.
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 6
5.3) Other Features
Author Manuscript
The set of features that do not fall into the two categories above include gender, height, BMI, approximate age on dn, and smoking status. The system identifies the patient’s Gender by counting the gender pronouns in each record associated with the patient in question. If the majority of the gender pronouns in all records for the patient are masculine, the system identifies that patient as male. Otherwise, the system identifies the patient as female.
Author Manuscript
Height values are extracted using a regular expression. All extracted values which are less than 54 or more than 84 in terms of inches are discarded. If any of the heights found in the records for a given patient vary by more than three inches (excluding the discarded heights), height is not recorded for that patient. Otherwise, the average of the values found by the regular expressions are used. The method we use to calculate BMI utilizes the time-based weighted average for the patient’s weight measurements and the patient’s height (both extracted using regular expressions) when both are mentioned in the patient’s records. This feature is represented by three nominal values representing ranges for BMI: {BMI < 25, 25 ≤ BMI < 30, 30 ≤ BMI}. We selected the thresholds based on BMI range classifications from the World Health Organization [12].
Author Manuscript
Calculating Approximate age on dk requires that the system be able to extract the patient’s age from at least one of the records for that patient. Age is extracted from records using a regular expression where all extracted values greater than 1251 are discarded. Assuming the patient’s age can be extracted from at least one record, the patient’s approximate date of birth, denoted by DOB, can be calculated as di – agei * (365.25) where agei is the age value extracted from di. DOB is accurate within one year. The approximate age of a patient on the prediction date is equal to (dn − DOB) in terms of years. This feature is represented by four nominal values denoting ranges for age: {26 – 50, 50 – 70, 71 – 80, 80+}. We did not attempt to directly extract the patient’s date of birth from the notes because it was rarely mentioned.
Author Manuscript
The system determines Smoking Status using a rule-based approach. Let S = 〈 s1, s2, · · ·sn〉 be the list of smoking status annotations for each patient record. Each si can be one of the following values: “unknown”, “current”, “past”, or “never”. Figure 4 shows the algorithm that determines smoking status for the purpose of the prediction model. This algorithm is necessary because the gold standard annotations for smoking status do not always reflect knowledge from previous records. For example, if si-m is equal to “past”, that would contradict si being equal to “never”.
1A reviewer justifiably expressed concern that “125 [years] seems to be on the high end for a cut-off”, given that “only one person in [recorded history] has ever lived to be 122”. A lower number would probably be a better choice.
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 7
5.4) Feature Selection
Author Manuscript
We selected features based on the output of a feature selection algorithm. The feature selection algorithm is implemented in the Java class “ClassifierSubsetEval” found in the Weka[8] library using a “Best First” ranking scheme with a backward search. This was done across 30 folds of the training data. Using cross validation on this training data, we discovered that including any feature not selected for more than 80% of the folds degraded performance. As such, these features were not used. See figure 5 for a list of selected features.
6.) Results
Author Manuscript
Weka’s implementation of Pearson’s correlation coefficient 2 is used to measure performance. A baseline classifier trained using only the slope direction of the trend-line for SBP yields a correlation coefficient of .39 on the test data. Using the actual slope of the trend-line to train the classifier yields a correlation coefficient of .07. We opted for the stronger baseline. Using the features extracted from the feature selection algorithm mentioned in section 5.4, the system achieves a correlation coefficient of .47 on the test data (p < .0001). In Figure 6, the model’s predictions are plotted along the graph’s vertical axis, and the actual changes in SBP from dk to dn are plotted along the graph’s horizontal axis.
7.) Discussion This section will first examine the prediction errors, then move on to examining the effectiveness of a few key features, including the presence of diabetes annotations, age, and medications.
Author Manuscript Author Manuscript
The test set contains 39 patients whose blood pressure increases and 43 patients whose blood pressure decreases. Error analysis uncovered that the model is more error prone when the absolute value of actual SBP change is high, and that the model has a tendency to overpredict SBP values. The model’s tendency to over-predict SBP may be explained by the different distribution of SBP changes in the training data vs the test data. This hypothesis is based on two things. First, the mean and median SBP change for each patient is +4.57 and +3.0 respectively for the training data and +1.8 and +0 respectively for the test data. Second, SBP increases or remains the same for 57% of the patients in the training data and 52% of patients in the test data. All patients in the i2b2/UTHealth 2014 dataset have diabetes, so a binary feature that indicates whether or not the patient has diabetes would always have the same value, and thus be uninformative. The presence of one or more mentions of diabetes within 24 months preceding dn was found to be a salient feature. This suggests the relative timeframe in which diabetes was diagnosed or mentioned is a predictor for future SBP. In contrast to the results shown in the Sparrow paper, age was not a salient feature in the present work. Using regular expressions, we determined the approximate age for 105 of the 138 patients in the training set. Given the relatively good coverage of this feature, it is unlikely that data sparseness is the reason it proved uninformative. One possible explanation
2http://weka.sourceforge.net/doc.dev/weka/attributeSelection/CorrelationAttributeEval.html
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 8
Author Manuscript
is that the age distribution for patients in the i2b2/UTHealth 2014 dataset and the Sparrow study are different. In the Sparrow study, the mean age of all patients at the time of first examination is 42 years. This means that the mean age of the patients at the time of the final examination was 52 years. In our study the mean age of the patient on dn is 67 years. Figure 7 provides a breakdown of the age distribution for the patients in the training set. It is important to note that we extracted age using regular expressions and did not manually verify the correctness of the extracted ages. As such, the patient ages we are using may not be totally correct.
Author Manuscript
Features that encode answers to the questions “How long has it been since the patient was diagnosed with a disorder or prescribed a medication?” and “How long has it been since the patient recovered from a disorder or discontinued a medication?” were not explicitly annotated in the i2b2/UTHealth 2014 dataset. Given a sufficiently large dataset where this information is available or extractable, we hypothesize features that answer these two questions would improve the performance of the model. The intuition behind this hypothesis is that the effects of disorders and medications vary with respect to the amount of time since diagnosis or prescription. The repEarliest variation of binary-time sensitive features gives a lower bound answer for the question: “How long has it been since the patient was diagnosed with a disorder or prescribed a medication?” by answering the question: “How long has it been since the first annotation of a medication or disorder?”. If the first annotation of a medication or disorder occurred at some date d, that disorder or medication must have been first diagnosed or prescribed no later than d. Although the feature selection algorithm did not find any features represented in this fashion to be salient, it’s plausible that given a larger amount of training data, a predictive model could learn from features represented in this fashion.
Author Manuscript
The prediction of medical events from notes that lack annotations using currently available NLP technology is a practical direction for future work. Existing clinical NLP tools like cTAKES [11] can be used to identify UMLS [9] and RxNorm [10] concepts in text. From these identified concepts, it is plausible that clinical prediction systems could be created or improved upon. Beyond serving as a proof of concept that predictive models can be developed by applying statistical models to the output of currently available NLP tools, such a system would not be as limited in the scope of medical concepts that could serve as potential features for the model. Thus, predictive features other than those explicitly annotated could be identified and exploited. Clinical notes are not the only source of data the presented model could exploit. A medical database containing measurements, prescriptions, and diagnoses on a patient by patient basis could also be leveraged in addition to or in place of textual data.
Author Manuscript
In conclusion, due to the time sensitive nature of the feature representation, a system that uses the presented model or a variation of it could be used to predict future medical events in a way that accounts for the amount of time that has passed between the medical events that inform the model and the prediction date. This is a particularly attractive feature in the face of real world data where the amount of time between the prediction date and the relevant observation is variable. If a likely adverse medical event can be predicted automatically, proper treatments and behavior changes are more likely to be implemented.
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 9
Author Manuscript
Acknowledgments The authors would like to thank Kathryn Farmer for her invaluable feedback on the paper.
References
Author Manuscript Author Manuscript
1. Stubbs A, Kotfila C, Hua X, Ozlem U. Practical applications for NLP in clinical research: the 2014 i2b2/UTHealth shared tasks. 2. David, Sparrow; Arthur, Garvey; Bernard, Rosner; Thomas, H. Factors in predicting blood pressure change. American Heart Association. Circulation. 1982; 65:789–94. [PubMed: 7060257] 3. Grossman EM, Longo WE, Virgo KS, et al. Morbidity and mortality of gastrectomy for cancer in department of veterans affairs medical centers. Surgery. 2002 May; 131(5):484–90. [PubMed: 12019399] 4. Teune MJ, Wassenaer AG, van Buuren S, Mol BW, Opmeer BC. Dutch POPS Collaborative Study Group. Perinatal risk-indicators for long-term respiratory morbidity among preterm or very low birth weight neonates. Eur J Obstet Gynecol Reprod Biol. 2012 Aug; 163(2):134–41. [PubMed: 22579229] 5. Bihorac A, Cobb R, Wang DZ, Korenkevych D, Baslanti TO, Puri S, Momcilovic P. 692: Computer algorithms are superior to physician assessment of the risk for postoperative complications. Critical Care Medicine. 2013; 41(12):A170. 6. Himes B, Dai Y, Kohane I, Wiess S, Ramoni M. Prediction of chronic obstructive pulmonary disease (COPD) in asthma patients using electronic medical records. Journal of the American Medical Informatics Association. 2009; 16(3):371–9. [PubMed: 19261943] 7. Chen, Lin; Elizabeth, Karlson; Helena, Canhao; Timothy, Miller. Automatic prediction of rheumatoid arthritis disease activity from the electronic medical records. PLoS ONE. 2013; 8:e69932. [PubMed: 23976944] 8. Hall M, Frank E, Holmes G, Pfahringer B, Reutermann P, Witten I. The weka data mining software: an update. ACM SIGKDD Explorations Newsletter. 2009; 11(1):10–18. 9. Lindberg DA, Humphreys BL, McCray AT. The Unified Medical Language System. Methods of information in medicine. 1993; 32(4):281–291. [PubMed: 8412823] 10. Liu S, Ma W, Moore R, Ganesan V, Nelson S. RxNorm: prescription for electronic drug information exchange. IT professional. 2005; 7(5):17–23. 11. Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, Chute CG. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informatics Association. 2010; 17(5):507–513. [PubMed: 20819853] 12. World Health Organization, & World Health Organization. Global database on body mass index. 2006 13. Fava C, Sjögren M, Montagnana M, Danese E, Almgren P, Engström G, … Melander O. Prediction of blood pressure changes over time and incidence of hypertension by a genetic risk score in Swedes. Hypertension. 2012 HYPERTENSIONAHA-112. 14. Annotating Risk Factors for Heart Disease in Clinical Narratives for Diabetic Patients Amber Stubbs and Ozlem Uzuner. 2014
Author Manuscript
Appendix (Regular Expressions) MAX – the measurement’s value must be less than our equal to this number MIN – the measurement’s value must be greater than or equal to this number. •
Age (years): “\bage\s*(\s+|:)\s*\d+” –
MAX: 125
J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 10
•
Author Manuscript
•
Height (inches): “\d/’\d{1,2}” –
MIN: 54 inches
–
MAX: 74 inches
Weight (pounds): “\bweight\s*(\s+|:)\s*\d+” –
MAX: 700 lbs
•
Sodium (mmol/L): “\bsodium\s*(\s+|:)\s*\d+”
•
Glucose (mg/dl): “\bglucose\s*(\s+|:)\s*\d+”
•
Blood Pressure (mmhg): “\d+/\d+”
Author Manuscript
–
MAX systolic: 300
–
MIN systolic: 40
–
MAX diastolic: 200
–
MIN diastolic: 30
Author Manuscript Author Manuscript J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 11
Author Manuscript
Highlights •
We present a model for predicting changes in systolic blood pressure (SBP).
•
The model is trained and tested using longitudinal clinical records.
•
Some features that inform the model account for the time of observation.
•
The model’s estimates have a correlation coefficient of 0.47 with actual SBP change.
Author Manuscript Author Manuscript Author Manuscript J Biomed Inform. Author manuscript; available in PMC 2016 August 18.
Solomon and Nielsen
Page 12
Author Manuscript
Figure 1.
hasMeas(r, c) returns 1 if document r contains a measurement of type c and 0 otherwise. Because documents dated later than dn are not considered (i
