GAMES FOR HEALTH JOURNAL: Research, Development, and Clinical Applications Volume 3, Number 4, 2014 ª Mary Ann Liebert, Inc. DOI: 10.1089/g4h.2014.0058
Editorial
Measurement Method Bias in Games for Health Research Tom Baranowski, PhD
A
s games for health scientists, we all want our research to test important hypotheses to arrive at clear incontrovertible results that will provide fresh new insights about games for health; and to influence others’ research about game design, or perhaps policy. Quality measurement is an important feature of good research. A good measure will have high validity (in the biological sciences, this is often called accuracy), high reliability (in the biological sciences, this is often called precision), and sensitivity to change (the units are appropriate for detecting the level of changes that are likely to occur). Threats to validity are usually called biases1 and manifest as a systematic distortion (e.g., the true value may indicate a person is very knowledgeable on a topic, but a biased measure may indicate the person is less knowledgeable, or even not knowledgeable at all). Threats to reliability are primarily random error or noise, which makes it difficult to detect relationships with other variables (often called attenuation). Some authors of Games for Health Journal (G4HJ) articles use single-item self-report measures, or a series of such measures, and appear to believe that participants/ respondents understand what is being asked and are accurately reporting it. Problems with such items include the following: The items will not have been validated (and so we cannot have confidence that they measure what we want/ think/expect); most items will not have been cognitively tested (and so we cannot be sure that our participants understand the question in the way we intended); and single items are often not enough to assess complex constructs with many facets (and we can’t assess the internal consistency reliability of the single response). The best measurement practice in this regard is to use multi-item scales that have been reported validated in samples comparable to those in our study. But there is another, perhaps more insidious, problem: Method bias. Any factor that threatens measurement validity is called a bias (i.e., it gives a systematically distorted/wrong answer). Method bias is shared error among measures due to a common error source. Self-reported measures share common error, often some form of socially desirable response, but could include common sources of errors in instructions, type of items, characteristics of the examiner or situation, etc. If method bias exists, then two measures will have some level of empirical correlation, not because the underlying
constructs are truly related, but to the extent they share common bias. The literature on this kind of bias has been recently reviewed2 and is relevant to our work. I very briefly report their findings and draw implications for G4HJ authors. The authors summarized several meta-analyses of the measurement literature and reported that somewhere between 18 percent and 32 percent of the variance in selfreported items was due to method factors or random error.2 This is high and threatens our ability to detect or properly interpret true effects/relationships that are really there. The authors report many alarming, but difficult to understand, facts. One more easily understood fact is that correlations of psychosocial variables with behavior are 239 percent larger when using measures from the same source (e.g., self-report) than when using measures coming from different sources. These are appalling findings and indicate our research can provide misleading results, even when we are think we are being conscientious (by using multi-item scales). The accepted way for minimizing method bias is to use validated measures from different sources (e.g., self-report and objective gadget), thereby with different sources of error. There are other ways,2 but they involve a substantial amount of extra data processing and analyses (which many of us are not excited about doing) and aren’t proven effective bias reduction techniques in all cases. So the bottom line for games for health investigators to avoid the problems associated with method bias is to use multiple measures from different sources. For example, selfreport measures might be used to assess the impact of a game on one or more psychosocial variables that would be expected to be affected by a game and best assessed by selfreport (e.g., transportation, self-efficacy, perceived safety of a neighborhood) and use of non–self-report measures of the behavior (e.g., accelerometry or step counts for physical activity, cafeteria observations of dietary intake, biomarkers of smoking, automated pill counts for medication compliance, etc.). We will likely get lower correlations between these measures, but the estimates will be closer to true effects or relationships. As researchers in search of truth, that is what we ultimately want, right? The article by Podsakoff et al.2 is quite interesting (at least for method geeks) and deserves a good read!
Pediatrics (Behavioral Nutrition & Physical Activity), USDA/ARS Children’s Nutrition Research Center, Department of Pediatrics, Baylor College of Medicine, Houston, Texas. Editor-in-Chief, Games for Health Journal.
193
194
Hope you enjoy the current issue as much as I did assembling it! References
1. Coggon D, Rose G, Barker DJP. Epidemiology for the uninitiated, 4th edition: 4. Measurement error and bias.
EDITORIAL
BMJ 2014. http://www.bmj.com/about-bmj/resources-readers/ publications/epidemiology-uninitiated/4-measurement-errorand-bias (accessed May 2, 2014). 2. Podsakoff PM, MacKenzie SB, Podsakoff NP. Sources of method bias in social science research and recommendations on how to control it. Annu Rev Psychol 2012; 63: 539–569.
Editorial
Measurement Method Bias in Games for Health Research Tom Baranowski, PhD
A
s games for health scientists, we all want our research to test important hypotheses to arrive at clear incontrovertible results that will provide fresh new insights about games for health; and to influence others’ research about game design, or perhaps policy. Quality measurement is an important feature of good research. A good measure will have high validity (in the biological sciences, this is often called accuracy), high reliability (in the biological sciences, this is often called precision), and sensitivity to change (the units are appropriate for detecting the level of changes that are likely to occur). Threats to validity are usually called biases1 and manifest as a systematic distortion (e.g., the true value may indicate a person is very knowledgeable on a topic, but a biased measure may indicate the person is less knowledgeable, or even not knowledgeable at all). Threats to reliability are primarily random error or noise, which makes it difficult to detect relationships with other variables (often called attenuation). Some authors of Games for Health Journal (G4HJ) articles use single-item self-report measures, or a series of such measures, and appear to believe that participants/ respondents understand what is being asked and are accurately reporting it. Problems with such items include the following: The items will not have been validated (and so we cannot have confidence that they measure what we want/ think/expect); most items will not have been cognitively tested (and so we cannot be sure that our participants understand the question in the way we intended); and single items are often not enough to assess complex constructs with many facets (and we can’t assess the internal consistency reliability of the single response). The best measurement practice in this regard is to use multi-item scales that have been reported validated in samples comparable to those in our study. But there is another, perhaps more insidious, problem: Method bias. Any factor that threatens measurement validity is called a bias (i.e., it gives a systematically distorted/wrong answer). Method bias is shared error among measures due to a common error source. Self-reported measures share common error, often some form of socially desirable response, but could include common sources of errors in instructions, type of items, characteristics of the examiner or situation, etc. If method bias exists, then two measures will have some level of empirical correlation, not because the underlying
constructs are truly related, but to the extent they share common bias. The literature on this kind of bias has been recently reviewed2 and is relevant to our work. I very briefly report their findings and draw implications for G4HJ authors. The authors summarized several meta-analyses of the measurement literature and reported that somewhere between 18 percent and 32 percent of the variance in selfreported items was due to method factors or random error.2 This is high and threatens our ability to detect or properly interpret true effects/relationships that are really there. The authors report many alarming, but difficult to understand, facts. One more easily understood fact is that correlations of psychosocial variables with behavior are 239 percent larger when using measures from the same source (e.g., self-report) than when using measures coming from different sources. These are appalling findings and indicate our research can provide misleading results, even when we are think we are being conscientious (by using multi-item scales). The accepted way for minimizing method bias is to use validated measures from different sources (e.g., self-report and objective gadget), thereby with different sources of error. There are other ways,2 but they involve a substantial amount of extra data processing and analyses (which many of us are not excited about doing) and aren’t proven effective bias reduction techniques in all cases. So the bottom line for games for health investigators to avoid the problems associated with method bias is to use multiple measures from different sources. For example, selfreport measures might be used to assess the impact of a game on one or more psychosocial variables that would be expected to be affected by a game and best assessed by selfreport (e.g., transportation, self-efficacy, perceived safety of a neighborhood) and use of non–self-report measures of the behavior (e.g., accelerometry or step counts for physical activity, cafeteria observations of dietary intake, biomarkers of smoking, automated pill counts for medication compliance, etc.). We will likely get lower correlations between these measures, but the estimates will be closer to true effects or relationships. As researchers in search of truth, that is what we ultimately want, right? The article by Podsakoff et al.2 is quite interesting (at least for method geeks) and deserves a good read!
Pediatrics (Behavioral Nutrition & Physical Activity), USDA/ARS Children’s Nutrition Research Center, Department of Pediatrics, Baylor College of Medicine, Houston, Texas. Editor-in-Chief, Games for Health Journal.
193
194
Hope you enjoy the current issue as much as I did assembling it! References
1. Coggon D, Rose G, Barker DJP. Epidemiology for the uninitiated, 4th edition: 4. Measurement error and bias.
EDITORIAL
BMJ 2014. http://www.bmj.com/about-bmj/resources-readers/ publications/epidemiology-uninitiated/4-measurement-errorand-bias (accessed May 2, 2014). 2. Podsakoff PM, MacKenzie SB, Podsakoff NP. Sources of method bias in social science research and recommendations on how to control it. Annu Rev Psychol 2012; 63: 539–569.
