A Readability Assessment of Online Stroke Information Nikhil Sharma, MBChB, MRCP, Andreas Tridimas, BMBS, and Paul R. Fitzsimmons, MBChB, MRCP
Background: Patients and carers increasingly access the Internet as a source of health information. Poor health literacy is extremely common and frequently limits patient’s comprehension of health care information literature. We aimed to assess the readability of online consumer-orientated stroke information using 2 validated readability measures. Methods: The 100 highest Google ranked consumer-oriented stroke Web pages were assessed for reading difficulty using the Flesch–Kincaid and Simple Measure of Gobbledygook (SMOG) formulae. Results: None of the included Web pages complied with the current readability guidelines when readability was measured using the gold standard SMOG formula. Mean Flesch–Kincaid grade level was 10.4 (95% confidence interval [CI] 9.97-10.9) and mean SMOG grade 12.1 (95% CI 11.7-12.4). Over half of the Web pages were produced at graduate reading levels or above. Not-for-profit Web pages were significantly easier to read (P 5 .0006). The Flesch–Kincaid formula significantly underestimated reading difficulty, with a mean underestimation of 1.65 grades (95% CI 1.49-1.81), P , .0001. Conclusions: Most consumer-orientated stroke information Web sites require major text revision to comply with readability guidelines and to be comprehensible to the average patient. The Flesch–Kincaid formula significantly underestimates reading difficulty, and SMOG should be used as the measure of choice. Key Words: Stroke—readability—Internet—online—literacy. Ó 2013 by National Stroke Association
Introduction Stroke is a common condition affecting over 150,000 people in the United Kingdom per year.1 It is a major cause of chronic neurologic disability and may be considered to be the most common cause of complex disability in the United Kingdom.2 Patient education is important in stroke and helps patients, their families, and carers to manage the long-term effects of a stroke and its treatment. Although patients receive information from health care staff, many From the Department of Gerontology, Royal Liverpool and Broadgreen Hospitals NHS Trust, Liverpool, Merseyside, UK. Received May 13, 2013; revision received October 20, 2013; accepted November 19, 2013. Sources of funding: None. Disclosures: None. Address correspondence to Nikhil Sharma, MBChB, MRCP, Department of Gerontology, Royal Liverpool and Broadgreen Hospitals NHS Trust, Prescot St, Liverpool, Merseyside L7 8XP. E-mail: [email protected]. 1052-3057/$ - see front matter Ó 2013 by National Stroke Association http://dx.doi.org/10.1016/j.jstrokecerebrovasdis.2013.11.017
seek further information independently, with the Internet being one of the most commonly used sources. There is a huge volume of stroke-related information available on the Internet, with a Google search for ‘‘stroke’’ returning 228 million Web pages. It is well recognized that the general population is increasingly turning to the Internet for health care information. In the United States, 82% of adults use the Internet,3 and of these, 72% sought online health information in the past year, with a total of 59% of American adults having browsed online for health information in the past year. Most of these users (77%) began their search for health information with an online search engine such as Google.4 Health information provision is 1 major function of the Internet with over 80% of American Internet users searching online for health information at some time. This is more than have been online to check the news (78%), buy products (71%), watch online video content (71%), or use social networking sites (69%).3 There is also evidence that those with chronic conditions and disability, such as stroke survivors, are more likely to search for health information than others.5
Journal of Stroke and Cerebrovascular Diseases, Vol. -, No. - (---), 2013: pp 1-6
1
N. SHARMA ET AL.
2
No peer-reviewed published estimates of UK Internet use for health care information or specifically by stroke patients are available. However, the number of households in the United Kingdom with Internet access has risen from around 42% to 80% in the past decade.6 Although the Internet is a popular and convenient source of consumer-oriented health care information, these resources are only useful if the consumer is able to comprehend the information presented. This is even more pertinent in stroke patients, where the capacity for language comprehension is often affected. Health literacy is ‘‘the constellation of skills, including the ability to perform basic reading and numerical tasks, required to function in the health care environment’’ as defined by the American Medical Association’s Ad Hoc Committee on Health Literacy in 1999.7 Similarly, a 2011 systematic review defined health literacy as ‘‘the degree to which individuals can obtain, process, and understand the basic health information and services they need to make appropriate health decisions.’’ Subsequently, it demonstrated that poor health literacy is associated with increased hospitalization, emergency care use, and among older people, poorer overall health status and higher mortality.8 Reading ability is an integral component of health literacy. Those with inadequate literacy have difficulty reading and understanding material written at the sixth-grade level (age 11-12 years), whereas those with marginal literacy have difficulty comprehending information written at the 10th-grade level (age 15-16 years).9 A national survey has shown that around 1 in 6 people in the United Kingdom have levels of general literacy below that expected of an 11-year old.10 A study of nearly 8000 older patients in the United Kingdom showed that almost a third had difficulties reading and understanding basic written health information. This study also demonstrated that those with lower health literacy had significantly higher mortality rates.11 A pooled analysis of 85 American studies representing 31,129 subjects showed a prevalence of poor health literacy of 26% and also demonstrated a significant association between older age and low health literacy with a third of the population older than 50 years having inadequate heath literacy skills.12 These findings have been supported by data published by the US National Center for education statistics, which undertook its National Assessment of Adult Literacy in 2003. Again adults in the oldest age group (older than 65 years) had the lowest levels of health literacy.13 In view of this, health literacy is a particular public health concern in stroke medicine as the incidence of stroke increases with advancing age. To date, there is only 1 small published study that has investigated health literacy in stroke patients. This Australian study of 57 patients found stroke patients to have a mean reading level of seventh to eighth grade,
although those with combined aphasia, as would be expected, read at a significantly lower level.14 In view of this high level of inadequate health literacy, the US Department of Health and Human Services (USDHHS) recommends that patient-orientated literature should be written at or below the sixth-grade level (age 11-12 years) and has developed an action plan to promote patient literacy.15 The USDHHS literacy classification system categorizes material as ‘‘easy to read’’ only if written at or below a sixth-grade level. Material between the 7th- and 9thgrade level is ‘‘average’’ and above the 9th-grade level is regarded as difficult. The United Kingdom currently has no quantitative guidelines regarding levels of readability for patient orientated literature; hence, the USDHHS guidelines outlined above have been taken as the international gold standard for the purpose of this study. Ours is the only study of readability in consumerorientated online general stroke literature of which we are aware. However, a recent study of 51 Web sites pertaining to patients with speech and language difficulties after a stroke showed that only 6% of Web sites had a reading level suitable for the sixth grade or below.16
Methods Sampling Method We analyzed the text content of the 100 highest ranked consumer-orientated Web pages, selected on the basis of Google search ranking and consumer orientation. A full description of the sampling method is given below. We performed a Google search on January 31, 2013, for the term ‘‘Stroke.’’ This returned about 228,000,000 Web page results. Google was selected as it is the most popular search engine, accounting for more than 90% of UK consumer health-related search activity in 2012, with a number of other search engines making up the remaining 10%.17 Starting with the highest ranking nonsponsored Web page link, each of the Web page results returned by this Google search was reviewed by a single researcher (A.T.). Initially, a decision was made as to whether the Web page contained consumer-orientated health care information (aimed primarily at lay persons) or was aimed at health care professionals (journal articles, textbooks, and so on). Only consumer-orientated Web pages containing health care information regarding stroke were included for further assessment. Each consumer-orientated Web page was then classified by the following criteria: 1. Commercial or not-for-profit—Not-for-profit was defined as a Web site of a registered charity, educational institution, or governmental organization.
READABILITY ASSESSMENT OF ONLINE STROKE INFORMATION
2. Web site country of origin—classified as Europe, United States, or other. The classification for each Web page was determined from the ‘‘about us’’ section of the hosting Web site. This process was repeated for each Google search Web page result in turn, until 100 unique Web pages containing consumer-orientated health care information were identified. A sample size of 100 Web pages was selected as it was felt unlikely that most stroke survivors would click on more than 100 consecutive links from a Google search. Where duplicate Web pages were identified, only the highest ranking Web page was included in the analysis. Unique Web pages from the same Web site were included, providing the information contained in the Web page did not duplicate previously analyzed text content. A second researcher (N.S.) independently reviewed the Web pages to confirm consumer orientation and to screen for duplicate results.
Readability Assessment Full Web page body text content was extracted into Microsoft Word 2007, and readability was assessed using 2 validated measures: the Flesch–Kincaid Grade Level and Flesch–Kincaid and Simple Measure of Gobbledygook (SMOG). These represent the most commonly used readability measure (Flesch–Kincaid) and the most exacting (SMOG). Titles, subtitles, references, Web links, and advertising text were excluded from the analysis, with only body text and bullet point text included. The Flesch–Kincaid Grade Level Formula is well established and conveniently is often located in wordprocessing packages such as MS Word. The Flesch– Kincaid formula gives a score that estimates the grade level (number of years of education) required to understand the text.18 Despite the formula’s widespread use, validation studies have shown that it may be of limited accuracy when grading low literacy-level materials.19 The formula is expressed as [(.39 3 ASL) 1(11.8 3 ASW) 2 15.59], where ASL is the average sentence length (number of words divided by number of sentences) and ASW is the average syllables per word (number of syllables divided by number of words).18 SMOG is a more exacting measure of readability and accurately grades for the grade level required for complete text comprehension (whereas the Flesch–Kincaid formula grades for less than complete comprehension), demonstrating strong correlation (r 5 .985) with the required reading level in validation studies.20 The SMOG formula, in brief terms, consists of counting the words of 3 syllables or more in three 10-sentence samples, calculating the count’s square root and adding 3 to obtain the grade level.20 The SMOG and Flesch–Kincaid grade levels were measured using an online calculator from online-utili-
3 21
ty.org. For the purpose of this study, the reading grade level required for complete comprehension of the text material (ie, the SMOG grade level) was taken as the gold standard reliability measure.
Statistical Analysis Reading level data were categorized in line with USDHHS standards and analyzed in GraphPad Prism 4.0 (GraphPad Software, San Diego, CA). The D’Agostino and Pearson omnibus normality test demonstrated that SMOG and Flesch–Kincaid Grade Level data followed parametric distributions. As such, the paired t test was used to compare paired SMOG and Flesch–Kincaid Grade Level values. The unpaired t test and 1-way analysis of variance were used to compare SMOG grade levels in Web pages of varying commercial nature (commercial and not-for-profit) and nationality (United Kingdom, European, and other), respectively. Relationships between Web page ranking, word count, and readability were analyzed using Pearson correlation. Statistical significance was set at .05.
Results Our analysis included 100 unique Web pages. The mean Web page word count was 747 words (95% confidence interval [CI] 551-931). Of the 100 Web pages analyzed, 44% were run as commercial interests and 56% on a not-for-profit basis. More than half (53%) of Web sites were based in the United States, with 30% based in Europe and 17% based in other countries. Table 1 shows the distribution of Web pages by USDHHS classification and grade level for the Flesch– Kincaid and SMOG formulae. Mean Flesch–Kincaid grade level was 10.4 (95% CI 9.9710.9) and mean SMOG grade 12.1 (95% CI 11.7-12.4). A strong and significant correlation was seen between the grade levels produced by the formulae (Pearson r 5 .952, P # .0001). The Flesch–Kincaid formula significantly underestimated reading difficulty when compared with the gold standard SMOG formula, with a mean underestimation of 1.65 grades (95% CI 1.49-1.81, P , .0001). No significant correlation was observed between SMOG readability and search engine ranking (r 5 .035, P 5 .73). A weak but significant correlation was observed between SMOG readability and Web page word count (r 5 .303, P 5 .002). Not-for-profit Web pages were significantly easier to read than commercial Web pages, with a mean commercial SMOG grade of 12.8 (95% CI 12.3-13.3) and a mean not-for-profit SMOG grade of 11.5 (95% CI 11.0-12.0, P 5 .0006). No significant difference in SMOG grade level was observed between Web pages taken from Web sites based
N. SHARMA ET AL.
4
Table 1. Number of Web sites in each USDHHS classification group according to SMOG and Flesch–Kincaid readability formulae USDHSS classification
Grade level
SMOG (% Web pages)
Flesch–Kincaid (% Web pages)
Easy Average Difficult Very difficult
4th-6th grade 7th-9th grade 10th-11th grade 12th grade and above
0 12 37 51
5 44 23 28
Abbreviations: SMOG, Simple Measure of Gobbledygook; USDHHS, US Department of Health and Human Services.
in different countries. The mean SMOG grade for US Web pages was 12.1 (95% CI 11.5-12.6), the mean SMOG grade for European Web pages was 12 (95% CI 11.3-12.7), and the mean SMOG grade for Web pages of other nationality was 12.3 (95% CI 11.5-13.2, P 5 .801).
Conclusions The Internet is rapidly growing and is a highly convenient source of information for patients. Irrespective of its quality, this information can only be helpful if patients are able to understand it. We found that most consumer-orientated stroke information Web pages were written at the 12th-grade level or above and that none complied with the USDHHS maximum recommended sixth-grade level when readability was measured with SMOG. This was in keeping with similar studies looking at readability in other areas of health care, including Parkinson disease,22 mental health,23 epilepsy,24 orthopedic surgery,25 fibromyalgia,26 and warfarin therapy.27 Similarly, excessive levels of reading difficulty have long been noted in printed patient information.28 The readability levels of the Web pages analyzed contrast sharply with the observed reading abilities of patients, with the average American adult reading between the seventh and eighth grade,13 inferring that only 12% of Web pages included in this study would be fully comprehensible to the average adult when measured by SMOG. Over half the Web pages (51%) were written at graduate level and above (SMOG grade level . 12). There was a significant difference in readability between commercial and not-for-profit Web sites. Interestingly not-for-profit Web sites were easier to read than commercial sites. This is in contrast to a recent similar study performed looking at Parkinson disease Web sites.22 The reasons for this are unclear; previously, it has been hypothesized that the increased ease of reading of commercial Web pages was because of different writing styles in each of these sectors, with commercial authors and editors aiming to ensure the widest possible commercial audience by minimizing reading difficulty.22 Our current study does not support this hypothesis. Similar studies on patient information for warfarin therapy29 and inflammatory bowel disease30 have shown no statistically significant difference between different Web
site provider categories. Further research is needed in this area. Readability levels were similarly poor in Web pages produced in the United States, Europe, or other countries. This suggests that readability is likely to be a limiting factor in the ability of patients to comprehend locally produced stroke information in most developed countries. There was only a weak correlation between SMOG score and word count and no significant correlation with search engine ranking. This is the first study to look at the readability of consumer-orientated online general stroke information on multiple Web sites. This study used the SMOG formula that accurately grades for complete comprehension of reading materials. The Flesch–Kincaid formula appears attractive because it is incorporated into commercial word-processing software. However, compared with the gold standard SMOG readability measure, Flesch–Kincaid significantly underestimated reading difficulty, presumably, because it grades for less than complete comprehension of written material. We recommend that SMOG should be the preferred measure of readability when evaluating consumer-orientated health care material. Our study is limited in that we did not undertake an assessment of the quality, accuracy, or relevance of the information presented. As such, we are unable to comment as to whether information quality is preserved in Web pages written at low literacy levels. However, although providing good quality information is of critical importance, its positive impact is negligible if that information is presented in an unintelligible format. We have identified 1 recent study incorporating strokespecific scoring systems for quantifying Web site quality and readability.16 The quality of Web-based information on stroke was found to be highly variable and did not correlate with readability. This trend is been seen across Web-based information for many other conditions.31-34 By reporting the discrepancy between levels of Web site readability and population estimates of literacy, we hope to encourage compliance with readability guidelines and, thus, improve accessibility to online information for stroke patients. We hope to specifically encourage those in clinical practice to consider readability as an important factor when developing patient literature. Studies have suggested involving patients in the design process of Web sites can be beneficial as they
READABILITY ASSESSMENT OF ONLINE STROKE INFORMATION 34,35
will ultimately be the target audience. Consideration of consumer needs in this way may act as an additional measure to ensure readability. Web site editors should introduce minimum readability policies based on USDHHS guidelines and also consider routine monitoring of content reading difficulty with SMOG in an attempt to increase the accessibility and ease of comprehension of online consumer-orientated stroke information. There is useful guidance to this end available from the Centre for Health Care Strategies36 and the National Institutes of Health.37 In summary, most consumer-orientated Stroke information Web sites require major text revision to comply with readability guidelines and to be comprehensible to the average stroke survivor.
References 1. Townsend N, Wickramasinghe K, Bhatnagar P, et al. Coronary heart disease statistics 2012 edition. London: British Heart Foundation 2012. 2. Adamson J, Beswick A, Ebrahim S. Stroke and disability. J Stroke Cerebrovasc Dis 2004;13:171-177. 3. Trend data (adults). Pew Internet & American Life Project. Available at: http://www.pewinternet.org/Trend-Data-% 28Adults%29/Online-Activites-Total.aspx. 2012. Accessed March 1, 2013. 4. Fox S, Duggan M. Health online 2013. Pew Internet & American Life Project. Available at: http://www.pewinternet. org/Reports/2013/Health-online/Summary-of-Findings. aspx 15 Jan 2013. Accessed January 19, 2013. 5. Bundorf MK, Wagner TH, Singer SJ, et al. Who searches the Internet for health information? HSR: Health Services Research 2008;41:819-836. 6. Office for National Statistics. Internet access—households and individuals 2012. Available at: http://www. ons.gov.uk/ons/dcp171778_275775.pdf. August 2012, Accessed January 19, 2013. 7. Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, American Medical Association. Health literacy: report of the Council on Scientific Affairs. JAMA 1999;281:552-557. 8. Berkman ND, Sheridan SL, Donahue KE, et al. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011;155:97-107. 9. Gazmararian JA, Baker DW, Williams MV, et al. Health literacy among Medicare enrolees in a managed care organization. JAMA 1999;281:545-551. 10. Williams J, Clemens S, Oleinikova J, Tarvin Kl. The skills for life survey: a national needs and impact survey of literacy, numeracy and ICT skills. London: Department for education and skills 2003. 11. Bostock S, Steptoe A. Association between low functional health literacy and mortality in older adults: longitudinal cohort study. BMJ open 2012;344:e1602. 12. Paasche-Orlow MK, Parker RM, Gazmararian JA, et al. The prevalence of limited health literacy. J Gen Intern Med 2005;20:175-184. 13. Kutner M, Greenberg E, Jin Y, Paulsen C. The Health Literacy of America’s Adults: results from the 2003 National Assessment of Adult Literacy (NCES 2006–483). Washington, DC: US Department of Education, National Center for Education Statistics 2006.
5
14. Hofmann T, McKenna K. Analysis of stroke patients’ and carers’ reading ability and the content and design of written materials: recommendations for improving written stroke information. Patient Educ Counsel 2006; 60:286-293. 15. National action plan to improve health literacy. Washington, DC: US Department of Health and Human Services, Office of Disease Prevention and Health Promotion 2010. 16. Surman R, Bath PA. An assessment of the quality of Information on stroke and speech and language difficulty websites. J Information Sci 2013;39:113-125. 17. Healthcare WebWatch 2012: an analysis of web activity in the health and medical sector. Private Healthcare UK website. Available at: http://www.privatehealth.co.uk/ aboutus/webwatch-2012. Accessed January 19, 2013. 18. Flesch R. How to write in plain English. New York: Harper and Row 1979. 19. McLaughlin GH. Temptations of the Flesch. Instr Sci 1974;2:367-384. 20. McLaughlin GH. SMOG grading: a new readability formula. J Reading 1969;12:639-646. 21. Tests document readability, readability calculator. Online utility-org website. Available at: http://www.online-utility. org/english/readability_test_and_improve.jsp. Accessed February 6, 2013. 22. Fitzsimmons PR, Michael BD, Hulley JL, Scott GO. A readability assessment of online Parkinson’s disease information. J R Coll Physicians Edinb 2010;40: 292-296. 23. Kalk NJ, Pothier DD. Patient information on Schizophrenia on the internet. Psychiatrist 2008;32:409-411. 24. Elliott JO, Shneker BF. A health literacy assessment of the epilepsy.com website. Seizure 2009;18:434-439. 25. Sabharwal S, Badarudeen S, Kunju SU. Readability of online patient education materials from the AAOS web site. Clin Orthop Relat Res 2008;466:1245-1250. 26. Daraz L, MacDermid JC, Wilkins S, et al. The quality of websites addressing fibromyalgia: an assessment of quality and readability using standardised tools. BMJ Open 2011;1:e000152. 27. Nasser S, Mullan J, Bajorek B. Assessing the quality, suitability and readability of internet-based health information about warfarin for patients. Austr Med J 2012; 5:194-203. 28. Davis TC, Crouch MA, Wills G, et al. The gap between patient reading comprehension and the readability of patient education materials. J Fam Pract 1990;31: 533-538. 29. Estrada CA, Hryniewicz MM, Higgs VB, et al. Anticoagulant patient information material is written at high readability levels. Stroke 2000;31:2966-2970. 30. van der Marel S, Duijvestein M, Hardwick JC, et al. Quality of web based information on inflammatory bowel diseases. Inflammatory Bowel Dis 2009;15: 1891-1896. 31. Bouchier H, Bath PA. Evaluation of web sites that provide information on Alzheimer’s disease. Health Informatics J 2003;9:17-31. 32. Bath PA, Bouchier H. Development and application of a tool designed to evaluate web sites providing information on Alzheimer’s disease. J Information Sci 2003; 29:279-297. 33. Harland J, Bath PA. Assessing the quality of web sites providing information on multiple sclerosis: evaluating tools and comparing sites. Health Information J 2007; 13:207-221.
6 34. Meric F, Bernstam EV, Mirza NQ, et al. Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of web sites. Br Med J 2002; 324:577-581. 35. Griffin E, McKenna K, Worrall L. Stroke education materials on the world wide web: an evaluation of their quality and suitability. Top Stroke Rehabil 2004;11:29-40.
N. SHARMA ET AL. 36. Center for Health Care Strategies. Health literacy fact sheets. Available at: http://www.chcs.org/publications3960/ publications_show.htm?doc_id5291711. Accessed March 8, 2013. 37. National Institutes of Health. How to write easy to read health materials. Available at: http://www.nlm. nih.gov/medlineplus/etr.html. Accessed March 8, 2013.
Background: Patients and carers increasingly access the Internet as a source of health information. Poor health literacy is extremely common and frequently limits patient’s comprehension of health care information literature. We aimed to assess the readability of online consumer-orientated stroke information using 2 validated readability measures. Methods: The 100 highest Google ranked consumer-oriented stroke Web pages were assessed for reading difficulty using the Flesch–Kincaid and Simple Measure of Gobbledygook (SMOG) formulae. Results: None of the included Web pages complied with the current readability guidelines when readability was measured using the gold standard SMOG formula. Mean Flesch–Kincaid grade level was 10.4 (95% confidence interval [CI] 9.97-10.9) and mean SMOG grade 12.1 (95% CI 11.7-12.4). Over half of the Web pages were produced at graduate reading levels or above. Not-for-profit Web pages were significantly easier to read (P 5 .0006). The Flesch–Kincaid formula significantly underestimated reading difficulty, with a mean underestimation of 1.65 grades (95% CI 1.49-1.81), P , .0001. Conclusions: Most consumer-orientated stroke information Web sites require major text revision to comply with readability guidelines and to be comprehensible to the average patient. The Flesch–Kincaid formula significantly underestimates reading difficulty, and SMOG should be used as the measure of choice. Key Words: Stroke—readability—Internet—online—literacy. Ó 2013 by National Stroke Association
Introduction Stroke is a common condition affecting over 150,000 people in the United Kingdom per year.1 It is a major cause of chronic neurologic disability and may be considered to be the most common cause of complex disability in the United Kingdom.2 Patient education is important in stroke and helps patients, their families, and carers to manage the long-term effects of a stroke and its treatment. Although patients receive information from health care staff, many From the Department of Gerontology, Royal Liverpool and Broadgreen Hospitals NHS Trust, Liverpool, Merseyside, UK. Received May 13, 2013; revision received October 20, 2013; accepted November 19, 2013. Sources of funding: None. Disclosures: None. Address correspondence to Nikhil Sharma, MBChB, MRCP, Department of Gerontology, Royal Liverpool and Broadgreen Hospitals NHS Trust, Prescot St, Liverpool, Merseyside L7 8XP. E-mail: [email protected]. 1052-3057/$ - see front matter Ó 2013 by National Stroke Association http://dx.doi.org/10.1016/j.jstrokecerebrovasdis.2013.11.017
seek further information independently, with the Internet being one of the most commonly used sources. There is a huge volume of stroke-related information available on the Internet, with a Google search for ‘‘stroke’’ returning 228 million Web pages. It is well recognized that the general population is increasingly turning to the Internet for health care information. In the United States, 82% of adults use the Internet,3 and of these, 72% sought online health information in the past year, with a total of 59% of American adults having browsed online for health information in the past year. Most of these users (77%) began their search for health information with an online search engine such as Google.4 Health information provision is 1 major function of the Internet with over 80% of American Internet users searching online for health information at some time. This is more than have been online to check the news (78%), buy products (71%), watch online video content (71%), or use social networking sites (69%).3 There is also evidence that those with chronic conditions and disability, such as stroke survivors, are more likely to search for health information than others.5
Journal of Stroke and Cerebrovascular Diseases, Vol. -, No. - (---), 2013: pp 1-6
1
N. SHARMA ET AL.
2
No peer-reviewed published estimates of UK Internet use for health care information or specifically by stroke patients are available. However, the number of households in the United Kingdom with Internet access has risen from around 42% to 80% in the past decade.6 Although the Internet is a popular and convenient source of consumer-oriented health care information, these resources are only useful if the consumer is able to comprehend the information presented. This is even more pertinent in stroke patients, where the capacity for language comprehension is often affected. Health literacy is ‘‘the constellation of skills, including the ability to perform basic reading and numerical tasks, required to function in the health care environment’’ as defined by the American Medical Association’s Ad Hoc Committee on Health Literacy in 1999.7 Similarly, a 2011 systematic review defined health literacy as ‘‘the degree to which individuals can obtain, process, and understand the basic health information and services they need to make appropriate health decisions.’’ Subsequently, it demonstrated that poor health literacy is associated with increased hospitalization, emergency care use, and among older people, poorer overall health status and higher mortality.8 Reading ability is an integral component of health literacy. Those with inadequate literacy have difficulty reading and understanding material written at the sixth-grade level (age 11-12 years), whereas those with marginal literacy have difficulty comprehending information written at the 10th-grade level (age 15-16 years).9 A national survey has shown that around 1 in 6 people in the United Kingdom have levels of general literacy below that expected of an 11-year old.10 A study of nearly 8000 older patients in the United Kingdom showed that almost a third had difficulties reading and understanding basic written health information. This study also demonstrated that those with lower health literacy had significantly higher mortality rates.11 A pooled analysis of 85 American studies representing 31,129 subjects showed a prevalence of poor health literacy of 26% and also demonstrated a significant association between older age and low health literacy with a third of the population older than 50 years having inadequate heath literacy skills.12 These findings have been supported by data published by the US National Center for education statistics, which undertook its National Assessment of Adult Literacy in 2003. Again adults in the oldest age group (older than 65 years) had the lowest levels of health literacy.13 In view of this, health literacy is a particular public health concern in stroke medicine as the incidence of stroke increases with advancing age. To date, there is only 1 small published study that has investigated health literacy in stroke patients. This Australian study of 57 patients found stroke patients to have a mean reading level of seventh to eighth grade,
although those with combined aphasia, as would be expected, read at a significantly lower level.14 In view of this high level of inadequate health literacy, the US Department of Health and Human Services (USDHHS) recommends that patient-orientated literature should be written at or below the sixth-grade level (age 11-12 years) and has developed an action plan to promote patient literacy.15 The USDHHS literacy classification system categorizes material as ‘‘easy to read’’ only if written at or below a sixth-grade level. Material between the 7th- and 9thgrade level is ‘‘average’’ and above the 9th-grade level is regarded as difficult. The United Kingdom currently has no quantitative guidelines regarding levels of readability for patient orientated literature; hence, the USDHHS guidelines outlined above have been taken as the international gold standard for the purpose of this study. Ours is the only study of readability in consumerorientated online general stroke literature of which we are aware. However, a recent study of 51 Web sites pertaining to patients with speech and language difficulties after a stroke showed that only 6% of Web sites had a reading level suitable for the sixth grade or below.16
Methods Sampling Method We analyzed the text content of the 100 highest ranked consumer-orientated Web pages, selected on the basis of Google search ranking and consumer orientation. A full description of the sampling method is given below. We performed a Google search on January 31, 2013, for the term ‘‘Stroke.’’ This returned about 228,000,000 Web page results. Google was selected as it is the most popular search engine, accounting for more than 90% of UK consumer health-related search activity in 2012, with a number of other search engines making up the remaining 10%.17 Starting with the highest ranking nonsponsored Web page link, each of the Web page results returned by this Google search was reviewed by a single researcher (A.T.). Initially, a decision was made as to whether the Web page contained consumer-orientated health care information (aimed primarily at lay persons) or was aimed at health care professionals (journal articles, textbooks, and so on). Only consumer-orientated Web pages containing health care information regarding stroke were included for further assessment. Each consumer-orientated Web page was then classified by the following criteria: 1. Commercial or not-for-profit—Not-for-profit was defined as a Web site of a registered charity, educational institution, or governmental organization.
READABILITY ASSESSMENT OF ONLINE STROKE INFORMATION
2. Web site country of origin—classified as Europe, United States, or other. The classification for each Web page was determined from the ‘‘about us’’ section of the hosting Web site. This process was repeated for each Google search Web page result in turn, until 100 unique Web pages containing consumer-orientated health care information were identified. A sample size of 100 Web pages was selected as it was felt unlikely that most stroke survivors would click on more than 100 consecutive links from a Google search. Where duplicate Web pages were identified, only the highest ranking Web page was included in the analysis. Unique Web pages from the same Web site were included, providing the information contained in the Web page did not duplicate previously analyzed text content. A second researcher (N.S.) independently reviewed the Web pages to confirm consumer orientation and to screen for duplicate results.
Readability Assessment Full Web page body text content was extracted into Microsoft Word 2007, and readability was assessed using 2 validated measures: the Flesch–Kincaid Grade Level and Flesch–Kincaid and Simple Measure of Gobbledygook (SMOG). These represent the most commonly used readability measure (Flesch–Kincaid) and the most exacting (SMOG). Titles, subtitles, references, Web links, and advertising text were excluded from the analysis, with only body text and bullet point text included. The Flesch–Kincaid Grade Level Formula is well established and conveniently is often located in wordprocessing packages such as MS Word. The Flesch– Kincaid formula gives a score that estimates the grade level (number of years of education) required to understand the text.18 Despite the formula’s widespread use, validation studies have shown that it may be of limited accuracy when grading low literacy-level materials.19 The formula is expressed as [(.39 3 ASL) 1(11.8 3 ASW) 2 15.59], where ASL is the average sentence length (number of words divided by number of sentences) and ASW is the average syllables per word (number of syllables divided by number of words).18 SMOG is a more exacting measure of readability and accurately grades for the grade level required for complete text comprehension (whereas the Flesch–Kincaid formula grades for less than complete comprehension), demonstrating strong correlation (r 5 .985) with the required reading level in validation studies.20 The SMOG formula, in brief terms, consists of counting the words of 3 syllables or more in three 10-sentence samples, calculating the count’s square root and adding 3 to obtain the grade level.20 The SMOG and Flesch–Kincaid grade levels were measured using an online calculator from online-utili-
3 21
ty.org. For the purpose of this study, the reading grade level required for complete comprehension of the text material (ie, the SMOG grade level) was taken as the gold standard reliability measure.
Statistical Analysis Reading level data were categorized in line with USDHHS standards and analyzed in GraphPad Prism 4.0 (GraphPad Software, San Diego, CA). The D’Agostino and Pearson omnibus normality test demonstrated that SMOG and Flesch–Kincaid Grade Level data followed parametric distributions. As such, the paired t test was used to compare paired SMOG and Flesch–Kincaid Grade Level values. The unpaired t test and 1-way analysis of variance were used to compare SMOG grade levels in Web pages of varying commercial nature (commercial and not-for-profit) and nationality (United Kingdom, European, and other), respectively. Relationships between Web page ranking, word count, and readability were analyzed using Pearson correlation. Statistical significance was set at .05.
Results Our analysis included 100 unique Web pages. The mean Web page word count was 747 words (95% confidence interval [CI] 551-931). Of the 100 Web pages analyzed, 44% were run as commercial interests and 56% on a not-for-profit basis. More than half (53%) of Web sites were based in the United States, with 30% based in Europe and 17% based in other countries. Table 1 shows the distribution of Web pages by USDHHS classification and grade level for the Flesch– Kincaid and SMOG formulae. Mean Flesch–Kincaid grade level was 10.4 (95% CI 9.9710.9) and mean SMOG grade 12.1 (95% CI 11.7-12.4). A strong and significant correlation was seen between the grade levels produced by the formulae (Pearson r 5 .952, P # .0001). The Flesch–Kincaid formula significantly underestimated reading difficulty when compared with the gold standard SMOG formula, with a mean underestimation of 1.65 grades (95% CI 1.49-1.81, P , .0001). No significant correlation was observed between SMOG readability and search engine ranking (r 5 .035, P 5 .73). A weak but significant correlation was observed between SMOG readability and Web page word count (r 5 .303, P 5 .002). Not-for-profit Web pages were significantly easier to read than commercial Web pages, with a mean commercial SMOG grade of 12.8 (95% CI 12.3-13.3) and a mean not-for-profit SMOG grade of 11.5 (95% CI 11.0-12.0, P 5 .0006). No significant difference in SMOG grade level was observed between Web pages taken from Web sites based
N. SHARMA ET AL.
4
Table 1. Number of Web sites in each USDHHS classification group according to SMOG and Flesch–Kincaid readability formulae USDHSS classification
Grade level
SMOG (% Web pages)
Flesch–Kincaid (% Web pages)
Easy Average Difficult Very difficult
4th-6th grade 7th-9th grade 10th-11th grade 12th grade and above
0 12 37 51
5 44 23 28
Abbreviations: SMOG, Simple Measure of Gobbledygook; USDHHS, US Department of Health and Human Services.
in different countries. The mean SMOG grade for US Web pages was 12.1 (95% CI 11.5-12.6), the mean SMOG grade for European Web pages was 12 (95% CI 11.3-12.7), and the mean SMOG grade for Web pages of other nationality was 12.3 (95% CI 11.5-13.2, P 5 .801).
Conclusions The Internet is rapidly growing and is a highly convenient source of information for patients. Irrespective of its quality, this information can only be helpful if patients are able to understand it. We found that most consumer-orientated stroke information Web pages were written at the 12th-grade level or above and that none complied with the USDHHS maximum recommended sixth-grade level when readability was measured with SMOG. This was in keeping with similar studies looking at readability in other areas of health care, including Parkinson disease,22 mental health,23 epilepsy,24 orthopedic surgery,25 fibromyalgia,26 and warfarin therapy.27 Similarly, excessive levels of reading difficulty have long been noted in printed patient information.28 The readability levels of the Web pages analyzed contrast sharply with the observed reading abilities of patients, with the average American adult reading between the seventh and eighth grade,13 inferring that only 12% of Web pages included in this study would be fully comprehensible to the average adult when measured by SMOG. Over half the Web pages (51%) were written at graduate level and above (SMOG grade level . 12). There was a significant difference in readability between commercial and not-for-profit Web sites. Interestingly not-for-profit Web sites were easier to read than commercial sites. This is in contrast to a recent similar study performed looking at Parkinson disease Web sites.22 The reasons for this are unclear; previously, it has been hypothesized that the increased ease of reading of commercial Web pages was because of different writing styles in each of these sectors, with commercial authors and editors aiming to ensure the widest possible commercial audience by minimizing reading difficulty.22 Our current study does not support this hypothesis. Similar studies on patient information for warfarin therapy29 and inflammatory bowel disease30 have shown no statistically significant difference between different Web
site provider categories. Further research is needed in this area. Readability levels were similarly poor in Web pages produced in the United States, Europe, or other countries. This suggests that readability is likely to be a limiting factor in the ability of patients to comprehend locally produced stroke information in most developed countries. There was only a weak correlation between SMOG score and word count and no significant correlation with search engine ranking. This is the first study to look at the readability of consumer-orientated online general stroke information on multiple Web sites. This study used the SMOG formula that accurately grades for complete comprehension of reading materials. The Flesch–Kincaid formula appears attractive because it is incorporated into commercial word-processing software. However, compared with the gold standard SMOG readability measure, Flesch–Kincaid significantly underestimated reading difficulty, presumably, because it grades for less than complete comprehension of written material. We recommend that SMOG should be the preferred measure of readability when evaluating consumer-orientated health care material. Our study is limited in that we did not undertake an assessment of the quality, accuracy, or relevance of the information presented. As such, we are unable to comment as to whether information quality is preserved in Web pages written at low literacy levels. However, although providing good quality information is of critical importance, its positive impact is negligible if that information is presented in an unintelligible format. We have identified 1 recent study incorporating strokespecific scoring systems for quantifying Web site quality and readability.16 The quality of Web-based information on stroke was found to be highly variable and did not correlate with readability. This trend is been seen across Web-based information for many other conditions.31-34 By reporting the discrepancy between levels of Web site readability and population estimates of literacy, we hope to encourage compliance with readability guidelines and, thus, improve accessibility to online information for stroke patients. We hope to specifically encourage those in clinical practice to consider readability as an important factor when developing patient literature. Studies have suggested involving patients in the design process of Web sites can be beneficial as they
READABILITY ASSESSMENT OF ONLINE STROKE INFORMATION 34,35
will ultimately be the target audience. Consideration of consumer needs in this way may act as an additional measure to ensure readability. Web site editors should introduce minimum readability policies based on USDHHS guidelines and also consider routine monitoring of content reading difficulty with SMOG in an attempt to increase the accessibility and ease of comprehension of online consumer-orientated stroke information. There is useful guidance to this end available from the Centre for Health Care Strategies36 and the National Institutes of Health.37 In summary, most consumer-orientated Stroke information Web sites require major text revision to comply with readability guidelines and to be comprehensible to the average stroke survivor.
References 1. Townsend N, Wickramasinghe K, Bhatnagar P, et al. Coronary heart disease statistics 2012 edition. London: British Heart Foundation 2012. 2. Adamson J, Beswick A, Ebrahim S. Stroke and disability. J Stroke Cerebrovasc Dis 2004;13:171-177. 3. Trend data (adults). Pew Internet & American Life Project. Available at: http://www.pewinternet.org/Trend-Data-% 28Adults%29/Online-Activites-Total.aspx. 2012. Accessed March 1, 2013. 4. Fox S, Duggan M. Health online 2013. Pew Internet & American Life Project. Available at: http://www.pewinternet. org/Reports/2013/Health-online/Summary-of-Findings. aspx 15 Jan 2013. Accessed January 19, 2013. 5. Bundorf MK, Wagner TH, Singer SJ, et al. Who searches the Internet for health information? HSR: Health Services Research 2008;41:819-836. 6. Office for National Statistics. Internet access—households and individuals 2012. Available at: http://www. ons.gov.uk/ons/dcp171778_275775.pdf. August 2012, Accessed January 19, 2013. 7. Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, American Medical Association. Health literacy: report of the Council on Scientific Affairs. JAMA 1999;281:552-557. 8. Berkman ND, Sheridan SL, Donahue KE, et al. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011;155:97-107. 9. Gazmararian JA, Baker DW, Williams MV, et al. Health literacy among Medicare enrolees in a managed care organization. JAMA 1999;281:545-551. 10. Williams J, Clemens S, Oleinikova J, Tarvin Kl. The skills for life survey: a national needs and impact survey of literacy, numeracy and ICT skills. London: Department for education and skills 2003. 11. Bostock S, Steptoe A. Association between low functional health literacy and mortality in older adults: longitudinal cohort study. BMJ open 2012;344:e1602. 12. Paasche-Orlow MK, Parker RM, Gazmararian JA, et al. The prevalence of limited health literacy. J Gen Intern Med 2005;20:175-184. 13. Kutner M, Greenberg E, Jin Y, Paulsen C. The Health Literacy of America’s Adults: results from the 2003 National Assessment of Adult Literacy (NCES 2006–483). Washington, DC: US Department of Education, National Center for Education Statistics 2006.
5
14. Hofmann T, McKenna K. Analysis of stroke patients’ and carers’ reading ability and the content and design of written materials: recommendations for improving written stroke information. Patient Educ Counsel 2006; 60:286-293. 15. National action plan to improve health literacy. Washington, DC: US Department of Health and Human Services, Office of Disease Prevention and Health Promotion 2010. 16. Surman R, Bath PA. An assessment of the quality of Information on stroke and speech and language difficulty websites. J Information Sci 2013;39:113-125. 17. Healthcare WebWatch 2012: an analysis of web activity in the health and medical sector. Private Healthcare UK website. Available at: http://www.privatehealth.co.uk/ aboutus/webwatch-2012. Accessed January 19, 2013. 18. Flesch R. How to write in plain English. New York: Harper and Row 1979. 19. McLaughlin GH. Temptations of the Flesch. Instr Sci 1974;2:367-384. 20. McLaughlin GH. SMOG grading: a new readability formula. J Reading 1969;12:639-646. 21. Tests document readability, readability calculator. Online utility-org website. Available at: http://www.online-utility. org/english/readability_test_and_improve.jsp. Accessed February 6, 2013. 22. Fitzsimmons PR, Michael BD, Hulley JL, Scott GO. A readability assessment of online Parkinson’s disease information. J R Coll Physicians Edinb 2010;40: 292-296. 23. Kalk NJ, Pothier DD. Patient information on Schizophrenia on the internet. Psychiatrist 2008;32:409-411. 24. Elliott JO, Shneker BF. A health literacy assessment of the epilepsy.com website. Seizure 2009;18:434-439. 25. Sabharwal S, Badarudeen S, Kunju SU. Readability of online patient education materials from the AAOS web site. Clin Orthop Relat Res 2008;466:1245-1250. 26. Daraz L, MacDermid JC, Wilkins S, et al. The quality of websites addressing fibromyalgia: an assessment of quality and readability using standardised tools. BMJ Open 2011;1:e000152. 27. Nasser S, Mullan J, Bajorek B. Assessing the quality, suitability and readability of internet-based health information about warfarin for patients. Austr Med J 2012; 5:194-203. 28. Davis TC, Crouch MA, Wills G, et al. The gap between patient reading comprehension and the readability of patient education materials. J Fam Pract 1990;31: 533-538. 29. Estrada CA, Hryniewicz MM, Higgs VB, et al. Anticoagulant patient information material is written at high readability levels. Stroke 2000;31:2966-2970. 30. van der Marel S, Duijvestein M, Hardwick JC, et al. Quality of web based information on inflammatory bowel diseases. Inflammatory Bowel Dis 2009;15: 1891-1896. 31. Bouchier H, Bath PA. Evaluation of web sites that provide information on Alzheimer’s disease. Health Informatics J 2003;9:17-31. 32. Bath PA, Bouchier H. Development and application of a tool designed to evaluate web sites providing information on Alzheimer’s disease. J Information Sci 2003; 29:279-297. 33. Harland J, Bath PA. Assessing the quality of web sites providing information on multiple sclerosis: evaluating tools and comparing sites. Health Information J 2007; 13:207-221.
6 34. Meric F, Bernstam EV, Mirza NQ, et al. Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of web sites. Br Med J 2002; 324:577-581. 35. Griffin E, McKenna K, Worrall L. Stroke education materials on the world wide web: an evaluation of their quality and suitability. Top Stroke Rehabil 2004;11:29-40.
N. SHARMA ET AL. 36. Center for Health Care Strategies. Health literacy fact sheets. Available at: http://www.chcs.org/publications3960/ publications_show.htm?doc_id5291711. Accessed March 8, 2013. 37. National Institutes of Health. How to write easy to read health materials. Available at: http://www.nlm. nih.gov/medlineplus/etr.html. Accessed March 8, 2013.
