e194(1) C OPYRIGHT 2014
BY
T HE J OURNAL
OF
B ONE
AND J OINT
S URGERY, I NCORPORATED
the
Orthopaedic forum
Quality of Online Pediatric Orthopaedic Education Materials Daniel P. Feghhi, BA, Daniel Komlos, PhD, Nitin Agarwal, BS, and Sanjeev Sabharwal, MD, MPH
Background: Increased availability of medical information on the Internet empowers patients to look up answers to questions about their medical conditions. However, the quality of medical information available on the Internet is highly variable. Various tools for the assessment of online medical information have been developed and used to assess the quality and accuracy of medical web sites. In this study we used the LIDA tool (Minervation) to assess the quality of pediatric patient information on the AAOS (American Academy of Orthopaedic Surgeons) and POSNA (Pediatric Orthopaedic Society of North America) web sites. Methods: The accessibility, usability, and reliability of online medical information in the “Children” section of the AAOS web site and on the POSNA web site were assessed with use of the LIDA tool. Flesch-Kincaid (FK) and Flesch Reading Ease (FRE) values were also calculated to assess the readability of the pediatric education material. Results: Patient education materials on each web site scored in the moderate range in assessments of accessibility, usability, and reliability. FK and FRE values indicated that the readability of each web site remained at a somewhat higher (more difficult) level than the recommended benchmark. Conclusions: The quality and readability of online information for children on the AAOS and POSNA web sites are acceptable but can be improved further. Clinical Relevance: The quality of online pediatric orthopaedic patient education materials may affect communication with patients and their caregivers, and further investigation and modification of quality are needed.
Peer Review: This article was reviewed by the Editor-in-Chief and one Deputy Editor, and it underwent blinded review by two or more outside experts. The Deputy Editor reviewed each revision of the article, and it underwent a final review by the Editor-in-Chief prior to publication. Final corrections and clarifications occurred during one or more exchanges between the author(s) and copyeditors.
Historically, patients had relied on office visits to seek answers regarding their medical problems. However, the dynamics of the physician-patient relationship have changed, in large part
because of greater access to and use of the Internet worldwide, including in the U.S.1-3. A survey conducted as part of the Pew Internet Project in May 2013 revealed that 85% of U.S. adults
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. None of the authors, or their institution(s), have had any financial relationship, in the thirty-six months prior to submission of this work, with any entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. Also, no author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.
J Bone Joint Surg Am. 2014;96:e194(1-7)
d
http://dx.doi.org/10.2106/JBJS.N.00043
e194(2) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
used the Internet4. Of those users, 72% stated that they had sought online health information within the prior year4. Health-care-related information available on the Internet is highly variable in terms of quality and accuracy 5. Along with easy access to medical information comes the risk that patients will be misinformed by it6. To prevent this, Internet-based medical information should be of high quality and easily comprehensible by the majority of readers. Readability, a metric that indicates the ease with which text can be read and understood, is a major factor affecting the ability of a patient to utilize health information7. The health literacy of an individual and the necessary readability of patient education material are closely related. Health literacy is defined by the U.S. Department of Health and Human Services (HHS) as the capacity to obtain, interpret, and understand basic medical information and services and the competence to use such information and services to enhance health8-10. Previous studies have focused on assessing the readability of patient education materials. In particular, a 2008 study by Badarudeen and Sabharwal found that patient education materials from the AAOS (American Academy of Orthopaedic Surgeons) and POSNA (Pediatric Orthopaedic Society of North America) web sites had a mean Flesch-Kincaid (FK) grade level of 8.911. However, readability is an imperfect tool for interpreting an individual’s ability to understand provided materials. It only accounts for the number of syllables in a word and the sentence structure. The impact that supporting figures, tables, and graphs have on the individual’s understanding of the online material is not taken into consideration. Another method of assessing the quality of online patient education material is needed to better evaluate the level at which such materials are presented to consumers of medical information. A widely used validation instrument for health-care web sites, the LIDA tool (Minervation), has been developed to assess the quality—including accessibility, usability, and reliability—of patient education resources12. Various web sites containing patient education information have been evaluated with use of this tool13-21. The LIDA tool contains forty-one questions that yield a percentage grade for each of the three quality categories12. A score of >90% is considered a good result, 50% to 90% is moderate, and 90%, whereas the usability and accessibility scores would be between 50% and 90%. Furthermore, we anticipated that the two orthopaedic web sites would have LIDA scores similar to those reported for sites in other medical specialties and that readability would have improved further since the 2008 study. Materials and Methods We did not seek institutional review board approval because of the open-access nature of the utilized web sites. The AAOS web site provides a link to information for patients, and one category in the patient education portion is entitled Children. The Children web page lists 104 distinct articles within eleven sections: General, Shoulder & Elbow, Hip & Thigh, Knee & Lower Leg, Foot & Ankle, Neck & Back, Treatments & Surgeries, Safety, Sports Injury Prevention, 23 Health & Fitness, and Patient Stories . Six evaluators applied the LIDA tool to the AAOS web page (http:// orthoinfo.aaos.org/menus/children.cfm) and to the POSNA web page (http:// www.posna.org/parents/articles/orthoInfo.cfm) in May 2013. Three of the evaluators (D.P.F., D.K., and N.A.) were senior medical students, and the remaining three were a business professional, a financial advisor, and a high school teacher. Each evaluator was blinded to the scores assigned by the other five evaluators. Each was sent a PDF file containing an uncompleted LIDA tool and instructed to read the first two pages to obtain a general understanding of the nature of the LIDA tool and what it is designed to measure. Each was then instructed to access the web pages and critically evaluate them by following the instructions in the LIDA tool. Evaluators did not share their completed PDF file or communicate with each other during the evaluations. The readability of the AAOS online materials was then assessed by creating eleven documents, each containing the text of the articles within one subsection of the Children portion of the web site. Content unrelated to patient education (e.g., copyright information, outside links, disclaimers, date stamps, authorship, and source information) was eliminated from the document. Illustrations, tables, and citations were also eliminated so that the readability score would represent only the relevant text. An article was excluded if the content was provided in a non-text format such as illustrations or multimedia. The readability analysis of the included text was performed by two of the authors (D.P.F. and N.A.) with use of Readability Studio software (Professional Edition Version 2012.1 for Windows; Oleander Software, Vandalia, Ohio). A similar process was performed for the POSNA web site. The FRE value of a text passage equals 206.835 – 1.015 · (total words/ total sentences) – 84.6 · (total syllables/total words). The value rates the
e194(3) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
Fig. 1
Bar graph of the LIDA scores of the AAOS and POSNA web sites. The mean and standard deviation of the scores assigned by the six evaluators are shown. *P < 0.005 for the comparison between the web sites.
readability of the text on a 100-point scale, with a lower score indicating that the text is more difficult to understand. A score of 90 to 100 corresponds to text that is easily understood by a typical eleven-year-old. In general, authors of consumer information should aim for an FRE score of 60 to 70, which is what a typical thirteen to fifteen-year-old student would 24,25 easily understand . The FK grade level of a text passage equals (0.39 · mean number of words per sentence) 1 (11.8 · mean number of syllables per word) – 15.59. The resulting value is an approximation of the number of years of schooling that 24 would be required for someone to understand the text . A function in Microsoft Word (Redmond, Washington) can also be used to calculate the FK value and FRE grade level.
FK and FRE values were calculated by two evaluators (D.P.F. and N.A.). The mean values were calculated, and the interobserver variability was determined (Table I).
Source of Funding There was no external funding for this study.
Results The quality (including accessibility, usability, and reliability) of the pediatric patient education section of each web site according to the LIDA tool is shown in Figures 1 and 2. The mean
Fig. 2
Box-and-whisker plot of the LIDA scores of the AAOS and POSNA web sites. The boxes show the 25th, 50th, and 75th percentiles of the scores assigned by the six evaluators, and the whiskers show the minimum and maximum values. *P < 0.005 for the comparison between the web sites.
e194(4) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
TABLE I FK and FRE Values of Pediatric Orthopaedic Patient Education Materials According to Section on the AAOS Web Site FK
FRE
Assessment of 103 articles by observer 1 (D.P.F.) General
9.4
50
Foot & Ankle
8.8
64
Knee & Lower Leg Hip & Thigh
8.5 8.3
65 66
Neck & Back
9.1
62
Shoulder & Elbow
9.0
51
Treatments & Surgeries
8.5
65
Safety
8.9
63
10.2
53
Sports Injury Prevention Health & Fitness Patient Stories
9.9
55
11.8
50
Mean
9.3
58.5
Standard deviation
1.0
6.7
Assessment of 103 articles by observer 2 (N.A.) General
9.0
53
Foot & Ankle
8.1
60
Knee & Lower Leg
8.0
62
Hip & Thigh
8.8
60
Neck & Back
8.8
57
Shoulder & Elbow Treatments & Surgeries
8.5 8.2
54 63
Mean
8.4
58.4
Standard deviation
0.4
3.9
8.8
59.4
Recalculated mean of the 52 articles used in the 2008 study
TABLE II Mean LIDA Scores of the AAOS and POSNA Web Sites AAOS LIDA (%)
POSNA LIDA (%)
P Value
Accessibility
77.67
66.50
0.004
Usability
84.33
72.50
0.15
Reliability
84.50
86.67
0.363
overall accessibility score (and standard deviation of the scores of the six evaluators) was rated as 78% ± 2.5% (range, 75% to 81%) for the AAOS site and 66% ± 4.8% (range, 60% to 73%) for the POSNA site (p < 0.005). The mean usability was 84% ± 4.8% (range, 77% to 91%) for the AAOS site and 73% ± 5.1% (range, 65% to 80%) for the POSNA site (p = 0.15). The mean reliability was 85% ± 5.6% (range, 77% to 92%) for the AAOS site and 87% ± 4.7% (range, 80% to 92%) for the POSNA site (p = 0.36) (Table II).
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
Although the structure and design of the POSNA web site26 differed from that of the AAOS site (leading to the observed differences in quality as assessed with the LIDA tool), the POSNA web site contains links to materials on the patient education portion of the AAOS web site (http://www.orthoinfo. aaos.org). As a result, the FK and FRE scores for the material on the two web sites were identical. We included 103 of the 104 AAOS articles in the present study; the remaining article was omitted because its content was provided in a video format23. The overall FK readability score, 8.8, was somewhat higher than the recommended sixth-grade level27, and it was nearly identical to the value of 8.9 reported in the 2008 study. The mean FRE score was 59.4, similar to the 2008 reported value of 57.3. Mean FK and FRE values for material related to each core orthopaedic area and for the Safety, Sports Injury Prevention, Health & Fitness, and Patient Stories sections are shown in Table I. The 2008 study by Badarudeen and Sabharwal included fifty-two articles. In that study, FK and FRE values for each article were calculated with use of a function in Microsoft Word, rather than with Readability Studio software as in the present study. However, the overall FK and FRE values in the two studies were nearly identical. Discussion The ease with which material on health-related web sites can be read, as assessed by “readability scores,” is not the only factor that affects the ability of patents and their caregivers to access online health information. Rather, there is a complex interplay involving readability and other factors such as accessibility, usability, and reliability. Accessibility of a web site (as determined by adherence to W3C, Bobby, and other accessibility standards) is now mandated in the U.S. and many other countries. The aim of such standards is to attempt to make web sites easier for public viewing, accessible on appropriate devices (including mobile devices and computers running various operating systems), and available for use by individuals with visual and other impairments. Usability refers to the ability of users to obtain the information they seek, and it assesses the ease of navigating a web site. Factors such as clear home page and overall web site design, clear text and context-appropriate color, consistency, and ease of navigation are all assessed to categorize the usability of a web site. The reliability of a web site is critically important in sites presenting health-carerelated information. Reliable health-care information is information that is current, evidence-based, and unbiased. For example, one strategy for improving the reliability of a health-care web site involves citing various sources of health information and disclosing any conflicts of interest that could be perceived as representing a potential bias in the selection of those resources. Our LIDA assessment yielded a mean accessibility score of 78% for the AAOS web site and 66% for the POSNA site. Although the latter was significantly lower, both accessibility scores were within the moderate range. Page setup, access restrictions, and HTML code lifetime were accessibility factors evaluated by the LIDA tool. The AAOS patient education section was fully accessible without any registration requirements. The POSNA web site could also generally be accessed freely,
e194(5) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
TABLE III Mean LIDA Scores of Various Medically Oriented Online Patient Education Resources
Specialty Orthopaedics
Mean LIDA Score (%) 79
Reference No. Present study
Cardiology
73
19
Coloproctology
62.50
17
Coloproctology
85.60
14
Interventional radiology
67
21
Otolaryngology
61.30
29
Otolaryngology
69.30
18
Vascular surgery
65
15
Vascular surgery
73
28
Urology
66
16
Aortic aneurysm and endovascular treatment
70.20
20
although the scoliosis handbook (accessed via a link in the Parents & Patients section of the site) required a free subscription to access the material. Overall, patient information on both web sites was provided in a black-and-white format, and the font was Arial size 8. Thus, colorblind individuals would be able to access the information without difficulty, but a larger font might offer a more favorable user experience to the visually impaired. The mean usability rating was 84% for the AAOS web site and 73% for the POSNA site. Evaluators of the POSNA web site noted occasional difficulty in ascertaining which section they were viewing within the web site. Also, the POSNA color scheme was rated as less engaging than that of the AAOS site. In general, both web sites were rated well with respect to aspects of functionality; they performed well regardless of the Internet browser, provided effective search and browsing facilities, and did not require third-party plug-ins other than perhaps a common and freely available PDF reader. Additionally, both sites were clear, well structured, and easy to navigate. The three aspects that were largely responsible for the lower usability score of the POSNA web site compared with the AAOS site all involved the ability to engage the user: the interactive qualities of the site, ability of users to personalize their experience, and integration of non-textual media. Reliability was the one LIDA measure on which the POSNA web site scored slightly (2%) higher than the AAOS site, although this difference did not reach significance. Reliability was the aspect that earned the highest LIDA score, approaching “good” (>90%) for both sites. Both sites posted current and unbiased health information supported with links to outside agencies, and both revealed the identity of the authors contributing to the provided text. However, the source of funding for the POSNA and AAOS web sites was scored less favorably by the evaluators, as was the frequency with which the provided information was updated. The latter could be
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
improved with a sidebar on the home page that provides the web site user with an alert regarding any new modifications made to patient information on the site. Currently, neither web site has a section for user feedback or comments that can be viewed by other users. Having a section where users can discuss their experiences with others would allow patients and parents to learn from each other’s unique experiences with orthopaedic conditions. We also compared the LIDA scores of the POSNA and AAOS online materials with those reported for web sites in other medical fields. However, the majority of the previous investigators did not evaluate individual specialty-specific web sites. Rather, keywords specific to a particular field were entered into popular search engines, and the first ten to fifty web sites pertaining to a specific specialty were then evaluated with the LIDA tool. The LIDA scores of the pediatric orthopaedic material on the AAOS web site were comparable with, and in some instances even better than, those of certain other medical specialties with regard to accessibility, usability, and reliability (Table III)14-19,20,21,28,29. Nevertheless, given the differences in methodology between the present study and some of the previous ones, further investigation will be required to compare the quality of online patient education materials between orthopaedic and other medical specialty web sites. Pediatric information on the AAOS and POSNA web sites had a mean FK value of 8.8, which is higher than the 6th-grade
TABLE IV LIDA Scores Assigned by the Six Users* Accessibility (%)
Usability (%)
Reliability (%)
AAOS LIDA Evaluator 1 Evaluator 2
75 77
85 77
92 77
Evaluator 3
81
81
81
Evaluator 4
75
91
83
Evaluator 5
80
86
90
Evaluator 6
78
86
84
Mean
77.67
84.33
84.50
Std. dev.
2.50
4.80
5.61
SEM
1.03
1.97
2.30
POSNA LIDA Evaluator 1
60
65
90
Evaluator 2
73
80
80
Evaluator 3
65
72
92
Evaluator 4
70
76
82
Evaluator 5
68
70
87
Evaluator 6
63
72
89
Mean
66.50
72.50
86.67
4.76 1.95
5.13 2.10
4.72 1.93
Std. dev. SEM
*SEM = standard error of measurement.
e194(6) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
level recommended for patient education materials27. Furthermore, the FK value was nearly identical to the value (8.9) obtained by Badarudeen and Sabharwal in 200811. Although the reasons for this lack of change are unclear, it does underscore the importance of periodically reevaluating how patientrelated material conforms to NIH (National Institutes of Health) and HHS recommendations27,30. Additionally, our study revealed a recalculated FRE value of 59.4, again similar to the value obtained in the previous 2008 study (57.3). This value is less than the minimum limit of 60, a value that implies easy understanding by most thirteen to fifteen-year-old children, who are typically 8th to 9th graders in the U.S.9,10. The FRE value for the assessed articles in the AAOS pediatrics section would need to be at least 80 (a score for 7th graders or twelve-year-olds), and ideally at least 90 (indicating that it is easily understood by eleven-year-olds, the average age of 6th graders in the U.S.), to conform to accepted standards. As previously noted, the POSNA site contains links to information on the AAOS web site; therefore, the FK and FRE values were identical for the two web sites. Our study has several limitations. First, whereas other investigators evaluated multiple medical web sites utilizing specialtyspecific search terms, we focused on only two pediatric orthopaedic web sites that are frequently accessed and referenced. It is possible that a study analyzing more web sites would have yielded different LIDA scores. Second, a relatively small pool of data was used to determine the mean LIDA score for the two web sites, thus making the results susceptible to selection bias. Third, a larger number of LIDA evaluators with varying degrees of educational backgrounds might have allowed further insights regarding the quality of the pediatric orthopaedic online patient information. However, the degree of variability among the scores of the six evaluators was acceptable for each of the three LIDA sections (Figs. 1 and 2, Table IV). Furthermore, only three of the six LIDA evaluators had a medical background, reducing the bias that could result if such a background influences the way in which a user evaluates the web sites.
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
Although the quality of the patient information on the AAOS and POSNA web sites was comparable to, or better than, that reported for web sites in other medical fields, it can be improved further. Moreover, the readability scores of the pediatric content of the orthopaedic web sites did not change substantially compared with a similar assessment performed five years earlier. As assessing only the readability of online health information may not be the most comprehensive method of assessing the relevance and comprehensibility of online patient education material, other metrics such as the LIDA tool that grade the quality of such web sites may be useful. Future studies should include a larger sample of pediatric orthopaedic web sites, and the patient education information should be evaluated by an even more diverse group of evaluators with a wide spectrum of backgrounds. We would recommend that LIDA evaluators for future studies possess a variety of degrees of medical knowledge and perhaps be of various ethnic and social backgrounds to represent the diversity of online users. Also, having a portion of LIDA evaluators with reading abilities at, or below, the NIH and HHS standards would offer more insight into possible correlations between the LIDA scores (which reflect the quality of the health-related web sites) and the FK and FRE values (which assess the readability of the online content). n
Daniel P. Feghhi, BA Daniel Komlos, PhD Nitin Agarwal, BS Sanjeev Sabharwal, MD, MPH Departments of Orthopaedics (D.P.F., D.K., and S.S.) and Neurological Surgery (N.A.), New Jersey Medical School, Rutgers Biomedical and Health Sciences, 90 Bergen Street, Suite 7300, Newark, NJ 07101. E-mail address for S. Sabharwal: [email protected]
References 1. McMullan M. Patients using the Internet to obtain health information: how this affects the patient-health professional relationship. Patient Educ Couns. 2006 Oct;63(1-2):24-8. Epub 2006 Jan 6. 2. Hart A, Henwood F, Wyatt S. The role of the Internet in patient-practitioner relationships: findings from a qualitative research study. J Med Internet Res. 2004 Sep 30;6(3):e36. Epub 2004 Sep 30. 3. Ilic D. The role of the internet on patient knowledge management, education, and decision-making. Telemed J E Health. 2010 Jul-Aug;16(6):664-9. 4. Fox S, Duggan M. Pew research Internet project. Heath online 2013. 2013 Jan 15. http://www.pewinternet.org/Reports/2013/Health-online.aspx. Accessed 2013 Jul 1. 5. Purcell GP, Wilson P, Delamothe T. The quality of health information on the internet. BMJ. 2002 Mar 9;324(7337):557-8. 6. Chalmers I. Invalid health information is potentially lethal. BMJ. 2001 Apr 21; 322(7292):998. 7. Badarudeen S, Sabharwal S. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res. 2010 Oct;468(10):2572-80. 8. Albright J, de Guzman C, Acebo P, Paiva D, Faulkner M, Swanson J. Readability of patient education materials: implications for clinical practice. Appl Nurs Res. 1996 Aug;9(3):139-43. 9. Baker DW, Parker RM, Williams MV, Clark WS, Nurss J. The relationship of patient reading ability to self-reported health and use of health services. Am J Public Health. 1997 Jun;87(6):1027-30.
10. Nielsen-Bohlman L, Panzer A, Kindig D. Health literacy: a prescription to end confusion. Washington, DC: National Academics Press; 2004. 11. Badarudeen S, Sabharwal S. Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites. J Bone Joint Surg Am. 2008 Jan;90(1):199-204. 12. Tomlin A, Badenoch D. The LIDA instrument. Minervation validation instrument for health care web sites. 2007. http://www.minervation.com/wp-content/ uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf. Accessed 2013 Jul 1. 13. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, Kanouse DE, Muñoz JA, Puyol JA, Lara M, Watkins KE, Yang H, McGlynn EA. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001 May 23-30;285(20):2612-21. 14. Grewal P, Alagaratnam S. The quality and readability of colorectal cancer information on the internet. Int J Surg. 2013;11(5):410-3. Epub 2013 Mar 19. 15. Grewal P, Williams B, Alagaratnam S, Neffendorf J, Soobrah R. Quality of vascular surgery web sites on the Internet. J Vasc Surg. 2012 Nov;56(5):1461-7. Epub 2012 Jul 15. 16. Wong LM, Yan H, Margel D, Fleshner NE. Urologists in cyberspace: a review of the quality of health information from American urologists’ websites using three validated tools. Can Urol Assoc J. 2013 Mar-Apr;7(3-4):100-7. 17. Soobrah R, Clark SK. Your patient information website: how good is it? Colorectal Dis. 2012 Mar;14(3):e90-4.
e194(7) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
18. Goslin RA, Elhassan HA. Evaluating Internet health resources in ear, nose, and throat surgery. Laryngoscope. 2013 Jul;123(7):1626-31. Epub 2013 May 29. 19. Kirthi V, Modi BN. Coronary angioplasty and the Internet: what can patients searching online expect to find? J Interv Cardiol. 2012 Oct;25(5):476-81. Epub 2012 Jun 7. 20. San Noberto EM, Taylor J, Salvador R, Revilla A, Merino B, Vaquero C. The quality of information available on the Internet about aortic aneurysm and its endovascular treatment. Rev Esp Cardiol (Engl Ed). 2011;64(10):869-75. 21. Tavare AN, Alsafi A, Hamady MS. Analysis of the quality of information obtained about uterine artery embolization from the Internet. Cardiovasc Intervent Radiol. 2012 Dec;35(6):1355-62. Epub 2012 Jan 25. 22. Henry SL. W3C web accessibility initiative. Introduction to web accessibility. http://www.w3.org/WAI/intro/accessibility.php. Accessed 2013 Jul 1. 23. The American Academy of Orthopaedic Surgeons. Children. http://orthoinfo. aaos.org/menus/children.cfm. Accessed 2013 Jul 1.
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
24. Juicy Studio. Readability test. http://juicystudio.com/services/readability.php. Accessed 2013 Jul 1. 25. Flesch R. A new readability yardstick. J Appl Psychol. 1948 Jun;32(3):221-33. 26. The Pediatric Orthopaedic Society of North America. Parents & patients. http://www.posna.org/parents/parents.cfm. Accessed 2013 Jul 1. 27. National Institutes of Health. How to write easy-to-read health materials. http://www.nlm.nih.gov/medlineplus/etr.html. 2012. Accessed 2013 Jul 1. 28. Alsafi A, Kaya G, Patel H, Hamady MS. A comparison of the quality of the information available on the Internet on interventional radiology, vascular surgery, and cardiology. J Postgrad Med. 2013 Jan-Mar;59(1):69-75. 29. Muthukumarasamy S, Osmani Z, Sharpe A, England RJ. Quality of information available on the World Wide Web for patients undergoing thyroidectomy: review. J Laryngol Otol. 2012 Feb;126(2):116-9. Epub 2011 Aug 26. 30. Walsh TM, Volsko TA. Readability assessment of Internet-based consumer health information. Respir Care. 2008 Oct;53(10):1310-5.
Copyright of Journal of Bone & Joint Surgery, American Volume is the property of Journal of Bone & Joint Surgery, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
BY
T HE J OURNAL
OF
B ONE
AND J OINT
S URGERY, I NCORPORATED
the
Orthopaedic forum
Quality of Online Pediatric Orthopaedic Education Materials Daniel P. Feghhi, BA, Daniel Komlos, PhD, Nitin Agarwal, BS, and Sanjeev Sabharwal, MD, MPH
Background: Increased availability of medical information on the Internet empowers patients to look up answers to questions about their medical conditions. However, the quality of medical information available on the Internet is highly variable. Various tools for the assessment of online medical information have been developed and used to assess the quality and accuracy of medical web sites. In this study we used the LIDA tool (Minervation) to assess the quality of pediatric patient information on the AAOS (American Academy of Orthopaedic Surgeons) and POSNA (Pediatric Orthopaedic Society of North America) web sites. Methods: The accessibility, usability, and reliability of online medical information in the “Children” section of the AAOS web site and on the POSNA web site were assessed with use of the LIDA tool. Flesch-Kincaid (FK) and Flesch Reading Ease (FRE) values were also calculated to assess the readability of the pediatric education material. Results: Patient education materials on each web site scored in the moderate range in assessments of accessibility, usability, and reliability. FK and FRE values indicated that the readability of each web site remained at a somewhat higher (more difficult) level than the recommended benchmark. Conclusions: The quality and readability of online information for children on the AAOS and POSNA web sites are acceptable but can be improved further. Clinical Relevance: The quality of online pediatric orthopaedic patient education materials may affect communication with patients and their caregivers, and further investigation and modification of quality are needed.
Peer Review: This article was reviewed by the Editor-in-Chief and one Deputy Editor, and it underwent blinded review by two or more outside experts. The Deputy Editor reviewed each revision of the article, and it underwent a final review by the Editor-in-Chief prior to publication. Final corrections and clarifications occurred during one or more exchanges between the author(s) and copyeditors.
Historically, patients had relied on office visits to seek answers regarding their medical problems. However, the dynamics of the physician-patient relationship have changed, in large part
because of greater access to and use of the Internet worldwide, including in the U.S.1-3. A survey conducted as part of the Pew Internet Project in May 2013 revealed that 85% of U.S. adults
Disclosure: None of the authors received payments or services, either directly or indirectly (i.e., via his or her institution), from a third party in support of any aspect of this work. None of the authors, or their institution(s), have had any financial relationship, in the thirty-six months prior to submission of this work, with any entity in the biomedical arena that could be perceived to influence or have the potential to influence what is written in this work. Also, no author has had any other relationships, or has engaged in any other activities, that could be perceived to influence or have the potential to influence what is written in this work. The complete Disclosures of Potential Conflicts of Interest submitted by authors are always provided with the online version of the article.
J Bone Joint Surg Am. 2014;96:e194(1-7)
d
http://dx.doi.org/10.2106/JBJS.N.00043
e194(2) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
used the Internet4. Of those users, 72% stated that they had sought online health information within the prior year4. Health-care-related information available on the Internet is highly variable in terms of quality and accuracy 5. Along with easy access to medical information comes the risk that patients will be misinformed by it6. To prevent this, Internet-based medical information should be of high quality and easily comprehensible by the majority of readers. Readability, a metric that indicates the ease with which text can be read and understood, is a major factor affecting the ability of a patient to utilize health information7. The health literacy of an individual and the necessary readability of patient education material are closely related. Health literacy is defined by the U.S. Department of Health and Human Services (HHS) as the capacity to obtain, interpret, and understand basic medical information and services and the competence to use such information and services to enhance health8-10. Previous studies have focused on assessing the readability of patient education materials. In particular, a 2008 study by Badarudeen and Sabharwal found that patient education materials from the AAOS (American Academy of Orthopaedic Surgeons) and POSNA (Pediatric Orthopaedic Society of North America) web sites had a mean Flesch-Kincaid (FK) grade level of 8.911. However, readability is an imperfect tool for interpreting an individual’s ability to understand provided materials. It only accounts for the number of syllables in a word and the sentence structure. The impact that supporting figures, tables, and graphs have on the individual’s understanding of the online material is not taken into consideration. Another method of assessing the quality of online patient education material is needed to better evaluate the level at which such materials are presented to consumers of medical information. A widely used validation instrument for health-care web sites, the LIDA tool (Minervation), has been developed to assess the quality—including accessibility, usability, and reliability—of patient education resources12. Various web sites containing patient education information have been evaluated with use of this tool13-21. The LIDA tool contains forty-one questions that yield a percentage grade for each of the three quality categories12. A score of >90% is considered a good result, 50% to 90% is moderate, and 90%, whereas the usability and accessibility scores would be between 50% and 90%. Furthermore, we anticipated that the two orthopaedic web sites would have LIDA scores similar to those reported for sites in other medical specialties and that readability would have improved further since the 2008 study. Materials and Methods We did not seek institutional review board approval because of the open-access nature of the utilized web sites. The AAOS web site provides a link to information for patients, and one category in the patient education portion is entitled Children. The Children web page lists 104 distinct articles within eleven sections: General, Shoulder & Elbow, Hip & Thigh, Knee & Lower Leg, Foot & Ankle, Neck & Back, Treatments & Surgeries, Safety, Sports Injury Prevention, 23 Health & Fitness, and Patient Stories . Six evaluators applied the LIDA tool to the AAOS web page (http:// orthoinfo.aaos.org/menus/children.cfm) and to the POSNA web page (http:// www.posna.org/parents/articles/orthoInfo.cfm) in May 2013. Three of the evaluators (D.P.F., D.K., and N.A.) were senior medical students, and the remaining three were a business professional, a financial advisor, and a high school teacher. Each evaluator was blinded to the scores assigned by the other five evaluators. Each was sent a PDF file containing an uncompleted LIDA tool and instructed to read the first two pages to obtain a general understanding of the nature of the LIDA tool and what it is designed to measure. Each was then instructed to access the web pages and critically evaluate them by following the instructions in the LIDA tool. Evaluators did not share their completed PDF file or communicate with each other during the evaluations. The readability of the AAOS online materials was then assessed by creating eleven documents, each containing the text of the articles within one subsection of the Children portion of the web site. Content unrelated to patient education (e.g., copyright information, outside links, disclaimers, date stamps, authorship, and source information) was eliminated from the document. Illustrations, tables, and citations were also eliminated so that the readability score would represent only the relevant text. An article was excluded if the content was provided in a non-text format such as illustrations or multimedia. The readability analysis of the included text was performed by two of the authors (D.P.F. and N.A.) with use of Readability Studio software (Professional Edition Version 2012.1 for Windows; Oleander Software, Vandalia, Ohio). A similar process was performed for the POSNA web site. The FRE value of a text passage equals 206.835 – 1.015 · (total words/ total sentences) – 84.6 · (total syllables/total words). The value rates the
e194(3) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
Fig. 1
Bar graph of the LIDA scores of the AAOS and POSNA web sites. The mean and standard deviation of the scores assigned by the six evaluators are shown. *P < 0.005 for the comparison between the web sites.
readability of the text on a 100-point scale, with a lower score indicating that the text is more difficult to understand. A score of 90 to 100 corresponds to text that is easily understood by a typical eleven-year-old. In general, authors of consumer information should aim for an FRE score of 60 to 70, which is what a typical thirteen to fifteen-year-old student would 24,25 easily understand . The FK grade level of a text passage equals (0.39 · mean number of words per sentence) 1 (11.8 · mean number of syllables per word) – 15.59. The resulting value is an approximation of the number of years of schooling that 24 would be required for someone to understand the text . A function in Microsoft Word (Redmond, Washington) can also be used to calculate the FK value and FRE grade level.
FK and FRE values were calculated by two evaluators (D.P.F. and N.A.). The mean values were calculated, and the interobserver variability was determined (Table I).
Source of Funding There was no external funding for this study.
Results The quality (including accessibility, usability, and reliability) of the pediatric patient education section of each web site according to the LIDA tool is shown in Figures 1 and 2. The mean
Fig. 2
Box-and-whisker plot of the LIDA scores of the AAOS and POSNA web sites. The boxes show the 25th, 50th, and 75th percentiles of the scores assigned by the six evaluators, and the whiskers show the minimum and maximum values. *P < 0.005 for the comparison between the web sites.
e194(4) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
TABLE I FK and FRE Values of Pediatric Orthopaedic Patient Education Materials According to Section on the AAOS Web Site FK
FRE
Assessment of 103 articles by observer 1 (D.P.F.) General
9.4
50
Foot & Ankle
8.8
64
Knee & Lower Leg Hip & Thigh
8.5 8.3
65 66
Neck & Back
9.1
62
Shoulder & Elbow
9.0
51
Treatments & Surgeries
8.5
65
Safety
8.9
63
10.2
53
Sports Injury Prevention Health & Fitness Patient Stories
9.9
55
11.8
50
Mean
9.3
58.5
Standard deviation
1.0
6.7
Assessment of 103 articles by observer 2 (N.A.) General
9.0
53
Foot & Ankle
8.1
60
Knee & Lower Leg
8.0
62
Hip & Thigh
8.8
60
Neck & Back
8.8
57
Shoulder & Elbow Treatments & Surgeries
8.5 8.2
54 63
Mean
8.4
58.4
Standard deviation
0.4
3.9
8.8
59.4
Recalculated mean of the 52 articles used in the 2008 study
TABLE II Mean LIDA Scores of the AAOS and POSNA Web Sites AAOS LIDA (%)
POSNA LIDA (%)
P Value
Accessibility
77.67
66.50
0.004
Usability
84.33
72.50
0.15
Reliability
84.50
86.67
0.363
overall accessibility score (and standard deviation of the scores of the six evaluators) was rated as 78% ± 2.5% (range, 75% to 81%) for the AAOS site and 66% ± 4.8% (range, 60% to 73%) for the POSNA site (p < 0.005). The mean usability was 84% ± 4.8% (range, 77% to 91%) for the AAOS site and 73% ± 5.1% (range, 65% to 80%) for the POSNA site (p = 0.15). The mean reliability was 85% ± 5.6% (range, 77% to 92%) for the AAOS site and 87% ± 4.7% (range, 80% to 92%) for the POSNA site (p = 0.36) (Table II).
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
Although the structure and design of the POSNA web site26 differed from that of the AAOS site (leading to the observed differences in quality as assessed with the LIDA tool), the POSNA web site contains links to materials on the patient education portion of the AAOS web site (http://www.orthoinfo. aaos.org). As a result, the FK and FRE scores for the material on the two web sites were identical. We included 103 of the 104 AAOS articles in the present study; the remaining article was omitted because its content was provided in a video format23. The overall FK readability score, 8.8, was somewhat higher than the recommended sixth-grade level27, and it was nearly identical to the value of 8.9 reported in the 2008 study. The mean FRE score was 59.4, similar to the 2008 reported value of 57.3. Mean FK and FRE values for material related to each core orthopaedic area and for the Safety, Sports Injury Prevention, Health & Fitness, and Patient Stories sections are shown in Table I. The 2008 study by Badarudeen and Sabharwal included fifty-two articles. In that study, FK and FRE values for each article were calculated with use of a function in Microsoft Word, rather than with Readability Studio software as in the present study. However, the overall FK and FRE values in the two studies were nearly identical. Discussion The ease with which material on health-related web sites can be read, as assessed by “readability scores,” is not the only factor that affects the ability of patents and their caregivers to access online health information. Rather, there is a complex interplay involving readability and other factors such as accessibility, usability, and reliability. Accessibility of a web site (as determined by adherence to W3C, Bobby, and other accessibility standards) is now mandated in the U.S. and many other countries. The aim of such standards is to attempt to make web sites easier for public viewing, accessible on appropriate devices (including mobile devices and computers running various operating systems), and available for use by individuals with visual and other impairments. Usability refers to the ability of users to obtain the information they seek, and it assesses the ease of navigating a web site. Factors such as clear home page and overall web site design, clear text and context-appropriate color, consistency, and ease of navigation are all assessed to categorize the usability of a web site. The reliability of a web site is critically important in sites presenting health-carerelated information. Reliable health-care information is information that is current, evidence-based, and unbiased. For example, one strategy for improving the reliability of a health-care web site involves citing various sources of health information and disclosing any conflicts of interest that could be perceived as representing a potential bias in the selection of those resources. Our LIDA assessment yielded a mean accessibility score of 78% for the AAOS web site and 66% for the POSNA site. Although the latter was significantly lower, both accessibility scores were within the moderate range. Page setup, access restrictions, and HTML code lifetime were accessibility factors evaluated by the LIDA tool. The AAOS patient education section was fully accessible without any registration requirements. The POSNA web site could also generally be accessed freely,
e194(5) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
TABLE III Mean LIDA Scores of Various Medically Oriented Online Patient Education Resources
Specialty Orthopaedics
Mean LIDA Score (%) 79
Reference No. Present study
Cardiology
73
19
Coloproctology
62.50
17
Coloproctology
85.60
14
Interventional radiology
67
21
Otolaryngology
61.30
29
Otolaryngology
69.30
18
Vascular surgery
65
15
Vascular surgery
73
28
Urology
66
16
Aortic aneurysm and endovascular treatment
70.20
20
although the scoliosis handbook (accessed via a link in the Parents & Patients section of the site) required a free subscription to access the material. Overall, patient information on both web sites was provided in a black-and-white format, and the font was Arial size 8. Thus, colorblind individuals would be able to access the information without difficulty, but a larger font might offer a more favorable user experience to the visually impaired. The mean usability rating was 84% for the AAOS web site and 73% for the POSNA site. Evaluators of the POSNA web site noted occasional difficulty in ascertaining which section they were viewing within the web site. Also, the POSNA color scheme was rated as less engaging than that of the AAOS site. In general, both web sites were rated well with respect to aspects of functionality; they performed well regardless of the Internet browser, provided effective search and browsing facilities, and did not require third-party plug-ins other than perhaps a common and freely available PDF reader. Additionally, both sites were clear, well structured, and easy to navigate. The three aspects that were largely responsible for the lower usability score of the POSNA web site compared with the AAOS site all involved the ability to engage the user: the interactive qualities of the site, ability of users to personalize their experience, and integration of non-textual media. Reliability was the one LIDA measure on which the POSNA web site scored slightly (2%) higher than the AAOS site, although this difference did not reach significance. Reliability was the aspect that earned the highest LIDA score, approaching “good” (>90%) for both sites. Both sites posted current and unbiased health information supported with links to outside agencies, and both revealed the identity of the authors contributing to the provided text. However, the source of funding for the POSNA and AAOS web sites was scored less favorably by the evaluators, as was the frequency with which the provided information was updated. The latter could be
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
improved with a sidebar on the home page that provides the web site user with an alert regarding any new modifications made to patient information on the site. Currently, neither web site has a section for user feedback or comments that can be viewed by other users. Having a section where users can discuss their experiences with others would allow patients and parents to learn from each other’s unique experiences with orthopaedic conditions. We also compared the LIDA scores of the POSNA and AAOS online materials with those reported for web sites in other medical fields. However, the majority of the previous investigators did not evaluate individual specialty-specific web sites. Rather, keywords specific to a particular field were entered into popular search engines, and the first ten to fifty web sites pertaining to a specific specialty were then evaluated with the LIDA tool. The LIDA scores of the pediatric orthopaedic material on the AAOS web site were comparable with, and in some instances even better than, those of certain other medical specialties with regard to accessibility, usability, and reliability (Table III)14-19,20,21,28,29. Nevertheless, given the differences in methodology between the present study and some of the previous ones, further investigation will be required to compare the quality of online patient education materials between orthopaedic and other medical specialty web sites. Pediatric information on the AAOS and POSNA web sites had a mean FK value of 8.8, which is higher than the 6th-grade
TABLE IV LIDA Scores Assigned by the Six Users* Accessibility (%)
Usability (%)
Reliability (%)
AAOS LIDA Evaluator 1 Evaluator 2
75 77
85 77
92 77
Evaluator 3
81
81
81
Evaluator 4
75
91
83
Evaluator 5
80
86
90
Evaluator 6
78
86
84
Mean
77.67
84.33
84.50
Std. dev.
2.50
4.80
5.61
SEM
1.03
1.97
2.30
POSNA LIDA Evaluator 1
60
65
90
Evaluator 2
73
80
80
Evaluator 3
65
72
92
Evaluator 4
70
76
82
Evaluator 5
68
70
87
Evaluator 6
63
72
89
Mean
66.50
72.50
86.67
4.76 1.95
5.13 2.10
4.72 1.93
Std. dev. SEM
*SEM = standard error of measurement.
e194(6) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
level recommended for patient education materials27. Furthermore, the FK value was nearly identical to the value (8.9) obtained by Badarudeen and Sabharwal in 200811. Although the reasons for this lack of change are unclear, it does underscore the importance of periodically reevaluating how patientrelated material conforms to NIH (National Institutes of Health) and HHS recommendations27,30. Additionally, our study revealed a recalculated FRE value of 59.4, again similar to the value obtained in the previous 2008 study (57.3). This value is less than the minimum limit of 60, a value that implies easy understanding by most thirteen to fifteen-year-old children, who are typically 8th to 9th graders in the U.S.9,10. The FRE value for the assessed articles in the AAOS pediatrics section would need to be at least 80 (a score for 7th graders or twelve-year-olds), and ideally at least 90 (indicating that it is easily understood by eleven-year-olds, the average age of 6th graders in the U.S.), to conform to accepted standards. As previously noted, the POSNA site contains links to information on the AAOS web site; therefore, the FK and FRE values were identical for the two web sites. Our study has several limitations. First, whereas other investigators evaluated multiple medical web sites utilizing specialtyspecific search terms, we focused on only two pediatric orthopaedic web sites that are frequently accessed and referenced. It is possible that a study analyzing more web sites would have yielded different LIDA scores. Second, a relatively small pool of data was used to determine the mean LIDA score for the two web sites, thus making the results susceptible to selection bias. Third, a larger number of LIDA evaluators with varying degrees of educational backgrounds might have allowed further insights regarding the quality of the pediatric orthopaedic online patient information. However, the degree of variability among the scores of the six evaluators was acceptable for each of the three LIDA sections (Figs. 1 and 2, Table IV). Furthermore, only three of the six LIDA evaluators had a medical background, reducing the bias that could result if such a background influences the way in which a user evaluates the web sites.
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
Although the quality of the patient information on the AAOS and POSNA web sites was comparable to, or better than, that reported for web sites in other medical fields, it can be improved further. Moreover, the readability scores of the pediatric content of the orthopaedic web sites did not change substantially compared with a similar assessment performed five years earlier. As assessing only the readability of online health information may not be the most comprehensive method of assessing the relevance and comprehensibility of online patient education material, other metrics such as the LIDA tool that grade the quality of such web sites may be useful. Future studies should include a larger sample of pediatric orthopaedic web sites, and the patient education information should be evaluated by an even more diverse group of evaluators with a wide spectrum of backgrounds. We would recommend that LIDA evaluators for future studies possess a variety of degrees of medical knowledge and perhaps be of various ethnic and social backgrounds to represent the diversity of online users. Also, having a portion of LIDA evaluators with reading abilities at, or below, the NIH and HHS standards would offer more insight into possible correlations between the LIDA scores (which reflect the quality of the health-related web sites) and the FK and FRE values (which assess the readability of the online content). n
Daniel P. Feghhi, BA Daniel Komlos, PhD Nitin Agarwal, BS Sanjeev Sabharwal, MD, MPH Departments of Orthopaedics (D.P.F., D.K., and S.S.) and Neurological Surgery (N.A.), New Jersey Medical School, Rutgers Biomedical and Health Sciences, 90 Bergen Street, Suite 7300, Newark, NJ 07101. E-mail address for S. Sabharwal: [email protected]
References 1. McMullan M. Patients using the Internet to obtain health information: how this affects the patient-health professional relationship. Patient Educ Couns. 2006 Oct;63(1-2):24-8. Epub 2006 Jan 6. 2. Hart A, Henwood F, Wyatt S. The role of the Internet in patient-practitioner relationships: findings from a qualitative research study. J Med Internet Res. 2004 Sep 30;6(3):e36. Epub 2004 Sep 30. 3. Ilic D. The role of the internet on patient knowledge management, education, and decision-making. Telemed J E Health. 2010 Jul-Aug;16(6):664-9. 4. Fox S, Duggan M. Pew research Internet project. Heath online 2013. 2013 Jan 15. http://www.pewinternet.org/Reports/2013/Health-online.aspx. Accessed 2013 Jul 1. 5. Purcell GP, Wilson P, Delamothe T. The quality of health information on the internet. BMJ. 2002 Mar 9;324(7337):557-8. 6. Chalmers I. Invalid health information is potentially lethal. BMJ. 2001 Apr 21; 322(7292):998. 7. Badarudeen S, Sabharwal S. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res. 2010 Oct;468(10):2572-80. 8. Albright J, de Guzman C, Acebo P, Paiva D, Faulkner M, Swanson J. Readability of patient education materials: implications for clinical practice. Appl Nurs Res. 1996 Aug;9(3):139-43. 9. Baker DW, Parker RM, Williams MV, Clark WS, Nurss J. The relationship of patient reading ability to self-reported health and use of health services. Am J Public Health. 1997 Jun;87(6):1027-30.
10. Nielsen-Bohlman L, Panzer A, Kindig D. Health literacy: a prescription to end confusion. Washington, DC: National Academics Press; 2004. 11. Badarudeen S, Sabharwal S. Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites. J Bone Joint Surg Am. 2008 Jan;90(1):199-204. 12. Tomlin A, Badenoch D. The LIDA instrument. Minervation validation instrument for health care web sites. 2007. http://www.minervation.com/wp-content/ uploads/2011/04/Minervation-LIDA-instrument-v1-2.pdf. Accessed 2013 Jul 1. 13. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, Kanouse DE, Muñoz JA, Puyol JA, Lara M, Watkins KE, Yang H, McGlynn EA. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001 May 23-30;285(20):2612-21. 14. Grewal P, Alagaratnam S. The quality and readability of colorectal cancer information on the internet. Int J Surg. 2013;11(5):410-3. Epub 2013 Mar 19. 15. Grewal P, Williams B, Alagaratnam S, Neffendorf J, Soobrah R. Quality of vascular surgery web sites on the Internet. J Vasc Surg. 2012 Nov;56(5):1461-7. Epub 2012 Jul 15. 16. Wong LM, Yan H, Margel D, Fleshner NE. Urologists in cyberspace: a review of the quality of health information from American urologists’ websites using three validated tools. Can Urol Assoc J. 2013 Mar-Apr;7(3-4):100-7. 17. Soobrah R, Clark SK. Your patient information website: how good is it? Colorectal Dis. 2012 Mar;14(3):e90-4.
e194(7) TH E JO U R NA L O F B O N E & JO I N T SU RG E RY J B J S . O RG V O L U M E 96-A N U M B E R 23 D E C E M B E R 3, 2 014 d
d
d
18. Goslin RA, Elhassan HA. Evaluating Internet health resources in ear, nose, and throat surgery. Laryngoscope. 2013 Jul;123(7):1626-31. Epub 2013 May 29. 19. Kirthi V, Modi BN. Coronary angioplasty and the Internet: what can patients searching online expect to find? J Interv Cardiol. 2012 Oct;25(5):476-81. Epub 2012 Jun 7. 20. San Noberto EM, Taylor J, Salvador R, Revilla A, Merino B, Vaquero C. The quality of information available on the Internet about aortic aneurysm and its endovascular treatment. Rev Esp Cardiol (Engl Ed). 2011;64(10):869-75. 21. Tavare AN, Alsafi A, Hamady MS. Analysis of the quality of information obtained about uterine artery embolization from the Internet. Cardiovasc Intervent Radiol. 2012 Dec;35(6):1355-62. Epub 2012 Jan 25. 22. Henry SL. W3C web accessibility initiative. Introduction to web accessibility. http://www.w3.org/WAI/intro/accessibility.php. Accessed 2013 Jul 1. 23. The American Academy of Orthopaedic Surgeons. Children. http://orthoinfo. aaos.org/menus/children.cfm. Accessed 2013 Jul 1.
Q UA L I T Y O F O N L I N E P E D I AT R I C O RT H O PA E D I C E D U C AT I O N M AT E R I A L S
24. Juicy Studio. Readability test. http://juicystudio.com/services/readability.php. Accessed 2013 Jul 1. 25. Flesch R. A new readability yardstick. J Appl Psychol. 1948 Jun;32(3):221-33. 26. The Pediatric Orthopaedic Society of North America. Parents & patients. http://www.posna.org/parents/parents.cfm. Accessed 2013 Jul 1. 27. National Institutes of Health. How to write easy-to-read health materials. http://www.nlm.nih.gov/medlineplus/etr.html. 2012. Accessed 2013 Jul 1. 28. Alsafi A, Kaya G, Patel H, Hamady MS. A comparison of the quality of the information available on the Internet on interventional radiology, vascular surgery, and cardiology. J Postgrad Med. 2013 Jan-Mar;59(1):69-75. 29. Muthukumarasamy S, Osmani Z, Sharpe A, England RJ. Quality of information available on the World Wide Web for patients undergoing thyroidectomy: review. J Laryngol Otol. 2012 Feb;126(2):116-9. Epub 2011 Aug 26. 30. Walsh TM, Volsko TA. Readability assessment of Internet-based consumer health information. Respir Care. 2008 Oct;53(10):1310-5.
Copyright of Journal of Bone & Joint Surgery, American Volume is the property of Journal of Bone & Joint Surgery, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
