Int Urogynecol J DOI 10.1007/s00192-014-2538-z
ORIGINAL ARTICLE
Quality of information on pelvic organ prolapse on the Internet Andrea B. Kakos & David A. Lovejoy & James L. Whiteside
Received: 8 July 2014 / Accepted: 3 October 2014 # The International Urogynecological Association 2014
Abstract Introduction and hypothesis This study aimed to determine the quality of available patient-centered information for pelvic organ prolapse (POP) on the Internet using a modified validated scale. Methods Two independent investigators using three search engines (Google, Yahoo, Bing) searched and reviewed the top 30 unique sites for four terms: bladder prolapse; dropped bladder; uterine prolapse; dropped uterus. A total of 219 websites were reviewed by both reviewers excluding redundancies. A two-stage, 6-point rating scale with score range per question of 0–5 was developed from the DISCERN instrument. Also recorded was whether a site had Health On the Net (HON) Foundation certification. The 400 sites were (as stated) a separate search where in the the domain suffix for the top 100 sites per serach term was recorded. Results The summary of 400 sites reviewed across the four search terms identified 64 % .com, 19 % .org, 8 % .edu, 6 % other and 3 % .gov; .gov yielded the highest quality information. Only 23 (9.5 %) sites were HON certified, yet these sites possessed higher DISCERN scores (p240,000 [1]. Indeed, that number may already have been met, as market data published on the US Food and Drug Administration (FDA) website states that in 2010 alone ~300,000 surgeries were performed for POP [2]. Those projections leave out the women who pursue conservative treatment options (e.g., watchful waiting, pelvic floor physical therapy, pessary) in order for these women to make informed health care decisions high-quality, patient-level information regarding POP treatment options is needed. Patients frequently use the Internet for health-care information [3]. Yet there are little data regarding the quality of information available about POP. Minaglia et al. found Internet sites sponsored by private parties or community groups were often incomplete, limiting patient awareness to all possible treatment options [4]. In contrast, government, university, and medical society sites are generally more comprehensive and significantly more likely to be relevant and accurate. That study, however, did not use any specific measure to quantify the quality of information, nor did it assess that information by domain suffix (e.g. .org, .com, .edu). Tools have been developed to assess the quality of healthrelated information. The DISCERN instrument is one such
Int Urogynecol J
tool [5] and is composed of 16 questions, each with five response levels, and is shown to be reliable in estimating the quality of web-based information [6]. In addition, the Health On the Net (HON) Foundation uses health-specific criteria to estimate website health information quality. HON’s mission is to identify and report reliable, comprehensive, relevant, and trustworthy sources of online health and medical information [7]. To date, such tools have not been used to evaluate the quality of such health information regarding POP. The purpose of this study was to use standardized instruments to estimate the overall quality of Internet-based health information on POP and to determine how that quality relates to domain suffix (.com, .org, .edu, .gov).
sites were combined with scores for sites reviewed by a single assessor. Total assessment score was then compared by search term, search engine, and site suffix using analysis of variance (ANOVA). Given differences in sample size across grouping variables, pairwise comparisons by grouping variables were conducted using the Tukey–Kramer honestly significant difference (HSD) test. For the 16 questions, a factor analysis identified relatedness across items. Post hoc adjustments were made in the loading values to identify unique domains without loss of question items. Domain subscores were then compared by search term, search engine, and site suffix also using ANOVA and the HSD test. Significance in statistical tests was set at p=0.05. Statistical analysis was performed using JMP v.11.1 (SAS Institute, Cary, NC, USA).
Methods During a single week at two geographically but regionally similar sites, two independent investigators—a second-year obstetrics and gynecology resident and a first-year pelvic medicine and reconstructive surgery fellow—used Google, Yahoo, and Bing to search the top 30 unique sites for four terms: bladder prolapse; dropped bladder; uterine prolapse, and dropped uterus. A two-stage, 6-point rating scale with score range per question 0–5 (0 = strongly disagree; 5 = strongly agree) was developed. There were 16 questions: 12 based on the DISCERN instrument, rendering a maximum score of 80, and four focused on information availability and quality regarding conservative treatment (pessary, watchful waiting, physical therapy). One of the latter four questions specifically asked whether images of a pessary procedure were provided. Also recorded was whether a site had HON certification, determined by the presence of the HON symbol. The DISCERN instrument’s original 5-point rating scale was modified to a 6-point, two-level scale designed to improve interrater reliability, expanding the options around partial credit. With the instrument created, the two raters reviewed ten sites and compared their scores. A third-party (JW) identified two-level differences per query per site and clarified interpretation of instrument queries prior to site searches. A separate Internet search was undertaken to identify the relative proportions of domain suffix for the top 100 sites identified per search term. Internal review board wavier was obtained, as no human subjects were involved. Statistical analysis Descriptive statistics were done across each variable, and where a skewed distribution was noted, median and interquartile range (IQR) was reported. For sites on which two reviews were conducted, the mean score between reviewers was used, and a kappa analysis was conducted wherein the two reviewers agreed on site suffix, search engine, and search term used. Mean assessment scores for those commonly reviewed
Results A total of 219 unique websites were reviewed, with 112 of these being done by both reviewers. Table 1 reports the descriptive features of the 219 websites, including median total and subscale scores. Table 2 lists the survey questions and Table 1 Descriptive features of 219 reviewed websites sites Search variables Site suffix .com .edu .gov .org Other Search engine Google Yahoo Bing Search term Bladder prolapse Uterine prolapse Dropped bladder Dropped uterus HON certification Total Internet site score out of 80 (median, IQR, range) Domain 1: subscore out of 30 (median, IQR, range) Domain 2: subscore out of 10 (median, IQR, range) Domain 3: subscore out of 15 (median, IQR, range) Domain 4: subscore out of 10 (median, IQR, range)
129 24 6 39 21 172 39 8 61 57 58 43 23 28, 16–39, 0–74 20, 11.8–24.5, 0–30 1.5, 0–5.5, 0–10 1, 0–3, 0–14 0, 0–3, 0–10
HON Health On the Net Foundation, IQR interquartile range
Int Urogynecol J Table 2 Item distribution and domain assignment Question (6-point scale, 0–5)
Median (IQR)
Min
Max
Domain
Kappaa
1. It is clear what sources were used to document treatment options for vaginal prolapse 2. It is clear when the document/its sources were produced describing treatment options for vaginal prolapse 3. Publication is balanced/unbiased in presenting all treatment options for vaginal prolapse 4. Details of additional sources/information are provided 5. Uncertainty of defining an overall “best” treatment for vaginal prolapse is described 6. How each treatment for vaginal prolapse works is described 7. Relative benefits of each treatment of vaginal prolapse are described 8. Relative risks of each treatment of vaginal prolapse are described 9. Consequences of no treatment (watchful waiting) are described 10. Effects of each treatment option on overall quality of life are described 11. It is clear that there is more than one treatment choice to manage vaginal prolapse 12. Support for shared decision making is provided by the site 13. Vaginal pessary is specifically described as a management option for vaginal prolapse 14. Images or graphics of a vaginal pessary are given on the site 15. Watchful waiting is specifically described as a management option for vaginal prolapse
0 (0–2) 0 (0–1.5)
0 0
5 5
4 4
0.53 0.48
4 (2–4.6) 0 (0–2) 1.5 (0–3) 3 (1.5–4) 0 (0–1.5) 0 (0–1) 0 (0–2) 0 (0–1) 4.5 (3–5) 1.3 (0–3) 5 (2.5–5) 0 0 (0–4)
0 0 0 0 0 0 0 0 0 0 0 0 0
5 5 5 5 5 5 5 4 5 5 5 5 5
1
2
0.17 0.34 0.05 0.05 0.08 0.17 0.18 0.03 0.20 0.13 0.70 0.92 0.40
16. Physical therapy is specifically described as a management option for vaginal prolapse
3 (0–5)
0
5
1
0.67
1 1 3 3 2 3 1 1
IQR, Interquartile range, Item reliability (Cronbach’s α): domain 1, 0.91; domain 2, 0.79; domain 3, 0.83; domain 4, 0.91 a Simple kappa statistic for 77 sites commonly reviewed; kappa conventions: 0.01–0.20 no to slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, 0.81–1 almost perfect agreement
response distributions for the 16 Internet site-assessment questions and shows which questions comprised the subscales identified in the factor analysis and kappa statistic for 77 commonly reviewed sites between the two reviewers. Thirtyfive commonly reviewed sites were not included in the agreement analysis due to disagreement over one or more site identifiers. Table 3 lists the top scoring sites by suffix. Notably, the interrater reliability of site-assessment questions was generally poor. Questions that displayed moderate or better reliability were questions 1, 2, 13, 14, and 16. Agreement between reviewers as to whether a site was HON certified was near perfect (simple kappa 0.96). Among the 400 top sites identified by search terms, the relative proportions by domain suffix were 64 % .com, 19 % .org, 8 % .edu, 6 % other, and 3 % .gov. Table 3 Top-scoring Internet sites by suffix and score URL
Score Suffix HON certified
http://www.patient.co.uk/doctor/ genitourinary-prolapse-pro www.physio-pedia.com/uterine_prolapse www.mayoclinic.org/diseases-conditions/ uterine-prolapse/basics/definition/con20027708 www.ncbi.nlm.nih.gov/pubmedhealth/ PMH0002477/ health.harvard.edu/fhg/updates/update0805c. shtml
74
Other
Yes
66.5 61
.com .org
No Yes
58
.gov
No
56.5
.edu
No
A factor analysis was conducted across the 219 unique sites for the 16 scored questions. Suppressing absolute loading values
ORIGINAL ARTICLE
Quality of information on pelvic organ prolapse on the Internet Andrea B. Kakos & David A. Lovejoy & James L. Whiteside
Received: 8 July 2014 / Accepted: 3 October 2014 # The International Urogynecological Association 2014
Abstract Introduction and hypothesis This study aimed to determine the quality of available patient-centered information for pelvic organ prolapse (POP) on the Internet using a modified validated scale. Methods Two independent investigators using three search engines (Google, Yahoo, Bing) searched and reviewed the top 30 unique sites for four terms: bladder prolapse; dropped bladder; uterine prolapse; dropped uterus. A total of 219 websites were reviewed by both reviewers excluding redundancies. A two-stage, 6-point rating scale with score range per question of 0–5 was developed from the DISCERN instrument. Also recorded was whether a site had Health On the Net (HON) Foundation certification. The 400 sites were (as stated) a separate search where in the the domain suffix for the top 100 sites per serach term was recorded. Results The summary of 400 sites reviewed across the four search terms identified 64 % .com, 19 % .org, 8 % .edu, 6 % other and 3 % .gov; .gov yielded the highest quality information. Only 23 (9.5 %) sites were HON certified, yet these sites possessed higher DISCERN scores (p240,000 [1]. Indeed, that number may already have been met, as market data published on the US Food and Drug Administration (FDA) website states that in 2010 alone ~300,000 surgeries were performed for POP [2]. Those projections leave out the women who pursue conservative treatment options (e.g., watchful waiting, pelvic floor physical therapy, pessary) in order for these women to make informed health care decisions high-quality, patient-level information regarding POP treatment options is needed. Patients frequently use the Internet for health-care information [3]. Yet there are little data regarding the quality of information available about POP. Minaglia et al. found Internet sites sponsored by private parties or community groups were often incomplete, limiting patient awareness to all possible treatment options [4]. In contrast, government, university, and medical society sites are generally more comprehensive and significantly more likely to be relevant and accurate. That study, however, did not use any specific measure to quantify the quality of information, nor did it assess that information by domain suffix (e.g. .org, .com, .edu). Tools have been developed to assess the quality of healthrelated information. The DISCERN instrument is one such
Int Urogynecol J
tool [5] and is composed of 16 questions, each with five response levels, and is shown to be reliable in estimating the quality of web-based information [6]. In addition, the Health On the Net (HON) Foundation uses health-specific criteria to estimate website health information quality. HON’s mission is to identify and report reliable, comprehensive, relevant, and trustworthy sources of online health and medical information [7]. To date, such tools have not been used to evaluate the quality of such health information regarding POP. The purpose of this study was to use standardized instruments to estimate the overall quality of Internet-based health information on POP and to determine how that quality relates to domain suffix (.com, .org, .edu, .gov).
sites were combined with scores for sites reviewed by a single assessor. Total assessment score was then compared by search term, search engine, and site suffix using analysis of variance (ANOVA). Given differences in sample size across grouping variables, pairwise comparisons by grouping variables were conducted using the Tukey–Kramer honestly significant difference (HSD) test. For the 16 questions, a factor analysis identified relatedness across items. Post hoc adjustments were made in the loading values to identify unique domains without loss of question items. Domain subscores were then compared by search term, search engine, and site suffix also using ANOVA and the HSD test. Significance in statistical tests was set at p=0.05. Statistical analysis was performed using JMP v.11.1 (SAS Institute, Cary, NC, USA).
Methods During a single week at two geographically but regionally similar sites, two independent investigators—a second-year obstetrics and gynecology resident and a first-year pelvic medicine and reconstructive surgery fellow—used Google, Yahoo, and Bing to search the top 30 unique sites for four terms: bladder prolapse; dropped bladder; uterine prolapse, and dropped uterus. A two-stage, 6-point rating scale with score range per question 0–5 (0 = strongly disagree; 5 = strongly agree) was developed. There were 16 questions: 12 based on the DISCERN instrument, rendering a maximum score of 80, and four focused on information availability and quality regarding conservative treatment (pessary, watchful waiting, physical therapy). One of the latter four questions specifically asked whether images of a pessary procedure were provided. Also recorded was whether a site had HON certification, determined by the presence of the HON symbol. The DISCERN instrument’s original 5-point rating scale was modified to a 6-point, two-level scale designed to improve interrater reliability, expanding the options around partial credit. With the instrument created, the two raters reviewed ten sites and compared their scores. A third-party (JW) identified two-level differences per query per site and clarified interpretation of instrument queries prior to site searches. A separate Internet search was undertaken to identify the relative proportions of domain suffix for the top 100 sites identified per search term. Internal review board wavier was obtained, as no human subjects were involved. Statistical analysis Descriptive statistics were done across each variable, and where a skewed distribution was noted, median and interquartile range (IQR) was reported. For sites on which two reviews were conducted, the mean score between reviewers was used, and a kappa analysis was conducted wherein the two reviewers agreed on site suffix, search engine, and search term used. Mean assessment scores for those commonly reviewed
Results A total of 219 unique websites were reviewed, with 112 of these being done by both reviewers. Table 1 reports the descriptive features of the 219 websites, including median total and subscale scores. Table 2 lists the survey questions and Table 1 Descriptive features of 219 reviewed websites sites Search variables Site suffix .com .edu .gov .org Other Search engine Google Yahoo Bing Search term Bladder prolapse Uterine prolapse Dropped bladder Dropped uterus HON certification Total Internet site score out of 80 (median, IQR, range) Domain 1: subscore out of 30 (median, IQR, range) Domain 2: subscore out of 10 (median, IQR, range) Domain 3: subscore out of 15 (median, IQR, range) Domain 4: subscore out of 10 (median, IQR, range)
129 24 6 39 21 172 39 8 61 57 58 43 23 28, 16–39, 0–74 20, 11.8–24.5, 0–30 1.5, 0–5.5, 0–10 1, 0–3, 0–14 0, 0–3, 0–10
HON Health On the Net Foundation, IQR interquartile range
Int Urogynecol J Table 2 Item distribution and domain assignment Question (6-point scale, 0–5)
Median (IQR)
Min
Max
Domain
Kappaa
1. It is clear what sources were used to document treatment options for vaginal prolapse 2. It is clear when the document/its sources were produced describing treatment options for vaginal prolapse 3. Publication is balanced/unbiased in presenting all treatment options for vaginal prolapse 4. Details of additional sources/information are provided 5. Uncertainty of defining an overall “best” treatment for vaginal prolapse is described 6. How each treatment for vaginal prolapse works is described 7. Relative benefits of each treatment of vaginal prolapse are described 8. Relative risks of each treatment of vaginal prolapse are described 9. Consequences of no treatment (watchful waiting) are described 10. Effects of each treatment option on overall quality of life are described 11. It is clear that there is more than one treatment choice to manage vaginal prolapse 12. Support for shared decision making is provided by the site 13. Vaginal pessary is specifically described as a management option for vaginal prolapse 14. Images or graphics of a vaginal pessary are given on the site 15. Watchful waiting is specifically described as a management option for vaginal prolapse
0 (0–2) 0 (0–1.5)
0 0
5 5
4 4
0.53 0.48
4 (2–4.6) 0 (0–2) 1.5 (0–3) 3 (1.5–4) 0 (0–1.5) 0 (0–1) 0 (0–2) 0 (0–1) 4.5 (3–5) 1.3 (0–3) 5 (2.5–5) 0 0 (0–4)
0 0 0 0 0 0 0 0 0 0 0 0 0
5 5 5 5 5 5 5 4 5 5 5 5 5
1
2
0.17 0.34 0.05 0.05 0.08 0.17 0.18 0.03 0.20 0.13 0.70 0.92 0.40
16. Physical therapy is specifically described as a management option for vaginal prolapse
3 (0–5)
0
5
1
0.67
1 1 3 3 2 3 1 1
IQR, Interquartile range, Item reliability (Cronbach’s α): domain 1, 0.91; domain 2, 0.79; domain 3, 0.83; domain 4, 0.91 a Simple kappa statistic for 77 sites commonly reviewed; kappa conventions: 0.01–0.20 no to slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, 0.81–1 almost perfect agreement
response distributions for the 16 Internet site-assessment questions and shows which questions comprised the subscales identified in the factor analysis and kappa statistic for 77 commonly reviewed sites between the two reviewers. Thirtyfive commonly reviewed sites were not included in the agreement analysis due to disagreement over one or more site identifiers. Table 3 lists the top scoring sites by suffix. Notably, the interrater reliability of site-assessment questions was generally poor. Questions that displayed moderate or better reliability were questions 1, 2, 13, 14, and 16. Agreement between reviewers as to whether a site was HON certified was near perfect (simple kappa 0.96). Among the 400 top sites identified by search terms, the relative proportions by domain suffix were 64 % .com, 19 % .org, 8 % .edu, 6 % other, and 3 % .gov. Table 3 Top-scoring Internet sites by suffix and score URL
Score Suffix HON certified
http://www.patient.co.uk/doctor/ genitourinary-prolapse-pro www.physio-pedia.com/uterine_prolapse www.mayoclinic.org/diseases-conditions/ uterine-prolapse/basics/definition/con20027708 www.ncbi.nlm.nih.gov/pubmedhealth/ PMH0002477/ health.harvard.edu/fhg/updates/update0805c. shtml
74
Other
Yes
66.5 61
.com .org
No Yes
58
.gov
No
56.5
.edu
No
A factor analysis was conducted across the 219 unique sites for the 16 scored questions. Suppressing absolute loading values
