An Online Quality Assurance Program for Colposcopy in a Population-Based Cervical Screening Setting in Italy: Results on Colposcopic Impression Paolo Cristiani, MD,1 Silvano Costa, MD,2 Patrizia Schincaglia, MD,3 Paola Garutti, MD,4 Priscilla Sassoli de Bianchi, BBiol,5 Carlo Naldoni, MD,5 Mario Sideri, MD,6 and Lauro Bucchi, MD7
Objective: To report the results of an Internet-based colposcopy quality assurance program from a population-based cervical screening service in a large region of northern Italy. Methods: In 2010 to 2011, a Web application was made accessible on the Web site of the regional administration. Fifty-nine colposcopists of the registered 65 participated. They logged-in, viewed a posted set of 50 high-quality digital colpophotographs selected by an expert committee, and rated them for colposcopic impression using a 4-tier classification (Negative; abnormal, grade 1 [G1]; abnormal, grade 2 [G2]; suspected invasive cancer [Cancer]) derived from the International Federation for Cervical Pathology and Colposcopy 2002 classification. kappa (J) coefficients for intercolposcopist agreement and colposcopist-committee agreement were calculated. Results: Colposcopist-committee agreement was greater than intercolposcopist agreement (overall J 0.69 vs 0.60, p G .001). The J values for colposcopist-committee agreement were 0.83 on Negative, 0.53 on G1, 0.66 on G2, and 0.80 on Cancer (all p values for pairwise comparisons G.001, except for Negative vs Cancer [ p = .078]). There was no systematic tendency for colposcopists to underestimate or overestimate the colposcopic findings (2-tailed sign test, p = .13). Overall colposcopistcommittee agreement was greater among patients 35 years or older ( p G .001) and for colposcopists with previous quality assurance experiences ( p G .01). Only 0.2% of Negative impressions were formulated for a cervical intraepithelial neoplasia grade 2 or worse. As a parallel finding, the impression of Cancer predicted cervical intraepithelial neoplasia grade 2 or less in 0.5% of cases. The histologic substrates of G1 were dispersed over a large spectrum. Conclusions: The reproducibility of colposcopic impression, when classified by trained colposcopists examining high-quality images, is higher than is generally thought. Key Words: quality assurance, colposcopy, cervical cancer screening, interobserver agreement, Internet (J Lower Gen Tract Dis 2014;18: 309Y313)
C
olposcopy is a critical step in the management of women with abnormal Pap smear results. However, the technique has often been reported to be associated with a low interobserver agreement in interpreting cervical lesions1Y4 and a high biopsy sampling error rate. Incorrect selection of biopsy site 1
Cervical Cancer Screening Unit, Bologna Health Care District, Bologna; Department of Obstetrics and Gynaecology, St. Orsola Hospital, Bologna; 4 3 Cancer Prevention Center, Ravenna Health Care District, Ravenna; Depart5 ment of Obstetrics and Gynaecology, University Hospital, Ferrara; Department of Health, Regione Emilia-Romagna, Bologna; 6Preventive Gynaecology Unit, European Institute of Oncology, Milan; and 7Romagna Cancer Registry, Romagna Cancer Institute (IRST), Meldola, Forlı`, Italy Reprint requests to: Lauro Bucchi, MD, Romagna Cancer Registry, IRCCS IRST, 47014 Meldola, Forlı`, Italy. E-mail: [email protected] The authors declare that they have no conflicts of interest. * 2014, American Society for Colposcopy and Cervical Pathology 2
Journal of Lower Genital Tract Disease
&
may lead to low sensitivity for high-grade cervical intraepithelial neoplasia (CIN).5Y7 There are experts who believe that, until these deficiencies are recognized and corrected, colposcopically guided biopsy cannot be considered the reference standard for the diagnosis of cervical disease.8 The proposed approaches to accomplish this task include the development of computer-assisted diagnosis, the collection of additional biopsy specimens from the abnormal cervical area (or areas), the collection of random specimens from all quadrants, and the identification and diffusion of a limited set of well-defined and highly reproducible colposcopic features of disease.8 The latter is a challenging process because it requires the widespread use of a standard classification, sufficient formal and practical apprenticeship, and quality assurance (QA) programs involving the largest possible number of colposcopists.8 These programs should be based on interactive retraining sessions and large agreement and accuracy studies to identify reproducible colposcopic features of cervical abnormalities and improve the single colposcopist’s competence as well as the appropriateness of his/her clinical decisions. These are also the objectives of a colposcopy QA program that is underway in the Emilia-Romagna Region of northern Italy in the context of a local population-based cervical screening service. In this article, we report the results on colposcopic impression.
MATERIALS AND METHODS Setting Detailed information of cervical screening service in the Emilia-Romagna Region is reported elsewhere.9Y11 Colposcopy assessment for women with abnormal Pap smear results is carried out by specially appointed gynecologists and gynecologistoncologists. QA activities are designed by experts from the screening centers (n = 11) and the public hospitals and are supervised by the regional Department of Health. For the past decade, several on-site QA initiatives were implemented for screening colposcopists, including courses on the distinction of abnormal from normal transformation zone and the grading of colposcopic abnormalities.
Design Between 2009 and 2010, an online QA program was developed. The program had no administrative functions (ranking, accreditation, etc.) and placed more emphasis on providing education and feedback to participants than on testing them. Participation was on a voluntary basis. A log-in Internet application was created and made accessible on the Web site of the regional administration. Between December 2010 and February 2011, screening colposcopists were invited to register, log-in, view a posted set of 50 colpophotographs selected by an expert steering committee, and classify them according to colposcopic impression, visibility of
Volume 18, Number 4, October 2014
309
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Journal of Lower Genital Tract Disease
Cristiani et al.
the squamocolumnar junction (SCJ), and need for biopsy. They were also asked to indicate the most appropriate site for biopsy with a left-button mouse click on the colposcopic image. In May 2011, a plenary seminar was held to discuss the results and to perform an interactive review of the test set of images. A detailed and extensive protocol of the program was published in an open-access journal.12 Given the accessibility of the document, only essential information is provided here.
Selection of the Test Set of Colpophotographs Two hundred fifty high-definition digital colpophotographs from a consecutive series of women with abnormal Pap smear results attending 2 random district screening centers were obtained. Technical specifications of the colposcopes, cameras, and video editing softwares that were used can be found in our previous article.12 The images were stored in the jpeg format. From this basic set, 50 images were selected based on the following criteria: (1) they were well representative of major normal and abnormal colposcopic findings; (2) they were of high technical quality; (3) the cervix was entirely visible; (4) there were no light reflections, color artifacts, shaded areas, or mucus accumulation; and (5) the patient had not been treated previously. The rationale for these criteria is discussed in our previous article.12
Classification of Colposcopic Impression Each colpophotograph in the test set was accompanied by a caption with information about the patient’s age, last Pap smear result, and human papillomavirus test result (which was actually performed only in 4 patients). Colposcopic impression was classified into 4 categories: Negative; abnormal, grade 1 (G1); abnormal, grade 2 (G2); and suspected invasive cancer (Cancer). Despite the simplified terminology, these categories were equivalent to the colposcopic patterns that the International Federation for Cervical Pathology and Colposcopy classification of 200213 designated as follows: normal colposcopic findings; abnormal colposcopic findings, minor changes; abnormal colposcopic findings, major changes; and colposcopic features suggestive of invasive cancer.
Data Analysis Data analysis was based on the observed percent agreement and the Cohen J coefficient and was centered on colposcopist-committee agreement. Given the high level of expertise of gynecologists and gynecologist-oncologists in the committee, this was considered a highly reliable reference but not an absolute one, thus justifying the use of agreement statistics.
&
Volume 18, Number 4, October 2014
Specific J values (e.g., for patient’s age group) and overall J values were used. The colposcopist’s age was used as a proxy for the years of professional experience. In the estimation of overall J values (a measure of overall agreement among all observers for all categories considered), one in each pair was chosen randomly as reviewer 1 and the other as reviewer 2. Comparison between J values was performed using the method by Soliani.14 The 2-tailed nonparametric sign test was used to determine whether colposcopists tended to significantly underrate or overrate the severity of colposcopic findings as compared with the committee. The degree and direction of disagreement were assessed by cross-tabulating the distribution of colposcopic impressions as formulated by the colposcopists compared to those formulated by the committee. Although colposcopic impression is not a direct predictor of histologic grading of a lesion,12 analysis included a crosstabulation with histology diagnoses to determine the approximate probability for a woman with an abnormal Pap smear result to receive a false-negative and false-positive colposcopy evaluation. When multiple biopsies and surgical specimens were available, the most severe diagnosis was considered.
RESULTS According to colposcopic impressions formulated by selectors, the test series of 50 images included 17 Negative cases, 10 cases of G1, 19 cases of G2, and 4 cases of Cancer. The median patient’s age was 35 years (range, 23Y58 y). Of the total staff of 65 colposcopists, 60 participated. Because one of these examined only part of the images, data were analyzed for 59 participants (median age, 56 y; range, 32Y63 y). Forty-five (76%) of these had participated in previous on-site QA initiatives. With 59 colposcopists and 50 colposcopic images, there were a total of 1,711 pairs of colposcopists and 85,550 paired observations. The number of paired colposcopistcommittee observations was 2,950. Both the overall and the impression-specific J values were significantly greater for colposcopist-committee agreement than for intercolposcopist agreement (see Table 1). The J values for Negative and Cancer were the highest and that for G1 was the lowest. The J values for colposcopist-committee agreement on Negative, G1, G2, and Cancer were mutually significantly different ( p G .001), with the exception of Negative vs Cancer ( p = .078). The distribution of impressions formulated by the colposcopists compared to those formulated by the committee (see Table 2) shows in greater detail that G1 was the least consistent impression. Over the entire table, however, negative
TABLE 1. Intercolposcopist Agreement and Colposcopist-Committee Agreement
Colposcopic impression Negative G1 G2 Cancer Overall
Intercolposcopist agreementa
Colposcopist-committee agreementb
Crude %
J
Crude %
J
p for comparison between J values
90 77 80 97 72
0.76 0.44 0.56 0.74 0.60
93 82 84 97 78
0.83 0.53 0.66 0.80 0.69
G.001 G.001 G.001 G.01 G.001
Cancer indicates suspected invasive cancer; G1, abnormal, grade 1; G2, abnormal, grade 2. a Data were based on 59 colposcopists and 50 colposcopic images for a total of 1,711 pairs of colposcopists and 85,550 paired observations. b Data were based on a total 2,950 paired observations.
310
* 2014, American Society for Colposcopy and Cervical Pathology
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Journal of Lower Genital Tract Disease
&
Volume 18, Number 4, October 2014
Colposcopy Quality Assurance in Italy
TABLE 2. Distribution of Colposcopic Impressions Formulated by the Colposcopists According to the Impressions Formulated by the Committee Impression formulated by the committee
Impression formulated by the colposcopists
Negative
G1
G2
Cancer
Total
Negative G1 G2 Cancer Total
826 (95.8) 139 (16.5) 38 (3.7) 0 (0.0) 1,003
34 (3.9) 457 (54.2) 98 (9.4) 1 (0.5) 590
2 (0.2) 246 (29.2) 849 (81.6) 24 (11.8) 1,121
0 (0.0) 1 (0.1) 56 (5.4) 179 (87.7) 236
862 (100.0) 843 (100.0) 1,041 (100.0) 204 (100.0) 2,950a
Cancer indicates suspected invasive cancer; G1, abnormal, grade 1; G2, abnormal, grade 2. Numbers in parentheses are percentages. Two-tailed nonparametric sign test: negative differences (with a lower grade being reported by colposcopists) n = 339; positive differences n = 300; Z statistic = j1.503; 2-tailed p = .13. a Number of paired observations.
differences between the screening colposcopists and the committee did not significantly exceed positive differences, which indicated the absence of a systematic component in disagreement. Table 3 shows the factors associated with colposcopistcommittee agreement. The J values for G1 and G2 were higher among images obtained from patients 35 years or older. An impression of G2 was more likely to be agreed on by the committee if the referral Pap smear showed high-grade atypia. The consistency of the impression of Negative was greater if associated with total or partial invisibility of the SCJ, whereas the opposite occurred for G2. The colposcopist’s age had no effect whatsoever on the results, whereas participation in previous QA initiatives was associated with increased agreement on G1 and G2. As shown in Table 4, Negative exhibited an almost perfect predictive value. Only 0.2% of Negative impressions were formulated for a CIN2 or worse (representing discrepancies of 2 or more degrees). As a parallel finding, the impression of Cancer predicted CIN2 or less in 0.5% of cases. The histologic substrates of G1 were dispersed over a large spectrum of lesions. No predominant counterpart was observed because the prevalence of normal histology was of the same order of magnitude as CIN1 and CIN3/adenocarcinoma in situ (AIS). Approximately one fourth of patients harbored CIN3 or worse. Data were then stratified according to whether the impression of G1 was agreed on by less or more than 50% of observers. In the latter subset of colposcopic impressions, no case of carcinoma was observed and CIN1 reached a prevalence of 45%. G2 predicted CIN3/AIS in two thirds of cases, and normal histology/CIN1 in less than 15% of cases. Among colposcopic impressions agreed on by more than 50% of observers, the prevalence of normal histology/CIN1 was only 6% and that of CIN3/AIS was greater than 70%.
DISCUSSION Although there have been studies reporting excellent levels of agreement between colposcopists,15 our results are generally better than those of most previous comparable investigations.2,3,16 In fact, agreement studies are always limited by the lack of an absolute gold standard, the subjectivity of the criteria for the selection of colposcopic images, and the artificiality of the test conditions. Whatever solutions are adopted, these problems are impossible to resolve completely.12 For this * 2014, American Society for Colposcopy and Cervical Pathology
reason, results cannot be safely used for external comparisons with other studies or for central control and administrative purposes. The images used for this program were free of imperfections and were considered by selectors to be well representative of the spectrum of colposcopic findings. The rationale for these TABLE 3. Factors Affecting J Values for Agreement Between the Colposcopists and the Committee Impression Factors
Negative
G1
G2
Cancer Overall
0.83 0.84
0.35 0.66b
0.57 0.72b
NC 0.82
0.60 0.76b
0.83 0.78
0.53 0.47
0.60 0.68c
NC 0.79
0.66 0.68
0.89
0.51
0.62
0.81
0.70
0.64b
0.56
0.80b
NC
0.67
0.83 0.86 0.82
0.51 0.52 0.49
0.65 0.65 0.64
0.77 0.82 0.79
0.68 0.70 0.63
0.86
0.57e
0.72e
0.81
0.71c
a
Patient age G35 y Q35 y Pap smear result ASCUS, LSIL HSIL, AGC, AIS Squamocolumnar junction Not or not entirely visible Entirely visible Colposcopist characteristics Aged 32Y50 yd Aged 59Y63 y With no previous QA experience With previous QA experience
AGC indicates atypical glandular cells; AIS, adenocarcinoma in situ; ASCUS, atypical squamous cells of undetermined significance; Cancer, suspected invasive cancer; G1, abnormal, grade 1; G2, abnormal, grade 2; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NC, not calculated because the committee did not formulate an impression of cancer for any image in this stratum. Data were based on a total of 2,950 paired observations. a Patients were divided by their median age. b p G .001. c p G .01 d The 12 youngest colposcopists were compared with the 12 oldest ones. e p G .05.
311
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Journal of Lower Genital Tract Disease
Cristiani et al.
&
Volume 18, Number 4, October 2014
TABLE 4. Predictive Value of Colposcopic Impressions Formulated by the Colposcopists for Histology Histologic diagnosis Impression Negative G1a Agreed by G50% of observers Agreed by Q50% of observers Total G2a Agreed by G50% of observers Agreed by Q50% of observers Total Cancer Total
Normal
CIN1
CIN2
CIN3/AIS
Carcinoma
Total
795 (92.2)
65 (7.5)
0 (0.0)
1 (0.1)
1 (0.1)
862 (100.0)
121 (40.1) 109 (20.1) 230 (27.3)
56 (18.5) 243 (44.9) 299 (35.5)
22 (7.3) 62 (11.5) 84 (10.0)
77 (25.5) 127 (23.5) 204 (24.2)
26 (8.6) 0 (0.0) 26 (3.1)
302 (100.0) 541 (100.0) 843 (100.0)
37 (17.2) 0 (0.0) 37 (3.6) 0 (0.0) 1,062
58 (27.0) 49 (5.9) 107 (10.3) 1 (0.5) 472
56 (26.0) 96 (11.6) 152 (14.6) 0 (0.0) 236
59 (27.4) 597 (72.3) 656 (63.0) 83 (40.7) 944
5 (2.3) 84 (10.2) 89 (8.5) 120 (58.8) 236
215 826 1,041 204 2,950b
(100.0) (100.0) (100.0) (100.0) (100.0)
AIS indicates adenocarcinoma in situ; Cancer, suspected invasive cancer; CIN, cervical intraepithelial neoplasia; G1, abnormal, grade 1; G2, abnormal, grade 2. Numbers in parentheses are percentages. a Colposcopic images were divided according to whether the impression was agreed by less or more than 50% of observers. b Number of paired observations.
criteria was that a QA program should target the basic presentations of normal and affected tissues of the cervix with which to compare the spectrum of colposcopic patterns encountered in practice.12 This condition further prevents extrapolation of results to daily practice. However, and more importantly, it reveals the potential reproducibility of colposcopic impression. If used by trained colposcopists examining high-quality images, a simplified and rational classification scheme can reach good levels of reproducibility. Another important finding was that interobserver agreement in a population of colposcopists is lower than between these and a panel of experts (see Table 1). Given the high number of colposcopists in this program, their range of variability was relatively large, and the experts’ approach was probably based on intermediate interpretative criteria. This would explain why colposcopists were more likely to agree with the committee than with each other. Data in Table 2 are in accordance with this hypothesis. There was no systematic component in disagreement. Negative differences in impression between the colposcopists and the committee, representing underestimation, were not significantly more than positive differences. In other words, the limited excess of negative differences was compatible with chance variation. We identified several factors affecting the colposcopistcommittee agreement (see Table 3). First, the level of agreement on G1 and G2 was greater if images were from patients 35 years or older. In younger patients, the maturation of the metaplastic epithelium may be asynchronous, and because of this, immature areas may be combined with more mature sections. If immature metaplastic areas react intensely to acetic acid and are uniformly iodine-negative, they may give an impression of intraepithelial lesion, particularly when they exhibit a fine punctation or mosaic due to stromal projections centered by capillaries. Second, the Negative impression was more consistent if associated with total or partial invisibility of the SCJ. Identifying the original squamous and glandular epithelium is relatively easy. Difficulties increase when the transformation zone shows complex squamous aspects because benign and abnormal colposcopic findings differ only in slight details. When the
312
transformation zone appears as a mature squamous epithelium confining the SCJ in the endocervical canal, interobserver agreement increases. Conversely, when there are areas of persistent immature metaplasia and the SCJ is on the ectocervix, variation in interpretation is larger. Third, although colposcopist’s ageVa proxy of the number of years of professional experienceVhad no influence whatsoever, participation in previous specific QA initiatives had a significant positive effect on the agreement on both G1 and G2, that is, the least consistent colposcopic impressions. These important findings strongly support the importance of active measures of QA and training. Although colposcopic impression is not a direct correlate of histologic grading,8,17 the (negative) predictive value of Negative and the positive predictive value of an impression of Cancer can reach high levels (see Table 4). No more than 0.5% of patients with colposcopic images interpreted as Negative and Cancer were diagnosed, respectively, with CIN2 or worse and CIN2 or less, representing discrepancies of 2 or more degrees in opposite directions. The observed predictive value of Negative is particularly worthy of note. It suggests that, when colposcopy is correctly interpreted as showing no abnormal findings, taking random biopsies would not increase appreciably the probability of detecting a high-grade disease. G1 was the least specific condition. Except for a low prevalence of invasive carcinoma, the underlying lesions followed a virtually random distribution. In particular, approximately one fourth of patients were diagnosed with CIN3 or worse, including some lesions that were located in the endocervix and were not visible. These findings emphasize the importance of examining through biopsy those cervical abnormalities that are colposcopically interpreted as G1. Current guidelines recommend that 2 or more biopsy specimens should be taken, coupled with an endocervical curettage when the transformation zone is not visible and in women older than 45 years with high-grade cytology.18 In a separate article, we will report the results of the QA program on the performance of biopsy, including the frequency by which the biopsy site was correctly chosen on digital colpophotographs, which can be taken as a proxy of biopsy sensitivity. * 2014, American Society for Colposcopy and Cervical Pathology
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Journal of Lower Genital Tract Disease
&
Volume 18, Number 4, October 2014
Both for G1 and G2, the consistency of histologic counterparts increased when the impression was agreed on by most colposcopists. The fact that abnormalities interpreted in the same way by most observers had a greater level of specificity suggests that they met shared definitions of G1 and G2. Of course, they were probably easier to recognize. However, this is equivalent to saying that the classic features of G1 and G2 have a higher predictive value and that their recognition makes colposcopic impressions more specific. This, in turn, confirms the validity of the classification scheme used in the current study. In conclusion, the current study demonstrated that the theoretical reproducibility of colposcopic impression is higher than is generally thought. This means that the low level of performance that is often found in colposcopy practice is at least partly due to insufficient training. The impression of G1 was found to be the least consistent one, with a considerable prevalence of high-grade lesions. This suggests that training should pay special attention to the recognition of low-grade colposcopic findings and that cervical abnormalities interpreted as such are an indication for biopsy. REFERENCES 1. Sellors JW, Nieminen P, Vesterinen E, Paavonen J. Observer variability in the scoring of colpophotographs. Obstet Gynecol 1990;76:1006Y8. 2. Etherington IJ, Luesley DM, Shafi MI, Dunn J, Hiller L, Jordan JA. Observer variability among colposcopists from the West Midlands region. Br J Obstet Gynaecol 1997;104:1380Y4. 3. Ferris DG, Litaker M. Interobserver agreement for colposcopy quality control using digitized colposcopic images during the ALTS trial. J Low Genit Tract Dis 2005;9:29Y35. 4. Massad LS, Jeronimo J, Schiffman M. Interobserver agreement in the assessment of components of colposcopic grading. Obstet Gynecol 2008;111:1279Y84. 5. ASCUS-LSIL Triage Study (ALTS) Group. Results of a randomized trial on the management of cytology interpretations of atypical squamous cells of undetermined significance. Am J Obstet Gynecol 2003;88:1383Y92. 6. Pretorius RG, Zhang WH, Belinson JL, Huang MN, Wu LY, Zhang X, et al. Colposcopically directed biopsy, random cervical biopsy, and endocervical curettage in the diagnosis of cervical intraepithelial neoplasia II or worse. Am J Obstet Gynecol 2004;19:430Y4.
* 2014, American Society for Colposcopy and Cervical Pathology
Colposcopy Quality Assurance in Italy
7. Gage JC, Hanson VW, Abbey K, Dippery S, Gardner S, Kubota J, et al. Number of cervical biopsies and sensitivity of colposcopy. Obstet Gynecol 2006;108:264Y72. 8. Jeronimo J, Schiffman M. Colposcopy at a crossroads. Am J Obstet Gynecol 2006;195:349Y53. 9. Bucchi L, Falcini F, Schincaglia P, Desiderio F, Bondi A, Farneti M, et al. Performance indicators of organized cervical screening in Romagna (Italy). Eur J Cancer Prev 2003;12:223Y8. 10. Prandi S, Beccati D, De Aloysio G, Fulgenzi P, Gabrielli M, Ghirardini C, et al. Applicability of the Bethesda System 2001 to a public health setting. Cancer Cytopathol 2006;108:271Y6. 11. Cristiani P, De Nuzzo M, Costa S, Prandi S, Davi D, Turci M, et al. Follow-up of screening patients conservatively treated for cervical intraepithelial neoplasia grade 2Y3. Eur J Obstet Gynecol Reprod Biol 2007;133:227Y31. 12. Bucchi L, Cristiani P, Costa S, Schincaglia P, Garutti P, Sassoli de Bianchi P, et al. Rationale and development of an on-line quality assurance programme for colposcopy in a population-based cervical screening setting in Italy. BMC Health Serv Res 2013;13:237. 13. Walker P, Dexeus S, De Palo G, Barrasso R, Campion M, Girardi F, et al. International terminology of colposcopy: an updated report from the International Federation for Cervical Pathology and Colposcopy. Obstet Gynecol 2003;101:175Y7. 14. Soliani L. Statistica univariata e bivariata, parametrica e non parametrica per le discipline ambientali e biologiche [Parametric and Nonparametric Univariate and Bivariate Statistics for Environmental and Biological Disciplines]. Parma, Italy: UNI.NOVA, Universita` di Parma; 2005. 15. Sideri M, Spolti N, Spinaci L, Sanvito F, Ribaldone R, Surico N, et al. Interobserver variability of colposcopic interpretations and consistency with final histologic results. J Low Genit Tract Dis 2004;8:212Y6. 16. Sellors JW, Jeronimo J, Sankaranarayanan R, Wright TC, Howard M, Blumenthal PD. Assessment of the cervix after acetic acid wash: inter-rater agreement using photographs. Obstet Gynecol 2002;99:635Y40. 17. Cagle AJ, Hu SY, Sellors JW, Bao YP, Lim JM, Li SM, et al. Use of an expanded gold standard to estimate the accuracy of colposcopy and visual inspection with acetic acid. Int J Cancer 2010;126:156Y61. 18. Bentley J. Colposcopic management of abnormal cervical cytology and histology. J Obstet Gynaecol Can 2012;34:1188Y202.
313
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Objective: To report the results of an Internet-based colposcopy quality assurance program from a population-based cervical screening service in a large region of northern Italy. Methods: In 2010 to 2011, a Web application was made accessible on the Web site of the regional administration. Fifty-nine colposcopists of the registered 65 participated. They logged-in, viewed a posted set of 50 high-quality digital colpophotographs selected by an expert committee, and rated them for colposcopic impression using a 4-tier classification (Negative; abnormal, grade 1 [G1]; abnormal, grade 2 [G2]; suspected invasive cancer [Cancer]) derived from the International Federation for Cervical Pathology and Colposcopy 2002 classification. kappa (J) coefficients for intercolposcopist agreement and colposcopist-committee agreement were calculated. Results: Colposcopist-committee agreement was greater than intercolposcopist agreement (overall J 0.69 vs 0.60, p G .001). The J values for colposcopist-committee agreement were 0.83 on Negative, 0.53 on G1, 0.66 on G2, and 0.80 on Cancer (all p values for pairwise comparisons G.001, except for Negative vs Cancer [ p = .078]). There was no systematic tendency for colposcopists to underestimate or overestimate the colposcopic findings (2-tailed sign test, p = .13). Overall colposcopistcommittee agreement was greater among patients 35 years or older ( p G .001) and for colposcopists with previous quality assurance experiences ( p G .01). Only 0.2% of Negative impressions were formulated for a cervical intraepithelial neoplasia grade 2 or worse. As a parallel finding, the impression of Cancer predicted cervical intraepithelial neoplasia grade 2 or less in 0.5% of cases. The histologic substrates of G1 were dispersed over a large spectrum. Conclusions: The reproducibility of colposcopic impression, when classified by trained colposcopists examining high-quality images, is higher than is generally thought. Key Words: quality assurance, colposcopy, cervical cancer screening, interobserver agreement, Internet (J Lower Gen Tract Dis 2014;18: 309Y313)
C
olposcopy is a critical step in the management of women with abnormal Pap smear results. However, the technique has often been reported to be associated with a low interobserver agreement in interpreting cervical lesions1Y4 and a high biopsy sampling error rate. Incorrect selection of biopsy site 1
Cervical Cancer Screening Unit, Bologna Health Care District, Bologna; Department of Obstetrics and Gynaecology, St. Orsola Hospital, Bologna; 4 3 Cancer Prevention Center, Ravenna Health Care District, Ravenna; Depart5 ment of Obstetrics and Gynaecology, University Hospital, Ferrara; Department of Health, Regione Emilia-Romagna, Bologna; 6Preventive Gynaecology Unit, European Institute of Oncology, Milan; and 7Romagna Cancer Registry, Romagna Cancer Institute (IRST), Meldola, Forlı`, Italy Reprint requests to: Lauro Bucchi, MD, Romagna Cancer Registry, IRCCS IRST, 47014 Meldola, Forlı`, Italy. E-mail: [email protected] The authors declare that they have no conflicts of interest. * 2014, American Society for Colposcopy and Cervical Pathology 2
Journal of Lower Genital Tract Disease
&
may lead to low sensitivity for high-grade cervical intraepithelial neoplasia (CIN).5Y7 There are experts who believe that, until these deficiencies are recognized and corrected, colposcopically guided biopsy cannot be considered the reference standard for the diagnosis of cervical disease.8 The proposed approaches to accomplish this task include the development of computer-assisted diagnosis, the collection of additional biopsy specimens from the abnormal cervical area (or areas), the collection of random specimens from all quadrants, and the identification and diffusion of a limited set of well-defined and highly reproducible colposcopic features of disease.8 The latter is a challenging process because it requires the widespread use of a standard classification, sufficient formal and practical apprenticeship, and quality assurance (QA) programs involving the largest possible number of colposcopists.8 These programs should be based on interactive retraining sessions and large agreement and accuracy studies to identify reproducible colposcopic features of cervical abnormalities and improve the single colposcopist’s competence as well as the appropriateness of his/her clinical decisions. These are also the objectives of a colposcopy QA program that is underway in the Emilia-Romagna Region of northern Italy in the context of a local population-based cervical screening service. In this article, we report the results on colposcopic impression.
MATERIALS AND METHODS Setting Detailed information of cervical screening service in the Emilia-Romagna Region is reported elsewhere.9Y11 Colposcopy assessment for women with abnormal Pap smear results is carried out by specially appointed gynecologists and gynecologistoncologists. QA activities are designed by experts from the screening centers (n = 11) and the public hospitals and are supervised by the regional Department of Health. For the past decade, several on-site QA initiatives were implemented for screening colposcopists, including courses on the distinction of abnormal from normal transformation zone and the grading of colposcopic abnormalities.
Design Between 2009 and 2010, an online QA program was developed. The program had no administrative functions (ranking, accreditation, etc.) and placed more emphasis on providing education and feedback to participants than on testing them. Participation was on a voluntary basis. A log-in Internet application was created and made accessible on the Web site of the regional administration. Between December 2010 and February 2011, screening colposcopists were invited to register, log-in, view a posted set of 50 colpophotographs selected by an expert steering committee, and classify them according to colposcopic impression, visibility of
Volume 18, Number 4, October 2014
309
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Journal of Lower Genital Tract Disease
Cristiani et al.
the squamocolumnar junction (SCJ), and need for biopsy. They were also asked to indicate the most appropriate site for biopsy with a left-button mouse click on the colposcopic image. In May 2011, a plenary seminar was held to discuss the results and to perform an interactive review of the test set of images. A detailed and extensive protocol of the program was published in an open-access journal.12 Given the accessibility of the document, only essential information is provided here.
Selection of the Test Set of Colpophotographs Two hundred fifty high-definition digital colpophotographs from a consecutive series of women with abnormal Pap smear results attending 2 random district screening centers were obtained. Technical specifications of the colposcopes, cameras, and video editing softwares that were used can be found in our previous article.12 The images were stored in the jpeg format. From this basic set, 50 images were selected based on the following criteria: (1) they were well representative of major normal and abnormal colposcopic findings; (2) they were of high technical quality; (3) the cervix was entirely visible; (4) there were no light reflections, color artifacts, shaded areas, or mucus accumulation; and (5) the patient had not been treated previously. The rationale for these criteria is discussed in our previous article.12
Classification of Colposcopic Impression Each colpophotograph in the test set was accompanied by a caption with information about the patient’s age, last Pap smear result, and human papillomavirus test result (which was actually performed only in 4 patients). Colposcopic impression was classified into 4 categories: Negative; abnormal, grade 1 (G1); abnormal, grade 2 (G2); and suspected invasive cancer (Cancer). Despite the simplified terminology, these categories were equivalent to the colposcopic patterns that the International Federation for Cervical Pathology and Colposcopy classification of 200213 designated as follows: normal colposcopic findings; abnormal colposcopic findings, minor changes; abnormal colposcopic findings, major changes; and colposcopic features suggestive of invasive cancer.
Data Analysis Data analysis was based on the observed percent agreement and the Cohen J coefficient and was centered on colposcopist-committee agreement. Given the high level of expertise of gynecologists and gynecologist-oncologists in the committee, this was considered a highly reliable reference but not an absolute one, thus justifying the use of agreement statistics.
&
Volume 18, Number 4, October 2014
Specific J values (e.g., for patient’s age group) and overall J values were used. The colposcopist’s age was used as a proxy for the years of professional experience. In the estimation of overall J values (a measure of overall agreement among all observers for all categories considered), one in each pair was chosen randomly as reviewer 1 and the other as reviewer 2. Comparison between J values was performed using the method by Soliani.14 The 2-tailed nonparametric sign test was used to determine whether colposcopists tended to significantly underrate or overrate the severity of colposcopic findings as compared with the committee. The degree and direction of disagreement were assessed by cross-tabulating the distribution of colposcopic impressions as formulated by the colposcopists compared to those formulated by the committee. Although colposcopic impression is not a direct predictor of histologic grading of a lesion,12 analysis included a crosstabulation with histology diagnoses to determine the approximate probability for a woman with an abnormal Pap smear result to receive a false-negative and false-positive colposcopy evaluation. When multiple biopsies and surgical specimens were available, the most severe diagnosis was considered.
RESULTS According to colposcopic impressions formulated by selectors, the test series of 50 images included 17 Negative cases, 10 cases of G1, 19 cases of G2, and 4 cases of Cancer. The median patient’s age was 35 years (range, 23Y58 y). Of the total staff of 65 colposcopists, 60 participated. Because one of these examined only part of the images, data were analyzed for 59 participants (median age, 56 y; range, 32Y63 y). Forty-five (76%) of these had participated in previous on-site QA initiatives. With 59 colposcopists and 50 colposcopic images, there were a total of 1,711 pairs of colposcopists and 85,550 paired observations. The number of paired colposcopistcommittee observations was 2,950. Both the overall and the impression-specific J values were significantly greater for colposcopist-committee agreement than for intercolposcopist agreement (see Table 1). The J values for Negative and Cancer were the highest and that for G1 was the lowest. The J values for colposcopist-committee agreement on Negative, G1, G2, and Cancer were mutually significantly different ( p G .001), with the exception of Negative vs Cancer ( p = .078). The distribution of impressions formulated by the colposcopists compared to those formulated by the committee (see Table 2) shows in greater detail that G1 was the least consistent impression. Over the entire table, however, negative
TABLE 1. Intercolposcopist Agreement and Colposcopist-Committee Agreement
Colposcopic impression Negative G1 G2 Cancer Overall
Intercolposcopist agreementa
Colposcopist-committee agreementb
Crude %
J
Crude %
J
p for comparison between J values
90 77 80 97 72
0.76 0.44 0.56 0.74 0.60
93 82 84 97 78
0.83 0.53 0.66 0.80 0.69
G.001 G.001 G.001 G.01 G.001
Cancer indicates suspected invasive cancer; G1, abnormal, grade 1; G2, abnormal, grade 2. a Data were based on 59 colposcopists and 50 colposcopic images for a total of 1,711 pairs of colposcopists and 85,550 paired observations. b Data were based on a total 2,950 paired observations.
310
* 2014, American Society for Colposcopy and Cervical Pathology
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Journal of Lower Genital Tract Disease
&
Volume 18, Number 4, October 2014
Colposcopy Quality Assurance in Italy
TABLE 2. Distribution of Colposcopic Impressions Formulated by the Colposcopists According to the Impressions Formulated by the Committee Impression formulated by the committee
Impression formulated by the colposcopists
Negative
G1
G2
Cancer
Total
Negative G1 G2 Cancer Total
826 (95.8) 139 (16.5) 38 (3.7) 0 (0.0) 1,003
34 (3.9) 457 (54.2) 98 (9.4) 1 (0.5) 590
2 (0.2) 246 (29.2) 849 (81.6) 24 (11.8) 1,121
0 (0.0) 1 (0.1) 56 (5.4) 179 (87.7) 236
862 (100.0) 843 (100.0) 1,041 (100.0) 204 (100.0) 2,950a
Cancer indicates suspected invasive cancer; G1, abnormal, grade 1; G2, abnormal, grade 2. Numbers in parentheses are percentages. Two-tailed nonparametric sign test: negative differences (with a lower grade being reported by colposcopists) n = 339; positive differences n = 300; Z statistic = j1.503; 2-tailed p = .13. a Number of paired observations.
differences between the screening colposcopists and the committee did not significantly exceed positive differences, which indicated the absence of a systematic component in disagreement. Table 3 shows the factors associated with colposcopistcommittee agreement. The J values for G1 and G2 were higher among images obtained from patients 35 years or older. An impression of G2 was more likely to be agreed on by the committee if the referral Pap smear showed high-grade atypia. The consistency of the impression of Negative was greater if associated with total or partial invisibility of the SCJ, whereas the opposite occurred for G2. The colposcopist’s age had no effect whatsoever on the results, whereas participation in previous QA initiatives was associated with increased agreement on G1 and G2. As shown in Table 4, Negative exhibited an almost perfect predictive value. Only 0.2% of Negative impressions were formulated for a CIN2 or worse (representing discrepancies of 2 or more degrees). As a parallel finding, the impression of Cancer predicted CIN2 or less in 0.5% of cases. The histologic substrates of G1 were dispersed over a large spectrum of lesions. No predominant counterpart was observed because the prevalence of normal histology was of the same order of magnitude as CIN1 and CIN3/adenocarcinoma in situ (AIS). Approximately one fourth of patients harbored CIN3 or worse. Data were then stratified according to whether the impression of G1 was agreed on by less or more than 50% of observers. In the latter subset of colposcopic impressions, no case of carcinoma was observed and CIN1 reached a prevalence of 45%. G2 predicted CIN3/AIS in two thirds of cases, and normal histology/CIN1 in less than 15% of cases. Among colposcopic impressions agreed on by more than 50% of observers, the prevalence of normal histology/CIN1 was only 6% and that of CIN3/AIS was greater than 70%.
DISCUSSION Although there have been studies reporting excellent levels of agreement between colposcopists,15 our results are generally better than those of most previous comparable investigations.2,3,16 In fact, agreement studies are always limited by the lack of an absolute gold standard, the subjectivity of the criteria for the selection of colposcopic images, and the artificiality of the test conditions. Whatever solutions are adopted, these problems are impossible to resolve completely.12 For this * 2014, American Society for Colposcopy and Cervical Pathology
reason, results cannot be safely used for external comparisons with other studies or for central control and administrative purposes. The images used for this program were free of imperfections and were considered by selectors to be well representative of the spectrum of colposcopic findings. The rationale for these TABLE 3. Factors Affecting J Values for Agreement Between the Colposcopists and the Committee Impression Factors
Negative
G1
G2
Cancer Overall
0.83 0.84
0.35 0.66b
0.57 0.72b
NC 0.82
0.60 0.76b
0.83 0.78
0.53 0.47
0.60 0.68c
NC 0.79
0.66 0.68
0.89
0.51
0.62
0.81
0.70
0.64b
0.56
0.80b
NC
0.67
0.83 0.86 0.82
0.51 0.52 0.49
0.65 0.65 0.64
0.77 0.82 0.79
0.68 0.70 0.63
0.86
0.57e
0.72e
0.81
0.71c
a
Patient age G35 y Q35 y Pap smear result ASCUS, LSIL HSIL, AGC, AIS Squamocolumnar junction Not or not entirely visible Entirely visible Colposcopist characteristics Aged 32Y50 yd Aged 59Y63 y With no previous QA experience With previous QA experience
AGC indicates atypical glandular cells; AIS, adenocarcinoma in situ; ASCUS, atypical squamous cells of undetermined significance; Cancer, suspected invasive cancer; G1, abnormal, grade 1; G2, abnormal, grade 2; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NC, not calculated because the committee did not formulate an impression of cancer for any image in this stratum. Data were based on a total of 2,950 paired observations. a Patients were divided by their median age. b p G .001. c p G .01 d The 12 youngest colposcopists were compared with the 12 oldest ones. e p G .05.
311
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Journal of Lower Genital Tract Disease
Cristiani et al.
&
Volume 18, Number 4, October 2014
TABLE 4. Predictive Value of Colposcopic Impressions Formulated by the Colposcopists for Histology Histologic diagnosis Impression Negative G1a Agreed by G50% of observers Agreed by Q50% of observers Total G2a Agreed by G50% of observers Agreed by Q50% of observers Total Cancer Total
Normal
CIN1
CIN2
CIN3/AIS
Carcinoma
Total
795 (92.2)
65 (7.5)
0 (0.0)
1 (0.1)
1 (0.1)
862 (100.0)
121 (40.1) 109 (20.1) 230 (27.3)
56 (18.5) 243 (44.9) 299 (35.5)
22 (7.3) 62 (11.5) 84 (10.0)
77 (25.5) 127 (23.5) 204 (24.2)
26 (8.6) 0 (0.0) 26 (3.1)
302 (100.0) 541 (100.0) 843 (100.0)
37 (17.2) 0 (0.0) 37 (3.6) 0 (0.0) 1,062
58 (27.0) 49 (5.9) 107 (10.3) 1 (0.5) 472
56 (26.0) 96 (11.6) 152 (14.6) 0 (0.0) 236
59 (27.4) 597 (72.3) 656 (63.0) 83 (40.7) 944
5 (2.3) 84 (10.2) 89 (8.5) 120 (58.8) 236
215 826 1,041 204 2,950b
(100.0) (100.0) (100.0) (100.0) (100.0)
AIS indicates adenocarcinoma in situ; Cancer, suspected invasive cancer; CIN, cervical intraepithelial neoplasia; G1, abnormal, grade 1; G2, abnormal, grade 2. Numbers in parentheses are percentages. a Colposcopic images were divided according to whether the impression was agreed by less or more than 50% of observers. b Number of paired observations.
criteria was that a QA program should target the basic presentations of normal and affected tissues of the cervix with which to compare the spectrum of colposcopic patterns encountered in practice.12 This condition further prevents extrapolation of results to daily practice. However, and more importantly, it reveals the potential reproducibility of colposcopic impression. If used by trained colposcopists examining high-quality images, a simplified and rational classification scheme can reach good levels of reproducibility. Another important finding was that interobserver agreement in a population of colposcopists is lower than between these and a panel of experts (see Table 1). Given the high number of colposcopists in this program, their range of variability was relatively large, and the experts’ approach was probably based on intermediate interpretative criteria. This would explain why colposcopists were more likely to agree with the committee than with each other. Data in Table 2 are in accordance with this hypothesis. There was no systematic component in disagreement. Negative differences in impression between the colposcopists and the committee, representing underestimation, were not significantly more than positive differences. In other words, the limited excess of negative differences was compatible with chance variation. We identified several factors affecting the colposcopistcommittee agreement (see Table 3). First, the level of agreement on G1 and G2 was greater if images were from patients 35 years or older. In younger patients, the maturation of the metaplastic epithelium may be asynchronous, and because of this, immature areas may be combined with more mature sections. If immature metaplastic areas react intensely to acetic acid and are uniformly iodine-negative, they may give an impression of intraepithelial lesion, particularly when they exhibit a fine punctation or mosaic due to stromal projections centered by capillaries. Second, the Negative impression was more consistent if associated with total or partial invisibility of the SCJ. Identifying the original squamous and glandular epithelium is relatively easy. Difficulties increase when the transformation zone shows complex squamous aspects because benign and abnormal colposcopic findings differ only in slight details. When the
312
transformation zone appears as a mature squamous epithelium confining the SCJ in the endocervical canal, interobserver agreement increases. Conversely, when there are areas of persistent immature metaplasia and the SCJ is on the ectocervix, variation in interpretation is larger. Third, although colposcopist’s ageVa proxy of the number of years of professional experienceVhad no influence whatsoever, participation in previous specific QA initiatives had a significant positive effect on the agreement on both G1 and G2, that is, the least consistent colposcopic impressions. These important findings strongly support the importance of active measures of QA and training. Although colposcopic impression is not a direct correlate of histologic grading,8,17 the (negative) predictive value of Negative and the positive predictive value of an impression of Cancer can reach high levels (see Table 4). No more than 0.5% of patients with colposcopic images interpreted as Negative and Cancer were diagnosed, respectively, with CIN2 or worse and CIN2 or less, representing discrepancies of 2 or more degrees in opposite directions. The observed predictive value of Negative is particularly worthy of note. It suggests that, when colposcopy is correctly interpreted as showing no abnormal findings, taking random biopsies would not increase appreciably the probability of detecting a high-grade disease. G1 was the least specific condition. Except for a low prevalence of invasive carcinoma, the underlying lesions followed a virtually random distribution. In particular, approximately one fourth of patients were diagnosed with CIN3 or worse, including some lesions that were located in the endocervix and were not visible. These findings emphasize the importance of examining through biopsy those cervical abnormalities that are colposcopically interpreted as G1. Current guidelines recommend that 2 or more biopsy specimens should be taken, coupled with an endocervical curettage when the transformation zone is not visible and in women older than 45 years with high-grade cytology.18 In a separate article, we will report the results of the QA program on the performance of biopsy, including the frequency by which the biopsy site was correctly chosen on digital colpophotographs, which can be taken as a proxy of biopsy sensitivity. * 2014, American Society for Colposcopy and Cervical Pathology
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
Journal of Lower Genital Tract Disease
&
Volume 18, Number 4, October 2014
Both for G1 and G2, the consistency of histologic counterparts increased when the impression was agreed on by most colposcopists. The fact that abnormalities interpreted in the same way by most observers had a greater level of specificity suggests that they met shared definitions of G1 and G2. Of course, they were probably easier to recognize. However, this is equivalent to saying that the classic features of G1 and G2 have a higher predictive value and that their recognition makes colposcopic impressions more specific. This, in turn, confirms the validity of the classification scheme used in the current study. In conclusion, the current study demonstrated that the theoretical reproducibility of colposcopic impression is higher than is generally thought. This means that the low level of performance that is often found in colposcopy practice is at least partly due to insufficient training. The impression of G1 was found to be the least consistent one, with a considerable prevalence of high-grade lesions. This suggests that training should pay special attention to the recognition of low-grade colposcopic findings and that cervical abnormalities interpreted as such are an indication for biopsy. REFERENCES 1. Sellors JW, Nieminen P, Vesterinen E, Paavonen J. Observer variability in the scoring of colpophotographs. Obstet Gynecol 1990;76:1006Y8. 2. Etherington IJ, Luesley DM, Shafi MI, Dunn J, Hiller L, Jordan JA. Observer variability among colposcopists from the West Midlands region. Br J Obstet Gynaecol 1997;104:1380Y4. 3. Ferris DG, Litaker M. Interobserver agreement for colposcopy quality control using digitized colposcopic images during the ALTS trial. J Low Genit Tract Dis 2005;9:29Y35. 4. Massad LS, Jeronimo J, Schiffman M. Interobserver agreement in the assessment of components of colposcopic grading. Obstet Gynecol 2008;111:1279Y84. 5. ASCUS-LSIL Triage Study (ALTS) Group. Results of a randomized trial on the management of cytology interpretations of atypical squamous cells of undetermined significance. Am J Obstet Gynecol 2003;88:1383Y92. 6. Pretorius RG, Zhang WH, Belinson JL, Huang MN, Wu LY, Zhang X, et al. Colposcopically directed biopsy, random cervical biopsy, and endocervical curettage in the diagnosis of cervical intraepithelial neoplasia II or worse. Am J Obstet Gynecol 2004;19:430Y4.
* 2014, American Society for Colposcopy and Cervical Pathology
Colposcopy Quality Assurance in Italy
7. Gage JC, Hanson VW, Abbey K, Dippery S, Gardner S, Kubota J, et al. Number of cervical biopsies and sensitivity of colposcopy. Obstet Gynecol 2006;108:264Y72. 8. Jeronimo J, Schiffman M. Colposcopy at a crossroads. Am J Obstet Gynecol 2006;195:349Y53. 9. Bucchi L, Falcini F, Schincaglia P, Desiderio F, Bondi A, Farneti M, et al. Performance indicators of organized cervical screening in Romagna (Italy). Eur J Cancer Prev 2003;12:223Y8. 10. Prandi S, Beccati D, De Aloysio G, Fulgenzi P, Gabrielli M, Ghirardini C, et al. Applicability of the Bethesda System 2001 to a public health setting. Cancer Cytopathol 2006;108:271Y6. 11. Cristiani P, De Nuzzo M, Costa S, Prandi S, Davi D, Turci M, et al. Follow-up of screening patients conservatively treated for cervical intraepithelial neoplasia grade 2Y3. Eur J Obstet Gynecol Reprod Biol 2007;133:227Y31. 12. Bucchi L, Cristiani P, Costa S, Schincaglia P, Garutti P, Sassoli de Bianchi P, et al. Rationale and development of an on-line quality assurance programme for colposcopy in a population-based cervical screening setting in Italy. BMC Health Serv Res 2013;13:237. 13. Walker P, Dexeus S, De Palo G, Barrasso R, Campion M, Girardi F, et al. International terminology of colposcopy: an updated report from the International Federation for Cervical Pathology and Colposcopy. Obstet Gynecol 2003;101:175Y7. 14. Soliani L. Statistica univariata e bivariata, parametrica e non parametrica per le discipline ambientali e biologiche [Parametric and Nonparametric Univariate and Bivariate Statistics for Environmental and Biological Disciplines]. Parma, Italy: UNI.NOVA, Universita` di Parma; 2005. 15. Sideri M, Spolti N, Spinaci L, Sanvito F, Ribaldone R, Surico N, et al. Interobserver variability of colposcopic interpretations and consistency with final histologic results. J Low Genit Tract Dis 2004;8:212Y6. 16. Sellors JW, Jeronimo J, Sankaranarayanan R, Wright TC, Howard M, Blumenthal PD. Assessment of the cervix after acetic acid wash: inter-rater agreement using photographs. Obstet Gynecol 2002;99:635Y40. 17. Cagle AJ, Hu SY, Sellors JW, Bao YP, Lim JM, Li SM, et al. Use of an expanded gold standard to estimate the accuracy of colposcopy and visual inspection with acetic acid. Int J Cancer 2010;126:156Y61. 18. Bentley J. Colposcopic management of abnormal cervical cytology and histology. J Obstet Gynaecol Can 2012;34:1188Y202.
313
Copyright © 2014 American Society for Colposcopy and Cervical Pathology. Unauthorized reproduction of this article is prohibited.
