Accepted Manuscript Overview of the repeatability, reproducibility and agreement of the biometry values provided by various ophthalmic devices Jos J. Rozema, Kristien Wouters, Danny GP. Mathysen, Marie-José Tassignon PII:
S0002-9394(14)00495-4
DOI:
10.1016/j.ajo.2014.08.014
Reference:
AJOPHT 9027
To appear in:
American Journal of Ophthalmology
Received Date: 14 March 2014 Revised Date:
8 August 2014
Accepted Date: 10 August 2014
Please cite this article as: Rozema JJ, Wouters K, Mathysen DG, Tassignon M-J, Overview of the repeatability, reproducibility and agreement of the biometry values provided by various ophthalmic devices, American Journal of Ophthalmology (2014), doi: 10.1016/j.ajo.2014.08.014. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Abstract
AC C
EP
TE D
M AN U
SC
RI PT
Purpose: To present an overview of the measurement errors for various biometric devices, as well as a meta-analysis of the agreement between biometric devices using the Pentacam, Orbscan and IOL Master as a reference. Design: Meta-analysis of the literature Methods: The meta-analysis is based on data from 216 articles that compare a total of 24 different devices with the reference devices for the following 9 parameters: mean, steep and flat curvature of the anterior and posterior cornea, central corneal thickness, anterior chamber depth and axial length. After the weighted average difference between devices have been determined, the two one-sided t test was used to test for equivalence between devices within certain thresholds defined by the measurement errors and the influence of these differences on the calculated refraction. Results: In only 17 of the 70 comparisons a device was equivalent with the reference device within the thresholds set by the measurement error. More lenient thresholds, based on a change in calculated refraction of ±0.25D, increased this number to a maximum of 25 / 50 comparisons (excluding pachymetry). High degrees of inconsistency were seen in the reported results, which could partially explain the low agreement between devices. Conclusion: As a rule biometry measurements taken by different devices should not be considered equivalent, although several exceptions could be identified. We therefore recommend that clinical studies involving multiple device types treat this as a within-subject variable to avoid bias. The follow-up of individual patients using different devices should be avoided at all times.
ACCEPTED MANUSCRIPT
Overview of the repeatability, reproducibility and agreement of the biometry values provided by various ophthalmic devices
RI PT
Jos J. Rozema,*† Kristien Wouters,‡ † Danny GP Mathysen,*† Marie-José Tassignon*† *
Dept of Ophthalmology, Antwerp University Hospital, Edegem, Belgium Dept of Medicine and Health Science, University of Antwerp, Wilrijk, Belgium ‡ Dept of Scientific Coordination and Biostatistics, Antwerp University Hospital, Edegem, Belgium †
Jos Rozema Department of Ophthalmology, Antwerp University Hospital, Wilrijkstraat 10, 2650 Edegem, Belgium + 32 3 821 48 15 + 32 3 825 19 26 [email protected]
TE D
Tel: Fax: Email:
M AN U
SC
Corresponding author:
Key words Ocular biometry; device equivalence; Pentacam; Orbscan; IOL Master
AC C
EP
Short title Meta-analysis of biometry provided by different devices
Supplemental Material available at AJO.com
1
ACCEPTED MANUSCRIPT
Introduction
AC C
Methods
EP
TE D
M AN U
SC
RI PT
Reliable biometry is an essential part of any ophthalmic practice. For this reason a wide range of instruments have been put on the market, each purporting to have the highest degree of accuracy. But although these devices have undoubtedly been developed and tested according to the state of the art, they will all suffer from errors and sensitivities intrinsic to the measurement process. These present themselves as variations between consecutive acquisitions, and between acquisitions made by different observers or devices. For individual clinics this large variety in available equipment is inconsequential, as they usually own only a few instruments operated consistently by a limited number of experienced individuals. This ensures that the best possible biometry is obtained in the clinical follow-up of each patient. But for research projects, often involving any number of clinics working together, each with their own choice of biometry devices, these equipment differences may form an important confounding factor to the study design. Typically this problem is handled in two ways: either only clinics that own a particular ‘preferred’ device are selected for participation, or equipment differences are simply ignored. Both approaches have their disadvantages, though, as selection may exclude clinics that could otherwise make interesting contributions to the study, and ignoring the differences may introduce systematic errors into the analysis. Hence, before ignoring equipment differences and inserting measurements from different devices into a single database, it is imperative that their agreement and measurement errors have been validated statistically. But although this has already been done quite extensively in the literature for many parameters and many combinations of devices, the results in these reports are often contradictory to each other and authors have very different opinions on what may be considered an acceptable error. It is therefore the aim of this work to present an overview of the measurement errors of the biometry devices available in the literature, as well as of the agreement between these devices and three of the most commonly used clinical instruments: the Pentacam Scheimpflug camera (Oculus Optikgeräte, Wetzlar, Germany) and Orbscan scanning slit corneal topographer (Bausch & Lomb, Bridgewater, NJ) for the anterior segment biometry, and the IOL Master (Carl Zeiss, Jena, Germany) for axial length measurements. Any systematic differences identified will then be compared to a number of clinical standards, in order to verify which of these differences may have an actual clinical impact.
Literature search
The literature search was performed in two parts: first a search for reports on the measurement errors of biometric devices, followed by a second, independent search for reports comparing the measurements of biometry devices with those of the reference instruments. Given the number of parameters and number of devices involved in this work, the literature search was performed through combinations of keywords. For the study on the measurement errors these keywords were structured as “[Device] [parameter] [repeatability/ reproducibility/ measurement error]”, with [Device] the name of the biometry device and [Parameter] the parameter of interest. For the meta-analysis of the
2
ACCEPTED MANUSCRIPT
TE D
M AN U
SC
RI PT
comparison between devices the structure of the keywords was “[Device 1] [Device 2] [Parameter]”, with [Device 1] either Pentacam, OrbScan or IOL Master, and [Device 2] the name of another device (e.g. Sirius, Galilei,…) or a generic technological name (e.g. Placido, ultrasound,…). The literature search was performed in June 2014 using PubMed and Google Scholar. It included both English and non-English references published after the year 2000, thus avoiding results obtained using outdated versions of the devices or their software. Since biometric devices are typically calibrated using eye models that are close to the epidemiological average (e.g. the Gullstrand eye model), only articles using healthy adult eyes were considered in this analysis. The only exception were papers comparing corneal or anterior chamber parameters in cataract patients, as these measurements are not influenced by the condition of the lens. For the comparison of axial length measurements, however, cataract patients were excluded. Finally a selection was made based on the way the remaining articles presented their data. For the study on measurement errors only those reporting the standard deviation of the repeated measurements, or parameters that can be calculated back to the standard deviation were included. For the meta-analysis of the comparison both the average and the standard deviation of the difference between devices (or the 95% limits of agreement) had to be reported, along the lines suggested by Altman and Bland,1 and McAlinden et al.2 Note that for the Orbscan a distinction was made between comparative articles that use the “acoustic factor” (AF) to report the central corneal thickness and articles that do not. This AF typically has a value of 0.92 (although different values are used as well) and aims to correct for an overestimation of the corneal thickness by the Orbscan. Also, for the anterior chamber depth (ACD) there were articles that included the cornea into the measurement value and other articles that did. However these articles were not considered separately due to an insufficient number of articles to warrant a separate analysis.
Measurement error
AC C
EP
The measurement error Emeas may be defined by Emeas = (E2repeat + E2reprod)1/2, which is comprised of the repeatability Erepeat, the standard deviation of multiple measurements performed by a single operator, and the reproducibility Ereprod, the standard deviation between measurements performed at different sessions or by different operators. Although in principle there could also be calibration differences between devices of the same type, we were unable to include this factor into our analysis as very little information on this type of error is available.3 In order to obtain a global estimate of the repeatability and reproducibility of the various biometry devices, the numerical values found in the literature review were tabulated (made available as supplemental data at AJO.com; Supplemental Table 1), from which the average of the repeatability and reproducibility, weighted by number of subjects used in each study, was calculated for each available device. From these values the measurement error Emeas was then estimated using the above equation.
Meta-analysis Based on the reported differences found in the literature a series of meta-analyses were performed per parameter and per device pair (i.e. Devices 1 and 2) using the procedures 3
ACCEPTED MANUSCRIPT
RI PT
described by Borenstein et al.4 This provides a global average and standard deviation of the difference between devices, a p value indicating its statistical significance and a coefficient I² to quantify the homogeneity between articles (with values of 0% indicating low homogeneity and 100% high inhomogeneity). The Bonferroni correction is used to adjust for multiple testing. Note that a difference that is not statistically significant from zero does not automatically imply both devices are to be considered equivalent, as the range of the differences may still be wider than what is considered clinically acceptable. For this reason equivalence margins were defined based on either the devices’ measurement error, or several common clinical calculations. These margins were used to test for equivalence by means of a “two one-side t test” (TOST).5, 6
SC
Analysis of significant differences
EP
TE D
M AN U
Although the measurement error of a biometric device (Emeas) may be considered as the lowest possible threshold for clinical importance, providing a reliable and well-founded set of equivalence margins, it is probably too strict for most clinical applications. This means that even though a difference between two devices may be considered too large according the Emeas margins, it may still be clinically inconsequential. For this reason several other, more lenient equivalence margins were defined based on the influence measurement differences would have on typical ophthalmic and optometric calculations (Table 1). These calculations included the refraction Scalc calculated from the ocular biometry using the thick lens formula, and the intraocular lens (IOL) powers calculated using the SRK/T,7 and Haigis8 formulas. Setting the tolerance for the changes in calculated power (∆Scalc) to 0.125D and 0.25D, corresponding with 50% and 100% of the smallest step in spectacle correction and the measurement error on most autorefractometers, we then determined the respective biometric changes that would cause just such a change in the calculations. These changes were then used as clinically based equivalence margins. The biometric parameters required for the calculations were taken from the Gullstrand eye model,9 while the IOL constants were those of the Alcon AcrySof SA60AT spherical, monofocal IOL.
Software
Results
AC C
All calculations were performed using Excel 2010 (Microsoft Corp, Redmond, WA, USA) and OpenMeta[Analyst],10 which is an R based open software package for meta-analysis (available at http://www.cebm.brown.edu/open_meta, accessed August 8, 2014).
Literature search After the selection process detailed in the Figure the literature search resulted in 124 articles for the measurement error analysis, and 127 for the comparison between the reference instruments (Oculus Pentacam, Bausch & Lomb Orbscan and Zeiss IOL Master) and a wide range of
4
ACCEPTED MANUSCRIPT
RI PT
methods, including Scheimpflug, optical coherence tomography (OCT), Placido topography, partial coherence interferometry, immersion ultrasound and applanation ultrasound devices. Note that due to the wide variety of Placido and ultrasound based devices reported in the literature the generic name of the technology was used rather than actual brand names. Also, for a number of less commonly used devices (e.g. Zeiss AC Master and slit lamp with Jaeger attachment) data was not included in the tables presented here. Instead, this information is made available as supplemental data at AJO.com (Supplemental Tables 1−4)
Overview of measurement errors
TE D
M AN U
SC
The combined measurement errors for the mean anterior keratometry Ka,m show a narrow range between 0.12 D (Galilei) and 0.18 D (Pentacam), with one outlier at 0.52 D (Orbscan). This is also seen for the mean posterior keratometry Kp,m, with values of 0.05 D (Pentacam and Galilei), and an outlier at 0.17 D (Orbscan). For the steep and flat keratometry data (Ka,s, Ka,f) the range lies between 0.10 D and 0.24 D, with the lowest error values for the Sirius and the IOL Master. Although from these data it would seem that the Orbscan gives considerably higher values for the mean anterior and posterior keratometry than the other devices, it is important to note that both deviating values were taken from the same study.11 Comparing these Orbscan values to those for Ka,s and Ka,f, which are derived from other studies and lie within the range of the other devices, it is likely that these outliers are coincidental in nature. The measurement error of the central corneal thickness CCT ranged between 1.76 µm (Galilei) and 16.71 µm (Artemis). Anterior chamber depth ACD had errors ranging between 0.05 mm and 0.09 mm, with an outlier at 0.25 mm (ultrasound), and a similar result was seen for the axial length L with error values around 0.02 mm. The repeatability of the ultrasound and the Tomey OA-1000 (0.11 mm and 0.08 mm, respectively) was higher than those of the IOL Master and the Lenstar (0.02 mm).
Meta-analysis of agreement between devices
AC C
EP
A total of 70 comparisons between devices were performed, excluding 7 duplicate comparisons between Pentacam and Orbscan (Tables 3−5). The average difference between devices was found to be significant for 17 comparisons at a significance level of p < 1 − 0.05/73 = 0.00068 (Bonferroni correction against α−inflation), which was in 9 / 17 cases based on data from a single study. As shown for the Pentacam in Table 3 these differences were mostly found for posterior keratometry (Kp,m: Pentacam vs. Sirius and TMS-5; Kp,s: Pentacam vs.Galilei), CCT (Pentacam vs. TMS-5, Orbscan with acoustic factor, Visante/ Stratus, SL-OCT and specular microscopy) and ACD (Pentacam vs. Visante/ Stratus, SL-OCT, Casia SS-1000 and Lenstar). The Orbscan showed fewer significant differences than the Pentacam (Table 4, most notably for Ka,s (Orbscan vs. Galilei), CCT (Orbscan with acoustic factor vs. Pentacam, Orbscan without acoustic factor vs. ultrasound) and ACD (Orbscan vs. Galilei). Finally, the IOL Master only showed a significant difference for the axial length with the Nidek AL-scan (Table 5). The meta-analysis also indicated that the homogeneity between studies is remarkably low in the 47 comparisons involving two or more articles, with only 5 / 47 I²-coefficients below 50% and 34 / 47 coefficients above 95%. This is also seen in the Forest plots for these comparisons (available as a supplemental file at AJO.com) and the often wide range of reported differences in 5
ACCEPTED MANUSCRIPT
the included articles (available as a supplemental data at AJO.com, Supplemental Tables 2−4). For this reason the results of this meta-analysis should be interpreted with caution.
Equivalence of results and clinical equivalence
AC C
EP
TE D
M AN U
SC
RI PT
The significant differences found in the meta-analysis cannot be used as an indication of (non)equivalence of devices, nor does it indicate the clinical relevance of the differences. For this reason a TOST procedure was performed using the thresholds of clinical equivalence in Table 1, which yielded the results presented in the last three columns of Tables 3−5. Using the thresholds defined by the measurement errors the TOST procedure demonstrated equivalence for only 17 / 70 comparisons (again, excluding duplicate comparisons between Pentacam and Orbscan). The thresholds based on a refraction change ΔScalc within ±0.125D demonstrated equivalence for 9 / 50 comparisons, which improved to 25 / 50 for a more lenient ΔScalc within ±0.25D. The thresholds for SRK/T IOL power change within ±0.125D equivalence was seen in only 3 / 28 comparisons, and in 10 / 28 comparisons for a power change within ±0.25D. For the Haigis IOL power these numbers were 15 / 44 and 22 / 44, respectively. Considering the most lenient thresholds in Table 1 (i.e. refraction change ΔScalc within ±0.25D) the Pentacam was found equivalent with Placido for the anterior keratometry (Ka,m, Ka,s, Ka,f) and ultrasound for CCT. Similarly the Pentacam was equivalent for some keratometry parameters with the Galilei (Ka,s, Kp,m, Kp,s, Kp,f), and for two parameters of the Sirius (Ka,f, ACD). The Pentacam and Orbscan were not found to be equivalent with each other for any the parameters considered. The Orbscan was however equivalent with the Galilei for the anterior keratometry (Ka,s and Ka,f). Finally the IOL Master was equivalent with the Lenstar, AL-Scan and the Aladdin for the axial length. For CCT an equivalence could only be found between ultrasound and the Pentacam at the threshold defined by the CCT measurement error (±8.12 µm). Even if this tolerance were doubled to ±16.24 µm, which would be considered clinically unacceptable for e.g. the follow-up of keratoconus, the number of equivalences would only marginally improve to 5 / 20 comparisons (Pentacam vs. Sirius, ultrasound and Lenstar, and Orbscan vs. ultrasound (with AF) and Artemis).
6
ACCEPTED MANUSCRIPT
Discussion
M AN U
SC
RI PT
This work demonstrates that, as a rule, measurements taken by different biometry devices should not be considered equivalent. Instead one should be aware that there may be differences between devices that can be detected through meta-analyses, or inconsistent variations between devices that may be picked up by the TOST procedure. Especially the latter is of great importance when one wishes to demonstrate the equivalence of two devices. Usually this is done by showing that the 95% CI of the average difference remains entirely within a predefined set of clinical thresholds representing the maximal discrepancy one is willing to accept (Table 1). Looking at the equivalence of the various devices with the Pentacam at the tolerance level of ±0.25D refractive change (last column of Table 3) it is seen that the Pentacam is equivalent with Placido for the anterior keratometry and with ultrasound for CCT, which are by many considered as the gold standard devices for these parameters. Given that these particular results are based on a large number of studies (15 and 35, respectively), this would suggest that the Pentacam conforms to these gold standards as well. The Galilei is equivalent with the Pentacam for four keratometric parameters, while the Sirius is equivalent with the Pentacam for all parameters except Ka,m and Ka,s. For the Orbscan only an equivalence with the Galilei was found for the keratometry. For the axial length the IOL Master is equivalent with the Lenstar, AL-Scan and Aladdin, but not to contact ultrasound.
AC C
EP
TE D
There are however a number of limitations to the analyses above, most important of which is the very high inconsistency in reported values (inhomogeneity I² in Tables 3−5). One typical example is the difference in Ka,m measurements reported by the Pentacam and the Orbscan, which differs by +0.27D according one study12 and by −0.69D according to another study.13 Although both studies clearly indicate a large and statistically significant difference between devices, their weighted average difference was not significantly different from zero due to their opposite signs and relative cohort sizes. Such large inhomogeneities makes that the results above are difficult to interpret and should be considered with caution. An important source of between-study inconsistencies is calibration differences in devices of the same type. Although at the time of production the manufacturer calibrated all devices according to the same standards, this calibration is not always preserved once the device is placed in an everyday clinical setting. As in such a setting devices may be knocked accidentally, moved around, or used in conditions different from what is described in the instruction manual, the actual calibration could drift over time from the original factory setting. This issue is however not limited to the comparison of devices, but also applies to every study that uses two or more devices of the same type, or studies that use the same machine for a long period of time. For this reason it would be recommended that device calibration is verified prior to and during any multicenter study. Alternatively, individual centers could be considered as a between-subjects factor in the statistical analysis, even if they have devices of the same type. A second source of inconsistency in the literature may be sign errors in the cited articles. Even though great care was taken to avoid these by comparing tables, graphs and text of the original manuscripts, and double checking articles with strongly deviating values, it is conceivable that a small number of sign errors may have gone unnoticed.
7
ACCEPTED MANUSCRIPT
M AN U
SC
RI PT
Another aspect to this study was that it aimed to find the agreement between the reference devices (Pentacam, Orbscan and IOL Master) and a wide range of other devices. This may however not be considered as a means to demonstrate the equivalence of two other devices that were each found equivalent with the reference devices. Similarly, in the situation where one device is e.g. equivalent with the Pentacam and another one is not, this cannot automatically lead to the conclusion that both devices are not equivalent. In both cases the conclusion can only be derived from a direct comparison between those specific devices, which in many cases can be found in the literature. It is also important to note that the clinical threshold values calculated in Table 1 only estimate how a change in one single parameter influences the calculated refraction. In a real-life setting, however, each of the parameters used in the calculation would have an influence on the refraction, which leads to a larger, compounded error. Through error propagation, we found that the combined error is about 73% higher than the threshold, meaning that if all thresholds were chosen to accept a refractive change of ±0.25D, this would correspond with a compounded error of ±0.434D. A similar increase was seen for the ±0.125D threshold. Finally, as all biometric devices make assumptions regarding intraocular distances and refractive indices based on idealized models of healthy adult eyes, only this type of eyes were included in the analysis. As these assumed parameter values of healthy adults may deviate considerably from those of children or pathological subjects it is conceivable that devices might react differently to eyes that deviate too far from the original assumptions. This could theoretically cause two devices to produce different results in pathological or children’s eyes, even if the results are in agreement for healthy adults. This could be an interesting subject for a follow-up study.
AC C
EP
TE D
This work is not the first attempt to provide an overview of the measurement errors and agreement between biometry devices in the literature. Earlier Doughty and Jonuscheit14 performed an analysis on 46 studies comparing average pachymetry values provided by contact ultrasound and the Orbscan with the aim to assess whether the use of the acoustic factor is warranted. They reported a very large variation between studies, and concluded that this variation essentially invalidates the use of the acoustic factor. As is seen in Table 4 we also found a large variation, but also a clear difference between studies that used the acoustic factor and studies that did not (-4.56 µm, SEM 1.71 µm and 30.22 µm, SEM 5.81 µm, respectively). This inconsistency between this work and that of Doughty and Jonuscheit may be due to methodical differences (analysis of mean values vs. a meta-analysis and TOST). Wu et al. also performed a meta-analysis15 using 19 studies to compare pachymetry values measured by Pentacam and ultrasound in normal eyes, and came to the conclusion that measurements of both devices were very similar. Their reported difference of 1.47 µm (SEM 1.93 µm) corresponds very well with the 2.12 µm (SEM 1.14 µm) reported in Table 3. Finally, Reinstein et al.16 recently provided an overview of the repeatability of various biometric devices based on information provided to them directly by the devices’ manufacturers, along with data found in the literature. Although that particular study did not provide an overall estimate of the repeatability, much of its data was included in the analysis of Table 1. In clinical practice these results mean that one should always avoid mixing devices in the follow-up of individual patients to prevent inter-device differences from obfuscating any real evolutions in pathology. For clinical studies equipment differences between participating centers could potentially influence the outcome. But while it is undoubtedly less complicated for clinical 8
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
study planners to only include centers that operate one particular ‘preferred’ device, it would prevent many centers from making an otherwise important contribution. Especially for studies on conditions with a relatively low prevalence, this may not be the optimal approach. It could therefore be more advantageous to include all interested centers and consider centers (rather than equipment) as a between-subjects factor in the statistical analysis, thus correcting for potential equipment and calibration bias.
9
ACCEPTED MANUSCRIPT
Acknowledgements
AC C
EP
TE D
M AN U
SC
RI PT
a. Funding/Support: none. b. Financial disclosures: None of the authors had financial relations with commercial companies, either as employee, consultant or in an advisory capacity. However, unrelated to current manuscript, Oculus GmbH (manufacturer of the Pentacam) recently acquired reference data from our department for use in their software. c. Contributions of authors: design of the study (JR, KW); conduct of the study (JR, KW); collection of the data (JR), management of the data (JR), analysis of the data (JR, KW, DM), and interpretation of the data (JR, KW, DM); preparation, review, or approval of the manuscript (JR, KW, DM, MJT). d. Other acknowledgments: none.
10
ACCEPTED MANUSCRIPT
References
7. 8. 9. 10. 11.
12.
13. 14. 15.
16.
RI PT
6.
SC
5.
M AN U
4.
TE D
3.
EP
2.
Altman DG, Bland JM. Measurement in medicine: the analysis of method comparison studies. Statistician 1983;32(3):307-317. McAlinden C, Khadka J, Pesudovs K. Statistical methods for conducting agreement (comparison of clinical tests) and precision (repeatability or reproducibility) studies in optometry and ophthalmology. Ophthalmic Physiol Opt 2011;31(4):330-338. Bullimore MA, Buehren T, Bissmann W. Agreement between a partial coherence interferometer and 2 manual keratometers. J Cataract Refract Surg 2013;39(10):1550-1560. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta-analysis: John Wiley & Sons, 2011. Westlake WJ. Symmetrical confidence intervals for bioequivalence trials. Biometrics 1976;32(4):741-744. Schuirmann DJ. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. J Pharmacokinet Biopharm 1987;15(6):657-680. Retzlaff JA, Sanders DR, Kraff MC. Development of the SRK/T intraocular lens implant power calculation formula. J Cataract Refract Surg 1990;16(3):333-340. Haigis W. The Haigis formula. In: Shammas HJ, editor. Intraocular lens calculation. Throfare, NJ: Slack, 2004:41-57. Gullstrand A. Appendix II and IV In: von Helmholtz H, editor. Handbuch der Physiologischen Optik. 3rd ed. Hamburg: Voss, 1909:301-358, 382-415. Wallace BC, Dahabreh IJ, Trikalinos TA, Lau J, Trow P, Schmid CH. Closing the gap between methodologists and end-users: R as a computational back-end. J Stat Software 2012;49(5):1-15 Kawamorita T, Uozato H, Kamiya K, et al. Repeatability, reproducibility, and agreement characteristics of rotating Scheimpflug photography and scanning-slit corneal topography for corneal power measurement. J Cataract Refract Surg 2009;35(1):127-133. Tajbakhsh Z, Salouti R, Nowroozzadeh MH, Aghazadeh-Amiri M, Tabatabaee S, Zamani M. Comparison of keratometry measurements using the Pentacam HR, the Orbscan IIz, and the TMS-4 topographer. Ophthalmic Physiol Opt 2012;32(6):539-546. Hashemi H, Mehravaran S. Corneal changes after laser refractive surgery for myopia: comparison of Orbscan II and Pentacam findings. J Cataract Refract Surg 2007;33(5):841-847. Doughty MJ, Jonuscheit S. The orbscan acoustic (correction) factor for central corneal thickness measures of normal human corneas. Eye Contact lens 2010;36(2):106-115. Wu W, Wang Y, Xu L. Meta-analysis of Pentacam vs. ultrasound pachymetry in central corneal thickness measurement in normal, post–LASIK or PRK, and keratoconic or keratoconus-suspect eyes. Graefe's Archive for Clin Exp Ophthalmol 2014;252(1):91-99. Reinstein DZ, Gobbe M, Archer TJ. Anterior segment biometry: a study and review of resolution and repeatability data. J Refract Surg 2012;28(7):509-520.
AC C
1.
11
ACCEPTED MANUSCRIPT
Figure captions
AC C
EP
TE D
M AN U
SC
RI PT
Figure: Flow diagram describing the selection of the articles used in the calculation of the measurement errors of the biometry devices (left) and the meta-analysis on agreement between biometry devices (right).
12
ACCEPTED MANUSCRIPT
Table 1: Thresholds to assess clinical equivalence of measurements provided by ophthalmic biometry devices, defined by effect on calculated parameters
0.093b
CCT µm
8.12
ACD mm
L mm
0.085
0.046
0.122 0.244
– –
0.095 0.192
– –
– –
– –
– –
– –
0.300 0.600
RI PT
Kp,m D
0.046 0.093
0. 042 0.082 0.037 0.074
SC
Parameter Ka,m Unit D Measurement error Emeas Emeasa 0.152b Change in calculated refraction ΔScalc 0.125D 0.122 0.250D 0.244 Change in IOL power (SRK/T)c 0.125D 0.118 0.250D 0.235 Change in IOL power (Haigis)‡ 0.125D 0.087 0.250D 0.174
AC C
EP
TE D
M AN U
Ka,m Ka,s, Ka,f: mean, steep and flat anterior keratometry; Kp,m Kp,s, Kp,f: mean, steep and flat posterior keratometry; CCT: central corneal thickness; ACD: anterior chamber depth; L: axial length; Emeas: total measurement error; ΔScalc: change in calculated refraction; IOL: intraocular lens; SRK/T: Sanders-Retzlaff-Kraff Theoretical IOL calculation formula. aDerived from the measurement errors in Table 2, averaged over all devices except those designated outliers in the text. bFor steep and flat axes different values were used, determined in the same manner (Ka,s: 0.185D; Ka,f: 0.178D; Kp,s: 0.062D; Kp,f: 0.062D). c Using IOL constants of the Alcon AcrySof SA60AT IOL (A = 118.4; Haigis a0 = −0.111, a1 = 0.249, a2 = 0.179), targeting emmetropia (Available at http://www.augenklinik.uni-wuerzburg.de/ulib/c1.htm, accessed August 8, 2014).
ACCEPTED MANUSCRIPT
Table 2: Estimation of the measurement errors of various commercial biometry devices, derived from previously published repeatability and reproducibility data
0.20 0.22 0.15 0.21 0.19 0.14 0.18
AC C
EP
TE D
0.22 0.19 0.13 0.19 0.17 0.10 0.24 0.05 0.06 0.17 0.07 0.06 -
Emeas 0.06 0.05 -
RI PT
0.18 0.12 0.15 0.52 0.14 0.16 -
Erepeat Ereprod (# studies) (# studies) Post. flat keratometry Kp,f (D) Pentacam 0.04 (3) 0.05 (4) Galilei 0.03 (1) 0.04 (1) Sirius 0.04 (1) Central corneal thickness CCT (µm) Pentacam 4.56 (19) 5.38 (11) Galilei 1.96 (10) 1.76 (3) Sirius 2.92 (7) 3.75 (1) Orbscan 5.60 (16) 8.57 (5) Ultrasound 4.66 (31) 8.68 (15) 5.23 (5) 15.87 (1) Artemis Visante 3.89 (18) 6.86 (10) RT-Vue 2.42 (14) 2.56 (6) SL-OCT 3.92 (3) 7.50 (1) Lenstar 3.77 (9) 7.09 (5) OA-1000 2.18 (1) Spec Micr 4.36 (7) 9.24 (4) Anterior chamber depth ACD (mm)a Pentacam 0.03 (9) 0.03 (5) 0.01 (5) 0.04 (2) Galilei Sirius 0.02 (4) Orbscan 0.03 (3) 0.04 (1) Ultrasound 0.10 (7) 0.23 (1) Artemis 0.02 (2) Visante 0.03 (4) 0.04 (3) SL-OCT 0.01 (2) IOL Master 0.05 (5) 0.04 (2) Lenstar 0.03 (8) 0.08 (2) OA-1000 0.09 (1) Aladdin 0.06 (1) Axial length L (mm) IOL Master 0.02 (5) 0.02 (3) Lenstar 0.02 (6) 0.02 (2) 0.08 (1) OA-1000 Aladdin 0.03 (1) Ultrasound 0.11 (2) Device
SC
Emeas
M AN U
Erepeat Ereprod (# studies) (# studies) Ant. mean keratometry Ka,m (D) Pentacam 0.09 (8) 0.15 (7) Galilei 0.10 (3) 0.07 (1) Sirius 0.12 (4) 0.09 (1) Orbscan 0.33 (1) 0.40 (1) Placido 0.10 (12) 0.10 (10) IOL Master 0.14 (5) 0.07 (2) Lenstar 0.10 (1) Aladdin 0.09 (1) Ant. steep keratometry Ka,s (D) Pentacam 0.12 (10) 0.17 (9) Galilei 0.21 (2) 0.06 (2) Sirius 0.11 (3) 0.10 (1) Orbscan 0.17 (2) 0.12 (2) Placido 0.14 (12) 0.13 (9) IOL Master 0.10 (3) 0.10 (2) Lenstar 0.14 (4) 0.12 (3) Ant. flat keratometry Ka,f (D) Pentacam 0.12 (10) 0.19 (9) Galilei 0.17 (2) 0.09 (2) Sirius 0.09 (3) 0.09 (1) Orbscan 0.15 (2) 0.12 (2) Placido 0.13 (12) 0.11 (9) IOL Master 0.07 (3) 0.08 (2) Lenstar 0.11 (4) 0.22 (2) Post. mean keratometry Kp,m (D) Pentacam 0.03 (4) 0.04 (5) Galilei 0.04 (4) 0.04 (2) Sirius 0.02 (1) Orbscan 0.13 (2) 0.11 (2) Post. steep keratometry Kp,s (D) Pentacam 0.04 (3) 0.06 (4) Galilei 0.04 (1) 0.04 (1) Sirius 0.04 (1) Device
7.05 2.64 4.76 10.24 9.85 16.71 7.88 3.52 8.46 8.03 10.21 0.05 0.05 0.05 0.25 0.05 0.06 0.09 0.03 0.02 -
The full version of this table, along with the references to the studies used, is available as supplemental material at AJO.com (Supplemental Table 1). Erepeat: repeatability; Ereprod: reproducibility; Emeas: total measurement error; Ka,m Ka,s, Ka,f: mean, steep and flat anterior keratometry; Kp,m Kp,s, Kp,f: mean, steep and flat posterior keratometry; CCT: central corneal thickness; ACD: anterior chamber depth; L: axial length; IOL: intraocular lens; SL-OCT: slit lamp optical coherence tomographer. a Includes studies that define ACD as the distance between the corneal epithelium and anterior lens, as well as those that define it as the distance between corneal endothelium and anterior lens.
ACCEPTED MANUSCRIPT
Table 3: Meta-analyses of the difference in biometric values provided by various ophthalmic devices compared to the Pentacam (Pentacam minus other device) Metaa
-0.22 ± 0.19 -0.09 ± 0.16 -0.23 ± 0.05 -0.16 ± 0.28 -0.03 ± 0.04 0.39 ± 0.20 -0.23 ± 0.08
99% 100% 99% 98% 97%
0.247 0.586
S0002-9394(14)00495-4
DOI:
10.1016/j.ajo.2014.08.014
Reference:
AJOPHT 9027
To appear in:
American Journal of Ophthalmology
Received Date: 14 March 2014 Revised Date:
8 August 2014
Accepted Date: 10 August 2014
Please cite this article as: Rozema JJ, Wouters K, Mathysen DG, Tassignon M-J, Overview of the repeatability, reproducibility and agreement of the biometry values provided by various ophthalmic devices, American Journal of Ophthalmology (2014), doi: 10.1016/j.ajo.2014.08.014. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Abstract
AC C
EP
TE D
M AN U
SC
RI PT
Purpose: To present an overview of the measurement errors for various biometric devices, as well as a meta-analysis of the agreement between biometric devices using the Pentacam, Orbscan and IOL Master as a reference. Design: Meta-analysis of the literature Methods: The meta-analysis is based on data from 216 articles that compare a total of 24 different devices with the reference devices for the following 9 parameters: mean, steep and flat curvature of the anterior and posterior cornea, central corneal thickness, anterior chamber depth and axial length. After the weighted average difference between devices have been determined, the two one-sided t test was used to test for equivalence between devices within certain thresholds defined by the measurement errors and the influence of these differences on the calculated refraction. Results: In only 17 of the 70 comparisons a device was equivalent with the reference device within the thresholds set by the measurement error. More lenient thresholds, based on a change in calculated refraction of ±0.25D, increased this number to a maximum of 25 / 50 comparisons (excluding pachymetry). High degrees of inconsistency were seen in the reported results, which could partially explain the low agreement between devices. Conclusion: As a rule biometry measurements taken by different devices should not be considered equivalent, although several exceptions could be identified. We therefore recommend that clinical studies involving multiple device types treat this as a within-subject variable to avoid bias. The follow-up of individual patients using different devices should be avoided at all times.
ACCEPTED MANUSCRIPT
Overview of the repeatability, reproducibility and agreement of the biometry values provided by various ophthalmic devices
RI PT
Jos J. Rozema,*† Kristien Wouters,‡ † Danny GP Mathysen,*† Marie-José Tassignon*† *
Dept of Ophthalmology, Antwerp University Hospital, Edegem, Belgium Dept of Medicine and Health Science, University of Antwerp, Wilrijk, Belgium ‡ Dept of Scientific Coordination and Biostatistics, Antwerp University Hospital, Edegem, Belgium †
Jos Rozema Department of Ophthalmology, Antwerp University Hospital, Wilrijkstraat 10, 2650 Edegem, Belgium + 32 3 821 48 15 + 32 3 825 19 26 [email protected]
TE D
Tel: Fax: Email:
M AN U
SC
Corresponding author:
Key words Ocular biometry; device equivalence; Pentacam; Orbscan; IOL Master
AC C
EP
Short title Meta-analysis of biometry provided by different devices
Supplemental Material available at AJO.com
1
ACCEPTED MANUSCRIPT
Introduction
AC C
Methods
EP
TE D
M AN U
SC
RI PT
Reliable biometry is an essential part of any ophthalmic practice. For this reason a wide range of instruments have been put on the market, each purporting to have the highest degree of accuracy. But although these devices have undoubtedly been developed and tested according to the state of the art, they will all suffer from errors and sensitivities intrinsic to the measurement process. These present themselves as variations between consecutive acquisitions, and between acquisitions made by different observers or devices. For individual clinics this large variety in available equipment is inconsequential, as they usually own only a few instruments operated consistently by a limited number of experienced individuals. This ensures that the best possible biometry is obtained in the clinical follow-up of each patient. But for research projects, often involving any number of clinics working together, each with their own choice of biometry devices, these equipment differences may form an important confounding factor to the study design. Typically this problem is handled in two ways: either only clinics that own a particular ‘preferred’ device are selected for participation, or equipment differences are simply ignored. Both approaches have their disadvantages, though, as selection may exclude clinics that could otherwise make interesting contributions to the study, and ignoring the differences may introduce systematic errors into the analysis. Hence, before ignoring equipment differences and inserting measurements from different devices into a single database, it is imperative that their agreement and measurement errors have been validated statistically. But although this has already been done quite extensively in the literature for many parameters and many combinations of devices, the results in these reports are often contradictory to each other and authors have very different opinions on what may be considered an acceptable error. It is therefore the aim of this work to present an overview of the measurement errors of the biometry devices available in the literature, as well as of the agreement between these devices and three of the most commonly used clinical instruments: the Pentacam Scheimpflug camera (Oculus Optikgeräte, Wetzlar, Germany) and Orbscan scanning slit corneal topographer (Bausch & Lomb, Bridgewater, NJ) for the anterior segment biometry, and the IOL Master (Carl Zeiss, Jena, Germany) for axial length measurements. Any systematic differences identified will then be compared to a number of clinical standards, in order to verify which of these differences may have an actual clinical impact.
Literature search
The literature search was performed in two parts: first a search for reports on the measurement errors of biometric devices, followed by a second, independent search for reports comparing the measurements of biometry devices with those of the reference instruments. Given the number of parameters and number of devices involved in this work, the literature search was performed through combinations of keywords. For the study on the measurement errors these keywords were structured as “[Device] [parameter] [repeatability/ reproducibility/ measurement error]”, with [Device] the name of the biometry device and [Parameter] the parameter of interest. For the meta-analysis of the
2
ACCEPTED MANUSCRIPT
TE D
M AN U
SC
RI PT
comparison between devices the structure of the keywords was “[Device 1] [Device 2] [Parameter]”, with [Device 1] either Pentacam, OrbScan or IOL Master, and [Device 2] the name of another device (e.g. Sirius, Galilei,…) or a generic technological name (e.g. Placido, ultrasound,…). The literature search was performed in June 2014 using PubMed and Google Scholar. It included both English and non-English references published after the year 2000, thus avoiding results obtained using outdated versions of the devices or their software. Since biometric devices are typically calibrated using eye models that are close to the epidemiological average (e.g. the Gullstrand eye model), only articles using healthy adult eyes were considered in this analysis. The only exception were papers comparing corneal or anterior chamber parameters in cataract patients, as these measurements are not influenced by the condition of the lens. For the comparison of axial length measurements, however, cataract patients were excluded. Finally a selection was made based on the way the remaining articles presented their data. For the study on measurement errors only those reporting the standard deviation of the repeated measurements, or parameters that can be calculated back to the standard deviation were included. For the meta-analysis of the comparison both the average and the standard deviation of the difference between devices (or the 95% limits of agreement) had to be reported, along the lines suggested by Altman and Bland,1 and McAlinden et al.2 Note that for the Orbscan a distinction was made between comparative articles that use the “acoustic factor” (AF) to report the central corneal thickness and articles that do not. This AF typically has a value of 0.92 (although different values are used as well) and aims to correct for an overestimation of the corneal thickness by the Orbscan. Also, for the anterior chamber depth (ACD) there were articles that included the cornea into the measurement value and other articles that did. However these articles were not considered separately due to an insufficient number of articles to warrant a separate analysis.
Measurement error
AC C
EP
The measurement error Emeas may be defined by Emeas = (E2repeat + E2reprod)1/2, which is comprised of the repeatability Erepeat, the standard deviation of multiple measurements performed by a single operator, and the reproducibility Ereprod, the standard deviation between measurements performed at different sessions or by different operators. Although in principle there could also be calibration differences between devices of the same type, we were unable to include this factor into our analysis as very little information on this type of error is available.3 In order to obtain a global estimate of the repeatability and reproducibility of the various biometry devices, the numerical values found in the literature review were tabulated (made available as supplemental data at AJO.com; Supplemental Table 1), from which the average of the repeatability and reproducibility, weighted by number of subjects used in each study, was calculated for each available device. From these values the measurement error Emeas was then estimated using the above equation.
Meta-analysis Based on the reported differences found in the literature a series of meta-analyses were performed per parameter and per device pair (i.e. Devices 1 and 2) using the procedures 3
ACCEPTED MANUSCRIPT
RI PT
described by Borenstein et al.4 This provides a global average and standard deviation of the difference between devices, a p value indicating its statistical significance and a coefficient I² to quantify the homogeneity between articles (with values of 0% indicating low homogeneity and 100% high inhomogeneity). The Bonferroni correction is used to adjust for multiple testing. Note that a difference that is not statistically significant from zero does not automatically imply both devices are to be considered equivalent, as the range of the differences may still be wider than what is considered clinically acceptable. For this reason equivalence margins were defined based on either the devices’ measurement error, or several common clinical calculations. These margins were used to test for equivalence by means of a “two one-side t test” (TOST).5, 6
SC
Analysis of significant differences
EP
TE D
M AN U
Although the measurement error of a biometric device (Emeas) may be considered as the lowest possible threshold for clinical importance, providing a reliable and well-founded set of equivalence margins, it is probably too strict for most clinical applications. This means that even though a difference between two devices may be considered too large according the Emeas margins, it may still be clinically inconsequential. For this reason several other, more lenient equivalence margins were defined based on the influence measurement differences would have on typical ophthalmic and optometric calculations (Table 1). These calculations included the refraction Scalc calculated from the ocular biometry using the thick lens formula, and the intraocular lens (IOL) powers calculated using the SRK/T,7 and Haigis8 formulas. Setting the tolerance for the changes in calculated power (∆Scalc) to 0.125D and 0.25D, corresponding with 50% and 100% of the smallest step in spectacle correction and the measurement error on most autorefractometers, we then determined the respective biometric changes that would cause just such a change in the calculations. These changes were then used as clinically based equivalence margins. The biometric parameters required for the calculations were taken from the Gullstrand eye model,9 while the IOL constants were those of the Alcon AcrySof SA60AT spherical, monofocal IOL.
Software
Results
AC C
All calculations were performed using Excel 2010 (Microsoft Corp, Redmond, WA, USA) and OpenMeta[Analyst],10 which is an R based open software package for meta-analysis (available at http://www.cebm.brown.edu/open_meta, accessed August 8, 2014).
Literature search After the selection process detailed in the Figure the literature search resulted in 124 articles for the measurement error analysis, and 127 for the comparison between the reference instruments (Oculus Pentacam, Bausch & Lomb Orbscan and Zeiss IOL Master) and a wide range of
4
ACCEPTED MANUSCRIPT
RI PT
methods, including Scheimpflug, optical coherence tomography (OCT), Placido topography, partial coherence interferometry, immersion ultrasound and applanation ultrasound devices. Note that due to the wide variety of Placido and ultrasound based devices reported in the literature the generic name of the technology was used rather than actual brand names. Also, for a number of less commonly used devices (e.g. Zeiss AC Master and slit lamp with Jaeger attachment) data was not included in the tables presented here. Instead, this information is made available as supplemental data at AJO.com (Supplemental Tables 1−4)
Overview of measurement errors
TE D
M AN U
SC
The combined measurement errors for the mean anterior keratometry Ka,m show a narrow range between 0.12 D (Galilei) and 0.18 D (Pentacam), with one outlier at 0.52 D (Orbscan). This is also seen for the mean posterior keratometry Kp,m, with values of 0.05 D (Pentacam and Galilei), and an outlier at 0.17 D (Orbscan). For the steep and flat keratometry data (Ka,s, Ka,f) the range lies between 0.10 D and 0.24 D, with the lowest error values for the Sirius and the IOL Master. Although from these data it would seem that the Orbscan gives considerably higher values for the mean anterior and posterior keratometry than the other devices, it is important to note that both deviating values were taken from the same study.11 Comparing these Orbscan values to those for Ka,s and Ka,f, which are derived from other studies and lie within the range of the other devices, it is likely that these outliers are coincidental in nature. The measurement error of the central corneal thickness CCT ranged between 1.76 µm (Galilei) and 16.71 µm (Artemis). Anterior chamber depth ACD had errors ranging between 0.05 mm and 0.09 mm, with an outlier at 0.25 mm (ultrasound), and a similar result was seen for the axial length L with error values around 0.02 mm. The repeatability of the ultrasound and the Tomey OA-1000 (0.11 mm and 0.08 mm, respectively) was higher than those of the IOL Master and the Lenstar (0.02 mm).
Meta-analysis of agreement between devices
AC C
EP
A total of 70 comparisons between devices were performed, excluding 7 duplicate comparisons between Pentacam and Orbscan (Tables 3−5). The average difference between devices was found to be significant for 17 comparisons at a significance level of p < 1 − 0.05/73 = 0.00068 (Bonferroni correction against α−inflation), which was in 9 / 17 cases based on data from a single study. As shown for the Pentacam in Table 3 these differences were mostly found for posterior keratometry (Kp,m: Pentacam vs. Sirius and TMS-5; Kp,s: Pentacam vs.Galilei), CCT (Pentacam vs. TMS-5, Orbscan with acoustic factor, Visante/ Stratus, SL-OCT and specular microscopy) and ACD (Pentacam vs. Visante/ Stratus, SL-OCT, Casia SS-1000 and Lenstar). The Orbscan showed fewer significant differences than the Pentacam (Table 4, most notably for Ka,s (Orbscan vs. Galilei), CCT (Orbscan with acoustic factor vs. Pentacam, Orbscan without acoustic factor vs. ultrasound) and ACD (Orbscan vs. Galilei). Finally, the IOL Master only showed a significant difference for the axial length with the Nidek AL-scan (Table 5). The meta-analysis also indicated that the homogeneity between studies is remarkably low in the 47 comparisons involving two or more articles, with only 5 / 47 I²-coefficients below 50% and 34 / 47 coefficients above 95%. This is also seen in the Forest plots for these comparisons (available as a supplemental file at AJO.com) and the often wide range of reported differences in 5
ACCEPTED MANUSCRIPT
the included articles (available as a supplemental data at AJO.com, Supplemental Tables 2−4). For this reason the results of this meta-analysis should be interpreted with caution.
Equivalence of results and clinical equivalence
AC C
EP
TE D
M AN U
SC
RI PT
The significant differences found in the meta-analysis cannot be used as an indication of (non)equivalence of devices, nor does it indicate the clinical relevance of the differences. For this reason a TOST procedure was performed using the thresholds of clinical equivalence in Table 1, which yielded the results presented in the last three columns of Tables 3−5. Using the thresholds defined by the measurement errors the TOST procedure demonstrated equivalence for only 17 / 70 comparisons (again, excluding duplicate comparisons between Pentacam and Orbscan). The thresholds based on a refraction change ΔScalc within ±0.125D demonstrated equivalence for 9 / 50 comparisons, which improved to 25 / 50 for a more lenient ΔScalc within ±0.25D. The thresholds for SRK/T IOL power change within ±0.125D equivalence was seen in only 3 / 28 comparisons, and in 10 / 28 comparisons for a power change within ±0.25D. For the Haigis IOL power these numbers were 15 / 44 and 22 / 44, respectively. Considering the most lenient thresholds in Table 1 (i.e. refraction change ΔScalc within ±0.25D) the Pentacam was found equivalent with Placido for the anterior keratometry (Ka,m, Ka,s, Ka,f) and ultrasound for CCT. Similarly the Pentacam was equivalent for some keratometry parameters with the Galilei (Ka,s, Kp,m, Kp,s, Kp,f), and for two parameters of the Sirius (Ka,f, ACD). The Pentacam and Orbscan were not found to be equivalent with each other for any the parameters considered. The Orbscan was however equivalent with the Galilei for the anterior keratometry (Ka,s and Ka,f). Finally the IOL Master was equivalent with the Lenstar, AL-Scan and the Aladdin for the axial length. For CCT an equivalence could only be found between ultrasound and the Pentacam at the threshold defined by the CCT measurement error (±8.12 µm). Even if this tolerance were doubled to ±16.24 µm, which would be considered clinically unacceptable for e.g. the follow-up of keratoconus, the number of equivalences would only marginally improve to 5 / 20 comparisons (Pentacam vs. Sirius, ultrasound and Lenstar, and Orbscan vs. ultrasound (with AF) and Artemis).
6
ACCEPTED MANUSCRIPT
Discussion
M AN U
SC
RI PT
This work demonstrates that, as a rule, measurements taken by different biometry devices should not be considered equivalent. Instead one should be aware that there may be differences between devices that can be detected through meta-analyses, or inconsistent variations between devices that may be picked up by the TOST procedure. Especially the latter is of great importance when one wishes to demonstrate the equivalence of two devices. Usually this is done by showing that the 95% CI of the average difference remains entirely within a predefined set of clinical thresholds representing the maximal discrepancy one is willing to accept (Table 1). Looking at the equivalence of the various devices with the Pentacam at the tolerance level of ±0.25D refractive change (last column of Table 3) it is seen that the Pentacam is equivalent with Placido for the anterior keratometry and with ultrasound for CCT, which are by many considered as the gold standard devices for these parameters. Given that these particular results are based on a large number of studies (15 and 35, respectively), this would suggest that the Pentacam conforms to these gold standards as well. The Galilei is equivalent with the Pentacam for four keratometric parameters, while the Sirius is equivalent with the Pentacam for all parameters except Ka,m and Ka,s. For the Orbscan only an equivalence with the Galilei was found for the keratometry. For the axial length the IOL Master is equivalent with the Lenstar, AL-Scan and Aladdin, but not to contact ultrasound.
AC C
EP
TE D
There are however a number of limitations to the analyses above, most important of which is the very high inconsistency in reported values (inhomogeneity I² in Tables 3−5). One typical example is the difference in Ka,m measurements reported by the Pentacam and the Orbscan, which differs by +0.27D according one study12 and by −0.69D according to another study.13 Although both studies clearly indicate a large and statistically significant difference between devices, their weighted average difference was not significantly different from zero due to their opposite signs and relative cohort sizes. Such large inhomogeneities makes that the results above are difficult to interpret and should be considered with caution. An important source of between-study inconsistencies is calibration differences in devices of the same type. Although at the time of production the manufacturer calibrated all devices according to the same standards, this calibration is not always preserved once the device is placed in an everyday clinical setting. As in such a setting devices may be knocked accidentally, moved around, or used in conditions different from what is described in the instruction manual, the actual calibration could drift over time from the original factory setting. This issue is however not limited to the comparison of devices, but also applies to every study that uses two or more devices of the same type, or studies that use the same machine for a long period of time. For this reason it would be recommended that device calibration is verified prior to and during any multicenter study. Alternatively, individual centers could be considered as a between-subjects factor in the statistical analysis, even if they have devices of the same type. A second source of inconsistency in the literature may be sign errors in the cited articles. Even though great care was taken to avoid these by comparing tables, graphs and text of the original manuscripts, and double checking articles with strongly deviating values, it is conceivable that a small number of sign errors may have gone unnoticed.
7
ACCEPTED MANUSCRIPT
M AN U
SC
RI PT
Another aspect to this study was that it aimed to find the agreement between the reference devices (Pentacam, Orbscan and IOL Master) and a wide range of other devices. This may however not be considered as a means to demonstrate the equivalence of two other devices that were each found equivalent with the reference devices. Similarly, in the situation where one device is e.g. equivalent with the Pentacam and another one is not, this cannot automatically lead to the conclusion that both devices are not equivalent. In both cases the conclusion can only be derived from a direct comparison between those specific devices, which in many cases can be found in the literature. It is also important to note that the clinical threshold values calculated in Table 1 only estimate how a change in one single parameter influences the calculated refraction. In a real-life setting, however, each of the parameters used in the calculation would have an influence on the refraction, which leads to a larger, compounded error. Through error propagation, we found that the combined error is about 73% higher than the threshold, meaning that if all thresholds were chosen to accept a refractive change of ±0.25D, this would correspond with a compounded error of ±0.434D. A similar increase was seen for the ±0.125D threshold. Finally, as all biometric devices make assumptions regarding intraocular distances and refractive indices based on idealized models of healthy adult eyes, only this type of eyes were included in the analysis. As these assumed parameter values of healthy adults may deviate considerably from those of children or pathological subjects it is conceivable that devices might react differently to eyes that deviate too far from the original assumptions. This could theoretically cause two devices to produce different results in pathological or children’s eyes, even if the results are in agreement for healthy adults. This could be an interesting subject for a follow-up study.
AC C
EP
TE D
This work is not the first attempt to provide an overview of the measurement errors and agreement between biometry devices in the literature. Earlier Doughty and Jonuscheit14 performed an analysis on 46 studies comparing average pachymetry values provided by contact ultrasound and the Orbscan with the aim to assess whether the use of the acoustic factor is warranted. They reported a very large variation between studies, and concluded that this variation essentially invalidates the use of the acoustic factor. As is seen in Table 4 we also found a large variation, but also a clear difference between studies that used the acoustic factor and studies that did not (-4.56 µm, SEM 1.71 µm and 30.22 µm, SEM 5.81 µm, respectively). This inconsistency between this work and that of Doughty and Jonuscheit may be due to methodical differences (analysis of mean values vs. a meta-analysis and TOST). Wu et al. also performed a meta-analysis15 using 19 studies to compare pachymetry values measured by Pentacam and ultrasound in normal eyes, and came to the conclusion that measurements of both devices were very similar. Their reported difference of 1.47 µm (SEM 1.93 µm) corresponds very well with the 2.12 µm (SEM 1.14 µm) reported in Table 3. Finally, Reinstein et al.16 recently provided an overview of the repeatability of various biometric devices based on information provided to them directly by the devices’ manufacturers, along with data found in the literature. Although that particular study did not provide an overall estimate of the repeatability, much of its data was included in the analysis of Table 1. In clinical practice these results mean that one should always avoid mixing devices in the follow-up of individual patients to prevent inter-device differences from obfuscating any real evolutions in pathology. For clinical studies equipment differences between participating centers could potentially influence the outcome. But while it is undoubtedly less complicated for clinical 8
ACCEPTED MANUSCRIPT
AC C
EP
TE D
M AN U
SC
RI PT
study planners to only include centers that operate one particular ‘preferred’ device, it would prevent many centers from making an otherwise important contribution. Especially for studies on conditions with a relatively low prevalence, this may not be the optimal approach. It could therefore be more advantageous to include all interested centers and consider centers (rather than equipment) as a between-subjects factor in the statistical analysis, thus correcting for potential equipment and calibration bias.
9
ACCEPTED MANUSCRIPT
Acknowledgements
AC C
EP
TE D
M AN U
SC
RI PT
a. Funding/Support: none. b. Financial disclosures: None of the authors had financial relations with commercial companies, either as employee, consultant or in an advisory capacity. However, unrelated to current manuscript, Oculus GmbH (manufacturer of the Pentacam) recently acquired reference data from our department for use in their software. c. Contributions of authors: design of the study (JR, KW); conduct of the study (JR, KW); collection of the data (JR), management of the data (JR), analysis of the data (JR, KW, DM), and interpretation of the data (JR, KW, DM); preparation, review, or approval of the manuscript (JR, KW, DM, MJT). d. Other acknowledgments: none.
10
ACCEPTED MANUSCRIPT
References
7. 8. 9. 10. 11.
12.
13. 14. 15.
16.
RI PT
6.
SC
5.
M AN U
4.
TE D
3.
EP
2.
Altman DG, Bland JM. Measurement in medicine: the analysis of method comparison studies. Statistician 1983;32(3):307-317. McAlinden C, Khadka J, Pesudovs K. Statistical methods for conducting agreement (comparison of clinical tests) and precision (repeatability or reproducibility) studies in optometry and ophthalmology. Ophthalmic Physiol Opt 2011;31(4):330-338. Bullimore MA, Buehren T, Bissmann W. Agreement between a partial coherence interferometer and 2 manual keratometers. J Cataract Refract Surg 2013;39(10):1550-1560. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta-analysis: John Wiley & Sons, 2011. Westlake WJ. Symmetrical confidence intervals for bioequivalence trials. Biometrics 1976;32(4):741-744. Schuirmann DJ. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. J Pharmacokinet Biopharm 1987;15(6):657-680. Retzlaff JA, Sanders DR, Kraff MC. Development of the SRK/T intraocular lens implant power calculation formula. J Cataract Refract Surg 1990;16(3):333-340. Haigis W. The Haigis formula. In: Shammas HJ, editor. Intraocular lens calculation. Throfare, NJ: Slack, 2004:41-57. Gullstrand A. Appendix II and IV In: von Helmholtz H, editor. Handbuch der Physiologischen Optik. 3rd ed. Hamburg: Voss, 1909:301-358, 382-415. Wallace BC, Dahabreh IJ, Trikalinos TA, Lau J, Trow P, Schmid CH. Closing the gap between methodologists and end-users: R as a computational back-end. J Stat Software 2012;49(5):1-15 Kawamorita T, Uozato H, Kamiya K, et al. Repeatability, reproducibility, and agreement characteristics of rotating Scheimpflug photography and scanning-slit corneal topography for corneal power measurement. J Cataract Refract Surg 2009;35(1):127-133. Tajbakhsh Z, Salouti R, Nowroozzadeh MH, Aghazadeh-Amiri M, Tabatabaee S, Zamani M. Comparison of keratometry measurements using the Pentacam HR, the Orbscan IIz, and the TMS-4 topographer. Ophthalmic Physiol Opt 2012;32(6):539-546. Hashemi H, Mehravaran S. Corneal changes after laser refractive surgery for myopia: comparison of Orbscan II and Pentacam findings. J Cataract Refract Surg 2007;33(5):841-847. Doughty MJ, Jonuscheit S. The orbscan acoustic (correction) factor for central corneal thickness measures of normal human corneas. Eye Contact lens 2010;36(2):106-115. Wu W, Wang Y, Xu L. Meta-analysis of Pentacam vs. ultrasound pachymetry in central corneal thickness measurement in normal, post–LASIK or PRK, and keratoconic or keratoconus-suspect eyes. Graefe's Archive for Clin Exp Ophthalmol 2014;252(1):91-99. Reinstein DZ, Gobbe M, Archer TJ. Anterior segment biometry: a study and review of resolution and repeatability data. J Refract Surg 2012;28(7):509-520.
AC C
1.
11
ACCEPTED MANUSCRIPT
Figure captions
AC C
EP
TE D
M AN U
SC
RI PT
Figure: Flow diagram describing the selection of the articles used in the calculation of the measurement errors of the biometry devices (left) and the meta-analysis on agreement between biometry devices (right).
12
ACCEPTED MANUSCRIPT
Table 1: Thresholds to assess clinical equivalence of measurements provided by ophthalmic biometry devices, defined by effect on calculated parameters
0.093b
CCT µm
8.12
ACD mm
L mm
0.085
0.046
0.122 0.244
– –
0.095 0.192
– –
– –
– –
– –
– –
0.300 0.600
RI PT
Kp,m D
0.046 0.093
0. 042 0.082 0.037 0.074
SC
Parameter Ka,m Unit D Measurement error Emeas Emeasa 0.152b Change in calculated refraction ΔScalc 0.125D 0.122 0.250D 0.244 Change in IOL power (SRK/T)c 0.125D 0.118 0.250D 0.235 Change in IOL power (Haigis)‡ 0.125D 0.087 0.250D 0.174
AC C
EP
TE D
M AN U
Ka,m Ka,s, Ka,f: mean, steep and flat anterior keratometry; Kp,m Kp,s, Kp,f: mean, steep and flat posterior keratometry; CCT: central corneal thickness; ACD: anterior chamber depth; L: axial length; Emeas: total measurement error; ΔScalc: change in calculated refraction; IOL: intraocular lens; SRK/T: Sanders-Retzlaff-Kraff Theoretical IOL calculation formula. aDerived from the measurement errors in Table 2, averaged over all devices except those designated outliers in the text. bFor steep and flat axes different values were used, determined in the same manner (Ka,s: 0.185D; Ka,f: 0.178D; Kp,s: 0.062D; Kp,f: 0.062D). c Using IOL constants of the Alcon AcrySof SA60AT IOL (A = 118.4; Haigis a0 = −0.111, a1 = 0.249, a2 = 0.179), targeting emmetropia (Available at http://www.augenklinik.uni-wuerzburg.de/ulib/c1.htm, accessed August 8, 2014).
ACCEPTED MANUSCRIPT
Table 2: Estimation of the measurement errors of various commercial biometry devices, derived from previously published repeatability and reproducibility data
0.20 0.22 0.15 0.21 0.19 0.14 0.18
AC C
EP
TE D
0.22 0.19 0.13 0.19 0.17 0.10 0.24 0.05 0.06 0.17 0.07 0.06 -
Emeas 0.06 0.05 -
RI PT
0.18 0.12 0.15 0.52 0.14 0.16 -
Erepeat Ereprod (# studies) (# studies) Post. flat keratometry Kp,f (D) Pentacam 0.04 (3) 0.05 (4) Galilei 0.03 (1) 0.04 (1) Sirius 0.04 (1) Central corneal thickness CCT (µm) Pentacam 4.56 (19) 5.38 (11) Galilei 1.96 (10) 1.76 (3) Sirius 2.92 (7) 3.75 (1) Orbscan 5.60 (16) 8.57 (5) Ultrasound 4.66 (31) 8.68 (15) 5.23 (5) 15.87 (1) Artemis Visante 3.89 (18) 6.86 (10) RT-Vue 2.42 (14) 2.56 (6) SL-OCT 3.92 (3) 7.50 (1) Lenstar 3.77 (9) 7.09 (5) OA-1000 2.18 (1) Spec Micr 4.36 (7) 9.24 (4) Anterior chamber depth ACD (mm)a Pentacam 0.03 (9) 0.03 (5) 0.01 (5) 0.04 (2) Galilei Sirius 0.02 (4) Orbscan 0.03 (3) 0.04 (1) Ultrasound 0.10 (7) 0.23 (1) Artemis 0.02 (2) Visante 0.03 (4) 0.04 (3) SL-OCT 0.01 (2) IOL Master 0.05 (5) 0.04 (2) Lenstar 0.03 (8) 0.08 (2) OA-1000 0.09 (1) Aladdin 0.06 (1) Axial length L (mm) IOL Master 0.02 (5) 0.02 (3) Lenstar 0.02 (6) 0.02 (2) 0.08 (1) OA-1000 Aladdin 0.03 (1) Ultrasound 0.11 (2) Device
SC
Emeas
M AN U
Erepeat Ereprod (# studies) (# studies) Ant. mean keratometry Ka,m (D) Pentacam 0.09 (8) 0.15 (7) Galilei 0.10 (3) 0.07 (1) Sirius 0.12 (4) 0.09 (1) Orbscan 0.33 (1) 0.40 (1) Placido 0.10 (12) 0.10 (10) IOL Master 0.14 (5) 0.07 (2) Lenstar 0.10 (1) Aladdin 0.09 (1) Ant. steep keratometry Ka,s (D) Pentacam 0.12 (10) 0.17 (9) Galilei 0.21 (2) 0.06 (2) Sirius 0.11 (3) 0.10 (1) Orbscan 0.17 (2) 0.12 (2) Placido 0.14 (12) 0.13 (9) IOL Master 0.10 (3) 0.10 (2) Lenstar 0.14 (4) 0.12 (3) Ant. flat keratometry Ka,f (D) Pentacam 0.12 (10) 0.19 (9) Galilei 0.17 (2) 0.09 (2) Sirius 0.09 (3) 0.09 (1) Orbscan 0.15 (2) 0.12 (2) Placido 0.13 (12) 0.11 (9) IOL Master 0.07 (3) 0.08 (2) Lenstar 0.11 (4) 0.22 (2) Post. mean keratometry Kp,m (D) Pentacam 0.03 (4) 0.04 (5) Galilei 0.04 (4) 0.04 (2) Sirius 0.02 (1) Orbscan 0.13 (2) 0.11 (2) Post. steep keratometry Kp,s (D) Pentacam 0.04 (3) 0.06 (4) Galilei 0.04 (1) 0.04 (1) Sirius 0.04 (1) Device
7.05 2.64 4.76 10.24 9.85 16.71 7.88 3.52 8.46 8.03 10.21 0.05 0.05 0.05 0.25 0.05 0.06 0.09 0.03 0.02 -
The full version of this table, along with the references to the studies used, is available as supplemental material at AJO.com (Supplemental Table 1). Erepeat: repeatability; Ereprod: reproducibility; Emeas: total measurement error; Ka,m Ka,s, Ka,f: mean, steep and flat anterior keratometry; Kp,m Kp,s, Kp,f: mean, steep and flat posterior keratometry; CCT: central corneal thickness; ACD: anterior chamber depth; L: axial length; IOL: intraocular lens; SL-OCT: slit lamp optical coherence tomographer. a Includes studies that define ACD as the distance between the corneal epithelium and anterior lens, as well as those that define it as the distance between corneal endothelium and anterior lens.
ACCEPTED MANUSCRIPT
Table 3: Meta-analyses of the difference in biometric values provided by various ophthalmic devices compared to the Pentacam (Pentacam minus other device) Metaa
-0.22 ± 0.19 -0.09 ± 0.16 -0.23 ± 0.05 -0.16 ± 0.28 -0.03 ± 0.04 0.39 ± 0.20 -0.23 ± 0.08
99% 100% 99% 98% 97%
0.247 0.586
