Acta Obstet Gynec Scand 54: 41-44, 1975
OBSERVER ERROR I N ULTRASONIC FETAL CEPHALOMETRY Per Bergsjn. Trygve Bakke and Lois Salamonsen
Absrract. Although ultrasonic fetal cephalometry gives
quite accurate results, a certain degree of observer error must be expected. In a series of duplicate measurements, the magnitude of this error has been assessed. The observer error was lowest when the fetal head was partly engaged in the pelvic inlet, when the midline echo was clearly observed, and when an obstetrician with experience in ultrasonography performed the examinations. The observer error should be taken into account when growth indices are evaluated in the individual patient.
Ultrasonic measurement of the biparietal diameter gives very accurate results. By measuring newborn children Willocks et al. (6) found that the values differed less than 1 mm from those obtained by caliper i n 73.5% of the cases. It is more difficult to test the accuracy of measurements on the fetus, because the biparietal diameter changes rapidly following vaginal delivery (6). Donald (5) stated that in the early years of fetal cephalometry accuracies to within 2 mm were confidently expected. In 1968 Campbell ( I ) introduced an improved method of fetal ultrasonic cephalometry combining one-dirnensional A-scans with two-dimensional B-scans. By comparing the results with caliper measurements following elective caesarean section, in which cases no moulding of the head takes place, he found less than 2 mm discrepancy in 94% and less than I mm in 60% of the cases. Measurements performed long before the child is born cannot be controlled by other procedures, but indirect evidence, such as the shape and standard deviation of the growth curves (2), indicates that these are also accurate. As in every physical measurement a source of error is introduced on the part of the examiner. The most reliable results are obviously obtained when every examination is performed by one ex4 -752861
pert examiner. which wiis stated to be the case in two reports ( 2 . 3). However. ultrasonic fetal cephalometry is a fairly simple procedure which in most cases can be done by a trained technician. When two or more. examiners use the same apparatus an estimate of the precision of the measurements can be obtained. We have not found any mention of the magnitude of the observer error in previously published reports, and we therefore decided to determine this error by letting two examiners independently perform fetal cephalometry on the same patient. MATERIAL AND METHODS One of us (L. S.) is a scientist with no previous experience in clinical medicine, working temporarily in the ultrasonic unit of the maternity clinic in Bergen. while P. B. and T. B. are obstetricians with some experience in ultrasonic fetal cephalometry. After a few weeks of training L. S. started to do a series of biparietal diameter measurements, each of which was repeated by either P. B. or T. B. The second examiner did not witness the first examination, and was not informed of the results before his own had been written down. If the case history was known to one of the examiners the other was also informed about the gestational age as well a s the results of possible previous measurements. Altogether 144 duplicate measurements were attempted, but 4 had to be discarded, as one or both of the examiners could not localise the biparietal diameter. The remaining 140 were divided into 7 consecutive series of 20 comparisons each. This subdivision will be further explained under “results”. The observer differences are expressed in millimeters, and the error of measurements is expressed as the standard deviations of the differences,
The apparatus used was made by Kretz-Technik, Zipf, Austria (Series 4100 MG). The two-dimensional B-scan Acta Obstet Gynec Scand 54 (1975)
42
P . Borgsj0 PI (11.
ERROR OF MEASUREMENT d = diff between two examiners
Exam 1
-
2N-2 (mn)
20
10
21 -40
I*
41
81
61 -80
- 60 214
81 -100 8'
m
101 -120
n
-120
PB v s L 5 TB vs L S PB v s 1 B Head engaged Head nd engaged Midlne e c b +
Fig. I . Mean observer errors in duplicate measurements
of the fetal biparietal diameter. Further explanations see text. was used for orientation. while the actual measurement was done on the A-scan beam. with the help of a millimeter scale on the oscilloscope screen. As a rule the measurements were performed from transversal B-scan sections. If the right plane for measurement was not found readily. the apparatus was moved, and a longitudinal scan obtained. The time taken for the single examination varied. with means of 4.9 min (L. S . ) , 3.8 min (P. B . ) and 3.5 min (T. B.).
RESULTS
each obstetrician (P. B. and T. B.) tested separately against those of L. S., the errors being of the same magnitude, 1.46 mm and 1.64 mm respectively. The next bar represents a series of 20 duplicate measurements, which were all performed by P. B. and T. B. This error, 1.17 mm, is low, which indicates that accuracy of measurements will increase with increasing experience. The following two bars show that the error was about twice as high when the fetal head was not engaged as when it was engaged (2.08 mm against 1.12 mm). Engagement of the fetal head is here used in its widest sense, meaning that the head is partly fixed in the pelvic inlet but that the biparietal diameter is still above the linea terminalis, permitting the measurement to be made. Engagement of the fetal head facilitated the measurements greatly. since head movements were then slight or absent during the examinations. On.the other hand. freely moving heads made the examinations more difficult, since they might disappear completely from the field of examination during adjustment of the apparatus. particularly i n breech presentation, hence the large error. The bottom bar shows that when both observers clearly observed the midline echo, the error was low (1.27 mm). The next question was whether the observer differences were due to a systematic error on the part of one examiner. The distribution shown in Fig. 2 represents the differences between L. S.'s measurements and those of P. B. and T. B. when the differences were signed as positive or negative. Statistical analysis showed that the mean of this distribution did not differ significantly from 0. The same holds true when the differences between
The results are shown graphically in Fig. I. The horizontal bars show the standard deviations of the differences of duplicate measurements. The upper bar, representing the first 20 examinations, shows a fairly high degree of observer error. This Table I. Distribution of positive and negative differences in 100 consecutive duplicate measurerepresents the end of L. S.'s training period. The ments of the fetal biparietal diameter (examinafollowing five series of 20 consecutive examina- tions 21-120) tions each (examinations no. 21-120), representing L. s. vs. L. s. vs. 71 women with gestational age ranging from 183P. B. T. B. Total 308 days, is thought to give a fair impression of the true observer error. This error ranges from No. of measurements 0.86 to 2.14 mm, and shows no clear trend with with positive difference 22 14 36 time. The over-all error of these last 100 measureNo. of measurements ments was 1.52 mm, wich means that 95% of the with negative dif10 28 ference 18 differences between two independent observer measurements were below 3.04 mm and 68% below No. of measurements 14 36 with equal results 22 1.52 mm. The next two'bars show the results for Acta Obstet Gynec Scand 54 (1975)
Fig. 2. Distribution of positive and negative differences in 62 duplicate cephalometric examinations performed by L. S. and P. B.. and 38 performed by L. S. and Difference between two EMrniners (rnrn)
L. S. and P. B. and between L. S . and T. B.. were analysed separately (Table I). Positive and negative differences were about equally distributed in both instances. It may be concluded that the observer error occurred at random, and that it was not caused by a systematic interpretation error on the part of one observer. DISCUSSION The observer error in fetal cephalometry may be due to several factors. The relative importance of each factor may vary from case to case. Firstly, the design of the apparatus, particularly its movability and the mode of obtaining the correct measurement on the oscilloscope is of some importance. The apparatus we used was easy to angulate, but difficult to move from a transverse to a longitudinal position. The actual measurement was read on a millimeter scale by counting. A built-in caliper system with display of the measurement by numerals might have been an improvement. Secondly, the position and the movability of the fetal head is of great importance, which is evident from the results. With a freely movable fetal head, particularly in breech presentation, it was often difficult to obtain the correct angulation of the transducer, and the fetal head might move while the final adjustments were made. In this investigation each examiner worked alone. The help of an assistant might have improved the accuracy of the measurements. Thirdly, the ability of the examiner is an important point. The examiner must be thoroughly familiar with the apparatus, but it is
T. B .
doubtful whether an important gain in diagnostic accuracy can be expected with increasing technical experience once a certain level is reached. Our results showed no clear improvement in observer error with time. On the other hand, experience in obstetrical examinations should ensure less observer error. Obviously, the maximum accuracy and the least observer error will be achieved if the examiner always adheres strictly to the rules laid down by Campbell ( I ) . but everyone who has tried the method knows that technical difficulties do arise from time to time, in which cases a greater observer error must be expected. I t is possible that the magnitude of observer error which we have reported here, will seem unacceptably high to other workers in the field. However, every ultrasonic unit should have an estimate of its own observer error, just as every laboratory should know the precision of its own analyses. Awareness of the observer error in ultrasonic fetal cephalometry is very important when growth indices are assessed. In cases of suspected intrauterine growth retardation the growth index has been shown to be a better prognostic sign than the urinary estrogen assays (4).To minimize the observer error in such cases the biparietal diameter should be measured frequently, and ideally by two examiners each time.
REFERENCES I . Campbell, S.: An improved method of fetal cephalometry by ultrasound. J Obstet Gynaecol Br Comm 75: 568-576, 1 M . Acfa Obsfef Gynec S c a d 54 (1975)
2. Campbell. S.: The prediction of fetal maturity by ultrasonic measurement of the biparietal diameter. J Obstet Gynaecol Br Comm 76: 603-609, 1969. 3. Campbell. S. & Dewhurst. C. J.: Diagnosis of the small-for-dates fetus by serial ultrasonic cephalometry. Lancet 2: 1002-1006. 1971. 4. Campbell. S . & Kurjak. A,: Comparison between urinary oestrogen assay and serial ultrasonic cephalometry in assessment of fetal growth retardation. Br Med J 4: 336350, 1972. 5. Davidson, J . M., Lind, T., Farr, V. & Whittingham, T. A.: The limitations of ultrasonic fetal cephalometry. J Obstet Gynecol Br Comm 80:769, 1973.
Acta Obstet Gynec S c a d 54 (1975)
6. Donald, I.: Ultrasonics in diagnosis (Sonar). Proc Roy SOCMed62: 442446. 1969. 7. Willocks, J., Donald, I., Duggan, T. C. & Day, N.: Foetal cephalometry by ultrasound. J Obstet Gynaecol Br Comm 71: 11-20, 1964. Siihm ittrd f b r pic bliccition Sept . 17. 1973.
Per Bergsje Department of Obstetrics and Gynecology Akershus Central Hospital 1474 Nordbyhagen Norway
OBSERVER ERROR I N ULTRASONIC FETAL CEPHALOMETRY Per Bergsjn. Trygve Bakke and Lois Salamonsen
Absrract. Although ultrasonic fetal cephalometry gives
quite accurate results, a certain degree of observer error must be expected. In a series of duplicate measurements, the magnitude of this error has been assessed. The observer error was lowest when the fetal head was partly engaged in the pelvic inlet, when the midline echo was clearly observed, and when an obstetrician with experience in ultrasonography performed the examinations. The observer error should be taken into account when growth indices are evaluated in the individual patient.
Ultrasonic measurement of the biparietal diameter gives very accurate results. By measuring newborn children Willocks et al. (6) found that the values differed less than 1 mm from those obtained by caliper i n 73.5% of the cases. It is more difficult to test the accuracy of measurements on the fetus, because the biparietal diameter changes rapidly following vaginal delivery (6). Donald (5) stated that in the early years of fetal cephalometry accuracies to within 2 mm were confidently expected. In 1968 Campbell ( I ) introduced an improved method of fetal ultrasonic cephalometry combining one-dirnensional A-scans with two-dimensional B-scans. By comparing the results with caliper measurements following elective caesarean section, in which cases no moulding of the head takes place, he found less than 2 mm discrepancy in 94% and less than I mm in 60% of the cases. Measurements performed long before the child is born cannot be controlled by other procedures, but indirect evidence, such as the shape and standard deviation of the growth curves (2), indicates that these are also accurate. As in every physical measurement a source of error is introduced on the part of the examiner. The most reliable results are obviously obtained when every examination is performed by one ex4 -752861
pert examiner. which wiis stated to be the case in two reports ( 2 . 3). However. ultrasonic fetal cephalometry is a fairly simple procedure which in most cases can be done by a trained technician. When two or more. examiners use the same apparatus an estimate of the precision of the measurements can be obtained. We have not found any mention of the magnitude of the observer error in previously published reports, and we therefore decided to determine this error by letting two examiners independently perform fetal cephalometry on the same patient. MATERIAL AND METHODS One of us (L. S.) is a scientist with no previous experience in clinical medicine, working temporarily in the ultrasonic unit of the maternity clinic in Bergen. while P. B. and T. B. are obstetricians with some experience in ultrasonic fetal cephalometry. After a few weeks of training L. S. started to do a series of biparietal diameter measurements, each of which was repeated by either P. B. or T. B. The second examiner did not witness the first examination, and was not informed of the results before his own had been written down. If the case history was known to one of the examiners the other was also informed about the gestational age as well a s the results of possible previous measurements. Altogether 144 duplicate measurements were attempted, but 4 had to be discarded, as one or both of the examiners could not localise the biparietal diameter. The remaining 140 were divided into 7 consecutive series of 20 comparisons each. This subdivision will be further explained under “results”. The observer differences are expressed in millimeters, and the error of measurements is expressed as the standard deviations of the differences,
The apparatus used was made by Kretz-Technik, Zipf, Austria (Series 4100 MG). The two-dimensional B-scan Acta Obstet Gynec Scand 54 (1975)
42
P . Borgsj0 PI (11.
ERROR OF MEASUREMENT d = diff between two examiners
Exam 1
-
2N-2 (mn)
20
10
21 -40
I*
41
81
61 -80
- 60 214
81 -100 8'
m
101 -120
n
-120
PB v s L 5 TB vs L S PB v s 1 B Head engaged Head nd engaged Midlne e c b +
Fig. I . Mean observer errors in duplicate measurements
of the fetal biparietal diameter. Further explanations see text. was used for orientation. while the actual measurement was done on the A-scan beam. with the help of a millimeter scale on the oscilloscope screen. As a rule the measurements were performed from transversal B-scan sections. If the right plane for measurement was not found readily. the apparatus was moved, and a longitudinal scan obtained. The time taken for the single examination varied. with means of 4.9 min (L. S . ) , 3.8 min (P. B . ) and 3.5 min (T. B.).
RESULTS
each obstetrician (P. B. and T. B.) tested separately against those of L. S., the errors being of the same magnitude, 1.46 mm and 1.64 mm respectively. The next bar represents a series of 20 duplicate measurements, which were all performed by P. B. and T. B. This error, 1.17 mm, is low, which indicates that accuracy of measurements will increase with increasing experience. The following two bars show that the error was about twice as high when the fetal head was not engaged as when it was engaged (2.08 mm against 1.12 mm). Engagement of the fetal head is here used in its widest sense, meaning that the head is partly fixed in the pelvic inlet but that the biparietal diameter is still above the linea terminalis, permitting the measurement to be made. Engagement of the fetal head facilitated the measurements greatly. since head movements were then slight or absent during the examinations. On.the other hand. freely moving heads made the examinations more difficult, since they might disappear completely from the field of examination during adjustment of the apparatus. particularly i n breech presentation, hence the large error. The bottom bar shows that when both observers clearly observed the midline echo, the error was low (1.27 mm). The next question was whether the observer differences were due to a systematic error on the part of one examiner. The distribution shown in Fig. 2 represents the differences between L. S.'s measurements and those of P. B. and T. B. when the differences were signed as positive or negative. Statistical analysis showed that the mean of this distribution did not differ significantly from 0. The same holds true when the differences between
The results are shown graphically in Fig. I. The horizontal bars show the standard deviations of the differences of duplicate measurements. The upper bar, representing the first 20 examinations, shows a fairly high degree of observer error. This Table I. Distribution of positive and negative differences in 100 consecutive duplicate measurerepresents the end of L. S.'s training period. The ments of the fetal biparietal diameter (examinafollowing five series of 20 consecutive examina- tions 21-120) tions each (examinations no. 21-120), representing L. s. vs. L. s. vs. 71 women with gestational age ranging from 183P. B. T. B. Total 308 days, is thought to give a fair impression of the true observer error. This error ranges from No. of measurements 0.86 to 2.14 mm, and shows no clear trend with with positive difference 22 14 36 time. The over-all error of these last 100 measureNo. of measurements ments was 1.52 mm, wich means that 95% of the with negative dif10 28 ference 18 differences between two independent observer measurements were below 3.04 mm and 68% below No. of measurements 14 36 with equal results 22 1.52 mm. The next two'bars show the results for Acta Obstet Gynec Scand 54 (1975)
Fig. 2. Distribution of positive and negative differences in 62 duplicate cephalometric examinations performed by L. S. and P. B.. and 38 performed by L. S. and Difference between two EMrniners (rnrn)
L. S. and P. B. and between L. S . and T. B.. were analysed separately (Table I). Positive and negative differences were about equally distributed in both instances. It may be concluded that the observer error occurred at random, and that it was not caused by a systematic interpretation error on the part of one observer. DISCUSSION The observer error in fetal cephalometry may be due to several factors. The relative importance of each factor may vary from case to case. Firstly, the design of the apparatus, particularly its movability and the mode of obtaining the correct measurement on the oscilloscope is of some importance. The apparatus we used was easy to angulate, but difficult to move from a transverse to a longitudinal position. The actual measurement was read on a millimeter scale by counting. A built-in caliper system with display of the measurement by numerals might have been an improvement. Secondly, the position and the movability of the fetal head is of great importance, which is evident from the results. With a freely movable fetal head, particularly in breech presentation, it was often difficult to obtain the correct angulation of the transducer, and the fetal head might move while the final adjustments were made. In this investigation each examiner worked alone. The help of an assistant might have improved the accuracy of the measurements. Thirdly, the ability of the examiner is an important point. The examiner must be thoroughly familiar with the apparatus, but it is
T. B .
doubtful whether an important gain in diagnostic accuracy can be expected with increasing technical experience once a certain level is reached. Our results showed no clear improvement in observer error with time. On the other hand, experience in obstetrical examinations should ensure less observer error. Obviously, the maximum accuracy and the least observer error will be achieved if the examiner always adheres strictly to the rules laid down by Campbell ( I ) . but everyone who has tried the method knows that technical difficulties do arise from time to time, in which cases a greater observer error must be expected. I t is possible that the magnitude of observer error which we have reported here, will seem unacceptably high to other workers in the field. However, every ultrasonic unit should have an estimate of its own observer error, just as every laboratory should know the precision of its own analyses. Awareness of the observer error in ultrasonic fetal cephalometry is very important when growth indices are assessed. In cases of suspected intrauterine growth retardation the growth index has been shown to be a better prognostic sign than the urinary estrogen assays (4).To minimize the observer error in such cases the biparietal diameter should be measured frequently, and ideally by two examiners each time.
REFERENCES I . Campbell, S.: An improved method of fetal cephalometry by ultrasound. J Obstet Gynaecol Br Comm 75: 568-576, 1 M . Acfa Obsfef Gynec S c a d 54 (1975)
2. Campbell. S.: The prediction of fetal maturity by ultrasonic measurement of the biparietal diameter. J Obstet Gynaecol Br Comm 76: 603-609, 1969. 3. Campbell. S. & Dewhurst. C. J.: Diagnosis of the small-for-dates fetus by serial ultrasonic cephalometry. Lancet 2: 1002-1006. 1971. 4. Campbell. S . & Kurjak. A,: Comparison between urinary oestrogen assay and serial ultrasonic cephalometry in assessment of fetal growth retardation. Br Med J 4: 336350, 1972. 5. Davidson, J . M., Lind, T., Farr, V. & Whittingham, T. A.: The limitations of ultrasonic fetal cephalometry. J Obstet Gynecol Br Comm 80:769, 1973.
Acta Obstet Gynec S c a d 54 (1975)
6. Donald, I.: Ultrasonics in diagnosis (Sonar). Proc Roy SOCMed62: 442446. 1969. 7. Willocks, J., Donald, I., Duggan, T. C. & Day, N.: Foetal cephalometry by ultrasound. J Obstet Gynaecol Br Comm 71: 11-20, 1964. Siihm ittrd f b r pic bliccition Sept . 17. 1973.
Per Bergsje Department of Obstetrics and Gynecology Akershus Central Hospital 1474 Nordbyhagen Norway
